[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

RHEL 6 and WMQ (Channels=1000)

Posted on 2011-04-25
24
Medium Priority
?
3,350 Views
Last Modified: 2013-11-17
Chapter 6 of the installation Guide contained in the Quick

Start Guide references the following link:

http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0m0/index.jsp

Which recommends max open files to be in excess of 1024,

and max processes to be 4096, amoung other kernal
recommendations.:

I have the following configurations on RHEL 6:

[root@WEST5 etc]# pwd
/etc
[root@WEST5 etc]# sysctl -p sysctl.conf
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
error: "net.bridge.bridge-nf-call-ip6tables" is an unknown 

key
error: "net.bridge.bridge-nf-call-iptables" is an unknown 

key
error: "net.bridge.bridge-nf-call-arptables" is an unknown 

key
kernel.msgmni = 1024
kernel.shmmni = 4096
kernel.shmall = 2097152
kernel.sem = 500 256000 250 1024
net.ipv4.tcp_keepalive_time = 300
fs.file-max = 32768
[root@WEST5 etc]# cd /proc/sys/kernel
[root@WEST5 kernel]# cat msgmni
1024
[root@WEST5 kernel]# cat shmmni
4096
[root@WEST5 kernel]# cat shmall
2097152
[root@WEST5 kernel]# cat sem
500     256000  250     1024
[root@WEST5 kernel]# cat file-max
cat: file-max: No such file or directory
[root@WEST5 kernel]# cd ..
[root@WEST5 sys]# cd fs
[root@WEST5 fs]# cat file-max
32768
[root@WEST5 fs]# ulimit -Ha
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1161386
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 32768
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1161386
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Open in new window


However, the error I am currently getting is:

| Build Type        :- IKAP - (Production)                  

                  |
| Effective UserID  :- 495 (mqm)                            

                  |
| Real UserID       :- UNKNOWN                              

                  |
| Program Name      :- amqzlaa0                             

                  |
| Addressing mode   :- 64-bit                               

                  |
| Process           :- 8562                                 

                  |
| Process(Thread)   :- 8562                                 

                  |
| Thread            :- 1                                    

                  |
| ThreadingModel    :- PosixThreads                         

                  |
| UserApp           :- FALSE                                

                  |
| Last HQC          :- 0.0.0-0                              

                  |
| Last HSHMEMB      :- 0.0.0-0                              

                  |
| Major Errorcode   :- xecF_E_UNEXPECTED_SYSTEM_RC          

                  |
| Minor Errorcode   :- OK                                   

                  |
| Probe Type        :- MSGAMQ6119                           

                  |
| Probe Severity    :- 2                                    

                  |
| Probe Description :- AMQ6119: An internal WebSphere MQ 

error has occurred   |
|   ('11 - Resource temporarily unavailable' from 

pthread_create.)            |
| FDCSequenceNumber :- 0                                    

                  |
| Arith1            :- 11 (0xb)                             

                  |
| Comment1          :- '11 - Resource temporarily 

unavailable' from           |
|   pthread_create.      

Open in new window

               


WHile there is no maximum number of threads per process in
Linux, there is a maximum number of threads:


[root@WEST5 ~]# cd /proc
[root@WEST5 proc]# cd sys
[root@WEST5 sys]# cd kernel
[root@WEST5 kernel]# cat threads-max
2322773

This seems will within the norm

The problems start occurring when I create a qmgr

(connected to a broker) with the following parameters:

Channels:
   MQIBindType=FASTPATH
   MaxChannels=1000
   MaxActiveChannels=1000

Open in new window


The large number of channels seems to thow it off.

This never happened with RHEL 5.x, only with RHEL 6.  RHEL
6 is known for having a very tight default configuration
setup.  I have examined all of the isntallation parameters
and they appear to be within the recommended configuration.
 
Any ideas ?
0
Comment
Question by:Los Angeles1
  • 12
  • 9
  • 2
  • +1
24 Comments
 
LVL 2

Expert Comment

by:advega
ID: 35462340
after the tunning of the parameters, did you restart the OS?
they will be effective until then... sorry, but I have to ask....
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 35462427
Infact you don't need to restart the system.

sysctl -p

will do. Will you post the output of the above command ? I think there's a problem with the contents of the file too. The problem is you did not ebnable IPV6 in your config but the sysctl file includes some presets ofr iptablesv6 which should be not installed. Also please post the actual log from the websphere.

BTW willyou post the output of getenforce command too ??

Cheers,
K.
0
 

Author Comment

by:Los Angeles1
ID: 35463545
[root@WEST5 log]# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key
error: "net.bridge.bridge-nf-call-iptables" is an unknown key
error: "net.bridge.bridge-nf-call-arptables" is an unknown key
kernel.msgmni = 1024
kernel.shmmni = 4096
kernel.shmall = 2097152
kernel.sem = 500 256000 250 1024
net.ipv4.tcp_keepalive_time = 300
fs.file-max = 32768
[root@WEST5 log]#
[root@WEST5 log]#
[root@WEST5 log]# getenforce
Enforcing
[root@WEST5 log]#

Open in new window

0
Transaction-level recovery for Oracle database

Veeam Explore for Oracle delivers low RTOs and RPOs with agentless transaction log backup and transaction-level recovery of Oracle databases. You can restore the database to a precise point in time, even to a specific transaction.

 
LVL 38

Expert Comment

by:wesly_chen
ID: 35463547
edit /etc/sysctl.conf
and comment out (put # in front the line
-------------
# net.bridge.bridge-nf-call-ip6tables = ...
# net.bridge.bridge-nf-call-iptables = ...
# net.bridge.bridge-nf-call-arptables = ...
-------
Then as root
# sysctl -p /etc/sysctl.conf


> AMQ6119: An internal WebSphere MQ error has occurred ('11 - Resource temporarily unavailable' from
Which Websphere MQ did you have?
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 35463575
Ok it seems that SELinux is getting in the way..  Will you check your /var/log/audit/audit.log for any AVC errors?

Or you might like to disable it till you get success with your MQ then we'll try to reenable it.

To disable execute:

setenforce Permissive
reboot

once you're sure that everything is OK then we'll reenable it with some polices added for it.

Cheers,
K.
0
 

Author Comment

by:Los Angeles1
ID: 35467999
I completely disabled SELinux, and rebooted.  Then I tried to start up WQM2, and had problems, so I tried to create a broker, as follows

bash-4.1$ mqsicreatebroker WQM3
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: Resource temporarily unavailable
bash-4.1$ ulimit -Ha
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1161386
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 32768
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Open in new window


bash-4.1$ pwd
/proc/sys/kernel
bash-4.1$ cat threads-max
2322773

Open in new window


So it looks like I ran out of processes (which is why it cant fork), even though I have hard limit of 4096.  I reported the max threads just in case the fork was using those.

Can it really be out of processes with a limit of 4096

Ulimit -Ha shows the limit of the processes, What command can I run to tell how many are currently taken up.

Should I increase the number of processes allowed over 4096 ?
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 35468234
Hi,

This error is interesting:

bash: fork: retry: Resource temporarily unavailable


You might be getting out of SWAP space. Will you post your df output too ??
0
 

Author Comment

by:Los Angeles1
ID: 35469444
[root@WEST5 security]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/vg_west5-lv_root
                      51606140  10653072  38331628  22% /
tmpfs                 74336116        88  74336028   1% /dev/shm
/dev/sda2               495844     60023    410221  13% /boot
/dev/sda1               204580       240    204340   1% /boot/efi
/dev/mapper/vg_west5-lv_home
                      21867732   1888692  18868208  10% /home

Open in new window

0
 

Author Comment

by:Los Angeles1
ID: 35469482
I am using the following version of WMQ

[root@WEST5 etc]#
[root@WEST5 etc]#
[root@WEST5 etc]# su mqm

MQSI 7.0.0.2
/opt/ibm/mqsi/7.0

bash-4.1$

Open in new window


I changed sysctl.conf as follows:

oot@WEST5 etc]# cat sysctl.conf
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and
# sysctl.conf(5) for more details.

# Controls IP packet forwarding
net.ipv4.ip_forward = 0

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1

# Disable netfilter on bridges.
#net.bridge.bridge-nf-call-ip6tables = 0
#net.bridge.bridge-nf-call-iptables = 0
#net.bridge.bridge-nf-call-arptables = 0

#  For MQSeries on Linux
kernel.msgmni = 1024
kernel.shmmni = 4096
kernel.shmall = 2097152
kernel.sem = 500 256000 250 1024
net.ipv4.tcp_keepalive_time = 300
fs.file-max = 32768
[root@WEST5 etc]#

Open in new window


and ran the command:

[root@WEST5 etc]# sysctl -p sysctl.conf
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmni = 1024
kernel.shmmni = 4096
kernel.shmall = 2097152
kernel.sem = 500 256000 250 1024
net.ipv4.tcp_keepalive_time = 300
fs.file-max = 32768

Open in new window


and set a huge no processes in limits.conf

mqm    soft     nofile   28866720
mqm    hard    nproc    1000000

Open in new window


but still et the following errors

bash-4.1$ mqsilist
JVMDUMP006I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" - please wait.
JVMDUMP032I JVM requested Snap dump using '/var/mqsi/common/errors/Snap.20110426.140148.7107.0001.trc' in response to an event
UTE001: Error starting trace thread for "Snap Dump Thread": -1
JVMDUMP010I Snap dump written to /var/mqsi/common/errors/Snap.20110426.140148.7107.0001.trc
JVMDUMP032I JVM requested Heap dump using '/var/mqsi/common/errors/heapdump.20110426.140148.7107.0002.phd' in response to an event
JVMDUMP010I Heap dump written to /var/mqsi/common/errors/heapdump.20110426.140148.7107.0002.phd
JVMDUMP032I JVM requested Java dump using '/var/mqsi/common/errors/javacore.20110426.140148.7107.0003.txt' in response to an event
JVMDUMP010I Java dump written to /var/mqsi/common/errors/javacore.20110426.140148.7107.0003.txt
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
BIP8075E: A Java exception was caught from class 'com/ibm/broker/config/util/MQSIList.main'. The text is 'java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 11'.
A Java exception was caught via the JNI.
Check preceding messages for related errors and retry the operation. If the exception still occurs, contact your IBM support center.

bash-4.1$ ulmit -Ha
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable

Open in new window


Am I running out of threads ?  memory ? etc ?
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 35470277
Your logs indicate:

JVMDUMP006I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" - please wait.

This is not an MQ error. You'd need to increase your java heap. The default is 768 Mb and you should increase it through WebSphere Manager..

Just to make sure will you post the output for the following commands:

vmstat
free -t -m
 
Cheers,
K.

0
 

Author Comment

by:Los Angeles1
ID: 35470322
bash-4.1$ vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0      0 142322240  39676 1015112    0    0     3     8   11   27  0  0 100  0  0
bash-4.1$ free -t -m
             total       used       free     shared    buffers     cached
Mem:        145187       6201     138986          0         38        991
-/+ buffers/cache:       5171     140016
Swap:        66411          0      66411
Total:      211599       6201     205398
bash-4.1$

Open in new window

0
 

Author Comment

by:Los Angeles1
ID: 35470353
How exactly can I increase the java heap size ?
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 35470541
You can configure the amount of memory available to the application server's JVM heap by setting the Initial Heap Size and Maximum Heap Size properties of the application server. For example, on the WebSphere Application Server administrative console panel: Servers > Application servers > server_name > Java and Process Management > Process Definition > Java Virtual Machine

Did you apply the latest FixPack I remember there was a PTF in the fixpack It addresess some OutOfMemory issues:

http://www-01.ibm.com/support/docview.wss?uid=swg1IZ60328

Cheers,
K.

0
 

Author Comment

by:Los Angeles1
ID: 35470563
Actually, this is the WebSphere Message Broker, Not the WebSphere Application Server, so it would have a different method to change the JVM
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 35470588
I know but it all runs over the WebSphere Application Server.
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 35470783
Here's the IBM documentation which suggests the same as I suggest for troubleshooting MQ problems:

http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm.websphere.express.doc/info/exp/ae/rmj_prob0.html

As I've told earlier WSMQ is built over WAS.

I'll strongly recomentd you to apply th suggested PTFs before going further. Also please note that Fxpacks include PTF addressing similar OurOfMemory conditions...
0
 

Author Comment

by:Los Angeles1
ID: 35470823
Actually WAS is on top of WMQ.   The possible configurations are:

WMQ alone
WAS alone
WAS plus WMQ
WMB plus WMQ

Unfortunately on my system I only have WMQ and WMB, which works well on RHEL 5.x, but I can not get this ocnfiguration to work on RHEL 6
0
 

Author Comment

by:Los Angeles1
ID: 35471167
I found the corresponding WMB command to check the heap size:

bash-4.1$ mqsilist
BIP1285I: Broker 'WQM1' on queue manager 'WQM1' is stopped.
BIP1285I: Broker 'WQM2' on queue manager 'WQM2' is stopped.
BIP8071I: Successful command completion.
bash-4.1$ mqsistart WQM2
WebSphere MQ queue manager 'WQM2' starting.
18 log records accessed on queue manager 'WQM2' during the log replay phase.
Log replay for queue manager 'WQM2' complete.
Transaction manager state recovered for queue manager 'WQM2'.
WebSphere MQ queue manager 'WQM2' started.
BIP8096I: Successful command initiation, check the system log to ensure that the component started without problem and that it continues to run without problem.
bash-4.1$ mqsireportproperties WQM2 -e e1 -o ComIbmJVMManager -a

ComIbmJVMManager
  uuid='ComIbmJVMManager'
  userTraceLevel='none'
  traceLevel='none'
  userTraceFilter='none'
  traceFilter='none'
  resourceStatsReportingOn='inactive'
  resourceStatsMeasurements='<ResourceStatsSwitches ResourceType="JVM" version='1'> <Measurement name="InitialMemoryInMB" collect="on" /> <Measurement name="UsedMemoryInMB" collect="on" /> <Measurement name="CommittedMemoryInMB" collect="on" /> <Measurement name="MaxMemoryInMB" collect="on" /> <Measurement name="CumulativeGCTimeInSeconds" collect="on" /> <Measurement name="CumulativeNumberOfGCCollections" collect="on" /> </ResourceStatsSwitches>'
  jvmVerboseOption='none'
  jvmDisableClassGC='false'
  jvmShareClasses='false'
  jvmNativeStackSize='-1'
  jvmJavaOSStackSize='-1'
  jvmMinHeapSize='33554432'
  jvmMaxHeapSize='-1'
  jvmDebugPort='0'
  jvmSystemProperty=''
  keystoreType=''
  keystoreFile=''
  keystorePass='********'
  truststoreType=''
  truststoreFile=''
  truststorePass='********'

BIP8071I: Successful command completion.

Open in new window


I get -1 on mqx heap size, so I assume it is unlimited

Any ideas what should I do from here ?
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 35471215
I'll suggest you to apply fixes before trying to play around..
0
 
LVL 38

Expert Comment

by:wesly_chen
ID: 35471244
I agree with KaremE.
The latest version of WMQ is 7.0.3.

RHEL 6 just came out and most of third party application are not fully tested on it.
Even though IBM website doesn't say what version/distribute of Linux they support,
it is highly recommended to upgrade to the latest version.
0
 

Author Comment

by:Los Angeles1
ID: 35473732
My problem is that the WMB is already at the highest level, and that is where the java out of memory errors are occuring

I will see if I can upgade the WMQ, but I think since RHEL 6 is a tighter configuration than RHEL 5.6, there must be some magic config parameter that must be set
0
 
LVL 30

Accepted Solution

by:
Kerem ERSOY earned 2000 total points
ID: 35473974
> I will see if I can upgade the WMQ, but I think since RHEL 6 is a tighter configuration than RHEL 5.6, > there must be some magic config parameter that must be set

RHEL is very new and I am not even sure the Java integration s at its full extent in x86_64. Apart from AIX I am not sure that 64 bit implementation of Java is so successful since it is also new.

Cheers,
K.
0
 

Author Comment

by:Los Angeles1
ID: 35474770
The java implementation is part of the WMB, as opposed to the WMQ, which is why I am not optimistic about upgrading the WMQ

The WMB is at its highest and most current level
0
 

Author Closing Comment

by:Los Angeles1
ID: 35477940
All advise was excellent, but I am switching focus at this time
0

Featured Post

Get free NFR key for Veeam Availability Suite 9.5

Veeam is happy to provide a free NFR license (1 year, 2 sockets) to all certified IT Pros. The license allows for the non-production use of Veeam Availability Suite v9.5 in your home lab, without any feature limitations. It works for both VMware and Hyper-V environments

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction We as admins face situation where we need to redirect websites to another. This may be required as a part of an upgrade keeping the old URL but website should be served from new URL. This document would brief you on different ways ca…
I. Introduction There's an interesting discussion going on now in an Experts Exchange Group — Attachments with no extension (http://www.experts-exchange.com/discussions/210281/Attachments-with-no-extension.html). This reminded me of questions tha…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
Suggested Courses
Course of the Month20 days, 3 hours left to enroll

873 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question