ashwanijain1981
asked on
OS becomes slow while Load AVG is still normal and IOWAIT is alow below 1
HI
I have one Oracle 11g server (member of RAC) running over RHEL 5. Its 16 processor and 204GB memory blade server and Fiber Channal SAN storage. Please refer to the screenshot attached, load avg and other performance parameters are normal but still we find this server performance very slow.
If we are logged in as user root and run normal commands on shell like 'ls' , vim, cat, df etc.... we don't see any performance issue.
But if we are logged in as user 'oracle' we see that even command 'ls' takes 10-15 seconds to returnd output. I ran 'top' command and it took about 20 seconds to show TOP screen as attached.
However there is not much load on server, still we see that if we are logged in as user 'oracle' , performance is really slow.
Can someone suggest why is it so happenign and whats the remedy ?
Regards
Ashwani Jain
TOP-Output.png
I have one Oracle 11g server (member of RAC) running over RHEL 5. Its 16 processor and 204GB memory blade server and Fiber Channal SAN storage. Please refer to the screenshot attached, load avg and other performance parameters are normal but still we find this server performance very slow.
If we are logged in as user root and run normal commands on shell like 'ls' , vim, cat, df etc.... we don't see any performance issue.
But if we are logged in as user 'oracle' we see that even command 'ls' takes 10-15 seconds to returnd output. I ran 'top' command and it took about 20 seconds to show TOP screen as attached.
However there is not much load on server, still we see that if we are logged in as user 'oracle' , performance is really slow.
Can someone suggest why is it so happenign and whats the remedy ?
Regards
Ashwani Jain
TOP-Output.png
Hi!
These values needs to be set in the "/etc/security/limits.conf
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 4096
oracle hard nofile 65536
oracle soft stack 10240
And these values in the "/etc/sysctl.conf" file.
fs.suid_dumpable = 1
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 536870912
kernel.shmmni = 4096
# semaphores: semmsl, semmns, semopm, semmni
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_ran
net.core.rmem_default=4194
net.core.rmem_max=4194304
net.core.wmem_default=2621
net.core.wmem_max=1048586
Regards,
Tomas Helgi
These values needs to be set in the "/etc/security/limits.conf
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 4096
oracle hard nofile 65536
oracle soft stack 10240
And these values in the "/etc/sysctl.conf" file.
fs.suid_dumpable = 1
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 536870912
kernel.shmmni = 4096
# semaphores: semmsl, semmns, semopm, semmni
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_ran
net.core.rmem_default=4194
net.core.rmem_max=4194304
net.core.wmem_default=2621
net.core.wmem_max=1048586
Regards,
Tomas Helgi
ASKER
HI
I had attached output of top command with original message. Please find below the ulimit output:
[oracle@oracle_server01 ~]$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1545926
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1545926
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
I had attached output of top command with original message. Please find below the ulimit output:
[oracle@oracle_server01 ~]$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1545926
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1545926
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
You need to tweak these settings with the recommendations from vendor as per your environment to get the best optimization.
TY/SA
TY/SA
ASKER
Here below is /etc/security/limits.conf:
oracle hard nofile 65536
oracle hard stack 32768
grid hard nofile 65536
grid hard stack 32768
oracle hard nofile 65536
oracle hard stack 32768
grid hard nofile 65536
grid hard stack 32768
ASKER
These have already been tweaked as per recommendations for Oracle RAC. Do you se any paramerer that still requirs tweaking.
But this is not working for you, Collect the system reports and re-open the case with them to get it more analyzed and get deployed as per your application needs.
@Tomas: the parameters you had given are these covered under best practices ? if yes then Ashwani these can be used.
to be specific below ones specified by Tomas....
And these values in the "/etc/sysctl.conf" file.
fs.suid_dumpable = 1
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 536870912
kernel.shmmni = 4096
# semaphores: semmsl, semmns, semopm, semmni
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_ran
net.core.rmem_default=4194
net.core.rmem_max=4194304
net.core.wmem_default=2621
net.core.wmem_max=1048586
TY/SA
@Tomas: the parameters you had given are these covered under best practices ? if yes then Ashwani these can be used.
to be specific below ones specified by Tomas....
And these values in the "/etc/sysctl.conf" file.
fs.suid_dumpable = 1
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 536870912
kernel.shmmni = 4096
# semaphores: semmsl, semmns, semopm, semmni
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_ran
net.core.rmem_default=4194
net.core.rmem_max=4194304
net.core.wmem_default=2621
net.core.wmem_max=1048586
TY/SA
Hi!
Yes, these are the recommended values provided by the Oracle install guide.
http://www.oracle.com/technetwork/articles/smiley-11gr1-install-092791.html
Regards,
Tomas Helgi
Yes, these are the recommended values provided by the Oracle install guide.
http://www.oracle.com/technetwork/articles/smiley-11gr1-install-092791.html
Regards,
Tomas Helgi
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
HI Tom
I only see below mentioned two parameters different from what you mentioned in sysctl.conf:
kernel.shmall = 53608931
kernel.shmmax = 109791090688
and below mentioned parameter does not exist in my sysctl.conf:
fs.suid_dumpable = 1
My server has 16 CPU and 204GB memory, do we still need to reduce these parameters ?
I only see below mentioned two parameters different from what you mentioned in sysctl.conf:
kernel.shmall = 53608931
kernel.shmmax = 109791090688
and below mentioned parameter does not exist in my sysctl.conf:
fs.suid_dumpable = 1
My server has 16 CPU and 204GB memory, do we still need to reduce these parameters ?
if the values are high i don't suggest to reduce..
and for fs.suid_dumpable ... you can refer to below link
http://www.cyberciti.biz/tips/tag/fs-suid_dumpable
TY/SA
and for fs.suid_dumpable ... you can refer to below link
http://www.cyberciti.biz/tips/tag/fs-suid_dumpable
TY/SA
ASKER
This was streight and accurate solution. There was one NFS mount that had latency due to some network issue and also there was one non-existant directory, after removing both of these from PATH, this issue got resolved.
TY/SA