OS becomes slow while Load AVG is still normal and IOWAIT is alow below 1

HI

I have one Oracle 11g server (member of RAC) running over RHEL 5.  Its 16 processor and 204GB memory blade server and Fiber Channal SAN storage.  Please refer to the screenshot attached, load avg and other performance parameters are normal but still we find this server performance very slow.

If we are logged in as user root and run normal commands on shell like 'ls' , vim, cat, df etc.... we don't see any performance issue.

But if we are logged in as user 'oracle' we see that even command 'ls' takes 10-15 seconds to returnd output.  I ran 'top' command and it took about 20 seconds to show TOP screen as attached.

However there is not much load on server, still we see that if we are logged in as user 'oracle' , performance is really slow.

Can someone suggest why is it so happenign and whats the remedy ?

Regards

Ashwani Jain
TOP-Output.png
ashwanijain1981Asked:
Who is Participating?

Improve company productivity with a Business Account.Sign Up

x
 
johnsoneConnect With a Mentor Senior Oracle DBACommented:
Check your PATH environment variable.  I have seen this where one of the directories listed in the PATH was an NFS mount and the mount point was unavailable.  The issue is you are waiting for a timeout before the command can even run.
0
 
SandyCommented:
Check the open file limits/process limits for oracle user in /etc/sysctl.conf and also check for the server resource utilization reports e.g. MRTG's etc...

TY/SA
0
 
Tomas Helgi JohannssonCommented:
Hi!

These values needs to be set in the  "/etc/security/limits.conf" file.

    oracle              soft    nproc   2047
    oracle              hard    nproc   16384
    oracle              soft    nofile  4096
    oracle              hard    nofile  65536
    oracle              soft    stack   10240

And these values  in the "/etc/sysctl.conf" file.

    fs.suid_dumpable = 1
    fs.aio-max-nr = 1048576
    fs.file-max = 6815744
    kernel.shmall = 2097152
    kernel.shmmax = 536870912
    kernel.shmmni = 4096
    # semaphores: semmsl, semmns, semopm, semmni
    kernel.sem = 250 32000 100 128
    net.ipv4.ip_local_port_range = 9000 65500
    net.core.rmem_default=4194304
    net.core.rmem_max=4194304
    net.core.wmem_default=262144
    net.core.wmem_max=1048586


Regards,
    Tomas Helgi
0
Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

 
ashwanijain1981Author Commented:
HI

I had attached output of top command with original message.  Please find below the ulimit output:

[oracle@oracle_server01 ~]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1545926
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1545926
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
0
 
SandyCommented:
You need to tweak these settings with the recommendations from vendor as per your environment to get the best optimization.

TY/SA
0
 
ashwanijain1981Author Commented:
Here below is /etc/security/limits.conf:

oracle hard nofile 65536
oracle hard stack 32768
grid hard nofile 65536
grid hard stack 32768
0
 
ashwanijain1981Author Commented:
These have already been tweaked as per recommendations for Oracle RAC.  Do you se any paramerer that still requirs tweaking.
0
 
SandyCommented:
But this is not working for you, Collect the system reports and re-open the case with them to get it more analyzed and get deployed as per your application needs.

@Tomas: the parameters you had given are these covered under best practices ? if yes then Ashwani these can be used.

to be specific below ones specified by Tomas....


And these values  in the "/etc/sysctl.conf" file.

    fs.suid_dumpable = 1
    fs.aio-max-nr = 1048576
    fs.file-max = 6815744
    kernel.shmall = 2097152
    kernel.shmmax = 536870912
    kernel.shmmni = 4096
    # semaphores: semmsl, semmns, semopm, semmni
    kernel.sem = 250 32000 100 128
    net.ipv4.ip_local_port_range = 9000 65500
    net.core.rmem_default=4194304
    net.core.rmem_max=4194304
    net.core.wmem_default=262144
    net.core.wmem_max=1048586


TY/SA
0
 
Tomas Helgi JohannssonCommented:
Hi!

Yes, these are the recommended values provided by the Oracle install guide.
http://www.oracle.com/technetwork/articles/smiley-11gr1-install-092791.html

Regards,
    Tomas Helgi
0
 
ashwanijain1981Author Commented:
HI Tom

I only see below mentioned two parameters different from what you mentioned in sysctl.conf:

kernel.shmall = 53608931
kernel.shmmax = 109791090688

and below mentioned parameter does not exist in my sysctl.conf:

fs.suid_dumpable = 1

My server has 16 CPU and 204GB memory, do we still need to reduce these parameters ?
0
 
SandyCommented:
if the values are high i don't suggest to reduce..

and for fs.suid_dumpable ... you can refer to below link

http://www.cyberciti.biz/tips/tag/fs-suid_dumpable

TY/SA
0
 
ashwanijain1981Author Commented:
This was streight and accurate solution.  There was one NFS mount that had latency due to some network issue and also there was one non-existant directory, after removing both of these from PATH, this issue got resolved.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.