Solved

OS becomes slow while Load AVG is still normal and IOWAIT is alow below 1

Posted on 2014-02-17
12
352 Views
Last Modified: 2014-02-23
HI

I have one Oracle 11g server (member of RAC) running over RHEL 5.  Its 16 processor and 204GB memory blade server and Fiber Channal SAN storage.  Please refer to the screenshot attached, load avg and other performance parameters are normal but still we find this server performance very slow.

If we are logged in as user root and run normal commands on shell like 'ls' , vim, cat, df etc.... we don't see any performance issue.

But if we are logged in as user 'oracle' we see that even command 'ls' takes 10-15 seconds to returnd output.  I ran 'top' command and it took about 20 seconds to show TOP screen as attached.

However there is not much load on server, still we see that if we are logged in as user 'oracle' , performance is really slow.

Can someone suggest why is it so happenign and whats the remedy ?

Regards

Ashwani Jain
TOP-Output.png
0
Comment
Question by:ashwanijain1981
  • 5
  • 4
  • 2
  • +1
12 Comments
 
LVL 13

Expert Comment

by:Sandy
ID: 39864389
Check the open file limits/process limits for oracle user in /etc/sysctl.conf and also check for the server resource utilization reports e.g. MRTG's etc...

TY/SA
0
 
LVL 24

Expert Comment

by:Tomas Helgi Johannsson
ID: 39864418
Hi!

These values needs to be set in the  "/etc/security/limits.conf" file.

    oracle              soft    nproc   2047
    oracle              hard    nproc   16384
    oracle              soft    nofile  4096
    oracle              hard    nofile  65536
    oracle              soft    stack   10240

And these values  in the "/etc/sysctl.conf" file.

    fs.suid_dumpable = 1
    fs.aio-max-nr = 1048576
    fs.file-max = 6815744
    kernel.shmall = 2097152
    kernel.shmmax = 536870912
    kernel.shmmni = 4096
    # semaphores: semmsl, semmns, semopm, semmni
    kernel.sem = 250 32000 100 128
    net.ipv4.ip_local_port_range = 9000 65500
    net.core.rmem_default=4194304
    net.core.rmem_max=4194304
    net.core.wmem_default=262144
    net.core.wmem_max=1048586


Regards,
    Tomas Helgi
0
 

Author Comment

by:ashwanijain1981
ID: 39864421
HI

I had attached output of top command with original message.  Please find below the ulimit output:

[oracle@oracle_server01 ~]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1545926
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1545926
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
0
 
LVL 13

Expert Comment

by:Sandy
ID: 39864430
You need to tweak these settings with the recommendations from vendor as per your environment to get the best optimization.

TY/SA
0
 

Author Comment

by:ashwanijain1981
ID: 39864433
Here below is /etc/security/limits.conf:

oracle hard nofile 65536
oracle hard stack 32768
grid hard nofile 65536
grid hard stack 32768
0
 

Author Comment

by:ashwanijain1981
ID: 39864484
These have already been tweaked as per recommendations for Oracle RAC.  Do you se any paramerer that still requirs tweaking.
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 13

Expert Comment

by:Sandy
ID: 39864489
But this is not working for you, Collect the system reports and re-open the case with them to get it more analyzed and get deployed as per your application needs.

@Tomas: the parameters you had given are these covered under best practices ? if yes then Ashwani these can be used.

to be specific below ones specified by Tomas....


And these values  in the "/etc/sysctl.conf" file.

    fs.suid_dumpable = 1
    fs.aio-max-nr = 1048576
    fs.file-max = 6815744
    kernel.shmall = 2097152
    kernel.shmmax = 536870912
    kernel.shmmni = 4096
    # semaphores: semmsl, semmns, semopm, semmni
    kernel.sem = 250 32000 100 128
    net.ipv4.ip_local_port_range = 9000 65500
    net.core.rmem_default=4194304
    net.core.rmem_max=4194304
    net.core.wmem_default=262144
    net.core.wmem_max=1048586


TY/SA
0
 
LVL 24

Expert Comment

by:Tomas Helgi Johannsson
ID: 39864519
Hi!

Yes, these are the recommended values provided by the Oracle install guide.
http://www.oracle.com/technetwork/articles/smiley-11gr1-install-092791.html

Regards,
    Tomas Helgi
0
 
LVL 34

Accepted Solution

by:
johnsone earned 500 total points
ID: 39864531
Check your PATH environment variable.  I have seen this where one of the directories listed in the PATH was an NFS mount and the mount point was unavailable.  The issue is you are waiting for a timeout before the command can even run.
0
 

Author Comment

by:ashwanijain1981
ID: 39876091
HI Tom

I only see below mentioned two parameters different from what you mentioned in sysctl.conf:

kernel.shmall = 53608931
kernel.shmmax = 109791090688

and below mentioned parameter does not exist in my sysctl.conf:

fs.suid_dumpable = 1

My server has 16 CPU and 204GB memory, do we still need to reduce these parameters ?
0
 
LVL 13

Expert Comment

by:Sandy
ID: 39876102
if the values are high i don't suggest to reduce..

and for fs.suid_dumpable ... you can refer to below link

http://www.cyberciti.biz/tips/tag/fs-suid_dumpable

TY/SA
0
 

Author Closing Comment

by:ashwanijain1981
ID: 39881770
This was streight and accurate solution.  There was one NFS mount that had latency due to some network issue and also there was one non-existant directory, after removing both of these from PATH, this issue got resolved.
0

Featured Post

Comprehensive Backup Solutions for Microsoft

Acronis protects the complete Microsoft technology stack: Windows Server, Windows PC, laptop and Surface data; Microsoft business applications; Microsoft Hyper-V; Azure VMs; Microsoft Windows Server 2016; Microsoft Exchange 2016 and SQL Server 2016.

Join & Write a Comment

It’s 2016. Password authentication should be dead — or at least close to dying. But, unfortunately, it has not traversed Quagga stage yet. Using password authentication is like laundering hotel guest linens with a washboard — it’s Passé.
From implementing a password expiration date, to datatype conversions and file export options, these are some useful settings I've found in Jasper Server.
This video shows, step by step, how to configure Oracle Heterogeneous Services via the Generic Gateway Agent in order to make a connection from an Oracle session and access a remote SQL Server database table.
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now