Solved

Swap issue on RHEL servers

Posted on 2014-01-23
23
808 Views
Last Modified: 2014-02-19
Hi ,

I am facing a very strange issues on one of my RHEL server:

We have 2 servers having same configuration / hardware / application and traffic but on one server swap utilization is almost 50 % used and same shows as cached while 4GB memory still free ?

On another server 100% used and same shows as cached and same 4.5 GB free?

Please clear these doubts?

One more strange thing is this that these are VM's and on first server no actvity or swap usage shown ?


I have already activated ballooning mechanism but no luck.

please advice!
0
Comment
Question by:apunkabollywood
  • 11
  • 7
  • 5
23 Comments
 
LVL 34

Expert Comment

by:Seth Simmons
ID: 39805517
how much memory and swap space do these servers have?
what application(s) are running (web server, java, mysql, etc.)?
if there is that much swap space used then likely insufficient memory
you can also adjust swappiness behavior but still appears to me not enough physical memory
0
 

Author Comment

by:apunkabollywood
ID: 39805855
8GB RAM on both and 8GB Swap on both.

APache/Java/BMC patrol etc.

RAM utilisation is 50 % max always ...and what ur input on diffrent data shows on VMware side?
0
 
LVL 34

Expert Comment

by:Seth Simmons
ID: 39807444
what do you mean 'different data on vmware side'?
have you tried increasing the memory for the guests?
doing that and adjusting swappiness will help
0
 

Author Comment

by:apunkabollywood
ID: 39808279
Please have a look on my initial problem summary as i have described that on server side swap usage shows 50% and at same time on vmware monitoring side no swap usage was there ? but still thats a diffrent point still we are not able to find the reason why swap usage is high on one and low on other.
0
 
LVL 61

Expert Comment

by:gheist
ID: 39808531
vmware swapping is not related to linux swap files.
It is a way to get more RAM in ESXi

Your virtual machine system runs short in RAM
Can you post:
cat /proc/meminfo
uname -r
free (from both systems, week or more since reboot)

As simple suggestion - apply all relevant updates and all security updates out there..
0
 
LVL 34

Expert Comment

by:Seth Simmons
ID: 39812219
i did look at your initial problem summary and it's confusing

you said you have an issue on ONE RHEL server but then mention 2 servers with same hardware, etc. with one 50% used and another server 100% used then no swap usage on the first server.  it is unclear what the specific issue is on this RHEL server

you also need to clarify what you are looking at - the guest or esx itself.  as gheist mentioned, these are 2 separate environments and are not related to each other
0
 

Author Comment

by:apunkabollywood
ID: 39814385
Hi - Issue is that on second server we are facing performance issue now while both servers having same configuration and applications running.

The only diffrence is that they are on difffrent ESX .

and what we observe that First one consuming very less swap while another using high swap
0
 

Author Comment

by:apunkabollywood
ID: 39814450
0
 

Author Comment

by:apunkabollywood
ID: 39814451
0
 

Author Comment

by:apunkabollywood
ID: 39814453
1.      The first show swap usage more than 50 % , why second show swap usage zero even in the presence of swap command ? is that because swap cached is used on second , if yes why it is not the same case on first ?
2.      Why on First we have more than 4GB  memory free , and Swap is used , is that means that all process running in swap are inactive process
0
 
LVL 61

Expert Comment

by:gheist
ID: 39814918
Run this (some kernel pool of priority higher than normal malloc is using up memory)
 vmstat -m | sort -n -k 5 | tail
Last few lines are of interest if they differ
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 34

Expert Comment

by:Seth Simmons
ID: 39815040
i would also be interested in your top output if you sorted by RES
what are the java memory parameters configured for?  i'm guessing your maximum heap size is 1gb?
either way, you clearly have insufficient memory and swap space (only 1gb)
maybe your vmstat output might yield something useful
0
 
LVL 61

Expert Comment

by:gheist
ID: 39815244
vmware best practives v5.5 says to use largepages whenever possible.
That implies locking whole -Xmx in VMs memory
So first we need to see if there is not any interference from RHEL bugs before saying - double ram and use large pages...

Also interesting - what version is your vmware at?
uname -r from rhel?
Do you use vmware-tools or open-vm-tools? At what version?
Please post text, png is unreadable.
0
 

Author Comment

by:apunkabollywood
ID: 39820094
vmstat -m | sort -n -k 5 | tail
size-32(DMA)                  0      0     32    112
tcp_bind_bucket             348    672     32    112
anon_vma                   1865   1872     24    144
dm_tio                     7539   7776     24    144
dm-tracked-chunk              0      0     24    144
fasync_cache                  0      0     24    144
journal_handle               17    144     24    144
numa_policy                  62    144     24    144
biovec-1                    565    808     16    202
revoke_table                 58    202     16    202



OS VErsion : RHEL 5.2


vmtools installed with balooning options

VM esx not confirm but looks 5.0
0
 

Author Comment

by:apunkabollywood
ID: 39820095
FREE on Problem VM

 free -m
             total       used       free     shared    buffers     cached
Mem:          7741       2571       5169          0        472        377
-/+ buffers/cache:       1722       6019
Swap:         1023        645        378


free on other VM

free -m
             total       used       free     shared    buffers     cached
Mem:          7741       3549       4192          0        511        808
-/+ buffers/cache:       2229       5512
Swap:         1023          0       1023


both have exact same OS and application configured
0
 
LVL 61

Expert Comment

by:gheist
ID: 39820230
Upgrade to supported RHEL
5u6 EUS
or
5U10
And make sure to install vmware tools after.
0
 

Author Comment

by:apunkabollywood
ID: 39820266
But can we have reasons behind this to support this upgrade ?

also we are not in condition to upgrade to whats an alternative ? and if atlast need to upgrade we have 5u5 and 5u8 which one would be better?
0
 
LVL 61

Accepted Solution

by:
gheist earned 450 total points
ID: 39823281
I mentioned two supported versions of RHEL.
They are equally good as being supported at all for the money you pay.
Redhat supports upgrades and downgrades for all packages, and leaves couple of old boot environments in case things go bad. Install CentOS 5.2 on your desktop VM and just upgrade...

They will let you apply VMWare performance best practices by locking java heaps in hugepages (consider your VM RAM to be twice java heaps -Xmx even if not using hugepages)
Now - it is up tp you - try hugepages, zram, whatever. If it crashes you get shaking hands and not even moral support from vendors.

Other best practices - hypervisor cpu usage should be <80%
Each machine should get 30 IOPS in average

Sorry you are to brief to help. You have to use vmware tools and say it loud that you do. Also exact vmware patchlevel helps identify patches that fixes bugs for outdated RHEL VMs
0
 

Author Comment

by:apunkabollywood
ID: 39843637
Thanks Ghiest ,
This is to confirm VM tools installed - i am not sure about patch level from vmware side but now situation is wrost then before it consumes more swap and seems like going to 100% soon - I just want to make sure
1- Upgrading RHEL 5.2 to RHEL 5.5 will it resolve the issue ? is there any bug exist in RHEL 5.2 related to swap or ram?

2. Can i say only upgrading RHEL and checking VMware is the only option left for us?

Please advice for the same - I am also checking from vmware side and will update once got all information
0
 
LVL 61

Expert Comment

by:gheist
ID: 39844678
I mean RHEL 5.10 or 5.7 depending on your support contract.
I was talking about vmware best practices for ESXi/Vi v5.5 that mandates use of large memory pages in guests if possible at all.
I also tried to enlighten you that you need 2 times all sum of -Xmx of RAM in system to get there (if you have less expect moderate swapping)

Your RHEL is obsolete. Especially 5.4 will be first RHEL that boots on vmware without tweaking boot parameters (but it is out of extended support schedule anyway)

In Red Hat Enterprise Linux 5, EUS is available for the following minor releases:
5.2 (ends March 31, 2010)
5.3 (ends November 30, 2010)
5.4 (ends July 31, 2011)
5.6 (ends July 31, 2013)
5.9 (available for 6 months after 5.11 availability)
0
 

Author Comment

by:apunkabollywood
ID: 39864529
Hi - I found these errors - but confuse doest relate to swap problem or above problem ? please advice what are these:


kernel: hda: tray open
kernel: end_request: I/O error, dev hda, sector 92
kernel: hda: tray open
kernel: end_request: I/O error, dev hda, sector 92
kernel: VFS: busy inodes on changed media or resized disk hda

ccess errors for portmap process :

portmap[2420]: warning: cannot open /etc/hosts.allow: Permission denied
portmap[2420]: warning: cannot open /etc/hosts.deny: Permission denied
portmap[2420]: warning: cannot open /etc/hosts.allow: Permission denied
portmap[2420]: warning: cannot open /etc/hosts.deny: Permission denied
portmap[17588]: connect from 10.238.13.99 to callit(nsrstat): request from unauthorized host
0
 
LVL 34

Assisted Solution

by:Seth Simmons
Seth Simmons earned 50 total points
ID: 39864727
that has nothing to do with it

the vfs/hda messages relate to the virtual cd device being ejected after vmware tools installation

portmap is for communications.  hosts.allow/deny control who can access the system over ssh
0
 
LVL 61

Expert Comment

by:gheist
ID: 39870698
nsrstat is part of emc legato networker. if the IP is from your backup server ask backup admin to help you with hosts.allow files
0

Featured Post

6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

Join & Write a Comment

Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
HOW TO: Install and Configure VMware vSphere Hypervisor 6.5 (ESXi 6.5), Step by Step Tutorial with screenshots. From Download, Checking Media, to Completed Installation.
Teach the user how to install and configure the vCenter Orchestrator virtual appliance Open vSphere Web Client: Deploy vCenter Orchestrator virtual appliance OVA file: Verify vCenter Orchestrator virtual appliance boots successfully: Connect to the …
This tutorial will walk an individual through the steps necessary to enable the VMware\Hyper-V licensed feature of Backup Exec 2012. In addition, how to add a VMware server and configure a backup job. The first step is to acquire the necessary licen…

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now