rhel not releasing chache memory for applications
Hi All,
I have applications configured on RHEL 5.5 on VMware machine.
Recently migrated from Physical to Vmware
- Problem is coming that now application used all RAM fast and then application crashed (Only application) and to avoid that either we need to restart the application or echo 3 to drop caches.
This is really painful to do .
What i checked that when the memory utilizatio reached 99% also then too it doesnt release the Cached memory for applications .
Please help why its happening and how to resolve this?
current swappiness value is : 60
Total RAM : 16 GB
Swap : 1 GB ( It always free - never used)
RHEL 5.5 OS
I have applications configured on RHEL 5.5 on VMware machine.
Recently migrated from Physical to Vmware
- Problem is coming that now application used all RAM fast and then application crashed (Only application) and to avoid that either we need to restart the application or echo 3 to drop caches.
This is really painful to do .
What i checked that when the memory utilizatio reached 99% also then too it doesnt release the Cached memory for applications .
Please help why its happening and how to resolve this?
current swappiness value is : 60
Total RAM : 16 GB
Swap : 1 GB ( It always free - never used)
RHEL 5.5 OS
ASKER
Memory info
MemTotal: 16436872 kB
MemFree: 89416 kB
Buffers: 381188 kB
Cached: 12983640 kB
SwapCached: 0 kB
Active: 3308144 kB
Inactive: 12825452 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 16436872 kB
LowFree: 89416 kB
SwapTotal: 1048568 kB
SwapFree: 1048464 kB
Dirty: 176 kB
Writeback: 0 kB
AnonPages: 2768628 kB
Mapped: 141028 kB
Slab: 168112 kB
PageTables: 19964 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 9267004 kB
Committed_AS: 5366696 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 264080 kB
VmallocChunk: 34359473527 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
MemTotal: 16436872 kB
MemFree: 89416 kB
Buffers: 381188 kB
Cached: 12983640 kB
SwapCached: 0 kB
Active: 3308144 kB
Inactive: 12825452 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 16436872 kB
LowFree: 89416 kB
SwapTotal: 1048568 kB
SwapFree: 1048464 kB
Dirty: 176 kB
Writeback: 0 kB
AnonPages: 2768628 kB
Mapped: 141028 kB
Slab: 168112 kB
PageTables: 19964 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 9267004 kB
Committed_AS: 5366696 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 264080 kB
VmallocChunk: 34359473527 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
How do you know that cache is not being released? Does Apache throw an "out of memory" error?
Out of interest, how much memory does the host machine have?
Out of interest, how much memory does the host machine have?
swappiness value 60, 1gb swap partition and not used doesn't sound much like an issue itself
the kernel will release cache as needed; won't automatically flush as you would manually to drop_caches
if the web application is crashing i would look at apache logs, any addins configured (php, etc.) and go from there - especially if apache processes are using that much memory. the cache could be used by the application.
another thing to check is messages to see if the kernel is invoking OOM killer on apache. i've seen this before with a badly configured application that causes a memory leak
the kernel will release cache as needed; won't automatically flush as you would manually to drop_caches
if the web application is crashing i would look at apache logs, any addins configured (php, etc.) and go from there - especially if apache processes are using that much memory. the cache could be used by the application.
another thing to check is messages to see if the kernel is invoking OOM killer on apache. i've seen this before with a badly configured application that causes a memory leak
ASKER
These are the logs coming from Kernel:
kernel: ll header: ff:ff:ff:ff:ff:ff:00:15:60
kernel: martian source 255.255.255.255 from 10.1.2.3, on dev eth3
kernel: ll header: ff:ff:ff:ff:ff:ff:00:15:60
kernel: martian source 255.255.255.255 from 10.1.2.3. on dev eth3
kernel: ll header: ff:ff:ff:ff:ff:ff:00:15:60
kernel: martian source 255.255.255.255 from 10.2.3.4, on dev eth2
kernel: ll header: ff:ff:ff:ff:ff:ff:00:17:a4
printk: 3 messages suppressed.
these are the logs i found nothing else
kernel: ll header: ff:ff:ff:ff:ff:ff:00:15:60
kernel: martian source 255.255.255.255 from 10.1.2.3, on dev eth3
kernel: ll header: ff:ff:ff:ff:ff:ff:00:15:60
kernel: martian source 255.255.255.255 from 10.1.2.3. on dev eth3
kernel: ll header: ff:ff:ff:ff:ff:ff:00:15:60
kernel: martian source 255.255.255.255 from 10.2.3.4, on dev eth2
kernel: ll header: ff:ff:ff:ff:ff:ff:00:17:a4
printk: 3 messages suppressed.
these are the logs i found nothing else
redhat (or any other Linux) will use all free RAM for cache.
vm.min_free_kbytes can be set to keep as much free memory as you want
namely i'd suggest dropping VM memory to 8GB as only <4G is used at any time
(With RHEL6 where RAM hotplug works i'd say to enable that and start at 4G and size up asneeded)
and either set guest mem warning threshold to something more appropriate
or use vm.min_free_kbytes to be 5% or more of MemTotal: form /proc/meminfo
Actually Linux kernel will stop at 64M in free kbytes, so you can rise it. e.g for DB2 IBM recommends setting that to 6% of MemTotal...
Also feel free to discuss with vmware support why they never counted cache for free memory, when actually their balloon driver will discard cache just like another free RAM...
vm.min_free_kbytes can be set to keep as much free memory as you want
namely i'd suggest dropping VM memory to 8GB as only <4G is used at any time
(With RHEL6 where RAM hotplug works i'd say to enable that and start at 4G and size up asneeded)
and either set guest mem warning threshold to something more appropriate
or use vm.min_free_kbytes to be 5% or more of MemTotal: form /proc/meminfo
Actually Linux kernel will stop at 64M in free kbytes, so you can rise it. e.g for DB2 IBM recommends setting that to 6% of MemTotal...
Also feel free to discuss with vmware support why they never counted cache for free memory, when actually their balloon driver will discard cache just like another free RAM...
ASKER
Hi Ghiest,
This is the current value on server:
cat /proc/sys/vm/min_free_kbyt
16261
and i have total ram of 16 GB and 1 GB swap
what you recommend?
and its RedHat RHEL 5.5 x86_64
This is the current value on server:
cat /proc/sys/vm/min_free_kbyt
16261
and i have total ram of 16 GB and 1 GB swap
what you recommend?
and its RedHat RHEL 5.5 x86_64
ASKER
Actually problem is this we need to drop cache in every 5 hours because RAM used to get full in 5 hours - if we dont drop cache application need to be crashed and then restart the app again
If dropping cache solves your problem (though I can't see why it helps), you could just add
*/3 * * * * echo 3 > /proc/sys/vm/drop_caches
to your crontab to drop cache every three hours.
*/3 * * * * echo 3 > /proc/sys/vm/drop_caches
to your crontab to drop cache every three hours.
ASKER
Hi Simon - its a production server and also memory full in hours ...i think as per redhat too this is not recommended for production servers?
Not ideal for production (disk access will spike after you do this), but if it keeps the server running, it's better than a crash.
Is there anything relevant in the logs for the host machine? If that starts to swap the VMs because *it* is running out of memory, that might cause a problem within the VMs.
Maybe you haven't installed the VMware Tools?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Thanks Ghiest for your support
ASKER