We help IT Professionals succeed at work.

Why system is swapping without reclaiming huge cached memory

Franck Pachot
on
804 Views
Last Modified: 2014-01-14
Hi,

I'm on a system that experiences swapping.
However, I have 4GB for 'cached' and I expected that the cache can be reclaimed under memory pressure.

top - 10:57:10 up 8 days,  2:49,  2 users,  load average: 0.18, 0.36, 0.46
Tasks: 645 total,   1 running, 644 sleeping,   0 stopped,   0 zombie
Cpu(s):  9.6%us,  8.9%sy,  0.0%ni, 81.0%id,  0.4%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:   8126148k total,  7963148k used,   163000k free,     6912k buffers
Swap: 25165816k total,  6066084k used, 19099732k free,  4103596k cached

Open in new window

Do I interpret that correctly when I say that only 4GB of physical memory is used by the processes and the unused part serves as cache ?

Here is detail from sar (big swapping issue at 09:50)

sar -r
            kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit
09:30:01 AM    134444   7991704     98.35     10332   4028216  19971452     59.99
09:40:01 AM    160784   7965364     98.02     10388   4048288  19818544     59.53
09:50:01 AM    128004   7998144     98.42      4868   4027960  19852052     59.63
10:00:01 AM    141856   7984292     98.25      6392   4151896  20006856     60.10
10:10:01 AM    195496   7930652     97.59     14352   4190944  19803992     59.49
10:20:04 AM    144424   7981724     98.22      4552   4256120  19811740     59.51
10:30:01 AM    155312   7970836     98.09      8248   4215552  19963548     59.97
Average:       193411   7932737     97.62     16036   3977477  19849017     59.62

sar -S
            kbswpfree kbswpused  %swpused  kbswpcad   %swpcad
09:30:01 AM  19083496   6082320     24.17    877932     14.43
09:40:01 AM  19099620   6066196     24.10    839472     13.84
09:50:01 AM  19130128   6035688     23.98    864112     14.32
10:00:01 AM  19206012   5959804     23.68    697464     11.70
10:10:01 AM  19228044   5937772     23.59    671576     11.31
10:20:04 AM  19245732   5920084     23.52    738748     12.48
10:30:01 AM  19188576   5977240     23.75    744344     12.45
Average:     18987125   6178691     24.55    934020     15.12

sar -B
             pgpgin/s pgpgout/s   fault/s  majflt/s  pgfree/s pgscank/s pgscand/s pgsteal/s    %vmeff
09:30:01 AM  11740.51    447.41  10604.60     15.65   3232.52    168.66     15.14    131.06     71.31
09:40:01 AM   7884.12   1225.61  10787.88     10.01   3223.09     87.22     18.81     83.75     78.99
09:50:01 AM   7657.48    475.55  10900.24     34.10   3402.93    309.62      9.57    243.68     76.34
10:00:01 AM   9923.26    677.34  11654.52     81.67   4279.32    835.99     17.22    632.46     74.13
10:10:01 AM   7943.10    399.23  10848.28     17.25   3200.36    108.03      9.29     88.78     75.66
10:20:04 AM  11178.82   1094.43  13837.84    146.95   5561.22   1372.39    123.54   1101.29     73.62
10:30:01 AM   8651.50    491.58  10718.29     16.86   3267.58    242.49     19.85    205.83     78.46
10:40:01 AM   8092.91    851.57  10575.16     19.10   3188.81    160.80     22.52    142.26     77.60
Average:     10435.52    622.72  10827.20     11.70   3233.49    102.95     24.56     96.19     75.44

Open in new window


But I'm not a specialist. What is the meaning of 'cached' and should I consider it as used or available memory ?

Thanks,
Franck.
Comment
Watch Question

Gerwin JansenTopic Advisor
CERTIFIED EXPERT
Most Valuable Expert 2016

Commented:
HI, there are previously answered questions on this subject at EE, like this one:

https://www.experts-exchange.com/OS/Linux/Q_27967317.html

Does that answer your question as well?
Franck PachotOracle DBA
CERTIFIED EXPERT

Author

Commented:
Hi,
Interresting link, but doesn't anwser. Let me rephrase my question.

1. From the 'top' result I've 8GB (8126148k total) of physical memory, and half of it (4103596k cached) is used for filesystem cache. is that right ?

2. I experience swapping because I use more memory than physical ones. in fact, there are several Oracle instances and the total of memory allocation is near 8GB
I expected that before swapping, the system should try to use the physical memory that is used for cache (i.e the 4GB). Am I missing something ?

Thanks,
Franck.
Lead Systems Administrator
CERTIFIED EXPERT
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION
Franck PachotOracle DBA
CERTIFIED EXPERT

Author

Commented:
Thanks a lot. I forgot to check swappiness . I will do it when I'm at the customer again next week.
just one additional precision, please: what is the best way to see - before the system starts swapping - if are low on physical memory.
I used to consider 'free' + 'buffers' + 'cached' but obviously, for the reason you gave me, it's not right. Monitoring page in/out will probably alert alert too late. Is there sonething else ?
Seth SimmonsLead Systems Administrator
CERTIFIED EXPERT

Commented:
free or top will show physical memory usage

Commented:
I can't actually see a problem from the reports you posted - I don't see any change at 09:50, though I/O stats might show something if the system is really paging/swapping.

Oracle uses shared memory, so adding up the memory usage reported by `ps` will overestimate the amount actually used. You need to look within Oracle to see how it is using that memory - and if the system is mainly used for Oracle, consider increasing the SGA size so Oracle can manage its own caching.

Mem:   8126148k total,  7963148k used,   163000k free,     6912k buffers
Swap: 25165816k total,  6066084k used, 19099732k free,  4103596k cached

I believe the 6GB of swap "used" is reserved in case the system _needs_ to swap rather than actively in use. As you know,  the "cached" figure should be part of the previous line, Mem stats.

`sar -B` shows a peak at 10.20, but it could be a new process pulling data in.

Also, how do the stats compare to your "benchmarks"? e.g Mem usage after a reboot & starting Oracle, plus normal, low and high usage?
Franck PachotOracle DBA
CERTIFIED EXPERT

Author

Commented:
@seth2740,
free or top will show physical memory usage
It show physical memory used. But part of it is used only because it is not needed for processes. How to know that part ?
Well I'll post the question again in a new thread.

@tfewster,
Thanks. Yes I'm aware that ps reports the shared memory for each process.
But I'm sure that SGA was above the physical memory and I'm sure the system was swapping at that time (system very long, kswapd busy) for at least 5 minutes around 09:50
What I did not understand was why it did not reclaim space from the filesystem cache.

Commented:
Aha, that's a different symptom. Search online for "kswapd high CPU". It's a known bug, and you should consider a kernel update.

I still suspect that, from the stats you posted, the system wasn't swapping and so didn't need to reclaim the memory used for cache.

Good luck!
Franck PachotOracle DBA
CERTIFIED EXPERT

Author

Commented:
Sorry, but the system was swapping. When you are on a system that starts swapping a lot when you do something that needs to read a large portion of memory, you know that it is swapping. No doubt... Operation was an Oracle Statspack snapshot level 7 ... reading lot of things from shared pool.
My doubt was about why that memory was swapped.
The answer from seth2740 makes sense: that part of memory was probably idle for a long time (low activity on the instance before I arrived) and the system preferred to keep data in cache because of swapiness.

I'll update this thread on tuesday when I'll check swapiness value.
Franck PachotOracle DBA
CERTIFIED EXPERT

Author

Commented:
Hi,
Jus a not to confirm that it was due to swappiness that was high:
$ cat /proc/sys/vm/swappiness
60

Open in new window

Thanks again seth2740
Regards,
Franck.

Gain unlimited access to on-demand training courses with an Experts Exchange subscription.

Get Access
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Empower Your Career
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.