Avatar of Franck Pachot
Franck Pachot
Flag for Switzerland asked on

Why system is swapping without reclaiming huge cached memory


I'm on a system that experiences swapping.
However, I have 4GB for 'cached' and I expected that the cache can be reclaimed under memory pressure.

top - 10:57:10 up 8 days,  2:49,  2 users,  load average: 0.18, 0.36, 0.46
Tasks: 645 total,   1 running, 644 sleeping,   0 stopped,   0 zombie
Cpu(s):  9.6%us,  8.9%sy,  0.0%ni, 81.0%id,  0.4%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:   8126148k total,  7963148k used,   163000k free,     6912k buffers
Swap: 25165816k total,  6066084k used, 19099732k free,  4103596k cached

Open in new window

Do I interpret that correctly when I say that only 4GB of physical memory is used by the processes and the unused part serves as cache ?

Here is detail from sar (big swapping issue at 09:50)

sar -r
            kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit
09:30:01 AM    134444   7991704     98.35     10332   4028216  19971452     59.99
09:40:01 AM    160784   7965364     98.02     10388   4048288  19818544     59.53
09:50:01 AM    128004   7998144     98.42      4868   4027960  19852052     59.63
10:00:01 AM    141856   7984292     98.25      6392   4151896  20006856     60.10
10:10:01 AM    195496   7930652     97.59     14352   4190944  19803992     59.49
10:20:04 AM    144424   7981724     98.22      4552   4256120  19811740     59.51
10:30:01 AM    155312   7970836     98.09      8248   4215552  19963548     59.97
Average:       193411   7932737     97.62     16036   3977477  19849017     59.62

sar -S
            kbswpfree kbswpused  %swpused  kbswpcad   %swpcad
09:30:01 AM  19083496   6082320     24.17    877932     14.43
09:40:01 AM  19099620   6066196     24.10    839472     13.84
09:50:01 AM  19130128   6035688     23.98    864112     14.32
10:00:01 AM  19206012   5959804     23.68    697464     11.70
10:10:01 AM  19228044   5937772     23.59    671576     11.31
10:20:04 AM  19245732   5920084     23.52    738748     12.48
10:30:01 AM  19188576   5977240     23.75    744344     12.45
Average:     18987125   6178691     24.55    934020     15.12

sar -B
             pgpgin/s pgpgout/s   fault/s  majflt/s  pgfree/s pgscank/s pgscand/s pgsteal/s    %vmeff
09:30:01 AM  11740.51    447.41  10604.60     15.65   3232.52    168.66     15.14    131.06     71.31
09:40:01 AM   7884.12   1225.61  10787.88     10.01   3223.09     87.22     18.81     83.75     78.99
09:50:01 AM   7657.48    475.55  10900.24     34.10   3402.93    309.62      9.57    243.68     76.34
10:00:01 AM   9923.26    677.34  11654.52     81.67   4279.32    835.99     17.22    632.46     74.13
10:10:01 AM   7943.10    399.23  10848.28     17.25   3200.36    108.03      9.29     88.78     75.66
10:20:04 AM  11178.82   1094.43  13837.84    146.95   5561.22   1372.39    123.54   1101.29     73.62
10:30:01 AM   8651.50    491.58  10718.29     16.86   3267.58    242.49     19.85    205.83     78.46
10:40:01 AM   8092.91    851.57  10575.16     19.10   3188.81    160.80     22.52    142.26     77.60
Average:     10435.52    622.72  10827.20     11.70   3233.49    102.95     24.56     96.19     75.44

Open in new window

But I'm not a specialist. What is the meaning of 'cached' and should I consider it as used or available memory ?


Avatar of undefined
Last Comment
Franck Pachot

8/22/2022 - Mon
Gerwin Jansen

HI, there are previously answered questions on this subject at EE, like this one:


Does that answer your question as well?
Franck Pachot

Interresting link, but doesn't anwser. Let me rephrase my question.

1. From the 'top' result I've 8GB (8126148k total) of physical memory, and half of it (4103596k cached) is used for filesystem cache. is that right ?

2. I experience swapping because I use more memory than physical ones. in fact, there are several Oracle instances and the total of memory allocation is near 8GB
I expected that before swapping, the system should try to use the physical memory that is used for cache (i.e the 4GB). Am I missing something ?

Seth Simmons

View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
Ask your own question & get feedback from real experts
Find out why thousands trust the EE community with their toughest problems.
Franck Pachot

Thanks a lot. I forgot to check swappiness . I will do it when I'm at the customer again next week.
just one additional precision, please: what is the best way to see - before the system starts swapping - if are low on physical memory.
I used to consider 'free' + 'buffers' + 'cached' but obviously, for the reason you gave me, it's not right. Monitoring page in/out will probably alert alert too late. Is there sonething else ?
This is the best money I have ever spent. I cannot not tell you how many times these folks have saved my bacon. I learn so much from the contributors.
Seth Simmons

free or top will show physical memory usage

I can't actually see a problem from the reports you posted - I don't see any change at 09:50, though I/O stats might show something if the system is really paging/swapping.

Oracle uses shared memory, so adding up the memory usage reported by `ps` will overestimate the amount actually used. You need to look within Oracle to see how it is using that memory - and if the system is mainly used for Oracle, consider increasing the SGA size so Oracle can manage its own caching.

Mem:   8126148k total,  7963148k used,   163000k free,     6912k buffers
Swap: 25165816k total,  6066084k used, 19099732k free,  4103596k cached

I believe the 6GB of swap "used" is reserved in case the system _needs_ to swap rather than actively in use. As you know,  the "cached" figure should be part of the previous line, Mem stats.

`sar -B` shows a peak at 10.20, but it could be a new process pulling data in.

Also, how do the stats compare to your "benchmarks"? e.g Mem usage after a reboot & starting Oracle, plus normal, low and high usage?
Franck Pachot

free or top will show physical memory usage
It show physical memory used. But part of it is used only because it is not needed for processes. How to know that part ?
Well I'll post the question again in a new thread.

Thanks. Yes I'm aware that ps reports the shared memory for each process.
But I'm sure that SGA was above the physical memory and I'm sure the system was swapping at that time (system very long, kswapd busy) for at least 5 minutes around 09:50
What I did not understand was why it did not reclaim space from the filesystem cache.
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.

Aha, that's a different symptom. Search online for "kswapd high CPU". It's a known bug, and you should consider a kernel update.

I still suspect that, from the stats you posted, the system wasn't swapping and so didn't need to reclaim the memory used for cache.

Good luck!
Franck Pachot

Sorry, but the system was swapping. When you are on a system that starts swapping a lot when you do something that needs to read a large portion of memory, you know that it is swapping. No doubt... Operation was an Oracle Statspack snapshot level 7 ... reading lot of things from shared pool.
My doubt was about why that memory was swapped.
The answer from seth2740 makes sense: that part of memory was probably idle for a long time (low activity on the instance before I arrived) and the system preferred to keep data in cache because of swapiness.

I'll update this thread on tuesday when I'll check swapiness value.
Franck Pachot

Jus a not to confirm that it was due to swappiness that was high:
$ cat /proc/sys/vm/swappiness

Open in new window

Thanks again seth2740
Experts Exchange has (a) saved my job multiple times, (b) saved me hours, days, and even weeks of work, and often (c) makes me look like a superhero! This place is MAGIC!
Walt Forbes