swap consumption questions

I have some questions regarding Linux's use of the swap partition.  I am using Linux ES 3 on a 2 CPU machine w/ 2 GB RAM and a 2 GB swap partition.

1.) I just killed a large process on my server, but 5 additonal MB of swap were consumed as a result (though about 100 MB of physical RAM was freed up).  Is this normal?  Why did this happen?

2.) After restarting the same process a few times, I noticed that, after one restart, a few MBs of swap were freed, but almost no physical RAM was freed, the second time, #1 happened.  Why?

3.) With a system running many looping processes (I have checked these procs for memory leaks) what is the normal behavior over time of the swap and physical memory (should one decrease w/ one increasing, etc)?

rnicholusAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Duncan RoeSoftware DeveloperCommented:
Linux will use all the memory it can for file caching. Do not expect to see much unused, ever.
rnicholusAuthor Commented:
I know this is true for physical memory, but i did not know that this applied to swap as well?  Really?  So if Linux reports that a large portion of my swap is in use after some time, I shouldn't be worried?
Tim_UtschigCommented:
> 1.) I just killed a large process on my server, but 5 additonal MB of swap were consumed as a result (though about 100 MB of physical RAM was freed up).  Is this normal?  Why did this happen?

When the large process consumed a lot of memory, Linux "swapped out" things which hadn't been used in a while (e.g: idle programs).  This is normal.

If you want to free up that swap all at once and bring the swapped processes back into main memory for whatever reason, turn off your swap for a moment:

    swapoff -a; swapon -a

> 2.) After restarting the same process a few times, I noticed that, after one restart, a few MBs of swap were freed, but almost no physical RAM was freed, the second time, #1 happened.  Why?

See duncan_roe's comment about caching.

> 3.) With a system running many looping processes (I have checked these procs for memory leaks) what is the normal behavior over time of the swap and physical memory (should one decrease w/ one increasing, etc)?

If memory is needed and there is no more cache available to free to make room for it, it will start swapping starting with the processes which haven't been used in the longest time.
Starting with Angular 5

Learn the essential features and functions of the popular JavaScript framework for building mobile, desktop and web applications.

ppfoongCommented:

It seems that Kernel 2.6 onwards will avoid using swap when there is no neccessary to use it. Earlier kernel will use some swap even though the physical memory is sufficient.

If you are using earlier version of kernel, you might be able to see the different after you upgrade it to 2.6.

Here is a quick and dirty way to reset the swap:

swapoff -a; swapon -a

rnicholusAuthor Commented:
Over time, it seems as if free swap is decreasing along w/ free physical memory, but I dont see the RSS value of any of the running procs increasing substantially...
rnicholusAuthor Commented:
...also, why would I want to reset the swap?
Tim_UtschigCommented:
> Over time, it seems as if free swap is decreasing along w/ free physical memory, but I dont see the RSS value of any of the running procs increasing substantially...

Are you looking at the "Mem:" line or the "-/+ buffers/cache:" line of the output of the "free" command?

> ...also, why would I want to reset the swap?

I can't think of a good reason.   But I do so once in a while after I do a memory-intensive operation.  Otherwise there's a delay and bunch of disk cranking (as the kernel loads the program's data back into main memory) when I go back to Firefox or some other app I had open.
rnicholusAuthor Commented:
>>Are you looking at the "Mem:" line or the "-/+ buffers/cache:" line of the output of the "free" command?
+/- buffers/cache

rnicholusAuthor Commented:
Ok, here's my theory on the Linux swap.  Please correct me wherever I may be incorrect:

The swap is used as needed, but pages are kept in the swap, and the swap will continue to grow.  If a situation arises where data must be swapped out and the swap does not have enough free space left, idle pages that may not be needed anymore are removed from memory altogether.  

For some reason, this doesn't make sense to me.  If my theory is true, doesn't the possibility of invalid page faults increase?  
Tim_UtschigCommented:
Your theory is sound.

> For some reason, this doesn't make sense to me.  If my theory is true, doesn't the possibility of invalid page faults increase?

The kernel knows where to find the pages when they're needed.
rnicholusAuthor Commented:
>>The kernel knows where to find the pages when they're needed.
But, if pages are released completely from memory, isn't there a possiblity that a running application will generate an interrupt/page fault, the OS will look at the swap for the requested page, and find that the page has been released from memory altogether in error, resulting in an invalid page fault?  I guess I don't know how the kernel can know if a page is never going to be used again...
Tim_UtschigCommented:
The kernel never frees a page unless it's not being used at all.
Tim_UtschigCommented:
> I guess I don't know how the kernel can know if a page is never going to be used again...

If a page is allocated, it is in use and the kernel knows that.  If it swaps it, all that changed is the physical location of the page.  Once it is freed it is free to be allocated again.
rnicholusAuthor Commented:
>>If a page is allocated, it is in use and the kernel knows that.  If it swaps it, all that changed is the physical location of the page.  Once it is freed it is free to be allocated again.

So once a page is freed it does not remain in swap as cached or something?  Is it removed from swap right away?
Tim_UtschigCommented:
> So once a page is freed it does not remain in swap as cached or something?

When it is free'd, probaby all that happens is some flag which indicated that the page was in use is cleared.  The data will remain right where it is, but that page is now available for use again, and should count towards the "free" total.

> Is it removed from swap right away?

If a swapped page is free'd, that area of the swap space should available immediately, yes.
rnicholusAuthor Commented:
So, it is or is not normal for free swap to decrease over time?
Tim_UtschigCommented:
> So, it is or is not normal for free swap to decrease over time?

If you have more and more idle pages being swapped out because the memory they occupy is needed for something else at some moment, then yes, it's normal.

Only you can determine if it is normal based on what your machine does hour by hour, day by day.

rnicholusAuthor Commented:
Is it ever OK for the swap to reach 0 free?  I've already patched a leak in a system of applications I'm running, so I'm somewhat certain that a memory leak doesn't exist anymore...
Tim_UtschigCommented:
> Is it ever OK for the swap to reach 0 free?

If you still have physical memory available, then yes, that's OK, but it might indicate that you want to increase your swap space, or figure out why you have so many idle processes laying around.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
rnicholusAuthor Commented:
Actually, it's pretty common for my linux server to have a large # of idle processes at once.  I have a large number of programs that perform an operation, sleep for a variable amount of time, and then iterate once more, etc.  I felt that it made more sense to keep these alive and in memory at all times to avoid the context switching associated with killing and restarting each process every 5 minutes (in some cases).
Tim_UtschigCommented:
That would explain it.

Anything else you need clarified?
rnicholusAuthor Commented:
Right now?  Nope.  I'm sure I'll have more later.  Thanks!

Time to close this one out...
rnicholusAuthor Commented:
Oops, almost forgot.  Hope someone is still monitoring this...

Is it conceivable that when a page is swapped in, the page is not flagged as free in the swap partition so that, in the event that the same page needs to swapped out again in the future, the kernel will not have to copy the page to disc again?
Tim_UtschigCommented:
> Hope someone is still monitoring this...

Anyone who has posted this question or otherwise subscribed to question will receive notifications of any replies unless they manually unsubscribe.

> Is it conceivable that when a page is swapped in, the page is not flagged as free in the swap partition so that,
> in the event that the same page needs to swapped out again in the future,
> the kernel will not have to copy the page to disc again?

I'm no kernel developer, and I have no idea whether they currently do such a thing, but if they did I bet they would mark it as "free" and just have some other indicator that the page has an identical copy at a certain location in the swap space.

Of course this is all just conjecture.  If you want the real deal, the kernel source is readily available ;-)
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.