Solved

Linux : disable disk caching

Posted on 2014-02-27
18
3,213 Views
Last Modified: 2014-03-13
I need to diable disk caching on RHEL

# free
             total       used       free     shared    buffers     cached
Mem:     528493876   86554832  441939044          0     930332   78327364
-/+ buffers/cache:    7297136  521196740
Swap:     16777208          0   16777208

in the output above, the system is using 86554832  bytes of memory, but almost all of it is disk cache, 78327364 bytes

I want to disable it entirely, and reboot.

how can i do this ?
0
Comment
Question by:c_hockland
  • 7
  • 4
  • 3
  • +2
18 Comments
 
LVL 11

Expert Comment

by:un1x86
ID: 39892380
There is no way you can directly disable this feature. But there is a few kernel parameters that help you shape its behaviour

http://unixadminschool.com/blog/2013/10/rhel-6-controlling-cache-memory-page-cache-size/
0
 
LVL 16

Expert Comment

by:Gerald Connolly
ID: 39894254
Why do you want to disable it?  What problem are you trying to solve?
0
 

Author Comment

by:c_hockland
ID: 39894859
My DBA has asked me to disable it because he thinks the performance issues he is having on his Oracle DB might be related to the cache value that i posted.
0
 
LVL 16

Expert Comment

by:Gerald Connolly
ID: 39896676
What on the basis that it is taking up RAM that Oracle could use?
0
 
LVL 26

Expert Comment

by:skullnobrains
ID: 39896962
given the fact that 441G out of 528G are free, that would be weird to say the least

if his point is that the ram is effectively divided by 2 because the os and the rdbm both cache the same data, this is true if oracle's tablespaces are on the filesystem, but since most of the ram is unused, you probably would not care.

having 78G of cache used means oracle hardly uses more than that in terms of disk access on the filesystem. it looks like the performance problem is elsewhere

btw gosh what the hell is that machine ? how big and intensely used a database must be to need this ?

you may want to run iostat and post information regarding the disk layout if you want to know why oracle is slow.

----

back to your question, you can disable write cache using -o sync in fstab. i do not know of a simple way to deactivate read cache. anyway, it is pretty foolish to even try.

any decent rdbm including oracle can use directio both for tablespaces and logs which effectively bypasses the caches for what oracle uses. you can also give oracle raw disks.
0
 
LVL 61

Expert Comment

by:gheist
ID: 39897884
OracleDB can skip disk cache using direct-io, are you sure its a DBA?

http://www.oracle-base.com/articles/misc/direct-and-asynchronous-io.php
0
 

Author Comment

by:c_hockland
ID: 39900812
i forgot to mentioned .... these are ASM ( oracle disks )

i dont see these asm disks @ /etc/fstab

any ideas ?
0
 
LVL 61

Expert Comment

by:gheist
ID: 39901208
So they use direct io and kernel aio. There is no OS buffer in between.
0
 

Author Comment

by:c_hockland
ID: 39901283
well , i realized that linux doesnt know and doesnt care about asm disks.
I will still have to disable write cache on the linux level
0
Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

 
LVL 61

Expert Comment

by:gheist
ID: 39901640
No, there is no linux write cache in the way of ASM

Only thing you can change is IO scheduler per disk or system default (tip - elevator=deadline is default on UEK)
That will change order aio requests are served, namely deadline makes sure they ground asap.
0
 
LVL 26

Expert Comment

by:skullnobrains
ID: 39901744
asm disks are by nature synchronous so "-o sync" from my previous comment does not apply, my bad

you'll find below a bunch of raw information but no real answer

using unbuffered IO is a matter of using directio or not which is something for the application to decide, not the OS. it is usually a parameter to the open system call and can be changed using ioctl on an open file descriptor. on linux there is also a family of functions (io_setup, io_getevent...) that will always be buffered by nature.

as far as i know, linux does not actually have a buffer cache separated from the filesystem's ( any one can confirm/infirm ? )

anyway, there is virtually no way that buffered IO will be slower than directIO. this is espetially true when you have that much free RAM. most OSes will use unused RAM for their buffer cache. this enhances disk writes speed a lot, and possibly writes with the use of prefetch.

there are high chances that the cached memory you see is totally unrelated to asm anyway. possibly your redo logs and the likes are on a regular filesystem ?

or oracle is configured to do asynchronous read/writes on asm (i have no idea wether this would appear as "cached" in top... if someone knows, thanks for sheding some light on the matter)

this links details different oracle behaviour in conjunction with asm or fiesystems. you'll probably find it usefull. https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Tuning_and_Optimizing_Red_Hat_Enterprise_Linux_for_Oracle_9i_and_10g_Databases/chap-Oracle_9i_and_10g_Tuning_Guide-Configuring_IO_for_Raw_Partitions.html

hope some of this helps

ps : maybe if you give more details about the actual performance problem (and run an iostat when something seems wrong), we could give much better hints regarding what to work on. disabling the buffer cache does not seem like a good candidate. if your dba has a theory, i'll be happy to know about it
0
 
LVL 61

Expert Comment

by:gheist
ID: 39901783
One can apply disk scheduler to block device backing raw device (actually there are heaps of tunable parameters, but "deadline" is what good disk firmware does, or noop if you trust disk/raid firmware, probably better test which does better - you can apply them to live partition, so if oracle admin has clean room benchmark let them help you)

That would operate on "buffers" seen in "free" (you cannot avoid those buffers, it is where data of disk request goes before or from disk)
Then there comes various filesystem cached, out of scope when handling raw devices, essentially free RAM will be used as filesystem cache.

If you want you can try disabling disk's/raid's internal cache (at the (huge) cost of performance)
0
 
LVL 16

Expert Comment

by:Gerald Connolly
ID: 39902152
So the consensus of the experts on this thread appears to be that none of us can understand why your DBA wants the cache disabled as it doesnt make any sense.

Yes the cache useage looks large, but there is a huge amount of RAM still free, so the system is not memory constrained.

And although you CAN disable the write cache, your whole system is going to suffer the performance degradation this will bring.

So lets see the DBA's justification of why disabling the caching on this system is going to solve whatever the original issue is.
0
 
LVL 61

Expert Comment

by:gheist
ID: 39903140
I got it (maybe)
Is your DBA complaining about oracle being swapped out? Do you use AMM too?
0
 
LVL 26

Expert Comment

by:skullnobrains
ID: 39906230
how could anything be swapped on a machine that has that much free ram ? if i'm not mistaken, even if a process could grow that big, it would be oem-killed before it would be swapped.
0
 
LVL 61

Expert Comment

by:gheist
ID: 39906376
It happens. there is cache pressure and swappiness settings.
Common scenario is oracle making buffered writes to backup (extfs or nfs) and verifying, to make sure less used /dev/shm pages go to swap file.
0
 
LVL 26

Accepted Solution

by:
skullnobrains earned 500 total points
ID: 39908771
@gheist : yes, these settings do exist, but how would a backup + backup check swap out anything in this case ? espetially shared pages which are unswappable most of the time (if not always on linux ?)

---

@c_lockland : i think we're moving away from your problem

there seems to be a mixup between buffers and ansynchronous writes in this thread :

when writing through a filesystem, you write through the kernel's VFS so there is no way you can bypass buffers, whatever you do

even with synchronous writes, buffers get written to. that does not mean these buffers are dirty (other than transiently during the write operation). there might be a way to force them to be deleted but the system will do so by itself whenever an app requires some ram before it will swap anything

-

when accessing disks directly, you bypass the vfs, and as far as i gather there is no separate buffer cache (well not that big anyway) on linux so the buffers are of no concern in this case

---

do you use asm for everything ?
does asm perform caching ?
do you use directIO for whatever is not managed by asm (logs ?) ?

what do you expect ?
0
 
LVL 61

Expert Comment

by:gheist
ID: 39926204
AMM uses tmpfs and not systemV ipc and SGA can go to swap....
0

Featured Post

Comprehensive Backup Solutions for Microsoft

Acronis protects the complete Microsoft technology stack: Windows Server, Windows PC, laptop and Surface data; Microsoft business applications; Microsoft Hyper-V; Azure VMs; Microsoft Windows Server 2016; Microsoft Exchange 2016 and SQL Server 2016.

Join & Write a Comment

I. Introduction There's an interesting discussion going on now in an Experts Exchange Group — Attachments with no extension (http://www.experts-exchange.com/discussions/210281/Attachments-with-no-extension.html). This reminded me of questions tha…
Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now