Solved

high percentage on 'iowait'

Posted on 2014-03-05
4
314 Views
Last Modified: 2014-05-01
I got the problem on the 'root' partition of the Linux RHEL3 server which is suddenly having the partition become read only. The case solve by rebooting the server.
On checking the 'top' status, it is found that an abnormal behavior is having the 'iowait' persist at a high rate and at the time of the incident, the 'iowait' always reaches 90-100 %
Will this be the cause of the partition becoming read only ?
If it is, what is the possible reason of the 'iowait' becoming high ?
Last but most important, how to solve this issue ?
Below showing that the iowait reaches 100% this morning and then the partition cannot write again.

03:44:02  up  7:02,  0 users,  load average: 0.00, 0.00, 0.00  
62 processes: 61 sleeping, 1 running, 0 zombie, 0 stopped
CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
           total    0.0%    0.0%    0.0%   0.0%     0.0%  100.0%    0.0%
Mem:  2056864k av,  175268k used, 1881596k free,       0k shrd,   67172k buff
       111756k active,              25436k inactive
Swap: 4192956k av,       0k used, 4192956k free                   66368k cached
0
Comment
Question by:francis_lai
4 Comments
 
LVL 34

Expert Comment

by:Seth Simmons
Comment Utility
have you looked at /var/log/messages?
when this happens, try running iostat 3 to see what device is being accessed

could also be a device causing i/o wait...maybe a failing disk?
0
 
LVL 13

Accepted Solution

by:
Daniel Helgenberger earned 500 total points
Comment Utility
My best guess is the I/O operations are timing out. Thus you have high I/O waits and high CPU load (on a low percentage, check with top). These IO - waits are causing the 'read only' state of the root volume.

This is most likely a failing disk; which could be since I assume the installation is about five to seven years old (RHEL3).

Is the root volume on a RAID device?

You should, however, use rsync or any other backup tool to constantly mirror the contents of your root device and switch disks ASAP.
0
 
LVL 34

Expert Comment

by:Duncan Roe
Comment Utility
I agree with Daniel. The only reason Linux flips a disk over to read-only is an I/O failure
0
 
LVL 48

Expert Comment

by:Tintin
Comment Utility
What's the output of

df -h

and

iostat -x 5
0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

The purpose of this article is to demonstrate how we can use conditional statements using Python.
It’s 2016. Password authentication should be dead — or at least close to dying. But, unfortunately, it has not traversed Quagga stage yet. Using password authentication is like laundering hotel guest linens with a washboard — it’s Passé.
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now