Solved

high percentage on 'iowait'

Posted on 2014-03-05
4
323 Views
Last Modified: 2014-05-01
I got the problem on the 'root' partition of the Linux RHEL3 server which is suddenly having the partition become read only. The case solve by rebooting the server.
On checking the 'top' status, it is found that an abnormal behavior is having the 'iowait' persist at a high rate and at the time of the incident, the 'iowait' always reaches 90-100 %
Will this be the cause of the partition becoming read only ?
If it is, what is the possible reason of the 'iowait' becoming high ?
Last but most important, how to solve this issue ?
Below showing that the iowait reaches 100% this morning and then the partition cannot write again.

03:44:02  up  7:02,  0 users,  load average: 0.00, 0.00, 0.00  
62 processes: 61 sleeping, 1 running, 0 zombie, 0 stopped
CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
           total    0.0%    0.0%    0.0%   0.0%     0.0%  100.0%    0.0%
Mem:  2056864k av,  175268k used, 1881596k free,       0k shrd,   67172k buff
       111756k active,              25436k inactive
Swap: 4192956k av,       0k used, 4192956k free                   66368k cached
0
Comment
Question by:francis_lai
4 Comments
 
LVL 34

Expert Comment

by:Seth Simmons
ID: 39908332
have you looked at /var/log/messages?
when this happens, try running iostat 3 to see what device is being accessed

could also be a device causing i/o wait...maybe a failing disk?
0
 
LVL 13

Accepted Solution

by:
Daniel Helgenberger earned 500 total points
ID: 39908773
My best guess is the I/O operations are timing out. Thus you have high I/O waits and high CPU load (on a low percentage, check with top). These IO - waits are causing the 'read only' state of the root volume.

This is most likely a failing disk; which could be since I assume the installation is about five to seven years old (RHEL3).

Is the root volume on a RAID device?

You should, however, use rsync or any other backup tool to constantly mirror the contents of your root device and switch disks ASAP.
0
 
LVL 34

Expert Comment

by:Duncan Roe
ID: 39910698
I agree with Daniel. The only reason Linux flips a disk over to read-only is an I/O failure
0
 
LVL 48

Expert Comment

by:Tintin
ID: 39911035
What's the output of

df -h

and

iostat -x 5
0

Featured Post

Best Practices: Disaster Recovery Testing

Besides backup, any IT division should have a disaster recovery plan. You will find a few tips below relating to the development of such a plan and to what issues one should pay special attention in the course of backup planning.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

SSH (Secure Shell) - Tips and Tricks As you all know SSH(Secure Shell) is a network protocol, which we use to access/transfer files securely between two networked devices. SSH was actually designed as a replacement for insecure protocols that sen…
Data center, now-a-days, is referred as the home of all the advanced technologies. In-fact, most of the businesses are now establishing their entire organizational structure around the IT capabilities.
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

840 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question