Solved

high percentage on 'iowait'

Posted on 2014-03-05
4
318 Views
Last Modified: 2014-05-01
I got the problem on the 'root' partition of the Linux RHEL3 server which is suddenly having the partition become read only. The case solve by rebooting the server.
On checking the 'top' status, it is found that an abnormal behavior is having the 'iowait' persist at a high rate and at the time of the incident, the 'iowait' always reaches 90-100 %
Will this be the cause of the partition becoming read only ?
If it is, what is the possible reason of the 'iowait' becoming high ?
Last but most important, how to solve this issue ?
Below showing that the iowait reaches 100% this morning and then the partition cannot write again.

03:44:02  up  7:02,  0 users,  load average: 0.00, 0.00, 0.00  
62 processes: 61 sleeping, 1 running, 0 zombie, 0 stopped
CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
           total    0.0%    0.0%    0.0%   0.0%     0.0%  100.0%    0.0%
Mem:  2056864k av,  175268k used, 1881596k free,       0k shrd,   67172k buff
       111756k active,              25436k inactive
Swap: 4192956k av,       0k used, 4192956k free                   66368k cached
0
Comment
Question by:francis_lai
4 Comments
 
LVL 34

Expert Comment

by:Seth Simmons
ID: 39908332
have you looked at /var/log/messages?
when this happens, try running iostat 3 to see what device is being accessed

could also be a device causing i/o wait...maybe a failing disk?
0
 
LVL 13

Accepted Solution

by:
Daniel Helgenberger earned 500 total points
ID: 39908773
My best guess is the I/O operations are timing out. Thus you have high I/O waits and high CPU load (on a low percentage, check with top). These IO - waits are causing the 'read only' state of the root volume.

This is most likely a failing disk; which could be since I assume the installation is about five to seven years old (RHEL3).

Is the root volume on a RAID device?

You should, however, use rsync or any other backup tool to constantly mirror the contents of your root device and switch disks ASAP.
0
 
LVL 34

Expert Comment

by:Duncan Roe
ID: 39910698
I agree with Daniel. The only reason Linux flips a disk over to read-only is an I/O failure
0
 
LVL 48

Expert Comment

by:Tintin
ID: 39911035
What's the output of

df -h

and

iostat -x 5
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Hyper-convergence systems have taken the IT world by storm and have quickly started to change our point of view of how the data center should and could be architected. In this article, I’ll explain the benefits of employing a hyper-converged system …
It’s 2016. Password authentication should be dead — or at least close to dying. But, unfortunately, it has not traversed Quagga stage yet. Using password authentication is like laundering hotel guest linens with a washboard — it’s Passé.
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

932 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now