[2 days left] What’s wrong with your cloud strategy? Learn why multicloud solutions matter with Nimble Storage.Register Now

x
?
Solved

high percentage on 'iowait'

Posted on 2014-03-05
4
Medium Priority
?
342 Views
Last Modified: 2014-05-01
I got the problem on the 'root' partition of the Linux RHEL3 server which is suddenly having the partition become read only. The case solve by rebooting the server.
On checking the 'top' status, it is found that an abnormal behavior is having the 'iowait' persist at a high rate and at the time of the incident, the 'iowait' always reaches 90-100 %
Will this be the cause of the partition becoming read only ?
If it is, what is the possible reason of the 'iowait' becoming high ?
Last but most important, how to solve this issue ?
Below showing that the iowait reaches 100% this morning and then the partition cannot write again.

03:44:02  up  7:02,  0 users,  load average: 0.00, 0.00, 0.00  
62 processes: 61 sleeping, 1 running, 0 zombie, 0 stopped
CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
           total    0.0%    0.0%    0.0%   0.0%     0.0%  100.0%    0.0%
Mem:  2056864k av,  175268k used, 1881596k free,       0k shrd,   67172k buff
       111756k active,              25436k inactive
Swap: 4192956k av,       0k used, 4192956k free                   66368k cached
0
Comment
Question by:francis_lai
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
4 Comments
 
LVL 35

Expert Comment

by:Seth Simmons
ID: 39908332
have you looked at /var/log/messages?
when this happens, try running iostat 3 to see what device is being accessed

could also be a device causing i/o wait...maybe a failing disk?
0
 
LVL 13

Accepted Solution

by:
Daniel Helgenberger earned 1000 total points
ID: 39908773
My best guess is the I/O operations are timing out. Thus you have high I/O waits and high CPU load (on a low percentage, check with top). These IO - waits are causing the 'read only' state of the root volume.

This is most likely a failing disk; which could be since I assume the installation is about five to seven years old (RHEL3).

Is the root volume on a RAID device?

You should, however, use rsync or any other backup tool to constantly mirror the contents of your root device and switch disks ASAP.
0
 
LVL 35

Expert Comment

by:Duncan Roe
ID: 39910698
I agree with Daniel. The only reason Linux flips a disk over to read-only is an I/O failure
0
 
LVL 48

Expert Comment

by:Tintin
ID: 39911035
What's the output of

df -h

and

iostat -x 5
0

Featured Post

Tech or Treat!

Submit an article about your scariest tech experience—and the solution—and you’ll be automatically entered to win one of 4 fantastic tech gadgets.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Moving your enterprise fax infrastructure from in-house fax machines and servers to the cloud makes sense — from both an efficiency and productivity standpoint. But does migrating to a cloud fax solution mean you will no longer be able to send or re…
New style of hardware planning for Microsoft Exchange server.
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.
Suggested Courses

649 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question