Got EXT3-fs error (device dm-4), how could I identify the hard disk and fix the issue?

I got a server failure on last friday/22th-23rd. The whole file system was unmounted and remounted as read-only. I have to restart the server and the fscheck found some errors and fixed them. Today when I am checking the root reason for this failure, I got confused and don't know which hard drive has the problem, could you help me how to identify the dm-4 device? thank you.


Aug 22 19:56:45 luna ntpd[3381]: synchronized to LOCAL(0), stratum 10
Aug 22 20:08:35 luna kernel: EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 8749201 in dir #91528712
Aug 22 20:08:35 luna kernel: Aborting journal on device dm-4.
Aug 22 20:08:35 luna kernel: EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 8749205 in dir #91528712
Aug 22 20:08:35 luna kernel: EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 8749211 in dir #91528712
Aug 22 20:08:35 luna kernel: EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 8749207 in dir #91528712
Aug 22 20:08:35 luna kernel: EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 8749198 in dir #91528712
Aug 22 20:08:35 luna kernel: EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 8749188 in dir #91528712
Aug 22 20:08:35 luna kernel: EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 8749213 in dir #91528712
Aug 22 20:08:35 luna kernel: EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 8749195 in dir #91528712
Aug 22 20:08:35 luna kernel: EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 8749193 in dir #91528712
Aug 22 20:08:35 luna kernel: EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 8749191 in dir #91528712
Aug 22 20:08:35 luna kernel: ext3_abort called.
Aug 22 20:08:35 luna kernel: EXT3-fs error (device dm-4): ext3_journal_start_sb: Detected aborted journal
Aug 22 20:08:35 luna kernel: Remounting filesystem read-only
Aug 22 20:15:13 luna kernel: printk: 21 messages suppressed.
Aug 22 20:15:13 luna kernel: EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 8749203 in dir #92275845
Aug 22 20:30:12 luna ntpd[3381]: synchronized to 10.10.4.10, stratum 3
Aug 22 20:42:08 luna kernel: EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 8749201 in dir #91528712
Aug 22 20:42:08 luna kernel: EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 8749205 in dir #91528712
Aug 22 20:42:08 luna kernel: EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 8749211 in dir #91528712
Aug 22 20:42:08 luna kernel: EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 8749207 in dir #91528712
Aug 22 20:42:08 luna kernel: EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 8749198 in dir #91528712
Aug 22 20:42:08 luna kernel: EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 8749188 in dir #91528712
Aug 22 20:42:08 luna kernel: EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 8749213 in dir #91528712
Aug 22 20:42:08 luna kernel: EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 8749195 in dir #91528712
Aug 22 20:42:08 luna kernel: EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 8749193 in dir #91528712
Aug 22 20:42:08 luna kernel: EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 8749191 in dir #91528712
Aug 22 21:04:25 luna ntpd[3381]: time reset +1.346179 s
Aug 22 21:08:27 luna ntpd[3381]: synchronized to LOCAL(0), stratum 10
Aug 22 21:09:33 luna ntpd[3381]: synchronized to 10.10.4.10, stratum 3
Aug 22 21:36:09 luna ntpd[3381]: time reset +0.312662 s
Aug 22 21:39:43 luna ntpd[3381]: synchronized to LOCAL(0), stratum 10
Aug 22 21:40:48 luna ntpd[3381]: synchronized to 10.10.4.10, stratum 3
Aug 22 21:43:21 luna kernel: printk: 21 messages suppressed.
Aug 22 21:43:21 luna kernel: EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 8749203 in dir #92275845
Aug 22 22:34:18 luna ntpd[3381]: synchronized to LOCAL(0), stratum 10
Aug 22 23:51:32 luna ntpd[3381]: synchronized to 10.10.4.10, stratum 3
Aug 23 00:02:14 luna ntpd[3381]: synchronized to LOCAL(0), stratum 10
Aug 23 00:08:08 luna ntpd[3381]: synchronized to 10.10.4.10, stratum 3
Aug 23 00:26:00 luna ntpd[3381]: time reset +0.925380 s
                                                                                                           345,1         26%
Jason YuAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Daniel HelgenbergerCommented:
1.: Hope you have backup in place, now would be the time.
2. Did you really successfully run e2fschk?

Boot your server off from SystemRescueCD and do a e2fsck if this happeds to be your root fs.

You can identify the device mapper dev 4 (dm-4) by using:
df -h
mount |grep ro

Open in new window


Though df should point you to the file system. Or with mount you just print out read only file systems. This will show you the 'real' device.
0
Jason YuAuthor Commented:
I already ran e2fsck when I was restarting this server. It's not the root fs, it's the partition for GIS software.

I ran the commands, the output is as below:

[root@luna ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/OS-root   2.0G  872M 1011M  47% /
/dev/mapper/OS-tmp    3.9G  137M  3.6G   4% /tmp
/dev/mapper/OS-usr    4.9G  3.0G  1.7G  65% /usr
/dev/mapper/OS-var    3.9G  3.5G  273M  93% /var
/dev/mapper/GIS-LogVol00
                      408G  331G   57G  86% /u1
/dev/sda3              99M   27M   68M  28% /boot
tmpfs                 3.0G  1.8G  1.3G  58% /dev/shm
[root@luna ~]# mount |grep ro
/dev/mapper/OS-root on / type ext3 (rw)
proc on /proc type proc (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
[root@luna ~]#
0
MazdajaiCommented:
It is a lvm device, try -

lvdisplay|awk  '/LV Name/{n=$3} /Block device/{d=$3; sub(".*:","dm-",d); print d,n;}'

Open in new window

0
Newly released Acronis True Image 2019

In announcing the release of the 15th Anniversary Edition of Acronis True Image 2019, the company revealed that its artificial intelligence-based anti-ransomware technology – stopped more than 200,000 ransomware attacks on 150,000 customers last year.

Daniel HelgenbergerCommented:
Yep, it's a LV. If e2fsck is running through and you still have these issues you might have a failing drive.

First run a run a 'deep' fschk:
umount /u1
e2fschk -c -v /dev/mapper/GIS-LogVol00 

Open in new window


If you still have the error messages I would recommend a full write test:

1. Copy  everything from /u1 to a save location as long as you can
2. Recreate (format) /u1 and write it full:
umount /u1
mkfs.ext3 /dev/mapper/GIS-LogVol00
mount /u1
dd if=/dev/zero of=/u1/test.file

Open in new window

While this runs open a second terminal and watch your logfile:
tail -f /var/log/messages

Open in new window


When you run into errors there replace the drive; or the RAID controller - depends on your setup.
0
Jason YuAuthor Commented:
It looks like a logic volume:

[root@luna pdfdocs]# lvdisplay|awk  '/LV Name/{n=$3} /Block device/{d=$3; sub(".*:","dm-",d); print d,n;}'
dm-4 /dev/GIS/LogVol00
dm-0 /dev/OS/root
dm-1 /dev/OS/tmp
dm-2 /dev/OS/usr
dm-3 /dev/OS/var
[root@luna pdfdocs]#
0
Daniel HelgenbergerCommented:
See, came to my mind at last! Its late, I go to sleep:
dmsetup ls

Open in new window


This way you find your dm-4
0
MazdajaiCommented:
Are you able to determine the hard disk setup and run e2fsck on the volume?
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Jason YuAuthor Commented:
I will try to run it this weekend. It's our EDI server, I can't run it on standard working hours.
0
Jason YuAuthor Commented:
[root@luna pdfprc]# dmsetup ls
OS-tmp  (253, 1)
GIS-LogVol00    (253, 4)
OS-usr  (253, 2)
OS-var  (253, 3)
OS-root (253, 0)
[root@luna pdfprc]#
0
MazdajaiCommented:
Sounds good. Make sure you have a back up.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.