How to fix inode errors on fedora 17 on my VPS

Hi,

Last night I was working on one of my VPS from (DigitalOcean) and suddenly the hard drive got into a read-only state.
I couldn't delete files even with root!?!

I rebooted and then the VPS went into a recovery mode. DigitalOcean Support was able to make it boot but  I am seeing errors in the log. Also I have tried backing up my files I created tar.gz and zip file but were corrupted.

Do you have any idea how to fix it the VPS? (I didn't have backups for that VPS)

cat /var/log/messages  | grep EXT4-fs
Jul 24 13:55:52 zeus kernel: [    2.258134] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
Jul 24 13:55:52 zeus kernel: [    2.678051] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Jul 24 13:55:52 zeus kernel: [    3.194671] EXT4-fs (vda): re-mounted. Opts: errors=remount-ro
Jul 24 15:21:09 zeus kernel: [    2.461322] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
Jul 24 15:21:09 zeus kernel: [    2.654158] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Jul 24 15:21:09 zeus kernel: [    3.721073] EXT4-fs (vda): re-mounted. Opts: errors=remount-ro
Jul 25 06:51:42 zeus kernel: [    1.480130] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
Jul 25 06:51:42 zeus kernel: [    1.557910] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Jul 25 06:51:42 zeus kernel: [    2.465928] EXT4-fs (vda): re-mounted. Opts: errors=remount-ro
Aug  5 22:09:18 zeus kernel: [    1.842273] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
Aug  5 22:09:18 zeus kernel: [    1.963574] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Aug  5 22:09:18 zeus kernel: [    2.669384] EXT4-fs (vda): re-mounted. Opts: errors=remount-ro
Sep  5 18:06:19 zeus kernel: [    1.838704] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
Sep  5 18:06:19 zeus kernel: [    1.934611] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Sep  5 18:06:19 zeus kernel: [    2.430845] EXT4-fs (vda): re-mounted. Opts: errors=remount-ro
Sep  5 18:14:01 sandbox kernel: [    1.708227] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
Sep  5 18:14:01 sandbox kernel: [    1.750590] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Sep  5 18:14:01 sandbox kernel: [    1.970721] EXT4-fs (vda): re-mounted. Opts: errors=remount-ro
Sep 19 06:04:47 sandbox kernel: [    1.929715] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
Sep 19 06:04:47 sandbox kernel: [    2.010668] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Sep 19 06:04:47 sandbox kernel: [    2.655082] EXT4-fs (vda): re-mounted. Opts: errors=remount-ro
Sep 19 08:07:06 sandbox kernel: [    1.677971] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
Sep 19 08:07:06 sandbox kernel: [  223.498300] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Sep 19 08:07:06 sandbox kernel: [  224.228788] EXT4-fs (vda): warning: mounting fs with errors, running e2fsck is recommended
Sep 19 08:07:06 sandbox kernel: [  224.229991] EXT4-fs (vda): re-mounted. Opts: errors=remount-ro
Sep 19 08:12:06 sandbox kernel: [  524.256107] EXT4-fs (vda): error count: 261
Sep 19 08:12:06 sandbox kernel: [  524.256285] EXT4-fs (vda): initial error at 1379575605: ext4_lookup:1044: inode 792674
Sep 19 08:12:06 sandbox kernel: [  524.256592] EXT4-fs (vda): last error at 1379576372: ext4_remount:4418: inode 544641

Open in new window



[root@sandbox slavi]# mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=509004k,nr_inodes=127251,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
/dev/vda on / type ext4 (rw,relatime,errors=remount-ro,user_xattr,barrier=1,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=21,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tmpfs on /media type tmpfs (rw,nosuid,nodev,noexec,relatime,mode=755)
mqueue on /dev/mqueue type mqueue (rw,relatime)

Open in new window

svetoslavmAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

slubekCommented:
Because it is root partition on remote server, you cannot unmount that partition before fsck. That means you have to force fsck on reboot. 'man 8 shutdown' says that you can do it with:
# shutdown -rF now

Open in new window

Another way is creating forcefsck file on partition you want to fsck on reboot:
# touch /forcefsck

Open in new window

Good luck!
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.