/var becomes read-only & can't create transaction lock on /var/lib/rpm

Anyone has any idea

# ls -l /var/lib/rpm/.rpm.lock
-rw-r--r--. 1 root root 0 Jul 25  2012 /var/lib/rpm/.rpm.lock
# chmod 755 /var/lib/rpm/.rpm.lock
chmod: changing permissions of `/var/lib/rpm/.rpm.lock': Read-only file system
# rpm -e ds_agent
error: can't create transaction lock on /var/lib/rpm/.rpm.lock (Read-only file system)

# chmod 755 /var/lib/rpm/.rpm.lock
chmod: changing permissions of `/var/lib/rpm/.rpm.lock': Read-only file system
Any suggestions to address the following?  I can't do 'rpm -e ...' nor 'rpm -Uvh ...'

# ls -lad /var
drwxr-xr-x. 22 root root 4096 Jul 25  2012 /var
# ls -lad /var/lib
drwxr-xr-x. 29 root root 4096 Oct 25  2012 /var/lib

# touch /var/lib/tst.dat
touch: cannot touch `/var/lib/tst.dat': Read-only file system
# touch /var/tst.dat
touch: cannot touch `/var/tst.dat': Read-only file system
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

andreasSystem AdminCommented:
Usually filesystems got read only when there is a problem in the logic structure of the filesystem or when the physical disc where the FS resides is faulty/defective.

You should check the physical disc where your /var filesystem is located for defective blocks, and run an fsck on the /var filesystem.

The fscheck (fsck) needs to be done when the filesystem is offline (not mounted).

If its a normal s-ata disk you might check the smart values for any problems (e.g. defective sectors, relocated sectors, pending sectors, etc.), it might give hints if the disk has problems. But good smart values does not in all cases mean the disk is not defective.
touch /forcefsck

That should fix filesystem damage if any.

Next enable remote syslog collection somewhere on other server, so that you see why and how /var becomes read-only (before it could capture the same information on the disk)

You can try mount -o remount,rw /var
to get it back but it may or may not work.

What is the hardware you are running on?

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
sunhuxAuthor Commented:
Is it possible to verify the FS first without reboot ie
in a non-disruptive manner as I don't have permission
to reboot yet
Active Protection takes the fight to cryptojacking

While there were several headline-grabbing ransomware attacks during in 2017, another big threat started appearing at the same time that didn’t get the same coverage – illicit cryptomining.

sunhuxAuthor Commented:
& fsck needs to be run, to run it in readonly (& report if there's issues) without
making any repair
andreasSystem AdminCommented:
No it needs to be rebooted, at least with /var unmounted but usually its not possible to unmount /vr on a running system.

Your system might get serious troubles if you run it prolonged time in this kind of state.
e.g. logging not working anymore. /var/lib/mysql cannot be written to, etc. pp.
sunhuxAuthor Commented:
Is there any logs that could indicate there's a problem with the FS?
I'll need justification to get a reboot
They should have been written to /var
I repeat - transfer logs to other machine over network.
andreasSystem AdminCommented:
As gheist said, logs are not there until you configure logging over the network.

Furthermore if /var lies on a HDD that also hosts other partitions you may endanger your data on the other partitions too if the HDD slowly dies. b4 reboot ensure you have a backup of your data and the servers important configuration. So you can restore the OS and Data more fast if the old installation wont boot due to a failed HDD.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux Security

From novice to tech pro — start learning today.