RPM completely hosed during CentOS upgrade from 4.4 to 5

jashcache used Ask the Experts™
A little history....

Install of CentOS-5 kept ejecting CD saying something like, no CD was found with CentOS install files. So I figured I'd install an older version and upgrade. Since I've never done an upgrade with this distro in this way I used the upgrade "not recommended" on the CentOS.org website.


There were some version differneces between what was on the migration guid and the actual RPM index, but they were all minor so I figured that it was ok to use the later ones in the current index. found here:


When I got to the section to "Remove deps/outdated packages", running the rpm command gave me this:

rpm: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory

Now anything using rpm give errors including yum and I'm to top it off I'm remote and to go reinstall is not impossible but fairly difficult as I am a couple hours away.

I'm currently attempting to reinstall rpm and just discovered that there is no compiler  installed (search for cc and gcc).

Any suggestions are greatly appreciated!!!

Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Michael WorshamCloud/Infrastructure Solutions Architect

Usually when you get a message like 'no CD was found with CentOS install files', it usually means you have a bad ISO.

I recommend downloading a new CentOS ISO. Another option would be to download the CentOS NetBoot ISO, burning it to a CD then using the Internet for downloading/installing CentOS in real-time from one of the CentOS image repositories.

CentOS NetInstall 5.3


I thought so too... I've tried the following list:

CentOS-5.3-i386-LiveCD.iso  - (3 different CD's)
CentOS-5.3-i386-netinstall.iso - (2 different CD's)
CentOS-5.3-i386-bin-DVD.torrent - (3 different DVD's)

Then I tried Debian Etch and that installed no problem, but the most of the software I want to install is not supported for Debian.

So, after that I tried CentOS 4.4. Installed without any trouble.

Thanks for the response though! It seems this is a pretty tough question. I haven't gotten much response on other forums either.
A clean install of the latest version using a good ISO, is probably the best, cleanest idea.   If it's not an option, I might consider using a different  (local)   Linux system for recovery.

* Get  the version of RPM off a known good copy of the CentOS 5  CD  
* Then copy RPM files off the CD  
* Then use rpm2cpio  to convert the RPM archive into a CPIO.
* Then copy the CPIO archive to the damaged system.

* If there is no valuable data on the system, and its uptime isn't critical,  you also have options like copying all binaries from the clean installed system to the damaged one (using rsync),  as long as you are careful to exclude certain items such as  /boot,  system drivers, network settings in /etc, etc.  Make sure to exclude /proc   /dev  and  /sys   /boot, also

For example,  I sometimes  RSYNC a complete system to another complete system, with those exclusions in place,  then after the rsyncs are all done, repeat the rsyncs  using the delete option.
   rsync -avle ssh  /lib  /usr   /var              root@system2:/
   rsync -avle ssh  --delete /lib/                 root@system2:/lib/
   rsync -avle ssh  --delete /usr/                 root@system2:/usr/
   rsync -avle ssh  --delete /var/                 root@system2:/var/
   rpm --erase --justdb kernel
   rpm -qa | grep kernel
   yum install kernel

The caveat is this won't correct config files,   you also need to copy the new major OS version's  /etc  in some way,    but without copying files like modprobe.conf and  network-scripts.      Rsync can do it,  if you get the commands right,  using the --exclude  option.

As for moving a CPIO from one system to the other:

# cd /tmp
# mkdir /mnt/centosdisk
# mount /dev/cdrom   /mnt/centosdisk
# cp /mnt/centosdisk/CentOS/rpm-*.rpm   .

# rpm2cpio rpm-libs-4.4.2-37.el5.i386.rpm  >  rpm-libs-4.4.2-37.el5.i386.cpio
# rpm2cpio rpm-4.4.2-37.el5.i386.rpm > rpm-4.4.2-37.el5.i386.cpio

# scp rpm*.cpio  root@remote-system-ip:/root
# ssh root@remote-system-ip

# mkdir unpack
# cd unpack
# cpio -i --make-directories
[Press Control + D]
Found end of tape.  To continue, type device/file name when ready.

Now instead of a  'RPM'   file,  you have  the files that are inside the RPM.  

If  you have all the files you need for your particular system, and there aren't conflicts   (E.g.  be  VERY careful when doing this to a RPM such as glibc)

You can just copy those files onto the running system.

You'll  want to repair your actual RPM database later,   depending on your circumstances you may use something like

rpm  --erase  --justdb   --nodeps  rpm  rpm-libs
yum install rpm rpm-libs

11/26 Forrester Webinar: Savings for Enterprise

How can your organization benefit from savings just by replacing your legacy backup solutions with Acronis' #CyberProtection? Join Forrester's Joe Branca and Ryan Davis from Acronis live as they explain how you can too.

Michael WorshamCloud/Infrastructure Solutions Architect

Just for kicks, please describe your hardware being used. It quite possibly could be something isn't compatible with the latest version of CentOS, thus causing the problems.

Also if you copy everything  over,  when doing /etc there are some more subtle hardware-specific items,  e.g.  you definitely don't want to copy over  /etc/fstab   .

And  files like /etc/modprobe.conf
You probably also want to leave /etc/ssh   /etc/shadow   and  /etc/passwd   alone.

Full system copying can be an effective strategy, but you definitely need to know what files contain crucial information  required to boot the system that won't work if you just copy them over, and you don't want to erase or overwrite.

/etc   is small,  so it  should be easy enough to make a backup,  to deal with any issues remotely  (short of  _system fails to boot_)

You will also want to run  'ldconfig'   on the new system,  and make sure you  re-install the kernel,  and that goes well before rebooting.

The files in /boot   are heavily system specific,  for example the initrd contains drivers and configs for  your specific hardware,  which is automatically re-generated every time you install a kernel,  you definitely don't want to be copying those.


Thank you very much for the guidance!

I don't have another comparable CentOS machine with access to that network. So, copying may be difficult. Is there a way I can do that from the disk? Even if I could just restore rpm I believe I can correct the repo's, kernal, etc.

It looks like I can do that with the CPIO instructions. I will give those a try.

Not sure about the Server hardware off hand Because I inherited it. Though it had a RAM problem then, so i'm 99% sure on is the motherboard - Intel s3000ah. Any tool you can suggest to tell me what's on it? All I know is "lspci".
Certainly.  Well, it can definitely be useful to have a reference system available, for other reasons.  

As for the error:
rpm: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory

You may want to know that 'libnss3.so'  is not part of RPM or RPM-LIBS, it's part of the NSS library

# rpm -qf /usr/lib/libnss3.so


You can find the actual RPM files on the CentOS  mirrors,  also

Converting the RPM file to a CPIO archive and then extracting the CPIO archive is a way of getting at the files inside any RPM

Here's a link to discussion that includes some alternatives, for extracting a RPM using a rescue disk (such as a Knoppix boot CD): http://linuxmafia.com/~karsten/Linux/FAQs/rpm-unpack-faq.html
That require only you have perl and compile  a special executable:  http://gentoo.osuosl.org/distfiles/rpmoffset.c

 *There is a Perl-based rpm2cpio script which does not  rely on Perl libs. However  it requires a Perl installation, which is unlikely to be found on most common boot disks: http://linuxmafia.com/pub/linux/utilities-general/rpm2cpio

*'Further research turns up an rpm2targz script by Patrick Volkering, of Slackware fame:  http://www.mirror.ac.uk/mirror/ftp.slackware.com/slackware-3.3/contrib/rpm2tgz.tgz


It took a while to get it done,  but after a couple of tries and thanks to the expert help, I was able to get it back to to a minimal OS and back to normal from there.

Thanks again for all your help!!

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial