Question

what caued 'cannot find /usr/lib/ld.so.1' error?

Asked by: hualing

Last week, I shutdown a server as usual, the next day, the server started up with the message
'cannot find /usr/lib/ld.so.1'.
With this message, almost all the command such as ls, mv, cp do not work.
It can not be boot as single mode either.

Has it be hacked? OR the HD error?


I also found a post on sun website, i don't understand it well

http://sunsolve.sun.com/pub-cgi/retrieve.pl?doc=fsalert%2F57469#Synopsis

I do hope to avoid this issue next time. How chould I improve?

thank you in advance!

This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.

Subscribe now for full access to Experts Exchange and get

Instant Access to this Solution

  • Plus...
  • 30 Day FREE access, no risk, no obligation
  • Collaborate with the world's top tech experts
  • Unlimited access to our exclusive solution database
  • Never be left without tech help again

Subscribe Now

Asked On
2004-03-07 at 21:51:41ID20910530
Tags

cannot

,

find

Topic

Sun Solaris

Participating Experts
3
Points
100
Comments
15

Trusted by hundreds of thousands everyday for fast, accurate and reliable tech support.

  • "The time we save is the biggest benefit of Experts Exchange to Warner Bros. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange." Mike Kapnisakis, Warner Bros.
  • "Our team likes having a resource that is more secure than just using Google and most experts using this service really know their stuff. It's nice to look here first versus using Google." Dayna Sellner, Lockheed Martin
  • "Anytime that I've been stumped with a problem, 9 out of 10 times Experts Exchange has either the accepted solution or an open discussion of the potential solution to the problem." Kenny Red, eBay Inc.

See what Experts Exchange can do for you.

Got a question?

We've got the answer.

Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.

Screenshot of Experts Exchange Knowledgebase

Need individual assistance?

Our experts are ready to help.

If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.

Screenshot of Experts Exchange Knowledgebase

Want to learn from the best?

Read articles from industry experts.

Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.

Screenshot of an Article

Working on a long term project?

Store your work and research.

Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.

Screenshot of Experts Exchange Knowledgebase

Access the answers to your technology questions today.

Subscribe Now

30-day free trial. Register in 60 seconds.

What Makes Experts Exchange Unique?

Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.

Trusted by the world's most respected brands.

image of each brand's logo

Faithfully serving IT professionals since 1996.

Experts Exchange Logo

Try it out and discover for yourself.

Subscribe Now

30-day free trial. Register in 60 seconds.

Related Solutions

  1. HELP! issued mv /usr /usrbk: nothing works.
    I issued the following command on Solaris: mv /usr /usrbk (I moved the /usr dir to /usrbk).. Now, I can't issue any real commands.. they all say "cannot find /usr/lib/ld.so.1" I thought I could rename it back when the problem started.. I was just going to mount a co...
  2. Unix /usr filesystem full
    1) What is the reason that /usr filesystem is full? Is there any files that can be removed in Sun Sparc Solaris 2) How can you find the filenames of files having size more than 1mb(say) in Sun Sparc Solaris. Please give me any readymade scripts if any
  3. renaming a dir (/usr  to /ousr and /nusr to /usr)
    My setup was really poorly done and I have been moving things around and renaming directories like /home to get more space, but it looks like I need to mount /usr on the new large partition so I have to boot up into linux 2 and do the work at command line. Too bad I can only...
  4. problem with '/usr/lib/sendmail'
    #!/usr/local/bin/perl ## ## printenv -- demo CGI program which just prints its environment ## #print "Content-type: text/html\n\n"; use CGI; use strict; use CGI::Carp qw(fatalsToBrowser); #### define all variables that we will use my $SENDMAIL = '...
  5. I am unable to unmount /usr
    I was trying to install gcc-3.3.2 on a sun ultra 10 sparcstation. But the /usr directory was full so i want to add the space used by /opt to /usr. When I try to umount /usr I got this error: umount: /usr busy. How can I unmount /usr and /opt so that I can increase the space t...

Free Tech Articles

  1. WARNING: 5 Reasons why you should NEVER fix a computer for free.
    It is in our nature to love the puzzle. We are obsessed. The lot of us. We love puzzles. We love the challenge. We thrive on finding the answer. We hate disarray. It bothers us deep in our soul. W...
  2. SCCM OSD Basic troubleshooting
    SCCM 2007 OSD is a fantastic way to deploy operating systems, however, like most things SCCM issues can sometimes be difficult to resolve due to the sheer volume of logs to sift through and the dispe...
  3. Migrate Small Business Server 2003 to Exchange 2010 and Windows 2008 R2
    This guide is intended to provide step by step instructions on how to migrate from Small Business Server 2003 to Windows 2008 R2 with Exchange 2010. For this migration to work you will need the fo...
  4. Create a Win7 Gadget
    This article shows you how to create a simple "Gadget" -- a sort of mini-application supported by Windows 7 and Vista. Gadgets can be dropped anywhere on the desktop to provide instant information, ...
  5. Outlook continually prompting for username and password
    There have been a lot of questions recently regarding Outlook prompting for a username and password whilst using Exchange 2007. There are a few reasons why this would happen and I will try to cover t...
  6. Backup Exchange 2010 Information Store using Windows Backup
    There seems to be quite a lot of confusion around the ability to backup Exchange 2010 using the built in Windows Backup feature. This stems from the omission of this feature prior to Exchange 2007 s...

Cloud Class Webinars

  1. Avoiding Bugs in Microsoft Access
    Alison Balter takes and in-depth look at avoiding bugs in Access. In this webinar you will learn about using the immediate window to debug your applications, invoking the debugger, using breakpoints to troubleshoot, stepping through code, setting the next statement to execute, ...
  2. Top 10 Best New Features in Visio 2010
    Scott Helmers gives live demonstrations of the top 10 new features in Visio 2010. This webinar will teach you how to create compelling diagrams by adding shapes to the page with a single click, linking the shapes in a diagram to data in Excel (or SQL Server, or SharePoint), ...
  3. IT Consultant Business Secrets Revealed
    Michael Munger, Experts Exchange tech pro and IT consultant, pulls back the curtain on his very successful businesses and answers question on every IT consultant and business owner should know about. He shares secrets on what he did to solve the 5 most common problems in IT, ...
  4. Disaster Recovery and Business Continuity
    Quest CTO, Mike Billon, gives an overview of the steps involved in building a dunamic disaster recovery plan. Through case studies and an examination of software/hardware tooles for monitoring and testing, you'll gain a better understandin of where you are, where you want ...
  5. Organize Your Visio Diagrams with Containers and Lists
    Scott Helmers uses cross functional flowcharts, wireframe diagrams, data graphic legends and seating charts to teach you: how to ustilize all three new structured diagram components in Visio 2010, the best practices for organizeing shapes in previous version of Visio, how to organize ...
  6. How to Us Objects, Properties, Events and Methods in Microsoft Access
    Alison Dalter gives an in-depbth look at objects, properties, events and methods in Microsoft Access. In this webinar you will learn about using the object browser, referring to objects, working with properties and methods, working with object variables, understanding the ...

Join the Community

Give a Little. Get a Lot.

Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.

Join the Community

Answers

 

by: yuzhPosted on 2004-03-07 at 22:40:03ID: 10538558

Do you have Sun Trunking 1.3 Utility installed on your system? If you have, please following the instructions in the sunsolve document to apply the patch(s).

If you don't have Sun Trunking, you need to have a look at /var/adm/messages* file (it is likely
your vi, more command does not work any more), boot up your system from CDROM, mount
the filesystem and have look at the file.

It could be your system has been HACKED, the hacker has replace some of your system binary.
if you have another system with the same patch level, you can compare the checksums value
of the binary files. eg, you mount /usr filesystem in /mnt,

cksum /mnt/bin/ls
in the other box (you think it is good)
chsum /usr/bin/ls

(please check ps, find, netstat, su, passwd as well)

If your system has been hacked, sorry, you need to do a fresh install (or restore from a reliable
backup -- you need to 100 sure that you backup is the good one, not after the system was hacked).

You need to harder Solaris, disable all the unwanted services.

Good luck!



 

by: hualingPosted on 2004-03-08 at 01:47:55ID: 10539236

yuzh
 
Thank you very much for you reply.
I am not sure whether I have trunking 1.3 utility, acturally I don't know what is it for.

I will follow your suggestion to see how it is going.

It is ok to re-install the system, but I really want to know how I could avoid this kind of attack as much as possiable.
I was afraid the server would be hacked, so the day before it could not startup, I did shutdown the server.
But the next day, ...
Is it possiable that the server is hacked and still running well before rebooting?

Could you teach me more on how to protect the server from hacking...

thank you again.

 

by: ahoffmannPosted on 2004-03-08 at 02:05:34ID: 10539314

listening ..

 

by: yuzhPosted on 2004-03-08 at 19:26:38ID: 10546807

To figure out if your system was hacked will take a lot of time. Since your system is not
functioning, you need to boot up to single user mode use CDROM or a boot server, then
mount the filesystem from the HD and check it out. (Assume that you know how to mount
the filesystem, and the mount point is /mnt, to find out the device name of the filesystem,
you can do: format -> verify to find out).

You need to check the etc dir to see if any configuration file change, check the passwd file
to see if the hacker has added some users, also, check the dev dir:

find /mnt/dev -type f -print
to see if strange file sitting there.

If you decide to rebuild your system, please rember change the root password, and ask your
users to change their password.

To make your system more secure:

1. Always keep your system up to date, apply the the latest Recommended Patch Clusters
    and read Security Sun Alerts from SunSolve:
    http://sunsolve.sun.com/pub-cgi/show.pl?target=home

    Sun update their Recommended Patch Clusters at least once a month, you should try to keep
your system up to date when possible.

2. Disable all the unwanted services, eg:
     vi /etc/inetd.conf
     command out (put the # in the begining out the line), telnet, ftp, uucp, talk .....etc, etc.

    Disable some of the unwanted service at startup:
    eg:
          cd /etc/rc2.d
              mv S88sendmail s88sendmail
              mv S70uucp s70uucp
              mv S47asppp s47asppp
       cd /etc/rc3.d
              mv S76snmpdx s76snmpdx     (/etc/rc3.d)
              mv S77dmi s77dmi           (/etc/rc3.d)

              vi /etc/default/login
              # Allow 3 login attempts
              RETRIES=3
   
Make system accounts have no login shell, add /bin/false shell
   to system accounts in /etc/passwd file : (eg, daemon:x:1:1::/:/bin/false)

   vi /etc/passwd
   daemon
   bin
   sys
   adm
   lp
   smtp
   uucp
   listen
   nobody
   noaccess
   nobody4

     Disable the ability to execute code from the stack.
       vi /etc/system file , add:

       * Disable the ability to execute code from the stack
       * This will actively prevent many buffer overflows
       set noexec_user_stack=1

       * This will report buffer overflows
       set noexec_user_stack_log=1

       Stop hackers find out server information via TCP/IP
   Make sure that hackers and DDoS attackers cannot gather information about
   the operating system of the server

   vi /etc/default/inetinit (TCP_STRONGS_ISS=2)
   Set the value here to 2, the RFC 1948 sequence number standard, and then
   reboot the server.

   To make TCP logging all the incoming TCP connetion:
   make sure /etc/init.d/inetsvc has
   /usr/sbin/inetd -s -t

   You can also  Disable kernel routing, vi /etc/rc2.d/S69inet (too much typing!)
   man ndd

 3. Install tcp_wrappers , Openssh on your system, use ssh and sftp instead of
     telnet and ftp. you can get them from:
     http://sunfreeware.com/
 
4. I suggest you to download the Solaris security tool -- JASS and install it on
    your system, easy to setup:
    http://wwws.sun.com/software/security/downloads.html


5. Other Docs and security tools:
   
    UNIX Security Checklist
    http://www.cert.org/tech_tips/usc20_essentials.html

    YASSP is "Yet Another Solaris Security package:
    http://www.yassp.org/

     Comparison of Solaris Hardening Scripts
     http://www.boran.com/security/sp/comparison1.html

     Solaris Security FAQ:
     http://www.itworld.com/Comp/2377/security-faq/


      Good luck!

   




 

by: NisusPosted on 2004-03-09 at 08:42:19ID: 10551862

Hi,

Assuming that this is not a security breach, it could be a problem with the filesystem.

When the server rebooted, did it prompt you to manually fsck any filesystems?

Is /usr a separate partition?

I see you can login to the box so do this please:

/sbin/sh

echo /*
echo /usr/*
echo /usr/lib/*

If the /usr/lib/ld.so.1 file is indeed missing or corrupt you can use the copy which is /etc/lib/ld.so.1

Another critical file is /usr/lib/libc.so.1

You will probably need to boot from another device to resolve this problem.  Can you boot from CD if necessary?

Regards, Nisus
http://www.omnimodo.com

 

by: hualingPosted on 2004-03-10 at 00:34:55ID: 10558894

Dear yuzh

I made a new installation on a new disk.
1. I have installed the latest patch.
2. I am not sure what service is not need, so i did not change anything in inetd.conf.
I know definatly we need telnet and FTP..
I disabled the unwanted service as you suggested at startup
/etc/init.d/inetsvc has a line
/usr/sbin/inetd -s &
i did not change it...

BTW : how can I change the umask of root , what does 027 for? Is it nessary to do that, I saw some article mentioned that.

I will try to add more security tools later.

Regarding the bad disk.
I tried to boot from cdrom, single user mode, mount the disk after that as /a
and copied /etc/lib/ld.so.1 to /a/usr/lib/
but during start up it still says can not find ld.so.1.

I also tried to fsck.
all slice are successfully done except s1
#fsck -F ufs /dev/dsk/c1t0d0s1
it says BAD SUPER BLOCK: MAGIC NUMBER WRONG

I did
# newfs -N /dev/rdsk/c1t0d0s1
had got many super-block backups at 32, 92592, 277712,../8138144 etc.
then
# fsck -F  ufs -o b=32 /dev/dsk/c1t0d0s1
it still show the same magic number wrong error.

Any Good suggesions?

 

by: ahoffmannPosted on 2004-03-10 at 04:34:22ID: 10560023

> .. umask .. what does 027 for?
enshures that all files and dirs created by root have no world permission and only read/write for group

> it still show the same magic number wrong error
sounds like your disk has bad blocks.
Either format the disk low-level, or exclude the reported blocks by newfs

 

by: hualingPosted on 2004-03-10 at 16:59:01ID: 10566854

ahoffmann,

>enshures that all files and dirs created by root have no world permission and only read/write for group
how to set the umask to 027?


>sounds like your disk has bad blocks.
>Either format the disk low-level, or exclude the reported blocks by newfs
with bad blocks, is it possiable to have it repaired and have all the data suvived?

If not, I will reinstall solaris, SHOULD me repair the block before installation?

thank you !

 

by: yuzhPosted on 2004-03-10 at 19:52:36ID: 10567551

--- "how to set the umask to 027?"

put:
umask  027

in root's .profile

-- "with bad blocks, is it possiable to have it repaired and have all the data suvived?"

You can try to use "format" command to repair the HD. try to copy the data to tape or
another filesystem.

"I know definatly we need telnet and FTP.."
    FTP and TELNET - plain text password! use ssh and sftp instead of telnet and ftp .
    In case you still have good reason to keep them,  install  tcp_wrappers and then
    configure only the trust user from trust can telnet/ftp to your system.
   
   DO NOT allow root ftp, create a  /etc/ftpusers file, and put the followings in the file:
root
daemon
bin
sys
adm
lp
uucp
nuucp
listen
nobody
noaccess
nobody4

    then
 chmod 600 /etc/ftpusers  

   Please read my comment (including the links):
   http://www.experts-exchange.com/Operating_Systems/Solaris/Q_20910530.html#10546807

   Good luck!









 

by: hualingPosted on 2004-03-10 at 21:19:21ID: 10567898

yuzh

Last question,
I am going to cpy everything from a good disk to the bad one.

Should I  repaire the bad one before copying?

How to copy between 2 HD?

How can I know which one is the good one.

IF I copy from bad to good one, that is too bad!!

 

by: ahoffmannPosted on 2004-03-10 at 23:24:01ID: 10568348

> Should I  repaire the bad one before copying?
no.

> How to copy between 2 HD?
2 possibilities:
  1. cd /disk2/dir; (cd /root-of-disk1 && tar cf - * .[^.]*)|tar xf -
  2. cd / && dd if=`df -k . | awk '{print $1}'` of=/disk2/disk1.dd.dump bs=1024 count=`df -k . | awk '{print $2}'`
while tar may fail with the bad blocks, dd copies everything including the bad blocks (if possible)

> reinstall solaris, SHOULD me repair the block before installation?
no.
You need to lowlevel format first, then use newfs on that partition excluding the bad blocks.

IMHO, a disk with bad blocks is subject for the trash-can

 

by: hualingPosted on 2004-03-11 at 05:31:18ID: 10570955

BAD news

I tried DD to copy everything from another disk

# dd if=/dev/dsk/c1t0d0s2 of=/dev/c1t1d0s2 bs=1024b
dd: unexpected short write, wrote 147456 bytes, expected 524288
7829+0 records in
7829+0 records out
# Mar 11 16:14:21 m001 ufs: NOTICE: alloc: /: file system full

i boot with that disk, failed.

I gave up and begain installation from DVD.

But I was not asked to divid the patition, and it finished.

What should I do with the patition?
This is the first time i use GUI to do the installation, the COM1 doesn't work.

SAD:(

 

by: NisusPosted on 2004-03-11 at 11:58:43ID: 10574737

Looks like you got the dd command wrong.  You wrote to /dev instead of /dev/dsk.  You could possibly remove /dev/c1t1d0s2  and try again.  The geometry of the disks should be identical for this procedure.

I would have done this (using the raw device):

dd if=/dev/rdsk/c1t0d0s2 of=/dev/rdsk/c1t1d0s2 bs=4096

Regards, Nisus.

 

by: yuzhPosted on 2004-03-11 at 17:52:07ID: 10577367

you need to get rid of the /dev/c1t1d0s2 file (output from you dd command!)

If the Harddisk are not identical, don't use "dd", you can find a lot of infor about how to clone
a disk in this TA, just do a  search.

 

by: ahoffmannPosted on 2004-03-12 at 00:00:25ID: 10578756

> If the Harddisk are not identical, don't use "dd", ..
hmm, as long as the target is as least as large as the source, and the same fs, I don't see a problem

20120131-EE-VQP-002

3 Ways to Join

30-Day Free Trial

The Experts

98% positive feedback on 31,087 answers since March 2000. angeliii is a Microsoft Most Valuable Professional for his work with MS SQL Server & Develoment.

He has also proven his knowledge of Visual Basic Programming, PHP Scripting and Oracle Databases.

The Experts

97% positive feedback on 10,752 answers since July 2000. lrmoore has more than 18 years experience in the networking industry.

The six-time Mircosoft MVPs specialties include firewalls, virtual private networking, and network management.

Testimonials

"...and excellent source for support... Kind of like having your very own IT dept." Electriciansnet

Testimonials

"I was apprehensive at signing up at first. However... it has already made my life as an IT administrator much easier." JaCrews

Testimonials

"WOW! You guys have great, active, and knowledgeable people on here." moore50

Business Clients

Business Clients

In the Press

"If you’ve got a question... Experts Exchange can supply an answer.”

In the Press

"...an invaluable aid for both IT professionals and those who require tech support."

In the Press

"where IT professionals provide quick answers on just about any topic"

Business Account Plans

Loading Advertisement...