Emergency Problem with Linux

Hi all,

I have an emergency issue that prevents my web server from booting.  I need to be able to get this thing back up and get it working for the next few weeks with out a reinstall.

At boot time, it enters run level 3, then comes up saying:

INIT: cannot execute "/etc/rc.d/rc.S"
and
INIT: cannot execute "/etc/rc.d/rc.M"

Then I get a login prompt like this:
(none) login:

If I attempt to log in, I type root on the console, it pauses then gets me back to the login prompt again.

I really need help on this and will be watching this for any updates.

I am running slackware 10.2 SMP kernel and have checked permissions, the two files are +X and chmod to 755.

Thanks


Thanks!

LVL 1
smyers051972Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Gregor LambertIT Manager / Systems EngineerCommented:
Hi symers051972

take a look at this link on the linuxquestions site
I hope it helps

http://www.linuxquestions.org/questions/linux-newbie-8/init-cannot-execute-etc-rc-d-rc-s-15517/
0
smyers051972Author Commented:
Im not seeing any of those errors.  What I get is the (none) login prompt and when I try to login, it wont even ask me my password.

I can boot to the partmagic cd and see the file system, did fsck on all partitions and all report clean.
0
smyers051972Author Commented:
here is a screen shot, can only locally connect...

login.png
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

venom96737Commented:
did you install sysvinit with permissions set to 755?
0
smyers051972Author Commented:
Im a little new to linux could you elaborate?
0
venom96737Commented:
during install it is a check box for sysvinit.  Is this a brand new install?   It actually sounds like it didnt install completely to me or that the partitions are incorrect.  If its brand new just clear it and take another stab at it.    Don't worry it took me more than one try to get my first one going also.
0
smyers051972Author Commented:
no it has being running for a few years, somebody else created the box, i just reboot it today and it did come back with that error
0
venom96737Commented:
Well then that would mean that /etc/rc.d is corrupt and that the system cant mount it correctly.  
0
smyers051972Author Commented:
i used partmagic to view the partitions and all the file are there in the etc/rc.d directory
0
smyers051972Author Commented:
Do you think we can call you, or help you dial into our server?  We'll pay you if you can fix it?
0
muffCommented:
Can you post your bootloader config and fdisk -l ?

If grub, it is /boot/grub/menu.lst

It says it is mounting root, but I am guessing that it isn't really, it is mounting something it thinks should be root and then finding nothing is there.

Is root on the raid array?
0
smyers051972Author Commented:
could you elaborate? we're new to this
0
gelonidaCommented:
can you download and burn some small linux installation and then boot fom CD
puppylinux would for example be an option.
you could download for example:
ftp://ibiblio.org/pub/linux/distributions/puppylinux/puppy-2.17.1-nolzma-seamonkey-fulldrivers.iso

but in fact: any linux on a disk would do.

You could then browse your local har disk and provide us with some more output.

What would be interesting is the output of
ls -ltra /etc/
ls -ltra /etc/rc.d

and the contents of
/etc/rc.d/rc.M
and
/etc/rc.d/rc.S
if these files are existing at all.

It might also be interesting to look t your system's log files to get an idea, when and how the machine was last booted successfully.
0
Duncan RoeSoftware DeveloperCommented:
You can get a shell prompt to allow you to do some investigations as follows:-
- get a boot prompt by using up / downarrow key when the boot menu displays
- uparrow to the top (default) kernel again. you should now see a prompt like "boot: mykernel" where mykernel is that top choice. (if there is only 1 kernel in the menu, you have to re-type it)
- Append to this line (i.e. type in) " init=/bin/bash", so the line now looks like "boot: mykernel init=/bin/bash"
- touch <Enter>. Linux boots then you get a root shell prompt. N.B. At this stage, only the root disk is mounted and it is mounted read-only

cd /etc/rc.d
cat -A rc.S | head

Check output fromthe above: lines should end "$". If you have lines ending "^M$" then you have carriage returns and that is your problem.
Otherwise, make sure there is still a /bin/sh:

ls -Fxl /bin/sh

You should see:

lrwxrwxrwx 1 root root 4 May 23  2009 /bin/sh -> bash*

i.e. sh is a symbolic link to bash, which is executable. Actually you would get "Permission Denied" if you only had a problem with execute access not being set
0
Duncan RoeSoftware DeveloperCommented:
If you do find the problem, you have to make the disk writable in order to fix it. 4 steps:
1. make disk read/write
2. fix problem(s)
3. make disk read-only again
4. continue init

Item 2 is up to you. Here are the other 3
1. mount -oremount,rw /
3. mount -oremount,ro /
4. exec /sbin/init 3

Open in new window

0
smyers051972Author Commented:
I run the ls -Fxl /bin/sh and this is the output
blackhole.JPG
0
Duncan RoeSoftware DeveloperCommented:
Looks like serious corruption - I would do an fsck -f on your root device. You can do it where you are (with disk read-only)
0
Duncan RoeSoftware DeveloperCommented:
Other possibility is make disk writable (step 1 of http:#1366836)

cd /bin
rm sh
ln -s bash sh

do steps 3 & 4 of http:#1366836
0
Duncan RoeSoftware DeveloperCommented:
I would do the fsck first - could be a doubly-allocated block
0
smyers051972Author Commented:
running fsck i get this
blackhole1.JPG
0
Duncan RoeSoftware DeveloperCommented:
Sorry - earlier link should be http:#31366836
0
smyers051972Author Commented:
how do i make disk read/write
0
Duncan RoeSoftware DeveloperCommented:
You must run fsck after you boot with init=/bin/bash. You may have to run it as /sbin/fsck.
Or are you saying that you ran fsck and it's no better? Running fsck won't fix your broken symbolic link - I thought you should run a forced fsck before modifying your disk further. Did it find any problems by the way?
Now you have run fsck, reboot with init=/bin/bash and apply the fix in http:#31369993
0
Duncan RoeSoftware DeveloperCommented:
You make the disk read/write as I posted earlier in http:#31366836
0
Duncan RoeSoftware DeveloperCommented:
WHOAH - I just saw your fsck didn't work
0
Duncan RoeSoftware DeveloperCommented:
I missed earlier that you actually can get in as root, but with read-only file system. You still should do the fsck -f I think.
I am surprised you can start fsck but it can't start fsck.ext2 - they a re both in /sbin.
try "type fsck" - where is it found?
try "echo $PATH" - does it contain /sbin? If not, type "PATH=$PATH:/sbin"
now try "type fsck.ext2" - is it now found as /sbin/fsck.ext2? Proceed with fsck -f if so.
0
Duncan RoeSoftware DeveloperCommented:
Alternatively, just type "/sbin/e2fsck -f"
0
Duncan RoeSoftware DeveloperCommented:
Did you get that? I'm about to go to bed
0
smyers051972Author Commented:
this is what i get
blackhole2.JPG
0
muffCommented:
Can you type

  ls /sbin/fsck*
  ls /usr/sbin/fsck*

I think the root partition is probably a bit trashed so you are missing some things.  Or perhaps fsck.ext2 may be in /usr/sbin in a distro that old, so you may need to add both to your path.

Can you give us a

  cat /etc/fstab

as well.
0
smyers051972Author Commented:
I can post screen shots only, I am connected via the KVM switch no network access is possible so I cant clip board things.  Give me a few and Ill try to post them.
0
Duncan RoeSoftware DeveloperCommented:
You should try to type EXACTLY what I put - I meant PATH to be upper case, but you fluked it because path is special.
It turns out fsck.ext2 is a shell script starting #!/bin/sh, but all it does is run e2fsck
You need to give the device on the command line as well

e2fsck -f <device>

where <device> is /dev/sda0 or whatever - "head /etc/fstab" to see what it is
0
Duncan RoeSoftware DeveloperCommented:
/sbin/e2fsck -f <device>
0
muffCommented:
According to the mount above /dev/sda1 is root

So /sbin/e2fsck -f /dev/sda1

You may as well do /usr too

/sbin/e2fsck -f /dev/sda7
0
smyers051972Author Commented:
im working on pasting the result from df -h , seems it cant find some of the mount points and ill do that e2fsck -f now and report the results. Give me a minute have to use a different pc to do this.
0
smyers051972Author Commented:
here is the mounts
mounts1.png
0
Duncan RoeSoftware DeveloperCommented:
Looking back at your erlier posts,  when you only typed "fsck -f" it would have tried to check all the entries in /etc/fstab in turn. So the first line in fstab refers to an ext2 file system (not ext3 - why not?). Hopefully that is the line for /, but it'd be worth checking. You must only use e2fsck on ext2, ext3 & ext4 file systems.
fsck is actually a driver for the real fixer programs. You can type "fsck /" but you must give an actual device to e2fsck.
0
smyers051972Author Commented:
and here is the results of /dev/sda7
captr1.png
0
smyers051972Author Commented:
what about rieserfs?
0
smyers051972Author Commented:
here is my fstab
captr1.png
0
Duncan RoeSoftware DeveloperCommented:
Good - your fsck is clean. I'm a bit suspicious that sda1, sda2, sda7 & sda9 have identical entries but that could be because of the read-only file system. All the df errors could be because of that.
What you need to do now is fix /bin/sh as I posted earlier, except miss out step 4 - just reboot when you're done.
0
smyers051972Author Commented:
there is a lot here, could you repost im sorry!
0
Duncan RoeSoftware DeveloperCommented:
Leave the reiserfs disks alone for now. Just fix /bin/sh to be a good link again and reboot. I didn't mean you to check all the disk partitions
0
smyers051972Author Commented:
Well since its a journaling FS I didnt think it was necessary but I thought I would ask any ways :)
0
Duncan RoeSoftware DeveloperCommented:
OK, repost. Lines starting # are comments. other lines, enter as-is
# make disk writeable

mount -oremount,rw /

# change directory

cd /bin

# fix the link

rm sh
ln -s bash sh

# flush changes to disk (new)

/usr/bin/sync

# reboot

/sbin/reboot

Open in new window

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
smyers051972Author Commented:
Worked great! THANK YOU!
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.