Solved

Critical!  Apache won't start need help asap

Posted on 2004-09-24
12
272 Views
Last Modified: 2010-04-20
We were in the process of using our vmware system (which is on a linux machine) and noticed that I could not login to the management interface for VM's.  I followed the online help and tried to restart the apache service for VMware (/etc/init.d/httpd.vmware)  The first time I restarted is said-

Stopping  [OK]

Starting [OK]

and then the website wouldn't even come up.  I did a su -c "/etc/init.d/httpd.vmware restart"


Next I did  su -c "/etc/init.d/httpd.vmware status" and it says "httpd is stopped"

I cannot start the service...  WHen I do a df to see freespace, I have 100% usage, even after I deleted some files using "rm"  I am guessing that the computer won't start apache because it has no freespace.  ANY suggestions would be greatly appreciated.  Thanks
0
Comment
Question by:cdesimone
  • 7
  • 4
12 Comments
 

Author Comment

by:cdesimone
Comment Utility
Here is what a df looks like

[root@esx2 VMFS2]# df
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sda2              1510060   1504116         0 100% /
/dev/sda1                46636     11586     32642  27% /boot
none                        256880     0        256880   0% /dev/shm





[root@esx2 VMFS2]# df /tmp
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sda2              1510060   1504120         0 100% /




and lastly

[root@esx2 VMFS2]# su -c "/etc/init.d/httpd.vmware start"
   Starting httpd.vmware:                                  [  OK  ]
[root@esx2 VMFS2]# su -c "/etc/init.d/httpd.vmware status"
httpd is stopped


HELP!
0
 
LVL 18

Expert Comment

by:liddler
Comment Utility
What is the output of df -k
You will need to remove some files
log file, tmp file, core files are all candidates
try
cd /
du -sk * |sort -n

this will show you which are the big directories, go into these and look to see where the disk is being used.

Post anything you are not sure about

If you are running out of disk space you might want to install some disk quota or even just monitoring software to catch it before it fails
0
 

Author Comment

by:cdesimone
Comment Utility
[root@esx2 VMFS2]# df -k
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sda2              1510060   1504324         0 100% /
/dev/sda1                46636     11586     32642  27% /boot
none                    256880         0    256880   0% /dev/shm


0
 

Author Comment

by:cdesimone
Comment Utility
heres the other one

[root@esx2 VMFS2]# du -sk |sort -n
128106496       .


I am not sure about a lot of stuff :)  I copied and then removed a fairly big file from vmfs/VMFS2 directory. Any suggestions?
0
 

Author Comment

by:cdesimone
Comment Utility
0       954
0       969
0       970
0       971
0       972
0       973
0       cmdline
0       cpuinfo
0       devices
0       dma
0       driver
0       execdomains
0       filesystems
0       fs
0       ide
0       interrupts
0       iomem
0       ioports
0       irq
0       kmsg
0       ksyms
0       loadavg
0       locks
0       meminfo
0       misc
0       modules
0       mounts
0       net
0       partitions
0       pci
0       scsi
0       self
0       slabinfo
0       stat
0       swaps
0       sys
0       sysvipc
0       tty
0       uptime
0       version
0       vmware
1       mtrr
16      bus
525712  kcore


0
 
LVL 18

Accepted Solution

by:
liddler earned 500 total points
Comment Utility
cd kcore
du -sk * |sort -n
and see what are the big directories, this is an iterative process, we just keep going down directories till we find the offending files
0
Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

 

Author Comment

by:cdesimone
Comment Utility
and some more...  it seems like nothing unusual.. we just have a lot of space taken up by VM's

du: `proc/15776/fd/3': No such file or directory
4       initrd
4       opt
8       tmp
8       vpx
16      lost+found
16      mnt
276     dev
568     home
2892    floppy images
3392    etc
3528    root
4720    sbin
5192    bin
7472    boot
17728   var
89004   lib
451780  usr
525729  proc
275379200       vmfs
[root@esx2 /]# du -sh * |sort -n
du: `proc/15784/fd/3': No such file or directory
2.9M    floppy images
3.4M    etc
3.5M    root
4.0k    initrd
4.0k    opt
4.7M    sbin
5.1M    bin
7.3M    boot
8.0k    tmp
8.0k    vpx
16k     lost+found
16k     mnt
16k     proc
18M     var
87M     lib
263G    vmfs
276k    dev
442M    usr
568k    home
0
 
LVL 18

Expert Comment

by:liddler
Comment Utility
so you need to reduced the size in your vmfs
0
 

Author Comment

by:cdesimone
Comment Utility
Thanks for the replies, I did remove one large file from the vmfs directory and it still says that SDA1 is 100% usage and still won't start. Wondering if I should start somewhere else or bounce the server. Doesn't "df" not take into account files that do not have their indexes tooken away whereas "du" does?

Thanks
0
 
LVL 18

Expert Comment

by:liddler
Comment Utility
Sometimes du & df give differnet values, you may a rogue process writing to a truncated log file, so a reboot may help.
What is in your vmfs?  du -sk  /vmfs/*
0
 
LVL 4

Expert Comment

by:EinarTh
Comment Utility
If what you've posted so far is correct, then this is not simply a disk full problem:

if 'df /' returns 1510060 Kb as the total size of the filesystem (~1.4Gb), and 'du -ks /vmfs' returns 263Gb then obviously something is very very wrong. My guess would be that /vmfs is a mountpoint for another filesystem which seems to be not mounted.

Check your /etc/fstab to see if I'm right, and if so, execute 'mount' to see which filesystems are actually mounted.  If vmfs is indeed a seperate mountpoint, and it is indeed not mounted, you can try to 'mount -a' to mount everything in /etc/fstab.  

Now, if the above holds, and you cannot mount the filesystem, you should run 'fsck' on the filesystem; e.g. fsck /dev/whatever_device_fstab_tells_you

Let me know if I'm on to something or if I'm just talking out of my as.....

cheers.

0
 

Author Comment

by:cdesimone
Comment Utility
The solution was to delete those DSK files that were incrementing in size and taking up all space on the root partition.  After that I rebooted and all is well.  Thx!
0

Featured Post

6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

Join & Write a Comment

I am a long time windows user and for me it is normal to have spaces in directory and file names. Changing to Linux I found myself frustrated when I moved my windows data over to my new Linux computer. The problem occurs when at the command line.…
Little introduction about CP: CP is a command on linux that use to copy files and folder from one location to another location. Example usage of CP as follow: cp /myfoder /pathto/destination/folder/ cp abc.tar.gz /pathto/destination/folder/ab…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now