Critical!  Apache won't start need help asap

Posted on 2004-09-24
Last Modified: 2010-04-20
We were in the process of using our vmware system (which is on a linux machine) and noticed that I could not login to the management interface for VM's.  I followed the online help and tried to restart the apache service for VMware (/etc/init.d/httpd.vmware)  The first time I restarted is said-

Stopping  [OK]

Starting [OK]

and then the website wouldn't even come up.  I did a su -c "/etc/init.d/httpd.vmware restart"

Next I did  su -c "/etc/init.d/httpd.vmware status" and it says "httpd is stopped"

I cannot start the service...  WHen I do a df to see freespace, I have 100% usage, even after I deleted some files using "rm"  I am guessing that the computer won't start apache because it has no freespace.  ANY suggestions would be greatly appreciated.  Thanks
Question by:cdesimone
  • 7
  • 4

Author Comment

ID: 12141454
Here is what a df looks like

[root@esx2 VMFS2]# df
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sda2              1510060   1504116         0 100% /
/dev/sda1                46636     11586     32642  27% /boot
none                        256880     0        256880   0% /dev/shm

[root@esx2 VMFS2]# df /tmp
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sda2              1510060   1504120         0 100% /

and lastly

[root@esx2 VMFS2]# su -c "/etc/init.d/httpd.vmware start"
   Starting httpd.vmware:                                  [  OK  ]
[root@esx2 VMFS2]# su -c "/etc/init.d/httpd.vmware status"
httpd is stopped

LVL 18

Expert Comment

ID: 12141462
What is the output of df -k
You will need to remove some files
log file, tmp file, core files are all candidates
cd /
du -sk * |sort -n

this will show you which are the big directories, go into these and look to see where the disk is being used.

Post anything you are not sure about

If you are running out of disk space you might want to install some disk quota or even just monitoring software to catch it before it fails

Author Comment

ID: 12141575
[root@esx2 VMFS2]# df -k
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sda2              1510060   1504324         0 100% /
/dev/sda1                46636     11586     32642  27% /boot
none                    256880         0    256880   0% /dev/shm

Ransomware-A Revenue Bonanza for Service Providers

Ransomware – malware that gets on your customers’ computers, encrypts their data, and extorts a hefty ransom for the decryption keys – is a surging new threat.  The purpose of this eBook is to educate the reader about ransomware attacks.


Author Comment

ID: 12141593
heres the other one

[root@esx2 VMFS2]# du -sk |sort -n
128106496       .

I am not sure about a lot of stuff :)  I copied and then removed a fairly big file from vmfs/VMFS2 directory. Any suggestions?

Author Comment

ID: 12141681
0       954
0       969
0       970
0       971
0       972
0       973
0       cmdline
0       cpuinfo
0       devices
0       dma
0       driver
0       execdomains
0       filesystems
0       fs
0       ide
0       interrupts
0       iomem
0       ioports
0       irq
0       kmsg
0       ksyms
0       loadavg
0       locks
0       meminfo
0       misc
0       modules
0       mounts
0       net
0       partitions
0       pci
0       scsi
0       self
0       slabinfo
0       stat
0       swaps
0       sys
0       sysvipc
0       tty
0       uptime
0       version
0       vmware
1       mtrr
16      bus
525712  kcore

LVL 18

Accepted Solution

liddler earned 500 total points
ID: 12141701
cd kcore
du -sk * |sort -n
and see what are the big directories, this is an iterative process, we just keep going down directories till we find the offending files

Author Comment

ID: 12141705
and some more...  it seems like nothing unusual.. we just have a lot of space taken up by VM's

du: `proc/15776/fd/3': No such file or directory
4       initrd
4       opt
8       tmp
8       vpx
16      lost+found
16      mnt
276     dev
568     home
2892    floppy images
3392    etc
3528    root
4720    sbin
5192    bin
7472    boot
17728   var
89004   lib
451780  usr
525729  proc
275379200       vmfs
[root@esx2 /]# du -sh * |sort -n
du: `proc/15784/fd/3': No such file or directory
2.9M    floppy images
3.4M    etc
3.5M    root
4.0k    initrd
4.0k    opt
4.7M    sbin
5.1M    bin
7.3M    boot
8.0k    tmp
8.0k    vpx
16k     lost+found
16k     mnt
16k     proc
18M     var
87M     lib
263G    vmfs
276k    dev
442M    usr
568k    home
LVL 18

Expert Comment

ID: 12142085
so you need to reduced the size in your vmfs

Author Comment

ID: 12142952
Thanks for the replies, I did remove one large file from the vmfs directory and it still says that SDA1 is 100% usage and still won't start. Wondering if I should start somewhere else or bounce the server. Doesn't "df" not take into account files that do not have their indexes tooken away whereas "du" does?

LVL 18

Expert Comment

ID: 12143790
Sometimes du & df give differnet values, you may a rogue process writing to a truncated log file, so a reboot may help.
What is in your vmfs?  du -sk  /vmfs/*

Expert Comment

ID: 12144370
If what you've posted so far is correct, then this is not simply a disk full problem:

if 'df /' returns 1510060 Kb as the total size of the filesystem (~1.4Gb), and 'du -ks /vmfs' returns 263Gb then obviously something is very very wrong. My guess would be that /vmfs is a mountpoint for another filesystem which seems to be not mounted.

Check your /etc/fstab to see if I'm right, and if so, execute 'mount' to see which filesystems are actually mounted.  If vmfs is indeed a seperate mountpoint, and it is indeed not mounted, you can try to 'mount -a' to mount everything in /etc/fstab.  

Now, if the above holds, and you cannot mount the filesystem, you should run 'fsck' on the filesystem; e.g. fsck /dev/whatever_device_fstab_tells_you

Let me know if I'm on to something or if I'm just talking out of my as.....



Author Comment

ID: 12205939
The solution was to delete those DSK files that were incrementing in size and taking up all space on the root partition.  After that I rebooted and all is well.  Thx!

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Daily system administration tasks often require administrators to connect remote systems. But allowing these remote systems to accept passwords makes these systems vulnerable to the risk of brute-force password guessing attacks. Furthermore there ar…
Little introduction about CP: CP is a command on linux that use to copy files and folder from one location to another location. Example usage of CP as follow: cp /myfoder /pathto/destination/folder/ cp abc.tar.gz /pathto/destination/folder/ab…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

830 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question