Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium


Critical!  Apache won't start need help asap

Posted on 2004-09-24
Medium Priority
Last Modified: 2010-04-20
We were in the process of using our vmware system (which is on a linux machine) and noticed that I could not login to the management interface for VM's.  I followed the online help and tried to restart the apache service for VMware (/etc/init.d/httpd.vmware)  The first time I restarted is said-

Stopping  [OK]

Starting [OK]

and then the website wouldn't even come up.  I did a su -c "/etc/init.d/httpd.vmware restart"

Next I did  su -c "/etc/init.d/httpd.vmware status" and it says "httpd is stopped"

I cannot start the service...  WHen I do a df to see freespace, I have 100% usage, even after I deleted some files using "rm"  I am guessing that the computer won't start apache because it has no freespace.  ANY suggestions would be greatly appreciated.  Thanks
Question by:cdesimone
  • 7
  • 4

Author Comment

ID: 12141454
Here is what a df looks like

[root@esx2 VMFS2]# df
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sda2              1510060   1504116         0 100% /
/dev/sda1                46636     11586     32642  27% /boot
none                        256880     0        256880   0% /dev/shm

[root@esx2 VMFS2]# df /tmp
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sda2              1510060   1504120         0 100% /

and lastly

[root@esx2 VMFS2]# su -c "/etc/init.d/httpd.vmware start"
   Starting httpd.vmware:                                  [  OK  ]
[root@esx2 VMFS2]# su -c "/etc/init.d/httpd.vmware status"
httpd is stopped

LVL 18

Expert Comment

ID: 12141462
What is the output of df -k
You will need to remove some files
log file, tmp file, core files are all candidates
cd /
du -sk * |sort -n

this will show you which are the big directories, go into these and look to see where the disk is being used.

Post anything you are not sure about

If you are running out of disk space you might want to install some disk quota or even just monitoring software to catch it before it fails

Author Comment

ID: 12141575
[root@esx2 VMFS2]# df -k
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sda2              1510060   1504324         0 100% /
/dev/sda1                46636     11586     32642  27% /boot
none                    256880         0    256880   0% /dev/shm

Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.


Author Comment

ID: 12141593
heres the other one

[root@esx2 VMFS2]# du -sk |sort -n
128106496       .

I am not sure about a lot of stuff :)  I copied and then removed a fairly big file from vmfs/VMFS2 directory. Any suggestions?

Author Comment

ID: 12141681
0       954
0       969
0       970
0       971
0       972
0       973
0       cmdline
0       cpuinfo
0       devices
0       dma
0       driver
0       execdomains
0       filesystems
0       fs
0       ide
0       interrupts
0       iomem
0       ioports
0       irq
0       kmsg
0       ksyms
0       loadavg
0       locks
0       meminfo
0       misc
0       modules
0       mounts
0       net
0       partitions
0       pci
0       scsi
0       self
0       slabinfo
0       stat
0       swaps
0       sys
0       sysvipc
0       tty
0       uptime
0       version
0       vmware
1       mtrr
16      bus
525712  kcore

LVL 18

Accepted Solution

liddler earned 2000 total points
ID: 12141701
cd kcore
du -sk * |sort -n
and see what are the big directories, this is an iterative process, we just keep going down directories till we find the offending files

Author Comment

ID: 12141705
and some more...  it seems like nothing unusual.. we just have a lot of space taken up by VM's

du: `proc/15776/fd/3': No such file or directory
4       initrd
4       opt
8       tmp
8       vpx
16      lost+found
16      mnt
276     dev
568     home
2892    floppy images
3392    etc
3528    root
4720    sbin
5192    bin
7472    boot
17728   var
89004   lib
451780  usr
525729  proc
275379200       vmfs
[root@esx2 /]# du -sh * |sort -n
du: `proc/15784/fd/3': No such file or directory
2.9M    floppy images
3.4M    etc
3.5M    root
4.0k    initrd
4.0k    opt
4.7M    sbin
5.1M    bin
7.3M    boot
8.0k    tmp
8.0k    vpx
16k     lost+found
16k     mnt
16k     proc
18M     var
87M     lib
263G    vmfs
276k    dev
442M    usr
568k    home
LVL 18

Expert Comment

ID: 12142085
so you need to reduced the size in your vmfs

Author Comment

ID: 12142952
Thanks for the replies, I did remove one large file from the vmfs directory and it still says that SDA1 is 100% usage and still won't start. Wondering if I should start somewhere else or bounce the server. Doesn't "df" not take into account files that do not have their indexes tooken away whereas "du" does?

LVL 18

Expert Comment

ID: 12143790
Sometimes du & df give differnet values, you may a rogue process writing to a truncated log file, so a reboot may help.
What is in your vmfs?  du -sk  /vmfs/*

Expert Comment

ID: 12144370
If what you've posted so far is correct, then this is not simply a disk full problem:

if 'df /' returns 1510060 Kb as the total size of the filesystem (~1.4Gb), and 'du -ks /vmfs' returns 263Gb then obviously something is very very wrong. My guess would be that /vmfs is a mountpoint for another filesystem which seems to be not mounted.

Check your /etc/fstab to see if I'm right, and if so, execute 'mount' to see which filesystems are actually mounted.  If vmfs is indeed a seperate mountpoint, and it is indeed not mounted, you can try to 'mount -a' to mount everything in /etc/fstab.  

Now, if the above holds, and you cannot mount the filesystem, you should run 'fsck' on the filesystem; e.g. fsck /dev/whatever_device_fstab_tells_you

Let me know if I'm on to something or if I'm just talking out of my as.....



Author Comment

ID: 12205939
The solution was to delete those DSK files that were incrementing in size and taking up all space on the root partition.  After that I rebooted and all is well.  Thx!

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Google Drive is extremely cheap offsite storage, and it's even possible to get extra storage for free for two years.  You can use the free account 15GB, and if you have an Android device..when you install Google Drive for the first time it will give…
In the first part of this tutorial we will cover the prerequisites for installing SQL Server vNext on Linux.
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
How to Install VMware Tools in Red Hat Enterprise Linux 6.4 (RHEL 6.4) Step-by-Step Tutorial
Suggested Courses
Course of the Month11 days, 6 hours left to enroll

572 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question