Go Premium for a chance to win a PS4. Enter to Win


Critical!  Apache won't start need help asap

Posted on 2004-09-24
Medium Priority
Last Modified: 2010-04-20
We were in the process of using our vmware system (which is on a linux machine) and noticed that I could not login to the management interface for VM's.  I followed the online help and tried to restart the apache service for VMware (/etc/init.d/httpd.vmware)  The first time I restarted is said-

Stopping  [OK]

Starting [OK]

and then the website wouldn't even come up.  I did a su -c "/etc/init.d/httpd.vmware restart"

Next I did  su -c "/etc/init.d/httpd.vmware status" and it says "httpd is stopped"

I cannot start the service...  WHen I do a df to see freespace, I have 100% usage, even after I deleted some files using "rm"  I am guessing that the computer won't start apache because it has no freespace.  ANY suggestions would be greatly appreciated.  Thanks
Question by:cdesimone
  • 7
  • 4

Author Comment

ID: 12141454
Here is what a df looks like

[root@esx2 VMFS2]# df
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sda2              1510060   1504116         0 100% /
/dev/sda1                46636     11586     32642  27% /boot
none                        256880     0        256880   0% /dev/shm

[root@esx2 VMFS2]# df /tmp
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sda2              1510060   1504120         0 100% /

and lastly

[root@esx2 VMFS2]# su -c "/etc/init.d/httpd.vmware start"
   Starting httpd.vmware:                                  [  OK  ]
[root@esx2 VMFS2]# su -c "/etc/init.d/httpd.vmware status"
httpd is stopped

LVL 18

Expert Comment

ID: 12141462
What is the output of df -k
You will need to remove some files
log file, tmp file, core files are all candidates
cd /
du -sk * |sort -n

this will show you which are the big directories, go into these and look to see where the disk is being used.

Post anything you are not sure about

If you are running out of disk space you might want to install some disk quota or even just monitoring software to catch it before it fails

Author Comment

ID: 12141575
[root@esx2 VMFS2]# df -k
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sda2              1510060   1504324         0 100% /
/dev/sda1                46636     11586     32642  27% /boot
none                    256880         0    256880   0% /dev/shm

NEW Veeam Agent for Microsoft Windows

Backup and recover physical and cloud-based servers and workstations, as well as endpoint devices that belong to remote users. Avoid downtime and data loss quickly and easily for Windows-based physical or public cloud-based workloads!


Author Comment

ID: 12141593
heres the other one

[root@esx2 VMFS2]# du -sk |sort -n
128106496       .

I am not sure about a lot of stuff :)  I copied and then removed a fairly big file from vmfs/VMFS2 directory. Any suggestions?

Author Comment

ID: 12141681
0       954
0       969
0       970
0       971
0       972
0       973
0       cmdline
0       cpuinfo
0       devices
0       dma
0       driver
0       execdomains
0       filesystems
0       fs
0       ide
0       interrupts
0       iomem
0       ioports
0       irq
0       kmsg
0       ksyms
0       loadavg
0       locks
0       meminfo
0       misc
0       modules
0       mounts
0       net
0       partitions
0       pci
0       scsi
0       self
0       slabinfo
0       stat
0       swaps
0       sys
0       sysvipc
0       tty
0       uptime
0       version
0       vmware
1       mtrr
16      bus
525712  kcore

LVL 18

Accepted Solution

liddler earned 2000 total points
ID: 12141701
cd kcore
du -sk * |sort -n
and see what are the big directories, this is an iterative process, we just keep going down directories till we find the offending files

Author Comment

ID: 12141705
and some more...  it seems like nothing unusual.. we just have a lot of space taken up by VM's

du: `proc/15776/fd/3': No such file or directory
4       initrd
4       opt
8       tmp
8       vpx
16      lost+found
16      mnt
276     dev
568     home
2892    floppy images
3392    etc
3528    root
4720    sbin
5192    bin
7472    boot
17728   var
89004   lib
451780  usr
525729  proc
275379200       vmfs
[root@esx2 /]# du -sh * |sort -n
du: `proc/15784/fd/3': No such file or directory
2.9M    floppy images
3.4M    etc
3.5M    root
4.0k    initrd
4.0k    opt
4.7M    sbin
5.1M    bin
7.3M    boot
8.0k    tmp
8.0k    vpx
16k     lost+found
16k     mnt
16k     proc
18M     var
87M     lib
263G    vmfs
276k    dev
442M    usr
568k    home
LVL 18

Expert Comment

ID: 12142085
so you need to reduced the size in your vmfs

Author Comment

ID: 12142952
Thanks for the replies, I did remove one large file from the vmfs directory and it still says that SDA1 is 100% usage and still won't start. Wondering if I should start somewhere else or bounce the server. Doesn't "df" not take into account files that do not have their indexes tooken away whereas "du" does?

LVL 18

Expert Comment

ID: 12143790
Sometimes du & df give differnet values, you may a rogue process writing to a truncated log file, so a reboot may help.
What is in your vmfs?  du -sk  /vmfs/*

Expert Comment

ID: 12144370
If what you've posted so far is correct, then this is not simply a disk full problem:

if 'df /' returns 1510060 Kb as the total size of the filesystem (~1.4Gb), and 'du -ks /vmfs' returns 263Gb then obviously something is very very wrong. My guess would be that /vmfs is a mountpoint for another filesystem which seems to be not mounted.

Check your /etc/fstab to see if I'm right, and if so, execute 'mount' to see which filesystems are actually mounted.  If vmfs is indeed a seperate mountpoint, and it is indeed not mounted, you can try to 'mount -a' to mount everything in /etc/fstab.  

Now, if the above holds, and you cannot mount the filesystem, you should run 'fsck' on the filesystem; e.g. fsck /dev/whatever_device_fstab_tells_you

Let me know if I'm on to something or if I'm just talking out of my as.....



Author Comment

ID: 12205939
The solution was to delete those DSK files that were incrementing in size and taking up all space on the root partition.  After that I rebooted and all is well.  Thx!

Featured Post

Visualize your virtual and backup environments

Create well-organized and polished visualizations of your virtual and backup environments when planning VMware vSphere, Microsoft Hyper-V or Veeam deployments. It helps you to gain better visibility and valuable business insights.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

How many times have you wanted to quickly do the same thing to a list but found yourself typing it again and again? I first figured out a small time saver with the up arrow to recall the last command but that can only get you so far if you have a bi…
SSH (Secure Shell) - Tips and Tricks As you all know SSH(Secure Shell) is a network protocol, which we use to access/transfer files securely between two networked devices. SSH was actually designed as a replacement for insecure protocols that sen…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Suggested Courses
Course of the Month10 days, 10 hours left to enroll

885 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question