Critical!  Apache won't start need help asap

Posted on 2004-09-24
Medium Priority
Last Modified: 2010-04-20
We were in the process of using our vmware system (which is on a linux machine) and noticed that I could not login to the management interface for VM's.  I followed the online help and tried to restart the apache service for VMware (/etc/init.d/httpd.vmware)  The first time I restarted is said-

Stopping  [OK]

Starting [OK]

and then the website wouldn't even come up.  I did a su -c "/etc/init.d/httpd.vmware restart"

Next I did  su -c "/etc/init.d/httpd.vmware status" and it says "httpd is stopped"

I cannot start the service...  WHen I do a df to see freespace, I have 100% usage, even after I deleted some files using "rm"  I am guessing that the computer won't start apache because it has no freespace.  ANY suggestions would be greatly appreciated.  Thanks
Question by:cdesimone
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 7
  • 4

Author Comment

ID: 12141454
Here is what a df looks like

[root@esx2 VMFS2]# df
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sda2              1510060   1504116         0 100% /
/dev/sda1                46636     11586     32642  27% /boot
none                        256880     0        256880   0% /dev/shm

[root@esx2 VMFS2]# df /tmp
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sda2              1510060   1504120         0 100% /

and lastly

[root@esx2 VMFS2]# su -c "/etc/init.d/httpd.vmware start"
   Starting httpd.vmware:                                  [  OK  ]
[root@esx2 VMFS2]# su -c "/etc/init.d/httpd.vmware status"
httpd is stopped

LVL 18

Expert Comment

ID: 12141462
What is the output of df -k
You will need to remove some files
log file, tmp file, core files are all candidates
cd /
du -sk * |sort -n

this will show you which are the big directories, go into these and look to see where the disk is being used.

Post anything you are not sure about

If you are running out of disk space you might want to install some disk quota or even just monitoring software to catch it before it fails

Author Comment

ID: 12141575
[root@esx2 VMFS2]# df -k
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sda2              1510060   1504324         0 100% /
/dev/sda1                46636     11586     32642  27% /boot
none                    256880         0    256880   0% /dev/shm

Veeam Task Manager for Hyper-V

Task Manager for Hyper-V provides critical information that allows you to monitor Hyper-V performance by displaying real-time views of CPU and memory at the individual VM-level, so you can quickly identify which VMs are using host resources.


Author Comment

ID: 12141593
heres the other one

[root@esx2 VMFS2]# du -sk |sort -n
128106496       .

I am not sure about a lot of stuff :)  I copied and then removed a fairly big file from vmfs/VMFS2 directory. Any suggestions?

Author Comment

ID: 12141681
0       954
0       969
0       970
0       971
0       972
0       973
0       cmdline
0       cpuinfo
0       devices
0       dma
0       driver
0       execdomains
0       filesystems
0       fs
0       ide
0       interrupts
0       iomem
0       ioports
0       irq
0       kmsg
0       ksyms
0       loadavg
0       locks
0       meminfo
0       misc
0       modules
0       mounts
0       net
0       partitions
0       pci
0       scsi
0       self
0       slabinfo
0       stat
0       swaps
0       sys
0       sysvipc
0       tty
0       uptime
0       version
0       vmware
1       mtrr
16      bus
525712  kcore

LVL 18

Accepted Solution

liddler earned 2000 total points
ID: 12141701
cd kcore
du -sk * |sort -n
and see what are the big directories, this is an iterative process, we just keep going down directories till we find the offending files

Author Comment

ID: 12141705
and some more...  it seems like nothing unusual.. we just have a lot of space taken up by VM's

du: `proc/15776/fd/3': No such file or directory
4       initrd
4       opt
8       tmp
8       vpx
16      lost+found
16      mnt
276     dev
568     home
2892    floppy images
3392    etc
3528    root
4720    sbin
5192    bin
7472    boot
17728   var
89004   lib
451780  usr
525729  proc
275379200       vmfs
[root@esx2 /]# du -sh * |sort -n
du: `proc/15784/fd/3': No such file or directory
2.9M    floppy images
3.4M    etc
3.5M    root
4.0k    initrd
4.0k    opt
4.7M    sbin
5.1M    bin
7.3M    boot
8.0k    tmp
8.0k    vpx
16k     lost+found
16k     mnt
16k     proc
18M     var
87M     lib
263G    vmfs
276k    dev
442M    usr
568k    home
LVL 18

Expert Comment

ID: 12142085
so you need to reduced the size in your vmfs

Author Comment

ID: 12142952
Thanks for the replies, I did remove one large file from the vmfs directory and it still says that SDA1 is 100% usage and still won't start. Wondering if I should start somewhere else or bounce the server. Doesn't "df" not take into account files that do not have their indexes tooken away whereas "du" does?

LVL 18

Expert Comment

ID: 12143790
Sometimes du & df give differnet values, you may a rogue process writing to a truncated log file, so a reboot may help.
What is in your vmfs?  du -sk  /vmfs/*

Expert Comment

ID: 12144370
If what you've posted so far is correct, then this is not simply a disk full problem:

if 'df /' returns 1510060 Kb as the total size of the filesystem (~1.4Gb), and 'du -ks /vmfs' returns 263Gb then obviously something is very very wrong. My guess would be that /vmfs is a mountpoint for another filesystem which seems to be not mounted.

Check your /etc/fstab to see if I'm right, and if so, execute 'mount' to see which filesystems are actually mounted.  If vmfs is indeed a seperate mountpoint, and it is indeed not mounted, you can try to 'mount -a' to mount everything in /etc/fstab.  

Now, if the above holds, and you cannot mount the filesystem, you should run 'fsck' on the filesystem; e.g. fsck /dev/whatever_device_fstab_tells_you

Let me know if I'm on to something or if I'm just talking out of my as.....



Author Comment

ID: 12205939
The solution was to delete those DSK files that were incrementing in size and taking up all space on the root partition.  After that I rebooted and all is well.  Thx!

Featured Post

Give Your Engineering Team a Productivity Boost

Learn why container technology is so powerful and how it can provide your team with productivity gains and other benefits.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Using 'screen' for session sharing, The Simple Edition Step 1: user starts session with command: screen Step 2: other user (logged in with same user account) connects with command: screen -x Done. Both users are connected to the same CLI sessio…
SSH (Secure Shell) - Tips and Tricks As you all know SSH(Secure Shell) is a network protocol, which we use to access/transfer files securely between two networked devices. SSH was actually designed as a replacement for insecure protocols that sen…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.
Suggested Courses

719 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question