How to free space on AWS Ubuntu 12.04.5 Instance

Firstly, while similar I do not believe my question is not a duplicate of other questions on this topic because all the suggestions in those questions have not resolved my problem.

I have a instance of Ubuntu running on EC2 that had been running fine for a couple of years. All the sudden the MySQL databases were unreachable. I ssh'd into the instance and tried sudo apt-get update just to try something and that was the first sign of the "No space left on device". The short of the long is I have done a lot of research trying to find a solution to my problem and have come up with zilch as most of the commands that seem to address this problem I can't get to run. I'm going to add a list of the commands with their corresponding results. Also, I have created a launched a second instance based on the offending AMI for testing suggestions. The results I give will be from the test instance.



sudo apt-get autoclean

sudo: unable to resolve host domU-12-31-39-0A-15-BE
Reading package lists... Error!
E: Could not open file /var/cache/apt/pkgcache.bin - open (28: No space left              on device)
E: Failed to truncate file - ftruncate (9: Bad file descriptor)
E: The package lists or status file could not be parsed or opened.

df -h

df: cannot read table of mounted file systems


dpkg -l | grep linux-image

ii  linux-image-3.2.0-23-virtual     3.2.0-23.36                       Linux kernel image for version 3.2.0 on 64 bit x86 Virtual Guests
ii  linux-image-3.2.0-39-virtual     3.2.0-39.62                       Linux kernel image for version 3.2.0 on 64 bit x86 Virtual Guests
ii  linux-image-3.2.0-58-virtual     3.2.0-58.88                       Linux kernel image for version 3.2.0 on 64 bit x86 Virtual Guests
ii  linux-image-3.2.0-64-virtual     3.2.0-64.97                       Linux kernel image for version 3.2.0 on 64 bit x86 Virtual Guests
ii  linux-image-3.2.0-69-virtual     3.2.0-69.103                      Linux kernel image for version 3.2.0 on 64 bit x86 Virtual Guests
ii  linux-image-3.2.0-70-virtual     3.2.0-70.105                      Linux kernel image for version 3.2.0 on 64 bit x86 Virtual Guests
ii  linux-image-3.2.0-74-virtual     3.2.0-74.109                      Linux kernel image for version 3.2.0 on 64 bit x86 Virtual Guests
ii  linux-image-3.2.0-75-virtual     3.2.0-75.110                      Linux kernel image for version 3.2.0 on 64 bit x86 Virtual Guests
ii  linux-image-3.2.0-83-virtual     3.2.0-83.120                      Linux kernel image for version 3.2.0 on 64 bit x86 Virtual Guests
ii  linux-image-3.2.0-88-virtual     3.2.0-88.126                      Linux kernel image for version 3.2.0 on 64 bit x86 Virtual Guests
ii  linux-image-3.2.0-89-virtual     3.2.0-89.127                      Linux kernel image for version 3.2.0 on 64 bit x86 Virtual Guests
ii  linux-image-virtual              3.2.0.89.103                      Linux kernel image for virtual machines


sudo apt-get purge linux-image-3.2.0-23.36-virtual

sudo: unable to resolve host domU-12-31-39-0A-15-BE
Reading package lists... Error!
E: Could not open file /var/cache/apt/pkgcache.bin - open (28: No space left    on device)
E: Failed to truncate file - ftruncate (9: Bad file descriptor)
E: The package lists or status file could not be parsed or opened.


lsblk

NAME  MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda1 202:1    0  12G  0 disk /



du -h /boot

4.0K    /boot/grub/locale
4.3M    /boot/grub
146M    /boot





cat /etc/fstab

LABEL=cloudimg-rootfs   /        ext4   defaults        0 0
/swapfile swap swap defaults 0 0

Open in new window

Dan BullAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

David Johnson, CD, MVPOwnerCommented:
This link is windows based but if you follow the instructions on how to increase the volume size you should be ok

1. Stop the instance
2. Create a snapshot of the boot drive
3. Using the above snapshot, create a new volume with the size you want
4. Detach the bootdrive volume from your instance
5. Attach the new larger volume to your instance
https://forums.aws.amazon.com/thread.jspa?threadID=51880

in ubuntu
sudo apt-get install gparted.
Launch it using Alt+F2, and typing gparted.
now resize your disk
https://askubuntu.com/questions/126153/how-to-resize-partitions
Dan BullAuthor Commented:
I left out the most important detail, the device is only 63% full.  That's what's so confusing to me about this situation.
Duncan RoeSoftware DeveloperCommented:
I think you need to run fsck over the affected partition. What kind of file system is it? (e.g. ext3, ext4, btrfs, ...)
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

Dan BullAuthor Commented:
So I got in touch with a local buddy of mine and this is the log of what he did.  Just in case anyone else runs in to this.

Connected successfully
"df: cannot read table of mounted file systems"

Open in new window

Looking for places to save space...
Trying to stop MySQL... fails: "unknown instance"
Sudo command also shows "unable to resolve host domU-12-31-39-0A-15-BE"
domU-12-31-39-0A-15-BE should be the host name, but /etc/hosts does not have this.
Trying to run
Vi

Open in new window

to edit /etc/hosts gets error:
  E303: Unable to open swap file for "/etc/hosts",recovery impossible
Successfully forced to write file
Gonna give restarting a try...
Sudo warning message gone, but df still gives problem
Issued command:
mysqld --verbose

Open in new window

 
151003 16:23:14 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
  151003 16:23:14 [Note] mysqld (mysqld 5.5.44-0ubuntu0.12.04.1) starting as process 1135 ...
  151003 16:23:14 [Warning] Can't create test file /var/lib/mysql/domU-12-31-39-0A-15-BE.lower-test
  151003 16:23:14 [Warning] Can't create test file /var/lib/mysql/domU-12-31-39-0A-15-BE.lower-test

Open in new window

issued command:
mysqladmin -u root status; "unable to access"

Open in new window

I'm guessing MySQL is not running at this point... (I was kinda thinking it was running but stuck)
Checking boot logs (dmesg)
Found
  [13567102.394629] EXT4-fs (xvda1): warning: maximal mount count reached, running e2fsck is recommended
So let's run
e2fsck!

Open in new window

Issued command:
shutdown -rF now

Open in new window

 * Curiously, the 'help' data is lacking on this command... but it runs, so let's hope.
...and now we wait.
Checked two minutes later, and it's accessible. Still error on df command, though, so I doubt fsck ran.
Google shows another method: creating an empty file called
 /forcefsck

Open in new window

and rebooting. Gonna try that...
But I cannot create the file because there's no room left on the disk! Ok, let's delete something trivial
Deleted fonts.css (copied contents into this folder) and touched
/forcefsck

Open in new window

--worked!
Rebooting...
...two minutes later, can connect again. Still having problems. Warning gone from
 dmesg

Open in new window

, though.
Ok, let's see where the actual space being taken up is. Saving discovered info in du.txt in this folder

Under /var/www/files/tmp, there's a LOT of files with names similar to: sess_l9sunrshor7i15cuf30313i416
The contents of this file is
  client|a:2:{s:11:"REMOTE_ADDR";s:15:"216.107.155.114";s:15:"HTTP_USER_AGENT";s:79:"Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)";}uGroups|a:1:{i:1;s:1:"1";}accessEntities|a:1:{i:0;O:27:"GroupPermissionAccessEntity":6:{s:8:"^@*^@group";O:5:"Group":12:{s:4:"ctID";N;s:13:"permissionSet";N;s:34:"^@Concrete5_Model_Group^@permissions";a:0:{}s:5:"error";s:0:"";s:3:"gID";s:1:"1";s:5:"gName";s:5:"Guest";s:12:"gDescription";s:62:"The guest group represents unregistered visitors to your site.";s:24:"gUserExpirationIsEnabled";s:1:"0";s:21:"gUserExpirationMethod";N;s:26:"gUserExpirationSetDateTime";N;s:23:"gUserExpirationInterval";s:1:"0";s:21:"gUserExpirationAction";N;}s:5:"error";s:0:"";s:5:"petID";s:1:"1";s:4:"peID";s:1:"1";s:9:"petHandle";s:5:"group";s:5:"label";s:5:"Guest";}}accessEntitiesUpdated|i:1433479931;
tried deleting them (rm /var/www/files/tmp/*), received error: "argument list too long."
Permissions: drwxr-xr-x  48; owner: www-data www-data
Issuing command: rm -fr /var/www/files/tmp/; mkdir /var/www/files/tmp/ ; chown www-data:www-data /var/www/files/tmp/
  Deleted everything just fine!
Issued command:
 sudo service mysql start

Open in new window

 SQl started successfully! Website is up, too!
Now, the important part... rebooting and having it all work still.
Well... the website is still up! Can't get in now, though:
    ssh -i TestKEY.pem  ubuntu@5X.1X7.2X6.1X3
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that a host key has just been changed.
    The fingerprint for the ECDSA key sent by the remote host is
    9x:15:4f:d0:2r:2g:e2:20:56:06:3a:b7:64:5f:1k:56.
    Please contact your system administrator.
    Add correct host key in /home/ryXXimXall/.ssh/known_hosts to get rid of this message.
    Offending ECDSA key in /home/ryXimXall/.ssh/known_hosts:2
      remove with: ssh-keygen -f "/home/ryXimXall/.ssh/known_hosts" -R 5X.1X7.2X6.1X3
    ECDSA host key for 5X.1X7.2X6.1X3 has changed and you have requested strict checking.
    Host key verification failed.
Cleared Known_hosts and it connected just fine. Must've been a AWS blip.

Anyway, I'm in and DF works just fine now, too. I would guess the root of the problem was the /var/www/files/tmp folder being filled with junk and overloading the filesystem.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Dan BullAuthor Commented:
I've requested that this question be closed as follows:

Accepted answer: 0 points for obtechsupport's comment #a41023346

for the following reason:

Because my buddy locally got it working.
Duncan RoeSoftware DeveloperCommented:
You ran fsck as I suggested. Now you have a working file system, I can give you commands (as I use) to track down where your disk space has gone.
Dan BullAuthor Commented:
We had to run /forcefsck and we tracked down the what was filling the hard drive.  It was a thousands of Amazon SES files.  What I'm assuming are some type of SES authentication hash.  We are using SES for outbound mail from a couple of sites on that instance.  What I don't understand is why when you first connect to the instance it shows that only 63% of the storage space was used.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.