Solved

ESXI Virtual Machine eating up Lun Space

Posted on 2009-06-29
15
1,402 Views
Last Modified: 2012-05-07
Hello,
I Currently have a Win server 2003 virtual machine setup for a Dev Sql environment. The Virtual machine is hosted by a ESXI server that uses a 500Gb Lun. All virtual machines are on these Luns (no local). My issue is that in one days time the Lun that the Dev Sql virtual machine was placed on lost over 10Gb in space for no reason. Im the ONLY person with access to the Lun associated with the Virtual machine and the ESXi environment, and I have not done any snapshots or file movement. I assume there is a massive log file placed on the ESXi server but where is the default location? and how can I prevent this from happening on another ESXi box?

**The Dev SQL Server is NOT full in its windows partition and has over 20Gb of space to use in Windows

**I have had (3) ESXi boxes running for several months now. The associated Luns often change within the 100mb range. They have never drastically changed like the issue above unless I do snapshots or otherwise.

ANY HELP IS APPRECIATED! Thanks


Vmware-Question.png
0
Comment
Question by:securemedical
  • 7
  • 3
  • 3
  • +2
15 Comments
 
LVL 24

Expert Comment

by:ryder0707
ID: 24737718
u can find all the logs in /var/log
perhaps u should rotate all vm logs, let me know if u need help on this
0
 

Author Comment

by:securemedical
ID: 24738609
Now when the ESXi server saves these logs does it follow the virtual machines path? or does it store it locally? I will SSH into the server and check.
0
 

Author Comment

by:securemedical
ID: 24738710
Since I have posted this question (1 Hour) my space has lost 1Gb.....
Vmware-Question1.png
0
 
LVL 42

Expert Comment

by:paulsolov
ID: 24738901
Did you lose 10GB and then it stopped or does it keep growing?  The reason I say this is because VMs have a swap file outside the VM depending on how much memory you give a VM ESXi will also create the same size swapfile.  

Also keep in mind that when you're creating ntfs partitions the vmdk encapsulation takes up 5% right off the bat so if you create a VM that's 100GB the vmdk file will be 105GB.

Another thing to check is your backups.  Are you using a backup application with snapshot capability and are you using vmware snapshots?

What type of storage are using FC iSCSI or NFS?  Also ensure that your LUNs are not thin provisioned (unless you're using vSphere you can't do it through the VI clietn by default_

Hope this helps

0
 

Author Comment

by:securemedical
ID: 24739249
Update--

--The log Files on the ESXI are very small (less then 2 GB)

-Im constantly loosing space past the 10Gb initial mark. Since then i have reduced the Swap file of the virtual machine by 2Gb and saw no change in available space. The strangest portion of this issue is that it only started 2 days ago. All of our virtual machines are backed up via Symantec backup exec at the Windows level and not the ESXI level. All Snapshots are done manually.

-Im currently using iSCSI Lun that is portioned 620Gb our of 4.6TB. The original size was 500Gb but i had since increased the LUNS volume size to fix the issue (failed)

Everything I try seems to only delay the pace. Starting/Stopping the virtual machine doesnt add/subtract any space but while the virtual machine is IDLE it will lose 1GB per hour.... By the way thank you for the quick ideas, i need them!
0
 
LVL 42

Expert Comment

by:paulsolov
ID: 24739375
There is a different swapfile outside the virtual machine used by ESX.  It is equal to the memory you provide the virtual machine..4GB memory 4GB initial swapfile, take a look at your datastore, with the VM you'll see vmswp file.  You can reduce it the swap file by reserving memory on a VM. Thus is you give a VM 4GB RAM and reserver 2GB for the VM itself the swap file shrinks to 2GB.


Take a look at the size of your vmdk files and post them as well the size of your NTFS partitions on the LUN
0
 

Author Comment

by:securemedical
ID: 24739803
I have since reduced the swapfile for the virtual machine use to Half of normal. Im not starting the virtual machine because it has gone under 1Gb of free space.Attached are the .vmdk's you requested and something doesn't add up...


vmware3.png
Vmware2-001.png
0
Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

 
LVL 6

Expert Comment

by:davismisbehavis
ID: 24739900
What's showing in your snapshot manager for this VM because it looks like you  do have some snapshot files there.  one of them looks like it's 191GB as well? which might explain the very small amount of space left.

Are you using the Vmware Data Recovery product in your vSphere implementation?  There is a current issue with snapshots being left behind,  however these wouldn't be visible in the GUI as these ones appear to be.
0
 

Author Comment

by:securemedical
ID: 24740575
I agree with you that the 191GB must go. I wonder what is still gradually taking 1Gb per hour though...I guess one thing at a time.

Basically my company uses the ESXi Free edition for minor server roles and have no supporting application like Data Recovery.
0
 

Author Comment

by:securemedical
ID: 24740782
ONE LAST Update --

I have since tried to delete the 191GB snapshot and received the error attached "invalid snapshot config" which cleared my entire snapshot Queue. Do i have the ability of manually deleting the snapshot files now that they are not available via snapshot manager? or will this create an issue with starting the Virtual machine...im VERY close!

**Long story short....can i manually delete a snapshot from the datastore after its not associated with snapshot manager?
Vmware-issue11111.png
0
 
LVL 6

Accepted Solution

by:
davismisbehavis earned 500 total points
ID: 24740905
The 1GB an hour is the rate of change on your VM. A snapshot in vmware Freezes the current virtual disks as they are, the then create these additional disks where every single disk write is the written. Every single write! SQL servers have to be watched with vmware, leave a snapshot running and it will grow to enormous sizes, like 191GB. The snapshot disks are currently the active disks. It's entirely possible that the next time you start your VM it won't start. Usually deleting the snapshots commits the writes being stored in the temporary DB to the original Virtual disk. I have seen large snapshots error but eventually commit, I'd wait till tomorrow see if the disks marked with 00001, 00002, etc have gone. I.e. They've been committed successfully. If they're not I would phone vmware support (assuming you have a SnS contract)
0
 

Author Comment

by:securemedical
ID: 24741075
Ok perfect. I will wait until tomorrow to delete the snapshots. Currently the Vm is offline due to 0% space. The snapshot scenario does make sense considering i have multiple users accessing the Sql Box. Thanks again, it seems a virtual Sql box does take quite a bit of work to have it rock solid.
0
 
LVL 6

Expert Comment

by:davismisbehavis
ID: 24741082
Just be wary of leaving snapshots for to long. Check out rvtools, it's a good tool for checking for old snapshots.
0
 
LVL 42

Expert Comment

by:paulsolov
ID: 24741119
If you have any issues applying the snapshots you could use vConverter to convert the VM to a different datastore and it will consolidate the vmdk files for you. Once done run the VM and test.  Once tested shutdown the VM, delete the old one and move the new one to the datastore.  
0
 
LVL 10

Expert Comment

by:dnilson
ID: 24768824
Your are going to hve to be carefull removing a snapshot that large - its going to take a long time, VI isnt going to wit for it to get don, and you will quite possibly lose connectiviety to the VM through VI for the duration.

I've seen snapshots smaller than this take 8 hours to commit their date.

As said above, the snapshot is what is beign riteen to.  After the snapshot is taken, the base disk to read-only and checksummed.  So the empty snapshot file continues to grow with time.  Taking a new snapshot startes the processo ver again so ulitimately you can have many times the base disk space eaten up.

Snapshots are inteneded to be used for HOURS and removed.  all the written data in a snapshot has to be put back into the base disk before the snap id deleted.  This takes time and VI wont wait, it has a timeout.

Better to open a command console and use the commandline tool so you can see if its done or not

Commandline is unsupported but works on ESXi

vim-cmd vmsvc/snapshot.removeall [VmId]
vim-cmd vmsvc/snapshot.remove [VmId] [removeChildren] [snapshotLevel] [snapshotIndex]

Heres some documentation on Vimsh VMware left ou.
http://engr.ucsb.edu/~duonglt/vmware/#vmware_vimsh

Also look at the VMware Snapshotmanager.pl to run from vimsh
http://www.vmware.com/support/developer/viperltoolkit/viperl15/doc/snapshotmanager.html

0

Featured Post

Get up to 2TB FREE CLOUD per backup license!

An exclusive Black Friday offer just for Expert Exchange audience! Buy any of our top-rated backup solutions & get up to 2TB free cloud per system! Perform local & cloud backup in the same step, and restore instantly—anytime, anywhere. Grab this deal now before it disappears!

Join & Write a Comment

HOW TO: Install and Configure VMware vSphere Hypervisor 6.5 (ESXi 6.5), Step by Step Tutorial with screenshots. From Download, Checking Media, to Completed Installation.
Veeam Backup & Replication has added a new integration – Veeam Backup for Microsoft Office 365.  In this blog, we will discuss how you can benefit from Office 365 email backup with the Veeam’s new product and try to shed some light on the needs and …
Teach the user how to use vSphere Update Manager to update the VMware Tools and virtual machine hardware version Open vSphere Client: Review manual processes for updating VMware Tools and virtual hardware versions: Create a new baseline group in vSp…
This video shows you how to use a vSphere client to connect to your ESX host as the root user. Demonstrates the basic connection of bypassing certification set up. Demonstrates how to access the traditional view to begin managing your virtual mac…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

23 Experts available now in Live!

Get 1:1 Help Now