Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

ESXI Virtual Machine eating up Lun Space

Posted on 2009-06-29
15
Medium Priority
?
1,421 Views
Last Modified: 2012-05-07
Hello,
I Currently have a Win server 2003 virtual machine setup for a Dev Sql environment. The Virtual machine is hosted by a ESXI server that uses a 500Gb Lun. All virtual machines are on these Luns (no local). My issue is that in one days time the Lun that the Dev Sql virtual machine was placed on lost over 10Gb in space for no reason. Im the ONLY person with access to the Lun associated with the Virtual machine and the ESXi environment, and I have not done any snapshots or file movement. I assume there is a massive log file placed on the ESXi server but where is the default location? and how can I prevent this from happening on another ESXi box?

**The Dev SQL Server is NOT full in its windows partition and has over 20Gb of space to use in Windows

**I have had (3) ESXi boxes running for several months now. The associated Luns often change within the 100mb range. They have never drastically changed like the issue above unless I do snapshots or otherwise.

ANY HELP IS APPRECIATED! Thanks


Vmware-Question.png
0
Comment
Question by:securemedical
  • 7
  • 3
  • 3
  • +2
15 Comments
 
LVL 24

Expert Comment

by:ryder0707
ID: 24737718
u can find all the logs in /var/log
perhaps u should rotate all vm logs, let me know if u need help on this
0
 

Author Comment

by:securemedical
ID: 24738609
Now when the ESXi server saves these logs does it follow the virtual machines path? or does it store it locally? I will SSH into the server and check.
0
 

Author Comment

by:securemedical
ID: 24738710
Since I have posted this question (1 Hour) my space has lost 1Gb.....
Vmware-Question1.png
0
VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

 
LVL 42

Expert Comment

by:Paul Solovyovsky
ID: 24738901
Did you lose 10GB and then it stopped or does it keep growing?  The reason I say this is because VMs have a swap file outside the VM depending on how much memory you give a VM ESXi will also create the same size swapfile.  

Also keep in mind that when you're creating ntfs partitions the vmdk encapsulation takes up 5% right off the bat so if you create a VM that's 100GB the vmdk file will be 105GB.

Another thing to check is your backups.  Are you using a backup application with snapshot capability and are you using vmware snapshots?

What type of storage are using FC iSCSI or NFS?  Also ensure that your LUNs are not thin provisioned (unless you're using vSphere you can't do it through the VI clietn by default_

Hope this helps

0
 

Author Comment

by:securemedical
ID: 24739249
Update--

--The log Files on the ESXI are very small (less then 2 GB)

-Im constantly loosing space past the 10Gb initial mark. Since then i have reduced the Swap file of the virtual machine by 2Gb and saw no change in available space. The strangest portion of this issue is that it only started 2 days ago. All of our virtual machines are backed up via Symantec backup exec at the Windows level and not the ESXI level. All Snapshots are done manually.

-Im currently using iSCSI Lun that is portioned 620Gb our of 4.6TB. The original size was 500Gb but i had since increased the LUNS volume size to fix the issue (failed)

Everything I try seems to only delay the pace. Starting/Stopping the virtual machine doesnt add/subtract any space but while the virtual machine is IDLE it will lose 1GB per hour.... By the way thank you for the quick ideas, i need them!
0
 
LVL 42

Expert Comment

by:Paul Solovyovsky
ID: 24739375
There is a different swapfile outside the virtual machine used by ESX.  It is equal to the memory you provide the virtual machine..4GB memory 4GB initial swapfile, take a look at your datastore, with the VM you'll see vmswp file.  You can reduce it the swap file by reserving memory on a VM. Thus is you give a VM 4GB RAM and reserver 2GB for the VM itself the swap file shrinks to 2GB.


Take a look at the size of your vmdk files and post them as well the size of your NTFS partitions on the LUN
0
 

Author Comment

by:securemedical
ID: 24739803
I have since reduced the swapfile for the virtual machine use to Half of normal. Im not starting the virtual machine because it has gone under 1Gb of free space.Attached are the .vmdk's you requested and something doesn't add up...


vmware3.png
Vmware2-001.png
0
 
LVL 6

Expert Comment

by:davismisbehavis
ID: 24739900
What's showing in your snapshot manager for this VM because it looks like you  do have some snapshot files there.  one of them looks like it's 191GB as well? which might explain the very small amount of space left.

Are you using the Vmware Data Recovery product in your vSphere implementation?  There is a current issue with snapshots being left behind,  however these wouldn't be visible in the GUI as these ones appear to be.
0
 

Author Comment

by:securemedical
ID: 24740575
I agree with you that the 191GB must go. I wonder what is still gradually taking 1Gb per hour though...I guess one thing at a time.

Basically my company uses the ESXi Free edition for minor server roles and have no supporting application like Data Recovery.
0
 

Author Comment

by:securemedical
ID: 24740782
ONE LAST Update --

I have since tried to delete the 191GB snapshot and received the error attached "invalid snapshot config" which cleared my entire snapshot Queue. Do i have the ability of manually deleting the snapshot files now that they are not available via snapshot manager? or will this create an issue with starting the Virtual machine...im VERY close!

**Long story short....can i manually delete a snapshot from the datastore after its not associated with snapshot manager?
Vmware-issue11111.png
0
 
LVL 6

Accepted Solution

by:
davismisbehavis earned 2000 total points
ID: 24740905
The 1GB an hour is the rate of change on your VM. A snapshot in vmware Freezes the current virtual disks as they are, the then create these additional disks where every single disk write is the written. Every single write! SQL servers have to be watched with vmware, leave a snapshot running and it will grow to enormous sizes, like 191GB. The snapshot disks are currently the active disks. It's entirely possible that the next time you start your VM it won't start. Usually deleting the snapshots commits the writes being stored in the temporary DB to the original Virtual disk. I have seen large snapshots error but eventually commit, I'd wait till tomorrow see if the disks marked with 00001, 00002, etc have gone. I.e. They've been committed successfully. If they're not I would phone vmware support (assuming you have a SnS contract)
0
 

Author Comment

by:securemedical
ID: 24741075
Ok perfect. I will wait until tomorrow to delete the snapshots. Currently the Vm is offline due to 0% space. The snapshot scenario does make sense considering i have multiple users accessing the Sql Box. Thanks again, it seems a virtual Sql box does take quite a bit of work to have it rock solid.
0
 
LVL 6

Expert Comment

by:davismisbehavis
ID: 24741082
Just be wary of leaving snapshots for to long. Check out rvtools, it's a good tool for checking for old snapshots.
0
 
LVL 42

Expert Comment

by:Paul Solovyovsky
ID: 24741119
If you have any issues applying the snapshots you could use vConverter to convert the VM to a different datastore and it will consolidate the vmdk files for you. Once done run the VM and test.  Once tested shutdown the VM, delete the old one and move the new one to the datastore.  
0
 
LVL 10

Expert Comment

by:dnilson
ID: 24768824
Your are going to hve to be carefull removing a snapshot that large - its going to take a long time, VI isnt going to wit for it to get don, and you will quite possibly lose connectiviety to the VM through VI for the duration.

I've seen snapshots smaller than this take 8 hours to commit their date.

As said above, the snapshot is what is beign riteen to.  After the snapshot is taken, the base disk to read-only and checksummed.  So the empty snapshot file continues to grow with time.  Taking a new snapshot startes the processo ver again so ulitimately you can have many times the base disk space eaten up.

Snapshots are inteneded to be used for HOURS and removed.  all the written data in a snapshot has to be put back into the base disk before the snap id deleted.  This takes time and VI wont wait, it has a timeout.

Better to open a command console and use the commandline tool so you can see if its done or not

Commandline is unsupported but works on ESXi

vim-cmd vmsvc/snapshot.removeall [VmId]
vim-cmd vmsvc/snapshot.remove [VmId] [removeChildren] [snapshotLevel] [snapshotIndex]

Heres some documentation on Vimsh VMware left ou.
http://engr.ucsb.edu/~duonglt/vmware/#vmware_vimsh

Also look at the VMware Snapshotmanager.pl to run from vimsh
http://www.vmware.com/support/developer/viperltoolkit/viperl15/doc/snapshotmanager.html

0

Featured Post

NFR key for Veeam Backup for Microsoft Office 365

Veeam is happy to provide a free NFR license (for 1 year, up to 10 users). This license allows for the non‑production use of Veeam Backup for Microsoft Office 365 in your home lab without any feature limitations.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Is your company's data protection keeping pace with virtualization? Here are 7 dynamic ways to adapt to rapid breakthroughs in technology.
In this article we will learn how to backup a VMware farm using Nakivo Backup & Replication. In this tutorial we will install the software on a Windows 2012 R2 Server.
Teach the user how to edit .vmx files to add advanced configuration options Open vSphere Web Client: Edit Settings for a VM: Choose VM Options -> Advanced: Add Configuration Parameters:
Teach the user how to install log collectors and how to configure ESXi 5.5 for remote logging Open console session and mount vCenter Server installer: Install vSphere Core Dump Collector: Install vSphere Syslog Collector: Open vSphere Client: Config…
Suggested Courses

971 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question