ESXI Virtual Machine eating up Lun Space

Hello,
I Currently have a Win server 2003 virtual machine setup for a Dev Sql environment. The Virtual machine is hosted by a ESXI server that uses a 500Gb Lun. All virtual machines are on these Luns (no local). My issue is that in one days time the Lun that the Dev Sql virtual machine was placed on lost over 10Gb in space for no reason. Im the ONLY person with access to the Lun associated with the Virtual machine and the ESXi environment, and I have not done any snapshots or file movement. I assume there is a massive log file placed on the ESXi server but where is the default location? and how can I prevent this from happening on another ESXi box?

**The Dev SQL Server is NOT full in its windows partition and has over 20Gb of space to use in Windows

**I have had (3) ESXi boxes running for several months now. The associated Luns often change within the 100mb range. They have never drastically changed like the issue above unless I do snapshots or otherwise.

ANY HELP IS APPRECIATED! Thanks


Vmware-Question.png
securemedicalAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

ryder0707Commented:
u can find all the logs in /var/log
perhaps u should rotate all vm logs, let me know if u need help on this
0
securemedicalAuthor Commented:
Now when the ESXi server saves these logs does it follow the virtual machines path? or does it store it locally? I will SSH into the server and check.
0
securemedicalAuthor Commented:
Since I have posted this question (1 Hour) my space has lost 1Gb.....
Vmware-Question1.png
0
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

Paul SolovyovskySenior IT AdvisorCommented:
Did you lose 10GB and then it stopped or does it keep growing?  The reason I say this is because VMs have a swap file outside the VM depending on how much memory you give a VM ESXi will also create the same size swapfile.  

Also keep in mind that when you're creating ntfs partitions the vmdk encapsulation takes up 5% right off the bat so if you create a VM that's 100GB the vmdk file will be 105GB.

Another thing to check is your backups.  Are you using a backup application with snapshot capability and are you using vmware snapshots?

What type of storage are using FC iSCSI or NFS?  Also ensure that your LUNs are not thin provisioned (unless you're using vSphere you can't do it through the VI clietn by default_

Hope this helps

0
securemedicalAuthor Commented:
Update--

--The log Files on the ESXI are very small (less then 2 GB)

-Im constantly loosing space past the 10Gb initial mark. Since then i have reduced the Swap file of the virtual machine by 2Gb and saw no change in available space. The strangest portion of this issue is that it only started 2 days ago. All of our virtual machines are backed up via Symantec backup exec at the Windows level and not the ESXI level. All Snapshots are done manually.

-Im currently using iSCSI Lun that is portioned 620Gb our of 4.6TB. The original size was 500Gb but i had since increased the LUNS volume size to fix the issue (failed)

Everything I try seems to only delay the pace. Starting/Stopping the virtual machine doesnt add/subtract any space but while the virtual machine is IDLE it will lose 1GB per hour.... By the way thank you for the quick ideas, i need them!
0
Paul SolovyovskySenior IT AdvisorCommented:
There is a different swapfile outside the virtual machine used by ESX.  It is equal to the memory you provide the virtual machine..4GB memory 4GB initial swapfile, take a look at your datastore, with the VM you'll see vmswp file.  You can reduce it the swap file by reserving memory on a VM. Thus is you give a VM 4GB RAM and reserver 2GB for the VM itself the swap file shrinks to 2GB.


Take a look at the size of your vmdk files and post them as well the size of your NTFS partitions on the LUN
0
securemedicalAuthor Commented:
I have since reduced the swapfile for the virtual machine use to Half of normal. Im not starting the virtual machine because it has gone under 1Gb of free space.Attached are the .vmdk's you requested and something doesn't add up...


vmware3.png
Vmware2-001.png
0
davismisbehavisCommented:
What's showing in your snapshot manager for this VM because it looks like you  do have some snapshot files there.  one of them looks like it's 191GB as well? which might explain the very small amount of space left.

Are you using the Vmware Data Recovery product in your vSphere implementation?  There is a current issue with snapshots being left behind,  however these wouldn't be visible in the GUI as these ones appear to be.
0
securemedicalAuthor Commented:
I agree with you that the 191GB must go. I wonder what is still gradually taking 1Gb per hour though...I guess one thing at a time.

Basically my company uses the ESXi Free edition for minor server roles and have no supporting application like Data Recovery.
0
securemedicalAuthor Commented:
ONE LAST Update --

I have since tried to delete the 191GB snapshot and received the error attached "invalid snapshot config" which cleared my entire snapshot Queue. Do i have the ability of manually deleting the snapshot files now that they are not available via snapshot manager? or will this create an issue with starting the Virtual machine...im VERY close!

**Long story short....can i manually delete a snapshot from the datastore after its not associated with snapshot manager?
Vmware-issue11111.png
0
davismisbehavisCommented:
The 1GB an hour is the rate of change on your VM. A snapshot in vmware Freezes the current virtual disks as they are, the then create these additional disks where every single disk write is the written. Every single write! SQL servers have to be watched with vmware, leave a snapshot running and it will grow to enormous sizes, like 191GB. The snapshot disks are currently the active disks. It's entirely possible that the next time you start your VM it won't start. Usually deleting the snapshots commits the writes being stored in the temporary DB to the original Virtual disk. I have seen large snapshots error but eventually commit, I'd wait till tomorrow see if the disks marked with 00001, 00002, etc have gone. I.e. They've been committed successfully. If they're not I would phone vmware support (assuming you have a SnS contract)
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
securemedicalAuthor Commented:
Ok perfect. I will wait until tomorrow to delete the snapshots. Currently the Vm is offline due to 0% space. The snapshot scenario does make sense considering i have multiple users accessing the Sql Box. Thanks again, it seems a virtual Sql box does take quite a bit of work to have it rock solid.
0
davismisbehavisCommented:
Just be wary of leaving snapshots for to long. Check out rvtools, it's a good tool for checking for old snapshots.
0
Paul SolovyovskySenior IT AdvisorCommented:
If you have any issues applying the snapshots you could use vConverter to convert the VM to a different datastore and it will consolidate the vmdk files for you. Once done run the VM and test.  Once tested shutdown the VM, delete the old one and move the new one to the datastore.  
0
dnilsonCommented:
Your are going to hve to be carefull removing a snapshot that large - its going to take a long time, VI isnt going to wit for it to get don, and you will quite possibly lose connectiviety to the VM through VI for the duration.

I've seen snapshots smaller than this take 8 hours to commit their date.

As said above, the snapshot is what is beign riteen to.  After the snapshot is taken, the base disk to read-only and checksummed.  So the empty snapshot file continues to grow with time.  Taking a new snapshot startes the processo ver again so ulitimately you can have many times the base disk space eaten up.

Snapshots are inteneded to be used for HOURS and removed.  all the written data in a snapshot has to be put back into the base disk before the snap id deleted.  This takes time and VI wont wait, it has a timeout.

Better to open a command console and use the commandline tool so you can see if its done or not

Commandline is unsupported but works on ESXi

vim-cmd vmsvc/snapshot.removeall [VmId]
vim-cmd vmsvc/snapshot.remove [VmId] [removeChildren] [snapshotLevel] [snapshotIndex]

Heres some documentation on Vimsh VMware left ou.
http://engr.ucsb.edu/~duonglt/vmware/#vmware_vimsh

Also look at the VMware Snapshotmanager.pl to run from vimsh
http://www.vmware.com/support/developer/viperltoolkit/viperl15/doc/snapshotmanager.html

0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.