Link to home
Start Free TrialLog in
Avatar of Adma1
Adma1

asked on

VM Ware error: Msg.hbacommon.outofspace

Hi All,

somehow we received the above error message (Msg.hbacommon.outofspace) when attempting to boot our virtual servers. However I have cleared space and even reduced the amount of RAM allocated to each of the 2 servers to ensure the amount of free space was greater than the avialable ram however we are still receiving this error. Looking in the event logs there vb isn't enough space for virtual disk 000004.

I am not sure what to do here and urgently need these servers up and running.

Any assistance would be greatly appreciated.

Many Thanks,
Adrian
Avatar of Adma1
Adma1

ASKER

Hi again,

One thing I forgot to mention in the previous post was that there are 2 Datastores sitting on this virtual machine one data store is empty and removed of its contents as the server was migrated to a new host. I am thinking of a temporary solution until we migrate the remaining 2 VM's can we delete the empty datastore and then expand the data store which currently host the remaining 2 VM's in turn bringing over the empty space into one consolidated data store. Sounds perfect in theory but wondering how this works in practice?

Thanks again,
Avatar of Adma1

ASKER

Hi All,

I thought I would add the error from the event log and a coupleof snap shots from the data store.

Any asssitance would be greatly appreciated.

Message from localhost.staging.solution-one.com.
au: There is no more space for virtual disk
File-Server-000004.vmdk.  You may be able to
continue this session by freeing disk space on the
relevant partition, and clicking Retry. Otherwise,
click Abort to terminate this session.
info
14/11/2012 5:08:30 PM
User
Datastore1.png
Datastore2.png
Avatar of Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
You need to DELETE the snapshot, Power OFF THE VM, and DELETE the Snapshot, and Be Patient whilst it deletes.

which datastore is the VM on, please also supply screen shot of the datastores with the VM files.

check if you are running on a snapshot, see my EE Article

HOW TO: VMware Snapshots :- Be Patient
Avatar of Adma1

ASKER

Hi Hanccocka,

Thankyou very much for your informative response.


Attached are 2 screenshots of the contents within the datastore. Looks like there are several snapshot files, can you please confirm those that I can delete and I will action this immediately and hopefully the deletion of snapshots is complete by the morning.

Thanks again
Adrian
Datastore-File-Server.png
Datastore-Mail-Server.png
Okay file server VM has 4 snapshots.

You cannot just delete the files, you must use the snapshot manager, and select Delete.

BUT, make sure the VM is Powered Off, You have enough storage space, and Be Patient, Snapshot deletion, can take minutes, hours or days to complete.
Avatar of Adma1

ASKER

OK will do,

What exactly do you mean by "You have enough storage space"?
Avatar of Adma1

ASKER

Looks like the 4 'vmsn' files are snapshot files and are only 28KB clearing these wont really give us the storage space we need in order to eliminate the error message in title of this thread. True?
Please do not fiddle, or manually delete any files!

How much free storage space do you have on the datastore?

looking at datastore, you may have 3-4GB free, it's very tight.

How many VMs on this datastore, you may need to turn OFF all VMs, and then check datastore space.
Avatar of Adma1

ASKER

Thanks so much for your help,

I have attached 2 screenshots.

Screenshot 1 = Snapshot manager
Screenshot 2 = datastore free space - remember both VM are currently residing in datastore 2 with only 9.33GB available.
snapshot-manager1.png
datastore-free-space.png
Okay, power off both VMs.

and then Click Delete ALL, and Wait, and Be Patient.

When the VM is off, it uses no extra disk space, and also the Memory Swap file is removed from the storage space, so you will have that extra space.
Avatar of Adma1

ASKER

OK to confirm selecting 'delete all' is done from within the snapshot manager correct.

It may also be worth noting at this point given that you mention 'Memory' today I reduced the RAM allocation for each server, as I tried to work around the issue and was told that the VM's need the same amount of free storage as what they have in allocated RAM.


Sorry just want to be 100% before I do anything.


Thanks again.
Yes, that's correct memory allocated to machine = swap file on disk , when VM powered ON, so by powering off all VMs, we get the storage back!

Yes, Delete ALL from, the Snapshot Manager to remove all Snapshots there.
Avatar of Adma1

ASKER

ok im going to give it a go now. will let u know how it goes.
Avatar of Adma1

ASKER

OK deletion of snapshots from file-server is underway. sorry one last question is it normal for the mail-server which sits in the same data store to not have any snapshots visible within the snapshot manager. I know there is mention of this in your article but just needed to ask.
it's very common for snapshots not to be displayed in the snapshot manager.

check my EE Article to check if you are running on a snapshot for your mailserver, I do not think you are.

the _1.vmdk is a second disk not a snapshot, snapshots have 00001.vmdk etc
Avatar of Adma1

ASKER

OK, will check that and thanks again for your help and assistance.

Much appreciated.
Adrian
Avatar of Adma1

ASKER

Sorry another dumb question.

Given that I am only deleting the snapshots from one of the servers i.e. the File-Server, progress is at 99% and I am just thinking if it is going to take a while to complete can I power up the second server which no snapshots are being deleted from?
No, Be Patient and wait for the process to complete, dealing with snapshots is all about Patience!

Power-ing up the other server, will use vital disk space you may need!

It could take minutes, hours or days......
Avatar of Adma1

ASKER

Deletion now complete, this is the test, im am going to attempt to turn both servers on and see what happens. Do I need to reboot the console first?
Avatar of Adma1

ASKER

just tried to power on the Mail-Server and failed due to insufficient space.
Avatar of Adma1

ASKER

Same thing keeps happening even after a a console reboot. Any ideas?
Avatar of Adma1

ASKER

Any ideas at all?, I am stuck and need these machines to boot up in approx 6 hours.
okay, check the store space, on the datastore.

no need to restart the host server.

do you have thin provisioned disk?
Avatar of Adma1

ASKER

Hi Hanccocka,

I cannot lkog into the server until I get into the office shortly as only the Terminal Server is up and running (on a seperate host) and I cannot login as the Mail-Server is also the domain controller.

I am thinking out aloud here can I just delete datastore and and then once remove I can simply extend datastore 2 to use the available space?
extending the datastores very much depends if they are local or SAN, shared storage based.

If SAN LUNs (shared storage) - yes.
Avatar of Adma1

ASKER

I think they are local i.e. using the servers physical HDD's  but what is the best to tell. Sorry I did not set these servers up.

Any other suggestions, I am desperate and I will be in some deep problems until I get these servers back up... Thanks
if they are local physical HDDs, you cannot extend datastores across partitions.

but can you rebuild ALL the HDDs, into a larger array?

send me screenshots of the datastores., free space an files.
Avatar of Adma1

ASKER

Is this what you were looking for (attached)?

Also I was planning to migrate these servers onto the new host this weekend, rebuilding the HDDs in a larger array sounds intense what exactly does that involve.
datastore-free-space.png
Datastore-File-Server.png
Datastore-Mail-Server.png
Avatar of Adma1

ASKER

OK, Im now back at my office and have logged into the Vshpere client. However alarm bells are going off, we now only have 3.00MB of storage avialable in datastore 2 which is where the 2 VM's are sitting (see attached) yesterday there was 9.3GB...I apologise in advance for being a pain but the urgency of this issue has me sitting on edge.

Any assitance would be greatly appreciated.

Many thanks
can you screenshot the datastores with the files.

can you also check your VM virtual disks are not thin
Avatar of Adma1

ASKER

just posted them, can u access them?

How do I check if the VM disks are thin? sorry not familiar with this.
okay the issue is with the server file server are these current screenshots, because the snapshot has not been deleted.
Avatar of Adma1

ASKER

yep just took the screenshots prior to the last post.
Avatar of Adma1

ASKER

what to do now.
are there any entries in the snapshot manager?
after you deleted the snapshot did it leave 3MB?
i think we should remove the iso file, that can be restored, and that will freeup some datastore space.
Avatar of Adma1

ASKER

No , they are both empty, see attahced. screen shot just taken.
Snapshot-Manager-File-Server-15-.png
Snapshot-manager-Mail-Server-15-.png
okay the snapshot chain was not deleted, you still have snapshots on your datastore.

1. delete the iso
2. how much space is free
3. power off VMs

report back
Avatar of Adma1

ASKER

Is this the ISO you want me to delete? (see attached)

If so then see 2nd screen shot the contents appears to be empty.
ISO-screenshot.png
ISO-contents.png
oh, I was hoping we could delete an iso file which would free up space.

any space on the other datastore?
your other datastore has approx 60gb free, but this is not enough to move mail server to.

can anything be removed on the other datastore?
okay I have a new idea.

please screenshot properties of the file server VM, right click vm, edit vm
Avatar of Adma1

ASKER

Yes Datastore1 has almost 60GB free, however has an ISO folder contianing the windows server 2K8 iso.

See attached.
Datastore1-free-space-15-11.png
Datastore-1-contents-15-11.png
what Im think is we could clone the first disk of file server to the other datastore, this would then be the new dis, it would not have a snapshot.
okay remove the iso, not needed at present.
do you know how to access console and login?

or via ssh?
Avatar of Adma1

ASKER

Is this the screenshot you wanted?

Can I really remove the ISO as well?
File-Server-properties-15-11.png
yes it is can you highlight virtual disk 1 and 2 and screenshot
okay the first disk is 50GB thick.

your other datastore has at least 50GB.

on Datastore1 can you create a folder called File-Server
Avatar of Adma1

ASKER

Yes will do now.
Avatar of Adma1

ASKER

Done.
Avatar of Adma1

ASKER

Do u mean on the actual phsical server itself, i.e. using ESXi?
Avatar of Adma1

ASKER

yep I can do that. I have the server here next to me. I just need to set up the KVM switch for keyboard/mouse and I can log into the ESXi.
Avatar of Adma1

ASKER

what do I need to do whilst in there?
you need to run a console command to clone the first virtual disk from datatore2 to datastore1
Avatar of Adma1

ASKER

OK, im good to go. Any idea how long the transfer will take?
in approx 30 mins, I'm going to had to get to bed, here in the UK, it's approaching 12.00.

the command you need to run is this:-

vmkfstools -i <most recent snapshot file name> /vmfs/volumes/<temp folder name>/<newfilename.vmdk>

I think this is the command for you formatted correctly with filenames and folders

you will need to execute this command from the /vmfs/Datastore2/File-Server folder, so you will need to use the

cd  /vmfs/Datastore2/File-Server

vmkfstools -i File-Server-000004.vmdk /vmfs/Datastore1/File-Server/File-Server.vmdk

start this going, and then let me know......you should see a new File-Server.vmdk be created on Datastore1/File-Server from the vSphere Client
speed, depends on how quick your storage is!

this is the first part, this will give us a new first virtual disk for the File-Server VM without a snapshot.

do you have backup of this VM?
Avatar of Adma1

ASKER

OK, thanks so much for your help,

So how do I actually access the command line interface within ESXi to run the command.
Avatar of Adma1

ASKER

No we don't have a back up of this VM, only a windows server backup.
Avatar of Adma1

ASKER

should I take a snapshot?
Avatar of Adma1

ASKER

Sorry which snapshot would be the latest (see attached)

Thanks,
file-server---datastore-15-11.png
Press ALT-F1 at the console screen.

I've given you the complete command above....


vmkfstools -i File-Server-000004.vmdk /vmfs/Datastore1/File-Server/File-Server.vmdk
Avatar of Adma1

ASKER

Ok thank you so much,

So this command:
vmkfstools -i File-Server-000004.vmdk /vmfs/Datastore1/File-Server/File-Server.vmdk

is this raw command with the gaps filled in, correct?
vmkfstools -i <most recent snapshot file name> /vmfs/volumes/<temp folder name>/<newfilename.vmdk>

so I run this command after changing directory: i.e running this command
cd  /vmfs/Datastore2/File-Server

is this all I need to do?
Avatar of Adma1

ASKER

should I have a backup, Im nervous about the whole operation.
Avatar of Adma1

ASKER

anoher dum question here, but there are 2 virtual drives for the file-server, but we are only moving 1 of them correct. The which is 50GB 000004.vdmk what will happen to the other virtual drive and will everything still work once this virtual drive is all moved to the new data store
this is the BIG RISK.

to create the space to be able to sort out the issue with the second virtual disk, 250GB, we need to create some space.

so, we are going to create a new first virtual disk (50GB) on datastore1, once this is complete.

you will create a new virtual machine, and attach this new virtual disk to this machine, to TEST the first virtual disk of VM File-Server, once you have confirmed this disk is good, you can delete the orginal virtual disk.

OR, if you do not feel confident in this....you could use VMware Converter, to EXPORT the entire File Server Virtual Machine to a Windows Workstation, Server Share, delete the entire VM from the Server and re-import.

or move to another server.

Problem is WE CANNOT

1. restart any machine - no storage
2. sort out the snapshot issue - no storage.
if you are nervous, I would call it a day here, and log a Support call with VMware Support, that will be able to WebEx you and do it for you.

Unfortunately, EE problems, must be posted, and I have no remote access.

I would certainly recommend backups.

Do you not run regular backups?
Avatar of Adma1

ASKER

OK I have another idea, and again thanks so much thus far for your help.

We have windows bare metal backups of both servers, and I mentioned we have a new identical host which we are planning to migrate to.

So I was thinking if we create a new virtual machine within the vshpere client of the new host then add the windows iso and point to the bare metal backups and restore the server that way?

What u think of that?
ah, it's not clever.

you could try using Veeam Quick Migration between servers.

make sure VMs are off.

See my EE Article
HOW TO: "Live Migrate" VMware Virtual Machines between ESX/ESXi hosts and/or datastores for FREE without licenses for vMotion or Storage vMotion

once last question, because I must now got to bed..
Avatar of Adma1

ASKER

problem is Veam is running on the Mail-Server and SQL server support Veeam is on the file server. hence the 2 servers we cannot access.

Should I attempt to reload Veeam on my local machine and use the default SQL express provided by Veeam installer?
Yes, that will do, any Workstation or Server.

I'm now off to bed now, but will be up in 6 hours!
Avatar of Adma1

ASKER

OK thanks.

Will update you at the end of our day here in Aust.
Avatar of Adma1

ASKER

Hi Hanccoka,

Just an update and also after some desperate advice.

After a 2 full days of workking at this issue I am no closer to a solution to getting the servers to either boot off the existing server or migrate onto the new server. As mentioned in the last post I installed Veeam locally however the when I attempt a 'quick migration' veeam will fail within the first 3 - 5 minutes with 0 progress. I contacted Veeam and there response was that they are unable to provide support as we are using the 'free' version of ESXi. So not much luck here.

After Speaking to our outsources IT support they advised using VM ware converter to migrate to the new server however when attempting to add the source server I receive an error message stating that VM converter could not identify the hardware for the virtual machine we are attempting to migrate. Reseraching the error on google returns an inconclusive trouble shooting attempts by others.

I then went for plan 'C' and attempt to download the datastore contents manually from within the vshpere client and save to USB and even my local machine. But  whenever I attempt to download from the datastore I recvieve a .vpx failed error.

Failing all of the above in a state of desperation I went last ditch effort and attempt to restore from the bare metal backups. I built a virtual machine on the new host and booted from an W2K8R2 iso and entered repair console, pointed the restore image to the network share but for some reason the connection to the share fails. (I being prompted for credentials etc...) and I receive an error I don't have the exact error but it was something similar to 'backup enumeration fail' I then added a USB device to the virtual machine and copied the backup over to the drive however Windows recovery fails to see the USB storage device.

Any further assistance on how I can get around the above issues you can offer and potentially migrate to the new server would be greatly appreciated.

Thanks again
Adrian.
I would still like to continue with our original plan, and once you have confirmed the newly merged vmdk (which we have cloned without snapshot) is working, you can delete the original disk.
Avatar of Adma1

ASKER

OK, was that to use Veeam to migrate? if so any ideas why veeam fails to initiate the migration?
No, that was not the Veeam Quick Migration route.

Veeam Quick Migration probably failed because it has no room to create the snapshot!

see http:#a38600798 above
Avatar of Adma1

ASKER

ok, there are a couple of queries here:

Our outsourced IT support has moved one vdmk file from the File-Server datastore into Datastore to free up enough space (think was about 3GB) in order to allow the Mail-Server to boot up. Will I need to move the vdmk file back to its orginal datastore? as running the command will probably overide anything already inside the datastore 1 correct?

secondly to run the command line on the console how to I get to command line alt F1 followed by something? or do I need to go into unsupported mode to run the script?

What are my chances of screwing up the operation, I guess I have no other option at this point anyway.

Thanks again.
You will need to go ""unsupported" mode, by selecting Alt-F1, then use the root username and password to login. What version of ESX is this, ESXi ? It's not really unsupported mode, it's where VMware Support, instruct people to go to run diagnostics commands, that cannot be run from the vSphere Client.

However, "your outsourced IT" should not have tampered with any files to get your other server up and running, they should have fixed the File Server first. Clearly they did not know how to do this.

There is a danger it's already been "messed" up.

to complete the clone all files need to be in the same datastore.
Avatar of Adma1

ASKER

ohh dear, it keeps getting worse. I did tell them not to touch the datastore. Hopefully its all still ok. Is just a mater of right clicking the file and copying out the datastore and dropping it back into the correct datastore?

We are running ESXi version 4.0 on exisitng server and 4.1 on the new one.

I will be going back into office in the morning to execute the above actions, hopefully it works.

Thanks,
Adrian
ASKER CERTIFIED SOLUTION
Avatar of Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Flag of United Kingdom of Great Britain and Northern Ireland image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of Adma1

ASKER

OK, thanks so much.

Unfortunately I cannot do this until first thing in the morning. as SSH to console was not working earlier today and still not, but don't have time to look into this isue its quicker to go into the office. Will be there in approx 8 - 9 hrs. its 10pm at night here in Australia.

I will touch base then and let u know how I am going.
Avatar of Adma1

ASKER

just a thought and I may have already brought this up in an earlier post. But would it be better to move the Mail Server to thenew datastore as its less then half the size of the File Server?
that depends on the size of the Mail-Server?

is there enough free space on the datastore to accomodate the mail-server?
Avatar of Adma1

ASKER

Hi Hanccocka, Thanks so much for your assistance I manged to get the servers fired up and running. I did not write back since our last conversation as it was a

What I did was the following.

Used Veeam to migrate the Mail-Server over to the new host tested that everything was fine and then deleted the Mail Server from the datastre on the old host. then moved the missing vdmk file back into the correct (file-server) datastore.

all servers now fire up and are running fine. Will plan for the Migration of the remaining (file-server) this weekend. hopefully all goes well.

Once again thanks so much for your help and patience.

 Adrian
did you manage to delete the snapshot on the File-Server?
Avatar of Adma1

ASKER

Hi hancoocka,

yes I did manage to delete the snap shot on the File-Server.

Thanks so much again for your help with this issue.

Adrian