VMware VM Won't Start After Moving to Different Datastore

Hi,
you went the wrong way to move your VM from one datastore to another.
You have ESXi Free or with icense?
I think you could have just power off the vm, right click on it and choose migrate, choose your destination datastore and go!

As you did it manually (copying files) the path information for your VMDK, included in the vmx file (the VM config file), must be targeting the old datastore.
So it is the wrong path and I think that's whay you got this error.

Now you can :
- move back and do it again correctly with the GUI as I described before
- or modify the VMX file to update the VMDK path

Be carefull editing the VMX.
First before all you got to have a full VM backup, but maybe it is already too late (hope not).
Then same, make a copy of the vmx before any modification.
Delete the VM FROM INVENTORY, not from disk!
Modify the vmx file.
Then browse your datastore and right click the vmx file to re add your VM to the inventory. Done.

I am using vSphere Essentials, so I do not have the ability to do a migration with vMotion.  

What doesn't make sense though is that the VM was running on the primary datastore prior to the upgrade, but now it will only run after manually moving it to the other datastore.  And then if i try and move it back to the original datastore it gives me an error.  I can move it back and forth, but it will only power on if it is located on the 2nd datastore -- not the 1st.

I'm wondering if the original datastore somehow got corrupted during the upgrade?  Could I just delete it and re-create it, since it is not hosting any VMs?  (Even though it has those folders I mentioned above on it.)

consider what I have told you modifying the vmx.
You can check if the label "52a5e8ed-500765ad-c940-000af73eb610" in your path does correspond to one datastore or another, by logging with putty on your ESXi and going to /vmfs/volumes then ls-l.

I'm wondering if the original datastore somehow got corrupted during the upgrade?

Data could be corrupted, I don't think there is really a chance your datastore is corrupt.
But if you got no files in it why not delete and recreate.
Check it by clicking on "browse datastore" option or going into putty at the path I told you.

I cannot connect to the ESXi host via SSH.  Perhaps it is not enabled?  Is there a way I can enable it from the vSphere client?

yes, please see by lovely step by step tutorial with screenshots

Part 5: HOW TO: Enable SSH Remote Access on a VMware vSphere Hypervisor 5.1 (ESXi 5.1)

this also works for 5.5, and 6.0.

You don't have any snapshots do you ?

or left any files behind?

how did you move them ?

yes it is not enabled by default, follow the Andrew's guide it will be fine  :-)

Andrew:  

Thanks for the guide.  
The VM does not have any snapshots.  (At least when I check via the client it does not show any :-)
What do you mean by "any files left behind"?

Hmm.  SSH was already enabled, but when I try to PUTTY into it I get "connection refused".

What do you mean by "any files left behind"?

You didn't leave any!

check the firewall is open and not blocking SSH client/server

As far as files, I just moved the entire VM folder.  So, I don't think any where left behind.

I do not have any firewall on my client workstation (where I am running putty).  Would there be firewall settings on the ESXi host I need to configure?

I downloaded the vmx file from the datastore and just opened it with notepad.  The only reference in that file to either datastore is to the secondary datastore (the one that the VM is currently stored and running on) under the setting sched.swap.derivedName =

Yes, there is a firewall setting on the server.

Okay, I would remove the VM from the inventory.

Create a new Configuration file, e.g. new Custom VM, but with no disks, and then add the original disk.

to be honest with you I would like screenshot of the VM folder of the files as well. I'm operating here in the dark!

Here is a screenshot of the VM folder.
Capture.PNG

is this VM on ? e.g. Powered ON currently ?

Yes.

So it's working now ?

It works fine, as long as it is on datastore2, but I want it on datastore1 (which is where it was originally).

okay....

so that screenshot is not the contents of datastore1, because that copy is in use, it's from datastore2.

I need to see a screenshot of the non-working version on datastore1

OK.  Moving it now.  It will probably take about 20 minutes.

Here you go.
Capture.PNG

is it added to the inventory ?

No.

If I just create a new VM (as you suggested) should I just delete all the files except the .vmdk from the folder, and then select that file as the HD for the new VM?

No need to delete the files.

Create a new configuration file, following the New VM Wizard, select Custom, and then browse and add the virtual disk.

I would create the configuration files, on the working datastore, the theory is to test this "datastore, where the virtual disk is currently stored" for issues

Andrew:  I created a new custom VM on the same datastore, per your suggestion, using the .vmdk from the VM in question.  It powers on and works fine.

Are you thinking I should be able to just move it to the preferred datastore now?

Okay, so currently working, and powered on.

So, where is the virtual machine files spread currently?

1. virtual machine disk on datastore2?
2. configuration files on datastore1 ?

and you want them all on

Currently they are both on DS2.  (That was my understanding of what you said to do.)  The goal is for them to be on DS1.

Okay, so all is working on DS2.

Can you created the Configuration files (new) on DS1, and link to the virtual disk on DS2.

So

config files on DS1
vmdk on DS2

OK.  It works with that config.

Okay, so what is the free space on datastore 1 ?

what is the size of the virtual machine disk, and how are you moving/copying it to ds1?

Datastore1 has plenty of free space.  The .vmdk is 200GB.  What I had done previously was remove the VM from inventory, move the files using the datastore browser (from DS2 to DS1), then add back to inventory.

So, with the current configuration (config files on DS1, but .vmdk on DS2) what is the recommended way to move the disk file so that it resides with the config files on the desired datastore?

I would first,

1. detach the virtual machine disk from the VM. (remove), DO NO DELETE FROM DISK.

2. if you have no storage vMotion, copy and paste the virtual machine disk, between datastores. (leave a working copy on ds2).

3. Add to virtual machine again.

4. Power On - screenshot and post any error message.

Hmm.  When I copy the 200GB vmdk file it completes instantly, but the resulting file (in datastore1) is less than 1MB.  I don't understand what is going on there.  Any ideas?

not copying completely, or error.

when you issue a datastore copy, it actually copies it to the client, and back to the datastore.

It allows me to select another datastore location to copy to.  It completes without error, but the resulting file is only 51KB.  If I do a move (vs. copy) it gives me an error.  I am copying to the client workstation now, but it will take a while to download and then upload to the other datastore.

I have uploaded the .vmdk file to the new VM on datastore1, but when I go to add it to the VM it does not appear in the list when it has me browse to the location of the existing disk.

Never mind that last comment.  I copied the -flat file, but forgot the header file.  :-)

Okay, so once I was able to attach the vmdk I try to power on the VM and this is what I get:

An error was received from the ESX host while powering on VM SBI3.
Failed to start the virtual machine.
Module Disk power on failed.
Cannot open the disk
'/vmfs/volumes/52a5e8ed-500765ad-c940-000af73eb610/SBI3/SBIx.vmdk' or one of the snapshot disks it depends on.
The specified device is not a valid physical disk device

Hi,
you should do a copy with command line in SSH, instead of using GUI.

1. log to your ESXi
2. browse to the source datastore
# cd /vmfs/volumes
# ls -l
then use the datastore alias by # cd datastore_name
# ls -l (again)
# cd VM_name
and ls -l again
3. make sur you got
- the vmdk file (descriptor)
- the flat.vmdk file (the big one..)
if the flat.vmdk file name finish with -000001 or -000002 for example this is beacause you have snapshot, and that is something you MUST solve before going further.
If its name is vm_name.vmdk or vm_name_1.vmdk or vm_name_2.vmdk it is fine.

4. copy the two files, flat and descriptor, and -ctk.vmdk file too if you got one with same name.
# copy /vmfs/volume/source_datastore_name/vm_name/filename /vmfs /vmfs/volumes/target_datastore_name

5. attach this copied VVMDK to your VM and run and tell if errors.

But seeing your error message it is possible that before all your changes you had snapshot that you should have deleted and you didn't have.

can you check the md5 sis of each vmdk

should be sigs

Sorry command to copy is "cp" instead of "copy"

I am still unable to connect to the ESXi host via ssh.  If it is a firewall on the host itself where would I make the exception to allow ssh?

did you follow my EE Article exactly ?

Just ran into something that perhaps would steer this in a different direction.  Just out of curiosity I tried creating a new VM (just a small Linux install of 16GB) on datastore1 and received the error "Invalid operation for device '4'".  However, If I create the VM without a disk it creates just fine.  I can then attach a .vmdk on another datastore and the VM runs fine.  I don't understand what is causing this, but everything seems to revolve around not being able to run the .vmdk from datastore1.

can we have a screenshot of datastore and it's properties?

any other VMs on this datastore.

Here you go Andrew.  
No other VMs on this datastore.

is this a single disk or RAID ?

it's not a 4k disk ?

It's RAID 10.
What do you mean by 4k disk?

Ah, 4k sector size.  :-)
No.

what are the physical disks in use?

Seagate, 300GB, SAS6, 15,000 RPM, 2.5"

Hi,

I am still unable to connect to the ESXi host via ssh.  If it is a firewall on the host itself where would I make the exception to allow ssh?

Well I really doubt that your ESXi firewall block SSH but maybe.
Look at the menu shown in my screenshot attached.

Or the reason is that there is another firewall (hardware or software) between you and the ESXi.

Just out of curiosity I tried creating a new VM (just a small Linux install of 16GB) on datastore1 and received the error "Invalid operation for device '4'"

Well i don't know why but you have something wrong on your this datastore, either with VMware or the storage system.
Could - might - possibly be an error message due to firewall bloccking some reequest from your vsphere web client to the ESXi, but I doubt that.
Did you try the same operations with "heavy" vsphere client?

Bye
Capture-ESXi-firewall-allow-ssh.PNG

Looks like it requires a reboot of the host.  I'll see about giving that a try later.  Odd that it only effected this one host.

do you know the exact Seagate model?

because if the disks are 4k the are not supported by VMware.

Go to the Seagate website, and check the model number, and if the advanced format is 4k sectors, this could be the issue.

This is not new hardware, so unless VMware dropped support for 4k in the 5.5 to 6.0 upgrade process, then it doesn't seem that would be the case (even if they are 4k disks).

What is odd, and WRONG, is after the upgrade from 5.5 to 6.0U1, you could not start your VMs in the Inventory, this is not right!

4k hard drives have never been supported.

what are the models, so we can rule this out?

can you also screenshot the Properties section of the firewall, so we can see why you cannot SSH, to check the MD5 sigs of the two virtual machine disks on ds1 and ds2.

I'm not seeing anywhere what is the server hardware ?

what is he storage controller ?

it could be firmware of the server, was it upgraded before you upgraded to 6.0U1.

At this point I'm leaning heavily toward the problem being caused by the issue I mentioned previously, as noted in this VMware KB article:  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2127554

I plan to apply the recommended solution this weekend when I can take the server down.

If there are still issues then we can revisit the hardware/firmware possibly causing the problem -- though I am doubtful that is the reason.

This will be your issue, because ESXi 6.0 uses the entire datastore as a scratch location. (not compatible with vmdks!)

have you checked your scratch location ?

if you do not have a scratch location, because you have installed on an SD card, you will not have one.

Do you not get a message about persistent location as a warning?

because the scratch location, should have been the same as for the previous installed ESXi 5.5, and maintained, and updated.

did you check your server and storage are compatible with ESXi 6.0?

No, I have not checked the scratch location.  Isn't there only 1 for the host?  If that were the problem how would the VMs run at all?

It is installed on SD.

I agree, it should all be the same.

Yes, the hardware is 6.0 compatible.

Okay, this is a change/issue with 6.0. Your previous Scratch location and Logging directory are possibly set to the root, based on the opening post they are, so ESXi is using the whole datastore as a logging scratch location.

Common with all SD and USB flash drive locations, because of the read/write limitations of USB and Flash, ESXi OS prefers to use spinny rust for these locations.

This makes it incompatible as a datastore for storage of VM (e.g. VMDKs).

More than likely your issue, create a folder on your datastore1 called scratch, and also create a defined logging location, e.g. logs, and changed advanced settings for scratch and logs.

You can confirm by checking advanced settings using the vSphere Client, change as per KB, and you'll be running.

did this reslove the issue ?

The solution is in the link to the VMware KB article I posted.