<

HOW TO: Add Local Storage (e.g. a SATA disk) as a Raw Disk Mapping (RDM) or Mapped RAW LUN to a virtual machine hosted on ESXi

Published on
101,403 Points
79,503 Views
9 Endorsements
Last Modified:
Awarded
Andrew Hancock (VMware vExpert / EE MVE^2)
MVE^2, Expert of the Year 2016-2011, Scribe 2016-2012, Author of the Year 2016,2013-2012 VMware vExpert 2016-2011 27 years of experience.
In this article, I am going to expose some of the hidden, undocumented, unsupported features and functions of the VMware vSphere Hypervisor (ESXi).

VMware vSphere ESXi 4.x, and ESXi 5.0 does not officially support Raw Disk Mapping of Local Storage e.g SATA, SAS or SCSI disks.

Please use at your own risk, and if you do escalate a Support Request (SR) to VMware Support, do not be surprised if they state this is unsupported.


VMware's definition of an RDM
'Introduced with VMware ESX Server 2.5, raw device mapping allows a special file in a VMFS volume to act as a proxy for a raw device. The mapping file contains metadata used to manage and redirect disk accesses to the physical device. The mapping file gives you some of the advantages of a virtual disk in the VMFS file system, while keeping some advantages of direct access to physical device characteristics. In effect, it merges VMFS manageability with raw device access.'

Traditionally and VMware supported, a Raw Disk Mapping (RDM) which may be used to present a LUN directly to a virtual machine from a SAN. The SAN could be Fibre Channel or iSCSI.

Why would we want to create a Raw Disk Mapping (RDM) of a local storage device?

An RDM can offer improved performance, which can be used for more disk I/O intensive applications such as database servers.
You may already have a data partition on the disk you want to present directly to the VM.
There is a limitation in VMware Hypervsior ESXi 4.x of a 2TB VMFS datastore limit, this can be overcome by using RDMs.
VMware Hypervisor ESXi 4.x and VMware Hypervisor ESXi 5.0 have a 2TB virtual disk (VMDK) limit.
You may want to have access directly to local storage to create a JBOD, because you do not have a supported storage controller,
SCSI controller may not be supported by VMware Hypervisor ESXi.
Creating a VMFS datastore on a single SATA drive can be slow.
Creating a 500GB virtual disk on a single SATA drive can be slow.

A few valid reasons why you may want to create an RDM of a local storage device.

This solution creates a RDM virtual disk proxy mapping file on an existing mounted VMFS datastore. This virtual disk proxy mapping "file" is then added to a Virtual Machine. I would recommend that RDM disks are used for data storage only e.g. SQL databases, large file servers.

The following procedure can be used to create the RDM on a local storage device. In the following example, a Seagate ST3500320AS 7200k rpm 500GB SATA disk is going to be used to create an RDM.

1. Connect to the VMware Hypervisor ESXi server via SSH

Using PuTTY a free telnet and SSH client or another SSH client Login and Connect to the VMware Hypervisor ESXi server, using IP address or hostname of the VMware Hypervisor ESXi server, using root username and password credentials.

putty SSH terminal session
logged in as root to ssh terminal session

2. Obtain the physical disk path of the storage device


There are a number of methods that can be used to find the physical disk path, the vSphere Client can be used, check Storage Adaptors, but the easiest way to find the physical disk path of your storage device is to use the commands

esxcfg-mpath -l

output from esxcg-mpath
or

ls -al /vmfs/devices/disks

output from ls -al /vmfs/devices/disks
using the above commands, we can cut and paste the physical disk path, which when using SATA devices can be very long and difficult to type.

In the above sessions the physical disk path is

t10.ATA_____ST3500320AS_________________________________________XXXXXXXX

(i've obsecured the screenshots above, because it includes a serial number for the Seagate ST3500320AS 7200k rpm SATA disk.). I've also replaced the serial number with Xs above.

If using a SCSI storage controller the physical disk path could be
example vmhba4:1:0:0


3. Using the vmkfstools command to create the RDM


Using the command vmkfstools we can create a RDM linked to a local storage device.

the command syntax is

vmkfstools -r <source> (space) <destination>

vmkfstools -r  /vmfs/devices/disks/<physical disk path> (space) /vmfs/volumes/[b]<vmfs datastore name>[/b]/<folder name>/<virtual disk name>.vmdk

at the console prompt type

vmkfstools -r  /vmfs/devices/disks/t10.ATA_____ST3500320AS_________________________________________XXXXXXXX /vmfs/volumes/esxdev004_local1_vmfs3/SATA500G/SATA500G.vmdk

esxdev004_local1_vmfs3 is the friendly VMFS datastore name
SATA500G is a folder created on the VMFS datastore to store the proxy mapping (vmdk) file.
SATA500G.vmdk is the name of the proxy mapping file (false virtual disk - vmdk.)

I've replaced the serial number with Xs above.

After entering the above command, vmkfstools does not respond with any output this is normal. If you check the datastore, you will see that a virtual machine disk named SATA500G.vmdk has been created.

SATA500G.vmdk

4. Adding the RDM to a Virtual Machine


Using the VMware vSphere Client, Login and Connect to the ESXi server, using IP address or hostname of the ESXi server, using root username and password credentials. If you have a VMware vSphere vCenter server, you could also specify IP address or hostname of the vCenter server.

Using the VMware vSphere Client, Login and Connect to the ESXi server

5. Select an existing virtual machine


Select an existing virtual machine, you want to add the RDM to, and select Edit Settings. The new hard disk can be added whilst the virtual machine is powered on or off. Hotplug is supported on some Windows OS.

 Edit Settings of Virtual Machine
Click Add to add a new device

Add a new Device
Click Add Hard Disk followed by Next

Add Hard Disk
Select Use an existing Virtual Disk
(please do not select Raw Device Mapping)

Use An Existing Disk
Browse the datastore and select the VMDK file and click OK, followed by Next

Browse Datastore for VMDK
Select a SCSI Virtual Device Mode. Select Mode - Independent - Persistent. Followed by Next.

Select Mode - Independent - Persistent.
Review the selected options and Click Finish

Disk Configuration Finished
The virtual machine will report a Mapped Raw LUN has been added.

Mapped Raw LUN has been added
The "magic" has been completed and we have successfully added a Mapped Raw LUN (RDM), which is the local SATA 500GB hard disk. Check that the words Mapped Raw LUN are present.

Click OK and wait several seconds for the device to be added to the virtual machine.

6. Checking the hard disk in the Virtual Machine


Open a console and login to the virtual machine OS. In this example Windows XP Professional.

Select Windows Disk Management (My Computer, Properties, Manage, Disk Management).

Before Rescan Disk Managment in Windows XP
The above screenshot, is before we have used the Rescan Disks option, as the hard disk was hot added.

Right Click Disk Management - Rescan Disks

After Rescan Disk Managment in Windows XP
A New Volume E: NTFS has been added to the virtual machine. This was an existing formatted NTFS disk with existing data we required in the VM.

Just also to note the virtual machine disk file VMDK, SATA500G.vmdk created although this file appears to be 500GB on disk, it's a small proxy mapping file, which does not take up this amount of disk space on the datastore. In this example above the local datastore is too small to accomodate a 500GB vmdk file.

You have successully created an RDM and added it to a virtual machine.

Thank you for reading my article, please leave valuable feedback.
9
Comment
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
12 Comments
 

Expert Comment

by:RFVDB
This guide is very comprehensive, but I have several questions about utilizing RDM for my deployment. I am attempting to utilize a RAID volume that is larger than 2TB within my datastore and was hoping that RDM would be a stable enough solution. I am running ESX 5.0 while following this guide and have run into an error.

In step 3, "Using the vmkfstools command to create the RDM", I am getting an error when I attempt to run the provided command. I have located my RAID volume and assembled my command, but I am receiving the following error with the accompanying debug info:

# vmkfstools -r  /vmfs/devices/disks/naa.XXXXXX/vmfs/volumes/datastore1/3700GB.vmdk -v 0

DISKLIB-LIB   : CREATE: "/vmfs/volumes/datastore1/3700GB.vmdk" -- vmfsRawDeviceMap capacity=0 (0 bytes) adapter=buslogic devicePath='/vmfs/devices/disks/naa.XXXXXX'
DISKLIB-LIB   : Only disks up to 2TB-512 are supported for 'vmfsRawDeviceMap' disk.
Failed to create virtual disk: The destination file system does not support large files (12).

In the head of this article, you stated the following:

There is a limitation in VMware Hypervsior ESXi 4.x of a 2TB VMFS datastore limit, this can be overcome by using RDMs.

Is there a necessary step to allow partitions greater than 2TB to be created as RDM disks? I was under the impression the change from MBR to GPT was what allowed this functionality, but I am not certain as to what that will require. For your 500GB drive there was no additional preparation, but to overcome the 2TB limitation, is there another step that I need to complete prior to running the vmkfstools command? Possibly creating a GUID Partition table for the drive?




Also, is there a way to use the remaining space on the installation drive for an RDM? For instance, with a large RAID 5 volume with ESX installed on it, the remaining space is partitioned. Is it possible to use the remaining space as an RDM with a size of > 2TB?
0
 
LVL 121

Author Comment

by:Andrew Hancock (VMware vExpert / EE MVE^2)
please also note this method is unsupported, and is not the traditional RDM from an  iSCSI or FC LUN for which RDMs are usually used. RDMs can be used to exceed the 2TB limit that existed in ESXi 4.x.

This VMFS limit in ESXi 5.0 no longer exists.

If you would like to discuss you issue further, please post a question, but as this is unsupported there may not be a working solution.
0
 

Expert Comment

by:NotMyRealNamee
I have just started playing around with vSphere 5 and tried several tutorials on the internet regarding how to add a local hard drive to a virtual machine - specifically WHS 2011.

I found your tutorial to be the best!  Although I ran into a couple of smaller issues, I was able to get through it and have my existing hard drive (with all my files intact) available in the Virtual Machine

I was a little confused about where you got the "datastore name" until I realized I needed to look at by datastore browser and see what the name was in brackets

The second problem was the command failed - but I was able to go into my server using WinSCP and manually create the folder where I wanted the new RDM files to be stored.  After this the command executed properly and I could immediately see the files in the folder via WinSCP

Thanks again - I am a happy camper
0
Efficient way to get backups off site to Azure

This user guide provides instructions on how to deploy and configure both a StoneFly Scale Out NAS Enterprise Cloud Drive virtual machine and Veeam Cloud Connect in the Microsoft Azure Cloud.

 

Expert Comment

by:bym007
Thanks for your very detailed post. I am trying to "mount" a local SATA disk to the ESXi. This is a 2TB disk, with 1 (Win 7) NTFS partition, which has a lot of valuable data, which I honestly do not want to lose.

So, when I try to "mount" the disk using PuTTY SSH using your given command vmkfstools -r , I get the error that "Failed to create virtual disk: The system cannot find the file specified (25)."

I am using this command exactly:

vmkfstools -r  /vmfs/devices/disks/t10.ATA_____SAMSUNG_HD204UI_________________________SERIAL_NUMBER______ /vmfs/volumes/esxdev004_local1_vmfs3/SATA2TB/SATA2TB.vmdk

I have also tried the above command without esxdev004_local1_vmfs3 folder as you mentioned in destination.

vmkfstools -r /vmfs/devices/disks/t10.ATA_____SAMSUNG_HD204UI_________________________S2H7J90B310611______ /vmfs/volumes/SATA2TB/SATA2TB.vmdk

Being adventurous, I went to the actual directory, /vmfs/volumes/ to try and manually create the SATA2TB directory, but it denied me permission. So I am stuck about loading this disk.

I have tried to copy the Physical Disk Path many times from multiple sources highlighted in your article, and get the same result everytime.

I have gone back to the vSphere client to confirm that the disk does exist and comes up, if I want to add it as a new datastore, which ofcourse I do not.

Any help will be appreciated. Many thanks.
0
 
LVL 121

Author Comment

by:Andrew Hancock (VMware vExpert / EE MVE^2)
@bym007 :- Please Ask a New Question for myself or other Experts to answer.
0
 

Expert Comment

by:bym007
@hancoccka: Sorry my account restrictions do not allow me to ask questions. Can you kindly answer in comments here ? Many thanks for understanding.
0
 

Expert Comment

by:SAGE Dining
@bym007
OK so I had the same problem as you. Turns out I needed to create the disk in a folder that already exists on the esxi host. I chose to use the SWAP drive. It's not actually located there, but it's just a pointer for the disk so it has a real space to land on the vmfs side of things. example:

disk1:
vmkfstools -r /vmfs/devices/disks/naa.5000cca6acc{removed}  /vmfs/volumes/SWAP-SASRAID1/Local-1TB-Disk1.vmdk

disk2:
vmkfstools -r /vmfs/devices/disks/naa.5000cca6acc{removed}  /vmfs/volumes/SWAP-SASRAID1/Local-1TB-Disk2.vmdk

and then you'll see the RDM pointer file noted as {file}-rdm:

Local-1TB-Disk1-rdm.vmdk
Local-1TB-Disk1.vmdk
Local-1TB-Disk2-rdm.vmdk
Local-1TB-Disk2.vmdk

after that you can add them and follow the rest of the tutorial above. This is just my experience...
0
 

Expert Comment

by:pdw8
Is there a solution for this?

DISKLIB-LIB   : Only disks up to 2TB-512 are supported for 'vmfsRawDeviceMap' disk.
Failed to create virtual disk: The destination file system does not support large files (12).

I have a 3TB drive with data on I wish to copy to a new vmware platform so can't reformat it.
0
 

Expert Comment

by:crjaq
hanccocka:
I'm having an issue with a new SSHD 1 TB drive that's installed on the ESXi 4.1 host not showing the sata disk i.e. "ls -al /vmfs/devices/disks". Please see output below. Thanks!

login as: root
The time and date of this activation have been sent to the system logs.

VMware offers supported, powerful system administration tools.  Please
see www.vmware.com/go/sysadmintools for details.


~ # ls -al /vmfs/devices/disks
drwxr-xr-x    1 root     root                512 Oct  5 17:20 .
drwxr-xr-x    1 root     root                512 Oct  5 17:20 ..
-rw-------    1 root     root       146163105792 Oct  5 17:20 naa.6001c230cd19b5                                                                                                                               0019df6a6d377dcabd
-rw-------    1 root     root          939524096 Oct  5 17:20 naa.6001c230cd19b5                                                                                                                               0019df6a6d377dcabd:1
-rw-------    1 root     root         4293918720 Oct  5 17:20 naa.6001c230cd19b5                                                                                                                               0019df6a6d377dcabd:2
-rw-------    1 root     root       140925468672 Oct  5 17:20 naa.6001c230cd19b5                                                                                                                               0019df6a6d377dcabd:3
-rw-------    1 root     root            4177920 Oct  5 17:20 naa.6001c230cd19b5                                                                                                                               0019df6a6d377dcabd:4
-rw-------    1 root     root          262127616 Oct  5 17:20 naa.6001c230cd19b5                                                                                                                               0019df6a6d377dcabd:5
-rw-------    1 root     root          262127616 Oct  5 17:20 naa.6001c230cd19b5                                                                                                                               0019df6a6d377dcabd:6
-rw-------    1 root     root          115326976 Oct  5 17:20 naa.6001c230cd19b5                                                                                                                               0019df6a6d377dcabd:7
-rw-------    1 root     root          299876352 Oct  5 17:20 naa.6001c230cd19b5                                                                                                                               0019df6a6d377dcabd:8
lrwxrwxrwx    1 root     root                 36 Oct  5 17:20 vml.02000000006001                                                                                                                               c230cd19b50019df6a6d377dcabd504552432035 -> naa.6001c230cd19b50019df6a6d377dcabd
lrwxrwxrwx    1 root     root                 38 Oct  5 17:20 vml.02000000006001                                                                                                                               c230cd19b50019df6a6d377dcabd504552432035:1 -> naa.6001c230cd19b50019df6a6d377dca                                                                                                                               bd:1
lrwxrwxrwx    1 root     root                 38 Oct  5 17:20 vml.02000000006001                                                                                                                               c230cd19b50019df6a6d377dcabd504552432035:2 -> naa.6001c230cd19b50019df6a6d377dca                                                                                                                               bd:2
lrwxrwxrwx    1 root     root                 38 Oct  5 17:20 vml.02000000006001                                                                                                                               c230cd19b50019df6a6d377dcabd504552432035:3 -> naa.6001c230cd19b50019df6a6d377dca                                                                                                                               bd:3
lrwxrwxrwx    1 root     root                 38 Oct  5 17:20 vml.02000000006001                                                                                                                               c230cd19b50019df6a6d377dcabd504552432035:4 -> naa.6001c230cd19b50019df6a6d377dca                                                                                                                               bd:4
lrwxrwxrwx    1 root     root                 38 Oct  5 17:20 vml.02000000006001                                                                                                                               c230cd19b50019df6a6d377dcabd504552432035:5 -> naa.6001c230cd19b50019df6a6d377dca                                                                                                                               bd:5
lrwxrwxrwx    1 root     root                 38 Oct  5 17:20 vml.02000000006001                                                                                                                               c230cd19b50019df6a6d377dcabd504552432035:6 -> naa.6001c230cd19b50019df6a6d377dca                                                                                                                               bd:6
lrwxrwxrwx    1 root     root                 38 Oct  5 17:20 vml.02000000006001                                                                                                                               c230cd19b50019df6a6d377dcabd504552432035:7 -> naa.6001c230cd19b50019df6a6d377dca                                                                                                                               bd:7
lrwxrwxrwx    1 root     root                 38 Oct  5 17:20 vml.02000000006001                                                                                                                               c230cd19b50019df6a6d377dcabd504552432035:8 -> naa.6001c230cd19b50019df6a6d377dca                                                                                                                               bd:8
~ #

Open in new window

0
 
LVL 121

Author Comment

by:Andrew Hancock (VMware vExpert / EE MVE^2)
@crjaq

Please post a question in the VMware Zone, for myself or other Experts to Answer.

Thanks
0
 
LVL 1

Expert Comment

by:Tech Savy
Hello

It is a pleasure to even ask you a question.

The file server itself is a virtual machine running on a VMware ESXi 5.1.0 host. The D: drive is a Mapped Raw LUN which resides on a HP Lefthand SAN cluster connected via an iSCSI network

The problem is intermittent, the drive is not accessible even from the virtual machine. I checked event viewer logs to get to know the problem, but there were no logs generated. Is there a way to find out why would it be not accessible.

Your advice would be a life saver.

Thanks
0
 
LVL 121

Author Comment

by:Andrew Hancock (VMware vExpert / EE MVE^2)
@rauldeshmukh

Please post a question in the VMware Zone, for myself or other Experts to Answer.

Thanks
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Join & Write a Comment

Teach the user how to install ESXi 5.5 and configure the management network System Requirements: ESXi Installation:  Management Network Configuration: Management Network Testing:
Teach the user how to convert virtaul disk file formats and how to rename virtual machine files on datastores. Open vSphere Web Client: Review VM disk settings: Migrate VM to new datastore with a thick provisioned (lazy zeroed) disk format: Rename a…
Suggested Courses

Keep in touch with Experts Exchange

Tech news and trends delivered to your inbox every month