Raw Device Mapping (RDM) DeepDive

What is it?

RDM is mapping file used to map a LUN directly to VM bypassing VMFS layer. This LUN can be formatted using any File System (NTFS or FAT32) without the need to format it using VMFS and placing VMDK file on top of it.

The mapping file is located with the virtual machine directory. Here is an example of RDM mapping file for a VM called VM-01 which has pRDM LUN and vRDM LUN (will cover them later in details).

~ #  cd /vmfs/volumes/SAN-DATASTORE/VM-01

/vmfs/volumes/507fd84d-e9b583ac-b370-001b244d6cb2/VM-01 # ls -la

drwxr-xr-x    1 root     root          2800 Nov  3 12:18 .

drwxr-xr-t    1 root     root          2100 Nov  2 21:33 ..

-rw-r--r--    1 root     root            73 Oct 20 10:21 VM-01-e61f5754.hlog

-rw-------    1 root     root     2147483648 Nov  3 11:39 VM-01-e61f5754.vswp

-rw-------    1 root     root     42949672960 Nov  3 12:49 VM-01-flat.vmdk

-rw-------    1 root     root          8684 Nov  3 11:40 VM-01.nvram

-rw-------    1 root     root           541 Nov  3 11:38 VM-01.vmdk

-rw-r--r--    1 root     root             0 Oct 19 15:10 VM-01.vmsd

-rwxr-xr-x    1 root     root          3679 Nov  3 12:18 VM-01.vmx

-rw-------    1 root     root             0 Oct 20 19:04 VM-01.vmx.lck

-rw-r--r--    1 root     root           260 Nov  3 12:18 VM-01.vmxf

-rwxr-xr-x    1 root     root          3680 Nov  3 12:18 VM-01.vmx~

-rw-------    1 root     root     32212254720 Nov  3 12:18 VM-01_1-rdm.vmdk ---------------------> vRDM mapping file

-rw-------    1 root     root           482 Nov  3 12:18 VM-01_1.vmdk ----------------------> Descriptor File

-rw-------    1 root     root     32212254720 Nov  3 12:18 VM-01_2-rdmp.vmdk ---------------------> pRDM mapping file

-rw-------    1 root     root           494 Nov  3 12:18 VM-01_2.vmdk ----------------------> Descriptor File

-rw-r--r--    1 root     root        579252 Oct 20 10:21 vmware-1.log

-rw-r--r--    1 root     root        139522 Oct 20 19:03 vmware-2.log

-rw-r--r--    1 root     root        307364 Nov  3 12:19 vmware.log

-rw-------    1 root     root     112197632 Oct 20 19:04 vmx-VM-01-3860813652-1.vswp

Notice the difference between the descriptor file of a VMDK Hard drive and RDM LUN Hard Drive.

Important Note:  The size of the mapping file equals to the size of the LUN. Although RDM LUN is obtained directly from SAN, the same space of the LUN should be maintained in VMFS datastore due to mapping file.

How it works?

When a LUN is opened for access from inside the VM, the mapping file is read to obtain the reference to the raw LUN (because mapping file is having the metadata for the LUN). Thereafter, reads and writes go directly to the raw LUN rather than going through the mapping file.

VM .vmx file is having reference pointing to RDM mapping file and not the LUN. For our example VM-01

scsi0:1.fileName = "VM-01_1.vmdk"

scsi0:1.mode = "persistent"

scsi0:1.ctkEnabled = "FALSE"

scsi0:1.deviceType = "scsi-hardDisk"

scsi0:1.present = "TRUE"

scsi0:1.redo = ""

scsi0:2.fileName = "VM-01_2.vmdk"

scsi0:2.mode = "independent-persistent"

scsi0:2.ctkEnabled = "FALSE"

scsi0:2.deviceType = "scsi-hardDisk"

scsi0:2.present = "TRUE"

scsi0:2.redo = ""

When to use?

Always VMFS is preferred over RDM. But there are some scenarios where RDM is required:

SAN-based features

- SAN snapshots: Using RDM enables backup software's to ‘see’ inside the LUN and to create a consistent snapshot of the data contained within the LUN. This allows for replication of data (snapshots) to a second SAN (on a different physical location) for Disaster Recovery. 

- In-band management software for your SAN running in Virtual Machines might require direct access to a management LUN. This usually requires physical compatibility mode.

Clustering

In any MSCS clustering scenario that spans physical hosts cluster data and quorum disks should be configured as RDMs rather than as virtual disks on a shared VMFS.  The reason is that cluster members need to have access to shared LUN (quorum).

What are the available types?

RDM has two compatibility modes:

1. Virtual Mode 

In this mode all SCSI commands from the VM to LUN will pass through VMFS layer except READ & WRITE. Those will be passed directly to the LUN.

If you check the device manager of the VM you will see the RDM LUN listed as VMware Virtual Disk SCSI Disk Device. This is exactly the same as VMDK Drive added to the VM. The physical characteristics of the LUN are hidden from the VM.

The advantage of vMode is that you can still use Distributed Locking Mechanism of VMFS as well as snapshots of RDM LUN drives.

2. Physical Mode

In this mode all SCSI commands are passed to RDM LUN directly from the VM expect REPORT command virtualized by VMkernal.

If you check the device manager of the VM you will see the RDM LUN listed based on the physical name/characteristics of the storage array.

The advantage of the full exposure of the LUN characteristics to the VM can be seen when using SAN Management Tools and MSCS services. Also, RDM LUNs in physical mode can have a size greater than 2TB.

 

Another important feature inherited with RDM is Dynamic Name Resolution. Each LUN is having unique ID in RDM mapping file used by VMFS. In case the path to the LUN modified due to failover, for example, the ID of the LUN will be change. Dynamic Name Resolution feature will automatically discover the new name and associate it in the RDM mapping file to resume operations.

VMFS vs. RDM

Here is a comparison between the types of disk drives available and the supported features.

RDM interoperatibility with vSphere Advanced Features

Both pRDM and vRDM support vSphere advnaced features such as HA, DRS, SDRS, VM, SVM. However there are some limitations for some features.

vMotion

- Files for a virtual machine are not relocated when it is vMotioned. 
- The virtual machine is re-registered to the destination host. 
- Any RDMs remain as RDMs when the virtual machine is registered to another host. That is, no changes to the virtual machine itself are made.

Cold Migration

Cold migration can be for resources, datastores, or both while the VM is powered-off. In case migrating datastore is selected, here are some considerations for RDM:

- Any non-RDM virtual disks are physically moved to the destination. 
- The virtual machine configuration files are physically moved to the destination. 
- Raw LUNs themselves cannot be moved, as they are raw disks presented from the SAN. However, the pointer files (RDMs) can be relocated if required. 
- During the migration you can use the advanced section of the migration wizard and select if you would like to maintain the same format of the files at the destination (remain as RDM) or convert it to a thick/thin disk. 
- The virtual machine registration changes, but the files are left untouched.

Cloning

When cloning a VM with RDMs, pRDM & vRDM contents will be copied into VMDK virtual disks. If you want to clone a virtual machine without its RDMs, remove them from the configuration of the virtual machine before cloning. You can delete the RDM from the disk when removing it (the raw LUN contents are not changed, only the RDM mapping file is deleted). Re-add them to the configuration when completed.

Storage vMotion

- When you perform Storage vMotion, the virtual machine files are physically relocated to a destination datastore. The same host retains ownership or registration of the virtual machine after Storage vMotion completes. 
- For vSphere 4.0 and higher: 
# Virtual disks and virtual mode RDM pointer files can be relocated to the destination datastore. 
# Virtual disks can be converted to thick-provisioned or thin-provisioned disks during migration as long as  the destination is not an NFS datastore. 
# Physical mode RDM pointer files can be relocated to the destination datastore, but physical mode RDM data cannot be migrated. 
# Virtual mode RDM data can be migrated with storage vMotion, provided sufficient licensing is present.

Notes:

- In ESX 5.0, during the Storage vMotion, virtual mode RDMs can be converted to thick/thin provisioned disks by selecting the appropriate options under the Advanced section of the migration wizard. Physical mode RDMs can only be converted with a cold migration. 
- If you are attempting a Storage vMotion of a virtual mode RDM using the Advanced method and the RDM pointer mapping file is already present in the target datastore, the Storage vMotion finishes quickly, but without moving the data. This is because the Storage vMotion detects that the source and target datastores for the mapping file are the same, and therefore concludes that no movement is needed.


To work around this issue: 
1. Remove the RDM from the virtual machine, and delete it from disk to ensure the mapping file is deleted. This will not delete the data in the RDM. 
2. Add the RDM again, and put the mapping file in a different datastore than the ultimate target of the storage vMotion. 
3. After the virtual machine reconfiguration is complete, attempt the storage vMotion again.

RDM Restrictions

1. RDM can't be used with NFS datastore. 
2. SVM can't be used with NPIV

RDM Tips

In case RDM is greyed out:

1. Make sure that you have a free LUN which support RDM 

2. Make sure that RDMFiler Advanced parameter is unchecked

Note: In case a LUN is used by ESXi host (even if partially), you can't use it as RDM to VM and vice versa. The reason is that OS or Hypervisor will apply lock into the LUN which will prevent it to be used by other system.

After adding the RDM to VM, you don't need to format it (assuming that it was used on previous system and was previously formatted). You need ONLY to bring it online using disk management. In case of fresh new RDM assigned to VM, it needs to be formatted for first use.

Note: RDM isn’t support with vCloud and should be avoided.