Symptoms
  • Powering on or migrating (vMotion) a virtual machine with a raw device mapping (RDM) may fail.
  • You see the error:

    Virtual Disk 'X' is a mapped direct access LUN that is not accessible
     
  • When checking the VML identifier for a RDM on two or more ESX hosts, you see they are not referring to the same VML ID.

    Note: This is referred to as as a VML mismatch.
 Cause

This is an informational event which indicates that the host attempting to start or receive the virtual machine cannot access the device (LUN) backing the raw disk mapping.

For more information on raw device mappings, see the Raw Device Mapping section of the ESX/ESXi Server Configuration Guide for your version.

A raw device mapping file contains metadata, specifically SCSI vital product data (VPD) page 0x83, for uniquely identifying a device. A host attempting to access the physical LUN reads the mapping file metadata, and searches the devices known to the host for a LUN with the same VPD page 0x83 information.

The contents of SCSI VPD page 0x83 should match when requested by all hosts in a vSphere cluster, and the RDM should be accessible by all hosts in the cluster. However, discrepancies in device presentation between hosts in a cluster may lead to different hosts receiving different contents for VPD page 0x83 for the same device. In this case, hosts which did not create the RDM mapping file may be unable to locate the raw device mapping LUN when powering on or migrating a virtual machine.

Note: Cold migrations of powered off virtual machines do not require access to the raw device mapping. Symptoms may be noticed after cold migration, when the virtual machine is powered on.

 Resolution

Identifying mis-presented devices using VML identifiers

The VPD page 0x83 information, when available, forms the basis of the unique VML identifier which the ESX/ESXi host assigns to each storage device. This VML identifier should be consistent for a given device across each host in a vSphere cluster.

Identify the device being used for a raw device mapping by its VML identifier, such as by reviewing the contents of /vmfs/devices/disks/, on each host in a cluster. Compare the VML identifier obtained for the device on each host. For more information, see Identifying disks when working with VMware ESX (1014953).

The highlighted sections of the VML identifiers in this example do not match between the hosts, indicating a LUN with different presentation to each host:

  • Host 1:

    vml.02000500006006048000019010406353303035344553594d4d4554 -> naa.60060480000190104063533030353445
     
  • Host 2:

    vml.02004d00006006048000019010406353303035344553594d4d4554 -> naa.60060480000190104063533030353445
Note: In the above example, the highlighted value equates to the device's LUN number represented in hexadecimal. On host 1, the LUN has been presented as LUN 5, and on Host 2, it has been presented as LUN 77.

LUN presentation (including LUN numbering) should be made consistent for every host participating in a cluster that could run the virtual machine. The raw device mapping metadata file should be consistent with that presentation, and vCenter Server's cache of this information should be accurate.

Segmented cluster:

If one segment of a vSphere Cluster has correct presentation of the LUN being used for a raw device mapping, but other segments do not, the virtual machine operates correctly only within the segment with correct presentation. Correct the presentation in the nonworking segment(s):

  1. Unpresent the LUNs from the hosts with incorrect presentation. For more information, see Removing a LUN containing a datastore from VMware ESXi/ESX 4.x (1029786) and Unmounting a LUN or detaching a datastore/storage device from multiple ESXi 5.x hosts (2004605).
  2. Represent the LUNs in the same manner as the working hosts. For more information, engage your storage array vendor.
  3. Rescan to pick up storage changes. For more information, see Performing a rescan of the storage (1003988).
  4. Verify that the host LUN ID matches on all the hosts in the cluster that have access to that LUN.
  5. Verify that the VML ID for the device matches on all the hosts in the cluster.
  6. If the host LUN ID matches and the VML ID does not match:
     
    1. Unpresent the LUN on the host where the VML ID does not reflect the correct host LUN ID. For more information, see Removing a LUN containing a datastore from VMware ESXi/ESX 4.x (1029786) or Unmounting a LUN or Detaching a Datastore/Storage Device from multiple ESXi 5.x hosts (2004605).
    2. Perform a rescan, then represent the LUN. Verify that the VML ID matches.
       
  7. Attempt migration of the virtual machine from the working segment to the corrected hosts.
Note: The above procedure may result in the VML mismatch issue still being present due to the way ESX caches vml devices. If the issue persists, repeat the procedure, but this time represent the LUN from the array with a different LUN ID. This forces a new VML identifier to be created for the LUN.

Note: An alternate method of correcting VML IDs for mispresented LUNs:
  1. Note the NAA_ID of the LUN.
  2. Detach RDM using vSphere client.
  3. Un-present the LUN from host on array.
  4. Rescan host storage.
  5. Remove LUN from detached list using these commands:

    esxcli storage core device detached list
    esxcli storage core device detached remove -d NAA_ID
     
  6. Rescan the host storage.
  7. Re-present LUN to host.
  8. Rescan host
If the LUN has been flagged as perennially reserved, this can prevent the removal from succeeding.

Run this command to remove the flag:
# esxcli storage core device setconfig -d NAA_ID --perennially-reserved=false

Now the command to remove the device should work.
# esxcli storage core device detached remove -d NAA_ID

There is no harm in setting the flag to false, even if it is not needed, so it might be simpler to always run both commands.

Whole cluster:

If a virtual machine cannot be started on any host in a vSphere Cluster, the presentation to each host may differ from the VPD Page 0x83 information stored within the raw device mapping file. Recreate the mapping file:

  1. Power down the virtual machine.
  2. Right-click the virtual machine and click Edit Settings.
  3. Make note of the raw device mapped disks which are attached to the virtual machine.
  4. Remove the raw device mapped disks from the virtual machine.
  5. Click OK.
  6. Right-click the virtual machine and click Edit Settings.
  7. Click Add to re-add the previously removed raw device(s).
  8. Click OK.
  9. Power on the virtual machine and re-attempt the operation which previously failed.

vCenter Server cache:

The vCenter Server maintains a cache of storage topology information for virtual machines. If this information has become out of sync with the presentation, power operations or migrations initiated from vCenter Server may fail. Remove the affected virtual machine from inventory and re-register it to clear the cached information:

  1. Power down the virtual machine.
  2. Right-click the virtual machine and remove it from the inventory.
  3. Re-register the virtual machine to the inventory. For more information, see Registering or adding a virtual machine to the inventory (1006160).
  4. Power on the virtual machine and retry the operation which previously failed.