VMFS-volumes with I/O-errors

Status: work in progress

Symptoms

In worst case an I/O-error on a VMFS-volume will result in an unmountable volume.
In less severe cases the I/O-error affects only a vmware.log or another file that can be recreated easily – such as vmx-files, vmdk-descriptorfiles or vmsd-files.
In a typical case the error affects a flat.vmdk or a delta.vmdk.
The last 2 cases will result in an unstartable VM or in a vmdk that can not be copied, cloned or used in a VM.
Effectively this often means that the content of the VM is lost.

What does the VMware documentation say about this problem ?

Unfortunately I am not aware of any knowledgebase entry or other official resource that gives any useful advice on this topic.
Please correct me if I am wrong.

My 2 cents …

Most admins will first of all consider hardware related issues when they see I/O errors.
In my experience only 1/3 or less of the I/O errors I see on VMFS-volumes really are caused by hardware-problems.
The majority of cases that I have seen are related to what I call “logical VMFS errors”
In this post I will only deal with this “logical VMFS errors”
These errors occur because there is an error in the VMFS-metadata that describe files and directories of the VMFS-volume.
I found ways to work around some of these error.

DISCLAIMER:
The procedures I explain in the following may contradict the VMware documentation or Knowledgebase.

Anamnese:

First of all lets check how seriously the problem really is.
Serious: there are I/O errors in the area used for the VMFS – that means the I/O-error is located at the start of the VMFS-volume up to about  1200-1500MB
A command like
dd if=/dev/disks/vmfs-partition bs=1M count=1536 of=/tmp/vmfs-header.dump
will fail and report an I/O error. A typical location of the first I/O error is at 29 MB into the VMFS-partition.
This problem typically result in an unmountable volume.

Manageable: the I/O error affects only a single file
A command like
vmkfstools -i name.vmdk clone.vmdk
will fail with an I/O error.
The VM that uses this VMDK may still work but you can not copy or clone it anymore.
It is also possible that the VM will start and report a more specific error.


Examples:

A flat.vmdk reports an I/O error

Lets assume we already tried
vmkfstools -i /vmfs/volumes/datastoreA/directoryA/name.vmdk  /vmfs/volumes/datastoreB/directoryB/clone.vmdk 
failed with an I/O error.
Next we try
dd bs=1M if=/vmfs/volumes/datastoreA/directoryA/name.vmdk of=/vmfs/volumes/datastoreB/directoryB/clone.vmdk
this also fails with an I/O error. We see that the error is at lets say 21 MB. dd reports 20 MB were copied and then it aborted.
Now we can split the dd-command in 2 steps: first one copies the part before the error, second step is to copy the part after the error.
Like this
dd bs=1M if=/vmfs/volumes/datastoreA/directoryA/name.vmdk of=/vmfs/volumes/datastoreB/directoryB/clone.vmdk count=20
dd bs=1M if=/vmfs/volumes/datastoreA/directoryA/name.vmdk of=/vmfs/volumes/datastoreB/directoryB/clone.vmdk skip=21 seek=21
If this works the flat.vmdk has only one error and the error-range is 1 MB.
Of course in real life it will not be so easy – we can have several I/O-errors with different error-ranges.
If ESXi had the tool ddrescue it would be easy – but we have to work with dd only.
Next thing to try is a vmkfstools command like
vmkfstools -p 0 /vmfs/volumes/datastoreA/directoryA/name.vmdk > mapping-name.txt
If that works we can use the mapping-name.txt file to create a dd-command that copies the vmdk fragment by fragment.
Unfortunately it is very rare that this command will work at this stage.

When we tried all the buildin ESXi tools and all of them failed the next step is to switch to Linux.
As always I use the MOA LiveCD
After the Linux-system has finished booting and the network is up you can access it via putty or WinSCP (user: root – password: sanbarrow)
create 2 directories:
sudo su
mkdir /esxi
mkdir /vmfs-out

connect to the target ESXi-host in readonly mode
sshfs -o ro root@ip-of-esxi:/ /esxi
connect to the same ESXi-host again in a writeable mode – mounting a single directory is enough.
sshfs  root@ip-of-esxi:/vmfs/volumes/datastoreB/directoryB /vmfs-out
Now we can use ddrescue to have an easy way to skip the error-areas.
ddrescue-syntax: ddrescue <file that needs to be copied> <output-file> <copy-log-file>
Make sure to create a log-file !
ddrescue /esxi/vmfs/volumes/datastoreA/directoryA/name-flat.vmdk /vmfs-out/clone-flat.vmdk /vmfs-out/copy.log
This command will try to copy the source file block by block and it can skip blocks that can not be copied.
The result will still be a damaged flat.vmdk – but this time we will at least get a vmdk that we can work with later.
This workaround will not always work – all I can say is that the attempt to try it is worth the effort.
Keep in mind that the VMDK will probably be a complete loss if you dont try …

There is a Plan C but that exceeds the scope of this blog.
Call me if necessary.


Examples:

The VMFS-metadata area has an  I/O error

In this case you will not be able to mount the volume at all or you see corrupted directories or a bunch of corrupt vmdks.
Lets assume we already tried the normal procedure to create a VMFS header dump
dd if=/dev/disks/vmfs-partition bs=1M count=1536 of=/tmp/vmfs-header.dump
and the command failed with an I/O error.
Trying to work around the I/O error with several dd-commands is very inconvenient using ESXi only.
So we switch to Linux again.
Use the same sshfs commands to connect to the ESXi host as we used before.
Again I use ddrescue to locate the errors.
To do that I simply run
ddrescue /esxi/dev/disks/vmfs-partition /tmp/vmfs-header.dump /tmp/copy-log
I watch how it goes and abort with CTRL + C once the command copied at least 1300MB.
Inspecting the ddrescue copy log now tells me how many errors there are and how large the error-area actually is.
Now it gets a bit tricky ….
The VMFS-metadata area (the first 1200-1500MB of the VMFS-volume) has some areas that are supposed to be filled with zeroes.
If our error is located in such an area we can fix the VMFS-metadata by simply injecting zeroes.
Errors in the more important cant be fixed that easily.
The basic approach I use here is to try to get a header-dump with an error-area as small as possible.
Once I have such a dump I use it in my lab-environment and try to create dd-commands for the VMs I need to extract.

Please understand that I do not go deeper into this matter at the moment.
If you need to investigate errors like this one bad dd-command can make matters much worse.
If you need help here – call me.

 

 

 

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *