Undelete VMDKs with VMFS

As you probably already know none of the VMFS-versions so far has an “Undelete” function.
So there is no way to restore a virtual disk if it was deleted manually by accident in a reliable way.

In the past few years I learned some procedures to work around this VMFS-limitation.
But to tell you the bad news first  – none of this procedures is 100% reliable.
So if you ask me : “Can you recover a vmdk that I deleted accidentaly ?” the answer will always be: “it depends …”

The most critical aspects are:

Condition Bad Good
activity on the VMFS-datastore after the VMDK was deleted production use no activity
Provisioning type of the VMDK thin thick
Size of the VMDK larger than 256 GB smaller than 256 GB
ESXi Version 6.0 3.5 – 5.*
free space on the datastore at the time the VMDK was created the more the better
free space on the datastore at the time the VMDK was deleted the more the better

To estimate the chances for a recovery it is required to understand that there are 2 possible cases:

Case 1:  the location of the VMDK inside the VMFS-volume is still referenced in the VMFS-Metadata.
Case 2:  the location of the VMDK inside the VMFS-volume has to be guessed

ESXi 3.5 upto 5.* usually simply dereferenced the name of the VMDK without cleaning the Inode and the mapping details.
The Linux-tool vmfs-fuse very often still displayed the VMDK as if it had never been deleted.
This behaviour seems to have changed with ESXi 6. With the new version the “delete file” process now does a cleaner job.
Now it looks like it dereferences the file-name, the Inode and the mapping details – as a result of the changed behaviour I could not recover
a single deleted VMDK from an ESXi 6 datastore so far.
It is a bit too early to say that ESXi 6 prevents a recovery with the help of “vmfs-fuse” – but right now it looks like that.

Let me give you a quick overview of the procedure that I use nowadays:

1. user deletes a VMDK
2. as soon as this gets noticed the following steps will improve the chances for a successful recovery:
– do not create new VMs on the datastore
– do not create new snapshots on the datastore
– if possible power off all VMs that use thin-provisioned VMDKs
– collect information about the lost VMDK such as size, filesystem, provisioning type, guestOS
3. create a dump of the first 1.5 Gb of the VMFS-volume

4. contact me
5. the dump of the VMFS-header-files can then be used to find out if an attempt to recover the deleted VMDK makes sense or not.
I need about an hour to analyse the dump and once I got the results we can discuss what to do next.

 

VMFS-Metadata Provision Size Chances Suggestion
available thick < 256 GB good do it yourself
missing thick < 256 GB depends on expected fragmentation “raw recovery procedure”
available thin < 256 GB good do it yourself
missing thin < 256 GB bad call Ontrack
available thick > 256 GB good “ask me later”
missing thick > 256 GB depends on expected fragmentation “raw recovery procedure”
available thin > 256 GB medium “ask me later”
missing thin > 256 GB bad call Ontrack

Leave a Reply

3 comments

  1. caymanwent says:

    Hello Ulli,

    I’ve sent you a Skype request and replied to a thread on communities.vmware.com.

    Are you available to chat sometime today briefly?

  2. mahesh says:

    Hi,

    One of my engineer accidentally deleted a VM. I want to recover it.

    Can you help me in this ?

    Thanks
    Mahesh

  3. paul.braren says:

    This is very helpful guidance, as I begin to tinker in the (largely undocumented) world of VMFS recovery!