Create a VMFS-Metadata-dump using an ESXi-Host in production

1. Required: root-access to an ESXi-host via ssh
2. Identify the device that corresponds to the affected datastore:

login with root account
cd /dev/disks
ls -lisa | grep -v vml

In many cases you can identify the correct device by inspecting the referenced filesize – typically several hundred of GBs or several TBs.
If lots of datastores with similar size are in used – use
esxcli-scsidevs -m
for a more detailed description of the available devices.

3. dd command to dump the first 1536 MB of DeviceX into a file
dd if=/dev/disks/Device:1 bs=1M count=1536 of=/tmp/Casename.1536

3a. Very often there is not enough free space available in /tmp
Workaround: dump into an archive:

dd if=/dev/disks/Device:1 bs=1M count=1536 | gzip -c /tmp/Casename.1536.gz

3b. if that still does not work use another datastore – BUT never use the affected datastore itself!!!
dd if=/dev/disks/Device:1 bs=1M count=1536 of=/vmfs/volumes/ANOTHER-UNAFFECTED-DATASTORE/Casename.1536

4. connect to ESXiHost via WinSCP
download /tmp/Casename.1536 to admin-host

5. download the tool strings.exe from
after download unzip strungs.exe and copy it to the same path that already has Casename.1536

open a cmd-box and execute
strings.exe Casename.1536 > Casename.1536.txt
Search through Casename.1536.txt
That file will contain information that is confidential and you may not be allowed to pass it to a 3rd party.
If you dont care about this – compress Casename.1536 with an effective packer like 7zip or rar.

6. you should now have an archive that varies in size – typically range is 50 MB – 800 MB

7. upload the archive to a freehostser, your webserver, or any other public location with a decent downloadrate.
(skype can be used too – but is a comparably slow option)
When upload is done – provide a downloadlink – typically this also is the perfect time for a short slype-chat

8. in most cases it takes one or two hours to get a solid overview of the prognosis and available recovery options.

There is a Knowledgebase-article that discuss the same topic – see

Continue reading →

Latest LiveCD Download

This is the first version of a Commandline-only-LiveCD.

There is quite a large number of good Linux LiveCDs to chose from when you are looking for a toolbox for system-administrators. My personal favorite sure is PartedMagic
Unfortunately there is none that has all the tools that I need for my recovery work
Often I could get away with a recent Ubuntu-LiveCD by adding the essential tools using apt-get.

In most recovery cases fighting to get your regular tools is nothing that you want to happen too often.
Booting a Linux LiveCD inside a remote production-environment by giving instructions via skye or phone can be tricky enough.

So the expected use-case for this LiveCD can be described in this typical dialogue:

CUSTOMER> I have a problem with my VMFS-Volume …
ME> Let me see what I can do for you …
CUSTOMER> What do you need to get started ?
ME> Enable ssh, get putty and winscp , download this ISO-file. When ready give me a Teamviewer-login to your admin-host.

Depending on the type of connection there are different ways to get access to the damaged volume:

– VMFS-volume is stored on a local SCSI or SATA drive
– VMFS-volume is stored on a remote iSCSI-LUN
– VMFS-volume is stored on FibreChannel-LUN
– VMFS-volume is stored in a VMDK-file

Sometimes a Volume has to remain active as it is used for production – sometimes it can be unmounted:

– Volume is active – no exclusive access allowed/possible
– Volume is unmounted – exclusive access allowed/possible

Depending on the local environment different types of hosts can be used for the Recoveryenvironment

– any spare physical machine
– a physical ESXi host
– a VM running on one of the local ESXi-hosts
– a VM running inside Workstation installed on the remote admin-host

In all the cases listed above the Recovery-Environment must be able to maintain reliable direct access to the damaged
VMFS-volume and offer several options to store the recovered VMs.

The term LiveCD suggests that it only can be started from a physical CD-drive or ISO-file.
This release is not limited to CD-drive or ISO-files thats why I prefer to call it “Recovery Environment”

– can boot from CD-drive or ISO
– can boot from USB-flash-drives or USB-disks
– can boot from harddisk (IDE, SATA or SCSI)
– can boot like a regular bootable vmdk – just add a vmdk-descriptor that references the ISO-file

– can be deployed as OVA

– can boot from MBR and UEFI – allows to boot it inside almost all recent VMs on ESXi, WS or Fusion
– needs 64bit support to be enabled inside the VM

My list of required tools – to name just the essentials – looks like this:

– latest version of vmfs-tools
– sshfs
– tools to manage GPT disks
– complete iSCSI-support
– complete NFS-support
– ddrescue, testdisk, photorec and other forensic essentials
– web-interface to make some common tasks easier

The experienced admin may miss the following addons:

– vmware-mount included in the VDDK-package
– esxcli included in the VMware-CLI
– vmrun included in the VMware-VIX package

Unfortunately these packages are not redistributable so I can not include them.
As a workaround I include the
– dependancies for non-distributable VMware-packages
so that the mentioned tools can be installed on the fly when the Recovery-environment is booted with sufficient memory.

Even without vmware-mount I think a collection of tools for the expected scenarios should be able to mount the most frequent types of virtual disks so I added further commandline-tools like guestfish …
This allows stunts that probably should only be attempted by experts like
– mount vmdks – even when the vmdks are locked by ESXi


This collection of tools was created because I need this set of tools for
– recovery of damaged VMFS-volumes
– P2V using the Coldclone-approach

I do not claim that the collection includes every tool that would be useful in the VMware-context.

This collection simply includes 99 % of the tools I use whenever I offer remote-support.
Feel free to use it if you have similar needs.


Try this first – should boot on most systems

Try this version when the EFI-version fails – old hosts may prefer this