Here are some numbers you can take as reference when backing up your VMware ESX(i) using MKSbackup or ghettoVCB.sh. MKSBackup use ghettoVCB.sh script to backup VMware VM to disk or to NFS shares. MKSBackup can upload or download your backed up disks using FTP or SSH together with the running backup as soon as the first disk is done.
The test server is running ESXi 4.0.0 on a ASUS P5KC motherboard with a Intel Core2 Quad CPU @ 2,4GHz and 8Go RAM. Hard disks are both SATA 300Mbps 7200 rpm, first one is a Hitachi 500GB Deskstar (HDP725050GL), the second one is a Seagate Barracuda 750 GB ( ST3750640AS ).
The remote server running FTP and NFS is PIV 2.8GHz HT with 1Go RAM and 300Go SATA disk running Fedora 14.
The Windows host in a Intel Dual Core CPU @ 1.86GHz with 3Go RAM running Windows XP.
The network is gigabits.
The tests are done on multiple 8Go VMDK sparse files full at 98%.
Rate calculations below, consider the file are exactly 8Go even if only 7988Mo are really copied to hard-disk or NFS.
The files belong to a up and running virtual machine lightly loaded.
VMware vmkfstools utility can use up to 4 formats : zeroedthick, eagerzeroedthick, thin and 2gbsparse.
thin format is very interesting for backup, because when the disk is not fully loaded, unused blocks are not allocated and then the file don't require all the space on disk. NFS handle well sparse file and these files can be used by VM running on ESX(i) server. This allows to start the VM from its backup NFS share if required without any disk conversion. On the other side if you are worried about runtime speed and backup VM can become the prod one at any time, without having time of any disk conversion, then zeroedthick and eagerzeroedthick are more appropriate.
FTP and SCP don't handle sparse file, and the copy will require the full size of the file on disk. Then a 400Go empty disk will take seconds to be backed up by vmkfstools in thin format and requires hours to be uploaded via FTP or SSH. RSYNC can handle sparse file efficiently !
2gbsparse format don't generate sparse files, unused blocks are left out of the files and the size correlates the real size of the data. Then 2gbsparse is the ideal format for archiving. These files can't be used by a ESX(i) VM and require to be converted into another format before to start the VM.
In short, if your are only backing up on local disk or NFS share and are worried about space, use thin format. If you need to archive your VM using non sparse aware protocol like FTP or SCP or are backing up on non sparse aware storage system and you are worried about transfer time and storage space use 2gbsparse format.
Direct backup to NFS storage is the fastest with 56 Mo/s. FTP is very fast with 29 Mo/s, but when running together with another backup, speed drops to 7 Mo/s. On the other hand, SSH appears to be slow standalone but is as fast as FTP when backups are running simultaneously ! This is misleading because as soon as VMware backups are done, FTP is near 4x time faster to reduce the backlog left by the disk backup that is 2 time faster than FTP or SSH. To backup 4 VM disks of 50Go, backup+FTP take 4H20, backup+SSH take 6H58, this is 2H38 more than FTP !
In other words, backup to NFS share if you can, use FTP if you have only local disks and want to upload your VMs somewhere else !
Don't use FTP or SSH when backing up on NFS storage, FTP and SSH are almost frozen until the end of the backup ! I think this is because vmkfstools
open 8 simultaneous connection to the target NFS file to grab all the bandwidth.
|Operation||Time in M:S||Rate in Mo/s||comment|
|vmkfstool to second HD||8:01||17.0Mo/s|
|vmkfstool to NFS||2:25||56.5Mo/s|
|FTP on Linux host||4:41||29.1Mo/s||running MKSBackup buit-in FTP|
|FTP download from Linux host||7:46||17.56Mo/s||using ftpget on ESXi and proftpd on Linux|
|FTP on Windows host||5:08||26.6Mo/s||running MKSBackup buit-in FTP|
|FTP on Windows host from NFS||8:51||15.5Mo/s||running MKSBackup buit-in FTP|
|SCP on Windows host||17:10||7.9Mo/s||running PuTTY pscp|
|SCP on Linux host||14:50||9.2Mo/s||running openssh|
|Web-Based Datastore||13:16||10.7Mo/s||using wget from Windows|
|vmkfstools and FTP running simultaneously|
|vmkfstool to second HD||8:31||16.3Mo/s|
|FTP on Windows host||18:36||7.3Mo/s||running MKSBackup buit-in FTP|
|vmkfstools and SSH running simultaneously|
|vmkfstool to second HD||8:08||16.8Mo/s|
|SSH on Windows host||18:20||7.4Mo/s||Windows HOST running PuTTY SCP|
|For your information|
|copy local disk to the same disk||12:56||10.5Mo/s||system slightly loaded|
|copy local disk to another local disk||10:15||13.3Mo/s||system slightly loaded|
|copy disk to NFS||14:37||9.3Mo/s||system slightly loaded|
|gzip on ESXi||26:30||5.1Mo/s||compressed size is 2082Mo|
|gzip on P4||10:53||12.5Mo/s||compressed size is 2099Mo|
|rsync from Linux host using rsyncd||6:55||23.0Mo/s||first run|
|rsync from Linux host using rsyncd||8:50||~ 15.5Mo/s||second run, no changes only 1Mo transferred|
|rsync from Linux host via SSH||1:18:13||1.7Mo/s||first run|
|rsync from Linux host via SSH||13:24||~ 10.2Mo/s||second run, no changes only 1Mo transferred|