VMware ESXi backup speed: compare NFS, FTP, RSYNC and SSH

Here are some numbers you can take as reference when backing up your VMware ESX(i) using MKSbackup or ghettoVCB.sh. MKSBackup use ghettoVCB.sh script to backup VMware VM to disk or to NFS shares. MKSBackup can upload or download your backed up disks using FTP or SSH together with the running backup as soon as the first disk is done.

The configuration

The test server is running ESXi 4.0.0 on a ASUS P5KC motherboard with a Intel Core2 Quad CPU @ 2,4GHz and 8Go RAM. Hard disks are both SATA 300Mbps 7200 rpm, first one is a Hitachi 500GB Deskstar (HDP725050GL), the second one is a Seagate Barracuda 750 GB ( ST3750640AS ).
The remote server running FTP and NFS is PIV 2.8GHz HT with 1Go RAM and 300Go SATA disk running Fedora 14.
The Windows host in a Intel Dual Core CPU @ 1.86GHz with 3Go RAM running Windows XP.
The network is gigabits.

The tests are done on multiple 8Go VMDK sparse files full at 98%.
Rate calculations below, consider the file are exactly 8Go even if only 7988Mo are really copied to hard-disk or NFS.
The files belong to a up and running virtual machine lightly loaded.

The format: 2gbsparse, thin and others

VMware vmkfstools utility can use up to 4 formats : zeroedthick, eagerzeroedthick, thin and 2gbsparse.

thin format is very interesting for backup, because when the disk is not fully loaded, unused blocks are not allocated and then the file don't require all the space on disk. NFS handle well sparse file and these files can be used by VM running on ESX(i) server. This allows to start the VM from its backup NFS share if required without any disk conversion. On the other side if you are worried about runtime speed and backup VM can become the prod one at any time, without having time of any disk conversion, then zeroedthick and eagerzeroedthick are more appropriate.

FTP and SCP don't handle sparse file, and the copy will require the full size of the file on disk. Then a 400Go empty disk will take seconds to be backed up by vmkfstools in thin format and requires hours to be uploaded via FTP or SSH. RSYNC can handle sparse file efficiently !

2gbsparse format don't generate sparse files, unused blocks are left out of the files and the size correlates the real size of the data. Then 2gbsparse is the ideal format for archiving. These files can't be used by a ESX(i) VM and require to be converted into another format before to start the VM.

In short, if your are only backing up on local disk or NFS share and are worried about space, use thin format. If you need to archive your VM using non sparse aware protocol like FTP or SCP or are backing up on non sparse aware storage system and you are worried about transfer time and storage space use 2gbsparse format.

Conclusion

Direct backup to NFS storage is the fastest with 56 Mo/s. FTP is very fast with 29 Mo/s, but when running together with another backup, speed drops to 7 Mo/s. On the other hand, SSH appears to be slow standalone but is as fast as FTP when backups are running simultaneously ! This is misleading because as soon as VMware backups are done, FTP is near 4x time faster to reduce the backlog left by the disk backup that is 2 time faster than FTP or SSH. To backup 4 VM disks of 50Go, backup+FTP take 4H20, backup+SSH take 6H58, this is 2H38 more than FTP !

In other words, backup to NFS share if you can, use FTP if you have only local disks and want to upload your VMs somewhere else !

Don't use FTP or SSH when backing up on NFS storage, FTP and SSH are almost frozen until the end of the backup ! I think this is because vmkfstools
open 8 simultaneous connection to the target NFS file to grab all the bandwidth.

Operation Time in M:S Rate in Mo/s comment
vmkfstool to second HD 8:01 17.0Mo/s  
vmkfstool to NFS 2:25 56.5Mo/s  
FTP on Linux host 4:41 29.1Mo/s running MKSBackup buit-in FTP
FTP download from Linux host 7:46 17.56Mo/s using ftpget on ESXi and proftpd on Linux
FTP on Windows host 5:08 26.6Mo/s running MKSBackup buit-in FTP
FTP on Windows host from NFS 8:51 15.5Mo/s running MKSBackup buit-in FTP
SCP on Windows host 17:10 7.9Mo/s running PuTTY pscp
SCP on Linux host 14:50 9.2Mo/s running openssh
Web-Based Datastore 13:16 10.7Mo/s using wget from Windows
vmkfstools and FTP running simultaneously
vmkfstool to second HD 8:31 16.3Mo/s  
FTP on Windows host 18:36 7.3Mo/s running MKSBackup buit-in FTP
vmkfstools and SSH running simultaneously
vmkfstool to second HD 8:08 16.8Mo/s  
SSH on Windows host 18:20 7.4Mo/s Windows HOST running PuTTY SCP
For your information
copy local disk to the same disk 12:56 10.5Mo/s system slightly loaded
copy local disk to another local disk 10:15 13.3Mo/s system slightly loaded
copy disk to NFS 14:37 9.3Mo/s system slightly loaded
gzip on ESXi 26:30 5.1Mo/s compressed size is 2082Mo
gzip on P4 10:53 12.5Mo/s compressed size is 2099Mo
rsync from Linux host using rsyncd 6:55 23.0Mo/s first run
rsync from Linux host using rsyncd 8:50 ~ 15.5Mo/s second run, no changes only 1Mo transferred
rsync from Linux host via SSH 1:18:13 1.7Mo/s first run
rsync from Linux host via SSH 13:24 ~ 10.2Mo/s second run, no changes only 1Mo transferred

Comments

I performed a quickt test with ftp upload on a small. this is much much faster then pscp!
many thanks!!

Thanks a lot for this great article
Corrine Dipas
corrine.dipas@mail.com

hello, sorry for posting here but it's been 3 hour of intense googling with no luck and I figured you're a good bet to solve my problem :) I'd like to use ghettovcb to copy (with vmkfstool) my VM from a local datastore to another local datastoreboth phisical disks, like in your very first test "vmkfstool to second HD, 17.0Mo/s".
Well, I expected it to be quite fast but.... copying a 1.8Tb thin VM (about 300Gb of real data) I got not more than 3Mb/sec.. The host is an IBM x3400 server, xeon something and 12gb ram with a LSI raid controller; the man datastore is on a RAID1 array and the "backup" datastore on a single SATA drive..
it sure isn't the fastest setup around, but... 3Mb/sec? there's something wrong here :)
can someone explain this? thank you!

aspineux's picture

Your source or destination disk is/are maybe very busy ?
3Mb/s is too slow !

Did you ever get a resolution to this? This looks like my exact setup (though I have actually tried a backup yet for the speed I might be getting - I'm in the research phase at the moment).

Add new comment