Several blogs and manuals with examples on kvm or xen setups use NFS as storage backend. Mostly they state that for production use iSCSI is recommended. However there are examples where NFS is part of the architecture, eg. OpenNebula. I tried to find specific statistics on the performance differences between NFS, iSCSI and local storage. During this search I encountered some pointers that NFS and Xen is not a good combination, but never a straight comparison.
I decided to invest some time and setup a small test environment and run some bonnie++ statistics. This is not a scientific designed experiment, but a test to show the differences between the platforms. Two test platforms are setup, 1 with a Xen server (DL360G6) (xen1) and a 12 disk SATA storage server (storage1), and another with a KVM server (DL360G5) (kvm1) and a 2 disk SATA storage server (storage2) . Both servers are connected with a gigabit network. I’ve also run a test with a 100mb/s network between the kvm1 and storage2 server. For reference I’ve also done tests with the images on localdisk.
I realize that LVM and iSCSI storage is most efficient, but storage with image files is very convenient and in case of cloud setups sometimes the only option.
| Seq output | Seq input | Random | |||||||||||
| Per Chr | Block | Rewrite | Per Chr | Block | Seeks | ||||||||
| Size | K/sec | %CP | K/sec | %CP | K/sec | %CP | K/sec | %CP | K/sec | %CP | /sec | %CP | |
| Xen-guest-via-nfs-tapaio | 1G | 3570 | 5 | 2436 | 0 | 1366 | 0 | 26474 | 41 | 24831 | 0 | 6719.0 | 1 |
| xen-guest-via-iscsi | 1G | 25242 | 40 | 12071 | 1 | 15175 | 0 | 32071 | 42 | 47742 | 0 | 7331.3 | 1 |
| kvm-guest-nfs-1gb-net | 1G | 8140 | 16 | 17308 | 3 | 11864 | 2 | 40861 | 81 | 71711 | 3 | 2126.6 | 54 |
| kvm-guest-nfs-qcow-100mb | 1G | 1922 | 3 | 9874 | 1 | 3994 | 0 | 10720 | 22 | 10441 | 0 | 595.4 | 33 |
| kvm-guest-nfs-qcow-100mb-2nd | 1G | 9735 | 21 | 2039 | 0 | 3197 | 0 | 10729 | 22 | 10463 | 0 | 685.3 | 38 |
| kvm-guest-nfs-qcow-100mb-3rd | 1G | 5327 | 10 | 7378 | 1 | 4421 | 0 | 10655 | 18 | 10512 | 0 | 706.3 | 39 |
| xenserver-nfsmount | 1G | 41507 | 60 | 60921 | 7 | 29687 | 1 | 33427 | 48 | 64147 | 0 | 4674.4 | 11 |
| kvmserver-nfs-1G | 20G | 31158 | 52 | 32044 | 17 | 10749 | 2 | 19152 | 28 | 18987 | 1 | 90.3 | 1 |
| localdisk-on-nfs-server-cloudtest3 | 4G | 41926 | 65 | 43805 | 7 | 18928 | 3 | 52943 | 72 | 56616 | 3 | 222.6 | 0 |
The conclusion of the tests is that local storage is fastest. NFS storage with Xen is not a good combination. Xen runs best with iSCSI backed storage. KVM with NFS runs significantly better. It is safe to say that if you want to use NFS use it with KVM, not with Xen. In any case iSCSI is always the best option for Xen. I have not yet tested KVM with iSCSI but I expect this to perform better than NFS.
Pingback: Xen en NFS performance in de guests