Update 2013-02-28: I’ve updated the driver to include the datastore id in the path, so it is now possible to use the driver with multiple datastores. Also the driver now correctly downloads drivers that are imported from URLs, eg. via the marketplace.

Based on this blog article I created an updated datastore driver that allows you to use a ZFS backend with OpenNebula. This datastore driver implements snapshot functionalitiy to clone images. I have created this driver to be able to start a VM with a persistent disk without having to wait untill the full image file is copied in the datastore.

The driver implements updated versions of the cp, the clone and the rm commands. I don’t use the mkfs command, so I have not implemented this with the ZFS datastore driver yet.

Due to the nature of the OpenNebula datastore layout and the ZFS snapshot capabilities, I had to make a workaround. For the sake of simplicity I dediced to use the filesystem driver as a basis. This means that the images are files in a datastore directory. ZFS snapshots are based on a directory level. To make snapshotting possible I created a link in the datastore location to the image file in a ZFS backed NFS directory.

Please find a short outline based on the blog article, and my own additions. The ZFS datastore driver is available as a tar donwload:zfs-datastore-v1.1

Setup the ZFS part
Install openindiana, create a zfs pool, create all the necessary ZFS volumes and share this as an NFS share with the frontend server.

# zfs create tank/export/home/cloud
# zfs set mountpoint=/srv/cloud tank/export/home/cloud
# zfs create tank/export/home/cloud/images
# chown -R oneadmin:cloud /srv/cloud
# zfs set sharenfs='rw=@' tank/export/home/cloud
# zfs set sharenfs='root=@' tank/export/home/cloud
# zfs allow oneadmin destroy,clone,create,mount,share,sharenfs tank/export/home/cloud

Setup ZFS volume per datastore
For each datastore that you want to host on the ZFS server you have to create a volume and allow oneadmin to manage it. Replace datastoreid with the id of the datastore you will create (eg. 100). Make sure to chown the directory to allow oneadmin to access it.

# zfs create tank/export/home/cloud/datastoreid
# zfs allow oneadmin destroy,clone,create,mount,share,sharenfs tank/export/home/cloud/datastoreid
# chown oneadmin:other /srv/cloud/datastoreid

Install ZFS datastore driver

Unpack the tar file with the driver in the datastore remote directory (/var/lib/one/remotes/datastore/)

$ tar xvf zfs-datastore.tar

Configure the ZFS datastore driver with the correct parameters and make sure passwordless connectivity is possible between the ZFS host and the frontend.


ZFS_BASE_PATH=/export/home/cloud ## this is the path that maps to /srv/cloud
ZFS_LOCAL_PATH=/srv/cloud ## relative one ZFS_BASE_PATH

Configure OpenNebula to use ZFS datastore driver

Update the datastore configuration in oned.conf with zfs driver. See example below:

executable = "one_datastore",
arguments = "-t 15 -d fs,vmware,iscsi,zfs"

Create new ZFS datastore

When the ZFS stuff is all done, make sure this NFS share is mounted on the frontend. In my example this is mounted on /srv/cloud. Now you can create a datastore with this new driver.


NAME = zfstest
DS_MAD = zfs
TM_MAD = ssh

$ onedatastore create zfstest.conf

Create new image in ZFS datastore

Create conf file to create new image:

# cat /tmp/centos63-5gb.conf
NAME = "Centos63-5GB-zfs"
PATH = /home/user/centos63-5gb.img
DESCRIPTION = "CentOS 6.3 5GB image contextualized"

Run oneimage command on the right datastore:

[oneadmin@cloudcontroller1 ~]$ oneimage create -d 100 /tmp/centos63-5gb.conf
ID: 111

The image path to be used to create a snapshot can be found by checking the image details of the newly created image (id 111 in our example):

$ oneimage show 111
SOURCE : /var/lib/one/datastores/100/4ce405866cf95a4d77b3a9dd9c54fa73

To use ths image as a golden image, create a snapshot on the ZFS server, so this snapshot can be the basis for the future clones. Instant cloning relies on the relevant ZFS capabilities, which allows creating a new ZFS dataset (clone) based on an existing snapshot. This means the snapshot has to be created first. So after you upload the golden image, manually create a snapshot of this image. This only needs to be done once as this snapshot can be used as many times as possible. This command needs to be run on the ZFS server, not the frontend!

# zfs snapshot tank/export/home/cloud/100/4ce405866cf95a4d77b3a9dd9c54fa73@golden

Now this image can be used for instand cloning.

Example ZFS datastore content on the frontend

[oneadmin@cloudcontroller1 ~]$ cd /var/lib/one/datastores/100/
[oneadmin@cloudcontroller1 100]$ ls -al
total 28
drwxr-xr-x 2 oneadmin oneadmin 4096 Nov 7 02:43 .
drwxr-xr-x 6 oneadmin oneadmin 4096 Oct 15 17:04 ..
lrwxrwxrwx 1 oneadmin oneadmin 76 Oct 16 03:21 25473f081ba733822f3e9ba1df347753 -> /srv/cloud/25473f081ba733822f3e9ba1df347753/25473f081ba733822f3e9ba1df347753
lrwxrwxrwx 1 oneadmin oneadmin 76 Oct 16 02:53 2bf829fedb6e1728f204be8a19ff8f8c -> /srv/cloud/2bf829fedb6e1728f204be8a19ff8f8c/2bf829fedb6e1728f204be8a19ff8f8c
lrwxrwxrwx 1 oneadmin oneadmin 76 Oct 19 17:41 4ce405866cf95a4d77b3a9dd9c54fa73 -> /srv/cloud/4ce405866cf95a4d77b3a9dd9c54fa73/4ce405866cf95a4d77b3a9dd9c54fa73
lrwxrwxrwx 1 oneadmin oneadmin 76 Oct 16 03:24 a00d08dd9b7447818e110115cbc33056 -> /srv/cloud/a00d08dd9b7447818e110115cbc33056/25473f081ba733822f3e9ba1df347753
lrwxrwxrwx 1 oneadmin oneadmin 76 Oct 16 03:18 cab665db977255c4c76c7aa3d687a6d6 -> /srv/cloud/cab665db977255c4c76c7aa3d687a6d6/2bf829fedb6e1728f204be8a19ff8f8c

Example output of ZFS volumes

root@openindiana:/home/rogierm# zfs list
rpool 5.68G 361G 45.5K /rpool
rpool/ROOT 1.56G 361G 31K legacy
rpool/ROOT/openindiana 1.56G 361G 1.55G /
rpool/dump 2.00G 361G 2.00G -
rpool/export 133K 361G 32K /export
rpool/export/home 101K 361G 33K /export/home
rpool/export/home/oneadmin 34K 361G 34K /export/home/oneadmin
rpool/export/home/rogierm 34K 361G 34K /export/home/rogierm
rpool/swap 2.12G 362G 133M -
tank 35.3G 1.04T 32K /tank
tank/export 35.3G 1.04T 32K /tank/export
tank/export/home 35.3G 1.04T 31K /tank/export/home
tank/export/home/cloud 35.3G 1.04T 15.3G /srv/cloud
tank/export/home/cloud/1872dba973eb2f13ef745fc8619d7c30 1K 1.04T 5.00G /srv/cloud/1872dba973eb2f13ef745fc8619d7c30
tank/export/home/cloud/25473f081ba733822f3e9ba1df347753 5.00G 1.04T 5.00G /srv/cloud/25473f081ba733822f3e9ba1df347753
tank/export/home/cloud/28e04ccbc4e55779964330a2131db466 1K 1.04T 5.00G /srv/cloud/28e04ccbc4e55779964330a2131db466
tank/export/home/cloud/2bf829fedb6e1728f204be8a19ff8f8c 40.0M 1.04T 40.0M /srv/cloud/2bf829fedb6e1728f204be8a19ff8f8c
tank/export/home/cloud/4ce405866cf95a4d77b3a9dd9c54fa73 5.00G 1.04T 5.00G /srv/cloud/4ce405866cf95a4d77b3a9dd9c54fa73
tank/export/home/cloud/8b86712ae314bc80eef1dfc303740a87 1K 1.04T 5.00G /srv/cloud/8b86712ae314bc80eef1dfc303740a87
tank/export/home/cloud/931492cd32cb96aad3b8dce4869412f3 1K 1.04T 5.00G /srv/cloud/931492cd32cb96aad3b8dce4869412f3
tank/export/home/cloud/a00d08dd9b7447818e110115cbc33056 5.00G 1.04T 5.00G /srv/cloud/a00d08dd9b7447818e110115cbc33056
tank/export/home/cloud/cab665db977255c4c76c7aa3d687a6d6 1K 1.04T 40.0M /srv/cloud/cab665db977255c4c76c7aa3d687a6d6
tank/export/home/cloud/images 5.00G 1.04T 32K /srv/cloud/images
tank/export/home/cloud/images/centos6 5.00G 1.04T 5.00G /srv/cloud/images/centos6
tank/export/home/cloud/one 63K 1.04T 32K /srv/cloud/one
tank/export/home/cloud/one/var 31K 1.04T 31K /srv/cloud/one/var

One thought on “OpenNebula with ZFS datastore

  1. Pingback: blog.opennebula.org » Archives » OpenNebula Newsletter – December 2012

Leave a reply