Notes on testing standard standard image-backed ephemeral instances vs. boot-volume-backed instances initialized from images in NDSLabs Kubernetes cluster.
Image-backed VMs copy the image to the hypervisor from glance, which is backed by gluster on nebula, which is further backed by a ZFS storage system. The system image is local to the hypervisor and is considered to be ephemeral.
Boot-volume backed instances use data from a cinder volume, which is initialized from another volume, snapshot, or image. Cinder volumes are mapped as block-devices from the storage system over iscsi, and are persistent.
Advantages to volumes:
- Migration - storage volume is remote mounted, so moving an instance is very quick as no data movement is required
- Snapshots - volume snapshots occur in cinder and typically don't require freezing the instance during the snapshot
- Sizing - volume sizing is more flexible than image/flavor sizing from image minimum to maximum volume provision limit in GB increments.
- Persistence - the system volume can be preserved even without the system, permitting openstack server resources to be released but maintaining the ability to rebuild the system. Image based VMs hold local hypervisor resources in the hypervisor until they are deleted, even when 'powered off'
- There can be some advantages to true block-devices that mapped volumes hide due to their higher level of abstraction - performance tuning, caching, block-layer mirroring, etc. which can be quite handy in certain circumstances but we are unlikely to use.
Disadvantages
- storage system can be single point of failure - similar to a single NFS server. Hypervisors running images are more independent (if they are not backed by the same storage subsystem).
- iscsi network traffic can compete with tcp network traffic depending on the switching fabric, or compete among iscsi clients doing heavy I/O
- Volume budget (#, size, total storage) is consumed by system-disk volumes in addition to pure data volumes
- Creates many more volumes (list at end)
Other
- We have been trying to unify volumes into a single persistent volume per system for logs, docker data, etc. The sys volume could subsume all of these, (besides glfs).
Tests Performed:
- system filesystem performance tests between identical coreos from image and with boot volume.
- File copies, large docker pulls, large docker runs, filesystem backups
- No significant differences in performance – which was somewhat surprising
- provision full ndslabs kubernetes test cluster
- provision was slower than usual, and included failures that are symptomatic of the openstack problems we have been seeing.
- unclear whether problem was volumes or other
- subsequent provision seemed more typical.
- No difference in performance was observed, sometimes deploying pods seemed faster, sometimes slower, but no exhaustive testing was done.
- openstack did not have problems after the provision - it is unclear whether the backing volumes immunize the systems from the gluster problems so far.
List of volumes after small provision:
ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+----------------+-----------------------+------+-------------+----------+--------------------------------------+
| 2910394d-f9f2-4ada-b5b7-acd5c365586a | in-use | dr-gfs3-glfs | 100 | - | false | 813afac6-94bd-4425-8b3c-599e3f284564 |
| 2aa773d8-1b1a-48d4-ba3a-d78c43039453 | in-use | dr-master1-storagevol | 10 | - | false | fc4e6ee9-46ed-4e34-beb7-7f82369d31f2 |
| 31aee772-0cbb-42ae-99ab-c078f266a5b0 | in-use | dr-lma-storagevol | 10 | - | false | 4daf8620-42cd-41b8-979f-7f9f021e8807 |
| 36c928e4-cb1a-450b-906a-a930f828da1a | in-use | dr-gfs3-storagevol | 10 | - | false | 813afac6-94bd-4425-8b3c-599e3f284564 |
| 37438dc8-31f6-4771-a79e-e1ef00dc8870 | in-use | dr-gfs2-storagevol | 10 | - | false | 7d75d01f-2b49-45bc-8ed0-35c96ad39bcf | | |
| 592b8d01-1233-42da-84d8-5c0eaddae470 | in-use | dr-gfs2-glfs | 100 | - | false | 7d75d01f-2b49-45bc-8ed0-35c96ad39bcf |
| 6355de6d-5269-4ef6-aa85-72dae846b82c | in-use | - | 50 | - | true | 02d6ab3d-9e5b-412d-9b57-cdd4be8b2e9e |
| 6a8e6110-ca04-4e3e-9509-eb5ce1a193a6 | in-use | - | 50 | - | true | 813afac6-94bd-4425-8b3c-599e3f284564 |
| 6beb71b1-8bed-44db-9fb6-2257b2ff3054 | in-use | - | 50 | - | true | fc4e6ee9-46ed-4e34-beb7-7f82369d31f2 |
| 75735f0c-b476-4030-b98c-c7789ae9f8e6 | in-use | dr-node1-dockervol | 40 | - | false | 72dbea82-875f-4cb9-9502-64d2c7e0b050 |
| 7e8660f5-bb20-4ef1-9b28-47d3a60ff7b0 | in-use | dr-gfs4-glfs | 100 | - | false | 0c5c314d-9ddc-474a-9d5a-f100e98efbb0 |
| 91ed17fa-69d2-4c98-b43c-d281742117c9 | in-use | dr-gfs1-glfs | 100 | - | false | da7364ef-0933-4364-b2d3-81629452cec7 |
| 9fbe00b0-d1b7-4414-9bca-e068ebc2274b | in-use | - | 50 | - | true | 72dbea82-875f-4cb9-9502-64d2c7e0b050 | |
| acee8491-ae28-450c-8347-11b4c1a5e1c5 | in-use | dr-loadbal-storagevol | 10 | - | false | 02d6ab3d-9e5b-412d-9b57-cdd4be8b2e9e |
| c030e2ee-e9d3-48da-a4e7-e8b42d1e33a1 | in-use | dr-gfs4-storagevol | 10 | - | false | 0c5c314d-9ddc-474a-9d5a-f100e98efbb0 |
| c4dac9f6-71c7-4b17-85d0-f41c7f2cd94b | in-use | dr-node1-storagevol | 10 | - | false | 72dbea82-875f-4cb9-9502-64d2c7e0b050 |
| c4f71f4b-9a19-41d1-80a8-1aace138e975 | in-use | - | 50 | - | true | 7d75d01f-2b49-45bc-8ed0-35c96ad39bcf | |
| ec17f70e-a612-4c66-b2b5-832fb4dc4425 | in-use | - | 50 | - | true | 4daf8620-42cd-41b8-979f-7f9f021e8807 |
| ecb1a5af-fdd8-40a3-8f21-9bb49f5027f4 | in-use | - | 50 | - | true | 0c5c314d-9ddc-474a-9d5a-f100e98efbb0 |
| f265f121-f8c0-44de-a2c9-c9915a0766d5 | in-use | - | 50 | - | true | da7364ef-0933-4364-b2d3-81629452cec7 |
| f381725f-79d7-4d3f-8496-51705e87bfba | in-use | dr-gfs1-storagevol | 10 | - | false | da7364ef-0933-4364-b2d3-81629452cec7 |