...
As you can see, these metadata files do not appear to be readable on disk and would likely need to be un-mangled by Rook to properly perform a backup.
Checking the kubelet logs...
Digging into the systemctl logs for kubectl, we can see it's complaining about the volume configuration:
...
Sadly, even setting this value explicitly did not fix my immediate issue.
Hacking Terraform
At this point, I decided to start hacking the Terraform deployment to get Rook working to the level we'll need for Workbench.
Out of the box, it seems to have everything we would need for Block storage, but this does not give us the ReadWriteMany capabilities that Workbench will require.
As a lesson from GlusterFS, the first stage in debugging a broken deployment process for a shared filesystem is to upgrade to the newest version as soon as humanly possible.
Instead of pulling directly from Zonca's GitHub, I opted to hard-code the entire YAML into the assets/deploy-rook.sh script for testing purposes:
| Code Block | ||
|---|---|---|
| ||
#!/bin/bash
git clone https://github.com/groundnuty/k8s-wait-for.git
sudo helm repo add rook-alpha https://charts.rook.io/alpha
sudo helm install rook-alpha/rook --name rook --version 0.7.1
... ... ... ...
# Now we can install the cluster
echo '
apiVersion: v1
kind: Namespace
metadata:
name: rook
---
apiVersion: rook.io/v1alpha1
kind: Cluster
metadata:
name: rook
namespace: rook
spec:
versionTag: v0.7.1
dataDirHostPath: /var/lib/rook
# cluster level storage configuration and selection
storage:
useAllNodes: false
useAllDevices: false
deviceFilter:
metadataDevice:
location:
storeConfig:
storeType: bluestore
databaseSizeMB: 1024 # this value can be removed for environments with normal sized disks (100 GB or larger)
journalSizeMB: 1024 # this value can be removed for environments with normal sized disks (20 GB or larger)
nodes:
- name: "mldev-storage0"
directories: # specific directories to use for storage can be specified for each node
- path: "/vol_b"
- name: "mldev-storage1"
directories: # specific directories to use for storage can be specified for each node
- path: "/vol_b"
' | kubectl create -f -
# And create a storage class
wget -q https://raw.githubusercontent.com/zonca/jupyterhub-deploy-kubernetes-jetstream/master/storage_rook/rook-storageclass.yaml
sudo kubectl create -f rook-storageclass.yaml
|
The embedded rook-cluster.yaml above differs from Zonca's original template in the following ways:
- Rook has been upgraded from v0.6.2 to v0.7.1, in the helm install and in rook-cluster.yaml
- Expanded
storagesection to include anodessubsection - this specifies which machines / directories should be part of the storage cluster - Turn off
useAllNodes
Checking the rook-operator logs...
Now, with the new version of rook up and running, I attempted to make a filesystem as before. This time, however, no pods were spawned following my filesystem's creation.
In the rook-operator logs, I noticed a pretty gnarly-sounding error message:
| Code Block | ||
|---|---|---|
| ||
2018-05-23 17:30:50.277799 W | op-osd: failed to create osd replica set for node mldev-storage0. ReplicaSet.apps "rook-ceph-osd-mldev-storage0" is invalid: [spec.template.spec.volumes[2].name: Invalid value: "vol_b": a DNS-1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name', or '123-abc', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?'), spec.template.spec.containers[0].volumeMounts[2].name: Not found: "vol_b"]
2018-05-23 17:30:50.941464 W | op-osd: failed to create osd replica set for node mldev-storage1. ReplicaSet.apps "rook-ceph-osd-mldev-storage1" is invalid: [spec.template.spec.volumes[2].name: Invalid value: "vol_b": a DNS-1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name', or '123-abc', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?'), spec.template.spec.containers[0].volumeMounts[2].name: Not found: "vol_b"]
2018-05-23 17:30:51.171199 I | exec: Running command: ceph osd unset noscrub --cluster=rook --conf=/var/lib/rook/rook/rook.config --keyring=/var/lib/rook/rook/client.admin.keyring --format json --out-file /tmp/479433030
2018-05-23 17:30:58.450482 I | exec: noscrub is unset
2018-05-23 17:30:58.450617 I | exec: Running command: ceph osd unset nodeep-scrub --cluster=rook --conf=/var/lib/rook/rook/rook.config --keyring=/var/lib/rook/rook/client.admin.keyring --format json --out-file /tmp/253765613
2018-05-23 17:31:06.351987 I | exec: nodeep-scrub is unset
2018-05-23 17:31:06.352146 E | op-cluster: failed to create cluster in namespace rook. failed to start the osds. 2 failures encountered while running osds in namespace rook: failed to create osd replica set for node mldev-storage0. ReplicaSet.apps "rook-ceph-osd-mldev-storage0" is invalid: [spec.template.spec.volumes[2].name: Invalid value: "vol_b": a DNS-1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name', or '123-abc', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?'), spec.template.spec.containers[0].volumeMounts[2].name: Not found: "vol_b"]
failed to create osd replica set for node mldev-storage1. ReplicaSet.apps "rook-ceph-osd-mldev-storage1" is invalid: [spec.template.spec.volumes[2].name: Invalid value: "vol_b": a DNS-1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name', or '123-abc', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?'), spec.template.spec.containers[0].volumeMounts[2].name: Not found: "vol_b"]
2018-05-23 17:31:06.352173 E | op-cluster: giving up creating cluster in namespace rook after 5m0s
|
Since Zonca's method would deploy JupyterHub (which doesn't have the same ReadWriteMany volume-mounting requirements), it is possible that the path: /vol_b identifier worked fine in the case of block storage, but for a Rook filesystem there are apparently other considerations.
Namely, in this case the path is somehow coerced into a volumeMount's name, which means that our path can't contain an underscore.
Changing /vol_b to /volb solved this problem - this must be adjusted both in the deploy-rook.sh script above, as well as the bootstrap-rook.sh script alongside of it.
Now we're getting somewhere...
After changing the volume path and redeploying (again), now myfs pods were being spawned after creating the filesystem in Kubernetes, as they should be:
| Code Block | ||
|---|---|---|
| ||
ubuntu@mldev-master:~$ kubectl get pods -n rook
NAME READY STATUS RESTARTS AGE
rook-api-6fc75cd555-rwcr9 1/1 Running 0 1h
rook-ceph-mds-myfs-6846dc79fb-5hmzx 1/1 Running 0 1h
rook-ceph-mds-myfs-6846dc79fb-jmdr9 1/1 Running 0 1h
rook-ceph-mgr0-5778b8b46b-2m4js 1/1 Running 0 1h
rook-ceph-mon0-bgzh7 1/1 Running 0 1h
rook-ceph-mon1-8bd9x 1/1 Running 0 1h
rook-ceph-mon2-cfq4f 1/1 Running 0 1h
rook-ceph-osd-mldev-storage0-pb6vs 1/1 Running 0 1h
rook-ceph-osd-mldev-storage1-62brv 1/1 Running 0 1h
|
We can also see that our data is now being stored in the correct place on /volb on storage0, instead of inside of in /var/lib/rook:
| Code Block | ||
|---|---|---|
| ||
ubuntu@mldev-storage0:~$ sudo ls -al /volb/osd0
total 2316
drwxr--r-- 3 root root 4096 May 23 19:43 .
drwxr-xr-x 4 root root 4096 May 23 19:43 ..
lrwxrwxrwx 1 root root 26 May 23 19:43 block -> /volb/osd0/bluestore-block
lrwxrwxrwx 1 root root 23 May 23 19:43 block.db -> /volb/osd0/bluestore-db
lrwxrwxrwx 1 root root 24 May 23 19:43 block.wal -> /volb/osd0/bluestore-wal
-rw-r--r-- 1 root root 2 May 23 19:43 bluefs
-rw-r--r-- 1 root root 79132893184 May 23 19:43 bluestore-block
-rw-r--r-- 1 root root 1073741824 May 23 19:43 bluestore-db
-rw-r--r-- 1 root root 603979776 May 23 20:21 bluestore-wal
-rw-r--r-- 1 root root 37 May 23 19:43 ceph_fsid
-rw-r--r-- 1 root root 37 May 23 19:43 fsid
-rw-r--r-- 1 root root 56 May 23 19:43 keyring
-rw-r--r-- 1 root root 8 May 23 19:43 kv_backend
-rw-r--r-- 1 root root 21 May 23 19:43 magic
-rw-r--r-- 1 root root 4 May 23 19:43 mkfs_done
-rw-r--r-- 1 root root 24 May 23 19:43 path_block.db
-rw-r--r-- 1 root root 25 May 23 19:43 path_block.wal
-rw-r--r-- 1 root root 6 May 23 19:43 ready
-rw-r--r-- 1 root root 1689 May 23 19:43 rook.config
srwxr-xr-x 1 root root 0 May 23 19:43 rook-osd.0.asok
drwxr--r-- 2 root root 4096 May 23 19:43 tmp
-rw-r--r-- 1 root root 10 May 23 19:43 type
-rw-r--r-- 1 root root 2 May 23 19:43 whoami |
Unfortunately my test case was still failing, as writing even the smallest amount of data to the shared filesystem would cause the container to lock up:
| Code Block | ||
|---|---|---|
| ||
ubuntu@mldev-master:~$ kubectl exec -it deployment-demo-c59b896c8-2llsl -c busybox -- sh
/ # ls -al /data/
total 4
drwxr-xr-x 1 root root 0 May 23 19:56 .
drwxr-xr-x 1 root root 4096 May 23 19:54 ..
-rw-r--r-- 1 root root 0 May 23 19:54 index.html
-rw-r--r-- 1 root root 0 May 23 19:56 testing.html
/ # cat /data/testing.html
/ # echo '<div>Hello, World!</div>' > /data/testing.html
^C^C^C
^C
^C
^C
^C
^C^C^C^C
^C
^C^C^C^C^C
rm -rf /data/^C^C^C
^C
^C^C
rm -rf /data/testing.html
^C^C^C^C
^C
^C
^C^C^C^Cubuntu@mldev-master:~$
|
Fortunately, kubectl describe pod gave me some kind of lead:
| Code Block | ||
|---|---|---|
| ||
ubuntu@mldev-master:~$ kubectl describe pod deployment-demo-c59b896c8-2llsl
Name: deployment-demo-c59b896c8-2llsl
Namespace: default
Node: mldev-storage0/192.168.0.3
Start Time: Wed, 23 May 2018 19:51:41 +0000
Labels: demo=deployment
pod-template-hash=715645274
version=v1
... ... ... ...Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 37m default-scheduler Successfully assigned deployment-demo-c59b896c8-2llsl to mldev-storage0
Normal SuccessfulMountVolume 37m kubelet, mldev-storage0 MountVolume.SetUp succeeded for volume "default-token-hx8hk"
Warning FailedMount 35m kubelet, mldev-storage0 Unable to mount volumes for pod "deployment-demo-c59b896c8-2llsl_default(b602a2dc-5ec2-11e8-92e6-fa163e8f23f7)": timeout expired waiting for volumes to attach/mount for pod "default"/"deployment-demo-c59b896c8-2llsl". list of unattached/unmounted volumes=[content]
Warning FailedMount 35m kubelet, mldev-storage0 Unable to mount volumes for pod "deployment-demo-c59b896c8-2llsl_default(b602a2dc-5ec2-11e8-92e6-fa163e8f23f7)": timeout expired waiting for volumes to attach/mount for pod "default"/"deployment-demo-c59b896c8-2llsl". list of unattached/unmounted volumes=[content]
Warning FailedMount 35m kubelet, mldev-storage0 MountVolume.SetUp failed for volume "content" : mount command failed, status: Failure, reason: failed to mount filesystem myfs to /var/lib/kubelet/pods/b602a2dc-5ec2-11e8-92e6-fa163e8f23f7/volumes/rook.io~rook/content with monitor 10.99.14.47:6790,10.104.100.195:6790,10.110.166.39:6790:/ and options [name=admin secret=AQBRwgVbTxaqNxAA9tGopJpso/BU8xEQO1jNgQ==]: mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/b602a2dc-5ec2-11e8-92e6-fa163e8f23f7/volumes/rook.io~rook/content --scope -- mount -t ceph -o name=admin,secret=AQBRwgVbTxaqNxAA9tGopJpso/BU8xEQO1jNgQ== 10.99.14.47:6790,10.104.100.195:6790,10.110.166.39:6790:/ /var/lib/kubelet/pods/b602a2dc-5ec2-11e8-92e6-fa163e8f23f7/volumes/rook.io~rook/content
Output: Running scope as unit run-rbddb8b0550494bf5a46ecf0173cdced7.scope.
mount: 10.99.14.47:6790,10.104.100.195:6790,10.110.166.39:6790:/: can't read superblock
|
Bah! What is it now?