...
| Code Block | ||
|---|---|---|
| ||
root@mldev-master:/var/lib/rook# ls -al total 20 drwxr-xr-x 5 root root 4096 May 22 17:40 . drwxr-xr-x 46 root root 4096 May 22 17:38 .. drwxr--r-- 3 root root 4096 May 22 17:41 osd1 drwxr--r-- 2 root root 4096 May 22 17:40 rook drwxr--r-- 3 root root 4096 May 22 17:38 rook-ceph-mon0 root@mldev-master:/var/lib/rook# ls -al rook total 16 drwxr--r-- 2 root root 4096 May 22 17:40 . drwxr-xr-x 5 root root 4096 May 22 17:40 .. -rw-r--r-- 1 root root 168 May 22 17:40 client.admin.keyring -rw-r--r-- 1 root root 1123 May 22 17:40 rook.config root@mldev-master:/var/lib/rook# ls -al rook-ceph-mon0/ total 24 drwxr--r-- 3 root root 4096 May 22 17:38 . drwxr-xr-x 5 root root 4096 May 22 17:40 .. drwxr--r-- 3 root root 4096 May 22 17:38 data -rw-r--r-- 1 root root 248 May 22 17:38 keyring -rw-r--r-- 1 root root 210 May 22 17:38 monmap -rw-r--r-- 1 root root 1072 May 22 17:38 rook.config srwxr-xr-x 1 root root 0 May 22 17:38 rook-mon.rook-ceph-mon0.asok root@mldev-master:/var/lib/rook# ls -al osd1 total 7896 drwxr--r-- 3 root root 4096 May 22 17:41 . drwxr-xr-x 5 root root 4096 May 22 17:40 .. lrwxrwxrwx 1 root root 34 May 22 17:40 block -> /var/lib/rook/osd1/bluestore-block lrwxrwxrwx 1 root root 31 May 22 17:40 block.db -> /var/lib/rook/osd1/bluestore-db lrwxrwxrwx 1 root root 32 May 22 17:40 block.wal -> /var/lib/rook/osd1/bluestore-wal -rw-r--r-- 1 root root 2 May 22 17:40 bluefs -rw-r--r-- 1 root root 74883726950 May 22 22:13 bluestore-block -rw-r--r-- 1 root root 1073741824 May 22 17:41 bluestore-db -rw-r--r-- 1 root root 603979776 May 22 22:14 bluestore-wal -rw-r--r-- 1 root root 37 May 22 17:41 ceph_fsid -rw-r--r-- 1 root root 37 May 22 17:40 fsid -rw-r--r-- 1 root root 56 May 22 17:40 keyring -rw-r--r-- 1 root root 8 May 22 17:40 kv_backend -rw-r--r-- 1 root root 21 May 22 17:41 magic -rw-r--r-- 1 root root 4 May 22 17:40 mkfs_done -rw-r--r-- 1 root root 6 May 22 17:41 ready -rw-r--r-- 1 root root 1693 May 22 17:40 rook.config srwxr-xr-x 1 root root 0 May 22 17:41 rook-osd.1.asok drwxr--r-- 2 root root 4096 May 22 17:40 tmp -rw-r--r-- 1 root root 10 May 22 17:40 type -rw-r--r-- 1 root root 2 May 22 17:41 whoami |
As you can see, the configuration these metadata files do not appear to be readable on disk and would likely need to be un-mangled by Rook to properly perform a backup.
Checking the kubelet logs...
Digging into the systemctl logs for kubectl, we can see it's complaining about the volume configuration:
| Code Block | ||
|---|---|---|
| ||
ubuntu@mldev-master:~$ ssh ubuntu@192.168.0.4
ubuntu@mldev-worker0:~$ sudo systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Tue 2018-05-22 17:36:36 UTC; 21h ago
Docs: http://kubernetes.io/docs/
Main PID: 4607 (kubelet)
Tasks: 18
Memory: 50.8M
CPU: 27min 25.718s
CGroup: /system.slice/kubelet.service
└─4607 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin
May 23 15:00:57 mldev-worker0 kubelet[4607]: E0523 15:00:57.308626 4607 reconciler.go:376] Could not construct volume information: no volume plugin matched
May 23 15:03:57 mldev-worker0 kubelet[4607]: E0523 15:03:57.344832 4607 reconciler.go:376] Could not construct volume information: no volume plugin matched
May 23 15:06:57 mldev-worker0 kubelet[4607]: E0523 15:06:57.376002 4607 reconciler.go:376] Could not construct volume information: no volume plugin matched
May 23 15:09:57 mldev-worker0 kubelet[4607]: E0523 15:09:57.424020 4607 reconciler.go:376] Could not construct volume information: no volume plugin matched
May 23 15:12:57 mldev-worker0 kubelet[4607]: E0523 15:12:57.476398 4607 reconciler.go:376] Could not construct volume information: no volume plugin matched
May 23 15:15:57 mldev-worker0 kubelet[4607]: E0523 15:15:57.511746 4607 reconciler.go:376] Could not construct volume information: no volume plugin matched
May 23 15:18:57 mldev-worker0 kubelet[4607]: E0523 15:18:57.545539 4607 reconciler.go:376] Could not construct volume information: no volume plugin matched
May 23 15:21:57 mldev-worker0 kubelet[4607]: E0523 15:21:57.602083 4607 reconciler.go:376] Could not construct volume information: no volume plugin matched
May 23 15:24:57 mldev-worker0 kubelet[4607]: E0523 15:24:57.660774 4607 reconciler.go:376] Could not construct volume information: no volume plugin matched
May 23 15:27:57 mldev-worker0 kubelet[4607]: E0523 15:27:57.717357 4607 reconciler.go:376] Could not construct volume information: no volume plugin matched
|
This is likely a reference to the Flex Volume Configuration steps that I breezed over, assuming they had been handled already by Terraform.
Examining our kubelet configuration on the worker node, we see that there is no reference to the --volume-plugin-dir mentioned in the Rook prerequisites:
| Code Block | ||
|---|---|---|
| ||
root@mldev-worker0:/etc/systemd/system/kubelet.service.d# cat 10-kubeadm.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS |
The following GitHub issues appear to confirm that the error message above seems to stem from a misconfigured --volume-plugin-dir: