Code Block

language	bash

$ ceph status                                                                                                                                              
  ...
  services:
    mds: myfs-1/1/1 up {[myfs:0]=mzw58b=up:active}, 1 up:standby-replay

Consume the Shared File System:

...

Busybox + NGINX Example

As an example, we will start the kube-registry pod with the shared file system as the backing store. Save the following spec as kube-registry.yaml:

...

Code Block

language	bash

ubuntu@mldev-master:~$ kubectl describe pod deployment-demo-c59b896c8-2llsl
Name:           deployment-demo-c59b896c8-2llsl
Namespace:      default
Node:           mldev-storage0/192.168.0.3
Start Time:     Wed, 23 May 2018 19:51:41 +0000
Labels:         demo=deployment
                pod-template-hash=715645274
                version=v1

   ...    ...    ...    ...Events:
  Type     Reason                 Age   From                     Message
  ----     ------                 ----  ----                     -------
  Normal   Scheduled              37m   default-scheduler        Successfully assigned deployment-demo-c59b896c8-2llsl to mldev-storage0
  Normal   SuccessfulMountVolume  37m   kubelet, mldev-storage0  MountVolume.SetUp succeeded for volume "default-token-hx8hk"
  Warning  FailedMount            35m   kubelet, mldev-storage0  Unable to mount volumes for pod "deployment-demo-c59b896c8-2llsl_default(b602a2dc-5ec2-11e8-92e6-fa163e8f23f7)": timeout expired waiting for volumes to attach/mount for pod "default"/"deployment-demo-c59b896c8-2llsl". list of unattached/unmounted volumes=[content]
  Warning  FailedMount            35m   kubelet, mldev-storage0  Unable to mount volumes for pod "deployment-demo-c59b896c8-2llsl_default(b602a2dc-5ec2-11e8-92e6-fa163e8f23f7)": timeout expired waiting for volumes to attach/mount for pod "default"/"deployment-demo-c59b896c8-2llsl". list of unattached/unmounted volumes=[content]
  Warning  FailedMount            35m   kubelet, mldev-storage0  MountVolume.SetUp failed for volume "content" : mount command failed, status: Failure, reason: failed to mount filesystem myfs to /var/lib/kubelet/pods/b602a2dc-5ec2-11e8-92e6-fa163e8f23f7/volumes/rook.io~rook/content with monitor 10.99.14.47:6790,10.104.100.195:6790,10.110.166.39:6790:/ and options [name=admin secret=AQBRwgVbTxaqNxAA9tGopJpso/BU8xEQO1jNgQ==]: mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/b602a2dc-5ec2-11e8-92e6-fa163e8f23f7/volumes/rook.io~rook/content --scope -- mount -t ceph -o name=admin,secret=AQBRwgVbTxaqNxAA9tGopJpso/BU8xEQO1jNgQ== 10.99.14.47:6790,10.104.100.195:6790,10.110.166.39:6790:/ /var/lib/kubelet/pods/b602a2dc-5ec2-11e8-92e6-fa163e8f23f7/volumes/rook.io~rook/content
Output: Running scope as unit run-rbddb8b0550494bf5a46ecf0173cdced7.scope.
mount: 10.99.14.47:6790,10.104.100.195:6790,10.110.166.39:6790:/: can't read superblock

Bah! What is it now?

This could be a side-effect of one or more of the following changes:

Upgrading to Rook 0.7.1
The adjustments to the directories configuration in rook-cluster.yaml are now writing data to the correct drive (/volb), but that drive may be improperly formatted for use with Rook

Narrowing it down...

Adjusting my rook-cluster.yaml to only include the adjustments to the directories configuration, and to use storeType: filestore instead of bluestore.

This will hopefully prevent Rook from mangling our files on disk, and simplify our backups process:

Code Block

language	bash

#!/bin/bash
git clone https://github.com/groundnuty/k8s-wait-for.git

sudo helm repo add rook-alpha https://charts.rook.io/alpha
sudo helm install rook-alpha/rook --name rook --version 0.6.2


   ...    ...    ...    ...


# Now we can install the cluster
echo '
apiVersion: v1
kind: Namespace
metadata:
  name: rook
---
apiVersion: rook.io/v1alpha1
kind: Cluster
metadata:
  name: rook
  namespace: rook
spec:
  versionTag: v0.6.2
  dataDirHostPath: /var/lib/rook
  # cluster level storage configuration and selection
  storage:                
    useAllNodes: false
    useAllDevices: false
    deviceFilter:
    metadataDevice:
    location:
    storeConfig:
      storeType: filestore
    nodes:
    - name: "mldev-storage0"
      directories:         # specific directories to use for storage can be specified for each node
      - path: "/rook"      # if changed, also adjust the mount path in bootstrap-rook.sh
    - name: "mldev-storage1"
      directories:         # specific directories to use for storage can be specified for each node
      - path: "/rook"      # if changed, also adjust the mount path in bootstrap-rook.sh
' | kubectl create -f -


# And create a storage class
wget -q https://raw.githubusercontent.com/zonca/jupyterhub-deploy-kubernetes-jetstream/master/storage_rook/rook-storageclass.yaml
sudo kubectl create -f rook-storageclass.yaml

Sadly, changing to filestore did not unmangle the files:

Code Block

language	bash

ubuntu@mldev-storage0:~$ sudo ls -al /rook/osd1/
total 5242956
drwxr--r--   4 root root       4096 May 24 16:03 .
drwxr-xr-x   4 root root       4096 May 24 16:03 ..
-rw-r--r--   1 root root         37 May 24 16:03 ceph_fsid
drwxr-xr-x 508 root root      20480 May 24 16:11 current
-rw-r--r--   1 root root         37 May 24 16:03 fsid
-rw-r--r--   1 root root 5368709120 May 24 16:12 journal
-rw-r--r--   1 root root         56 May 24 16:03 keyring
-rw-r--r--   1 root root         21 May 24 16:03 magic
-rw-r--r--   1 root root          6 May 24 16:03 ready
-rw-r--r--   1 root root       1283 May 24 16:03 rook.config
srwxr-xr-x   1 root root          0 May 24 16:03 rook-osd.1.asok
-rw-r--r--   1 root root          4 May 24 16:03 store_version
-rw-r--r--   1 root root         53 May 24 16:03 superblock
drwxr--r--   2 root root       4096 May 24 16:03 tmp
-rw-r--r--   1 root root         10 May 24 16:03 type
-rw-r--r--   1 root root          2 May 24 16:03 whoami
ubuntu@mldev-storage0:~$ sudo ls -al /rook/osd1/current
total 4080
drwxr-xr-x 508 root root 20480 May 24 16:11 .
drwxr--r--   4 root root  4096 May 24 16:03 ..
drwxr-xr-x   2 root root  4096 May 24 16:04 1.0_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.10_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.10_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.11_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.11_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.12_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.12_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.13_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.13_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.14_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.14_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.15_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.15_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.16_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.16_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.17_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.17_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.18_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.18_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.19_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.19_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.1a_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.1a_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.1b_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.1b_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.1c_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.1c_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.1d_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.1d_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.1e_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.1e_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.1f_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.1f_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.1_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.1_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.20_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.20_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.21_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.21_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.22_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.22_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.23_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.23_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.24_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.24_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.25_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.25_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.26_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.26_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.27_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.27_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.28_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.28_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.29_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.29_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.2a_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.2a_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.2b_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.2b_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.2c_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.2c_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.2d_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.2d_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.2e_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.2e_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.2f_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.2f_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.2_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.2_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.30_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.30_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.31_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.31_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.32_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.32_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.33_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.33_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.34_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.34_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.35_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.35_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.36_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.36_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.37_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.37_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.38_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.38_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.39_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.39_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.3a_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.3a_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3b_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3b_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3c_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3c_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3d_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3d_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3e_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3e_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3f_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3f_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.40_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.40_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.41_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.41_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.42_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.42_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.43_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.43_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.44_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.44_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.45_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.45_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.46_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.46_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.47_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.47_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.48_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.48_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.49_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.49_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.4a_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.4a_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.4b_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.4b_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.4c_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.4c_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.4d_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.4d_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.4e_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.4e_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.4f_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.4f_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.4_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.4_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.50_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.50_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.51_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.51_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.52_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.52_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.53_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.53_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.54_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.54_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.55_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.55_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.56_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.56_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:05 1.57_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.57_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:05 1.58_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.58_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.59_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.59_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.5a_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.5a_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.5b_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.5b_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:05 1.5c_head
drwxr-xr-x   2 root root  4096 May 24 16:05 1.5c_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:05 1.5d_head
drwxr-xr-x   2 root root  4096 May 24 16:05 1.5d_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.5e_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.5e_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.5f_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.5f_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.5_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.5_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.60_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.60_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:05 1.61_head
drwxr-xr-x   2 root root  4096 May 24 16:05 1.61_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.62_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.62_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:05 1.63_head
drwxr-xr-x   2 root root  4096 May 24 16:05 1.63_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.6_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.6_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.7_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.7_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.8_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.8_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.9_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.9_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.a_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.a_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.b_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.b_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.c_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.c_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.d_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.d_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.e_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.e_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.f_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.f_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.0_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.10_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.10_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.11_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.11_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.12_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.12_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 2.13_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.13_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.14_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.14_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.15_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.15_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 2.16_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.16_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.17_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.17_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 2.18_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.18_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.19_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.19_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.1a_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.1a_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.1b_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.1b_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.1c_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.1c_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.1d_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.1d_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.1e_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.1e_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.1f_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.1f_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.1_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.1_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.20_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.20_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.21_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.21_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.22_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.22_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.23_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.23_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.24_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.24_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.25_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.25_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.26_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.26_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.27_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.27_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.28_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.28_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.29_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.29_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.2a_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.2a_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.2b_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.2b_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.2c_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.2c_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.2d_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.2d_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.2e_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.2e_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.2f_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.2f_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.2_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.2_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.30_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.30_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.31_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.31_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.32_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.32_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.33_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.33_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.34_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.34_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.35_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.35_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.36_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.36_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.37_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.37_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.38_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.38_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.39_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.39_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.3a_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.3a_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.3b_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.3b_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.3c_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.3c_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.3d_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.3d_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.3e_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.3e_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.3f_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.3f_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.3_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.3_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.40_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.40_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.41_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.41_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.42_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.42_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.43_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.43_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.44_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.44_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.45_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.45_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.46_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.46_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.47_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.47_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.48_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.48_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.49_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.49_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.4a_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.4a_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.4b_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.4b_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.4c_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.4c_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.4d_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.4d_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.4e_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.4e_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.4f_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.4f_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 2.4_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.4_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.50_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.50_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.51_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.51_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.52_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.52_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.53_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.53_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.54_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.54_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.55_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.55_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.56_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.56_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.57_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.57_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.58_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.58_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.59_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.59_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.5a_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.5a_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.5b_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.5b_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.5c_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.5c_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.5d_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.5d_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.5e_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.5e_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.5f_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.5f_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 2.5_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.5_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.60_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.60_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.61_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.61_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.62_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.62_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.63_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.63_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 2.6_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.6_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.7_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.7_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.8_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.8_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 2.9_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.9_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 2.a_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.a_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.b_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.b_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 2.c_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.c_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 2.d_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.d_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 2.e_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.e_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.f_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.f_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.0s0_head
drwxr-xr-x   2 root root  4096 May 24 16:08 3.0s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.12s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.12s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.14s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.14s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.18s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.18s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.19s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.19s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.1as0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.1as0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.1bs0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.1bs0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.1cs0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.1cs0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.1es0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.1es0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.1fs0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.1fs0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.1s0_head
drwxr-xr-x   2 root root  4096 May 24 16:08 3.1s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.20s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.20s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.21s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.21s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.22s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.22s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.24s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.24s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.27s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.27s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.28s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.28s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.29s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.29s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.2ds0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.2ds0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.2es0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.2es0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.30s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.30s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.32s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.32s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.38s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.38s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.39s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.39s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.3es0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.3es0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.3s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.3s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.40s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.40s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.41s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.41s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.42s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.42s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.45s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.45s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.46s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.46s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.47s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.47s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.49s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.49s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.4es0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.4es0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.4s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.4s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.50s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.50s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.51s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.51s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.52s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.52s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.55s0_head
drwxr-xr-x   2 root root  4096 May 24 16:10 3.55s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.56s0_head
drwxr-xr-x   2 root root  4096 May 24 16:10 3.56s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.57s0_head
drwxr-xr-x   2 root root  4096 May 24 16:10 3.57s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.59s0_head
drwxr-xr-x   2 root root  4096 May 24 16:10 3.59s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.5as0_head
drwxr-xr-x   2 root root  4096 May 24 16:10 3.5as0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.5ds0_head
drwxr-xr-x   2 root root  4096 May 24 16:10 3.5ds0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.5es0_head
drwxr-xr-x   2 root root  4096 May 24 16:10 3.5es0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.5fs0_head
drwxr-xr-x   2 root root  4096 May 24 16:10 3.5fs0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.62s0_head
drwxr-xr-x   2 root root  4096 May 24 16:10 3.62s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.63s0_head
drwxr-xr-x   2 root root  4096 May 24 16:10 3.63s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.6s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.6s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.7s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.7s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.bs0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.bs0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.es0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.es0_TEMP
-rw-r--r--   1 root root     4 May 24 16:12 commit_op_seq
drwxr-xr-x   2 root root 12288 May 24 16:12 meta
-rw-r--r--   1 root root     0 May 24 16:03 nosnap
drwxr-xr-x   2 root root  4096 May 24 16:03 omap

But it did actively prevent me from using a shared filesystem:

Code Block

language	bash

2018-05-24 16:08:39.653246 I | exec: Error EINVAL: pool must only be stored on bluestore for scrubbing to work: osd.1 uses filestore
2018-05-24 16:08:39.653337 W | cephmds: failed to set ec pool property. failed to set pool property allow_ec_overwrites on pool myfs-data0, exit status 22
2018-05-24 16:08:39.653422 I | exec: Running command: ceph fs flag set enable_multiple true --yes-i-really-mean-it --cluster=rook --conf=/var/lib/rook/rook/rook.config --keyring=/var/lib/rook/rook/client.admin.keyring --format json --out-file /tmp/566066704
2018-05-24 16:08:43.644068 I | exec: Running command: ceph fs new myfs myfs-metadata myfs-data0 --cluster=rook --conf=/var/lib/rook/rook/rook.config --keyring=/var/lib/rook/rook/client.admin.keyring --format json --out-file /tmp/808516655
2018-05-24 16:08:45.443925 I | exec: Error EINVAL: pool 'myfs-data0' (id '3') is an erasure-coded pool, with no overwrite support
2018-05-24 16:08:45.444031 E | op-mds: failed to create file system myfs. failed to create file system myfs: failed enabling ceph fs myfs. exit status 22

Confirmed by this GitHub issue: https://github.com/rook/rook/issues/1604

Back to bluestore...

Switching back to storeType: bluestore on Rook v0.6.2 with the correct nodes/directories configuration:

Code Block

language	bash

#!/bin/bash
git clone https://github.com/groundnuty/k8s-wait-for.git

sudo helm repo add rook-alpha https://charts.rook.io/alpha
sudo helm install rook-alpha/rook --name rook --version 0.6.2


   ...    ...    ...    ...


# Now we can install the cluster
echo '
apiVersion: v1
kind: Namespace
metadata:
  name: rook
---
apiVersion: rook.io/v1alpha1
kind: Cluster
metadata:
  name: rook
  namespace: rook
spec:
  versionTag: v0.6.2
  dataDirHostPath: /var/lib/rook
  # cluster level storage configuration and selection
  storage:                
    useAllNodes: false
    useAllDevices: false
    deviceFilter:
    metadataDevice:
    location:
    storeConfig:
      storeType: bluestore
      databaseSizeMB: 1024 # this value can be removed for environments with normal sized disks (100 GB or larger)
      journalSizeMB: 1024  # this value can be removed for environments with normal sized disks (20 GB or larger)
    - name: "mldev-storage0"
      directories:         # specific directories to use for storage can be specified for each node
      - path: "/rook"      # if changed, also adjust the mount path in bootstrap-rook.sh
    - name: "mldev-storage1"
      directories:         # specific directories to use for storage can be specified for each node
      - path: "/rook"      # if changed, also adjust the mount path in bootstrap-rook.sh
' | kubectl create -f -


# And create a storage class
wget -q https://raw.githubusercontent.com/zonca/jupyterhub-deploy-kubernetes-jetstream/master/storage_rook/rook-storageclass.yaml
sudo kubectl create -f rook-storageclass.yaml

Edge Cases and Quirks

There are many pitfalls here, particularly surrounding my perceived fragility of the shared filesystem

DO NOT delete the filesystem before shutting down the pods consuming it

Deleting the shared filesystem out from under the pod will confuse the kubelet, and prevent it from being able to properly unmount and terminate your containers.

If you need to shut down your shared filesystem, please ensure that you first shut down any pods consuming it's storage.

You must follow these cleanup steps before `terraform destroy` will work

Expanding on the above topic, terraform destroy will hang on destroying your cluster if you fail to cleanup your filesystems properly:

Shut down any pods that are consuming the shared filesystem (e.g. any clients)
Shut down the filesystem pods (e.g. any servers)
Run terraform destroy to cleanup all other cluster resources

WARNING: this seems like a very tenuous/tedious process... I am hoping that later versions of terraform/rook will improve the stability of cleanup under these scenarios. Perhaps we can expand their cleanup to first drain all nodes of their running pods (if this is not already the case), although this would not fix the case of a user deleting the filesystem before a running pod that is consuming it - in this case, the pods will fail to terminate indefinitely, which I think is what is leading terraform to fail.

Cleaning up failed terraform deployments

Here is a quick checklist of the items that you will need to manually if you are unable to terraform destroy your cluster:

Under Compute: Instances, then Volumes
Under Compute → Access & Security: Security Groups, then Keypair
Under Network: Router, then Network

You will then need to manually delete your terraform state:

rm -rf terraform.tfstate*

Page tree

Versions Compared

Old Version 9

New Version 10

Key

Consume the Shared File System:

Busybox + NGINX Example

Narrowing it down...

Back to bluestore...

Edge Cases and Quirks

DO NOT delete the filesystem before shutting down the pods consuming it

You must follow these cleanup steps before `terraform destroy` will work

Cleaning up failed terraform deployments

Page tree

Page History

Versions Compared

Old Version 9

New Version 10

Key

Consume the Shared File System:

Busybox + NGINX Example

Narrowing it down...

Back to bluestore...

Edge Cases and Quirks

DO NOT delete the filesystem before shutting down the pods consuming it

You must follow these cleanup steps before terraform destroy will work

Cleaning up failed terraform deployments

You must follow these cleanup steps before `terraform destroy` will work