...

#!/bin/bash
git clone https://github.com/groundnuty/k8s-wait-for.git

sudo helm repo add rook-alpha https://charts.rook.io/alpha
sudo helm install rook-alpha/rook --name rook --version 0.6.2


   ...    ...    ...    ...


# Now we can install the cluster
echo '
apiVersion: v1
kind: Namespace
metadata:
  name: rook
---
apiVersion: rook.io/v1alpha1
kind: Cluster
metadata:
  name: rook
  namespace: rook
spec:
  versionTag: v0.6.2
  dataDirHostPath: /var/lib/rook
  # cluster level storage configuration and selection
  storage:                
    useAllNodes: false
    useAllDevices: false
    deviceFilter:
    metadataDevice:
    location:
    storeConfig:
      storeType: bluestore
      databaseSizeMB: 1024 # this value can be removed for environments with normal sized disks (100 GB or larger)
      journalSizeMB: 1024  # this value can be removed for environments with normal sized disks (20 GB or larger)
    - name: "mldev-storage0"
      directories:         # specific directories to use for storage can be specified for each node
      - path: "/rook"      # if changed, also adjust the mount path in bootstrap-rook.sh
    - name: "mldev-storage1"
      directories:         # specific directories to use for storage can be specified for each node
      - path: "/rook"      # if changed, also adjust the mount path in bootstrap-rook.sh
' | kubectl create -f -


# And create a storage class
wget -q https://raw.githubusercontent.com/zonca/jupyterhub-deploy-kubernetes-jetstream/master/storage_rook/rook-storageclass.yaml
sudo kubectl create -f rook-storageclass.yaml

Edge Cases and Quirks

There are many pitfalls here, particularly surrounding my perceived fragility of the shared filesystem

DO NOT delete the filesystem before shutting down the pods consuming it

Deleting the shared filesystem out from under the pod will confuse the kubelet, and prevent it from being able to properly unmount and terminate your containers.

If you need to shut down your shared filesystem, please ensure that you first shut down any pods consuming it's storage.

You must follow these cleanup steps before terraform destroy will work

With this configuration, the filesystem pods came up just fine, but this exhibited the same behavior as the case above, where writing to a file causes the container to hang indefinitely.

The only way to recover in this case is to Ctrl+P, Ctrl+Q to abandon the kubectl exec command, Ctrl+C-ing as necessary - this leaves a hanging exec process, but it is still unclear whether there are performance concerns about executing this case repeatedly without rebooting.

I have noticed that cluster with pods in an error state such as this one will fail to terraform destroy (the operation never completes even after waiting 15+ minutes)

Recovering from backup

This feature is currently in the planning stages: https://github.com/rook/rook/issues/1552

Unofficial Python script for creating / restoring backups from Rook: https://gitlab.com/costrouc/kubernetes-rook-backup

Edge Cases and Quirks

There are many pitfalls here, particularly surrounding my perceived fragility of the shared filesystem

DO NOT delete the filesystem before shutting down all of the pods consuming it

Deleting the shared filesystem out from under the pod will confuse the kubelet, and prevent it from being able to properly unmount and terminate your containers.

If you need to shut down your shared filesystem, please ensure that you first shut down any pods consuming it's storage.

You must follow these cleanup steps before terraform destroy will work

Expanding on the Expanding on the above topic, terraform destroy will hang on destroying your cluster if you fail to cleanup your filesystems properly:

...

WARNING: this seems like a very tenuous/tedious process... I am hoping that later versions of terraform/rook will improve the stability of cleanup under these scenarios. Perhaps we can expand their cleanup to first drain all nodes of their running pods (if this is not already the case), although this would not fix the case of a user deleting the filesystem before a running pod that is consuming it - in this case, the pods will fail to terminate indefinitely, which I think is what is leading terraform to fail.

...

Kill (or hide) a hanging pod

There are a few ways to kill a pod with fire:

# Attempt to shorten the grace period
ubuntu@mldev-master:~$ kubectl delete pod deployment-demo-c59b896c8-8hgnm --grace-period=1
pod "deployment-demo-c59b896c8-8hgnm" deleted


# If that doesn't work, you can try to force it
ubuntu@mldev-master:~$ kubectl delete pod deployment-demo-c59b896c8-8hgnm --grace-period=1 --force
pod "deployment-demo-c59b896c8-8hgnm" deleted


# If your pod still doesn't terminate, you can change try it with --now instead:
ubuntu@mldev-master:~$ kubectl delete pod deployment-demo-c59b896c8-8hgnm --now
pod "deployment-demo-c59b896c8-8hgnm" deleted
ubuntu@mldev-master:~$ kubectl delete pod deployment-demo-c59b896c8-8hgnm --now --force
pod "deployment-demo-c59b896c8-8hgnm" deleted


# As a last resort, drop the grace period to zero
# WARNING: Only use this as a last resort, and never use it in production!
ubuntu@mldev-master:~$ kubectl delete pod deployment-demo-c59b896c8-8hgnm --grace-period=0 --force
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "deployment-demo-c59b896c8-8hgnm" deleted

Note that the last method may not actually cleanup all resources properly, as noted in the output above.

Only use this as a last resort on test clusters, and NEVER use --grace-period=0  on a production cluster.

Cleaning up failed runs of terraform destroy

Here is a quick checklist of the items that you will need to manually if you are unable to terraform destroy your cluster:

...