...
Code Block | ||
---|---|---|
| ||
#!/bin/bash git clone https://github.com/groundnuty/k8s-wait-for.git sudo helm repo add rook-alpha https://charts.rook.io/alpha sudo helm install rook-alpha/rook --name rook --version 0.6.2 ... ... ... ... # Now we can install the cluster echo ' apiVersion: v1 kind: Namespace metadata: name: rook --- apiVersion: rook.io/v1alpha1 kind: Cluster metadata: name: rook namespace: rook spec: versionTag: v0.6.2 dataDirHostPath: /var/lib/rook # cluster level storage configuration and selection storage: useAllNodes: false useAllDevices: false deviceFilter: metadataDevice: location: storeConfig: storeType: bluestore databaseSizeMB: 1024 # this value can be removed for environments with normal sized disks (100 GB or larger) journalSizeMB: 1024 # this value can be removed for environments with normal sized disks (20 GB or larger) - name: "mldev-storage0" directories: # specific directories to use for storage can be specified for each node - path: "/rook" # if changed, also adjust the mount path in bootstrap-rook.sh - name: "mldev-storage1" directories: # specific directories to use for storage can be specified for each node - path: "/rook" # if changed, also adjust the mount path in bootstrap-rook.sh ' | kubectl create -f - # And create a storage class wget -q https://raw.githubusercontent.com/zonca/jupyterhub-deploy-kubernetes-jetstream/master/storage_rook/rook-storageclass.yaml sudo kubectl create -f rook-storageclass.yaml |
Edge Cases and Quirks
There are many pitfalls here, particularly surrounding my perceived fragility of the shared filesystem
DO NOT delete the filesystem before shutting down the pods consuming it
Deleting the shared filesystem out from under the pod will confuse the kubelet, and prevent it from being able to properly unmount and terminate your containers.
If you need to shut down your shared filesystem, please ensure that you first shut down any pods consuming it's storage.
You must follow these cleanup steps before terraform destroy
will work
With this configuration, the filesystem pods came up just fine, but this exhibited the same behavior as the case above, where writing to a file causes the container to hang indefinitely.
The only way to recover in this case is to Ctrl+P, Ctrl+Q to abandon the kubectl exec
command, Ctrl+C-ing as necessary - this leaves a hanging exec process, but it is still unclear whether there are performance concerns about executing this case repeatedly without rebooting.
I have noticed that cluster with pods in an error state such as this one will fail to terraform destroy (the operation never completes even after waiting 15+ minutes)
Recovering from backup
This feature is currently in the planning stages: https://github.com/rook/rook/issues/1552
Unofficial Python script for creating / restoring backups from Rook: https://gitlab.com/costrouc/kubernetes-rook-backup
Edge Cases and Quirks
There are many pitfalls here, particularly surrounding my perceived fragility of the shared filesystem
DO NOT delete the filesystem before shutting down all of the pods consuming it
Deleting the shared filesystem out from under the pod will confuse the kubelet, and prevent it from being able to properly unmount and terminate your containers.
If you need to shut down your shared filesystem, please ensure that you first shut down any pods consuming it's storage.
You must follow these cleanup steps before terraform destroy
will work
Expanding on the Expanding on the above topic, terraform destroy
will hang on destroying your cluster if you fail to cleanup your filesystems properly:
...
WARNING: this seems like a very tenuous/tedious process... I am hoping that later versions of terraform/rook will improve the stability of cleanup under these scenarios. Perhaps we can expand their cleanup to first drain all nodes of their running pods (if this is not already the case), although this would not fix the case of a user deleting the filesystem before a running pod that is consuming it - in this case, the pods will fail to terminate indefinitely, which I think is what is leading terraform to fail.
...
Kill (or hide) a hanging pod
There are a few ways to kill a pod with fire:
Code Block | ||
---|---|---|
| ||
# Attempt to shorten the grace period
ubuntu@mldev-master:~$ kubectl delete pod deployment-demo-c59b896c8-8hgnm --grace-period=1
pod "deployment-demo-c59b896c8-8hgnm" deleted
# If that doesn't work, you can try to force it
ubuntu@mldev-master:~$ kubectl delete pod deployment-demo-c59b896c8-8hgnm --grace-period=1 --force
pod "deployment-demo-c59b896c8-8hgnm" deleted
# If your pod still doesn't terminate, you can change try it with --now instead:
ubuntu@mldev-master:~$ kubectl delete pod deployment-demo-c59b896c8-8hgnm --now
pod "deployment-demo-c59b896c8-8hgnm" deleted
ubuntu@mldev-master:~$ kubectl delete pod deployment-demo-c59b896c8-8hgnm --now --force
pod "deployment-demo-c59b896c8-8hgnm" deleted
# As a last resort, drop the grace period to zero
# WARNING: Only use this as a last resort, and never use it in production!
ubuntu@mldev-master:~$ kubectl delete pod deployment-demo-c59b896c8-8hgnm --grace-period=0 --force
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "deployment-demo-c59b896c8-8hgnm" deleted
|
Note that the last method may not actually cleanup all resources properly, as noted in the output above.
Only use this as a last resort on test clusters, and NEVER use --grace-period=0
on a production cluster.
Cleaning up failed runs of terraform destroy
Here is a quick checklist of the items that you will need to manually if you are unable to terraform destroy
your cluster:
...