The purpose of this document is to familiarize you with running GlusterFS under Kubernetes.
GlusterFS is an open-source distributed filesystem that allows for PVCs that support ReadWriteMany.
Overview
Running GlusterFS in Kubernetes with PVC support is easier than ever with the GlusterFS Simple Provisioner!
Prerequisites
- 2+ node Kubernetes cluster
- No PVC support currently installed
The Long Way
The external-storage repo gives instructions for bringing this all up by hand.
First Steps
First, you'll need to clone the external-storage repo from the kubernetes-incubator:
$ git clone https://github.com/kubernetes-incubator/external-storage && cd external-storage
Locate the gluster/glusterfs subdirectory, which contains these same instructions on getting things up and running:
$ cd gluster/glusterfs/
Apply the correct node label to each of your storage nodes:
$ kubectl label nodes <storage-node-name> storagenode=glusterfs node/<storage-node-name> labeled
Start GlusterFS
Bring up the GlusterFS DaemonSet and wait for them to come online:
$ kubectl create -f deploy/glusterfs-daemonset.yaml daemonset.extensions/glusterfs created $ kubectl get pods -l glusterfs-node=pod --watch
Locate your pod IPs once they are online:
$ kubectl get pods -o wide | grep glusterfs | grep -v provisioner NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE glusterfs-t44m5 1/1 Running 0 4h 192.168.0.9 nfstest-storage1 <none> glusterfs-v64wn 1/1 Running 0 4h 192.168.0.4 nfstest-storage0 <none> $ kubectl get pods -o wide | grep glusterfs | grep -v provisioner | awk '{print $6}' 192.168.0.9 192.168.0.4
Exec into each glusterfs pod and perform a gluster peer probe
on the other pod's IP:
$ kubectl exec -it glusterfs-t44m5 -- gluster peer probe 192.168.0.4 peer probe: success. $ kubectl exec -it glusterfs-v64wn -- gluster peer probe 192.168.0.9 peer probe: success. Host 192.168.0.9 port 24007 already in peer list
Congratulations! You now have a GlusterFS cluster running on top of Kubernetes!
Start GlusterFS Simple Provisioner
To install PVC support on your GlusterFS, you need to build up a custom StorageClass containing your GlusterFS pod IPs.
This will also require you to choose a "brick path", a directory on the host where your gluster bricks should be housed.
Normally this path would be mounted from an external volume, but for this example we are just using /tmp
(NOTE: this is obviously not advised in production, as /tmp
is typically cleared upon restart):
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: annotations: storageclass.kubernetes.io/is-default-class: "true" name: glusterfs-simple parameters: brickrootPaths: "192.168.0.9:/tmp,192.168.0.4:/tmp" forceCreate: "true" volumeType: "replica 2" provisioner: gluster.org/glusterfs-simple
For Kubernetes 1.8+, you will also need to install RBAC permissions for the provisioner:
$ kubectl create -f deploy/rbac.yaml serviceaccount/glfs-provisioner created clusterrole.rbac.authorization.k8s.io/glfs-provisioner-runner created clusterrolebinding.rbac.authorization.k8s.io/run-glfs-provisioner created
You are now ready to run the GlusterFS Simple Provisioner deployment:
$ kubectl create -f deploy/deployment.yaml deployment.extensions/glusterfs-simple-provisioner created
The Short Way
Execute the following bash script from your Kubernetes master node to set everything up for you:
$ chmod +x ./deploy-glfs.sh $ ./deploy-glfs <number_of_storage_nodes>
#!/bin/bash # # Usage: ./deploy-glfs.sh <number_of_storage_nodes> # # DEBUG ONLY: Set this to "echo" to neuter the script and perform a dry-run DEBUG="" # The host directory to store brick files BRICK_HOSTDIR="/tmp" # Read in the desired number of storage nodes from first arg NODE_COUNT="$1" # Ensure that we have enough storage nodes to run GLFS if [ "$NODE_COUNT" -lt 2 ]; then echo "ERROR: Cannot deploy GlusterFS with less than 2 nodes" exit 1 fi # Clone external-storage repo for NFS provisioner templates $DEBUG git clone https://github.com/kubernetes-incubator/external-storage # Label storage nodes appropriately STORAGE_NODES=$(kubectl get nodes --no-headers | grep storage | awk '{print $1}') for node in $STORAGE_NODES; do $DEBUG kubectl label nodes $node storagenode=glusterfs done # Create the GLFS cluster $DEBUG kubectl apply -f external-storage/gluster/glusterfs/deploy/glusterfs-daemonset.yaml # Wait for the GLFS cluster to come up count="$(kubectl get pods --no-headers | grep glusterfs | grep -v provisioner | awk '{print $3}' | grep Running | wc -l)" while [ "$count" -lt "$NODE_COUNT" ]; do echo "Waiting for GLFS: $count / $NODE_COUNT" sleep 5 count="$(kubectl get pods --no-headers | grep glusterfs | grep -v provisioner | sed -e s/[\\n\\r]//g | awk '{print $3}' | grep -o Running | wc -l)" done echo "GlusterFS is now Running: $count / $NODE_COUNT" # Retrieve GlusterFS pod IPs PEER_IPS=$(kubectl get pods -o wide | grep glusterfs | grep -v provisioner | awk '{print $6}') # Use pod names / IPs to exec in and perform `gluster peer probe` for pod_ip in ${PEER_IPS}; do for peer_ip in ${PEER_IPS}; do # Skip each node probing itself if [ "$pod_ip" == "$peer_ip" ]; then continue; fi # Perform a gluster peer probe pod_name=$(kubectl get pods -o wide | grep $pod_ip | awk '{print $1}') $DEBUG kubectl exec -it $pod_name gluster peer probe $peer_ip done; done; # Dynamically build StorageClass from pod IPs (see below) BRICK_PATHS="" for pod_ip in ${PEER_IPS[@]}; do # Insert comma if we already started accumlating ips/paths if [ "$BRICK_PATHS" != "" ]; then BRICK_PATHS="$BRICK_PATHS," fi # Build up brickrootPaths one host at a time BRICK_PATHS="${BRICK_PATHS}${pod_ip}:${BRICK_HOSTDIR}" done # Modify StorageClass to contain our GlusterFS brickrootPaths echo "--- kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: glusterfs-simple provisioner: gluster.org/glusterfs-simple parameters: forceCreate: \"true\" volumeType: \"replica 2\" brickrootPaths: \"$BRICK_PATHS\" " > external-storage/gluster/glusterfs/deploy/storageclass.yaml # Create the storage class $DEBUG kubectl apply -f external-storage/gluster/glusterfs/deploy/storageclass.yaml # Bind the necessary ServiceAccount / ClusterRole $DEBUG kubectl apply -f external-storage/gluster/glusterfs/deploy/rbac.yaml # Create the GLFS Simple Provisioner $DEBUG kubectl apply -f external-storage/gluster/glusterfs/deploy/deployment.yaml
Testing it Out
You can create a test PVC to ensure that your GlusterFS and provisioner are working correctly together with Kubernetes:
$ kubectl create -f deploy/pvc.yaml persistentvolumeclaim/gluster-simple-claim created
You should see that you PVC is created with an initial state of Pending and no PersistentVolume has been provisioned for it:
$ kubectl get pvc,pv NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/gluster-simple-claim Pending glusterfs-simple 2s
After a few seconds, a volume will be provisioned for your PVC:
$ kubectl get pvc,pv NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/gluster-simple-claim Bound pvc-e519c597-a195-11e8-82d6-fa163e59d79f 1Gi RWX glusterfs-simple 5s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pvc-e519c597-a195-11e8-82d6-fa163e59d79f 1Gi RWX Delete Bound default/gluster-simple-claim glusterfs-simple 2s
You can exec into the glusterfs pods to verify that a gluster volume was create for your PVC, and check the provisioner pod logs to see how it all happened under the hood:
$ kubectl get pods NAME READY STATUS RESTARTS AGE glusterfs-simple-provisioner-86c6d8c8cd-75bk4 1/1 Running 0 5h glusterfs-t44m5 1/1 Running 0 5h glusterfs-v64wn 1/1 Running 0 5h $ kubectl exec -it glusterfs-t44m5 -- gluster volume list pvc-e519c597-a195-11e8-82d6-fa163e59d79f $ kubectl logs -f glusterfs-simple-provisioner-86c6d8c8cd-75bk4 I0816 15:50:58.969822 1 main.go:47] Provisioner gluster.org/glusterfs-simple specified I0816 15:50:58.969896 1 main.go:56] Building kube configs for running in cluster... I0816 15:50:58.988158 1 provision.go:45] Creating NewGlusterfsProvisioner. I0816 15:50:58.988635 1 leaderelection.go:185] attempting to acquire leader lease kube-system/gluster.org-glusterfs-simple... I0816 15:50:59.000100 1 leaderelection.go:194] successfully acquired lease kube-system/gluster.org-glusterfs-simple I0816 15:50:59.000155 1 event.go:221] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"gluster.org-glusterfs-simple", UID:"2b4eef67-a16c-11e8-82d6-fa163e59d79f", APIVersion:"v1", ResourceVersion:"1165", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' glusterfs-simple-provisioner-86c6d8c8cd-75bk4_2b4e2946-a16c-11e8-ab87-0a580af40102 became leader I0816 15:50:59.000203 1 controller.go:596] Starting provisioner controller gluster.org/glusterfs-simple_glusterfs-simple-provisioner-86c6d8c8cd-75bk4_2b4e2946-a16c-11e8-ab87-0a580af40102! I0816 15:50:59.100536 1 controller.go:645] Started provisioner controller gluster.org/glusterfs-simple_glusterfs-simple-provisioner-86c6d8c8cd-75bk4_2b4e2946-a16c-11e8-ab87-0a580af40102! ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... I0816 20:49:40.074522 1 provision.go:183] mkdir -p 192.168.0.9:/tmp/default/gluster-simple-claim-pvc-e519c597-a195-11e8-82d6-fa163e59d79f I0816 20:49:40.074750 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gluster-simple-claim", UID:"e519c597-a195-11e8-82d6-fa163e59d79f", APIVersion:"v1", ResourceVersion:"37040", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/gluster-simple-claim" I0816 20:49:40.080309 1 exec.go:108] Pod selecterd: default/glusterfs-t44m5 I0816 20:49:40.182105 1 exec.go:81] Result: I0816 20:49:40.182132 1 exec.go:82] Result: I0816 20:49:40.277435 1 exec.go:81] Result: I0816 20:49:40.277462 1 exec.go:82] Result: I0816 20:49:40.375121 1 exec.go:81] Result: I0816 20:49:40.375158 1 exec.go:82] Result: I0816 20:49:40.375171 1 provision.go:183] mkdir -p 192.168.0.4:/tmp/default/gluster-simple-claim-pvc-e519c597-a195-11e8-82d6-fa163e59d79f I0816 20:49:40.378560 1 exec.go:108] Pod selecterd: default/glusterfs-v64wn I0816 20:49:40.501549 1 exec.go:81] Result: I0816 20:49:40.501579 1 exec.go:82] Result: I0816 20:49:40.630585 1 exec.go:81] Result: I0816 20:49:40.630608 1 exec.go:82] Result: I0816 20:49:40.737097 1 exec.go:81] Result: I0816 20:49:40.737193 1 exec.go:82] Result: I0816 20:49:40.741076 1 exec.go:108] Pod selecterd: default/glusterfs-t44m5 I0816 20:49:41.072344 1 exec.go:81] Result: volume create: pvc-e519c597-a195-11e8-82d6-fa163e59d79f: success: please start the volume to access data I0816 20:49:41.072370 1 exec.go:82] Result: I0816 20:49:43.536546 1 exec.go:81] Result: volume start: pvc-e519c597-a195-11e8-82d6-fa163e59d79f: success I0816 20:49:43.536585 1 exec.go:82] Result: I0816 20:49:43.559744 1 controller.go:1043] volume "pvc-e519c597-a195-11e8-82d6-fa163e59d79f" for claim "default/gluster-simple-claim" created I0816 20:49:43.568855 1 controller.go:1060] volume "pvc-e519c597-a195-11e8-82d6-fa163e59d79f" for claim "default/gluster-simple-claim" saved I0816 20:49:43.568887 1 controller.go:1096] volume "pvc-e519c597-a195-11e8-82d6-fa163e59d79f" provisioned for claim "default/gluster-simple-claim" I0816 20:49:43.569213 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gluster-simple-claim", UID:"e519c597-a195-11e8-82d6-fa163e59d79f", APIVersion:"v1", ResourceVersion:"37040", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-e519c597-a195-11e8-82d6-fa163e59d79f