The purpose of this document is to familiarize you with running GlusterFS under Kubernetes.
GlusterFS is an open-source distributed filesystem that allows for PVCs that support ReadWriteMany.
Overview
Running GlusterFS in Kubernetes with PVC support is easier than ever with the GlusterFS Simple Provisioner!
Prerequisites
- 1+ node Kubernetes cluster
- No PVC support currently installed
The Long Way
The external-storage repo gives instructions for bringing this all up by hand.
First Steps
First, you'll need to clone the external-storage repo from the kubernetes-incubator:
$ git clone https://github.com/kubernetes-incubator/external-storage && cd external-storage
Locate the gluster/glusterfs subdirectory, which contains these same instructions on getting things up and running:
$ cd gluster/glusterfs/
Apply the correct node label to each of your storage nodes:
$ kubectl label nodes <storage-node-name> storagenode=glusterfs node/<storage-node-name> labeled
Start GlusterFS
Bring up the GlusterFS DaemonSet and wait for them to come online:
$ kubectl create -f deploy/glusterfs-daemonset.yaml daemonset.extensions/glusterfs created $ kubectl get pods -l glusterfs-node=pod --watch
Locate your pod IPs once they are online:
$ kubectl get pods -o wide | grep glusterfs | grep -v provisioner NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE glusterfs-t44m5 1/1 Running 0 4h 192.168.0.9 nfstest-storage1 <none> glusterfs-v64wn 1/1 Running 0 4h 192.168.0.4 nfstest-storage0 <none> $ kubectl get pods -o wide | grep glusterfs | grep -v provisioner | awk '{print $6}' 192.168.0.9 192.168.0.4
Exec into each glusterfs pod and perform a gluster peer probe
on the other pod's IP:
$ kubectl exec -it glusterfs-t44m5 -- gluster peer probe 192.168.0.4 peer probe: success. $ kubectl exec -it glusterfs-v64wn -- gluster peer probe 192.168.0.9 peer probe: success. Host 192.168.0.9 port 24007 already in peer list
Congratulations! You now have a GlusterFS cluster running on top of Kubernetes!
Start GlusterFS Simple Provisioner
To install PVC support on your GlusterFS, you need to build up a custom StorageClass containing your GlusterFS pod IPs.
This will also require you to choose a "brick path", a directory on the host where your gluster bricks should be housed.
Normally this path would be mounted from an external volume, but for this example we are just using /tmp
(NOTE: this is obviously not advised in production, as /tmp
is typically cleared upon restart):
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: annotations: storageclass.kubernetes.io/is-default-class: "true" name: glusterfs-simple parameters: brickrootPaths: "192.168.0.9:/tmp,192.168.0.4:/tmp" forceCreate: "true" volumeType: "replica 2" provisioner: gluster.org/glusterfs-simple
For Kubernetes 1.8+, you will also need to install RBAC permissions for the provisioner:
$ kubectl create -f deploy/rbac.yaml serviceaccount/glfs-provisioner created clusterrole.rbac.authorization.k8s.io/glfs-provisioner-runner created clusterrolebinding.rbac.authorization.k8s.io/run-glfs-provisioner created
You are now ready to run the GlusterFS Simple Provisioner deployment:
$ kubectl create -f deploy/deployment.yaml deployment.extensions/glusterfs-simple-provisioner created
The Short Way
Execute the following bash script from your Kubernetes master node to set everything up for you:
$ chmod +x ./deploy-glfs.sh $ ./deploy-glfs <number_of_storage_nodes>
#!/bin/bash # # Usage: ./deploy-glfs.sh <number_of_storage_nodes> # # DEBUG ONLY: Set this to "echo" to neuter the script and perform a dry-run DEBUG="" # The host directory to store brick files BRICK_HOSTDIR="/tmp" # Read in the desired number of storage nodes from first arg NODE_COUNT="$1" # Ensure that we have enough storage nodes to run GLFS if [ "$NODE_COUNT" -lt 2 ]; then echo "ERROR: Cannot deploy GlusterFS with less than 2 nodes" exit 1 fi # Clone external-storage repo for NFS provisioner templates $DEBUG git clone https://github.com/kubernetes-incubator/external-storage # Label storage nodes appropriately STORAGE_NODES=$(kubectl get nodes --no-headers | grep storage | awk '{print $1}') for node in $STORAGE_NODES; do $DEBUG kubectl label nodes $node storagenode=glusterfs done # Create the GLFS cluster $DEBUG kubectl create -f external-storage/gluster/glusterfs/deploy/glusterfs-daemonset.yaml # Wait for the GLFS cluster to come up count="$(kubectl get pods --no-headers | grep glusterfs | grep -v provisioner | awk '{print $3}' | grep Running | wc -l)" while [ "$count" -lt "$NODE_COUNT" ]; do echo "Waiting for GLFS: $count / $NODE_COUNT" sleep 5 count="$(kubectl get pods --no-headers | grep glusterfs | grep -v provisioner | sed -e s/[\\n\\r]//g | awk '{print $3}' | grep -o Running | wc -l)" done echo "GlusterFS is now Running: $count / $NODE_COUNT" # Retrieve GlusterFS pod IPs PEER_IPS=$(kubectl get pods -o wide | grep glusterfs | grep -v provisioner | awk '{print $6}') # Use pod names / IPs to exec in and perform `gluster peer probe` for pod_ip in ${PEER_IPS}; do for peer_ip in ${PEER_IPS}; do # Skip each node probing itself if [ "$pod_ip" == "$peer_ip" ]; then continue; fi # Perform a gluster peer probe pod_name=$(kubectl get pods -o wide | grep $pod_ip | awk '{print $1}') $DEBUG kubectl exec -it $pod_name gluster peer probe $peer_ip done; done; # Dynamically build StorageClass from pod IPs (see below) BRICK_PATHS="" for pod_ip in ${PEER_IPS[@]}; do # Insert comma if we already started accumlating ips/paths if [ "$BRICK_PATHS" != "" ]; then BRICK_PATHS="$BRICK_PATHS," fi # Build up brickrootPaths one host at a time BRICK_PATHS="${BRICK_PATHS}${pod_ip}:${BRICK_HOSTDIR}" done # Modify StorageClass to contain our GlusterFS brickrootPaths echo "--- kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: glusterfs-simple provisioner: gluster.org/glusterfs-simple parameters: forceCreate: \"true\" volumeType: \"replica 2\" brickrootPaths: \"$BRICK_PATHS\" " > external-storage/gluster/glusterfs/deploy/storageclass.yaml # Create the storage class $DEBUG kubectl apply -f external-storage/gluster/glusterfs/deploy/storageclass.yaml # Bind the necessary ServiceAccount / ClusterRole $DEBUG kubectl apply -f external-storage/gluster/glusterfs/deploy/rbac.yaml # Create the GLFS Simple Provisioner $DEBUG kubectl apply -f external-storage/gluster/glusterfs/deploy/deployment.yaml