The purpose of this document is to familiarize you with running GlusterFS under Kubernetes.

GlusterFS is an open-source distributed filesystem that allows for PVCs that support ReadWriteMany.

Overview

Running GlusterFS in Kubernetes with PVC support is easier than ever with the GlusterFS Simple Provisioner!

Prerequisites

1+ node Kubernetes cluster
No PVC support currently installed

The Long Way

The external-storage repo gives instructions for bringing this all up by hand.

First Steps

First, you'll need to clone the external-storage repo from the kubernetes-incubator:

$ git clone https://github.com/kubernetes-incubator/external-storage && cd external-storage

Locate the gluster/glusterfs subdirectory, which contains these same instructions on getting things up and running:

$ cd gluster/glusterfs/

Apply the correct node label to each of your storage nodes:

$ kubectl label nodes <storage-node-name> storagenode=glusterfs
node/<storage-node-name> labeled

Start GlusterFS

Bring up the GlusterFS DaemonSet and wait for them to come online:

$ kubectl create -f deploy/glusterfs-daemonset.yaml
daemonset.extensions/glusterfs created


$ kubectl get pods -l glusterfs-node=pod --watch

Locate your pod IPs once they are online:

$ kubectl get pods -o wide | grep glusterfs | grep -v provisioner
NAME                                            READY     STATUS    RESTARTS   AGE       IP            NODE               NOMINATED NODE
glusterfs-t44m5                                 1/1       Running   0          4h        192.168.0.9   nfstest-storage1   <none>
glusterfs-v64wn                                 1/1       Running   0          4h        192.168.0.4   nfstest-storage0   <none>


$ kubectl get pods -o wide | grep glusterfs | grep -v provisioner | awk '{print $6}'
192.168.0.9
192.168.0.4

Exec into each glusterfs pod and perform a gluster peer probe on the other pod's IP:

$ kubectl exec -it glusterfs-t44m5 -- gluster peer probe 192.168.0.4
peer probe: success.


$ kubectl exec -it glusterfs-v64wn -- gluster peer probe 192.168.0.9 
peer probe: success. Host 192.168.0.9 port 24007 already in peer list

Congratulations! You now have a GlusterFS cluster running on top of Kubernetes!

Start GlusterFS Simple Provisioner

To install PVC support on your GlusterFS, you need to build up a custom StorageClass containing your GlusterFS pod IPs.

This will also require you to choose a "brick path", a directory on the host where your gluster bricks should be housed.

Normally this path would be mounted from an external volume, but for this example we are just using /tmp (NOTE: this is obviously not advised in production, as /tmp is typically cleared upon restart):

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: glusterfs-simple
parameters:
  brickrootPaths: "192.168.0.9:/tmp,192.168.0.4:/tmp"
  forceCreate: "true"
  volumeType: "replica 2"
provisioner: gluster.org/glusterfs-simple

For Kubernetes 1.8+, you will also need to install RBAC permissions for the provisioner:

$ kubectl create -f deploy/rbac.yaml
serviceaccount/glfs-provisioner created
clusterrole.rbac.authorization.k8s.io/glfs-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-glfs-provisioner created

You are now ready to run the GlusterFS Simple Provisioner deployment:

$ kubectl create -f deploy/deployment.yaml
deployment.extensions/glusterfs-simple-provisioner created

The Short Way

Execute the following bash script from your Kubernetes master node to set everything up for you:

$ chmod +x ./deploy-glfs.sh
$ ./deploy-glfs <number_of_storage_nodes>

#!/bin/bash
#
# Usage: ./deploy-glfs.sh <number_of_storage_nodes>
# 

# DEBUG ONLY: Set this to "echo" to neuter the script and perform a dry-run
DEBUG=""

# The host directory to store brick files
BRICK_HOSTDIR="/tmp"

# Read in the desired number of storage nodes from first arg
NODE_COUNT="$1"

# Ensure that we have enough storage nodes to run GLFS
if [ "$NODE_COUNT" -lt 2 ]; then
  echo "ERROR: Cannot deploy GlusterFS with less than 2 nodes"
  exit 1
fi

# Clone external-storage repo for NFS provisioner templates
$DEBUG git clone https://github.com/kubernetes-incubator/external-storage 

# Label storage nodes appropriately
STORAGE_NODES=$(kubectl get nodes --no-headers | grep storage | awk '{print $1}')
for node in $STORAGE_NODES; do
  $DEBUG kubectl label nodes $node storagenode=glusterfs 
done

# Create the GLFS cluster
$DEBUG kubectl create -f external-storage/gluster/glusterfs/deploy/glusterfs-daemonset.yaml

# Wait for the GLFS cluster to come up
count="$(kubectl get pods --no-headers | grep glusterfs | grep -v provisioner | awk '{print $3}' | grep Running | wc -l)"
while [ "$count" -lt "$NODE_COUNT" ]; do
  echo "Waiting for GLFS: $count / $NODE_COUNT"
  sleep 5
  count="$(kubectl get pods --no-headers | grep glusterfs | grep -v provisioner | sed -e s/[\\n\\r]//g | awk '{print $3}' | grep -o Running | wc -l)"
done
echo "GlusterFS is now Running: $count / $NODE_COUNT"

# Retrieve GlusterFS pod IPs
PEER_IPS=$(kubectl get pods -o wide | grep glusterfs | grep -v provisioner | awk '{print $6}')

# Use pod names / IPs to exec in and perform `gluster peer probe`
for pod_ip in ${PEER_IPS}; do
  for peer_ip in ${PEER_IPS}; do
    # Skip each node probing itself
    if [ "$pod_ip" == "$peer_ip" ]; then
      continue;
    fi

    # Perform a gluster peer probe
    pod_name=$(kubectl get pods -o wide | grep $pod_ip | awk '{print $1}')
    $DEBUG kubectl exec -it $pod_name gluster peer probe $peer_ip
  done;
done;

# Dynamically build StorageClass from pod IPs (see below)
BRICK_PATHS=""
for pod_ip in ${PEER_IPS[@]}; do
  # Insert comma if we already started accumlating ips/paths
  if [ "$BRICK_PATHS" != "" ]; then
    BRICK_PATHS="$BRICK_PATHS,"
  fi

  # Build up brickrootPaths one host at a time
  BRICK_PATHS="${BRICK_PATHS}${pod_ip}:${BRICK_HOSTDIR}"
done

# Modify StorageClass to contain our GlusterFS brickrootPaths
echo "---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: glusterfs-simple
provisioner: gluster.org/glusterfs-simple
parameters:
  forceCreate: \"true\"
  volumeType: \"replica 2\"
  brickrootPaths: \"$BRICK_PATHS\"
" > external-storage/gluster/glusterfs/deploy/storageclass.yaml

# Create the storage class
$DEBUG kubectl apply -f external-storage/gluster/glusterfs/deploy/storageclass.yaml

# Bind the necessary ServiceAccount / ClusterRole
$DEBUG kubectl apply -f external-storage/gluster/glusterfs/deploy/rbac.yaml

# Create the GLFS Simple Provisioner
$DEBUG kubectl apply -f external-storage/gluster/glusterfs/deploy/deployment.yaml