Overview

Rook is an open-source distributed filesystem designed for use under Kubernetes, and is only supported on Kubernetes 1.7 or higher.

...

Source code is also available here: https://github.com/rook/rook

Prerequisites

Minimum Version: Kubernetes v1.7 or higher is supported by Rook.

...

You will also need to set up RBAC, and ensure that the Flex volume plugin has been configured.

Set the `dataDirHostPath`

If you are using dataDirHostPath to persist Rook data on Kubernetes hosts, make sure your host has at least 5GB of space available on the specified path.

Setting up RBAC

On Kubernetes 1.7+, you will need to configure Rook to use RBAC appropriately.

See https://rook.github.io/docs/rook/master/rbac.html

Flex Volume Configuration

The Rook agent requires setup as a Flex volume plugin to manage the storage attachments in your cluster. See the Flex Volume Configuration topic to configure your Kubernetes deployment to load the Rook volume plugin.

Getting Started

Now that we've examined each of the pieces, let's zoom out and see what we can do with the whole cluster.

For the quickest quick start, check out the Rook QuickStart guide: https://rook.github.io/docs/rook/master/quickstart.html

Getting Started without an Existing Kubernetes cluster

The easiest way to deploy a new Kubernetes cluster with Rook support on OpenStack (Nebula / SDSC) is to use the https://github.com/nds-org/kubeadm-terraform repository.

This may work for other cloud providers as well, but has not yet been thoroughly tested.

Getting Started on an Existing Kubernetes cluster

If you’re feeling lucky, a simple Rook cluster can be created with the following kubectl commands. For the more detailed install, skip to the next section to deploy the Rook operator.

...

For a more detailed look at the deployment process, see below.

Deploy the Rook Operator

The first step is to deploy the Rook system components, which include the Rook agent running on each node in your cluster as well as Rook operator pod.

...

You can also deploy the operator with the Rook Helm Chart.

Restart Kubelet (Kubernetes 1.7.x only)

For versions of Kubernetes prior to 1.8, the Kubelet process on all nodes will require a restart after the Rook operator and Rook agents have been deployed. As part of their initial setup, the Rook agents deploy and configure a Flexvolume plugin in order to integrate with Kubernetes’ volume controller framework. In Kubernetes v1.8+, the dynamic Flexvolume plugin discovery will find and initialize our plugin, but in older versions of Kubernetes a manual restart of the Kubelet will be required.

Create a Rook Cluster

Now that the Rook operator and agent pods are running, we can create the Rook cluster. For the cluster to survive reboots, make sure you set the dataDirHostPath property. For more settings, see the documentation on configuring the cluster.

...

Code Block

language	bash

$ kubectl -n rook get pod
NAME                              READY     STATUS    RESTARTS   AGE
rook-ceph-mgr0-1279756402-wc4vt   1/1       Running   0          5m
rook-ceph-mon0-jflt5              1/1       Running   0          6m
rook-ceph-mon1-wkc8p              1/1       Running   0          6m
rook-ceph-mon2-p31dj              1/1       Running   0          6m
rook-ceph-osd-0h6nb               1/1       Running   0          5m

Monitoring Your Rook Cluster

A glimpse into setting up Prometheus for monitoring Rook: https://rook.github.io/docs/rook/master/monitoring.html

Advanced Configuration

Advanced Configuration options are also documented here: https://rook.github.io/docs/rook/master/advanced-configuration.html

Debugging

For common issues, see https://github.com/rook/rook/blob/master/Documentation/common-issues.md

For more help debugging, see https://github.com/rook/rook/blob/master/Documentation/toolbox.md

Cluster Teardown

See https://rook.github.io/docs/rook/master/teardown.html for thorough steps on destroying / cleaning up your Rook cluster

Components

Rook runs a number of smaller microservices that run on different nodes in your Kubernetes cluster:

The Rook Operator + API
Ceph Managers / Monitors / OSDs
Rook Agents

The Rook Operator

The Rook operator is a simple container that has all that is needed to bootstrap and monitor the storage cluster.

...

The Rook operator also creates the Rook agents as a daemonset, which runs a pod on each node.

Ceph Managers / Monitors / OSDs

The operator will start and monitor ceph monitor pods and a daemonset for the OSDs, which provides basic Reliable Autonomic Distributed Object Store (RADOS) storage.

...

Ceph monitors (aka "Ceph mons") will be started or failed over when necessary, and other adjustments are made as the cluster grows or shrinks.

Rook Agents

Each agent is a pod deployed on a different Kubernetes node, which configures a Flexvolume plugin that integrates with Kubernetes’ volume controller framework.

All storage operations required on the node are handled such as attaching network storage devices, mounting volumes, and formatting the filesystem.

Storage

Rook provides three types of storage to the Kubernetes cluster:

Block Storage: Mount storage to a single pod
Object Storage: Expose an S3 API to the storage cluster for applications to put and get data that is accessible from inside or outside the Kubernetes cluster
Shared File System: Mount a file system that can be shared across multiple pods

Custom Resource Definitions

Rook also allows you to create and manage your storage cluster through custom resource definitions (CRDs). Each type of resource has its own CRD defined.

Cluster: A Rook cluster provides the basis of the storage platform to serve block, object stores, and shared file systems.
Pool: A pool manages the backing store for a block store. Pools are also used internally by object and file stores.
Object Store: An object store exposes storage with an S3-compatible interface.
File System: A file system provides shared storage for multiple Kubernetes pods.

Shared Storage Example

Shamelessly stolen from https://rook.github.io/docs/rook/master/filesystem.html

Prerequisites

This guide assumes you have created a Rook cluster as explained in the main Kubernetes guide

Multiple File Systems Not Supported

By default only one shared file system can be created with Rook. Multiple file system support in Ceph is still considered experimental and can be enabled with the environment variable ROOK_ALLOW_MULTIPLE_FILESYSTEMS defined in rook-operator.yaml.

Please refer to cephfs experimental features page for more information.

Create the File System

Create the file system by specifying the desired settings for the metadata pool, data pools, and metadata server in the FilesystemCRD. In this example we create the metadata pool with replication of three and a single data pool with erasure coding. For more options, see the documentation on creating shared file systems.

...

Code Block

language	yml

apiVersion: rook.io/v1alpha1
kind: Filesystem
metadata:
  name: myfs
  namespace: rook
spec:
  metadataPool:
    replicated:
      size: 3        <------------ this was eventually the problem
  dataPools:
    - erasureCoded:
       dataChunks: 2
       codingChunks: 1
  metadataServer:
    activeCount: 1
    activeStandby: true

...

Code Block

language	bash

$ ceph status                                                                                                                                              
  ...
  services:
    mds: myfs-1/1/1 up {[myfs:0]=mzw58b=up:active}, 1 up:standby-replay

Consume the Shared File System: Busybox + NGINX Example

As an example, we will start the kube-registry pod with the shared file system as the backing store. Save the following spec as kube-registry.yaml:

...

NOTE: I had to explicitly specify clusterName in the YAML above... newer versions of Rook will fallback to clusterNamespace

Kernel Version Requirement

If the Rook cluster has more than one filesystem and the application pod is scheduled to a node with kernel version older than 4.7, inconsistent results may arise since kernels older than 4.7 do not support specifying filesystem namespaces.

Testing Shared Storage

After creating our above example, we should now have 2 pods each with 2 containers running on 2 separate nodes:

...

You have just set up your first shared filesystem under Rook!

Under the Hood

For more information on the low-level processes involved in the above example, see https://github.com/rook/rook/blob/master/design/filesystem.md

...

The directories section is supposed to list the paths that will be included in the storage cluster. (Note that using two directories on the same physical device can cause a negative performance impact.)

Investigating Storage `directories`

Checking the logs for one of the Rook agents, we can see a success message shows us where the data really lives:

...

Obviously this is not where we want the shared filesystem data stored long-term, so I'll need to figure out why these files are persisted into /var/lib/kubelet and not into the directories specified in the Cluster configuration.

Digging Deeper into `dataDirHostPath`

Checking /var/lib/rook directory, we see a few sub-directories:

...

As you can see, these metadata files do not appear to be readable on disk and would likely need to be un-mangled by Rook to properly perform a backup.

Checking the `kubelet` logs...

Digging into the systemctl logs for kubectl, we can see it's complaining about the volume configuration:

...

Sadly, even setting this value explicitly did not fix my immediate issue.

Hacking Terraform

At this point, I decided to start hacking the Terraform deployment to get Rook working to the level we'll need for Workbench.

...

Rook has been upgraded from v0.6.2 to v0.7.1, in the helm install and in rook-cluster.yaml
Expanded storage section to include a nodes subsection - this specifies which machines / directories should be part of the storage cluster
Turn off useAllNodes

Checking the `rook-operator` logs...

Now, with the new version of rook up and running, I attempted to make a filesystem as before. This time, however, no pods were spawned following my filesystem's creation.

...

Changing /vol_b to /volb solved this problem - this must be adjusted both in the deploy-rook.sh script above, as well as the bootstrap-rook.sh script alongside of it.

Now we're getting somewhere...

After changing the volume path and redeploying (again), now myfs pods were being spawned after creating the filesystem in Kubernetes, as they should be:

...

Upgrading to Rook 0.7.1
The adjustments to the directories configuration in rook-cluster.yaml are now writing data to the correct drive (/volb), but that drive may be improperly formatted for use with Rook

Narrowing it down...

Adjusting my rook-cluster.yaml to only include the adjustments to the directories configuration, and to use storeType: filestore instead of bluestore.

...

Code Block

language	bash

ubuntu@mldev-storage0:~$ sudo ls -al /rook/osd1/
total 5242956
drwxr--r--   4 root root       4096 May 24 16:03 .
drwxr-xr-x   4 root root       4096 May 24 16:03 ..
-rw-r--r--   1 root root         37 May 24 16:03 ceph_fsid
drwxr-xr-x 508 root root      20480 May 24 16:11 current
-rw-r--r--   1 root root         37 May 24 16:03 fsid
-rw-r--r--   1 root root 5368709120 May 24 16:12 journal
-rw-r--r--   1 root root         56 May 24 16:03 keyring
-rw-r--r--   1 root root         21 May 24 16:03 magic
-rw-r--r--   1 root root          6 May 24 16:03 ready
-rw-r--r--   1 root root       1283 May 24 16:03 rook.config
srwxr-xr-x   1 root root          0 May 24 16:03 rook-osd.1.asok
-rw-r--r--   1 root root          4 May 24 16:03 store_version
-rw-r--r--   1 root root         53 May 24 16:03 superblock
drwxr--r--   2 root root       4096 May 24 16:03 tmp
-rw-r--r--   1 root root         10 May 24 16:03 type
-rw-r--r--   1 root root          2 May 24 16:03 whoami
ubuntu@mldev-storage0:~$ sudo ls -al /rook/osd1/current
total 4080
drwxr-xr-x 508 root root 20480 May 24 16:11 .
drwxr--r--   4 root root  4096 May 24 16:03 ..
drwxr-xr-x   2 root root  4096 May 24 16:04 1.0_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.10_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.10_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.11_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.11_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.12_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.12_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.13_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.13_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.14_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.14_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.15_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.15_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.16_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.16_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.17_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.17_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.18_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.18_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.19_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.19_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.1a_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.1a_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.1b_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.1b_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.1c_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.1c_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.1d_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.1d_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.1e_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.1e_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.1f_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.1f_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.1_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.1_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.20_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.20_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.21_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.21_TEMP
drwxr-xr-x    ...    ...    ...    2... root root  4096... May 24 16:06 1...22_head
drwxr-xr-x   2 root root  4096 May 24 16:0609 13.2245s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:0609 13.2346s0_head
drwxr-xr-x   2 root root  4096 May 24 16:0609 13.2346s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:0609 13.2447s0_head
drwxr-xr-x   2 root root  4096 May 24 16:0609 13.2447s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:0409 13.2549s0_head
drwxr-xr-x   2 root root  4096 May 24 16:0409 13.2549s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:0609 13.264es0_head
drwxr-xr-x   2 root root  4096 May 24 16:0609 13.264es0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:0609 13.274s0_head
drwxr-xr-x   2 root root  4096 May 24 16:0609 13.274s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:0409 13.2850s0_head
drwxr-xr-x   2 root root  4096 May 24 16:0409 13.2850s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:0610 13.2951s0_head
drwxr-xr-x   2 root root  4096 May 24 16:0609 13.2951s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:0410 13.2a52s0_head
drwxr-xr-x   2 root root  4096 May 24 16:0409 13.2a52s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:0410 13.2b55s0_head
drwxr-xr-x   2 root root  4096 May 24 16:0410 13.2b55s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:0610 13.2c56s0_head
drwxr-xr-x   2 root root  4096 May 24 16:0610 13.2c56s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:0610 13.2d57s0_head
drwxr-xr-x   2 root root  4096 May 24 16:0610 13.2d57s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:0610 13.2e59s0_head
drwxr-xr-x   2 root root  4096 May 24 16:0610 13.2e59s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:0610 13.2f5as0_head
drwxr-xr-x   2 root root  4096 May 24 16:0610 13.2f5as0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:0410 13.25ds0_head
drwxr-xr-x   2 root root  4096 May 24 16:0410 13.25ds0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:0410 13.305es0_head
drwxr-xr-x   2 root root  4096 May 24 16:0410 13.305es0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:0610 13.315fs0_head
drwxr-xr-x   2 root root  4096 May 24 16:0610 13.315fs0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:0610 13.3262s0_head
drwxr-xr-x   2 root root  4096 May 24 16:0610 13.3262s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:0410 13.3363s0_head
drwxr-xr-x   2 root root  4096 May 24 16:0410 13.3363s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:0609 13.346s0_head
drwxr-xr-x   2 root root  4096 May 24 16:0609 13.346s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:0709 13.357s0_head
drwxr-xr-x   2 root root  4096 May 24 16:0609 13.357s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:0409 13.36bs0_head
drwxr-xr-x   2 root root  4096 May 24 16:0409 13.36bs0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:0409 13.37es0_head
drwxr-xr-x   2 root root  4096 May 24 16:0409 13.37es0_TEMP
drwxr-xr-x-rw-r--r--   21 root root  4096   4 May 24 16:04 1.38_head12 commit_op_seq
drwxr-xr-x   2 root root  409612288 May 24 16:04 1.38_TEMP
drwxr-xr-x12 meta
-rw-r--r--   21 root root    4096 0 May 24 16:07 1.39_head03 nosnap
drwxr-xr-x   2 root root  4096 May 24 16:07 1.39_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.3a_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.3a_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3b_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3b_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3c_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3c_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3d_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3d_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3e_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3e_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3f_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3f_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.3_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.40_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.40_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.41_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.41_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.42_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.42_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.43_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.43_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.44_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.44_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.45_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.45_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.46_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.46_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.47_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.47_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.48_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.48_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.49_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.49_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.4a_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.4a_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.4b_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.4b_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.4c_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.4c_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.4d_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.4d_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.4e_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.4e_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.4f_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.4f_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.4_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.4_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.50_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.50_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.51_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.51_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.52_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.52_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.53_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.53_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.54_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.54_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.55_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.55_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.56_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.56_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:05 1.57_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.57_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:05 1.58_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.58_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.59_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.59_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.5a_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.5a_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.5b_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.5b_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:05 1.5c_head
drwxr-xr-x   2 root root  4096 May 24 16:05 1.5c_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:05 1.5d_head
drwxr-xr-x   2 root root  4096 May 24 16:05 1.5d_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.5e_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.5e_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 1.5f_head
drwxr-xr-x   2 root root  4096 May 24 16:07 1.5f_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.5_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.5_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.60_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.60_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:05 1.61_head
drwxr-xr-x   2 root root  4096 May 24 16:05 1.61_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.62_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.62_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:05 1.63_head
drwxr-xr-x   2 root root  4096 May 24 16:05 1.63_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.6_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.6_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.7_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.7_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.8_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.8_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.9_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.9_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.a_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.a_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.b_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.b_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.c_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.c_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.d_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.d_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:06 1.e_head
drwxr-xr-x   2 root root  4096 May 24 16:06 1.e_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:04 1.f_head
drwxr-xr-x   2 root root  4096 May 24 16:04 1.f_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.0_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.10_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.10_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.11_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.11_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.12_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.12_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 2.13_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.13_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.14_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.14_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.15_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.15_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 2.16_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.16_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.17_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.17_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 2.18_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.18_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.19_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.19_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.1a_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.1a_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.1b_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.1b_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.1c_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.1c_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.1d_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.1d_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.1e_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.1e_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.1f_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.1f_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.1_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.1_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.20_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.20_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.21_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.21_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.22_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.22_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.23_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.23_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.24_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.24_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.25_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.25_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.26_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.26_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.27_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.27_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.28_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.28_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.29_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.29_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.2a_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.2a_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.2b_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.2b_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.2c_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.2c_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.2d_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.2d_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.2e_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.2e_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.2f_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.2f_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.2_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.2_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.30_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.30_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.31_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.31_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.32_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.32_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.33_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.33_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.34_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.34_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.35_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.35_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.36_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.36_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.37_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.37_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.38_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.38_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.39_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.39_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.3a_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.3a_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.3b_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.3b_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.3c_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.3c_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.3d_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.3d_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.3e_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.3e_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.3f_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.3f_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.3_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.3_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.40_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.40_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.41_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.41_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.42_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.42_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.43_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.43_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.44_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.44_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.45_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.45_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.46_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.46_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.47_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.47_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.48_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.48_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.49_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.49_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.4a_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.4a_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.4b_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.4b_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.4c_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.4c_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.4d_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.4d_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.4e_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.4e_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.4f_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.4f_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 2.4_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.4_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.50_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.50_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.51_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.51_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.52_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.52_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.53_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.53_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.54_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.54_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.55_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.55_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.56_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.56_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.57_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.57_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.58_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.58_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.59_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.59_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.5a_head
drwxr-xr-x   2 root root  4096 May 24 16:11 2.5a_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.5b_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.5b_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.5c_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.5c_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.5d_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.5d_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.5e_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.5e_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.5f_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.5f_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 2.5_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.5_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.60_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.60_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.61_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.61_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.62_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.62_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:08 2.63_head
drwxr-xr-x   2 root root  4096 May 24 16:08 2.63_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 2.6_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.6_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.7_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.7_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.8_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.8_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 2.9_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.9_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 2.a_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.a_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 2.b_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.b_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 2.c_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.c_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 2.d_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.d_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:07 2.e_head
drwxr-xr-x   2 root root  4096 May 24 16:07 2.e_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:11 2.f_head
drwxr-xr-x   2 root root  4096 May 24 16:10 2.f_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.0s0_head
drwxr-xr-x   2 root root  4096 May 24 16:08 3.0s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.12s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.12s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.14s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.14s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.18s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.18s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.19s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.19s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.1as0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.1as0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.1bs0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.1bs0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.1cs0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.1cs0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.1es0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.1es0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.1fs0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.1fs0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.1s0_head
drwxr-xr-x   2 root root  4096 May 24 16:08 3.1s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.20s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.20s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.21s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.21s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.22s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.22s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.24s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.24s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.27s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.27s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.28s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.28s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.29s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.29s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.2ds0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.2ds0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.2es0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.2es0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.30s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.30s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.32s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.32s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.38s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.38s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.39s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.39s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.3es0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.3es0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.3s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.3s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.40s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.40s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.41s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.41s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.42s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.42s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.45s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.45s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.46s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.46s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.47s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.47s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.49s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.49s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.4es0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.4es0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.4s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.4s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.50s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.50s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.51s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.51s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.52s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.52s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.55s0_head
drwxr-xr-x   2 root root  4096 May 24 16:10 3.55s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.56s0_head
drwxr-xr-x   2 root root  4096 May 24 16:10 3.56s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.57s0_head
drwxr-xr-x   2 root root  4096 May 24 16:10 3.57s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.59s0_head
drwxr-xr-x   2 root root  4096 May 24 16:10 3.59s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.5as0_head
drwxr-xr-x   2 root root  4096 May 24 16:10 3.5as0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.5ds0_head
drwxr-xr-x   2 root root  4096 May 24 16:10 3.5ds0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.5es0_head
drwxr-xr-x   2 root root  4096 May 24 16:10 3.5es0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.5fs0_head
drwxr-xr-x   2 root root  4096 May 24 16:10 3.5fs0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.62s0_head
drwxr-xr-x   2 root root  4096 May 24 16:10 3.62s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:10 3.63s0_head
drwxr-xr-x   2 root root  4096 May 24 16:10 3.63s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.6s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.6s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.7s0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.7s0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.bs0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.bs0_TEMP
drwxr-xr-x   2 root root  4096 May 24 16:09 3.es0_head
drwxr-xr-x   2 root root  4096 May 24 16:09 3.es0_TEMP
-rw-r--r--   1 root root     4 May 24 16:12 commit_op_seq
drwxr-xr-x   2 root root 12288 May 24 16:12 meta
-rw-r--r--   1 root root     0 May 24 16:03 nosnap
drwxr-xr-x   2 root root  4096 May 24 16:03 omap

But it did actively prevent me from using a shared filesystem:

Code Block

language	bash

2018-05-24 16:08:39.653246 I | exec: Error EINVAL: pool must only be stored on bluestore for scrubbing to work: osd.1 uses filestore
2018-05-24 16:08:39.653337 W | cephmds: failed to set ec pool property. failed to set pool property allow_ec_overwrites on pool myfs-data0, exit status 22
2018-05-24 16:08:39.653422 I | exec: Running command: ceph fs flag set enable_multiple true --yes-i-really-mean-it --cluster=rook --conf=/var/lib/rook/rook/rook.config --keyring=/var/lib/rook/rook/client.admin.keyring --format json --out-file /tmp/566066704
2018-05-24 16:08:43.644068 I | exec: Running command: ceph fs new myfs myfs-metadata myfs-data0 --cluster=rook --conf=/var/lib/rook/rook/rook.config --keyring=/var/lib/rook/rook/client.admin.keyring --format json --out-file /tmp/808516655
2018-05-24 16:08:45.443925 I | exec: Error EINVAL: pool 'myfs-data0' (id '3') is an erasure-coded pool, with no overwrite support
2018-05-24 16:08:45.444031 E | op-mds: failed to create file system myfs. failed to create file system myfs: failed enabling ceph fs myfs. exit status 22

Confirmed by this GitHub issue: https://github.com/rook/rook/issues/1604

Back to bluestore...

Switching back to storeType: bluestore on Rook v0.6.2 with the correct nodes/directories configuration:

Code Block

language	bash

#!/bin/bash
git clone https://github.com/groundnuty/k8s-wait-for.git

sudo helm repo add rook-alpha https://charts.rook.io/alpha
sudo helm install rook-alpha/rook --name rook --version 0.6.2


   ...    ...    ...    ...


# Now we can install the cluster
echo '
apiVersion: v1
kind: Namespace
metadata:
  name: rook
---
apiVersion: rook.io/v1alpha1
kind: Cluster
metadata:
  name: rook
  namespace: rook
spec:
  versionTag: v0.6.2
  dataDirHostPath: /var/lib/rook
  # cluster level storage configuration and selection
  storage:                
    useAllNodes: false
    useAllDevices: false
    deviceFilter:
    metadataDevice:
    location:
    storeConfig:
      storeType: bluestore
      databaseSizeMB: 1024 # this value can be removed for environments with normal sized disks (100 GB or larger)
      journalSizeMB: 1024  # this value can be removed for environments with normal sized disks (20 GB or larger)
    - name: "mldev-storage0"
      directories:         # specific directories to use for storage can be specified for each node
      - path: "/rook"      # if changed, also adjust the mount path in bootstrap-rook.sh
    - name: "mldev-storage1"
      directories:         # specific directories to use for storage can be specified for each node
      - path: "/rook"      # if changed, also adjust the mount path in bootstrap-rook.sh
' | kubectl create -f -


# And create a storage class
wget -q https://raw.githubusercontent.com/zonca/jupyterhub-deploy-kubernetes-jetstream/master/storage_rook/rook-storageclass.yaml
sudo kubectl create -f rook-storageclass.yaml

Edge Cases and Quirks

There are many pitfalls here, particularly surrounding my perceived fragility of the shared filesystem

DO NOT delete the filesystem before shutting down the pods consuming it

Deleting the shared filesystem out from under the pod will confuse the kubelet, and prevent it from being able to properly unmount and terminate your containers.

If you need to shut down your shared filesystem, please ensure that you first shut down any pods consuming it's storage.

You must follow these cleanup steps before `terraform destroy` will work

Expanding on the above topic, terraform destroy will hang on destroying your cluster if you fail to cleanup your filesystems properly:

Shut down any pods that are consuming the shared filesystem (e.g. any clients)
Shut down the filesystem pods (e.g. any servers)
Run terraform destroy to cleanup all other cluster resources

WARNING: this seems like a very tenuous/tedious process... I am hoping that later versions of terraform/rook will improve the stability of cleanup under these scenarios. Perhaps we can expand their cleanup to first drain all nodes of their running pods (if this is not already the case), although this would not fix the case of a user deleting the filesystem before a running pod that is consuming it - in this case, the pods will fail to terminate indefinitely, which I think is what is leading terraform to fail.

...

03 omap

But it did actively prevent me from using a shared filesystem:

Code Block

language	bash

2018-05-24 16:08:39.653246 I | exec: Error EINVAL: pool must only be stored on bluestore for scrubbing to work: osd.1 uses filestore
2018-05-24 16:08:39.653337 W | cephmds: failed to set ec pool property. failed to set pool property allow_ec_overwrites on pool myfs-data0, exit status 22
2018-05-24 16:08:39.653422 I | exec: Running command: ceph fs flag set enable_multiple true --yes-i-really-mean-it --cluster=rook --conf=/var/lib/rook/rook/rook.config --keyring=/var/lib/rook/rook/client.admin.keyring --format json --out-file /tmp/566066704
2018-05-24 16:08:43.644068 I | exec: Running command: ceph fs new myfs myfs-metadata myfs-data0 --cluster=rook --conf=/var/lib/rook/rook/rook.config --keyring=/var/lib/rook/rook/client.admin.keyring --format json --out-file /tmp/808516655
2018-05-24 16:08:45.443925 I | exec: Error EINVAL: pool 'myfs-data0' (id '3') is an erasure-coded pool, with no overwrite support
2018-05-24 16:08:45.444031 E | op-mds: failed to create file system myfs. failed to create file system myfs: failed enabling ceph fs myfs. exit status 22

Confirmed by this GitHub issue: https://github.com/rook/rook/issues/1604

Back to bluestore...

Switching back to storeType: bluestore on Rook v0.6.2 with the correct nodes/directories configuration:

Code Block

language	bash

#!/bin/bash
git clone https://github.com/groundnuty/k8s-wait-for.git

sudo helm repo add rook-alpha https://charts.rook.io/alpha
sudo helm install rook-alpha/rook --name rook --version 0.6.2


   ...    ...    ...    ...


# Now we can install the cluster
echo '
apiVersion: v1
kind: Namespace
metadata:
  name: rook
---
apiVersion: rook.io/v1alpha1
kind: Cluster
metadata:
  name: rook
  namespace: rook
spec:
  versionTag: v0.6.2
  dataDirHostPath: /var/lib/rook
  # cluster level storage configuration and selection
  storage:                
    useAllNodes: false
    useAllDevices: false
    deviceFilter:
    metadataDevice:
    location:
    storeConfig:
      storeType: bluestore
      databaseSizeMB: 1024 # this value can be removed for environments with normal sized disks (100 GB or larger)
      journalSizeMB: 1024  # this value can be removed for environments with normal sized disks (20 GB or larger)
    - name: "mldev-storage0"
      directories:         # specific directories to use for storage can be specified for each node
      - path: "/rook"      # if changed, also adjust the mount path in bootstrap-rook.sh
    - name: "mldev-storage1"
      directories:         # specific directories to use for storage can be specified for each node
      - path: "/rook"      # if changed, also adjust the mount path in bootstrap-rook.sh
' | kubectl create -f -


# And create a storage class
wget -q https://raw.githubusercontent.com/zonca/jupyterhub-deploy-kubernetes-jetstream/master/storage_rook/rook-storageclass.yaml
sudo kubectl create -f rook-storageclass.yaml

With this configuration, the filesystem pods came up just fine, but this exhibited the same behavior as the case above, where writing to a file causes the container to hang indefinitely.

The only way to recover in this case is to Ctrl+P, Ctrl+Q to abandon the kubectl exec command, Ctrl+C-ing as necessary - this leaves a hanging exec process, but it is still unclear whether there are performance concerns about executing this case repeatedly without rebooting.

I have noticed that cluster with pods in an error state such as this one will fail to terraform destroy (the operation never completes even after waiting 15+ minutes)

Resolution

After pouring through the docs and GitHub issues and tediously reading the source code, we found a concerning comment in a GitHub issue: https://github.com/rook/rook/issues/1220#issuecomment-343342515

It turns out that this example shared filesystem requires 3 nodes to function - this is why it was mysteriously failing when only 2 OSDs were present.

All in all, as long as you have more OSDs than the spec.metadataPool.replicated.size you have configured in rook-filesystem.yaml

In our sample cluster we have 2 storage nodes, so setting this value to 2 fixed our test case.

The final working configuration can be found below:

Code Block

language	yml
title	rook-cluster.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: rook
---
apiVersion: rook.io/v1alpha1
kind: Cluster
metadata:
  name: rook
  namespace: rook
spec:
  versionTag: v0.6.2
  dataDirHostPath: /var/lib/rook
  # cluster level storage configuration and selection
  storage:                
    useAllNodes: false
    useAllDevices: false
    deviceFilter:
    metadataDevice:
    location:
    storeConfig:
      storeType: bluestore
      databaseSizeMB: 1024 # this value can be removed for environments with normal sized disks (100 GB or larger)
      journalSizeMB: 1024  # this value can be removed for environments with normal sized disks (20 GB or larger)
    - name: "mldev-storage0"
      directories:         # specific directories to use for storage can be specified for each node
      - path: "/rook"      # if changed, also adjust the mount path in bootstrap-rook.sh
    - name: "mldev-storage1"
      directories:         # specific directories to use for storage can be specified for each node
      - path: "/rook"      # if changed, also adjust the mount path in bootstrap-rook.sh

Code Block

language	yml
title	rook-filesystem.yaml

apiVersion: rook.io/v1alpha1
kind: Filesystem
metadata:
  name: myfs
  namespace: rook
spec:
  metadataPool:
    replicated:
      size: 2
  dataPools:
    - erasureCoded:
       dataChunks: 2
       codingChunks: 1
  metadataServer:
    activeCount: 1
    activeStandby: true

Recovering from backup

This feature is currently in the planning stages: https://github.com/rook/rook/issues/1552

Unofficial Python script for creating / restoring backups from Rook: https://gitlab.com/costrouc/kubernetes-rook-backup

Edge Cases and Quirks

There are many pitfalls here, particularly surrounding my perceived fragility of the shared filesystem

DO NOT delete the filesystem before shutting down all of the pods consuming it

Deleting the shared filesystem out from under the pod will confuse the kubelet, and prevent it from being able to properly unmount and terminate your containers.

If you need to shut down your shared filesystem, please ensure that you first shut down any pods consuming it's storage.

This will hopefully be improved in later versions of Kubernetes (1.9+?)

You must follow these cleanup steps before `terraform destroy` will work

Expanding on the above topic, terraform destroy will hang on destroying your cluster if you fail to cleanup your filesystems properly:

Shut down any pods that are consuming the shared filesystem (e.g. any clients)
Shut down the filesystem pods (e.g. any servers)
Run terraform destroy to cleanup all other cluster resources

WARNING: this seems like a very tenuous/tedious process... I am hoping that later versions of terraform/rook will improve the stability of cleanup under these scenarios. Perhaps we can expand their cleanup to first drain all nodes of their running pods (if this is not already the case), although this would not fix the case of a user deleting the filesystem before a running pod that is consuming it - in this case, the pods will fail to terminate indefinitely, which I think is what is leading terraform to fail.

Kill (or hide) a hanging pod

There are a few ways to kill a pod with fire:

Code Block

language	bash

# Attempt to shorten the grace period
ubuntu@mldev-master:~$ kubectl delete pod deployment-demo-c59b896c8-8hgnm --grace-period=1
pod "deployment-demo-c59b896c8-8hgnm" deleted


# If that doesn't work, you can try to force it
ubuntu@mldev-master:~$ kubectl delete pod deployment-demo-c59b896c8-8hgnm --grace-period=1 --force
pod "deployment-demo-c59b896c8-8hgnm" deleted


# If your pod still doesn't terminate, you can change try it with --now instead:
ubuntu@mldev-master:~$ kubectl delete pod deployment-demo-c59b896c8-8hgnm --now
pod "deployment-demo-c59b896c8-8hgnm" deleted
ubuntu@mldev-master:~$ kubectl delete pod deployment-demo-c59b896c8-8hgnm --now --force
pod "deployment-demo-c59b896c8-8hgnm" deleted


# As a last resort, drop the grace period to zero
# WARNING: Only use this as a last resort, and never use it in production!
ubuntu@mldev-master:~$ kubectl delete pod deployment-demo-c59b896c8-8hgnm --grace-period=0 --force
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "deployment-demo-c59b896c8-8hgnm" deleted

Note that the last method may not actually cleanup all resources properly, as noted in the output above.

Only use this as a last resort on test clusters, and NEVER use --grace-period=0 on a production cluster.

Cleaning up failed runs of `terraform destroy`

Here is a quick checklist of the items that you will need to manually if you are unable to terraform destroy your cluster:

...

Page tree

Page History

Versions Compared

Old Version 10

New Version Current

Key

Overview

Prerequisites

Set the dataDirHostPath

Setting up RBAC

Flex Volume Configuration

Getting Started

Getting Started without an Existing Kubernetes cluster

Getting Started on an Existing Kubernetes cluster

Deploy the Rook Operator

Restart Kubelet (Kubernetes 1.7.x only)

Create a Rook Cluster

Monitoring Your Rook Cluster

Advanced Configuration

Debugging

Cluster Teardown

Components

The Rook Operator

Ceph Managers / Monitors / OSDs

Rook Agents

Storage

Custom Resource Definitions

Shared Storage Example

Prerequisites

Multiple File Systems Not Supported

Create the File System

Consume the Shared File System: Busybox + NGINX Example

Kernel Version Requirement

Testing Shared Storage

Under the Hood

Investigating Storage directories

Digging Deeper into dataDirHostPath

Checking the kubelet logs...

Hacking Terraform

Checking the rook-operator logs...

Now we're getting somewhere...

Narrowing it down...

Back to bluestore...

Edge Cases and Quirks

DO NOT delete the filesystem before shutting down the pods consuming it

You must follow these cleanup steps before terraform destroy will work

Back to bluestore...

Resolution

Recovering from backup

Edge Cases and Quirks

DO NOT delete the filesystem before shutting down all of the pods consuming it

You must follow these cleanup steps before terraform destroy will work

Kill (or hide) a hanging pod

Cleaning up failed runs of terraform destroy

Set the `dataDirHostPath`

Investigating Storage `directories`

Digging Deeper into `dataDirHostPath`

Checking the `kubelet` logs...

Checking the `rook-operator` logs...

You must follow these cleanup steps before `terraform destroy` will work

You must follow these cleanup steps before `terraform destroy` will work

Cleaning up failed runs of `terraform destroy`