Table of Contents

Clowder Spec

Latest Stable Tag: 0.9.2

See https://github.com/nds-org/ndslabs-specs/tree/master/clowder

Required Docker Images

The following images will be automatically pulled from Docker Hub if they are not present on your machine:

Docker Image Build

If you would rather build up your own version from source, rather than pulling from Docker Hub:

Clone the https://github.com/nds-org/ndslabs-clowder repository.
Change directories to dockerfiles and run the ./build.sh command.
You should see the images start building from the Dockerfiles present.
Coming Soon: Core Extractors!
- audio-preview
- audio-speech2text
- image-metadata
- pdf-preview

WARNING: plantcv may take 10 - 25 minutes to complete its build. Plan accordingly.

Test Setup

Start Tool Server

Start up a new toolserver using the following command:
- NOTE: Ensure that the image is tagged as ncsa/clowder-toolserver
  - This is a manual step until the image gets tagged and pushed to Docker Hub.
- Save the Public IP of the node where this is running.
  Code Block
  language bash
  docker run -d -p 8082:8082 --name toolserver

TODO: Update this

NDSLabs Test Cases

Prerequisites

Start Tool Server

Start up a new toolserver using the following command:
- NOTE: Ensure that the image is tagged as ncsa/clowder-toolserver
  - This is a manual step until the image gets tagged and pushed to Docker Hub.
- Save the Public IP of the node where this is running.
  Code Block
  language bash
  docker run -d -p 8082:8082 --name toolserver -v /var/run/docker.sock:/var/run/docker.sock ndslabs/toolserver:terra toolserver

Start NDSLabs

Start up NDSLabs as described below:

Run the NDSLabs System Shell:

...

language	bash

...

 -v /var/run/docker.sock:/var/run/docker.sock ndslabs/toolserver:0.9.2 toolserver

Start NDSLabs

Start up NDSLabs as described below:

Run the NDSLabs System Shell:

Code Block

language	bash

docker run  -it --volume=/:/rootfs:ro --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:rw --volume=/var/lib/kubelet/:/var/lib/kubelet:rw --volume=/var/run:/var/run:rw --net=host --pid=host --privileged=true -v /var/run/docker.sock:/var/run/docker.sock ndslabs/system-shell 
Unable to find image 'ndslabs/system-shell:latest' locally
latest: Pulling from ndslabs/system-shell
ff12aecbe22a: Already exists
287750ad6625: Already exists
ca98bdf222fa: Already exists
a3ed95caeb02: Already exists
97ef68d67ea6: Pull complete
8c53c989a967: Pull complete
79d911a06f41: Pull complete
807cecd8f466: Pull complete
7f887f3746f8: Pull complete
0cadab32de06: Pull complete
aff97fd2a6c1: Pull complete
Digest: sha256:4128fff8a0234ee6cc25d077b7f607358e681370e5a483b6c89fe1a3dfc3e77e
Status: Downloaded newer image for ndslabs/system-shell:latest
[root@default NDSLabsSystem ] / #

From the NDSLabsSystem shell:system-shell Unable to find image 'ndslabs/system-shell:latest' locally latest: Pulling from ndslabs/system-shell ff12aecbe22a: Already exists 287750ad6625: Already exists ca98bdf222fa: Already exists a3ed95caeb02: Already exists 97ef68d67ea6: Pull complete 8c53c989a967: Pull complete 79d911a06f41: Pull complete 807cecd8f466: Pull complete 7f887f3746f8: Pull complete 0cadab32de06: Pull complete aff97fd2a6c1: Pull complete Digest: sha256:4128fff8a0234ee6cc25d077b7f607358e681370e5a483b6c89fe1a3dfc3e77e Status: Downloaded newer image for ndslabs/system-shell:latest [root@default NDSLabsSystem ] / # From the NDSLabsSystem shell:

Start Kubernetes by running kube-up.sh
- This will pull the necessary Kubernetes images and start up a single-node (development) Kubernetes cluster.
Start NDSLabs by running ndslabs-up.sh
- This will start the API Server and GUI in Kubernetes
- NOTE: You may need to wait 30 seconds or more for the GUI server to start while npm and bower download the GUI's dependencies

Check the API server logs and verify that the specs were loaded correctly:

Code Block

language	bash

Cloning into '/specs'...
I0325 01:52:58.670716      15 server.go:127] Starting NDS Labs API server (0.1alpha 2016-03-24 14:03)
I0325 01:52:58.671129      15 server.go:128] etcd PRIVATE_IP:4001
I0325 01:52:58.671145      15 server.go:129] kube-apiserver https://PRIVATE_IP:6443
I0325 01:52:58.671159      15 server.go:130] volume dir /volumes
I0325 01:52:58.671170      15 server.go:131] specs dir /specs
I0325 01:52:58.671181      15 server.go:132] host PUBLIC_IP
I0325 01:52:58.671192      15 server.go:133] port 8083
I0325 01:52:58.671205      15 server.go:134] V1
I0325 01:52:58.671216      15 server.go:135] V2
I0325 01:52:58.671227      15 server.go:136] V3
I0325 01:52:58.671237      15 server.go:137] V4
I0325 01:52:58.671257      15 server.go:675] GetEtcdClient PRIVATE_IP:4001
I0325 01:52:58.672692      15 server.go:165] Using local storage
I0325 01:52:58.672746      15 server.go:175] CORS origin http://PUBLIC_IP:30000
I0325 01:52:58.673061      15 server.go:276] Loading service specs from /specs
I0325 01:52:58.673910      15 server.go:1875] Adding /specs/clowder/clowder.json
I0325 01:52:58.674621      15 server.go:1875] Adding /specs/clowder/elastic.json
I0325 01:52:58.675383      15 server.go:1875] Adding /specs/clowder/extractors/image-preview.json
I0325 01:52:58.675999      15 server.go:1875] Adding /specs/clowder/extractors/plantcv.json
I0325 01:52:58.676513      15 server.go:1875] Adding /specs/clowder/extractors/video-preview.json
I0325 01:52:58.677121      15 server.go:1875] Adding /specs/clowder/mongo.json
I0325 01:52:58.677516      15 server.go:1875] Adding /specs/clowder/rabbitmq.json
I0325 01:52:58.678111      15 server.go:1875] Adding /specs/dataverse/dataverse.json
I0325 01:52:58.679353      15 server.go:1875] Adding /specs/dataverse/postgres.json
I0325 01:52:58.679768      15 server.go:1875] Adding /specs/dataverse/rserve.json
I0325 01:52:58.680253      15 server.go:1875] Adding /specs/dataverse/solr.json
I0325 01:52:58.682642      15 server.go:1875] Adding /specs/dataverse/tworavens.json
I0325 01:52:58.684168      15 server.go:1875] Adding /specs/elk/elastic.json
I0325 01:52:58.684700      15 server.go:1875] Adding /specs/elk/kibana.json
I0325 01:52:58.685192      15 server.go:1875] Adding /specs/elk/logspout.json
I0325 01:52:58.686697      15 server.go:1875] Adding /specs/elk/logstash.json
I0325 01:52:58.689697      15 server.go:1875] Adding /specs/irods/cloudbrowser.json
I0325 01:52:58.691582      15 server.go:1875] Adding /specs/irods/cloudbrowserui.json
I0325 01:52:58.692906      15 server.go:1875] Adding /specs/irods/icat.json
I0325 01:52:58.693939      15 server.go:283] Listening on PUBLIC_IP:8083

You should now be able to reach the GUI by navigating to http://CLUSTER_IP:30000
Create an account on the NDSLabs GUI as described below:
1. GUI Test Cases: Create an Account

Test Cases

TERRA Clowder Configuration

...

From the NDSLabs Dashboard, Click "View Logs" next to the Video Preview Extractor

The logs should will show that this extractor has ignored this file upload, since it is not a video file:

Code Block

language	bash

2016-03-27 00:40:30,299 INFO    : pika.adapters.base_connection - Connecting to 10.0.0.56:5672
2016-03-27 00:40:30,303 INFO    : pika.adapters.blocking_connection - Created channel=1
2016-03-27 00:40:30,320 INFO    : pyclowder.extractors - Waiting for messages. To exit press CTRL+C

Testing Text-Based Search (ElasticSearch)

Verify that elasticsearch is enabled by navigating to Clowder's endpoint
- You should see elasticsearch: connected listed under the "plugins" section of http://YOUR_OPENSTACK_IP:30291/api/status
- You should see a "Search" box at the top-right of the Clowder UI. This indicates that elasticsearch is enabled.
After uploading a file (as described above), attempt to search for the file extensions, such as "jpg" or "png".
- You should see any matching file(s) that you have uploaded listed under the results of the search.

Testing the Tool Server

Navigate to the Dataset that you created above.
On the right side of the page, you should see the Tool Manager section.
Choose a tool (Jupyter / Rstudio) from the drop-down and press "Launch"
Once the image downloads and the container starts (this may take several minutes):
- Rstudio:
  - Navigate to and log into the Rstudio instance
    - username: rstudio
    - password: rstudio
  - You should see the Dataset that you uploaded listed here
- Jupyter:
  - Navigate to the Jupyter instance
  - You should see the Dataset that you uploaded listed here

Clowder Docker Images

Docker Image Build

Clone the https://github.com/nds-org/ndslabs-clowder repository.
Change directories to dockerfiles.
From dockerfiles, run the make command.
You should see the images start building from the Dockerfiles present.
Images that will be built include:
- clowder
- toolserver
- extractors:
  - audio-preview
  - audio-speech2text
  - image-metadata
  - image-preview
  - pdf-preview
  - plantcv
  - video-preview

...

- CTRL+C

Testing Text-Based Search (ElasticSearch)

Verify that elasticsearch is enabled by navigating to Clowder's endpoint
- You should see elasticsearch: connected listed under the "plugins" section of http://YOUR_OPENSTACK_IP:30291/api/status
- You should see a "Search" box at the top-right of the Clowder UI. This indicates that elasticsearch is enabled.
After uploading a file (as described above), attempt to search for the file extensions, such as "jpg" or "png".
- You should see any matching file(s) that you have uploaded listed under the results of the search.

Testing the Tool Server

Navigate to the Dataset that you created above.
On the right side of the page, you should see the Tool Manager section.
Choose a tool (Jupyter / Rstudio) from the drop-down and press "Launch"
Once the image downloads and the container starts (this may take several minutes):
- Rstudio:
  - Navigate to and log into the Rstudio instance
    - username: rstudio
    - password: rstudio
  - You should see the Dataset that you uploaded listed here
- Jupyter:
  - Navigate to the Jupyter instance
  - You should see the Dataset that you uploaded listed here

Archived Test Cases

These test cases are kept for historical purposes and , but can be used to run and / test the Clowder stack in raw Kubernetes (without running NDSLabs).

Getting Started

Start a Kubernetes cluster using kube-up.sh
- Ensure that kubectl has been downloaded and added to your PATH
Clone the https://github.com/nds-org/ndslabs-clowder repository
Change directories to kubernetes
You are now ready to run the test cases described below

Basic Clowder Startup

Run . ./start-clowder.sh with no arguments to spin up a vanilla Clowder, with only a MongoDB instance attached.
Navigate your browser to http://YOUR_OPENSTACK_IP:30291. You should see the Clowder homepage.
Verify MongoDB attachment by navigating to http://YOUR_OPENSTACK_IP:30291/api/status.
- You should see mongodb: true listed under the "plugins" section.

...

Space shortcuts

Page tree

Versions Compared

Old Version 9

New Version Current

Key

Clowder Spec

Required Docker Images

Docker Image Build

Test Setup

Start Tool Server

NDSLabs Test Cases

Prerequisites

Start Tool Server

Start NDSLabs

Start NDSLabs

Test Cases

TERRA Clowder Configuration

Testing Text-Based Search (ElasticSearch)

Testing the Tool Server

Clowder Docker Images

Testing Text-Based Search (ElasticSearch)

Testing the Tool Server

Archived Test Cases

Getting Started

Space shortcuts

Page tree

Page History

Versions Compared

Old Version 9

New Version Current

Key

Clowder Spec

Required Docker Images

Docker Image Build

Test Setup

Start Tool Server

NDSLabs Test Cases

Prerequisites

Start Tool Server

Start NDSLabs

Start NDSLabs

Test Cases

TERRA Clowder Configuration

Testing Text-Based Search (ElasticSearch)

Testing the Tool Server

Clowder Docker Images

Testing Text-Based Search (ElasticSearch)

Testing the Tool Server

Archived Test Cases

Getting Started