Table of Contents |
---|
Clowder Spec
Latest Stable Tag: 0.9.2
See https://github.com/nds-org/ndslabs-specs/tree/master/clowder
Required Docker Images
The following images will be automatically pulled from Docker Hub if they are not present on your machine:
Docker Image Build
If you would rather build up your own version from source, rather than pulling from Docker Hub:
- Clone the https://github.com/nds-org/ndslabs-clowder repository.
- Change directories to dockerfiles and run the ./build.sh command.
- You should see the images start building from the Dockerfiles present.
- Coming Soon: Core Extractors!
- audio-preview
- audio-speech2text
- image-metadata
- pdf-preview
WARNING: plantcv may take 10 - 25 minutes to complete its build. Plan accordingly.
Test Setup
Start Tool Server
- Start up a new toolserver using the following command:
- NOTE: Ensure that the image is tagged as ncsa/clowder-toolserver
- This is a manual step until the image gets tagged and pushed to Docker Hub.
Save the Public IP of the node where this is running.
Code Block language bash docker run -d -p 8082:8082 --name toolserver -v /var/run/docker.sock:/var/run/docker.sock ndslabs/toolserver:0.9.2 toolserver
- NOTE: Ensure that the image is tagged as ncsa/clowder-toolserver
Start NDSLabs
Start up NDSLabs as described below:
Run the NDSLabs System Shell:
Code Block language bash docker run -it --volume=/:/rootfs:ro --volume=/sys:/sys:ro --volume=/
TODO: Update this
NDSLabs Test Cases
Prerequisites
Start Tool Server
- Start up a new toolserver using the following command:
- NOTE: Ensure that the image is tagged as ncsa/clowder-toolserver
- This is a manual step until the image gets tagged and pushed to Docker Hub.
Save the Public IP of the node where this is running.
Code Block language bash docker run -d -p 8082:8082 --name toolserver -v /var/run/docker.sock:/var/run/docker.sock ndslabs/toolserver:terra toolserver
- NOTE: Ensure that the image is tagged as ncsa/clowder-toolserver
Start NDSLabs
Start up NDSLabs as described below:
Run the NDSLabs System Shell:
Code Block language bash docker run -it --volume=/:/rootfs:ro --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:rw --volume=/var/lib/kubelet/:/var/lib/kubelet:rw --volume=/var/run:/var/run:rw --net=host --pid=host --privileged=true -v /var/run/docker.sock:/var/run/docker.sock ndslabs/system-shell Unable to find image 'ndslabs/system-shell:latest' locally latest: Pulling from ndslabs/system-shell ff12aecbe22a: Already exists 287750ad6625: Already exists ca98bdf222fa: Already exists a3ed95caeb02: Already exists 97ef68d67ea6: Pull complete 8c53c989a967: Pull complete 79d911a06f41: Pull complete 807cecd8f466: Pull complete 7f887f3746f8: Pull complete 0cadab32de06: Pull complete aff97fd2a6c1: Pull complete Digest: sha256:4128fff8a0234ee6cc25d077b7f607358e681370e5a483b6c89fe1a3dfc3e77e Status: Downloaded newer image for ndslabs/system-shell:latest [root@default NDSLabsSystem ] / #
- From the NDSLabsSystem shell:
- Start Kubernetes by running kube-up.sh
- This will pull the necessary Kubernetes images and start up a single-node (development) Kubernetes cluster.
- Start NDSLabs by running ndslabs-up.sh
- This will start the API Server and GUI in Kubernetes
- NOTE: You may need to wait 30 seconds or more for the GUI server to start while npm and bower download the GUI's dependencies
- This will start the API Server and GUI in Kubernetes
- Check the API server logs and verify that the specs were loaded correctly:
Code Block language bash Cloning into '/specs'... I0325 01:52:58.670716 15 server.go:127] Starting NDS Labs API server (0.1alpha 2016-03-24 14:03) I0325 01:52:58.671129 15 server.go:128] etcd PRIVATE_IP:4001 I0325 01:52:58.671145 15 server.go:129] kube-apiserver https://PRIVATE_IP:6443 I0325 01:52:58.671159 15 server.go:130] volume dir /volumes I0325 01:52:58.671170 15 server.go:131] specs dir /specs I0325 01:52:58.671181 15 server.go:132] host PUBLIC_IP I0325 01:52:58.671192 15 server.go:133] port 8083 I0325 01:52:58.671205 15 server.go:134] V1 I0325 01:52:58.671216 15 server.go:135] V2 I0325 01:52:58.671227 15 server.go:136] V3 I0325 01:52:58.671237 15 server.go:137] V4 I0325 01:52:58.671257 15 server.go:675] GetEtcdClient PRIVATE_IP:4001 I0325 01:52:58.672692 15 server.go:165] Using local storage I0325 01:52:58.672746 15 server.go:175] CORS origin http://PUBLIC_IP:30000 I0325 01:52:58.673061 15 server.go:276] Loading service specs from /specs I0325 01:52:58.673910 15 server.go:1875] Adding /specs/clowder/clowder.json I0325 01:52:58.674621 15 server.go:1875] Adding /specs/clowder/elastic.json I0325 01:52:58.675383 15 server.go:1875] Adding /specs/clowder/extractors/image-preview.json I0325 01:52:58.675999 15 server.go:1875] Adding /specs/clowder/extractors/plantcv.json I0325 01:52:58.676513 15 server.go:1875] Adding /specs/clowder/extractors/video-preview.json I0325 01:52:58.677121 15 server.go:1875] Adding /specs/clowder/mongo.json I0325 01:52:58.677516 15 server.go:1875] Adding /specs/clowder/rabbitmq.json I0325 01:52:58.678111 15 server.go:1875] Adding /specs/dataverse/dataverse.json I0325 01:52:58.679353 15 server.go:1875] Adding /specs/dataverse/postgres.json I0325 01:52:58.679768 15 server.go:1875] Adding /specs/dataverse/rserve.json I0325 01:52:58.680253 15 server.go:1875] Adding /specs/dataverse/solr.json I0325 01:52:58.682642 15 server.go:1875] Adding /specs/dataverse/tworavens.json I0325 01:52:58.684168 15 server.go:1875] Adding /specs/elk/elastic.json I0325 01:52:58.684700 15 server.go:1875] Adding /specs/elk/kibana.json I0325 01:52:58.685192 15 server.go:1875] Adding /specs/elk/logspout.json I0325 01:52:58.686697 15 server.go:1875] Adding /specs/elk/logstash.json I0325 01:52:58.689697 15 server.go:1875] Adding /specs/irods/cloudbrowser.json I0325 01:52:58.691582 15 server.go:1875] Adding /specs/irods/cloudbrowserui.json I0325 01:52:58.692906 15 server.go:1875] Adding /specs/irods/icat.json I0325 01:52:58.693939 15 server.go:283] Listening on PUBLIC_IP:8083
- Start Kubernetes by running kube-up.sh
- You should now be able to reach the GUI by navigating to http://CLUSTER_IP:30000
- Create an account on the NDSLabs GUI as described below:
Test Cases
TERRA Clowder Configuration
...
- Verify that elasticsearch is enabled by navigating to Clowder's endpoint
- You should see elasticsearch: connected listed under the "plugins" section of http://YOUR_OPENSTACK_IP:30291/api/status
- You should see a "Search" box at the top-right of the Clowder UI. This indicates that elasticsearch is enabled.
- After uploading a file (as described above), attempt to search for the file extensions, such as "jpg" or "png".
- You should see any matching file(s) that you have uploaded listed under the results of the search.
Testing the Tool Server
- Navigate to the Dataset that you created above.
- On the right side of the page, you should see the Tool Manager section.
- Choose a tool (Jupyter / Rstudio) from the drop-down and press "Launch"
- Once the image downloads and the container starts (this may take several minutes):
- Rstudio:
- Navigate to and log into the Rstudio instance
- username: rstudio
- password: rstudio
- You should see the Dataset that you uploaded listed here
- Navigate to and log into the Rstudio instance
- Jupyter:
- Navigate to the Jupyter instance
- You should see the Dataset that you uploaded listed here
- Rstudio:
Clowder Docker Images
- Clone the https://github.com/nds-org/ndslabs-clowder repository.
- Change directories to dockerfiles.
- From dockerfiles, run the make command.
- You should see the images start building from the Dockerfiles present.
- Images that will be built include:
- clowder
- toolserver
- extractors:
- image-preview
- plantcv
- video-preview
- Coming Soon: New Extractors!
- audio-preview
- audio-speech2text
- image-metadata
- pdf-preview
...
- elasticsearch: connected listed under the "plugins" section of http://YOUR_OPENSTACK_IP:30291/api/status
- You should see a "Search" box at the top-right of the Clowder UI. This indicates that elasticsearch is enabled.
- After uploading a file (as described above), attempt to search for the file extensions, such as "jpg" or "png".
- You should see any matching file(s) that you have uploaded listed under the results of the search.
Testing the Tool Server
- Navigate to the Dataset that you created above.
- On the right side of the page, you should see the Tool Manager section.
- Choose a tool (Jupyter / Rstudio) from the drop-down and press "Launch"
- Once the image downloads and the container starts (this may take several minutes):
- Rstudio:
- Navigate to and log into the Rstudio instance
- username: rstudio
- password: rstudio
- You should see the Dataset that you uploaded listed here
- Navigate to and log into the Rstudio instance
- Jupyter:
- Navigate to the Jupyter instance
- You should see the Dataset that you uploaded listed here
- Rstudio:
Archived Test Cases
These test cases are kept for historical purposes and , but can be used to run and / test the Clowder stack in raw Kubernetes (without running NDSLabs).
Getting Started
- Start a Kubernetes cluster using kube-up.sh
- Ensure that kubectl has been downloaded and added to your PATH
- Clone the https://github.com/nds-org/ndslabs-clowder repository
- Change directories to kubernetes
- You are now ready to run the test cases described below
- Run . ./start-clowder.sh with no arguments to spin up a vanilla Clowder, with only a MongoDB instance attached.
- Navigate your browser to http://YOUR_OPENSTACK_IP:30291. You should see the Clowder homepage.
- Verify MongoDB attachment by navigating to http://YOUR_OPENSTACK_IP:30291/api/status.
- You should see mongodb: true listed under the "plugins" section.
...