Clowder Spec

Latest Stable Tag: 0.9.2

See https://github.com/nds-org/ndslabs-specs/tree/master/clowder

Test Setup

Start Tool Server

Required Docker Images

The following images will be automatically pulled from Docker Hub if they are not present on your machine:

• clowder
• toolserver
• extractors:
  • image-preview
  • plantcv
  • video-preview

Docker Image Build

If you would rather build up your own version from source, rather than pulling from Docker Hub:

• Clone the https://github.com/nds-org/ndslabs-clowder repository.
• Change directories to dockerfiles and run the ./build.sh command.
• You should see the images start building from the Dockerfiles present.
• Coming Soon: Core Extractors!
  • audio-preview
  • audio-speech2text
  • image-metadata
  • pdf-preview

WARNING: plantcv may take 10 - 25 minutes to complete its build. Plan accordingly.

Test Setup

Start Tool Server

• Start up a new toolserver using the following command:
  • NOTE: Ensure that the image is tagged as ncsa/clowder-toolserver
    • This is a manual step until the image gets tagged and pushed to Docker Hub.
  Start up a new toolserver using the following command:
  • NOTE: Ensure that the image is tagged as ncsa/clowder-toolserver
    • This is a manual step until the image gets tagged and pushed to Docker Hub.

  • Save the Public IP of the node where this is running.

    Code Block
    language bash
    docker run -d -p 8082:8082 --name toolserver -v /var/run/docker.sock:/var/run/docker.sock ndslabs/toolserver:0.9.2 toolserver

...

• Verify that elasticsearch is enabled by navigating to Clowder's endpoint
  • You should see elasticsearch: connected listed under the "plugins" section of http://YOUR_OPENSTACK_IP:30291/api/status
  • You should see a "Search" box at the top-right of the Clowder UI. This indicates that elasticsearch is enabled.
• After uploading a file (as described above), attempt to search for the file extensions, such as "jpg" or "png".
  • You should see any matching file(s) that you have uploaded listed under the results of the search.

Testing the Tool Server

• Navigate to the Dataset that you created above.
• On the right side of the page, you should see the Tool Manager section.
• Choose a tool (Jupyter / Rstudio) from the drop-down and press "Launch"
• Once the image downloads and the container starts (this may take several minutes):
  • Rstudio:
    • Navigate to and log into the Rstudio instance
      • username: rstudio
      • password: rstudio
    • You should see the Dataset that you uploaded listed here
  • Jupyter:
    • Navigate to the Jupyter instance
    • You should see the Dataset that you uploaded listed here

Clowder Docker Images

Docker Image Build

• Clone the https://github.com/nds-org/ndslabs-clowder repository.
• Change directories to dockerfiles.
• From dockerfiles, run the make command.
• You should see the images start building from the Dockerfiles present.
• Images that will be built include:
  • clowder
  • toolserver
  • extractors:
    • image-preview
    • plantcv
    • video-preview
• Coming Soon: Core Extractors!
  • audio-preview
  • audio-speech2text
  • image-metadata
  • pdf-preview

...

• • elasticsearch: connected listed under the "plugins" section of http://YOUR_OPENSTACK_IP:30291/api/status
  • You should see a "Search" box at the top-right of the Clowder UI. This indicates that elasticsearch is enabled.
• After uploading a file (as described above), attempt to search for the file extensions, such as "jpg" or "png".
  • You should see any matching file(s) that you have uploaded listed under the results of the search.

Testing the Tool Server

• Navigate to the Dataset that you created above.
• On the right side of the page, you should see the Tool Manager section.
• Choose a tool (Jupyter / Rstudio) from the drop-down and press "Launch"
• Once the image downloads and the container starts (this may take several minutes):
  • Rstudio:
    • Navigate to and log into the Rstudio instance
      • username: rstudio
      • password: rstudio
    • You should see the Dataset that you uploaded listed here
  • Jupyter:
    • Navigate to the Jupyter instance
    • You should see the Dataset that you uploaded listed here

Archived Test Cases

These test cases are kept for historical purposes and , but can be used to run and / test the Clowder stack in raw Kubernetes (without running NDSLabs).

Getting Started

• Start a Kubernetes cluster using kube-up.sh
  • Ensure that kubectl has been downloaded and added to your PATH
• Clone the https://github.com/nds-org/ndslabs-clowder repository
• Change directories to kubernetes
• You are now ready to run the test cases described below

Basic Clowder Startup 

• Run . ./start-clowder.sh with no arguments to spin up a vanilla Clowder, with only a MongoDB instance attached.
• Navigate your browser to http://YOUR_OPENSTACK_IP:30291. You should see the Clowder homepage.
• Verify MongoDB attachment by navigating to http://YOUR_OPENSTACK_IP:30291/api/status.
  • You should see mongodb: true listed under the "plugins" section.

...