You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 8 Next »

TODO: Update this

NDSLabs Test Cases

Prerequisites

Start Tool Server

  • Start up a new toolserver using the following command:

    • Save the Public IP of the node where this is running.

      docker run -d -p 8082:8082 --name toolserver -v /var/run/docker.sock:/var/run/docker.sock ndslabs/toolserver:terra toolserver

Start NDSLabs

  1. Start up NDSLabs as described below:

    • Run the NDSLabs System Shell:

      docker run  -it --volume=/:/rootfs:ro --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:rw --volume=/var/lib/kubelet/:/var/lib/kubelet:rw --volume=/var/run:/var/run:rw --net=host --pid=host --privileged=true -v /var/run/docker.sock:/var/run/docker.sock ndslabs/system-shell 
Unable to find image 'ndslabs/system-shell:latest' locally
latest: Pulling from ndslabs/system-shell
ff12aecbe22a: Already exists
287750ad6625: Already exists
ca98bdf222fa: Already exists
a3ed95caeb02: Already exists
97ef68d67ea6: Pull complete
8c53c989a967: Pull complete
79d911a06f41: Pull complete
807cecd8f466: Pull complete
7f887f3746f8: Pull complete
0cadab32de06: Pull complete
aff97fd2a6c1: Pull complete
Digest: sha256:4128fff8a0234ee6cc25d077b7f607358e681370e5a483b6c89fe1a3dfc3e77e
Status: Downloaded newer image for ndslabs/system-shell:latest
[root@default NDSLabsSystem ] / #
  2. From the NDSLabsSystem shell:
    • Start Kubernetes by running kube-up.sh
    • Start NDSLabs by running ndslabs-up.sh
      • This will start the API Server and GUI in Kubernetes
      • NOTE: You may need to wait 30 seconds or more for the GUI server to start while npm and bower download the GUI's dependencies
    • Check the API server logs and verify that the specs were loaded correctly:
      • Cloning into '/specs'...
I0325 01:52:58.670716      15 server.go:127] Starting NDS Labs API server (0.1alpha 2016-03-24 14:03)
I0325 01:52:58.671129      15 server.go:128] etcd PRIVATE_IP:4001
I0325 01:52:58.671145      15 server.go:129] kube-apiserver https://PRIVATE_IP:6443
I0325 01:52:58.671159      15 server.go:130] volume dir /volumes
I0325 01:52:58.671170      15 server.go:131] specs dir /specs
I0325 01:52:58.671181      15 server.go:132] host PUBLIC_IP
I0325 01:52:58.671192      15 server.go:133] port 8083
I0325 01:52:58.671205      15 server.go:134] V1
I0325 01:52:58.671216      15 server.go:135] V2
I0325 01:52:58.671227      15 server.go:136] V3
I0325 01:52:58.671237      15 server.go:137] V4
I0325 01:52:58.671257      15 server.go:675] GetEtcdClient PRIVATE_IP:4001
I0325 01:52:58.672692      15 server.go:165] Using local storage
I0325 01:52:58.672746      15 server.go:175] CORS origin http://PUBLIC_IP:30000
I0325 01:52:58.673061      15 server.go:276] Loading service specs from /specs
I0325 01:52:58.673910      15 server.go:1875] Adding /specs/clowder/clowder.json
I0325 01:52:58.674621      15 server.go:1875] Adding /specs/clowder/elastic.json
I0325 01:52:58.675383      15 server.go:1875] Adding /specs/clowder/extractors/image-preview.json
I0325 01:52:58.675999      15 server.go:1875] Adding /specs/clowder/extractors/plantcv.json
I0325 01:52:58.676513      15 server.go:1875] Adding /specs/clowder/extractors/video-preview.json
I0325 01:52:58.677121      15 server.go:1875] Adding /specs/clowder/mongo.json
I0325 01:52:58.677516      15 server.go:1875] Adding /specs/clowder/rabbitmq.json
I0325 01:52:58.678111      15 server.go:1875] Adding /specs/dataverse/dataverse.json
I0325 01:52:58.679353      15 server.go:1875] Adding /specs/dataverse/postgres.json
I0325 01:52:58.679768      15 server.go:1875] Adding /specs/dataverse/rserve.json
I0325 01:52:58.680253      15 server.go:1875] Adding /specs/dataverse/solr.json
I0325 01:52:58.682642      15 server.go:1875] Adding /specs/dataverse/tworavens.json
I0325 01:52:58.684168      15 server.go:1875] Adding /specs/elk/elastic.json
I0325 01:52:58.684700      15 server.go:1875] Adding /specs/elk/kibana.json
I0325 01:52:58.685192      15 server.go:1875] Adding /specs/elk/logspout.json
I0325 01:52:58.686697      15 server.go:1875] Adding /specs/elk/logstash.json
I0325 01:52:58.689697      15 server.go:1875] Adding /specs/irods/cloudbrowser.json
I0325 01:52:58.691582      15 server.go:1875] Adding /specs/irods/cloudbrowserui.json
I0325 01:52:58.692906      15 server.go:1875] Adding /specs/irods/icat.json
I0325 01:52:58.693939      15 server.go:283] Listening on PUBLIC_IP:8083
  3. You should now be able to reach the GUI by navigating to http://CLUSTER_IP:30000
  4. Create an account on the NDSLabs GUI as described below:
    1. GUI Test Cases: Create an Account

TERRA Clowder Configuration 

Account Registration

  • Start Clowder (as described above) and navigate to its endpoint link
  • At the top right of the page, click Login and then choose Sign Up on the bottom of the panel.
  • Enter your e-mail address in the box and press Submit.
    • You should receive an e-mail with a link to confirm your account registration
  • Click the link in the e-mail to be brought back to Clowder.
  • Enter your First/Last name, enter/confirm your desired password, then click Submit.
  • You should now be able to log in with the credentials that you have entered (email / password).

Create a Dataset / Upload a File

  • After registering for an account (see above), create a new dataset by choosing Datasets > Create from the navbar at the top of the page.
  • Choose a picture file to upload to this dataset. The contents of the picture do not matter.
  • After choosing Start Upload, check the logs of the mongo container and you should see the uploaded files being added to the database

Extractor(s)

Now that you've seen the basic setup, let's try something a little more complex:

  • Once Clowder starts, verify that the extractors are present by navigating to http://YOUR_OPENSTACK_IP:30291/api/status
    • You should see rabbitmq: true listed under the "plugins" section.
    • You should see the extractors you specified listed at the bottom
  • After uploading a file, as described above:
    • View http://CLOWDER_IP/admin/extractions in your browser to verify that the extractors are working
    • You should also be able to see per-image extraction events listed under each image in a dataset
  • From the NDSLabs Dashboard, Click "View Logs" next to the PlantCV Extractor
    • The logs should show the extractor reading the image and attempting to attach metadata
    • Insert snippet
  • From the NDSLabs Dashboard, Click "View Logs" next to the Image Preview Extractor
    • The logs should show the extractor reading the image and attempting to create a thumbnail from it
    • Insert snippet
  • From the NDSLabs Dashboard, Click "View Logs" next to the Video Preview Extractor
    • The logs should show the extractor reading the image, realize it is a video and not an image, and skip processing the file
    • Insert snippet

Text-Based Search (ElasticSearch)

  • Once Clowder starts, verify that elasticsearch is enabled by navigating to its endpoint
    • You should see elasticsearch: true listed under the "plugins" section.
    • You should see a "Search" box at the top-right of the Clowder UI. This indicates that elasticsearch is enabled.
  • After uploading a file (as described above), attempt to search for the file extensions, such as "jpg" or "png".
    • You should see the file that you uploaded listed under the results of the search.

Clowder Docker Images

Docker Image Build

  • Clone the https://github.com/nds-org/ndslabs-clowder repository.
  • Change directories to dockerfiles.
  • From dockerfiles, run the make command.
  • You should see the images start building from the Dockerfiles present.
  • Images that will be built include:
    • python-base
    • clowder
    • image-preview
    • video-preview
    • plantcv

 

WARNING: plantcv may take up to 25 minutes to complete its build. Plan accodingly.

Archived Test Cases

These test cases are kept for historical purposes and can be used to run and test the Clowder stack in raw Kubernetes (without NDSLabs).

Basic Clowder Startup 

  • Run . ./start-clowder.sh with no arguments to spin up a vanilla Clowder, with only a MongoDB instance attached.
  • Navigate your browser to http://YOUR_OPENSTACK_IP:30291. You should see the Clowder homepage.
  • Verify MongoDB attachment by navigating to http://YOUR_OPENSTACK_IP:30291/api/status.
    • You should see mongodb: true listed under the "plugins" section.

Account Registration

  • Start Clowder (as described above)
  • At the top right of the page, click Login and then choose Sign Up on the bottom of the panel.
  • Enter your e-mail address in the box and press Submit.
    • You should receive an e-mail with a link to confirm your account registration
  • Click the link in the e-mail to be brought back to Clowder.
  • Enter your First/Last name, enter/confirm your desired password, then click Submit.
  • You should now be able to log in with the credentials that you have entered (email / password).

Create a Dataset / Upload a File

  • After registering for an account (see above), create a new dataset by choosing Datasets > Create from the navbar at the top of the page.
  • Choose a picture file to upload to this dataset. The contents of the picture do not matter.
  • After choosing Start Upload, check the logs of the mongo container and you should see

Extractor(s)

Now that you've seen the basic setup, let's try something a little more complex:

  • Stop any running Clowder / plugin instances: . ./stop-clowder.sh -m
  • Restart Clowder with some extractors: . ./start-clowder.sh -w image-preview plantcv video-preview
    • The script should automatically start RabbitMQ for you as well, since you have specified that you would like to utilize extractors.
  • Wait for everything to finish starting everything up (this may take up to ~1 minute)
  • Once Clowder starts, verify that the extractors are present by navigating tohttp://YOUR_OPENSTACK_IP:30291/api/status
    • You should see rabbitmq: true listed under the "plugins" section.
    • You should see the extractors you specified listed at the bottom
  • Create a Dataset and upload a file as described above.
    • View http://CLOWDER_IP/admin/extractions in your browser to verify that the extractors are working.
    • If anything strange appears on the UI, check the log(s) of each extractor and you should see it doing work on the file(s) you chose to upload

Text-Based Search (ElasticSearch)

  • Stop any running Clowder / plugin instances: . ./stop-clowder.sh -m
  • Restart Clowder with elasticsearch enabled: . ./start-clowder.sh elasticsearch
  • Once Clowder starts, verify that elasticsearch is enabled by navigating tohttp://YOUR_OPENSTACK_IP:30291/api/status
    • You should see elasticsearch: true listed under the "plugins" section.
    • You should see a "Search" box at the top-right of the Clowder UI. This indicates that elasticsearch is enabled.
  • After uploading a file (as described above), attempt to search for the file extensions, such as "jpg" or "png".
    • You should see the file that you uploaded listed under the results of the search.

Start up NDSLabs as described below:

  • Run the NDSLabs System Shell:

    docker run  -it --volume=/:/rootfs:ro --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:rw --volume=/var/lib/kubelet/:/var/lib/kubelet:rw --volume=/var/run:/var/run:rw --net=host --pid=host --privileged=true -v /var/run/docker.sock:/var/run/docker.sock ndslabs/system-shell 
Unable to find image 'ndslabs/system-shell:latest' locally
latest: Pulling from ndslabs/system-shell
ff12aecbe22a: Already exists
287750ad6625: Already exists
ca98bdf222fa: Already exists
a3ed95caeb02: Already exists
97ef68d67ea6: Pull complete
8c53c989a967: Pull complete
79d911a06f41: Pull complete
807cecd8f466: Pull complete
7f887f3746f8: Pull complete
0cadab32de06: Pull complete
aff97fd2a6c1: Pull complete
Digest: sha256:4128fff8a0234ee6cc25d077b7f607358e681370e5a483b6c89fe1a3dfc3e77e
Status: Downloaded newer image for ndslabs/system-shell:latest
[root@default NDSLabsSystem ] / #
  • From the System Shell, start Kubernetes
  • No labels