Signing Up

1. (Done) User/Developer signs up on BrownDog webpage
   • http://browndog.ncsa.illinois.edu/
2. User (Done) User receives email with more information and link to survey
3. (Automatic email) User fills out survey
4. (Done) User is contacted by Brown Dog developer and account is activated
5. (Not for April 4) Tools catalog support for single sign-in not for April 4 demo

Creating and Adding a New Tool

1. (Gregory Jansen, Rui Liu) Developer downloads BrownDog development environment in docker container (in dockerhub, link on BD webpage)

   Jira
   server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-956

   Jira
   server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-957

   Jira
   server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-958

   Jira
   server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-959

   Jira
   server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-960

   Jira
   server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-1009

   Jira
   server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-1017

   Jira
   server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-1007

   Jira
   server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-1016

   • Contains running Clowder instance, Polyglot instance, basic development tools, mabye even an IDE configured and ready to go
   • Contains a bare-bones extractor ready to be modified for a new tool (with comments suggesting where to edit) (Rui Liu has started this, in various languages e.g. python, C++, Java, ..., and docker sample file)
   • Contains a bare-bones converter ready to be modified for a new tool (with comments suggesting where to edit)
   • Formerly Rob's work. Gregory Jansen to drive out this step.
   • , in various languages e.g. bash, bat, ahk, python, ..., and docker sample file)
   • (Done) Polyglot changes from redirecting to SS to downloading and serving converted files to users' "/file/" requests, to support 1 Pol + multi SS docker containers on the same VM.
   • (Done - Rui provided the files/images.) docker-compose.yml file, docker images for Polyglot server + 2 different SS to work together on the same VM.
2. (Gregory Jansen, Smruti Padhy) Developer installs tool, creates BrownDog interface script(s)
   • Adds sample input file(s)
   • Example extractor: imagemagick edge detector
   • Example converter: imagemagick jpg to ico converter
3. (Gregory Jansen, Smruti Padhy and Inna Zharnitsky will assist Gregory Jansen here)Developer creates dockerfile
   • For extractors this should contain only the tool, its dependencies, and the extractor
   • For converters this should contain the basic Polyglot setup, ideally only the Software Server, the tool, its dependencies, and the converter interface script

   • Jira
     server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-983

4. (Rui Liu, Inna Zharnitsky) Developer Developer submits tool to Tools Catalog (using BrownDog account)
   

   Jira
   server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-961

   Jira
   server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-962

   
   
   • (Done) BrownDog interface Interface script(s), dockerfile, sample input, sample output
   • TODO: Documentation on how to add tool to Tools Catalog
   • (Done) Through web interface
   • Through script in development environment (probably the ideal way)

     Jira
     server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-962

5. (Inna Zharnitsky, Marcus Slavenas) Admin reviews and tests tool by pressing Admin presses button which deploys tool in a seperate separate Clowder/Polyglot sandbox, passes in the given input file, and checks the obtained output file against what is provided
   • Returns pass/fail
   • Inna Zharnitsky has started this UI, Marcus Slavenas to support
   1. Manual review source code and docker file
   2. If ok, reviewer submits tool for testing from tool catalog ui 

      Jira
      server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-968

      Jira
      server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-969

      Jira
      server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-1055

      1. Build docker image using local parameters
      2. If user provides sample output run script to test inputs/outputs
         1. if not, manual inspections is required
      3. Scripts show pass/fail in tool script
   • Admin click accept or reject
6. (Rui Liu, Inna Zharnitsky) Admin deploys tool by selecting instances to deploy interface on 

   Jira
   server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-916

   Jira
   server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-954

   Jira
   server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-733

   Admin selects instances to deploy interface on
   • e.g. dts-dev or dts
   • selects minimum number of instances to provision (e.g. 5 instances)
   • presses button to deploy
   • Elasticity backend:  deploy docker images

     Jira
     server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-890

     , deploy VM images

     Jira
     server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-930

     , stop

     Jira
     server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-941

     .
   • Tool Catalog UI: BD-733 (deploy), BD-954 (stop).
   • Task: integration testing after the web UI part is done.

     Jira
     server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-975

7. (Sandeep Puthanveetil Satheesan) Admin Admin receives status page showing number of tools deployed and where, load on each tool, etc...
   • No support for this at this time.
   • How to pull this from MongoDB
   • Sandeep Puthanveetil Satheesan to investigate

Using a Tool

• (Luigi Marini) API Gateway with keys and tokens (uses Crowd credentials)
  • (Not for April 4) Forwarding user credentials to Clowder
  • Clients
    • (Marcus Slavenas) Matlab 

      Jira
      server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-620

      • Working and includes test script: DAP Outputs, DAP Convert, DTS Extract, and Download File
      • Not yet translated from bd.py: index, find, descriptor distance
    • (Smruti Padhy) R – available
    • (Luigi Marini) bash script using curl 

      Jira
      server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-984

    • (Luigi Marini) Python – available bd-cli
    • (Smruti Padhy) Windows – Dap is there, for DTS not sure 

      Jira
      server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-1004

      Jira
      server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-1005

    • (Smruti Padhy) Bookmarklet - 
    • (Smruti Padhy) Chrome Extension
    • Add keys and tokens to all clients

      • Jira
        server JIRA serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-986

        Jira
        server JIRA serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-987

        Jira
        server JIRA serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-988

        Jira
        server JIRA serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-988

        Jira
        server JIRA serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-989

        Jira
        server JIRA serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-990

        Jira
        server JIRA serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-991

• (Jong Lee, Eugene Roeder) Simple web interface (requires BrownDog account) Luigi Marini 's API work applies here

  Jira
  server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-967

  Jira
  server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-970

  Jira
  server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key BD-976

  
  
  • Web form with field box for file URL or upload
  • Tool selection dropdown box (means of selecting a specific tool)
  • Output select dropdown box (means of allowing system to select tool)
    • Clowder will run all relevant extractors (obtained by selecting "json" as the output)
    • Polyglot will use I/O-graph
  • Suggetions Suggestions pane will update below
    • REST API performing this same operation
    • Python code in Jupyter lab notebook
    • R code
    • Matlab code
    • curl commands
  Everyone with a client will need to describe how this will apply (jupyter, R, etc)
  • • php