8/31/2017
From 2017-08-31 Workbench planning, we'll consider the following use cases:
- Running ETK on Comet from WB
- TERRA-REF, either stitching or Genomics workflows
- Other: TERRA Fusion, LSST
8/3/2017
Brainstorming:
- Use case:
- HPC operator installs WB to provide:
- Interactive analysis/development environments with access to shared data
- Access to Hadoop and traditional TORQUE/PBS resources (headnode replacement)
- WB is just 1+ nodes in the cluster with access to shared filesystem, authentication, etc.
- User
- Develops in WB, creates and deploys containers (Docker, Singularity) to local registry (or Dockerhub)
- Writes PBS/qsub scripts and submits job from within WB, can monitor progress. Similar for Hadoop.
- Data is available to WB for analysis.
- HPC operator installs WB to provide:
- Benefits:
- User has one-stop-shop for development, exploratory analysis, visualization and data access that isn't the head node
- HPC operator no longer needs to provide access to dedicated head node. Users can launch custom software without requiring additional installation.
- HPC operator can easily support running workshops/tutorials etc.
- Essentially, users would have qsub/qdel/qstat all available in containers in WB instead of SSHing into the head node.
- We could do this easily via SCP/SSH (ssh clusterserver qsub /tmp/foo.sh), very easy if we share the home directory.
- Could also develop (or leverage existing?) method for remote job submission API (Agave?)
...