Page History

...

8/3/2017 

Brainstorming: 

• Use case: 
  • HPC operator installs WB to provide:
    • Interactive analysis/development environments with access to shared data
    • Access to Hadoop and traditional HPC TORQUE/PBS clusterresources (headnode replacement)
  • WB is just 1+ nodes in the cluster with access to shared filesystem, authentication, etc.
  • User 
    • Develops
    User develops
    • in WB, creates and deploys containers (Docker
    container
    • , Singularity) to local registry (or Dockerhub)
    User writes
    • Writes PBS/qsub
    script
    • scripts and submits
    the
    • job from
    with
    • within WB,
    can
    •  can monitor progress. Similar for Hadoop.
    • Data is available to WB for analysis.
  What if WB replaces the head node in a traditional TORQUE or Hadoop cluster?
• Benefits:
  • User has one-stop-shop for development, exploratory analysis, visualization and data access that isn't the head node
  • HPC operator no longer needs to provide access to dedicated head node. Users can launch custom software without requiring additional installation.
• Essentially, users would have qsub/qdel/qstat all available in containers in WB instead of SSHing into the head node. 
  • We could do this easily via SCP/SSH (ssh clusterserver qsub /tmp/foo.sh), very easy if we share the home directory.
  • Could also develop (or leverage existing?) method for remote job submission API (Agave?)
  
  

...