Thoughts on generalizing workbench as Project X based on recent discussions.
...
what we currently call "Labs Workbench" as a general platform that can be used to support multiple distinct use cases.
Potential Use Cases
NDS Labs Workbench
This is the existing primary use case. NDS Labs Workbench powered by "Project X".
Education and training
One of the clearest proven uses of the platform is for education and training purposes. Labs Workbench was used for:
- IASSIST 2016 for a workshop on integrating Dataverse and iRODS (30 users)
- NDSC6 for a workshop for the development of Docker containers (20 users)
- Phenome 2017 for a workshop on how to use the TERRA-REF reference data platform and tools (40+ users)
- Planned iSchool pilot for data curation educators
- Possible deployment by Big Data Hub on commercial provider, such as Microsoft Azure, AWS, GCE to provide sandbox
Each environment is unique, but there are a few basic requirements:
- Custom catalog only of the tools needed for the training environment
- User accounts that can be created with and without requiring registration (e.g., batch import).
- Authentication that ties to existing systems (e.g., Shibboleth, Oauth)
- Short term scalable resources (e.g. 40 users, 4 hours) as well as longer-term stable resources (11 weeks, 24x7, maintenance allowed)
- Custom documentation and branding/skinning
- Custom data, API keys, etc accessible by users. For example, Phenome 2017 wanted sample data for each user and pre-defined API keys.
- Configurable quotas (not one-size fits all)
- Ability to deploy a dedicated environment, scale it up, and tear it down. At the end of the workshop/semester, access can be revoked.
- Ability to backup/download data
- Ability to deploy system under a variety of architectures (commercial, local, OpenStack, etc)
- Ability to host/manage system at NDSC/SDSC/TACC
- Security/TLS/vulnerability assessmentmanagement
- Monitoring
Scalable analysis environment
We can also envision the platform working as a replacement for the TERRA-REF toolserver or as a DataDNS analysis service. In this case, the requirements areinclude:
- Custom catalog of tools supported for the environment .as well as user-defined tools
- User accounts that can be created without requiring registration (API). For example, shared auth with Clowder for TERRA-REF.
- Authentication that ties to existing systems (e.g., Shibboleth, Oauth)
- Long-term stable and scalable resources. Ability to add/remove nodes as needed.
- Ability to terminate long-running containers to reclaim resources
- Custom documentation and branding, although the UI itself may be optional
- Ability to mount data stored on remote systems (e.g., ROGER) as read-only and possibly read-write scratch space
- Ability to add data to a running container, retrieved from a remote system?
- Clear REST API to
- List tools; list environments for a user; launch tools; stop tools;
- Security/TLS/vulnerability assessment
...
The Angular Web UI includes a facility for executing automated Selenium smoke tests.
What would need to change?
- Deployment process needs to be generalized to support more environments than OpenStack and likely more OSes than CoreOS
- Volume/storage will not work on cloud providers
- Potentially allow for Web UI customization
- Better custom catalog support
- Confirm ingress (including DNS/TLS) support with Commercial providers