With the advent of per-user Docker, we've got a few different options open here.

We are not limited to only one build scheme, but the more we offer, the more work it will be to keep it all in repair / in-sync when we want to make changes.

I am personally still on the fence about whether it would be beneficial or detrimental to give the user access to the Docker CLI, so I have foregone that as a possible pro / con in the following scenarios.

Solution A: Per-User Docker Sidecar

Summary: Run a single-build Docker Pod on the user's behalf using their personal docker instance/socketSee 

Example: 

• ndslabs-docker

Pros:

• lightweight - no service(s) required to run when no jobs are building
• User cannot access the raw Docker socket used by Kubernetes

Cons:

• No access to Docker CLI
• requires user to manually set up the build in order to integrate back into NDS Labs

Result: Ideal and seemingly low-effort solution

Solution B: Per-User Docker + NDS Labs Spec

Summary: Allow user to run and manage their own Jenkins instance using their personal docker instance/socket

See jenkins.json

Example: 

• jenkins.json

Pros:

• highly customizable build system
• rich plugin support for different languages, frameworks, and workflows
• could easily be enabled - assuming 

  Jira
  server JIRA serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key NDS-439

• User cannot access the raw Docker socket used by Kubernetes

Cons:

• CI is very complex - we don't want to be in the business of supporting / fixing Jenkins problems
• we would need to enforce auth, although the 2.7.1 build of jenkins used for the dind tag makes you set up an admin user on first-run

Result: I feel like this should be a service that we offer to host for them, but not necessarily force it on users or manage it for them 

Solution C: Centralized Internal CI

Summary: Run a centralized jenkins instance within Kubernetes to run Docker builds on all users' behalfSee 

Example: 

• service.yaml

...

• controller.yaml

Pros:

• complexity is hidden from the user
• no authentication is needed unless we wish to expose this to the public internet (we may need to tunnel in)
• User cannot access the raw Docker socket used by Kubernetes
  

Cons:

• would likely force tight coupling between our API and Jenkins'

Result: Tight Coupling => Not Ideal 

Solution D: Exo-Kubernetes CI

Summary: Spin up a raw VM on Nebula and run a jenkins instance on it outside of Kubernetes

Example: See 

• run-ci.sh

Pros:

• complexity is hidden from the user
• no authentication is needed unless we wish to expose this to the public internet (we may need to tunnel in)
• User cannot access the raw Docker socket used by Kubernetes

Cons:

• would likely force tight coupling between our API and Jenkins'

Result: Tight Coupling => Not Ideal

Solution E: Exo-Nebula CI

Summary: Use some CI server that does not reside on Nebula

...

• DockerHub
• TravisCI
• CircleCI
• etc

...

Example: 

• bodom0015/ndslabs-web-ui

Pros:

• we support this workflow right now

Cons:

• requires user to manually set up one of the following in order to integrate back into NDS Labs
  • automated build
  • Docker CLI (for manual pushes)

Result: No reason why we can't support this one, but it would be beneficial for us to offer alternatives