Features of an Effective a Developer Environment

 Some Developer Use Cases

5 minutes to a running multi-node cluster that can run in your laptop.  VBox machines running CoreOS on host-local networking, not exposed to internet.  Notes below for other options.

Here is the recipe, adapted from the official instructions here: https://coreos.com/kubernetes/docs/latest/kubernetes-on-vagrant.html

  1. Install VirtualBox
  2. Install Vagrant
    1. Check vagrant: "vagrant box list"
  3. Install kubectl
    1. curl -O https://storage.googleapis.com/kubernetes-release/release/v1.5.2/bin/linux/amd64/kubectl
    2. install in /usr/local/bin or your favorite spot in your PATH
  4. git clone https://github.com/coreos/coreos-kubernetes.git
  5. cd multinode/vagrant
  6. vagrant box add http://alpha.release.core-os.net/amd64-usr/current/coreos_production_virtualbox.json
    1. will download CoreOS for virtualbox
  7. cp config.rb.sample to config.rb and edit:
    1. Pick the channel and node sizes and counts according to your laptop resources (primarily memory), here's one example:
  8. Good to check your networking for collisions on unrouted address ranges that you'll encounter at work, home, etc.
    1. The default VM network specs are in the Vagrantfile:  172.17.4/24
  9. vagrant up --provider virtualbox

  10. export KUBECONFIG="${KUBECONFIG}:$(pwd)/kubeconfig"

    1. Alt:   Add the config to any existing ~/.kube/config if you are working with multiple clusters and or namespaces (kubectl set-cluster, kubectl set-context, etc.)

  11. Should be operational, some useful commands (use in the multinode/vagrant path)

    1. vagrant ssh c1 (or the other names) to get into coreos

    2. vagrant status

    3. vagrant suspend/resume

      1. Once in a while net doesn't come back, and kubectl says:  

        The connection to the server was refused - did you specify the right host or port?

      2. In this case you can vagrant halt, vagrant up

    4. vagrant destroy (obliterates)

      1. Just vagrant up and restart afresh

Comparison to Hyperkube (container-based) local install

Advantages to local container options:    

HyperkubeSingle dependencyLowdocker install vs Vagrant and VirtualBox
 EfficiencyLow for development class systemsContainer-based is more eficient in terms of resources.  VM's are always overprovisioned
VagrantFamiliarity with processModerate

Use of VM's by developers is a common development model/process that's widely understood

Known familiar model is easy to adopt and faster to get started - more self-serving w.r.t. problems.

 Isolation between instances and from developer system:
better stability, reproducability, easier support

No entertwining with local docker environment - avoids widespread docker troubles/inconsistencies/breaking updates

easier cleanup/reinit of environments without local docker cleanups
ability to run more than one environment

Overloading Docker will not kill developers system - observed problem in NDS

Avoids potential collision of WB images and other local docker images unrelated to WB devenv.

 More accurate w.r.t. hosted and self-hosted production environmentsLow - App Development
High - Systems Development
True emulation of multi-machine deplolyment, true multi-network capabilities with routing,  true ingress without hostport collision, true operations emulation of machine failure/upgrade/maintenance, ability to work with node device-layer storage.
 FlexibilityLow - App Development
High - Systems Development

Changeout capabilities:   network layer, docker layer, OS layer

Can experiment with different OS and/or multiple OS on different machines  in single cluster

Complex developments can be exported/shared and disseminated for evaluation and/or group development purposes.  Accelerates development in comparison to the starting-from-ground-zero/replay model and allows more involved experimentation with complex environments.

Can support addition non-cluster vm's with tooling/ops ennvironments - like the gcloud shell

Supports low-level OS and networking debug tools that are

 SupportabilityVery High and potential non-value-add timesink

Uniform support through VM image gives full intentional control of configurations and upgrades - this is a huge ongoing issue in the k8s community for a reason (not simply an issue of opinionation) docker change has been an enormous support timesink - and NDS has direct experience on multiple occasions with this -  avoid docker update dependency if possible.

Self-test by devs having trouble:   stop the trouble env, start a new one to test basic functionality and A/B test to discover where things went wrong

State of devenv is tranferrable - can checkpoint the VMs and share the state with support to reproduce troublesome/important issues - yes semi large-files (but compare to jupyter pull?), but easy in comparison to trying to share a hyperkube environment.


  1. Can be used for volume/fs work by adding VBox disks to nodes.   Kubernetes doesn't have a VBox/vagrant volume driver (yet), so you get raw disk on nodes like we have in OpenStack.

  2. Can be used for ingress/network by adding additional interface to the node to be LB.

  3. It can be left running, or vagrant suspend/resume can freeze/thaw the whole cluster in a few seconds if you want it toiling around in the background.  Can also vagrant halt/up to shutdown/reboot and cluster state (etcd) is preserved.