Thoughts on developer and single-node installation as per Locally-managed workbench.

Use case

The primary use case is for a developer to run a local copy of Labs Workbench on their laptop or a remote VM (AWS, GCE, Azure, OpenStack, etc).  The basic requirements are simple: 

• Bring up a (virtual) Kubernetes cluster 
• Deploy Labs Workbench
• Ideally, process should mimic deployment in other environments

Current environment

For developers, we achieve this with the ndslabs-startup repo. A simple shell script  kube-up.sh brings up Kubernetes via Hyperkube.  This requires access to /var/lib/docker and /var/lib/kubelet, which has proven problematic on MacOS and Windows with the newer Docker installs.  A second shell script ndslabs-up.sh collects information from the user (domain name, IP address, email address) and uses kubectl to deploy a number of templated YAML files.  

Pros:

• Simple, scripted process gets things running quickly

Cons:

• Doesn't currently work on MacOS/Windows
• Repo has unused stuff
• "Mustache" variable replacement is ridiculous
• Doesn't include support for DNS (can be added easily)
• Templates duplicated in deploy-tools repository

Options:

• Minikube (works on MacOS, Windows, Linux – doesn't require Docker; doesn't work on VM)
• Hyperkube (works well on VMs running Docker; doesn't work well on MacOS/Windows due to filesystem permission problems)
• Vagrant (works on MacOS)
• Deploy-tools

Minikube

Minikube runs Kubernetes in a single VM.

• Requires virtualization environment, like VirtualBox
• Download binary
• minikube start creates a VM in VirtualBox (or other virtualization environment)
• Sets up kubeconfig and local environment to access VM via kubectl

MacOS

• Install VirtualBox
• Download binary
• minikube start
• git clone ndslabs-up.sh
• Don't need kube-up.sh, since we're running Minikube
• ndslabs-up.sh 
  • Local DNS via bind works!
  • IP detection, etcd, etc, won't work as-is
  • Labeling won't work as is.

Windows 10

• Install VirtualBox (can also use Hyper-V, but not tested here)
  • Disable Hyper-V: Settings > Turn Windows Features on or off; uncheck Hyper-V
• Download minikube for Windows
• minikube.exe start
  • Will make changes to VirtualBox
• Works well on Windows

Linux/VM

• Minikube will work on Linux, but so far we haven't been able to use it on a VM (e.g., OpenStack). Hyperkube works for this.

Hyperkube

Our current ndslabs-startup/kube-up.sh relies on Hyperkube (Kubernetes in Docker)

• This works well on OpenStack VMs
• Doesn't work well for Docker on Windows/MacOS
  
  
• Our current ndslabs-startup/kube-up relies on Hyperkube, which is basically Kubernetes in Docker. 
  • gcr.io/google_containers/hyperkube-amd64
• Hyperkube works fine on CoreOS boxes in OpenStack, but has proven problematic under Docker for Mac and Docker for Windows due to permissions problems.
• The CoreOS project provides a process for launching Kubernetes on CoreOS, using another hyperkube image
  • https://coreos.com/kubernetes/docs/latest/kubernetes-on-generic-platforms.html
  • Uses quay.io/coreos/hyperkube image

Vagrant

Vagrant is a tool to manage virtual machines. The approach described below comes from CoreOS as a method to deploy a Kubernetes cluster via VirtualBox (or similar). Vagrant has multiple providers, so it might be possible to use the same Vagrant configuration to deploy a Kubernetes cluster on VirtualBox on your laptop and deploy a full cluster on OpenStack. However, it's not clear how useful this might be.

MacOS

The VirtualBox/Vagrant approach suggested by David works on MacOS. As with Minikube, this will not work on an OpenStack or other VM without running a different virtualization package (VirtualBox won't work in a VM). It's not clear that Vagrant will support this configuration.

Following instructions from Multi Node Kubernetes developer environment - using VBox/Vagrant/CoreOS

1. Install Vagrant
   1. vagrant version

   2. Got error "failed generating SSL artifacts", solved with https://github.com/mitchellh/vagrant/issues/7747

      sudo ln -sf /usr/local/bin/openssl /opt/vagrant/embedded/bin/openssl

2. git clone https://github.com/coreos/coreos-kubernetes.git
3. cd coreos-kubernetes/multi-node/vagrant/
4. cp config.rb.sample config.rb, uncomment values
   1. Want 2GB for compute for us
5. vagrant up --provider virtualbox
6. vagrant status
   1. Should show three instances e1 (etcd), w1 (compute), c1 (master/controller)
   2. vagrant ssh c1 will take you into the controller node, will need to 
7. Takes a few minutes for things to come up
   1. kubectl get nodes

ndslabs-up

• Domain name (dnsmasq)? For now add /etc/hosts entry
• IP address – IP of master node – same as used by kubeconfig
• ndslabs-up.sh
• kubectl label will change
• Email fails, but could output to log?
• No addons, currently
• Added bind service to support local DNS, based on instructions in 
  • http://www.damagehead.com/blog/2015/04/28/deploying-a-dns-server-using-docker/
  • Worked like a charm, did have to add DNS server to network settings

Windows 10

• Install Virtualbox
• Install vagrant
• Install git