Thoughts on developer and single-node installation as per Locally-managed workbench.
Use case
The primary use case is for a developer to run a local copy of Labs Workbench on their laptop or a remote VM (AWS, GCE, Azure, OpenStack, etc). The basic requirements are simple:
- Bring up a (virtual) Kubernetes cluster
- Deploy Labs Workbench
Current environment
For developers, we achieve this with the ndslabs-startup repo. A simple shell script kube-up.sh brings up Kubernetes via Hyperkube. This requires access to /var/lib/docker and /var/lib/kubelet, which has proven problematic on MacOS and Windows with the newer Docker installs. A second shell script ndslabs-up.sh collects information from the user (domain name, IP address, email address) and uses kubectl to deploy a number of templated YAML files.
Pros:
- Simple, scripted process gets things running quickly
Cons:
- Doesn't currently work on MacOS/Windows
- Repo has unused stuff
- "Mustache" variable replacement is ridiculous
- Doesn't include support for DNS (can be added easily)
- Templates duplicated in deploy-tools repository
Options:
- Minikube (works on MacOS, Windows, Linux – doesn't require Docker; doesn't work on VM)
- Hyperkube (works well on VMs running Docker; doesn't work well on MacOS/Windows due to filesystem permission problems)
- Vagrant (works on MacOS)
- Deploy-tools
Minikube
Minikube runs Kubernetes in a single VM.
- Requires virtualization environment, like VirtualBox
- Download binary
- minikube start creates a VM in VirtualBox (or other virtualization environment)
- Sets up kubeconfig and local environment to access VM via kubectl
MacOS
- Install VirtualBox
- Download binary
- minikube start
- git clone ndslabs-up.sh
- Don't need kube-up.sh, since we're running Minikube
- ndslabs-up.sh
- Local DNS via bind works!
- IP detection, etcd, etc, won't work as-is
- Labeling won't work as is.
Windows 10
- Install VirtualBox (can also use Hyper-V, but not tested here)
- Disable Hyper-V: Settings > Turn Windows Features on or off; uncheck Hyper-V
- Download minikube for Windows
- minikube.exe start
- Will make changes to VirtualBox
- Works well on Windows
Linux/VM
- Minikube will work on Linux, but so far we haven't been able to use it on a VM (e.g., OpenStack). Hyperkube works for this.
Hyperkube
Our current ndslabs-startup/kube-up.sh relies on Hyperkube (Kubernetes in Docker)
- This works well on OpenStack VMs
- Doesn't work well for Docker on Windows/MacOS
- Our current ndslabs-startup/kube-up relies on Hyperkube, which is basically Kubernetes in Docker.
- Hyperkube works fine on CoreOS boxes in OpenStack, but has proven problematic under Docker for Mac and Docker for Windows due to permissions problems.
- The CoreOS project provides a process for launching Kubernetes on CoreOS, using another hyperkube image
Vagrant
MacOS
The VirtualBox/Vagrant approach suggested by David works on MacOS. As with Minikube, this will not work on an OpenStack or other VM without running a different virtualization package (VirtualBox won't work in a VM). It's not clear that Vagrant will support this configuration.
Following instructions from Multi Node Kubernetes developer environment - using VBox/Vagrant/CoreOS
- Install Vagrant
- vagrant version
Got error "failed generating SSL artifacts", solved with https://github.com/mitchellh/vagrant/issues/7747
sudo ln -sf /usr/local/bin/openssl /opt/vagrant/embedded/bin/openssl
- git clone https://github.com/coreos/coreos-kubernetes.git
- cd coreos-kubernetes/multi-node/vagrant/
- cp config.rb.sample config.rb, uncomment values
- Want 2GB for compute for us
- vagrant up --provider virtualbox
- vagrant status
- Should show three instances e1 (etcd), w1 (compute), c1 (master/controller)
- vagrant ssh c1 will take you into the controller node, will need to
- Takes a few minutes for things to come up
- kubectl get nodes
ndslabs-up
- Domain name (dnsmasq)? For now add /etc/hosts entry
- IP address – IP of master node – same as used by kubeconfig
- ndslabs-up.sh
- kubectl label will change
- Email fails, but could output to log?
- No addons, currently
- Added bind service to support local DNS, based on instructions in
- http://www.damagehead.com/blog/2015/04/28/deploying-a-dns-server-using-docker/
- Worked like a charm, did have to add DNS server to network settings
Windows 10
- Install Virtualbox
- Install vagrant
- Install git