Introduction
This document assumes basic familiarity with running containers using Docker and Kubernetes.
Interactive Tutorials for Docker
- https://www.katacoda.com/courses/docker
- https://training.play-with-docker.com/
- https://learndocker.online/demo
Interactive Tutorials for Kubernetes
- https://www.katacoda.com/courses/kubernetes
- https://kubernetes.io/docs/tutorials/kubernetes-basics/
- https://kubernetesbootcamp.github.io/kubernetes-bootcamp/
- https://labs.play-with-k8s.com/
What are Secrets?
Sounds great... what's the problem?
What is Vault and how does it solve the problem?
Setup
Vault can run locally on your machine using the vault
binary, or it can be run directly inside of your Kubernetes cluster using the Vault Helm Chart.
NOTE: Even if you plan on using the Helm deployment method, you may want to download and configure the vault
binary anyway as it can be helpful for debugging or for checking on your Vault's status.
Helm Install (recommended)
Assuming you have a Kubernetes cluster up and running with Helm installed and configured, you need only run the following commands:
# Clone the chart repo $ git clone https://github.com/hashicorp/vault-helm.git $ cd vault-helm # Checkout a tagged version $ git checkout v0.1.2 # Run Helm $ helm install --name vault ./ ...
NOTE: You can pass the --dry-run
flag to helm install
to simply print our the resources that will be created, instead of actually triggering their creation.
You should then be able to open a port-forward
using kubectl
to the Vault pod:
$ kubectl port-forward vault-0 8200:8200
As long as this port-forward is running, navigating your browser to http://localhost:8200 should allow you to view the Vault UI.
Similarly, pointing your vault
binary at http://localhost:8200 over the port-forward will allow you to run vault status
to view the cluster status.
NOTE: For larger Vault cluster, configuring Vault to be exposed via a Kubernetes Service may be preferential.
Local Install
A local install has quite a few more manual steps to get things running on Kubernetes.
Download the vault binary and run vault server -dev
.
NOTE: For production setups, leave off the -dev
flag. Developer mode is far less secure, providing an unsealed Vault by default and requiring less keys to unseal in the future.
WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory and starts unsealed with a single unseal key. The root token is already authenticated to the CLI, so you can immediately begin using Vault.
Once the dev server is up and running, MAKE SURE that you do the following
Launch a new terminal session.
Copy and run the
export VAULT_ADDR ...
command from the terminal output. This will configure the Vault client to talk to our dev server.Save the unseal key somewhere. Don't worry about how to save this securely. For now, just save it anywhere.
Copy the generated Root Token value and set is as
VAULT_DEV_ROOT_TOKEN_ID
environment variable:$ export VAULT_DEV_ROOT_TOKEN_ID="s.SomeLongStringThatIsUniqueToYourVault"
You now have a running/unsealed dev Vault. Let's try manually storing some secrets to learn the basics.
Learning the Basics
Storing and Retrieving Secrets
Vault offers an excellent "first steps" tutorial that walks you through reading and writing secrets to the Vault: https://learn.hashicorp.com/vault/getting-started/first-secret
You can put
one or more key-value pairs to a Vault secret using the vault kv put
command:
$ vault kv put secret/hello foo=world exciting=yes Key Value --- ----- created_time 2019-10-30T16:42:05.269502Z deletion_time n/a destroyed false version 2
You can retrieve these values again using vault kv get
:
$ vault kv get secret/hello ====== Metadata ====== Key Value --- ----- created_time 2019-10-30T16:42:05.269502Z deletion_time n/a destroyed false version 2 === Data === Key Value --- ----- excited yes foo world
Working with Secrets Engines
See https://learn.hashicorp.com/vault/getting-started/secrets-engines
Our above examples worked fine for secret/hello
, but what if we attempt to change this to someotherpath/hello
?
$ vault kv put someotherpath/hello foo=world Error making API request. URL: GET http://127.0.0.1:8200/v1/sys/internal/ui/mounts/someotherpath/hello Code: 403. Errors: * preflight capability check returned 403, please ensure client's policies grant access to path "someotherpath/hello/"
This is because we don't have a secrets engine
enabled that matches someotherpath/
. A secrets engine is Vault's way of writing these secrets to the underlying filesystem. This way, we don't need to worry about proper file formats or manually ingesting values.
To see all of the currently-enabled secrets engines, you can use vault secrets list
:
$ vault secrets list Path Type Accessor Description ---- ---- -------- ----------- cubbyhole/ cubbyhole cubbyhole_d1212f15 per-token private secret storage identity/ identity identity_910c214b identity store secret/ kv kv_19353bba key/value secret storage sys/ system system_c58ea5a1 system endpoints used for control, policy and debugging
To enable a new secrets engine:
$ vault secrets enable -path=kv/ kv Success! Enabled the kv secrets engine at: kv/ $ vault secrets list Path Type Accessor Description ---- ---- -------- ----------- cubbyhole/ cubbyhole cubbyhole_d1212f15 per-token private secret storage identity/ identity identity_910c214b identity store kv/ kv kv_aa513af3 n/a secret/ kv kv_19353bba key/value secret storage sys/ system system_c58ea5a1 system endpoints used for control, policy and debugging
We are now able to store values prefixed with kv/
or whatever path was provided when enabling the secrets engine:
$ vault write kv/my-secret value="s3c(eT" Success! Data written to: kv/my-secret $ vault write kv/hello target=world Success! Data written to: kv/hello $ vault write kv/airplane type=boeing class=787 Success! Data written to: kv/airplane $ vault list kv Keys ---- airplane hello my-secret
We can disable a secrets engine using a similar syntax:
$ vault secrets disable kv/ Success! Disabled the secrets engine (if it existed) at: kv/
Managing Users and Policies
By default, you are authenticated into the dev cluster as the admin user - this user can access all paths regardless of the policies set.
The admin defines a set of paths that other users should be able to access and assigns a list of capabilities
(verbs) that a user is allowed to perform on that path.
Create a read-only policy for a particular key or set of keys and write some test keys to it as the admin user:
# Create a policy file, myapp-kv-ro.hcl $ tee myapp-kv-ro.hcl <<EOF # If working with K/V v1 path "secret/myapp/*" { capabilities = ["read", "list"] } # If working with K/V v2 path "secret/data/myapp/*" { capabilities = ["read", "list"] } EOF # Create a policy named myapp-kv-ro $ vault policy write myapp-kv-ro myapp-kv-ro.hcl # Write some test keys as the admin user $ vault kv put secret/myapp/config username='appuser' \ password='suP3rsec(et!' \ ttl='30s'
Enable the userpass
auth method, then create and login as a new test user:
# Enable userpass auth method $ vault auth enable userpass # Create a user named "test-user" $ vault write auth/userpass/users/test-user \ password=training \ policies=myapp-kv-ro # Login as test-user $ vault login -method=userpass \ username=test-user \ password=training
We can now test that our policy is working as we have defined. Try to read from, then write to secret/myapp/config
and you should see that your request is denied with a 403 - this means that our policy is working correctly!
# Verify that test-user can read secret/myapp path, as policy has written $ vault kv get secret/myapp/config ====== Metadata ====== Key Value --- ----- created_time 2019-10-29T20:52:41.15247Z deletion_time n/a destroyed false version 1 ====== Data ====== Key Value --- ----- password suP3rsec(et! ttl 30s username appuser # Verify that test-user CANNOT write secret/myapp path, as policy has written $ vault kv put secret/myapp/config another=one Error writing data to secret/data/myapp/config: Error making API request. URL: PUT http://127.0.0.1:8200/v1/secret/data/myapp/config Code: 403. Errors: * 1 error occurred: * permission denied
After testing the policy, you will need to log back into the privileged user to configure Kubernetes auth:
# Log back in using the token you saved from Vault's startup logs $ vault login ${VAULT_DEV_ROOT_TOKEN_ID}
Vault Agent for Kubernetes
The final step is to configure Vault's Kubernetes authentication method.
See https://www.vaultproject.io/docs/auth/kubernetes.html#configuration
https://learn.hashicorp.com/vault/identity-access-management/vault-agent-k8s