Introduction

This document assumes basic familiarity with running containers using Docker and Kubernetes.

Interactive Tutorials for Docker

Interactive Tutorials for Kubernetes

What are Secrets?

Sounds great... what's the problem?

What is Vault and how does it solve the problem?

Setup

Vault can run locally on your machine using the vault binary, or it can be run directly inside of your Kubernetes cluster using the Vault Helm Chart.

NOTE: Even if you plan on using the Helm deployment method, you may want to download and configure the vault binary anyway as it can be helpful for debugging or for checking on your Vault's status.

Helm Install (recommended)

Assuming you have a Kubernetes cluster up and running with Helm installed and configured, you need only run the following commands:

# Clone the chart repo
$ git clone https://github.com/hashicorp/vault-helm.git
$ cd vault-helm

# Checkout a tagged version
$ git checkout v0.1.2

# Run Helm
$ helm install --name vault ./
...

NOTE: You can pass the --dry-run flag to helm install to simply print our the resources that will be created, instead of actually triggering their creation.

You should then be able to open a port-forward using kubectl to the Vault pod:

$ kubectl port-forward vault-0 8200:8200

As long as this port-forward is running, navigating your browser to http://localhost:8200 should allow you to view the Vault UI.

Similarly, pointing your vault binary at http://localhost:8200 over the port-forward will allow you to run vault status to view the cluster status.

NOTE: For larger Vault cluster, configuring Vault to be exposed via a Kubernetes Service may be preferential.

Local Install

A local install has quite a few more manual steps to get things running on Kubernetes.

Download the vault binary and run vault server -dev .

NOTE: For production setups, leave off the -dev flag. Developer mode is far less secure, providing an unsealed Vault by default and requiring less keys to unseal in the future.

WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.

Once the dev server is up and running, MAKE SURE that you do the following

Launch a new terminal session.
Copy and run the export VAULT_ADDR ... command from the terminal output. This will configure the Vault client to talk to our dev server.
Save the unseal key somewhere. Don't worry about how to save this securely. For now, just save it anywhere.
Copy the generated Root Token value and set is as VAULT_DEV_ROOT_TOKEN_ID environment variable:
```
$ export VAULT_DEV_ROOT_TOKEN_ID="s.SomeLongStringThatIsUniqueToYourVault"
```

You now have a running/unsealed dev Vault. Let's try manually storing some secrets to learn the basics.

Learning the Basics

Storing and Retrieving Secrets

Vault offers an excellent "first steps" tutorial that walks you through reading and writing secrets to the Vault: https://learn.hashicorp.com/vault/getting-started/first-secret

You can put one or more key-value pairs to a Vault secret using the vault kv put command:

$ vault kv put secret/hello foo=world exciting=yes
Key              Value
---              -----
created_time     2019-10-30T16:42:05.269502Z
deletion_time    n/a
destroyed        false
version          2

You can retrieve these values again using vault kv get:

$ vault kv get secret/hello
====== Metadata ======
Key              Value
---              -----
created_time     2019-10-30T16:42:05.269502Z
deletion_time    n/a
destroyed        false
version          2

=== Data ===
Key    Value
---    -----
excited    yes
foo        world

Working with Secrets Engines

Our above examples worked fine for secret/hello, but what if we attempt to change this to someotherpath/hello?

$ vault kv put someotherpath/hello foo=world
Error making API request.

URL: GET http://127.0.0.1:8200/v1/sys/internal/ui/mounts/someotherpath/hello
Code: 403. Errors:

* preflight capability check returned 403, please ensure client's policies grant access to path "someotherpath/hello/"

This is because we don't have a secrets engine enabled that matches someotherpath/. A secrets engine is Vault's way of writing these secrets to the underlying filesystem. This way, we don't need to worry about proper file formats or manually ingesting values.

To see all of the currently-enabled secrets engines, you can use vault secrets list:

$ vault secrets list
Path          Type         Accessor              Description
----          ----         --------              -----------
cubbyhole/    cubbyhole    cubbyhole_d1212f15    per-token private secret storage
identity/     identity     identity_910c214b     identity store
secret/       kv           kv_19353bba           key/value secret storage
sys/          system       system_c58ea5a1       system endpoints used for control, policy and debugging

To enable a new secrets engine:

$ vault secrets enable -path=kv/ kv
Success! Enabled the kv secrets engine at: kv/

$ vault secrets list
Path          Type         Accessor              Description
----          ----         --------              -----------
cubbyhole/    cubbyhole    cubbyhole_d1212f15    per-token private secret storage
identity/     identity     identity_910c214b     identity store
kv/           kv           kv_aa513af3           n/a
secret/       kv           kv_19353bba           key/value secret storage
sys/          system       system_c58ea5a1       system endpoints used for control, policy and debugging

We are now able to store values prefixed with kv/ or whatever path was provided when enabling the secrets engine:

$ vault write kv/my-secret value="s3c(eT"
Success! Data written to: kv/my-secret

$ vault write kv/hello target=world
Success! Data written to: kv/hello

$ vault write kv/airplane type=boeing class=787
Success! Data written to: kv/airplane

$ vault list kv
Keys
----
airplane
hello
my-secret

We can disable a secrets engine using a similar syntax:

$ vault secrets disable kv/
Success! Disabled the secrets engine (if it existed) at: kv/

Managing Users and Policies

By default, you are authenticated into the dev cluster as the admin user - this user can access all paths regardless of the policies set.

The admin defines a set of paths that other users should be able to access and assigns a list of capabilities (verbs) that a user is allowed to perform on that path.

Create a read-only policy for a particular key or set of keys and write some test keys to it as the admin user:

# Create a policy file, myapp-kv-ro.hcl
$ tee myapp-kv-ro.hcl <<EOF
# If working with K/V v1
path "secret/myapp/*" {
    capabilities = ["read", "list"]
}

# If working with K/V v2
path "secret/data/myapp/*" {
    capabilities = ["read", "list"]
}
EOF

# Create a policy named myapp-kv-ro
$ vault policy write myapp-kv-ro myapp-kv-ro.hcl

# Write some test keys as the admin user
$ vault kv put secret/myapp/config username='appuser' \
        password='suP3rsec(et!' \
        ttl='30s'

Enable the userpass auth method, then create and login as a new test user:

# Enable userpass auth method
$ vault auth enable userpass

# Create a user named "test-user"
$ vault write auth/userpass/users/test-user \
        password=training \
        policies=myapp-kv-ro

# Login as test-user
$ vault login -method=userpass \
        username=test-user \
        password=training

We can now test that our policy is working as we have defined. Try to read from, then write to secret/myapp/config and you should see that your request is denied with a 403 - this means that our policy is working correctly!

# Verify that test-user can read secret/myapp path, as policy has written
$ vault kv get secret/myapp/config
====== Metadata ======
Key              Value
---              -----
created_time     2019-10-29T20:52:41.15247Z
deletion_time    n/a
destroyed        false
version          1

====== Data ======
Key         Value
---         -----
password    suP3rsec(et!
ttl         30s
username    appuser

# Verify that test-user CANNOT write secret/myapp path, as policy has written
$ vault kv put secret/myapp/config another=one
Error writing data to secret/data/myapp/config: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/secret/data/myapp/config
Code: 403. Errors:

* 1 error occurred:
	* permission denied

After testing the policy, you will need to log back into the privileged user to configure Kubernetes auth:

# Log back in using the token you saved from Vault's startup logs
$ vault login ${VAULT_DEV_ROOT_TOKEN_ID}

Vault Agent for Kubernetes

https://learn.hashicorp.com/vault/identity-access-management/vault-agent-k8s

Page tree

Encrypting Kubernetes Secrets with Vault