Notes for NDS-385 exploring answers to the following questions:

  • Q. How can we enable access to TCP (non-HTTP) services?
    • A. The easiest way is to use NodePort
  • Q. Can we use the ingress controller for authentication to services?
    • A. Yes, by upgrading the ingress controller we get HTTP-basic authentication.
  • Q. Can we use the ingress controller for monitoring usage for service timeouts?
    • Open

TCP support

We have two obvious options for TCP service support

  1. Use Kubernetes NodePort mechanism for non-HTTP services
  2. Use the nginx-ingress-controller ConfigMap option

If we use the ConfigMap option, we'll need to manage port assignments ourselves. We might as well use NodePort.

Note about security

We should not allow unsecured services to be exposed via raw TCP. The only services that can be exposed via this method should be those with SSL/TLS support enabled.

We've investigated TLS/SSL termination at the loadbalancer, and it is not currently supported for TCP services in nginx.  We'll table this for later.

David has suggested that we can use SSH tunneling as a solution.  In this case, we'd run some sort of SSH tunneling service that will allow the user to create a tunnel from their local system to the remote service in NDS Labs.

NodePort

We already support access to these services through the Kubernetes NodePort model.  We simply need to open the NodePort security group on the cluster.  We should also restrict NodePort assignment to non-HTTP services to have as many ports as possible available.

A few things to consider:

  • There is a limit of 65536 (2^16) ports for a single IP address.  
  • Ports < 1024 are protected (run by root)
  • Kubernetes defaults to the range 30000-32767, configurable during startup.  
  • IANA recommends 49152-65535 as the ephemeral port range

If we do not plan to expose services other than 80/443 on the loadbalancer, we should be able to assign 1025-65536.  If we grow beyond this, we can assign multiple IP addresses to the cluster.

ConfigMap

The contrib ingress controller supports TCP services through the use of a config map:

        args:
        - /nginx-ingress-controller
        - --default-backend-service=default/default-http-backend
        - --tcp-services-configmap=default/tcp-configmap-example

Where the config-map is a map of external ports to service/ports:

apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-configmap-example
data:
  9000: "default/example-go:8080"

To support TCP services, we will need to manage this config map.  This could be in conflict with NodePort.   Each TCP service will need to be assigned a port and added to the config map when added and removed when deleted.

Chisel Baby

Import the following spec, add and start the service.

{
  "key": "chisel",
  "label": "Chisel",
  "description": "Chisel server for TCP tunneling",
  "image": {
    "registry": "",
    "name": "craigwillis/chisel",
    "tags": [
      "dev"
    ]
  },
  "display": "stack",
  "access": "external",
  "ports": [
    {
      "port": 8080,
      "protocol": "http"
    }
  ],
  "config": [
    {
      "name": "CHISEL_PASSWORD",
      "value": "",
      "label": "Password",
      "isPassword": true,
      "canOverride": false
    }
  ],
  "resourceLimits": {
    "cpuMax": 500,
    "cpuDefault": 100,
    "memMax": 1000,
    "memDefault": 50
  }
}

Download chisel from my fork, since we need to disable cert verification:

https://github.com/craig-willis/chisel/releases/download/1.1.3-noverify/chisel-darwin-amd64


Run the client (with TLS verify off because certs are self-signed):

./chisel-darwin-amd64 client -auth namespace:password -v https://stackid.test.ndslabs.org/ localhost:3000:rabbitmqip:15672

Open browser to localhost:3000

Authentication

Authentication is supported by the latest version in contrib, using HTTP basic or digest auth:

https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx/examples/auth

A secret is created using htpasswd and can be attached to an ingress object.  We currently create a single ingress object per stack service with endpoint. The simplest approach would be to create add a username/password to the ingress for individual stacks or services.

One approach would be to enable the password for the stack:

  • The stack displays a "lock" icon or similar
  • User selects "lock" icon to protect or unprotect stack
  • If protected, the user enters a username/password (generated by default)

This would require a change to the stack service object:

...
"protected": {
    "username": "<username>",
    "password": "<password>"
}

When the stack is started, a secret is created containing the username and password and attached to the various stack services.

One open issue is whether we can configure nginx to share the same htpassword prompt for multiple services. Also, we cannot password protect TCP services.

We might also consider having "protect by default" option on services.

 

 

  • No labels