Overview

This page defines test cases for the NDSLabs command line utility (ndslabsctl). 

See also

Starting the API server

  • eval $(docker run --rm -t ndslabs/system-shell usage docker)
  • kube-up.sh
  • If needed, download the latest copy of ndslabsctl
  • If testing a new version of API server
    • vi /usr/local/lib/ndslabs/apiserver
    • Change latest -> test
    • docker rmi ndslabs/apiserver:test
  • ndslabs-up.sh
  • "ndslabsctl list projects" to confirm things are working

Global preconditions

  • Kubernetes is running
  • API Server is running either as a Kubernetes service or as a standalone Docker container 
  • ndslabsctl has been installed and is in your path

Help

  • Preconditions: none
  • Test steps:

    • ndslabsctl

    • ndslabsctl -h

    • ndslabsctl --help

  • Expected results
    • Any of these commands should display the help/usage information

Account administration

Add account (admin only)

    • Test steps
      • ndslabsctl add account <name> <password>
      • ndslabsctl add account -f <account.json>
      • ndslabsctl add account --file <account.json>
    • Expected results
      • Prompted for admin password
      • Account is added. User can login with specified name and password
    • Alternate flows
      • Account exists with same name (409 conflict)
      • Incorrect admin password (401 unauthorized)
    • Sample Data:
      • {
    "id": "demo",
    "name": "demo account",
    "description": "demo account description",
    "namespace": "demo",
    "password": "12345",
	"storageQuota": "10",
	"resourceLimits": {
		"cpuMax": "2",
		"cpuDefault": "1",
		"memMax": "8Gi",
		"memDefault": "100Mi",
		"storageQuota": "10"
	}
}

Delete account (admin only)

    • Test steps
      • ndslabsctl account project <account>
    • Expected results
      • Prompted for admin password
      • Account is deleted
    • Alternate flows:
      • Account does not exist (404 not found) 

      • Incorrect admin password (401 unauthorized)

Get account details (admin or current project only)

    • Preconditions
      • Admin is logged in or current account user is logged in
    • Test steps
      • ndslabsctl get account <account>
    • Expected results
      • Displays account details (as json)
      • Password is redacted
    • Alternate flows:
      • Account does not exist (404)
      • Account is not current project (401 unauthorized)

List accounts

  • Test steps
    • ndslabsctl list accounts
  • Expected results
    • Prompted for admin password
    • Accounts are listed
  • Alternate flows:
    • Incorrect admin password (401 unauthorized)

Authentication/authorization

Login

  • Preconditions:

    • Account has been created

  • Test steps:
    • ndslabsctl login <user>
    • Prompted for password
  • Expected results:
    • Login succeeded
  • Alternative flows
    • Invalid username / password (401 unauthorized)
  • Notes:
    • Login information is stored in ~/.apictl/.passwd. This files contains the currently logged in user and an authentication token. Removing this file is the equivalent of logging out
  • Possible Issues:
    • It is possible to execute login while already logged in.
      • This may be intentional to allow you to easily switch namespaces.
      • Won't fix, for now

Logout

  • Preconditions
    • User is logged in
  • Test steps
    • ndslabsctl logout
    • ndslabsctl list services
  • Expected results:
    • List command results in "unauthorized"

Change password

  • Preconditions:
    • User is logged in
  • Test steps
    • ndslabsctl login <user>
    • ndslabsctl passwd
      • Enter current password
      • Enter new password
      • Confirm new password
  • Alternate flows:
    • User not logged in (401 unauthorized)
    • Invalid password
    • Password mismatch yield error message: "Passwords do not match"
  • Expected results
    • Password is changed

Service administration

Get service details

    • Preconditions
      • User is logged in
    • Test steps
      • ndslabsctl get service <key>
    • Expected results
      • Displays service spec (as json).  Defaults to user catalog
    • Alternate flows:
      • Service does not exist (404)

List services 

  • Preconditions
    • User is logged in
  • Test steps
    • ndslabsctl list services
    • ndslabsctl list services -c system
    • ndslabsctl list services -c user
    • ndslabsctl list services -c all
  • Expected results
    • Services are listed for the specified catalog (Default: user)
  • Alternate flows:
    • User not logged in (401 unauthorized)

Add service

    • Test steps
      • ndslabsctl add service -f <service.json> -c <catalog>
    • Expected results
      • Prompted for admin password
      • Service is added to the specified catalog (default: user). Only admin user can add to system catalog
      • "ndslabsctl list services" now shows added service
    • Alternate flows
      • Service exists with same key (409 conflict)
        • Problem: value is replaced in etcd instead of error thrown - this may be intentional
          • Should not replace service if being used in a stack
          • Prompt for confirmation on replace?
      • Incorrect admin password (401 unauthorized)
    • Sample Data:
      • {
    "key": "whalesay",
    "label": "Docker Whalesay Example",
    "image": "ndslabs/cowsay-php",
    "description": "An example of a custom spec loaded into NDSLabs",
    "isStack": true,
    "isService": true,
    "isPublic": true,
    "isStandalone": true,
    "ports": [
        { "port": 80, "protocol": "http" }
    ]
}

Delete service 

    • Test steps
      • ndslabsctl delete service <key> -c <catalog>
    • Expected results
      • Prompted for admin password
      • Service is deleted from the specified catalog
      • Default catalog is user
      • Admin user required to delete from system catalog
      • "ndslabsctl list services" no longer shows deleted service
    • Alternate flows:
      • Incorrect admin password (401 unauthorized)
      • Service is in use by projects (409)
      • Service required by other services (409)
      • No such service (404)

Volume administration

Create volume

    • Preconditions
      • Logged in
      • Sufficient quota remaining (not implemented)
      • Service exists with no existing attachments
      • Service not running
    • Test steps
      • ndslabsctl add volume <vol name> <vol size GB>
        • Creates a detached volume
      • ndslabsctl add volume <vol name> <vol size GB> <sid>
        • Creates and attaches volume
    • Expected results
      • Volume is created
      • Volume is created and attached
    • Alternate flows
      • Not logged in (401)
      • Insufficient quota (409) (not implemented)
      • Duplicate name (409)
        • NDS-181 now allows duplicate volume names
      • Stack service already has a volume attached (409)
      • Stack service doesn't exist (404)

Attach volume

    • Preconditions:
      • Logged in 
      • Volume exists
      • Service exists with no existing attachments
      • Service not running
    • Test steps
      • ndslabsctl attach <vol name> <sid>
    • Expected results
      • Volume is attached to specified service
    • Alternate flows
      • Not logged in (401)
      • Volume doesn't exist (404)
      • Service doesn't exist (404)
      • Service is running (409)
      • Volume is attached (409)
        • Problem: Reattaches volume to new service without throwing error
    • Issues:
      • It is possible to attach a single volume to any kind of service. This may be intentional.

List volumes

    • Preconditions
      • Logged in
      • Volumes exist
    • Test steps
      • ndslabsctl list volumes
    • Expected results
      • Volume details are displayed
    • Alternate flows
      • Not logged in (401)
      • No volumes exist (200)

Detach volume

    • Preconditions
      • Logged in
      • Volume exists and is attached
      • Stack not running
    • Test steps
      • ndslabsctl detach <vol name>
    • Expected results
      • Volume is detached
    • Alternative flows
      • Not logged in (401)
      • Volume doesn't exist (404)
      • Stack is running (409)
        • Problem: Was able to detach from running stack without error
        • Resolved
      • Volume isn't attached (409)
        • Problem: Was able to detach multiple times without error
        • Resolved

Delete volume

    • Preconditions
      • Logged in
      • Volume exists
      • Stack not running
    • Test case:
      • ndslabsctl delete <vol name>
    • Expected results:
      • Volume is deleted
    • Alternative flows
      • Not logged in (401)
      • Volume doesn't exist (404)
      • Stack is running (409)

Stack administration

Add stack

    • Preconditions
      • Logged in
      • Service exists
    • Test steps
      • ndslabsctl add stack <key> <name> 
      • ndslabsctl add stack <key> <name> --opt=<comma-separated list of optional services>
    • Expected results
      • Stack is added
    • Alternate flows
      • Not logged in (401)
      • Service doesn't exist (404)
      • Duplicate stack name (409)
        • NDS-181 now allows duplicate stack names

List stacks

List stacks

    • Preconditions:
      • Logged in
      • Stack has been added
    • Test steps
      • ndslabsctl list stacks
    • Expected results
      • Summary of stacks is displayed (name, services, status, SIDs)
    • Alternate flow:
      • Not logged in (401)
      • No stacks (empty list)

Delete stack

    • Preconditions:
      • Logged in
      • Stack exists
      • Stack not running
    • Test steps
      • ndslabsctl delete stack <sid>
      • Stack is deleted (201)
    • Alternate flows
      • Not logged in (401)
      • Stack doesn't exist (404)
      • Stack is running (409)
      • Attached volumes are orphaned

Get stack details

    • Preconditions:
      • Logged in
      • Stack exists
    • Test steps
      • ndslabsctl get stack <sid>
    • Expected results
      • Stack details, including endpoints and config, are displayed
    • Alternate flows
      • Not logged in (401)
      • Stack doesn't exist (404)

Start stack

    • Preconditions
      • Logged in
      • Stack exists and started
    • Test steps
      • ndslabsctl start <sid>
    • Expected results
      • All stack services (service, replication controller, pods) are started and ready
    • Alternate flows
      • Not logged in (401)
      • Stack doesn't exist (404)
      • Stack not started or stopping (409)

Stop stack

    • Preconditions
      • Stack started
    • Test steps
      • ndslabsctl stop <sid>
    • Expected results
      • Stack service, replication controller, pods are stopped (200)
    • Alternative flow
      • Not logged in (401)
      • Stack does not exist (404)
      • Stack not started (200)

Configuration

List configuration options for spec

    • Preconditions
      • Logged in
      • Service exists
    • Test case:
      • ndslabsctl list configs <service key(s)>
    • Expected results:
      • Possible service configuration options are displayed
    • Alternative flow
      • Not logged in (401)
      • Service does not exist (404)
        • Problem: currently throws 500 error
        • Resolved – if one fails, all fail
    • Issues:
      • Help text references "service ids", should be "service keys" for consistency
      • Resolved

Get stack service configuration

    • Preconditions
      • Logged in
      • Stack exists
    • Test case:
      • ndslabsctl get stack <stack id>
    • Expected results:
      • Current service configuration options and settings are displayed
    • Alternative flow
      • Not logged in (401)
      • Stack does not exist (404)
        • Problem: currently throws 500 error
        • Works for me

Set stack service configuration option

    • Preconditions
      • Logged in
      • Stack exists
      • Config exists
    • Test case:
      • ndslabsctl set <stack service id> <name> <value>
    • Expected results:
      • Specified configuration setting os overriden for the service
    • Alternative flow
      • Not logged in (401)
      • Stack doesn't exist (404)
        • Problem: currently throws 500 error
        • Resolved

      • Config doesn't exist or is not overridable (409)

        • Problem: No output given, appears to be no-op?
        • Resolved

Other

View stack service logs

    • Preconditions
      • Logged in
      • Stack exists
      • Stack started
      • Config exists
    • Test case:
      • ndslabsctl logs <sid> 
      • ndslabsctl logs <sid> --lines=<lines>
    • Expected results:
      • Logs are displayed
      • Last n log lines are displayed
    • Alternative flow
      • Not logged in (401)
      • Stack doesn't exist (404)
        • Problem: currently throws 500 error
      • Stack is not running (404)
        • Problem: No output given, appears to be no-op?

Console access

    • Preconditions
      • Logged in
      • Stack exists
      • Stack started
    • Test case
      • ndslabsctl console <ssid>
    • Alternative flow
      • Not logged in (401)
      • Stack service doesn't exist (404)
      • Stack not running (404)

API Server Startup

Kubernetes token authentication

    • Preconditions
      • Kubernetes started with token authentication enabled for kube-apiserver
      • Tokens are 
    • Test case
      • Configure apiserver.conf Kubernetes section
      • Start NDS Labs (e.g., ndslabs-up)
      • Tail the apiserver log and look for successful startup
    • Expected results
      • Successful server startup (i.e., you do not see the error "Kubernertes API not availble"
    • Alternative flow
      • Invalid token path
      • Token authentication not supported by kube-apiserver

 

Resource management

Add account with resource limits

    • Test steps
      • ndslabsctl add account -f <account.json>
    • Expected results
      • Prompted for admin password
      • Account is added
      • kubectl describe quota --namespace=demo returns specified quotas
      • kubectl describe limitranges --namespace=demo returns specified limit ranges
    • Sample Data:
    { 
    "id": "demo",
    "name": "demo project",
    "description": "demo project description",
    "namespace": "demo",
    "password": "12345",
	"storageQuota": 10,
	"resourceLimits": {
		"cpuMax": "2",
		"cpuDefault": "1",
		"memMax": "8Gi",
		"memDefault": "100Mi"
	}
}
    • 5/19: OK (NDS-205)

     

Start stack with resource limits

    • Prerequisites:
      • Test services have been added
      • User is logged in
    • Test steps
      • Add "Dataverse" service
    • Expected results
      • Services start as expected

      • For each pod, "kubectl get pod <pod> --namespace=<ns> -o wide" shows a "resources" section, such as:

                        "resources": {
                    "limits": {
                        "cpu": "100m",
                        "memory": "500Mi"
                    },
                    "requests": {
                        "cpu": "100m",
                        "memory": "500Mi"
                    }
                }
      • Resources and limits match those specified on the service spec
    • 5/19: OK (NDS-205)

Start stack: Too big

    • Prerequisites:
      • Test services have been added
      • User is logged in
    • Test steps
      • Add "Test - Too Big" service
    • Expected results
      • Stack service is in error state
    • 5/19: OK (NDS-205)


Start stack: Memory Hog

    • Prerequisites:
      • Test services have been added
      • User is logged in
    • Test steps
      • Add "Test - Memory Hog" service
    • Expected results
      • Stack service starts, but quickly enters error state
      • Log shows 
        • Reason="OOMKilled"
    • 5/19: OK (NDS-205)

Start stack: No such image

    • Prerequisites:
      • Test services have been added
      • User is logged in
    • Test steps
      • Add "Test - No Image" service
    • Expected results
      • Stack service starts, but quickly enters error state
      • Log shows 

        • Reason=Failed, Message=Failed to pull image "xyzzy": Error: image library/xyzzy not found

    • 5/19: OK (NDS-205)

Start stack: Bad dependency

    • Prerequisites:
      • Test services have been added
      • User is logged in
    • Test steps
      • Add "Test - Bad dependency" service
    • Expected results
      • "No image" service is in error state, "Bad dependency" service is in "stopped" state
      • Log shows 

        • Reason=Failed, Message=Failed to pull image "xyzzy": Error: image library/xyzzy not found

    • 5/19: OK (NDS-205)

Ingress/LoadBalancer support

Single-node, NodePort, no TLS

This first test case demonstrates how to configure a single-node instance using NodePort without TLS;

    • Prerequisites
      • Kubernetes is running
      • Assumes single-node installation
    • Test steps
      • Modify apiserver.yaml
        • CORS_ORIGIN_ADDR = http://<public ip>:30000
        • INGRESS = NodePort
        • PREFX=/api
      • Modify gui.yaml
        • APISERVER_HOST = <public ip>
        • APISERVER_PORT = 30001
        • UI_BASE_PATH = /ui
      • kubectl create -f gui.yaml
      • kubectl create -f apiserver.yaml
      • kubectl label nodes 127.0.0.1 ndslabs-role=compute
      • Open browser to: http://<public ip>:30000/ui/

Single-node, LoadBalancer + TLS

    • Prerequisites
      • Kubernetes is running
      • Assumes single-node installation
      • Wildcard CNAME entry for *.test.ndslabs.org pointing to single node IP
    • Test steps
      • Generate wildcard TLS certificate and key
openssl genrsa 2048 > certs/ndslabs.key
openssl req -new -x509 -nodes -sha1 -days 3650 -key certs/ndslabs.key > certs/ndslabs.cert
#[enter *.test.ndslabs.org for the Common Name]
cat certs/ndslabs.cert certs/ndslabs.key > certs/ndslabs.pem
chmod 400 certs/ndslabs.key certs/ndslabs.pem
        • git clone https://github.com/craig-willis/ndslabs-startup
        • cd ndslabs-startup
        • ./ndslabs-up.sh
          • Domain name: test.ndslabs.org
          • Internal IP: accept default
        • Open browser to:
        • A few kubectl checks
          • kubectl get secrets --namespace=default 
            • ndslabs-tls-secret
          • kubectl get ingress --namespace=default
            • ndslabs-ingress with rules for /api and /ui
          • kubectl get secret --namespace=demo 
            • demo-tls-secret
          • kubectl get ingress --namespace=demo
            • stack-clowder-ingress with rule for clowder "/"
        • Stop the stack
          • Confirm ingress deleted
            • kubectl get ingress --namespace=demo
        • Delete project
          • ndslabsctl delete project demo
          • Confirm secret deleted
            • kubectl get secret --namespace=demo 

Multi-node, LoadBalancer + TLS

    • Prerequisites
      • Kubernetes is running 
      • Assumes multi-node installation
      • Wildcard CNAME entry for *.test.ndslabs.org pointing to load balancer node
      • Assumes loadbalancer and default-backend services already deployed
    • Test steps
      • Generate wildcard TLS certificate and key
openssl genrsa 2048 > certs/ndslabs.key
openssl req -new -x509 -nodes -sha1 -days 3650 -key certs/ndslabs.key > certs/ndslabs.cert
#[enter *.test.ndslabs.org for the Common Name]
cat certs/ndslabs.cert certs/ndslabs.key > certs/ndslabs.pem
chmod 400 certs/ndslabs.key certs/ndslabs.pem
      • git clone https://github.com/craig-willis/ndslabs-startup
      • cd ndslabs-startup
      • ./ndslabs-up-multinode.sh
      • Open browser to:
      • A few kubectl checks
        • kubectl get secrets --namespace=default 
          • ndslabs-tls-secret
        • kubectl get ingress --namespace=default
          • ndslabs-ingress with rules for /api and /ui
        • kubectl get secret --namespace=demo 
          • demo-tls-secret
        • kubectl get ingress --namespace=demo
          • stack-clowder-ingress with rule for clowder "/"
      • Stop the stack
        • Confirm ingress deleted
          • kubectl get ingress --namespace=demo
      • Delete project
        • ndslabsctl delete project demo
        • Confirm secret deleted
          • kubectl get secret --namespace=demo 

 

  • No labels