...
- Preconditions: none
- Test steps:
ndslabsctl
ndslabsctl -h
ndslabsctl --help
- Expected results
- Any of these commands should display the help/usage information
- Issues:
- help command still references "apictl"
...
Account administration
Add
...
account (admin only)
- Test steps
- ndslabsctl add project account <name> <password>
- ndslabsctl add project account -f <project<account.json>
- ndslabsctl add project account --file <project<account.json>
- Expected results
- Prompted for admin password
- Project Account is added. User can login with specified name and password
- Alternate flows
- Project Account exists with same name (409 conflict)
- Incorrect admin password (401 unauthorized)
- Sample Data:
Code Block language js { "nameid": "testdemo", "descriptionname": "testacctdemo account", "namespacedescription": "testdemo account description", "passwordnamespace": "123456demo", "storageQuotapassword": "12345", "storageQuota": "10 }
- Test steps
Delete project (admin only)
- Test steps
- ndslabsctl delete project <project>
- Expected results
- Prompted for admin password
- Project is deleted
- Alternate flows:
- Project does not exist (404 not found)
Incorrect admin password (401 unauthorized)
- Test steps
Get project details (admin or current project only)
- Preconditions
- Admin is logged in or current project user is logged in
- Test steps
- ndslabsctl get project <project>
- Expected results
- Displays project details (as json)
- Password is redacted
- Alternate flows:
- Project does not exist (404)
- Problem: Can only get details of currently logged in project -> admin user can no longer get project details
- Project is not current project (401 unauthorized)
- Project does not exist (404)
- Preconditions
List projects
- Test steps
- ndslabsctl list projects
- Expected results
- Prompted for admin password
- Projects are listed
- Alternate flows:
- Incorrect admin password (401 unauthorized)
Authentication/authorization
Login
", "resourceLimits": { "cpuMax": "2", "cpuDefault": "1", "memMax": "8Gi", "memDefault": "100Mi", "storageQuota": "10" } }
Delete account (admin only)
- Test steps
- ndslabsctl account project <account>
- Expected results
- Prompted for admin password
- Account is deleted
- Alternate flows:
- Account does not exist (404 not found)
Incorrect admin password (401 unauthorized)
- Test steps
Get account details (admin or current project only)
- Preconditions
- Admin is logged in or current account user is logged in
- Test steps
- ndslabsctl get account <account>
- Expected results
- Displays account details (as json)
- Password is redacted
- Alternate flows:
- Account does not exist (404)
- Account is not current project (401 unauthorized)
- Preconditions
List accounts
- Test steps
- ndslabsctl list accounts
- Expected results
- Prompted for admin password
- Accounts are listed
- Alternate flows:
- Incorrect admin password (401 unauthorized)
Authentication/authorization
Login
Preconditions:
Account
Preconditions:
Project has been created
- Test steps:
- ndslabsctl login <user>
- Prompted for password
- Expected results:
- Login succeeded
- Alternative flows
- Invalid username / password (401 unauthorized)
- Notes:
- Login information is stored in ~/.apictl/.passwd. This files contains the currently logged in user and an authentication token. Removing this file is the equivalent of logging out
- Possible Issues:
- It is possible to execute login while already logged in.
- This may be intentional to allow you to easily switch namespaces.
- Won't fix, for now
- It is possible to execute login while already logged in.
Logout
- Preconditions
- User is logged in
- Test steps
- ndslabsctl logout
- ndslabsctl list services
- Expected results:
- List command results in "unauthorized"
- Issues
- Attempting to logout without logging in first yields:
"Error removing passwd data: remove /home/core/.apictl/.passwd: no such file or directory"
- Attempting to logout without logging in first yields:
Change password
Change password
- Preconditions:
- User
- User is logged in
- Test steps
- ndslabsctl login <user>
- ndslabsctl passwd
- Enter current password
- Enter new password
- Confirm new password
- Alternate flows:
- User not logged in (401 unauthorized)
- Invalid password
- Password mismatch yield error message: "Passwords do not match"
- Expected results
- Password is changed
Service administration
...
Get service details
- Preconditions
- User is logged in
- Test steps
- ndslabsctl get service <key>
- Expected results
- Displays service spec (as json). Defaults to user catalog
- Alternate flows:
- Service does not exist (404)
- Service does not exist (404)
- Preconditions
List servicesservices
- Preconditions
- User is logged in
- Test steps
- ndslabsctl list services
- ndslabsctl list services -c system
- ndslabsctl list services -c user
- ndslabsctl list services -c all
- Expected results
- Services are listed for the specified catalog (Default: user)
- Services are listed
- Alternate flows:
- User not logged in (401 unauthorized)
Add service
...
- Test steps
- ndslabsctl add service -f <service.json> ndslabsctl add service --file <service.json>-c <catalog>
- Expected results
- Prompted for admin password
- Service is added to the specified catalog (default: user). Only admin user can add to system catalog
- "ndslabsctl list services" now shows added service
- Alternate flows
- Service exists with same key (409 conflict)
- Problem: value is replaced in etcd instead of error thrown - this may be intentional
- Should not replace service if being used in a stack
- Prompt for confirmation on replace?
- Problem: value is replaced in etcd instead of error thrown - this may be intentional
- Incorrect admin password (401 unauthorized)
- Service exists with same key (409 conflict)
- Sample Data:
Code Block language js { "key": "whalesay", "label": "Docker Whalesay Example", "image": "ndslabs/cowsay-php", "description": "An example of a custom spec loaded into NDSLabs", "isStack": true, "isService": true, "isPublic": true, "isStandalone": true, "ports": [ { "port": 80, "protocol": "http" } ] }
- Test steps
Delete
...
service
- Test steps
- ndslabsctl delete service <key> -c <catalog>
- Expected results
- Prompted for admin password
- Service is deleted from the specified catalog
- Default catalog is user
- Admin user required to delete from system catalog
- "ndslabsctl list services" no longer shows deleted service
- Alternate flows:
- Incorrect admin password (401 unauthorized)
- Service is in use by projects (409)
- Service required by other services (409)
- No such service (404)
- Test steps
...
- Preconditions
- Logged in
- Sufficient quota remaining (not implemented)
- Service exists with no existing attachments
- Service not running
- Test steps
- ndslabsctl add volume <vol name> <vol size GB>
- Creates a detached volume
- ndslabsctl add volume <vol name> <vol size GB> <sid>
- Creates and attaches volume
- ndslabsctl add volume <vol name> <vol size GB>
- Expected results
- Volume is created
- Volume is created and attached
- Alternate flows
- Not logged in (401)
- Insufficient quota (409) (not implemented)
Duplicate name (409)- NDS-181 now allows duplicate volume names
- Stack service already has a volume attached (409)
- Stack service doesn't exist (404)
- Preconditions
...
- Preconditions
- Logged in
- Volume exists and is attached
- Stack not running
- Test steps
- ndslabsctl detach <vol name>
- Expected results
- Volume is detached
- Alternative flows
- Not logged in (401)
- Volume doesn't exist (404)
- Stack is running (409)
- Problem: Was able to detach from running stack without error
- Resolved
- Volume isn't attached (409)
- Problem: Was able to detach multiple times without error
- Resolved
- Preconditions
Delete volume
- Preconditions
- Logged in
- Volume exists
- Stack not running
- Test case:
- ndslabsctl delete <vol name>
- Expected results:
- Volume is deleted
- Alternative flows
- Not logged in (401)
- Volume doesn't exist (404)
- Stack is running (409)
- Preconditions
Stack administration
- Admin user can create / start / stop stacks. We will need to revisit the workflow for admin.
- Resolution: Admin user can no longer login. User is prompted for admin password for add/delete project/service.
Add stack
Add stack
- Preconditions
- Logged in
- Service
- Logged in
- Service exists
- Test steps
- ndslabsctl add stack <key> <name>
- ndslabsctl add stack <key> <name> --opt=<comma-separated list of optional services>
- Expected results
- Stack is added
- Alternate flows
- Not logged in (401)
- Service doesn't exist (404)
Duplicate stack name (409)- NDS-181 now allows duplicate stack names
- Preconditions
List stacks
List stacks
- Preconditions:
- Logged in
- Stack has been added
- Test steps
- ndslabsctl list stacks
- Expected results
- Summary of stacks is displayed (name, services, status, SIDs)
- Alternate flow:
- Not logged in (401)
- No stacks (empty list)
- Preconditions:
...
- Preconditions
- Stack started
- Test steps
- ndslabsctl stop <sid>
- Expected results
- Stack service, replication controller, pods are stopped (200)
- Alternative flow
- Not logged in (401)
- Stack does not exist (404)
- Stack not started (200)
- Preconditions
Configuration
...
List configuration options for spec
- Preconditions
- Logged in
- Stack Service exists
- Test case:
- ndslabsctl list configs <sid><service key(s)>
- Expected results:
- Current Possible service configuration options and settings are displayed
- Alternative flow
- Not logged in (401)
- Service does not exist (404)
- Stack doesn't exist (404)
- Preconditions
Set stack service configuration option
- Preconditions
- Logged in
- Stack exists
- Config exists
- Test case:
- ndslabsctl set <sid> <name> <value>
- Expected results:
- Specified configuration setting os overriden for the service
- Alternative flow
- Not logged in (401)
- Stack doesn't exist (404)
Config doesn't exist or is not overridable (409)
- Preconditions
Other
View stack service logs
- Preconditions
- Logged in
- Stack exists
- Stack started
- Config exists
- Test case:
- ndslabsctl logs <sid>
- ndslabsctl logs <sid> --lines=<lines>
- Expected results:
- Logs are displayed
- Last n log lines are displayed
- Alternative flow
- Not logged in (401)
- Stack doesn't exist (404)
- Stack is not running (404)
- Preconditions
Other problems:
- Volumes are referred by name not ID
- Quota stuff
- Problem: currently throws 500 error
- Resolved – if one fails, all fail
- Issues:
- Help text references "service ids", should be "service keys" for consistency
- Resolved
Get stack service configuration
- Preconditions
- Logged in
- Stack exists
- Test case:
- ndslabsctl get stack <stack id>
- Expected results:
- Current service configuration options and settings are displayed
- Alternative flow
- Not logged in (401)
- Stack does not exist (404)
- Problem: currently throws 500 error
- Works for me
- Preconditions
Set stack service configuration option
- Preconditions
- Logged in
- Stack exists
- Config exists
- Test case:
- ndslabsctl set <stack service id> <name> <value>
- Expected results:
- Specified configuration setting os overriden for the service
- Alternative flow
- Not logged in (401)
- Stack doesn't exist (404)
- Problem: currently throws 500 error
- Resolved
Config doesn't exist or is not overridable (409)
- Problem: No output given, appears to be no-op?
- Resolved
- Preconditions
Other
View stack service logs
- Preconditions
- Logged in
- Stack exists
- Stack started
- Config exists
- Test case:
- ndslabsctl logs <sid>
- ndslabsctl logs <sid> --lines=<lines>
- Expected results:
- Logs are displayed
- Last n log lines are displayed
- Alternative flow
- Not logged in (401)
- Stack doesn't exist (404)
- Problem: currently throws 500 error
- Stack is not running (404)
- Problem: No output given, appears to be no-op?
- Preconditions
Console access
- Preconditions
- Logged in
- Stack exists
- Stack started
- Test case
- ndslabsctl console <ssid>
- Alternative flow
- Not logged in (401)
- Stack service doesn't exist (404)
- Stack not running (404)
- Preconditions
API Server Startup
Kubernetes token authentication
- Preconditions
- Kubernetes started with token authentication enabled for kube-apiserver
- Tokens are
- Test case
- Configure apiserver.conf Kubernetes section
- TokenPath=/run/secrets/kubernetes.io/serviceaccount/token
- Start NDS Labs (e.g., ndslabs-up)
- Tail the apiserver log and look for successful startup
- Configure apiserver.conf Kubernetes section
- Expected results
- Successful server startup (i.e., you do not see the error "Kubernertes API not availble"
- Alternative flow
- Invalid token path
- Token authentication not supported by kube-apiserver
- Preconditions
Resource management
Add account with resource limits
- Test steps
- ndslabsctl add account -f <account.json>
- Expected results
- Prompted for admin password
- Account is added
- kubectl describe quota --namespace=demo returns specified quotas
- kubectl describe limitranges --namespace=demo returns specified limit ranges
- Sample Data:
Code Block { "id": "demo", "name": "demo project", "description": "demo project description", "namespace": "demo", "password": "12345", "storageQuota": 10, "resourceLimits": { "cpuMax": "2", "cpuDefault": "1", "memMax": "8Gi", "memDefault": "100Mi" } }
- 5/19: OK (NDS-205)
- Test steps
Start stack with resource limits
- Prerequisites:
- Test services have been added
- User is logged in
- Test steps
- Add "Dataverse" service
- Expected results
- Services start as expected
For each pod, "kubectl get pod <pod> --namespace=<ns> -o wide" shows a "resources" section, such as:
Code Block "resources": { "limits": { "cpu": "100m", "memory": "500Mi" }, "requests": { "cpu": "100m", "memory": "500Mi" } }
- Resources and limits match those specified on the service spec
- 5/19: OK (NDS-205)
- Prerequisites:
Start stack: Too big
- Prerequisites:
- Test services have been added
- User is logged in
- Test steps
- Add "Test - Too Big" service
- Expected results
- Stack service is in error state
- 5/19: OK (NDS-205)
- Prerequisites:
Start stack: Memory Hog
- Prerequisites:
- Test services have been added
- User is logged in
- Test steps
- Add "Test - Memory Hog" service
- Expected results
- Stack service starts, but quickly enters error state
- Log shows
- Reason="OOMKilled"
- 5/19: OK (NDS-205)
- Prerequisites:
Start stack: No such image
- Prerequisites:
- Test services have been added
- User is logged in
- Test steps
- Add "Test - No Image" service
- Expected results
- Stack service starts, but quickly enters error state
- Log shows
Reason=Failed, Message=Failed to pull image "xyzzy": Error: image library/xyzzy not found
- 5/19: OK (NDS-205)
- Prerequisites:
Start stack: Bad dependency
- Prerequisites:
- Test services have been added
- User is logged in
- Test steps
- Add "Test - Bad dependency" service
- Expected results
- "No image" service is in error state, "Bad dependency" service is in "stopped" state
- Log shows
Reason=Failed, Message=Failed to pull image "xyzzy": Error: image library/xyzzy not found
- 5/19: OK (NDS-205)
- Prerequisites:
Ingress/LoadBalancer support
Single-node, NodePort, no TLS
This first test case demonstrates how to configure a single-node instance using NodePort without TLS;
- Prerequisites
- Kubernetes is running
- Assumes single-node installation
- Test steps
- Modify apiserver.yaml
- CORS_ORIGIN_ADDR = http://<public ip>:30000
- INGRESS = NodePort
- PREFX=/api
- Modify gui.yaml
- APISERVER_HOST = <public ip>
- APISERVER_PORT = 30001
- UI_BASE_PATH = /ui
- kubectl create -f gui.yaml
- kubectl create -f apiserver.yaml
- kubectl label nodes 127.0.0.1 ndslabs-role=compute
- Open browser to: http://<public ip>:30000/ui/
- Configure basic Clowder, launch stack
- Confirm Clowder endpoint link points to NodePort (http://publicip:NodePort)
- Modify apiserver.yaml
- Prerequisites
Single-node, LoadBalancer + TLS
- Prerequisites
- Kubernetes is running
- Assumes single-node installation
- Wildcard CNAME entry for *.test.ndslabs.org pointing to single node IP
- Test steps
- Generate wildcard TLS certificate and key
- Prerequisites
Code Block |
---|
openssl genrsa 2048 > certs/ndslabs.key
openssl req -new -x509 -nodes -sha1 -days 3650 -key certs/ndslabs.key > certs/ndslabs.cert
#[enter *.test.ndslabs.org for the Common Name]
cat certs/ndslabs.cert certs/ndslabs.key > certs/ndslabs.pem
chmod 400 certs/ndslabs.key certs/ndslabs.pem |
- git clone https://github.com/craig-willis/ndslabs-startup
- cd ndslabs-startup
- ./ndslabs-up.sh
- Domain name: test.ndslabs.org
- Internal IP: accept default
- Open browser to:
- https://www.test.ndslabs.org/ui/ (Accept certificate)
- Configure basic Clowder, launch stack
- Confirm endpoint URI link points to ingress host (https://stackservice.test.ndslabs.org)
- A few kubectl checks
- kubectl get secrets --namespace=default
- ndslabs-tls-secret
- kubectl get ingress --namespace=default
- ndslabs-ingress with rules for /api and /ui
- kubectl get secret --namespace=demo
- demo-tls-secret
- kubectl get ingress --namespace=demo
- stack-clowder-ingress with rule for clowder "/"
- kubectl get secrets --namespace=default
- Stop the stack
- Confirm ingress deleted
- kubectl get ingress --namespace=demo
- Confirm ingress deleted
- Delete project
- ndslabsctl delete project demo
- Confirm secret deleted
- kubectl get secret --namespace=demo
Multi-node, LoadBalancer + TLS
- Prerequisites
- Kubernetes is running
- Assumes multi-node installation
- Wildcard CNAME entry for *.test.ndslabs.org pointing to load balancer node
- Assumes loadbalancer and default-backend services already deployed
- Test steps
- Generate wildcard TLS certificate and key
- Prerequisites
Code Block |
---|
openssl genrsa 2048 > certs/ndslabs.key
openssl req -new -x509 -nodes -sha1 -days 3650 -key certs/ndslabs.key > certs/ndslabs.cert
#[enter *.test.ndslabs.org for the Common Name]
cat certs/ndslabs.cert certs/ndslabs.key > certs/ndslabs.pem
chmod 400 certs/ndslabs.key certs/ndslabs.pem |
- git clone https://github.com/craig-willis/ndslabs-startup
- cd ndslabs-startup
- ./ndslabs-up-multinode.sh
- Domain name: test.ndslabs.org
- Internal IP: accept default
- Open browser to:
- https://www.test.ndslabs.org/ui/ (Accept certificate)
- Configure basic Clowder, launch stack
- Confirm endpoint URI link points to ingress host (https://stackservice.test.ndslabs.org)
- A few kubectl checks
- kubectl get secrets --namespace=default
- ndslabs-tls-secret
- kubectl get ingress --namespace=default
- ndslabs-ingress with rules for /api and /ui
- kubectl get secret --namespace=demo
- demo-tls-secret
- kubectl get ingress --namespace=demo
- stack-clowder-ingress with rule for clowder "/"
- kubectl get secrets --namespace=default
- Stop the stack
- Confirm ingress deleted
- kubectl get ingress --namespace=demo
- Confirm ingress deleted
- Delete project
- ndslabsctl delete project demo
- Confirm secret deleted
- kubectl get secret --namespace=demo
...