...
Starting the API server
- eval $(docker run --rm -t ndslabs/system-shell usage docker)
- git clone https://github.com/craig-willis/ndslabs-startup
- cd ndslabs-startup
- ./kube-up.sh
- ./ndslabs-up.sh
...
- If needed, download the latest
...
- copy of
...
- ndslabsctl
- wget https://github.
...
...
...
- /ndslabsctl-linux-amd64 -O /usr/local/bin/ndslabsctl
- chmod +x ndslabsctl
- If testing a new version of API server
- vi /usr/local/lib/ndslabs/apiserver
- Change latest -> test
- docker rmi ndslabs/apiserver:test
- ndslabs-up.sh
- "ndslabsctl list projects" to confirm things are working
Global preconditions
- Kubernetes is running
- API Server is running either as a Kubernetes service or as a standalone Docker container
- ndslabsctl has been installed and is in your path
Help
- Preconditions: none
- Test steps:
ndslabsctl
ndslabsctl -h
ndslabsctl --help
- Expected results
- Any of these commands should display the help/usage information
Authentication/authorization
Login
Preconditions:
Project has been created
If the API server is running under docker or Kubernetes, you will need to specify the server address. This can be done on the commandline using the --server flag or by creating a .ndslabsctl.yaml file in your home directory. For example:
APISERVER=`kubectl describe svc ndslabs-apiserver | grep ^IP | awk '{print $2}'`
echo "server: http://$APISERVER:8083" > ~/.ndslabsctl.yaml
Global preconditions
- Kubernetes is running
- API Server is running either as a Kubernetes service or as a standalone Docker container
- ndslabsctl has been installed and is in your path
Help
- Preconditions: none
- Test steps:
ndslabsctl
ndslabsctl -h
ndslabsctl --help
- Expected results
- Any of these commands should display the help/usage information
Authentication/authorization
Login
Preconditions:
Project has been created or admin user
- Test steps:
- ndslabsctl login admin
- ndslabsctl login <user>
- Prompted for password
- Expected results:
- Login succeeded
- Alternative flows
- Invalid username or password results in login error
- Notes:
- Login information is stored in ~/.apictl/.passwd. This files contains the currently logged in user and an authentication token. Removing this file is the equivalent of logging out
...
- Preconditions:
- User is logged in
- Test steps
- ndslabsctl login <user>
- ndslabsctl passwd
- Enter current password
- Enter new password
- Confirm new password
- Expected results
- Password is changed
- Issues:
- Attempting to change the admin password yielded:
Code Block | ||
---|---|---|
| ||
$ ndslabsctl passwd
Error changing password: %s
404 Not Found |
Project administration
Add project (admin only)
- Resolution:
Admin user cannot login. Only a regular project user can use passwd command
Code Block language bash # ndslabsctl login admin Admin login not allowed # ndslabsctl login willis8 Password: Login succeeded # ndslabsctl passwd Current password: New password: Confirm new password: Password changed
Project administration
Add project (admin only)
- Preconditions:
- Admin is logged in
- Test steps
- ndslabsctl add project <name> <password>
- ndslabsctl add project -f <project.json>
- ndslabsctl add project --file <project.json>
- Expected results
- Project is added. User can login with specified name and password
- Alternate flows
- Project
- Preconditions:
- Preconditions:
- Admin is logged in
- Test steps
- ndslabsctl add project <name> <password>
- ndslabsctl add project -f <project.json>
- ndslabsctl add project --file <project.json>
- Expected results
- Project is added. User can login with specified name and password
Alternate flowsProject - exists with same name (409 conflict)
- Logged in user is not admin (401 unauthorized)
Sample project file:
...
{
"name": "test",
"description": "testacct",
"
...
namespace": "test",
"
...
password": "
...
123456"
}
Delete project (admin only)
- Preconditions
- Admin is logged in
- Test steps
- ndslabsctl delete project <project>
- Expected results
- Project is deleted
- Alternate flows:
- Project does not exist (404 not found)
- Problem: currently results in 500
- User is not admin (401 unauthorized)
- Resolution:
Should now result in 404. Also, add project will prompt for admin password for add/delete project
Code Block language bash # ndslabsctl add project -f /tmp/project.json Admin password: Added project # ndslabsctl delete project test Admin password: Project test deleted # ndslabsctl delete project testnotexist Admin password: Unable to delete project testnotexist: 404 Not Found
- Preconditions
Delete project (admin only)
- Preconditions
- Admin is logged in
- Test steps
- ndslabsctl delete project <project>
- Expected results
- Project is deleted
Alternate flows: - Project does not exist (404 not found)
- Problem: currently results in 500
User is not admin (401 unauthorized)
Get project details (admin or current project only)
- Preconditions
- Admin is logged in or current project user is logged in
- Test steps
- ndslabsctl get project <project>
- Expected results
- Displays project details (as json)
- Password is redacted
- Alternate flows:
- Project does not exist (404)
- User is not admin and project is not current project (401 unauthorized)
- Preconditions
List projects
- Preconditions
- Admin is logged in
- Test steps
- ndslabsctl list projects
- Expected results
- Projects are listed
- Problem: ID is displayed
- Resolution:
- Now list shows Namespace, Quota, Description
- Preconditions
Service administration
Add service (admin only)
- Preconditions:
- Admin is logged in
- Test steps
- ndslabsctl add service -f <service.json>
- ndslabsctl add service --file <service.json>
- Expected results
- Service is added. User can add service/stack.
- Alternate flows
- Service exists with same key (409 conflict)
- Logged in user is not admin (401 unauthorized)
- Notes:
- See ndslabs-specs repos for sample services
- Preconditions:
Delete service (admin only)
- Preconditions
- Admin is logged in
- Test steps
- ndslabsctl delete service <key>
- Expected results
- Service is deleted
- Alternate flows:
- Project does not exist (404 not found)
- User is not admin (401 unauthorized)
- Not logged in (401)
- Not admin user (401)
- Service is in use by projects (409)
- Service required by other services (409)
- No such service (404)
- Issues:
- Throws a 500 error on delete (wget https://raw.githubusercontent.com/nds-org/ndslabs-specs/master/elk/elastic.json, change key, then add / delete):
- Preconditions
Code Block | ||||
---|---|---|---|---|
| ||||
runtime error: invalid memory address or nil pointer dereference goroutine 7674 [running]: runtime/debug.Stack(0x0, 0x0, 0x0) /usr/local/go/src/runtime/debug/stack.go:24 +0x80 github.com/ant0ine/go-json-rest/rest.(*RecoverMiddleware).MiddlewareFunc.func1.1(0xc820014ee0, 0x7f8379174910, 0xc8203dbb00) /go/src/github.com/ant0ine/go-json-rest/rest/recover.go:41 +0x53 panic(0xf83620, 0xc820014090) /usr/local/go/src/runtime/panic.go:426 +0x4e9 main.(*Server).serviceInUse(0xc820143110, 0xc82027d711, 0x17, 0x0) /go/src/github.com/ndslabs/apiserver/server.go:661 +0x2ed main.(*Server).DeleteService(0xc820143110, 0x7f8379174950, 0xc82027d740, 0xc820527840) /go/src/github.com/ndslabs/apiserver/server.go:642 +0x458 main.(*Server).DeleteService-fm(0x7f8379174950, 0xc82027d740, 0xc820527840) /go/src/github.com/ndslabs/apiserver/server.go:253 +0x3e github.com/ant0ine/go-json-rest/rest.(*router).AppFunc.func1(0x7f8379174950, 0xc82027d740, 0xc820527840) /go/src/github.com/ant0ine/go-json-rest/rest/router.go:56 +0xf1 github.com/StephanDollberg/go-json-rest-middleware-jwt.(*JWTMiddleware).middlewareImpl(0xc8201c6a80, 0x7f8379174950, 0xc82027d740, 0xc820527840, 0xc82025adc0) /go/src/github.com/StephanDollberg/go-json-rest-middleware-jwt/auth_jwt.go:103 +0x2bf github.com/StephanDollberg/go-json-rest-middleware-jwt.(*JWTMiddleware).MiddlewareFunc.func2(0x7f8379174950, 0xc82027d740, 0xc820527840) /go/src/github.com/StephanDollberg/go-json-rest-middleware-jwt/auth_jwt.go:82 +0x47 github.com/ant0ine/go-json-rest/rest.(*IfMiddleware).MiddlewareFunc.func1(0x7f8379174950, 0xc82027d740, 0xc820527840) /go/src/github.com/ant0ine/go-json-rest/rest/if.go:47 +0x6b github.com/ant0ine/go-json-rest/rest.(*CorsMiddleware).MiddlewareFunc.func1(0x7f8379174950, 0xc82027d740, 0xc820527840) /go/src/github.com/ant0ine/go-json-rest/rest/cors.go:86 +0xc2 github.com/ant0ine/go-json-rest/rest.(*ContentTypeCheckerMiddleware).MiddlewareFunc.func1(0x7f8379174950, 0xc82027d740, 0xc820527840) /go/src/github.com/ant0ine/go-json-rest/rest/content_type_checker.go:38 +0x1e9 github.com/ant0ine/go-json-rest/rest.(*JsonIndentMiddleware).MiddlewareFunc.func1(0x7f8379174910, 0xc8203dbb00, 0xc820527840) /go/src/github.com/ant0ine/go-json-rest/rest/json_indent.go:35 +0x118 github.com/ant0ine/go-json-rest/rest.(*RecoverMiddleware).MiddlewareFunc.func1(0x7f8379174910, 0xc8203dbb00, 0xc820527840) /go/src/github.com/ant0ine/go-json-rest/rest/recover.go:57 +0x81 github.com/ant0ine/go-json-rest/rest.(*PoweredByMiddleware).MiddlewareFunc.func1(0x7f8379174910, 0xc8203dbb00, 0xc820527840) /go/src/github.com/ant0ine/go-json-rest/rest/powered_by.go:26 +0xa7 github.com/ant0ine/go-json-rest/rest.(*RecorderMiddleware).MiddlewareFunc.func1(0x7f83791748d0, 0xc820527860, 0xc820527840) /go/src/github.com/ant0ine/go-json-rest/rest/recorder.go:22 +0xbc github.com/ant0ine/go-json-rest/rest.(*TimerMiddleware).MiddlewareFunc.func1(0x7f83791748d0, 0xc820527860, 0xc820527840) /go/src/github.com/ant0ine/go-json-rest/rest/timer.go:20 +0x117 github.com/ant0ine/go-json-rest/rest.(*AccessLogApacheMiddleware).MiddlewareFunc.func1(0x7f83791748d0, 0xc820527860, 0xc820527840) /go/src/github.com/ant0ine/go-json-rest/rest/access_log_apache.go:87 +0x55 github.com/ant0ine/go-json-rest/rest.adapterFunc.func1(0x7f8379174898, 0xc820215790, 0xc82021ae00) /go/src/github.com/ant0ine/go-json-rest/rest/middleware.go:70 +0x132 net/http.HandlerFunc.ServeHTTP(0xc82025b130, 0x7f8379174898, 0xc820215790, 0xc82021ae00) /usr/local/go/src/net/http/server.go:1618 +0x3a net/http.serverHandler.ServeHTTP(0xc820081800, 0x7f8379174898, 0xc820215790, 0xc82021ae00) /usr/local/go/src/net/http/server.go:2081 +0x19e net/http.(*conn).serve(0xc8203fad80) /usr/local/go/src/net/http/server.go:1472 +0xf2e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2137 +0x44e 29/Mar/2016:20:31:55 +0000 500 2890μs "DELETE /services/hellomynameisnewservice HTTP/1.1" admin "Go-http-client/1.1" |
...
- Resolution:
- This was likely caused by logging in as admin user. Since admin user can't login with this version, this shouldn't happen.
- Resolution:
Get service details
- Preconditions
- User is logged in
- Test steps
- ndslabsctl get service <key>
- Expected results
- Displays service spec (as json)
- Alternate flows:
- Service does not exist (404
- Preconditions
List services
- Preconditions
- User is logged in
- Test steps
- ndslabsctl list services
- Expected results
- Services are listed
- Issues:
List services does not seem to show custom services:
Code Block language bash $ etcdctl ls /ndslabs/services /ndslabs/services/rabbitmq /ndslabs/services/dataverse /ndslabs/services/dvicat /ndslabs/services/logspout /ndslabs/services/logstash /ndslabs/services/mongo /ndslabs/services/rserve /ndslabs/services/icat /ndslabs/services/thisisnewservice <<<<< Custom Service added /ndslabs/services/plantcv /ndslabs/services/videopreview /ndslabs/services/solr /ndslabs/services/kibana /ndslabs/services/cloudbrowser /ndslabs/services/cloudbrowserui /ndslabs/services/clowder /ndslabs/services/elasticsearch /ndslabs/services/imagepreview /ndslabs/services/postgres /ndslabs/services/tworavens /ndslabs/services/elasticsearch2 $ ndslabsctl list services SERVICE DEPENDENCY REQUIRED icat cloudbrowserui (optional) kibana elasticsearch2 (required) logstash (required) logspout (required) clowder mongo (required) imagepreview (optional) videopreview (optional) elasticsearch (optional) plantcv (optional) dataverse postgres (required) solr (required) rserve (required) tworavens (optional) dvicat (optional)
- Resolution:
- List command was limiting to only services with IsStack=true.
- Not limits to services with IsStandalone = true
- Preconditions
Volume administration
- Create / delete volume feel a bit inconsistent:
- ndslabsctl create <volumeName> <volumeSize>
ndslabsctl delete volume <volumeName>- Create / delete volume feel a bit inconsistent:
- ndslabsctl delete volume <volumeName>
- Resolution
- Now use "ndslabsctl add volume ..."
- Create is no longer.
Create volume
- Preconditions
- Logged in
- Sufficient quota remaining (not implemented)
- Test steps
- ndslabsctl create <vol name> <vol size GB>
- Creates a detached volume
- ndslabsctl create <vol name> <vol size GB> <sid>
- Creates and attaches volume
- Expected results
- Volume is created
- Volume is created and attached
- Alternate flow
- Not logged in (401)
- Insufficient quota (409) (not implemented)
- Duplicate name (409)
- Stack doesn't exist (404)
- Preconditions
Attach volume
- Preconditions:
- Logged in
- Volume exits
- Service exists with no existing attachments
- Service not running
- Test steps
- ndslabsctl attach <vol name> <sid>
- Expected results
- Volume is attached to specified service
- Alternate flow
- Not logged in (401)
- Volume doesn't exist (404)
- Service doesn't exist (404)
- Service is running (409)
- Volume is attached (409)
- Preconditions:
List volumes
- Preconditions
- Logged in
- Volumes exist
- Test steps
- ndslabsctl list volumes
- Expected results
- Volume details are displayed
- Alterate flow
- Not logged in (401)
- No volumes exist (200)
- Preconditions
Detach volume
- Preconditions
- Logged in
- Volume exists and is attached
- Stack not running
- Test steps
- ndslabsctl detach <vol name>
- Expected results
- Volume is detached
- Alternative flow
- Not logged in (401)
- Volume doesn't exist (404)
- Stack is running (409)
- Volume isn't attached (409)
- Preconditions
Delete volume
- Preconditions
- Logged in
- Volume exists
- Stack not running
- Test case:
- ndslabsctl delete <vol name>
- Expected results:
- Volume is deleted
- Alternative flow
- Not logged in (401)
- Volume doesn't exist (404)
- Stack is running (409)
- Preconditions
Stack administration
- Admin user can create / start / stop stacks. We will need to revisit the workflow for admin.
- Resolution: Admin user can no longer login. User is prompted for admin password for add/delete project/service.
Add stack
- Preconditions
- Logged in
- Service exists
- Test steps
- ndslabsctl add stack <key> <name>
- ndslabsctl add stack <key> <name> --opt=<comma-separated list of optional services>
- Expected results
- Stack is added
- Alternate flow
- Not logged in (401)
- Service doesn't exist (404)
- Duplicate stack name (409)
- Preconditions
List stacks
List stacks
- Preconditions:
- Logged in
- Stack has been added
- Test steps
- ndslabsctl list stacks
- Expected results
- Summary of stacks is displayed (name, services, status, SIDs)
- Alternate flow:
- Not logged in (401)
- No stacks (empty list)
- Preconditions:
Delete stack
- Preconditions:
- Logged in
- Stack exists
- Stack not running
- Test steps
- ndslabsctl delete stack <sid>
- Stack is deleted (201)
- Alternate flow
- Not logged in (401)
- Stack doesn't exist (404)
- Stack is running (409)
- Attached volumes are orphaned
- Preconditions:
Get stack details
- Preconditions:
- Logged in
- Stack exists
- Test steps
- ndslabsctl get stack <sid>
- Expected results
- Stack details, including endpoints and config, are displayed
- Alternate flow
- Not logged in (401)
- Stack doesn't exist (404)
- Preconditions:
Start stack
- Preconditions
- Logged in
- Stack exists and started
- Test steps
- ndslabsctl start <sid>
- Expected results
- All stack services (service, replication controller, pods) are started and ready
- Alternate flow
- Not logged in (401)
- Stack doesn't exist (404)
- Stack not started or stopping (409)
- Issues:
Throws a 500 error, but still seems to send stack to "starting" state
Code Block language bash $ ndslabsctl start sc6c63 Error starting sc6c63: 500 Internal Server Error core@lambert-test ~/ndslabs-startup $ ndslabsctl list stacks STACK SERVICE STATUS SID ELK stopped sb8rvt kibana sb8rvt-kibana elasticsearch2 sb8rvt-elasticsearch2 logstash sb8rvt-logstash logspout sb8rvt-logspout TERRA starting sc6c63 clowder sc6c63-clowder mongo sc6c63-mongo
- Resolution
- None thus far. Hard to repeat case.
- Preconditions
Stop stack
- Preconditions
- Stack started
- Test steps
- ndslabsctl stop <sid>
- Expected results
- Stack service, replication controller, pods are stopped (200)
- Alternative flow
- Not logged in (401)
- Stack does not exist (404)
- Stack not started (200)
- Preconditions
Configuration
Get stack service configuration
- Preconditions
- Logged in
- Stack exists
- Test case:
- ndslabsctl get configs <sid>
- Expected results:
- Current service configuration options and settings are displayed
- Problem: resulting in 500 now
- Alternative flow
- Not logged in (401)
- Stack doesn't exist (404)
- Issues:
Only prints usage:
Code Block language bash $ ndslabsctl get configs sc6c63 Get service or resource details Usage: apictl get [command] Available Commands: service Get service details stack Get stack details project Get project details Global Flags: --config string config file (default is $HOME/.ndslabsctl.yaml) -s, --server string API server host address (default "http://localhost:8083") -v, --verbose Verbose output Use "apictl get [command] --help" for more information about
a command- Preconditions
a command.
- Resolution
- get configs doesn't exist. Use "ndslabsctl get stack <stack id>", config information is printed at the bottom.
Set stack service configuration option
- Preconditions
- Logged in
- Stack exists
- Config exists
- Test case:
- ndslabsctl set <sid> <name> <value>
- Expected results:
- Specified configuration setting os overriden for the service
- Alternative flow
- Not logged in (401)
- Stack doesn't exist (404)
- Config doesn't exist or is not overridable (409)
- Issues:
Throws a stack trace when trying to set config on clowder:
Code Block language bash $ ndslabsctl set sc6c63 TOOLMANAGER_URI
"123.456.789.0:8082"
panic: runtime error: slice bounds out of range
goroutine 1 [running]:
panic(0x8ed380, 0xc820012060)
/usr/local/go/src/runtime/panic.go:464 +0x3e6
github.com/ndslabs/apictl/cmd.glob.func23(0xc198a0, 0xc82000fa40, 0x3, 0x3- Preconditions
"123.456.789.0:8082" panic: runtime error: slice bounds out of range goroutine 1 [running]: panic(0x8ed380, 0xc820012060) /usr/local/go/src/runtime/panic.go:464 +0x3e6 github.com/ndslabs/apictl/cmd.glob.func23(0xc198a0, 0xc82000fa40, 0x3, 0x3) /go/src/github.com/ndslabs/apictl/cmd/set.go:26 +0x1052 github.com/spf13/cobra.(*Command).execute(0xc198a0, 0xc82000f920, 0x3, 0x3, 0x0, 0x0) /go/src/github.com/spf13/cobra/command.go:569 +0x85a github.com/spf13/cobra.(*Command).ExecuteC(0xc196a0, 0xc198a0, 0x0, 0x0) /go/src/github.com/
ndslabsspf13/
apictlcobra/
cmd/setcommand.go:
26656 +
0x10520x55c github.com/spf13/cobra.(*Command).
execute(0xc198a0, 0xc82000f920Execute(0xc196a0,
0x3, 0x3, 0x0, 0x0) /go/src/github.com/spf13/cobra/command.go:
569615 +
0x85a0x2d github.com/
spf13ndslabs/apictl/
cobracmd.Execute(
*Command).ExecuteC(0xc196a0, 0xc198a0, 0x0, 0x0) /go/src/github.com/ndslabs/apictl/cmd/root.go:73 +0x27 main.main() /go/src/github.com/
spf13ndslabs/
cobraapictl/
commandmain.go:
656 +0x55c
github.com/spf13/cobra.(*Command).Execute(0xc196a0, 0x0, 0x0)
/go/src/github.com/spf13/cobra/command.go:615 +0x2d
github.com/ndslabs/apictl/cmd.Execute()
/go/src/github.com/ndslabs/apictl/cmd/root.go:73 +0x27
main.main()
/go/src/github.com/ndslabs/apictl/main.go:12 +0x2512 +0x25
- Resolution:
- This is due to the fact that you're using the stack ID, not the stack service ID.
Code Block #ndslabsctl get stack s80ex8 ... s80ex8-icat RODS_ZONE=test ... #ndslabsctl set s80ex8-icat RODS_ZONE fedZone RODS_ZONE fedZone true # ndslabsctl set s80ex8 RODS_ZONE fedZone Invalid stack service id (looks like a stack Id?): s80ex8
Other
View stack service logs
- Preconditions
- Logged in
- Stack exists
- Stack started
- Config exists
- Test case:
- ndslabsctl logs <sid>
- ndslabsctl logs <sid> --lines=<lines>
- Expected results:
- Logs are displayed
- Last n log lines are displayed
- Alternative flow
- Not logged in (401)
- Stack doesn't exist (404)
- Stack is not running (404)
- Preconditions
Other problems:
- Volumes are referred by name not ID
- Quota stuff