Details
-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
None
-
None
-
NDS Sprint 42, NDS Sprint 43
Description
I've noticed a lot of 500 errors coming back from GET /accounts/{account-id} lately:
Request URL:https://www.local.ndslabs.org/api/accounts/lambert8
Request Method:GET
Status Code:500
Remote Address:127.0.0.1:443
Referrer Policy:no-referrer-when-downgrade
Response Headers
cache-control:no-cache, no-store, must-revalidate
content-type:application/json; charset=utf-8
date:Mon, 11 Dec 2017 17:26:21 GMT
expires:0
pragma:no-cache
server:nginx/1.13.7
status:500
strict-transport-security:max-age=15724800; includeSubDomains;
x-powered-by:go-json-rest
Request Headers
:authority:www.local.ndslabs.org
:method:GET
:path:/api/accounts/lambert8
:scheme:https
accept:application/json, text/plain, /
accept-encoding:gzip, deflate, br
accept-language:en-US,en;q=0.9
authorization:Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1MTMwMTQ5ODAsImlkIjoibGFtYmVydDgiLCJvcmlnX2lhdCI6MTUxMzAxMzE3OSwic2VydmVyIjoibmRzbGFicy1hcGlzZXJ2ZXItNTdjNGoiLCJ1c2VyIjoibGFtYmVydDgifQ.gjHRshphACLh0GO_xFA4p9X_Z6x7zbmbD9upeN1TIZA
cache-control:no-cache
cookie:namespace=lambert8; token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1MTMwMTQ5ODAsImlkIjoibGFtYmVydDgiLCJvcmlnX2lhdCI6MTUxMzAxMzE3OSwic2VydmVyIjoibmRzbGFicy1hcGlzZXJ2ZXItNTdjNGoiLCJ1c2VyIjoibGFtYmVydDgifQ.gjHRshphACLh0GO_xFA4p9X_Z6x7zbmbD9upeN1TIZA
dnt:1
pragma:no-cache
referer:https://www.local.ndslabs.org/dashboard/home
user-agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36
Looking at the API server logs, I am getting the following stack trace:
runtime error: index out of range
|
goroutine 10244 [running]:
|
runtime/debug.Stack(0xc420799098, 0x103dd00, 0xc4200180a0)
|
/usr/local/go/src/runtime/debug/stack.go:24 +0x79
|
github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*RecoverMiddleware).MiddlewareFunc.func1.1(0xc4203690b0, 0x19c8fe0, 0xc4202043f0)
|
/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/recover.go:41 +0x6e
|
panic(0x103dd00, 0xc4200180a0)
|
/usr/local/go/src/runtime/panic.go:458 +0x243
|
main.(*Server).GetAccount(0xc4200e4790, 0x19c8fa0, 0xc420267a00, 0xc4201a2e60)
|
/go/src/github.com/ndslabs/apiserver/cmd/server/server.go:547 +0x11fd
|
main.(*Server).GetAccount-fm(0x19c8fa0, 0xc420267a00, 0xc4201a2e60)
|
/go/src/github.com/ndslabs/apiserver/cmd/server/server.go:300 +0x48
|
github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*router).AppFunc.func1(0x19c8fa0, 0xc420267a00, 0xc4201a2e60)
|
/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/router.go:56 +0xa4
|
github.com/ndslabs/apiserver/vendor/github.com/StephanDollberg/go-json-rest-middleware-jwt.(*JWTMiddleware).middlewareImpl(0xc4204b6780, 0x19c8fa0, 0xc420267a00, 0xc4201a2e60, 0xc420430270)
|
/go/src/github.com/ndslabs/apiserver/vendor/github.com/StephanDollberg/go-json-rest-middleware-jwt/auth_jwt.go:103 +0x256
|
github.com/ndslabs/apiserver/vendor/github.com/StephanDollberg/go-json-rest-middleware-jwt.(*JWTMiddleware).MiddlewareFunc.func2(0x19c8fa0, 0xc420267a00, 0xc4201a2e60)
|
/go/src/github.com/ndslabs/apiserver/vendor/github.com/StephanDollberg/go-json-rest-middleware-jwt/auth_jwt.go:82 +0x51
|
github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*IfMiddleware).MiddlewareFunc.func1(0x19c8fa0, 0xc420267a00, 0xc4201a2e60)
|
/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/if.go:47 +0x78
|
github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*CorsMiddleware).MiddlewareFunc.func1(0x19c8fa0, 0xc420267a00, 0xc4201a2e60)
|
/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/cors.go:86 +0x681
|
github.com/ndslabs/apiserver/pkg/middleware.(*NoCacheMiddleware).MiddlewareFunc.func1(0x19c8fa0, 0xc420267a00, 0xc4201a2e60)
|
/go/src/github.com/ndslabs/apiserver/pkg/middleware/no_cache.go:17 +0x135
|
github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*ContentTypeCheckerMiddleware).MiddlewareFunc.func1(0x19c8fa0, 0xc420267a00, 0xc4201a2e60)
|
/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/content_type_checker.go:38 +0x1fe
|
github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*JsonIndentMiddleware).MiddlewareFunc.func1(0x19c8fe0, 0xc4202043f0, 0xc4201a2e60)
|
/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/json_indent.go:35 +0xd5
|
github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*RecoverMiddleware).MiddlewareFunc.func1(0x19c8fe0, 0xc4202043f0, 0xc4201a2e60)
|
/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/recover.go:57 +0x8a
|
github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*PoweredByMiddleware).MiddlewareFunc.func1(0x19c8fe0, 0xc4202043f0, 0xc4201a2e60)
|
/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/powered_by.go:26 +0xae
|
github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*RecorderMiddleware).MiddlewareFunc.func1(0x19c9020, 0xc4201a2e80, 0xc4201a2e60)
|
/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/recorder.go:22 +0xc4
|
github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*TimerMiddleware).MiddlewareFunc.func1(0x19c9020, 0xc4201a2e80, 0xc4201a2e60)
|
/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/timer.go:20 +0x11e
|
github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*AccessLogApacheMiddleware).MiddlewareFunc.func1(0x19c9020, 0xc4201a2e80, 0xc4201a2e60)
|
/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/access_log_apache.go:87 +0x54
|
github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.adapterFunc.func1(0x19c7ea0, 0xc42040e270, 0xc420467e00)
|
/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/middleware.go:70 +0x10c
|
net/http.HandlerFunc.ServeHTTP(0xc420430650, 0x19c7ea0, 0xc42040e270, 0xc420467e00)
|
/usr/local/go/src/net/http/server.go:1726 +0x44
|
net/http.(*ServeMux).ServeHTTP(0x1a06c80, 0x19c7ea0, 0xc42040e270, 0xc420467e00)
|
/usr/local/go/src/net/http/server.go:2022 +0x7f
|
net/http.serverHandler.ServeHTTP(0xc420487400, 0x19c7ea0, 0xc42040e270, 0xc420467e00)
|
/usr/local/go/src/net/http/server.go:2202 +0x7d
|
net/http.(*conn).serve(0xc4201f6d80, 0x19c8d20, 0xc420267900)
|
/usr/local/go/src/net/http/server.go:1579 +0x4b7
|
created by net/http.(*Server).Serve
|
/usr/local/go/src/net/http/server.go:2293 +0x44d
|
11/Dec/2017:17:26:21 +0000 500 149761μs "GET /api/accounts/lambert8 HTTP/1.1" lambert8 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36"
|
I1211 17:26:21.158875 18 server.go:260] remoteAddr: 172.17.0.14:48172
|
My account object in etcd is as follows:
|
"id": "", |
"name": "Mike Lambert", |
"description": "Testing", |
"namespace": "lambert8", |
"email": "lambert8@illinois.edu", |
"password": "$apr1$mAjbjKYF$vFnCo8yZ\/rO0J3egF6rm2.", |
"resourceLimits": { |
"cpuMax": 0, |
"cpuDefault": 0, |
"memMax": 0, |
"memDefault": 0, |
"storageQuota": 0 |
},
|
"resourceUsage": { |
"cpu": "", |
"memory": "", |
"storage": "", |
"cpuPct": "", |
"memPct": "" |
},
|
"status": "approved", |
"token": "TIY0cPSACM7zYEt7os6KHwtAcFE", |
"organization": "NCSA", |
"created": 0, |
"lastLogin": 1513013481, |
"inactiveTimeout": 480, |
"nexturl": "https:\/\/www.local.ndslabs.org\/#\/home?quickstart=cloudcmd" |
}
|
Gliffy Diagrams
Attachments
Issue Links
- is needed for
-
NDS-1071 Reprovisioning the Workbench public beta
-
- Open
-
- is related to
-
NDS-1133 Stack trace starting standard Docker application
-
- Open
-
-
NDS-1168 API server stack trace + crash + restart when starting toolmanager
-
- Open
-
-
NDS-1170 GET /api/stacks returning 500 when stopping FileManager
-
- Open
-
- relates to
-
NDS-1125 API server errors and restarts while trying to shutdown inactive service
-
- Open
-
-
NDS-998 Intermittent etcd timeouts from apiserver
-
- Resolved
-