Loading...

Details

Type: Bug
Resolution: Unresolved
Priority: Minor
Fix Version/s: None
Affects Version/s: None
Component/s: Backend
Labels:
None

Sprint:
NDS Sprint 42, NDS Sprint 43

Description

I've noticed a lot of 500 errors coming back from GET /accounts/{account-id} lately:

Request URL:https://www.local.ndslabs.org/api/accounts/lambert8
Request Method:GET
Status Code:500
Remote Address:127.0.0.1:443
Referrer Policy:no-referrer-when-downgrade

Response Headers
cache-control:no-cache, no-store, must-revalidate
content-type:application/json; charset=utf-8
date:Mon, 11 Dec 2017 17:26:21 GMT
expires:0
pragma:no-cache
server:nginx/1.13.7
status:500
strict-transport-security:max-age=15724800; includeSubDomains;
x-powered-by:go-json-rest

Request Headers
:authority:www.local.ndslabs.org
:method:GET
:path:/api/accounts/lambert8
:scheme:https
accept:application/json, text/plain, /
accept-encoding:gzip, deflate, br
accept-language:en-US,en;q=0.9
authorization:Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1MTMwMTQ5ODAsImlkIjoibGFtYmVydDgiLCJvcmlnX2lhdCI6MTUxMzAxMzE3OSwic2VydmVyIjoibmRzbGFicy1hcGlzZXJ2ZXItNTdjNGoiLCJ1c2VyIjoibGFtYmVydDgifQ.gjHRshphACLh0GO_xFA4p9X_Z6x7zbmbD9upeN1TIZA
cache-control:no-cache
cookie:namespace=lambert8; token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1MTMwMTQ5ODAsImlkIjoibGFtYmVydDgiLCJvcmlnX2lhdCI6MTUxMzAxMzE3OSwic2VydmVyIjoibmRzbGFicy1hcGlzZXJ2ZXItNTdjNGoiLCJ1c2VyIjoibGFtYmVydDgifQ.gjHRshphACLh0GO_xFA4p9X_Z6x7zbmbD9upeN1TIZA
dnt:1
pragma:no-cache
referer:https://www.local.ndslabs.org/dashboard/home
user-agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36

Looking at the API server logs, I am getting the following stack trace:

runtime error: index out of range

goroutine 10244 [running]:

runtime/debug.Stack(0xc420799098, 0x103dd00, 0xc4200180a0)

	/usr/local/go/src/runtime/debug/stack.go:24 +0x79

github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*RecoverMiddleware).MiddlewareFunc.func1.1(0xc4203690b0, 0x19c8fe0, 0xc4202043f0)

	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/recover.go:41 +0x6e

panic(0x103dd00, 0xc4200180a0)

	/usr/local/go/src/runtime/panic.go:458 +0x243

main.(*Server).GetAccount(0xc4200e4790, 0x19c8fa0, 0xc420267a00, 0xc4201a2e60)

	/go/src/github.com/ndslabs/apiserver/cmd/server/server.go:547 +0x11fd

main.(*Server).GetAccount-fm(0x19c8fa0, 0xc420267a00, 0xc4201a2e60)

	/go/src/github.com/ndslabs/apiserver/cmd/server/server.go:300 +0x48

github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*router).AppFunc.func1(0x19c8fa0, 0xc420267a00, 0xc4201a2e60)

	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/router.go:56 +0xa4

github.com/ndslabs/apiserver/vendor/github.com/StephanDollberg/go-json-rest-middleware-jwt.(*JWTMiddleware).middlewareImpl(0xc4204b6780, 0x19c8fa0, 0xc420267a00, 0xc4201a2e60, 0xc420430270)

	/go/src/github.com/ndslabs/apiserver/vendor/github.com/StephanDollberg/go-json-rest-middleware-jwt/auth_jwt.go:103 +0x256

github.com/ndslabs/apiserver/vendor/github.com/StephanDollberg/go-json-rest-middleware-jwt.(*JWTMiddleware).MiddlewareFunc.func2(0x19c8fa0, 0xc420267a00, 0xc4201a2e60)

	/go/src/github.com/ndslabs/apiserver/vendor/github.com/StephanDollberg/go-json-rest-middleware-jwt/auth_jwt.go:82 +0x51

github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*IfMiddleware).MiddlewareFunc.func1(0x19c8fa0, 0xc420267a00, 0xc4201a2e60)

	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/if.go:47 +0x78

github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*CorsMiddleware).MiddlewareFunc.func1(0x19c8fa0, 0xc420267a00, 0xc4201a2e60)

	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/cors.go:86 +0x681

github.com/ndslabs/apiserver/pkg/middleware.(*NoCacheMiddleware).MiddlewareFunc.func1(0x19c8fa0, 0xc420267a00, 0xc4201a2e60)

	/go/src/github.com/ndslabs/apiserver/pkg/middleware/no_cache.go:17 +0x135

github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*ContentTypeCheckerMiddleware).MiddlewareFunc.func1(0x19c8fa0, 0xc420267a00, 0xc4201a2e60)

	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/content_type_checker.go:38 +0x1fe

github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*JsonIndentMiddleware).MiddlewareFunc.func1(0x19c8fe0, 0xc4202043f0, 0xc4201a2e60)

	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/json_indent.go:35 +0xd5

github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*RecoverMiddleware).MiddlewareFunc.func1(0x19c8fe0, 0xc4202043f0, 0xc4201a2e60)

	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/recover.go:57 +0x8a

github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*PoweredByMiddleware).MiddlewareFunc.func1(0x19c8fe0, 0xc4202043f0, 0xc4201a2e60)

	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/powered_by.go:26 +0xae

github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*RecorderMiddleware).MiddlewareFunc.func1(0x19c9020, 0xc4201a2e80, 0xc4201a2e60)

	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/recorder.go:22 +0xc4

github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*TimerMiddleware).MiddlewareFunc.func1(0x19c9020, 0xc4201a2e80, 0xc4201a2e60)

	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/timer.go:20 +0x11e

github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*AccessLogApacheMiddleware).MiddlewareFunc.func1(0x19c9020, 0xc4201a2e80, 0xc4201a2e60)

	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/access_log_apache.go:87 +0x54

github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.adapterFunc.func1(0x19c7ea0, 0xc42040e270, 0xc420467e00)

	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/middleware.go:70 +0x10c

net/http.HandlerFunc.ServeHTTP(0xc420430650, 0x19c7ea0, 0xc42040e270, 0xc420467e00)

	/usr/local/go/src/net/http/server.go:1726 +0x44

net/http.(*ServeMux).ServeHTTP(0x1a06c80, 0x19c7ea0, 0xc42040e270, 0xc420467e00)

	/usr/local/go/src/net/http/server.go:2022 +0x7f

net/http.serverHandler.ServeHTTP(0xc420487400, 0x19c7ea0, 0xc42040e270, 0xc420467e00)

	/usr/local/go/src/net/http/server.go:2202 +0x7d

net/http.(*conn).serve(0xc4201f6d80, 0x19c8d20, 0xc420267900)

	/usr/local/go/src/net/http/server.go:1579 +0x4b7

created by net/http.(*Server).Serve

	/usr/local/go/src/net/http/server.go:2293 +0x44d

11/Dec/2017:17:26:21 +0000 500 149761μs "GET /api/accounts/lambert8 HTTP/1.1" lambert8 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36"

I1211 17:26:21.158875      18 server.go:260] remoteAddr: 172.17.0.14:48172

My account object in etcd is as follows:

  "id": "",

  "name": "Mike Lambert",

  "description": "Testing",

  "namespace": "lambert8",

  "email": "lambert8@illinois.edu",

  "password": "$apr1$mAjbjKYF$vFnCo8yZ\/rO0J3egF6rm2.",

  "resourceLimits": {

    "cpuMax": 0,

    "cpuDefault": 0,

    "memMax": 0,

    "memDefault": 0,

    "storageQuota": 0

},

  "resourceUsage": {

    "cpu": "",

    "memory": "",

    "storage": "",

    "cpuPct": "",

    "memPct": ""

},

  "status": "approved",

  "token": "TIY0cPSACM7zYEt7os6KHwtAcFE",

  "organization": "NCSA",

  "created": 0,

  "lastLogin": 1513013481,

  "inactiveTimeout": 480,

  "nexturl": "https:\/\/www.local.ndslabs.org\/#\/home?quickstart=cloudcmd"

Gliffy Diagrams

Attachments

Issue Links

is needed for

NDS-1071 Reprovisioning the Workbench public beta

Open

is related to

NDS-1133 Stack trace starting standard Docker application

Open

NDS-1168 API server stack trace + crash + restart when starting toolmanager

Open

NDS-1170 GET /api/stacks returning 500 when stopping FileManager

Open

relates to

NDS-1125 API server errors and restarts while trying to shutdown inactive service

Open

NDS-998 Intermittent etcd timeouts from apiserver

Resolved

(1 relates to)

Frequent 500 errors from /accounts/{account-id}

Details

Description

Gliffy Diagrams

Attachments

Issue Links

Activity

People

Dates

Tasks