Uploaded image for project: 'National Data Service'
  1. National Data Service
  2. NDS-1130

Frequent 500 errors from /accounts/{account-id}

XMLWordPrintableJSON

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • None
    • None
    • Backend
    • None
    • NDS Sprint 42, NDS Sprint 43

      I've noticed a lot of 500 errors coming back from GET /accounts/{account-id} lately:

      Request URL:https://www.local.ndslabs.org/api/accounts/lambert8
      Request Method:GET
      Status Code:500
      Remote Address:127.0.0.1:443
      Referrer Policy:no-referrer-when-downgrade

      Response Headers
      cache-control:no-cache, no-store, must-revalidate
      content-type:application/json; charset=utf-8
      date:Mon, 11 Dec 2017 17:26:21 GMT
      expires:0
      pragma:no-cache
      server:nginx/1.13.7
      status:500
      strict-transport-security:max-age=15724800; includeSubDomains;
      x-powered-by:go-json-rest

      Request Headers
      :authority:www.local.ndslabs.org
      :method:GET
      :path:/api/accounts/lambert8
      :scheme:https
      accept:application/json, text/plain, /
      accept-encoding:gzip, deflate, br
      accept-language:en-US,en;q=0.9
      authorization:Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1MTMwMTQ5ODAsImlkIjoibGFtYmVydDgiLCJvcmlnX2lhdCI6MTUxMzAxMzE3OSwic2VydmVyIjoibmRzbGFicy1hcGlzZXJ2ZXItNTdjNGoiLCJ1c2VyIjoibGFtYmVydDgifQ.gjHRshphACLh0GO_xFA4p9X_Z6x7zbmbD9upeN1TIZA
      cache-control:no-cache
      cookie:namespace=lambert8; token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1MTMwMTQ5ODAsImlkIjoibGFtYmVydDgiLCJvcmlnX2lhdCI6MTUxMzAxMzE3OSwic2VydmVyIjoibmRzbGFicy1hcGlzZXJ2ZXItNTdjNGoiLCJ1c2VyIjoibGFtYmVydDgifQ.gjHRshphACLh0GO_xFA4p9X_Z6x7zbmbD9upeN1TIZA
      dnt:1
      pragma:no-cache
      referer:https://www.local.ndslabs.org/dashboard/home
      user-agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36

      Looking at the API server logs, I am getting the following stack trace:

      runtime error: index out of range
      		 
      goroutine 10244 [running]:
      		 
      runtime/debug.Stack(0xc420799098, 0x103dd00, 0xc4200180a0)
      		 
      	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
      		 
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*RecoverMiddleware).MiddlewareFunc.func1.1(0xc4203690b0, 0x19c8fe0, 0xc4202043f0)
      		 
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/recover.go:41 +0x6e
      		 
      panic(0x103dd00, 0xc4200180a0)
      		 
      	/usr/local/go/src/runtime/panic.go:458 +0x243
      		 
      main.(*Server).GetAccount(0xc4200e4790, 0x19c8fa0, 0xc420267a00, 0xc4201a2e60)
      		 
      	/go/src/github.com/ndslabs/apiserver/cmd/server/server.go:547 +0x11fd
      		 
      main.(*Server).GetAccount-fm(0x19c8fa0, 0xc420267a00, 0xc4201a2e60)
      		 
      	/go/src/github.com/ndslabs/apiserver/cmd/server/server.go:300 +0x48
      		 
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*router).AppFunc.func1(0x19c8fa0, 0xc420267a00, 0xc4201a2e60)
      		 
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/router.go:56 +0xa4
      		 
      github.com/ndslabs/apiserver/vendor/github.com/StephanDollberg/go-json-rest-middleware-jwt.(*JWTMiddleware).middlewareImpl(0xc4204b6780, 0x19c8fa0, 0xc420267a00, 0xc4201a2e60, 0xc420430270)
      		 
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/StephanDollberg/go-json-rest-middleware-jwt/auth_jwt.go:103 +0x256
      		 
      github.com/ndslabs/apiserver/vendor/github.com/StephanDollberg/go-json-rest-middleware-jwt.(*JWTMiddleware).MiddlewareFunc.func2(0x19c8fa0, 0xc420267a00, 0xc4201a2e60)
      		 
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/StephanDollberg/go-json-rest-middleware-jwt/auth_jwt.go:82 +0x51
      		 
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*IfMiddleware).MiddlewareFunc.func1(0x19c8fa0, 0xc420267a00, 0xc4201a2e60)
      		 
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/if.go:47 +0x78
      		 
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*CorsMiddleware).MiddlewareFunc.func1(0x19c8fa0, 0xc420267a00, 0xc4201a2e60)
      		 
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/cors.go:86 +0x681
      		 
      github.com/ndslabs/apiserver/pkg/middleware.(*NoCacheMiddleware).MiddlewareFunc.func1(0x19c8fa0, 0xc420267a00, 0xc4201a2e60)
      		 
      	/go/src/github.com/ndslabs/apiserver/pkg/middleware/no_cache.go:17 +0x135
      		 
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*ContentTypeCheckerMiddleware).MiddlewareFunc.func1(0x19c8fa0, 0xc420267a00, 0xc4201a2e60)
      		 
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/content_type_checker.go:38 +0x1fe
      		 
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*JsonIndentMiddleware).MiddlewareFunc.func1(0x19c8fe0, 0xc4202043f0, 0xc4201a2e60)
      		 
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/json_indent.go:35 +0xd5
      		 
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*RecoverMiddleware).MiddlewareFunc.func1(0x19c8fe0, 0xc4202043f0, 0xc4201a2e60)
      		 
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/recover.go:57 +0x8a
      		 
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*PoweredByMiddleware).MiddlewareFunc.func1(0x19c8fe0, 0xc4202043f0, 0xc4201a2e60)
      		 
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/powered_by.go:26 +0xae
      		 
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*RecorderMiddleware).MiddlewareFunc.func1(0x19c9020, 0xc4201a2e80, 0xc4201a2e60)
      		 
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/recorder.go:22 +0xc4
      		 
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*TimerMiddleware).MiddlewareFunc.func1(0x19c9020, 0xc4201a2e80, 0xc4201a2e60)
      		 
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/timer.go:20 +0x11e
      		 
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*AccessLogApacheMiddleware).MiddlewareFunc.func1(0x19c9020, 0xc4201a2e80, 0xc4201a2e60)
      		 
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/access_log_apache.go:87 +0x54
      		 
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.adapterFunc.func1(0x19c7ea0, 0xc42040e270, 0xc420467e00)
      		 
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/middleware.go:70 +0x10c
      		 
      net/http.HandlerFunc.ServeHTTP(0xc420430650, 0x19c7ea0, 0xc42040e270, 0xc420467e00)
      		 
      	/usr/local/go/src/net/http/server.go:1726 +0x44
      		 
      net/http.(*ServeMux).ServeHTTP(0x1a06c80, 0x19c7ea0, 0xc42040e270, 0xc420467e00)
      		 
      	/usr/local/go/src/net/http/server.go:2022 +0x7f
      		 
      net/http.serverHandler.ServeHTTP(0xc420487400, 0x19c7ea0, 0xc42040e270, 0xc420467e00)
      		 
      	/usr/local/go/src/net/http/server.go:2202 +0x7d
      		 
      net/http.(*conn).serve(0xc4201f6d80, 0x19c8d20, 0xc420267900)
      		 
      	/usr/local/go/src/net/http/server.go:1579 +0x4b7
      		 
      created by net/http.(*Server).Serve
      		 
      	/usr/local/go/src/net/http/server.go:2293 +0x44d
      		 
      11/Dec/2017:17:26:21 +0000 500 149761μs "GET /api/accounts/lambert8 HTTP/1.1" lambert8 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36"
      		 
      I1211 17:26:21.158875      18 server.go:260] remoteAddr: 172.17.0.14:48172

      My account object in etcd is as follows:

       
      		 
        "id": "",
      		 
        "name": "Mike Lambert",
      		 
        "description": "Testing",
      		 
        "namespace": "lambert8",
      		 
        "email": "lambert8@illinois.edu",
      		 
        "password": "$apr1$mAjbjKYF$vFnCo8yZ\/rO0J3egF6rm2.",
      		 
        "resourceLimits": {
      		 
          "cpuMax": 0,
      		 
          "cpuDefault": 0,
      		 
          "memMax": 0,
      		 
          "memDefault": 0,
      		 
          "storageQuota": 0
      		 
        },
      		 
        "resourceUsage": {
      		 
          "cpu": "",
      		 
          "memory": "",
      		 
          "storage": "",
      		 
          "cpuPct": "",
      		 
          "memPct": ""
      		 
        },
      		 
        "status": "approved",
      		 
        "token": "TIY0cPSACM7zYEt7os6KHwtAcFE",
      		 
        "organization": "NCSA",
      		 
        "created": 0,
      		 
        "lastLogin": 1513013481,
      		 
        "inactiveTimeout": 480,
      		 
        "nexturl": "https:\/\/www.local.ndslabs.org\/#\/home?quickstart=cloudcmd"
      		 
      }

              Unassigned Unassigned
              lambert8 Sara Lambert
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: