Uploaded image for project: 'National Data Service'
  1. National Data Service
  2. NDS-1170

GET /api/stacks returning 500 when stopping FileManager

    XMLWordPrintableJSON

Details

    • Bug
    • Resolution: Unresolved
    • Normal
    • None
    • None
    • Backend
    • None
    • NDS Sprint 42, NDS Sprint 43

    Description

      Running the dashboard e2e suite against ndslabs/apiserver:latest, I am encountering seemingly intermittent issues auto-refreshing while stopping a FileManager stack... More investigation is needed to determine a cause

      Steps to Reproduce
      Unable to reliably reproduce, but seems to occur every so often when running the dashboard e2e suite.

      Expected Results
      The FileManager should stop/start when requested in a reasonable amount of time

      Actual Results
      GET /stacks appears to occasionally return a 500. Possibly related to NDS-1130

      Verbose log output can be found below:

      E0126 16:32:05.482455      18 etcd.go:254] client: etcd cluster is unavailable or misconfigured; error #0: client: endpoint http://10.0.0.129:4001 exceeded header timeout
      E0126 16:32:05.493502      18 server.go:2308] client: etcd cluster is unavailable or misconfigured; error #0: client: endpoint http://10.0.0.129:4001 exceeded header timeout
      runtime error: invalid memory address or nil pointer dereference
      goroutine 102266 [running]:
      runtime/debug.Stack(0xc4202331a8, 0x10a5560, 0x1a35170)
      	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*RecoverMiddleware).MiddlewareFunc.func1.1(0x1a34db0, 0x1a53ba0, 0xc420c1aa20)
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/recover.go:41 +0x6e
      panic(0x10a5560, 0x1a35170)
      	/usr/local/go/src/runtime/panic.go:489 +0x2cf
      main.(*Server).getStackWithStatus(0xc42008d8c0, 0xc4203528a8, 0x8, 0xc420352be0, 0x6, 0x0, 0x0, 0xc4202b9400)
      	/go/src/github.com/ndslabs/apiserver/cmd/server/server.go:2312 +0x3b9
      main.(*Server).getStacks(0xc42008d8c0, 0xc4203528a8, 0x8, 0x4, 0xc4204b8c08, 0xc4201c0980)
      	/go/src/github.com/ndslabs/apiserver/cmd/server/server.go:1354 +0x169
      main.(*Server).GetAllStacks(0xc42008d8c0, 0x1a53b60, 0xc42081e340, 0xc420695d80)
      	/go/src/github.com/ndslabs/apiserver/cmd/server/server.go:1338 +0xcf
      main.(*Server).GetAllStacks-fm(0x1a53b60, 0xc42081e340, 0xc420695d80)
      	/go/src/github.com/ndslabs/apiserver/cmd/server/server.go:314 +0x48
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*router).AppFunc.func1(0x1a53b60, 0xc42081e340, 0xc420695d80)
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/router.go:56 +0xa3
      github.com/ndslabs/apiserver/vendor/github.com/StephanDollberg/go-json-rest-middleware-jwt.(*JWTMiddleware).middlewareImpl(0xc42045bb00, 0x1a53b60, 0xc42081e340, 0xc420695d80, 0xc420141500)
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/StephanDollberg/go-json-rest-middleware-jwt/auth_jwt.go:103 +0x226
      github.com/ndslabs/apiserver/vendor/github.com/StephanDollberg/go-json-rest-middleware-jwt.(*JWTMiddleware).MiddlewareFunc.func2(0x1a53b60, 0xc42081e340, 0xc420695d80)
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/StephanDollberg/go-json-rest-middleware-jwt/auth_jwt.go:82 +0x51
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*IfMiddleware).MiddlewareFunc.func1(0x1a53b60, 0xc42081e340, 0xc420695d80)
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/if.go:47 +0x78
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*CorsMiddleware).MiddlewareFunc.func1(0x1a53b60, 0xc42081e340, 0xc420695d80)
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/cors.go:86 +0xdc
      github.com/ndslabs/apiserver/pkg/middleware.(*NoCacheMiddleware).MiddlewareFunc.func1(0x1a53b60, 0xc42081e340, 0xc420695d80)
      	/go/src/github.com/ndslabs/apiserver/pkg/middleware/no_cache.go:17 +0x135
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*ContentTypeCheckerMiddleware).MiddlewareFunc.func1(0x1a53b60, 0xc42081e340, 0xc420695d80)
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/content_type_checker.go:38 +0x200
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*JsonIndentMiddleware).MiddlewareFunc.func1(0x1a53ba0, 0xc420c1aa20, 0xc420695d80)
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/json_indent.go:35 +0xd8
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*RecoverMiddleware).MiddlewareFunc.func1(0x1a53ba0, 0xc420c1aa20, 0xc420695d80)
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/recover.go:57 +0x8a
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*PoweredByMiddleware).MiddlewareFunc.func1(0x1a53ba0, 0xc420c1aa20, 0xc420695d80)
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/powered_by.go:26 +0xae
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*RecorderMiddleware).MiddlewareFunc.func1(0x1a53be0, 0xc420695da0, 0xc420695d80)
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/recorder.go:22 +0xa2
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*TimerMiddleware).MiddlewareFunc.func1(0x1a53be0, 0xc420695da0, 0xc420695d80)
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/timer.go:20 +0x122
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*AccessLogApacheMiddleware).MiddlewareFunc.func1(0x1a53be0, 0xc420695da0, 0xc420695d80)
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/access_log_apache.go:87 +0x54
      github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.adapterFunc.func1(0x1a52a60, 0xc42011a9a0, 0xc420a4ad00)
      	/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/middleware.go:70 +0x122
      net/http.HandlerFunc.ServeHTTP(0xc4201418a0, 0x1a52a60, 0xc42011a9a0, 0xc420a4ad00)
      	/usr/local/go/src/net/http/server.go:1942 +0x44
      net/http.serverHandler.ServeHTTP(0xc420164160, 0x1a52a60, 0xc42011a9a0, 0xc420a4ad00)
      	/usr/local/go/src/net/http/server.go:2568 +0x92
      net/http.(*conn).serve(0xc4202dc960, 0x1a538e0, 0xc42081e100)
      	/usr/local/go/src/net/http/server.go:1825 +0x612
      created by net/http.(*Server).Serve
      	/usr/local/go/src/net/http/server.go:2668 +0x2ce
      26/Jan/2018:16:32:04 +0000 500 1478679μs "GET /api/stacks HTTP/1.1" lambert8 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
      

      Gliffy Diagrams

        Attachments

          Issue Links

            Activity

              People

                Unassigned Unassigned
                lambert8 Sara Lambert
                Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                  Created:
                  Updated:

                  Tasks