Uploaded image for project: 'National Data Service'
  1. National Data Service
  2. NDS-1168

API server stack trace + crash + restart when starting toolmanager

XMLWordPrintableJSON

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • None
    • Backend
    • None
    • NDS Sprint 42, NDS Sprint 43

      Steps to Reproduce
      Could not reproduce reliably - I had started a toolmanager stack during testing today and the API server crashed with an odd stack trace. Performing the same steps again, the stack started up just fine without crashing the apiserver pod.

      Expected Results
      All catalog specs should be able to be added and started reliably

      Actual Results
      ndslabs-toolmanager went into the "Starting" state, but just before it was about to change to "Started", the API server crashed with a stack trace:

      panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x41f2a7]
       
      goroutine 15096 [running]:
      panic(0x103e0c0, 0xc420018060)
      	/usr/local/go/src/runtime/panic.go:500 +0x1a1
      main.(*Server).startController(0xc420094a50, 0xc42044b2a8, 0x8, 0xc42042ca60, 0xb, 0xc4203dad90, 0xc420026638, 0x46afa3, 0xfca400, 0x100000000004608)
      	/go/src/github.com/ndslabs/apiserver/cmd/server/server.go:2018 +0x26f7
      created by main.(*Server).startStack
      	/go/src/github.com/ndslabs/apiserver/cmd/server/server.go:2215 +0x1344
      

      Verbose log output surrounding this scenario can be found below:

      22/Jan/2018:23:16:44 +0000 200 114μs "GET /api/version HTTP/1.1" - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
      I0122 23:17:12.010988      18 server.go:261] remoteAddr: 172.17.0.1:56502
      {
          "kind": "ReplicationController",
          "apiVersion": "v1",
          "metadata": {
              "name": "s5vj4l-toolmanager",
              "creationTimestamp": null,
              "labels": {
                  "name": "s5vj4l-toolmanager",
                  "service": "toolmanager",
                  "stack": "s5vj4l"
              }
          },
          "spec": {
              "replicas": 1,
              "selector": {
                  "name": "s5vj4l-toolmanager"
              },
              "template": {
                  "metadata": {
                      "creationTimestamp": null,
                      "labels": {
                          "name": "s5vj4l-toolmanager",
                          "service": "toolmanager",
                          "stack": "s5vj4l"
                      }
                  },
                  "spec": {
                      "volumes": [
                          {
                              "name": "home",
                              "hostPath": {
                                  "path": "/ndslabs/data/volumes/lambert8"
                              }
                          },
                          {
                              "name": "shared",
                              "hostPath": {
                                  "path": "/shared"
                              }
                          }
                      ],
                      "containers": [
                          {
                              "name": "toolmanager",
                              "image": "ndslabs/toolmanager:latest",
                              "ports": [
                                  {
                                      "containerPort": 8082
                                  }
                              ],
                              "env": [
                                  {
                                      "name": "NDSLABS_HOSTNAME",
                                      "value": "s5vj4l-toolmanager.local.ndslabs.org"
                                  },
                                  {
                                      "name": "NDSLABS_DOMAIN",
                                      "value": "local.ndslabs.org"
                                  },
                                  {
                                      "name": "NDSLABS_EMAIL",
                                      "value": "lambert8@illinois.edu"
                                  },
                                  {
                                      "name": "NDSLABS_STACK",
                                      "value": "s5vj4l"
                                  },
                                  {
                                      "name": "NAMESPACE",
                                      "value": "lambert8"
                                  },
                                  {
                                      "name": "NDSLABS_HOME",
                                      "value": "/home/lambert8"
                                  },
                                  {
                                      "name": "TERM",
                                      "value": "linux"
                                  },
                                  {
                                      "name": "COLUMNS",
                                      "value": "100"
                                  },
                                  {
                                      "name": "LINES",
                                      "value": "30"
                                  },
                                  {
                                      "name": "NDSLABS_SMTP_HOST",
                                      "value": "10.0.0.51"
                                  }
                              ],
                              "resources": {
                                  "limits": {
                                      "cpu": "1",
                                      "memory": "2G"
                                  },
                                  "requests": {
                                      "cpu": "100m",
                                      "memory": "1G"
                                  }
                              },
                              "volumeMounts": [
                                  {
                                      "name": "home",
                                      "mountPath": "/home/lambert8"
                                  },
                                  {
                                      "name": "shared",
                                      "mountPath": "/shared"
                                  }
                              ],
                              "readinessProbe": {
                                  "tcpSocket": {
                                      "port": 8082
                                  },
                                  "initialDelaySeconds": 30,
                                  "failureThreshold": 3
                              },
                              "imagePullPolicy": "Always",
                              "securityContext": {
                                  "privileged": true
                              }
                          }
                      ],
                      "nodeSelector": {
                          "ndslabs-role-compute": "true"
                      },
                      "serviceAccountName": ""
                  }
              }
          },
          "status": {
              "replicas": 0
          }
      }
      I0122 23:17:19.307578      18 server.go:261] remoteAddr: 172.17.0.1:56572
      I0122 23:17:19.308008      18 server.go:261] remoteAddr: 172.17.0.1:56592
      22/Jan/2018:23:17:19 +0000 200 66μs "GET /api/version HTTP/1.1" - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
      22/Jan/2018:23:17:19 +0000 200 422196μs "GET /api/stacks HTTP/1.1" lambert8 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
      panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x41f2a7]
       
      goroutine 15096 [running]:
      panic(0x103e0c0, 0xc420018060)
      	/usr/local/go/src/runtime/panic.go:500 +0x1a1
      main.(*Server).startController(0xc420094a50, 0xc42044b2a8, 0x8, 0xc42042ca60, 0xb, 0xc4203dad90, 0xc420026638, 0x46afa3, 0xfca400, 0x100000000004608)
      	/go/src/github.com/ndslabs/apiserver/cmd/server/server.go:2018 +0x26f7
      created by main.(*Server).startStack
      	/go/src/github.com/ndslabs/apiserver/cmd/server/server.go:2215 +0x1344
      

              Unassigned Unassigned
              lambert8 Sara Lambert
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: