Uploaded image for project: 'National Data Service'
  1. National Data Service
  2. NDS-580

API server "latest" image will not start correctly

    XMLWordPrintableJSON

Details

    • Bug
    • Resolution: Not a Bug
    • Normal
    • Labs Workbench - Beta
    • None
    • Backend
    • None

    Description

      When trying to boot the "latest" API server image, I am getting a stack trace on my single-node (and yes, ingress is still enabled):

      core@lambert-dev ~ $ kubectl logs ndslabs-apiserver-dsry4
      Cloning into '/specs'...
      I0922 15:33:14.459679      22 etcd.go:219] GetEtcdClient 192.168.100.202:4001
      I0922 15:33:14.460812      22 kube.go:63] Reading token from /run/secrets/kubernetes.io/serviceaccount/token
      I0922 15:33:14.500596      22 server.go:203] Starting NDS Labs API server (1.0.5  2016-09-21 14:58)
      I0922 15:33:14.500617      22 server.go:204] etcd 192.168.100.202:4001 
      I0922 15:33:14.500620      22 server.go:205] kube-apiserver https://192.168.100.202:6443
      I0922 15:33:14.500623      22 server.go:206] volume dir /ndslabs/data/volumes
      I0922 15:33:14.500625      22 server.go:207] specs dir /specs
      I0922 15:33:14.500628      22 server.go:208] port 30001
      I0922 15:33:14.500644      22 server.go:215] prefix /api/
      I0922 15:33:14.500647      22 server.go:218] CORS origin http://www.mldev.ndslabs.org
      I0922 15:33:14.500653      22 server.go:238] session timeout 30m0s
      I0922 15:33:14.500661      22 server.go:240] domain www.mldev.ndslabs.org
      I0922 15:33:14.500664      22 server.go:241] ingress LoadBalancer
      I0922 15:33:14.500982      22 server.go:345] Loading service specs from /specs
      I0922 15:33:14.501483      22 server.go:2293] Adding /specs/cantaloupe/cantaloupe.json
      I0922 15:33:14.502585      22 server.go:2293] Adding /specs/clowder/clowder-pycharm-xpra.json
      I0922 15:33:14.503520      22 server.go:2293] Adding /specs/clowder/clowder.json
      I0922 15:33:14.504693      22 server.go:2293] Adding /specs/clowder/extractors/audio-preview.json
      I0922 15:33:14.505634      22 server.go:2293] Adding /specs/clowder/extractors/audio-speech2text.json
      I0922 15:33:14.506481      22 server.go:2293] Adding /specs/clowder/extractors/image-metadata.json
      I0922 15:33:14.507380      22 server.go:2293] Adding /specs/clowder/extractors/image-preview.json
      I0922 15:33:14.508438      22 server.go:2293] Adding /specs/clowder/extractors/pdf-preview.json
      I0922 15:33:14.509478      22 server.go:2293] Adding /specs/clowder/extractors/plantcv.json
      I0922 15:33:14.510306      22 server.go:2293] Adding /specs/clowder/extractors/video-preview.json
      I0922 15:33:14.511353      22 server.go:2293] Adding /specs/dataverse/dataverse.json
      I0922 15:33:14.512594      22 server.go:2293] Adding /specs/dataverse/dvicat.json
      I0922 15:33:14.513676      22 server.go:2293] Adding /specs/dataverse/rserve.json
      I0922 15:33:14.514514      22 server.go:2293] Adding /specs/dataverse/solr.json
      I0922 15:33:14.515484      22 server.go:2293] Adding /specs/dataverse/tworavens.json
      I0922 15:33:14.516481      22 server.go:2293] Adding /specs/devenvs/cloud9-cpp.json
      I0922 15:33:14.517277      22 server.go:2293] Adding /specs/devenvs/cloud9-go.json
      I0922 15:33:14.518277      22 server.go:2293] Adding /specs/devenvs/cloud9-java.json
      I0922 15:33:14.519159      22 server.go:2293] Adding /specs/devenvs/cloud9-nodejs.json
      I0922 15:33:14.520069      22 server.go:2293] Adding /specs/devenvs/cloud9-php.json
      I0922 15:33:14.520859      22 server.go:2293] Adding /specs/devenvs/pycharm-xpra.json
      I0922 15:33:14.521774      22 server.go:2293] Adding /specs/docker/docker.json
      I0922 15:33:14.522676      22 server.go:2293] Adding /specs/dspace/dspace.json
      I0922 15:33:14.523817      22 server.go:2293] Adding /specs/elasticsearch/elastic.json
      I0922 15:33:14.524786      22 server.go:2293] Adding /specs/elasticsearch/elastic2.json
      I0922 15:33:14.525816      22 server.go:2293] Adding /specs/fedora/commons.json
      I0922 15:33:14.526537      22 server.go:2293] Adding /specs/girder/girder.json
      I0922 15:33:14.527370      22 server.go:2293] Adding /specs/iqvoc/iqvoc.json
      I0922 15:33:14.528519      22 server.go:2293] Adding /specs/irods/cloudbrowser.json
      I0922 15:33:14.529884      22 server.go:2293] Adding /specs/irods/cloudbrowserui.json
      I0922 15:33:14.530760      22 server.go:2293] Adding /specs/irods/icat.json
      I0922 15:33:14.532004      22 server.go:2293] Adding /specs/jenkins/jenkins.json
      I0922 15:33:14.532655      22 server.go:2293] Adding /specs/jupyter/dsnotebook.json
      I0922 15:33:14.533606      22 server.go:2293] Adding /specs/jupyter/minimal.json
      I0922 15:33:14.534530      22 server.go:2293] Adding /specs/kibana/kibana.json
      I0922 15:33:14.536016      22 server.go:2293] Adding /specs/logstash/logspout.json
      I0922 15:33:14.537048      22 server.go:2293] Adding /specs/logstash/logstash.json
      I0922 15:33:14.538247      22 server.go:2293] Adding /specs/mongo/mongo.json
      I0922 15:33:14.539604      22 server.go:2293] Adding /specs/mysql/mysql.json
      I0922 15:33:14.540593      22 server.go:2293] Adding /specs/ontowiki/ontowiki.json
      I0922 15:33:14.542392      22 server.go:2293] Adding /specs/owncloud/owncloud.json
      I0922 15:33:14.544470      22 server.go:2293] Adding /specs/postgres/postgres.json
      I0922 15:33:14.545475      22 server.go:2293] Adding /specs/protege/webprotege.json
      I0922 15:33:14.546807      22 server.go:2293] Adding /specs/rabbitmq/rabbitmq.json
      I0922 15:33:14.548145      22 server.go:2293] Adding /specs/rstudio/rstudio.json
      I0922 15:33:14.549163      22 server.go:2293] Adding /specs/system/chisel.json
      I0922 15:33:14.550142      22 server.go:2293] Adding /specs/system/cloudcmd.json
      I0922 15:33:14.551452      22 server.go:2466] Adding vocabulary /specs/vocab/tags.json
      I0922 15:33:14.552748      22 server.go:2576] Creating admin user
      I0922 15:33:14.552769      22 etcd.go:280] GetAccounts()
      I0922 15:33:14.553107      22 etcd.go:289] node.Key /ndslabs/accounts/lambert8
      I0922 15:33:14.553669      22 etcd.go:289] node.Key /ndslabs/accounts/admin
      I0922 15:33:14.553953      22 etcd.go:36] GetAccount /ndslabs//accounts/admin/account
      I0922 15:33:14.572628      22 etcd.go:36] GetAccount /ndslabs//accounts/admin/account
      I0922 15:33:14.573019      22 kube.go:1436] https://192.168.100.202:6443/api/v1/namespaces/kube-system/secrets/basic-auth
      I0922 15:33:14.574244      22 kube.go:1407] https://192.168.100.202:6443/api/v1/namespaces/kube-system/secrets/basic-auth
      I0922 15:33:14.576644      22 kube.go:1417] Deleted secret kube-system basic-auth
      I0922 15:33:14.576749      22 kube.go:1377] https://192.168.100.202:6443/api/v1/namespaces/kube-system/secrets
      I0922 15:33:14.578922      22 kube.go:1387] Added secret kube-system basic-auth
      I0922 15:33:14.578970      22 kube.go:1277] https://192.168.100.202:6443/apis/extensions/v1beta1/namespaces/kube-system/ingresses
      I0922 15:33:14.580919      22 server.go:618] Touching ingress kube-lma-ingress
      panic: assignment to entry in nil map
       
      goroutine 1 [running]:
      panic(0x112c560, 0xc820428b90)
      	/golang/go/src/runtime/panic.go:464 +0x3e6
      github.com/ndslabs/apiserver/pkg/kube.(*KubeHelper).CreateUpdateIngress(0xc820094730, 0x15fd920, 0xb, 0xc820166140, 0x1, 0x1, 0x0, 0x0)
      	/go/src/github.com/ndslabs/apiserver/pkg/kube/kube.go:1207 +0x11a
      main.(*Server).updateIngress(0xc820172000, 0x15fd920, 0xb, 0x0, 0x0)
      	/go/src/github.com/ndslabs/apiserver/cmd/server/server.go:619 +0x395
      main.(*Server).createLMABasicAuthSecret(0xc820172000, 0x0, 0x0)
      	/go/src/github.com/ndslabs/apiserver/cmd/server/server.go:645 +0x334
      main.(*Server).createAdminUser(0xc820172000, 0x7ffc43e6c953, 0x1e, 0x0, 0x0)
      	/go/src/github.com/ndslabs/apiserver/cmd/server/server.go:2615 +0x409
      main.(*Server).start(0xc820172000, 0xc820429508, 0x5, 0xc820410c00, 0x1c, 0xc820410c40, 0x15, 0xc820429c20, 0x6, 0xc820429e90, ...)
      	/go/src/github.com/ndslabs/apiserver/cmd/server/server.go:354 +0x53bc
      main.main()
      	/go/src/github.com/ndslabs/apiserver/cmd/server/server.go:197 +0xd65

      More investigation is needed into the cause of this issue.
      Clearing out my existing state via kubectl and etcdctl does not seem to help.

      This ticket is complete when we once again have a "latest" apiserver image that starts properly.

      Gliffy Diagrams

        Attachments

          Activity

            People

              willis8 Craig Willis
              lambert8 Sara Lambert
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Tasks