Uploaded image for project: 'National Data Service'
  1. National Data Service
  2. NDS-715

Kubernetes addons terminate themselves automatically in Kubernetes 1.5

XMLWordPrintableJSON

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • None
    • None
    • Infrastructure
    • None
    • NDS Sprint 18, NDS Sprint 19

      After deploying a Kubernetes 1.5 cluster at SDSC, we noticed that the addons were not all running as they should be. Performing a kubectl create manually on the .yml file showed the services would run properly for several minutes before terminating themselves for seemingly no reason.

      Step to recreate:

      1. Deploy a Kubernetes 1.5 cluster via deploy-tools
      2. Create grafana via kubectl

        kubectl create -f /tmp/grafana.yml

      3. Watch kube get pods for changes (~5 min)

        watch kubectl get pods

        • You should see the grafana / heapster / influx pods all go into a "Terminating" state.

      This may be related to the addons and their associated update script, but more investigation will certainly be needed.

      This ticket is complete when the addons (grafana and kibana, at minimum) continue to run for longer than 10 minutes on a multi-node Kubernetes 1.5 cluster without terminating themselves.

              raila David Raila
              lambert8 Sara Lambert
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved:

                  Estimated:
                  Original Estimate - 6 hours
                  6h
                  Remaining:
                  Time Spent - 3 hours Remaining Estimate - 3 hours
                  3h
                  Logged:
                  Time Spent - 3 hours Remaining Estimate - 3 hours
                  3h