...
Date/Time | What happened | How was it resolved |
---|---|---|
1/19/2018 | Disk space warnings gfs3 | The registry cache was using ~34GB disk. kubectl exec -it regsitry sh wget localhost:5001/v2/_catalog -O - (lists images in cache) cd /var/lib/registry/docker/registry/v2 find something that can be removed (e.g., repositories/craigwillis/apiserver) rm -r repositories/craigwillis/apiserver /bin/registry garbage-collect /etc/docker/registry/config.yml Deletes cached blobs |
1/8/2018 | transport connection errors | Started receiving alerts about exceeded pod restart thresholds for two mongo containers. Noticed I/O errors in mongo logs. Exec'd into Gluster server and noted that two bricks (node1, node2) were offline. Restarted both pods, one at a time. |
1/14/2018 | gfs4 load warnings | Ongoing load warning on gfs4. Noticed gfs2 brick not connected. Restarted gfs2 gluster server. Rebooted gfs4 node. Ran gluster volume heal global info gluster volume heal global to heal files |
2/12/2018 | LMA disk space warnings | LMA node on public beta does not appears to have a /var/lib/docker mount.. this would be fine, except that the node also has "ndslabs-role-compute: true", so client pods have been scheduled there. This included one instance each of NBI and MDF Forge, each of which have huge images (~4GB) with NBI also having a larger-than-average docker overlay folder. Short term: I have temporarily removed the compute label from LMA and deleted the MDF Forge pod and image - the NBI instance is Akshay's, so I will leave it running to avoid interruptions Long term: Once the user services are gone from this node (e.g. timeout), we can stop the docker daemon on LMA and remount /var/lib/docker as a bind-mount from /media/storage, as is standard on the other nodes |
2016-2017
Date/Time | What happened | How was it resolved | |||||
---|---|---|---|---|---|---|---|
10/31/2016 ~8am | NAGIOS errors on gfs2, node1, node3 | Attempted to reboot nodes, but encountered error: > "Error: Failed to perform requested operation on instance "workbench- Emailed Nebula group – apparently a problem with their glusterfs. Resolved at 10:30 AM | |||||
11/3/2016 | NAGIOS errors "could not complete SSH Handshake" node6 | Looked at node6 console via Nebula. Appears to be OOM problem (maybe old swap issue?). kubectl get nodes says all nodes except node6 are ready. Node is totally inaccessible. Tried soft reboot via Horizon, but node was then in error state. Spoke with Nebula group, this was related to the error from Monday. They resolved the underlying problem, but I still wasn't able to start the instance. Using cli: nova show <instance> nova reset-state --active <instance> nova start instance Did the trick | |||||
11/4/2016 | NAGIOS error for labstest-lma | Same as above. Nebula team resolved the glusterfs issue. Did not have permission to issue the reset state command. | |||||
11/8/2016 | API server not accessible – all Kubernetes services down on workbench-master1 | It again appears that etcd2 went down, probably due to memory problems. Rebooted the node. | |||||
12/26/2016 | GFS1 not accessible. Rebooting via Nebula put node in error state | Resolved on 1/3 by Nebula team – apparent problem with Gluster server. Node was able to restart. | |||||
1/4/2017 | GFS4 not accessible. | Resolved 1/4 by Nebula team – continued problem with Gluster server. | |||||
1/12/2017 | Loadbalancer sluggish | workbench-loadbal has been sluggish, slow response times resulting in numerous false positive nagios alerts. At some point this afternoon, it was unresponsive. Hard reboot via Horizon took >30 minutes for CoreOS1122 (which takes ~30 seconds on a normal day). Login was slow after reboot, services never fully revived. David suggests that this is a storage problem, but Nebula team can find no apparent cause. Starting standalone CoreOS instances works without error. Tried two different approaches: 1. shutdown -h of the instance and restart to see if hypervisor moves somewhere more friendly. 2. create a snapshot of another node (lma) and use this to create a new instance from it. After boot, edit /etc/kubernetes/kubelet change KUBELET_HOSTNAME from lma to loabal, systemctl restart kubelet. After this, kubectl get nodes showed loadbal in ready state with correct label. Disassociated the IP, associated with new instance. Shutdown bad instance.
| |||||
1/20/2017 | Node1/Node3 unavailable | Nodes 1 and 3 where not accessible via SSH from the Nagios instances. Node3 was totally unaccessible – Horizon console indicated OOM. Hard reboot succeeded, but CoreOS upgraded to 1235, introducing the flannel error. Copied the flannel config to /run/flannel and restarted. Node3 was accessible, but docker was down. Restarting docker failed until /var/lib/docker was deleted. Also upgraded to 1235, requiring the flannel change. | |||||
OPS node read-only | OPS node is currently in read-only state (same old Nebula problem). Should be resolved by reboot when needed. | ||||||
Master Kubelet down | Master Kubelet died due to etcd memory error (known issue). Rebooted, CoreOS upgrade required flannel fix. | ||||||
1/25/2017 | GFS nodes | Multiple incidents of GFS server pods not responding during healthz. In all cases, one or more glfs-server pods will not respond to exec. SSH to GFS node is find, but docker is unresponsive (docker ps hangs). journalctl shows errors related to registry cache Jan 28 11:10:23 workbench-gfs4.os.ncsa.edu dockerd[26174]: time="2017-01-28T11:10:23.365820903-06:00" level=warning msg="Error getting v2 registry: Get http://localhost:5001/v2/: read tcp 127.0.0.1:36906->127.0.0.1:5001: read: connection reset by peer" Generally, restarting docker daemon temporarily resolves problem. | |||||
1/29 | node1 | -bash: /usr/bin/wc: Input/output error Gluster problems on Nebula | |||||
2/9 | Multiple instances | Multiple instance I/O errors across projects, apparently due to Gluster outage on Nebula. Problem first detected at 3AM, reported at 6AM. No updates as of 10:30AM. | |||||
7/4 | gfs4 out of disk | /media/storage ran out of disk, due to the docker cache pod filling up the disk... pod had already been recreated so I looked up the uuid of the broken pod, deleted it, and SSH'd into gfs4 to delete its folder from core@workbench-master1 ~ $ kubectl get pods -o wide core@workbench-master1 ~ $ kubectl get pod -o yaml docker-cache-fwkd6 | grep uid core@workbench-gfs4 ~ $ sudo su ---- nagios pod was missing on gfs4 after this, core@workbench-master1 ~ $ kubectl get ds nagios-nrpe --namespace=kube-system core@workbench-master1 ~ $ kubectl apply -f nagios-nrpe-ds.yaml core@workbench-master1 ~ $ kubectl get ds nagios-nrpe --namespace=kube-system | |||||
7/22 | Single Pod restart threshold surpassed | NAGIOS started complaining shortly after 7pm: "workbench-master1/Kubernetes Pods is WARNING: 1 pods exceeding WARNING restart threshold." A Fedora Commons pod had restarted a sixth time (due to OOMKilled), which started triggering these warnings. Solution was to delete the pod in question to reset the restart count.
| |||||
8/8 | loadbal out of disk | NAGIOS alerted that node was nearly out of disk space. Craig restarted the ilb pod to clear out the huge 9.5GB log file. | |||||
8/9 | loadbal | NAGIOS alerted that node was nearly out of disk space (again). Mike restarted the loadbalancer node with a sudo reboot | |||||
9/24 ~1pm | loadbal | NAGIOS alerted that node was nearly out of disk space (again). Mike restarted the ilb pod to clear out the log file. This did not appear to alleviate the symptom, so he also restarted the node with a sudo reboot. NOTE: This reboot reset the MTU settings on the node. Please remember to verify MTU settings after reboot | |||||
10/13 ~2:30pm | master | NAGIOS was complaining: "workbench-master1/Kubernetes Pods is UNKNOWN: CRITICAL: Get http://localhost:8080/api/v1/pods: dial tcp 127.0.0.1:8080: getsockopt: connection refused" Fixed by running the following:
| |||||
10/20 ~4:30pm | lma / node1 pods hung in Terminating / ContainerCreating state | Node1 hit a weird zombie problem where Kenton's mongo instance would not terminate. Rebooting node1 allowed the shutdowns to complete properly, but this caused some odd behavior in the MTU settings... LMA node was running ElasticSearch / kibana, which filled up /media/storage with 19GB of log data, with 2 replicas took up 38GB of the 40GB storage drive. Unable to kill ElasticSearch due to hung/zombie pods. Looking into it a bit further, zombies seemed to be caused by MTU mismatch between docker0 (1500 == incorrect) and flannel.1 (1408 == correct) Further inspection revealed that /etc/systemd/system/docker.service.d/10-docker0.conf had specified 1454 as the MTU (also incorrect). Changing this to the correct value of 1408 and running docker network inspect bridge / ifconfig now shows the correct docker0 MTU. Resetting the MTU allowed the hung pods to finish shutting down/starting up, and I was then able to shut down the running elasticsearch / kibana to automatically clear out the storage drive. | |||||
10/24 10:30am | node1 acting sluggish almost out of space on /var/lib/docker (88%) | Ran the following on node1 to remove any images that no longer have valid tags (i.e. images that have been updated that have stale references cached that will never be started.. mysql releasing a new tag for 5.7, rebuilding cloud9, etc): docker images | grep none && docker rmi $(docker images | grep none | awk '{print $3}') | |||||
10/31 8:30am | loadbal out of disk | I was able to clear out ~500MB of space by deleting Dead/Exited containers, but this problem is still ongoing. Craig is experimenting with enabling docker logs rotation on this node, to prevent us from needing to check on it once every two weeks. | |||||
11/16/2017 | net/http request failures pulling images | Ongoing issue with ETK instance on Nebula, large image pulls are exceeding Kubernetes timeouts. We've decided to migrate the ETK instance to the new Jetstream allocation until the Nebula filesystem problems are resolved. | |||||
...