Starting to track incidents and how they were resolved.
| Table of Contents |
|---|
2019
| Date/Time | What happened | How was it resolved |
|---|---|---|
| 3/1/2019 | Disk space errors | Kube registry and GLFS client pods were using ~1.5GB each. MISTAKE: Deleted pods from master to clear old log files. Found that Docker doesn't actually release space from deleted resources until the daemon is restarted. Required a restart of the Docker daemon on gfs2 to resolve after deleting pods See https://github.com/moby/moby/issues/21925 In the future, truncating huge log files with the following method is preferred:
|
...