2022/05/23 Update plan

Groups of cluster-prod nodes:

(1) descut.cosmology.illinois.edu        141.142.161.46
(1) descut4.cosmology.illinois.edu       141.142.161.56
(2) descut5.cosmology.illinois.edu       141.142.161.57
(2) descut6.cosmology.illinois.edu       141.142.161.58
( ) desgpu1.cosmology.illinois.edu       141.142.161.18
(3) desrelease.cosmology.illinois.edu    141.142.161.51
(3) *descut3.ncsa.illinois.edu           141.142.161.48
(3) *descut2.cosmology.illinois.edu      141.142.161.54

*desgpu1 was already updated and so is skipped

Update procedure:

1. Drain group (1). Update and reboot.
2. Cordon groups (2), (3), and (4).
3. Drain group (2). Update and reboot. Pods will primarily respawn on updated nodes in group (1).
4. Uncordon group (2).
5. Drain group (3). Update and reboot. Pods will primarily respawn on updated nodes in groups (1) and (2).
6. Uncordon group (3).

cluster-dev nodes:

Update and reboot without coordination.