Date: Fri, 29 Mar 2024 03:45:01 -0500 (CDT) Message-ID: <1199068476.318.1711701901225@os-confluence.ncsa.illinois.edu> Subject: Exported From Confluence MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_Part_317_1817978176.1711701901225" ------=_Part_317_1817978176.1711701901225 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Location: file:///C:/exported.html
Sometimes hosts need to be shut down for updates, what we would = like is a webpage that will be shown instead. This page could give some exp= lanations of why the site is down.
Plan is to have a server that can handle and impersonate the web traffic= for different sites. This single server will be the backup server and only= serve a single page that explains the outage and who to contact. There are= 2 pieces needed for this to work. The site configuration files needed for = the server to respond to requests for a specific server (see below in Site = Configs). The second piece needed is a method to switch DNS entries from on= e server to another.
For this to work we need to change the DNS entry to point to the mainten= ance server. There are different methods to do this. Two methods we have ex= perience with is to ask the neteng team to change the DNS manually or have = loadbalancing service that checks to see if the server is up.
Since most servers are set to have a timeout of 24 hours, which requires= the neteng team to be notified at least 24 hours before hand. The steps ar= e in general:
We can leverage off a load balancing service that will automatically che= ck to see if a server is up, and will fail back to the maintenance page if = the server is not reachable.
One of such examples is Global Server Load Balanc= ing hosted by Illinois. GSLB is a load balancing system that will resp= ond to DNS queries and returns an IP address associated with the name. GSLB= will continuously check the main IP address to see if this up and respondi= ng. Once it detects no response it will switch to the backup IP address. On= ce the main IP address is back up, GSLB will switch back to the regular IP = address.
A discussion has started with the neteng team to see how we can support = a system like this at NCSA. Examples being considered is a nginx high avail= ability, keepalived, etc.
All files will be hosted in /home/maintenance/<server>, the follow= ing is what is recommended:
The nginx configuration file in /home/maintenance/<server>/ng= inx.conf and a symbolic link (ln -s /home/maintenance/<server>/nginx.= conf /etc/nginx/sites-enabled/<server>)
server = { listen 80; server_name browndog.ncsa.illinois.edu; client_max_body_size 0; server_tokens off; root /home/maintenance/browndog/html; # create 503 Service Unavailable location / { return 503; } error_page 503 @maintenance; location @maintenance { rewrite ^(.*)$ /maintenance.html break; } } server { listen 443; server_name browndog.ncsa.illinois.edu client_max_body_size 0; server_tokens off; proxy_read_timeout 300; # answer from server, 5 min proxy_send_timeout 300; # chunks to server, 5 min ssl on; ssl_session_timeout 5m; ssl_certificate /home/maintenance/browndog/ssl/nginx.crt; ssl_certificate_key /home/maintenance/browndog/ssl/nginx.key; #ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_protocols TLSv1.2; ssl_prefer_server_ciphers on; #ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EEC= DH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH ED= H+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4"; ssl_ciphers "ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-= RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECD= HE-RSA-AES256-SHA:DHE-RSA-AES256-SHA"; root /home/maintenance/browndog/html; # create 503 Service Unavailable location / { return 503; } error_page 503 @maintenance; location @maintenance { rewrite ^(.*)$ /maintenance.html break; } }
A single file called maintenance.html in /home/maintenance/<serv= er>/html/maintenance.html
<html= > <head> <title>Maintenance</title> </head> <body> <h1>Down for maintenance</h1> <p>The service you are looking for is currently down for maintena= nce. Please try again soon.</p> <p>A list of known outages can be found at <a href=3D"http://s= tatus.ncsa.illinois.edu/">NCSA Status Page</a></p> <h1>Please pardon the dust!</h1> <p>If you reach this page and want to let us know about your expe= rience during the outage or have any questions about why you received this = page, please contact us at <a href=3D"mailto:opensource+maintenance@ncsa= .illinois.edu">browndog maintenance</a>.</p> </body> </html>
And the SSL certificate needed to return https calls, place these in&nbs= p;/home/maintenance/<server>/ssl/nginx.key and /home/maintenance= /<server>/ssl/nginx.crt
Once the site is configured you can test it using:
curl --= insecure --verbose --header 'Host: browndog.ncsa.illinois.edu' https://141.= 142.210.188/clowder/ * Trying 141.142.210.188... * Connected to 141.142.210.188 (141.142.210.188) port 443 (#0) * found 148 certificates in /etc/ssl/certs/ca-certificates.crt * found 594 certificates in /etc/ssl/certs * ALPN, offering http/1.1 * SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384 * =09 server certificate verification SKIPPED * =09 server certificate status verification SKIPPED * =09 common name: browndog.ncsa.illinois.edu (does not match '141.142.210.= 188') * =09 server certificate expiration date OK * =09 server certificate activation date OK * =09 certificate public key: RSA * =09 certificate version: #3 * =09 subject: C=3DUS,postalCode=3D61801,ST=3DIL,L=3DUrbana,street=3D1205 W= . Clark St,O=3DUniversity of Illinois,OU=3DNCSA,CN=3Dbrowndog.ncsa.illinois= .edu * =09 start date: Fri, 16 Feb 2018 00:00:00 GMT * =09 expire date: Mon, 15 Feb 2021 23:59:59 GMT * =09 issuer: C=3DUS,ST=3DMI,L=3DAnn Arbor,O=3DInternet2,OU=3DInCommon,CN= =3DInCommon RSA Server CA * =09 compression: NULL * ALPN, server accepted to use http/1.1 > GET /clowder/ HTTP/1.1 > Host: browndog.ncsa.illinois.edu > User-Agent: curl/7.47.0 > Accept: */* > < HTTP/1.1 503 Service Temporarily Unavailable < Server: nginx < Date: Tue, 26 Feb 2019 16:27:08 GMT < Content-Type: text/html < Content-Length: 630 < Connection: keep-alive < ETag: "5c75661f-276" < <html> <head> <title>Maintenance</title> </head> <body> <h1>Down for maintenance</h1> <p>The service you are looking for is currently down for maintena= nce. Please try again soon.</p> <p>A list of known outages can be found at <a href=3D"http://s= tatus.ncsa.illinois.edu/">NCSA Status Page</a></p> <h1>Please pardon the dust!</h1> <p>If you reach this page and want to let us know about your expe= rience during the outage or have any questions about why you received this = page, please contact us at <a href=3D"mailto:opensource+maintenance@ncsa= .illinois.edu">browndog maintenance</a>.</p> </body> </html> * Connection #0 to host 141.142.210.188 left intact
This example shows that it will pretend the browndog webserver in this c= ase, and returns the actual webpage. The curl command requires the --= insecure flag since the IP address does not match the actual hostname. The&= nbsp;--header 'Host: browndog.ncsa.illinois.edu' will allow the curl comman= d to pretend to connect to a specific host and will use the right configura= tion section for nginx (the servername in the nginx.conf).