...

This seems like the most obvious approach, given that NGINX is already along the central entrypoint of traffic into the Clowder ecosystem.

...

I looked into applying the ratelimit one level above NGINX, going to Amazon's Route53 which I believe currently handles our can be used to handle DNS resolution (among other things).

Sadly, it looks like Route53 applies its own ratelimiting, which is not necessarily configurable or even exposed to the user.

Error-Handling Options

From Google:

The HTTP 429 Too Many Requests response status code indicates the user has sent too many requests in a given amount of time ("rate limiting").

A Retry-After header might be included to this response indicating how long to wait before making a new request.

There are some outliers - for example, Google Drive uses 403 Forbidden instead of 429. This could be confusing to users, as we already use 403 to indicate that the user lacks permission to perform the operation.

My recommendation would be to stick with this 429 as an error code if we find that for ratelimiting because it is commonly used to indicate that a user is ratelimited, and to differentiate it from Forbidden (aka "Permission Denied").  

Other Concerns

• How can we tell the difference between a user being ratelimited vs being blocked by the firewall (HTTP 429 should only be a symptom for ratelimit?)

...