Requirements
- Rate Limiting (against flooding)
- Time stamp each request and make sure there is not more than a fixed amount over the last time interval (seconds/minutes/hours/etc)
- Requests Quota (against user overusing the system)
- Keep a counter of how many requests a user can make
- Decrease counter with every new request
- When at 0 notify user they are out of requests
- This could be a fixed number or could be reset every year/month/week
- Byte count
- We could do the same as with Requests Quota, but by counting bytes
- Or we could to byte size limit for individual files
- Limits are by user
- If we want them by key we need to limit keys as well
Tasks
- Move storing of events to MongoDB or Postgres
- It's taking up too much space in Redis
- Add key users:{user}:uploaded:bytes and users:{user}:uploaded:count
- Add check when uploading. If current users:{user}:uploaded:bytes > config value, then send error
- More accurate byte counts
- Change Clowder to return file size after upload
- Change Polyglot to return file size after upload
Other
- Return appropriate message when token has expired so user can request a new one and retry the request
Background Resources
{"serverDuration": 86, "requestCorrelationId": "19373ca204043247"}
3 Comments
Kenton McHenry
Luigi Marini, all options seem good. Would you support all of them or pick one or two?
Luigi Marini
Kenton McHenry I am thinking of doing them in this order 2 > 1 > 3. I don't think doing quotas by key buys us much. I think by user should be enough.
Kenton McHenry
Sounds good.