...
BD-703. To add Docker containers as a new granularity in the VM elasticity module, thus expands its functionality.
- Design questions and answers:
- Should an extractor type be managed by both an VM image and docker, or only one of them?
A: only one for simplicity. So need to specify this piece of information in the config file. - Docker image storage: docker hub, or a private registry?
A: go with the docker hub for now. Setting up a private registry takes time and a secure one requires getting a certificate from a CA. Can do it later when needed. Use ncsa/clowder-ocr, ncsa/clowder-python-base, ncsa/clowder-opencv-closeups, etc. for now. Done. - How to restart a docker container if the application crashed/stopped? A: docker run --restart=always .
...
- .. This will retry indefinitely, but with a delay that doubles before each retry, starting from 100 ms (0.1 second), to avoid flooding the server. Can also consider using "--restart=on-failure[:max-retries]" to limit the number of retries, but then that could leave a container in the stopped state, without any component to restart it. Usually a RabbitMQ server restart would cause an error, and the error was observed to persist for about 2 minutes.
- How to scale up an application managed by docker?
- How to scale down? A: see above.
- Do we suspend and resume docker VMs, or always keep them running? A: We suspend and resume docker VMs, but keep at least 1.
- We need to run some VMs exclusively to host the docker containers. How do we start them? Externally bootstrap, or start them using the elasticity module? A: need to add the docker VM image info in the config file, so the module knows how to start a new docker VM. Can start one at the beginning of the elasticity. Later on as needed start more.
- how to detect idle extractors managed by docker? A: use the RabbitMQ API as before.
- how to detect idle docker machines if no container runs on them? A: add data structure for docker machines. Find docker VMs that have no extractors running on them, add them to the idle machine list, or somehow signal that they can be suspended.
- how to specify mapping of docker images with extractors? A: add a [Docker] section in the config file. extr1 -> dockerimg1. When starting elasticity module, load the config file, and check for errors: one extractor type should be managed only by one method: either docker or a VM image. If such configuration errors exist, print out, and use docker (make this choice configurable too).
- Algorithm / Logic
Assumptions:
...