Page History

...

Brown Dog VM elasticity project needs to support multiple OSes, so OpenStack seems a viable high level solution. Currently considering using OpenStack.  May consider using Docker on the VMs at a low level if needed.

• Algorithm / Logic

Assumptions:

The following assumptions are made in the design:

1. the extractor is installed as a service on a VM, so when a VM starts, all the extractors that the VM contains as services will start automatically and successfully;
2. the resource limitation of using extractors to process input data is CPU processing, not memory, hard disk I/O, or network I/O, so the design is only for scaling for CPU usage;
3. the system needs to support multiple OS types, including both Linux and Windows;
4. the system uses RabbitMQ as the messaging technology.

Algorithm:

The VM elasticity module maintains and uses the following data:

...

Periodically (configurable, such as every minute), the system checks whether we need to scale up, and whether we need to scale down. These 2 checks can be done in parallel, but if so, the system needs to protect and synchronize the shared data, such as the list of running VMs.

Scaling up:

The criterion for the need of scaling up is: RabbitMQ queue lengths > pre-defined thresholds, such as 100 or 1000.

...

1. If the threshold is reached for a given queue, say, q1, then use the data item 2 above, find the corresponding extractor (e1). Currently this is hardcoded in the extractors, so that queue name == extractor name.
2. Look up e1 to find the corresponding running VM list, say, (vm1, vm2, vm3).
3. Go through the list one by one. If there's an open slot in the VM, meaning its #vCPUs > loadavg + <cpu_buffer_room> (configurable, such as 0.5), for example, vm1 #vCPUs == 2, loadavg = 1.2, then start another instance of e1 on vm1. Return. If there's no open slot on vm1, look at the next VM in the list. Return if an open slot is found and another instance of e1 is started.
4. If we go through the entire list and there's no open slot, or the list is empty, then look up e1 to find the corresponding suspended VM list, say, (vm4, vm5).  If the list is not empty, resume the first VM in the list. Return.
5. If the above suspended VM list is empty, then we need to start a new VM to have more e1 instances. Look up e1 to find a VM image that contains it. Start a new VM using that image.

Scaling down:

Get the list of IPs of the running VMs. Iterate through them:

...