To start elasticity module evaluation
--------------------------------------------------
>> Connect to Rabbitmq from outside NCSA to use Rabbitmq Management Interface:
ssh -L 5673:rabbitmq.ncsa.illinois.edu:5672 -L 15673:rabbitmq.ncsa.illinois.edu:15672 isda.ncsa.illinois.edu
Open the management interface on the browser typing: localhost:15673
>> Connect to Mongo from outside NCSA:
ssh -L 27017:mongo:27017 isda.ncsa.illinois.edu
>> on bd-elasticity.ncsa.illinois.edu:
- To query the status:
initctl list | grep bd
- To start or stop an individual bdelas service:
sudo start bdelas CLOWDER_INSTANCE=dts-dev
sudo stop bdelas CLOWDER_INSTANCE=dts-dev
- To start or stop all bdelas service instances:
sudo start bdelas-all-instances
sudo stop bdelas-all-instances
- To see the logs:
cd /var/log/upstart
sudo tail -f bdelas-dts-dev.log
- source BROWNDOG-openrc.sh (Obtain this from Nebula web interface under BROWNDOG project>Access & Security > API access> Download OpenStack RC file)
- To see the VMs used for an instance, e.g., for dts-dev:
nova list | grep dts_dev
- To be able to ssh into the VM, change to browndog user
sudo -u browndog -H -s /bin/bash
- Then ssh into a docker VM.
To see the docker containers:
docker ps -a
To stop one container:
docker rm -f opencv-faces-2
To start a container:
docker run --name=ocr-1 -d --restart=always -e 'RABBITMQ_URI="amqp://user1:password1@rabbitmq.ncsa.illinois.edu:5672/clowder-dev?heartbeat_interval=30"' -e 'RABBITMQ_EXCHANGE="dts"' -e 'TZ=/usr/share/zoneinfo/US/Central' ncsa/clowder-ocr:jsonld
To list container images:
docker images
To remove all containers:
docker rm $(docker ps -a -q)
To remove all images:
docker rmi $(docker images -q)
>> To start the tests:
Run the test script
python dts_benchmark_file_atRate.py <num of testfiles>
>> Obtain the data from the database for analysis:
query_elas.js
//print("# queue name,time,num_msgs,num_consumers,total_num_msgs_published,total_num_msgs_delivered, num_vms");
db = db.getSiblingDB('browndog_dev');
db.extr.find({"$and": [{"time": {"$gte": ISODate("2016-03-31T11:11:00.00Z")}}, {"time": {"$lte": ISODate("2016-03-31T11:40:00.000Z")}},{"name":"ncsa.image.ocr"}]}).sort({time:1}).forEach(function(jo){print(jo.time.getTime() + " "+ jo.num_msgs.toString() + " " + jo.num_consumers.toString() + " " + jo.hosts.length.toString());});
query_noelas.js
Note the collection is different when EM module is switched off and we only ran rabbitmq-monitor to collect statistics.
db = db.getSiblingDB('browndog_dev');
db.queues.find({"$and": [{"time": {"$gte": ISODate("2016-04-08T20:35:00.00Z")}}, {"time": {"$lte": ISODate("2016-04-08T21:45:00.000Z")}},{"name":"ncsa.cv.profiles"}]}).sort({time:1}).forEach(function(jo){print(jo.time.getTime() + " "+ jo.num_msgs.toString() + " " + jo.num_consumers.toString() + " " + jo.hosts.length.toString());});
>> Shell script to obtain number of consumers and hosts as queue length vary over time from the query results of database.
>> Matlab plot script : MATLAB script to plot the data