This document is currently under development.
This document details the instructions for installing the Brown Dog auxiliary service Extractor Info Fetcher in BD-Clowder-dev (dev) and BD-Clowder (prod) machines.
- SSH into to bd-clowder.ncsa.illinois.edu or bd-clowder-dev.ncsa.illinois.edu. Rest of the instructions are identical for both these machines unless specified otherwise.
- Clone extractor-info-service code repository into /home/browndog
git clone opensource.ncsa.illinois.edu/bitbucket/scm/bd/bd-aux-services.git
Install pyvenv if needed. One can also create Python virtual environment using virtualenv
sudo apt-get install python3-venv
Install extractor-info-service
cd /home/browndog/bd-aux-services/extractor-info-fetcher
sudo pyvenv prototype-endpoint-env
source ./prototype-endpoint-env/bin/activate
sudo -H ./prototype-endpoint-env/bin/pip install -r requirements.txt
Update config.py so that the variables, especially rabbitmq_vhost, clowder_url, clowder_key store details about development instances or production instances based on the machine (i.e. dev or prod) in which the service is being installed. I.e. Extractor Info Fetcher in bd-clowder machine should use information about Brown Dog production instances where as in bd-clowder-dev machine, it should use information about Brown Dog development instances.
Convert source code from Python2.7 to Python3 compatible by running (or something similar):
sudo 2to3-3.5 -w extractor-info-fetcher.py
- Update file permissions:
sudo chown -R browndog /home/browndog/bd-aux-services/
sudo chgrp -R users /home/browndog/bd-aux-services/
Make sure that the flask app can be run locally:
sh ./run-scripscript.sh
From a new terminal window in the same machine, test the endpoint using the command below:
curl localhost:5000/get-extractors-info?file_type=image/png
If everything looks fine, kill the Flask app by pressing Ctrl + C.
Setup the Flask app as a systemd service
sudo cp extractor-info-fetcher.service /lib/systemd/system
sudo systemctl daemon-reload
sudo systemctl start extractor-info-fetcher.service
Check the systemd service status and make sure it is running:
sudo systemctl status extractor-info-fetcher.service
Update iptables Update system to allow input traffic to port 5000 on this machine from within NCSA networkin the machine within NCSA network
If the machine is a Nebula OpenStack VM, update iptables by following the steps below:
Create a file in /etc/iptables/rules.d, say 51local.rules with content:
# Extractor Info Service (Flask App)
-A INPUT -s 141.142.0.0/16 -p tcp -m state --state NEW -m tcp --dport 5000 -j ACCEPTsudo /etc/iptables/rules.sh
If the machine is a Nebula OpenStack VM, you will need to create a security group in Nebula based on 51local.rules mentioned above and add that group to the VM.
Try to access http://bd-clowder.ncsa.illinois.edu:5000 or http://bd-clowder-dev.ncsa.illinois.edu:5000 from NCSA network and you should see a welcome message.
- Now, if you visit http://bd-clowder.ncsa.illinois.edu:5000/get-extractors-info?file_type=image/png or http://bd-clowder-dev.ncsa.illinois.edu:5000/get-extractors-info?file_type=image/png from NCSA network, you should be able to see a JSON formatted array of details about currently running extractors that can process PNG format image files.