Page History

This document is currently under development. 

This document details the instructions for installing the Brown Dog auxiliary service Extractor Info Fetcher in BD-Clowder-dev (dev) and BD-Clowder (prod) machines. 

1. SSH into to bd-clowder.ncsa.illinois.edu or bd-clowder-dev.ncsa.illinois.edu. Rest of the instructions are identical for both these machines unless specified otherwise.
2. Clone extractor-info-service code repository into /home/browndog

   1. git clone opensource.ncsa.illinois.edu/bitbucket/scm/bd/bd-aux-services.git

3. Install pyvenv if needed. One can also create Python virtual environment using virtualenv

   1. sudo apt-get install python3-venv

4. Install extractor-info-service

   1. cd /home/browndog/bd-aux-services/extractor-info-fetcher

   2. sudo pyvenv prototype-endpoint-env

   3. source ./prototype-endpoint-env/bin/activate

   4. sudo -H ./prototype-endpoint-env/bin/pip install -r requirements.txt

   5. Update config.py so that the variables, especially rabbitmq_vhost, clowder_url, clowder_key store details about development or production instances based on the machine (i.e. dev or prod) in which the service is being installed. I.e. Extractor Info Fetcher in bd-clowder machine should use information about Brown Dog production instances where as in bd-clowder-dev machine, it should use information about Brown Dog development instances.

5. Convert source code from Python2.7 to Python3 compatible by running (or something similar):

   1. sudo 2to3-3.5 -w extractor-info-fetcher.py

6. Update file permissions:
   

   1. sudo chown -R browndog /home/browndog/bd-aux-services/

   2. sudo chgrp -R users /home/browndog/bd-aux-services/

7. Make sure that the flask app can be run locally:

   1. sh ./run-script.sh

   2. From a new terminal window in the same machine, test the endpoint using the command below:

   3. curl localhost:5000/get-extractors-info?file_type=image/png

   4. If everything looks fine, kill the Flask app by pressing Ctrl + C.

8. Setup the Flask app as a systemd service

   1. sudo cp extractor-info-fetcher.service /lib/systemd/system

   2. sudo systemctl daemon-reload

   3. sudo systemctl start extractor-info-fetcher.service

9. Check the systemd service status and make sure it is running:

   1. sudo systemctl status extractor-info-fetcher.service

10.  Update system to allow input traffic to port 5000 in the machine within NCSA network

    1. If the machine is a Nebula OpenStack VM, update iptables by following the steps below:

       1. Create a file in /etc/iptables/rules.d, say 51local.rules with content:

          # Extractor Info Service (Flask App)
          -A INPUT -s 141.142.0.0/16 -p tcp -m state --state NEW -m tcp --dport 5000 -j ACCEPT
          

       2. sudo /etc/iptables/rules.sh

    2. If the machine is a Nebula OpenStack VM, you will need to create a security group in Nebula based on 51local.rules mentioned above and add that group to the VM.

11. Try to access http://bd-clowder.ncsa.illinois.edu:5000 or http://bd-clowder-dev.ncsa.illinois.edu:5000 from NCSA network and you should see a welcome message.

12. Now, if you visit http://bd-clowder.ncsa.illinois.edu:5000/get-extractors-info?file_type=image/png or http://bd-clowder-dev.ncsa.illinois.edu:5000/get-extractors-info?file_type=image/png from NCSA network, you should be able to see a JSON formatted array of details about currently running extractors that can process PNG format image files.