Instructions for Creating VM in Nebula

  1. Create a Windows VM in Nebula.
    1. Choose the instance boot source to be "Boot from image (create new volume option)". 
    2. Choose appropriate VM flavor to get sufficient disk size and number of CPUs
    3. Choose appropriate security and networking
    4. Launch instance
    5. Associate a Floating IP address
    6. Login to the VM and enable remote desktop connection

Install the Extractor

1. Install python

instructions: https://docs.python.org/2/using/windows.html

make sure “C:\Python27\” is on path for system user (start->right click on “computer”->properties->advanced->environment variables), edit PATH if necessary

For now, use Python 2.7 instead of Python 3.x for compatibility reasons.


2. Install pip

download file “get-pip.py” from https://pip.pypa.io/en/latest/installing.html#id7

open cmd as admin

python get-pip.py

after install is done, type pip in the cmd prompt, if “not recognized” error, add “C:\Python27\Scripts" to path for system user (like described in 1)

 

3. Install pika, requests

pip install pika

pip install requests

 

4. Install pyclowder

clone from the repo: https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder/browse

 python setup.py install

If you need to make any changes to pyclowder during development, you will need to rerun setup.py install.


5. Install any additional software/packages required

 

6. Ensure Clowder and RabbitMQ are running [This is not required if Clowder is not running in the same VM]

In clowder\conf\play.plugins, 9992:services.RabbitmqPlugin should be enabled (line 3 may be commented out by default).

RabbitMQ needs a topic exchange called clowder - this can be created via RabbitMQ browser UI (default is http://localhost:15672/)

The extractor config.py should include the rabbitmqURL (default is "amqp://guest:guest@localhost:5672/%2f")

 

7. Test that the extractor works 

Run it from the cmd prompt and watch the outputs to be sure it is working properly. Also advised to upload some files to Clowder and observe that the extractor is receiving, processing and uploading the results back correctly.

Build Windows Service [Uses NSSM]

  1. Create a simple bat file to run extractor and save it in some directory that is easily accessible:

    cd /d <full_path_to_extractor_directory>
python Landsat7mosaic.py


  2. Download and install NSSM. At the time of writing this document, the latest stable version is 2.24.
    Make sure that NSSM is added to your PATH for easy access from the command prompt.


  3. From Windows Command Prompt, run the command to install a new service. You will need to provde administrative privileges to execute this command:

    nssm install <service_name>

    Here the <service_name> is the unique name of the service. This is different from the display name that shows up when you open the Windows Service Manager.

  4. Fill in the details of the extractor and the service settings in various tabs

    1. In the Application Path, put the path of the bat file that was created in step 1. Startup directory will be the directory containing the bat file by default. Provide any arguments if needed.
       

    2. Add the service details including the display name


    3. Set exit actions. Optionally, provide a delay to restart the service.


    4. Optionally, choose log files to write stdout and stderr


    5. If needed change any of the other settings and press "Install Service" button


    6. Start the service from Windows Service Manager


      Or you can use the "nssm start <service_name>" command. Other commands needed to manage the service are listed below:

      For example, 

      nssm.exe restart terex-floodplain


    7. Make sure that the corresponding extractor queue was created in RabbitMQ. Done!

Build Windows Service:

[This section exists only because of historical reasons. The method described here uses srvany and is not recommended because of issues with restarting and stopping the service cleanly. DO NOT USE THIS METHOD!]

1. Prepare a simple bat file to run extractor, like:

 cd /d <full_path_to_extractor_directory>

 python cellprofiler_extractor.py >>  log_<some_extractor_specific_name> 2>&1

 

2. Convert bat file to exe file

For that can bat_to_exe_converter can be used: http://www.computerhope.com/issues/ch000435.htm

 

3. Get svrany.exe 

It can be found on Windows server 2003 resource kit tools: http://www.microsoft.com/en-us/download/details.aspx?id=17657

 

4. Use install.bat to build a Windows Service:

install.bat can be found here: https://opensource.ncsa.illinois.edu/stash/projects/CBI/repos/cyberintegrator-2/browse/edu.uiuc.ncsa.cyberintegrator.server.feature/root/service

It assumes srvany.exe is on a subdirectory called service (create this subdirectory and copy srvany there)

Edit install.bat file changing NAME, DESC and EXE to reflect the name, description and executable file of your service.

Set HOME to the directory containing install.bat script

Run install.bat as administrator and install the service

 

5. Check that the correspondent queue was created on the rabbitmq server that the extractor connects to:

Delete queue first if already exists and no one else is using

If not working, try checking log errors in the event viewer (find it by typing eventviewer on the start search box) under windows logs, in system or in application and filtering for errors (filter current log on right panel, check box for “Error”) which have “event sources”=Service control manager. (http://stackoverflow.com/questions/1067531/are-there-any-log-file-about-windows-services-status)

 

6. Change windows service configuration to “delayed start”

Open “services” from win start menu (type services on the start search box and click enter)

Right click on the extractor->properties

Change startup type to "automatic (Delayed Start)"

 

7. Restart VM

Do not login and see if the queue was created in the rabbitmq; it will take a couple of minutes because of the “delayed start”

Delete queue first if it already exists to be sure the extractor service can create the queue if necessary