Page History

Below are documentation on Polyglot and Softwareserver, including how they work together. Intended for developers or interested users. Started with what Rui learned as part of work of BD-460 "Software Server not setting correct public IP on checkins".

• Installation

  1. On Mac, use brew to install maven, imagemagick, rabbitmq. Install mongo either manually or using brew.
  2. git clone the repo.
  3. In the top dir, do "mvn package -Dmaven.test.skip=true -Dmaven.javadoc.skip=true".
  4. Start mongod on localhost.
  5. Start RabbitMQ on localhost.
  6. In the top dir, in one shell, run "bin/PolyglotRestlet.sh" to start Polyglot. In another shell tab, run "bin/SoftwareServerRestlet.sh" to start SoftwareServer.

• User Interface

A user interacts with Polyglot. Polyglot works internally with software servers. By default Polyglot runs on port 8184, Softwareserver on port 8182. A user can use "http://<polyglot_ip>:8184/" to see the available endpoints / URLs, such as "http://<polyglot_ip>:8184/servers" to see the server IP list, or "http://<polyglot_ip>:8184/form" to use a form to submit a conversion request.

• Internal Working

• The Java Classes

  1. Software Server:
     1. SoftwareServer.java: handles wrapper scripts;
     2. SoftwareServerRestlet.java: handles the Restlet service;
     3. SoftwareServerRESTUtilities.java: the rabbitMQHandler() method handles all the interaction with RabbitMQ.
  2. Polyglot:
     1. Polyglot.java: abstract class;
     2. PolyglotStewardAMQ.java: handles IOGraph and interaction with RabbitMQ;
     3. PolyglotRestlet.java: handles the Restlet interface.
        process_jobs(): at the end, writes the ".url" file.

• SS Registration

A Polyglot process goes to RabbitMQ, gets the consumer IPs, connects to these IP's softwareserver at the URL "<softewareserver_ip>:8182/applications". If the URL is accessible and contains valid content, Polyglot adds the IP to its server list.

• SS checkin

A Softwareserver connects to RabbitMQ, picks up jobs (aka msgs) in the queues, processes them, and sends the results back by accessing Polyglot's endpoint at "<polyglot_ip>:8184/checkin/<jobid>/<result_url>".

• SS capabilities

SoftwareServer uses SoftwareServer.conf + scripts/*/.aliases.txt to configure which applications it will process. For example, SS on dap-dev is configured to convert only demclip and streamclip:

SoftwareServer.conf:
#BatchScripts=scripts/bat
#ShellScripts=/home/polyglot/scripts/sh
#RScripts=scripts/R
#AHKScripts=scripts/csr-configured
#AppleScripts=scripts/applescript
#SikuliScripts=scripts/sikuli
PythonScripts=scripts/py

$ grep -v ^# */.aliases.txt
ahk/.aliases.txt:A3DReviewer
ahk/.aliases.txt:ImgMgk
ahk/.aliases.txt:IrfanView
py/.aliases.txt:demclip_convert.py
py/.aliases.txt:streamclip_convert.py
R/.aliases.txt:PEcAn
sh/.aliases.txt:avconv-audio
sh/.aliases.txt:avconv-video

• Implementation Details

Polyglot and SS are implemented in Java, currently using Restlet. Current configuration files are:

...

• When Polyglot starts, it does:
  1. read the configuration file;
  2. start the PolyglotStewardAMQ thread;
  3. start a thread to update Mongo; call PolyglotRESTUtilities.updateMongo(). By default updates every 2 sec.
  4. start the restlet service.
• When PolyglotStewardAMQ thread starts, it starts 3 threads:
  1. discoveryAMQ(), every 30 s.
  2. process_jobs(), every 3 s,
  3. heartbeat(), to remove unresponsive SSs. every Heartbeat secs, default to 10 s.
• Two queryEndpoint() methods.
  In SoftwareServerRESTUtilities.java: returns pure text; in PolyglotStewardAMQ.java, returns json.

• Installation of Required Software Packages on Ubuntu Trusty (14.04)

Need to install the programs used in Softwareserver scripts:

...