Versions Compared

Old Version 15

changes.mady.by.user Rui Liu

Saved on

compared with

New Version Current

changes.mady.by.user Rui Liu

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • Installation

    1. On Mac, use brew to install maven, imagemagick, rabbitmq. Install mongo either manually or using brew.
    2. git clone the repo.
    3. In the top dir, do "mvn compile" to compile the code, should not take long. Then do "mvn package -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -q".
    4. Start mongod on localhost.
    5. Start RabbitMQ on localhost.
    6. In the top dir, in one shell, run "bin/PolyglotRestlet.sh" to start Polyglot. In another shell tab, run "bin/SoftwareServerRestlet.sh" to start SoftwareServer. If the script fails, change its content from "2.1.0" to "2.2.0". Then you can use the URLs http://localhost:8184 to check Polyglot, and http://localhost:8182 to check SS.

  • User Interface

A user interacts with Polyglot. Polyglot works internally with software servers. By default Polyglot runs on port 8184, Softwareserver on port 8182. A user can use "http://<polyglot_ip>:8184/" to see the available endpoints / URLs, such as "http://<polyglot_ip>:8184/servers" to see the server IP list, or "http://<polyglot_ip>:8184/form" to use a form to submit a conversion request.

  • Internal Working

  • The Java Classes
    1. Software Server:
      1. SoftwareServer.java: handles wrapper scripts;
      2. SoftwareServerRestlet.java: handles the Restlet service;
      3. SoftwareServerRESTUtilities.java: the rabbitMQHandler() method handles all the interaction with RabbitMQ.
    2. Polyglot:
      1. The entry point is PolyglotRestlet.java.
        java -cp polyglot.jar:lib/* -Xmx1g edu.illinois.ncsa.isda.softwareserver.polyglot.PolyglotRestlet
      2. Polyglot.java: abstract class;
      3. PolyglotStewardAMQ.java: handles IOGraph and interaction with RabbitMQ;
      4. PolyglotRestlet.java: handles the Restlet interface.
        process_jobs(): at the end, writes the ".url" file.
  • SS Registration

...

Code Block
languagebash
titleSS conf and .aliases.txt content
collapsetrue
SoftwareServer.conf:
#BatchScripts=scripts/bat
#ShellScripts=/home/polyglot/scripts/sh
#RScripts=scripts/R
#AHKScripts=scripts/csr-configured
#AppleScripts=scripts/applescript
#SikuliScripts=scripts/sikuli
PythonScripts=scripts/py

$ grep -v ^# */.aliases.txt
ahk/.aliases.txt:A3DReviewer
ahk/.aliases.txt:ImgMgk
ahk/.aliases.txt:IrfanView
py/.aliases.txt:demclip_convert.py
py/.aliases.txt:streamclip_convert.py
R/.aliases.txt:PEcAn
sh/.aliases.txt:avconv-audio
sh/.aliases.txt:avconv-video
  • Polyglot REST APIs
  • POL REST endpoints that POL handles on its own without accessing/redirecting to SSes:

    GET:

    /           Returns a list of supported endpoints.

    /alive      Returns "yes".

    /checkin

    /convert    Returns all supported output formats

    /convert/output_format1   returns all supported input formats that can be converted to output_format1

    /convert/output_format1/file_url1   do the conversion: download file_url1

    /form

    /image

    /inputs

    /inputs/<format1>

    /outputs

    /requests

    /servers

    /software

  • POL REST endpoints that access or redirect to SSes:

    GET:

    /file/<file1>      # If file1 doesn't exist and file1.url exists.

    /servers/<server1_ip>[/...]   # Redirects to server1_ip:8182/software/...

    /software/<sw1>    # Accesses all SS:8182/software until finding one that contains sw1 and redirects to <ss_ip1>:8182/software/<sw1>.

 

    POST:

    /servers/<ip1>[/...]   # Redirects to ip1:8182/software/<...>.

    /software/<sw1>        # Accesses all SS:8182/software until finding one that contains sw1 and redirects to <ss_ip1>:8182/software/<sw1>.

  • Implementation Details

  • Configuration files

Polyglot and SS are implemented in Java, currently using Restlet. Current configuration files are:

...

Softwareserver job checkin is in SoftwareServerRESTUtilities.java, Polyglot accessing RabbitMQ part is in polyglot/PolyglotStewardAMQ.java.

  • Start-up / Initialization
  • When Polyglot starts, it does:
    1. read the configuration file;
    2. start the PolyglotStewardAMQ thread;
    3. start a thread to update Mongo; call PolyglotRESTUtilities.updateMongo(). By default updates every 2 sec.
    4. start the restlet service.
  • When PolyglotStewardAMQ thread starts, it starts 3 threads:
    1. discoveryAMQ(), every 30 s.
    2. process_jobs(), every 3 s,
    3. heartbeat(), to remove unresponsive SSs. every Heartbeat secs, default to 10 s.
  • Two queryEndpoint() methods.
    In SoftwareServerRESTUtilities.java: returns pure text; in PolyglotStewardAMQ.java, returns json.
  • kgm.utility.Utility source code

    https://isda.ncsa.illinois.edu/svn/isda/trunk/kgm/Utilities/src/kgm/utility/Utility.java

...