Page History

For BD-1007: Polyglot refactoring for dockerizing SSes, mainly in the POL-SS communication part.

Current Polyglot-SS communication architecture:

In the existing architecure, both Polyglot and SS are servers. Polyglot-SS communication mainly occurs at SS registration, SS heartbeat, SS checking in converted files, and Polyglot redirecting file-getting requests to a SS.

1. SS Registration and heartbeat.
   Refer to the development notes at the "Polyglot / SoftwareServer Documentation" page, POL goes to RabbitMQ, gets the consumer IPs as the SSes, connects to the URL "<ip>:8182/applications". If the URL is accessible and contains valid content, Polyglot adds the IP to its server list.
   
   SS supports many REST endpoints. PolyglotStewardAMQ uses

...

1. "/alive" and "/applications" for SS registration and "/alive" for heartbeat:
   
   • • /alive
       This returns a timestamp (a long integer) to indicate the start time of the SS. (long int, plain text)
       Used for adding a SS in discoveryAMQ() if POL gets a valid long int, and removing a SS in heartbeat() if POL does not get a valid long int from the ep.
     • /applications
       This returns a JSON array, such as "[{"alias":"daffodil","conversions":[{"inputs":["csv"],"outputs":["xml"]},{"inputs":["pgm"],"outputs":["xml"]}]},{"alias":"flac","conversions":[{"inputs":["aif","aiff","fla","flac","wav"],"outputs":["aif","aiff","fla","flac","wav"]}]".
       Used for adding a SS and updating the IO graph in discoveryAMQ().

...

The following Polyglot REST endpoints in PolyglotRestlet.java accesses/redirects to SS.

• /file/<file1>
• servers/<server_ip1>[/...]
• software/<sw1>[/...]

1. SS checkin.

  Proposed design:

1. SS registration with polyglot.
   
   • Add 1 exchange/queue for SS registration with polyglot.
     Set its TTL to say 5 secs, so msgs are discarded in such time.
   • SS publishes msgs to this queue, regarding aliveness, and conversion capabilities.

2. In "checkin", SS uploads the converted file – instead of sending only the URL – and Polyglot saves it, instead of creating a "<file>.url" file.

3. SS heartbeat with polyglot.

What need to be added or changed:

1. POL now adds SSes in discoveryAMQ(), removes (non-responsive) SSes in heartbeat(). Change to: POL listens to a new queue, adds SSes and updates timestamps in discovery thread, and removes SSes with expired timestamps in heartbeat thread – with NO querying of SSes.
2. In SS: add code to publish conversion capability msgs in a new queue in RabbitMQ for POLs to listen to for SS registration and heartbeat.
3. In SS: add code to upload converted files, in polyglot: add code to save them;
4. In PolyglotStewardAMQ: change the code that creates "<file1.url>".

With this design, a SS can keep its REST endpoints, but POL does not query them. So it does not matter whether multiple SS containers run on the same VM or different VMs.

Notes:

1. use Java ConcurrentSkipListMap for the server_map in PolyglotStewardAMQ discoveryAMQ() for multi-threaded access.
2. The following Polyglot REST endpoints in PolyglotRestlet.java accesses/redirects to SS.

...

• /file/<file1>
• servers/<server_ip1>[/...]
• software/<sw1>[/...]