For BD-1007: Polyglot refactoring for dockerizing SSes, mainly in the POL-SS communication part.
In the existing architecure, Polyglot is a server, while a SS is also a server. Refer to the development notes at the "Polyglot / SoftwareServer Documentation" page, POL goes to RabbitMQ, gets the consumer IPs as the SSes, connects to the URL "<ip>:8182/applications". If the URL is accessible and contains valid content, Polyglot adds the IP to its server list.
SS supports many REST endpoints. PolyglotStewardAMQ uses the following SS endpoints:
- /alive
This returns a timestamp (a long integer) to indicate the start time of the SS. (long int, plain text)
Used for adding a SS in discoveryAMQ() if POL gets a valid long int, and removing a SS in heartbeat() if POL does not get a valid long int from the ep. - /applications
This returns a JSON array, such as "[{"alias":"daffodil","conversions":[{"inputs":["csv"],"outputs":["xml"]},{"inputs":["pgm"],"outputs":["xml"]}]},{"alias":"flac","conversions":[{"inputs":["aif","aiff","fla","flac","wav"],"outputs":["aif","aiff","fla","flac","wav"]}]".
Used for adding a SS and updating the IO graph in discoveryAMQ().
The following Polyglot REST endpoints in PolyglotRestlet.java accesses/redirects to SS:
- /file/<file1>
- servers/<server_ip1>[/...]
- software/<sw1>[/...]
Proposed design:
- SS registration with polyglot.
- Add 1 exchange/queue for SS registration with polyglot.
Set its TTL to say 5 secs, so msgs are discarded in such time. - SS publishes msgs to this queue, regarding aliveness, and conversion capabilities.
- Add 1 exchange/queue for SS registration with polyglot.
In "checkin", SS uploads the converted file – instead of sending only the URL – and Polyglot saves it, instead of creating a "<file>.url" file.
What need to be added or changed:
- POL now adds SSes in discoveryAMQ(), removes (non-responsive) SSes in heartbeat(). Change to: POL listens to a new queue, adds SSes and updates timestamps in discovery thread, and removes SSes with expired timestamps in heartbeat thread – with NO querying of SSes.
- In SS: add code to publish conversion capability msgs in a new queue in RabbitMQ for POLs to listen to for SS registration and heartbeat.
- In SS: add code to upload converted files, in polyglot: add code to save them;
- In PolyglotStewardAMQ: change the code that creates "<file1.url>".
With this design, a SS can keep its REST endpoints, but POL does not query them. So it does not matter whether multiple SS containers run on the same VM or different VMs.
Notes: use Java ConcurrentSkipListMap for the server_map in PolyglotStewardAMQ discoveryAMQ() for multi-threaded access.