Page History

...

• Rui's notes while working on Polyglot, including how to test, the bi-hourly tests.py, etc.

• Polyglot REST APIs

• POL REST endpoints that POL handles on its own without accessing/redirecting to SSes:

    GET:

    /           Returns a list of supported endpoints.

    /alive      Returns "yes".

    /checkin

    /convert    Returns all supported output formats

    /convert/output_format1   returns all supported input formats that can be converted to output_format1

    /convert/output_format1/file_url1   do the conversion: download file_url1

    /form

    /image

    /inputs

    /inputs/<format1>

    /outputs

    /requests

    /servers

    /software

• POL REST endpoints that access or redirect to SSes:

    GET:

    /file/<file1>      # If file1 doesn't exist and file1.url exists.

    /servers/<server1_ip>[/...]   # Redirects to server1_ip:8182/software/...

    /software/<sw1>    # Accesses all SS:8182/software until finding one that contains sw1 and redirects to <ss_ip1>:8182/software/<sw1>.

    POST:

    /servers/<ip1>[/...]   # Redirects to ip1:8182/software/<...>.

    /software/<sw1>        # Accesses all SS:8182/software until finding one that contains sw1 and redirects to <ss_ip1>:8182/software/<sw1>.

• Implementation Details

• Configuration files

...

• When Polyglot starts, it does:
  1. read the configuration file;
  2. start the PolyglotStewardAMQ thread;
  3. start a thread to update Mongo; call PolyglotRESTUtilities.updateMongo(). By default updates every 2 sec.
  4. start the restlet service.
• When PolyglotStewardAMQ thread starts, it starts 3 threads:
  1. discoveryAMQ(), every 30 s.
  2. process_jobs(), every 3 s,
  3. heartbeat(), to remove unresponsive SSs. every Heartbeat secs, default to 10 s.
• Two queryEndpoint() methods.
  In SoftwareServerRESTUtilities.java: returns pure text; in PolyglotStewardAMQ.java, returns json.

• kgm.utility.Utility source code

  :

  https://isda.ncsa.illinois.edu/svn/isda/trunk/kgm/Utilities/src/kgm/utility/Utility.java

• Testing

• Testing conversion on dap-dev

  .

• Commands:

  • Send the conversion request:
    curl http://browndog.user:password1@dap-dev.ncsa.illinois.edu:8184/convert/jpg/http%3A%2F%2Fbrowndog.ncsa.illinois.edu%2Fexamples%2Fbrowndog.png

  • Get the converted file:
    curl -u browndog.user:password1 -O http://dap-dev.ncsa.illinois.edu:8184/file/200598567_browndog.jpg

• Used a png to test.
  http://browndog.ncsa.illinois.edu/examples/browndog.png
  The converion URL requires an escaped URL.
     To generate an escaped URL using perl:
         perl -e 'use URI::Escape; print uri_escape("http://browndog.ncsa.illinois.edu/examples/browndog.png")'
  
  http://www.perlhowto.com/encode_and_decode_url_strings
     man URI::Escape

• Bi-hourly tests.py

  .

  On dap-dev in /var/www/html/dap/tests/. It used the Polyglot GET API
      using the URL to convert, got a returned URL, then downloaded that URL to
      "tmp/<count>_<file_basename>.<output_format>", then checked
      whether that file existed and was not empty.
  
      Python requests:
          r = requests.get(api_call, auth=(username, password),
          headers=headers, timeout=timeout)
  
          result = r.text
      contains only the text, not the HTML tags.
      That is, the actual content returned is:
          <a href=http://dap-dev.ncsa.illinois.edu:8184/file/200598567_browndog.jpg>http://dap-dev.ncsa.illinois.edu:8184/file/200598567_browndog.jpg</a>
      but result is only:
  http://dap-dev.ncsa.illinois.edu:8184/file/200598567_browndog.jpg
  
      python requests API:
  http://docs.python-requests.org/en/master/api/

...