Uploaded image for project: 'BrownDog'
  1. BrownDog
  2. BD-1896

Investigate polyglot conversion timeout.

    XMLWordPrintableJSON

    Details

    • Type: Task
    • Status: Done
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: None
    • Component/s: None
    • Labels:
      None
    • Sprint:
      Brown Dog - Sprint 5, Brown Dog - Sprint 6, Brown Dog - Sprint 7

      Description

      Sometimes, polyglot conversion will take 30 minutes to complete conversion. For example, DOCX to TXT. It does not happen everytime.

      some idea:

      First, need to test DOCX to TXT unoconv converter itself on this test input file.

       

      http://bd-test.ncsa.illinois.edu/report.html?id=5a0d91c127f528615a890965

      logs of bd-test

      Name test_get_convert[Test document conversions, DOCX to TXT] Classname test_conversion Time 300.167852879 seconds Message AssertionError: File was not downloaded System Out Description : Test document conversions, DOCX to TXT
      Converting from : http://browndog.ncsa.illinois.edu/examples/Astell1700.docx
      Converting to : txt
      API Call : https://bd-api-dev.ncsa.illinois.edu/conversions/txt/http%3A%2F%2Fbrowndog.ncsa.illinois.edu%2Fexamples%2FAstell1700.docx
      Output path : /tmp/tmp5MuFpO.txt
      File url : https://bd-api-dev.ncsa.illinois.edu/dap/file/205742088_Astell1700.txt Trace

      host = 'https://bd-api-dev.ncsa.illinois.edu' api_token = 'api_token', request_timeout = 5 processing_timeout = 300 conversion_data = ordereddict([('description', 'Test document conversions, DOCX to TXT'), ('file...browndog.ncsa.illinois.edu/examples/Astell1700.docx'), ('output_type', 'txt')]) def test_get_convert(host, api_token, request_timeout, processing_timeout, conversion_data): # should this test be skipped if 'skip' in conversion_data: pytest.skip(conversion_data['skip']) if 'file_url' in conversion_data: > convert(host, api_token, request_timeout, processing_timeout, conversion_data, convert_by_url, 'file_url') test_conversion.py:14: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ host = 'https://bd-api-dev.ncsa.illinois.edu' api_token = 'api_token', request_timeout = 5 processing_timeout = 300 conversion_data = ordereddict([('description', 'Test document conversions, DOCX to TXT'), ('file...browndog.ncsa.illinois.edu/examples/Astell1700.docx'), ('output_type', 'txt')]) convert_func = file_field = 'file_url' def convert(host, api_token, request_timeout, processing_timeout, conversion_data, convert_func, file_field): print "Description :", conversion_data['description'] print "Converting from :", conversion_data[file_field] print "Converting to :", conversion_data['output_type'] stoptime = time.time() + processing_timeout input_filename = conversion_data['file_url'] output = conversion_data['output_type'] tf = tempfile.NamedTemporaryFile(dir='/tmp') output_path = tf.name + '.' + output r = convert_func(host, api_token, input_filename, output, request_timeout) if r.status_code == 200: print "Output path :", output_path print "File url :", r.text if basename(output_path): output_filename = output_path else: output_filename = output_path + basename(r.text) downloaded_filename = download_file(r.text, output_filename, api_token, stoptime) > assert os.path.isfile(downloaded_filename), "File was not downloaded" E AssertionError: File was not downloaded E assert False E + where False = ('/tmp/tmp5MuFpO.txt') E + where = <module 'posixpath'="" from="" '="" home="" browndog="" bd-test="" lib="" python2.7="" posixpath.pyc'="">.isfile E + where <module 'posixpath'="" from="" '="" home="" browndog="" bd-test="" lib="" python2.7="" posixpath.pyc'=""> = os.path test_conversion.py:39: AssertionError

       

      logs:

      [Thu Nov 16 07:06:13 2017] [restlet] [205742088]: 141.142.209.84 request for http://browndog.ncsa.illinois.edu/examples/Astell1700.docx->txt will be at http://dap-dev.ncsa.illinois.edu:8184/file/205742088_Astell1700.txt [Thu Nov 16 07:36:15 2017] [restlet] [205742088]: Software Server at 141.142.208.85 checked in result for job-205742088, http://141.142.227.81:8184/file/205742088_Astell1700.txt (111.8 KB) [Thu Nov 16 07:36:15 2017] [restlet] [205742088]: [Begin Software Server Log - 141.142.208.85] ========= [Thu Nov 16 13:06:14 2017] [restlet] [2]: Setting session to session-2, result will be at http://172.17.0.8:8182/file/2_Astell1700.txt [Thu Nov 16 13:06:14 2017] [restlet] [2]: Downloading http://browndog.ncsa.illinois.edu/examples/Astell1700.docx (84.0 KB) ... [Thu Nov 16 13:06:14 2017] [sserver] [2]: Executing, scripts/sh/unoconv_convert.sh /home/polyglot/polyglot/tmp/SoftwareServer/Cache/2_Astell1700.docx /home/polyglot/polyglot/tmp/SoftwareServer/Cache/2_Astell1700.txt /home/polyglot/polyglot/tmp/SoftwareServer/Temp/2_1510837574537_ ... [Thu Nov 16 13:36:14 2017] [sserver] [2]: Killing, scripts/sh/unoconv_convert.sh /home/polyglot/polyglot/tmp/SoftwareServer/Cache/2_Astell1700.docx /home/polyglot/polyglot/tmp/SoftwareServer/Cache/2_Astell1700.txt /home/polyglot/polyglot/tmp/SoftwareServer/Temp/2_1510837574537_ ... [Thu Nov 16 13:36:14 2017] [restlet] [2]: Execution complete, result at /home/polyglot/polyglot/tmp/SoftwareServer/Cache/2_Astell1700.txt (111.8 KB) [Thu Nov 16 13:36:14 2017] [restlet] [2]: Copied result to public folder [Thu Nov 16 07:36:15 2017] [restlet] [205742088]: ============ [End Software Server Log - 141.142.208.85] [Thu Nov 16 07:36:16 2017] [restlet] [205742088]: [Begin Siegfried Log - 205742088_Astell1700.txt] =========

       

        Gliffy Diagrams

          Attachments

            Activity

              People

              Assignee:
              bing Bing Zhang
              Reporter:
              bing Bing Zhang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 4 hours Original Estimate - 4 hours
                  4h
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 1 day, 2 hours
                  1d 2h

                    Tasks