Uploaded image for project: 'Daffodil'
  1. Daffodil
  2. DFDL-1799

Enable data stremaing in the CLI

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Normal
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 2.1.0
    • Component/s: CLI, Performance
    • Labels:
      None

      Description

      This CLI currently reads the input data as a byte array. This is simple and allows for ensuring all data is read into a memory, reducing disk overhead during the preformance command. However, this means the CLI is limited to the maximum size of an array, which is INT_MAX. In order to support the CLI parsing/unparsing larger files, we should instead work on InputStreams rather than array buffers. For the performance subcommand, this will mean requiring something like a SplittalbeInputStream that will allow multiple consumers of a single InputStream.

      Some SplittableInputStream implementations do exist, for example in JMRTD and on stack overflow, but licensing issues make it so these aren't an option. Either need to find a solution compatible with our license or implement our own.

      This work should be done concurrently with changes to improve the efficiency of the I/O layer.

        Gliffy Diagrams

          Attachments

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              slawrence Steve Lawrence
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:

                  Tasks

                  Progress: 
                   0/0