Uploaded image for project: 'Daffodil'
  1. Daffodil
  2. DFDL-934

Streaming parser: Need to stream input data in, and infoset out to handle arbitrarily large data.

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: s13
    • Fix Version/s: 2.1.0
    • Component/s: Performance
    • Labels:
      None

      Description

      Currently Daffodil requires that all incoming data fit in one java.nio.ByteBuffer. A separate issue (DFDL-881) is about allowing > 4GB files, but data sizes would still be limited by available address space.

      A streaming approach has great advantages. It requires that the input can be streamed in (e.g., from a java.io.InputStream), but also requires that the DFDL Infoset can be streamed out. (Think SAX parser 'events' coming out). This is complicated by the DFDL notion of points of uncertainty. E.g., until a choice branch has been resolved none of the elements on any branch can be emitted since "backtracking" may invalidate them.

        Gliffy Diagrams

          Attachments

          There are no Sub-Tasks for this issue.

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              mbeckerle.dfdl Mike Beckerle
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:

                  Tasks