Page History

...

There are several related problems. For parsing:

1. Support for input streams or files that are larger than any particular fixed limit. (e.g. larger than the largest Java/Scala Array[Byte], or larger than 4Gbytes)
2. Support for individual DFDL Infoset objects larger than the largest Java/Scala object size. E.g, within a particular data format an MPEG video is found which is larger than 4Gbytes.

3. Support for DFDL Infosets larger than any particular Java/Scala JVM can hold in its virtual memory at one time.

4. Conversion of DFDL Infosets to XML (and JSON) as incrementally produced, so as to avoid the need to hold the entire XML document (or JSON) in memory.

For unparsing, the problem is simpler.

1. There is no analog to problem (1) above, as one can simply write data to a Java OutputStream.
2. Support for providing access to a large data object (larger than largest Java/Scala object size) via some sort of handle object that is placed into the DFDL Infoset Item, and having the Daffodil unparser obtain data from that handle for unparsing. This must not require bringing the entire object into memory even in pieces.
3. Support for incremental delivery of the Infoset to the unparser.
4. Incremental conversion of XML input data (or JSON) to the DFDL Infoset, so that we don't require the entire incoming XML document (or JSON) to reside in memory for unparsing.

Streaming Input

The I/O layer input system must be modified to do streaming - that is, all reading operations must be on finite buffers (CharBuffer and ByteBuffer) and must handle the underflow/overflow protocol so as to be restartable so that if one does not get "enough data", one can extend the buffer and fill it from the input, or make room for the data in the receiving buffer.

...

When projecting the DFDL infoset into XML, these handle objects would have to show up as the XML serialization of the handle object, with usable members so that other software can access the data the handle is referring to. One example would be that the handle contains a fileName or URI and an offset (type Long) into it, and a length (type Long), and possibly the first N bytes/characters of the data.

This mechanism needs to work both for parsing and unparsing; hence, an API way of constructing these large-data handle objects is needed.

Infoset Events

Infoset elements must be produced incrementally by the parser. These can only be produced once surrounding points of uncertainty are resolved fully. An architecture for this is needed. There may be some limitations.
This is closely related to the backtrack issues with being able to over-write state rather than allocate new state when parsing - or the changes should be in the same area of the code anyway.

The Daffodil API ProcessorFactory class has an onPath("...") method. (Currently only "/" is allowed as  a path.) This is intended to enable a cursor-like behavior if given a path that identifies an array. Successive calls to the DataProcessor's parse method should advance through the data one element of the array at a time, returning an Infoset each time which has as its root the successive InfosetElement items. Using this along with co-routines an event-based API can be produced.

Infoset elements must be deleted from the infoset once no longer needed so that the accumulated infoset does not grow endlessly.  To insure they aren't needed forever, any value that is to be referenced by an expression must be stored into a variable, and the referencing expression changed to dereference that variable. This requires the dfdl:newVariableInstance functionality, and some Daffodil compiler capabilities to identify and insert new variables at the proper scopes.

...