Versions Compared

Old Version 13

changes.mady.by.user Mike Beckerle

Saved on

compared with

New Version 14

changes.mady.by.user Steve Lawrence

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Expand
titleSee Answer

Use CDATA with expressions and regular expressions, and generally to stop XML editors from messing with your DFDL schema layouts

Most XML editors will wrap long lines. So your

Code Block
languagexml
<a>foobar</a>

just might get turned into

Code Block
languagexml
<a>foobar
</a>

Now most of the time that is fine. But sometimes the whitespace really matters. One such place is when you type a regular expression.

In DFDL this can come up in this way:

Code Block
languagexml
<dfdl:assert testKind="pattern"> *</dfdl:assert>

Now the contents of that element is " *", i.e., a single space, and the "*" character. That means zero or more spaces in regex language.

If you don't want your XML tooling to mess with the whitespace do this instead:

Code Block
languagehtml/xml
<dfdl:assert testKind="pattern"><![CDATA[ *]]></dfdl:assert>

CDATA informs XML processors that you very much care about this. Any decent XML tooling/editor will see this and decide it cannot line-wrap this or in any way mess with the whitespace.

Also useful if you want to write a complex DFDL expression in the expression language, and you want indentation and lines to be respected. Here's an example:

Code Block
languagexml
<dfdl:discriminator><![CDATA[{
    if (daf:trace((daf:trace(../../ex:presenceBit,"presenceBit") = 0),"pbIsZero")) then false()
    else if
    (daf:trace(daf:trace(dfdl:occursIndex(),"occursIndex") = 1,"indexIsOne")) then true()
    else if
    (daf:trace(daf:trace(xs:int(daf:trace(../../ex:A1[daf:trace(dfdl:occursIndex()-1,"indexMinusOne")],
                                       "occursIndexMinusOneNode")/ex:repeatBit),
                       "priorRepeatBit") = 0,
              "priorRepeatBitIsZero")) 
    then false()
    else true()  
}]]></dfdl:discriminator>

If you get done writing something very deeply nested like this (and XPath style languages require this all the time), then you do NOT want anything messing with the whitespace.

About the xml:space='preserve' attribute: According to this thread on the stack overflow web site, xml:space is only about whitespace-only nodes, not nodes that are part whitespace. Within element-only content, the text nodes found between the elements are whitespace-only nodes. Unless you use xml:space='preserve', those are eliminated. None of the above discussion is about whitespace-only nodes. It's about value nodes containing text strings with surrounding whitespace.

Q: Why doesn't DFDL allow me to express my format using offsets into a file, instead of lengths?

Expand
titleSee Answer
Anchor
FAQ-topic-DFDL-offsets
FAQ-topic-DFDL-offsets

Long answer, unfortunately. Perhaps belongs in an article somewhere, but for now, here it is in this FAQ.

(If I lapse into first person, this answer is by Mike Beckerle, DFDL Workgroup Co-chair.)

With some study, the DFDL workgroup concluded that these formats nearly always require the full complexity of a transformation system AND a data format description system. DFDL is only about the latter problem.

In other words, it was left out for complexity reasons, not because we didn't think there were examples.

It is a much more complex issue than people think. As we got into it we kept falling down the slippery slope of needing rich transformations to express such things.

We certainly have seen formats where there are a bunch of fields, in the ordinary manner, but instead of expressing their lengths, the forrmat specifies only their starting positions relative to start of record. There are also formats where there are tables of offsets into a subsequent data array.

DFDL requires one to recast such a specification as lengths.

It is not a "either or" scenario where lengths and offsets are equivalent so you can pick one.

Use of lengths is simply a superior and more precise way of expressing the format because use of offsets can obscure aliasing, which is the term for when there are two elements (or more) that describe the same part of the data representation. With lengths, it's clear what every bit means, and that every bit is in fact described or explicitly skipped. You can't just use an offset to skip past a bunch of data leaving it not described at all. You can't have aliasing of  the same data.

Aliasing is a difficult issue when parsing. When unparsing it is a nightmare, as it introduces non-determinacy in what the data written actually comes out like. It depends on who writes it last with what alias.

Structures like

Code Block
<offset to start><length of thing>
<offset to start2><length of thing2>
...
<offset to startN><length of thingN>
thing
thing2
...
thingN

So long as the things and the corresponding descriptor pairs are in order, these can be described. The lengths need not even be there as they are redundant. If present they can be checked for validity. Overlap can be checked for and deemed invalid.

But, in DFDL the above *must* be represented as two vectors. One of the offsets table, the other of the things. If you want an array of things and then want DFDL to convert that into the offsets and things separately, well DFDL doesn't do transformations of that sort. Do that first in XSLT or other transformation system when unparsing. When parsing, you first parse with DFDL, then transform the data into the logical single vector using XSLT (or other).

Tip
titleUse XProc
XProc is a language for expressing chains of XML-oriented transformations like this. Calabash is an open-source XProc implementation, and the daffodil-calabash-extension provides Daffodil stages that have been created to enable creation of XProc pipelines that glue together transformations like XSLT with DFDL parse/unparse steps. This can be used to create a unit that runs both DFDL and an XSLT together for parse or for unparse (they would be different XSLTs).

If the things are potentially out of order, especially if the lengths are not stored, but just implied by "from this offset to the start of the next one, whichever one that is", that is simply too complex a transformation for DFDL. 

If you think about what is required mentally to decode this efficiently, you must grab all the entries, sort them by offset, and then compute lengths, etc.  Shy of building a real programming language (e.g., XQuery) into DFDL there has to be a limit to what level of complexity we allow DFDL to express directly.   And unparsing is entirely non-deterministic... you have to stage an array/blob filled with fill bytes, write pieces to it one by one, potentially overwriting sections. It's really quite hard. Even if you supported this in DFDL somehow, would it in fact write these things out in the order an application does? So will you even be able to re-create data?

There is a sense in which formats expressed as these sorts of "potentially overlapping regions" are simply not adequately specified unless they specify the exact order things are to be written so that the contents of overlap regions is deterministic.

I believe there are formats where there are offset tables like this, where in principle things could be out of order, or overlapping/aliased, but they simply never are, and allowing them to be is effectively a bad idea as it allows people to do very obscure things - information hiding, polyglot files, etc. PDF is heavily criticized for this. It may be an unstated principle that such formats *do not* do this sort of out-of-order or aliasing stuff.

All that said. Practically speaking people have data with offset tables, and out-of-order might be a possibility that needs to be allowed at least on parsing. So what to do in DFDL?

In this case, DFDL can describe the table of offsets, and a big blob of data. Beyond that something else (e.g., XSLT, or a program) must take over for expressing the sort and extraction of chunks out of the larger blob.

If you think about this, if you want deterministic unparsing behavior, that is what has to be presented to the DFDL unparser anyway, since presenting the resolved content blob means the application has dealt with the order to which the various chunks (which may overlap) have been written.