Daffodil is the first DFDL implementation to provide the advanced features of
- dfdl:inputValueCalc (IVC for short)
- dfdl:outputValueCalc (OVC for short)
- dfdl:hiddenGroupRef
In addition Daffodil implements almost the complete DFDL expression language, including all the xpath functions, and the DFDL-namespaced functions such as dfdl:valueLength and dfdl:contentLength (though with some restrictions - as of this writing - on when length units of 'characters' can be specified.)
These features are essential for highly complex binary formats like:
- PCAP
- Asterix
- MIL-STD-2045 and related MIL-STD and STANAGs
This page is a location to centralize notes about implementation of advanced DFDL features, and to highlight places where the DFDL specification may need to be clarified, augmented, or corrected.
SDE for element in hidden group that has no default, nor dfdl:outputValueCalc
Within a hidden group, if an element has no fixed/default value, and no OVC, then because it is hidden and so does not appear in the infoset, there's no way for it to get a value at all when unparsing.
Daffodil issues an SDE in this situation as is required by the DFDL spec.
This SDE is too strong: If the element has IVC, then it doesn't need a value for unparsing unless an expression that is used at unparse time (i.e., not in an assert or discriminator) references the element. In many cases the value of an IVC may be used only in dfdl:occursCount expressions, or in assert/discriminator test expressions. Since those aren't evaluated when unparsing, there is no unparsing situation would never need the value, so the SDE is too strong.
IVC and OVC on the same element
In many situations we have found that we need both IVC and OVC on the same element. This occurs when the element is in a hidden group, and is essentially being used as a variable.
The dfdl:newVariableInstance doesn't eliminate this problem, as there are situations where one needs an "array variable", that is a variable associated with each index of an array, but referenced from expressions used to compute things outside of the scope of that array element.
TBD: is OVC truly needed here, or is this a result of the SDE above for elements that have no way to get a value at unparse time - but the element turns out to be unused at unparse time.
Array Variables and dfdl:newVariableInstance
In many cases dfdl:newVariableInstance is not sufficient because what are needed are array variables.
.e., when parsing element i+1 you have to refer to a "variable" defined as part of element i.
You can't do that with dfdl:newVariableInstance because of the scope it has. If you introduce a dfdl:newVariableInstance for an array element, it's scope will be that element only. If you introduce it outside the array element there will be only one instance for the whole array.
But you can do an "array variable" with a hidden element. (Doesn't have to be hidden, but that's the sensible way to use it.)
The headache being the relative path to it., and then the fact that the most natural thing to do is to put both IVC and OVC on it (though the OVC may not be needed in many cases).
Forward reference from dfdl:setVariable and dfdl:newVariableInstance Expressions
Consider unparsing with dfdl:setVariable expression referring to OVC forward-referencing element.
<xs:element name="len" type="xs:int" dfdl:outputValueCalc="{ dfdl:valueLength(../data) + 10 }"/>
<xs:sequence>
<xs:annotation><xs:appinfo ...>
<dfdl:setVariable name="var">{ ../len }" />
</xs:appinfo></xs:annotation>
<xs:element name="data" dfdl:length="{ $var }">
....
</xs:element>
....
</xs:sequence>
In the above, when unparsing, the output value calc for len can be evaluated, but we must delay its evaluation and unparsing until the subsequent data element is available.
The next thing the unparse has to do, after delaying the unparsing of 'len' is set the var variable. This requires the value of len, which has been deferred.
We have real schemas (e.g., even PCAP) where this occurs.
This breaks the rule that when variables are evaluated, things they reference must have already been evaluated. Basically, when we delay evaluating the OVC for len we are suspending anything that depends on len having a value as well.
fn:error(...) Function needed to issue errors when unparsing
When parsing, one has the alternative of creating a dfdl:assert that tests something and fails if it doesn't hold.
However, dfdl:assert expressions aren't evaluated when unparsing; hence, a way to indicate an error in data during unparse-time expression evaluation is needed.
The XPath fn:error(...) function is suitable.