...
| Expand | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||
Long answer, unfortunately. Perhaps belongs in an article somewhere, but for now, here it is in this FAQ. (If I lapse into first person, this answer is by Mike Beckerle, DFDL Workgroup Co-chair.) With some study, the DFDL workgroup concluded that these formats nearly always require the full complexity of a transformation system AND a data format description system. DFDL is only about the latter problem. In other words, it was left out for complexity reasons, not because we didn't think there were examples. It is a much more complex issue than people think. As we got into it we kept falling down the slippery slope of needing rich transformations to express such things. We certainly have seen formats where there are a bunch of fields, in the ordinary manner, but instead of expressing their lengths, the forrmat specifies only their starting positions relative to start of record. There are also formats where there are tables of offsets into a subsequent data array. DFDL requires one to recast such a specification as lengths. It is not a "either or" scenario where lengths and offsets are equivalent so you can pick one. Use of lengths is simply a superior and more precise way of expressing the format because use of offsets can obscure aliasing, which is the term for when there are two elements (or more) that describe the same part of the data representation. With lengths, it's clear what every bit means, and that every bit is in fact described or explicitly skipped. You can't just use an offset to skip past a bunch of data leaving it not described at all. You can't have aliasing of the same data. Aliasing is a difficult issue when parsing. When unparsing it is a nightmare, as it introduces non-determinacy in what the data written actually comes out like. It depends on who writes it last with what alias. Structures like
So long as the things and the corresponding descriptor pairs are in order, these can be described. The lengths need not even be there as they are redundant. If present they can be checked for validity. Overlap can be checked for and deemed invalid. But, in DFDL the above *must* be represented as two vectors. One of the offsets table, the other of the things. If you want an array of things and then want DFDL to convert that into the offsets and things separately, well DFDL doesn't do transformations of that sort. Do that first in XSLT or other transformation system when unparsing. When parsing, you first parse with DFDL, then transform the data into the logical single vector using XSLT (or other).
If the things are potentially out of order, especially if the lengths are not stored, but just implied by "from this offset to the start of the next one, whichever one that is", that is simply too complex a transformation for DFDL. If you think about what is required mentally to decode this efficiently, you must grab all the entries, sort them by offset, and then compute lengths, etc. Shy of building a real programming language (e.g., XQuery) into DFDL there has to be a limit to what level of complexity we allow DFDL to express directly. And unparsing is entirely non-deterministic... you have to stage an array/blob filled with fill bytes, write pieces to it one by one, potentially overwriting sections. It's really quite hard. Even if you supported this in DFDL somehow, would it in fact write these things out in the order an application does? So will you even be able to re-create data? There is a sense in which formats expressed as these sorts of "potentially overlapping regions" are simply not adequately specified unless they specify the exact order things are to be written so that the contents of overlap regions is deterministic. I believe there are formats where there are offset tables like this, where in principle things could be out of order, or overlapping/aliased, but they simply never are, and allowing them to be is effectively a bad idea as it allows people to do very obscure things - information hiding, polyglot files, etc. PDF is heavily criticized for this. It may be an unstated principle that such formats *do not* do this sort of out-of-order or aliasing stuff. All that said. Practically speaking people have data with offset tables, and out-of-order might be a possibility that needs to be allowed at least on parsing. So what to do in DFDL? In this case, DFDL can describe the table of offsets, and a big blob of data. Beyond that something else (e.g., XSLT, or a program) must take over for expressing the sort and extraction of chunks out of the larger blob. If you think about this, if you want deterministic unparsing behavior, that is what has to be presented to the DFDL unparser anyway, since presenting the resolved content blob means the application has dealt with the order to which the various chunks (which may overlap) have been written. |
Q: I want these strings in the data to become element names... how can I do that?
| Expand | ||||||
|---|---|---|---|---|---|---|
If the data contains tags/strings, and you want those strings to become element names in XML, then you *must* do pass 1 to extract the tag information, use them as element names when you create a DFDL schema dynamically, and then parse the data again with this new specialized DFDL schema. Or you can parse the data with a generic schema where your tag names will be in element values someplace, and do a transformation outside of DFDL to convert them to element names. Consider the common "comma separated values" or CSV formats. If you have
and you want
That's what you would get from a generic CSV DFDL schema. If you want this:
That's a specific-to-exactly-these-column-names CSV DFDL schema that is required. If you have lots of files with this exact structure you would create this DFDL schema once. If you have no idea what CSV is coming at you, but want this sort of XML elements anyway, then you have to generate a DFDL schema on the fly from the data (parse just the headers with a generic DFDL schema first - then use that to create the DFDL schema (from a template - generally this is pretty easy.) Or you parse using the generic schema, then use XSLT or something to convert the result of the generic parse. Keep in mind that this problem has little to do with DFDL. Given an XML document like the generic one above, but you didn't want that XML, you wanted the specific style XML. Well you have the same problem. You need to grab the column names first, then transform the data using them as the element names. |