Steve Lawrence

Bump version to 1.1-SNAPSHOT for development

Add support for reloading pre-compiled schemas

- Adds a new boolean property "Pre-compiled Schema" that specified

whether or not the DFDL Schema File property is a path to an XML

schema or a pre-compiled schema.

- Modifies the cache to accept a cache key, which is now a tuple of

dfdlSchema and preCompiled properties.

DFDL-1709

Update to Daffodil 2.0.0

Ensure Bit/ByteOrderChange parsers always exist at root

If the dfdl:byteOrder or dfdl:bitOrder properties were not provided,

Daffodil did not complain, and it would not insert a Bit/ByteOrderChange

parser at root. This meant that Daffodil would use a default value,

which in some cases caused incorrect behavior. It also meant that the

check for valid bit and byte orders (e.g. not bigEndian &

leastSignificantBitFirst) never occurred since that only happend in

ByteOrderChange parsers, meaning it could allow invalid bit/byte order

combinations.

This changes Daffodil so the root element will always insert Bit and

ByteOrderChange parsers, ensuring that the validity of the properties

are correct and ensuring correct parsing/unparsing. This means that

byteOrder is now always required, even in some cases where it might not

technically be needed.

This also modifies the initial UState/PState creation to not set

bit/byte order, but to instead rely on the Bit/ByteOrderChange parsers

to always be inserted first thing.

DFDL-1835

Ensure Bit/ByteOrderChange parsers always exist at root

If the dfdl:byteOrder or dfdl:bitOrder properties were not provided,

Daffodil did not complain, and it would not insert a Bit/ByteOrderChange

parser at root. This meant that Daffodil would use a default value,

which in some cases caused incorrect behavior. It also meant that the

check for valid bit and byte orders (e.g. not bigEndian &

leastSignificantBitFirst) never occurred since that only happend in

ByteOrderChange parsers, meaning it could allow invalid bit/byte order

combinations.

This changes Daffodil so the root element will always insert Bit and

ByteOrderChange parsers, ensuring that the validity of the properties

are correct and ensuring correct parsing/unparsing. This means that

byteOrder is now always required, even in some cases where it might not

technically be needed.

This also modifies the initial UState/PState creation to not set

bit/byte order, but to instead rely on the Bit/ByteOrderChange parsers

to always be inserted first thing.

DFDL-1835

Calculate the approximate length of delimited and pattern lengthKinds for alignment

By assuming the length of delimited and pattern lengthKinds was 1,

Daffodil was unable to optimize out some aligngmnet unparsers, which

could lead to deadlocks.

This changes that so the length of delimited or pattern lengthKinds is a

multiple of the length of the encoding. This allows Daffodil to optimize

out some alignment unparsers and prevent deadlocks.

DFDL-1836

Calculate the approximate length of delimited and pattern lengthKinds for alignment

By assuming the length of delimited and pattern lengthKinds was 1,

Daffodil was unable to optimize out some alignment unparsers, which

could lead to deadlocks.

This changes that so the length of delimited or pattern lengthKinds is a

multiple of the length of the encoding. This allows Daffodil to optimize

out some alignment unparsers and prevent deadlocks.

DFDL-1836

Delete calabash-server and make this repo just about the extension

The server is more of a prototype and isn't maintained. The XML calabash

CLI is capable of using the extension with some configuration. The

piperack command, which comes with XML Calbash is a more fleshed out

server for communicating with XML calabash.

  1. … 72 more files in changeset.
fixup! Fix Xerces resource resolution for includes/imports

    • -0
    • +24
    /test-stdLayout/src/test/scala/org2/TestOrg2.scala
Fix Xerces resource resolution for includes/imports

After we found a resource (like from an include/import), we reported the

location to Xerces incorrectly. For example, say we resolved the

systemID at

org1/xsd/payload.xsd

to the absolute URI at

file:/foo/bar/baz/org1/xsd/payload.xsd

When we created the new Input to return where we found payload.xsd to

Xerces, we set the systemID to the original relative systemID

(org/xsd/payload.xsd), and set the baseURI to the absolute URI

(file:/foo/bar/baz/org1/xsd/payload.xsd). To Xerces, this meant that the

resource was found at the first path relative to the second path. But

the first path relative to the second path is

file:/foo/bar/baz/org1/xsd/org1/xsd/payload.xsd

So parts of the path end up doubled, and Xerces cannot actually find the

resource. We thought this was a bug in Xerces and added heuristics to

remove these doubled paths, but upon further inspection, it was just a

bug in how we created Input's that Xerces used.

This patch fixes this so that the Input sets the systemId to the

resolved absolute URI, and does not set a baseURI. Xerces will not try

to do any relative stuff and the absolute resource will work. This also

means we can remove the heuristics that tried to remove doubled paths

(but which did not actually work in non-trivial cases).

Also, fix how arrayIndexStack and occursBoundsStack modifications are

undone when errors occurr with Arrays. Rather than the Array pushing and

a child rep parser popping on error, the array should do both the push

and pop. The logic was wrong before and things were not popped correctly

in some cases.

DFDL-1832, DFDL-1183

    • -0
    • +0
    /test-stdLayout/src/test/scala/TestOrg2.scala
    • -0
    • +23
    /test-stdLayout/src/test/scala/org1/TestOrg1.scala
Fix Xerces resource resolution for includes/imports

After we found a resource (like from an include/import), we reported the

location to Xerces incorrectly. For example, say we resolved the

systemID at

org1/xsd/payload.xsd

to the absolute URI at

file:/foo/bar/baz/org1/xsd/payload.xsd

When we created the new Input to return where we found payload.xsd to

Xerces, we set the systemID to the original relative systemID

(org/xsd/payload.xsd), and set the baseURI to the absolute URI

(file:/foo/bar/baz/org1/xsd/payload.xsd). To Xerces, this meant that the

resource was found at the first path relative to the second path. But

the first path relative to the second path is

file:/foo/bar/baz/org1/xsd/org1/xsd/payload.xsd

So parts of the path end up doubled, and Xerces cannot actually find the

resource. We thought this was a bug in Xerces and added heuristics to

remove these doubled paths, but upon further inspection, it was just a

bug in how we created Input's that Xerces used.

This patch fixes this so that the Input sets the systemId to the

resolved absolute URI, and does not set a baseURI. Xerces will not try

to do any relative stuff and the absolute resource will work. This also

means we can remove the heuristics that tried to remove doubled paths

(but which did not actually work in non-trivial cases).

Also, fix how arrayIndexStack and occursBoundsStack modifications are

undone when errors occurr with Arrays. Rather than the Array pushing and

a child rep parser popping on error, the array should do both the push

and pop. The logic was wrong before and things were not popped correctly

in some cases.

DFDL-1832, DFDL-1183

    • -0
    • +23
    /test-stdLayout/src/test/scala/org1/TestOrg1.scala
    • -0
    • +24
    /test-stdLayout/src/test/scala/org2/TestOrg2.scala
Ensure the infoset exists before trying to display it in the debugger

If you run 'info infoset' before stepping at least once, there will be

no infoset to visit. This checks to make sure that an infoset exists

before trying to print it in the debugger.

DFDL-1834

Modify test to verify that dfdl:inputValueCalc and dfdl:outputValueCalc cannot be on the same element

DFDL-1027

SDE when arrays or optional elements contain dfdl:outputValueCalc property

DFDL-1546

SDE when arrays or optional elements contain dfdl:outputValueCalc property

DFDL-1546

Add license information for ICU

The ICU package does contain license information, but the link it points

to is dead. Manually override it to a link in the svn repo that works.

DFDL-1825

Throw an Not Yet Implmemented SDE for encodingErrorPolicy="error"

It isn't supported and can result in weird behavior and mark pool leaks.

Best to just not allow it.

DFDL-1828

Throw an Not Yet Implmemented SDE for encodingErrorPolicy="error"

It isn't supported and can result in weird behavior and mark pool leaks.

Best to just not allow it.

DFDL-1828

Improve nil checking for infoset outputters

The InfosetOutputters use _.isNilled to determine if an infoset element

is nil. However, _.isNilled can throw an exception if the nilled flag

has not been set (e.g. while debugging). This could lead to thrown

exceptions during the 'info infoset' debugger command. This patch adds a

new isNilled helper function to the InfosetOutputter trait that performs

the correct logic to check if the nilled flag is set and determine if an

element is nilled, and updates all the InfosetOutputters to use this

function.

DFDL-1829

Update to latest Daffodil version and API

Modify isError API logic to support validation errors

- isError now returns true if either there are processing errors OR

validation errors

- New methods, isProcessingError and isValidationError, are added to

give the ability to differentiate why isError is true

- Remove the canProceed() method and deprecate it in the java/scala API.

This is just a wrapper around !isError

- Rename status to processorStatus to clearly differentiate between the

new validationStatus

- Remove unused isValidationSuccess method

- Remove status from the processor constructors--this is a holdover from

when PState was immutable

DFDL-1821

  1. … 13 more files in changeset.
Add option to enable validation

Also, fix new namespace path to built-in-formats.xsd

DFDL-1819

Add option to enable validation

Also, fix new namespace path to built-in-formats.xsd

DFDL-1819

Move TestOutputValueCalcAndAlignment from scala-new to scala

Verifies DFDL-1518

Add CLI delimiterStack test

DFDL-1283

Comment out unused imports in scala-new

Ignore all *.sbt files in the root directory except for build.sbt

This is useful for automated build systems to add custom *.sbt files to

add customizations to the build. For example, adding custom sbt

configurations to publish to a local repository. Note that this is

necessary since our sbt configurations makes a release a SNAPSHOT if

there are any untracked files (e.g. custom sbt file) or files with

changes. If we don't ignore this *.sbt files, then it will also build a

SNAPSHOT. This all helps Daffodil build cleanly on a jenkins automated

build and publishing to artifactory.

DFDL-1666

Fix test to have correct value and move out of debug

F7FF came frome the specification, but it is wrong. It should be F7DC.

Fix the tests and moved it out of debug.

DFDL-1455

Use a ThreadLocal for w3c dom infoset inputters in CLI

w3c Documents are not thread-safe, even for reading. This is because the

implementation lazily evaluates parts of the tree and creates/destroys

various caches while reading. It is still safe to read the same tree

multiple times, just not at the same time. This patch adds a ThreadLocal

variable when creating w3c documents in the CLI, so that each thread has

a unique copy that it can use. This prevents any threading issue while

still keeping the total number of copies to minimum to save memory.

DFDL-1795

Use a ThreadLocal for w3c dom infoset inputters in CLI

w3c Documents are not thread-safe, even for reading. This is because the

implementation lazily evaluates parts of the tree and creates/destroys

various caches while reading. It is still safe to read the same tree

multiple times, just not at the same time. This patch adds a ThreadLocal

variable when creating w3c documents in the CLI, so that each thread has

a unique copy that it can use. This prevents any threading issue while

still keeping the total number of copies to minimum to save memory.

Also add W3CDOMInfosetInputter/Outputter to the Java and Scala API's,

alont with a comment describing the non-thread-safe nature of the

inputter.

DFDL-1795