Skip to end of metadata
Go to start of metadata

Obtain Source Code

The source code is available from the NCSA opensource Git repository:

git clone https://opensource.ncsa.illinois.edu/stash/scm/dfdl/daffodil.git

Our Development Workflow uses review branches before integrating back into the master branch.

Requirements & Building from Source

Daffodil requires Java 1.8 or higher. All scripts provided for running and testing are written in bash and tested in GNU Linux. Those scripts are not required though, so Daffodil should be able to run in other platforms as long as Java is installed

SBT is required for building, and dependency management.  See the README and BUILD files in the root of your local clone of the git repository.  In short, you need to download and put the correct SBT Launcher in your path.  Daffodil is currently using version 0.13.12 of SBT.  SBT is available here: http://www.scala-sbt.org/release/docs/Getting-Started/Setup.html#installing-sbt

Daffodil is written in Scala 2.11.8, which also must be installed in order to run Daffodil.

Note: You will likely need to set available JVM memory larger. For example, on a 64-bit Linux system, I have this in my .bashrc:

export JAVA_OPTS="-Xmx6000m -Xss3m"

This might seem generous (6Gig!) The Xss3m option makes the default stack size larger.

Compiling in eclipse

There are IDE's and eclipse plug-ins that can be used to build daffodil. See:

  http://www.scala-lang.org/node/91

Tricks/Tips: You must increase the stack-size that Eclipse's JVM will use. This is done by editing eclipse.ini in the eclipse install directory. Below is an example of an eclipse.ini that works.

The key change is the last line. You need to enlarge this stack size option from the default because the scala compiler is highly recursive, as is the Daffodil schema compiler.

It is also very helpful when dealing with XML files to load the XML and XML-schema tools that are available in Eclipse.

-startup
plugins/org.eclipse.equinox.launcher_1.2.0.v20110502.jar
--launcher.library
plugins/org.eclipse.equinox.launcher.gtk.linux.x86_1.1.100.v20110505
-showsplash
org.eclipse.platform
--launcher.defaultAction
openFile
-vmargs
-Xss3m

Eclipse XML Setup

Getting eclipse to validate DFDL Schemas using its XML validation capabilities requires some configuration. Without this you will get hundreds (at least) of XML and XML Schema validation errors from the daffodil code base.

There are two files of interest to this

  • daffodil-lib/src/main/resources/eclipse-xml-catalog.xml
  • daffodil-core/src/main/resources/eclipse-settings.zip

The eclipse-xml-catalog is imported into Eclipse via File->Import->XML->XML Catalog.

The eclipse-settings zip file contains a .settings directory to be restored in your Eclipse workspace's ".metadata/.plugins/org.eclipse.core.runtime/.settings" directory.

These settings and catalog will change the configuration of eclipse substantially, and may not be suitable for projects other than Daffodil, so save your existing eclipse install directory and workspace's .metadata/.plugins/org.eclipse.core.runtime/.settings directory, if you are trying to share it across more than just Daffodil. You'll likely need to split out into two distinct workspaces and two distinct copies of eclipse.

With the settings and XML Catalog above, you can edit DFDL Schemas, and get support from Eclipse.

For example,

  • Edit core/tests/AA.dfdl.xsd in eclipse. Notice that it does not use xmlns="http://www.w3.org/2001/XMLSchema", which is the usual name space for XML Schema. Rather, it uses "http://www.ogf.org/dfdl/dfdl-1.0/XMLSchemaSubset" which provides much more DFDL-oriented schema validation.
  • Note that the file extension ".dfdl.xsd" is the standard way to identify a DFDL schema file.
  • Change one of the dfdl:inputValueCalc properties and misspell the property name. Save the file and you should see a validation error.
  • Validation using the DFDL XML Schema Subset URL will validate short form and long form DFDL annotations, and enforce DFDL's subset of XML Schema.

Contrarywise....

  • Edit core/tests/AH.dfdl.xsd in eclipse. Notice that it uses xmlns="http://www.w3.org/2001/XMLSchema", which is the usual name space for XML Schema. 
  • Note that the file extension ".dfdl.xsd" is the standard way to identify a DFDL schema file.
  • Change one of the dfdl:inputValueCalc properties and misspell the property name. Save the file and you will NOT see any validation error.
  • Change one of the attributes of the dfdl:defineVariable element, such as remove the name attribute and put in a ref="foo" attribute. Save, and you should see a validation error.
  • Validation using the standard XML Schema URL will NOT validate short form DFDL annotations, nor will it care if you put a long form annotation element named dfdl:foobar. But if you do get the name of a DFDL annotation element correct, then it will validate the properties on that annotation element.

Eclipse Unicode Setup

On Linux, UTF-8 and good unicode fonts are the default.

On Windows, a US install normally doesn't default to a Unicode friendly environment. One of the features of the Scala programming language is that it is completely unicode aware.

But, to make a USA windows default install of Windows itself, and Eclipse unicode friendly, you have to take a few steps.

These are described in the solution to this issue: https://opensource.ncsa.illinois.edu/jira/browse/DFDL-81. See the comments at the bottom for the settings. Basically, you have to install the Microsoft Japanese Language Pack.

Intellij Setup

Intellij comes ready-equipped to support Daffodil with only a few changed required to add the plugins for Scala and SBT support.

In order to make sure all the required libraries, including Scala, are downloaded you should first run sbt from the command line.  This will call Ivy to retrieve any missing libraries so that when you run Intellij you can point to the right libraries for the project build.

Next, in the Intellij UI, open File->Settings and in the left pane scroll down to the section labeled Plugins (this is below the middle divider).  In plugins select "Browse repositories,,," and from this list select "Scala" and "SBT" (use the search feature to help find them).  Then click the Download and install icon (the disk or box with the arrow toolbar button or via context menu).  When the two plugins are loaded, close Intellij and restart.

Once restarted, reopen settings and open the SBT option under the project settings (you can ignore the Scala settings as these defaults are sufficient).  Under "SBT launcher JAR file" enter the path to where you downloaded the sbt-launch,jar above and under the VM parameters enter the value specified by the JAVA_OPTS export.

Next you need to configure daffodil to use the version of the Scala compiler downloaded by Ivy.  This will be a local setting and not the global FCS setting so each module has to be updated individually.  Right-click on daffodil-core and select "Open Module Settings".  Then for each module, open up it's children, select "Scala" from the list of children and click the "Use ordinary compiler" for the Compiler Installation.  This should default to scala-2.9.1 which is the version downloaded by Ivy and configured in the Intellij project.

Finally, under the Edit Configurations menu (drop-down) make sure you have a target.  You can create a new target to build all tests by clicking the plus sign and then selecting "JUnit" as the test type.  Then in the dialog make sure you select the correct class path (i.e. daffodil-core) and path to the core and then check "Run SBT Action" and in the "..." dialog select "test:products".

  • No labels

3 Comments

  1. Also you'll want to turn off HTML Validation for eclipse under Window->Preferences->Validation. There's tons of validation errors in some of the libsrc materials, and they're just annoying.

  2. I just did a fresh Eclipse install for Daffodil development on windows.

    I got some very wierd error messages until I changed it from using Java-7 compilation to Java-6 compilation.

    This is under Window->Preferences->Java->Compiler->Compiler Compliance Level should be 1.6, not 1.7. This then requires a clean and build to clear out any java 1.7-binary code.

     

  3. After struggling with SBT for the past day I've found it better not to give Java a hint as to how much memory SBT will actually need.  It fails on a 32-bit machine to allow when set to 6GB.  So I recommend removing the -Xmx6000m option from the call to Java.