Brown Dog (BD) data transformation services, Data Access Proxy (DAP) and Data Tilling Service (DTS), provide data format conversions and metadata extractions from data content functionalities, respectively, through a set of consistent and programmable REST APIs. These services also exist as framework and are extensible by design. DAP, built on top of Polyglot framework and DTS, built on top of Clowder framework, support contribution of new BD tools from users across various research domains and making the tools available for other users to use. A BD tool is a Brown Dog script/program that wraps any piece of code, software, library or webservice and exposes their data format conversions and/or metadata extraction or analysis capabilities. The tool is customized to fit into the ecosystem of either Polyglot framework (data format conversions) or Clowder framework (metadata extractions), and is made available through DAP/DTS APIs.
Within the Polyglot framework, a tool is alternatively known as converter to emphasize on the data format conversion capabilities. A converter resides within a software server which hosts the third party software/library with convert functionality. The software server is the component that automates the use of conversion capabilities of any piece of code through the converter. A converter is a wrapper script, written in any scripting languages such as bash, R, Python, on any existing piece of code to leverage its convert capability.
Similarly, a tool is known as extractor within Clowder framework to emphasize on the extraction of metadata from data content. An extractor is a program that extracts metadata within the file content, analyzes the file’s contents and tags it according to some specific classification or criteria, etc. It resides in distributed environment such as cloud, as extraction service and listens to a message queue for extraction requests from DTS API. The extraction process is triggered based on specific file types and the metadata extracted are made available through the DTS API.
In the BD framework, extractor and converter are the extensible units, and can be contributed by researchers across different research domains. To enable contributions from research community, a Tools Catalogue was designed and implemented where a user can register and share their BD tool with other users.
How can I contribute to BD data transformation services?
- Identify your tool capabilities - extractions/analysis or conversions.
- Decide if your tool is best fit as converter within DAP or as extractor within DTS. Look into some existing examples in BD Tools catalogue (http://browndog.ncsa.illinois.edu/tools).
- Download bd-template project from the NCSA opensource repository (http://opensource.ncsa.illinois.edu/bitbucket/projects/BD/repos/bd-templates/)
- Follow the README instructions within bd-converter-templates and bd-extractor-templates, to set up the BD development environment for converter and extractor, respectively.
- Test the bd-templates with the test scripts provided within bd-templates.
- Follow the README instructions to write a new extractor/converter with your tool, thereby converting it to BD tool.
- Once tested in the local development environment, push the bd-tool code to the NCSA open-source repository(or github??).
- Register the new BD tool in the Tools catalogue. This will involve you filling up forms providing description, link to the code repository, a dockerfile, input /output file. Submit for admin approval.
- The admin goes through your tool, test it and approves it.
- On approval, your tool is visible others and you can share the tool across different domain.
Why should I contribute?
Researchers often build/develop new tools for their study/research to extract useful information from unstructured/semi-structured data. A lot of efforts goes into that and are often unacknowledged. The value of their research is usually measured in terms of publications and the analysis done on the data obtained from those unstructured data. Also, some of these extraction/conversion tasks are repeated by multiple researchers within the same domain of science. Towards acknowledging such efforts, we built BD Tools Catalogue where members of different research community could contribute and share their tools within BD framework. So, you get proper credit for your effort in creating the new tool.
How can I contribute if my tool uses a proprietary software?
Yes, you can submit your tool which uses third proprietary software. However, before it is being available as Data Transformation Services, we will require to review the license agreement. Our current focus has been on using opensource software for building bd tools.