The Brown Dog Data Transformation Service (DTS) is a highly extensible/distributed service providing a uniform means of managing and accessing transformation capabilities within the web. Utilized tools can come in the form of command line applications, GUI driven applications, libraries, and/or other services. Here we go over the process of preparing a new transformation tool, either an extractor or a converter, for usage with the DTS.
...
BD-base runs the necessary dockerized Brown Dog Data Transformation Services (Clowder, Polyglot, Fence, RabbitMQ, MongoDB, Redis, an example extractor, an example converter, and the BD CLI) and combines them into one integrated program. You can get the BD-base by cloning the git repo:
| Code Block |
|---|
git |
...
clone https://opensource.ncsa.illinois.edu/bitbucket/scm/bd/bd-base.git |
or download the VirtualBox VM image and run it:
httphttps://browndog.ncsa.illinois.edu/downloads/bd-base.ova
After downloading BD-base, users can simply run the bash script in the command-line to start up the BD development base.
| Code Block |
|---|
cd bd-base ./bd |
The BD-base script will split your terminal into panes and start each of the services needed for the Brown Brown Dog DTS. This provides a useful and convenient way to view the logs of running services in panes.
...
Users can switch between panes using Tmux commands. The panes are as follows: Fence (top), Clowder (middle-left), example extractor (middle-right), Polyglot (middle-left), example convert (middle-right), and the BD CLI (bottom). Users can run BDCLI BD-CLI commands to interact with Brown Dog Data Transformation Services (username: bd, password: browndog):
...
Here we described the process for taking a working piece of code and deploying it as a Brown Dog extractor. It It is assumed that the method can be invoked from a single call. In In this example, we are using the python extractor wrapper and will invoke a python function. In In a very similar fashion, a method developed in a language other than python can be invoked using subprocess.
...
Rename and edit ImageMagick_convert.sh script to wrap your conversion tool. This script file should be named in the format <alias>_convert.<script_type>. Here <alias> needs to be replaced by the name of the conversion tool with which the converter registers with Polyglot and <script_type> needs to be replaced by the extension for the type of script this wrapper is written in. Polyglot currently supports scripts written in Python, Bash, R, AutoHotKey, AutoIT, and Sikuli (e.g. *.py, *.sh, etc.). For the sake of ease of explanation, we will rename the script file as MyTool_convert.sh. This script accepts three parameters:
...