The Brown Dog DTS is a highly extensible/distributed service providing a uniform means of managing and accessing transformation capabilities within the web. Utilized tools can come in the form of command line applications, GUI driven applications, libraries, and/or other services. Here we go over the process of preparing a new transformation tool, either an extractor or a converter, for usage with the DTS.
Extractors
Here is we described the entire process for taking a working piece of code and deploying it as a Brown Dog Extractorextractor. It is assumed that the method can be invoked from a single call. In this example, we are using the python extractor wrapper and will invoke a python function. In a very similar fashion, a method developed in a language other than python can be invoked using subprocess.
...
- Wrap the tool for use as an extractor in Clowder (and through that Brown Dog)
- Dockerize the extractor
- Deploy the extractor
- Add the extractor to the Tools Catalog
...
Code Block | ||
---|---|---|
| ||
USER root RUN apt-get update && apt-get install -y imagemagick |
6. Test
...
the Extractor
You can test your extractor as follows:
...
Code Block | ||
---|---|---|
| ||
INFO : pyclowder.extractors - Waiting for messages. To exit press CTRL+C |
Converters
In this sectionHere we described the process for taking a working piece of code (an application, library, other service, etc) and deploying it as a Brown Dog converter. In this example, we describe the creation of a converter using the popular image converter written using tool, ImageMagick.
...
1. Get Your Code Together
We have developed a template
...
converter. It is a simple image converter that
...
converts between different image formats using ImageMagick tool. Clone the template converter and rename the directory to an appropriate name that reflects the purpose of your converter
Code Block | ||
---|---|---|
| ||
git clone https://opensource.ncsa.illinois.edu/bitbucket/scm/bd/convertors-template.git
mv convertors-template/ <your_converter_name>
cd <your_converter_name> |
Rename and edit ImageMagick_convert.sh script to wrap your
...
conversion tool. This script file should be named in the format <alias>_convert.<script_type>. Here <alias> needs to be replaced by the name of the conversion tool with which the converter registers with Polyglot and <script_type> needs to be replaced by the extension for the type of
...
script this wrapper is written in. Polyglot currently supports scripts written in Python, Bash, R, AutoHotKey, AutoIT, and Sikuli (e.g. *.py, *.sh, etc.). For the sake of ease of explanation, we will rename the script file as MyTool_convert.sh. This script accepts three parameters:
- Full path to input file
- Full path to output file (including filename)
Full local path to available scratch space (optional)
This script will be used by the Software Server to
...
run the tool and carry out any requested conversions. The example script ImageMagick_convert.
...
sh that uses ImageMagick tool to convert images between different formats is shown below. The conversion script follows a specific header and is written as comments:
- First line is the shebang line
- Second line contains the name of the converter followed by the version if any
- Third line refers to the type of the data that it can convert
- Fourth line contains a comma-separated list of input file formats accepted by this converter
- Fifth line contains a comma-separated list of output file formats that this converter can generate
- This is followed by the actual code that does conversion.
2. Edit the Dockerfile
Modify the Dockerfile in the converter directory to
...
replace ImageMagick with MyTool. Specifically change line numbers 11, 15, 16 and 17. You need to also change other fields like maintainer and may need to add instructions to install any specific software required by your converter. For example, you can see instruction to install ImageMagick software in the example Dockerfile:
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
# Create softwareserver for polyglot.
FROM ncsapolyglot/polyglot:develop
MAINTAINER Rob Kooper <kooper@illinois.edu>
USER root
# - install requirements
# - enable shellscripts to be scanned
# - enable imagemagick conversion by adding to .aliases.txt
RUN apt-get update && apt-get -y install vim nano imagemagick && \
/bin/sed -i -e 's/^\([^#]*Scripts=\)/#\1/' -e 's/^#\(ShellScripts=\)/\1/' /home/polyglot/polyglot/SoftwareServer.conf && \
echo "ImageMagick" > /home/polyglot/polyglot/scripts/sh/.aliases.txt
# copy convert file to scripts/sh folder in container
# this is done to keep cache so you can debug script easily
COPY ImageMagick_convert.sh /home/polyglot/polyglot/scripts/sh/
RUN chown polyglot /home/polyglot/polyglot/scripts/sh/ImageMagick_convert.sh && \
chmod +x /home/polyglot/polyglot/scripts/sh/ImageMagick_convert.sh
# back to polyglot
CMD ["softwareserver"] |
...
Specifically, modify:
Code Block | ||
---|---|---|
| ||
echo "ImageMagick" > /home/polyglot/polyglot/scripts/sh/.aliases.txt |
...
to:
Code Block | ||
---|---|---|
| ||
echo "MyTool" > /home/polyglot/polyglot/scripts/sh/.aliases.txt |
...
modify:
Code Block | ||
---|---|---|
| ||
COPY ImageMagick_convert.sh /home/polyglot/polyglot/scripts/sh/ |
...
to:
Code Block | ||
---|---|---|
| ||
COPY MyTool_convert.sh /home/polyglot/polyglot/scripts/sh/ |
...
and modify:
Code Block | ||
---|---|---|
| ||
RUN chown polyglot /home/polyglot/polyglot/scripts/sh/ImageMagick_convert.sh && \
chmod +x /home/polyglot/polyglot/scripts/sh/ImageMagick_convert.sh |
...
to:
Code Block | ||
---|---|---|
| ||
RUN chown polyglot /home/polyglot/polyglot/scripts/sh/MyTool_convert.sh && \
chmod +x /home/polyglot/polyglot/scripts/sh/MyTool_convert.sh |
3. Test the Converter
Build the Dockerfile and start the converter
Code Block | ||
---|---|---|
| ||
docker-compose stop
docker build –t mytool .
docker-compose up |
Usage of BD-tmux
BD-tmux runs the necessary dockerized Brown Dog Data Transformation Services (Polyglot, Clowder, Fence, ImageMagick converter and OCR extractor) and combines them into one integrated program.
...