This story board concentrates on the way a developer would contribute an extractor or a format converter to the Brown Dog service. It includes setup of the development environment, testing, packaging/description and submission.
I am leaning more toward having the developer begin from the template project, i.e. an empty or bare-bones extractor or converter. We can include all the docker files that are necessary for the runtime environment with the template project, making it easy install/start/stop whatever is needed via docker-compose. See the Docker Compose workflow: https://docs.docker.com/compose/gettingstarted/#step-2-create-a-docker-image
Question: Is there any way to make the docker container describe itself in more detail and maybe include some sample files, such that a single upload will capture everything we need for submission of the contribution?
- Developer visits the Brown Dog developer/contributor web page, which provides an overview of what and how they can contribute. (TODO: create or enhance contributor web page)
- Tools catalogue page has some information on this. TODO: A link to the contributor web page with instructions - Developer installs prerequisites: possibly a development VM, Docker, python, python-pip TODO: Provide minimum installation instructions for these at a single place.
- Developer downloads a project template with all of the basics for an example extractor or converter project. (TODO: need a template project for converters) (Rui Liu has started this) (TODO: Create standalone extractor template project)
- Extractor template project: imagemagick edge detector
- Converter template project: imagemagick jpg to ico converter
Create a local project from the template:
git clone http://bitbucket.ncsa.illinois.edu/path/to/template.git myextractor
Note: If the user downloads the docker-extractor bare image, it is not required to git clone the template project. The docker image should have that extractor template project.
- Developer creates and activates a python virtual environment for their new project.
virtualenv myextractor source myextractor/bin/activate pip install docker-compose
- Developer runs "docker-compose up" to start runtime environment of docker containers (TODO: docker-compose.yml for extractor runtime environment) (TODO: docker-compose.yml for converter runtime environment) (TODO: Create any missing Docker containers)
cd myextractor docker-compose up
- Runtime containers for extractors: Clowder instance
- Runtime containers for converters: Polyglot instance
- Alternatively, to Step (3-4), Developer looks for docker-extractor or docker-converter to start with and for writing interface script for new tools. TODO: create basic docker-extractor and docker-converter.
- Example extractors and converters are there in (https://opensource.ncsa.illinois.edu/bitbucket/projects/BD/repos/dockerfiles/browse)
- TODO: use those as reference to create an image for basic docker-extractor and docker-converter.
- Include these template docker-images in the docker-compose.yml file for setting up the runtime-environment for developer to start writing custom scripts.
- Developer adds custom code to template project, creates interface script(s)
- Adds sample input file(s) (TODO: README )
- Smruti Padhy and Inna Zharnitsky will assist Gregory Jansen here
- Developer creates dockerfile
- For extractors this should contain only the tool, its dependencies, and the extractor
- For converters this should contain the basic Polyglot setup, ideally only the Software Server, the tool, its dependencies, and the converter interface script
- I have not done a converter, they are packaged as VMs right? How can they be contributed/distributed/packaged? - Gregory Jansen
- Developer submits tool to Tools Catalog (using BrownDog account)
- Interface script(s), dockerfile, sample input, sample output
- Through web interface
- Through script in development environment (probably the ideal way)