You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 29 Next »

This story board concentrates on the way a developer would contribute an extractor or a format converter to the Brown Dog service. It includes setup of the development environment, testing, packaging/description and submission.

I am leaning more toward having the developer begin from the template project, i.e. an empty or bare-bones extractor or converter. We can include all the docker files that are necessary for the runtime environment with the template project, making it easy install/start/stop whatever is needed via docker-compose. See the Docker Compose workflow: https://docs.docker.com/compose/gettingstarted/#step-2-create-a-docker-image

Question: Is there any way to make the docker container describe itself in more detail and maybe include some sample files, such that a single upload will capture everything we need for submission of the contribution?

  1. Developer visits the Brown Dog developer/contributor web page, which provides an overview of what and how they can contribute. 
    1.  TODO: create or enhance contributor web page
    2. Tools catalogue page has some information on this. TODO: A link to the contributor web page with instructions from Tools Catalogue and Brown Dog webpages.
  2. Developer installs prerequisites: possibly a development VM, Docker, python, python-pip 
    1. TODO: Provide minimum installation instructions for these at a single place.
  3. Developer downloads a project template with all of the basics for an example extractor or converter project. 
    1. TODO: need a template project for converters(Rui Liu has started this). Scripts within Polyglot can be found here (https://opensource.ncsa.illinois.edu/bitbucket/projects/POL/repos/polyglot/browse/scripts). Use these as reference to write the template (point (e)). 
    2. TODO: Create standalone extractor template project. Sample-extractors can be found in the pyClowder Library repo (https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder/browse) . Use this as reference to write the template extractor (point (d)).
    3.  I think, there are two approaches to this: developer downloads example template such as imagemagick to run example extractor, see its working, study the code and later customizes it. But this will need the dependencies for that extractor which the developer may not need in future with the customization. So, the other way is to give just the skeleton of the extractor and the developer can install dependencies and write code. – Smruti Padhy
    4. Extractor template project: imagemagick edge detector -  
    5. Converter template project: imagemagick jpg to ico converter
    6. Extractor template written in Java, C++

    7. Create a local project from the template:

      git clone http://bitbucket.ncsa.illinois.edu/path/to/template.git myextractor

      Note: If the user downloads the docker-extractor bare image, it is not required to git clone the template project. The docker-extractor image should have that extractor template project.

  4. Developer creates and activates a python virtual environment for their new project.
    1. virtualenv myextractor
source myextractor/bin/activate
pip install docker-compose
  5. Developer runs "docker-compose up" to start runtime environment of docker containers (TODO: docker-compose.yml for extractor runtime environment) (TODO: docker-compose.yml for converter runtime environment) (TODO: Create any missing Docker containers)
    1. cd myextractor
docker-compose up
    2. Runtime containers for extractors: Clowder instance
    3. Runtime containers for converters: Polyglot instance

  6. Alternatively, to Step (3-4), Developer looks for docker-extractor or docker-converter to start with  and for writing interface script for new tools.  
    1. Example docker extractors and converters are there in  (https://opensource.ncsa.illinois.edu/bitbucket/projects/BD/repos/dockerfiles/browse) 
    2. TODO: use those as reference to create an image for basic docker-extractor and docker-converter.  While creating basic docker image for extractor, the template project should consists of  interface scripts with comments where to modify the code ( Follow Step 3-4, within a docker container). 
    3. upload the image to dockerhub and create dockerfile and README and upload to the repository. 
    4. Include these template docker-images in the docker-compose.yml file for setting up the runtime-environment for developer to enable start writing custom scripts.
  7. Developer adds custom code to template project, creates interface script(s)
    • The template interface script should consists of 
    • Adds sample input file(s)
    • TODO: Add  README to the template as how to modify the template, install path for any dependencies and source code, input file description, expected output.
  8. Developer creates dockerfile
    • For extractors this should contain only the tool, its dependencies, and the extractor
    • For converters this should contain the basic Polyglot setup, ideally only the Software Server, the tool, its dependencies, and the converter interface script
      • I have not done a converter, they are packaged as VMs right? How can they be contributed/distributed/packaged? - Gregory Jansen
  9. Developer submits tool to Tools Catalog (using BrownDog account)
    • Interface script(s), dockerfile, sample input, sample output
    • Through web interface
    • Through script in development environment (probably the ideal way)

 

  • No labels