This story board concentrates on the way a developer would contribute an extractor or a format converter to the Brown Dog service. It includes setup of the development environment, testing, packaging/description and submission.

I am leaning more toward having the developer begin from the template project, i.e. an empty or bare-bones extractor or converter. We can include all the docker files that are necessary for the runtime environment with the template project, making it easy install/start/stop whatever is needed via docker-compose. See the Docker Compose workflow: https://docs.docker.com/compose/gettingstarted/#step-2-create-a-docker-image

Question: Is there any way to make the docker container describe itself in more detail and maybe include some sample files, such that a single upload will capture everything we need for submission of the contribution?

Overview of Steps

  1. Developer visits the contributor web page  BD-956 - Getting issue details... STATUS
    1. Contributor page explains at a high level how they can develop and contribute a two types of tools to the Brown Dog catalog.
    2. Explains benefits of contributing
    3. Explains intellectual property around contributing
    4. Examples and links to contributions by others 
    5. Tools catalogue page has some information on this.
    6. Provide a link to the contributor web page from Tools Catalogue and Brown Dog web pages.
  2. Developer visits either extractor or converter how-to page:
    1. Explains where their module fits within system diagram
    2. Explains development instructions step-by-step
      1. how to modify the template
      2. install path for any dependencies and source code
      3. input file description
      4. expected output
      5. how to upload the script to opensource repository.
    3. Provides links to resources they need 
    4. BD-957 - Getting issue details... STATUS
    5. BD-958 - Getting issue details... STATUS
  3. Developer installs prerequisites
    1. VirtualBox or other VM kit
    2. Development VM (if they need a foreign OS)
    3. Docker
    4. Python packages: python, python-pip
  4. Developer downloads a project template with all of the basics for an extractor or converter project.
    1. This is an empty project without functionality or dependencies, equivalent to "hello world".  (What does exactly Project Template mean?  I am assuming that it means a sample for writing extractor/converter- Smruti Padhy)
    2. Template project is documented, with placeholders where contributed code might go.
    3. Might include template files for tools catalog info (A tool metadata file that would be posted upon submission)
    4. BD-959 - Getting issue details... STATUS
      1. Converter Scripts within Polyglot can be found here (https://opensource.ncsa.illinois.edu/bitbucket/projects/POL/repos/polyglot/browse/scripts). Use these as reference to write the template (point (e)).
      2. May need a template project per target OS?
    5. BD-960 - Getting issue details... STATUS  
      1. Sample-extractors can be found in the pyClowder Library repo (https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder/browse) . Use this as reference to write the template extractor (point (d)).
    6. I think, there are two approaches to this: developer downloads example template such as imagemagick to run example extractor, see its working, study the code and later customizes it. But this will need the dependencies for that extractor which the developer may not need in future with the customization. So, the other way is to give just the skeleton of the extractor and the developer can install dependencies and write code. – Smruti Padhy 
      1. I agree, I think the empty template is more useful. We can document it thoroughly and throw in some dummy extracted data ("hello world"), without introducing more dependencies.
  5. Developer writes and tests code within template project.
    1. This involves edits to python or customization of a VM image (converters)
    2. Involves edits to the supplied Dockerfile for new dependencies, etc..
      1. For extractors this should contain only the tool, its dependencies, and the extractor
      2. For converters this should contain the basic Polyglot setup, ideally only the Software Server, the tool, its dependencies, and the converter interface script 
    3. Involves docker-compose up for testing
    4. BD-961 - Getting issue details... STATUS
      1. Developer could edit a template test script for this.
      2. Tests could be run on catalog submission as a validation step.
  6. Developer add sample data files to the sample data folder.
  7. Developer edits tool info file (for tools catalog entry)
  8. Developer runs Brown Dog contributor script, which submits the key data in the project  BD-962 - Getting issue details... STATUS
    1. Uses their Brown Dog account
    2. Includes interface script(s), dockerfile, sample input, sample output
    3. If you are submitting a dockerfile file, it should contain the interface scripts. 

Notes

  • Extractor example project: imagemagick edge detector
  • Converter example project: imagemagick jpg to ico converter
  • TODO: Create an extractor template written in Java, C++

Extractor How-To Details

  1. Create a local project from the template:

    git clone http://bitbucket.ncsa.illinois.edu/path/to/extractor-template.git myextractor

    Note: If the user downloads the docker-extractor bare image, it is not required to git clone the template project. The docker-extractor image should have that extractor template project.

  2. Developer creates and activates a python virtual environment for their new project.
    1. virtualenv myextractor
source myextractor/bin/activate
pip install docker-compose
  3. Developer runs "docker-compose up" to start runtime environment of docker containers (TODO: docker-compose.yml for extractor runtime environment) (TODO: docker-compose.yml for converter runtime environment) (TODO: Create any missing Docker containers)
    1. cd myextractor
docker-compose up
./run-tests.py (or something like that)
    2. Runtime containers for extractors: Clowder instance
    3. Runtime containers for converters: Polyglot instance

  4. Alternatively, to Step (3-4), Developer looks for docker-extractor or docker-converter to start with  and for writing interface script for new tools.  
    1. Example docker extractors and converters are there in  (https://opensource.ncsa.illinois.edu/bitbucket/projects/BD/repos/dockerfiles/browse) 
    2. TODO: Create an repository for a template docker-extractor and a template docker-converter. 
    3. While creating basic docker image for extractor, the template project should consists of  interface scripts with comments where to modify the code ( Follow Step 3-4, within a docker container). 
    4. upload the image to dockerhub and create dockerfile and README and upload to the repository. 
    5. Include these template docker-images in the docker-compose.yml file for setting up the runtime-environment for developer to enable start writing custom scripts.

TODO: Test the deployment based on Rob's script.

 

  • No labels