...

We have this ongoing task 

Definitions

This section will define the main concepts that will need to be clearly understood for easily interacting with the Tools CatalogBD Transformations. An example can be Tool Software vs Script Transformation or any new terminology that will be used in Tools CatalogBD Transformations.

Converter: Software Program that does data format conversion.

Extractor: Software that Program that process and extract metadata from data.

Transformation: Any converter or extractor or extractor program.

Software: Software tool / library that is used by an extractor or converter to do data transformation.

...

Minimum Viable Product (MVP): Version of Tools Catalog 2.0 BD Transformations with only a few basic features. This will be developed through some kind of a hackathon.

User Roles

The user roles in the Tool Catalog 2.0 BD Transformations will be:

1. Administrator
   1. Update / Delete transformations created by contributors
   2. Update / Delete tools added software added by contributors?
   3. All Viewer permissions
2. Contributor
   
   1. Create new transformation and link it to one or more existing software.
   2. Add a new software to the Tools CatalogBD Transformations
   3. Update created transformations [adding the word "created" to make it clear that a contributor can only manage the transformations they contributed]
   4. Delete created transformations
   5. View created transformations
   6. All Viewer permissions
3. Viewer (can be any user who does not login)
   1. View all transformations
   2. View all software

...

• All users can view all transformations that are linked to a software.
• All users can view a single transformation and the software that it uses.
• Contributors will be able to add a transformation through the Tools Catalog interfaceBD Transformations interface.
  • This will be made available through a form that the contributor can fill and submit.
  • Ability to include icon
  • Ability to include link to software web page (e.g. Image JImageJ)
  • Ability to add a URL to the transformation  - source code / docker image name / website
• Contributors will also be able to import a new transformation into the catalog by uploading an extractors_info.json or converters_info.json file.
  • Any missing piece of information can be filled in by the contributor.
• Contributors will be able to link their transformation with one more existing software in the Tools CatalogBD Transformations. They will be able to add a new tool software if it doesn't already exist in BD Transformations.
• There won't be any formal approval process for the submitted transformations. All transformations that get submitted will be available to the public.
• Administrators can update or delete any transformation as they feel appropriate.

Wireframes

...

Wireframe
initialResourceID 2278E287-509B-183B-1098-2EC38DDDB7D8 platformArchiveID 137433183 Alignment Left view grid platformArchiveName BalsamiqProject_137432356 initialBranchID Master

1. Home (Yan)

2. View Transformation (Sandeep)

3. Login (Bing)

4. POST Endpoint

5. Administrator

6. Contributor (Low priority for now)

Technologies 

• Flask Templates
• Bootstrap
• JQuery
• Flask Services
• MongoDB
• Python LDAP Library
• Python MongoDB Library

Tasks

• Script for exporting data from Tools Catalog to BD Transformations
• Code to talk to NCSA LDAP for authentication
• Clusterman endpoint to read from BD Transformations database
• Flask Services
  
  • POST <extractors/converters>_info.json to MongoDB
  • GET <extractors/converters>_info.json for a specific transformation
  • GET all transformations
  • GET all transformations that uses a specific software

JSON Document Example

Mapping Table

Note: Changing to camel case notation for field names as MongoDB

...

• Code to talk to NCSA LDAP for authentication

documents follow JSON.

status

creationDate

dateCreated

updateDate

dateUpdated

downloads

license

whatsnew

description

compatibility

url

url

sample_input

sample_output

interfaceLevel

dockerimageName

dockerImageName

vmImageName

dockerfile

author

author

dependencies

dependencies

transformation_type

transformationType

transformation_id

transformationId

title

title

Transformations Catalog Example

{
    "_id" : ObjectId("53864aee7400009900ec8d22"),
    "dateCreated" : ISODate("2014-05-28T20:45:34.412Z"),
    "dateUpdated" : ISODate("2016-11-21T04:09:47.836Z"),
    "description" : "Identify tumors in microscopy images.",
    "url" : "https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler/cellprofiler-tumor?raw",
    "dockerImageName" : "",
    "author" : "ruiliu@illinois.edu",
    "dependencies" : [
           "Cell Profiler"
    ],
    "externalServices" : [
    ],
    "transformationType" : "extractor",
    "transformationId" : "",
    "title" : "Tumors",
    "status" : "submitted"
}

Extractor Info Example

{
    "@context": "http://clowder.ncsa.illinois.edu/contexts/extractors.jsonld",
    "name": "ncsa.csvheaders",
    "version": "1.0",
    "description": "Extracts the headers of CSV file. Only looks at the first row.",
    "author": "Yan Zhao <yanzhao3@illinois.edu>",
    "contributors": ["Luigi Marini <lmarini@illinois.edu>"],
    "contexts": [
        {
            "csv_headers": "http://clowder.ncsa.illinois.edu/metadata/ncsa.csvheaders#csv_headers"
        }
    ],
    "repository": [
        {
            "repType": "git",
            "repUrl": "https://opensource.ncsa.illinois.edu/bitbucket/scm/iarp/extractors.git"
        },
        {
            "repType": "docker",
            "repUrl": "clowder/extractors-csvheaders"
        }
    ],
    "process": {
        "file": [
          "text/csv"
        ]
    },
    "external_services":[],
    "dependencies": [],
    "bibtex": []
}

Converter Info Example

{
    "name": "ImageMagick",
    "version": "1.0",
    "description": "Converts images from one format to another",
    "author": "Kenton McHenry <mchenry@illinois.edu>",
    "contributors": ["Bing Zhang <bing@illinois.edu>"],
    "repository": [
        {
            "repType": "git",
            "repUrl": "opensource.ncsa.illinois.edu/bitbucket/scm/pol/converters-imagemagick.git"
        },
        {
            "repType": "docker",
            "repUrl": "ncsapolyglot/converters-imagemagick"
        }
    ],
    "input_formats": [
          "png", "bmp"
        ],
    "output_formats": [
          "jpeg", "jpg", "pdf"
        ],
    "external_services":[],
    "dependencies": ["ImageMagick"],
    "bibtex": []
}

Current Tools Catalog Example

{
    "_id" : ObjectId("53864aee7400009900ec8d22"),
    "status" : 4,
    "creationDate" : ISODate("2014-05-28T20:45:34.412Z"),
    "updateDate" : ISODate("2016-11-21T04:09:47.836Z"),
    "downloads" : 2,
    "license" : "UIUC/NCSA Open Source License.\r\nhttp://otm.illinois.edu/uiuc_openSource",
    "whatsnew" : "Identify tumors in microscopy images.",
    "compatibility" : "",
    "url" : "https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler/cellprofiler-tumor?raw",
    "sample_input" : "",
    "sample_output" : "",
    "interfaceLevel" : 2,
    "dockerimageName" : "",
    "vmImageName" : "",
    "dockerfile" : "",
    "author" : "ruiliu@illinois.edu",
    "dependencies" : [
           "Cell Profiler"
    ],
    "transformation_type" : "extractor",
    "transformation_id" : "",
    "title" : "Tumors"
}

...