You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 26 Current »

This page is under construction.

Background

Tools Catalog is the main web application that the tool contributors of Brown Dog will interact with. Current version of Tools Catalog (0.4.0) needs to be simplified and made more user friendly, in order for that to happen. Tools Catalog 0.4.0 is written in Scala using the Play framework, which is less popular than languages like Python, JavaScript, etc. There are also many features in Tools Catalog 0.4.0, which are either not needed or not clearly understood by the users. This will be an opportunity to re-design and re-implement Tools Catalog with only the required set of functionalities, which are user friendly as well.

Name

We have this ongoing task  BD-736 - Getting issue details... STATUS  and will tentatively use the name "Brown Dog Transformations" (previously suggested by Ben) or "BD Transformations" in short as we describe Tools Catalog "2.0" on this page. For the remaining part of this document we will use BD Transformations in place of Tools Catalog 2.0. This will help clarify thoughts related to Tool vs Software.

Definitions

This section will define the main concepts that will need to be clearly understood for easily interacting with BD Transformations. An example can be Software vs Transformation or any new terminology that will be used in BD Transformations.

Converter: Program that does data format conversion.

Extractor: Program that process and extract metadata from data.

Transformation: Any converter or extractor or extractor program.

Software: Software tool / library that is used by an extractor or converter to do data transformation.

converters_info.json: A JSON file that describes a converter. 

extractors_info.json: A JSON file that describes an extractor.

Minimum Viable Product (MVP): Version of BD Transformations with only a few basic features. This will be developed through some kind of a hackathon.

User Roles

The user roles in BD Transformations will be:

  1. Administrator
    1. Update / Delete transformations created by contributors
    2. Update / Delete software added by contributors?
    3. All Viewer permissions
  2. Contributor
    1. Create new transformation and link it to one or more existing software.
    2. Add a new software to BD Transformations
    3. Update created transformations [adding the word "created" to make it clear that a contributor can only manage the transformations they contributed]
    4. Delete created transformations
    5. View created transformations
    6. All Viewer permissions
  3. Viewer (can be any user who does not login)
    1. View all transformations
    2. View all software

Features

  • All users can view all transformations that are linked to a software.
  • All users can view a single transformation and the software that it uses.
  • Contributors will be able to add a transformation through the BD Transformations interface.
    • This will be made available through a form that the contributor can fill and submit.
    • Ability to include icon
    • Ability to include link to software web page (e.g. ImageJ)
    • Ability to add a URL to the transformation  - source code / docker image name / website
  • Contributors will also be able to import a new transformation into the catalog by uploading an extractors_info.json or converters_info.json file.
    • Any missing piece of information can be filled in by the contributor.
  • Contributors will be able to link their transformation with one more existing software in BD Transformations. They will be able to add a new software if it doesn't already exist in BD Transformations.
  • There won't be any formal approval process for the submitted transformations. All transformations that get submitted will be available to the public.
  • Administrators can update or delete any transformation as they feel appropriate.

Wireframes

Exported image for the selected wireframe not found. Did someone delete it?

  1. Home (Yan)

  2. View Transformation (Sandeep)

  3. Login (Bing)

  4. POST Endpoint

  5. Administrator

  6. Contributor (Low priority for now)

Technologies 

  • Flask Templates
  • Bootstrap
  • JQuery
  • Flask Services
  • MongoDB
  • Python LDAP Library
  • Python MongoDB Library

Tasks

  • Script for exporting data from Tools Catalog to BD Transformations
  • Code to talk to NCSA LDAP for authentication
  • Clusterman endpoint to read from BD Transformations database
  • Flask Services
    • POST <extractors/converters>_info.json to MongoDB
    • GET <extractors/converters>_info.json for a specific transformation
    • GET all transformations
    • GET all transformations that uses a specific software

JSON Document Example

Mapping Table

Note: Changing to camel case notation for field names as MongoDB documents follow JSON.

Sl. No.Tools Catalog FieldTransformations Catalog FieldTypeRemarks
1.
status
N/AInteger
2.
creationDate
dateCreated
Date
3.
updateDate
dateUpdated
Date
4.
downloads
N/AInteger
5.
license
N/AString
6.
whatsnew
description
String
7.
compatibility
N/AString
8.
url
url
StringURL
9.
sample_input
N/AStringURL
10.
sample_output
N/AStringURL
11.
interfaceLevel
N/AInteger
12.
dockerimageName
dockerImageName
String
13.
vmImageName
N/AString
14.
dockerfile
N/AStringEmail ID
15.
author
author
String
16.
dependencies
dependencies
ArrayArray of Strings
17.
transformation_type
transformationType
StringOriginally this field type was an integrer with values 1 for converter and 2 for extractor
18.
transformation_id
transformationId
StringExtractor name / converter name
19.
title
title
String

Transformations Catalog Example

{
    "_id" : ObjectId("53864aee7400009900ec8d22"),
    "dateCreated" : ISODate("2014-05-28T20:45:34.412Z"),
    "dateUpdated" : ISODate("2016-11-21T04:09:47.836Z"),
    "description" : "Identify tumors in microscopy images.",
    "url" : "https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler/cellprofiler-tumor?raw",
    "dockerImageName" : "",
    "author" : "ruiliu@illinois.edu",
    "dependencies" : [
           "Cell Profiler"
    ],
    "externalServices" : [
    ],
    "transformationType" : "extractor",
    "transformationId" : "",
    "title" : "Tumors",
    "status" : "submitted"
}

Extractor Info Example

{
    "@context": "http://clowder.ncsa.illinois.edu/contexts/extractors.jsonld",
    "name": "ncsa.csvheaders",
    "version": "1.0",
    "description": "Extracts the headers of CSV file. Only looks at the first row.",
    "author": "Yan Zhao <yanzhao3@illinois.edu>",
    "contributors": ["Luigi Marini <lmarini@illinois.edu>"],
    "contexts": [
        {
            "csv_headers": "http://clowder.ncsa.illinois.edu/metadata/ncsa.csvheaders#csv_headers"
        }
    ],
    "repository": [
        {
            "repType": "git",
            "repUrl": "https://opensource.ncsa.illinois.edu/bitbucket/scm/iarp/extractors.git"
        },
        {
            "repType": "docker",
            "repUrl": "clowder/extractors-csvheaders"
        }
    ],
    "process": {
        "file": [
          "text/csv"
        ]
    },
    "external_services":[],
    "dependencies": [],
    "bibtex": []
}

Current Tools Catalog Example

{
    "_id" : ObjectId("53864aee7400009900ec8d22"),
    "status" : 4,
    "creationDate" : ISODate("2014-05-28T20:45:34.412Z"),
    "updateDate" : ISODate("2016-11-21T04:09:47.836Z"),
    "downloads" : 2,
    "license" : "UIUC/NCSA Open Source License.\r\nhttp://otm.illinois.edu/uiuc_openSource",
    "whatsnew" : "Identify tumors in microscopy images.",
    "compatibility" : "",
    "url" : "https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler/cellprofiler-tumor?raw",
    "sample_input" : "",
    "sample_output" : "",
    "interfaceLevel" : 2,
    "dockerimageName" : "",
    "vmImageName" : "",
    "dockerfile" : "",
    "author" : "ruiliu@illinois.edu",
    "dependencies" : [
           "Cell Profiler"
    ],
    "transformation_type" : "extractor",
    "transformation_id" : "",
    "title" : "Tumors"
}


Converter Info Example

{
    "name": "ImageMagick",
    "version": "1.0",
    "description": "Converts images from one format to another",
    "author": "Kenton McHenry <mchenry@illinois.edu>",
    "contributors": ["Bing Zhang <bing@illinois.edu>"],
    "repository": [
        {
            "repType": "git",
            "repUrl": "opensource.ncsa.illinois.edu/bitbucket/scm/pol/converters-imagemagick.git"
        },
        {
            "repType": "docker",
            "repUrl": "ncsapolyglot/converters-imagemagick"
        }
    ],
    "input_formats": [
          "png", "bmp"
        ],
    "output_formats": [
          "jpeg", "jpg", "pdf"
        ],
    "external_services":[],
    "dependencies": ["ImageMagick"],
    "bibtex": []
}


  • No labels