Page tree
Skip to end of metadata
Go to start of metadata

Code: Is there code?  Is it submitted or existing? If so add URL to parent wiki page!!!
Py1/Py2: Is it using PyClowder1, PyClowder 2, or neither.  Expectation for Beta is most will be in Py1, moving forward with new ones we will use Py2.
JSON-LD: Is proper JSON-LD generated in the MetaData and registration document (make sure to use full URI).
DCKR: Dockerized?
Test File: Sample file available?

TC: Is the tool in the tools catalog?
DPL: Is the tool dockerized (if not Windows), managed by the elasticity service, and passing tests every other hour?
DL: Is the tool downloadable via Sandeep's modifications to move compute to the data?

Complete & Deployed
Needs work
Blocked

Scientific

Domain  ToolType

Description                                     

CodePy1/Py2JSON-LDDCKRTest FileTCDPLDLAssignment
Hydrology




 

 








1Advection Diffusion Solve a general advection-dispersion equation.

No

(Don't have a general tool. Seems like this was not done as per the description.)

     

 

 

Sandeep Puthanveetil Satheesan

Per Dr K this will be post beta

2Chemical Mean AgeExtractorDetermine the mean age of chemical constituents with inputs of chemical dynamics.

 No

(This is a simulation tool - don't have code.)

 

 

 

 

 

 

 

Sandeep Puthanveetil Satheesan

Per Dr K this will be post beta

3Document Tables Extractor (ncsa.nlp.wordtables)ExtractorExtract tables from documents.

Yes 

No

No

No

No 

No 

No 

No 


4ESRI Shapefile Extractor by using GDAL (ncsa.geoshp.preview)ExtractorCreate a Geoserver layer for WMS service. It can be used in previewer

Yes

Py1

Yes

Yes

 Yes

 No

Yes

 No

5GeoTiff Extractor by using GDAL (ncsa.geotiff.preview)ExtractorCreate a Geoserver layer for WMS service. It can be used in previewer

Yes

Py1YesYes

 Yes

 No

Yes

 No

6GeoTiff Metadata Extractor by using GDAL (Another implementation) (ncsa.geotiff.metadata)ExtractorExtract geospatial metadata from Geotiff. It should be combined with #5 Geotiff extractor.

 Yes

Py1

Yes

Yes

 Yes

 Yes

Yes

 No

7Historical River Extractor (ncsa.cv.river)ExtractorExtract the river networks from the ancient hand-drawing maps and compare them with current river networks

 Yes

Py1

 Yes

Yes

 Yes

 Yes

 Yes

No

8Normalized Difference Vegetation Index (ncsa.arcgis.landsat7mosaic)Extractor

Create a NDVI layer from Landsat data. Calculating vegetation indices such as NDVI and Surface Temperature from Landsat 7 and 8 satellite data.

 Yes

Py1

 Yes

N/A

(Windows,

ArcGIS)

 Yes

 No

 Yes

(Windows VM)

 No

9River Chi Index (ncsa.chi_analysis)ExtractorIdentify the river dynamics in a river basin and evaluate human activities' influences through Chi index in the streams.

 Yes

Py1

 Yes

Yes

 Yes

Yes

Yes

Yes

10GDAL ConverterConverterConverts between TIFF and BIL (zip) and FLT (zip)YesN/AN/AYesYesYesYesYes
11River SinuosityExtractorStudy the maturity and equilibrium conditions of a stream through the sinuosity index.

 Yes

 

 

 

 

 

 

 

Sandeep Puthanveetil Satheesan

Qina supposed to be delivering code - Dr K thinks it should make it in Beta

12Soil Moisture ChangeExtractorDetermine role of hydraulic redistribution in AZ (riparian site / upland site) by studying soil moisture change throughout different seasons.

 No

 

 

 

 

 

 

 

Sandeep Puthanveetil Satheesan

Per Dr. K - will need discussed and can not be in Beta

13Species ClassifierExtractorSAM based Species Classification from Hyperspectral data, Hyperspectral Indices, NDVI, SAVI, MSAVI, etc.

 Yes 

 

 

 

 

 

 

 

Sandeep Puthanveetil Satheesan

Dr K does not think this will make it in Beta

14Floodplain Extractor by using TerEx (ncsa.arcgis.floodplain)ExtractorIdentify the flat polygons and the heights inside a river valley.

Yes (but procedure?)

Py1

Yes 

No (windows)

Yes 

 No

Yes

(Windows)

 no

15Topographic DepressionsExtractorIdentify topographic depressions (TDs) and their distribution on landscape (Number, location, area, volume of TDs).

 Yes - ready to review

 

 

 

 

 

 

 

16

Tree Delineation

(ncsa.arcgis.treedelin)

ExtractorCreate a shape-file with polygons of tree canopy polygon from LiDAR data

 Yes

Py1

 Yes

N/A

(Windows,

ArcGIS)

Yes

 Yes

 Yes

(Windows VM)

 

No
17Tree-wise voxelization Tree-wise voxelization of waveform data for lidar metrics that describes canopy structure (max intensity, height, etc...).  Individual tree delineation, tree leaf area density to describe vertical leaf distribution.

Yes

(But extractor has to be developed)

No

No

No

No 

No 

No 

No 

18Valley Safety ZonesExtractorEstimate submerging areas and water depths under extreme floods and map the safety zones in a river valley.

 Yes  

 

 

 

 

 

 

 

Sandeep Puthanveetil Satheesan

emailed March and April.

Sandeep will follow up with Qina.

Ecology

 






1netcdf (ncdump)ConverterConvert from binary netcdf to text.N/AN/A

 N/A

YesYes

Yes

Yes

Yes
2PEcAn (PEcAn#Ameriflux)ConverterConvert Ameriflux data to PEcAn's netcdf CF format.

Yes

N/A

 N/A


YesYes

Yes

Yes

Yes
3PEcAn (PEcAn#DALEC)ConverterConvert PEcAn's netcdf CF format to the format required by the DALEC model.

Yes

N/A

  N/A

YesYes

 Yes

Yes

Yes
4PEcAn (PEcAn#ED2)ConverterConvert PEcAn's netcdf CF format to the format required by the ED model.

 Yes

N/A

  N/A

YesYes

 Yes

Yes

Yes
5PEcAn (PEcAn#LINKAGES)ConverterConvert PEcAn's netcdf CF format to the format required by the LINKAGES model.

 Yes

N/A

 N/A

YesYes

 Yes

Yes

Yes
6PEcAn (PEcAn#Sipnet)ConverterConvert PEcAn's netcdf CF format to the format required by the Sipnet model.

 Yes

N/A

  N/A

YesYes

 Yes

 Yes

Yes
7

PEcAn (PEcAn#AmerifluxBNL)

PEcAn (PEcAn#FLUXNET2015)

PEcAn (PEcAn#FACE)

PEcAn (PEcAn#PALEON)

PEcAn (PEcAn#NLDAS)

PEcAn (PEcAn#CURNCEP)

PEcAn (PEcAn#GLDAS)

PEcAn (PEcAn#GFDL)

PEcAn (PEcAn#BIOCRO)

PEcAn (PEcAn#CLM)

PEcAn (PEcAn#GDAY)

PEcAn (PEcAn#JULES)

PEcAn (PEcAn#LPJ-GUESS)

PEcAn (PEcAn#MAAT)

PEcAn (PEcAn#MAESPA)

PEcAn (PEcAn#PRELES)

  

 Yes

N/A

 N/A


NoYes

 Yes

Yes

No
8PlantCV (terra.plantcv)ExtractorExtract plant height, area, and color distribution from photographs.

 Yes

Py1

 yes

yes

yes 

yes 

no

(bamboo test available, not being deployed)

 no

9BIL (hyperspectral)ConverterConvert bil.zip (contains raw & *.hdr) to *.terra.nc

Yes

(The original code has been updated. Need to figure out if we want to update this)

N/AN/Ayesno, only test files available are very large and would take a long timeyes

no

(bamboo test available, not being deployed)

no
Civil & Environmental Engineering



 

1

Body of Water Detector (ncsa.image.ponddetect)

ExtractorLand coverage, extract locations of bodies of water from satellite data.

Yes

No

 No

No

 Yes

 No

No 

 No

??who??
2GI IdentificationExtractor 

Yes

Py1

 Yes

Yes

 Yes

 Yes

 Yes

No

3Human Preference Score (ncsa.image.humanpref)ExtractorAssign a model derived human preference score to a given image of an urban environment.

 Yes

no

 no

no

 yes

 no

 yes

 no

??who??

4

Route Greenness (ncsa.greenindex)

ExtractorDerive the green index of a city route.

 Yes

Py1

 Yes

yes

 yes

 Yes

 Yes

 No

5Social Media GI PreferencesExtractorDetermine if text contains references to visual or functional green infrastructure.

 Yes

Py1

 Yes

Yes

Yes 

No 

 no

 No

6Stanford CoreNLP - Sentiment (ncsa.nlp.SNLPSentiment)ExtractorAssign a sentiment score to a piece of text.

 Yes

Py2 Java equiv.

Yes

YesYes

 

Yes

 

No

 

No

 

7TUV TriaxusExtractor

Towed Undulating Vehicle Data Analyzing Tools

 Yes

Py1

 yes

yes

yes

 yes

 yes

No 

Social Science & Humanities













1Bertillon Card Cell Extractor (ncsa.image.dmp)ExtractorExtract table cells from a Bertillon Card.

 Yes

Py1

Yes

No

 No

No 

No

 No

2Census Form Cell Extractor (census-section-segmentor) ExtractorExtract table cells from a 1930s Census form.

 Yes

No

 No

No

 Yes

No 

 No

 No

3Handwritten Decimals Extractor (ncsa.image.sphog.debod)ExtractorExtract handwritten decimal values from an image.

 Yes

Py1

 Yes

No

 No

 No

No 

No

4

Killed Photos

(ncsa.image.killedphoto)

ExtractorIdentify depression era photos "killed" by Farm Security Administration director Roy Stryker, indicated by a hole punch in the image.

 Yes

Py2

Yes

Yes

Yes

Yes

no 

 no

5Mean Grey (ncsa.cv.meangrey)ExtractorMean grey values of black and white photos.Yes

 Py1

 Yes

YesYes

Yes 

Yes

 no

6Movie Slice (ncsa.movieslice)ExtractorGenerates movie slice visualization from video files

 Yes

Py1/Py2 not used

No

No

 No

 No

No 

No 

7Person Detector (ncsa.video.person_detector)ExtractorExtract locations of people in an image.

 Yes

Py2

Yes

Yes

(MATLAB external)

 Yes

 Yes

Yes 

No 

8Person Tracker (ncsa.video.person_tracking)ExtractorExtract locations and paths of people moving in videos.

 Yes

Py2

Yes

Yes

(MATLAB external)

 Yes

Yes 

 Yes

No 

9Video Analytics Toolbox (ncsa.cinemetrics_batch)ExtractorExtract shot descriptors from videos.

 Yes

Py1/Py2 not used

No

No

 No

No 

 No

No 

Biology, Genomics, Medicine1FSL (mri2mesh)ExtractorCreates a finite element mesh from a set of mr images

 Yes

Py2

 no

yes

 yes

 no

 no

 no

2Glomeruli (ncsa.msc.diagnosis)ExtractorExtract glomeruli from Kidney biopsy images.

 Yes

Py1

Yes

Yes

Yes

Yes

no

no

General

Domain 

Tool

TypeDescription                                     CodePy1/Py2JSON-LDDCKRTest FileTCDPLDLAssignment

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

General

1Calibre (ebook-converter)ConverterConvert e-books to a number of document formats.YesN/AN/AYesYes

Yes

No

Yes
2

CMU Sphinx (ncsa.audio.speech2text)

ExtractorAudio recognition to extract text for speech within audio.YesN/AYesYesYes

 Yes

No 

No 

??who??

3Daffodil (daffodil)ConverterConvert formats with a provided DFDL schema to XML.Yes    

 Yes

 Yes

Yes
4DBPedia (ncsa.dbpedia)ExtractorFind and define named entities within the given text.

 Yes

Py1YesYesYes

Yes

 Yes

 

5FFmpeg (ffmpeg)ConverterConvert between a large number of video formats.YesN/AN/AYesYes

 Yes

No

Yes
6FLAC (flac)ConverterConvert to and from the FLAC format from other audio formats.YesN/AN/AYesYes

 Yes

No

YesBing Zhang
7Ghostscript (ghostscript)ConverterConvert between document formats.YesN/AN/AYesYes

 Yes

No

YesBing Zhang
8htmldoc (htmldoc)ConverterConvert HTML to a number of document formats.YesN/AN/AYesYes

 Yes

No

Yes
9ImageMagick (ImageMagick)ConverterConvert between a large number of image formats.YesN/AN/AYesYes

 Yes

 No

Yes


10ImageMagick (ncsa.image.metadata)ExtractorPull available EXIF image metadata from a given image.

 Yes

Py2YesYesYes

Yes

 Yes

 Yes


11Kabeja (kabeja)ConverterConvert between a handful of 3D and image formats.YesN/AN/AYesYes

 Yes

 Yes

Yes
12

OpenCV - Faces (ncsa.cv.faces)

ExtractorFind faces in an image and return their locations.

 Yes

Py1YesYesYes

 Yes

 Yes

 Yes

13

OpenCV - Eyes (ncsa.cv.eyes)

ExtractorFind eyes in an image and return their locations.

 Yes

Py1YesYesYes

 Yes

Yes

 Yes

14

OpenCV - Closeups (ncsa.cv.closeups)

ExtractorDetermine whether an image is a closeup of a person or not.

 Yes

Py1YesYesYes

 Yes

 Yes

 Yes

15

OpenCV - Profiles (ncsa.cv.profiles)

ExtractorFind human face profiles in an image and return their locations.

 Yes

Py1YesYesYes

Yes

 Yes

 Yes

16Langid (ncsa.nlp.simplelanguage)ExtractorIdentify the language of the given text.

 Yes

Py2YesYesYes

 Yes

 No

No

17LibreOffice (unoconv)ConverterConvert to and from a variety of document formats.YesN/AN/AYesYes

 Yes

No

YesBing Zhang
18NLTK - Summary (ncsa.nlp.simplesummary)ExtractorSummarize a body of text.

 Yes

Py2Yes

Yes

Yes

 Yes

No

 No

19Siegfried (siegfried)ExtractorExtract information about a given file relevant to identifying its type and validating its format.

 Yes

Py2YesYesYes

 Yes

No

 No

20Stanford CoreNLP (ncsa.nlp.SNLP)ExtractorNatural Language Process extractions such as parts of speech, named entities, langauge, etc.

 Yes

Py2 Java equiv.YesYesYes

 Yes

 No

 No

21Tesseract (ncsa.image.ocr)ExtractorObject Character Recognition (OCR) to extract text from images containing text.

 Yes

Py1YesYesYes

 Yes

 Yes

Yes 

22Tika (ncsa.nlp.tika)ExtractorDocument extractions such as language identification, ...

 Yes

older Py1YesYesYesYes

 

No

 

Yes

 

23txt2html (txt2html)ConverterConvert text documents to HTML.YesN/AN/AYesYes

Yes

Yes

Yes
24Versus - Color Distribution (ncsa.versus.image)ExtractorGenerate a distribution of color values within an image to be used for comparing how similar two images are.

 Yes

Py1YesYesYes

 Yes

Yes

 Yes

25

VLFeat (ncsa.image.caltech101)

ExtractorClassify images as to whether they contain objects from the Caltech101 dataset (e.g. people, airplanes, motorcycles, cougars, ...).

 Yes

Py2yesyesyes

 Yes

 yes

no

26Zip (zip)ConverterUnzip zip archives.YesN/AN/AYesYes

 Yes

No

YesBing Zhang

https://opensource.ncsa.illinois.edu/confluence/display/BD/Transformations (Under development, Being Refactored)

GeoTiff Extractor by using

  • No labels

4 Comments

  1. Kenton McHenry I believe that in the case of the Ecology section all of the code is now in a single convertor that supports all models and met processes, including those added in the future. See https://github.com/PecanProject/pecan/blob/master/contrib/browndog/PEcAn.R

  2. Identify items and prioritize the ones that just have to be dockerized or that just need tweaking of metadata JSON-LD info - and confirm they can register with clowder

  3. To test your extractor:

    docker run -rm -t -i -e "RABBITMQ_URI=amqp://user:password@rabbitmq.ncsa.illinois.edu/clowder-test" -e "RABBITMQ_EXCHANGE=dts" -e "REGISTRATION_ENDPOINTS=http://browndog.ncsa.illinois.edu/clowder-dev/extractors?key=commKey"

    Now connect to http://browndog.ncsa.illinois.edu/clowder-dev upload your testfile and check the result of the rextractor 

  4. Next week
    YAML - testing - need an input and output file
    everyone should get in the habit of adding each to the YAML file when the tool goes green