Page History

This page is for the refactoring of the existing extractors. The original wiki page Hosted VMs is still used for the deployments. 

As we figure out who's working on what, please start with the following steps for the extractor(s) you chose:

• be able to run the extractor
• add a README, specifically a readme.md (i.e. in markdown), with information on how to install dependencies and run the extractor (in its current shape)
• start looking at dbpedia extractor for template
• Learn about jsonld by playing in the playground here http://json-ld.org/
• Go through the README for the docker extractors template: https://opensource.ncsa.illinois.edu/bitbucket/projects/BD/repos/bd-extractor-template/browse

Steps to take for every extractor in this list:

1. Docker containers
2. JSONLD
3. Extractor info registration
4. Use pyclowder (for python extractors)
5. Add status messages to all extractors and fix level granularity
   1. Make status constants (DONE, ERROR)
   2. Arcgis multiprocessing extractor
6. Register on on demand execution queues
   1. Add on demand key binding to configuration file: messageType = "*.file.text.plain", "extractors."+extractorName
7. Standardize around python logging
   1. Figure out what to log and what format to follow
8. Add logstash to docker compose
9. Add sample input/ouput to git repository
10. Add icon for tools catalog to git repository
11. Add entry to Tools catalog, with icon

ID (Extractor Name from config file,

same as queue name)

Programming

Language

SoftwareOSAssigned ToLink to repoWho wrote or worked on the codeDEPLOYED      ncsa.image.ocrPythonTesseractLinuxRuiocr 

ncsa.cv.faces

PythonOpenCVLinuxInna (may be?)

ID (Extractor Name from config file, same as queue name)	Programming Language	Software	OS	Can be Dockerized?	Assigned To	Repo	Author
DEPLOYED
ncsa.image.ocr	Python	Tesseract	Linux		Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/ocr
ncsa.cv.faces	Python	OpenCV	Linux		Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencv	Liana
ncsa.cv.eyes	Python	OpenCV	Linux		Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencv	Liana
ncsa.cv.closeups	Python	OpenCV	Linux		Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencv	Liana
ncsa.cv.profiles	Python	OpenCV	Linux		Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencv	Liana
ncsa.cellprofiler.fluorescentcomet	Python	pymedici	Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.fly	Python		Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.human	Python		Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.silvercomet	Python		Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.speckle	Python		Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.trackobject	Python		Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.tumor	Python		Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.yeast	Python		Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.image.sphog	Python	Matlab, mnist-sphog	Linux		Gregory Jansen	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/handwritten/HandwrittenNumbers
ncsa.image.caltech101					Gregory Jansen
ncsa.bisque.histogram (notes: disabled)	Python		Linux
ncsa.bisque.metadata (notes: disabled)	Python		Linux
census-section-segmentor	Java		Linux		Sandeep Puthanveetil Satheesan	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/census	Liana, Inna
ncsa.cv.river	Python	OpenCV (python), convert (from imagemagick), and Gdal	Linux		Smruti Padhy	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/river	Liana
ncsa.geo.shpExtractor	Python	gdal	Linux		Jong Lee	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-geo/browse	Jong Lee
ncsa.geo.tiffExtractor	Python	gdal	Linux		Jong Lee	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-geo/browse	Jong Lee
ncsa.image.geotiff	Python	GDAL, Cython, numpy, pygeoprocessing	Linux		Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-geotiff/browse	Rui, Mostafa Elag
ncsa.image.ponddetect	Python	Matlab	Linux		Marcus Slavenas	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-maps/browse/feature_detection	Marcus, Ankit
ncsa.image.humanpref	Python	Matlab	Linux		Marcus Slavenas	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-maps/browse/humanpref	Marcus, Ankit
ncsa.xml.greenindexroute, ncsa.csv.greenindexroute	Python	OpenCV	Linux		Marcus Slavenas	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-maps/browse/greenroute	Marcus
ncsa.image.knn_numerals	Python	OpenCV	Linux		Marcus Slavenas		Marcus
ncsa.audio.speech2text	Java	CMU Sphinx, ffmpeg, sox	Linux		Marcus Slavenas	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/audio/speech2text	Marcus
ncsa.audio.preview	Python				Inna	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/audio/preview
ncsa.nlp.simplelanguage	Python	numpy			Inna	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/SimpleLanguage	Liana
ncsa.nlp.simplesummary	Python	Natural Language Toolkit (NLTK) for Python, NLTK Data or at least:  nltk.corpus,nltk.stem.porter and nltk.tokenize.punkt.			Gregory Jansen	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/SimpleSummary	Liana
ncsa.nlp.SNLPSentiment	Java	Stanford CoreNLP tool, java, maven			Marcus Slavenas	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/SNLP/SNLPSentimentExtractor	Liana, Marcus(?)
ncsa.nlp.wordtables	Python	requests, pika, win32com				https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/WordTablesExtractor	Liana
siegfried	Python					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-siegfried/browse	Gregory Jansen
ncsa.versus.image	Java	Versus	Linux		Smruti Padhy	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-versus/browse	Kenton, Smruti
ncsa.image.preview (note: check if really deployed. there is an extractor in Hosted VMs list with a similar name.)	Python					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/image/preview	Rob, Sandeep
ncsa.pdf.preview (note: check if really deployed. there is an extractor in Hosted VMs list with a similar name.)	Python					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/pdf/preview	Rob
ncsa.video.preview (note: check if really deployed. there is an extractor in Hosted VMs list with a similar name.)	Python					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/video/preview	Rob
NOT DEPLOYED
ncsa.image.digitpy	Python	opencv				https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/handwritten/SimpleDigitPython

...

ncsa.cv.pdfimages		pdfimages, from poppler-utils				https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/poppler
ncsa.cv.caltech101	Python	Matlab and VLFeat	64-bit Mac OS			https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/vlfeat
dbpedia	Python	Natural Language Toolkit (NLTK) and rdflib.			Luigi Marini

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

dbpedia/browse

/opencvLiana

ncsa.cv.eyes

PythonOpenCVLinuxInna

	Luigi Marini
digest	Python					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

digest/browse

/opencvLiana

ncsa.

cv.closeups

hpc	Python

OpenCVLinuxInna

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

hpc/browse

/opencvLiana

	Sandeep Puthanveetil Satheesan
LSVA	Java

ncsa.cv.profiles

PythonOpenCVLinuxInna

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

lsva/browse

/opencv

Liana

ncsa.cellprofiler.fluorescentcomet

Pythonpymedici

, Constantinos
LSVA integrated

Windows

Sandeep Puthanveetil Satheesan

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-lsva-

cv

integrated/browse

/cellprofilerLiana

Sandeep Puthanveetil Satheesan
ncsa.

cellprofiler.fly

movieslice	Python

Windows

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

movieslice/browse

/cellprofilerLiana

	Sandeep
mri2mesh	Python	pymedici, subprocess, logging, os, numpy, shutil, zipfile

ncsa.cellprofiler.human

Python Windows

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

mri/browse/

cellprofiler

mri2mesh

Lianancsa.cellprofiler.silvercomet

Marcus
msc-ChemCBCExtractor	Python	requests,

pika, openpyxl, xlrd, pymongo

Linux

Windows

Yan Zhao

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

msc/browse/

cellprofilerLiana

ncsa.cellprofiler.speckle

Python Windows

ChemCBCExtractor	Yan
msc-IsletExtractor	Python	requests, pika, openpyxl, xlrd, pymongo	Linux		Yan Zhao

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

msc/browse/

cellprofiler

IsletExtractor

Lianancsa.cellprofiler.trackobject

Yan
msc-MonitorExtractor	Python	requests, pika, openpyxl, xlrd, pymongo	Linux

Windows

Yan Zhao

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

msc/browse/

cellprofiler

MonitorExtractor

Liana

Yan
ncsa.

cellprofiler

msc.

tumor

dailymonitor	Python	requests, pika, openpyxl, xlrd, pymongo

Windows 

not used

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

msc/browse/

cellprofiler

OldMonitorExtractor

Lianancsa.cellprofiler.yeast

Ashwini
msc-PhenotypeExtractor	Python	requests, pika, openpyxl, xlrd, pymongo	Linux

Windows

Yan Zhao

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

msc/browse/

cellprofiler

PhenotypeExtractor

Liana

Yan
ncsa.

image

nlp.

sphog

SNLP

Python

Java

Matlab, mnist-sphog

Stanford CoreNLP tool, java, maven

Linux

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

nlp/browse/

handwritten

SNLP/

HandwrittenNumbers

SNLPExtractor

 

Liana
ncsa.

image.caltech101      ncsa.bisque.histogram (notes: disabled)Python Linux   ncsa.bisque.metadata (notes: disabled)Linux

nlp.tika	Python

Tika project page, pymedici

 

Kenton McHenry	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/tika	Liana
person-detector	Python	MATLAB, FFMPEG, requests and pika

census-section-segmentorJava Linux

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-person-

cv

detector/browse/

census

python

Liana, Inna

Sandeep
ncsa.

cv.river PythonOpenCV (python), convert (from imagemagick), and Gdal

person-tracker	Python	python, MATLAB, FFMPEG requests and pika

Linux

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

person-tracking/browse/

river

python

Liana

Sandeep

ncsa

terra.

geo.shpExtractor

plantcv	Python

gdal

pika requests wheel

Linux

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

geo

plantcv/browse

Jong Lee

	Yan
medici_PTM_thumbnails	Java

ncsa.geo.tiffExtractorPythongdalLinux

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

geo

ptm/browse

Jong Lee

ncsa.image.ponddetect

PythonMatlabLinux

/PTMThumbnailExtractor	Constantinos
medici_PTM_metadata	Java

Marcus

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

maps

ptm/browse

/feature_detectionMarcus, Ankit

/PTMMetadataExtractor	Constantinos
Name not clear PtmMetadata(?)	Java

ncsa.image.humanprefPythonMatlabLinuxMarcus

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

maps

ptm/browse/

humanprefMarcus, Ankit

ncsa.xml.greenindexroute, ncsa.csv.greenindexroute

PythonOpenCVLinux

PTMMetadata	Constantinos
medici_ptm_maps	Java

Marcus

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

maps

ptm/browse/

greenroute

PTMMapsExtractor

Marcus

Constantinos
medici_ptm_3d	Java

ncsa.image.knn_numerals

PythonOpenCVLinuxMarcus Marcus

ncsa.audio.speech2text

JavaCMU Sphinx, ffmpeg, soxLinuxMarcus Marcusncsa.nlp.simplelanguagePythonnumpy

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

nlp

ptm/browse/

SimpleLanguage

PTM3DExtractor

Lianancsa.nlp.simplesummaryPython

Natural Language Toolkit (NLTK) for Python, NLTK Data or at least:

Constantinos
medici_images_ptm	Java

 nltk.corpus,nltk.stem.porter and nltk.tokenize.punkt.

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

nlp

ptm/browse/

SimpleSummaryLianancsa.nlp.SNLPSentimentJava Stanford CoreNLP tool, java, maven

ImagesPTMExtractor	Constantinos
extractors-rabbitmq (look like examples)						https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

nlp

rabbitmq/browse

/SNLP/SNLPSentimentExtractorLiana, Marcus(?)ncsa.nlp.wordtablesPython requests, pika, win32com

Name not clear extractors-seabird/	Scala					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS

/repos/extractors-nlp/browse/WordTablesExtractorLiana

/repos/extractors-seabird/browse	Luigi
medici_3d_x3d (one of extractors-3d)	Java

                     NOT DEPLOYED   ncsa.image.digitpy (notes: not in the Wiki page)Pythonopencv 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

3d/browse/

handwritten/SimpleDigitPython

ObjJSONExtractor

 

Constantinos
medici_3d_obj_merger (one of extractors-3d)	Java

ncsa.cv.pdfimages (not in the wiki page) pdfimages, from poppler-utils

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

3d/browse/

poppler ncsa.cv.caltech101PythonMatlab and VLFeat 

OBJMergerExtractor	Constantinos
medici_oni (one of extractors-3d)	Java

64-bit Mac OS 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

3d/browse/

vlfeat dbpediaPython Natural Language Toolkit (NLTK) and rdflib. 

OniExtractor	Constantinos
medici_ply_obj (one of extractors-3d)	Java

Luigi Marini

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

dbpedia

3d/browse

Luigi Marini

/PlyObjExtractor	Constantinos
medici_3d_metadata (one of extractors-3d)	Java

digestPython

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

digest

3d/browse/ThreeDMetadataExtractor

 

Constantinos
medici_x3d_html (one of extractors-3d)	Java

ncsa.image.geotiffPython

GDAL, Cython, numpy,
pygeoprocessing,
pika,
requests

LinuxRuihttps://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-geotiff/browseRui, Mostafa Elagncsa.hpcPython

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

hpc

3d/browse

 LSVAJava  

/X3DhtmlExtractor	Constantinos
ncsa.arcgis.landsat7mosaic	Python	ArcGIS	Windows	No	Smruti Padhy

 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

lsva

bd-cz/browse

 LSVA integrated   

/ndviextractor	Smruti
ncsa.arcgis.floodplain	Python	ArcGIS	Windows	No	Smruti Padhy	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

lsva

bd-

integrated

cz/browse

 

/terex_floodplain/config.py	Smruti
medici_book	Java

ncsa.movieslicePython

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

movieslice

books/browse/BookPreviewExtractor

Sandeepmri2meshPythonpymedici, subprocess, logging, os, numpy, shutil, zipfile

Theerasit Issaranon
medici_image_pyramid	Java					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

mri

books/browse/

mri2meshMarcus

ImagePreviewPyramidExtractor-shebook	Theerasit Issaranon
shebook	Java

msc-ChemCBCExtractorPython

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

msc

books/browse/

ChemCBCExtractorYanmsc-IsletExtractorPythonrequests, pika, openpyxl, xlrd, pymongo 

SheBookPreviewExtractor/src/BookPreviewExtractor   https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

msc/browse/IsletExtractorYanmsc-MonitorExtractorPythonrequests, pika, openpyxl, xlrd, pymongo

books/browse/SheBookPreviewExtractor/src/bookpreviewextractor	Theerasit Issaranon
lsva-cedd	Java					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

msc

cedd/browse

/MonitorExtractorYan

Constantinos
ncsa.

msc.dailymonitor

cinemetrics	Python

requests, pika, openpyxl, xlrd, pymongo

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

msc

cinemetrics/browse

/OldMonitorExtractorAshwini

Constantinos
ncsa.image.metadata

msc-PhenotypeExtractor

	Python

requests, pika, openpyxl, xlrd, pymongo

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

msc

core/browse/image/

PhenotypeExtractor

metadata

Yan

Max. Rob
ncsa.

nlp

debod.

SNLP

segmentor

JavaStanford CoreNLP tool, java, maven

https://opensource.ncsa.illinois.edu/bitbucket/projects/

CATS

DEBOD/repos/extractors-

nlp

cellsegmentor/browse

/SNLP/SNLPExtractorLiana

ncsa.

nlp

image.

tika

dmp

PythonTika project page, pymedici

https://opensource.ncsa.illinois.edu/bitbucket/projects/

CATS

DEBOD/repos/extractors-

nlp

debod/browse

/tikaLianaperson-detectorPython MATLAB, FFMPEG, requests and pika  

https://opensource.ncsa.illinois.edu/bitbucket/projects/

CATS

DEBOD/repos/extractors-

person-detector

dmp/browse

/pythonSandeep

ncsa

.person-trackerPythonpython, MATLAB, FFMPEG requests and pika

.image.sphog.debod						https://opensource.ncsa.illinois.edu/bitbucket/projects/

CATS

DEBOD/repos/extractors-

person-tracking

handwrittendecimals/browse

/pythonSandeepterra.plantcvPython

pika
requests
wheel

ncsa.image.iarp_remove_circle						https://opensource.ncsa.illinois.edu/bitbucket/projects/

CATS

IARP/repos/

extractors-plantcv/browseYan

image_fetcher/browse/extractors/remove_circle	Marcus
ncsa.cv.meangrey

medici_PTM_thumbnails     COnstantinos         

https://opensource.ncsa.illinois.edu/bitbucket/projects/IARP/repos/image_fetcher/browse/extractors/mean_grey

Marcus