Page History

This page is for the refactoring of the existing extractors. The original wiki page Hosted VMs is still used for the deployments. 

As we figure out who's working on what, please start with the following steps for the extractor(s) you chose:

• be able to run the extractor
• add a README, specifically a readme.md (i.e. in markdown), with information on how to install dependencies and run the extractor (in its current shape)
• start looking at dbpedia extractor for template
• Learn about jsonld by playing in the playground here http://json-ld.org/
• Go through the README for the docker extractors template: https://opensource.ncsa.illinois.edu/bitbucket/projects/BD/repos/bd-extractor-template/browse

Steps to take for every extractor in this list:

1. Docker containers
2. JSONLD
3. Extractor info registration
4. Use pyclowder (for python extractors)
5. Add status messages to all extractors and fix level granularity
   1. Make status constants (DONE, ERROR)
   2. Arcgis multiprocessing extractor
6. Register on on demand execution queues
   1. Add on demand key binding to configuration file: messageType = "*.file.text.plain", "extractors."+extractorName
7. Standardize around python logging
   1. Figure out what to log and what format to follow
8. Add logstash to docker compose
9. Add sample input/ouput to git repository
10. Add icon for tools catalog to git repository
11. Add entry to Tools catalog, with icon

ID (Extractor Name from config file,

same as queue name)

Programming

Language

SoftwareOSAssigned ToLink to repoWho wrote or worked on the codeDEPLOYED      ncsa.image.ocrPythonTesseractLinuxRuiocr 

ncsa.cv.faces

PythonOpenCVLinuxInna (may be?)

ID (Extractor Name from config file, same as queue name)	Programming Language	Software	OS	Can be Dockerized?	Assigned To	Repo	Author
DEPLOYED
ncsa.image.ocr	Python	Tesseract	Linux		Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/ocr
ncsa.cv.faces	Python	OpenCV	Linux		Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencv	Liana
ncsa.cv.eyes	Python	OpenCV	Linux		Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencv	Liana
ncsa.cv.closeups	Python	OpenCV	Linux		Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencv	Liana
ncsa.cv.profiles	Python	OpenCV	Linux		Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencv	Liana
ncsa.cellprofiler.fluorescentcomet	Python	pymedici	Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.fly	Python		Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.human	Python		Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.silvercomet	Python		Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.speckle	Python		Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.trackobject	Python		Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.tumor	Python		Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.yeast	Python		Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.image.sphog	Python	Matlab, mnist-sphog	Linux		Gregory Jansen	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/handwritten/HandwrittenNumbers
ncsa.image.caltech101					Gregory Jansen
ncsa.bisque.histogram (notes: disabled)	Python		Linux
ncsa.bisque.metadata (notes: disabled)	Python		Linux
census-section-segmentor	Java		Linux		Sandeep Puthanveetil Satheesan	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/census	Liana, Inna
ncsa.cv.river	Python	OpenCV (python), convert (from imagemagick), and Gdal	Linux		Smruti Padhy	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/river	Liana
ncsa.geo.shpExtractor	Python	gdal	Linux		Jong Lee	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-geo/browse	Jong Lee
ncsa.geo.tiffExtractor	Python	gdal	Linux		Jong Lee	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-geo/browse	Jong Lee
ncsa.image.geotiff	Python	GDAL, Cython, numpy, pygeoprocessing	Linux		Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-geotiff/browse	Rui, Mostafa Elag
ncsa.image.ponddetect	Python	Matlab	Linux		Marcus Slavenas	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-maps/browse/feature_detection	Marcus, Ankit
ncsa.image.humanpref	Python	Matlab	Linux		Marcus Slavenas	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-maps/browse/humanpref	Marcus, Ankit
ncsa.xml.greenindexroute, ncsa.csv.greenindexroute	Python	OpenCV	Linux		Marcus Slavenas	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-maps/browse/greenroute	Marcus
ncsa.image.knn_numerals	Python	OpenCV	Linux		Marcus Slavenas		Marcus
ncsa.audio.speech2text	Java	CMU Sphinx, ffmpeg, sox	Linux		Marcus Slavenas	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/audio/speech2text	Marcus
ncsa.audio.preview	Python				Inna	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/audio/preview
ncsa.nlp.simplelanguage	Python	numpy			Inna	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/SimpleLanguage	Liana
ncsa.nlp.simplesummary	Python	Natural Language Toolkit (NLTK) for Python, NLTK Data or at least:  nltk.corpus,nltk.stem.porter and nltk.tokenize.punkt.			Gregory Jansen	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/SimpleSummary	Liana
ncsa.nlp.SNLPSentiment	Java	Stanford CoreNLP tool, java, maven			Marcus Slavenas	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/SNLP/SNLPSentimentExtractor	Liana, Marcus(?)
ncsa.nlp.wordtables	Python	requests, pika, win32com				https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/WordTablesExtractor	Liana
siegfried	Python					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-siegfried/browse	Gregory Jansen
ncsa.versus.image	Java	Versus	Linux		Smruti Padhy	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-versus/browse	Kenton, Smruti
ncsa.image.preview (note: check if really deployed. there is an extractor in Hosted VMs list with a similar name.)	Python					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/image/preview	Rob, Sandeep
ncsa.pdf.preview (note: check if really deployed. there is an extractor in Hosted VMs list with a similar name.)	Python					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/pdf/preview	Rob
ncsa.video.preview (note: check if really deployed. there is an extractor in Hosted VMs list with a similar name.)	Python					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/video/preview	Rob
NOT DEPLOYED
ncsa.image.digitpy	Python	opencv				https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/handwritten/SimpleDigitPython
ncsa.cv.pdfimages		pdfimages, from poppler-utils				https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/poppler
ncsa.cv.caltech101	Python	Matlab and VLFeat	64-bit Mac OS			https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/vlfeat
dbpedia	Python	Natural Language Toolkit (NLTK) and rdflib.			Luigi Marini

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

dbpedia/browse

/opencvLiana

Luigi Marini
digest

ncsa.cv.eyes

	Python

OpenCVLinuxInna

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

digest/browse

/opencvLiana

ncsa.

cv.closeups

hpc	Python

OpenCVLinuxInna

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

hpc/browse

/opencvLiana

	Sandeep Puthanveetil Satheesan
LSVA	Java

ncsa.cv.profiles

PythonOpenCVLinuxInna

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

lsva/browse

/opencv

Liana

ncsa.cellprofiler.fluorescentcomet

Pythonpymedici

, Constantinos
LSVA integrated

Windows

Sandeep Puthanveetil Satheesan

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

lsva-integrated/browse

/cellprofilerLiana

Sandeep Puthanveetil Satheesan
ncsa.

cellprofiler.fly

movieslice	Python

Windows

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

movieslice/browse

/cellprofilerLiana

Sandeep
mri2mesh	Python	pymedici, subprocess, logging, os, numpy, shutil, zipfile

ncsa.cellprofiler.human

PythonWindows

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

mri/browse/

cellprofiler

mri2mesh

Lianancsa.cellprofiler.silvercomet

Marcus
msc-ChemCBCExtractor	Python	requests, pika, openpyxl, xlrd, pymongo

Windows

Linux

Yan Zhao

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

msc/browse/

cellprofiler

ncsa.cellprofiler.speckle

Python Windows

ChemCBCExtractor

Liana

Yan
msc-IsletExtractor	Python	requests, pika, openpyxl, xlrd, pymongo	Linux		Yan Zhao

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

msc/browse/

cellprofilerLiana

ncsa.cellprofiler.trackobject

Python Windows

IsletExtractor	Yan
msc-MonitorExtractor	Python	requests, pika, openpyxl, xlrd, pymongo	Linux		Yan Zhao

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

msc/browse/

cellprofiler

MonitorExtractor

Liana

Yan
ncsa.

cellprofiler

msc.

tumor

dailymonitor	Python	requests, pika, openpyxl, xlrd, pymongo

Windows 

not used

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

msc/browse/

cellprofilerLiana

ncsa.cellprofiler.yeast

Python Windows

OldMonitorExtractor	Ashwini
msc-PhenotypeExtractor	Python	requests, pika, openpyxl, xlrd, pymongo	Linux		Yan Zhao

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

msc/browse/

cellprofiler

PhenotypeExtractor

Liana

Yan
ncsa.

image

nlp.

sphog

SNLP

Python

Java

Matlab, mnist-sphog

Stanford CoreNLP tool, java, maven

Linux

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

nlp/browse/

handwritten/HandwrittenNumbers 

SNLP/SNLPExtractor

Liana

• ncsa.image.caltech101

      

ncsa.

bisque.histogram (notes: disabled)Linux

nlp.tika	Python

Tika project page, pymedici

 Python Linux

Kenton McHenry

https://opensource.ncsa.

bisque.metadata (notes: disabled)

illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/tika	Liana
person-detector	Python	MATLAB, FFMPEG, requests and pika

census-section-segmentorJava 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-person-detector/browse/python	Sandeep
ncsa.person-tracker	Python	python, MATLAB, FFMPEG requests and pika

Linux

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-person-

cv

tracking/browse/

census

python

Liana, Inna

Sandeep

ncsa.cv.river PythonOpenCV (python), convert (from imagemagick), and GdalLinux

terra.plantcv	Python	pika requests wheel				https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

plantcv/browse

/riverLiana

	Yan
medici_PTM_thumbnails	Java

ncsa.geo.shpExtractorPythongdalLinux

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

geo

ptm/browse

Jong Lee

/PTMThumbnailExtractor	Constantinos
medici_PTM_metadata	Java

ncsa.geo.tiffExtractorPythongdalLinux

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

geo

ptm/browse

Jong Lee

ncsa.image.ponddetect

PythonMatlabLinux

/PTMMetadataExtractor	Constantinos
Name not clear PtmMetadata(?)	Java

Marcus

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

maps

ptm/browse/

feature_detectionMarcus, Ankitncsa.image.humanprefPythonMatlabLinuxMarcus

PTMMetadata	Constantinos
medici_ptm_maps	Java

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

maps

ptm/browse/

humanpref

PTMMapsExtractor

Marcus, Ankit

ncsa.xml.greenindexroute, ncsa.csv.greenindexroute

PythonOpenCVLinux

Constantinos
medici_ptm_3d	Java

Marcus

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

maps

ptm/browse/

greenroute

PTM3DExtractor

Marcus

Constantinos
medici_images_ptm	Java					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-ptm/browse/ImagesPTMExtractor	Constantinos
extractors-rabbitmq (look like examples)

ncsa.image.knn_numerals

PythonOpenCVLinuxMarcus Marcus

ncsa.audio.speech2text

JavaCMU Sphinx, ffmpeg, soxLinuxMarcus Marcusncsa.nlp.simplelanguagePythonnumpy

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

nlp

rabbitmq/browse

/SimpleLanguageLianancsa.nlp.simplesummaryPython

Natural Language Toolkit (NLTK) for Python, NLTK Data or at least:

Name not clear extractors-seabird/	Scala

 nltk.corpus,nltk.stem.porter and nltk.tokenize.punkt.

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/

repos/extractors-nlp/browse/SimpleSummaryLianancsa.nlp.SNLPSentimentJava Stanford CoreNLP tool, java, maven

repos/extractors-seabird/browse	Luigi
medici_3d_x3d (one of extractors-3d)	Java					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

nlp

3d/browse/

SNLP/SNLPSentimentExtractorLiana, Marcus(?)ncsa.nlp.wordtablesPython requests, pika, win32com

ObjJSONExtractor	Constantinos
medici_3d_obj_merger (one of extractors-3d)	Java					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

nlp/browse/WordTablesExtractorLiana                     NOT DEPLOYED      ncsa.image.digitpy (notes: not in the Wiki page)Pythonopencv

3d/browse/OBJMergerExtractor	Constantinos
medici_oni (one of extractors-3d)	Java					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

3d/browse/

handwritten/SimpleDigitPython

OniExtractor

 ncsa.cv.pdfimages (not in the wiki page) pdfimages, from poppler-utils

Constantinos
medici_ply_obj (one of extractors-3d)	Java					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

3d/browse/

poppler ncsa.cv.caltech101PythonMatlab and VLFeat 64-bit Mac OS 

PlyObjExtractor	Constantinos
medici_3d_metadata (one of extractors-3d)	Java					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

3d/browse/

vlfeat

ThreeDMetadataExtractor	Constantinos
medici_x3d_html (one of extractors-3d)

dbpedia

Java

PythonNatural Language Toolkit (NLTK) and rdflib.Luigi Marini

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

dbpedia

3d/browse/X3DhtmlExtractor

Luigi Marini

Constantinos
ncsa.arcgis.landsat7mosaic

digest

	Python

 

ArcGIS

Windows

 

No

 

Smruti Padhy

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-bd-

digest

cz/browse/ndviextractor

 

Smruti
ncsa.

image

arcgis.

geotiff

floodplain	Python

GDAL, Cython, numpy,
pygeoprocessing,
pika,
requests

Linux

ArcGIS

Windows

No

Smruti Padhy

Rui

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-bd-

geotiff

cz/browse

Rui, Mostafa Elag

/terex_floodplain/config.py	Smruti
medici_book	Java

ncsa.hpcPython

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

hpc

books/browse/BookPreviewExtractor

 

Theerasit Issaranon
medici_image_pyramid

LSVA

	Java					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

lsva

books/browse

 

/ImagePreviewPyramidExtractor-shebook	Theerasit Issaranon
shebook	Java

LSVA integrated

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

lsva-integrated

books/browse/SheBookPreviewExtractor/src/BookPreviewExtractor

ncsa.movieslicePython   

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors

-movieslice/browseSandeepmri2meshPythonpymedici, subprocess, logging, os, numpy, shutil, zipfile

-books/browse/SheBookPreviewExtractor/src/bookpreviewextractor	Theerasit Issaranon
lsva-cedd	Java					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

mri

cedd/browse

/mri2meshMarcus

Constantinos
ncsa.cinemetrics

msc-ChemCBCExtractor

	Python					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

msc

cinemetrics/browse

/ChemCBCExtractorYan

Constantinos
ncsa.image.metadata

msc-IsletExtractor

	Python

requests, pika, openpyxl, xlrd, pymongo

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

msc

core/browse/

IsletExtractorYanmsc-MonitorExtractorPythonrequests, pika, openpyxl, xlrd, pymongo

image/metadata	Max. Rob
ncsa.debod.segmentor						https://opensource.ncsa.illinois.edu/bitbucket/projects/

CATS

DEBOD/repos/extractors-

msc

cellsegmentor/browse

/MonitorExtractorYan

ncsa.

msc

image.

dailymonitor

dmp

Pythonrequests, pika, openpyxl, xlrd, pymongo

https://opensource.ncsa.illinois.edu/bitbucket/projects/

CATS

DEBOD/repos/extractors-

msc

debod/browse

/OldMonitorExtractorAshwinimsc-PhenotypeExtractorPythonrequests, pika, openpyxl, xlrd, pymongo  

https://opensource.ncsa.illinois.edu/bitbucket/projects/

CATS

DEBOD/repos/extractors-

msc

dmp/browse

/PhenotypeExtractorYan

ncsa.

nlp.SNLPJava Stanford CoreNLP tool, java, maven

image.sphog.debod						https://opensource.ncsa.illinois.edu/bitbucket/projects/

CATS

DEBOD/repos/extractors-

nlp

handwrittendecimals/browse

/SNLP/SNLPExtractorLiana

ncsa.

nlp.tikaPython Tika project page, pymedici

image.iarp_remove_circle						https://opensource.ncsa.illinois.edu/bitbucket/projects/

CATS

IARP/repos/

extractors-nlp

image_fetcher/browse/

tikaLianaperson-detectorPython MATLAB, FFMPEG, requests and pika

extractors/remove_circle	Marcus
ncsa.cv.meangrey						https://opensource.ncsa.illinois.edu/bitbucket/projects/

CATS

IARP/repos/

extractors-person-detector

image_fetcher/browse/extractors/

pythonSandeep

mean_grey

Marcus