Page History

This page is for the refactoring of the existing extractors. The original wiki page Hosted VMs is still used for the deployments. 

As we figure out who's working on what, please start with the following steps for the extractor(s) you chose:

• be able to run the extractor
• add a README, specifically a readme.md (i.e. in markdown), with information on how to install dependencies and run the extractor (in its current shape)
• start looking at dbpedia extractor for template
• Learn about jsonld by playing in the playground here http://json-ld.org/
• Go through the README for the docker extractors template: https://opensource.ncsa.illinois.edu/bitbucket/projects/BD/repos/bd-extractor-template/browse

Steps to take for every extractor in this list:

1. Docker containers
2. JSONLD
3. Extractor info registration
4. Use pyclowder (for python extractors)
5. Add status messages to all extractors and fix level granularity
   1. Make status constants (DONE, ERROR)
   2. Arcgis multiprocessing extractor
6. Register on on demand execution queues
   1. Add on demand key binding to configuration file: messageType = "*.file.text.plain", "extractors."+extractorName
7. Standardize around python logging
   1. Figure out what to log and what format to follow
8. Add logstash to docker compose
9. Add sample input/ouput to git repository
10. Add icon for tools catalog to git repository
11. Add entry to Tools catalog, with icon

ID (Extractor Name from config file, same as queue name)	Programming Language	Software	OS	Can be Dockerized?	Assigned To	Repo	Author
DEPLOYED
ncsa.image.ocr	Python	Tesseract	Linux		Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/ocr
ncsa.cv.faces	Python	OpenCV	Linux		Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencv	Liana
ncsa.cv.eyes	Python	OpenCV	Linux		Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencv	Liana
ncsa.cv.closeups	Python	OpenCV	Linux		Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencv	Liana
ncsa.cv.profiles	Python	OpenCV	Linux		Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencv	Liana
ncsa.cellprofiler.fluorescentcomet	Python	pymedici	Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.fly	Python		Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.human	Python		Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.silvercomet	Python		Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.speckle	Python		Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.trackobject	Python		Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.tumor	Python		Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.yeast	Python		Windows	No		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.image.sphog	Python	Matlab, mnist-sphog	Linux		Gregory Jansen	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/handwritten/HandwrittenNumbers
ncsa.image.caltech101					Gregory Jansen
ncsa.bisque.histogram (notes: disabled)	Python		Linux
ncsa.bisque.metadata (notes: disabled)	Python		Linux
census-section-segmentor	Java		Linux		Sandeep Puthanveetil Satheesan	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/census	Liana, Inna
ncsa.cv.river	Python	OpenCV (python), convert (from imagemagick), and Gdal	Linux		Smruti Padhy	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/river	Liana
ncsa.geo.shpExtractor	Python	gdal	Linux		Jong Lee	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-geo/browse	Jong Lee
ncsa.geo.tiffExtractor	Python	gdal	Linux		Jong Lee	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-geo/browse	Jong Lee
ncsa.image.geotiff	Python	GDAL, Cython, numpy, pygeoprocessing	Linux		Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-geotiff/browse	Rui, Mostafa Elag
ncsa.image.ponddetect	Python	Matlab	Linux		Marcus Slavenas	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-maps/browse/feature_detection	Marcus, Ankit
ncsa.image.humanpref	Python	Matlab	Linux		Marcus Slavenas	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-maps/browse/humanpref	Marcus, Ankit
ncsa.xml.greenindexroute, ncsa.csv.greenindexroute	Python	OpenCV	Linux		Marcus Slavenas	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-maps/browse/greenroute	Marcus
ncsa.image.knn_numerals	Python	OpenCV	Linux		Marcus Slavenas		Marcus
ncsa.audio.speech2text	Java	CMU Sphinx, ffmpeg, sox	Linux		Marcus Slavenas	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/audio/speech2text	Marcus
ncsa.audio.preview	Python				Inna	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/audio/preview
ncsa.nlp.simplelanguage	Python	numpy			Inna	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/SimpleLanguage	Liana
ncsa.nlp.simplesummary	Python	Natural Language Toolkit (NLTK) for Python, NLTK Data or at least:  nltk.corpus,nltk.stem.porter and nltk.tokenize.punkt.			Gregory Jansen	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/SimpleSummary	Liana
ncsa.nlp.SNLPSentiment	Java	Stanford CoreNLP tool, java, maven			Marcus Slavenas	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/SNLP/SNLPSentimentExtractor	Liana, Marcus(?)
ncsa.nlp.wordtables	Python	requests, pika, win32com				https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/WordTablesExtractor	Liana
siegfried	Python					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-siegfried/browse	Gregory Jansen
ncsa.versus.image	Java	Versus	Linux		Smruti Padhy	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-versus/browse	Kenton, Smruti
ncsa.image.preview (note: check if really deployed. there is an extractor in Hosted VMs list with a similar name.)	Python					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/image/preview	Rob, Sandeep
ncsa.pdf.preview (note: check if really deployed. there is an extractor in Hosted VMs list with a similar name.)	Python					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/pdf/preview	Rob
ncsa.video.preview (note: check if really deployed. there is an extractor in Hosted VMs list with a similar name.)	Python					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/video/preview	Rob
NOT DEPLOYED
ncsa.image.digitpy	Python	opencv

...

	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/handwritten/SimpleDigitPython
ncsa.cv.pdfimages		pdfimages, from poppler-utils				https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/poppler
ncsa.cv.caltech101	Python	Matlab and VLFeat	64-bit Mac OS			https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/vlfeat
dbpedia	Python	Natural Language Toolkit (NLTK) and rdflib.			Luigi Marini

ID (Extractor Name from config file,

same as queue name)

Programming

Language

SoftwareOSAssigned ToLink to repoWho wrote or worked on the codeDEPLOYED      ncsa.image.ocrPythonTesseractLinuxRuiocr 

ncsa.cv.faces

PythonOpenCVLinuxInna (may be?)

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

dbpedia/browse

/opencvLiana

Luigi Marini
digest

ncsa.cv.eyes

	Python

OpenCVLinuxInna

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

digest/browse

/opencvLiana

ncsa.

cv.closeups

hpc	Python

OpenCVLinuxInna

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

hpc/browse

/opencvLiana

	Sandeep Puthanveetil Satheesan
LSVA	Java

ncsa.cv.profiles

PythonOpenCVLinuxInna

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

lsva/browse

/opencv

Liana

ncsa.cellprofiler.fluorescentcomet

Pythonpymedici

, Constantinos
LSVA integrated

Windows

Sandeep Puthanveetil Satheesan

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

lsva-integrated/browse

/cellprofilerLiana

Sandeep Puthanveetil Satheesan
ncsa.

cellprofiler.fly

movieslice	Python

Windows

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

movieslice/browse

/cellprofilerLiana

Sandeep
mri2mesh

ncsa.cellprofiler.human

	Python	pymedici, subprocess, logging, os, numpy, shutil, zipfile

Windows

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

mri/browse/

cellprofiler

mri2mesh

Liana

Marcus
msc-ChemCBCExtractor	Python	requests, pika, openpyxl, xlrd, pymongo	Linux		Yan Zhao

ncsa.cellprofiler.silvercomet

Python Windows 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

msc/browse/

cellprofiler

ncsa.cellprofiler.speckle

Python Windows

ChemCBCExtractor

Liana

Yan
msc-IsletExtractor	Python	requests, pika, openpyxl, xlrd, pymongo	Linux		Yan Zhao

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

msc/browse/

cellprofilerLiana

ncsa.cellprofiler.trackobject

Python Windows

IsletExtractor	Yan
msc-MonitorExtractor	Python	requests, pika, openpyxl, xlrd, pymongo	Linux		Yan Zhao

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

msc/browse/

cellprofiler

MonitorExtractor

Liana

Yan
ncsa.

cellprofiler

msc.

tumor

dailymonitor	Python	requests, pika, openpyxl, xlrd, pymongo

Windows 

not used

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

msc/browse/

cellprofilerLiana

ncsa.cellprofiler.yeast

Python Windows

OldMonitorExtractor	Ashwini
msc-PhenotypeExtractor	Python	requests, pika, openpyxl, xlrd, pymongo	Linux		Yan Zhao

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

msc/browse/

cellprofiler

PhenotypeExtractor

Liana

Yan
ncsa.

image

nlp.

sphog

SNLP

Python

Java

Matlab, mnist-sphog

Stanford CoreNLP tool, java, maven

Linux

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

nlp/browse/

handwritten/HandwrittenNumbers 

SNLP/SNLPExtractor

Liana

• ncsa.image.caltech101

      

ncsa.

bisque.histogram (notes: disabled)

nlp.tika	Python	Tika project page, pymedici

Linux 

Kenton McHenry

https://opensource.ncsa.

bisque.metadata (notes: disabled)

illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/tika	Liana
person-detector	Python	MATLAB, FFMPEG, requests and pika

Python Linuxcensus-section-segmentorJava 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-person-detector/browse/python	Sandeep
ncsa.person-tracker	Python	python, MATLAB, FFMPEG requests and pika

Linux

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

person-tracking/browse/

census

python

Liana, Inna

Sandeep

ncsa.cv.river PythonOpenCV (python), convert (from imagemagick), and Gdal

terra.plantcv	Python	pika requests wheel

Linux

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

plantcv/browse

/riverLiana

	Yan
medici_PTM_thumbnails	Java

ncsa.geo.shpExtractorPythongdalLinux

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

geo

ptm/browse

Jong Lee

/PTMThumbnailExtractor	Constantinos
medici_PTM_metadata	Java

ncsa.geo.tiffExtractorPythongdalLinux

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

geo

ptm/browse

Jong Lee

ncsa.image.ponddetect

PythonMatlabLinux

/PTMMetadataExtractor	Constantinos
Name not clear PtmMetadata(?)	Java

Marcus

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

maps

ptm/browse/

feature_detectionMarcus, Ankit

PTMMetadata	Constantinos
medici_ptm_maps	Java

ncsa.image.humanprefPythonMatlabLinuxMarcus

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

maps

ptm/browse/

humanpref

PTMMapsExtractor

Marcus, Ankit

ncsa.xml.greenindexroute, ncsa.csv.greenindexroute

PythonOpenCVLinux

Constantinos
medici_ptm_3d	Java

Marcus

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

maps

ptm/browse/

greenrouteMarcus

PTM3DExtractor	Constantinos
medici_images_ptm	Java

ncsa.image.knn_numerals

PythonOpenCVLinuxMarcus Marcus

ncsa.audio.speech2text

JavaCMU Sphinx, ffmpeg, soxLinuxMarcus Marcusncsa.nlp.simplelanguagePythonnumpy

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

nlp

ptm/browse/

SimpleLanguage

ImagesPTMExtractor

Lianancsa.nlp.simplesummaryPython

Natural Language Toolkit (NLTK) for Python, NLTK Data or at least:

Constantinos
extractors-rabbitmq (look like examples)

 nltk.corpus,nltk.stem.porter and nltk.tokenize.punkt.

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

nlp

rabbitmq/browse

/SimpleSummaryLianancsa.nlp.SNLPSentimentJava Stanford CoreNLP tool, java, maven

Name not clear extractors-seabird/	Scala					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

nlp

seabird/browse

/SNLP/SNLPSentimentExtractorLiana, Marcus(?)ncsa.nlp.wordtablesPython requests, pika, win32com

	Luigi
medici_3d_x3d (one of extractors-3d)	Java					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

nlp

3d/browse/

WordTablesExtractorLiana

ObjJSONExtractor	Constantinos
medici_3d_obj_merger (one of extractors-3d)	Java

                 NOT DEPLOYED 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-3d/browse/OBJMergerExtractor	Constantinos
medici_oni (one of extractors-3d)	Java

 

https://opensource.ncsa.

image.digitpy (notes: not in the Wiki page)

illinois.edu/bitbucket/projects/CATS/repos/extractors-3d/browse/OniExtractor	Constantinos
medici_ply_obj (one of extractors-3d)	Java

Pythonopencv

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

3d/browse

/handwritten

/

SimpleDigitPython

PlyObjExtractor

 

Constantinos
medici_3d_metadata (one of extractors-3d)	Java

ncsa.cv.pdfimages (not in the wiki page) pdfimages, from poppler-utils

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

3d/browse/

poppler 

ThreeDMetadataExtractor	Constantinos
medici_x3d_html (one of extractors-3d)	Java

ncsa.cv.caltech101PythonMatlab and VLFeat 64-bit Mac OS 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

3d/browse/

vlfeat

X3DhtmlExtractor

 

Constantinos
ncsa.arcgis.landsat7mosaic

dbpedia

	Python

 Natural Language Toolkit (NLTK) and rdflib. 

ArcGIS

Windows

No

Smruti Padhy

Luigi Marini

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

dbpedia

bd-cz/browse/ndviextractor

Luigi Marini

Smruti
ncsa.arcgis.floodplain

digest

	Python	ArcGIS

 

Windows

 

No

Smruti Padhy

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

digest

bd-cz/browse

 ncsa.image.geotiffPython

GDAL, Cython, numpy,
pygeoprocessing,
pika,
requests

Linux

/terex_floodplain/config.py	Smruti
medici_book	Java

Rui

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

geotiff

books/browse

Rui, Mostafa Elag

/BookPreviewExtractor	Theerasit Issaranon
medici_image_pyramid	Java

ncsa.hpcPython

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

hpc

books/browse

 

/ImagePreviewPyramidExtractor-shebook	Theerasit Issaranon
shebook

LSVA

	Java					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

lsva

books/browse/SheBookPreviewExtractor/src/BookPreviewExtractor

LSVA integrated    

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

lsva-integrated

books/browse

 

/SheBookPreviewExtractor/src/bookpreviewextractor	Theerasit Issaranon
lsva-cedd	Java

ncsa.movieslicePython

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

movieslice

cedd/browse

Sandeep

Constantinos

mri2mesh

ncsa.cinemetrics	Python

pymedici, subprocess, logging, os, numpy, shutil, zipfile

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

mri

cinemetrics/browse

/mri2meshMarcus

Constantinos
ncsa.image.metadata

msc-ChemCBCExtractor

	Python					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

msc

core/browse

/ChemCBCExtractorYanmsc-IsletExtractorPythonrequests, pika, openpyxl, xlrd, pymongo

/image/metadata	Max. Rob
ncsa.debod.segmentor						https://opensource.ncsa.illinois.edu/bitbucket/projects/

CATS

DEBOD/repos/extractors-

msc

cellsegmentor/browse

/IsletExtractorYanmsc-MonitorExtractorPythonrequests, pika, openpyxl, xlrd, pymongo

ncsa.image.dmp						https://opensource.ncsa.illinois.edu/bitbucket/projects/

CATS

DEBOD/repos/extractors-

msc

debod/browse

/MonitorExtractorYanncsa.msc.dailymonitorPythonrequests, pika, openpyxl, xlrd, pymongo  

https://opensource.ncsa.illinois.edu/bitbucket/projects/

CATS

DEBOD/repos/extractors-

msc

dmp/browse

/OldMonitorExtractorAshwinimsc-PhenotypeExtractorPythonrequests, pika, openpyxl, xlrd, pymongo

ncsa.image.sphog.debod						https://opensource.ncsa.illinois.edu/bitbucket/projects/

CATS

DEBOD/repos/extractors-

msc

handwrittendecimals/browse

/PhenotypeExtractorYan

ncsa.

nlp.SNLPJava Stanford CoreNLP tool, java, maven

image.iarp_remove_circle						https://opensource.ncsa.illinois.edu/bitbucket/projects/

CATS

IARP/repos/

extractors-nlp

image_fetcher/browse/

SNLP

extractors/

SNLPExtractor

remove_circle

Liana

Marcus
ncsa.

nlp

cv.

tika

meangrey

PythonTika project page, pymedici

https://opensource.ncsa.illinois.edu/bitbucket/projects/

CATS

IARP/repos/

extractors-nlp

image_fetcher/browse/

tikaLiana                    

extractors/mean_grey

Marcus