Page History

...

• be able to run the extractor
• add a README, specifically a readme.md (i.e. in markdown), with information on how to install dependencies and run the extractor (in its current shape)
• start looking at dbpedia extractor for template
• Learn about jsonld by playing in the playground here http://json-ld.org/
• Go through the README for the docker extractors template: https://opensource.ncsa.illinois.edu/bitbucket/projects/BD/repos/bd-extractor-template/browse

...

1. Docker containers
2. JSONLD
3. Extractor info registration
4. Use pyclowder (for python extractors)
5. Add status messages to all extractors and fix level granularity
   1. Make status constants (DONE, ERROR)
   2. Arcgis multiprocessing extractor
6. Register on on demand execution queues
   1. Add on demand key binding to configuration file: messageType = "*.file.text.plain", "extractors."+extractorName
7. Standardize around python logging
   1. Figure out what to log and what format to follow
8. Add logstash to docker compose
9. Add sample input/ouput to git repository
10. Add icon for tools catalog to git repository
11. Add entry to Tools catalog, with icon, sample input/output

ID (Extractor Name from config file,

same as queue name)	Programming

Language	Software	OS	Can be Dockerized

?Can be upload to Docker Hub DEPLOYED        

?	Assigned To

Link to repoWho wrote or worked on the code

Repo	Author
DEPLOYED

ncsa.image.ocr	Python	Tesseract	Linux

 

Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/ocr
ncsa.cv.faces	Python	OpenCV	Linux

 

Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencv	Liana
ncsa.cv.eyes	Python	OpenCV	Linux

 

		Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencv	Liana
ncsa.cv.closeups	Python	OpenCV	Linux

 

		Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencv	Liana
ncsa.cv.profiles	Python	OpenCV	Linux

 

		Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencv	Liana
ncsa.cellprofiler.fluorescentcomet	Python	pymedici	Windows	No

 

		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.fly	Python		Windows	No

 

		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.human	Python		Windows	No

 

		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.silvercomet	Python		Windows	No

 https

		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.speckle	Python		Windows	No

 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.trackobject	Python		Windows	No

 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.tumor	Python		Windows	No

 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.cellprofiler.yeast	Python		Windows	No

 

		https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler	Liana
ncsa.image.sphog	Python	Matlab, mnist-sphog	Linux

 

Gregory Jansen	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/handwritten/HandwrittenNumbers
ncsa.image.caltech101

 

					Gregory Jansen
ncsa.bisque.histogram (notes: disabled)	Python		Linux

 

ncsa.bisque.metadata (notes: disabled)	Python		Linux

 

census-section-segmentor	Java		Linux

 

		Sandeep Puthanveetil Satheesan	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/census	Liana, Inna
ncsa.cv.river	Python	OpenCV (python), convert (from imagemagick), and Gdal	Linux

 

Smruti Padhy	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/river	Liana
ncsa.geo.shpExtractor	Python	gdal	Linux

 

Jong Lee	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-geo/browse	Jong Lee
ncsa.geo.tiffExtractor	Python	gdal	Linux

 

		Jong Lee	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-geo/browse	Jong Lee
ncsa.image.geotiff	Python	GDAL, Cython, numpy, pygeoprocessing	Linux

 

Rui	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-geotiff/browse	Rui, Mostafa Elag
ncsa.image.ponddetect	Python	Matlab	Linux

 

		Marcus Slavenas	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-maps/browse/feature_detection	Marcus, Ankit
ncsa.image.humanpref	Python	Matlab	Linux

 

		Marcus Slavenas	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-maps/browse/humanpref	Marcus, Ankit
ncsa.xml.greenindexroute, ncsa.csv.greenindexroute	Python	OpenCV	Linux

 

		Marcus Slavenas	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-maps/browse/greenroute	Marcus
ncsa.image.knn_numerals	Python	OpenCV	Linux

 

Marcus Slavenas		Marcus
ncsa.audio.speech2text	Java	CMU Sphinx, ffmpeg, sox	Linux

 

Marcus Slavenas	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/audio/speech2text	Marcus
ncsa.audio.preview	Python

 

Inna	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/audio/preview
ncsa.nlp.simplelanguage	Python	numpy

 

Inna	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/SimpleLanguage	Liana
ncsa.nlp.simplesummary	Python	Natural Language Toolkit (NLTK) for Python, NLTK Data or at least:  nltk.corpus,nltk.stem.porter and nltk.tokenize.punkt.

  

			Gregory Jansen	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/SimpleSummary	Liana
ncsa.nlp.SNLPSentiment	Java	Stanford CoreNLP tool, java, maven

 

Marcus Slavenas	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/SNLP/SNLPSentimentExtractor	Liana, Marcus(?)
ncsa.nlp.wordtables	Python	requests, pika, win32com

 

				https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/WordTablesExtractor	Liana
siegfried	Python

 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-siegfried/browse	Gregory Jansen
ncsa.versus.image	Java	Versus	Linux

 

		Smruti Padhy	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-versus/browse	Kenton, Smruti
ncsa.image.preview (note: check if really deployed. there is an extractor in Hosted VMs list with a similar name.)	Python

 

					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/image/preview	Rob, Sandeep
ncsa.pdf.preview (note: check if really deployed. there is an extractor in Hosted VMs list with a similar name.)	Python

 

					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/pdf/preview	Rob
ncsa.video.preview (note: check if really deployed. there is an extractor in Hosted VMs list with a similar name.)	Python

 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/video/preview

Rob

     

NOT DEPLOYED
ncsa.image.digitpy	Python	opencv

 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/handwritten/SimpleDigitPython

 

ncsa.cv.pdfimages

 

pdfimages, from poppler-utils

       

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/poppler
ncsa.cv.caltech101	Python	Matlab and VLFeat	64-bit Mac OS			https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/vlfeat
dbpedia	Python	Natural Language Toolkit (NLTK) and rdflib.			Luigi Marini	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-dbpedia/browse	Luigi Marini
digest	Python

NOT DEPLOYED

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-digest/browse

 

ncsa.hpc

 

Python

 

https://opensource.ncsa.

image.digitpy

illinois.edu/bitbucket/projects/CATS/repos/extractors-hpc/browse	Sandeep Puthanveetil Satheesan
LSVA	Java

Pythonopencv

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

lsva/browse

/handwritten/SimpleDigitPython 

	Liana, Constantinos
LSVA integrated

ncsa.cv.pdfimagespdfimages, from poppler-utils 

Sandeep Puthanveetil Satheesan

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-lsva-

cv

integrated/browse

/poppler 

Sandeep Puthanveetil Satheesan
ncsa.

cv.caltech101

movieslice	Python

Matlab and VLFeat64-bit Mac OS 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

cv

movieslice/browse

/vlfeat 

Sandeep

dbpedia

mri2mesh	Python

 Natural Language Toolkit (NLTK) and rdflib.

pymedici, subprocess, logging, os, numpy, shutil, zipfile

Luigi Marini

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

dbpedia

mri/browse/mri2mesh

Luigi Marini

Marcus
msc-ChemCBCExtractor

digest

	Python	requests,

 

pika, openpyxl, xlrd, pymongo

Linux

  

Yan Zhao

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

digest

msc/browse/ChemCBCExtractor

 

Yan
msc-IsletExtractor

ncsa.hpc

	Python

 

requests, pika, openpyxl, xlrd, pymongo

Linux

 

Yan Zhao

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

hpc

msc/browse/IsletExtractor

 

Yan

LSVA

msc-MonitorExtractor

Java 

Python

requests, pika, openpyxl, xlrd, pymongo

Linux

 

Yan Zhao

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

lsva

msc/browse/MonitorExtractor

 LSVA integrated 

Yan
ncsa.msc.dailymonitor	Python	requests, pika, openpyxl, xlrd, pymongo

   

not used

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

lsva-integrated

msc/browse/OldMonitorExtractor

 

Ashwini
msc-PhenotypeExtractor

ncsa.movieslice

	Python

 

requests, pika, openpyxl, xlrd, pymongo

Linux

 

Yan Zhao

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

movieslice

msc/browse/PhenotypeExtractor

Sandeepmri2meshPythonpymedici, subprocess, logging, os, numpy, shutil, zipfile

Yan
ncsa.nlp.SNLP	Java	Stanford CoreNLP tool, java, maven

 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

mri

nlp/browse/SNLP/

mri2mesh

SNLPExtractor

Marcus

Liana
ncsa.nlp.tika

msc-ChemCBCExtractor

	Python

requests,Linuxpika, openpyxl, xlrd, pymongo

Tika project page, pymedici

Yan

Kenton McHenry

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

msc

nlp/browse/

ChemCBCExtractor

tika

Yan

Liana

msc

person-

IsletExtractor

detector	Python	MATLAB, FFMPEG, requests

, pika, openpyxl, xlrd, pymongo

and pika

LinuxYan

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-person-

msc

detector/browse/

IsletExtractor

python

Yan

Sandeep

msc

ncsa.person-

MonitorExtractor

tracker	Python

requests, pika, openpyxl, xlrd, pymongo

python, MATLAB, FFMPEG requests and pika

LinuxYan

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-person-

msc

tracking/browse/

MonitorExtractor

python

Yan

Sandeep

ncsa

terra.

msc.dailymonitor

plantcv	Python	pika requests

, pika, openpyxl, xlrd, pymongo

wheel

not used

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

msc

plantcv/browse

/OldMonitorExtractorAshwinimsc-PhenotypeExtractorPython

requests, pika, openpyxl, xlrd, pymongo

Yan
medici_PTM_thumbnails	Java

LinuxYan

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

msc

ptm/browse/

PhenotypeExtractor

PTMThumbnailExtractor

Yan

Constantinos
medici_PTM_metadata

ncsa.nlp.SNLP

	Java

Stanford CoreNLP tool, java, maven 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

nlp

ptm/browse/

SNLP/SNLPExtractor

PTMMetadataExtractor

Lianancsa.nlp.tikaPython Tika project page, pymedici

Constantinos
Name not clear PtmMetadata(?)	Java					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

nlp

ptm/browse/

tikaLianaperson-detectorPython MATLAB, FFMPEG, requests and pika

PTMMetadata	Constantinos
medici_ptm_maps	Java					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

person-detector

ptm/browse/

python

PTMMapsExtractor

Sandeepncsa.person-trackerPythonpython, MATLAB, FFMPEG requests and pika

Constantinos
medici_ptm_3d	Java					https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

person-tracking

ptm/browse/

python

PTM3DExtractor

Sandeep

Constantinos
medici_images_ptm	Java

terra.plantcvPythonpika
requests
wheel

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

plantcv

ptm/browse/ImagesPTMExtractor

Yan

Constantinos
extractors-rabbitmq (look like examples)

medici_PTM_thumbnailsJava

						https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

ptm

rabbitmq/browse

/PTMThumbnailExtractorConstantinos

Name not clear extractors-seabird/	Scala

medici_PTM_metadataJava 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

ptm

seabird/browse

/PTMMetadataExtractorConstantinos

Name not clear

Luigi
medici_3d_x3d (one of extractors-3d

PtmMetadata(?

)	Java

 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

ptm

3d/browse/

PTMMetadata

ObjJSONExtractor	Constantinos
medici_

ptm_maps

3d_obj_merger (one of extractors-3d)	Java

 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

ptm

3d/browse/

PTMMapsExtractor

OBJMergerExtractor	Constantinos
medici_

ptm_3d

oni (one of extractors-3d)	Java

 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

ptm

3d/browse/

PTM3DExtractor

OniExtractor	Constantinos
medici_

images_ptm

ply_obj (one of extractors-3d)	Java

 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

ptm

3d/browse/

ImagesPTMExtractor

PlyObjExtractor

Constantinos

extractors-rabbitmq

(look like examples)

 

medici_3d_metadata (one of extractors-3d)	Java

 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

rabbitmq

3d/browse/ThreeDMetadataExtractor

 

Constantinos
medici_x3d_html (one of extractors-3d)	Java

Name not clear extractors-seabird/Scala 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

seabird

3d/browse/X3DhtmlExtractor

Luigimedici_3d_x3d (one of extractors-3d)Java    

Constantinos
ncsa.arcgis.landsat7mosaic	Python	ArcGIS	Windows	No	Smruti Padhy

 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

3d

bd-cz/browse/

ObjJSONExtractor

ndviextractor

Constantinosmedici_3d_obj_merger (one of extractors-3d)Java    

Smruti
ncsa.arcgis.floodplain	Python	ArcGIS	Windows	No	Smruti Padhy	https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

3d

bd-cz/browse

/OBJMergerExtractor

/terex_floodplain/config.py

Smruti

Constantinos

medici_

oni (one of extractors-3d)

book	Java

 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

3d

books/browse/

OniExtractor

BookPreviewExtractor

Constantinos

Theerasit Issaranon
medici_

ply_obj (one of extractors-3d)

image_pyramid	Java

 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

3d

books/browse/

PlyObjExtractorConstantinosmedici_3d_metadata (one of extractors-3d) 

ImagePreviewPyramidExtractor-shebook	Theerasit Issaranon
shebook	Java

 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors

-3d/browse/ThreeDMetadataExtractorConstantinosmedici_x3d_html (one of extractors-3d) Java     

-books/browse/SheBookPreviewExtractor/src/BookPreviewExtractor

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

3d

books/browse/

X3DhtmlExtractorConstantinosncsa.arcgis.landsat7mosaicPythonArcGISWindowsNo 

SheBookPreviewExtractor/src/bookpreviewextractor	Theerasit Issaranon
lsva-cedd	Java

Smruti Padhy

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

bd-cz

cedd/browse

/ndviextractorSmruti

Constantinos
ncsa.

arcgis.floodplain

cinemetrics	Python

ArcGISNoWindowsSmruti Padhy

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

bd-cz

cinemetrics/browse

/terex_floodplain/config.pySmruti

	Constantinos
ncsa.image.metadata	Python

medici_bookJava 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-

books

core/browse/image/

BookPreviewExtractorTheerasit Issaranonmedici_image_pyramidJava

metadata	Max. Rob
ncsa.debod.segmentor						https://opensource.ncsa.illinois.edu/bitbucket/projects/

CATS

DEBOD/repos/extractors-

booksTheerasit IssaranonshebookJava

cellsegmentor/browse

/ImagePreviewPyramidExtractor-shebook

ncsa.image.dmp

https://opensource.ncsa.illinois.edu/bitbucket/projects/

CATS

DEBOD/repos/extractors-

books

debod/browse

/SheBookPreviewExtractor/src/BookPreviewExtractor 

https://opensource.ncsa.illinois.edu/bitbucket/projects/

CATS

DEBOD/repos/extractors-

books

dmp/browse

/SheBookPreviewExtractor/src/bookpreviewextractor

ncsa.image.sphog.debod

Theerasit Issaranonlsva-ceddJava

						https://opensource.ncsa.illinois.edu/bitbucket/projects/

CATS

DEBOD/repos/extractors-

cedd

handwrittendecimals/browse

Constantinos

ncsa

.cinemetricsPython

.image.iarp_remove_circle						https://opensource.ncsa.illinois.edu/bitbucket/projects/

CATS

IARP/repos/image_fetcher/browse/extractors

-cinemetrics

/

browse

remove_circle

Constantinos

Marcus
ncsa.

image

cv.

metadataPython

meangrey						https://opensource.ncsa.illinois.edu/bitbucket/projects/

CATS

IARP/repos/

extractors-core

image_fetcher/browse/

image

extractors/

metadataMax. Rob

mean_grey

Marcus