...
- be able to run the extractor
- add a README, specifically a readme.md (i.e. in markdown), with information on how to install dependencies and run the extractor (in its current shape)
- start looking at dbpedia extractor for template
- Learn about jsonld by playing in the playground here http://json-ld.org/
- Go through the README for the docker extractors template: https://opensource.ncsa.illinois.edu/bitbucket/projects/BD/repos/bd-extractor-template/browse
...
- Docker containers
- JSONLD
- Extractor info registration
- Use pyclowder (for python extractors)
- Add status messages to all extractors and fix level granularity
- Make status constants (DONE, ERROR)
- Arcgis multiprocessing extractor
- Register on on demand execution queues
- Add on demand key binding to configuration file: messageType = "*.file.text.plain", "extractors."+extractorName
- Standardize around python logging
- Figure out what to log and what format to follow
Add logstash to docker compose- Add sample input/ouput to git repository
- Add icon for tools catalog to git repository
- Add entry to Tools catalog, with icon
ID (Extractor Name from config file, |
---|
same as queue name) | Programming |
---|
Language | Software | OS | Can be Dockerized |
---|
? | Assigned To |
---|
Repo | Author | ||||||
---|---|---|---|---|---|---|---|
DEPLOYED |
ncsa.image.ocr | Python | Tesseract | Linux |
---|
Rui |
https://opensource.ncsa. |
illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/ocr | |||
ncsa.cv.faces | Python | OpenCV | Linux |
---|
Rui | https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencv | Liana | ||
ncsa.cv.eyes | Python | OpenCV | Linux |
---|
Rui | https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencv | Liana | |
ncsa.cv.closeups | Python | OpenCV | Linux |
---|
Rui | https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencv | Liana | ||
ncsa.cv.profiles | Python | OpenCV | Linux |
---|
Rui | https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencv | Liana | ||
ncsa.cellprofiler.fluorescentcomet | Python | pymedici | Windows | No |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler | Liana | ||||
ncsa.cellprofiler.fly | Python | Windows | No |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler | Liana | ||||
ncsa.cellprofiler.human | Python | Windows | No |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler | Liana | ||||
ncsa.cellprofiler.silvercomet | Python | Windows | No |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler | Liana | |||
ncsa.cellprofiler.speckle | Python | Windows | No |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler | Liana | ||||
ncsa.cellprofiler.trackobject | Python | Windows | No |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler | Liana | |||
ncsa.cellprofiler.tumor | Python | Windows | No |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler | Liana | |||
ncsa.cellprofiler.yeast | Python | Windows | No |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofiler | Liana | ||
ncsa.image.sphog | Python | Matlab, mnist-sphog | Linux |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/handwritten/HandwrittenNumbers | ||||
ncsa.image.caltech101 |
---|
ncsa.bisque.histogram (notes: disabled) | Python | Linux |
---|
ncsa.bisque.metadata (notes: disabled) | Python | Linux |
---|
census-section-segmentor | Java | Linux |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/census | Liana, Inna | |||
ncsa.cv.river | Python | OpenCV (python), convert (from imagemagick), and Gdal | Linux |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/river | Liana | |||
ncsa.geo.shpExtractor | Python | gdal | Linux |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-geo/browse | Jong Lee | |||
ncsa.geo.tiffExtractor | Python | gdal | Linux |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-geo/browse | Jong Lee | |||
ncsa.image.geotiff | Python | GDAL, Cython, numpy, | Linux |
---|
Rui | https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-geotiff/browse | Rui, Mostafa Elag | ||
ncsa.image.ponddetect | Python | Matlab | Linux |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-maps/browse/feature_detection | Marcus, Ankit | |||
ncsa.image.humanpref | Python | Matlab | Linux |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-maps/browse/humanpref | Marcus, Ankit | |||
ncsa.xml.greenindexroute, ncsa.csv.greenindexroute | Python | OpenCV | Linux |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-maps/browse/greenroute | Marcus | ||
ncsa.image.knn_numerals | Python | OpenCV | Linux |
---|
Marcus | ||||
ncsa.audio.speech2text | Java | CMU Sphinx, ffmpeg, sox | Linux |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/audio/speech2text | Marcus | |
ncsa.audio.preview | Python |
---|
Inna | https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/audio/preview | |||||
ncsa.nlp.simplelanguage | Python | numpy |
---|
Inna | https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/SimpleLanguage | Liana | |||
ncsa.nlp.simplesummary | Python | Natural Language Toolkit (NLTK) for Python, NLTK Data or at least: nltk.corpus,nltk.stem.porter and nltk.tokenize.punkt. |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/SimpleSummary | Liana | ||||
ncsa.nlp.SNLPSentiment | Java | Stanford CoreNLP tool, java, maven |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/SNLP/SNLPSentimentExtractor | Liana, Marcus(?) | ||||
ncsa.nlp.wordtables | Python | requests, pika, win32com |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/WordTablesExtractor | Liana | ||||
siegfried | Python |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-siegfried/browse | Gregory Jansen | ||
ncsa.versus.image | Java | Versus | Linux |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-versus/browse | Kenton, Smruti | ||||
ncsa.image.preview (note: check if really deployed. there is an extractor in Hosted VMs list with a similar name.) | Python |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/image/preview | Rob, Sandeep |
ncsa.pdf.preview (note: check if really deployed. there is an extractor in Hosted VMs list with a similar name.) | Python |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/pdf/preview | Rob | |||||
ncsa.video.preview (note: check if really deployed. there is an extractor in Hosted VMs list with a similar name.) | Python |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/video/preview | Rob |
NOT DEPLOYED | |||||||
---|---|---|---|---|---|---|---|
ncsa.image.digitpy | Python | opencv |
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/handwritten/SimpleDigitPython | |||||||
ncsa.cv.pdfimages | pdfimages, from poppler-utils | https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/poppler | |||||
---|---|---|---|---|---|---|---|
ncsa.cv.caltech101 | Python | Matlab and VLFeat | 64-bit Mac OS | https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/vlfeat | |||
dbpedia | Python | Natural Language Toolkit (NLTK) and rdflib. | https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-dbpedia/browse | ||||
digest | Python |
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-digest/browse |
ncsa.hpc |
---|
Python |
https://opensource.ncsa. |
illinois.edu/bitbucket/projects/CATS/repos/extractors-hpc/browse | |
LSVA | Java |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
lsva/browse | Liana, Constantinos | |||
LSVA integrated |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
lsva-integrated/browse |
ncsa. |
---|
movieslice | Python |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
movieslice/browse |
Sandeep |
mri2mesh | Python |
---|
pymedici, subprocess, logging, os, numpy, shutil, zipfile |
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
mri/browse/mri2mesh |
Marcus |
msc-ChemCBCExtractor |
---|
Python | requests, |
pika, openpyxl, xlrd, pymongo | Linux |
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
msc/browse/ChemCBCExtractor |
Yan |
msc-IsletExtractor |
---|
Python |
requests, pika, openpyxl, xlrd, pymongo | Linux |
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
msc/browse/IsletExtractor |
Yan |
msc-MonitorExtractor |
---|
Python | requests, pika, openpyxl, xlrd, pymongo | Linux |
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
msc/browse/MonitorExtractor |
Yan | ||
ncsa.msc.dailymonitor | Python | requests, pika, openpyxl, xlrd, pymongo |
---|
not used | https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
msc/browse/OldMonitorExtractor |
Ashwini |
msc-PhenotypeExtractor |
---|
Python |
requests, pika, openpyxl, xlrd, pymongo | Linux |
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
msc/browse/PhenotypeExtractor |
Yan | |||||
ncsa.nlp.SNLP | Java | Stanford CoreNLP tool, java, maven |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
nlp/browse/SNLP/ |
SNLPExtractor |
Liana |
ncsa.nlp.tika |
---|
Python |
Tika project page, pymedici |
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
nlp/browse/ |
tika |
Liana |
person- |
---|
detector | Python | MATLAB, FFMPEG, requests |
---|
and pika |
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-person- |
detector/browse/ |
python |
Sandeep |
ncsa.person- |
---|
tracker | Python |
---|
python, MATLAB, FFMPEG requests and pika |
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-person- |
tracking/browse/ |
python |
Sandeep |
terra. |
---|
plantcv | Python | pika |
---|
|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
plantcv/browse |
Yan | |
medici_PTM_thumbnails | Java |
---|
requests, pika, openpyxl, xlrd, pymongo
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
ptm/browse/ |
PTMThumbnailExtractor |
Constantinos |
medici_PTM_metadata |
---|
Java |
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
ptm/browse/ |
PTMMetadataExtractor |
Constantinos | ||||||
Name not clear PtmMetadata(?) | Java | https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
---|
ptm/browse/ |
PTMMetadata | Constantinos | |||||
medici_ptm_maps | Java | https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
---|
ptm/browse/ |
PTMMapsExtractor |
Constantinos | ||||||
medici_ptm_3d | Java | https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
---|
ptm/browse/ |
PTM3DExtractor |
Constantinos | |
medici_images_ptm | Java |
---|
requests
wheel
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
ptm/browse/ImagesPTMExtractor |
Constantinos |
extractors-rabbitmq (look like examples) |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
rabbitmq/browse |
Name not clear extractors-seabird/ | Scala |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
seabird/browse |
Name not clear
Luigi |
medici_3d_x3d (one of extractors-3d |
---|
) | Java |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
3d/browse/ |
ObjJSONExtractor | Constantinos |
medici_ |
---|
3d_obj_merger (one of extractors-3d) | Java |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
3d/browse/ |
OBJMergerExtractor | Constantinos |
medici_ |
---|
oni (one of extractors-3d) | Java |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
3d/browse/ |
OniExtractor | Constantinos |
medici_ |
---|
ply_obj (one of extractors-3d) | Java |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
3d/browse/ |
PlyObjExtractor | Constantinos |
extractors-rabbitmq
(look like examples)
medici_3d_metadata (one of extractors-3d) | Java |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
3d/browse/ThreeDMetadataExtractor |
Constantinos | |
medici_x3d_html (one of extractors-3d) | Java |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
3d/browse/X3DhtmlExtractor |
Constantinos | |||||
ncsa.arcgis.landsat7mosaic | Python | ArcGIS | Windows | No |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
bd-cz/browse/ |
ndviextractor |
Smruti | ||||||
ncsa.arcgis.floodplain | Python | ArcGIS | Windows | No | https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
---|
bd-cz/browse |
/terex_floodplain/config.py | Smruti |
medici_ |
---|
book | Java |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
books/browse/ |
BookPreviewExtractor |
Theerasit Issaranon |
medici_ |
---|
image_pyramid | Java |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
books/browse/ |
ImagePreviewPyramidExtractor-shebook | Theerasit Issaranon |
shebook | Java |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors |
-books/browse/SheBookPreviewExtractor/src/BookPreviewExtractor
|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
Theerasit Issaranon | |||||
lsva-cedd | Java |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
cedd/browse |
Constantinos |
ncsa. |
---|
cinemetrics | Python |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
cinemetrics/browse |
Constantinos | |
ncsa.image.metadata | Python |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors- |
core/browse/image/ |
metadata | Max. Rob | |||||
ncsa.debod.segmentor | https://opensource.ncsa.illinois.edu/bitbucket/projects/ |
---|
DEBOD/repos/extractors- |
cellsegmentor/browse |
ncsa.image.dmp |
---|
ncsa.image.sphog.debod |
---|
https://opensource.ncsa.illinois.edu/bitbucket/projects/ |
DEBOD/repos/extractors- |
handwrittendecimals/browse |
ncsa |
---|
.image.iarp_remove_circle | https://opensource.ncsa.illinois.edu/bitbucket/projects/ |
---|
IARP/repos/image_fetcher/browse/extractors |
/ |
remove_circle |
Marcus |
ncsa. |
---|
cv. |
---|
meangrey | https://opensource.ncsa.illinois.edu/bitbucket/projects/ |
---|
IARP/repos/ |
image_fetcher/browse/ |
extractors/ |
mean_grey | Marcus |