Page tree
Skip to end of metadata
Go to start of metadata

This page is for the refactoring of the existing extractors. The original wiki page Hosted VMs is still used for the deployments. 

As we figure out who's working on what, please start with the following steps for the extractor(s) you chose:

Steps to take for every extractor in this list:

  1. Docker containers
  2. JSONLD
  3. Extractor info registration
  4. Use pyclowder (for python extractors)
  5. Add status messages to all extractors and fix level granularity
    1. Make status constants (DONE, ERROR)
    2. Arcgis multiprocessing extractor
  6. Register on on demand execution queues
    1. Add on demand key binding to configuration file: messageType = "*.file.text.plain", "extractors."+extractorName
  7. Standardize around python logging
    1. Figure out what to log and what format to follow
  8. Add logstash to docker compose
  9. Add sample input/ouput to git repository
  10. Add icon for tools catalog to git repository
  11. Add entry to Tools catalog, with icon

 

ID (Extractor Name from config file, same as queue name)

Programming Language

SoftwareOSCan be Dockerized?Assigned ToRepoAuthor
DEPLOYED
ncsa.image.ocrPythonTesseractLinux Ruihttps://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/ocr 

ncsa.cv.faces

PythonOpenCVLinux Ruihttps://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencvLiana

ncsa.cv.eyes

PythonOpenCVLinux Ruihttps://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencvLiana

ncsa.cv.closeups

PythonOpenCVLinux Ruihttps://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencvLiana

ncsa.cv.profiles

PythonOpenCVLinux Ruihttps://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/opencvLiana

ncsa.cellprofiler.fluorescentcomet

Pythonpymedici (question)WindowsNo https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofilerLiana

ncsa.cellprofiler.fly

Python WindowsNo https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofilerLiana

ncsa.cellprofiler.human

Python WindowsNo https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofilerLiana

ncsa.cellprofiler.silvercomet

Python WindowsNo https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofilerLiana

ncsa.cellprofiler.speckle

Python WindowsNo https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofilerLiana

ncsa.cellprofiler.trackobject

Python WindowsNo https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofilerLiana

ncsa.cellprofiler.tumor

Python WindowsNo https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofilerLiana

ncsa.cellprofiler.yeast

Python WindowsNo https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/cellprofilerLiana

ncsa.image.sphog

Python Matlab, mnist-sphog Linux https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/handwritten/HandwrittenNumbers 

ncsa.image.caltech101

      
ncsa.bisque.histogram (notes: disabled)Python Linux    
ncsa.bisque.metadata (notes: disabled)Python Linux    
census-section-segmentorJava Linux https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/censusLiana, Inna
ncsa.cv.river PythonOpenCV (python), convert (from imagemagick), and GdalLinux https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/riverLiana
ncsa.geo.shpExtractorPythongdalLinux https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-geo/browseJong Lee
ncsa.geo.tiffExtractorPythongdalLinux https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-geo/browseJong Lee
ncsa.image.geotiffPython

GDAL, Cython, numpy,
pygeoprocessing

Linux Ruihttps://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-geotiff/browseRui, Mostafa Elag

ncsa.image.ponddetect

PythonMatlabLinux https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-maps/browse/feature_detectionMarcus, Ankit
ncsa.image.humanprefPythonMatlabLinux https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-maps/browse/humanprefMarcus, Ankit

ncsa.xml.greenindexroute, ncsa.csv.greenindexroute

PythonOpenCVLinux https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-maps/browse/greenrouteMarcus

ncsa.image.knn_numerals

PythonOpenCVLinux  Marcus

ncsa.audio.speech2text

JavaCMU Sphinx, ffmpeg, soxLinux https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/audio/speech2textMarcus
ncsa.audio.previewPython   Innahttps://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/audio/preview 
ncsa.nlp.simplelanguagePythonnumpy  Innahttps://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/SimpleLanguageLiana
ncsa.nlp.simplesummaryPython

Natural Language Toolkit (NLTK) for Python, NLTK Data or at least:

 nltk.corpus,nltk.stem.porter and nltk.tokenize.punkt.

  https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/SimpleSummaryLiana
ncsa.nlp.SNLPSentimentJava Stanford CoreNLP tool, java, maven  https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/SNLP/SNLPSentimentExtractorLiana, Marcus(?)
ncsa.nlp.wordtablesPython requestspikawin32com   https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/WordTablesExtractorLiana
siegfriedPython    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-siegfried/browseGregory Jansen
ncsa.versus.imageJavaVersusLinux https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-versus/browseKenton, Smruti
ncsa.image.preview (note: check if really deployed. there is an extractor in Hosted VMs list with a similar name.)Python    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/image/previewRob, Sandeep
ncsa.pdf.preview (note: check if really deployed. there is an extractor in Hosted VMs list with a similar name.)Python    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/pdf/previewRob
ncsa.video.preview (note: check if really deployed. there is an extractor in Hosted VMs list with a similar name.)Python    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/video/previewRob
NOT DEPLOYED
ncsa.image.digitpyPythonopencv   https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/handwritten/SimpleDigitPython 
ncsa.cv.pdfimages pdfimages, from poppler-utils   https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/poppler 
ncsa.cv.caltech101PythonMatlab and VLFeat 64-bit Mac OS   https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cv/browse/vlfeat 
dbpediaPython Natural Language Toolkit (NLTK) and rdflib.  https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-dbpedia/browse
digestPython    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-digest/browse 
ncsa.hpcPython    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-hpc/browse
LSVAJava    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-lsva/browseLiana, Constantinos
LSVA integrated    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-lsva-integrated/browse
ncsa.movieslicePython    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-movieslice/browseSandeep
mri2meshPythonpymedici, subprocess, logging, os, numpy, shutil, zipfile   https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-mri/browse/mri2meshMarcus
msc-ChemCBCExtractorPythonrequests, pika, openpyxl, xlrd, pymongoLinux https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-msc/browse/ChemCBCExtractorYan
msc-IsletExtractorPythonrequests, pika, openpyxl, xlrd, pymongoLinux https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-msc/browse/IsletExtractorYan
msc-MonitorExtractorPythonrequests, pika, openpyxl, xlrd, pymongoLinux https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-msc/browse/MonitorExtractorYan
ncsa.msc.dailymonitorPythonrequests, pika, openpyxl, xlrd, pymongo  not usedhttps://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-msc/browse/OldMonitorExtractorAshwini
msc-PhenotypeExtractorPython

requests, pika, openpyxl, xlrd, pymongo

Linux https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-msc/browse/PhenotypeExtractorYan
ncsa.nlp.SNLPJava Stanford CoreNLP tool, java, maven   https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/SNLP/SNLPExtractorLiana
ncsa.nlp.tikaPython Tika project page, pymedici  https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-nlp/browse/tikaLiana
person-detectorPython MATLAB, FFMPEG, requests and pika   https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-person-detector/browse/pythonSandeep
ncsa.person-trackerPythonpython, MATLAB, FFMPEG requests and pika   https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-person-tracking/browse/pythonSandeep
terra.plantcvPython

pika
requests
wheel

   https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-plantcv/browseYan
medici_PTM_thumbnailsJava    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-ptm/browse/PTMThumbnailExtractorConstantinos
medici_PTM_metadataJava    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-ptm/browse/PTMMetadataExtractorConstantinos

Name not clear

PtmMetadata(?)

Java    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-ptm/browse/PTMMetadataConstantinos
medici_ptm_mapsJava    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-ptm/browse/PTMMapsExtractorConstantinos
medici_ptm_3dJava    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-ptm/browse/PTM3DExtractorConstantinos
medici_images_ptmJava    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-ptm/browse/ImagesPTMExtractorConstantinos

extractors-rabbitmq

(look like examples)

     https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-rabbitmq/browse 
Name not clear extractors-seabird/Scala    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-seabird/browseLuigi
medici_3d_x3d (one of extractors-3d)Java    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-3d/browse/ObjJSONExtractorConstantinos
medici_3d_obj_merger (one of extractors-3d)Java    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-3d/browse/OBJMergerExtractorConstantinos
medici_oni (one of extractors-3d)Java    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-3d/browse/OniExtractorConstantinos
medici_ply_obj (one of extractors-3d)Java    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-3d/browse/PlyObjExtractorConstantinos
medici_3d_metadata (one of extractors-3d) Java    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-3d/browse/ThreeDMetadataExtractorConstantinos
medici_x3d_html (one of extractors-3d) Java    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-3d/browse/X3DhtmlExtractorConstantinos
ncsa.arcgis.landsat7mosaicPythonArcGISWindowsNohttps://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-bd-cz/browse/ndviextractorSmruti
ncsa.arcgis.floodplainPythonArcGISWindowsNohttps://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-bd-cz/browse/terex_floodplain/config.pySmruti
medici_bookJava    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-books/browse/BookPreviewExtractorTheerasit Issaranon
medici_image_pyramidJava    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-books/browse/ImagePreviewPyramidExtractor-shebookTheerasit Issaranon
shebookJava    

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-books/browse/SheBookPreviewExtractor/src/BookPreviewExtractor

 

https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-books/browse/SheBookPreviewExtractor/src/bookpreviewextractor

Theerasit Issaranon
lsva-ceddJava    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cedd/browseConstantinos
ncsa.cinemetricsPython    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-cinemetrics/browseConstantinos
ncsa.image.metadataPython    https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-core/browse/image/metadataMax. Rob
ncsa.debod.segmentor     https://opensource.ncsa.illinois.edu/bitbucket/projects/DEBOD/repos/extractors-cellsegmentor/browse 
ncsa.image.dmp     

https://opensource.ncsa.illinois.edu/bitbucket/projects/DEBOD/repos/extractors-debod/browse

https://opensource.ncsa.illinois.edu/bitbucket/projects/DEBOD/repos/extractors-dmp/browse

 
ncsa.image.sphog.debod     https://opensource.ncsa.illinois.edu/bitbucket/projects/DEBOD/repos/extractors-handwrittendecimals/browse 

ncsa.image.iarp_remove_circle

     https://opensource.ncsa.illinois.edu/bitbucket/projects/IARP/repos/image_fetcher/browse/extractors/remove_circleMarcus
ncsa.cv.meangrey     https://opensource.ncsa.illinois.edu/bitbucket/projects/IARP/repos/image_fetcher/browse/extractors/mean_greyMarcus
  • No labels

2 Comments

  1. Other things to add maybe:

    • queue used
    • deployed dts on server XYZ
    • deployed dts-dev on server UVW
  2. This page is just for the refactoring of the existing extractors. The original wiki page is still used for the deployments. Maybe we should add this information that page?

    Hosted VMs