Extractors are services that run silently alongside Clowder. They can be configured to wait for specific file types to be uploaded into Clowder, and automatically execute scripts and processing on those files to extract metadata or generate new files.
In order to develop and test an extractor, it's useful to have a local instance of Clowder running that you can test against. This will allow you to upload target files to trigger your extractor, and verify any outputs are being submitted back into Clowder correctly.
The easiest way to get a local Clowder instance up and running is via Docker. We have created a docker image with the full Clowder stack already installed:
It's possible to install these elements individually, but unless you want to pursue Clowder development this is unnecessary.
docker-compose up
. This should start Clowder and its components.docker network ls
and ifconfig
to determine Docker's IP address. docker-machine ip
.<dockerIP>:9000
.<dockerIP>:15672
.docker ps
to list running Docker containersdocker logs <clowder container name>
to see the logs for your Clowder containerIn addition to Clowder, another package that makes extractor development easier is pyClowder 2. This package is not required - extractors can be written in any language that RabbitMQ supports.
pyClowder 2 provides helpful Python wrapper functions for many standard Clowder API calls and communication with RabbitMQ using pika. If you want to write extractors in other languages, you'll need to implement that functionality on your own.
git clone https://opensource.ncsa.illinois.edu/bitbucket/scm/cats/pyclowder2.git
python setup.py install
from inside the new pyclowder2 directoryThis will install pyClowder 2 on your system. In Python scripts you will refer to the package as:
import pyclowder
from pyclowder.extractors import Extractor
import pyclowder.files
...etc.
Extractors use the RabbitMQ message bus to communicate with Clowder instances. Queues are created for each extractor, and the queue bindings filter the types of Clowder event messages the extractor is notified about. The following non-exhaustive list of events exist in Clowder (messages begin with an asterisk because the exchange name is not required to be 'clowder'):
message type | trigger event | message payload | examples |
---|---|---|---|
*.file.# | when any file is uploaded |
| clowder.file.image.png clowder.file.text.csv clowder.file.application.json |
*.file.image.# *.file.text.# ... | when any file of the given MIME type is uploaded (this is just a more specific matching) |
| see above |
*.dataset.file.added | when a file is added to a dataset |
| clowder.dataset.file.added |
*.dataset.file.removed | when a file is removed from a dataset |
| clowder.dataset.file.removed |
*.metadata.added | when metadata is added to a file or dataset |
| clowder.metadata.added |
*.metadata.removed | when metadata is removed from a file or dataset |
| clowder.metadata.removed |
sudo -s export RABBITMQ_URL="amqp://guest:guest@localhost:5672/%2F" export EXTRACTORS_HOME="/home/clowder" apt-get -y install git python-pip pip install pika requests cd ${EXTRACTORS_HOME} git clone https://opensource.ncsa.illinois.edu/stash/scm/cats/pyclowder.git chown -R clowder.users pyclowder |
apt-get -y install python-opencv opencv-data cd ${EXTRACTORS_HOME} git clone https://opensource.ncsa.illinois.edu/stash/scm/cats/extractors-cv.git for x in opencv-closeups opencv-eyes opencv-faces opencv-profiles; do ln -s ${EXTRACTORS_HOME}/pyclowder/pyclowder ${EXTRACTORS_HOME}/extractors-cv/opencv/$x sed -i -e "s#rabbitmqURL = .*#rabbitmqURL = '${RABBITMQ_URL}'#" \ -e "s#/usr/local/share/OpenCV#/usr/share/opencv#" ${EXTRACTORS_HOME}/extractors-cv/opencv/$x/config.py cp ${EXTRACTORS_HOME}/extractors-cv/opencv/$x/*.conf /etc/init done chown -R clowder.users extractors-cv |
apt-get -y install tesseract-ocr cd ${EXTRACTORS_HOME} git clone https://opensource.ncsa.illinois.edu/stash/scm/cats/extractors-cv.git ln -s ${EXTRACTORS_HOME}/pyclowder/pyclowder ${EXTRACTORS_HOME}/extractors-cv/ocr sed -i -e "s#rabbitmqURL = .*#rabbitmqURL = '${RABBITMQ_URL}'#" ${EXTRACTORS_HOME}/extractors-cv/ocr/config.py cp ${EXTRACTORS_HOME}/extractors-cv/ocr/clowder-ocr.conf /etc/init chown -R clowder.users pyclowder |
apt-get -y install sox libsox-fmt-mp3 cd ${EXTRACTORS_HOME} git clone https://opensource.ncsa.illinois.edu/stash/scm/cats/extractors-audio.git ln -s ${EXTRACTORS_HOME}/pyclowder/pyclowder ${EXTRACTORS_HOME}/extractors-audio/preview/ sed -i -e "s#Binary = .*#Binary = '`which sox`'#" -e "s#rabbitmqURL = .*#rabbitmqURL = '${RABBITMQ_URL}'#" extractors-audio/preview/config.py cp ${EXTRACTORS_HOME}/extractors-audio/preview/clowder-audio-preview.conf /etc/init chown -R clowder.users extractors-audio |
apt-get -y install imagemagick cd ${EXTRACTORS_HOME} git clone https://opensource.ncsa.illinois.edu/stash/scm/cats/extractors-image.git ln -s ${EXTRACTORS_HOME}/pyclowder/pyclowder ${EXTRACTORS_HOME}/extractors-image/preview/ sed -i -e "s#imageBinary = .*#imageBinary = '`which convert`'#" -e "s#rabbitmqURL = .*#rabbitmqURL = '${RABBITMQ_URL}'#" extractors-image/preview/config.py cp ${EXTRACTORS_HOME}/extractors-image/preview/clowder-image-preview.conf /etc/init chown -R clowder.users extractors-image |
apt-get -y install imagemagick cd ${EXTRACTORS_HOME} git clone https://opensource.ncsa.illinois.edu/stash/scm/cats/extractors-pdf.git ln -s ${EXTRACTORS_HOME}/pyclowder/pyclowder /home/clowder/extractors-pdf/preview/ sed -i -e "s#Binary = .*#Binary = '`which convert`'#" -e "s#rabbitmqURL = .*#rabbitmqURL = '${RABBITMQ_URL}'#" extractors-pdf/preview/config.py cp ${EXTRACTORS_HOME}/extractors-pdf/preview/clowder-pdf-preview.conf /etc/init chown -R clowder.users extractors-pdf |
apt-get -y install libav-tools cd /home/clowder git clone https://opensource.ncsa.illinois.edu/stash/scm/cats/extractors-video.git ln -s /home/clowder/pyclowder/pyclowder /home/clowder/extractors-video/preview/ sed -i -e "s#Binary = .*#Binary = '`which convert`'#" -e "s#rabbitmqURL = .*#rabbitmqURL = '${RABBITMQ_URL}'#" extractors-video/preview/config.py cp /home/clowder/extractors-video/preview/clowder-video-preview.conf /etc/init chown -R clowder.users extractors-video |
cd /etc/init for x in clowder-*.conf; do start `basename $x .conf` done |