Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.


message typetrigger eventmessage payloadexamples
*.file.#when any file is uploaded
  • added file ID
  • added filename
  • destination dataset ID, if applicable







when any file of the given MIME type is uploaded

(this is just a more specific matching)

  • added file ID
  • added filename
  • destination dataset ID, if applicable
see above
*.dataset.file.addedwhen a file is added to a dataset
  • added file ID
  • dataset ID
  • full list of files in dataset
*.dataset.file.removedwhen a file is removed from a dataset
  • removed file ID
  • dataset ID
  • full list of files in dataset
*.metadata.addedwhen metadata is added to a file or dataset
  • file or dataset ID
  • the metadata that was added
*.metadata.removedwhen metadata is removed from a file or dataset
  • file or dataset ID


Typical extractor structure


Code Block
cd /etc/init
for x in clowder-*.conf; do
  start `basename $x .conf`


Converting from pyClowder to pyClowder2


  1. If there are parameters in that don't use the default values in the link under Key differences, they should be listed as command line parameters in your new extractor class __init__ or simply coded into the script. It's possible to make the parameters read from environment variables as well.
    1. - in this example, 
      1. is moved to
    2. ...and below the self.parser.add_argument the input values can be adjusted before assinging to self.args (e.g. cast a string to an int):
    3. Add the messageType from into extractor_info.json
      1. Before:
      2. After: 
  2. Move your extractor python functions into a new Extractor subclass 
    1. Here a new extractor class called StereoBin2JpgTiff is created.
    2. main() -> __init__(self) (but only for handling inputs)
    3. check_message() and process_message() must be named as such now, and receive explicit inputs:
      1. self, connector, host, secret_key, resource, parameters
      2. typically, old references to parameters['xyz'] can be replaced either with resource['xyz'] or with secret_key, host, etc.
      3. if you aren't sure when writing, you can use print(resource) in your extractor testing to see what fields are included.
  3. Modify old extractor.method() to use the new pyclowder.files.method() or pyclowder.datasets.method()
    6. more to come
  4. finally, the call to main() is replaced with a simple instantiation of your extractor class.
    2. extractor = StereoBin2JpgTiff(); extractor.start()

Registering an Extractor

If you create an extractor and want to register it for widespread use Clowder has a catalog of extractors. To let the community know about your extractor you may submit it to the catalog (registration required) for publication alongside the extractors created by the Clowder developers, and other Clowder community developers.

  1. If you haven't registered you may do so at the registration page (to be linked soon). Registration will need to be approved, so this will not be an instantaneous process. Once approved you will be able to sign in, and submit extractors.
  2. Sign in to the catalog at the sign in page.
  3. Click the Contribute link at the top, or visit the contribution page.
  4. Paste your extractor info into the text box on the contribution page. The extractor info has all the information that we use in the catalog, so having a fully detailed extractor info is important for the catalog.
  5. Select Extractor from the radio buttons under the text box (Converter is for converting one file type to another, and there may be more options added for other tools at a later date).
  6. Submit the extractor.
  7. On the home page you may not see your extractor immediately. Once submitted it must be approved by one of the Clowder development team to show up on the home page. As long as your links to where to get the extractor check out you will likely be approved, and if there are correctable issues we may contact you.