The ability to track extractors by space is already in master. Go to a project space and click on "Extractors."

Next we need to send the message over rabbitmq to that specific extractor. Since we have the ability to do this now by extractors this part should be easy.

Questions


Clowder and RabbitMQ


DISCUSSION DRAFT

 Following are some steps that we can do to create a per space extractor. The thinking of this list is that this is path to implement.

  1. List of extractors is kept per space.
  2. List of bindings is kept in clowder, removes exchanges
  3. List of extractors is queried
    1. Extractors will bind themselves to a queue and have a command queue. The command queue is non-persistant. Each extractor will pick up messages from both queues, however give preference to the command queue. Clowder will send a message to the extractor exchange and send a command message (cmd) that is picked up by all command queues, allowing them to register them than with that specific clowder instance. All extractor.* messages will go the normal extractor queue.
    2. Can easily get a list of all extractors and refresh this list every 15 or so minutes.
  4. More complex logic for extractors
    1. Extractors can now have more complex logic, such as (file added, mimetype=image/jpg, filesize>5MB). This is part of the extractor_info.json, clowder can use this more complex logic to see if an extractor should fire.