Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The second architecture is to create a Box client application that will interact with Brown Dog using pdbd.py.

draw.io Diagram
bordertrue
viewerToolbartrue
fitWindowfalse
diagramNameBoxSkillsWithBD.py
simpleViewerfalse
widthdiagramWidth899
revision1

Comparison of Approaches

Box in Pyclowderbd.py

File is downloaded once from box to the extractor container

File is deleted at end of process_message

File

If using the /extractions/file endpoint the file is transferred three times:

  1. From Box to BoxClient
  2. From BoxClient to Clowder (via fence)
  3. From Clowder to extractor container

If using the /extractions/url endpoint the file is trasferred ?? times:

  1. ??

File lives in clowder until the cleanup script is run

Box SDK has to be introduced into Pyclowder library. Any other repos we want to support would also have to be included.Box SDK only lives in the BoxClient service. No changes are required for pyclowder
Custom metadata structure for box would be implemented in the extractor.An automated translation of clowder metadata to box skills cards would have to be developed.

Potential Bottlenecks for massive scaling:

  1. Fence
  2. RabbitMQ
  3. Extractors

Notes:

Everything apart from Rabbit is stateless and can be horizontally scaled.

Potential Bottlenecks for massive scaling:

  1. Fence
  2. RabbitMQ
  3. Extractors
  4. BoxClient
  5. Clowder
  6. MongoDB

Notes:

We can't rely on threading in the BoxClient to do the polling since we would risk running out of threads.

Would need to add some endpoints to fence

Would need to deploy a new service and proxy it behind Apache.

BoxClient would need to be allocated a service account and handle Brown Dog tokens

Limited error logging and reporting

Unsure about retry if an extraction fails

Eventually we could create an app where user logs in via their Box credentials to see the history of extractions

Errors can be reported in Clowder (not visible to the Box user)

The BoxClient could potentially retry


Skills Invocation

When a file is uploaded to a Box folder that has an attached skill, the following payload is POSTed to the fence endpoint:

...