This page outlines an approach for Brown Dog to become the definitive source for scientific community curated box skills.
Architecture
There are two possible architectures:
The first one is to enhance pyclowder to enable it to download files and upload metadata to Box in addition to clowder. Eventually support would be added to Google Drive and Dataverse, etc.
| draw.io Diagram | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
When a Skill is registered with a Box account, the invocation URL is provided. This URL will resolve to an endpoint in Fence.
The second architecture is to create a Box client application that will interact with Brown Dog using pd.py.
| draw.io Diagram | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Comparison of Approaches
| Box in Pyclowder | bd.py |
|---|---|
File is downloaded once from box to the extractor container File is deleted at end of process_message | File is transferred three times:
File lives in clowder until the cleanup script is run |
| Box SDK has to be introduced into Pyclowder library. Any other repos we want to support would also have to be included. | Box SDK only lives in the BoxClient service. No changes are required for pyclowder |
| Custom metadata structure for box would be implemented in the extractor. | An automated translation of clowder metadata to box skills cards would have to be developed. |
Potential Bottlenecks for massive scaling:
Notes: Everything apart from Rabbit is stateless and can be horizontally scaled. | Potential Bottlenecks for massive scaling:
Notes: We can't rely on threading in the BoxClient to do the polling since we would risk running out of threads. |
| Would need to add some endpoints to fence | Would need to deploy a new service and proxy it behind Apache. BoxClient would need to be allocated a service account and handle Brown Dog tokens |
Limited error logging and reporting Unsure about retry if an extraction fails Eventually we could create an app where user logs in via their Box credentials to see the history of extractions | Errors can be reported in Clowder (not visible to the Box user) The BoxClient could potentially retry |
Skills Invocation
When a file is uploaded to a Box folder that has an attached skill, the following payload is POSTed to the fence endpoint:
...