...
A user wishes to convert and then extract some statistics on a locally hosted 8 GB weather dataset. The user is using the Brown Dog command line interface to do this. Below we outline the modifications needed by each of the components in order to allow these transformations to be carried out locally (i.e. moving the compute to the local data).
DTS (Clowder)
- An endpoint (see this page for more details) needs to be created either in Clowder or in Fence which returns the list of docker containers containing each extractor that would be fired off given a file/dataset of a given type. Parameters will be the mime type (requiring the client to determine this) or the file extension (requiring the server side to fill in the full mime type for Clowder).
- A minor modification will be needed to the extractors such that if the extractor is executed with a single argument (i.e. the path to the locally stored file to operate on) the extractor will bypass connecting to rabbitmq, use the argument as the input, run on it, print the generated JSON to the screen, and immediately exit. Might be possible to add a single method to pyClowder to handle this overall (will need to explore).
- The default entrypoint.sh file for the dockerized extractor will need to be modified to allow for an optional argument to be passed in.
...