...
Code Block | ||
---|---|---|
| ||
cd /etc/init for x in clowder-*.conf; do start `basename $x .conf` done |
Converting from pyClowder to pyClowder2
Given an extractor that is written to use pyClowder 1, the process of migrating to pyClowder 2 is fairly straightforward.
Key differences
- config.py is no longer used or needed.
- Several of the common entries in config.py are accessible to all extractors via the basic Extractor class: https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder2/browse/pyclowder/extractors.py#66 (here you can also see defaults)
- You can implement your own command line arguments to include any special parameters in config.py. Another option is to read them from environment variables.
- the 'messageType' parameter (telling what types of messages to listen for) will go into extractor_info.json and uses a a more MIME-like definition format.
- your extractor will now be an extension of pyClowder2's Extractor class, which contains many useful methods.
- init is where you can define custom command line arguments beyond the standard ones.
- check_message and process_message now get explicit parameters such as clowder host and secret key, rather than embedding them in a 'params' object. information about the entity in clowder that triggered the extraction (file, dataset, etc.) is in the 'resource' parameter. The old 'parameters' is kept for back compatibility, but is deprecated.
- As a result of config.py going away, you should provide parameters at runtime
- python my_extractor.py --rabbitmqExchange="terra" --rabbitmqURI="rabbitmw.ncsa.illinois.edu/clowder-dev"
- new cleaner functions in pyClowder 2 for interacting with clowder, including packages for files, datasets, etc.
- OLD - extractors.upload_file_to_dataset(outfile, parameters)
- NEW - pyclowder.files.upload_to_dataset(connector, host, secret_key, resource['id'], outfile)
Migration steps
- If there are parameters in config.py that don't use the default values in the link under Key differences, they should be listed as command line parameters in your new extractor class __init__ or simply coded into the script. It's possible to make the parameters read from environment variables as well.
- https://github.com/terraref/extractors-stereo-rgb/pull/3/files - in this example,
- ...and below the self.parser.add_argument the input values can be adjusted before assinging to self.args (e.g. cast a string to an int):
- Add the messageType from config.py into extractor_info.json
- Move your extractor python functions into a new Extractor subclass
- https://github.com/terraref/extractors-stereo-rgb/pull/3/files#diff-924a575b0595fcd52d5531433471b109R23 Here a new extractor class called StereoBin2JpgTiff is created.
- main() -> __init__(self) (but only for handling inputs)
- check_message() and process_message() must be named as such now, and receive explicit inputs:
- self, connector, host, secret_key, resource, parameters
- typically, old references to parameters['xyz'] can be replaced either with resource['xyz'] or with secret_key, host, etc.
- if you aren't sure when writing, you can use print(resource) in your extractor testing to see what fields are included.
- Modify old extractor.method() to use the new pyclowder.files.method() or pyclowder.datasets.method()
- https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder2/browse/pyclowder/files.py
- https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder2/browse/pyclowder/datasets.py
- https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder2/browse/pyclowder/collections.py
- https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder2/browse/pyclowder/sections.py
- https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder2/browse/pyclowder/utils.py
- more to come
- https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder2/browse/pyclowder/files.py
- finally, the call to main() is replaced with a simple instantiation of your extractor class.
- https://github.com/terraref/extractors-stereo-rgb/pull/3/files#diff-924a575b0595fcd52d5531433471b109R174
- extractor = StereoBin2JpgTiff(); extractor.start()