...
If neither of the above two extractors fit your use case, pyclowder can be used to quickly create a new archival extractor that fits your needs.
| Table of Contents |
|---|
Configuration Options / Defaults for Clowder
...
| Environment Variable | Command-Line Flag | Default Value | Description |
|---|---|---|---|
ARCHIVE_SOURCE_DIRECTORY | --archive-source | $HOME/clowder/data/uploads/ | The current directory where Clowder stores it's uploaded files |
ARCHIVE_TARGET_DIRECTORY | --archive-target | $HOME/clowder/data/archive/ | The target directory where the archival extractor should store the files that it archives. Note that this path can be on a network or other persistent storage. |
Example Configuration: Archive to another folder
| Code Block |
|---|
# disk storage #clowder.diskStorage.path="C:/Users/lambert8/clowder/data" # Win10 #clowder.diskStorage.path="/Users/lambert8/clowder/data" # MacOSX clowder.diskStorage.path="/home/clowder/clowder/data" # Linux |
Configuration Options: ncsa.archival.s3
...
| Environment Variable | Command-Line Flag | Default Value | Description |
|---|---|---|---|
AWS_S3_SERVICE_ENDPOINT | --service-endpoint <value> | https://s3.amazonaws.com | Which AWS Service Endpoint to use to connect to S3. Note that this may depend on the region used, but can also be used to point at a running MinIO instance. |
AWS_ACCESS_KEY | --access-key <value> | "" | The AccessKey that should be used to authorize with AWS or MinIO |
AWS_SECRET_KEY | --secret-key <value> | "" | The SecretKey that should be used to authorize with AWS or MinIO |
AWS_BUCKET_NAME | --bucket-name <value> | clowder-archive | The name of the bucket where the files are stored in Clowder. |
AWS_REGION | --region <value> | us-east-1 | AWS only: the region where the S3 bucket exists |
Example Configuration: S3 on AWS in us-east-2 Region
| Code Block |
|---|
# AWS S3
clowder.s3.serviceEndpoint="https://s3-us-east-2.amazonaws.com"
clowder.s3.accessKey="AWSACCESSKEYKASOKD"
clowder.s3.secretKey="aWSseCretKey+asAfasf90asdASDADAOaisdoas"
clowder.s3.bucketName="bucket-on-aws"
clowder.s3.region="us-east-2" |
NOTE: Changing the Region also requires changing the S3 Service Endpoint.
Example Configuration: MinIO
| Code Block |
|---|
# Minio S3
clowder.s3.serviceEndpoint="http://localhost:8000"
clowder.s3.accessKey="AMINIOACCESSKEYKASOKD"
clowder.s3.secretKey="aMinIOseCretKey+asAfasf90asdASDADAOaisdoas"
clowder.s3.bucketName="bucket-on-minio" |
Point the S3ByteStorageDriver and the archival extractor at your running MinIO instance.
Process Overview
When a file is first uploaded, it is placed into a temp folder and created in the DB with the state CREATED.
...
The extractor performs whatever operation it deems as "archiving" - for example, copying to a network filesystemfile system.
Finally the file is marked as ARCHIVED, and (if configured) the user is given the option to Unarchive the file.
...
By default, files that are over 100MB 1MB and have not been downloaded in that last 90 days will be automatically archived.
...