...
| Environment Variable | Command-Line Flag | Default Value | Description |
|---|---|---|---|
ARCHIVE_SOURCE_DIRECTORY | --archive-source | $HOME/clowder/data/uploads/ | The current directory where Clowder stores it's uploaded files |
ARCHIVE_TARGET_DIRECTORY | --archive-target | $HOME/clowder/data/archive/ | The target directory where the archival extractor should store the files that it archives. Note that this path can be on a network or other persistent storage. |
To build the Disk archival extractor's Docker image, execute the following commands:
| Code Block | ||
|---|---|---|
| ||
git clone https://opensource.ncsa.illinois.edu/bitbucket/scm/cats/extractors-archival-disk.git
cd extractors-archival-disk/
docker build -t clowder/extractors-archival-disk . |
Example Configuration: Archive to another folder
In Clowder, configure the following:
| Code Block |
|---|
# storage driver service.byteStorage=services.filesystem.DiskByteStorageService # disk storage path #clowder.diskStorage.path="C:/Users/lambert8/clowder/data" # Win10MacOSX #clowderclowder.diskStorage.path="/Usershome/lambert8clowder/clowder/data" # MacOSX clowder.diskStorage.path # Linux |
To run the Disk archival extractor with this configuration:
| Code Block |
|---|
docker run -itd --rm -e ARCHIVE_SOURCE_DIRECTORY="/home/clowder/clowder/data/uploads/" -e # LinuxARCHIVE_TARGET_DIRECTORY="/home/clowder/clowder/data/archive/" clowder/extractors-archival-disk |
NOTE: on MacOSX, you may need to run the extractor with the --net=host option to connect to RabbitMQ
Configuration Options: ncsa.archival.s3
...
| Environment Variable | Command-Line Flag | Default Value | Description |
|---|---|---|---|
AWS_S3_SERVICE_ENDPOINT | --service-endpoint <value> | https://s3.amazonaws.com | Which AWS Service Endpoint to use to connect to S3. Note that this may depend on the region used, but can also be used to point at a running MinIO instance. |
AWS_ACCESS_KEY | --access-key <value> | "" | The AccessKey that should be used to authorize with AWS or MinIO |
AWS_SECRET_KEY | --secret-key <value> | "" | The SecretKey that should be used to authorize with AWS or MinIO |
AWS_BUCKET_NAME | --bucket-name <value> | clowder-archive | The name of the bucket where the files are stored in Clowder. |
AWS_REGION | --region <value> | us-east-1 | AWS only: the region where the S3 bucket exists |
To build the S3 archival extractor's Docker image, execute the following commands:
| Code Block | ||
|---|---|---|
| ||
git clone https://opensource.ncsa.illinois.edu/bitbucket/scm/cats/extractors-archival-s3.git
cd extractors-archival-s3/
docker build -t clowder/extractors-archival-s3 . |
Example Configuration: S3 on AWS in us-east-2 Region
In Clowder, configure the following:
| Code Block |
|---|
# AWS S3 clowder.s3.serviceEndpoint="https://s3-us-east-2.amazonaws.com" clowder.s3.accessKey="AWSACCESSKEYKASOKD" clowder.s3.secretKey="aWSseCretKey+asAfasf90asdASDADAOaisdoas" clowder.s3.bucketName="bucket-on-aws" clowder.s3.region="us-east-2" |
NOTE: Changing the Region also typically requires changing the S3 Service Endpoint.
To run the S3 archival extractor with this configuration:
| Code Block |
|---|
docker run --net=host -it --rm -e AWS_S3_SERVICE_ENDPOINT='https://s3-us-east-2.amazonaws.com' -e AWS_ACCESS_KEY='AWSACCESSKEYKASOKD' -e AWS_SECRET_KEY='aWSseCretKey+asAfasf90asdASDADAOaisdoas' -e AWS_BUCKET_NAME='bucket-on-aws' -e AWS_REGION='us-east-2' clowder/extractors-archival-s3 |
NOTE: on MacOSX, you may need to run the extractor with the --net=host option to connect to RabbitMQ
Example Configuration: MinIO
In Clowder, configure the following to point the S3ByteStorageDriver and the archival extractor at your running MinIO instance:
| Code Block |
|---|
# Minio S3 clowder.s3.serviceEndpoint="http://localhost:8000" clowder.s3.accessKey="AMINIOACCESSKEYKASOKD" clowder.s3.secretKey="aMinIOseCretKey+asAfasf90asdASDADAOaisdoas" clowder.s3.bucketName="bucket-on-minio" |
NOTE: MinIO does not use the value for "Region", if one was specified.
To run the S3 archival extractor with this configuration:
| Code Block |
|---|
docker run --net=host -it --rm -e AWS_S3_SERVICE_ENDPOINT='http://localhost:8000' -e AWS_ACCESS_KEY='AMINIOACCESSKEYKASOKD' -e AWS_SECRET_KEY='aMinIOseCretKey+asAfasf90asdASDADAOaisdoas' -e AWS_BUCKET_NAME='bucket-on-minio' clowder/extractors-archival-s3 |
NOTE: on MacOSX, you may need to run the extractor with the --net=host option to connect to RabbitMQPoint the S3ByteStorageDriver and the archival extractor at your running MinIO instance.
Process Overview
When a file is first uploaded, it is placed into a temp folder and created in the DB with the state CREATED.
...