...
| Configuration Path | Default | Description |
|---|---|---|
| false | If true, Clowder should perform a lookup once per day to see if any files uploaded the past hour are candidates for archival. |
archiveDebug | false | If true, Clowder should temporarily use "5 minutes" as the archive check interval (instead of once per day). In addition, it only considers candidate files that were uploaded in the past hour. |
archiveExtractorId | "ncsa.archival.disk" | The id of the Extractor to use for archival |
archiveAllowUnarchive | false | If true, the UI should offer a way to Unarchive a file that is ARCHIVED |
archiveAutoAfterDaysInactive | 90 | The number of days that an item can go without being downloaded before it is automatically archived. |
archiveMinimumStorageSize | 1000000 | The minimum number of bytes for a file to be considered as a candidate for automatic archival. |
...
ncsa.archival.disk
To build the Disk archival extractor's Docker image, execute the following commands:
| Code Block | ||
|---|---|---|
| ||
git clone https://opensource.ncsa.illinois.edu/bitbucket/scm/cats/extractors-archival-disk.git
cd extractors-archival-disk/
docker build -t clowder/extractors-archival-disk . |
Configuration Options
The following configuration options must match your configuration of the DiskByteStorageDriver:
| Environment Variable | Command-Line Flag | Default Value | Description |
|---|---|---|---|
ARCHIVE_SOURCE_DIRECTORY | --archive-source | $HOME/clowder/data/uploads/ | The current directory where Clowder stores it's uploaded files |
ARCHIVE_TARGET_DIRECTORY | --archive-target | $HOME/clowder/data/archive/ | The target directory where the archival extractor should store the files that it archives. Note that this path can be on a network or other persistent storage. |
To build the Disk archival extractor's Docker image, execute the following commands:
| Code Block | ||
|---|---|---|
| ||
git clone https://opensource.ncsa.illinois.edu/bitbucket/scm/cats/extractors-archival-disk.git
cd extractors-archival-disk/
docker build -t clowder/extractors-archival-disk . |
Example Configuration: Archive to another folder
...
To run the Disk archival extractor with this configuration:
| Code Block |
|---|
docker run -itd --rmit -v /Users/lambert8/clowder/data/:/home/clowder/clowder/data -e ARCHIVE_SOURCE_DIRECTORY="/home/clowder/clowder/data/uploads/" -e ARCHIVE_TARGET_DIRECTORY="/home/clowder/clowder/data/archive/" clowder/extractors-archival-disk |
NOTE: on MacOSX, you may need to run the extractor with the --net=host option to connect to RabbitMQ
...
ncsa.archival.s3
To build the S3 archival extractor's Docker image, execute the following commands:
| Code Block | ||
|---|---|---|
| ||
git clone https://opensource.ncsa.illinois.edu/bitbucket/scm/cats/extractors-archival-s3.git
cd extractors-archival-s3/
docker build -t clowder/extractors-archival-s3 . |
Configuration Options
The following configuration options must match your configuration of the S3ByteStorageDriver:
| Environment Variable | Command-Line Flag | Default Value | Description |
|---|---|---|---|
AWS_S3_SERVICE_ENDPOINT | --service-endpoint <value> | https://s3.amazonaws.com | Which AWS Service Endpoint to use to connect to S3. Note that this may depend on the region used, but can also be used to point at a running MinIO instance. |
AWS_ACCESS_KEY | --access-key <value> | "" | The AccessKey that should be used to authorize with AWS or MinIO |
AWS_SECRET_KEY | --secret-key <value> | "" | The SecretKey that should be used to authorize with AWS or MinIO |
AWS_BUCKET_NAME | --bucket-name <value> | clowder-archive | The name of the bucket where the files are stored in Clowder. |
AWS_REGION | --region <value> | us-east-1 | AWS only: the region where the S3 bucket exists |
To build the S3 archival extractor's Docker image, execute the following commands:
...
| language | bash |
|---|
...
Example Configuration: S3 on AWS in us-east-2 Region
...