...
Detailed information on how to use Acacia can be found at https://support.pawsey.org.au/documentation/display/US/Acacia+-+Common+Usage.
| Table of Contents |
|---|
...
Setting up
Creating keys
- Go to https://portal.pawsey.org.au/origin and login using your Pawsey username and password
- Go to the ACACIA tab:
...
- If you lose the secret key, the easiest recovery method is to simply delete the key and create a new one.
Pawsey environment
The next step is to setup your environment on Pawsey so that you can access the Acacia "S3" system from Garrawarla, etc. There are two clients available for doing so:
MinIO Client ("mc")As at March 2024, minio client is no longer supported or available on Pawsey systems.- rclone
The set up for each client is different, as detailed in the following subsections. It has been reported that rclone seems to be the more robust choice for very large file transfers, and that mc does not seem to be able to handle large file transfers.
MinIO Client
| Code Block | ||||
|---|---|---|---|---|
| ||||
mc alias set <ALIAS_NAME> https://acacia.pawsey.org.au <ACCESS_ID> <SECRET_KEY> |
...
.
You are now set up to use the mc command to move data to/from your Acacia storage. You do not need to load any special modules to use mc.
rclone
Create the following file:
...
You are now set up to use rclone to move data to/from your Acacia storage.
AWS Client
Configuration for aws
| Code Block | ||
|---|---|---|
| ||
user@setonix:~> tee -a $HOME/.aws/credentials <<EOF
[<profilename>]
aws_access_key_id=
aws_secret_access_key=
EOF
user@setonix:~> tee -a $HOME/.aws/config <<EOF
[profile <profilename>]
output=json
EOF |
To use aws client commands, you will need to load the aws client module:
| Code Block | ||||
|---|---|---|---|---|
| ||||
module load awscli/1.16.308 #Garrawarla
module load aws-cli/2.13.0 #Setonix |
To create a new bucket using AWS S3 CLI, use the S3 mb command:
...
> aws --endpoint-url=https://projects.pawsey.org.au --profile=<PROFILE_NAME> s3 cp <SOURCE> s3://<bucket>/<TARGET>
Where:
<SOURCE>is the filesystem path and name of the file you want to upload.<TARGET>is the key of the object on Acacia, so bucket name, pseudo folder (optional), and object name. You can specify any object name to “rename” the file on upload.
...
> aws --endpoint-url=https://projects.pawsey.org.au --profile=<PROFILE_NAME> s3 cp s3://<bucket>/<SOURCE> <TARGET>
Where:
<SOURCE>is the key of the object on Acacia you want to download, so bucket name, pseudo folder (optional), and object name.<TARGET>is the filesystem path where you want the object to be downloaded to.
...
<PROFILE_NAME>is the name you gave to the account credentials when configuring AWS S3 CLI.<BUCKET_NAME>is the name of the bucket containing the object.<OBJECT_NAME>is the name of the object to share.
Example workflows
Some example workflows are given at https://support.pawsey.org.au/documentation/display/US/Supercomputing+project+example. These examples are designed for Setonix, but can be used almost exactly as is from hpc-data. The only (necessary?) thing that needs to change is the name of the partition, which is "copy" on Setonix, but "copyq" elsewhere:
...
Note that these example scripts may require access to /scratch, which is accessible from hpc-data, but not from garrawarla or galaxy.
mc cheatsheet
Creating a bucket
| Code Block | ||||
|---|---|---|---|---|
| ||||
mc mb <ALIAS_NAME>/<BUCKET_NAME> |
Uploading data
| Code Block | ||||
|---|---|---|---|---|
| ||||
mc cp <PATH_TO_FILE> <ALIAS_NAME>/<BUCKET_NAME>/<NAME_OF_FILE> |
Downloading data
| Code Block | ||||
|---|---|---|---|---|
| ||||
mc cp <ALIAS_NAME/BUCKET_NAME/NAME_OF_FILE> <PATH_TO_DOWNLOAD_DIRECTORY> |
List buckets
| Code Block | ||||
|---|---|---|---|---|
| ||||
mc ls <ALIAS_NAME> |
List bucket contents
...
| language | bash |
|---|---|
| theme | Midnight |
...
...