Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 11 Next »

A disk-based system powered by Dell, named Acacia after Australia’s national floral emblem the Golden Wattle – Acacia pycnantha, providing 60PB of high-speed object storage for hosting research data online. This multi-tiered cluster separates different types of data to improve data availability.

Detailed information on how to use Acacia can be found at https://support.pawsey.org.au/documentation/display/US/Acacia+-+Common+Usage.

Setting up

Creating keys

  • Click View Keys:

  • You will need to create a key for each "Storage" which you would like to use. Select the desired Storage Name from the dropdown, click Create New Key, and click Yes when prompted to confirm.

  • You will be shown an Access ID and a Secret Key that you will need later. The Access ID is easily obtained on the portal website, but the Secret Key will not be shown again. COPY THE ACCESS ID AND SECRET KEY AND KEEP THEM SOMEWHERE SAFE.

  • If you lose the secret key, the easiest recovery method is to simply delete the key and create a new one.

Pawsey environment

The next step is to setup your environment on Pawsey so that you can access the Acacia "S3" system from Garrawarla, etc. There are two clients available for doing so:

  1. MinIO Client ("mc")
  2. rclone

The set up for each client is different, as detailed in the following subsections. It has been reported that rclone seems to be the more robust choice for very large file transfers, and that mc does not seem to be able to handle large file transfers.

MinIO Client

mc alias set <ALIAS_NAME> https://acacia.pawsey.org.au <ACCESS_ID> <SECRET_KEY>
  • ALIAS_NAME can be anything, but a sensible choice is the Storage Name (e.g. mwasci in the screenshots above)
  • ACCESS_ID is the Access ID of the key created to access the given storage
  • SECRET_KEY is the Secret Key given to you at the time your created the key.

You are now set up to use the mc command to move data to/from your Acacia storage. You do not need to load any special modules to use mc.

rclone

Create the following file:

~/.config/rclone/rclone.conf
[<ALIAS_NAME>]
type = s3
provider = Other
access_key_id = <ACCESS_ID>
secret_access_key = <SECRET_KEY>
endpoint = https://projects.pawsey.org.au
acl = public-read-write
bucket_acl = public-read-write
  • ALIAS_NAME can be anything, but a sensible choice is the Storage Name (e.g. mwasci in the screenshots above)
  • ACCESS_ID is the Access ID of the key created to access the given storage
  • SECRET_KEY is the Secret Key given to you at the time your created the key.

To use rclone commands, you will need to load the rclone module:

module load rclone

You are now set up to use rclone to move data to/from your Acacia storage.

Example workflows

Some example workflows are given at https://support.pawsey.org.au/documentation/display/US/Supercomputing+project+example. These examples are designed for Setonix, but can be used almost exactly as is from hpc-data. The only (necessary?) thing that needs to change is the name of the partition, which is "copy" on Setonix, but "copyq" elsewhere:

#SBATCH --partition=copyq

Note that these example scripts may require access to /scratch, which is accessible from hpc-data, but not from garrawarla or galaxy.

mc cheatsheet

Creating a bucket

mc mb <ALIAS_NAME>/<BUCKET_NAME>

Uploading data

mc cp <PATH_TO_FILE> <ALIAS_NAME>/<BUCKET_NAME>/<NAME_OF_FILE> 

Downloading data

mc cp <ALIAS_NAME/BUCKET_NAME/NAME_OF_FILE> <PATH_TO_DOWNLOAD_DIRECTORY>

List buckets

mc ls <ALIAS_NAME>

List bucket contents

mc ls <ALIAS_NAME>/<BUCKET_NAME>



  • No labels