...

Detailed information on how to upload to acacia is given in the pawsey help/support page!

Setting up ACACIA

Use the link: can be found at https://support.pawsey.org.au/documentation/display/US/Acacia+-+Common+Usage.

...

Setting up

Creating keys

• Go to https://portal.pawsey.org.au/origin

...

• and login using your Pawsey username and password

...

Once you are in the Acacia tab, Storage name shows all the relevant storage that is assigned under the specific user. For example, USERNAME, mwavcs etc

• Click on the storage name where you want to upload the data. 
• Press on the View Keys button. This shows the existing keys for the user. 
• As first time user, click on the Create New Key button. 
• Go to the ACACIA tab

...

• On doing so, it will show :

Image Added

• Click View Keys:

Image Added

• You will need to create a key for each "Storage" which you would like to use. Select the desired Storage Name from the dropdown, and click Create New Key, and click Yes when prompted to confirm.

Image Added

...

• You will be shown an Access ID and a Secret Key that is very very important to set up acacia on garrawarla etc. 

NOTE: Make sure that you copy the Access ID and the Secret Key before closing the pop up window. 

• Once you have the ID and Key, login to Garrawarla/magnus etc. Then type: 

                                                               mc -h 

mc is MinIO Client for cloud storage and filesystems. -h option shows all the functions of mc. 

• The next step is to setup an alias for the storage:

• you will need later. The Access ID is easily obtained on the portal website, but the Secret Key will not be shown again. COPY THE ACCESS ID AND SECRET KEY AND KEEP THEM SOMEWHERE SAFE.

Image Added

• If you lose the secret key, the easiest recovery method is to simply delete the key and create a new one.

Pawsey environment

The next step is to setup your environment on Pawsey so that you can access the Acacia "S3" system from Garrawarla, etc. There are two clients available for doing so:

1. MinIO Client ("mc")
2. rclone

The set up for each client is different, as detailed below.

MinIO Client

mc alias set <ALIAS_NAME> https://acacia.pawsey.org.au <ACCESS_ID> <SECRET_KEY>

• ALIAS_NAME can be anything, but a sensible choice is the Storage Name (e.g. mwasci in the screenshots above)
• ACCESS_ID is the Access ID of the key created to access the given storage
• SECRET_KEY is the Secret Key

...

The alias name that you set will be used from now on to upload and access data on acacia. 

Uploading the data

• The next step is to create a bucket where all your data files will be uploaded. This can be thought of like a directory in our system. It can be done by:

                                                             set +o history

...

• given to you at the time your created the key.

You are now set up to use the mc command to move data to/from your Acacia storage. You do not need to load any special modules to use mc.

rclone

Create the following file:

[<ALIAS_NAME>]
type = s3
provider = Other
access_key_id = <ACCESS_ID>
secret_access_key = <SECRET_KEY>
endpoint = https://projects.pawsey.org.au
acl = public-read-write
bucket_acl = public-read-write

• ALIAS_NAME can be anything, but a sensible choice is the Storage Name (e.g. mwasci in the screenshots above)
• ACCESS_ID is the Access ID of the key created to access the given storage
• SECRET_KEY is the Secret Key given to you at the time your created the key.

To use rclone commands, you will need to load the rclone module:

module load rclone

You are now set up to use rclone to move data to/from your Acacia storage.

mc cheatsheet

Creating a bucket

mc mb <ALIAS_NAME>/<BUCKET_NAME>

...

                                                            set -o history

This will create the bucket according to the name you have given. Once the bucket is created we can start uploading our data there. 

• To upload the data:

...

Uploading data

mc cp <PATH_TO_FILE> <ALIAS_NAME>/<BUCKET_NAME>/<NAME_OF_FILE> 

...

Other essentials

• To download an object from Acacia:

...

Downloading data

mc cp <ALIAS_NAME/BUCKET_NAME/NAME_OF_FILE> <PATH_TO_DOWNLOAD_DIRECTORY>

• To list the buckets in your account using MC, use the mc ls command:

                                                           mc ls <ALIAS_NAME>

• To list the objects in a bucket:

...

List buckets

mc ls <ALIAS_NAME>

List bucket contents

mc ls <ALIAS_NAME>/<BUCKET_NAME>

...

Essential links:

• To create buckets:

https://support.pawsey.org.au/documentation/pages/viewpage.action?pageId=104302738#

• To upload and download data:

https://support.pawsey.org.au/documentation/pages/viewpage.action?pageId=104302741

• To list buckets and data in buckets:

...

...