giant-squid (ASVO downloading code)

giant-squid is an alternative ASVO client (the official client being manta-ray).


Reasons you would want to use giant-squid over manta-ray:

  • It's twice as efficient at downloading data.
    • Technical details: It doesn't download a zip by default; it transparently unzips the data from ASVO, resulting in half the amount of IO required to actually get at the data.
  • It's written in Rust, not Python
    • No strange Python dependency issues!
  • It allows you to control how long your jobs live. The default is 7 days, but it might be useful to do a large batch of data with a 1 day lifetime.
    • (I don't think manta-ray allows you to control this, but I could be wrong)


Usage on Pawsey's garrawarla cluster

Get access to non-Pawsey-built software:

module use /pawsey/mwa/software/python3/modulefiles

Load the default module:

module load giant-squid

Look at the help text to help you decide what to do next:

giant-squid -h

More information

https://github.com/MWATelescope/giant-squid