giant-squid (ASVO downloading code)
giant-squid
is an alternative ASVO client (the official client being manta-ray
).
Reasons you would want to use giant-squid
over manta-ray
:
- It's twice as efficient at downloading data.
- Technical details: It doesn't download a zip by default; it transparently unzips the data from ASVO, resulting in half the amount of IO required to actually get at the data.
- It's written in Rust, not Python
- No strange Python dependency issues!
- It allows you to control how long your jobs live. The default is 7 days, but it might be useful to do a large batch of data with a 1 day lifetime.
- (I don't think
manta-ray
allows you to control this, but I could be wrong)
- (I don't think
Usage on Pawsey's garrawarla cluster
Get access to non-Pawsey-built software:
module use /pawsey/mwa/software/python3/modulefiles
Load the default module:
module load giant-squid
Look at the help text to help you decide what to do next:
giant-squid -h