Usage on Pawsey's garrawarla cluster

Get access to non-Pawsey-built software:

module use /pawsey/mwa/software/python3/modulefiles

List available hyperdrive versions:

module avail hyperdrive

Example module avail output

---------------------------------- /pawsey/mwa/software/python3/modulefiles ----------------------------------
hyperdrive/chj hyperdrive/v0.2.0-alpha11 (L,D)

Load a hyperdrive module:

module load hyperdrive # this will load the default version
module load hyperdrive/chj # load CHJ's development version

Example Slurm script

#!/bin/bash -l
#SBATCH --job-name=hyp-$1
#SBATCH --output=hyperdrive.out
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=40
#SBATCH --time=01:00:00
#SBATCH --clusters=garrawarla
#SBATCH --partition=gpuq
#SBATCH --account=mwaeor
#SBATCH --export=NONE
#SBATCH --gres=gpu:1,tmp:50g

module use /pawsey/mwa/software/python3/modulefiles
module load hyperdrive

set -eux
command -v hyperdrive

cd /astro/mwaeor/MWA/data/1090008640

# Get calibration solutions. Use the top 1000 sources.
hyperdrive di-calibrate \
    -s /pawsey/mwa/software/python3/srclists/master/srclist_pumav3_EoR0aegean_fixedEoR1pietro+ForA_phase1+2.txt \
    -n 1000 \
    -d *gpubox*.fits *.metafits *.mwaf \
    -o hyp_sols.fits

# Apply the solutions and write out a measurement set.
# Write it to /nvmetmp as that's much faster than /astro.
hyperdrive solutions-apply \
    -d *gpubox*.fits *.metafits *.mwaf \
    -s hyp_sols.fits \
    -o /nvmetmp/hyp_calibrated.ms \
    --time-average 8s \
    --freq-average 80kHz

# Move the measurement set to /astro.
mv /nvmetmp/hyp_calibrated.ms .

This example script reserves 50 GB of space for node local storage (/nvmetmp). If your output visibilities are bigger than this, then the write will fail; you should adjust the #SBATCH --gres=gpu:1,tmp:50g line to account for this, e.g. #SBATCH --gres=gpu:1,tmp:200g

hyperdrive (calibration software)

Usage on Pawsey's garrawarla cluster

Example module avail output

Example Slurm script