Currently still in heavy development, but is able to perform direction-independent calibration on the CPU.
More documentation: https://mwatelescope.github.comio/MWATelescope/mwa_hyperdrive/wikiindex.html
Project homepage: https://github.com/MWATelescope/mwa_hyperdrive
Usage on Pawsey's garrawarla cluster
Get access to non-Pawsey-built software:
| Code Block |
|---|
| language | bash |
|---|
| theme | Midnight |
|---|
|
module use /pawsey/mwa/software/python3/modulefiles |
List available hyperdrive versions:
| Code Block |
|---|
| language | bash |
|---|
| theme | Midnight |
|---|
|
module avail hyperdrive |
| Code Block |
|---|
| language | bash |
|---|
| theme | Midnight |
|---|
| title | Example module avail output |
|---|
| collapse | true |
|---|
|
---------------------------------- /pawsey/mwa/software/python3/modulefiles ----------------------------------
hyperdrive/chj hyperdrive/v0.2.0-alpha11 (L,D) |
Load a hyperdrive module:
| Code Block |
|---|
| language | bash |
|---|
| theme | Midnight |
|---|
|
module load hyperdrive # this will load the default version
module load hyperdrive/chj # load CHJ's development version |
Example Slurm script
| Code Block |
|---|
| language | bash |
|---|
| theme | Midnight |
|---|
|
#!/bin/bash -l
#SBATCH --job-name=hyp-$1
#SBATCH --output=hyperdrive.out
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=40
#SBATCH --time=01:00:00
#SBATCH --clusters=garrawarla
#SBATCH --partition=gpuq
#SBATCH --account=mwaeor
#SBATCH --export=NONE
#SBATCH --gres=gpu:1,tmp:50g
module use /pawsey/mwa/software/python3/modulefiles
module load hyperdrive
set -eux
command -v hyperdrive
cd /astro/mwaeor/MWA/data/1090008640
# Get calibration solutions. Use the top 1000 sources.
hyperdrive di-calibrate \
-s /pawsey/mwa/software/python3/srclists/master/srclist_pumav3_EoR0aegean_fixedEoR1pietro+ForA_phase1+2.txt \
-n 1000 \
-d *gpubox*.fits *.metafits *.mwaf \
-o hyp_sols.fits
# Apply the solutions and write out a measurement set.
# Write it to /nvmetmp as that's much faster than /astro.
hyperdrive solutions-apply \
-d *gpubox*.fits *.metafits *.mwaf \
-s hyp_sols.fits \
-o /nvmetmp/hyp_calibrated.ms \
--time-average 8s \
--freq-average 80kHz
# Move the measurement set to /astro.
mv /nvmetmp/hyp_calibrated.ms . |
This example script reserves 50 GB of space for node local storage (/nvmetmp). If your output visibilities are bigger than this, then the write will fail; you should adjust the #SBATCH --gres=gpu:1,tmp:50g line to account for this, e.g. #SBATCH --gres=gpu:1,tmp:200g