Tutorial: Visibility Data Reduction on Setonix
- Dev Null
Starting with raw MWA data, we will preprocess, calibrate, and produce an image. For a more thorough tutorial including some quality analysis steps, check out mwa-demo.
For this tutorial, we will use already-provided raw data in an interactive session, but typically, data reduction at scale involves a slurm script that uses giant-squid to download raw, or calibrated visibilties from ASVO.
Users in the mwaeor , mwasci and mwavcs groups can request gpu nodes with the salloc command:
# request 1/8 of a GPU node for an hour
salloc --nodes=1 --partition=mwa-gpu --account=${PAWSEY_PROJECT}-gpu -t 01:00:00 --gres=gpu:1
# load our modules
module load giant-squid/1.0.3 mwalib/1.5.0 birli/0.15.1 hyperdrive-amd-gfx90a/0.4.1 wsclean/3.4-idg-everybeamother users can request a CPU node on the mwa queue, and use the CPU version of the module, hyperdrive/0.4.1 instead.
Setup
Create a directory on the scratch filesystem to store the data we need for the tutorial, and change directory into it. Files on scratch are subject to a purge policy, which deletes the files Pawsey Filesystems and their Use
mkdir -p $MYSCRATCH/deleteme
cd $_ # "$_" is short-hand for the last argument of the previous commandDownload our calibration sky model, storing the filename in the srclist environment variable
export srclist="GGSM_updated.fits"
wget 'https://github.com/GLEAM-X/GLEAM-X-pipeline/raw/refs/heads/master/models/'$srclistWe will skip over obtaining raw data, but see MWA ASVO Use with HPC Systems and https://github.com/MWATelescope/mwa-demo/blob/main/demo/02_download.sh for details. This tutorial uses raw data, which allows for more flexibility in preprocessing, flagging and calibration options, but the calibrated visibilities that can be obtained by ASVO are often sufficient for most science cases. We’ll process http://ws.mwatelescope.org/observation/obs/?obsid=1121334536
Preprocessing
Birli is the MWA preprocessor. It takes raw files, does RFI flagging, instrument corrections and format conversion, resulting in a preprocessed visibility file in the uvfits format (measurement set is supported too).
Calibration
hyperdrive is the MWA calibration suite. it has excellent documentation .
The first step in direction-independent calibration is to produce calibration solutions.
At this point, you would typically inspect the calibration solutions for quality issues with hyperdrive solutions-plot , other quality analysis tools are explored in https://github.com/MWATelescope/mwa-demo/blob/1b044e185e0594027253ec9aa9a7f13c0a35df78/demo/06_cal.sh#L106 . Once you are happy with the solutions, you can apply them to the data. In this case, we will produce a measurement set, as this is required by wsclean. However, writing measurement sets to scratch is not recommended due to deficiencies in the format itself.
imaging
you can then view the image with CARTA (without needing to download the file locally)
