1 Overview
2 Hardware
- 2.1 CUDA GPU
3 Software
- 3.1 CUDA
- 3.2 CFITSIO library
- 3.3 PSRDADA library
- 3.4 mwax_xGPU library
- 3.5 mwax_common
- 3.6 mwax_db2correlate2db
- 3.7 mwax_db2fits
- 3.8 Example build script on Garrawarla
  - 3.8.1 build_mwax_offline_corr.sh
4 How to Run
- 4.1 Setup
  - 4.1.1 Prepare input data
  - 4.1.2 Create Ring Buffers
  - 4.1.3 Launch Programs
- 4.2 Processing
- 4.3 Clean up

Overview

The MWAX correlator was built to be modular and to be run real-time in production at the MRO, or offline using archived voltage subfiles instead of live multicast UDP packets.

High level view of offline and online MWAX correlator flows

Hardware

CUDA GPU

A significant fraction of the xGPU library and the mwax_db2correlate2db process is written in CUDA, so a CUDA-compatible GPU is essential.

Software

CUDA

Visit the Homepage. CUDA is required to be installed prior to PSRDADA, and mwax_xGPU.

CFITSIO library

Visit the Homepage.

$ CFLAGS="-O3" ./configure --prefix=/usr/local --enable-reentrant --enable-sse2 --enable-ssse3 --disable-curl
$ make clean && make
$ make install

PSRDADA library

Visit the Homepage

The MWAX correlator uses, and has tested, this commit (ca505cdb519afbd63ae91c00e4d86af0f3313b68) from the PSRDADA repository.

For Ubuntu 20.04 systems, the following packages also need to be installed in order to compile PSRDADA successfully (this was discovered through trial and error) - other Debian based systems should have similar requirements:

$ sudo apt update
$ sudo apt install automake autoconf csh hwloc libhwloc-dev libhwloc-plugins libtool numactl pkg-config python2.7-dev python

This is how we build it (note: some paths may be different on your system):

$ ./bootstrap
$ ./configure --with-cuda-dir=/usr/local/cuda --with-cuda-lib-dir=/usr/local/cuda/lib64 --with-hwloc-lib-dir=/usr/lib/x86_64-linux-gnu --prefix=/usr/local
$ make
$ make install

If all goes well, you should be able to run commands, such as:

mwax_xGPU library

Visit the github page: https://github.com/MWATelescope/mwax-xGPU

The mwax-xGPU fork must be compiled with very specific flags in order to work. This is for a 128T system:

Replace {{gpu_sm_arch}} with the correct SM_ARCH value for your GPU. See CUDA GPU List (wikipedia). Read off the "Compute capability (version)" value, remove the dot and prefix with "sm". E.g. an Nvidia V100 would be sm_70.

For cases where tiles >192T and up to 256T:

xGPU's NSTATION has a minimum of 16 and MUST be in increments of 16. So, for example, if you need to correlate 136T, then NSTATION needs to be 144 because 136 is not a multiple of 16. 144 is the next closest multiple.

mwax_common

Visit the github page: https://github.com/MWATelescope/mwax_common

This repo is required by mwax_db2correlate2db and mwax_db2fits. It contains some common header and c files which both programs require.

mwax_db2correlate2db

Visit the github page: https://github.com/MWATelescope/mwax_cbf

mwax_db2fits

Visit the github page: https://github.com/MWATelescope/mwax_db2fits

Example build script on Garrawarla

This is mostly for posterity given that Garrawarla's decommissioning at the end of 2024. It should help those down the line on other compatible systems (i.e., where xGPU is available).

build_mwax_offline_corr.sh

How to Run

Setup

Prepare input data

Ensure all the subfiles you wish to process are in one directory e.g. /path/to/subfiles
For each subfile you will need to modify the following Key/Value pairs to your use case:
One way to modify these values is to use the mwax_update_subfile_header utility in the mwax_user_tools github repo.

Key	Existing Value	Set To	Notes

Key	Existing Value	Set To	Notes
MODE	MWAX_VCS	MWAX_CORRELATOR	Required
INT_TIME_MSEC	a default set by the M&C system	250-8000	This is the correlator integration time (in ms)
FINE_CHAN_WIDTH_HZ	a default set by the M&C system	200-1280000	This is the correlator frequency resolution (in Hz). E.g. 10000 would be 10 kHz.
FSCRUNCH_FACTOR	a default set by the M&C system	1-6400	The number of 200 Hz ultrafine channels to scrunch together into a fine channel. E.g. for 10 kHz fine channels set this to 50
NFINE_CHAN	a default set by the M&C system	1-6400	Redundant info but must be compatible with FSCRUNCH_FACTOR and FINE_CHAN_WIDTH_HZ. E.g. 128 for 10kHz fine channels
EXPOSURE_SECS	the duration of the VCS observation	The duration of the correlator observation you want.	Must be in unit of 8 seconds, matching the total number of subfiles you wish to correlate per coarse channel * 8.
OBS_ID	The original obs_id of the VCS observation	The first SUBOBS_ID of the data you want to correlate	So if you have a VCS observation and want to correlate only from 80 seconds in, then you need to make OBS_ID==SUBOBS_ID==the gps time at 80 seconds into the VCS obs.
OBS_OFFSET	The 8 second offset for this subfile for the original VCS observation.	The offset you need for your correlator observation.	The first subfile in your correlator observation should be 0. Next will be 8, and so on.

See also: MWAX PSRDADA header for more information on the definition of each key within the subfile header.

Create Ring Buffers

For the INPUT ringbuffer, the -b parameter depends on the number of tiles present in the subfiles with the following relation:

INPUT_BUFFER_SIZE = TILES * 256,000 bytes

For the OUTPUT ringbuffer , the -b parameter depends on the number of tiles present in the output visibilities with the following relation:

OUTPUT_BUFFER_SIZE = (TILES + 1) * TILES * 102408 bytes

Launch Programs

Processing

It is important to process each coarse channel separately. Do not mix coarse channel subfiles- process all of one channel before processing all of another channel. You should then have a FITS file per coarse channel

For this you will need a bash script to execute a for/while loop through all subfiles and for each one run:

Each time this is executed it will fill the input ringbuffer with the subfile, and that will kickstart mwax_db2correlate2db processing data and filling the output ringbuffer with visibilities and weights, which will then kickstart mwax_db2fits writing out FITS files.

MWA Telescope

MWAX Offline Correlator