Offline correlation

In some instances, it is extremely useful to be able to produce visibilities from recorded voltage data, where online processing hardware limitations and real-time requirements are relaxed, allowing for greater time and frequency averaging options.

MWAX offline correlator

See the MWAX Offline Correlator page for some details of that specific setup.

MWAX Offline Correlator

 

BLINK correlator

The BLINK correlator software, part of the BLINK project, offers GPU accelerated correlation working on both AMD and NVIDIA GPUs. With a similar interface and functionality as the legacy offline correlator, the blink-correlatorexecutable is the only tool available at the moment to perform correlation on Pawsey’s Setonix supercomputer.

The BLINK correlator is a software under development as part of Cristian Di Pietrantonio’s PhD project. While its output is thoroughly tested, it is subject to fast development and its interface and error handling may be unstable. The source code will be released once Cristian Di Pietrantonio’s first paper is published.

If you encounter any issue please contact cristian.dipietrantonio [at] postgrad.curtin.edu.au

The BLINK correlator currently only supports the old legacy VCS .dat format. There exists a tool to convert newer .sub to the .dat format as part of the VCSBeam software. The plan is to natively support the newer formats in the future.

How to run the offline correlation on Setonix

The blink-correlator module is installed under the mwavcsproject on Setonix.

cdipietrantonio@setonix-01:~> module avail blink-correlator --------------------- /software/projects/mwavcs/setonix/2024.05/modules/zen3/gcc/12.2.0 --------------------- blink-correlator/9Dec2024

 

Running the executable without arguments prints the help message:

cdipietrantonio@setonix-01:~> module load blink-correlator/9Dec2024 cdipietrantonio@setonix-01:~> blink-correlator blink-correlator -t <int time> [-c <cnls>] [-o <outdir>] DATFILE1 [DATFILE2 [DATFILE3 [...]]] Program options: ------------- -t <integration time>: duration of the time interval to integrate over. Accepts a timespec (see below). -c <channels to average>: number of contiguous frequency channels to average. Must be >= 1. Default is 1, that is, no averaging. -o <output directory>: path to a directory where to save output files. If the directory does not exist, it will be created. Default is current directory. -i [mwa | eda2]: choose which data type is given in input (default: mwa). is applied. Time specification (timespec) ----------------------------- A timespec is a convenient way of specifing a time duration. It is made up of a real number followed by a unit of time; for instance '2ms' is a timespec representing 2 milliseconds. Valid unit of times are: 'ms', 'cs', 'ds', and 's'.

 

Here is an example of how to correlate 2 seconds of data at 50ms time and 40kHz frequency resolutions.

cdipietrantonio@setonix-01:~> salloc -p gpu -Apawsey1045-gpu --gres=gpu:1 --time=00:10:00 salloc: Pending job allocation 19070518 salloc: job 19070518 queued and waiting for resources salloc: job 19070518 has been allocated resources salloc: Granted job allocation 19070518 salloc: Waiting for resource configuration salloc: Nodes nid002180 are ready for job cdipietrantonio@nid002180:~> cd /scratch/director2183/cdipietrantonio/1276619416/combined_flash/ cdipietrantonio@nid002180:/scratch/director2183/cdipietrantonio/1276619416/combined_flash> ls -1 | head -n 50 1276619416_1276619418_ch133.dat 1276619416_1276619418_ch134.dat 1276619416_1276619418_ch135.dat 1276619416_1276619418_ch136.dat [...] 1276619416_1276619419_ch133.dat 1276619416_1276619419_ch134.dat 1276619416_1276619419_ch135.dat 1276619416_1276619419_ch136.dat [...] cdipietrantonio@nid002180:/scratch/director2183/cdipietrantonio/1276619416/combined_flash> blink-correlator -t 50ms -c 4 1276619416_127661941[89]_ch*.dat -o vis Running the correlator program with the following options: Integration time interval: 0.05s Number of channels to average: 4 Output directory: vis Correlating voltages in 1276619416_1276619418_ch133.dat.. Correlation is happening on GPU.. Correlation took 1191ms Correlating voltages in 1276619416_1276619418_ch134.dat.. Correlation is happening on GPU.. Correlation took 150ms Correlating voltages in 1276619416_1276619418_ch135.dat.. Correlation is happening on GPU.. Correlation took 151ms [...] cdipietrantonio@nid002180:/scratch/director2183/cdipietrantonio/1276619416/combined_flash> cd vis cdipietrantonio@nid002180:/scratch/director2183/cdipietrantonio/1276619416/combined_flash/vis> ls 1276619416_20200619163000_gpubox01_00.fits 1276619416_20200619163000_gpubox13_00.fits 1276619416_20200619163001_gpubox01_00.fits 1276619416_20200619163001_gpubox13_00.fits 1276619416_20200619163000_gpubox02_00.fits 1276619416_20200619163000_gpubox14_00.fits 1276619416_20200619163001_gpubox02_00.fits 1276619416_20200619163001_gpubox14_00.fits 1276619416_20200619163000_gpubox03_00.fits 1276619416_20200619163000_gpubox15_00.fits 1276619416_20200619163001_gpubox03_00.fits 1276619416_20200619163001_gpubox15_00.fits 1276619416_20200619163000_gpubox04_00.fits 1276619416_20200619163000_gpubox16_00.fits 1276619416_20200619163001_gpubox04_00.fits 1276619416_20200619163001_gpubox16_00.fits 1276619416_20200619163000_gpubox05_00.fits 1276619416_20200619163000_gpubox17_00.fits 1276619416_20200619163001_gpubox05_00.fits 1276619416_20200619163001_gpubox17_00.fits 1276619416_20200619163000_gpubox06_00.fits 1276619416_20200619163000_gpubox18_00.fits 1276619416_20200619163001_gpubox06_00.fits 1276619416_20200619163001_gpubox18_00.fits 1276619416_20200619163000_gpubox07_00.fits 1276619416_20200619163000_gpubox19_00.fits 1276619416_20200619163001_gpubox07_00.fits 1276619416_20200619163001_gpubox19_00.fits 1276619416_20200619163000_gpubox08_00.fits 1276619416_20200619163000_gpubox20_00.fits 1276619416_20200619163001_gpubox08_00.fits 1276619416_20200619163001_gpubox20_00.fits 1276619416_20200619163000_gpubox09_00.fits 1276619416_20200619163000_gpubox21_00.fits 1276619416_20200619163001_gpubox09_00.fits 1276619416_20200619163001_gpubox21_00.fits 1276619416_20200619163000_gpubox10_00.fits 1276619416_20200619163000_gpubox22_00.fits 1276619416_20200619163001_gpubox10_00.fits 1276619416_20200619163001_gpubox22_00.fits 1276619416_20200619163000_gpubox11_00.fits 1276619416_20200619163000_gpubox23_00.fits 1276619416_20200619163001_gpubox11_00.fits 1276619416_20200619163001_gpubox23_00.fits 1276619416_20200619163000_gpubox12_00.fits 1276619416_20200619163000_gpubox24_00.fits 1276619416_20200619163001_gpubox12_00.fits 1276619416_20200619163001_gpubox24_00.fits