Offline correlation
In some instances, it is extremely useful to be able to produce visibilities from recorded voltage data, where online processing hardware limitations and real-time requirements are relaxed, allowing for greater time and frequency averaging options.
MWAX offline correlator
See the MWAX Offline Correlator page for some details of that specific setup.
BLINK correlator
The BLINK correlator software, part of the BLINK project, offers GPU accelerated correlation working on both AMD and NVIDIA GPUs. With a similar interface and functionality as the legacy offline correlator, the blink-correlator
executable is the only tool available at the moment to perform correlation on Pawsey’s Setonix supercomputer.
The BLINK correlator is a software under development as part of Cristian Di Pietrantonio’s PhD project. While its output is thoroughly tested, it is subject to fast development and its interface and error handling may be unstable. The source code will be released once Cristian Di Pietrantonio’s first paper is published.
If you encounter any issue please contact cristian.dipietrantonio [at] postgrad.curtin.edu.au
The BLINK correlator currently only supports the old legacy VCS .dat
format. There exists a tool to convert newer .sub
to the .dat
format as part of the VCSBeam software. The plan is to natively support the newer formats in the future.
How to run the offline correlation on Setonix
The blink-correlator
module is installed under the mwavcs
project on Setonix.
cdipietrantonio@setonix-01:~> module avail blink-correlator
--------------------- /software/projects/mwavcs/setonix/2024.05/modules/zen3/gcc/12.2.0 ---------------------
blink-correlator/9Dec2024
Running the executable without arguments prints the help message:
cdipietrantonio@setonix-01:~> module load blink-correlator/9Dec2024
cdipietrantonio@setonix-01:~> blink-correlator
blink-correlator -t <int time> [-c <cnls>] [-o <outdir>] DATFILE1 [DATFILE2 [DATFILE3 [...]]]
Program options:
-------------
-t <integration time>: duration of the time interval to integrate over. Accepts a timespec (see below).
-c <channels to average>: number of contiguous frequency channels to average. Must be >= 1.
Default is 1, that is, no averaging.
-o <output directory>: path to a directory where to save output files. If the directory does
not exist, it will be created. Default is current directory.
-i [mwa | eda2]: choose which data type is given in input (default: mwa).
is applied.
Time specification (timespec)
-----------------------------
A timespec is a convenient way of specifing a time duration. It is made up of a real number
followed by a unit of time; for instance '2ms' is a timespec representing 2 milliseconds.
Valid unit of times are: 'ms', 'cs', 'ds', and 's'.
Here is an example of how to correlate 2 seconds of data at 50ms time and 40kHz frequency resolutions.
cdipietrantonio@setonix-01:~> salloc -p gpu -Apawsey1045-gpu --gres=gpu:1 --time=00:10:00
salloc: Pending job allocation 19070518
salloc: job 19070518 queued and waiting for resources
salloc: job 19070518 has been allocated resources
salloc: Granted job allocation 19070518
salloc: Waiting for resource configuration
salloc: Nodes nid002180 are ready for job
cdipietrantonio@nid002180:~> cd /scratch/director2183/cdipietrantonio/1276619416/combined_flash/
cdipietrantonio@nid002180:/scratch/director2183/cdipietrantonio/1276619416/combined_flash> ls -1 | head -n 50
1276619416_1276619418_ch133.dat
1276619416_1276619418_ch134.dat
1276619416_1276619418_ch135.dat
1276619416_1276619418_ch136.dat
[...]
1276619416_1276619419_ch133.dat
1276619416_1276619419_ch134.dat
1276619416_1276619419_ch135.dat
1276619416_1276619419_ch136.dat
[...]
cdipietrantonio@nid002180:/scratch/director2183/cdipietrantonio/1276619416/combined_flash> blink-correlator -t 50ms -c 4 1276619416_127661941[89]_ch*.dat -o vis
Running the correlator program with the following options:
Integration time interval: 0.05s
Number of channels to average: 4
Output directory: vis
Correlating voltages in 1276619416_1276619418_ch133.dat..
Correlation is happening on GPU..
Correlation took 1191ms
Correlating voltages in 1276619416_1276619418_ch134.dat..
Correlation is happening on GPU..
Correlation took 150ms
Correlating voltages in 1276619416_1276619418_ch135.dat..
Correlation is happening on GPU..
Correlation took 151ms
[...]
cdipietrantonio@nid002180:/scratch/director2183/cdipietrantonio/1276619416/combined_flash> cd vis
cdipietrantonio@nid002180:/scratch/director2183/cdipietrantonio/1276619416/combined_flash/vis> ls
1276619416_20200619163000_gpubox01_00.fits 1276619416_20200619163000_gpubox13_00.fits 1276619416_20200619163001_gpubox01_00.fits 1276619416_20200619163001_gpubox13_00.fits
1276619416_20200619163000_gpubox02_00.fits 1276619416_20200619163000_gpubox14_00.fits 1276619416_20200619163001_gpubox02_00.fits 1276619416_20200619163001_gpubox14_00.fits
1276619416_20200619163000_gpubox03_00.fits 1276619416_20200619163000_gpubox15_00.fits 1276619416_20200619163001_gpubox03_00.fits 1276619416_20200619163001_gpubox15_00.fits
1276619416_20200619163000_gpubox04_00.fits 1276619416_20200619163000_gpubox16_00.fits 1276619416_20200619163001_gpubox04_00.fits 1276619416_20200619163001_gpubox16_00.fits
1276619416_20200619163000_gpubox05_00.fits 1276619416_20200619163000_gpubox17_00.fits 1276619416_20200619163001_gpubox05_00.fits 1276619416_20200619163001_gpubox17_00.fits
1276619416_20200619163000_gpubox06_00.fits 1276619416_20200619163000_gpubox18_00.fits 1276619416_20200619163001_gpubox06_00.fits 1276619416_20200619163001_gpubox18_00.fits
1276619416_20200619163000_gpubox07_00.fits 1276619416_20200619163000_gpubox19_00.fits 1276619416_20200619163001_gpubox07_00.fits 1276619416_20200619163001_gpubox19_00.fits
1276619416_20200619163000_gpubox08_00.fits 1276619416_20200619163000_gpubox20_00.fits 1276619416_20200619163001_gpubox08_00.fits 1276619416_20200619163001_gpubox20_00.fits
1276619416_20200619163000_gpubox09_00.fits 1276619416_20200619163000_gpubox21_00.fits 1276619416_20200619163001_gpubox09_00.fits 1276619416_20200619163001_gpubox21_00.fits
1276619416_20200619163000_gpubox10_00.fits 1276619416_20200619163000_gpubox22_00.fits 1276619416_20200619163001_gpubox10_00.fits 1276619416_20200619163001_gpubox22_00.fits
1276619416_20200619163000_gpubox11_00.fits 1276619416_20200619163000_gpubox23_00.fits 1276619416_20200619163001_gpubox11_00.fits 1276619416_20200619163001_gpubox23_00.fits
1276619416_20200619163000_gpubox12_00.fits 1276619416_20200619163000_gpubox24_00.fits 1276619416_20200619163001_gpubox12_00.fits 1276619416_20200619163001_gpubox24_00.fits