Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Offline PFB

*** WARNING: Must use VCSBeam >= v2.18 and mwalib >= v0.11.0 ***

This step is currently necessary for processing MWAX data, although it is intended to be subsumed into the beamforming step so that any intermediate channelisation that is required as part of the beamforming process is made invisible to the user. This is also important for obviating the need to write out the .dat files to disk, as the pre-beamformed data are sufficiently volumous that even our generous allotment of disk space on Pawsey's systems would be quickly exhausted.

The Offline PFB implements the weighted overlap add algorithm described in McSweeney et al. (2020).. It uses GPUs for the fine PFB operation (NVIDIA/CUDA), including cuFFT for the Fourier Transform step, and operates on one second of data at a time. The GPUs must have at least 3.5 GB of device memory available. Operating on smaller chunks of data is not (yet) implemented.

A single call to Offline PFB operates on a single coarse channel and an arbitrary number of timesteps, and produces output files in the legacy .dat format. The example SBATCH script below shows it being applied to 600 seconds of data (starting at GPS second 1313388760) for 5 coarse channels. The output files are written to the current working directory.

The Offline PFB uses the same polyphase filter that was used in the legacy system by default (see McSweeney et al. 2020), but alternative filters will be made available in the future. The filters are always applied on the second boundaries, and the tap size is determined from the length of the filter and the number of desired output channels. No attempt is made apply any time or phase adjustments to the voltages either before or after the PFB is applied.

The default mode of the PFB, and the FINEPFB filter, are described in the beginning of McSweeney et al. (2020).

Command line options

fine_pfb_offline -h 
usage: fine_pfb_offline [OPTIONS]

REQUIRED OPTIONS

	-m, --metafits=FILE        FILE is the metafits file for the target observation

OPTIONAL OPTIONS

	-b, --begin=GPSTIME        Begin time of observation, in GPS seconds
	                           If GPSTIME starts with a '+' or a '-', then the time
	                           is taken relative to the start or end of the observation
	                           respectively. [default: "+0"]
	-d, --data-location=PATH   PATH is the directory containing the recombined data
	                           [default: current directory]
	-f, --coarse-chan=CHAN     Coarse channel number
	                           If CHAN starts with a '+' or a '-', then the channel is taken
	                           relative to the first or last channel in the observation
	                           respectively. Otherwise, it is treated as a receiver channel number
	                           (0-255) [default: "+0"]
	-S, --synth_filter=FILTER  Apply the named filter during high-time resolution synthesis.
	                           File [RUNTIME_DIR]/FILTER.dat must exist [default: FINEPFB]
	-T, --nseconds=VAL         Process VAL seconds of data [default: as many as possible]

OTHER OPTIONS

	-h, --help                 Print this help and exit
	-V, --version              Print version number and exit


Example of use on Garrawarla

(This example is intended to show how to pack multiple jobs onto the same compute node, but my testing of this script did not appear to produce the desired parallelisation that I was hoping for. At the very least, the user will be able to amend this example so that it requests multiple compute nodes.)

fine_pfb_example.sh 
#!/bin/bash -l

#SBATCH --nodes=1
#SBATCH --ntasks=5
#SBATCH --ntasks-per-node=5
#SBATCH --mem=370gb
#SBATCH --partition=gpuq
#SBATCH --gres=gpu:1
#SBATCH --time=02:00:00
#SBATCH --account=mwavcs
#SBATCH --export=NONE

module use /pawsey/mwa/software/python3/modulefiles
module load vcsbeam/master
module load openmpi-ucx-gpu

srun -N 1 -n 1 fine_pfb_offline -m /path/to/1313388760_metafits.fits -b 1313388760 -T 600 -f 144 -d /path/to/subfiles &
srun -N 1 -n 1 fine_pfb_offline -m /path/to/1313388760_metafits.fits -b 1313388760 -T 600 -f 145 -d /path/to/subfiles &
srun -N 1 -n 1 fine_pfb_offline -m /path/to/1313388760_metafits.fits -b 1313388760 -T 600 -f 146 -d /path/to/subfiles &
srun -N 1 -n 1 fine_pfb_offline -m /path/to/1313388760_metafits.fits -b 1313388760 -T 600 -f 147 -d /path/to/subfiles &
srun -N 1 -n 1 fine_pfb_offline -m /path/to/1313388760_metafits.fits -b 1313388760 -T 600 -f 148 -d /path/to/subfiles &

wait
  • No labels