VCSBeam version

Required: >= v2.18.1.8

Code Block

language	bash
theme	Midnight

# On Garrawarla, test vcsbeam version
$ module use /pawsey/mwa/software/python3/modulefiles
$ module load vcsbeam
$ make_mwa_tied_array_beam -V
MWA Beamformer v2.18.18_e79c6bd

...

99df2a4

Offline FinePFB

Code Block

language	bash
theme	Midnight

$ cd /astro/mwavcs/asvo/251861
$ wget -O 1318345216.fits http://ws.mwatelescope.org/metadata/fits?obs_id=1318345216
$ cd /astro/mwavcs/vcs/1318345216/combined
$ for freq in {109..120}
do
    sed "s/FREQ/${freq}/" < make_legacy.sbatch > make_legacy_ch${freq}.sbatch
    sbatch make_legacy_ch${freq}.sbatch
done

...

Code Block

language	bash
theme	Midnight
title	make_legacy.sbatch
collapse	true

#!/bin/bash -l

#SBATCH --nodes=1
#SBATCH --cpus-per-task=1
#SBATCH --ntasks-per-node=1
#SBATCH --mem=370gb
#SBATCH --partition=gpuq
#SBATCH --gres=gpu:1
#SBATCH --time=01:00:00
#SBATCH --account=mwavcs
#SBATCH --export=NONE

module use /pawsey/mwa/software/python3/modulefiles
module load vcsbeam
module load openmpi-ucx-gpu

DATAPATH=/astro/mwavcs/asvo/251861

srun -N 1 -n 1 fine_pfb_offline -m $DATAPATH/1318345216.metafits -b 1318345216 -T 600 -f FREQ -d $DATAPATH

Incoherent Beam

Code Block

language	bash
theme	Midnight

$ cd /astro/mwavcs/vcs/1318345216/vcsbeam/incoh
$ sbatch vcsbeam_incoh.sbatch

Code Block

language	bash
theme	Midnight
title	/astro/mwavcs/vcs/1318345216/vcsbeam/incoh/vcsbeam_incoh.sbatch
collapse	true

#!/bin/bash -l

#SBATCH --nodes=12
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=370gb
#SBATCH --partition=gpuq
#SBATCH --gres=gpu:1
#SBATCH --time=00:30:00
#SBATCH --account=mwavcs
#SBATCH --nice=0
#SBATCH --export=NONE
 
module use /pawsey/mwa/software/python3/modulefiles
module load vcsbeam
 
srun -N 12 -n 12 make_mwa_incoh_beam \
    -m /astro/mwavcs/vcs/1318345216/combined/1318345216_legacy.metafits \
    -b 1318345216 \
    -T 600 \
    -f 109 \
    -d /astro/mwavcs/vcs/1318345216/combined \
    -o incoh

After beamforming finishes, fold on B0031-07:

Code Block

language	bash
theme	Midnight

$ module load singularity
$ /pawsey/mwa/singularity/presto/presto.sif prepfold -psr J0034-0721 -nosearch -nodmsearch incoh*.fits

Image Added

The pulsar is clearly there, but there is a lot of RFI. After some experimenting, I find that the last 38% or so is RFI free:

Code Block

language	bash
theme	Midnight

$ pawsey/mwa/singularity/presto/presto.sif prepfold -psr J0034-0721 -start 0.62 -nosearch -nodmsearch incoh*.fits

Image Added

Therefore, I should only use this section of the data to get a calibration solution.

Calibration

Offline Correlator

Code Block

language	bash
theme	Midnight

$ cd /astro/mwavcs/vcs/1318345216/cal/1318345216/vis
$ for freq in {109..120}
do
    sed "s/FREQ/${freq}/" < offline_correlate.sbatch > offline_correlate_${freq}.sbatch
    sbatch offline_correlate_${freq}.sbatch
done

...

Code Block

language	bash
theme	Midnight
title	offline_correlate.sbatch
collapse	true

#!/bin/bash -l

#SBATCH --nodes=1
#SBATCH --mem=370gb
#SBATCH --partition=gpuq
#SBATCH --gres=gpu:1
#SBATCH --time=01:00:00
#SBATCH --account=mwavcs
#SBATCH --export=NONE

module use /pawsey/mwa/software/python3/modulefiles
module load offline_correlator

for t in {0..299599}
do

    gpssecond=$(echo "$t + 13183452201318345216" | bc)
    gpubox=$(echo "FREQ - 108" | bc)

    INPUT_DATA_FILE=/astro/mwavcs/vcs/1318345216/combined/1318345216_${gpssecond}_chFREQ.dat
    START_SECOND=${gpssecond}
    DUMPS_PER_SECOND=1 # This sets the output time resolution
                    # (e.g. 20 --> 1/20 = 0.05s = 50 ms)
                    # Minimum allowed resolution is 2 ms
    CHANS_TO_AVERAGE=4 # This sets the output frequency resolution
                   # (e.g. 4 --> 4x10 kHz = 40 kHz)
    GPUBOX_CHANNEL_NUMBER=${gpubox} # This should be chosen to "match" the input channel
                         # This is not easy! (mwalib handles this, but at the
                         # moment, offline_correlator is not using mwalib)
    OUTPUT_PREFIX=1318345216 # Output files begin with this

    srun -N 1 -n 1 offline_correlator \
        -d ${INPUT_DATA_FILE} \
        -s ${START_SECOND} \
        -r ${DUMPS_PER_SECOND} \
        -n ${CHANS_TO_AVERAGE} \
        -c ${GPUBOX_CHANNEL_NUMBER} \
        -o ${OUTPUT_PREFIX}

done

RTS

Create source list

Code Block

language	bash
theme	Midnight
title	Create source list

$ cd /astro/mwavcs/vcs/1318345216/cal/1318345216
$ module load srclists/v1.0.0
$ srclist_by_beam.py -m /astro/mwavcs/vcs/1318345216/1318345216_metafits_ppds.fits -n 1000 -s ${SRCLISTS_DIR}/srclist_pumav3_EoR0aegean_EoR1pietro+ForA.txt

Set up flagged tiles

To find which tiles need to be flagged, find the broken tiles by going to the observation status page (http://ws.mwatelescope.org/observation/obs/?obs_id=1318345216). In the section "Tile errors and flags", you will find the "TileNames" for any tiles that should be flagged. Then, in the metafits file for the observation, you will find the "Input" numbers associated with the X and Y polarisations of each tile. The numbers you need to put in the "flagged_tiles.txt" are the Input numbers divided by two (and possibly rounded down). For example, in 1318345216, Tile064 is listed as broken, and in the metafits file, that tile's polarisations are associated with inputs 144 & 145. This means that the number that goes into flagged_tiles.txt for this tile is 72. In this way, the following list is constructed for this observation.

Code Block

language	bash
theme	Midnight
title	/astro/mwavcs/vcs/1318345216/cal/1318345216/rts/flagged_tiles.txt

Set up flagged channels

Setting up the flagged channels is more straightforward. As a rule, we always flag the edge channels (80 kHz worth on either side) and the centre channel. Since we have averaged up to 40 kHz when we did the offline correlation step above, that means we will be flagging the outer two channels on either side (as 2 x 40 kHz = 80 kHz) and the single centre channel. Since there are 32 channels in this case, the final list of flagged channels is:

Code Block

language	bash
theme	Midnight
title	/astro/mwavcs/vcs/1318345216/cal/1318345216/rts/flagged_channels.txt

Selecting only the last 38% of the observation for calibration

One of the parameters passed to the RTS is the "basename" of the files containing the visibilities. In order to distinguish between the RFI-ridden visibilities and the clean ones, I will create soft links to the clean gpubox files (created in an earlier step) and change the case of the filenames (gpubox → GPUBOX). The first and last filenames are 1318345216_20111011150016_gpubox01_00.fits and 1318345216_20111011151015_gpubox12_00.fits, with the timestamp in the format YYYYMMDDHHMMSS. To get the last 38%, I need to start 6:12 (mm:ss) into the observation, which means that my first clean file will be 1318345216_20111011150628_gpubox01_00.fits.

Code Block

language	bash
theme	Midnight
title	Create source list

$ cd /astro/mwavcs/vcs/1318345216/cal/1318345216/vis
$ ls 1318345216_2011101115062[89]_gpubox*.fits 1318345216_201110111506[3-5]?_gpubox*.fits 1318345216_20111011150[7-9]??_gpubox*.fits 1318345216_20111011151???_gpubox*.fits | while read f; do ln -s ${f} $(echo ${f} | tr 'gpubox' 'GPUBOX'); done

After doing this, the RTS ".in" file must be altered so that it's only looking for files with GPUBOX in the filename (see below).

Extra flags

After running the RTS (see below) and checking the solutions (using the python scripts supplied in VCSTools), I found that I also had to add 14 to the list of flagged channels, and 52, 53, 54, 59 and 61 to the list of flagged tiles.

Code Block

language	bash
theme	Midnight
title	/astro/mwavcs/vcs/1318345216/cal/1318345216/rts/flagged_tiles.txt
collapse	true

Run the RTS

Code Block

language	bash
theme	Midnight
title	Run the RTS

$ cd /astro/mwavcs/vcs/1318345216/cal/1318345216/rts
$ sbatch /astro/mwavcs/vcs/1318345216/batch/RTS_1318345216.batch

Code Block

language	bash
theme	Midnight
title	/astro/mwavcs/vcs/1318345216/batch/RTS_1318345216.batch
collapse	true

#!/bin/bash -l

#SBATCH --export=NONE
#SBATCH --output=/astro/mwavcs/vcs/1318345216/batch/RTS_1318345216.out
#SBATCH --account=mwavcs
#SBATCH --clusters=garrawarla
#SBATCH --partition=gpuq
#
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=10240MB
#SBATCH --nice=0
#SBATCH --chdir=/astro/mwavcs/vcs/1318345216/cal/1318345216/rts
#SBATCH --time=0:45:00
#SBATCH --nodes=13
#SBATCH --cpus-per-gpu=1
#SBATCH --gres=gpu:1

ncpus=1
export OMP_NUM_THREADS=1


module use /pawsey/mwa/software/python3/modulefiles/
module load vcstools/master

module load RTS/master


cd /astro/mwavcs/vcs/1318345216/cal/1318345216/rts
export UCX_MEMTYPE_CACHE=n
srun --export=all -N 13 -n 13 rts_gpu /astro/mwavcs/vcs/1318345216/cal/1318345216/rts/rts_1318345216.in
# ^^^^ 13 = 12 coarse channels (1 node per coarse channel) + 1 master node

Code Block

language	bash
theme	Midnight
title	/astro/mwavcs/vcs/1318345216/cal/1318345216/rts/rts_1318345216.in
collapse	true

ReadAllFromSingleFile=
BaseFilename=/astro/mwavcs/vcs/1318345216/cal/1318345216/vis/*_GPUBOX
ReadGpuboxDirect=0
UseCorrelatorInput=1

ReadMetafitsFile=1
MetafitsFilename=/astro/mwavcs/vcs/1318345216/1318345216

DoCalibration=
doMWArxCorrections=1
doRawDataCorrections=1
doRFIflagging=0
useFastPrimaryBeamModels=0
generateDIjones=1
applyDIcalibration=1
UsePacketInput=0
UseThreadedVI=1

MaxFrequency=170.24
ObservationFrequencyBase=138.895
ObservationTimeBase=2455846.1252314816
ObservationPointCentreHA=-0.35952189154909175
ObservationPointCentreDec=-5.98757073636307
ChannelBandwidth=0.04
NumberOfChannels=32

CorrDumpsPerCadence=128
CorrDumpTime=1.0
NumberOfIntegrationBins=6
NumberOfIterations=1

StartProcessingAt=0

ArrayPositionLat=-26.7033194
ArrayPositionLong=116.6708152
ArrayNumberOfStations=128

ArrayFile=

SourceCatalogueFile=/astro/mwavcs/vcs/1318345216/cal/1318345216/srclist_pumav3_EoR0aegean_EoR1pietro+ForA_1318345216_patch1000.txt
NumberOfCalibrators=1
NumberOfSourcesToPeel=0
calBaselineMin=20.0
calShortBaselineTaper=40.0
FieldOfViewDegrees=1

Make a "calibration" metafits

In order to be able to use this calibration solution for beamforming, mwalib has to be "tricked" into getting the frequency resolution right (i.e. whatever we set during the offline correlation step). To do this, make a copy of the metafits file and change

FINECHAN → 40 (kHz)

NCHANS → 768

MODE → 'HWLFILES'

I then saved this file to /astro/mwavcs/vcs/1318345216/cal/1318345216_cal.metafits.

(MEMO TO SELF: write a utility script to do this for you.)

Beamform

For the time being, we have to "trick" the beamformer into thinking this is a legacy VCS observation, instead of an MWAX VCS observation. I did this by creating a copy of the metafits file and changing the "MODE" and "NCHANS" fields to 'VOLTAGE_START' and 3072, respectively. [MEMO TO SELF: add an option to "force" the beamformer to treat it as a legacy obs, if you want to beamform from the .dat files produced by the offline PFB.]

Transfer the flagged tiles from the RTS

Code Block

language	bash
theme	Midnight

$ cd /astro/mwavcs/vcs/1318345216/vcsbeam
$ /PATH/TO/rts_flag_ant_to_tilenames.py 1318345216_cal.metafits flagged_tiles.txt > flagged_tilenames.txt

Code Block

language	bash
theme	Midnight
title	/astro/mwavcs/vcs/1318345216/vcsbeam/flagged_tilenames.txt
collapse	true

HexS31
HexE16
Tile017
Tile091
Tile098
Tile097
Tile096
Tile081
Tile088
Tile087
Tile064
HexE7
HexE6
HexS16

Code Block

language	bash
theme	Midnight

$ cd /astro/mwavcs/vcs/1318345216/vcsbeam
$ echo "00:34:08.8703 -07:21:53.409" > pointings.txt
$ sbatch vcsbeam.sbatch

Code Block

language	bash
theme	Midnight

$ module load singularity
$ /pawsey/mwa/singularity/presto/presto.sif prepfold -psr J0034-0721 -nosearch -nodmsearch C001_1318345216_00:34:08.87_-07:21:53.41_ch109-120_000?.fits

Versions Compared

Old Version 3

New Version Current

Key

VCSBeam version

Offline FinePFB

Incoherent Beam

Calibration

Offline Correlator

RTS

Create source list

Set up flagged tiles

Set up flagged channels

Selecting only the last 38% of the observation for calibration

Extra flags

Run the RTS

Make a "calibration" metafits

Beamform

Transfer the flagged tiles from the RTS

Page Comparison

Versions Compared

Old Version 3

New Version Current

Key

VCSBeam version

Offline FinePFB

Incoherent Beam

Calibration

Offline Correlator

RTS

Create source list

Set up flagged tiles

Set up flagged channels

Selecting only the last 38% of the observation for calibration

Extra flags

Run the RTS

Make a "calibration" metafits

Beamform

Transfer the flagged tiles from the RTS