...
| Code Block |
|---|
| language | bash |
|---|
| theme | Midnight |
|---|
| title | offline_correlate.sbatch |
|---|
| collapse | true |
|---|
|
#!/bin/bash -l
#SBATCH --nodes=1
#SBATCH --mem=370gb
#SBATCH --partition=gpuq
#SBATCH --gres=gpu:1
#SBATCH --time=01:00:00
#SBATCH --account=mwavcs
#SBATCH --export=NONE
module use /pawsey/mwa/software/python3/modulefiles
module load offline_correlator
for t in {0..299}
do
gpssecond=$(echo "$t + 1318345220" | bc)
gpubox=$(echo "FREQ - 108" | bc)
INPUT_DATA_FILE=/astro/mwavcs/vcs/1318345216/combined/1318345216_${gpssecond}_chFREQ.dat
START_SECOND=${gpssecond}
DUMPS_PER_SECOND=1 # This sets the output time resolution
# (e.g. 20 --> 1/20 = 0.05s = 50 ms)
# Minimum allowed resolution is 2 ms
CHANS_TO_AVERAGE=4 # This sets the output frequency resolution
# (e.g. 4 --> 4x10 kHz = 40 kHz)
GPUBOX_CHANNEL_NUMBER=${gpubox} # This should be chosen to "match" the input channel
# This is not easy! (mwalib handles this, but at the
# moment, offline_correlator is not using mwalib)
OUTPUT_PREFIX=1318345216 # Output files begin with this
srun -N 1 -n 1 offline_correlator \
-d ${INPUT_DATA_FILE} \
-s ${START_SECOND} \
-r ${DUMPS_PER_SECOND} \
-n ${CHANS_TO_AVERAGE} \
-c ${GPUBOX_CHANNEL_NUMBER} \
-o ${OUTPUT_PREFIX}
done |
RTS
Create source list
| Code Block |
|---|
| language | bash |
|---|
| theme | Midnight |
|---|
| title |
|---|
| # | | |
|
$ cd /astro/mwavcs/vcs/1318345216/cal/1318345216
$ module load srclists/v1.0.0
$ srclist_by_beam.py -m /astro/mwavcs/vcs/1318345216/1318345216_metafits_ppds.fits -n 1000 -s ${SRCLISTS_DIR}/srclist_pumav3_EoR0aegean_EoR1pietro+ForA.txt |
Set up flagged tiles
To find which tiles need to be flagged, find the broken tiles by going to the observation status page (http://ws.mwatelescope.org/observation/obs/?obs_id=1318345216). In the section "Tile errors and flags", you will find the "TileNames" for any tiles that should be flagged. Then, in the metafits file for the observation, you will find the "Input" numbers associated with the X and Y polarisations of each tile. The numbers you need to put in the "flagged_tiles.txt" are the Input numbers divided by two (and possibly rounded down). For example, in 1318345216, Tile064 is listed as broken, and in the metafits file, that tile's polarisations are associated with inputs 144 & 145. This means that the number that goes into flagged_tiles.txt for this tile is 72. In this way, the following list is constructed for this observation.
| Code Block |
|---|
| language | bash |
|---|
| theme | Midnight |
|---|
| title | /astro/mwavcs/vcs/1318345216/cal/1318345216/rts/flagged_tiles.txt |
|---|
|
1
# Run RTS
12
45
51
60
72
93
94
112 |
Set up flagged channels
Setting up the flagged channels is more straightforward. As a rule, we always flag the edge channels (80 kHz worth on either side) and the centre channel. Since we have averaged up to 40 kHz when we did the offline correlation step above, that means we will be flagging the outer two channels on either side (as 2 x 40 kHz = 80 kHz) and the single centre channel. Since there are 32 channels in this case, the final list of flagged channels is:
| Code Block |
|---|
| language | bash |
|---|
| theme | Midnight |
|---|
| title | /astro/mwavcs/vcs/1318345216/cal/1318345216/rts/flagged_tiles.txt |
|---|
|
0
1
16
30
31 |
Run the RTS
| Code Block |
|---|
| language | bash |
|---|
| theme | Midnight |
|---|
| title | Run the RTS |
|---|
|
$ cd /astro/mwavcs/vcs/1318345216/cal/1318345216/rts
$ sbatch /astro/mwavcs/vcs/1318345216/batch/RTS_1318345216.batch |
...
| Code Block |
|---|
| language | bash |
|---|
| theme | Midnight |
|---|
| title | /astro/mwavcs/vcs/1318345216/batch/RTS_1318345216.batch |
|---|
| collapse | true |
|---|
|
#!/bin/bash -l
#SBATCH --export=NONE
#SBATCH --output=/astro/mwavcs/vcs/1318345216/batch/RTS_1318345216.out
#SBATCH --account=mwavcs
#SBATCH --clusters=garrawarla
#SBATCH --partition=gpuq
#
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=10240MB
#SBATCH --nice=0
#SBATCH --chdir=/astro/mwavcs/vcs/1318345216/cal/1318345216/rts
#SBATCH --time=2:00:00
#SBATCH --nodes=13
#SBATCH --cpus-per-gpu=1
#SBATCH --gres=gpu:1
ncpus=1
export OMP_NUM_THREADS=1
module use /pawsey/mwa/software/python3/modulefiles/
module load vcstools/master
module load RTS/master
cd /astro/mwavcs/vcs/1318345216/cal/1318345216/rts
export UCX_MEMTYPE_CACHE=n
srun --export=all -N 13 -n 13 rts_gpu /astro/mwavcs/vcs/1318345216/cal/1318345216/rts/rts_1318345216.in
# ^^^^ 13 = 12 coarse channels (1 node per coarse channel) + 1 master node |
| Code Block |
|---|
| language | bash |
|---|
| theme | Midnight |
|---|
| title | /astro/mwavcs/vcs/1318345216/cal/1318345216/rts/rts_1318345216.in |
|---|
| collapse | true |
|---|
|
ReadAllFromSingleFile=
BaseFilename=/astro/mwavcs/vcs/1318345216/cal/1318345216/vis/*_gpubox
ReadGpuboxDirect=0
UseCorrelatorInput=1
ReadMetafitsFile=1
MetafitsFilename=/astro/mwavcs/vcs/1318345216/1318345216
DoCalibration=
doMWArxCorrections=1
doRawDataCorrections=1
doRFIflagging=0
useFastPrimaryBeamModels=0
generateDIjones=1
applyDIcalibration=1
UsePacketInput=0
UseThreadedVI=1
MaxFrequency=170.24
ObservationFrequencyBase=138.895
ObservationTimeBase=2455846.1252314816
ObservationPointCentreHA=-0.35952189154909175
ObservationPointCentreDec=-5.98757073636307
ChannelBandwidth=0.04
NumberOfChannels=32
CorrDumpsPerCadence=128
CorrDumpTime=1.0
NumberOfIntegrationBins=6
NumberOfIterations=1
StartProcessingAt=0
ArrayPositionLat=-26.7033194
ArrayPositionLong=116.6708152
ArrayNumberOfStations=128
ArrayFile=
SourceCatalogueFile=/astro/mwavcs/vcs/1318345216/cal/1318345216/srclist_pumav3_EoR0aegean_EoR1pietro+ForA_1318345216_patch1000.txt
NumberOfCalibrators=1
NumberOfSourcesToPeel=0
calBaselineMin=20.0
calShortBaselineTaper=40.0
FieldOfViewDegrees=1 |
...