Recombining means converting legacy VCS data from one format to another. The two formats are both headerless binary formats, with each byte representing a (4+4)-bit complex signed sample. The only main difference between the formats is the ordering of the bytes and how they are distributed between multiple files. For the purposes of this document, the "from" format will be called the PFB or unrecombined format, and the "to" format will be called the VCS or recombined format. The format conversion is necessary because the beamformer only (currently) supports the VCS format as input.
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
recombine -o <obsid> -t <secondid> -m <meta-data fits> -i <output dir>
-c <skip course chan> -s <skip ICS> -f <file list> or -g <input file list>
<obsid>: observation id of the data being processed.
<secondid>: the second which is being processed
<meta-data fits>: meta-data fits file containing tile flag information and various orther useful
information regarding the observation. To obtain the meta-data fits file for a particular observation
use the following:
wget -O <obsid>.metafits http://ws.mwatelescope.org/metadata/fits?obs_id=<obsid>
<output dir>: output product directory
<skip course chan>: 1 will skip the generation of the recombined course channel data
<skip ICS>: 1 will skip the generation of the incoherent sum
<input file list>: location of 32 raw uncombined input files for a single seconds worth of data (separate each with a space)
<input file list>: a file containing the location of the 32 raw uncombined input files (separate by newline) |
...
Processing on Garrawarla
Examples of using recombine to process both single-second and multiple-second jobs on Garrawarla is now provided as part of the documentation included in the recombine repository.
Other (wrapper) scripts
vcs_download.nf
vcs_download.nf is a Nextflow script provided by the mwa_search repo. Its use is described on the main Documentation page. As a quick reference, however, the following template can be followed on Garrawarla:
| Code Block | ||||
|---|---|---|---|---|
| ||||
#!/bin/bash -l
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --mem=370gb
#SBATCH --time=00:10:00
#SBATCH --partition=workq
#SBATCH --account=mwavcs
#SBATCH --job-name=recombine_test
#SBATCH --output=recombine_test.out
module use /pawsey/mwa/software/python3/modulefiles
module load recombine
srun -n 1 -N 1 recombine \
-o 1267111608 \
-t 1267111610 \
-m /astro/mwavcs/vcs/1267111608/1267111608_metafits_ppds.fits \
-i . \
-s 1 \
-f /astro/mwavcs/vcs/1267111608/raw/1267111608_1267111610_vcs*.dat |
Other scripts
vcs_download.nf
...
module load vcstools/devel
module load mwa_search/devel
vcs_download.nf --obsid <obsid> --begin <begin gps> --end <end gps> --download_dir /astro/mwavcs/asvo/<ASVO ID> --vcstools_version devel -resume |
Note that vcs_download.nf will remove the PFB files once they have been successfully recombined.
process_vcs.py (deprecated)
...