Currently still in heavy development, but is able to perform direction-independent calibration on the CPU.
More documentation: https://mwatelescope.github.com/MWATelescopeio/mwa_hyperdrive/wikiindex.html
Project homepage: https://github.com/MWATelescope/mwa_hyperdrive
...
| Code Block | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
---------------------------------- /pawsey/mwa/software/python3/modulefiles ---------------------------------- hyperdrive/chj hyperdrive/v0.2.0-alpha1alpha11 (L,D) |
Load a hyperdrive module:
| Code Block | ||||
|---|---|---|---|---|
| ||||
module load hyperdrive # this will load the default version |
hyperdrive prefers to use the FEE beam when its applicable. The associated beam code (hyperbeam) requires that the MWA FEE beam file be available at runtime; this is either done manually with a command-line argument to hyperdrive, or with the MWA_BEAM_FILE environment variable. garrawarla users typically don't need to worry about this, because hyperdrive modules automatically set MWA_BEAM_FILE.
How do I get started?
Have a look at the help text!
The following is current as of 12 August 2021.
See help text:
| Code Block | ||||
|---|---|---|---|---|
| ||||
hyperdrive -h # -h could also be --help module load hyperdrive/chj # load CHJ's development version |
Example Slurm script
| Code Block | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
hyperdrive 0.2.0-alpha1
Compiled on git commit hash: f76c8f6499349bff83b1953abf2f923f5a46ff46
head ref: refs/heads/di
at: Fri, 06 Aug 2021 07:43:20 +0000
with compiler: rustc 1.54.0 (a178d0322 2021-07-26)
Christopher H. Jordan <christopherjordan87@gmail.com>
USAGE:
hyperdrive <SUBCOMMAND>
FLAGS:
-h, --help Prints help information
-V, --version Prints version information
SUBCOMMANDS:
di-calibrate Perform direction-independent calibration on the input MWA data
simulate-vis Simulate visibilities of a sky-model source list
srclist-by-beam Reduce a sky-model source list to the top N brightest sources, given pointing information
srclist-convert Convert a sky-model source list from one format to another
srclist-verify Verify that sky-model source lists can be read by hyperdrive
help Prints this message or the help of the given subcommand(s)
|
hyperdrive is broken up into many subcommands. Each of these have their own help; e.g.
| Code Block | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
hyperdrive-simulate-vis 0.2.0-alpha1
Simulate visibilities of a sky-model source list
USAGE:
hyperdrive simulate-vis [FLAGS] [OPTIONS] --metafits <metafits> --source-list <source-list>
FLAGS:
--no-beam Should we use a beam? Default is to use the FEE beam
-v, --verbosity The verbosity of the program. The default is to print high-level information
--dry-run Don't actually do any work; just verify that the input arguments were correctly ingested and
print out high-level information
-c, --cpu Use the CPU for visibility generation. This is deliberately made non-default because using a GPU
is much faster
-h, --help Prints help information
-V, --version Prints version information
OPTIONS:
-s, --source-list <source-list> Path to the sky-model source list used for simulation
-m, --metafits <metafits> Path to the metafits file
--model-file <model-file> Path to the output visibilities file [default: model.uvfits]
-r, --ra <ra>
The phase centre right ascension \[degrees\]. If this is not specified, then the metafits phase/pointing
centre is used
-d, --dec <dec>
The phase centre declination \[degrees\]. If this is not specified, then the metafits phase/pointing centre
is used
--num-fine-channels <num-fine-channels>
The total number of fine channels in the observation [default: 384]
-f, --freq-res <freq-res> The fine-channel resolution \[kHz\] [default: 80]
--middle-freq <middle-freq>
The middle frequency of the simulation \[MHz\]. If this is not specified, then the middle frequency
specified in the metafits is used
-t, --time-steps <time-steps>
The number of time steps used from the metafits epoch [default: 14]
--time-res <time-res> The time resolution \[seconds\] [default: 8.0]
--beam-file <beam-file>
The path to the HDF5 MWA FEE beam file. If not specified, this must be provided by the MWA_BEAM_FILE
environment variable
|
DI calibration
Available with hyperdrive di-calibrate
Two main things are required to calibrate visibilities:
- A data container (e.g. measurement set); and
- A sky-model source list.
Discussion on the source lists and the applicable formats can be found here.
By default, hyperdrive will attempt to use all sources in the source list file. If there are more than 1,000 sources in the file, then it may take a long time! In order to keep the number of sources used low, one could use the -n/--num-sources and/or --veto-threshold flags, or use a source list with fewer sources in the first place (see hyperdrive srclist-by-beam).
As hyperdrive is still in heavy development, not all features are currently available. An indication of what is available is below.
- Reads raw MWA data
- Reads a single uvfits file as input
- Reads multiple uvfits files as input
- Reads a single measurement set file as input
- Reads multiple measurement set files as input
- Calibrates on the CPU
- Calibrates on a GPU
- Writes calibration solutions to the "André Offringa calibrate format"
- Writes calibration solutions in the "RTS format"
- Writes calibrated visibilities directly to uvfits output
- Writes calibrated visibilities directly to measurement set output
#!/bin/bash -l
#SBATCH --job-name=hyp-$1
#SBATCH --output=hyperdrive.out
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=40
#SBATCH --time=01:00:00
#SBATCH --clusters=garrawarla
#SBATCH --partition=gpuq
#SBATCH --account=mwaeor
#SBATCH --export=NONE
#SBATCH --gres=gpu:1,tmp:50g
module use /pawsey/mwa/software/python3/modulefiles
module load hyperdrive
set -eux
command -v hyperdrive
cd /astro/mwaeor/MWA/data/1090008640
# Get calibration solutions. Use the top 1000 sources.
hyperdrive di-calibrate \
-s /pawsey/mwa/software/python3/srclists/master/srclist_pumav3_EoR0aegean_fixedEoR1pietro+ForA_phase1+2.txt \
-n 1000 \
-d *gpubox*.fits *.metafits *.mwaf \
-o hyp_sols.fits
# Apply the solutions and write out a measurement set.
# Write it to /nvmetmp as that's much faster than /astro.
hyperdrive solutions-apply \
-d *gpubox*.fits *.metafits *.mwaf \
-s hyp_sols.fits \
-o /nvmetmp/hyp_calibrated.ms \
--time-average 8s \
--freq-average 80kHz
# Move the measurement set to /astro.
mv /nvmetmp/hyp_calibrated.ms . |
This example script reserves 50 GB of space for node local storage (/nvmetmp). If your output visibilities are bigger than this, then the write will fail; you should adjust the #SBATCH --gres=gpu:1,tmp:50g line to account for this, e.g. #SBATCH --gres=gpu:1,tmp:200g