...
For general information on WODEN, including installation notes, in-depth operating principles, primary beam details and a user guide, see https://woden.readthedocs.io/en/latest/. This should be your first port of call to understand WODEN as this will be kept up to date before all other documentation. Read on below on for cluster specifics.
For an idea of what you can do with WODEN, here is the --help from run_woden.py. Most options can be read in automagically from a metafits file.
...
| language | bash |
|---|
| theme | Midnight |
|---|
...
Garrawarla guide
An example sbatch script on Garrwarla looks like the below. This will setup an 'array job' to launch 24 separate GPU process, one for each 'coarse band' (think equivalent to GPUbox files). This will simulate a high-band EoR1 observation on 'CHIPS settings, with a resolution of 80kHz and 8s, creating uvfits files that can be directly used by CHIPS. The rest of the observational settings we'll get from the metafits file. I've commented each line of the command below to explain what's it's doing - if you copy and edit this example, you'll need to get rid of the comments as they will confuse sbatch.
| Code Block |
|---|
| language | bash |
|---|
| theme | Midnight |
|---|
| title | run_woden_EoR1.sh |
|---|
|
#!/bin/bash --login
#SBATCH --nodes=1 ##Ask for one node per job in the array
#SBATCH --partition=gpuq ##Use Set the frequencyGPU (Hz)queue
of the lowest channel for band 1. If#SBATCH --gres=gpu:1 ##Only ask for one GPU per process
#SBATCH --time=00:10:00 ##This simulation takes less than 10 mins per usingGPU
a metafits file, this will override the frequency in the
#SBATCH --account=mwaeor ##If you aren't MWA EoR, you'll need a different account
#SBATCH --mem=10gb ##Probably more memory that needed, but some calcs happen on CPU
#SBATCH metafits--ntasks=1 --coarse_band_width COARSE_BAND_WIDTH ##Only one process per job in array
#SBATCH --array=1-24 ##Launch 24 jobs, with env variable Set the width of each coarse band If using a metafits file,
$SLURM_ARRAY_TASK_ID=1 through 24 inclusive
module use /pawsey/mwa/software/python3/modulefiles ##Use Chris' fancy module load things
module load woden this will override the frequency in the metafits --num_freq_channels NUM_FREQ_CHANNELS ##Load up everything you need for WODEN
cd Number of fine frequency channels to simulate - defaults to/astro/mwaeor/jline/test_WODEN/MWA_analytic-vs-FEE ##Where you want to run WODEN
mkdir -p data --coarse_band_width / --freq_res --freq_res FREQ_RES Fine channel frequnecy resolution (Hz) - will default to what##If you want to store outputs somewhere else, make a dir
time run_woden.py \
--ra0=60.0 --dec0=-27.0 \ ##Set the phase iscentre inof the metafits
TIME OPTIONS:simulation to ra, dec = 60, -27.0 degrees
--num_timefreq_steps NUM_TIME_STEPS
channels=16 --freq_res=80e+3 \ ##Simulate 16 frequency channels per coarse band, of resolution 80kHz
--num_time_steps=14 --time_res=8.0 \ ##Simulate 14 time steps, of resolution The8 numberseconds
of time steps to simualte. Defaults --cat_filename=/astro/mwaeor/jline/software/WODEN/woden-srclist_pumav3.txt \ ##Path to howa manyWODEN-style aresky model
inthe metafits if using metafits
--time_res TIME_RES Time resolution (s) - will default to what is in the metafits
if the metafits if using metafits
OBSERVATION OPTIONS:
--ra0 RA0 RA of the desired phase centre (deg)
--dec0 DEC0 Dec of the desired phase centre (deg)
--date DATE Initial UTC date of the observatio in format YYYY-MM-
DDThh:mm:ss This is used to set the LST and array precession.
--metafits_filename=/astro/mwaeor/jline/software/WODEN/examples/metafits/1136380296_metafits_ppds.fits \ ##Path to a metafits file containing observational settings
--band_nums=$SLURM_ARRAY_TASK_ID \ ##Use the array job number to simulate different coarse bands - 1 is lowest freq, 24 highest. Uses values in metafits file to set freq.
--output_uvfits_prepend=./data/MWA_EoR1_FEE \ ##What to name the outputs. For --band_nums=1, this will make ./data/MWA_EoR1_FEE_band01.uvfits
--primary_beam=MWA_FEE \ ##Which primary beam to use - this one uses the MWA Fully Embedded Element Beam (with no interpolation)
--sky_crop_components ##WODEN defaults to horizon cropping sky model by SOURCE, but this catalogue happens to only have one giant SOURCE
|
Simply run this with the command
| Code Block |
|---|
| language | bash |
|---|
| theme | Midnight |
|---|
|
$ sbatch run_woden_EoR1.sh |
and it will launch your 24 GPU jobs separately.
WODEN options
For an idea of what you can do with WODEN, here is the --help from run_woden.py. Most options can be read in automagically from a metafits file (again, check out https://woden.readthedocs.io/en/latest/ for examples and explanations on how to run WODEN).
| Code Block |
|---|
| language | text |
|---|
| theme | Midnight |
|---|
|
$ FREQUENCY OPTIONS:
--band_nums BAND_NUMS
This is set automatically when reading a metafits but including Defaults to running 24 coarse bands. Alternatively, enter
this will override the date in the metafits TELESCOPE OPTIONS: required numbers --latitudedelineated LATITUDEby commas, Latitude (deg) of the array - defaults to MWA at -26.7033194444e.g. --band_nums=1,7,9
--longitude LONGITUDElowest_channel_freq LOWEST_CHANNEL_FREQ
LongitudeSet the frequency (degHz) of the arraylowest -channel defaultsfor toband MWA1. atIf
116.670813889 --array_height ARRAY_HEIGHT using a metafits file, this Height (m) ofwill override the frequency in the
array above sea level - defaults to MWA at metafits
--coarse_band_width COARSE_BAND_WIDTH
377.0 --array_layout ARRAY_LAYOUT Set the width of each coarse band If using a Insteadmetafits offile,
reading the array layout from the metafits file, this will override the frequency in the metafits
read from a text file. Store antenna positions as offset from --num_freq_channels NUM_FREQ_CHANNELS
Number of fine frequency channels to simulate array- centre,defaults into
east, north, height coords (metres) --primary_beam PRIMARY_BEAM --coarse_band_width / --freq_res
--freq_res FREQ_RES WhichFine primarychannel beamfrequnecy toresolution use(Hz) in- thewill simulation.default to what
Options are: is in the metafits
TIME OPTIONS:
--num_time_steps NUM_TIME_STEPS
- MWA_FEE (MWA fully embedded element model) The number of time steps to simualte. Defaults to how many are
- MWA_analy (MWA analytic model) inthe metafits if using metafits
--time_res TIME_RES Time resolution (s) - Gaussianwill (Analyticdefault symmetricto Gaussian)what is in the metafits
see --gauss_beam_FWHM and if the metafits if using metafits
OBSERVATION OPTIONS:
--ra0 RA0 and --gauss_beam_ref_freq for
RA of the desired phase centre (deg)
--dec0 DEC0 Dec of the desired phase centre (deg)
fine control --date DATE Initial UTC date of the observatio in format YYYY-MM-
- EDA2 (Analytic dipole with a ground mesh) DDThh:mm:ss This is used to set the LST and array precession.
- none (Don't use a primary beam at all) This is set automatically when reading a metafits but including
Defaults to --primary_beam=none --gauss_beam_FWHM GAUSS_BEAM_FWHM this will override the date in the metafits
TheTELESCOPE OPTIONS:
FWHM of--latitude theLATITUDE Gaussian beam inLatitude (deg) of the array - WODEN defaults to using MWA at -26.7033194444
--longitude LONGITUDE
20 deg if this isLongitude not(deg) setof the array --gauss_beam_ref_freq GAUSS_BEAM_REF_FREQ
defaults to MWA at 116.670813889
--array_height ARRAY_HEIGHT
The frequency at which the gaussHeight beam(m) FWHMof isthe setarray at.above Ifsea notlevel - defaults to MWA at
set, WODEN will default to 150MHz377.0
--gaussarray_ra_pointlayout GAUSSARRAY_RA_POINTLAYOUT
TheInstead of initialreading RAthe (deg)array tolayout pointfrom the Gaussianmetafits beamfile,
at. This will read from a betext usedfile. toStore calculateantenna anpositions houras angleoffset atfrom
which the beam will array centre, in east, remainnorth, pointedheight atcoords for(metres)
the duration of the observation. Defaults--primary_beam PRIMARY_BEAM
Which primary beam to theuse RAin of the metafitssimulation.
if available, or the RA of the phase Options are:
centre if not --gauss_dec_point GAUSS_DEC_POINT - MWA_FEE (MWA fully embedded element model)
The initial Dec (deg) to point the Gaussian beam at. Defaults - MWA_analy (MWA analytic model)
to the Dec of the metafits if available, or the Dec of the - Gaussian (Analytic symmetric Gaussian)
phase centre if not see --hdf5gauss_beam_path HDF5_BEAM_PATHFWHM and
Location of the hdf5 file holding the FEE beam coefficients
--MWA_FEE_delays MWA_FEE_DELAYSand --gauss_beam_ref_freq for
fine control
A list of 16 delays to point the- MWAEDA2 primary(Analytic beamdipole with a ground mesh)
model enter as as list like:- none (Don't use a primary beam at all)
--MWA_FEE_delays=[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0] Defaults to --primary_beam=none
--gauss_beam_FWHM GAUSS_BEAM_FWHM
for a zenith pointing. This is read directly from The FWHM of the Gaussian beam in deg - WODEN defaults to using
the20 metafitsdeg if usingthis ais metafitsnot fileset
--telescope_name TELESCOPE_NAMEgauss_beam_ref_freq GAUSS_BEAM_REF_FREQ
NameThe offrequency telescopeat writtenwhich outthe togauss thebeam uvfitsFWHM file,is defaultsset toat. If not
MWA INPUT/OUTPUT OPTIONS:set, WODEN will default to 150MHz.
--catgauss_ra_filenamepoint CATGAUSS_RA_FILENAMEPOINT
PathThe toinitial WODENRA style(deg) skyto modelpoint the Gaussian --metafits_filename METAFITS_FILENAME
beam at. This will
MWA style metafitsbe fileused to basecalculate thean simulationhour on.angle Arrayat which the beam will
layout, frequency and time parametersremain arepointed allat setfor bythe thisduration of the observation. Defaults
option, but can be overriddento usingthe otherRA argumentsof the --output_uvfits_prepend OUTPUT_UVFITS_PREPEND
metafits if available, or the RA of the phase
Prepend name for uvfits - will appendcentre band%02d.uvfits %band_numif not
--gauss_dec_point GAUSS_DEC_POINT
atThe theinitial end. DefaultsDec (deg) to "output".
--sky_crop_components
point the Gaussian beam at. Defaults
WODEN willto cropthe outDec skyof modelthe informationmetafits thatif isavailable, belowor the Dec of the
horizon for the given LST. By default, for each SOURCE in thephase centre if not
--hdf5_beam_path HDF5_BEAM_PATH
Location of skythe model,hdf5 iffile anyholding COMPONENTthe isFEE belowbeam thecoefficients
horizon, the entire
--MWA_FEE_delays MWA_FEE_DELAYS
source willA belist flagged.of If --sky_crop_components is included
16 delays to point the MWA primary beam
WODEN will include any COMPONENT abovemodel theenter horizon,as regardlessas list like:
of which SOURCE it belongs to.
SIMULATOR OPTIONS:
--removeMWA_phase_tracking
FEE_delays=[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
By adding this flag, remove the phase tracking of the for a zenith pointing. This is read directly from
visibilities - use this to feed uvfits into the RTSmetafits if using a metafits file
--notelescope_tidyname TELESCOPE_NAME
Defaults to deleting output binary files from woden and json Name of telescope written out to the uvfits file, defaults to
files. Add this flag to not delete those files --chunking_size CHUNKING_SIZE MWA
INPUT/OUTPUT OPTIONS:
--cat_filename CAT_FILENAME
The chunk size to break up the point sources intoPath forto WODEN style sky model
--metafits_filename METAFITS_FILENAME
processing - defaults MWA style metafits file to 0base (usethe defaultsimulation chunkingon. inArray
WODEN) --dry_run Add this to NOT call the WODEN executable -layout, thisfrequency willand justtime parameters are all set by this
write out the .json file and dooption, nothing else |
Garrawarla guide
An example sbatch script on Garrwarla looks like the below. This will setup an 'array job' to launch 24 separate GPU process, one for each 'coarse band' (think equivalent to GPUbox files). This will simulate a high-band EoR1 observation on 'CHIPS settings, with a resolution of 80kHz and 8s, creating uvfits files that can be directly used by CHIPS. The rest of the observational settings we'll get from the metafits file. I've commented each line of the command below to explain what's it's doing - if you copy and edit this example, you'll need to get rid of the comments as they will confuse sbatch.
| Code Block |
|---|
| language | bash |
|---|
| theme | Midnight |
|---|
| title | run_woden_EoR1.sh |
|---|
|
but can be overridden using other arguments
--output_uvfits_prepend OUTPUT_UVFITS_PREPEND
Prepend name for uvfits - will append band%02d.uvfits %band_num
at the end. Defaults to "output".
--sky_crop_components
WODEN will crop out sky model information that is below the
horizon for the given LST. By default, for each SOURCE in the
sky model, if any COMPONENT is below the horizon, the entire
source will be flagged. If --sky_crop_components is included
WODEN will include any COMPONENT above the horizon, regardless
of which SOURCE it belongs to.
SIMULATOR OPTIONS:
--remove_phase_tracking
By adding this flag, remove the phase tracking of the
visibilities - use this to feed uvfits into the RTS
--no_tidy Defaults to deleting output binary files from woden and json
files. Add this flag to not delete those files
--chunking_size CHUNKING_SIZE
The chunk size to break up the point sources into for
processing - defaults to 0 (use default chunking in WODEN)
--dry_run Add this to NOT call the WODEN executable - this will just
write out the .json file and do nothing else |
Alternative Garrawarla script
Provided by Himanshu Tiwari
| Code Block |
|---|
#!/bin/bash --login
#SBATCH --nodes=1 ##Ask for one node per job in the array
#SBATCH --partition=gpuq ##Use the GPU queue
#SBATCH --gres=gpu:1 ##Only ask for one GPU per process
#SBATCH --time=00:10:00 ##This simulation takes less than 10 mins per GPU
#SBATCH --account=mwaeor ##If you aren't MWA EoR, you'll need a different account
#SBATCH --mem=10gb ##Probably more memory that needed, but some calcs happen on CPU
#SBATCH --ntasks=1 ##Only one process per job in array
#SBATCH --array=1-24 ##Launch 24 jobs, with env variable $SLURM_ARRAY_TASK_ID=1 through 24 inclusive
module use /pawsey/mwa/software/python3/modulefiles ##Use
Chris' fancy module load things
module load wodensingularity ##Load up everything you need for WODEN
cd /astro/mwaeor/jline/test_WODEN/MWA_analytic-vs-FEE ##Where you want to##Load runup WODENsingularity
mkdir -p data ##If you want to store outputs somewhere else, make a dir
singularity exec
time--nv path/to/woden/dockerimage run_woden.py \
--ra0=60.0 --dec0=-27.0 \ ##Set the phase centre of the simulation to ra, dec = 60, -27.0 degrees
--num_freq_channels=16 --freq_res=80e+3 \ ##Simulate 16 frequency channels per coarse band, of resolution 80kHz
--num_time_steps=14 --time_res=8.0 \ ##Simulate 14 time steps, of resolution 8 seconds
--cat_filename=/astro/mwaeor/jline/software/WODEN/woden-srclist_pumav3.txt \ ##Path to a WODEN-style sky model
--metafits_filename=/astro/mwaeor/jline/software/WODEN/examples/metafits/1136380296_metafits_ppds.fits \ ##Path to a metafits file containing observational settings
--band_nums=$SLURM_ARRAY_TASK_ID \ ##Use the array job number to simulate different coarse bands - 1 is lowest freq, 24 highest. Uses values in metafits file to set freq.
--output_uvfits_prepend=./data/MWA_EoR1_FEE \ ##What to name the outputs. For --band_nums=1, this will make ./data/MWA_EoR1_FEE_band01.uvfits
--primary_beam=MWA_FEE \ ##Which primary beam to use - this one uses the MWA Fully Embedded Element Beam (with no interpolation)
--sky_crop_components ##WODEN defaults to horizon cropping sky model by SOURCE, but this catalogue happens to only have one giant SOURCE
|
Simply run this with the command
| Code Block |
|---|
| language | bash |
|---|
| theme | Midnight |
|---|
|
$ sbatch run_woden_EoR1.sh |
...
Related articles
...