The PSRDADA format is used internally within the MWAX correlator and beamformer components. It is not user-facing.
Pre Correlator
This is the header that is created by mwax_u2s when creating a subfile from the multicast UDP. It therefore describes headers that MWAX VCS users would see in the VCS subfiles.
| Keyword | Valid values or Example | Used By | Description |
|---|---|---|---|
| HDR_SIZE | 4096 | PSRDADA | Required by PSRDADA - size of header in bytes (should always be 4096) |
| POPULATED | 0 | 1 | mwax_db2correlate2db | Used to ensure the correlator does not read an incomplete file. 0 means the file is not yet complete, 1 means the file is ready for consumption |
| OBS_ID | 1244973688 | mwax-db2fits | Obs_id of this observation |
| SUBOBS_ID | 1244973688 | mwax-db2fits | GPS second of this part of the observation. Increments in 8 sec intervals |
| MODE | MWAX_CORRELATOR MWAX_VCS NO_CAPTURE | mwax_db2correlate2db mwax-db2fits | This matches the "mode" from the existing metadata database/schedule. |
| UTC_START | 2018-10-11-05:26:14 | PSRDADA mwax-db2fits | Required by PSRDADA. This is the UTC start date/time of the subobservation(?). Used by mwax-db2fits to append a human readable date/time to the filename produced. |
| OBS_OFFSET | 0 (n seconds since start of observation) | PSRDADA mwax-db2fits | Required by PSRDADA. I believe this is the number of seconds offset from the start of the observation. First subobservation will have this set to 0. Second subobservation will be 8, etc. |
NBIT | 8 | mwax_db2correlate2db | Bits per sample (should always be 8) |
| NPOL | 2 | mwax_db2correlate2db mwax-db2fits | Polarisations per tile (i.e. rf_chains) (should always be 2) |
| NTIMESAMPLES | 64000 | mwax_db2correlate2db | Number of timesamples in each input sub-block of a subobservation (nominally sub-blocks are 50 ms) |
| NINPUTS | 256 | n rfinputs | mwax_db2correlate2db | Number of signal paths present in the input subobservation file, to be correlated (must be <= NINPUTS_XGPU) |
| NINPUTS_XGPU | 256 | n rfinputs in multiples of 16 | mwax_db2correlate2db | Number of signal paths actually correlated by xGPU (must be a multiple of 16; must match the number of paths that xGPU has been configured for) |
APPLY_PATH_WEIGHTS | 0 | 1 | mwax_db2correlate2db | Should scalar path weights be applied prior to correlation (for gain normalisation and/or sub-arraying) |
| APPLY_PATH_DELAYS | 0 | 1 | mwax_db2correlate2db | Should fractional path delays be applied prior to correlation (for phasing up to a correlation pointing centre) |
| APPLY_PATH_PHASE_OFFSETS | 0 | 1 | mwax_db2correlate2db | Should path phase offsets be applied prior to correlation (for fringe-stopping, in conjunction with applying path delays) |
| INT_TIME_MSEC | 200 - 8000 | mwax_db2correlate2db mwax-db2fits | Correlator integration time (milliseconds) |
| FSCRUNCH_FACTOR | 40 (1 - 6400) | mwax_db2correlate2db mwax-db2fits | Correlator frequency scrunch factor: number of ultrafine 200 Hz fine channels to average together |
| APPLY_VIS_WEIGHTS | 0 | 1 | mwax_db2correlate2db | Should data occupancy weights be applied to normalise the output visibilities |
TRANSFER_SIZE | 5275652096 | PSRDADA mwax_db2correlate2db | Total number of bytes of input data for one subobservation (of nominally 8 seconds) |
PROJ_ID | G0008 | mwax-db2fits | Project ID of observation |
| EXPOSURE_SECS | 8 (in incremements of 8 seconds) | mwax-db2fits | Duration of observation in seconds |
| COARSE_CHANNEL | 76 (0 - max receiver channel number == 255) | mwax-db2fits | Receiver channel number |
| CORR_COARSE_CHANNEL | 9 | (1 - coarse channel max) | mwax-db2fits | Correlator channel number - which correlator channel "index" is this? e.g. first channel would be 1. In a 24 channel system, last channel would be 24. This would map to gpubox01-24 in the legacy Correlator |
| SECS_PER_SUBOBS | 8 | mwax_db2correlate2db mwax-db2fits | Number of seconds per subobservation |
UNIXTIME | 1539235574 | mwax-db2fits | Unix time of the start of this observation. This field is used by mwax-db2fits to provide a UNIX timestep for each integration throughout the observation. |
| UNIXTIME_MSEC | 0 | mwax-db2fits | Unix time milliseconds offset of the start of this observation. This will always be 0 with the current setup, but is here for completeness. |
FINE_CHAN_WIDTH_HZ | 10000 | mwax-db2fits | Correlator output fine channel width in Hz |
| NFINE_CHAN | 128 | mwax_db2correlate2db mwax-db2fits | Correlator output number of fine channels |
| BANDWIDTH_HZ | 1280000 | mwax-db2fits | Coarse channel passband width in Hz, i.e. the channel spacing |
SAMPLE_RATE | 1280000 | mwax_db2correlate2db | Actual sample rate of input data in samples/s (whether critically-sampled or oversampled). |
| MC_IP | 0.0.0.0 | mwax-db2fits | Multicast IP for this coarse channel (for debug purposes) |
| MC_PORT | 0 | mwax-db2fits | Multicast port for this coarse channel (for debug purposes) |
| MWAX_U2S_VER | X.Y.Z | mwax_db2fits | Version number of mwax_u2s used to capture the UDP packets for this subfile |
| MWAX_SUB_VER | 1 | mwax_u2s | Version number of subfile format. A value of 2 indicates the presence of the following fields: |
| IDX_PACKET_MAP | (OFFSET)+(SIZE)¹ | mwax_u2s | Location and size of packet map in block 0 metadata |
| IDX_DELAY_TABLE | (OFFSET)+(SIZE)¹ | mwax_u2s | Location and size of delay table in block 0 metadata |
| IDX_MARGIN_DATA | (OFFSET)+(SIZE)¹ | mwax_u2s | Location and size of margin data section in block 0 metadata |
¹ These fields constitute an index into the block 0 metadata. For example a value of 1234+567 denotes a section beginning at byte 1234 which is 567 bytes long. This is to avoid the need for downstream consumers to calculate these locations based on other fields, and allows for future restructuring without breaking applications that use these index fields to locate the information they are interested in. Currently inserted by mwax_u2s and supported by subtool, but not yet used by mwax_db2correlate2db or mwalib.
Post Correlator changes
The below changes are made during the mwax_db2correlate2db process. The only recipient of this updated header is mwax_db2fits, which uses it to produce the output visibility FITS files.
| Keyword | Valid values or Example | Used By | Description |
|---|---|---|---|
NBIT | 32 | mwax-db2fits | Bits per complex visibility component (real | imaginary) |
| NTIMESAMPLES | 1 - 40 (for 8 s subobservations) | ? | Number of integration times (visibility sets produced) per subobservation |
TRANSFER_SIZE | n bytes | mwax-db2fits | Total number of bytes of output data for this 8 second subobservation. Calculated by: bytes = visibilities + weights visibilities = integrations * baselines * fine channels * polarisations * bytes_real * bytes_imaginary weights = integrations * baselines * polarisations * bytes_real |
| MWAX_DB2CORR2DB_VER | X.Y.Z | mwa-db2fits | Version number of the correlator executable used to perform the F and X stages of correlation |
Block 0
Voltage data samples are stored in blocks 1-160, but the PSRDADA block numbering scheme is 0-indexed, so the contents of block 0 are user-defined. In our case, a number of data tables are stored in this area. They are:
- The delay table, storing whole- and fractional-sample delay values, used for cable delay corrections and fringe stopping.
- The margin data table, storing the complete payload of the packet immediately before and after the subobservation, as well as the first and last packet within it. This permits whole-sample delays applied by u2s to be undone, or a different set of delays to be applied.
- The UDP packet map. This is a bitmap recording precisely which of the expected packets for the subobservation were seen, and which were missing.
- The occupancy table. A coarser analogue of the UDP packet map, storing a count of packets received per 5ms time slice, which can be used directly by the correlator for calculating weights.
- (Not currently implemented) A copy of the metafits file at the time the subfile was created.
Block 0: Delay table
The delay table records whole- and fractional-sample delays for cable delays and geometric delays. It is stored in a densely-packed binary format, with each row corresponding to an RF source, in the same order as the voltage data. There are 10 fixed columns per row, followed by a column for each fractional delay according to a granularity specified externally (currently 1600, for a total of 1610 columns) but also calculable from data in the table (the num_pointings column). All numbers are little-endian. The schema is as follows:
| # | Name | Type | Offset | Length | Description |
|---|---|---|---|---|---|
| 0 | rf_input | uint16 | 0 | 2 | RF input ID. This is the tile ID right-shifted by 1,with the lower bit indicating polarity: 0 for X,1 for Y. |
| 1 | ws_delay | int16 | 2 | 2 | Whole sample delay. This indicates how many samples that u2s has shifted the data stream forward (negative values indicate a backwards shift). |
| 2 | initial_delay | float64 | 4 | 8 | The calculated delay at the start of the subobservation, in milliseconds. |
| 3 | delta_delay | float64 | 12 | 8 | The rate of change of the delay, in milliseconds per FFT block (currently 5ms). |
| 4 | delta_delta_delay | float64 | 20 | 8 | The rate of change of the delta_delay, in milliseconds per FFT block (currently 5ms). |
| 5 | start_total_delay | float64 | 28 | 8 | Equivalent to initial_delay, first point for quadratic interpolation of fractional delay values. |
| 6 | middle_total_delay | float64 | 36 | 8 | Second point for quadratic interpolation of fractional delay values, delay at middle of subobservation. |
| 7 | end_total_delay | float64 | 44 | 8 | Third point for quadratic interpolation of fractional delay values, delay at end of subobservation. |
| 8 | num_pointings | uint16 | 52 | 2 | The number of fractional delay values in the row, N. |
| 9 | _reserved | uint16 | 54 | 2 | Reserved for future use. Must always be 0. Used by subtool for automatic detection of delay table format in exported delay tables. |
| 10 .. N | frac_delay[N] | float32 | 56 | 4N | Fractional delay values, in microsamples. |