Legacy Correlator Issues
This page is to capture the general types of issues (with some examples) encountered with the Legacy MWA correlator. Many of these issued had previously gone unnoticed, because of the overly permissive assumptions it made about gpubox aligmnent. With Birli, we see these issues more clearly.
- 1 Diagnostic Tools
- 2 Issue Details
- 2.1 NAXIS2 Mismatch
- 2.1.1 Symptoms
- 2.1.2 Example
- 2.1.3 Potential fixes
- 2.1.4 Historical info
- 2.2 MILLITIM header fixed to zero gives duplicate HDU timestamps
- 2.2.1 Symptoms
- 2.2.2 Example
- 2.2.3 Potential fixes
- 2.3 Tried to move past end of file
- 2.4 HDU timestamps out of sync between gpubox files
- 2.4.1 Symptoms
- 2.4.2 Example
- 2.4.3 Potential fixes
- 2.5 Data ends ahead of or after schedule
- 2.1 NAXIS2 Mismatch
Diagnostic Tools
Astropy / FitsHeader
One of the simplest ways to extract detailed diagnostic information from the metafits and raw gpubox file formats of the legacy correlator is to use the fitsheader command line tool provided by Astropy. This Python library is also useful for correcting erroneous header values so that the files can be correctly processed. To load Astropy on Setonix:
module use py-pip/default
pip install --user astropy
Note that Pawsey policy says don’t use default module versions, but if I provided instructions with a specific version number, then those instructions would need to be updated every year or so and I’m not doing that.
Birli dry run
module use birli/default
birli --dry-run -m *.metafits -- 1*_2*.fits
Issue Details
NAXIS2 Mismatch
Symptoms
number of frequency channels that we would expect based on the value of the FINECHAN header does not match the dimensions of the raw files.
metafits number of frequency channels is 1280/FINECHAN
raw file hdus should have dimensions (number of baselines, nubmer of fine channels).
Example
> fitsheader 1107478352.metafits --keyword FINECHAN --extension 0
# HDU 0 in 1107478352.metafits:
FINECHAN= 40.0 / [kHz] Fine channel width - correlator freq_res
# ^ we would expect there to be 32 fine channels.
> fitsheader 1107478352_20150209005219_gpubox01_00.fits --keyword NAXIS* --extension 0
# HDU 1 in 1107478352_20150209005219_gpubox01_00.fits:
NAXIS = 2 / number of data axes
NAXIS1 = 66048 / length of data axis 1
NAXIS2 = 256 / length of data axis 2
# ^ but there are actually 256 fine channels.
Potential fixes
update the metafits
export obsid=...
python - <<EoF
from astropy.io import fits
hdus=fits.open('${obsid}.metafits')
hdus[0].header['FINECHAN']=10
hdus.writeto('${obsid}.fixed.metafits')
EoF
Historical info
For the old correlator, the NAXIS values represent the actual, working correlator mode when the data was collected, while the metafits file represents what mode the user wanted the correlator to be in. Back in those days, the correlator never saw or knew anything about metafits files - the mode was changed using magic 'mode change' observations, so if one of those mode change observations was accidentally deleted, it would keep taking data in the previous mode. That happened quite often. If there's a discrepancy, you should use those, and fix (or ignore) the metafits file.
MILLITIM header fixed to zero gives duplicate HDU timestamps
Symptoms
Groups of adjacent HDUs in the same gpubox file have the timestamp, according to the combination of the
TIME
andMILLITIME
headers of each HDUOnly visible with with non-integer
INTTIME
(e.g. 0.5s)
Example
> fitsheader 1059505936_20130802191158_gpubox01_00.fits --keyword '*TIM*'
# HDU 0 in 1059505936_20130802191158_gpubox01_00.fits:
TIME = 1375470717 / Unix time (seconds)
MILLITIM= 0 / Milliseconds since TIME
# HDU 1 in 1059505936_20130802191158_gpubox01_00.fits:
TIME = 1375470717 / Unix time (seconds)
MILLITIM= 0 / Milliseconds since TIME
Potential fixes
Use a script to update the incorrect
MILLITIM
headers
Tried to move past end of file
Symptoms
first N bytes of a fits file are all nulls
fitsino can't read the file
fitscheck shows error code 1
Example
> xxd -a 1351246440_20221031101342_ch118_001.fits | head
0000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................
*
05fe000: 45a0 0abb 43fe f3d9 c3b1 0e54 4429 92db E...C......TD)..
05fe010: c3f2 8c0c c49e 098a 445a 93da c39f 4862 ........DZ....Hb
05fe020: c564 433a c50f 3a33 43e8 8d3c c46b 351b .dC:..:3C..<.k5.
05fe030: 44c1 5521 43a5 8126 43ca b4ae c458 5af6 D.U!C..&C....XZ.
05fe040: c4e9 0f1d 44e6 e0b2 4550 a628 43af bdf6 ....D...EP.(C...
05fe050: c49d 81ae 4488 ce0e 4472 516b 42ba 47d3 ....D...DrQkB.G.
05fe060: 453b 3915 c51b e58c 44bb 51b6 c439 c772 E;9.....D.Q..9.r
05fe070: 4479 888e c3c9 8fc3 c454 74b2 43ce 69c0 Dy.......Tt.C.i.
> fitscheck -ci 1351246440_20221031101342_ch118_001.fits
NONCOMPLIANT '1351246440_20221031101342_ch118_001.fits' .. Verification reported errors: HDUList's 0th element is not a primary HDU. Primary HDU does not contain an EXTEND keyword equal to T even though there are extension HDUs. HDUList's element 1 is not an extension HDU. HDU 0: 'SIMPLE' card does not exist. 'BITPIX' card does not exist. 'NAXIS' card does not exist. Card 88931: Card 'D5' is not FITS standard (equal sign not at column 8). Card 'D5' is not FITS standard (invalid value string: '??????\x7f?Da7?D??ND??\x1d?L7?E\x0c?\x1bD??D?[??C(??CP&\x1aD??R?\x03?\x1c?,?\x04?4????AuD\x07\x03???"?D?E='). Card 96164: Unprintable string '\x0f\x07>\x16\x7f???K??\x16?????Z?U@???Y?????@ko1@\x1e@\x05?/???7?\x1c?>?6>?A\x05?K\x00%>?j?>??\x04?3\x16s?3.\x0b???'; commentary cards may only contain printable ASCII characters Card 97343: Card '?H??' is not FITS standard (equal sign not at column 8). Card keyword '?h??' is not upper case. Illegal keyword name '?H??' Card '?H??' is not FITS standard (invalid value string: 'B?\x16,r???\x11??\x03?????>8?????/?8???\\?t>.|?>?\x18????U???e?b?w>?J%>????\x05?*>??????a'). Card 109720: Card '>?\n@' is not FITS standard (equal sign not at column 8). Illegal keyword name '>?\n@' Card '>?\n@' is not FITS standard (invalid value string: '??@A\\\x01??2???i??3?z??O??\x0c???N??>?P??\nZ?=1?g?Q}???!??[m?@??\x03>%?\x10?z?\x11@\x16???\x1e\tY'). Card 114951: Card '????A?' is not FITS standard (equal sign not at column 8). Illegal keyword name '????A?' Card '????A?' is not FITS standard (invalid value string: '????????DHq?DE???\x11??C???C?<?D}(\x16??sHDE??C????u?????#D?\t\x1eD?wr??Vq??cl??1*'). Card 118755: Illegal keyword name '?"???\x19??' Card '?"???\x19??' is not FITS standard (invalid value string: ':???zg?*?)?M?e???\x12?#?????m?.????\n[?#T?@,\x1e??\x1d??>K?\x10???\x1f?\x0b\x08??\x18?o?&\x0cm?B?\\'). Card 119530: Card keyword '@+r??;??' is not upper case. Illegal keyword name '@+R??;??' Card '@+R??;??' is not FITS standard (invalid value string: '?\x1f?\x10\x16N>?7??p&???????????f???Ad??\x1e4?#G??(S???w_????@\x19\x1e??[g?????=????KIk'). Card 123398: Card '>?' is not FITS standard (equal sign not at column 8). Illegal keyword name '>?' Card '>?' is not FITS standard (invalid value string: 'A\x14??@\x02?a??BA??\x7fY=??Z???Z?????_?\x12?\x7fg0@??%@tf?????@????\x1c>??????\x0c3?@*???d\x14??U5?'). Card 123910: Card '?;\x06?=\x0f??' is not FITS standard (equal sign not at column 8). Illegal keyword name '?;\x06?=\x0f??' Card '?;\x06?=\x0f??' is not FITS standard (invalid value string: '\x10??\x1f?I\x00\x00\x00\x00?\x00\x00\x00\x00\x00\x00\x00?\x00\x00\x00?_N???????7\x1f>?PV\x00\x00\x00\x00?\x00\x00\x00\x00\x00\x00\x00?\x00\x00\x00>?;??\x02?\x0e>\x13\x07?<anj'). Card 124608: Illegal keyword name '>?8J??\x11?' Card '>?8J??\x11?' is not FITS standard (invalid value string: '~ ?5??\x00\x00\x00\x00?\x00\x00\x00\x00\x00\x00\x00?\x00\x00\x00?+d?=??\n??!Y?r??\x00\x00\x00\x00?\x00\x00\x00\x00\x00\x00\x00?\x00\x00\x00???X<???<IG!?*@?'). Card 128964: Card '=????J3N' is not FITS standard (equal sign not at column 8). Card keyword '=????J3n' is not upper case. Illegal keyword name '=????J3N' Card '=????J3N' is not FITS standard (invalid value string: '??=0?`\x00\x00\x00\x00?\x00\x00\x00\x00\x00\x00\x00?\x00\x00\x00??????\x17??\x1f?????s\x00\x00\x00\x00?\x00\x00\x00\x00\x00\x00\x00?\x00\x00\x00??+?????>??????7'). Card 131088: Card keyword '??\x11?>??e' is not upper case. Illegal keyword name '??\x11?>??E' Card '??\x11?>??E' is not FITS standard (invalid value string: "??@Nq??\x1a1N>????\x1f?\x7f?$?^??\x11???ZY?@B'??g\x15?zo[??s\x0f?D\x19????#>?\x1fa?????\x08#o?d?!"). Card 132518: Unprintable string '|h??y@??\r??e?\x1e@Q~??Ul5?Ae?@=???.?B??T?@????s???>???y??>K?n=?zR@\x13PY@\x03?\x18<????&\x14F'; commentary cards may only contain printable ASCII characters Card 133575: Card '?P\x7fC' is not FITS standard (equal sign not at column 8). Illegal keyword name '?P\x7fC' Card '?P\x7fC' is not FITS standard (invalid value string: '??????G?\x14??\x7f??$9@?\x05/?=???>\x11?????\x19?\x0f)?>??????h?????GQZ>???=????\x05? ??\x19[?R\x019'). Card 154186: Card 'D{?SE\x18' is not FITS standard (equal sign not at column 8). Illegal keyword name 'D{?SE\x18' Card 'D{?SE\x18' is not FITS standard (invalid value string: 'D?c????.D??\x03?#\x00??\x0cE{??p???\x05???mu?D=??xD?D_?B?\x00???\x10?CC?G\x1b?/\x04?????E\x02\r%?C??'). Card 210578: Card '??\x00C?N' is not FITS standard (equal sign not at column 8). Illegal keyword name '??\x00C?N' Card '??\x00C?N' is not FITS standard (invalid value string: 'D??f?\x05\x7f]?\x0fc"C?.\x0e?\x02\x08d?D=d???\x1e??3f?\'2\x0f???\x14E\x17?K????D?w+E=t:??U?E!>???n?E(E?'). Card 214222: Card '??\x1dDD\x14' is not FITS standard (equal sign not at column 8). Illegal keyword name '??\x1dDD\x14' Card '??\x1dDD\x14' is not FITS standard (invalid value string: '?6????R?????D?A~C?\t0??g??O?eDa?\x1aE\x07G??\t??E?A|ETi?B?I\n?$?JECz??W??B?>?D?{?'). Card 226452: Card 'C?' is not FITS standard (equal sign not at column 8). Illegal keyword name 'C?' Card 'C?' is not FITS standard (invalid value string: '?c?\x16?tS???y?C ?\x13E\x089\x1cD??x?????\x1c\rP?\x11??D??\x1aEL?yE\x08??CE??E\x05??EN?\x0eD:n?D?qi?+?<CKN\x16'). Note: astropy.io.fits uses zero-based indexing.
Context
something corrupted raw file in its journey from the correlator to your filesystem
HDU timestamps out of sync between gpubox files
Symptoms
One or more gpubox files have timestamps that fall in between the timestamps of the other gpuboxes
Only visible with
INTTIME
> 1sNoCommonTimesteps
when processing with MWALib, Birli or Hyperdrive
Example
# TODO 1092080416
Potential fixes
Process groups of gpuboxes with common timesteps separately
Data ends ahead of or after schedule
Symptoms
some or all gpubox files have more or less timesteps than specified in the metafits
in severe cases, the data can run out before GOODTIME, meaning all data is considered contaminated.
more or less timestamps that expected when processing with MWALib, Birli or Hyperdrive
Example
birli --dry-run -m *.metafits -- 1*_2*.fits
Scheduled start: 2017-08-01 19:42:46.000 UTC, unix=1501616566.000, gps=1185651784.000, mjd=5008333366.000, lmst= 2.9687°, lmst2k= 2.7492°, lat2k=-26.8000°
Scheduled end: 2017-08-01 19:44:38.000 UTC, unix=1501616678.000, gps=1185651896.000, mjd=5008333478.000, lmst= 3.4366°, lmst2k= 3.2176°, lat2k=-26.7999°
Scheduled duration: 112.000s = 56 * 2.000s
Quack duration: 2.000s = 1 * 2.000s
Output duration: 114.000s = 57 * 2.000s
...
Timestep details (all=57, provided=57, common=57, good=56, select=57, flag=1):
2017-08-01 UTC + unix [s] gps [s] p c g s f
ts0: 19:42:47.000 1501616567.000 1185651785.000 p c s f
ts1: 19:42:49.000 1501616569.000 1185651787.000 p c g s
...
ts55: 19:44:37.000 1501616677.000 1185651895.000 p c g s
ts56: 19:44:39.000 1501616679.000 1185651897.000 p c g s
each timestamp is the START of the integration, so according to what was scheduled, the last timestamp should have started at GPS 1185651894
it started 1 second late and ended 3 seconds late relative to what was scheduled
observatory recommends flagging 2 seconds (1 timestamp) after scheduled start of observation, so it only flags the first timestamp
birli selects all "common" timestamps, and flags any timestamps that aren't "good", these are mwalib conventions:
common = common to all gpu box files
good = outside of quack time