Legacy Correlator Issues

Owned by Greg Sleap
Last updated: Mar 16, 2023 by Dev Null

This page is to capture the general types of issues (with some examples) encountered with the Legacy MWA correlator. Many of these issued had previously gone unnoticed, because of the overly permissive assumptions it made about gpubox aligmnent. With Birli, we see these issues more clearly.

Diagnostic Tools

Astropy / FitsHeader

One of the simplest ways to extract detailed diagnostic information from the metafits and raw gpubox file formats of the legacy correlator is to use the fitsheader  command line tool provided by Astropy. This Python library is also useful for correcting erroneous header values so that the files can be correctly processed. To load Astropy on Garrawarla:

module use /pawsey/mwa/software/python3/modulefiles
module load python astropy

Issue Details

NAXIS2 Mismatch

Symptoms

  • number of frequency channels that we would expect based on the value of the FINECHAN header does not match the dimensions of the raw files.
    • metafits number of frequency channels is 1280/FINECHAN
    • raw file hdus should have dimensions (number of baselines, nubmer of fine channels).

Example

> fitsheader 1107478352.metafits --keyword FINECHAN --extension 0
# HDU 0 in 1107478352.metafits:
FINECHAN=                 40.0 / [kHz] Fine channel width - correlator freq_res 


# ^ we would expect there to be 32 fine channels.

> fitsheader 1107478352_20150209005219_gpubox01_00.fits --keyword NAXIS* --extension 0
# HDU 1 in 1107478352_20150209005219_gpubox01_00.fits:
NAXIS   =                    2 / number of data axes                            
NAXIS1  =                66048 / length of data axis 1                          
NAXIS2  =                  256 / length of data axis 2  

# ^ but there are actually 256 fine channels.

Potential fixes

update the metafits

export obsid=...
python - <<EoF
from astropy.io import fits
hdus=fits.open('${obsid}.metafits')
hdus[0].header['FINECHAN']=10
hdus.writeto('${obsid}.fixed.metafits')
EoF

Historical info

For the old correlator, the NAXIS values represent the actual, working correlator mode when the data was collected, while the metafits file represents what mode the user wanted the correlator to be in. Back in those days, the correlator never saw or knew anything about metafits files - the mode was changed using magic 'mode change' observations, so if one of those mode change observations was accidentally deleted, it would keep taking data in the previous mode. That happened quite often. If there's a discrepancy, you should use those, and fix (or ignore) the metafits file.

MILLITIM header fixed to zero gives duplicate HDU timestamps

Symptoms

  • Groups of adjacent HDUs in the same gpubox file have the timestamp, according to the combination of the TIME  and MILLITIME headers of each HDU
  • Only visible with with non-integer INTTIME  (e.g. 0.5s)

Example

> fitsheader 1059505936_20130802191158_gpubox01_00.fits --keyword '*TIM*'
# HDU 0 in 1059505936_20130802191158_gpubox01_00.fits:
TIME    =           1375470717 / Unix time (seconds)
MILLITIM=                    0 / Milliseconds since TIME 

# HDU 1 in 1059505936_20130802191158_gpubox01_00.fits:
TIME    =           1375470717 / Unix time (seconds)
MILLITIM=                    0 / Milliseconds since TIME

Potential fixes

  • Use a script to update the incorrect MILLITIM  headers

Tried to move past end of file

Symptoms

  • first N bytes of a fits file are all nulls
  • fitsino can't read the file
  • fitscheck shows error code 1

Example

> xxd -a 1351246440_20221031101342_ch118_001.fits | head

0000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
*
05fe000: 45a0 0abb 43fe f3d9 c3b1 0e54 4429 92db  E...C......TD)..
05fe010: c3f2 8c0c c49e 098a 445a 93da c39f 4862  ........DZ....Hb
05fe020: c564 433a c50f 3a33 43e8 8d3c c46b 351b  .dC:..:3C..<.k5.
05fe030: 44c1 5521 43a5 8126 43ca b4ae c458 5af6  D.U!C..&C....XZ.
05fe040: c4e9 0f1d 44e6 e0b2 4550 a628 43af bdf6  ....D...EP.(C...
05fe050: c49d 81ae 4488 ce0e 4472 516b 42ba 47d3  ....D...DrQkB.G.
05fe060: 453b 3915 c51b e58c 44bb 51b6 c439 c772  E;9.....D.Q..9.r
05fe070: 4479 888e c3c9 8fc3 c454 74b2 43ce 69c0  Dy.......Tt.C.i.


> fitscheck -ci 1351246440_20221031101342_ch118_001.fits

NONCOMPLIANT '1351246440_20221031101342_ch118_001.fits' ..  Verification reported errors: HDUList's 0th element is not a primary HDU. Primary HDU does not contain an EXTEND keyword equal to T even though there are extension HDUs. HDUList's element 1 is not an extension HDU. HDU 0:     'SIMPLE' card does not exist.     'BITPIX' card does not exist.     'NAXIS' card does not exist.     Card 88931:         Card 'D5' is not FITS standard (equal sign not at column 8).         Card 'D5' is not FITS standard (invalid value string: '??????\x7f?Da7?D??ND??\x1d?L7?E\x0c?\x1bD??D?[??C(??CP&\x1aD??R?\x03?\x1c?,?\x04?4????AuD\x07\x03???"?D?E=').     Card 96164:         Unprintable string '\x0f\x07>\x16\x7f???K??\x16?????Z?U@???Y?????@ko1@\x1e@\x05?/???7?\x1c?>?6>?A\x05?K\x00%>?j?>??\x04?3\x16s?3.\x0b???'; commentary cards may only contain printable ASCII characters     Card 97343:         Card '?H??' is not FITS standard (equal sign not at column 8).         Card keyword '?h??' is not upper case.         Illegal keyword name '?H??'         Card '?H??' is not FITS standard (invalid value string: 'B?\x16,r???\x11??\x03?????>8?????/?8???\\?t>.|?>?\x18????U???e?b?w>?J%>????\x05?*>??????a').    Card 109720:         Card '>?\n@' is not FITS standard (equal sign not at column 8).         Illegal keyword name '>?\n@'         Card '>?\n@' is not FITS standard (invalid value string: '??@A\\\x01??2???i??3?z??O??\x0c???N??>?P??\nZ?=1?g?Q}???!??[m?@??\x03>%?\x10?z?\x11@\x16???\x1e\tY').     Card 114951:         Card '????A?' is not FITS standard (equal sign not at column 8).         Illegal keyword name '????A?'         Card '????A?' is not FITS standard (invalid value string: '????????DHq?DE???\x11??C???C?<?D}(\x16??sHDE??C????u?????#D?\t\x1eD?wr??Vq??cl??1*').     Card 118755:         Illegal keyword name '?"???\x19??'         Card '?"???\x19??' is not FITS standard (invalid value string: ':???zg?*?)?M?e???\x12?#?????m?.????\n[?#T?@,\x1e??\x1d??>K?\x10???\x1f?\x0b\x08??\x18?o?&\x0cm?B?\\').     Card 119530:         Card keyword '@+r??;??' is not upper case.         Illegal keyword name '@+R??;??'         Card '@+R??;??' is not FITS standard (invalid value string: '?\x1f?\x10\x16N>?7??p&???????????f???Ad??\x1e4?#G??(S???w_????@\x19\x1e??[g?????=????KIk').     Card 123398:         Card '>?' is not FITS standard (equal sign not at column 8).         Illegal keyword name '>?'         Card '>?' is not FITS standard (invalid value string: 'A\x14??@\x02?a??BA??\x7fY=??Z???Z?????_?\x12?\x7fg0@??%@tf?????@????\x1c>??????\x0c3?@*???d\x14??U5?').     Card 123910:         Card '?;\x06?=\x0f??' is not FITS standard (equal sign not at column 8).        Illegal keyword name '?;\x06?=\x0f??'         Card '?;\x06?=\x0f??' is not FITS standard (invalid value string: '\x10??\x1f?I\x00\x00\x00\x00?\x00\x00\x00\x00\x00\x00\x00?\x00\x00\x00?_N???????7\x1f>?PV\x00\x00\x00\x00?\x00\x00\x00\x00\x00\x00\x00?\x00\x00\x00>?;??\x02?\x0e>\x13\x07?<anj').     Card 124608:         Illegal keyword name '>?8J??\x11?'         Card '>?8J??\x11?' is not FITS standard (invalid value string: '~ ?5??\x00\x00\x00\x00?\x00\x00\x00\x00\x00\x00\x00?\x00\x00\x00?+d?=??\n??!Y?r??\x00\x00\x00\x00?\x00\x00\x00\x00\x00\x00\x00?\x00\x00\x00???X<???<IG!?*@?').     Card 128964:         Card '=????J3N' is not FITS standard (equal sign not at column 8).         Card keyword '=????J3n' is not upper case.         Illegal keyword name '=????J3N'         Card '=????J3N' is not FITS standard (invalid value string: '??=0?`\x00\x00\x00\x00?\x00\x00\x00\x00\x00\x00\x00?\x00\x00\x00??????\x17??\x1f?????s\x00\x00\x00\x00?\x00\x00\x00\x00\x00\x00\x00?\x00\x00\x00??+?????>??????7').     Card 131088:         Card keyword '??\x11?>??e' is not upper case.         Illegal keyword name '??\x11?>??E'         Card '??\x11?>??E' is not FITS standard (invalid value string: "??@Nq??\x1a1N>????\x1f?\x7f?$?^??\x11???ZY?@B'??g\x15?zo[??s\x0f?D\x19????#>?\x1fa?????\x08#o?d?!").     Card 132518:         Unprintable string '|h??y@??\r??e?\x1e@Q~??Ul5?Ae?@=???.?B??T?@????s???>???y??>K?n=?zR@\x13PY@\x03?\x18<????&\x14F'; commentary cards may only contain printable ASCII characters     Card 133575:         Card '?P\x7fC' is not FITS standard (equal sign not at column 8).         Illegal keyword name '?P\x7fC'         Card '?P\x7fC' is not FITS standard (invalid value string: '??????G?\x14??\x7f??$9@?\x05/?=???>\x11?????\x19?\x0f)?>??????h?????GQZ>???=????\x05? ??\x19[?R\x019').     Card 154186:         Card 'D{?SE\x18' is not FITS standard (equal sign not at column 8).         Illegal keyword name 'D{?SE\x18'         Card 'D{?SE\x18' is not FITS standard (invalid value string: 'D?c????.D??\x03?#\x00??\x0cE{??p???\x05???mu?D=??xD?D_?B?\x00???\x10?CC?G\x1b?/\x04?????E\x02\r%?C??').     Card 210578:         Card '??\x00C?N' is not FITS standard (equal sign not at column 8).         Illegal keyword name '??\x00C?N'         Card '??\x00C?N' is not FITS standard (invalid value string: 'D??f?\x05\x7f]?\x0fc"C?.\x0e?\x02\x08d?D=d???\x1e??3f?\'2\x0f???\x14E\x17?K????D?w+E=t:??U?E!>???n?E(E?').     Card 214222:       Card '??\x1dDD\x14' is not FITS standard (equal sign not at column 8).         Illegal keyword name '??\x1dDD\x14'         Card '??\x1dDD\x14' is not FITS standard (invalid value string: '?6????R?????D?A~C?\t0??g??O?eDa?\x1aE\x07G??\t??E?A|ETi?B?I\n?$?JECz??W??B?>?D?{?').     Card 226452:         Card 'C?' is not FITS standard (equal sign not at column 8).         Illegal keyword name 'C?'         Card 'C?' is not FITS standard (invalid value string: '?c?\x16?tS???y?C ?\x13E\x089\x1cD??x?????\x1c\rP?\x11??D??\x1aEL?yE\x08??CE??E\x05??EN?\x0eD:n?D?qi?+?<CKN\x16'). Note: astropy.io.fits uses zero-based indexing.

Context

something corrupted raw file in its journey from the correlator to your filesystem

HDU timestamps out of sync between gpubox files

Symptoms

  • One or more gpubox files have timestamps that fall in between the timestamps of the other gpuboxes
  • Only visible with INTTIME > 1s
  • NoCommonTimesteps  when processing with MWALib, Birli or Hyperdrive

Example

# TODO 1092080416

Potential fixes

  • Process groups of gpuboxes with common timesteps separately

Data ends ahead of schedule

Symptoms

  • some or all gpubox files have less timesteps than specified in the metafits
  • in severe cases, the data can run out before GOODTIME, meaning all data is considered contaminated.

Example

# TODO 1088203096