Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • Use a script to update the incorrect MILLITIM  headers

HDU timestamps out of sync between gpubox files

Symptoms

...

Tried to move past end of file

Symptoms

  • first N bytes of a fits file are all nulls
  • fitsino can't read the file
  • fitscheck shows error code 1

Example

Code Block
# TODO 1092080416

Potential fixes

  • Process groups of gpuboxes with common timesteps separately

Data ends ahead of schedule

Symptoms

  • some or all gpubox files have less timesteps than specified in the metafits
  • in severe cases, the data can run out before GOODTIME, meaning all data is considered contaminated.

Example

Code Block
# TODO 1088203096

Tried to move past end of file

Symptoms

  • first N bytes of a fits file are all nulls
  • fitsino can't read the file
  • fitscheck shows error code 1

Image Removed

Example

Code Block
> fitscheck -ci 1351246440_20221031101342_ch118_001.fits

NONCOMPLIANT '1351246440_20221031101342_ch118_001.fits' ..  Verification reported errors: HDUList's 0th element is not a primary HDU. Primary HDU does not contain an EXTEND keyword equal to T even though there are extension HDUs. HDUList's element 1 is not an extension HDU. HDU 0:     'SIMPLE' card does not exist.     'BITPIX' card does not exist.     'NAXIS' card does not exist.     Card 88931:         Card 'D5' is not FITS standard (equal sign not at column 8).         Card 'D5' is not FITS standard (invalid value string: '??????\x7f?Da7?D??ND??\x1d?L7?E\x0c?\x1bD??D?[??C(??CP&\x1aD??R?\x03?\x1c?,?\x04?4????AuD\x07\x03???"?D?E=').     Card 96164:         Unprintable string '\x0f\x07>\x16\x7f???K??\x16?????Z?U@???Y?????@ko1@\x1e@\x05?/???7?\x1c?>?6>?A\x05?K\x00%>?j?>??\x04?3\x16s?3.\x0b???'; commentary cards may only contain printable ASCII characters     Card 97343> xxd -a 1351246440_20221031101342_ch118_001.fits | head

0000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
*
05fe000: 45a0 0abb 43fe f3d9 c3b1 0e54 4429 92db  E...C......TD)..
05fe010: c3f2 8c0c c49e 098a 445a 93da c39f 4862  ........DZ....Hb
05fe020: c564 433a c50f 3a33 43e8 8d3c c46b 351b  .dC:..:3C..<.k5.
05fe030: 44c1 5521 43a5 8126 43ca b4ae c458 5af6  D.U!C..&C....XZ.
05fe040: c4e9 0f1d 44e6 e0b2 4550 a628 43af bdf6  ....D...EP.(C...
05fe050: c49d 81ae 4488 ce0e 4472 516b 42ba 47d3  ....D...DrQkB.G.
05fe060: 453b 3915 c51b e58c 44bb 51b6 c439 c772  E;9.....D.Q..9.r
05fe070: 4479 888e c3c9 8fc3 c454 74b2 43ce 69c0  Dy.......Tt.C.i.


> fitscheck -ci 1351246440_20221031101342_ch118_001.fits

NONCOMPLIANT '1351246440_20221031101342_ch118_001.fits' ..  Verification reported errors: HDUList's 0th element is not a primary HDU. Primary HDU does not contain an EXTEND keyword equal to T even though there are extension HDUs. HDUList's element 1 is not an extension HDU. HDU 0:     'SIMPLE' card does not exist.     'BITPIX' card does not exist.     'NAXIS' card does not exist.     Card 88931:         Card '?H??D5' is not FITS standard (equal sign not at column 8).         Card keyword '?h??D5' is not upperFITS case.standard (invalid        Illegal keyword name '?H??'         Card '?H??' is not FITS standard (invalid value string: 'B?\x16,r???\x11??\x03?????>8?????/?8???\\?t>.|?>?\x18????U???e?b?w>?J%>????\x05?*>??????avalue string: '??????\x7f?Da7?D??ND??\x1d?L7?E\x0c?\x1bD??D?[??C(??CP&\x1aD??R?\x03?\x1c?,?\x04?4????AuD\x07\x03???"?D?E=').     Card 10972096164:         Unprintable Cardstring '>?\n@' is not FITS standard (equal sign not at column 8).         Illegal keyword name '>?\n@'\x0f\x07>\x16\x7f???K??\x16?????Z?U@???Y?????@ko1@\x1e@\x05?/???7?\x1c?>?6>?A\x05?K\x00%>?j?>??\x04?3\x16s?3.\x0b???'; commentary cards may only contain printable ASCII characters     Card 97343:         Card '>?H??\n@' is not FITS standard (invalidequal sign valuenot string: '??@A\\\x01??2???i??3?z??O??\x0c???N??>?P??\nZ?=1?g?Q}???!??[m?@??\x03>%?\x10?z?\x11@\x16???\x1e\tYat column 8).         Card keyword '?h??' is not upper case.         Illegal keyword name '?H??'         Card '?H??' is not FITS standard (invalid value string: 'B?\x16,r???\x11??\x03?????>8?????/?8???\\?t>.|?>?\x18????U???e?b?w>?J%>????\x05?*>??????a').     Card 114951109720:         Card '????A?>?\n@' is not FITS standard (equal sign not at column 8).         Illegal keyword name '????A?>?\n@'         Card '????A?>?\n@' is not FITS standard (invalid value string: '??@A\\\x01??2???i??3?DHqz?DE?O??\x11x0c??C?N??C>?<P??D}(\x16nZ?=1?sHDEg?Q}?C??!??u[m?@??\x03>%?\x10?#Dz?\tx11@\x1eDx16?wr??Vq??cl??1*\x1e\tY').     Card 118755114951:         Illegal keyword nameCard '?"???\x19A??' is not FITS standard (equal sign not at Card '?"column 8).         Illegal keyword name '????A?'         Card '???\x19?A?' is not FITS standard (invalid value string: ':?????zg?*?)?MDHq?eDE???\x12x11?#?C???C?m?.<?D}(\x16??sHDE??\n[C?#T?@,\x1e??\x1du??>K?\x10??#D?\t\x1f?\x0b\x08x1eD?wr??\x18Vq?o?&\x0cmcl?B?\\1*').     Card 119530118755:         CardIllegal keyword '@+rname '?"???;\x19??' is not upper case.     Card '?"???\x19??' is  Illegal keyword name '@+R??;??'         Card '@+R??;??' is not not FITS standard (invalid value string: ':?\x1f?\x10\x16N>?7zg?*?p&)?M?e???\x12?#?????fm?.??Ad??\x1e4?#G??(S???w_n[?#T?@,\x1e??\x1d??>K?\x10???\x1f?@\x19x0b\x1ex08??[g\x18?o?&\x0cm???=????KIkB?\\').     Card 123398119530:         Card '>keyword '@+r??;??' is not FITSupper standardcase. (equal sign not at column 8).   Illegal      Illegal keyword keyword name '>@+R??;??'         Card '>@+R??;??' is not FITS standard (invalid value string: 'A\x14??@\x02?a??BA??\x7fY=??Z???Z?????_?\x12?\x7fg0@??%@tf?????@????\x1c>??????\x0c3?@*???d\x14??U5??\x1f?\x10\x16N>?7??p&???????????f???Ad??\x1e4?#G??(S???w_????@\x19\x1e??[g?????=????KIk').     Card 123910123398:         Card '?;\x06?=\x0f??>?' is not FITS standard (equal sign not at column 8).         Illegal keyword name '?;\x06?=\x0f?>?'         Card '?;\x06?=\x0f??>?' is not FITS standard (invalid value string: 'A\x10x14??\x1f?I\x00\x00\x00\x00?\x00\x00\x00\x00\x00\x00\x00?\x00\x00\x00?_N?@\x02?a??BA??\x7fY=??Z???Z?????_?7\x1f>x12?PV\x00\x00\x00\x00?\x00\x00\x00\x00\x00\x00\x00?\x00\x00\x00>?;??\x02?\x0e>\x13\x07?<anj\x7fg0@??%@tf?????@????\x1c>??????\x0c3?@*???d\x14??U5?').     Card 124608123910:         Illegal keyword nameCard '>?8J;\x06??\x11?'         Card '>?8J??\x11=\x0f??' is not FITS standard (invalid value string: '~ ?5??\x00\x00\x00\x00?\x00\x00\x00\x00\x00\x00\x00?\x00\x00\x00?+d?=??\n??!Y?r??equal sign not at column 8).        Illegal keyword name '?;\x06?=\x0f??'         Card '?;\x06?=\x0f??' is not FITS standard (invalid value string: '\x10??\x1f?I\x00\x00\x00\x00?\x00\x00\x00\x00\x00\x00\x00?\x00\x00\x00?_N???X<???<IG!?*@?').  ?7\x1f>?PV\x00\x00\x00\x00?\x00\x00\x00\x00\x00\x00\x00?\x00\x00\x00>?;??\x02?\x0e>\x13\x07?<anj').     Card 128964124608:         Illegal Cardkeyword name '=>?8J??\x11?J3N'  is not FITS standard (equal sign not at column 8).         Card keyword '=>?8J??\x11?J3n' is not upperFITS case.standard (invalid value string:      Illegal keyword name '=????J3N'         Card '=????J3N' is not FITS standard (invalid value string: '??=0?`'~ ?5??\x00\x00\x00\x00?\x00\x00\x00\x00\x00\x00\x00?\x00\x00\x00?+d?=??\n??!Y?r??\x00\x00\x00\x00?\x00\x00\x00\x00\x00\x00\x00?\x00\x00\x00???X<???\x17<IG!??\x1f?????s\x00\x00\x00\x00?\x00\x00\x00\x00\x00\x00\x00?\x00\x00\x00??+?????>??????7'*@?').     Card 131088128964:         Card keyword '=??\x11?>??eJ3N' is not upper FITS standard (equal sign not at column 8).         Card keyword '=????J3n' is not upper case.         Illegal keyword name '?=?\x11?>??EJ3N'         Card '=??\x11?>??EJ3N' is not FITS standard (invalid value string: "'??@Nq??\x1a1N>????\x1f?\x7f?$?^??\x11???ZY?@B'??g\x15?zo[??s\x0f?D\x19????#>?\x1fa?????\x08#o?d?!"=0?`\x00\x00\x00\x00?\x00\x00\x00\x00\x00\x00\x00?\x00\x00\x00??????\x17??\x1f?????s\x00\x00\x00\x00?\x00\x00\x00\x00\x00\x00\x00?\x00\x00\x00??+?????>??????7').     Card 132518131088:         UnprintableCard stringkeyword '|h??y@??\r??e?\x1e@Q~??Ul5?Ae?@=???.?B??T?@????s???>???y??>K?n=?zR@\x13PY@\x03?\x18<????&\x14F'; commentary cards may only contain printable ASCII characters     Card 133575:         Card '?P\x7fC' is not FITS standard (equal sign not at column 8).         Illegal keyword name '?P\x7fC'         Card '?P\x7fC' is not FITS standard (invalid value string: '??????G?\x14??\x7f??$9@?\x05/?=???>\x11?????\x19?\x0f)?>??????h?????GQZ>???=????\x05? ??\x19[?R\x019').\x11?>??e' is not upper case.         Illegal keyword name '??\x11?>??E'         Card '??\x11?>??E' is not FITS standard (invalid value string: "??@Nq??\x1a1N>????\x1f?\x7f?$?^??\x11???ZY?@B'??g\x15?zo[??s\x0f?D\x19????#>?\x1fa?????\x08#o?d?!").     Card 132518:         Unprintable string '|h??y@??\r??e?\x1e@Q~??Ul5?Ae?@=???.?B??T?@????s???>???y??>K?n=?zR@\x13PY@\x03?\x18<????&\x14F'; commentary cards may only contain printable ASCII characters     Card 154186133575:         Card 'D{?SEP\x18x7fC' is not FITS standard (equal sign not at column 8).         Illegal keyword name 'D{?SEP\x18x7fC'         Card 'D{?SEP\x18x7fC' is not FITS standard (invalid value string: 'D???c???G?.D\x14??\x03x7f??$9@?#\x00x05/?=???>\x0cE{x11??p???\x05x19?\x0f)?>?mu?D=??xD?D_?Bh?\x00???\x10?CCGQZ>?G\x1b?/\x04?=????E\x02\r%x05?C ??\x19[?R\x019').     Card 210578154186:         Card 'D{??SE\x00C?Nx18' is not FITS standard (equal sign not at column 8).         Illegal keyword name 'D{??SE\x00C?Nx18'         Card 'D{??SE\x00C?Nx18' is not FITS standard (invalid value string: 'D?c?f?\x05\x7f]?\x0fc"C?.D??\x0ex03?#\x02\x08d?D=d???\x1e??3f?\'2\x0f???\x14E\x17?K????D?w+E=t:??U?E!>???n?E(Ex00??\x0cE{??p???\x05???mu?D=??xD?D_?B?\x00???\x10?CC?G\x1b?/\x04?????E\x02\r%?C??').     Card 214222210578:         Card '??\x1dDD\x14x00C?N' is not FITS standard (equal sign not at column 8).         Illegal keyword name '??\x1dDD\x14x00C?N'         Card '??\x1dDD\x14x00C?N' is not FITS standard (invalid value string: 'D?6?f?\x05\x7f]?\x0fc"C??R?.\x0e?\x02\x08d?D=d???\x1e?D?A~C3f?\'2\t0x0f???g\x14E\x17?K?O?eDa?\x1aE\x07G?D?\w+E=t:??U?E!>?A|ETi?B?I\n?$?JECz??W??B?>?D?{?E(E?').     Card 226452214222:         Card 'C??\x1dDD\x14' is not FITS standard (equal sign not at column 8).         Illegal keyword name 'C??\x1dDD\x14'         Card 'C??\x1dDD\x14' is not FITS standard (invalid value string: '?c6?\x16?tS??R?y?C ?\x13E\x089\x1cD??xD?A~C?\t0??g?\x1c\rP?\x11??D??\x1aEL?yE\x08??CE??E\x05??EN?\x0eD:n?D?qi?+?<CKN\x16'). Note: astropy.io.fits uses zero-based indexing.  

Context

...

?O?eDa?\x1aE\x07G??\t??E?A|ETi?B?I\n?$?JECz??W??B?>?D?{?').     Card 226452:         Card 'C?' is not FITS standard (equal sign not at column 8).         Illegal keyword name 'C?'         Card 'C?' is not FITS standard (invalid value string: '?c?\x16?tS???y?C ?\x13E\x089\x1cD??x?????\x1c\rP?\x11??D??\x1aEL?yE\x08??CE??E\x05??EN?\x0eD:n?D?qi?+?<CKN\x16'). Note: astropy.io.fits uses zero-based indexing.  

Context

something corrupted raw file in its journey from the correlator to your filesystem

HDU timestamps out of sync between gpubox files

Symptoms

  • One or more gpubox files have timestamps that fall in between the timestamps of the other gpuboxes
  • Only visible with INTTIME > 1s
  • NoCommonTimesteps  when processing with MWALib, Birli or Hyperdrive

Example

Code Block
# TODO 1092080416

Potential fixes

  • Process groups of gpuboxes with common timesteps separately

Data ends ahead of schedule

Symptoms

  • some or all gpubox files have less timesteps than specified in the metafits
  • in severe cases, the data can run out before GOODTIME, meaning all data is considered contaminated.

Example

Code Block
# TODO 1088203096