Update to MWA Clock reticulation

In response to Marcin's recent comments about unusual accumulated delays ('stretched cables') in calibration solutions at times when some (but not all) receivers are rebooted, I have made some small changes to the MWA station clock reticulation, to see if we can reduce/remove these creeping errors as well as build some resilience to "upsets" to the CSIRO station clock network immediately after building power failures/cycles.

Also, it seemed to me that these occasional "length changes" have only been apparent after the installation of a (MWA) Rubidium source and White-rabbit clock retic hardware earlier this year (designed to retic the 10MHz and 1pps signals to the Telstra hut over buried fibre). I was very careful about installing this to isolate any possible impacts to MWA, but could've inadvertently upset something.

We suspect that the issue Marcin is seeing could be caused by a slow drift of the 163.84MHz MWA station clock, relative to the GPS/H-Maser derived 11pps "wall clock ticks". The HP Synth is disciplined by a 10MHz reference from CSIRO, and the CSIRO 1-pps is separately encoded onto the clock fibres via an independent path. This station clock drift could be due to CSIRO master clock settling time, or changes in stability or drive levels after (or caused by) the recent power issues, or I could've introduced a problem when the Rubidium was installed.

Partial evidence supporting this theory was a 'message' observed today on the error log of the MWA HewlettPackard Synth which indicates a loss-of-external-sync at some unknown time since the Synth was last powered up. Unfortunately there is no way to time-stamp the message(s) since the Synth has no idea of world-clock time (it only accepts a 10MHz ref input).

In any case, the previous retic layout was:

  CSIRO 10MHz -> HP Synth (163.84MHz sig gen) -> MWA Rx clock drivers

  CSIRO 1-pps -> BNC "T"

    BNC T.1 -> MWA 1-pps encoder -> MWA Rx clock drivers.

    BNC T.2 -> MWA Rubidium source.

(Note that while there are multiple separate 10MHz and 1pps slave outputs from our Rubidium, and some of these are used for MWA External instruments support, the Rube guarantees good isolation between these slave outputs/instruments).

I'm happy that our Rubidium is maintaining long-term lock with the CSIRO 1-pps (it has front panel and serial console indicators to track the lock). In addition the Rube does not "jolt" on loss/re-establishment of the 1pps external reference. I believe that with the Rubidium thus disciplined, its 10MHz and 1pps slave outputs are as good as (or possibly better than!) the CSIRO provided 10MHz/1-pps signals, as well as being local to the MWA racks instead of carried over long coax cables from the CSIRO clock system.

I am also fairly happy that the MWA receivers' regenerated 1-pps faithfully tracks the CSIRO 1-pps whenever an "arm-clock" is issued. It's barely possible the drive levels weren't quite right with "two loads" from the one "source" (via the BNC T) but we don't think we are seeing any evidence of that.

However I wasn't happy with the cable crimp on our end of the (long) 10MHz cable from CSIRO, so I decided lock our HP synth of one of the local MWA Rubidium 10MHz outputs over a new, shorter, properly crimped BNC Cable. And while I was at it, altered the 1-pps path as well to make our Rube the only "load" on CSIRO's source.

I had then to re-arm-clock all the receivers since their timing was a little scrambled when the HP synth temporarily lost its 10MHz ref as I recabled things.

Therefore the new retic layout is:

  CSIRO 10MHz (with dodgy crimp) - not connected

  CSIRO 1pps -> MWA Rubidium (direct connection)

    MWA Rube 10MHz slave -> MWA HP Synth (163.84MHz) -> MWA Rx clock drivers.

    MWA Rube 1-pps slave -> MWA 1-pps encoder -> MWA Rx clock drivers.

Time will tell if this fixes the "stretched cables" problem and or results in an improvement in reliability.