Human-readable calibration analysis services

There are a couple of web services to support human-readable tables and plots summarising recent calibration fits. These allow:

• The quality of each fit to be judged, based on the differences between residual delays for a given tile, and either the residuals for all other tiles, or residuals for the same tile from other observations over the previous week, or between X and Y cable length residuals for all tiles, etc.

• Statistics including the residual median delay for all 8 tiles on each receiver (to adjust the receiver clock fibre lengths in the database) and for each tile (to adjust the tile cable lengths in the database).

To view these plots, open this URL in a browser:

https://ws.mwatelescope.org/calib/summary

A form at the top of the page allows you to specify a time range, to view only the fits for observations in that time range. Below that form is a table showing the fits. Note that it’s possible to have more than one calibration fit calculated for the same observation ID/s. The table columns are:

• Fit ID - for current fits, this is the time (in Unix ‘microseconds since epoch’) that the calibration fit process was performed. Click on the fit ID to view tables of median cable length residuals, both per-receiver median’s over all slots, and per-tile medians of all observations in the fit. You can use these values to update the clock fibre and tile cable lengths in the configuration database.

• Creator - the name of the pipeline that calculated the fit, currently one of ‘msok’ or ‘mwa’ for Marcin Sokolowski’s pipeline running from building 610, or ‘calvin’ for Greg Sleap’s pipeline running on site.

• Code version of the fit pipeline, eg 0.17.69

• Time the fit pipeline executed, in AWST.

• Observations - the individual observations included in that fit, expressed as ‘<obsid> =<obsname>’, where <obsid> is the observation start time in GPS seconds, and <obsname> is the name of the observation. Note that only ‘mwa’ (Sokolowski pipeline) fits will contain more than one observation in the fit. You can click on an obsid to download the calibration fit for that observation or click on the observation name to view the full information page for that observation.

• Good/Total Tiles - the numbers of good (no errors) and total tiles in that observation.

• Plot links:

  • tilediff: X and Y cable residuals for each tile (large circles), superimposed over small points showing the residuals from all other tiles in the array, in that same observation.

  • timediff: X and Y cable residuals for each tile (large circles), superimposed over small points showing the residuals from other fits for that same tile over the previous 7 days. The ‘msok’ and ‘mwa’ fits are in blue (X) and green (Y), while the ‘calvin’ fits are in red (X) and orange (Y). Data from previous fits is shown as partially transparent points, getting more transparent further back in time.

  • xydiff: The difference between the X and Y residuals for each tile, over the previous 30 days before that observation.

  • gains: X and Y gains for ech tile, over every coarse channel included in that fit.

  • gaintimes: X and Y gains for each tile, for each MWA channel in the fit, compared to previous fit data for that tile/channel. The ‘msok’ and ‘mwa’ fits are in blue (X) and green (Y), while the ‘calvin’ fits are in red (X) and orange (Y). Data from previous fits is shown as partially transparent points, getting more transparent further back in time.

Get the best calibration solution available for a target observation ID

e.g. https://ws.mwatelescope.org/calib/get_cal_solution?obs_id=1120493008 

Where:

• obs_id is an observation ID you wish to calibrate. The get_cal_solution service will search the calibration solution database for the best calibration solution available for this observation. See MWA ASVO: Calibration Option for some background information on the calibration database and how it is populated. 

• max_time_separation is a maximum offset in seconds between the given obs_id and the observation start time of the calibrator chosen to use for that observation. The default is 86400 (one day).

• allow_after is an optional integer indicating whether calibrator observations after the given obsid are allowed. Values are 0 or 1 (defaults to 1, meaning True). 

• prefer_fitid is an optional integer fit-id value. If provided, that fit will be used (assuming it’s a valid fit for that obsid). If not provided, the service will return the most recent fit for the calibration observation that is closest in time to the given target obsid, that matches the coarse channels for the target.

Note that obs_id is the target observation - one that you wish to find a calibration fit for - it doesn’t need to be flagged as a calibrator observation. However, if you provide the obsid for an observation that IS a calibrator, and it has a valid fit in the database, the service for that fit will be returned. If you want a different fit (from some other calibration observation) use the prefer_fitid option above.

This service will return:

• On Success:

  • A zip file containing:

    • <CALIBRATION_ID>.bin: Binary file (in AO-cal format) containing the calibration solution. This can be used in Andre Offringa's calibrate and Cotter tools. Note the filename contains the observation ID of the calibrator observation used.

    • <OBSERVATION_ID>_flagged_tiles.txt: A plain text file listing the tile_id's of tiles that failed to calibrate. Note the filename contains the the observation ID of the TARGET observation that you requested a calibration for.

    • <OBSERVATION_ID>_flags.py: A simple python script which you can run to apply the calibration solution (in CASA).

    • <OBSERVATION_ID>_fit_metadata.txt: A small file with name: value pairs, one per line, containing metadata about the fit (fitid, fit quality, etc). This metadata is described below in the dos for get_cal_json.

    • <OBSERVATION_ID>_fit_metadata.json: The same information in JSON format.

• On Error:
  
  

  • http status 400 if the obs_id does not exist in the schedule database, or there is any other fault in the request.

  • http status 404 if no valid calibration fit was found for an observation within 'max_time_separation' seconds from the given obs_id.

  • http status 500 if an internal error occurred.

Note that this service will only look for calibration observations taken with the same analogue attenuation ('gain_control_value') as the observation.

This service does NOT return a metafits file for the observation in the zip file - if needed, one should be requested separately using the /metadata/fits web service.

ALSO - this service does NOT request a calibration fit be calculated for the given obs_id, if one wasn't found. Use the 'add_cal_request' service (below) instead.

If you wish to request a fit for a specific calibrator observation, just pass that calibrator observations starttime in obs_id, and a 'max_time_separation' value of 0.

Get a JSON structure with the calibration fit for a given calibration observation ID

e.g. https://ws.mwatelescope.org/calib/get_cal_json?obs_id=1450126248

Where:

• obs_id is an observation ID you wish to get the calibration fit for. It must have been flagged as a calibration observation.

• prefer_fitid is an optional fitid integer (unit timestamp when the fit was run) to use, instead of the default (the most recent fit on the calibrator observation closest in time to the given obs_id).

This service will return:

• On Success, returns a JSON dictionary with tileid as key,

  • Each value is a dict with pol ('X' or 'Y') as key, and each of those values is a dict with:

    • 'delay_m':float - Fitted residual error in cable length, in metres at the speed of light. Used to generate a phase slope which is then multiplied by a frequency index.

    • ‘intercept':float - phase intercept in radians. For each fine channel, the phase correction in radians would be ((2 * pi * delay_m / c) * freq - intercept) where 'c’ is the speed of light in m/s and freq is the frequency in Hz for that fine channel.

    • 'gains':list of 24 floats - phase correction for each of the 24 coarse channels in this observation.

    • 'gains_pol1':list of 24 floats - only found in old fits - linear slope of gain across each coarse channel. For a given fine channel, this slope would be multiplied by the fine channel index from 0-767 (assuming a 40kHz calibrator resolution) across the whole observation.

    • 'gains_pol0':list of 24 floats - only found in old fits - intercept of gain across each coarse channel. Note that this value is the fitted gain at fine channel index 0 (the first fine channel in the whole observation), NOT the intercept at zero Hz, or at the first fine channel in the current coarse channel.

    • 'phase_sigma_resid':float - ?

    • 'phase_chi2dof':float - ?

    • 'phase_fit_quality':float - Phase fit quality from 0-1 (fits > 0.3 or so seem to be OK)

    • 'gains_sigma_resid':list of 24 floats - ?

    • 'gains_fit_quality':float - Gains fit quality from 0-1 (fits > 0.3 or so seem to be OK)

  • As well as tile_ids as keys, there is an extra key 'metadata' containing a dict with information about the calibration fit - it contains:

    • 'fitid':integer - The ID of the specific calibration fit

    • 'target_obsid':integer - The original observation ID for which this fit was requested.

    • 'filename:str - The hyperdrive filename with the full fit, to be requested from the S3 bucket

    • 'freq_res_MHz' - The frequency resolution of the fit, in MHz

    • 'calid':integer - The starttime of the observation used to generate this fit

    • 'freq_list' - The list of frequencies channels in the observation

    • 'code_version':str - The code used to generate this fit

    • 'fit_time':integer - unix timestamp when this fit was generated

    • 'creator':str - who created this fit

    • 'fit_niter':integer - number of iterations performed

    • 'fit_limit':float - fit quality limit

• On Error:

  • http status 400 if the obs_id is not provided, or there is any other fault in the request.

  • http status 404 if no valid calibration fit was found for an observation with the given obs_id.

  • http status 500 if an internal error occurred.

Add a request for a calibration fit to be calculated for a target observation ID

e.g. https://ws.mwatelescope.org/calib/add_cal_request?obs_id=1120493008 

Where:

• obs_id is an observation ID you wb ish to calibrate. The add_cal_request service will search the schedule database for the best calibration observation available for this observation. 

• asvo_job_id is the (optional) ASVO job for which this calibration fit is being requested. Will be in the data dictionary submitted in JSON format with the callback URL.

• max_time_separation is a maximum offset in seconds between the given obs_id and the observation start time of the calibrator chosen to use for that observation. The default is 86400 (one day).

• callback_url is an optional string with a URL to be called (hours or days later) when the requested fit has been calculated. See below for callback details.

This service will return:

• On Success:

  • A 200 status code, and no data in the response

• On Error:

  • http status 400 if the obs_id does not exist in the schedule database, or if there is any other fault in the request.

  • http status 404 if no valid calibration observation was found within 'max_time_separation' seconds from the given obs_id, meaning no fit can be calculated.

  • http status 409 if there was an error adding the request to the database

  • http status 500 if an internal error occurred.

Note that this service will only look for calibration observations taken with the same analogue attenuation ('gain_control_value') as the observation.

If you wish to request that a fit be calculated for a specific calibrator observation, just pass that calibrator observations starttime in obs_id, and a 'max_time_separation' value of 0.

Get all calibration fits already calculated for a target observation ID

e.g. https://ws.mwatelescope.org/calib/get_cal_fits?obs_id=1120493008 

Where:

• obs_id is an observation ID you wish to calibrate. The get_cal_fits service will search the calibration solution database for all calibration solutions possible for this observation. See MWA ASVO: Calibration Option for some background information on the calibration database and how it is populated. 

• max_time_separation is a maximum offset in seconds between the given obs_id and the observation start time of the calibrator chosen to use for that observation. The default is 86400 (one day).

• allow_after is an optional integer indicating whether calibrator observations after the given obsid are allowed. Values are 0 or 1 (defaults to 1, meaning True). 

This service will return:

• On Success:

  • A JSON string containing a dictionary, with fitid as the key, and tuples of (calid, offset, creator, filename) as values, where calid is the observation ID of a possible calibrator observation, offset is abs(obsid - calid), creator is either ‘mwa’ or ‘msok’ (for old Marcin pipeline fits) or ‘calvin’ for new fits, and filename is the name of the hyperdrive calibration fit file in the calibration S3 bucket.  

• On Error:

  • http status 400 if the obs_id does not exist in the schedule database, or there is any other fault in the request.

  • http status 404 if no valid calibration fits were found for the given obsid.

  • http status 500 if an internal error occurred.

Note that this service will only look for calibration fits for observations taken with the same analogue attenuation ('gain_control_value') as the observation.

Callbacks

When a callback URL is provided to the add_cal_request() service above, after the calibration fit is finished (hours or days later), the given URL is called, and a data dictionary is provided, containing:

• cal_id: The observation ID of the calibration observation that was fit.

• asvo_job_id: The ASVO job number, if provided.

• target_obs_id: The observation ID being reduced, that needed a calibration solution

• finished_datetime: The time that the calibration fit finished (successfully or not)

• success: Boolean, True if the calibration fit succeeded

• message: Text string describing the fit result, with any errors if applicable

JSON calibration data service for Grafana dashboard

Two web services support the Grafana calibration dashboard using the JSON data source. The ‘grafana/metrics’ service is used by Grafana to build the user interface for designing queries, telling it what targets (receivers) and options are available. The ‘grafana/query’ service returns the actual data points to plot.

These services are used by the Grafana dashboard here: https://grafana.mwatelescope.org/d/defeb8ik3jtogd/calibration-fits-improved

Get all metrics that can be used as targets for the JSON query

eg: https://ws.mwatelescope.org/calib/grafana/metrics

Returns a JSON structure containing a list of dictionaries, where each dictionary describes a metric that can be passed as a target to the query() service. It looks like:

This indicates that valid targets for the query service are rec01 to rec16 (returning calibration data for that receiver over the given time period), or ‘sources’ (returning a list of calibration sources over the given time period.

Get payload options for the metrics

This service is used to tell Grafana how to build queries, indicating what values are possible options to any payload option with a ‘type’ of either ‘select’ or ‘multi-select’.

eg https://mro.mwa128t.org/calib/grafana/metric-payload-options

The value passed is a JSON structure carrying the metric and payload set so far in the query builder, and the name of the payload option for which the possible options are to be returned. For example, when passed:

The service will return:

Get a JSON structure with calibration data (or a list of sources)

eg: https://mro.mwatelescope.org/calib/grafana/query

When passed a JSON structure containing the target metric, the time range to plot, and payload options defining what sort of data should be returned, returns a JSON structure with the requested data. For example, Grafana would call it with a JSON structure looking like:

Most of this data passed by Grafana is just ignored by the calibration query web service. If calling from your own code, just pass, for example:

Or, to get a list of source names for all observations in a given time range, do:

What you’ll get back from the query service is JSON structure is a list of dictionaries, one for each tile in the receiver. Each will have two keys - ‘schema’ and ‘data’, where the ‘schema’ points to a dictionary with the metadata for that tile, and the ‘data’ contains two lists of values, one with timestamps and the other with the corresponding values to plot. For example:

Old, deprecated web services for calibration solutions:

Get the best calibration solution available for a target observation ID (deprecated - do not use!)

e.g. https://ws.mwatelescope.org/calib/get_calfile_for_obsid?obs_id=1120493008 

Where:

• obs_id is an observation ID you wish to calibrate. The get_calfile_for_obsid service will search the calibration solution database for the best calibration solution available for this observation. See MWA ASVO: Calibration Option for some background information on the calibration database and how it is populated. 

This service will return:

• On Success:

  • A zip file containing:

    • cal_id.bin: Binary file (in AO-cal format) containing the calibration solution. This can be used in Andre Offringa's calibrate and Cotter tools. Note the filename contains the observation ID of the calibrator observation used.

    • obs_id_flagged_tiles.txt: A plain text file listing the tile_id's of tiles that failed to calibrate.

    • obs_id_flags.py: A simple python script which you can run to apply the calibration solution (in CASA).

    • obs_id.metafits: A metafits file, with the calibration solution applied to the data.

• On Error:
  
  

  • http status 400 if the cal_id is not found in the calibration solution database.

  • http status 500 if an internal error occurred.

Note that this service will only look for calibration observations taken with the same analogue attenuation ('gain_control_value') as the observation. If you wish to search for /any/ calibration solution, regardless of analogue attenuation, add the 'any_gain' parameter to the URL string - for example, http://ws.mwatelescope.org/calib/get_calfile_for_obsid?obs_id=1120493008&any_gain

Get the calibration solution for a specific calibrator observation ID (deprecated - do not use!)

e.g. https://ws.mwatelescope.org/calib/get_calfile_for_calid?cal_id=1120493424 

Where:

• cal_id is the observation ID of an observation that has been marked as a calibrator observation.

This service will return:

• On Success:

  • A zip file containing:

    • cal_id.bin: Binary file (in AO-cal format) containing the calibration solution. This can be used in Andre Offringa's calibrate and Cotter tools.

    • cal_id_flagged_tiles.txt: A plain text file listing the tile_id's of tiles that failed to calibrate.

    • cal_id_flags.py: A simple python script which you can run to apply the calibration solution (in CASA).

• On Error:
  
  

  • http status 400 if the cal_id is not found in the calibration solution database.

  • http status 500 if an internal error occurred.