Calibration web services
- Andrew Williams
- Greg Sleap
Human-readable calibration analysis services
There are a couple of web services to support human-readable tables and plots summarising recent calibration fits. These allow:
The quality of each fit to be judged, based on the differences between residual delays for a given tile, and either the residuals for all other tiles, or residuals for the same tile from other observations over the previous week, or between X and Y cable length residuals for all tiles, etc.
Statistics including the residual median delay for all 8 tiles on each receiver (to adjust the receiver clock fibre lengths in the database) and for each tile (to adjust the tile cable lengths in the database).
To view these plots, open this URL in a browser:
https://ws.mwatelescope.org/calib/summary
A form at the top of the page allows you to specify a time range, to view only the fits for observations in that time range. Below that form is a table showing the fits. Note that it’s possible to have more than one calibration fit calculated for the same observation ID/s. The table columns are:
Fit ID - the time (in Unix ‘seconds since epoch’) that the calibration fit process was performed. Click on the fit ID to view tables of median cable length residuals, both per-receiver median’s over all slots, and per-tile medians of all observations in the fit. You can use these values to update the clock fibre and tile cable lengths in the configuration database.
Creator - the name of the pipeline that calculated the fit, currently either ‘mwa’ for Marcin Sokolowski’s pipeline, or ‘calvin’ for Greg Sleap’s pipeline running on site.
Code version of the fit pipeline, eg 0.17.69
Time the fit pipeline executed, in AWST.
Observations - the individual observations included in that fit, expressed as ‘<obsid> =<obsname>’, where <obsid> is the observation start time in GPS seconds, and <obsname> is the name of the observation. Note that only ‘mwa’ (Sokolowski pipeline) fits will contain more than one observation in the fit. You can click on an obsid to download the calibration fit for that observation or click on the observation name to view the full information page for that observation.
Niter/Limit - fit parameters
Plot links:
tilediff: X and Y cable residuals for each tile (large circles), superimposed over small points showing the residuals from all other tiles in the array, in that same observation.
timediff: X and Y cable residuals for each tile (large circles), superimposed over small points showing the residuals from other fits for that same tile over the previous 7 days. The ‘mwa’ fits are in blue (X) and green (Y), while the ‘calvin’ fits are in red (X) and orange (Y). Data from previous fits is shown as partially transparent points, getting more transparent further back in time.
xydiff: The difference between the X and Y residuals for each tile, over the previous 30 days before that observation.
gains: X and Y gains for ech tile, over every coarse channel included in that fit.
gaintimes: X and Y gains for each tile, for each MWA channel in the fit, compared to previous fit data for that tile/channel. The ‘mwa’ fits are in blue (X) and green (Y), while the ‘calvin’ fits are in red (X) and orange (Y). Data from previous fits is shown as partially transparent points, getting more transparent further back in time.
Get the best calibration solution available for a target observation ID
e.g. http://ws.mwatelescope.org/calib/get_cal_solution?obs_id=1120493008
Where:
obs_id is an observation ID you wish to calibrate. The get_cal_solution service will search the calibration solution database for the best calibration solution available for this observation. See MWA ASVO: Calibration Option for some background information on the calibration database and how it is populated.
prefer_fitid is an optional fitid integer (unit timestamp when the fit was run) to use, instead of the default (the most recent fit on the calibrator observation closest in time to the given obs_id).
max_time_separation is a maximum offset in seconds between the given obs_id and the observation start time of the calibrator chosen to use for that observation. The default is 86400 (one day).
allow_after is an optional integer indicating whether calibrator observations after the given obsid are allowed. Values are 0 or 1 (defaults to 1, meaning True).
nocache is an optional integer indicating whether cached calibration fits can be returned, or if the fit should be generated directly from the calibration_solutions table. Values are 0 or 1 (defaults to 0, meaning False).
output_format is an optional string, one of 'aocal', 'auto' or 'hyperdrive', where 'hyperdrive' format is not yet implemented.
This service will return:
On Success:
A zip file containing:
cal_id.bin: Binary file (in AO-cal format) containing the calibration solution. This can be used in Andre Offringa's calibrate and Cotter tools. Note the filename contains the observation ID of the calibrator observation used.
obs_id_flagged_tiles.txt: A plain text file listing the tile_id's of tiles that failed to calibrate.
obs_id_flags.py: A simple python script which you can run to apply the calibration solution (in CASA).
On Error:
http status 400 if the obs_id does not exist in the schedule database, or there is any other fault in the request.
http status 404 if no valid calibration fit was found for an observation within 'max_time_separation' seconds from the given obs_id.
http status 500 if an internal error occurred.
Note that this service will only look for calibration observations taken with the same analogue attenuation ('gain_control_value') as the observation.
This service does NOT return a metafits file for the observation in the zip file - if needed, one should be requested separately using the /metadata/fits web service.
ALSO - this service does NOT request a calibration fit be calculated for the given obs_id, if one wasn't found. Use the 'add_cal_request' service (below) instead.
If you wish to request a fit for a specific calibrator observation, just pass that calibrator observations starttime in obs_id, and a 'max_time_separation' value of 0.
Add a request for a calibration fit to be calculated for a target observation ID
e.g. http://ws.mwatelescope.org/calib/add_cal_request?obs_id=1120493008
Where:
obs_id is an observation ID you wb ish to calibrate. The add_cal_request service will search the schedule database for the best calibration observation available for this observation.
asvo_job_id is the (optional) ASVO job for which this calibration fit is being requested. Will be in the data dictionary submitted in JSON format with the callback URL.
max_time_separation is a maximum offset in seconds between the given obs_id and the observation start time of the calibrator chosen to use for that observation. The default is 86400 (one day).
callback_url is an optional string with a URL to be called (hours or days later) when the requested fit has been calculated. See below for callback details.
This service will return:
On Success:
A 200 status code, and no data in the response
On Error:
http status 400 if the obs_id does not exist in the schedule database, or if there is any other fault in the request.
http status 404 if no valid calibration observation was found within 'max_time_separation' seconds from the given obs_id, meaning no fit can be calculated.
http status 409 if there was an error adding the request to the database
http status 500 if an internal error occurred.
Note that this service will only look for calibration observations taken with the same analogue attenuation ('gain_control_value') as the observation.
If you wish to request that a fit be calculated for a specific calibrator observation, just pass that calibrator observations starttime in obs_id, and a 'max_time_separation' value of 0.
Get all calibration fits already calculated for a target observation ID
e.g. http://ws.mwatelescope.org/calib/get_cal_fits?obs_id=1120493008
Where:
obs_id is an observation ID you wish to calibrate. The get_cal_fits service will search the calibration solution database for all calibration solutions possible for this observation. See MWA ASVO: Calibration Option for some background information on the calibration database and how it is populated.
max_time_separation is a maximum offset in seconds between the given obs_id and the observation start time of the calibrator chosen to use for that observation. The default is 86400 (one day).
allow_after is an optional integer indicating whether calibrator observations after the given obsid are allowed. Values are 0 or 1 (defaults to 1, meaning True).
This service will return:
On Success:
A JSON string containing a dictionary, with fitid as the key, and tuples of (calid, offset, filename) as values, where calid is the observation ID of a possible calibrator observation, offset is abs(obsid - calid), and filename is the name of the hyperdrive calibration fit file in the calibration S3 bucket.
On Error:
http status 400 if the obs_id does not exist in the schedule database, or there is any other fault in the request.
http status 404 if no valid calibration fits were found for the given obsid.
http status 500 if an internal error occurred.
Note that this service will only look for calibration fits for observations taken with the same analogue attenuation ('gain_control_value') as the observation.
Callbacks
When a callback URL is provided to the add_cal_request() service above, after the calibration fit is finished (hours or days later), the given URL is called, and a data dictionary is provided, containing:
cal_id: The observation ID of the calibration observation that was fit.
asvo_job_id: The ASVO job number, if provided.
target_obs_id: The observation ID being reduced, that needed a calibration solution
finished_datetime: The time that the calibration fit finished (successfully or not)
success: Boolean, True if the calibration fit succeeded
message: Text string describing the fit result, with any errors if applicable
Old, deprecated web services for calibration solutions:
Get the best calibration solution available for a target observation ID (deprecated - do not use!)
e.g. http://ws.mwatelescope.org/calib/get_calfile_for_obsid?obs_id=1120493008
Where:
obs_id is an observation ID you wish to calibrate. The get_calfile_for_obsid service will search the calibration solution database for the best calibration solution available for this observation. See MWA ASVO: Calibration Option for some background information on the calibration database and how it is populated.
This service will return:
On Success:
A zip file containing:
cal_id.bin: Binary file (in AO-cal format) containing the calibration solution. This can be used in Andre Offringa's calibrate and Cotter tools. Note the filename contains the observation ID of the calibrator observation used.
obs_id_flagged_tiles.txt: A plain text file listing the tile_id's of tiles that failed to calibrate.
obs_id_flags.py: A simple python script which you can run to apply the calibration solution (in CASA).
obs_id.metafits: A metafits file, with the calibration solution applied to the data.
On Error:
http status 400 if the cal_id is not found in the calibration solution database.
http status 500 if an internal error occurred.
Note that this service will only look for calibration observations taken with the same analogue attenuation ('gain_control_value') as the observation. If you wish to search for /any/ calibration solution, regardless of analogue attenuation, add the 'any_gain' parameter to the URL string - for example, http://ws.mwatelescope.org/calib/get_calfile_for_obsid?obs_id=1120493008&any_gain
Get the calibration solution for a specific calibrator observation ID (deprecated - do not use!)
e.g. http://ws.mwatelescope.org/calib/get_calfile_for_calid?cal_id=1120493424
Where:
cal_id is the observation ID of an observation that has been marked as a calibrator observation.
This service will return:
On Success:
A zip file containing:
cal_id.bin: Binary file (in AO-cal format) containing the calibration solution. This can be used in Andre Offringa's calibrate and Cotter tools.
cal_id_flagged_tiles.txt: A plain text file listing the tile_id's of tiles that failed to calibrate.
cal_id_flags.py: A simple python script which you can run to apply the calibration solution (in CASA).
On Error:
http status 400 if the cal_id is not found in the calibration solution database.
http status 500 if an internal error occurred.