Observation Quality Web Services

Owned by Andrew Williams
Last updated: Aug 07, 2023
6 min read

The EoR group maintain a set of observation quality measurements for each observation reduced by their pipeline, and there is a separate set of web services to write quality data to, and read quality data from, the database tables storing this data. All of these services need parameters passed in JSON format as HTTP 'POST' parameters, not embedded in the URL. The best way to call them is using quality.py, attached to this page.

These web services use a common set of parameters, described here:

  • user_name: A string specifying a user name.
  • secure_key: A string specifying the password associated with that user name.
  • constraints: A string containing the JSON representation of a constraint structure, described below.
  • pretty: If passed, and True, any returned JSON structure will be indented to make it more easily readable by humans.

Constraints are passed as a nested hierarchy of tuples or lists, each defining an operator and its argument/s.

Each tuple or list consists of:

  • An operator string, currently one of '=', '<', '<=', '>', '>=', '!=', 'and', 'or', 'not' or 'like'
  • An argument, which could be a string value, a numeric value, a column name, OR another constraint tuple.
  • (Sometimes) another argument, which could be a string value, a numeric value, a column name, OR another constraint tuple.

In the case of a unary operator, the list has two elements - the operator and one argument, and for all other operators, the list has three elements. Currently, the only unary operator is 'not'.

For example, this constraint:

('or',
    ('and',
        ('>', 'obsid', 500),
        ('<', 'obsid', 1000)),
    ('not',
        ('like', 'projectid', '%000%')))


Would translate to SQL looking like:

    WHERE (((obsid > 500) AND (obsid < 1000)) OR (NOT (projectid LIKE '%000%')))


The columns in the quality database table are:

  • obsid: Integer observation ID (start time in GPS seconds).
  • projectid: Project ID code, eg 'G0009'.
  • lowest_channel: Integer channel number.
  • gridpoint_number: Sweet-spot gridpoint number for primary beam pointing.
  • duration_seconds: Integer observation duration.
  • calibration_qa: String describing calibration. Not currently used?
  • rts_cal_qa: Boolean - good/bad. Not currently used?
  • noise_qa: Float - not currently used?
  • iono_magnitude: Float - the median ionospheric offset, scaled as if the offset was observed at 200 MHz.  See Jordan et al. 2017.
  • iono_pca: Float - the dominant eigenvalue in a PCA of ionospheric offsets.  See Jordan et al. 2017.
  • iono_qa: Float - the single-number metric describing how "bad" the ionosphere is.  Rule of thumb: bigger than 5 is active ionosphere.
  • window_power: Float - not currently used
  • iono_abs_tec: Float - not currently used
  • uvfits_path: String - if populated, the path to where RTS calibrated and peeled uvfits files are located on Pawsey
  • rts_cal_source: Integer - not currently used
  • rts_peel_source: Integer - not currently used
  • sourcelist: String - the name of the sky-model source list file used for calibration
  • eor_field: Integer - the EoR field used (i.e. EoR-0, EoR-1 or EoR-2)
  • bad_states: Integer - the number of tiles where the receiver did not respond to commands
  • bad_pointings: Integer - the number of tiles with bad beamformer communications
  • bad_freqs: Integer - the number of tiles where the receiver did not set the coarse channel set correctly
  • bad_gains: Integer - the number of tiles where the receiver did not set the analog attenuation correctly
  • bad_beamshape: Integer - the number of tiles with two or more bad dipoles in the same polarisation
  • bad_flagged: Integer - the number of tiles flagged (manually) in the tile_flags table for known faults
  • good_tiles: Integer - the number of tiles that are NOT in any of the above bad_* counts
  • total_tiles: Integer - the total number of tiles in the observation (good and bad combined)
  • error_info: A JSON structure describing the tile errors (can not be used as a constraint field)
  • total_prep_occupancy: Total flag occupancy of pre-processed file (0.0 to 1.0)
  • non_preflagged_occupancy: Flag occupancy of non-pre-flagged visibilities (ignores timesteps, channels and antennas that are flagged by default)  (0.0 to 1.0)
  • total_ssins_occupancy: Total occupancy of pre-processed visibilities according to SSINS (0.0 to 1.0)
  • ssins_streak_occupancy: Occupancy of broadband streaks according to ssins (0.0 to 1.0)


All of the web services return a JSON'd dictionary, with the following contents:

  • 'errors': A dictionary with incrementing keys, and error strings as values.
  • 'success': A Boolean, True if the web service succeeded.
  • 'query': A string containing the actual SQL query executed.

In addition, the 'Select' web service results dictionary also contains:

  • 'rows': A list of tuples containing the values in each row.
  • 'columns': A list of column names, matching the order of the values in the 'rows' data.

The 'Update' and 'Delete' web services results dictionary also contains:

  • 'rowcount': Integer, the number of rows updated or deleted.


The web services themselves:

The URLs for these services all have a similar format, starting with the base URL, e.g.

Base Webservice URL

http://ws.mwatelescope.org/

Insert - inserts a row into the quality table.

BASEURL/quality/insert - eg https://ws.mwatelescope.org/quality/insert

Parameters are:

  • pretty: As above
  • user_name: As above
  • secure_key: As above
  • row: A JSON string representing a dictionary with a set of key/value pairs, where the key is the column name and the value is that column's value.

This service will insert the given row into the quality table. Any key not in the list of column names above will be ignored (the row will be inserted, with a warning returned to the caller). The 'obsid' and 'projectid' keys are compulsory, but other keys can be omitted, and the inserted row will contain NULLs for those columns.

Select - returns one or more rows from the quality table, matching the constraints provided.

BASEURL/quality/select - eg https://ws.mwatelescope.org/quality/select

Parameters are:

  • pretty: As above
  • user_name: As above
  • desc: If provided, and True, then the results will be returned in decreasing obsid order, otherwise in increasing obsid order.
  • limit: Maximum number of rows returned, defaults to 100. 
  • column_list: A list of column names to return, default is all columns (as listed above).
  • constraints: As described above.

The service will return matching rows from the table in the 'rows' and 'columns' entries in the results dictionary (in JSON format), as described above.

Note that this service does not require a secure_key parameter, as it's read only. Any value for the user_name field will be accepted.

Update - modified one or more rows in the quality table, subject to the constraints provided.

BASEURL/quality/update - eg https://ws.mwatelescope.org/quality/update

Parameters are:

  • pretty: As above
  • user_name: As above
  • secure_key: As above
  • constraints: As described above.
  • data: A JSON string representing a dictionary with a set of key/value pairs, where the key is the column name and the value is that column's value.

Unlike the 'row' parameter in the 'Insert' service, the 'data' dictionary usually only has a few key/value pairs, representing the one or two column values in each row that should be changed. The rest of the data in each matching row will be left unchanged.

This service will return the number of rows changed in the 'rowcount' entry in the results dictionary (in JSON format) as described above.

Delete - delete one or more rows from the quality table, subject to the constraints provided.

BASEURL/quality/delete - eg https://ws.mwatelescope.org/quality/delete

Parameters are:

  • pretty: As above
  • user_name: As above
  • secure_key: As above
  • constraints: As described above.

This service will return the number of rows deleted from the 'rowcount' entry in the results dictionary (in JSON format) as described above.


Example Python code to use this web service directly

See the example code at the bottom of the main MWA web services page: Web Services


Using the interface library, quality.py: (click on the file name to download)

The interface library looks for a configuration file ('quality.conf') in /usr/local/etc, and in the current directory. If it finds one, it will read the user_name and secure_key parameters to use from that file. If that file is not found, you must pass them as additional parameters to the functions described below. Contact Andrew Williams (Andrew.Williams@curtin.edu.au) for username/password information for the quality database. Note that the 'Select' query is read-only, and does not require a secure_key password.

The library defines four functions (insert(), select(), update() and delete(), matching the web services described above. The parameters for each function match the parameter names and descriptions above, except that the 'pretty' parameter is missing from all of the functions, and hte values of the parameters are actual Python structures (lists, dictionaries, etc), not JSON encoded versions of those structures.

Examples for using quality.py to carry out queries:

from quality import insert, select, update, delete

result = insert(row={'obsid': 999,
                     'eor_field': 9,
                     'projectid':'C001'})

result = insert(row={'obsid': 998,
                     'eor_field': 9,
                     'projectid':'C001'})


result = update(constraints=('=', 'obsid', 998),
                data={'eor_field':8})


result = select(constraints=('or',
                                 ('and',
                                     ('>', 'obsid', 500),
                                     ('<', 'obsid', 1000)),
                                 ('not',
                                     ('like', 'projectid', '%000%'))),
                column_list=['obsid', 'projectid', 'lowest_channel',
                             'eor_field'],
                limit=10,
                desc=True)


result = delete(constraints=('or', ('=', 'obsid', 998), ('=', 'obsid', 999)))