Find observations

Search the database for observations that satisfy given criteria.

For manual (human) use, omit all parameters and use the base URL to get a form where you can just fill in the relevant criteria: http://ws.mwatelescope.org/metadata/find

If you are writing code to search for observations, append the search criteria parameters to the URL.

eg, http://ws.mwatelescope.org/metadata/find?creator=randall&future=1 

The full list of allowed constraints is:

• mintime, maxtime: Minimum and maximum values of 'starttime' for the observation. Times are INCLUSIVE of the endpoints, e.g. comparisons use 'starttime>=mintime and startttime<=maxtime'.

• mintime_utc, maxtime_utc: As above, but expressed as strings (passed directly to astropy.time.Time() for parsing).

• minduration: Minimum observation duration (stoptime - starttime), in seconds.

• minra, maxra: Minimum and maximum values for ra_pointing for the observation, in degrees. NOTE - if maxra < minra, the values returned wrap around 0/360, e.g. use maxra=10 and minra=350.

• mindec, maxdec: Minimum and maximum values for dec_pointing for the observation, in degrees.

• mingal_long, maxgal_long: Minimum and maximum values for gal_long for the observation, in degrees. NOTE - if maxgal_long< mingal_long, the values returned wrap around 0/360, e.g. use maxgal_long=10 and mingal_long=350.

• mingal_lat, maxgal_lat: Minimum and maximum values for gal_lat for the observation, in degrees.

• minel, maxel: Minimum and maximum values for elevation_pointing for the observation, in degrees.

• minaz, maxaz: Minimum and maximum values for azimuth_pointing for the observation, in degrees. NOTE - if maxaz < minaz, the values returned wrap around 0/360, e.g. use maxaz=10 and minaz=350.

• minlst, maxlst: Minimum and maximum values for Local Sidereal Time for the observation, in degrees. NOTE - if maxlst < minlst, the values returned wrap around 0/360, e.g. use maxlst=10 and minlst=350.

• minsunel, maxsunel: Minimum and maximum values for sun_elevation for the observation, in degrees.

• minsunpd, maxsunpd: Minimum and maximum values for sun_pointing_distance for the observation, in degrees.

• projectid: A string to match against the projectid field for the observation.

• groupid: The observation ID (GPS statart time) of the first observation in the same observation block (scheduled with the same command). 

• mode: A string to match against the observing mode, e.g. 'HW_LFILES', 'VOLTAGE_START', etc.

• creator: A string to match against the name of the person creating that observation.

• obsname: A string to match against the observation name.

• recomined, notrecombined: Whether or not the raw VCS observation files have been pre-processed.

• deleted, notdeleted: Whether or not the observation data files have been deleted.

• contigfreq: Pass 1 to match only observations with 24 contiguous frequency channels, or 0 to match only observations with gaps in the frequency channels.

• calibration: Pass 1 to match only calibration observations, or 0 to match only observations that are NOT calibrations.

• cenchan: Select only observations with this centre frequency channel.

• anychan: Select only observations that include this channel in their list of 24 frequency channels.

• freq_res: Select only observations with the correlator set to this frequency resolution (currently, the only valid numbers are 10, 20, and 40 kHz).

• int_time: Select only observations with the correlator set to this integration time (currently, the only valid numbers are 0.5, 1.0, 2.0 and 4.0 seconds).

• future: Pass 1 to match only observations that have not yet happened, or 0 to match only observations that are in the past.

• dataquality: Pass one or more possible data quality flags (digits from 1 to 6) to match against. The values are 1 (Good), 2 (Some Issues), 3 (Unusable), 4 (Deleted), 5 (Marked for deletion) or 6 (Processed). For example, passing 1 returns only 'Good' observations, 12returns only 'Good' or 'Some Issues', 345 returns only unusable/deleted/marked-for-deletion observations, etc.

• gridpoint: An integer to match against the gridpoint_number column in the schedule_metadata table.

• minfiles: Select only observations with at least this many data files recorded.

• archived: Only return observations where all data files have been archived.

• min_total_tiles, max_total_tiles: Specify minimum and maximum values for the total number of tiles in the array (always 128, before the MWAX correlator was deployed).

• min_good_tiles: The minimum number of good (no errors, not flagged) tiles in the observation.

• max_bad_tiles: The maximum number of bad (errors, or flagged) tiles in the observation.

• delete_id: Only return observations that are part of the given deletion request (the ID is an integer). 

• any_deletion: Only return observations that are part of (any) deletion request - even cancelled and actioned ones.

Other parameters control the output, and output format:

• extended: If specified, an extended set of parameters are returned in the output list (see below).

• dict: If specified, a dictionary is returned, instead of a list, with even more parameters (see below).

• pretty: If passed, and non-zero, the JSON output is padded with spaces and newlines to make it more human-readable.

• sortorder: Either 'asc' or 'desc'. Determines the sort order for both grouping results into pages, and for list and html output, the order within the returned structure.

• pagesize: By default, the web service returns a maximum of 200 observations at a time. Use this parameter to specify a smaller page size, or (with care) a larger one.

• page: By default, the web service returns the first pagesize results. If you pass a number in the page argument, then that 'page' of results is returned instead.

• html: If this parameter is given, the results are returned in human-readable format as a web page, instead of a JSON structure. The SQL query that generated the result will be shown at the top, and if there are more than pagesize observations, buttons to skip to other pages will be shown as well. Clicking on the obsid in the results takes you to the observation summary page (described below).

Any constraints that take strings (projectid, mode, obsname and creator) accept the standard PostgreSQL wildcard characters. A percent character (%) means 'match zero or more of any character', and an underscore (_) means 'match exactly one of any character'. If you wish to match an actual percent or underscore character, escape it with a single leading backslash (e.g. foo\_bar).

Normally, the parameters returned are:

• starttime (obsid in GPS seconds)

• obsname

• creator

• projectid

• ra_pointing (in degrees)

• dec_pointing (in degrees)

if you pass the the extended parameter, then more parameters are returned for each matching observation:

• starttime (obsid in GPS seconds)

• stoptime (in GPS seconds)

• obsname

• creator

• projectid

• ra_pointing (in degrees)

• dec_pointing (in degrees)

• azimuth (in degrees)

• elevation (in degrees)

• ra_phase_center (in degrees)

• dec_phase_center (in degrees)

• local_sidereal_time_deg (in degrees)

• gridpoint_number

If you pass the dict parameter, then a dictionary is returned, containing even more parameters:

• starttime (obsid in GPS seconds)

• stoptime (in GPS seconds)

• obsname

• creator

• projectid

• ra_pointing (in degrees)

• dec_pointing (in degrees)

• azimuth (in degrees)

• elevation (in degrees)

• ra_phase_center (in degrees)

• dec_phase_center (in degrees)

• local_sidereal_time_deg (in degrees)

• gridpoint_name ('EOR1' or 'sweet')

• gridpoint_number

• dataquality

• dataqualitycomment

• mode (eg 'HW_LFILES')

• int_time (correlator integration time)

• freq_res (correlator frequency resolution)

• frequencies (list of 24 coarse channel numbers)

• processed (whether VCS raw data files have been pre-processed/recombined)

• deleted_timestamp (when the observation was deleted, or None if it has not been deleted)

• calibration (True if this is a calibration observation)

• calibrators (the names of the calibration source in this observation)

• groupid (The observation ID - GPS statart time - of the first observation in the same observation block)

• good_tiles (number of tiles without flags or errors in this observation)

• total_tiles (total number of tiles, good and bad, at the time of the observation)

• bad_tiles (number of tiles that had errors, or were flagged)

If you pass 1 to the pretty parameter, than the JSON results are indented to make them more human readable (the default when a GUI web browser is used to make the query). If 0 is passed, then the results are not indented.

Search for or view MWA project IDs

eg, https://ws.mwatelescope.org/metada/projects or https://ws.mwatelescope.org/metadata/projects?projectid=G00% 

This service returns details about all MWA projects (if called with no options) or all project IDs matching the given projectid parameter, where the '%' character is a wild card, matching zero or more characters in the project ID.

If the 'html' parameter is provided, and is 'on', 'true', or 1, or if the web service is called from a web browser, then a human-readable HTML table is returned, including a link in each row to search for all observations matching that project ID. Otherwise, a JSON structure is returned, for use in your code.

Possible parameters are:

• projectid or project_id: The project ID to search for, or a string with one or more '%' characters as wildcards, eg, 'G006%' will match G0060, G0061, ... G0069

• html: Pass  'on', 'true', or 1 for HTML format output, otherwise JSON data will be returned.

Get a 'metafits' file with observation AND telescope configuration data, for a given observation

eg, http://ws.mwatelescope.org/metadata/fits?obs_id=1096952256 or http://ws.mwatelescope.org/metadata/fits?filename=1095675784_20140925102450_gpubox23_01.fits 

This service takes returns a metafits file (described HERE) for a given observation. The observation can be specified by passing either an observation ID (starttime in GPS seconds) with the 'obs_id' parameter, or by passing the full name of a data file recorded by that observation using the 'filename' parameter. If neither obs_id or filename are passed, a metafits file is returned for the most recent observation actually observed by the telescope.

There are optional parameters that affect the contents of the metafits file:

• min_bad_dipoles: A number from 1-16. If either polarisation on a single tile has this many dipoles (or more), flag it as bad. The default (min_bad_dipoles=2) means that if a tile has more than one bad dipole in either or both of X and Y, it will be flagged (deprecated).

• max_bad_dipoles: A number from 1-16. If either polarisation on a single tile has more than this many dipoles, flag it as bad. The default (max_bad_dipoles=1) means that if a tile has more than one bad dipole in either or both of X and Y, it will be flagged.

• nocache: Don't use cached observation data.

• include_calib: Include calibration fit data from the nearest calibration observation that has already been reduced, if one is within plus or minus one day of the given obs_id.

• include_ppds: Include two extra HDUs ('DigGains' and 'PPDs') with digital gains for each cable flavour, and PPD data (full 0-32768 MHz average powers in every coarse channel, for every tile, for the duration of the observation, recorded every 64 seconds for each tile).

• flag_faults: Defaults to 1. If flag_faults=0 then tiles with observing faults (comms errors, too many bad dipoles, etc) are NOT flagged in the metafits file data table.

If you are making a query, and you don't want the results taken from the cache (you want them to be read directly from the database) than add the nocache parameter to your query. Note that this can be hundreds of times slower than using the cached data, so this parameter should be used rarely.

If you are using the offline correlator to turn all or part of a voltage-capture observation into visibilities, you'll need a customised metafits file describing the data to pass to the offline correlator (and to reduce the visibilities after they are created). To produce one of these customised metafits files, call the 'fits' web service with the obs_id or filename parameter for the full (voltage capture) observation, along with one or more of these parameters:

• new_starttime: Must be an integer multiple of 8 GPS seconds, inside the main observation. Becomes the new, fictitious 'obsid' of the customised observation fragment.

• new_stoptime: Must be an integer multiple of 8 GPS seconds, inside the main observation. Becomes the new, 'stoptime' of the customised observation fragment.

• new_freqres: New correlator frequency resolution, in kHz. Must be a valid MWAX frequency resolution value, (0.2, 0.4, ... 10, 20, 40, ... 1280).

• new_inttime: New correlator integration dump time, in seconds. Must be a valid MWAX integration time (0.25, 0.5, 1.0, 2.0, 4.0, 8.0).

• new_mode: New correlator mode string - must be one of 'MWAX_VCS' or 'MWAX_CORRELATOR'.

These fields in the metafits file will be customised for the new observing details: GPSTIME, MODE, EXPOSURE, FINECHAN, INTTIME, NAV_FREQ, NCHANS, NSCANS, DATESTRT, DATE-OBS, MJD, RA, DEC.

These fields in the metafits file will NOT be changed, and retain the original values - calculated for the midpoint of the full, original observation time: SUN-ALT, SUN-DIST, MOONDIST, JUP-DIST, LST

Get observation/schedule data for an observation

eg, http://ws.mwatelescope.org/metadata/obs?obs_id=1096952256 or http://ws.mwatelescope.org/metadata/obs?filename=1095675784_20140925102450_gpubox23_01.fits 

This service returns details about a single observation. The observation can be specified by passing either an observation ID (starttime in GPS seconds) with the obs_id or obsid parameter, or by passing the full name of a data file recorded by that observation using the filename parameter. If neither obs_id/obsid or filename are passed, data is returned for the most recent observation actually observed by the telescope.

If the filetype parameter is passed, only files of that type are returned in the observation info structure (e.g. filetype=8 gives you normal correlated data files, filetype=1 gives you flag files). This parameter is only likely to be useful for voltage capture observations, which can contain thousands of files - you can use the filetype argument to have some (or none) of these file details returned, to speed up the query.

If an observation taken more than one week ago is queried, the details are cached on the server, to speed up subsequent queries about the same object. Observations in the future, or less than a few days old, will have details in the database change as tiles are flagged or observation logs are analysed, so it's not worth saving the data in the query cache.

If you are making a query, and you don't want the results taken from the cache (you want them to be read directly from the database) than add the nocache parameter to your query. Note that this can be hundreds of times slower than using the cached data, so this parameter should be used rarely.

If you pass 1 to the pretty parameter, than the JSON results are indented to make them more human readable (the default when a GUI web browser is used to make the query). If 0 is passed, then the results are not indented.

This service returns a hierarchical structure (as a string in JSON format) that contains every detail about that observation. That string can then be converted from JSON format to a Python dictionary - an example would be:

Note that this hierarchical structure is almost identical to the structure returned by schedule.MWA_Setting() - the main difference is that any values uses as keys to dictionaries (e.g. the 'number' in the .rfstreams dictionary, or the tile ID in the bad dipoles structure) is a string representing a decimal number, instead of an integer value.

Get telescope configuration for a given observation

eg, http://ws.mwatelescope.org/metadata/con?obs_id=1096952256 or http://ws.mwatelescope.org/metadata/con?filename=1095675784_20140925102450_gpubox23_01.fits 

This service takes an observation ID, in gpsseconds, and returns a hierarchical structure (as a string in JSON format) that contains every detail about the configuration of the array at the time of that observation (the value passed can actually be any time in gpsseconds, it doesn't have to be the start time of an actual observation). The time can be passed as obs_id, obsid or reftime parameter, all are equivalent. If you only want a summary of the configuration (to know whether the array is in 'PHASE1', 'COMPACT' or 'LB' configuration) then pass the 'summary' parameter in the URL.

If an observation ID more than one week in the past is queried, the details are cached on the server, to speed up subsequent queries about the same object. Observations in the future, or less than a few days old, will have details in the database change as tiles are flagged or observation logs are analysed, so it's not worth saving the data in the query cache.

If you pass 1 to the pretty parameter, than the JSON results are indented to make them more human readable (the default when a GUI web browser is used to make the query). If 0 is passed, then the results are not indented.

If you are making a query, and you DON'T want the results taken from the cache (you want them to be read directly from the database) than add the nocache parameter to your query. Note that this can be hundreds of times slower than using the cached data, so this parameter should be used rarely.

The structure is a dictionary, with tile ID as a string and an entry for every tile. Each entry is another dictionary, containing data about that tile - the connectivity (receiver number, receiver slot, and correlator 'inputnum'), location ('pos' is a tuple containing east/north in metres, 'alt' is the altitude in metres), and cable data (digital gain multipliers, total tile electrical delay ('ted'), cable and beamformer attenuation). 

Get a list of the last N and next N observations in the schedule, as of the time the service is called

eg, https://ws.mwatelescope.org/metadata/schedstatus?html

This service lets the caller find out what the telescope is doing 'now'. By default, it returns a JSON structure containing a full observation record for each of the last 10 and next 10 observations in the schedule. The current observation will be marked (in colour in the HTML, or with the current=True attribute in JSON results.

Additional options are:

• num_rows - the total number of observations to return (half before the current time, half after the current time, plus whatever observation, if any, is currently being observed).

• html - if this option is present, a human-readable HTML page is returned instead of a JSON structure

• inverse - if this option is present, the colour scheme of the HTML page will be light-on-dark instead of dark-on-light.

• refresh - integer value in seconds. If this option is given, along with the html option, then the HTML output will include an HTML refresh tag with the given refresh time.

• desc - if present, any output (HTML or JSON) will be in reverse order (most recent first).

• terse - if present, JSON output will only contain a few fields per observation (starttime, stoptime, projectid, current, vcsmode, errorstring, obsname).

• pretty - if present, JSON output will include whitespace to make it more human readable (also happens by default if the request is from a web browser).

Get a structure summarising the current telescope health.

eg. http://ws.mwatelescope.org/metadata/tile_health

This service will return a JSON dictionary, these entries:

• obs_id - The observation ID of the recent observation analysed.

• good_tiles - The number of unflagged tiles without pointing or other errors during that observation.

• total_tiles - The total number of tiles configured to be part of the MWA at the time of the observation.

• bad_states - List of tiles where the receiver state is wrong (implying a receiver is offline).

• bad_pointings - List of tiles with pointing errors in this observation.

• bad_freqs - List of tiles with incorrect coarse channel setting in the receiver.

• bad_gains - List of tiles with incorrect analogue attenuation in the receiver.

• bad_beamshape - List of tiles with two or more bad dipoles in the same polarisation.

• bad_flagged - List of manually flagged tiles in the tile_flags table.

• all_tiles - List of all configured tiles.

• bad_tiles - List of all tiles that are bad for any reason (in one or more of the above lists).

For a given observation, get a structure summarising the observation data, and whether it is ready to be processed

eg. http://ws.mwatelescope.org/metadata/data_ready/?obs_id=1336930080

Where:

• obs_id is an observation ID you wish to determine the data status for (or you can instead pass the name of any data file, with the filename option instead of an obsid). If not provided, the most recent observation will be used instead (this includes observations that are still in progress).

This service will return a JSON dictionary, with four entries:

• dataready - True if the observation is ready to process or download from ASVO (all data files recorded are in storage at Pawsey).

• num_data_files - total number of data files recorded for this observation (some or all may not have made it to Pawsey yet, and some or all might have since been deleted).

• num_data_files_archived - the number of data files for this observation actually present in the MWA archive, now.

• good_tiles - The number of unflagged tiles without pointing or other errors during that observation.

• total_tiles - The total number of tiles configured to be part of the MWA at the time of the observation.

• files - a JSON dict containing all data files, with filename as key, and a dict with filetype, size, source host name, site_path, remote_archived and deleted as fields.

Convert times/dates between UTC strings and GPS seconds values

eg, http://ws.mwatelescope.org/metadata/tconv?gpssec=1096952256 or http://ws.mwatelescope.org/metadata/tconv/?utciso=2014-10-10T04:57:20  

This service translates between UTC times (as an ISO format string) and times in 'GPS seconds' - seconds since the GPS epoch, without any leap seconds. The parameters are:

• gpssec: An integer number, the time in GPS seconds, e.g. 1096952256. An ISO format time string is returned.

• utciso: An ISO format date/time string in UTC - e.g. "2014-10-10T04:57:20". The 'T' separating the date and time parts can be replaced by a space character, but if so it must be URL encoded ('%20').

Get beamformer temperatures for a given observation ID

e.g. http://ws.mwatelescope.org/metadata/temps/?starttime=1174096840 

This service takes a time in gpsseconds (typically an observation ID) and returns a structure indexed by the tile ID, giving the tile name, the receiver ID (1-16), the slot ID (1-8) and the beamformer temperature in degrees C for that beamformer at the given time (actually the average of all measurements over the next 5 minutes after the given time).

Note that the beamformer temperatures are only actually read from the beamformer when a pointing command is sent (at the start of each observation), so if you specify a time in between observations, the vlaues will be stale - possibly many hours old. A temperature value of 'None' indicates communications errors with the tile, showing that no reading was returned.

The parameters are:

• starttime: An integer time in GPS seconds, typically the starttime of an observation.

If you pass 1 to the pretty parameter, than the JSON results are indented to make them more human readable (the default when a GUI web browser is used to make the query). If 0 is passed, then the results are not indented.

Delete the server-side cache entries associated with a given observation ID

eg, http://ws.mwatelescope.org/metadata/invalcache?obs_id=1096952256 or http://ws.mwatelescope.org/metadata/invalcache?filename=1095675784_20140925102450_gpubox23_01.fits 

Deletes the data cached on the web server about a given observation, so that the next query will re-load the data from the SQL database.  The observation can be specified by passing either an observation ID (starttime in GPS seconds) with the obs_id or obsid parameter, or by passing the full name of a data file recorded by that observation using the filename parameter.

Get all of the data files associated with a given observation ID

eg, http://ws.mwatelescope.org/metadata/data_files?obs_id=1096952256 or http://ws.mwatelescope.org/metadata/data_files?filename=1095675784_20140925102450_gpubox23_01.fits 

This service returns details about a single observation. The observation can be specified by passing either an observation ID (starttime in GPS seconds) with the obs_id or obsid parameter, or by passing the full name of a data file recorded by that observation using the filename parameter.

The parameters are:

• obs_id: An observation ID (GPS start time in seconds)

• filename: One data file name associated with anobservation, instead of an obs_id

• mintime: Minimum data file timestamp, as an integer GPS second value. Only works for VCS observations.

• maxtime: Maximum data file timestamp, as an integer GPS second value. Only works for VCS observations.

• filetype_id: If supplied, return only files of this integer type code.

• pretty: 1 to have the JSON results are indented to make them more human readable (the default when a GUI web browser is used to make the query). If 0 is passed (the default), then the results are not indented.

• location: Integer - if supplied, only return files in this location. Here 2 is Acacia, 3 is Banksia.

• all_files: If supplied, and True (1, y, Y, true, True, etc) then include files that have been deleted, or not yet archived at Pawsey, otherwise omit them from the file list.

• terse: If supplied, and True (1, y, Y, true, True, etc) then only return the bucket, folder and filename for each file, instead of the full list of attributes.

• max_files: Integer - if supplied, limit the number of file structures returned to the value given.

Note that mintime is inclusive (>=), while maxtime is not inclusive (<), like Python ranges.

Example output is:

Get MWA project information for a given project ID or observation

eg, http://ws.mwatelescope.org/metadata/project?obs_id=1096952256 or http://ws.mwatelescope.org/metadata/project?project_id=G0002 

This service returns details about a single MWA project. You can either pass a project ID code (eg G0008), or you can specify a single observation (by passing either an observation ID (starttime in GPS seconds) with the obs_id or obsid parameter, or by passing the full name of a data file recorded by that observation using the filename parameter).

The parameters are:

• obs_id: An observation ID (GPS start time in seconds)

• filename: One data file name associated with anobservation, instead of an obs_id

• project_id: MWA project ID code, eg G0008.

• pretty: 1 to have the JSON results are indented to make them more human readable (the default when a GUI web browser is used to make the query). If 0 is passed (the default when called by a script), then the results are not indented.

Example output is: