Triggering web services
- Andrew Williams
These web services are the back-end of the MWA triggering system, described in detail Hancock et al, 2019 (https://ui.adsabs.harvard.edu/link_gateway/2019PASA...36...46H/EPRINT_PDF)
All of these web services accept parameters that can either be URL-encoded in the request URL, or POSTed in the request body. Most of these web services require the user to pass a 'project_id' parameter containing an MWA project ID code (used to determine whether that project is allowed to override whatever observations are currently in the schedule), and a 'secure_key' parameter containing a password that proves that the caller is authorised to interrupt the MWA on behalf of the given project ID.
All of them return a JSON structure in the body of the reply.
Busy - ask whether the telescope can currently be interrupted.
eg: http://mro.mwa128t.org/trigger/busy?project_id=C001&obstime=180
Parameters:
project_id: MWA project ID string, eg G0055, that would be interrupting the existing observations
obstime: Number of seconds from 'now' to check for in the schedule
pretty: True if the returned JSON should be rendered to be more human-readable
The reply is a JSON string containing a boolean - True if the telescope is 'busy' at this instant, and can't be interrupted by the given project ID between now and 'obstime' seconds in the future.
Obslist - get a list of observations currently in the schedule over the next <obstime> seconds.
eg: http://mro.mwa128t.org/trigger/obslist?obstime=180
Parameters:
obstime: Number of seconds from 'now' to check for in the schedule
pretty: True if the returned JSON should be rendered to be more human-readable
current: If true (a non-zero integer), only return a single observation - either the observation currently in progress, or the last observation in the schedule. This observation will determine the current settings (channels, attenuation, etc) in use if a buffer trigger is called.
The data returned is a JSON string containing a list of observations, where each observation is a tuple of (starttime, stoptime, obsname, creator, project_id, mode, groupid)
TriggerObs - Trigger a set of correlator observations with the MWA telescope, on one or more targets.
Parameters can be passed in the URL, as with the other services, but there are usually a lot of them, so POSTing them is preferred.
Base parameters are:
project_id: MWA project ID string, eg G0055, that would be interrupting the existing observations. Defaults to 'C001'.
secure_key: The password associated with that project ID. No default.
pretty: True if the returned JSON should be rendered to be more human-readable. Defaults to False if called from a script, True if called from a web browser.
pretend: Boolean, integer, or string (eg 'y', 'n', etc) - if true, then do everything except clear out existing observations and schedule new ones. Defaults to True.
calibrator: Boolean, integer, or string (eg 'y', 'n', etc), or the name of a source - if a source name, observe that source as a calibrator observation for each frequency specified, after the target observations. If a boolean, and True, choose the best available calibrator source to observe. Defaults to False.
calexptime: The duration in seconds of each calibrator observation, if specified. Defaults to 120 seconds.
groupid: Used to specify the observation ID marking the start of this 'observation group' - useful if this trigger is adding to or changing a set of previously triggered observations, so the new ones can be grouped with the original set. Defaults to using the obsid of the first observation scheduled with this trigger call.
Many of the parameters can be used to specify one or more observing targets, in several different formats:
ra and dec: Specify a target coordinate in J2000 coordinates. Each can be either a single value (in degrees), or a JSON encoded list of values in degrees to specify more than one target. If so, the number of elements in the 'ra' and 'dec' lists must be equal.
alt and az: Specify a target coordinate in altitude/elevation and azimuth. Each can be either a single value (in degrees), or a JSON encoded list of values in degrees to specify more than one target. If so, the number of elements in the 'alt' and 'az' lists must be equal.
source: A string specifying a source name, or a JSON encoded list of source names (to be resolved by the name resolver in astropy.coordinates).
subarrays: A JSON encoded list of pre-defined MWA subarray names. If provided, the number of subarray names must be equal to the total number of observing targets provided (all ra/dec, alt/az and source names), and one of the given subarrays will be assigned to each of the target positions provided. If the subarrays parameter is NOT provided, then the MWA will observe each of the provided target positions in turn, with the entire array. Defaults to empty (no subarrays). Valid subarray names currently include 'all_north', 'all_south', 'all_east' and 'all_west' (each including approximately half the tiles in the array), as well as 'all_ne', 'all_nw', 'all_se', and 'all_sw' (each including approximately one quarter of the tiles in the array).
The rest of the observations provide the rest of the observation details:
freqspecs: A single MWA coarse channel selection specifier (eg '121,24' for 24 coarse channels centered on channel 121), or a JSON encoded list of several channel selection strings. If more than one is provided, multiple observations of each specified target location will be taken, for each frequency channel set provided (as well as one at each frequency for the post-observation calibrator, if specified). Defaults to '145,24'.
creator: The name of the person or system generating this trigger - an arbitrary string. Defaults to 'djmwa/trigger'.
freqres: Frequency resolution for the correlator, in kHz - must be one of the standard MWAX settings. Defaults to 10.0 kHz.
inttime: Integration time for the correlator, in seconds - must be one of the standard MWAX settings. Defaults to 0.5 seconds.
avoidsun: Boolean, integer, or string (eg 'y', 'n', etc) - if true, try to put the Sun in a beam null for each target location (except alt/az positions). Defaults to False.
obsname: The name the observation to be inserted (another arbitrary string). Defaults to 'trigger'.
exptime: The duration of each individual observation (for each observing target, at each frequency channel selection). Defaults to 120 seconds.
nobs: The number of observations (for each observing target, at each frequency channel selection). The total observing time (if not using subarrays) will be nobs * (number of target locations) * (number of frequency specifiers) * exptime (not including calibrator observations). Defaults to 15 observations.
atten: The analogue ASC attenuation for the observation - defaults to 1dB.
The structure returned is a JSON encoded dictionary, containing the following:
result['success'] - a Boolean, True if the observations were scheduled successfully, False if there was an error.
result['errors'] - a dictionary, containing integer keys from 0-N, where each value is an error message. Normally empty.
result['params'] - a dictionary containing all parameters passed to the web service, after parsing, and some extra parameters calculated by the web service (the name of the automatically chosen calibrator, etc).
result['clear'] - the commands used to clear the schedule. It contains the keys/values:
'command': The full clear_schedule.py command line
'retcode': The integer return code from that command
'stderr': The output to STDERR from that command
'stdout': The output to STDOUT from that command
result['schedule'] - the commands used to schedule the triggered observations. It contains the keys/values:
'command': A string containing all of the schedule_observation.py command lines
'retcode':The integer return code from the shell spawned to run those commands
'stderr': The output to STDERR from those commands
'stdout': The output to STDOUT from those commands
result['trigger_id'] - A unique integer identifying this particular call to the trigger service. You can use this to look up more details using the trigger 'find' web service (see below).
result['obsid_list'] - A list of integer observation IDs for every observation generated by this trigger call.
TriggerVCS - Trigger a set of voltage-capture observations with the MWA telescope, on one or more targets.
Parameters can be passed in the URL, as with the other services, but there are usually a lot of them, so POSTing them is preferred.
Base parameters are:
project_id: MWA project ID string, eg G0055, that would be interrupting the existing observations. Defaults to 'C001'.
secure_key: The password associated with that project ID. No default.
pretty: True if the returned JSON should be rendered to be more human-readable. Defaults to False if called from a script, True if called from a web browser.
pretend: Boolean, integer, or string (eg 'y', 'n', etc) - if true, then do everything except clear out existing observations and schedule new ones. Defaults to True.
calibrator: Boolean, integer, or string (eg 'y', 'n', etc), or the name of a source - if a source name, observe that source as a calibrator observation for each frequency specified, after the target observations. If a boolean, and True, choose the best available calibrator source to observe. Defaults to False.
calexptime: The duration in seconds of each calibrator observation, if specified. Defaults to 120 seconds.
groupid: Used to specify the observation ID marking the start of this 'observation group' - useful if this trigger is adding to or changing a set of previously triggered observations, so the new ones can be grouped with the original set. Defaults to using the obsid of the first observation scheduled with this trigger call.
Many of the parameters can be used to specify one or more observing targets, in several different formats. If no target positions are specified, then observations will be triggered in 'all sky' mode, with a single dipole active on each tile:
ra and dec: Specify a target coordinate in J2000 coordinates. Each can be either a single value (in degrees), or a JSON encoded list of values in degrees to specify more than one target. If so, the number of elements in the 'ra' and 'dec' lists must be equal.
alt and az: Specify a target coordinate in altitude/elevation and azimuth. Each can be either a single value (in degrees), or a JSON encoded list of values in degrees to specify more than one target. If so, the number of elements in the 'alt' and 'az' lists must be equal.
source: A string specifying a source name, or a JSON encoded list of source names (to be resolved by the name resolver in astropy.coordinates).
subarrays: A JSON encoded list of pre-defined MWA subarray names. If provided, the number of subarray names must be equal to the total number of observing targets provided (all ra/dec, alt/az and source names), and one of the given subarrays will be assigned to each of the target positions provided. If the subarrays parameter is NOT provided, then the MWA will observe each of the provided target positions in turn, with the entire array. Defaults to empty (no subarrays). Valid subarray names currently include 'all_north', 'all_south', 'all_east' and 'all_west' (each including approximately half the tiles in the array), as well as 'all_ne', 'all_nw', 'all_se', and 'all_sw' (each including approximately one quarter of the tiles in the array).
The rest of the observations provide the rest of the observation details:
freqspecs: A single MWA coarse channel selection specifier (eg '121,24' for 24 coarse channels centered on channel 121), or a JSON encoded list of several channel selection strings. If more than one is provided, multiple observations of each specified target location will be taken, for each frequency channel set provided (as well as one at each frequency for the post-observation calibrator, if specified). Defaults to '145,24'.
creator: The name of the person or system generating this trigger - an arbitrary string. Defaults to 'djmwa/trigger'.
avoidsun: Boolean, integer, or string (eg 'y', 'n', etc) - if true, try to put the Sun in a beam null for each target location (except alt/az positions). Defaults to False.
obsname: The name the observation to be inserted (another arbitrary string). Defaults to 'trigger'.
exptime: The duration of each individual observation (for each observing target, at each frequency channel selection). Defaults to 120 seconds.
nobs: The number of observations (for each observing target, at each frequency channel selection). The total observing time (if not using subarrays) will be nobs * (number of target locations) * (number of frequency specifiers) * exptime (not including calibrator observations). Defaults to 15 observations.
atten: The analogue ASC attenuation for the observation - defaults to 1dB.
The structure returned is a JSON encoded dictionary, containing the following:
result['success'] - a Boolean, True if the observations were scheduled successfully, False if there was an error.
result['errors'] - a dictionary, containing integer keys from 0-N, where each value is an error message. Normally empty.
result['params'] - a dictionary containing all parameters passed to the web service, after parsing, and some extra parameters calculated by the web service (the name of the automatically chosen calibrator, etc).
result['clear'] - the commands used to clear the schedule. It contains the keys/values:
'command': The full clear_schedule.py command line
'retcode': The integer return code from that command
'stderr': The output to STDERR from that command
'stdout': The output to STDOUT from that command
result['schedule'] - the commands used to schedule the triggered observations. It contains the keys/values:
'command': A string containing all of the schedule_observation.py command lines
'retcode':The integer return code from the shell spawned to run those commands
'stderr': The output to STDERR from those commands
'stdout': The output to STDOUT from those commands
result['trigger_id'] - A unique integer identifying this particular call to the trigger service. You can use this to look up more details using the trigger 'find' web service (see below).
result['obsid_list'] - A list of integer observation IDs for every observation generated by this trigger call.
TriggerBuffer - Trigger a buffer-dump in the correlator to retro-actively save the last few minutes of data from the past, and keep voltage-capture going for a set time into the future.
Parameters can be passed in the URL, as with the other services, or you can POST them if preferred.
Base parameters are:
project_id: MWA project ID string, eg G0055, that would be interrupting the existing observations. Defaults to 'C001'.
secure_key: The password associated with that project ID. No default.
pretty: True if the returned JSON should be rendered to be more human-readable. Defaults to False if called from a script, True if called from a web browser.
pretend: Boolean, integer, or string (eg 'y', 'n', etc) - if true, then do everything except clear out existing observations and schedule new ones. Defaults to True.
The rest of the observations provide the time/s to start and stop capture. All other observation details are inherited from whatever observation/s are currently in progress, or if there is no current observation, whatever the last observation used - this includes pointing direction/s and frequency settings. Note that all future observations (from the time called, through to 'end_time' or the end of the 'obstime' provided) are deleted by the triggerbuffer() call, to prevent the telescope state from being (further) changed during the capture.
start_time: The earliest time in GPS seconds for which data should be saved - if zero, or before the earliest buffer file available save all data. This value is inclusive, so .free files later that, or starting at this time, are included.
end_time: The latest time in GPS seconds for which data should be saved. If start_time and end_time are both zero, don't save any data. This time may be in the past. If not specified, keep capturing until 'obstime' seconds into the future, from the time of calling.
obstime: How many seconds to keep capturing data for, after the buffer dump (from 'now' into the future, not including past data dumped from the buffers). If end_time is specified, the value of obstime is ignored.
For triggers requiring continued capturing after the trigger, you would normally pass zero to start_time, and a duration to obstime as a capture duration, counted from the time that the trigger takes place. For captures requiring only one or two subfiles at a specific time in the past, you would pass specific GPS times to start_time and stop_time, and not use obstime.
Note that if start_time and end_time are both specified, and both zero, no subfiles are captured, but otherwise the full end-to-end process takes place as a null operation, including calls to all 24 MWAX servers.
The structure returned is a JSON encoded dictionary, containing the following:
result['success'] - a Boolean, True if the observations were scheduled successfully, False if there was an error.
result['errors'] - a dictionary, containing integer keys from 0-N, where each value is an error message. Normally empty.
result['params'] - a dictionary containing all parameters passed to the web service, after parsing, and some extra parameters calculated by the web service (the name of the automatically chosen calibrator, etc).
result['clear'] - the commands used to clear the schedule. It contains the keys/values:
'command': The full clear_schedule.py command line
'retcode': The integer return code from that command
'stderr': The output to STDERR from that command
'stdout': The output to STDOUT from that command
result['schedule'] - the commands used to schedule the triggered observations. It contains the keys/values:
'command': A string containing all of the schedule_observation.py command lines
'retcode':The integer return code from the shell spawned to run those commands
'stderr': The output to STDERR from those commands
'stdout': The output to STDOUT from those commands
result['trigger_id'] - A unique integer identifying this particular call to the trigger service. You can use this to look up more details using the trigger 'find' web service (see below).
result['obsid_list'] - The observation IDs of all MWA observations covered by the buffer dump and subsequent voltage capture. Use these observation IDs to download the voltage capture files, and to determine the telescope setting/s during the time span including the captured data.
Find - Search for and view past calls to trigger observations using these web services
eg: https://ws.mwatelescope.org/trigger/find
When called with no parameters, this service returns a web form for entering parameters to search for previous triggers, like the 'find observatons' metadata web service.
If called with parameters, this service returns (depending on the html parameter) either an HTML page or a JSON structure, with a list of trigger calls.
Optional parameters are:
pagesize: By default, the web service returns a maximum of 200 triggers at a time. Use this parameter to specify a smaller page size, or (with care) a larger one.
page: By default, the web service returns the first pagesize results. If you pass a number in the
pageargument, then that 'page' of results is returned instead.html: If this parameter is given, the results are returned in human-readable format as a web page, instead of a JSON structure. If there are more than
pagesizeobservations, buttons to skip to other pages will be shown as well. Clicking on the obsid in the results takes you to the observation summary page (described below).trigger_id: Returns a single result, with this trigger ID.
trigger_mode: one of 'MWAX_CORRELATOR', 'MWAX_VCS' or 'BUFFER'
pretend: One of 'on', 'true', or a non-zero integer to find only 'pretend mode' triggers, or anything else to find only 'not pretend mode' triggers.
success: One of 'on', 'true', or a non-zero integer to find only successful triggers, or anything else to find only not-successful triggers (that failed generate observations or data).
mintime_utc, maxtime_utc: Time constraints for when the trigger happened, in ISOT string format.
project_id: Return only triggers called with this project ID code.
obsname: Matches the observation name passed with the trigger call. Use the '%' character as a wildcard matching 0 or more occurrences of any character.
creator: Matches the creator name passed with the trigger call. Use the '%' character as a wildcard matching 0 or more occurrences of any character.
desc: If supplied, with any value, reverse the sort order.
pretty: If supplied and evaluates to True, pretty-format the JSON stucture for human readability.
Show - View detailed information about a single past call to trigger observations
eg: https://ws.mwatelescope.org/trigger/show?trigger_id=17
Given a single trigger ID, this service returns detailed information about that trigger. Currently this service only returns JSON structure, but it will be upgraded to return a web page instead.
Parameters are:
html: No effect.
trigger_id: The ID code of the trigger to return
pretty: If supplied and evaluates to True, pretty-format the JSON stucture for human readability.
Returns a JSON structure containing trigger_id, project_id, pretend, success, creator, obsname, trigger_mode, obsids, params, errors, and created_datetime, where:
obsids is a list of observation IDs that were either scheduled by this trigger (in MWAX_CORRELATOR and MWAX_VCS mode), or contain data created by a buffer dump from this trigger (in BUFFER mode).
params is a list of all parameters passed in the call that scheduled this trigger.
errors is a list of all errors reported when processing this trigger
created_datetimeis the UTC timestamp in ISOT format.