TAP mwa.obs_download_history Schema and Examples
- Greg Sleap
Owned by Greg Sleap
Examples
select * from mwa.obs_download_history where obs_id = 1222441344
Definition
Provides details of each download of an observation.
| Column Name | Data Type | Description |
|---|---|---|
| datetime_mro | TIMESTAMP | Datetime of download (local MRO time) |
| obs_id | BIGINT | Observation ID- unique identifier of an MWA observation |
| downloaded_bytes | BIGINT | Bytes downloaded |
| source_system_id | INTEGER | System used for this download 1 = MWA ASVO v3 2 = MWA ASVO v1 and v2 3 = NGAS (includes obsdownload and voltdownload) 4 = NGAS via Chen Wu's stats database (includes obsdownload and voltdownload) 5 = NGAS via obsdownload.py proxy logs |
Important Notes About Download History
- The download history is not entirely complete, due to some system logs that could not be recovered.
- The MWA ASVO v1, v2 and v3 are 100% complete.
- The remainder of the logs are missing some data:
- From ~June 2018 to December 2019 are missing many VCS downloads as these logs which were on the FE4 server could not be recovered.
- Prior to 2014, the logs appear incomplete and downloads pertaining to observations prior to commissioning is scant.
- As a result of there being gaps in the data, especially for NGAS, we rely on three related but different sources of logs and merge them together to get a more complete picture.
- For the three NGAS sources, the logs do not contain one entry per downloaded obs_id, but instead contain one row per file downloaded. Sometimes, downloading an entire observation may span hours or days, so we have done our best to estimate a grouping of files into a single date/time to become a download record in our table. This process is most definitely going to cause under or over-reporting of the number of downloads for a particular obs_id (but it will be counted).
- The downloaded_bytes column is, in some cases an estimate:
- For MWA ASVO v1, v2 and v3 it is 100% accurate and represents the final data product (raw files or CASA MS or UVFITS file) that was made available to be downloaded by the end user.
- For all three types of NGAS downloads, downloaded_bytes is estimated based on the data that was archived. In the case of VCS observations, where an observation was recombined and thus the original raw data and recombined data both were stored in the archive for some time, we assume the recombined data was downloaded, not both.