Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 80 Next »

The following are the official, recent release notes for the MWA ASVO.For older release notes, see MWA ASVO Past Release Notes.

Changes/fixes are tagged with:

WEB This item relates to the front end web site of the MWA ASVO and the associated VO TAP service.

CLIENT This item relates to the python API and command line clients of the MWA ASVO (MWA ASVO: Command Line Clients).

BACKEND This item relates to the server back-end of the MWA ASVO system, which includes the dispatch service, processor service and cotter.

BREAKING This represents a breaking change to existing functionality


9 Aug 2024

  • BACKEND Github actions failing
  • WEB BACKEND Allow scratch delivery to deliver as a tar file. User can choose the delivery format as tar when deliver is selected as scratch. 
  • BACKEND birli manifest hash verification was failing due to a bug.
  • WEB BACKEND Allow MWAX_VCS jobs to specify a range of coarse channels. User can select a range of channels they want to download instead of all the channels.
  • CLIENT submitting jobs should be able to continue after the failure of a single job. when multiple jobs are submitted through Giant-squid, a failure in a single job caused giant squid to quit, which is now fixed. Giant-squid now will proceed to the next job if a job fails.

2 Jul 2024

  • BACKEND Fix SQL bug in db.py
  • WEB Jobs page data fetching logic has been updated
  • WEB Fixed blank login page
  • BACKEND Migrating Dev environment onto setonix(on going)

14 May 2024

  • WEB BACKEND Admin setting: This is an admin feature enables the admin to schedule maintenance , post maintenance message , partially outage some of the jobs. for example, blocking only the jobs that delivers to acacia when acacia is down. This will allow us to keep working parts of the system operational when only one Pawsey component is offline or not working.
  • WEB BACKEND Added ORCID field to the registration and my profile, this is an optional field. Existing users can go to their profile and update their ORCID. We capture ORCIDs from users to better understand how MWA data is used by researchers.
  • WEB BACKEND Allow users to unsubscribe from email status updates from MWA ASVO team. The user can go to their profile and uncheck the option to opt out. We will only send status emails when there is a major issue or disruption to users.
  • WEB BACKEND CLIENT Allow resubmission / retry of completed or errored or cancelled jobs.
    • Retry feature: For failed or cancelled voltage jobs, "retry" allows a partially completed voltage job to be resumed from where it left off. (via web front end only).
    • Resubmit: All type of jobs that are failed or cancelled or completed can be resubmitted using the same parameters.
    • Use -ar, --allow-resubmit to resubmit the jobs through the manta-ray-client CLI
    • Use -r, --allow-resubmit to resubmit the jobs through the giant-squid  CLI
  • CLIENT giant-squid CLI now has a new download command line argument --download-dir. This allows the user to specify which directory to download data to. If omitted, it will default to the current working directory ('.') which was the behaviour prior to this release.
  • WEB BACKEND Cosmetic changes: Date formatting; redirect to home after password reset; error message handling; removed some deprecated code; removed a column an internal table data_files that was no longer needed.
  • BACKEND Distinguish between different job types in SLURM queue. 
    • Visibility Download jobs are named {MWA ASVO jobid}vis
    • Voltage Download jobs are named {MWA ASVO jobid}volt
    • Conversion jobs have several steps:
      • Download data step jobs are named {MWA ASVO jobid}convdload
      • Birli step jobs are named {MWA ASVO jobid}convbirli
      • Delivery of data step jobs are named {MWA ASVO jobid}convdelivery
      • Cleanup/Error step jobs are named {MWA ASVO jobid}converror

05 March 2024

  • BACKEND Refining the way we choose different VSS server when attempt fails downloading from Banksia
  • BACKEND staging only required files from Banksia for VCS and Visibility meta jobs - to decrease the waiting time of a job in the queue.
  • BACKEND Add download and upload throughput measurements to processor logs to find the speed of download.
  • WEB Fixing sorting issue in admin interface.
     

18 January 2024

  • BACKEND Package updates: Python packages are updated from 3.8 to 3.11 except processor.
  • WEB Package updates: React is updated to v18 except axios.
  • WEB BACKEND  Removal of /astro as a delivery option and processing. Please be aware that the /astro option will be removed from ASVO after the January 18th release. If any jobs with the delivery option as /astro were submitted just before the release, they might encounter issues after Pawsey removes the /astro filesystem on the 22nd. Also, any job data that is left in astro will be lost.
  • WEB  UI / UX changes: fixed in search button observation page ,  filtering  issue in jobs page. 
  • WEB  BACKEND  log in locally using email address.
  • BACKEND  fixing permission issues with readme.

13 December 2023

  • WEB Fix delete request filter issues
  • WEB Remove column "Flags" from search results grid
  • BACKEND Priority penalty for ASVO jobs according to the Env 

29 November 2023

  • BACKEND Fix to download VCS data directly into the delivery location.

22 November 2023

  • WEB Removal of SSO Login
  • WEB New login page
  • WEB Scratch Delivery option
  • BACKEND Restrict SSO Login from ASVO CLI clients
  • BACKEND Binding Scratch to SLURM
  • CLIENT manta-ray-client and giant-squid will support scratch delivery option

07 November 2023

  • WEB Added new wiki  URLs.
  • WEB Internal admin page,  user activation and security fixes.
  • BACKEND New MWAX voltage download validations.
  • BACKEND Hash Manifest of files created by Birli.
  • BACKEND Restricting Jobs from SSO users.

03 October 2023

  • WEB Frontend / REACT refactor.
  • WEB Registration of new account via single sign on (SSO) has been removed. See: Removal of Single Sign on for MWA Services for more info.
  • BACKEND v1 of Admin interface merged into main website for superusers.
  • BACKEND Conversion job products now ship a manifest file containing the filenames and sha1 checksum before the files are delivered to /astro or Acacia so filesystem errors can be more easily detected by users.
  • BACKEND Backend logs now contain more information about the Banksia endpoint used to download data from tape in order to allow easier debugging when there are issues.

05 September 2023

  • WEB Many minor text changes, fixes and updates to the static pages of the MWA ASVO website.
  • WEB Updated calibration option text and tooltip.
  • WEB Removed twitter widget from front page and replaced with text and a link to our outages page, as there is still no fix from X.
  • WEB Profile page: Added list of projects that user is a PI of to profile page. These are the projects where you are able to submit delete requests.
  • WEB Profile page: Show user ID, superuser status and MWA Member status.
  • WEB Profile page: Show list of projects the user has special permissions for.
  • WEB Observations (search) page: MWAX_BUFFER has been added to the "Correlator Mode" dropdown (and corresponding "ANY VCS" option.
  • WEB Observations (search) page: New search criteria: "Archived Data Files"- a range allowing you to restrict results to only those observations with that many data files (note: Metafits files or flags are NOT included in that number).
  • WEB Observations (search) page: New search criteria: "Archived Data Size"- a range allowing you to restrict results to only those observations with that total size of data (note: Metafits files or flags are NOT included in that number). Choose units (B, KB, MB, GB, TB). NOTE units are SI/metric prefix not binary prefix.
  • WEB Observations (search) page: "Total archived data bytes" now provides a human readable value instead of a huge unformatted number of bytes. E.g 1234567890 bytes will be shown as 1.2 GB. NOTE units are SI/metric prefix not binary prefix.
  • WEB Observations (search) page: issues with the search table's pagination have now been fixed.
  • BACKEND New user registration info will now also include the Institution/affiliation, allowing admins to more easily approve new users.

01 August 2023

  • BACKEND Various OS patches
  • BACKEND SSL certificates updated
  • BACKEND If an internal server error occurs with status code 500. It will try an alternative webserver before giving an error.
  • BACKEND Logging messages are updated with descriptive messages and provide absolute paths for jobs.
  • BACKEND Added MWA_ASVO_README.md to downloaded files. Now, users will have a new file in their downloads that gives information about MWA ASVO.
  • CLIENT giant-squid client now supports voltage jobs (note: usual restrictions apply- please Contact Support if you need to be able to download voltages)
  • WEB ASVO is added with filters in the "My Jobs" page for Job type and status.
  • WEB Total Archieved Bytes(Size of observation) in the Observations page is formatted.
  • WEB BACKEND Pre-submission validations are added in both web-UI and the backend to capture pre-processor errors in advance. 

26 July 2023

  • BACKEND Birli version bumped to v0.9.2 which resolves an issue where CASA measurement sets created by Birli may contain files which have a last access time in 2021. This was causing issues for users who are on filesystems with a purge policy which deletes those files as they are too old. Now Birli writes the files with up to date timestamps.

21 July 2023

  • CLIENT  manta-ray-client now prints the line/job number with submission issues in order to make finding and fixing issues in the csv file easier.

19th July 2023

  • BACKEND Changed copyq  partition to asvo-copyq  partition, Updating the Slurm output logs, and adding birli_manifest to check for better astro filesystem debugging. 

4th July 2023

  • WEB New feature: If you are the PI of an MWA project/proposal (or an MWA ASVO superuser), you will see a new icon on the "Observations" page called "Request Deletion". If, as a PI, you want to nominate one or many observations from your project for deletion, click on the button and provide a name for the request and the reason for nominating for deletion. At a future date, the details of observations in this (and other) delete requests will be emailed to the MWA collaboration for a 4 week consultation process. If no objection is received in writing by the Director, the request will be approved, and then the data will be deleted. You can view existing Delete Requests by clicking on your profile (name) in the top right and selecting "Delete Requests". You can then view or cancel your delete requests. There are also filters available to manage the list if it becomes too large (useful for superusers especially).
  • WEB The 2 new ARDC Collections have been added to the About→Data Collections menu & search options.

24th January 2023

  • BREAKING Following a discussion at the 2022A MWA project meeting, Cotter has been removed from the MWA ASVO entirely. Birli will now be used for all conversion jobs.
  • BACKEND Following user feedback, metafits files are now included with the output products of conversion jobs.
  • BREAKING The names of parameters that are passed to conversion jobs have changed.
    • Users can still use the web frontend to submit jobs
    • Users will not need to update the manta-ray-client
    • Users will need to update the CSV that they use to submit jobs with the manta-ray-client.
    • See here for details MWA ASVO: Command Line Client (manta-ray-client)


  • No labels