Using the Recipe Test Workflow (RTW) for regular testing

[ehogan]

Introduction

The Recipe Test Workflow (RTW) is a Cylc workflow that is used to regularly run recipes so issues can be discovered during the development process sooner rather than later.

Since the RTW prototype was merged, the RTW has been developed further to enable the RTW to be run on DKRZ.

Now, decisions need to be made on how the RTW should be run and monitored in preparation for the next release (v2.13.0).

How recipes are currently tested for a release (i.e. not using the RTW)

• During the release process, the release manager runs all recipes on DKRZ
• The recipe outputs are shared via a web page, e.g. https://esmvaltool.dkrz.de/shared/esmvaltool/v2.12.0/
• Recipe maintainers check the recipe outputs and report whether the recipe outputs are as expected

How recipes would be tested using the RTW

• Fast recipes (i.e. those that take less than 10 minutes to complete) would run nightly on DKRZ using the RTW (using the latest (main) versions of ESMValTool and ESMValCore)
• If a recipe failed to complete, a new GitHub issue would be opened containing all the relevant information (e.g. recipe name, traceback, any useful logs)
• If the recipe completed but the comparison step failed, a new GitHub issue would be opened with before / after / difference plots
  • If the change is approved, the KGO would be updated
  • If the change is not approved, an appropriate fix would be introduced

How the approach of using GitHub issue would work still needs to be finalised. For example:

• Should a "RTW" user be created to submit these GitHub issues?
• Should one GitHub issue be opened per recipe, or per day?
  • Multiple recipes might fail due to a single change to ESMValCore.
  • Additional comments could be added by the RTW if the recipe is still failing a week later
• Who should be notified about the issues?
  • It would depend on the cause of the failure. They may need to be triaged in the first instance.

Questions

• Is it necessary to continue to maintain the current web page if the approach of using GitHub issues to report problems is introduced?
• Would it be useful for anyone to be able to view the status of the RTW?
• Would it be possible to obtain a shared account on DKRZ? If so, we could use a shared account to run the RTW, and a pre-defined group of people could log in to that account to view the status of the RTW.

Considerations

• Cylc workflows are typically monitored using one of the Cylc UIs. However, the Cylc GUI and Cylc TUI can only be used to monitor Cylc workflows running under your own user account. This means, if my user account is running the RTW, only I would be able to monitor the RTW using the Cylc GUI and / or the Cylc TUI. Cylc Hub supports multi-user access, but would need to installed on the machine that currently hosts the web page. Also, only people with accounts on DKRZ would be able to log in and view the RTW.
• GraphQL can be used to obtain information from the RTW, which could then be presented on a web page. However, this would require authentication.
• Disk space on the machine that currently hosts the web page is limited, so re-creating the current web page daily, with access to all recipe outputs, would not be viable (unless they were overwritten every day).

[ehogan]

A RTW status report web page was created at the May 2025 workshop.

We will update the RTW status report to include useful information and monitor the page manually to triage any issues.

Once we are confident the monitoring is working effectively, we will work on automatically opening GitHub issues.

At the workshop, V suggested using the ESMValBot to automatically open issues 👍

Please let us know if you have any feedback on the RTW status report! 🥳