Skip to content

[build-tools] Add eas/report_maestro_test_results build function#3388

Open
hSATAC wants to merge 1 commit intomainfrom
ash/eng-19032-report-workflowtestcaseresult-in-reuse_device-true-route
Open

[build-tools] Add eas/report_maestro_test_results build function#3388
hSATAC wants to merge 1 commit intomainfrom
ash/eng-19032-report-workflowtestcaseresult-in-reuse_device-true-route

Conversation

@hSATAC
Copy link
Contributor

@hSATAC hSATAC commented Feb 9, 2026

Why

Maestro test jobs produce per-flow results on the build worker (pass/fail, duration, error messages, retries), but this data is lost after the job finishes.

We want to persist these as WorkflowDeviceTestCaseResult records to surface test outcomes in the dashboard.

How

Add a new eas/report_maestro_test_results build function that:

  1. Parses JUnit XML reports from the junit_report_directory (primary data source)
  2. Enriches with flow file paths from Maestro's ai-*.json debug output in tests_directory
  3. Calls the createWorkflowDeviceTestCaseResults GraphQL mutation via gql.tada + graphqlClient

Key details:

  • JUnit XML as primary data source — ai-*.json only used for flow_file_path mapping and retry count
  • junit_report_directory added as output on eas/__maestro_test (set when output_format=junit) and as optional input on eas/report_maestro_test_results
  • Retry detection by counting occurrences of the same flow across timestamp directories (works for both reuse_devices: true and false patterns)
  • Duplicate testcase name detection — skips report when Maestro config-level name collisions make flow-to-file mapping ambiguous
  • Path relativization uses fs.realpath to handle symlinks (/tmp/private/tmp on macOS)
  • Best-effort reporting — errors are logged but never fail the build step
  • graphqlClient exposed on CustomBuildContext for step functions to use
  • 39 unit tests covering JUnit parsing, flow metadata, retry detection, properties/tags, duplicate detection, and error resilience

Test Plan

@linear
Copy link

linear bot commented Feb 9, 2026

@github-actions
Copy link

github-actions bot commented Feb 9, 2026

Size Change: +3.48 kB (0%)

Total Size: 72.7 MB

Filename Size Change
./packages/eas-cli/dist/eas-linux-x64.tar.gz 72.7 MB +3.48 kB (0%)

compressed-size-action

@hSATAC hSATAC force-pushed the ash/eng-19032-report-workflowtestcaseresult-in-reuse_device-true-route branch from 2e32515 to 3201ab1 Compare February 9, 2026 16:52
@hSATAC hSATAC added the no changelog PR that doesn't require a changelog entry label Feb 9, 2026
@hSATAC hSATAC force-pushed the ash/eng-19032-report-workflowtestcaseresult-in-reuse_device-true-route branch 2 times, most recently from 51c142a to cc45900 Compare February 10, 2026 06:46
@codecov
Copy link

codecov bot commented Feb 10, 2026

Codecov Report

❌ Patch coverage is 92.98246% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 52.76%. Comparing base (1604b7b) to head (1476a49).

Files with missing lines Patch % Lines
...d-tools/src/steps/functions/maestroResultParser.ts 91.41% 11 Missing ⚠️
...ls/src/steps/functions/reportMaestroTestResults.ts 97.44% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3388      +/-   ##
==========================================
+ Coverage   52.26%   52.76%   +0.50%     
==========================================
  Files         804      806       +2     
  Lines       33450    33621     +171     
  Branches     6975     7014      +39     
==========================================
+ Hits        17480    17737     +257     
+ Misses      15885    15799      -86     
  Partials       85       85              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@hSATAC hSATAC force-pushed the ash/eng-19032-report-workflowtestcaseresult-in-reuse_device-true-route branch from cc45900 to f171e9e Compare February 10, 2026 07:12
@hSATAC hSATAC marked this pull request as ready for review February 10, 2026 07:19
@hSATAC hSATAC requested a review from sjchmiela February 10, 2026 07:19
@github-actions
Copy link

Subscribed to pull request

File Patterns Mentions
**/* @douglowder

Generated by CodeMention

@hSATAC hSATAC force-pushed the ash/eng-19032-report-workflowtestcaseresult-in-reuse_device-true-route branch 3 times, most recently from 275434c to 7fb1035 Compare February 11, 2026 14:35
@hSATAC
Copy link
Contributor Author

hSATAC commented Feb 11, 2026

The Size / compare check is failing now due the the yarn gql prebuild script. The graphql was not deployed yet so it can't fetch the required new gql schema.

@hSATAC hSATAC requested a review from sjchmiela February 11, 2026 14:42
Comment on lines +63 to +64
// the same name. JUnit XML only contains names (not file paths), making it impossible
// to map duplicates back to their original flow files. Skip and let the user fix it.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nnnnnooooooo, really??

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, the JUnit is super simple like this:

<?xml version='1.0' encoding='UTF-8'?>
<testsuites>
  <testsuite name="Test Suite" device="iPhone 16 - iOS 18.6 - 59572B21-FEAF-4BEC-B8EB-5793C9A31034" tests="6" failures="3" time="78.0">
    <testcase id="settings_screen" name="settings_screen" classname="settings_screen" time="22.0" status="ERROR">
      <properties>
        <property name="priority" value="high"/>
        <property name="suite" value="smoke"/>
        <property name="jira" value="APP-101"/>
      </properties>
      <failure>Element not found: Text matching regex: Settings</failure>
    </testcase>
  </testsuite>
</testsuites>

}

// Maestro allows overriding flow names via config, so different flow files can share
// the same name. JUnit XML only contains names (not file paths), making it impossible
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After having parsed both commands and JUnit, what is your professional opinion on which one should we use?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After looking at both, I’m leaning toward using the original JSON as the source of truth.

It simply has more complete data, and we can support all formats without forcing users to adopt JUnit.

With JUnit, we’d still need JSON anyway (no tags / flowFileName), and duration is only second-level precision.

The only real advantage of JUnit is stability — it’s less likely to change upstream. JSON doesn’t give us that guarantee, so if we go with it, we might want some automation to detect schema changes early.

@hSATAC hSATAC force-pushed the ash/eng-19032-report-workflowtestcaseresult-in-reuse_device-true-route branch 2 times, most recently from b6efd20 to 198a91d Compare February 13, 2026 14:09
@hSATAC hSATAC force-pushed the ash/eng-19032-report-workflowtestcaseresult-in-reuse_device-true-route branch from 198a91d to 1476a49 Compare February 13, 2026 14:14
@github-actions
Copy link

⏩ The changelog entry check has been skipped since the "no changelog" label is present.

@hSATAC hSATAC requested a review from sjchmiela February 13, 2026 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no changelog PR that doesn't require a changelog entry

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants