[build-tools] Add eas/report_maestro_test_results build function#3388
[build-tools] Add eas/report_maestro_test_results build function#3388
Conversation
|
Size Change: +3.48 kB (0%) Total Size: 72.7 MB
|
2e32515 to
3201ab1
Compare
51c142a to
cc45900
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3388 +/- ##
==========================================
+ Coverage 52.26% 52.76% +0.50%
==========================================
Files 804 806 +2
Lines 33450 33621 +171
Branches 6975 7014 +39
==========================================
+ Hits 17480 17737 +257
+ Misses 15885 15799 -86
Partials 85 85 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
cc45900 to
f171e9e
Compare
|
Subscribed to pull request
Generated by CodeMention |
packages/build-tools/src/steps/functions/reportMaestroTestResults.ts
Outdated
Show resolved
Hide resolved
packages/build-tools/src/steps/functions/maestroResultParser.ts
Outdated
Show resolved
Hide resolved
packages/build-tools/src/steps/functions/maestroResultParser.ts
Outdated
Show resolved
Hide resolved
packages/build-tools/src/steps/functions/maestroResultParser.ts
Outdated
Show resolved
Hide resolved
packages/build-tools/src/steps/functions/maestroResultParser.ts
Outdated
Show resolved
Hide resolved
packages/build-tools/src/steps/functions/maestroResultParser.ts
Outdated
Show resolved
Hide resolved
packages/build-tools/src/steps/functions/reportMaestroTestResults.ts
Outdated
Show resolved
Hide resolved
275434c to
7fb1035
Compare
|
The |
packages/build-tools/src/steps/functions/reportMaestroTestResults.ts
Outdated
Show resolved
Hide resolved
| // the same name. JUnit XML only contains names (not file paths), making it impossible | ||
| // to map duplicates back to their original flow files. Skip and let the user fix it. |
There was a problem hiding this comment.
Yeah, the JUnit is super simple like this:
<?xml version='1.0' encoding='UTF-8'?>
<testsuites>
<testsuite name="Test Suite" device="iPhone 16 - iOS 18.6 - 59572B21-FEAF-4BEC-B8EB-5793C9A31034" tests="6" failures="3" time="78.0">
<testcase id="settings_screen" name="settings_screen" classname="settings_screen" time="22.0" status="ERROR">
<properties>
<property name="priority" value="high"/>
<property name="suite" value="smoke"/>
<property name="jira" value="APP-101"/>
</properties>
<failure>Element not found: Text matching regex: Settings</failure>
</testcase>
</testsuite>
</testsuites>| } | ||
|
|
||
| // Maestro allows overriding flow names via config, so different flow files can share | ||
| // the same name. JUnit XML only contains names (not file paths), making it impossible |
There was a problem hiding this comment.
After having parsed both commands and JUnit, what is your professional opinion on which one should we use?
There was a problem hiding this comment.
After looking at both, I’m leaning toward using the original JSON as the source of truth.
It simply has more complete data, and we can support all formats without forcing users to adopt JUnit.
With JUnit, we’d still need JSON anyway (no tags / flowFileName), and duration is only second-level precision.
The only real advantage of JUnit is stability — it’s less likely to change upstream. JSON doesn’t give us that guarantee, so if we go with it, we might want some automation to detect schema changes early.
packages/build-tools/src/steps/functions/reportMaestroTestResults.ts
Outdated
Show resolved
Hide resolved
packages/build-tools/src/steps/functions/reportMaestroTestResults.ts
Outdated
Show resolved
Hide resolved
packages/build-tools/src/steps/functions/reportMaestroTestResults.ts
Outdated
Show resolved
Hide resolved
b6efd20 to
198a91d
Compare
Signed-off-by: Ash Wu <hsatac@gmail.com>
198a91d to
1476a49
Compare
|
⏩ The changelog entry check has been skipped since the "no changelog" label is present. |
Why
Maestro test jobs produce per-flow results on the build worker (pass/fail, duration, error messages, retries), but this data is lost after the job finishes.
We want to persist these as
WorkflowDeviceTestCaseResultrecords to surface test outcomes in the dashboard.How
Add a new
eas/report_maestro_test_resultsbuild function that:junit_report_directory(primary data source)ai-*.jsondebug output intests_directorycreateWorkflowDeviceTestCaseResultsGraphQL mutation viagql.tada+graphqlClientKey details:
ai-*.jsononly used forflow_file_pathmapping and retry countjunit_report_directoryadded as output oneas/__maestro_test(set whenoutput_format=junit) and as optional input oneas/report_maestro_test_resultsreuse_devices: trueandfalsepatterns)fs.realpathto handle symlinks (/tmp→/private/tmpon macOS)graphqlClientexposed onCustomBuildContextfor step functions to useTest Plan