Open
Conversation
|
arrdem
force-pushed
the
arrdem/uv-sdist-builds-with-exec-groups
branch
from
ee0bafb to
7a3dbaa
Compare
arrdem
added a commit
that referenced
this pull request
Jan 15, 2026
The Python ecosystem presently lacks ([PEP-720](https://peps.python.org/pep-0720/)) a comprehensive crossbuilding story. As a workaround for this and issues which could arise by "faking" crossbuilds eg by lying about the current interpreter platform, we can instead lean on RBE to ignore the issue by executing sdist to bdist build actions on workers matching the target configuration. With thanks to @fmeum for the guidance, this is achieved by using Bazel's [execution groups](https://bazel.build/extending/exec-groups) feature. We define a dummy toolchain of no files which can be resolved _only_ in a configuration matching the target configuration, and consequently inherits the `exec_properties` of the target platform. Then we create an execution group which contains that dummy toolchain. `ctx.actions.run` calls referencing an exec group inherit the exec properties of the chosen platform for which the toolchains in the group were resolved. Thus we have a way to enforce that the action will be placed on a host matching the target's configuration. The new e2e case exercises this behavior by forcibly marking a dependency which has no bdists (it was installed via sdist url) as being native and thus requiring platform-specific builds which must occur as platform-specific actions. This behavior can be observed in the following query ``` bazel aquery \ --platforms=//cases/uv-deps-650/source_crossbuild:arm64_linux \ --host_platform=//cases/uv-deps-650/source_crossbuild:amd64_linux \ --extra_execution_platforms=//cases/uv-deps-650/source_crossbuild/... \ 'deps(cases/uv-deps-650/source_crossbuild:image_index)' ``` The image index depends on both configurations of the binary, each of which has a dependency on the "native" source build of `cowsay`. Consequently we expect to see two `PySdistBuild` actions, In order to make performing these builds more efficient in RBE, this patch also reworks the sdist build helper so that we use Python to unzip the sdist archive in the same action step as we perform the build. In RBE this eliminates the time and transit overhead of using one action to run `tar.bzl` to unzip the sdist only to re-pack the resulting tree artifact and shovel it to a different worker for the subsequent sdist build action. ### Changes are visible to end-users: yes - Searched for relevant documentation and updated as needed: yes - Breaking change (forces users to change their own code or config): no - Suggested release notes appear below: yes The UV extension will now attempt to place sdist builds of native extensions on the target platform via RBE. ### Test plan See #771 for the test case code. Inspect the build plan for the image index, which should show one sdist build on arm64 and one on amd64. ``` ❯ bazel aquery \ --platforms=//cases/uv-deps-650/source_crossbuild:arm64_linux \ --extra_execution_platforms=//cases/uv-deps-650/source_crossbuild/... \ 'deps(cases/uv-deps-650/source_crossbuild:image_index)' \ | grep -A14 'PySdistBuild' | sed 's/Inputs: \(\[.*\]\)/Inputs: .../g' Mnemonic: PySdistBuild Target: @@aspect_rules_py++uv+sbuild__pypi__source-build__cowsay//:whl Configuration: k8-fastbuild-ST-93c492559deb Execution platform: //cases/uv-deps-650/source_crossbuild:arm64_linux ActionKey: 4d5bf8cafcebd3c553c2e38548995188c20986fd614ce920ba4e90f351ac255f Inputs: ... Outputs: [bazel-out/k8-fastbuild-ST-93c492559deb/bin/external/aspect_rules_py++uv+sbuild__pypi__source-build__cowsay/build (TreeArtifact)] Command Line: (exec bazel-out/k8-fastbuild-ST-93c492559deb/bin/external/aspect_rules_py++uv+sbuild__pypi__source-build__cowsay/.build_venv/bin/python3 \ external/aspect_rules_py+/uv/private/sdist_build/build_helper.py \ external/aspect_rules_py++uv+sdist__cowsay__47445cb2/file/cowsay-6.0.tar.gz \ bazel-out/k8-fastbuild-ST-93c492559deb/bin/external/aspect_rules_py++uv+sbuild__pypi__source-build__cowsay/build) # Configuration: 7329af2666e1bf46b2f1b89d88d6e90d5311d6fd69df686f2a70377db1254561 # Execution platform: //cases/uv-deps-650/source_crossbuild:arm64_linux ExecutionInfo: {Arch: arm64, OSFamily: Linux} -- Mnemonic: PySdistBuild Target: @@aspect_rules_py++uv+sbuild__pypi__source-build__cowsay//:whl Configuration: k8-fastbuild-ST-01f0fdcf49e8 Execution platform: //cases/uv-deps-650/source_crossbuild:amd64_linux ActionKey: 0b683ffca3367a28a07bee2fdc24a0913268d83a46b3444cc9f6a3e724571807 Inputs: ... Outputs: [bazel-out/k8-fastbuild-ST-01f0fdcf49e8/bin/external/aspect_rules_py++uv+sbuild__pypi__source-build__cowsay/build (TreeArtifact)] build__cowsay/.build_venv/bin/python3 \ external/aspect_rules_py+/uv/private/sdist_build/build_helper.py \ external/aspect_rules_py++uv+sdist__cowsay__47445cb2/file/cowsay-6.0.tar.gz \ bazel-out/k8-fastbuild-ST-01f0fdcf49e8/bin/external/aspect_rules_py++uv+sbuild__pypi__source-build__cowsay/build) # Configuration: 9475f7f1bf5753f90f43a325cac5dd2ab8bef1567797b087a6cb1a77b871b825 # Execution platform: //cases/uv-deps-650/source_crossbuild:amd64_linux ExecutionInfo: {Arch: amd64, OSFamily: Linux} ``` --------- Co-authored-by: aspect-marvin[bot] <marvin@aspect.build>
Base automatically changed from
arrdem/uv-sdist-builds-with-exec-groups
to
main
January 15, 2026 20:43
arrdem
force-pushed
the
arrdem/uv-sdist-builds-with-exec-groups-test
branch
from
f1c9e45 to
ba03701
Compare
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test cases extracted from #770 which we can't land because they (correctly!) require ARM RBE which we don't have set up.