uv.lock support

MODULE.bazel

@@ -50,9 +50,13 @@ python.defaults(
 python.toolchain(
     python_version = "3.11",
 )
+python.toolchain(
+    python_version = "3.14",
+)
 use_repo(
     python,
     "python_3_11",
+    "python_3_14_host",
     "pythons_hub",
     python = "python_versions",
 )
@@ -341,7 +345,12 @@ dev_pip.parse(
     python_version = "3.11",
     requirements_lock = "//tests/multi_pypi/beta:requirements.txt",
 )
-use_repo(dev_pip, "dev_pip", "pypi_alpha", "pypi_beta", "pypiserver")
+dev_pip.parse(
+    hub_name = "uv_pypi",
+    python_version = "3.12",
+    uv_lock = "//tests/uv_pypi:uv.lock",
+)
+use_repo(dev_pip, "dev_pip", "pypi_alpha", "pypi_beta", "pypiserver", "uv_pypi")
 
 # Bazel integration test setup below
 

plan.md

@@ -0,0 +1,42 @@
+# Plan for adding uv.lock support to pip.parse
+
+## Part 1: Analyze and Plan (Done)
+- Analyzed `python/extensions/pip.bzl`, `python/private/pypi/hub_builder.bzl`, `python/private/pypi/uv_lock.bzl`.
+- Confirmed `uv.lock` structure contains wheel URLs and resolution markers.
+- Identified need to modify `pip_repository_attrs.bzl` and `hub_builder.bzl`.
+
+## Part 2: Basic Implementation
+1.  **Modify `python/private/pypi/pip_repository_attrs.bzl`**:
+    - Add `uv_lock` attribute (label, allow_single_file=True).
+    - Add `_toml2json` attribute (label, default pointing to a tool). Note: Need to verify if `_toml2json` is already available or needs to be added. The `uv_lock.bzl` helper uses `attr._toml2json`, so it must be present on the calling rule/tag.
+
+2.  **Modify `python/private/pypi/hub_builder.bzl`**:
+    - In `_pip_parse`:
+        - Check if `pip_attr.uv_lock` is set.
+        - If set, call `convert_uv_lock_to_json`.
+        - Parse the JSON result.
+        - Iterate over packages in the JSON.
+        - Group packages by name.
+        - For this step, select the highest version for each package name.
+        - Select the first wheel URL for that version.
+        - Create `whl_library` repositories for these wheels.
+        - Ensure these `whl_library` calls are integrated into `self._whl_libraries`.
+
+3.  **Verify**:
+    - Run `bazel run //tests/uv_pypi:bin`.
+
+## Part 3: Advanced Implementation (Multiple Versions)
+1.  **Handle Resolution Markers**:
+    - Parse `resolution-markers` from `uv.lock` packages.
+    - Instead of picking one version, keep all versions that have distinct resolution markers.
+
+2.  **Use `wheel_tags_settings`**:
+    - In `hub_builder.bzl`, when constructing the hub repository content (via `hub_repository`), we need to pass information about these multiple versions.
+    - The `hub_repository` rule (or the macros creating it) needs to generate `define_wheel_tag_settings` in the `BUILD.bazel` of the hub.
+    - Generate `alias` targets using `select()` based on the defined settings.
+
+3.  **Refactor Hub Generation**:
+    - Update `hub_repository.bzl` (or the template it uses) to support this new "multi-version via select" pattern, if it doesn't already. The prompt suggests modifying the "hub build file for a package".
+
+4.  **Verify**:
+    - Run the test again. It should correctly pick `absl-py` 2.3.1 or 2.4.0 based on the environment/platform.

prompt.md

@@ -0,0 +1,71 @@
+
+We're adding uv.lock support to pip.parse.
+
+A test has been added to `//test/uv_pypi`
+
+Verify functionality by running `bazel run //tests/uv_pypi:bin`
+
+The desired interface is by setting a `uv_lock` attribute in `pip.parse`. This
+has been configured in MODULE.bazel for the test already:
+
+```
+dev_pip.parse(
+    hub_name = "uv_pypi",
+    python_version = "3.12",
+    uv_lock = "//tests/uv_pypi:uv.lock",
+)
+```
+
+A key helper to accomplish this is to use the toml2json tool, which handles
+converting the toml-format of uv.lock to JSON, which can then be processed
+by Starlark code. The name of this helper is `convert_uv_lock_to_json`
+in `python/private/pypi/uv_lock.bzl`
+
+Some background:
+
+The pypi integration uses a "hub" and "spokes" design. The hub is the
+`@uv_pypi` repo. This is basically a collection of BUILD files that
+have aliases that point to spoke repos. Spoke repos use the `whl_library`
+repository rule which downloads and extracts that actual wheel.
+
+Part 1:
+
+Analyze the pypi extension code base and write a plan of changes to make to
+`plan.md` that accomplish the different parts of what needs to be done.
+
+Part 2:
+
+Modify the pip.parse extension to convert the uv.lock file to JSON.
+
+From that, create a whl_library for each wheel URL. For now, if there
+are multiple versions of a package, take the highest version package. If
+there are multiple wheel URLs, use the first one.
+
+Part 3:
+
+Modify the whl_library generation logic to handle when there are multiple
+versions of a package available. This should use the "resolution-markers"
+information to generate select() expressions to pick between the different
+packages available. The wheel_tags_settings rule is a key helper for this:
+it can take a resolution-markers expression and evaluate it so that select()
+expressions can match it.
+
+As part of implementing this logic, the BUILD file for a package in the hub
+repo should use wheel_tags_settings. For example, if the absl_py package
+has two wheels, then the hub build file should look something like:
+
+```
+# File: @uv_pypi//absl_py:BUILD.bazel
+
+define_wheel_tag_settings([
+  ("@absl_py_a//:pkg", "resolution marker expr for a"),
+  ("@absl_py_b//:pkg", "resolution marker expr for b"),
+])
+alias(
+  name = "absl_py",
+  actual = select({
+    "pick_0": "@absl_py_a//:pkg",
+    "pick_1": "@absl_py_b//:pkg",
+  })
+)
+```

python/extensions/uv_external_deps.bzl

@@ -0,0 +1,152 @@
+"""A module extension for uv external dependencies.
+
+This extension allows users to define external dependencies using uv.
+"""
+
+load("//python/private:repo_utils.bzl", "REPO_DEBUG_ENV_VAR", "repo_utils")
+load("//python/private:text_util.bzl", "render")
+
+def _wheel_repo_impl(rctx):
+    rctx.download(rctx.attr.urls, output = "output.zip")
+    rctx.file("BUILD.bazel", "exports_files(glob(['**']))")
+
+wheel_repo = repository_rule(
+    implementation = _wheel_repo_impl,
+    attrs = {
+        "urls": attr.string_list(),
+    },
+)
+
+_PACKAGE_BUILD_TEMPLATE = """
+
+load("@rules_python//python/private/pypi:uv_lock_targets.bzl", "define_targets")
+
+package(
+    default_visibility = ["//visibility:public"]
+)
+exports_files(glob(["**"]))
+
+define_targets(
+    name = "{package}",
+    selectors = {selectors}
+)
+"""
+
+def _hub_package_build_file(rctx, package, selectors):
+    rctx.file(
+        "{}/BUILD.bazel".format(package),
+        _PACKAGE_BUILD_TEMPLATE.format(
+            package = package,
+            selectors = render.list(selectors),
+        ),
+    )
+
+def _hub_repo_impl(rctx):
+    hub_selectors = json.decode(rctx.attr.selectors)
+
+    for package, selectors in hub_selectors.items():
+        _hub_package_build_file(rctx, package, selectors)
+
+    rctx.file("BUILD.bazel", "")
+
+hub_repo = repository_rule(
+    implementation = _hub_repo_impl,
+    attrs = {
+        "selectors": attr.string(),
+    },
+)
+
+def wheel_tags_from_wheel_url(url):
+    _, _, basename = url.rpartition("/")
+    basename = basename.removesuffix(".whl")
+    distro, _, tail = basename.partition("-")
+    version, _, tail = tail.partition("-")
+    py_tag, _, tail = tail.partition("-")
+    abi, _, tail = tail.partition("-")
+    platform, _, tail = tail.partition("-")
+    return {
+        "python_tag": py_tag,
+        "abi_tag": abi,
+        "platform_tag": platform,
+    }
+    #charset_normalizer-3.4.4-cp310-cp310-macosx_10_9_universal2.whl",
+
+def _uv_external_deps_extension_impl(mctx):
+    logger = repo_utils.logger(mctx, "uvextdeps")
+
+    # sources[distro][version][type][url] = any_of_conditions
+    # url -> info
+    sources = {}
+    hub_name = None
+    for mod in mctx.modules:
+        for hub in mod.tags.hub:
+            out = _convert_uv_lock_to_json(mctx, hub, logger)
+            lock_info = json.decode(out)
+            break
+
+    # Basically what we're doing is:
+    # URLs are given name R (repo name). Thus, we never duplicate a download.
+    # Repo R is used if conditions C are met (wheel tags, marker, and
+    # custom config settings for that lock file).
+    url_to_repo = {}
+    hub_selectors = {}
+
+    packages = lock_info["package"]
+    for distro in packages:
+        # todo: handle source{virtual = "."}
+        if "wheels" not in distro:
+            continue
+        for i, wheel in enumerate(distro["wheels"]):
+            url = wheel["url"]
+            if url not in url_to_repo:
+                # todo: normalize dash to underscore etc
+                name = "{distro}_{version}_{i}".format(
+                    distro = distro["name"],
+                    version = distro["version"],
+                    i = i,
+                )
+                wheel_repo(
+                    name = name,
+                    urls = [url],
+                )
+            else:
+                name = url_to_repo[url]
+
+            hub_selectors.setdefault(distro["name"], [])
+            wheel_tags = wheel_tags_from_wheel_url(url)
+            resolution_markers = distro.get("resolution-markers")
+            if resolution_markers:
+                for marker in resolution_markers:
+                    hub_selectors[distro["name"]].append(struct(
+                        wheel_tags = wheel_tags,
+                        config_settings = hub.config_settings,
+                        marker = marker,
+                        actual_repo = name,
+                    ))
+            else:
+                hub_selectors[distro["name"]].append(struct(
+                    wheel_tags = wheel_tags,
+                    config_settings = hub.config_settings,
+                    marker = None,
+                    actual_repo = name,
+                ))
+
+    hub_repo(
+        name = hub.name,
+        selectors = json.encode(hub_selectors),
+    )
+
+uv_external_deps = module_extension(
+    implementation = _uv_external_deps_extension_impl,
+    tag_classes = {
+        "hub": tag_class(attrs = {
+            "name": attr.string(),
+            "srcs": attr.label_list(),
+            "config_settings": attr.label_list(),
+            "_toml2json": attr.label(default = "//tools/toml2json:toml2json.py"),
+            "_python_interpreter_target": attr.label(
+                default = "@python_3_14_host//:python",
+            ),
+        }),
+    },
+)

python/private/pypi/extension.bzl

@@ -272,6 +272,7 @@ You cannot use both the additive_build_content and additive_build_content_file a
     exposed_packages = {}
     extra_aliases = {}
     whl_libraries = {}
+    uv_selectors = {}
     for hub in pip_hub_map.values():
         out = hub.build()
 
@@ -285,6 +286,7 @@ You cannot use both the additive_build_content and additive_build_content_file a
         extra_aliases[hub.name] = out.extra_aliases
         hub_group_map[hub.name] = out.group_map
         hub_whl_map[hub.name] = out.whl_map
+        uv_selectors[hub.name] = out.uv_selectors
 
     return struct(
         config = config,
@@ -294,6 +296,7 @@ You cannot use both the additive_build_content and additive_build_content_file a
         hub_whl_map = hub_whl_map,
         whl_libraries = whl_libraries,
         whl_mods = whl_mods,
+        uv_selectors = uv_selectors,
         platform_config_settings = {
             hub_name: {
                 platform_name: sorted([str(Label(cv)) for cv in p.config_settings])
@@ -386,6 +389,7 @@ def _pip_impl(module_ctx):
                 key: whl_config_settings_to_json(values)
                 for key, values in whl_map.items()
             },
+            uv_selectors = mods.uv_selectors.get(hub_name, {}),
             packages = mods.exposed_packages.get(hub_name, []),
             platform_config_settings = mods.platform_config_settings.get(hub_name, {}),
             groups = mods.hub_group_map.get(hub_name),