Skip to content

Conversation

@luke-lombardi
Copy link
Contributor

@luke-lombardi luke-lombardi commented Oct 29, 2025

Note

Introduce OCI image indexing and runtime with checkpointed lazy reads, layered caching, and pluggable registry auth, plus extensive tests and build updates.

  • OCI v2 Support:
    • Implement metadata-only OCI indexing (CreateFromOCIImage, IndexOCIImage) with layer whiteout handling, inode generation, and gzip checkpointing; writes OCIStorageInfo and RemoteRef entries.
    • Extend archive read/write to encode/decode OCI storage info; add gob registrations.
    • Add image metadata embedding (ImageMetadata, LayerMetadata).
  • Storage:
    • New OCIClipStorage with 3-tier lazy reads: disk cache → content cache (range reads) → registry; optional checkpoint-based partial decompression.
    • Route in NewClipStorage to OCI backend; support cache/config options.
  • Auth:
    • Pluggable RegistryCredentialProvider (env, Docker config, keychain, static, ECR, AWS helper, callback, caching) and helpers to parse/construct providers.
  • Filesystem/API:
    • FS read path supports RemoteRef; cache pipeline updates; new SetLogLevel and extended MountOptions (checkpoints, content cache, registry creds).
  • Common/Infra:
    • Add gzip/zstd index types, metrics collection, and new storage mode constants.
  • Build/Housekeeping:
    • Dockerfile builds all packages; Makefile uses docker build.
  • Tests:
    • Extensive tests for OCI format, storage, performance, credentials, and cache behavior.

Written by Cursor Bugbot for commit e0fd643. This will update automatically on new commits. Configure here.

cursoragent and others added 30 commits October 26, 2025 21:02
This commit introduces support for OCI container images, enabling lazy loading and zero data duplication. It also includes the `clipctl` command-line tool for indexing and mounting these images.

Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Add delay and verification for FUSE mounts. Remove unused symlink tests.

Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Introduces layer caching for OCI storage and enables direct indexing from OCI registries for Clip v2. This significantly speeds up image builds and reduces archive sizes.

Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Replaces in-memory layer cache with disk cache and integrates with remote content cache. This significantly reduces memory usage and improves performance by avoiding repeated decompression.

Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
cursor[bot]

This comment was marked as outdated.

cursor[bot]

This comment was marked as outdated.

luke-beamcloud and others added 7 commits October 29, 2025 14:14
* Refactor: Remove verbose flag, use debug logging

Co-authored-by: luke <luke@smartshare.io>

* feat: Add SetLogLevel function and logging documentation

Removes verbose flags and adds a function to control log levels.

Co-authored-by: luke <luke@smartshare.io>

* feat: Add OCI image indexing progress reporting

Co-authored-by: luke <luke@smartshare.io>

* remove logging test

* remove usage

* remove test

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Luke Lombardi <luke@beam.cloud>
* Refactor: Improve OCI caching memory efficiency and consistency

Co-authored-by: luke <luke@smartshare.io>

* remove files

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Luke Lombardi <luke@beam.cloud>
cursor[bot]

This comment was marked as outdated.

cursor[bot]

This comment was marked as outdated.

luke-lombardi and others added 8 commits October 30, 2025 16:46
* Refactor: Fix OCI cache lookup and hash generation

Co-authored-by: luke <luke@smartshare.io>

* feat: Pass ContentCache to OCI storage layer

Fixes ContentCache passthrough issue, enabling cluster-wide layer sharing.

Co-authored-by: luke <luke@smartshare.io>

* Refactor: Standardize OCI cache key format to sha256_hash

Co-authored-by: luke <luke@smartshare.io>

* Refactor: Use pure hex hash for cache keys

Use just the hex hash for cache keys, enabling true content-addressing and deduplication.

Co-authored-by: luke <luke@smartshare.io>

* Refactor: Use decompressed hash for content caching

This change introduces a new cache for mapping layer digests to their decompressed hashes. This allows for true content-addressable storage and sharing of decompressed layers across different CLIP images. The cache key for remote content cache is now the decompressed hash, enabling efficient range reads and deduplication.

Co-authored-by: luke <luke@smartshare.io>

* feat: Compute and store decompressed layer hashes during indexing

Co-authored-by: luke <luke@smartshare.io>

* Refactor: Improve OCI indexer and layer caching tests (#37)

This commit refactors the OCI indexer for better performance and readability. It also enhances layer caching tests and fixes existing storage tests.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* clean up

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Luke Lombardi <luke@beam.cloud>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat: Add checkpoint-based OCI layer decompression

Co-authored-by: luke <luke@smartshare.io>

* Refactor: Move NearestCheckpoint to common and enable checkpoints

Co-authored-by: luke <luke@smartshare.io>

* remove md files

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Luke Lombardi <luke@beam.cloud>
* feat: Implement pluggable OCI registry authentication

Co-authored-by: luke <luke@smartshare.io>

* cleanup

* more cleanup

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Luke Lombardi <luke@beam.cloud>
* Refactor OCI auth to use authn.FromConfig

Co-authored-by: luke <luke@smartshare.io>

* feat: Add ECR and AWS credential providers

Introduces new credential providers for AWS ECR and general AWS credentials.
Enhances static provider with wildcard pattern matching for registries.
Improves credential type detection and parsing logic.

Co-authored-by: luke <luke@smartshare.io>

* Refactor credential provider logic for better registry support

Co-authored-by: luke <luke@smartshare.io>

* Refactor: Improve credential parsing and logging

This commit enhances the credential parsing logic to handle nested JSON structures, particularly those generated by Beta9. It also refines logging within the ECR provider and OCI storage components for better visibility into credential retrieval and usage.

Co-authored-by: luke <luke@smartshare.io>

* Refactor ParseCredentialsFromJSON to support beta9 format

Co-authored-by: luke <luke@smartshare.io>

* Refactor logging for credential provider creation

Co-authored-by: luke <luke@smartshare.io>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
cursor[bot]

This comment was marked as outdated.

* feat: Embed OCI image metadata in clip index

Co-authored-by: luke <luke@smartshare.io>

* remove md

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Luke Lombardi <luke@beam.cloud>
@luke-lombardi luke-lombardi merged commit 1c761e3 into master Nov 1, 2025
2 checks passed
@luke-lombardi luke-lombardi deleted the ll/oci-index branch November 1, 2025 19:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants