-
Notifications
You must be signed in to change notification settings - Fork 2
OCI indexer #30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
OCI indexer #30
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces support for OCI container images, enabling lazy loading and zero data duplication. It also includes the `clipctl` command-line tool for indexing and mounting these images. Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Add delay and verification for FUSE mounts. Remove unused symlink tests. Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Introduces layer caching for OCI storage and enables direct indexing from OCI registries for Clip v2. This significantly speeds up image builds and reduces archive sizes. Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Replaces in-memory layer cache with disk cache and integrates with remote content cache. This significantly reduces memory usage and improves performance by avoiding repeated decompression. Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
Co-authored-by: luke <luke@smartshare.io>
* Refactor: Remove verbose flag, use debug logging Co-authored-by: luke <luke@smartshare.io> * feat: Add SetLogLevel function and logging documentation Removes verbose flags and adds a function to control log levels. Co-authored-by: luke <luke@smartshare.io> * feat: Add OCI image indexing progress reporting Co-authored-by: luke <luke@smartshare.io> * remove logging test * remove usage * remove test --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Luke Lombardi <luke@beam.cloud>
* Refactor: Improve OCI caching memory efficiency and consistency Co-authored-by: luke <luke@smartshare.io> * remove files --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Luke Lombardi <luke@beam.cloud>
* Refactor: Fix OCI cache lookup and hash generation Co-authored-by: luke <luke@smartshare.io> * feat: Pass ContentCache to OCI storage layer Fixes ContentCache passthrough issue, enabling cluster-wide layer sharing. Co-authored-by: luke <luke@smartshare.io> * Refactor: Standardize OCI cache key format to sha256_hash Co-authored-by: luke <luke@smartshare.io> * Refactor: Use pure hex hash for cache keys Use just the hex hash for cache keys, enabling true content-addressing and deduplication. Co-authored-by: luke <luke@smartshare.io> * Refactor: Use decompressed hash for content caching This change introduces a new cache for mapping layer digests to their decompressed hashes. This allows for true content-addressable storage and sharing of decompressed layers across different CLIP images. The cache key for remote content cache is now the decompressed hash, enabling efficient range reads and deduplication. Co-authored-by: luke <luke@smartshare.io> * feat: Compute and store decompressed layer hashes during indexing Co-authored-by: luke <luke@smartshare.io> * Refactor: Improve OCI indexer and layer caching tests (#37) This commit refactors the OCI indexer for better performance and readability. It also enhances layer caching tests and fixes existing storage tests. Co-authored-by: Cursor Agent <cursoragent@cursor.com> * clean up --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Luke Lombardi <luke@beam.cloud>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat: Add checkpoint-based OCI layer decompression Co-authored-by: luke <luke@smartshare.io> * Refactor: Move NearestCheckpoint to common and enable checkpoints Co-authored-by: luke <luke@smartshare.io> * remove md files --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Luke Lombardi <luke@beam.cloud>
* feat: Implement pluggable OCI registry authentication Co-authored-by: luke <luke@smartshare.io> * cleanup * more cleanup --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Luke Lombardi <luke@beam.cloud>
* Refactor OCI auth to use authn.FromConfig Co-authored-by: luke <luke@smartshare.io> * feat: Add ECR and AWS credential providers Introduces new credential providers for AWS ECR and general AWS credentials. Enhances static provider with wildcard pattern matching for registries. Improves credential type detection and parsing logic. Co-authored-by: luke <luke@smartshare.io> * Refactor credential provider logic for better registry support Co-authored-by: luke <luke@smartshare.io> * Refactor: Improve credential parsing and logging This commit enhances the credential parsing logic to handle nested JSON structures, particularly those generated by Beta9. It also refines logging within the ECR provider and OCI storage components for better visibility into credential retrieval and usage. Co-authored-by: luke <luke@smartshare.io> * Refactor ParseCredentialsFromJSON to support beta9 format Co-authored-by: luke <luke@smartshare.io> * Refactor logging for credential provider creation Co-authored-by: luke <luke@smartshare.io> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat: Embed OCI image metadata in clip index Co-authored-by: luke <luke@smartshare.io> * remove md --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Luke Lombardi <luke@beam.cloud>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note
Introduce OCI image indexing and runtime with checkpointed lazy reads, layered caching, and pluggable registry auth, plus extensive tests and build updates.
CreateFromOCIImage,IndexOCIImage) with layer whiteout handling, inode generation, and gzip checkpointing; writesOCIStorageInfoandRemoteRefentries.ImageMetadata,LayerMetadata).OCIClipStoragewith 3-tier lazy reads: disk cache → content cache (range reads) → registry; optional checkpoint-based partial decompression.NewClipStorageto OCI backend; support cache/config options.RegistryCredentialProvider(env, Docker config, keychain, static, ECR, AWS helper, callback, caching) and helpers to parse/construct providers.RemoteRef; cache pipeline updates; newSetLogLeveland extendedMountOptions(checkpoints, content cache, registry creds).docker build.Written by Cursor Bugbot for commit e0fd643. This will update automatically on new commits. Configure here.