Cursor tree optimization + beauty refactoring (#43)#63
Open
theronic wants to merge 18 commits intofeature/write-schema-dslfrom
Open
Cursor tree optimization + beauty refactoring (#43)#63theronic wants to merge 18 commits intofeature/write-schema-dslfrom
theronic wants to merge 18 commits intofeature/write-schema-dslfrom
Conversation
7 new tests verifying v2 cursor structure, intermediate advancement, backward compat, pagination correctness, performance (index-range call counting), lookup-subjects, and exhausted path preservation. All fail against current implementation as expected — no implementation code yet. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Cursor-based pagination now tracks per-path intermediate state via a v2
cursor format {:v 2 :e <eid> :p {<path-idx> <intermediate-cursor-eid>}},
allowing later pages to skip exhausted intermediates via d/index-range
seek (arrow-to-relation) or drop-while (arrow-to-permission). Includes
v1 backward compatibility, spiceomic coercion for v2 cursors, 12 new
cursor tree tests + 5 stress tests (30 total, 327 assertions, 0 failures),
cleanup of unused Cursor defrecord and dead code, and a detailed critique
report.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Seven concrete proposals to reduce indexed.clj by ~200 lines while making the cursor tree feel foundational rather than bolted-on. Key ideas: extract tracking-min transducer, unify arrow-via-intermediates pattern (written 4x), kill unused [eid path] tuples, merge traverse functions, unify forward/reverse lookup via direction parameter. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Detailed TDD implementation plan for 7 proposals from the beauty report. Phases 1-3 are fully specified with exact test code and implementation steps. Phases 4-7 have summaries and key subtleties identified but need the same level of detail. Includes deep structural analysis of forward/reverse asymmetries that constrain the unification approach. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phases 4-7 now have the same level of precision as Phases 1-3: - Phase 4: arrow-via-intermediates extraction with TDD tests - Phase 5: traverse-permission-path merge with visited-paths - Phase 6: opaque cursor tokens with TTL and backward compat - Phase 7: unified lookup directions with direction maps Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add tracking-min private fn (7 lines) that wraps a collection with volatile side-channel tracking of the minimum contributing value - Replace 4 identical 4-line vswap blocks in traverse-permission-path-via-subject and traverse-permission-path-reverse with single tracking-min calls - Add tracking-min-test and count-resources-propagates-volatile-state-test - Update plan status with implementation progress - 30 tests, 286 assertions, 0 failures Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…r helpers Helpers are defined but call sites not yet replaced. Next session should wire them in and fix count-resources volatile propagation. - Add extract-cursor-eid: extracts eid from v1 or v2 cursor - Add build-v2-cursor: builds cursor with carry-forward semantics - Add count-resources-propagates-volatile-state-test - Update plan with detailed implementation progress and next steps Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase 2: Wire extract-cursor-eid + build-v2-cursor into 5 call sites, fix count-resources volatile propagation Phase 3: Kill [eid path] tuples — traverse-permission-path returns bare eids Phase 4: Extract arrow-via-intermediates — unify 4 arrow branches into 1 function Phase 5: Reduce traverse-permission-path from ~100 lines to ~15-line thin wrapper with visited-paths cycle detection Phase 6: Add opaque cursor tokens (eacl1_<base64>) with TTL to core.clj, wire into all spiceomic functions Phase 7: Unify lookup-resources/lookup-subjects with direction maps (forward-direction, reverse-direction) indexed.clj: 740 → 616 lines (17% reduction) Tests: 34 indexed (308 assertions) + 3 spice (62 assertions) = 0 failures Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Owner
Author
|
Addressed in #63, in human review. |
Remove per-element vswap! tracking in arrow-via-intermediates. Since intermediates are scanned in ascending EID order via d/index-range, the first non-empty sub-sequence's intermediate is always the minimum. Pre-compute it once at setup instead of tracking per-element. Fix can? map arity: default nil consistency to fully-consistent instead of asserting equality with nil. Multi-path benchmark (4 arrow paths, 375k servers) shows v6.2 is 7-15x faster than v6.1: pagination avg 13ms vs 90ms, deep page 1999 at 3.7ms vs 56ms. Simple 2-hop limit=1000 improved from 20ms to 15ms. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add `lazy-merge2-dedupe-longs` with primitive `==`/`<` comparison instead of boxed `Numbers.lt(Object,Object)` through keyfn indirection - Eager `mapv` fold2 tree construction (was lazy `map`) - Key-cmp on pre-extracted keys (halves keyfn invocations) - `seq` nil checks instead of `(empty?)` double-negation - Heap-based PriorityQueue merge implemented but rejected (slower at all k) - Dispatch: `identical? keyfn identity` → fold2-longs fast path - 14 new correctness tests (oracle comparison, edge cases, laziness, tracking-min) - Benchmark harness in lazy_merge_sort_bench.clj - Gen 1 oracle copy in lazy_merge_sort/legacy.clj - All 48 tests pass (375 assertions, 0 failures) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The reported "2.7x slower first page" was a JIT compilation outlier from a single cold sample. After warmup: v6.2 page 0 median is 14ms vs v6.1's 84ms. No trade-offs — v6.2 is faster on all pages. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…-sort Gen 1 (linear scan), Gen 2 (pairwise no-dedup), heap, and dedupe-by moved to legacy namespace. Production lazy_merge_sort.clj now contains only the Gen 3 optimized fold2 with primitive long fast path. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace nil cursor + drop-while with (dec intermediate-cursor-eid) cursor threading in forward arrow-to-permission branch. The recursive traverse-permission-path call now receives the cursor, pushing skip logic into d/index-range B-tree seeks instead of O(k) linear scan. Adds 2 new tests for seek efficiency and inclusive boundary semantics. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace (or cursor-eid 0) + post-filter with (if cursor-eid (inc cursor-eid) 0) in all d/index-range start tuples. Since d/index-range is inclusive on start, (inc cursor-eid) makes the range exclusive, eliminating per-element (filter #(> % cursor-eid)). Also replace (map extract-*) (filter some?) chains with (keep extract-*). Changes across all traversal branches: - Forward :relation, :self-permission, :arrow (both to-relation and to-permission) - Reverse :relation, :arrow to-relation result-fn - traverse-single-path (unused but consistent) Reverse direction: fixed O(k) -> O(log N) for lookup-subjects pagination. Previously used hardcoded 0 in start tuple, scanning from beginning and filtering client-side. Now uses cursor-eid for B-tree seek. Benchmark (30 accounts x 500 servers, limit=50): Before: first page 1.27ms, pagination 1.39ms/page After: first page 1.00ms, pagination 1.05ms/page (-24%) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Regression detection benchmark for cursor-tree pagination performance. Tests 4-path permission graph (server.view = account->admin + team->admin + vpc->admin + shared_admin) with 30 accounts x 500 servers = 15k total. Run via: clojure -M:bench Thresholds: first page <75ms, per-page <50ms (generous for CI/slow hardware). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Benchmark comparison of feature/write-schema-dsl vs feature/cursor-tree on multi-path pagination (30 accounts x 500 servers, limit=50). Results: first page 2.70ms → 0.88ms (3.1x), pagination 2.53ms → 0.99ms/page (2.6x). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
min(intermediate eid)per permission path during pagination, enabling efficient B-tree seek on subsequent pages instead of scanning from the beginningindexed.cljfrom ~740 to 616 lines while adding opaque tokens, direction maps, and cycle detection"eacl1_<base64(edn)>"strings with 300s TTL — internals no longer leak through the API boundaryKey changes
tracking-min— extracts 4 inlinevswap!patterns into 1 functionarrow-via-intermediates— unifies 4 arrow branches (2 forward + 2 reverse) into 1 parameterized functiontraverse-permission-path— reduced from ~100 lines to ~15-line thin wrapper withvisited-pathscycle detectionforward-direction/reverse-directionmaps — unifylookup-resourcesandlookup-subjectsinto sharedlazy-merged-lookup+lookupcursor->token/token->cursorincore.clj— opaque Base64 tokens with TTLcount-resourcesbug fix — now propagates volatile state to cursor:p(was pass-through only)Test plan
🤖 Generated with Claude Code