feat: Add paginated loader to entity-database-adapter-knex by wschurman · Pull Request #422 · expo/entity

wschurman · 2026-02-05T22:14:04Z

Why

This is the main goal of the stack that started in #407. It adds a paginated loader for entities for applications that use the knex postgres database adapter.

How

Many many iterations of claude.

The general principle is hoisted from Expo server application code: fetch limit + 1 entities to know if there are more pages.

This makes use of the sql loader added in #414 to do the actual page load. Exact implementation described in docblocks.

Next PRs:

Add includeTotal option and use postgres windows if possible SELECT *, COUNT(*) OVER() as total_count
Add search (ilike and trigram)

Test Plan

Full test coverage and example usage in tests.

wschurman · 2026-02-05T22:14:28Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

codecov · 2026-02-05T22:17:13Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (56ec27d) to head (7f45f60).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##              main      #422    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files          108       108            
  Lines        15061     15606   +545     
  Branches       766       812    +46     
==========================================
+ Hits         15061     15606   +545

Flag	Coverage Δ
integration	`24.97% <96.89%> (+2.65%)`	⬆️
unittest	`95.90% <65.51%> (-0.44%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

quinlanj

lgtm, but will need to account for the tricky Date case in the future, if we want to support that

packages/entity-database-adapter-knex/src/internal/EntityKnexDataManager.ts

ide

Quin mentioned specifying an entity ID alone as a cursor. I think this is a pattern we will want to use often as it's compact and stable.

packages/entity-database-adapter-knex/src/AuthorizationResultBasedKnexEntityLoader.ts

packages/entity-database-adapter-knex/src/internal/EntityKnexDataManager.ts

wschurman · 2026-02-12T03:31:08Z

Updated to use ID alone as the opaque external cursor, and then a sub-query to derive the postgres cursor. A similar approach was already in progress for search trigram similarity so this makes them consistent.

wschurman · 2026-02-12T03:33:38Z

(requesting re-review, easiest to see the diff on graphite by using the versions compare. no tests needed to change which is a good sign)

# Why This PR adds search capabilities to the existing cursor-based pagination system added in #422 which only supported orderBy ordering. This adds: 1. Case-insensitive pattern matching (ILIKE) - Useful for basic text search across entity fields 2. Trigram similarity search - Provides fuzzy matching capabilities for more advanced search use cases like handling typos or finding similar text This enables building user-facing search features while maintaining the benefits of cursor-based pagination (stable results, efficient queries). # How The implementation introduces a unified PaginationSpecification that supports three strategies: 1. Standard pagination - The existing orderBy-based pagination 2. ILIKE search - Pattern matching with automatic wildcard escaping 3. Trigram search - PostgreSQL trigram similarity with configurable threshold Key implementation details: - Search terms are properly parameterized to prevent SQL injection - ILIKE special characters (%, _, \) are escaped to prevent pattern injection - Results maintain stable cursor-based pagination with proper ordering: - ILIKE: Ordered by search fields, then ID - Trigram: Ordered by exact match priority, similarity score, optional extra fields, then ID # Test Plan The PR includes comprehensive test coverage: - Unit tests for both AuthorizationResultBasedKnexEntityLoader and EnforcingKnexEntityLoader - Integration tests covering: - Forward/backward pagination with search - Multi-field search - Case-insensitive matching - Trigram similarity with various thresholds - Cursor stability across pages - Edge cases (empty results, special characters) - Combined with WHERE clauses - Security validation ensuring proper escaping and parameterization All existing tests pass, confirming backward compatibility for standard pagination when using the new API.

…#453) # Why The one downside of using id-based cursor pagination and subqueries (added in #422, #431) is that it risks the row referenced by the cursor being deleted between pagination requests. It's an edge case though to be clear. Encoding the full cursor alleviates this, but has it's own downsides since trigram similarity isn't encodable and neither are things like Date objects. It is still preferable to use ID-based cursors for all pagination. But it becomes the library's responsibility to explicitly document what the behavior is when a row referenced by the cursor is no longer present. This PR does this. # How When the cursor row is no longer present, the tuple evaluates to `NULL`, which produces a empty page. The alternative to this behavior is to run an id check query ahead of the pagination query, and throw an error if it doesn't exist, or even return the first page of data if it doesn't exist. Both of these are less optimal since they cause unexpected behavior during results consumption. So we keep the behavior as is and explicitly document it. # Test Plan Run new tests.

wschurman mentioned this pull request Feb 5, 2026

feat!: Add sql template and loader method #414

Merged

wschurman changed the base branch from wschurman/02-01-feat_add_sql_template_and_loader_method to graphite-base/422 February 6, 2026 19:37

wschurman force-pushed the wschurman/02-05-feat_add_paginated_loader_to_entity-database-adapter-knex branch from 5d121ae to aabff5d Compare February 6, 2026 19:38

wschurman changed the base branch from graphite-base/422 to wschurman/02-06-fix_use_tselectedfields_for_knex_loader_order_by_method February 6, 2026 19:38

wschurman mentioned this pull request Feb 6, 2026

fix: Use TSelectedFields for knex loader order by method #424

Merged

wschurman changed the base branch from wschurman/02-06-fix_use_tselectedfields_for_knex_loader_order_by_method to graphite-base/422 February 6, 2026 20:26

wschurman force-pushed the wschurman/02-05-feat_add_paginated_loader_to_entity-database-adapter-knex branch from aabff5d to da0673d Compare February 6, 2026 20:26

wschurman changed the base branch from graphite-base/422 to wschurman/02-06-fix_type_idfield_in_entityconfiguration_as_tidfield February 6, 2026 20:26

wschurman mentioned this pull request Feb 6, 2026

fix: Type idField in EntityConfiguration as TIDField #425

Merged

wschurman force-pushed the wschurman/02-05-feat_add_paginated_loader_to_entity-database-adapter-knex branch 4 times, most recently from 75f7914 to 4be6e84 Compare February 9, 2026 23:16

wschurman force-pushed the wschurman/02-06-fix_type_idfield_in_entityconfiguration_as_tidfield branch from 6164b63 to fbce886 Compare February 9, 2026 23:28

wschurman force-pushed the wschurman/02-05-feat_add_paginated_loader_to_entity-database-adapter-knex branch 2 times, most recently from 586186d to 806058c Compare February 9, 2026 23:31

wschurman force-pushed the wschurman/02-06-fix_type_idfield_in_entityconfiguration_as_tidfield branch 2 times, most recently from 29a3d8c to 0c6daeb Compare February 10, 2026 00:02

wschurman force-pushed the wschurman/02-05-feat_add_paginated_loader_to_entity-database-adapter-knex branch from 806058c to 11222bc Compare February 10, 2026 00:03

wschurman changed the base branch from wschurman/02-06-fix_type_idfield_in_entityconfiguration_as_tidfield to graphite-base/422 February 10, 2026 03:01

wschurman mentioned this pull request Feb 10, 2026

fix: move EntityPrivacyUtils back into core package #429

Merged

wschurman force-pushed the wschurman/02-05-feat_add_paginated_loader_to_entity-database-adapter-knex branch from 11222bc to 33e8e6f Compare February 10, 2026 03:33

wschurman force-pushed the graphite-base/422 branch from 0c6daeb to 64712a7 Compare February 10, 2026 03:33

wschurman changed the base branch from graphite-base/422 to wschurman/02-09-fix_move_entityprivacyutils_back_into_core_package February 10, 2026 03:33

wschurman requested review from ide and quinlanj February 10, 2026 04:07

This was referenced Feb 10, 2026

feat: add includeTotal option to pagination #430

Closed

feat: Add ilike and trigram similarity search to pagination #431

Merged

wschurman changed the title ~~feat!: Add paginated loader to entity-database-adapter-knex~~ feat: Add paginated loader to entity-database-adapter-knex Feb 10, 2026

wschurman force-pushed the wschurman/02-05-feat_add_paginated_loader_to_entity-database-adapter-knex branch from 0b4f87d to 1218be7 Compare February 11, 2026 03:30

quinlanj approved these changes Feb 11, 2026

View reviewed changes

packages/entity-database-adapter-knex/src/internal/EntityKnexDataManager.ts Outdated Show resolved Hide resolved

ide reviewed Feb 11, 2026

View reviewed changes

packages/entity-database-adapter-knex/src/AuthorizationResultBasedKnexEntityLoader.ts Outdated Show resolved Hide resolved

packages/entity-database-adapter-knex/src/internal/EntityKnexDataManager.ts Outdated Show resolved Hide resolved

wschurman force-pushed the wschurman/02-09-fix_move_entityprivacyutils_back_into_core_package branch 2 times, most recently from e63584e to 3e7e550 Compare February 12, 2026 01:31

wschurman force-pushed the wschurman/02-05-feat_add_paginated_loader_to_entity-database-adapter-knex branch from 1218be7 to 93efc2f Compare February 12, 2026 01:31

wschurman force-pushed the wschurman/02-09-fix_move_entityprivacyutils_back_into_core_package branch from 3e7e550 to 30fb9dc Compare February 12, 2026 03:14

wschurman force-pushed the wschurman/02-05-feat_add_paginated_loader_to_entity-database-adapter-knex branch from 93efc2f to d216f87 Compare February 12, 2026 03:14

wschurman changed the base branch from wschurman/02-09-fix_move_entityprivacyutils_back_into_core_package to graphite-base/422 February 12, 2026 03:20

wschurman force-pushed the wschurman/02-05-feat_add_paginated_loader_to_entity-database-adapter-knex branch from d216f87 to 0829b03 Compare February 12, 2026 03:20

wschurman force-pushed the graphite-base/422 branch from 30fb9dc to 56ec27d Compare February 12, 2026 03:20

graphite-app bot changed the base branch from graphite-base/422 to main February 12, 2026 03:21

wschurman force-pushed the wschurman/02-05-feat_add_paginated_loader_to_entity-database-adapter-knex branch 2 times, most recently from 351382a to b43bb56 Compare February 12, 2026 03:28

wschurman requested a review from quinlanj February 12, 2026 03:32

feat!: Add paginated loader to entity-database-adapter-knex

7f45f60

wschurman force-pushed the wschurman/02-05-feat_add_paginated_loader_to_entity-database-adapter-knex branch from b43bb56 to 7f45f60 Compare February 12, 2026 03:41

ide approved these changes Feb 12, 2026

View reviewed changes

wschurman merged commit 58deffd into main Feb 12, 2026
5 checks passed

wschurman deleted the wschurman/02-05-feat_add_paginated_loader_to_entity-database-adapter-knex branch February 12, 2026 19:49

wschurman mentioned this pull request Feb 17, 2026

fix: explicitly document behavior when cursor entity no longer exists #453

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add paginated loader to entity-database-adapter-knex#422

feat: Add paginated loader to entity-database-adapter-knex#422
wschurman merged 1 commit intomainfrom
wschurman/02-05-feat_add_paginated_loader_to_entity-database-adapter-knex

wschurman commented Feb 5, 2026 •

edited

Loading

Uh oh!

wschurman commented Feb 5, 2026 •

edited

Loading

Uh oh!

codecov bot commented Feb 5, 2026 •

edited

Loading

Uh oh!

quinlanj left a comment

Uh oh!

Uh oh!

ide left a comment

Uh oh!

Uh oh!

Uh oh!

wschurman commented Feb 12, 2026

Uh oh!

wschurman commented Feb 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

Conversation

wschurman commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

How

Test Plan

Uh oh!

wschurman commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

quinlanj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ide left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

wschurman commented Feb 12, 2026

Uh oh!

wschurman commented Feb 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

wschurman commented Feb 5, 2026 •

edited

Loading

wschurman commented Feb 5, 2026 •

edited

Loading

codecov bot commented Feb 5, 2026 •

edited

Loading