Skip to content

Conversation

@sahar-fehri
Copy link
Contributor

@sahar-fehri sahar-fehri commented Dec 9, 2025

Description

Optimizes TokenListController storage to reduce write amplification by persisting tokensChainsCache via StorageService using per-chain files instead of a single monolithic state property.

Mobile: MetaMask/metamask-mobile#24019

Extension: MetaMask/metamask-extension#39250

Related: https://github.com/MetaMask/metamask-mobile/pull/22943/files

Related: https://github.com/MetaMask/decisions/pull/110

Related: #7192

Explanation

The tokensChainsCache (~5MB total, containing token lists for all chains) was persisted as part of the controller state. Every time a single chain's token list was updated (~100-500KB), the entire ~5MB cache was rewritten to disk, causing:

  • Startup performance issues (loading large state on app initialization)
  • Runtime performance degradation (frequent large writes during token fetches)
  • Impacts both extension

Solution

Per-Chain File Storage:
Each chain's cache is now stored in a separate file (e.g., tokensChainsCache:0x1, tokensChainsCache:0x89)
Only the updated chain (~100-500KB) is written on each token fetch, reducing write operations by ~90-95%
All chains are loaded in parallel at startup to maintain compatibility with TokenDetectionController
Key Changes:

  • Set tokensChainsCache metadata to persist: false to prevent framework-managed persistence
  • Added #loadCacheFromStorage() to load all per-chain files in parallel on initialization
  • Added #saveChainCacheToStorage(chainId) to persist only the specific chain that changed
  • Added #migrateStateToStorage() to automatically migrate existing cache data on first launch after upgrade
  • Updated clearingTokenListData() to remove all per-chain files

References

Checklist

  • I've updated the test suite for new or updated code as appropriate
  • I've updated documentation (JSDoc, Markdown, etc.) for new or updated code as appropriate
  • I've communicated my changes to consumers by updating changelogs for packages I've changed
  • I've introduced breaking changes in this PR and have prepared draft pull requests for clients and consumer packages to resolve them

Note

BREAKING

  • TokenListController now persists tokensChainsCache via StorageService using per‑chain keys (e.g., tokensChainsCache:0x1) and requires clients to call await initialize() after construction
  • State metadata sets tokensChainsCache persist: false; state changes are auto‑persisted via debounced subscription

Behavior/impl changes

  • Loads all chain caches in parallel on startup; tracks chains loaded from storage to avoid redundant writes
  • Persists only changed chains; guards against race conditions; enhanced error handling for save/load/clear paths
  • clearingTokenListData() now async and removes all per‑chain files; deprecated start/restart/stop wrappers maintained

Project updates

  • Adds @metamask/storage-service dependency and TS references; extensive unit tests added/updated; changelog updated with breaking note

Written by Cursor Bugbot for commit d6cac27. This will update automatically on new commits. Configure here.

tokensChainsCache: {
includeInStateLogs: false,
persist: true,
persist: false, // Persisted separately via StorageService
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Making this false to block disk writes

salimtb
salimtb previously approved these changes Dec 16, 2025
@sahar-fehri
Copy link
Contributor Author

@metamaskbot publish-preview

// Only remove loaded chains, not chains from initial state that need first persist
for (const chainId of Object.keys(loadedCache) as Hex[]) {
this.#changedChainsToPersist.delete(chainId);
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initialization removes chains from persistence queue incorrectly

Medium Severity

The #loadCacheFromStorage method removes ALL chains in loadedCache from #changedChainsToPersist, but loadedCache contains all chains from storage, not just chains that were actually added to state. When a chain exists in both initial state and storage, the update() call preserves the initial state's data (correctly), but the chain is still removed from the persistence queue (incorrectly). This means initial state data that should be persisted to storage is silently dropped if the same chain also exists in storage with older data.

Fix in Cursor Fix in Web

@sahar-fehri
Copy link
Contributor Author

@metamaskbot publish-preview

@github-actions
Copy link
Contributor

Preview builds have been published. See these instructions for more information about preview builds.

Expand for full list of packages and versions.
{
  "@metamask-previews/account-tree-controller": "4.0.0-preview-821afcb8",
  "@metamask-previews/accounts-controller": "35.0.1-preview-821afcb8",
  "@metamask-previews/address-book-controller": "7.0.1-preview-821afcb8",
  "@metamask-previews/analytics-controller": "1.0.0-preview-821afcb8",
  "@metamask-previews/announcement-controller": "8.0.0-preview-821afcb8",
  "@metamask-previews/app-metadata-controller": "2.0.0-preview-821afcb8",
  "@metamask-previews/approval-controller": "8.0.0-preview-821afcb8",
  "@metamask-previews/assets-controller": "0.0.0-preview-821afcb8",
  "@metamask-previews/assets-controllers": "95.1.0-preview-821afcb8",
  "@metamask-previews/base-controller": "9.0.0-preview-821afcb8",
  "@metamask-previews/bridge-controller": "64.4.1-preview-821afcb8",
  "@metamask-previews/bridge-status-controller": "64.4.2-preview-821afcb8",
  "@metamask-previews/build-utils": "3.0.4-preview-821afcb8",
  "@metamask-previews/chain-agnostic-permission": "1.4.0-preview-821afcb8",
  "@metamask-previews/claims-controller": "0.4.1-preview-821afcb8",
  "@metamask-previews/composable-controller": "12.0.0-preview-821afcb8",
  "@metamask-previews/controller-utils": "11.18.0-preview-821afcb8",
  "@metamask-previews/core-backend": "5.0.0-preview-821afcb8",
  "@metamask-previews/delegation-controller": "2.0.0-preview-821afcb8",
  "@metamask-previews/earn-controller": "11.0.0-preview-821afcb8",
  "@metamask-previews/eip-5792-middleware": "2.1.0-preview-821afcb8",
  "@metamask-previews/eip-7702-internal-rpc-middleware": "0.1.0-preview-821afcb8",
  "@metamask-previews/eip1193-permission-middleware": "1.0.3-preview-821afcb8",
  "@metamask-previews/ens-controller": "19.0.1-preview-821afcb8",
  "@metamask-previews/error-reporting-service": "3.0.1-preview-821afcb8",
  "@metamask-previews/eth-block-tracker": "15.0.0-preview-821afcb8",
  "@metamask-previews/eth-json-rpc-middleware": "22.0.1-preview-821afcb8",
  "@metamask-previews/eth-json-rpc-provider": "6.0.0-preview-821afcb8",
  "@metamask-previews/foundryup": "1.0.1-preview-821afcb8",
  "@metamask-previews/gas-fee-controller": "26.0.1-preview-821afcb8",
  "@metamask-previews/gator-permissions-controller": "0.8.0-preview-821afcb8",
  "@metamask-previews/json-rpc-engine": "10.2.0-preview-821afcb8",
  "@metamask-previews/json-rpc-middleware-stream": "8.0.8-preview-821afcb8",
  "@metamask-previews/keyring-controller": "25.0.0-preview-821afcb8",
  "@metamask-previews/logging-controller": "7.0.1-preview-821afcb8",
  "@metamask-previews/message-manager": "14.1.0-preview-821afcb8",
  "@metamask-previews/messenger": "0.3.0-preview-821afcb8",
  "@metamask-previews/multichain-account-service": "5.0.0-preview-821afcb8",
  "@metamask-previews/multichain-api-middleware": "1.2.5-preview-821afcb8",
  "@metamask-previews/multichain-network-controller": "3.0.1-preview-821afcb8",
  "@metamask-previews/multichain-transactions-controller": "7.0.0-preview-821afcb8",
  "@metamask-previews/name-controller": "9.0.0-preview-821afcb8",
  "@metamask-previews/network-controller": "28.0.0-preview-821afcb8",
  "@metamask-previews/network-enablement-controller": "4.0.0-preview-821afcb8",
  "@metamask-previews/notification-services-controller": "21.0.0-preview-821afcb8",
  "@metamask-previews/permission-controller": "12.2.0-preview-821afcb8",
  "@metamask-previews/permission-log-controller": "5.0.0-preview-821afcb8",
  "@metamask-previews/phishing-controller": "16.1.0-preview-821afcb8",
  "@metamask-previews/polling-controller": "16.0.1-preview-821afcb8",
  "@metamask-previews/preferences-controller": "22.0.0-preview-821afcb8",
  "@metamask-previews/profile-metrics-controller": "2.0.0-preview-821afcb8",
  "@metamask-previews/profile-sync-controller": "27.0.0-preview-821afcb8",
  "@metamask-previews/ramps-controller": "3.0.0-preview-821afcb8",
  "@metamask-previews/rate-limit-controller": "7.0.0-preview-821afcb8",
  "@metamask-previews/remote-feature-flag-controller": "4.0.0-preview-821afcb8",
  "@metamask-previews/sample-controllers": "4.0.1-preview-821afcb8",
  "@metamask-previews/seedless-onboarding-controller": "7.1.0-preview-821afcb8",
  "@metamask-previews/selected-network-controller": "26.0.1-preview-821afcb8",
  "@metamask-previews/shield-controller": "4.1.0-preview-821afcb8",
  "@metamask-previews/signature-controller": "38.0.1-preview-821afcb8",
  "@metamask-previews/storage-service": "0.0.1-preview-821afcb8",
  "@metamask-previews/subscription-controller": "5.4.0-preview-821afcb8",
  "@metamask-previews/token-search-discovery-controller": "4.0.0-preview-821afcb8",
  "@metamask-previews/transaction-controller": "62.9.1-preview-821afcb8",
  "@metamask-previews/transaction-pay-controller": "11.0.1-preview-821afcb8",
  "@metamask-previews/user-operation-controller": "41.0.1-preview-821afcb8"
}

@sahar-fehri
Copy link
Contributor Author

@metamaskbot publish-preview

@github-actions
Copy link
Contributor

Preview builds have been published. See these instructions for more information about preview builds.

Expand for full list of packages and versions.
{
  "@metamask-previews/account-tree-controller": "4.0.0-preview-009e026d",
  "@metamask-previews/accounts-controller": "35.0.1-preview-009e026d",
  "@metamask-previews/address-book-controller": "7.0.1-preview-009e026d",
  "@metamask-previews/analytics-controller": "1.0.0-preview-009e026d",
  "@metamask-previews/announcement-controller": "8.0.0-preview-009e026d",
  "@metamask-previews/app-metadata-controller": "2.0.0-preview-009e026d",
  "@metamask-previews/approval-controller": "8.0.0-preview-009e026d",
  "@metamask-previews/assets-controller": "0.0.0-preview-009e026d",
  "@metamask-previews/assets-controllers": "95.1.0-preview-009e026d",
  "@metamask-previews/base-controller": "9.0.0-preview-009e026d",
  "@metamask-previews/bridge-controller": "64.4.1-preview-009e026d",
  "@metamask-previews/bridge-status-controller": "64.4.2-preview-009e026d",
  "@metamask-previews/build-utils": "3.0.4-preview-009e026d",
  "@metamask-previews/chain-agnostic-permission": "1.4.0-preview-009e026d",
  "@metamask-previews/claims-controller": "0.4.1-preview-009e026d",
  "@metamask-previews/composable-controller": "12.0.0-preview-009e026d",
  "@metamask-previews/controller-utils": "11.18.0-preview-009e026d",
  "@metamask-previews/core-backend": "5.0.0-preview-009e026d",
  "@metamask-previews/delegation-controller": "2.0.0-preview-009e026d",
  "@metamask-previews/earn-controller": "11.0.0-preview-009e026d",
  "@metamask-previews/eip-5792-middleware": "2.1.0-preview-009e026d",
  "@metamask-previews/eip-7702-internal-rpc-middleware": "0.1.0-preview-009e026d",
  "@metamask-previews/eip1193-permission-middleware": "1.0.3-preview-009e026d",
  "@metamask-previews/ens-controller": "19.0.1-preview-009e026d",
  "@metamask-previews/error-reporting-service": "3.0.1-preview-009e026d",
  "@metamask-previews/eth-block-tracker": "15.0.0-preview-009e026d",
  "@metamask-previews/eth-json-rpc-middleware": "22.0.1-preview-009e026d",
  "@metamask-previews/eth-json-rpc-provider": "6.0.0-preview-009e026d",
  "@metamask-previews/foundryup": "1.0.1-preview-009e026d",
  "@metamask-previews/gas-fee-controller": "26.0.1-preview-009e026d",
  "@metamask-previews/gator-permissions-controller": "0.8.0-preview-009e026d",
  "@metamask-previews/json-rpc-engine": "10.2.0-preview-009e026d",
  "@metamask-previews/json-rpc-middleware-stream": "8.0.8-preview-009e026d",
  "@metamask-previews/keyring-controller": "25.0.0-preview-009e026d",
  "@metamask-previews/logging-controller": "7.0.1-preview-009e026d",
  "@metamask-previews/message-manager": "14.1.0-preview-009e026d",
  "@metamask-previews/messenger": "0.3.0-preview-009e026d",
  "@metamask-previews/multichain-account-service": "5.0.0-preview-009e026d",
  "@metamask-previews/multichain-api-middleware": "1.2.5-preview-009e026d",
  "@metamask-previews/multichain-network-controller": "3.0.1-preview-009e026d",
  "@metamask-previews/multichain-transactions-controller": "7.0.0-preview-009e026d",
  "@metamask-previews/name-controller": "9.0.0-preview-009e026d",
  "@metamask-previews/network-controller": "28.0.0-preview-009e026d",
  "@metamask-previews/network-enablement-controller": "4.0.0-preview-009e026d",
  "@metamask-previews/notification-services-controller": "21.0.0-preview-009e026d",
  "@metamask-previews/permission-controller": "12.2.0-preview-009e026d",
  "@metamask-previews/permission-log-controller": "5.0.0-preview-009e026d",
  "@metamask-previews/phishing-controller": "16.1.0-preview-009e026d",
  "@metamask-previews/polling-controller": "16.0.1-preview-009e026d",
  "@metamask-previews/preferences-controller": "22.0.0-preview-009e026d",
  "@metamask-previews/profile-metrics-controller": "2.0.0-preview-009e026d",
  "@metamask-previews/profile-sync-controller": "27.0.0-preview-009e026d",
  "@metamask-previews/ramps-controller": "3.0.0-preview-009e026d",
  "@metamask-previews/rate-limit-controller": "7.0.0-preview-009e026d",
  "@metamask-previews/remote-feature-flag-controller": "4.0.0-preview-009e026d",
  "@metamask-previews/sample-controllers": "4.0.1-preview-009e026d",
  "@metamask-previews/seedless-onboarding-controller": "7.1.0-preview-009e026d",
  "@metamask-previews/selected-network-controller": "26.0.1-preview-009e026d",
  "@metamask-previews/shield-controller": "4.1.0-preview-009e026d",
  "@metamask-previews/signature-controller": "38.0.1-preview-009e026d",
  "@metamask-previews/storage-service": "0.0.1-preview-009e026d",
  "@metamask-previews/subscription-controller": "5.4.0-preview-009e026d",
  "@metamask-previews/token-search-discovery-controller": "4.0.0-preview-009e026d",
  "@metamask-previews/transaction-controller": "62.9.1-preview-009e026d",
  "@metamask-previews/transaction-pay-controller": "11.0.1-preview-009e026d",
  "@metamask-previews/user-operation-controller": "41.0.1-preview-009e026d"
}

Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

this.#persistDebounceTimer = undefined;
}
this.#changedChainsToPersist.clear();
this.#chainsLoadedFromStorage.clear();
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pending state changes lost when controller is destroyed

Medium Severity

The destroy() method cancels the debounce timer and clears #changedChainsToPersist without flushing pending changes to storage. If the controller is destroyed within 500ms of a token fetch or state update, the updated token cache data will be discarded and never persisted. On next app launch, stale data would be loaded from storage instead of the fresher data that was fetched but not persisted.

Fix in Cursor Fix in Web

@sahar-fehri
Copy link
Contributor Author

@metamaskbot publish-preview

@github-actions
Copy link
Contributor

Preview builds have been published. See these instructions for more information about preview builds.

Expand for full list of packages and versions.
{
  "@metamask-previews/account-tree-controller": "4.0.0-preview-a94732c3",
  "@metamask-previews/accounts-controller": "35.0.2-preview-a94732c3",
  "@metamask-previews/address-book-controller": "7.0.1-preview-a94732c3",
  "@metamask-previews/ai-controllers": "0.0.0-preview-a94732c3",
  "@metamask-previews/analytics-controller": "1.0.0-preview-a94732c3",
  "@metamask-previews/announcement-controller": "8.0.0-preview-a94732c3",
  "@metamask-previews/app-metadata-controller": "2.0.0-preview-a94732c3",
  "@metamask-previews/approval-controller": "8.0.0-preview-a94732c3",
  "@metamask-previews/assets-controller": "0.0.0-preview-a94732c3",
  "@metamask-previews/assets-controllers": "96.0.0-preview-a94732c3",
  "@metamask-previews/base-controller": "9.0.0-preview-a94732c3",
  "@metamask-previews/bridge-controller": "64.8.1-preview-a94732c3",
  "@metamask-previews/bridge-status-controller": "64.4.4-preview-a94732c3",
  "@metamask-previews/build-utils": "3.0.4-preview-a94732c3",
  "@metamask-previews/chain-agnostic-permission": "1.4.0-preview-a94732c3",
  "@metamask-previews/claims-controller": "0.4.1-preview-a94732c3",
  "@metamask-previews/composable-controller": "12.0.0-preview-a94732c3",
  "@metamask-previews/connectivity-controller": "0.1.0-preview-a94732c3",
  "@metamask-previews/controller-utils": "11.18.0-preview-a94732c3",
  "@metamask-previews/core-backend": "5.0.0-preview-a94732c3",
  "@metamask-previews/delegation-controller": "2.0.0-preview-a94732c3",
  "@metamask-previews/earn-controller": "11.1.0-preview-a94732c3",
  "@metamask-previews/eip-5792-middleware": "2.1.0-preview-a94732c3",
  "@metamask-previews/eip-7702-internal-rpc-middleware": "0.1.0-preview-a94732c3",
  "@metamask-previews/eip1193-permission-middleware": "1.0.3-preview-a94732c3",
  "@metamask-previews/ens-controller": "19.0.2-preview-a94732c3",
  "@metamask-previews/error-reporting-service": "3.0.1-preview-a94732c3",
  "@metamask-previews/eth-block-tracker": "15.0.1-preview-a94732c3",
  "@metamask-previews/eth-json-rpc-middleware": "23.0.0-preview-a94732c3",
  "@metamask-previews/eth-json-rpc-provider": "6.0.0-preview-a94732c3",
  "@metamask-previews/foundryup": "1.0.1-preview-a94732c3",
  "@metamask-previews/gas-fee-controller": "26.0.2-preview-a94732c3",
  "@metamask-previews/gator-permissions-controller": "1.1.0-preview-a94732c3",
  "@metamask-previews/json-rpc-engine": "10.2.1-preview-a94732c3",
  "@metamask-previews/json-rpc-middleware-stream": "8.0.8-preview-a94732c3",
  "@metamask-previews/keyring-controller": "25.1.0-preview-a94732c3",
  "@metamask-previews/logging-controller": "7.0.1-preview-a94732c3",
  "@metamask-previews/message-manager": "14.1.0-preview-a94732c3",
  "@metamask-previews/messenger": "0.3.0-preview-a94732c3",
  "@metamask-previews/multichain-account-service": "5.1.0-preview-a94732c3",
  "@metamask-previews/multichain-api-middleware": "1.2.6-preview-a94732c3",
  "@metamask-previews/multichain-network-controller": "3.0.2-preview-a94732c3",
  "@metamask-previews/multichain-transactions-controller": "7.0.0-preview-a94732c3",
  "@metamask-previews/name-controller": "9.0.0-preview-a94732c3",
  "@metamask-previews/network-controller": "29.0.0-preview-a94732c3",
  "@metamask-previews/network-enablement-controller": "4.1.0-preview-a94732c3",
  "@metamask-previews/notification-services-controller": "21.0.0-preview-a94732c3",
  "@metamask-previews/permission-controller": "12.2.0-preview-a94732c3",
  "@metamask-previews/permission-log-controller": "5.0.0-preview-a94732c3",
  "@metamask-previews/perps-controller": "0.0.0-preview-a94732c3",
  "@metamask-previews/phishing-controller": "16.1.0-preview-a94732c3",
  "@metamask-previews/polling-controller": "16.0.2-preview-a94732c3",
  "@metamask-previews/preferences-controller": "22.0.0-preview-a94732c3",
  "@metamask-previews/profile-metrics-controller": "3.0.0-preview-a94732c3",
  "@metamask-previews/profile-sync-controller": "27.0.0-preview-a94732c3",
  "@metamask-previews/ramps-controller": "4.1.0-preview-a94732c3",
  "@metamask-previews/rate-limit-controller": "7.0.0-preview-a94732c3",
  "@metamask-previews/remote-feature-flag-controller": "4.0.0-preview-a94732c3",
  "@metamask-previews/sample-controllers": "4.0.2-preview-a94732c3",
  "@metamask-previews/seedless-onboarding-controller": "7.1.0-preview-a94732c3",
  "@metamask-previews/selected-network-controller": "26.0.2-preview-a94732c3",
  "@metamask-previews/shield-controller": "5.0.0-preview-a94732c3",
  "@metamask-previews/signature-controller": "39.0.1-preview-a94732c3",
  "@metamask-previews/storage-service": "0.0.1-preview-a94732c3",
  "@metamask-previews/subscription-controller": "5.4.0-preview-a94732c3",
  "@metamask-previews/token-search-discovery-controller": "4.0.0-preview-a94732c3",
  "@metamask-previews/transaction-controller": "62.9.2-preview-a94732c3",
  "@metamask-previews/transaction-pay-controller": "11.1.0-preview-a94732c3",
  "@metamask-previews/user-operation-controller": "41.0.2-preview-a94732c3"
}

Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants