fix: prevent heap overflow in large monorepo scans #1041
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add streaming-based filtering to globWithGitIgnore to prevent heap overflow when scanning large monorepos with 100k+ files. Instead of accumulating all file paths and filtering afterwards, files are now filtered during streaming which dramatically reduces memory usage. Changes: - Add `filter` option to globWithGitIgnore for early filtering during streaming - Add createSupportedFilesFilter helper to create filter from supported files - Update getPackageFilesForScan to use streaming filter Fixes SMO-522 Ported from v1.x commit 9bbb8e8 ([SMO-522] Fix heap overflow in large monorepo scans #1026) Co-authored-by: Mikola Lysenko <mikolalysenko@gmail.com>
Resolved conflict in CHANGELOG.md by keeping both changes: - Changed: Updated Coana CLI to v14.12.148 - Fixed: Heap overflow prevention in large monorepo scans
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Prevent heap overflow in large monorepo scans by using streaming-based filtering.
Note
Improves memory efficiency of large scans by filtering during glob streaming instead of after accumulating all paths.
filtertoglobWithGitIgnoreto apply early, maintain fast path when no negation/filter, and preserve negated ignore support via streamingcreateSupportedFilesFilterand use it ingetPackageFilesForScanto stream-filter supported files rather than post-filteringCHANGELOG.mdunder Unreleased with the fixWritten by Cursor Bugbot for commit ee2fb52. Configure here.