Limitation of aio-files

[agraharishub]

Hi Team,
I have a requirement to generate a CSV file from stock data.
The processed stock data is stored in Adobe I/O Files in the following structure:
`inv/logs/stock///.json``

My goal is to read all files for a specific date and generate a single CSV.
Problem
When I list files for a given date folder using @adobe/aio-lib-files, I only receive 1000 files. There doesn’t appear to be any pagination mechanism available for listing beyond this limit.
Constraints
I cannot delete the original files, as they need to be retained.
Earlier, when the number of files was small, this approach worked fine.
Now that the file count exceeds 1000+, I’m unable to process all files.

Attempted Approach

I tried copying the date folder into a temporary folder, then reading and deleting files from the temp folder in batches. However, the copy() operation also handles only 1000 files at a time, so the same limitation applies.

Question
What is the recommended approach to:
Read and process more than 1000 files from Adobe I/O Files for a given directory/date?
Any guidance or best practices would be greatly appreciated.

Thank you so much for your attention and participation.