Chunking Blockchain Range Queries Safely
Split large block ranges into bounded chunks, preserve inclusive boundary rules and adapt to response-size limits without skipping or overlapping records incorrectly.
Overview
When translating these concepts into an integration, keep the scope described by the wallet history api with the result. The examples in wallet transaction history and smart contract event logs show how the same distinction affects neighbouring data questions.
Large historical queries can exceed time, row or response-size limits. Chunking makes progress manageable, but the client must preserve the original range and completion contract.
Define chunk boundaries
For inclusive block ranges, a chunk ending at B is followed by B plus one. For half-open ranges, the next start can equal the previous end. Do not mix conventions. Keep the selected accepted upper boundary stable while scanning, and preserve each chunk’s success or failure independently.
Worked example: an inclusive split
A query covers blocks 100 through 199. It splits into 100–149 and 150–199. Reusing 149 as the second start creates overlap; starting at 151 creates a gap. Deduplication can tolerate deliberate overlap, but it cannot recover a skipped block.
Adapt without hiding failures
Reduce chunk size when documented limits require it, while bounding attempts and preserving the pending range. Test a dense block that alone exceeds assumptions and a service returning truncated results. A successful chunk request is not complete unless pagination and response limits are satisfied. Persist processed ranges so an interruption resumes safely.
A practical acceptance test
Split inclusive blocks 100–199 into 100–149 and 150–199, then compare evidence IDs with a reference scan. Inject a dense block requiring pagination and fail one page. The chunk must remain incomplete until every required page is durable. Retrying should neither skip the dense block nor silently move the requested upper boundary to latest.
Questions about the topic
Frequently asked questions
Can deduplication repair skipped ranges?
No. It removes repeats but cannot create missing evidence.
Is one successful response always a completed chunk?
No. Check pagination and truncation semantics.
Standards & technical documentation
