Crawl pages until exhausted
Requirement
A paged orders API returns three orders per page and an empty list once the data runs out (page 4). Crawl page by page until the empty page, flatten the orders into one list, and print them plus how many pages were actually fetched — exactly four; the crawl must never request page 5. The fake API is in the code; both versions must print the lines shown under Expected output.
Expected output
order#1 order#2 order#3 order#4 order#5 order#6 order#7 order#8 order#9 pages fetched: 4
Side by side
RxDart
FxDart
Why they differ
Pagination is the pull model: fetch a page, look at it,
decide whether to ask for another. The FxDart side writes that down
directly — an endless sync* cursor of page numbers that
only advances when the pipeline demands the next one,
map(fetchPage), takeWhile(isNotEmpty),
flatten. Nothing bounds the cursor because demand is the bound: when
takeWhile sees the empty page it simply stops pulling,
and page 5 is never even generated.
The stream side gets to the same place, but only by borrowing pull
mechanics: an endless async* cursor — plain Dart rather
than an Rx operator — paused into demand-driven behavior by
asyncMap's backpressure, and a takeWhile
whose cancellation stops the crawl at the empty page. It works, and
prints the same pages fetched: 4 — because pause,
resume and cancel are exactly the stream model's back-channel for
simulating "ask again when ready". The pull side did not need the
simulation: demand is its normal mode. Jobs where the consumer's
state decides whether more input should exist are pull-shaped, and
this is the cleanest case of it.
Benchmark
Async case: the headline scale is N = 10,000, not 1,000,000. Every element costs an event-loop turn on both sides, so a million real awaits would measure Dart's event loop for minutes — not the pipeline. Delays are zero-length and the example's concurrency limit is kept; what the bars compare is the pipeline machinery.
N = 100
Time Tie
Peak memory Tie
N = 10,000
Time Tie
Peak memory Tie
Bars are medians of repeated timed iterations in fresh processes per side (small N is batched for timer resolution). Sides within 5% of each other — or within 0.6 ms, a difference no person can perceive — count as a tie; close relative races are re-measured up to 5 runs. In an app, anything under a few milliseconds is invisible to the user regardless of which bar is shorter. Memory is peak process RSS. The Dart VM and the dataset are identical on both sides, so the difference between the two bars is what the pipeline itself holds onto.