Dedupe a paged feed by id
Requirement
A product feed arrives in three pages whose boundaries overlap, so some items appear on two pages. Flatten the pages and print each item exactly once — first occurrence wins, arrival order kept — keyed by its numeric id. The pages are in the code; both versions must print the lines shown under Expected output.
Expected output
#101 Desk lamp #102 Notebook #103 Pen set #104 Stapler #105 Monitor arm #106 Desk mat
Side by side
RxDart
FxDart
Why they differ
The interesting fact is where the duplicates sit. The page boundaries
overlap, so a repeated id arrives on a different page —
never next to its first occurrence once the pages are flattened.
That is exactly the job plain Stream.distinct silently
gets wrong: it is adjacent-only, so on this feed it would wave every
repeat straight through. The dedup here needs a whole-feed memory,
while the flattening itself is fluent in both models —
expand on the stream, flatMap on the pull
chain.
As in the unique-visitors pair, the global dedup is
distinctUnique against uniqBy — an
equals/hashCode pair versus one key
function — and the adjacent-vs-global naming split is the one the
status-changes page walks through. Both sides keep the same set of
seen keys and preserve first-arrival order, so the verdict is a tie —
the pull version just stays synchronous because the pages are
sitting in a local list.
Benchmark
N = 100
Time Tie
Peak memory Tie
N = 1,000,000
Time FxDart wins
Peak memory FxDart wins
Bars are medians of repeated timed iterations in fresh processes per side (small N is batched for timer resolution). Sides within 5% of each other — or within 0.6 ms, a difference no person can perceive — count as a tie; close relative races are re-measured up to 5 runs. In an app, anything under a few milliseconds is invisible to the user regardless of which bar is shorter. Memory is peak process RSS. The Dart VM and the dataset are identical on both sides, so the difference between the two bars is what the pipeline itself holds onto.