Writing fast pipelines

The library can make a shape fast; it cannot pick the shape for you. These are the ones that pay.

// Shapes that let a terminal operator take a fast path fx(list).map(f).toList() // hands the copy to the SDK's bulk fill fx(list).filter(p).toList() // same fx(list).map(f).distinct() // fuses into one stage, one loop foldBy(key, seed, f, list) // no per-key List, unlike groupBy + fold countBy(key, list) // foldBy with a counter, pre-named uniqStrict(list) // dedupe once when the result is reused

Lecture

Every example in the Dart vs FxDart comparison is measured against a hand-written imperative loop, and most land at or near it. Where a chain is slower, it is almost never the algorithm — both sides do the same work — and almost always one of four things: a stage boundary paid per element, an allocation per element, a callback run more often than it needs to be, or an upstream walked more than once.

1. End in a terminal operator

A terminal like toList sees the whole chain and can take a route no element-by-element consumer can. Over a List source, map(f).toList() and filter(p).toList() hand the copy to the SDK's own bulk fill, which writes the result without the per-element type check that package code is forced to pay. Pulling the same chain by hand with a for-in and collecting as you go gives up that route entirely.

2. Filter before you map

Stages run in the order you write them, and each element that survives a filter pays for every stage after it. Putting the cheap test first and the expensive transform second is free to do and often the single biggest win available.

3. Ask for the answer, not the ingredients

groupBy builds a List for every key — allocation proportional to the input — and if all you wanted was a total per key, those lists are built and discarded. foldBy accumulates straight into the result map, and countBy is the pre-named counter version. Reach for groupBy when you genuinely want the members.

These two are also the rare case where the operator is faster than the loop you would have written. The obvious hand-written line, counts[k] = (counts[k] ?? 0) + 1, touches the hash map twice per element — once to read, once to write back — and on a counting workload the map is essentially the whole cost. Both operators count into a mutable cell held in the map instead, so the map is written once per distinct key rather than once per element: ~1.5× on a million-row count, and it is why Most frequent log level beats a hand loop rather than trailing it.

4. A lazy chain re-runs on every pass

Laziness means the chain is a recipe, not a result: iterate it twice and the upstream runs twice. When the answer is used more than once, materialize it once — with toList, or uniqStrict where the dedupe itself is the thing you keep.

What laziness is still buying you

None of this argues against lazy chains. A take or head after a filter stops the source as soon as it has enough — the work simply never happens — and that is a category of saving no eager pipeline can match. Laziness costs a little per element and can save all of it.

Records are not free. A stage that produces a record — zip, zipWithIndex, pairwise, or a map to a tuple — allocates one per element, and a record cannot be reused. If the very next stage throws most of them away, consider whether the pipeline can carry indices or a single value instead; mapWithIndex exists precisely so that zipWithIndex().map(...) does not have to allocate a pair per element.

Demo 1 · Terminals and filter order

Demo 2 · foldBy over groupBy, and paying for a second pass

import 'package:fxdart/fxdart.dart';

// Rule 3: pick the operator that produces the answer you actually want.
// groupBy keeps every element; foldBy keeps only the running total.
void main() {
  final txns = [
    ('food', 12.5),
    ('rent', 800.0),
    ('food', 8.0),
    ('travel', 30.0),
    ('rent', 200.0),
  ];

// Allocates a List per key, then throws the lists away.
  final viaGroup = fx(txns)
      .groupBy((t) => t.$1)
      .map((k, v) => MapEntry(k, fx(v).sumBy((t) => t.$2)));
  print(viaGroup);

// Accumulates straight into the result map — no per-key lists at all.
  final viaFold = fx(txns).foldBy((t) => t.$1, 0.0, (sum, t) => sum + t.$2);
  print(viaFold); // same answer

// Same idea: countBy is foldBy with a counter, already named.
  print(fx(txns).countBy((t) => t.$1));

// Rule 4: a chain you iterate more than once re-runs its upstream every
  // time. Materialize once when you need the result twice.
  var runs = 0;
  final lazy = fx(txns).map((t) {
    runs++;
    return t.$1;
  }).distinct();
  lazy.toList();
  lazy.toList();
  print('lazy, two passes: $runs callback calls');

runs = 0;
  final once = fx(txns).map((t) {
    runs++;
    return t.$1;
  }).uniqStrict();
  once.toList();
  once.toList();
  print('uniqStrict, two passes: $runs callback calls');
}

Try it yourself

Exercise: the snippet walks its readings twice and builds a list it throws away. Rewrite it as one chain ending in a single terminal operator.

Related: toList — the terminal that most fast paths hang off · foldBy — aggregate without the groups · uniqStrict — dedupe once, reuse many times · mapWithIndex — the index without the record