Flushing

Keep in mind that the functions defined in the jobs are run in web browsers and not servers. They are run in separate Web Workers to avoid interrupting the UI but still they are limited to the memory and CPU available on the host machine. It is possible to flush emits mid-processing to keep the resultant output file sizes small and to help keep memory consumption low. This ensures that the browsers processing the next stage will not have to load very large files into memory.

The following example will flush every 10000 emits:

execution.addStage(new MapStage(async function map(x) {
  for (let i = 0, n = x.length; i < n; i++) {
    let [k, v] = x[i];
    for (let j = 0, m = k.length; j < m; j++) {
      context.emit(k.charAt(j), v);
    }
    if (i % 10000 === 0) {
      await context.flush();
    }
  }
  await context.onComplete();
}));