Node.js Streams

Node.js Streams: Processing Large Data in Constant Memory

A Node.js stream lets your code read or write data piece by piece as chunks arrive instead of loading an entire gigabyte file into memory at once. Streams power fs.createReadStream(), HTTP request/response bodies, and zlib compression pipelines.

Four Core Stream Types and pipeline()

Node.js organizes streams into four interfaces depending on data direction:

• Readable: Emits chunks from a source (fs.createReadStream(), http.IncomingMessage).
• Writable: Consumes chunks into a destination (fs.createWriteStream(), http.ServerResponse).
• Duplex: Both readable and writable independently (TCP sockets).
• Transform: Modifies chunks as they pass through (zlib.createGzip(), crypto ciphers).

Always connect streams with pipeline() from node:stream/promises instead of .pipe(). pipeline() automatically handles backpressure (pausing fast readers when disk writes lag) and cleans up all stream handles if any step errors.

import { createReadStream, createWriteStream } from 'node:fs';
import { createGzip } from 'node:zlib';
import { pipeline } from 'node:stream/promises';

async function compressLogFile(inputPath, outputPath) {
  await pipeline(
    createReadStream(inputPath),
    createGzip(),
    createWriteStream(outputPath)
  );
  console.log('Compression complete:', outputPath);
}

Iterating Over Readable Streams with for await...of

Readable streams implement JavaScript's async iterator protocol, letting you process chunks cleanly with for await (const chunk of readable) without wrestling with 'data' and 'end' event listeners.

Common Mistakes to Avoid

• Chaining .pipe() without error handlers on every stream: If the source file fails mid-stream, .pipe() does not destroy downstream streams or propagate errors, leaking file descriptors. Always use pipeline().
• Writing in a tight loop without respecting backpressure: Ignoring writable.write(chunk) === false buffers gigabytes in RAM; await 'drain' or use pipeline().
• Forgetting Readable.from() for quick in-memory streams: Readable.from(['line1 ', 'line2 ']) turns any array or async generator directly into a stream.

Frequently Asked Questions

How large is each chunk in a Node.js file stream?

By default, fs.createReadStream() reads 64 KB chunks (highWaterMark: 64 * 1024), keeping RAM usage flat even on a 10 GB file.

What is backpressure in Node.js streams?

Backpressure is the signal from a slow writable destination (like disk or network) telling a fast readable source to pause reading until the write buffer drains.

Can I convert a Node.js stream to a Web ReadableStream?

Yes. Readable.toWeb() and Readable.fromWeb() bridge Node streams with fetch() Web Streams cleanly.