Node.js Streams

Node.js Streams: Processing Large Datasets & Memory-Efficient Pipelines

Streams in Node.js handle reading and writing data sequentially in chunks without loading entire files into memory at once. Node.js provides four fundamental stream types: Readable, Writable, Duplex, and Transform.

Stream Pipeline & Backpressure Mechanics

flowchart LR
    A["Readable Stream (Disk File / Network)"] -->|"Chunks of Buffer Data"| B["Transform Stream (Gzip / Cipher)"]
    B -->|"Processed Data Chunks"| C["Writable Stream (Destination File / HTTP Response)"]
    C -.->|"HighWaterMark Full (Backpressure)"| A

Practical Code Example

Using node:stream/promises pipeline() for memory-safe stream transformation:

import fs from 'node:fs';
import zlib from 'node:zlib';
import { pipeline } from 'node:stream/promises';
import path from 'node:path';

const sourceFile = path.join(process.cwd(), 'large_dataset.json');
const targetCompressedFile = path.join(process.cwd(), 'large_dataset.json.gz');

// Helper script to compress large file efficiently
async function compressFilePipeline() {
    console.log('Starting stream compression pipeline...');

    try {
        await pipeline(
            fs.createReadStream(sourceFile),  // Readable stream
            zlib.createGzip(),                // Transform stream
            fs.createWriteStream(targetCompressedFile) // Writable stream
        );

        console.log('File compressed successfully without loading whole file into RAM!');
    } catch (error) {
        console.error('Stream Pipeline Failed:', error.message);
    }
}

compressFilePipeline();

Best Practices & Gotchas

• Always Use pipeline() from node:stream/promises: Avoid .pipe() because .pipe() does not automatically clean up streams or handle error callbacks if an intermediary stream fails.
• Respect Backpressure: If writing directly to streams, check if stream.write() returns false, and wait for the 'drain' event before pushing more chunks.
• Tune highWaterMark: Adjust the stream buffer chunk size (highWaterMark, default 64KB) based on network throughput requirements.

Self-Check Challenge

What are the four core types of streams available in Node.js? Give one real-world example for each.