Streams in Node.js: processing large files without memory drain

1. Overview

Reading an entire file into memory with fs.readFile is convenient when the file is small. With log files several GB in size or CSV files with millions of lines, this approach causes the process to bloat RAM or crash. This article shows you how to process large files using streams in Node.js: read each chunk, split lines, filter data, compress, and write to a new file. You’ll also understand backpressure and learn the right way to handle errors with pipeline.

All examples use Node.js 20+ and ES modules. Your package.json needs "type": "module". The article requires no external libraries since fs, stream, zlib, and readline are built-in.

2. The Problem with readFile

First, let’s create an orders.csv file with about 5 million lines for testing:

// generate-orders.mjs
import { createWriteStream } from 'node:fs';
import { once } from 'node:events';

const out = createWriteStream('orders.csv');
out.write('id,customer,amount,status\n');

for (let i = 1; i <= 5_000_000; i++) {
  const status = i % 10 === 0 ? 'FAILED' : 'PAID';
  const ok = out.write(`${i},customer-${i % 1000},${(i % 500) + 0.99},${status}\n`);
  if (!ok) await once(out, 'drain'); // wait for the buffer to drain before writing more
}
out.end();

This code already uses backpressure. Section 5 will explain the mechanism in detail. The resulting file is about 180 MB.

Now let’s count the number of FAILED orders the “naive” way:

// naive-count.mjs
import { readFile } from 'node:fs/promises';

const content = await readFile('orders.csv', 'utf8');
const failed = content.split('\n').filter(line => line.endsWith(',FAILED')).length;

console.log('FAILED:', failed);
console.log('RSS MB:', Math.round(process.memoryUsage().rss / 1024 / 1024));

FAILED: 500000
RSS MB: 1046

A 180 MB file consumes about 1 GB of RAM. The process has to hold three things at the same time: the raw buffer, the UTF-8 string, and an array of 5 million substrings. When the file exceeds about 2 GB, readFile throws ERR_FS_FILE_TOO_LARGE and can’t read it anymore.

3. Four Basic Stream Types

A stream is data that flows through in chunks. Node.js has four types of streams:

Type	Role	Example
Readable	Data source	fs.createReadStream, process.stdin
Writable	Data destination	fs.createWriteStream, process.stdout
Duplex	Both readable and writable, independent directions	TCP socket
Transform	Duplex with output depending on input	zlib.createGzip()

Each stream has an internal buffer. The limit of this buffer is determined by highWaterMark. By default, fs.createReadStream uses 64 KiB. Streams in object mode use 16 objects.

4. Reading Large Files Chunk by Chunk

4.1. Using for await on Readable

The most concise way to consume a Readable is async iteration:

// read-chunks.mjs
import { createReadStream } from 'node:fs';

const stream = createReadStream('orders.csv', { highWaterMark: 64 * 1024 });

let bytes = 0;
let chunks = 0;
for await (const chunk of stream) {
  bytes += chunk.length;
  chunks++;
}

console.log({ chunks, mb: (bytes / 1024 / 1024).toFixed(1) });
console.log('RSS MB:', Math.round(process.memoryUsage().rss / 1024 / 1024));

{ chunks: 2853, mb: 178.3 }
RSS MB: 52

Each chunk is a Buffer of at most 64 KiB. The loop only requests the next chunk after finishing the current one. Therefore, RAM is almost independent of file size.

4.2. Reading Line by Line with readline

Chunks are cut by byte, not by line. If you need to process line by line, use readline:

// count-failed.mjs
import { createReadStream } from 'node:fs';
import readline from 'node:readline';

const rl = readline.createInterface({
  input: createReadStream('orders.csv'),
  crlfDelay: Infinity, // treat \r\n as a single line break
});

let failed = 0;
for await (const line of rl) {
  if (line.endsWith(',FAILED')) failed++;
}
console.log('FAILED:', failed);

FAILED: 500000

The result is the same as section 2, but RSS is only about 50–60 MB. readline automatically joins line fragments cut between chunks, so you don’t have to worry about it.

5. Writing Files and Backpressure

Backpressure occurs when the data source is faster than the write destination. Disk, network, or gzip can all be slower than read speed. The writable.write() function returns false when the internal buffer exceeds highWaterMark. That’s the signal “stop, wait for the drain event”.

The following example ignores this signal:

// bad-write.mjs — DON'T do this
import { createWriteStream } from 'node:fs';

const out = createWriteStream('big.txt');
for (let i = 0; i < 10_000_000; i++) {
  out.write(`line ${i}\n`); // ignore the return value
}
out.end();

The loop runs synchronously, so all 10 million strings are buffered in out before the disk can write them. RAM spikes into gigabytes. The fix is to check the return value of write() and await once(out, 'drain'), like in the generate-orders.mjs file in section 2.

If you chain streams together with pipeline, backpressure is handled automatically. This is the approach you should use in most cases.

6. Connecting Streams with pipeline

6.1. Copying and Compressing Files

// gzip-orders.mjs
import { createReadStream, createWriteStream } from 'node:fs';
import { createGzip } from 'node:zlib';
import { pipeline } from 'node:stream/promises';

await pipeline(
  createReadStream('orders.csv'),
  createGzip(),
  createWriteStream('orders.csv.gz'),
);
console.log('Done');

Done

pipeline from node:stream/promises returns a Promise. This Promise resolves when the last stream finishes writing. If any stream encounters an error, such as a missing file or full disk, the Promise rejects. When this happens, all streams are destroyed, so file descriptors don’t leak.

Note: Avoid using a.pipe(b).pipe(c). pipe doesn’t propagate errors across streams. If a errors, c stays open forever. Use pipeline instead.

6.2. Writing a Transform to Split Lines

Suppose you want to filter for FAILED orders and then compress. You need a line-splitting step that you can put in pipeline. Let’s write a Transform:

// split-lines.mjs
import { Transform } from 'node:stream';
import { StringDecoder } from 'node:string_decoder';

export function splitLines() {
  const decoder = new StringDecoder('utf8');
  let tail = '';

  return new Transform({
    readableObjectMode: true, // output is individual lines (strings), not Buffers

    transform(chunk, _encoding, callback) {
      const lines = (tail + decoder.write(chunk)).split('\n');
      tail = lines.pop(); // the last part might be an incomplete line
      for (const line of lines) this.push(line);
      callback();
    },

    flush(callback) {
      const rest = tail + decoder.end();
      if (rest) this.push(rest);
      callback();
    },
  });
}

Here’s the explanation of each part:

• readableObjectMode: true: The readable side emits each line as a separate object. The writable side still receives Buffer.
• StringDecoder: A chunk might cut through the middle of a multi-byte UTF-8 character, like the “ệ” in “Nguyễn Văn Hiệp”. Calling chunk.toString() directly produces garbage characters �. decoder.write() keeps incomplete bytes and joins them on the next call.
• tail: Stores the incomplete line fragment to join with the next chunk.
• flush: Runs when input ends. This step emits the final line if the file doesn’t end with \n.

6.3. Filtering with Async Generators

pipeline accepts an async generator as a step. This way you can filter without writing an extra Transform class:

// export-failed.mjs
import { createReadStream, createWriteStream } from 'node:fs';
import { createGzip } from 'node:zlib';
import { pipeline } from 'node:stream/promises';
import { splitLines } from './split-lines.mjs';

let count = 0;

await pipeline(
  createReadStream('orders.csv'),
  splitLines(),
  async function* (lines) {
    for await (const line of lines) {
      if (line.endsWith(',FAILED')) {
        count++;
        yield line + '\n';
      }
    }
  },
  createGzip(),
  createWriteStream('failed-orders.csv.gz'),
);

console.log(`Exported ${count} failed orders`);

Exported 500000 failed orders

Data flows through five steps. Each step holds only a few chunks in memory. When gzip is slow, the generator hangs at yield and splitLines stops receiving chunks. As a result, createReadStream pauses reading from disk. That’s backpressure working automatically.

6.4. Canceling Midway with AbortSignal

For long tasks, you should allow cancellation or set a timeout:

// export-with-timeout.mjs
import { createReadStream, createWriteStream } from 'node:fs';
import { createGzip } from 'node:zlib';
import { pipeline } from 'node:stream/promises';

try {
  await pipeline(
    createReadStream('orders.csv'),
    createGzip(),
    createWriteStream('orders.csv.gz'),
    { signal: AbortSignal.timeout(500) }, // cancel after 500 ms
  );
} catch (err) {
  console.error(err.name); // AbortError if timed out
}

AbortError

When canceled, all streams are closed. The file orders.csv.gz is only partially written at this point. In real code, delete that file in the catch block, or write to a temporary file and rename it when done.

7. Testing Transform

The logic that’s easiest to get wrong is joining line fragments cut between chunks. Let’s test using the built-in node:test runner. The fake data source is Readable.from with chunks intentionally cutting across lines and UTF-8 characters:

// split-lines.test.mjs
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { Readable } from 'node:stream';
import { splitLines } from './split-lines.mjs';

async function collect(stream) {
  const result = [];
  for await (const item of stream) result.push(item);
  return result;
}

test('join lines cut across chunks', async () => {
  const source = Readable.from([
    Buffer.from('1,an,10,PAID\n2,bi'),
    Buffer.from('nh,20,FAILED\n3,chi,5,PAID'), // last line has no \n
  ]);

  const lines = await collect(source.pipe(splitLines()));

  assert.deepEqual(lines, ['1,an,10,PAID', '2,binh,20,FAILED', '3,chi,5,PAID']);
});

test('preserve multi-byte UTF-8 characters', async () => {
  const bytes = Buffer.from('1,Nguyễn Văn Hiệp,10,PAID\n');
  const source = Readable.from([bytes.subarray(0, 9), bytes.subarray(9)]); // cut in middle of "ễ"

  const lines = await collect(source.pipe(splitLines()));

  assert.deepEqual(lines, ['1,Nguyễn Văn Hiệp,10,PAID']);
});

Run the tests:

node --test

✔ join lines cut across chunks (3.1ms)
✔ preserve multi-byte UTF-8 characters (0.6ms)
ℹ tests 2
ℹ pass 2
ℹ fail 0

In this test, using .pipe() is fine because Readable.from runs in memory with no I/O to clean up. If you drop StringDecoder and use chunk.toString(), the second test fails with garbage characters.

8. Common Mistakes and When Not to Use Streams

Using on('data') with an async handler. For example, stream.on('data', async chunk => await db.insert(chunk)). Event emitters don’t wait for Promises, so the stream keeps pushing data at maximum speed and thousands of insert statements run in parallel. Use for await or an async generator step in pipeline instead.

Forgetting error handling. An 'error' event with no listener crashes the entire process. With Promise-based pipeline, you only need one try/catch.

Assuming a chunk is a line or a record. Chunks are cut by byte. Always split lines with readline or a Transform with tail buffering like section 6.2.

Setting highWaterMark too large. Setting it to 100 MB “for faster reading” barely helps and defeats the memory advantage. 64 KiB to 1 MiB is usually enough.

Accumulating results in a large array. Code like for await (...) rows.push(row) then process later is back to the readFile problem. Process or write data as soon as you receive it.

Note: Streams aren’t always the best choice. For config files of a few KB, or when you need all data at once (sort everything, parse a single JSON file), readFile is simpler and faster. Use streams only when data is large, unknown in size, or comes from a network.

9. Conclusion

You’ve seen readFile consume many times the file size in RAM. Streams keep memory stable by processing chunks. Key takeaways:

• Use createReadStream with for await to read chunks, and readline to read lines.
• Respect backpressure: check the return value of write(), or let pipeline handle it.
• Use pipeline from node:stream/promises instead of .pipe() so errors propagate through streams and resources are cleaned up properly.
• Write Transform classes that handle incomplete lines and UTF-8 bytes, then test with chunks intentionally cut across them.

With these built-in tools, you can filter, transform, and compress multi-GB files while the process uses only tens of megabytes of RAM.