← Back to ArticlesBackend & Database
Backend & Database6 min read

MongoDB Change Streams in NestJS Keep Dropping Events. Here's the Fix

RENOVATEAPI ARCHITECTURAL SPEC
CANONICAL GUIDE
Published 15 August 20266 min read

Change Streams look production-ready in a demo and fall apart on deploy day. Here's how to persist resume tokens, batch under load, and survive replica set elections in NestJS.

Abhishek

Abhishek

Full-Stack & AI Product Engineer

MongoDB Change Streams are one of those features that look finished the moment they work. You wire up .watch(), log a change event to the console, and it feels like real-time architecture is basically solved. Then you deploy on a Friday, a pod restarts mid-shift, and you discover that every order update that happened during those four seconds of downtime just vanished. No error, no retry, nothing in the logs. It's just gone.

That gap between "works in a demo" and "survives production" is almost entirely about three things: resume tokens, backpressure, and reconnection. Get those three right and Change Streams are genuinely one of the best tools available for event-driven sync in a NestJS + MongoDB stack — better than polling, and lighter than standing up a full CDC pipeline with Debezium if your scale doesn't need it yet.

Resume tokens are the whole game

Every Change Stream event ships with an opaque _id — the resume token. It's MongoDB's bookmark. Pass it back in on your next watch() call via resumeAfter, and the stream picks up exactly where it left off instead of only listening from "now."

The failure mode is almost embarrassingly simple: if your app crashes or redeploys without saving that token somewhere durable, the next watch() call has no memory of it. It starts listening from the current moment. Whatever wrote to your orders collection during the restart window is invisible to your app forever — no error thrown, no exception to catch, because from Mongo's point of view nothing went wrong. You just weren't listening.

The fix is to persist the resume token to Redis (or any store that survives a restart) right after you've successfully processed each event — not before, because if processing fails you want to retry that same event on next boot, not skip past it:

@Injectable()
export class OrderChangeStreamService implements OnModuleInit, OnModuleDestroy {
  private changeStream: ChangeStream;

  constructor(
    @InjectModel(Order.name) private orderModel: Model<OrderDocument>,
    private redisService: RedisService,
  ) {}

  async onModuleInit() {
    await this.startListening();
  }

  private async startListening() {
    const lastResumeToken = await this.redisService.get('mongo:resume:orders');

    const options: ChangeStreamOptions = {
      fullDocument: 'updateLookup',
      ...(lastResumeToken ? { resumeAfter: JSON.parse(lastResumeToken) } : {}),
    };

    this.changeStream = this.orderModel.watch(
      [{ $match: { operationType: { $in: ['insert', 'update', 'replace'] } } }],
      options,
    );

    this.changeStream.on('change', async (change: ChangeEvent<OrderDocument>) => {
      try {
        await this.handleOrderEvent(change);
        await this.redisService.set('mongo:resume:orders', JSON.stringify(change._id));
      } catch (err) {
        console.error('Failed to process change event:', err);
      }
    });

    this.changeStream.on('error', async (error) => {
      console.warn('Change stream disconnected. Reconnecting with exponential backoff...', error);
      this.reconnect();
    });
  }

  onModuleDestroy() {
    this.changeStream?.close();
  }
}

One detail worth calling out: fullDocument: 'updateLookup' isn't optional if your downstream consumers need the full document on updates. Without it, an update event only gives you the changed fields, and you'd end up querying the database manually to reconstruct state — which defeats half the point of using a Change Stream in the first place.

At real throughput, one-event-at-a-time will hurt you

A Change Stream listener that processes events synchronously, one by one, works fine at low volume and falls over the moment you hit real traffic. A few thousand writes a second is enough to saturate the Node.js event loop and start generating unhandled promise rejections — not because the logic is wrong, but because you're doing too much synchronous work per tick.

The fix is boring but effective: buffer events in memory and flush in batches, either when the buffer hits a size threshold or a timeout fires, whichever comes first.

private eventBuffer: ChangeEvent[] = [];
private flushInterval = setInterval(() => this.flushBuffer(), 50);

private async handleIncomingEvent(change: ChangeEvent) {
  this.eventBuffer.push(change);
  if (this.eventBuffer.length >= 100) {
    await this.flushBuffer();
  }
}

private async flushBuffer() {
  if (this.eventBuffer.length === 0) return;

  const batch = [...this.eventBuffer];
  this.eventBuffer = [];

  await this.downstreamQueue.addBulk(batch.map(event => ({ name: 'sync-event', data: event })));
  await this.redisService.set('mongo:resume:orders', JSON.stringify(batch[batch.length - 1]._id));
}

100 events or 50ms, whichever comes first, is a reasonable starting point for most order/inventory-style workloads — tune it against your own p99 write rate rather than treating it as a fixed constant. And notice the resume token save moved here too: you commit the token for the last event in the batch only after the whole batch has been successfully queued downstream, not per-event. That's the same "save after success" rule from the resume-token section, just applied at batch granularity.

Replica set elections will disconnect you — plan for it, don't just log it

MongoDB replica sets fail over. A primary steps down, an election happens, and for a window of a few seconds your Change Stream listener gets disconnected with something like MongoNetworkError or, worse, MongoError: resume token not found. This isn't a bug. It's a normal part of running a replica set, and a listener that just logs the error and dies is not production-ready.

Two things need to happen:

  1. Exponential backoff on reconnect. Don't hammer the primary the instant it errors — wait Math.min(1000 * Math.pow(2, attempt), 30000) milliseconds before the next attempt, capping around 30 seconds.
  2. Handle expired resume tokens explicitly. If the oplog has rolled over past your stored token — which happens if your app was down longer than the oplog retention window — MongoDB throws InvalidResumeToken. At that point resumeAfter can't help you. Log an alert, fall back to startAtOperationTime, and treat it as a signal that you may need a full reconciliation sync to backfill whatever was missed.

That second case is the one teams tend to skip, because it only shows up when things have already gone wrong for a while — a long deploy freeze, an incident, a stuck pod. It's exactly the scenario where silently losing data hurts the most, so it's worth writing the fallback path before you need it, not after.

The checklist, if you're auditing an existing setup

Operational requirement Anti-pattern Resilient solution
Downtime continuity Reconnect with a fresh cursor (drops events) Store the resumeAfter token in Redis after each event is processed
Event loop safety Unbounded, synchronous per-event execution In-memory batching with a timed flush
Failover handling Unhandled stream crash on election Exponential backoff reconnect listener
Data lookup overhead Manually querying the DB for the full document fullDocument: 'updateLookup' in stream options

None of these are individually hard. What makes Change Streams painful in production is that all three failure modes are silent — no stack trace tells you a resume token expired three days ago, or that you've been dropping every fifth event under load. You find out from a support ticket about a missing order, not from your monitoring. Build the resume-token persistence, batching, and reconnect logic in from day one, and Change Streams stop being a liability and start being one of the more reliable pieces of an event-driven NestJS stack.

Tags:#MongoDB#NestJS#Change Streams#Event-Driven Architecture#Redis#Microservices

Related Articles

View All Articles ↗