Backend & APIs · PulseMetrics
A realtime pipeline crunching 2 billion events a day
PulseMetrics ran customer analytics on overnight batch jobs that broke weekly. We rebuilt it as a streaming pipeline on Kafka and ClickHouse: 2B events a day, dashboards fresh within seconds.
Client
PulseMetrics
Industry
SaaS analytics
Timeline
10 weeks
Year
2026
0B
events processed daily
0s
from event to dashboard
0%
lower infrastructure cost
01 · The challenge
PulseMetrics sells analytics dashboards, but the pipeline behind them was a chain of overnight batch jobs. Customers looked at yesterday's data, and when a job failed at 3am, they looked at the day before yesterday's.
Event volume had grown to two billion a day and the nightly window physically ran out of night. Each retry pushed morning dashboards later, and the biggest customers were asking for realtime numbers in contract renewals.
02 · The solution
We rebuilt ingestion as a streaming pipeline: events flow through Kafka, get validated and enriched in stream processors, and land in ClickHouse where dashboard queries answer in milliseconds.
Every stage is horizontally scalable and observable. Bad events go to a dead-letter queue instead of killing the run, backfills replay from the log without downtime, and the old nightly window simply stopped existing.
03 · How we did it
The approach
Trace the data first
Two weeks mapping every producer, schema, and consumer, finding the undocumented feeds everyone forgot about.
Run both in parallel
The stream pipeline shadowed the batch jobs for three weeks, with results diffed nightly until they matched.
Design for bad data
Schema validation at the door, dead-letter queues, and replay tooling, because one malformed event should never cost a night.
Cut over by customer
Dashboards switched to the new pipeline cohort by cohort, largest accounts last, with a one-click path back.
We stopped apologizing for stale dashboards. Two billion events a day, live within seconds, and the infra bill went down instead of up.
Priya Raman
VP Engineering, PulseMetrics
The stack
Want results like this?
Tell us where your numbers are today and we will show you what they could look like.