Flux Analytics

A privacy-first analytics engine ingesting millions of events per day with sub-second dashboards.

Live demo Source

12M

Events/day

780ms

p99 query

-64%

Cost cut

The problem

Teams wanted product insight without shipping user data to third parties or waiting minutes for queries to return.

The solution

A columnar ingestion pipeline with a streaming layer and pre-aggregated rollups that keep p99 dashboard latency under 800ms.

Architecture

Events flow through Kafka into a Node.js ingestion service that batches into ClickHouse. Materialized views maintain rollups; a gRPC query gateway fans out and caches hot windows.

Node.jsClickHouseReactgRPCKafka

Challenges

Backpressure handling during traffic spikes.
Schema evolution without downtime.
Balancing cardinality against query cost.

Lessons learned

Pre-aggregation is the cheapest performance win at scale.
Observability must be built before you need it.

Timeline

Prototype

Validated ClickHouse rollup model.

Pipeline

Kafka + Node.js ingestion hardening.

Product

Dashboard builder and funnels.

Future improvements

Anomaly detection

SQL notebooks

Managed cloud tier