Role

Solo Developer

Data Analyst

Role

Solo Developer

Data Analyst

Tools

Python

Kafka

Spark

MongoDB

Tools

Python

Kafka

Spark

MongoDB

Reflex

Real-Time Competitive Demand-Shock Detection & Response Platform

A distributed data pipeline that detects unplanned e-commerce demand shocks in real time — validated against two real-world events I didn't know about in advance.

Real-Time Competitive Demand-Shock Detection & Response Platform

A distributed data pipeline that detects unplanned e-commerce demand shocks in real time — validated against two real-world events I didn't know about in advance.

The Problem

E-commerce companies plan well for predictable demand spikes like Black Friday, but struggle with unplanned shifts — a competitor's flash sale, a surprise price drop, a product suddenly trending. By the time internal teams notice, the critical early window to respond has already passed. I've seen this firsthand working in e-commerce, and this project is my attempt to actually solve it.

What it Does

  • Trend — Builds a rolling baseline of expected demand per category

  • Shock Detection — Flags statistically significant deviations as they happen

  • Propagation — Tests whether spillover demand follows to related categories

  • Response — Combines it all into one ranked, actionable output

Key Findings

  • 8 demand shocks detected across 61 days of real transaction data

  • Real-world validation: the largest anomaly aligns exactly with the Huawei Mate X foldable smartphone launch (Nov 15, 2019); a second shock captures Black Friday

  • Consistent leading indicator: every shock was driven by smartphones, concentrated among Samsung, Apple, Xiaomi, and Huawei

  • Honest, nuanced result: spillover to related categories is real but uneven — headphones showed a clean signal, cameras showed none

Under the Hood

Built on a genuinely distributed architecture: Kafka for real-time ingestion, Spark Streaming for continuous aggregation, Chroma for vector similarity search, MongoDB Atlas for cloud storage, and NewsAPI for a live external signal — plus a second, fully isolated pipeline demonstrating continuous, non-terminating production-style operation.