As of 2015, New Relic received half a trillion data points from customers daily. All of these need to be processed, stored and made available for charts, queries and alerts as soon as possible. Getting from humble roots to querying more than a billion events a second has meant radically reinventing and re-architecting the whole platform multiple times. Matthew Flaming and Nic Benders shared a journey designing scalable services, optimizing for predictability, creating software architecture through team structure, and what it takes to continuously rebuild a system that can never stop. youtube
YOUTUBE MlSRmY8elgc Matthew Flaming and Nic Benders, FutureStack15: Building a System that Never Stops- New Relic at Scale, Dec 21, 2015
We are drawn back to this talk somewhat often because it captures a truth about modern software engineering. Any company with software running in the cloud faces this fundamental challenge of continually updating a living system.
This is much more like a biology or ecosystem problem than what we widely accept as true for computer problems.