Experimentation as Infrastructure: Designing a Centralized A/B Testing Platform for 500 Million Users

Abstract

Building valid experimentation infrastructure at internet scale requires solving engineering problems whose solutions cannot be derived from statistical methodology alone. This article documents the architecture and design principles of a centralized Experimentation-as-a-Service (EaaS) platform supporting 500 million users, 1,000+ concurrent experiments, and sub-second assignment latencies across Yahoo, AOL, and affiliated properties following their 2015 acquisition. The platform addresses three interconnected engineering challenges: deterministic reproducible traffic assignment through a multi-layer orthogonal hash-based bucketing architecture; statistical validity assurance via the discovery and remediation of a systematic non-uniformity bias in the Fowler-Noll-Vo (FNV) hash function and the development of a pre-validated bucket system eliminating the traditional 4–5 day per-experiment A/A gating bottleneck; and dual batch and real-time event processing pipelines sustaining petabyte-scale data volumes required for high-power experiment analysis. Continuous platform health monitoring through identifier-level discrepancy detection reduced systematic bucket inconsistency from approximately 6% to below 1%, a quality improvement invisible to per-experiment validation. The pre-validated bucket system and its monitoring architecture were subsequently recognized in U.S. Patent Application Publication No. US 2019/0057108 A1 filed by Yahoo Holdings, Inc., independently confirming the engineering novelty of the contributions described here.