Revolutionizing Data Ingestion: Meta's Large-Scale Migration to a New Architecture

Introduction

Meta’s social graph, one of the largest MySQL deployments in the world, relies on a robust data ingestion system to provide up-to-date snapshots for analytics, reporting, and downstream products. Every day, this system incrementally scrapes petabytes of data into the warehouse, powering everything from daily decisions to machine learning model training. Recently, Meta performed a major revamp of this ingestion architecture to enhance efficiency and reliability at scale. The transition from a legacy system to a new self-managed service involved migrating thousands of jobs and fully deprecating the old pipelines. This article shares the solutions, strategies, and key architectural decisions that made this large-scale migration successful.

Revolutionizing Data Ingestion: Meta's Large-Scale Migration to a New Architecture — Source: engineering.fb.com

The Legacy System and Its Limitations

The original data ingestion system consisted of customer-owned pipelines. While these worked well for smaller operations, they began to show instability as Meta’s scale grew. The primary issue was meeting increasingly strict data landing time requirements under hyperscale loads. As the volume of social graph data expanded, the legacy architecture struggled with resource utilization and latency, leading to frequent quality concerns. It became clear that a new approach was needed—one that could handle petabytes efficiently without compromising reliability.

The New Architecture: A Self-Managed Data Warehouse Service

The revamped architecture replaces customer-owned pipelines with a simpler, self-managed data warehouse service. This shift moves away from fragmented ownership to a centralized system that operates efficiently at hyperscale. The new design reduces complexity, improves maintainability, and enables automatic scaling. By abstracting away pipeline management, Meta’s engineering teams can focus on data consumption rather than infrastructure. The self-managed service also provides consistent performance, making it easier to enforce data quality and landing time guarantees.

The Migration Challenge

Migrating a system of this magnitude posed two main challenges: ensuring each individual job transitioned seamlessly, and executing the migration at scale across thousands of jobs. Without careful planning, the process could introduce data quality issues, latency regressions, or resource conflicts. Meta needed a robust framework to track progress, verify correctness, and handle rollbacks. The solution was a formal migration lifecycle with clear success criteria at every stage.

The Migration Lifecycle

The migration lifecycle was designed to ensure data integrity and operational reliability. Each job had to pass a series of verification steps before progressing to the next phase. The key criteria included:

No data quality issues: The new system must deliver identical data as the old one. Meta verified this by comparing both row counts and checksums of the output, ensuring complete consistency.
No landing latency regression: The new system must match or improve the data landing time compared to the legacy system. Performance benchmarks were run to confirm no degradation.
No resource utilization regression: The new architecture should not consume more compute or storage resources than necessary. Metrics were monitored to ensure efficient scaling.

Only after meeting all three criteria was a job considered fully migrated. This stepwise approach minimized risk and allowed teams to catch issues early.

Robust Rollout and Rollback Controls

Another critical component was the implementation of rollout and rollback controls. For each job, the migration could be toggled between the old and new system in real time. If any anomaly was detected—such as a sudden spike in latency or a data mismatch—the system automatically reverted the job to the legacy pipeline. This safety net ensured that production workloads remained unaffected during the transition. Additionally, dashboards provided visibility into the migration progress, allowing engineers to monitor thousands of jobs concurrently.

Conclusion

Meta successfully migrated 100% of its data ingestion workload to the new self-managed warehouse service and fully deprecated the legacy system. The key to success was a structured lifecycle with strict verification requirements and powerful rollback mechanisms. By moving away from customer-owned pipelines, Meta achieved a simpler, more reliable architecture that can scale with the social graph’s continued growth. This migration serves as a blueprint for organizations tackling large-scale system transitions, proving that careful planning and automated safeguards can turn a daunting challenge into a seamless upgrade.