Meta’s Global Delivery Infrastructure & how it works!

Meta’s (Facebook’s) platform relies on a highly customized infrastructure of servers, networks, and software. Their data centers deploy tailored hardware and networks, combined with extensive caching and distribution to deliver content worldwide. Below we examine each layer in detail, citing Meta’s own disclosures and research.

1. Server Hardware

Meta uses rack-scale custom servers built to their Open Compute designs. For AI, they build massive GPU clusters: e.g. two 24,576-GPU clusters (NVIDIA) announced in 2023 , and plan to deploy ~350,000 NVIDIA H100 GPUs by end of 2024 . They also run an earlier “RSC” cluster with ~16,000 A100 GPUs. General-purpose servers are standardized: each server typically has one high-end CPU (Intel Xeon) and large RAM (originally 64 GB, now 256 GB) , following a one-CPU-per-node policy to simplify deployment. Meta is even co-designing custom CPUs with ARM for future workloads . In addition to GPUs and CPUs, Meta builds its own AI accelerators and NICs. The Meta Training & Inference Accelerator (MTIA) chip (an in-house ASIC) is deployed for deep-learning inference (doubling compute and memory bandwidth over prior designs) . MTIA v1 handles ranking/recommendation models, and v2 extends to more complex AI tasks. In the latest AI clusters, MTIA chips function as smart NICs supporting RoCE (RDMA) alongside GPUs .

• GPU Accelerators: Large AI clusters use NVIDIA GPUs (Tesla A100 and H100). 2×24,576-GPU clusters were built (2022–23) . Meta expects ~350,000 H100 GPUs by end-2024 for AI training .

• Standard Servers: Non-AI servers are single-socket x86 (Intel) with large DRAM (now ~256 GB) . Meta standardizes on one server type (mono-socket) per rack to maximize efficiency. These OCP-based servers (e.g. “Zion” and successors) are designed and open-sourced via the Open Compute Project .

• Custom ASICs: Meta’s custom MTIA chips are added to racks for ML workloads . They are designed in-house (as part of OCP) and plugged into OCP-standard chassis (35 W per module ). Meta’s future roadmap includes more custom silicon (for inference and generative AI) to improve power efficiency .

• Networking Hardware: At the server rack level, Meta uses OCP white-box switches (e.g. the Wedge and Minipack families). For example, their 51.2 Tbps “Minipack3” switch (Broadcom Tomahawk5 ASIC, Celestica-built) provides 64 × 400Gb/800Gb ports for high-bandwidth fabrics . Meta runs its FBOSS network OS on these switches (as well as Cisco Silicon One gear) .

2. Networking Infrastructure

Meta operates a vast private network interconnecting its data centers and edge sites. They have built dedicated long-haul fiber routes (terrestrial and submarine) and use custom routing/traffic-engineering. Their global backbone (Backbone Engineering or BBE) spans tens of thousands of fiber-optic kilometers, linking dozens of data-center regions. They often deploy 100% outside fiber (dark fiber) and have been a partner in >20 subsea cables. Recent projects include “Project Waterworth,” a planned 50,000 km, five-continent cable (24 fiber pairs) connecting the US, India, Brazil, South Africa, and East Asia . They also helped build the 2Africa cable (3 billion population reach) and US–Japan Pacific cables . Meta extends regional networks as well: for example, the Malbec cable across South America (Argentina–Brazil) has been extended to Porto Alegre, adding a local PoP to speed Brazilian traffic .

Internally, Meta uses custom network fabrics. Within and between data centers, they run an “Express Backbone”: a multi-plane, MPLS/segment-routing network that isolates internal DC-DC traffic from user-facing egress . The control plane evolved to use Meta’s Open/R routing software for distributed IGP (and a central TE controller) . In data centers, switches are arranged in Clos/fat-tree fabrics with minimal oversubscription . Meta’s switches all use the FBOSS operating system (open-sourced) on OCP hardware, and they have moved from decentralized protocols (BGP/RSVP-TE) to centralized controllers for congestion-aware routing . At the network edge, hundreds of Points-of-Presence (PoPs) run on multiple ISPs; Meta advertises BGP routes preferring paths by measured capacity and latency . Traffic engineering tools constantly adjust PoP-to-DC mappings to optimize latency and capacity . In short, Meta has built its own optical backbone and routing stack (FBOSS/Open/R) rather than relying solely on the public Internet .

Meta’s Minipack3 is an example of their open-network hardware: a Celestica-built, Broadcom Tomahawk5-based 800 Gbps switch providing 51.2 Tbps of fabric bandwidth . Meta deploys Minipack3 (and similar 51.2T Cisco switches) with FBOSS to create non-blocking 400G/800G datacenter fabrics. This open design achieves high throughput with reduced power per bit (no retimer chips needed) .

3. Backend Software Stack

Meta’s site backend is built on an internally standardized software stack. The web servers run PHP code (converted to Hack) on HHVM (JIT compiler) for the Facebook front-end. Product engineers primarily write in Hack/PHP, C++, and Python . For data APIs, Meta developed GraphQL: a query language that powers most app data fetching (mobile apps and web). GraphQL handles hundreds of billions of requests per day and was open-sourced in 2015 . In practice, a user request first hits a pool of stateless front-end services (written in Hack/C++) which call various backend services via Thrift or RPC.

Key software components include:

• Service Deployment: Nearly all services run in containers managed by Meta’s in-house orchestrator Twine . Twine bundles code/images and schedules containers across millions of servers globally, providing features like cross-DC failover and real-time autoscaling. (Legacy systems also use Tupperware containers with Chef for orchestration .)

• Databases: The main user data is stored in a massively sharded MySQL fleet. The entire user profile dataset is split across thousands of MySQL instances and data centers for scale . Meta automates MySQL administration using tools like MPS (MySQL Pool Scanner) to handle replication and failover . On top of MySQL (the “persistent store”), Meta uses a distributed TAO cache: a custom graph storage layer that implements the social-graph abstraction with an integrated cache . This combination (Memcached + MySQL + TAO) handles the high read/write rates of the social graph. For analytics and batch workloads, Meta runs Hadoop/HDFS clusters (hundreds of PB) and uses Presto (open-source SQL engine built at Meta) for interactive queries .

• Caching: To reduce load on databases and storage, Meta employs multilevel caching. A global Memcached layer fronts the MySQL/TAO backend (inherited from Facebook’s early design ). For media (photos/videos), Meta uses a hierarchical CDN: browser cache → edge caches at ISP PoPs → a global “origin cache” cluster (photo servers like Haystack) → durable blob stores . Research has shown this stack serves over 90% of photo requests from cache layers . Likewise, video and other static content use similar CDN tiers.

• Configuration and Tools: Meta uses a monorepo with continuous deployment. Host-level config is managed via Chef, and service-level configs use a Python-based “Configerator” that outputs JSON objects . They have built common services (e.g. an internal TFTP for boot images, binary distribution system) and job pipelines in Python to automate operations . Real-time debugging is aided by systems like Scuba (in-memory analytics), Dapper (distributed tracing), and a wide network of metrics collectors.

4. Data Center Design

Meta’s data centers are custom-engineered for efficiency, scalability, and sustainability. All operational sites are LEED Gold or higher, and in 2023 achieved an average PUE of ~1.09 . They purchase or match 100% of their electricity with renewable energy . The data halls use cold/hot-aisle containment and a two-tier “penthouse” cooling system with 100% outside-air economization . In this design, air enters through upper louvers, mixes as needed, is cooled via evaporative (mist) chambers, then blown through the raised-floor cold-aisle supply plenum. Server exhaust goes into contained hot aisles and is expelled via roof relief fans . Because of this fresh-air approach (no traditional chillers), each center uses minimal additional cooling: Meta has applied AI-based controls (reinforcement learning) to adjust fan speeds and dampers, reducing cooling energy by ~20% in pilot sites . Water usage is also optimized: average WUE is ~0.20, reflecting dripless humidification.

Meta data centers use outside air economization. The two-tier penthouse layout shown above (with mixed-air and evaporative misting) keeps server inlet temps at 18–30 °C using only outside air and controlled humidity . Such advanced HVAC gives Meta some of the world’s most energy- and water-efficient data halls .

Other design highlights: the facilities use modular Open Rack v3 architecture (developed in OCP) with high-voltage 48V distribution. Each rack (or “pod”) can draw up to ~15 kW, enabled by the 48V system . For the highest-density racks (e.g. GPU pods), Meta has experimented with Air-Assisted Liquid Cooling (AALC) – a hybrid air/liquid cooling connector for server trays – to further boost power density without overheating. Overall, Meta continuously measures and optimizes power profiles, reusing idle capacity and pushing workload-aware power allocation to minimize waste .

5. Global Distribution (CDN and Edge)

To deliver content with low latency worldwide, Meta operates a multi-layer CDN and edge network. When a user makes a request (e.g. for an image or News Feed), DNS selects a nearby Meta Point-of-Presence (PoP) or CDN node. Meta runs thousands of small CDN sites co-located with Internet providers to cache photos, videos, and static files . (For example, a photo URL is rewritten to cdnNN.meta.com; if that edge cache misses, it fetches from the nearest origin.) Edge PoPs typically have tens of servers (some up to hundreds) and serve dynamic requests as well.

• Static Content (CDN): User requests for static assets go to the nearest Meta CDN site. Each CDN site has limited storage, so on a miss it forwards the request to a PoP or regional cache. Behind the scenes, Meta runs a global origin cache layer spanning multiple data centers (e.g. Prineville, Luleå, Forest City) . Only about 10% of photo requests ever reach the origin store, thanks to high cache hit rates at the browser, edge, and origin layers . All retrieved media is stored back up the chain (origin caches and edge caches save fetched items) to speed future requests . Meta’s CDN operates alongside third-party CDNs, but it carries the majority of Facebook media traffic with finely tuned caching policies.

• Dynamic Content (PoPs): For API/dynamic queries (News Feed, messages, etc.), DNS maps the user to a local PoP. The PoP runs Meta’s edge routing (FBOSS on white-box switches) and applies traffic engineering: a central controller computes the best data-center region (based on latency, load, capacity) . The request is then forwarded over Meta’s private backbone to that data center region. Crucially, Meta’s internal WAN (tens of thousands of fiber km, including submarine cables) carries all PoP↔DC traffic . This private WAN provides much higher bandwidth and predictability than public Internet links. Within regions, PoPs and DCs use multiple redundant paths; Meta’s PoPs even advertise their preferred BGP routes based on real-time network metrics . The result is that nearly all content (dynamic or static) is served from very close sources (edge caches or local data centers), minimizing user latency and offloading backbone.

Finally, Meta continually expands its edge footprint. For instance, the Malbec subsea cable to South America was extended to Porto Alegre, creating a new Brazilian PoP to serve Latin American users locally . Similar investments in local caches and network nodes ensure fast global delivery.

6. Security and Proprietary Systems

Meta’s infrastructure includes many custom security and monitoring tools (many of them internal to Meta’s SRE teams). Key aspects include:

• Cryptography Monitoring: Meta developed an internal crypto library called FBCrypto used by most core services . They built a large-scale logging/monitoring system for cryptographic operations. Instead of logging every event (which would be enormous), FBCrypto aggregates counts of crypto calls and periodically flushes summaries to Meta’s logging pipeline (Scribe → Scuba/Hive) . This gives engineers visibility into algorithm/key usage. For example, they track the use of each cipher and key, enabling automated detection of weak keys or algorithms and timely rotation. Monitoring FBCrypto usage has helped ensure compliance (e.g. phasing out RSA1024, planning post-quantum transitions) . In short, Meta has built advanced, proprietary crypto telemetry to maintain high security and future-proof cryptographic integrity.

• Automated Remediation: Meta emphasizes uptime and self-healing. Many routine failure fixes are automated. For example, a Python-based FBAR (Facebook Automatic Rebooting) system auto-detects failed hardware or services and initiates fixes (reboot, failover, etc.) without human intervention . Each server runs watchdog agents (e.g. MachineChecker) that report faults. Their production engineering teams have built tooling (often in Python) for everything from BIOS imaging to RAID rebuilds. These tools are custom and not publicly disclosed. Continuous integration, canarying, and blue-green rollouts are standard so that regressions are caught quickly.

• Network and Service Security: Meta’s custom software (FBOSS, Open/R, ServiceRouter, etc.) is developed internally with strict security reviews. All inter-service communication is encrypted (via TLS) and authenticated, and storage subsystems are encrypted at rest using FBCrypto-managed keys. Internal APIs are gated behind service-auth proxies. (While details are private, Meta does share that user data is strongly protected: e.g. end-to-end encryption for Messenger and rigorous internal access controls.) For anomaly detection, Meta uses its monitoring stack (Scuba for metrics analytics, Dapper for tracing) to flag unusual patterns in real time. If a sudden traffic spike or error rate occurs, alerts fire and SREs can drill down. In sum, Meta employs a suite of proprietary tools to protect data and maintain performance, but does not publicly disclose all details. The above points illustrate the high-level strategy (comprehensive crypto logging, auto-remediation, encrypted services) backed by internal systems .

7. Open Source and Innovation

Meta drives and contributes heavily to open infrastructure. As a founder of the Open Compute Project (OCP), Meta has open-sourced its data-center designs from servers to racks to power systems . For example, the Grand Teton GPU server (a successor to the older Zion design) was released via OCP; it doubles network and compute capability in one chassis . They also released the Open Rack v3 spec (48 V distribution for >15 kW racks) . At OCP summits Meta has shown new open hardware: in 2024 they unveiled two disaggregated Ethernet fabrics (DSF) for AI clusters and a new NIC design for MTIA . All of these allow other organizations to leverage Meta’s hardware innovations.

On software, Meta is equally open. They have open-sourced many key projects used in their stack:

• AI/Analytics: PyTorch and the Llama family of models (deep-learning libraries) . Presto (SQL query engine for petabyte data) .

• Data Stores: RocksDB (embedded key-value store) and Cassandra (distributed DB) . Relay (GraphQL client) and GraphQL itself (query language) .

• Infrastructure Tools: FBOSS (switch OS) and Open/R (routing software) are on GitHub. Many internal tools (Tupperware/Twine, Configerator, etc.) have been open-sourced or documented in research publications.

• Energy/Design Collaboration: Through OCP and initiatives like the iMasons Climate Accord, Meta works on shared standards (e.g. disclosure of embodied carbon) .

Meta’s strategy is “open by design”: they publish research papers on their systems and frequently contribute code and designs. As noted by a recent ACM survey, nearly all Meta products now converge on a common infrastructure (e.g. the same K/V store ZippyDB, shared deployment tools) and many of those components are open source . In summary, Meta not only uses cutting-edge technology in-house but also shares much of it with the industry via OCP and open-source software .

Sources: Meta’s engineering blogs and reports (Meta’s official disclosures and publications).