How OpenAI Kept Azure Cosmos DB Standing Under the Weight of 1 Billion ChatGPT Users

Habitat: OpenAI's Internal Data Backbone

Habitat is OpenAI's internal storage platform, handling requests from ChatGPT, the API, Codex, and dozens of internal services. It needs to respond in milliseconds while serving more than a billion users worldwide, and it stores over 500 petabytes of data. The engine under the hood is Azure Cosmos DB. OpenAI engineers John Lee, Grach Hakobyan, and Chaoming Yu described how they rebuilt the way load gets distributed across products inside this database — three times — before landing on a scheme that actually worked.

The Shared Database Problem and "Noisy Neighbors"

Originally, every product wrote to a single shared Cosmos DB database (container). Each key — a user ID, for example — has a limit of 10,000 operations per second, but in practice that limit applies not to the key itself, but to the physical disk partition where the key happens to be stored alongside other keys. That created two problems:

• Different products using semantically similar keys (like user_id) ended up sharing the same throughput limit, even though they had no logical reason to overlap.
• If one client suddenly spiked load on "its" partition, every other user and feature whose data happened to sit nearby suffered — sometimes to the point of login failures across the board.

A secondary issue surfaced too: "weak partitions." When a disk partition grew large enough that Cosmos DB split it in two, the throughput of each half was cut in proportion — meaning data growth alone could slow things down, even if request volume stayed flat.

Isolation Through Separate Databases — and Its Cost

The fix was to spin up a separate database for every major new product or feature. That genuinely eliminated the noisy-neighbor effect — one service overloading no longer dragged down the rest. But the approach had a downside:

• Product teams had to request the right throughput ahead of launch themselves — and often forgot, turning launches into last-minute emergency scaling meetings.
• Load that would have been a rounding error in a shared database — going from 1,000 to a million requests per second is just 5% of total shared capacity — could blow past a dedicated database's provisioned resources by a thousandfold.
• Capacity planning became scattered across the entire company instead of living with one central team, and the quality of that planning varied wildly from team to team.

Back to a Shared Database — This Time With Smart Balancing

Realizing that neither approach scaled organizationally, the Habitat team worked with Microsoft engineers to rework the mechanics of Cosmos DB itself:

• They added throttling that targets the "hottest" keys first, so one aggressive client can't claim an entire partition's throughput.
• They introduced request prioritization (QoS): background jobs like bulk data exports now get throttled first, while interactive user requests are the last to be cut.
• Together with Azure, they improved Cosmos DB's serverless mode, raising the per-partition limit from 5,000 to 10,000 operations per second and adding automatic splitting based on load rather than just data volume.

This let the team return to a single shared database without manual capacity planning — and according to the authors, there hasn't been a single outage caused by overload or noisy neighbors in the months since the switch.

Joint Engineering With Microsoft

OpenAI's largest Cosmos DB account stores tens of petabytes and handles more than 20 million requests per second — the authors call it the largest deployment of the service they're aware of. At that scale, standard cloud-provider support stops being enough: a power outage in one region in late 2024 showed that failing over to a backup region for an account that size takes several hours — a failure of that magnitude had never been tested before. A month later, the reverse cutover caused a second incident: clients rushed to reconnect to the original region en masse and overwhelmed its gateway, even though only part of the data had actually come back online.

After these incidents, OpenAI and Azure built out direct collaboration: shared dashboards, joint testing against a copy of the production account, direct Teams channels, and the ability to page each other directly, bypassing standard support. Over the past year, the teams have jointly improved the scalability of the Cosmos DB gateway, added metadata caching, reduced the number of connections from Habitat's Python services, and separated critical and background workloads onto different networks.

The main takeaway for engineering teams: multi-tenant (shared) systems are cheaper and simpler to operate, but they require built-in isolation mechanisms from the provider itself — without them, the only alternative is manually splitting into separate databases, which just shifts the capacity-planning burden onto dozens of teams.