OpenAI Rewrites Habitat Storage: 500 PB of Data and 1 Billion Users on Cosmos DB

Habitat, the platform behind ChatGPT, the API, Codex, and OpenAI's internal services, serves more than 1 billion users across 40 geographic regions and stores over 500 petabytes of data. Responses are needed in milliseconds. In the second part of their engineering write-up, Jon Lee, Hrachya Hakobyan, and Chaomin Yu describe how the team squeezed the most out of Azure Cosmos DB, went through three multitenancy models, and ultimately returned to a single shared container.

Phase I: a shared container and three systemic problems

At first, OpenAI admitted any client workload and relied on the documented Cosmos DB limit of 10K RU/s per logical partition key as natural back pressure. Each account had a single container, the default. This worked until scale exposed three problems.

First, because of early API decisions, clients were given the choice of partition key. Popular keys such as user_id or workspace_id overlapped across different product use cases, and those use cases shared the limit of a single key. Second, limits are in fact enforced at the physical partition level, so a logical key never reaches 10 000 RU/s, and neighbors on the same partition interfere with each other. Side features affected critical login and sign-up flows. Third, a container's throughput is divided equally among its physical partitions. As data grows and partitions split, total provisioning stays the same while the per-partition limit drops. For heavy blob-like workloads, storage growth alone reduced available throughput even though traffic was stable.

Phase II: isolation that didn't scale organizationally

Separate containers solved some of these problems. Each one has its own physical partitions, so only workloads inside it can be noisy neighbors. For critical new use cases, containers were allocated, and a script synchronized provisioning with the number of physical partitions and repaired weak partitions. This prevented site-wide outages: failures were limited to a degraded feature or service.

But the overhead kept growing. The main reason: in a shared container, a new use case going from 0 to 1M QPS barely moved the aggregate. In a dedicated container at 10K RU/s, the same jump from 1K RU/s during load testing to 1M RU/s at launch meant 10 000% of headroom, whereas in a shared container at 100M RU/s with 80% utilization it amounted to about 5%. Teams shipped features and forgot to scale ahead of time, and on launch day engineers ended up in SEV rooms. Capacity planning required product knowledge from every team, and the quality of that work varied widely.

Phase III: back to a shared container, together with Azure

OpenAI returned to multitenancy by default and, together with the Azure team, fixed the original shortcomings:

• Hot key throttling. The hottest keys are throttled first, which keeps a single key from consuming all of a physical partition's throughput.
• Quality of Service. Requests are tiered: offline workloads such as bulk inserts and exports get low priority and are throttled first, while interactive user traffic is left untouched.
• New serverless mode. The previous mode scaled only by storage and capped each physical partition at 5K RU/s. The new one lets each partition grow to 10K RU/s and adds splits based on RU load in addition to splits based on size. Manual monitoring and provisioning increases are no longer needed.

The result: one shared container, much simpler capacity planning for product teams, and workload isolation that remains intact. In the months since the switch, there have been no provisioning or noisy-neighbor incidents, even as product development has sped up.

What you can take away

OpenAI's story is worth reading as a checklist for any team building a shared data platform.

• Don't let clients choose the partition key. That early API decision later had to be compensated for with an entire infrastructure.
• Watch the effective limit per physical partition, not per logical key. It drops with splits even when traffic doesn't change.
• Container isolation is expensive organizationally: every team becomes a capacity planner, and the stable aggregate growth is gone.
• Prioritize traffic. Bulk inserts and exports should be throttled before user requests.
• At large scale, push the vendor for improvements. OpenAI got a dedicated serverless mode from Azure rather than working around the limits on its own side.

OpenAI itself says it is still early in its multitenancy journey. But for now the payoff is clear: fewer SEV rooms at launches and less manual work for product teams.