Shared vs. Dedicated Infrastructure for Embedding and Reranking: How to Choose Based on Workload

Request Profile: Shape Matters More Than Volume

Cohere conducted an analysis of the choice between shared (pay-as-you-go) and dedicated (reserved infrastructure) deployments for embedding and reranking. The key finding: people typically compare only price per token and requests per minute, but these metrics say almost nothing about the right choice. The logic is simple — Request Profile → Traffic Pattern → Utilization → Deployment Choice.

The same embedding model can perform completely different jobs. On an e-commerce site, embedding is used in two ways: indexing a catalog (hundreds of product descriptions at once, roughly 150 words or about 200 tokens per text) and processing a search query (a single short query while the user watches a loading spinner). Batch processing optimizes for throughput and cost; a search query optimizes for responsiveness. These are opposite requirements, and one piece of infrastructure can't efficiently serve both.

Reranking adds a critical parameter: the number of candidates being reranked. Vector search returns 50 products, and the reranker re-scores each one against the query. The model compares the query against every candidate independently, so the workload scales strictly with the number of candidates. A typical profile: a 20-token query plus 50 documents at 200 tokens each equals roughly 11,000 tokens per operation. Go from 20 to 100 candidates, and the user notices nothing — but the workload grows from 4,400 to 22,000 tokens. That's a decision about search depth, not scaling to more users.

Traffic Pattern and Latency: What Determines Real Capacity

Two applications with the same peak traffic can have wildly different infrastructure costs. Steady traffic (global search, a constant stream throughout the day) makes dedicated capacity more economical: the infrastructure is well-utilized, with no idle time and no unhandled bursts. Bursty traffic (financial reports, batch jobs, spikes in specific windows) favors shared infrastructure, because you're not paying for capacity that sits idle most of the time.

The key question isn't "what's the peak traffic?" but "how predictably and consistently is capacity used?" A service with 10,000 requests per second for 5 minutes a day has completely different economics than a service with 100 requests per second continuously.

Maximum GPU throughput doesn't equal real production capacity, because different workloads have different latency requirements. Batch indexing can run at maximum capacity (completion matters more than speed). Search queries require strict latency budgets (delay affects UX and revenue). Reranking with expanded candidate sets increases documents/sec but decreases searches/sec. Infrastructure needs to deliver throughput that respects latency requirements, not just hit a benchmark peak.

Break-Even and the Economics of Choice

Cohere calculated the economic crossover points using a single NVIDIA A10 GPU with standard consumption-based pricing (excluding discounts, 24/7 traffic):

• Query Embedding (single request, 8 tokens): break-even at ~tens of thousands of req/min • Catalog Embedding (100 texts × 200 tokens, 20K per request): break-even at ~20 req/min • Long-document Embedding (100 texts × 1000 tokens, 100K per request): break-even at ~4 req/min • Reranking (1 query + 50 documents, 11K tokens): break-even at ~29 req/min

One model, one price — and the break-even point shifts by hundreds of times depending on request size. A catalog profile at 30 req/min looks modest, but it's already above the 20 req/min break-even point — dedicated is already more economical. Change the request size, the number of candidates, the latency requirements, or the traffic pattern, and the break-even point shifts dramatically. That's why there's no universal answer: the choice between shared and dedicated depends on the specific workload, required latency, and traffic patterns — not on headline metrics like price per token or RPM.