GPU Clouds, Serverless Inference and Vector Databases Compared: 8 LLM Infrastructure Options in 2026

Every LLM API price is quoted per million tokens, which makes it look like a software cost. It is not. Behind each token is a GPU rented by the hour, and the three ways to buy that GPU - spot market, per-second cloud, and reserved contract - differ by more than 10x for the same hardware. The two discontinued products in this list are here on purpose: both were once reasonable choices, and both are now cautionary data points.

1. Vast.ai: The Spot Market, With Spot Market Risk

Vast.ai is a marketplace where individual hosts rent out their own GPUs. Prices float and are often half of what the branded clouds charge, because you are buying capacity from someone's spare machine rather than a data centre contract.

Pricing: There is no rate card, only live listings. As of this writing an RTX 3060 starts near $0.03 per hour with a median around $0.07, an RTX 4070 Super from $0.08, an L40S from $0.33 with a median of $0.73, and an A100 SXM4 80 GB from $0.40 with a median of $0.84.

Where it falls short: Hosts can and do disappear mid-job. Reliability, network speed and whether the machine is actually the GPU you paid for vary by host, and you are responsible for verifying rather than the platform verifying for you. It is the cheapest option and also the one requiring the most operational maturity.

2. RunPod: Per-Second Billing and Fixed Rates

RunPod sits between the marketplace and the contract cloud: a fixed rate card, per-second billing, and both community and secure tiers plus serverless endpoints.

Pricing: RTX 3090 runs $0.50 per hour, RTX A6000 $0.59, L40 $0.82, RTX 4090 $0.89 and RTX 5090 $1.19. Per-second billing means a nine-minute job costs nine minutes, which matters enormously for development and profiling where you start and stop constantly.

Where it falls short: Per-second billing encourages you to leave instances running because stopping is cheap - and a forgotten instance billing by the second all month is not cheap. Availability of high-end cards also fluctuates, so a fixed rate card does not guarantee a fixed supply.

3. Lambda: Reserved Capacity for Real Workloads

Lambda sells reserved GPU capacity on two-week to one-year terms, which is the structure you use when the job is not a demo but a training run that has to finish.

Pricing: On-demand rates run $1.99 per GPU hour for an A100 SXM 40 GB, $3.29 for H100 PCIe, $4.29 for H100 SXM, $2.29 for GH200 and $6.99 for a B200 SXM6. Reserved contracts are cheaper per hour - an H100 at 16 GPUs is $6.16 per GPU hour and drops to $5.54 at 256 GPUs, while a B200 at 16 GPUs is $9.86.

Where it falls short: The reserved pricing is priced per GPU and quoted at volume - the $5.54 H100 rate requires committing to 256 GPUs, which is not a decision a small team can make. On-demand rates are among the highest here, so Lambda is the right answer only when you actually need guaranteed capacity.

4. DeepInfra: Serverless Tokens, No Instances to Manage

DeepInfra sells the thing most people actually want: tokens, not GPUs. You send a request, you get a completion, you pay per million tokens, and you never think about whether a machine is warm.

Pricing: Per-token rates vary by model and are among the lowest available for open models. Phi-4 runs $0.07 per million input tokens and $0.14 per million output; Mistral-Nemo-Instruct around $0.029 input and $0.03 output. Larger hosted models price higher - Claude Sonnet 5 is $3 input and $15 output per million.

Where it falls short: Serverless means shared capacity, so latency under load is not something you control, and cold starts appear at the least convenient moments. You also inherit the provider's model list and version choices rather than running whatever weights you want.

5. Chroma: Open Source Plus a Metered Cloud

Chroma is the vector store you can run yourself - the open-source version is free and local - and Chroma Cloud is the hosted version billed by usage rather than by seat.

Pricing: Starter is $0 per month plus $5 in free credits, then usage: $2.50 per GiB written, $0.33 per GiB per month stored, $0.0075 per TiB queried and $0.09 per GiB returned. Team is $250 per month including $100 of credits with 100 databases and 30 team members.

Where it falls short: Writing is the expensive operation - $2.50 per GiB - so reindexing a large corpus repeatedly gets costly fast, while querying barely registers. The pricing rewards you for writing once and querying often, which is the opposite of how a lot of prototype code behaves.

6. Pinecone: Managed, With a Real Jump Between Tiers

Pinecone is the mature managed vector database: no infrastructure, predictable operations, and the tier structure splits at a price point worth knowing before you adopt it.

Pricing: Starter is free with 1,000 ingestion units per month. Builder is $20 per month flat with 10,000 units included. Standard jumps to $190 per month. Beyond the included units, ingestion runs $0.0005 per unit and embedding inference $0.16 per million tokens for llama-text-embed-v2.

Where it falls short: That $20 to $190 step is a 9.5x increase, and it is not obvious which feature forces it - teams regularly discover they need Standard for a requirement they did not anticipate. Starter also caps you at a single region.

7. Anyscale Endpoints: Retired

Anyscale Endpoints was a hosted API for open models like Llama, positioned as production-grade at a time when running Llama yourself was impractical. The endpoint no longer exists: in July 2026 Anyscale signed a definitive agreement to join Nscale, and Anyscale now sells GPU compute for Ray workloads rather than hosted LLM endpoints. If you are still routing traffic to it, the five current substitutes are compared in 5 Anyscale Endpoints Alternatives.

What this teaches: "Production-grade API for open models" turned out to be a transitional product. The moment open weights became cheap to serve yourself, the value of a hosted wrapper around them collapsed, and the business pivoted to the orchestration layer it actually owned.

8. AI21 Jamba and Maestro: Sunset in August 2026

AI21 built Jurassic, then Jamba, then Maestro. In May 2026 it cut about 60% of its staff and halted Jamba development, and on August 9, 2026 both the Jamba and Maestro APIs were formally sunset. Five current substitutes are compared in 5 AI21 API Alternatives.

What this teaches: A well-funded model vendor with a $1.4 billion valuation is not a guarantee of an API surviving. If a model provider is not among the three or four with genuine scale, the migration risk belongs in your planning, not in a footnote.

How to Choose

Match the purchase to the workload shape. Intermittent development and profiling: RunPod, because per-second billing makes start-stop cheap. A long training run that must finish: Lambda reserved, accepting the volume commitment. Batch inference where cost dominates and you can tolerate host variance: Vast.ai.

Serving real traffic without wanting to manage machines: DeepInfra, and budget for the latency variance. Vector storage: Chroma if you want the open-source escape hatch and write-once-query-many access; Pinecone if you want managed operations and can absorb the tier jump.

The number worth remembering is the spread, not any single rate: an A100 80 GB costs roughly $0.84 per hour at the marketplace median and $1.99 at a reserved cloud - and that ratio, not the token price, is what determines whether self-hosting beats an API.