Two ways to run the same kind of model, with completely different bills, control, and blast radius when something goes wrong.
"Which model should we use" is only half the decision — where it runs is the other half, and it changes almost everything about the system built around it. Calling a hosted API means sending a request over the network to a provider's data center, paying per token, and never touching the weights yourself. Self-hostingmeans running inference on infrastructure you own or control, with the model's weights sitting on your own hardware. Same category of technology, opposite tradeoffs in latency, cost structure, data privacy, and who controls what version of the model you're actually running.
A hosted API trades away control — of the exact weights running, of when a model version changes underneath you, of where your request data physically travels — for zero infrastructure to buy, patch, or scale, and typically same-day access to whatever the provider considers their frontier model. Self-hosting trades that convenience back for genuine control: you decide exactly which weights are loaded and exactly when that changes, your request data never has to leave a boundary you define, and your latency depends on your own hardware and network topology instead of someone else's API queue. Neither side of that trade is free — the hosted side pays in reduced control and per-token bills that scale with usage; the self-hosted side pays in GPU capex, the engineering time to deploy and keep a model serving reliably, and usually a ceiling on capability, since the strongest frontier models are frequently only offered as a hosted API in the first place.
Not necessarily — this compares a marginal cost (per-token API pricing) against a picture that leaves out most of the real bill. Self-hosting means paying for GPU capacity whether or not it's actively serving requests, plus the engineering and operations time to deploy, monitor, patch, and scale a model-serving stack — costs that exist even at zero traffic. At low-to-moderate, bursty usage, a pay-as-you-go API is very often cheaper in practice, because idle self-hosted GPU capacity is still being paid for. Self-hosting tends to win on raw cost specifically at sustained high utilization, where the fixed infrastructure cost is spread across enough continuous inference volume to undercut per-token pricing — which is a real, common outcome for high-traffic production systems, just not a universal one. The honest comparison has to include utilization, not just the per-token number on the API pricing page.
Explains the real tradeoffs between calling a hosted model API and self-hosting a model on your own infrastructure — latency, cost model (per-token opex vs upfront capex plus ops), data privacy, and control over the exact weights and version running. Includes a side-by-side illustration of a request traveling over the network to a provider's data center versus inference running entirely inside a boundary you control, plus why 'self-hosting is always cheaper' depends heavily on utilization.
Both approaches can serve the same open-weights model, produce comparable output quality for a given model, and be wired into an application through what looks like an almost identical HTTP request from the calling code's point of view. That surface similarity hides a real architectural split: one path sends your request and data across a network boundary to infrastructure someone else owns and controls; the other keeps it inside a boundary you define, on infrastructure you're responsible for keeping running. The decision is really about who takes on operational responsibility and who controls the model artifact, not about which option is inherently "better AI."
Calling a hosted model API means your application sends a request over the network to a provider's infrastructure, which runs the model on their hardware and returns a response — you never touch a GPU, never manage model-serving software, and typically get access to frontier-capability models the same day they're released. In exchange, you pay per token used (an operating expense that scales directly with usage), your request data leaves your network boundary and travels to the provider (even if handled under a data-processing agreement or zero-retention policy), and the provider controls which exact model version is running behind that API endpoint — including when it gets updated, changed, or eventually deprecated.
Self-hosting means deploying a model's weights on infrastructure you own or control — on-premises hardware or a private cloud instance under your management — and running inference there directly. Your request data never has to leave a network boundary you define, you decide exactly which weights are loaded and exactly when that version changes (it doesn't change underneath you without your action), and your latency depends on your own hardware and network topology rather than a shared API queue. In exchange, you carry the capital cost of GPU hardware (or the ongoing cost of GPU cloud instances), the engineering effort to deploy and keep a model-serving stack running reliably at whatever scale you need, and — for many use cases — a capability ceiling, since the most capable frontier models are frequently offered only as a hosted API, not as downloadable open weights.
The right choice usually comes down to which constraint bites hardest for a given system: if regulatory or contractual requirements mean data genuinely cannot leave your network, or if request volume is high and sustained enough that dedicated GPU capacity is reliably kept busy, self-hosting's tradeoffs tend to pay off. If usage is bursty or low-to-moderate, the team doesn't want to own GPU operations, or the task benefits from the strongest available frontier model rather than an open-weights alternative, a hosted API is usually the pragmatic default. Many production systems end up hybrid — a hosted API for general capability, with a self-hosted smaller or fine-tuned model handling a specific high-volume, latency-sensitive, or data-sensitive sub-task.
Self-hosting keeps request data inside a boundary you define by default, which is a genuine privacy advantage. But reputable API providers increasingly offer data-retention opt-outs, zero-retention agreements, and enterprise data-handling terms that materially close that gap for many compliance needs — and a poorly secured self-hosted deployment is not automatically more private just because it's "yours." The comparison has to be made against the specific provider's actual data terms, not an assumption either way.
Only models whose weights are actually released for you to download and run — generally described as open-weights models. Many of the most capable frontier models are only made available through a hosted API, with their weights never released publicly, which is itself one of the practical factors pushing teams that need maximum capability toward the API path.
Not automatically. Self-hosting removes the round trip to a third party's data center, which can lower latency — but only if your own infrastructure and network are actually well provisioned for the load. Under-provisioned self-hosted infrastructure, or infrastructure geographically far from your users, can easily be slower than a well-optimized, geographically distributed hosted API.
Quantization — reducing a model's weight precision to shrink its memory footprint and speed up inference — is one of the most common techniques used specifically to make self-hosting practical, since it can let a model fit on GPUs you can actually afford to buy or rent, at some accuracy cost. It is far less relevant to the hosted-API side, where the provider handles that engineering decision internally.
The provider does. They decide when a model version is updated, when an older version is deprecated, and how much advance notice is given — which means your application's behavior can shift over time even if you never change a line of your own code. Self-hosting removes that risk entirely: your model version changes only when you deliberately update it.
Try our AI Engineering Studio
More calculators, simulators, and guides for this discipline.