Is an open-weight model free?

The weights are free to download, not free to run. You pay for the GPUs (rented or owned), the electricity, the ops and the people who keep it up. At low volume, a hosted API is usually cheaper than self-hosting once you count the full total cost of ownership.

What is the difference between open weights and open source?

Open weights means the parameters are published, so you can run and fine-tune the model. Open source is a stricter license bar: free use, modification and redistribution with no field-of-use restriction (the OSI definition). Apache-2.0 and MIT qualify; many lab community licenses, including Llama, do not, because they carry caps or use conditions.

When does self-hosting actually pay off?

At sustained, predictable volume. As an order of magnitude, self-hosting tends to break even around 500K tokens a day for a small model on a single GPU, and a few million tokens a day for a 70B-class model. Against budget APIs you may need tens of millions of tokens a month. Below that, the ops cost outweighs the token savings.

Can I use a US model and stay GDPR-compliant in the EU?

Often yes, but not by default. A US-hosted API can create CLOUD Act exposure, so you pin an EU region (Bedrock Frankfurt, Vertex Frankfurt, Azure West Europe), sign a data-processing agreement, and manage lawful basis, retention and subprocessors. The alternative is an EU-native provider such as Mistral, or self-hosting an open-weight model in your own EU infrastructure.

What changes with the EU AI Act in 2026?

The substantive obligations of the EU AI Act apply from 2 August 2026. In practice you classify your AI use, ensure AI literacy in your teams, and for higher-risk uses add documentation, risk management and human oversight. The deployment choice feeds into this: where the model runs and which provider you use affects how you demonstrate control and data handling.

← Stack

Deploying models

Lab API, managed open-weight, or self-hosted: where your model runs, and who pays for what.

Overview

There is a binary you hear everywhere: pay a proprietary provider, or run a free open model yourself. It is wrong on both ends. Running a model in production is a spectrum with three positions, and the choice is rarely about the model itself, it is about where the inference happens and who carries the cost, the control and the compliance.

Position one: a closed model behind a lab API (Anthropic, OpenAI, Google). You pay per token, you get the highest capability with zero infrastructure, and your data leaves your perimeter. Position two, the one most framings skip: an open-weight model on a managed inference provider (Together, Fireworks, Groq, Bedrock, Vertex, Mistral hosted). You still pay per token, but the weights are open, the catalogue is wide, and you can pin an EU region. Position three: the same open-weight model self-hosted on your own GPUs, on-prem or at the edge. Now you pay for infrastructure instead of tokens, you hold full control and data sovereignty, and you only come out ahead at sustained volume.

Two myths to retire before any of this makes sense. Open-weight is not free: downloading the weights costs nothing, but running them costs compute, ops and people. And open-weight is not open-source: published parameters do not imply an OSI-grade license. The Llama license carries a monthly-active-users cap and EU-use conditions; Qwen, DeepSeek and several Mistral models ship under genuinely permissive Apache-2.0 or MIT. The license is part of the deployment decision, not a footnote.

Architecture

Read the spectrum left to right. As you move from a lab API toward self-hosting, three things shift together: you stop paying per token and start paying for infrastructure, control and data sovereignty rise from low to high, and the time to get started goes from instant to a real engineering project. There is no universally correct position, only the one that matches your volume, your latency budget and your compliance constraints.

The deployment spectrum: lab API → managed open-weight → self-hosted, and the cost, control and sovereignty gradient across it.

Key concepts

Open weights: The trained model parameters are published and downloadable, so anyone can run the model. It says nothing about the license terms: open weights can still carry commercial caps or use restrictions.
Open source (OSI): A stricter bar than open weights: the license must meet the Open Source Initiative definition (free use, modification and redistribution, no field-of-use restriction). Apache-2.0 and MIT qualify; most lab "community" licenses do not.
Inference provider: A managed service that hosts open-weight models and serves them through an API, so you pay per token without owning GPUs. Together, Fireworks, Groq, AWS Bedrock and Google Vertex are examples; some add fine-tuning and EU regions.
Self-hosting: You run the model on infrastructure you control: rented GPUs, your own datacenter, or edge hardware. You trade per-token billing for capital and operational cost, and gain full control over data and uptime.
Total cost of ownership (TCO): The real cost of self-hosting, not just the GPU bill: add electricity, cooling, networking, tooling and the staff time to run it. A common rule of thumb puts true cost at the raw GPU cost multiplied by 1.3 to 3.
Prompt caching: Reusing a previously processed prefix (system prompt, tool definitions, prior turns) at a reduced price on later calls. It changes agent-loop economics, where the same large context repeats across many steps.
Data sovereignty: Control over where data is processed and which jurisdiction governs it. For EU buyers, a US-hosted API can create CLOUD Act exposure; an EU region or an EU-native provider keeps processing within EU jurisdiction.

When to use

Good fit

Lab API when you want the highest capability with zero infrastructure, your volume is low to moderate, and your data is not subject to a residency constraint you cannot satisfy with a region or a data-processing agreement.
Managed inference when you want an open-weight model (for portability, fine-tuning, or to avoid lab lock-in) without operating GPUs, and you value a wide catalogue or an EU region over running it yourself.
Self-hosting when volume is high and predictable, when data must never leave your perimeter, or when latency and uptime requirements justify owning the stack. It pays off at scale, not before.
A mix, routed per task: a frontier lab API for the hard reasoning steps, a cheap managed or self-hosted small model for high-volume classification and routing. The deployment choice is per workload, not per company.

Anti-patterns

Self-hosting at low volume to save money. Below roughly 500K tokens a day for a small model, or a few million for a large one, an API is cheaper once you count the full TCO. You will spend more on ops than you save on tokens.
Assuming open-weight means free. The weights cost nothing to download and everything to run. Budget the GPUs, the ops and the people before you commit.
Sending regulated EU personal data to a US-hosted API with no region pin and no data-processing agreement. The model quality is irrelevant if the data path is non-compliant.
Picking an open-weight model on capability alone without reading the license. A monthly-active-users cap or a field-of-use restriction can rule it out for your product after you have already built on it.

Code examples

Position 1: a closed model via lab API

from anthropic import Anthropic

client = Anthropic()  # ANTHROPIC_API_KEY in the environment
msg = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=512,
    messages=[{"role": "user", "content": "Summarise this ticket."}],
)
print(msg.content[0].text)
# You pay per token. No GPU, no ops. Data transits the provider.

Fastest path to the highest capability. The tradeoff is data egress and per-token cost that grows linearly with volume.

Position 2: an open-weight model via managed inference

from openai import OpenAI

# Same OpenAI-compatible client, pointed at a managed provider's
# base_url (Together, Fireworks, Groq, Bedrock-compatible gateways...).
client = OpenAI(
    base_url="https://api.example.com/v1",
    api_key="...",  # provider key
)
resp = client.chat.completions.create(
    model="Qwen/Qwen3-235B",  # open weights, Apache-2.0
    messages=[{"role": "user", "content": "Summarise this ticket."}],
)
print(resp.choices[0].message.content)
# Still per-token, but open weights and a provider-chosen region.

The middle ground: an open-weight model and a portable, OpenAI-compatible interface, without running GPUs yourself. Many providers let you pin an EU region.

Position 3: self-hosting the same open-weight model

# Serve an open-weight model on your own GPU with vLLM.
pip install vllm
vllm serve Qwen/Qwen3-235B --port 8000

# It exposes the same OpenAI-compatible API on localhost:8000.
# Now you pay for the GPU and the ops, not per token.
# Below sustained volume this costs MORE than the managed call above.

Identical API surface, opposite cost structure: infrastructure instead of tokens, full control instead of a shared endpoint. Worth it only at sustained, predictable volume.

Comparison

vs Lab API

Cost model, control, time to start

Per-token billing, highest capability, instant to start, zero ops. You give up data residency control and your cost grows linearly with volume. Best for low-to-moderate volume and the hardest reasoning, when compliance allows it.

vs Managed inference

Cost model, openness, region

Per-token billing on open-weight models, wide catalogue, portable OpenAI-compatible API, often an EU region. The pragmatic middle: open weights and no GPUs to run. You depend on the provider for uptime and pricing, but you can move the same weights elsewhere.

vs Self-hosting

Cost model, sovereignty, operational burden

Pay for infrastructure instead of tokens, full control and data sovereignty, and the heaviest operational burden. Breaks even only at sustained, predictable volume. The right call when data must stay in your perimeter or volume is large enough to amortise the GPUs.

Resources

FAQ

Is an open-weight model free?: The weights are free to download, not free to run. You pay for the GPUs (rented or owned), the electricity, the ops and the people who keep it up. At low volume, a hosted API is usually cheaper than self-hosting once you count the full total cost of ownership.
What is the difference between open weights and open source?: Open weights means the parameters are published, so you can run and fine-tune the model. Open source is a stricter license bar: free use, modification and redistribution with no field-of-use restriction (the OSI definition). Apache-2.0 and MIT qualify; many lab community licenses, including Llama, do not, because they carry caps or use conditions.
When does self-hosting actually pay off?: At sustained, predictable volume. As an order of magnitude, self-hosting tends to break even around 500K tokens a day for a small model on a single GPU, and a few million tokens a day for a 70B-class model. Against budget APIs you may need tens of millions of tokens a month. Below that, the ops cost outweighs the token savings.
Can I use a US model and stay GDPR-compliant in the EU?: Often yes, but not by default. A US-hosted API can create CLOUD Act exposure, so you pin an EU region (Bedrock Frankfurt, Vertex Frankfurt, Azure West Europe), sign a data-processing agreement, and manage lawful basis, retention and subprocessors. The alternative is an EU-native provider such as Mistral, or self-hosting an open-weight model in your own EU infrastructure.
What changes with the EU AI Act in 2026?: The substantive obligations of the EU AI Act apply from 2 August 2026. In practice you classify your AI use, ensure AI literacy in your teams, and for higher-risk uses add documentation, risk management and human oversight. The deployment choice feeds into this: where the model runs and which provider you use affects how you demonstrate control and data handling.