Same model, different behaviour: the hidden cost of OpenRouter's provider lottery

Motivation

OpenRouter is what we call a “re-router”. It allows you to use Anthropic’s, OpenAI’s or even open-weight models with a single API key. The platform includes hundreds of models, and you might have already used it in your analysis. In addition to the model selection, OpenRouter also lets you choose which provider you use. For example, Llama 3 is an open-weight model, meaning anyone with sufficient computational resources is able to download the model and serve it. As a result, you can have up to 10 providers for an open-weight model, with potentially different pricing. Depending on the quantization, context window, hardware and many other parameters, the operating cost can vary, resulting in a x2 price in some cases. OpenRouter is widely used, especially for coding and role-play (which includes commercial chatbots). It allegedly redirects 20 trillion tokens per week.

This blogpost will explore why you should be careful when using OpenRouter to call open-weight LLMs.

Quantization

Recent LLMs contain many billions of parameters, and each parameter is a floating point number. When dealing with large scale computation, we fix the number of bits used by each parameter. The more bits you have, the more precise these parameters will be. The typical maximum size we use when releasing LLMs is 16 bits, i.e. 2 bytes. As a side note, this allows you to know how much disk storage a model uses: 2 bytes/param × 70B params = 140GB.

However, a commonly used technique to reduce the space needed (and to be able to run more instances of it) is quantization. Quantization compresses each parameter so that it uses fewer bits. Common sizes include 8 bits or 4 bits. When doing so, you trade disk storage for precision and capabilities. It’s well known that quantization often makes you lose a few percent on the benchmarks.

On OpenRouter, one can check that by looking at the details per provider. Many explicitly mention the quantization level.

For Llama 3.3 70B alone, we counted providers serving the model at fp16, bf16, fp8, and a few that simply don’t disclose the number (“unknown”). Groq is one of them, likely because it runs on its own custom chips instead of regular GPUs, which makes the usual quantization labels less meaningful.

It would be tempting to stop here and conclude “lower precision, worse behaviour, pick the highest precision provider and move on”. As we will see when covering the results, quantization is one visible knob, but other parameters also influence the results.

Putting it to the test

To move from “this could matter” to “this does matter”, we ran a controlled experiment on our own refusal benchmark (a suite of translation and summarization tasks derived from real, sometimes sensitive, legal documents). The paper which resulted from this analysis can be found on arXiv. We took a single model, Llama 3.3 70B, and a single set of tasks, and we forced OpenRouter to route every request to one specific provider at a time, instead of letting it pick automatically. Same model name, same prompts, same benchmark. The only thing we changed was who was actually running the model on the other end.

We then measured the refusal rate: out of all the tasks we asked the model to perform, what fraction did it decline to do (e.g. “I can’t help with that”)?

The results:

Provider	Refusal rate
Groq	11%
Cloudflare	55%
Novita	82%
Nebius	83%

That is roughly a 7.6x difference between the least and most “cautious” provider, for what is marketed, sold and billed as the same model. This isn’t noise or a fluke of a handful of unlucky samples: we ran this across hundreds of tasks per provider, and the gap is large and consistent.

Quantization alone doesn’t explain it

This is where the quantization story from earlier comes back in, and where it falls short as a full explanation. Nebius and Cloudflare, in our data, both run this model at the same fp8 quantization. Yet Nebius refused 83% of the time and Cloudflare 55% of the time, a 28-point gap between two providers that, on paper, look identical on the one dimension OpenRouter makes easy to check.

That doesn’t mean quantization is irrelevant. It’s a real, visible factor, and it’s worth checking. It means that quantization is, at best, one ingredient among several (context window, hardware, serving stack and, as we discuss below, possibly the exact checkpoint being served) that combine to produce very different real-world behaviour. You cannot infer “these two providers will behave the same” just because their quantization badge matches.

Where the disagreement actually comes from

TF-RefusalBench isn’t testing “write me a poem” against “give me a bomb recipe”. By design, it probes the gray zone: legal-adjacent, dual-use requests where a reasonable model could plausibly land on either side. That design choice turns out to be exactly why provider choice matters as much as it does.

We lined up all four providers’ verdicts, question by question, on the same 221 tasks. Only 10% got a unanimous call, meaning all four providers agreed to either answer or refuse. The single most common outcome, at 42% of questions, was a dead-even 2-2 split.

This is the mechanism, not a coincidence. Differences in quantization, hardware and serving stack perturb the model’s outputs by a small amount. On a request that’s unambiguously fine, or unambiguously off-limits, that perturbation doesn’t matter: every provider lands on the same side regardless. But for a request sitting right on the boundary, that same small perturbation is enough to tip the verdict either way. Since our benchmark deliberately leans into boundary cases, a large share of it turns out to be sensitive to exactly this kind of noise. We’d expect the same experiment on a set of obviously benign or obviously forbidden requests to show far less disagreement. The effect is a symptom of testing near the edge, not evidence that these models are randomly unreliable across the board.

It’s also worth ruling out a simpler explanation: maybe this isn’t the model’s behaviour changing at all, just our own scoring being noisy. We checked that directly, by freezing the model’s answers completely and only changing which provider ran the judge that scores them.

There, agreement was high: the different judge variants reached the same verdict on the same fixed answer 94% of the time. So the measurement itself is reliable, and the disagreement we’re reporting upstream is a real property of the model being served, not an artifact of how we’re scoring it.

Practically, this means the effect will hit hardest on exactly the use cases where getting the nuanced call right matters most: content moderation, compliance review, anything living near a policy boundary. It will barely register on requests that are obviously fine either way. It’s also precisely on these boundary cases that a stronger, higher-fidelity model would be expected to earn its keep by resolving the ambiguity correctly. A few bits of numerical precision are a small thing to lose on an easy question, but they matter most on the hard ones, which is exactly where quantization noise hits.

What about finetuning?

A possible explanation is finetuning. “Same model” on OpenRouter doesn’t guarantee that providers actually run the same model behind the scenes. They might apply their own additional safety finetuning, quantization-aware retraining, or small patches on top of the base checkpoint before deploying it, and none of that is visible anywhere in OpenRouter’s listing the way quantization is. We don’t have direct evidence this is happening here, but unlike quantization noise, a genuine finetuning difference would be a much more direct explanation for large, systematic behavioural gaps between providers, and it’s currently impossible to verify from outside. We’re flagging it as an open question worth investigating, not a confirmed cause.

Cost

The differences aren’t limited to behaviour. As mentioned in the introduction, running the same model through different providers can also mean a meaningfully different bill. In our own judge-cost comparison, the cheapest provider for a given (much smaller) model came in at roughly a third of the cost of the most expensive one, for comparable volume. Switching to a completely different model as judge multiplied the cost several times over on top of that.

Practical takeaways

“Same model” is not a guarantee of “same behaviour” on OpenRouter. The provider actually serving the request can matter as much as the model name itself. What to do about it depends on what you’re using the model for:

• I only need the model to parse a semi-structured document (pull fields out of an invoice, reformat a table): whatever provider works, the task isn’t near any decision boundary, so provider noise won’t show up.
• I’m building something that touches legal, medical, safety or policy-adjacent content: pin the provider explicitly, or self-host. This is exactly the gray zone where provider choice swings behaviour.
• I need to benchmark a model’s capability for an internal decision (“is this model good enough for X?”): the provider must be pinned, or the model run locally. Otherwise your benchmark number reflects whichever provider you happened to hit that day, not the model itself.
• I need results other people can reproduce later: self-host, or at minimum record and pin the exact provider and quantization you used, so the setup can be reconstructed.
• I’m prototyping and not shipping anything yet: whatever provider works, iterate freely and revisit the choice once it matters.
• I’m running a large one-off batch job where cost matters more than perfect consistency: let OpenRouter auto-route for the best price, but spot-check a sample of the output afterwards.

Whichever path you pick, check context windows against your real input sizes. That failure mode is silent regardless of provider, and pinning a provider doesn’t fix it if the one you pinned has a small context window.

Conclusion

OpenRouter’s promise (one API, any model, any provider) is genuinely useful, but it hides a layer of real variation underneath a single model name. For open-weight models especially, “which provider” can matter as much as “which model”. If your use case cares about consistent, auditable behaviour, it’s worth treating the provider choice as a first-class decision, not an implementation detail OpenRouter can be trusted to handle invisibly on your behalf.