What Is a Private LLM?

Most discussions of private AI start with infrastructure. That is the wrong end of the problem. The useful definition is about where your data goes.

A private LLM is a large language model running on infrastructure your organization controls. The prompts your staff write, the documents the system retrieves, and the output it generates all stay inside your boundary. Nothing is sent to a third-party endpoint, and nothing depends on another company's retention policy being what its marketing page says it is.

That is the whole idea. Everything else is implementation detail.

The distinction people get wrong

"Private" and "on-premise" are used interchangeably and they are not the same thing.

On-premise means the model runs on hardware in your own facilities. You own the accelerators, they sit in your data centre, and you are responsible for power, cooling and replacement.

Private is the broader category. It includes on-premise, but it also covers self-hosting inside your own cloud tenancy, where the infrastructure is yours even though the building is not. Your VPC, your storage, your access controls, running an open-weight model you deployed.

Every on-premise deployment is private. Not every private deployment is on-premise. The distinction matters because the two have very different cost profiles and very different lead times, and organizations frequently commit to the harder one when the easier one would have satisfied the actual requirement.

What sits between the extremes

Deployment is a spectrum, not a binary:

• Public API. Requests to a vendor endpoint on shared infrastructure.
• Enterprise API tier. The same, with contractual terms on retention, training exclusion and residency.
• Dedicated capacity. Reserved capacity in your tenancy or a vendor's isolated deployment.
• Self-hosted in your cloud. Open-weight models on infrastructure you control, inside your own account.
• On-premise. Your hardware, your facilities.
• Air-gapped. On-premise with no external network path at all.

Most large organizations end up applying different tiers to different workload classes rather than picking one for everything. Sensitive work on self-hosted or on-premise infrastructure, general knowledge work on an enterprise API tier, experimental work on public APIs with non-sensitive data. That costs more to operate than a single choice and is usually still correct, because it matches control to actual exposure.

The governance requirement is a clear classification rule telling a developer which tier a given workload belongs in. Without that rule, everything drifts to whichever tier is easiest to reach.

When it is genuinely required

Private deployment is often a preference. Sometimes it is a hard requirement. The difference is worth establishing early, because it changes the cost conversation entirely.

It is required when any of these bind:

1. A legal or regulatory boundary. Some law, regulation or accreditation prohibits the data leaving a defined physical or jurisdictional boundary.
2. A contractual restriction. Customer or partner agreements limit processing to named infrastructure, or prohibit third-party processors. Check the actual executed contracts, not the standard template.
3. Privilege or confidentiality. Exposure would breach legal privilege, trade secret protection or a confidentiality undertaking.
4. Disconnected operation. The workload must function during a network outage or in an isolated environment.
5. Reproducibility for audit. You need to explain months later why a system produced a particular output. Hosted providers deprecate model versions on their schedule, not yours. In regulated decisioning this one alone often forces self-hosting, and it is the requirement organizations discover last.

If none of those bind, the decision is economics, and economics is mostly about volume and utilisation.

The capability question

The most common objection is that open-weight models are not good enough. That was a stronger argument two years ago than it is now.

The gap has narrowed considerably, though it has not closed at the very top end. More importantly, for most enterprise tasks the deciding factor is not raw model capability. It is retrieval quality, prompt design, and whether the system has access to the right documents with the right permissions applied. A better model retrieving the wrong context still produces the wrong answer.

The honest approach is to test on your actual task with your actual data before assuming either direction. Benchmark scores are a poor proxy for performance on your specific workload.

What it actually takes to run

The line most often understated in a private LLM business case is staffing. Someone has to patch, monitor, tune, manage capacity and respond when it breaks outside business hours. That capability is scarce and expensive to retain, and if you cannot hire and keep it, the cost model is fiction regardless of what the spreadsheet says.

The second most understated line is utilisation. Owned hardware costs the same whether it is saturated or idle. A workload that is busy for six hours on weekdays has very different unit economics from one running continuously, and comparing steady-state self-hosted costs against first-year API consumption is how organizations talk themselves into the wrong answer.

Where to go next

If you are working out whether private deployment is required for your case, the constraint test in the Enterprise AI Readiness Playbook walks through the six questions in order, and the deployment comparison in Private LLM vs Cloud API: Total Cost covers the economics in more depth.

Common questions

What is a private LLM?

A private LLM is a large language model that runs on infrastructure your organization controls, either on your own hardware or inside your own cloud tenancy. Prompts, retrieved documents and generated output stay inside your boundary rather than being sent to a third-party endpoint.

Is a private LLM the same as an on-premise LLM?

Not quite. On-premise means the model runs on hardware in your own facilities. Private is broader: it includes on-premise, but also self-hosting inside your own cloud account, where the infrastructure is yours even though the data centre is not. Every on-premise deployment is private, but not every private deployment is on-premise.

Do private LLMs perform worse than commercial APIs?

The gap has narrowed considerably but has not closed at the very top end. For most enterprise tasks the deciding factor is retrieval quality and prompt design rather than raw model capability, so the difference is often smaller than expected. Test on your actual task with your actual data before assuming either way.

When is a private LLM actually required rather than just preferred?

When a law, regulation or accreditation prohibits the data leaving a defined boundary; when customer or partner contracts restrict processing to named infrastructure; when the data would breach privilege or confidentiality if exposed; when the system must run without external connectivity; or when you need to reproduce a specific output months later for audit and cannot rely on a hosted model version remaining available.

What does a private LLM deployment cost?

The economics depend almost entirely on volume and utilisation. Owned hardware costs the same whether it is busy or idle, so a spiky daytime-only workload has very different unit economics from a continuous one. Model both options over the same three-year window with the same adoption assumptions, and include staffing, power and redundancy rather than just hardware.