Does using Azure OpenAI mean our data stays in our tenant?

It means processing happens within Azure under your subscription and region selection, with enterprise terms excluding your data from training. That is meaningfully stronger than a consumer API. It is not the same as the data never leaving your boundary, because the model and the service are still operated by a third party on infrastructure you do not control. Whether that distinction matters depends entirely on what your obligations actually say.

Is Azure OpenAI enough for regulated workloads?

Often yes, and many regulated organizations run substantial workloads on it. It stops being enough when an obligation requires the data to remain inside a boundary you control, when you must reproduce a specific output months later and cannot depend on a model version remaining available, or when the workload must function without external connectivity.

What does self-hosting give us that Azure OpenAI does not?

Three things contract terms cannot supply: the data physically staying inside your boundary, control over model versioning so an output can be reproduced for audit, and independence from a provider's pricing, packaging and deprecation decisions. Everything else, including residency and training exclusion, is generally obtainable contractually.

Should we move everything off Azure OpenAI?

Almost certainly not. The workloads that genuinely require private deployment are usually a minority of volume and a majority of risk. Moving everything means paying the operational cost of self-hosting for work that never needed it. Size the private deployment for the subset that requires it.

What is the migration risk?

Prompts tuned against one model rarely transfer unchanged, so expect a tuning pass per workload. Build an evaluation set that runs against either backend before you start, put an abstraction layer in front of the provider, and move one workload class at a time behind a flag with both paths measurable.

Azure OpenAI vs Self-Hosted: Where the Boundary Actually Sits

Most enterprise AI in large organizations starts on Azure OpenAI, and usually for good reasons: the commercial relationship already exists, the security review is shorter, and the identity and networking integration is already understood.

The question that arrives later is whether some workloads need to leave. That question gets debated as though it were ideological. It is not. It comes down to what your specific obligations require, and for most organizations the honest answer is that some workloads need to move and most do not.

What Azure OpenAI actually gives you

It is worth being precise, because this is where the argument usually goes wrong in both directions.

Processing happens within Azure, under your subscription, in a region you select. Enterprise terms exclude your content from training the underlying models. You get the identity, networking and monitoring integration of a cloud you already run, which is not a small thing: a private deployment that nobody integrated properly into your access controls is worse, not better.

That combination satisfies a great many regulated workloads. Organizations under financial services and healthcare regulation run real production systems on it.

What it does not give you

Three things, and they are the same three regardless of which commercial provider you are comparing against.

The data physically staying inside your boundary. Azure's isolation model is strong, but the service is operated by a third party on infrastructure you do not control. If an obligation says the data must not leave a boundary you own, contractual assurance does not satisfy it. Whether your obligation actually says that is a question for your legal and compliance functions, not for an architecture diagram.

Control over model versioning. Providers deprecate model versions on their own schedule. If you need to explain to a regulator months later why a system produced a particular output, and the version that produced it is gone, you cannot reproduce it. Version pinning helps within the provider's support window and not beyond it. In regulated decisioning this requirement alone frequently forces self-hosting, and it is the one organizations discover last.

Independence from the roadmap. Pricing, packaging, quota and feature availability are outside your control. For an experiment that is fine. For a capability that becomes load-bearing across the business, it is a dependency your risk function will eventually raise.

The test that settles it

Work these in order, per workload class rather than for the organization:

Does a law, regulation, accreditation or executed customer contract prohibit the data leaving a boundary you control?
Would exposure breach privilege, trade secret protection or a confidentiality undertaking in a way contract terms do not adequately transfer?
Must the workload function without external connectivity?
Do you need to reproduce a specific output months later for audit?

Any yes points at self-hosting for that class. All no, and Azure OpenAI is very likely the faster, cheaper and better-integrated answer, and moving would be paying a real operational cost for a theoretical gain.

What self-hosting actually costs

The reason not to move everything is that self-hosting is not free once the hardware is bought.

Someone has to patch the serving stack, monitor quality, manage capacity, handle model upgrades and respond outside business hours. That capability is scarce and expensive to retain, and it is the line most often missing from the business case. Utilisation compounds it: owned capacity costs the same idle, so a daytime-only internal workload has very different unit economics from a continuous one.

Against that, you are also giving up integration you already have. A private deployment needs its own identity integration, its own permission-aware retrieval, its own observability. Those are all solvable and none of them are free.

The pattern that works

Keep Azure OpenAI for the general case. Stand up a private deployment sized only for the workload classes that genuinely require it. Put an abstraction layer in front of both so the routing decision is configuration rather than a rewrite.

Then write the classification rule that tells a developer which path a given workload takes. Without that rule everything drifts to whichever is easier to reach, and you end up with the operational cost of two systems and the compliance posture of the weaker one.

If you do migrate

Do not treat it as a swap. Build an evaluation set that runs against either backend before you begin, because without it you cannot tell whether a change helped. Expect a prompt tuning pass per workload, since prompts encode assumptions about a specific model's behaviour. Move one workload class at a time behind a flag with both paths measurable, and start with the one whose output is easiest to evaluate objectively rather than the one with the most volume.

Where to go next

The same framework applied to other providers is in Private LLM vs ChatGPT Enterprise and Private LLM vs Microsoft Copilot. For the migration mechanics see migrating off a hosted API.