Do we need LangChain or LlamaIndex for an enterprise AI system?

For a straightforward retrieval and generation pipeline, frequently not. That path is a modest amount of code you fully understand: embed, search with permission filtering, assemble context, call the model, log everything. Frameworks earn their place when you need many integrations you would otherwise write yourself, or when complex orchestration is genuinely part of the requirement.

What do orchestration frameworks actually give you?

Breadth of connectors, established patterns for chaining and agent loops, and a shared vocabulary that helps a team move quickly. For prototyping and for systems touching many data sources, that breadth is real value and rewriting it yourself is waste.

What is the main risk of adopting one?

The abstraction becomes the thing you debug. When output is wrong, you need to see the exact prompt sent, the exact context retrieved and the exact response received. Any layer that makes those harder to inspect costs you more in production than it saved in development, so evaluate on how easily you can see through it.

Does using a framework lock us in?

Less than teams fear, provided you keep your own boundaries. Keep prompts, evaluation sets and retrieval logic as your own artefacts rather than framework-shaped ones, and put your own interface between the application and whatever orchestration you use. Then the framework is an implementation detail you can replace.

What should we build ourselves regardless?

The evaluation harness, the permission filtering in retrieval, and the logging. Those three encode requirements specific to your organization, they are what you will rely on when something goes wrong, and they are the parts you least want to depend on someone else's release cycle for.

LangChain, LlamaIndex or Custom: Do You Need a Framework?

This decision is usually made by default. Someone builds a prototype with a framework because the tutorial used one, and that choice quietly becomes the architecture.

It is worth making deliberately, because the answer differs by component rather than for the system as a whole.

What a basic RAG pipeline actually is

Strip the abstraction away and the core path is short:

Take the question.
Embed it.
Search the vector store, filtering by the requesting user's permissions.
Assemble the retrieved context into a prompt.
Call the model.
Log the question, the retrieved documents, the prompt and the response.

That is a modest amount of code, and every line of it is something you will eventually need to reason about when an answer is wrong. Teams routinely adopt a framework to avoid writing it, then spend considerably longer working out what the framework did to their prompt.

If your system is essentially this path, writing it yourself is a defensible and often better choice.

Where frameworks genuinely earn their place

Breadth of connectors. If you need to pull from a dozen source systems, the integrations already exist and writing them yourself is waste with no upside.

Complex orchestration. Multi-step flows with branching, tool use and retries involve genuinely fiddly state management. Established patterns for that are worth using rather than reinventing.

Prototyping speed. Getting to something demonstrable quickly has real value, particularly when you are still establishing whether a use case works at all.

Shared vocabulary. A team that knows the same framework moves faster together than one inventing house conventions from scratch.

None of these are small. The mistake is not using a framework, it is using one without deciding what it is for.

The failure mode to watch

The abstraction becomes the thing you debug.

When an answer is wrong in production, you need three things quickly: the exact prompt that was sent, the exact context that was retrieved, and the exact response that came back. Any layer that makes those harder to see costs more during an incident than it saved during development.

So evaluate any framework primarily on transparency. Can you log the final prompt without effort? Can you see which documents were retrieved and why? Can you step through the flow without reading library source? If those are awkward, that is a stronger signal than any feature comparison.

The second failure mode is upgrade churn. Fast-moving libraries change interfaces, and a system built deeply against one spends real time tracking those changes rather than improving the product.

The pattern that works

Not framework or custom, but a boundary between them.

Keep three things as your own artefacts, whatever you use underneath:

The evaluation harness. It encodes what correct means for your organization. It is what you will rely on for every model change and every quality investigation. It should not depend on anyone's release cycle.

Permission filtering in retrieval. This is a security control, and it is the defect most commonly found at review. Own it, understand it, and be able to test it directly rather than trusting that a library applies your rules correctly.

Logging. Question, retrieved documents, final prompt, response, user. This is your audit trail and your debugging surface, and generic framework tracing rarely captures exactly what a regulator or an incident review will ask for.

Then put your own interface between the application and whatever orchestration sits below. If the framework becomes a liability, you replace the layer rather than the system.

A reasonable default

Prototype with a framework, because speed matters while you are learning whether the use case works.

Before production, look at what you are actually using. If it is a thin slice of a broad library, consider whether owning that slice is simpler than carrying the dependency. If you are using the breadth genuinely, keep it and invest in the boundaries above.

Either way, make it a decision with a reason attached, rather than an inheritance from whichever tutorial the prototype started with.

Where to go next

RAG at scale covers the architecture around this, permission-aware retrieval covers the control you should own, and multi-agent systems covers when orchestration complexity is genuinely warranted.