I Rebuilt My RAG Pipeline Without LangChain — What Got Better and What Got Worse

The first time I seriously doubted the framework was not because the model hallucinated. It was because the answer looked plausible, contained a citation, and was still wrong. The assistant had retrieved a chunk from a deprecated help page because one part of the pipeline applied a metadata filter, another part did not, and the final prompt assembly made the whole thing look coherent. Debugging it meant stepping through wrappers, runnable compositions, and framework-specific assumptions instead of asking the real question: why did retrieval favor the wrong document? That was the point where I stopped treating LangChain as the core of the RAG system and started treating it as an optional integration layer. This is not an anti-framework article. LangChain solved a real problem: it gave developers a fast way to compose LLM applications when the ecosystem was young and everyone was still figuring out the basics. But once RAG moved from demo to production, the problems changed. The hard parts stopped being “call the model” and became: permission-aware retrieval stable chunking hybrid search reranking evaluation document ingestion failures embedding migrations traceability when an answer goes wrong Rebuilding the pipeline without LangChain made some things dramatically better. It also made some things more annoying, more expensive, and more time-consuming than I expected. This article is about both. If you are deciding whether to keep, adopt, or remove LangChain from a production RAG system: Removing LangChain improved debugging, retrieval control, evaluation, observability, and cost discipline. Removing LangChain made harder document loading, integration maintenance, and the long tail of “small” pipeline decisions. The biggest win was not performance. It was that the pipeline became explicit. The biggest downside was that I became responsible for a lot of boring glue code that frameworks usually hide. My current rule: prototype with high-level tools, but own the retrieval core when the product depends on answer quality. 📋 Table of Contents 1. The abstraction stopped being a shortcut and became a boundary 2. Chunking stopped being “split by 800 characters” 3. Retrieval became a small query planner 4. Hybrid search was the unglamorous fix for exact identifiers 5. Reranking became the highest-leverage quality gate 6. Embedding generation became a data-engineering job 7. Evaluation got easier once the pipeline had seams 8. Observability changed from “the answer looks weird” to “chunk 7f2a was dropped” 9. What got worse: the long tail of boring integration work 10. Where I draw the line now Scenario: A user asks, “What changed in webhook authentication?” The system retrieves something that mentions authentication, but not the correct product version. The final answer sounds confident. The problem is not the model. The problem is that the retrieval request did not carry the right filters, and the abstraction made that hard to see. Why it matters: In early RAG projects, abstractions help you move quickly. You connect a loader, a splitter, an embedding model, a vector store, and a prompt template. But in production, the interesting failures happen in the spaces between those components. When those spaces are hidden behind generic chain-like composition, you end up debugging the composition layer instead of the retrieval behavior. Solution: I rebuilt the pipeline around explicit stages with small interfaces. Not a huge framework. Just enough structure to make each stage testable. from dataclasses import dataclass from typing import Protocol @dataclass(frozen=True) class RetrievedChunk: chunk_id: str doc_id: str text: str score: float metadata: dict class Retriever(Protocol): def retrieve( self, query: str, *, filters: dict | None = None, limit: int = 20, ) -> list[RetrievedChunk]: ... The pipeline then became a sequence of ordinary functions: def answer_question(user_query: str, user_context: UserContext) -> FinalAnswer: plan = plan_query(user_query, user_context) candidates = retriever.retrieve( plan.retrieval_query, filters=plan.filters, limit=40, ) evidence = select_evidence(plan.raw_query, candidates) prompt = build_prompt(plan.raw_query, evidence) return generate(prompt, request_id=plan.request_id) Why this works: The important part is not that this code is “framework-free.” The important part is that the seams are visible. If retrieval is bad, I look at plan_query and retriever.retrieve. If the prompt is bad, I look at build_prompt. If the answer is unfaithful, I inspect evidence. There is no chain abstraction sitting between me and the failure. 💡 Practical note: If your LangChain usage already has clear boundaries around retrieval, parsing, and prompt construction, removing the framework may not help much. The problem is not the library itself. It is whether the library hides the decisions you now need to debug. Scenario: A support article contains a table of error codes. The user asks about one specific code. The retriever returns a chunk that includes the correct code, but not the header row explaining what the columns mean. The model guesses. Sometimes it guesses wrong. Why it matters: A lot of early RAG advice treated chunking as a text-length problem: Pick a chunk size, add overlap, repeat. That works for simple prose. It falls apart for real documents: tables code blocks numbered steps headings with nested context FAQs legal clauses API reference docs product changelogs In production, chunking is not a text problem. It is a document-structure problem. Solution: I stopped thinking of chunks as “pieces of text” and started treating them as evidence units. An evidence unit should carry enough context to be interpreted without its surrounding document. For Markdown-like documents, that usually means: preserve heading hierarchy keep tables intact when possible attach column/header context to table rows keep code blocks with their immediately preceding explanation avoid splitting a numbered step away from its introductory sentence A simplified version of the chunk model: @dataclass(frozen=True) class Chunk: chunk_id: str doc_id: str heading_path: tuple[str, ...] text: str block_type: str # paragraph, table, code, list token_estimate: int source_url: str updated_at: str For tables, I do not only store the raw row. I store enough surrounding structure to make the row meaningful: row_text = ( "Error Code: E1042\n" "Meaning: Webhook signature expired\n" "Resolution: Regenerate signing key and replay event\n" "From table: Error reference / Webhooks / Common failures" ) Why this works: The model does not only need the right passage. It needs the right passage in a form where the meaning is self-contained. A chunk like: E1042 | Webhook signature expired | Regenerate signing key is much weaker than: Error Code: E1042 Meaning: Webhook signature expired Resolution: Regenerate signing key and replay event From table: Error reference / Webhooks / Common failures The second one is easier to retrieve, easier to rerank, and easier for the model to use faithfully. ⚠️ Gotcha: Overlap is not a substitute for context. Overlap helps at sentence boundaries, but it does not recover lost table headers, section titles, or document-level metadata. Scenario: A customer asks, “How do I rotate API keys?” The system retrieves a generic security page instead of the tenant-specific admin guide. Another user asks, “What changed in v2?” The retriever returns v1 and v3 documentation because the query does not carry version intent. Why it matters: A raw user query is rarely the best retrieval query. Users are terse. They use pronouns. They assume context. They mix product names, abbreviations, and incomplete descriptions. If you pass the raw query straight into a vector store, you are asking semantic search to solve problems that often belong to query planning. Solution: I introduced a lightweight planning stage before retrieval. It does three things: rewrite the query for retrieval extract filters choose the retrieval mode @dataclass(frozen=True) class QueryPlan: request_id: str raw_query: str retrieval_query: str filters: dict mode: str # "hybrid", "keyword", "semantic" A simplified planner: def plan_query(raw_query: str, user_context: UserContext) -> QueryPlan: filters = { "tenant_id": user_context.tenant_id, "allowed_doc_types": user_context.allowed_doc_types, } if user_context.product_version: filters["product_version"] = user_context.product_version retrieval_query = rewrite_for_search(raw_query) mode = choose_retrieval_mode(raw_query) return QueryPlan( request_id=user_context.request_id, raw_query=raw_query, retrieval_query=retrieval_query, filters=filters, mode=mode, ) The important part is not the exact implementation. The important part is that retrieval now receives structured intent. For example: { "tenant_id": "acme", "product_version": "v2", "doc_type": ["admin_guide", "api_reference"] } That is much better than hoping the embedding model figures it out. Why this works: Most production RAG failures are not “the model is dumb.” They are retrieval-context failures. A query planner lets you separate: what the user asked what should be searched what documents are allowed whether exact matching matters more than semantic matching That separation becomes critical once you add multi-tenancy, permissions, or versioned documentation. 🔍 Why this matters: Permission filtering should be part of retrieval, not a post-processing idea. If you retrieve first and filter later, you often lose the best candidates to inaccessible documents. Scenario: A user searches for SKU-8842 or error E1042 or endpoint /v2/webhooks/signatures. Vector search returns conceptually related content, but not the exact item. Semantic search is good at meaning, but it can be surprisingly bad at identifiers. Why it matters: A lot of RAG demos focus on natural language questions: “How do refunds work?” “What is your billing policy?” “Summarize this document.” But production systems also get queries like: ORD-55213 stuck in pending ERR_TIMEOUT during sync Stripe webhook 500 after upgrade INV-2049 tax calculation mismatch These are not purely semantic queries. They often contain exact tokens that matter more than conceptual similarity. Solution: I stopped pretending vector search was enough by itself and moved to a hybrid retrieval model: vector search for semantic recall keyword search for exact terms, IDs, and rare tokens fusion to combine the result sets A simple reciprocal rank fusion implementation looks like this: from collections import defaultdict def reciprocal_rank_fusion( ranked_lists: list[list[str]], k: int = 60, ) -> list[str]: scores: dict[str, float] = defaultdict(float) for ranked_list in ranked_lists: for rank, chunk_id in enumerate(ranked_list): scores[chunk_id] += 1.0 / (k + rank + 1) return sorted(scores, key=scores.get, reverse=True) The idea is simple: semantic search contributes candidates that match meaning keyword search contributes candidates that match exact strings RRF gives extra credit to chunks that appear in multiple result lists Why this works: Hybrid search improves recall in a way that feels almost unfair when you first see it. Vector search is great at: paraphrasing conceptual questions intent-level matching Keyword search is great at: product names error codes function names filenames SKUs API routes rare domain terms Together, they cover failure modes the other one misses. Practical note: The biggest hidden cost of hybrid search is not the code. It is maintaining the lexical side properly: analyzers stemming rules stopwords synonyms field weights tokenization of identifiers If you treat keyword search as an afterthought, it will disappoint you. Scenario: The retriever returns 30 chunks. The correct one is in the set, but it is buried below several plausible-looking alternatives. The prompt builder takes the top 6 by raw retrieval score. The model uses the wrong one because it appeared first. Why it matters: Retrieval score is not the same as answer usefulness. A chunk can be semantically close and still be the wrong evidence because: it is from the wrong version it is too generic it is outdated it lacks the exact detail required it is a sibling section, not the target section This is where reranking earns its place. Solution: The pipeline changed from: retrieve top-k → put into prompt to: retrieve broad → rerank narrow → select evidence A reranker can be any model or function that scores query-document relevance more carefully than the first-stage retriever. A simple interface: class Reranker(Protocol): def score(self, query: str, text: str) -> float: ... def select_evidence( query: str, candidates: list[RetrievedChunk], reranker: Reranker, *, top_n: int = 6, min_score: float | None = None, ) -> list[RetrievedChunk]: scored = [(chunk, reranker.score(query, chunk.text)) for chunk in candidates] scored.sort(key=lambda item: item[1], reverse=True) selected: list[RetrievedChunk] = [] for chunk, score in scored[:top_n]: if min_score is not None and score < min_score: break selected.append(chunk) return selected Why this works: The first retrieval stage should optimize for recall. The reranking stage should optimize for precision. The prompt should receive only the strongest evidence. That separation gives you much more control than trying to make one retrieval pass do everything. In practice, reranking often improves answer quality more than: prompt wording changes minor chunk size tweaks changing temperature adding more verbose instructions It is not glamorous, but it is one of the highest-leverage upgrades you can make. 🧠 The important part: Do not rerank hundreds of chunks per request. Retrieve a broader candidate pool, then rerank a manageable subset. Reranking is powerful, but it should still be treated as a scarce resource. Scenario: A document source changes. A few hundred pages are updated. Later, you switch embedding models or change chunking. Suddenly you need to re-embed a large corpus without duplicating content, losing traceability, or overwhelming rate limits. Why it matters: In demos, embedding is a function call. In production, embedding is a data pipeline. You need to know: which document version produced this chunk which embedding model produced this vector whether the chunk changed since last index whether reindexing is incremental or full whether two documents are accidentally producing duplicate chunks whether a failed batch can be retried safely Solution: I started treating chunks and embeddings as derived artifacts with deterministic identifiers. The most important change was using stable chunk IDs based on content and position, not random IDs. import hashlib def chunk_identity( doc_id: str, chunk_position: str, text: str, ) -> str: canonical = f"{doc_id}|{chunk_position}|{text.strip()}".encode("utf-8") return hashlib.sha256(canonical).hexdigest() Then the database row becomes something you can upsert safely: INSERT INTO chunks ( chunk_id, doc_id, heading_path, content, embedding, embedding_model_version, updated_at ) VALUES ( $1, $2, $3, $4, $5, $6, now() ) ON CONFLICT (chunk_id) DO UPDATE SET content = EXCLUDED.content, embedding = EXCLUDED.embedding, embedding_model_version = EXCLUDED.embedding_model_version, updated_at = EXCLUDED.updated_at; Why this works: When chunk identity is stable, reindexing becomes manageable. You can: re-embed only changed documents keep model version in metadata compare old and new embeddings during migrations avoid duplicate chunks from repeated ingestion runs roll back to a previous embedding model more easily This sounds boring until you have to migrate an index in production. Then it becomes one of the best decisions you made. Production warning: If your chunk IDs are random, every ingestion failure eventually becomes a data integrity problem. You will either accumulate duplicates or lose traceability between chunks and source documents. Scenario: You change the prompt. The answers look better on a few examples. But you do not know whether retrieval improved, whether the model is just writing more confidently, or whether you quietly broke a previously working case. Why it matters: RAG systems fail in multiple places: The retriever did not find the right chunk. The reranker buried the right chunk. The prompt included too much noise. The model ignored the evidence. The evidence was wrong because the source data was stale. If you only evaluate final answers, you cannot tell which of those happened. Solution: Once the pipeline had explicit stages, I started evaluating them separately. For retrieval, the most useful metrics were simple: Hit@k MRR recall over known golden chunks failure analysis by document type A small MRR helper: def mean_reciprocal_rank( golden: list[tuple[str, set[str]]], retriever: Retriever, k: int = 10, ) -> float: if not golden: return 0.0 total = 0.0 for query, relevant_chunk_ids in golden: results = retriever.retrieve(query, limit=k) result_ids = [chunk.chunk_id for chunk in results] for rank, chunk_id in enumerate(result_ids, start=1): if chunk_id in relevant_chunk_ids: total += 1.0 / rank break return total / len(golden) That alone changed the debugging conversation. Instead of saying: “The answers feel worse.” we could say: “Retrieval MRR dropped from 0.82 to 0.61 on API reference questions.” That is a real engineering signal. Why this works: Separating retrieval evaluation from generation evaluation prevents you from optimizing the wrong thing. If retrieval is bad, prompt engineering only hides the problem temporarily. If retrieval is good but generation is bad, then prompt and model choices matter more. Once those boundaries were clear, improvements became much less emotional. 💡 Practical note: Keep a small golden set that is adversarial: exact IDs, version-specific questions, permission-sensitive documents, ambiguous phrasing, and multi-step questions. A golden set full of easy questions will flatter your pipeline and teach you nothing. Scenario: A user reports a bad answer. You try the same question later and it works. Or you try it again and it fails, but you do not know why. The final response gives you almost nothing to debug with. Why it matters: RAG failures are often stateful and data-dependent. They can depend on: which retriever was used which filters were applied which candidates were retrieved which chunks survived reranking which prompt template was selected which model version answered whether the source document had changed that morning If you only log the final prompt and final answer, you are missing most of the story. Solution: I started logging a structured trace for every request. The fields that proved most useful: trace = { "request_id": plan.request_id, "raw_query": plan.raw_query, "retrieval_query": plan.retrieval_query, "filters": plan.filters, "mode": plan.mode, "candidate_chunk_ids": [c.chunk_id for c in candidates], "selected_chunk_ids": [c.chunk_id for c in evidence], "rerank_scores": rerank_scores, "prompt_template_id": prompt_template.id, "model_name": model.name, "latency_ms": latency_ms, } The important detail is that I log chunk IDs, not only text. That lets me answer questions like: Was the right chunk retrieved but dropped by reranking? Was it retrieved but filtered out by permissions? Was it retrieved but excluded because of prompt budget? Did two queries hit different document versions? Did the source document change between incidents? Why this works: Observability in RAG is not just about LLM calls. It is about tracing evidence flow. A good trace tells you where the evidence was lost: query → filters → retrieval → fusion → rerank → prompt → generation Without that, you end up guessing. With it, you can often reduce a “bad answer” report to one of a few concrete causes in minutes. This is the part people often skip in “I quit the framework” articles. Removing LangChain did improve the core pipeline. But it also transferred a lot of maintenance burden back to me. The easy part of a loader is reading a file. The hard part is handling: PDFs with multi-column layouts scanned documents tables that break across pages headers and footers polluting content HTML navigation menus mixed into article text Markdown with embedded components PowerPoint slides with text boxes in random order Confluence pages with macros Notion blocks with nested toggles tickets with rich text and attachments permission metadata that lives outside the document itself LangChain and the broader ecosystem do not solve all of this perfectly, but they give you a starting point. When you remove that layer, you either: build your own parsers maintain wrappers around third-party extraction tools accept lower quality ingestion None of those are free. Once the framework is gone, you also own more of the surrounding plumbing: retry logic for embedding APIs batching and rate-limit handling ingestion workers schema migrations for vector indexes versioned prompt templates feature flags for retrieval modes cache invalidation multi-tenant filtering source synchronization failed document quarantine This is not glamorous work. But it is the work that keeps a RAG system alive after the demo. One underrated benefit of popular frameworks is that common patterns are already visible: loaders retrievers splitters record managers evaluators tracers example recipes When you build everything yourself, you lose some of that shared vocabulary. That can slow down onboarding and make it easier to reinvent mediocre solutions. So no, the rebuild was not a pure win. It was a trade. I gained control, but I also accepted responsibility for parts of the system I would have preferred to outsource. After going through this, I no longer ask: “Should we use LangChain or not?” That question is too broad. The better question is: “Which parts of the RAG pipeline need to be explicit, and which parts can remain high-level?” For me, the answer depends on what the product actually needs. you are still prototyping your main goal is speed retrieval quality is not the core product differentiator you need many quick integrations your documents are relatively simple you are still exploring whether RAG is even the right approach you need agent/tool orchestration more than retrieval precision In those cases, LangChain or a similar framework can still be a very reasonable choice. answer quality is directly tied to product value retrieval must respect permissions, tenancy, or versioning your corpus contains messy documents exact identifiers matter as much as semantic meaning you need rigorous evaluation you need fine-grained observability you are optimizing cost and latency carefully you want stable interfaces across models and vendors That is where owning the pipeline starts to pay off. Dimension LangChain / high-level framework Custom pipeline Speed to first demo Very strong Slower Debugging retrieval behavior Can be indirect Usually clearer Integration breadth Strong Requires more work Control over chunking/retrieval Medium High Evaluation discipline Possible, but not automatic Easier to enforce Operational ownership Partially shared Mostly yours Best fit Exploration, broad tooling, agent-heavy systems Production systems where retrieval quality is critical If I am building a system where the user mostly needs an assistant over many tools, and retrieval is just one supporting feature, I am still open to high-level orchestration. If I am building a system where the core promise is: accurate answers correct citations permission-aware search version-specific behavior reliable evidence selection then I want the retrieval core to be explicit, boring, and fully owned. That is the real lesson. The benefit of removing LangChain was not that the pipeline became smaller. It became legible. And once the pipeline was legible, every hard problem became easier to locate: bad chunking weak query planning missing hybrid recall noisy reranking stale documents broken permissions prompt overflow At that point, improving RAG stopped feeling like prompt roulette and started feeling like engineering. That is the state you want to reach, whether you use a framework or not.