Embedding Pipeline Chunking and Indexing Guide

An embedding pipeline turns a document into retrievable vectors in two decisions: how you cut the text into pieces (chunking), and how you store the resulting vectors so a query can find the right piece fast (indexing). Most RAG tutorials treat both as a solved default — split every 500 tokens, drop everything into whatever vector database the tutorial already uses — and most production retrieval problems trace back to exactly that default.

Key facts

Splitting text into short, self-contained "propositions" instead of fixed-size passages improved exact-match QA accuracy by 4.9–7.8 points and passage-retrieval Recall@20 by up to 10.1 points for unsupervised retrievers, per a 2023 study from researchers at the University of Pennsylvania and collaborators (Chen et al., "Dense X Retrieval").
The same study found the gain was largest for rare entities — 17–25% relative improvement — because a fixed-size chunk about a rare topic usually drags in unrelated sentences that dilute the embedding.
For index choice, the independent ANN-Benchmarks project found FAISS-IVF can build an index in about 2 seconds and still serve roughly 655,000 queries per second at 0.7 recall, while graph-based HNSW needs close to 5 hours to build the same scale of index but tends to win on recall-per-query once built — and on some data distributions, HNSW's recall ceiling drops as low as 0.86 regardless of tuning.
Langoedge's own vector-embedding tools (create_qdrant_embedding, create_pinecone_embedding) take a single raw text string as input. The chunking decision happens entirely upstream, in the Text Graph node that calls the tool — the tool has no opinion on chunk size.

Why fixed-size chunking is the default, and why that's a problem

Split every N tokens, usually with some overlap, is the first thing most embedding tutorials show, and it's reasonable as a starting point: it's simple, deterministic, and doesn't require an extra model call. The problem shows up downstream, in retrieval quality, not in the pipeline code.

A fixed-size chunk doesn't know where an idea starts or ends. It can split a sentence in half, bundle three unrelated facts together because they happened to land in the same 500-token window, or — the case that hurts most — surround a specific, rare fact with enough unrelated text that the resulting embedding represents "a bit of everything" rather than the fact a query is actually looking for.

flowchart TD A["📄 Source document"] --> B{"How is it split?"} B -->|"Fixed-size window"| C["🧩 Chunk mixes 2–3 unrelated ideas"] B -->|"Proposition-level"| D["🎯 Chunk is one self-contained fact"] C --> E["Embedding averages the mixed signal"] D --> F["Embedding matches the query precisely"] E --> G["🔍 Retrieval misses or ranks low"] F --> H["🔍 Retrieval finds it"] style A fill:#2a78d6,stroke:#2a78d6,color:#fff style B fill:#4a3aa7,stroke:#4a3aa7,color:#fff style C fill:#e34948,stroke:#e34948,color:#fff style D fill:#1baf7a,stroke:#1baf7a,color:#fff style E fill:#e34948,stroke:#e34948,color:#fff style F fill:#1baf7a,stroke:#1baf7a,color:#fff style G fill:#e34948,stroke:#e34948,color:#fff style H fill:#1baf7a,stroke:#1baf7a,color:#fff

The split happens once, before embedding. Everything downstream — ranking, retrieval, the answer the LLM eventually gives — inherits whatever that split got wrong.

What the research says to do instead

Two findings, from two different research groups, point the same direction: smaller, more self-contained units beat bigger, looser ones.

Propositions. Chen et al.'s "Dense X Retrieval" paper tested retrieval at three granularities — passage, sentence, and "proposition" (an LLM-generated, atomic, self-contained statement of a single fact) — across six different retrievers (SimCSE, Contriever, DPR, ANCE, TAS-B, GTR). Proposition-level retrieval won on every one, with the biggest gains on questions about less-common entities, where a fixed-size chunk is most likely to dilute the one sentence that actually answers the question. The paper's own sweet spot for chunk size, when propositions aren't an option, was 100–200 words — roughly 10 propositions or 5 sentences, not the 400–500 token chunks many pipelines default to.

Late chunking. A separate 2024 paper from Jina AI, "Late Chunking: Contextual Chunk Embeddings", attacks the same problem from the embedding side rather than the splitting side. Instead of chunking text and then embedding each chunk in isolation, late chunking embeds the entire document first with a long-context model, and only pools the token-level embeddings into chunks afterward. Each chunk's embedding then carries context from the rest of the document, which the authors report produces better retrieval than embedding pre-split chunks independently — without retraining the embedding model.

Neither finding says "always do this." Proposition extraction and late chunking both cost an extra model call or a long-context embedding model you may not already have wired up. The honest takeaway is narrower: fixed-size chunking is a reasonable default for a first pass, and if retrieval quality is the thing actually failing in your pipeline, the fix usually isn't a bigger vector database — it's revisiting how the text was split before it ever got embedded.

Index choice: the tradeoff tutorials skip

Chunking decides what gets embedded. Indexing decides how fast you can find the right embedding later, and here the tradeoff is build time versus query time, not "which index is best."

The ANN-Benchmarks project (Aumüller, Bernhardsson and Faithfull) ran the same nearest-neighbor search across multiple algorithms and datasets and published the raw recall/throughput curves rather than a single verdict. Two results matter for a team choosing an index:

Build time is real and very different between index types. FAISS-IVF (an inverted-file, cluster-based index) built an index in about 2 seconds in their benchmark and still served roughly 655,000 queries per second at 0.7 recall. HNSW (a graph-based index) took close to 5 hours to build an index at the same scale. If your embeddings change constantly — new documents arriving continuously — that build-time gap is the first thing to check, not recall.
No index universally wins. Graph-based methods like HNSW generally beat inverted-file and tree-based methods on recall-per-query once built, but the paper also found HNSW's recall ceiling drops as low as 0.86 on certain data distributions (their Rand-Euclidean benchmark), where a tree-based method (Annoy) did better regardless of tuning. The "best" index depends on your actual vector distribution, not a leaderboard.

flowchart TD A["How often does your embedded data change?"] --> B{"Mostly static, rebuilt rarely"} A --> C{"Growing or updating continuously"} B --> D["Graph-based index (e.g. HNSW): slower to build, strong recall per query once built"] C --> E["Inverted-file index (e.g. IVF): seconds to build, very high throughput, test recall on your own data"] D --> F["✅ Benchmark both on your real embeddings before committing"] E --> F style A fill:#2a78d6,stroke:#2a78d6,color:#fff style B fill:#4a3aa7,stroke:#4a3aa7,color:#fff style C fill:#4a3aa7,stroke:#4a3aa7,color:#fff style D fill:#eb6834,stroke:#eb6834,color:#fff style E fill:#eb6834,stroke:#eb6834,color:#fff style F fill:#1baf7a,stroke:#1baf7a,color:#fff

Neither index is "better." The ANN-Benchmarks data says to test recall on your own vectors before picking one, because the gap between algorithms is distribution-dependent, not fixed.

Where chunking actually happens in a Langoedge Text Graph

Langoedge's native embedding tools — create_qdrant_embedding and create_pinecone_embedding — each take exactly one required text parameter plus a target collection or index name. There's no chunk-size setting on the tool itself, by design: the tool's job is "embed this string and write it," not "decide how this document should be split." That decision belongs to the node (or nodes) in your Text Graph that run before the embedding tool is called.

Concretely, a Text Graph ingesting a knowledge-base article for RAG retrieval looks like: a node that splits the raw document (by fixed window, by sentence, or by proposition, per the research above) → a loop or fan-out that calls create_pinecone_embedding once per chunk, each insert carrying the same index_name → a separate retrieval-time Text Graph tool that queries that index against an incoming question. Getting the chunking step right happens entirely in your own graph logic, before the first embedding call — which is also why a vague "it's not retrieving the right answer" complaint about a RAG setup is almost always a chunking problem, not an embedding-model or index problem.

FAQ

What's the difference between chunking and indexing in an embedding pipeline?

Chunking is splitting source text into pieces before anything gets embedded — it decides what each vector represents. Indexing is how those vectors get stored and searched afterward — it decides how fast and how accurately a query can find the right vector. They're separate decisions with separate tradeoffs, and a lot of retrieval problems get misdiagnosed as an indexing issue when the real cause is how the text was chunked.

Is semantic or proposition-based chunking always better than fixed-size chunking?

It wins on the benchmarks in the Dense X Retrieval paper, but it costs an extra LLM call per document to extract propositions, which fixed-size splitting doesn't. For a first pipeline, or for homogeneous short documents where little gets diluted by fixed windows, fixed-size chunking with reasonable overlap is a defensible starting point. It's worth revisiting only once retrieval quality is actually measured and found wanting.

Should I use HNSW or IVF for my vector index?

Neither is categorically better. ANN-Benchmarks' own data shows IVF-style indexes build in seconds and handle very high query throughput, while graph-based indexes like HNSW generally win on recall per query but can take hours to build and occasionally underperform on specific data distributions. Benchmark recall on your own embeddings before choosing, rather than picking the index that wins on someone else's dataset.

Does Langoedge's embedding tooling handle chunking automatically?

No. create_qdrant_embedding and create_pinecone_embedding each embed one text string you provide; chunking is decided by the Text Graph logic that runs before the tool is called, not by the tool itself. That keeps the chunking strategy fully under your control rather than hidden inside a managed step.

How big should a chunk be?

There's no single right number, but the Dense X Retrieval paper's own optimum — when using plain passages rather than LLM-extracted propositions — was 100 to 200 words, noticeably smaller than the 400–500 token chunks common in many RAG tutorials. Treat any fixed number as a starting point to test against your own retrieval queries, not a settled rule.

Sources

About the author. Arnab Chakraborty is the founder of Langoedge, a platform for building production voice and text AI agents with LangGraph. This post was substantially rewritten on 2026-10-04 as part of a weekly content audit — the original November 2025 version used a fictional engineering-team narrative with unsourced performance figures that didn't reflect any real benchmark or Langoedge product behavior.