Embedding Models vs LLMs: Understand the Two Engines Behind RAG

1. Start with Two Different Jobs

Many new AI developers treat an embedding model and a large language model as interchangeable because both process text. They are not interchangeable. Their outputs, responsibilities, cost profiles, and places in an application are different.

2. See the Complete RAG Flow

Retrieval-augmented generation, usually shortened to RAG, connects retrieval and generation. The application prepares knowledge before a question arrives, then performs a smaller runtime pipeline for each question.

Indexing time

Approved documents
  -> extract useful text
  -> split text into meaningful chunks
  -> create an embedding for each chunk
  -> store vector + text + metadata

Question time

User question
  -> create question embedding
  -> compare it with stored chunk embeddings
  -> retrieve the best supported context
  -> send instructions + context + question to the LLM
  -> generate an answer with source references

The vector store may be a dedicated system such as Qdrant, Milvus, Pinecone, Weaviate, Chroma, or FAISS-based infrastructure. A small application can also compare vectors from a structured local store when its scale and latency requirements allow it.

3. What Is an Embedding?

An embedding is a numeric representation produced by a model. The output is a vector, which is an ordered array of numbers. Its length is determined by the selected model.

Text:
"My vehicle stopped working."

Illustrative vector:
[0.021, -0.184, 0.733, 0.092, ...]

The individual numbers are not human-readable labels such as vehicle or repair. Meaning is distributed across the vector. The useful property is geometric: text with related meaning tends to occupy nearby regions in that model's vector space.

For example, My car is broken and My vehicle stopped working use different words but express similar intent. A retrieval-oriented embedding model should place them closer than an unrelated sentence about quarterly finance results.

Dimensions are model-specific

Some models return hundreds of values and others return more. More dimensions do not automatically mean better retrieval. Language coverage, training objective, context length, domain fit, latency, memory, and evaluation results all matter.

4. What Happens Inside an Embedding Model?

The exact architecture varies, but a transformer-based text embedding pipeline can be understood through five stages.

1. Tokenization: text is divided into model-specific tokens. Tokens may be complete words, parts of words, punctuation, or other learned units.
2. Token IDs: each token is mapped to an integer in the tokenizer's vocabulary. Example IDs in diagrams are illustrative; real IDs depend on the tokenizer.
3. Contextual representation: transformer layers use attention and learned transformations so each token representation reflects surrounding context.
4. Pooling: token-level representations are combined into one fixed-length vector. The method may use a designated token, mean pooling, or another model-specific strategy.
5. Optional normalization: some providers return unit-length vectors, which simplifies cosine-based comparison. Follow the model or provider documentation.

Context is why the model can represent the word apple differently in Apple released a device and I ate an apple. The surrounding sequence changes the contextual token representations and therefore the final passage vector.

The embedding does not store a readable miniature copy of the sentence. It is a learned representation optimized for tasks such as retrieval, clustering, classification, recommendation, or semantic similarity.

5. How Semantic Search Uses Vectors

At query time, the application embeds the question and compares the question vector with stored document vectors. A similarity function produces a score or distance, and the system returns the nearest candidates.

The nearest result is not automatically a good result. Every collection has a nearest item, even for an unrelated question. Production retrieval therefore needs evaluation, metadata filters, confidence handling, and a clear no-answer policy.

Dense meaning is useful but not sufficient

Semantic vectors are good at paraphrases, but exact identifiers, product names, error codes, dates, and regulated terms may benefit from lexical search or metadata matching. A hybrid retriever can combine exact signals with vector similarity before reranking the strongest candidates.

6. How a Generative LLM Produces an Answer

A generative LLM receives a sequence of tokens and predicts a probability distribution for the next token. The selected token is appended to the sequence, and the process repeats until a stopping condition is reached.

Prompt:
"The capital of France is"

Illustrative next-token probabilities:
Paris   0.98
London  0.01
Rome    0.005
...

This example is simplified. A decoding strategy may select the highest-probability token or sample from likely candidates according to settings such as temperature and top-p. The model then predicts again using the expanded sequence.

Where its capability comes from

During training, a language model learns statistical patterns from its training data and training objective. It develops useful representations of language, structure, relationships, and recurring reasoning patterns. The exact data sources, update process, and capabilities differ by model, so they should not be assumed without provider documentation.

Why generation needs grounding

An LLM can produce fluent language even when its information is unsupported, incomplete, or outdated. In RAG, the retrieved passages provide task-specific context. Instructions should tell the model to use that context, cite it where appropriate, and admit when the context does not contain a reliable answer.

7. Embedding Model vs Generative LLM

These are architectural roles, not absolute product categories. Some model families expose multiple capabilities, and tool-using AI systems can orchestrate search. Your application should still keep retrieval evidence and answer generation observable as separate stages.

8. Walk Through a Document Assistant

Suppose an internal support assistant must answer questions from approved database operations documentation.

Before questions arrive

1. Collect the approved SQL operations manual, backup policy, recovery guide, and support notes.
2. Remove navigation noise and split the documents by meaningful headings and procedures.
3. Attach metadata such as source, section, audience, system, and access classification.
4. Generate an embedding for each searchable chunk.
5. Store each vector beside its text, source link, metadata, and parent section.

When the user asks a question

Question:
"Where is the approved SQL Server backup procedure?"

Retrieval:
1. Embed the question.
2. Apply access and system filters.
3. Combine exact terms with vector similarity.
4. Rerank the strongest chunks.
5. Resolve each child chunk to useful parent context.

Generation:
1. Send only approved retrieved context.
2. Ask the LLM to answer from that context.
3. Include source links.
4. Return "not found" when evidence is insufficient.

The assistant should not invent backup steps that are missing from the approved documents. Its value comes from finding the right procedure and presenting it clearly, not from replacing operational authority.

9. Practical Design Rules

Evaluate retrieval separately from generation

First test whether the correct source appears in the candidate set. Then test whether the answer accurately reflects that source. A polished answer cannot repair missing evidence, and excellent retrieval can still be damaged by poor composition.

Keep vectors tied to their model identity

Store the provider, model name, dimensions, normalization behavior, and generation date with the knowledge version. Rebuild vectors when the embedding model or incompatible preprocessing changes.

Chunk around meaning

Prefer headings, procedures, FAQ answers, policy clauses, and coherent topic boundaries. Extremely large chunks dilute retrieval; fragments without enough context make answers difficult to compose. Parent-child retrieval can search focused children and answer from a fuller parent section.

Preserve metadata

Source URL, title, section, entity, audience, date, permissions, and content type can improve filtering, ranking, citation, and administration. A vector should not become detached from the text and governance context that gives it meaning.

Use confidence as a decision, not decoration

Calibrate thresholds against representative questions. High-confidence results can answer normally; uncertain matches can be presented cautiously; weak results should produce a clear no-answer response or human handoff.

10. Common Misunderstandings

11. Final Recap

An embedding model and a generative LLM solve different problems. The embedding model converts text into a comparable semantic representation. The vector search layer uses that representation to retrieve useful evidence. The generative LLM turns instructions, evidence, and the question into a readable response.

Embedding model = represent meaning
Vector search   = retrieve evidence
Generative LLM  = communicate an answer
RAG application = govern and connect all three

Official References

• Ollama embeddings documentation
• Ollama embedding API reference
• Qdrant similarity search documentation
• Qdrant semantic and text search guide
• BAAI BGE large English v1.5 model card