Guides

Retrieval-Augmented Generation (RAG) Guide: Concept, Components, and Evaluation

Retrieval-Augmented Generation (RAG) converts documents and user queries into vector embeddings, then retrieves the documents that are semantically closest to add them to the model’s context. This guide explains the concept, its roles in multimodal applications and agents, the importance of filtering, tracing, and evaluation, and the limitations identified by the sources.

Retrieval-Augmented Generation (RAG) is a method that converts documents and user queries into vector embeddings, then searches for the documents that are semantically closest and adds them to the model’s context before generation. This enables the system to use information outside the model’s training data, such as documents, support records, product details, internal knowledge, and codebase context.

How does RAG work?

The basic idea is to convert content and queries into vector representations, then use semantic search to retrieve the items closest to the query. The results are then passed to a generative model so it can use them to provide context and reasoning.

  • Convert documents and user queries into vector embeddings.
  • Search for the documents that are semantically closest.
  • Add the retrieved results to the model’s context.
  • Generate an answer based on the retrieved context, while recognizing that result quality remains tied to retrieval relevance and data accuracy.

Where is RAG used?

The sources mention using RAG for document search, enterprise-data-based chatbots, knowledge bases, agentic applications, codebase search, and financial research and legal services platforms.

RAG can also be combined with other tools. An agent may use MCP to access a tool or data, follow Skills to apply project-specific instructions, and use retrieval to fetch supporting context. According to the source, these components are not necessarily competing alternatives; they may instead perform complementary roles within a single workflow.

Multimodal and on-device retrieval

Google announced EmbeddingGemma 2, a multimodal embedding model that places text, images, audio, video, and code in a unified representation space. The company says it is intended for building semantic search, retrieval, and RAG applications locally on devices, without sending data to external servers.

  • The model has 740 million parameters and supports a context of up to 8,000 tokens.
  • According to the announcement, it supports up to 5.5 minutes of audio, 29 images, or 58 video frames.
  • Output vectors can be reduced from 768 dimensions to 512, 256, or 128 dimensions. Google says this may reduce storage requirements and memory consumption in vector databases by up to six times.
  • Google mentions using it to search a media library with text or an image, or to identify moments in a video through a text or audio query.
  • The performance and memory figures cited come from Google, and the provided sources do not include independent test results confirming them.

Metadata filtering before search

AWS added pre-filtering of metadata to Amazon S3 Vectors before similarity search. In ENHANCED mode, conditions such as tenant, category, status, or date are applied first, after which vector search is performed within the matching vector set.

  • AWS says the feature may provide up to five times more matching vectors for queries with selective filters.
  • Each vector can carry up to 2 KB of filterable metadata.
  • A single query supports up to 100 filter conditions according to the article.
  • Filtering includes equality, Boolean logic, numeric comparisons, set membership, existence checks, and the startsWith operator.
  • Existing indexes need to be converted to ENHANCED mode through UpdateIndexMode, and the announcement says that re-ingesting vectors is not required.

This may be important in multitenant systems or situations where search must be limited to a specific set of documents. However, the actual result remains tied to the data type, index configuration, and nature of the filters, and the provided sources do not offer a general guarantee for every application.

RAG in agent applications

In applications that combine retrieval, generation, and tool calls, the error may not be limited to the final text. It may result from selecting the wrong database, retrieving irrelevant documents, or using a source outside the required scope.

Rig, a Rust library mentioned in one of the sources, explains that it supports RAG by converting documents and user queries into vector embeddings and searching for the documents that are semantically closest to add them to the model’s context. The article also mentions integrations with databases and abstractions for vector stores, with the ability to implement the vector-store interface when using a database that is not directly supported.

How do you evaluate a RAG system?

The sources emphasize that checking only the final answer is insufficient. The process that led to it should also be evaluated, including retrieval, tools, context, and generation.

  • Log vector and keyword searches and the retrieved data.
  • Trace the tools called by the agent and the sequence of decisions.
  • Measure answer relevance, completeness, and factual correctness when suitable criteria are available.
  • Use deterministic checks for schema validity, tool arguments, citation identifiers, and authorization enforcement.
  • Use LLM-as-a-judge cautiously for open-ended or ambiguous tasks, because it may produce inconsistent results or fail to detect specific errors.
  • Add human review and domain expertise when building test data and calibrating metrics.
  • Turn user-reported failures into new examples in regression tests.

An Elastic experiment mentions starting with a small set of 20 to 50 test examples, while noting that this figure is not a fixed general standard for every application.

Version and change management

Releasing a RAG application is not limited to model weights. Prompts, the index, the embedding model, the chunking and reranking pipeline, runtime settings, data, and tool contracts may also change.

One article proposes creating a release manifest that stores references to the components tested together, such as the application version, model version, prompt version, index version, embedding version, runtime settings, and evaluation set. Tests should also run across the full path from retrieval through generation, with output validation.

This practice does not guarantee literal reproducibility, because external services may change and generation may remain nondeterministic. Rolling back to an older application is also insufficient if the retrieval index or another path component has changed elsewhere. Therefore, compatible dependencies, data, and retrieval settings should be treated as part of the release.

Limitations to know

  • Answer quality is tied to the relevance and completeness of the retrieved documents, not the generative model alone.
  • Retrieval may fail because the wrong source or database was selected.
  • Performance and memory metrics for models may be provided by the producer and do not constitute independent testing.
  • LLM-as-a-judge is not a final authority and may be affected by evaluator variability or bias.
  • Passing a limited test set does not prove the absence of rare failures or security defects.
  • The provided sources do not identify one universal result or best RAG architecture for all applications.

Conclusion

RAG is a pattern for building systems that add context retrieved from documents or other sources to a model. Its benefits are apparent in semantic search, knowledge bases, agents, and code search, and it can be extended to multimodal or on-device retrieval. However, building a reliable system requires more than creating embeddings and an index: the search scope must be controlled, retrieval and tools must be traced, the complete task must be evaluated, and model, index, embedding, and configuration versions must be managed together.