What Is Retrieval-Augmented Generation (RAG)?
An employee asks the company's AI assistant whether a hotel expense needs approval. The assistant says no. But the travel policy was updated last week, and the new rules say otherwise.
The answer sounds reasonable. It is also wrong, and the employee has no way of knowing unless they check the policy themselves.
This is a common problem with large language models (LLMs). They don't automatically have access to internal documents or information published after their training. When the necessary facts are missing, a model may produce a plausible answer anyway, a problem known as an artificial intelligence (AI) hallucination.
Retrieval-augmented generation addresses this by giving the model relevant information before it answers.
In a RAG-based policy assistant, the employee's question triggers a search of current company documents. The system retrieves the relevant passages and includes them in the model's prompt. The answer can then cite the policy it used, allowing the employee to verify the decision.
Using external evidence to support an answer is known as grounding.
The RAG framework was introduced by Patrick Lewis and colleagues in their 2020 research paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.
RAG still depends on reliable source information. Our article on how AI is reshaping enterprise data pipelines explores the engineering work behind it.
RAG vs Fine-Tuning
RAG retrieves information when someone asks a question. Fine-tuning changes a model through additional training, generally to improve how it performs a particular task.
Freshness
Uses updated content once it's indexed.
New knowledge generally requires further training.
Cost
Requires retrieval infrastructure and ongoing content maintenance.
Involves training costs and potentially further tuning.
Access control
Can restrict retrieved documents according to user permissions.
Doesn't inherently enforce document-level permissions.
Traceability
Can provide citations to source documents.
Doesn't inherently identify the source of an answer.
Best use
Questions involving current or private information.
Improving model performance on specific tasks.
If the travel policy changes, the company can update the document and refresh its search index. RAG can then retrieve the new version without retraining the model.
Fine-tuning may still help the assistant handle a specialized task more consistently. The two approaches can be combined.
How RAG Works in Practice
Before the assistant can answer from company documents, those documents need to be searchable. From there, RAG follows four stages: indexing, retrieval, generation, and evaluation.
1. Indexing
Engineers prepare the source documents, organize their metadata, and divide longer files into smaller passages. This is called chunking.
Those passages can be converted into numerical representations called embeddings. They allow the system to search for similarities in meaning rather than relying entirely on matching words.
A vector database stores and searches the embeddings. Some applications use a hybrid index that combines semantic search with keyword matching, useful when someone asks about an exact policy number or document title.
2. Retrieval
When a question arrives, the system searches for passages likely to contain the answer. Semantic search finds related meaning, while hybrid search also considers keywords. Reranking helps prioritize the most relevant results.
The search must also respect access rights. A document may match the question perfectly and still be off-limits to the employee asking it. Permission checks should happen before restricted content reaches the model.
3. Generation
The application places the retrieved passages alongside the employee's question in a prompt. The language model uses that context to produce its response.
The answer can include citations to the original documents, giving the employee a way to check the evidence. A citation alone, however, doesn't guarantee that the model interpreted the source correctly.
4. Evaluation
Testing needs to establish whether the system found useful information and whether its answer was supported by that information. These measures are called retrieval relevance and answer faithfulness.
Teams can assess them using real questions, reference answers, and supporting documents. Tests should also cover missing information, incorrect citations, and attempts to access restricted content.
Some applications use agents to choose which sources to search or whether another retrieval step is needed. Our article on the decision layer between your agents examines the coordination involved.
Why RAG Matters for Enterprise Leaders
The value of RAG becomes clear when the information behind an answer changes.
A company can revise its travel policy and update the search index without retraining the language model. Employees can then receive answers based on the revised rules, with citations that let them check the source.
That depends on keeping the index current. If an outdated policy remains searchable alongside its replacement, the assistant may retrieve the wrong version. Document quality and maintenance remain part of the job.
Document permissions also need to carry through to retrieval, so restricted information doesn't reach the model. Without those controls, an assistant could disclose material its user isn't authorized to see.
There are operational tradeoffs. Retrieval adds processing time, indexes need maintenance, and even relevant documents can be misinterpreted. RAG can improve the accuracy of company-specific answers, but the application still needs testing before people can rely on it.
Getting Started With RAG
Start with a knowledge domain where employees ask recurring questions and someone owns the answers. An internal policy library is a reasonable candidate because the documents provide a clear reference for checking responses.
Before building the application, inspect the source material. Are the documents current? Do different versions contradict one another? Can the system search them reliably? Who should be allowed to see each document?
Use real user questions to build an evaluation set, recording the expected answers and their supporting sources. Include questions the assistant cannot answer and requests involving restricted information.
A pilot should show whether the system finds the right passages, cites them accurately, and respects permissions. When an answer fails, trace the problem to its source. Outdated documents, poor retrieval, and incorrect interpretation require different fixes.
For help taking RAG into production, see Exadel's AI engineering services.
Key Takeaways: RAG
Retrieval-augmented generation gives language models access to relevant external information when a question is asked. The process covers indexing, retrieval, generation, and evaluation. Unlike fine-tuning, RAG can use updated information without retraining the model, although the two approaches can work together. Reliable results depend on prepared source content, appropriate permissions, and testing that checks both retrieval quality and the accuracy of generated answers.
Frequently Asked Questions About RAG
What is the difference between RAG and fine-tuning?
RAG retrieves external information when a user asks a question, while fine-tuning changes a model through additional training. RAG is useful when answers depend on current documents, private knowledge, or verifiable sources. Fine-tuning is generally better suited to improving performance on specific tasks. Both can be used together when an application needs specialized model behavior and access to updated information.
Does RAG eliminate AI hallucinations?
No. RAG can reduce hallucinations, but it cannot eliminate them. The system might retrieve the wrong document, overlook relevant information, or misinterpret a passage that contains the answer. Citations help users check the evidence but don't guarantee accuracy. Evaluation needs to examine what the system retrieved and whether the response is supported by those sources.
What is a vector database's role in RAG?
A vector database stores and searches numerical representations of content called embeddings. These allow a RAG system to find passages related to a question's meaning, even when the wording differs. Vector databases may also support metadata filtering and work alongside keyword search. They're common in RAG applications, although other search technologies can perform retrieval.
What is agentic RAG?
Agentic RAG allows an AI agent to decide how to retrieve information before answering a question. The agent might choose a data source, revise its search query, or look for more evidence when the first results are insufficient. This can help with complex questions, but each additional decision creates something else that needs to be tested and controlled.
How do you secure RAG on sensitive data?
Secure RAG applications check user permissions before sending retrieved information to the model. They also need content classification, appropriate access controls, and logging to track how information is accessed. Teams should test for unintended disclosure, including attempts to retrieve restricted documents or expose sensitive material through generated answers. Security must cover the full retrieval and generation process.
