Multimodal RAG Systems
Class Notes: Retrieval-Augmented Generation with Text, Tables & Images
Compiled study notes — includes teacher's annotations marked 🎓
- Introduction to RAG and Multimodal RAG
What is RAG (Retrieval-Augmented Generation)?
RAG is a technique that enhances Large Language Models (LLMs) by providing them with relevant context from external knowledge sources before generating responses.
Instead of an AI relying solely on its training data (which might be outdated), RAG allows it to “look up” information from a knowledge base — similar to how you'd search the web before answering a question.
Traditional RAG Workflow
What is Multimodal RAG?
Multimodal RAG extends traditional RAG to handle multiple types of data:
📝 Text (documents, articles)
📊 Tables (structured data with rows/columns)
🖼️ Images (charts, diagrams, photographs)
📈 Graphs and visualizations
Why Multimodal RAG?
Most documents contain a mixture of content — text, tables, and images. Traditional RAG ignores images, losing valuable information. Multimodal RAG captures ALL information types and uses them together for better answers.
🎓 Teacher's Note
Before this section, you should be comfortable with: what an LLM is, what a vector embedding is (a numeric representation of meaning), and basic Python. If any of those are new, pause and look them up — everything below builds on them.
Unimodal vs. Multimodal RAG, side by side: a unimodal system can only answer questions about text it indexed (it would miss a bar chart entirely). A multimodal system 'reads' the chart too, turning it into searchable text via a vision model, so a question like 'which year had the most fires?' can be answered even if that number only appears in an image.
- System Architecture
The pipeline has two phases: an offline INDEXING phase (process the document once) and an online QUERY phase (answer questions using what was indexed).
Phase A — Indexing (runs once per document)
PDF / Document is loaded
EXTRACT elements — text, tables, and images are pulled out separately
SUMMARIZE content — each element gets a short, retrieval-friendly summary
STORE in two places at once (see Two-Store System below)
Phase B — Query (runs every time a user asks a question)
MULTI-VECTOR RETRIEVER searches the vector store using the query's embedding, finds matching summaries, then fetches the corresponding raw content from the document store
CONTENT SEPARATION splits the retrieved content into images (base64) vs. text/tables, using split_image_text_types()
GPT-4o MULTIMODAL GENERATION processes text + tables + images together and writes the final answer