Retrieval-Augmented Generation (RAG) systems power many modern AI search and question answering applications.
However, traditional RAG pipelines struggle when dealing with visually rich documents such as research papers, financial reports, technical manuals, and enterprise PDFs.

These documents contain:

• tables
• figures
• diagrams
• structured layouts
• typography information

Traditional text-based pipelines often lose this information.

A new approach called ColPali proposes a different idea: instead of extracting text first, embed document pages directly as images using Vision Language Models (VLMs).

This allows retrieval systems to preserve both visual and textual semantics.

The Problem with Traditional RAG Pipelines

Most RAG pipelines ingest documents using a multi-stage preprocessing workflow.

Typical ingestion pipelines include:

• Running OCR on scanned documents
• Performing document layout detection
• Segmenting pages into paragraphs, titles, figures, and tables
• Reconstructing reading order
• Captioning figures and tables
• Chunking text passages
• Generating embeddings using models such as BGE-M3
• Storing embeddings in a vector database

While tools such as Unstructured and Surya help automate these steps, several limitations remain:

• indexing pipelines can be slow
• errors propagate across stages
• visual layout information is often lost
• complex documents are difficult to represent accurately

For organizations working with large document collections, these limitations reduce retrieval performance.

A Different Idea: Embed the Page Directly

Instead of reconstructing documents into text, ColPali embeds the entire document page as an image.

This preserves:

• layout structure
• tables
• diagrams
• visual hierarchy
• typography

The method is enabled by advances in Vision Language Models, specifically PaliGemma, developed by the Google Zürich team.

ColPali also uses late interaction retrieval, inspired by the ColBERT architecture.

Model Architecture

The ColPali architecture consists of two main phases.

Indexing Phase

During indexing:

1. Document pages are converted into images
2. Images are split into patches
3. Patches are processed using a Vision Transformer (SigLIP-So400M)
4. Patch embeddings are projected into a language model space (Gemma 2B)
5. Final embeddings are compressed to 128 dimensional vectors

The system stores a multi-vector representation for each document page.

Query Phase

When a user submits a query:

1. The query is embedded using the language model
2. Token embeddings are generated
3. A late interaction scoring mechanism matches query tokens to document patches

Late Interaction Retrieval

Late interaction works differently from traditional dense retrieval.

Instead of compressing an entire passage into a single vector:

• each query token interacts with document patch embeddings
• the best matching patch is selected
• scores are aggregated

This allows richer matching between queries and document regions while maintaining efficient retrieval.

Production Architecture with Qdrant

A production-ready system requires more than just the model.

A typical architecture may include:

• Document ingestion service
• ColPali embedding service
• Qdrant vector database
• Retrieval service
• LLM inference layer
• API layer

Why Use Qdrant Vector Database

ColPali produces multi-vector embeddings, which require efficient similarity search.

Qdrant is well suited for this because it provides:

• high performance Approximate Nearest Neighbor search
• support for HNSW graph indexing
• scalable vector storage
• metadata payload support
• filtering capabilities

HNSW graph indexing enables efficient retrieval even when storing millions of embeddings.

This makes Qdrant a strong choice for production-scale RAG systems.

Metadata Filtering

Enterprise search often requires filtering results based on structured attributes.

Examples include:

• department
• document type
• publication year
• language
• access permissions

Qdrant supports storing payload metadata alongside embeddings.

Example filter query: department = “research” AND year > 2022 AND language = “english”

This allows combining:

• semantic vector search
• structured filtering

The result is hybrid retrieval, which is essential for enterprise knowledge systems.

Adding an LLM Layer: The Hidden Bottleneck

Even though ColPali improves document retrieval, most RAG systems still rely on an LLM to generate final answers.

Adding an LLM layer introduces new infrastructure challenges.

Running modern LLMs requires:

• GPU resources
• high memory bandwidth
• efficient batching

If many users query the system simultaneously, the LLM inference service may become a bottleneck.

In poorly designed systems, it can even become a single point of failure (SPOF).

Improving Throughput with vLLM

To handle high query throughput, many production systems use vLLM.

vLLM improves inference efficiency using:

• Paged attention
• efficient KV cache management
• continuous batching

Compared to traditional inference frameworks, vLLM can significantly increase throughput.

Other strategies include:

• horizontal scaling of inference nodes
• request batching
• response caching
• semantic caching layers

Accuracy vs Cost Tradeoff

While ColPali improves retrieval quality, it introduces new computational costs.

Sources of cost include:

• Vision language model encoding
• multi-vector storage
• GPU inference for LLMs
• additional infrastructure complexity

Organizations must balance:

• retrieval accuracy
• system latency
• infrastructure cost
• compute requirements

In many cases, the optimal system is not the most accurate one, but the one that achieves the best balance between performance and cost.

ColPali vs Traditional RAG vs Multimodal RAG

ColPali particularly excels when working with visually complex documents such as:

• research papers
• technical documentation
• infographics
• financial reports

Why This Matters for RAG Systems

Most current RAG pipelines are designed around text-only retrieval.

However, real-world documents contain rich visual information.

By embedding document pages directly, ColPali enables systems to understand:

• layout
• structure
• diagrams
• tables

This opens the door to more accurate document retrieval systems.

Final Thoughts

ColPali represents an important shift in how retrieval systems process documents.

Instead of forcing documents into purely textual formats, it treats them as visual artifacts.

As Vision Language Models continue to improve, approaches like ColPali may become core components of future retrieval architectures.

However, deploying such systems requires careful consideration of:

• infrastructure design
• compute cost
• scalability
• operational reliability

For AI engineers building document-heavy RAG systems, the challenge is not just building better models, but designing efficient, scalable retrieval architectures.

References

Paper: https://arxiv.org/abs/2407.01449
Model: https://huggingface.co/vidore/colpali
Benchmark: https://huggingface.co/vidore
Benchmark Code: https://github.com/illuin-tech/vidore-benchmark
Training Code: https://github.com/ManuelFay/colpali