Multimodal RAG

Multimodal RAG is a retrieval-augmented-generation variant in which the indexed corpus and the retrieval step span multiple modalities — text, images, tables, figures, audio, or video — not just plain text. Implementations take one of two shapes. The first uses a single multimodal embedding model (such as CLIP-style encoders or a vision-language embedding) so text queries can retrieve images or image queries can retrieve text in a shared vector space. The second indexes each modality with its own specialist retriever and merges results at query time. Multimodal RAG is most useful when source material is inherently mixed — product catalogs with photos, financial reports with charts, clinical data with scans — and when pure text extraction would lose information. The generator is usually a vision-language model so it can actually consume the retrieved non-text payloads.