A vision-language model (VLM) is an AI model that takes images as well as text and answers in text. You can give it a photo, a scanned invoice or a chart and ask what is on it, or have it copy the figures into a table. For a business, it can read documents that exist only as images. Its answers still need checking, because it can misread or invent details.
How does it work?
Most VLMs join two parts that were first trained separately. A vision encoder, often a Vision Transformer (ViT), cuts an image into small patches and turns each into a list of numbers, the way a transformer handles words. A connector translates those numbers into visual tokens that a large language model reads together with the text of the question.
LLaVA (Liu et al., 2023) shows the recipe. It used the image encoder of OpenAI's CLIP, which had learned from 400 million image-text pairs which caption belongs to which picture, a single projection matrix as the connector, and the Vicuna language model. Training took two steps: first only the connector, on 595,000 image-caption pairs, then the connector and the language model together, on 158,000 instruction examples that GPT-4 had written.
According to IBM, most VLMs feed the visual information into the language model at a single point. Its Granite 4.0 3B Vision instead feeds it in at several layers, a technique called DeepStack injection.
Which VLMs can you use?
Large commercial models accept images too: OpenAI describes GPT-4 (March 2023) as a model that takes image and text inputs. Open models can run on your own hardware. Google's Gemma 4 models (April 2026) all process images and video, according to Google, and IBM built Granite 4.0 3B Vision (March 2026) specifically to extract tables, chart data and form fields from documents. On VAREX, a set of 1,777 US government forms, IBM measured that it extracted 85.5% of the key-value pairs exactly.
Where does it go wrong?
- It sees things that are not there. In a study by Li et al. (2023), VLMs "mostly suffer from severe object hallucination": their descriptions named objects that were not in the image, especially objects that often appear together with the ones that were.
- It misses fine visual detail. In Vision language models are blind (2024), four leading VLMs, including GPT-4o and Gemini 1.5 Pro, averaged 58.07% on seven tasks that are easy for people, such as counting how often two lines cross. Claude 3.5 Sonnet did best with 77.84%. The information was present in the vision encoder, but the language model failed to put it into words.
Example
Suppose an accounting firm receives supplier invoices as scans. A VLM gets each scan with the instruction to return the supplier, invoice number, date and total as structured output, such as JSON. IBM describes this use for Granite 4.0 3B Vision, together with its Docling tool, which splits multi-page PDFs and crops out tables and figures for the model. Because a VLM can misread a digit, a simple check, such as whether the line items add up to the total, flags the invoices that a person should look at.
Sources
- Liu et al.: Visual Instruction Tuning, the LLaVA paper (2023)arXiv
- IBM Granite: Granite 4.0 3B Vision, compact multimodal intelligence for enterprise documentsRSS
- Dosovitskiy et al.: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020)arXiv
- Radford et al. (OpenAI): Learning Transferable Visual Models From Natural Language Supervision, the CLIP paper (2021)arXiv
- Li et al.: Evaluating Object Hallucination in Large Vision-Language Models (2023)arXiv
- Rahmanzadehgervi et al.: Vision language models are blind (2024)arXiv