Multimodal AI

Multimodal AI is AI that works with more than one kind of information: besides text, also photos, speech, sound or video. You can show it a photo of a broken part and ask which one to order, or talk to it instead of typing. For a business that means an assistant can read a scanned invoice, look at a photo a customer sends, or answer a spoken question.

What does multimodal mean?

Each kind of information is called a modality: text, images, audio and video. A model is multimodal when it takes in or produces more than one of them. Many chat assistants accept text and images and answer in text; such a model is a vision-language model. Others also listen and speak (speech AI), or make images and video. Google's Gemini Omni, for example, makes video from any mix of text, images, video and voice (May 2026).

How does it work?

There are three common designs, from loosely to tightly connected:

1. A chain of separate models. Before May 2024, ChatGPT's Voice Mode was a pipeline of three models: one turned speech into text, GPT-3.5 or GPT-4 wrote a text answer, and a third read it aloud. An answer took 2.8 seconds (GPT-3.5) or 5.4 seconds (GPT-4) on average, and the language model could not hear tone, several speakers or background noise (OpenAI).
2. An encoder per modality, connected to a language model. A vision encoder turns an image into tokens that the language model reads along with the text. DeepMind's Flamingo (2022) linked a pretrained vision model and a pretrained language model in this way.
3. One model trained on all modalities together. Google calls its Gemini models (December 2023) natively multimodal: trained jointly on text, images, audio and video, with video read as a series of frames and audio taken in directly, which keeps nuances that get lost when speech is first turned into text. OpenAI's GPT-4o (May 2024) is a single network trained end to end on text, images and audio, and answers speech in as little as 232 milliseconds, 320 on average.

An image also costs tokens. In September 2026, Anthropic's documentation counts images in blocks of 28 by 28 pixels, so a photo of 1,000 by 1,000 pixels counts as 1,296 tokens, priced like text.

Where does it go wrong?

• It sees things that are not there, or misses small details. The entry on vision-language models gives the measurements.
• Instructions can hide in an image. OpenAI's GPT-4V system card (September 2023) describes jailbreaks placed in images, such as a screenshot of written instructions, which filters that only read text cannot catch. OWASP lists the same trick under prompt injection: an attacker hides instructions in an image that comes with harmless text.

Example

From March 2023, Be My Eyes and OpenAI built Be My AI, which uses GPT-4V to describe photos that blind and low-vision people take with their phones. By September 2023, 16,000 beta testers requested 25,000 descriptions a day on average, according to the system card. Testers also reported confident mistakes: "It very confidently told me there was an item on a menu that was in fact not there." Be My Eyes tells users not to rely on it for prescriptions or allergens, and lets them leave the AI and reach a human volunteer straight away.