Multimodal

Multimodal AI refers to a machine learning system capable of processing, understanding, and generating information from multiple types of data—known as modalities—simultaneously. While unimodal systems are limited to a single data type, such as text or audio, multimodal models integrate diverse inputs to create a more comprehensive representation of the data.

Common modalities include text, images, audio, video, and sensor data (such as LiDAR or thermal imaging). These systems typically use a shared embedding space, where different data types are converted into numerical vectors that allow the model to find correlations between disparate inputs, such as linking the word "apple" to an image of a red fruit.

This concept differs from multi-task learning, which focuses on performing different functions, by focusing instead on the variety of the input and output formats. It is the foundational technology behind Large Multimodal Models (LMMs), which expand upon Large Language Models (LLMs) by incorporating visual or auditory processing capabilities.

A concrete example is a visual question-answering (VQA) system. In this application, the model accepts an image and a text-based query as simultaneous inputs and generates a text response based on the visual content of the image.