Edge AI

Edge AI means an AI model runs on the device itself, such as a phone, laptop, camera or machine, instead of on a server in the cloud. The data stays on the device, answers come without a trip over the internet, it works offline and there is no fee per request. Face ID on an iPhone works this way. The limit: a device only fits a small model.

Where does the name come from?

From edge computing: processing data close to where it is produced, at the edge of the network, instead of in a central data centre (Wikipedia). Edge AI applies that to inference, the moment a trained model does its work. Face ID is a clear case: a part of the iPhone's Neural Engine compares your face with the stored data, and that face data never leaves the device (Apple).

Which devices run AI models?

• Phones. In December 2023 Google called the Pixel 8 Pro the first smartphone built to run Gemini Nano, for summaries in the Recorder app and suggested replies in the Gboard keyboard. Apple Intelligence runs an on-device model of about 3 billion parameters (Apple, June 2025).
• Laptops. Microsoft's Copilot+ PCs (May 2024) have an NPU (neural processing unit), a chip for AI calculations, capable of more than 40 trillion operations per second.
• Machines and sensors. NVIDIA's Jetson modules bring AI to robots and machines, and TensorFlow Lite for Microcontrollers runs models on chips with very little memory.

How does a model fit on a device?

A large language model in a data centre does not fit on a phone, so edge models are smaller and compressed:

• Fewer parameters. Gemini Nano came in versions of 1.8 and 3.25 billion parameters (Gemini report); Meta's Llama 3.2 (September 2024) in 1 and 3 billion, with quantized versions for on-device use (model card).
• Quantization: the weights are stored with fewer bits. Gemini Nano is 4-bit quantized; Apple compressed its on-device model to 2 bits per weight.
• Distillation: a small model is trained to imitate a large one (Hinton et al., 2015). Gemini Nano was distilled from larger Gemini models, and Apple also trained its on-device model with distillation.

The Phi-3 report (Microsoft, April 2024) shows what that yields: phi-3-mini, with 3.8 billion parameters, takes about 1.8 GB after 4-bit quantization and ran fully offline on an iPhone 14 at more than 12 tokens per second.

Edge or cloud?

Many products combine the two. Apple Intelligence works on the device and, for heavier requests, draws on larger server models through Private Cloud Compute (Apple). Microsoft pairs small models on the Copilot+ PC with large models in its Azure cloud.

The small model is the weak point. The Phi-3 authors write that phi-3-mini lacks the capacity to store much factual knowledge, visible in low scores on the TriviaQA quiz, and suggest letting it use a search engine.

Example

Suppose a heating installer gives its technicians an iPhone app that turns spoken notes into a short job report. Through Apple's Foundation Models framework, the app can use the on-device model behind Apple Intelligence: the feature works offline, in a basement without signal too, and costs nothing per request (Apple, September 2026). The customer's details stay on the phone. For a question outside the notes, such as a part number, the app looks it up in the company's own system instead of relying on the model.