Multi-tenancy in AI

Multi-tenancy means that one software system serves many customers at once, each with their own data and settings, like tenants who share a building but each have a locked flat. A company that rents out an AI chatbot to many webshops can run all of them on the same servers and the same model. That keeps the price down, as long as no shop can see another shop's data.

What is a tenant?

Microsoft's architecture guide for multitenant solutions defines a tenant as a customer, and separates tenants from users: all users of one company form a single tenant. Its examples include accounting software and other SaaS products, and a platform that several business units of one company share. In a multitenant architecture one instance of the software serves all tenants and gives each its own share, with its own data, configuration and user management (Wikipedia). The alternative is a separate instance per tenant.

What does it look like with AI?

Microsoft's guidance for Azure OpenAI compares four set-ups, from a separate model instance per tenant (high data isolation) to one shared model deployment for all tenants (the simplest, with the least isolation). In between is a shared instance with a separate model deployment per tenant, which lets you set a quota, a content filter, a model version or a fine-tuned model per tenant. The guidance adds a few rules:

• A shared deployment can suffer from noisy neighbours: heavy use by one tenant raises the waiting time for others.
• To bill customers for a shared deployment, the application has to count the tokens each tenant uses.
• Azure OpenAI does not enforce access per model deployment, so the application must check which tenant may use which deployment.
• Documents for RAG need a separate index per tenant, or a tenant filter on every query. Pinecone, for example, advises one namespace per customer in its vector database.
• The application code verifies the tenant ID; it should not rely on the language model to pass tenant information along.

Customised models can share hardware too. A LoRA adapter adapts a base model with a small set of extra weights, so one copy of the base model can serve many adapters. S-LoRA (2023) serves thousands of adapters on a single GPU, and Punica (2023) reports 12 times the throughput of existing serving systems at 2 milliseconds of extra latency per token.

Where does it go wrong?

Shared components can mix up customers. In March 2023, a bug in an open-source library that ChatGPT used for its cache let some users see titles from another active user's chat history. According to OpenAI, the same bug may have made payment details of 1.2% of ChatGPT Plus subscribers who were active during a nine-hour window visible to others: name, email address, payment address, card type, the last four digits and the expiry date. OpenAI then added checks that data from the cache matches the requesting user.

Shared caches can also leak through timing. Prompt caching makes a repeated prompt faster, so if the cache is shared across customers, response times can reveal what others asked. Gu et al. (2025) found caches shared across all users at seven API providers, among them OpenAI's embedding API. After they reported this in October 2024, at least five providers made changes, such as no longer sharing caches between organisations. In September 2026, Anthropic's documentation states that different organisations never share caches.

Example

Suppose a software company offers a support chatbot to many webshops. Each shop's help articles go into its own namespace in the vector database. Every request carries the shop's ID from the login, and the code, not the model, picks the namespace to search. The tokens of each request are stored per shop for the monthly invoice, and each shop has a quota, so that a sale at one shop does not slow down the chatbots of the others.