AI orchestration is the coordination of several AI steps into one working process. A customer's question to a webshop may need a model that recognises what the customer wants, a search in the help centre, a model that writes the answer and a check before it is sent. The orchestration layer decides which step runs when, passes the information along and deals with errors.
Which patterns are there?
Anthropic's Building effective agents (December 2024) describes the patterns it saw in production, from fixed workflows to agents:
- Prompt chaining: each step processes the output of the previous one, with checks in between (prompt chaining).
- Routing: classify the input and send it to a specialised follow-up, for example general questions, refund requests and technical support.
- Parallelisation: split a task into independent parts that run at the same time, or run the same task several times and compare the outputs.
- Orchestrator-workers: a central model breaks the task down, delegates the parts to worker models and combines their results.
- Evaluator-optimiser: one model writes a response, another evaluates it and gives feedback, in a loop.
- Agents: the model directs its own steps and tool use until the task is done (AI agent).
Who decides the next step: code or the model?
OpenAI's Agents SDK documentation defines orchestration as the flow of agents in an application: which agents run, in what order, and how the next step is decided. There are two ways. The language model can decide, planning the task, using tools and handing parts to specialised agents. Or your code fixes the flow, which makes speed, cost and performance more predictable. The two can be combined. LangGraph, for example, lets you mix hand-coded steps and steps driven by a language model in one graph.
When several agents work together (multi-agent systems), the SDK names two common patterns. With agents as tools, a manager agent keeps the conversation and calls specialists for subtasks. With handoffs, a triage agent passes the conversation to a specialist, which then answers the user directly.
Which frameworks do this?
- LangGraph: a low-level framework and runtime for long-running agents that keep state, with durable execution and human-in-the-loop steps.
- Haystack (deepset): modular pipelines and agent workflows with explicit control over retrieval, routing, memory and generation.
- LlamaIndex Workflows: an event-driven way to control the flow, in which each step receives an event and returns the next one.
- Semantic Kernel (Microsoft): ready-made multi-agent patterns such as sequential, concurrent, handoff and group chat; in September 2026 its documentation still labels these features experimental.
Where does it go wrong?
- Every step costs time and money. Anthropic notes that prompt chaining trades speed for accuracy, and that agentic systems often trade speed and cost for better results.
- Frameworks can hide what happens. According to Anthropic, extra layers of abstraction can obscure the prompts and responses underneath and make them harder to debug; it advises starting with the model's API directly and understanding the code of any framework you use. OpenAI's documentation adds that you should monitor the application to see where things go wrong, for example by tracing every step (AI observability).
- The more freedom the model has, the more errors can compound. Anthropic recommends testing agents extensively in a sandbox, with guardrails.
Example
Suppose a webshop automates its customer service. A classifier decides whether a message is about an order, a return or a product. For product questions, a search step fetches the relevant help articles and a language model writes the answer from them (RAG). A guardrail checks the draft before it goes out. Complaints and anything the classifier is unsure about go to a staff member (human-in-the-loop). The orchestration is the code or configuration that connects these steps and decides what happens when one of them fails.