Beyond the Agent Loop: Building the Infrastructure Behind an AI Agent
Orchestration often gets most of the attention when building AI agents, but it’s only one part of a much larger system.
Think of orchestration as the brain of an agent. The brain is essential, but it needs a body to function. Similarly, an agent needs supporting infrastructure to handle requests, manage deployments, maintain execution state, and remain observable in production.
In this post, I’ll walk through how we designed our agent infrastructure, the challenges we encountered, and the trade-offs we made to keep it scalable, reliable, and easy to evolve.
Written by Heng Cui, Mingyong Ma, reviewed by Mingyong Ma
Separate What Needs to Scale Differently
Initially, we kept things simple. Each pod ran a copy of the agent runtime, along with the Slack listener and scheduler. Incoming requests were routed to one of these pods, where the entire workflow—from orchestration to tool execution—would take place.
This worked well for our early use cases, but as we introduced more complex agent workflows, we started running into several problems:
- Noisy Neighbor Problem: Resource-intensive scheduled jobs and latency-sensitive agent requests shared the same compute resources. A single heavy task could consume significant CPU or memory, degrading the performance of unrelated requests.
- Coupled Scaling: Scaling the backend to handle more agent requests also meant creating additional Slack listeners and scheduler loops, even though neither needed to scale with request volume.
- Coupled Lifecycle: A restart or deployment of the agent backend could also interrupt Slack connections and background scheduling, making it harder to manage each component independently.
Rather than introducing micro-services for every part of the agent, we decided to separate workloads based on their responsibilities and scaling requirements.
Slack: Decouple Connection Management from Agent Execution
Slack uses Socket Mode to deliver events over WebSocket connections. Initially, the listener lived alongside the agent runtime, so scaling the backend also created additional socket connections.
We extracted the listener into a lightweight Socket Connector service. Its responsibility is simple: maintain the Slack connection, receive events, and forward requests to the Agent Backend over HTTP.
This lets us scale the backend based on request volume without changing how we manage Slack connections. We currently keep one connector replica per Slack app, although Slack itself supports multiple concurrent Socket Mode connections.
Scheduler: Isolate Background Scheduling from Interactive Requests
The scheduler has a different lifecycle from user-facing requests. It runs continuously, periodically checking for pending jobs and triggering scheduled agent executions.
Running this loop inside every backend pod introduced unnecessary resource contention and meant that adding backend replicas could also create additional scheduler instances.
We moved the scheduler into its own Kubernetes deployment. It now periodically polls the backend and triggers scheduled runs through HTTP, while the Agent Backend remains responsible for the actual orchestration and tool execution.
This gives the scheduling loop its own resource limits and restart lifecycle, without tying its replica count to backend traffic.
Keep Agent Execution in One Place
Both the Slack Connector and the Web UI serve as entry points rather than separate agent runtimes. The Web UI and its BFF run together in a dedicated deployment, with the BFF proxying requests to the Agent Backend.
Regardless of where a request originates—Slack, the browser, or the scheduler—it eventually reaches the same Agent Backend.
The goal was not to distribute every piece of agent logic across microservices. It was to separate connection management, scheduling, and request execution so each could scale and operate according to its own requirements.
Making Deployment Repeatable
As our agent evolved, we were frequently updating orchestration logic, deployment configurations, and domain knowledge. We wanted these changes to evolve independently without turning every update into a full application deployment.
We first separated application code from deployment configuration using two repositories:
- Application Repository: Contains the agent runtime, orchestration, and tools. CI builds, scans, and publishes container images.
- Deployment Repository: Manages Helm configurations and GitOps, controlling image versions, replicas, resources, and environment-specific settings. Changes are verified in staging before being promoted to production.
However, separating code from configuration wasn’t enough.
With skill-based orchestration, much of our agent’s domain knowledge and operational procedures now live in SKILL.md files. As we introduced Recursive Self-Improvement (RSI) and skill evolution, these instructions began changing much more frequently than the underlying code.
Rebuilding and redeploying the backend for every skill update would slow down this iteration cycle. Instead, we moved skills into PostgreSQL, allowing us to update them independently while maintaining version history to track how our knowledge evolves.
We ended up with three independent lifecycles:
- Code defines how the agent executes.
- Configuration defines how the agent is deployed and its secrets.
- Knowledge defines how the agent approaches tasks.
By separating these layers, we can evolve agent behavior without constantly rebuilding the infrastructure underneath it.
Isolating Model-Specific Logic
We also introduced a lightweight provider abstraction between the agent runtime and our LLM gateway.
Different models come with different authentication, streaming, tool-calling, and usage-reporting behaviors. Rather than handling these differences throughout the agent runtime, we keep them behind a common interface.
This allows us to change model configurations without modifying the orchestration logic, while keeping provider-specific integration changes isolated from Slack, Web UI, and other infrastructure components.
Supporting Tracing and State Recovery
Unlike traditional API requests, agent workflows can span multiple LLM turns, tool calls, and skill transitions. When execution is interrupted, we need to know not just what the user asked, but what the agent has already completed and where it left off.
To support this, we use PostgreSQL to persist execution state.
Rather than waiting until the entire workflow finishes, we incrementally append completed model turns and tool results to our message history, along with tool-call identifiers and execution metadata. When a conversation resumes, the backend reconstructs its context from this history, handles incomplete tool calls, and restores previously loaded skills and tool availability.
The same execution events also power a live trace graph, while structured logs capture tool activity and interruptions. For evaluation runs, we additionally export step-level traces to Langfuse.
As conversations grow, however, retaining every message becomes increasingly expensive. Our next step is checkpoint compaction: merging older execution history into a persisted summary of completed work, remaining tasks, and relevant state, while keeping recent messages in full detail.
What’s Next: Continuous Evaluation
One area we’re still working on is continuous evaluation. We previously used the free version of Langfuse for tracing and evaluation, but with LangSmith now available for Adobe, we’re planning to migrate our evaluation workflows and run them in a dedicated Kubernetes deployment.
Similar to the scheduler, evaluation has its own lifecycle and resource requirements. A separate evaluation worker would periodically run test suites covering skill selection, tool execution, and answer correctness, while tracking latency, token usage, and experiment results in LangSmith.
This also lays the foundation for our next step: Recursive Self-Improvement (RSI), where evaluation results can guide how our agents and skills evolve over time.