Building Multi-Agent Systems That Actually Scale: Lessons from Hermes, LobeHub, and the 2025 AI Agent Explosion

Originally published on tamiz.pro. The AI agent landscape shifted dramatically in 2025. What began as single-agent chat interfaces exploded into multi-agent ecosystems where dozens or hundreds of specialized agents coordinate, debate, and execute complex workflows. Hermes and LobeHub emerged as two distinctive approaches to this problem—neither a toy demo nor an enterprise suite—and their architectural decisions reveal what it actually takes to build systems that scale beyond a handful of concurrent agents. This is not a survey of agent frameworks. It's an engineering post-mortem on the hard problems that appear when you stop treating agents as isolated LLM calls and start thinking about them as networked services. A single agent calling an LLM is a well-understood pattern. You send a prompt, you get a response, you handle latency and token budgets. The mental model is straightforward because the topology is trivial: one request, one agent, one model call. Multi-agent systems introduce three compounding difficulties: Communication complexity: Agents need to exchange messages, share state, and coordinate actions. This is a distributed systems problem dressed in conversational clothing. Orchestration overhead: Deciding which agent runs when, in what order, and under what conditions adds a control plane on top of an already non-deterministic LLM layer. Cost and latency multiplication: Every inter-agent message is a potential LLM call. A 3-agent round-robin with 5 messages each can easily generate 15+ model invocations for a single user request. The naive approach—fire all agents in parallel, let them talk through a shared message bus, aggregate results—works until you try to run it at scale. Then you hit resource contention, unbounded fan-out, and prompt injection attacks that flow through trust boundaries between agents. Hermes and LobeHub solved this differently. Understanding both approaches is the best way to internalize the design space. Hermes approaches multi-agent coordination as a message-passing system with typed interaction protocols. Its key insight is that most agent-to-agent communication follows predictable patterns—subtask delegation, result synthesis, conflict resolution—and these patterns should be explicit, not emergent. ┌─────────────────────────────────────────────────┐ │ User Request │ └──────────────────────┬──────────────────────────┘ ▼ ┌────────────────┐ │ Orchestrator │ ← Static routing table + dynamic load balancing └────────┬───────┘ ▼ ┌──────────────────────────────┐ │ Message Router │ ← Typed message queues per agent group └────────┬─────────────┬───────┘ ▼ ▼ ┌────────────────┐ ┌────────────────┐ │ Worker Pool │ │ Specialist │ │ (parallel) │ │ (sequential) │ └────────┬───────┘ └────────┬───────┘ ▼ ▼ ┌────────────────────────────────────┐ │ Result Aggregator │ ← Deterministic merge + LLM reconciliation └────────────────────────────────────┘ Hermes separates agents into two tiers: Worker agents: Stateless, parallel, focused on narrow subtasks (retrieval, code execution, validation). These are the workhorses. Specialist agents: Stateful, sequential, responsible for reasoning-heavy steps that depend on prior results. The critical design choice is that workers never talk to workers directly. All inter-agent communication flows through the orchestrator, which enforces a directed acyclic communication graph. This prevents the combinatorial explosion of agent-to-agent messaging and makes the execution plan auditable. Hermes introduces a minimal schema for agent messages: // Core message types in Hermes interface AgentMessage { id: string; type: 'delegation' | 'result' | 'conflict' | 'escalation'; sender: AgentId; recipient: AgentId | 'orchestrator'; payload: AgentPayload; contextRef?: string; // Reference to shared context blob ttl: number; // Time-to-live in seconds } interface AgentPayload { task?: string; result?: AgentResult; error?: AgentError; vote?: { agentId: AgentId; confidence: number; reasoning: string }; } interface AgentResult { output: string; tools_used: string[]; tokens_consumed: number; confidence: number; citations?: SourceReference[]; } This might look like unnecessary boilerplate next to free-form agent conversations, but it's the difference between a system you can monitor and debug versus one you cannot. When an agent call fails at 3 AM, you want structured error propagation, not a black-box reasoning trace you have to parse from prose. The cost model is where Hermes gets interesting. Instead of letting agents burn tokens freely, it implements a budget-aware routing layer: class BudgetAwareRouter: def __init__(self, agent_registry, cost_tracker): self.agents = agent_registry self.cost = cost_tracker self.default_budget_per_request = 5000 # tokens self.max_concurrent_agents = 8 async def route(self, user_request, context): plan = self._decompose_task(user_request) cost_estimate = self._estimate_cost(plan) if cost_estimate > self.default_budget_per_request: # Trigger agent compression: merge low-value agents plan = self._compress(plan) return await self._execute_plan(plan) The router estimates token costs before execution by analyzing the task decomposition. If a plan exceeds the budget, it compresses the agent graph—merging redundant specialists or falling back to cheaper models for lower-priority steps. This is not a hard limit; it's a heuristic optimization that prevents runaway costs on complex requests. LobeHub takes a fundamentally different approach. Rather than enforcing a rigid top-down orchestration, it treats agent coordination as a peer-to-peer mesh with gossip-style consensus. ┌─────────┐ │ Agent A │◄──────────────────────────────────┐ └────┬────┘ │ │ gossip │ shared state ┌────▼────┐ ┌─────────────┐ │ Agent B │◄─────────────────────────────│ State Bus │ └────┬────┘ └──────┬──────┘ │ │ ┌────▼────┐ │ │ Agent C │◄─────────────────────────────────────┘ └─────────┘ All agents read/write to a shared state bus. Consensus emerges through voting rounds. LobeHub's design philosophy is that complex problems benefit from diverse, concurrent reasoning rather than sequential decomposition. Multiple agents work on the same problem independently, then reach consensus through a voting mechanism. interface ConsensusRound { roundId: string; question: string; participants: AgentId[]; deadline: number; // Unix timestamp quorumSize: number; strategy: 'majority' | 'weighted' | 'unanimous'; votes: Vote[]; result?: ConsensusResult; } interface Vote { agentId: AgentId; position: string; confidence: number; reasoning: string; timestamp: number; } interface ConsensusResult { agreedPosition: string; confidence: number; dissentingViews: DissentingView[]; roundId: string; converged: boolean; } The consensus mechanism is the heart of LobeHub. When agents disagree—and they will, because LLMs are non-deterministic—the system doesn't default to a simple majority. It runs weighted voting where agents with higher historical accuracy on similar tasks get more influence. class WeightedConsensusEngine: def __init__(self, agent_reputation_store): self.reputation = agent_reputation_store self.confidence_threshold = 0.75 self.max_rounds = 3 async def reach_consensus(self, round: ConsensusRound) -> ConsensusResult: votes = await self.collect_votes(round) if not self._has_quorum(votes, round.quorumSize): return await self.expand_participants(round) weighted = self._apply_weights(votes) agreed = self._find_agreement(weighted) if agreed.confidence < self.confidence_threshold and round.round_num < self.max_rounds: # Trigger a refinement round with targeted follow-up questions return await self.run_refinement_round(round, agreed) return agreed This is essentially a distributed reasoning protocol—borrowing from consensus algorithms like Raft and Paxos, but adapted for probabilistic, non-deterministic participants. The refinement rounds are particularly clever: instead of just re-voting, the system identifies specific disagreement points and asks agents to address them directly. Strengths: Emergent behavior: Complex solutions arise from simple local interactions Resilience: No single point of failure in the orchestration layer Creativity: Divergent thinking paths produce novel solutions Weaknesses: Higher latency: Multiple reasoning rounds add up Cost: More agents, more votes, more tokens Debuggability: Harder to trace why a specific output was produced Convergence guarantees: Non-deterministic inputs mean convergence is probabilistic Three shifts in 2024–2025 made multi-agent systems viable at production scale: Early agent frameworks struggled with parsing LLM outputs reliably. Function calling was inconsistent, JSON extraction failed silently, and error handling was an afterthought. By 2025, the major providers (OpenAI, Anthropic, Google) had matured their structured output APIs to the point where deterministic parsing became tractable. This is the single most important infrastructure development for multi-agent systems—if you can't reliably parse an agent's response, you can't build a protocol around it. 200K+ context windows mean agents can share substantial state without constant round-trips. Hermes leverages this with a shared context blob pattern: instead of re-explaining the problem to every agent, a compressed context representation is passed along, and agents operate on a shared understanding. This reduces both latency and token costs significantly. The early 2024 trend of building "general purpose" AI assistants collapsed under its own weight. Too many agentic loops, too little specialization, too much hallucination. The 2025 winners are systems where each agent has a clearly defined scope, well-specified tool access, and measurable accuracy. This is why both Hermes and LobeHub start with agent specialization as a first principle, not a retrospective cleanup. Despite their architectural differences, Hermes and LobeHub converge on several patterns that appear to be necessary for any production multi-agent system: Every agent needs a stable identity, not just a name. This includes: A unique, versioned agent ID for tracking across sessions A capability manifest declaring what tools and models the agent uses A reputation or accuracy score that decays over time Explicit lifecycle states: idle → assigned → active → completed / failed interface AgentManifest { agentId: string; version: string; capabilities: Capability[]; model: ModelSpec; toolAccess: ToolAccessPolicy; maxContextTokens: number; timeoutMs: number; } interface AgentState { agentId: string; status: 'idle' | 'assigned' | 'active' | 'completed' | 'failed' | 'timeout'; currentTask?: string; sessionHistory: MessageHistory; lastActiveAt: number; errorCount: number; } Both systems implement fallback chains at every layer: Model fallback: If the primary model exceeds latency or cost thresholds, drop to a cheaper model with degraded (but acceptable) quality Agent fallback: If a specialist agent fails or times out, route to a general-purpose agent or a cached result Protocol fallback: If consensus fails to converge, fall back to the highest-confidence individual agent's output with dissenting views attached You cannot debug what you cannot observe. Both systems treat observability as a first-class concern: Tracing: Every agent invocation is traced with a span that captures input, output, tokens, latency, and tool calls Cost tracking: Per-agent, per-request, and per-session cost breakdowns Quality metrics: Confidence scores, consistency checks, and human-in-the-loop feedback loops Agent health dashboards: Error rates, latency percentiles, and reputation scores per agent # Example: Structured logging for agent lifecycle class AgentTelemetry: def __init__(self): self.tracer = OpenTelemetryTracer("multi-agent-system") self.metrics = PrometheusMetrics("agent_system") async def record_agent_call(self, call: AgentCall) -> None: with self.tracer.start_span("agent.invocation", trace_id=call.trace_id) as span: span.set_attribute("agent.id", call.agent_id) span.set_attribute("agent.model", call.model) span.set_attribute("tokens.input", call.input_tokens) span.set_attribute("tokens.output", call.output_tokens) span.set_attribute("latency_ms", call.latency_ms) span.set_attribute("confidence", call.result.confidence) span.set_attribute("cost_cents", call.estimated_cost_cents) if call.error: span.record_exception(call.error) self.metrics.increment("agent.errors", labels={"agent": call.agent_id}) else: self.metrics.histogram("agent.latency", call.latency_ms, labels={"agent": call.agent_id}) Agents must never share mutable state directly. Each agent operates on an immutable snapshot of the context it needs, and any state changes are communicated through explicit messages. This prevents race conditions, makes debugging tractable, and enables replay of agent interactions for quality assurance. Based on post-mortems from both Hermes and LobeHub deployments, these are the failure modes that appear most frequently: The temptation is to spawn an agent for every subtask, every validation step, every edge case. This creates exponential cost and latency. Both systems enforce fan-out caps at the orchestrator level: interface OrchestrationPolicy { maxDepth: number; // Max nesting of agent calls maxFanOut: number; // Max concurrent agents maxTotalAgents: number; // Hard cap per request budgetCapTokens: number; // Token budget budgetCapUSD: number; // Cost cap circuitBreaker: { errorRateThreshold: number; // Trigger if >X% of agents fail timeoutThreshold: number; // Trigger if avg latency >Y ms }; } When agents communicate freely, a malicious user input can propagate through the agent network. If Agent A generates a message to Agent B, and that message contains instructions that Agent B treats as authoritative, you have a prompt injection attack vector. Both systems implement: Input sanitization at every agent boundary Authority scoping: Agents can only modify state they're explicitly authorized to change Separation of instruction and data: Agent messages are parsed and reconstructed, never directly exec LLM outputs are probabilistic. Two identical requests to the same agent can produce different results. This makes testing nearly impossible and debugging frustrating. Mitigation strategies: Seed fixation for reproducibility during debugging (not production) Deterministic wrappers around stochastic operations where possible Diff-based testing: Compare outputs semantically, not token-for-token Regression test suites with large golden datasets As agents accumulate conversation history, the context window fills up quickly. Both systems use aggressive context compression: Summarization: Older conversation turns are summarized, not truncated Relevance filtering: Only context relevant to the current agent's task is included Deduplication: Redundant information across agent contexts is identified and removed Neither Hermes nor LobeHub is universally better. The right architecture depends on your requirements: Criterion Hermes (Top-Down) LobeHub (Peer-to-Peer) Latency sensitivity Better—sequential specialists are fast Worse—consensus rounds add latency Cost predictability Better—bounded execution plans Worse—dynamic agent counts Solution quality Good for well-defined problems Better for creative/open-ended problems Debuggability Excellent—linear traces Poor—emergent behavior Scalability Linear with worker pool size Super-linear with participant count Resilience Orchestrator is a bottleneck True fault tolerance Best for Transactional workflows (support, analytics, coding) Exploratory workflows (research, design, strategy) The most promising systems in 2025 combine both approaches. They use Hermes-style orchestration for the high-level task decomposition and LobeHub-style consensus for the reasoning-critical subtasks. The result is a system that is both fast and creative, bounded and exploratory. A typical hybrid flow: Decompose the user request into a task graph (Hermes-style) Route each node to the appropriate agent type (specialist vs. worker) Execute independent nodes in parallel (Hermes-style) Consensus on dependent nodes where multiple interpretations exist (LobeHub-style) Aggregate results with deterministic merge logic, falling back to LLM reconciliation only when necessary This hybrid approach is becoming the default pattern for production multi-agent systems because it captures the strengths of both paradigms while mitigating their weaknesses. If you're planning to build a multi-agent system in 2025, here's what the evidence from Hermes, LobeHub, and related systems suggests you need: [ ] Define your agent type taxonomy (specialist vs. worker vs. orchestrator) [ ] Implement typed message protocols (don't use raw strings) [ ] Build a deterministic task decomposition engine [ ] Set up structured observability before writing any agent logic [ ] Implement the orchestrator with fan-out controls and circuit breakers [ ] Build the agent registry with capability manifests and reputation tracking [ ] Create the budget-aware routing layer [ ] Implement context compression and shared state management [ ] Add fallback chains at every layer [ ] Implement prompt injection defenses at agent boundaries [ ] Build the consensus protocol if your use case requires divergent reasoning [ ] Create a replay/debug mode for agent interactions [ ] Add agent pooling and connection management [ ] Implement horizontal scaling for the orchestrator [ ] Build automated agent evaluation and A/B testing infrastructure [ ] Deploy guardrails and human-in-the-loop checkpoints for high-stakes operations Q: Do I really need a custom multi-agent framework, or can I use LangGraph or AutoGen? For simple workflows, yes—those frameworks are fine. But they abstract away the scaling concerns that matter in production: cost control, observability, prompt injection defense, and resilience under load. Hermes and LobeHub's approaches show that these concerns require architectural decisions that generic frameworks don't address. If you're building something that needs to run reliably at scale, investing in purpose-built infrastructure pays off. Q: How do I choose between sequential orchestration and peer-to-peer consensus? The rule of thumb: use sequential orchestration when the problem has clear sub-components that can be decomposed and solved independently (data processing, code generation, support ticket resolution). Use consensus-based approaches when the problem requires creative synthesis, trade-off analysis, or when multiple valid answers exist and you need to find the best one (strategy planning, research synthesis, design review). Hybrid approaches get you the best of both worlds. Q: What's the realistic cost per complex multi-agent request? In production systems observed in 2025, a well-optimized multi-agent request with 5–8 agents typically costs $0.50–$3.00 depending on complexity and model choices. Unoptimized systems can easily exceed $10 per request. The key differentiator is whether you're using budget-aware routing and context compression. If your costs are higher, you're likely spawning too many agents or failing to compress shared context effectively. The 2025 AI agent explosion isn't about making individual agents smarter—it's about building systems where many specialized agents coordinate effectively. Hermes and LobeHub represent two proven points in that design space, and the hybrid architectures they're converging toward suggest the field is still evolving rapidly. The engineers who internalize these patterns now will have a significant advantage as multi-agent systems move from experimental to essential infrastructure.