Back to Projects
Architecting a Cloud-Native Autonomous VS Code Agent: A Distributed, Human-in-the-Loop Framework
Agent Building
4 min read
TypeScriptVS Code ExtensionFastAPIGemini SDKDistributed Systems

Architecting a Cloud-Native Autonomous VS Code Agent: A Distributed, Human-in-the-Loop Framework

Project Overview

Architecting a Cloud-Native Autonomous VS Code Agent: A Distributed, Human-in-the-Loop Framework

While the industry has seen significant advancements with tools like GitHub Copilot Workspace and Cursor, I wanted to explore the architectural requirements for building a truly autonomous, project-specific stateful agent from the ground up. This is the second part of the experimental introduction of agentic coding to our team.

The result is Hermes AI 2.0—a custom, self-hosted, autonomous coding agent designed to function not just as a chatbot, but as an integrated engineering teammate capable of codebase exploration, terminal execution, and direct file manipulation.

System Architecture

The Hermes AI architecture is built on a decoupled, distributed model to ensure scalability and security:

  • Orchestration Layer (Google Cloud Run): A high-performance Python/FastAPI backend powered by the Gemini 1.5 Pro SDK. It utilizes a dynamic Tool Registry and leverages Server-Sent Events (SSE) to stream real-time reasoning and thought processes to the client.
  • Client-Side Integration (VS Code Extension): A custom TypeScript extension that interfaces directly with the native VS Code Chat API.
  • The Command-Response Proxy Loop: To bridge the gap between the cloud-based brain and the local development environment, the system utilizes a JSON-based command protocol. The cloud backend issues specific "Commands" via the SSE stream, which the local extension intercepts to execute file system or terminal operations, subsequently feeding the execution results back to the cloud for continuous reasoning.

    • Core Capabilities

    1. Multimodal Contextual Awareness

    Hermes supports multimodal input, allowing developers to drag and drop UI screenshots directly into the chat. The agent performs visual analysis to identify discrepancies and autonomously hunts down the corresponding CSS or React components to resolve visual regressions.

    2. Autonomous Tool Use & Self-Correction

    The agent possesses full terminal integration. It can execute package installations, run test suites, parse stderr outputs, and iteratively rewrite code to resolve failing tests without manual intervention.

    3. Granular Code Manipulation Modes

    To provide developers with precise control over code changes, I implemented a multi-mode editing interface:

  • Ghost Mode: Utilizes the WorkspaceEdit API to perform seamless, real-time code injections directly into the active editor.
  • Diff Mode: Designed for large-scale refactors; the agent generates a temporary file and triggers the native VS Code Split-Screen Diff Viewer, allowing for line-by-line validation before commits.
  • Chat Mode: Operates as a high-level architectural advisor, providing markdown-based guidance.

    • 4. Adaptive User Profiling

    To optimize the Developer Experience (DX), Hermes employs an individualized user profile prompt. This profile is synthesized from a dedicated one-hour observation period, analyzing the user's specific coding patterns, syntax preferences, and instruction-following styles to ensure the AI's output is highly personalized.

    5. Governance & Human-in-the-Loop (HITL)

    To maintain safety during complex or long-running operations, Hermes implements a mandatory intervention threshold. If a task exceeds five consecutive autonomous loops, the agent pauses execution, presents a collapsible audit log of all terminal commands, and awaits explicit user authorization to proceed.

    Intelligence & Observability

    Hermes is designed with a "flywheel effect" in mind, focusing on both individual efficiency and collective intelligence:

  • Distributed Project Intelligence: Every interaction contributes to a shared, global project context. By aggregating individual developer insights, the system builds a specialized knowledge base that effectively fine-tunes the model’s understanding of the specific project architecture and team conventions.
  • Granular Observability: The system includes dedicated monitoring for each user, tracking individual coding metrics and token consumption to ensure efficient resource management and cost transparency.

    • Conclusion

    Building a distributed, stateful, and streaming agent architecture presented significant engineering challenges, particularly regarding the synchronization of cloud-based reasoning with local execution.

    However, beyond the architectural complexity inherent in interactive AI systems, the true value of this platform lies in its ability to earn user trust and personalize interactions based on individual coding profiles and identities. The agent maintains awareness of each user’s specific coding and instructional preferences, as well as their name.

    Specifications

    Tech Stack

    TypeScriptVS Code ExtensionFastAPIGemini SDKDistributed Systems
    DomainAgent Building
    StatusActive Production
    Date Released2026
    Hardware BaseCUDA GPU Nodes

    Need help in research integration?

    Get in touch to discuss specialized development and consulting opportunities.