Building an AI Coding IDE from Scratch: A Full Open-Source Architecture

AI coding assistants have fundamentally changed how developers write software. The best ones do one thing well: collapse the distance between the developer's intent and the code that expresses it. But under the hood, they are largely orchestration layers — the individual primitives they use (language servers, vector search, tool-calling LLMs, sandboxed execution) are all open-source or replicable.

This post lays out a complete open-source architecture that covers every capability a modern AI coding IDE ships: autocomplete, chat-in-editor, multi-file agent editing, repo understanding, system design reasoning, and a safe code execution loop. No proprietary APIs required.

---

The Full Picture

Before going layer by layer, here is how the system fits together:

%%{init: {'theme': 'dark'}}%%
flowchart TB
    IDE["IDE Frontend\nMonaco Editor · Chat Panel · Agent Diff View"]
    OC["Orchestration Core\nLangGraph · Tool Router · Context Assembler"]
    FIM["FIM Engine"]
    CHAT["Chat / RAG Engine"]
    AGENT["Agent Engine"]
    EXEC["Execution Sandbox"]
    ML["Model Layer\nOllama · vLLM · StarCoder2 · DeepSeek-Coder-V2"]
    REPO["Repo Intelligence Layer\nTree-sitter · Chroma · nomic-embed-text · LSP"]

    IDE -->|"JSON-RPC / WebSocket"| OC
    OC --> FIM & CHAT & AGENT & EXEC
    FIM & CHAT & AGENT & EXEC --> ML
    ML --> REPO

Each vertical slice is independently deployable. You can run only the autocomplete engine on a laptop, or scale the agent engine on a GPU cluster. The layers communicate over a local JSON-RPC bus (same protocol VS Code uses for LSP).

---

Layer 1 — Repo Understanding

Every other capability is only as good as the system's understanding of the codebase. This is where most open-source IDE projects are weakest. Getting it right means going beyond naive file-splitting.

Parsing with Tree-sitter

Tree-sitter produces a concrete syntax tree for 40+ languages in under 5ms per file. Rather than splitting on character count, we split on semantic boundaries: functions, classes, method bodies. This keeps each chunk self-contained and reduces context fragmentation at retrieval time.

%%{init: {'theme': 'dark'}}%%
flowchart TB
    SRC["Raw Source Files"]
    TS["Tree-sitter\n(per language)"]
    SC["Semantic Chunks\nfunction_definition → 1 chunk\nclass_body → 1 chunk\nimport_block → 1 chunk"]
    META["Metadata Overlay\nfile path + git blame\nsymbol name + docstring\ncall graph edges via LSP references"]
    EMBED["Embedding Model\nnomic-embed-text-v1.5 · 768-dim · runs locally\nor CodeBERT for code-specific tasks"]
    VS["Vector Store\nChroma (local dev) · Qdrant (self-hosted, production)"]

    SRC --> TS --> SC --> META --> EMBED --> VS

The Call Graph Layer

Pure vector search finds semantically similar code but misses structural relationships. A symbol reference graph (built from LSP textDocument/references calls) lets you answer questions like: "find every function that touches the auth middleware" with a graph traversal rather than a fuzzy search.

Store this as an adjacency list in SQLite — lightweight, zero infrastructure, always in sync with the repo.

---

Layer 2 — Autocomplete (Fill-in-the-Middle)

Autocomplete in a modern AI IDE is not next-token prediction. It is fill-in-the-middle (FIM): the model sees the prefix (everything before the cursor) and the suffix (everything after), and generates the completion that bridges them.

%%{init: {'theme': 'dark'}}%%
flowchart TB
    subgraph req["FIM Request Assembly"]
        direction TB
        PRE["fim_prefix\nrepo context: top-3 retrieved chunks\ncurrent file: lines 0 to cursor"]
        SUF["fim_suffix\ncurrent file: cursor to EOF"]
        MID["fim_middle — model generates here"]
    end
    LAT["Latency Budget\nless than 150ms P50 for single-line completions\nless than 400ms P95 for multi-line blocks"]

    req -->|"speculative decode: draft model → verify model"| LAT

Model choices

Serve them with Ollama for local dev or vLLM in production (PagedAttention cuts memory by ~40%, continuous batching removes queuing).

Speculative Decoding

Pair a small draft model (StarCoder2-1B) with a large verifier (DeepSeek-Coder-V2-Lite). The draft generates K tokens; the verifier accepts or rejects in a single forward pass. Effective throughput: 3–5× faster than the large model alone for typical completion lengths.

---

Layer 3 — Chat-in-Editor

Chat works differently from autocomplete. The latency bar is 2–5 seconds (acceptable for a conversational exchange), but the context window must be carefully assembled to fit within model limits while including what's most relevant.

%%{init: {'theme': 'dark'}}%%
flowchart TB
    DEV["Developer Message"]
    CA["Context Assembler\nSlot 1: System prompt — 500 tokens\nSlot 2: Active file + selection — 2 000 tokens\nSlot 3: LSP diagnostics + errors — 500 tokens\nSlot 4: Retrieved chunks from vector store — 4 000 tokens\nSlot 5: Recent conversation history — 2 000 tokens\nSlot 6: User message — remaining budget\nTotal budget: 16k / 32k / 128k depending on model"]
    LLM["LLM — streaming SSE"]
    RP["Response Parser\nplain prose → render in chat panel\ncode blocks → diff-preview in editor\ntool_call JSON → route to Tool Engine"]

    DEV --> CA --> LLM --> RP

The critical UX insight: stream tokens to the chat panel in real time, but buffer code blocks and only apply them to the editor after the complete block arrives. Partial code blocks applied live cause flickering and make the diff unreadable.

For the model, any instruction-tuned model with a large context window works here: Qwen2.5-Coder-32B-Instruct, DeepSeek-V3, or Llama-3.3-70B-Instruct via Ollama / vLLM.

---

Layer 4 — Multi-File Agent Editing

This is the hardest layer to get right. The agent must plan, act across multiple files, observe outcomes (compiler errors, test failures), and revise — all without losing context of the original goal.

The Plan-Act-Observe Loop

%%{init: {'theme': 'dark'}}%%
flowchart TB
    REQ["Developer Request\nRefactor auth to use JWT, update all call sites"]
    PLAN["Planner\nreasoning model · CoT + outline"]

    subgraph loop["Agent Loop"]
        direction LR
        THINK["Think\nLLM"] -->|" "| ACT["Act\nTools"]
        ACT -->|"Observe"| THINK
    end

    DIFF["Diff Preview UI\nAccept / Reject"]

    REQ --> PLAN -->|"ordered task list"| loop --> DIFF

Tool Set

Orchestration: LangGraph

LangGraph models the agent loop as a directed graph of nodes (think, act, observe, plan, verify). Edges are conditional — the observe node routes back to think on errors, or forwards to verify on success.

The key advantage over a simple while loop: checkpointing. LangGraph can pause the loop mid-execution, serialize state to disk, and resume — critical for long refactors that might span dozens of file edits.

---

Layer 5 — System Design and Reasoning

Architecture-level questions ("should I use event sourcing here?", "draw the service dependency graph") require a different mode: long-horizon reasoning over the entire codebase context, not just a few files.

%%{init: {'theme': 'dark'}}%%
flowchart TB
    Q["Developer: Design a rate-limiting layer for this API"]
    RSB["Repo Summary Builder\nsummarise each module — 1-paragraph LLM call per dir\nbuild project map: module to responsibilities\nidentify integration points from call graph"]
    RM["Reasoning Model — extended thinking\nDeepSeek-R1 671B quantised or QwQ-32B 32B 24GB VRAM\nGenerates: options analysis, tradeoffs, recommendation"]
    DG["Diagram Generation\nModel outputs Mermaid or PlantUML markup\nRendered inline in IDE panel · Exportable as SVG"]

    Q --> RSB -->|"~8k token project map"| RM --> DG

The repo summary is the critical artifact. Build it once on first index, then update incrementally using git diff — only re-summarise modules that changed in the last commit.

---

Layer 6 — Safe Code Execution Loop

Agents that can write code must be able to run it. But running arbitrary LLM-generated code on the host machine is a hard no. The execution layer must be:

• Isolated: no access to host filesystem, network, or env vars outside the project
• Ephemeral: container torn down after each run
• Auditable: all stdin/stdout/stderr captured and shown to the developer

%%{init: {'theme': 'dark'}}%%
flowchart TB
    REQ["Agent: run_command('pytest tests/auth_test.py')"]

    subgraph sandbox["Execution Sandbox"]
        DC["Docker Container — ephemeral\nproject files: read-only bind mount\n/tmp: writable scratch space only\nnetwork: none\nuser: nobody uid 65534\nmemory limit: 512MB\nCPU: 1 core · 30s timeout\nseccomp profile: restricted syscalls"]
    end

    subgraph router["Result Router"]
        R0["exit 0 → tests passed → agent proceeds"]
        R1["exit 1 → test failures → agent sees stderr, re-plans"]
        RT["timeout → hard kill → agent informed, retries or stops"]
    end

    REQ --> sandbox -->|"exit code + stdout + stderr"| router

The Self-Healing Loop

%%{init: {'theme': 'dark'}}%%
flowchart LR
    WRITE["Write code"] --> RUN["Run tests"]
    RUN -->|"✅ Pass"| DIFF["Offer diff to developer"]
    RUN -->|"❌ Failure"| OBS["observe error"]
    OBS -->|"re-plan fix"| WRITE

When tests fail, the output becomes the next observation in the agent loop. The agent sees the exact error, reasons about the fix, edits the file, and re-runs — typically converging in 2–3 iterations for straightforward bugs.

For an even tighter sandbox, use gVisor (Google's container runtime that intercepts syscalls in user space) or Firecracker (AWS's micro-VM used in Lambda) instead of vanilla Docker.

---

The Full Open-Source Stack

---

What You Don't Get For Free

An honest architecture post should name the hard parts:

Latency at low VRAM. A 32B model doing chat on a single 24GB GPU hits 15–20 tokens/second. Acceptable for most workflows, but noticeably slower than cloud-hosted alternatives. The fix is speculative decoding, quantisation (GGUF Q4), or offloading to a small cloud GPU when needed.

Prompt cache invalidation. Managed AI coding services almost certainly implement prompt caching across requests. Replicating this without a managed inference provider requires careful key-value cache management in vLLM — possible, but non-trivial.

Index freshness. Keeping the vector store in sync with active edits (every keystroke rewrites files) requires debounced incremental re-indexing — easy to get wrong and end up with stale retrieval.

Security surface. The Docker sandbox is safe for test runners. But agents that can write_file anywhere in the repo, modify CI configs, or touch secrets files are a different risk level. Implement a path allowlist and require developer confirmation for writes outside the current working directory.

---

Closing Thoughts

The individual components here — Tree-sitter, vLLM, LangGraph, Docker — are each battle-tested in production at scale. The architecture challenge is the orchestration: assembling the right context, routing to the right model at the right latency budget, and designing a UX that keeps the developer in control of what the agent actually touches.

The moat of any great AI coding tool is not its architecture. It's the years of UX iteration on top of this architecture. The open-source community now has every primitive it needs to build something just as capable.

The next post in this series will walk through implementing the FIM autocomplete engine end-to-end: Tree-sitter chunking, nomic embeddings, and a StarCoder2-3B server with speculative decoding — all running on a single laptop.