I remember the first time I discovered that VS Code's IntelliSense, go to definition, and auto completion features weren't magic built into the editor. They were powered by something called a Language Server Protocol (LSP). This revelation changed how I thought about developer tooling entirely.

The Language Server Protocol is one of those brilliant ideas that seems obvious in hindsight. Before LSP, every editor had to implement language support from scratch. If you wanted Python IntelliSense in VS Code, Sublime, Vim, and Emacs, you needed four separate implementations. LSP changed everything by creating a standardized protocol that allows any editor to communicate with any language server.

In this article, I'll walk you through building a simple language server from scratch, exploring how LSP works under the hood, and understanding why it's become the foundation of modern developer tooling.

What is the Language Server Protocol?

At its core, LSP is a JSON-RPC based protocol that defines how editors (clients) communicate with language analysis tools (servers). Microsoft introduced it in 2016, and it's now supported by virtually every major code editor.

The genius of LSP lies in its separation of concerns:

Editors handle the UI, file management, and user interactions
Language Servers handle language specific intelligence: parsing, type checking, symbol resolution, etc.

This means you write your language server once, and it works in VS Code, Vim, Emacs, Sublime, and dozens of other editors without modification.

The LSP Request Response Cycle

Communication between the editor and language server happens through JSON-RPC messages over stdin/stdout or TCP. Here's a simplified example:

Editor (Client) Request:

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "textDocument/completion",
  "params": {
    "textDocument": { "uri": "file:///path/to/file.ts" },
    "position": { "line": 10, "character": 5 }
  }
}

Language Server Response:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "items": [
      {
        "label": "console",
        "kind": 6,
        "detail": "Global console object",
        "insertText": "console"
      }
    ]
  }
}

Core LSP Capabilities

A language server can implement various capabilities:

1. Text Synchronization

The editor notifies the server when files are opened, modified, or closed:

// File opened
textDocument/didOpen

// File changed
textDocument/didChange

// File saved
textDocument/didSave

// File closed
textDocument/didClose

2. Code Intelligence

These are the features users interact with:

// Auto-completion
textDocument/completion

// Go to definition
textDocument/definition

// Find references
textDocument/references

// Hover information
textDocument/hover

// Signature help (parameter hints)
textDocument/signatureHelp

// Code actions (quick fixes)
textDocument/codeAction

// Diagnostics (errors/warnings)
textDocument/publishDiagnostics

3. Workspace Operations

// Rename symbol across files
textDocument/rename

// Find all symbols in workspace
workspace/symbol

// Format document
textDocument/formatting

Building a Simple Language Server

Let's build a minimal language server for a hypothetical language. We'll use Node.js and the vscode-languageserver package:

import {
  createConnection,
  TextDocuments,
  ProposedFeatures,
  InitializeParams,
  CompletionItem,
  CompletionItemKind,
  TextDocumentPositionParams,
  TextDocumentSyncKind,
} from 'vscode-languageserver/node';

import { TextDocument } from 'vscode-languageserver-textdocument';

// Create a connection for the server
const connection = createConnection(ProposedFeatures.all);

// Create a document manager
const documents = new TextDocuments(TextDocument);

// Initialize handler - declare capabilities
connection.onInitialize((params: InitializeParams) => {
  return {
    capabilities: {
      textDocumentSync: TextDocumentSyncKind.Incremental,
      completionProvider: {
        resolveProvider: true,
        triggerCharacters: ['.'],
      },
      hoverProvider: true,
      definitionProvider: true,
    },
  };
});

// Completion provider
connection.onCompletion(
  (_textDocumentPosition: TextDocumentPositionParams): CompletionItem[] => {
    return [
      {
        label: 'console',
        kind: CompletionItemKind.Keyword,
        data: 1,
      },
      {
        label: 'function',
        kind: CompletionItemKind.Keyword,
        data: 2,
      },
    ];
  }
);

// Resolve completion item details
connection.onCompletionResolve((item: CompletionItem): CompletionItem => {
  if (item.data === 1) {
    item.detail = 'Global console object';
    item.documentation = 'Provides access to the browser debugging console';
  }
  return item;
});

// Hover provider
connection.onHover((params) => {
  const document = documents.get(params.textDocument.uri);
  if (!document) return null;

  const position = params.position;
  const text = document.getText();
  
  // Simple hover implementation
  return {
    contents: {
      kind: 'markdown',
      value: '**Hover Information**\n\nThis is a simple hover example.',
    },
  };
});

// Listen for document changes
documents.onDidChangeContent((change) => {
  validateDocument(change.document);
});

function validateDocument(document: TextDocument): void {
  const text = document.getText();
  const diagnostics = [];

  // Simple validation: find "TODO" comments
  const lines = text.split('\n');
  lines.forEach((line, i) => {
    if (line.includes('TODO')) {
      diagnostics.push({
        severity: 2, // Warning
        range: {
          start: { line: i, character: 0 },
          end: { line: i, character: line.length },
        },
        message: 'TODO comment found',
        source: 'my-lsp',
      });
    }
  });

  connection.sendDiagnostics({ uri: document.uri, diagnostics });
}

// Make the text document manager listen on the connection
documents.listen(connection);

// Start listening
connection.listen();

Advanced LSP Features

Incremental Text Synchronization

For performance, LSP supports incremental updates instead of sending the entire file:

textDocumentSync: TextDocumentSyncKind.Incremental

The editor sends only the changed ranges:

{
  "range": { "start": { "line": 5, "character": 0 }, "end": { "line": 5, "character": 10 } },
  "text": "new code here"
}

Semantic Tokens

Modern language servers provide semantic highlighting:

connection.languages.semanticTokens.on((params) => {
  const document = documents.get(params.textDocument.uri);
  // Return token information for syntax highlighting
  return {
    data: [
      // [line, char, length, tokenType, tokenModifiers]
      0, 0, 5, 0, 0,  // "const" keyword at line 0
      0, 6, 4, 1, 0,  // "name" variable at line 0
    ],
  };
});

Code Actions and Quick Fixes

Provide automated fixes for diagnostics:

connection.onCodeAction((params) => {
  const diagnostics = params.context.diagnostics;
  
  return diagnostics.map((diagnostic) => ({
    title: 'Fix this issue',
    kind: 'quickfix',
    diagnostics: [diagnostic],
    edit: {
      changes: {
        [params.textDocument.uri]: [{
          range: diagnostic.range,
          newText: 'corrected code',
        }],
      },
    },
  }));
});

Performance Considerations

1. Caching

Language servers should cache parsed ASTs and type information:

const fileCache = new Map<string, ParsedFile>();

function parseDocument(uri: string, text: string) {
  if (fileCache.has(uri)) {
    const cached = fileCache.get(uri);
    if (cached.version === getCurrentVersion(uri)) {
      return cached.ast;
    }
  }
  
  const ast = parse(text);
  fileCache.set(uri, { ast, version: getCurrentVersion(uri) });
  return ast;
}

2. Debouncing

Avoid expensive operations on every keystroke:

let validationTimer: NodeJS.Timer | null = null;

documents.onDidChangeContent((change) => {
  if (validationTimer) {
    clearTimeout(validationTimer);
  }
  
  validationTimer = setTimeout(() => {
    validateDocument(change.document);
  }, 500); // Wait 500ms after last change
});

3. Incremental Parsing

For large files, only reparse changed sections:

function incrementalParse(oldAst: AST, changes: TextChange[]) {
  // Only reparse affected nodes
  for (const change of changes) {
    const affectedNode = findNodeAtPosition(oldAst, change.range.start);
    reparseBranch(affectedNode, change.text);
  }
  return oldAst;
}

Testing Your Language Server

Unit Testing

Test individual features in isolation:

import { createConnection } from 'vscode-languageserver/node';

describe('Completion Provider', () => {
  it('should provide keyword completions', async () => {
    const completions = await getCompletions({
      textDocument: { uri: 'test.ts' },
      position: { line: 0, character: 0 },
    });
    
    expect(completions).toContainEqual({
      label: 'function',
      kind: CompletionItemKind.Keyword,
    });
  });
});

Integration Testing

Use the LSP testing framework:

import { TestClient } from 'vscode-languageserver-protocol/lib/test/client';

const client = new TestClient();
await client.start();

await client.sendRequest('textDocument/completion', {
  textDocument: { uri: 'file:///test.ts' },
  position: { line: 0, character: 0 },
});

const result = await client.receiveResponse();
expect(result.items.length).toBeGreaterThan(0);

Real World Language Servers

Some excellent open source language servers to study:

TypeScript Language Server (tsserver)

One of the most sophisticated LSP implementations
Handles complex type inference and project references
Source: microsoft/TypeScript

rust-analyzer

Modern Rust language server
Excellent example of incremental compilation
Source: rust-lang/rust-analyzer

gopls

Official Go language server
Shows good patterns for caching and workspace management
Source: golang/tools

Common Pitfalls and Solutions

1. Memory Leaks

Language servers run continuously and must manage memory carefully:

// Bad: Unbounded cache
const cache = new Map();

// Good: LRU cache with size limit
const cache = new LRUCache({ max: 1000 });

2. Blocking Operations

Never block the main event loop:

// Bad: Synchronous parsing
const ast = parseSync(largeFile);

// Good: Async or worker threads
const ast = await parseAsync(largeFile);
// Or use worker threads for CPU intensive tasks

3. Workspace Scalability

Handle large workspaces efficiently:

// Index files on demand, not all at once
async function indexWorkspace(rootUri: string) {
  const files = await findRelevantFiles(rootUri);
  
  // Index in batches
  for (let i = 0; i < files.length; i += 100) {
    const batch = files.slice(i, i + 100);
    await Promise.all(batch.map(indexFile));
    
    // Allow other operations between batches
    await setImmediate();
  }
}

Debugging Your Language Server

Enable Logging

import { createConnection, Logger } from 'vscode-languageserver/node';

const connection = createConnection(ProposedFeatures.all);

connection.console.log('Server started');
connection.console.error('Error occurred');

Attach Debugger

In VS Code, create a launch configuration:

{
  "type": "node",
  "request": "attach",
  "name": "Attach to Language Server",
  "port": 6009,
  "restart": true,
  "outFiles": ["${workspaceFolder}/out/**/*.js"]
}

Start your server with debugging enabled:

node --inspect=6009 out/server.js

The Future of LSP

The Language Server Protocol continues to evolve. Recent additions include:

Semantic tokens for better syntax highlighting
Call hierarchy for understanding function call relationships
Type hierarchy for navigating type systems
Inline values for debugging integration
Notebook support for Jupyter-style notebooks

Conclusion

Building a language server might seem daunting, but the Language Server Protocol provides a well structured framework that makes it manageable. Whether you're creating support for a new language, building specialized tooling, or just curious about how your editor works, understanding LSP is invaluable.

The best way to learn is to start small: implement just completion or hover support, then gradually add features. Study existing language servers, leverage the excellent vscode-languageserver npm package, and don't be afraid to dive into the LSP specification.

The developer tooling ecosystem is better because of LSP, and with the knowledge to build your own language server, you can contribute to making it even better.

Further Reading:

Try it yourself: Clone the lsp-sample repo and experiment with building your first language server.

Building Your Own Language Server: A Deep Dive into LSP