Open Source

vite-plugin-llm-spider: Make Your SPA Visible to AI Search

Kingsley

Kingsley

— 2 min read
vite-plugin-llm-spider: Make Your SPA Visible to AI Search

Your single page app is invisible to AI.

While ChatGPT, Perplexity, and Claude reshape how people find information, SPAs remain hidden behind JavaScript walls. These AI systems can't execute your Vue, React, or Svelte code—they just see an empty shell.

vite-plugin-llm-spider fixes this. One plugin, zero configuration, and suddenly your SPA is discoverable by AI search engines.

The Problem: SPAs and AI Don't Mix

Traditional SEO optimizes for Google's crawler. But a new frontier is emerging: Generative Engine Optimization (GEO)—optimizing for AI systems that synthesize answers from your content.

When someone asks ChatGPT or Perplexity a question your site answers, you want to be cited. You want to be the source. But if your content only exists after JavaScript executes, AI crawlers see nothing.

Most solutions involve complex SSR setups, prerendering services, or entirely rearchitecting your app. That's overkill for most projects.

The Solution: Build-Time Snapshots

Our Vite plugin takes a simpler approach:

  1. At build time, it launches a headless browser
  2. Crawls your SPA just like a user would
  3. Captures each rendered page as clean markdown
  4. Generates an llms.txt index following the llmstxt.org spec

Result: your dist/ folder includes machine-readable markdown snapshots that AI crawlers can consume directly.

Installation

npm i -D @happyalienai/vite-plugin-llm-spider

Usage

Zero configuration required:

// vite.config.js
import llmSpider from "@happyalienai/vite-plugin-llm-spider";

export default defineConfig({
  plugins: [
    llmSpider(),  // That's it
  ],
});

After npm run build, you get:

  • dist/llms.txt — auto-generated index of all discovered pages
  • dist/*.md — markdown snapshot for each page

What Gets Generated

The llms.txt file follows the emerging standard for AI-readable site indexes:

# My Site

> LLM-friendly index of important pages and their Markdown equivalents.

## Product

- [Home](index.html.md)
- [Pricing](pricing.md)
- [Features](features.md)

## Docs

- [Getting Started](docs/getting-started.md)
- [API Reference](docs/api.md)

Each linked .md file contains the clean, extracted content from that page—no navigation, no footers, no cookie banners. Just the content AI systems need.

Smart Defaults

The plugin automatically:

  • Crawls from root — follows links to discover pages
  • Strips noise — removes nav, footer, modals, banners
  • Respects limits — max depth and page count to prevent runaway crawls
  • Blocks analytics — no Google Analytics or tracking scripts during capture

Customization

Need more control? Everything is configurable:

llmSpider({
  exclude: ["/login", "/admin", "/account"],
  crawl: {
    maxDepth: 3,
    maxPages: 100,
  },
  extract: {
    mainSelector: ["main", "#content"],
    removeSelectors: ["nav", "footer", ".modal"],
  },
})

Works Everywhere

Any Vite framework: Vue, React, Svelte, Solid, Astro. If it builds with Vite, this plugin works.

For SSG sites that don't need JavaScript rendering, use static: true to skip Puppeteer entirely—perfect for CI environments without Chrome.

Why llms.txt?

The llms.txt specification is an emerging standard for making websites AI-readable. Think of it like robots.txt but for LLMs—a standardized way to tell AI agents where your important content lives and how to access it.

As AI search becomes mainstream, having an llms.txt is table stakes for discoverability.

Get Started

Install it, add one line to your Vite config, and your next build includes AI-friendly snapshots. No architecture changes. No SSR complexity. Just discoverability.

npm i -D @happyalienai/vite-plugin-llm-spider

Full documentation and source: npm package

Make your SPA visible to the future of search.

Read more