Clean HTML and Markdown delivery drive visibility to AI agents

Evil Martians’ server logs show a split in how AI tools consume websites: chat assistants primarily fetch HTML, while coding agents request Markdown through HTTP content negotiation. Claude Code accounted for 94% of Markdown traffic on the consultancy’s site, while ChatGPT-User requested Markdown on under 1% of its requests. The findings favor clean, server-rendered HTML alongside Markdown endpoints rather than replacing either format with the other.

The recommended foundation includes auditing robots.txt and CDN restrictions, using semantic page structure, providing .md routes, and advertising alternate formats through HTML links and HTTP Link headers. Servers should honor Accept: text/markdown preferences and send Vary: Accept so caches distinguish representations. Generating both formats from a shared content source helps prevent stale information. By contrast, llms.txt remains a low-priority addition: no major LLM provider has committed to reading it, and observed assistant traffic was limited.

Unsupported AI meta tags, hidden discovery hints, HTML comments, and dedicated AI information pages lack evidence of improving discovery. JSON-LD can still help indirectly through search indexes and shopping systems, provided it matches visible content. Measurement requires server-side logs because AI fetchers generally do not execute analytics JavaScript. Tracking should distinguish training crawlers from tools fetching pages for users, rather than treating all bot traffic as evidence of visibility.