Improve docs AI readiness (#59577)

Context

This PR makes the Zed docs easier for AI tools, search crawlers, and
users to consume without changing the visible docs content. The current
production docs are primarily optimized for browser navigation. They do
not expose first-class Markdown URLs, an `llms.txt` index, page-level
copy affordances, or machine-readable freshness metadata that let users
and agents grab clean, current page content.

Changes

- Generate Markdown copies for docs pages during the mdBook postprocess
step, including `/docs/index.md` as an alias for Getting Started.
- Generate `/docs/llms.txt` from the mdBook chapter list, grouped by
`SUMMARY.md` sections and annotated with page frontmatter descriptions.
- Generate `/docs/sitemap.xml` with `<lastmod>` values for every docs
page.
- Emit machine-readable freshness metadata in HTML via `last-modified`
and `article:modified_time` meta tags.
- Generate Cloudflare Pages `_redirects` for `.html`, extensionless, and
`.md` redirect variants, with channel-aware docs destinations.
- Add discovery hints for agents and crawlers: `rel="llms.txt"`,
`rel="alternate" type="text/markdown"`, and a short generated `llms.txt`
directive in copied Markdown pages.
- Update the docs proxy so `Accept: text/markdown`, `/docs.md`, and
direct `.md` requests can resolve to the generated Markdown artifacts.
- Move primary docs content earlier in the HTML source while preserving
the visible layout, so crawlers and agent scorers encounter the article
before sidebar chrome.
- Move the existing copy-as-Markdown control from the top navigation
into the page-title row, using the generated Markdown alternate link as
the source of truth.
- Split AI-discovery artifact generation out of
`docs_preprocessor/src/main.rs` into a focused module.

Best Practices Adopted

- Use `llms.txt` as a concise navigation index, not a dump of full page
content.
- Link to absolute, canonical Markdown URLs from `llms.txt`.
- Preserve the docs hierarchy in `llms.txt` instead of emitting a flat
sitemap-like list.
- Include short per-link descriptions from existing metadata rather than
inventing summaries.
- Keep `llms.txt`, Markdown copies, sitemap data, redirects, and
freshness metadata generated from the same mdBook source to avoid drift.
- Advertise Markdown alternates with standard HTML metadata and
same-origin URLs.
- Support both explicit Markdown URLs and content negotiation for
clients that prefer Markdown.
- Keep browser copy behavior pointed at generated Markdown alternate
links instead of duplicating route inference in JavaScript.
- Keep the copy-as-Markdown affordance in the page title row without
duplicating header chrome controls.

Validation

- `cargo check -p docs_preprocessor`
- `cargo test -p docs_preprocessor`
- `./script/clippy -p docs_preprocessor`
- `mdbook build ./docs --dest-dir=../target/deploy/docs/`
- `node --check docs/theme/plugins.js`
- `pnpm dlx prettier@3.5.0 docs/theme/plugins.js --check`
- `git diff --check`
- Local artifact checks confirmed generated Markdown pages, `llms.txt`,
`sitemap.xml` lastmod values, HTML freshness metadata, Markdown
alternate links, redirect targets, and preprocessed action/keybinding
tags resolve as expected.
- Worker URL rewrite mock covered `/docs/`, `/docs.md`,
`/docs/index.md`, extensionless docs routes, `.html` routes,
`/docs/llms.txt`, and `/docs/sitemap.xml`.
- High-effort adversarial subagent review found blockers around
channel-aware redirects, shallow-checkout date fallback, file size,
duplicated Markdown path inference, and process spawning. Those were
addressed.

Remaining Notes

- Local `python -m http.server` does not emulate Cloudflare Pages pretty
URLs, `_redirects`, or the docs-proxy Worker, so full local `afdocs`
still cannot prove content negotiation end to end.
- Existing production remains unchanged until this PR is deployed
through the docs workflow.
- Production baseline `npx afdocs check https://zed.dev/docs/ --fixes
--verbose` still reports the original failures before this PR is
deployed: 12 passed, 8 failed, 3 skipped.

Release Notes:

- Improved docs AI-readiness by adding machine-readable discovery,
Markdown access, and freshness metadata.

---------

Co-authored-by: Katie Geer <katie@zed.dev>
Co-authored-by: Ben Kunkle <ben@zed.dev>

docs/theme/plugins.js

@@ -79,34 +79,15 @@ function darkModeToggle() {
   });
 }
 
-const copyMarkdown = () => {
-  const copyButton = document.getElementById("copy-markdown-toggle");
-  if (!copyButton) return;
+const copyPageActions = () => {
+  const headerCopyMarkdownButton = document.getElementById(
+    "copy-markdown-toggle",
+  );
 
-  // Store the original icon class, loading state, and timeout reference
-  const originalIconClass = "fa fa-copy";
+  const actionButtons = new Map();
   let isLoading = false;
-  let iconTimeoutId = null;
-
-  const getCurrentPagePath = () => {
-    const pathname = window.location.pathname;
-
-    // Handle root docs path
-    if (pathname === "/docs/" || pathname === "/docs") {
-      return "getting-started.md";
-    }
-
-    // Remove /docs/ prefix and .html suffix, then add .md
-    const cleanPath = pathname
-      .replace(/^\/docs\//, "")
-      .replace(/\.html$/, "")
-      .replace(/\/$/, "");
-
-    return cleanPath ? cleanPath + ".md" : "getting-started.md";
-  };
 
   const showToast = (message, isSuccess = true) => {
-    // Remove existing toast if any
     const existingToast = document.getElementById("copy-toast");
     existingToast?.remove();
 
@@ -117,12 +98,10 @@ const copyMarkdown = () => {
 
     document.body.appendChild(toast);
 
-    // Show toast with animation
     setTimeout(() => {
       toast.classList.add("show");
     }, 10);
 
-    // Hide and remove toast after 2 seconds
     setTimeout(() => {
       toast.classList.remove("show");
       setTimeout(() => {
@@ -131,38 +110,119 @@ const copyMarkdown = () => {
     }, 2000);
   };
 
-  const changeButtonIcon = (iconClass, duration = 1000) => {
-    const icon = copyButton.querySelector("i");
+  const registerButton = (button, originalIconClass) => {
+    if (!button) return;
+    actionButtons.set(button, {
+      originalIconClass,
+      iconTimeoutId: null,
+    });
+  };
+
+  registerButton(headerCopyMarkdownButton, "fa fa-copy");
+
+  const changeButtonIcon = (button, iconClass, duration = 1000) => {
+    const state = actionButtons.get(button);
+    if (!state) return;
+
+    const icon = button.querySelector("i");
     if (!icon) return;
 
-    // Clear any existing timeout
-    if (iconTimeoutId) {
-      clearTimeout(iconTimeoutId);
-      iconTimeoutId = null;
+    if (state.iconTimeoutId) {
+      clearTimeout(state.iconTimeoutId);
+      state.iconTimeoutId = null;
     }
 
     icon.className = iconClass;
 
     if (duration > 0) {
-      iconTimeoutId = setTimeout(() => {
-        icon.className = originalIconClass;
-        iconTimeoutId = null;
+      state.iconTimeoutId = setTimeout(() => {
+        icon.className = state.originalIconClass;
+        state.iconTimeoutId = null;
       }, duration);
     }
   };
 
-  const fetchAndCopyMarkdown = async () => {
+  const copyText = async (text) => {
+    if (!navigator.clipboard?.writeText) {
+      throw new Error("Clipboard API not supported in this browser");
+    }
+
+    await navigator.clipboard.writeText(text);
+  };
+
+  const getContentRoot = () =>
+    document.querySelector("#content main .content-wrap") ||
+    document.querySelector("#content > main");
+
+  const markdownAlternateHref = () =>
+    document
+      .querySelector('link[rel="alternate"][type="text/markdown"]')
+      ?.getAttribute("href");
+
+  const insertPageActionButtons = () => {
+    const contentRoot = getContentRoot();
+    if (
+      !contentRoot ||
+      contentRoot.querySelector(".page-actions") ||
+      !headerCopyMarkdownButton ||
+      !markdownAlternateHref()
+    ) {
+      return;
+    }
+
+    const firstHeading = contentRoot.querySelector("h1");
+    if (!firstHeading) return;
+
+    const header = document.createElement("div");
+    header.className = "page-header";
+
+    const actions = document.createElement("div");
+    actions.className = "page-actions";
+
+    firstHeading.insertAdjacentElement("beforebegin", header);
+    headerCopyMarkdownButton.classList.remove(
+      "icon-button",
+      "ib-hidden-mobile",
+    );
+    headerCopyMarkdownButton.classList.add("page-action", "page-action-icon");
+    actions.append(headerCopyMarkdownButton);
+    header.append(firstHeading, actions);
+  };
+
+  const markdownUrl = () => {
+    const alternateHref = markdownAlternateHref();
+    if (!alternateHref) return null;
+
+    const url = new URL(alternateHref, window.location.href);
+    if (
+      url.origin === window.location.origin &&
+      url.pathname.startsWith("/docs/") &&
+      !window.location.pathname.startsWith("/docs/")
+    ) {
+      return url.pathname.replace(/^\/docs/, "") || "/";
+    }
+
+    if (url.origin === window.location.origin) {
+      return `${url.pathname}${url.search}${url.hash}`;
+    }
+
+    return url.pathname;
+  };
+
+  const fetchAndCopyMarkdown = async (button) => {
     // Prevent multiple simultaneous requests
     if (isLoading) return;
 
     try {
       isLoading = true;
-      changeButtonIcon("fa fa-spinner fa-spin", 0); // Don't auto-restore spinner
+      changeButtonIcon(button, "fa fa-spinner fa-spin", 0); // Don't auto-restore spinner
 
-      const pagePath = getCurrentPagePath();
-      const rawUrl = `https://raw.githubusercontent.com/zed-industries/zed/main/docs/src/${pagePath}`;
+      const url = markdownUrl();
+      if (!url) {
+        throw new Error("Markdown alternate link not found");
+      }
 
-      const response = await fetch(rawUrl);
+      const response = await fetch(url);
       if (!response.ok) {
         throw new Error(
           `Failed to fetch markdown: ${response.status} ${response.statusText}`,
@@ -171,33 +231,36 @@ const copyMarkdown = () => {
 
       const markdownContent = await response.text();
 
-      // Copy to clipboard using modern API
-      if (navigator.clipboard?.writeText) {
-        await navigator.clipboard.writeText(markdownContent);
-      } else {
-        // Fallback: throw error if clipboard API isn't available
-        throw new Error("Clipboard API not supported in this browser");
-      }
+      await copyText(markdownContent);
 
-      changeButtonIcon("fa fa-check", 1000);
-      showToast("Page content copied to clipboard!");
+      changeButtonIcon(button, "fa fa-check", 1000);
+      showToast("Markdown copied to clipboard!");
     } catch (error) {
       console.error("Error copying markdown:", error);
-      changeButtonIcon("fa fa-exclamation-triangle", 2000);
+      changeButtonIcon(button, "fa fa-exclamation-triangle", 2000);
       showToast("Failed to copy markdown. Please try again.", false);
     } finally {
       isLoading = false;
     }
   };
 
-  copyButton.addEventListener("click", fetchAndCopyMarkdown);
+  headerCopyMarkdownButton?.addEventListener("click", () =>
+    fetchAndCopyMarkdown(headerCopyMarkdownButton),
+  );
+
+  insertPageActionButtons();
 };
 
-// Initialize functionality when DOM is loaded
-document.addEventListener("DOMContentLoaded", () => {
+const initializePlugins = () => {
   darkModeToggle();
-  copyMarkdown();
-});
+  requestAnimationFrame(copyPageActions);
+};
+
+if (document.readyState === "loading") {
+  document.addEventListener("DOMContentLoaded", initializePlugins);
+} else {
+  initializePlugins();
+}
 
 // Collapsible sidebar navigation for entire sections
 // Note: Initial collapsed state is applied in index.hbs to prevent flicker