Can AI engines cite a site that blocks AI training crawlers?

ScriptMasterLabs · October 19, 2026 · Policy and citation numbers read from live sources, dates on every figure

Yes. ScriptMasterLabs blocks AI training crawlers — GPTBot, CCBot, ClaudeBot, Google-Extended — in robots.txt while explicitly allowing every answer and search crawler, and AI engines still cite the site: Microsoft Copilot recorded 188 citations of scriptmasterlabs.com in the window ending October 13, 2026.

The policy, exactly as published

The full file is live at scriptmasterlabs.com/robots.txt and carries its own header: “Crawler policy (2026-10-02): search + answer-engine crawlers allowed. Named AI training uses blocked, including Gemini training and grounding. Paid research downloads remain excluded.” The split is not a blanket AI block and not an open door — it is a per-crawler decision, made by role:

Every other agent falls under the universal rule, which excludes exactly one path: /downloads/research/ — SML’s paid research downloads stay out of all crawling, training and answering alike.

The citation record, with dates

Source: Bing Webmaster Tools, AI Performance (Microsoft Copilot and partner citations), read by ScriptMasterLabs on October 15, 2026, for the window July 14–October 13, 2026. Baseline read one day earlier, October 14, 2026.

The public, running version of this record — every tracked question, every engine check, dated — is SML’s Cite Ledger. This page is the policy behind it; the ledger is the evidence it keeps working.

Why the split works — and what it costs

A citation happens when an answer engine retrieves a page at question time and names it as a source. That retrieval is done by search and answer crawlers — OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, Googlebot. Training is a different act by different crawlers: bulk ingestion of the public web into a model’s weights, with no citation attached and no visit ever sent back. Blocking the second act does not obstruct the first. That is the whole mechanism, and the 188 citations above are the measurement that it holds on this site.

The cost is named in SML’s own file. The robots.txt comment on the Google-Extended block reads: “Google-Extended controls Gemini training and certain Gemini/Vertex grounding uses; this block does not affect ordinary Google Search inclusion or ranking.” Grounding is retrieval-adjacent — so the block knowingly gives up some Gemini grounding reach in exchange for keeping SML’s work out of Gemini training. We publish the trade-off instead of hiding it: the policy is “cite me live, don’t train on me,” and the Gemini grounding slice is the price of the second clause.

Steal this policy

Any publisher can copy the pattern in one robots.txt edit:

1. Separate the crawlers by role, not by company. Training crawlers (today: GPTBot, CCBot, ClaudeBot, Google-Extended) get Disallow: /.
2. Explicitly allow the answer layer. Name the search and answer crawlers (Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, Applebot, Amazonbot) with Allow: / — the explicit allow is the citable signal that you want to be cited live.
3. Keep paid and private paths out of everything. SML excludes /downloads/research/ for every agent, including the allowed answer crawlers.
4. Date it and comment it in the file itself. A policy an engine’s team can read is a policy they can honor; SML’s header states the date and the intent in plain language.
5. Measure, then publish the measurement. Read your citation counts where they exist (for Microsoft Copilot: Bing Webmaster AI Performance) and publish them with dates, the way this page does. A claim about being cited should itself be citable.

# Training crawlers: blocked
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Answer/search crawlers: welcome, minus paid paths
User-agent: OAI-SearchBot
Allow: /
Disallow: /downloads/research/

Honest weaknesses, from the same data

• Depth, not breadth: SML’s entire Bing citation presence still rides on exactly 1 grounding query (“payment networks supporting x402 protocol,” 27.54% share). 188 citations on one question is concentration risk, not authority — growing the query count is the work ahead, tracked openly in the Cite Ledger.
• The money page is not the cited page: /verdict-protocol — the page carrying SML’s live x402 settlement evidence — has earned no citations yet; the most-cited page is a news analysis (/robinhood-agents-approval-setting, 46). Engines cite the commentary before the receipts. We say so here because this page argues receipts are what matter.
• Citation ≠ traffic ≠ revenue: a citation is a named source inside an answer. It is not a click, a visit, or a sale, and this page claims none of those.
• The Gemini cost is real: blocking Google-Extended also blocks certain Gemini/Vertex grounding uses (per the policy’s own comment). If Gemini grounding is your primary channel, this exact split costs more than it costs SML.

Canonical: https://scriptmasterlabs.com/cite-me-live
Script Master Labs LLC · Service-Disabled Veteran-Owned Small Business · Policy: robots.txt · Record: Cite Ledger