This organizes how URLs with and without -aiarticle are handled by RAG systems and search engines. There are broadly two control mechanisms.
- Crawler filter (
.*-aiarticle/): control internal to the RAG indexer. Directly decides which URLs get registered in the index. rel=canonical: control aimed at search engines. Declares "the canonical URL is here," preventing ranking signals from being diluted across duplicate content.
Since these two play different roles, use them separately according to purpose, or combine them.
1. Overall flow
RAG indexer processing flow (flowchart)
- The crawler's filter setting effectively determines "which one gets picked up"
- The LLM only sees "URLs that are registered in the index and rank highly in search"
Timeline from query to answer (sequence diagram)
- The LLM only receives "the content of URLs chosen by the search API"
- The URL's path name (whether or not it has
-aiarticle) carries no meaning for the LLM — it's just a string
Patterns the designer can control (use case diagram)
| Use case | Main setting |
|---|---|
| Want both URLs to be referenceable by the LLM | Crawler: all paths targeted / Filter: none |
| Want the LLM to use only aiarticle | Crawler: limited to .*-aiarticle/ / Filter: aiarticle allowed |
| Want to completely exclude aiarticle from the LLM | Crawler: exclude .*-aiarticle/ / noindex via robots.txt |
| Want to treat normal articles and aiarticle as identical | Unify to one with rel=canonical |
Supplement: scope of the LLM's own involvement
The LLM only sees the "context that is passed to it." The URL's path name or the position of the slug carries no meaning for it; "which one gets picked up" is determined entirely by the design of the RAG/search infrastructure. In practice, then, the question isn't "which one will the LLM pick up?" but rather a matter of designing and configuring "which one gets registered in the RAG indexer?"
2. Role of the control mechanisms: crawler filter vs. canonical
Crawler filter (.*-aiarticle/)
A setting that controls which URLs the RAG crawler registers in its index.
- If you exclude
.*-aiarticle/, the LLM cannot reference that page at all - If you limit to
.*-aiarticle/, the LLM can only reference that page - Directly controls behavior inside the RAG system, and is self-contained within the RAG, independent of search engines
- Suited for: cases where you want to clearly separate AI-facing articles from human-facing articles
rel=canonical
An HTML tag that declares to search engines, "this page's canonical URL is here." Placed on the duplicate page (the one with -aiarticle).
<!-- Placed on the page with -aiarticle -->
<head>
<link rel="canonical" href="https://example.com/docs/xj6w3n/llm-rag-hallucination-guide-2026/" />
</head>
- Prevents search engines from treating it as duplicate content, and consolidates ranking signals onto a single URL
- This is a control aimed at search engines, and whether the RAG indexer respects it depends on the implementation (many crawlers do reference canonical, but it isn't mandatory)
- Suited for: cases where identical content exists at multiple URLs and you want search engines to recognize only one
Should you use both?
Using both is the most robust approach. Control things explicitly inside the RAG with the .*-aiarticle/ filter, and for search engines, declare the canonical URL via rel=canonical on the -aiarticle page — that way "which one is authoritative" stays consistent across both the RAG and search engines.
There are also cases where just one is enough.
- Filter only: control that's self-contained within the RAG is sufficient, you don't care about SEO, or it's already handled elsewhere
- canonical only: the main goal is deduplication in search engines, on the premise that the RAG side's implementation references canonical
3. Comparing crawler filter behaviors
Whether you "apply" or "don't apply" the .*-aiarticle/ filter fundamentally changes which documents the LLM can reference.
- Not applied: both URLs get registered in the index, and the LLM can reference both
- Applied (exclude): the aiarticle version is excluded, and the LLM can only reference the normal article
- Applied (limit): only the aiarticle version gets registered, and the LLM can only reference aiarticle
Without a filter (default)
| Step | Behavior |
|---|---|
| Crawling | Registers both URLs (normal article / -aiarticle) |
| Index | Both get registered as searchable |
| At search time | Whichever ranks higher in the ranking algorithm gets passed to the LLM |
| LLM reference | Which one ranks higher is unknown (varies with content similarity, SEO factors, etc.) |
The advantage is that no configuration is needed and both versions are kept, but which one gets referenced is unpredictable, carrying the risk of competition in search results and of the LLM referencing the unintended version.
With a filter (excluding .*-aiarticle/)
| Step | Behavior |
|---|---|
| Crawling | Excludes URLs matching .*-aiarticle/ |
| Index | Only normal articles get registered |
| At search time | The aiarticle version doesn't exist as a search target |
| LLM reference | References normal articles only |
You can clearly control which documents the LLM references, and there's no competition in search results. On the other hand, you can't let the LLM use the aiarticle version, so optimizations specific to the aiarticle version may go to waste.
With a filter (limited to .*-aiarticle/)
| Step | Behavior |
|---|---|
| Crawling | Registers only URLs matching .*-aiarticle/ |
| Index | Only the aiarticle version gets registered |
| At search time | Normal articles don't exist as a search target |
| LLM reference | References aiarticle only |
You can fully leverage the aiarticle version's optimizations, and there's no competition in search results. On the other hand, you can't let the LLM use the normal article version, so the normal article version's SEO ranking signals aren't reflected on the LLM side.
Comparison table of the three patterns
| Setting | Index registration | Documents the LLM references | Search result competition | Controllability |
|---|---|---|---|---|
| No filter | Both | Either (depends on ranking) | Yes | Low |
| Filter (exclude) | Normal articles only | Normal articles only | None | High |
| Filter (limit) | aiarticle only | aiarticle only | None | High |
Implementation example (regex filter)
# Exclude pattern
CRAWL_FILTER = {
"exclude": [r".*-aiarticle/"]
}
# Limit pattern
CRAWL_FILTER = {
"include": [r".*-aiarticle/"]
}
| Setting | Matching URL | Non-matching URL |
|---|---|---|
| Exclude | With -aiarticle ❌ |
Normal article ✅ |
| Limit | With -aiarticle ✅ |
Normal article ❌ |
Recommendations by case
| Case | Recommended setting | Reason |
|---|---|---|
| Want to use aiarticle as an "AI-only version" | Limit | Fully leverages aiarticle's optimizations |
| Don't want the AI to see aiarticle | Exclude | Have the LLM reference only normal articles |
| Want to treat both as "the same content" | Exclude + canonical | Consolidates ranking signals onto the normal article while preventing search competition |
In practice, "controlling things clearly with a filter" is the basic policy. No filter is not recommended, since you can't predict which one the LLM will reference.
4. Comparing rel=canonical behaviors
Whether you "apply" or "don't apply" rel=canonical changes which URL search engines treat as canonical.
- Not applied: search engines decide automatically (risk of competition and diluted ranking signals)
- Applied (set on the
-aiarticleversion): explicitly declares the normal article side as canonical (consolidates ranking signals)
The basic syntax is as follows. Specify the URL you want to be canonical (the normal article side) in href; only one per page, written inside the <head> tag.
<head>
<link rel="canonical" href="https://example.com/docs/xj6w3n/llm-rag-hallucination-guide-2026/" />
</head>
Without canonical (default)
| Step | Behavior |
|---|---|
| Index | Both URLs may get registered in the index separately |
| Canonical determination | The search engine automatically judges "which is the original" |
| Search results | Risk of both being shown (ranking signals diluted) |
| SEO ranking | Backlink/internal-link signals get diluted across 2 URLs |
No configuration is needed, but there's a risk of competition in search results, diluted SEO ranking signals, and the unintended version being judged canonical.
With canonical (set on the -aiarticle version)
| Step | Behavior |
|---|---|
| Index | Both URLs get indexed, but the -aiarticle version is treated as a "copy" |
| Canonical determination | The normal article side is explicitly designated as the canonical URL |
| Search results | Only the normal article is shown (the -aiarticle version is not shown) |
| SEO ranking | Backlink/internal-link signals get consolidated onto the normal article |
<!-- Written in the <head> of the -aiarticle page -->
<head>
<title>LLM RAG Hallucination Guide 2026 (AI Article)</title>
<link rel="canonical" href="https://example.com/docs/xj6w3n/llm-rag-hallucination-guide-2026/" />
</head>
While this avoids competition in search results and consolidates SEO ranking signals onto a single URL, the aiarticle version stops appearing in search results.
Comparison table of the three patterns
| Setting | Index registration | Search result display | SEO ranking | Controllability |
|---|---|---|---|---|
| No canonical | Both | Both (competition risk) | Diluted | Low |
| canonical set (normal article designated) | Both (aiarticle treated as a copy) | Normal article only | Consolidated onto the normal article | High |
| canonical set (self-referencing) | Both (treated independently) | Both | Independent per URL | Medium |
Common mistakes
- Setting canonical on both pages pointing to each other → the search engine may ignore it
- Specifying a nonexistent URL in href → treated as an error
- Setting multiple canonicals on a single page → invalid
- Setting canonical on a noindex page → results in contradictory instructions
5. Combining the filter and canonical
Combination patterns
| Filter setting | canonical setting | Search results | LLM reference within the RAG |
|---|---|---|---|
| None | None | Both (competition) | Either (depends on ranking) |
| Exclude | None | Normal article only | Normal article only |
| Exclude | Normal article designated | Normal article only | Normal article only |
| Limit | None | aiarticle only | aiarticle only |
| Limit | Normal article designated | Normal article only | aiarticle only (contradiction) |
The last row (the filter is set to limit, yet canonical designates the normal article) is a combination to avoid. The RAG registers only aiarticle, yet you're declaring to search engines that "the normal article is canonical" — so search results and what the RAG references end up mismatched.
Recommended patterns by objective
| Objective | Recommended setting | Reason |
|---|---|---|
| Want to use aiarticle as an "AI-only version" | Filter: limit + canonical: self-referencing | Fully leverages aiarticle's optimizations while preventing search competition |
| Don't want the AI to see aiarticle | Filter: exclude + canonical: normal article designated | Has the LLM reference only the normal article while consolidating SEO ranking signals |
| Want to treat both as "the same content" | Filter: exclude + canonical: normal article designated | Consolidates ranking signals onto the normal article while preventing search competition |
Implementation examples
Pattern A: want to use aiarticle as an "AI-only version"
<head>
<title>LLM RAG Hallucination Guide 2026 (AI Article)</title>
<!-- Designate itself as canonical (or don't specify one) -->
<link rel="canonical" href="https://example.com/docs/xj6w3n/llm-rag-hallucination-guide-2026-aiarticle/" />
</head>
CRAWL_FILTER = {
"include": [r".*-aiarticle/"]
}
Pattern B: don't want the AI to see aiarticle
<head>
<title>LLM RAG Hallucination Guide 2026 (AI Article)</title>
<!-- Designate the normal article side as canonical -->
<link rel="canonical" href="https://example.com/docs/xj6w3n/llm-rag-hallucination-guide-2026/" />
</head>
CRAWL_FILTER = {
"exclude": [r".*-aiarticle/"]
}
6. Summary
| Control mechanism | Target | Purpose | Recommended case |
|---|---|---|---|
.*-aiarticle/ filter |
RAG indexer | Technical URL control | Want clear separation within the RAG |
rel=canonical |
Search engine | SEO deduplication measure | Want to unify the URL in search results |
Rather than asking "which is better," the practical answer is to combine both according to your objective. If you want clear control within the RAG, use the .*-aiarticle/ filter; if you want to prevent duplication in search engines, set rel=canonical on the -aiarticle page. Setting both keeps behavior consistent across both the RAG and search engines.