robots.txt Wildcard Rules Explained: Syntax, Precedence, and Examples

The robots.txt file is the front gate of your website. Located at the root of your domain, this plain text file provides crawl directives to search engine bots, web crawlers, and automated indexing agents. When configured accurately, it protects your server resources, prevents the crawling of internal utility scripts, and focuses crawler attention on indexable, high-value content.

However, writing rules for websites with thousands or millions of dynamic URLs, faceted parameters, and diverse file formats quickly becomes impossible using static paths alone. To address this complexity, the Robots Exclusion Protocol supports wildcard pattern-matching operators.

Under the official IETF RFC 9309 standard^[1]Source 1Internet Engineering Task Force (IETF). RFC 9309: Robots Exclusion Protocol.View source ↗ and Google Search Central robots.txt specifications^[2]Source 2Google Search Central. robots.txt Specifications and File Format Guidelines.View source ↗, webmasters can use two primary pattern-matching symbols: the asterisk (*) and the dollar sign ($). When combined with Allow and Disallow directives, these characters allow you to target complex URL patterns across entire directories.

Yet wildcards are often misunderstood. A single stray character can inadvertently block your entire website from search engine crawlers, or fail to block sensitive parameters because of subtle path-matching nuances.

This guide provides a comprehensive technical breakdown of robots.txt wildcard rules. You will learn how path matching works by default, how to combine the asterisk and dollar sign operators, how search engines resolve conflicting directives using the RFC 9309 longest-match rule, and how to verify your syntax using Google Search Console.

---

How Path Matching Works by Default: Implicit Prefix Matching

Before introducing wildcards, it is essential to understand how search engine crawlers evaluate standard URL paths in robots.txt.

In standard crawl parsing, all path rules are prefix matches by default.

When you write a directive like Disallow: /catalog, you are not matching only the exact URL /catalog. You are instructing the crawler to block any URL path that starts with /catalog.

Target Directive: Disallow: /catalog
 
Matching Behavior:
- https://example.com/catalog             [BLOCKED - Exact match]
- https://example.com/catalog/            [BLOCKED - Trailing slash match]
- https://example.com/catalog/shoes       [BLOCKED - Subdirectory match]
- https://example.com/catalog.html        [BLOCKED - Extension match]
- https://example.com/catalogue           [BLOCKED - Prefix word match!]

Because search engine crawlers treat the end of any rule as an implicit wildcard, writing Disallow: /catalog* is functionally redundant. The trailing asterisk is already assumed.

The Critical Difference Between /path and /path/

One of the most common beginner mistakes in technical SEO is confusing a prefix without a slash with a prefix with a slash:

• Disallow: /private blocks /private, /private/, /private-access, and /privacypolicy.
• Disallow: /private/ blocks only URLs within the /private/ directory (such as /private/report.html), while leaving /privacypolicy completely crawlable.

Unless you intend to match all filenames starting with a specific word stem, always terminate directory paths with a forward slash.

---

The Two Official Wildcard Operators: Asterisk and Dollar Sign

The Robots Exclusion Protocol supports two wildcard operators to modify default prefix behavior.

1. The Asterisk Wildcard (*)

The asterisk represents zero or more instances of any valid character within a URL path.

You can use the asterisk in three primary ways:

• Inside a directory path: To match dynamic folders. Disallow: /users/*/settings/ This matches /users/john/settings/ and /users/sarah/settings/.
• To target query parameters: To match parameters regardless of preceding path structure. Disallow: /*?sort= This matches any URL containing the ?sort= query string anywhere in the path.
• To target file extensions anywhere on the site: Disallow: /*.pdf This matches any URL that contains .pdf. Note that without an end anchor, this rule will also match /document.pdf.bak or /document.pdf?version=2.

2. The Dollar Sign End Anchor ($)

The dollar sign asserts the exact end of the URL string. It tells the crawler: "This rule only applies if the URL terminates immediately after this character."

By anchoring a pattern with $, you override the default implicit prefix behavior:

• Disallow: /exact-page$ matches https://example.com/exact-page, but does NOT match https://example.com/exact-page/ or https://example.com/exact-page?utm=1.
• Disallow: /*.pdf$ matches any URL that ends strictly with the .pdf file extension, but does NOT match /file.pdf?download=true.

---

Pattern-Matching Reference Table

The table below demonstrates how different combinations of wildcards, slashes, and anchors interact with sample URLs:

Directive in robots.txt	Target Evaluated URL	Result	Technical Explanation
Disallow: /shoes	https://example.com/shoes	BLOCKED	Exact prefix match
Disallow: /shoes	https://example.com/shoes/running	BLOCKED	Subdirectory matches default prefix
Disallow: /shoes	https://example.com/shoestrings	BLOCKED	Prefix matches without requiring slash
Disallow: /shoes/	https://example.com/shoestrings	ALLOWED	Trailing slash requires directory boundary
Disallow: /shoes/	https://example.com/shoes/running	BLOCKED	Matches path within shoes directory
Disallow: /shoes$	https://example.com/shoes	BLOCKED	Exact match terminated by dollar sign anchor
Disallow: /shoes$	https://example.com/shoes/	ALLOWED	Dollar sign prevents trailing slash match
Disallow: /*.xls$	https://example.com/data.xls	BLOCKED	Asterisk matches filename; dollar sign anchors .xls
Disallow: /*.xls$	https://example.com/data.xlsx	ALLOWED	Dollar sign prevents matching trailing x
Disallow: /*?*	https://example.com/page?id=12	BLOCKED	Asterisk matches query string presence
Disallow: /*?*	https://example.com/page	ALLOWED	URL lacks question mark query separator

---

Conflict Resolution: The Longest Match Rule (RFC 9309)

When building a scalable robots.txt file, you will inevitably have overlapping Allow and Disallow rules that both match a single URL.

For example, you might want to disallow an entire directory while allowing a specific file inside that directory, or disallow all parameters while allowing a specific pagination parameter.

Historically, different search engines handled rule conflicts differently. Some read sequentially from top to bottom, while others prioritized Allow directives arbitrarily.

Under IETF RFC 9309, the conflict resolution mechanism is standardized across all compliant search engines, including Google, Bing, and Yandex: the most specific rule (the rule with the longest character match) always wins.

How Character Match Length Is Calculated

To determine which rule takes precedence, the crawler calculates the number of characters in the matching directive path:

1. The crawler compares all applicable directives against the candidate URL.
2. It counts the number of octets (characters) in each matching rule path.
3. The directive with the highest character count is enforced.
4. Line order in the file does not matter. An Allow rule at the bottom of the file will defeat a Disallow rule at the top if the Allow rule has a longer matching path.

The Tie-Breaker Rule

If a candidate URL matches both an Allow and a Disallow directive that have the exact same character length, the Allow directive takes precedence.

Google and the RFC 9309 specification treat access permission as the preferred default in the event of an exact mathematical tie.

---

Specificity in Action: Match Length Evaluation Table

The table below illustrates how crawlers evaluate character length and determine the winning directive across common URL conflict scenarios:

Disallow Rule	Allow Rule	Candidate URL	Disallow Length	Allow Length	Winning Rule	Final Crawl State
Disallow: /	Allow: /blog/	https://example.com/blog/seo	1 char (/)	6 chars (/blog/)	Allow (6 > 1)	ALLOWED
Disallow: /catalog/	Allow: /catalog/sale.html	https://example.com/catalog/sale.html	9 chars	18 chars	Allow (18 > 9)	ALLOWED
Disallow: /admin/users/	Allow: /admin/	https://example.com/admin/users/list	13 chars	7 chars	Disallow (13 > 7)	BLOCKED
Disallow: /private	Allow: /private	https://example.com/private	8 chars	8 chars	Allow (Tie-breaker)	ALLOWED
Disallow: /*?	Allow: /*?page=	https://example.com/shop?page=2	3 chars	8 chars	Allow (8 > 3)	ALLOWED
Disallow: /temp/*.pdf	Allow: /temp/	https://example.com/temp/doc.pdf	11 chars	6 chars	Disallow (11 > 6)	BLOCKED

Notice the fifth row in the table above: Disallow: /*? (3 characters) competing against Allow: /*?page= (8 characters). This specificity mechanism makes clean parameter filtering possible.

---

Real-World Wildcard Configurations for Technical SEO

Below are tested, production-grade wildcard rules addressing complex real-world crawl scenarios.

1. Blocking All Tracking Parameters While Allowing Pagination

E-commerce websites often generate infinite parameter combinations from marketing campaigns (?utm_source, ?fbclid, ?gclid, ?sessionid) alongside vital category pagination (?page=2, ?p=3).

To block all parameter variations without trapping paginated product archives:

User-agent: *
# Step 1: Block any URL that contains a question mark
Disallow: /*?
 
# Step 2: Create a longer, more specific Allow rule for pagination
Allow: /*?page=
Allow: /*?p=

Because /*?page= (8 characters) and /*?p= (5 characters) are longer than /*? (3 characters), Googlebot will crawl paginated URLs while blocking all tracking and session queries.

2. Blocking Non-HTML Document Downloads Site-Wide

To prevent search engines from crawling and indexing PDF brochures, spreadsheets, and presentations:

User-agent: *
# Block all URLs ending strictly with designated file extensions
Disallow: /*.pdf$
Disallow: /*.xlsx$
Disallow: /*.docx$
Disallow: /*.pptx$

The dollar sign ($) is critical here. It ensures that an HTML page located at /products/pdf-reader/ is not blocked, while /downloads/spec-sheet.pdf is blocked.

3. Blocking Internal Site Search and Filtering Combinations

Internal search results pages should never be crawled or indexed:

User-agent: *
# Block standard search queries
Disallow: /search?*
Disallow: /search/*
 
# Block multi-attribute filter sorting
Disallow: /*?*sort=*
Disallow: /*?*filter=*

4. Permitting Image Crawling While Blocking Media Folders

If you store assets in an /assets/ directory that contains sensitive PDFs, scripts, and images, but want Google Image Search to index product photos:

# General web crawlers are blocked from the assets directory
User-agent: *
Disallow: /assets/
 
# Google Image Search crawler is granted specific access to images
User-agent: Googlebot-Image
Allow: /assets/*.jpg$
Allow: /assets/*.png$
Allow: /assets/*.webp$
Disallow: /assets/

Because specific User-agent blocks override the universal User-agent: * block for that designated bot, Googlebot-Image will follow its dedicated rules while all other crawlers respect the general disallow.

---

Common Wildcard Pitfalls and Anti-Patterns

Implementing wildcards requires strict attention to protocol limits. Avoid the following common traps:

1. Assuming Full Regular Expression Support

A frequent misconception among developers is that robots.txt supports full POSIX or PCRE regular expressions.

It does not.

The Robots Exclusion Protocol only supports the asterisk (*) and dollar sign ($). Operators common in regular expressions like +, ? (as a single-character wildcard), [0-9], (abc|def), or {1,3} are treated as literal characters, not regex commands!

• Writing Disallow: /product/[0-9]+ will look for a URL containing literal brackets and plus signs.
• It will NOT match /product/123.

2. Forgetting Case Sensitivity

Under RFC 9309, path matching is strictly case-sensitive.

• Disallow: /*.pdf$ blocks /manual.pdf.
• It does NOT block /manual.PDF or /manual.Pdf.

If your web server or CMS serves uppercase or mixed-case file extensions, declare rules for both:

Disallow: /*.pdf$
Disallow: /*.PDF$

3. Forgetting the Leading Forward Slash

Every path directive in robots.txt must begin with a forward slash (/).

• Disallow: /admin/ is valid.
• Disallow: admin/ is invalid under RFC 9309 and may be ignored by crawlers.

4. Expecting robots.txt to Deindex Existing Pages

Remember that blocking a URL in robots.txt does not remove it from Google Search. If external backlinks or internal anchor links point to that URL, Google may still index the page without content.

As detailed in our guide to URLs blocked by robots.txt, search engines cannot read a noindex tag or HTTP header if the page is disallowed in robots.txt. If you need a page removed from the search index, use a noindex directive and ensure the URL is crawlable.

---

Auditing and Testing Wildcard Rules

Never publish changes to robots.txt without rigorous testing against live URL samples.

1. Using the Google Search Console robots.txt Report

According to official Google Search Console robots.txt documentation^[3]Source 3Google Search Console Help. robots.txt Report: Checking Fetch Status and Errors.View source ↗, you can inspect how Googlebot parses your production file:

1. Open Google Search Console and navigate to Settings > robots.txt.
2. Review the Fetch status and Last crawl time to confirm Google has fetched your latest version.
3. Check for syntax warnings or unparsed lines.

2. Verifying URLs with the URL Inspection Tool

According to official Google Search Console URL Inspection documentation^[4]Source 4Google Search Console Help. URL Inspection Tool: Testing Live URLs Against robots.txt.View source ↗, you can test individual URLs in real time:

1. Paste your target URL into the top search bar.
2. Click Test Live URL.
3. Expand the Page availability section.
4. Check Crawl allowed?:
   • If permitted, it will display: Yes.
   • If blocked by a wildcard rule, it will display: No: blocked by robots.txt.

3. Pre-Flight Testing in Screaming Frog

Before pushing a modified robots.txt file to your production server:

1. Open Screaming Frog SEO Spider.
2. Navigate to Configuration > Robots.txt > Custom.
3. Paste your draft robots.txt content into the testing window.
4. In the Test URLs tab, enter a batch of URLs representing allowed pages, blocked parameters, and edge cases.
5. Review the Status column to verify that each URL matches the intended directive before going live.

---

Actionable Validation Checklist for Webmasters

Before updating robots.txt on a production environment, complete this pre-flight verification:

• Check Operator Syntax: Ensure you have only used the supported * and $ wildcards, with no unsupported regex syntax.
• Verify Leading Slashes: Every path in every Allow and Disallow rule must start with a leading forward slash (/).
• Anchor Extensions with Dollar Signs: Ensure file extension blocks terminate with $ (/*.pdf$) to avoid blocking unintended subdirectories.
• Calculate Match Lengths: When pairing Allow and Disallow rules, verify that the exception rule has a higher character count than the general block.
• Account for Case Sensitivity: Include uppercase variants (.PDF, .JPG) if your server supports mixed-case URLs.
• Preserve Critical Crawl Paths: Confirm that wildcards targeting parameters (/*?) do not inadvertently block pagination (?page=) or language selectors.
• Test Live in Search Console: Run sample URL checks in Google Search Console's URL Inspection tool to confirm that Googlebot matches the expected rule.

Sources

1. Internet Engineering Task Force (IETF). RFC 9309: Robots Exclusion Protocol.

2. Google Search Central. robots.txt Specifications and File Format Guidelines.

3. Google Search Console Help. robots.txt Report: Checking Fetch Status and Errors.

4. Google Search Console Help. URL Inspection Tool: Testing Live URLs Against robots.txt.