Question 1

What is robots.txt?

Accepted Answer

Robots.txt is a file at the root of a website that tells web crawlers which pages they can and cannot access.

Question 2

How does this tester work?

Accepted Answer

Paste your robots.txt content and enter a URL to test. The tool parses the rules and checks if the URL is allowed or blocked.

Question 3

Which user-agents are supported?

Accepted Answer

You can test against any user-agent including Googlebot, Bingbot, and wildcard (*).

Question 4

What does Disallow mean?

Accepted Answer

Disallow tells crawlers not to access matching URLs. A path with Disallow alone blocks that path, while Disallow with no path allows everything.

Question 5

Can I test multiple URLs at once?

Accepted Answer

Currently, you test one URL at a time. Enter a new URL and click Test to check another path.

Question 6

How are wildcards handled?

Accepted Answer

The tool supports * wildcard in paths (e.g., Disallow: /private/* blocks all paths starting with /private/) and $ for end-of-path matching.

Question 7

What is the Sitemap directive?

Accepted Answer

The Sitemap directive in robots.txt points crawlers to your XML sitemap. It does not control access.

Question 8

Is this tool free?

Accepted Answer

Yes, completely free with no signup required.

🤖 Robots.txt Tester