CompareTexts

CompareTexts / Technical SEO

Compare two robots.txt files

Paste an old and new robots.txt to catch changed crawler instructions before a launch or migration. Results keep directives attached to their user-agent groups.

Your inputs

Files and pasted text are read locally. Nothing is sent for comparison.

Comparison results

Choose inputs and run the comparison.

Worked example

A crawl rule changed at launch

Select Load example above to run this sample in your browser.

Original

User-agent: *; Disallow: /private/; Sitemap: /sitemap.xml

Revised

User-agent: *; Disallow: /staging/; Sitemap: /sitemap.xml

What the result means: The comparison marks /private/ removed and /staging/ added under the same crawler group. The unchanged Sitemap line is omitted.

How to use it

How to compare two robots.txt files

  1. STEP 01

    Paste the old and new robots.txt contents.

  2. STEP 02

    Compare directives by group rather than line number.

  3. STEP 03

    Review added and removed rules, especially Disallow and Sitemap entries.

Catch accidental crawling changes

A staging Disallow rule or a missing sitemap line can be easy to overlook in a file diff. This tool ignores blank lines and comments, normalizes directive names and shows rules that exist on only one side. Group context matters: the same path under a different user-agent is reported as a change.

What to check in the results

This is a directive comparison, not a search-engine crawler simulator. It does not resolve conflicting rules or fetch the live file. Paste text stays in your browser.

For a general line and character comparison, use the text diff tool. The text comparison guide explains a worked example, line counts, and how to keep shared links private.

Common questions

How do I spot a new Disallow rule after migration?

Paste the previous robots.txt on the left and the new file on the right. New rules appear as added and retain their user-agent group.

Does changing the order of robots.txt lines count as a change?

No. The tool compares directives within their user-agent groups rather than line positions. It still reports a rule moved to a different group.