DE

↑ ↓ select, Enter open, Esc close

Fetch robots.txt and show the Disallow rules

harmlessSEO checksdebian, ubuntu
Adjust the values, the command updates live
user@server
curl -s https://example.com/robots.txt | grep -iE '^\s*(user-agent|disallow|allow|sitemap):'

Fetches robots.txt and filters out comments and empty lines. This quickly shows whether a Disallow: / from the development phase was accidentally left in place and whether the sitemap is listed. The URL must end with a slash.

Note: On WordPress without a physical file, WordPress generates robots.txt dynamically. curl -sI {{URL}}robots.txt checks the status code, and a 5xx there can stop crawling entirely.

Also searched as

  • check robots.txt disallow rules
  • what does robots.txt block
  • is my site blocked for google
  • show robots.txt command line

Related one-liners

All in SEO checks

Check for soft 404s: does a missing page really return 404

$curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' https://example.com/diese-seite-gibt-es-nicht-$RANDOM
harmlessSEO checks

Check maintenance mode for status 503 and Retry-After

$curl -s -o /dev/null -D - https://example.com/ | grep -iE '^(HTTP/|retry-after:)'
harmlessSEO checks

Check the status code of all sitemap.xml URLs in parallel

$curl -s https://example.com/sitemap.xml | grep -oP '(?<=<loc>)[^<]+' | xargs -P 8 -I{} curl -o /dev/null -s -w '%{http_code} %{url_effective}\n' {} | grep -v '^200 ' | sort
harmlessSEO checks

Check the X-Robots-Tag header of a URL

$curl -s -o /dev/null -D - https://example.com/ | grep -i '^x-robots-tag'
harmlessSEO checks

Check whether the server returns 304 Not Modified

$curl -s -o /dev/null -w '%{http_code}\n' -H "If-Modified-Since: $(curl -sI https://example.com/ | grep -i '^last-modified:' | cut -d' ' -f2- | tr -d '\r')" https://example.com/
harmlessSEO checks

Collect all URLs from a sitemap index

$curl -s https://example.com/sitemap_index.xml | grep -oP '(?<=<loc>)[^<]+' | xargs -n1 curl -s | grep -oP '(?<=<loc>)[^<]+' | sort -u
harmlessSEO checks

Read first, then run.

The commands on myline.de act directly on servers, files and databases. A wrong path or placeholder can delete data irreversibly or make a server unreachable.

  • All commands are provided without warranty and are not tested on every system.
  • Understand what a command does before running it, and check every placeholder.
  • Make a backup first and, if possible, try it on a test system.
  • You run commands at your own risk. Liability for damages is excluded to the extent permitted by law.