DE

↑ ↓ select, Enter open, Esc close

Check the X-Robots-Tag header of a URL

harmlessSEO checksdebian, ubuntu
Adjust the values, the command updates live
user@server
curl -s -o /dev/null -D - https://example.com/ | grep -i '^x-robots-tag'

Requests the URL via GET and shows only the X-Robots-Tag header. It works like a meta robots tag, but lives in the HTTP header and is therefore easily overlooked when checking the page source. Also important for PDFs and images, which have no HTML head.

Note: No output means no such header is set. Typical sources are Header set X-Robots-Tag in .htaccess, add_header in nginx, and staging plugins.

Also searched as

  • noindex in http header
  • why is my page not indexed
  • check pdf for noindex
  • show x-robots-tag header curl

Related one-liners

All in SEO checks

Find meta robots and noindex in the HTML of a page

$curl -s https://example.com/ | tr '\n' ' ' | grep -oiE '<meta[^>]*name=.?(robots|googlebot)[^>]*>'
harmlessSEO checks

Check for soft 404s: does a missing page really return 404

$curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' https://example.com/diese-seite-gibt-es-nicht-$RANDOM
harmlessSEO checks

Check maintenance mode for status 503 and Retry-After

$curl -s -o /dev/null -D - https://example.com/ | grep -iE '^(HTTP/|retry-after:)'
harmlessSEO checks

Check the status code of all sitemap.xml URLs in parallel

$curl -s https://example.com/sitemap.xml | grep -oP '(?<=<loc>)[^<]+' | xargs -P 8 -I{} curl -o /dev/null -s -w '%{http_code} %{url_effective}\n' {} | grep -v '^200 ' | sort
harmlessSEO checks

Collect all URLs from a sitemap index

$curl -s https://example.com/sitemap_index.xml | grep -oP '(?<=<loc>)[^<]+' | xargs -n1 curl -s | grep -oP '(?<=<loc>)[^<]+' | sort -u
harmlessSEO checks

Extract all URLs from a sitemap.xml

$curl -s https://example.com/sitemap.xml | grep -oP '(?<=<loc>)[^<]+' | sort -u
harmlessSEO checks

Read first, then run.

The commands on myline.de act directly on servers, files and databases. A wrong path or placeholder can delete data irreversibly or make a server unreachable.

  • All commands are provided without warranty and are not tested on every system.
  • Understand what a command does before running it, and check every placeholder.
  • Make a backup first and, if possible, try it on a test system.
  • You run commands at your own risk. Liability for damages is excluded to the extent permitted by law.