DE

↑ ↓ select, Enter open, Esc close

Check the status code of all sitemap.xml URLs in parallel

harmlessSEO checksdebian, ubuntu
Adjust the values, the command updates live
user@server
curl -s https://example.com/sitemap.xml | grep -oP '(?<=<loc>)[^<]+' | xargs -P 8 -I{} curl -o /dev/null -s -w '%{http_code} %{url_effective}\n' {} | grep -v '^200 ' | sort

Checks all sitemap URLs with several concurrent requests and lists only those deviating from 200, with their status code. A clean sitemap contains only indexable pages with status 200, redirects and error pages in it waste crawl budget. No output means everything is fine.

Note: Choose the parallelism carefully, on weak servers or behind a WAF too many concurrent requests trigger blocks. For an arbitrary URL list from a file, replace curl -s ... | grep with cat datei.txt.

Also searched as

  • check sitemap for 404 errors
  • sitemap contains redirects
  • status codes of all sitemap urls
  • find broken urls in sitemap

Related one-liners

All in SEO checks

Collect all URLs from a sitemap index

$curl -s https://example.com/sitemap_index.xml | grep -oP '(?<=<loc>)[^<]+' | xargs -n1 curl -s | grep -oP '(?<=<loc>)[^<]+' | sort -u
harmlessSEO checks

Check for soft 404s: does a missing page really return 404

$curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' https://example.com/diese-seite-gibt-es-nicht-$RANDOM
harmlessSEO checks

Check maintenance mode for status 503 and Retry-After

$curl -s -o /dev/null -D - https://example.com/ | grep -iE '^(HTTP/|retry-after:)'
harmlessSEO checks

Extract all URLs from a sitemap.xml

$curl -s https://example.com/sitemap.xml | grep -oP '(?<=<loc>)[^<]+' | sort -u
harmlessSEO checks

Check the X-Robots-Tag header of a URL

$curl -s -o /dev/null -D - https://example.com/ | grep -i '^x-robots-tag'
harmlessSEO checks

Extract and count the internal links of a page

$curl -s https://example.com/ | tr '\n' ' ' | grep -oiE 'href="[^"]+"' | sed -E 's/^href="//I; s/"$//; s/#.*//' | grep -E '^(/|https?://(www\.)?example.com)' | sort | uniq -c | sort -rn
harmlessSEO checks

Read first, then run.

The commands on myline.de act directly on servers, files and databases. A wrong path or placeholder can delete data irreversibly or make a server unreachable.

  • All commands are provided without warranty and are not tested on every system.
  • Understand what a command does before running it, and check every placeholder.
  • Make a backup first and, if possible, try it on a test system.
  • You run commands at your own risk. Liability for damages is excluded to the extent permitted by law.