DE

↑ ↓ select, Enter open, Esc close

Find broken links on a website with wget --spider

harmlessSEO checksdebian, ubuntu
Adjust the values, the command updates live
user@server
LC_ALL=C wget --spider -r -l 2 -nd -nv -w 1 -o /tmp/spider.log https://example.com/; grep -A 100 'broken link' /tmp/spider.log

Follows all links up to the given depth, only checks whether they are reachable and writes the log to a file. At the end, all unreachable URLs are listed under “Found N broken links”. Internal 404s waste crawl budget and hurt the user experience.

Note: -w 1 waits one second between requests, so large sites take a while. wget respects robots.txt, so disallowed areas are not checked. LC_ALL=C forces English messages so that grep matches.

Also searched as

  • find broken links command line
  • wget spider broken link check
  • find 404 links on website linux
  • crawl website for dead links

Related one-liners

All in SEO checks

Fetch robots.txt and show the Disallow rules

$curl -s https://example.com/robots.txt | grep -iE '^\s*(user-agent|disallow|allow|sitemap):'
harmlessSEO checks

Find the most frequent 404 errors in the access log

#awk '$9 == 404 {print $7}' /var/www/vhosts/system/example.com/logs/access_ssl_log | sort | uniq -c | sort -rn | head -20
harmlessLogs

Regenerate permalinks and rewrite rules

$wp rewrite flush --path=/var/www/vhosts/example.com/httpdocs
cautionWP-CLI

Check for soft 404s: does a missing page really return 404

$curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' https://example.com/diese-seite-gibt-es-nicht-$RANDOM
harmlessSEO checks

Check maintenance mode for status 503 and Retry-After

$curl -s -o /dev/null -D - https://example.com/ | grep -iE '^(HTTP/|retry-after:)'
harmlessSEO checks

Check the status code of all sitemap.xml URLs in parallel

$curl -s https://example.com/sitemap.xml | grep -oP '(?<=<loc>)[^<]+' | xargs -P 8 -I{} curl -o /dev/null -s -w '%{http_code} %{url_effective}\n' {} | grep -v '^200 ' | sort
harmlessSEO checks

Read first, then run.

The commands on myline.de act directly on servers, files and databases. A wrong path or placeholder can delete data irreversibly or make a server unreachable.

  • All commands are provided without warranty and are not tested on every system.
  • Understand what a command does before running it, and check every placeholder.
  • Make a backup first and, if possible, try it on a test system.
  • You run commands at your own risk. Liability for damages is excluded to the extent permitted by law.