SEO checks

Technical SEO straight from the command line: uncover redirect chains, check the status codes of many URLs, test robots.txt, sitemaps, canonical tags and response times. Faster than any browser extension and easy to automate.

24 one-liners: 24 harmless, 0 caution, 0 destructive

curl 22 seo 21 status-code 3 sitemap 3 crawling 3 xargs 2 noindex 2 googlebot 2 https 2 html 2 redirect 2

Check for soft 404s: does a missing page really return 404

$curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' https://example.com/diese-seite-gibt-es-nicht-$RANDOM

Check maintenance mode for status 503 and Retry-After

$curl -s -o /dev/null -D - https://example.com/ | grep -iE '^(HTTP/|retry-after:)'

Check the status code of all sitemap.xml URLs in parallel

$curl -s https://example.com/sitemap.xml | grep -oP '(?<=<loc>)[^<]+' | xargs -P 8 -I{} curl -o /dev/null -s -w '%{http_code} %{url_effective}\n' {} | grep -v '^200 ' | sort

Check the X-Robots-Tag header of a URL

$curl -s -o /dev/null -D - https://example.com/ | grep -i '^x-robots-tag'

Check whether the server returns 304 Not Modified

$curl -s -o /dev/null -w '%{http_code}\n' -H "If-Modified-Since: $(curl -sI https://example.com/ | grep -i '^last-modified:' | cut -d' ' -f2- | tr -d '\r')" https://example.com/

Collect all URLs from a sitemap index

$curl -s https://example.com/sitemap_index.xml | grep -oP '(?<=<loc>)[^<]+' | xargs -n1 curl -s | grep -oP '(?<=<loc>)[^<]+' | sort -u

Server is clean. Is the site?

How fast does the website really load?

GENLOC.SEO measures PageSpeed and Core Web Vitals on mobile and desktop and tells you clearly what to fix first. Free, no time limit.

by GENLOC.NETWORK, the team behind myline.de

Extract all URLs from a sitemap.xml

$curl -s https://example.com/sitemap.xml | grep -oP '(?<=<loc>)[^<]+' | sort -u

Extract and count the internal links of a page

$curl -s https://example.com/ | tr '\n' ' ' | grep -oiE 'href="[^"]+"' | sed -E 's/^href="//I; s/"$//; s/#.*//' | grep -E '^(/|https?://(www\.)?example.com)' | sort | uniq -c | sort -rn

Extract the H1 headings of a page

$curl -s https://example.com/ | tr '\n' ' ' | grep -oiP '<h1[^>]*>.*?</h1>' | sed -E 's/<[^>]+>//g; s/^\s+|\s+$//g'

Extract the structured data (JSON-LD) of a page

$curl -s https://example.com/ | tr '\n' ' ' | grep -oP '<script[^>]*application/ld\+json[^>]*>.*?</script>'

Fetch a page as Googlebot and compare with a normal request

$for ua in 'Mozilla/5.0 (X11; Linux x86_64)' 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'; do curl -s -o /dev/null -A "$ua" -w "%{http_code} %{size_download} Bytes $ua\n" https://example.com/; done

Fetch robots.txt and show the Disallow rules

$curl -s https://example.com/robots.txt | grep -iE '^\s*(user-agent|disallow|allow|sitemap):'

Find broken links on a website with wget --spider

$LC_ALL=C wget --spider -r -l 2 -nd -nv -w 1 -o /tmp/spider.log https://example.com/; grep -A 100 'broken link' /tmp/spider.log

Find images without an alt attribute on a page

$curl -s https://example.com/ | tr '\n' ' ' | grep -oiE '<img[^>]*>' | grep -viE '\balt='

Find meta robots and noindex in the HTML of a page

$curl -s https://example.com/ | tr '\n' ' ' | grep -oiE '<meta[^>]*name=.?(robots|googlebot)[^>]*>'

Find mixed content in the HTML of an HTTPS page

$curl -s https://example.com/ | tr '\n' ' ' | grep -oiE '(src|srcset|data-src)="http://[^"]+"|<link[^>]+href="http://[^"]+"|url\(.?http://[^)]+\)' | sort -u

Measure the HTML and header size of a page

$curl -s -o /dev/null -w 'Status: %{http_code} Header: %{size_header} Bytes HTML: %{size_download} Bytes\n' https://example.com/

Measure TTFB repeatedly with minimum, median and maximum

$for i in $(seq 10); do curl -o /dev/null -s -w '%{time_starttransfer}\n' https://example.com/; done | LC_ALL=C sort -n | awk '{a[NR]=$1; s+=$1} END {printf "Min: %.3fs Median: %.3fs Max: %.3fs Schnitt: %.3fs\n", a[1], a[int((NR+1)/2)], a[NR], s/NR}'

Read the canonical tag of a page with curl

$curl -s https://example.com/ | tr '\n' ' ' | grep -oiE '<link[^>]*canonical[^>]*>'

Read the hreflang tags of a page

$curl -s https://example.com/ | tr '\n' ' ' | grep -oiE '<link[^>]*hreflang[^>]*>'

Read the title and meta description of a page with curl

$curl -s https://example.com/ | tr '\n' ' ' | grep -oiE '<title[^>]*>[^<]*</title>|<meta[^>]*name=.?description[^>]*>'

Show the number of redirects and the final URL

$curl -sL -o /dev/null -w 'Weiterleitungen: %{num_redirects}\nZiel: %{url_effective}\nStatus: %{http_code}\n' https://example.com/

Test http, https, www and non-www at once

$for u in http://example.com/ http://www.example.com/ https://example.com/ https://www.example.com/; do curl -s -o /dev/null -m 10 -w "%{http_code} $u -> %{redirect_url}\n" "$u"; done

Verify a real Googlebot via reverse DNS

$n=$(host 66.249.66.1 | awk '/pointer/ {print $NF}'); echo "PTR: $n"; host "$n"