DE

↑ ↓ select, Enter open, Esc close

Verify a real Googlebot via reverse DNS

harmlessSEO checksdebian, ubuntu
Adjust the values, the command updates live
user@server
n=$(host 66.249.66.1 | awk '/pointer/ {print $NF}'); echo "PTR: $n"; host "$n"

First resolves the IP in reverse to a hostname, then resolves that hostname forward to an IP again. A real Googlebot has a name under googlebot.com or google.com, and the forward lookup returns exactly the IP being checked. Many scrapers only pretend to be Googlebot via the user agent.

Note: host is part of the bind9-host package and is missing on minimal systems. The entry on Googlebot requests in the logs category counts requests by user agent.

Also searched as

  • verify googlebot ip reverse dns
  • is this really googlebot
  • detect fake googlebot
  • fake crawler in access log

Related one-liners

All in SEO checks

Fetch a page as Googlebot and compare with a normal request

$for ua in 'Mozilla/5.0 (X11; Linux x86_64)' 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'; do curl -s -o /dev/null -A "$ua" -w "%{http_code} %{size_download} Bytes $ua\n" https://example.com/; done
harmlessSEO checks

Analyze Googlebot requests in the access log

#grep -i "googlebot" /var/www/vhosts/system/example.com/logs/access_ssl_log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
harmlessLogs

Check for soft 404s: does a missing page really return 404

$curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' https://example.com/diese-seite-gibt-es-nicht-$RANDOM
harmlessSEO checks

Check maintenance mode for status 503 and Retry-After

$curl -s -o /dev/null -D - https://example.com/ | grep -iE '^(HTTP/|retry-after:)'
harmlessSEO checks

Check the status code of all sitemap.xml URLs in parallel

$curl -s https://example.com/sitemap.xml | grep -oP '(?<=<loc>)[^<]+' | xargs -P 8 -I{} curl -o /dev/null -s -w '%{http_code} %{url_effective}\n' {} | grep -v '^200 ' | sort
harmlessSEO checks

Check the X-Robots-Tag header of a URL

$curl -s -o /dev/null -D - https://example.com/ | grep -i '^x-robots-tag'
harmlessSEO checks

Read first, then run.

The commands on myline.de act directly on servers, files and databases. A wrong path or placeholder can delete data irreversibly or make a server unreachable.

  • All commands are provided without warranty and are not tested on every system.
  • Understand what a command does before running it, and check every placeholder.
  • Make a backup first and, if possible, try it on a test system.
  • You run commands at your own risk. Liability for damages is excluded to the extent permitted by law.