DE

↑ ↓ select, Enter open, Esc close

Analyze Googlebot requests in the access log

harmless rootLogsdebian, ubuntu
Adjust the values, the command updates live
root@server
grep -i "googlebot" /var/www/vhosts/system/example.com/logs/access_ssl_log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Filters all lines with the Googlebot user agent and lists the 20 most crawled paths. This shows whether Google reaches important pages or wastes its crawl budget on parameter URLs and redirects. Replacing $7 with $9 shows the status codes the bot receives.

Note: The user agent is easy to fake. Genuine Googlebot IPs resolve via reverse DNS (host <IP>) to googlebot.com or google.com.

Also searched as

  • is google crawling my site
  • show googlebot hits in access log
  • which pages does googlebot visit
  • check crawl budget in log

Related one-liners

All in Logs

Find the most frequent 404 errors in the access log

#awk '$9 == 404 {print $7}' /var/www/vhosts/system/example.com/logs/access_ssl_log | sort | uniq -c | sort -rn | head -20
harmlessLogs

Extract all URLs from a sitemap.xml

$curl -s https://example.com/sitemap.xml | grep -oP '(?<=<loc>)[^<]+' | sort -u
harmlessSEO checks

Fetch a page as Googlebot and compare with a normal request

$for ua in 'Mozilla/5.0 (X11; Linux x86_64)' 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'; do curl -s -o /dev/null -A "$ua" -w "%{http_code} %{size_download} Bytes $ua\n" https://example.com/; done
harmlessSEO checks

Verify a real Googlebot via reverse DNS

$n=$(host 66.249.66.1 | awk '/pointer/ {print $NF}'); echo "PTR: $n"; host "$n"
harmlessSEO checks

Analyze server errors (5xx) in the access log

#awk '$9 ~ /^5[0-9][0-9]$/ {print $9, $7}' /var/www/vhosts/system/example.com/logs/access_ssl_log | sort | uniq -c | sort -rn | head -20
harmlessLogs

Count HTTP status codes in the access log

#awk '{print $9}' /var/www/vhosts/system/example.com/logs/access_ssl_log | sort | uniq -c | sort -rn
harmlessLogs

Read first, then run.

The commands on myline.de act directly on servers, files and databases. A wrong path or placeholder can delete data irreversibly or make a server unreachable.

  • All commands are provided without warranty and are not tested on every system.
  • Understand what a command does before running it, and check every placeholder.
  • Make a backup first and, if possible, try it on a test system.
  • You run commands at your own risk. Liability for damages is excluded to the extent permitted by law.