DE

↑ ↓ select, Enter open, Esc close

Find the most common user agents and bots in the access log

harmless rootLogsdebian, ubuntu
Adjust the values, the command updates live
root@server
awk -F'"' '{print $6}' /var/www/vhosts/system/example.com/logs/access_ssl_log | sort | uniq -c | sort -rn | head -20

Splits each log line at the double quotes and prints the sixth field, which is the user agent in the combined format. The list shows immediately whether crawlers like AhrefsBot, SemrushBot or GPTBot account for a large share of the traffic. From there you can add targeted blocks in robots.txt or in the web server.

Also searched as

  • which bots are crawling my site
  • count user agents in access log
  • find aggressive crawlers
  • which bot is causing load

Related one-liners

All in Logs

Analyze server errors (5xx) in the access log

#awk '$9 ~ /^5[0-9][0-9]$/ {print $9, $7}' /var/www/vhosts/system/example.com/logs/access_ssl_log | sort | uniq -c | sort -rn | head -20
harmlessLogs

Count HTTP status codes in the access log

#awk '{print $9}' /var/www/vhosts/system/example.com/logs/access_ssl_log | sort | uniq -c | sort -rn
harmlessLogs

Count requests per minute in the access log

#awk '{print substr($4, 2, 17)}' /var/www/vhosts/system/example.com/logs/access_ssl_log | sort | uniq -c | sort -rn | head -20
harmlessLogs

Filter the access log by time range

#awk -v von="09/Oct/2026:14:00" -v bis="09/Oct/2026:15:00" 'substr($4, 2) >= von && substr($4, 2) < bis' /var/www/vhosts/system/example.com/logs/access_ssl_log
harmlessLogs

Find the most frequent 404 errors in the access log

#awk '$9 == 404 {print $7}' /var/www/vhosts/system/example.com/logs/access_ssl_log | sort | uniq -c | sort -rn | head -20
harmlessLogs

Find the most requested URLs in the access log

#awk '{print $7}' /var/www/vhosts/system/example.com/logs/access_ssl_log | sort | uniq -c | sort -rn | head -20
harmlessLogs

Read first, then run.

The commands on myline.de act directly on servers, files and databases. A wrong path or placeholder can delete data irreversibly or make a server unreachable.

  • All commands are provided without warranty and are not tested on every system.
  • Understand what a command does before running it, and check every placeholder.
  • Make a backup first and, if possible, try it on a test system.
  • You run commands at your own risk. Liability for damages is excluded to the extent permitted by law.