#crawling

3 one-liners with this tag.

Check whether the server returns 304 Not Modified

$curl -s -o /dev/null -w '%{http_code}\n' -H "If-Modified-Since: $(curl -sI https://example.com/ | grep -i '^last-modified:' | cut -d' ' -f2- | tr -d '\r')" https://example.com/

harmlessSEO checks

Fetch robots.txt and show the Disallow rules

$curl -s https://example.com/robots.txt | grep -iE '^\s*(user-agent|disallow|allow|sitemap):'

harmlessSEO checks

Find broken links on a website with wget --spider

$LC_ALL=C wget --spider -r -l 2 -nd -nv -w 1 -o /tmp/spider.log https://example.com/; grep -A 100 'broken link' /tmp/spider.log

harmlessSEO checks