HTTP Status Codes

Check status codes with curl

The browser only reveals the status code in its developer tools and caches redirects. With curl you get the server's unvarnished response. Only the headers, including the status line:

curl -sI https://example.com/page/

A whole list of URLs at once, for example after a site move (one URL per line in urls.txt):

while read -r u; do
  printf '%s %s\n' "$(curl -s -o /dev/null -w '%{http_code}' "$u")" "$u"
done < urls.txt

An analysis of the access log shows which codes your server actually delivers. The path applies to Plesk, column 9 to the usual combined log format:

awk '{print $9}' /var/www/vhosts/example.com/logs/access_ssl_log | sort | uniq -c | sort -rn

404 or 410?

Both say "does not exist". 404 leaves open whether the page will come back, 410 explicitly says it is gone for good. Google treats both similarly, but usually removes 410 URLs from the index a bit faster. For normal deleted pages, 404 is perfectly fine. 410 is worth it when many unwanted URLs need to disappear quickly, typically after a hack that injected spam pages. If an old page has a successor with matching content, a 301 redirect is better than either.

Soft 404: error page with status 200

A soft 404 is a page that says "not found" but is delivered with status 200. Google detects such pages and reports them in Search Console. Common causes: custom error pages included via a redirect instead of ErrorDocument, empty category or search pages, and blanket redirects of all old URLs to the home page. Use curl to check that a non-existent URL really returns 404.

Announce maintenance properly: 503 with Retry-After

During maintenance the server should return 503 Service Unavailable, not 200 with a notice and not 404. The Retry-After header tells search engines when to try again (in seconds or as a date). That way your rankings stay untouched. An example for nginx:

# nginx (Plesk: Additional nginx directives): maintenance mode
# while the file .maintenance-on exists in the web root
if (-f $document_root/.maintenance-on) {
    return 503;
}
add_header Retry-After 3600 always;

During updates, WordPress itself returns a 503 with Retry-After as long as the file .maintenance exists in the web root. If it is left behind after an aborted update, the site stays in maintenance mode: delete the file and you are done.

Narrowing down 5xx errors on the server

• 500: start with the domain's error log (/var/www/vhosts/example.com/logs/error_log in Plesk) and the PHP log. For WordPress, WP_DEBUG_LOG helps.
• 502: is PHP-FPM running? systemctl status plesk-php83-fpm (adjust the version), plus journalctl -u plesk-php83-fpm --since "1 hour ago".
• 503 and 504: are all PHP-FPM workers busy? The FPM log then says server reached pm.max_children. The MySQL slow query log finds slow queries.