DE

↑ ↓ select, Enter open, Esc close

Find duplicate files by checksum

harmlessFilesystemalle
Adjust the values, the command updates live
user@server
find /var/www/vhosts/example.com/httpdocs/wp-content/uploads -type f -size +0 -exec md5sum {} + | sort | uniq -w32 --all-repeated=separate

Computes the MD5 checksum of every non-empty file and shows only files whose checksum occurs more than once. Each group of identical files is separated by a blank line. MD5 is sufficient here, since the goal is finding duplicates, not tamper resistance.

Note: With many large files this takes a while, because every file is read completely. The fdupes package does the same job faster.

Also searched as

  • find duplicate files linux
  • find duplicate images in uploads
  • find identical files
  • same file stored twice

Related one-liners

All in Filesystem

Delete empty directories recursively

$find /var/www/vhosts/example.com/httpdocs -mindepth 1 -type d -empty -print -delete
destructiveFilesystem

Delete files older than X days

$find /var/backups/example.com -type f -name '*.tar.gz' -mtime +30 -print -delete
destructiveFilesystem

Find empty directories

$find /var/www/vhosts/example.com/httpdocs -mindepth 1 -type d -empty
harmlessFilesystem

Find files older than X days

$find /var/www/vhosts/example.com -type f -mtime +30 -printf '%TY-%Tm-%Td %p\n' | sort | head -100
harmlessFilesystem

Find the largest files in a directory

$find /var/www/vhosts -type f -size +500M -exec ls -lh {} + 2>/dev/null | sort -k5 -rh | head -20
harmlessFilesystem

Recursively delete all files with a given name

$find /var/www/vhosts/example.com/httpdocs -type f -name 'error_log' -print -delete
destructiveFilesystem

Read first, then run.

The commands on myline.de act directly on servers, files and databases. A wrong path or placeholder can delete data irreversibly or make a server unreachable.

  • All commands are provided without warranty and are not tested on every system.
  • Understand what a command does before running it, and check every placeholder.
  • Make a backup first and, if possible, try it on a test system.
  • You run commands at your own risk. Liability for damages is excluded to the extent permitted by law.