↑ ↓ select, Enter open, Esc close

Remove duplicate lines without changing the order

Adjust the values, the command updates live
user@server
awk '!seen[$0]++' list.txt

Remembers each line in an array and prints it only on its first occurrence. Unlike sort -u, the original order is kept, which matters for configuration files, host lists or authorized_keys. The result is printed to the console, the file itself stays unchanged.

Note: All unique lines are held in memory, so very large files with millions of distinct lines need a corresponding amount of RAM.

Also searched as

  • remove duplicate lines without sorting
  • delete duplicate lines keep order
  • awk unique lines preserve order

Related one-liners

All in Text processing

Read first, then run.

The commands on myline.de act directly on servers, files and databases. A wrong path or placeholder can delete data irreversibly or make a server unreachable.

  • All commands are provided without warranty and are not tested on every system.
  • Understand what a command does before running it, and check every placeholder.
  • Make a backup first and, if possible, try it on a test system.
  • You run commands at your own risk. Liability for damages is excluded to the extent permitted by law.