Remove duplicate lines without changing the order
remove duplicate lines without sorting | delete duplicate lines keep order | awk unique lines preserve order | awk | duplicates | deduplicateawk '!seen[$0]++' list.txtRemembers each line in an array and prints it only on its first occurrence. Unlike sort -u, the original order is kept, which matters for configuration files, host lists or authorized_keys. The result is printed to the console, the file itself stays unchanged.
Note: All unique lines are held in memory, so very large files with millions of distinct lines need a corresponding amount of RAM.
Also searched as
- remove duplicate lines without sorting
- delete duplicate lines keep order
- awk unique lines preserve order