Removing Duplicates


A common task when editing files is to remove duplicate lines. While this can be done manually, it is often easier to leverage Vim's command-line mode to leverage the uniq command.

Suppose we have the following file in a buffer and want to remove duplicates:

Initial Conditions
bananas
apples
apples
oranges
apples
COMMANDTop1:1
:%!uniq

The command to execute is:

:%!uniq

Let's break this down to review what is happening: To start, this follows the pattern:

:[range]!{cmd} {args}

where

:%

specifies the range as the entire file, although this can be replaced with a range specifying a subset of lines, or a visual selection. Next,

!{cmd}

indicates that an external command is to be called with the specified range and arguments. In this case the external command to be called is:

uniq

with (initially) no arguments. By default uniq takes input from stdin, so there is no need to specify the input source.

Executing this command produces the following output:

Execute uniq
bananas
apples
oranges
apples
NORMALTop1:1
5 lines filtered

which is not exactly what we were looking for. It turns out that by default uniq only detects adjacent repeating lines, so apple still appears twice in the output.

If we take a look at the help page for uniq we see that the -u option can be used to print only unique lines, which is what we were looking for. Lets undo to get back to the original buffer, then try again using the -u argument:

Execute uniq -u
bananas
oranges
apples
NORMALTop1:1
5 lines filtered

That is more like it. We should note that uniq also offers the -d option, which performs the opposite function and returns only the duplicate lines. Starting from the original buffer we can execute this command to see the result:

Execute uniq -d
apples
NORMALTop1:1
5 lines filtered