7.3 uniq: Uniquify files
uniq writes the unique lines in the given input, or
standard input if nothing is given or for an input name of
‘-’. Synopsis:
uniq [option]... [input [output]]
By default, uniq prints its input lines, except that
it discards all but the first of adjacent repeated lines, so that
no output lines are repeated. Optionally, it can instead discard
lines that are not repeated, or all repeated lines.
The input need not be sorted, but repeated input lines are detected
only if they are adjacent. If you want to discard non-adjacent
duplicate lines, perhaps you want to use sort -u
.
See sort invocation.
Comparisons honor the rules specified by the LC_COLLATE
locale category.
If no output file is specified, uniq writes to standard
output.
The program accepts the following options. Also see Common options.
- ‘-f n’
- ‘--skip-fields=n’
- Skip n fields on each line before checking for uniqueness. Use
a null string for comparison if a line has fewer than n fields. Fields
are sequences of non-space non-tab characters that are separated from
each other by at least one space or tab.
For compatibility uniq supports an obsolete option syntax
-n. New scripts should use -f n instead.
- ‘-s n’
- ‘--skip-chars=n’
- Skip n characters before checking for uniqueness. Use a null string
for comparison if a line has fewer than n characters. If you use both
the field and character skipping options, fields are skipped over first.
On older systems, uniq supports an obsolete option syntax
+n.
This obsolete behavior can be enabled or disabled with the
_POSIX2_VERSION environment variable (see Standards conformance), but portable scripts should avoid commands whose
behavior depends on this variable.
For example, use ‘uniq ./+10’ or ‘uniq -s 10’ rather than
the ambiguous ‘uniq +10’.
- ‘-c’
- ‘--count’
- Print the number of times each line occurred along with the line.
- ‘-i’
- ‘--ignore-case’
- Ignore differences in case when comparing lines.
- ‘-d’
- ‘--repeated’
- Discard lines that are not repeated. When used by itself, this option
causes uniq to print the first copy of each repeated line,
and nothing else.
- ‘-D’
- ‘--all-repeated[=delimit-method]’
- Do not discard the second and subsequent repeated input lines,
but discard lines that are not repeated.
This option is useful mainly in conjunction with other options e.g.,
to ignore case or to compare only selected fields.
The optional delimit-method tells how to delimit
groups of repeated lines, and must be one of the following:
- ‘none’
- Do not delimit groups of repeated lines.
This is equivalent to --all-repeated (-D).
- ‘prepend’
- Output a newline before each group of repeated lines.
With --zero-terminated (-z), use a zero
byte (ASCII nul) instead of a newline.
- ‘separate’
- Separate groups of repeated lines with a single newline.
With --zero-terminated (-z), use a zero
byte (ASCII nul) instead of a newline.
This is the same as using ‘prepend’, except that
no delimiter is inserted before the first group, and hence
may be better suited for output direct to users.
Note that when groups are delimited and the input stream contains
two or more consecutive blank lines, then the output is ambiguous.
To avoid that, filter the input through ‘tr -s '\n'’ to replace
each sequence of consecutive newlines with a single newline.
This is a gnu extension.
- ‘-u’
- ‘--unique’
- Discard the first repeated line. When used by itself, this option
causes uniq to print unique lines, and nothing else.
- ‘-w n’
- ‘--check-chars=n’
- Compare at most n characters on each line (after skipping any specified
fields and characters). By default the entire rest of the lines are
compared.
- ‘-z’
- ‘--zero-terminated’
- Delimit items with a zero byte rather than a newline (ASCII lf).
I.E. treat input as items separated by ASCII nul
and terminate output items with ASCII nul.
This option can be useful in conjunction with ‘perl -0’ or
‘find -print0’ and ‘xargs -0’ which do the same in order to
reliably handle arbitrary file names (even those containing blanks
or other special characters).
An exit status of zero indicates success,
and a nonzero value indicates failure.