Questions tagged [uniq]
174 questions
44
votes
4 answers
How is uniq not unique enough that there is also uniq --unique?
Here are commands on a random file from pastebin:
wget -qO - http://pastebin.com/0cSPs9LR | wc -l
350
wget -qO - http://pastebin.com/0cSPs9LR | sort -u | wc -l
287
wget -qO - http://pastebin.com/0cSPs9LR | sort | uniq | wc -l
287
wget -qO -…
enfascination
- 581
15
votes
2 answers
What did `uniq -t` do?
I have some old code from 2003 which uses -t option for uniq command. It throws an error since that option is probably not supported anymore.
Here's the piece which uses the command:
egrep -n "{ IA32_OP" ia32-decode.c | \
awk '{ print $1 $3 $4…
Babken Vardanyan
- 885
14
votes
3 answers
What is the point of uniq -u and what does it do?
uniq seems to do something different than uniq -u, even though the description for both is "only unique lines".
What's the difference here, what do they do?
user11350058
- 159
6
votes
2 answers
Count & sum no. of occurrences of each unique line in a file
I think the answer to this problem might be some sort of variant of the uniq function that allows you to count the number of times each line appears in a file:
sort file.txt | uniq -c
The problem for me is that I have used this uniq function to…
Simonmdr
- 141
5
votes
2 answers
Why doesn't "uniq --unique" remove all duplicate lines?
Running
printf "lol\nlol\nfoo\n\n\n\n\nbar\nlol\nlol\nfoo\nlol\nfoo" | uniq --unique
prints
foo
bar
foo
lol
foo
Why is foo printed three times? Shouldn't uniq --unique remove them?
Also, notably, it seems all duplicates of lol were removed. Why…
hanshenrik
- 615
4
votes
1 answer
Generalization of uniq
The uniq(1) command can be viewed as a tiny map/reduce-style program:
map consecutive, identical input lines into groups;
reduce the list of lines (which are all identical) to a single line.
Are there more examples of programs in this (functional)…
miku
- 673
2
votes
2 answers
How can I remove blocks of duplicated lines with uniq?
Is there anyway to use uniq (or similar) to filter/remove sets of repeating lines from log type output? I am debugging an MPI code where multiple processors often print the same exact output. uniq works great when the output is one line, but…
rheo
- 31
- 4
2
votes
3 answers
finding all non-unique lines in a file
I'm trying to use uniq to find all non-unique lines in a file. By non-unique, I mean any line that I have already seen on the previous line. I thought that the "-D" option would do this:
-D print all duplicate lines
But instead of just…
Michael
- 544
2
votes
2 answers
Using 1st[N] characters when checking for duplicate
I have a set of Data in file:
AAAPOL.0001
AAAPOL.0002
AAAPRO.0001
AAAPRO.0002
AAAPRO.0003
AAAPRO.0004
AAAXEL.0002
AAAJOK.1111
AAAJOK.2222
I only need the first occurrence using the pattern of the 1st 6 characters so I need to know…
0
votes
1 answer
How to use uniq Unix command and specify input and output file?
I'm confused on the use of the uniq command. Can anyone explain this command and how to use it?
How can I use uniq without specifying any option? Any example would be helped and appreciated.
user2105542
- 1
- 1
- 1
0
votes
2 answers
0
votes
2 answers
find unique elements restricted by column
The unix command uniq -u returns truly unique elements (as opposed to the default behavior of uniq). For example:
echo -e "a\na\nb\nc\nc\nd\ne\ne\ne\nf" | uniq -u
b
d
f
How can this command be emulated in a column-restricted manner (i.e.,…
user001
- 3,698
0
votes
1 answer
uniq and sed, delete lines with pattern similar in multiple files
I need delete lines with pattern similar, in multiple files (2500 .in files). example I have an input file -- "transmission-gtk.in" -- whose content is something like, but it is variable, I do not…
davidva
- 160