print most of repeated numbers from the second field

Question

we have the following very long file

    llvmpipe-  36141  36258           root  mem       REG              253,0    534488  201604324 /usr/lib64/libgcrypt.so.11.8.2
    llvmpipe-  36141  36258           root  mem       REG              253,0     27504  201943960 /usr/lib64/libfontenc.so.1.0.0
    llvmpipe-  36141  36258           root  mem       REG              253,0    691680  201604095 /usr/lib64/libfreetype.so.6.10.0
    llvmpipe-  36141  36258           root  mem       REG              253,0     90632  201604087 /usr/lib64/libz.so.1.2.7
    llvmpipe-  36141  36258           root  mem       REG              253,0    153192  201604070 /usr/lib64/liblzma.so.5.0.99
    llvmpipe-  36141  36258           root  mem       REG              253,0    398272  201604073 /usr/lib64/libpcre.so.1.2.0
    llvmpipe-  36141  36258           root  mem       REG              253,0    142304  201603881 /usr/lib64/libpthread-2.17.so
    llvmpipe-  36141  36258           root  mem       REG              253,0     88720  201326789 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
    llvmpipe-  36141  36258           root  mem       REG              253,0    297464  201654760 /usr/lib64/libdw-0.163.so
    llvmpipe-  36141  36258           root  mem       REG              253,0     20024  201610087 /usr/lib64/libcap.so.2.22
    llvmpipe-  36141  36258           root  mem       REG              253,0     44096  201603885 /usr/lib64/librt-2.17.so
    llvmpipe-  36141  36258           root  mem       REG              253,0   2107800  201596079 /usr/lib64/libc-2.17.so
    llvmpipe-  36141  36258           root  mem       REG              253,0   1141560  201596087 /usr/lib64/libm-2.17.so
    llvmpipe-  36141  36258           root  mem       REG              253,0    118792  201604116 /usr/lib64/libaudit.so.1.0.0
    llvmpipe-  36141  36258           root  mem       REG              253,0     61648  202097293 /usr/lib64/libpam.so.0.83.1
    llvmpipe-  36141  36258           root  mem       REG              253,0     15616  202097295 /usr/lib64/libpam_misc.so.0.82.0
    llvmpipe-  36141  36258           root  mem       REG              253,0     23888  201868640 /usr/lib64/libXdmcp.so.6.0.0
    llvmpipe-  36141  36258           root  mem       REG              253,0      6976  202124274 /usr/lib64/libxshmfence.so.1.0.0
    llvmpipe-  36141  36258           root  mem       REG              253,0     15512  201686410 /usr/lib64/libXau.so.6.0.0
    llvmpipe-  36141  36258           root  mem       REG              253,0    216840  201943962 /usr/lib64/libXfont.so.1.4.1
    llvmpipe-  36141  36258           root  mem       REG              253,0    715344  201610107 /usr/lib64/libpixman-1.so.0.32.6
    llvmpipe-  36141  36258           root  mem       REG              253,0     54008  202770732 /usr/lib64/libdrm.so.2.4.0
    llvmpipe-  36141  36258           root  mem       REG              253,0     40784  202770730     /usr/lib64/libdl-2.17.so
    gnome-ses  36280                   gdm  mem       REG              253,0     11384  203387653 /usr/lib64/libXinerama.so.1.0.0
    gnome-ses  36280                   gdm  mem       REG              253,0    251800  201983581 /usr/lib64/libfontconfig.so.1.7.0
    gnome-ses  36280                   gdm  mem       REG              253,0     88440  203491884 /usr/lib64/libpangoft2-1.0.so.0.3600.8
    gnome-ses  36280                   gdm  mem       REG              253,0    180632  203673069 /usr/lib64/libatk-bridge-2.0.so.0.0.0
    gnome-ses  36280                   gdm  mem       REG              253,0     23768  203387616 /usr/lib64/libXfixes.so.3.1.0
    gnome-ses  36280                   gdm  mem       REG              253,0     15616  201610125 /usr/lib64/libgmodule-2.0.so.0.4200.2
    gnome-ses  36280                   gdm  mem       REG              253,0     40736  203387649 /usr/lib64/libXrandr.so.2.2.0
    gnome-ses  36280                   gdm  mem       REG              253,0   1141560  201596087 /usr/lib64/libm-2.17.so
    gnome-ses  36280                   gdm  mem       REG              253,0     44096  201603885 /usr/lib64/librt-2.17.so

.
.
.

we want to find the top of 20 repeated numbers of the second field

so first we print the list of lines with most repeated numbers on the second field

then the second of repeated numbers of the second field and so on

Isn't this what you just asked at https://unix.stackexchange.com/questions/584165/lsof-how-to-find-which-are-the-opened-files-that-printed-from-lsof-with-the-hu ? — Jeff Schaller, May 03 '20 at 11:52
Can you provide a minimal, reproducible example with expected output? — Quasímodo, May 03 '20 at 17:13
Relating https://unix.stackexchange.com/q/365922/117549 and https://unix.stackexchange.com/q/199479/117549 — Jeff Schaller, May 03 '20 at 21:35

score 1 · Answer 1 · answered May 03 '20 at 18:39

awk '
  FNR==NR && NR<=20{ a[$2]; b[++cnt]=$2; next }
  $2 in a{
     if ($2 in c) c[$2]=c[$2] ORS $0 # append line to existing line(s)
     else c[$2]=$0                   # add first line
  }
  END{ for (i=1; i<=cnt; i++) print c[b[i]] }
' <(awk '{ print $2 }' file | sort | uniq -c | sort -rn) file

Process substitution

awk: print the second field
sort: not needed in this example since the PIDs are already adjacent, not sure if this is always the case
uniq: print unique count and PIDs
sort: reverse numeric (highest count first)

Main script

Use the result of the process substitution (limited to 20 lines, ignore the count) and the original file as input. Save the PIDs in two arrays a and b, then process the input file and store the desired lines in array c and print these.

array a: the PIDs as indices used for lookup
array b: index to PID mapping to maintain the order in the output
array c: stores the lines to print (PID as index)

score 1 · Accepted Answer · answered May 03 '20 at 21:30

Given the data in your question in the file called file:

$ awk '{ print $2 }' <file | sort | uniq -c | sort -rn | head -n 20
  23 36141
   9 36280

In the given data, there is only two unique numbers in the second field (36141 occurs 23 times and 36280 occurs 9 times).

The pipeline extracts the second column using awk (use cut -f 2 instead, if the data is tab-delimited). This list of numbers is then sorted and uniq -c counts how many times each number occurs. The sort -rn sorts them on the count in the order larges to smallest, and the final head -n 20 cuts the list short at 20 unique numbers (only two numbers are unique in the example data).

print most of repeated numbers from the second field

2 Answers2