find a pair of words that appear the most of the times together

Question

I have 10 text files, in each file i have a chapter from a book, i want to find the pair of words that appear the most of the time together in a line i.e:

chapter1:

hello world good boy green sun

good green boy sun world hello

chapter2:

chapter3:

.....etc

Output wanted for chapter1:

hello world (alphabet order)

what about if you had a line like Hello world Hello? it should come in output? and how ? Hello world or world Hello? — αғsнιη, Apr 17 '19 at 15:02
hello world , and this will count for 2 to the pair "hello world" always alphabetical — John B, Apr 17 '19 at 15:10
how about Hello\nworld\nHello? \n is actual new-line character. please [edit] your question to answer for comments asking for clarifications — αғsнιη, Apr 17 '19 at 15:31
Ohhh, sounds this is a followup of your recent question https://unix.stackexchange.com/q/513006/72456 — αғsнιη, Apr 17 '19 at 16:08

score 2 · Answer 1 · answered Apr 17 '19 at 16:21

awk '
  {
    $0 = tolower($0)
    for (i = 1; i < NF; i++) {
      pair = $i"" < $(i+1) ? $i" "$(i+1) : $(i+1)" "$i
      c = ++count[pair]
      if (c > max) max = c
    }
  }
  END {
    for (pair in count)
      if (count[pair] == max)
        print pair
  }'

pLumo · Answer 2 · 2019-04-17T15:38:16.410

0

Try this,

Use awk to print each pair of words.
Use perl to sort the words in a pair (via).
Use sort and uniq -c to count occurrences each pair.

awk '{for (i=1;i<NF;i++) { print tolower($i)" "tolower($(i+1)) }}' file \
| perl -ane '$,=" "; print sort @F; print "\n";' \
| sort | uniq -c | sort -b -k1nr -k2

Output:

  2 boy green
  2 hello world
  1 boy good
  1 boy sun
  1 good green
  1 good world
  1 green sun
  1 sun world

edited Apr 17 '19 at 15:38

answered Apr 17 '19 at 15:21

pLumo

22,565

can not use perl or pipeline.. – John B Apr 17 '19 at 15:35
2

Why can't you, @John? Those are standard utilities on most Linux systems. – Jeff Schaller Apr 17 '19 at 15:37
1

Yes, this kind of information should be in your question. – pLumo Apr 17 '19 at 15:37

find a pair of words that appear the most of the times together

2 Answers2