Mfrhtyj

It's a pretty straight-forward perl script:

#!/usr/bin/perl -w

use strict;



my %words = ();

while (<>) {

  chomp;

  my %linewords = ();

  map { $linewords{$_}=1 } split / /;

  foreach my $word (keys %linewords) {

    $words{$word}++;

  }

}



foreach my $word (sort keys %words) {

  print "$word:$words{$word}n";

}

The basic idea is to loop over the input; for each line, split it into words, then save those words into a hash (associative array) in order to remove any duplicates, then loop over that array of words and add one to an overall counter for that word. At the end, report on the words and their counts.

Straightfoward-ish in bash:

declare -A wordcount

while read -ra words; do 

    # unique words on this line

    declare -A uniq

    for word in "${words[@]}"; do 

        uniq[$word]=1

    done

    # accumulate the words

    for word in "${!uniq[@]}"; do 

        ((wordcount[$word]++))

    done

    unset uniq

done < file

Looking at the data:

$ declare -p wordcount

declare -A wordcount='([possible]="1" [one]="1" [different]="1" [this]="1" [a]="1" [hello]="1" [world]="2" [man]="2" [0]="1" [1]="1" [2]="1" [is]="3" [the]="3" )'

and formatting as you want:

$ printf "%sn" "${!wordcount[@]}" | sort | while read key; do echo "$key:${wordcount[$key]}"; done

0:1

1:1

2:1

a:1

different:1

hello:1

is:3

man:2

one:1

possible:1

the:3

this:1

world:2

A solution that calls several programs from a shell:

fmt -1 words.txt | sort -u | xargs -Ipattern sh -c 'echo "pattern:$(grep -cw pattern words.txt)"'

A little explanation:

The fmt -1 words.txt prints out all the words, 1 per line, and the | sort -u sorts this output and extracts only the unique words from it.

In order to count the occurences of a word in a file, one can use grep (a tool meant to search files for patterns). By passing the -cw option, grep gives the number of word matches it finds. So you can find the total number of occurrences of pattern using grep -cw pattern words.txt.

The tool xargs allows us to do this for each and every single word output by sort. The -Ipattern means that it will execute the following command multiple times, replacing each occurrence of pattern with a word it reads from standard input, which is what it gets from sort.

The indirection with sh is needed because xargs only knows how to execute a single program, given it's name, passing everything else as arguments to it. xargs does not handle things like command substitution. The $(...) is command substitution in the above snippet, as it substitutes the output from grep into echo, allowing it to be formatted correctly. Since we need the command substitution, we must use the sh -c command which runs whatever it recieves as an argument in its own shell.

Another simple alternative would be to use Python (>3.6). This solution has the same problem as the one mentioned by @Larry in his comment.

from collections import Counter



with open("words.txt") as f:

    c = Counter(word for line in [line.strip().split() for line in f] for word in set(line))

    for word, occurrence in sorted(c.items()):

        print(f'{word}:{occurrence}')

        # for Python 2.7.x compatibility you can replace the above line with 

        # the following one:

        # print('{}:{}'.format(word, occurrence))

A more explicit version version of the above:

from collections import Counter





FILENAME = "words.txt"





def find_unique_words():

    with open(FILENAME) as f:

        lines = [line.strip().split() for line in f]



    unique_words = Counter(word for line in lines for word in set(line))

    return sorted(unique_words.items())





def print_unique_words():

    unique_words = find_unique_words()

    for word, occurrence in unique_words:

        print(f'{word}:{occurrence}')





def main():

    print_unique_words()





if __name__ == '__main__':

    main()

Output:

0:1

1:1

2:1

a:1

different:1

hello:1

is:3

man:2

one:1

possible:1

the:3

this:1

world:2

The above also assumes that words.txt is on the same directory as script.py. Note that this is not much different from other solutions provided here, but perhaps somebody will find it useful.

Trying to do it with awk:

count.awk:

#!/usr/bin/awk -f

# count line containing word



{

    for (i = 1 ; i <= NF ; i++) {

        word_in_a_line[$i] ++

        if (word_in_a_line[$i] == 1) {

            word_line_count[$i] ++

        }

    }



    delete word_in_a_line

}



END {

    for (word in word_line_count){

        printf "%s:%dn",word,word_line_count[word]

    }

}

Run it by:

$ awk -f count.awk ./test.data | sort

A pure bash answer

echo "0 hello world the man is world

1 this is the world

2 a different man is the possible one" | while IFS=$'n' read -r line; do echo $line | tr ' ' 'n' | sort -u; done | sort | uniq -c





   1 0

   1 1

   1 2

   1 a

   1 different

   1 hello

   3 is

   2 man

   1 one

   1 possible

   3 the

   1 this

   2 world

I looped unique words on each line and passed it to uniq -c

edit: I did not see glenn's answer. I found it strange to not see a bash answer

Simple, though doesn't care if it reads the file many times:

sed 's/ /n/g' file.txt | sort | uniq | while read word; do

  printf "%s:%dn" "$word" "$(grep -Fw $word file.txt | wc -l)"

done