Unix and Linux Stack Exchange
Unix and Linux Stack Exchange

Questions tagged [awk]

A pattern-directed scanning and processing language.

Awk is a text processing tool with a far more expressive manipulation language than .
It has named scalar and array variables, functions, and common imperative control structures.

Implementations and documentation

Related tags

  • Text processing in general, when you don't know which tool is best
  • , , and other text processing tools
  • , for more general programming languages that can be used for text processing

Further reading

8013 questions
107
votes
5 answers

Using awk to sum the values of a column, based on the values of another column

I am trying to sum certain numbers in a column using awk. I would like to sum just column 3 of the "smiths" to get a total of 212. I can sum the whole column using awk but not just the "smiths". I have: awk 'BEGIN {FS = "|"} ; {sum+=$3} END {print…
jake
  • 1,215
  • 2
  • 10
  • 9
68
votes
4 answers

How to view all the content in an awk array?

In my understanding, awk array is something like python dict. So I write down the code bellow to explore it: awk '{my_dict[$1] = $2} END { print my_dict}' zen And I got: awk: can't read value of my_dict; it's an array name. As the first column…
Zen
  • 7,537
47
votes
16 answers

How to print certain columns by name?

I have the following file: id name age 1 ed 50 2 joe 70 I want to print just the id and age columns. Right now I just use awk: cat file.tsv | awk '{ print $1, $3 }' However, this requires knowing the column numbers. Is there a way…
Brett Thomas
  • 573
40
votes
3 answers

What is the meaning of '1' at the end of an awk script

I was reading this awk script awk -F"=" '{OFS="=";gsub(",",";",$2)}1' I want to know what is the function of 1 at the end of it.
Mirage
  • 1,557
  • 7
  • 19
  • 19
26
votes
5 answers

Extract value between double quotes

My query is to extract the value between double quotes "". Sample input is: 10.219.41.68 - - - [11/Jun/2014:10:23:04 -0400] Sec:0 MicSec:1797 "GET /balancer-manager HTTP/1.1" 200 28980 "-" "curl/7.15.5 (i386-redhat-linux-gnu) libcurl/7.15.5…
user79658
  • 271
26
votes
7 answers

How to remove duplicate lines with awk whilst keeping all empty lines?

Below awk command removes all duplicate lines as explained here: awk '!seen[$0]++' If the text contains empty lines, all but one empty line will be deleted. How can I keep all empty lines whilst deleting all non-empty duplicate lines, using only…
Serge Stroobandt
  • 2,454
23
votes
4 answers

Why does AWK print "0xffffffffbb6002e0" as "ffffffffbb600000" using printf?

I have been experimenting with hex numbers in AWK (gawk), but sometimes when I print them using e.g. printf, they are printed with some LSBs masked out, like in the following example: awk 'BEGIN { x=0xffffffffbb6002e0; printf("%x\n", x);…
Shuzheng
  • 4,411
23
votes
2 answers

How does ` ... | awk '$1=$1'` remove extra spaces?

From my understanding, $1 is the first field. But strangely enough, awk '$1=$1' omits extra spaces. $ echo "$string" foo foo bar bar $ echo "$string" | awk '$1=$1' foo foo bar bar Why is this happening?
annahri
  • 2,075
23
votes
3 answers

When using awk /pattern/ { print "text"} /patern/ {print ""} is there an ELSE pattern?

Let's say I have text file like: R1 12 324 3453 36 457 4 7 8 R2 34 2342 2525 25 25 26 26 2 2 R3 23 2342 32 52 54 543 643 63 R4 25 234 2342 4 234242 I want to use awk to process these lines differently, like awk '/R1/ { print "=>" $0} /R2/ { print…
Ali
  • 6,943
21
votes
1 answer

Slurp-mode in awk?

Tools like sed, awk or perl -n process their input one record at a time, records being lines by default. Some, like awk with RS, GNU sed with -z or perl with -0ooo can change the type of record by selecting a different record separator. perl -n can…
Stéphane Chazelas
  • 544,893
21
votes
2 answers

Conditional block vs conditional statement (if)

Say I have a file: PRO 1 GLN 5.55112e-17 ILE -6.245e-17 THR 5.55112e-17 I want every line that has a number unequal to 1 in the second column to change it to 0 and keep the rest. If I use if (i.e conditional statement), everything is OK: awk…
Ooker
  • 667
19
votes
2 answers

How to keep the field separator when printing $0 with awk

I try to modify one column of my file, then print the result. awk -F"|" '{ if(NR!=1){$5 = $5+0.1} print $0}' myfile It does what I want, but when printing, only the first line keeps its field separator.( the one, i don't modify). So I could use…
Damien
  • 375
18
votes
5 answers

Print last element of each row

How can one print the element after the last tab in a file? Exemple: File1 A 3 8 6 7 B 4 6 2 3 6 8 c 1 9 would return: 7 8 9
dovah
  • 1,717
17
votes
4 answers

find length of longest line in all text files in a directory

I know how to get length of the longest line in a text file with awk awk ' { if ( length > L ) { L=length} }END{ print L}' file.txt but how can I get the length of the longest line of all files in a directory?
trupty
  • 173
17
votes
4 answers

BEGIN and END with the awk command

According to the awk manual, BEGIN and END are not used to match input, but rather to provide start-up and clean-up information to the awk script. Here is the example given: ls -l | \ awk 'BEGIN { print "Files found:\n" } /\<[a|x].*\.conf$/ { print…
JohnMerlino
  • 6,161
1
2 3
…
56 57