Linux awk Command Tutorial for Beginners: Fields, Patterns, Sums and Reports

By Ravindra Bagale, trainer · LinkedIn · Last updated: 25 October 2026

1. An everyday example: a milk delivery notebook

Why do we need this?

Ganesh the milkman writes every delivery as: date, house, litres, rate. At month end he wants: the total litres, the bill for each house, only days above 20 litres, the average per day, and a neat printed bill with columns lined up.

What is it?

Each line = a record, each column = a field ($1 date, $2 house, $3 litres). Total = sum+=$3 in END. Bill per house = bill[$2]+=$3*$4. Days above 20 = $3>20. Neat bill = printf.

How to do it (step by step)

2. Fields: $1, $NF and the separator -F

Why do we need this?

Command output and logs are columns separated by spaces or commas. cut fails when spaces are uneven; awk handles them.

What is it?

awk '{print $1}' prints field 1. By default fields are split by any run of spaces or tabs, so uneven spacing is fine. $0 is the whole line, $NF the last field, $(NF-1) the one before. -F, sets a comma separator, -F: a colon, -F'[,;]' a regex. OFS is the output separator; print $1,$2 uses it (default space).

How to do it (step by step)

1. First column of ps output.

ps aux | awk '{print $1}' | sort | uniq -c | sort -nr | head -3

Output:

     98 root
     12 nginx
      7 ravi

What it means: Uneven spaces do not matter.

2. Users and shells.

awk -F: '{print $1, $7}' /etc/passwd | tail -2

Output:

ravi /bin/bash
nginx /sbin/nologin

What it means: -F: splits on colons.

3. Last field.

df -h | awk '{print $NF}'

Output:

Mounted
/
/boot
/data

What it means: $NF is the mount point.

4. Change the output separator.

awk -F: 'BEGIN{OFS=","} {print $1,$3}' /etc/passwd | head -2

Output:

root,0
bin,1

What it means: OFS="," makes CSV.

5. Reorder columns.

echo 'Pune 4500 2026-10-01' | awk '{print $3, $1, $2}'

Output:

2026-10-01 Pune 4500

What it means: Any order you like.

6. Field count per line.

awk -F, '{print NR": "NF}' sales.csv | head -3

Output:

1: 4
2: 4
3: 3

What it means: Line 3 has a missing column. Good for data checks.

3. Patterns: pick lines by condition

Why do we need this?

You want rows where amount > 5000, users with UID ≥ 1000, or lines matching a regex in one column.

What is it?

pattern { action }. If the action is missing, awk prints the line. Patterns: /regex/, $3 > 100, $2 == "Pune", $1 ~ /^web/ (matches), $1 !~ /test/ (does not), && and || to combine, NR==1, NR>1 to skip a header, /start/,/end/ ranges.

How to do it (step by step)

1. Rows above 5000.

awk -F, '$4 > 5000' sales.csv

Output:

2026-10-02,Mumbai,Asha,7200
2026-10-03,Pune,Imran,5600

What it means: No action means print.

2. Real users (UID 1000+).

awk -F: '$3 >= 1000 && $3 < 65000 {print $1}' /etc/passwd

Output:

ravi
anita

What it means: Two conditions joined with &&.

3. Text compare.

awk -F, '$2 == "Pune" {print $3}' sales.csv

Output:

Imran
Sneha

What it means: Use == with quotes for text.

4. Regex on one field.

awk '$1 ~ /^web/' hosts.txt

Output:

web1 10.0.0.11
web2 10.0.0.12

What it means: ~ means 'matches'.

5. Skip the header.

awk -F, 'NR > 1 {print $2}' sales.csv | sort -u

Output:

Mumbai
Nashik
Pune

What it means: NR is the line number.

6. HTTP 5xx from an access log.

awk '$9 >= 500 {print $7, $9}' access.log | head -2

Output:

/pay 503
/cart 500

What it means: In the nginx combined format, $9 is the status and $7 the URL.

4. BEGIN, END, NR and running totals

Why do we need this?

Reports need a header first and totals at the end.

What is it?

BEGIN { } runs before the first line (set FS, OFS, print headers). END { } runs after the last line (print totals). Variables start at 0 or empty, so sum += $3 just works. NR in END is the total number of lines. Average = sum/NR (or sum/count if you filtered).

How to do it (step by step)

1. Total litres.

awk '{sum += $3} END {print "Total:", sum}' milk.txt

Output:

Total: 642

What it means: sum adds field 3 of every line.

2. Average response time.

awk '{s += $NF} END {printf "avg %.2f ms over %d requests\n", s/NR, NR}' resp.log

Output:

avg 182.40 ms over 1200 requests

What it means: printf gives 2 decimals.

3. Min and max.

awk 'NR==1{min=max=$2} {if($2<min)min=$2; if($2>max)max=$2} END{print min, max}' temps.txt

Output:

18 41

What it means: Start min and max from line 1.

4. Header and footer.

awk -F, 'BEGIN{print "CITY AMOUNT"} NR>1{print $2, $4; t+=$4} END{print "TOTAL", t}' sales.csv

Output:

CITY AMOUNT
Pune 4500
Nashik 3200
TOTAL 7700

What it means: A tiny report in one line.

5. Count matching lines.

awk '/ERROR/ {n++} END {print n+0}' app.log

Output:

27

What it means: n+0 prints 0 instead of empty if none.

6. Disk used percent above 80.

df -h | awk 'NR>1 && $5+0 > 80 {print $6, $5}'

Output:

/data 91%

What it means: $5+0 turns '91%' into the number 91.

5. Arrays: totals per group

Why do we need this?

Bill per house, requests per IP, sales per city: you need one total for each key.

What is it?

awk arrays use text keys: t[$2] += $4. Loop in END with for (k in t) print k, t[k]. The order is random, so pipe into sort. count[$1]++ counts. if (k in arr) checks a key. delete arr[k] removes it. !seen[$0]++ is the famous one-liner that prints each line only the first time (dedupe without sorting).

How to do it (step by step)

1. Sales per city.

awk -F, 'NR>1 {t[$2]+=$4} END {for (c in t) print c, t[c]}' sales.csv | sort -k2 -nr

Output:

Pune 10100
Mumbai 7200
Nashik 3200

What it means: One total per key.

2. Requests per IP.

awk '{c[$1]++} END {for (ip in c) print c[ip], ip}' access.log | sort -nr | head -3

Output:

812 10.0.0.7
430 10.0.0.2
95 49.36.12.4

What it means: Same as sort | uniq -c, in one tool.

3. Bill per house.

awk '{b[$2] += $3*$4} END {for (h in b) printf "%-6s Rs %d\n", h, b[h]}' milk.txt | sort

Output:

A-101  Rs 1680
B-204  Rs 2240

What it means: Litres × rate, summed by house.

4. Remove duplicates, keep the order.

awk '!seen[$0]++' emails.txt | wc -l

Output:

151

What it means: First time seen[$0] is 0, so !0 is true and it prints.

5. Average per group.

awk '{s[$1]+=$2; n[$1]++} END {for (k in s) printf "%s %.1f\n", k, s[k]/n[k]}' resp_by_api.txt

Output:

/orders 210.5
/login 95.0

What it means: Two arrays: sum and count.

6. printf, variables and small programs

Why do we need this?

Reports must line up, and sometimes you need a value from the shell or an if/else.

What is it?

printf "%-10s %8.2f\n", a, b: %s text, %d integer, %f decimal, %-10s left aligned width 10, %8.2f width 8 with 2 decimals. Pass shell values with -v name="$VAR" (never paste $VAR inside the program). awk has if/else, for, while, next (skip to the next line), and functions: length(), substr(s,i,n), tolower(), toupper(), split(s,a,sep), gsub(/re/,"x") and sub(), index(), sprintf(), int().

How to do it (step by step)

1. A neat table.

awk -F, 'NR>1 {printf "%-8s %-8s %8.2f\n", $2, $3, $4}' sales.csv

Output:

Pune     Imran     5600.00
Mumbai   Asha      7200.00

What it means: Columns line up.

2. Pass a shell value safely.

LIMIT=5000; awk -F, -v lim="$LIMIT" 'NR>1 && $4 > lim {print $3}' sales.csv

Output:

Asha
Imran

What it means: -v sets an awk variable.

3. if/else labels.

awk '{if ($2 >= 35) r="PASS"; else r="FAIL"; print $1, r}' marks.txt

Output:

Sneha PASS
Raj FAIL

What it means: Normal programming logic.

4. Replace inside a field.

echo 'Poona,4500' | awk -F, 'BEGIN{OFS=","} {gsub(/Poona/,"Pune",$1); print}'

Output:

Pune,4500

What it means: gsub replaces all matches in $1.

5. Substring and upper case.

echo '2026-10-25 web01' | awk '{print substr($1,1,7), toupper($2)}'

Output:

2026-10 WEB01

What it means: substr(s, start, length).

6. Skip lines with next.

awk '/^#/ {next} NF {print}' app.conf

Output:

port=8080
mode=prod

What it means: next jumps to the next line; NF skips blank lines.

7. A script file.

printf 'BEGIN{FS=","}\nNR>1{t+=$4}\nEND{print t}\n' > total.awk && awk -f total.awk sales.csv

Output:

22500

What it means: Long programs go in a .awk file.

7. Real log analysis with awk

Why do we need this?

During an incident you have minutes to answer: which URL is slow, which status codes, what changed in the last hour.

What is it?

Know your log format. nginx combined: $1 IP, $4 time, $7 URL, $9 status, $10 bytes; add $request_time at the end to use $NF as seconds. Combine filters, arrays and sort. For CSV with quoted commas, use a real CSV tool (csvkit, Miller mlr, or gawk --csv in gawk 5.3+).

How to do it (step by step)

1. Status code counts.

awk '{c[$9]++} END {for (s in c) print s, c[s]}' access.log | sort

Output:

200 9120
301 210
404 77
500 14

What it means: A quick health picture.

2. Slowest URLs by average time.

awk '{t[$7]+=$NF; n[$7]++} END {for (u in t) printf "%.3f %s\n", t[u]/n[u], u}' access.log | sort -nr | head -3

Output:

2.140 /api/report
0.910 /api/search
0.120 /cart

What it means: $NF is request time.

3. Bytes served per hour.

awk '{split($4,a,":"); b[a[2]]+=$10} END {for (h in b) print h, int(b[h]/1048576) " MB"}' access.log | sort

Output:

09 512 MB
10 830 MB

What it means: split breaks the time into parts; a[2] is the hour.

4. Top memory processes.

ps aux | awk 'NR>1 {print $4, $11}' | sort -nr | head -3

Output:

12.4 /usr/bin/java
3.1 /usr/sbin/mysqld
1.2 nginx:

What it means: $4 is %MEM, $11 the command.

5. Lines in a time window.

awk '$2 >= "10:00:00" && $2 <= "10:05:00"' app.log | wc -l

Output:

412

What it means: Text compare works for fixed-width times.

8. Real examples from many industries

Why do we need this?

The same Linux skill shows up in every kind of company. Seeing it in many places helps you remember it.

What is it?

Each row is a made-up team. It shows who uses this and how.

How to do it (step by step)

9. Common errors and how to fix them

Why do we need this?

Everyone hits these errors. Knowing the cause saves hours.

What is it?

Read the exact message first. Then match it to a row below.

How to do it (step by step)

10. Troubleshooting step by step

Why do we need this?

When an awk report is wrong, look at the fields awk sees, one step at a time.

What is it?

Go top to bottom. Stop at the first step that shows the problem.

How to do it (step by step)

1. See how many fields each line has.

awk -F, '{print NR, NF}' data.csv | head

2. Print the field with markers to see spaces.

awk -F, '{print "["$4"]"}' data.csv | head -3

3. Check for hidden \r.

head -2 data.csv | cat -A

4. Check the filter alone.

awk -F, '$4 > 5000' data.csv | head

5. Check which awk you have.

awk --version 2>/dev/null | head -1 || awk -W version

6. Guard against empty input in END.

awk '{n++} END{print n+0}' data.csv

11. Lab 1: Monthly milk bill report

Why do we need this?

Build a full report with totals, groups and printf.

What is it?

A full hands-on lab. Run every step on a test Linux machine (an EC2 instance, a VM, or WSL).

How to do it (step by step)

1. Create the delivery file.

mkdir -p ~/lab11 && cd ~/lab11 && for d in $(seq -w 1 30); do echo "2026-10-$d A-101 2 56"; echo "2026-10-$d B-204 $(( (10#$d % 3) + 1 )) 56"; done > milk.txt; head -2 milk.txt

Output:

2026-10-01 A-101 2 56
2026-10-01 B-204 2 56

2. Total litres.

awk '{s+=$3} END{print s}' milk.txt

Output:

120

3. Bill per house.

awk '{b[$2]+=$3*$4} END{for(h in b) printf "%-6s Rs %6d\n", h, b[h]}' milk.txt | sort

Output:

A-101  Rs   3360
B-204  Rs   3360

4. Days when B-204 took 3 litres.

awk '$2=="B-204" && $3==3 {n++} END{print n}' milk.txt

Output:

10

5. Average litres per day per house.

awk '{s[$2]+=$3; n[$2]++} END{for(h in s) printf "%s %.2f\n", h, s[h]/n[h]}' milk.txt | sort

Output:

A-101 2.00
B-204 2.00

12. Lab 2: Web log incident report

Why do we need this?

Answer real incident questions: status codes, slow URLs, top IPs.

What is it?

A full hands-on lab. Run every step on a test Linux machine (an EC2 instance, a VM, or WSL).

How to do it (step by step)

1. Make a sample log (IP, URL, status, seconds).

cd ~/lab11 && for i in $(seq 1 500); do echo "10.0.0.$((i%5)) /api/p$((i%4)) $([ $((i%25)) -eq 0 ] && echo 500 || echo 200) 0.$((i%9))"; done > web.log

Output:

(no output)

2. Status counts.

awk '{c[$3]++} END{for(s in c) print s, c[s]}' web.log | sort

Output:

200 480
500 20

3. Average time per URL.

awk '{t[$2]+=$4; n[$2]++} END{for(u in t) printf "%.3f %s\n", t[u]/n[u], u}' web.log | sort -nr

Output:

0.404 /api/p1 ...(4 lines)

4. Which URLs return 500?

awk '$3==500 {u[$2]++} END{for(k in u) print u[k], k}' web.log | sort -nr

Output:

20 /api/p0

5. Top 2 IPs.

awk '{c[$1]++} END{for(i in c) print c[i], i}' web.log | sort -nr | head -2

Output:

100 10.0.0.0
100 10.0.0.1

13. Frequently asked questions

14. Interview questions and answers

Beginner level

Mid level level

Senior level

15. Related chapters

• Linux Ops Refresh for DevOps
• Bash Scripting for Automation
• Linux grep and find Commands with Examples
• How to Set Up Passwordless SSH Between Linux Servers

Revision, comparison and interview cards

Revise this lesson in a few minutes. Save the cards on your phone or download them as PNG files. The QR code on each card opens this lesson.

More cards: Ravi Sir's Cards.