Ravindra BagaleCourses & study guides मराठी Track your progress

6. Linux Advanced Commands

6.3 Text Processing: awk and sed

By Last updated:

Also in the AWS course: this lesson is the same as 4.3 Text processing: awk and sed in the AWS course, which is the main version of this page.

awk columns sathi, sed find-and-replace sathi. Donhi log analysis aani config automation madhe roj lagtat.

awk — column-based processing

awk splits each line into fields $1, $2, ... (space-separated by default; -F sets the separator).

awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head
#  ^ top 10 client IPs hitting your web server

awk -F: '{print $1, $7}' /etc/passwd          # user name and shell
awk -F: '$3 >= 1000 {print $1}' /etc/passwd   # normal (non-system) users
df -h | awk 'NR>1 {print $5, $6}'             # usage % and mount point
awk '{sum += $10} END {print sum/1024/1024 " MB"}' /var/log/nginx/access.log  # bytes served

sed — stream editor (find & replace)

sed 's/http/https/' file.txt              # replace first match per line (prints result)
sed 's/http/https/g' file.txt             # replace all matches
sed -i 's/Listen 80/Listen 8080/' /etc/httpd/conf/httpd.conf   # edit file in place
sed -i.bak 's/old/new/g' config.ini       # in place, keep backup config.ini.bak
sed -n '10,20p' file.txt                  # print only lines 10-20
sed '/^#/d' file.txt                      # delete comment lines

Test sed before using -i

Run the sed command without -i first and check the output. Once you are happy, add -i (or -i.bak to keep a backup).

Why this matters for security

The first awk one-liner above (top client IPs) is how you spot a brute-force or scanning source in a web log within seconds – the attacker's IP jumps to the top of the list.

Ravindra Bagale's Tip

Many students run sed -i directly on a live config – and one wrong pattern ruins the whole file. First look at the output without -i, then use -i.bak so a backup is created automatically.

Practice task

From /etc/passwd, print the user names whose shell is /bin/bash. Using sed, print lines 5–10 of any config file and delete comment lines in a copy of it.

Extended teaching notes

Why this lesson matters

Use this lesson to strengthen systems you own: understand the risk, practise safely in a lab, then write and apply the fix.

Apply the fix across stacks

Same class of bug, same idea in every language.

Java / Spring Boot

Parameterise queries, encode output, CSRF tokens, limited Actuator, dependency scanning.

ASP.NET Core

EF Core / parameterised SQL, Razor encoding, HTTPS, Data Protection.

Node.js / Express

helmet, parameterised SQL, npm audit, avoid eval and shell concat.

Python / Django / Flask / FastAPI

ORM, autoescape, pip-audit, Bandit; never shell=True with user input.

PHP / Laravel

PDO/Eloquent, Blade escaping, composer audit.

Go / Ruby

database/sql placeholders / ActiveRecord; html/template / ERB escaping; govulncheck / Brakeman.

nginx / Apache / Tomcat

Hide versions, TLS, security headers, least-open ports.

MySQL / PostgreSQL / MongoDB / Redis

Private networks, auth on, least-privilege users, no 0.0.0.0/0.

AWS / Azure / GCP

Roles not long-lived keys, private data stores, cloud audit logs on.

Mini case study

Suppose we are on a purple-team exercise for an Indian mid-size company (sample data). Red notes a technique from this lesson against a lab clone. Blue must show the control that stops it, the log that would have fired, and the ticket that hardens production. That is the standard this course trains you for.

Defender's worksheet (do this today)

  1. Write the lesson title, date and the exact lab target you are allowed to use.
  2. List three owned systems (VMs, apps, or cloud accounts) where this lesson applies.
  3. Run one safe check from the lesson on a lab system only. Paste the command and output.
  4. Write a two-sentence fix you would give a teammate.
  5. Name one related public advisory (CERT-In, CISA, NVD, or a vendor bulletin) with its date.

Indian workplace (sample data)

Suppose we are a junior analyst at Infosys, TCS, Wipro or a fintech SOC in Bengaluru/Pune. Turn this lesson into one checklist item with an owner, a due date and a place to store evidence (ticket + screenshot).

Global practice

Compare your notes with OWASP, CIS Benchmarks, NIST CSF/IR guidance or a cloud Well-Architected security question. Write five lines on what overlaps.

Everyday analogy

Explain the risk and the fix to a family member without jargon. If they can repeat the warning, you understand it well enough to teach juniors.

Key terms

  • Control: a safeguard (patch, config, process, detection).
  • Finding: a weakness backed by evidence.
  • Remediation: the fix plus how you verified it.
  • Scope: systems you have written permission to touch.
  • Least privilege: only the access the task needs.
  • Defence in depth: several layers so one failure does not end the story.

Common mistakes

  • Skipping written permission "just for a quick scan".
  • Practising on hostel, office or public targets without authorisation.
  • Collecting tool output without writing the fix.
  • Copying attack steps onto production to "verify". Defenders verify with versions, configs and patches.
  • Publishing screenshots that contain secrets, cookies or internal hostnames.

Recap

  • Evidence, fix, owner, date.
  • Own lab only.
  • Fix > spectacle.