Shell Scripting Complete 10: Redirection, Pipes, tee, Here-docs and Process Substitution

By Ravindra Bagale, trainer LinkedIn Last updated: 25 October 2026

Every Linux command has three doors: input (stdin), normal output (stdout) and error output (stderr). Redirection decides where each door points. In this chapter you learn > and >> to write files, 2> and &> for errors, < to read, pipes | to join commands, tee to save and show, here-docs <

What you'll learn in this chapter

Know stdin, stdout and stderr (0, 1, 2)
Write and append with > and >>
Split or join errors with 2> and 2>&1
Build pipes and use tee
Write files with here-docs
Compare outputs with <( )

1. An everyday example: water pipes in a building

Why do we need this?

In a Kothrud society, water comes from the tank (input), goes to the kitchen tap (normal output), and waste water goes to a separate drain (errors). A plumber can send the tap water into a bucket instead, join two pipes, or add a T-joint so water goes two ways at once. Redirection and pipes are plumbing for data.

What is it?

Tank = stdin (0). Tap = stdout (1). Drain = stderr (2). Bucket = a file with >. Joining pipes = |. T-joint = tee. A tanker delivering many litres at once = a here-doc.

How to do it (step by step)

Order matters: > file 2>&1, not 2>&1 > file.

2. Write and append: > and >>

Why do we need this?

Reports, logs and generated configs must land in files, not just on the screen.

What is it?

cmd > file writes stdout to file and replaces old content. cmd >> file adds to the end. > file alone (or : > file) empties a file without deleting it, good for log files a program has open. set -o noclobber stops > from overwriting by mistake; >| forces it.

Form	Does
> f	overwrite
>> f	append
: > f	empty the file
< f	read input from f
>\| f	overwrite even with noclobber

How to do it (step by step)

Save a list.

ls /etc > etc-list.txt; wc -l etc-list.txt

Output:

212 etc-list.txt

What it means: Screen shows nothing.

Append a line.

date >> run.log; date >> run.log; wc -l run.log

Output:

2 run.log

What it means: Two lines.

Empty a log.

: > /var/log/app/app.log; ls -l /var/log/app/app.log

Output:

-rw-r--r-- 1 app app 0 Oct 25 10:12 /var/log/app/app.log

What it means: Size 0, file kept.

Protect files.

set -o noclobber; echo hi > etc-list.txt

Output:

bash: etc-list.txt: cannot overwrite existing file

What it means: Safety on.

Read from a file.

wc -l < etc-list.txt

Output:

What it means: No file name printed.

3. Errors: 2>, &> and 2>&1

Why do we need this?

In cron jobs you want errors in a log, and noisy warnings away from your report.

What is it?

2> file sends only errors to file. 2>/dev/null throws errors away. cmd > out.log 2>&1 sends both to the same file: first move 1, then point 2 at 1. &> file is the bash short form for both. |& pipes both to the next command. Do not hide errors you need; hide only known noise.

How to do it (step by step)

Errors only.

find / -name '*.pem' 2> find-errors.txt | head -2

Output:

/etc/pki/tls/cert.pem
/workspace/key.pem

What it means: Permission errors went to the file.

Hide noise.

find / -name sshd_config 2>/dev/null

Output:

/etc/ssh/sshd_config

What it means: Clean output.

Both in a log.

./backup.sh > backup.log 2>&1; tail -2 backup.log

Output:

tar: /data/x: Cannot open: Permission denied
backup done

What it means: Errors and output together.

Short form.

./backup.sh &>> backup.log

What it means: Append both.

Separate files.

./import.sh > ok.log 2> err.log

What it means: Easy to check err.log size.

4. Pipes | and tee

Why do we need this?

Small tools joined together answer big questions: which IP hit us most, which product sold most.

What is it?

a | b sends a's stdout into b's stdin. Chain as many as you need. tee file writes to file and passes data on, so you can save a middle step and still see it. tee -a appends. sudo tee is the way to write a root-owned file, because sudo echo x > file does not give the > root rights.

How to do it (step by step)

Top IPs.

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -3

Output:

  540 49.36.12.8
  311 103.21.4.9
   97 66.249.66.1

What it means: Five tools, one answer.

Save and show.

df -h | tee disk-report.txt

Output:

Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 20G 8.1G 12G 41% /

What it means: Seen and saved.

Root file.

echo '10.0.1.5 db1' | sudo tee -a /etc/hosts

Output:

10.0.1.5 db1

What it means: Works where sudo echo > fails.

Count lines in a pipe.

journalctl -u nginx --since today | grep -c error

Output:

What it means: Errors today.

5. Here-docs: <

Why do we need this?

Writing a config file, an SQL script or an email body line by line with echo is ugly and error-prone.

What is it?

cmd <

Form	Effect
<	expand $vars
<<'EOF'	no expansion
<<-EOF	strip leading tabs
<<<	one string as input

How to do it (step by step)

Write a config.

app=shop; cat > /tmp/app.conf <<EOF
name=$app
port=8080
EOF
cat /tmp/app.conf

Output:

name=shop
port=8080

What it means: $app expanded.

Keep $ as text.

cat > /tmp/hello.sh <<'EOF'
echo "Hello $USER"
EOF
cat /tmp/hello.sh

Output:

echo "Hello $USER"

What it means: Quoted word, no expansion.

SQL in one go.

psql -h db1 -U app shop <<'SQL'
SELECT city, count(*) FROM orders GROUP BY city;
SQL

Output:

  city  | count
 Pune   |  1200

What it means: Many lines to psql.

Root file with here-doc.

sudo tee /etc/motd > /dev/null <<'EOF'
Authorized users only. Managed by ops team.
EOF

What it means: tee writes as root.

6. Here-strings <<<

Why do we need this?

Sometimes you have one value in a variable and a command wants it on stdin.

What is it?

cmd <<< "$v" feeds the string plus a newline to cmd. It avoids echo "$v" | cmd and, unlike a pipe, read runs in the current shell, so variables it sets stay after.

How to do it (step by step)

Split a CSV line.

IFS=, read -r id city amt <<< '101,Pune,499'; echo "$city $amt"

Output:

Pune 499

What it means: Variables stay set.

Count words.

wc -w <<< 'build test deploy'

Output:

What it means: No echo needed.

bc input.

bc <<< 'scale=2; 22/7'

Output:

3.14

What it means: Short maths.

7. Process substitution <( ) and >( )

Why do we need this?

diff and comm want two files, but you have two command outputs, for example packages on two servers.

What is it?

<(cmd) runs cmd and gives a file name like /dev/fd/63 that reads its output. You can pass it to any command that wants a file. >(cmd) is the reverse: what you write goes into cmd. while read ... done < <(cmd) loops over output without the subshell problem of a pipe, so counters survive.

How to do it (step by step)

Diff two servers.

diff <(ssh web1 'rpm -qa | sort') <(ssh web2 'rpm -qa | sort')

Output:

< nginx-1.24.0-1
> nginx-1.22.1-1

What it means: Version drift found.

What is it?

echo <(true)

Output:

/dev/fd/63

What it means: A file name.

Loop keeps counter.

n=0; while read -r u; do ((n++)); done < <(cut -d: -f1 /etc/passwd); echo $n

Output:

What it means: With a pipe n would be 0.

Write to two places.

./export.sh | tee >(gzip > export.gz) | wc -l

Output:

What it means: Count and compress at once.

8. File descriptors and exec

Why do we need this?

Long scripts want all output logged without adding > log to every line.

What is it?

exec > file redirects the rest of the script's stdout. exec > >(tee -a run.log) 2>&1 logs everything while still showing it. exec 3> file opens a new descriptor 3; write with >&3 and close with exec 3>&-. /dev/null is the black hole; /dev/stderr is a name for fd 2.

How to do it (step by step)

Log the whole script.

exec > >(tee -a /var/log/nightly.log) 2>&1
echo 'nightly start'

Output:

nightly start

What it means: Every later line is logged.

Extra descriptor.

exec 3>> audit.log; echo 'user asha ran deploy' >&3; exec 3>&-; cat audit.log

Output:

user asha ran deploy

What it means: Audit kept apart.

Read from fd.

exec 4< servers.txt; read -r first <&4; echo $first

Output:

web1

What it means: Read one line.

9. Real examples from many industries

Why do we need this?

The same Linux skill shows up in every kind of company. Seeing it in many places helps you remember it.

What is it?

Each row is a made-up team. It shows who uses this and how.

How to do it (step by step)

Team and city	How they use it	Command
Rohan, banking, Mumbai	Separate error log for EOD batch	`./eod.sh > eod.log 2> eod.err`
Emma, media, Berlin	Top referrers from access log	`awk '{print $11}' a.log \| sort \| uniq -c \| sort -rn`
Kenji, robotics, Tokyo	Write device config with here-doc	`cat > dev.conf <<EOF`
Fatima, government, Riyadh	Compare user lists of two servers	`comm -3 <(sort a) <(sort b)`
Lucas, SaaS, Sao Paulo	Log whole deploy script	`exec > >(tee -a deploy.log) 2>&1`
Pooja, pharma, Hyderabad	Append to root-owned hosts	`echo "$ip db" \| sudo tee -a /etc/hosts`

10. Common errors and how to fix them

Why do we need this?

Everyone hits these errors. Knowing the cause saves hours.

What is it?

Read the exact message first. Then match it to a row below.

How to do it (step by step)

Error you see	Why it happens	How to fix
`Permission denied with sudo echo x > file`	> runs as you, not root	Use sudo tee. `echo x \| sudo tee file`
`Errors still on screen after 2>&1 > f`	Wrong order	Put > f first. `cmd > f 2>&1`
`File became empty`	> on a file you read	Write to a temp then move. `sort f > f.tmp && mv f.tmp f`
`warning: here-document delimited by end-of-file`	Closing word missing or indented	Put EOF alone at line start. `EOF`
`Variables empty after while loop`	Pipe runs loop in a subshell	Use < <(cmd). `done < <(cmd)`
`cannot overwrite existing file`	noclobber is on	Use >\| or set +o noclobber. `echo x >\| f`
syntax error near unexpected token `('	Process substitution in sh	Run with bash. `bash script.sh`
`$VAR printed literally in file`	Used <<'EOF'	Use unquoted EOF to expand. `cat <<EOF`
`Disk full from a log`	>> never rotated	Empty or rotate it. `: > big.log`

11. Troubleshooting step by step

Why do we need this?

When output goes to the wrong place, test each stream on its own.

What is it?

Go top to bottom. Stop at the first step that shows the problem.

How to do it (step by step)

Only stdout.

cmd 2>/dev/null

Only stderr.

cmd > /dev/null

See open fds.

ls -l /proc/$$/fd

Check order.

cmd > f 2>&1

Pipe statuses.

echo "${PIPESTATUS[@]}"

Lint.

shellcheck s.sh

12. Lab 1: Log report pipeline with tee

Why do we need this?

Build a small access log, find the top IPs and status codes with a pipe, and save the report while seeing it.

What is it?

A full hands-on lab. Run every step on a test Linux machine (an EC2 instance, a VM, or WSL).

How to do it (step by step)

Make a log with a here-doc.

cat > ~/access.log <<'EOF'
49.36.12.8 - - [25/Oct/2026] "GET / HTTP/1.1" 200
49.36.12.8 - - [25/Oct/2026] "GET /cart HTTP/1.1" 500
103.21.4.9 - - [25/Oct/2026] "GET / HTTP/1.1" 200
49.36.12.8 - - [25/Oct/2026] "GET /pay HTTP/1.1" 200
EOF

Top IPs.

awk '{print $1}' ~/access.log | sort | uniq -c | sort -rn

Output:

      3 49.36.12.8
      1 103.21.4.9

Status codes saved.

awk '{print $NF}' ~/access.log | sort | uniq -c | tee ~/status.txt

Output:

      3 200
      1 500

Errors to a file.

grep ' 500$' ~/access.log > ~/errors.txt; wc -l < ~/errors.txt

Output:

Clean up.

rm ~/access.log ~/status.txt ~/errors.txt

13. Lab 2: Compare two configs with process substitution

Why do we need this?

Write two configs with here-docs, then compare them sorted and without comments using <( ) and no temp files.

What is it?

A full hands-on lab. Run every step on a test Linux machine (an EC2 instance, a VM, or WSL).

How to do it (step by step)

Config A.

cat > ~/a.conf <<'EOF'
# prod
port=80
workers=4
EOF

Config B.

cat > ~/b.conf <<'EOF'
workers=8
port=80
# staging
EOF

Plain diff is noisy.

diff ~/a.conf ~/b.conf | wc -l

Output:

Clean compare.

diff <(grep -v '^#' ~/a.conf | sort) <(grep -v '^#' ~/b.conf | sort)

Output:

2c2
< workers=4
---
> workers=8

Clean up.

rm ~/a.conf ~/b.conf

Ravindra Bagale's Tip

Remember the three doors 0, 1 and 2. Write > file 2>&1 in that order, use tee when you need to save and see, use <<'EOF' when writing scripts or configs into files, and use < <(cmd) when a loop must keep its variables.

14. Frequently asked questions

Q. What is /dev/null?

A file that throws data away.

Q. > vs >>?

Replace vs add.

Q. &> in sh?

No, use > f 2>&1.

Q. Here-doc indent?

Use <<- with tabs.

Q. Why tee with sudo?

Redirect runs as you.

Q. Is <( ) a real file?

A pipe shown as /dev/fd/N.

Q. Pipe both streams?

Use |&.

Q. Edit a file in place with >?

No; temp file then mv.

15. Interview questions and answers

Beginner level

Q. What is fd 2?

stderr.

Q. Append to a file?

>>.

Q. Hide errors?

2>/dev/null.

Q. Join commands?

Pipe |.

Mid level level

Q. Both streams to one file?

> f 2>&1.

Q. Save and show?

tee.

Q. Stop $ expanding in here-doc?

Quote the word.

Q. Diff two command outputs?

diff <(a) <(b).

Senior level

Q. Why does 2>&1 > f fail?

2 copies the old target of 1.

Q. Pipe and subshell?

Each part may run in a subshell; vars are lost.

Q. Log a whole script?

exec > >(tee -a log) 2>&1.

Q. Safe in-place edit?

Temp file and atomic mv.

Try at home

Use a here-doc to create a shopping list file, then pipe it through sort and tee into a sorted copy.
Compare the list of files in two folders with diff <( ) <( ).

Practice Tasks

About this scenario

London Legal LLP is a law business in London. Oliver Hughes looks after its IT and cloud work. They count money in GBP and run their AWS resources in the eu-west-2 Region. In the tasks below you do real jobs for them using what this lesson teaches. All names, account IDs, IP addresses and numbers are made up for practice.

0 / 21