83.6 Project 96: Distributed tracing with OpenTelemetry + Jaeger
Friends, logs alone do not show a slow hop. Today we emit OpenTelemetry traces and view a request path in Jaeger.
मित्रांनो, फक्त logs slow hop दाखवत नाहीत. आज OpenTelemetry traces आणि Jaeger मध्ये request path.
दोस्तों, सिर्फ logs slow hop नहीं दिखाते. आज OpenTelemetry traces और Jaeger में request path.
What you will build in this project
- Run Jaeger all-in-one.
- Instrument a sample service.
- Send a trace.
- Find the slow span.
1. Start from a real company story
At PhonePe in Bengaluru, a checkout timeout looked like the API but the real delay was a downstream KYC call.
Rough cost: about Rs 30 lakh in abandoned payments that evening.
Sana added OTel traces and found the KYC span in Jaeger.
Today we follow one request across services.
Ravindra Bagale's Tip
Students sample 100 percent in prod forever. Use head sampling and raise only during incidents. Keep that habit.
Students prod मध्ये नेहमी 100 percent sample. Head sampling; incident वेळी वाढवा. सवय ठेवा.
Students prod में हमेशा 100 percent sample. Head sampling; incident पर बढ़ाएं. आदत रखिए.
2. Start and prepare
Step 1: Why this project
Why: Real outages teach. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: echo why. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 1: a1
Type this:
echo 'Today we build otel96'
You will see:
Today we build otel96
What it means: Why.
Example 2: a2
Type this:
mkdir -p /tmp/otel96 && echo 'Scope ready'
You will see:
Scope ready
What it means: Scope.
Example 3: a3
Type this:
echo 'Keep commands short and check every output'
You will see:
Keep commands short and check every output
What it means: Habit.
Step 2: Prepare workspace
Why: Folders and notes. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: prep. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 4: b1
Type this:
mkdir -p /tmp/otel96 && printf '%s\n' 'lab notes' > /tmp/otel96/NOTES.txt && cat /tmp/otel96/NOTES.txt
You will see:
lab notes
What it means: Notes.
Example 5: b2
Type this:
echo 'Work only under /tmp/otel96 for this project'
You will see:
Work only under /tmp/otel96 for this project
What it means: Path.
Example 6: b3
Type this:
echo 'Write ACCEPT.txt at the end'
You will see:
Write ACCEPT.txt at the end
What it means: Accept file.
Step 3: Jaeger all-in-one
Why: Lab collector. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: jaeger. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 7: c1
Type this:
echo 'docker run -d --name jaeger -p 16686:16686 -p 4318:4318 jaegertracing/all-in-one:1.53'
You will see:
docker run -d --name jaeger -p 16686:16686 -p 4318:4318 jaegertracing/all-in-one:1.53
What it means: Jaeger.
Example 8: c2
Type this:
echo 'UI http://localhost:16686 OTLP http://localhost:4318'
You will see:
UI http://localhost:16686 OTLP http://localhost:4318
What it means: Ports.
Example 9: c3
Type this:
echo 'Prod often uses a collector gateway and object storage'
You will see:
Prod often uses a collector gateway and object storage
What it means: Prod.
3. Build the main pieces
Step 4: Instrument
Why: SDK tip. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: sdk. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 10: d1
Type this:
printf '%s\n' 'Use OpenTelemetry SDK for your language' 'Export OTLP to the collector or Jaeger' 'Create a span per inbound request and outbound call' > /tmp/otel96/SDK.txt && cat /tmp/otel96/SDK.txt
You will see:
Create a span per inbound request and outbound call
What it means: SDK.
Example 11: d2
Type this:
echo 'Auto-instrumentation exists for many frameworks'
You will see:
Auto-instrumentation exists for many frameworks
What it means: Auto.
Example 12: d3
Type this:
echo 'Always propagate traceparent headers'
You will see:
Always propagate traceparent headers
What it means: Headers.
Step 5: Send a trace
Why: Smoke. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: send. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 13: e1
Type this:
printf '%s\n' 'Hit the demo /checkout endpoint once' 'Open Jaeger UI search service shop-api' > /tmp/otel96/SEND.txt && cat /tmp/otel96/SEND.txt
You will see:
Open Jaeger UI search service shop-api
What it means: Send.
Example 14: e2
Type this:
echo 'Click the trace id to see the span timeline'
You will see:
Click the trace id to see the span timeline
What it means: UI.
Example 15: e3
Type this:
printf '%s\n' 'Acceptance: checkout trace shows api and kyc spans' > /tmp/otel96/ACCEPT.txt && cat /tmp/otel96/ACCEPT.txt
You will see:
Acceptance: checkout trace shows api and kyc spans
What it means: Gate.
Step 6: Sampling
Why: Cost control. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: sample. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 16: f1
Type this:
printf '%s\n' 'Head sampling: decide at the root span' 'Tail sampling: keep interesting traces after the fact' > /tmp/otel96/SAMPLE.txt && cat /tmp/otel96/SAMPLE.txt
You will see:
Head sampling: decide at the root span
What it means: Sample.
Example 17: f2
Type this:
echo 'Start at 1 to 5 percent in busy prod services'
You will see:
Start at 1 to 5 percent in busy prod services
What it means: Rate.
Example 18: f3
Type this:
echo 'Raise sampling temporarily during an incident'
You will see:
Raise sampling temporarily during an incident
What it means: Incident.
4. Automate and verify
Step 7: Attributes
Why: Useful fields. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: attrs. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 19: g1
Type this:
printf '%s\n' 'http.route, http.status_code, enduser.id hashed' 'Never put secrets or full card numbers in spans' > /tmp/otel96/ATTR.txt && cat /tmp/otel96/ATTR.txt
You will see:
Never put secrets or full card numbers in spans
What it means: Attrs.
Example 20: g2
Type this:
echo 'Consistent attribute names help search'
You will see:
Consistent attribute names help search
What it means: Names.
Example 21: g3
Type this:
echo 'Error spans should set status and record exceptions'
You will see:
Error spans should set status and record exceptions
What it means: Errors.
Step 8: Collector
Why: Gateway. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: collector. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 22: h1
Type this:
echo 'otelcol can receive OTLP and export to Jaeger or a vendor'
You will see:
otelcol can receive OTLP and export to Jaeger or a vendor
What it means: Collector.
Example 23: h2
Type this:
echo 'Put the collector in the mesh path to centralize sampling'
You will see:
Put the collector in the mesh path to centralize sampling
What it means: Path.
Example 24: h3
Type this:
echo 'Batch export to cut network chatter'
You will see:
Batch export to cut network chatter
What it means: Batch.
Step 9: With metrics and logs
Why: Three pillars. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: pillars. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 25: i1
Type this:
printf '%s\n' 'Metrics: is it broken' 'Logs: what did it say' 'Traces: where did time go' > /tmp/otel96/PILLARS.txt && cat /tmp/otel96/PILLARS.txt
You will see:
Traces: where did time go
What it means: Pillars.
Example 26: i2
Type this:
echo 'Share trace_id into logs for jumps'
You will see:
Share trace_id into logs for jumps
What it means: Link.
Example 27: i3
Type this:
echo 'Dashboards plus traces close incidents faster'
You will see:
Dashboards plus traces close incidents faster
What it means: Speed.
5. Harden and accept
Step 10: Story PhonePe
Why: Bengaluru lesson. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: story. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 28: j1
Type this:
printf '%s\n' 'At PhonePe in Bengaluru, checkout timeouts were blamed on the API until traces showed KYC.' 'Sana added OTel and found the KYC span.' > /tmp/otel96/STORY.txt && wc -l /tmp/otel96/STORY.txt
You will see:
2 /tmp/otel96/STORY.txt
What it means: Story.
Example 29: j2
Type this:
echo 'Rough cost: about Rs 30 lakh in abandoned payments that evening.'
You will see:
Rough cost: about Rs 30 lakh in abandoned payments that evening.
What it means: Cost.
Example 30: j3
Type this:
echo 'Sana: PASS'
You will see:
Sana: PASS
What it means: Sign-off.
Step 11: Proof
Why: Files. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: ls. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 31: k1
Type this:
ls /tmp/otel96
You will see:
NOTES.txt\nSTORY.txt\nACCEPT.txt
What it means: Tree.
Example 32: k2
Type this:
test -f /tmp/otel96/ACCEPT.txt && echo ACCEPT_OK
You will see:
ACCEPT_OK
What it means: Accept present.
Example 33: k3
Type this:
echo 'Monitoring projects 91-96 complete; next Capstone 97'
You will see:
Monitoring projects 91-96 complete; next Capstone 97
What it means: Bridge.
Step 12: Acceptance
Why: Trace shows the slow span. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: final. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 34: z1
Type this:
echo 'Jaeger ready'
You will see:
Jaeger ready
What it means: A.
Example 35: z2
Type this:
echo 'Trace received'
You will see:
Trace received
What it means: B.
Example 36: z3
Type this:
echo 'Slow span found'
You will see:
Slow span found
What it means: C.
Check yourself
Q1. Who found PhonePe KYC delay?
Sana
Q2. API?
OpenTelemetry
Q3. UI?
Jaeger
Q4. Prod sample 100% forever?
no
Q5. Context?
propagate
Q6. Shows?
spans
Friends, you traced a request end to end. Did you understand? Let us try it once more. Add an artificial sleep span and prove Jaeger highlights it.
मित्रांनो, तुम्ही request end to end trace केली. Artificial sleep span टाका आणि Jaeger highlight करते हे सिद्ध करा.
दोस्तों, आपने request end to end trace की. Artificial sleep span डालें और Jaeger highlight करे, यह साबित करें.
Lab: try at home
- Jaeger up.
- Trace emitted.
- UI shows path.
- Slow span visible.
Recap
- Traces show hops.
- OTel is the standard.
- Sample wisely.
- Propagate context.
- Combine with metrics.
Lab
- Finish all six projects; keep ACCEPT notes for each.
- Never page on raw noisy counters without a period.
- Keep Loki labels low-cardinality; put ids in the line body.
- After each project, write one paragraph: what broke and how you fixed it.
- Next batch is the Capstone projects 97-100.
Practice task
Pick the Alertmanager drill and the Jaeger slow-span hunt. Prove both before you call the chapter done.
Ravindra Bagale's Tip
Golden signals first. Alert on symptoms. Logs need retention. Low-cardinality labels. Test alarms. Sample traces.
Golden signals प्रथम. Symptoms वर alert. Logs ला retention. Low-cardinality labels. Alarms test. Traces sample.
Golden signals पहले. Symptoms पर alert. Logs को retention. Low-cardinality labels. Alarms test. Traces sample.
In short
- Prometheus scrapes.
- Grafana shows.
- Alertmanager routes.
- EFK searches logs.
- Loki beside metrics.
- CloudWatch for AWS.
- OTel traces hops.
- Sample wisely.
- Test every alarm.
- Link the three pillars.