83.3 Project 93: ELK/EFK stack for logs
Friends, SSH grepping does not scale. Today we ship container logs to Elasticsearch through Fluent Bit and search in Kibana.
मित्रांनो, SSH grep scale होत नाही. आज Fluent Bit ने Elasticsearch, Kibana search.
दोस्तों, SSH grep scale नहीं होता. आज Fluent Bit से Elasticsearch, Kibana search.
What you will build in this project
- Run Elasticsearch.
- Ship logs with Fluent Bit.
- Open Kibana.
- Find an error by request id.
1. Start from a real company story
At Ola in Bengaluru, engineers SSH-ed into twenty nodes during an outage and still missed the error.
Rough cost: about Rs 25 lakh in prolonged downtime.
Priya centralized logs with EFK and a request-id field.
Today we search one error across pods.
Ravindra Bagale's Tip
Students keep every log forever. Set ILM retention in days. Keep that habit.
Students सर्व logs forever ठेवतात. ILM retention days मध्ये. सवय ठेवा.
Students सभी logs forever रखते हैं. ILM retention days में. आदत रखिए.
2. Start and prepare
Step 1: Why this project
Why: Real outages teach. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: echo why. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 1: a1
Type this:
echo 'Today we build efk93'
You will see:
Today we build efk93
What it means: Why.
Example 2: a2
Type this:
mkdir -p /tmp/efk93 && echo 'Scope ready'
You will see:
Scope ready
What it means: Scope.
Example 3: a3
Type this:
echo 'Keep commands short and check every output'
You will see:
Keep commands short and check every output
What it means: Habit.
Step 2: Prepare workspace
Why: Folders and notes. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: prep. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 4: b1
Type this:
mkdir -p /tmp/efk93 && printf '%s\n' 'lab notes' > /tmp/efk93/NOTES.txt && cat /tmp/efk93/NOTES.txt
You will see:
lab notes
What it means: Notes.
Example 5: b2
Type this:
echo 'Work only under /tmp/efk93 for this project'
You will see:
Work only under /tmp/efk93 for this project
What it means: Path.
Example 6: b3
Type this:
echo 'Write ACCEPT.txt at the end'
You will see:
Write ACCEPT.txt at the end
What it means: Accept file.
Step 3: Elasticsearch
Why: Single node lab. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: es. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 7: c1
Type this:
echo 'docker run -d --name es -p 9200:9200 -e discovery.type=single-node -e xpack.security.enabled=false docker.elastic.co/elasticsearch/elasticsearch:8.12.0'
You will see:
docker run -d --name es -p 9200:9200 -e discovery.type=single-node -e xpack.security.enabled=false docker.elastic.co/elasticsearch/elasticsearch:8.12.0
What it means: ES.
Example 8: c2
Type this:
echo 'curl -s http://localhost:9200 | head'
You will see:
curl -s http://localhost:9200 | head
What it means: Health.
Example 9: c3
Type this:
echo 'Lab only: security disabled; enable it in real clusters'
You will see:
Lab only: security disabled; enable it in real clusters
What it means: Sec.
3. Build the main pieces
Step 4: Fluent Bit
Why: Ship stdout. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: fb. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 10: d1
Type this:
printf '%s\n' '[INPUT]' ' Name tail' ' Path /var/log/containers/*.log' '[OUTPUT]' ' Name es' ' Host es' ' Port 9200' ' Index shop-logs' > /tmp/efk93/fluent-bit.conf && cat /tmp/efk93/fluent-bit.conf
You will see:
Index shop-logs
What it means: FB.
Example 11: d2
Type this:
echo 'Run Fluent Bit as a DaemonSet in Kubernetes'
You will see:
Run Fluent Bit as a DaemonSet in Kubernetes
What it means: DS.
Example 12: d3
Type this:
echo 'Parse JSON logs so fields become searchable'
You will see:
Parse JSON logs so fields become searchable
What it means: JSON.
Step 5: Kibana
Why: Discover. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: kibana. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 13: e1
Type this:
echo 'docker run -d --name kibana -p 5601:5601 -e ELASTICSEARCH_HOSTS=http://es:9200 docker.elastic.co/kibana/kibana:8.12.0'
You will see:
docker run -d --name kibana -p 5601:5601 -e ELASTICSEARCH_HOSTS=http://es:9200 docker.elastic.co/kibana/kibana:8.12.0
What it means: Kibana.
Example 14: e2
Type this:
echo 'Create a data view on shop-logs*'
You will see:
Create a data view on shop-logs*
What it means: View.
Example 15: e3
Type this:
printf '%s\n' 'Acceptance: search request.id:demo-123 returns the error line' > /tmp/efk93/ACCEPT.txt && cat /tmp/efk93/ACCEPT.txt
You will see:
Acceptance: search request.id:demo-123 returns the error line
What it means: Gate.
Step 6: Structured logs
Why: JSON fields. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: json. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 16: f1
Type this:
printf '%s\n' 'Log JSON: level, msg, request_id, service' 'Avoid free-text only lines in new services' > /tmp/efk93/JSON.txt && cat /tmp/efk93/JSON.txt
You will see:
Log JSON: level, msg, request_id, service
What it means: JSON.
Example 17: f2
Type this:
echo 'Propagate request_id across service calls'
You will see:
Propagate request_id across service calls
What it means: Propagate.
Example 18: f3
Type this:
echo 'Redact tokens and passwords before shipping'
You will see:
Redact tokens and passwords before shipping
What it means: Redact.
4. Automate and verify
Step 7: ILM retention
Why: Disk budget. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: ilm. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 19: g1
Type this:
printf '%s\n' 'Hot 3 days, warm 7 days, delete after 14 days for app logs' 'Keep audit logs longer if compliance needs it' > /tmp/efk93/ILM.txt && cat /tmp/efk93/ILM.txt
You will see:
Hot 3 days, warm 7 days, delete after 14 days for app logs
What it means: ILM.
Example 20: g2
Type this:
echo 'Without ILM, disks fill and searches die'
You will see:
Without ILM, disks fill and searches die
What it means: Risk.
Example 21: g3
Type this:
echo 'Alert when disk used exceeds 75 percent'
You will see:
Alert when disk used exceeds 75 percent
What it means: Alert.
Step 8: Index pattern
Why: Naming. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: index. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 22: h1
Type this:
echo 'Name indexes shop-logs-YYYY.MM.DD or use data streams'
You will see:
Name indexes shop-logs-YYYY.MM.DD or use data streams
What it means: Name.
Example 23: h2
Type this:
echo 'One data stream per service tier works well'
You will see:
One data stream per service tier works well
What it means: Stream.
Example 24: h3
Type this:
echo 'Avoid one giant index for all apps forever'
You will see:
Avoid one giant index for all apps forever
What it means: Avoid.
Step 9: Access control
Why: Who can see. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: acl. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 25: i1
Type this:
printf '%s\n' 'Prod logs may contain PII; restrict Kibana spaces' 'Devs get their service indexes only' > /tmp/efk93/ACL.txt && cat /tmp/efk93/ACL.txt
You will see:
Prod logs may contain PII; restrict Kibana spaces
What it means: ACL.
Example 26: i2
Type this:
echo 'Enable security features outside the lab'
You will see:
Enable security features outside the lab
What it means: Sec.
Example 27: i3
Type this:
echo 'Audit who ran sensitive queries'
You will see:
Audit who ran sensitive queries
What it means: Audit.
5. Harden and accept
Step 10: Story Ola
Why: Bengaluru lesson. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: story. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 28: j1
Type this:
printf '%s\n' 'At Ola in Bengaluru, SSH across twenty nodes still missed the error.' 'Priya centralized logs with EFK and request-id.' > /tmp/efk93/STORY.txt && wc -l /tmp/efk93/STORY.txt
You will see:
2 /tmp/efk93/STORY.txt
What it means: Story.
Example 29: j2
Type this:
echo 'Rough cost: about Rs 25 lakh in prolonged downtime.'
You will see:
Rough cost: about Rs 25 lakh in prolonged downtime.
What it means: Cost.
Example 30: j3
Type this:
echo 'Priya: PASS'
You will see:
Priya: PASS
What it means: Sign-off.
Step 11: Proof
Why: Files. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: ls. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 31: k1
Type this:
ls /tmp/efk93
You will see:
NOTES.txt\nSTORY.txt\nACCEPT.txt
What it means: Tree.
Example 32: k2
Type this:
test -f /tmp/efk93/ACCEPT.txt && echo ACCEPT_OK
You will see:
ACCEPT_OK
What it means: Accept present.
Example 33: k3
Type this:
echo 'Next: Loki Kubernetes logs project 94'
You will see:
Next: Loki Kubernetes logs project 94
What it means: Bridge.
Step 12: Acceptance
Why: Central log search works. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: final. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 34: z1
Type this:
echo 'ES green'
You will see:
ES green
What it means: A.
Example 35: z2
Type this:
echo 'Fluent shipping'
You will see:
Fluent shipping
What it means: B.
Example 36: z3
Type this:
echo 'Kibana hit'
You will see:
Kibana hit
What it means: C.
Check yourself
Q1. Who fixed Ola SSH hunt?
Priya
Q2. Shipper?
Fluent Bit
Q3. Store?
Elasticsearch
Q4. UI?
Kibana
Q5. Forever logs?
no
Q6. Key field?
request id
Friends, you searched central logs. Did you understand? Let us try it once more. Drop a unique request id and prove Kibana finds it in under a minute.
मित्रांनो, तुम्ही central logs search केले. Unique request id टाका आणि Kibana एका मिनिटात शोधते हे सिद्ध करा.
दोस्तों, आपने central logs search किए. Unique request id डालें और Kibana एक मिनट में खोजे, यह साबित करें.
Lab: try at home
- Elasticsearch up.
- Fluent Bit shipping.
- Kibana search.
- Request id found.
Recap
- Centralize logs.
- Structured JSON.
- Request ids.
- ILM retention.
- Search beats SSH.