Ravindra BagaleCourses & study guides मराठी Track your progress

21. Web Application Testing Tools

21.5 Gobuster and Dirb: Finding Hidden Content

By Last updated:

Web servers var khup pages links madhun disat nahit – /admin, /backup, /config.php, old files. Directory brute-forcing tools ek wordlist gheun pratyek naav try kartat aani konte exist kartat te sangtat (HTTP status codes: 200 found, 301/302 redirect, 403 forbidden-but-there, 404 not found).

# Gobuster (fast, written in Go)
gobuster dir -u http://192.168.56.20 -w /usr/share/wordlists/dirb/common.txt
gobuster dir -u http://192.168.56.20/dvwa -w /usr/share/wordlists/dirbuster/directory-list-2.3-medium.txt -x php,txt,bak
gobuster dns -d lab.local -r 192.168.56.20 -w /usr/share/wordlists/subdomains-top1million-5000.txt   # subdomains

# Dirb (classic, simple)
dirb http://192.168.56.20 /usr/share/wordlists/dirb/common.txt

Useful flags: -x php,txt,bak adds file extensions, -t 50 sets threads, -s 200,301,403 filters status codes.

Ravindra Bagale's Tip

Ignore all the 404s, but never ignore a 403 – "forbidden" means the page exists, you just don't have access. Those pages often turn out to be interesting later. And always start with a small wordlist (common.txt), then move to a bigger one.

Lab

Run Gobuster against http://192.168.56.20 with common.txt and -x php,txt,bak. List every path that returns 200, 301 or 403. On your own reels app, run the same scan and check whether any backup file (.bak) or config is exposed – if so, remove it.

What directory discovery teaches defenders

If Gobuster finds /backup.zip or /.env on your server, attackers will find them too. The lesson is clean deployments.

  1. Do not leave backup files inside the web root.
  2. Block dotfiles in nginx/Apache.
  3. Review robots.txt and remove sensitive paths from the server entirely (not just from robots).
  4. Store backups in private object storage (S3) instead.

Real examples

Global: countless breaches began with an exposed .git directory or a leftover admin panel. Indian company (sample): a university result portal left /phpinfo.php online. Gobuster found it in seconds during an authorised test. Removing the file was the whole fix.

Recap

  • Discovery tools find leftovers. Clean builds prevent them.
  • Own servers only.

Defender's worksheet (do this today)

  1. Write the lesson title and date in your notebook.
  2. List three systems you own (VMs, personal projects, or lab apps) where this lesson applies.
  3. Run one safe command or checklist item from the lesson on a lab system only.
  4. Paste the command and its output under an Evidence heading.
  5. Write one fix you would give a teammate in two sentences.
  6. Write one mistake students make with this topic and how to avoid it.

More real-world pictures

Indian workplace (sample data)

Suppose we are a junior analyst at Tech Mahindra supporting a client SOC. After reading this lesson you add one detection or one hardening item to the team weekly checklist, with the owner name and a due date. That is how learning becomes operations.

Global practice

Public bodies such as CERT-In (India), CISA (US) and NCSC (UK) publish advisories that map to the same ideas: patch, least privilege, logging and user awareness. Bookmark one advisory feed and read it every Monday.

Home and family

Explain this lesson to a family member in four sentences without jargon. If they can repeat the main warning back to you, you understand it well enough to teach.

Connect to earlier chapters

Parts 1 to 9 built Linux, web servers, databases and cloud pieces. This lesson only makes sense when you can point to a concrete config file, package name or architecture box from those parts. If you cannot, revisit the matching earlier chapter before you practise any tool command.

Common mistakes

  • Skipping written permission just for a quick scan.
  • Practising on hostel or office Wi-Fi targets without authorisation.
  • Collecting tool output without writing the fix.
  • Copying exploit commands onto production to verify. Verification for defenders means checking versions and patches, not firing modules.

Recap

  • Evidence, fix, owner, date.
  • Own lab only.
  • Teach one other person the warning from this lesson.