Chapter 15: Amazon VPC — Subnets, Route Tables, NACL, NAT Gateway and Bastion Host
15.12 Project: test every hop, troubleshoot and clean up
Let's see the result! Open the web server's public IP in the browser – if you see the students list, then the three tiers, three subnets, route tables, NAT and the security group chain are all correct. If not, don't panic, we will test hop by hop. So the very first job – the browser.
आता result बघूया! Browser मध्ये web server चा public IP उघडा – students list दिसली तर तीन tiers, तीन subnets, route tables, NAT आणि security group chain सगळं बरोबर आहे. नाही दिसली तर घाबरू नका, एक एक hop test करू. तर सगळ्यात पहिलं काम – browser.
अब result देखते हैं! Browser में web server का public IP खोलो – students list दिखी तो तीन tiers, तीन subnets, route tables, NAT और security group chain सब सही है. नहीं दिखी तो घबराओ मत, एक एक hop test करेंगे. तो सबसे पहला काम – browser.
The browser reaches Nginx in the public subnet on port 80. Nginx passes the PHP request with fastcgi_pass to PHP-FPM on port 9000, PHP reads the students from MySQL on port 3306, and the HTML page goes back. Each security group allows only the tier before it.
Browser public subnet मधल्या Nginx पर्यंत port 80 वर पोहोचतो. Nginx PHP request fastcgi_pass ने port 9000 वरच्या PHP-FPM कडे पाठवतो, PHP port 3306 वरच्या MySQL मधून students वाचतो, आणि HTML page परत जातं. प्रत्येक security group फक्त त्याच्या आधीच्या tier ला allow करतो.
Browser public subnet में Nginx तक port 80 पर पहुँचता है. Nginx PHP request को fastcgi_pass से port 9000 वाले PHP-FPM को भेजता है, PHP port 3306 वाले MySQL से students पढ़ता है, और HTML page वापस जाता है. हर security group सिर्फ अपने पहले वाले tier को allow करता है.
Open http://<WEB_PUBLIC_IP>/ in your browser. You should see the Students list table with three rows and the line "PHP ran on ip-10-0-2-10…". That single page proves the full path: internet → IGW → public subnet → Nginx → (port 9000) → PHP-FPM in the private app subnet → (port 3306) → MariaDB in the private DB subnet.
Verify each hop on its own
When something fails, test hop by hop from the server before it. Bash can test a TCP port with /dev/tcp, so no extra tools are needed:
| Hop | Run on | Command | Good result |
|---|---|---|---|
| You → web (HTTP) | Your laptop | curl -I http://<WEB_PUBLIC_IP>/ |
HTTP/1.1 200 OK, Server: nginx |
| You → bastion (SSH) | Your laptop | ssh ec2-user@<BASTION_PUBLIC_IP> |
Shell prompt |
| Web → app (9000) | web | timeout 3 bash -c 'echo > /dev/tcp/10.0.2.10/9000' && echo OPEN |
OPEN |
| App → db (3306) | app | timeout 3 bash -c 'echo > /dev/tcp/10.0.3.10/3306' && echo OPEN |
OPEN |
| App → MySQL login | app | mysql -h 10.0.3.10 -u app -p -e "SELECT COUNT(*) FROM appdb.students;" |
3 |
| Private → internet (NAT) | app or db | curl -s https://checkip.amazonaws.com |
The NAT gateway's Elastic IP |
| Nginx → PHP locally | web | curl -s localhost \| grep -c "<tr>" |
4 (header + 3 students) |
Prove the blocks work too
A secure design is not only "what works" but also "what is refused". These must fail (time out):
| Test | Run on | Expected | Why |
|---|---|---|---|
timeout 3 bash -c 'echo > /dev/tcp/10.0.3.10/3306' |
web | times out | db-sg allows 3306 only from app-sg |
ssh ec2-user@10.0.2.10 directly from your laptop |
laptop | no route | Private IP, and no public IP on app |
curl http://<APP_PRIVATE_IP>:9000 from the internet |
laptop | impossible | Private subnet has no route from the IGW |
Add NACL deny rule 90 for 198.51.100.7/32 on web-public-1a |
console | your browser stops loading | NACL denies before the allow rule 100 (remove it afterwards!) |
Troubleshooting table
| Symptom | Where to look | Cause and fix |
|---|---|---|
502 Bad Gateway + connect() failed (111: Connection refused) in /var/log/nginx/error.log |
app | PHP-FPM not running or still on the socket. Check grep ^listen /etc/php-fpm.d/www.conf, sudo service php-fpm restart |
502 / 504 + (110: Connection timed out) |
Security group | app-sg has no rule for TCP 9000 from web-sg |
502 + (104: Connection reset by peer) |
app | listen.allowed_clients does not contain 10.0.1.10 |
File not found. and Primary script unknown in the log |
app | index.php is not at /var/www/html/ on the app server, or SCRIPT_FILENAME path is wrong |
| Nginx welcome page instead of the students list | web | Missing placeholder /usr/share/nginx/html/index.php or php-app.conf not in /etc/nginx/default.d/. Run sudo nginx -t and reload |
yum install hangs, Curl error (28): Timeout was reached |
Private route table | No 0.0.0.0/0 → nat-… route, or NAT gateway not Available |
SQLSTATE[HY000] [2002] Connection timed out |
db-sg | No 3306 rule from app-sg |
SQLSTATE[HY000] [2002] Connection refused |
db | MariaDB still bound to 127.0.0.1: check bind-address and sudo service mariadb restart |
SQLSTATE[HY000] [1045] Access denied for user 'app'@'10.0.2.10' |
db | Wrong password or user created for another host; recreate 'app'@'10.0.2.%' |
ssh -J: Permission denied (publickey) on the private server |
Laptop | Key not in the agent: run ssh-add society-key.pem and retry |
Clean up (in this order)
The NAT gateway and the Elastic IP are billed every hour, even when you are sleeping. Delete everything once you have finished:
- EC2 → Instances: terminate
web,bastion,appanddb. - VPC → NAT gateways: select
society-nat→ Actions → Delete NAT gateway. Wait until the state is Deleted. - EC2 → Elastic IPs: select the NAT's address → Actions → Release Elastic IP addresses. Deleting the NAT gateway does not release it.
- VPC → Your VPCs: select
society-vpc→ Actions → Delete VPC. This also deletes its subnets, route tables, internet gateway, network ACLs and security groups. - Next day, open Billing and Cost Management and confirm that no NAT gateway or public IPv4 charges are still growing.
Ravindra Bagale's Tip
Clean up right after the project – that is the rule. My students' biggest bills come from the NAT gateway and a forgotten Elastic IP. Note: even after you delete the NAT, the Elastic IP has to be released. And while troubleshooting, test hop by hop – from web to app, from app to db. No guessing; get proof with /dev/tcp.
Ravindra Bagale's Tip – मराठी
Project झाला की लगेच cleanup – हा नियम. माझ्या students चं सगळ्यात जास्त bill NAT gateway आणि विसरलेल्या Elastic IP मुळे येतं. NAT delete केला तरी Elastic IP release करावा लागतो, लक्ष द्या. आणि troubleshooting करताना एक एक hop test करा – web वरून app, app वरून db. अंदाज नको, /dev/tcp ने proof घ्या.
Ravindra Bagale's Tip – हिंदी
Project होते ही तुरंत cleanup – यह नियम है. मेरे students का सबसे ज़्यादा bill NAT gateway और भूले हुए Elastic IP की वजह से आता है. NAT delete करने के बाद भी Elastic IP release करना पड़ता है, ध्यान दो. और troubleshooting करते समय एक एक hop test करो – web से app, app से db. अंदाज़ा नहीं, /dev/tcp से proof लो.
Try it at home
Build the whole project again from an empty account, this time without looking at the notes for the security group chain. Then add a second page add.php on the app server with a small form that inserts a student (the app user already has INSERT). Finally, block your own IP with NACL rule 90 for five minutes, watch the site stop, remove the rule, and clean up everything.