Chapter 46: Terraform 3-Tier Architecture Multi-AZ with ALBs Auto Scaling Bastion NAT and RDS

46.6 Internal ALB and app tier Auto Scaling group

By Ravindra Bagale · Last updated: 25 October 2026

Story. Suppose we are separating the shop's business logic from the web pages, the way large retail sites do. App servers sit in private app subnets in both zones behind an internal ALB that has no public address. Web servers call the internal ALB, which spreads requests across healthy app servers. The app tier scales on its own, so a heavy checkout load does not need more web servers.

Steps

1. Write app.tf and userdata/app.sh.
2. Run terraform validate.
3. After apply, curl /api/ several times and see answers from both zones.

# ---- app.tf ----
resource "aws_lb" "internal" {
  name               = "${var.project}-int-alb"
  load_balancer_type = "application"
  internal           = true
  security_groups    = [aws_security_group.int_alb.id]
  subnets            = [for s in aws_subnet.app : s.id]
}

resource "aws_lb_target_group" "app" {
  name     = "${var.project}-app-tg"
  port     = 8080
  protocol = "HTTP"
  vpc_id   = aws_vpc.main.id
  health_check {
    path    = "/"
    port    = "8080"
    matcher = "200"
  }
}

resource "aws_lb_listener" "internal_http" {
  load_balancer_arn = aws_lb.internal.arn
  port              = 80
  protocol          = "HTTP"
  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.app.arn
  }
}

resource "aws_launch_template" "app" {
  name_prefix            = "${var.project}-app-"
  image_id               = data.aws_ami.al2023.id
  instance_type          = var.instance_type
  key_name               = aws_key_pair.ops.key_name
  vpc_security_group_ids = [aws_security_group.app.id]
  metadata_options {
    http_tokens = "required"
  }
  user_data = filebase64("${path.module}/userdata/app.sh")
}

resource "aws_autoscaling_group" "app" {
  name                      = "${var.project}-app-asg"
  min_size                  = 2
  desired_capacity          = 2
  max_size                  = 6
  vpc_zone_identifier       = [for s in aws_subnet.app : s.id]
  target_group_arns         = [aws_lb_target_group.app.arn]
  health_check_type         = "ELB"
  health_check_grace_period = 120
  launch_template {
    id      = aws_launch_template.app.id
    version = aws_launch_template.app.latest_version
  }
  instance_refresh {
    strategy = "Rolling"
  }
  tag {
    key                 = "Name"
    value               = "${var.project}-app"
    propagate_at_launch = true
  }
}

resource "aws_autoscaling_policy" "app_cpu" {
  name                   = "app-cpu-60"
  autoscaling_group_name = aws_autoscaling_group.app.name
  policy_type            = "TargetTrackingScaling"
  target_tracking_configuration {
    predefined_metric_specification {
      predefined_metric_type = "ASGAverageCPUUtilization"
    }
    target_value = 60
  }
}

# ---- userdata/app.sh ----
#!/bin/bash
mkdir -p /opt/app
echo "app tier: $(hostname)" > /opt/app/index.html
printf '%s\n' '[Unit]' 'Description=demo app' '[Service]' \
  'ExecStart=/usr/bin/python3 -m http.server 8080 --directory /opt/app' \
  'Restart=always' '[Install]' 'WantedBy=multi-user.target' \
  > /etc/systemd/system/app.service
systemctl daemon-reload
systemctl enable --now app

What each line does

• internal = true — No public IP; only reachable inside the VPC.
• subnets = [for s in aws_subnet.app : s.id] — The internal ALB lives in the private app subnets.
• port = 8080 (target group) — App servers listen on 8080.
• health_check { port = "8080" } — Health checks hit the app port.
• aws_lb_listener "internal_http" port = 80 — Web calls the internal ALB on 80; it forwards to 8080.
• user_data = filebase64(".../app.sh") — The app boot script needs no variables, so filebase64 is enough.
• max_size = 6 — The app tier can grow more than the web tier.
• health_check_type = "ELB" — Unhealthy app servers are replaced.
• app.sh: systemd unit — Runs the demo app on 8080 and restarts it if it stops. Replace it with your real app.

When it is green.

$ curl -s $(terraform output -raw app_url)/api/
app tier: ip-10-40-22-81.ap-south-1.compute.internal
$ curl -s $(terraform output -raw app_url)/api/
app tier: ip-10-40-21-37.ap-south-1.compute.internal

When it is red.

$ curl -s $(terraform output -raw app_url)/api/
<html><head><title>502 Bad Gateway</title></head>

Fix. This 502 comes from nginx on the web tier: it cannot reach the internal ALB, or the ALB has no healthy app targets. Check int_alb_from_web and app_from_int_alb, then check app target health.

What: every term in plain English

• internal ALB — A load balancer with internal = true. Its DNS name resolves only to private IPs.
• app tier — The business logic servers, listening on 8080 here.
• filebase64() — Reads a file and returns base64, ready for a launch template user_data.
• systemd unit — A service file that starts the app at boot and restarts it on crash.
• port mapping — The internal ALB listens on 80 and forwards to targets on 8080.
• decoupling — Web servers call one stable DNS name; app servers can scale or be replaced behind it.
• connection draining — deregistration_delay: how long the ALB waits for in-flight requests before removing a target.
• scale in protection — Stops the ASG from terminating a server that is still doing work.

Example 1: Create the internal ALB in the app subnets

At PhonePe, internal services talk through internal load balancers so a caller never needs the IP of a single server.

1. Set internal = true.
2. Use both app subnets.
3. Give it the int_alb security group.

resource "aws_lb" "internal" {
  name               = "${var.project}-int-alb"
  load_balancer_type = "application"
  internal           = true
  security_groups    = [aws_security_group.int_alb.id]
  subnets            = [for s in aws_subnet.app : s.id]
}

dig +short $(terraform output -raw internal_alb_dns)

Output

10.40.21.88
10.40.22.143

What it shows: Both answers are private 10.40.2x addresses, one per app subnet. From the internet this name is useless.

Example 2: Write the app boot script with printf

Course rule: no heredocs. printf with one quoted argument per line writes the systemd unit cleanly and passes shellcheck.

1. Create userdata/app.sh.
2. Write the unit file with printf.
3. Load it with filebase64.

#!/bin/bash
# userdata/app.sh
mkdir -p /opt/app
echo "app tier: $(hostname)" > /opt/app/index.html
printf '%s\n' '[Unit]' 'Description=demo app' '[Service]' \
  'ExecStart=/usr/bin/python3 -m http.server 8080 --directory /opt/app' \
  'Restart=always' '[Install]' 'WantedBy=multi-user.target' \
  > /etc/systemd/system/app.service
systemctl daemon-reload
systemctl enable --now app

resource "aws_launch_template" "app" {
  name_prefix            = "${var.project}-app-"
  image_id               = data.aws_ami.al2023.id
  instance_type          = var.instance_type
  vpc_security_group_ids = [aws_security_group.app.id]
  user_data              = filebase64("${path.module}/userdata/app.sh")
}

Output

$ shellcheck userdata/app.sh && echo clean
clean

What it shows: filebase64 is used because the script has no variables. Use templatefile when you need to inject values.

Example 3: Target group on 8080 with a health check

The internal ALB listens on 80 but the app listens on 8080. The target group port and the health check port must both be 8080.

1. Create the target group on 8080.
2. Health check path / on 8080.
3. Set a short deregistration delay for labs.

resource "aws_lb_target_group" "app" {
  name                 = "${var.project}-app-tg"
  port                 = 8080
  protocol             = "HTTP"
  vpc_id               = aws_vpc.main.id
  deregistration_delay = 30
  health_check {
    path     = "/"
    port     = "8080"
    matcher  = "200"
    interval = 15
  }
}

Output

aws_lb_target_group.app: Creation complete after 1s [id=arn:aws:elasticloadbalancing:ap-south-1:111122223333:targetgroup/shop3t-app-tg/def456]

What it shows: deregistration_delay 30 makes refreshes faster in a lab. Production often keeps the 300 second default.

Example 4: Test the full path web to app

The real proof is a request that enters the public ALB, passes nginx on a web server and lands on an app server.

1. Curl /api/ on the public URL.
2. Repeat to see different app hosts.
3. Compare with web hostnames.

URL=$(terraform output -raw app_url)
for i in 1 2 3 4; do curl -s $URL/api/; done

Output

app tier: ip-10-40-21-34.ap-south-1.compute.internal
app tier: ip-10-40-22-91.ap-south-1.compute.internal
app tier: ip-10-40-21-34.ap-south-1.compute.internal
app tier: ip-10-40-22-91.ap-south-1.compute.internal

What it shows: Answers alternate between 10.40.21.x (zone a) and 10.40.22.x (zone b). Both tiers are balanced.

Example 5: Survive the loss of an app server

Self-healing is the reason for ASGs. Ola engineers test it by terminating a server in office hours and watching traffic carry on.

1. Terminate one app instance.
2. Keep curling /api/.
3. Watch the ASG launch a replacement.

ID=$(aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names shop3t-app-asg \
  --query "AutoScalingGroups[0].Instances[0].InstanceId" --output text)
aws ec2 terminate-instances --instance-ids $ID --query "TerminatingInstances[0].CurrentState.Name"
sleep 180
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names shop3t-app-asg \
  --query "length(AutoScalingGroups[0].Instances)"

Output

"shutting-down"
2

What it shows: During those three minutes every curl still answered, from the server in the other zone.

Example 6: Scale the app tier separately

At Paytm, payment logic needs more servers than the web front during a cashback offer. Separate ASGs let each tier grow on its own.

1. Change app desired_capacity with a variable.
2. Plan and apply.
3. Confirm the web ASG did not change.

terraform apply -var app_desired=4 -auto-approve | grep -E "aws_autoscaling_group|Apply"

Output

aws_autoscaling_group.app: Modifying... [id=shop3t-app-asg]
aws_autoscaling_group.app: Modifications complete after 2s [id=shop3t-app-asg]
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

What it shows: Only the app ASG changed. In production let the policy scale and use ignore_changes on desired_capacity.

Command cheat sheet

Verify: did it work?

1. Internal name is private

   dig +short $(terraform output -raw internal_alb_dns)
   

   Output

   10.40.21.88 / 10.40.22.143
   

2. API answers

   curl -s $(terraform output -raw app_url)/api/
   

   Output

   app tier: ip-10-40-2x...
   

3. Two app servers

   aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names shop3t-app-asg --query "length(AutoScalingGroups[0].Instances)"
   

   Output

   2
   

When it is red 1

502 Bad Gateway

Cause. nginx on web cannot reach the internal ALB.

Fix, step by step

1. Check int_alb ingress from the web group on 80.
2. Check app_lb_dns in the rendered nginx file.

When it is red 2

Target.FailedHealthChecks: Health checks failed

Cause. The app did not start or the app group misses 8080 from int_alb.

Fix, step by step

1. Read /var/log/cloud-init-output.log.
2. Check app_from_int_alb rule.

When it is red 3

Error: Cycle: aws_launch_template.web, aws_lb.internal

Cause. A reference loop between resources.

Fix, step by step

1. Make sure aws_lb.internal does not refer to web resources.
2. Run terraform graph to find the loop.

Cleanup

terraform destroy -auto-approve

Practice tasks

Homework: try at home

Recap

• Internal ALB in app subnets, private IPs only.
• Boot script in userdata/app.sh loaded with filebase64.
• Listener 80, targets 8080.
• Web reaches app only through the internal ALB.
• Each tier scales and heals on its own.

FAQ

Why not call app servers by IP?

IPs change whenever the ASG replaces a server.

Can the internal ALB use HTTPS?

Yes, with a private certificate from ACM PCA.

Why filebase64 here and templatefile for web?

The app script has no variables.

Does the app tier need the bastion?

Only for debugging; Session Manager also works.

Interview questions

1. What is an internal load balancer?

   A load balancer with only private IPs, used for traffic inside the VPC.

2. Why use separate ASGs for web and app?

   Each tier has different load and can scale and deploy independently.

3. How does the web tier find the app tier?

   Through the internal ALB DNS name passed with templatefile.

4. What is deregistration delay?

   Time the ALB keeps sending in-flight requests to a target before removing it.

Samajla ka?

Ravi Sir's Cards for this lesson

Revision cardInterview card

मराठी cards