AWS SDK Error Handling, Retries, Pagination and Waiters (Python, PHP, Node.js)

By Ravindra Bagale, trainer · LinkedIn · Last updated: 24 October 2026

1. The big picture: a courier delivery

Why do we need this?

Carlos runs a bakery in Madrid and sends cakes by courier. If nobody answers the door, the courier tries again later (retry). If the address does not exist, trying again is pointless (a real error). A big order goes in several vans (pages). And he waits until the parcel shows "delivered" before he calls the customer (waiter).

What is it?

Every SDK raises a typed error with an error code (NoSuchBucket, AccessDenied, ThrottlingException) and an HTTP status. Throttling and 5xx errors are retried automatically. 4xx errors like AccessDenied are not.

How to do it (step by step)

2. Catch errors by code

Why do we need this?

A user uploads to a bucket name with a typo. You want a clear message, not a stack trace.

What is it?

boto3 raises botocore.exceptions.ClientError; PHP throws Aws\Exception\AwsException; Node throws an error whose name is the code and $metadata.httpStatusCode is the status.

How to do it (step by step)

1. Python: catch ClientError and read the code.

import boto3
from botocore.exceptions import ClientError

s3 = boto3.client("s3", region_name="eu-west-1")
try:
    s3.head_bucket(Bucket="carlos-bakery-no-such-bucket")
except ClientError as e:
    code = e.response["Error"]["Code"]
    status = e.response["ResponseMetadata"]["HTTPStatusCode"]
    print("Failed:", code, status)

Expected output:

Failed: 404 404

2. Python: handle specific codes differently.

import boto3
from botocore.exceptions import ClientError

ddb = boto3.client("dynamodb", region_name="eu-west-1")
try:
    ddb.describe_table(TableName="Orders-Missing")
except ClientError as e:
    if e.response["Error"]["Code"] == "ResourceNotFoundException":
        print("Table not found, create it first")
    else:
        raise

Expected output:

Table not found, create it first

3. PHP: catch AwsException.

<?php
require "vendor/autoload.php";
use Aws\DynamoDb\DynamoDbClient;
use Aws\Exception\AwsException;

$ddb = new DynamoDbClient(["region" => "eu-west-1"]);
try {
    $ddb->describeTable(["TableName" => "Orders-Missing"]);
} catch (AwsException $e) {
    echo "Failed: ", $e->getAwsErrorCode(), " ", $e->getStatusCode(), PHP_EOL;
}

Expected output:

Failed: ResourceNotFoundException 400

4. Node.js: check error.name and the HTTP status.

import { DynamoDBClient, DescribeTableCommand } from "@aws-sdk/client-dynamodb";

const ddb = new DynamoDBClient({ region: "eu-west-1" });
try {
  await ddb.send(new DescribeTableCommand({ TableName: "Orders-Missing" }));
} catch (err) {
  console.log("Failed:", err.name, err.$metadata?.httpStatusCode);
}

Expected output:

Failed: ResourceNotFoundException 400

5. Verify: the CLI shows the same code.

aws dynamodb describe-table --table-name Orders-Missing --region eu-west-1 2>&1 | head -2

Expected output:

An error occurred (ResourceNotFoundException) when calling the DescribeTable operation: Requested resource not found

3. Retries and timeouts

Why do we need this?

A flash sale sends thousands of writes per second and some get throttled. A slow network should not hang your web request for a minute.

What is it?

Retry modes: legacy, standard (default in newer SDKs) and adaptive (adds client-side rate limiting). Set max attempts and connect/read timeouts on the client.

How to do it (step by step)

1. Python: pass a botocore Config.

import boto3
from botocore.config import Config

cfg = Config(
    retries={"max_attempts": 5, "mode": "adaptive"},
    connect_timeout=3,
    read_timeout=10,
)
s3 = boto3.client("s3", region_name="us-east-1", config=cfg)
print(s3.meta.config.retries)

Expected output:

{'max_attempts': 5, 'mode': 'adaptive'}

2. PHP: set retries and http timeouts.

<?php
require "vendor/autoload.php";
use Aws\S3\S3Client;

$s3 = new S3Client([
    "region"  => "us-east-1",
    "retries" => ["mode" => "standard", "max_attempts" => 5],
    "http"    => ["connect_timeout" => 3, "timeout" => 10],
]);
echo "Client ready with 5 attempts", PHP_EOL;

Expected output:

Client ready with 5 attempts

3. Node.js: maxAttempts, retryMode and a request handler with timeouts.

import { S3Client } from "@aws-sdk/client-s3";
import { NodeHttpHandler } from "@smithy/node-http-handler";

const s3 = new S3Client({
  region: "us-east-1",
  maxAttempts: 5,
  retryMode: "adaptive",
  requestHandler: new NodeHttpHandler({ connectionTimeout: 3000, requestTimeout: 10000 }),
});
console.log("maxAttempts:", await s3.config.maxAttempts());

Expected output:

maxAttempts: 5

4. Or set retries for all SDKs with env vars.

export AWS_RETRY_MODE=standard
export AWS_MAX_ATTEMPTS=5
python -c "import boto3;print(boto3.client('s3').meta.config.retries)"

Expected output:

{'mode': 'standard', 'total_max_attempts': 5}

5. Verify: count how many retries happened on a response.

import boto3
r = boto3.client("s3").list_buckets()
print("Retries:", r["ResponseMetadata"]["RetryAttempts"])

Expected output:

Retries: 0

4. Pagination

Why do we need this?

A London agency has 2,500 objects in a bucket, but one ListObjectsV2 call returns at most 1,000. Without pagination you silently miss data.

What is it?

Each SDK has paginators that follow NextToken / ContinuationToken for you: boto3 get_paginator, PHP getPaginator, Node paginateListObjectsV2.

How to do it (step by step)

1. Python: count every object with a paginator.

import boto3

s3 = boto3.client("s3")
pages = s3.get_paginator("list_objects_v2").paginate(Bucket="london-agency-assets")
total = 0
for page in pages:
    total += page.get("KeyCount", 0)
print("Objects:", total)

Expected output:

Objects: 2500

2. PHP: getPaginator.

<?php
require "vendor/autoload.php";
$s3 = new Aws\S3\S3Client(["region" => "eu-west-2"]);
$total = 0;
foreach ($s3->getPaginator("ListObjectsV2", ["Bucket" => "london-agency-assets"]) as $page) {
    $total += $page["KeyCount"];
}
echo "Objects: $total", PHP_EOL;

Expected output:

Objects: 2500

3. Node.js: for await over paginateListObjectsV2.

import { S3Client, paginateListObjectsV2 } from "@aws-sdk/client-s3";

const s3 = new S3Client({ region: "eu-west-2" });
let total = 0;
for await (const page of paginateListObjectsV2({ client: s3 }, { Bucket: "london-agency-assets" })) {
  total += page.KeyCount ?? 0;
}
console.log("Objects:", total);

Expected output:

Objects: 2500

4. Verify: the CLI count matches.

aws s3api list-objects-v2 --bucket london-agency-assets --query "length(Contents)"

Expected output:

2500

5. Waiters

Why do we need this?

After creating a DynamoDB table or launching EC2 you must wait until it is ready before using it.

What is it?

A waiter polls a describe call until a success state, with a delay and max attempts, and fails cleanly if it never gets there.

How to do it (step by step)

1. Python: wait for a table to exist.

import boto3

ddb = boto3.client("dynamodb", region_name="eu-west-1")
waiter = ddb.get_waiter("table_exists")
waiter.wait(TableName="Orders", WaiterConfig={"Delay": 5, "MaxAttempts": 24})
print("Orders table is ACTIVE")

Expected output:

Orders table is ACTIVE

2. PHP: waitUntil.

<?php
require "vendor/autoload.php";
$ddb = new Aws\DynamoDb\DynamoDbClient(["region" => "eu-west-1"]);
$ddb->waitUntil("TableExists", ["TableName" => "Orders", "@waiter" => ["delay" => 5, "maxAttempts" => 24]]);
echo "Orders table is ACTIVE", PHP_EOL;

Expected output:

Orders table is ACTIVE

3. Node.js: waitUntilTableExists.

import { DynamoDBClient, waitUntilTableExists } from "@aws-sdk/client-dynamodb";

const ddb = new DynamoDBClient({ region: "eu-west-1" });
const r = await waitUntilTableExists({ client: ddb, maxWaitTime: 120 }, { TableName: "Orders" });
console.log("Waiter state:", r.state);

Expected output:

Waiter state: SUCCESS

4. Verify: status is ACTIVE.

aws dynamodb describe-table --table-name Orders --region eu-west-1 --query Table.TableStatus --output text

Expected output:

ACTIVE

6. Logging and request IDs

Why do we need this?

When AWS Support asks "what was the request ID?", or a call is slow, you need to see what the SDK sent.

What is it?

boto3 has set_stream_logger; PHP has the debug option; Node takes a logger such as console. Every response carries a RequestId.

How to do it (step by step)

1. Python: turn on debug logs for botocore.

import boto3, logging

boto3.set_stream_logger("botocore", logging.DEBUG)
r = boto3.client("sts").get_caller_identity()
print("RequestId:", r["ResponseMetadata"]["RequestId"])

Expected output:

DEBUG - botocore.endpoint - Making request for OperationModel(name=GetCallerIdentity)
...
RequestId: 4c1d7a2e-9b1f-4f5e-a3c2-1d2e3f4a5b6c

2. PHP: debug option prints the HTTP wire.

<?php
require "vendor/autoload.php";
$sts = new Aws\Sts\StsClient(["region" => "us-east-1", "debug" => ["logfn" => fn($m) => error_log($m), "http" => true]]);
$r = $sts->getCallerIdentity();
echo "RequestId: ", $r["@metadata"]["headers"]["x-amzn-requestid"], PHP_EOL;

Expected output:

RequestId: 4c1d7a2e-9b1f-4f5e-a3c2-1d2e3f4a5b6c

3. Node.js: pass console as logger.

import { STSClient, GetCallerIdentityCommand } from "@aws-sdk/client-sts";
const sts = new STSClient({ region: "us-east-1", logger: console });
const r = await sts.send(new GetCallerIdentityCommand({}));
console.log("RequestId:", r.$metadata.requestId);

Expected output:

{ clientName: 'STSClient', commandName: 'GetCallerIdentityCommand', ... }
RequestId: 4c1d7a2e-9b1f-4f5e-a3c2-1d2e3f4a5b6c

7. The same call in Java, Go, .NET and the CLI

Why do we need this?

Teams often mix languages. Once you know the call in Python, PHP and Node.js, the other SDKs are the same API with different syntax.

What is it?

All of them call the ListObjectsV2 with pagination API. Use the CLI line to test it quickly from a terminal.

How to do it (step by step)

8. Clean up so nothing keeps costing

Why do we need this?

Lab resources left running cost money and can be a security risk.

What is it?

Delete in reverse order: children first, then parents.

How to do it (step by step)

1. Unset the retry env vars.

unset AWS_RETRY_MODE AWS_MAX_ATTEMPTS
env | grep -c AWS_MAX_ATTEMPTS

Expected output:

0

9. Interview questions and answers

Revision, comparison and interview cards

Revise this lesson in a few minutes. Save the cards on your phone or download them as PNG files. The QR code on each card opens this lesson.

More cards: Ravi Sir's Cards.