26. How to Read APIs, JSON and Zipped CSVs into pandas

26.4 Nested and Paged Payloads (Keep It Light)

By Ravindra Bagale, trainer LinkedIn Last updated: 24 October 2026

Why. Real APIs paginate (page=2) or wrap data in data.results. Beginners try to DataFrame(entire_dict) and get one ugly row.

What.

Walk keys until you reach a list of similar dicts.
Loop pages only with a hard cap in class (e.g. max 5 pages).
Prefer saving each page JSON under data/raw/ for debugging.

How.

Find the record list

Print payload.keys().
Drill one level at a time (payload["data"].keys()).
When you see a list, check type(payload["data"]["results"][0]).
Normalize that list only.
Stop pagination after a classroom limit.

Worked example A — Wrapped results.

import pandas as pd

payload = {
    "ok": True,
    "data": {
        "results": [
            {"order_id": "BLK-30", "city": "Pune", "amount_inr": 64},
            {"order_id": "BLK-31", "city": "Sambhaji Nagar", "amount_inr": 110},
        ]
    },
}
df = pd.json_normalize(payload["data"]["results"])
print(df)

Worked example B — Fake pagination loop (local).

import pandas as pd

pages = [
    [{"order_id": "BLK-40", "city": "Pune"}],
    [{"order_id": "BLK-41", "city": "Nashik"}],
]
frames = [pd.DataFrame(p) for p in pages]
df = pd.concat(frames, ignore_index=True)
print(df)

Worked example C — One-row disaster.

import pandas as pd

payload = {"orders": [{"order_id": "BLK-1", "city": "Pune"}]}
print("bad shape", pd.DataFrame(payload).shape)      # often wrong grain
print("good shape", pd.DataFrame(payload["orders"]).shape)

Ravindra Bagale's Tip

Dict keys print kara. List of dicts = rows. Ekach row dista ki path chukla.