Ravindra BagaleCourses & study guides मराठी Track your progress

26. How to Read APIs, JSON and Zipped CSVs into pandas

26.6 Common Mistakes

Why. Two classroom disasters: (1) API key in Git; (2) downloading a 2 GB extract into RAM on a 8 GB laptop without filters.

What.

  • No timeout → hung kernel.
  • Ignoring status_code → parsing HTML as JSON.
  • DataFrame(payload) on a wrapped dict → wrong grain.
  • Reading zip path wrong (orders.csv inside but you pointed at folder).
  • Logging full response text that contains tokens.

How.

Safer pull habit

  1. Env var for secrets; never commit .env if it has real keys.
  2. Always timeout= and raise_for_status().
  3. Save raw JSON/CSV to disk before heavy cleaning.
  4. Prefer filtered API query params (city, date) — same idea as SQL Chapter 24.
  5. Check df.shape immediately.
  6. For huge files, read columns subset / chunksize later — start small in class.

Worked example A — Status check.

import requests

r = requests.get("https://httpbin.org/status/404", timeout=30)
print(r.status_code)
try:
    r.raise_for_status()
except requests.HTTPError as e:
    print("caught", type(e).__name__)

Worked example B — Wrong grain reminder.

import pandas as pd
payload = {"count": 2, "orders": [{"order_id": "BLK-1"}, {"order_id": "BLK-2"}]}
print(pd.DataFrame(payload))
print(pd.DataFrame(payload["orders"]))

Worked example C — Zip member pick.

Shraddha’s zip held orders.csv and README.txt. She listed names, then pd.read_csv(z.open("orders.csv")) pattern — or extracted one file first.

Ravindra Bagale's Tip

Key, timeout, status, shape — char checks. Galicha JSON parse = aadhi HTML error asel.