Ravindra BagaleCourses & study guides Track your progress

7. First Spark Job (PySpark) on a Small CSV

7.2 Read CSV and groupBy

Why. groupBy on a small CSV proves the verbs.

# Tiny local sketch — may need pyspark installed; not a live cluster claim
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("raja-orders-lab").master("local[*]").getOrCreate()
df = spark.read.option("header", True).csv("data/raw/orders_nagpur.csv")
summary = df.groupBy("city").count()
summary.show()
# summary.write.mode("overwrite").parquet("data/clean/orders_by_city")
spark.stop()

Fictional data

Retail tables and city names here are fictional teaching data. Not real company metrics.