Beginning Apache Spark 3 Pdf May 2026

query.awaitTermination() Structured Streaming uses checkpointing and write‑ahead logs to guarantee end‑to‑end exactly‑once processing. 6.4 Event Time and Watermarks Handle late data efficiently:

Example:

from pyspark.sql.functions import udf def squared(x): return x * x

df.createOrReplaceTempView("sales") result = spark.sql("SELECT region, COUNT(*) FROM sales WHERE amount > 1000 GROUP BY region") This makes Spark accessible to analysts familiar with SQL. 4.1 Reading and Writing Data Supported formats: Parquet, ORC, Avro, JSON, CSV, text, JDBC, and more.

df = spark.read.parquet("sales.parquet") df.filter("amount > 1000").groupBy("region").count().show() You can register DataFrames as temporary views and run SQL:

General rule: 2–3 tasks per CPU core.