24 Structured Streaming

Structured Streaming

Outline

Programming Model
Input Sources
Starting Stream Queries
Output Modes
Output Sinks
Sample Structured Stream Processing
Operations on Streaming DataFrames
Window Operations on Event Time
Handling Late Data
Watermarking
Types of Time Window

1. The "Unbounded Table" Concept

In traditional batch processing, you run a query against a static dataset (a table with a fixed number of rows). In Structured Streaming, Spark treats the incoming data stream as an Input Table that is infinitely long.

Batch: You process rows 1 through 1,000 and the job ends.

Streaming: You start at row 1, and as new data arrives at the source, it is simply "appended" to the end of the table. The table is unbounded because it never stops growing.

2. From Static Query to Incremental Execution

Because the stream is viewed as a table, you can write your logic using standard SQL or DataFrame APIs (like SELECT, GROUP BY, or JOIN).

The Query: You define a "Result Table" by applying a query to the Input Table.

The Trigger: At a set interval (e.g., every 1 second), Spark checks for new data.

Incremental Processing: Instead of re-reading the entire table, Spark’s engine identifies only the new rows (the "delta"), updates the result, and writes it to the output.

3. How Data Flows: Input to Output

The model follows a three-step cycle to ensure the "Result Table" stays up to date:

Component Description

Input Table The live stream where new data arrives as rows.

Query The logic (e.g., count()) you apply to the data.

Result Table The final state of your data after the query is applied.

Output What gets written to the external sink (a database, a file, or a console).

4. Output Modes

Since the Result Table is constantly changing, you have to decide how you want to see the results. Spark provides three main modes:

Complete Mode: The entire Result Table is written to storage every time. (Best for aggregations like "total count since the start").

Append Mode: Only the new rows added to the Result Table since the last trigger are written. (Best for simple transformations).

Update Mode: Only the rows that were changed in the Result Table since the last trigger are written.

Key Benefit: This model eliminates the need for the developer to worry about "streaming-specific" problems like event-time ordering, state management, or fault tolerance. Spark handles those under the hood so you can focus on the business logic.

Programming Model

The key idea in Structured Streaming is to treat a live data stream as a table that is being continuously appended
This leads to a new stream processing model that is very similar to a batch processing model
Users can express the streaming computation as standard batch-like query as on a static table, and Spark runs it as an incremental query on the unbounded Input Table
A new data item arriving on the stream is like a new row being appended to Input Table

From Static Query to Incremental Execution

Because the stream is viewed as a table, you can write your logic using standard SQL or DataFrame APIs (like SELECT, GROUP BY, or JOIN).

The Query: You define a "Result Table" by applying logic to the Input Table.

The Trigger: At a set interval (e.g., every 1 second), Spark checks for new data.

Incremental Processing: Spark doesn't re-process the entire history. It identifies only the new rows (the "delta"), updates the result, and writes it to the output.

The model follows a cycle to ensure the "Result Table" stays up to date as the data evolves:

Component Description

Input Table The live stream where new data arrives as rows.

Query The logic (e.g., SUM or COUNT) you apply to the data.

Result Table The final state of your data after the query is applied.

Output Sink Where the results are saved (e.g., a Database, S3 bucket, or Kafka topic).

Since the "Result Table" is constantly changing, you must tell Spark how to write those changes to your destination.

Complete Mode: The entire Result Table is written to storage every time there is a change. This is usually used for aggregations (e.g., "Give me the total count of all events since the start").

Append Mode: Only the new rows added to the Result Table since the last trigger are written. This is the most common mode for simple transformations.

Update Mode: Only the rows that were modified in the Result Table since the last trigger are written.

Why This Matters

This model is powerful because it handles the hardest parts of streaming for you:

Fault Tolerance: If the system crashes, Spark uses "checkpointing" and "write-ahead logs" to know exactly which row it left off on.

Event-Time Processing: It can handle data that arrives late or out of order by looking at a timestamp column within the row rather than just when the data reached the server.

Programming Model

A query on the input will generate Result Table
Every trigger interval, let us say, every X seconds, the new rows get appended to Input Table
It will eventually updates Result Table
Whenever Result Table gets updated, we would want to write the changed result rows to an external sink

Programming Model

A complete process

Structured Streaming

In Structured Streaming, the Input Source is the starting point of your pipeline. To treat a stream as a table, Spark needs a way to "read" the data continuously. This is handled by the DataStreamReader object.

Here is a breakdown of how to create these streams and the specific sources available:

1. The readStream() Interface

In standard Spark (Batch), you use spark.read. In Structured Streaming, you use spark.readStream(). This tells Spark to keep the connection open and watch for new data rather than doing a one-time pull.

# Basic syntax example

df = spark.readStream \

.format("source_type") \

.option("key", "value") \

.load()

2. Common Data Sources

Spark provides several built-in sources, ranging from testing tools to production-grade connectors:

File Source

Reads files written in a directory as a stream of data.

How it works: Spark monitors a folder (e.g., S3, HDFS, or local). When a new file is dropped into that folder, Spark treats the content of that file as new rows appended to the Input Table.

Supported formats: CSV, JSON, ORC, Parquet, Text.

Kafka Source

This is the most common production source. It polls data from Apache Kafka (a distributed messaging system).

Key Feature: It allows you to subscribe to one or more "topics." Spark handles the offsets (the position in the stream) automatically to ensure no data is lost if the system restarts.

Socket Source (Testing Only)

Reads UTF-8 text data from a network socket connection.

Use Case: Ideal for learning or quick debugging. You can use a tool like nc -lk 9999 to type words into a terminal and see Spark process them in real-time.

Limitation: It is not fault-tolerant and should never be used in production.

Rate Source

A "dummy" source used for testing and benchmarking.

How it works: It automatically generates data at a specific number of rows per second.

Structure: Each generated row contains two columns: timestamp and value (a sequential number). This is perfect for testing how your system handles a specific volume of traffic.

Rate Per Micro-Batch Source

Similar to the Rate source, but instead of rows-per-second, it generates a fixed number of rows per trigger (micro-batch).

Use Case: Useful for testing backpressure and seeing how your transformations behave when a specific "chunk" of data arrives all at once.

Input Source

Structured Streaming is based on creating streaming DataFrames
Streaming DataFrames can be created through the DataStreamReader interface returned by SparkSession.readStream()
Structured Streaming may have the followign data sources
- File source: it reads files written in a directory as a stream of data
- Kafka source: it reads data from Kafka
- Socket source: it reads text data from a socket connection
- Rate source: it generates data at the specified number of rows per second, each output row contains a timestamp and value
- Rate Per Micro-Batch source: it generates data at the specified number of rows per micro-batch, each output row contains a timestamp and value

Structured Streaming

Outline

Programming Model
Input Sources
Starting Stream Queries
Output Modes
Output Sinks
Sample Structured Stream Processing
Operations on Streaming DataFrames
Window Operations on Event Time
Handling Late Data
Watermarking
Types of Time Window

Apache Spark Structured Streaming, the Dataframe.writeStream() method is the entry point for turning a static DataFrame into a streaming query. Once you call .start(), the engine begins processing data continuously as it arrives.

Here is a breakdown of the key components you need to configure:

1. Output Sink

This defines where the data goes. You must specify the format (e.g., parquet, kafka, console, or memory) and often a path for file-based sinks.

Example: .format("parquet").option("path", "path/to/destination")

2. Output Mode

This defines what is written to the sink when new data arrives. There are three main types:

Append Mode: Only new rows added to the result table since the last trigger are written. (Default)

Complete Mode: The entire updated result table is written to the sink every time.

Update Mode: Only the rows that were updated since the last trigger are written.

3. Trigger Interval (Optional)

This defines the timing of the stream processing. If not specified, Spark will process data as soon as the previous micro-batch completes.

Fixed interval: e.g., .trigger(Trigger.ProcessingTime("10 seconds"))

One-time: .trigger(Trigger.Once()) processes all available data and then stops.

4. Checkpoint Location (Critical for Production)

This is a directory in an HDFS-compatible file system where Spark saves the state and progress of the query.

Why it matters: If the query fails, Spark uses the checkpoint to pick up exactly where it left off, ensuring "exactly-once" delivery guarantees.

Example: .option("checkpointLocation", "path/to/checkpoint/dir")

5. Query Name (Optional)

A unique string used to identify the query. This is particularly useful if you are running multiple queries in a single Spark application and want to monitor them via the Spark UI.

Basic Code Structure

query = df.writeStream \

.format("console") \

.outputMode("append") \

.queryName("my_streaming_query") \

.trigger(processingTime='5 seconds') \

.option("checkpointLocation", "/mnt/checkpoints/") \

.start()

query.awaitTermination()

Starting Stream Queries

To start the streaming computation we use Dataframe.writeStream() method
One or more of the following details must be provided
- Details of the output sink: data format, location, etc.
- Output mode: what gets written to the output sink
- Query name: unique name of the query for identification (optional)
- Trigger interval: trigger interval (optional)
- Checkpoint location: location where the system will write all the checkpoint information (optional)

Structured Streaming

Outline

Programming Model
Input Sources
Starting Stream Queries
Output Modes
Output Sinks
Sample Structured Stream Processing
Operations on Streaming DataFrames
Window Operations on Event Time
Handling Late Data
Watermarking
Types of Time Window

In Apache Spark Structured Streaming, the Dataframe.writeStream() method is the entry point for turning a static DataFrame into a streaming query. Once you call .start(), the engine begins processing data continuously as it arrives.

Here is a breakdown of the key components you need to configure:

1. Output Sink

This defines where the data goes. You must specify the format (e.g., parquet, kafka, console, or memory) and often a path for file-based sinks.

Example: .format("parquet").option("path", "path/to/destination")

2. Output Mode

This defines what is written to the sink when new data arrives. There are three main types:

Append Mode: Only new rows added to the result table since the last trigger are written. (Default)

Complete Mode: The entire updated result table is written to the sink every time.

Update Mode: Only the rows that were updated since the last trigger are written.

3. Trigger Interval (Optional)

This defines the timing of the stream processing. If not specified, Spark will process data as soon as the previous micro-batch completes.

Fixed interval: e.g., .trigger(Trigger.ProcessingTime("10 seconds"))

One-time: .trigger(Trigger.Once()) processes all available data and then stops.

4. Checkpoint Location (Critical for Production)

This is a directory in an HDFS-compatible file system where Spark saves the state and progress of the query.

Why it matters: If the query fails, Spark uses the checkpoint to pick up exactly where it left off, ensuring "exactly-once" delivery guarantees.

Example: .option("checkpointLocation", "path/to/checkpoint/dir")

5. Query Name (Optional)

A unique string used to identify the query. This is particularly useful if you are running multiple queries in a single Spark application and want to monitor them via the Spark UI.

Basic Code Structure

query = df.writeStream.format("console").outputMode("append").queryName("my_streaming_query").trigger(processingTime='5 seconds').option("checkpointLocation", "/mnt/checkpoints/").start()

query.awaitTermination()

Output Modes

Output from stream processing is what gets written out to the persistent storage
Output can be one of the following modes
- Complete Mode: the entire updated Result Table is written to the external storage
- Append Mode: only the new rows appended to the Result Table since the last trigger activation are written to the external storage
- Update Mode: only the rows that were updated in the Result Table since the last trigger are written to the external storage
Event-time is the time embedded in the data itself
It is possible to operate on event-time, for example, to get the number of events generated by IoT devices every minute
Each event from the devices is a row in the table, and event-time is a column value in the row
Watermarking is used to specify the threshold of late data and it allows clean up old states

Structured Streaming

Outline

Programming Model
Input Sources
Starting Stream Queries
Output Modes
Output Sinks
Sample Structured Stream Processing
Operations on Streaming DataFrames
Window Operations on Event Time
Handling Late Data
Watermarking
Types of Time Window

In Structured Streaming (like in Apache Spark), an Output Sink is the final destination for your processed data. Once the engine processes a "micro-batch" or a stream of data, it needs to know where to send the results.

Think of it as the exit point of a pipeline. Depending on your use case—whether you're archiving data, powering a real-time dashboard, or triggering another process—you choose a different sink.

1. File Sink

The File Sink is used to store the output of your stream into a permanent storage system (like HDFS, S3, or a local file system). It is ideal for long-term data storage and batch-processing later on.

Common Formats: Parquet (standard for performance), ORC, JSON, and CSV.

writeStream

.format("parquet")

.option("path", "path/to/destination/dir")

.option("checkpointLocation", "path/to/checkpoint/dir")

.start()

2. Kafka Sink

The Kafka Sink pushes your processed data into one or more Apache Kafka topics. This is the gold standard for "event-driven" architectures where one stream's output becomes another system's input.

Use Case: Sending alerts to a notification service or feeding data into a real-time analytics engine.

Requirements: You must provide the Kafka brokers (bootstrap servers) and the target topic name.

3. Foreach (and ForeachBatch) Sink

The Foreach Sink is the "Swiss Army Knife" of sinks. Use this when the built-in sinks (File, Kafka, etc.) don't meet your needs. It allows you to write custom logic for every row.

Use Case: Writing to a NoSQL database (like Cassandra or MongoDB), calling an external API for every record, or sending custom emails.

ForeachBatch: A more common variant that allows you to process an entire micro-batch at once using standard Batch Dataframe operations.

4. Console Sink

The Console Sink prints the output directly to your terminal or standard output.

Warning: This is strictly for debugging. Because it prints to the console, it cannot handle high volumes of data and does not provide "fault tolerance" (if it crashes, the data shown on screen is gone).

Modes: It supports Append (only show new rows) and Complete (show the entire result table every time).

Output Sinks

There are a few types of built-in output sinks

File sink: stores the output to a directory

  
writeStream.format("parquet").option("path", "path/to/destination/dir").start()
	 // format can be "orc", "json", "csv", etc.

Kafka sink: stores the output to one or more topics in Kafka

  
writeStream.format("kafka").option("kafka.bootstrap.servers", "host1:port1,host2:port2")
	                   .option("topic", "updates").start()

Foreach sink: runs arbitrary computation on the records in the output
Console sink: prints the output to the console/stdoutfor every activation of a trigger, supports Append and Complete output modes

What is a Memory Sink?

When you use the memory sink, Spark stores the output of your streaming query in the driver's memory as an in-memory table. This allows you to run standard SQL queries (or DataFrame operations) against the results in real-time.

Key Characteristics

Temporary Storage: The data is stored in the system memory (RAM). If the Spark driver crashes or the application stops, the data is lost.

Queryability: You must provide a queryName. This name becomes the string you use in the FROM clause of your SQL queries.

Not for Production: Because it stores all data in the driver's memory, it can easily cause OutOfMemory (OOM) errors with large datasets. It is strictly intended for low-volume testing.

Supported Output Modes

The memory sink supports two primary modes that define how data is written to the in-memory table:

Append Mode: Only the new rows added to the result table since the last trigger are added to the memory table.

Complete Mode: The entire result table is recalculated and replaces the previous version in memory every time there is new data. This is typically used for streaming aggregations (e.g., keeping a running count of words).

Syntax Breakdown

The code snippet you provided is the standard way to initialize this sink:

writeStream \

.format("memory") \

.queryName("tableName") \

.start()

.format("memory"): Tells Spark to use the memory sink provider.

.queryName("tableName"): Defines the name of the temporary table created in the Spark catalog.

.start(): Begins the execution of the streaming query.

How to view the results: Once the stream is running, you can view the data using a standard Spark SQL command:

spark.sql("SELECT * FROM tableName").show()

Output Sinks

Memory sink: outputs the results memory as an in-memory table, supports Append and Complete output modes

Structured Streaming

Outline

Programming Model
Input Sources
Starting Stream Queries
Output Modes
Output Sinks
Sample Structured Stream Processing
Operations on Streaming DataFrames
Window Operations on Event Time
Handling Late Data
Watermarking
Types of Time Window

In Structured Streaming, the example you provided is the "Hello World" of stream processing. It treats a live data stream as an unbounded table—a table that continuously grows as new data arrives.

Step-by-Step Breakdown

1. Setup and Imports

First, you bring in the necessary tools. explode and split are essential for transforming a single string (a sentence) into multiple rows (individual words).

from pyspark.sql import SparkSession

from pyspark.sql.functions import explode, split

2. The SparkSession

This is the entry point for Spark functionality. The appName helps you identify this specific job in the Spark UI.

spark = SparkSession.builder.appName("StructuredNetworkWordCount").getOrCreate()

3. Creating the Input DataFrame

The readStream command defines your Source. In this case, it’s a network socket.

lines = spark.readStream \

.format("socket") \

.option("host", "localhost") \

.option("port", 9999) \

.load()

lines: This is not a static DataFrame; it is a Streaming DataFrame.

Schema: By default, the socket source provides a DataFrame with a single column named value containing the text strings.

The Transformation Logic (The "Word Count")

To finish the logic you started, you need to transform the lines into word counts. Here is how that looks conceptually:

Split: Break the sentences into arrays of words.

Explode: Take that array and turn each element into its own row.

Group & Count: Aggregate the results.

# Split the lines into words

words = lines.select(

explode(

split(lines.value, " ")

).alias("word") )

# Generate running word count

wordCounts = words.groupBy("word").count()

Sample Structured Stream Processing

Assume that we want to count the number of words in text data received from a data server listening on a TCP socket

Import SparkSession and string processing functions explode and split

  
from pyspark.sql import SparkSession
from pyspark.sql.functions import explode
from pyspark.sql.functions import split

Create SparkSession

   
spark = SparkSession.builder.appName("StructuredNetworkWordCount").getOrCreate()

Create DataFrame lines representing the stream of input lines from connection to localhost:9999

   		
lines = spark.readStream.format("socket").option("host", "localhost").option("port", 9999).load()

This completes the logic for the Structured Network Word Count. At this stage, you are transforming the raw input stream into a summarized result and defining how that result is delivered to you.

1. Splitting the Lines into Words

The input from the socket arrives as a single column named value (e.g., "hello world").

words = lines.select( explode( split(lines.value, " ") ).alias("word") )

split(lines.value, " "): This breaks the string into an array of strings: ["hello", "world"].

explode(...): This is a powerful function that takes an array and turns every element into a new row.

2. Counting the Words

This is where the "Structured" part of Structured Streaming shines. You use the same syntax as a static DataFrame.

wordCounts = words.groupBy("word").count()

Spark maintains a state in the background. It remembers the counts of every word seen since the stream started and updates them as new data arrives.

3. Starting the Query (The Console Sink)

The query won't actually do anything until you call .start().

query = wordCounts.writeStream.outputMode("complete").format("console").start()

.outputMode("complete"): Because we are doing an aggregation (counting), "complete" mode ensures that every time the console updates, you see the entire result table (all words and their current total counts), not just the changes.

.format("console"): This tells Spark to print the results directly to your terminal or IDE log.

4. Awaiting Termination

Streaming jobs are designed to run forever.

query.awaitTermination()

This line tells the driver script to sit and wait while the background streaming process runs. Without this, the script would reach the end of the file and exit immediately, killing your stream before it even has a chance to process data.

Sample Structured Stream Processing

Split the lines into words

 
words = lines.select( explode( split(lines.value, " ") ).alias("word") )

Count the words

 
wordCounts = words.groupBy("word").count()

Start running the query that prints the running counts to the console

 
query = wordCounts.writeStream.outputMode("complete").format("console").start()

Wait for query termination

This section describes the practical, hands-on execution of the Word Count stream. It involves two separate processes communicating over a network port: a Data Producer (Netcat) and a Data Consumer (Spark).

1. The Data Server: Netcat (nc)

Netcat is a utility used to read from and write to network connections.

nc -lk 9999

-l: Tells Netcat to "listen" for an incoming connection.

-k: Keeps the server alive even after a client disconnects.

9999: The port number.

Anything you type into this terminal window after starting it acts as a "live feed" for Spark.

2. The Processor: Spark Submit

In a different terminal, you launch the Spark application.

./bin/spark-submit examples/src/main/python/sql/streaming/structured_network_wordcount.py localhost 9999

spark-submit: The script used to launch applications on a Spark cluster.

localhost 9999: These are arguments passed to the Python script telling it where to look for the data.

3. The Execution Flow

Here is what happens inside Spark's engine as you type:

First Input: "abc def"

You type abc def in the Netcat window.

Spark triggers a batch. It splits the line and counts the words.

Output Table: | word | count | | :--- | :--- | | abc | 1 | | def | 1 |

Second Input: "abc def"

You type abc def again.

Spark triggers a new batch. Because we are in Complete Mode, Spark combines the new data with the old data.

Updated Output Table: | word | count | | :--- | :--- | | abc | 2 | | def | 2 |

Sample Structured Stream Processing

The input is simulated by Netcat (a small utility found in most Unix-like systems) as a data server

In a diferent terminal start a sample stream processing application

 
./bin/spark-submit examples/src/main/python/sql/streaming/structured_network_wordcount.py localhost 9999

In Netcat window we type first ...
... and later on again

This visual output is the final result of your Console Sink. It shows exactly how Spark Structured Streaming processes data in discrete "micro-batches" and updates the results over time.

Understanding the Console Output

The output is divided into distinct sections, each representing a Trigger or a "Batch."

Batch 0: Initialization

-------------------------------------------

Batch: 0

-------------------------------------------

+----+-----+

|word|count|

+----+-----+

What happened: When you first start the script, Spark initializes the engine and the empty result table. Since no data has been typed into Netcat yet, the table is empty.

Batch 1: The First Input

-------------------------------------------

Batch: 1

-------------------------------------------

+----+-----+

|word|count|

+----+-----+

| abc|1|

| def|1|

+----+-----+

Trigger: You typed abc def into the Netcat terminal.

Processing: Spark detected the data, split the string into two rows, grouped them by word, and calculated the count.

Batch 2: The State Update

-------------------------------------------

Batch: 2

-------------------------------------------

+----+-----+

|word|count|

+----+-----+

| abc|2|

| def|2|

+----+-----+

Trigger: You typed abc def a second time.

Stateful Aggregation: Because we used .outputMode("complete"), Spark doesn't just show the new words. It looks at its internal State Store, retrieves the previous counts (1 and 1), adds the new data, and displays the updated totals (2 and 2).

Sample Structured Stream Processing

Then we could see the following output in an application Terminal window

-------------------------------------------                                     
Batch: 0
-------------------------------------------
+----+-----+
|word|count|
+----+-----+
+----+-----+
-------------------------------------------                                     
Batch: 1
-------------------------------------------
+----+-----+
|word|count|
+----+-----+
| abc|    1|
| def|    1|
+----+-----+
------------------------------------------                                     
Batch: 2
-------------------------------------------
+----+-----+
|word|count|
+----+-----+
| abc|    2|
| def|    2|
+----+-----+

This scenario shifts from text processing to numerical stream processing. Instead of counting every item, you are now applying a transformation (casting data types) and a filter (conditional logic) to the incoming data stream.

1. The Setup

The code starts by initializing the environment and connecting to the data source.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("StructuredNetworkWordCount").getOrCreate()

socketDF = spark.readStream.format("socket") \

.option("host", "localhost") \

.option("port", 9999) \

.load()

2. Data Type Transformation (Casting)

By default, the socket source reads everything as a String. To perform numerical comparisons (like >= 100), you must convert the data type.

numbers = socketDF.select(socketDF.value.cast("Int").alias("number"))

.cast("Int"): Converts the raw string from the socket into an integer. If you send "105", it becomes the number 105. If you send "abc", it becomes null.

.alias("number"): Names the column so you can reference it easily in the next step.

3. Applying the Filter

This is where the business logic is applied. Spark will evaluate every incoming number against your criteria.

numbersge100 = numbers.filter(numbers.number >= 100)

filter(...): Acts as a gatekeeper. Only rows where the condition is true are passed to the next stage of the pipeline.

Stateless Processing: Unlike the "Word Count" example which had to remember previous totals, this specific filter is stateless. Spark looks at each number individually, decides if it's ≥100, and either keeps it or drops it.

Sample Structured Stream Processing

Assume that instead of words we would like to count the numbers greater or equal 100 coming as an input stream of numbers
Import SparkSession

Create SparkSession

   
spark = SparkSession.builder.appName("StructuredNetworkWordCount").getOrCreate()

Create DataFrame socketDF representing the stream of input numbers from connection to localhost:9999

   		
socketDF = spark.readStream.format("socket").option("host", "localhost").option("port", 9999).load()
numbers = socketDF.select(socketDF.value.cast("Int").alias("number"))

Create DataFrame numbersge100

   		
numbersge100= numbers.filter(numbers.number>=100)

This part of the process defines the Aggregation and the Execution of your numerical filter stream. Even though the logic is similar to the word count example, the way Spark handles the integers is distinct.

1. Grouping and Counting

numberCounts = numbersge100.groupBy("number").count()

Aggregation by Value: This doesn't just count how many "big numbers" arrived; it counts the occurrences of each specific number that is ≥100.

State Management: Spark keeps a running tally in its "State Store." If you send the number 105 now, and another 105 five minutes later, Spark will update the count for that specific key to 2.

2. The Output Sink (Console)

query = numberCounts.writeStream.outputMode("complete").format("console").start()

Output Mode "complete": This is vital for aggregations. It tells Spark to print the entire summary table every time a new number arrives. This allows you to see the full list of numbers ≥100 and how many times each has appeared.

Format "console": The results will appear in your terminal window in a formatted table.

Sample Structured Stream Processing

Count the numbers greater or equal 100

   		
numberCounts = numbersge100.groupBy("number").count()

Start running the query that prints the running counts to the console

 
query = numberCounts.writeStream.outputMode("complete").format("console").start()

This scenario transitions from a Network Source (volatile) to a File Source (persistent). In this setup, Spark monitors a specific directory on your computer or cluster and automatically processes any new files as they are dropped into that folder.

1. The Imports

from pyspark.sql import SparkSession

from pyspark.sql.types import StructType

Note: In your snippet, StructType is imported from functions, but it actually resides in pyspark.sql.types. It is used to define the "shape" of your data.

2. Why do we need a userSchema?

When reading from a network socket, Spark assumes the data is a simple string. However, when reading files, Structured Streaming requires you to define the schema manually.

userSchema = StructType().add("number", "integer")

Spark does not perform "schema inference" (guessing the data types) on streaming file sources by default. This is because:

Performance: It would be too slow to scan all files to guess types every time new data arrives.

Consistency: It ensures that your streaming pipeline doesn't break if a new file accidentally contains a different format.

3. How the File Source Works

In a file-based stream, Spark treats the folder as the table.

Existing Files: If there are files in the folder when you start the stream, Spark processes them first.

New Files: As soon as you move or copy a new file into that folder, Spark detects it and triggers a new micro-batch.

4. Setting up the Stream

To complete the logic you started, you would use readStream with the json, csv, or parquet format, depending on your file type:

# Example for CSV files

inputDF = spark.readStream \

.schema(userSchema) \

.option("header", "false") \

.csv("path/to/your/input/directory")

Key Rules for File Streaming

Atomicity: Files must be placed in the folder in their entirety. If you are writing a file directly into the folder, Spark might try to read it before you finish writing. It is best to write to a temporary folder and "move" it into the stream folder.

Immutability: Spark assumes that once a file is in the folder, it will not change. If you edit an existing file, Spark will not re-process it.

Sample Structured Stream Processing

Assume that we would like to process a stream of files inserted into a given folder
Each file consists of lines with numbers

Import SparkSession and StructType

  
from pyspark.sql import SparkSession
from pyspark.sql.functions import StructType

Create SparkSession

   
spark = SparkSession.builder.appName("StructuredNetworkWordCount").getOrCreate()

Create userSchema

   
userSchema = StructType().add("number", "integer")

This example demonstrates how to use the File Source in Structured Streaming. Instead of listening to a live network connection, Spark is now "watching" a directory for new data.

1. Reading the CSV Stream

csvDF = spark.readStream.schema(userSchema).csv("/user/bigdata")

The Directory: Spark monitors the folder /user/bigdata. Any new file dropped into this folder will be treated as a new batch of data.

The Schema: As discussed previously, userSchema tells Spark exactly how to interpret the bytes in those CSV files (in this case, treating the data as an integer named "number").

2. The Logic: Grouping and Counting

numberCounts = csvDF.groupBy("number").count()

This transformation creates a running tally. It is important to note how this behaves with files:

If File A arrives: Contains 100, 200. Spark shows 100: 1, 200: 1.

If File B arrives: Contains 100, 300. Spark updates the state and shows 100: 2, 200: 1, 300: 1.

3. Execution and Output

query = numberCounts.writeStream.outputMode("complete").format("console").start()

Output Mode "complete": This ensures that every time a new file is added to the folder, the console displays the entire global count of all numbers found in all files processed so far.

Fault Tolerance: Because the data is coming from files (which are persistent), if your Spark job crashes and you restart it, Spark can use its "checkpointing" (if configured) to pick up exactly where it left off without reprocessing the old files.

Sample Structured Stream Processing

Create DataFrame csvDF representing a stream of csv files inserted into a folder /user/bigdata
Count the numbers in each file inserted into a folder /user/bigdata

Start running the query that prints the running counts to the console

 
query = numberCounts.writeStream.outputMode("complete").format("console").start()

Structured Streaming

Outline

Programming Model
Input Sources
Starting Stream Queries
Output Modes
Output Sinks
Sample Structured Stream Processing
Operations on Streaming DataFrames
Window Operations on Event Time
Handling Late Data
Watermarking
Types of Time Window

Operating on a Streaming DataFrame feels almost identical to operating on a static one. This is the core philosophy of Structured Streaming: you write your logic as if the data were static, and Spark handles the "streaming" complexity under the hood.

1. Basic Operations (Selection & Projection)

You can use standard DataFrame methods to filter and transform your data.

df.select("device").where("signal > 10")

Selection: Choosing specific columns (like device).

Projection/Filtering: Using .where() or .filter() to narrow down the data. In this case, only devices with a high signal strength are passed through.

Statelessness: These operations are "stateless," meaning Spark doesn't need to remember previous rows to process the current ones.

2. Aggregations (Stateful Operations)

Aggregations allow you to calculate running totals, averages, or counts.

df.groupBy("deviceType").count()

State Management: Unlike filters, Spark must remember the counts for each deviceType across all previous batches. This is stored in an internal state store.

Incremental Updates: Every time a new batch of IOT data arrives, Spark updates only the relevant device types in the result table.

3. Using SQL Queries

If you prefer standard SQL syntax over the DataFrame API, you can register the stream as a temporary view.

df.createOrReplaceTempView("updates")

spark.sql("select count(*) from updates")

Note: When you run spark.sql(...) on a view created from a streaming DataFrame, the resulting object is also a streaming DataFrame. It won't actually execute until you define a sink (like .writeStream) and call .start().

4. Verification: isStreaming()

Because streaming and static DataFrames use the same API, it can sometimes be confusing which one you are working with.

df.isStreaming() # Returns True or False

Operations on Streaming DataFrames

Basic Operations - Selection, Projection, Aggregation

   
df = ...  # streaming DataFrame with IOT device data with schema
      	  { device: string, deviceType: string, signal: double, time: DateType }

Select the devices which have signal more than 10
Running count of the number of updates for each device type

Registering a streaming DataFrame/Dataset as a temporary view and then applying SQL

   
df.createOrReplaceTempView("updates")
spark.sql("select count(*) from updates")

Checking if DataFrame is streaming

This scenario involves processing a continuous stream of CSV-like data where each file contains a list of names and their corresponding grades. This is a classic "ETL" (Extract, Transform, Load) streaming pattern.

1. The Data Structure

The input consists of raw text files where each line follows a specific pattern: Name,Grade.

Example content: James,20, Alice,55, Harry,0, Charlie,99.

Since the data has a fixed structure, we treat this as a CSV source.

2. The Imports

from pyspark.sql import SparkSession

from pyspark.sql.types import StructType, StringType, IntegerType

Note on Syntax: While the snippet suggests importing StructType from functions, it is technically part of pyspark.sql.types. You will also need StringType and IntegerType to define the columns for the name and the grade.

3. Creating the SparkSession

The SparkSession is the driver that coordinates the workers reading the files from your directory.

spark = SparkSession.builder \

.appName("GradeProcessingStream") \

.getOrCreate()

4. Defining the Schema (The Next Step)

Because Spark is reading a stream of files, it cannot "guess" the data types efficiently. You must provide a Schema. Based on your description, the schema would look like this:

# Define the structure: Name (String) and Grade (Integer)

userSchema = StructType() \

.add("name", StringType()) \

.add("grade", IntegerType())

How Spark Sees the Folder

Once the stream starts, Spark treats the folder as an Unbounded Table.

File Name Content Spark's View (Internal Table)

file1.txt James,20 `James

file2.txt Alice,55 `James

Operations on Streaming DataFrames

Assume that we would like to process a stream of files inserted into a given folder
Each file consists of lines with names and grades separated with a comma

Import SparkSession and StructType

  
from pyspark.sql import SparkSession
from pyspark.sql.functions import StructType

Create SparkSession

   
spark = SparkSession.builder.appName("StructuredNetworkWordCount").getOrCreate()

This example demonstrates the transition from raw file data to high-level SQL analytics in a streaming environment. By registering a streaming DataFrame as a temporary view, you can use standard SQL syntax to perform complex aggregations.

1. Defining the Schema and Source

userSchema = StructType().add("name", "string").add("score", "integer")

gradesDF = spark.readStream.schema(userSchema).csv("/user/bigdata")

Spark needs to know that the first column is a String (the name) and the second is an Integer (the score). It then monitors the folder /user/bigdata for any new CSV files.

2. Bridging DataFrames and SQL

gradesDF.createOrReplaceTempView("grades")

This is a crucial step. It maps the streaming DataFrame to a table name (grades) in Spark's internal catalog. This allows the SQL engine to "see" the stream as if it were a regular database table.

3. The Analytics Query

sqlDF = spark.sql("select count(*), max(score), min(score), avg(score) from grades")

Even though this looks like a static query, because it is based on a Streaming View, the resulting sqlDF is a Streaming DataFrame. Spark will incrementally update these four metrics:

count(*): Total number of students processed.

max(score): The highest grade seen so far.

min(score): The lowest grade seen so far.

avg(score): The running mathematical average of all scores.

4. Outputting the Result

query = sqlDF.writeStream.outputMode("complete").format("console").start()

Complete Mode: Since we are calculating "Global" aggregates (max, min, avg for the whole history), we use complete mode so the console always shows the most up-to-date summary of the entire dataset.

Operations on Streaming DataFrames

Create userSchema

userSchema = StructType().add("name", "string").add("score", "integer")

Create DataFrame gradesDF representing a stream of csv files inserted into a folder /user/bigdata
Create a temporary view grades of DataFrame gradesDF

Use SQL to create a Dataframe with the counter of grades, maximum, minium, and average grade

   		
sqlDF=spark.sql("select count(*), max(score), min(score), avg(score) from grades")

Start running the query that prints the running counts to the console

 
query = sqlDF.writeStream.outputMode("complete").format("console").start()

This output shows the real-time evolution of your statistics as Spark processes new files. Because you used Complete Mode, Spark doesn't just show you the data from the latest file; it recalculates the "global" state across all files seen so far.

Analysis of the Batches

Batch 0: Initial Files

Input Data: Likely the first file containing James, 20 and Alice, 55.

Calculation: * Count: 2

Max: 55

Min: 20

Average: (20+55)/2=37.5

Batch 1: New Data Arrives

Input Data: A second file is added, containing Harry, 0 and Charlie, 99.

State Update: Spark combines these with the previous two records.

Count: Now 4.

Max: Updated from 55 to 99.

Min: Updated from 20 to 0.

Average: (20+55+0+99)/4=43.5.

Batch 2: Continued Incremental Processing

Input Data: A third file with two more records arrives (e.g., scores like 30 and 40).

Observation: Notice that while the Count increased to 6, the Max (99) and Min (0) stayed the same because the new scores didn't break the existing records. The Average adjusted slightly to 41.5.

Why the output looks like this

Micro-Batch Triggers: Even though the data source is a folder, Spark checks that folder at a specific interval (the trigger). Each time it finds new files, it increments the Batch ID.

State Store: In the background, Spark is maintaining an "In-memory State Store." It keeps track of the running sum and running count to calculate the average, and the current absolute min/max.

Operations on Streaming DataFrames

Sample results

   
------------------------------------------                                     
Batch: 0
-------------------------------------------
+--------+----------+----------+----------+
|count(1)|max(score)|min(score)|avg(score)|
+--------+----------+----------+----------+
|       2|        55|        20|      37.5|
+--------+----------+----------+----------+

-------------------------------------------
Batch: 1
-------------------------------------------
+--------+----------+----------+----------+
|count(1)|max(score)|min(score)|avg(score)|
+--------+----------+----------+----------+
|       4|        99|         0|      43.5|
+--------+----------+----------+----------+

-------------------------------------------
Batch: 2
-------------------------------------------
+--------+----------+----------+----------+
|count(1)|max(score)|min(score)|avg(score)|
+--------+----------+----------+----------+
|       6|        99|         0|      41.5|
+--------+----------+----------+----------+

Structured Streaming

Outline

Programming Model
Input Sources
Starting Stream Queries
Output Modes
Output Sinks
Sample Structured Stream Processing
Operations on Streaming DataFrames
Window Operations on Event Time
Handling Late Data
Watermarking
Types of Time Window

In Structured Streaming, Window Operations allow you to perform aggregations (like counting words or averaging temperatures) over specific "chunks" of time. Instead of grouping by a category like deviceType, you group by a time interval.

1. The Concept of Event Time

Event Time is the time embedded within the data itself (e.g., a timestamp generated by an IoT sensor or a log entry), rather than the time the data reaches Spark. This is crucial because data often arrives out of order or delayed due to network lag.

2. Grouping by Windows

In a basic aggregation, a word like "apple" has one total count. In a windowed aggregation, "apple" has a different count for every time window in which it appeared.

The Example: 10-minute windows, updating every 5 minutes

This is known as a Sliding Window.

Window Duration (10 min): How long the window lasts.

Slide Duration (5 min): How often a new window starts.

Because the slide (5 min) is smaller than the duration (10 min), the windows overlap. This means a single event—like a word sent at 12:07—will actually fall into two different windows:

The 12:00 - 12:10 window.

The 12:05 - 12:15 window.

3. How the Result Table Looks

Spark represents these windows as a special column containing a struct with start and end times.

Window Word Count

12:00 - 12:10 hello 5

12:05 - 12:15 hello 3

12:10 - 12:20 hello 1

4. Why Use Sliding Windows?

Sliding windows are perfect for "moving averages" or detecting trends. If you only used Tumbling Windows (where windows don't overlap, e.g., 10:00-10:10 and 10:10-10:20), you might miss a spike in activity that happens exactly at 10:10. Sliding windows ensure that continuous activity is captured accurately.

Window Operations on Event Time

In a simple grouped aggregation, aggregate values like for example counts, are maintained for each unique value in the user-specified grouping column
In a case of window-based aggregations, aggregate values are maintained for each window the event-time of a row falls into
For example we want to count words within 10 minute windows, updating every 5 minutes

In stream processing, windowing allows you to group data that arrives within specific time slices. When we talk about Event Time, we are grouping data based on the timestamp attached to the data point itself (when the event happened), rather than when it reached the processing engine.

The code snippet uses a Sliding Window in Spark Structured Streaming. Here is the breakdown of how it works.

1. Breaking Down the Function

The function window(words.timestamp, "10 minutes", "5 minutes") defines two critical parameters:

Window Duration ("10 minutes"): How long the window "lasts." This determines the range of time covered by each bucket.

Slide Duration ("5 minutes"): How often a new window is started.

Because the slide duration (5m) is smaller than the window duration (10m), the windows will overlap. In this case, every event will fall into two different windows.

2. How the Data is Aggregated

When you run words.groupBy(window(...), words.word).count(), Spark maintains a running count for every unique word within every active time window.

3. Visualizing the Timeline

Imagine two words, "Apple" and "Banana," arriving with different timestamps:

12:00: Window 1 starts (12:00 - 12:10).

12:05: Window 2 starts (12:05 - 12:15). Window 1 is still open.

12:07: An event "Apple" arrives with timestamp 12:07.

It falls into Window 1 (12:00–12:10).

It falls into Window 2 (12:05–12:15).

12:10: Window 1 closes/completes. Window 3 starts (12:10 - 12:20).

Window Range Words Included Count

12:00 - 12:10 Apple (12:07) 1

12:05 - 12:15 Apple (12:07) 1

Why use Event Time instead of Processing Time?

Using event time is crucial for accuracy when data is delayed. If a "word" is generated at 12:01 but delayed by network lag and arrives at 12:11, Spark will use the 12:01 timestamp to put it in the correct 12:00–12:10 window, rather than grouping it based on the 12:11 arrival time.

Note on Late Data: In a real-world scenario, you would typically pair this with a .withWatermark() to tell Spark how long to wait for late-arriving data before closing a window permanently.

Window Operations on Event Time

In window-based aggregation it is possible to use groupBy() and window() operations

    
words = ...  # streaming DataFrame of schema { timestamp: Timestamp, word: String }

      
windowedCounts = words.groupBy(window(words.timestamp,
                               "10 minutes", "5 minutes"),
                               words.word).count()

Structured Streaming

Outline

Programming Model
Input Sources
Starting Stream Queries
Output Modes
Output Sinks
Sample Structured Stream Processing
Operations on Streaming DataFrames
Window Operations on Event Time
Handling Late Data
Watermarking
Types of Time Window

In stream processing, Late Data refers to events that arrive at the system significantly after the timestamp recorded in the event itself. Because Spark Structured Streaming uses Event Time (as seen in your windowing example), it can intelligently handle these "stragglers."

1. Updating Past Aggregations

When late data arrives, Spark looks at the event’s timestamp, identifies which window(s) it belongs to, and re-calculates the aggregation for those specific windows.

State Store: Spark keeps the intermediate counts for previous windows in a "State Store."

Update: If an event from 12:03 arrives at 12:15, Spark finds the 12:00–12:10 window in the state store and increments the count.

Output: The system then emits an updated result for that window.

2. The Problem: Unbounded State

If Spark kept every window open forever to wait for late data, the system would eventually run out of memory. This is where Watermarking becomes essential.

A Watermark is a threshold that tells Spark how late data can be. It is defined as: Max Event Time seen by the engine - Late Threshold

3. How it works with Code

To handle late data efficiently, you chain the .withWatermark() operator before your grouping:

windowedCounts = words \

.withWatermark("timestamp", "10 minutes") \

.groupBy(

window(words.timestamp, "10 minutes", "5 minutes"),

words.word) \

.count()

The Lifecycle of Late Data:

Arrival: An event arrives with a timestamp older than the current time.

Validation: Spark checks the Watermark.

If the event is newer than the watermark, the state is updated.

If the event is older than the watermark, it is considered "too late" and is dropped (ignored) to save memory.

Cleanup: Once the watermark moves past the end of a window (plus the late threshold), Spark clears that window from its memory.

Handling Late Data

What happens if one of the events arrives late to the application ?
Late data can be used to update the past aggregations

Structured Streaming

Outline

Programming Model
Input Sources
Starting Stream Queries
Output Modes
Output Sinks
Sample Structured Stream Processing
Operations on Streaming DataFrames
Window Operations on Event Time
Handling Late Data
Watermarking
Types of Time Window

In stream processing, Watermarking is the mechanism used to balance two competing needs: the ability to handle late data and the need to free up system memory (State Store).

Without watermarking, Spark would have to keep the counts for every window in memory forever, because it would never know if a very old piece of data might eventually show up.

1. The Watermark Logic

The "Watermark" is a moving boundary calculated as:

Watermark=(Maximum event time seen so far)−(Allowed late threshold)

As Spark processes new data, it tracks the highest timestamp it has encountered. The watermark trails behind that maximum time by the threshold you define (e.g., 10 minutes).

2. The "Window Dropping" Rule

A window's state is maintained in memory only as long as the watermark hasn't passed the window's end time.

If Event Time > Watermark: The data is accepted, and the window aggregate is updated.

If Event Time < Watermark: The data is considered "too late" and is dropped.

Cleanup: Once the Watermark exceeds the End Time of a window, Spark assumes no more relevant data will arrive for that period and deletes that window from the state store to save memory.

3. Step-by-Step Example

Imagine we have a 10-minute window and a 5-minute watermark threshold.

Time Max Event Time Seen Calculated Watermark Action

12:15 12:15 12:10 (12:15 - 5m) Any data for the 12:00-12:10 window is still accepted.

12:20 12:20 12:15 (12:20 - 5m) The Watermark (12:15) is now greater than the end of the 12:00-12:10 window.

Result State Cleanup: The 12:00-12:10 window is cleared from memory.

Watermarking

Keeping old data for a very long time is not possible due to the transient memory limitations
Watermarking allows to track the current event time in the data and attempt to clean up old aggregations
A watermark of a query determines an event time column and the threshold on how late the data is expected to be in terms of event time
For example, if a window ends at time T, the system maintains state and allow late data to update the state until (max event time seen by the engine - late threshold > T)
Late data within the threshold are aggregated, but data later than the threshold are dropped

In Spark Structured Streaming, Watermarking combined with Update Mode is a powerful way to handle late data while keeping the system's memory usage under control.

1. How Watermarking Works here

The watermark is a moving threshold that lags behind the maximum event time seen by the engine. In your example:

Watermark=(Max Event Time seen)−10 minutes

Any data with a timestamp older than this calculated watermark is considered "too late" and is dropped. Any data newer than the watermark is accepted and used to update the counts.

2. Update Mode: The "Continuous Update" Logic

In Update Mode, the behavior is focused on providing the most recent results as soon as possible:

Immediate Output: As soon as a micro-batch is processed, any window that received new data (including late data) is updated in the Result Table and pushed to the sink.

Late Data Integration: If an event is late but still newer than the watermark, Spark retrieves the old window from its internal State Store, increments the count, and emits that updated row.

State Eviction (Cleanup): Once the watermark moves past the end of a specific window, Spark knows that no more "legal" late data can arrive for that window. It deletes that window from the State Store to free up memory. Once deleted, any future data for that window—no matter what—will be dropped.

3. Visualizing the Process

Imagine the engine has seen events up to 12:15. With a 10-minute watermark, the threshold is 12:05.

Event Arrival Time Event Timestamp (Event Time) Watermark Action

12:15 12:15 12:05 Watermark updates to 12:05.

12:16 12:08 (Late) 12:05 Accepted: 12:08 > 12:05. The 12:00-12:10 and 12:05-12:15 windows are updated and sent to the sink.

12:17 12:02 (Too Late) 12:05 Dropped: 12:02 < 12:05. This data is ignored.

Watermarking

In the example below the threshold of how late is the data allowed to be is set to 10 minutes

    
words = ...  # streaming DataFrame of schema { timestamp: Timestamp, word: String }

      
windowedCounts = words.withWatermark("timestamp", "10 minutes").groupBy(
                           window(words.timestamp, "10 minutes", "5 minutes"),
                           words.word).count()

If a query is run in Update output mode, the system keeps updating counts of a window in the Result Table until the window is older than the watermark

In Spark Structured Streaming, Update Mode is often the preferred choice when you need real-time dashboards or low-latency monitoring. When combined with Watermarking, it allows you to see results immediately while still managing memory efficiently.

1. The Core Behavior

In Update Mode, Spark outputs only the rows that were updated in the most recent micro-batch.

Low Latency: Unlike Append Mode (which waits for the watermark to pass before showing anything), Update Mode emits the "current best count" for a window as soon as the first event for that window arrives.

Continuous Refinement: If late data arrives later (but still within the watermark), Spark simply updates that row and sends it to the sink again.

2. How Watermarking Works in Update Mode

The watermark serves one primary purpose in Update Mode: State Cleanup.

Without a watermark, Spark would have to keep the counts for every window (e.g., 8:00 AM, 9:00 AM, 10:00 AM...) in its internal memory forever, just in case a stray event from 8:00 AM arrives tomorrow. This would eventually cause an Out of Memory (OOM) error.

The Cleanup Rule: A window is removed from the internal state store only when:

Watermark>Window End Time

Once a window is evicted from state, Spark "forgets" it entirely. If an event for that window arrives after eviction, it is dropped and no update is sent to the sink.

3. Comparison: Update Mode vs. Append Mode

Understanding the difference is critical for choosing the right sink (like a database or a console).

Feature Update Mode Append Mode

When is data emitted? Immediately (every trigger). Only after the watermark passes the window end.

Can rows be updated? Yes. The same window can be sent to the sink multiple times. No. Each window is emitted exactly once.

Memory Cleanup? When watermark > window end. When watermark > window end.

Best Use Case: Real-time dashboards (Grafana, PowerBI). Writing to files (Parquet/CSV) or non-updateable sinks.

Watermarking

Update mode

In Append Mode, the behavior of Watermarking is fundamentally different from Update Mode. While Update Mode focuses on low latency (showing you results as they change), Append Mode focuses on finality and consistency.

1. The Core Rule of Append Mode

In Append Mode, Spark will only output a row to the sink when it is certain that the result for that window will never change again.

Since late data could arrive at any moment and change the count, Spark uses the Watermark as a "guarantee" of completion. A window is only finalized and "appended" to the result table once the Watermark has moved past the end of that window.

2. How the Process Works

Let’s use the example: 10-minute windows with a 10-minute watermark.

Data Arrives: Events with timestamps between 12:00 and 12:10 arrive.

Internal State: Spark calculates the counts internally but does not output anything to the sink yet.

Watermark Advances: The engine sees a new event with a timestamp of 12:20.

The Trigger: The Watermark is now 12:20−10 minutes=12:10.

Final Output: Because the Watermark (12:10) is now equal to (or greater than) the end of the 12:00–12:10 window, Spark:

Finalizes the count for that window.

Appends the result to the sink.

Drops the state from memory (cleanup).

3. Trade-offs: Latency vs. Stability

Append Mode creates a "lag" in your data equal to the window length plus the watermark threshold.

Feature Append Mode Detail

Output Latency High. You must wait for the "late threshold" to pass before seeing data.

Record Uniqueness Guaranteed. Each window/key pair appears in your output exactly once.

Sink Compatibility Perfect for Files (Parquet/JSON) or Append-only Databases where you cannot go back and "update" a row that was already written.

Watermarking

Append mode

Structured Streaming

Outline

Programming Model
Input Sources
Starting Stream Queries
Output Modes
Output Sinks
Sample Structured Stream Processing
Operations on Streaming DataFrames
Window Operations on Event Time
Handling Late Data
Watermarking
Types of Time Window

In Spark Structured Streaming, time windows are used to segment an infinite stream of data into finite chunks for aggregation. Depending on how you want to analyze your data, you can choose between three distinct types:

1. Tumbling Window (Fixed)

Tumbling windows are fixed-size, non-overlapping, and contiguous time intervals.

Behavior: Every event belongs to exactly one window.

When to use: When you need a simple summary of what happened in distinct blocks of time (e.g., "How many sales did we make every hour?").

Syntax: window(timestamp, "10 minutes")

2. Sliding Window

Sliding windows are also fixed-size, but they can overlap. They are defined by two parameters: the window duration (how long the window is) and the slide interval (how often a new window starts).

Behavior: If the slide interval is smaller than the duration, an event can belong to multiple windows.

When to use: When you need a "moving" average or a smoother trend (e.g., "What is the average CPU usage over the last 10 minutes, updated every 5 minutes?").

Syntax: window(timestamp, "10 minutes", "5 minutes")

3. Session Window

Session windows have dynamic sizes and are based on periods of activity. They do not have a fixed start or end time.

Behavior: A session window starts when an event arrives and stays open as long as new events keep arriving within a specified inactivity gap. If no data arrives for longer than that gap, the window closes.

When to use: When events are naturally grouped by user activity (e.g., "Track a user's website visit from the moment they land until they haven't clicked anything for 30 minutes").

Syntax: session_window(timestamp, "30 minutes")

Comparison Table

Feature Tumbling Sliding Session

Window Size Fixed Fixed Dynamic

Overlapping? No Yes No

Interval Regular Regular Irregular (Data-driven)

Event Mapping 1 Event → 1 Window 1 Event → Multiple 1 Event → 1 Window

Types of Time Window

Spark supports three types of time windows: tumbling (fixed), sliding and session

In data processing and stream analytics (like Apache Flink, Spark, or Kafka Streams), Time Windows are used to bucket continuous data into manageable chunks for analysis.

Here is a breakdown of the three primary types of windows based on your description:

1. Tumbling Windows

Tumbling windows are the simplest form of windowing. They segment a data stream into fixed-size, non-overlapping, and contiguous time intervals.

Behavior: Once a window ends, a new one begins immediately.

Key Characteristic: Each data point belongs to exactly one window.

Use Case: Calculating total sales per hour where you don't want any data counted twice.

2. Sliding Windows

Sliding windows are also fixed-sized, but they add a "slide" parameter that defines how frequently a new window starts.

Behavior: If the slide duration is smaller than the window size, the windows overlap.

Key Characteristic: A single data point can be bound to multiple windows.

Use Case: Calculating a "moving average." For example, a 10-minute window that recalculates every 5 minutes.

Shutterstock

3. Session Windows

Unlike the first two, Session windows do not have a fixed duration. Their size is dynamic and determined by the arrival of data.

Behavior: A window opens when an input arrives. It stays open as long as new data points arrive within a specified Gap Duration.

The "Gap": If no data is received for a period longer than the gap duration, the window closes.

Key Characteristic: They are ideal for activity that comes in "bursts" followed by periods of silence.

Use Case: Tracking user behavior on a website. The session stays active while the user clicks links but "times out" after 30 minutes of inactivity.

Types of Time Window

Tumbling windows are a series of fixed-sized, non-overlapping and contiguous time intervals
Sliding windows are “fixed-sized”, but windows can overlap if the duration of slide is smaller than the duration of window, and in this case an input can be bound to the multiple windows
Tumbling and sliding window use window function
Session windows have have a dynamic size of the window length, depending on the inputs
A session window starts with an input, and expands itself if following input has been received within gap duration
For static gap duration, a session window closes when there’s no input received within gap duration after receiving the latest input

Building on your code snippet, it appears you are looking at Spark Structured Streaming syntax. Session windows are unique because they don't have a pre-defined start and end time; they are "data-driven."

Here is an explanation of how Session Windows function, specifically focusing on the dynamic gap features you mentioned.

1. The Session Window Mechanism

A session window groups events that occur close together in time. Unlike Tumbling or Sliding windows, these are defined by periods of activity separated by periods of inactivity (gaps).

The "Gap" Duration: In your code, "5 minutes" is the threshold. If a user triggers an event, the window stays open. If 5 minutes pass without another event from that same user, the session closes.

Watermarking: The .withWatermark("timestamp", "10 minutes") is crucial here. It tells Spark how long to wait for "late" data before finalizing the session and dropping it from memory.

2. Dynamic Gap Duration

The most advanced form of session windowing is when the gap is not a static string (like "5 minutes") but an expression based on the data itself.

How it works:

Instead of a fixed 5-minute rule for everyone, you can vary the gap based on the input row (e.g., a "Premium" user might get a longer session timeout than a "Guest" user).

Shift in Logic: When gaps are dynamic, the window is no longer just waiting for the "latest input + gap."

The "Union" Rule: Spark treats each event as having its own potential range: [startTime,startTime+dynamicGap]. If these ranges overlap, Spark unions them into a single session.

Note: With dynamic gaps, the closing of the window depends on the collective "coverage" of all events in that session, rather than just a timer starting after the very last event.

3. Implementation Example

In Spark, you would implement the dynamic gap by passing a column expression to the session_window function instead of a hardcoded string.

from pyspark.sql import functions as F

# Assume 'userType' determines the gap

# Premium users get 10 mins, others get 5 mins

dynamic_gap = F.when(events.userType == "premium", "10 minutes") \

.otherwise("5 minutes")

sessionizedCounts = events \

.withWatermark("timestamp", "20 minutes") \

.groupBy(

F.session_window(events.timestamp, dynamic_gap),

events.userId

).count()

Types of Time Window

Session window uses session_window function

    
events = ...  # streaming DataFrame of schema {timestamp: Timestamp, userId: String}

     
sessionizedCounts = events.withWatermark("timestamp", "10 minutes").groupBy(
                                  session_window(events.timestamp, "5 minutes"),
                                  events.userId).count()

It is possible to provide an expression to specify gap duration dynamically based on the input row
With dynamic gap duration, the closing of a session window does not depend on the latest input anymore
A session window’s range is the union of all events’ ranges which are determined by event start time and evaluated gap duration during the query execution

This example moves into the most advanced form of windowing in Spark Structured Streaming: Dynamic Session Windows.

In our code, the "inactivity gap" is no longer a constant value for the entire stream. Instead, it is evaluated per row based on the userId.

1. The Concept of "Range Union"

When you use a dynamic gap, Spark calculates a potential range for every single incoming event.

Individual Event Range: For any event i, its range is [timestampi,timestampi+gapi].

The Union Rule: If the range of Event A and the range of Event B overlap, Spark merges them into a single session.

Result: A session window stays open as long as there is a continuous "chain" of overlapping ranges.

2. Code Breakdown

In your specific snippet, the system treats users with different "patience" levels or session requirements:

session_window = session_window(events.timestamp,

sf.when(events.userId == "user1", "5 seconds")# Very short gap

.when(events.userId == "user2", "20 seconds") # Medium gap

.otherwise("5 minutes") # Default long gap

)

How this executes:

User1: If User1 triggers an event at 12:00:00, their session range is 12:00:00 - 12:00:05. If the next event is at 12:00:06, it starts a new session because the ranges don't overlap.

User2: If User2 triggers an event at 12:00:00, their range is 12:00:00 - 12:00:20. They have more "breathing room" to keep the session alive.

Default Users: Anyone else gets a 5-minute gap. Their sessions will likely be much larger and contain many more events.

3. Why "Closing" Logic Changes

As you noted in your text, with dynamic gaps, the closing of the window no longer depends solely on the latest input.

In a static window, you just wait for Last_Event + Fixed_Gap. In a dynamic window, an event that occurred earlier in the sequence might have a massive gap that "covers" later events, or a late event might have a very small gap that doesn't extend the session at all. The window boundary is determined by the maximum extension provided by any event within that merged group.

Types of Time Window

A session window’s range is the union of all events’ ranges which are determined by event start time and evaluated gap duration during the query execution

from pyspark.sql import functions as sf

    
events = ...  # streaming DataFrame of schema { timestamp: Timestamp, userId: String }

session_window = session_window(events.timestamp,
sf.when(events.userId == "user1", "5 seconds")
  .when(events.userId == "user2", "20 seconds")
  .otherwise("5 minutes"))

      
sessionizedCounts = events \
    .withWatermark("timestamp", "10 minutes") \
    .groupBy(
        session_window,
        events.userId) \
    .count()

References

Structured Streaming Programming Guide
Spark SQL, DataFrames and Datasets Guide
Perrin J-G., Spark in Action, 2nd ed., Manning Publications Co. 2020