23 Spark Stream Processing (DStreams)

Spark Stream Processing (DStreams)

Outline

Spark Stream Processing Modules
Stream Processing
Sample Stream Processing

The shift from Spark Streaming to Structured Streaming was primarily about moving from "functional" programming to "declarative" programming.

1. Spark Streaming (DStreams)

The Concept: It treats live data as a sequence of small batches called Micro-batches.

The Unit: Based on RDDs (Resilient Distributed Datasets).

The Pain Point: Developers had to manage time intervals manually, and it was difficult to synchronize streaming data with static historical data.

2. Structured Streaming

The Concept: It treats a live data stream as an Unbounded Table that is continuously appended to.

The Unit: Based on the Dataset/DataFrame API.

The Advantage: It uses the Catalyst Optimizer (the same engine behind Spark SQL) to optimize queries. It also handles "event-time" processing and late-arriving data much more gracefully than RDDs ever could.

Why Spark 3.5.1 is Great for This

In version 3.5.1, you benefit from several advanced features that weren't available in the early days:

Enhanced Python Support: Significant performance improvements for PySpark users via Pandas UDFs and Arrow.

RocksDB State Store: Better management of "state" (like keeping track of a running count) for very large streams.

Trigger.AvailableNow: A smart way to process all available data in a batch-like fashion but using the streaming API, which is great for cost-saving in cloud environments.

A Quick Example

To see it in action on your VM, you can use a simple socket source to simulate a stream. In a PySpark shell, the code looks like this:

Python

# Read streaming data from a port

lines = spark.readStream.format("socket").option("host", "localhost").option("port", 9999).load()

# Perform a simple transformation

words = lines.selectExpr("explode(split(value, ' ')) as word")

wordCounts = words.groupBy("word").count()

# Write the output to the console

query = wordCounts.writeStream.outputMode("complete").format("console").start()

query.awaitTermination()

Spark Stream Processing Modules

Stream processing is a key requirement in many big data applications
As soon as an application computes something of value, for example, a report or a machine learning model, an organization may want to compute this result continuously in a production environment
This capability is lacked in Hadoop MapReduce framework due to slowness of hard-disk IO
In-memory computation implemented in Spark make stream processing possible
Spark Streaming based on its low-level API Resilient Distributed Dataset is available since Spark 1.2
Spark Structured Streaming based on the Spark SQL engine is available since Spark 2.1
Luckily, our VM has installation of Spark 3.5.1

Spark Stream Processing (DStreams)

Stream processing is the practice of taking action on data while it is in motion. Unlike traditional methods that store data first and ask questions later, stream processing computes results (like averages, alerts, or model updates) incrementally as each new record arrives.

2. Real-World Data Sources

Modern organizations generate "unbounded" data (data that has no defined end) from various sources:

Financial Transactions: Monitoring credit card swipes or stock trades to detect fraud or market shifts within milliseconds.

Web Clickstreams: Tracking user behavior on a website (clicks, scrolls, hovers) to provide instant product recommendations.

IoT & Sensor Networks: Readings from smart thermostats, factory machinery, or wind turbines to predict hardware failure before it happens.

Scientific Observations: Real-time data from telescopes or particle accelerators that produce more data than can be stored long-term.

Log Files: Server and application logs that security teams monitor for hacking attempts or system crashes.

3. Comparison: Stream vs. Batch Processing

The choice between these two depends on how fast you need the answer and how much data you are looking at.

Feature Batch Processing Stream Processing

Data Scope Bounded: Processes a fixed, historical set of data (e.g., "All of October"). Unbounded: Processes data continuously as it is generated with no end.

Latency High: Minutes, hours, or days (e.g., generating a monthly payroll). Low: Milliseconds to seconds (e.g., a real-time fraud alert).

Computation Stateful/Global: Can easily look at the entire dataset at once. Incremental: Updates the current result based on only the newest data.

Hardware Use Disk-Heavy: Often reads/writes to HDFS or S3 (classic Hadoop style). Memory-Heavy: Relies on RAM for speed (modern Spark/Flink style).

Complexity Lower: Easier to handle errors by simply re-running the job. Higher: Must handle out-of-order data and system failures in real-time.

The "Golden Rule" of Choosing

Use Batch if you need a deep, exhaustive analysis of history where time isn't the primary factor (e.g., "What were our total profits last year?").

Use Stream if the value of the data diminishes quickly over time (e.g., "Is this specific login attempt a hack occurring right now?").

Stream Processing

Stream processing is the act of continuously incorporating the new data in the stream to compute a result
Sample sources of streams
- Bank transactions
- Clicks on a website
- Sensor readings from IoT devices
- Scientific observations and experiments
- Manufacturing processes, and the others
Stream processing vs. batch processing
- Batch processing runs to a fixed set of data, but stream processing handles an unbounded set of data
- Batch processing has low timeliness requirement, but stream processing requires to work at near realtime

1. Notifications and Alerting

This is the most time-critical use case. Instead of finding out a server went down an hour ago via a batch log, stream processing allows you to detect the "event" the millisecond it happens.

Example: A bank's security system detects a transaction from a country where the user isn't located and instantly sends a "Push Notification" to the user's phone to verify the purchase.

2. Real-Time Reporting

Traditional Business Intelligence (BI) often relies on "Yesterday's Data." Real-time reporting provides a live "heartbeat" of the business.

Example: A live dashboard for a ride-sharing app (like Uber) showing exactly how many drivers are active in downtown London vs. the current demand.

How it works: Data is aggregated in "windows" (e.g., every 5 minutes) and pushed to a dashboarding tool like Grafana or Power BI.

3. Incremental ETL (Extract, Transform, Load)

In the past, ETL was a massive "Nightly Batch" that slowed down databases. Incremental ETL processes data as it arrives, spreading the workload throughout the day and keeping the data warehouse always up to date.

Example: Taking raw JSON logs from a web server, cleaning the data, and loading it into a structured SQL table every few seconds.

Benefit: No more "maintenance windows" where the database is slow due to heavy batch loading.

4. Update Data to Serve in Real Time

This involves maintaining a "current state" that other applications can query instantly.

Example: An e-commerce site showing "Only 3 items left in stock!" As soon as someone buys an item, the stream processor updates the inventory count in the serving database so the next user sees the correct number.

5. Real-Time Decision Making

This moves beyond just alerting a human; it allows the system to make a choice instantly.

Example: Dynamic Pricing. An airline or hotel booking site sees a sudden spike in searches for a specific date and automatically raises the price by $10 in real-time to maximize revenue.

6. Online Machine Learning

Most ML models are "Offline" (trained once and deployed). Online ML allows a model to learn or adapt as new data comes in.

Example: A recommendation engine that notices you just clicked on three "hiking boot" videos and immediately changes your "Recommended for You" section to show outdoor gear, rather than waiting for a weekly model refresh.

Stream Processing

Use cases of stream processing
- Notifications and alerting
- Real-time reporting
- Incremental ETL
- Update data to serve in real time
- Real-time decision making
- Online machine learning

1. Single Values vs. Pattern Detection

The logic required for a single data point is very different from the logic required for a sequence.

Single Values (The "Threshold" Problem): When you receive {value: 5}, the system only needs to look at that one record.

Action: Simple filtering. "If value > 4, send alert."

Patterns (The "Stateful" Problem): To detect the pattern 2 -> 10 -> 5, the system must remember that it saw a 2, then wait for a 10, then wait for a 5.

Challenge: This requires State Management. The system must store "partial matches" in memory. If the system crashes, it needs to recover exactly where it was in that sequence.

2. The "Event Time" vs. "Processing Time" Challenge

Look closely at your data sequence:

time: 01:00:00 (Value: 2)

time: 02:00:00 (Value: 5)

time: 01:30:00 (Value: 10) <-- Out of Order!

In a perfect world, the 10 should have arrived before the 5.

The Problem: If your application calculates a "running average" for every hour, it might have already finished the 01:00:00 - 02:00:00 window by the time the 10 arrives.

The Solution: Spark Structured Streaming uses Watermarking. It tells the engine, "Wait for late data for up to X minutes before finalizing the result."

3. Operational Challenges

Beyond the data itself, the distributed nature of Spark introduces "Big Data" headaches:

Fault Tolerance & State LossWhat if the worker node holding the "memory" of your 2 -> 10 pattern crashes?

The Fix: Spark uses Checkpointing. It periodically saves the "state" and the "read position" (offset) to reliable storage (like HDFS or S3). If a failure occurs, Spark restarts and reloads the exact state from the last checkpoint.

Load Imbalance (Data Skew)

If you are processing sensors from 1,000 factories, but one factory is sending 100x more data than the others, one Spark worker will be overwhelmed while others sit idle.

The Fix: This requires careful Partitioning strategies to ensure the work is distributed evenly across the cluster.

Downstream Signaling (End-of-Event)

How does a dashboard know that the "hourly report" is final if data can always arrive late?

The Fix: Using the Watermark mentioned above. Once the system clock passes the "Watermark" threshold, Spark "emits" the final result to the downstream consumer (like a database or an email trigger), signaling that no more updates for that time period are expected.

Stream Processing

To see the challenges of stream processing, we consider the following example

Suppose we received the following data from a sensor


{value: 1,  time: "2017-04-07T00:00:00"}
{value: 2,  time: "2017-04-07T01:00:00"}
{value: 5,  time: "2017-04-07T02:00:00"}
{value: 10, time: "2017-04-07T01:30:00"}
{value: 7,  time: "2017-04-07T03:00:00"}

What actions should be performed when receiving single values, say, 5 ?
How to react to a pattern, say, 2 -> 10 -> 5
What if data arrives out-of-order, for example, 10 before 5
Other issues: What if a machine in the system fails, losing some state? What if the load is imbalanced? How can an application signal downstream consumers when analysis for some event is done, and so on

1. Event Time vs. Processing Time

As you noted with the sensor data, the time a sensor recorded an event (Event Time) is often different from the time the server received it (Processing Time).

The Challenge: If data arrives 10 minutes late due to a network glitch, the system must "reach back in time" to update the correct aggregate.

The Tool: Watermarking allows the system to track how far "behind" the data is and decide when to stop waiting for late arrivals.

2. Maintaining Large Amounts of State

If you are calculating the "running average" for 1 million sensors, Spark has to keep the current sum and count for every single sensor in memory.

The Challenge: What happens if the state (RAM) grows larger than the available memory?

The Tool: Spark uses State Stores (like RocksDB) to spill state to disk efficiently when it doesn't fit in RAM.

3. "Exactly-Once" Guarantees

If a machine fails halfway through processing a batch, you don't want to count the same transaction twice when the machine restarts.

The Challenge: Ensuring that even if a system crashes, the final output looks as if every piece of data was processed exactly one time.

The Tool: Spark achieves this via Checkpointing and Idempotent Sinks, which track the "offset" (position) of the data stream.

4. Handling Load Imbalance and Stragglers

In a distributed cluster, one "worker" might be slower than the others (a "straggler"), or one specific sensor might send way more data than others (skew).

The Challenge: The entire pipeline can only move as fast as its slowest worker.

The Tool: Dynamic Allocation and clever partitioning help redistribute the work.

5. Joins and External Data

Streaming data is often "thin"��for example, a sensor ID and a temperature. To make it useful, you might need to join it with a "thick" database that contains the sensor's location and owner.

The Challenge: Joining a fast-moving stream with a static database without slowing down the stream to a crawl.

The Tool: Stream-Static Joins allow Spark to cache the reference data and join it in-memory.

6. Transactional Writes to Output Sinks

When you write the results to a database (the "Sink"), you want to ensure that the database doesn't see partial or "dirty" results if the stream job fails mid-write.

The Challenge: Coordinating the stream's "checkpoint" with the database's "commit."

The Tool: ForeachBatch and specialized connectors (like the Kafka or Delta Lake sinks) support ACID transactions to ensure data integrity.

Stream Processing

Main challenges of stream processing are the following
- Processing out-of-order data based on application timestamps (also called event time)
- Maintaining large amounts of states
- Supporting high-data throughput
- Processing each event exactly once despite machine failures
- Handling load imbalance and stragglers
- Responding to events at low latency
- Joining with external data in other storage systems
- Determining how to update output sinks as new events arrive
- Writing data transactionally to output systems
- Updating application business logic at runtime

Spark Stream Processing (DStreams)

Outline

Spark Stream Processing Modules
Stream Processing
Sample Stream Processing

1. The StreamingContext

The StreamingContext is the main entry point for Spark Streaming.

local[2]: This tells Spark to run locally with two threads. In streaming, you must have at least two threads: one to "listen" to the socket and one to "process" the data.

batch interval (1): The 1 means Spark will collect data every 1 second and turn it into a mini-batch (an RDD).

2. Creating the Input DStream

lines = ssc.socketTextStream("localhost", 9999)

This line tells Spark to open a connection to your VM's local port 9999.

Reality Check: If you run this code without an active server on port 9999, it will keep trying to connect. (You can use the terminal command nc -lk 9999 to start a sender).

3. Transformation: flatMap

words = lines.flatMap(lambda line: line.split(" "))

This is where the "logic" happens.

lines: This is a DStream, which is essentially a continuous series of RDDs.

flatMap: Unlike a regular map, which returns one item per input, flatMap can return multiple items. If a line is "Hello Spark Streaming", the split creates a list, and flatMap "flattens" that list into three separate word records.

4. How this provides "Fast, Scalable, Fault-Tolerant"

Snippet is the start of a pipeline that Spark manages automatically:

Fast (Micro-batching): By processing data every 1 second, Spark achieves "near real-time" latency.

Scalable: If you move this from your VM to a cluster, Spark will automatically distribute the words across different worker nodes to process them in parallel.

Fault-Tolerant: If a worker node crashes while splitting words, Spark knows which batch was lost and re-runs the computation on another node using the original data from the socket (if the source supports replay) or from a checkpoint.

Exactly-Once: In modern Spark (especially with Structured Streaming), the system ensures that even if a failure occurs, the final word count isn't double-counted.

# Count each word in each batch

pairs = words.map(lambda word: (word, 1))

wordCounts = pairs.reduceByKey(lambda x, y: x + y)

# Print the first ten elements of each RDD generated in this DStream to the console

wordCounts.pprint()

# Start the computation

ssc.start()

# Wait for the computation to terminate

ssc.awaitTermination()

Sample Stream Processing

Stream Processing suppose to provide fast, scalable, fault-tolerant, end-to-end exactly-once stream processing without the user having to reason about streaming
Assume that we want to count the number of words in text data received from a data server listening on a TCP socket

Import StreamingContext and create a local StreamingContext

  
from pyspark import SparkContext
from pyspark.streaming import StreamingContext
sc = SparkContext("local[2]", "NetworkWordCount")
ssc = StreamingContext(sc, 1))

Create a stream that represents streaming data from a TCP source

Split the lines by space into words

 
words = lines.flatMap(lambda line: line.split(" "))

The Word Count Logic

pairs = words.map(lambda word: (word, 1))

wordCounts = pairs.reduceByKey(lambda x, y: x + y)

map: This transforms every individual word into a tuple: (word, 1). For example, "apple" becomes ("apple", 1). This prepares the data for counting.

reduceByKey: This groups all tuples with the same word (the key) and adds their values together. If you have three instances of ("apple", 1), this function reduces them to ("apple", 3).

2. Output and Visualization

wordCounts.pprint()

pprint(): This is a debug-friendly method that prints the first ten elements of every batch of data to the console. It allows you to see the results of your computation in real-time as the stream flows in.

Execution Control

ssc.start()

ssc.awaitTermination()

ssc.start(): In Spark Streaming, nothing actually happens until you call this. It tells the Streaming Context to begin receiving and processing data.

ssc.awaitTermination(): This keeps the script running. Without it, the Python script would finish and exit immediately after starting, killing the background stream. This ensures the app stays alive until you manually stop it (e.g., via Ctrl+C).

Sample Stream Processing

Count the words

  
pairs = words.map(lambda word: (word, 1))
wordCounts = pairs.reduceByKey(lambda x, y: x + y)

Print the first ten elements of each RDD to the console
Start the computations
Wait for the computation to terminate
Finally, we save an application in a file dstream.py

This process describes how to test a Spark Streaming application locally using a Client-Server model. You are essentially setting up one terminal to "talk" and another terminal to "listen" and process data.

1. Setting up the Data Source (The Server)

nc -lk 9999

nc (Netcat): This utility is used to send data over network connections.

-l (Listen): Tells Netcat to act as a server and wait for a connection.

-k (Keep-alive): Forces Netcat to stay permit multiple connections and stay open even after a client disconnects.

9999: This is the port number. Netcat will "broadcast" whatever you type in this terminal to this specific port.

2. Feeding Data into the Stream

aaa aaa aaa aa aa

Once Netcat is running, anything you type and enter into that terminal becomes the "Live Data." In this case, you are feeding the string "aaa aaa aaa aa aa" into the stream for Spark to count.

3. Running the Spark App (The Client)

$SPARK_HOME/bin/spark-submit dstream.py localhost 9999

spark-submit: This is the command-line tool used to launch Spark applications.

dstream.py: This is the Python file containing the word count logic we discussed previously.

localhost 9999: These are arguments passed to your script. They tell the Spark application to look at your own machine (localhost) on port 9999 to find the data stream.

How the Workflow Looks

The relationship between these two terminals is a continuous loop:

Terminal 1 (Netcat) Direction Terminal 2 (Spark)

Acts as the Producer �� Acts as the Consumer

You type: aaa bbb �� Processes: (aaa, 1), (bbb, 1)

Holds the connection open ? Listens for new batches of text

Sample Stream Processing

The input is simulated by Netcat (a small utility found in most Unix-like systems) as a data server
In a different Terminal, we start Spark-shell and input the Python code from the previous slides
Next, we input in the first Terminal window

Then we run the application in the second Terminal window

  	    
$SPARK_HOME/bin/spark-submit dstream.py localhost 9999

This output is the visual result of the Micro-Batching architecture used by Spark Streaming. It shows exactly how Spark "slices" the continuous stream of data into small, manageable chunks of time.

Here is a breakdown of what that output tells you:

1. The Micro-Batch Timeline

The dashed lines represent the boundaries of each batch interval. Even if no data is being typed in the Netcat terminal, Spark continues to create these batches based on a set frequency (usually 1 second).

19:07:34: The stream is active, but no data was received during this specific second. The output is empty.

19:07:35: This is where the "action" happened. Spark collected the text you typed (aaa aaa aaa aa aa), processed it, and printed the result.

19:07:36: The system moves to the next interval, waiting for more input.

2. Interpreting the Results

Inside the 19:07:35 batch, you see the final result of your reduceByKey operation:

('aaa', 3): Spark found three instances of "aaa".

('aa', 2): Spark found two instances of "aa".

Sample Stream Processing

Then we could see the following output in application Terminal window

 
-------------------------------------------
Time: 2026-05-23 19:07:34
-------------------------------------------

-------------------------------------------
Time: 2026-05-23 19:07:35
-------------------------------------------
('aaa', 3)
('aa', 2)

-------------------------------------------
Time: 2026-05-23 19:07:36
-------------------------------------------

References

Spark Streaming Programming Guide
RDD Programming Guide
Perrin J-G., Spark in Action, 2nd ed., Manning Publications Co. 2020
Chambers B., Zaharia M.,Spark: The Definitive Guide, O'Reilly 2017