22 Spark Operations

Spark Operations

Outline

Quick Start
Self Contained Application
Web User Interface
Operations on Resilient Distributed Datasets (RDDs)
Operations on DataFrames
SQL Module

pyspark: This is the command you executed. It tells your system to start the Python version of the Spark shell. By default, it runs Spark in a standalone mode (locally on your machine) suitable for quick testing and development.

Python Environment

Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux: This confirms the specific Python interpreter that PySpark is using. It shows the version (3.10.12), the build details, and the operating system (Linux).

Spark Configuration and Banner

Setting default log level to "WARN".: This is a standard initial message. PySpark is telling you that it will only display log messages that are classified as WARNING or more severe (ERROR, FATAL). This keeps the console less cluttered.

Welcome to / The ASCII Art Banner: This is the official Apache Spark welcome graphic.

version 3.5.1: This clearly indicates the version of Apache Spark that has been initialized.

Active Spark Contexts

These lines are the most critical, as they confirm that the necessary Spark objects have been automatically created and are ready for you to use.

Spark context Web UI available at http://10.0.2.15:4040 Spark launches a small web server (Web UI) that you can access in a browser to monitor the status of your jobs, check storage usage, and view the execution timeline. The address provided is where you can find it.

Spark context available as 'sc' (master = local[*], app id = local-1713320306456).:

sc: This is the pre-defined variable name for the SparkContext. The SparkContext is the main entry point for all Spark functionality in older versions, and it's how Spark connects to a cluster.

master = local[*]: This confirms that Spark is running in local mode (standalone), using all available cores on your machine (*) for parallel processing.

app id: A unique ID automatically generated for this specific interactive session.

SparkSession available as 'spark'.:

spark: This is the pre-defined variable name for the SparkSession. Since Spark 2.0, this has been the unified entry point for all Spark functionality (SQL, streaming, and core RDD operations) and is the primary object you'll use to start your work.

In summary, the output confirms that Apache Spark (v3.5.1) is now running in a local, interactive Python environment, and you can immediately begin typing PySpark commands using the spark and sc variables.

Quick Start

To open Python version of Spark shell in standalone mode process the following command

  
pyspark

 
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). 
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.5.1
      /_/

Using Python version 3.10.12 (main, Nov 20 2023 15:14:05)
Spark context Web UI available at http://10.0.2.15:4040
Spark context available as 'sc' (master = local[*], app id = local-1713320306456).
SparkSession available as 'spark'.

The line "A SparkSession instance is an entry to a Spark application" is the key concept.

The spark Variable

When you are in the PySpark shell, the variable spark is automatically created for you.

This output confirms two things:

spark is not an empty variable; it holds a value.

That value is an instance of the SparkSession class from the pyspark.sql.session module. The hexadecimal address (0x726c633ca740) is just the memory location of that object.

The SparkSession is the unified starting point for everything you do with Spark, replacing the need to separately create a SparkContext, SQLContext, and HiveContext.

Developing the Data Processing Pipeline

The second part of the example demonstrates how you use the spark variable to start your data processing work.

Creating a DataFrame

range1000 = spark.range(1000).toDF("number")

spark.range(1000): This is a method of the SparkSession that efficiently creates a simple DataFrame containing a single column of numbers from 0 up to (but not including) 1000.

.toDF("number"): This method takes the default column name of the newly created DataFrame and explicitly renames it to "number".

range1000: This is the Python variable that now holds the resulting DataFrame object. A DataFrame in Spark is a distributed collection of data organized into named columns, conceptually equivalent to a table in a relational database.

Viewing the Data

range1000.show(2)

.show(2): This is an action on the DataFrame. It instructs Spark to display the first 2 rows of the data in a neatly formatted, tabular output. This is a common way to quickly inspect data.

The output is the visual representation of the first two rows of the DataFrame named range1000.

Quick Start

A SparkSession instance is an entry to a Spark application
If you type spark in the spark-shell interface then you get the following messages
You can use SparkSession instance spark to interact with Spark and to develop your data processing pipeline

For example,

   
range1000 = spark.range(1000).toDF("number")

range1000.show(2)
+------+                                                                        
|number|
+------+
|     0|
|     1|
+------+
only showing top 2 rows

This code snippet demonstrates a quick start example of processing a file named README.txt using Apache Spark's Python API (PySpark), likely within an interactive shell like the Spark Shell or a Jupyter Notebook.

Code Line Explanation

lines =sc.textFile('README.txt') This is the initial loading step. It instructs Spark to read the file README.txt from a specific location on the Hadoop Distributed File System (HDFS) (indicated by the hdfs:// path and port 9000). The loaded data is stored in a Spark data structure called a Resilient Distributed Dataset (RDD), which is assigned to the variable lines. sc stands for SparkContext, the entry point to Spark functionality.

lines.first() This is an Action that retrieves and displays the first line of the RDD.

'For the latest information about Hadoop, please visit our website at:' This is the output from the lines.first() command, showing the content of the first line.

lines.count() This is an Action that calculates and returns the total number of lines (elements) in the RDD.

7 This is the output from the lines.count() command, indicating the file has 7 lines.

lines.filter(lambda x: "apache" in x).collect()

This performs a Transformation followed by an Action:

1. filter(...): This Transformation creates a new RDD containing only the lines (x) that have the substring "apache" in them. The lambda function defines the filtering logic.

2. .collect(): This Action sends the resulting data from the distributed Spark cluster back to the local machine (the client program/shell) as a standard Python list, allowing it to be displayed.

['http://hadoop.apache.org/', 'https://cwiki.apache.org/confluence/display/HADOOP/'] This is the output from the combined filter and collect operations, showing the two lines in the file that contain the word "apache."

Summary of Concepts

SparkContext (sc): The main connection point to the Spark cluster.

RDDs (lines): Spark's fundamental data structure for resilient, fault-tolerant, and parallel processing of data.

Transformations (.filter()): Operations that create a new RDD from an existing one (e.g., mapping, filtering). These are lazy and only execute when an Action is called.

Actions (.first(), .count(), .collect()): Operations that trigger the computation of transformations and return a result back to the driver program or write data to an external storage system.

Quick Start

Sample processing of a file README.txt

       
lines =sc.textFile('README.txt')

   
lines.first()

 
'For the latest information about Hadoop, please visit our website at:'

   
lines.count()

		
lines.filter(lambda x: "apache" in x).collect()

	      
['http://hadoop.apache.org/', 'https://cwiki.apache.org/confluence/display/HADOOP/']

The first part is a repeat of the setup from the quick start, followed by an Action to view the entire RDD content.

lines = sc.textFile("https://url.au.m.mimecastprotect.com/s/jrFwCYW8EEtLDoj2lsVs1FxWIoc?domain=127.0.0.1"): This loads the file into an RDD named lines, where each element is a line of the file.

lines.collect(): This is an Action that retrieves all elements (lines) from the distributed RDD back to the local driver program and presents them as a standard Python list.

Output: ['For the latest information about Hadoop, please visit our website at:','', 'http://hadoop.apache.org/, '', 'and our wiki, at:', '', ]

Word Counting Operations

The core of this example is the two-step process to count the occurrences of each word in the file.

1. The Transformation: flatMap

counters = lines.flatMap(lambda x: x.split()).countByValue()

flatMap(...) (Transformation): This is the key step. It applies a function (lambda x: x.split()) to every element (x is a line) in the RDD, and then flattens the result.

The x.split() part splits each line into a list of words.

Since it's flatMap, if the input line is "hello world", x.split() returns ['hello', 'world']. flatMap then combines all these lists into one single RDD of words.

The resulting intermediate RDD contains one element for every single word in the file.

2. The Action: countByValue

.countByValue() (Action): This is a specialized action on RDDs of key-value pairs (or, in this case, simply values) that returns the count of every unique element in the RDD.

It executes the necessary distributed counting and returns the result to the driver program as a standard Python dictionary (or collections.Counter object).

The result is stored in the counters variable, where keys are the words and values are their counts.

Output and Interpretation

The subsequent for loop is simply iterating through the resulting counters dictionary to print the word and its count.

Python

for i, (word, count) in enumerate(counters.items()):... print(word,count)

Output Snippet Interpretation:

Word Count Meaning

For 1 The word "For" appears once.

the 1 The word "the" appears once.

our 2 The word "our" appears twice (once in the first line: "...visit our website..." and once in the fifth line: "...and our wiki...").

Quick Start

More operations on RDD

   	    
lines = sc.textFile('README.txt')

   	    
lines.collect()
['For the latest information about Hadoop, please visit our website at:','',
'http://hadoop.apache.org/, '', 'and our wiki, at:', '', ]

    	    	    
counters = lines.flatMap(lambda x: x.split()).countByValue()

    	    	    
for i, (word, count) in enumerate(counters.items()):
...     print(word,count)

    	    	    
For 1
the 1
latest 1
information 1
about 1
Hadoop, 1
please 1
visit 1
our 2
...    ...

Spark Operations

Outline

Quick Start
Self Contained Application
Web User Interface
Operations on Resilient Distributed Datasets (RDDs)
Operations on DataFrames
SQL Module

The SimpleApp.py script is designed to count the number of lines in a file that contain the letter 'a' and the letter 'b'.

1. Initialization and Setup

from pyspark.sql import SparkSession: Imports the essential class for modern Spark programming.

spark = SparkSession.builder.appName("SimpleApp").getOrCreate(): This is the unified entry point to all Spark functionality.

It creates a SparkSession, which manages the connection to the cluster and is configured with the name "SimpleApp."

logFile = "README.txt": Specifies the file to be processed, assumed to be accessible by the Spark cluster (in this case, on HDFS at /user/bigdata).

2. Data Loading and Caching

logData = spark.read.text(logFile).cache():

spark.read.text(logFile): Loads the text file. This creates a DataFrame (the modern successor to RDDs) where each line of the file is a row in a single column named value.

.cache(): This is a Transformation that marks the resulting DataFrame to be saved in memory after its first calculation. This is crucial for performance since the data will be accessed twice (once for counting 'a' and once for counting 'b').

3. Processing (Counting)

numAs = logData.filter(logData.value.contains('a')).count():

This uses DataFrame methods to count lines containing 'a'.

.filter(...) is a Transformation that keeps only the rows where the condition (logData.value.contains('a')) is true.

.count() is an Action that triggers the distributed computation and returns the final number of rows back to the local program.

numBs = logData.filter(logData.value.contains('b')).count(): Repeats the counting process for lines containing 'b'.

print(...): Prints the final results.

4. Cleanup

spark.stop(): Shuts down the SparkSession and releases the cluster resources used by the application.

Execution and Results

The self-contained application is launched using the spark-submit utility, which correctly packages the application and submits it to the cluster manager.

Execution Command

$SPARK_HOME/bin/spark-submit --master local[4] SimpleApp.py

spark-submit: The utility for launching applications.

--master local[4]: Tells Spark to run in local mode on the current machine using 4 CPU cores. This is used for development and testing.

Output Verification

Application Output: Lines with a: 4, lines with b: 1

Source Data Check: The hdfs dfs -cat command confirms the file contents:

Lines containing 'a': 4 (The 1st, 3rd, 5th, and 7th lines all contain 'a' in words like 'latest', 'Hadoop', 'apache', 'at', 'wiki', etc.)

Lines containing 'b': 1 (Only the 1st line contains 'b' in the word 'about').

This entire process is the standard way to deploy and run a production-ready, distributed data processing application using PySpark.

Self-Contained Application

A sample self-contained application SimpleApp.py

""SimpleApp.py"""
from pyspark.sql import SparkSession

logFile = "README.txt" # In home folder '/user/bigdat/README.txt')
spark = SparkSession.builder.appName("SimpleApp").getOrCreate()

logData = spark.read.text(logFile).cache()
numAs = logData.filter(logData.value.contains('a')).count()
numBs = logData.filter(logData.value.contains('b')).count()
print("Lines with a: %i, lines with b: %i" % (numAs, numBs))
spark.stop()

Processing of an application SimpleApp.py

$SPARK_HOME/bin/spark-submit --master local[4] SimpleApp.py

Lines with a: 4, lines with b: 1

hdfs dfs -cat /user/bigdata/README.txt

For the latest information about Hadoop, please visit our website at:
http://hadoop.apache.org/
and our wiki, at:
https://cwiki.apache.org/confluence/display/HADOOP/

This example demonstrates another self-contained PySpark application, AnotherApp.py, which calculates the total number of characters across all lines in a file. Crucially, this script uses the older RDD API and initializes the SparkContext directly, unlike the previous example which used the newer SparkSession.

🐍 AnotherApp.py: The Application Code

This application performs three main RDD operations: load, map, and reduce.

1. Initialization (RDD API)

from pyspark import SparkConf, SparkContext: Imports the classic classes needed for RDD processing.

spark_conf = SparkConf(): Creates a configuration object.

sc = SparkContext(conf=spark_conf): Initializes the SparkContext (sc), which is the entry point for RDD-based applications.

2. Data Loading

lines = sc.textFile("README.txt"): Loads the file from the specified path (assumed HDFS) into an RDD named lines, where each element is a line of text.

3. Transformation: map

lineLengths = lines.map(lambda s: len(s)): This is a Transformation that applies a function to every element in the input RDD and creates a new RDD.

The lambda s: len(s) function takes each line (s) and returns its length (the character count).

The new RDD, lineLengths, contains only the integer lengths of the original lines.

4. Action: reduce

totalLength = lineLengths.reduce(lambda a, b: a + b): This is an Action that aggregates all elements in the RDD by applying a binary function repeatedly.

The lambda a, b: a + b function is an addition operator. Spark distributes the addition across the cluster, combining the lengths of all the lines in parallel.

The final result, a single integer representing the total character count, is returned to the driver program and stored in totalLength.

5. Cleanup

sc.stop(): Stops the SparkContext, releasing cluster resources.

Execution and Result

The application is run using spark-submit, and the output confirms the calculation.

Execution Command

$SPARK_HOME/bin/spark-submit --master local[4] AnotherApp.py

The command launches the application on the local machine using 4 CPU cores, similar to the previous example.

Output Verification

Application Output: 168

Source Data: The file content is:

For the latest information about Hadoop, please visit our website at:

https://url.au.m.mimecastprotect.com/s/P0TQC3QN11HpXYWzKsEFqFQmBGj?domain=hadoop.apache.org our wiki, at:

https://url.au.m.mimecastprotect.com/s/m2IfC1WLwwtM6wLNGF1uOFVI5lF?domain=cwiki.apache.org/

Verification of Count: The individual line lengths are:

Line 1: 75 characters

Line 2 (blank): 0 characters

Line 3: 30 characters

Line 4 (blank): 0 characters

Line 5: 17 characters

Line 6 (blank): 0 characters

Line 7: 46 characters

Total Length: 75+0+30+0+17+0+46=168. The output is correct.

This script is a classic, clear demonstration of the MapReduce programming model implemented using Spark's core RDD API.

Self-Contained Application

Another sample self-contained application AnotherApp.py

"""AnotherApp.py"""
from pyspark import SparkConf, SparkContext

spark_conf = SparkConf()
sc = SparkContext(conf=spark_conf)

lines = sc.textFile("README.txt") # In home folder  '/user/bigdat/README.txt'
lineLengths = lines.map(lambda s: len(s))
totalLength = lineLengths.reduce(lambda a, b: a + b)
print(totalLength)

sc.stop()

Processing of an application AnotherApp.py

$SPARK_HOME/bin/spark-submit --master local[4] AnotherApp.py

hdfs dfs -cat /user/bigdata/README.txt

For the latest information about Hadoop, please visit our website at:
http://hadoop.apache.org/
and our wiki, at:
https://cwiki.apache.org/confluence/display/HADOOP/

Spark Operations

Outline

Quick Start
Self Contained Application
Web User Interface
Operations on Resilient Distributed Datasets (RDDs)
Operations on DataFrames
SQL Module

The Spark Web UI is a set of monitoring dashboards automatically launched with every Spark driver program (the process that runs the main() function of your application).

Access: It is typically accessed via a web browser at port 4040 on the machine where the Spark driver is running. For example, if your Spark driver is running on localhost, you would open your browser to http://localhost:4040.

Purpose: It gives you a real-time, visual overview of the progress, status, and resource consumption of your Spark application.

Key Information Displayed

The Spark Web UI is organized into several tabs, each providing a different set of monitoring details:

Tab Description Key Information Provided

Jobs Shows a summary of all executed Spark Jobs. A Job is triggered by a Spark action (like collect() or saveAsTextFile). Job ID, description, duration, status (running, succeeded, failed), and a breakdown of Stages.

Stages Shows a list of all Stages defined in your Job(s). A Stage is a set of tasks that all compute the same function but on different partitions of data. Stage ID, status, total time, a visual DAG (Directed Acyclic Graph) of the job, and a summary of Tasks by status.

Tasks (Viewable within the Stages tab) The lowest level of execution. A Task runs on an Executor and processes a single partition of data. Task status, start time, executor ID, duration, input/output data metrics, and shuffle read/write metrics.

Executors Displays detailed information about the Spark Executors that are running the tasks across the cluster. Executor ID, Hostname, status, total cores, memory usage, disk usage, and cumulative statistics for tasks run.

Storage Shows the persistence level of RDDs and DataFrames that have been cached/persisted in memory or on disk. Cached RDD/DataFrame names, size in memory/disk, storage level, and number of partitions.

Environment Displays important configuration variables and system properties for the Spark application. Spark version, configuration parameters (e.g., spark.executor.memory), and system properties.

In short, the Spark Web UI is essential for monitoring resource usage, identifying performance bottlenecks (like data skew or slow stages), and debugging failures by examining task logs and error messages.

Web User Interface

Each driver program has a Web UI, typically on port 4040
Spark Web UI displays information about running tasks, executors, and storage usage.

Spark Operations

Outline

Quick Start
Self Contained Application
Web User Interface
Operations on Resilient Distributed Datasets (RDDs)
Operations on DataFrames
SQL Module

In Apache Spark, Resilient Distributed Datasets (RDDs) are the fundamental data structure, representing an immutable, fault-tolerant, distributed collection of objects. Operations on RDDs are performed using raw Java or Scala objects (or their Python equivalents) and are categorized into two main types:

Transformations: Create a new RDD from an existing one. They are lazily evaluated—meaning they don't execute until an action is called.

Actions: Compute a result or write to an external storage system. They trigger the execution of all pending transformations.

Example RDD Operations

The code snippetsillustrate common RDD transformations used for basic text processing.

1. Creating an RDD

The first step is reading the data into an RDD. Assuming a file named README.txt is available in the Hadoop Distributed File System (HDFS):

Bash

hdfs dfs -cat /user/bigdata/README.txt

(This command is for viewing the file content on HDFS, before creating the RDD.)

Python

lines = sc.textFile("README.txt")

Transformation: sc.textFile("README.txt") is a function on the SparkContext (sc) that reads the file and creates an RDD where each element is a line of the file. This RDD is called lines.

2. Flattening and Mapping (flatMap)

This transformation is crucial for breaking lines into individual words.

words = lines.flatMap(lambda line: line.split(' '))

Transformation: The flatMap operation applies a function (here, splitting the line by spaces) to each element of the source RDD (lines) and then flattens the result.

For example, if a line is "Hello World", line.split(' ') returns a list ["Hello", "World"].

flatMap takes all these resulting lists and combines them into a single RDD (words) containing all the individual words.

3. Eliminating Duplicates and Counting (distinct, count)

These operations are used to find the number of unique words.

distinct_words = words.distinct()

Transformation: distinct() returns a new RDD (distinct_words) containing only the unique elements from the source RDD (words).

distinct_words.count()

Action: count() is an action that returns the number of elements (the final result) in the RDD. This action forces the execution of all preceding transformations (textFile, flatMap, and distinct).

Filtering (filter)

This transformation selects a subset of elements based on a condition.

a_words = words.filter(lambda word: word.startswith("a"))

Transformation: filter() applies a Boolean condition (a function that returns True or False) to every element in the source RDD (words). It returns a new RDD (a_words) containing only the words that start with the letter "a".

Operations on Resilient Distributed Datasets (RDDs)

Operations on RDDs are performed on raw Java or Scala objects

Creating a simple RDD with words

 
hdfs dfs -cat /user/bigdata/README.txt

Creating lines and words

 
lines = sc.textFile("README.txt")
words = lines.flatMap(lambda line: line.split(' '))

Eliminating duplicates and counting words

 
distinct_words = words.distinct()
distinct_words.count()

Filtering

 
a_words = words.filter(lambda word: word.startswith("a"))

The sortBy transformation in Apache Spark is used to sort the elements of an RDD based on a custom key or an attribute derived from each element. It returns a new RDD that is globally sorted.

The core of sortBy is the lambda function (or a regular Python function) you pass to it, which dictates the value used for sorting.

1. Sorting Single Values

When sorting an RDD of simple elements (like strings or integers), the lambda function defines the sorting key.

A. Ascending Sort (Default)

Syntax: sc.parallelize([...]).sortBy(lambda x: x).collect()

When the lambda function simply returns the element itself (lambda x: x), the RDD is sorted in its natural ascending order.

Code Example Sorting Key Result

sc.parallelize(['a', 'd', 'c', 'e', 'b']).sortBy(lambda x: x).collect() The string itself ['a', 'b', 'c', 'd', 'e']

sc.parallelize([1, 2, 5, 3, 4]).sortBy(lambda x: x).collect() The integer itself [1, 2, 3, 4, 5]

B. Descending Sort using a Negative Key

Syntax: sc.parallelize([...]).sortBy(lambda x: -key).collect()

To achieve descending order, you can apply the negative sign (-) to the numeric key used for sorting.

Code Example Sorting Key Result

sc.parallelize([1, 2, 5, 3, 4]).sortBy(lambda x: -x).collect() Negative of the number [5, 4, 3, 2, 1]

sc.parallelize(['a', 'd', 'c', 'e', 'b']).sortBy(lambda x: -ord(x)).collect() Negative of the ASCII/Unicode value of the character (ord(x)) ['e', 'd', 'c', 'b', 'a']

2. Sorting Pair RDDs (Tuples)

When sorting an RDD of key-value pairs (tuples), the lambda function must specify which element of the tuple should be used as the sorting key.

Assume the RDD is composed of tuples in the format (key, value).

A. Sorting by Key (First Element: Index [0])

Python

sc.parallelize([('1', 3), ('2', 5), ('a', 1), ('b', 2), ('d', 4)]).sortBy(lambda x: x[0]).collect()

Key: lambda x: x[0] uses the first element of the tuple ('1', '2', 'a', etc.) as the sorting key.

Result (Ascending): [('1', 3), ('2', 5), ('a', 1), ('b', 2), ('d', 4)]

Note: The sort order is based on string comparison, where '1' comes before 'a'.

B. Sorting by Value (Second Element: Index [1])

sc.parallelize([('1', 3), ('2', 5), ('a', 1), ('b', 2), ('d', 4)]).sortBy(lambda x: x[1]).collect()

Key: lambda x: x[1] uses the second element of the tuple (3, 5, 1, etc.) as the sorting key.

Result (Ascending): [('a', 1), ('b', 2), ('1', 3), ('d', 4), ('2', 5)]

Operations on Resilient Distributed Datasets (RDDs)

Sorting with sortBy

 
sc.parallelize(['a', 'd', 'c', 'e', 'b']).sortBy(lambda x: x).collect()

 
sc.parallelize(['a', 'd', 'c', 'e', 'b']).sortBy(lambda x: -ord(x)).collect()

 
sc.parallelize([1, 2, 5, 3, 4]).sortBy(lambda x: -x).collect()

 
sc.parallelize([1, 2, 5, 3, 4]).sortBy(lambda x: x).collect()

Sorting pairs

	     
sc.parallelize([('1', 3), ('2', 5), ('a', 1), ('b', 2), ('d', 4)]).sortBy(lambda x: x[0]).collect()

	     
sc.parallelize([('1', 3), ('2', 5), ('a', 1), ('b', 2), ('d', 4)]).sortBy(lambda x: x[1]).collect()

The sortByKey transformation is specifically designed for Pair RDDs—RDDs composed of key-value pairs (tuples like (key, value)). It sorts the RDD entirely based on the key element of each pair.

This operation is more efficient than a generic sortBy(lambda x: x[0]) because Spark optimizes internal data shuffling based on the known structure of Pair RDDs.

1. Basic sortByKey()

The basic usage sorts the RDD in ascending order based on the natural order of the keys.

Code Example Explanation Result

sc.parallelize([('a', 1), ('b', 2), ('1', 3), ('d', 4), ('2', 5)]).sortByKey().first() Sorts by key and returns the first element. Keys are sorted as strings: '1', '2', 'a', 'b', 'd'. ('1', 3)

2. Controlling Sort Order and Parallelism

The sortByKey method accepts optional parameters to control the sort behavior:

ascending: A boolean that specifies the sort order (default is True).

numPartitions: An integer that specifies the number of partitions for the resulting RDD. This can help control parallelism (default is determined by Spark).

keyfunc: A function that can be applied to the key before sorting, allowing complex sorting logic.

The examples below show how changing the number of partitions (numPartitions) does not change the final sorted order of the RDD's collected elements; it only affects how the data is internally distributed and processed.

Code Example ascending numPartitions Result

sc.parallelize([...]).sortByKey(True, 1).collect() True (Ascending) 1 [('1', 3), ('2', 5), ('a', 1), ('b', 2), ('d', 4)]

sc.parallelize([...]).sortByKey(True, 2).collect() True (Ascending) 2 [('1', 3), ('2', 5), ('a', 1), ('b', 2), ('d', 4)]

3. Sorting with a Custom Key Function (keyfunc)

The keyfunc argument allows you to transform the key before the sort operation is applied, enabling case-insensitive or complex sorting.

In the example below, the original keys are a mix of capitalized and lowercase strings:

tmp2 = [('Mary', 1), ('had', 2), ('a', 3), ('little', 4), ('lamb', 5)]

tmp2.extend([('whose', 6), ('fleece', 7), ('was', 8), ('white', 9)])

sc.parallelize(tmp2).sortByKey(True, 3, keyfunc=lambda k: k.lower()).collect()

keyfunc=lambda k: k.lower(): This is applied to the key (k) before the comparison.

The key 'Mary' is converted to 'mary' for sorting.

The key 'a' remains 'a' for sorting.

Result (Sorted based on the lowercase key): [('a', 3), ('fleece', 7), ('had', 2), ('lamb', 5), ('little', 4), ('Mary', 1), ('was', 8), ('white', 9), ('whose',

Operations on Resilient Distributed Datasets (RDDs)

Sorting with sortByKey

  
sc.parallelize([('a', 1), ('b', 2), ('1', 3), ('d', 4), ('2', 5)]).sortByKey().first()
('1', 3)

  
sc.parallelize([('a', 1), ('b', 2), ('1', 3), ('d', 4), ('2', 5)]).sortByKey(True, 1).collect()
[('1', 3), ('2', 5), ('a', 1), ('b', 2), ('d', 4)]

  
sc.parallelize([('a', 1), ('b', 2), ('1', 3), ('d', 4), ('2', 5)]).sortByKey(True, 2).collect()
[('1', 3), ('2', 5), ('a', 1), ('b', 2), ('d', 4)]

  
tmp2 = [('Mary', 1), ('had', 2), ('a', 3), ('little', 4), ('lamb', 5)]
tmp2.extend([('whose', 6), ('fleece', 7), ('was', 8), ('white', 9)])

 
sc.parallelize(tmp2).sortByKey(True, 3, keyfunc=lambda k: k.lower()).collect()
[('a', 3), ('fleece', 7), ('had', 2), ('lamb', 5),...('white', 9), ('whose', 6)]

The final RDD operations showcase methods for distributing data for testing/sampling (randomSplit) and aggregating data into a single value (reduce).

1. Random Split (randomSplit)

The randomSplit transformation divides a single RDD into multiple RDDs, with each element being placed into one of the new RDDs based on specified weights. This is commonly used for creating training and test datasets.

rdd = sc.parallelize(range(500), 1)

rdd1, rdd2 = sc.parallelize(range(500), 1).randomSplit([2, 3], 17)

ource RDD: sc.parallelize(range(500), 1) creates an RDD with 500 integer elements (0 through 499).

Weights ([2, 3]): This is a list of weights (not percentages) defining the relative size of the resulting RDDs. The total weight is 2+3=5.

rdd1 is expected to receive roughly 2/5 (or 40%) of the data.

rdd2 is expected to receive roughly 3/5 (or 60%) of the data.

Seed (17): The seed ensures the split is reproducible. If you run the split again with the same seed, the same elements will end up in rdd1 and rdd2.

len(rdd1.collect() + rdd2.collect()) is 500. This confirms that all elements from the original RDD are distributed across the new RDDs without loss.

The count checks confirm the split is approximately 40% and 60%:

150 < rdd1.count() < 250 (Expected range around 200) ⟹ True

250 < rdd2.count() < 350 (Expected range around 300) ⟹ True

2. Reduce RDD to One Value (reduce)

The reduce action is one of the most fundamental operations in distributed computing. It aggregates all elements of the RDD using an associative and commutative binary operator to produce a single value.

A. Simple Summation

from operator import add

sc.parallelize([1, 2, 3, 4, 5]).reduce(add)

Operator: add (which is equivalent to lambda x, y: x + y).

Process: The reduce operation applies the function iteratively to pairs of elements: (1+2)→3→(3+3)→6→(6+4)→10→(10+5)→15.

Result: 15 (The sum of all elements).

B. Reduction on a Transformed RDD

This example is slightly more complex, involving transformations and caching before the reduction.

sc.parallelize((2 for _ in range(10))).map(lambda x: 1).cache().reduce(add)

sc.parallelize((2 for _ in range(10))): Creates an RDD of 10 elements, all of which are the number 2.

.map(lambda x: 1): Transformation. Converts every element in the RDD from 2 to 1. The RDD now contains ten 1s: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1].

.cache(): Storage Level Action. Marks the RDD to be persisted in memory after it's computed for the first time.

,reduce(add): Action. Sums all the elements. Since there are ten 1s, the sum is 10.

Result: 10

Operations on Resilient Distributed Datasets (RDDs)

Random split

 	    
rdd = sc.parallelize(range(500), 1)
rdd1, rdd2 = sc.parallelize(range(500), 1).randomSplit([2, 3], 17)

 	    
len(rdd1.collect() + rdd2.collect())
500
150 < rdd1.count() < 250
True
250 < rdd2.count() < 350
True

Reduce RDD to one value

 
from operator import add
sc.parallelize([1, 2, 3, 4, 5]).reduce(add)
15

 
sc.parallelize((2 for _ in range(10))).map(lambda x: 1).cache().reduce(add)
10

The Difference Between map and flatMap

Both map and flatMap are transformations that apply a function to every element in the RDD. The key difference lies in how they handle the output of the function, especially when the function returns a collection (like a list).

A. map Transformation

The map transformation applies the function to each element and collects the results into a new RDD. It maintains a one-to-one correspondence between input and output elements.

sc.parallelize(["a b c","c d e"]).map(lambda line: line.split(" ")).collect()

# Result: [['a', 'b', 'c'], ['c', 'd', 'e']]

Process: The lambda function splits the input string into a list of words. The map transformation keeps each of these lists as a distinct element in the resulting RDD. The input RDD had 2 elements (lines), and the output RDD has 2 elements (lists of words).

B. flatMap Transformation

The flatMap transformation first applies the function to each element (like map), and then it flattens the resulting collection of collections into a single, cohesive RDD. It is a one-to-many (or one-to-zero) transformation.

sc.parallelize(["a b c","c d e"]).flatMap(lambda line: line.split(" ")).collect()

# Result: ['a', 'b', 'c', 'c', 'd', 'e']

Process:

The function splits the first line into ['a', 'b', 'c'].

The function splits the second line into ['c', 'd', 'e'].

flatMap takes these two lists and merges them into a single RDD of individual words. The input RDD had 2 elements, and the output RDD has 6 elements.

2. Iteration Operation (foreach)

The foreach operation is an action that executes a function on every element of the RDD. Since RDDs are distributed across a cluster, the function executes on the worker nodes.

def f(x):

print(x)

sc.parallelize([1, 2, 3, 4, 5]).foreach(f)

Purpose: It's used for performing side effects, such as writing data to a database, sending logs, or, as in this case, printing.

Important Note: When running in a distributed environment, the print(x) output occurs on the worker nodes and is usually not visible to the driver program's console (or the Spark shell/Jupyter environment) unless special logging configurations are set up.

3. Emptiness Check (isEmpty)

The isEmpty operation is a simple action that checks if the RDD contains any elements. It returns a Boolean value.

sc.parallelize([]).isEmpty() # Result: True

sc.parallelize(["a"]).isEmpty() # Result: False

Efficiency: Because it's an action, it triggers the computation of any preceding transformations. However, it is generally very fast as it only needs to check the size or existence of the RDD's partitions.

Operations on Resilient Distributed Datasets (RDDs)

map versus flatMap

 	    
sc.parallelize(["a b c","c d e"]).map(lambda line: line.split(" ")).collect()
[['a', 'b', 'c'], ['c', 'd', 'e']]

 	    
sc.parallelize(["a b c","c d e"]).flatMap(lambda line: line.split(" ")).collect()
['a', 'b', 'c', 'c', 'd', 'e']

Iteration operation foreach

def f(x):
    print(x)

sc.parallelize([1, 2, 3, 4, 5]).foreach(f)

isEmpty operation

sc.parallelize([]).isEmpty()
True
sc.parallelize(["a"]).isEmpty()
False

The operations reduce and reduceByKey are critical for aggregating data in Spark RDDs. They both apply a specified function to combine values, but they differ significantly in what they operate on and what they return.

1. reduce Operation

The reduce operation is an Action that aggregates all elements of an RDD into a single, non-distributed value using a specified binary function (e.g., addition, multiplication, finding the maximum).

Input RDD: An RDD of any type (e.g., numbers, strings).

Output: A single, local value.

Purpose: To compute a final summary (like a total sum) for the entire dataset.

Examples:

from operator import add

sc.parallelize([1, 2, 3, 4, 5]).reduce(add)

# Result: 15

The add function is applied iteratively across the RDD elements: (1+2)→3→(3+3)→6→(6+4)→10→(10+5)→15.

sc.parallelize((2 for _ in range(10))).map(lambda x: 1).cache().reduce(add)

# Result: 10

The input RDD is first transformed to contain ten 1s.

The reduce(add) action then sums all ten 1s, resulting in 10.

2. reduceByKey Operation

The reduceByKey operation is a Transformation designed specifically for Pair RDDs (RDDs of (key, value) tuples). It aggregates the values within each key group using a specified binary function, combining all matching keys into a single key-value pair.

Input RDD: A Pair RDD, RDD[(K, V)].

Output: A new Pair RDD, RDD[(K, V')], where the key is preserved and the value is the result of the aggregation.

Purpose: To compute a summary per key (e.g., counting word occurrences, totaling sales per region).

Example:

from operator import add

sorted(sc.parallelize([("a", 1), ("b", 1), ("a", 1)]).reduceByKey(add).collect())

# Result: [('a', 2), ('b', 1)]

Grouping: Spark groups the elements by key:

Key "a": Values [1, 1]

Key "b": Values [1]

Reduction: The add function is applied to the values for each key:

"a": 1+1=2

"b": 1 (no other value to add)

Output: A new RDD containing the pairs ('a', 2) and ('b', 1).

Operations on Resilient Distributed Datasets (RDDs)

reduce operation

 	    
from operator import add
sc.parallelize([1, 2, 3, 4, 5]).reduce(add)
15

 	    	     
sc.parallelize((2 for _ in range(10))).map(lambda x: 1).cache().reduce(add)
10

reduceByKey operation

 	    
from operator import add
sorted(sc.parallelize([("a", 1), ("b", 1), ("a", 1)]).reduceByKey(add).collect())
[('a', 2), ('b', 1)]

An RDD is a fault-tolerant collection of elements that can be operated on in parallel. The operations you listed are transformations, which create a new RDD from an existing one.

1. groupBy Operation

The groupBy transformation groups the elements of an RDD based on a user-defined function that determines the grouping key.

Key Characteristics:

Input: An RDD of any type of element.

Key Function: You must provide a function (a lambda in your example) that takes an element and returns the key by which it should be grouped.

Output RDD: An RDD of (K, Iterable) pairs, where:

K is the key returned by your grouping function.

Iterable is an iterable collection of all the original values that produced that key.

Shuffle: This operation is a wide transformation and requires a full data shuffle across the cluster, which can be computationally expensive.

Initial RDD: [("a", 1), ("b", 1), ("a", 1)]

Example 1 (Counting Values):

# 1. groupByKey: Groups all values for the same key.

# Result: [('a', Iterable<1, 1>), ('b', Iterable<1>)]

# 2. mapValues(len): Applies the 'len' function to the Iterable (the values) for each key.

# len([1, 1]) = 2

# len([1]) = 1

sorted(sc.parallelize([...]).groupByKey().mapValues(len).collect()) # Output: [('a', 2), ('b', 1)]

Example 2 (Listing Values):

# 1. groupByKey: Groups all values for the same key.

# Result: [('a', Iterable<1, 1>), ('b', Iterable<1>)]

# 2. mapValues(list): Converts the Iterable into a Python list for each key.

sorted(sc.parallelize([...]).groupByKey().mapValues(list).collect()) # Output: [('a', [1, 1]), ('b', [1])]

3. join Operation

The join transformation is used to combine two Pair RDDs based on their keys, similar to an SQL JOIN.

Key Characteristics:

Input: Two Pair RDDs, R1 of (K, V1) and R2 of (K, V2).

Type of Join: By default, RDD join performs an inner join. It only returns pairs where the key exists in both input RDDs.

Output RDD: An RDD of (K, (V1, V2)) pairs. If a key K appears m times in R1 and n times in R2, the output RDD will contain m×n combined pairs for that key.

Shuffle: This is a wide transformation and requires shuffling the data to bring matching keys together on the same machine.

RDD 1: [("a", 1), ("b", 4)] (Keys: 'a', 'b') RDD 2: [("a", 2), ("a", 3)] (Keys: 'a', 'a')

rdd1 = sc.parallelize([("a", 1), ("b", 4)])

rdd2 = sc.parallelize([("a", 2), ("a", 3)])

# Key 'b' only exists in rdd1, so it is dropped (Inner Join).

# Key 'a' exists in both. We pair every value from rdd1's 'a' with every value from rdd2's 'a'.

# From rdd1: ('a', 1)

# From rdd2: ('a', 2) and ('a', 3)

# Resulting pairs for key 'a':

# 1 paired with 2 -> ('a', (1, 2))

# 1 paired with 3 -> ('a', (1, 3))

sorted(rdd1.join(rdd2).collect()) # Output: [('a', (1, 2)), ('a', (1, 3))]

Operations on Resilient Distributed Datasets (RDDs)

groupBy operation

 	    
result = sc.parallelize([1, 1, 2, 3, 5, 8]).groupBy(lambda x: x % 2)
sorted([(x, sorted(y)) for (x, y) in result])                               
[(0, [2, 8]), (1, [1, 1, 3, 5])]

groupByKey operation

 	    
sorted(sc.parallelize([("a", 1), ("b", 1), ("a", 1)]).groupByKey().mapValues(len).collect())
[('a', 2), ('b', 1)]

 	    
sorted(sc.parallelize([("a", 1), ("b", 1), ("a", 1)]).groupByKey().mapValues(list).collect())
[('a', [1, 1]), ('b', [1])]

join operation

 	    
rdd1 = sc.parallelize([("a", 1), ("b", 4)])
rdd2 = sc.parallelize([("a", 2), ("a", 3)])
sorted(rdd1.join(rdd2).collect())
[('a', (1, 2)), ('a', (1, 3))]

Set-Like Transformations on RDDs

These transformations treat RDDs as distributed collections of items, performing standard set arithmetic (combining, finding common elements, and finding differences).

1. union Operation

The union transformation returns a new RDD that contains all the elements from the source RDD and the other RDD, including duplicates.

Key Characteristics:

Input: Two RDDs of the same data type.

Result: A new RDD containing all elements from both RDDs.

Duplicates: Unlike standard mathematical set union, the RDD union operation does not eliminate duplicate elements. If an element appears N times in RDD1 and M times in RDD2, it will appear N+M times in the resulting RDD.

Shuffle: This is a narrow transformation (no data exchange between partitions), making it very fast.

rdd = sc.parallelize([1, 1, 2, 3])

# rdd.union(rdd) combines the list [1, 1, 2, 3] with itself.

# The contents of the first RDD: [1, 1, 2, 3]

# Followed by the contents of the second RDD: [1, 1, 2, 3]

rdd.union(rdd).collect() # Output: [1, 1, 2, 3, 1, 1, 2, 3]

2. intersection Operation

The intersection transformation returns a new RDD that contains only the elements that are present in both the source RDD and the other RDD.

Key Characteristics:

Input: Two RDDs of the same data type.

Result: A new RDD containing only the common elements.

Duplicates: The operation removes duplicates. The resulting RDD will only contain distinct elements.

Shuffle: This is a wide transformation as it requires comparing elements across partitions to find common items, which is generally more costly than union.

RDD 1 (Distinct elements): {1, 10, 2, 3, 4, 5} RDD 2 (Distinct elements): {1, 6, 2, 3, 7, 8}

rdd1 = sc.parallelize([1, 10, 2, 3, 4, 5])

rdd2 = sc.parallelize([1, 6, 2, 3, 7, 8])

# Common elements (intersection): {1, 2, 3}

# The order of elements in the output is not guaranteed due to distributed processing.

rdd1.intersection(rdd2).collect() # Output: [2, 1, 3] (or any permutation of 1, 2, 3)

3. subtract Operation

The subtract transformation returns a new RDD containing all the elements from the source RDD (R1) that are not present in the other RDD (R2). It is equivalent to the set operation R1−R2.

Input: Two RDDs, R1 (source) and R2 (to subtract).

Data Type: Both RDDs must contain the same data type.

Result: The resulting RDD contains elements from R1 only if the entire element (or the key, in the case of Pair RDDs) does not exist in R2.

Crucial Note for Pair RDDs: When used on Pair RDDs, the operation checks if the entire (K, V) tuple is present in R2. Your example, however, is a common variant where subtract is often implicitly understood to be subtractByKey in concept, but based on the code, it's operating on the full tuple. Let's analyze your specific example:

RDD 1: [("a", 1), ("b", 4), ("b", 5), ("a", 3)] RDD 2: [("a", 3), ("c", None)]

The operation checks which full tuples in RDD 1 are not present in RDD 2:

("a", 1): Not in RDD 2. Kept.

("b", 4): Not in RDD 2. Kept.

("b", 5): Not in RDD 2. Kept.

("a", 3): IS in RDD 2. Removed.

rdd1 = sc.parallelize([("a", 1), ("b", 4), ("b", 5), ("a", 3)])

rdd2 = sc.parallelize([("a", 3), ("c", None)])

# ('a', 3) is the only tuple in rdd1 that is also found in rdd2.

sorted(rdd1.subtract(rdd2).collect()) # Output: [('a', 1), ('b', 4), ('b', 5)]

Operations on Resilient Distributed Datasets (RDDs)

union operation

 	    
rdd = sc.parallelize([1, 1, 2, 3])
rdd.union(rdd).collect()
[1, 1, 2, 3, 1, 1, 2, 3]

intersection operation

 	    
rdd1 = sc.parallelize([1, 10, 2, 3, 4, 5])
rdd2 = sc.parallelize([1, 6, 2, 3, 7, 8])
rdd1.intersection(rdd2).collect()
[2, 1, 3]

substract operation

 	    
rdd1 = sc.parallelize([("a", 1), ("b", 4), ("b", 5), ("a", 3)])
rdd2 = sc.parallelize([("a", 3), ("c", None)])
sorted(rdd1.subtract(rdd2).collect())
[('a', 1), ('b', 4), ('b', 5)]

Spark Operations

Outline

Quick Start
Self Contained Application
Web User Interface
Operations on Resilient Distributed Datasets (RDDs)
Operations on DataFrames
SQL Module

DataFrames, like RDDs, are subject to two types of operations that define Spark's execution model: Transformations and Actions.

1. Transformations

Transformations are operations that define a new DataFrame based on an existing one. They are used to build the data processing workflow.

Key Characteristics:

Output: Returns a new DataFrame (or Dataset).

Examples: select(), filter(), groupBy(), withColumn(), join().

Lazy Evaluation: This is the most critical feature. When you call a transformation, Spark does not immediately execute the computation. Instead, it just records the operation in a logical plan (or a lineage graph). This means calling a transformation is very fast.

Narrow Transformations: Each input partition contributes to only one output partition (e.g., filter(), select()). These are faster and don't require shuffling data.

Wide Transformations: Input partitions contribute to many output partitions, often requiring a shuffle of data across the network (e.g., groupBy(), join()).

2. Actions

Actions are operations that instruct Spark to perform the actual computation and return a result to the driver program or write data to an external storage system.

Key Characteristics:

Output: Returns a non-DataFrame data type (e.g., a Python list, an integer, or prints to the console).

Examples: show(), count(), collect(), save(), first().

Eager Execution: Calling an action is what triggers the full execution of the entire lineage of transformations that preceded it. Spark converts the logical plan into a physical plan (a Directed Acyclic Graph, or DAG) and executes it on the cluster.

The Return Type Difference and Lazy Evaluation

The difference in return type (DataFrame vs. a non-DataFrame object) is a direct consequence of the different execution models.

Why Spark Treats Them Differently

Feature Transformation Action

Purpose Define the processing logic Execute the logic and produce a result

Execution Lazy (Defers work) Eager (Triggers work)

Return Type DataFrame (A set of instructions) Standard Data Type (A concrete result)

Performance Instantaneous (No data processing) Varies (Involves cluster computation)

Operations on DataFrames

Spark DataFrame support two types of operations: transformations and actions
Transformations are operations on DataFrames that return a new DataFrame
- For example select(), groupBy(), map(), and filter()
Actions are operations that return a result to the driver program or write it to storage, and kick off a computation
- For example show(), count(), and first()
Return type difference: transformations return DataFrames/Datasets, whereas actions return some other data type
Spark treats the two operations very differently

The fact that Transformations are lazily evaluated is the single most important design principle behind Spark's efficiency and optimization capabilities.

1. The Transformation: Defining the Recipe

When you execute a transformation, Spark does not read any data or perform any significant computation. Instead, it records the metadata about the operation, effectively building a plan or a "recipe" for the final result.

Logical Plan/Lineage Graph: Spark builds a Directed Acyclic Graph (DAG) of all transformations. This DAG represents the lineage—the sequence of steps needed to transform the initial data source into the final desired result.

sqlDF = spark.sql("SELECT * FROM vpeople"))

What Spark Does: It parses the SQL query and creates a new DataFrame object, sqlDF.

What Spark Doesn't Do: It does not actually run the query, access the data source (vpeople), or load any rows into memory. It merely notes: "To get the result for sqlDF, I need to execute the query SELECT * FROM vpeople."

Result: sqlDF is a DataFrame—a new set of instructions.

2. The Action: Triggering the Execution

An Action is the command that tells Spark to stop planning and start working. When Spark encounters an action, it triggers the entire lineage of transformations that led up to that point.

sqlDF.show()

What Spark Does: Optimization (The Catalyst Optimizer): Spark takes the complete, recorded transformation plan (the DAG) and submits it to the Catalyst Optimizer. This component analyzes the plan and optimizes it, for instance, by pushing down filters, combining projections, and selecting the most efficient physical execution strategy.

Physical Plan Generation: The optimized logical plan is converted into a physical execution plan.

Execution: The physical plan is sent to the cluster's executors, where the data is actually read, processed across partitions, and the final result is computed.

Result: The computed result is brought back to the driver program and displayed (in the case of show()).

Concept Transformation Action

Execution Lazy Eager

Metaphor Writing a recipe Baking the cake

Output New DataFrame (Instructions) Non-DataFrame Value (Result)

This distinction is what allows Spark to handle massive datasets efficiently, as it avoids unnecessary I/O and computation by only executing the minimal set of steps required to satisfy the final action.

Operations on DataFrames

Transformations are lazily evaluated, meaning that Spark will not begin to execute until it sees an action
Instead, Spark internally records metadata to indicate that some transformation operation has been requested

For example transformation creates another DataFrame

  
sqlDF = spark.sql("SELECT * FROM vpeople"))

Action triggers the computation

Lazy Evaluation and Optimization in Spark

Apache Spark uses a technique called lazy evaluation to significantly improve performance by optimizing the entire sequence of operations before any data is processed.

1. Lazy Evaluation

What it is: When you define an operation on a Spark DataFrame (like filter(), select(), or groupBy()), the operation is not executed immediately. Instead, Spark records the transformation in a logical plan.

Action vs. Transformation:

Transformations (like map, filter, join) are lazy; they build the execution plan.

Actions (like count(), show(), write()) trigger the execution of the recorded plan.

The Benefit: Since Spark knows the entire program structure before it starts, it can look for ways to optimize the steps.

2. Reduction in Passes (Catalyst Optimizer)

The text mentions: "The lazy evaluation to reduce the number of passes it has to take over the dataset."

This is the job of Spark's Catalyst Optimizer .

The optimizer takes the chain of operations (the logical plan) and rearranges, combines, and simplifies them to be as efficient as possible. This optimization process leads to two key outcomes:

Fewer Passes/Stages: It minimizes the number of times data has to be read from disk or shuffled across the network. For example, if you have two filtering operations and then a selection, Spark can often combine these into a single pass over the data.

Code Flexibility: Because Spark handles the heavy optimization, the user is given a high degree of "freedom to organize their program into smaller, more manageable operations.

Operations on DataFrames

The lazy evaluation to reduce the number of passes it has to take over the dataset
In Hadoop MapReduce, developers often have to consider how to group together operations to minimize the number of MapReduce passes
In Spark, there is no substantial benefit to writing a single complex map instead of chaining together many simple operations
Thus, users are free to organize their program into smaller, more manageable operations

In modern Spark (versions 2.0 and later), the DataFrame is essentially a specific form of a Dataset.

1. Dataset: The Foundation

Definition: A Dataset is a distributed collection of data in Spark. It represents the data distributed across the cluster nodes.

Typing: A Dataset is typed, meaning its structure and data types are known and enforced.

Compile-time Check: "Typing is checked at compiling time." This is a significant advantage, as it catches errors (like trying to add a string to an integer) before the code is even run, making development faster and safer.

Flexibility: It supports the use of self-defined functions (User Defined Functions or UDFs) like map and reduce to process data.

Use Case: Datasets are primarily used when working in Scala or Java, where strong typing is native and beneficial.

2. DataFrame: The Structured View

Definition: "DataFrame is a Dataset organized into named columns."

Conceptual Equivalent: It is "conceptually equivalent to a table in a relational database or a data frame in R/Python."

Typing: A DataFrame is considered untyped or structurally typed.

Runtime Check: "Typing is checked at runtime." While the structure (schema) is known, accessing a column that doesn't exist or performing a mismatched operation is checked only when the code executes.

Language Use: This untyped nature is why DataFrames are the standard abstraction used in Python and R, as these languages are dynamically typed.

Self-Defined Functions: To use custom logic, "you need to register them with Spark" as UDFs. Spark then knows how to integrate this custom logic into its execution plan.

Performance: "DataFrame is more performance-optimal than Dataset."

This is the most crucial distinction. DataFrames, by being structured into named columns, allow Spark's Catalyst Optimizer to perform powerful, engine-level optimizations like predicate pushdown, column pruning, and codegen.

Summary of Differences

Feature Dataset (Typed API) DataFrame (Untyped/Structurally Typed API)

Structure Distributed collection of objects (e.g., a list of Person objects). Organized into named columns (like a database table).

Typing Strongly Typed Structurally Typed (Untyped API for the programmer)

Error Checking Compile-time (errors caught early) Runtime (errors only appear when the code runs)

Performance Good, but relies more on manual optimization. Superior. Enables Spark's Catalyst Optimizer to maximize performance.

Primary Languages Scala, Java Python, R, Scala, Java

Operations on DataFrames

DataFrame is a data abstraction for Spark SQL
Dataset is a distributed collection of data
- It supports the use of self-defined functions to process data
- For example, map and reduce functions in the previous slides
- Dataset is typed; typing is checked at compiling time
DataFrame is a Dataset organized into named columns.
- It is conceptually equivalent to a table in a relational database or a data frame in R/Python
- To use self-defined functions, you need to register them with Spark
- DataFrame is untyped, i.e., typing is checked at runtime
- DataFrame is more performance-optimal than Dataset

Creating and Inspecting a Spark DataFrame

The process involves reading a file using a Spark Session and then using methods to visualize the data and its inferred structure.

1. The Source Data (people.json)

The first part shows the source data file, which is a collection of JSON records stored in a distributed file system (like HDFS):

hdfs dfs -cat people.json

{"name":"James","age":25}

{"name":"Harry","age":24}

{"name":"Robin","age":50}

Format: The data is stored in JSON Lines format, where each line is a self-contained JSON object.

Content: It contains two fields for each record: name (a string) and age (a number).

2. Reading the Data into a DataFrame

The next step is the actual command used in Spark (typically in a tool like spark-shell or a Jupyter notebook using PySpark) to create the DataFrame:

Python

df = spark.read.load("people.json", format="json")

spark: This is the entry point object for Spark SQL, usually an instance of SparkSession.

.read: This is the DataFrameReader used to load data from various external sources.

.load("people.json", format="json"): This instructs Spark to:

Load the data from the file path "people.json".

Specify the file format as "json".

Crucially: Spark automatically infers the schema (the names and types of the columns) by sampling the JSON data.

3. Displaying the Data (df.show())

The show() action is used to display the contents of the DataFrame, similar to a SELECT * query in SQL:

df.show(): This is an Action that triggers the execution of the read operation, loads the data, and formats the result in a clean, tabular output.

Output: The output shows the DataFrame's structure with two columns (age and name) and the three rows of data.

4. Inspecting the Schema (df.printSchema())

The printSchema() method is a vital debugging and inspection tool. It reveals the structure (schema) that Spark inferred during the load process:

df.printSchema(): This shows the DataFrame's metadata.

root: The top level of the structure.

age: long (nullable = true): Spark inferred that the age field contains whole numbers, mapping them to the long data type (a 64-bit integer) and that the field can contain null values (nullable = true).

name: string(nullable = true): Spark inferred that the name field contains text, mapping it to the string data type.

This entire process demonstrates how easily Spark can ingest semi-structured data like JSON and immediately provide a structured, queryable DataFrame abstraction.

Operations on DataFrames

A DataFrame can be created in the following way

hdfs dfs -cat people.json
{"name":"James","age":25}
{"name":"Harry","age":24}
{"name":"Robin","age":50}

   
df = spark.read.load("people.json", format="json")

   
>>> df.show()
+---+-----+
|age| name|
+---+-----+
| 25|James|
| 24|Harry|
| 50|Robin|
+---+-----+

	 
df.printSchema()
root
 |-- age: long (nullable = true)
 |-- name: string (nullable = true)

Saving DataFrames to HDFS

The general syntax for saving a DataFrame is to select the columns you want, then use the .write property to specify the output format and destination.

1. Saving to CSV (Comma Separated Values)

This example selects only the name column and saves it as CSV files into a directory named names on HDFS.

Python

df.select("name").write.save("names", format="csv")

df.select("name"): This is a transformation that creates a new DataFrame containing only the name column.

.write: Accesses the DataFrameWriter interface.

.save("names", format="csv"): This Action triggers the save operation:

The data is written to the HDFS directory names.

The output format is specified as csv.

The subsequent HDFS commands demonstrate the result of this operation:

hdfs dfs -ls names

Partitioned Output: Spark is a distributed system, so it writes the output in one or more part-files (e.g., part-00000-....csv). The number of part-files depends on the number of partitions in the DataFrame.

This confirms that only the name column was saved, with each name on a new line.

2. Reading and Writing Parquet Files

Parquet is the preferred file format for most large-scale data processing in Spark because it is highly optimized for performance and space efficiency.

df = spark.read.load("examples/src/main/resources/users.parquet")

If the file extension is omitted, Spark often defaults to inferring the format, but here, the explicit .load() function is used, which typically relies on the file extension or optional parameters. However, in Spark, Parquet is the default format for reading and writing if no format is specified. The users.parquet name strongly implies the format.

Writing Parquet

df.select("name", "favorite_color").write.save("namesAndFavColors.parquet")

Columnar Storage: Parquet is a columnar storage format, meaning it groups data by column instead of by row. This allows for:

High Compression: Since data within a column is often homogeneous, compression is much more effective.

Projection Pushdown: Spark can read only the columns you select (in this case, name and favorite_color) without reading the entire row, drastically improving I/O performance.

Save Name: The output is saved to the directory namesAndFavColors.parquet.

Operations on DataFrames

A DataFrame can be saved in HDFS in the following way

 
df.select("name").write.save("names", format="csv")

 
hdfs dfs -ls names
Found 2 items
-rw-r--r--   1 bigdata supergroup          0 2026-04-20 13:59 names/_SUCCESS
-rw-r--r--   1 bigdata supergroup         18 2026-04-20 13:59 names/part-00000-f1e3483e-810b-4be6-8875-
                                                                                  4587cb6be954-c000.csv

 
hdfs dfs -cat names/part*
James
Harry
Robin

Reading and writing a Parquet file

 
df = spark.read.load("examples/src/main/resources/users.parquet")

 	    
df.select("name", "favorite_color").write.save("namesAndFavColors.parquet")

PySpark DataFrame Operations Explained

The operations showcased in the examples primarily involve basic DataFrame introspection, filtering, and selection.

1. DataFrame Creation

First, two DataFrames, people and department, are created from lists of dictionaries.

2. Counting Rows

people.count()

Operation: count()

Purpose: This is an Action operation that returns the total number of rows in the DataFrame. It triggers computation to read the data.

Result: The people DataFrame has 4 rows, so the result is 4.

3. Filtering Rows (filter and where)

The operations filter() and where() are aliases for each other and are used to select a subset of rows based on a specified condition. They are Transformations, meaning they define a plan but do not execute until an action is called.

Using Column Expressions (Type 1)

This method uses PySpark's built-in column object notation, which is generally recommended as it is more robust and safer against parsing errors.

df1 = people.filter(people.age > 50)

Operation: Filters the people DataFrame where the age column is greater than 50.

Result (df1): Contains the person whose age is 60.

df2 = people.where(people.age == 60) # Note: The code snippet used 'df.age', but it should be 'people.age' based on context.

Operation: Filters the people DataFrame where the age column is equal to 60.

Result (df2): Contains the person whose age is 60.

Using SQL Expression Strings (Type 2)

This method passes a SQL-style conditional expression as a string.

df3 = people.filter('age > 30')

Operation: Filters the people DataFrame where the age is greater than 30.

Result (df3): Contains the people with ages 40, 50, and 60 (3 rows).

df4 = people.where('age = 60')

Operation: Filters the people DataFrame where the age is equal to 60.

Result (df4): Contains the person whose age is 60.

Summary of DataFrame Filters

The filter() and where() methods create a new DataFrame containing only the rows that satisfy the condition, which is a core concept in data wrangling.

Variable Method Condition Resulting Row Count

df1 filter(people.age > 50) Age greater than 50 1

df2 where(people.age == 60) Age equals 60 1

df3 filter('age > 30') Age greater than 30 3

df4 where('age = 60') Age equals 60 1

Operations on DataFrames

Operations on DataFrames included in a class pyspark.sql.DataFrame

people = spark.createDataFrame([
	 {"deptId": 1, "age": 40, "name": "Hyukjin Kwon", "gender": "M", "salary": 50},
         {"deptId": 1, "age": 50, "name": "Takuya Ueshin", "gender": "M", "salary": 100},
         {"deptId": 2, "age": 60, "name": "Xinrong Meng", "gender": "F", "salary": 150},
         {"deptId": 3, "age": 20, "name": "Haejoon Lee", "gender": "M", "salary": 200}
])

department = spark.createDataFrame([
         {"id": 1, "name": "PySpark"},
         {"id": 2, "name": "ML"},
         {"id": 3, "name": "Spark SQL"}
])

people.count()

df1 = people.filter(people.age > 50)
df2 = people.where(df.age == 60)

df3 = people.filter('age > 30')
df4 = people.where('age = 60')

PySpark DataFrame Operations: Selection & Aggregation

1. Selecting Columns (select)

The select() transformation is used to choose which columns to keep, rename columns, or perform basic column-wise transformations.

Python

df5 = people.select('*')

Operation: select('*')

Purpose: Selects all columns from the people DataFrame. This results in a new DataFrame (df5) that is structurally identical to the original people DataFrame.

df6 = people.select(people.name, (people.age + 10).alias('age'))

Operation: select() with an expression and an alias.

Purpose

Selects the existing name column.

Creates a new column by taking the existing age column and adding 10 to every value (people.age + 10).

Renames this new calculated column to 'age' using .alias('age').

Result (df6): A DataFrame with two columns: name and a new calculated age (e.g., Hyukjin Kwon's new age is 40+10=50).

2. Aggregation (groupBy)

Aggregation operations combine multiple rows into a single summary row based on a grouping key. They are typically two-step processes: grouping followed by an aggregation function (like avg, count, sum, max).

Simple Aggregation (Without Grouping Key)

df7 = people.groupBy().avg()

Operation: groupBy().avg()

Purpose: When no grouping key is specified in groupBy(), the entire DataFrame is treated as a single group. The avg() function then calculates the average of all numeric columns across the entire dataset.

Result (df7): A single row with columns like avg(deptId), avg(age), avg(salary).

Grouping and Counting

df8 = people.groupBy("age").count()

Operation: groupBy("age").count()

Purpose: Groups the rows based on the unique values in the age column. For each unique age group, the count() function calculates the number of rows belonging to that group.

Result (df8): A DataFrame with two columns: age and count. It will show, for example, how many people are 40, 50, 60, and 20 (one person for each age in the sample data).

rouping and Aggregating with a Dictionary

df9 = people.groupBy("name").agg({"age": "sum"}).sort("name")

Operation: groupBy().agg()

Purpose: This is a flexible way to apply multiple, specific aggregation functions.

Groups the data by the name column.

Uses the dictionary {"age": "sum"} to apply the sum aggregation function only to the age column within each group.

Since name is unique in this dataset, the "sum of age" is just the age itself for each person. The result is then sorted by name.

Result (df9): A DataFrame with name and sum(age).

Grouping and Finding Maximum

df10 = people.groupBy(people.name).max().sort("name")

Operation: groupBy().max()

Purpose:

Groups the data by name.

Applies the max() function to find the maximum value of all other numeric columns (i.e., deptId, age, salary) within each name group.

Result (df10): A DataFrame with columns name, max(deptId), max(age), and max(salary).

Grouping by Multiple Columns

df11 = people.groupBy(["name", people.age]).count().sort("name", "age")

Operation: groupBy(list_of_columns).count()

Purpose: Groups the data based on the combination of values in the name and age columns. The count() function then counts the number of rows for each unique (name, age) pair.

Result (df11): A DataFrame with name, age, and count. Since the sample data has unique people, the count for every resulting row will be 1. This is often used when the grouping keys are not guaranteed to be unique individually (e.g., grouping by deptId and gender).

Operations on DataFrames

Operations on DataFrames included in a class pyspark.sql.DataFrame

df5 = people.select('*')
df6 = people.select(people.name, (people.age + 10).alias('age'))

 
df7 = people.groupBy().avg()

	      
df8 = people.groupBy("age").count()

	    
df9 = people.groupBy("name").agg({"age": "sum"}).sort("name")

	      
df10 = people.groupBy(people.name).max().sort("name")

 	      
df11 = people.groupBy(["name", people.age]).count().sort("name", "age")

For set operations like union, intersect, and subtract, the DataFrames involved must have the same number of columns and the columns must be compatible (ideally having the same order and data types).

1. Combining DataFrames (union)

df12 = spark.createDataFrame([(1, 'A'), (2, 'B')], ['id', 'value'])

df13 = spark.createDataFrame([(3, 'C'), (4, 'D')], ['id', 'value'])

df14 = df12.union(df13)

Operation: union()

Purpose: Appends the rows of the second DataFrame (df13) to the end of the first DataFrame (df12).

Key Feature: The union() operator in modern Spark (since version 2.0) is equivalent to unionAll in older versions. It does not eliminate duplicate rows by default.

Result (df14): A new DataFrame with the four rows from both df12 and df13: | id | value | | :---: | :---: | | 1 | A | | 2 | B | | 3 | C | | 4 | D |

Note on unionByName: If the columns have the same names but are in a different order, you should use df12.unionByName(df13).

2. Finding Common Rows (intersect)

df15 = spark.createDataFrame([("a", 1), ("a", 1), ("b", 3), ("c", 4)], ["C1", "C2"])

df16 = spark.createDataFrame([("a", 1), ("a", 1), ("b", 3)], ["C1", "C2"])

df17 = df16.intersect(df15).sort(df16.C1.desc()).show()

Operation: intersect()

Purpose: Returns a new DataFrame containing only the distinct rows that are present in both DataFrames.

Key Feature: intersect() implicitly performs a distinct() operation on the result.

3. Finding Unique Rows (subtract)

df18 = spark.createDataFrame([("a", 1), ("a", 1), ("b", 3), ("c", 4)], ["C1", "C2"])

df19 = spark.createDataFrame([("a", 1), ("a", 1), ("b", 3)], ["C1", "C2"])

df20 = df18.subtract(df19).show()

Operation: subtract()

Purpose: Returns a new DataFrame containing the distinct rows that are present in the first DataFrame (df18) but not present in the second DataFrame (df19). This is often called the set difference.

Key Feature: subtract() considers rows as distinct elements.

Operations on DataFrames

Operations on DataFrames included in a class pyspark.sql.DataFrame

 
df12 = spark.createDataFrame([(1, 'A'), (2, 'B')], ['id', 'value'])
df13 = spark.createDataFrame([(3, 'C'), (4, 'D')], ['id', 'value'])
df14 = df12.union(df13)

 
df15 = spark.createDataFrame([("a", 1), ("a", 1), ("b", 3), ("c", 4)], ["C1", "C2"])
df16 = spark.createDataFrame([("a", 1), ("a", 1), ("b", 3)], ["C1", "C2"])
df17 = df16.intersect(df15).sort(df16.C1.desc()).show()

 
df18 = spark.createDataFrame([("a", 1), ("a", 1), ("b", 3), ("c", 4)], ["C1", "C2"])
df19 = spark.createDataFrame([("a", 1), ("a", 1), ("b", 3)], ["C1", "C2"])
df20 = df18.subtract(df19).show()

Joining DataFrames is necessary when you have related data spread across multiple tables (DataFrames) that you want to bring together. This is similar to SQL JOINs.

First, let's look at the four DataFrames created for this section:

DataFrame Schema (Columns) Data Sample

df1 age, name (2, "Alice"), (5, "Bob")

df2 height, name (80, "Tom"), (85, "Bob")

df3 age, name (2, "Alice"), (5, "Bob")

df4 age, height, name (10, 80, "Alice"), (5, null, "Bob"), (null, null, "Tom"), (null, null, null)

1. Inner Join (Implicit)

An inner join selects only the rows that have matching values in both DataFrames based on the join key(s).

df21 = df1.join(df2, 'name').select(df2.name, df2.height)

peration: join(df2, 'name')

Join Type: Inner Join (default).

Join Key: The common column is 'name'.

Result (df21): It finds names present in both df1 (Alice, Bob) and df2 (Tom, Bob). The only match is "Bob". The final DataFrame only contains Bob's name and height. | name | height | | :---: | :---: | | Bob | 85 |

df22 = df1.join(df4, ['name', 'age']).select(df4.name, df4.age).show()

Operation: join(df4, ['name', 'age'])

Join Key: A compound key consisting of both 'name' and 'age'.

Result (df22):

df1 rows: (2, "Alice"), (5, "Bob")

df4 rows: (10, 80, "Alice"), (5, null, "Bob"), ...

The joins require an exact match on both columns. No exact match exists for (2, "Alice") in df4. The closest is (5, "Bob"), but since df4 has other rows with nulls, and the join is inner, the result is likely empty because the non-null records in df1 don't find a full match in the non-null records of df4 (Alice's age is 2 in df1 but 10 in df4).

2. Outer Join (Full Outer Join)

An outer join returns all records when there is a match in either the left or the right DataFrame. Where there is no match, the columns from the non-matching side are filled with null.

df23 = df1.join(df2, df1.name == df2.name, 'outer').select(df2.name, df2.height).sort(desc("name")).

Operation: join(df2, join_expression, 'outer')

Join Type: Full Outer Join ('outer').

Join Condition: Using an expression df1.name==df2.name.

Result (df23): It combines all names from both DataFrames: Alice, Bob, and Tom.

Operations on DataFrames

Operations on DataFrames included in a class pyspark.sql.DataFrame

 
from pyspark.sql import Row
from pyspark.sql.functions import desc
df1 = spark.createDataFrame([(2, "Alice"), (5, "Bob")]).toDF("age", "name")
df2 = spark.createDataFrame([Row(height=80, name="Tom"), Row(height=85, name="Bob")])
df3 = spark.createDataFrame([Row(age=2, name="Alice"), Row(age=5, name="Bob")])
df4 = spark.createDataFrame([Row(age=10, height=80, name="Alice"),
                             Row(age=5, height=None, name="Bob"),
                             Row(age=None, height=None, name="Tom"),
                             Row(age=None, height=None, name=None) ])

 
df21 = df1.join(df2, 'name').select(df2.name, df2.height)
df22 = df1.join(df4, ['name', 'age']).select(df4.name, df4.age).show()

df23 = df1.join(df2, df1.name == df2.name, 'outer')
	  .select(df2.name, df2.height).sort(desc("name")).
df24 = df.join(df2, 'name', 'outer').select('name', 'height').sort(desc("name")).show()

Spark Operations

Outline

Quick Start
Self Contained Application
Web User Interface
Operations on Resilient Distributed Datasets (RDDs)
Operations on DataFrames
SQL Module

1. Registering the DataFrame as a Temporary View

people.createOrReplaceTempView("vpeople")

Method: createOrReplaceTempView(viewName)

Purpose: This method registers the people DataFrame as a temporary, logical view named "vpeople".

Key Features:

Temporary: The view is available only during the life of the current SparkSession. It is not stored persistently.

Logical: It doesn't physically copy the data. It just registers a name in Spark's catalog that maps to the DataFrame's data plan.

OrReplace: If a view with the name "vpeople" already exists, it is dropped and replaced with the new one.

2. Executing SQL Query

sqlDF = spark.sql("SELECT * FROM vpeople")

Method: spark.sql(sqlQueryString)

Purpose: This is the entry point for executing standard SQL queries. The query accesses the data registered via the temporary view "vpeople".

Result (sqlDF): The result of the SQL query is a new DataFrame (sqlDF). The query SELECT * FROM vpeople simply selects all columns and all rows from the registered people data.

3. Displaying the Result

sqlDF.show()

Method: show()

Purpose: This is an Action that triggers the execution of the SQL query plan, retrieves the data from the underlying DataFrame, and prints the result to the console in a formatted table.

SQL Module

 
people.createOrReplaceTempView("vpeople")

  
sqlDF = spark.sql("SELECT * FROM vpeople")
sqlDF.show()

References

RDD Programming Guide
Spark SQL, DataFrames and Datasets Guide
A Gentle Introduction to Spark, Databricks, (Available in READINGS folder)
Karau H., Fast data processing with Spark Packt Publishing, 2013 (Available from UOW Library)
Srinivasa, K.G., Guide to High Performance Distributed Computing: Case Studies with Hadoop, Scalding and SparkSpringer, 2015 (Available from UOW Library)
Chambers B., Zaharia M.,Spark: The Definitive Guide, O'Reilly 2017
Perrin J-G., Spark in Action, 2nd ed., Manning Publications Co. 2020