21 Spark Data Model

Introduction to Spark

Outline

Resilient Distributed Data Sets (RDDs)
DataFrames
SQL Tables/View
Datasets

Resilient Distributed Datasets (RDDs) are the original and lowest-level data structure in Spark, representing an immutable, partitioned collection of elements that can be operated on in parallel across a cluster.

Key Characteristics

Immutable and Partitioned: An RDD is an immutable collection, meaning once created, its contents cannot be changed. All transformations create a new RDD. It is partitioned, meaning the data is split into logical chunks distributed across the cluster's Worker Nodes. This partitioning enables parallel processing.

Collection of Objects: Every element (row) in an RDD is a Java object (or a Scala/Python equivalent). This generic object structure is what gives RDDs their flexibility.

No Schema: A key feature is that the RDD does not need to have any schema defined in advance. This makes RDDs extremely flexible for handling raw, unstructured, or semi-structured data where the exact format isn't known or fixed.

Flexibility vs. Complexity

This flexibility comes with a trade-off in complexity:

Complicated Manipulations: Since there's no predefined schema or column names, manipulating the data is more complicated. Users must implement their own custom functions (like map and reduceByKey) to perform even simple tasks like aggregation functions (e.g., calculating the average, count, or maximum).

Control: RDDs offer the highest level of control to the user over how data is distributed, cached, and operated on across the cluster. Developers can manually control partitioning, serialization, and persistence strategy, which is useful for highly specialized performance tuning.

While RDDs offer low-level control, the higher-level abstractions (DataFrames and Datasets) are generally preferred today because they automate optimization and provide simpler, schema-based APIs.

Resilient Distributed Data Sets (RDDs)

Resilient Distributed Datasets (RDDs) is the lowest level (and the oldest) data abstraction available to the users
An RDD represents an immutable, partitioned collection of elements that can be operated on in parallel
Every row in an RDD is a Java object
RDD does not need to have any schema defined in advance
It makes RDD very flexible for various applications but in the same moment makes the manipulations on data more complicated
Users must implement their own functions to perform simple tasks like for example aggregation functions: average, count, maximum, etc
RDDs provide more control on how data is distributed over a cluster and how it is operated

An RDD is fully defined by five main properties that the Spark scheduler uses to determine how to compute, distribute, and recover the data.

1. A List of Partitions

An RDD is logically split into smaller, manageable chunks called partitions.

This property specifies the exact number and identity of these partitions.

The partitions are the fundamental unit of parallelism in Spark; each partition is processed by a single Task on an Executor.

2. A Function for Computing Each Split

This is the function that defines how the data in each partition is computed or rebuilt.

For a base RDD (one loaded from a file), this function describes how to read the data. For an RDD created through a transformation (like map or filter), this function describes the transformation to be applied to the parent RDD's partition.

3. A List of Dependencies on Other RDDs

This property maintains the lineage graph, which is the sequence of transformations used to create the current RDD.

Dependencies allow Spark to implement Resilience: if a partition is lost due to a node failure, Spark can re-trace the lineage path (the sequence of transformations) and recompute only the lost partition from its parent RDDs, rather than recomputing the entire dataset.

4. Optionally, a Partitioner for Key-Value RDDs

For RDDs that are composed of key-value pairs (e.g., the output of a map operation before a reduceByKey), this property dictates how the data is logically grouped and distributed across the partitions based on the keys.

Custom partitioners can be specified by the user to ensure that related keys land on the same partition. This is essential for operations like joins or aggregations and can provide significant performance improvements by reducing the amount of data shuffling (network I/O) required across the cluster.

5. Optionally, a List of Preferred Locations to Compute Each Split

This list identifies the worker nodes where the partition's data is physically closest (e.g., where the data resides in HDFS).

Spark uses this information for data locality: the scheduler tries to run the task on the worker node that already holds the data. This minimizes network traffic and is a crucial part of Spark's optimization for speed.

These properties collectively define the RDD and directly instruct the Spark scheduler on how to process and distribute the application's work, allowing for fault tolerance, parallel execution, and performance optimization. Different RDDs in the same application may use different properties based on the required outcome of that specific computation step.

Resilient Distributed Data Sets (RDDs)

RDD is characterized by the following properties
- A list of partitions
- A function for computing each split
- A list of dependencies on other RDDs
- Optionally, a Partitioner for key-value RDDs
- Optionally, a list of preferred locations to compute each split
Specification of custom partitions may provide significant performance improvements when using key-value RDDs
The properties of RDDs determine how Spark schedules and processes the applications
Different RDDs may use different properties depending on the required outcomes

The SparkContext is the core object in every Spark application and acts as the entry point for all Spark functionality. It's the mechanism that:

Connects to the Cluster: The SparkContext connects your application (the Driver Program) to the Cluster Manager (like YARN or Standalone) to acquire resources (Executors) on the cluster.

Coordinates Parallel Processing: Once connected, it is used to create RDDs and coordinate the execution of tasks across the worker nodes.

SparkContext Creation

In Standalone Applications: You must explicitly create the SparkContext object, providing at least the application name and the master URL (which specifies where the cluster resides, e.g., 'local', 'yarn', or the master node's URL).Python

sc = SparkContext(appName="RDD Examples")

In PySpark Shell/Notebooks: When you launch the pyspark shell or use a managed environment (like Databricks), the SparkContext object is automatically created and is made available under the variable name sc. This lets you start working immediately without boilerplate code.

Creating RDDs

An RDD can be created from any data source, but the simplest way to start is by parallelizing an existing collection in the driver program's memory.

The parallelize() Method

The sc.parallelize() method takes a local Python collection (like a list) and distributes its elements across the cluster, turning it into a parallel collection (an RDD).

numbers = sc.parallelize([1, 2, 3, 4, 5])

names = sc.parallelize(["James", "Harry", "Robin"])

The data is now partitioned and ready for distributed processing by the executors.

Listing RDD Elements (Actions)

To view the elements of an RDD or retrieve its results, you must use an Action. Actions trigger the execution of the entire sequence of transformations defined on the RDD.

Action Purpose Warning

.collect() Retrieves ALL elements of the RDD from all worker nodes and returns them as a local list to the Driver Program. Use with caution! If the RDD is very large, this can cause the Driver machine to run out of memory (OutOfMemoryError).

.take(n) Retrieves the first n elements of the RDD as a local list to the Driver Program. This is much safer than collect() for large RDDs, as it only returns a small sample. It's often used for quick inspection or debugging.

# Listing RDD

names.collect() # Output: ['James', 'Harry', 'Robin']

# Listing the first 3 elements

names.take(3) # Output: ['James', 'Harry', 'Robin']

Resilient Distributed Data Sets (RDDs)

Every Spark script includes an object SparkContext that connects to the cluster and allows processing to be parallelized

Typical creation of SparkContext object

 
sc = SparkContext(appName="RDD Examples")

When connecting from pyspark shell SparkContext object is created automatically

RDD can be created from a list

   
numbers = sc.parallelize([1, 2, 3, 4, 5])

   
names = sc.parallelize(["James","Harry", "Robin"])

Listing RDD

Creating RDDs from HDFS Files

The primary way to ingest large-scale data into Spark is by pointing the SparkContext (sc) to a file path using the textFile() method. Spark handles the complexity of distributing the file read operation across the cluster.

1. Simple Path (Default HDFS)

If the file is located in the HDFS home directory of the current user (e.g., /user/bigdata), you can often use a simple file name or relative path.

# Assuming NOTICE.txt is in the /user/bigdata folder in HDFS

# and the user running the Spark shell is 'bigdata'

notice_lines = sc.textFile("NOTICE.txt")

Mechanism: Spark automatically resolves this path based on the user's default HDFS configuration and the default HDFS NameNode address defined in the cluster setup.

2. Explicit HDFS Path

When dealing with files located in non-default directories or external HDFS clusters, you should use the full HDFS URI.

# Assuming README.txt is in the /user folder on a local HDFS instance

read_lines = sc.textFile("hdfs://localhost:9000/user/README.txt")

hdfs://: The protocol identifier.

localhost:9000: The address and port of the HDFS NameNode (the master service).

/user/README.txt: The full path to the file within HDFS.

Saving RDDs to HDFS

The saveAsTextFile() action is used to write the contents of an RDD back out to a text file in a distributed filesystem, typically HDFS.

read_lines.saveAsTextFile("hdfs://localhost:9000/user/readme")

Output in HDFS

When Spark saves an RDD, it does not create a single file. Instead, it creates a directory (named /user/readme in this case) containing multiple files, where each file represents a partition of the RDD processed by an Executor.

hdfs dfs -ls /user/readme

Found 2 items

-rw-r--r-- 1 bigdata supergroup 0 2026-04-15 19:00 /user/readme/_SUCCESS

-rw-r--r-- 1 bigdata supergroup 175 2026-04-15 19:00 /user/readme/part-00000

_SUCCESS: A zero-byte marker file created by Spark (and other Hadoop tools) to indicate that the job completed successfully and that the data is ready to be used.

part-00000 (and subsequent files): These are the actual output files containing the data. The number of these files is determined by the number of partitions in the RDD at the time the saveAsTextFile action was executed.

Resilient Distributed Data Sets (RDDs)

Assume that a file NOTICE.txt is localed in /user/bigdata folder in HDFS and it is a home folder of bigdata user

Creating RDD from a file in HDFS

   
notice_lines = sc.textFile("NOTICE.txt")

Assume that a file README.txt is localed in /user folder in HDFS on the local host

Creating RDD from a file in HDFS

   
read_lines = sc.textFile("hdfs://localhost:9000/user/README.txt")

Saving RDD in HDFS

read_lines.saveAsTextFile("hdfs://localhost:9000/user/readme")

hdfs dfs -ls /user/readme
Found 2 items
-rw-r--r--   1 bigdata supergroup          0 2026-04-15 19:00 /user/readme/_SUCCESS
-rw-r--r--   1 bigdata supergroup        175 2026-04-15 19:00 /user/readme/part-00000

Word Count with RDDs

The provided code uses a chain of transformations on an RDD to count the occurrences of every word in the words.txt file located in HDFS.

Step 1: Ingest Data (Action/Creation)

lines_rdd = sc.textFile("words.txt")

Method: sc.textFile() is an action (that initiates the reading) and a method of the SparkContext that creates the initial RDD.

Result: lines_rdd is an RDD wher each element is a single line from the words.txt file. The lines are partitioned across the cluster's worker nodes.

Step 2: Flatten Lines into Words (Transformation)

.flatMap(lambda line: line.split(" "))

Method: flatMap() is a transformation that applies a function to each element of the RDD and then "flattens" the result.

Function: line.split(" ") splits each line into a list of individual words using a space as a delimiter.

Result: The RDD is transformed from a collection of lines into a single, combined RDD of individual words.

Step 3: Map Words to Key-Value Pairs (Transformation)

.map(lambda word: (word, 1))

Method: map() is a simple transformation that applies a function to each element of the RDD, maintaining the one-to-one count.

Function: (word, 1) converts each word into a key-value pair, where the word is the key and 1 is the initial count/value.

Result: The RDD now looks like: [('the', 1), ('quick', 1), ('brown', 1), ('fox', 1), ('the', 1), ...]

Step 4: Aggregate Counts (Wide Transformation)

.reduceByKey(lambda a, b: a + b)

Method: reduceByKey() is a wide transformation (meaning it requires a shuffle of data). It groups all values associated with the same key and applies an aggregation function.

Function: lambda a, b: a + b specifies the aggregation: for any given word, it sums up all the associated 1 values.

Result: This is the core counting step. The RDD now contains the final word counts as key-value pairs: [('the', 2), ('quick', 1), ('brown', 1), ('fox', 1), ...]. This is saved in the counters RDD.

Step 5: Output Result (Action)

counters.collect()

Method: collect() is an action that triggers the entire chain of transformations (from textFile to reduceByKey) and brings the final result from the distributed Executors back to the Driver Program as a local list.

Result: The final output is a Python list containing the complete word counts.

Resilient Distributed Data Sets (RDDs)

Assume that a file words.txt is localed in /user/bigdata folder in HDFS and it is a home folder of bigdata user
A file words.txt contains the words separated with blanks

The following program computes the count of each word in a file words.txt

   
lines_rdd = sc.textFile("words.txt")

 
counters = (lines_rdd.flatMap(lambda line: line.split(" "))
                     .map(lambda word: (word, 1))
                     .reduceByKey(lambda a, b: a + b) )

counters.collect()

Introduction to Spark

A DataFrame is the most commonly used and arguably the simplest data model in modern Spark. It represents a distributed collection of data organized into named columns, much like a table in a relational database or a data frame in R/Python (e.g., pandas).

Core Characteristics

Tabular Structure: A DataFrame is fundamentally a table of data with rows and columns.

Schema Defined: Unlike the low-level RDD, a DataFrame has a schema. The schema is the list of columns in the DataFrame along with the defined data types for those columns (e.g., String, Integer, Timestamp). Having a schema allows Spark to perform optimizations (via the Catalyst Optimizer) that are not possible with RDDs.

Distributed: A DataFrame is designed to span over many systems in a cluster. This distributed nature is what allows Spark to handle Big Data efficiently.

Parallelism and Partitions

DataFrames achieve parallel processing through partitions:

Partitions: A DataFrame consists of zero or more partitions. To parallelize data processing, all data is divided into logical chunks called partitions.

Physical Location: A partition is a collection of rows located on a single system (a Worker Node) in the cluster.

Parallel Processing: When operating on a DataFrame (e.g., applying a filter or join), Spark simultaneously processes all relevant partitions using separate Tasks across the cluster.

Data Sharing and Shuffle

Shuffle Operation: A Shuffle operation is performed to share data among the systems in a cluster. This is necessary for wide transformations like join, groupBy, or orderBy, where data with the same key must be moved from various partitions on different worker nodes to a single new partition for aggregation or grouping.

DataFrames

A DataFrame is a table of data with rows and columns
A list of columns in DataFrame together with the types of columns is called as a schema
A DataFrame can span over many systems in a cluster
The concepts similar to DataFrame are used in Python and R
A DataFrame is the simplest data model available in Spark
A DataFrame consist of zero or more partitions
- To parallelize data processing all data are divided into chunks called as partitions
- A partition is a collection of rows located on a single system in a cluster
- When operating DataFrame Spark simultaneously processes all relevant partitions
- Shuffle operation is performed to share data among the systems in a cluster

Creating the SparkSession

Before you can work with DataFrames or any structured data in modern Spark, you need an entry point called the SparkSession. It supersedes the older SparkContext and SQLContext by unifying all the context objects used for data processing.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("Demo").getOrCreate()

from pyspark.sql import SparkSession: Imports the necessary class.

SparkSession.builder: Returns a builder object used to configure the session.

.appName("Demo"): Assigns a name to your Spark application, which is useful for monitoring in the cluster manager.

.getOrCreate(): This is the key method. It checks if a SparkSession already exists; if it does, it returns the existing one. If not, it creates a new one based on the configuration provided. The resulting object, typically named spark, is your gateway to all Spark SQL and DataFrame functionality.

Creating a DataFrame

DataFrames can be created from many sources (files, databases, etc.), but the simplest way is by parallelizing a local collection and providing schema hints.

Python

people = spark.createDataFrame([("James", 25),("Harry",17),("Robin",40)],["name","age"])

spark.createDataFrame(): A method of the SparkSession used to create a DataFrame.

First Argument (Data): A list of tuples [("James", 25),...] provides the rows and values for the data.

Second Argument (Schema): A list of strings ["name", "age"] provides the column names (the schema) for the DataFrame. Spark infers the data types (e.g., String for name and Integer for age) based on the provided values.

Listing a DataFrame

The easiest way to inspect the data and schema of a DataFrame is using the show() action.

people.show()

show(): This is an action that triggers the computation of the DataFrame and prints the first 20 rows (by default) to the console in a neatly formatted, tabular layout. This is invaluable for debugging and quick inspection.

The output clearly displays the column names and the data rows, confirming the structure of the DataFrame

DataFrames

Working with DataFrames starts from creating Spark Session

 
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("Demo").getOrCreate()

A DataFrame can be created in the following way

 
people = spark.createDataFrame([("James", 25),("Harry",17),("Robin",40)],["name","age"])

A DataFrame can be listed in the following way

 
people.show()

 
+-----+---+
| name|age|
+-----+---+
|James| 25|
|Harry| 17|
|Robin| 40|
+-----+---+

The schema defines the structure of your data and is crucial for Spark's optimization. You can easily view it using the printSchema() action.

people.printSchema()

root

|-- name: string (nullable = true)

|-- age: long (nullable = true)

Purpose: printSchema() displays the hierarchy of the data (root), lists each column name, specifies its data type (e.g., string, long), and indicates whether the column accepts null values (nullable = true).

Optimization: Spark's Catalyst Optimizer uses this schema to generate highly optimized physical plans for execution.

Adding a New Column with Conditional Logic

The withColumn() transformation is used to add a new column to a DataFrame or replace an existing one. It is often combined with conditional functions like when() to define the new column's values based on existing data.

Defining the Logic

The example code creates a new column called life_stage based on the values in the age column:

new_people = people.withColumn("life_stage",

when(col("age") < 13, "child")

.when(col("age").between(13, 19), "teenager")

.otherwise("adult") )

people.withColumn("life_stage", ...): Starts the transformation, defining the new column name as "life_stage".

when(col("age") < 13, "child"): If the value in the age column is less than 13, the new column value is "child".

.when(col("age").between(13, 19), "teenager"): If the first condition is false, check if age is between 13 and 19 (inclusive); if so, the value is "teenager".

.otherwise("adult"): If none of the preceding conditions are met, the value defaults to "adult".

Showing the Result

The resulting DataFrame, new_people, now includes the derived column

DataFrames

A schema of a DataFrame can be listed in the following way

 
people.printSchema()

	    
root
 |-- name: string (nullable = true)
 |-- age: long (nullable = true)

It is possible to add a column an already existing DataFrame

new_people = people.withColumn("life_stage", when(col("age") < 13, "child")
...                                          .when(col("age").between(13, 19), "teenager")
...                                          .otherwise("adult") )

 
new_people.show()

    
+-----+---+----------+
| name|age|life_stage|
+-----+---+----------+
|James| 25|     adult|
|Harry| 17|  teenager|
|Robin| 40|     adult|
+-----+---+----------+

Creating a DataFrame from Dictionaries

The code snippet shows how to create a DataFrame by passing a list of Python dictionaries to the spark.createDataFrame() method:

people = spark.createDataFrame([{"name":"James","age":25},

{"name":"Harry","age":24},

{"name":"Robin","age":50}])

Row Representation: Each dictionary in the list represents a single row in the resulting DataFrame.

Automatic Schema Inference: Spark automatically infers the column names from the dictionary keys ("name" and "age") and the data types from the associated values (e.g., "James" suggests string, 25 suggests long). This eliminates the need to manually specify a separate schema list.

Displaying Data and Schema

The following output confirms that the DataFrame was created correctly with an inferred schema.

Data Output (people.show())

The data is displayed in a standard tabular format.

The column ordering in the output (e.g., age then name) is often alphabetical by default when created this way, but the key point is that the data is correctly mapped to its respective column name.

Schema Output (people.printSchema())

root

|-- age: long (nullable = true)

|-- name: string (nullable = true)

Data Types: Spark correctly inferred that name is a string and age is a long (a 64-bit integer type often used by Spark for numerical columns without a specific type hint).

Advantage: This method is preferred for its simplicity and the automatic, correct schema inference, especially for development and smaller datasets.

DataFrames

Creating a DataFrame in a different way

 
people = spark.createDataFrame([{"name":"James","age":25},
...                             {"name":"Harry","age":24},
...                             {"name":"Robin","age":50}])

 
people.show()

 
+---+-----+                                                                     
|age| name|
+---+-----+
| 25|James|
| 24|Harry|
| 50|Robin|
+---+-----+

 
people.printSchema()

 
root
 |-- age: long (nullable = true)
 |-- name: string (nullable = true)

Reading DataFrames from HDFS Files

The modern way to load data into Spark is by using the spark.read interface. This interface is part of the SparkSession object (spark) and allows you to specify the data format, path, and various options.

1. Reading JSON Files

The JSON format is naturally semi-structured, which allows Spark to easily infer the schema.

# Creates a DataFrame by reading the people.json file from the default HDFS location.

people = spark.read.json("people.json")

people.show()

Data and Mechanism

The content of people.json is:

{"name":"James","age":25}

{"name":"Harry","age":24}

{"name":"Robin","age":50}

Schema Inference: Spark reads the file and automatically infers the schema by scanning the data. It sees that name holds strings and age holds numbers.

Result: A DataFrame is created with the correct column names and data types, organized into rows:

(Note: The output provided in the prompt shows only two rows, which is common if the show() command only displays a subset.)

2. Reading Generic Text Files (CSV/Delimiter-Separated)

Reading a simple .txt file using the default methods requires additional steps, as the spark.read.text() method would treat each line as a single string column.

To correctly read a delimited file like people.txt (which appears to be a simple CSV/text format):

james,23

john,34

lisa,23

You would typically use the csv format reader:

# Example of reading a comma-separated text file

people_csv = spark.read.csv("people.txt", header=False, inferSchema=True)

people_csv.show()

header=False: Tells Spark that the first line is data, not column headers.

inferSchema=True: Instructs Spark to scan the file to determine the correct data types (e.g., string for names, integer for ages).

Delimiter: Spark automatically uses the comma (,) as the delimiter by default in the CSV reader.

While the prompt only showed the JSON example, understanding how to handle different file types demonstrates the flexibility of the spark.read interface.

DataFrames

Assume that the files people.json and people.txt are uploaded to HDFS

hdfs dfs -cat /user/bigdata/people.json
{"name":"James","age":25}
{"name":"Harry","age":24}
{"name":"Robin","age":50}

hdfs dfs -cat /user/bigdata/people.txt
james,23
john,34
lisa,23

Creating a DataFrame from a file people.json

 
people = spark.read.json("people.json")

people.show()                                                               
+---+-----+
|age| name|
+---+-----+
| 25|James|
| 24|Harry|
| 50|Robin|
+---+-----+

Reading a Raw Text File (spark.read.text)

The spark.read.text() method is designed to read a file where the entire content of each line is treated as a single value.

people = spark.read.text("people.txt")

people.show()

Mechanism and Output

Single Column: The resulting DataFrame will have exactly one column, which Spark automatically names value.

Data Type: The data type of the value column will be string.

Raw Content: Each row in the DataFrame contains the entire line from the source file, including any delimiters (like the comma ,) and newline characters.

This method is generally used when you need to perform custom processing (like splitting or parsing) on the lines using DataFrame functions. If the file is a standard CSV, the spark.read.csv() method (as discussed previously) is usually the correct and more efficient choice.

Creating a DataFrame from Row Objects

You can explicitly construct a DataFrame by defining data elements using the pyspark.sql.Row class, which allows you to define the column names and values for each record.

from pyspark.sql import Row

df = spark.createDataFrame([Row(name="James", age=25)])

df.show()

Mechanism and Output

Row Object: The Row object is a representation of a single record (row) in a DataFrame. When using Row(...), you are directly providing the column names (name, age) as keyword arguments and their corresponding values.

Explicit Schema: This method provides an explicit way to define the schema without relying on inference. Spark uses the field names and data types defined in the Row objects.

Result: A DataFrame is created with the specified column names and a single row of data.

DataFrames

Creating a DataFrame from a file people.txt

 
people = spark.read.text("people.txt")

people.show()                                                               
+--------+
|   value|
+--------+
|james,23|
| john,34|
| lisa,23|
+--------+

One more way of creating a DataFrame

from pyspark.sql import Row

df = spark.createDataFrame([Row(name="James", age=25)])

df.show()
+-----+---+                                                                     
| name|age|
+-----+---+
|James| 25|
+-----+---+

The process of saving a DataFrame is done using the DataFrame.write property, which is an action that triggers the execution of the DataFrame's logic and writes the result to a specified location and format.

1. The write Operation

The core command used to save the people DataFrame is:

people.write.save("/user/bigdata/people", format="csv", mode="append")

Component Description

people.write This is the entry point for writing data to an external sink (location).

.save(...) The method that executes the write operation.

Path (/user/bigdata/people) Specifies the target directory in HDFS where the data will be written. Spark creates this directory if it doesn't exist.

Format (format="csv") Specifies the output file format. Here, the data is written as plain comma-separated values (CSV). Other common formats include Parquet, JSON, and ORC.

Mode (mode="append") Specifies how Spark should handle the target directory if it already contains data. append adds the new DataFrame's data to the existing data in the directory. Other common modes are overwrite (deletes and recreates the directory) and error (throws an exception if the directory exists).

2. Output Structure in HDFS

Similar to saving RDDs, Spark saves the DataFrame as a directory containing multiple part files, reflecting the distributed nature of the operation.

hdfs dfs -ls /user/bigdata/people/*

-rw-r--r-- 1 bigdata supergroup 0 2026-04-17 10:21 /user/bigdata/people/_SUCCESS

-rw-r--r-- 1 bigdata supergroup 27 2026-04-17 10:21 /user/bigdata/people/part-00000-*-c000.csv

_SUCCESS File: This is an empty marker file indicating that the Spark job finished successfully.

part-****.csv File(s): These are the actual data files. There is typically one part file for each partition in the DataFrame at the time of writing. The long, complex filename includes the partition ID and a unique job ID, ensuring uniqueness.

3. Viewing the Content

The hdfs dfs -cat command confirms that the contents of the DataFrame were successfully written in the CSV format:

hdfs dfs -cat /user/bigdata/people/part*

DataFrames

Saving a DataFrame in HDFS

 
hdfs dfs -mkdir /user/bigdata/people

  
people.write.save("/user/bigdata/people", format="csv", mode="append")

 
hdfs dfs -ls /user/bigdata/people/*
-rw-r--r--   1 bigdata supergroup          0 2026-04-17 10:21 /user/bigdata/people/_SUCCESS
-rw-r--r--   1 bigdata supergroup         27 2026-04-17 10:21 /user/bigdata/people/
                                                part-00000-ab78decf-d976-4bcb-b2c9-1e9f7c178fa5-c000.csv

 
hdfs dfs -cat /user/bigdata/people/part*
James,25
Harry,24
Robin,50

Introduction to Spark

Outline

Resilient Distributed Data Sets (RDDs)
DataFrames
SQL Tables/Views
Datasets

Spark allows you to treat a distributed DataFrame as if it were a traditional relational SQL Table or View. This process involves registering the DataFrame with the SparkSession's internal catalog.

1. Registering the DataFrame as a View

Since DataFrames are inherently distributed structures and not literal tables in a database, they are registered as Views (often called temporary tables) for SQL access.

The command to register a DataFrame is:

people.createOrReplaceTempView("people_table")

createOrReplaceTempView(...): This method takes the DataFrame (people) and registers it in Spark's metadata catalog under the specified name ("people_table").

Temporary View: The view is temporary, meaning it exists only for the duration of the current SparkSession. It is not saved to disk like a permanent Hive table.

No Overhead: Critically, there is no performance overhead when registering a DataFrame as a temporary view. The view is simply a pointer to the underlying DataFrame's data and execution plan.

2. Executing SQL Queries

Once the view is registered, you can use the spark.sql() method to execute standard SQL queries against it.

spark.sql("SELECT * from people_table").show()

spark.sql(...): This method takes a SQL string and executes it using the Spark SQL engine.

Result: The output of spark.sql() is always a new DataFrame. This means you can easily switch between DataFrame API methods and SQL throughout your pipeline.

The query returns the full content of the underlying people DataFrame:

This capability allows data teams to use the simple, declarative power of SQL while still benefiting from Spark's distributed processing and optimization capabilities.

SQL Tables/Views

SQL can be used to operate on DataFrames
It is possible to register any DataFrame as a table or view (a temporary table) and apply SQL to it
There is no performance overhead when registering a DataFrame as SQL Table and when processing it

A DataFrame dataFrame can be registered as SQL temporary view people in the following way

 
people.createOrReplaceTempView("people_table")

	      
spark.sql("SELECT * from people_table").show()
+---+-----+
|age| name|
+---+-----+
| 25|James|
| 24|Harry|
| 50|Robin|
+---+-----+

Once you register a DataFrame as a temporary view (e.g., people_table), you can execute any standard SQL query against it. Spark's SQL engine processes these queries in a distributed manner.

1. Running Aggregate Queries

You can use standard SQL aggregation functions (like COUNT, SUM, AVG, etc.).

spark.sql("SELECT count(*) from people_table").show()

Query: This is a basic aggregate query that returns the total number of rows in the people_table.

Output: The result is a single row DataFrame showing the count.

2. Running Filtered Queries

You can use the WHERE clause to filter the data, retrieving only the rows that satisfy a specific condition.

spark.sql("SELECT * from people_table WHERE age>25").show()

Query: This selects all columns (*) from the view where the age is greater than 25.

Output: Only the row for 'Robin' (age 50) satisfies the condition.

3. Creating a New DataFrame from a Query

The key takeaway is that the result of every spark.sql() command is always a new DataFrame. This allows you to chain operations easily.

table_df = spark.sql("SELECT * from people_table WHERE age>25")

table_df.show()

Mechanism: The SQL query is executed, and the resulting data (the filtered set) is wrapped back into a new DataFrame object, named table_df.

Benefit: You can now apply further DataFrame transformations (like withColumn, groupBy, etc.) to table_df or register it as a new temporary view for subsequent SQL queries.

SQL Tables/Views

Accessing a temporary view in SQL

	      
spark.sql("SELECT count(*) from people_table").show()
+--------+
|count(1)|
+--------+
|       3|
+--------+

	      
spark.sql("SELECT * from people_table WHERE age>25").show()
+---+-----+
|age| name|
+---+-----+
| 50|Robin|
+---+-----+

 
table_df = spark.sql("SELECT * from people_table WHERE age>25")
table_df.show()
+---+-----+
|age| name|
+---+-----+
| 50|Robin|
+---+-----+

Introduction to Spark

Outline

Resilient Distributed Data Sets (RDDs)
DataFrames
SQL Tables/Views
Datasets

The Dataset abstraction in Spark is designed to provide the best features of both the low-level RDD (type safety and functional programming) and the high-level DataFrame (performance optimization via the Catalyst Optimizer and columnar storage).

Core Definition and Relationship to DataFrames

Distributed Collection: A Dataset is a distributed collection of data and represents an immutable, partitioned set of objects.

Typed API: Datasets are strongly typed. This means the compiler knows the exact data structure and type of every column, enabling compile-time error checking (a major advantage over DataFrames/RDDs).

DataFrame is a Dataset: A DataFrame is simply a Dataset of type Row (i.e., Dataset[Row]). The Row object is Spark's generic, schemaless internal representation of structured data. When you use a DataFrame, you're relying on Spark to handle the types at runtime.

JVM-Specific Feature

JVM Language Feature: Datasets are a strictly JVM language feature and currently only work with Scala and Java.

Python/R Limitation: Python does not have support for the Datasets abstraction because Python lacks the strong type-checking features and compile-time safety that the Dataset API relies upon. When using PySpark, you are working with DataFrames.

Row-Object Mapping: Datasets consist of objects included in the rows; one object per row. In Scala, a case class object defines the schema of an object, providing the strong typing.

Performance and Serialization

Encoder: Datasets use a specialized Encoder to serialize the objects for processing or transmitting over the network. This is the key performance feature.

The Encoder generates code to serialize and deserialize JVM objects to/from Spark's efficient internal Tungsten binary format. This is faster and uses less memory than standard Java serialization or RDD's object serialization.

Manipulation: A Dataset can be constructed from JVM objects and then manipulated using functional transformations (like map, filter, etc.), but with the benefit of Spark's optimization.

Compatibility

Full Functionality: A Dataset supports all operations of DataFrame (e.g., SQL-like methods like select, where, groupBy). You can always convert a Dataset back to a DataFrame simply by casting it as Dataset[Row].

Datasets

A Dataset is a distributed collection of data
A DataFrame, is a Dataset of type Row
Datasets consists of the objects included in the rows; one object per row
Datasets are a strictly JVM language feature that only work with Scala and Java
Python does not have the support for the Datasets
A Dataset can be constructed from JVM and then manipulated using functional transformations
In Scala a case class object defines a schema of an object
Datasets use a specialized Encoder to serialize the objects for processing or transmitting over the network
Dataset supports all operations of DataFrame

Defining and Creating a Dataset

The example demonstrates the key steps for creating a type-safe Dataset:

1. Defining the Schema with a Case Class

case class Person(name: String, age: Long)

Case Class: The Scala case class is used to define the schema and the data type for the Dataset. It acts as the strongly typed object that will represent each row.

Strong Typing: The types for the fields (String for name and Long for age) are fixed at compile time, which is the foundation of the Dataset's type safety.

2. Creating the Dataset

val caseClassDS = Seq(Person("Andy", 32)).toDS()

Local Collection: A standard Scala Seq (sequence/list) of Person objects is created.

Conversion: The powerful .toDS() method converts this local, strongly-typed collection into a distributed Dataset of Person objects (Dataset[Person]).

Result: caseClassDS is now a distributed, type-safe collection, ready for parallel processing by Spark.

Using the Dataset

Displaying the Data (.show())

caseClassDS.show()

The .show() action triggers the computation and displays the content in a familiar tabular format, just like a DataFrame.

Selecting Columns (.select())

caseClassDS.select($"name").show()

Column Selection: The .select() transformation is used to pick specific columns.

Column Access: The syntax $"name" is shorthand in Scala for accessing the column named "name".

Performance: Although the operation uses the friendly, high-level API syntax (like a DataFrame), under the hood, Spark's Encoders handle the efficient serialization and execution, ensuring high performance.

Datasets

A Dataset can be defined, created, and used in the following way

	      
case class Person(name: String, age: Long)

	      
val caseClassDS = Seq(Person("Andy", 32)).toDS()

	      
caseClassDS.show()

	      
+----+---+
|name|age|
+----+---+
|Andy| 32|
+----+---+


caseClassDS.select($"name").show()

	      
+-----+
| name|
+-----+
|James|
+-----+

References

A Gentle Introduction to Spark, Databricks, (Available in READINGS folder)
RDD Programming Guide
Spark SQL, DataFrames and Datasets Guide
Karau H., Fast data processing with Spark Packt Publishing, 2013 (Available from UOW Library)
Srinivasa, K.G., Guide to High Performance Distributed Computing: Case Studies with Hadoop, Scalding and Spark Springer, 2015 (Available from UOW Library)
Chambers B., Zaharia M.,Spark: The Definitive Guide, O'Reilly 2017