20 SparkIntroduction

Introduction to Spark

Outline

MapReduce Challenges
Meet Spark !
Features of Spark
Spark Architecture
Overview of Spark Components
Spark Glossary

While Google's MapReduce concept (and its implementation in Hadoop) was revolutionary for batch processing of large datasets, two primary challenges quickly emerged: its low-level programming model and its inefficiency for iterative/interactive workloads.

1. Low-Level Programming API

The core Hadoop MapReduce framework requires developers to manually define the map and reduce functions using a low-level Java API.

Complexity: Implementing common data operations becomes non-trivial. For example, implementing an inner join involves:

Map Phase: Reading records from both input datasets (tables), tagging each record with its source (e.g., 'A' or 'B'), and emitting the join key and the tagged record value.

Shuffle Phase: Grouping all records with the same join key together.

Reduce Phase: Iterating over the list of values for a given key, pairing records from source 'A' with records from source 'B', and emitting the joined result.

Verbosity and Debugging: This process is verbose, prone to errors, and requires developers to think in terms of keys, values, and distributed shuffling, rather than higher-level SQL-like logic.

The Solution: Frameworks like Pig (a scripting language) and Hive (a SQL-like query language, or HiveQL) were created on top of MapReduce. These tools allow users to express complex data transformations in a much simpler way, which the framework then compiles into a sequence of MapReduce jobs.

2. Inefficiency for Interactive and Iterative Computation

The most significant performance challenge of MapReduce stems from its reliance on the Hadoop Distributed File System (HDFS) for intermediate results.

Persistent Storage I/O: Every MapReduce job is a standalone process. Data must be read from HDFS at the start of the job and written back to HDFS at the end.

Interactive Computation: If you run a quick query and then another follow-up query based on the first result, the second query must re-read the result from disk, incurring high I/O latency.

Iterative Computation (e.g., Machine Learning/Graph Processing): Algorithms like K-means or PageRank require multiple passes over the same dataset, with the output of one pass serving as the input for the next. In MapReduce, each iteration involves a full read from HDFS and a full write to HDFS. This constant disk I/O is the bottleneck.

The Solution: This fundamental inefficiency led to the development of new computing frameworks, most notably Apache Spark. Spark introduced the concept of Resilient Distributed Datasets (RDDs), which allows intermediate data to be stored in-memory across the cluster, drastically improving the performance of iterative and interactive workloads by eliminating the constant read/write latency to disk.

In summary, MapReduce excels at large-scale batch processing but is limited by its low-level API (solved by Hive/Pig) and its reliance on disk I/O between job steps, making it slow for interactive and iterative tasks (solved by frameworks like Spark).

MapReduce Challenges

Google introduced MapReduce, which was a breakthrough in the history of big data technologies
- It makes the processing of big data feasible and practical
However, Hadoop MapReduce framework releases the developer from the distributing computing trickiness, its Java API is still too low-level
- Think how would you implement an inner join in MapReduce ?
- Hive, Pig and other frameworks based on MapReduce can help
More importantly, the persistent storage I/O makes the interactive and iterative computation (two important forms of data processing) very inefficient and leads to to potential I/O latency

The most significant performance issue in Hadoop MapReduce is the reliance on the Hadoop Distributed File System (HDFS) for every intermediate data transfer, which creates substantial overhead.

1. Persistent Storage I/O

The Problem: In the MapReduce model, the output of one job (or the intermediate output of iterative computations) must be written to HDFS before the next job or iteration can begin.

The Impact: This mandatory writing and subsequent reading from disk (persistent storage) introduces high I/O latency. As the summary notes, up to 90% of the time spent running a MapReduce job can be dedicated to these HDFS read-write operations, making the overall process extremely slow.

Inefficiency: This design decision prioritizes fault tolerance and scalability at the expense of speed. If a node fails, the data is safe on HDFS, but the constant disk access slows down the execution dramatically, especially for multi-step or iterative jobs.

2. Substantial Overhead Components

Beyond the raw disk speed, several other factors contribute to the significant overhead when sharing data:

Overhead Component Description Impact

Data Replication HDFS, by default, stores at least three copies of every data block across the cluster for fault tolerance. Increases network traffic and write time, as data must be copied to multiple nodes.

Serialization Data structures in memory must be converted into a byte stream (serialized) before being written to HDFS, and then converted back (deserialized) when read by the next job. Adds CPU processing overhead at both the write and read stages of data transfer.

Persistent Storage I/O The fundamental time spent waiting for the physical hard drives to write and read the data. The primary time bottleneck (the 90% mentioned), especially compared to reading/writing from faster RAM.

The Solution: In-Memory Computing

The inability of MapReduce to efficiently reuse data across computation steps led directly to the rise of Apache Spark.

Spark's core innovation is the Resilient Distributed Dataset (RDD), which allows intermediate results to be kept in-memory across the cluster's nodes. This eliminates the persistent storage I/O between computation steps, meaning:

No HDFS Write/Read: Data is read from HDFS once, processed in memory, and only written back to HDFS when the final result is needed.

Speed Boost: Iterative algorithms (like those used in machine learning) become orders of magnitude faster because the data stays resident in RAM instead of being constantly written to and read from disk.

MapReduce Challenges

Data Sharing in Hadoop
- In the Hadoop MapReduce framework, the reuse data between computations (e.g., between two MapReduce jobs) is to write it to an external stable storage system, for example HDFS)
- Although Hadoop provides numerous abstractions for data access, data sharing is slow due to replication, serialisation and persistent storage IO
- More than 90% of the time for running a MapReduce job is doing HDFS read-write operations
- Substantial overhead is caused by data replication, persistent storage I/O, and serialization

Introduction to Spark

The Speed of Spark: In-Memory Computing

Spark achieves its dramatic speed improvement��often cited as up to 100x faster for iterative jobs��by prioritizing in-memory computing.

In-Memory Data Sharing: Unlike MapReduce, which writes intermediate results to persistent storage (like HDFS) after every map or reduce step, Spark uses Resilient Distributed Datasets (RDDs), DataFrames, and Datasets to hold data in the cluster's RAM between operations.

For Iterative Computations (e.g., machine learning algorithms like K-means, which repeatedly pass over the same data), this eliminates the bottleneck of constant disk I/O, allowing subsequent iterations to access the data at memory speed.

For Interactive Computations (e.g., ad-hoc queries), the results of previous queries can be cached in memory for much faster follow-up analysis.

Optimized Execution: Spark uses a Directed Acyclic Graph (DAG) scheduler to build an optimized plan for a series of transformations, avoiding unnecessary shuffles and operations. This is far more efficient than MapReduce's fixed, two-stage execution model.

The Generality of Spark: A Unified Engine

Spark is a general-purpose platform, meaning it is not limited to just batch processing. It provides a unified environment that allows a single application to seamlessly combine multiple types of data processing workloads.

Spark Component Primary Workload Description

Spark Core General Processing The engine providing in-memory computation, task scheduling, and fault tolerance.

Spark SQL Interactive Queries/Batch Enables querying structured data using standard SQL syntax.

Spark Streaming (or Structured Streaming) Real-Time Processing Processes live data streams in small batches or continuously for near real-time analytics.

MLlib Advanced Analytics/Iterative A machine learning library for tasks like classification, regression, and clustering.

GraphX Graph Computation An API for graph-parallel computation (e.g., PageRank).

Multi-Language Support

Spark provides rich, high-level APIs in several popular languages, which significantly lowers the barrier to entry and simplifies application development compared to the low-level Java API of MapReduce:

Scala, Python (PySpark), Java, R, SQL

This versatility makes Spark the central data processing engine for modern data engineering, data science, and business intelligence workloads.

Meet Spark !

Apache Spark, as a cluster computing platform, is designed to be fast and general-purpose
On the speed, Spark extends the MapReduce model to efficient support more types of computations, for example, interactive and iterative computations
- This is mainly due to Spark's in-memory computing, although Spark is also more efficient than MapReduce for persistent storage complex applications
On the generality, Spark covers a wide ranges of workloads, including batch applications, iterative queries, streaming and advanced analytics
- It provides a unified environment makes the combination of those different engines easy and inexpensive
- It provides multiple language APIs, including Scala, Python, Java, SQL, and R, as well as a very rich (yet fast-growing) built-in libraries

Spark's Role for Data Professionals

Spark's speed and versatility make it the go-to platform for both Data Scientists (focused on analysis and modeling) and Data Engineers (focused on production and infrastructure).

Data Scientists and Spark

Data Scientists use Spark to handle the heavy computational lifting required for sophisticated analysis and model building on massive datasets. They leverage Spark's libraries and distributed processing to:

Analyze and Model Data: Use Spark's distributed processing capabilities to efficiently run complex statistical analysis and exploratory data analysis (EDA) on terabytes or petabytes of data. They often use the MLlib library for distributed machine learning algorithms.

Make Predictions Based on Data: Train and evaluate machine learning models (e.g., classification, regression, clustering) using tools like PySpark (Spark's Python API) or the R API. They scale their modeling efforts far beyond what a single machine could handle.

Build Data Pipelines for Analysis: Create transformation pipelines��often using Spark DataFrames��to clean, prepare, and feature-engineer raw data into the structured inputs required for their models.

Data Engineers and Spark

Data Engineers are the architects and builders of the data infrastructure. They use Spark as the core engine for moving, transforming, and managing data at scale. Their primary Spark tasks include:

Develop Data Processing Applications: Build robust, scalable, and fault-tolerant Extract, Transform, Load (ETL) or Extract, Load, Transform (ELT) pipelines. This involves using Spark SQL or the DataFrame API to join, aggregate, filter, and cleanse data from various sources (like HDFS, S3, or databases) and prepare it for consumption.

Deploy the Output of Data Scientists in Production: Take the trained machine learning models (the "output") developed by Data Scientists and integrate them into production systems. This involves:

Creating scoring pipelines that apply the model to new, incoming data in real-time or in batches.

Ensuring the necessary data preprocessing steps defined by the Data Scientist are correctly and efficiently implemented on the production data flow using Spark.

Meet Spark !

Data Scientists use Spark to:
- Analyse and model data
- Make prediction based on data
- Build data pipelines to fulfil certain tasks
Data Engineers use Spark to:
- Develop data processing applications
- Deploy the output of data scientists in production

Introduction to Spark

Outline

MapReduce Challenges
Meet Spark !
Features of Spark
Spark Architecture
Overview of Spark Components
Spark Glossary

The features of Apache Spark make it a powerful, fast, and versatile cluster computing engine designed to address the limitations of traditional MapReduce.

Speed Through In-Memory Computing

Spark's most fundamental feature is its ability to perform in-memory cluster computing, which drastically increases application speed.

Eliminates I/O Bottlenecks: Spark stores intermediate processing data in memory (RAM) across the cluster nodes, unlike MapReduce, which requires writing data back to persistent storage (like HDFS) between every step.

Performance Gains: This reduction in read/write operations to disk allows Spark applications to run up to 100 times faster in memory for iterative jobs (which repeat a sequence of operations) and still be about 10 times faster when running on disk compared to MapReduce.

Versatile Computing: This speed allows Spark to support not just traditional batch computing, but also interactive, iterative, and real-time computing, which were inefficient or impossible with MapReduce.

Flexible and Unified Programming Model

Spark uses a more flexible and powerful programming model than the rigid MapReduce framework.

Directed Acyclic Graph (DAG) Model: Spark's processing is based on a DAG, which is a sequence of computation stages optimized by a scheduler. This allows Spark to build a highly efficient execution plan for complex multi-step transformations, unlike MapReduce's fixed, two-stage (Map then Reduce) model.

High-Level APIs for Structured Data: Spark provides built-in high-level APIs like DataFrames and Datasets that allow developers to process structured data (e.g., tables in Hive) much more easily and efficiently than the low-level Key-Value pairs of MapReduce.

Reduced Management Burden: By supporting a wide variety of workloads (batch, interactive, streaming, and analytics) within a single framework, Spark reduces the management burden of maintaining separate tools and systems for each type of task.

Features of Spark

As mentioned, Spark supports not only batch computing (as MapReduce does) but also interactive, iterative, and real-time computing
The main feature of Spark is its in-memory cluster computing that increases the processing speed of an application
The programming model of Spark is based on Directed Acyclic Graphs (DAG) and it is more flexible than the MapReduce model
Unlike MapReduce, Spark has built-in high-level APIs to process structured data, e.g., tables in Hive
Spark reduces the management burden of maintaining separate tools

Features of Spark

Speed
Spark helps to run an application in Hadoop cluster, up to 100 times faster in memory, and 10 times faster when running on disk
- This is achieved by reducing number of read/write operations to persistent storage
- It stores the intermediate processing data in memory
Support for multiple languages
Spark provides built-in APIs in Java, Scala, or Python
Advanced Analytics
Spark not only supports MapReduce It also supports SQL queries, Streaming data, Machine Learning (ML), and Graph algorithms

Introduction to Spark

Outline

MapReduce Challenges
Meet Spark !
Features of Spark
Spark Architecture
Overview of Spark Components
Spark Glossary

Spark Architecture Components

A Spark application runs as an independent set of processes on a cluster, coordinated by the SparkContext in the Driver Program.

1.Driver Program

The Driver Program is the main Spark application. It runs the user's main() function and is responsible for:

Logic: It contains the overall data processing logic, where you define the transformations (like map, filter, join) and actions (like collect, count, save).

Coordination: It houses the SparkContext, which connects to the cluster manager (like YARN, Mesos, or Standalone) and coordinates the distributed computation.

Job Planning: It translates the high-level application code into a series of Stages and Tasks (a Directed Acyclic Graph or DAG) and schedules them to be executed by the executors.

2.Executor

An Executor is a JVM (Java Virtual Machine) process that runs on each Worker Node in the cluster. It is the distributed "worker" component responsible for the actual data processing:

Task Execution: It executes the tasks assigned to it by the Driver Program.

Data Storage: It holds the data that is cached or persisted by the user's application (Spark's key in-memory storage).

Communication: Executors report their status and results back to the Driver Program.

3.Task

A Task is the smallest unit of work in a Spark application.

Sub-component of a Job: A Spark Job (triggered by an action like collect) is broken down into Stages, and each Stage is broken down into a set of independent Tasks.

Execution Unit: Each task is sent to an Executor to run on a partition of the data. For example, a map transformation task might be applied to one data partition while other identical map tasks run on other partitions concurrently across the cluster.

Spark Architecture

Driver program is the main Spark application that consists of the data processing logic
Executor is a JVM process that runs on each worker node and processes the jobs that the driver program submits
Task is a subcomponent of a data processing job

1.Client Mode (Interactive Mode)

In Client Mode, the Spark Driver Program runs on the machine where the application was submitted (the client or gateway node).

Driver Location: Runs outside the cluster, typically on the client machine.

Execution Flow:

The client launches the Driver Program.

The Driver requests resources from the YARN Resource Manager.

YARN grants the resources, launching Executors on the worker nodes.

The Driver communicates directly with the Executors to coordinate tasks.

Use Case: This mode is best for interactive workloads, such as using the Spark Shell, where the user needs immediate feedback and the Driver's logs are visible locally.

Risk: If the client machine that runs the Driver goes down, the entire Spark application will fail.

2.Cluster Mode (Production Mode) ??

In Cluster Mode, the Spark Driver Program runs inside the cluster on one of the worker nodes, managed by YARN.

Driver Location: Runs inside a YARN container (called the Application Master) on a cluster worker node.

Execution Flow:

The client submits the application and immediately terminates.

YARN launches an Application Master container on a worker node, which in turn launches the Driver Program.

The Driver requests and launches the Executors on the other worker nodes.

The Driver manages the job completely within the cluster.

Use Case: This is the preferred mode for production jobs (non-interactive, long-running batch jobs) because the application is fully independent of the client machine.

Benefit: The Spark job continues to run even if the client machine that initiated the submission disconnects or goes offline, making it more fault-tolerant.

Spark Architecture

Two modes of Building Spark on Hadoop
These modes are different from full-distributed mode and pseudo-distributed mode

Spark Architecture

Spark on YARN: Client-Mode

Spark on YARN: Cluster Mode

Cluster Mode is the preferred deployment configuration for running production-ready, long-running batch jobs on a Hadoop YARN cluster. In this mode, the crucial Spark Driver Program runs inside the cluster, making the application robust and fault-tolerant against client failures.

1. The Role of Components

Component Location Responsibility

Client User's Submission Machine Submits the application JAR/code to the YARN Resource Manager and then disconnects.

Application Master (AM) YARN Worker Node Runs the Driver Program inside its container. It coordinates resources and task execution.

Driver Program Application Master Container Analyzes the code, creates the execution plan (DAG), requests Executor containers from YARN, and coordinates tasks.

Executors YARN Worker Nodes Execute the individual tasks, store cached data, and report results back to the Driver (which is inside the AM).

2. Execution Flow

Submission: The user submits the Spark application using the spark-submit command, specifying "cluster" deployment mode.

AM Launch (Driver): The YARN Resource Manager finds a suitable Worker Node and launches the Application Master (AM) container. Unlike Client Mode, the Driver Program is immediately launched inside this AM container.

Client Disconnect: Once the AM/Driver is running, the client machine that submitted the job can disconnect without affecting the application's execution.

Executor Request: The Driver Program (inside the AM) connects to the YARN Resource Manager and requests the necessary Executor containers.

Execution: Once the Executors are allocated, the Driver dispatches tasks to them. The entire application runs entirely within the cluster's resource management system.

3. Key Characteristics

Fault Tolerance: This is the primary advantage. Since the Driver is running in a managed YARN container on a Worker Node, the application is isolated from the submission client. If the client machine crashes or loses network connectivity, the job continues to run to completion.

Production Use: It is the standard mode for production ETL (Extract, Transform, Load) pipelines and machine learning model training jobs due to its reliability and independence.

Resource Management: The Application Master/Driver is responsible for managing its own resource allocation, providing a clean separation of concerns within the cluster.

Spark Architecture

Spark on YARN: Cluster-Mode

In-Memory Data Persistence

Spark's core design feature is its capability to store intermediate data and application results in the cluster's distributed RAM (Random Access Memory), rather than writing to and reading from disk (like HDFS) after every operation.

1. Reduced I/O Overhead

The Problem in MapReduce: In MapReduce, every task or job step involves reading data from disk and writing intermediate results back to disk. This continuous persistent storage I/O is slow, as disk access speeds are magnitudes slower than memory access.

The Spark Solution: Spark utilizes its core data structures (RDDs, DataFrames, Datasets) to cache or persist data in memory. When a subsequent operation needs this data, it's accessed directly from RAM, bypassing the slow disk I/O step. This is the source of Spark's reported 10x to 100x speedup over MapReduce.

2. Fast Access for Iterative and Interactive Computing

Iterative Computations: Algorithms common in Machine Learning (ML) and graph processing (like PageRank) involve many passes (iterations) over the same dataset. By persisting the dataset in memory, each subsequent iteration starts immediately without waiting for a full disk read. This makes complex ML training feasible at scale.

Interactive Queries: When a user runs an ad-hoc query and then runs several follow-up queries based on the first result, Spark can keep the result set cached in memory. This enables fast access for interactive analysis and allows Data Scientists and Analysts to explore data much more quickly.

In short, data persists in memory in Spark, enabling fast access and dramatically reducing the need for costly disk reads and writes, which is the definition of reducing I/O overhead.

Spark Architecture

Writing to and reading from memory in Spark reduces I/O overhead
Data persists in memory and enables fast access

High-Level API Eases Development

Spark moves beyond the rigid, low-level key-value pair programming model of MapReduce, offering APIs that are more abstract and expressive.

No Specific Pattern (MapReduce): Developers are not forced to conform to the specific MapReduce pattern (always Map then Shuffle then Reduce). Instead, they can express their data transformations using familiar concepts like filtering, joining, grouping, and selecting columns, similar to writing SQL or using methods in Python/Scala collections.

Intuitive Code: APIs like DataFrames organize data into named columns, making the code more readable, intuitive, and closer to how data is managed in a traditional database. This drastically reduces the complexity, verbosity, and debugging time associated with writing data pipelines.

Intelligent Execution Planning and Optimization

This simplicity for the developer doesn't sacrifice performance because Spark employs a sophisticated optimization framework "under the hood."

Logical Plan: When a developer writes a series of transformations (e.g., filter, join, groupBy), Spark first creates a Logical Plan. This plan is an abstract, step-by-step description of the requested computation without regard for how it will be executed on the cluster. It represents the what of the job.

Optimized Logical Plan: Spark's Catalyst Optimizer then optimizes this logical plan. It performs smart transformations like predicate pushdown (filtering data early) and column pruning (ignoring unneeded columns) to reduce the amount of data processed.

Physical Plan: Spark then translates the optimized logical plan into a Physical Plan. This plan determines the how��it specifies the actual low-level execution details, such as how data will be shuffled, which join algorithms will be used, and how tasks will be distributed to executors. Spark often tries different physical plans and selects the most efficient one.

This separation allows developers to focus on what they want to achieve (the business logic) while Spark's engine focuses on how to execute it most efficiently across the distributed cluster.

Spark Architecture

High-level API of Spark eases the development of a data processing pipeline
- No need to confirm to a specific pattern, for example MapReduce
- Under the hood, Spark determines the best logical plan and translate it to a physical plan

The five main components of the Apache Spark ecosystem are the Spark Core engine and the four specialized libraries built on top of it. This layered structure allows Spark to handle a wide variety of data workloads within a single, unified environment.

1.Spark Core (The Foundation)

This is the fundamental execution engine for all Spark functionality. It provides the low-level functions for distributed data processing.

Key Functions: Task dispatching, scheduling, basic I/O (Input/Output), fault recovery, and interaction with various storage systems (like HDFS, S3, etc.).

API: It provides the core API for Resilient Distributed Datasets (RDDs), Spark's original distributed data collection abstraction.

2.Spark SQL

This component allows users to interact with structured data using familiar relational concepts.

Workload: Interactive queries and batch processing of structured data.

Key Abstractions: DataFrames and Datasets, which are optimized, column-oriented views of data.

Functionality: Enables running standard SQL queries against data stored anywhere, including Hive tables, JSON files, or Parquet files. It utilizes the Catalyst Optimizer for efficient execution.

3.Spark Streaming / Structured Streaming

This library processes live streams of data from sources like Kafka, Flume, or Kinesis.

Workload: Real-time data processing and analytics.

Key Concept: Structured Streaming treats a live data stream as an continuously appended table, allowing you to use the same DataFrame/Dataset and Spark SQL APIs for stream processing as you would for static batch data.

4.MLlib (Machine Learning Library)

This is Spark's scalable machine learning library, offering a wide range of algorithms and utilities.

Workload: Advanced analytics, predictive modeling, and iterative computation.

Functionality: Provides distributed algorithms for tasks like classification, regression, clustering, and collaborative filtering, enabling data scientists to build models on massive datasets.

5.GraphX

This is an API and optimized execution engine for graph-parallel computation.

Workload: Graph analytics (social networks, routing).

Functionality: Includes a collection of graph algorithms and graph builders, enabling distributed processing of graph-structured data (nodes and edges) for analysis like PageRank.

Spark Architecture

Components of Spark

Introduction to Spark

Outline

MapReduce Challenges
Meet Spark !
Features of Spark
Spark Architecture
Overview of Spark Components
Spark Glossary

Spark Core

Spark Core is the heart of the Spark platform; it's the fundamental execution engine upon which all other Spark functionalities are built.

Foundation: It handles the basic functionality, including task scheduling, memory management, fault recovery, and interacting with external storage systems.

In-Memory Computing: It provides the mechanism for in-memory computing by managing distributed collections called Resilient Distributed Datasets (RDDs). This is key to Spark's speed, allowing data to be held in RAM for fast access.

Storage Access: It provides methods for referencing datasets in external storage systems (like HDFS, S3, Cassandra, etc.) and reading that data into the RDD abstraction for parallel processing.

Spark SQL

Spark SQL is a component layered on top of Spark Core that is designed to work with structured and semi-structured data.

Abstraction: It introduced the DataFrame and Dataset abstractions (which superseded the initial SchemaRDD concept). These high-level abstractions organize data into named columns, making it much easier to work with, similar to a table in a relational database.

Functionality: It enables the use of standard SQL queries to analyze data. It integrates with existing systems like Hive and allows complex relational operations to be executed with high performance due to the Catalyst Optimizer.

Spark Streaming

Spark Streaming is a library that enables processing of live data streams by utilizing Spark Core's speed and fault-tolerance.

Streaming Analytics: It leverages Spark Core's fast scheduling capability (low-latency task execution) to perform streaming analytics.

Micro-Batching: The original Spark Streaming worked by dividing the live stream into tiny, continuous batches (called DStreams). These batches are then processed by the Spark engine as a sequence of quick, short batch jobs. (Note: The newer, preferred approach is Structured Streaming, which treats the incoming stream as an infinite table).

Overview of Spark Components

Spark Core
- Spark Core is the underlying general execution engine for Spark platform that all other functionality is built upon
- It provides In-Memory computing and referencing datasets in external storage systems
Spark SQL
- Spark SQL is a component on top of Spark Core that introduces a new data abstraction called SchemaRDD, which provides support for structured and semi-structured data
Spark Streaming
- Spark Streaming leverages Spark Core fast scheduling capability to perform streaming analytics

SparkML (Machine Learning Library)

SparkML is Spark's scalable and distributed machine learning (ML) framework.

Distributed ML: It was built on Spark because the distributed, memory-based Spark architecture is perfectly suited for the intensive, iterative computations required by most machine learning algorithms.

Pipeline API: It provides high-level APIs for building ML pipelines. This allows data scientists to easily combine multiple steps��such as feature extraction, transformation, and model training��into a single, unified workflow that can be executed across the cluster.

Functionality: It includes a comprehensive set of algorithms for common ML tasks, including classification, regression, clustering, and dimensionality reduction, all optimized for big data.

GraphX (Graph Processing)

GraphX is a distributed graph-processing framework that runs on top of Spark. It enables large-scale, fault-tolerant analysis of data modeled as graphs (networks of interconnected objects).

Graph Abstraction: It provides a specialized API for expressing graph computation, defining data in terms of vertices (nodes) and edges (connections).

Optimized Runtime: It leverages Spark's core engine to provide an optimized runtime for this abstraction, allowing algorithms like PageRank or connected components to run efficiently across a distributed cluster.

SparkR (R Package)

SparkR is an R package that allows users to seamlessly interact with Apache Spark from the R environment.

Light-Weight Frontend: It acts as a light-weight frontend to use Spark from R. Data scientists who prefer the statistical and graphical capabilities of the R language can utilize Spark's distributed computation engine without needing to switch to Scala or Python.

DataFrames: It enables R users to interact with Spark DataFrames, making it possible to run large-scale data manipulation and machine learning tasks using familiar R syntax, but with the performance and scalability of Spark.

Overview of Spark Components

SparkML
- SparkML is a distributed machine learning framework above Spark because of the distributed memory-based Spark architecture
GraphX
- GraphX is a distributed graph-processing framework on top of Spark
- It provides an API for expressing graph computation and provides an optimized runtime for this abstraction
SparkR
- SparkR is an R package that provides a light-weight frontend to use Spark from R

Introduction to Spark

Outline

MapReduce Challenges
Meet Spark !
Features of Spark
Spark Architecture
Overview of Spark Components
Spark Glossary

1. Application

An Application is the overall program or job written by a user in one of Spark's supported languages (Scala, Python, Java, R, or SQL).

It contains the complete data processing logic, which starts with the user's main() function.

It is executed across a cluster of machines and coordinated by a Driver Program and distributed across Executors.

2. Application JAR

The Application JAR (Java Archive) is the self-contained package of your Spark code and all its dependencies.

When you write a Spark program in Java or Scala, you compile it into a .jar file.

This file is submitted to the cluster via spark-submit and contains all the instructions and logic necessary for the Driver Program and Executors to run the application. (For Python applications, this is often a collection of .py or .zip files).

3. Driver Program

The Driver Program is the process that runs the main() method of the application and coordinates the parallel operations across the cluster. It is the "brain" of the Spark application.

It holds the SparkContext, which connects to the Cluster Manager.

It analyzes the user code, converts the high-level logic into an optimized execution plan (a DAG), and schedules the tasks to run on the Worker Nodes.

4. Cluster Manager

The Cluster Manager is an external service (separate from Spark) that is responsible for acquiring resources (CPU, memory) on the cluster and managing their allocation to different applications.

Examples include YARN (Yet Another Resource Negotiator, commonly used with Hadoop), Mesos, and Spark's built-in Standalone Scheduler.

The Driver Program connects to the Cluster Manager to request resources for its Executors.

5. Worker Node

A Worker Node is any machine in the cluster that runs the actual distributed computations for the application.

It receives instructions from the Driver Program.

It hosts the Executors, which are the processes that run the individual tasks (the smallest units of work) on partitions of the data.

Spark Glossary

1. Deploy Mode

The Deploy Mode determines where the Driver Program (the brain of the application) executes relative to the cluster resources managed by the Cluster Manager (like YARN or Mesos).

Client Mode: The Driver runs on the machine where the application was submitted (the client). Best for interactive sessions.

Cluster Mode: The Driver runs inside a container on one of the Worker Nodes in the cluster. Best for production jobs as it's more fault-tolerant.

2. Executor

An Executor is a process that runs on a Worker Node in the cluster and is responsible for executing the application's individual tasks and storing the data cached by the application.

Each Executor is a JVM process (or a separate process in PySpark).

It reports the status of its tasks back to the Driver Program.

3. Task

A Task is the smallest unit of work in a Spark application. It is the actual work that is sent to and executed by an Executor.

Tasks are usually a piece of code that runs on a single partition (a segment) of the distributed data.

A Stage is broken down into a set of Tasks that are run in parallel.

4. Job

A Job is the highest-level unit of work in a Spark application. A Job is triggered when a Spark action is called on an RDD, DataFrame, or Dataset.

Actions include operations like count(), collect(), or save().

A single Job consists of one or more Stages that must be executed to compute the final result of the action.

5. Stage

A Stage is a set of parallel Tasks that perform the same operation (e.g., all map operations, or all reduce operations) and can be executed together without any data shuffling.

A Spark Job is broken down into Stages based on the required shuffling of data across the network.

Whenever a wide dependency (like a groupByKey or join) is encountered that requires data to be moved between Executors, the execution plan breaks into a new Stage.

Spark Glossary

References

A Gentle Introduction to Spark, Databricks, (Available in READINGS folder)
Spark Overview
Karau H., Fast data processing with Spark Packt Publishing, 2013 (Available from UOW Library)
Srinivasa, K.G., Guide to High Performance Distributed Computing: Case Studies with Hadoop, Scalding and SparkSpringer, 2015 (Available from UOW Library)
Chambers B., Zaharia M.,Spark: The Definitive Guide, O'Reilly 2017