03HadoopArchitecture

Hadoop Architecture

Outline

Hadoop Distributed File System (HDFS)
NameNode metadata
DataNode and Secondary node
Yet Another Resource Negotiator (YARN)
ResourceManger
NodeManager
ApplicationMaster
Summary

Hadoop Distributed File System (HDFS) is a central component of Hadoop's implementation of the MapReduce framework.

HDFS presents a unified file system view across numerous individual file systems distributed over the nodes within a computer cluster. Essentially, HDFS implements a single logical file system across many computer nodes in a cluster.

HDFS is designed for the following:

Very large files: A very large file is partitioned into smaller chunks, with each chunk residing on a single node in the computer cluster. HDFS provides a view of these chunks as a single, monolithic file.

Stream data access: A stream of data items, such as signals received from outer space, can be recorded directly into HDFS.

Commodity hardware: HDFS is implemented on computer clusters where individual nodes are off-the-shelf, low-cost commodity computer systems.

HDFS is not designed for the following:

Low-latency data access: HDFS does not optimize for or speed up access to medium-sized data files.

Numerous small files: HDFS does not improve access to many small files. Typically, multiple small files are integrated into a single larger file before being uploaded to HDFS.

Multiple writers, arbitrary file modifications: HDFS is primarily designed for reading large data files. It offers no performance improvements when operating on a large number of smaller files, nor is it optimized for frequent, arbitrary modifications by multiple writers.

HDFS: Hadoop Distributed File System

HDFS is designed for:
- Very large files
- Stream data access
- Commodity hardware
But not for:
- Low-latency data access
- Lots of small files
- Multiple writers, arbitrary file modifications

HDFS is implemented on a computer cluster comprising multiple nodes, each of which is a computer system. The nodes within a cluster fulfill distinct roles.

A node can be a "NameNode". A NameNode plays a role of HDFS master node process. It manages the filesystem metadata. Metadata contains information about the distribution of the chunks of large files over the nodes in a cluster. A NameNode does not store any chunk of file itself.

A node can be a NameNode, which serves as the HDFS master node process. It manages the filesystem metadata, which includes information about the distribution of large file chunks across the cluster's nodes. A NameNode does not store any file chunks itself.

Alternatively, a node may function as a DataNode, operating as the HDFS slave node process. Its responsibilities include managing block storage and facilitating data access for read and write operations, as well as handling block replication.

HDFS: Hadoop Distributed File System

HDFS contains the following key components:
NameNode:
- HDFS master node process
- manages the filesystem metadata
- does not store a file itself
SecondaryNameNode and Standby NameNode
- SecondaryNameNode expedites the filesystem metadata recovery
- Standby NameNode (optional) provides high availability
DataNode
- runs HDFS slave node process
- manages block storage and access for reading or writing of data, block replication

HDFS: Hadoop Distributed File System

Architecture of HDFS

HDFS functions as a virtual filesystem, meaning it presents a unified and seamless view of data to a client, appearing as a single, contiguous file system. However, this is an abstraction. Underneath this virtual layer, the actual data is distributed and stored across multiple different physical locations, specifically on various nodes within a Hadoop cluster.

Furthermore, HDFS is not a standalone operating system filesystem; rather, it is deployed on top of existing native filesystems on each node. For Linux-based systems, these underlying native filesystems commonly include ext3, ext4, and xfs. This layered architecture allows HDFS to leverage the stability and features of the underlying local filesystems while providing its unique distributed storage capabilities.

In HDFS each file is fundamentally divided into smaller, manageable units called blocks. Unlike traditional file systems where block sizes are typically much smaller (e.g., 4KB), HDFS blocks are significantly larger to accommodate the needs of big data processing. The default size for each HDFS block is 128 megabytes (MB), a setting chosen to minimize the overhead associated with disk seeks and network transfers for large datasets. However, this block size is not fixed and is fully configurable, allowing administrators to adjust it based on their specific cluster hardware, workload characteristics, and typical file sizes.

To ensure data reliability and fault tolerance, HDFS implements block replication. By default, the system maintains three identical copies of each block and distributes these replicates across different nodes in the cluster. This redundancy means that if a node fails or data corruption occurs on one block, the data remains accessible from the other replicated copies. Similar to block size, the default number of replicates (3) is also configurable, enabling users to increase it for higher data durability requirements or decrease it to save storage space, depending on their risk tolerance and resource availability

HDFS: Hadoop Distributed File System

HDFS is a virtual filesystem
- appears to a client as one file system, but the data is stored in multiple different locations
- deployed on the top of the native filesystems (such as ext3, ext4 and xfs in Linux)
Each file in HDFS consists of blocks
- The size of each block defaults to 128MB but is configurable
- The default number of replicates for blocks is 3, but it is also configurable

When a physical and very large file is uploaded to HDFS (Hadoop Distributed File System), it undergoes a crucial process of segmentation and distribution. Instead of storing the entire file on a single machine, HDFS automatically divides the file into a number of smaller, fixed-size units known as blocks (or chunks). As previously mentioned, the default size of these blocks is 128MB, though this is configurable.

Once the file is broken down into these blocks, HDFS intelligently distributes them across the cluster. Each of these blocks (or chunks) is then located on a separate DataNode. This distribution is not random; HDFS aims to spread the blocks evenly across the available DataNodes to maximize parallel processing capabilities and ensure data availability. Furthermore, for fault tolerance and data reliability, each block is typically replicated multiple times (by default, three times) and these replicates are stored on different DataNodes.

The NameNode plays a critical supervisory role in this process. It contains the definitive repository of metadata for all data chunks. This "repository" isn't the actual data itself, but rather a comprehensive index or map. For every single data chunk (or block) in the entire HDFS, the NameNode meticulously records its unique identifier, its size, the file it belongs to, and, most importantly, the exact locations (i.e., the specific DataNodes) where each replica of that data chunk is stored. When a client wishes to read a file, it first consults the NameNode to get this location information, allowing it to directly access the necessary blocks from the appropriate DataNodes. This centralized metadata management by the NameNode is fundamental to HDFS's ability to present a single, coherent file system view despite the underlying distributed storage.

HDFS: Hadoop Distributed File System

Logical view of data storage

Physical implementation of data file storage.

File Segmentation into Blocks:

At the outset, any file uploaded to HDFS is logically divided into a series of fixed-size data units called blocks. While the default block size in modern Hadoop versions is 128 megabytes (MB), this is a configurable parameter that can be adjusted based on the nature of the data and the cluster's hardware. This large block size is a deliberate design choice in HDFS. It significantly reduces the overhead associated with disk seeks (as fewer seeks are needed to read a large file) and minimizes the amount of metadata the NameNode needs to manage compared to smaller block sizes. Even if the last part of a file is smaller than the full block size, it will still occupy its own block.

Distributed Block Storage on DataNodes:

Each of these individual blocks is then stored independently on separate DataNodes across the cluster. The NameNode orchestrates this placement, attempting to distribute blocks evenly to balance storage utilization and maximize data locality for future computations. This distributed storage is crucial for HDFS's ability to handle petabytes of data and enable highly parallelized processing.

Data Replication for Fault Tolerance and Availability:

A cornerstone of HDFS's reliability is its replication mechanism. Each block is not stored just once, but is replicated on two or more DataNodes. The default replication factor is three, meaning that for every block, there are three identical copies residing on distinct DataNodes.

HDFS employs intelligent replica placement policies. For a replication factor of three, the typical strategy is:

- The first replica is placed on the same node as the client (if possible) or a node in the same rack.

- The second replica is placed on a node in a different rack.

- The third replica is placed on a different node within the same rack as the second replica. This "rack-aware" placement helps to protect against entire rack failures while minimizing inter-rack network traffic for writes.

Inter-DataNode Network Connectivity:

For this distributed and replicated storage to function, all DataNodes are robustly networked, allowing for high-speed data transfer between them. This direct DataNode-to-DataNode communication is vital for several operations:

HDFS: Hadoop Distributed File Systeme

Physical implementation of data file storage

Hadoop Architecture

The NameNode is the absolute core of the Hadoop Distributed File System (HDFS) because it stores and manages all the metadata pertaining to the files within HDFS. This metadata is critical for enabling HDFS to present a unified file system view to clients, even though the data itself is distributed across numerous DataNodes. Without the NameNode, clients would have no way to locate their data or understand the file system's structure.

The NameNode's functions can be elaborated as follows:

- Maintaining the File System Metadata: It maintains the entire file system hierarchy �C a complete directory tree of all files and folders stored in HDFS. For every file, it keeps track of its attributes such as permissions, modification times, and importantly, the mapping of its data blocks to their physical locations on specific DataNodes.

- Managing User Access to Data Files: The NameNode serves as the gatekeeper for all client interactions with the HDFS file system. When a user or application wants to read, write, create, delete, or rename a file, they first communicate with the NameNode.

- Mapping Data Blocks to the DataNodes in the Cluster: When a client requests to read a file, the NameNode, using its block-to-DataNode mapping, identifies the optimal DataNodes where the required data blocks reside. It then provides the client with a list of these DataNodes.

- Performing File System Operations: This includes opening and closing files and directories, handling block reports, managing replication.

- Providing Registration Services and Periodic Heartbeats for DataNodes: DataNodes register themselves with the NameNode upon startup, declaring their presence and availability. After registration, DataNodes send periodic "heartbeat" signals to the NameNode.

NameNode Metadata

NameNode stores the metadata of the files in HDFS
NameNode functions:
- Maintain the metadata pertaining to the file system (e.g., the file hierarchy and the block locations for each file)
- Manage user access to the data files
- Map the data blocks to the DataNodes in the cluster
- Perform file system operations (e.g., opening and closing the files and directories)
- Provide registration services and periodic heartbeats for DataNodes

Hadoop Architecture

Outline

Hadoop Distributed File System (HDFS)
NameNode metadata
DataNode and Secondary node
Yet Another Resource Negotiator (YARN)
ResourceManger
NodeManager
ApplicationMaster
Summary

DataNode Functions

DataNodes are responsible for the actual storage and retrieval of data. DataNodes provide the following functionality:

- Access to block storage: DataNodes physically store the data blocks that make up HDFS files on their local file systems (like ext4 or xfs). They manage the disk space and ensure data integrity at the block level.

- Handling Read/Write Requests: When a client wants to read or write data, it directly interacts with the DataNodes. The DataNodes handle the high-throughput streaming of data in response to these requests, often simultaneously serving multiple clients.

- Replicating Data: DataNodes actively participate in data replication. After receiving a block from a client during a write operation, a DataNode will then pipeline that block to other DataNodes to create the necessary replicas (e.g., three copies by default), ensuring data durability and fault tolerance.

Communicating with the NameNode: DataNodes maintain constant communication with the NameNode.

- Sending periodic heartbeats to confirm they are alive, healthy, and operational. If a DataNode stops sending heartbeats, the NameNode assumes it has failed.

- Sending block reports periodically. A block report is a comprehensive list of all the data blocks currently managed by that specific DataNode. This report helps the NameNode keep its metadata up-to-date and detect any missing or corrupted blocks that need re-replication.

Secondary NameNode and Standby NameNode Functions

Both the Secondary NameNode and Standby NameNode play crucial roles in safeguarding the NameNode's metadata, but they do so in distinct ways.

Secondary NameNode: This node serves as a checkpointing assistant for the primary NameNode. It periodically fetches the NameNode's EditLog (transaction log) and merges it with the FsImage (filesystem image) to create a new, up-to-date FsImage. This process reduces the size of the EditLog and creates more frequent, consistent snapshots of the metadata. This backup is primarily used for disaster recovery �C allowing the NameNode to recover its state more quickly if it crashes, as it only needs to apply a smaller EditLog. It's not a hot standby.

Standby NameNode: In contrast to the Secondary NameNode, the Standby NameNode is a hot active node that runs alongside the primary (active) NameNode in a High-Availability (HA) cluster setup. It maintains a continuously synchronized copy of the NameNode's metadata by receiving a stream of edits from the active NameNode. If the primary NameNode fails, the Standby NameNode can rapidly take over its role, becoming the new active NameNode. This facilitates high-availability, minimizing downtime and preventing the NameNode from being a single point of failure for the entire HDFS cluster

DataNode and Secondary node

DataNode functions:
- Provide the block storage by storing blocks on the local file system
- Fulfil the read/write requests
- Replicating data across the cluster
- Keeping in touch with the NameNode by sending periodic block reports and heartbeats
- A heartbeat confirms the DataNode is alive and healthy, and a block report shows the blocks being managed by the DataNode
Secondary NameNode and Standby NameNode functions:
- Without a NameNode, there is no way to know to which files the blocks stored on the DataNodes correspond to
- In essence, all files in HDFS are lost
- Secondary NameNode periodically backups the metadata in the (primary) NameNode, which is usually for recovery
- Standby NameNode is a hot node that running together with the (primary) NameNode in the cluster, facilitating high-availability

Hadoop Architecture

Outline

Hadoop Distributed File System (HDFS)
NameNode metadata
DataNode and Secondary node
Yet Another Resource Negotiator (YARN)
ResourceManger
NodeManager
ApplicationMaster
Summary

YARN (Yet Another Resource Negotiator)

YARN is the fundamental resource management layer of Hadoop. It acts as the core subsystem responsible for governing, allocating, and managing the finite distributed processing resources (like CPU cores and memory) available across a Hadoop cluster. Essentially, YARN transforms Hadoop from just a storage system (HDFS) and a single processing engine (MapReduce) into a more versatile, general-purpose distributed operating system.

YARN was a significant architectural shift, introduced in Hadoop 2, primarily to improve the scalability and flexibility of the MapReduce implementation. Before YARN, MapReduce's resource management was tightly coupled within the MapReduce framework itself, limiting its ability to run other types of distributed applications.

YARN provides its core services through two types of long-running daemon processes:

- Resource Manager: There is one ResourceManager per cluster. This is the master daemon that oversees the entire cluster's resources. Its primary role is to manage the use of resources across the cluster, making high-level scheduling decisions and allocating "containers" (resource bundles) to applications based on their requests and available capacity.

- NodeManager: NodeManagers run on all the worker nodes in the cluster. These are the per-node agents responsible for launching and monitoring containers. A container is a logical bundle of resources (e.g., specific CPU cores and an amount of RAM) allocated to an application's task. The NodeManager also monitors resource usage by these containers and reports this information back to the ResourceManager.

Yet Another Resource Negotiator (YARN)

YARN: the core subsystem in Hadoop responsible for governing, allocating, and managing the finite distributed processing resources available on a Hadoop cluster
- introduced in Hadoop 2 to improve the MapReduce implementation, but general enough to support other distributed computing paradigms
YARN provides its core services via two types of long-running daemons:
- A ResourceManager (one per cluster) to manage the use of resources across the cluster, and
- NodeManagers running on all the nodes in the cluster to launch and monitor containers

The architecture of YARN (Yet Another Resource Negotiator) orchestrates how computational resources are allocated and utilized.

Client: The client is the entity that initiates and submits jobs or applications to the YARN cluster. This is typically a software program or script that defines the workload to be executed.

Job (Application): In YARN's terminology, a "job" is often referred to as an "application." An application represents a complete unit of work submitted to the cluster, which is broken down into one or more constituent tasks. For instance, in the specific context of a MapReduce job, an application would comprise its individual mapper tasks (which process input data) and reducer tasks (which aggregate intermediate results).

]Task: A task is the smallest, executable unit of work within a larger job or application.

Container: Every individual task (be it a mapper, reducer, or any other application task) runs within a dedicated container. A container is a logical abstraction that encapsulates a specific, isolated bundle of resources allocated on a worker node by the ResourceManager. These resources are quantifiable and primarily include a defined amount of memory (RAM) and a certain number of processing cores (CPU). For example, a container might be configured to provide 2GB of memory and access to 2 CPU cores.

Yet Another Resource Negotiator (YARN)

Architecture of YARN
A client is the program that submits jobs to the cluster
- May also be the gateway machine that the client program runs on
A job, also called an application, contains one or more tasks
- A task in a MapReduce job can be either a mapper and a reducer task
Each mapper and reducer task runs within a container
- Containers are logical constructs that represent a specific amount of memory and other resources, such as processing cores (CPU)
- For example, a container can represent 2GB memory and 2 processing cores
- Containers may also refer to the running environment of an application

Architecture of YARN (Continued)

Yarn consists of the following daemons.

- ResourceManager: The ResourceManager is YARN's central daemon, running on a designated master node of the Hadoop cluster. It serves as authority for resource arbitration across the entire cluster. Its primary responsibility is granting cluster computing resources to the various applications that are running or want to run on the cluster.

- NodeManager: The NodeManager is YARN's daemon that runs on every slave (or worker) node within the cluster. It acts as the agent for the ResourceManager on its local machine. The NodeManager manages the containers on its particular slave node, launching them, monitoring their resource consumption (CPU, memory), and reporting their status back to the ResourceManager.

- ApplicationMaster: For each application submitted to YARN, the ApplicationMaster is a special container that is the first one allocated by the ResourceManager to run on a NodeManager. This container is unique because it's responsible for managing the lifecycle of its specific application. It negotiates resources from the ResourceManager for its application's tasks, tracks their progress, monitors their health, and handles faults.

Yet Another Resource Negotiator (YARN)

Architecture of YARN
ResourceManager: YARN's daemon running on a master node
- ResourceManager is responsible for granting cluster computing resources to applications running on the cluster
- Resources are granted the items of containers
NodeManager: YARN's daemon running on a slave node.
- NodeManager manages containers on a slave node
- ApplicationMaster: the first container allocated by the ResourceManager to run on a NodeManager for each application

Communications among Yarn's daemons

Communication between Clients and ResourceManager:

Clients initiate the process by submitting application requests (jobs) to the ResourceManager.

They communicate with the ResourceManager to get an application ID, negotiate initial resources for the ApplicationMaster, and later query the status and progress of their submitted applications.

Communications between ResourceManager and NodeManager:

The ResourceManager and NodeManagers maintain a continuous dialogue.

NodeManagers periodically send heartbeats to the ResourceManager, signaling their health, available resources, and the status of containers running on their node.

The ResourceManager sends instructions to NodeManagers, such as commands to launch or kill containers for specific applications.

Communications between ResourceManager and ApplicationMaster:

When an application is submitted, the ResourceManager allocates the initial container to launch the ApplicationMaster.

The ApplicationMaster then negotiates resources (more containers) from the ResourceManager for its application's tasks, communicating its resource needs and receiving allocations.

The ApplicationMaster also reports its application's overall progress and status back to the ResourceManager.

Communications between NodeManager and Containers:

The NodeManager is directly responsible for managing the lifecycle of containers on its local node.

It launches containers as instructed by the ApplicationMaster (via ResourceManager's allocation and subsequent instructions), monitors their resource usage (CPU, memory), and manages their execution environment.

The NodeManager also terminates containers when their tasks are complete or if they exceed their allocated resources.

Communications between ApplicationMaster and Containers:

While the NodeManager launches containers, the ApplicationMaster indirectly manages its application's containers by instructing the NodeManager.

Once launched, the application code running inside a container communicates its progress, status, and any errors back to its controlling ApplicationMaster via an application-specific protocol. The ApplicationMaster uses this information to track the job's overall execution and to request new containers if tasks fail.

Yet Another Resource Negotiator (YARN)

Architecture of YARN

Clients initiate the process by submitting application requests (jobs) to the ResourceManager.

They communicate with the ResourceManager to get an application ID, negotiate initial resources for the ApplicationMaster, and later query the status and progress of their submitted applications.

The ResourceManager sends instructions to NodeManagers, such as commands to launch or kill containers for specific applications.

The ApplicationMaster then negotiates resources (more containers) from the ResourceManager for its application's tasks, communicating its resource needs and receiving allocations. The ApplicationMaster also reports its application's overall progress and status back to the ResourceManager.

While the NodeManager launches containers, the ApplicationMaster indirectly manages its application's containers by instructing the NodeManager.

Yet Another Resource Negotiator (YARN)

Architecture of YARN

Hadoop Architecture

Outline

Hadoop Distributed File System (HDFS)
NameNode metadata
DataNode and Secondary node
Yet Another Resource Negotiator (YARN)
ResourceManger
NodeManager
ApplicationMaster
Summary

There is one ResourceManager per cluster, which consists of two key components: Scheduler and ApplicationManager

Key functions of ResourceManager:

- Creates the initial container for an application to launch and run its ApplicationMaster. This is the first step in an application's lifecycle on YARN.

- Tracks heartbeats from NodeManagers to monitor the health and available resources of all worker nodes in the cluster. This allows it to keep an up-to-date view of the cluster's capacity. (Note: It tracks NodeManagers, not DataNodes directly, although DataNodes reside on the same physical machines as NodeManagers).

- Runs the Scheduler component to intelligently determine how computing resources (containers) should be allocated among all competing applications running on the cluster. The Scheduler operates based on configurable policies (e.g., capacity scheduler, fair scheduler).

- Manages cluster-level security, including authentication and authorization for applications and users accessing YARN resources.

- Manages the resource requests from ApplicationMasters. ApplicationMasters continuously negotiate with the ResourceManager for more containers as their application's tasks require them.

- Monitors the status of ApplicationMasters and, upon their failure, orchestrates the restart of that container to ensure application resilience.

- Deallocates containers when the application successfully completes its execution or when individual containers expire after their allocated task finishes.

- The role of the ResourceManager is purely management and scheduling. It acts as the central orchestrator of resources.

- Crucially, it does not perform any actual data processing itself. For example, it does not execute the Map or Reduce functions in a MapReduce application; that work is delegated to tasks running within containers managed by NodeManagers.

ResourceManager

There is one ResourceManager per cluster, which consists of two key components: Scheduler and ApplicationManager
Key functions of ResourceManager:
- Creates the first container for an application to run ApplicationMaster for that application
- Tracks the heartbeats from NodeManagers to manage DataNodes
- Runs Scheduler to determine resource allocation among the clusters
- Manages cluster level security
- Manages the resource requests from ApplicationMasters
- Monitors the status of ApplicationMasters and restarts that container upon its failure
- Deallocates the containers when the application completes or after they expire
The role of ResourceManager is pure management and scheduler
It does not perform any actual data processing, for example the Map and Reduce functions in a MapReduce application

Hadoop Architecture

Outline

Hadoop Distributed File System (HDFS)
NameNode metadata
DataNode and Secondary node
Yet Another Resource Negotiator (YARN)
ResourceManger
NodeManager
ApplicationMaster
Summary

Each DataNode runs a NodeManager daemon for performing YARN functions

Main functions of a NodeManager daemon:

- Communicates with the ResourceManager: The NodeManager maintains constant communication with the central ResourceManager. It sends health heartbeats to confirm its operational status and reports container status notifications (e.g., container started, running, completed, failed) to keep the ResourceManager updated on the cluster's health and resource utilization.

- Registers and Manages Application Processes: Upon startup, the NodeManager registers itself with the ResourceManager, announcing its availability and capacity. It is then responsible for launching and managing the individual application processes (tasks) that run within containers on its node.

- Launches and Oversees Containers: On request from the ApplicationMaster (which represents a specific application), the NodeManager is responsible for launching both the ApplicationMaster's own container (the first one for that application) and subsequently all other resource containers for that application (e.g., the Map and Reduce tasks for a MapReduce job). It then oversees the entire lifecycle of these application containers, from their initial setup to their eventual termination.

- Monitors and Manages Resource Consumption: A critical function is to continuously monitor, manage, and provide information regarding the resource consumption (CPU and memory) by the containers running on its node. If containers exceed their allocated resources, the NodeManager can enforce limits or even terminate them to maintain cluster stability.

- Tracks DataNode Health: While the NodeManager's primary role is YARN-specific, it typically resides on the same machine as the DataNode. Therefore, it also often tracks the health of the underlying DataNode by running disk health checks and reporting issues to the ResourceManager. This provides an integrated view of the worker node's overall health.

- Provides Auxiliary Services: The NodeManager can provide auxiliary services to YARN applications. A prime example is the MapReduce framework's shuffle service, which runs within the NodeManager process. This service facilitates the transfer of intermediate map output data to reducer tasks, a crucial step in the MapReduce paradigm, demonstrating how YARN supports diverse frameworks.

NodeManager

Each DataNode runs a NodeManager daemon for performing YARN functions
Main functions of a NodeManager daemon:
- Communicates with ResourceManager through health heartbeats and container status notifications.
- Registers and starts the application processes
- Launches both ApplicationMaster and the rest of an application's resource containers (that is, the map and reduce tasks that run in the containers) on request from ApplicationMaster
- Oversees the lifecycle of the application containers
- Monitors, manages and provides information regarding the resource consumption (CPU/memory) by the containers
- Tracks the health of DataNode
- Provides auxiliary services to YARN applications, such as services used by the MapReduce framework for its shuffle and sort operations

Hadoop Architecture

Outline

Hadoop Distributed File System (HDFS)
NameNode metadata
DataNode and Secondary node
Yet Another Resource Negotiator (YARN)
ResourceManger
NodeManager
ApplicationMaster
Summary

For each YARN application, there is a dedicated ApplicationMaster.

Functions of the ApplicationMaster:

- Manages Task Scheduling and Execution: The ApplicationMaster is responsible for coordinating the tasks that make up its application. It tracks the progress of each task, handles failures, and re-schedules tasks if necessary to ensure the application completes successfully.

- Allocates Resources Locally for the Application's Tasks: While the ResourceManager grants overall cluster resources, the ApplicationMaster handles the granular resource allocation for its specific application's tasks. It communicates with the ResourceManager to acquire containers and then instructs the appropriate NodeManagers to launch those containers for its tasks.

Key Characteristics of the ApplicationMaster:

- ApplicationMaster is running within a container

- ApplicationMaster's existence is associated with the running application

- When an application is completed, its ApplicationMaster no longer exists

- Once created, ApplicationMaster is in charge of requesting resources with ResourceManager to run the application

The resource request are very specific, for example:

- the file blocks needed to process the job,

- the amount of the resource, in terms of the number of containers to create for the application,

- the size of the containers, etc.

ApplicationMaster

For each YARN application, there is a dedicated ApplicationMaster
Functions of ApplicationMaster:
- Managing task scheduling and execution
- Allocating resources locally for the application's tasks
ApplicationMaster is running within a container
ApplicationMaster's existence is associated with the running application
When an application is completed, its ApplicationMaster no longer exists
Once created, ApplicationMaster is in charge of requesting resources with ResourceManager to run the application
The resource request are very specific, for example:
- the file blocks needed to process the job,
- the amount of the resource, in terms of the number of containers to create for the application,
- the size of the containers, etc.

Hadoop Architecture

Outline

Hadoop Distributed File System (HDFS)
NameNode metadata
DataNode and Secondary node
Yet Another Resource Negotiator (YARN)
ResourceManger
NodeManager
ApplicationMaster
Summary

Terminologies (see the text from the image)

Hadoop: A Leading Platform for Big Data

Hadoop has emerged as a leading open-source platform specifically engineered for the challenges of big data. It provides a robust, scalable, and cost-effective solution for both storing and processing enormous volumes of diverse data across distributed clusters of readily available, commodity hardware.

At its core, Hadoop's architecture is intelligently segmented into three distinct yet integrated layers:

- Storage Layer (HDFS): This is the foundation, provided by the Hadoop Distributed File System (HDFS). HDFS is designed for high-throughput access to very large files, offering fault-tolerant storage by distributing data across multiple machines.

- Coordination and Management Layer (YARN): Yet Another Resource Negotiator (YARN) acts as the cluster's operating system. It's responsible for efficiently governing, allocating, and managing the cluster's computational resources (like CPU and memory) for all running applications, ensuring optimal utilization.

- Processing Layer (e.g., MapReduce): This layer encompasses the various computational frameworks that execute data processing tasks. While MapReduce remains a historically fundamental computing model within Hadoop (specializing in large-scale batch processing), YARN's flexibility has opened the door for many other powerful processing engines like Apache Spark, Apache Flink, and more to operate within the same ecosystem.

Summary

Terminologies
Hadoop is a leading platform for big data
Hadoop consists of a storage layer (HDFS), a coordination and management layer (YARN) and a processing layer (e.g., MapReduce)
HDFS and YARN have key services (daemons)
MapReduce is a fundamental computing model (i.e., batch processing) for big data
Next: Interaction with Hadoop and "dive" into the MapReduce framework

References

White T., Hadoop The Definitive Guide: Storage and analysis at Internet scale, O'Reilly, 2015 (Available through UOW library)
Vohra D., Practical Hadoop ecosystem: a definitive guide to Hadoop-related frameworks and tools, Apress, 2016 (Available through UOW library)
Aven J., Hadoop in 24 Hours, SAMS Teach Yourself, SAMS 2017
Alapati S. R., Expert Hadoop Adiministration: Managing, tuning, and securing Spark, YARN and HDFS, Addison-Wesley 2017
Apache Hadoop, HDFS Interface
Apache Hadoop, YARN