05MapReduceModel

MapReduce Data Processing Model

Outline

Key-value pairs
MapReduce model
Map phase
Reduce phase
Shuffle and sort
Combiner phase
Example
Running MapReduce jobs

Key-Value Pairs

MapReduce processes data in the form of key-value pairs. Think of it like a dictionary or a simple table with two columns:

Key: An identifier or label.

Value: The actual data associated with that identifier.

For example:

Key Value

City Sydney

Employer Cloudera

In this context:

City is the key, and Sydney is its corresponding value.

Employer is the key, and Cloudera is its corresponding value.

Key Characteristics in MapReduce:

Ubiquitous Representation: Key-value pairs are the fundamental unit of data throughout the entire MapReduce processing pipeline:

Input: The raw data that you feed into your MapReduce job is interpreted as a collection of key-value pairs. The format of these initial key-value pairs depends on how your input data is structured and the input format you configure.

Intermediate Data: The output of the Map phase and the input to the Reduce phase are also structured as key-value pairs. The Map function transforms the input key-value pairs into new intermediate key-value pairs.

Output: The final result produced by the Reduce phase is also a set of key-value pairs.

Why Key-Value Pairs in MapReduce?

The choice of the key-value pair model is central to MapReduce's design and enables its parallel and distributed processing capabilities:

Parallel Processing (Mapping): The Map function operates on each input key-value pair independently. This allows for massive parallelization across a cluster of machines.

Data Shuffling and Grouping: The MapReduce framework automatically shuffles and groups intermediate key-value pairs based on their keys. All values associated with the same key are brought together to the same Reducer task.

Parallel Processing (Reducing): The Reduce function processes all the values associated with a single key. This allows for aggregation, summarization, or other computations on related data.

Key-value pairs

Key-Value pairs: MapReduce basic data model

Key                          Value
City                         Sydney
Employer                     Cloudera

Input, output, and intermediate records in MapReduce are represented as key-value pairs (aka name-value/attribute-value pairs)
A key is an identifier, for example, a name of attribute
- In MapReduce, a key is not required to be unique.
A value is a data associated with a key
- It may be simple value or a complex object

MapReduce Data Processing Model

The MapReduce data processing model is a sequence of distinct stages that work together to process large datasets in a distributed and parallel manner.

1. Map Stage:

Input: The Map stage takes the input data, which is initially presented as a collection of key-value pairs. The format of these initial key-value pairs depends on the input format and how the data is read.

Processing: The Map function, defined by the user, is applied to each input key-value pair independently. Its primary goal is to process the input record and produce one or more new intermediate key-value pairs.

Output: The output of the Map stage is a collection of these intermediate key-value pairs. The keys and values in these intermediate pairs can be of different types than the input key-value pairs.

Example: Consider processing log files where each line is an input record. The Map function might take the line (potentially with a byte offset as the initial key) and emit key-value pairs where the key is a word found in the line, and the value is '1' (representing a single occurrence of that word).

2. Partition Stage:

Input: The intermediate key-value pairs generated by the Map tasks.

Processing: The Partition function takes the intermediate key and determines which Reducer task should receive that key-value pair. The goal is to distribute the intermediate data across the Reducer tasks.

Default Behavior: The default partitioner typically uses a hash function on the key (e.g., hash(key) mod num_reducers) to distribute keys evenly across the available Reducers.

Custom Partitioning: Users can define custom partitioners to control how keys are distributed, which can be useful for performance optimization or specific processing requirements.

Output: The output of the Partition stage is a set of intermediate key-value pairs, where each pair is tagged with the ID of the Reducer task it should be sent to.

Example (continuing from Map): If you have two Reducer tasks, the Partition function might send all key-value pairs where the word (key) starts with 'A' to 'M' to Reducer 1, and words starting with 'N' to 'Z' to Reducer 2.

3. Shuffle and Sort Stage:

Input: The partitioned intermediate key-value pairs from all the Map tasks.

Processing: This stage happens on the Reducer side. It involves two main operations:

Shuffle: Data from all the Map tasks that is destined for a particular Reducer is transferred (shuffled) across the network to that Reducer machine.

Sort: Once all the key-value pairs for a Reducer have arrived, they are sorted by the intermediate key. This grouping of values by key makes it easier for the Reduce function to process all related values together.

Output: The output of this stage is a set of key-value pairs, where all the values associated with the same key are grouped together and sorted by key.

Example (continuing): Reducer 1 would receive all the (word, 1) pairs where the 'word' starts with 'A' to 'M'. The Shuffle stage brings all these pairs to Reducer 1, and the Sort stage then arranges them so that all the 'apple', 'apple', 'apricot', 'ball', 'ball', etc., are together.

4. Reduce Stage:

Input: The sorted and grouped intermediate key-value pairs from the Shuffle and Sort stage. The input to the Reduce function is typically a key and an iterator over all the values associated with that key.

Processing: The Reduce function, defined by the user, processes each unique key and its corresponding list of values. It performs an aggregation, summarization, or other computation on these values to produce the final output.

Output: The output of the Reduce stage is a collection of final key-value pairs, which represent the result of the MapReduce job.

Example (continuing): Reducer 1 receives the key 'apple' and the list of values [1, 1]. The Reduce function might sum these values to produce the output key-value pair ('apple', 2). Similarly, for 'ball' and the list [1, 1], it would output ('ball', 2), and so on.

MapReduce Model

MapReduce data processing model is a sequence of Map, Partition, Shuffle and Sort, and Reduce stages

Input: The Map function takes two arguments for each input record (in this case, each line of text):

lineNo (Long): This represents the line number or the byte offset of the line within the input file. While provided, this key is often not directly used in the core logic of a basic WordCount. The framework provides this context.

line (String): This is the actual line of text being processed.

Processing: The function iterates through each word (w) present in the input line. How the line is split into words (e.g., by spaces, punctuation) is typically handled by the MapReduce framework's input format configuration or within the Map function itself (though this example abstractly assumes it).

For each encountered word (w), the Map function emits an intermediate key-value pair: (w, 1).

The word (w) becomes the key.

The integer 1 becomes the value, representing a single occurrence of that word in the current line.

Output: The Map function produces a list of intermediate key-value pairs, where each word from the input text is paired with a count of 1.

Input: The Reduce function receives two arguments:

w (String): This is a unique key (a word) that was generated by one or more Map tasks. The MapReduce framework ensures that all intermediate key-value pairs with the same key are grouped together and sent to the same Reduce task.

loc (List): This is a list of values associated with the key w. In the context of WordCount, each value in this list will be 1, representing an occurrence of the word w found by different Map tasks across the input data.

Processing:The function initializes a variable sum to 0.

It iterates through each count (c) in the input list loc.

For each count, it adds it to the sum.

Output: The Reduce function emits a single final key-value pair: (word, sum).

The input word (w) remains the key.

The calculated sum becomes the value, representing the total number of times the word w appeared in the entire input data.

MapReduce Model

An abstract MapReduce program: WordCount

	  
function Map(Long lineNo, String line):
  lineNo: the position no. of a line in the text
  line: a line of text
    for each word w in line:
      emit (w, 1)

function Reduce(String w, List loc):
  w: a word
  loc: a list of counts outputted from map instances
    sum = 0
    for each c in loc:
      sum += c
    emit (word, sum)

ORM

Input Data (e.g., Files in HDFS):

This is the initial large dataset that needs to be processed. It could reside in a distributed file system like Hadoop Distributed File System (HDFS) or other data sources.

Splitting: The input data is divided into smaller, manageable chunks called splits. Each split is typically processed by a separate Map task.

Map Tasks: Multiple Map tasks run in parallel across the data splits. Each Map task takes an input split and applies the user-defined Map function to each record (often a line in a text file). The Map function processes each input key-value pair (though the initial key might be a byte offset) and emits one or more intermediate key-value pairs.

Intermediate Key-Value Pairs: The output of the Map tasks is a collection of intermediate key-value pairs. These pairs are the result of the transformation performed by the Map function.

Shuffle: The MapReduce framework then performs a shuffle operation. This involves transferring the intermediate key-value pairs from the Map tasks to the appropriate Reduce tasks. All key-value pairs with the same intermediate key are sent to the same Reducer.

Sort: Within each Reducer, the received intermediate key-value pairs are sorted based on the intermediate keys. This ensures that all values associated with the same key are presented to the Reduce function in a contiguous manner.

Reduce Tasks: Multiple Reduce tasks run in parallel. Each Reduce task receives a unique key and a list of all the values associated with that key (as outputted by the Map tasks and grouped by the shuffle and sort phases). The Reduce task applies the user-defined Reduce function to this key-value list to produce the final output.

Output Data (e.g., Files in HDFS): The final output of the Reduce tasks is a set of key-value pairs, which represents the result of the MapReduce job. This output is typically stored back in a distributed file system like HDFS.

MapReduce Model

A diagram of data processing in MapReduce model

MapReduce Data Processing Model

Outline

Key-value pairs
MapReduce model
Map phase
Reduce phase
Shuffle and sort
Combiner phase
Example
Running MapReduce jobs

Responsibilities of the Map phase in MapReduce:

1. Reading and Parsing Input Data:

2. Applying the User-Defined map() Function:

The map() function takes one of the initial key-value pairs (from the Record Reader) as input.

Transformation: Inside the map() function, the following operations are performed:

Filtering: Selecting only relevant data.

Data Extraction: Pulling out specific pieces of information.

Data Transformation: Changing the format or structure of the data.

Calculation: Performing initial computations.

Focus on Individual Records: Importantly, each invocation of the map() function is independent of others. It processes one input key-value pair at a time.

3. Generating Intermediate Key-Value Pairs:

The Output of map(): The map() function doesn't produce the final result. Instead, it emits one or more intermediate key-value pairs. These pairs represent a transformed view of the input data, designed to facilitate the subsequent aggregation in the Reduce phase.

New Keys and Values: The keys and values in these intermediate pairs are often different from the initial input key-value pairs. The choice of these intermediate keys is critical because they determine how the data will be grouped and processed in the Reduce phase.

4. Achieving Parallelism and Leveraging Data Locality:

Divide and Conquer: The splitting of the input data into chunks (input splits/HDFS blocks) is fundamental for parallelism. Each Map task works on its own independent chunk.

Concurrent Execution: The MapReduce framework can launch and manage many Map tasks running simultaneously across a cluster of machines. This significantly speeds up the processing of large datasets.

Data Locality Benefit: By ideally running Map tasks on the same nodes where the corresponding data blocks are stored in HDFS, the framework minimizes the need to move large amounts of data across the network. This dramatically improves performance.

Setting the Stage for Shuffle and Reduce:

The Map phase is not the end goal. Its primary purpose is to prepare the data for the next crucial phases:

Shuffle: The intermediate key-value pairs generated by all the Map tasks are redistributed based on their keys. All pairs with the same key are brought together to the same Reduce task.

Reduce: The Reduce phase then aggregates and processes these grouped values for each unique key to produce the final output.

Map phase

Map phase uses input format and record reader functions to derive records in the form of key-value pairs for the input data
Map phase applies a function or functions to each key-value pair over a portion of the dataset
- In the case of a dataset hosted in HDFS, this portion is usually called as a block
- If there are n blocks of data in the input dataset, there will be at least n Map tasks (also referred to as Mappers)
Each Map task operates against one filesystem (HDFS) block
In the diagram fragment, a Map task will call its map() function, represented by M in the diagram, once for each record, or key-value pair; for example, rec1, rec2, and so on.

The map() function is the workhorse of the Map phase. It processes individual input records, transforms them into intermediate key-value pairs, and prepares the data for the next stage. Its ability to emit zero or more key-value pairs, combined with the independent operation of Mappers, enables powerful and scalable data processing

Each call of the map() function accepts one key-value pair and emits zero or more key-value pairs

map (in_key, in_value) -> list (intermediate_key, intermediate_value)

The emitted data from Mapper, also in the form of lists of key-value pairs, will be subsequently processed in the Reduce phase

Different Mappers do not communicate or share data with each other

Common Map() functions include filtering of specific keys, such as filtering log messages if you only wanted to count or analyse ERROR log messages

Map (k, v) = if (ERROR in v) then emit (k, v)

Another example of Map() function would be to manipulate values, such as a function that converts a text value to lowercase

Map (k, v) = emit (k, v.toLowercase ( ))

Map phase

Each call of the map() function accepts one key-value pair and emits zero or more key-value pairs
The emitted data from Mapper, also in the form of lists of key-value pairs, will be subsequently processed in the Reduce phase
Different Mappers do not communicate or share data with each other
Common Map() functions include filtering of specific keys, such as filtering log messages if you only wanted to count or analyse ERROR log messages
Another example of Map() function would be to manipulate values, such as a function that converts a text value to lowercase

The primary responsibility of the Partition function (Partitioner) is to determine which Reduce task (Reducer) should receive each intermediate key-value pair generated by the Map tasks. It acts as a traffic controller, ensuring that all values associated with the same intermediate key are sent to the same Reducer.

Imagine a mail sorting facility. The Mappers are like the people writing letters (emitting intermediate key-value pairs). The Partition function is like the sorting machine that looks at the address (the intermediate key) and decides which delivery truck (Reducer) the letter should go on. All letters going to the same address (same intermediate key) are put on the same truck. The number of trucks (Reducers) determines the number of sorting bins (partitions). A standard sorting machine might just use the zip code (default Partitioner), but a special handling department (custom Partitioner) might sort based on specific recipient names or delivery routes for efficiency.

In summary, the Partition function is a critical component that ensures the correct grouping of intermediate data by key, enabling the Reduce tasks to perform meaningful aggregations and computations. Custom Partitioners provide the flexibility to optimize data distribution and cater to specific processing needs.

Map phase

Partition function, or Partitioner, ensures each key and its list of values is passed to one and only one Reduce task or Reducer
The number of partitions is determined by the (default or user-defined) number of Reducers
Custom Partitioners are developed for various practical purposes

MapReduce Data Processing Model

Outline

Key-value pairs
MapReduce model
Map phase
Reduce phase
Shuffle and sort
Combiner phase
Example
Running MapReduce jobs

Input to the Reduce Phase:

Output of Map Phase (via Shuffle and Sort): The input to each Reduce task is the culmination of the Map phase's output, specifically after it has undergone the crucial shuffle and sort operations.

Shuffle: The intermediate key-value pairs generated by all the Map tasks are transferred across the network to the machines running the Reduce tasks. The framework ensures that all pairs with the same intermediate key are sent to the same Reducer.

Sort: Once the data arrives at a Reducer, it is sorted by the intermediate key. This groups all the values associated with the same key together in a structured manner.

Key and Value List: Consequently, the reduce() function within a Reducer receives:

intermediate_key: A unique key that was generated by one or more Map tasks.

list(intermediate_value): A list containing all the intermediate values that were associated with that particular intermediate_key across all the Map tasks. The values in this list are typically sorted based on the key they arrived with (due to the sort phase).

2. Execution of the reduce() Function:

Per Key: Each Reduce task invokes the user-defined reduce() function once for each unique intermediate_key it receives.

Processing the Value List: The reduce() function's primary responsibility is to process the list of intermediate_value associated with the given intermediate_key. This is where the core aggregation, summarization, or final computation takes place.

Emitting Output: The reduce() function can emit zero or more final key-value pairs as its output:

(out_key, out_value): The out_key and out_value produced by the Reducer represent the result of the processing for that specific intermediate_key. The types of the output key and value can be different from the intermediate key and value.

Example (WordCount): The reduce() function receives an intermediate_key (a word) and a list(intermediate_value) (a list of '1's representing the counts from the Mappers). It then sums up the '1's to produce the out_value (the total count of the word), and the out_key remains the same (the word).

3. Output of the Reduce Phase:

Final Result: The collection of (out_key, out_value) pairs emitted by all the Reduce tasks constitutes the final output of the MapReduce job.

Storage: This output is typically written to a distributed file system like HDFS.

Reduce Phase

Input of the Reduce phase is output of the Map phase (via shuffle-and sort)
Each Reduce task (or Reducer) executes a reduce() function for each intermediate key and its list of associated intermediate values

The output from each reduce() function is zero or more key-values

		    
reduce (intermediate_key, list (intermediate_value)) -> (out_key, out_value)

Note that, in the reality, an output from Reducer may be an input to another Map phase in a complex multistage computational workflow

Example of Reduce Functions

The simplest and most common reduce() function is the Sum Reducer, which simply sums a list of values for each key
A count operation is as simple as summing a set of numbers representing instances of the values you wish to count
Other examples of reduce() function are max() and average()

MapReduce Data Processing Model

Outline

Key-value pairs
MapReduce model
Map phase
Reduce phase
Shuffle and sort
Combiner phase
Example
Running MapReduce jobs

Shuffle-and-sort is the process where data are transferred from the Mapper to the Reducer. It is often considered the "heart" of MapReduce, where the "magic" of data aggregation and processing truly happens.

The most important purpose of Shuffle-and-sort is to minimize data transmission through a network. This is achieved by intelligently grouping and sorting data before it is sent across the cluster, reducing the overall network overhead and improving efficiency.

In general, during Shuffle-and-Sort, the Mapper output is sent to the target Reduce task according to a partitioning function. This function determines which Reducer will receive a particular key-value pair, ensuring that all values for a given key are directed to the same Reducer. Following partitioning, the data destined for each Reducer is then sorted by key, allowing for efficient processing and aggregation on the Reducer side.

Shuffle and Sort

Shuffle-and-sort is the process where data are transferred from Mapper to Reducer
- It is the heart of MapReduce where the "magic" happens
The most important purpose of Shuffle-and-sort is to minimise data transmission through a network
In general, in Shuffle-and-Sort, the Mapper output is sent to the target Reduce task according to the partitioning function

MapReduce Data Processing Model

Outline

Key-value pairs
MapReduce model
Map phase
Reduce phase
Shuffle and sort
Combiner phase
Example
Running MapReduce jobs

The Combiner phase in MapReduce is an optional, but highly beneficial, optimization step that acts as a "mini-reducer" at the Mapper's local machine. Its primary purpose is to reduce the volume of intermediate data generated by the Mappers before it's sent across the network to the Reducers. This significantly minimizes network traffic, which is often the most expensive operation in a distributed computing environment.

Here's a breakdown of the Combiner phase's structure and how it fits into the MapReduce workflow. The Combiner runs after the Mapper and before the Partitioner and the Shuffle-and-Sort phase. The typical order of execution in a MapReduce job with a Combiner is:

Mapper: Processes the raw input data and emits intermediate key-value pairs.

Combiner (Optional): Takes the output of the Mapper and performs a local aggregation or reduction on the key-value pairs with the same key.

Partitioner: Determines which Reducer will receive each key-value pair. This happens after the Combiner has potentially reduced the data volume.

Shuffle-and-Sort: Data is transferred to the appropriate Reducers and sorted by key.

Reducer: Processes the aggregated and sorted data from multiple Mappers (or Combiners) to produce the final output.

Combiner phase

A structure of Combiner phase

If the Reduce function is both commutative (the order of operands does not change the result, e.g., a+b=b+a) and associative (the grouping of operands does not change the result, e.g., (a+b)+c=a+(b+c)), then a portion of the reduction logic can be performed before the Shuffle-and-Sort phase. In this specific scenario, this pre-reduction function is called a Combiner function.

For example, sum and count are commutative and associative operations. You can sum numbers in any order or group them in any way and still get the same total. Similarly, you can count items incrementally. However, average is not commutative and associative in the context of partial aggregation. If you calculate the average of subsets of data, you cannot simply average those partial averages to get the overall average; you need the sum of values and the count of values.

The appropriate use of a Combiner can significantly minimize the amount of data transferred to the Reduce phase, thereby substantially reducing network transmit overhead. This is because the Combiner performs a local aggregation on the output of each Mapper before the data is sent across the network. Instead of sending numerous individual key-value pairs, the Combiner sends fewer, partially aggregated key-value pairs, leading to considerable bandwidth savings.

Combiner phase

If the Reduce function is commutative and associative then it can be performed before the Shuffle-and-Sort phase
In this case, the Reduce function is called a Combiner function
For example, sum and count is commutative and associative, but average is not
The use of a Combiner can minimise the amount of data transferred to Reduce phase and in such a way reduce the network transmit overhead
A MapReduce application may contain zero Reduce tasks
In this case, it is a Map-Only application
Examples of Map-only MapReduce jobs
- ETL routines without data summarization, aggregation and reduction
- File format conversion jobs
- Image processing jobs

A Map-Only MapReduce job is a specialized type of MapReduce application where the entire processing is completed within the Mapper phase, and there are zero Reduce tasks. This means that after the Mappers produce their output, that output is directly written to the final output location without any subsequent aggregation, summarization, or grouping by Reducers.

Essentially, in a Map-Only job:

Input: The job starts with input data, just like any other MapReduce job.

Mapping: The Mappers read the input data, perform the required transformations, filtering, or processing on each record or block of data independently.

Direct Output: The output of each Mapper is directly written to HDFS (or the configured output file system). There is no "shuffle" or "sort" phase designed to group data for Reducers, because there are no Reducers to group data for. While some internal sorting might still occur for the Mapper's output buffers, it's not for the purpose of Reducer input.

No Reducing: The Reduce phase is completely bypassed.

When is Map-Only MapReduce Used?

Map-Only jobs are ideal for scenarios where:

Transformations are record-level and independent: Each input record can be processed in isolation without needing information from other records.

No aggregation or summarization is required: The goal is not to count, sum, average, or group data across multiple records.

Data filtering or cleansing: Removing unwanted records or correcting malformed ones.

Format conversion: Changing the structure or encoding of data.

Loading data into another system: Preparing data for import into a database or another processing pipeline.

Combiner phase

Map-Only MapReduce

Let's use an election analogy to explain the MapReduce framework, specifically focusing on how it handles counting votes in a large-scale election.

Imagine a country with millions of voters spread across thousands of polling booths. Your goal is to find out the total number of votes each candidate received.

1. The Overall Goal (The Job):

Counting all votes for each candidate across the entire country. This is your big data problem.

2. The Polling Booths (Mappers):

Map Phase: Each polling booth (Mapper) is responsible for counting the votes cast in its specific location.

As votes come in, the clerks at each booth (Mapper instances) record them. Instead of just a tick mark, imagine they write down pairs: (Candidate Name, 1 vote).

So, if "Alice" gets a vote, they write (Alice, 1). If "Bob" gets a vote, (Bob, 1).

Local Aggregation (Combiner - Optional but Smart): To be efficient, before sending their tally away, the clerks at a single polling booth might do a preliminary count. If Alice gets 50 votes at their booth, instead of sending 50 separate (Alice, 1) slips, they consolidate it into one (Alice, 50) slip. This is like the Combiner, reducing the local data before transmission.

3. The Regional Counting Centers (Reducers):

Reduce Phase: You can't have every polling booth send its tallies directly to one central office, as it would be overwhelmed. Instead, you set up regional counting centers (Reducers).

The Assignment (Partitioner): How do you decide which polling booth's tally goes to which regional center? You need a rule. A common rule is "all tallies for Candidate A go to Regional Center 1, all for Candidate B go to Regional Center 2," and so on. This is the Partitioner's job. It ensures that all votes for a single candidate ultimately end up at the same regional counting center.

Transportation (Shuffle): Once a polling booth has its local tallies (potentially combined), they are transported (shuffled) to their assigned regional counting centers.

Sorting (Sort): When the tallies arrive at a regional center, they might come in a random order (e.g., Alice's votes from Booth X, then Bob's from Booth Y, then Alice's from Booth Z). The regional center clerks first organize these tallies, putting all of Alice's votes together, then all of Bob's, and so on. This is the sorting step, ensuring that all (Candidate, VoteCount) pairs for a specific candidate are grouped together.

Final Count (Reduce): Now, at each regional center, all the slips for a specific candidate (e.g., all slips saying "Alice, 50", "Alice, 75", "Alice, 30" etc.) are finally combined by a clerk. They add up all the numbers for Alice to get Alice's total votes for that region (or, more accurately, the total votes for Alice from all booths that sent their data to this regional center).

4. The Final Report (Output):

Each regional counting center then publishes its final tally for the candidate(s) it was assigned (e.g., Regional Center 1 reports: "Alice: 1,500,000 votes").

The sum of these totals across all regional centers gives you the national total for each candidate.

Combiner phase

An election Analogy for MapReduce

MapReduce Data Processing Model

Outline

Key-value pairs
MapReduce model
Map phase
Reduce phase
Shuffle and sort
Combiner phase
Example
Running MapReduce jobs

The Problem: Average Social Contacts by Age

We have a massive database of 1 billion people (social.person). Each record likely contains at least two pieces of information relevant to our task:

age: The age of the person.

contacts: The number of social contacts that person has.

Our goal is to find, for each distinct age, the average number of contacts.

SQL-like Approach

The SQL query provided is a perfect, concise way to express this calculation in a relational database context:

SELECT age, AVG(contacts)

FROM social.person

GROUP BY age;

MapReduce Implementation for Distributed Data

Now, let's consider the scenario where these 1 billion records are not in a single centralized database but are distributed across different data nodes in a Hadoop-like distributed file system. This is where MapReduce comes in.

The core idea for calculating an average in MapReduce is that you cannot directly average values in the Map phase (because you only see a subset of the data). Instead, you need to:

Emit the sum and count for each group in the Map phase.

Aggregate the sums and counts in the Reduce phase.

Calculate the final average in the Reduce phase using the aggregated sum and count.

The Map function provided correctly implements the first step:

function Map is

input: integer K between 1 and 1000, representing a batch of 1

million social.person records

for each social.person record in the K-th batch do

let Y be the person age

let N be the number of contacts the person has

produce one output record (Y,(N,1)) -- (Age, (Sum_of_Contacts, Count_of_People))

repeat

end function

Explanation of the Map Function:

Input: Each Map task receives a "batch" of social.person records. Since the data is distributed, a Mapper will typically process a "split" of the input data, which could be a block of files or a logical chunk. The K here refers to one such batch/split.

Processing Each Record: For every social.person record within its assigned batch, the Mapper:

Extracts Y (the person's age).

Extracts N (the number of contacts that person has).

Output Record: The Mapper produces a key-value pair:

Key: Y (the age). This is the grouping key. All records with the same age will eventually be sent to the same Reducer.

Value: (N, 1). This is a tuple or pair.

N: Represents the sum of contacts for this single person.

1: Represents the count of people (i.e., one person) for this record.

Why (N, 1)?

We emit (N, 1) because to calculate the average later, we need two pieces of information for each age group:

The total sum of contacts for all people of that age.

The total count of people of that age.

Example

For a database of 1 billion people, compute the average number of social contacts a person has according to age

In SQL like language


SELECT age, AVG(contacts)
FROM social.person
GROUP BY age

If the records are stored in different datanodes then in Map function is the following

function Map is
      input: integer K between 1 and 1000, representing a batch of 1
      million social.person records
      for each social.person record in the K-th batch do
        let Y be the person age
        let N be the number of contacts the person has
          produce one output record (Y,(N,1))
      repeat
end function

The Reduce Function (Crucial for Average Calculation)

While not explicitly provided in the prompt, the Reduce function is essential to complete the average calculation:

function Reduce is

input: key Y (an age), list_of_values [(N1,1), (N2,1), ..., (N_k,1)]

let total_contacts = 0

let total_people = 0

for each (N, C) in list_of_values do -- C will always be 1 from the Mapper

total_contacts = total_contacts + N

total_people = total_people + C

repeat

let average_contacts = total_contacts / total_people

produce one output record (Y, average_contacts)

end function

Explanation of the Reduce Function:

Input: Each Reduce task receives one unique Y (age) as its key, and a list of all the (N, 1) tuples produced by all Mappers for that specific age.

Aggregation: The Reducer iterates through this list:

It sums up all the N values (total contacts for this age group).

It sums up all the 1 values (total number of people for this age group).

Final Calculation: Once all (N, 1) pairs for that Y are processed, it calculates the average_contacts by dividing total_contacts by total_people.

Output: The Reducer produces a final key-value pair: (Age, Average_Contacts).

This MapReduce pattern effectively distributes the computation of sums and counts, then centralizes the final averaging, allowing for scalable processing of massive datasets.

Example

Then Reduce function is the following

function Reduce is
       input: age (in years) Y
       for each input record (Y,(N,C)) do
         Accumulate in S the sum of N
         Accumulate in C_new the sum of C
       repeat
       let A be S/C_new
       produce one output record (Y,(A,C_new ))
end function

MapReduce sends the codes to the location of each data batch (not the other way around)
Question: the output from Map is multiple copies of (Y, (N, 1)), but the input to Reduce is (Y, (N, C)), so what fills the gap?

A MapReduce application in Hadoop is fundamentally a Java implementation of the abstract MapReduce programming model, tailored to solve a specific data processing problem. It defines how data should be mapped and reduced to achieve a desired outcome. The quintessential example, and often the "hello world" of Hadoop, is the Word Count application.

Beyond just the Map and Reduce functions, a complete Hadoop MapReduce application in Java typically involves several key components that orchestrate the entire distributed process:

Driver Program (Job Configuration):

Mapper Class:

Reducer Class:

Combiner Class (Optional):

Partitioner Class (Optional):

InputFormat and OutputFormat:

When a Hadoop MapReduce application is submitted, the Hadoop YARN (Yet Another Resource Negotiator) resource manager takes over, allocates resources (containers) on the cluster nodes, and orchestrates the execution of these Map and Reduce tasks in a distributed and fault-tolerant manner. This entire structure allows developers to focus on the core business logic (what to Map and what to Reduce) while Hadoop handles the complexities of distributed execution, data movement, and fault recovery.

Example

A MapReduce application in Hadoop is a Java implementation of the MapReduce model for a specific problem, for example, word count

Example

Sample processing on a screen

Sample processing on a screen

1 Processing messages

Why Hadoop displays so many messages ?

here are several key reasons for this verbosity, rooted in its design as a distributed, fault-tolerant, and massively scalable system:

Distributed Nature and Coordination:

Orchestration Details: Hadoop jobs involve many moving parts: a JobClient submitting the job, the YARN ResourceManager allocating resources, the NodeManagers launching containers, and individual Map and Reduce tasks executing on various nodes. Each of these components needs to communicate its status and progress.

Heartbeats and Health Checks: Components (like NodeManagers) constantly send "heartbeat" messages to the ResourceManager to indicate they are alive and well. These might not always be logged directly to the console, but they are part of the background chatter.

Task Lifecycle: Every Map and Reduce task goes through a lifecycle (submitted, running, successful, failed). Messages are generated for each stage of each task to provide visibility.

Progress Tracking and Monitoring:

Visibility into Large Jobs: Hadoop is designed for jobs that can run for minutes, hours, or even days, processing petabytes of data. Users need to know if the job is progressing, if it's stuck, or if tasks are failing. Verbose messages provide this essential feedback.

Counter Updates: Hadoop tracks numerous counters (e.g., bytes read, bytes written, records processed, custom counters). These counters are updated and often displayed periodically, giving a high-level summary of job progress.

Percentage Completion: You'll frequently see messages about the percentage completion of Map and Reduce phases. This is critical for understanding the overall job status.

Fault Tolerance and Debugging:

Error Reporting: When a task fails on a specific node, Hadoop will attempt to retry it. The messages explain which task failed, why it failed (e.g., stack trace), and if it's being retried. This is crucial for debugging.

Diagnostics: Detailed logs help operators and developers diagnose issues. Without sufficient messages, pinpointing the cause of a problem in a distributed environment would be incredibly difficult. Imagine a single line error message for a failure that happened across 1000 nodes!

Resource Management: Messages might indicate resource contention, memory issues, or other system-level problems that need attention.

Logging Levels:

Configurable Verbosity: Hadoop uses standard logging frameworks (like Log4j in Java). By default, the logging level is often set to INFO, which includes many operational messages.

Changing Levels: You can configure Hadoop's logging levels to be less verbose (e.g., WARN or ERROR only), but this is often done for production deployments where monitoring tools aggregate logs, not for interactive command-line use where immediate feedback is desired. For initial development and troubleshooting, INFO is usually preferred.

Historical Context (Console vs. Logs):

In earlier versions and for many command-line utilities, printing messages directly to stdout (the console) was the primary way to provide immediate feedback to the user running the job.

While modern Hadoop environments rely heavily on centralized log aggregation (like Splunk or ELK stack) for in-depth analysis, the console output still serves as a quick "dashboard" during job execution.

In summary, Hadoop's verbosity is a deliberate design choice that prioritizes transparency, debuggability, and operational monitoring in a complex distributed system. While it can seem overwhelming at first, every message serves a purpose in ensuring the robust and efficient execution of large-scale data processing jobs.

Example

Sample processing on a screen

MapReduce Data Processing Model

Outline

Key-value pairs
MapReduce model
Map phase
Reduce phase
Shuffle and sort
Combiner phase
Example
Running MapReduce jobs

1. Client Submits MapReduce Job

The process begins with the Client, which is typically a user running a Hadoop command or a program that uses Hadoop's API (e.g., a Java application, a Python script using a Hadoop client library).

Job Definition: The client application first defines the MapReduce job. This involves:

Specifying the input and output paths in HDFS.

Identifying the Mapper, Reducer, Combiner (optional), and Partitioner (optional) classes.

Setting various job configuration parameters (e.g., number of reducers, memory limits for tasks).

Job Submission: The client then sends the job definition, along with the necessary JAR files containing the application code, to the YARN ResourceManager. This submission effectively tells YARN, "Here's a job I want to run; please manage its execution."

2. YARN ResourceManager Coordinates Resource Allocation

The YARN ResourceManager (RM) is the central authority in a Hadoop cluster responsible for managing and allocating computing resources. It's like the air traffic controller for all applications running on the cluster.

Receives Job Submission: The RM receives the job submission from the client.

Schedules ApplicationMaster: For each MapReduce job (or any YARN application), the RM initiates a dedicated process called the ApplicationMaster (AM). The RM finds a suitable NodeManager to launch this AM.

Resource Negotiation: The AM, once launched, is responsible for negotiating resources (CPU, memory, disk I/O) with the RM on behalf of its application. It tells the RM, "I need X number of Map tasks and Y number of Reduce tasks, each requiring Z amount of memory."

Tracking Resources: The RM maintains a global view of all available resources in the cluster and tracks which resources are being used by which applications.

Granting Resources: Based on its scheduling policies (e.g., fairness, capacity), the RM grants resource containers to the AM. A "container" is a logical bundle of resources (CPU, memory) on a specific NodeManager.

3. YARN NodeManager(s): Launch & Monitor Containers on Machines in the Cluster

The YARN NodeManager (NM) runs on each data node (or worker node) in the Hadoop cluster. It's the "agent" that interacts directly with the physical machine's resources.

Registers with ResourceManager: Each NodeManager periodically sends "heartbeat" messages to the ResourceManager, reporting its resource availability and the status of its containers.

Receives Container Launch Commands: When the ResourceManager grants a container to an ApplicationMaster, the ApplicationMaster then instructs a specific NodeManager (that was assigned the container by the RM) to launch that container.

Launches & Manages Containers: The NodeManager is responsible for:

Launching Containers: It creates isolated environments (containers) on its machine, allocating the specified CPU and memory.

Downloading Code: It pulls the necessary JAR files (containing the Mapper, Reducer, etc.) from HDFS to run the application code within the container.

Executing Tasks: It then launches the actual Map or Reduce task (a Java Virtual Machine process) inside the container.

Monitoring Resources: It continuously monitors the resource consumption (CPU, memory) of the running tasks within its containers. If a task exceeds its allocated resources, the NodeManager can terminate it.

Reporting Status: It reports the status and progress of the tasks running in its containers back to the ApplicationMaster.

Health Checks: It constantly checks the health of the physical machine and reports any issues to the ResourceManager.

Flow Summary:

Client submits job to ResourceManager.

ResourceManager launches ApplicationMaster on a NodeManager.

ApplicationMaster negotiates resources with ResourceManager.

ResourceManager grants containers to ApplicationMaster.

ApplicationMaster instructs NodeManagers to launch Map/Reduce tasks in allocated containers.

NodeManagers launch and monitor Map/Reduce tasks.

Map/Reduce tasks process data, write intermediate output, and send status updates back through NodeManagers to the ApplicationMaster.

ApplicationMaster oversees the entire job, handling failures, tracking progress, and reporting overall job status back to the ResourceManager (and eventually the Client).

Running MapReduce Jobs

Client submits Mapreduce job
YARN resource manager coordinates the allocation of computing resources in the cluster
YARN node manager(s): launch & monitor containers on machines in the cluster

The MapReduce ApplicationMaster (AM) is a pivotal component in the YARN architecture. It's not just a passive placeholder; it's the dedicated, on-site manager for a single MapReduce job.

Runs in a Container: Crucially, the AM itself runs as a process within a YARN container, just like any other Map or Reduce task. This means it's treated as a first-class citizen of the cluster, subject to resource allocation and monitoring by the NodeManager where it's launched. This design makes the AM fault-tolerant: if an AM fails, the ResourceManager can restart it in a new container, allowing the job to resume.

Coordinates the Tasks in a MapReduce Job: Once launched, the AM takes full responsibility for the lifecycle of its specific MapReduce job. Its responsibilities include:

Task Scheduling: Breaking down the job into individual Map and Reduce tasks. It then requests containers from the ResourceManager to run these tasks.

Task Launching: Once containers are granted by the ResourceManager, the AM instructs the respective NodeManagers to launch the Map and Reduce tasks within those containers.

Resource Negotiation: Continuously communicating with the ResourceManager to acquire necessary CPU and memory resources for its tasks.

Progress Tracking: Monitoring the execution of all Map and Reduce tasks, tracking their percentage completion, and keeping tabs on their status (running, succeeded, failed).

Fault Tolerance (Task Level): If a Map or Reduce task fails, the AM detects this failure and can decide to re-launch that task on another available NodeManager. It manages the retry logic for failed tasks.

Heartbeats and Status Updates: Tasks periodically send heartbeats and status updates to the AM, which in turn reports overall job progress back to the ResourceManager and ultimately to the client.

Results Aggregation (for MapReduce): For MapReduce jobs, it's the AM that monitors when all Mappers complete and then orchestrates the Shuffle-and-Sort phase to prepare data for Reducers. It also manages the final output commit.

Job Completion: Once all tasks are successfully completed, the AM reports job completion to the ResourceManager and then shuts itself down.

HDFS (Hadoop Distributed File System) is not just for storing the massive input and output data; it's absolutely essential for sharing all the files required for the MapReduce job's execution across the distributed cluster.

Running MapReduce Jobs

MapReduce application master runs in a container, and coordinates the tasks in a MapReduce job
HDFS is used for sharing job files between the other files

References

White T., Hadoop The Definitive Guide: Storage and Analysis at Internet Scale, O'Reilly, 2015