06JavaMapReduceApplication

Java MapReduce Application

Outline

Building blocks of MapReduce program
Hadoop streaming
Word Count: The "Hello, World" of MapReduce
Miscellaneous

The present slide shows MapReduce execution flow

Job Submission: The Driver submits the job to the Hadoop YARN (Yet Another Resource Negotiator) ResourceManager.

Resource Allocation: YARN allocates resources (containers) for Mapper and Reducer tasks.

Splitting: The input data is logically split into smaller chunks. Each split is processed by a single Mapper task.

Mapping Phase: Mapper tasks execute, processing their assigned input splits and emitting intermediate key-value pairs.

Shuffling and Sorting: Hadoop collects the intermediate output from all Mappers. It then shuffles these pairs (grouping values by key) and sorts them. This sorted data is then sent to the appropriate Reducers. (This is where the Partitioner plays a role).

Reducing Phase: Reducer tasks execute, taking the grouped and sorted intermediate key-value pairs as input. They perform the aggregation logic and emit the final output.

Output Writing: The final output from the Reducers is written to the specified output directory in HDFS (Hadoop Distributed File System).

Job Completion: The Driver monitors the job and reports its success or failure.

This structured approach allows MapReduce to efficiently process massive datasets by distributing computation across a cluster of machines, making it a powerful paradigm for big data analytics.

In Hadoop MapReduce, ToolRunner is a utility class that helps in running classes that implement the Tool interface. It's an extremely common and recommended way to structure your MapReduce driver code because it simplifies command-line argument parsing and configuration management.

Building blocks of MapReduce program

Structure of MapReduce Java Application

Building blocks of MapReduce program

Mapper, Reducer, Combiner, and Partitioner classes correspond to their counterparts in the MapReduce model
- These classes implement the MapReduce logic
The Driver or ToolRunner in a MapReduce program represents the client program
- The main method of a MapReduce program is in the Driver or ToolRunner
- The code of the two is very standard

At its most fundamental level, an elementary MapReduce program indeed consists of just three core components: a Mapper class, a Reducer class, and a Driver class. These three pieces form the minimal set required to define and execute a basic MapReduce job.

The Flow of an Elementary MapReduce Program

- Driver: You run the Driver class.

- Driver: The Driver configures a Job object, specifying:

Which Mapper class to use.

Which Reducer class to use.

The data types for the Mapper's output.

The data types for the Reducer's output.

The input data location.

The output data location.

- Driver: The Driver submits the configured Job to the Hadoop cluster.

- Hadoop: Hadoop's YARN (Yet Another Resource Negotiator) component allocates resources and starts Mapper tasks.

- Mapper Tasks: Each Mapper task reads a portion of the input data, processes it using its map() method, and emits intermediate key-value pairs.

- Hadoop (Shuffle & Sort): Hadoop automatically collects all intermediate pairs, groups them by key, and sorts them. It then sends each unique key and its associated values to a Reducer.

- Reducer Tasks: Each Reducer task receives a key and all values for that key, processes them using its reduce() method, and emits the final output key-value pairs.

- Hadoop: The final output is written to the specified output directory in HDFS.

- Driver: The Driver waits for the job to complete and reports its status.

As the main method is contained in the Driver, sometimes (but not always) it is convenient to make Mapper and Reducer as inner classes in Driver, which contains routine codes.

Building blocks of MapReduce program

An elementary MapReduce program consists only of a Mapper class, a Reducer class and a Driver
As the main method is contained in the Driver, sometimes (but not always) it is convenient to make Mapper and Reducer as inner classes in Driver, which contains routine codes

Building blocks of MapReduce program: Driver

Driver is the program which sets up and starts a MapReduce application
Driver code is executed on the client; this code submits the application to the ResourceManager along with the application's configuration
Driver can submit the job asynchronously (in a non-blocking fashion) or synchronously (waiting for the application to complete before performing another action)– the second way will be used in our examples
Driver can also configure and submit more than one application; for instance, running a workflow consisting of multiple MapReduce applications

Building blocks of MapReduce program: Mapper

Mapper Java class contains a map() method
Its object instance iterates through the input to execute a map() method, using the InputFormat and its associated RecordReader
The number of HDFS blocks for the file determines the number of input splits, which, in turn, determines the number of Mapper objects (or Map tasks) in a MapReduce application (also see the previous lecture)
Mappers do most of the heavy lifting in data processing in MapReduce, as they read the entire input file for the application
Mappers can also include setup and cleanup code to run in any given object lifespan
Setup is called before the map() method; cleanup is called after it

Building blocks of MapReduce program: Reducer

Reducer runs against a partition and each key and its associated values are passed to a reduce() method inside Reducer class
Reduce's InputFormat matches Mapper's OutputFormat
While Mapper usually do the data preparation, for example, filtering and extracting), Reducer usually contains the main application logic
- For example, summation, counting, and averaging operations are implemented in Reducers
The runtime of Reducer instances is usually faster (and much faster in some cases) than the runtime of Mapper instances

Java MapReduce Application

Outline

Building blocks of MapReduce program
Hadoop streaming
Word Count: The "Hello, World" of MapReduce
Miscellaneous

Hadoop Streaming is a utility that comes with the Apache Hadoop distribution. It allows you to write MapReduce programs in any language you want (like Python, Perl, Ruby, or even Bash scripts) instead of being limited to Java.

The Core Idea: Standard Input/Output (STDIN/STDOUT)

Hadoop Streaming leverages the Unix philosophy of standard input and output. It treats your custom scripts as a "black box" that reads data from stdin and writes results to stdout. Hadoop then takes care of all the complex distributed computing aspects.

How it Works (Simplified):

Input Data: Hadoop reads your input data (files in HDFS) and feeds lines of it to your mapper script's stdin.

Mapper Script: Your mapper script reads lines from stdin. For each line, it processes the data and typically emits key-value pairs to stdout, with the key and value separated by a tab character

Shuffle and Sort: Hadoop collects the output from all mappers, shuffles them across the cluster, and sorts them by key. All values for the same key are grouped together.

Reducer Script: Hadoop feeds the grouped key-value pairs (where all values for a given key are presented sequentially) to your reducer script's stdin. Your reducer script reads these key-value pairs. It performs aggregation or other operations on the values for each key.

It then emits the final results to stdout (again, typically tab-separated key-value pairs).

Output Data: Hadoop collects the output from all reducers and writes it to output files in HDFS.

Key Components and Options:

When running a Hadoop Streaming job, you typically specify:

–mapper : The path to your mapper script (can be local or on HDFS).

–reducer : The path to your reducer script.

–input : The HDFS path to your input data.

–output : The HDFS path where the results will be written.

–file ,...: Used to distribute your scripts and any other necessary files (like configuration files or libraries) to the Hadoop cluster nodes.

–D mapreduce.job.reduces=: (Optional) To specify the number of reducers.

–inputformat : (Optional) To specify a custom input format.

–outputformat : (Optional) To specify a custom output format.

–numReduceTasks : Another way to specify the number of reducers.

–partitioner : (Optional) To specify a custom partitioner for controlling which reducer receives which key.

–combiner : (Optional) To specify a combiner script (like a local reducer) to optimize performance.

Hadoop streaming

Hadoop streaming is an utility that allows to create and process Map/Reduce jobs with any executable or script as the mapper and/or the reducer

For example counting lines and words and characters in a file located in Input folder in HDFS

 
mapred streaming  -input Input  -output Output  -mapper /bin/cat  -reducer /usr/bin/wc

mapred is an utility located in $HADOOP_HOME/bin
A parameter -input determines input folder or file name in HDFS
A parameter -output determines output folder in HDFS
A parameter -mapper determines executable or Java Class name
A parameter -reducer determines executable or Java Class name
cat is a program that concatenates and prints the files to standard output
wc is a program that counts lines, words, and characters in each input file

In the example below a command the mapper and the reducer are executables that read the input from standard input (line by line) and emit the output to standard output

mapred utility creates a Map/Reduce job, submits the job to a cluster, and monitors processing of the job until it completed

mapred streaming -input /user/words.txt -output /user/output -mapper /bin/cat -reducer /usr/bin/wc

Let a file words.txt in /user in HDFS contains

word1 word2

word3

Then the outcomes from mapred in /user/output/part-00000 are the following

2 3 20

Why the total number of characters is 20?

mapper emits:

word1 word2TABNULL (13 characters) (key-value are separated with TAB, (see above) missing is represented by NULL placeholder)

word3TABNULL (7 characters)

Hadoop streaming

In the example below

 
mapred streaming  -input /user/words.txt  -output /user/output  -mapper /bin/cat  -reducer /usr/bin/wc

the mapper and the reducer are executables that read the input from standard input (line by line) and emit the output to standard output
mapred utility creates a Map/Reduce job, submits the job to a cluster, and monitors processing of the job until it completed
Let a file words.txt in /user in HDFS contains
Then the outcomes from mapred in /user/output/part-00000 are the following

Hadoop streaming

Processing of mapper is performed in the following way:
- A mapper starts processing the executable (in our example /bin/cat) as a separate process when the mapper is initialized
- The mapper converts its inputs into lines and feeds the lines to the standard input of the process
- Simultaneously, the mapper collects the lines from the standard output of the process and converts each line into a key/value pair
- The key/value pairs are collected as the output of the mapper
- By default, the prefix of a line up to the first tab character is the key and the rest of the line (excluding the tab character) is the value

Hadoop streaming

Processing of reducer is performed in the following way:
- A reducer task starts its executable (in our example /user/bin/wc as a separate process
- The reducer task converts its input key/values pairs into lines and submits the lines to the standard input of the process
- Simultaneously, the reducer collects the lines from the standard output of the process and converts each line into a key/value pair
- The key/value is collected as the output of the reducer
- By default, the prefix of a line up to the first tab character is the key and the rest of the line (excluding the tab character) is the value

Hadoop streaming

Hadoop streaming supports streaming command options as well as generic command options

Generic command options include

   
-conf configuration_file      Specify an application configuration file
-D property=value 	      Use value for given property
-fs host:port or local 	      Specify a namenode
-files 	Optional 	      Specify comma-separated files to be copied to the Map/Reduce cluster
-libjars 	              Specify comma-separated jar files to include in the classpath
-archives 	              Specify comma-separated archives to be unarchived on the compute machines

Hadoop streaming

Sample applications of generic command options
Specifying map-only jobs
Specifying number of reducers

Specifying how lines are Split into key/value Pairs

   
-D stream.map.output.field.separator=.
-D stream.num.map.output.key.fields=4

Streaming command options include

-input directoryname or filename Input location for mapper

-output directoryname Required Output location for reducer

-mapper executable or JavaClassName Mapper executable, default IdentityMapper

-reducer executable or JavaClassName Reducer executable, default IdentityMapper

-file filenameMake the mapper, reducer, or combiner executable available locally

-inputformat JavaClassName Class you supply should return key/value pairs of Text class

-outputformat JavaClassName Class you supply should take key/value pairs of Text class

-partitioner JavaClassName Class that determines which reduce a key is sent to

-combiner streamingCommand or JavaClassName Combiner executable for map output

-cmdenv name=value Pass environment variable to streaming commands

-inputreader For backwards-compatibility: specifies a record reader class

-verbose Verbose output

-lazyOutput Create output lazily

-numReduceTasks Specify the number of reducers

-mapdebug Script to call when map task fails

-reducedebug Script to call when reduce task fails

Hadoop streaming

Streaming command options include

   
-input directoryname or filename      Input location for mapper
-output directoryname 	Required      Output location for reducer
-mapper executable or JavaClassName   Mapper executable, default IdentityMapper
-reducer executable or JavaClassName  Reducer executable, default IdentityMapper
-file filename                        Make the mapper, reducer, or combiner executable available locally
-inputformat JavaClassName 	      Class you supply should return key/value pairs of Text class 
-outputformat JavaClassName 	      Class you supply should take key/value pairs of Text class
-partitioner JavaClassName 	      Class that determines which reduce a key is sent to
-combiner streamingCommand or JavaClassName  Combiner executable for map output
-cmdenv name=value 	              Pass environment variable to streaming commands
-inputreader 	                      For backwards-compatibility: specifies a record reader class
-verbose 	                      Verbose output
-lazyOutput 	                      Create output lazily
-numReduceTasks 	              Specify the number of reducers
-mapdebug 	                      Script to call when map task fails
-reducedebug 	                      Script to call when reduce task fails

Sample applications of streaming command options

Specifying Java class as the mapper and/or the reducer

mapred streaming -input Input -output Output \

-inputformat org.apache.hadoop.mapred.KeyValueTextInputFormat \

-mapper org.apache.hadoop.mapred.lib.IdentityMapper \

-reducer /usr/bin/wc

Packaging files with job submissions

mapred streaming -input Input -output Output \

-mapper myPythonScript.py \

-reducer /usr/bin/wc \

-file myPythonScript.py

Specifying other plugins for jobs

-inputformat JavaClassName

-outputformat JavaClassName

-partitioner JavaClassName

-combiner streamingCommand or JavaClassName

Advantages of Hadoop Streaming:

Language Agnostic: The biggest advantage. You don't need to learn Java to write MapReduce jobs. This allows teams to leverage existing skills in other scripting languages.

Rapid Prototyping: Often faster to write and test simple MapReduce logic in scripting languages.

Leverages Existing Tools: You can use command-line utilities (like grep, awk, sed) directly in your mapper or reducer scripts, or chain them together.

Easier Debugging: Debugging individual mapper and reducer scripts can be simpler as they are essentially standalone programs.

Disadvantages of Hadoop Streaming:

Performance Overhead: There's a slight performance overhead due to the constant spawning of processes and inter-process communication, compared to native Java MapReduce jobs. This overhead is usually negligible for most use cases unless you have extremely tight performance requirements.

Limited API Access: You don't have direct access to the full Hadoop MapReduce API. This means you can't easily use advanced features like counters, custom Writable objects, or complex data types that are readily available in Java.

Error Handling: Error handling can be trickier. If your script crashes, Hadoop will mark the task as failed, but debugging the specific script error might require careful logging within your script.

Serialization: Data is serialized as text (strings), which can be less efficient than Hadoop's native binary serialization (Writable objects).

When to Use Hadoop Streaming:

When you need to perform simple data transformations or aggregations.

When your team is more proficient in scripting languages like Python or Perl than Java.

For quick prototyping and experimentation with data on Hadoop.

When you have existing shell scripts or utilities that can be easily adapted to the MapReduce paradigm.

In summary, Hadoop Streaming is a powerful and flexible t

Hadoop streaming

Sample applications of streaming command options

Specifying Java class as the mapper and/or the reducer

   
mapred streaming -input Input -output Output \
	         -inputformat org.apache.hadoop.mapred.KeyValueTextInputFormat \
                 -mapper org.apache.hadoop.mapred.lib.IdentityMapper \
                 -reducer /usr/bin/wc

Packaging files with job submissions

   
mapred streaming -input Input -output Output \
                 -mapper myPythonScript.py \
                 -reducer /usr/bin/wc \
                 -file myPythonScript.py

Specifying other plugins for jobs

   
-inputformat JavaClassName
-outputformat JavaClassName
-partitioner JavaClassName
-combiner streamingCommand or JavaClassName

Java MapReduce Application

Outline

Building blocks of MapReduce program
Hadoop streaming
Word Count: The "Hello, World" of MapReduce
Miscellaneous

Mapper:

It is a classic example of a mapper script for a Hadoop MapReduce word count program written in Python. This script is designed to be used with Hadoop Streaming, which allows you to write MapReduce jobs in languages other than Java by reading from standard input and writing to standard output.

#!/usr/bin/env python3

Shebang Line: This line is called a "shebang." It tells the operating system (specifically, the shell) that this script should be executed using the python3 interpreter found in the system's PATH. This is crucial for making the script executable directly.

import sys

Import sys module: The sys module provides access to system-specific parameters and functions. In this context, it's primarily used to access sys.stdin (standard input) and sys.stdout (standard output).

for line in sys.stdin:

Reading from Standard Input: This is the core of how the mapper receives data from Hadoop. When Hadoop runs your mapper, it feeds lines of text from your input files (which are typically stored in HDFS) one by one into the mapper's standard input. The for line in sys.stdin: loop iterates over each line of text provided by Hadoop.

line = line.strip():

The strip() method is a string method that removes leading and trailing whitespace (spaces, tabs, newlines, etc.) from the string.

When you read a line from sys.stdin, it typically includes the newline character (\n) at the end. strip() removes this, making it easier to process the actual content of the line.

words = line.split():

The split() method is another string method that splits a string into a list of substrings (words).

When called without any arguments, split() splits the string by any whitespace and discards empty strings. For example, "hello world".split() would result in ['hello', 'world']. Multiple spaces between words are treated as a single delimiter.

for word in words::

This is an inner for loop that iterates through each word in the words list that was created in the previous step.

print ('%s\t%s' % (word, 1)):

This line prints each word followed by a tab character (\t) and then the number 1.

%s is a placeholder for a string, and %d would be for an integer, though 1 is implicitly converted to a string here for %s. % (word, 1) is the old-style string formatting in Python, where the values in the tuple are substituted into the placeholders in the string. The output format of word\t1 is characteristic of the intermediate output often seen in MapReduce frameworks (like Hadoop). In a MapReduce context, this code would likely be the "mapper" part, where each word is emitted with a count of 1, and then a "reducer" would aggregate these counts for each unique word.

Word Count: The "Hello, World" of MapReduce

Implementation of word count program in Python

Implementation of mapper mapper.py

   
#!/usr/bin/env python3

import sys

for line in sys.stdin:
	line = line.strip()
	words = line.split()
	for word in words:
		print ('%s\t%s' % (word, 1))

Reducer:

current_word = None: Initializes a variable to store the word being currently processed. It's set to None initially because no word has been encountered yet.

current_count = 0: Initializes a variable to store the aggregated count for current_word.

word = None: This variable word is actually re-assigned within the loop. While it's initialized here, its purpose is primarily within the loop's scope. It will hold the word from the current input line.

for line in sys.stdin:: This loop iterates over each line received from the standard input. In a MapReduce context, this input would typically be the sorted output from the "Shuffle and Sort" phase, where all occurrences of the same word are grouped together.

line = line.strip(): Removes any leading or trailing whitespace (including the newline character \n) from the line.

word, count = line.split('\t', 1):

This is crucial. It splits the line string into two parts based on the tab character \t.

1 as the second argument to split() means "split at most once".

try...except ValueError:: This block attempts to convert the count (which is initially a string) into an integer.

If int(count) fails (i.e., count is not a valid integer), a ValueError is raised. In that case, continue skips the rest of the current iteration and moves to the next line of input. This handles malformed input gracefully.

Aggregation Logic: This is the core of the reducer.

if current_word == word:: If the word from the current input line is the same as the current_word being aggregated (meaning we're still processing the same group of words), then:

current_count += count: The count from the current line is added to the current_count.

else:: If the word from the current input line is different from current_word (meaning we've encountered a new unique word), then:

if current_word:: This check ensures that we don't print anything on the very first iteration when current_word is still None.

print ('%s\t%s' % (current_word, current_count)): The aggregated current_word and its current_count are printed to standard output. This completes the processing for the previous word.

current_count = count: The current_count is reset to the count of the new word.

current_word = word: The current_word is updated to the new word.

Final Output Handling: This block outside the loop is crucial. After the loop finishes (meaning all lines from sys.stdin have been processed), it's necessary to print the very last current_word and its current_count. The else block inside the loop only prints when a new word is encountered, so the final word processed would otherwise be missed.

The condition if current_word == word: here might seem redundant, as after the loop word would typically hold the last word processed by the loop, which should be equal to current_word. However, a more robust check might just be if current_word is not None: to ensure something was processed at all. In practice, for a reducer receiving sorted input, current_word will indeed be the last word from the stream, and word will also hold that last word's value after the final iteration.

Word Count: The "Hello, World" of MapReduce

Implementation of reducer reducer.py

#!/usr/bin/env python3

import sys

current_word = None
current_count = 0
word = None
for line in sys.stdin:
	line = line.strip()
	word, count = line.split('\t', 1)
	try:
		count = int(count)
	except ValueError:
		continue
	if current_word == word:
		current_count += count
	else:
		if current_word:
			print ('%s\t%s' % (current_word, current_count))
		current_count = count
		current_word = word
if current_word == word:
	print ('%s\t%s' % (current_word, current_count))

Implementation of word count program in modern Python

Implementation of mapper mapper.py

def read_input(file):: This defines a function named read_input that takes one argument, file. In the context of the main function, this file argument will typically be sys.stdin, which represents the standard input stream (where the program receives its data).

for line in file:: This loop iterates through each line in the provided file object. When file is sys.stdin, it reads lines from whatever is piped into the script or typed at the command line.

yield line.split(): This is the most interesting part of this function.

line.split(): This string method splits the line into a list of words. By default, split() splits on any whitespace (spaces, tabs, newlines) and removes empty strings from the result. For example, if line is "hello world\tpython", line.split() will return ['hello', 'world', 'python'].

yield: This keyword makes read_input a generator function. Instead of returning a single list of all words at once (which could consume a lot of memory for large files), it "yields" one list of words (from a single line) at a time. This is highly memory-efficient for processing large datasets, as it only processes one line's worth of data in memory at any given moment.

def main(separator='\t'):: Defines the main function of the script. It takes an optional argument separator, which defaults to a tab character (\t). This separator will be used to delimit the word and its count in the output.

data = read_input(sys.stdin): Calls the read_input generator function, passing sys.stdin as the file to read from. The data variable now holds the generator object, which will yield lists of words as it's iterated over.

for words in data:: This loop iterates through the lists of words yielded by the read_input generator. words here represents a list of words from a single line of the input.

for word in words:: This inner loop iterates through each individual word within the words list (i.e., each word on the current line).

print '%s%s%d' % (word, separator, 1): This is the core output part of the mapper.

'%s%s%d' % (word, separator, 1): This is string formatting.

%s: Placeholder for a string (the word).

%s: Placeholder for another string (the separator, which is \t).

The print statement outputs a line in the format word1. This is the standard output format for a MapReduce mapper in a word count scenario: each word found is emitted with a count of 1.

if __name__ == "__main__":

main()

This is a standard Python idiom. It ensures that the main() function is called only when the script is executed directly (not when it's imported as a module into another script).

Word Count: The "Hello, World" of MapReduce

Implementation of word count program in modern Python

Implementation of mapper mapper.py

   
#!/usr/bin/env python3

import sys

def read_input(file):
    for line in file:
        yield line.split()

def main(separator='\t'):
    data = read_input(sys.stdin)
    for words in data:
        for word in words:
            print '%s%s%d' % (word, separator, 1)

if __name__ == "__main__":
    main()

Reducer:

def read_mapper_output(file, separator='\t'):: This defines a generator function that reads the output from a mapper. It takes file (typically sys.stdin) and an optional separator (defaulting to a tab \t).

for line in file:: This loop iterates through each line received from the input file. In a MapReduce context, this file represents the sorted and grouped output from the "Shuffle and Sort" phase, which comes after the Mappers. Each line is expected to be in the format keyvalue.

yield line.rstrip().split(separator, 1): This is the core of this generator.

line.rstrip(): This removes any trailing whitespace, especially the newline character (\n), from the end of the line. This is important before splitting to avoid including the newline as part of the value.

.split(separator, 1): This splits the cleaned line string into a list of two elements based on the separator. The 1 as the second argument is crucial: it tells split() to split at most once. This ensures that if your key or value itself contains the separator, it's only split at the first occurrence, correctly separating the key from its associated value.

For example, if a line is hello\t1 (from a word count mapper), it will yield ['hello', '1'].

If a line were long_phrase with\ttabs\t1, split('\t', 1) would result in ['long_phrase with\ttabs', '1'].

yield: Like in the Mapper, yield makes this a generator. It produces one [key, value] pair (as a list) at a time, making it memory-efficient for large datasets.

In essence: This function is designed to parse the keyvalue lines generated by the mapper and provide them as an iterable of [key, value] lists.

from itertools import groupby: This imports the groupby function from the itertools module. groupby is incredibly useful for processing sorted sequences, as it groups consecutive identical elements. In a MapReduce reducer, the input is guaranteed to be sorted by key, making groupby perfect for this task.

from operator import itemgetter: This imports itemgetter from the operator module. itemgetter(0) is a function that, when applied to an iterable (like ['hello', '1']), returns the item at index 0 (i.e., 'hello'). It's used as the "key" function for groupby.

data = read_mapper_output(sys.stdin, separator=separator): This calls the read_mapper_output generator function, passing sys.stdin as the input source. data now holds the generator object that will yield [word, count] lists.

for current_word, group in groupby(data, itemgetter(0)):: This is the heart of the reducer's aggregation.

groupby(data, itemgetter(0)):

data: The input stream of [word, count] pairs.

itemgetter(0): This tells groupby to group elements based on their first element (index 0), which is the word (the key).

groupby yields pairs:

current_word: This is the key (the word) that groupby is currently processing. groupby ensures that all items in the group iterator have this same current_word.

group: This is an iterator that yields all the [word, count] pairs that have the current_word as their key. Because the input from "Shuffle and Sort" is sorted, all identical keys will be consecutive, which is exactly what groupby needs to work correctly.

try...except ValueError:: This block handles potential errors during count conversion.

rtotal_count = sum(int(count) for current_word, count in group): This line calculates the total count for the current_word.

for current_word, count in group: This iterates through each [word, count] pair within the group iterator. Remember, all words in this group will be the same as current_word.

int(count): Converts the string count (e.g., '1') to an integer.

sum(...): Sums up all these integer counts.

print "%s%s%d" % (current_word, separator, total_count): Prints the aggregated result to standard output in the format wordtotal_count.

except ValueError: pass: If any count within a group cannot be converted to an integer (e.g., if it's malformed), this try-except block catches the ValueError and simply passes, meaning it skips that particular group or problematic line without crashing the script.

Word Count: The "Hello, World" of MapReduce

Implementation of reducer reducer.py

#!/usr/bin/env python3

from itertools import groupby
from operator import itemgetter
import sys

def read_mapper_output(file, separator='\t'):
    for line in file:
        yield line.rstrip().split(separator, 1)

def main(separator='\t'):
    data = read_mapper_output(sys.stdin, separator=separator)
    for current_word, group in groupby(data, itemgetter(0)):
        try:
            total_count = sum(int(count) for current_word, count in group)
            print "%s%s%d" % (current_word, separator, total_count)
        except ValueError:
            pass

if __name__ == "__main__":
    main()

Java MapReduce Application

Outline

Building blocks of MapReduce program
Hadoop streaming
Word Count: The "Hello, World" of MapReduce
Miscellaneous

mapred streaming -input /user/words.txt -output /user/output -mapper ./mapper.py -reducer ./reducer.py

This is a more concise (and potentially older or shell-aliased) way to invoke Hadoop Streaming.

mapred streaming: This is the command to initiate a MapReduce streaming job. It's often an alias or a wrapper around the full hadoop jar ... command.

-input /user/words.txt:

-input: Specifies the input path(s) for the MapReduce job.

/user/words.txt: This is the path to the input data file or directory within the Hadoop Distributed File System (HDFS). The mapper will read data from this location. So, mapper.py will receive the contents of words.txt as its standard input.

-output /user/output:

-output: Specifies the output directory for the MapReduce job.

/user/output: This is the HDFS directory where the results of the reducer will be stored. Important: This directory must not exist before you run the job; Hadoop will create it. If it exists, the job will fail.

-mapper ./mapper.py:

-mapper: Specifies the executable script or program to be used as the mapper.

./mapper.py: This indicates that the mapper.py script (presumably a Python script, like the word count mapper we discussed) located in the current working directory will be used. Hadoop will distribute this script to all the Mappers (worker nodes) and execute it, piping input data to its standard input and capturing its standard output.

-reducer ./reducer.py:

-reducer: Specifies the executable script or program to be used as the reducer.

./reducer.py: This indicates that the reducer.py script (like the word count reducer we discussed) located in the current working directory will be used. Hadoop will distribute this script to all the Reducers (worker nodes), provide them with the shuffled and sorted output from the mappers as standard input, and capture their standard output.

$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-3.3.6.jar -input /user/numbers.txt -output /user/outputm -mapper ./filter.py -reducer ./reducer.py

This is the more explicit and standard way to invoke Hadoop Streaming, particularly useful if mapred isn't directly in your PATH or you need to specify a precise Hadoop version.

$HADOOP_HOME/bin/hadoop: This refers to the main Hadoop command-line utility. $HADOOP_HOME is an environment variable that points to your Hadoop installation directory.

jar $HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-3.3.6.jar:

jar: A subcommand of hadoop used to run a Java application packaged in a .jar file.

$HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-3.3.6.jar: This is the exact path to the Hadoop Streaming JAR file for version 3.3.6. This JAR file contains the necessary Java code to bridge external executables with the Hadoop MapReduce framework.

-input /user/numbers.txt:

Input data is /user/numbers.txt in HDFS. This suggests the data might be numerical, perhaps one number per line or a set of numbers on each line.

-output /user/outputm:

Output directory is /user/outputm in HDFS.

-mapper ./filter.py:

The mapper script is named filter.py. This suggests its role might be to filter data, perhaps only emitting certain numbers or transforming them based on some criteria.

-reducer ./reducer.py:

The reducer script is named reducer.py. This script would then aggregate or process the filtered output from filter.py.

Word Count: The "Hello, World" of MapReduce

Sample processing of MapReduce job

 
mapred streaming  -input /user/words.txt  -output /user/output \
         	  -mapper ./mapper.py  -reducer ./reducer.py

 
$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-3.3.6.jar \
                  -input /user/numbers.txt -output /user/outputm \
                  -mapper ./filter.py -reducer ./reducer.py

Miscellaneous

Compound MapReduce jobs
How to turn a (possibly complex) data processing problem into the MapReduce model ?
As a rule of thumb, for complex data processing problems, think about adding more jobs, rather than adding the complexity to jobs
For example, to find the mean of maximum daily temperature in each month in different weather stations, consider these two jobs:
- Compute the maximum daily temperature for every station-day key
- Compute the mean of the maximum daily temperature for every station-day-month key
A library ChainMapper allows to develop chained MapReduce Jobs

References

White T., Hadoop The Definitive Guide: Storage and Analysis at Internet Scale, O'Reilly, 2015
Hadoop Streaming
Hadoop with Python
Writing An Hadoop MapReduce Program In Python