04HDFSInterfaces

HDFS Interfaces

Outline

Hadoop Cluster vs. Pseudo-Distributed Hadoop
Shell Interface to HDFS
Web Interface to HDFS
Python Interface to HDFS
Internals of HDFS

A Hadoop cluster is deployed across a cluster of computer nodes. As Hadoop is developed in Java, all Hadoop services operate within Java Virtual Machines (JVMs) running on these cluster nodes.

Hadoop also offers a pseudo-distributed mode, where all necessary Hadoop services and their corresponding JVMs run on a single machine. In our specific setup, this machine is a Virtual Machine running either Ubuntu 14.04 or 22.04.

HDFS (Hadoop Distributed File System) provides several interfaces for interacting with the filesystem, allowing users to read, write, interrogate, and manage files. These interfaces include:

- The file system shell (Command-Line Interface): This can be accessed using either hadoop fs or hdfs dfs commands.

- Hadoop Filesystem Java API: A programmatic interface for Java applications.

- Hadoop simple Web User Interface: A web-based interface for basic monitoring and management.

- Snakebite pure Python HDFS client library: A client library specifically for Python applications to interact with HDFS

Hadoop Cluster vs. Pseudo-Distributed Hadoop

A Hadoop cluster is deployed in a cluster of computer nodes
- As Hadoop is developed in Java, all Hadoop services sit on Java Virtual Machines running on the cluster nodes
Hadoop provides a pseudo-distributed mode on a single machine
- All Java Virtual Machines for necessary Hadoop services are running on a single machine
- In our case this machine is a Virtual Machine running under Ubuntu 14.04 or 22.04
HDFS provides the following interfaces to read, write, interrogate, and manage the filesystem
- The file system shell (Command-Line Interface): hadoop fs or hdfs dfs
- Hadoop Filesystem Java API
- Hadoop simple Web User Interface
- Snakebite pure Python HDFS client library

HDFS Interfaces

Outline

Hadoop Cluster vs. Pseudo-Distributed Hadoop
Shell Interface to HDFS
Web Interface to HDFS
Python Interface to HDFS
Internals of HDFS

When working with Hadoop, you'll primarily interact with it through the Bash shell. To get started, it's a good practice to navigate to Hadoop's home directory:

$ cd $HADOOP_HOME

You'll mostly be using programs located in the bin and shell scripts in the sbin folders. Additionally, you'll find Java Archive (JAR) files in the share folder, which are crucial for Hadoop's functionality.

Starting and Verifying Hadoop Services

To fire up all the necessary Hadoop processes, you'll use the following command:

$ ./sbin/start-all.sh

For Hadoop to function correctly, all its core services must be running. You can verify the status of these services using the jps command, which lists the running Java processes. A healthy Hadoop cluster will show output similar to this:

$ jps

28530 SecondaryNameNode

11188 NodeManager

28133 NameNode

28311 DataNode

10845 ResourceManager

3542 Jps

This output confirms that the SecondaryNameNode, NodeManager, NameNode, DataNode, and ResourceManager are all active, indicating Hadoop is running properly.

Shell Interface to HDFS

Commands are provided in the shell Bash
Moving to Hadoop's home directory
You will mostly use programs in the bin and shell scripts in sbin folders, and use jar files in the share folder
Starting all Hadoop processes

Hadoop is running properly only if the above services are running

$ jps
28530 SecondaryNameNode
11188 NodeManager
28133 NameNode
28311 DataNode
10845 ResourceManager
3542 Jps

Even though a user account might be pre-configured in your virtual machine environment, it's essential to understand how HDFS user directories are managed.

Creating an HDFS User Account Directory

While the "bigdata" user account itself is assumed to be set up on the virtual machine, you need to create a corresponding home directory for this user within HDFS. This is done using the hdfs dfs -mkdir command with the -p flag, which creates parent directories if they don't already exist:

$ bin/hdfs dfs -mkdir -p /user/bigdata

This command establishes /user/bigdata as the home directory for the bigdata user within HDFS.

Creating a Folder within the HDFS Home Directory

Once your HDFS user directory is set up, you can create subfolders within it. For instance, to create an "input" folder in your HDFS home directory, you'd use:

$ hdfs dfs -mkdir input

When you execute this command from what HDFS considers your "home" directory (which, by default, is /user/, in this case /user/bigdata), it will create the input directory directly within /user/bigdata.

Viewing Folders in Your HDFS Home Directory

To verify the creation of the input folder and see other contents of your HDFS home directory, you can use the hdfs dfs -ls command without specifying a path. This command lists the contents of your current HDFS working directory, which defaults to your HDFS home directory:

$ hdfs dfs -ls

Found 1 item

drwxr-xr-x - bigdata supergroup 0 2017-07-17 16:33 input

The output confirms that an input directory exists, owned by bigdata and belonging to the supergroup. The drwxr-xr-x indicates it's a directory with read, write, and execute permissions for the owner, and read and execute permissions for the group and others.

Shell Interface to HDFS

Create a HDFS user account (already created in a virtual machine used by us)
Create a folder input in a home folder

View the folders in Hadoop home

$ hdfs dfs -ls
Found 1 item
drwxr-xr-x - bigdata supergroup 0 2017-07-17 16:33 input

What is a path to Hadoop home folder ?

$ hdfs dfs -ls /user/bigdata
Found 1 item
drwxr-xr-x - bigdata supergroup 0 2017-07-17 16:33 /user/bigdata/input

Uploading a File to HDFS

To transfer a local file from your machine to the HDFS input folder, you use the hdfs dfs -put command. For instance, to upload a file named README.txt (which should be present in your current local directory) into the HDFS input folder, you would execute:

$ hdfs dfs -put README.txt input

This command copies README.txt from your local filesystem into the input directory within your HDFS home directory (e.g., /user/bigdata/input/README.txt).

Verifying Contents of the HDFS input Folder

To confirm that the README.txt file has been successfully uploaded to the input folder in HDFS, you can list its contents using the hdfs dfs -ls command. While the user's provided example shows an extra hdfs dfs -put README.txt input command before the ls, the ls command is what actually verifies the upload:

$ bin/hdfs dfs -ls input

-rw-r--r-- 1 bigdata supergroup 1494 2017-07-12 17:53 input/README.txt

The output confirms the presence of README.txt within the input directory. It also provides details such as file permissions (-rw-r--r--), replication factor (implied as 1 in pseudo-distributed mode), owner (bigdata), group (supergroup), file size (1494 bytes), and the last modification timestamp.

Reading a File Located in HDFS

To view the contents of a file stored in HDFS directly through the shell, you use the hdfs dfs -cat command. For example, to display the contents of README.txt from the input folder:

$ bin/hdfs dfs -cat input/README.txt

< Contents of README.txt is displayed here >

This command will stream the entire content of the README.txt file to your console.

Getting Help on HDFS Shell Commands

Hadoop's HDFS shell interface provides a comprehensive help utility. To get a complete list of all available commands and their usage, you can use the hdfs dfs -help command:

$ bin/hdfs dfs -help

< A long list of commands of shell interface to HDFS displayed here >

This will display extensive documentation covering various operations you can perform on HDFS using the command-line interface, including options for file manipulation, directory management, permissions, and more.

Shell Interface to HDFS

Upload a file to HDFS to input folder

Check the contents of input folder

$ hdfs dfs -put README.txt input
$ bin/hdfs dfs -ls input
-rw-r--r-- 1 bigdata supergroup 1494 2017-07-12 17:53 input/README.txt

Read a file located in HDFS

$ bin/hdfs dfs -cat input/README.txt
< Contents of README.txt is displayed here >

Get help on all commands of shell interface to HDFS

$ bin/hdfs dfs -help
< A long list of commands of shell interface to HDFS displayed here >

HDFS Path Representation as a URI

In HDFS, file and directory paths are formally represented as Uniform Resource Identifiers (URIs), which include the hdfs:// prefix. This prefix indicates that the resource is located within the Hadoop Distributed File System.

For example:

hdfs://:/user/bigdata/input

This URI specifically refers to the input folder located within the /user/bigdata directory in HDFS. Here, would be the network name or IP address of your NameNode (the master node in HDFS), and is the RPC port the NameNode listens on (typically 9000).

hdfs://:/user/bigdata/input/README.txt

Similarly, this URI points directly to the README.txt file situated inside the input directory, which itself is under /user/bigdata in HDFS.

Simplified Path Notation in HDFS Shell

When you are interacting with the HDFS command-line interface (using hdfs dfs or hadoop fs), Hadoop provides a convenient simplification for path notation in its default configuration.

You can often omit the hdfs://:/user/ part of the URI.

This means that if you are the bigdata user and your current working directory in HDFS is set to /user/bigdata (which is typically the default), you can simply refer to files and directories relative to that location.

Therefore, the command:

hdfs dfs -ls input

is effectively a shorthand for the full URI:

hdfs dfs -ls hdfs://:/user/bigdata/input

Hadoop automatically resolves the relative path input to the absolute URI hdfs://:/user/bigdata/input based on the user's default HDFS home directory and the cluster's configuration.

Shell Interface to HDFS

The path in HDFS is represented as a URI with the prefix hdfs://

For example

 
hdfs://<hostname>:<port>/user/bigdata/input

refers to the input folder in HDFS under the user of bigdata

 
hdfs ://<hostname>:<port>/user/bigdata/input/README.txt

refers to the file README.txt in the above input directory in HDFS
When interacting with HDFS interface in the default setting, one can omitt IP, port, and user, and simply mention the directory or file

Thus, the full-spelling of hdfs dfs -ls input is

hdfs dfs -ls hdfs://<hostname>:<port>/user/bigdata/input

Let us extend a list of frequently used shell interface commands for HDFS with some more useful ones. I

Command Description

-du Display the disk usage statistics for a given path in HDFS.

-dus Display a summary of the disk usage statistics for a given path in HDFS.

-chown : Change the owner and/or group of a file or directory in HDFS.

-chmod Change the permissions of a file or directory in HDFS (e.g., 755, 644).

-chgrp Change the group of a file or directory in HDFS.

-mv Move a file or directory from one path to another within HDFS.

-cp Copy files or directories from one path to another within HDFS.

-get Copy files from HDFS to the local filesystem (similar to copyToLocal).

-getmerge Concatenate and copy files from HDFS to a single local file.

-test -e Check if a file or directory exists in HDFS. Returns 0 if exists, 1 otherwise.

-test -d Check if the path in HDFS is a directory. Returns 0 if it is, 1 otherwise.

-test -f Check if the path in HDFS is a file. Returns 0 if it is, 1 otherwise.

-tail Display the last 1KB of the file to stdout.

-tail -f Display appended data as the file grows.

-expunge Empty the trash directory in HDFS.

-setrep -w Wait for the block replication to reach the specified number.

-setrep [-R] Change the replication factor of files in HDFS. Use -R for recursive changes.

-df -h Display the capacity, free space, and used space of the HDFS filesystem in a human-readable format.

-help Display help information for a specific HDFS command.

This extended list covers common operations for managing files, directories, permissions, and disk usage within the Hadoop Distributed File System. Remember that these commands are typically prefixed with hdfs dfs or hadoop fs when executed in the shell. For example, to list files in the root directory of HDFS, you would use:

Bash

hdfs dfs -ls /

Bash

hadoop fs -ls /

Shell Interface to HDFS

Some of frequently used commands of shell interface to HDFS

 
Command          Description
-put             Upload a file (or files) from the local filesystem to HDFS
-mkdir           Create a directory in HDFS
-ls              List the files in a directory in HDFS
-cat             Read the content of a file (or files) in HDFS
-copyFromLocal   Copy a file from the local filesystem to HDFS (similar to
                 put)
-copyToLocal     Copy a file (or files) from HDFS to the local filesystem
-rm              Delete a file (or files) in HDFS
-rm -r           Delete a directory in HDFS

HDFS Interfaces

Outline

Hadoop Cluster vs. Pseudo-Distributed Hadoop
Shell Interface to HDFS
Web Interface to HDFS
Python Interface to HDFS
Internals of HDFS

The web interface to HDFS provides a browser-based way to interact with the Hadoop Distributed File System. It runs on the NameNode and DataNodes, offering a user-friendly alternative to the command-line interface.

Here's a breakdown of the HDFS web interface:

Accessing the Web Interface:

- NameNode UI: Typically accessible on port 9870 (for Hadoop 2.x it was 50070) of the NameNode's hostname or IP address. The URL would be something like http://:9870.

- DataNode UI: Each DataNode also runs a web server, usually on port 9864 (for Hadoop 2.x it was 50075). You can often find links to the DataNode UIs from the NameNode web interface.

- HttpFS UI: If the HttpFS service is running, it provides a REST API over HTTP and usually has a web UI on a configured port (e.g., 14000).

Web Interface of HDFS

Features of the Web Interface:

- File System Navigation: Browse the HDFS directory structure similar to a regular file explorer.

- File Information: View details of files and directories, such as:

Permissions, Owner and Group, Size, Last Modified Time, Replication Factor, Block Size

- Basic File Operations: Depending on the configuration and permissions, you might be able to perform actions like:

Creating Directories: Create new folders within HDFS;

Uploading Files: Upload files from your local machine to HDFS.; Downloading Files: Download files from HDFS to your local machine.

Deleting Files and Directories: Remove files and folders from HDFS.

Changing Permissions: Modify the access rights for files and directories.

Setting Replication Factor: Change the number of copies of a file's blocks.

Cluster Monitoring: Get a high-level overview of the cluster's health and resource utilization.

DataNode Status: Monitor the status and storage capacity of individual DataNodes.

Log Viewing: Access and view the logs of different HDFS components for troubleshooting.

Configuration Inspection: Examine the current HDFS configuration settings.

Metrics Display: View various performance metrics for the NameNode and DataNodes.

Safe Mode Monitoring: Check if the HDFS cluster is in Safe Mode and when it will exit.

Advantages of using the Web Interface:

User-Friendly: Provides a graphical interface that is easier to navigate for users who are not comfortable with the command line.

Remote Access: Allows monitoring and basic management of HDFS from any machine with a web browser and network access to the NameNode.

Quick Overview: Provides a quick visual summary of the cluster's health and status.

Limitations:

Limited Functionality: The web interface offers a subset of the operations available through the command-line interface. For advanced tasks, the shell is still necessary.

Security Considerations: Access to the web interface should be properly secured to prevent unauthorized access and modifications to HDFS.

Web Interface of HDFS

HDFS Interfaces

Outline

Hadoop Cluster vs. Pseudo-Distributed Hadoop
Shell Interface to HDFS
Web Interface to HDFS
Python Interface to HDFS
Internals of HDFS

Snakebite: Programmatic HDFS Access from Python

Snakebite is a Python library developed by Spotify that enables Python applications to interact with Hadoop's HDFS. It offers a more Pythonic way to perform HDFS operations compared to executing shell commands.

Key Features and Details:

- Direct NameNode Communication: Snakebite communicates directly with the HDFS NameNode using protocol buffer messages. This offers several advantages:

Efficiency: Bypassing the overhead of shell command execution and parsing, Control: Fine-grained control over HDFS operations, Integration: Seamless integration within Python applications.

- Protocol Buffers: Protocol Buffers are a language-neutral, platform-neutral, extensible mechanism for serializing structured data, developed by Google. HDFS uses them for communication between clients (like Snakebite) and the NameNode.

- Pythonic Interface: Snakebite provides a set of Python classes and methods that map to HDFS operations, making it easier for Python developers to work with HDFS.

- Command-Line Interface (CLI): As highlighted, Snakebite includes its own CLI (snakebite) that mirrors many of the standard HDFS shell commands. This can be useful for quick interactions and scripting directly from the terminal using Snakebite.

Extending the Sample Application and Concepts:

1. Understanding the Output:

The output you provided for listing /user shows a Python dictionary representing the metadata of a directory named bigdata within /user. Let's break down the keys:

'file_type': 'd': Indicates that this is a directory ('f' would signify a file).

'permission': 493: The octal representation of the file permissions (equivalent to rwxr-xr-x).

'path': '/user/bigdata': The full path of the directory in HDFS.

'length': 0: For a directory, the length is typically 0. For a file, it would be the file size in bytes.

'owner': 'bigdata': The user who owns the directory.

'group': 'supergroup': The group associated with the directory.

'block_replication': 0: For a directory, the block replication is usually 0. For a file, it indicates the number of copies of each block.

'modification_time': 1711847570566: The last modification time of the directory, represented as milliseconds since the epoch.

'access_time': 0: The last access time of the directory (may not be tracked in all HDFS configurations).

'blocksize': 0: For a directory, the block size is typically 0. For a file, it's the size of each HDFS block.

Python interface to HDFS

Snakebite is a Python package, created by Spotify, that allows for HDFS to be accessed programmatically from Python applications
Protocol buffer messages are used to communicate directly with the NameNode
Snakebite package also includes its own command-line interface to HDFS

A sample application of Snakebite that lists the contents of /user folder in HDFS

 
#!/usr/bin/env python3
	      
from snakebite.client import Client
	      
client = Client('localhost', 9000)
for x in client.ls(['/user']):
      print(x)

 
{'file_type': 'd', 'permission': 493, 'path': '/user/bigdata', 'length': 0, 'owner': 'bigdata',
'group': 'supergroup', 'block_replication': 0, 'modification_time': 1711847570566, 'access_time': 0,
'blocksize': 0}

This Python script provides a basic example of using the Snakebite library to retrieve and display the metadata (parameters) of a specific file, NOTICE.txt, located in the /user directory of your Hadoop Distributed File System (HDFS). Let's break down the code step by step:

1. #!/usr/bin/env python3:

This is a shebang line. It's typically the first line of a script and tells the operating system that this script should be executed using the python3 interpreter.

2. from snakebite.client import Client:

This line imports the Client class from the snakebite.client module. The Client class is the primary interface provided by Snakebite to interact with the HDFS NameNode. It allows you to perform various HDFS operations programmatically.

3. client = Client('localhost', 9000):

This line creates an instance of the Client class.

'localhost' specifies the hostname or IP address of the HDFS NameNode. In this case, it assumes the NameNode is running on the local machine.

9000 specifies the port number on which the NameNode's client RPC server is listening. This is the port Snakebite uses to communicate with the NameNode. Note: The default port for the NameNode's client RPC in newer Hadoop versions (2.x and later) is often 8020. If you are using a more recent Hadoop setup, you might need to change 9000 to 8020.

4. for x in client.ls(['/user/NOTICE.txt'])::

This line initiates a loop that iterates over the results of the client.ls() method.

client.ls(['/user/NOTICE.txt']) calls the ls() method of the Client object. The ls() method is analogous to the hdfs dfs -ls or hadoop fs -ls shell command. It takes a list of HDFS paths as arguments and returns an iterator that yields information about the files and directories at those paths.

In this specific case, we are providing a list containing a single path: '/user/NOTICE.txt'. This means Snakebite will request information about the file (or directory, if it existed) at this exact path.

The loop iterates through each item returned by the ls() method. Since we are listing a specific file, the iterator will typically yield only one item (a dictionary containing the file's metadata) if the file exists. This item is assigned to the variable x in each iteration.

5. print(x):

Inside the loop, this line prints the value of x. As seen in the output below the code, x is a Python dictionary containing various parameters (metadata) of the /user/NOTICE.txt file.

Output Explanation:

The output you provided is a Python dictionary representing the metadata of the /user/NOTICE.txt file as returned by Snakebite's ls() method:

{'file_type': 'f', 'permission': 420, 'path': '/user/NOTICE.txt', 'length': 1541, 'owner': 'bigdata','group': 'supergroup', 'block_replication': 1, 'modification_time': 1711864616929,'access_time': 1711864616715, 'blocksize': 134217728}

In summary, this script uses the Snakebite library to connect to the HDFS NameNode, requests information about the /user/NOTICE.txt file, and then prints a dictionary containing the file's metadata, such as its type, permissions, size, owner, group, replication factor, modification and access times, and block size. This demonstrates a fundamental way to programmatically access file information in HDFS using Python and Snakebite.

Python interface to HDFS

A sample application of Snakebite that lists the parameters of a file NOTICE.txt located at /user in HDFS

 
#!/usr/bin/env python3

from snakebite.client import Client

client = Client('localhost', 9000)
for x in client.ls(['/user/NOTICE.txt']):
      print(x)

 
{'file_type': 'f', 'permission': 420, 'path': '/user/NOTICE.txt', 'length': 1541, 'owner': 'bigdata',
'group': 'supergroup', 'block_replication': 1, 'modification_time': 1711864616929,
'access_time': 1711864616715, 'blocksize': 134217728}

Script 1: Listing a Specific File's Path and Type

#!/usr/bin/env python3

from snakebite.client import Client

client = Client('localhost', 9000)

for x in client.ls(['/user/NOTICE.txt']):

print(x['path'], x['file_type'])

# Output: /user/NOTICE.txt f

Explanation:

client.ls(['/user/NOTICE.txt']):

The ls() method is called with a list containing a single HDFS path: '/user/NOTICE.txt'.

This tells Snakebite to fetch the metadata specifically for the NOTICE.txt file within the /user directory.

The ls() method returns an iterator. If the file exists, this iterator will yield a single dictionary containing all the metadata for NOTICE.txt (like permissions, owner, size, etc., as seen in the previous explanation).

for x in ...: The loop iterates over the items yielded by client.ls(). In this case, x will be the dictionary containing the metadata for NOTICE.txt.

print(x['path'], x['file_type']):

Instead of printing the entire dictionary x, this line specifically accesses two keys from the dictionary:

x['path']: Retrieves the full HDFS path of the entry.

x['file_type']: Retrieves the type of the entry. 'f' stands for "file", and 'd' stands for "directory".

The print() function then outputs these two values, separated by a space.

Output:

/user/NOTICE.txt f: This output confirms that the entry at /user/NOTICE.txt is indeed a file (f).

Script 2: Listing Multiple Specific Paths' Names and Types

Python

#!/usr/bin/env python3

from snakebite.client import Client

client = Client('localhost', 9000)

for x in client.ls(['/user/NOTICE.txt', '/user']): # Assuming '/user' was the intended second path

print(x['path'], x['file_type'])

client.ls(['/user/NOTICE.txt', '/user']):

This is the key difference. The ls() method is now called with a list containing two HDFS paths: '/user/NOTICE.txt' (a file) and '/user' (a directory).

Snakebite will fetch metadata for both of these specified paths.

The iterator returned by ls() will yield a dictionary for each of these paths. The order of results might not always strictly match the input list order, but both will be present.

for x in ...: The loop processes each metadata dictionary as it's yielded.

print(x['path'], x['file_type']): Similar to the first script, this extracts and prints the path and file type for each item.

Output:

/user/NOTICE.txt f: This line shows that /user/NOTICE.txt is a file.

/user d: This line shows that /user is a directory (d).

Python interface to HDFS

The sample applications of Snakebite that list the names and type of folders/files located at the end of a path

 
#!/usr/bin/env python3

from snakebite.client import Client

client = Client('localhost', 9000)
for x in client.ls(['/user/NOTICE.txt']):
      print(x['path'], x['file_type'])

/user/NOTICE.txt f

 
#!/usr/bin/env python3

from snakebite.client import Client

client = Client('localhost', 9000)
for x in client.ls(['/user/NOTICE.txt',/]):
      print(x['path'], x['file_type'])

 
/user/NOTICE.txt f
/user d

This Python script demonstrates how to use the Snakebite library to copy a file from HDFS (Hadoop Distributed File System) to your local file system. This is equivalent to the hdfs dfs -copyToLocal or hadoop fs -get shell commands.

1. #!/usr/bin/env python3:

This is the shebang line, which tells the operating system to execute the script using the python3 interpreter.

2. from snakebite.client import Client:

This line imports the Client class from the snakebite.client module. The Client class is the main interface provided by Snakebite for interacting with the HDFS NameNode.

3. client = Client('localhost', 9000):

This line creates an instance of the Client class. 'localhost' specifies the hostname or IP address of your HDFS NameNode. 9000 specifies the RPC port on which the NameNode is listening for client connections. (As a reminder, for Hadoop 2.x and later, this is often 8020).

4. for x in client.copyToLocal(['/user/NOTICE.txt'],'.')::

This is the core of the script. It calls the copyToLocal() method of the Client object.

copyToLocal() takes two main arguments:

['/user/NOTICE.txt']: This is a list of one or more HDFS source paths. In this case, we are specifying that we want to copy the file named NOTICE.txt located in the /user directory of HDFS.

'.': This is the local destination path. The . (dot) signifies the current working directory where the Python script is being executed. So, the file will be copied to the same directory where you run this script.

The copyToLocal() method returns a generator. You need to iterate over this generator (even if there's only one file being copied) for the operation to execute. Each x in the loop will be a dictionary containing the result of the copy operation for a specific file.

5. print(x):

Inside the loop, this line prints the dictionary x that contains the status and details of the copy operation for each file.

Python interface to HDFS

A sample application of Snakebite that copies a file from HDFS to a local file system

 
#!/usr/bin/env python3

from snakebite.client import Client

client = Client('localhost', 9000)
for x in client.copyToLocal(['/user/NOTICE.txt'],'.'):
      print(x)

 
{'path': '/home/bigdata/NOTICE.txt', 'result': True, 'error': '', 'source_path': '/user/NOTICE.txt'}

  
bigdata@bigdata-VirtualBox:~$ ls

 
Desktop    Downloads  hadoop-3.3.6  NOTICE.txt  Public     snap       UMLet15-1
Documents  ex.py      Music         Pictures    shrink.sh  Templates  Videos

This Python script demonstrates how to use the Snakebite library to read and print the content of a file located in HDFS (/user/NOTICE.txt) to your standard output.

1. #!/usr/bin/env python3:

This is the shebang line, which tells the operating system to execute the script using the python3 interpreter.

2. from snakebite.client import Client:

This line imports the Client class from the snakebite.client module. The Client class is the primary interface provided by Snakebite for interacting with the HDFS NameNode.

client = Client('localhost', 9000):

4. for file in client.cat(['/user/NOTICE.txt'])::

This line initiates the first for loop. It calls the cat() method of the Client object.

client.cat(['/user/NOTICE.txt']) is analogous to the hdfs dfs -cat /user/NOTICE.txt or hadoop fs -cat /user/NOTICE.txt shell command. It reads the content of the specified file(s) in HDFS.

The cat() method returns a generator that yields the content of the file in chunks (blocks). Each file in this outer loop represents a chunk of data read from the HDFS file.

5. for y in file::

This is a nested for loop. In each iteration of the outer loop, file (a chunk of data) is itself an iterable (specifically, a bytes object in Python).

This inner loop iterates through each individual byte within the current chunk of data (file).

6. print(y):

Inside the inner loop, this line prints the value of y. Since y represents a single byte from the file's content, it will be printed as an integer representing the ASCII value of that byte. This is likely not the desired output for human-readable text.

Python interface to HDFS

A sample application of Snakebite that lists a file located in HDFS

 
#!/usr/bin/env python3

from snakebite.client import Client

client = Client('localhost', 9000)
for file in client.cat(['/user/NOTICE.txt']):
      for y in file:
	  print(y)

 
b"Apache Hadoop\nCopyright 2006 and onwards The Apache Software Foundation.\n\nThis product includes
software developed at\nThe Apache Software Foundation (https://url.au.m.mimecastprotect.com/s/2HlmC5QPyyHGPEZzhyiQhk89x1?domain=apache.org).\n\nExport Control Notice
\n---------------------\n\nThis distribution ... "

This Python script demonstrates how to use the Snakebite library to:

List files: Specifically target files within a given HDFS path (/user/bigdata/NOTICE.txt).

Read and print the content of those files: Use the client.text() method to efficiently read the content of the identified files.

1. #!/usr/bin/env python3:

This is the shebang line, indicating that the script should be executed using the python3 interpreter.

2. from snakebite.client import Client:

This line imports the Client class from the snakebite.client module, which is the primary interface for interacting with HDFS using Snakebite.

3. client = Client('localhost', 9000):

This line creates an instance of the Client class, establishing a connection to the HDFS NameNode running on localhost at port 9000. (Remember that the default port for newer Hadoop versions might be 8020).

4. files = []:

This line initializes an empty Python list named files. This list will be used to store the HDFS paths of the files we want to read.

5. for x in client.ls(['/user/bigdata/NOTICE.txt'])::

This loop uses the client.ls() method to list the contents of the specified HDFS path: '/user/bigdata/NOTICE.txt'.

Even though the path points to a specific file, client.ls() still returns an iterator that yields metadata for the entry at that path.

For each item x yielded by the iterator (in this case, likely just one item if the file exists):

if x['file_type'] == 'f':: This condition checks if the file_type in the metadata dictionary x is 'f', indicating that it's a file.

files.append(x['path']): If it's indeed a file, its full HDFS path (x['path']) is appended to the files list.

6. for cat in client.text(files)::

This is the second loop, which uses the client.text() method.

client.text(files) takes a list of HDFS file paths as input (in this case, the files list we populated in the previous loop).

The text() method is designed to efficiently read the content of text files in HDFS. It returns an iterator that yields the content of each file in the provided list, typically line by line (decoded as UTF-8 by default).

For each cat yielded by the client.text(files) iterator, which represents a line of text from one of the files:

print(cat): This line prints the current line of text to the standard output.

Python interface to HDFS

A sample application of Snakebite that lists several files located in HDFS

#!/usr/bin/env python3

from snakebite.client import Client

client = Client('localhost', 9000)

files = []
for x in client.ls(['/user/bigdata/NOTICE.txt']):
        if x['file_type'] == 'f':
                files.append(x['path'])

for cat in client.text(files):
      print(cat)

 
b'\n                      Apache License\n                Version 2.0, January 2004\n
https://url.au.m.mimecastprotect.com/s/Bj0mC6XQ88iK6zrgT5sph5H-n2?domain=apache.org\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n
1. Definitions.\n\n      "License" shall mean ... "

 
b"Apache Hadoop\nCopyright 2006 and onwards The Apache Software Foundation.\n\n
This product includes software developed at\nThe Apache Software Foundation (https://url.au.m.mimecastprotect.com/s/2HlmC5QPyyHGPEZzhyiQhk89x1?domain=apache.org/).
\n\nExport Control Notice\n----------------\n\nThis distribution includes cryptographic software. ..."

Python interface to HDFS

A sample application that reads and filters a file located in HDFS
The application uses Hadoop Streaming technique to process the files located in HDFS
Assume that a file numbers.txt contains a sequence of numbers
The file is uploaded to HDFS

Explanation of the Python Script:

#!/usr/bin/env python3: This is the shebang line, indicating that the script should be executed using the python3 interpreter.

import sys: This line imports the sys module, which provides access to system-specific parameters and functions, including standard input (sys.stdin).

for number in sys.stdin:: This loop reads data from the standard input, which, in the context of Hadoop Streaming's mapper, will be lines from the input file (/user/numbers.txt in HDFS), one line at a time.

try...except ValueError:: This block handles potential errors that might occur if a line in numbers.txt cannot be converted to an integer.

num = int(number.strip()):

number.strip(): This removes any leading or trailing whitespace (like spaces or newline characters) from the input line.

int(...): This attempts to convert the stripped line into an integer and stores it in the num variable.

if num > 50:: This is the filtering condition. It checks if the integer value of num is greater than 50.

print(num): If the condition in the if statement is true (i.e., the number is greater than 50), this line prints the number to the standard output. In Hadoop Streaming's mapper, anything printed to standard output in the format "key\tvalue" (or just "value" if no explicit key is needed) becomes the output of the mapper. In this case, we are just printing the filtered number, so Hadoop Streaming will treat each printed number as a value with a default key.

Hadoop Streaming Command:

$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-3.3.6.jar \

-D mapred.reduce.tasks=0 \

-input /user/numbers.txt \

-output /user/output \

-mapper ./filter.py

Python interface to HDFS

We would like to filter out all numbers greater than 50

 
#!/usr/bin/env python3

import sys

for number in sys.stdin:
      if int(number) > 50:
                  print(int(number))

The file in HDFS is processed with Hadoop Streaming feature

 
$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-3.3.6.jar \
 -D mapred.reduce.tasks=0 \
 -input /user/numbers.txt \
 -output /user/output     \
 -mapper ./filter.py

Python interface to HDFS

The results are retrieved from HDFS

 
hdfs dfs -ls /user/output

 
Found 2 items
-rw-r--r--   1 bigdata supergroup          0 2024-04-01 20:09 /user/output/_SUCCESS
-rw-r--r--   1 bigdata supergroup          8 2024-04-01 20:09 /user/output/part-00000

 
bigdata@bigdata-VirtualBox:~/hadoop-3.3.6$ hdfs dfs -cat /user/output/part*

 
56    
78

HDFS Interfaces

Outline

Hadoop Cluster vs. Pseudo-Distributed Hadoop
Shell Interface to HDFS
Web Interface to HDFS
Python Interface to HDFS
Internals of HDFS

Internals of HDFS

What happens "inside" when we read data into HDFS ?

Internals of HDFS

Read data from HDFS
Step 1: The client opens the file it wishes to read by calling open() on the FileSystem object, which for HDFS is an instance of DistributedFileSystem
Step 2: DistributedFileSystem calls the namenode, using remote procedure calls (RPCs), to determine the locations of the first few blocks in the file
Step 3: The DistributedFileSystem returns an FSDataInputStream to the client and the client calls read() on the stream

Internals of HDFS

Step 4: FSDataInputStream connects to the first datanode for the first block in the file, and then data is streamed from the datanode back to the client, by calling read() repeatedly on the stream
Step 5: When the end of the block is reached, FSDataInputStream will close the connection to the datanode, then find the best (possibly the same) datanode for the next block
Step 6: When the client has finished reading, it calls close() on the FSDataInputStream

Internals of HDFS

Write data into HDFS

Internals of HDFS

Step 1: The client creates the file by calling create() on DistributedFileSystem
Step 2: DistributedFileSystem makes an RPC call to the namenode to create a new file in the file system namespace and returns an FSDataOutputStream for the client to start writing data to
Step 3: The client writes data into the FSDataOutputStream
Step 4: Data wrapped by the FSDataOutputStream is split into packages, which are flushed into a queue; data packages are sent to the blocks in a datanode and forwarded to other (usually two) datanodes

Internals of HDFS

Step 5: If FSDataStream receives an ack signal from the datanode the data packages are removed from the queue
Step 6: When the client has finished writing data, it calls close() on the stream
Step 7: The client signals the namenode that the writing is completed

References

Vohra D., Practical Hadoop ecosystem: a definitive guide to Hadoop-related frameworks and tools, Apress, 2016 (Available through UOW library)
Aven J., Hadoop in 24 Hours, SAMS Teach Yourself, SAMS 2017
Hadoop with Python
Snakebite Documentation, Client Library