15 HBaseDataModel

HBase Data Model

Outline

Background
Logical view of data
Design fundamentals
Physical implementation

HBase is the open-source implementation of the concepts outlined in the seminal Google BigTable paper. BigTable is Google's proprietary highly available, high-performance distributed storage system.

As a distributed database, its data is spread across many commodity servers (a cluster), which allows it to handle petabytes of data and scale out horizontally.

HBase provides a BigTable view of data stored in HDFS:

HDFS (Hadoop Distributed File System) is the underlying storage layer for HBase. HBase doesn't store data directly on individual disks; it uses HDFS for persistent storage and fault tolerance.

HBase acts as a layer on top of HDFS, providing the structured, indexed, random-access capabilities that HDFS lacks (HDFS is primarily for large, sequential file access).

HBase is also called as Hadoop DataBase:

It's a common nickname because of its tight integration with the Apache Hadoop ecosystem (specifically HDFS and MapReduce), serving as the primary NoSQL database for large-scale data processing within that stack.

Data Model: The Sparse, Multidimensional Map

HBase still provides a tabular view of data however it is also very different from the traditional relational data model:

It's tabular in the sense that data is organized into rows and columns, like a spreadsheet.

However, it's a NoSQL (Not only SQL) database, fundamentally different from relational databases (like MySQL or PostgreSQL). It doesn't have fixed schemas, relationships (joins), or transactions in the traditional SQL sense, making it much more flexible and scalable for massive datasets.

HBase data model is a sparse, distributed, persistent multidimensional sorted map: This is the most formal and accurate definition:

Sparse: Tables can have a huge number of potential columns, but any given row only needs to use a small subset of them. Unused columns don't take up space, making it efficient for rows with highly variable data structures.

Distributed: The map (data) is partitioned and spread across the servers in the cluster (called RegionServers).

Persistent: Data is stored durably on HDFS.

Multidimensional Sorted Map: This defines how the data is uniquely addressed and stored:

The data structure is a map (key-value store) where the values are stored based on a composite key.

It is indexed by a row key, column key, and timestamp: These three components form the coordinates that uniquely identify any single data point (cell value) in HBase:

Row Key: A byte array that uniquely identifies a row. All rows are lexicographically sorted by the row key, which is critical for efficient range-based scanning and data locality.

Column Key (Column Family:Qualifier): A combination of a mandatory Column Family (defined at table creation, groups related columns, and determines storage properties) and a flexible Column Qualifier (user-defined name for the actual "column").

Timestamp: The time (version) at which the cell value was written. HBase keeps multiple versions of data, indexed by timestamp, allowing for time-based retrieval.

In essence, the unique address for any data value in HBase is: (Table, RowKey, ColumnFamily, ColumnQualifier, Timestamp) → CellValue.

Background

Hbase is open source distributed database based on a data model of Google's BigTable
HBase provides a BigTable view of data stored in HDFS
HBase is also called as Hadoop DataBase
HBase still provides a tabular view of data however it is also very different from the traditional relational data model
HBase data model is a sparse, distributed, persistent multidimensional sorted map
It is indexed by a row key, column key, and timestamp

HBase Data Model

Data Grouping: Column Families

Data within a row is grouped by a column family:

Column Families are the most critical structural and physical component. They are defined when the table is created and are fixed.

They serve as the physical storage unit. All data belonging to the same column family is stored together on HDFS in a separate set of files called HFiles.

Column families have an important impact on the physical implementation of HBase table:

This is the key difference from relational databases. Different column families can have different storage properties (e.g., compression, block size, number of versions to keep).

Because they are stored separately, accessing data from one column family does not require reading data from another, which greatly optimizes I/O performance.

Every row has the same column families although some of them can be empty:

While every row conceptually includes all defined column families, a row is sparse. If a row doesn't have data for a column family, it simply doesn't take up any physical space for that family, supporting the flexible NoSQL model.

Cell Identification and Data Typing

Data within a column family is addressed via its column qualifier, or simply, column name:

The Column Qualifier (or Column Name) is a user-defined tag (a byte array) that identifies a specific data field within a Column Family. Unlike the Column Family, qualifiers can be defined on the fly without needing a schema update.

Hence, a combination of row key, column family, and column qualifier uniquely identifies a cell:

This combination (along with the optional Timestamp) is the full address for a piece of data: (Row Key, Column Family:Qualifier) → Value.

Values in cells do not have a data type and are always treated as sequences of bytes:

HBase is type-agnostic. It simply stores and retrieves raw byte arrays.

It is the client application's responsibility to serialize (convert) application-level types (like integers, floats, strings, JSON objects) into bytes before writing and deserialize them back after reading. This keeps HBase fast and flexible.

Logical view of data

HBase organizes data into tables
HBase table consists of rows
Each row is uniquely identified by a row key
Data within a row is grouped by a column family
Column families have an important impact on the physical implementation of HBase table
Every row has the same column families although some of them can be empty
Data within a column family is addressed via its column qualifier, or simply, column name
Hence, a combination of row key, column family, and column qualifier uniquely identifies a cell
Values in cells do not have a data type and are always treated as sequences of bytes

Data Versioning in HBase

Values within a cell have multiple versions:

Unlike many traditional databases that only store the current value of a cell, HBase can retain multiple historical values (versions) for the same cell coordinate (Row Key, Column Family, and Column Qualifier).

This built-in versioning allows users to easily retrieve the value of a cell as it existed at a specific point in time, which is useful for audit logging, time-series data, or simply retrieving the previous state of data.

Versions are identified by their version number, which by default is a timestamp when the cell was written:

The timestamp is the default and most common way a version is defined. When a new value is written to a cell, HBase automatically assigns the current server time (in milliseconds) as the version number.

Crucially, when data is read, the versions are retrieved in descending order (most recent first). If a user doesn't specify a timestamp, they automatically get the latest version.

Customization: While the default is the millisecond timestamp, users can optionally specify their own custom version number when writing data, as long as it's a positive 64-bit integer, allowing for more controlled version tracking.

Summary of the Cell Address

Combining all the points previously discussed, the complete coordinate that uniquely identifies a specific piece of data (Value) in HBase is:

Address=(Row Key,Column Family,Column Qualifier,Timestamp)

This structure is what makes HBase a sparse, distributed, persistent multidimensional sorted map.

Logical view of data

ORM

Values within a cell have multiple versions
Versions are identified by their version number, which by default is a timestamp when the cell was written

Timestamp and Version Defaults

If a timestamp is not determined at write time, then the current timestamp is used:

When an application writes data to an HBase cell and does not explicitly provide a timestamp, HBase automatically uses the current time on the server (in milliseconds since the Unix epoch) as the version identifier for that new value.

This ensures every write receives a unique, ordered version, which is the default and most common behavior.

If a timestamp is not determined during a read, the latest one is returned:

When an application reads a cell and does not specify a particular timestamp (version number) it's looking for, HBase, by default, returns only the single most recent version (the one with the highest timestamp).

This is the standard behavior for simple "get" operations, providing the "current" view of the data. You must explicitly request a historical version or request multiple versions during a read operation to see older data.

Version Control

The maximum allowed number of cell value versions is determined for each column family:

The Column Family is the administrative unit for versioning. When you create or alter an HBase table, you set the maximum number of versions (MAX_VERSIONS) that HBase should keep for every cell within that specific column family.

Once this limit is reached, the oldest version is automatically and eventually discarded (deleted) during compaction to conserve storage space.

The default number of cell versions is three:

f no specific MAX_VERSIONS parameter is set when defining a column family, HBase uses a default of three versions. This means it keeps the latest value and the two previous values for every cell in that family.

For applications requiring a full history (like time-series), this is often increased to a very large number, while for simple mutable data, it might be set to one.

Logical view of data

If a timestamp is not determined at write time, then the current timestamp is used

If a timestamp is not determined during a read, the latest one is returned
The maximum allowed number of cell value versions is determined for each column family
The default number of cell versions is three

Nested Structure Breakdown

The structure shows how all the key identifiers—Row Key, Column Family, Column Qualifier, and Timestamp—nest together to locate a specific Cell Value.

Nesting Level HBase Component Example Value Explanation

Level 1 Row Key "Row-0001" This is the unique identifier for the entire row of data. All other data is contained within this row.

Level 2 Column Family (CF) "Home", "Office" Data is physically grouped into these families. Notice that the Home CF contains personal info, while Office CF contains work info.

Level 3 Column Qualifier (CQ) "Name", "Phones", "Phone", "Address" These are the specific data fields within their respective Column Families.

Level 4 Timestamp (Version) "timestamp-1", "timestamp-2", etc. These are the unique version identifiers. The values are stored under these timestamps.

Level 5 Cell Value "James", "2 42 214339", etc. This is the raw sequence of bytes stored at the unique address.

Key Insights from the Example

1. Unique Cell Identification

To retrieve the specific value "James", you must navigate the full address:

Row Key: Row-0001→CF: Home→CQ: Name→Timestamp: timestamp-1

2. Versioning in Action

Look at the Home:Phones cell:

It shows three distinct versions (timestamp-1, timestamp-2, timestamp-3).

Assuming the timestamps are sequential (e.g., timestamp-3 is the latest), a simple read request for Row-0001, Home:Phones (without specifying a timestamp) would return +61 2 4567890.

This illustrates the concept that a cell can hold multiple versions, and the latest one is returned by default.

3. Sparse Nature

The structure highlights the sparse nature of the data:

The Home column family has columns Name and Phones.

The Office column family has columns Phone and Address.

Crucially, Office does not have a Name or Phones column, and Home does not have an Address column.

Because HBase is schemaless regarding qualifiers, rows only store data for the qualifiers they actually use, saving space and providing flexibility.

This hierarchical model allows HBase to efficiently store and retrieve massive amounts of highly dynamic and flexible data.

Logical view of data

A view of HBase table as a nested structure

{"Row-0001":
            {"Home":
                    {"Name":
                            {"timestamp-1":"James"}
                     "Phones":
                              {"timestamp-1":"2 42 214339"
                               "timestamp-2":"2 42 213456"
                               "timestamp-3":"+61 2 4567890"}
                    }
             "Office":
                      {"Phone":
                               {"timestamp-4":"+64 345678"}
                       "Address":
                                 {"timestamp-5":"10 Ellenborough Pl"}
                      }
            }
}

Key Observations

1. Unique Cell Address

Every piece of data is identified by the combination of its identifiers. For example, the value "Harry" is uniquely located at:

(Row-0002, Home, Name, timestamp-6)

2. Sparsity (Flexible Schema)

The structure demonstrates sparsity by showing that Row-0002 has different columns populated under different families:

The Home CF contains Name and Phones.

The Office CF contains Phone and Address.

If this row had another Column Family (e.g., "Social"), and it wasn't present here, it simply wouldn't take up any storage space.

3. Versioning Control (The Office:Address Cell)

The cell under the coordinate Office:Address clearly demonstrates the concept of multiple versions for a single data point:

It has two versions: "timestamp-9":"10 Bong Bong Rd" and "timestamp-10":"23 Victoria Rd".

If we assume timestamp-10 is later than timestamp-9, a standard read of the Office:Address cell (without specifying a timestamp) would return only the latest version: "23 Victoria Rd".

The older version, "10 Bong Bong Rd", is retained for historical lookups, up to the maximum number of versions configured for the Office column family.

Logical view of data

A view of HBase table as a nested structure

{"Row-0002":
            {"Home":
                    {"Name":
                            {"timestamp-6":"Harry"}
                     "Phones":
                              {"timestamp-7":"2 42 214234"}
                    }
             "Office":
                      {"Phone":
                               {"timepstamp-8":"+64 345678"}
                       "Address":
                                 {"timestamp-9":"10 Bong Bong Rd"
                                  "timestamp-10":"23 Victoria Rd"}
                      }
            }
}

This explains how data retrieval in HBase relies on specifying the correct level of granularity through the key used in the read operation. Because HBase is built on a multidimensional sorted map, the "key" you supply determines the scope of the data returned.

Defining the Retrieval Key

The term "key" in an HBase read operation is flexible and acts as the address range or coordinate needed to isolate the desired data. It can range from a very broad row identifier to a very specific cell identifier.

Retrieval Goal Required Key Components HBase Operation Explanation

Entire Row Row Key Get (on the Row Key) If all the cells in a row are of interest, you only need the Row Key. HBase will return all Column Families, all Qualifiers, and, by default, the latest version of every cell within that specific row.

Specific Cells Row Key + Column Family and/or Qualifier Get (on Row Key + Scopes) If only specific cells are of interest, you must include the appropriate Column Families and/or Qualifiers along with the Row Key to narrow the scope.

Specific Version Row Key + CF + CQ + Timestamp Get (with Version Specified) To retrieve a historical value, you must add the specific Timestamp (version number) to the key components.

Granularity of the ReadHBase uses the provided components to target its underlying storage files (HFiles), making reads extremely efficient when you provide a precise address:

1. Broad Read: Row Key Only

When you provide just the Row Key, HBase identifies the RegionServer and Region holding that row. It then sequentially checks all the HFiles associated with every Column Family for that row.

Example: Retrieve everything about user "Row-0001".

Key: Row-0001

Result: You get the name, all phones, office phone, and address data.

2. Narrow Read: Row Key + Scope

When you add Column Families and/or Qualifiers, you significantly narrow the search scope:

Filtering by Column Family (CF): HBase only has to check the HFiles belonging to the specified CFs, ignoring all others. This provides a huge performance gain since CFs are physically separated.

Example: Retrieve only personal information for "Row-0001".

Key: Row-0001 + Column Family: Home

Filtering by Column Qualifier (CQ): You can narrow it further to a specific data field within a CF.

Example: Retrieve only the person's name.

Key: Row-0001 + Column Family: Home + Qualifier: Name

By allowing the user to construct a key with varying levels of detail, HBase supports flexible and highly performant access to its sparse, multidimensional data structure.

Logical view of data

A key can be row key or a combination of a row key, column family, qualifier, and timestamp depending on what supposed to be retrieved

If all the cells in a row are of interest then a key is a row key
If only specific cells are of interest, the appropriate column families and qualifiers are a part of a key

HBase Data Model

Outline

Background
Logical view of data
Design fundamentals
Physical implementation

HBase Table Design Questions

Designing an HBase table is essentially making decisions that impact data distribution, read/write performance, and storage efficiency. Here's why those questions are crucial:

What should be a row key and what should it contain?

The Row Key is the most important design decision. It determines how data is sorted (lexicographically) and distributed across the cluster (sharded into Regions).

A good Row Key design ensures even data distribution (avoiding hotspots) and allows for efficient range scans (retrieving related rows quickly). For example, using reversed timestamps or salting can prevent hotspots when data is naturally sequential.

How many column families should a table have?

Column families are the physical storage unit. Data from different families are stored in separate HFiles.

Keep the number low (ideally 1–3). Too many CFs lead to a proliferation of files and inefficient resource management.

Grouping: Families should group columns that are often accessed together and have the same storage characteristics (e.g., compression, number of versions).

What columns (qualifiers) should be included in each column family?

The qualifiers are flexible and can be added on the fly. The main decision is to ensure that a qualifier belongs to the CF that best suits its access pattern and storage properties.

What information should go into the cells?

HBase is type-agnostic; cells only store raw bytes. The information should be structured (e.g., serialized into JSON, Avro, or protobuf) to minimize storage size and facilitate quick deserialization by the client application.

How many versions should be stored for each cell?

The maximum number of versions (MAX_VERSIONS) is set per Column Family.

This decision is a trade-off between data history retention (higher versions) and storage space/compaction time (lower versions). Set it to 1 if you only need the latest value, or to a high number for time-series data.

Design Fundamentals

When designing Hbase table we have to consider the following questions:
- What should be a row key and what should it contain?
- How many column families should a table have?
- What columns (qualifiers) should be included in each column family?
- What information should go into the cells?
- How many versions should be stored for each cell?
In fact HBase table is a four level hierarchical structure where a table consists of rows, rows consists of column families, column families consist of columns and columns consists of versions
If cells contain the keys then HBase table becomes a network/graph structure

These are the most fundamental and crucial design principles and operational constraints of HBase, defining its capabilities as a NoSQL, Big Data store.

Data Access and Indexing

Indexing is performed only for a row key:

HBase is not a general-purpose relational database. It only provides an index on the Row Key.

This makes retrieval extremely fast when accessing a single row by its key or scanning a range of rows based on key prefixes.

Searching or querying data based on any column value (Qualifier) requires scanning large portions of the table (a full table scan), which is inherently slow. If secondary indexing is required, external tools like Apache Phoenix or ElasticSearch must be used.

HBase tables are stored sorted based on a row key:

All rows are stored in lexicographical order by the Row Key.

This is the basis of its scalability, as contiguous blocks of rows (Regions) can be stored on separate servers, enabling efficient range scans of related data.

Data Type and Atomicity

Everything in HBase tables is stored as untyped sequence of bytes:

This applies to Row Keys, Column Families, Column Qualifiers, and Cell Values. HBase is type-agnostic.

The client application is responsible for serializing (converting) complex data types (integers, strings, JSON objects) into bytes before writing and deserializing them back when reading.

Atomicity is guaranteed only at a row level and there are no multi-row transactions:

Atomicity means that all changes within a single row either succeed completely or fail completely. You will never see a partially updated row.

However, HBase does not support transactions that span multiple rows. This is a common trade-off in NoSQL databases to achieve massive horizontal scalability and high availability (sacrificing strict ACID compliance for a single-server system).

Column Family vs. Qualifier

Column families must be defined at HBase table creation time:

Column Families are the physical organizational unit and are part of the table's fixed schema. They determine how data is stored on HDFS (separate files) and managed (compression, versions).

They cannot be added or changed dynamically without altering the table schema.

Column qualifiers are dynamic and can be defined at write time:

Column Qualifiers are the logical columns within a family and are schemaless. You can define a new qualifier simply by writing data to it.

This allows for the storage of sparse and highly flexible data structures within the strict physical framework of the Column Families.

Column qualifiers are stored as sequences of bytes such that they can represent data:

Since qualifiers are bytes, they can also store variable data. For instance, in time-series data, the qualifier might represent a device ID or a specific metric name, making the table highly dynamic.

Design Fundamentals

Important facts to remember:
- Indexing is performed only for a row key
- Hbase tables are stored sorted based on a row key
- Everything in Hbase tables is stored as untyped sequence of bytes
- Atomicity is guaranteed only at a row level and there are no multi-row transactions
- Column families must be defined at Hbase table creation time
- Column qualifiers are dynamic and can be defined at write time
- Column qualifiers are stored as sequences of bytes such that they can represent data

The JSON-like structure you provided illustrates a common and effective way to implement an Entity Type within an HBase table. The key is using the Row Key to uniquely identify the entity and a single Column Family to hold all its attributes.

HBase Implementation of an Entity Type

In this model, the HBase table is designed to store records for a specific type of real-world object (the "Entity," in this case, a CUSTOMER).

1. Entity Identification (The Row Key)

Row Key: "007"

The Row Key is the unique identifier for the entire entity instance (i.e., a specific customer).

Every piece of information about this entity is stored under this single Row Key, ensuring that a simple Get operation on "007" retrieves the complete customer record atomically and efficiently.

2. Entity Grouping (The Column Family)

Column Family: "CUSTOMER"

A single Column Family (CF) is typically used to group all related attributes of one entity type. Since all these attributes (name, phone, email) are always accessed together when looking up a customer, they should be stored in the same CF.

This ensures that all customer data for this row is stored physically together on disk (in the same HFiles), maximizing read performance.

3. Entity Attributes (The Column Qualifiers)

Column Qualifiers (CQs): "first-name", "last-name", "phone", "email"The Column Qualifiers represent the specific attributes (fields or properties) of the CUSTOMER entity.

They are flexible and define the structure of the customer's data within the immutable structure of the Column Family.

4. Versioning (The Timestamps and Values)

Timestamps: "timestamp-1", "timestamp-2"

These identify the version of the attribute value. Notice that some attributes, like first-name, phone, and email, were likely written at the same time (timestamp-1), while the last-name was potentially updated later (timestamp-2).

Cell Values: "James", "Bond", "007-007", "jb@mi6.com"

These are the actual, untyped byte sequences representing the data for each attribute.

Design Fundamentals

Implementation of Entity type

ORM

{"007":
       {"CUSTOMER":
                   {"first-name": {"timestamp-1":"James"},
                    "last-name": {"timestamp-2":"Bond"},
                    "phone": {"timestamp-1":"007-007"},
                    "email": {"timestamp-1":"jb@mi6.com"}
                   }
       }
}

The provided structure illustrates how a one-to-one (1:1) relationship can be implemented in HBase by leveraging a single Row Key to store two distinct, but related, entities within different Column Families.

In this example, the 1:1 relationship is between a DEPARTMENT and its MANAGER: every department has exactly one manager, and every manager manages exactly one department. HBase 1:1 Relationship Implementation

The core principle here is co-location: storing the related entities together under one key for atomic, high-speed retrieval.

1. Co-location via Row Key

Row Key: "Sales"

The row key represents the primary entity in the relationship (the Department).

By using the department name ("Sales") as the row key, all data pertaining to both the department and its manager are stored within this single row. This ensures that when you retrieve the Sales department, you atomically retrieve all its related data, including the manager's details.

2. Entity Separation via Column Families

The two entities involved in the 1:1 relationship are separated using different Column Families (CFs):

CF 1: "DEPARTMENT"

This family stores the attributes belonging strictly to the Department entity (e.g., dname, budget).

CF 2: "MANAGER"

This family stores the attributes belonging strictly to the Manager entity (e.g., enumber, first-name, last-name).

3. Attribute Storage (Qualifiers)

Qualifiers (e.g., budget, first-name) act as the specific fields for each entity.

4. Versioning

Timestamps (e.g., timestamp-1, timestamp-2) are used to version the attributes of each separate entity.

Design Fundamentals

Implementation of one-to-one relationship

ORM

{"Sales":
         {"DEPARTMENT":
                       {"dname": {"timestamp-1":"Sales"},
                       {"budget": {"timestamp-1":"1000"}
                       }
         }
         {"MANAGER":
                    {"enumber": {"timestamp-2":"007"},
                     "first-name": {"timestamp-3":"James"},
                     "last-name": {"timestamp-4":"Bond"}
                    }
         }
}

Design Fundamentals

Implementation of one-to-many relationship

ORM

{"007":
       {"EMPLOYEE":
                   {"enumber": {"timestamp-1":"007"},
                    "first-name": {"timestamp-2":"James"},
                    "last-name": {"timestamp-3":"Bond"},
                    "department":{"timestamp-4":"Sales"}
                   }
       }
}

Implementation of One-to-Many

The core idea is to use the Column Qualifiers (CQs) to store the identifiers of the "many" entities, turning the entire Column Family into a collection or list.

1. The "One" Side (Department)

Row Key: "Sales"

The single Row Key identifies the "one" entity: the Department.

This is the natural key for querying: "Get me the Sales Department and all its employees."

Column Family 1: "DEPARTMENT"

This CF stores the primary attributes of the department (e.g., dname, budget).

2. The "Many" Side (Employees)

Column Family 2: "EMPLOYEE"

This is the collection point for all the employees belonging to the "Sales" department.

Column Qualifiers (CQs): "007", "008", "009", etc.

Crucially, the Column Qualifiers are used as the unique identifiers (Employee IDs) for the "many" entities.

Because CQs are dynamic and virtually limitless, this family can easily scale to hold thousands of employees.

Cell Values: "James Bond", "Harry Potter", etc.

The cell values store denormalized, essential attributes of the employee (e.g., their full name).

Design Fundamentals

Another implementation of one-to-many relationship

ORM

{"Sales":
       {"DEPARTMENT":
                     {"dname": {"timestamp-1":"Sales"},
                     {"budget": {"timestamp-1":"1234567"}
                     }
       }
       {"EMPLOYEE":
                   {"007": {"timestamp-2":"James Bond"},
                    "008": {"timestamp-3":"Harry Potter"},
                    "009": {"timestamp-4":"Robin Hood" }
                    ...
                   }
       }
}

This third structure is a variation of the denormalized one-to-many (1:M) relationship, but it leverages versioning and a single Column Qualifier (CQ) to manage the list of "many" entities. While it implements the relationship, it introduces some significant functional trade-offs compared to the previous example.

1:M Implementation via Versioning

In this implementation, the relationship is established by focusing on the Department (the "One" side) and storing the list of Employees (the "Many" side) within a single cell:

1. The "One" Side (Department)

Row Key: "Sales"

This remains the identifier for the primary entity (the Department).

Column Family 1: "DEPARTMENT"

Stores core department attributes (e.g., dname, budget).

2. The "Many" Side (Employees)

Column Family 2: "HAS-EMPLOYEES"

The CF dedicated to storing the list of employees.

Column Qualifier (CQ): "employees"

Unlike the previous example where the Employee ID was the CQ, here there is only one fixed CQ named "employees".

Timestamps (Versions): "timestamp-2", "timestamp-3", "timestamp-4", etc.

The list of employees is modeled using multiple versions of the same cell. Each new employee is written as a new version of the "employees" cell.

Cell Value: "007 James Bond", "008 Harry Potter", etc.

The value contains the denormalized employee details.

Design Fundamentals

Yet another implementation of one-to-many relationship

ORM

{"Sales":
         {"DEPARTMENT":
                       {"dname": {"timestamp-1":"Sales"},
                       {"budget": {"timestamp-1":"1000"}
         }
         {"HAS-EMPLOYEES":
                          {"employees": {"timestamp-2":"007 James Bond"},
                                        {"timestamp-3":"008 Harry Potter"},
                                        {"timestamp-4":"009 Robin Hood"}
                                        ...
                          }
         }
}

Design Fundamentals

Implementation of many-to-many relationship

ORM

{"participation-1":
                   {"EMPLOYEE":
                               {"enumber": {"timestamp-1":"007"},
                                "first-name": {"timestamp-2":"James"},
                                "last-name": {"timestamp-3":"Bond"},
                                "pnumber": {"timestamp-4":"project-1"},
                                           {"timestamp-5":"project-2"}
                                            ...
                               }
                  }  
}

M:M Implementation via Project-Centric Versioning

In this design, a single Project is treated as the primary entity, and the multiple participating Employees are stored as different versions of a specific attribute.

1. The Primary Entity (Project)

Row Key: "participation-1"

As with the previous example, this generic key suggests a single transaction or relationship instance. A more robust design would use the Project ID as the Row Key (e.g., "project-A").

Column Family: "PROJECT"

This CF holds all the attributes related to the project itself.

Core Attributes: Attributes like "pnumber" and "budget" are stored in their own specific cells.

2. The Related Entities (Employees)

Column Qualifier (CQ): "employee"

A single, fixed qualifier is used to represent the list of employees on the project.

Timestamps (Versions): "timestamp-3", "timestamp-4", "timestamp-5", etc.

Each participating employee is stored as a new version of the same "employee" cell.

Cell Values: "007", "008", "009" (Employee IDs)

The value stored is the identifier of the related employee.

Design Fundamentals

Another implementation of many-to-many relationship

ORM

{"participation-1":
                   {"PROJECT":
                              {"pnumber": {"timestamp-1":"project-1"},
                               "budget:": {"timestamp-2":"12345.25"},
                               "employee": {"timestamp-3":"007"},
                                           {"timestamp-4":"008"},
                                           {"timestamp-5":"009"},
                                            ...
                              }
                   }  
}

M:M Implementation: Intersection Rows

This approach models the relationship itself as an independent entity, often called an Intersection Record or Join Table record in relational database terminology.

1. The Intersection Row Key

Row Key: "participation-1", "participation-2", etc.

The Row Key uniquely identifies a single link between an employee and a project.

In a real-world HBase design, this key would typically be a Composite Key made up of the two related IDs, for example: P#project-1_E#employee-007. This makes the row key descriptive and easy to find via a scan.

2. The Relationship Details

Column Family: "PARTICIPATION"

A single CF is used to group the relationship's attributes.

Column Qualifiers (CQs): "pnumber" and "employee"

These CQs hold the keys of the two entities being linked (the Project ID and the Employee ID).

In essence, each row represents the simple fact: "Employee X is working on Project Y."

Design Fundamentals

Another implementation of many-to-many relationship

ORM

{"participation-1":
                   {"PARTICIPATION":
                                   {"pnumber": {"timestamp-1":"project-1"},
                                    "employee": {"timestamp-2":"employee-007"}
                                   }
                   }
}

{"participation-2":
                   {"PARTICIPATION":
                                   {"pnumber": {"timestamp-1":"project-1"},
                                    "employee": {"timestamp-2":"employee-008"}
                                   }
                   }
}

Design Fundamentals

Note, that it is possible to group in one Hbase table rows of different types

{"employee-007":
                {"EMPLOYEE":
                            {"enumber": {"timestamp-1":"007"},
                             "first-name": {"timestamp-2":"James"},
                             "last-name": {"timestamp-3":"Bond"}
                            }
                }
},
{"project-1":
             {"PROJECT":
                        {"pnumber": {"timestamp-4":"1"},
                         "budget": {"timestamp-5":"12345.25"}
                                   }
                   }
},
{"participation-2":
                   {"PARTICIPATION":
                                   {"pnumber": {"timestamp-1":"project-1"},
                                    "employee": {"timestamp-2":"employee-007"}
                                   }
                   }
}

This HBase structure perfectly illustrates the implementation of a Fact Table with Denormalized Dimensions, a common pattern in data warehousing known as a star schema or denormalization, adapted for HBase.

Fact with Denormalized Dimensions

In data warehousing, a Fact is a core event or transaction (like a sale or a measurement), and Dimensions are the descriptive attributes related to that event (like the buyer, the seller, or the time).

Here's how the HBase design maps to this concept:

1. The Fact Record

Row Key: "1234567"

The Row Key represents the unique identifier for the Fact—the actual transaction or event that occurred. It's the primary key of the transaction record itself.

Fact Measures (Column Family: MEASURE)

This Column Family stores the quantifiable metric of the fact.

Qualifier: "amount" (the fact measure)

Value: "1000000"

This is the central data point, the "what" of the transaction.

2. The Denormalized Dimensions

Instead of storing only a foreign key (like a buyer_id) and requiring a separate lookup to get the buyer's details (which would be slow in HBase), the related dimensional data is denormalized and stored directly in the same row using dedicated Column Families.

Dimension 1: Buyer (Column Family: BUYER)

This CF stores the descriptive details of the person initiating the transaction.

Qualifiers: "phone", "first-name", "last-name".

Advantage: When you retrieve the fact "1234567", you get the buyer's full contact information immediately, avoiding a second query.

Dimension 2: Seller (Column Family: SELLER)

This CF stores the descriptive details of the person completing the transaction.

Qualifiers: "phone", "first-name", "last-name".

Design Fundamentals

Implementation of fact with dimensions

ORM

{"1234567":
                {"MEASURE":
                           {"amount": {"timestamp-1":"1000000"}
                           },
                 "BUYER":
                           {"phone": {"timestamp-1":"242214339"},
                            "first-name": {"timestamp-1":"James"},
                            "last-name": {"timestamp-1":"Bond"}
                           },
                 "SELLER":
                           {"phone": {"timestamp-1":"242215612"},
                            "first-name": {"timestamp-1":"Harry"},
                            "last-name": {"timestamp-1":"potter"}
                           }
                }
}

This structure demonstrates a method for implementing a graph database structure using HBase, where the graph is modeled as a collection of Nodes (entities) and Edges (relationships). This is achieved by leveraging the Row Key and Column Qualifiers to represent the graph components.

HBase Implementation of a Graph Structure

In this design, HBase is used to store the graph's adjacency list, defining the neighbors (nodes) connected to a central node via specific relationships (edges).

1. Nodes (Entities)

Row Key: The Row Key is used to uniquely identify a Node (or Vertex) in the graph.

Examples: "A", "B". These are the entities whose outgoing connections are being listed.

2. Edge Labels (Relationship Types)

Column Family (CF): The Column Family is used to represent the Edge Label or the Type of Relationship that connects the central node to its neighbors.

Examples: "R", "S". The relationships originating from Node "A" are categorized as type "R" and type "S".

3. Edges (The Link)

Column Qualifier (CQ): The Column Qualifier acts as the Edge ID or a unique identifier for a specific instance of the relationship.

Examples: "1", "2". These differentiate individual edges of the same type (CF) originating from the same node (Row Key).

4. Target Nodes (The Neighbors)

Cell Value: The Cell Value stores the Row Key (ID) of the Target Node (the neighbor) reached via the Edge.

Examples: "B", "D", "C". These are the nodes connected to the Row Key node.

Breakdown of the Example

Let's trace the connections from Node A:

Starting Node Relationship Type (CF) Edge ID (CQ) Target Node (Value) Interpretation

"A" "R" "1" "B" Node A has relationship R to Node B.

"A" "R" "2" "D" Node A has relationship R to Node D.

"A" "S" "1" "C" Node A has relationship S to Node C.

Design Fundamentals

Implementation of graph structure

ORM

{"A":
     {"R":
          {"1":{"timestamp-1":"B"},
           "2":{"timestamp-1":"D"}
          },
      "S":
          {"1":{"timestamp-1":"C"}
          }
     }
}

{"B":
     {"R":
          {"1":{"timestamp-1":"D"}
          }
     }
}

Design Fundamentals

Implementation of graph structure

ORM

	      
{"C":
     {"S":
          {"1":{"timestamp-1":"D"}
          }
     }
}

{"D":
}

HBase Data Model

Outline

Background
Logical view of data
Design fundamentals
Physical implementation

HBase Architecture: Built on HDFS

HBase is a database built on top of HDFS:

HDFS (Hadoop Distributed File System) provides the persistence and fault tolerance for HBase. HBase does not manage its own files; it delegates this job to HDFS.

HBase acts as the random-access layer, providing features like indexing, versioning, and atomic row updates that HDFS (a sequential file system) lacks.

HBase tables can scale up to billions of rows and millions of columns:

This is the goal of the architecture. By horizontally scaling, HBase can handle datasets that grow to terabytes or even petabytes in size, far beyond the capacity of a single traditional database server.

Data Partitioning: Regions and RegionServers

HBase tables are split into smaller chunks of data that are distributed across multiple servers:

This partitioning is the mechanism for distributing the data and workload, enabling parallel processing and high scalability.

Chunks of data are called as regions and servers that host regions are called as region servers:

A Region is the fundamental unit of scaling and distribution. It represents a contiguous, sorted range of rows from a single table.

A RegionServer is a Java application that runs on a commodity server. Its job is to host and manage several Regions, handle read and write requests for the data stored in them, and communicate with HDFS.

Region servers are usually collocated with data nodes of HDFS:

This is a critical best practice called data locality. By running the RegionServer (which processes the data) on the same physical machine as the HDFS DataNode (which stores the data), HBase minimizes network traffic. When a RegionServer needs data, it can access it locally without having to fetch it across the network.

Types of Data Splits

The splits of HBase tables are usually horizontal, however, it is also possible to benefit from vertical splits separating column families:

Horizontal Split (Regions): This is the primary split. The table is divided by its Row Key range (e.g., Row Keys A-M go to Region 1, N-Z go to Region 2). This is how data is distributed across different RegionServers.

Vertical Split (Column Families): While not a physical split across servers, data within a single Region is separated vertically by Column Family (CF). All data belonging to one CF is stored in a separate set of files (HFiles) from another CF. This separation is crucial for I/O efficiency, as querying one CF does not require reading the files of other CFs.

Dynamic Management

Region assignments happen when HBase table grows in size or when a region server is malfunctioning or when a new region server is added:

Automatic Splitting: When a Region accumulates too much data (exceeds a configured size limit), it automatically splits into two new Regions. The HBase Master Server then reassigns one of the new Regions to a different RegionServer to balance the load.

Load Balancing and Recovery: If a RegionServer fails, its Regions are quickly reassigned to other active RegionServers. Likewise, adding a new RegionServer causes the Master to reassign existing Regions to the new server to improve cluster utilization. This ensures fault tolerance and elastic scalability.

The combination of HDFS for durability, Regions for horizontal scalability, and Column Families for vertical I/O efficiency is what allows HBase to operate as a high-performance, petabyte-scale database.

Physical implementation

HBase is a database built on top of HDFS
HBase tables can scale up to billions of rows and millions of columns
Because Hbase tables can grow up to terabytes or even petabytes, Hbase tables are split into smaller chunks of data that are distributed across multiple servers
Chunks of data are called as regions and servers that host regions are called as region servers
Region servers are usually collocated with data nodes of HDFS
The splits of Hbase tables are usually horizontal, however, it is also possible to benefit from vertical splits separating column families
Region assignments happen when Hbase table grows in size or when a region server is malfunctioning or when a new region server is added

References

Kerzner M., Maniyam S., HBase Design Patterns, Packt Publishing 2014 (Available from UoW Library)
Jiang, Y. HBase Administration Cookbook, Packt Publishing, 2012 (Available from UoW Library)
Dimiduk N., Khurana A., HBase in Action, Mannig Publishers, 2013
Spaggiari J-M., O'Dell K., Architecting HBase Applications, O'Reilly, 2016