A schema in Pig is essentially the blueprint or structure of your data. It defines the names and data types of the fields within your tuples (records). Explicitly defining a schema is the best way to ensure Pig correctly interprets your data.
1. Explicit Schema Definition (With Data Types)
This is the preferred method as it provides Pig with the most complete information, allowing it to perform efficient type-checking and optimization.
Example
dividends = load 'NYSE_dividends' as
(exchange:chararray, symbol:chararray, date:chararray, dividend:float);
Key Points
You assign a field name (e.g., exchange) and a data type (e.g., :chararray) for every expected field.
The data is treated as having four fields with the specified types.
2. Explicit Schema Definition (Without Data Types)
You can still define a schema by only providing the field names.
Example
dividends = load 'NYSE_dividends' as (exchange, symbol, date, dividend);
Key Point
In this case, Pig assumes the data type for all fields is bytearray. A bytearray is a raw, untyped collection of bytes. While flexible, you'll need to explicitly cast or convert these fields later in your script to perform most calculations or comparisons.
Impact of Schema on Data Loading
When you define a schema, you are setting a strict expectation for the structure of your input data. Pig enforces this structure as the data is loaded:
Too Many Fields in the Data:
If the input record has more fields than specified in the schema, Pig will truncate the extra fields. Only the fields matching the schema definition are kept.
Too Few Fields in the Data:
If the input record has fewer fields than specified, Pig will pad the end of the record with NULLs for the missing fields.
For example, if the schema expects four fields but the data only has three, the fourth field will be set to NULL (unknown value).
Defining a schema is critical for data integrity and predictable processing in Pig Latin scripts.