Benchmarking of Database Systems
Outline
Database benchmark ? What is it ?
- A database benchmark is a sample database and a group of database applications able to run on several different database systems in order to measure performance of each system and relate it to price of each system (price/ performance ratio)
- Performance metric is typically a throughput metric (work/second, e.g number of transactions executed per second, number of queries executed per hour)
- Price metric is typically five years cost of ownership
Benchmarking of Database Systems
Outline
Domain specific benchmarks
- A domain specific benchmark specifies a synthetic workload characterizing typical applications in the problem domain
- For example, numeric computations benchmark (number of floating point operations per second), transaction processing benchmark (number of transactions processed per second), query processing benchmark (number of queries processed per hour)
Benchmarking of Database Systems
Outline
Properties of good benchmark
- Relevance
- A benchmark should be meaningful with a target domain
- Clarity
- A benchmark should clearly determine the measured characteristics
- Scalability
- It should be possible to scale up a benchmark
- Linearity
- Linear scaling up a benchmark linearly increases its complexity
- Orthogonality
- Benchmark parameters should be independent on each other
- Monotonicity
- Scaling benchmark up should always make it more demanding
Properties of good benchmark
- Coverage
- A benchmark should not oversimplify a typical environment
- Acceptance
- A benchmark should be accepted by the vendors and users
Benchmarking of Database Systems
Outline
TPC-H and TPC-R benchmarks (1999)
- TPC-H and TPC-R replaced older TPC-D benchmark
- TPC-R is used to measure reporting load
- TPC-H is used to measure ad hoc-querying load
- TPC-H and TPC-R operate on the same database as TPC-D
- TPC-H and TPC-R extend the number of queries, updates and streams
TPC-H and TPC-R benchmarks (1999)
- TPC-H and TPC-R processing rules:
- Power test
- Queries and updates are submitted in a single stream of query sets (no concurrency)
- Each query set consist of 17 queries and it is followed by a sequence of updates
- Throughput test
- Queries are submitted in a number of concurrent streams versus one update stream
- Load test
- Load test measures the time to go from an empty database to reproducible query runs
- It starts from empty database then it loads synthetic data, builds indexes, gathers statistics, and runs queries
TPC-H and TPC-R benchmarks (1999)
- Performance metrics
- Power metric
TPC-H and TPC-R benchmarks (1999)
- Performance metrics
- Throughput metric
TPC-H and TPC-R benchmarks (1999)
- Performance metrics
- Composite query per hour rating
Benchmarking of Database Systems
Outline
TPC-W benchmark (2000)
- TPC-W is designed to measure performance of both hardware and software in e-commerce environments
- TPC-W is based on a business model that employs a shopping scenario typical of an online bookstore
- Metrics reported include Web interactions per second
- Configuration of TPC-W includes both database server and Web server
Benchmarking of Database Systems
Outline
TPC-DS benchmark (2012)
- TPC-DS is a Decision Support Benchmark
- TPC-DS benchmark models several generally applicable aspects of a decision support system, including queries and data maintenance
- The benchmark measures query response time in single user mode, query throughput in multi user mode and data maintenance performance for a given hardware, operating system, and data processing system configuration under a controlled, complex, multi-user decision support workload
- TPC-DS Version 2 enables emerging technologies, such as Big Data systems, to execute the benchmark
References
- Transaction Performance Council
- B.Scalzo Database Benchmarking and Stress Testing, Apress, 2018