OpenText Analytics Database 26.2.x – Query optimization

Data-Analysis: Initial process for improving query performance

Mon, 01 Jan 0001 00:00:00 +0000

To optimize query performance, begin by performing the following tasks:

Run Database Designer

Database Designer creates a physical schema for your database that provides optimal query performance. The first time you run Database Designer, you should create a comprehensive design that includes relevant sample queries and data on which to base the design. If you develop performance issues later, consider loading additional queries that you run frequently and then rerunning Database Designer to create an incremental design.

When you run Database Designer, choose the option, Update Statistics. The query optimizer uses statistics about the data to create a query plan. Statistics help the optimizer determine:

Multiple eligible projections to answer the query
The best order in which to perform joins
Data distribution algorithms, such as broadcast and resegmentation

If your statistics become out of date, run the function ANALYZE_STATISTICS function to update statistics for a schema, table, or columns. For more information, see Collecting database statistics.

Check query events proactively

The QUERY_EVENTS system table returns information on query planning, optimization, and execution events.

The EVENT_TYPE column provides various event types:

Some event types are informational.
Others you should review for possible corrective action.
Several are most important to address.

Review the query plan

A query plan is a sequence of step-like paths that the query optimizer selects to access or alter information in your database. There are two ways to get information about the query plan:

Run the EXPLAIN command. Each step (path) represents a single operation that the optimizer uses for its execution strategy.
Query the QUERY_PLAN_PROFILES system table. This table provides detailed execution status for currently running queries. Output from the QUERY_PLAN_PROFILES table shows the real-time flow of data and the time and resources consumed for each path in each query plan.

Data-Analysis: Column encoding

Mon, 01 Jan 0001 00:00:00 +0000

You can potentially make queries faster by changing column encoding. Encoding reduces the on-disk size of your data so the amount of I/O required for queries is reduced, resulting in faster execution times. Make sure all columns and projections included in the query use the correct data encoding. To do this, take the following steps:

Run Database Designer to create an incremental design. Database Designer implements the optimum encoding and projection design.
After creating the incremental design, update statistics using the ANALYZE_STATISTICS function.
Run EXPLAIN with one or more of the queries you submitted to the design to make sure it is using the new projections.

Alternatively, run DESIGNER_DESIGN_PROJECTION_ENCODINGS to re-evaluate the current encoding and update it if necessary.

Data-Analysis: Projections for queries with predicates

Mon, 01 Jan 0001 00:00:00 +0000

If your query contains one or more predicates, you can modify the projections to improve the query's performance, as described in the following two examples.

Queries that use date ranges

This example shows how to encode data using RLE and change the projection sort order to improve the performance of a query that retrieves all data within a given date range.

Suppose you have a query that looks like this:

=> SELECT * FROM trades
   WHERE trade_date BETWEEN '2016-11-01' AND '2016-12-01';

To optimize this query, determine whether all of the projections can perform the SELECT operation in a timely manner. Run SELECT COUNT(*) statement for each projection, specifying the date range, and note the response time. For example:

=> SELECT COUNT(*) FROM [ projection_name ]
   WHERE trade_date BETWEEN '2016-11-01' AND '2016-12-01;

If one or more of the queries is slow, check the uniqueness of the trade_date column and determine if it needs to be in the projection’s ORDER BY clause and/or can be encoded using RLE. RLE replaces sequences of the same data values within a column by a pair that represents the value and a count. For best results, order the columns in the projection from lowest cardinality to highest cardinality, and use RLE to encode the data in low-cardinality columns.

Note

For an example of using sorting and RLE, see Choosing sort order: best practices.

If the number of unique columns is unsorted, or if the average number of repeated rows is less than 10, trade_date is too close to being unique and cannot be encoded using RLE. In this case, add a new column to minimize the search scope.

The following example adds a new column trade_year:

Determine if the new column trade_year returns a manageable result set. The following query returns the data grouped by trade_year:
```
=> SELECT DATE_TRUNC('trade_year', trade_date), COUNT(*)
   FROM trades
   GROUP BY DATE_TRUNC('trade_year',trade_date);
```
Assuming that trade_year = 2007 is near 8k, add a column for trade_year to the trades table. The SELECT statement then becomes:
```
=> SELECT * FROM trades
   WHERE trade_year = 2007
   AND trade_date BETWEEN '2016-11-01' AND '2016-12-01';
```
As a result, you have a projection that is sorted on trade_year, which can be encoded using RLE.

Queries for tables with a high-cardinality primary key

This example demonstrates how you can modify the projection to improve the performance of queries that select data from a table with a high-cardinality primary key.

Suppose you have the following query:

=> SELECT FROM [table]
   WHERE pk IN (12345, 12346, 12347,...);

Because the primary key is a high-cardinality column, the database has to search a large amount of data.

To optimize the schema for this query, create a new column named buckets and assign it the value of the primary key divided by 10000. In this example, buckets=(int) pk/10000. Use the buckets column to limit the search scope as follows:

=> SELECT FROM [table]
   WHERE buckets IN (1,...)
   AND pk IN (12345, 12346, 12347,...);

Creating a lower cardinality column and adding it to the query limits the search scope and improves the query performance. In addition, if you create a projection where buckets is first in the sort order, the query may run even faster.

Data-Analysis: GROUP BY queries

Mon, 01 Jan 0001 00:00:00 +0000

The following sections include several examples that show how you can design your projections to optimize the performance of your GROUP BY queries.

Data-Analysis: DISTINCT in a SELECT query list

Mon, 01 Jan 0001 00:00:00 +0000

This section describes how to optimize queries that have the DISTINCT keyword in their SELECT list. The techniques for optimizing DISTINCT queries are similar to the techniques for optimizing GROUP BY queries because when processing queries that use DISTINCT, the optimizer rewrites the query as a GROUP BY query.

The following sections below this page describe specific situations:

Examples in these sections use the following table:

=> CREATE TABLE table1 (
    a INT,
    b INT,
    c INT
);

Data-Analysis: JOIN queries

Mon, 01 Jan 0001 00:00:00 +0000

In general, you can optimize execution of queries that join multiple tables in several ways:

Create projections for the joined tables that are sorted on join predicate columns. This facilitates use of the merge join algorithm, which generally joins tables more efficiently than the hash join algorithm.
Create projections that are identically segmented on the join keys.

Other best practices

The database executes joins more efficiently if the following conditions are true:

Query construction enables the query optimizer to create a plan where the larger table is defined as the outer input.
The columns on each side of the equality predicate are from the same table. For example in the following query, the left and right sides of the equality predicate include only columns from tables T and X, respectively:
```
=> SELECT * FROM T JOIN X ON T.a + T.b = X.x1 - X.x2;
```
Conversely, the following query incurs more work to process, because the right side of the predicate includes columns from both tables T and X:
```
=> SELECT * FROM T JOIN X WHERE T.a = X.x1 + T.b
```

Data-Analysis: ORDER BY queries

Mon, 01 Jan 0001 00:00:00 +0000

You can improve the performance of queries that contain only ORDER BY clauses if the columns in a projection's ORDER BY clause are the same as the columns in the query.

If you define the projection sort order in the CREATE PROJECTION statement, the query optimizer does not have to sort projection data before performing certain ORDER BY queries.

The following table, sortopt, contains the columns a, b, c, and d. Projection sortopt_p specifies to order on columns a, b, and c.

CREATE TABLE sortopt (
    a INT NOT NULL,
    b INT NOT NULL,
    c INT,
    d INT
);
CREATE PROJECTION sortopt_p (
   a_proj,
   b_proj,
   c_proj,
   d_proj )
AS SELECT * FROM sortopt
ORDER BY a,b,c 
UNSEGMENTED ALL NODES;
INSERT INTO sortopt VALUES(5,2,13,84);
INSERT INTO sortopt VALUES(14,22,8,115);
INSERT INTO sortopt VALUES(79,9,401,33);

Based on this sort order, if a SELECT * FROM sortopt query contains one of the following ORDER BY clauses, the query does not have to resort the projection:

ORDER BY a
ORDER BY a, b
ORDER BY a, b, c

For example, the database does not have to resort the projection in the following query because the sort order includes columns specified in the CREATE PROJECTION..ORDER BY a, b, c clause, which mirrors the query's ORDER BY a, b, c clause:

=> SELECT * FROM sortopt ORDER BY a, b, c;
 a  | b  |  c  |  d
----+----+-----+-----
  5 |  2 |  13 |  84
 14 | 22 |   8 | 115
 79 |  9 | 401 |  33
(3 rows)

If you include column d in the query, the database must re-sort the projection data because column d was not defined in the CREATE PROJECTION..ORDER BY clause. Therefore, the ORDER BY d query won't benefit from any sort optimization.

You cannot specify an ASC or DESC clause in the CREATE PROJECTION statement's ORDER BY clause. The database always uses an ascending sort order in physical storage, so if your query specifies descending order for any of its columns, the query still causes the database to re-sort the projection data. For example, the following query requires the database to sort the results:

=> SELECT * FROM sortopt ORDER BY a DESC, b, c;
 a  | b  |  c  |  d
----+----+-----+-----
 79 |  9 | 401 |  33
 14 | 22 |   8 | 115
  5 |  2 |  13 |  84
(3 rows)

Data-Analysis: Analytic functions

Mon, 01 Jan 0001 00:00:00 +0000

The following sections describe how to optimize SQL-99 analytic functions that OpenText™ Analytics Database supports.

Data-Analysis: LIMIT queries

Mon, 01 Jan 0001 00:00:00 +0000

A query can include a LIMIT clause to limit its result set in two ways:

Return a subset of rows from the entire result set.
Set window partitions on the result set and limit the number of rows in each window.

Limiting the query result set

Queries that use the LIMIT clause with ORDER BY return a specific subset of rows from the queried dataset. The database processes these queries efficiently using Top-K optimization, which is a database query ranking process. Top-K optimization avoids sorting (and potentially writing to disk) an entire data set to find a small number of rows. This can significantly improve query performance.

For example, the following query returns the first 20 rows of data in table customer_dimension, as ordered by number_of_employees:

=> SELECT store_region, store_city||', '||store_state location, store_name, number_of_employees
     FROM store.store_dimension ORDER BY number_of_employees DESC LIMIT 20;
 store_region |       location       | store_name | number_of_employees
--------------+----------------------+------------+---------------------
 East         | Nashville, TN        | Store141   |                  50
 East         | Manchester, NH       | Store225   |                  50
 East         | Portsmouth, VA       | Store169   |                  50
 SouthWest    | Fort Collins, CO     | Store116   |                  50
 SouthWest    | Phoenix, AZ          | Store232   |                  50
 South        | Savannah, GA         | Store201   |                  50
 South        | Carrollton, TX       | Store8     |                  50
 West         | Rancho Cucamonga, CA | Store102   |                  50
 MidWest      | Lansing, MI          | Store105   |                  50
 West         | Provo, UT            | Store73    |                  50
 East         | Washington, DC       | Store180   |                  49
 MidWest      | Sioux Falls, SD      | Store45    |                  49
 NorthWest    | Seattle, WA          | Store241   |                  49
 SouthWest    | Las Vegas, NV        | Store104   |                  49
 West         | El Monte, CA         | Store100   |                  49
 SouthWest    | Fort Collins, CO     | Store20    |                  49
 East         | Lowell, MA           | Store57    |                  48
 SouthWest    | Arvada, CO           | Store188   |                  48
 MidWest      | Joliet, IL           | Store82    |                  48
 West         | Berkeley, CA         | Store248   |                  48
(20 rows)

Important

If a LIMIT clause omits ORDER BY, results can be nondeterministic.

Limiting window partitioning results

You can use LIMIT to set window partitioning on query results, and limit the number of rows that are returned in each window:

SELECT ... FROM dataset LIMIT num-rows OVER ( PARTITION BY column-expr-x, ORDER BY column-expr-y [ASC | DESC] )

where querying dataset returns num-rows rows in each column-expr-x partition with the highest or lowest values of column-expr-y.

For example, the following statement queries table store.store_dimension and includes a LIMIT clause that specifies window partitioning. In this case, the database partitions the result set by store_region, where each partition window displays for one region the two stores with the fewest employees:

=> SELECT store_region, store_city||', '||store_state location, store_name, number_of_employees FROM store.store_dimension
     LIMIT 2 OVER (PARTITION BY store_region ORDER BY number_of_employees ASC);
 store_region |      location       | store_name | number_of_employees
--------------+---------------------+------------+---------------------
 West         | Norwalk, CA         | Store43    |                  10
 West         | Lancaster, CA       | Store95    |                  11
 East         | Stamford, CT        | Store219   |                  12
 East         | New York, NY        | Store122   |                  12
 SouthWest    | North Las Vegas, NV | Store170   |                  10
 SouthWest    | Phoenix, AZ         | Store228   |                  11
 NorthWest    | Bellevue, WA        | Store200   |                  19
 NorthWest    | Portland, OR        | Store39    |                  22
 MidWest      | South Bend, IN      | Store134   |                  10
 MidWest      | Evansville, IN      | Store30    |                  11
 South        | Mesquite, TX        | Store124   |                  10
 South        | Beaumont, TX        | Store226   |                  11
(12 rows)

Data-Analysis: INSERT-SELECT operations

Mon, 01 Jan 0001 00:00:00 +0000

An INSERT-SELECT query selects values from a source and inserts them into a target, as in the following example:

=> INSERT /*+direct*/ INTO destination SELECT * FROM source;

You can optimize an INSERT-SELECT query by matching sort orders or using identical segmentation.

Matching sort orders

To prevent the INSERT operation from sorting the SELECT output, make sure that the sort order for the SELECT query matches the projection sort order of the target table.

For example, on a single-node database:

=> CREATE TABLE source (col1 INT, col2 INT, col3 INT);

=> CREATE PROJECTION source_p (col1, col2, col3)
     AS SELECT col1, col2, col3 FROM source
     ORDER BY col1, col2, col3
     SEGMENTED BY HASH(col3)
     ALL NODES;

=> CREATE TABLE destination (col1 INT, col2 INT, col3 INT);

=> CREATE PROJECTION destination_p (col1, col2, col3)
     AS SELECT col1, col2, col3 FROM destination
     ORDER BY col1, col2, col3
     SEGMENTED BY HASH(col3)
     ALL NODES;

The following INSERT operation does not require a sort because the query result has the column order of the projection:

=> INSERT /*+direct*/ INTO destination SELECT * FROM source;

In the following example, the INSERT operation does require a sort because the column order does not match the projection order:

=> INSERT /*+direct*/ INTO destination SELECT col1, col3, col2 FROM source;

The following INSERT does not require a sort. The order of the columns does not match, but the explicit ORDER BY clause causes the output to be sorted by col1, col3, and col2:

=> INSERT /*+direct*/ INTO destination SELECT col1, col3, col2 FROM source
      GROUP BY col1, col3, col2
      ORDER BY col1, col2, col3 ;

Identical segmentation

When selecting from a segmented source table and inserting into a segmented destination table, segment both projections on the same column to avoid resegmenting the data, as in the following example:

=> CREATE TABLE source (col1 INT, col2 INT, col3 INT);

=> CREATE PROJECTION source_p (col1, col2, col3) AS
     SELECT col1, col2, col3 FROM source
     SEGMENTED BY HASH(col3) ALL NODES;

=> CREATE TABLE destination (col1 INT, col2 INT, col3 INT);

=> CREATE PROJECTION destination_p (col1, col2, col3) AS
     SELECT col1, col2, col3 FROM destination
     SEGMENTED BY HASH(col3) ALL NODES;

=> INSERT /*+direct*/ INTO destination SELECT * FROM source;

Data-Analysis: DELETE and UPDATE queries

Mon, 01 Jan 0001 00:00:00 +0000

OpenText™ Analytics Database is optimized for query-intensive workloads, so DELETE and UPDATE queries might not achieve the same level of performance as other queries. DELETE and UPDATE operations must update all projections, so these operations can be no faster than the slowest projection. For details, see Optimizing DELETE and UPDATE.

Data-Analysis: Data collector table queries

Mon, 01 Jan 0001 00:00:00 +0000

The Data Collector extends system table functionality by gathering and retaining information about your database cluster. The Data Collector makes this information available in system tables.

OpenText™ Analytics Database stores Data Collection data in the Data Collector directory under the vertica or catalog path. Use Data Collector information to query the past state of system tables and extract aggregate information.

In general, queries on Data Collector tables are more efficient when they include only the columns that contain the desired data. Queries are also more efficient when they:

Avoiding resegmentation

You can avoid resegmentation when you join the following DC tables on session_id or transaction_id, because all data is local:

dc_session_starts
dc_session_ends
dc_requests_issued
dc_requests_completed

Resegmentation is not required when a query includes the node_name column. For example:

=> SELECT dri.transaction_id, dri.request, drc.processed_row_count
    FROM dc_requests_issued dri
    JOIN dc_requests_completed drc
    USING (node_name, session_id, request_id)
    WHERE dri.time between 'April 7,2015'::timestamptz and 'April 8,2015'::timestamptz
    AND drc.time between 'April 7,2015'::timestamptz and 'April 8,2015'::timestamptz;

This query runs efficiently because:

The initiator node writes only to dc_requests_issued and dc_requests_completed.
Columns session_id and node_name are correlated.

Using time predicates

Use non-volatile functions and TIMESTAMP for the time range predicates. OpenText™ Analytics Database optimizes SQL performance for DC tables that use the time predicate.

Each DC table has a time column. Use this column to enter the time range as the query predicate.

For example, this query returns data for dates between September 1 and September 10: select * from dc_foo where time > 'Sept 1, 2015::timestamptz and time < 'Sept 10 2015':: timestamptz; You can change the minimum and maximum time values to adjust the time for which you want to retrieve data.

You must use non-volatile functions as time predicates. Volatile functions cause queries to run inefficiently. This example returns all queries that started and ended on April 7, 2015. However, the query runs at less than optimal performance because trunc and timestamp are volatile:

=> SELECT dri.transaction_id, dri.request, drc.processed_row_count
    FROM dc_requests_issued dri
    LEFT JOIN dc_requests_completed drc
    USING (session_id, request_id)
    WHERE trunc(dri.time, ‘DDD’) > 'April 7,2015'::timestamp
    AND trunc(drc.time, ‘DDD’) < 'April 8,2015'::timestamp;

OpenText Analytics Database 26.2.x – Query optimization

Data-Analysis: Initial process for improving query performance

Run Database Designer

Check query events proactively

Review the query plan

See also

Data-Analysis: Column encoding

Data-Analysis: Projections for queries with predicates

Queries that use date ranges

Note

Queries for tables with a high-cardinality primary key

Data-Analysis: GROUP BY queries

Data-Analysis: DISTINCT in a SELECT query list

Data-Analysis: JOIN queries

Other best practices

Data-Analysis: ORDER BY queries

See also

Data-Analysis: Analytic functions

Data-Analysis: LIMIT queries

Limiting the query result set

Important

Limiting window partitioning results

Data-Analysis: INSERT-SELECT operations

Matching sort orders

Identical segmentation

Data-Analysis: DELETE and UPDATE queries

Data-Analysis: Data collector table queries

Avoiding resegmentation

Using time predicates