OpenText Analytics Database 26.2.x – Creating a database design

Admin: About Database Designer

Mon, 01 Jan 0001 00:00:00 +0000

Database Designer uses sophisticated strategies to create a design that provides excellent performance for ad-hoc queries and specific queries while using disk space efficiently.

During the design process, Database Designer analyzes the logical schema definition, sample data, and sample queries, and creates a physical schema (projections) in the form of a SQL script that you deploy automatically or manually. This script creates a minimal set of superprojections to ensure K-safety.

In most cases, the projections that Database Designer creates provide excellent query performance within physical constraints while using disk space efficiently.

General design options

When you run Database Designer, several general options are available:

Create a comprehensive or incremental design.
Optimize for query execution, load, or a balance of both.
Require K-safety.
Recommend unsegmented projections when feasible.
Analyze statistics before creating the design.

Design input

Database Designer bases its design on the following information that you provide:

Design queries that you typically run during normal database operations.
Design tables that contain sample data.

Output

Database Designer yields the following output:

Design script that creates the projections for the design in a way that meets the optimization objectives and distributes data uniformly across the cluster.
Deployment script that creates and refreshes the projections for your design. For comprehensive designs, the deployment script contains commands that remove non-optimized projections. The deployment script includes the full design script.
Backup script that contains SQL statements to deploy the design that existed on the system before deployment. This file is useful in case you need to revert to the pre-deployment design.

Design restrictions

Database Designer-generated designs:

Exclude live aggregate or Top-K projections. You must create these manually. See CREATE PROJECTION.
Do not sort, segment, or partition projections on LONG VARBINARY and LONG VARCHAR columns.
Do not support operations on complex types.

Post-design options

While running Database Designer, you can choose to deploy your design automatically after the deployment script is created, or to deploy it manually, after you have reviewed and tested the design. It is recommended that you test the design on a non-production server before deploying the design to your production server.

Admin: How Database Designer creates a design

Mon, 01 Jan 0001 00:00:00 +0000

Design recommendations

Database Designer-generated designs can include the following recommendations:

Sort buddy projections in the same order, which can significantly improve load, recovery, and site node performance. All buddy projections have the same base name so that they can be identified as a group.

Note
If you manually create projections, Database Designer recommends a buddy with the same sort order, if one does not already exist. By default, Database Designer recommends both super and non-super segmented projections with a buddy of the same sort order and segmentation.
Accepts unlimited queries for a comprehensive design.
Identifies similar design queries and assigns them a signature.

For queries with the same signature, Database Designer weights the queries, depending on how many queries have that signature. It then considers the weighted query when creating a design.
Recommends and creates projections in a way that minimizes data skew by distributing data uniformly across the cluster.
Produces higher quality designs by considering UPDATE, DELETE, and SELECT statements.

Admin: Database Designer access requirements

Mon, 01 Jan 0001 00:00:00 +0000

By default, only users with the DBADMIN role can run Database Designer. Non-DBADMIN users can run Database Designer only if they are granted the necessary privileges and DBDUSER role, as described below. You can also enable users to run Database Designer on the Management Console (see Enabling Users to run Database Designer on Management Console).

Add a temporary folder to all cluster nodes with CREATE LOCATION:
```
=> CREATE LOCATION '/tmp/dbd' ALL NODES;
```
Grant the desired user CREATE privileges to create schemas on the current (DEFAULT) database, with GRANT DATABASE:
```
=> GRANT CREATE ON DATABASE DEFAULT TO dbd-user;
```
Grant the DBDUSER role to dbd-user with GRANT ROLE:
```
=> GRANT DBDUSER TO dbd-user;
```
On all nodes in the cluster, grant dbd-user access to the temporary folder with GRANT LOCATION:
```
=> GRANT ALL ON LOCATION '/tmp/dbd' TO dbd-user;
```

Grant dbd-user privileges on one or more database schemas and their tables, with GRANT SCHEMA and GRANT TABLE, respectively:

=> GRANT ALL ON SCHEMA this-schema[,...] TO dbd-user;
=> GRANT ALL ON ALL TABLES IN SCHEMA this-schema[,...] TO dbd-user;

Enable the DBDUSER role on dbd-user in one of the following ways:
- As dbd-user, enable the DBDUSER role with SET ROLE:
```
=> SET ROLE DBDUSER;
```
- As DBADMIN, automatically enable the DBDUSER role for dbd-user on each login, with ALTER USER:
```
=> ALTER USER dbd-user DEFAULT ROLE DBDUSER;
```

Important

When you grant the DBDUSER role, be sure to associate a resource pool with that user to manage resources while Database Designer runs.

Multiple users can run Database Designer concurrently without interfering with each other or exhausting cluster resources. When a user runs Database Designer, either with Management Console or programmatically, execution is generally contained by the user's resource pool, but might spill over into system resource pools for less-intensive tasks.

Enabling users to run Database Designer on Management Console

Users who are already granted the DBDUSER role and required privileges, as described above, can also be enabled to run Database Designer on Management Console:

Log in as a superuser to Management Console.
Click MC Settings.
Click User Management.
Specify an MC user:
- To create an MC user, click Add.
- To use an existing MC user, select the user and click Edit.
Next to the DB access level window, click Add.
In the Add Permissions window:
1. From the Choose a database drop-down list, select the database on which to create a design.
2. In the Database username field, enter the dbd-user user name that you created earlier.
3. In the Database password field, enter the database password.
4. In the Restrict access drop-down list, select the level of MC user for this user.
Click OK to save your changes.
Log out of the MC Super user account.

The MC user is now mapped to dbd-user. Log in as the MC user and use Database Designer to create an optimized design for your database.

DBDUSER capabilities and limitations

As a DBDUSER, the following constraints apply:

Designs must set K-safety to be equal to system K-safety. If a design violates K-safety by lacking enough buddy projections for tables, the design does not complete.
You cannot explicitly advance the ancient history mark (AHM)—for example, call MAKE_AHM_NOW—until after deploying the design.

When you create a design, you automatically have privileges to manipulate that design. Other tasks might require additional privileges:

Task	Required privileges
Submit design tables	USAGE on the design table schema OWNER on the design table
Submit a single design query	EXECUTE on the design query
Submit a file of design queries	READ privilege on the storage location that contains the query file EXECUTE privilege on all queries in the file
Submit design queries from results of a user query	EXECUTE privilege on the user queries EXECUTE privilege on each design query retrieved from the results of the user query
Create design and deployment scripts	WRITE privilege on the storage location of the design script WRITE privilege on the storage location of the deployment script

Admin: Logging projection data for Database Designer

Mon, 01 Jan 0001 00:00:00 +0000

When you run Database Designer, the Optimizer proposes a set of ideal projections based on the options that you specify. When you deploy the design, Database Designer creates the design based on these projections. However, space or budget constraints may prevent Database Designer from creating all the proposed projections. In addition, Database Designer may not be able to implement the projections using ideal criteria.

To get information about the projections, first enable the Database Designer logging capability. When enabled, Database Designer stores information about the proposed projections in two Data Collector tables. After Database Designer deploys the design, these logs contain information about which proposed projections were actually created. After deployment, the logs contain information about:

Projections that the Optimizer proposed
Projections that Database Designer actually created when the design was deployed
Projections that Database Designer created, but not with the ideal criteria that the Optimizer identified.
The DDL used to create all the projections
Column optimizations

If you do not deploy the design immediately, review the log to determine if you want to make any changes. If the design has been deployed, you can still manually create some of the projections that Database Designer did not create.

To enable the Database Designer logging capability, see Enabling logging for Database Designer.

To view the logged information, see Viewing Database Designer logs.

Admin: General design settings

Mon, 01 Jan 0001 00:00:00 +0000

Before you run Database Designer, you must provide specific information on the design to create.

Design name

All designs that you create with Database Designer must have unique names that conform to the conventions described in Identifiers, and are no more than 32 characters long (16 characters if you use Database Designer in Administration Tools or Management Console).

The design name is incorporated into the names of files that Database Designer generates, such as its deployment script. This can help you differentiate files that are associated with different designs.

Design type

Database Designer can create two distinct design types: comprehensive or incremental.

Comprehensive design

A comprehensive design creates an initial or replacement design for all the tables in the specified schemas. Create a comprehensive design when you are creating a new database.

To help Database Designer create an efficient design, load representative data into the tables before you begin the design process. When you load data into a table, the database creates an unoptimized superprojection so that Database Designer has projections to optimize. If a table has no data, Database Designer cannot optimize it.

Optionally, supply Database Designer with representative queries that you plan to use so Database Designer can optimize the design for them. If you do not supply any queries, Database Designer creates a generic optimization of the superprojections that minimizes storage, with no query-specific projections.

During a comprehensive design, Database Designer creates deployment scripts that:

Create projections to optimize query performance.
Create replacement buddy projections when Database Designer changes the encoding of existing projections that it decides to keep.

Incremental design

After you create and deploy a comprehensive database design, your database is likely to change over time in various ways. Consider using Database Designer periodically to create incremental designs that address these changes. Changes that warrant an incremental design can include:

Significant data additions or updates
New or modified queries that you run regularly
Performance issues with one or more queries
Schema changes

Optimization objective

Database Designer can optimize the design for one of three objectives:

Load: Designs that are optimized for loads minimize database size, potentially at the expense of query performance.
Query: Designs that are optimized for query performance. These designs typically favor fast query execution over load optimization, and thus result in a larger storage footprint.
Balanced: Designs that are balanced between database size and query performance.

A fully optimized query has an optimization ratio of 0.99. Optimization ratio is the ratio of a query's benefits achieved in the design produced by the Database Designer to that achieved in the ideal plan. The optimization ratio is set in the OptRatio parameter in designer.log.

Design tables

Database Designer needs one or more tables with a moderate amount of sample data—approximately 10 GB—to create optimal designs. Design tables with large amounts of data adversely affect Database Designer performance. Design tables with too little data prevent Database Designer from creating an optimized design. If a design table has no data, Database Designer ignores it.

Note

If you drop a table after adding it to the design, Database Designer cannot build or deploy the design.

Design queries

A database design that is optimized for query performance requires a set of representative queries, or design queries. Design queries are required for incremental designs, and optional for comprehensive designs. You list design queries in a SQL file that you supply as input to Database Designer. Database Designer checks the validity of the queries when you add them to your design, and again when it builds the design. If a query is invalid, Database Designer ignores it.

If you use Management Console to create a database design, you can submit queries either from an input file or from the system table QUERY_REQUESTS. For details, see Creating a design manually.

The maximum number of design queries depends on the design type: ≤200 queries for a comprehensive design, ≤100 queries for an incremental design. Optionally, you can assign weights to the design queries that signify their relative importance. Database Designer uses those weights to prioritize the queries in its design.

Segmented and unsegmented projections

When creating a comprehensive design, Database Designer creates projections based on data statistics and queries. It also reviews the submitted design tables to decide whether projections should be segmented (distributed across the cluster nodes) or unsegmented (replicated on all cluster nodes).

By default, Database Designer recommends only segmented projections. You can enable Database Designer to recommend unsegmented projections . In this case, Database Designer recommends segmented superprojections for large tables when deploying to multi-node clusters, and unsegmented superprojections for smaller tables.

Database Designer uses the following algorithm to determine whether to recommend unsegmented projections. Assuming that largest-row-count equals the number of rows in the design table with the largest number of rows, Database Designer recommends unsegmented projections if any of the following conditions is true:

largest-row-count< 1,000,000 ANDnumber-table-rows≤10%-largest-row-count
largest-row-count≥ 10,000,000 ANDnumber-table-rows≤1%-largest-row-count
1,000,000 ≤ largest-row-count< 10,000,000 ANDnumber-table-rows ≤ 100,000

Database Designer does not segment projections on:

Single-node clusters
LONG VARCHAR and LONG VARBINARY columns

For more information, see High availability with projections.

Statistics analysis

By default, Database Designer analyzes statistics for design tables when they are added to the design. Accurate statistics help Database Designer optimize compression and query performance.

Analyzing statistics takes time and resources. If you are certain that design table statistics are up to date, you can specify to skip this step and avoid the overhead otherwise incurred.

For more information, see Collecting Statistics.

Admin: Building a design

Mon, 01 Jan 0001 00:00:00 +0000

After you have created design tables and loaded data into them, and then specified the parameters you want Database Designer to use when creating the physical schema, direct Database Designer to create the scripts necessary to build the design.

Note

You cannot stop a running database if Database Designer is building a database design.

When you build a database design, the database generates two scripts:

Deployment script: design-name_deploy.sql—Contains the SQL statements that create projections for the design you are deploying, deploy the design, and drop unused projections. When the deployment script runs, it creates the optimized design. For details about how to run this script and deploy the design, see Deploying a Design.
Design script: design-name_design.sql—Contains the CREATE PROJECTION statements that Database Designeruses to create the design. Review this script to make sure you are happy with the design.

The design script is a subset of the deployment script. It serves as a backup of the DDL for the projections that the deployment script creates.

When you create a design using Management Console:

If you submit a large number of queries to your design and build it right immediately, a timing issue could cause the queries not to load before deployment starts. If this occurs, you might see one of the following errors:
- No queries to optimize for
- No tables to design projections for
To accommodate this timing issue, you may need to reset the design, check the Queries tab to make sure the queries have been loaded, and then rebuild the design. Detailed instructions are in:
- Using the wizard to create a design
- Creating a design manually
The scripts are deleted when deployment completes. To save a copy of the deployment script after the design is built but before the deployment completes, go to the Output window and copy and paste the SQL statements to a file.

Admin: Resetting a design

Mon, 01 Jan 0001 00:00:00 +0000

You must reset a design when:

You build a design and the output scripts described in Building a Design are not created.
You build a design but Database Designer cannot complete the design because the queries it expects are not loaded.

Resetting a design discards all the run-specific information of the previous Database Designer build, but retains its configuration (design type, optimization objectives, K-safety, etc.) and tables and queries.

After you reset a design, review the design to see what changes you need to make. For example, you can fix errors, change parameters, or check for and add additional tables or queries. Then you can rebuild the design.

You can only reset a design in Management Console or by using the DESIGNER_RESET_DESIGN function.

Admin: Deploying a design

Mon, 01 Jan 0001 00:00:00 +0000

After running Database Designer to generate a deployment script, it is recommended that you test your design on a non-production server before you deploy it to your production server.

Both the design and deployment processes run in the background. This is useful if you have a large design that you want to run overnight. Because an active SSH session is not required, the design/deploy operations continue to run uninterrupted, even if the session is terminated.

Note

You cannot stop a running database if Database Designer is building or deploying a database design.

Database Designer runs as a background process. Multiple users can run Database Designer concurrently without interfering with each other or using up all the cluster resources. However, if multiple users are deploying a design on the same tables at the same time, Database Designer may not be able to complete the deployment. To avoid problems, consider the following:

Schedule potentially conflicting Database Designer processes to run sequentially overnight so that there are no concurrency problems.
Avoid scheduling Database Designer runs on the same set of tables at the same time.

There are two ways to deploy your design:

Admin: How to create a design

Mon, 01 Jan 0001 00:00:00 +0000

There are three ways to create a design using Database Designer:

From Management Console, open a database and select the Design page at the bottom of the window.

For details about using Management Console to create a design, see Creating a database design in Management Console
Programmatically, using the techniques described in About Running Database Designer Programmatically. To run Database Designer programmatically, you must be a DBADMIN or have been granted the DBDUSER role and enabled that role.
From the Administration Tools menu, by selecting Configuration Menu > Run Database Designer. You must be a DBADMIN user to run Database Designer from the Administration Tools.

For details about using Administration Tools to create a design, see Using administration tools to create a design.

The following table shows what Database Designer capabilities are available in each tool:

Database Designer Capability	Management Console	Running Database Designer Programmatically	Administrative Tools
Create design	Yes	Yes	Yes
Design name length (# of characters)	16	32	16
Build design (create design and deployment scripts)	Yes	Yes	Yes
Create backup script			Yes
Set design type (comprehensive or incremental)	Yes	Yes	Yes
Set optimization objective	Yes	Yes	Yes
Add design tables	Yes	Yes	Yes
Add design queries file	Yes	Yes	Yes
Add single design query		Yes
Use query repository	Yes	Yes
Set K-safety	Yes	Yes	Yes
Analyze statistics	Yes	Yes	Yes
Require all unsegmented projections	Yes	Yes
View event history	Yes	Yes
Set correlation analysis mode (Default = 0)		Yes

Admin: Running Database Designer programmatically

Mon, 01 Jan 0001 00:00:00 +0000

The database provides a set of meta-functions that enable programmatic access to Database Designer functionality. Run Database Designer programmatically to perform the following tasks:

Optimize performance on tables that you own.
Create or update a design without requiring superuser or DBADMIN intervention.
Add individual queries and tables, or add data to your design, and then rerun Database Designer to update the design based on this new information.
Customize the design.
Use recently executed queries to set up your database to run Database Designer automatically on a regular basis.
Assign each design query a query weight that indicates the importance of that query in creating the design. Assign a higher weight to queries that you run frequently so that Database Designer prioritizes those queries in creating the design.

For more details about Database Designer functions, see Database Designer function categories.

Admin: Creating custom designs

Mon, 01 Jan 0001 00:00:00 +0000

It is recommended that you use the physical schema design produced by Database Designer, which provides K-safety, excellent query performance, and efficient use of storage space. If any queries run less as efficiently than you expect, consider using the Database Designer incremental design process to optimize the database design for the query.

If the projections created by Database Designer still do not meet your needs, you can write custom projections, from scratch or based on projection designs created by Database Designer.

If you are unfamiliar with writing custom projections, start by modifying an existing design generated by Database Designer.