Resolved issues

Operations over directories containing large numbers of external data files on object stores consumed more CPU than expected. This issue affected planning for external table queries, COPY statements, and license auditing. A change in the algorithm reduced the amount of CPU time used to perform these operations.

Actions

Unlike a regular outer join, an event series join pads output with some cached values that represent the columns of the mismatched side. Padding with nulls is required if no previous rows are available for interpolation. To process event series joins, cached rows are always initialized with nulls, then updated with values to represent the mismatched side if applicable.

When initializing cached rows with a schema that differed from what the join expected at that stage of the query plan, Vertica occasionally crashed. This issue has been resolved: now cached rows are initially set to null tuples, using the correct tuple schema.

Users can now control configuration parameter MaxDVROSPerContainer knob and set it to any value>1. The new formula is:

max(2, (MaxDVROSPerContainer+1)/2)

A two-level strata helps avoid excessive DVMergeouts: DVs with fewer deleted rows than (TBL.nrows/MaxDVROSPerStratum), where maxDVROSPerStratun is max(2, (MaxDVROSPerContainer+1)/2), are placed at stratum 0; if the number of these DVs exceeds MaxDVROSPerStratum, they are merged together. As before, larger DVs at stratum 1 are not merged.

The configuration parameter MaxParsedQuerySizeMB knob is set in MB (as documented). The optimizer can require very large amounts of memory for a given query, much of it consumed by internal objects that the parser creates when it converts the query into a query tree for optimization.

Other issues were identified as contributing to excessive memory consumption, and these have been addressed, including freeing memory allocated to query tree objects when they are no longer in use.

Several Vertica partition management operations support a force-split option: copy_partitions_to_table, drop_partitions, move_partitions_to_table, and swap_partitions_between_tables. In all cases, if force-split is true, and the operation specifies a range of partition keys that span multiple containers or part of a single container, Vertica executes the operation in two steps:

1. Breaks open the ROS containers
2. Reorganizes/merges the ROS containers as per operation requirements

Occasionally, the Tuple Mover would attempt to merge the split ROS containers before step 2 was complete, and the operation would fail. This issue has been resolved: the two stages are now part of a single transaction. Until the transaction is complete, the Tuple Mover does not queue merge requests on the split containers.

If admintools called create_db and the database creation process was lengthy, admintools sometimes prompted users to confirm whether to continue waiting. If the user did not answer the prompt--for example, when create_db was called by a script--create_db completed execution without creating all database nodes and properly updating the configuration file admintools.conf. In this case, the database was incomplete and unusable.

Now, the prompt times out after 120 seconds. If the user doesn't respond within that time period, create_db exits.

It was unsafe to reorder query predicates when the following conditions were true:

• The query contained a predicate on a projection's leading sort order columns that restricted the leading columns to constant values, where the leading columns also were not run-length encoded.
• A SIPS predicate from a merge join was applied to non-leading sort order columns of that projection.

This issue has been resolved: query predicates can no longer be reordered when it contains a predicate on a projection's leading sort order columns that restrict leading columns to constant values, where the leading columns are not run-length encoded.

Vertica was unable to optimize queries on v_internal tables, where equality predicates (with operators = or <=>) filtered on columns relname or nspname, in the following cases:

• The predicate specified expressions with embedded characters such as underscore (_) or percentage (%). For example:

  SELECT * FROM v_internal.vs_columns WHERE nspname = 'x_yz';
  

• The query contained multiple predicates separated by AND operators, and more than one predicate queried the same column, either nspname or relname. For example:

  SELECT * FROM v_internal.vs_columns WHERE nspname = 'xyz' AND nspname <> 'vs_internal';  
  

In this case, Vertica was unable to optimize equality predicate nspname = 'xyz'. In all these cases, the queries are now optimized as expected.

Previously, when migrate_enterprise_to_eon found projections that were Inconsistent with cluster segmentation, it wrote to the migration log file an incorrect anchor table name; it also provided only one projection, _b0. This issue has been resolved: now the log file error log file lists the names of all tables with problematic projections. You can use these names as arguments to meta-function REBALANCE_TABLE.