This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Before you upgrade

Before you upgrade the Vertica database, perform the following steps:.

Before you upgrade the Vertica database, perform the following steps:

  • Verify that you have enough RAM available to run the upgrade. The upgrade requires approximately three times the amount of memory your database catalog uses.

    You can calculate catalog memory usage on all nodes by querying system table RESOURCE_POOL_STATUS:

    => SELECT node_name, pool_name, memory_size_kb FROM resource_pool_status WHERE pool_name = 'metadata';
    
  • Perform a full database backup. This precautionary measure allows you to restore the current version if the upgrade is unsuccessful.

  • Perform a backup of your grants.

  • Verify platform requirements for the new version.

  • Determine whether you are using any third-party user-defined extension libraries (UDxs). UDx libraries that are compiled (such as those developed using C++ or Java) may need to be recompiled with a new version of the Vertica SDK libraries to be compatible with the new version of Vertica. See UDx library compatibility with new server versions.

  • Check catalog storage space.

  • Note that any user or role with the same name as a predefined role is renamed to OLD_n_name, where n is an integer that increments from zero until the resulting name is unique and name is the previous name of the user or role.

  • If you're upgrading from Vertica 9.2.x and have set the PasswordMinCharChange or PasswordMinLifeTime system-level security parameters, take note of their current values. You will have to set these parameters again, this time at the PROFILE-level, to reproduce your configuration. To view the current values for these parameters, run the following query:

    => SELECT parameter_name,current_value from CONFIGURATION_PARAMETERS
        WHERE parameter_name IN ('PasswordMinCharChange', 'PasswordMinLifeTime');
    

After you complete these tasks, shut down the database gracefully.

1 - Verifying platform requirements

The Vertica installer checks the target platform as it runs, and stops whenever it determines the platform fails to meet an installation requirement.

The Vertica installer checks the target platform as it runs, and stops whenever it determines the platform fails to meet an installation requirement. Before you update the server package on your systems, manually verify that your platform meets all hardware and software requirements (see Platform and hardware requirements and recommendations).

By default, the installer stops on all warnings. You can configure the level where the installer stops installation, through the installation parameter --failure-threshold. If you set the failure threshold to FAIL, the installer ignores warnings and stops only on failures.

2 - Checking catalog storage space

Use the commands documented here to determine how much catalog space is available before upgrading.

Use the commands documented here to determine how much catalog space is available before upgrading. This helps you determine how much space the updated catalog may take up.

Compare how much space the catalog currently uses against space that is available in the same directory:

  1. Use the du command to determine how much space the catalog directory currently uses:

    $ du -s -BG v_vmart_node0001_catalog
    2G      v_vmart_node0001_catalog
    
  2. Determine how much space is available in the same directory:

    $ df -BG v_vmart_node0001_catalog
    Filesystem     1G-blocks  Used Available Use% Mounted on
    /dev/sda2            48G   19G       26G  43% /
    

3 - Verify license compliance for ORC and Parquet data

If you are upgrading from a version before 9.1.0 and:.

If you are upgrading from a version before 9.1.0 and:

  • Your database has external tables based on ORC or Parquet files (whether stored locally on the Vertica cluster or on a Hadoop cluster)

  • Your Vertica license has a raw data allowance

follow the steps in this topic before upgrading.

Background

Vertica licenses can include a raw data allowance. Since 2016, Vertica licenses have allowed you to use ORC and Parquet data in external tables. This data has always counted against any raw data allowance in your license. Previously, the audit of data in ORC and Parquet format was handled manually. Because this audit was not automated, the total amount of data in your native tables and external tables could exceed your licensed allowance for some time before being spotted.

Starting in version 9.1.0, Vertica automatically audits ORC and Parquet data in external tables. This auditing begins soon after you install or upgrade to version 9.1.0. If your Vertica license includes a raw data allowance and you have data in external tables based on Parquet or ORC files, review your license compliance before upgrading to Vertica 9.1.x. Verifying your database is compliant with your license terms avoids having your database become non-compliant soon after you upgrade.

Verifying your ORC and Parquet usage complies with your license terms

To verify your data usage is compliant with your license, run the following query as the database administrator:

SELECT (database_size_bytes + file_size_bytes) <= license_size_bytes
       "license_compliant?"
       FROM (SELECT database_size_bytes,
                    license_size_bytes FROM license_audits
                    WHERE audited_data='Total'
                    ORDER BY audit_end_timestamp DESC LIMIT 1) dbs,
            (SELECT sum(total_file_size_bytes) file_size_bytes
                    FROM external_table_details
                    WHERE source_format IN ('ORC', 'PARQUET')) ets;

This query returns one of three values:

  • If you do not have any external data in ORC or Parquet format, the query returns 0 rows:

     license_compliant?
    --------------------
    (0 rows)
    

    In this case, you can proceed with your upgrade.

  • If you have data in external tables based on ORC or Parquet format, and that data does not cause your database to exceed your raw data allowance, the query returns t:

     license_compliant?
    --------------------
     t
    (1 row)
    

    In this case, you can proceed with your upgrade.

  • If the data in your external tables based on ORC and Parquet causes your database to exceed your raw data allowance, the query returns f:

     license_compliant?
    --------------------
     f
    (1 row)
    

    In this case, resolve the compliance issue before you upgrade. See below for more information.

Resolving non-compliance

If query in the previous section indicates that your database is not in compliance with your license, you should resolve this issue before upgrading. There are two ways you can bring your database into compliance:

  • Contact Vertica to upgrade your license to a larger data size allowance. See Obtaining a license key file.

  • Delete data (either from ORC and Parquet-based external tables or Vertica native tables) to bring your data size into compliance with your license. You should always backup any data you are about to delete from Vertica. Dropping external tables is a less disruptive way to reduce the size of your database, as the data is not lost—it is still in the files that your external table is based on.

4 - Backing up and restoring grants

After an upgrade, if the prototypes of UDx libraries change, Vertica will drop the grants on those libraries since they aren't technically the same function anymore.

After an upgrade, if the prototypes of UDx libraries change, Vertica will drop the grants on those libraries since they aren't technically the same function anymore. To resolve these types of issues, it's best practice to back up the grants on these libraries so you can restore them after the upgrade.

  1. Save the following SQL to a file named user_ddl.sql. It creates a view named user_ddl which contains the grants on all objects in the database.

    CREATE OR REPLACE VIEW user_ddl AS
    (
    SELECT 0 as grant_order,
           name principal_name,
           'CREATE ROLE "' || name || '"' ||    ';' AS sql,
           'NONE' AS object_type,
           'NONE' AS object_name
      FROM v_internal.vs_roles vr
     WHERE NOT vr.predefined_role -- Exclude system roles
       AND ldapdn = ''            -- Limit to NON-LDAP created roles
    )
    UNION ALL
    (
      SELECT 1, -- CREATE USERs
             user_name,
             'CREATE USER "' || user_name || '"' ||
             DECODE(is_locked, TRUE, ' ACCOUNT LOCK', '') ||
             DECODE(grace_period, 'undefined', '', ' GRACEPERIOD  ''' || grace_period || '''') ||
             DECODE(idle_session_timeout, 'unlimited', '', ' IDLESESSIONTIMEOUT ''' || idle_session_timeout || '''') ||
             DECODE(max_connections, 'unlimited', '', ' MAXCONNECTIONS ' || max_connections || ' ON ' || connection_limit_mode) ||
             DECODE(memory_cap_kb, 'unlimited', '', ' MEMORYCAP ''' || memory_cap_kb || 'K''') ||
             DECODE(profile_name, 'default', '', ' PROFILE ' || profile_name) ||
             DECODE(resource_pool, 'general', '', ' RESOURCE POOL ' || resource_pool) ||
             DECODE(run_time_cap, 'unlimited', '', ' RUNTIMECAP ''' || run_time_cap || '''') ||
             DECODE(search_path, '', '', ' SEARCH_PATH ' || search_path) ||
             DECODE(temp_space_cap_kb, 'unlimited', '', ' TEMPSPACECAP ''' || temp_space_cap_kb || 'K''') || ';' AS sql,
             'NONE' AS object_type,
             'NONE' AS object_name
        FROM v_catalog.users
       WHERE NOT is_super_user -- Exclude database superuser
         AND ldap_dn = ''      -- Limit to NON-LDAP created users
    )
    UNION ALL
    (
      SELECT 2, -- GRANTs
             grantee,
             'GRANT ' || REPLACE(TRIM(BOTH ' ' FROM words), '*', '') ||
             CASE
               WHEN object_type = 'RESOURCEPOOL' THEN ' ON RESOURCE POOL '
               WHEN object_type = 'STORAGELOCATION' THEN ' ON LOCATION '
               WHEN object_type = 'CLIENTAUTHENTICATION' THEN 'AUTHENTICATION '
               WHEN object_type IN ('DATABASE', 'LIBRARY', 'MODEL', 'SEQUENCE', 'SCHEMA') THEN ' ON ' || object_type || ' '
               WHEN object_type = 'PROCEDURE' THEN (SELECT ' ON ' || CASE REPLACE(procedure_type, 'User Defined ', '')
                                                                       WHEN 'Transform' THEN 'TRANSFORM FUNCTION '
                                                                       WHEN 'Aggregate' THEN 'AGGREGATE FUNCTION '
                                                                       WHEN 'Analytic' THEN 'ANALYTIC FUNCTION '
                                                                       ELSE UPPER(REPLACE(procedure_type, 'User Defined ', '')) || ' '
                                                                     END
                                                      FROM vs_procedures
                                                     WHERE proc_oid = object_id)
               WHEN object_type = 'ROLE' THEN ''
               ELSE ' ON '
             END ||
             NVL2(object_schema, object_schema || '.', '') || CASE WHEN object_type = 'STORAGELOCATION' THEN (SELECT '''' || location_path || ''' ON ' || node_name FROM storage_locations WHERE location_id = object_id) ELSE object_name END ||
             CASE
               WHEN object_type = 'PROCEDURE' THEN (SELECT CASE WHEN procedure_argument_types = '' OR procedure_argument_types = 'Any' THEN '()' ELSE '(' || procedure_argument_types || ')' END
                                                      FROM vs_procedures
                                                     WHERE proc_oid = object_id)
               ELSE ''
             END ||
             ' TO ' || grantee ||
             CASE WHEN INSTR(words, '*') > 0 THEN ' WITH GRANT OPTION' ELSE '' END
             || ';',
             object_type,
             object_name
      FROM (SELECT grantee, object_type, object_schema, object_name, object_id,
                     v_txtindex.StringTokenizerDelim(DECODE(privileges_description, '', ',' , privileges_description), ',')
                       OVER (PARTITION BY grantee, object_type, object_schema, object_name, object_id)
              FROM v_catalog.grants) foo
     ORDER BY CASE REPLACE(TRIM(BOTH ' ' FROM words), '*', '') WHEN 'USAGE' THEN 1 ELSE 2 END
    )
    UNION ALL
    (
      SELECT 3, -- Default ROLEs
             user_name,
             'ALTER USER "' || user_name || '"' ||
               DECODE(default_roles, '', '', ' DEFAULT ROLE ' || REPLACE(default_roles, '*', ''))  || ';' ,
             'NONE' AS object_type,
             'NONE' AS object_name
        FROM v_catalog.users
       WHERE default_roles <> ''
    )
    UNION ALL -- GRANTs WITH ADMIN OPTION
    (
      SELECT 4, user_name, 'GRANT ' || REPLACE(TRIM(BOTH ' ' FROM words), '*', '') || ' TO ' || user_name || ' WITH ADMIN OPTION;',
             'NONE' AS object_type ,
             'NONE' AS object_name
        FROM (SELECT user_name, v_txtindex.StringTokenizerDelim(DECODE(all_roles, '', ',', all_roles), ',') OVER (PARTITION BY user_name)
                FROM v_catalog.users
               WHERE all_roles <> '') foo
       WHERE INSTR(words, '*') > 0
    )
    UNION ALL
    (
      SELECT 5, 'public', 'ALTER SCHEMA ' || name || ' DEFAULT ' || CASE WHEN defaultinheritprivileges THEN 'INCLUDE PRIVILEGES;' ELSE 'EXCLUDE PRIVILEGES;' END, 'SCHEMA', name
        FROM v_internal.vs_schemata
       WHERE NOT issys -- Exclude system schemas
    )
    UNION ALL
    (
      SELECT 6, 'public', 'ALTER DATABASE ' || database_name ||  ' SET disableinheritedprivileges = ' || current_value || ';',
             'DATABASE', database_name
        FROM v_internal.vs_configuration_parameters
       CROSS JOIN v_catalog.databases
       WHERE parameter_name = 'DisableInheritedPrivileges'
    )
    UNION ALL -- TABLE PRIV INHERITENCE
    (
      SELECT 7, 'public' , 'ALTER TABLE ' || table_schema || '.' || table_name ||
             CASE WHEN inheritprivileges THEN ' INCLUDE PRIVILEGES;' ELSE ' EXCLUDE PRIVILEGES;' END,
             'TABLE' AS object_type,
             table_schema || '.' || table_name AS object_name
        FROM v_internal.vs_tables
        JOIN v_catalog.tables ON (table_id = oid)
    )
    UNION ALL -- VIEW PRIV INHERITENCE
    (
      SELECT 8, 'public', 'ALTER VIEW ' || table_schema || '.' || table_name || CASE WHEN inherit_privileges THEN ' INCLUDE PRIVILEGES;' ELSE ' EXCLUDE PRIVILEGES; ' END,
             'TABLE' AS object_type, table_schema || '.' || table_name AS object_name
        FROM v_catalog.views
    )
    UNION ALL
    (
      SELECT 9, owner_name, 'ALTER TABLE ' || table_schema || '.' || table_name || ' OWNER TO ' || owner_name || ';',
             'TABLE', table_schema || '.' || table_name
        FROM v_catalog.tables
    )
    UNION ALL
    (
      SELECT 10, owner_name, 'ALTER VIEW ' || table_schema || '.' || table_name || ' OWNER TO ' || owner_name || ';', 'TABLE',
             table_schema || '.' || table_name
        FROM v_catalog.views
    );
    
  2. From the Linux command line, run the script in the user_ddl.sql file:

    $ vsql -f user_ddl.sql
    CREATE VIEW
    
  3. Connect to Vertica using vsql.

  4. Export the content of the user_ddl's sql column ordered on the grant_order column to a file:

    => \o pre-upgrade.txt
    => SELECT sql FROM user_ddl ORDER BY grant_order ASC;
    => \o
    
  5. Upgrade Vertica.

  6. Select and save to a different file the view's SQL column with the same command.

    => \o post-upgrade.txt
    => SELECT sql FROM user_ddl ORDER BY grant_order ASC;
    => \o
    
  7. Create a diff between pre-upgrade.txt and post-upgrade.txt. This collects the missing grants into grants-list.txt.

    $ diff pre-upgrade.txt post-upgrade.txt > grants-list.txt
    
  8. To restore any missing grants, run the remaining grants in grants-list.txt, if any:

    => \i 'grants-list.txt'
    

5 - Nonsequential FIPS database upgrades

As of Vertica 10.1.1, FIPS support has been reinstated.

As of Vertica 10.1.1, FIPS support has been reinstated. Prior to this, the last version to support FIPS was Vertica 9.2.x. If you are upgrading from 9.2.x and want to maintain your FIPS certification, you must first perform a direct upgrade from 9.2.x to 10.1.1 before performing further upgrades.

The following procedure performs a direct upgrade from Vertica 9.2.x running on RHEL 6.x to Vertica 10.1.1 on RHEL 8.1.

  1. Create a full backup of your Vertica 9.2.x database. This example uses the configuration file fullRestore.ini.

    $ vbr --config-file=/tmp/fullRestore.ini -t init
    $ vbr --config-file=/tmp/fullRestore.ini -t backup
    
    [Transmission]
    concurrency_backup = 1
    port_rsync = 50000
    encrypt = False
    serviceAccessPass = rsyncpw
    hardLinkLocal = False
    checksum = False
    total_bwlimit_restore = 0
    serviceAccessUser = rsyncuser
    total_bwlimit_backup = 0
    concurrency_restore = 1
    
    [Misc]
    snapshotName = full_restore
    restorePointLimit = 1
    retryDelay = 1
    objects =
    retryCount = 0
    tempDir = /tmp/vbr
    
    [Mapping]
    v_fips_db_node0001 = 198.51.100.0:/home/release/backup/
    v_fips_db_node0002 = 198.51.100.1:/home/release/backup/
    v_fips_db_node0003 = 198.51.100.2:/home/release/backup/
    
    [Database]
    dbPort = 5433
    dbPromptForPassword = False
    dbUser =
    dbPassword =
    dbName = fips_db
    
  2. Shut down the database gracefully. Do not start the database until instructed.

  3. Acquire a RHEL 8.1 cluster with one of the following methods:

    1. Upgrade in place
    2. Reimage your machines
    3. Use a completely different RHEL 8.1 cluster
  4. Enable FIPS on your RHEL 8.1 machines and reboot.

    $ fips-mode-setup --enable
    
  5. Install Vertica 10.1.1 on the RHEL 8.1 cluster.

    $ install_vertica --hosts node0001, node0002, node0003 \
        --rpm /tmp/vertica-10.1.1.x86_64.RHEL8.rpm
    
  6. If you acquired your RHEL 8.1 cluster by reimaging or using a different cluster, you must restore your database.

    $ vbr -c /tmp/fullRestore.ini -t restore
    

    If you encounter the following warning, you can safely ignore it.

    Warning: Vertica versions do not match: v9.2.1-xx -> v10.1.1-xxxxxxxx. This operation may not be supported.
    
  7. Start the Vertica 10.1.1 database to trigger the upgrade. This should be the first time you've started your database since shutting it down in step 2.

    $ admintools -t start_db -d fips_db