Upgrading Vertica on Kubernetes
The operator automates Vertica server version upgrades for a custom resource (CR). Use the upgradePolicy setting in the CR to determine whether your cluster remains online or is taken offline during the version upgrade.
Note
You cannot downgrade Vertica server versions. The VerticaDB operator validates the pre- and post-upgrade versions before it proceeds with any changes to the server version. Vertica uses semantic versioning for each release in the major.minor.patch format.
For a comprehensive list of containerized server versions, see the Docker Hub repository.
Prerequisites
Before you begin, complete the following:
-
Install the VerticaDB operator.
-
Create a VerticaDB custom resource definition manifest.
Setting the policy
The upgradePolicy CR parameter setting determines how the operator upgrades Vertica server versions. It provides the following options:
Setting | Description |
---|---|
Offline |
The operator shuts down the cluster to prevent multiple versions from running simultaneously. The operator performs all server version upgrades using the
|
ReadOnlyOnline |
The cluster continues to operate during a read-only online upgrade. The database is in read-only mode while the operator upgrades the image for the primary subcluster. ImportantTo perform a read-only online upgrade, you must upgrade from Vertica server version 11.1.0 and higher. |
Online |
The cluster continues to operate during an online upgrade. You can modify the data while the operator upgrades the database. ImportantTo perform an online upgrade, you must upgrade from Vertica server version 24.3.0-2 and higher. |
Auto |
The default setting. The operator selects either Offline or ReadOnlyOnline depending on the configuration. The operator performs a ReadOnlyOnline upgrade if all of the following are true:
If the current configuration does not meet all of the previous requirements, the operator performs an Offline upgrade. |
Reconcile loop iteration time
During an upgrade, the operator runs the reconcile loop to compare the actual state of the objects to the desired state defined in the CR. The operator requeues any unfinished work, and the reconcile loop compares states with a set period of time between each reconcile iteration.
Online upgrade
An online upgrade allows you to load data with minimal downtime, keeping the database active with continuous writes though replication. By leveraging sandboxes, instead of shutting down the primary subclusters and limiting secondary subclusters to read-only mode, you can sandbox a secondary subcluster. This allows ongoing read and write access to the database while the primary subcluster is being upgraded.
Note
- Vertica only supports versions v24.3.0-2 or higher for online upgrades. If the version is lower, it will default to a read-only online upgrade.
- The local catalog directory, including all logs in the old database, will be removed. We recommend that you backup any important files before proceeding with the upgrade.
- The database must not have any sandboxes before performing an online upgrade.
- If incorrect images are detected, an error will be displayed. You must update the image tag in the VerticaDB to proceed with online upgrade.
- All user connections, except JDBC connections, will be terminated prior to database replication during an online upgrade.
Online upgrade workflow
The following outlines the workflow during an online upgrade:
- Enable no-ddl mode: This mode restricts certain actions, such as creating new users or views. You can only insert data into existing tables or create new tables.
- Create a sandbox: The operator creates a new sandbox that replicates the main cluster. This requires additional resources temporarily.
In the following example
vertica-db-sc1
is the original cluster whilevertica-db-sc1-sb
is the sandboxed copy. - Upgrade the sandbox: The sandbox is upgraded in
Offline
mode. - Replicate data and redirect connections: Changes are synchronized by replicating data from the main cluster to the sandbox and connections are redirected to the sandbox environment.
- Promote the sandbox: The sandbox is now promoted to the main cluster.
- Remove the old cluster: After redirect is complete, the old cluster is removed. The new StatefulSet name and Vertica node names will differ from those of the old cluster.
Client session transfer
Important
This is a beta feature.During an online upgrade, the operator pauses write operations to replicate data from the main cluster to the sandbox. After data replication is complete, client sessions are transferred from the existing Vertica version on the main cluster to a sandboxed subcluster on the upgraded Vertica version.
Note
- Session transfer is supported in Vertica version 24.3.0-4 and later, except for version 24.4.0-0.
- When using client proxy (by setting
vertica.com/use-client-proxy
totrue
), session transfers are supported for all types of Vertica clients. Without client proxy, session transfers are limited to JDBC connections only. - Session transfers may result in the loss of temporary tables.
- Clients external to the Kubernetes cluster must connect through a load balancer or NodePort (load balancer or NodePort can be set in subclusters[i].serviceType) for connections to be redirected to the upgraded database.
- Connections running queries that exceed the timeout value (default 300 seconds) will be closed instead of being paused or redirected.
- SQL UI tools such as DBeaver do not receive notifications for paused connections and may hang during the pause.
- Connections without transaction commits will be closed as they cannot be paused.
- When the old cluster is removed, connections will be briefly disrupted as they disconnect from the old cluster and reconnect to the new cluster.
Routing client traffic during a ReadOnlyOnline upgrade
During a read-only online upgrade, the operator begins by upgrading the Vertica server version in the primary subcluster to form a cluster with the new version. When the operator restarts the primary nodes, it places the secondary subclusters in read-only mode. Next, the operator upgrades any secondary subclusters one at a time. During the upgrade for any subcluster, all client connections are drained, and traffic is rerouted to either an existing subcluster or a temporary subcluster.
Read-only online upgrades require more than one subcluster so that the operator can reroute client traffic for the subcluster while it is upgrading. By default, the operator selects which subcluster receives the rerouted traffic using the following rules:
-
When rerouting traffic for the primary subcluster, the operator selects the first secondary subcluster defined in the CR.
-
When restarting the first secondary subcluster after the upgrade, the operator selects the first subcluster that is defined in the CR that is up.
-
If no secondary subclusters exist, you cannot perform a read-only online upgrade. The operator selects the first primary subcluster defined in the CR and performs an offline upgrade.
Route to an existing subcluster
You might want to control which subclusters handle rerouted client traffic due to subcluster capacity or licensing limitations. You can set the temporarySubclusterRouting.names parameter to specify an existing subcluster to receive the rerouted traffic:
In the previous example, subcluster-2
accepts traffic when the other subcluster-1
is offline. When subcluster-2
is down, subcluster-1
accepts its traffic.
Route to a temporary subcluster
To create a temporary subcluster that exists for the duration of the upgrade process, use the temporarySubclusterRouting.template parameter to provide a name and size for the temporary subcluster:
If you choose to upgrade with a temporary subcluster, ensure that you have the necessary resources.
Migrating deployment types
Note
For Online upgrades, the image version must be v24.3.0-2 or higher.Beginning with Vertica server version 24.1.0, the operator manages deployments with vclusterops
, a Go library that uses a high-level REST interface to perform database operations with the Node Management Agent (NMA) and HTTPS service. The vclusterops
library replaces Administration tools (admintools), a traditional command-line interface that executes administrator commands through STDIN and required SSH keys for internal node communications. The vclusterops
deployment is more efficient in containerized environments than the admintools deployment.
Because version 24.1.0 does not include admintools, you must migrate to the vcluster deployment type when you upgrade from an earlier server version.
Migrate the VerticaDB CR
Before you can migrate deployment types, you must upgrade the VerticaDB operator to version 2.0.0.
Important
The following requirements and restrictions apply to deployment type migrations:
- You can only migrate a custom resource from admintools to vclusterops. You cannot migrate a vclusterops deployment to admintools.
- All migrations require that you use the
Offline
upgrade policy. If you try to migrate deployments with aReadOnlyOnline
upgrade, the operator transitions to anOffline
upgrade automatically. Migrating deployments with anOnline
upgrade will result in a failure.
To migrate deployment types, update the manifest and apply it:
-
Update the manifest to a vcluster deployment. The following sample manifest includes all fields that are required to migrate to a
vclusterops
deployment:This manifest sets the following parameters:
apiVersion
: By default,v1
supports vcluster deployments. Deprecated API versionv1beta1
also supports vcluster, but Vertica recommends that you change tov1
.vertica.com/vcluster-ops
: Set totrue
. With API versionv1
, this field and setting are optional. If you use the deprecatedv1beta1
, this setting is required or the migration fails.vertica.com/run-nma-in-sidecar
: You must set this tofalse
for vcluster deployments. For additional details, see VerticaDB custom resource definition.spec.image
: Set this to a 24.1.0 image version. For a list images, see Vertica images.
-
Apply the updated manifest to complete the migration:
Upgrade the Vertica server version
Important
If you are upgrading to 24.1.0 from version 23.4.0 or earlier, you must migrate deployment types.After you select your upgrade policy, use the kubectl command line tool to perform the upgrade and monitor its progress. The following steps demonstrate an online upgrade:
-
Set the upgrade policy to
Online
: -
Update the image setting in the CR:
-
Use
kubectl wait
to wait until the operator leaves upgrade mode:
View the upgrade process
To view the current phase of the upgrade process, use kubectl get
to inspect the upgradeStatus
status field:
To view the entire upgrade process, use kubectl describe
to list the events the operator generated during the upgrade: