Sandboxing on K8s

Sandboxing on Kubernetes allows you to create isolated testing environments without the need to set up a new database or reload data, making it easier to test Vertica features in new versions. Sandboxing enables seamless online upgrades in Kubernetes. While users stay connected to the main cluster, the upgrade is performed on the sandbox. Once the upgrade is complete, the sandbox is promoted to the main cluster. The operator automates the sandboxing process for Vertica subclusters within a custom resource (CR). For more information, see Subcluster sandboxing.

Prerequisites

Install the VerticaDB operator
Create a VerticaDB custom resource definition manifest

Sandboxing a Subcluster

Note

You can only sandbox secondary subclusters. If your existing cluster does not have a secondary subcluster, then you can scale your subclusters to add one. See, Scaling subclusters.
Only existing subclusters can be added to a sandbox. Any subclusters created after the sandbox is established cannot be added to it.

The following specification in VerticaDB CR (VerticaDB custom resource definition) has the sandbox information:

Parameter	Description
spec.sandboxes[i].name	Name of the sandbox.
spec.sandboxes[i].subclusters[i].name	Name of the secondary subcluster to be added to the sandbox. The sandbox must include at least one secondary subcluster.
spec.sandboxes[i].image	Name of the image to use for the sandbox. If omitted, image from the main cluster will be used. Changing this will force an upgrade for the sandbox where it is defined.
spec.sandboxes[i].shutdown	Indicates that the sandbox must remain in a shutdown state. When set to `true`, the sandbox will be stopped (if it is running). The operator will stop all the subclusters in the sandbox using draining shutdown and will not attempt to restart it.

To sandbox a subcluster

Use kubectl edit to open your default text editor and update the YAML file for the specified custom resource. The following command opens a custom resource named vdb for editing:
```
$ kubectl edit vdb
```

In the spec section of the custom resource, locate the subclusters subsection and identify the secondary subcluster that you want to sandbox. In the following example, we will sandbox the secondary subcluster, sc2:

apiVersion: vertica.com/v1
kind: VerticaDB
metadata:
 name: vertica-db
spec:
...
 subclusters:
 - affinity: {}
   name: sc1
   resources: {}
   serviceName: sc1
   serviceType: ClusterIP
   size: 3
   type: primary
 - affinity: {}
   name: sc2
   resources: {}
   serviceName: sc2
   serviceType: ClusterIP
   size: 1
   type: secondary
 - affinity: {}
   name: sc3
   resources: {}
   serviceName: sc3
   serviceType: ClusterIP
   size: 1
   type: secondary

Now, add an entry for the sandbox. Provide a sandbox name and name of the subcluster(s) that you want to sandbox. For example, we will sandbox subcluster sc2 and name it sandbox1:

Note

The first subcluster added to the sandbox becomes the primary subcluster, any subsequent subclusters are designated as secondary. If multiple subclusters are added to the sandbox simultaneously, K8s operator selects the first subcluster in the list as the primary subcluster.

spec:
...
 subclusters:
 - affinity: {}
   name: sc1
   resources: {}
   serviceName: sc1
   serviceType: ClusterIP
   size: 3
   type: primary
 - affinity: {}
   name: sc2
   resources: {}
   serviceName: sc2
   serviceType: ClusterIP
   size: 3
   type: secondary
 - affinity: {}
   name: sc3
   resources: {}
   serviceName: sc3
   serviceType: ClusterIP
   size: 3
   type: secondary
 sandboxes:
 - name: sandbox1
   subclusters:
    - name: sc2

Save and close the custom resource file. When the update completes, you will receive the following message:
```
verticadb.vertica.com/vertica-db edited
```

If you want to include another subcluster in the sandbox, go back to VerticaDB and modify the sandbox information. Following are the contents of the VerticaDB after adding sc3:

spec:
...
  subclusters:
  - affinity: {}
    name: sc1
    resources: {}
    serviceName: sc1
    serviceType: ClusterIP
    size: 3
    type: primary
  - affinity: {}
    name: sc2
    resources: {}
    serviceName: sc2
    serviceType: ClusterIP
    size: 3
    type: secondary
  - affinity: {}
    name: sc3
    resources: {}
    serviceName: sc3
    serviceType: ClusterIP
    size: 3
    type: secondary
  sandboxes:
  - name: sandbox1
	image: opentext/vertica-k8s:24.3.0-0
    subclusters:
      - name: sc2
	  - name: sc3

Checking sandboxing status

You can check the status of sandboxing as follows:

$ kubectl describe vdb
Name:         vertica-db
...
Events:
  Type     Reason                      Age                From                Message
  ----     ------                      ----               ----                -------
  ...
  Normal   SandboxSubclusterStart      8m1s               verticadb-operator  Starting add subcluster "sc2" to sandbox "sandbox1"
  Normal   SandboxSubclusterSucceeded  7m39s              verticadb-operator  Successfully added subcluster "sc2" to sandbox "sandbox1"

You can verify if sandboxing is successful by checking the VerticaDB CR to see if the subcluster type changed from secondary to sandboxprimary.

spec:
...
  subclusters:
  - affinity: {}
    name: sc1
    resources: {}
    serviceName: sc1
    serviceType: ClusterIP
    size: 3
    type: primary
  - affinity: {}
    name: sc2
    resources: {}
    serviceName: sc2
    serviceType: ClusterIP
    size: 3
    type: sandboxprimary
  - affinity: {}
    name: sc3
    resources: {}
    serviceName: sc3
    serviceType: ClusterIP
    size: 3
    type: secondary
  sandboxes:
  - name: sandbox1
	image: opentext/vertica-k8s:24.3.0-0
    subclusters:
      - name: sc2
	  - name: sc3

Alternatively, you can connect to any node in the subcluster using vsql client and query the system table subclusters to check that sandboxing is successful.

$ vsql -h 10.244.2.166 -U dbadmin
Welcome to vsql, the Vertica Analytic Database interactive terminal.

Type:  \h or \? for help with vsql commands
       \g or terminate with semicolon to execute query
       \q to quit

SSL connection (cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, protocol: TLSv1.2)

select distinct subcluster_name, is_primary, sandbox from subclusters where subcluster_name = 'sc2';
 subcluster_name | is_primary | sandbox
-----------------+------------+----------
 sc2             | t          | sandbox1
(1 row)

Upgrading the subcluster

To upgrade the sandbox, simply update the spec.sandboxes.image field.

Note

Sandboxes can run different Vertica versions. For example, sandbox1 uses version 24.3.0-0, while sandbox2 operates on version 24.3.0-1.

spec:
...
  sandboxes:
  - image: opentext/vertica-k8s:24.3.0-1
    name: sandbox1
    subclusters:
    - name: sc2
    - name: sc3

Stopping and shutting down a sandbox

You can gracefully shut down the sandbox and the nodes where the sandbox was running. We recommend running sandboxes on separate nodes or node groups. This allows you to shut down the nodes or node groups when the sandboxes are stopped, helping to reduce costs.

A sandbox can remain in a shutdown state for as long as required.

In the following example, sandbox1 will be stopped and remain in the shutdown state as long as shutdown is set to true.

spec:
...
  sandboxes:
  - image: opentext/vertica-k8s:24.3.0-1
    name: sandbox1
    shutdown: true
    subclusters:
    - name: sc2

All pods in the subclusters within the sandbox will be deleted and will not be recreated until shutdown is set to false.

Checking sandbox status

You can check if the sandbox was stopped as follows:

$ kubectl describe vdb
Name:         vertica-db
...
Events:
  Type    Reason           Age   From                Message
  ----    ------           ----  ----                -------
  Normal  StopDBStart      24s   verticadb-operator  Starting stop database
  Normal  StopDBSucceeded  22s   verticadb-operator  Successfully stopped the database.  It took 2s

Note that spec.subclusters[].shutdown and status.subclusters[].shutdown are set to true for all subclusters of a sandbox that has been shut down.

$ kubectl describe vdb
Name:         vertica-db
...
spec:
...
  subclusters:
  ...
    name: sc2
    shutdown: true
...
status:
  ...
  subclusters:
    ...
    name:           sc2
    oid:            54043195528448686
    shutdown:       true
    UpNodeCount:    0

You can start the sandbox again by setting shutdown to false.

spec:
...
  sandboxes:
  - image: opentext/vertica-k8s:24.3.0-1
    name: sandbox1
    shutdown: false
    subclusters:
    - name: sc2

Note

Ensure shutdown is not set to true when adding a new subcluster or sandbox to a vdb.
A subcluster cannot be unsandboxed if shutdown is true or part of a sandbox with shutdown set to true.
image cannot be changed if the sandbox or any of its subclusters have shutdown set to true.

Removing sandboxes

Removing sandboxes allows you to remove a subcluster from the sandbox and return it to the main cluster.

Note

You must remove secondary subclusters from a sandbox before removing the sandboxprimary subcluster.
To remove a sandboxed subcluster from the database, make sure to unsandbox the subcluster first. Removing a sandboxed subcluster without unsandboxing it will cause a failure. You can remove the complete sandbox and all its subclusters (from the spec) at the same time.
A sandbox cannot be removed if shutdown is set to true.

To remove a subcluster from the sandbox, you need to remove the subcluster name from spec.sandboxes.subclusters in VerticaDB. In the following example, subcluster sc3 will be removed from the sandbox:

spec:
...
  sandboxes:
  - image: opentext/vertica-k8s:24.3.0-1
    name: sandbox1
    subclusters:
    - name: sc2

To remove the complete sandbox, remove its information from the VerticaDB:

spec:
...
  sandboxes:[]

Checking unsandboxing status

You can check if the sandbox was removed as follows:

$ kubectl describe vdb
Name:         vertica-db
...
Events:
  Type    Reason                        Age   From                Message
  ----    ------                        ----  ----                -------
  Normal  UnsandboxSubclusterStart      2m3s  verticadb-operator  Starting unsandbox subcluster "sc2"
  Normal  UnsandboxSubclusterSucceeded  111s  verticadb-operator  Successfully unsandboxed subcluster "sc2"
  Normal  UnsandboxSubclusterStart      111s  verticadb-operator  Starting unsandbox subcluster "sc3"
  Normal  UnsandboxSubclusterSucceeded  99s   verticadb-operator  Successfully unsandboxed subcluster "sc3"
  Normal  NodeRestartStarted            90s   verticadb-operator  Starting database restart node of the following pods: vertica-db-sc2-0, vertica-db-sc3-0
  Normal  NodeRestartSucceeded          65s   verticadb-operator  Successfully restarted database nodes and it took 24s

You can verify if the sandbox was removed successfully by opening the VerticaDB CR and checking that subcluster type has changed from sandboxprimary to secondary.

spec:
...
  subclusters:
  - affinity: {}
    name: sc1
    resources: {}
    serviceName: sc1
    serviceType: ClusterIP
    size: 3
    type: primary
  - affinity: {}
    name: sc2
    resources: {}
    serviceName: sc2
    serviceType: ClusterIP
    size: 3
    type: secondary
  - affinity: {}
    name: sc3
    resources: {}
    serviceName: sc3
    serviceType: ClusterIP
    size: 3
    type: secondary

Alternatively, you can connect to any node in the subcluster using the vsql client and query the subclusters system table to verify if unsandboxing was successful.

$ vsql -h 10.244.2.166 -U dbadmin
Welcome to vsql, the Vertica Analytic Database interactive terminal.

Type:  \h or \? for help with vsql commands
       \g or terminate with semicolon to execute query
       \q to quit

SSL connection (cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, protocol: TLSv1.2)

vertdb=> select distinct subcluster_name, is_primary, sandbox from subclusters where subcluster_name = 'sc2';
 subcluster_name | is_primary | sandbox
-----------------+------------+---------
 sc2             | f          |
(1 row)