Create an Eon Mode database on-premises with Ceph

Vertica connects to Ceph using the Ceph Object Gateway (RGW), which exposes an S3-compatible REST interface. All Vertica nodes must be able to reach the RGW endpoint over the network.

Step 1: Configure Ceph RGW and create credentials

Ensure that you have the following before you create your Eon Mode database:

• A running Ceph cluster with at least one radosgw RGW daemon deployed. The Ceph cluster must be accessible from all Vertica nodes.

• The IP address and port of the RGW endpoint. The default RGW ports are 9000 and 9001.

• A Ceph RGW user account with an access key and secret key that has read and write access to the communal storage bucket.

• A bucket on the Ceph cluster to use for communal storage.

For Ceph cluster installation and RGW configuration instructions, see the Ceph documentation.

Create an RGW user

Use the radosgw-admin tool to create a user account for Vertica. Replace ceph-user, access-key, and secret-key with your chosen values:

radosgw-admin user create --uid=ceph-user --display-name=ceph-user \
  --access-key=access-key --secret-key=secret-key

Create the communal storage bucket

Use the AWS CLI to create the bucket.

aws --endpoint=http://RGW_IP:RGW_Port s3 mb s3://BUCKET_NAME --region us-east-1

To remove a bucket and all its contents, use radosgw-admin:

radosgw-admin bucket rm --bucket=BUCKET_NAME --purge-objects --bypass-gc

For information about RGW performance tuning, see the Ceph configuration recommendations section below.

Step 2: install Vertica on your cluster

To install Vertica:

1. Ensure your nodes are configured properly by reviewing all of the content in the Before you install Vertica section.

2. Use the install_vertica script to verify that your nodes are correctly configured and to install the Vertica binaries on all of your nodes. Follow the steps under Install Vertica using the command line to install Vertica.

   Note

   These installation steps are the same ones you follow to install Vertica in Enterprise Mode. The difference between Eon Mode and Enterprise Mode on-premises databases is how you create the database, not how you install the Vertica software.

Step 3: Create an authorization file

Prior to creating your Eon Mode on-premises database, you must create an authorization file that admintools can use to authenticate with the Ceph RGW endpoint.

1. On the node where you will run admintools to create your database, use a text editor to create a file named auth_params.conf. The location of this file must be readable by the Linux user you use to create the database (typically dbadmin).

   Important

   The auth_params.conf file contains the secret key to access the bucket containing your Eon Mode database's data. This information is sensitive, and can be used to access the raw data in your database. Be sure this file is not readable by unauthorized users. After you have created your database, you can delete this file.

2. Add the following lines to the file:

   awsauth = Ceph_Access_Key:Ceph_Secret_Key
   awsendpoint = RGW_IP:RGW_Port
   awsregion = us-east-1
   

   Note

   You do not need to supply a port number in the awsendpoint setting if you configured your RGW to use the default HTTP ports (80 for an unencrypted connection or 443 for an encrypted connection). The default RGW port is 9000.

3. If you are not using TLS encryption for the connection between Vertica and Ceph RGW, add the following line to the file:

   awsenablehttps = 0
   

4. Save the file and exit the editor.

The following example auth_params.conf file is for an unencrypted connection to a Ceph RGW endpoint at IP address 10.20.30.40 using port 9000.

awsauth = PIWHSNDGSHVRPIQ:339068001+e904816E02E5fe9103f8MQOEAEHFFVPKBAAL
awsendpoint = 10.20.30.40:9000
awsenablehttps = 0
awsregion = us-east-1

Step 4: choose a depot path on all nodes

Choose or create a directory on each node for the depot storage path. The directory you supply for the depot storage path parameter must:

• Have the same path on all nodes in the cluster (i.e. /home/dbadmin/depot).

• Be readable and writable by the dbadmin user.

• Have sufficient storage. By default, Vertica uses 60% of the filesystem space containing the directory for depot storage. You can limit the size of the depot by using the --depot-size argument in the create_db command. See Configuring your Vertica cluster for Eon Mode for guidelines on choosing a size for your depot.

The admintools create_db tool will attempt to create the depot path for you if it doesn't exist.

Step 5: Create the Eon on-premises database

Use the admintools create_db tool to create the database. You must pass this tool the following arguments:

To view all arguments for the create_db tool, run the command:

admintools -t create_db --help

The following example demonstrates creating a three-node database named verticadb, with depot stored in the dbadmin home directory:

$ admintools -t create_db -x auth_params.conf \
  --communal-storage-location=s3://verticadbbucket \
  --depot-path=/home/dbadmin/depot --shard-count=6 \
  -s vnode01,vnode02,vnode03 -d verticadb -p 'YourPasswordHere'

Alternative: Use vcluster

You can also create the database using vcluster create_db, passing the Ceph connection parameters directly with --config-param rather than using an auth_params.conf file:

vcluster create_db \
  --hosts host1,host2,host3 \
  --db-name dbname \
  --data-path /path/to/data \
  --catalog-path /path/to/catalog \
  --shard-count 6 \
  --depot-path /path/to/depot \
  --depot-size 50G \
  --communal-storage-location s3://bucketname \
  --config-param AWSAuth=access-key:secret-key,awsendpoint=RGW_IP:RGW_Port,awsenablehttps=0,awsregion=us-east-1

Step 6: Disable streaming limitations

After creating the database, disable the AWSStreamingConnectionPercentage configuration parameter. This setting is unnecessary for an Eon Mode on-premises install with communal storage on Ceph. Set it to 0 to disable it:

=> ALTER DATABASE DEFAULT SET PARAMETER AWSStreamingConnectionPercentage = 0;
ALTER DATABASE

Ceph configuration recommendations

The following Ceph-side configuration settings were used in validated testing environments and are provided as a starting point. Adjust these values based on your hardware and workload.

RGW thread pool size

You can increase the RGW thread pool size for workloads with high concurrency using the following configuration:

ceph config set client.rgw rgw_thread_pool_size 4096

Replication factor and consistency

The validated pool configuration uses a replication factor of 3 (size 3, min_size 2). With this configuration, Ceph continues accepting writes as long as at least 2 replicas are available. If only 1 replica is available, the pool becomes read-only.

Ceph provides strong consistency: writes are only acknowledged after all replica copies are written, guaranteeing read-after-write consistency. Background deep-scrub operations detect and repair bit rot or corruption across replicas.

Placement groups

Red Hat recommends 100 to 200 placement groups (PGs) per OSD. Use values that are powers of two to avoid POOL_PG_NUM_NOT_POWER_OF_TWO health warnings.

Network

Provision at least 10 Gb/s networking between Vertica nodes and Ceph nodes.

Troubleshooting

Multipart upload error (ERROR 6966)

Symptom: Queries or data loads fail with:

ERROR 6966: StorageBundleWriter::close:: error closing writer: This multipart completion is already in progress

Resolution: Check the health of your Ceph cluster with ceph health detail and allow any active recovery operations to complete before retrying. This error is commonly associated with OSD daemons that are under load or recovering.

If the error persists on a healthy cluster, set AWSStreamingConnectionPercentage to 75:

=> ALTER DATABASE DEFAULT SET PARAMETER AWSStreamingConnectionPercentage = 75;
ALTER DATABASE

Bucket deletion and Ceph recovery

If you accidentally delete objects or a bucket using the AWS CLI, Ceph will enter a background recovery state. This is reflected in slow OSD operations (osd_op_complaint_time exceeded, CephDaemonSlowOps health warnings). The cluster recovers automatically, but performance is degraded until recovery is complete.

To prevent this, always use radosgw-admin for bucket and object management operations outside of normal Vertica database activity.