Troubleshooting Kerberos authentication

These tips can help you avoid issues related to Kerberos authentication with Vertica and to troubleshoot any problems that occur.

JDBC client authentication fails

If Kerberos authentication fails on a JDBC client, check the JAAS login configuration file for syntax issues. If syntax is incorrect, authentication fails.

Working domain name service (DNS) not configured

Verify that the DNS entries and the system host file (/etc/hosts or /etc/hostnames) on the network are all properly configured for your environment. If you are using a fully qualified domain name, ensure that is properly configured as well. Refer to the Kerberos documentation for your platform for details.

System clocks out of sync

System clocks in your network must remain in sync for Kerberos authentication to work properly. If you access data in HDFS, then Vertica nodes must also be in sync with Hadoop.

All systems except red hat 7/CentOS 7

To keep system clocks in sync:

Install NTP on the Kerberos server (KDC).
Install NTP on each server in your network.
Synchronize system clocks on all machines that participate in the Kerberos realm within a few minutes of the KDC and each other

Clock skew can be a problem on Linux virtual machines that need to sync with the Windows Time Service. Use the following stepsto keep time in sync:

Using any text editor, open /etc/ntp.conf.
Under the Undisciplined Local Clock section, add the IP address for the Vertica server. Then, remove existing server entries.
Log in to the server as root, and set up a cron job to sync time with the added IP address every half hour, or as often as needed. For example:
```
# 0 */2 * * * /etc/init.d/ntpd restart
```
Alternatively, run the following command to force clock sync immediately:
```
$ sudo /etc/init.d/ntpd restart
```

For more information, see Enabling network time protocol (NTP) and the Network Time Protocol website.

Red hat 7/CentOS 7 systems

In Red Hat 7/CentOS 7, ntpd is deprecated in favor of chrony. To keep system clocks in your network in sync for Kerberos authentication to work properly, do the following:

Install chrony on the Kerberos server (KDC).
Install chrony on each server in your network.
Synchronize system clocks on all machines that participate in the Kerberos realm within a few minutes of the KDC and each other.

Clock skew on Linux virtual machines

Clock skew can be problematic on Linux virtual machines that need to sync with the Windows Time Service. Try the following to keep time in sync:

Using any text editor, open /etc/chrony.conf.
Under the Undisciplined Local Clock section, add the IP address for the Vertica server. Then, remove existing server entries.
Log in to the server as root, and set up a cron job to sync time with the added IP address every half hour, or as often as needed. For example:
```
# 0 */2 * * * systemctl start chronyd
```
Alternatively, run the following command to force clock sync immediately:
```
$ sudo systemctl start chronyd
```

For more information, see the Red Hat chrony guide.

Kerberos ticket is valid, but Hadoop access fails

Vertica uses Kerberos tickets to obtain Hadoop tokens. It then uses the Hadoop tokens to access the Hadoop data. Hadoop tokens expire after a period of time, so Vertica periodically refreshes them. However, if your Hadoop cluster is set to expire tokens frequently, it is possible that tokens might not be refreshed in time. If the token expires, you cannot access data.

Setting the HadoopFSTokenRefreshFrequency configuration parameter allows you to specify how often Vertica should refresh the token. Specify this value, in seconds, to be smaller than the expiration period set for Hadoop. For example:

=> ALTER DATABASE exampledb SET HadoopFSTokenRefreshFrequency = '86400';

For another cause of Hadoop access failure, see System Clocks Out of Sync.

Encryption algorithm choices

Kerberos is based on symmetric encryption. Be sure that all Kerberos parties used in the Kerberos realm agree on the encryption algorithm to use. If they do not agree, authentication fails. You can review the exceptions in the vertica.log.

On a Windows client, be sure the encryption types match the types set on Active Directory. See Configure Vertica for Kerberos authentication.

Be aware that Kerberos is used only for securing the login process. After the login process completes, by default, information travels between client and server without encryption. If you want to encrypt traffic, use SSL. For details, see Implementing SSL.

Kerberos passwords not recognized

If you change your Kerberos password, you must re-create all of your keytab files.

Using the ODBC data source configuration utility

On Windows vsql clients, you may choose to use the ODBC Data Source Configuration utility and supply a client Data Source. If so, be sure you enter a Kerberos host name in the Client Settings tab to avoid client connection failures with the Vertica Analytic Database server.

Authentication failure in backup, restore, or admin tools

This problem can arise in configurations where each Vertica node uses its own Kerberos principal. (This configuration is recommended.) When using vbr or admintools you might see an error such as the following:

$ vsql: GSSAPI continuation error: Miscellaenous failure
GSSAPI continuation error: Server not found in Kerberos database

Backup/restore and the admin tools use the value of KerberosHostname, if it is set, in the Kerberos principal used to authenticate. The same value is used on all nodes. If you have defined one Kerberos principal per node, as recommended, this value does not match. To correct this, unset the KerberosHostname parameter:

=> ALTER DATABASE DEFAULT CLEAR KerberosHostname;

Server's principal name does not match host name

This problem can arise in configurations where a single Kerberos principal is used for all nodes. Vertica recommends against using a single Kerberos principal for all nodes. Instead, use one principal per node and do not set the KerberosHostname parameter.

In some cases during client connection, the Vertica server's principal name might not match the host name in the connection string. (See also Using the ODBC Data Source Configuration Utility in this topic.)

On ODBC, JDBC, and ADO.NET clients, set the host name portion of the server's principal using the KerberosHostName connection string.

Tip

On vsql clients, you set the host name portion of the server's principal name using the -K KRB HOST command-line option. The default value is specified by the -h switch, which is the host name of the machine on which the Vertica server is running. -K is equivalent to the drivers' KerberosHostName connection string value.

For details, see Command Line Options.

Principal/host mismatch issues and resolutions

The following issues can occur if the principal and host are mismatched.

The KerberosHostName configuration parameter has been overridden

For example, consider the following connection string:

jdbc:vertica://v_vmart_node0001.example.com/vmart?user=kuser

Because the this connection string includes no explicit KerberosHostName parameter, the driver defaults to the host in the URL (v_vmart_node0001.example.com). If you overwrite the server-side KerberosHostName parameter as “abc”, the client generates an incorrect principal.

To resolve this issue, explicitly set the client’s KerberosHostName to the connection string, as in this example:

jdbc:vertica://v_vmart_node0001.example.com/vmart?user=kuser&kerberoshostname=abc

Connection load balancing is enabled... but the node against which the client authenticates might not be the node in the connection string.

In this situation, consider changing all nodes to use the same KerberosHostName setting. When you use the default to the host that was originally specified in the connection string, load balancing cannot interfere with Kerberos authentication.

A DNS name does not match the Kerberos host name

For example, imagine a cluster of six servers, where you want hr-servers and finance-servers to connect to different nodes on the Vertica cluster. Kerberos authentication, however, occurs on a single (the same) KDC. In the following example, the Kerberos service host name of the servers is server.example.com.

Suppose you have the following list of example servers:

server1.example.com 192.16.10.11
server2.example.com 192.16.10.12
server3.example.com 192.16.10.13
server4.example.com 192.16.10.14
server5.example.com 192.16.10.15
server6.example.com 192.16.10.16

Now, assume you have the following DNS entries:

finance-servers.example.com 192.168.10.11, 192.168.10.12, 192.168.10.13
hr-servers.example.com 192.168.10.14, 192.168.10.15, 192.168.10.16

When you connect to finance-servers.example.com, specify:

Kerberos -h host name option as server.example.com
-K host option for hr-servers.example.com

For example:

$ vsql -h finance-servers.example.com -K server.example.com

No DNS is set up on the client machine... so you must connect by IP only

To resolve this issue, specify:

Kerberos -h host name option for the IP address
-K host option for server.example.com

For example:

$ vsql -h 192.168.1.12 -K server.example.com

There is a load balancer involved (Virtual IP)... but there is no DNS name for the VIP

Specify:

Kerberos -h host name option for the Virtual IP address
-K host option for server.example.com

For example:

$ vsql -h <virtual IP> -K server.example.com

You connect to Vertica using an IP address... but there is no host name to construct the Kerberos principal name.

Provide the instance or host name for the Vertica as described in Inform Vertica about the Kerberos principal

The server-side KerberosHostName configuration parameter is set to a name other than the Vertica node's host name... but the client cannot determine the host name based on the host name in the connection string alone.

Reset KerberosHostName to match the name of the Vertica node's host name. For more information, see the following topics: