OpenText Analytics Database 26.2.x – User-defined extensions

Extending: Loading UDxs

Mon, 01 Jan 0001 00:00:00 +0000

User-defined extensions (UDxs) are contained in libraries. A library can contain multiple UDxs. To add UDxs to OpenText™ Analytics Database, you must:

Deploy the library (once per library).
Create each UDx (once per UDx).

If you are using UDxs written in Java, you must also set up a Java runtime environment. See Installing Java on OpenText Analytics Database hosts.

Deploying libraries

To deploy a library to your database:

Copy the UDx shared library file (.so), Python file, Java JAR file, or R functions file that contains your function to a node on your database cluster. You do not need to copy it to every node.
Connect to the node where you copied the library (for example, using vsql).
Add your library to the database catalog using the CREATE LIBRARY statement.
```
=> CREATE LIBRARY libname AS '/path_to_lib/filename'
   LANGUAGE 'language';
```
libname is the name you want to use to reference the library. path_to_lib/filename is the fully-qualified path to the library or JAR file you copied to the host. language is the implementation language.

For example, if you created a JAR file named TokenizeStringLib.jar and copied it to the dbadmin account's home directory, you would use this command to load the library:
```
=> CREATE LIBRARY tokenizelib AS '/home/dbadmin/TokenizeStringLib.jar'
   LANGUAGE 'Java';
```

You can load any number of libraries into the database.

Privileges

Superusers can create, modify, and drop any library. Users with the UDXDEVELOPER role or explicit grants can also act on libraries, as shown in the following table:

Operation	Requires
CREATE LIBRARY	UDXDEVELOPER
Replace a library (CREATE OR REPLACE LIBRARY)	UDXDEVELOPER and one of: owner of library being replaced DROP privilege on the target library
DROP LIBRARY	UDXDEVELOPER and one of: owner of library being dropped DROP privilege on the target library
ALTER LIBRARY	UDXDEVELOPER and owner

Creating UDx functions

After the library is loaded, define individual UDxs using SQL statements such as CREATE FUNCTION and CREATE SOURCE. These statements assign SQL function names to the extension classes in the library. They add the UDx to the database catalog and remain available after a database restart.

The statement you use depends on the type of UDx you are declaring, as shown in the following table:

UDx Type	SQL Statement
Aggregate Function (UDAF)	CREATE AGGREGATE FUNCTION
Analytic Function (UDAnF)	CREATE ANALYTIC FUNCTION
Scalar Function (UDSF)	CREATE FUNCTION (scalar)
Transform Function (UDTF)	CREATE TRANSFORM FUNCTION
Load (UDL): Source	CREATE SOURCE
Load (UDL): Filter	CREATE FILTER
Load (UDL): Parser	CREATE PARSER

If a UDx of the given name already exists, you can replace it or instruct the database to not replace it. To replace it, use the OR REPLACE syntax, as in the following example:


=> CREATE OR REPLACE TRANSFORM FUNCTION tokenize
   AS LANGUAGE 'C++' NAME 'TokenFactory' LIBRARY TransformFunctions;
CREATE TRANSFORM FUNCTION

You might want to replace an existing function to change between fenced and unfenced modes.

Alternatively, you can use IF NOT EXISTS to prevent the function from being created again if it already exists. You might want to use this in upgrade or test scripts that require, and therefore load, UDxs. By using IF NOT EXISTS, you preserve the original definition including fenced status. The following example shows this syntax:

--- original creation:
=> CREATE TRANSFORM FUNCTION tokenize
   AS LANGUAGE 'C++' NAME 'TokenFactory' LIBRARY TransformFunctions NOT FENCED;
CREATE TRANSFORM FUNCTION

--- function is not replaced (and is still unfenced):
=> CREATE TRANSFORM FUNCTION IF NOT EXISTS tokenize
   AS LANGUAGE 'C++' NAME 'TokenFactory' LIBRARY TransformFunctions FENCED;
CREATE TRANSFORM FUNCTION

After you add the UDx to the database, you can use your extension within SQL statements. The database superuser can grant access privileges to the UDx for users. See GRANT (user defined extension) for details.

When you call a UDx, the database creates an instance of the UDx class on each node in the cluster and provides it with the data it needs to process.

Extending: Installing Java on OpenText Analytics Database hosts

Mon, 01 Jan 0001 00:00:00 +0000

If you are using UDxs written in Java, follow the instructions in this section.

You must install a Java Virtual Machine (JVM) on every host in your cluster in order for OpenText™ Analytics Database to be able to execute your Java UDxs.

Installing Java on your Vertica cluster is a two-step process:

Install a Java runtime on all of the hosts in your cluster.
Set the JavaBinaryForUDx configuration parameter to tell Vertica the location of the Java executable.

Installing a Java runtime

For Java-based features, Vertica requires a 64-bit Java 6 (Java version 1.6). Vertica supports runtimes from either Oracle or OpenJDK. You can choose to install either the Java Runtime Environment (JRE) or Java Development Kit (JDK), since the JDK also includes the JRE.

Many Linux distributions include a package for the OpenJDK runtime. See your Linux distribution's documentation for information about installing and configuring OpenJDK.

To install the Oracle Java runtime, see the Java Standard Edition (SE) Download Page. You usually run the installation package as root in order to install it. See the download page for instructions.

Once you have installed a JVM on each host, ensure that the java command is in the search path and calls the correct JVM by running the command:

$ java -version

This command should print something similar to:

java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)

Note

Any previously installed Java VM on your hosts may interfere with a newly installed Java runtime. See your Linux distribution's documentation for instructions on configuring which JVM is the default. Unless absolutely required, you should uninstall any incompatible version of Java before installing the Java 6 or Java 7 runtime.

Setting the JavaBinaryForUDx configuration parameter

The JavaBinaryForUDx configuration parameter tells Vertica where to look for the JRE to execute Java UDxs. After you have installed the JRE on all of the nodes in your cluster, set this parameter to the absolute path of the Java executable. You can use the symbolic link that some Java installers create (for example /usr/bin/java). If the Java executable is in your shell search path, you can get the path of the Java executable by running the following command from the Linux command line shell:

$ which java
/usr/bin/java

If the java command is not in the shell search path, use the path to the Java executable in the directory where you installed the JRE. Suppose you installed the JRE in /usr/java/default (which is where the installation package supplied by Oracle installs the Java 1.6 JRE). In this case the Java executable is /usr/java/default/bin/java.

You set the configuration parameter by executing the following statement as a database superuser:

=> ALTER DATABASE DEFAULT SET PARAMETER JavaBinaryForUDx = '/usr/bin/java';

See ALTER DATABASE for more information on setting configuration parameters.

To view the current setting of the configuration parameter, query the CONFIGURATION_PARAMETERS system table:

=> \x
Expanded display is on.
=> SELECT * FROM CONFIGURATION_PARAMETERS WHERE parameter_name = 'JavaBinaryForUDx';
-[ RECORD 1 ]-----------------+----------------------------------------------------------
node_name                     | ALL
parameter_name                | JavaBinaryForUDx
current_value                 | /usr/bin/java
default_value                 |
change_under_support_guidance | f
change_requires_restart       | f
description                   | Path to the java binary for executing UDx written in Java

Once you have set the configuration parameter, Vertica can find the Java executable on each node in your cluster.

Note

Since the location of the Java executable is set by a single configuration parameter for the entire cluster, you must ensure that the Java executable is installed in the same path on all of the hosts in the cluster.

Extending: UDx restrictions

Mon, 01 Jan 0001 00:00:00 +0000

Some UDx types have special considerations or restrictions.

UDxs written in Java and R do not support complex types.

Aggregate functions

You cannot use the DISTINCT clause in queries with more than one aggregate function or provide inputs or return values containing complex types.

Analytic functions

UDAnFs do not support framing windows using ROWS.

Only UDAnFs written in C++ can use complex types.

As with OpenText™ Analytics Database's built-in analytic functions, UDAnFs cannot be used with MATCH clause functions.

Scalar functions

If the result of applying a UDSF is an invalid record, COPY aborts the load even if CopyFaultTolerantExpressions is set to true.

A ROW returned from a UDSF cannot be used as an argument to COUNT.

Transform functions

A query that includes a UDTF cannot:

Include statements other than the SELECT statement that calls the UDTF and a PARTITION BY expression unless the UDTF is marked as a one-to-many UDTF
Call an analytic function
Call another UDTF
Include one of the following clauses:

Load functions

Installing an untrusted UDL function can compromise the security of the server. UDxs can contain arbitrary code. In particular, user-defined parser functions can read data from any arbitrary location. It is up to the developer of the function to enforce proper security limitations. Superusers must not grant access to UDxs to untrusted users.

You cannot ALTER UDL functions.

UDFilter and UDSource functions do not support complex types.

Extending: Fenced and unfenced modes

Mon, 01 Jan 0001 00:00:00 +0000

User-defined extensions (UDxs) written in the C++ programming language have the option of running in fenced or unfenced mode. Fenced mode runs the UDx code outside of the main OpenText™ Analytics Database process in a separate zygote process. UDxs that use unfenced mode run directly within the OpenText™ Analytics Database process.

Fenced mode

You can run most C++ UDxs in fenced mode. Fenced mode uses a separate zygote process, so fenced UDx crashes do not impact the core database process. There is a small performance impact when running UDx code in fenced mode. On average, using fenced mode adds about 10% more time to execution compared to unfenced mode.

Fenced mode is currently available for all C++ UDxs with the exception of user-defined aggregates. All UDxs developed in the Python, R, and Java programming languages must run in fenced mode, since the Python, R, and Java runtimes cannot run directly within the database process.

Using fenced mode does not affect the development of your UDx. Fenced mode is enabled by default for UDxs that support fenced mode. Optionally, you can issue the CREATE FUNCTION command with the NOT FENCED modifier to disable fenced mode for the function. Additionally, you can enable or disable fenced mode on any fenced mode-supported C++ UDx by using the ALTER FUNCTION command.

Unfenced mode

Unfenced UDxs run within the database, so they have little overhead, and can perform almost as fast as the database's own built-in functions. However, because they run within the database directly, any bugs in their code (memory leaks, for example) can destabilize the main database process and bring one or more database nodes down.

Important

Always carefully review code downloaded from third-party sources and test UDxs in a test or staging environment before deciding to run them in unfenced mode in production.

About the zygote process

The OpenText™ Analytics Database zygote process starts when the database starts. Each node has a single zygote process. Side processes are created "on demand". The zygote listens for requests and spawns a UDx side session that runs the UDx in fenced mode when a UDx is called by the user.

About fenced mode logging:

UDx code that runs in fenced mode is logged in the UDxZygote.log and is stored in the UDxLogs directory in the catalog directory of the database. Log entries for the side process are denoted by the UDx language (for example, C++), node, zygote process ID, and the UDxSideProcess ID.

For example, for the following query returns the current fenced processes:

=> SELECT * FROM UDX_FENCED_PROCESSES;
    node_name     |   process_type   |            session_id            |  pid  | port  | status
------------------+------------------+----------------------------------+-------+-------+--------
 v_vmart_node0001 | UDxZygoteProcess |                                  | 27468 | 51900 | UP
 v_vmart_node0001 | UDxSideProcess   | localhost.localdoma-27465:0x800b |  5677 | 44123 | UP

Below is the corresponding log file for the fenced processes returned in the previous query:

2016-05-16 11:24:43.990 [C++-localhost.localdoma-27465:0x800b-5677]  0x2b3ff17e7fd0 UDx side process started
 11:24:43.996 [C++-localhost.localdoma-27465:0x800b-5677]  0x2b3ff17e7fd0 Finished setting up signal handlers.
 11:24:43.996 [C++-localhost.localdoma-27465:0x800b-5677]  0x2b3ff17e7fd0 My port: 44123
 11:24:43.996 [C++-localhost.localdoma-27465:0x800b-5677]  0x2b3ff17e7fd0 My address: 0.0.0.0
 11:24:43.996 [C++-localhost.localdoma-27465:0x800b-5677]  0x2b3ff17e7fd0 Vertica port: 51900
 11:24:43.996 [C++-localhost.localdoma-27465:0x800b-5677]  0x2b3ff17e7fd0 Vertica address: 127.0.0.1
 11:25:19.749 [C++-localhost.localdoma-27465:0x800b-5677]  0x41837940 Setting memory resource limit to -1
 11:30:11.523 [C++-localhost.localdoma-27465:0x800b-5677]  0x41837940 Exiting UDx side process

The last line indicates that the side process was killed. In this case it was killed when the user session (vsql) closed.

About fenced mode configuration parameters

Fenced mode supports the following configuration parameters:

FencedUDxMemoryLimitMB: The maximum memory size, in MB, to use for fenced mode processes. The default is -1 (no limit). The side process is killed if this limit is exceeded.
ForceUDxFencedMode: When set to 1, force all UDxs that support fenced mode to run in fenced mode even if their definitions specify NOT FENCED. The default is 0 (disabled).
UDxFencedBlockTimeout: The maximum time, in seconds, that the database server waits for a UDx to return before aborting with ERROR 3399. The default is 60.
UDxJVMIdleTimeout: The maximum time, in seconds, that a Java UDx JVM is allowed to remain idle (no active tasks and no incoming requests) before it exits automatically. When this timeout is reached, the JVM shuts down gracefully and logs the inactivity event in the Java UDx logs. The default is 120.

Extending: Updating UDx libraries

Mon, 01 Jan 0001 00:00:00 +0000

There are two cases where you need to update libraries that you have already deployed:

When you have upgraded OpenText™ Analytics Database to a new version that contains changes to the SDK API. For your libraries to work with the new server version, you need to recompile them with new version of the SDK. See UDx library compatibility with new server versions for more information.
When you have made changes to your UDxs and you want to deploy these changes. Before updating your UDx library, you need to determine if you have changed the signature of any of the functions contained in the library. If you have, you need to drop the functions from the database catalog before you update the library.

Extending: Listing the UDxs contained in a library

Mon, 01 Jan 0001 00:00:00 +0000

Once a library has been loaded using the CREATE LIBRARY statement, you can find the UDxs and UDLs it contains by querying the USER_LIBRARY_MANIFEST system table:

=> CREATE LIBRARY ScalarFunctions AS '/home/dbadmin/ScalarFunctions.so';
CREATE LIBRARY
=> \x
Expanded display is on.
=> SELECT * FROM USER_LIBRARY_MANIFEST WHERE lib_name = 'ScalarFunctions';
-[ RECORD 1 ]-------------------
schema_name | public
lib_name    | ScalarFunctions
lib_oid     | 45035996273792402
obj_name    | RemoveSpaceFactory
obj_type    | Scalar Function
arg_types   | Varchar
return_type | Varchar
-[ RECORD 2 ]-------------------
schema_name | public
lib_name    | ScalarFunctions
lib_oid     | 45035996273792402
obj_name    | Div2intsInfo
obj_type    | Scalar Function
arg_types   | Integer, Integer
return_type | Integer
-[ RECORD 3 ]-------------------
schema_name | public
lib_name    | ScalarFunctions
lib_oid     | 45035996273792402
obj_name    | Add2intsInfo
obj_type    | Scalar Function
arg_types   | Integer, Integer
return_type | Integer

The obj_name column lists the factory classes contained in the library. These are the names you use to define UDxs and UDLs in the database catalog using statements such as CREATE FUNCTION and CREATE SOURCE.

Extending: Using wildcards in your UDx

Mon, 01 Jan 0001 00:00:00 +0000

OpenText™ Analytics Database supports wildcard * characters in the place of column names in user-defined functions.

You can use wildcards when:

Your query contains a table in the FROM clause.
You are using a OpenText™ Analytics Database-supported development language.
Your UDx is running in fenced or unfenced mode.

Supported SQL statements

The following SQL statements can accept wildcards:

DELETE
INSERT
SELECT
UPDATE

Unsupported configurations

The following situations do not support wildcards:

You cannot pass a wildcard in the OVER clause of a query
You cannot us a wildcard with a DROP statement
You cannot use wildcards with any other arguments

Examples

These examples show wildcards and user-defined functions in a range of data manipulation operations.

DELETE statements:

=> DELETE FROM tablename WHERE udf(tablename.*) = 5;

INSERT statements:

=> INSERT INTO table1 SELECT udf(*) FROM table2;

SELECT statements:

=> SELECT udf(*) FROM tablename;
=> SELECT udf(tablename.*) FROM tablename;
=> SELECT udf(f.*) FROM table f;
=> SELECT udf(*) FROM table1,table2;
=> SELECT udf1( udf2(*) ) FROM table1,table2;
=> SELECT udf( db.schema.table.*) FROM tablename;
=> SELECT udf(sub.*) FROM (select col1, col2 FROM table) sub;
=> SELECT x FROM tablename WHERE udf(*) = y;
=> WITH sub as (SELECT * FROM tablename) select x, udf(*) FROM sub;
=> SELECT udf( * using parameters x=1) FROM tablename;
=> SELECT udf(table1.*, table2.col2) FROM table1,table2;

UPDATE statements:

=> UPDATE tablename set col1 = 4 FROM tablename WHERE udf(*) = 3;

OpenText Analytics Database 26.2.x – User-defined extensions

Extending: Loading UDxs

Deploying libraries

Privileges

Creating UDx functions

Extending: Installing Java on OpenText Analytics Database hosts

Installing a Java runtime

Note

Setting the JavaBinaryForUDx configuration parameter

Note

Extending: UDx restrictions

Aggregate functions

Analytic functions

Scalar functions

Transform functions

Load functions

Extending: Fenced and unfenced modes

Fenced mode

Unfenced mode

Important

About the zygote process

About fenced mode logging:

About fenced mode configuration parameters

See also

Extending: Updating UDx libraries

Extending: Listing the UDxs contained in a library

Extending: Using wildcards in your UDx

Supported SQL statements

Unsupported configurations

Examples