OpenText Analytics Database 26.2.x – User-defined load (UDL)

Extending: User-defined source

Mon, 01 Jan 0001 00:00:00 +0000

A user-defined source allows you to process a source of data using a method that is not built into OpenText™ Analytics Database. For example, you can write a user-defined source to access the data from an HTTP source using cURL. While a given COPY statement can use specify only one user-defined source statement, the source function itself can pull data from multiple sources.

The UDSource class acquires data from an external source. It reads data from an input stream and produces an output stream to be filtered and parsed. If you implement a UDSource, you must also implement a corresponding SourceFactory.

Extending: User-defined filter

Mon, 01 Jan 0001 00:00:00 +0000

User-defined filter functions allow you to manipulate data obtained from a source in various ways. For example, a filter can:

Process a compressed file in a compression format not natively supported by OpenText™ Analytics Database.
Take UTF-16-encoded data and transcode it to UTF-8 encoding.
Perform search-and-replace operations on data before it is loaded into the database.

You can also process data through multiple filters before it is loaded into the database. For instance, you could unzip a file compressed with GZip, convert the content from UTF-16 to UTF-8, and finally search and replace certain text strings.

If you implement a UDFilter, you must also implement a corresponding FilterFactory.

See UDFilter class and FilterFactory class for API details.

Extending: User-defined parser

Mon, 01 Jan 0001 00:00:00 +0000

A parser takes a stream of bytes and passes a corresponding sequence of tuples to the OpenText™ Analytics Database load process. You can use user-defined parser functions to parse:

Data in formats not understood by the OpenText™ Analytics Database built-in parser.
Data that requires more specific control than the built-in parser supplies.

For example, you can load a CSV file using a specific CSV library. See the OpenText™ Analytics Database SDK for two CSV examples.

COPY supports a single user-defined parser that you can use with a user-defined source and zero or more instances of a user-defined filter. If you implement a UDParser class, you must also implement a corresponding ParserFactory.

Sometimes, you can improve the performance of your parser by adding a chunker. A chunker divides up the input and uses multiple threads to parse it. Chunkers are available only in the C++ API. For details, see Cooperative parse and UDChunker class. Under special circumstances you can further improve performance by using apportioned load, an approach where multiple database nodes parse the input.

Extending: Load parallelism

Mon, 01 Jan 0001 00:00:00 +0000

OpenText™ Analytics Database can divide the work of loading data, taking advantage of parallelism to speed up the operation. The database supports several types of parallelism:

Distributed load: The database distributes files in a multi-file load to several nodes to load in parallel, instead of loading all of them on a single node. The database manages distributed load; you do not need to do anything special in your UDL.
Cooperative parse: A source being loaded on a single node uses multi-threading to parallelize the parse. Cooperative parse divides a load at execution time, based on how threads are scheduled. You must enable cooperative parse in your parser. See Cooperative parse.
Apportioned load: The database divides a single large file or other single source into segments, which it assigns to several nodes to load in parallel. Apportioned load divides the load at planning time, based on available nodes and cores on each node. You must enable apportioned load in your source and parser. See Apportioned load.

You can support both cooperative parse and apportioned load in the same UDL. The database decides which to use for each load operation and might use both. See Combining cooperative parse and apportioned load.

Extending: Continuous load

Mon, 01 Jan 0001 00:00:00 +0000

The ContinuousUDSource, ContinuousUDFilter, and ContinuousUDParser classes allow you to write and process data as needed instead of having to iterate through the data. The Python API does not support continuous load.

Each class includes the following functions:

initialize() - Invoked before run(). You can optionally override this function to perform setup and initialization.
run() - Processes the data.
deinitialize() - Invoked after run() has returned. You can optionally override this function to perform tear-down and destruction.

Do not override the setup(), process(), and destroy() functions that are inherited from parent classes.

You can use the yield() function to yield control back to the server during idle or busy loops so the server can check for status changes or query cancellations.

These three classes use associated ContinuousReader and ContinuousWriter classes to read input data and write output data.

ContinuousUDSource API (C++)

The ContinuousUDSource class extends UDSource and adds the following methods for extension by subclasses:

virtual void initialize(ServerInterface &srvInterface);

virtual void run();

virtual void deinitialize(ServerInterface &srvInterface);

ContinuousUDFilter API (C++)

The ContinuousUDFilter class extends UDFilter and adds the following methods for extension by subclasses:

virtual void initialize(ServerInterface &srvInterface);

virtual void run();

virtual void deinitialize(ServerInterface &srvInterface);

ContinuousUDParser API

C++
Java

The ContinuousUDParser class extends UDParser and adds the following methods for extension by subclasses:

virtual void initialize(ServerInterface &srvInterface);

virtual void run();

virtual void deinitialize(ServerInterface &srvInterface);

The ContinuousUDParser class extends UDParser and adds the following methods for extension by subclasses:


public void initialize(ServerInterface srvInterface, SizedColumnTypes returnType);

public abstract void run() throws UdfException;

public void deinitialize(ServerInterface srvInterface, SizedColumnTypes returnType);

See the API documentation for additional utility methods.

Extending: Buffer classes

Mon, 01 Jan 0001 00:00:00 +0000

Buffer classes are used as handles to the raw data stream for all UDL functions. The C++ and Java APIs use a single DataBuffer class for both input and output. The Python API has two classes, InputBuffer and OutputBuffer.

DataBuffer API (C++, java)

C++
Java

The DataBuffer class has a pointer to a buffer and size, and an offset indicating how much of the stream has been consumed.

/**
* A contiguous in-memory buffer of char *
*/
    struct DataBuffer {
    /// Pointer to the start of the buffer
    char * buf;

    /// Size of the buffer in bytes
    size_t size;

    /// Number of bytes that have been processed by the UDL
    size_t offset;
};

The DataBuffer class has an offset indicating how much of the stream has been consumed. Because Java is a language whose strings require attention to character encodings, the UDx must decode or encode buffers. A parser can interact with the stream by accessing the buffer directly.

/**
* DataBuffer is a a contiguous in-memory buffer of data.
*/
public class DataBuffer {

/**
* The buffer of data.
*/
public byte[] buf;

/**
* An offset into the buffer that is typically used to track progress
* through the DataBuffer. For example, a UDParser advances the
* offset as it consumes data from the DataBuffer.
*/
public int offset;}

InputBuffer and OutputBuffer APIs (python)

The Python InputBuffer and OutputBuffer classes replace the DataBuffer class in the C++ and Java APIs.

InputBuffer class

The InputBuffer class decodes and translates raw data streams depending on the specified encoding. Python natively supports a wide range of languages and codecs. The InputBuffer is an argument to the process() method for both UDFilters and UDParsers. A user interacts with the UDL's data stream by calling methods of the InputBuffer

If you do not specify a value for setEncoding(), OpenText™ Analytics Database assumes a value of NONE.

class InputBuffer:
    def getSize(self):
        ...
    def getOffset(self):
    ...

    def setEncoding(self, encoding):
        """
        Set the encoding of the data contained in the underlying buffer
        """
        pass

    def peek(self, length = None):
        """
        Copy data from the input buffer into Python.
        If no encoding has been specified, returns a Bytes object containing raw data.
        Otherwise, returns data decoded into an object corresponding to the specified encoding
        (for example, 'utf-8' would return a string).
        If length is None, returns all available data.
        If length is not None then the length of the returned object is at most what is requested.
        This method does not advance the buffer offset.
        """
        pass

    def read(self, length = None):
        """
        See peek().
        This method does the same thing as peek(), but it also advances the
        buffer offset by the number of bytes consumed.
        """
        pass

        # Advances the DataBuffer offset by a number of bytes equal to the result
        # of calling "read" with the same arguments.
        def advance(self, length = None):
        """
        Advance the buffer offset by the number of bytes indicated by
        the length and encoding arguments.  See peek().
    Returns the new offset.
        """
        pass

OutputBuffer class

The OutputBuffer class encodes and outputs data from Python to OpenText™ Analytics Database. The OutputBuffer is an argument to the process() method for both UDFilters and UDParsers. A user interacts with the UDL's data stream by calling methods of the OutputBuffer to manipulate and encode data.

The write() method transfers all data from the Python client to the database. The output buffer can accept any size object. If a user writes an object to the OutputBuffer larger than the database can immediately process, the database stores the overflow. During the next call to process(), the database checks for leftover data. If there is any, the database copies it to the DataBuffer before determining whether it needs to call process() from the Python UDL.

If you do not specify a value for setEncoding(), the database assumes a value of NONE.

class OutputBuffer:
def setEncoding(self, encoding):
"""
Specify the encoding of the data which will be written to the underlying buffer
"""
pass
def write(self, data):
"""
Transfer bytes from the data object into OpenText&trade; Analytics Database.
If an encoding was specified via setEncoding(), the data object will be converted to bytes using the specified encoding.
Otherwise, the data argument is expected to be a Bytes object and is copied to the underlying buffer.
"""
pass