OpenText Analytics Database 26.2.x – Errors, warnings, and logging

Extending: Sending messages

Mon, 01 Jan 0001 00:00:00 +0000

A UDx can handle a problem by reporting an error and terminating execution, but in some cases you might want to send a warning and proceed. For example, a UDx might ignore or use a default for an unexpected input and report that it did so. The C++ and Python messaging APIs support reporting messages at different severity levels.

A UDx has access to a ServerInterface instance. This class has the following methods for reporting messages, in order of severity:

reportError (also terminates execution)
reportWarning
reportNotice
reportInfo

Each method produces messages with the following components:

ID code: an identification code, any integer. This code does not interact with OpenText™ Analytics Database error codes.
Message string: a succinct description of the issue.
Optional details string: provides more contextual information.
Optional hint string: provides other guidance.

Duplicate messages are condensed into a single report if they have the same code and message string, even if the details and hint strings differ.

Constructing messages

The UDx should report errors immediately, usually during a process call. For all other message types, record information during processing and call the reporting methods from the UDx's destroy method. The other reporting methods do not produce output if called during processing.

The process of constructing messages is language-specific.

C++

Each ServerInterface reporting method takes a ClientMessage argument. The ClientMessage class has the following methods to set the code and message, detail, and hint:

makeMessage: sets the ID code and message string.
setDetail: sets the optional detail string.
setHint: sets the optional hint string.

These method calls can be chained to simplify creating and passing the message.

All strings support printf-style arguments and formatting.

In the following example, a function records issues in processBlock and reports them in destroy:


class PositiveIdentity : public Vertica::ScalarFunction
    {
public:
    using ScalarFunction::destroy;
    bool hitNotice = false;

    virtual void processBlock(Vertica::ServerInterface &srvInterface,
                              Vertica::BlockReader &arg_reader,
                              Vertica::BlockWriter &res_writer)
    {
        do {
            const Vertica::vint a = arg_reader.getIntRef(0);
            if (a < 0 && a != vint_null) {
                hitNotice = true;
                res_writer.setInt(null);
            } else {
                res_writer.setInt(a);
            }
            res_writer.next();
        } while (arg_reader.next());
    }

    virtual void destroy(ServerInterface &srvInterface,
                         const SizedColumnTypes &argTypes) override
    {
        if (hitNotice) {
            ClientMessage msg = ClientMessage::makeMessage(100, "Passed negative argument")
                                .setDetail("Value set to null");
            srvInterface.reportNotice(msg);
        }
    }
}

Python

Each ServerInterface reporting method has the following positional and keyword arguments:

idCode: integer ID code, positional argument.
message: message text, positional argument.
hint: optional hint text, keyword argument.
detail: optional detail text, keyword argument.

All arguments support str.format() and f-string formatting.

In the following example, a function records issues in processBlock and reports them in destroy:


class PositiveIdentity(vertica_sdk.ScalarFunction):
    def __init__(self):
        self.hitNotice = False

    def processBlock(self, server_interface, arg_reader, res_writer):
        while True:
            arg = arg_reader.getInt(0)
            if arg < 0 and arg is not None:
                self.hitNotice = True
                res_writer.setNull()
            else:
                res_writer.setInt(arg)
            res_writer.next()
            if not arg_reader.next():
                break

    def destroy(self, srv, argType):
        if self.hitNotice:
            srv.reportNotice(100, "Passed negative arguement", detail="Value set to null")
        return

API

C++
Python

Before calling a ServerInterface reporting method, construct and populate a message with the ClientMessage class.

The ServerInterface API provides the following methods for reporting messages:


// ClientMessage methods
template<typename... Argtypes>
static ClientMessage makeMessage(int errorcode, const char *fmt, Argtypes&&... args);

template <typename... Argtypes>
ClientMessage & setDetail(const char *fmt, Argtypes&&... args);

template <typename... Argtypes>
ClientMessage & setHint(const char *fmt, Argtypes&&... args);

// ServerInterface reporting methods
virtual void reportError(ClientMessage msg);

virtual void reportInfo(ClientMessage msg);

virtual void reportNotice(ClientMessage msg);

virtual void reportWarning(ClientMessage msg);

The ServerInterface API provides the following methods for reporting messages:


def reportError(self, code, text, hint='', detail=''):

def reportInfo(self, code, text, hint='', detail=''):

def reportNotice(self, code, text, hint='', detail=''):

def reportWarning(self, code, text, hint='', detail=''):

Extending: Handling errors

Mon, 01 Jan 0001 00:00:00 +0000

If your UDx encounters an unrecoverable error, it should report the error and terminate. How you do this depends on the language:

C++: Consider using the API described in Sending messages, which is more expressive than the error-handling described in this topic. Alternatively, you can use the vt_report_error macro to report an error and exit. The macro takes two parameters: an error number and an error message string. Both the error number and message appear in the error that OpenText™ Analytics Database reports to the user. The error number is not defined by the database. You can use any value.
Java: Instantiate and throw a UdfException, which takes a numeric code and a message string to report to the user.
Python: Consider using the API described in Sending messages, which is more expressive than the error-handling described in this topic. Alternatively, raise an exception built into the Python language; the SDK does not include a UDx-specific exception.
R: Use stop to halt execution with a message.

An exception or halt causes the transaction containing the function call to be rolled back.

The following examples demonstrate error-handling:

The following function divides two integers. To prevent division by zero, it tests the second parameter and fails if it is zero:

class Div2ints : public ScalarFunction
{
public:
  virtual void processBlock(ServerInterface &srvInterface,
                            BlockReader &arg_reader,
                            BlockWriter &res_writer)
  {
    // While we have inputs to process
    do
      {
        const vint a = arg_reader.getIntRef(0);
        const vint b = arg_reader.getIntRef(1);
        if (b == 0)
          {
            vt_report_error(1,"Attempted divide by zero");
          }
        res_writer.setInt(a/b);
        res_writer.next();
      }
    while (arg_reader.next());
  }
};

Loading and invoking the function demonstrates how the error appears to the user. Fenced and unfenced modes use different error numbers.

=> CREATE LIBRARY Div2IntsLib AS '/home/dbadmin/Div2ints.so';
CREATE LIBRARY
=> CREATE FUNCTION div2ints AS LANGUAGE 'C++' NAME 'Div2intsInfo' LIBRARY Div2IntsLib;
CREATE FUNCTION
=> SELECT div2ints(25, 5);
 div2ints
----------
        5
(1 row)
=> SELECT * FROM MyTable;
 a  | b
----+---
 12 | 6
  7 | 0
 12 | 2
 18 | 9
(4 rows)
=> SELECT * FROM MyTable WHERE div2ints(a, b) > 2;
ERROR 3399:  Error in calling processBlock() for User Defined Scalar Function
div2ints at Div2ints.cpp:21, error code: 1, message: Attempted divide by zero

In the following example, if either of the arguments is NULL, the processBlock() method throws an exception:

@Override
public void processBlock(ServerInterface srvInterface,
                         BlockReader argReader,
                         BlockWriter resWriter)
            throws UdfException, DestroyInvocation
{
  do {
      // Test for NULL value. Throw exception if one occurs.
      if (argReader.isLongNull(0) || argReader.isLongNull(1) ) {
          // No nulls allowed. Throw exception
          throw new UdfException(1234, "Cannot add a NULL value");
     }

When your UDx throws an exception, the side process running your UDx reports the error back to the database and exits. The database displays the error message contained in the exception and a stack trace to the user:

=> SELECT add2ints(2, NULL);
ERROR 3399:  Failure in UDx RPC call InvokeProcessBlock(): Error in User Defined Object [add2ints], error code: 1234
com.vertica.sdk.UdfException: Cannot add a NULL value
        at com.example.Add2intsFactory$Add2ints.processBlock(Add2intsFactory.java:37)
        at com.vertica.udxfence.UDxExecContext.processBlock(UDxExecContext.java:700)
        at com.vertica.udxfence.UDxExecContext.run(UDxExecContext.java:173)
        at java.lang.Thread.run(Thread.java:662)

In this example, if one of the arguments is less than 100, then the Python UDx throws an error:

    while(True):
        # Example of error checking best practices.
        product_id = block_reader.getInt(2)
        if product_id < 100:
            raise ValueError("Invalid Product ID")

An error generates a message like the following:

=> SELECT add2ints(prod_cost, sale_price, product_id) FROM bunch_of_numbers;
ERROR 3399:  Failure in UDx RPC call InvokeProcessBlock(): Error calling processBlock() in User Defined Object [add2ints]
at [/udx/PythonInterface.cpp:168], error code: 0,
message: Error [/udx/PythonInterface.cpp:385] function ['call_method']
(Python error type [<class 'ValueError'>])
Traceback (most recent call last):
  File "/home/dbadmin/py_db/v_py_db_node0001_catalog/Libraries/02fc4af0ace6f91eefa74baecf3ef76000a0000000004fc4/pylib_02fc4af0ace6f91eefa74baecf3ef76000a0000000004fc4.py",
line 13, in processBlock
    raise ValueError("Invalid Product ID")
ValueError: Invalid Product ID

In this example, if the third column of the data frame does not match the specified Product ID, then the R UDx throws an error:


Calculate_Cost_w_Tax <- function(input.data.frame) {
  # Must match the Product ID 11444
  if ( !is.numeric(input.data.frame[, 3]) == 11444 ) {
    stop("Invalid Product ID!")
  } else {
    cost_w_tax <- data.frame(input.data.frame[, 1] * input.data.frame[, 2])
  }
  return(cost_w_tax)
}

Calculate_Cost_w_TaxFactory <- function() {
  list(name=Calculate_Cost_w_Tax,
       udxtype=c("scalar"),
       intype=c("float","float", "float"),
       outtype=c("float"))
}

An error generates a message like the following:

=> SELECT Calculate_Cost_w_Tax(item_price, tax_rate, prod_id) FROM Inventory_Sales_Data;
vsql:sql_test_multiply.sql:21: ERROR 3399:  Failure in UDx RPC call InvokeProcessBlock():
Error calling processBlock() in User Defined Object [mul] at
[/udx/RInterface.cpp:1308],
error code: 0, message: Exception in processBlock :Invalid Product ID!

To report additional diagnostic information about the error, you can write messages to a log file before throwing the exception (see Logging).

Your UDx must not consume exceptions that it did not throw. Intercepting server exceptions can lead to database instability.

Extending: Logging

Mon, 01 Jan 0001 00:00:00 +0000

Each UDx written in C++, Java, or Python has an associated instance of ServerInterface. The ServerInterface class provides a function to write to the OpenText™ Analytics Database log, and the C++ implementation also provides a function to log events in a system table.

Writing messages to the OpenText™ Analytics Database log

You can write to log files using the ServerInterface.log() function. The function acts similarly to printf(), taking a formatted string and an optional set of values and writing the string to the log file. Where the message is written depends on whether your function runs in fenced mode or unfenced mode:

Functions running in unfenced mode write their messages into the vertica.log file in the catalog directory.
Functions running in fenced mode write their messages into a log file named UDxLogs/UDxFencedProcesses.log in the catalog directory.

To help identify your function's output, the database adds the SQL function name bound to your UDx to the log message.

The following example logs a UDx's input values:

    virtual void processBlock(ServerInterface &srvInterface,
                              BlockReader &argReader,
                              BlockWriter &resWriter)
    {
        try {
            // While we have inputs to process
            do {
                if (argReader.isNull(0) || argReader.isNull(1)) {
                    resWriter.setNull();
                } else {
                    const vint a = argReader.getIntRef(0);
                    const vint b = argReader.getIntRef(1);
                    srvInterface.log("got a: %d and b: %d", (int) a, (int) b);
                    resWriter.setInt(a+b);
                }
                resWriter.next();
            } while (argReader.next());
        } catch(std::exception& e) {
            // Standard exception. Quit.
            vt_report_error(0, "Exception while processing block: [%s]", e.what());
        }
    }

        @Override
        public void processBlock(ServerInterface srvInterface,
                                 BlockReader argReader,
                                 BlockWriter resWriter)
                    throws UdfException, DestroyInvocation
        {
            do {
                // Get the two integer arguments from the BlockReader
                long a = argReader.getLong(0);
                long b = argReader.getLong(1);

                // Log the input values
                srvInterface.log("Got values a=%d and b=%d", a, b);

                long result = a+b;
                resWriter.setLong(result);
                resWriter.next();
            } while (argReader.next());
        }
    }

    def processBlock(self, server_interface, arg_reader, res_writer):
        server_interface.log("Python UDx - Adding 2 ints!")
        while(True):
            first_int = block_reader.getInt(0)
            second_int = block_reader.getInt(1)
            block_writer.setInt(first_int + second_int)
            server_interface.log("Values: first_int is {} second_int is {}".format(first_int, second_int))
            block_writer.next()
            if not block_reader.next():
                break

The log() function generates entries in the log file like the following:

$ tail /home/dbadmin/py_db/v_py_db_node0001_catalog/UDxLogs/UDxFencedProcesses.log
 07:52:12.862 [Python-v_py_db_node0001-7524:0x206c-40575]  0x7f70eee2f780 PythonExecContext::processBlock
 07:52:12.862 [Python-v_py_db_node0001-7524:0x206c-40575]  0x7f70eee2f780 [UserMessage] add2ints - Python UDx - Adding 2 ints!
 07:52:12.862 [Python-v_py_db_node0001-7524:0x206c-40575]  0x7f70eee2f780 [UserMessage] add2ints - Values: first_int is 100 second_int is 100

For details on viewing the database log files, see Monitoring log files.

Writing messages to the UDX_EVENTS table (C++ only)

In the C++ API, you can write messages to the UDX_EVENTS system table instead of or in addition to writing to the log. Writing to a system table allows you to collect events from all nodes in one place.

You can write to this table using the ServerInterface.logEvent() function. The function takes one argument, a map. The map is written into the __RAW__ column of the table as a Flex VMap. The following example shows how the Parquet exporter creates and logs this map.

// Log exported parquet file details to v_monitor.udx_events
std::map<std::string, std::string> details;
details["file"] = escapedPath;
details["created"] = create_timestamp_;
details["closed"] = close_timestamp_;
details["rows"] = std::to_string(num_rows_in_file);
details["row_groups"] = std::to_string(num_row_groups_in_file);
details["size_mb"] = std::to_string((double)outputStream->Tell()/(1024*1024));
srvInterface.logEvent(details);

You can select individual fields from the VMap as in the following example.

=> SELECT __RAW__['file'] FROM UDX_EVENTS;
                                   __RAW__
-----------------------------------------------------------------------------
 /tmp/export_tmpzLkrKq3a/450c4213-v_vmart_node0001-139770732459776-0.parquet
 /tmp/export_tmpzLkrKq3a/9df1c797-v_vmart_node0001-139770860660480-0.parquet
(2 rows)

Alternatively, you can define a view to make it easier to query fields directly, as columns. See Monitoring exports for an example.