stream_microbatch_history

This table contains a history of every microbatch executed within this scheduler configuration.

This table contains a history of every microbatch executed within this scheduler configuration.

Column Data Type Description
source_name VARCHAR The name of the source.
source_cluster VARCHAR The name of the source cluster. The clusters are defined in stream_clusters.
source_partition INTEGER The number of the data streaming partition.
start_offset INTEGER The starting offset of the microbatch.
end_offset INTEGER The ending offset of the microbatch.
end_reason VARCHAR

An explanation for why the batch ended.The following are valid end reasons:

  • DEADLINE - The batch ran out of time.

  • END_OFFSET - The load reached the ending offset specified in the KafkaSource. This reason is never used by the scheduler, as it does specify an end offset.

  • END_OF_STREAM - There are no messages available to the scheduler or the eof_timeout has been reached.

  • NETWORK_ERROR - The scheduler could not connect to Kafka.

  • RESET_OFFSET - The start offset was changed using the --update and --offset parameters to the KafkaSource. This state does not occur during normal scheduler operations.

  • SOURCE_ISSUE - The Kafka service returned an error.

  • UNKNOWN - The batch ended for an unknown reason.

end_reason_message VARCHAR If the end reason is a network or source issue, this column contains a brief description of the issue.
partition_bytes INTEGER The number of bytes transferred from a source partition to the database target table.
partition_messages INTEGER The number of messages transferred from a source partition to the database target table.
microbatch_id INTEGER The database transaction id for the batch session.
microbatch VARCHAR The name of the microbatch.
target_schema VARCHAR The name of the target schema.
target_table VARCHAR The name of the target table.
timeslice INTERVAL The amount of time spent in the KafkaSource operator.
batch_start TIMESTAMP The time the batch executed.
batch_end TIMESTAMP The time the batch completed.
last_batch_duration INTERVAL The length of time required to run the complete COPY statement.
last_batch_parallelism INTEGER The number of parallel COPY statements generated to process the microbatch during the last frame.
microbatch_sub_id INTEGER The identifier for the COPY statement that processed the microbatch.
consecutive_error_count INTEGER (Currently not used.) The number of times a microbatch has encountered an error on an attempt to load. This value increases over multiple attempts.
transaction_id INTEGER The identifier for the transaction within the session.
frame_start TIMESTAMP The time the frame started. A frame can contain multiple microbatches.
frame_end TIMESTAMP The time the frame completed.

Examples

This example shows typical rows from the stream_microbatch_history table.


=> SELECT * FROM stream_config.stream_microbatch_history;

-[ RECORD 1 ]--+---------------------------
source_name             | streamsource1
source_cluster          | kafka-1
source_partition        | 0
start_offset            | 196
end_offset              | 196
end_reason              | END_OF_STREAM
partition_bytes         | 0
partition_messages      | 0
microbatch_id           | 1
microbatch              | mb_0
target_schema           | public
target_table            | kafka_flex_0
timeslice               | 00:00:09.892
batch_start             | 2016-07-28 11:31:25.854221
batch_end               | 2016-07-28 11:31:26.357942
last_batch_duration     | 00:00:00.379826
last_batch_parallelism  | 1
microbatch_sub_id       | 0
consecutive_error_count |
transaction_id          | 45035996275130064
frame_start             | 2016-07-28 11:31:25.751
frame_end               |
end_reason_message      |

-[ RECORD 2 ]--+---------------------------
source_name             | streamsource1
source_cluster          | kafka-1
source_partition        | 1
start_offset            | 197
end_offset              | 197
end_reason              | NETWORK_ISSUE
partition_bytes         | 0
partition_messages      | 0
microbatch_id           | 1
microbatch              | mb_0
target_schema           | public
target_table            | kafka_flex_0
timeslice               | 00:00:09.897
batch_start             | 2016-07-28 11:31:45.84898
batch_end               | 2016-07-28 11:31:46.253367
last_batch_duration     | 000:00:00.377796
last_batch_parallelism  | 1
microbatch_sub_id       | 0
consecutive_error_count |
transaction_id          | 45035996275130109
frame_start             | 2016-07-28 11:31:45.751
frame_end               |
end_reason_message      | Local: All brokers are down