Collecting diagnostics: scrutinize command

• 1: Running scrutinize
• 2: Informational options
• 3: Redirecting scrutinize output
• 4: Scrutinize security
• 5: Data collection scope
• 6: Uploading scrutinize results
• 7: Troubleshooting scrutinize

1 - Running scrutinize

You can run scrutinize with the following command:

$ /opt/vertica/bin/scrutinize

Unqualified, scrutinize collects a wide range of information from all cluster nodes. It stores the results in a .tar file (VerticaScrutinize.NumericID.tar), with minimal effect on database performance. scrutinize output can help diagnose most issues and yet reduces upload size by omitting fine-grained profiling data.

Command options

scrutinize options support the following tasks:

• Obtain version information about scrutinize and Vertica, and online help.

• Redirect output.

• Access a password-protected database.

• Control the scope of data collection.

• Upload results to Vertica Customer Support.

Privileges

In order for scrutinize to collect data from all system tables, you must have superuser or SYSMONITOR privileges; otherwise, scrutinize collects data only from the system tables that you have privileges to access. If you run scrutinize as root when the dbadmin user exists, Vertica returns an error.

Disk space requirements

scrutinize requires temporary disk space where it can collect data before posting the final compressed (.tar) output. How much space depends on variables such as the size of the Vertica log and extracted system tables, as well as user-specified options that limit the scope of information collected. Before scrutinize runs, it verifies that the temporary directory contains at least 1 GB of space; however, the actual amount needed can be much higher.

You can redirect scrutinize output to another directory. For details, see Redirecting scrutinize output.

Database specification

If multiple databases are defined on the cluster and more than one is active, or none is active, you must run scrutinize with one of the following options:

$ /opt/vertica/bin/scrutinize {--database=database | -d database}

If you omit this option when these conditions are true, scrutinize returns with an error.

2 - Informational options

scrutinize supports two informational options that cannot be combined with any other options:

--version

Obtains the version number of the Vertica server and the scrutinize version number, and then exits. For example:

$ scrutinize --version
Scrutinize Version 12.0.2-20221107
  

--help -h

Lists all scrutinize options to the console, and then exits:

$ scrutinize -h
Usage: scrutinize [options]
  
Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -X LIST, --exclude-tasks=LIST
                        Skip tasks of a particular type. Provide a comma-
                        separated lists of types to skip. Types are case-
                        sensitive. Possible types are: Command, File,
                        VerticaLog, DC, SystemTable, CatalogObject, Query,
                        UdxLog, KafkaLog, MemoryReportLog, all.
  -v, --vsql-off        Does -X Query,SystemTable and skips vsql checks.
                        Useful if vertica is running, but slow to respond.
  -s, --local_diags     Gather diagnostics for local machine only
  -d DB, --database=DB  Only report on database <DB>
  -n HOST_LIST, --hosts=HOST_LIST
                        Gather diagnostics for these hosts only. Host list
                        must be a comma-separated list. Ex. host1,host2,host3
                        or 'host1, host2, host3'
  -m MESSAGE, --message=MESSAGE
                        Reason for gathering diagnostics
  -o OUTPUT_DIR, --output_dir=OUTPUT_DIR
                        redirect output to somewhere other than the current
                        directory
  -U USERNAME, --user=USERNAME
                        Specify DB user
  -P PASSWORD, --password=PASSWORD
                        Specify DB user password
  -W, --prompt-password
                        Force Scrutinize to prompt for DB user password
  ...

3 - Redirecting scrutinize output

By default, scrutinize uses the temporary directory /opt/vertica/tmp execution to compile output while it executes. On completing its collection, it saves the collection to a tar file to the current directory. You can redirect scrutinize output with two options:

--tmpdir=path

Directs temporary output to the specified path, where the following requirements apply to path:

• The directory must have at least 1 GB of free space.

• You must have write permission to it.

--output_dir=path
-o path

Saves scrutinize results to a tar file in path. For example:

$ scrutinize --output_dir="/my_diagnostics/"

4 - Scrutinize security

scrutinize can specify user names and passwords as follows:

--user=username
-U username

Specifies the dbadmin user name. By default, scrutinize uses the user name of the invoking user.

--password=password -P password

Sets the database password as an argument to the scrutinize command. Use this option if the administrator account (default dbadmin) has password authentication. If you omit this option on a password-protected database, scrutinize returns a warning, unless the environment variable VSQL_PASSWORD is set.

Passwords with special characters must be enclosed with single quotes. For example:

$ scrutinize -P '@passWord**'
$ scrutinize --password='$password1*'

`-prompt-password` `-W`

Specifies to prompt users for their database password before scrutinize begins to collect data.

5 - Data collection scope

scrutinize options let you control the scope of the data collection. You can specify the scope of the data collection according to the following criteria:

• Amount of data, including its level of granularity

• Specific nodes

• Types of data to include and exclude

You can use these options singly or in combination, to achieve the desired level of granularity.

Amount of collected data

Several options let you limit how much data scrutinize collects:

--by-second

Collect data every second. This is the highest level of granularity when collecting from Data Collector tables.

--by-minute=boolean-value

Collect data every minute (if the value is true) or every hour (if the value is false).

--get-files file-list

Collect the specified additional files, including globs, where file-list is a semicolon-delimited list of files.

--include_gzlogs=num-files

-z num-files

Number of rotated log files (vertica.log*.gz) to include in the scrutinize output, or all.

By default, scrutinize includes three rotated log files.

--log-limit=limit

-l limit

How much data to collect from Vertica logs, in gigabytes, starting from the most recent log entry. By default, scrutinize collects unlimited log data.

Node-specific collection

By default, scrutinize collects data from all cluster nodes. You can specify that scrutinize collect from individual nodes in three ways:

--local_diags

-s

Collect diagnostics only from the host on which scrutinize was invoked. To collect data from multiple nodes in the cluster, use the --hosts option.

--hosts=host-list

-n host-list

Collect diagnostics only from the hosts specified in host-list, a comma-separated list of IP addresses or host names.

For example:

$ scrutinize --hosts=127.0.0.1,host_3,host_1

--ignore-unreachable-nodes

Ignore nodes that are not reachable. Continue collection with all other nodes with which communication can be established.

Types of data to include

scrutinize provides several options that let you specify the type of data to collect:

--debug

Collects debug information for the log.

--diag-dump

Limits the collection to database design, system tables, and Data Collector tables. Use this option to collect data to analyze system performance.

--diagnostics

Limits the collection to log file data and output from commands that are run against Vertica and its host system. Use this option to collect data to evaluate unexpected behavior in your Vertica system.

--include-ros-info

Includes ROS related information from system tables.

--no-active-queries | --with-active-queries

Whether to exclude diagnostic information from system tables and Data Collector tables about currently running queries. By default, scrutinize collects this information (--with-active-queries).

--tasks=tasks

-T tasks

Gathers diagnostics on one or more tasks, as specified in a file or JSON list. This option is typically used together with --exclude.

Note

Use this option only in consultation with Vertica Customer Support

--type=type

-t type

Type of diagnostics collection to perform, one of the following:

• profiling: Gather profiling data.

• context: Gather summary information.

Types of data to exclude

scrutinize options also let you specify the types of data to exclude from its collection:

--exclude=tasks

-X tasks

Excludes one or more types of tasks from the diagnostics collection, where tasks is a comma-separated list of the tasks to exclude:

• all: All default tasks

• DC: Data Collector tables

• File: Log files from the installation process, the database, and Administration Tools, such as vertica.log, dbLog, and adminTools.log

• VerticaLog: Vertica logs

• CatalogObject: Vertica catalog metadata, such as system configuration parameters

• SystemTable: Vertica system tables that contain information about system, resources, workload, and performance

• Query: Vertica meta-functions that use vsql to connect to the database, such as EXPORT_CATALOG()

• Command: Operating system information, such as the length of time that a node has been up

Note

This option is typically used only in consultation with your Vertica Customer Support contact.

--no-active-queries

Omits diagnostic information from system tables and Data Collector tables about currently running queries. By default, scrutinize always collects active query information (--with-active-queries).

--vsql-off

-v

Excludes Query and SystemTable tasks, which are used to connect to the database. This option can help you deal with problems that occur during an upgrade, and is typically used in the following cases:

• Vertica is running but is slow to respond.

• You haven't yet created a database but need help troubleshooting other cluster issues.

6 - Uploading scrutinize results

scrutinize provides several options for uploading data to Vertica customer support.

Upload packaging

When you use an upload option, scrutinize does not bundle all output in a single tar file. Instead, each node posts its output directly to the specified URL as follows:

1. Uploads a smaller, context file, enabling Customer Support to review high-level information.

2. On completion of scrutinize execution, uploads the complete diagnostics collection.

Upload prerequisites

Before you run scrutinize with an upload option:

• Install the cURL program in the path for the database administrator user who is running scrutinize.

• Verify each node in the cluster can make an HTTP or FTP connection directly to the Internet.

Upload options

--auth-upload=url
-A url

Uses your Vertica license to authenticate with the Vertica server, by uploading your customer name. Customer Support uses this information to verify your identity on receiving your uploaded file. This option requires a valid Vertica license.

--url=url
-u url

Requires url to include a user name and password that is supplied by Vertica Customer Support.

--message=message
-m message

Includes a message with the scrutinize output, where message is a text string, a path to a text file, or PROMPT to open an input stream in which to compose a message. scrutinize reads input until you type a period (.) on a new line. This closes the input stream, and scrutinize writes the message to the collected output.

The message is written in the output directory in reason.txt. If no message is specified, scrutinize generates the default message Unknown reason for collection. Messages typically include the following information:

• Reason for gathering/submitting diagnostics.

• Support-supplied case number and other issue-specific information, to help Vertica Customer Support identify your case and analyze the problem.

Examples

The --auth-upload option uses your Vertica to identify yourself:

$ scrutinize -U username -P 'password' --auth-upload="support-provided-url"

The --url option includes the FTP username and password, supplied by support, in the URL:

$ scrutinize -U username -P 'password' --url='ftp://username/password@customers.vertica.com/'

You can supply a message as a text string or in a text file:

$ scrutinize --message="re: case number #ABC-12345"
$ scrutinize --message="/path/to/msg.txt"

Alternatively, you can open an input stream and type a message:

$ scrutinize --message=PROMPT
Enter reason for collecting diagnostics; end with '.' on a line by itself:
Query performance degradation noticed around 9AM EST on Saturday
.
Vertica Scrutinize Report
-----------------------------
Result Dir:              /home/dbadmin/VerticaScrutinize.20131126083311
...

7 - Troubleshooting scrutinize

The troubleshooting advice in this section can help you resolve common issues that you might encounter when using scrutinize.

Collection time is too slow

To speed up collection time, omit system tables when running an instance of scrutinize. Be aware that collecting from fewer nodes does not necessarily speed up the collection process.

Output size is too large

Output size depends on system table size and vertica log size.

To create a smaller scrutinize output, omit some system tables or truncate the vertica log. For more information, see Narrowing the Scope of scrutinize Data Collection.

System tables not collected on databases with password

Running scrutinize on a password-protected database might require you to supply a user name and password:

$ scrutinize -U username -P 'password'