Cloudera

Overview

This page covers how to add and configure a Cloudera connector using GroundWork Cloud Hub. The connection requires a unique set of parameters (e.g., endpoint, credentials). You will need your GroundWork server and virtual environment connector parameters handy.

1.0 Adding a New Connection

To access Cloud Hub, log in to GroundWork Monitor as an Administrator and select (1) GroundWork Administration > GroundWork Cloud Hub.

The initial Cloud Hub screen is used to (2) Add new, or Start, Stop, Modify, obtain Status for, or Delete configured connectors.

The Start option enables a configured connector to begin the discovery and data collection process. If you decide you do not want to monitor a particular region, simply select Stop for the corresponding connector, the connectors configuration will be maintained for a subsequent start. Modify opens the Configuration page with a link to the Metrics screen. Status provides connection status information including error details. If a configured connector fails to connect, a connector-specific service will be updated to a Warning state, or Critical if you run out of retries (hosts will still become Unreachable and services Unknown if retries are exhausted). To stop and completely Delete a connection, see How to delete hosts. To keep a Cloud Hub connector configuration and temporarily suspend its monitoring, see Black List.

To (3) add a new connection click +Add corresponding to the Cloudera connector icon. You will need to create a new connection in this way for each region to be monitored.

Figure: Adding a connection


Next, in the configuration page (shown below) you will need to enter both the GroundWork server and remote server parameters. The data the GroundWork server receives comes from the remote virtualization server. The information is pulled from the API on a periodic basis based on the check interval that is set.

The (1) Groundwork Server is where Cloud Hub will store Cloudera metrics. Often, this is the same server as where Cloud Hub is running. However, Cloud Hub can also be run in a distributed environment, on its own node in a GroundWork cluster. Here we enter the GroundWork server parameters, each described in the first table below.

Next enter values for the remote (2) Cloudera Connector shown below and described in the second table.

Validate both server configurations by selecting the (3) Test buttons which will check if the connections are accessible with the given credentials. A dialog will be displayed with either a success message or, if the server cannot be contacted, an error message will be displayed with information describing why the connection failed. When a successful connection is made, the Connection Status buttons will change to green.

(4) Service Views are optional features of Cloudera. These features are the core components, or services that are managed by Cloudera. Services include Cluster, Host, HBase, HDFS, Hive, Hue, Impala, KSIndexer, Oozie, Solr, Spark on Yarn, Zookeeper, Yarn, and Kafka. Each of these services has their own rich set of metrics. By default, all services are selected. If checked, service will be monitored. Cloudera also provides Cluster and Host metrics which can be optionally collected. If there are one or more clusters or hosts in the system, they will be automatically detected and collected. If you were collecting metrics for a service, and then unchecked that Cloudera service, the existing hosts and metrics stored in the GroundWork server will be deleted.

And after the remote server parameters have been entered and verified, click (5) Save in the upper right corner to save and write the entries to an XML file in the GroundWork server /usr/local/groundwork/config/cloudhub directory. The Cloud Hub connector is assigned an agent ID and that in turn becomes a record locator in Foundation when you begin monitoring. After the credentials have been validated select the Metrics link (top navigation) to start customizing metrics for the connection.

Figure: Cloudera connector

Table: GroundWork server values

Table: Cloudera server values

2.0 Navigating

From the Configuration page, navigations are on displayed in the top navigation bar:

From here, you can navigate to:

• Home - Cloud Hub home page
• Metrics - Metrics configuration page associated with this Cloud Hub connection
• Configuration - Cloud Hub configuration page

When creating a new Cloudera configuration, the Metrics link is not visible until you successfully save the configuration parameters. Once you are satisfied with your configuration settings, click Save, then click the Metrics link in the navigation bar to start customizing your metrics for this connection. Also, the Save button is not enabled until all required fields are validated. If you make changes on the configuration page, and forget to save, you will be prompted.

3.0 Determining Metrics To Be Monitored

The Cloudera Metrics page is where you customize the lists of metrics being gathered for a connection. Out of the box, a complete list of metrics is provided for clusters, hosts, and Cloudera services. You can customize these metric lists by adding metrics to the list, deleting metrics, as well as creating calculated metric fields called Synthetic metrics.

The Metrics page is displayed in groups of metrics grouped by Cluster, Host, and Cloudera Service collections. The counts of metrics are displayed in the Group bar, and summarized by:

• # metrics per group
• # active metrics per group
• # synthetic metrics per group

A metric is considered inactive if it is not monitored.

Figure: Cloudera metrics


You can configure the metrics for any group by clicking on the (1) group bar. Each row in the grid represents a metric. Metrics can be added, edited or deleted. You can directly edit metrics in the grid or use the advanced metric dialog by clicking the (2) Edit button and then configuring all properties of a metric in the dialog. When editing metrics in the grid directly, you will need to click the (3) Save button in the top navigation to commit your changes to Cloud Hub. To add new metrics use (4) Add Normal Metric or Add Synthetic Metric.

For example, if we click on the bottom Zookeeper group bar, the display automatically expands to show all metrics for the Zookeeper Cloudera service.

Figure: Zookeeper Cloudera


The grid displays the following fields:

Leaving the threshold fields blank will disable threshold triggers.

Metrics come in two flavors: they can either be Normal or Synthetic metrics.

Normal Metrics

Normal metrics can be:

• Single Metric Names
• Computed Metric Names
• Health Checks metrics
• Configuration metrics (not monitored)

Single Metric Names

Figure: Single metric name with display name


In this example, we have a Host metric named load_5. This is the unique name of the metric in Cloudera. In the Display field, we renamed this metric to HostLoad5Minute. Renaming metrics is an optional feature. In this case, we renamed load_5 to have a more descriptive metric name displayed in the GroundWork Status viewer. We recommend filling out the description field to describe the metric. This metric represents the Host CPU Load averaged over 5 minutes. We have also setup warning and critical thresholds. Note that the metric will be monitored and graphed.
Note that we never use dashes in metric names, only underscores. This is because dashes are not valid variable names in a Cloudera or synthetic expression.

As you type into the Metric Name field, the valid names of metrics available are automatically auto-suggested. This ensures that you use a valid Cloudera metric.

Computed Normal Metrics

Normal metrics can also be computed. They differ from Synthetic metrics in that the value of the metric is an expression, and it is computed on the Cloudera server, not by Cloud Hub.

Figure: Normal metrics - computed Cloudera expression


In this example, the Metric name is a computed Cloudera expression. The expression includes two Cloudera metrics: physical_memory_used and physical_memory_total. The expression takes the memory used metric, divides it by the total memory metric and multiplies that by 100 to return a computed metric named memory_usage_percent. The AS keyword is required. It defines an alias for the expression to uniquely name the metric:

(physical_memory_used / physical_memory_total) * 100 as memory_usage_percent

The Metric Format String is an optional C-style formatting string. Here we limit the floating point number to 2 decimal places, and then append a percent sign to the computed metric value:

%.2f%%

See the section below on Example Formatting for more examples.

As you type into the Metric Name field, the valid names of metrics available are automatically auto-suggested. This ensures that you use a valid Cloudera metric name in your expression.

Health Check Metrics

Health Check metrics are a special type of metric that only report back Health Check status.

These metrics do not have numeric values, but instead have health check statuses that map to GroundWork statuses.

Figure: Health check metrics


Health Check metrics are flagged with the Health Check checkbox. As you type into the Metric Name field, the valid names of Health Check metrics available are automatically auto-suggested. This ensures that you use a valid Cloudera Health Check metric.
See the section below on Health Check Status Mappings for the complete list of health check status mappings.

Configuration Metrics

Configuration metrics are only used in synthetic computations. They are not reported back to the GroundWork server. To create a configuration metric, simply do not check the Monitor checkbox.

Configuration metrics are used in synthetic calculations, where the value is required, for example to perform a to megabyte or to gigabyte conversion, but you do not want to report back the byte value to the GroundWork server.

Figure: Configuration metrics


The Monitor check box is left unchecked. Note that we still use thresholds, as they are useful in the Synthetic Expression evaluator.

Synthetic Metrics

A Synthetic metric is a metric that is computed by Cloud Hub. It has one additional field, expression, that normal metrics do not have.

Figure: Synthetic metrics


The synthetic metric name is a simple metric name conforming to the GroundWork service name requirements. No spaces are allowed. By convention, we name synthetic metrics with the prefix syn_.
The Metric expression field contains the synthetic expression. In this example, we use a GroundWork function, GW:GB2 to convert the value of the physical_memory_used Cloudera metric, a value in bytes, to a gigabyte value:

GW:GB2(physical_memory_used)

The Metric Format String is an optional C-style formatting string. Here we limit the floating-point number to 2 decimal places:

%.2f

See the section below on Example Formatting for more examples.

As you type into the Metric Expression field, the valid names of metrics available are automatically auto-suggested. This ensures that you use a valid Cloudera metric name in your expression. Synthetic expressions are limited to the Normal metrics defined for the current group. Additionally, the auto-suggest feature displays all GroundWork functions.

The Synthetic Expression

This field contains an actual programmable expression that is parsed by Cloud Hub. The expression is made up of:

• Normal Metrics (not health checks)
• Expression Operations (addition, subtraction, multiplication, division, parenthesis for grouping)
• GroundWork Functions
• Math Functions

Example expression with division and multiplier operators, parenthesis for grouping, and data conversion of integers to double values. The two normal Host metrics are fd_open and fd_max. Note that both normal metrics must be defined for this group. Other synthetic metrics cannot be included in a synthetic expression.

(GW:toDouble(fd_open) / GW:toDouble(fd_max)) * 100.0

The data types of Cloudera Metrics are typically floating point (double) values for any measurements. For counters, like the example above, are usually integers or longs. Consult the Cloud Hub documentation for a complete reference guide to metrics.

https://www.cloudera.com/documentation/enterprise/5-5-x/topics/cm_metrics.html

Type conversion is supported as GroundWork functions for both floating point and integer numbers. See the section below GroundWork Functions for a complete list.

The Expression Evaluator

The synthetics dialog has an expression evaluator to try out and test your expressions before saving them. The evaluator is displayed at the bottom of the dialog. Each variable in the expression is evaluated based on the check boxes.

Given the expression:

(GW:toDouble(fd_open) / GW:toDouble(fd_max)) * 100.0

there are two variables, fd_open and fd_max. These variables are displayed in the Input Metric Values section of the dialog. There are three ways to evaluate the expression based on:

• Warning Threshold - Select the Warning Threshold option then click Evaluate
• Critical Threshold - Select the Critical Threshold option then click Evaluate
• Override - Enter values into the Override Value fields, then click Evaluate

In the image below, the fd_max and fd_open metric fields are predefined with warning threshold values of 800 and 400. Clicking Evaluate yields the formatted output: 50.00% used.

Figure: Warning threshold


In the image below, the fd_max and fd_open metric fields are predefined with critical threshold values of 1000 and 600. Clicking Evaluate yields the formatted output: 60.00%

Figure: Critical threshold


In the image below, the fd_max and fd_open metric fields are entered with values of 2000 and 1500. Clicking Evaluate yields the formatted output: 75.00% used.

Figure: Override values

Cloudera Computed Examples

Cloudera computed (normal) metrics are calculated in the Cloudera server. Here are some examples of Cloudera computed metrics.

Example 1: Cloudera computes the memory usage of physical memory of a host

Example 2: Cloudera converts bytes to MB for a host metric

Example 3: Cloudera calculates Host CPU Load Percentage over 1 minute

Note that Cloudera currently has the following functions that can be used in a metric computation:

dt(metric) - Derivative with negative values.

The change of the underlying metric expression, per second.

Example:

dt(jvm_gc_count)

dt0(metric) - Derivative where negative values are skipped (useful for dealing with counter resets). The change of the underlying metric expression, per second.

Example:

dt0(jvm_gc_time_ms) / 10

getHostFact(string factName, double defaultValue) - Retrieves a fact about a host.

Example:

dt(total_cpu_user) /   getHostFact(numCores, 2)

This example divides the results of dt(total_cpu_user) by the current number of cores for each host. If the number of cores cannot be determined, the default "2" will be used.

getHostFact currently supports one fact, numCores.

Synthetics examples

Example 1: Cloud Hub computes the physical memory used from bytes to GB with GW function

Example 2: Cloud Hub computes the physical memory used from bytes to GB with GW functions to convert integer values to double values

Example 3: Cloud Hub computes the percentage of memory used with the divideToPercentage function. Note this function returns an integer.

GroundWork Functions

Table: Byte Conversion Functions Using Strict Hexadecimal Values (1024..)

Table: Byte Conversion Functions Using Decimal Values (1000..)

Table: Byte Conversion Functions Using Decimal Values (1000..)

Table: Type Conversion

GW:scalePercentageUsed

This Function provides percentage usage synthetic values.
Calculates the usage percentage for a given used metric and a corresponding available metric.
Both the used metric and available metric can be scaled by corresponding scale factor parameters.

Example:
scalePercentageUsed(summary.quickStats.overallMemoryUsage,summary.hardware.memorySize, 1.0, 1.0)

Parameters:
used - Represents a 'used' metric value of how much of this resource has been used such as 'overallMemoryUsage'
available - Represents the totality of a resource, such as all memory available
usedScaleFactor - multiply usage parameter by this value, or pass in null to not scale. Passing in 1.0 will also not scale
availableScaleFactor - multiply available parameter by this value, or pass in null to not scale. Passing in 1.0 will also not scale

Returns the percentage usage as an integer

GW:scalePercentageUnused

This Function provides percentage unused/free synthetic values.
Calculates the unused(free) percentage for a given unused metric and a corresponding available metric.
Both the unused metric and available metric can be scaled by corresponding scale factor parameters.

Example:
scalePercentageUnused(summary.freeSpace,summary.capacity, 1.0, null, true)

Parameters:
unused - Represents a metric reference value of how much of this resource has not be used (free)
available - Represents the totality of a resource, such as all disk space available
usageScaleFactor - multiply usage parameter by this value, or pass in null to not scale. Passing in 1.0 will also not scale
availableScaleFactor - multiply available parameter by this value, or pass in null to not scale. Passing in 1.0 will also not scale

Returns the percentage not used (free) as an integer

GW:percentageUsed

This Function provides percentage usage synthetic values.
Calculates the usage percentage for a given used metric and a corresponding available metric.

Example:
scalePercentageUsed(summary.quickStats.overallMemoryUsage,summary.hardware.memorySize)

Parameters:
used - Represents a 'used' metric value of how much of this resource has been used such as 'overallMemoryUsage'
available - Represents the totality of a resource, such as all memory available

Returns the percentage usage as an integer

GW:percentageUnused

This Function provides percentage unused/free synthetic values.
Calculates the unused(free) percentage for a given unused metric and a corresponding available metric.
Both the unused metric and available metric can be scaled by corresponding scale factor parameters.

Example:
scalePercentageUnused(summary.freeSpace, summary.capacity)

Parameters
unused - Represents a metric reference value of how much of this resource has not be used (free)
available - Represents the totality of a resource, such as all disk space available

Returns he percentage not used (free) as an integer

GW:divideToPercentage

Given two metrics, dividend and divisor divides them and returns a percentage ratio

Example:
GW:divideToPercentage(summary.quickStats.overallMemoryUsage,summary.hardware.memorySize)

Parameters:
dividend - typically a usage or free type metric
divisor - typically a totality type metric, such as total disk space

Returns the percentage ratio as an integer

GW:toPercentage

Turns a number such as .87 into an integer percentage (87). Also handles rounding of percentages

Example:
GW:toPercentage(summary.quickStats.overallMemoryUsage)

Parameters:
value - the value to be rounded to a full integer percentage

Returns the percentage value as an integer

Math Functions

Functions from Java Math library sample:

• min(n1,n2), max(n1,n2)
• abs
• cos, sin, tanexp, log, sort
• ceil, floor, round
• rint
• pow

See docs: https://docs.oracle.com/javase/8/docs/api/java/lang/Math.html

Math functions should be prefixed by:
Math:
Example:
Math:abs(metric)

Example Formatting

The formatting field uses standard C/Java style formatting strings. Typically, you will only be formatting one number, so the formatting strings should be very simple. Data types used are:

• Integer Numbers %d
• Floating Point Numbers %f

Example of formatting an integer value 2175:

Example of formatting a floating point value 3.141593:

Normal Metric Discovery

When entering a metric name in the Metric Name field, metrics are auto-discovered. As you type into the Metric name field, the names of metrics will be auto-suggested. Cloudera has thousands of metrics. The auto-discovery feature can be very useful in finding the right metric.

Figure: Metric name

Synthetic Metric Auto Suggest

Functions are available in the auto-suggestion list:

Figure: Metric expression


Also, when entering a synthetic expression, configured metrics will be auto-suggested. As you type into the Metric name field, the names of metrics will be auto-suggested.

Figure: Auto suggest

Health Check Status Mappings

Cloudera Health Check statuses are mapped to GroundWork monitor status values in the Status Viewer based on the tables below:

Table: Cluster Status Mapping

Table: Host and Cloudera Service Status Mapping

Table: Metric Status Mapping