Overview

This section contains information common to all GroundWork RAPID-based Feeders.

CONTENTS 1.0 Common Architecture 2.0 Common Operations 3.0 Common Configuration

RELATED RESOURCES RAPID-based Feeders

WAS THIS PAGE HELPFUL? Leave Feedback

1.0 Common Architecture

A GroundWork RAPID-based feeder generally has this architecture. The GroundWork RAPID-based feeder application is built on top of the Feeder library, which is in turn built on top of the RAPID library, which ultimately provides a development API to the GroundWork REST API.

Figure: Common Architecture

2.0 Common Operations

All GroundWork RAPID-based feeders have a common set of functions and operations. For example, all of them have a set of hosts and services for reporting errors and feeder-specific metrics through. This section describes these commonalities.

2.1 Feeder Health Services and Metrics

A RAPID feeder's health is reported by itself through GroundWork host groups, hosts, services, events and notifications. Upon state changes of these hosts and services, GroundWork events and notifications are generated just like other services in the system. The following GroundWork objects structure is created by all GroundWork RAPID feeders:

• A health host group
• A health host for each GroundWork REST server configured (aka endpoint)
• A feeder health service
• A set of retry cache services
• A set of feeder specific metrics

For example, a Cacti feeder is configured to integrate with two GroundWork servers (aka endpoints) called 'parent' and 'standby', with the health host group configured to be called 'cacti_feeder'. When everything is working normally without error, Status Viewer on both 'parent' and 'standby' will include these GroundWork objects:

Figure: Health Host Group

In general, these health services and metrics are published to all GroundWork endpoints configured for the feeder. If an endpoint is unreachable, this information is reported to all reachable endpoints via that endpoint host's health service.

2.2 Common Health Services

The following feeder services are common to all RAPID-based feeders (1).

2.3 Failure Mode and Retry caches

In the event of the feeder failing to reach a GroundWork endpoint because it is unavailable or unreachable for some reason, the feeder will continue to store data in a retry cache for that endpoint. When the GroundWork endpoint is reachable and operational again, the retry cache will processed by prepending it to the data retrieved for the cycle.

Features

A number of features are available to support feeder resiliency through retry caching:

• Configurable cache size monitoring and alerting
• Configurable and automatic cache truncation of oldest entries
• Configurable and automatic time-based aging out of cache entries
• Configurable file system location

For details on configuring these features, see the Retry Caching configuration section.

Retry Cache Structure

A retry cache is a file containing JSON encoded timestamped rows. Each timestamped row contains a set of data that is consumable by the feeder that owns it. The name of the retry cache file follows the pattern: <host on which feeder is running> <feeder name> <endpoint name>. For example, if a cacti feeder is running on host 'cactichild1', and is configured with a GroundWork endpoint ‘parent’, then the file name will be cactichild1_cacti_feeder_parent.

Oldest timestamped rows are stored at the start of the cache file, and newest rows at the end, much like a web server log file.

2.4 Feeder Endpoint Uniqueness

RAPID feeders create a UUID for each GroundWork server endpoint. For example, a Cacti feeder has two GroundWork endpoints configured, 'parent' and 'standby'. Both of these endpoints will have it's own configuration, in which a UUID is created. Whenever GroundWork objects such as hosts, services, host groups etc, are created on 'parent', then the 'parent' UUID is assigned to those objects, and similarly the 'standby' UUID is attached to objects created on 'standby'.

2.5 Interactive Mode - a How-To

Normally the feeder runs as a daemonized process controlled by the GroundWork supervise subsystem. All RAPID feeders allow only one instance of their feeder process to be running at a time. However, sometimes it is necessary to run the feeder interactively to use a feeder-specific command line option. To use a RAPID feeder in interactive mode, follow these steps:

1. Disable the GroundWork supervise service and terminate the feeder process
   1. Locate the feeder's subdirectory in /usr/local/groundwork/core/services, and move the 'run' file to 'norun'. For example, the Cacti feeder is in /usr/local/groundwork/core/services/feeder-cacti.
   2. Identify the feeder's process id and use kill -s SIGHUP <pid> to terminate it.
2. Modify logging to see output in the terminal (optional step)
   1. See the Common Configuration Logging section for details and instructions.
   2. If the logging configuration is not modified, watch the log file for output.
3. Run the feeder interactively
   1. Identify the fully qualified feeder name by looking in the 'norun' file created above. Usually the feeder scripts are located in /usr/local/groundwork/foundation/feeder and the names are self explanatory.
   2. Identify which user the supervise subsystem runs the feeder as, and switch user to that user. Usually this user is user nagios. This information is also in the 'norun' file above.
   3. Run the feeder interactively with options. All GroundWork RAPID feeders support the -h option which provide details about which command line options they support.
4. To restore the feeder back to a supervised daemonized process, revert any logging configuration changes so that output only goes to the feeder logfile, and move the 'norun' file back to 'run'. The supervise subsystem will automatically start the process back up.

2.6 Cleanup Mode - a How-To

All GroundWork RAPID feeders support a cleanup mode of operation. In this mode, all objects created by the feeder are removed from each configured GroundWork server endpoint. To use this mode, run the feeder in interactive mode with the -clean command line option. The feeder will then display what it will remove and ask for confirmation before proceeding (unless the -yes option was also supplied).

3.0 Common Configuration

3.1 General Configuration Architecture

All GroundWork RAPID feeders use the same configuration architecture.

Figure: General Configuration Architecture

Master Configuration

Each feeder has it's own master configuration file which defines general feeder settings such as:

• Which GroundWork servers (endpoints) the feeder should work against
• Whether the feeder is enabled/disabled globally
• How frequently to process things
• Logging settings

The master configuration of a feeder also contains endpoint-independent settings specific to that feeder. These are usually related to controlling data that the feeder integrates into GroundWork. Examples of feeders and their master config feeder-specific settings are:

• Cacti feeder: database connection and threshold qualification settings
• Log Bridge feeder: elasticsearch node(s) settings
• SCOM feeder: database connection settings, processed events save options, and maximum event age settings

The master configuration files are located in /usr/local/groundwork/config. They typically have a self explanatory reference to the feeder name in the file name. For example, the Cacti feeder uses cacti_feeder.conf, and the SCOM feeder scom_feeder.conf.

For more details and instructions on configuring endpoints, see the comments and information in each feeder master configuration.

Endpoint Configuration

Each endpoint configuration file contains general settings specific to GroundWork server endpoints such as:

• A web services configuration file setting which in turn tells the feeder how to communicate with the GroundWork server endpoint
• Various REST API operation bundle size options
• The UUID for this feeder's endpoint (also known as the agentId)
• Retry cache age limits
• Host group for feeder health services metrics

The endpoint configuration file can also contain endpoint-specific settings for it's feeder. For example, the SCOM feeder contains many SCOM event to GroundWork data model mapping rules for each endpoint.

Web Services Configuration

A web services configuration file defines GroundWork REST API credentials and URL's. Each endpoint configuration points to its own web services configuration file. When the feeder is processing an endpoint, it uses these properties from its web services configuration:

• webservices_user / webservices_password - these are the web services credentials
• foundation_rest_url - this url gives defines the web protocol, hostname, port and api url

Full Configuration Example

A Cacti feeder is configured with two GroundWork server endpoints, 'parent' and 'standby', in the Cacti feeder's master configuration, cacti_feeder.conf. The parent endpoint configuration file is defined as parent.conf, and the standby as standby.conf. In each of these endpoint configurations, the ws_client_config_file setting defines what the web services configuration file is ws_parent.properties and ws_standby.properties.

Figure: Full Configuration Example

3.2 Make Configuration Changes - a How-To

Generally making changes to either the master configuration of endpoint configuration files, the feeder process needs restarting. To do this, find the process id of the feeder, and use kill -s SIGHUP <process id> to stop it. The GroundWork supervise subsystem will automatically start the feeder back up, reading in the new configuration.

The only exceptions to this requirement apply to the following settings in the master configuration. Changes to these are picked up automatically without a restart of the feeder:

• Enabling/disabling the feeder (feeder_enabled setting)
• The endpoint directives (endpoint = …) - note: depending on which feeder, changes can take a few feeder cycles to take effect

3.3 Creating Endpoint and Web Services Configurations - a How-To

When a new endpoint needs creating follow these steps in this order, in the /usr/local/groundwork/config directory:

1. Copy an existing web services properties file. There will usually be a ws_client.properties file which can be used. Copy it to ws_client_<endpoint name>.properties and then edit it as follows:
   1. Update the foundation_rest_url setting to point to the GroundWork server endpoint.
   2. Update webservices_user and webservices_password if necessary.
2. Copy an existing endpoint config file for this feeder. It usually has the format <feeder name>_<endpoint name>.conf. In this new endpoint configuration file, do the following:
   1. Important: Set the guid property to 'undefined' ie guid = undefined. This will cause the feeder to generate a new UUID for this endpoint.
   2. Set the ws_client_config_file property value to be the fully qualified name of web services property file created in step 1 above.
   3. Review and change other settings appropriately for this new endpoint.
3. Make sure the new files have ownership nagios:nagios.
4. Create a new endpoint entry in the feeder's master configuration. This new entry consists of the new endpoint name, and a fully qualified path to the new endpoint configuration file.

3.4 Enabling/Disabling the Feeder and its Endpoints

GroundWork RAPID-based feeders are by default disabled at the master configuration level. This means that no processing of any endpoint configurations is done. However, while disabled, the feeder is waiting quietly to be enabled by setting feeder_enabled = yes.

Processing of endpoints defined in the master configuration can also be turned on and off. This is done by changing the enable_processing setting to yes or no. By default, the endpoint configurations are enabled.

3.5 Defining Multiple GroundWork Server Endpoints

Any number of GroundWork server endpoints can be defined in a feeder's master configuration. They will be processed in the order they are defined. Once all endpoints have been processed, the feeder typically waits for a configurable amount of time before beginning its processing cycle again.

3.6 Retry Caching

Cache size limits and notifications

Size limits can be defined for retry caches and are optional per endpoint. The configuration allows warning and/or critical thresholds to be defined. Thresholds work as follows:

• If a warning limit is defined, and the cache size exceeds that limit, then the <feeder name>.retry.cache.size feeder health service will report a warning state and message.
• If a critical limit is defined, and the cache size exceeds that limit, then the <feeder name>.retry.cache.size feeder health service will report a critical state and message. Also, the cache file will be automatically truncated down to no more than this size limit, from the top of the cache file, whilst maintaining the integrity of the cache rows.

Caches location

The location of the directory that contains the retry caches is configurable.

3.7 Logging

All GroundWork RAPID-based feeders use the same system for logging. The configuration for each feeder is located in the feeder's master configuration file under the Logger Settings section. See the comments and instructions in this section for more details.

All feeder’s logs are also configured to be rotated via the 'logrotate' Unix service. See the .../logrotate.d/groundwork file for details.

3.8 Operational Frequency

All GroundWork RAPID-based feeders follow a similar pattern: process enabled endpoints, wait, and repeat. The 'wait' duration here is configurable through the system_indicator_check_frequency setting present in all feeder's master configurations.

3.9 Health Host Group Naming

Feeder health services metrics for each endpoint are reported through a feeder health host group. This is configured using the health_hostgroup setting in each endpoint's configuration. Note that each feeder must have it's own individual and unique health_hostgroup value across all enabled feeders in the GroundWork ecosystem.

3.10 Endpoint Naming

By default, a feeder is configured with one endpoint, 'localhost'. The endpoint name is used as the feeder health hostname for that endpoint.

Single endpoint

If there is only one endpoint configured, and it is the GroundWork server on which the feeder is running, leaving the endpoint name set to localhost is ok. If the GroundWork server endpoint is a different server however, to avoid operator confusion on that remote GroundWork server, it makes more sense to rename the endpoint to something that more clearly identifies it, since localhost is now no longer a 'local host'.

Multi endpoint

If there are multiple endpoints configured, it's highly recommended that to avoid hostname confusion, rename the endpoints to something that clearly identifies them within the entire monitoring environment. For example, rather than leaving one called 'localhost', and adding 'standby', rename 'localhost' to be 'parent'.

3.11 Configuring the Feeder to use SSL - a How-To

To enable the feeder to use SSL:

• Configure the GroundWork server on which the feeder is running to use SSL. Then configure all endpoints used by the feeder to use SSL. To do this, follow GroundWork instructions on configuring a GroundWork server to use SSL.
• If the configured feeder endpoint is the same host on which the feeder is running, no further configuration is necessary for that endpoint.
• Otherwise, update the foundation_rest_url property in the web services properties file for the endpoint to use https. No port definition is required since port 80 access is automatically redirected to the configured SSL port via the Apache configuration. For example, foundation_rest_url = https://standby/foundation-webapp/api|https://standby/foundation-webapp/api.
• See also the force_crl_check option in the endpoint configuration for Certificate Revocation List checking.

3.12 Auditing

All RAPID-based feeders by default create audit trails for the creation and deletion of GroundWork host groups, hosts and services. This can be switched off using the auditing option in the master configuration. The audit trail can be seen GroundWork Administration > Audit Log.

3.13 API Operations Bundling

For efficiency, all RAPID-based feeders provide options for n-ary REST API operations. For example, if a feeder needed to add 150 hosts, it is more efficient to add them in bundles of 50 per REST API call, rather than one at a time. Bundling options exist for REST API operations used for the following GroundWork objects: hosts, host groups, services, events and notifications.

3.14 Configuring Feeder Cycle Frequency

All RAPID-based feeder applications run as daemonized processes under the control of the GroundWork supervise subsystem. They both perform their main functions cycle, then go to sleep for a configurable amount of time before starting a new cycle. The feeder applications follow a similar pattern of performing a sequence of operations in one cycle, waiting for an amount of time, and then moving on to do another cycle. The amount of time that the feeder waits between cycles can be configured using the system_indicator_check_frequency setting in the master configuration file.