Overview

This document contains important information about the GroundWork Monitor Enterprise 7.2.0 release. While this information is not required to install or upgrade, it may be useful to know when examining the new features of GroundWork Monitor that have been added.

This document only discusses changes since 7.1.1. This release of GroundWork Monitor fixes many existing issues, and adds several new features. It is a major upgrade in that we have not only upgraded and updated, but added significant functionality.

In general new features and improvements in GroundWork Monitor 7.2.0 include:

• The option to use Grafana for graphing performance data and InfluxDB to store it (GrafBridge).
• A new Cloud Hub connector for monitoring Cloudera installations
• Many included packages, such as RRDtool, syslog-ng, fping, and lots of supporting libraries, have been upgraded in this release.

SECTION I: Features and Improvements

Cacti

• We have not updated the Cacti version supplied in this release for various reasons. Instead, we have added support for independent Cacti servers to connect to GroundWork servers loosely by feeding data. This allows Cacti users to have the benefits of unified monitoring using the latest releases of Cacti. If you need to use external Cacti servers, contact GroundWork support for instructions on how to do this.

Configuration (Monarch)

• Access to Nagios CGIs
  • Nagios CGIs can now be controlled at the level of contactgroups, not just at the level of individual contacts/users. This can be seen on the second CGI-configuration screen, under Configuration > Control > Nagios cgi configuration > Save and Next >>.
  • Access to the Nagios CGIs can now be set to read-only for certain users (or users in particular contactgroups).
• NoMa commands
  • Standard host-notify-by-noma and service-notify-by-noma commands are now added to the monarch database for use with NoMa notifications. The definitions of these commands are slightly different from what was documented in previous releases, to fix operational problems with the old formulations.
• Monarch externals
  • Monarch externals generation has been extended to support creating per-service-instance copies, along with instance-level and service-level macro substitutions, as part of supporting the forthcoming GDMA Auto-Setup feature. However, this capability can also be used with purely manual setup. In particular, these macro references will now be expanded.
    • $SERVICEDESC$ expands to the full name of the service, as you would expect, including the appropriate instance name suffix in an instance-level external.
    • $BASESERVICEDESC$ expands to the name of the service. For a service without instances, this will be the same value as $SERVICEDESC$. For a service with instances, this will be the name of the base service, without any instance name suffix appended.
    • $INSTANCE$ expands to a simple integer which counts the number of copies of a given external as possibly multiple copies are created for service instances. This macro will generally be used within the square brackets of service-externals lines. For a base service without any service instances defined, this number will be a constant 1.
    • $INSTANCESUFFIX$ expands to the instance name suffix as defined in Monarch, but with a leading underscore (if any) elided. This makes it easier to customize the service externals definitions with a string reflecting just the critical part of the service instance name suffix, while keeping an underscore in the instance name suffix for overall readability of the full service name.
    • $ARG#$ macros ($ARG1$, $ARG2$, etc.) expand to values defined in the Monarch UI at either the generic-service, host-service, or service-instance level, depending on the complexity of your setup and what inheritance options you have chosen.

Cloud Hub

• There is a new connector included in Cloud Hub for Cloudera. This is a new type of connector, and features updated UI screens for configuration, synthetic metrics, support for tsquery, math functions, units conversion, and much more.
• We have also improved reporting of connection failures. If a connector is stopped or fails to connect for some reason, hosts will become "Unreachable" and services "Unknown", with status text that informs the user as to why (e.g., retries exhausted). Prior iterations did not update service status in this situation, leading to errors, especially with infinite retries configured. The default retries is now set to 10, for this reason.
• The VMware connector now supports snapshot monitoring as a service.
• All connectors now support service sync by default, which will remove services tied to API metrics that are selected, but not present (e.g., snapshot metrics). You can turn this off/on in the cloudhub.properties file, either globally or by connector.

Foundation

• We have made some changes to the data types of certain database columns in the GroundWork Foundation database in order to support diverse time-stamp values. This change may impact users who directly utilize database queries rather than the REST API to extract data, or those who use the archive database (also changed) to generate reports. For details, please contact GroundWork Support.
• We added support for turning off (or on) the prefixes added to hosts and services in the status viewer by application type. This allows you to de-clutter the status view if you prefer not to see the prefixes. These are set in the status-viewer.properties file.

Fping

• Fping has been updated to the latest upstream release (4.0). This provides efficiency improvements and makes it easier to support IPv6 hosts. In general, an IPv6 address of a target host will be preferred during fping probing, if the GroundWork server is IPv6-enabled.
• The fping_process.conf config file has been modified to clarify the usage of the delay_between_sends parameter. It applies to all paths for sending data, not just sending via NSCA.
• There is a new fping_exclude_groups option in the fping_process.conf file, to allow excluding hosts directly or indirectly associated with some set of Monarch Groups from the set of those that will be pinged. This might be used on a parent server, for instance, to ignore all hosts that will be pinged instead from child servers.

GroundWork Distributed Monitoring Agent (GDMA)

• Building externals for GDMA can now be simplified by the use of substitutable service-level externals arguments. See the discussion above for Monarch changes.

Notifications (NoMa)

• Standard commands
  • The monarch database now has standard host-notify-by-noma and service-notify-by-noma commands defined (see above). These standard commands are present in a fresh install, and are added during an upgrade if not already present.
• NoMa fixes
  • A large number of fixes have been applied to NoMa to greatly improve the reliability of handling notifications.
  • It is recommended that all notification rules be configured to check the "Rollover if the last rule is reached" option. This will cover extreme cases of repeated alerts by resetting the notification count and ensuring that some additional notifications eventually make it out.
  • Note that Cloud Hub currently only generates alerts on state transitions, not on an ongoing basis when a host is not UP or a service is not OK. This means that notification counting for such hosts and services will not operate as it would for a Nagios-monitored host or service. You will want to construct your notification rules accordingly:
    • Filter on hostgroups and servicegroups that are known to be associated with Cloud Hub.
    • Ensure that a notification for such hosts and services is sent out on the first alert that NoMa sees (make sure the "Notify after # number of notifications (all types)" option contains a 1).
    • Higher-count specifications, the rollover option, and escalations will have no effect, since there will be no follow-on alerts until a final recovery alert is received as the host returns to a UP state or a service returns to an OK state.

Performance Data Handling

• GroundWork GrafBridge (Grafana and InfluxDB)
  • GrafBridge has been added as a new feature. GrafBridge is an integration of several open source components, allowing users to take advantage of the advanced features of Grafana, along with other capable packages we have added to enhance the collection and storage of performance metrics, the graphing of these metrics, and their presentation in dashboards.
  • GrafBridge gets you better graphs and dashboards. The RRD-based graphs we have supplied up till now have been useful and easy to manage, but they have disadvantages as well, including the inherent data summarization that all RRDs commonly enforce. With GrafBridge, performance data is stored in InfluxDB, which saves every point it receives, until it is aged off after 13 months (or optionally longer). No summarization of data occurs.
  • RRD is also significantly slower to render graphs in GroundWork Monitor, due to the server-side generation of a bitmap image for each updated graph. Grafana uses client-side rendering, and is far less impactful on the GroundWork server for the same kind of operation. This improves the server’s speed and scalability, especially when generating and displaying a lot of graphs.
  • The transition from RRD is seamless, and the RRD data can be optionally imported into InfluxDB.
  • InfluxDB time-series data store allows GroundWork customers to use other open source agent and analysis packages, including components of the TICK stack to set up data transfers and graphing for in-depth analysis and monitoring. These additional packages take advantage of the APIs for performance data that are inherent in the InfluxDB project, an open source time series database.
  • GrafBridge includes the GroundWork data source, making it easy to select the individual hostgroups, servicegroups, hosts, and services that you want to graph on your dashboards.
  • After a GroundWork Monitor 7.2.0 installation, InfluxDB and Grafana monitoring profiles are added to localhost, and will be active after the first configuration Commit operation. After upgrading to GroundWork Monitor 7.2.0, these profiles need to be added manually. This can be done using the grafbridge-control using the -add_service_profiles option.

Security Improvements

• Authentication
  • We have closed a security hole related to exposed API tokens in the DOM on certain screens. These could potentially have been used to execute a privileged elevation exploit whereby a logged-in user might have been able to see or change data they were not granted access to.
  • We added a read-only API token to the REST API for dashboard and other read-centric uses.

Component Upgrades

With the release of GroundWork Monitor 7.2.0, many components have been upgraded to the latest available versions. The following list shows the new version of the key components that have been updated in this release:

• Upgraded to PostgreSQL 9.6.5
• Upgraded PHP to 5.6.31
• Upgraded to Nagios 4.3.4
  • Addresses a number of security issues with older Nagios releases
  • The Bronx event broker has been revised to interoperate with the upgraded Nagios version
  • Bronx has been made more reliable in its treatment of data across a Nagios stop/start cycle, in two ways:
    1. Check results that have been received and queued by Bronx but not yet processed are now saved if Nagios is shut down before it gets to them, and picked up when Nagios restarts. This prevents possible data loss around the time of an ordinary Commit operation.
    2. In previous releases, there has inadvertently been a very small (up to 1 second) window right after Bronx starts up when it could receive but would then discard incoming data. This window has now been discovered and closed.
  • nagios2collage_eventlog.pl now supports sending certain events with alternate app types.
• Upgraded NeDi to 1.7.090
  • NeDi upgrade plus both upstream and GroundWork patches
• GroundWork Distributed Monitoring Agents (GDMA)
  • This GroundWork Monitor Enterprise 7.2.0 release upgrades the included Solaris and AIX GDMA builds to the 2.5.0 or 2.5.1 release, respectively, which folds in the builds that were provided separately after the GroundWork Monitor Enterprise 7.1.1 release was made available.
• Setting Up a Newer JDK
  • For those who need a newer JDK for security reasons you can install the Zulu OpenJDK from Azul Systems. Internet access is required because packages are pulled from the Azul Systems repository:

    cd /usr/local/groundwork/tools/system/setup/scripts/
    ./install-zulu.py

• Upgraded RRDTool to 1.7.0
• Upgraded NET-SNMP to 5.7.3
• Upgraded syslog-ng to 3.12.1
• Many more upgrades to underlying packages

SECTION II: Known Issues and Limitations

Installation Issues

• Installer needs Root Account Login: The root user login is required on installation to set up aspects of the Grafana package. If you instead enter the admin user and password and not root, the installer will complete without error, but will not complete the product setup. Please enter the root credentials when prompted (default is root, password root).
• NeDi has a dependency on the netstat command which is not satisfied by GroundWork libraries. If you encounter an error in NeDi that lists netstat as not found, this can be solved by installing the net-tools package (e.g., apt-get install net-tools in Ubuntu).
• The installer has an option to set the timezone --php_timezone that should not be used. If used, it can perturb the function of several services, notably BSM.

Upgrade Issues

• You may need a new license: The license version restrictions for certain licenses do not allow 7.2.0 to work. Please ask support if you want to check if you need a new key.
• RAPID-based Feeder files: If you use the RAPID feeders for Cacti, LogBridge, or other systems, read this section:
  • The installer when upgrading to 7.2.0 does not restore various RAPID-based Feeder files in the /usr/local/groundwork/config directory. You will need to manually do the following steps if you are using the Cacti or LogBridge feeders:
    • Copy the master configuration files from the installer backup config directory (e.g., /usr/local/groundwork/backup-2017-11-01/config):
      • Master configuration files: cacti_feeder.conf, logbridge_feeder.conf, and gwevents_to_es.conf
    • Copy the feeder GroundWork endpoint config files:
      • Defined in the endpoint directives in the master config files: (e.g., cacti_feeder_localhost.conf)
    • Copy the web services properties files that are defined in the endpoint config files with the ws_client_config_file (e.g., /usr/local/groundwork/config/ws_client.properties):
      • Make sure all of the files you just copied have nagios:nagios ownership
• RAPID-based Feeder log file symlinks: The installer when upgrading to 7.2.0 does not restore various RAPID-based Feeder log file symlinks in the /usr/local/groundwork/logs directory. The logfile names are defined with the logfile directive in the master config files.
• Custom portal root user: The installer when upgrading to 7.2.0 does not restore any custom portal root user name definition that was in the gatein.properties file. To fix:
  • Restore config/gatein.properties from the backup.
  • Run the the following (Note: The name of the jarfile will be different in the final release and should not have -SNAPSHOT in it):

    /usr/local/groundwork/java/bin/java -cp \
        /usr/local/groundwork/jpp/modules/com/groundwork/security/main/groundwork-jboss-security-7.2.0-SNAPSHOT.jar \
        com.groundwork.core.security.GateinConfigurationUtils -superuser svc_gwroot

• Custom portal changes for LogBridge: If you use LogBridge, read this section and the next:
  • The installer when upgrading to 7.2.0 does not restore custom GroundWork portal changes made for the LogBridge feeders. Without these changes, you'll get an error when trying to access Kibana from the Log Analytics menus. To restore previous changes that will fix this:

    cd /tmp
    cp {the groundwork backup directory}/foundation/container/jpp/standalone/deployments/portal-groundwork-base.war .
    jar xf portal-groundwork-base.war WEB-INF/portlet.xml
    rm -f portal-groundwork-base.war
    cp /usr/local/groundwork/foundation/container/jpp/standalone/deployments/portal-groundwork-base.war /tmp
    jar uf portal-groundwork-base.war WEB-INF/portlet.xml
    cp portal-groundwork-base.war /usr/local/groundwork/jpp/standalone/deployments/portal-groundwork-base.war
    service groundwork restart

• LogBridge: After upgrading to 7.2.0, if using the LogBridge module, then install the latest and greatest logbridge_feeder.pl and gwevents_to_es.pl scripts. Note that 7.2.0 includes the Search::Elasticsearch Perl modules so no rebuilding/reinstallation of them is required.
• LOGSTASH: After upgrading from earier versions of the LogBridge feeders, you might see the LOGSTASH application type next to hosts and services that these feeders created. This application type has been updated to LOGBRIDGE. You'll need to do some SQL to update the applicationtype on tables like servicestatus, host, and hostgroup.
  • To find the new applicationtypeid value for LOGBRIDGE and the old id for LOGSTASH:

    select * from applicationtype;

  • Show the agent id and show potential records to be updated. Copy the agentid for the new LOGBRIDGE group:

    select * from host where applicationtypeid = [old applicationid] or applicationtypeid = [new applicationid];
    update host set agentid = '[new agentid]' where applicationtypeid = [old applicationtypeid];
    update host set applicationtypeid = [new applicationtypeid] where applicationtypeid = [old applicationtypeid];
    select * from servicestatus where applicationtypeid = [old applicationtypeid] or applicationtypeid = [new applicationtypeid];
    update servicestatus set agentid = [new agentid] where applicationtypeid = [old applicationtypeid];
    update servicestatus set applicationtypeid = [new applicationtypeid] where applicationtypeid = [old applicationtypeid];

  • Once unassigned, you should be able to delete the LOGSTASH application type from the applicationtype table with:

    delete from applicationtype where applicationtypeid = [old applicationtypeid];

• Centos 7 known issues: If you install GroundWork on Centos 7 in a virtual machine, read this section:
  • Centos has published known issues that may impact performance and booting on virtual environments, see Centos 7 Release Notes.

Limitations and Work-arounds

• SNMP trap processing
  • Read the following if you use SNMP trap processing:
    • Best Practice: The SNMPTrap process is a brittle and difficult to configure subsystem. If too many traps come in, the disk creates enough traps that even stating the spool directory where they are temporarily stored by snmptrapd causes a huge delay enough that most processes would either time out or hang. If you get behind, where the rate of incoming traps is greater than the rate you are processing them, problems will show up around disk space reporting, inode availability, disk io, application and network timeouts. You will eventually run out of inodes. Since we turn on this the snmptrap processes at startup but don't have any definitions defined, this is an easy attack vector to take out GroundWork from the network without needing to authenticate. Until a remedy, it is recommended that all systems where snmptraps are not being used (e.g., child servers) have the processes disabled:

      service groundwork stop snmptrapd
      chmod -x /usr/local/groundwork/common/scripts/ctl-snmptrapd.sh
      service groundwork stop snmptt
      chmod -x /usr/local/groundwork/common/scripts/ctl-snmptt.sh

      Note, typical fresh install of Linux has iptables local firewall blocking UDP. A further best practice is to ensure continued control of iptables and other firewalls, limiting incoming access to only what is needed. See How to determine ports used by GroundWork. Typically Linux as installed will block all but SSH. Here is the iptables file content for a fresh installation:

      # Generated by iptables-save v1.4.7 on Sun Nov  5 21:37:28 2017
      *filter
      :INPUT ACCEPT [0:0]
      :FORWARD ACCEPT [0:0]
      :OUTPUT ACCEPT [3626:1478814]
      -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
      -A INPUT -p icmp -j ACCEPT
      -A INPUT -i lo -j ACCEPT
      -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
      -A INPUT -j REJECT --reject-with icmp-host-prohibited
      -A FORWARD -j REJECT --reject-with icmp-host-prohibited
      COMMIT
      # Completed on Sun Nov  5 21:37:28 2017
      

      Here is the typical iptables command to permit snmptrap UDP port 162 (you won't want to do this unless you are looking to use SNMP TRAPS)

      iptables -A INPUT -m state --state NEW -m udp -p udp --dport 162 -j ACCEPT
      

• GroundWork GrafBridge
  • There are some known limitations to the initial release of GrafBridge:
    • Data source selection to service level:
      While it is possible to have several metrics per service in GroundWork, when using Grafana with the GroundWork data source for InfluxDB (the default), one can select a service as the finest level of detail in choosing what to graph. You can turn thresholds on or off, and you can suppress unwanted metrics with a display option called series-specific overrides, so this should not affect your ability to graph what you want, but we may decide to enhance the selection to metric level in future releases or technical bulletins.
    • No support for Counter data types:
      The GroundWork Grafana data source only supports "Gauge" type metrics, and renders counters as increasing in status viewer.

• Host Identity
  • There are some known issues when using host identities.
    • If you add a host alias record for a non-existent host (e.g., if you mistype the hostname or don't accept a suggested name), the system allows this but does not deal with the database identity cleanly. While there is no resultant data corruption, the results for the aliased hosts will be identified as coming from a different host. It is not recommended to use host identity records in this way.
    • If you create an alias, the alias host will be in all the hostgroups of the aliased host(s). This is by design. If, however, you delete the alias, all the aliased hosts will re-appear properly, but the alias host will retain the hostgroups of the aliased hosts. This can be fixed by deleting the hostgroups or connectors associated, or with a manual correction. Contact GroundWork Support for instructions if needed.

• Cloud Hub (UPDATED)
  • The AWS connector for Cloud Hub sometimes adds services (typically starting with "info.") to AWS hosts which are not updated and eventually fall to the "unknown" state. This can be fixed by enabling the parameter:

    synchronizer.services.amazon.enabled=true

    in the file:

    /usr/local/groundwork/config/cloudhub.properties

    and redeploying cloudhub with:

    touch /usr/local/groundwork/jpp/standalone/deployments/cloudhub.war.dodeploy

    After a few moments, these services will vanish from the Status viewer, as will any services specified in the Cloud Hub profile that no longer exist (e.g., metrics on deleted volumes).

• Miscellaneous changes from previous releases
  • Uninstalling GroundWork Monitor is now more aggressive about removing files that may have been installed outside a standard installation or upgrade of GroundWork Monitor. It is important to backup any customizations before performing a new installation or an upgrade, or uninstalling using the groundwork uninstall program.
  • The Cloud Hub connector pages display a "GroundWork Server Version" field with a grayed out drop-down. This is a carry over from previous versions and can be ignored.
  • Aliasing feature is improved. However if you happen to delete an Alias that included a Nagios monitored host, that host will disappear from Status Viewer tree until the next Monarch Commit is performed. Monitoring will continue but alerts based on Hostgroup for that host may fail within the NoMa tool. This issue is slated to be addressed in a patch.

SECTION III: Announcements as of Version 7.2.0

Supported versions of GroundWork

• GroundWork Monitor versions 7.0.2 (including 7.0.2-SP03) are now end-of-life. Customers using older versions are advised to contact GroundWork Support regarding upgrade options.
• GroundWork Monitor version 7.1.0 will be end-of-life with the next release.

Linux supported versions

The versions of Linux now supported by GroundWork have changed. See the System Requirements for details. If you try to install GroundWork 7.2.0 on an unsupported version it will warn you, but it will allow you to proceed. This warning does not appear when using unattended mode in the installer.

Supported architectures

• GroundWork Monitor Enterprise only supports installation on 64-bit platforms. Monitored nodes can be 32-bit or 64-bit but GroundWork itself must be deployed to 64-bit.

Browser compatibility

• This version of GroundWork has been tested with the latest versions of Firefox and Google Chrome, as well as Internet Explorer. Chrome is recommended.
• Recent versions of Firefox and Chrome browsers require that all certificates contain a Subject Alternative Name in order to be valid. Most commercial certificates populate this field when generated. Self signed certificates created using instructions from previous versions do not populate this field. The new procedure for https creates certificates with this field automatically if commercial certificates are not provided.

SECTION IV: Details of Fixed Issues Since Release 7.1.1

This release incorporates a good bit more than the usual assortment of bug fixes. For a full listing, please contact GroundWork support. A partial listing of the most relevant (from a user perspective) issues fixed are listed here. If you want to know if a particular issue you have reported is fixed in GroundWork Monitor 7.2.0, please see your support case where you reported it, or contact support with your question.

Copyright 2004-2017 GroundWork Open Source, Inc. ("GroundWork"). All rights reserved. Use is subject to GroundWork commercial license terms. GroundWork Monitor products are released under the terms of the various public and commercial licenses. For information on licensing and open source elements please see Licenses used in GroundWork. GroundWork, GroundWork Open Source, GroundWork Monitor Professional, GroundWork Monitor Open Source, GroundWork Community Edition, GroundWork Monitor Enterprise, GroundWork Foundation, GroundWork Status Viewer, Monarch, and GroundWork Guava are trademarks of GroundWork Open Source, Inc. Other trademarks, logos and service marks (each, a "Mark") used in GroundWork’s products, including Nagios, which is a registered trademark of Ethan Galstad, are the property of other third parties. These Marks may not be used without the prior written consent of GroundWork Open Source or the third party that owns the respective Mark.