Overview

This document contains important information about the GroundWork Monitor Enterprise 7.2.1 release.

This is where we discuss changes made since 7.2.0. The 7.2.1 release of GroundWork Monitor fixes many existing issues, adds a few new features, and enhances several areas. It is a minor upgrade from 7.2.0, and is recommended for 7.1.1 and 7.2.0 users.

The new features and improvements in GroundWork Monitor 7.2.1 include:

• A new Cloud Hub connector for monitoring Microsoft Azure
• A new Cloud Hub connector for NeDi
• Updates to NeDi, and the addition of NoDi
• Updated versions of Grafana and InfluxDB
• Updates to supporting libraries
• Updates to GDMA, including GDMA Auto-Setup
• New SLA dashboards

SECTION I: Features and Improvements

Cloud Hub

• There is a new connector included in Cloud Hub for Microsoft Azure. This connector features updated UI screens for configuration, synthetic metrics, math functions, units conversion, and more.
• There is a new connector for NeDi, allowing you to use the native policy monitoring and device stats monitoring NeDi can do in GroundWork.
• We have also further improved reporting of connection failures. If a connector fails to connect for some reason, a connector-specific service will be updated to a warning state, or critical if you run out of retries (hosts will still become "Unreachable" and services "Unknown" if retries are exhausted).
• Several other connectors have been updated to use the new UI for connection and configuration, and to expose synthetic metrics as much as possible.

NeDi

• For GWMEE 7.2.1, NeDi has been upgraded to the 1.8.100 release, plus a few GroundWork patches.
• For the first time in a GWMEE upgrade, the content of the nedi database will be preserved across the upgrade, if that database was first created in GWMEE 7.2.0. This will happen only in a direct upgrade from GWMEE 7.2.0 to 7.2.1. The upgrade process can handle a database version of either 1.7.090 or 1.7.090p1. You can check your current nedi database version using the following commands:

  su nagios
  echo "select value from system where name='version'" | \
      /usr/local/groundwork/postgresql/bin/psql -t -U nedi
  exit
  

  If you are upgrading from a release prior to GWMEE 7.2.0, you will have a nedi database version of 1.6.100 or earlier. In that case, the upgrade code cannot handle the conversion, and it will generate an error message during the upgrade process. To avoid that, when you reach the stage in a multi-release upgrade where you have GWMEE 7.2.0 installed, run these commmands to initialize the nedi database:

  su nagios
  /usr/local/groundwork/nedi/nedi.pl -i
  exit
  

  You will be prompted for the PostgreSQL administrative user (postgres) and its password. All data in the nedi database will be lost, but this is unavoidable in this case. It will, however, be repopulated the next time you run NeDi.

• Starting with GWMEE 7.1.0, the scheduling of NeDi cron jobs was changed from once every four hours to once every hour. However, the RRD file creation parameters were not changed at the same time, resulting in a discrepancy of the resolution of the collected data. To address that, after you have upgraded to GWMEE 7.2.1, you should delete all of the existing NeDi RRD files and allow them to be re-created using the new RRD parameters in the nedi/nedi.conf and nedi/nodi.conf files.

  rm -rf /usr/local/groundwork/nedi/rrd/*
  

  This also means that when you are merging old pre-upgrade configuration data into the post-upgrade configuration files, you should leave the rrdstep value in the /usr/local/groundwork/nedi/nedi.conf file set to 3600.

• nfdump is added to GroundWork Monitor. This is useful in monitoring and graphing Netflow and Sflow data, as well as monitoring for particular types of flows and traffic, and getting alerted when they occur. Please see the Bookshelf entry for NeDi for details, or contact GroundWork Support.
• We now provide a NeDi Connector, and as such we are officially deprecating use of the NeDi feeder in favor of the (much easier to use) connector. We recommend the old feeder be disabled in the config/nedi_feeder.conf file:

  enable_processing = no
  

  when you are merging in old config-file values.

• A NeDi add-on, called NoDi (for Node Discovery), is also included in GWMEE 7.2.1. See the System > NoDi screen in NeDi. Please note that using this tool will allow SQL to be entered into the user interface directly. Enable it only if you are ok with the risk this entails.
• NoDi will use a new PostgreSQL database, named nedi_nodes. The nedi/nodi.conf file will contain the access credentials for this database. The access credentials will also be stored in the /usr/local/groundwork/users/nagios/.pgpass file. If you want to change those credentials, both files should be changed in tandem.
• The Nedi flowi.pl and moni.pl processes will be set to run automatically, as possible sources of additional monitoring data.
• NeDi relies on netstat, a deprecated utility no longer included by default in some Linux distributions. You will need to install it using your package manager, for example:

  yum install net-tools
  

GroundWork Distributed Monitoring Agent (GDMA)

• GDMA 2.6.1 agents now support Auto-Setup. GDMA Auto-Setup is an advanced facility for having each GDMA client discover the facilities that need monitoring on that machine, reporting back that data, and having the server use it to configure or reconfigure the GDMA monitoring on that client. You can now assign host and service profiles as well as individual services to hosts, based on the results of the Auto-Setup "sensors", e.g., a file's presence or absence, its content, or the presence or absence of an open port, running services, etc.
  
  This means that GDMA can be fully automated in deployments, and can adapt to changes in stack components as they are deployed.
• Windows and Linux GDMA have both been updated to the 2.6.1 release, which includes support for GDMA Auto-Setup. The other platforms will follow as demand arises.
• Logfile rotation is now built into GDMA 2.6.1, and operates automatically. With this release, it is no longer necessary to disable logging to prevent unlimited logfile growth.
• Starting with GDMA 2.6.1, permissions on the GDMA client config files are now restricted, to block general access to sensitive information.
• With respect to Windows GDMA 2.6.1 specifically:
  • This is the first official release to support TLS 1.2 without the use of any overlay files.
  • This release may be installed to run the GDMA service as a non-administrative user, see Windows GDMA as non-admin user. The customer may choose from a variety of different types of users for this purpose. We expect the NT SERVICE\GDMA virtual user to become the most popular choice.
  • On a 64-bit platform, 64-bit Powershell and VB (cscript) will be invoked in preference to the 32-bit versions, when running service checks that invoke these languages. This solves a longstanding issue with accessing certain long-width counters.
  • It is now possible to install Windows GDMA without immediately starting the GDMA service. This mirrors what has been available on the other platforms for a long time. It allows you to adjust the gdma\config\gdma_auto.conf configuration before the GDMA service is started the first time. You might want to do this, for instance, to enable logging or to enable GDMA Auto-Setup. For unattended installs, this is controlled by the command-line --gdma_service_start value option, the value being one of yes, no, true, false, 1, or 0.
• In each GDMA 2.6.1 build, OpenSSL has been updated to the current upstream release.
• Hostgroup-assignment processing during Auto Registration is now modified so it will not assign hostgroups to existing hosts that are already assigned to some hostgroup. This makes it easier to move GDMA hosts out of the single default hostgroup (typically "Auto-Registration") and not have them pop back up there at some future time. It is still possible to have full local control over such behavior; see the new assign_hostgroups_to_existing_hostgroup_hosts option in the register_agent.properties configuration file.

Nagios

• There has been no new upstream Nagios Core release since GWMEE 7.2.0, so the version of Nagios that we include remains the same.
• We are including the Monitoring Plugins 2.2 release, thereby providing a long-overdue uplift to the available Nagios plugins.
• We are also including the Nagios Plugins 2.2.1 release, in case there are any significant differences in the plugins you might need to take advantage of.
• Except for a few particular plugins that are simply being upgraded in place, the content of the nagios/libexec/ file tree remains the same, with the older release of the Nagios plugins. We are doing this for stability across upgrades. The new plugins are all being provided under the /usr/local/groundwork/plugins/ directory, with various subdirectories for different identifiable collections of plugins.
• The /usr/local/groundwork/plugins/customer/ directory is being provided as a standard safe place for customers to locally install additional plugins, either from other third-party sources or written by you, our customers. GroundWork itself will not populate any content into this directory. In future releases, the contents of this directory will be preserved across an upgrade, simplifying the upgrade process.

Configuration (Monarch)

• Support for Auto-Setup
  • Monarch now includes the server side components to fully support the GDMA Auto-Setup feature.
• Check period Override is now host-specific
  • In the Host Detail screen, you may now override the host-template "Check period" value with a host-specific value.
  • In a fresh install, the value for the host-check interval in the generic-host host template has been changed from 0 to 10. (In order not to disturb working existing customer configurations, this value is not changed during an upgrade.) This modification reflects an underlying change in the behavior of Nagios, which we had not previously recognized: running regularly-scheduled host checks is no longer actively discouraged. The pop-up help text for this option has been updated to reflect our new understanding.

Performance Data Handling

We have extended the perf-data processing to allow sending to multiple Foundation REST API endpoints. This facility may find use in some advanced distributed monitoring setups.

Grafana

As of 7.2.0, we have included Grafana for creating graphical dashboard displays. Grafana can do much more than this, and we have updated to the 5.1.3 version in 7.2.1, so you can now access several more features:

• Additional data sources are supported (including Postgres)
• Dashboards can be arranged into folders
• Trend and threshold enhancements
• A new look and feel for the Grafana user interface
  Of course, you can still use the GroundWork templates and the GroundWork data source to access the performance data you get from Cloud Hub and Nagios, etc. in GroundWork.

SLAs and SLA Dashboards

Perhaps the most radical change in this release of GroundWork is SLA management and SLA dashboards. These facilities are not available in the GroundWork Monitor Core edition, and require a commercial license.
SLA management can use any GroundWork Service as a base, though it works best when paired with Business Service Management (BSM). With SLA management, you can track compliance with many different SLAs. Compliance is as measured by GroundWork Monitor, and any other sources you input.
SLA Dashboards are a new way to display data in GroundWork Monitor, specifically for SLA compliance measures. The dashboards are highly flexible and can be adapted to many scenarios.

Component Upgrades

With the release of GroundWork Monitor 7.2.1, many components have been upgraded to the latest available versions. The following list shows the new version of the key components that have been updated in this release:

• Upgraded to PostgreSQL 9.6.9
• Upgraded PHP to 5.6.36
• Upgraded Grafana to 5.1.3
• Upgraded InfluxDB to 1.5.3
• Upgraded NeDi to 1.8.100
• Upgraded GDMA to 2.6.1
• Many more upgrades to underlying packages

SECTION II: Known Issues and Limitations

Installation Issues

• Installer needs Root Account Login: The root user login is required on installation to set up aspects of the SLA Dashboards. If you instead enter the admin user and password and not root, the installer will complete without error, but will not complete the product setup. Please enter the root credentials when prompted (default is root, password root, but this will likely be different, especially if you use LDAP/AD integration).
• NeDi has a dependency on the netstat command which is not satisfied by GroundWork libraries. If you encounter an error in NeDi that lists netstat as not found, this can be solved by installing the net-tools package (e.g., apt-get install net-tools in Ubuntu).
• The installer has an option to set the timezone --php_timezone that should not be used. If used, it can perturb the function of several services, notably BSM.

Upgrade Issues

• You may need a new license: The license version restrictions for certain licenses do not allow 7.2.1 to work. Please ask support if you want to check if you need a new key.
• RAPID-based Feeder files: If you use the RAPID feeders for Cacti, LogBridge, or other systems, read this section:
  • The installer when upgrading to 7.2.1 does not restore various RAPID-based Feeder files in the /usr/local/groundwork/config directory. You will need to manually do the following steps if you are using the Cacti or LogBridge feeders:
    • Copy the master configuration files from the installer backup config directory (e.g., /usr/local/groundwork/backup-2018-06-21/config):
      • Master configuration files: cacti_feeder.conf, logbridge_feeder.conf, and gwevents_to_es.conf
    • Copy the feeder GroundWork endpoint config files:
      • Defined in the endpoint directives in the master config files: (e.g., cacti_feeder_localhost.conf)
    • Copy the web services properties files that are defined in the endpoint config files with the ws_client_config_file (e.g., /usr/local/groundwork/config/ws_client.properties):
      • Make sure all of the files you just copied have nagios:nagios ownership
• RAPID-based Feeder log file symlinks: The installer when upgrading to 7.2.1 does not restore various RAPID-based Feeder log file symlinks in the /usr/local/groundwork/logs directory. The logfile names are defined with the logfile directive in the master config files.
• Custom portal root user: The installer when upgrading to 7.2.1 does not restore any custom portal root user name definition that was in the gatein.properties file. To fix:
  • Restore config/gatein.properties from the backup.
  • Run the the following:

    /usr/local/groundwork/java/bin/java -cp \
        /usr/local/groundwork/jpp/modules/com/groundwork/security/main/groundwork-jboss-security-7.2.1.jar \
        com.groundwork.core.security.GateinConfigurationUtils -superuser svc_gwroot

• Remote Postgres Server Upgrade throws an error about postgres-xtra-functions.sql. This happens because there is a missing file on the remote postgres installation on some systems.
  • The Upgrade instructions for Remote Postgres installations completely address fixing this error.
• Custom portal changes for LogBridge: If you use LogBridge, read this section and the next:
  • The installer when upgrading to 7.2.1 does not restore custom GroundWork portal changes made for the LogBridge feeders. Without these changes, you'll get an error when trying to access Kibana from the Log Analytics menus. To restore previous changes that will fix this:

    cd /tmp
    cp {the groundwork backup directory}/foundation/container/jpp/standalone/deployments/portal-groundwork-base.war .
    jar xf portal-groundwork-base.war WEB-INF/portlet.xml
    rm -f portal-groundwork-base.war
    cp /usr/local/groundwork/foundation/container/jpp/standalone/deployments/portal-groundwork-base.war /tmp
    jar uf portal-groundwork-base.war WEB-INF/portlet.xml
    cp portal-groundwork-base.war /usr/local/groundwork/jpp/standalone/deployments/portal-groundwork-base.war
    service groundwork restart

• LogBridge: After upgrading to 7.2.1, if using the LogBridge module, then install the latest and greatest logbridge_feeder.pl and gwevents_to_es.pl scripts. Note that 7.2.1 includes the Search::Elasticsearch Perl modules so no rebuilding/reinstallation of them is required.
• LOGSTASH: After upgrading from earier versions of the LogBridge feeders, you might see the LOGSTASH application type next to hosts and services that these feeders created. This application type has been updated to LOGBRIDGE. You'll need to do some SQL to update the applicationtype on tables like servicestatus, host, and hostgroup.
  • To find the new applicationtypeid value for LOGBRIDGE and the old id for LOGSTASH, access the psql command line and connect to the gwcollagedb database, then:

    select * from applicationtype;

  • Show the agent id and show potential records to be updated. Copy the agentid for the new LOGBRIDGE group:

    select * from host where applicationtypeid = [old applicationid] or applicationtypeid = [new applicationid];
    update host set agentid = '[new agentid]' where applicationtypeid = [old applicationtypeid];
    update host set applicationtypeid = [new applicationtypeid] where applicationtypeid = [old applicationtypeid];
    select * from servicestatus where applicationtypeid = [old applicationtypeid] or applicationtypeid = [new applicationtypeid];
    update servicestatus set agentid = [new agentid] where applicationtypeid = [old applicationtypeid];
    update servicestatus set applicationtypeid = [new applicationtypeid] where applicationtypeid = [old applicationtypeid];

  • Once unassigned, you should be able to delete the LOGSTASH application type from the applicationtype table with:

    delete from applicationtype where applicationtypeid = [old applicationtypeid];

• Centos 7 known issues: If you install GroundWork on Centos 7 in a virtual machine, read this section:
  • Centos has published known issues that may impact performance and booting on virtual environments, see Centos 7 Release Notes.
  • In addition, some customers have encountered issues with RedHat Enterprise Linux 7 under VMware. These issues are unrelated to GroundWork, but may impact network stability in such systems.

Limitations and Work-arounds

• SNMP trap processing
  • Read the following if you use SNMP trap processing:
    • Best Practice: The SNMPTrap process is a brittle and difficult to configure subsystem. If too many traps come in, the disk creates enough traps that even stating the spool directory where they are temporarily stored by snmptrapd causes a huge delay enough that most processes would either time out or hang. If you get behind, where the rate of incoming traps is greater than the rate you are processing them, problems will show up around disk space reporting, inode availability, disk io, application and network timeouts. You will eventually run out of inodes. Since we turn on this the snmptrap processes at startup but don't have any definitions defined, this is an easy attack vector to take out GroundWork from the network without needing to authenticate. Until a remedy, it is recommended that all systems where snmptraps are not being used (e.g., child servers) have the processes disabled:

      service groundwork stop snmptrapd
      chmod -x /usr/local/groundwork/common/scripts/ctl-snmptrapd.sh
      service groundwork stop snmptt
      chmod -x /usr/local/groundwork/common/scripts/ctl-snmptt.sh

      Note, typical fresh install of Linux has iptables local firewall blocking UDP. A further best practice is to ensure continued control of iptables and other firewalls, limiting incoming access to only what is needed. See How to determine ports used by GroundWork. Typically Linux as installed will block all but SSH. Here is the iptables file content for a fresh installation:

      # Generated by iptables-save v1.4.7 on Sun Nov  5 21:37:28 2017
      *filter
      :INPUT ACCEPT [0:0]
      :FORWARD ACCEPT [0:0]
      :OUTPUT ACCEPT [3626:1478814]
      -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
      -A INPUT -p icmp -j ACCEPT
      -A INPUT -i lo -j ACCEPT
      -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
      -A INPUT -j REJECT --reject-with icmp-host-prohibited
      -A FORWARD -j REJECT --reject-with icmp-host-prohibited
      COMMIT
      # Completed on Sun Nov  5 21:37:28 2017
      

      Here is the typical iptables command to permit snmptrap UDP port 162 (you won't want to do this unless you are looking to use SNMP TRAPS)

      iptables -A INPUT -m state --state NEW -m udp -p udp --dport 162 -j ACCEPT
      

• GroundWork GrafBridge
  • There are some known limitations to the current release of GrafBridge:
    • Data source selection to service level:
      While it is possible to have several metrics per service in GroundWork, when using Grafana with the GroundWork data source for InfluxDB (the default), one can select a service as the finest level of detail in choosing what to graph. You can turn thresholds on or off, and you can suppress unwanted metrics with a display option called series-specific overrides, so this should not affect your ability to graph what you want, but we may decide to enhance the selection to metric level in future releases or technical bulletins.
    • No support for Counter data types:
      The GroundWork Grafana data source only supports "Gauge" type metrics, and renders counters as increasing in status viewer.

• Host Identity
  • There are some known issues when using host identities.
    • If you add a host alias record for a non-existent host (e.g., if you mistype the hostname or don't accept a suggested name), the system allows this but does not deal with the database identity cleanly. While there is no resultant data corruption, the results for the aliased hosts will be identified as coming from a different host. It is not recommended to use host identity records in this way.
    • If you create an alias, the alias host will be in all the hostgroups of the aliased host(s). This is by design. If, however, you delete the alias, all the aliased hosts will re-appear properly, but the alias host will retain the hostgroups of the aliased hosts. This can be fixed by deleting the hostgroups or connectors associated, or with a manual correction. Contact GroundWork Support for instructions if needed.

• Miscellaneous changes from previous releases
  • The Cloud Hub connector pages display a "GroundWork Server Version" field with a grayed out drop-down. This is a carry over from previous versions and can be ignored.

SECTION III: Announcements as of Version 7.2.1

Supported versions of GroundWork

• GroundWork Monitor version 7.1.0 is now end-of-life. Customers using older versions are advised to contact GroundWork Support regarding upgrade options.
• GroundWork Monitor version 7.1.1 will be end-of-life with the next release.

Linux supported versions

The versions of Linux now supported by GroundWork have changed. See the System Requirements for details. If you try to install GroundWork 7.2.1 on an unsupported version it will warn you, but it will allow you to proceed. This warning does not appear when using unattended mode in the installer.

Supported architectures

• GroundWork Monitor Enterprise only supports installation on 64-bit platforms. Monitored nodes can be 32-bit or 64-bit but GroundWork itself must be deployed to 64-bit.

Browser compatibility

• This version of GroundWork has been tested with the latest versions of Firefox and Google Chrome, as well as Internet Explorer. Chrome is recommended.

SECTION IV: Details of Fixed Issues Since Release 7.2.0

This release incorporates the usual assortment of bug fixes. For a full listing, please contact GroundWork support. A partial listing of the most relevant (from a user perspective) issues fixed are listed here. If you want to know if a particular issue you have reported is fixed in GroundWork Monitor 7.2.1, please see your support case where you reported it, or contact support with your question.

Copyright 2004-2018 GroundWork Open Source, Inc. ("GroundWork"). All rights reserved. Use is subject to GroundWork commercial license terms. GroundWork Monitor products are released under the terms of the various public and commercial licenses. For information on licensing and open source elements please see Licenses used in GroundWork. GroundWork, GroundWork Open Source, GroundWork Monitor Professional, GroundWork Monitor Open Source, GroundWork Community Edition, GroundWork Monitor Enterprise, GroundWork Foundation, GroundWork Status Viewer, Monarch, and GroundWork Guava are trademarks of GroundWork Open Source, Inc. Other trademarks, logos and service marks (each, a "Mark") used in GroundWork’s products, including Nagios, which is a registered trademark of Ethan Galstad, are the property of other third parties. These Marks may not be used without the prior written consent of GroundWork Open Source or the third party that owns the respective Mark.