Auto Registration Troubleshooting

WAS THIS PAGE HELPFUL? Leave Feedback

CONTENTS

1.0 Common Issues

  • If auto-registration is completing but monitoring data from GDMA clients is obviously not being processed by the server, perhaps the problem is that the server does not have the Bronx listener port (by default, port 5667) open to the clients. See the listener_port value in the config/bronx.cfg file on the server, and the Spooler_NSCA_Port value in the gdma/config/gdma_auto.conf file on the GDMA client. These values must match, and you must not have a firewall (or packet filtering such as iptables on the server) blocking this port.

    The mechanism for allowing access to a given server port will depend on the particular Linux distribution you are running on the server. On RHEL 6.3, for instance, you would need a line such as the following in the /etc/sysconfig/iptables file, and then have iptables restarted to pick up this configuration: 
    -A INPUT -m state --state NEW -m tcp -p tcp --dport 5667 -j ACCEPT
  • If you think that auto-registration is not doing what it ought to, first look in the server-side log file below. This pathname is configured in the config/register_agent.properties file on the server, and listed also in the /etc/logrotate.d/groundwork file to enable regular log rotation. 
    /usr/local/groundwork/foundation/container/logs/register_agent.log
  • Auto-registration error messages listed in the server log are generally also returned to the GDMA client, and will be recorded in the client's poller log file if logging is enabled on the client by enabling it in the gdma/config/gdma_auto.conf file. If the client is having trouble contacting the server in the first place, that may not be evident without enabling logging on the client and looking in the poller log file. If you do enable logging on the client, it is highly recommended that you disable it once you have diagnosed and fixed the problem, because the current version of GDMA never rotates its log files.Enable_Local_Logging = "on"
  • If an auto-registration request doesn't yield any positive results, the poller won't keep banging on the server every iteration. It waits at least an hour before trying again. This interval is currently hardcoded. Due to slight delays in processing a single auto-registration request, in practice this means it won't try again for 70 minutes.
  • If you want to speed things up for testing, especially to avoid the initial 10-minute wait before the first auto-registration attempt after the poller starts and finds it has no local copy of the host configuration file, you will probably want to set Poller_Proc_Interval to 60. If you change this value for testing, be sure to change it back (typically to 600) for production use! If auto-registration fails on a given attempt and you don't want to wait the extra hour for the following attempt, just bounce GDMA on the client machine. The How to start and stop services page describes the various platform-specific commands used to start and stop the GDMA client.
  • Time sync: Many installations do not properly synchronize time with NTP. This is particularly an issue with Virtual environments, where time slicing may introduce latency and drift to otherwise stable time signatures. For this reason, GDMA has been made tolerant of time drift. In the setup instructions, the file /usr/local/groundwork/config/bronx.cfg was modified to allow up to 900 seconds latency and 90 seconds imminence, or future timestamps, on incoming messages. If the time synchronization is sufficiently off to be outside this 16.5 minute envelope, the GroundWork server will reject the incoming message. To correct this situation, you must either:
    1. Synchronize the time on all monitored systems and the GroundWork server, or
    2. Widen the envelope by changing the parameters in bronx.cfg. It is highly recommended that the time be synchronized. You will need to restart Nagios after you make a change, /etc/init.d/groundwork restart nagios.
      The GroundWork Monitor version of send_nsca supplied with GDMA allows for the sending of old time stamps, which are the time stamps associated with when the check was run, not when the spooler has sent the data, which is potentially much later. This allows the system to accurately process performance data for checks that have been spooled, potentially for long periods.
  • Encryption mismatch: A common security requirement GroundWork Monitor supports is the encryption of data sent over the NSCA channel. By default, however, GDMA is installed with encryption turned off, which is also the default on the GroundWork server. Thus it is possible to have an encryption mismatch, for example, if you enable encryption on the GroundWork server (in bronx.cfg), and fail to enable it on the GDMA in send_nsca.cfg. Always make sure these files are consistent between GDMA (send_nsca.cfg) and GroundWork (bronx.cfg).

    One way to manage this situation is to set up GDMA to use an alternate send_nsca.cfg file. This file can be downloaded to the GDMA as if it were a plugin, and specified for use via the host external using the GDMA parameter called Spooler_NSCA_Config. For example:

    If you create a file called: 
    send_nsca_gdma_encrypted.cfg

    and you set it up for download as a plugin for Linux, it will be placed on the Linux GDMA in:

    /usr/local/groundwork/gdma/tmp/send_nsca_gdma_encrypted.cfg

    That means you could set up the GDMA to start using this file at the same time you switch the GroundWork server to use encryption, just by changing:

    Spooler_NSCA_Config="/usr/local/groundwork/gdma/tmp/send_nsca_gdma_encrypted.cfg"

2.0 Generating Ouput

Generating logs

In general, GDMA will not output much in the way of log information unless you specify that it should do so. If you want logs, you can use the Enable_Local_Logging parameter to output log data to the file specified in the Logdir parameter, which is the log subdirectory by default. You should not leave this enabled permanently, as there is no facility to remove old logs.

Client logs

The gdma_auto.conf file is located in the /usr/local/groundwork/gdma/config directory.

  • Auto-registration error messages listed in the server log are generally also returned to the GDMA client, and will be recorded in the client's poller log file, if logging is enabled (e.g., Enable_Local_Logging = "on") on the client to output log data to the file specified in the Logdir parameter, which is the log subdirectory by default. If the client is having trouble contacting the server in the first place, that may not be evident without enabling logging on the client and looking in the poller log file. If you do enable logging on the client, it is highly recommended that you disable it once you have diagnosed and fixed the problem, because the current version of GDMA never rotates its log files.
  • Example with logging parameter "on": linux gdma client gdma_auto.conf example.txt
  • Example: linux gdma client log directory example.txt
Nagios log

After a Commit operation, the auto registration process begins and a Nagios log message will announce the new GDMA. As soon as the auto-registration cron job runs, the message is matched to the profile, the host definition is created with that profile, and the Monarch group gets the host added.

  • Nagios Log - The nagios.log file is located in the /usr/local/groundwork/nagios/var directory.
Agent log

If you think auto registration is not operating as it should you can look in the register_agent.log file.

Running GDMA poller and spooler by hand

You can also generate useful output (the same as what gets placed in the log) by running the GDMA Poller and/or Spooler by hand.

  • In Windows
    • Poller output - Open a command shell, and type: 
      cd \Program Files (x86)\groundwork\gdma\bin
      gdma_poll.exe -i -d2 -x
    • As the Windows GDMA can be installed in alternate locations, you may need to enter a different path. For example in 64-bit Windows, GDMA is installed in \Program Files (x86)\groundwork\gdma by default.
    • This will give you output similar to the following: windows gdma client poller output example.txt
    • There are several diagnostic steps GDMA goes through each time it runs. Errors in the retrieval of the config file, or with a syntax error in that file, will be presented near the top of this output. Any problems running the checks will appear in the last sections.
    • If all of that is working ok, then you might try running the spooler: 
      gdma_spool_processor.exe -i -d2 -x
    • This will give you output similar to the following: windows gdma client spooler output example.txt
    • The spooler merely takes output from the checks that have been run and sends it to the Target server or servers. Of interest here is if it succeeded in contacting the Target, which will be indicated as Processing live targets. Dead targets are Target servers that can't be reached for some reason, and indicate a down primary or standby system, or network issues in contacting the Target. The spooler will keep trying until it reaches a dead target, or times out according to the Spooler_Retention_Time and Spooler_Max_Retries parameters.
    • For UNIX, the command is similar, but you must run as the gdma user, with the full execution environment of that user, and specify the GroundWork-supplied perl interpreter. The output is essentially the same for UNIX as for Windows.
  • On Linux and AIX
  • On Solaris
    • Poller output - Open a terminal session, and enter: 
      su - gdma
      cd /opt/groundwork/gdma/bin
      /opt/groundwork/perl/bin/perl gdma_poll.pl -i -d2 -x
    • Spooler ouput - Open a terminal session, and enter: 
      su - gdma
      cd /opt/groundwork/gdma/bin
      /opt/groundwork/perl/bin/perl gdma_spool_processor.pl -i -d2 -x

Labels

gdma gdma Delete
troubleshooting troubleshooting Delete
autoregistration autoregistration Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.