Automated Agent Registration

1.0 General Description

The GDMA installer asks for an Automated Agent Registration username/password at install time. The defaults presented to the user are blank, but gdma/gdma will work in a fresh install out of the box. Changing the password on the server, in Group > Organization > Users and Groups Management, is recommended for customers before deploying GDMA. If both values are given, then the auto-registration is enabled on that GDMA client. If one or both are blank, then the auto-registration is disabled and the old auto-configuration mechanism will still be in play, albeit perhaps at a slower pace than before (see the hour delay noted below).

This new gdma user on the server has limited (generally read-only) privileges for accessing most server functions, and those restrictions should be left in place. It is intended that this username be used only for this Automated Agent Registration purpose, partly so that if any trouble arises with it, the account can be disabled without impacting any other aspect of the monitoring.

The GDMA client installer also asks for host and service profile names. Both are optional. The installer suggests a standard default value for the host profile, which happens to be the same as the initial default we have configured on the server side. That default is gdma-linux-host.xml or equivalent for other platforms gdma-windows-host.xml, gdma-solaris-host.xml, and gdma-aix-host.xml. If no service profile is specified, then no extra service profile will be applied beyond any service profiles referenced directly by the host profile.

The new values requested by the GDMA client installer are stored on the client in gdma/config/gdma_auto.conf:

Auto_Register_Host_Profile = ""
Auto_Register_Pass = ""
Auto_Register_Service_Profile = ""
Auto_Register_User = ""

These settings can be adjusted after installation, if need be. Like anything else in the gdma_auto.conf file, modifying values there will in due course cause the client to recognize a configuration change and attempt to reload its own configuration — now in up to three files: gdma_auto.conf, gdma_override.conf, and the gwmon_{hostname}.conf file.

The gdma/config/gdma_override.conf file is new as of the GDMA 2.3.0 release. It is dynamically created as the place where the poller will store the hostname returned by the server, after a successful auto-registration request. The poller uses a Forced_Hostname option to save this centrally-agreed-upon hostname. This gets picked up by the spooler as well, so both poller and spooler know the registered name of the host. Any use of the Forced_Hostname option in the gdma_auto.conf file will be overridden by the value in the gdma_override.conf file, if the latter file is present.

When the client makes an auto-registration call to the server, it passes along a small amount of information about the client system, including the configured Auto_Register_Host_Profile and Auto_Register_Service_Profile values. The foundation/scripts/registerAgentByProfile.pl script on the server is called to validate the host attributes, add the host to Monarch, build externals for the host (based on whatever externals were attached to the host profile, or to the services attached to the service profile(s), and return the final hostname. Building externals means that the associated host and service externals are collected together, processed via possible macro substitution, and written to a location where the client can pick them up.

In practice, a green system operates in the following way. An unconfigured poller will reach for a local copy of the host-config file, find none, and go to sleep for 10 minutes (configured in gdma_auto.conf as the Poller_Proc_Interval). On waking up, when it again sees it still cannot find its config file, if it has auto-registration credentials, it will do the auto-registration thing; otherwise, it will fall back to the old auto-configure protocol. All the auto-registration thing does is to make a call to the server, and look at the results. If the results include a hostname, the poller stores the hostname in the gdma_override.conf file. At that moment, it still doesn't have a host config file in hand. On the next cycle, it should pull the externals file from the GroundWork server, save it locally, and start monitoring the client machine.

The spooler, in its own cycle time, eventually sees that the configuration files have changed, and re-reads them. Then it will see the forced hostname and adapt to it.

When an auto-registration request comes into the server, it first tries to look up in Monarch to see if the host is already registered. There is a special Perl package installed (foundation/scripts/AutoRegistration.pm) that implements a pass of possible host-attribute recoding before this lookup. If that lookup fails, a second possible recoding (presumably with different logic this time) and lookup is attempted. If both passes fail and the package has not concluded that the attributes are totally invalid, the host will be added to Monarch.

The second pass of recoding allows IP => hostname lookups based on specific addresses listed in the server config/register_agent.properties file, for cases where DNS just isn't operating right on the client and there's no way to fix that easily. (We've tried to cover all the bases, for situations seen in the field. If we've missed any, let us know and we'll see about folding more capability into the next version.)

Auto-registered hosts will begin monitoring as soon as they receive their own host configuration file from the server. They will then start sending on the results of those checks to the GroundWork server. At this point, however, Nagios won't know of the existence of such hosts, and it will casually drop those results on the floor (probably with a notice to that effect in the nagios/var/nagios.log file). Results from such hosts will continue to be ignored until the administrator (or perhaps some automated process) runs a Commit operation.

Additional Information

The config/register_agent.properties file on the server contains a number of configuration options to control the server-side scripting.

The AutoRegistration.pm module on the server contains documentation regarding the internal routines, which can help both with understanding its actions and for making possible modifications:

/usr/local/groundwork/perl/bin/perldoc /usr/local/groundwork/foundation/scripts/AutoRegistration.pm

2.0 GDMA Automated Agent Registration Initial Setup

2.1 Change the default password

Change the password of the gdma user from the default value that is shipped in the standard release. This is a basic security measure you should take to protect your site.

Log in to GroundWork Monitor as an administrator with the defaults; user name: admin and password: admin
To change the default password select Group from the top toolbar and navigate to Users and groups management.

Figure: Users and Groups Management
Within the User Management tab, select the Edit icon, under the Action column, for user name gdma.

Figure: Edit Action
Enter a check in the box for Change Password and type a new password, select Save. The length of the text in fields for Password must be between 6 and 30 characters.

Figure: Change Password

2.2 Enable externals

Enable externals within Monarch, so you have access to all the externals-related screens and options.

Select Configuration > Control > Setup >.
Check the box to Enable externals and select Save. A variety of screens will now display extra tabs and navigational links allowing you to edit and manage host and service externals.

Figure: Enable Externals

2.3 Add an Auto-Registration hostgroup

Select Configuration > Hosts > Host groups > New.
Enter Auto-Registration as the hostname. You can choose a different name, if you want, but then you will also need to modify the default_hostgroup setting in the /usr/local/groundwork/config/register_agent.properties file.

Figure: Add Hostgroup

2.4 Add an auto-registration Monarch configuration group

Select Configuration > Groups > New.
Enter auto-registration as the Monarch configuration group. Select Add. You can choose a different name, if you want, but then you will also need to modify the default_monarch_group setting in the /usr/local/groundwork/config/register_agent.properties file.

Figure: Add Monarch Group
In the Detail tab for the new Monarch configuration group, set the Build folder to /usr/local/groundwork/apache2/htdocs/gdma so externals files will be placed where GDMA clients will be looking to pick them up. Save this change.

Figure: Set Build Folder
It is generally recommended that you add the Auto-Registration hostgroup to the auto-registration Monarch configuration group. While still in the Manage Group section, select the Hosts tab and add this hostgroup to the Monarch configuration group. This will simplify the actions when new hosts are added, by allowing the individual hosts to be indirectly associated with the Monarch configuration group via the hostgroup, instead of being assigned directly to the Monarch configuration group. Make adjustments as needed if you use alternative hostgroup or Monarch configuration group names.

Figure: Add Hostgroup to Configuration Group

2.5 Add new host profiles

The new host profiles will form the basis for configuring new auto-registered GDMA hosts running the respective operating-system types. You are not required to use these profiles; the GDMA client installer allows you to select an alternate host profile, and an optional additional service profile, to be applied on the server to the auto-registered client's configuration. See also the default_host_profile setting in the /usr/local/groundwork/config/register_agent.properties file.

To add new host profiles and related objects, select Configuration > Profiles > Profile importer > Import.
Choose the GDMA category and select the four new host-profile files gdma-aix-host.xml, gdma-linux-host.xml, gdma-solaris-host.xml, gdma-windows-host.xml, and click Import at the bottom of the screen. This will bring in corresponding new host profiles and related host templates, service profiles, service templates, services, host and service externals, commands, performance configuration entries, and a few other minor objects.

Figure: Host Profiles

2.6 Add default contact group to the host templates

Edit each of the new host templates just imported (gdma-aix-host.xml, gdma-linux-host.xml, gdma-solaris-host.xml, gdma-windows-host.xml), to add one or more default contact groups.

Navigate to Configuration > Hosts > Host templates > Modify, then select each one of the host templates listed here and establish the basic contact group(s) you will want for hosts using this host template (perhaps just nagiosadmin at this point, as a kind of safe default).
Save each host template after making this change. The setting here can be overridden later on at other levels, but this provides at least a standard setting for all auto-registered hosts.

Figure: Set Host Templates with Contact Group

3.0 Automated Agent Registration FAQs

Q: Which GroundWork Monitor server is the auto-registration request sent to?
A: It's the server mentioned as the first value of the Target_Server option in the gdma/config/gdma_auto.conf file on the client. Also see the GDMA>Configuration Reference.
Q: What happens during auto-registration if the GDMA client machine is already present in Monarch, and doesn't need to be freshly added to the configuration?
A: The server makes up to two separate attempts to look up the host within Monarch, to find it using possibly different host attributes. If one of those attempts succeeds, the existing hostname in Monarch will be returned to the client as its official, final hostname. But even in this case, the server attempts to ensure that the desired host profile, service profile, hostgroup, and Monarch configuration group assignments are made to this host.
Q: My GDMA client cannot reliably determine its own hostname to send to the server, because DNS is not working at the client. What can I do?
A: Many failures of this type can be corrected by the server during the auto-registration process. Logic in the AutoRegistration.pm module looks at the IP address sent by the client as well, and tries to determine a correct match between the IP address and the hostname before settling on a final hostname to add to Monarch and to publish back to the client machine. See also the <hardcoded_hostnames> option in the config/register_agent.properties file, for handling particularly pernicious problems.
Q: I don't like the form of the hostname that my client machines get registered under. How can I change it?

A: An out-of-the-box setup will cause auto-registered hosts to have fully qualified hostnames within GroundWork Monitor. This form was chosen to provide the least chance of ambiguity in large installations, especially those that serve multiple domains that might otherwise have hostname collisions. But it is not a requirement of operation. The client tries to determine its own hostname using standard heuristics, and it sends that hostname to the server as part of the auto-registration request. The server may or may not accept that hostname as-is, but in any case, the server's decision is final. There are two controls, and possibly other settings, to play with.

On the client side, you can set the Forced_Hostname option in the gdma/config/gdma_auto.conf file. See the documentation for this option in the GDMA>Configuration Reference.

With auto-registration in play, this option in the gdma_auto.conf file really only affects the name passed to the server by the client during its auto-registration attempt. Afterward, the value of this option dynamically saved by the client in the gdma_override.conf file will rule.
The Use_Long_Hostname option will have no practical effect when auto-registration is in effect. The client always attempts to send a fully-qualified hostname to the server as part of an auto-registration attempt, to give the server the best chance of accurately identifying the client.

On the server side, you can adjust the hostname_qualification option in the config/register_agent.properties file, to globally select long or short names, or to pass the decision to your own custom logic.
If the hostname is being determined by the <hardcoded_hostnames> section of the config/register_agent.properties file, that will be the place you will need to change it.

In the extreme, if you need some kind of specialized hostname determination that is not covered by the options above, you can modify the logic by which the server chooses the client hostname. You might do this, for instance, if you want to use unqualified hostnames for most of your machines, and fully qualified hostnames only for machines on particular subnets. The client sends multiple attributes to the server as part of an auto-registration request: the type of agent making the request (e.g., "GDMA"), its own hostname (as best it knows it), its IP address, its MAC address, and the generic type of operating system running on the host. Custom logic can use any or all of these attributes to make a final determination of the client hostname. To implement custom server logic in this area, you will need to:

Copy the foundation/scripts/AutoRegistration.pm file into some sibling file of your own name (e.g., AcmeCorporationAutoReg.pm).
Change the package declaration in your file to reflect that name:
```
package AcmeCorporationAutoReg;
```
Change the logic in the soft_recode_host_attributes() and/or hard_recode_host_attributes() routines as you need.
Edit the documentation in the file at the end of your own version of this package to reflect your logic changes.
Change the customer_network_package option in the config/register_agent.properties file to name your package instead of the AutoRegistration package.
```
customer_network_package = "AcmeCorporationAutoReg"
```

This last step will put your package into play.

GroundWork Monitor will overwrite the AutoRegistration.pm file in future releases, which is part of why you are advised to change the filename and package name if you change the content.
You should be aware that both client-side GDMA and the server-side scripting always lowercase whatever hostname they use, to provide consistency in the face of potential confusion. This behavior is not configurable, as it just causes too much headache to loosen up on this convention.

Q: I don't like the form of the hostname that one of my client machines has already been registered under. How can I change it?

A: You can rename the host within Monarch (Configuration > Hosts > {hostgroup} > {host} > Detail > Rename). After doing so, go to Configuration > Control > Build externals, to rebuild the externals file using the modified hostname. After some time, the GDMA client will recognize that its externals file is no longer available on the server under the old name, and that will eventually cause it to exercise the auto-registration logic. At that time, the client should be recognized by the auto-registration request as being already in Monarch (given that the IP address has not changed, even though the hostname did), and the new name will be returned to the client for further operations there.

If need be, you can speed up this process by bouncing GDMA on the client after the externals are rebuilt on the server. (See the GDMA>GDMA Quick Start page in the Bookshelf for the commands to do this on each platform.) It will take about 10 minutes (one Poller_Proc_Interval) for the auto-registration to occur on the client.
Of course, you will need to run a Commit operation (Configuration > Control > Commit) for Nagios to understand that the hostname has changed, so it doesn't start dropping the data for this host on the floor.

Renaming an existing host like this is subject to a caveat, namely that in the current release of GroundWork Monitor, performance data collected under the old hostname will not be transferred to the new hostname. The old data still exists, but it is stored in RRD files using the old hostname, while new performance data will be stored in RRD files using the new hostname. We don't currently have a utility to merge the old data into the new RRD files.

Renaming a GDMA Host

Known issue:
Using the Rename button, located at the bottom of the host detail screen (Configuration > Hosts > Hosts > {hostgroup} > {host} > Detail), to rename a GDMA host will cause the host to stop receiving checks. This action fails to overwrite the agent configuration on the remote machine and restart the remote agent to allow it to pull the new configuration. Once the workaround below has been completed, a new configuration file will present itself with the corrected name, and the forced config file is also updated with the correct name.

Workaround:

This must happen on the GDMA client only after the next Commit after the server-side Rename occurs. That way, the new setup will be in place on the server when the updated client reaches for it. Also note that, after the Commit, the administrator should re-build externals, so make sure they are all up-to-date, before making changes on the GDMA client.

Stop GDMA service
Delete the forced config and the host named config files
Start GDMA service

GDMA client files to delete:

gdma/config/gdma_override.conf (which file is supplied to the GDMA client by the GroundWork server during auto-registration, and holds a Forced_Hostname directive specifying the form and content of the hostname that the client is to use when interacting with the server)
gdma/config/gwmon_hostname.cfg (which file, named using the precise form of the client hostname which the client should be using, is supplied to the GDMA client by the GroundWork server, and contains host and service externals from the server which are relevant to this client, note this is the actual hostname of the GDMA client, not the literal characters "hostname")

Q: What configuration objects are associated with an auto-registered host?

A: Multiple connections will be made during auto-registration:

Common practice is to have the host profile specified by the client be assigned and applied to the host. If the client does not specify a host profile, the server will assign a host profile based on the default_host_profile option in the config/register_agent.properties file. The server can modify these standard choices via the host_profile_to_assign() routine in a customized version of the AutoRegistration.pm module.
Any service profiles associated with the applied host profile will be applied to the host.
The client can optionally specify one additional service profile to be applied to the host.
Host and service externals from the host profile and from the services applied to the host will be copied to the configuration for the host. This is a critical link in how the individual GDMA options and service checks for the host are made known to the client to run its monitoring commands.
The host will become a member of the hostgroup listed as default_hostgroup in the config/register_agent.properties file. If a more-complex setup is required, a customized version of the AutoRegistration.pm module can be used to return a list of hostgroups in the result of the hostgroups_to_assign() routine. If you truly don't want any auto-registered host to be a member of any hostgroup (probably not a practical setup), that can also be accomplished by having your custom version of the AutoRegistration.pm module have no hostgroups_to_assign() routine.
If the host is not already a member of some Monarch configuration group, the host will become a member of the Monarch configuration group listed as default_monarch_group in the config/register_agent.properties file. This standard behavior can be modified via a customized version of the AutoRegistration.pm module, where the monarch_groups_to_assign() routine can be used to determine a list of Monarch groups to which the host should be assigned.

Also see the assign_monarch_groups_to_existing_group_hosts option in the config/register_agent.properties file, if you need to allow a host which is already assigned to some Monarch configuration group to also be assigned to the Monarch configuration group(s) determined by the auto-registration logic.

The host must be assigned to at least one Monarch group in order to have externals built and published on the server, where the client can subsequently find them. Some customers might want to create and distribute the client host-configuration files via some other mechanism. A future version of GDMA will include the ability to assign an empty string to the default_monarch_group option in the config/register_agent.properties file, to suppress the assignment of the host to a Monarch group, for all hosts. In the meantime, the same effect can be achieved in a custom version of the AutoRegistration.pm module by either having the monarch_groups_to_assign() routine return an empty list of Monarch groups, or having no monarch_groups_to_assign() routine in your package.

Administration of the Monarch configuration group can be made easier by manually assigning at least one hostgroup the host will be a member of to the Monarch group. Then all hosts that are auto-registered and that are a member of such a hostgroup won't have to be directly assigned to the Monarch group; the indirect association will suffice. This is generally the preferred setup, as it keeps the configuration much simpler.

Q: GroundWork's standard profiles provide a basic set of monitoring checks for each platform (type of operating system). How do I customize the set of checks used by GDMA clients?
A: Use custom profiles instead of the standard profiles that GroundWork provides. You can adjust the sets of checks however you like, selecting the sets of host and service checks you are most interested in.
Q: My monitoring-check definitions need to be written in a generic form so they apply across many hosts. But then how do I get them automatically customized for each particular host?
A: Host and service externals can reference several different types of macros, and have those references be expanded when the externals files are built. Macros are currently expanded in the following order:
- Nagios user-resource macros (the macros named as USER1 through USER32 in the Configuration > Control > Nagios resource macros screen).
- Monarch configuration group macros (created in the Configuration > Groups > Macros screen, and associated with groups and possibly redefined in the Configuration > Groups > Groups> {groupname} > Detail > Macros screen). Since a host can be a member of multiple configuration groups arranged in parent-group/sub-group hierarchies, a standard order has to be defined in which such macros are applied. The rule is that Monarch configuration group macros are substituted in ascending order from subgroups to parent groups associated with the given host. This allows the most-specific value of a given macro (generally associated with a subgroup) to be used in preference to a more-general value of the same macro name (generally associated with a parent group, and used as a default value).
- Certain standard macros: HOSTNAME, HOSTADDRESS, and HOSTALIAS.
  
  In all of these cases, the macro references are recognized by the presence of both leading and trailing $ characters around the macro names. So they would appear in your raw externals as references such as shown below. Lots of capabilities can be enabled and customized via this mechanism. GroundWork Professional Services can be engaged to help with complex requirements.
```
$USER32$
$MYGROUPMACRO$
$HOSTADDRESS$
```

Q: How do I change the set of monitoring checks for a host that is already being monitored by GDMA?
A: Modify the monitoring checks as you would for any other host — for instance, adding new service checks. Just make sure that each new host-service has associated externals applied to the host-service by the time you are done with your modifications, to define things like the command to run on the GDMA host. That way, the GDMA host will find out what it needs to do to perform the desired monitoring. Once your changes are done, build externals (to make the changes available to the GDMA host), then run a Commit (to let Nagios know that the set of configured checks has changed). Soon thereafter, the GDMA host will discover that its cached copy of the externals file is out-of-date, and it will download a fresh copy and begin monitoring according to the new setup. About 5 minutes after the Commit operation (per the setting of the Service performance data file processing interval, on page 3 of the Control > Nagios main configuration), Nagios will start the performance-processing script. Performance data will start to be processed again, which will pick up any new checks for the host, create new RRD files if necessary, and make graphs for new metrics available in Status Viewer.
Q: Are there other improvements in this area provided since the GWMEE 6.7.0 release?
A: Building externals is much more sophisticated now. By default, it compares the old and new payload of each generated externals file, and only updates the timestamp of the file if the content has changed. This should cut down dramatically on the amount of noise in the event system from GDMA clients notifying the server of supposed configuration changes. Also, we have improved the code in some places to take very targeted action and only rebuild externals for a few hosts that might actually need it, so we no longer build indiscriminately for all hosts.
Q: How can I disable Automated Agent Registration on a given GDMA client machine?
A: Disabling auto-registration for an individual host can be effected on the client side by editing the gdma/config/gdma_auto.conf file and either setting the auto-registration access credentials to empty strings, or just commenting out those lines in the file. Such a setup will instead invoke the older auto-configure protocol if the client cannot find its own host-configuration file on the server.
```
Auto_Register_Pass = ""
Auto_Register_User = ""
```
Q: How can I completely disable Automated Agent Registration?
A: There are two simple server-side controls that will stop all Automated Agent Registration requests from succeeding. Either one alone suffices to disable this facility.
- You can change the password for the auto-registration user that the clients were installed with, without telling the clients the new password.
- In the config/register_agent.properties file, you can set:
```
enable_processing = no
```
Q: Is auto-registration of hosts with IPv6 addresses supported?
A: Not yet, in this release. Some initial support is present, but not to the point where GroundWork considers it to be functional and tested. This capability will evolve in future releases.

4.0 Automated Agent Registration Troubleshooting

If auto-registration is completing but monitoring data from GDMA clients is obviously not being processed by the server, perhaps the problem is that the server does not have the Bronx listener port (by default, port 5667) open to the clients. See the listener_port value in the config/bronx.cfg file on the server, and the Spooler_NSCA_Port value in the gdma/config/gdma_auto.conf file on the GDMA client. These values must match, and you must not have a firewall (or packet filtering such as iptables on the server) blocking this port.

The mechanism for allowing access to a given server port will depend on the particular Linux distribution you are running on the server. On RHEL 6.3, for instance, you would need a line such as the following in the /etc/sysconfig/iptables file, and then have iptables restarted to pick up this configuration:
```
-A INPUT -m state --state NEW -m tcp -p tcp --dport 5667 -j ACCEPT
```
If you think that auto-registration is not doing what it ought to, first look in the server-side log file below. This pathname is configured in the config/register_agent.properties file on the server, and listed also in the /etc/logrotate.d/groundwork file to enable regular log rotation.
```
/usr/local/groundwork/foundation/container/logs/register_agent.log
```
Auto-registration error messages listed in the server log are generally also returned to the GDMA client, and will be recorded in the client's poller log file if logging is enabled on the client by enabling it in the gdma/config/gdma_auto.conf file below. If the client is having trouble contacting the server in the first place, that may not be evident without enabling logging on the client and looking in the poller log file. If you do enable logging on the client, it is highly recommended that you disable it once you have diagnosed and fixed the problem, because the current version of GDMA never rotates its log files.
```
Enable_Local_Logging = "on"
```
If an auto-registration request doesn't yield any positive results, the poller won't keep banging on the server every iteration. It waits at least an hour before trying again. This interval is currently hardcoded. Due to slight delays in processing a single auto-registration request, in practice this means it won't try again for 70 minutes.
If you want to speed things up for testing, especially to avoid the initial 10-minute wait before the first auto-registration attempt after the poller starts and finds it has no local copy of the host configuration file, you will probably want to set Poller_Proc_Interval to 60. If you change this value for testing, be sure to change it back (typically to 600) for production use! If auto-registration fails on a given attempt and you don't want to wait the extra hour for the following attempt, just bounce GDMA on the client machine. The GDMA>GDMA Quick Start page describes the various platform-specific commands used to start and stop the GDMA client.