Overview

Monitoring with SSH has advantages in security and flexibility that you can leverage with GroundWork Monitor. This section reviews how to remotely monitor servers with the SSH plugin.

1.0 About Secure Shell (SSH)

Secure Shell (SSH) provides encrypted terminal access to remote hosts, and also provides the ability for users to execute commands on the remote hosts, using the resulting encrypted sessions. In simple terms, this means that a user on one host can use SSH to execute commands on another host in the same basic manner as if those commands were being executed locally, with the extra benefit of having the commands and response data being exchanged across a secure communications channel.

Moreover, since Nagios uses a lightweight command and response interface to gather monitoring data, Nagios is also able to use SSH technology to process remote commands and response data just as easily, and with the same benefits.

Specifically, Nagios provides an SSH plugin interface that can be used to build a variety of service profiles in GroundWork Monitor. In this setup, the administrator defines one or more profiles that describe the commands that need to be executed on the remote hosts, and also defines a global SSH username for remote login purposes. From there, Nagios is able to call the SSH plugin with the necessary parameters as needed, with the plugin establishing a secure connection to the specified host, executing the specified command, and then returning the program output to Nagios for processing. The only substantive difference between this approach and regular command processing is that the service profiles require some additional parameters, and the fact that Nagios calls upon the SSH plugin to perform the remote command processing instead of handling that task directly, but it otherwise preserves the traditional Nagios model.

This model is especially important with services that do not offer a network monitoring interface. For example, many modern application servers maintain statistics for important attributes such as the number of requests processed per/day, but will only make the information available through local management utilities. In those situations, the easiest and most secure way to incorporate the data into GroundWork Monitor is by using the Nagios SSH plugin to automate the process of remotely executing the necessary management commands.

Apart from specific services that cannot be easily monitored, many administrators also use the SSH plugin to monitor system-level resources by simply copying the entire set of Nagios local commands to a remote host and then using the SSH plugin to login to the remote server and execute those commands. Although this is not technically required (there are several other means of obtaining most of the same data), this route can simplify data collection if an administrator has to use the SSH plugin for other tasks on those systems anyway.

For these reasons, GroundWork Monitor includes several predefined SSH service profiles that are capable of monitoring many common applications and system-level resources, and new service profiles can also be easily developed if they are needed.

2.0 Building the SSH Infrastructure

Using the SSH plugin to execute commands on remote hosts is fairly simple in practice, although there are some one-time setup tasks that must be performed first. In particular, network administrators must define a user account for Nagios to use for remote login purposes, and must also configure the private and public keys for that account. Once those tasks are performed, the appropriate service definitions must also be created on the GroundWork Monitor server.

Define the login account

All of the SSH-based command definitions in GroundWork Monitor use the username stored in the user17 nagios macro definition for remote login purposes. The default value for this macro assignment is nagios although any username can be used.

If you wish to define a different username, perform the following steps:

1. Log in to GroundWork Monitor as an Administrator (e.g., admin/admin).
2. Select Configuration > Control > Nagios resource macros.
3. From the list of macros, select the USER17 entry, which will cause the main window to be reloaded with the macro value in an edit field at the top of the screen.
4. Change the USER17 value to a desired username value, click Update.
5. After a commit operation (Configuration > Control > Commit), the new username will be used for all subsequent SSH service definitions that reference the user17 macro.
   
   Figure: Nagios Resource Macros
   

Create the login accounts

The SSH username must have a corresponding user account on each machine that will be used for monitoring purposes. If the user account does not already exist, it must be created before that system can be accessed by the Nagios SSH plugin.

User account management routines are platform-specific, so you should refer to your system documentation for information on how to create a new user. The username assigned to the account must be identical to the username specified in the user17 nagios macro (the default is nagios), and must also have a home directory that can store the SSH configuration directory and public key.

Most modern Linux systems typically use the following sequence of commands:

1. Login to the target system with an administrative account, such as root.
2. Create the nagios user account, requesting that the user's home directory also be created. With the default username of nagios, the command would be useradd -m nagios.
3. Verify that the user's home directory is owned by the new user account, and has permissions that let the user read, write, and enter the directory. For example, if the user's home directory is /home/nagios, the ls -lad /home/nagios command should show that the directory is owned by nagios, with rwx permissions for the owner:

   $ ls -lad /home/nagios
   drwxr-x--  9 nagios Users 512 Apr 26 2007 /home/nagios

   Some level of access for the account's primary group are often allowed, and this is typically a matter of local administrative policy. However, some SSH servers are known to prohibit logins for accounts with insufficiently secure home directories (refer to your system's SSHD documentation for details).

4. You should assign a password for the user account to prevent unwanted access, although a password is not used by the SSH public-key authentication mechanism. The command to assign a password to the default nagios username would be passwd nagios, which would then prompt for a new user password.
5. Once the user account has been created and enabled, verify that the user is able to login to the system with the assigned password, and that the user is able to create and edit files in their home directory.

3.0 Configuring Public Key Authentication

SSH supports several different mechanisms for authenticating user logins, including the use of traditional password authentication. However, the Nagios SSH plugin cannot use password authentication, since password prompts interfere with remote command execution. In order to use the SSH plugin with Nagios, you must configure the associated user account for public-key authentication.

Public-key authentication uses two pieces of data to authenticate a user account.

• First, the SSH client maintains a file that contains the user account's private key, which it presents to the SSH server during the authentication process.
• Meanwhile, the SSH server looks for a public key in the SSH subdirectory of the specified user's home directory, and uses the private key to decrypt the contents of the public key. If the decryption operation succeeds, the user is allowed to login with the specified account, under the assumption that possession of the private key has been authorized.

This model allows a user to login to a system without typing or transmitting their password. This is important, since password authentication would require that the SSH plugin be able to read it, which in turn would require that the password be stored somewhere as clear-text data. By using public-key authentication, the encrypted private key can be stored on disk and used as needed, without exposing the account's actual password.

Public Keys Process

Several steps are required in order for this to process work correctly, itemized below:

1. Login to the remote system with an administrative account such as root, and verify that the SSH server has public key authentication enabled. SSH server configuration mechanisms are system-specific, so you may need to review your system documentation for more information. Most Linux systems use the open source OpenSSH package, which typically uses a configuration file named sshd_config, although its location will often vary by distribution. Most Linux distributions with OpenSSH also have public key authentication enabled by default, but if you need to specifically enable it you can do so by changing or adding a PubkeyAuthentication configuration directive with value of yes. You will need to restart the SSH server if this directive is modified.
2. Now you need to create the public and private key-pair for the Nagios SSH plugin to use. Login to the GroundWork Monitor server system with a system administrator account such as root, and then change credentials to the local nagios account, using the command su nagios.
3. Change to the nagios account's home directory, using the command cd ~.
4. Execute the command ssh-keygen -t dsa -b 2048. This will generate a 2048-bit DSA key-pair in the .ssh subdirectory under the nagios account's home directory. These files will be called id_dsa (the private key), and id_dsa.pub (the public key).
   
   The private key file will be used to request and prove user authorization in all subsequent SSH connection requests, and as such must be carefully secured. This file will usually be created with very strict permissions that only allow the owner to read and write the file, and some SSH clients will even refuse to use the private key if the file has overly lax permission. Since this file is only needed to request authentication, and is not used to grant access, this file does not need to leave its current location or have its permissions modified. On the other hand, however, the public key will be used to grant authorization requests, and as such that contents of the public key file must be added to each remote system that will be used with the Nagios SSH plugin.
5. On UNIX systems that use version 3 of the OpenSSH server, a user's public keys are stored in a file called authorized_keys that is located in the .ssh subdirectory under their home directory (other servers and versions may have different repository mechanisms). In order to allow the Nagios SSH plugin to successfully authenticate with each remote server, the contents of the id_dsa.pub file created in the preceding step must be added to the nagios user's authorized_keys file on each target system.

   The public key data can be copied into the authorized_keys file using any means available, but you must be careful to avoid adding any extra characters such as line feeds or carriage returns.

   Once the authorized_keys file has been updated, verify that the specified user account still owns the file, and still has sufficient permissions to the file, using the ls -la ~/.ssh/authorized_keys command.

   $ ls -la ~/.ssh/authorized_keys
   -rw-r--r-- 1 nagios Users 604 Apr 26 2007 /home/nagios/.ssh/authorized_keys

   Group and world file permissions are not as important on this file as they are for the private keys, since it only contains public key data which is harmless by itself. Users cannot use this file to obtain access to other systems, or to learn any secrets about the user account. In fact, some SSH servers may even require the file to be world-readable in order for the server itself to be able to access its contents.

6. Return to the GroundWork Monitor server, and switch to the nagios user account with the command su nagios. Verify that the user is able to use SSH to login to the correct account on the remote host, and that no passwords are exchanged during this process.

Once the steps above have been successfully completed, the Nagios SSH plugin should be able to seamlessly login to any target host that has the correct username and public key file. From this point forward, you will only need to be concerned with maintaining the key files themselves, and you can even change the passwords on the individual systems to random strings if you want, since they will not be needed by the plugin. Additional measures for further tightening access to the SSH plugin are discussed here http://www.linuxjournal.com/article/8759, if higher levels of security are needed or desired.

4.0 Managing Remote Commands

When Nagios issues a command on a remote server, it tries to ensure that the correct command is being called by looking for in a specific subdirectory under the primary user account's home directory. In order to ensure that the default set of SSH service definitions function as expected, it is necessary to ensure that these commands are copied into the appropriate subdirectory on the remote hosts. By default, this subdirectory is specified in the Nagios user22 macro definition, with the value of libexec (Configuration > Control > Nagios resource macros).

If the target host platform runs a nearly identical version of Linux and has the same processing architecture as the platform that is running the GroundWork Monitor server, then you can simply copy the contents of the local command directory to the appropriate target system by executing the following commands:

1. Login to the GroundWork server and switch to the nagios user account by using the su nagios command.
2. Copy the entire /usr/local/groundwork/nagios/libexec subdirectory to the target host with the command scp -pr /usr/local/groundwork/nagios/libexec destination:~, where destination is the IP address or hostname of the target system. The tilde character indicates that the input directory should be copied into the user's home directory, which will result in the creation of a libexec subdirectory under the user's home directory.

If the operating system and/or processor architecture of the target host is different from the GroundWork server platform, you will need to download the correct set of Nagios commands from Nagios Exchange http://www.nagiosexchange.org/, and install them on the target host system. Make sure to put them into the libexec directory under the SSH user account's home directory, and make sure that account has sufficient permission to execute the programs.

Once all the files have been put into the correct directory on the target host, you can verify that they will work as expected by logging into the GroundWork server under the Nagios SSH account, and then providing one of the remote commands as a parameter to an ssh command. This will cause the SSH client to login to the remote host with the current user's credentials, execute the specified command, display any output that was returned and then close the SSH session. For example, the following command will look at the amount of free swap space in use on the destination host, and generate a warning if there are less than 100 bytes of free space, or generate a critical error if there are less than 50 bytes of free space:

nagios$ ssh destination libexec/check_swap -w 100 -c 50
SWAP OK - 100% free (1349 MB out of 1349 MB) |swap=1349MB;0;0;0;1349

If this command completes successfully, the Nagios SSH plugin commands should also be able to work as expected.

The Perl scripts in the libexec directory are configured to use a Perl interpreter that is expected to be located in /usr/local/groundwork/common/bin, but that directory will not exist on the remote machine by default. You may need to update the Perl scripts to use a different interpreter, or create a symbolic link from an existing Perl interpreter to that directory so that one will be located.