Notification Configuration

Contents

This document describes how one configures and uses Notification Manager (NoMa) with GroundWork Monitor.

1.0 Notification Rule Components

In this section we focus on the components, including Contacts, Holidays, Contactgroups, Timeframes and Methods, which make up a notification rule used to send out alert notifications. The image below illustrates a notification rule and points out the tabs where the components are defined and where they are incorporated into the rule.

Throughout the various tabs; the Create button is used to produce a new definition, the Pencil icon to update an existing definition, the Plus icon to copy an existing definition, and the X icon is used to delete a definition. Defined components can then be applied within the Notifications tab. This section describes each of the components directives. 

Figure: Notification rule

1.1 Contacts

Contacts are individual definitions indicating who should get notified and how and when they should get notified in the event of a problem on your network. Contacts are applied in the Contactgroups, Holidays and Notifications tabs.

Directive Description
Full Name Full name of the person to receive the notification, displayed in various screens as the contact name.
Username User name, displayed in the Logs tab as a notification recipient.
Timezone Sets the time zone of the contact.
NoMa makes use of the Perl DateTime::TimeZone to provide proper time zone support (including winter/summer time) and simplifies worldwide support for working hours , (e.g., America/Los_Angeles).
Timeframe Sets the contacts availability to receive notifications, (e.g., 24x7).
Suppress multiple alerts If checked, no more than one notification for the same alert condition will be sent out to this particular contact.
If unchecked, indicates the contact will receive more than one notification for the same alert condition, if more than one alert is sent in for that condition.
Contact addresses Sets the contact's information for notifications: email, phone, mobile, Growl address.
Holiday Indicates a period of time a contact should not receive notifications. Selecting this option opens the Holidays tab and returns to the Contacts tab.

1.2 Contactgroups

Contact groups are definitions of one or more contacts and can be used to send alert or recovery notifications to a group of contacts. Contact groups can be created for an area of expertise (e.g., network-administrators) or geographic location (e.g., san_francisco-support). When a host or service has a problem or recovers, NoMa, as configured, will find the appropriate contact groups to send notifications to and notify all contacts. These Groups are applied in the Notifications tab.

Directive Description
Name (short) Name of the contact group.
Name (long) Descriptive name.
Timezone Sets the time zone of the contact group.
NoMa makes use of the Perl DateTime::TimeZone to provide proper timezone support (including winter/summer time) and simplifies worldwide support for working hours , (e.g., America/Los_Angeles).
Notification hours (Timeframe) Sets the contact groups availability to receive notifications, (e.g., 24x7).
Do not send notifications (to members)
If checked, contact group is a view-only group and notifications are not sent. The contact group is ignored while evaluating notification rules in response to incoming alerts.
If unchecked, notifications are sent to the contact group per notification rules.
Members Sets the contacts to be included in the contact group, the individuals that will be notified.

1.3 Holidays

Holiday definitions Indicate a period of time a contact should not receive notifications. The Holiday option is only displayed for previously defined contacts. Holidays are applied in the Contacts tab.

Directive Description
Name Name for holiday.
Start Start day and time.
End End day and time.
Timeframe Sets time frame in which holiday is effective.
Contact Sets defined contact associated with the holiday adding holiday to Contact's definition.

1.4 Timeframes

Schedule information including day of week, times and monthly occurrence. Timeframes are applied in the Contacts, Holidays, Contactgroups and Notifications tabs.

Directive Description
Name Name of time frame.
Schedule Information Sets time frame including day of week, from and to times with invert option (outside of from and to times), and choice of monthly occurance.
Validity Information Valid dates and time of time frame.

1.5 Methods

Individual notifications are configured in Methods. The name of a method definition is associated with a command, contact and in some instances a sender. The system defaults for each method are listed in the table below. Methods are applied in the Notifications tab.

Directive Description
Name Name of method, e.g., E-Mail, Growl, SMS
Command, Contact Field, Sender (method defaults) E-Mail: Emails are sent using the standard system mailer, you may need to configure your mail relay to accept mail from the system.
Command: sendemail, Contact: email, Sender: root@localhost, Fallback method: none, Acknowledable: No
See [How to enable sendEmail to work with TLS].

Growl: Growl notifications are sent using an included script, it requires the Perl module Net::Growl and by standard it uses UDP 9887 by default. After install, allow notifications from network, optional require password from LAN, server only requires Perl module Net::Growl as stated earlier. Client, depending on OS, requires an extra client: Mac:http://growl.info Linux: http://mattn.github.com/growl-for-linux/ Windows:http://www.growlforwindows.com/gfw
Command: growl, Contact: growladdress, Sender: none, Fallback method: none, Acknowledable: No

SMS: SMS alerts are sent via an attached SMS-capable modem using smstool3 or an iSMS/SMSFinder hardware device.
Command: sendsms_, Contact_: mobile, Sender: none, Fallback method: none, Acknowledable: No

Voice: Voice alerts are generated with an Asterisk (or Starface) soft PBX.
Command: voicecall_, Contact_: phone, Sender: none, Fallback method: none, Acknowledable: Yes

Voice + E-Mail fallback: Voice alert with email method fallback.
Command: voicecall_, Contact_: phone, Sender: none, Fallback method: E-Mail, Acknowledable: Yes

Voice + SMS fallback: Voice alert with SMS method fallback.
Command: voicecall_, Contact_: phone,_ Sender: none, Fallback method_: SMS,_ Acknowledable: Yes
Fallback method Method used if first method returns an error. This is currently only used by the voicecall plugin, to provide fallback to SMS or Email.
Acknowledgable Any method that is configured as acknowledable will cause the escalation chain to end, typically only voice should use this method as E-Mail and SMS provide no guarantee that a message has been received and understood.

2.0 Configuring Notifications

Essentially, notification rules define the who, what, where, when, why, and how of alert notifications. Rules contain various directives including contacts and contact groups defining who is to receive notifications, what monitored hosts and services should be included or excluded in the notifying, where and when notifications should be sent, and which event states to notify upon. Notification rules can be enabled (active) and disabled, and rules can be set to escalate notifications after a specified number of notifications is reached.

2.1 Notification Rules

Follow the steps below to create a notification rule. You may want to configure notification components to be used in the notification rule before you start.

  1. Go to Configuration > NoMa.
  2. From the Notifications tab, select the Create button to add a new notification rule, or to edit select the pencil icon corresponding to an existing rule.
  3. On the next screen, enter the notification information (the directives are described in the table below), then select Create/Save.

    Table: Notification rule directives
    Notification Information Name - Identifies the notification and is displayed on the Notifications front page.
    Description - Optional notification description.
    Activated - If checked means the notification is enabled, (a notification rule can also be enabled from the Manage Notification Rules page).
    Owner - Identifies the owner of a rule and allows the user to edit his own rules, administrators can edit rules of every user.
    Hosts and Services (comma-separated) Includes and Excludes - Indicates which hosts, host groups, services, service groups and recipients are to be included or excluded in the notification rule. An applied include constraint will only allow the notification rule to pass if the host or service belongs to at least one of the listed hostgroups or servicegroups. Conversely, an applied exclude constraint will only allow the notification rule to pass if the host or service does not belong to any of the listed hostgroups or servicegroups. See Filter Examples documented at the end of this page.
    Wildcard - The " * " wildcard character indicates "all" possibilities, (e.g., all hosts). Any include fields that are empty are assigned an implicit " * " and will automatically match all possibilities. In specifying comma-separated lists of hosts, hostgroups, services, servicegroups, or recipients for inclusion or exclusion filtering, '*' can be used as a multi-character wildcard and '?' can be used as a single-character wildcard, within any particular item in such a list.
    Hosts and Services - For Hosts and Services fields, typing a host name and pressing Tab will display a list of valid hosts and services.
    Contacts and Methods Contacts - Selected contact(s) will be notified.
    Groups - Selected contact group(s) will be notified.
    Methods - Selected method(s) will be used to send the notification.
    An element in each of the Contacts, Groups, and Methods is only associated with the present notification rule if its background is grayed indicating it has been selected. To select a single element click on the element, to select more than one element use the Control, Command, or Shift keys while making a selection.
    Timezone and Timeframe Timezone - Selected time zone is used for the notification (e.g., America/Los_Angeles).
    Timeframe - Selected timeframe indicates hours allowed for notifications to be sent, (e.g., 24x7).
    Which events to notify The checked filter options specifically define the recovery, problem, state transitions, and other states of change for which to send notifications.
    Numbers and more Notify after - Sets the notify after number indicating when notifications will start being sent.
    Rollover if the last rule is reached - If checked, enables NoMa to start again at number 1 when the last notification has been reached. If unchecked, NoMa stops notifying when the last notification/escalation has been reached.
    Let notifier handle escalations - If checked, enables NoMa to simulate the reception of notifications from Nagios and is useful for one-off alerts.
    Add Escalation Used to define escalations for the notification rule. Continues notifications after specified notify after directive.
  4. The escalations option is only displayed for previously defined notification rules. To add an escalation, make sure to save the notification rule, then go back to Configuration > NoMa > Notifications, and select the pencil icon corresponding to the rule to update.
  5. Then, select add Escalation toward the bottom of the screen.
  6. Next, select the Contacts and or Groups for the escalation, those to be notified, and the Methods of communication. The Notify after number indicates when this notification should be escalated. An element in each of the ContactsGroups, and Methods lists is only associated with the present escalation if its background is grayed which indicates it has been selected. To select a single element click on the element, to select more than one element use the Control, Command, or Shift keys while making a selection.

2.2 NoMa Commands

A script is provided for inclusion with Nagios as the notification command (host-notify-by-noma, service-notify-by-noma) that will be triggered for every host or service you desire to be controlled via NoMa.

The following updated commands are included in the GroundWork Monitor 7.2.0 release. Administrators should check existing commands and update if necessary for prior versions of GroundWork Monitor.
  • host-notify-by-noma
    /usr/local/groundwork/noma/notifier/alert_via_noma.pl -c h -s "$HOSTSTATE$" -H "$HOSTNAME$" -G "$HOSTGROUPNAMES$" -n "$NOTIFICATIONTYPE$" -i "$HOSTADDRESS$" -o "$HOSTOUTPUT$" -t "$TIMET$" -u "$$(( $HOSTPROBLEMID$ ? $HOSTPROBLEMID$ : $LASTHOSTPROBLEMID$ ))" -A "$$([ -n "$NOTIFICATIONAUTHORALIAS$" ] && echo "$NOTIFICATIONAUTHORALIAS$" || echo "$NOTIFICATIONAUTHOR$")" -C "$NOTIFICATIONCOMMENT$" -R "$NOTIFICATIONRECIPIENTS$"
  • service-notify-by-noma
    /usr/local/groundwork/noma/notifier/alert_via_noma.pl -c s -s "$SERVICESTATE$" -H "$HOSTNAME$" -G "$HOSTGROUPNAMES$" -E "$SERVICEGROUPNAMES$" -S "$SERVICEDESC$" -o "$SERVICEOUTPUT$" -n "$NOTIFICATIONTYPE$" -a "$HOSTALIAS$" -i "$HOSTADDRESS$" -t "$TIMET$" -u "$$(( $SERVICEPROBLEMID$ ? $SERVICEPROBLEMID$ : $LASTSERVICEPROBLEMID$ ))" -A "$$([ -n "$NOTIFICATIONAUTHORALIAS$" ] && echo "$NOTIFICATIONAUTHORALIAS$" || echo "$NOTIFICATIONAUTHOR$")" -C "$NOTIFICATIONCOMMENT$" -R "$NOTIFICATIONRECIPIENTS$"

You will need to associate the NoMa commands with a contact, that contact with a contact group, and that contact group with every object that will be using NoMa to alert, along with the appropriate notify conditions and time periods. On the occasion of a state change, this notification transmits the specifics to the NoMa daemon by calling the script; the daemon uses the specifics passed in the script compared to the stored filters in the NoMa database and on a match sends out a NoMa message to the contacts named in that matching filter. NoMa sends the message using a different script, one of several available for choice in the filter setup.

In Nagios/Monarch you will need to add/edit the contact to be associated with the forwarding of the alerting. In the example below we use the nagiosadmin contact which has the assigned nagiosadmin contact group, and notify-by-noma commands. This will associate the alert via NoMa script as a notification command for hosts and for services with nagiosadmin contact. You will also need to make sure that all hosts for NoMa alserts are setup with this contact group, (e.g., nagiosadmin). You may want to create a new contact and contact group.

  1. Go to Configuration > Contacts > ContactsModify, and select the contact to modify, (e.g., nagiosadmin).
  2. Set the Host notification commands box to host-notify-by-noma and the Service notification commands box to service-notify-by-service.
  3. Scroll down and set the contact group, (e.g., nagiosadmin), you may need to uncheck the inherit box to do so.
  4. Click Save.

    Figure: Set notify-by-noma commands

2.3 Receiving Notifications

Once notifications are enabled they can be received based on the methods defined and can be viewed in the NoMa application Logs tab.

  1. In Nagios, the Enable notification directive on the Nagios main configuration page controls the enabling and disabling of system wide notifications. Go to Configuration > Control, expand Nagios main configuration, select Notification.
  2. To enable check the box for Enable notifications, then click Save and Next >> 3 times, and Save and Done.
  3. From the left side navigation select Commit and follow the process to commit the configuration change.

    Figure: Enable notifications
  4. In NoMa, notification definitions can be turned on and off. To activate a notification definition, go to Configuration > NoMa > Notifications.
  5. Toggle the check mark icon to be green (activate) for the corresponding notification rule.

    Figure: Activate a notification rule


  6. Notifications will then be sent based on the directives set in the notification rule. The notification example email below shows the host ad-demo service windows-net in a CRITICAL state. The Link: is active and (if logged in) will go directly into the Status application for this device where you can acknowledge and or apply actions.

    Figure: Example NoMa email notification 
    ***** NoMa *****
    ID: 7
    Notification Type: PROBLEM
    Service: windows_net
    Host: ad-demo
    Host Alias: ad-demo
    State: CRITICAL
    Address: 172.28.111.16
    Link: [http://docs.groundwork.groundworkopensource.com/portal-statusviewer/urlmap?host=ad-demo&service=windows_net|http://docs.groundwork.groundworkopensource.com/portal-statusviewer/urlmap?host=ad-demo&service=windows_net]
    Info: A valid MODE and/or SUBMODE must be specified
    Date/Time: Thu Aug 24 08:53:29 2017
  7. The Logs tab displays NoMa notifications in a log which is accessible during runtime and displays alerts with status information in the order they are generated. The boxes at the top of the list enable filtering by content for any of the columns, for instance you can enter "Service" for the Check type or a recipient's name under Recipient to filter by a specific notification recipient.

    Figure: NoMa notification log

3.0 Examples

3.1 How Filters Work

Following is an example notification rule with host configuration for two hosts, applied filters, and the notification results.

Host Configuration
  Host 1 Host 2
Host mysql-lnxsrv01 mysql-lnxsrv02
Hostgroup(s) mysql, linux-servers mysql, linux-servers
Servicegroup(s)
mysql-svcs
mysql-svcs
Service (desc)
MySQL Service
MySQL Query Cache Hitrate
Host and Services Filters
  Includes Excludes
Hosts    
Hostgroup mysql  
Servicegroup mysql-svcs  
Service   mysql query*
Notification Results

For Host 1 the rule matches and the notification is sent. For Host 2 the rule does not match and the notification is not sent to members of this notification rule.

  Host 1 Host 2
Host    
Hostgroup Matches include
Matches include
Servicegroup Matches include
Matches include
Service   Matches exclude
Notification sent? Yes
No
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.