Tutorial A Real-World Adapter for SNMP Traps

Overview

Foundation Adapter Development - The following text describes step by step how to develop a new Feeder and Adapter for inserting custom data streams into the Foundation framework. The input source will be SNMP Traps. Development Environment - The Adapters are distributed as Java libraries (jar files) and therefore you need the JAVA SDK available for compiling and packaging the Adapters. The tutorial uses ANT and MAVEN as build tools. You can get ANT and Maven binaries from the Apache site. The tutorial uses the Foundation source distribution which is available from http://sourceforge.net/projects/gwfoundation.

CONTENTS

RELATED RESOURCES

WAS THIS PAGE HELPFUL?

A. Before You Start

  1. Install and configure the build tools and Foundation.
  2. Make sure that foundations build runs without errors.
  3. Define the data that needs to be integrated.
  4. What is the Application scope? SNMP traps are treated as a separate application and therefore we have to create a new ApplicationType: SNMPTRAP.
  5. What is the Entity Type? SNMP trap messages will be stored in the LogMessage table and therefore the EntityType is LOG_MESSAGE.
  6. What fields need to be stored? Define the attributes that are generated by the application and specify the if required and the default values if not required.

    Table: Fields to be stored
    Attribute Type Property Required/Default Value
    Host String No Required
    Severity Class Severity No Required
    IpAddress String Yes Required
    MonitorStatus Class Severity No Same as Severity
    ReportDate Date No Default set at time inserted
    LastInsertDate Date No Default set at time inserted. Can be set by SNMP_LOG message
    Event_OID_numeric String Yes Not required, default to NULL
    Event_OID_symbolic String Yes Not required, default to NULL
    Event_Name String Yes
    		Not required, default to NULL
    Category String Yes Not required, default to NULL
    Variable_Bindings String Yes Not required, default to NULL
    TextMessage String No Not required, default to NULL
  7. Select fields that are application specific. A list of fields needs to be defined that are properties attached to the table defined as the Entity Type. The properties are application specific and are not part of the base table. For SNMP traps the following fields and types will be stored.

    Table: Defined fields
    Property/Name Type
    IpAddress String
    Event_OID_numeric String
    Event_OID_symbolic String
    Event_Name String
    Category String
    Variable_Bindings String
  8. Select fields for data consolidation. This feature reduces the number of identical messages in the LogMessage table. For each insert, the consolidation criteria will be applied to the incoming message as long as the flag in the XML stream is set to a consolidation criteria name (consolidate="SNMPTRAP" ). By default no consolidation criteria is applied. If the consolidation criteria matches that of an existing message in the Log Message table, the message counter for the existing message will increased and the date fields will be updated as following:

    Table: Fields for data consolidation
    Field Change
    FirstInsertDate Unchanged
    LastInsertDate ReportDate
    ReportDate System (current time)

Fields that Need to Match before a Message Gets Consolidated:

  • OperationStatus
  • Host
  • Severity
  • IPaddress
  • MonitorStatus
  • Event_OID_numeric
  • Event_Name
  • Category
  • Variable_Bindings

Once the data set is defined and properties and consolidation criteria are defined, the next step will be to update the database Metadata and write an Adapter for SNMP data normalization.

B. Database Updates

The following database updates can be integrated into one database script that ships with the new Adapter. It's always a good idea to check for the existence of an entry before attempting to insert that entry into the database, and abort if they are already present. But for readability of the tutorial these steps are not documented. Since a new application was created the Application needs to be added to the database; INSERT INTO ApplicationType(Name, Description) VALUES ("SNMPTRAP", "SNMP Trap application").

The new properties need to be created and assigned to the ApplicationType and EntityType:

INSERT INTO PropertyType(Name, Description, isString) VALUES ("ipaddress", "ipdddress of snmp device", 1);
INSERT INTO PropertyType(Name, Description, isString) VALUES ("Event_OID_numeric", "Event_OID_numeric", 1);

Continue for all property types.

INSERT INTO  ApplicationEntityProperty(ApplicationTypeID, EntityTypeID,  PropertyTypeID, SortOrder) VALUES ((SELECT ApplicationTypeID FROM  ApplicationType WHERE Name='SNMPTRAP'),(SELECT EntityTypeID FROM  EntityType WHERE Name='LOG_MESSAGE'),(SELECT PropertyTypeID FROM  PropertyType WHERE Name = 'ipaddress'), 1);
INSERT INTO  ApplicationEntityProperty(ApplicationTypeID, EntityTypeID,  PropertyTypeID, SortOrder) VALUES ((SELECT ApplicationTypeID FROM  ApplicationType WHERE Name='SNMPTRAP'),(SELECT EntityTypeID FROM  EntityType WHERE Name='LOG_MESSAGE'),(SELECT PropertyTypeID FROM  PropertyType WHERE Name = 'Event_OID_numeric'), 1);

Continue for all properties.

The consolidation criteria needs to be named and the criteria of matching fields is a semicolon separated list.

INSERT INTO ConsolidationCriteria(Name, Criteria)  VALUES ("SNMPTRAP", "OperationStatus;Host;Severity;ipaddress,  MonitorStatus;  Event_OID_numeric;Event_Name;Category;Variable_Bindings");

C. Writing The Feeder

The Feeder captures SNMP trap and sends XML formatted messages where all the fields to monitor are XML attributes to the Foundation listener component. The Listener listens on a configurable socket for incoming messages. The default is set to port 4913. For SNMP traps the XML messages look like the following:

<SNMPTRAP
MonitorServerName="localhost"
Host="cisco2900.itgroundwork.com"
Severity="Normal"
MonitorStatus="Normal"
ReportDate=""2005-10-25 04:20.44"
LastInsertDate="2005-10-25 04:20.44"
ipaddress="192.168.2.203"
Event_OID_numeric=".1.3.6.1.4.1.9.0.1"
Event_OID_symbolic="enterprises.9.0.1"
Event_Name="tcpConnectionClose"
Category="Status Events"
Variable_Bindings="enterprises.9.2.9.3.1.1.1.1:5  tcpConnState.192.168.2.203.23.192.168.2.249.38591:synReceived  enterprises.9.2.6.1.1.5.192.168.2.203.23.192.168.2.249.38591:600  enterprises.9.2.6.1.1.1.192.168.2.203.23.192.168.2.249.38591:70  enterprises.9.2.6.1.1.2.192.168.2.203.23.192.168.2.249.38591:101  enterprises.9.2.9.2.1.18.1:"
TextMessage="A tty trap signifies that a TCP connection, previously  established with the sending protocol entity for the purposes of a tty  session, has been terminated. 5 synReceived 600 70 101 " />

D. Writing An Adapter

Creating the Java Project
  1. Inside the expanded Foundation package, go into collagefeeder/adapter and create a new folder called snmp.
  2. Step inside the snmp directory and create a source/java directory.
  3. Create a new maven.xml file that looks like the following: 
    <project default="jar:install"
    xmlns:j="jelly:core"
    xmlns:maven="jelly:maven"
    xmlns:ant="jelly:ant">
        <goal name='distro'>
            <attainGoal name='clean'/>
            <attainGoal name='jar'/>
            <delete dir='./dist' />
            <mkdir dir='./dist' />
            <mkdir dir='./dist/lib' />
            <copy todir="./dist/lib" file="${maven.build.dir}/${maven.final.name}.jar"/>
            <j:forEach var="lib" items="${pom.artifacts}">
                <j:set var="dep" value="${lib.dependency}"/>
                <j:if test="${dep.getProperty('war.bundle')=='true'}">
                <copy todir="./dist/lib" file="${lib.path}"/>
                </j:if>
            </j:forEach>
        </goal>
        <goal name='allBuild'>
            <attainGoal name='distro'/>
        </goal>
</project>
  4. Create a project.xml that looks like the following. Note that the version and the libraries are inherited from the main project file: 
    <project>
    <pomVersion>3</pomVersion>
    <extend>../project.xml</extend>
    <id>collage-adapter-snmp</id>
    <name>Groundwork Collage Adapters for SNMP</name>
    <package>com.groundwork.feeder</package>
        <build>
        <sourceDirectory>src/java</sourceDirectory>
          <resources>
            <resource>
              <directory>${basedir}/src/java</directory>
              <excludes>
                <exclude>**/*.java</exclude>
              </excludes>
            </resource>
          </resources>
        </build>
</project>
  5. Create a new package by creating the following directories under src/java: com.groundwork.feeder.adapter.impl.
    Now the setup of the new java project that will include the SNMP Adapter is done. The next step will show what classes need to be implemented.
Classes and Method to Overwrite

The Adapter has to implement the FeederBase interface which is part of the adapter-api. Class creation:

  1. Create a new class SNMPTrap inside com/groundwork/feeder/adapter/impl that implements the FeederBase (Note: will be renamed to AdapterBase).
  2. Implement GetName that returns the Adapter name. The name has to match the node name of the XML fragment sent to the listener. For snmp traps, it is SNMPTRAP.
  3. Implement initialize() and uninitialize() for any actions that need to be executed when the Adapter gets loaded or unloaded by the framework. Implement the method process() that gets called by the framework for each incoming XML stream that is of the Adapter name (SNMPTRAP). Into this method goes the normalization code that transforms the XML message to a database call. The main steps are:
    • Parse the XML stream and extract the attributes. Use the utils.getAttributes() method.
    • Get the API object by calling into the bean factory.
    • Create a properties map and call into the API.
Spring Assemblies for SNMPTrap Adapter

The new SNMPTrap Adapter will be loaded into the Spring container. What you have to include into your JAR file is the spring assembly file which has to be created in the src/java/META-INF directory. Steps:

  1. Create a new folder META-INF inside your project's snmp/src/java directory
  2. In META-INF create a file called assembly-adapter-snmptrap.xml. A sample of the whole file can be seen below under Spring Assembly for SNMPTRAP Adapter.
Building the Adapter Package

Inside the adapter/snmp directory execute: maven jar. This will create the jar file in the target directory. Executing: maven jar:install This will copy the jar file into the local repository.

E. Installing And Configuring The SNMPTRAP Adapter

Once the Adapter is compiled successfully it needs to be deployed into the listener installation and the adapter.properties need to be updated with the new Adapter entry.

  1. Copy the jar file collage-adapter-snmp-1.1.jar into the listener library path /usr/local/groundwork/collage/feeder/lib. Edit adapter.properties for the new adapter as following:
    First, increment the counter for assemblies: 
    # Spring assemblies
nb.assemblies = 3

    Add the assembly name and the Property Bean name:

    # SNMPTrap Beans
adapter.assembly3 = META-INF/assembly-adapter-snmptrap.xml
adapter.properties.assembly3 = SNMPTrapAdapterProperties
  2. Start the listener.
  3. Feed data to the listener and verify that the data shows up in the database. Check the log files for any errors.

F. SQL Script For SNMPTRAP Metadata Creation

# Database changes for SNMPTRAP messages
# Add new ApplicationType for SNMPTRAP
DELETE FROM ApplicationEntityProperty WHERE  ApplicationTypeID = (SELECT ApplicationTypeID FROM ApplicationType WHERE  Name='SNMPTRAP') %% EntityTypeID = (SELECT EntityTypeID FROM EntityType  WHERE Name='LOG_MESSAGE');
INSERT INTO ApplicationType (Name, Description) VALUES ("SNMPTRAP","SNMP Trap application");
# Add the properties specific to SNMPTRAP
INSERT INTO PropertyType(Name, Description, isString) VALUES ("IpAddress", "ipadddress of snmp device", 1);
INSERT INTO PropertyType(Name, Description, isString) VALUES ("Event_OID_numeric", "Event_OID_numeric", 1);
INSERT INTO PropertyType(Name, Description, isString) VALUES ("Event_OID_symbolic", "Event_OID_symbolic of snmp device", 1);
INSERT INTO PropertyType(Name, Description, isString) VALUES ("Event_Name", "Event_Name", 1);
INSERT INTO PropertyType(Name, Description, isString) VALUES ("Category", "Category of snmp device", 1);
INSERT INTO PropertyType(Name, Description, isString) VALUES ("Variable_Bindings", "Variable_Bindings", 1);
# Assign the SNMP properties to Application Type SNMPTRAP and Entity LOG_MESSAGE
INSERT INTO ApplicationEntityProperty(ApplicationTypeID, EntityTypeID,  PropertyTypeID, SortOrder) VALUES ((SELECT ApplicationTypeID FROM  ApplicationType WHERE Name='SNMPTRAP'),(SELECT EntityTypeID FROM  EntityType WHERE Name='LOG_MESSAGE'), (SELECT PropertyTypeID FROM  PropertyType WHERE Name = 'IpAddress'), 1);
INSERT INTO ApplicationEntityProperty(ApplicationTypeID, EntityTypeID,  PropertyTypeID, SortOrder) VALUES ((SELECT ApplicationTypeID FROM  ApplicationType WHERE Name='SNMPTRAP'),(SELECT EntityTypeID FROM  EntityType WHERE Name='LOG_MESSAGE'), (SELECT PropertyTypeID FROM  PropertyType WHERE Name = 'Event_OID_numeric'), 1);
INSERT INTO ApplicationEntityProperty(ApplicationTypeID, EntityTypeID, PropertyTypeID, SortOrder) VALUES ((SELECT ApplicationTypeID FROM  ApplicationType WHERE Name='SNMPTRAP'),(SELECT EntityTypeID FROM  EntityType WHERE Name='LOG_MESSAGE'),(SELECT PropertyTypeID FROM  PropertyType WHERE Name = 'Event_OID_symbolic'), 1);
INSERT INTO ApplicationEntityProperty(ApplicationTypeID, EntityTypeID,  PropertyTypeID, SortOrder) VALUES ((SELECT ApplicationTypeID FROM  ApplicationType WHERE Name='SNMPTRAP'), (SELECT EntityTypeID FROM  EntityType WHERE Name='LOG_MESSAGE'),(SELECT PropertyTypeID FROM  PropertyType WHERE Name = 'Event_Name'), 1);
INSERT INTO ApplicationEntityProperty(ApplicationTypeID, EntityTypeID, PropertyTypeID, SortOrder) VALUES ((SELECT ApplicationTypeID FROM  ApplicationType WHERE Name='SNMPTRAP'), (SELECT EntityTypeID FROM  EntityType WHERE Name='LOG_MESSAGE'), (SELECT PropertyTypeID FROM  PropertyType WHERE Name = 'Category'), 1);
INSERT INTO ApplicationEntityProperty(ApplicationTypeID, EntityTypeID,  PropertyTypeID, SortOrder) VALUES ((SELECT ApplicationTypeID FROM  ApplicationType WHERE Name='SNMPTRAP'),(SELECT EntityTypeID FROM  EntityType WHERE Name='LOG_MESSAGE'),(SELECT PropertyTypeID FROM  PropertyType WHERE Name = 'Variable_Bindings'), 1);
#Create consolidation criteria
INSERT INTO ConsolidationCriteria(Name, Criteria)  VALUES ('SNMPTRAP',  'Host;Severity;IpAddress;MonitorStatus;Event_OID_numeric;Event_Name;Category;Variable_Bindings')

G. Spring Assembly For SNMPTrap Adapter

The following file assembly-adapter-snmptrap.xml needs to be included into the jar package for the SNMPTrap Adapter inside META-INF:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">
<beans>
<!--
List all the BeanID that have implemented the initialize method. The bean ID's defined as a comma
separated list will be called during the loading of the assembly
-->
<bean id="SNMPTrapAdapterProperties" class="com.groundwork.feeder.adapter.impl.AdapterProperties">
<constructor-arg type="java.lang.String"><value>adapter.snmptrap</value></constructor-arg>
 </bean>
<bean id="adapter.snmptrap" singleton="false"
      class="com.groundwork.feeder.adapter.impl.SNMPTrap" />
</beans>

H. Creating A Feeder For LOG4J And Using The Generic Log Adapter

The Foundation Adapters are pluggable modules, written in Java, that normalize data from an Application before it gets inserted into the data store. Normalization is needed for complex data structures, but a simple Generic Adapter with several pre-defined fields might be sufficient for many external log messages, including some application logs. Foundation provides a GenericLog Adapter that can be used to add log data into Foundation without writing any code. The values assigned to pre-defined fields will be stored in the database under a user defined Application Type. The following example shows how LOG4J (used by Application Servers) can be fed into Foundation.

Installing and Using the Generic Log Adapter

The Generic Log Adapter is included in all distributions of Foundation.

Definitions

An ApplicationType for the Adapter to log data against must be defined. In this example the ApplicationType will be LOG4J, but this is user defined and may be adjusted to any type.

Generic Adapter Input

The following attributes are required as part of the XML feed, otherwise the message will be rejected: ApplicationType, MonitorServerName, Device, Severity, TextMessage

In addition to the required properties the following properties are accepted but optional: Host, MonitorStatus, ReportDate, OperationStatus, ApplicationSeverity, Component, Priority, ServiceDescription, Priority, and TypeRule.

Notes about some fields:

  • If OperationStatus is not defined it will be set to Open.
  • If ReportDate is not set the current time (system) will be used.
  • If MonitorStatus is not defined it will be set to UNKNOWN. MonitorStatus defines the color in the console: OK = Green, DOWN = Red, WARNING = Yellow, and UNKNOWN = Blue.
  • If the FeederScript defines values for the Host and ServiceDescription attributes, the LogMessage will be linked (via a Foreign Key) to the matching ServiceStatus. This can be used in the GroundWork Monitor UI to link from an Event Message to the ServiceStatus.
  • If Host is set the LogMessage will be linked to HostStatus.
  • If Host and Service Description are set the message will be linked to an existing ServiceStatus entry.
  • If the FeederScript defines a value for the Host attribute, the LogMessage will be linked (via a Foreign Key) to the matchng HostStatus entry.
System Setup

Create a new ApplicationType (LOG4J) in the database:

Stop the listener and update the database with a SQL statement. In GroundWork Monitor, these statements will work:

Stop gwservices from the bash command line:

# /usr/local/groundwork/ctlscript.sh stop gwservices

PostgreSQL command line:

# psql -U postgres gwcollagedb
gwcollagedb=# INSERT INTO ApplicationType (Name, Description) VALUES("LOG4J","LOG4J Events");
gwcollagedb=#\q

Start gwservices:

# /usr/local/groundwork/ctlscript.sh start gwservices

I. Feeder Code Example

Here is an example of the simplest possible script that can be used to send one message to the Event table:

#!/usr/local/groundwork/perl/bin/perl --
#
#Copyright 2003-2011 Groundwork, Inc.
#[http://www.gwos.com]
#
#Unless required by applicable law or agreed to in writing, software
#distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
#WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
#License for the specific language governing permissions and limitations under the License.
#
use IO::Socket;
my $debug =1 ;
my $remote_host = "localhost";
my $remote_port = 4913;
my $socket;
if ( $socket = IO::Socket::INET->new(PeerAddr => $remote_host,
PeerPort => $remote_port,
Proto    => "tcp",
Type     => SOCK_STREAM)
) {
my $xml_message = "<GENERICLOG ApplicationType='LOG4J' MonitorServerName='localhost' Host='dashboard.itgroundwork.com' Device='dashboard.itgroundwork.com' Severity='WARNING' MonitorStatus='WARNING' ErrorType='LogRotation' SubComponent='LOG4J Integrator' TextMessage='16:15:54,593 [WARN ] com.groundwork.collage.impl.LogMessageDAOImpl - Consolidation criteria matches with more than one record. Make sure the criteria is better defined. If the consolidation criteria was turned on after identical messages were inserted you have to run consolidate existing messages. Contact support for more information about database maintenance.' />" ;
              print $xml_message."\n\n" if $debug;
              print $socket $xml_message;
              my $xml_message = "<SERVICE-MAINTENANCE command=\"close\" />";
              print $xml_message."\n\n" if $debug;
              print $socket $xml_message;
             } else {
print "Couldn't connect to $remote_host:$remote_port : $@\n";
}

Labels

adapter adapter Delete
snmp snmp Delete
trap trap Delete
tutorial tutorial Delete
foundation foundation Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.