GroundWork System Log Files

Overview

This page describes the various GroundWork system log files and components, their settings, and locations. Included are log files for Java, GroundWork Monitor and Third Party applications.

CONTENTS

RELATED RESOURCES

WAS THIS PAGE HELPFUL?

1.0 Log Files

Java Applications

The logging libraries log4j is used by all the GroundWork Java applications. The log files (name, location, log rotation and format) generated by log4j are configured in the /usr/local/groundwork/config/log4j.xml file.

What is to be included in the logfile and what log level (ERROR, WARN, INFO, DEBUG) is defined by the log4j logger directives which is the namespace of the component including all sub-components, for example log4j.logger.org.groundwork.foundation.

Logger directives for the same namespace can be defined for multiple log files and therefore the same log message can show up in multiple log files at the same time.

The idea behind the different log files is to group messages by application type (e.g., framework, report, console) for better troubleshooting. For example, when running a report an error might be generated in the framework. The log file framework.log will show the error, but the file report.log will include it as well because it was related to the report application.

GroundWork Monitor Applications

All GroundWork Monitor applications have their own logs:

  • Event Console: console.log
  • Report Server: report.log
  • Status: statusviewer.log
  • Foundation Framework: framework.log
  • Foundation API, Internals: collage.log
  • Foundation Web Service API: foundation-webapp.log
Third Party Applications

In addition third party applications included in the GroundWork Monitor distribution have their own log files. Unfortunately we don't have much control over these files because their configuration is set inside the application that we distribute:

  • Jetty Servlet Container
    • HTTP requests to container: YYYY_MM_DD.request.log
  • BIRT Report Viewer
    • Report execution: ReportEngine_YYYY_MM_DD_HH_MM_SS.log
    • Report WS API: org.eclipse.datatools.enablement.oda_YYYY_MM_DD_HH_MM_SS.log

2.0 Log File Components

The following table documents all log files that are generated by GroundWork Monitor processes and scheduled jobs. The first column lists the name of the log component, the second column lists the log file description, settings (where the level and file name are set) and the location of the log file. The last column list the components retention control.

Table: Log Files generated by GroundWork Monitor

Component Log File Description, Settings, and Location Retention Control
Apache Description
Access and error logs for the Apache web server.

Settings
/usr/local/groundwork/apache2/conf/httpd.conf
(set in the CustomLog and ErrorLog options)

/usr/local/groundwork/apache2/conf/extra/httpd-ssl.conf
(set in the TransferLog and ErrorLog options)

Location
/usr/local/groundwork/apache2/logs/access_log
/usr/local/groundwork/apache2/logs/error_log 		logrotate
  Description
Secure-request access log for the Apache web server.

Settings
/usr/local/groundwork/apache2/conf/extra/httpd-ssl.conf
(set in the CustomLog option)

Location
/usr/local/groundwork/apache2/logs/ssl_request_log 		logrotate
autoimport module Description
Output summary of execution.

Settings
/usr/local/groundwork/common/etc/autoimport.conf
(set in the logfile option)

Location
/usr/local/groundwork/nagios/var/autoimport.log 		 
dashboard_avail_load.pl Description
Output summary of execution.

Settings
/usr/local/groundwork/common/etc/gwir.cfg
(set in the dashboard_avail_log option, indirectly referred to by the dashboard_avail_debug option)
This script is run via crontab for the nagios user. Stderr and stdout are redirected to a log file.

Location
/usr/local/groundwork/core/reports/utils/log/dashboard_avail.log
(configured log file for the dashboard_avail_load.pl script)

/usr/local/groundwork/core/reports/utils/log/dashboard_avail_load.log
(output and error streams from the dashboard_avail_load.pl script cron job) 		No retention control. By default, the log file is overwritten on each pass.
dashboard_data_load.pl Description
Output summary of execution.

Settings
/usr/local/groundwork/common/etc/gwir.cfg
(set in the dashboard_data_log option; indirectly referred to by the dashboard_data_debug option)
This script is run via crontab for the nagios user.  Stderr and stdout are redirected to a log file.

Location
/usr/local/groundwork/core/reports/utils/log/dashboard_data.log
(configured log file for the dashboard_data_load.pl script)

/usr/local/groundwork/core/reports/utils/log/dashboard_data_load.log
(output and error streams from the dashboard_data_load.pl script cron job) 		No retention control. By default, the log file is overwritten on each pass.
dashboard_lwp_load.pl Description
Output summary of execution.

Settings
/usr/local/groundwork/common/etc/gwir.cfg
(set in the dashboard_lwp_log option; indirectly referred to by the dashboard_lwp_debug option)

Location
/usr/local/groundwork/core/reports/utils/log/dashboard_lwp.log 		No retention control. By default, the log file is overwritten on each pass.
fping feeder module Description
Output summary of execution.

Settings
/usr/local/groundwork/config/fping_process.conf
(set in the logfile option)

Location
/usr/local/groundwork/foundation/container/logs/fping.log 		logrotate
gwprocesstrap.pl Description
Output summary of execution, including error messages.

Settings
/usr/local/groundwork/nagios/eventhandlers/gwprocesstrap.pl
(set in scripted code as the logfile option)

The gwprocesstrap.pl script is now obsolescent in the GW5.2.1 release.  Its functionality has been integrated into snmptt.  The standalone script is provided only to assist during migrations from older releases. New setups should not reference this script, and the configuration on upgraded systems should be modified so it no longer uses this script.

Location
/usr/local/groundwork/common/var/log/snmp/gwprocesstrap.log 		logrotate
importer.pl Description
The log records the file names/types that are detected and the number of rows matched in addition to configuration changes that are made to the system.

Settings
/usr/local/groundwork/log-reporting/lib/GWLogger.pm
(set in the scripting code)

Location
/usr/local/groundwork/log-reporting/logs/log-reporting.log 		logrotate
JBoss Portal (Foundation, BIRT Reports) Description
See Configuring Foundation.

Settings
Log levels for various Java components (almost all set by default to WARN) can be independently adjusted by editing corresponding lines in: /usr/local/groundwork/config/log4j.xml

See http://logging.apache.org/log4j/1.2/manual.html for a full description of log4j capabilities and the format and meaning of the configuration file entries. The precise meanings of the configuration entries depend on an understanding of the logging classes used within the Java application code, and so are beyond the scope of this documentation.

The other Foundation Container log file names are buried inside the compiled code and are not externally adjustable.

Location
log4j.xml configuration file
The Java code in GroundWork Monitor uses a set of logging classes to output messages, all ultimately derived from the log4j package. log4j effectively supports tagging each message with a hierarchy of logging classes and a severity level, which are matched against the configuration properties to determine which messages are actually emitted and to where they are sent. The properties file may be set up to route the same message to multiple destinations, and to apply different filtering to each destination.

In the standard setup that GroundWork ships in the log4j.xml file, various output files reflect different categories and filtering of the log messages:
  • framework.log should contain all log messages in all categories (subject only to the severity level that is set, with the default being 'warn').
  • report.log will contain all messages in the 'groundwork' class.
  • collage.log will contain messages particular to collage and foundation within the 'groundwork' class. There may be some overlap with the messages in the report.log file, so you may see entries in report.log that also show up in collage.log.statusviewer.log will contain only messages particular to the Status application.console.log will contain only messages particular to the Event Console application. The hierarchy with which such messages are tagged is outside of the 'groundwork' class, so you will only see duplicates of these messages in the framework.log file.
  • foundation-webapp.log similarly will contain only messages particular to Foundation activities.
    Log files named in the /usr/local/groundwork/config/log4j.xml config file are:
    FrameworkAppender produces:
    /usr/local/groundwork/foundation/container/logs/framework.log

    CollageAppender produces:
    /usr/local/groundwork/foundation/container/logs/collage.log

    FoundationAppender produces:
    /usr/local/groundwork/foundation/container/logs/foundation-webapp.log

    ReportAppender produces:
    /usr/local/groundwork/foundation/container/logs/report.log

    ConsoleAppender produces:
    /usr/local/groundwork/foundation/container/logs/console.log

    These files are rotated automatically, with old copies named with numeric suffixes, as specified in the log4j.xml file.

    Foundation Container Logs

    Rotation of files where the saved copies of these files have numeric suffixes will happen only when the files grow past a certain configured size. So, if there is no activity, the log does not get rotated.

    Files in /usr/local/groundwork/foundation/container/logs/ are:
    ReportEngine_YYYY_MM_DD_hh_mm_ss.log
    org.eclipse.datatools.connectivity.oda_YYYY_MM_DD_hh_mm_ss.log
    org.eclipse.datatools.enablement.oda_YYYY_MM_DD_hh_mm_ss.log
Controlled through log4j.xml settings.
  Description
Logging for JBoss Portal server under extreme failure conditions

Settings
/usr/local/groundwork/config/jboss-log4j.xml

Location
/usr/local/groundwork/foundation/container/logs/server.log 		Controlled through jboss-log4j.xml settings.
  Description
This file is generated by Foundation as it starts up, and therefore is the place to look if it does not start.

Location
/usr/local/groundwork/foundation/container/logs/boot.log 		This file is not under retention control because it is simply truncated and overwritten with each new restart.
postgres Settings
/usr/local/groundwork/postgresql/data/postgresql.conf

By default logging is turned off. To turn on logging on the following setting needs to be enabled:
logging_collector = on
log_rotation_age = 1d        #by days OR
log_rotation_size = 10MB  #by size}}

Changes to this file require a restart of the postgreSQL instance:
/etc/init.d/groundwork restart postgresql 		None of these files are under rotation or retention control.
Nagios Description
Variable detail on all input that Nagios processes and output it generates.

Settings
/usr/local/groundwork/nagios/etc/nagios.cfg
(set in the log_file option; indirectly referred to by the use_syslog, log_notifications, log_service_retries, log_host_retries, log_event_handlers, log_initial_states, log_external_commands, and log_passive_checks options)
(set in Monarch as Configuration -> Control -> Nagios Main Configuration, Page 1 -> Log file)

/usr/local/groundwork/nagios/scripts/ctl.sh
(set in the scripting code)

/usr/local/groundwork/core/reports/etc/gwir.cfg
(set in the nagios_event_log option)

/usr/local/groundwork/common/etc/autoimport.conf
(set in the nagioslogfile option)

Location
/usr/local/groundwork/nagios/var/nagios.log 		Nagios archiving.
  Description
Rotated/archived copies of the nagios.log file.

Settings
/usr/local/groundwork/nagios/etc/nagios.cfg
(base path set in the log_archive_path option; indirectly referred to by the log_rotation_method option)
(directory set in Monarch as Configuration -> Control -> Nagios Main Configuration, Page 1 -> Log archive path)

Location
/usr/local/groundwork/nagios/var/archives/nagios-*.log 		File rotation under control of Nagios; file retention under control of a nagios cron job.
  Description
Not a true (continuously growing) log file, despite the name; instead, contains a periodically updated snapshot of current internal Nagios state.

Settings
/usr/local/groundwork/nagios/etc/nagios.cfg
(set in the status_file option; indirectly referred to by the aggregate_status_updates and status_update_interval options)
(set in Monarch as Configuration -> Control -> Nagios Main Configuration, Page 1 -> Status file)

/usr/local/groundwork/feeder/nagios2collage_socket.pl
(set in the scripting code)

Location
/usr/local/groundwork/nagios/var/status.log 		Nagios
  Description
Stores performance data from all host checks for periodic processing. See Performance.

Settings
/usr/local/groundwork/nagios/etc/nagios.cfg
(set in the host_perfdata_file option, if host performance data logging is enabled; indirectly referred to by the perfdata_timeout, host_perfdata_command, host_perfdata_file_template, host_perfdata_file_mode, host_perfdata_file_processing_interval, and host_perfdata_file_processing_command options)
(set in Monarch as Configuration -> Control -> Nagios Main Configuration, Page 3 -> Host performance data file)

Location
/usr/local/groundwork/nagios/var/host-perfdata.dat 		Nagios; handled by logrotate as well only as a fail-safe mechanism.
  Description
Stores performance data from all service checks for periodic processing. See Performance.

Settings
/usr/local/groundwork/nagios/etc/nagios.cfg
(set in the service_perfdata_file option, if service performance data logging is enabled; indirectly referred to by the perfdata_timeout, service_perfdata_command, service_perfdata_file_template, service_perfdata_file_mode, service_perfdata_file_processing_interval, and service_perfdata_file_processing_command options)
(set in Monarch as Configuration -> Control -> Nagios Main Configuration, Page 3 -> Service performance data file)

/usr/local/groundwork/nagios/eventhandlers/launch_perf_data_processing
(set in the scripting code as the source_file option)

Location
/usr/local/groundwork/nagios/var/service-perfdata.dat 		Nagios; handled by logrotate as well only as a fail-safe mechanism.
Nagios-Foundation Feeders Description
Debug level defines output level: 0=debug off, 1=print summary, 2=also print xml messages.

Settings
/usr/local/groundwork/foundation/feeder/nagios2collage_eventlog.pl
(set in the logfile option; indirectly referred to by the debug option)

Location
/usr/local/groundwork/foundation/container/logs/nagios2collage_eventlog.log 		logrotate
  Description
Debug level defines output level: 0=minimal, 1=summary, 2=basic, 3=debug, 4=ridiculous

Settings
/usr/local/groundwork/foundation/feeder/nagios2collage_socket.pl
(set in the logfile option; indirectly referred to by the debug option)

Location
/usr/local/groundwork/foundation/container/logs/nagios2collage_socket.log 		logrotate
NSCA Description
Logs show connection attempts and submitted commands.

Settings
/usr/local/groundwork/common/etc/nsca.cfg
(logging level controlled by the debug option)

Location
Log messages are sent to the syslog facility (syslog-ng, on a GroundWork server). Such messages can be found in /var/log/messages by searching for the string "nsca". 		logrotate
process_service_perfdata_file Description
Variable detail on the results of processing the service-perfdata.dat file to create and update RRD files for graphing.

Settings
/usr/local/groundwork/nagios/eventhandlers/process_service_perfdata_file
(set in the debuglog option; indirectly referred to by the debug option)

Location
/usr/local/groundwork/nagios/var/log/process_service_perfdata_file.log 		logrotate
snmptrapd Description
Shows all traps received by snmptrapd.

Settings
/usr/local/groundwork/common/etc/snmp/snmpttd.init
(set in the scripting code as the LOGFILE option)

/usr/local/groundwork/common/scripts/ctl-snmptrapd.sh
(set in the scripting code as the SNMPTRAPD_LOGFILE option)

Location
/usr/local/groundwork/common/var/log/snmp/snmptrapd.log 		logrotate
snmptt Description
Output and debug messages generated during SNMP trap handling.

Settings
/usr/local/groundwork/common/etc/snmp/snmptt.ini
(pathnames set in the DEBUGGING_FILE, DEBUGGING_FILE_HANDLER, log_file, log_system_file, and unknown_trap_log_file options; see related DEBUGGING, stdout_enable, log_enable, log_system_enable, unknown_trap_log_enable, syslog_enable, syslog_facility, syslog_level, syslog_level_*, syslog_system_enable, syslog_system_facility, and syslog_system_level options)

Note: Alternate debugging-level flag and debug logging filename values (--debug=# and --debugfile=filename) can be passed in as parameters when starting snmptt (by adjusting the SNMPTT_OPTION1 string in the /usr/local/groundwork/common/scripts/ctl-snmptt.sh script). This is not generally recommended for production use, but may be helpful while isolating problems (though adjusting the snmptt.ini file should be the first resort). One might also modify the /usr/local/groundwork/common/sbin/snmptt script to directly set the default values for the debug level and debug file settings, though again such a hack is not recommended.

Location
/usr/local/groundwork/common/var/log/snmp/snmptt.debug
/usr/local/groundwork/common/var/log/snmp/snmptt.log
/usr/local/groundwork/common/var/log/snmp/snmptthandler.debug
/usr/local/groundwork/common/var/log/snmp/snmpttunknown.log
/var/log/snmpttsystem.log 		logrotate
supervise Description
The log files represent the output created by the scripts and applications that are launched by supervise.

Settings
The supervise process itself doesn't have any log messages reflecting its own status.  But messages are generated for each process that supervise is monitoring.

Location
Log files for standard GroundWork services are:
/usr/local/groundwork/core/services/foundation/log/main/log
/usr/local/groundwork/core/services/feeder-nagios-log/log/main/log
/usr/local/groundwork/core/services/feeder-nagios-status/log/main/log

Log files for optional services may include the folowing as well as similar log files for other modules placed under control of gwservices:
/usr/local/groundwork/core/services/check_ganglia/log/main/log
/usr/local/groundwork/core/services/feeder-nagios-fping/log/main/log 		logrotate
syslog-ng Description
Logs are for any application that submits messages to syslog-ng.

Settings
/usr/local/groundwork/common/etc/syslog-ng.conf
(message input defined by source options; message output defined by destination options, message routing and filtering defined by log options)

Location
Log entries written to specified files, including:
/var/log/messages
/var/log/kern.log
/usr/local/groundwork/common/var/log/syslog-ng/$HOST/$HOST.$YEAR-$MONTH-$DAY 		Generally under logrotate control.
postgres Description
Logs are for the postgres database system.

Settings
/usr/local/groundwork/postgresql/data/postgresql.conf
(all logging options available in this file.)

Location
Depends on options set. Postgres can log to syslog. In GroundWork by default we use:
/usr/local/groundwork/postgresql/data/pg_log/postmaster.log 		Postgres rotation.
grafana Description
Log for the Grafana graphing system.

Settings
/usr/local/groundwork/grafana/scripts/ctl.sh
(all logging options available in this script.)

Location
Depends on options set. In GroundWork by default we use:
/usr/local/groundwork/grafana/data/log/grafana.log 		 
influxdb Description
Log for the Influx database system.

Settings
/usr/local/groundwork/influxdb/etc/influxdb.conf
(all logging options available in this file.)

Location
Depends on options set. In GroundWork by default we use:
/usr/local/groundwork/influxdb/var/log/influxd.log 		logrotate

Labels

log log Delete
files files Delete
java java Delete
application application Delete
system system Delete
maintenance maintenance Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.