How to import RRDs into InfluxDB

WAS THIS PAGE HELPFUL? Leave Feedback

Overview

It is useful to import RRD data into InfluxDB in order to provide a smooth transition from RRD’s to InfluxDB.

For additional reference about related RRD import options, see the section RRD Data Importation Related Options from grafbridge-control -help. This utility is located in /usr/local/groundwork/grafana/scripts. You can enter ./grafbridge-control --help to view the built-in manual page.

Examples

RRD Importation Example - Scenario 1

In this scenario, a GroundWork Monitor 7.2.0 RRD-only system is converted over to an InfluxDB-only system.

GroundWork Monitor 7.2.0 is configured with performance data being sent to just RRD's. This might be the case if during an upgrade to 7.2.0, RRD's were chosen as the only time series database.
Later, GroundWork Monitor is re-configured to send performance data to just InfluxDB. To do this reconfiguration, grafbridge-control can be used as follows:
```
grafbridge-control -tsdb influxdb -rendering client -restart
```

RRD's can now be imported into InfluxDB to create a seamless visualization of the performance data over time, from when just RRD's were in operation, to when InfluxDB is in operation. To perform the RRD import into InfluxDB, grafbridge-control can be used as follows:

By entering the command grafbridge-control -import_rrds, the output will look something like this (host and service names will vary depending on your system's configuration):

root@gwserver:/usr/local/groundwork/grafana/scripts# ./grafbridge-control -import_rrds
--------------------------------------------------------------------------------
./grafbridge-control version v1.3.8
Arguments: -import_rrds
--------------------------------------------------------------------------------
[Mon Oct  2 08:43:25 2017] Checking prerequisites
[Mon Oct  2 08:43:25 2017] Migrating RRDs into InfluxDB ...
[Mon Oct  2 08:43:25 2017]   Getting a list of all hosts and their services from GroundWork (this can take a while of very large systems) ...
[Mon Oct  2 08:43:27 2017]   Processing RRD's for host 172.28.113.12:
[Mon Oct  2 08:43:27 2017]        icmp_ping_alive
[Mon Oct  2 08:43:27 2017]                Extracting RRD data from RRD /usr/local/groundwork/rrd/172.28.113.12_icmp_ping_alive.rrd
[Mon Oct  2 08:43:27 2017]                Writing extracted RRD data for RRD RRA rra0 to InfluxDB database groundwork
[Mon Oct  2 08:43:27 2017]                Writing extracted RRD data for RRD RRA rra1 to InfluxDB database groundwork
[Mon Oct  2 08:43:27 2017]        ssh_alive
[Mon Oct  2 08:43:27 2017]                Extracting RRD data from RRD /usr/local/groundwork/rrd/172.28.113.12_ssh_alive.rrd
[Mon Oct  2 08:43:27 2017]                Writing extracted RRD data for RRD RRA rra0 to InfluxDB database groundwork
[Mon Oct  2 08:43:27 2017]                Writing extracted RRD data for RRD RRA rra1 to InfluxDB database groundwork
[Mon Oct  2 08:43:27 2017]   Processing RRD's for host 172.28.113.2:
[Mon Oct  2 08:43:27 2017]        http_alive
[Mon Oct  2 08:43:27 2017]                Extracting RRD data from RRD /usr/local/groundwork/rrd/172.28.113.2_http_alive.rrd
[Mon Oct  2 08:43:27 2017]                Writing extracted RRD data for RRD RRA rra0 to InfluxDB database groundwork
[Mon Oct  2 08:43:27 2017]                Writing extracted RRD data for RRD RRA rra1 to InfluxDB database groundwork
…
…
[Mon Oct  2 08:50:39 2017]   Processing RRD's for host wingdma-dev:
[Mon Oct  2 08:50:39 2017]        icmp_ping_alive
[Mon Oct  2 08:50:39 2017]                Extracting RRD data from RRD /usr/local/groundwork/rrd/wingdma-dev_icmp_ping_alive.rrd
[Mon Oct  2 08:50:40 2017]                Writing extracted RRD data for RRD RRA rra0 to InfluxDB database groundwork
[Mon Oct  2 08:50:41 2017]                Writing extracted RRD data for RRD RRA rra1 to InfluxDB database groundwork
[Mon Oct  2 08:50:41 2017] Done.
root@gwserver:/usr/local/groundwork/grafana/scripts#

After the above steps have been completed, you can see the Service Availability and Performance Measurement in Status, for host localhost, and service local_memory, as shown below. In this example, the system was turned on Thursday 28th with only RRD configured. The next day it was re-configured to use only InfluxDB, and then the RRD's were imported. Note that since an RRD is found, it is also automatically rendered here too.

Figure: Status Service Availability and Performance Measurement

It is interesting to visualize in Grafana the imported RRD data and the RRD/InfluxDB data boundary. In this Grafana graph, the style has been changed to bars to aid visualization.

Figure: Status Service Availability and Performance Measurement

RRD Importation Example - Scenario 2

In this scenario, a GroundWork Monitor 7.2.0 system that is using both RRD and InfluxDB is converted over to an InfluxDB-only system. The steps are the same as from Scenario 1, except in step 1, GroundWork 7.2.0 is configured with performance data being sent to both RRD's and InfluxDB, as might be the case if during an upgrade to 7.2.0, both RRD's and InfluxDB were chosen as the only time series database.

After the steps have been completed, you can see the Service Availability and Performance Measurement in Status, for host localhost, and service local_memory, as shown below. In this example, the system was turned on Thursday 28th with RRD and InfluxDB configured. The next day it was re-configured to use only InfluxDB, and then the RRD's were imported. Note that since an RRD is found, it is also automatically rendered here too.

Figure: Status Service Availability and Performance Measurement

Again, it is interesting to visualize in Grafana the imported RRD data and the InfluxDB data. In this Grafana graph, the style has been changed to bars to aid visualization.

Figure: Status Service Availability and Performance Measurement

RRD Importation Process notes

The following algorithm is used:

          Gather a list of hosts and their services from GroundWork via the GroundWork API
          For each of the hosts services RRD files :
              Search for an associated RRD file for this host/service combination, and if found:
                 Extract datasources and their associated consolidation functions
                 For each of the Round Robin Archive (RRA)'s contained therein :
                     Extract the RRD data and build it into data that is InfluxDB-consumable
                     Write the data for the RRA to InfluxDB

The GroundWork instance is determined by settings in the file /usr/local/groundwork/config/ws_client.properties, which can be adjusted with the grafbridge-control -ws_props_file argument.
A /usr/local/groundwork/common/bin/rrdtool dump is used for data extraction.

InfluxDB notes

Imported RRD data is tagged with fromRRD=true

Imported RRD metrics are of the format:

<servicename>_<RRDdatasourcename>_<RRAconsoliationfunction>_<RRAprimarydatapointseconds>

Example:

local_load_load15_AVERAGE_3600seconds

Since GroundWork RRD file names are of the format <hostname_servicename>.rrd, the host name and service name that generated the RRD are not deducible since host name and service name can contain underscores too. Also, only importing data that belongs to hosts in the GroundWork system is sufficient.
The importation processing time will depend on the number of RRDs to be imported, and how much non NaN data is in the RRD's.