Overview

This page reviews how to set up and monitor your Amazon EC2 infrastructure using the GroundWork Cloud Hub AWS connector. The connection requires a unique set of parameters (e.g., endpoint, credentials). You will need your GroundWork server and virtual environment connector parameters handy.

1.0 Adding a New Connection

The initial Cloud Hub screen is used to add, start, stop, modify, or delete available connectors. Follow the steps below to add a connection. You will need to create a new connection in this way for each region to be monitored.

1. Log in to GroundWork Monitor as an Administrator.
2. Select GroundWork Administration > GroundWork Cloud Hub.
3. Click +Add corresponding to the AWS connector icon.
   
   Figure: Adding a connection
   

2.0 Configuring the Connection

In the configuration page you will need to enter both the GroundWork server and remote server parameters.

The data the GroundWork server receives comes from the remote virtualization server. The GroundWork server is where Cloud Hub will store the connectors metrics. The information is pulled from the API on a periodic basis based on the check interval that is set. Selected views such as Network View determine metric options and become visible in the monitoring output.

2.1 GroundWork Server Parameters

1. Here we enter the GroundWork server parameters, each described in the table below.
   
   Figure: GroundWork server values
   
   
   Table: GroundWork server values

   GroundWork Server Version	The GroundWork server version number which is set to the latest release installed. Cloud Hub can be configured to talk to versions from GroundWork Monitor version 7.0.
   Display Name	This is the configuration’s name displayed in the list of Cloud Hub connectors on the Cloud Hub home page.
   GroundWork Server Name	The host name or IP address of the GroundWork server that will integrate the Cloud Hub messages. A port number should not be entered here. If Cloud Hub is running on the same server as the portal the name can be localhost, or as preferred the server name. Often the GroundWork server is the same server where Cloud Hub is running however, Cloud Hub can also run in a distributed environment on its own node in a GroundWork cluster.
   Is SSL enabled on GroundWork Server?	Check this box if the GroundWork server is provisioned with a secure HTTPS transport.
   GroundWork Web Services Username	This is the provisioned user name granted API access on the GroundWork server. This value defaults to RESTAPIACCESS, and can be located by opening a tab to the GroundWork Administration > GroundWork License page. The value is under Webservices API Account Info in the Access field. For all systems, make sure the Web Services Username and Password match with the entries in the ws_client.properties file.
   GroundWork Web Services Password	This is the corresponding API Token (password) for the given user name on the GroundWork server. This is obtained by opening a tab to the GroundWork Administration > GroundWork License page. The value is under Webservices API Account Info in the Token (encrypted) field. Along with the GroundWork Web Services Username, these are the same credentials set within the /usr/local/groundwork/config/ws_client.properties file. For all systems, make sure the Web Services Username and Password match with the entries in the ws_client.properties file.
   Merge hosts on GroundWork Server?	If checked, this option combines all metrics of same named hosts under one host. For example, if there is a Nagios configured host named demo1 and a Cloud Hub discovered host named demo1, the services for both configured and discovered hosts will be combined under the hostname demo1 (case-sensitive).
   Enable HostGroup Tagging?	Check this option to enable support for Amazon EC2 and EBS tagging as a mapping mechanism for host groups, defined in the next directive.
   HostGroup Tag Name	Cloud Hub can map your AWS resource tags to GroundWork host groups. This entry represents the key name for the GroundWork tag. All instances with this tag key will be mapped to the GroundWork Monitor host group (e.g., GWTraining) with the EC2 tag value (e.g., trainingservers). There may be multiple host groups with the same tag name and tag names are case-sensitive. Refer to Appendix A: Host Group Tagging on this page for a detailed description of this feature.

2.2 Remote Server Parameters

1. Here we enter the remote server parameters, each described in the table below.
   
   Figure: Remote server values
   
   
   Table: AWS server values

   Is SSL enabled on Amazon Server?	Check this box if the Amazon server is configured for secure HTTPS.
   AWS Region Endpoint Domain	This is the Web Service endpoint for a region (e.g., us-west-2.amazonaws.com for region Oregon). Create a connector like this for each region deployed to point to the endpoint for that region. The endpoint is where you access the CloudWatch API.
   AWS Access Key ID	This can also be a common user name set up for command line access (does not have to be the master account). It should be clear that the user name must be assigned rights to the region that you wish to monitor.
   AWS Secret Access Key	This can also be a common password set up for command line access (does not have to be the master account). It should be clear that the password must be assigned rights to the region that you wish to monitor.
   Check Interval (in mins)	This is the polling interval for collecting monitoring data from the virtual instance and sending it to the GroundWork server. It defines how often Cloud Hub will query Amazon CloudWatch for change updates. The value is in minutes.
   Connection Retries (-1 infinite)	This entry is the number of retries for the connection and sets a limit on how many attempts are made after a failure. If you set this to -1, the retrying goes on forever. The number set indicates how many connections are attempted before the connection is left inactive (until you restart it).
   Views	When checked, specifies a view to report upon. When unchecked the metrics will be deleted. Availability Zone View - Displays instances organized by the availability zone within connector region. Storage View - Shows instances according to the Amazon EBS resources where they are assigned. Network View - Shows instances as they are assigned to Virtual Private Cloud subnets and Relational Database Service (RDS) metrics. Custom View - Enables a query that will retrieve all active custom CloudWatch metrics in the zone being connected. The AWS connector now provides custom metrics as a way to have your applications report performance into an arbitrary metric that you can design. You can tell EC2 what you want to keep track of and have your applications send what is being tracked to Amazon to record regularly.

2. After the remote server parameters have been entered, click SAVE which saves and writes the entries to an XML file in the GroundWork server /usr/local/groundwork/conifg/cloudhub directory. The Cloud Hub connector is assigned an agent ID and that in turn becomes a record locator in Foundation when you begin monitoring.
3. Next, validate the configuration by selecting TEST CONNECTION which will check if the connection is accessible with the given credentials. A dialog will be displayed with either a success message or, if the server cannot be contacted, an error message will be displayed with information describing why the connection failed.
4. After the credentials have been validated, select NEXT to display an associated connection metrics screen where you can determine the metrics to be monitored.

3.0 Determining Metrics To Be Monitored

Each management system provides metrics for specific checks that can be defined for the instance or the container. The property name and the thresholds are defined in a monitoring profile in an XML format.

In the UI the metrics are separated into categories, e.g., AWS CloudWatch Thresholds and AWS Custom Metrics, where you can define if a metric should be monitored, graphed, and also set the values for Warning and Critical thresholds at which to trigger alerts. By default some of these thresholds are set to -1 which turns them off so you can get an idea by watching the data coming in and then setting a value appropriate for the environment. The Service Name field enables overriding of the automatically generated service name. 

The selections made are applied to every instance discovered in the region. The set of selections is saved on the GroundWork server in the /usr/local/groundwork/config/cloudhub/profiles directory as a profile in an XML file. The selections become effective both against new instances that may be discovered as well as already monitored instances.

For additional information regarding Amazon EBS, Amazon RDS, and Amazon CloudWatch Custom metrics see Appendix B: Notes on AWS Metrics.

1. Here we adjust any profile selections, each described in the table below.
   
   Figure: AWS profile metrics
   
   
   Table: AWS profile attribute options

   Attribute	The name of the service attribute (the metric name reported by the virtualization server).
   Monitored	When on (checked) the service will be monitored.
   Graphed	When on (checked) the service will be graphed.
   Warning and Critical Thresholds	These values control the triggering of alerts. A Warning number larger than the Critical value will cause Cloud Hub to detect the metric as a trigger. Choosing a -1 in a threshold box will disable triggering on that alert.
   Service Name	Cloud Hub automatically creates service names based on the metric name gathered from a virtualization server. The Service Name option adds the ability to report the polled metrics under a unique name that is set by the Administrator. For example, in the image above you can see the attribute EBS.VolumeIdleTime has been assigned the service name IDLE. This custom name will then override the name that was automatically created for the service based on the metric name gathered from a virtualization server, EBS.VolumeIdleTime. Leaving the Service Name field blank defaults to the metric name reported by the virtualization server. All Cloud Hub connectors now support the editable Service Name feature.
   Description	A description of the service attribute.
   Refresh	The AWS connector now provides custom metrics as a way to have your applications report performance into an arbitrary metric that you can design. You can tell EC2 what you want to keep track of and have your applications send what you're tracking to Amazon and record regularly. If you selected Custom View in the previous screen you will see the AWS Custom Metrics category. Click Refresh to list all active custom CloudWatch metrics.

2. When you are satisfied with the profile selections click SAVE to write out the profile.
3. Click HOME to return to the main Cloud Hub panel.
4. Click START for the specific connector to begin the discovery and data collection process.

   At anytime, if you decide you do not want to monitor a particular region, simply navigate back to this point (GroundWork Administration > GroundWork Cloud Hub) and select STOP for the corresponding connector, the connectors configuration will be maintained for a subsequent START. To stop and completely delete a connection see How to delete GroundWork Hub hosts.

Appendix A: Host Group Tagging

To help manage and group your EC2 instances and EBS images, AWS allows you to assign your own metadata to each resource in the form of tags. A tag is a key and value pair of user defined strings that can be assigned to AWS resources as mapping mechanism for host groups. Tags enable you to categorize your AWS resources in different ways, for example, by purpose, owner, or environment. This is useful when you have many resources of the same type — you can quickly identify a specific resource based on the tags you've assigned to it. For example, with a tag Key name of GWTraining, and a tag Value trainingservers, all instances assigned the tag key GWTraining are placed in the host group called trainingservers.

Tagging Notes

• Host Groups are deleted when empty as is the case for all such host groups automatically created by Cloud Hub.
• A host can be in more than one group, so if you create tagging and only want to display the tagged group - you can allow access to specific Host Groups and Service Groups in the portal membership management. For example, restricting user operator to just the trainingservers host group would result in the Cloud Hub discovered host groups to not display in such applications as Status.
• Tag key and value strings are case-sensitive.
• Resources (e.g., EC2 instances, EBS volumes) can have one or more (up to 50) tags each.

Enabling Tagging

Cloud Hub can map your AWS resource tags to GroundWork Host Groups. To utilize this feature you need to enable tagging and set a Host Group tag name.

1. To enable tagging, go to GroundWork Administration > GroundWork Cloud Hub and Add or Modify an AWS connection.
2. To enable tagging, check the Enable HostGroup Tagging? checkbox.
3. In the next field, enter a Host Group Tag Name to match a tag key name configured in EC2.
   
   Figure: Enabling Host Group Tagging
   

Configuring Tagging in the Console

Along with the Cloud Hub configuration above, to tag EC2 instances or EBS volumes you will need to configure tags in the EC2 console.

1. Launch and sign in to the Amazon EC2 console at https://console.aws.amazon.com/ec2.
2. If necessary, in the upper right corner change the location of the resources to the desired region.
3. In the navigation pane, select Tags.
4. Click the Manage Tags button.
   
5. Select the instances you would like to tag (e.g., checked in blue) and assign a tag key (e.g., GWTraining) and a tag value (e.g., trainingservers), and select Add Tag.
   
   The tag key (e.g., GWTraining) is then used in the Cloud Hub connector configuration (from above), and the tag value (e.g., trainingservers) becomes the name of the host group for the associated instances. Alternatively, you can navigate to individual EC2 instances and assign tags specifically to that one instance from the Tag tab. Similarly, Elastic Block Store volumes can also be assigned tags.
   
   Figure: Connector console (e.g., AWS EC2)
   

Appendix B: Notes on AWS Metrics

Elastic Block Store (EBS) Metrics

• EBS metrics are now attached to corresponding hosts. A host can have one or more EBS metrics. Previously, EBS metrics were gathered in their own groups which was causing unused volume metrics to be reported.
• EBS metric naming has been changed to prefix service names on EC2 instance with: EBS.{volume-name}.{metricname}
• EBS metrics are only added to EC2 instances when Storage View on the connection configuration page is enabled
• EBS host groups and hosts have been removed
• When unchecking Storage View, all EBS services are deleted
• Supported EBS metrics:
  • EBS.VolumeIdleTime - Total seconds spent by in Idle time on EBS device
  • EBS.VolumeQueueLength - Total Read and Write Operations on EBS device
  • EBS.VolumeReadBytes - Total Bytes Read on EBS device
  • EBS.VolumeReadOps - Total Read Operations on EBS device
  • EBS.VolumeTotalReadTime - Total seconds spent by Read operations completed on EBS device
  • EBS.VolumeTotalWriteTime - Total seconds spent by Write operations completed on EBS device
  • EBS.VolumeWriteBytes - Total Bytes Written on an EBS device
  • EBS.VolumeWriteOps - Total Write Operations on EBS device

Relational Database Service (RDS) Metrics

• When Storage View is enabled, RDS hosts are added to a host group named AWS-RDS:storage
• No unused RDS volume metrics are reported, such as volumes that exist in the zone but are not attached to any running instance
• When unchecking Storage View, all RDS hosts and services are deleted, and the RDS host group is deleted
• RDS metrics are only associated with an RDS host
• Supported RDS metrics:
  • RDS.BinLogDiskUsage - Disk space occupied by binary logs on the master RDS node
  • RDS.DatabaseConnections - Number of database connections currently in use on RDS
  • RDS.DiskQueueDepth - Number of outstanding IOs (read/write requests) waiting to access RDS disks
  • RDS.FreeStorageSpace - Amount (bytes) of available Storage space on RDS service
  • RDS.FreeableMemory - Amount (bytes) of available RAM on RDS service
  • RDS.NetworkReceiveThroughput - Incoming (Receive) network traffic on the DB instance in bytes/second, includes both database and RDS traffic
  • RDS.NetworkTransmitThroughput - Outgoing (Transmit) network traffic on the DB instance in bytes/second, includes both database and RDS traffic
  • RDS.ReadIOPS - Average number of disk I/O read operations per second (RDS)
  • RDS.ReadLatency - Average amount of time in seconds taken per disk Read I/O operation (RDS)
  • RDS.ReadThroughput - Average number of bytes read from disk per second (RDS)
  • RDS:ReplicaLag - Amount of time a Read Replica DB Instance lags behind the source DB Instance
  • RDS.SwapUsage - Amount (bytes) of swap space used on the DB instance
  • RDS.WriteIOPS - Average number of disk I/O write operations per second (RDS)
  • RDS.WriteLatency -  Average amount of time in seconds taken per disk Write I/O operation (RDS)
  • RDS.WriteThroughput - Average number of bytes written to disk per second (RDS)

CloudWatch Custom Metrics

Amazon CloudWatch is a monitoring service for AWS cloud resources and the applications you run on AWS. Amazon CloudWatch is used to collect and track metrics for all standard AWS resources such as EC2 instances, Relational Database Service (RDS), and Elastic Block Store (EBS) storage volumes.

Additionally, CloudWatch can be extended to gather Custom Metrics generated by your applications and services and monitor these metrics for you. If your application is generating custom metrics, to have GroundWork Cloud Hub retrieve these metrics you'll need to turn on Custom View in your AWS connection configuration. To retrieving custom metrics follow the steps below.

1. Go to GroundWork Administration > GroundWork Cloud Hub.
2. Click MODIFY corresponding to the AWS connection.
3. Enable (check) the Custom View option, as shown in the image below.
4. Click SAVE to save your configuration.
5. Click NEXT to enter into the profile screen.
   
   Figure: Enabling Custom view
   
   
   
6. Towards the bottom of the screen you should see the section labeled AWS Custom Metrics. Click Refresh to retrieve a list of custom metrics from your application.
   • In our example, we show 3 metrics starting with System/Linux.DiskSpaceUtilization.
   • Note that the prefix, System/Linux, is a CloudWatch Namespace. It is best practice to use namespaces when registering your custom metrics with CloudWatch. This can help you separate your custom metrics from other metrics in the system.
   • In the Cloud Hub configuration, you can treat these metrics like any other:
     • Check Monitored to start monitoring this metric in GroundWork Monitor
     • Check Graphed to start graphing this metric in GroundWork Monitor
     • Enter your Warning and Critical thresholds
     • The Service Name field enables you to rename the metric to a different name in GroundWork Monitor
   • After any edits, click SAVE and Cloud Hub will immediately start gathering your custom metrics.
     
     Figure: AWS CloudWatch custom metrics