Collecting Java Application Metrics
With the release of our Orchestrated Netprobe solution, collecting useful metrics from Java applications has become much easier. No longer is it necessary to manage JMX connections between Geneos and all of your applications. Instead, the Orchestrated Netprobe uses a push model where n applications publish metrics to a central agent via the Statsd protocol. This greatly reduces the operational overhead involved when spinning up new applications.
Statsd operates in a client/server fashion, where the client (the application) sends metrics to a server that maintains the current value of each metric. Every ten seconds, the server reports each metric to the visualization layer, which in our case is the Geneos Active Console. The benefit of this architecture is that the client does as little work as possible with a minimal resource footprint.
We provide a Java library that includes a Statsd client and a JVM metrics collector which can be used to instrument your applications.
Metric Types
The following types of metrics can be collected:
- Counter: Non-negative integer value that can be incremented and decremented.
- Gauge: Positive or negative floating point value that can be arbitrarily adjusted.
- Timer: Elapsed time in milliseconds. The recorded values are reported as a histogram.
- Set: Bucket of unique string values. The set is reported as a counter whose value is the number of unique elements in the set.
- Event: User-defined event. Unlike other metrics which are reported at ten-second intervals, events are immediately reported upstream by the server.
- Attribute: Static information that doesn't change during runtime.
Identifying Metric Sources
In order to identify its source, a Statsd metric is tagged with one or more "dimensions". A dimension is simply a string that has meaning within the context of your environment, and a set of dimensions comprise the unique identifier for a metric's source. The dimensions drive the hierarchical organization of data when displayed in Geneos Active Console.
Before instrumenting an application, you must choose a dimension scheme that makes sense for your environment. In many cases the hostname and app name may be enough, while in larger environments it may be necessary to use some combination of hostname, datacenter name, city etc.
We do provide some out-of-the-box schemes that can be employed with minimal configuration. They are driven by environment variables that the Statsd client looks for automatically. Kubernetes and Pivotal Cloud Foundry are supported (more details), but we'll use the simplest scheme of just HOSTNAME and APP_NAME. The client will translate these variables into dimensions of the same name (in lowercase).
Using the Client
There are two ways to collect metrics:
- Instrument Your Application: Add the client as an application dependency, create an instance of the client and instrument your code where needed. This method is the only way to collect custom metrics.
- Java Agent: Initialize the client automatically by adding a -javaagent flag to your Java startup command. This method supports the collection of JVM metrics only. Since no code modifications are required, it's a great way to start collecting useful metrics with zero development time and minimal operational changes. Just set some environment variables, change the startup command and you're done.
Instrumenting Your Application For Custom Metrics
- Include the Statsd client library in your application:
- To add as a Maven dependency, first define the ITRS Maven repository in your pom.xml or ~/.m2/settings.xml:
<repositories>
<repository>
<id>itrsgroup-collection</id>
<url>https://itrsgroup.bintray.com/collection</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories> - Then add the dependency in the POM:
<dependencies>
<dependency>
<groupId>com.itrsgroup.collection</groupId>
<artifactId>statsd-client</artifactId>
<version>1.2.0</version>
</dependency>
</dependencies> - Or manually download statsd-client-VERSION.jar from ITRS Resources and add it to the classpath.
- To add as a Maven dependency, first define the ITRS Maven repository in your pom.xml or ~/.m2/settings.xml:
- Build an instance of the Statsd client:
For many environments the default builder options will suffice. The resulting client publishes to localhost:8125 via UDP and auto-detects the dimensions based on certain environment variables:StatsdClientBuilder builder = new StatsdClientBuilder();
However, if you need to make customizations:StatsdClient statsd = builder.build();// Use TCP instead of UDP
builder.useTcp();// Send to a different host and port
builder.server(InetAddress.getByName("192.168.1.1")).port(9000);// Define a custom dimension that will be added to all metrics
builder.defaultDimension("region", "Americas");// If opting not to use any of the built-in dimension schemes, they can be disabled
builder.disableEnvironmentalDimensions();// Define an error callback
builder.errorCallback(e -> logger.error("Statsd error", e));
A few notes about the client:// Finally create the client
StatsdClient statsd = builder.build();- It is thread safe — a single instance can be used throughout the application.
- All methods are non-blocking with very little overhead and minimal garbage creation. Delivery to the server is handled in a different thread.
- Its methods do not throw — by default, exceptions are silently caught and ignored. A callback can be provided if you wish to log these errors.
- Instrument your application:
// Increment a counter
void login() {
if (failed) {
statsd.increment("failed_logins");
}
}// Set a gauge
void cacheObject(Object obj) {
cache.put(obj);
statsd.gauge("cache_size", cache.getSizeInMB(), Unit.MEGABYTES);
}// Register periodic gauge sampling (call this only once per gauge)
void startup() {
statsd.gauge("messages_in_buffer", buffer::getSize);
}// Raise a custom event
statsd.event("MyCustomEvent", "Something has just occurred", Severity.WARN);// Count unique values in a set
void login(String username) {
if (success) {
statsd.unique("unique_logins", username);
}
}// Record a timer
void receiveMessage(Message m) {
long t1 = m.timestamp();
long t2 = System.currentTimeMillis();
statsd.timer("message_latency", t2-t1);
}// Report a static attribute
void startup() {
statsd.attribute("product_version", "1.2");
}
- Collecting JVM Metrics:
All that's required is to provide the client created in step 2:
The collector will poll for memory, GC, threading and runtime metrics every 10 seconds and report them via the Statsd client. Refer to the docs for a list of all metrics collected.// Create the collector
new JvmMetricsBuilder().client(statsd).build();
Using the Java Agent
To quickly collect JVM metrics from existing applications without writing any code, run the Statsd client and JVM metrics collector as a Java Agent. Simply download statsd-client-VERSION.jar from ITRS Resources and add the -javaagent flag to your startup command:
java -javaagent:statsd-client-VERSION.jar -jar app.jar
By default, the agent will publish to localhost:8125 via UDP. See the docs for more configuration details.
Dimension Environment Variables
Whether you instrumented your code or opted to use the Java Agent, one last important step remains before starting your application. Environment variables must be defined in order for the client to detect and tag each metric with dimensions.
If using an orchestration platform like Kubernetes or Pivotal Cloud Foundry, the variables can be defined in the application's deployment manifest.
In a non-orchestrated environment, define the variables in your startup script:
#!/bin/sh
export APP_NAME=JavaApp
export HOSTNAME=`hostname`
java -jar app.jar
Viewing the Metrics
After starting your application, metrics will start to appear in Active Console after about 10 seconds. Here are some screenshots showing data from the examples above:
Custom Metrics:
Heap Memory Usage:
GC Stats:
Thread Stats:
