Statsd Metrics – Nutanix Epoch Documentation

Overview¶

This section will describe how to send custom metrics and events to AOC through StatsD. Custom metrics and events can be used as data sources within dashboards and alerts in AOC.

How it works¶ A

local StatsD daemon is built into every Epoch AOC collector. Your application’s metrics are sent via UDP to the StatsD daemon, which adds and flushes the metrics every 10 seconds (default flush period) to AOC over HTTPS. The main job of the daemon is to buffer and aggregate many data points into a single metric (by metric name and unique tags) at each download interval. The aggregation depends on the statsd metric type. With Count, the daemon would increment the counter for each new data point. For example, counting the number of times a function is called for page rendering.

If you have already instrumented your application with StatsD, the

same implementation would work with AOC by pointing your application to the StatsD port (8125) of the Epoch collector, which is also the default StatsD port used by most StatsD clients.

Types of metrics¶

Counters¶

This type

of metric

is used to maintain counts over time. The application sends the increment to the counter, and the collector maintains the count and periodically resets it. You can use AOC Analytics to see the trend of counter increments over time and also to add up the increments over a period of time.

Meters

¶ Meters

measure an amount over time. The application sends the value of the quantity, the StatsD daemon records the average of the quantity as a single data point in each download interval. For example, record the page size from the page rendering function.

Histograms¶

Histograms measure the statistical distribution of a set of values. The distribution is calculated each flush interval for the data points sent in that time period. A histogram named metric_name produces the following metrics for a single histogram:

metric_name.avg metric_name.median metric_name.max metric_name.95percentile

For example, timing calls to the page rendering function

.

Sets¶

Assemblies are used to count the number of unique elements in a set of data points in each flush interval. For example, track the number of unique users who call the page rendering feature.

Metric format¶

StatsD metrics are sent over UDP. The format is very simple and textual. You can send metrics over raw UDP or use one of the various open source StatsD libraries available for most languages and frameworks.

Tags¶

Tags can be added to a metric to provide more context. Multiple labels can be applied as key/values or only keys in each metric. Tag keys can be used in groups by and tag keys/values can be filtered in Analytics queries. Epoch uses the DogStatsD extension of the StatsD protocol that is compatible with the original StatsD protocol. This extension allows you to label the metric with a dimension that makes sense for that metric. You can split and split the data by metric label dimensions in AOC Query Builder. For example, label your metrics by deployment environment to differentiate between metrics coming from staging and production environments.

Sampling¶

Sampling is useful when the performance overhead of using StatsD on busy areas of application code is a concern. When sampling is used, it means that the metric is reported only a fraction of the time. The StatsD protocol supports sending the sample rate of a metric, the StatsD daemon takes care of the normalization of the sampled metrics.

Cable format¶

metric.name:value|type|@sample_rate|#tag1:value,tag2

• metric.name — a string without colons, bars, or @ characters

. value — an

• integer or float

.

• Type — C for counter, G for gauge, MS for timer, H for histogram, S for set

.

• Sampling frequency (optional): a float between 0 and 1, inclusive. It only works with counter, histogram, and timer metrics. The default value is 1 (that is, it shows 100% of the time).
• Tags (optional): A comma-separated list of tags. Use a colon for key/value tags, that is, env:prod. The key device is reserved.

Here are some examples of the metric format over UDP:

// Increment the function.render counter. page.render.count:1|c // Record page size from page render function page.render.size:100|g // Record time spent on page render function with a sample rate of 0.5 page.render.time:250|h|@0.5 // Track unique users calling page rendering function, tagged by production environment page.render.unique.users:20|s|#environment: production

Sending metrics

to AOC¶

Metrics can be sent to AOC using any StatsD open source client library for your framework/language. If you need support for tagging, you can use a DogStatsD client library for your framework/language. You can also send metrics in plain UDP using the format described in Cable format.

Bash

command¶

Here is an example bash command to send a metric to a local collector

: echo -n “custom_metric:60|g|#shell” >/dev/udp/localhost/8125 Using StatsD Metrics in AOC¶

In Query Builder, find the metric name that you sent with StatsD in the list of data sources

.

To filter the data source by tags, you can apply a serial filter. Find the tag by typing the tag key in the filter drop-down menu.

For example, if the metric is page.render.count:

1|c#environment:production, the data source would be page.render.count, the tag key would be environment, and the tag value would be production.

Events¶

The statsd server of the collector can also accept events. Events are unique entities over time and can be overlaid on metrics in graphs. In addition, one can do regular analysis, dashboards and alerts on events just like metrics.

Wire Format¶

_e{title.length,text.length}:title|text|d:

timestamp|p:priority|t:alert_type|#tag1,tag2

_e

• – The datagram must begin with _e

title — Event title. text

• — Event

text.

• Insert line breaks with an escape forward slash (n)
• |d:timestamp (optional): Adds a timestamp to the event. The default value is the current Unix-era timestamp.
• |k:aggregation_key (optional): Add an aggregation key to group the event with others that have the same key. There is no default.
• |p:priority (optional) — Set to ‘normal’ or ‘low’. Default value ‘normal’.
• |s:source_type_name (optional): Add a source type to the event. There is no default.
• |t:alert_type (optional) — Set to ‘error’, ‘warning’, ‘info’ or ‘success’. default ‘info’.
• |#tag1:value1,tag2,tag3:value3… (Optional): The colon of the tags are part of the tag list string and have no parsing purpose as for the other parameters. There is no default.

Bash

command¶

Here is an example of a bash command to send an event to a local collector, notice how we calculate the tile size and event body

: title=”Shell Event” text=”This was sent from Bash!” echo “_e{${#title},${#text}}:$title|$text|#shell,bash” >/dev/udp/localhost/8125

Configuration¶

You can change the UDP port of StatsD (8125 is the default) that the collector listens on by passing the port as an environment EPOCH_STATSD_PORT variable to the collector process. Make sure that this port is available on the host. If you are running the container version of the collector, the port would be available on the host, because the recommended installation of collectors requires host-mode networking. Epoch’s StatsD integration is a direct solution to your current StatsD instrumentation, so the defaults should work in most cases.