* [Brian Troutwine](https://github.com/Feuerlabs/exometer/pulls/blt?q=is%3Aclosed)
* [Tino Breddin](https://github.com/Feuerlabs/exometer/pulls/tolbrino?q=is%3Aclosed)
* [Artem Teslenko](https://github.com/Feuerlabs/exometer/pulls/ates?q=is%3Aclosed)
* [Sébastien Merle](https://github.com/Feuerlabs/exometer/pulls/sylane?q=is%3Aclosed)
* [Diana Corbacho](https://github.com/Feuerlabs/exometer/pulls/dcorbacho?q=is%3Aclosed)
* [Roland Karlsson](https://github.com/Feuerlabs/exometer/pulls/roland-karlsson-erlang-solutions-com?q=is%3Aclosed)
* [Aaron France](https://github.com/Feuerlabs/exometer/pulls/aeronotix?q=is%3Aclosed)
* [Maas-Maarten Zeeman](https://github.com/Feuerlabs/exometer/pulls/mmzeeman?q=is%3Aclosed)
* [Vladimir G. Sekissov](https://github.com/Feuerlabs/exometer/pulls/eryx67?q=is%3Aclosed)
* [Mark Steele](https://github.com/Feuerlabs/exometer/pulls/marksteele?q=is%3Aclosed)
================================================
FILE: LICENSE
================================================
Mozilla Public License, version 2.0
1. Definitions
1.1. “Contributor”
means each individual or legal entity that creates, contributes to the
creation of, or owns Covered Software.
1.2. “Contributor Version”
means the combination of the Contributions of others (if any) used by a
Contributor and that particular Contributor’s Contribution.
1.3. “Contribution”
means Covered Software of a particular Contributor.
1.4. “Covered Software”
means Source Code Form to which the initial Contributor has attached the
notice in Exhibit A, the Executable Form of such Source Code Form, and
Modifications of such Source Code Form, in each case including portions
thereof.
1.5. “Incompatible With Secondary Licenses”
means
a. that the initial Contributor has attached the notice described in
Exhibit B to the Covered Software; or
b. that the Covered Software was made available under the terms of version
1.1 or earlier of the License, but not also under the terms of a
Secondary License.
1.6. “Executable Form”
means any form of the work other than Source Code Form.
1.7. “Larger Work”
means a work that combines Covered Software with other material, in a separate
file or files, that is not Covered Software.
1.8. “License”
means this document.
1.9. “Licensable”
means having the right to grant, to the maximum extent possible, whether at the
time of the initial grant or subsequently, any and all of the rights conveyed by
this License.
1.10. “Modifications”
means any of the following:
a. any file in Source Code Form that results from an addition to, deletion
from, or modification of the contents of Covered Software; or
b. any new file in Source Code Form that contains any Covered Software.
1.11. “Patent Claims” of a Contributor
means any patent claim(s), including without limitation, method, process,
and apparatus claims, in any patent Licensable by such Contributor that
would be infringed, but for the grant of the License, by the making,
using, selling, offering for sale, having made, import, or transfer of
either its Contributions or its Contributor Version.
1.12. “Secondary License”
means either the GNU General Public License, Version 2.0, the GNU Lesser
General Public License, Version 2.1, the GNU Affero General Public
License, Version 3.0, or any later versions of those licenses.
1.13. “Source Code Form”
means the form of the work preferred for making modifications.
1.14. “You” (or “Your”)
means an individual or a legal entity exercising rights under this
License. For legal entities, “You” includes any entity that controls, is
controlled by, or is under common control with You. For purposes of this
definition, “control” means (a) the power, direct or indirect, to cause
the direction or management of such entity, whether by contract or
otherwise, or (b) ownership of more than fifty percent (50%) of the
outstanding shares or beneficial ownership of such entity.
2. License Grants and Conditions
2.1. Grants
Each Contributor hereby grants You a world-wide, royalty-free,
non-exclusive license:
a. under intellectual property rights (other than patent or trademark)
Licensable by such Contributor to use, reproduce, make available,
modify, display, perform, distribute, and otherwise exploit its
Contributions, either on an unmodified basis, with Modifications, or as
part of a Larger Work; and
b. under Patent Claims of such Contributor to make, use, sell, offer for
sale, have made, import, and otherwise transfer either its Contributions
or its Contributor Version.
2.2. Effective Date
The licenses granted in Section 2.1 with respect to any Contribution become
effective for each Contribution on the date the Contributor first distributes
such Contribution.
2.3. Limitations on Grant Scope
The licenses granted in this Section 2 are the only rights granted under this
License. No additional rights or licenses will be implied from the distribution
or licensing of Covered Software under this License. Notwithstanding Section
2.1(b) above, no patent license is granted by a Contributor:
a. for any code that a Contributor has removed from Covered Software; or
b. for infringements caused by: (i) Your and any other third party’s
modifications of Covered Software, or (ii) the combination of its
Contributions with other software (except as part of its Contributor
Version); or
c. under Patent Claims infringed by Covered Software in the absence of its
Contributions.
This License does not grant any rights in the trademarks, service marks, or
logos of any Contributor (except as may be necessary to comply with the
notice requirements in Section 3.4).
2.4. Subsequent Licenses
No Contributor makes additional grants as a result of Your choice to
distribute the Covered Software under a subsequent version of this License
(see Section 10.2) or under the terms of a Secondary License (if permitted
under the terms of Section 3.3).
2.5. Representation
Each Contributor represents that the Contributor believes its Contributions
are its original creation(s) or it has sufficient rights to grant the
rights to its Contributions conveyed by this License.
2.6. Fair Use
This License is not intended to limit any rights You have under applicable
copyright doctrines of fair use, fair dealing, or other equivalents.
2.7. Conditions
Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in
Section 2.1.
3. Responsibilities
3.1. Distribution of Source Form
All distribution of Covered Software in Source Code Form, including any
Modifications that You create or to which You contribute, must be under the
terms of this License. You must inform recipients that the Source Code Form
of the Covered Software is governed by the terms of this License, and how
they can obtain a copy of this License. You may not attempt to alter or
restrict the recipients’ rights in the Source Code Form.
3.2. Distribution of Executable Form
If You distribute Covered Software in Executable Form then:
a. such Covered Software must also be made available in Source Code Form,
as described in Section 3.1, and You must inform recipients of the
Executable Form how they can obtain a copy of such Source Code Form by
reasonable means in a timely manner, at a charge no more than the cost
of distribution to the recipient; and
b. You may distribute such Executable Form under the terms of this License,
or sublicense it under different terms, provided that the license for
the Executable Form does not attempt to limit or alter the recipients’
rights in the Source Code Form under this License.
3.3. Distribution of a Larger Work
You may create and distribute a Larger Work under terms of Your choice,
provided that You also comply with the requirements of this License for the
Covered Software. If the Larger Work is a combination of Covered Software
with a work governed by one or more Secondary Licenses, and the Covered
Software is not Incompatible With Secondary Licenses, this License permits
You to additionally distribute such Covered Software under the terms of
such Secondary License(s), so that the recipient of the Larger Work may, at
their option, further distribute the Covered Software under the terms of
either this License or such Secondary License(s).
3.4. Notices
You may not remove or alter the substance of any license notices (including
copyright notices, patent notices, disclaimers of warranty, or limitations
of liability) contained within the Source Code Form of the Covered
Software, except that You may alter any license notices to the extent
required to remedy known factual inaccuracies.
3.5. Application of Additional Terms
You may choose to offer, and to charge a fee for, warranty, support,
indemnity or liability obligations to one or more recipients of Covered
Software. However, You may do so only on Your own behalf, and not on behalf
of any Contributor. You must make it absolutely clear that any such
warranty, support, indemnity, or liability obligation is offered by You
alone, and You hereby agree to indemnify every Contributor for any
liability incurred by such Contributor as a result of warranty, support,
indemnity or liability terms You offer. You may include additional
disclaimers of warranty and limitations of liability specific to any
jurisdiction.
4. Inability to Comply Due to Statute or Regulation
If it is impossible for You to comply with any of the terms of this License
with respect to some or all of the Covered Software due to statute, judicial
order, or regulation then You must: (a) comply with the terms of this License
to the maximum extent possible; and (b) describe the limitations and the code
they affect. Such description must be placed in a text file included with all
distributions of the Covered Software under this License. Except to the
extent prohibited by statute or regulation, such description must be
sufficiently detailed for a recipient of ordinary skill to be able to
understand it.
5. Termination
5.1. The rights granted under this License will terminate automatically if You
fail to comply with any of its terms. However, if You become compliant,
then the rights granted under this License from a particular Contributor
are reinstated (a) provisionally, unless and until such Contributor
explicitly and finally terminates Your grants, and (b) on an ongoing basis,
if such Contributor fails to notify You of the non-compliance by some
reasonable means prior to 60 days after You have come back into compliance.
Moreover, Your grants from a particular Contributor are reinstated on an
ongoing basis if such Contributor notifies You of the non-compliance by
some reasonable means, this is the first time You have received notice of
non-compliance with this License from such Contributor, and You become
compliant prior to 30 days after Your receipt of the notice.
5.2. If You initiate litigation against any entity by asserting a patent
infringement claim (excluding declaratory judgment actions, counter-claims,
and cross-claims) alleging that a Contributor Version directly or
indirectly infringes any patent, then the rights granted to You by any and
all Contributors for the Covered Software under Section 2.1 of this License
shall terminate.
5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user
license agreements (excluding distributors and resellers) which have been
validly granted by You or Your distributors under this License prior to
termination shall survive termination.
6. Disclaimer of Warranty
Covered Software is provided under this License on an “as is” basis, without
warranty of any kind, either expressed, implied, or statutory, including,
without limitation, warranties that the Covered Software is free of defects,
merchantable, fit for a particular purpose or non-infringing. The entire
risk as to the quality and performance of the Covered Software is with You.
Should any Covered Software prove defective in any respect, You (not any
Contributor) assume the cost of any necessary servicing, repair, or
correction. This disclaimer of warranty constitutes an essential part of this
License. No use of any Covered Software is authorized under this License
except under this disclaimer.
7. Limitation of Liability
Under no circumstances and under no legal theory, whether tort (including
negligence), contract, or otherwise, shall any Contributor, or anyone who
distributes Covered Software as permitted above, be liable to You for any
direct, indirect, special, incidental, or consequential damages of any
character including, without limitation, damages for lost profits, loss of
goodwill, work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses, even if such party shall have been
informed of the possibility of such damages. This limitation of liability
shall not apply to liability for death or personal injury resulting from such
party’s negligence to the extent applicable law prohibits such limitation.
Some jurisdictions do not allow the exclusion or limitation of incidental or
consequential damages, so this exclusion and limitation may not apply to You.
8. Litigation
Any litigation relating to this License may be brought only in the courts of
a jurisdiction where the defendant maintains its principal place of business
and such litigation shall be governed by laws of that jurisdiction, without
reference to its conflict-of-law provisions. Nothing in this Section shall
prevent a party’s ability to bring cross-claims or counter-claims.
9. Miscellaneous
This License represents the complete agreement concerning the subject matter
hereof. If any provision of this License is held to be unenforceable, such
provision shall be reformed only to the extent necessary to make it
enforceable. Any law or regulation which provides that the language of a
contract shall be construed against the drafter shall not be used to construe
this License against a Contributor.
10. Versions of the License
10.1. New Versions
Mozilla Foundation is the license steward. Except as provided in Section
10.3, no one other than the license steward has the right to modify or
publish new versions of this License. Each version will be given a
distinguishing version number.
10.2. Effect of New Versions
You may distribute the Covered Software under the terms of the version of
the License under which You originally received the Covered Software, or
under the terms of any subsequent version published by the license
steward.
10.3. Modified Versions
If you create software not governed by this License, and you want to
create a new license for such software, you may create and use a modified
version of this License if you rename the license and remove any
references to the name of the license steward (except to note that such
modified license differs from this License).
10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses
If You choose to distribute Source Code Form that is Incompatible With
Secondary Licenses under the terms of this version of the License, the
notice described in Exhibit B of this License must be attached.
Exhibit A - Source Code Form License Notice
This Source Code Form is subject to the
terms of the Mozilla Public License, v.
2.0. If a copy of the MPL was not
distributed with this file, You can
obtain one at
http://mozilla.org/MPL/2.0/.
If it is not possible or desirable to put the notice in a particular file, then
You may include the notice in a location (such as a LICENSE file in a relevant
directory) where a recipient would be likely to look for such a notice.
You may add additional accurate notices of copyright ownership.
Exhibit B - “Incompatible With Secondary Licenses” Notice
This Source Code Form is “Incompatible
With Secondary Licenses”, as defined by
the Mozilla Public License, v. 2.0.
================================================
FILE: Makefile
================================================
.PHONY: all clean clean_plt deps compile test doc dialyzer xref devnode_snmp_agent devnode_snmp_manager compile_examples ci
EXOMETER_PLT=exometer.plt
DIALYZER_OPTS = # -Wunderspecs
DIALYZER_APPS = erts kernel stdlib compiler syntax_tools snmp ssl ssh \
crypto public_key test_server webtool xmerl common_test \
lager goldrush afunix netlink folsom mnesia parse_trans \
setup exometer_core
all: deps compile
ci: compile xref dialyzer test
deps:
rebar get-deps
compile:
rebar compile
clean: clean_plt
rebar clean
clean-all: clean
rm -rf deps
test: compile_examples
ERL_LIBS=./examples rebar ct skip_deps=true
xref:
ERL_LIBS=./deps rebar xref skip_deps=true
edown_deps:
rebar get-deps compile edown=true
doc: edown_deps
rebar doc edown=true skip_deps=true
$(EXOMETER_PLT):
rebar get-deps compile
ERL_LIBS=deps dialyzer --build_plt --output_plt $(EXOMETER_PLT) \
--apps $(DIALYZER_APPS)
clean_plt:
rm -f $(EXOMETER_PLT)
dialyzer: deps compile $(EXOMETER_PLT)
dialyzer -r ebin --plt $(EXOMETER_PLT) $(DIALYZER_OPTS)
compile_examples:
erlc +'{parse_transform, lager_transform}' -pz deps/lager/ebin -I src -o examples/snmp_manager/ examples/snmp_manager/*.erl
devnode_snmp_agent:
erl -sname agent -pa deps/*/ebin ebin -config examples/snmp_agent/sys.config -boot start_sasl -s lager -s crypto -s exometer
devnode_snmp_manager: compile_examples
erl -sname manager -pz examples/snmp_manager -pz deps/*/ebin ebin -config examples/snmp_manager/sys.config \
-boot start_sasl -s lager -s crypto -s snmp
================================================
FILE: README.md
================================================
# Exometer - Erlang instrumentation package #
Copyright (c) 2014 Basho Technologies, Inc. All Rights Reserved.
__Version:__ Feb 1 2015 23:02:37
__Authors:__ Ulf Wiger ([`ulf.wiger@feuerlabs.com`](mailto:ulf.wiger@feuerlabs.com)), Magnus Feuer ([`magnus.feuer@feuerlabs.com`](mailto:magnus.feuer@feuerlabs.com)).
[](https://travis-ci.org/Feuerlabs/exometer)
__NOTE: Exometer has been split into [exometer_core](https://github.com/Feuerlabs/exometer_core), and exometer (as well as separate reporter applications). The latest monolithic version of Exometer is 1.1.__
The Exometer package allows for easy and efficient instrumentation of
Erlang code, allowing crucial data on system performance to be
exported to a wide variety of monitoring systems.
Exometer comes with a set of pre-defined monitor components, and can
be expanded with custom components to handle new types of Metrics, as
well as integration with additional external systems such as
databases, load balancers, etc.
This document gives a high level overview of the Exometer system. For
details, please see the documentation for individual modules, starting
with `exometer`.
Note the section on [Dependency Management](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Dependency_Management) for how to deal with
optional packages, both users and developers.
### Table of Content ###
1. [Concept and definitions](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Concept_and_definitions)
1. [Metric](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Metric)
2. [Data Point](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Data_Point)
3. [Metric Type](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Metric_Type)
4. [Entry Callback](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Entry_Callback)
5. [Probe](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Probe)
6. [Caching](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Caching)
7. [Subscriptions and Reporters](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Subscriptions_and_Reporters)
2. [Built-in entries and probes](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Built-in_entries_and_probes)
1. [counter (exometer native)](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#counter_(exometer_native))
2. [fast_counter (exometer native)](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#fast_counter_(exometer_native))
3. [gauge (exometer native)](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#gauge_(exometer_native))
4. [exometer_histogram (probe)](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#exometer_histogram_(probe))
5. [exometer_uniform (probe)](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#exometer_uniform_(probe))
6. [exometer_spiral (probe)](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#exometer_spiral_(probe))
7. [exometer_folsom [entry]](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#exometer_folsom_[entry])
8. [exometer_function [entry]](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#exometer_function_[entry])
3. [Built in Reporters](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Built_in_Reporters)
1. [exometer_report_graphite](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#exometer_report_graphite)
2. [exometer_report_opentsdb](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#exometer_report_opentsdb)
3. [exometer_report_amqp](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#exometer_report_amqp)
4. [exometer_report_snmp](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#exometer_report_snmp)
4. [Instrumenting Erlang code](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Instrumenting_Erlang_code)
1. [Exometer Start](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Exometer_Start)
2. [Creating metrics](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Creating_metrics)
3. [Deleting metrics](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Deleting_metrics)
4. [Setting metric values](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Setting_metric_values)
5. [Retrieving metric values](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Retrieving_metric_values)
6. [Setting up subscriptions](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Setting_up_subscriptions)
7. [Set metric options](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Set_metric_options)
5. [Configuring Exometer](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_Exometer)
1. [Configuring type - entry maps](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_type_-_entry_maps)
2. [Configuring statically defined entries](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_statically_defined_entries)
3. [Configuring static subscriptions](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_static_subscriptions)
4. [Configuring reporter plugins](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_reporter_plugins)
5. [Configuring opentsdb reporter](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_opentsdb_reporter)
6. [Configuring amqp reporter](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_amqp_reporter)
7. [Configuring graphite reporter](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_graphite_reporter)
8. [Configuring snmp reporter](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_snmp_reporter)
6. [Creating custom exometer entries](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Creating_custom_exometer_entries)
7. [Creating custom probes](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Creating_custom_probes)
8. [Creating custom reporter plugins](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Creating_custom_reporter_plugins)
9. [Dependency management](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Dependency_management)
### Concepts and Definitions ###
Exometer introduces a number of concepts and definitions used
throughout the documentation and the code.
#### Metric ####
A metric is a specific measurement sampled inside an Erlang system and
then reported to the Exometer system. An example metric would be
"transactions_per_second", or "memory_usage".
Metrics are identified by a list of terms, such as given below:
`[ xml_front_end, parser, file_size ]`
A metric is created through a call by the code to be instrumented to
`exometer:new()`. Once created, the metric can be updated through
`exometer:update()`, or on its own initiative through the
`exometer_probe:sample` behavior implementation.
#### Data Point ####
Each metric can consist of multiple data points, where each point has
a specific value.
A typical example of data points would be a
`transactions_per_second` (tps) metric, usually stored as a
histogram covering the last couple of minutes of tps samples. Such a
histogram would host multiple values, such as `min`, `max`,
`median`, `mean`, `50_percentile`, `75_percentile`,
etc.
It is up to the type of the metric, and the data probe backing that
type (see below), to specify which data points are available under the
given metric.
#### Metric Type ####
The type of a metric, specified when the metric is created through
`exometer:new()`, determines which `exometer_entry`
callback to use.
The link between the type and the entry to use is configured
through the `exometer_admin` module, and its associated exometer
defaults configuration data.
The metric type, in other words, is mainly used to map a metric to a
configurable `exometer_entry` callback, but it can also be referenced
in queries using `exometer:select/1`. An entry callback can also support
multiple types (the type is provided as an argument in the callback functions).
Exometer provides default mappings for a number of metric types. It is
possible to select different callbacks for each metric instance, as well
as modify metrics using callback-specific options. Please see
[Configuring type - entry maps](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_type_-_entry_maps) for details on how to do this.
#### Entry Callback ####
An exometer entry callback will receive values reported to a metric through the
`exometer:update()` call and compile it into one or more data points.
The entry callback can either be a counter (implemented natively
in `exometer`), or a more complex statistical analysis such
as a uniform distribution or a regular histogram.
The various outputs from these entries are reported as data points
under the given metric.
An entry can also interface external analytics packages.
`exometer_folsom`, for example, integrates with the
`folsom_metrics` package found at [`https://github.com/boundary/folsom`](https://github.com/boundary/folsom).
#### Probe ####
Probes are a further specialization of exometer entries that run in
their own Erlang processes and have their own state (like a
gen_server). A probe is implemented through the `exometer_probe`
behavior.
A probe can be used if independent monitoring is needed of,
for example, `/proc` trees, network interfaces, and other subsystems
that need periodic sampling. In these cases, the
`exometer_probe:probe_sample()` call is invoked regularly by exometer,
in the probe's own process, in order to extract data from
the given subsystem and add it to the metric's data points.
#### Caching ####
Metric and data point values are read with the `exometer:get_value()`
function. In the case of counters, this operation is very fast. With probes,
the call results in a synchronous dialog with the probe process, and the
cost of serving the request depends on the probe implementation and the
nature of the metric being served.
If the cost of reading the value is so high that calling the function often
would result in prohibitive load, it is possible to cache the value. This is
done either explicitly from the probe itself (by calling
`exometer_cache:write()`), or by specifying the option `{cache, Lifetime}`
for the entry. If an entry has a non-zero cache lifetime specified, the
`get_value()` call will try fetching the cached value before calling the
actual entry and automatically caching the result.
Note that if `{cache, Lifetime}` is not specified, `exometer:get_value()`
will neither read nor write to the cache. It is possible for the probe
to periodically cache a value regardless of how the cache lifetime is set,
and the probe may also explicitly read from the cache if it isn't done
automatically.
#### Subscriptions and Reporters ####
The subscription concept, managed by `exometer_report` allows metrics
and their data points to be sampled at given intervals and delivered
to one or more recipients, which can be either an arbitrary process
or a Reporter plugin.
Each subscription ties a specific metric-datapoint pair to a reporter
and an interval (given in milliseconds). The reporter system will, at
the given interval, send the current value of the data point to the
subscribing reporter. The subscription, with all its parameters,
is setup through a call to `exometer_report:subscribe()`.
In the case of processes, subscribed-to values will be delivered as a
message. Modules, which implement the `exometer_report` callback
behavior, will receive the plugins as a callbacks within the
`exometer_report` process.
Subscriptions can either be setup at runtime, through
`exometer_report:subscribe()` calls, or statically through the
`exometer_report` configuration data.
### Built-in entries and probes ###
There are a number of built-in entries and probes shipped
with the Exometer package, as described below:
#### counter (exometer native) ####
The counter is implemented directly in `exometer` to provide simple
counters. A call to `exometer:update()` will add the provided value
to the counter.
The counter can be reset to zero through `exometer:reset()`.
The available data points under a metric using the counter entry
are `value` and `ms_since_reset`.
#### fast_counter (exometer native) ####
A fast counter implements the counter functionality, through the
`trace_info` system, yielding a speed increase of about 3.5 in
comparison to the regular counter.
The tradeoff is that running tracing and/or debugging may interfere
with the counter functionality.
A call to `exometer:update()` will add the provided value to the
counter.
The counter can be reset to zero through `exometer:reset()`.
The available data points under a metric using the fast_counter
entry are `value` and `ms_since_reset`.
#### gauge (exometer native) ####
The gauge is implemented directly in `exometer` to provide simple
gauges. A call to `exometer:update()` will set the gauge's value
to the provided value. That is, the value of the gauge entry is
always the most recently provided value.
The gauge can be reset to zero through `exometer:reset()`.
The available data points under a metric using the gauge entry
are `value` and `ms_since_reset`.
#### histogram (probe) ####
The histogram probe stores a given number of updates, provided through
`exometer:update()`, in a histogram. The histogram maintains a log
derived from all values received during a configurable time span and
provides min, max, median, mean, and percentile analysis data points
for the stored data.
Exometer supports a number of different histogram implementations, each
with different performance and accuracy trade-offs.
In order to save memory, the histogram is divided into equal-sized
time slots, where each slot spans a settable interval. All values
received during a time slot will be averaged into a single value to be
stored in the histogram once the time slot expires. The averaging
function (which can be replaced by the caller), allows for
high-frequency update metrics to have their resolution traded against
resource consumption.
#### exometer_uniform (probe) ####
The uniform probe provides a uniform sample over a pool of values
provided through `exometer:update()`. When the pool reaches its configurable
max size, existing values will be replaced at random to make space for
new values. Much like `exometer_histogram`, the uniform probe
provides min, max, median, mean, and percentile analysis data points
for the stored data.
#### exometer_spiral (probe) ####
The spiral probe maintains the total sum of all values stored in its
histogram. The histogram has a configurable time span, all values
provided to the probe, through `exometer:update()`, within that time
span will be summed up and reported. If, for example, the histogram
covers 60 seconds, the spiral probe will report the sum of all
values reported during the last minute.
The grand total of all values received during the lifetime of the
probe is also available.
#### exometer_folsom [entry] ####
`exometer_folsom` is an entry behavior which implements most metric types
supported by the [folsom](https://github.com/boundary/folsom)
metrics package: Specifically, the metric types `counter`, `spiral`,
`histogram`, `meter`, `meter_reader`, `gauge`, `duration` and `history`.
The folsom entry integrates with the folsom metrics package provided
by the boundary repo at github. Updated values sent to the folsom entry
can be forwarded to folsom's counter, histogram, duration, meter,
and spiral.
Folsom integration is provided as a backup. New code using Exometer
should use the native probes that duplicate folsom.
#### exometer_function [entry] ####
The function entry allows for an existing erlang function to be wrapped
as an exometer entry. The [`exometer_function`](https://github.com/Feuerlabs/exometer_core/blob/master/doc/exometer_function.md) module supports a number
of options for passing arguments and matching out data points from the
result.
The function entry provides an easy way of integrating an external
system without having to write a complete entry.
### Built in Reporters ###
Exometer ships with some built-in reporters which can be used to forward updated
metrics and their data points to external systems. They can also
serve as templates for custom-developed reporters.
#### exometer_report_graphite ####
The graphite reporter uses the TCP/IP protocol to forward
subscribed-to metrics and data points to a graphite server, such as
the one provided by [`http://hostedgraphite.com`](http://hostedgraphite.com). When the graphite
reporter receives a metric-datapoint value (subscribed to through
`exometer_report:subscriber()`), the reporter will immediately
forward the key-value pair to the graphite server.
#### exometer_report_opentsdb ####
The OpenTSDB reporter sends metrics to an OpenTSDB server using
the telnet API. All subscribed-to metric-datapoint values received
by the reporter are immediately forwarded to OpenTSDB.
If the OpenTSDB connection is lost, the reporter will attempt to reconnect to it
at a configurable interval.
The data sent to OpenTSDB will be formatted as follows:
```
put metric timestamp value host=host type=datapoint
```
Where the value for the host tag will be the configured host in the reporter
configuration (defaults to the value returned by `netadm:localhost`), and
datapoint tags as specified by the subscriber.
Please see [Configuring opentsdb reporter](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_opentsdb_reporter) for details on the
application environment parameters listed above.
#### exometer_report_amqp ####
The AMQP reporter sends metrics to an AMQP broker as a json-encoded payload. All
subscribed-to metric-datapoint values received by the reporter are forwarded to AMQP.
If the AMQP connection is lost, the reporter will attempt to reconnect to it
at a configurable interval.
The data sent to AMQP will be formatted as follows:
```
{
"type":"exometer_metric",
"body":
{"name":"messages_per_second",
"value":0,"timestamp":1414006826,
"host":"testhost",
"instance":"max"}
}
```
Where the value for the host tag will be the configured host in the reporter
configuration (defaults to the value returned by `netadm:localhost`), the
instance tag represents the datapoint for the metric.
Please see [Configuring amqp reporter](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_amqp_reporter) for details on the
application environment parameters listed above.
#### exometer_report_snmp ####
The SNMP reporter enables the export of metrics and their datapoints to SNMP managers.
The export needs to be enabled for each metric through their options.
Moreover, SNMP notifications can be created using the options to send periodic reports
on datapoints to SNMP managers. All SNMP protocol handling is done by the snmp application
shipped with Erlang/OTP. Thus, the snmp application needs to be started and
the local SNMP master agent needs to be configured correctly for SNMP export to work
properly.
To configure SNMP export for a single metric use these options:
+ `{snmp, disabled}` (default) Disables SNMP export for the metric. Same as not specifying the option at all.
+ `{snmp, []}` Enables SNMP export for the metric. No subscriptions are setup.
+ `{snmp, [{Datapint, Interval}]}` Enables SNMP export for the metric. Subscriptions are setup for the given Datapoint/Interval pairs. Each subscription report will be forwarded to SNMP mangers as notifications.
+ `{snmp, [{Datapint, Interval, Extra}]}` Same as above, but using an addition extra identification for the subscriptions. Allow the creation ofmultiple subscriptions for a single datapoint.
Please see [Configuring snmp reporter](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_snmp_reporter) for details on how to configure the
SNMP reporter.
### Instrumenting Erlang code ###
The code using Exometer needs to be instrumented in order to setup and
use metrics reporting.
#### Exometer Start ####
The system using Exometer must start the `exometer` application prior to using it:
```erlang
application:start(lager),
application:start(exometer).
```
Note that dependent applications need to be started first. On newer OTP versions
(R16B or later), you can use `application:ensure_all_started(exometer)`.
For testing, you can also use [`exometer:start/0`](https://github.com/Feuerlabs/exometer_core/blob/master/doc/exometer.md#start-0).
If you make use of e.g. folsom metrics, you also need to start `folsom`. Exometer
will not do that automatically, nor does it contain an application dependency for it.
See [Configuring Exometer](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_Exometer) for details on configuration data
format.
#### Creating metrics ####
A metric, can be created through a call to
```erlang
exometer:new(Name, Type)
```
`Name` is a list of atoms, uniquely identifying the metric created.
The type of the metric, specified by `Type` will be mapped
to an exometer entry through the table maintained by
`exometer_admin` Please see the [Configuring type - entry
maps](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_type_-_entry_maps) for details.
The resolved entry to use will determine the data points available
under the given metric.
#### Deleting metrics ####
A metric previously created with `exometer:new()` can be deleted by
`exometer:delete()`.
All subscriptions to the deleted metrics will be cancelled.
#### Setting metric values ####
A created metric can have its value updated through the
`exometer:update()` function:
```erlang
exometer:update(Name, Value)
```
The `Name` parameter is the same atom list provided to a previous
`exometer:new()` call. The `Value` is an arbitrarty element that is
forwarded to the `exometer:update()` function of the entry/probe that the
metric is mapped to.
The receiving entry/probe will process the provided value and modify
its data points accordingly.
#### Retrieving metric values ####
Exometer-using code can at any time retrieve the data point values
associated with a previously created metric. In order to find out which
data points are available for a metric, the following call can be used:
```erlang
exometer:info(Name, datapoints)
```
The `Name` parameter is the same atom list provided to a previous
`exometer:new()` call. The call will return a list of data point
atoms that can then be provided to `exometer:get_value()` to
retrieve their actual value:
```erlang
exometer:get_value(Name, DataPoint)
```
The `Name` paramer identifies the metric, and `DataPoints`
identifies the data points (returned from the previous `info()` call)
to retrieve the value for.
If no DataPoints are provided, the values of a default list of data points,
determined by the backing entry / probe, will be returned.
#### Setting up subscriptions ####
A subscription can either be statically configured, or dynamically
setup from within the code using Exometer. For details on statically
configured subscriptions, please see [Configuring static subscriptions](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_static_subscriptions).
A dynamic subscription can be setup with the following call:
```erlang
exometer_report:subscribe(Recipient, Metric, DataPoint, Inteval)
```
`Recipient` is the name of a reporter.
#### Set metric options ####
Each created metric can have options setup for it through the following call:
```erlang
exometer:setopts(Name, Options)
```
The `Name` paramer identifies the metric to set the options for, and
Options is a proplist (`[{ Key, Value },...]`) with the options to be
set.
Exometer looks up the the backing entry that hosts the metric with the given Name, and will
invoke the entry\'s `setopts/4` function to set the actual options. Please see the
`setopts/4` function for the various entries for details.
### Configuring Exometer ###
Exometer defaults can be changed either through OTP application environment
variables or through the use of Basho's `cuttlefish`
([`https://github.com/basho/cuttlefish`](https://github.com/basho/cuttlefish)).
#### Configuring type - entry maps ####
The dynamic method of configuring defaults for `exometer` entries is:
```erlang
exometer_admin:set_default(NamePattern, Type, Default)
```
Where `NamePattern` is a list of terms describing what is essentially
a name prefix with optional wildcards (`'_'`). A pattern that
matches any legal name is `['_']`.
`Type` is an atom defining a type of metric. The types already known to
`exometer`, `counter`, `fast_counter`, `ticker`, `uniform`, `histogram`,
`spiral`, `netlink`, and `probe` may be redefined, but other types can be
described as well.
`Default` is either an `#exometer_entry{}` record (unlikely), or a list of
`{Key, Value}` options, where the keys correspond to `#exometer_entry` record
attribute names. The following attributes make sense to preset:
```erlang
{module, atom()} % the callback module
{status, enabled | disabled} % operational status of the entry
{cache, non_neg_integer()} % cache lifetime (ms)
{options, [{atom(), any()}]} % entry-specific options
```
Below is an example, from `exometer/priv/app.config`:
```erlang
{exometer, [
{defaults, [
{['_'], function , [{module, exometer_function}]},
{['_'], counter , [{module, exometer}]},
{['_'], histogram, [{module, exometer_histogram}]},
{['_'], spiral , [{module, exometer_spiral}]},
{['_'], duration , [{module, exometer_folsom}]},
{['_'], meter , [{module, exometer_folsom}]},
{['_'], gauge , [{module, exometer_folsom}]}
]}
]}
```
In systems that use CuttleFish, the file `exometer/priv/exometer.schema`
contains a schema for default settings. The setup corresponding to the above
defaults would be as follows:
```ini
exometer.template.function.module = exometer_function
exometer.template.counter.module = exometer
exometer.template.histogram.module = exometer_histogram
exometer.template.spiral.module = exometer_spiral
exometer.template.duration.module = exometer_folsom
exometer.template.meter.module = exometer_folsom
exometer.template.gauge.module = exometer_folsom
```
#### Configuring statically defined entries ####
Using the `exometer` environment variable `predefined`, entries can be added
at application startup. The variable should have one of the following values:
* `{script, File}` - `File` will be processed using `file:script/2`. The return
value (the result of the last expression in the script) should be a list of`{Name, Type, Options}` tuples.
* `{apply, M, F, A}` - The result of `apply(M, F, A)` should be `{ok, L}` where`L` is a list of `{Name, Type, Options}` tuples.
* `L`, where L is a list of `{Name, Type, Options}` tuples or extended
instructions (see below).
The list of instructions may include:
* `{delete, Name}` - deletes `Name` from the exometer registry.
* `{select_delete, Pattern}` - applies a select pattern and
deletes all matching entries.
* `{re_register, {Name, Type, Options}}` - redefines an entry if present,
otherwise creates it.
Exometer will also scan all loaded applications for the environment
variables `exometer_defaults` and `exometer_predefined`, and process
as above. If an application is loaded and started after exometer has started,
it may call the function `exometer:register_application()` or
`exometer:register_application(App)`. This function will do nothing if
exometer isn't already running, and otherwise process the `exometer_defaults`
and `exometer_predefined` variables as above. The function can also be
called during upgrade, as it will re-apply the settings each time.
#### Configuring static subscriptions ####
Static subscriptions, which are automatically setup at exometer
startup without having to invoke `exometer_report:subscribe()`, are
configured through the report sub section under exometer.
Below is an example, from `exometer/priv/app.config`:
```erlang
{exometer, [
{report, [
{subscribers, [
{exometer_report_collectd, [db, cache, hits], mean, 2000, true},
{exometer_report_collectd, [db, cache, hits], max, 5000, false}
]}
]}
]}
```
The `report` section configures static subscriptions and reporter
plugins. See [Configuring reporter plugins](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_reporter_plugins) for details on
how to configure individual plugins.
The `subscribers` sub-section contains all static subscriptions to be
setup att exometer applications start. Each tuple in the prop list
should be of one of the following formats:
* `{Reporter, Metric, DataPoint, Interval}`
* `{Reporter, Metric, DataPoint, Interval, RetryFailedMetrics}`
* `{Reporter, Metric, DataPoint, Interval, RetryFailedMetrics, Extra}`
* `{apply, {M, F, A}}`
* `{select, {MatchPattern, DataPoint, Interval [, Retry [, Extra] ]}}`
In the case of `{apply, M, F, A}`, the result of `apply(M, F, A)` must
be a list of `subscribers` tuples.
In the case of `{select, Expr}`, a list of metrics is fetched using
`exometer:select(MatchPattern)`, where the result must be on the form
`{Key, Type, Status}` (i.e. what corresponds to `'$_'`).
The rest of the items will be applied to each of the matching entries.
The meaning of the above tuple elements is:
+ `Reporter :: module()` Specifies the reporter plugin module, such as`exometer_report_collectd` that is to receive updated metric's data
points.
+ `Metric :: [atoms()]` Specifies the path to a metric previously created with an`exometer:new()` call.
+ `DataPoint` :: atom() | [atom()]' Specifies the data point within the given metric to send to the
receiver. The data point must match one of the data points returned by`exometer:info(Name, datapoints)` for the given metrics name.
+ `Interval` :: integer()' (milliseconds) Specifies the interval, in milliseconds, between each update of the
given metric's data point. At the given interval, the data point will
be samples, and the result will be sent to the receiver.
+ `RetryFailedMetrics :: boolean()` Specifies if the metric should be continued to be reported
even if it is not found during a reporting cycle. This would be
the case if a metric is not created by the time it is reported for
the first time. If the metric will be created at a later time,
this value should be set to true. Set this value to false if all
attempts to report the metric should stop if when is not found.
The default value is `true`.
+ `Extra :: any()` Provides a means to pass along extra information for a given
subscription. An example is the `syntax` option for the SNMP reporter,
in which case `Extra` needs to be a property list.
Example configuration in sys.config, using the `{select, Expr}` pattern:
```erlang
[
{exometer, [
{predefined,
[{[a,1], counter, []},
{[a,2], counter, []},
{[b,1], counter, []},
{[c,1], counter, []}]},
{report,
[
{reporters,
[{exometer_report_tty, []}]},
{subscribers,
[{select, {[{ {[a,'_'],'_','_'}, [], ['$_']}],
exometer_report_tty, value, 1000}}]}
]}
]}
].
```
This will activate a subscription on `[a,1]` and `[a,2]` in the
`exometer_report_tty` reporter, firing once per second.
#### Configuring reporter plugins ####
The various reporter plugins to be loaded by exometer are configured
in the `report` section under `reporters`
Each reporter has an entry named after its module, and the content of
that entry is dependent on the reporter itself. The following chapters
specifies the configuration parameters for the reporters shipped with
exometer.
#### Configuring opentsdb reporter ####
Below is an example of the opentsdb reporter application environment, with
its correct location in the hierarchy:
```erlang
{exometer, [
{report, [
{reporters, [
{exometer_report_opentsdb, [
{reconnect_interval, 10},
{connect_timeout, 8000},
{hostname, "testhost"},
{host, {"127.0.0.1", 4242}}
]}
]}
]}
]}
```
The following attributes are available for configuration:
+ `reconnect_interval` (seconds - default: 30) Specifies the duration between each reconnect attempt to an opentsdb
server that is not available. Should the server either be unavailable
at exometer startup, or become unavailable during exometer's
operation, exometer will attempt to reconnect at the given number of
seconds.
+ `connect_timeout` (milliseconds - default: 5000) Specifies how long the opentsdb reporter plugin shall wait for a
socket connection to complete before timing out. A timed out
connection attempt will be retried after the reconnect interval has
passed see item 1 above).
+ `hostname` (string - default: `net_adm:localhost()`) Specifies the host name to use for the host tag in the OpenTSDB tags.
Please see [Configuring opentsdb reporter](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_opentsdb_reporter) for details.
+ `host` (ip - default: {"127.0.0.1", 4242}) Specifies the host and port to connect to OpenTSDB.
#### Configuring amqp reporter ####
Below is an example of the amqp reporter application environment, with
its correct location in the hierarchy:
```erlang
{exometer, [
{report, [
{reporters, [
{exometer_report_amqp, [
{reconnect_interval, 10},
{hostname, "testhost"},
{amqp_url, "amqp://user:pass@host:5672/%2f"},
{exchange, "metrics"},
{routing_key, "metrics"},
{buffer_size, 0}
]}
]}
]}
]}
```
The following attributes are available for configuration:
+ `reconnect_interval` (seconds - default: 30) Specifies the duration between each reconnect attempt to an amqp
broker that is not available. Should the server either be unavailable
at exometer startup, or become unavailable during exometer's
operation, exometer will attempt to reconnect at the given number of
seconds.
+ `hostname` (string - default: `net_adm:localhost()`) Specifies the host name to use for the host property in the JSON payload.
Please see [Configuring amqp reporter](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_amqp_reporter) for details.
+ `amqp_url` (string - default: `amqp://guest:guest@localhost:5672/%2f`) Specifies the amqp url to connect to.
+ `exchange` (string - default: `exometer`) Specifies the exchange to publish messages to.
+ `routing_key` (string - default: `exometer`) Specifies the routing key to use when publishing messages.
+ `buffer_size` (integer - default: `0`) Specifies the size in bytes of payload to buffer before sending to AMQP.
#### Configuring graphite reporter ####
Below is an example of the a graphite reporter application environment, with
its correct location in the hierarchy:
```erlang
{exometer, [
{report, [
{reporters, [
{exometer_report_graphite, [
{connect_timeout, 5000},
{prefix, "web_stats"},
{host, "carbon.hostedgraphite.com"},
{port, 2003},
{api_key, "267d121c-8387-459a-9326-000000000000"}
]}
]}
]}
]}
```
The following attributes are available for configuration:
+ `connect_timeout` (milliseconds - default: 5000) Specifies how long the graphie reporter plugin shall wait for a tcp
connection to complete before timing out. A timed out connection will
not be reconnected to automatically. (To be fixed.)
+ `prefix` (string - default: "") Specifies an optional prefix to prepend all metric names with before
they are sent to the graphite server.
+ `host` (string - default: "carbon.hostedgraphite.com") Specifies the name (or IP address) of the graphite server to report to.
+ `port` (integer - default: 2003) Specifies the TCP port on the given graphite server to connect to.
+ `api_key` (string - default: n/a) Specifies the api key to use when reporting to a hosted graphite server.
If `prefix` is not specified, but `api_key` is, each metrics will be reported as `ApiKey.Metric`.
If `prefix` is specified, but `api_key` is not, each metrics will be reported as `Prefix.Metric`.
if neither `prefix` or `api_key` is specified, each metric will be reported simply as `Metric`.
#### Configuring snmp reporter ####
Below is an example of the a snmp reporter application environment, with
its correct location in the hierarchy:
```erlang
{exometer, [
{report, [
{reporters, [
{exometer_report_snmp, [
{mib_template, "priv/MYORG-EXOMETER-METRICS.mib"},
{mib_dir, "/tmp/exometer"}
]}
]}
]}
]}
```
The following attributes are available for configuration:
+ `mib_template` (string - default: "mibs/EXOMETER-METRICS-MIB.mib") Specifies where to find the MIB template used for dynamically assembline an internal MIB. Take a look at the MIB template shipped with Exometer for reference in case you want to define your own template.
+ `mib_dir` (string - default: "tmp/exometer_report_snmp") Specifies temporary direction which will be used by Exometer to store dymanically created MIB files.
### Creating custom exometer entries ###
Please see @see exometer_entry documentation for details.
### Creating custom probes ###
Please see @see exometer_probe documentation for details.
### Creating custom reporter plugins ###
Please see @see exometer_report documentation for details.
### Dependency management ###
Exometer dependencies can be controlled using the `EXOMETER_PACKAGES`
unix environment variable: a string listing packages or applications to
either keep or remove, separated using space, tab or comma.
#### Syntax ####
+ `(Package)` - use `Package` as a base. This will implicitly exclude all
applications not included in `Package`. See below for supported packages.
+ `+(Package)` - add applications included in `Package`.
+ `-(Package)` - remove applications in `Package` (except mandatory deps).
+ `App` - keep application `App`.
+ `+App` - keep application `App`.
+ `-App` - exclude application `App`.
#### Supported packages ####
+ `minimal` - only the mandatory deps: `lager`, `parse_trans`, `setup`.
+ `basic` - (mandatory deps and) `folsom`.
+ `amqp` - (mandatory deps and) `amqp_client`, `jiffy`.
+ `full` - all of the above, plus `afunix` and `netlink`.
Example - use only basic deps plus `afunix`
```
EXOMETER_PACKAGES="(basic), +afunix" make
```
Example - use all deps except the AMQP-related deps:
```
export EXOMETER_PACKAGES="(full) -(amqp)"
```
#### Conditional defines ####
For each optional dependency that is included, a macro is defined,
named `dep_App` - e.g. `dep_afunix`. Developers must not include
compile-time dependencies to optional applications, without checking
the corresponding macro and ensuring that the module compiles even
when the dependent application is not included. See `exometer_report_amqp.erl`
for an example.
#### Customizing rebar.config ####
The OS environment variables `EXOMETER_CONFIG_PREPROCESS` and
`EXOMETER_CONFIG_POSTPROCESS` can be used to insert a script, similar to
`rebar.config.script` in the processing flow of the exometer build.
As the names imply, the script given by `EXOMETER_CONFIG_PREPROCESS` (if any)
will be run before exometer does any processing of its own, and the
`EXOMETER_CONFIG_POSTPROCESS` script (if any) will be run after all other
processing is complete.
Things that could be done in preprocessing: re-targeting a dependency,
modifying the list of predefined packages, etc.
## Modules ##
================================================
FILE: TODO.md
================================================
# Improvements
- report plugin processes are currently manually monitored by exometer_report, this should be changed to a more robust appraoch. Could use simple_one_for_one, which lacks restarts. Can't use other normal supervision strategies without some changes as we don't want external systems to trigger a supervision tree collapse.
# Bugs
- fix all dialyzer errors
- type specs for SNMP stuff
- subscription parameter extra needs to be sent to manager as part of inform
# Misc
- use proper module organization in all modules, e.g. exports, includes, external API, internal API
- update documentation
================================================
FILE: doc/README.md
================================================
# Exometer - Erlang instrumentation package #
Copyright (c) 2014 Basho Technologies, Inc. All Rights Reserved.
__Version:__ Feb 1 2015 23:02:37
__Authors:__ Ulf Wiger ([`ulf.wiger@feuerlabs.com`](mailto:ulf.wiger@feuerlabs.com)), Magnus Feuer ([`magnus.feuer@feuerlabs.com`](mailto:magnus.feuer@feuerlabs.com)).
[](https://travis-ci.org/Feuerlabs/exometer)
__NOTE: Exometer has been split into [exometer_core](https://github.com/Feuerlabs/exometer_core), and exometer (as well as separate reporter applications). The latest monolithic version of Exometer is 1.1.__
The Exometer package allows for easy and efficient instrumentation of
Erlang code, allowing crucial data on system performance to be
exported to a wide variety of monitoring systems.
Exometer comes with a set of pre-defined monitor components, and can
be expanded with custom components to handle new types of Metrics, as
well as integration with additional external systems such as
databases, load balancers, etc.
This document gives a high level overview of the Exometer system. For
details, please see the documentation for individual modules, starting
with `exometer`.
Note the section on [Dependency Management](#Dependency_Management) for how to deal with
optional packages, both users and developers.
### Table of Content ###
1. [Concept and definitions](#Concept_and_definitions)
1. [Metric](#Metric)
2. [Data Point](#Data_Point)
3. [Metric Type](#Metric_Type)
4. [Entry Callback](#Entry_Callback)
5. [Probe](#Probe)
6. [Caching](#Caching)
7. [Subscriptions and Reporters](#Subscriptions_and_Reporters)
2. [Built-in entries and probes](#Built-in_entries_and_probes)
1. [counter (exometer native)](#counter_(exometer_native))
2. [fast_counter (exometer native)](#fast_counter_(exometer_native))
3. [gauge (exometer native)](#gauge_(exometer_native))
4. [exometer_histogram (probe)](#exometer_histogram_(probe))
5. [exometer_uniform (probe)](#exometer_uniform_(probe))
6. [exometer_spiral (probe)](#exometer_spiral_(probe))
7. [exometer_folsom [entry]](#exometer_folsom_[entry])
8. [exometer_function [entry]](#exometer_function_[entry])
3. [Built in Reporters](#Built_in_Reporters)
1. [exometer_report_graphite](#exometer_report_graphite)
2. [exometer_report_opentsdb](#exometer_report_opentsdb)
3. [exometer_report_amqp](#exometer_report_amqp)
4. [exometer_report_snmp](#exometer_report_snmp)
4. [Instrumenting Erlang code](#Instrumenting_Erlang_code)
1. [Exometer Start](#Exometer_Start)
2. [Creating metrics](#Creating_metrics)
3. [Deleting metrics](#Deleting_metrics)
4. [Setting metric values](#Setting_metric_values)
5. [Retrieving metric values](#Retrieving_metric_values)
6. [Setting up subscriptions](#Setting_up_subscriptions)
7. [Set metric options](#Set_metric_options)
5. [Configuring Exometer](#Configuring_Exometer)
1. [Configuring type - entry maps](#Configuring_type_-_entry_maps)
2. [Configuring statically defined entries](#Configuring_statically_defined_entries)
3. [Configuring static subscriptions](#Configuring_static_subscriptions)
4. [Configuring reporter plugins](#Configuring_reporter_plugins)
5. [Configuring opentsdb reporter](#Configuring_opentsdb_reporter)
6. [Configuring amqp reporter](#Configuring_amqp_reporter)
7. [Configuring graphite reporter](#Configuring_graphite_reporter)
8. [Configuring snmp reporter](#Configuring_snmp_reporter)
6. [Creating custom exometer entries](#Creating_custom_exometer_entries)
7. [Creating custom probes](#Creating_custom_probes)
8. [Creating custom reporter plugins](#Creating_custom_reporter_plugins)
9. [Dependency management](#Dependency_management)
### Concepts and Definitions ###
Exometer introduces a number of concepts and definitions used
throughout the documentation and the code.
#### Metric ####
A metric is a specific measurement sampled inside an Erlang system and
then reported to the Exometer system. An example metric would be
"transactions_per_second", or "memory_usage".
Metrics are identified by a list of terms, such as given below:
`[ xml_front_end, parser, file_size ]`
A metric is created through a call by the code to be instrumented to
`exometer:new()`. Once created, the metric can be updated through
`exometer:update()`, or on its own initiative through the
`exometer_probe:sample` behavior implementation.
#### Data Point ####
Each metric can consist of multiple data points, where each point has
a specific value.
A typical example of data points would be a
`transactions_per_second` (tps) metric, usually stored as a
histogram covering the last couple of minutes of tps samples. Such a
histogram would host multiple values, such as `min`, `max`,
`median`, `mean`, `50_percentile`, `75_percentile`,
etc.
It is up to the type of the metric, and the data probe backing that
type (see below), to specify which data points are available under the
given metric.
#### Metric Type ####
The type of a metric, specified when the metric is created through
`exometer:new()`, determines which `exometer_entry`
callback to use.
The link between the type and the entry to use is configured
through the `exometer_admin` module, and its associated exometer
defaults configuration data.
The metric type, in other words, is mainly used to map a metric to a
configurable `exometer_entry` callback, but it can also be referenced
in queries using `exometer:select/1`. An entry callback can also support
multiple types (the type is provided as an argument in the callback functions).
Exometer provides default mappings for a number of metric types. It is
possible to select different callbacks for each metric instance, as well
as modify metrics using callback-specific options. Please see
[Configuring type - entry maps](#Configuring_type_-_entry_maps) for details on how to do this.
#### Entry Callback ####
An exometer entry callback will receive values reported to a metric through the
`exometer:update()` call and compile it into one or more data points.
The entry callback can either be a counter (implemented natively
in `exometer`), or a more complex statistical analysis such
as a uniform distribution or a regular histogram.
The various outputs from these entries are reported as data points
under the given metric.
An entry can also interface external analytics packages.
`exometer_folsom`, for example, integrates with the
`folsom_metrics` package found at [`https://github.com/boundary/folsom`](https://github.com/boundary/folsom).
#### Probe ####
Probes are a further specialization of exometer entries that run in
their own Erlang processes and have their own state (like a
gen_server). A probe is implemented through the `exometer_probe`
behavior.
A probe can be used if independent monitoring is needed of,
for example, `/proc` trees, network interfaces, and other subsystems
that need periodic sampling. In these cases, the
`exometer_probe:probe_sample()` call is invoked regularly by exometer,
in the probe's own process, in order to extract data from
the given subsystem and add it to the metric's data points.
#### Caching ####
Metric and data point values are read with the `exometer:get_value()`
function. In the case of counters, this operation is very fast. With probes,
the call results in a synchronous dialog with the probe process, and the
cost of serving the request depends on the probe implementation and the
nature of the metric being served.
If the cost of reading the value is so high that calling the function often
would result in prohibitive load, it is possible to cache the value. This is
done either explicitly from the probe itself (by calling
`exometer_cache:write()`), or by specifying the option `{cache, Lifetime}`
for the entry. If an entry has a non-zero cache lifetime specified, the
`get_value()` call will try fetching the cached value before calling the
actual entry and automatically caching the result.
Note that if `{cache, Lifetime}` is not specified, `exometer:get_value()`
will neither read nor write to the cache. It is possible for the probe
to periodically cache a value regardless of how the cache lifetime is set,
and the probe may also explicitly read from the cache if it isn't done
automatically.
#### Subscriptions and Reporters ####
The subscription concept, managed by `exometer_report` allows metrics
and their data points to be sampled at given intervals and delivered
to one or more recipients, which can be either an arbitrary process
or a Reporter plugin.
Each subscription ties a specific metric-datapoint pair to a reporter
and an interval (given in milliseconds). The reporter system will, at
the given interval, send the current value of the data point to the
subscribing reporter. The subscription, with all its parameters,
is setup through a call to `exometer_report:subscribe()`.
In the case of processes, subscribed-to values will be delivered as a
message. Modules, which implement the `exometer_report` callback
behavior, will receive the plugins as a callbacks within the
`exometer_report` process.
Subscriptions can either be setup at runtime, through
`exometer_report:subscribe()` calls, or statically through the
`exometer_report` configuration data.
### Built-in entries and probes ###
There are a number of built-in entries and probes shipped
with the Exometer package, as described below:
#### counter (exometer native) ####
The counter is implemented directly in `exometer` to provide simple
counters. A call to `exometer:update()` will add the provided value
to the counter.
The counter can be reset to zero through `exometer:reset()`.
The available data points under a metric using the counter entry
are `value` and `ms_since_reset`.
#### fast_counter (exometer native) ####
A fast counter implements the counter functionality, through the
`trace_info` system, yielding a speed increase of about 3.5 in
comparison to the regular counter.
The tradeoff is that running tracing and/or debugging may interfere
with the counter functionality.
A call to `exometer:update()` will add the provided value to the
counter.
The counter can be reset to zero through `exometer:reset()`.
The available data points under a metric using the fast_counter
entry are `value` and `ms_since_reset`.
#### gauge (exometer native) ####
The gauge is implemented directly in `exometer` to provide simple
gauges. A call to `exometer:update()` will set the gauge's value
to the provided value. That is, the value of the gauge entry is
always the most recently provided value.
The gauge can be reset to zero through `exometer:reset()`.
The available data points under a metric using the gauge entry
are `value` and `ms_since_reset`.
#### histogram (probe) ####
The histogram probe stores a given number of updates, provided through
`exometer:update()`, in a histogram. The histogram maintains a log
derived from all values received during a configurable time span and
provides min, max, median, mean, and percentile analysis data points
for the stored data.
Exometer supports a number of different histogram implementations, each
with different performance and accuracy trade-offs.
In order to save memory, the histogram is divided into equal-sized
time slots, where each slot spans a settable interval. All values
received during a time slot will be averaged into a single value to be
stored in the histogram once the time slot expires. The averaging
function (which can be replaced by the caller), allows for
high-frequency update metrics to have their resolution traded against
resource consumption.
#### exometer_uniform (probe) ####
The uniform probe provides a uniform sample over a pool of values
provided through `exometer:update()`. When the pool reaches its configurable
max size, existing values will be replaced at random to make space for
new values. Much like `exometer_histogram`, the uniform probe
provides min, max, median, mean, and percentile analysis data points
for the stored data.
#### exometer_spiral (probe) ####
The spiral probe maintains the total sum of all values stored in its
histogram. The histogram has a configurable time span, all values
provided to the probe, through `exometer:update()`, within that time
span will be summed up and reported. If, for example, the histogram
covers 60 seconds, the spiral probe will report the sum of all
values reported during the last minute.
The grand total of all values received during the lifetime of the
probe is also available.
#### exometer_folsom [entry] ####
`exometer_folsom` is an entry behavior which implements most metric types
supported by the [folsom](https://github.com/boundary/folsom)
metrics package: Specifically, the metric types `counter`, `spiral`,
`histogram`, `meter`, `meter_reader`, `gauge`, `duration` and `history`.
The folsom entry integrates with the folsom metrics package provided
by the boundary repo at github. Updated values sent to the folsom entry
can be forwarded to folsom's counter, histogram, duration, meter,
and spiral.
Folsom integration is provided as a backup. New code using Exometer
should use the native probes that duplicate folsom.
#### exometer_function [entry] ####
The function entry allows for an existing erlang function to be wrapped
as an exometer entry. The [`exometer_function`](https://github.com/Feuerlabs/exometer_core/blob/master/doc/exometer_function.md) module supports a number
of options for passing arguments and matching out data points from the
result.
The function entry provides an easy way of integrating an external
system without having to write a complete entry.
### Built in Reporters ###
Exometer ships with some built-in reporters which can be used to forward updated
metrics and their data points to external systems. They can also
serve as templates for custom-developed reporters.
#### exometer_report_graphite ####
The graphite reporter uses the TCP/IP protocol to forward
subscribed-to metrics and data points to a graphite server, such as
the one provided by [`http://hostedgraphite.com`](http://hostedgraphite.com). When the graphite
reporter receives a metric-datapoint value (subscribed to through
`exometer_report:subscriber()`), the reporter will immediately
forward the key-value pair to the graphite server.
#### exometer_report_opentsdb ####
The OpenTSDB reporter sends metrics to an OpenTSDB server using
the telnet API. All subscribed-to metric-datapoint values received
by the reporter are immediately forwarded to OpenTSDB.
If the OpenTSDB connection is lost, the reporter will attempt to reconnect to it
at a configurable interval.
The data sent to OpenTSDB will be formatted as follows:
```
put metric timestamp value host=host type=datapoint
```
Where the value for the host tag will be the configured host in the reporter
configuration (defaults to the value returned by `netadm:localhost`), and
datapoint tags as specified by the subscriber.
Please see [Configuring opentsdb reporter](#Configuring_opentsdb_reporter) for details on the
application environment parameters listed above.
#### exometer_report_amqp ####
The AMQP reporter sends metrics to an AMQP broker as a json-encoded payload. All
subscribed-to metric-datapoint values received by the reporter are forwarded to AMQP.
If the AMQP connection is lost, the reporter will attempt to reconnect to it
at a configurable interval.
The data sent to AMQP will be formatted as follows:
```
{
"type":"exometer_metric",
"body":
{"name":"messages_per_second",
"value":0,"timestamp":1414006826,
"host":"testhost",
"instance":"max"}
}
```
Where the value for the host tag will be the configured host in the reporter
configuration (defaults to the value returned by `netadm:localhost`), the
instance tag represents the datapoint for the metric.
Please see [Configuring amqp reporter](#Configuring_amqp_reporter) for details on the
application environment parameters listed above.
#### exometer_report_snmp ####
The SNMP reporter enables the export of metrics and their datapoints to SNMP managers.
The export needs to be enabled for each metric through their options.
Moreover, SNMP notifications can be created using the options to send periodic reports
on datapoints to SNMP managers. All SNMP protocol handling is done by the snmp application
shipped with Erlang/OTP. Thus, the snmp application needs to be started and
the local SNMP master agent needs to be configured correctly for SNMP export to work
properly.
To configure SNMP export for a single metric use these options:
+ `{snmp, disabled}` (default) Disables SNMP export for the metric. Same as not specifying the option at all.
+ `{snmp, []}` Enables SNMP export for the metric. No subscriptions are setup.
+ `{snmp, [{Datapint, Interval}]}` Enables SNMP export for the metric. Subscriptions are setup for the given Datapoint/Interval pairs. Each subscription report will be forwarded to SNMP mangers as notifications.
+ `{snmp, [{Datapint, Interval, Extra}]}` Same as above, but using an addition extra identification for the subscriptions. Allow the creation ofmultiple subscriptions for a single datapoint.
Please see [Configuring snmp reporter](#Configuring_snmp_reporter) for details on how to configure the
SNMP reporter.
### Instrumenting Erlang code ###
The code using Exometer needs to be instrumented in order to setup and
use metrics reporting.
#### Exometer Start ####
The system using Exometer must start the `exometer` application prior to using it:
```erlang
application:start(lager),
application:start(exometer).
```
Note that dependent applications need to be started first. On newer OTP versions
(R16B or later), you can use `application:ensure_all_started(exometer)`.
For testing, you can also use [`exometer:start/0`](https://github.com/Feuerlabs/exometer_core/blob/master/doc/exometer.md#start-0).
If you make use of e.g. folsom metrics, you also need to start `folsom`. Exometer
will not do that automatically, nor does it contain an application dependency for it.
See [Configuring Exometer](#Configuring_Exometer) for details on configuration data
format.
#### Creating metrics ####
A metric, can be created through a call to
```erlang
exometer:new(Name, Type)
```
`Name` is a list of atoms, uniquely identifying the metric created.
The type of the metric, specified by `Type` will be mapped
to an exometer entry through the table maintained by
`exometer_admin` Please see the [Configuring type - entry
maps](#Configuring_type_-_entry_maps) for details.
The resolved entry to use will determine the data points available
under the given metric.
#### Deleting metrics ####
A metric previously created with `exometer:new()` can be deleted by
`exometer:delete()`.
All subscriptions to the deleted metrics will be cancelled.
#### Setting metric values ####
A created metric can have its value updated through the
`exometer:update()` function:
```erlang
exometer:update(Name, Value)
```
The `Name` parameter is the same atom list provided to a previous
`exometer:new()` call. The `Value` is an arbitrarty element that is
forwarded to the `exometer:update()` function of the entry/probe that the
metric is mapped to.
The receiving entry/probe will process the provided value and modify
its data points accordingly.
#### Retrieving metric values ####
Exometer-using code can at any time retrieve the data point values
associated with a previously created metric. In order to find out which
data points are available for a metric, the following call can be used:
```erlang
exometer:info(Name, datapoints)
```
The `Name` parameter is the same atom list provided to a previous
`exometer:new()` call. The call will return a list of data point
atoms that can then be provided to `exometer:get_value()` to
retrieve their actual value:
```erlang
exometer:get_value(Name, DataPoint)
```
The `Name` paramer identifies the metric, and `DataPoints`
identifies the data points (returned from the previous `info()` call)
to retrieve the value for.
If no DataPoints are provided, the values of a default list of data points,
determined by the backing entry / probe, will be returned.
#### Setting up subscriptions ####
A subscription can either be statically configured, or dynamically
setup from within the code using Exometer. For details on statically
configured subscriptions, please see [Configuring static subscriptions](#Configuring_static_subscriptions).
A dynamic subscription can be setup with the following call:
```erlang
exometer_report:subscribe(Recipient, Metric, DataPoint, Inteval)
```
`Recipient` is the name of a reporter.
#### Set metric options ####
Each created metric can have options setup for it through the following call:
```erlang
exometer:setopts(Name, Options)
```
The `Name` paramer identifies the metric to set the options for, and
Options is a proplist (`[{ Key, Value },...]`) with the options to be
set.
Exometer looks up the the backing entry that hosts the metric with the given Name, and will
invoke the entry\'s `setopts/4` function to set the actual options. Please see the
`setopts/4` function for the various entries for details.
### Configuring Exometer ###
Exometer defaults can be changed either through OTP application environment
variables or through the use of Basho's `cuttlefish`
([`https://github.com/basho/cuttlefish`](https://github.com/basho/cuttlefish)).
#### Configuring type - entry maps ####
The dynamic method of configuring defaults for `exometer` entries is:
```erlang
exometer_admin:set_default(NamePattern, Type, Default)
```
Where `NamePattern` is a list of terms describing what is essentially
a name prefix with optional wildcards (`'_'`). A pattern that
matches any legal name is `['_']`.
`Type` is an atom defining a type of metric. The types already known to
`exometer`, `counter`, `fast_counter`, `ticker`, `uniform`, `histogram`,
`spiral`, `netlink`, and `probe` may be redefined, but other types can be
described as well.
`Default` is either an `#exometer_entry{}` record (unlikely), or a list of
`{Key, Value}` options, where the keys correspond to `#exometer_entry` record
attribute names. The following attributes make sense to preset:
```erlang
{module, atom()} % the callback module
{status, enabled | disabled} % operational status of the entry
{cache, non_neg_integer()} % cache lifetime (ms)
{options, [{atom(), any()}]} % entry-specific options
```
Below is an example, from `exometer/priv/app.config`:
```erlang
{exometer, [
{defaults, [
{['_'], function , [{module, exometer_function}]},
{['_'], counter , [{module, exometer}]},
{['_'], histogram, [{module, exometer_histogram}]},
{['_'], spiral , [{module, exometer_spiral}]},
{['_'], duration , [{module, exometer_folsom}]},
{['_'], meter , [{module, exometer_folsom}]},
{['_'], gauge , [{module, exometer_folsom}]}
]}
]}
```
In systems that use CuttleFish, the file `exometer/priv/exometer.schema`
contains a schema for default settings. The setup corresponding to the above
defaults would be as follows:
```ini
exometer.template.function.module = exometer_function
exometer.template.counter.module = exometer
exometer.template.histogram.module = exometer_histogram
exometer.template.spiral.module = exometer_spiral
exometer.template.duration.module = exometer_folsom
exometer.template.meter.module = exometer_folsom
exometer.template.gauge.module = exometer_folsom
```
#### Configuring statically defined entries ####
Using the `exometer` environment variable `predefined`, entries can be added
at application startup. The variable should have one of the following values:
* `{script, File}` - `File` will be processed using `file:script/2`. The return
value (the result of the last expression in the script) should be a list of`{Name, Type, Options}` tuples.
* `{apply, M, F, A}` - The result of `apply(M, F, A)` should be `{ok, L}` where`L` is a list of `{Name, Type, Options}` tuples.
* `L`, where L is a list of `{Name, Type, Options}` tuples or extended
instructions (see below).
The list of instructions may include:
* `{delete, Name}` - deletes `Name` from the exometer registry.
* `{select_delete, Pattern}` - applies a select pattern and
deletes all matching entries.
* `{re_register, {Name, Type, Options}}` - redefines an entry if present,
otherwise creates it.
Exometer will also scan all loaded applications for the environment
variables `exometer_defaults` and `exometer_predefined`, and process
as above. If an application is loaded and started after exometer has started,
it may call the function `exometer:register_application()` or
`exometer:register_application(App)`. This function will do nothing if
exometer isn't already running, and otherwise process the `exometer_defaults`
and `exometer_predefined` variables as above. The function can also be
called during upgrade, as it will re-apply the settings each time.
#### Configuring static subscriptions ####
Static subscriptions, which are automatically setup at exometer
startup without having to invoke `exometer_report:subscribe()`, are
configured through the report sub section under exometer.
Below is an example, from `exometer/priv/app.config`:
```erlang
{exometer, [
{report, [
{subscribers, [
{exometer_report_collectd, [db, cache, hits], mean, 2000, true},
{exometer_report_collectd, [db, cache, hits], max, 5000, false}
]}
]}
]}
```
The `report` section configures static subscriptions and reporter
plugins. See [Configuring reporter plugins](#Configuring_reporter_plugins) for details on
how to configure individual plugins.
The `subscribers` sub-section contains all static subscriptions to be
setup att exometer applications start. Each tuple in the prop list
should be of one of the following formats:
* `{Reporter, Metric, DataPoint, Interval}`
* `{Reporter, Metric, DataPoint, Interval, RetryFailedMetrics}`
* `{Reporter, Metric, DataPoint, Interval, RetryFailedMetrics, Extra}`
* `{apply, {M, F, A}}`
* `{select, {MatchPattern, DataPoint, Interval [, Retry [, Extra] ]}}`
In the case of `{apply, M, F, A}`, the result of `apply(M, F, A)` must
be a list of `subscribers` tuples.
In the case of `{select, Expr}`, a list of metrics is fetched using
`exometer:select(MatchPattern)`, where the result must be on the form
`{Key, Type, Status}` (i.e. what corresponds to `'$_'`).
The rest of the items will be applied to each of the matching entries.
The meaning of the above tuple elements is:
+ `Reporter :: module()` Specifies the reporter plugin module, such as`exometer_report_collectd` that is to receive updated metric's data
points.
+ `Metric :: [atoms()]` Specifies the path to a metric previously created with an`exometer:new()` call.
+ `DataPoint` :: atom() | [atom()]' Specifies the data point within the given metric to send to the
receiver. The data point must match one of the data points returned by`exometer:info(Name, datapoints)` for the given metrics name.
+ `Interval` :: integer()' (milliseconds) Specifies the interval, in milliseconds, between each update of the
given metric's data point. At the given interval, the data point will
be samples, and the result will be sent to the receiver.
+ `RetryFailedMetrics :: boolean()` Specifies if the metric should be continued to be reported
even if it is not found during a reporting cycle. This would be
the case if a metric is not created by the time it is reported for
the first time. If the metric will be created at a later time,
this value should be set to true. Set this value to false if all
attempts to report the metric should stop if when is not found.
The default value is `true`.
+ `Extra :: any()` Provides a means to pass along extra information for a given
subscription. An example is the `syntax` option for the SNMP reporter,
in which case `Extra` needs to be a property list.
Example configuration in sys.config, using the `{select, Expr}` pattern:
```erlang
[
{exometer, [
{predefined,
[{[a,1], counter, []},
{[a,2], counter, []},
{[b,1], counter, []},
{[c,1], counter, []}]},
{report,
[
{reporters,
[{exometer_report_tty, []}]},
{subscribers,
[{select, {[{ {[a,'_'],'_','_'}, [], ['$_']}],
exometer_report_tty, value, 1000}}]}
]}
]}
].
```
This will activate a subscription on `[a,1]` and `[a,2]` in the
`exometer_report_tty` reporter, firing once per second.
#### Configuring reporter plugins ####
The various reporter plugins to be loaded by exometer are configured
in the `report` section under `reporters`
Each reporter has an entry named after its module, and the content of
that entry is dependent on the reporter itself. The following chapters
specifies the configuration parameters for the reporters shipped with
exometer.
#### Configuring opentsdb reporter ####
Below is an example of the opentsdb reporter application environment, with
its correct location in the hierarchy:
```erlang
{exometer, [
{report, [
{reporters, [
{exometer_report_opentsdb, [
{reconnect_interval, 10},
{connect_timeout, 8000},
{hostname, "testhost"},
{host, {"127.0.0.1", 4242}}
]}
]}
]}
]}
```
The following attributes are available for configuration:
+ `reconnect_interval` (seconds - default: 30) Specifies the duration between each reconnect attempt to an opentsdb
server that is not available. Should the server either be unavailable
at exometer startup, or become unavailable during exometer's
operation, exometer will attempt to reconnect at the given number of
seconds.
+ `connect_timeout` (milliseconds - default: 5000) Specifies how long the opentsdb reporter plugin shall wait for a
socket connection to complete before timing out. A timed out
connection attempt will be retried after the reconnect interval has
passed see item 1 above).
+ `hostname` (string - default: `net_adm:localhost()`) Specifies the host name to use for the host tag in the OpenTSDB tags.
Please see [Configuring opentsdb reporter](#Configuring_opentsdb_reporter) for details.
+ `host` (ip - default: {"127.0.0.1", 4242}) Specifies the host and port to connect to OpenTSDB.
#### Configuring amqp reporter ####
Below is an example of the amqp reporter application environment, with
its correct location in the hierarchy:
```erlang
{exometer, [
{report, [
{reporters, [
{exometer_report_amqp, [
{reconnect_interval, 10},
{hostname, "testhost"},
{amqp_url, "amqp://user:pass@host:5672/%2f"},
{exchange, "metrics"},
{routing_key, "metrics"},
{buffer_size, 0}
]}
]}
]}
]}
```
The following attributes are available for configuration:
+ `reconnect_interval` (seconds - default: 30) Specifies the duration between each reconnect attempt to an amqp
broker that is not available. Should the server either be unavailable
at exometer startup, or become unavailable during exometer's
operation, exometer will attempt to reconnect at the given number of
seconds.
+ `hostname` (string - default: `net_adm:localhost()`) Specifies the host name to use for the host property in the JSON payload.
Please see [Configuring amqp reporter](#Configuring_amqp_reporter) for details.
+ `amqp_url` (string - default: `amqp://guest:guest@localhost:5672/%2f`) Specifies the amqp url to connect to.
+ `exchange` (string - default: `exometer`) Specifies the exchange to publish messages to.
+ `routing_key` (string - default: `exometer`) Specifies the routing key to use when publishing messages.
+ `buffer_size` (integer - default: `0`) Specifies the size in bytes of payload to buffer before sending to AMQP.
#### Configuring graphite reporter ####
Below is an example of the a graphite reporter application environment, with
its correct location in the hierarchy:
```erlang
{exometer, [
{report, [
{reporters, [
{exometer_report_graphite, [
{connect_timeout, 5000},
{prefix, "web_stats"},
{host, "carbon.hostedgraphite.com"},
{port, 2003},
{api_key, "267d121c-8387-459a-9326-000000000000"}
]}
]}
]}
]}
```
The following attributes are available for configuration:
+ `connect_timeout` (milliseconds - default: 5000) Specifies how long the graphie reporter plugin shall wait for a tcp
connection to complete before timing out. A timed out connection will
not be reconnected to automatically. (To be fixed.)
+ `prefix` (string - default: "") Specifies an optional prefix to prepend all metric names with before
they are sent to the graphite server.
+ `host` (string - default: "carbon.hostedgraphite.com") Specifies the name (or IP address) of the graphite server to report to.
+ `port` (integer - default: 2003) Specifies the TCP port on the given graphite server to connect to.
+ `api_key` (string - default: n/a) Specifies the api key to use when reporting to a hosted graphite server.
If `prefix` is not specified, but `api_key` is, each metrics will be reported as `ApiKey.Metric`.
If `prefix` is specified, but `api_key` is not, each metrics will be reported as `Prefix.Metric`.
if neither `prefix` or `api_key` is specified, each metric will be reported simply as `Metric`.
#### Configuring snmp reporter ####
Below is an example of the a snmp reporter application environment, with
its correct location in the hierarchy:
```erlang
{exometer, [
{report, [
{reporters, [
{exometer_report_snmp, [
{mib_template, "priv/MYORG-EXOMETER-METRICS.mib"},
{mib_dir, "/tmp/exometer"}
]}
]}
]}
]}
```
The following attributes are available for configuration:
+ `mib_template` (string - default: "mibs/EXOMETER-METRICS-MIB.mib") Specifies where to find the MIB template used for dynamically assembline an internal MIB. Take a look at the MIB template shipped with Exometer for reference in case you want to define your own template.
+ `mib_dir` (string - default: "tmp/exometer_report_snmp") Specifies temporary direction which will be used by Exometer to store dymanically created MIB files.
### Creating custom exometer entries ###
Please see @see exometer_entry documentation for details.
### Creating custom probes ###
Please see @see exometer_probe documentation for details.
### Creating custom reporter plugins ###
Please see @see exometer_report documentation for details.
### Dependency management ###
Exometer dependencies can be controlled using the `EXOMETER_PACKAGES`
unix environment variable: a string listing packages or applications to
either keep or remove, separated using space, tab or comma.
#### Syntax ####
+ `(Package)` - use `Package` as a base. This will implicitly exclude all
applications not included in `Package`. See below for supported packages.
+ `+(Package)` - add applications included in `Package`.
+ `-(Package)` - remove applications in `Package` (except mandatory deps).
+ `App` - keep application `App`.
+ `+App` - keep application `App`.
+ `-App` - exclude application `App`.
#### Supported packages ####
+ `minimal` - only the mandatory deps: `lager`, `parse_trans`, `setup`.
+ `basic` - (mandatory deps and) `folsom`.
+ `amqp` - (mandatory deps and) `amqp_client`, `jiffy`.
+ `full` - all of the above, plus `afunix` and `netlink`.
Example - use only basic deps plus `afunix`
```
EXOMETER_PACKAGES="(basic), +afunix" make
```
Example - use all deps except the AMQP-related deps:
```
export EXOMETER_PACKAGES="(full) -(amqp)"
```
#### Conditional defines ####
For each optional dependency that is included, a macro is defined,
named `dep_App` - e.g. `dep_afunix`. Developers must not include
compile-time dependencies to optional applications, without checking
the corresponding macro and ensuring that the module compiles even
when the dependent application is not included. See `exometer_report_amqp.erl`
for an example.
#### Customizing rebar.config ####
The OS environment variables `EXOMETER_CONFIG_PREPROCESS` and
`EXOMETER_CONFIG_POSTPROCESS` can be used to insert a script, similar to
`rebar.config.script` in the processing flow of the exometer build.
As the names imply, the script given by `EXOMETER_CONFIG_PREPROCESS` (if any)
will be run before exometer does any processing of its own, and the
`EXOMETER_CONFIG_POSTPROCESS` script (if any) will be run after all other
processing is complete.
Things that could be done in preprocessing: re-targeting a dependency,
modifying the list of predefined packages, etc.
## Modules ##
## Function Details ##
### exometer_call/3 ###
`exometer_call(Unknown, From, St) -> any()`
### exometer_cast/2 ###
`exometer_cast(Unknown, St) -> any()`
### exometer_info/2 ###
`exometer_info(Unknown, St) -> any()`
### exometer_init/1 ###
`exometer_init(Opts) -> any()`
### exometer_newentry/2 ###
`exometer_newentry(Entry, St) -> any()`
### exometer_report/5 ###
`exometer_report(Metric, DataPoint, Extra, Value, St) -> any()`
### exometer_setopts/4 ###
`exometer_setopts(Metric, Options, Status, St) -> any()`
### exometer_subscribe/5 ###
`exometer_subscribe(Metric, DataPoint, Extra, Interval, St) -> any()`
### exometer_terminate/2 ###
`exometer_terminate(X1, X2) -> any()`
### exometer_unsubscribe/4 ###
`exometer_unsubscribe(Metric, DataPoint, Extra, St) -> any()`
================================================
FILE: doc/overview.edoc
================================================
@author Ulf Wiger
@author Magnus Feuer
@copyright 2014 Basho Technologies, Inc. All Rights Reserved.
@version {@version}
@title Exometer - Erlang instrumentation package
@doc
[](https://travis-ci.org/Feuerlabs/exometer)
NOTE: Exometer has been split into exometer_core, and exometer (as well as separate reporter applications). The latest monolithic version of Exometer is 1.1.
The Exometer package allows for easy and efficient instrumentation of
Erlang code, allowing crucial data on system performance to be
exported to a wide variety of monitoring systems.
Exometer comes with a set of pre-defined monitor components, and can
be expanded with custom components to handle new types of Metrics, as
well as integration with additional external systems such as
databases, load balancers, etc.
This document gives a high level overview of the Exometer system. For
details, please see the documentation for individual modules, starting
with `exometer'.
Note the section on {@section Dependency Management} for how to deal with
optional packages, both users and developers.
== Table of Content ==
1. {@section Concept and definitions}
1. {@section Metric}
2. {@section Data Point}
3. {@section Metric Type}
4. {@section Entry Callback}
5. {@section Probe}
6. {@section Caching}
7. {@section Subscriptions and Reporters}
2. {@section Built-in entries and probes}
1. {@section counter (exometer native)}
2. {@section fast_counter (exometer native)}
3. {@section gauge (exometer native)}
4. {@section exometer_histogram (probe)}
5. {@section exometer_uniform (probe)}
6. {@section exometer_spiral (probe)}
7. {@section exometer_folsom [entry]}
8. {@section exometer_function [entry]}
3. {@section Built in Reporters}
1. {@section exometer_report_graphite}
2. {@section exometer_report_opentsdb}
3. {@section exometer_report_amqp}
4. {@section exometer_report_snmp}
4. {@section Instrumenting Erlang code}
1. {@section Exometer Start}
2. {@section Creating metrics}
3. {@section Deleting metrics}
4. {@section Setting metric values}
5. {@section Retrieving metric values}
6. {@section Setting up subscriptions}
7. {@section Set metric options}
5. {@section Configuring Exometer}
1. {@section Configuring type - entry maps}
2. {@section Configuring statically defined entries}
3. {@section Configuring static subscriptions}
4. {@section Configuring reporter plugins}
5. {@section Configuring opentsdb reporter}
6. {@section Configuring amqp reporter}
7. {@section Configuring graphite reporter}
8. {@section Configuring snmp reporter}
6. {@section Creating custom exometer entries}
7. {@section Creating custom probes}
8. {@section Creating custom reporter plugins}
9. {@section Dependency management}
== Concepts and Definitions ==
Exometer introduces a number of concepts and definitions used
throughout the documentation and the code.
=== Metric ===
A metric is a specific measurement sampled inside an Erlang system and
then reported to the Exometer system. An example metric would be
"transactions_per_second", or "memory_usage".
Metrics are identified by a list of terms, such as given below:
`[ xml_front_end, parser, file_size ]'
A metric is created through a call by the code to be instrumented to
`exometer:new()'. Once created, the metric can be updated through
`exometer:update()', or on its own initiative through the
`exometer_probe:sample' behavior implementation.
=== Data Point ===
Each metric can consist of multiple data points, where each point has
a specific value.
A typical example of data points would be a
`transactions_per_second' (tps) metric, usually stored as a
histogram covering the last couple of minutes of tps samples. Such a
histogram would host multiple values, such as `min', `max',
`median', `mean', `50_percentile', `75_percentile',
etc.
It is up to the type of the metric, and the data probe backing that
type (see below), to specify which data points are available under the
given metric.
=== Metric Type ===
The type of a metric, specified when the metric is created through
`exometer:new()', determines which `exometer_entry'
callback to use.
The link between the type and the entry to use is configured
through the `exometer_admin' module, and its associated exometer
defaults configuration data.
The metric type, in other words, is mainly used to map a metric to a
configurable `exometer_entry' callback, but it can also be referenced
in queries using `exometer:select/1'. An entry callback can also support
multiple types (the type is provided as an argument in the callback functions).
Exometer provides default mappings for a number of metric types. It is
possible to select different callbacks for each metric instance, as well
as modify metrics using callback-specific options. Please see
{@section Configuring type - entry maps} for details on how to do this.
=== Entry Callback ===
An exometer entry callback will receive values reported to a metric through the
`exometer:update()' call and compile it into one or more data points.
The entry callback can either be a counter (implemented natively
in `exometer'), or a more complex statistical analysis such
as a uniform distribution or a regular histogram.
The various outputs from these entries are reported as data points
under the given metric.
An entry can also interface external analytics packages.
`exometer_folsom', for example, integrates with the
`folsom_metrics' package found at [https://github.com/boundary/folsom].
=== Probe ===
Probes are a further specialization of exometer entries that run in
their own Erlang processes and have their own state (like a
gen_server). A probe is implemented through the `exometer_probe'
behavior.
A probe can be used if independent monitoring is needed of,
for example, `/proc' trees, network interfaces, and other subsystems
that need periodic sampling. In these cases, the
`exometer_probe:probe_sample()' call is invoked regularly by exometer,
in the probe's own process, in order to extract data from
the given subsystem and add it to the metric's data points.
=== Caching ===
Metric and data point values are read with the `exometer:get_value()'
function. In the case of counters, this operation is very fast. With probes,
the call results in a synchronous dialog with the probe process, and the
cost of serving the request depends on the probe implementation and the
nature of the metric being served.
If the cost of reading the value is so high that calling the function often
would result in prohibitive load, it is possible to cache the value. This is
done either explicitly from the probe itself (by calling
`exometer_cache:write()'), or by specifying the option `{cache, Lifetime}'
for the entry. If an entry has a non-zero cache lifetime specified, the
`get_value()' call will try fetching the cached value before calling the
actual entry and automatically caching the result.
Note that if `{cache, Lifetime}' is not specified, `exometer:get_value()'
will neither read nor write to the cache. It is possible for the probe
to periodically cache a value regardless of how the cache lifetime is set,
and the probe may also explicitly read from the cache if it isn't done
automatically.
=== Subscriptions and Reporters ===
The subscription concept, managed by `exometer_report' allows metrics
and their data points to be sampled at given intervals and delivered
to one or more recipients, which can be either an arbitrary process
or a Reporter plugin.
Each subscription ties a specific metric-datapoint pair to a reporter
and an interval (given in milliseconds). The reporter system will, at
the given interval, send the current value of the data point to the
subscribing reporter. The subscription, with all its parameters,
is setup through a call to `exometer_report:subscribe()'.
In the case of processes, subscribed-to values will be delivered as a
message. Modules, which implement the `exometer_report' callback
behavior, will receive the plugins as a callbacks within the
`exometer_report' process.
Subscriptions can either be setup at runtime, through
`exometer_report:subscribe()' calls, or statically through the
`exometer_report' configuration data.
== Built-in entries and probes ==
There are a number of built-in entries and probes shipped
with the Exometer package, as described below:
=== counter (exometer native) ===
The counter is implemented directly in `exometer' to provide simple
counters. A call to `exometer:update()' will add the provided value
to the counter.
The counter can be reset to zero through `exometer:reset()'.
The available data points under a metric using the counter entry
are `value' and `ms_since_reset'.
=== fast_counter (exometer native) ===
A fast counter implements the counter functionality, through the
`trace_info' system, yielding a speed increase of about 3.5 in
comparison to the regular counter.
The tradeoff is that running tracing and/or debugging may interfere
with the counter functionality.
A call to `exometer:update()' will add the provided value to the
counter.
The counter can be reset to zero through `exometer:reset()'.
The available data points under a metric using the fast_counter
entry are `value' and `ms_since_reset'.
=== gauge (exometer native) ===
The gauge is implemented directly in `exometer' to provide simple
gauges. A call to `exometer:update()' will set the gauge's value
to the provided value. That is, the value of the gauge entry is
always the most recently provided value.
The gauge can be reset to zero through `exometer:reset()'.
The available data points under a metric using the gauge entry
are `value' and `ms_since_reset'.
=== histogram (probe) ===
The histogram probe stores a given number of updates, provided through
`exometer:update()', in a histogram. The histogram maintains a log
derived from all values received during a configurable time span and
provides min, max, median, mean, and percentile analysis data points
for the stored data.
Exometer supports a number of different histogram implementations, each
with different performance and accuracy trade-offs.
In order to save memory, the histogram is divided into equal-sized
time slots, where each slot spans a settable interval. All values
received during a time slot will be averaged into a single value to be
stored in the histogram once the time slot expires. The averaging
function (which can be replaced by the caller), allows for
high-frequency update metrics to have their resolution traded against
resource consumption.
=== exometer_uniform (probe) ===
The uniform probe provides a uniform sample over a pool of values
provided through `exometer:update()'. When the pool reaches its configurable
max size, existing values will be replaced at random to make space for
new values. Much like `exometer_histogram', the uniform probe
provides min, max, median, mean, and percentile analysis data points
for the stored data.
=== exometer_spiral (probe) ===
The spiral probe maintains the total sum of all values stored in its
histogram. The histogram has a configurable time span, all values
provided to the probe, through `exometer:update()', within that time
span will be summed up and reported. If, for example, the histogram
covers 60 seconds, the spiral probe will report the sum of all
values reported during the last minute.
The grand total of all values received during the lifetime of the
probe is also available.
=== exometer_folsom [entry] ===
`exometer_folsom' is an entry behavior which implements most metric types
supported by the folsom
metrics package: Specifically, the metric types `counter', `spiral',
`histogram', `meter', `meter_reader', `gauge', `duration' and `history'.
The folsom entry integrates with the folsom metrics package provided
by the boundary repo at github. Updated values sent to the folsom entry
can be forwarded to folsom's counter, histogram, duration, meter,
and spiral.
Folsom integration is provided as a backup. New code using Exometer
should use the native probes that duplicate folsom.
=== exometer_function [entry] ===
The function entry allows for an existing erlang function to be wrapped
as an exometer entry. The {@link exometer_function} module supports a number
of options for passing arguments and matching out data points from the
result.
The function entry provides an easy way of integrating an external
system without having to write a complete entry.
== Built in Reporters ==
Exometer ships with some built-in reporters which can be used to forward updated
metrics and their data points to external systems. They can also
serve as templates for custom-developed reporters.
=== exometer_report_graphite ===
The graphite reporter uses the TCP/IP protocol to forward
subscribed-to metrics and data points to a graphite server, such as
the one provided by [http://hostedgraphite.com]. When the graphite
reporter receives a metric-datapoint value (subscribed to through
`exometer_report:subscriber()'), the reporter will immediately
forward the key-value pair to the graphite server.
=== exometer_report_opentsdb ===
The OpenTSDB reporter sends metrics to an OpenTSDB server using
the telnet API. All subscribed-to metric-datapoint values received
by the reporter are immediately forwarded to OpenTSDB.
If the OpenTSDB connection is lost, the reporter will attempt to reconnect to it
at a configurable interval.
The data sent to OpenTSDB will be formatted as follows:
put metric timestamp value host=host type=datapoint
Where the value for the host tag will be the configured host in the reporter
configuration (defaults to the value returned by `netadm:localhost'), and
datapoint tags as specified by the subscriber.
Please see {@section Configuring opentsdb reporter} for details on the
application environment parameters listed above.
=== exometer_report_amqp ===
The AMQP reporter sends metrics to an AMQP broker as a json-encoded payload. All
subscribed-to metric-datapoint values received by the reporter are forwarded to AMQP.
If the AMQP connection is lost, the reporter will attempt to reconnect to it
at a configurable interval.
The data sent to AMQP will be formatted as follows:
Where the value for the host tag will be the configured host in the reporter
configuration (defaults to the value returned by `netadm:localhost'), the
instance tag represents the datapoint for the metric.
Please see {@section Configuring amqp reporter} for details on the
application environment parameters listed above.
=== exometer_report_snmp ===
The SNMP reporter enables the export of metrics and their datapoints to SNMP managers.
The export needs to be enabled for each metric through their options.
Moreover, SNMP notifications can be created using the options to send periodic reports
on datapoints to SNMP managers. All SNMP protocol handling is done by the snmp application
shipped with Erlang/OTP. Thus, the snmp application needs to be started and
the local SNMP master agent needs to be configured correctly for SNMP export to work
properly.
To configure SNMP export for a single metric use these options:
+ `{snmp, disabled}' (default)
Disables SNMP export for the metric. Same as not specifying the option at all.
+ `{snmp, []}'
Enables SNMP export for the metric. No subscriptions are setup.
+ `{snmp, [{Datapint, Interval}]}'
Enables SNMP export for the metric.
Subscriptions are setup for the given Datapoint/Interval pairs.
Each subscription report will be forwarded to SNMP mangers as notifications.
+ `{snmp, [{Datapint, Interval, Extra}]}'
Same as above, but using an addition extra identification for the subscriptions.
Allow the creation ofmultiple subscriptions for a single datapoint.
Please see {@section Configuring snmp reporter} for details on how to configure the
SNMP reporter.
== Instrumenting Erlang code ==
The code using Exometer needs to be instrumented in order to setup and
use metrics reporting.
=== Exometer Start ===
The system using Exometer must start the `exometer' application prior to using it:
Note that dependent applications need to be started first. On newer OTP versions
(R16B or later), you can use `application:ensure_all_started(exometer)'.
For testing, you can also use {@link exometer:start/0}.
If you make use of e.g. folsom metrics, you also need to start `folsom'. Exometer
will not do that automatically, nor does it contain an application dependency for it.
See {@section Configuring Exometer} for details on configuration data
format.
=== Creating metrics ===
A metric, can be created through a call to
exometer:new(Name, Type)
`Name' is a list of atoms, uniquely identifying the metric created.
The type of the metric, specified by `Type' will be mapped
to an exometer entry through the table maintained by
`exometer_admin' Please see the {@section Configuring type - entry
maps} for details.
The resolved entry to use will determine the data points available
under the given metric.
=== Deleting metrics ===
A metric previously created with `exometer:new()' can be deleted by
`exometer:delete()'.
All subscriptions to the deleted metrics will be cancelled.
=== Setting metric values ===
A created metric can have its value updated through the
`exometer:update()' function:
exometer:update(Name, Value)
The `Name' parameter is the same atom list provided to a previous
`exometer:new()' call. The `Value' is an arbitrarty element that is
forwarded to the `exometer:update()' function of the entry/probe that the
metric is mapped to.
The receiving entry/probe will process the provided value and modify
its data points accordingly.
=== Retrieving metric values ===
Exometer-using code can at any time retrieve the data point values
associated with a previously created metric. In order to find out which
data points are available for a metric, the following call can be used:
exometer:info(Name, datapoints)
The `Name' parameter is the same atom list provided to a previous
`exometer:new()' call. The call will return a list of data point
atoms that can then be provided to `exometer:get_value()' to
retrieve their actual value:
exometer:get_value(Name, DataPoint)
The `Name' paramer identifies the metric, and `DataPoints'
identifies the data points (returned from the previous `info()' call)
to retrieve the value for.
If no DataPoints are provided, the values of a default list of data points,
determined by the backing entry / probe, will be returned.
=== Setting up subscriptions ===
A subscription can either be statically configured, or dynamically
setup from within the code using Exometer. For details on statically
configured subscriptions, please see {@section Configuring static subscriptions}.
A dynamic subscription can be setup with the following call:
`Recipient' is the name of a reporter.
=== Set metric options ===
Each created metric can have options setup for it through the following call:
exometer:setopts(Name, Options)
The `Name' paramer identifies the metric to set the options for, and
Options is a proplist (`[{ Key, Value },...]') with the options to be
set.
Exometer looks up the the backing entry that hosts the metric with the given Name, and will
invoke the entry\'s `setopts/4' function to set the actual options. Please see the
`setopts/4' function for the various entries for details.
== Configuring Exometer ==
Exometer defaults can be changed either through OTP application environment
variables or through the use of Basho's `cuttlefish'
([https://github.com/basho/cuttlefish]).
=== Configuring type - entry maps ===
The dynamic method of configuring defaults for `exometer' entries is:
Where `NamePattern' is a list of terms describing what is essentially
a name prefix with optional wildcards ('_'). A pattern that
matches any legal name is ['_'].
`Type' is an atom defining a type of metric. The types already known to
`exometer', `counter', `fast_counter', `ticker', `uniform', `histogram',
`spiral', `netlink', and `probe' may be redefined, but other types can be
described as well.
`Default' is either an `#exometer_entry{}' record (unlikely), or a list of
`{Key, Value}' options, where the keys correspond to `#exometer_entry' record
attribute names. The following attributes make sense to preset:
{module, atom()} % the callback module
{status, enabled | disabled} % operational status of the entry
{cache, non_neg_integer()} % cache lifetime (ms)
{options, [{atom(), any()}]} % entry-specific options
Below is an example, from `exometer/priv/app.config':
In systems that use CuttleFish, the file `exometer/priv/exometer.schema'
contains a schema for default settings. The setup corresponding to the above
defaults would be as follows:
=== Configuring statically defined entries ===
Using the `exometer' environment variable `predefined', entries can be added
at application startup. The variable should have one of the following values:
* `{script, File}' - `File' will be processed using `file:script/2'. The return
value (the result of the last expression in the script) should be a list of
`{Name, Type, Options}' tuples.
* `{apply, M, F, A}' - The result of `apply(M, F, A)' should be `{ok, L}' where
`L' is a list of `{Name, Type, Options}' tuples.
* `L', where L is a list of `{Name, Type, Options}' tuples or extended
instructions (see below).
The list of instructions may include:
* `{delete, Name}' - deletes `Name' from the exometer registry.
* `{select_delete, Pattern}' - applies a select pattern and
deletes all matching entries.
* `{re_register, {Name, Type, Options}}' - redefines an entry if present,
otherwise creates it.
Exometer will also scan all loaded applications for the environment
variables `exometer_defaults' and `exometer_predefined', and process
as above. If an application is loaded and started after exometer has started,
it may call the function `exometer:register_application()' or
`exometer:register_application(App)'. This function will do nothing if
exometer isn't already running, and otherwise process the `exometer_defaults'
and `exometer_predefined' variables as above. The function can also be
called during upgrade, as it will re-apply the settings each time.
=== Configuring static subscriptions ===
Static subscriptions, which are automatically setup at exometer
startup without having to invoke `exometer_report:subscribe()', are
configured through the report sub section under exometer.
Below is an example, from `exometer/priv/app.config':
The `report' section configures static subscriptions and reporter
plugins. See {@section Configuring reporter plugins} for details on
how to configure individual plugins.
The `subscribers' sub-section contains all static subscriptions to be
setup at exometer applications start. Each tuple in the prop list
should be of one of the following formats:
* `{Reporter, Metric, DataPoint, Interval}'
* `{Reporter, Metric, DataPoint, Interval, RetryFailedMetrics}'
* `{Reporter, Metric, DataPoint, Interval, RetryFailedMetrics, Extra}'
* `{apply, {M, F, A}}'
* `{select, {MatchPattern, DataPoint, Interval [, Retry [, Extra] ]}}'
In the case of `{apply, M, F, A}', the result of `apply(M, F, A)' must
be a list of `subscribers' tuples.
In the case of `{select, Expr}', a list of metrics is fetched using
`exometer:select(MatchPattern)', where the result must be on the form
`{Key, Type, Status}' (i.e. what corresponds to '$_').
The rest of the items will be applied to each of the matching entries.
The meaning of the above tuple elements is:
+ `Reporter :: module()'
Specifies the reporter plugin module, such as
`exometer_report_collectd' that is to receive updated metric's data
points.
+ `Metric :: [atoms()]'
Specifies the path to a metric previously created with an
`exometer:new()' call.
+ `DataPoint :: atom() | [atom()]'
Specifies the data point within the given metric to send to the
receiver. The data point must match one of the data points returned by
`exometer:info(Name, datapoints)' for the given metrics name.
+ `Interval :: integer()' (milliseconds)
Specifies the interval, in milliseconds, between each update of the
given metric's data point. At the given interval, the data point will
be samples, and the result will be sent to the receiver.
+ `RetryFailedMetrics :: boolean()'
Specifies if the metric should be continued to be reported
even if it is not found during a reporting cycle. This would be
the case if a metric is not created by the time it is reported for
the first time. If the metric will be created at a later time,
this value should be set to true. Set this value to false if all
attempts to report the metric should stop if when is not found.
The default value is `true'.
+ `Extra :: any()'
Provides a means to pass along extra information for a given
subscription. An example is the `syntax' option for the SNMP reporter,
in which case `Extra' needs to be a property list.
Example configuration in sys.config, using the `{select, Expr}' pattern:
This will activate a subscription on `[a,1]' and `[a,2]' in the
`exometer_report_tty' reporter, firing once per second.
=== Configuring reporter plugins ===
The various reporter plugins to be loaded by exometer are configured
in the `report' section under `reporters'
Each reporter has an entry named after its module, and the content of
that entry is dependent on the reporter itself. The following chapters
specifies the configuration parameters for the reporters shipped with
exometer.
=== Configuring opentsdb reporter ===
Below is an example of the opentsdb reporter application environment, with
its correct location in the hierarchy:
The following attributes are available for configuration:
+ `reconnect_interval' (seconds - default: 30)
Specifies the duration between each reconnect attempt to an opentsdb
server that is not available. Should the server either be unavailable
at exometer startup, or become unavailable during exometer's
operation, exometer will attempt to reconnect at the given number of
seconds.
+ `connect_timeout' (milliseconds - default: 5000)
Specifies how long the opentsdb reporter plugin shall wait for a
socket connection to complete before timing out. A timed out
connection attempt will be retried after the reconnect interval has
passed see item 1 above).
+ `hostname' (string - default: `net_adm:localhost()')
Specifies the host name to use for the host tag in the OpenTSDB tags.
Please see {@section Configuring opentsdb reporter} for details.
+ `host' (ip - default: `{"127.0.0.1", 4242}')
Specifies the host and port to connect to OpenTSDB.
=== Configuring amqp reporter ===
Below is an example of the amqp reporter application environment, with
its correct location in the hierarchy:
The following attributes are available for configuration:
+ `reconnect_interval' (seconds - default: 30)
Specifies the duration between each reconnect attempt to an amqp
broker that is not available. Should the server either be unavailable
at exometer startup, or become unavailable during exometer's
operation, exometer will attempt to reconnect at the given number of
seconds.
+ `hostname' (string - default: `net_adm:localhost()')
Specifies the host name to use for the host property in the JSON payload.
Please see {@section Configuring amqp reporter} for details.
+ `amqp_url' (string - default: `amqp://guest:guest@localhost:5672/%2f')
Specifies the amqp url to connect to.
+ `exchange' (string - default: `exometer')
Specifies the exchange to publish messages to.
+ `routing_key' (string - default: `exometer')
Specifies the routing key to use when publishing messages.
+ `buffer_size' (integer - default: `0')
Specifies the size in bytes of payload to buffer before sending to AMQP.
=== Configuring graphite reporter ===
Below is an example of the a graphite reporter application environment, with
its correct location in the hierarchy:
The following attributes are available for configuration:
+ `connect_timeout' (milliseconds - default: 5000)
Specifies how long the graphie reporter plugin shall wait for a tcp
connection to complete before timing out. A timed out connection will
not be reconnected to automatically. (To be fixed.)
+ `prefix' (string - default: "")
Specifies an optional prefix to prepend all metric names with before
they are sent to the graphite server.
+ `host' (string - default: "carbon.hostedgraphite.com")
Specifies the name (or IP address) of the graphite server to report to.
+ `port' (integer - default: 2003)
Specifies the TCP port on the given graphite server to connect to.
+ `api_key' (string - default: n/a)
Specifies the api key to use when reporting to a hosted graphite server.
If `prefix' is not specified, but `api_key' is, each metrics will be reported as `ApiKey.Metric'.
If `prefix' is specified, but `api_key' is not, each metrics will be reported as `Prefix.Metric'.
if neither `prefix' or `api_key' is specified, each metric will be reported simply as `Metric'.
=== Configuring snmp reporter ===
Below is an example of the a snmp reporter application environment, with
its correct location in the hierarchy:
=== Metric ===
A metric is a specific measurement sampled inside an Erlang system and
then reported to the Exometer system. An example metric would be
"transactions_per_second", or "memory_usage".
Metrics are identified by a list of terms, such as given below:
`[ xml_front_end, parser, file_size ]'
A metric is created through a call by the code to be instrumented to
`exometer:new()'. Once created, the metric can be updated through
`exometer:update()', or on its own initiative through the
`exometer_probe:sample' behavior implementation.
=== Data Point ===
Each metric can consist of multiple data points, where each point has
a specific value.
A typical example of data points would be a
`transactions_per_second' (tps) metric, usually stored as a
histogram covering the last couple of minutes of tps samples. Such a
histogram would host multiple values, such as `min', `max',
`median', `mean', `50_percentile', `75_percentile',
etc.
It is up to the type of the metric, and the data probe backing that
type (see below), to specify which data points are available under the
given metric.
=== Metric Type ===
The type of a metric, specified when the metric is created through
`exometer:new()', determines which `exometer_entry'
callback to use.
The link between the type and the entry to use is configured
through the `exometer_admin' module, and its associated exometer
defaults configuration data.
The metric type, in other words, is mainly used to map a metric to a
configurable `exometer_entry' callback, but it can also be referenced
in queries using `exometer:select/1'. An entry callback can also support
multiple types (the type is provided as an argument in the callback functions).
Exometer provides default mappings for a number of metric types. It is
possible to select different callbacks for each metric instance, as well
as modify metrics using callback-specific options. Please see
{@section Configuring type - entry maps} for details on how to do this.
=== Entry Callback ===
An exometer entry callback will receive values reported to a metric through the
`exometer:update()' call and compile it into one or more data points.
The entry callback can either be a counter (implemented natively
in `exometer'), or a more complex statistical analysis such
as a uniform distribution or a regular histogram.
The various outputs from these entries are reported as data points
under the given metric.
An entry can also interface external analytics packages.
`exometer_folsom', for example, integrates with the
`folsom_metrics' package found at [https://github.com/boundary/folsom].
=== Probe ===
Probes are a further specialization of exometer entries that run in
their own Erlang processes and have their own state (like a
gen_server). A probe is implemented through the `exometer_probe'
behavior.
A probe can be used if independent monitoring is needed of,
for example, `/proc' trees, network interfaces, and other subsystems
that need periodic sampling. In these cases, the
`exometer_probe:probe_sample()' call is invoked regularly by exometer,
in the probe's own process, in order to extract data from
the given subsystem and add it to the metric's data points.
=== Caching ===
Metric and data point values are read with the `exometer:get_value()'
function. In the case of counters, this operation is very fast. With probes,
the call results in a synchronous dialog with the probe process, and the
cost of serving the request depends on the probe implementation and the
nature of the metric being served.
If the cost of reading the value is so high that calling the function often
would result in prohibitive load, it is possible to cache the value. This is
done either explicitly from the probe itself (by calling
`exometer_cache:write()'), or by specifying the option `{cache, Lifetime}'
for the entry. If an entry has a non-zero cache lifetime specified, the
`get_value()' call will try fetching the cached value before calling the
actual entry and automatically caching the result.
Note that if `{cache, Lifetime}' is not specified, `exometer:get_value()'
will neither read nor write to the cache. It is possible for the probe
to periodically cache a value regardless of how the cache lifetime is set,
and the probe may also explicitly read from the cache if it isn't done
automatically.
=== Subscriptions and Reporters ===
The subscription concept, managed by `exometer_report' allows metrics
and their data points to be sampled at given intervals and delivered
to one or more recipients, which can be either an arbitrary process
or a Reporter plugin.
Each subscription ties a specific metric-datapoint pair to a reporter
and an interval (given in milliseconds). The reporter system will, at
the given interval, send the current value of the data point to the
subscribing reporter. The subscription, with all its parameters,
is setup through a call to `exometer_report:subscribe()'.
In the case of processes, subscribed-to values will be delivered as a
message. Modules, which implement the `exometer_report' callback
behavior, will receive the plugins as a callbacks within the
`exometer_report' process.
Subscriptions can either be setup at runtime, through
`exometer_report:subscribe()' calls, or statically through the
`exometer_report' configuration data.
== Built-in entries and probes ==
There are a number of built-in entries and probes shipped
with the Exometer package, as described below:
=== counter (exometer native) ===
The counter is implemented directly in `exometer' to provide simple
counters. A call to `exometer:update()' will add the provided value
to the counter.
The counter can be reset to zero through `exometer:reset()'.
The available data points under a metric using the counter entry
are `value' and `ms_since_reset'.
=== fast_counter (exometer native) ===
A fast counter implements the counter functionality, through the
`trace_info' system, yielding a speed increase of about 3.5 in
comparison to the regular counter.
The tradeoff is that running tracing and/or debugging may interfere
with the counter functionality.
A call to `exometer:update()' will add the provided value to the
counter.
The counter can be reset to zero through `exometer:reset()'.
The available data points under a metric using the fast_counter
entry are `value' and `ms_since_reset'.
=== gauge (exometer native) ===
The gauge is implemented directly in `exometer' to provide simple
gauges. A call to `exometer:update()' will set the gauge's value
to the provided value. That is, the value of the gauge entry is
always the most recently provided value.
The gauge can be reset to zero through `exometer:reset()'.
The available data points under a metric using the gauge entry
are `value' and `ms_since_reset'.
=== histogram (probe) ===
The histogram probe stores a given number of updates, provided through
`exometer:update()', in a histogram. The histogram maintains a log
derived from all values received during a configurable time span and
provides min, max, median, mean, and percentile analysis data points
for the stored data.
Exometer supports a number of different histogram implementations, each
with different performance and accuracy trade-offs.
In order to save memory, the histogram is divided into equal-sized
time slots, where each slot spans a settable interval. All values
received during a time slot will be averaged into a single value to be
stored in the histogram once the time slot expires. The averaging
function (which can be replaced by the caller), allows for
high-frequency update metrics to have their resolution traded against
resource consumption.
=== exometer_uniform (probe) ===
The uniform probe provides a uniform sample over a pool of values
provided through `exometer:update()'. When the pool reaches its configurable
max size, existing values will be replaced at random to make space for
new values. Much like `exometer_histogram', the uniform probe
provides min, max, median, mean, and percentile analysis data points
for the stored data.
=== exometer_spiral (probe) ===
The spiral probe maintains the total sum of all values stored in its
histogram. The histogram has a configurable time span, all values
provided to the probe, through `exometer:update()', within that time
span will be summed up and reported. If, for example, the histogram
covers 60 seconds, the spiral probe will report the sum of all
values reported during the last minute.
The grand total of all values received during the lifetime of the
probe is also available.
=== exometer_folsom [entry] ===
`exometer_folsom' is an entry behavior which implements most metric types
supported by the folsom
metrics package: Specifically, the metric types `counter', `spiral',
`histogram', `meter', `meter_reader', `gauge', `duration' and `history'.
The folsom entry integrates with the folsom metrics package provided
by the boundary repo at github. Updated values sent to the folsom entry
can be forwarded to folsom's counter, histogram, duration, meter,
and spiral.
Folsom integration is provided as a backup. New code using Exometer
should use the native probes that duplicate folsom.
=== exometer_function [entry] ===
The function entry allows for an existing erlang function to be wrapped
as an exometer entry. The {@link exometer_function} module supports a number
of options for passing arguments and matching out data points from the
result.
The function entry provides an easy way of integrating an external
system without having to write a complete entry.
== Built in Reporters ==
Exometer ships with some built-in reporters which can be used to forward updated
metrics and their data points to external systems. They can also
serve as templates for custom-developed reporters.
=== exometer_report_graphite ===
The graphite reporter uses the TCP/IP protocol to forward
subscribed-to metrics and data points to a graphite server, such as
the one provided by [http://hostedgraphite.com]. When the graphite
reporter receives a metric-datapoint value (subscribed to through
`exometer_report:subscriber()'), the reporter will immediately
forward the key-value pair to the graphite server.
=== exometer_report_opentsdb ===
The OpenTSDB reporter sends metrics to an OpenTSDB server using
the telnet API. All subscribed-to metric-datapoint values received
by the reporter are immediately forwarded to OpenTSDB.
If the OpenTSDB connection is lost, the reporter will attempt to reconnect to it
at a configurable interval.
The data sent to OpenTSDB will be formatted as follows: