\n\n* [Brian Troutwine](https://github.com/Feuerlabs/exometer/pulls/blt?q=is%3Aclosed)\n\n* [Tino Breddin](https://github.com/Feuerlabs/exometer/pulls/tolbrino?q=is%3Aclosed)\n\n* [Artem Teslenko](https://github.com/Feuerlabs/exometer/pulls/ates?q=is%3Aclosed)\n\n* [Sébastien Merle](https://github.com/Feuerlabs/exometer/pulls/sylane?q=is%3Aclosed)\n\n* [Diana Corbacho](https://github.com/Feuerlabs/exometer/pulls/dcorbacho?q=is%3Aclosed)\n\n* [Roland Karlsson](https://github.com/Feuerlabs/exometer/pulls/roland-karlsson-erlang-solutions-com?q=is%3Aclosed)\n\n* [Aaron France](https://github.com/Feuerlabs/exometer/pulls/aeronotix?q=is%3Aclosed)\n\n* [Maas-Maarten Zeeman](https://github.com/Feuerlabs/exometer/pulls/mmzeeman?q=is%3Aclosed)\n\n* [Vladimir G. Sekissov](https://github.com/Feuerlabs/exometer/pulls/eryx67?q=is%3Aclosed)\n\n* [Mark Steele](https://github.com/Feuerlabs/exometer/pulls/marksteele?q=is%3Aclosed)\n"
},
{
"path": "LICENSE",
"content": "Mozilla Public License, version 2.0\n\n1. Definitions\n\n1.1. “Contributor”\n\n means each individual or legal entity that creates, contributes to the\n creation of, or owns Covered Software.\n\n1.2. “Contributor Version”\n\n means the combination of the Contributions of others (if any) used by a\n Contributor and that particular Contributor’s Contribution.\n\n1.3. “Contribution”\n\n means Covered Software of a particular Contributor.\n\n1.4. “Covered Software”\n\n means Source Code Form to which the initial Contributor has attached the\n notice in Exhibit A, the Executable Form of such Source Code Form, and\n Modifications of such Source Code Form, in each case including portions\n thereof.\n\n1.5. “Incompatible With Secondary Licenses”\n means\n\n a. that the initial Contributor has attached the notice described in\n Exhibit B to the Covered Software; or\n\n b. that the Covered Software was made available under the terms of version\n 1.1 or earlier of the License, but not also under the terms of a\n Secondary License.\n\n1.6. “Executable Form”\n\n means any form of the work other than Source Code Form.\n\n1.7. “Larger Work”\n\n means a work that combines Covered Software with other material, in a separate\n file or files, that is not Covered Software.\n\n1.8. “License”\n\n means this document.\n\n1.9. “Licensable”\n\n means having the right to grant, to the maximum extent possible, whether at the\n time of the initial grant or subsequently, any and all of the rights conveyed by\n this License.\n\n1.10. “Modifications”\n\n means any of the following:\n\n a. any file in Source Code Form that results from an addition to, deletion\n from, or modification of the contents of Covered Software; or\n\n b. any new file in Source Code Form that contains any Covered Software.\n\n1.11. “Patent Claims” of a Contributor\n\n means any patent claim(s), including without limitation, method, process,\n and apparatus claims, in any patent Licensable by such Contributor that\n would be infringed, but for the grant of the License, by the making,\n using, selling, offering for sale, having made, import, or transfer of\n either its Contributions or its Contributor Version.\n\n1.12. “Secondary License”\n\n means either the GNU General Public License, Version 2.0, the GNU Lesser\n General Public License, Version 2.1, the GNU Affero General Public\n License, Version 3.0, or any later versions of those licenses.\n\n1.13. “Source Code Form”\n\n means the form of the work preferred for making modifications.\n\n1.14. “You” (or “Your”)\n\n means an individual or a legal entity exercising rights under this\n License. For legal entities, “You” includes any entity that controls, is\n controlled by, or is under common control with You. For purposes of this\n definition, “control” means (a) the power, direct or indirect, to cause\n the direction or management of such entity, whether by contract or\n otherwise, or (b) ownership of more than fifty percent (50%) of the\n outstanding shares or beneficial ownership of such entity.\n\n\n2. License Grants and Conditions\n\n2.1. Grants\n\n Each Contributor hereby grants You a world-wide, royalty-free,\n non-exclusive license:\n\n a. under intellectual property rights (other than patent or trademark)\n Licensable by such Contributor to use, reproduce, make available,\n modify, display, perform, distribute, and otherwise exploit its\n Contributions, either on an unmodified basis, with Modifications, or as\n part of a Larger Work; and\n\n b. under Patent Claims of such Contributor to make, use, sell, offer for\n sale, have made, import, and otherwise transfer either its Contributions\n or its Contributor Version.\n\n2.2. Effective Date\n\n The licenses granted in Section 2.1 with respect to any Contribution become\n effective for each Contribution on the date the Contributor first distributes\n such Contribution.\n\n2.3. Limitations on Grant Scope\n\n The licenses granted in this Section 2 are the only rights granted under this\n License. No additional rights or licenses will be implied from the distribution\n or licensing of Covered Software under this License. Notwithstanding Section\n 2.1(b) above, no patent license is granted by a Contributor:\n\n a. for any code that a Contributor has removed from Covered Software; or\n\n b. for infringements caused by: (i) Your and any other third party’s\n modifications of Covered Software, or (ii) the combination of its\n Contributions with other software (except as part of its Contributor\n Version); or\n\n c. under Patent Claims infringed by Covered Software in the absence of its\n Contributions.\n\n This License does not grant any rights in the trademarks, service marks, or\n logos of any Contributor (except as may be necessary to comply with the\n notice requirements in Section 3.4).\n\n2.4. Subsequent Licenses\n\n No Contributor makes additional grants as a result of Your choice to\n distribute the Covered Software under a subsequent version of this License\n (see Section 10.2) or under the terms of a Secondary License (if permitted\n under the terms of Section 3.3).\n\n2.5. Representation\n\n Each Contributor represents that the Contributor believes its Contributions\n are its original creation(s) or it has sufficient rights to grant the\n rights to its Contributions conveyed by this License.\n\n2.6. Fair Use\n\n This License is not intended to limit any rights You have under applicable\n copyright doctrines of fair use, fair dealing, or other equivalents.\n\n2.7. Conditions\n\n Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in\n Section 2.1.\n\n\n3. Responsibilities\n\n3.1. Distribution of Source Form\n\n All distribution of Covered Software in Source Code Form, including any\n Modifications that You create or to which You contribute, must be under the\n terms of this License. You must inform recipients that the Source Code Form\n of the Covered Software is governed by the terms of this License, and how\n they can obtain a copy of this License. You may not attempt to alter or\n restrict the recipients’ rights in the Source Code Form.\n\n3.2. Distribution of Executable Form\n\n If You distribute Covered Software in Executable Form then:\n\n a. such Covered Software must also be made available in Source Code Form,\n as described in Section 3.1, and You must inform recipients of the\n Executable Form how they can obtain a copy of such Source Code Form by\n reasonable means in a timely manner, at a charge no more than the cost\n of distribution to the recipient; and\n\n b. You may distribute such Executable Form under the terms of this License,\n or sublicense it under different terms, provided that the license for\n the Executable Form does not attempt to limit or alter the recipients’\n rights in the Source Code Form under this License.\n\n3.3. Distribution of a Larger Work\n\n You may create and distribute a Larger Work under terms of Your choice,\n provided that You also comply with the requirements of this License for the\n Covered Software. If the Larger Work is a combination of Covered Software\n with a work governed by one or more Secondary Licenses, and the Covered\n Software is not Incompatible With Secondary Licenses, this License permits\n You to additionally distribute such Covered Software under the terms of\n such Secondary License(s), so that the recipient of the Larger Work may, at\n their option, further distribute the Covered Software under the terms of\n either this License or such Secondary License(s).\n\n3.4. Notices\n\n You may not remove or alter the substance of any license notices (including\n copyright notices, patent notices, disclaimers of warranty, or limitations\n of liability) contained within the Source Code Form of the Covered\n Software, except that You may alter any license notices to the extent\n required to remedy known factual inaccuracies.\n\n3.5. Application of Additional Terms\n\n You may choose to offer, and to charge a fee for, warranty, support,\n indemnity or liability obligations to one or more recipients of Covered\n Software. However, You may do so only on Your own behalf, and not on behalf\n of any Contributor. You must make it absolutely clear that any such\n warranty, support, indemnity, or liability obligation is offered by You\n alone, and You hereby agree to indemnify every Contributor for any\n liability incurred by such Contributor as a result of warranty, support,\n indemnity or liability terms You offer. You may include additional\n disclaimers of warranty and limitations of liability specific to any\n jurisdiction.\n\n4. Inability to Comply Due to Statute or Regulation\n\n If it is impossible for You to comply with any of the terms of this License\n with respect to some or all of the Covered Software due to statute, judicial\n order, or regulation then You must: (a) comply with the terms of this License\n to the maximum extent possible; and (b) describe the limitations and the code\n they affect. Such description must be placed in a text file included with all\n distributions of the Covered Software under this License. Except to the\n extent prohibited by statute or regulation, such description must be\n sufficiently detailed for a recipient of ordinary skill to be able to\n understand it.\n\n5. Termination\n\n5.1. The rights granted under this License will terminate automatically if You\n fail to comply with any of its terms. However, if You become compliant,\n then the rights granted under this License from a particular Contributor\n are reinstated (a) provisionally, unless and until such Contributor\n explicitly and finally terminates Your grants, and (b) on an ongoing basis,\n if such Contributor fails to notify You of the non-compliance by some\n reasonable means prior to 60 days after You have come back into compliance.\n Moreover, Your grants from a particular Contributor are reinstated on an\n ongoing basis if such Contributor notifies You of the non-compliance by\n some reasonable means, this is the first time You have received notice of\n non-compliance with this License from such Contributor, and You become\n compliant prior to 30 days after Your receipt of the notice.\n\n5.2. If You initiate litigation against any entity by asserting a patent\n infringement claim (excluding declaratory judgment actions, counter-claims,\n and cross-claims) alleging that a Contributor Version directly or\n indirectly infringes any patent, then the rights granted to You by any and\n all Contributors for the Covered Software under Section 2.1 of this License\n shall terminate.\n\n5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user\n license agreements (excluding distributors and resellers) which have been\n validly granted by You or Your distributors under this License prior to\n termination shall survive termination.\n\n6. Disclaimer of Warranty\n\n Covered Software is provided under this License on an “as is” basis, without\n warranty of any kind, either expressed, implied, or statutory, including,\n without limitation, warranties that the Covered Software is free of defects,\n merchantable, fit for a particular purpose or non-infringing. The entire\n risk as to the quality and performance of the Covered Software is with You.\n Should any Covered Software prove defective in any respect, You (not any\n Contributor) assume the cost of any necessary servicing, repair, or\n correction. This disclaimer of warranty constitutes an essential part of this\n License. No use of any Covered Software is authorized under this License\n except under this disclaimer.\n\n7. Limitation of Liability\n\n Under no circumstances and under no legal theory, whether tort (including\n negligence), contract, or otherwise, shall any Contributor, or anyone who\n distributes Covered Software as permitted above, be liable to You for any\n direct, indirect, special, incidental, or consequential damages of any\n character including, without limitation, damages for lost profits, loss of\n goodwill, work stoppage, computer failure or malfunction, or any and all\n other commercial damages or losses, even if such party shall have been\n informed of the possibility of such damages. This limitation of liability\n shall not apply to liability for death or personal injury resulting from such\n party’s negligence to the extent applicable law prohibits such limitation.\n Some jurisdictions do not allow the exclusion or limitation of incidental or\n consequential damages, so this exclusion and limitation may not apply to You.\n\n8. Litigation\n\n Any litigation relating to this License may be brought only in the courts of\n a jurisdiction where the defendant maintains its principal place of business\n and such litigation shall be governed by laws of that jurisdiction, without\n reference to its conflict-of-law provisions. Nothing in this Section shall\n prevent a party’s ability to bring cross-claims or counter-claims.\n\n9. Miscellaneous\n\n This License represents the complete agreement concerning the subject matter\n hereof. If any provision of this License is held to be unenforceable, such\n provision shall be reformed only to the extent necessary to make it\n enforceable. Any law or regulation which provides that the language of a\n contract shall be construed against the drafter shall not be used to construe\n this License against a Contributor.\n\n\n10. Versions of the License\n\n10.1. New Versions\n\n Mozilla Foundation is the license steward. Except as provided in Section\n 10.3, no one other than the license steward has the right to modify or\n publish new versions of this License. Each version will be given a\n distinguishing version number.\n\n10.2. Effect of New Versions\n\n You may distribute the Covered Software under the terms of the version of\n the License under which You originally received the Covered Software, or\n under the terms of any subsequent version published by the license\n steward.\n\n10.3. Modified Versions\n\n If you create software not governed by this License, and you want to\n create a new license for such software, you may create and use a modified\n version of this License if you rename the license and remove any\n references to the name of the license steward (except to note that such\n modified license differs from this License).\n\n10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses\n If You choose to distribute Source Code Form that is Incompatible With\n Secondary Licenses under the terms of this version of the License, the\n notice described in Exhibit B of this License must be attached.\n\nExhibit A - Source Code Form License Notice\n\n This Source Code Form is subject to the\n terms of the Mozilla Public License, v.\n 2.0. If a copy of the MPL was not\n distributed with this file, You can\n obtain one at\n http://mozilla.org/MPL/2.0/.\n\nIf it is not possible or desirable to put the notice in a particular file, then\nYou may include the notice in a location (such as a LICENSE file in a relevant\ndirectory) where a recipient would be likely to look for such a notice.\n\nYou may add additional accurate notices of copyright ownership.\n\nExhibit B - “Incompatible With Secondary Licenses” Notice\n\n This Source Code Form is “Incompatible\n With Secondary Licenses”, as defined by\n the Mozilla Public License, v. 2.0.\n\n"
},
{
"path": "Makefile",
"content": ".PHONY: all clean clean_plt deps compile test doc dialyzer xref devnode_snmp_agent devnode_snmp_manager compile_examples ci\n\nEXOMETER_PLT=exometer.plt\nDIALYZER_OPTS = # -Wunderspecs\nDIALYZER_APPS = erts kernel stdlib compiler syntax_tools snmp ssl ssh \\\n\t\tcrypto public_key test_server webtool xmerl common_test \\\n\t\tlager goldrush afunix netlink folsom mnesia parse_trans \\\n\t\tsetup exometer_core\n\nall: deps compile\n\nci: compile xref dialyzer test\n\ndeps:\n\trebar get-deps\n\ncompile:\n\trebar compile\n\nclean: clean_plt\n\trebar clean\n\nclean-all: clean\n\trm -rf deps\n\ntest: compile_examples\n\tERL_LIBS=./examples rebar ct skip_deps=true\n\nxref:\n\tERL_LIBS=./deps rebar xref skip_deps=true\n\nedown_deps:\n\trebar get-deps compile edown=true\n\ndoc: edown_deps\n\trebar doc edown=true skip_deps=true\n\n$(EXOMETER_PLT):\n\trebar get-deps compile\n\tERL_LIBS=deps dialyzer --build_plt --output_plt $(EXOMETER_PLT) \\\n\t--apps $(DIALYZER_APPS)\n\nclean_plt:\n\trm -f $(EXOMETER_PLT)\n\ndialyzer: deps compile $(EXOMETER_PLT)\n\tdialyzer -r ebin --plt $(EXOMETER_PLT) $(DIALYZER_OPTS)\n\ncompile_examples:\n\terlc +'{parse_transform, lager_transform}' -pz deps/lager/ebin -I src -o examples/snmp_manager/ examples/snmp_manager/*.erl\n\ndevnode_snmp_agent:\n\terl -sname agent -pa deps/*/ebin ebin -config examples/snmp_agent/sys.config -boot start_sasl -s lager -s crypto -s exometer\n\ndevnode_snmp_manager: compile_examples\n\terl -sname manager -pz examples/snmp_manager -pz deps/*/ebin ebin -config examples/snmp_manager/sys.config \\\n\t\t-boot start_sasl -s lager -s crypto -s snmp\n"
},
{
"path": "README.md",
"content": "\n\n# Exometer - Erlang instrumentation package #\n\nCopyright (c) 2014 Basho Technologies, Inc. All Rights Reserved.\n\n__Version:__ Feb 1 2015 23:02:37\n\n__Authors:__ Ulf Wiger ([`ulf.wiger@feuerlabs.com`](mailto:ulf.wiger@feuerlabs.com)), Magnus Feuer ([`magnus.feuer@feuerlabs.com`](mailto:magnus.feuer@feuerlabs.com)).\n\n[](https://travis-ci.org/Feuerlabs/exometer)\n\n__NOTE: Exometer has been split into [exometer_core](https://github.com/Feuerlabs/exometer_core), and exometer (as well as separate reporter applications). The latest monolithic version of Exometer is 1.1.__\n\nThe Exometer package allows for easy and efficient instrumentation of\nErlang code, allowing crucial data on system performance to be\nexported to a wide variety of monitoring systems.\n\nExometer comes with a set of pre-defined monitor components, and can\nbe expanded with custom components to handle new types of Metrics, as\nwell as integration with additional external systems such as\ndatabases, load balancers, etc.\n\nThis document gives a high level overview of the Exometer system. For\ndetails, please see the documentation for individual modules, starting\nwith `exometer`.\n\nNote the section on [Dependency Management](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Dependency_Management) for how to deal with\noptional packages, both users and developers.\n\n\n### Table of Content ###\n\n\n1. [Concept and definitions](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Concept_and_definitions)\n 1. [Metric](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Metric)\n 2. [Data Point](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Data_Point)\n 3. [Metric Type](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Metric_Type)\n 4. [Entry Callback](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Entry_Callback)\n 5. [Probe](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Probe)\n 6. [Caching](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Caching)\n 7. [Subscriptions and Reporters](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Subscriptions_and_Reporters)\n2. [Built-in entries and probes](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Built-in_entries_and_probes)\n 1. [counter (exometer native)](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#counter_(exometer_native))\n 2. [fast_counter (exometer native)](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#fast_counter_(exometer_native))\n 3. [gauge (exometer native)](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#gauge_(exometer_native))\n 4. [exometer_histogram (probe)](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#exometer_histogram_(probe))\n 5. [exometer_uniform (probe)](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#exometer_uniform_(probe))\n 6. [exometer_spiral (probe)](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#exometer_spiral_(probe))\n 7. [exometer_folsom [entry]](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#exometer_folsom_[entry])\n 8. [exometer_function [entry]](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#exometer_function_[entry])\n3. [Built in Reporters](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Built_in_Reporters)\n 1. [exometer_report_graphite](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#exometer_report_graphite)\n 2. [exometer_report_opentsdb](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#exometer_report_opentsdb)\n 3. [exometer_report_amqp](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#exometer_report_amqp)\n 4. [exometer_report_snmp](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#exometer_report_snmp)\n4. [Instrumenting Erlang code](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Instrumenting_Erlang_code)\n 1. [Exometer Start](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Exometer_Start)\n 2. [Creating metrics](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Creating_metrics)\n 3. [Deleting metrics](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Deleting_metrics)\n 4. [Setting metric values](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Setting_metric_values)\n 5. [Retrieving metric values](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Retrieving_metric_values)\n 6. [Setting up subscriptions](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Setting_up_subscriptions)\n 7. [Set metric options](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Set_metric_options)\n5. [Configuring Exometer](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_Exometer)\n 1. [Configuring type - entry maps](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_type_-_entry_maps)\n 2. [Configuring statically defined entries](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_statically_defined_entries)\n 3. [Configuring static subscriptions](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_static_subscriptions)\n 4. [Configuring reporter plugins](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_reporter_plugins)\n 5. [Configuring opentsdb reporter](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_opentsdb_reporter)\n 6. [Configuring amqp reporter](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_amqp_reporter)\n 7. [Configuring graphite reporter](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_graphite_reporter)\n 8. [Configuring snmp reporter](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_snmp_reporter)\n6. [Creating custom exometer entries](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Creating_custom_exometer_entries)\n7. [Creating custom probes](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Creating_custom_probes)\n8. [Creating custom reporter plugins](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Creating_custom_reporter_plugins)\n9. [Dependency management](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Dependency_management)\n\n\n### Concepts and Definitions ###\n\nExometer introduces a number of concepts and definitions used\nthroughout the documentation and the code.\n\n\n\n\n#### Metric ####\n\nA metric is a specific measurement sampled inside an Erlang system and\nthen reported to the Exometer system. An example metric would be\n\"transactions_per_second\", or \"memory_usage\".\n\nMetrics are identified by a list of terms, such as given below:\n\n`[ xml_front_end, parser, file_size ]`\n\nA metric is created through a call by the code to be instrumented to\n`exometer:new()`. Once created, the metric can be updated through\n`exometer:update()`, or on its own initiative through the\n`exometer_probe:sample` behavior implementation.\n\n\n#### Data Point ####\n\nEach metric can consist of multiple data points, where each point has\na specific value.\n\nA typical example of data points would be a\n`transactions_per_second` (tps) metric, usually stored as a\nhistogram covering the last couple of minutes of tps samples. Such a\nhistogram would host multiple values, such as `min`, `max`,\n`median`, `mean`, `50_percentile`, `75_percentile`,\netc.\n\nIt is up to the type of the metric, and the data probe backing that\ntype (see below), to specify which data points are available under the\ngiven metric.\n\n\n#### Metric Type ####\n\nThe type of a metric, specified when the metric is created through\n`exometer:new()`, determines which `exometer_entry`\ncallback to use.\n\nThe link between the type and the entry to use is configured\nthrough the `exometer_admin` module, and its associated exometer\ndefaults configuration data.\n\nThe metric type, in other words, is mainly used to map a metric to a\nconfigurable `exometer_entry` callback, but it can also be referenced\nin queries using `exometer:select/1`. An entry callback can also support\nmultiple types (the type is provided as an argument in the callback functions).\n\nExometer provides default mappings for a number of metric types. It is\npossible to select different callbacks for each metric instance, as well\nas modify metrics using callback-specific options. Please see\n[Configuring type - entry maps](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_type_-_entry_maps) for details on how to do this.\n\n\n#### Entry Callback ####\n\nAn exometer entry callback will receive values reported to a metric through the\n`exometer:update()` call and compile it into one or more data points.\nThe entry callback can either be a counter (implemented natively\nin `exometer`), or a more complex statistical analysis such\nas a uniform distribution or a regular histogram.\n\nThe various outputs from these entries are reported as data points\nunder the given metric.\n\nAn entry can also interface external analytics packages.\n`exometer_folsom`, for example, integrates with the\n`folsom_metrics` package found at [`https://github.com/boundary/folsom`](https://github.com/boundary/folsom).\n\n\n#### Probe ####\n\nProbes are a further specialization of exometer entries that run in\ntheir own Erlang processes and have their own state (like a\ngen_server). A probe is implemented through the `exometer_probe`\nbehavior.\n\nA probe can be used if independent monitoring is needed of,\nfor example, `/proc` trees, network interfaces, and other subsystems\nthat need periodic sampling. In these cases, the\n`exometer_probe:probe_sample()` call is invoked regularly by exometer,\nin the probe's own process, in order to extract data from\nthe given subsystem and add it to the metric's data points.\n\n\n#### Caching ####\n\nMetric and data point values are read with the `exometer:get_value()`\nfunction. In the case of counters, this operation is very fast. With probes,\nthe call results in a synchronous dialog with the probe process, and the\ncost of serving the request depends on the probe implementation and the\nnature of the metric being served.\n\nIf the cost of reading the value is so high that calling the function often\nwould result in prohibitive load, it is possible to cache the value. This is\ndone either explicitly from the probe itself (by calling\n`exometer_cache:write()`), or by specifying the option `{cache, Lifetime}`\nfor the entry. If an entry has a non-zero cache lifetime specified, the\n`get_value()` call will try fetching the cached value before calling the\nactual entry and automatically caching the result.\n\nNote that if `{cache, Lifetime}` is not specified, `exometer:get_value()`\nwill neither read nor write to the cache. It is possible for the probe\nto periodically cache a value regardless of how the cache lifetime is set,\nand the probe may also explicitly read from the cache if it isn't done\nautomatically.\n\n\n#### Subscriptions and Reporters ####\n\nThe subscription concept, managed by `exometer_report` allows metrics\nand their data points to be sampled at given intervals and delivered\nto one or more recipients, which can be either an arbitrary process\nor a Reporter plugin.\n\nEach subscription ties a specific metric-datapoint pair to a reporter\nand an interval (given in milliseconds). The reporter system will, at\nthe given interval, send the current value of the data point to the\nsubscribing reporter. The subscription, with all its parameters,\nis setup through a call to `exometer_report:subscribe()`.\n\nIn the case of processes, subscribed-to values will be delivered as a\nmessage. Modules, which implement the `exometer_report` callback\nbehavior, will receive the plugins as a callbacks within the\n`exometer_report` process.\n\nSubscriptions can either be setup at runtime, through\n`exometer_report:subscribe()` calls, or statically through the\n`exometer_report` configuration data.\n\n\n### Built-in entries and probes ###\n\n\nThere are a number of built-in entries and probes shipped\nwith the Exometer package, as described below:\n\n\n#### counter (exometer native) ####\n\n\nThe counter is implemented directly in `exometer` to provide simple\ncounters. A call to `exometer:update()` will add the provided value\nto the counter.\n\nThe counter can be reset to zero through `exometer:reset()`.\n\nThe available data points under a metric using the counter entry\nare `value` and `ms_since_reset`.\n\n\n#### fast_counter (exometer native) ####\n\nA fast counter implements the counter functionality, through the\n`trace_info` system, yielding a speed increase of about 3.5 in\ncomparison to the regular counter.\n\nThe tradeoff is that running tracing and/or debugging may interfere\nwith the counter functionality.\n\nA call to `exometer:update()` will add the provided value to the\ncounter.\n\nThe counter can be reset to zero through `exometer:reset()`.\n\nThe available data points under a metric using the fast_counter\nentry are `value` and `ms_since_reset`.\n\n\n#### gauge (exometer native) ####\n\nThe gauge is implemented directly in `exometer` to provide simple\ngauges. A call to `exometer:update()` will set the gauge's value\nto the provided value. That is, the value of the gauge entry is\nalways the most recently provided value.\n\nThe gauge can be reset to zero through `exometer:reset()`.\n\nThe available data points under a metric using the gauge entry\nare `value` and `ms_since_reset`.\n\n\n#### histogram (probe) ####\n\nThe histogram probe stores a given number of updates, provided through\n`exometer:update()`, in a histogram. The histogram maintains a log\nderived from all values received during a configurable time span and\nprovides min, max, median, mean, and percentile analysis data points\nfor the stored data.\n\nExometer supports a number of different histogram implementations, each\nwith different performance and accuracy trade-offs. \n\nIn order to save memory, the histogram is divided into equal-sized\ntime slots, where each slot spans a settable interval. All values\nreceived during a time slot will be averaged into a single value to be\nstored in the histogram once the time slot expires. The averaging\nfunction (which can be replaced by the caller), allows for\nhigh-frequency update metrics to have their resolution traded against\nresource consumption.\n\n\n#### exometer_uniform (probe) ####\n\nThe uniform probe provides a uniform sample over a pool of values\nprovided through `exometer:update()`. When the pool reaches its configurable\nmax size, existing values will be replaced at random to make space for\nnew values. Much like `exometer_histogram`, the uniform probe\nprovides min, max, median, mean, and percentile analysis data points\nfor the stored data.\n\n\n#### exometer_spiral (probe) ####\n\nThe spiral probe maintains the total sum of all values stored in its\nhistogram. The histogram has a configurable time span, all values\nprovided to the probe, through `exometer:update()`, within that time\nspan will be summed up and reported. If, for example, the histogram\ncovers 60 seconds, the spiral probe will report the sum of all\nvalues reported during the last minute.\n\nThe grand total of all values received during the lifetime of the\nprobe is also available.\n\n\n#### exometer_folsom [entry] ####\n\n`exometer_folsom` is an entry behavior which implements most metric types\nsupported by the [folsom](https://github.com/boundary/folsom)\nmetrics package: Specifically, the metric types `counter`, `spiral`,\n`histogram`, `meter`, `meter_reader`, `gauge`, `duration` and `history`.\n\nThe folsom entry integrates with the folsom metrics package provided\nby the boundary repo at github. Updated values sent to the folsom entry\ncan be forwarded to folsom's counter, histogram, duration, meter,\nand spiral.\n\nFolsom integration is provided as a backup. New code using Exometer\nshould use the native probes that duplicate folsom.\n\n\n#### exometer_function [entry] ####\n\nThe function entry allows for an existing erlang function to be wrapped\nas an exometer entry. The [`exometer_function`](https://github.com/Feuerlabs/exometer_core/blob/master/doc/exometer_function.md) module supports a number\nof options for passing arguments and matching out data points from the\nresult.\n\nThe function entry provides an easy way of integrating an external\nsystem without having to write a complete entry.\n\n\n### Built in Reporters ###\n\nExometer ships with some built-in reporters which can be used to forward updated\nmetrics and their data points to external systems. They can also\nserve as templates for custom-developed reporters.\n\n\n#### exometer_report_graphite ####\n\nThe graphite reporter uses the TCP/IP protocol to forward\nsubscribed-to metrics and data points to a graphite server, such as\nthe one provided by [`http://hostedgraphite.com`](http://hostedgraphite.com). When the graphite\nreporter receives a metric-datapoint value (subscribed to through\n`exometer_report:subscriber()`), the reporter will immediately\nforward the key-value pair to the graphite server.\n\n\n#### exometer_report_opentsdb ####\n\nThe OpenTSDB reporter sends metrics to an OpenTSDB server using\nthe telnet API. All subscribed-to metric-datapoint values received\nby the reporter are immediately forwarded to OpenTSDB.\n\nIf the OpenTSDB connection is lost, the reporter will attempt to reconnect to it\nat a configurable interval.\n\nThe data sent to OpenTSDB will be formatted as follows:\n\n```\nput metric timestamp value host=host type=datapoint\n```\n\nWhere the value for the host tag will be the configured host in the reporter\nconfiguration (defaults to the value returned by `netadm:localhost`), and\ndatapoint tags as specified by the subscriber.\n\nPlease see [Configuring opentsdb reporter](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_opentsdb_reporter) for details on the\napplication environment parameters listed above.\n\n\n#### exometer_report_amqp ####\n\nThe AMQP reporter sends metrics to an AMQP broker as a json-encoded payload. All\nsubscribed-to metric-datapoint values received by the reporter are forwarded to AMQP.\n\nIf the AMQP connection is lost, the reporter will attempt to reconnect to it\nat a configurable interval.\n\nThe data sent to AMQP will be formatted as follows:\n\n```\n{\n \"type\":\"exometer_metric\",\n \"body\":\n {\"name\":\"messages_per_second\",\n \"value\":0,\"timestamp\":1414006826,\n \"host\":\"testhost\",\n \"instance\":\"max\"}\n}\n```\n\nWhere the value for the host tag will be the configured host in the reporter\nconfiguration (defaults to the value returned by `netadm:localhost`), the\ninstance tag represents the datapoint for the metric.\n\nPlease see [Configuring amqp reporter](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_amqp_reporter) for details on the\napplication environment parameters listed above.\n\n\n#### exometer_report_snmp ####\n\nThe SNMP reporter enables the export of metrics and their datapoints to SNMP managers.\nThe export needs to be enabled for each metric through their options.\nMoreover, SNMP notifications can be created using the options to send periodic reports\non datapoints to SNMP managers. All SNMP protocol handling is done by the snmp application\nshipped with Erlang/OTP. Thus, the snmp application needs to be started and\nthe local SNMP master agent needs to be configured correctly for SNMP export to work\nproperly.\n\nTo configure SNMP export for a single metric use these options:\n\n+ `{snmp, disabled}` (default) Disables SNMP export for the metric. Same as not specifying the option at all.\n\n+ `{snmp, []}` Enables SNMP export for the metric. No subscriptions are setup.\n\n+ `{snmp, [{Datapint, Interval}]}` Enables SNMP export for the metric. Subscriptions are setup for the given Datapoint/Interval pairs. Each subscription report will be forwarded to SNMP mangers as notifications.\n\n+ `{snmp, [{Datapint, Interval, Extra}]}` Same as above, but using an addition extra identification for the subscriptions. Allow the creation ofmultiple subscriptions for a single datapoint.\n\nPlease see [Configuring snmp reporter](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_snmp_reporter) for details on how to configure the\nSNMP reporter.\n\n\n### Instrumenting Erlang code ###\n\nThe code using Exometer needs to be instrumented in order to setup and\nuse metrics reporting.\n\n\n#### Exometer Start ####\n\nThe system using Exometer must start the `exometer` application prior to using it:\n\n```erlang\n\napplication:start(lager),\napplication:start(exometer).\n```\n\nNote that dependent applications need to be started first. On newer OTP versions\n(R16B or later), you can use `application:ensure_all_started(exometer)`.\n\nFor testing, you can also use [`exometer:start/0`](https://github.com/Feuerlabs/exometer_core/blob/master/doc/exometer.md#start-0).\n\nIf you make use of e.g. folsom metrics, you also need to start `folsom`. Exometer\nwill not do that automatically, nor does it contain an application dependency for it.\n\nSee [Configuring Exometer](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_Exometer) for details on configuration data\nformat.\n\n\n#### Creating metrics ####\n\nA metric, can be created through a call to\n\n```erlang\n\nexometer:new(Name, Type)\n```\n\n`Name` is a list of atoms, uniquely identifying the metric created.\nThe type of the metric, specified by `Type` will be mapped\nto an exometer entry through the table maintained by\n`exometer_admin` Please see the [Configuring type - entry\nmaps](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_type_-_entry_maps) for details.\n\nThe resolved entry to use will determine the data points available\nunder the given metric.\n\n\n#### Deleting metrics ####\n\nA metric previously created with `exometer:new()` can be deleted by\n`exometer:delete()`.\n\nAll subscriptions to the deleted metrics will be cancelled.\n\n\n#### Setting metric values ####\n\nA created metric can have its value updated through the\n`exometer:update()` function:\n\n```erlang\n\nexometer:update(Name, Value)\n```\n\nThe `Name` parameter is the same atom list provided to a previous\n`exometer:new()` call. The `Value` is an arbitrarty element that is\nforwarded to the `exometer:update()` function of the entry/probe that the\nmetric is mapped to.\n\nThe receiving entry/probe will process the provided value and modify\nits data points accordingly.\n\n\n#### Retrieving metric values ####\n\nExometer-using code can at any time retrieve the data point values\nassociated with a previously created metric. In order to find out which\ndata points are available for a metric, the following call can be used:\n\n```erlang\n\nexometer:info(Name, datapoints)\n```\n\nThe `Name` parameter is the same atom list provided to a previous\n`exometer:new()` call. The call will return a list of data point\natoms that can then be provided to `exometer:get_value()` to\nretrieve their actual value:\n\n```erlang\n\nexometer:get_value(Name, DataPoint)\n```\n\nThe `Name` paramer identifies the metric, and `DataPoints`\nidentifies the data points (returned from the previous `info()` call)\nto retrieve the value for.\n\nIf no DataPoints are provided, the values of a default list of data points,\ndetermined by the backing entry / probe, will be returned.\n\n\n#### Setting up subscriptions ####\n\nA subscription can either be statically configured, or dynamically\nsetup from within the code using Exometer. For details on statically\nconfigured subscriptions, please see [Configuring static subscriptions](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_static_subscriptions).\n\nA dynamic subscription can be setup with the following call:\n\n```erlang\n\nexometer_report:subscribe(Recipient, Metric, DataPoint, Inteval)\n```\n\n`Recipient` is the name of a reporter.\n\n\n#### Set metric options ####\n\n\nEach created metric can have options setup for it through the following call:\n\n```erlang\n\nexometer:setopts(Name, Options)\n```\n\nThe `Name` paramer identifies the metric to set the options for, and\nOptions is a proplist (`[{ Key, Value },...]`) with the options to be\nset.\n\nExometer looks up the the backing entry that hosts the metric with the given Name, and will\ninvoke the entry\\'s `setopts/4` function to set the actual options. Please see the\n`setopts/4` function for the various entries for details.\n\n\n### Configuring Exometer ###\n\nExometer defaults can be changed either through OTP application environment\nvariables or through the use of Basho's `cuttlefish`\n([`https://github.com/basho/cuttlefish`](https://github.com/basho/cuttlefish)).\n\n\n#### Configuring type - entry maps ####\n\nThe dynamic method of configuring defaults for `exometer` entries is:\n\n```erlang\n\nexometer_admin:set_default(NamePattern, Type, Default)\n```\n\nWhere `NamePattern` is a list of terms describing what is essentially\na name prefix with optional wildcards (`'_'`). A pattern that\nmatches any legal name is `['_']`.\n\n`Type` is an atom defining a type of metric. The types already known to\n`exometer`, `counter`, `fast_counter`, `ticker`, `uniform`, `histogram`,\n`spiral`, `netlink`, and `probe` may be redefined, but other types can be\ndescribed as well.\n\n`Default` is either an `#exometer_entry{}` record (unlikely), or a list of\n`{Key, Value}` options, where the keys correspond to `#exometer_entry` record\nattribute names. The following attributes make sense to preset:\n\n```erlang\n\n{module, atom()} % the callback module\n{status, enabled | disabled} % operational status of the entry\n{cache, non_neg_integer()} % cache lifetime (ms)\n{options, [{atom(), any()}]} % entry-specific options\n```\n\nBelow is an example, from `exometer/priv/app.config`:\n\n```erlang\n\n{exometer, [\n {defaults, [\n {['_'], function , [{module, exometer_function}]},\n {['_'], counter , [{module, exometer}]},\n {['_'], histogram, [{module, exometer_histogram}]},\n {['_'], spiral , [{module, exometer_spiral}]},\n {['_'], duration , [{module, exometer_folsom}]},\n {['_'], meter , [{module, exometer_folsom}]},\n {['_'], gauge , [{module, exometer_folsom}]}\n ]}\n]}\n```\n\nIn systems that use CuttleFish, the file `exometer/priv/exometer.schema`\ncontains a schema for default settings. The setup corresponding to the above\ndefaults would be as follows:\n\n```ini\n\nexometer.template.function.module = exometer_function\nexometer.template.counter.module = exometer\nexometer.template.histogram.module = exometer_histogram\nexometer.template.spiral.module = exometer_spiral\nexometer.template.duration.module = exometer_folsom\nexometer.template.meter.module = exometer_folsom\nexometer.template.gauge.module = exometer_folsom\n```\n\n\n#### Configuring statically defined entries ####\n\nUsing the `exometer` environment variable `predefined`, entries can be added\nat application startup. The variable should have one of the following values:\n\n* `{script, File}` - `File` will be processed using `file:script/2`. The return\n value (the result of the last expression in the script) should be a list of`{Name, Type, Options}` tuples.\n\n* `{apply, M, F, A}` - The result of `apply(M, F, A)` should be `{ok, L}` where`L` is a list of `{Name, Type, Options}` tuples.\n\n* `L`, where L is a list of `{Name, Type, Options}` tuples or extended\ninstructions (see below).\n\nThe list of instructions may include:\n\n* `{delete, Name}` - deletes `Name` from the exometer registry.\n\n* `{select_delete, Pattern}` - applies a select pattern and\ndeletes all matching entries.\n\n* `{re_register, {Name, Type, Options}}` - redefines an entry if present,\notherwise creates it.\n\nExometer will also scan all loaded applications for the environment\nvariables `exometer_defaults` and `exometer_predefined`, and process\nas above. If an application is loaded and started after exometer has started,\nit may call the function `exometer:register_application()` or\n`exometer:register_application(App)`. This function will do nothing if\nexometer isn't already running, and otherwise process the `exometer_defaults`\nand `exometer_predefined` variables as above. The function can also be\ncalled during upgrade, as it will re-apply the settings each time.\n\n\n#### Configuring static subscriptions ####\n\n\nStatic subscriptions, which are automatically setup at exometer\nstartup without having to invoke `exometer_report:subscribe()`, are\nconfigured through the report sub section under exometer.\n\nBelow is an example, from `exometer/priv/app.config`:\n\n```erlang\n\n{exometer, [\n {report, [\n {subscribers, [\n {exometer_report_collectd, [db, cache, hits], mean, 2000, true},\n {exometer_report_collectd, [db, cache, hits], max, 5000, false}\n ]}\n ]}\n]}\n```\n\nThe `report` section configures static subscriptions and reporter\nplugins. See [Configuring reporter plugins](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_reporter_plugins) for details on\nhow to configure individual plugins.\n\nThe `subscribers` sub-section contains all static subscriptions to be\nsetup att exometer applications start. Each tuple in the prop list\nshould be of one of the following formats:\n\n* `{Reporter, Metric, DataPoint, Interval}`\n\n* `{Reporter, Metric, DataPoint, Interval, RetryFailedMetrics}`\n\n* `{Reporter, Metric, DataPoint, Interval, RetryFailedMetrics, Extra}`\n\n* `{apply, {M, F, A}}`\n\n* `{select, {MatchPattern, DataPoint, Interval [, Retry [, Extra] ]}}`\n\nIn the case of `{apply, M, F, A}`, the result of `apply(M, F, A)` must\nbe a list of `subscribers` tuples.\n\nIn the case of `{select, Expr}`, a list of metrics is fetched using\n`exometer:select(MatchPattern)`, where the result must be on the form\n`{Key, Type, Status}` (i.e. what corresponds to `'$_'`).\nThe rest of the items will be applied to each of the matching entries.\n\nThe meaning of the above tuple elements is:\n\n+ `Reporter :: module()` Specifies the reporter plugin module, such as`exometer_report_collectd` that is to receive updated metric's data\npoints.\n\n+ `Metric :: [atoms()]` Specifies the path to a metric previously created with an`exometer:new()` call.\n\n+ `DataPoint` :: atom() | [atom()]' Specifies the data point within the given metric to send to the\n receiver. The data point must match one of the data points returned by`exometer:info(Name, datapoints)` for the given metrics name.\n\n+ `Interval` :: integer()' (milliseconds) Specifies the interval, in milliseconds, between each update of the\ngiven metric's data point. At the given interval, the data point will\nbe samples, and the result will be sent to the receiver.\n\n+ `RetryFailedMetrics :: boolean()` Specifies if the metric should be continued to be reported\n even if it is not found during a reporting cycle. This would be\n the case if a metric is not created by the time it is reported for\n the first time. If the metric will be created at a later time,\n this value should be set to true. Set this value to false if all\n attempts to report the metric should stop if when is not found.\n The default value is `true`.\n\n+ `Extra :: any()` Provides a means to pass along extra information for a given\n subscription. An example is the `syntax` option for the SNMP reporter,\n in which case `Extra` needs to be a property list.\n\nExample configuration in sys.config, using the `{select, Expr}` pattern:\n\n```erlang\n\n[\n {exometer, [\n {predefined,\n [{[a,1], counter, []},\n {[a,2], counter, []},\n {[b,1], counter, []},\n {[c,1], counter, []}]},\n {report,\n [\n {reporters,\n [{exometer_report_tty, []}]},\n {subscribers,\n [{select, {[{ {[a,'_'],'_','_'}, [], ['$_']}],\n exometer_report_tty, value, 1000}}]}\n ]}\n ]}\n].\n\n```\n\nThis will activate a subscription on `[a,1]` and `[a,2]` in the\n`exometer_report_tty` reporter, firing once per second.\n\n\n#### Configuring reporter plugins ####\n\n\nThe various reporter plugins to be loaded by exometer are configured\nin the `report` section under `reporters`\n\nEach reporter has an entry named after its module, and the content of\nthat entry is dependent on the reporter itself. The following chapters\nspecifies the configuration parameters for the reporters shipped with\nexometer.\n\n\n#### Configuring opentsdb reporter ####\n\n\nBelow is an example of the opentsdb reporter application environment, with\nits correct location in the hierarchy:\n\n```erlang\n\n{exometer, [\n {report, [\n {reporters, [\n {exometer_report_opentsdb, [\n {reconnect_interval, 10},\n {connect_timeout, 8000},\n {hostname, \"testhost\"},\n {host, {\"127.0.0.1\", 4242}}\n ]}\n ]}\n ]}\n]}\n```\n\nThe following attributes are available for configuration:\n\n+ `reconnect_interval` (seconds - default: 30) Specifies the duration between each reconnect attempt to an opentsdb\nserver that is not available. Should the server either be unavailable\nat exometer startup, or become unavailable during exometer's\noperation, exometer will attempt to reconnect at the given number of\nseconds.\n\n+ `connect_timeout` (milliseconds - default: 5000) Specifies how long the opentsdb reporter plugin shall wait for a\nsocket connection to complete before timing out. A timed out\nconnection attempt will be retried after the reconnect interval has\npassed see item 1 above).\n\n+ `hostname` (string - default: `net_adm:localhost()`) Specifies the host name to use for the host tag in the OpenTSDB tags.\n Please see [Configuring opentsdb reporter](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_opentsdb_reporter) for details.\n\n+ `host` (ip - default: {\"127.0.0.1\", 4242}) Specifies the host and port to connect to OpenTSDB.\n\n\n#### Configuring amqp reporter ####\n\n\nBelow is an example of the amqp reporter application environment, with\nits correct location in the hierarchy:\n\n```erlang\n\n{exometer, [\n {report, [\n {reporters, [\n {exometer_report_amqp, [\n {reconnect_interval, 10},\n {hostname, \"testhost\"},\n {amqp_url, \"amqp://user:pass@host:5672/%2f\"},\n\t\t{exchange, \"metrics\"},\n\t\t{routing_key, \"metrics\"},\n\t\t{buffer_size, 0}\n ]}\n ]}\n ]}\n]}\n```\n\nThe following attributes are available for configuration:\n\n+ `reconnect_interval` (seconds - default: 30) Specifies the duration between each reconnect attempt to an amqp\nbroker that is not available. Should the server either be unavailable\nat exometer startup, or become unavailable during exometer's\noperation, exometer will attempt to reconnect at the given number of\nseconds.\n\n+ `hostname` (string - default: `net_adm:localhost()`) Specifies the host name to use for the host property in the JSON payload.\n Please see [Configuring amqp reporter](https://github.com/Feuerlabs/exometer/blob/master/doc/README.md#Configuring_amqp_reporter) for details.\n\n+ `amqp_url` (string - default: `amqp://guest:guest@localhost:5672/%2f`) Specifies the amqp url to connect to.\n\n+ `exchange` (string - default: `exometer`) Specifies the exchange to publish messages to.\n\n+ `routing_key` (string - default: `exometer`) Specifies the routing key to use when publishing messages.\n\n+ `buffer_size` (integer - default: `0`) Specifies the size in bytes of payload to buffer before sending to AMQP.\n\n\n#### Configuring graphite reporter ####\n\n\nBelow is an example of the a graphite reporter application environment, with\nits correct location in the hierarchy:\n\n```erlang\n\n{exometer, [\n {report, [\n {reporters, [\n {exometer_report_graphite, [\n {connect_timeout, 5000},\n {prefix, \"web_stats\"},\n {host, \"carbon.hostedgraphite.com\"},\n {port, 2003},\n {api_key, \"267d121c-8387-459a-9326-000000000000\"}\n ]}\n ]}\n ]}\n]}\n```\n\nThe following attributes are available for configuration:\n\n+ `connect_timeout` (milliseconds - default: 5000) Specifies how long the graphie reporter plugin shall wait for a tcp\nconnection to complete before timing out. A timed out connection will\nnot be reconnected to automatically. (To be fixed.)\n\n+ `prefix` (string - default: \"\") Specifies an optional prefix to prepend all metric names with before\nthey are sent to the graphite server.\n\n+ `host` (string - default: \"carbon.hostedgraphite.com\") Specifies the name (or IP address) of the graphite server to report to.\n\n+ `port` (integer - default: 2003) Specifies the TCP port on the given graphite server to connect to.\n\n+ `api_key` (string - default: n/a) Specifies the api key to use when reporting to a hosted graphite server.\n\nIf `prefix` is not specified, but `api_key` is, each metrics will be reported as `ApiKey.Metric`.\n\nIf `prefix` is specified, but `api_key` is not, each metrics will be reported as `Prefix.Metric`.\n\nif neither `prefix` or `api_key` is specified, each metric will be reported simply as `Metric`.\n\n\n#### Configuring snmp reporter ####\n\n\nBelow is an example of the a snmp reporter application environment, with\nits correct location in the hierarchy:\n\n```erlang\n\n{exometer, [\n {report, [\n {reporters, [\n {exometer_report_snmp, [\n {mib_template, \"priv/MYORG-EXOMETER-METRICS.mib\"},\n {mib_dir, \"/tmp/exometer\"}\n ]}\n ]}\n ]}\n]}\n```\n\nThe following attributes are available for configuration:\n\n+ `mib_template` (string - default: \"mibs/EXOMETER-METRICS-MIB.mib\") Specifies where to find the MIB template used for dynamically assembline an internal MIB. Take a look at the MIB template shipped with Exometer for reference in case you want to define your own template.\n\n+ `mib_dir` (string - default: \"tmp/exometer_report_snmp\") Specifies temporary direction which will be used by Exometer to store dymanically created MIB files.\n\n\n### Creating custom exometer entries ###\n\n\nPlease see @see exometer_entry documentation for details.\n\n\n### Creating custom probes ###\n\n\nPlease see @see exometer_probe documentation for details.\n\n\n### Creating custom reporter plugins ###\n\n\nPlease see @see exometer_report documentation for details.\n\n\n### Dependency management ###\n\n\nExometer dependencies can be controlled using the `EXOMETER_PACKAGES`\nunix environment variable: a string listing packages or applications to\neither keep or remove, separated using space, tab or comma.\n\n\n#### Syntax ####\n\n+ `(Package)` - use `Package` as a base. This will implicitly exclude all\n applications not included in `Package`. See below for supported packages.\n\n+ `+(Package)` - add applications included in `Package`.\n\n+ `-(Package)` - remove applications in `Package` (except mandatory deps).\n\n+ `App` - keep application `App`.\n\n+ `+App` - keep application `App`.\n\n+ `-App` - exclude application `App`.\n\n\n#### Supported packages ####\n\n+ `minimal` - only the mandatory deps: `lager`, `parse_trans`, `setup`.\n\n+ `basic` - (mandatory deps and) `folsom`.\n\n+ `amqp` - (mandatory deps and) `amqp_client`, `jiffy`.\n\n+ `full` - all of the above, plus `afunix` and `netlink`.\n\nExample - use only basic deps plus `afunix`\n\n```\n EXOMETER_PACKAGES=\"(basic), +afunix\" make\n```\n\nExample - use all deps except the AMQP-related deps:\n\n```\n export EXOMETER_PACKAGES=\"(full) -(amqp)\"\n```\n\n\n#### Conditional defines ####\n\nFor each optional dependency that is included, a macro is defined,\nnamed `dep_App` - e.g. `dep_afunix`. Developers must not include\ncompile-time dependencies to optional applications, without checking\nthe corresponding macro and ensuring that the module compiles even\nwhen the dependent application is not included. See `exometer_report_amqp.erl`\nfor an example.\n\n\n#### Customizing rebar.config ####\n\nThe OS environment variables `EXOMETER_CONFIG_PREPROCESS` and\n`EXOMETER_CONFIG_POSTPROCESS` can be used to insert a script, similar to\n`rebar.config.script` in the processing flow of the exometer build.\n\nAs the names imply, the script given by `EXOMETER_CONFIG_PREPROCESS` (if any)\nwill be run before exometer does any processing of its own, and the\n`EXOMETER_CONFIG_POSTPROCESS` script (if any) will be run after all other\nprocessing is complete.\nThings that could be done in preprocessing: re-targeting a dependency,\nmodifying the list of predefined packages, etc.\n\n## Modules ##\n\n\n
\n\n"
},
{
"path": "TODO.md",
"content": "# Improvements\n\n- report plugin processes are currently manually monitored by exometer_report, this should be changed to a more robust appraoch. Could use simple_one_for_one, which lacks restarts. Can't use other normal supervision strategies without some changes as we don't want external systems to trigger a supervision tree collapse.\n\n# Bugs\n\n- fix all dialyzer errors\n- type specs for SNMP stuff\n- subscription parameter extra needs to be sent to manager as part of inform\n\n# Misc\n\n- use proper module organization in all modules, e.g. exports, includes, external API, internal API\n- update documentation\n"
},
{
"path": "doc/README.md",
"content": "\n\n# Exometer - Erlang instrumentation package #\n\nCopyright (c) 2014 Basho Technologies, Inc. All Rights Reserved.\n\n__Version:__ Feb 1 2015 23:02:37\n\n__Authors:__ Ulf Wiger ([`ulf.wiger@feuerlabs.com`](mailto:ulf.wiger@feuerlabs.com)), Magnus Feuer ([`magnus.feuer@feuerlabs.com`](mailto:magnus.feuer@feuerlabs.com)).\n\n[](https://travis-ci.org/Feuerlabs/exometer)\n\n__NOTE: Exometer has been split into [exometer_core](https://github.com/Feuerlabs/exometer_core), and exometer (as well as separate reporter applications). The latest monolithic version of Exometer is 1.1.__\n\nThe Exometer package allows for easy and efficient instrumentation of\nErlang code, allowing crucial data on system performance to be\nexported to a wide variety of monitoring systems.\n\nExometer comes with a set of pre-defined monitor components, and can\nbe expanded with custom components to handle new types of Metrics, as\nwell as integration with additional external systems such as\ndatabases, load balancers, etc.\n\nThis document gives a high level overview of the Exometer system. For\ndetails, please see the documentation for individual modules, starting\nwith `exometer`.\n\nNote the section on [Dependency Management](#Dependency_Management) for how to deal with\noptional packages, both users and developers.\n\n\n### Table of Content ###\n\n\n1. [Concept and definitions](#Concept_and_definitions)\n 1. [Metric](#Metric)\n 2. [Data Point](#Data_Point)\n 3. [Metric Type](#Metric_Type)\n 4. [Entry Callback](#Entry_Callback)\n 5. [Probe](#Probe)\n 6. [Caching](#Caching)\n 7. [Subscriptions and Reporters](#Subscriptions_and_Reporters)\n2. [Built-in entries and probes](#Built-in_entries_and_probes)\n 1. [counter (exometer native)](#counter_(exometer_native))\n 2. [fast_counter (exometer native)](#fast_counter_(exometer_native))\n 3. [gauge (exometer native)](#gauge_(exometer_native))\n 4. [exometer_histogram (probe)](#exometer_histogram_(probe))\n 5. [exometer_uniform (probe)](#exometer_uniform_(probe))\n 6. [exometer_spiral (probe)](#exometer_spiral_(probe))\n 7. [exometer_folsom [entry]](#exometer_folsom_[entry])\n 8. [exometer_function [entry]](#exometer_function_[entry])\n3. [Built in Reporters](#Built_in_Reporters)\n 1. [exometer_report_graphite](#exometer_report_graphite)\n 2. [exometer_report_opentsdb](#exometer_report_opentsdb)\n 3. [exometer_report_amqp](#exometer_report_amqp)\n 4. [exometer_report_snmp](#exometer_report_snmp)\n4. [Instrumenting Erlang code](#Instrumenting_Erlang_code)\n 1. [Exometer Start](#Exometer_Start)\n 2. [Creating metrics](#Creating_metrics)\n 3. [Deleting metrics](#Deleting_metrics)\n 4. [Setting metric values](#Setting_metric_values)\n 5. [Retrieving metric values](#Retrieving_metric_values)\n 6. [Setting up subscriptions](#Setting_up_subscriptions)\n 7. [Set metric options](#Set_metric_options)\n5. [Configuring Exometer](#Configuring_Exometer)\n 1. [Configuring type - entry maps](#Configuring_type_-_entry_maps)\n 2. [Configuring statically defined entries](#Configuring_statically_defined_entries)\n 3. [Configuring static subscriptions](#Configuring_static_subscriptions)\n 4. [Configuring reporter plugins](#Configuring_reporter_plugins)\n 5. [Configuring opentsdb reporter](#Configuring_opentsdb_reporter)\n 6. [Configuring amqp reporter](#Configuring_amqp_reporter)\n 7. [Configuring graphite reporter](#Configuring_graphite_reporter)\n 8. [Configuring snmp reporter](#Configuring_snmp_reporter)\n6. [Creating custom exometer entries](#Creating_custom_exometer_entries)\n7. [Creating custom probes](#Creating_custom_probes)\n8. [Creating custom reporter plugins](#Creating_custom_reporter_plugins)\n9. [Dependency management](#Dependency_management)\n\n\n### Concepts and Definitions ###\n\nExometer introduces a number of concepts and definitions used\nthroughout the documentation and the code.\n\n\n\n\n#### Metric ####\n\nA metric is a specific measurement sampled inside an Erlang system and\nthen reported to the Exometer system. An example metric would be\n\"transactions_per_second\", or \"memory_usage\".\n\nMetrics are identified by a list of terms, such as given below:\n\n`[ xml_front_end, parser, file_size ]`\n\nA metric is created through a call by the code to be instrumented to\n`exometer:new()`. Once created, the metric can be updated through\n`exometer:update()`, or on its own initiative through the\n`exometer_probe:sample` behavior implementation.\n\n\n#### Data Point ####\n\nEach metric can consist of multiple data points, where each point has\na specific value.\n\nA typical example of data points would be a\n`transactions_per_second` (tps) metric, usually stored as a\nhistogram covering the last couple of minutes of tps samples. Such a\nhistogram would host multiple values, such as `min`, `max`,\n`median`, `mean`, `50_percentile`, `75_percentile`,\netc.\n\nIt is up to the type of the metric, and the data probe backing that\ntype (see below), to specify which data points are available under the\ngiven metric.\n\n\n#### Metric Type ####\n\nThe type of a metric, specified when the metric is created through\n`exometer:new()`, determines which `exometer_entry`\ncallback to use.\n\nThe link between the type and the entry to use is configured\nthrough the `exometer_admin` module, and its associated exometer\ndefaults configuration data.\n\nThe metric type, in other words, is mainly used to map a metric to a\nconfigurable `exometer_entry` callback, but it can also be referenced\nin queries using `exometer:select/1`. An entry callback can also support\nmultiple types (the type is provided as an argument in the callback functions).\n\nExometer provides default mappings for a number of metric types. It is\npossible to select different callbacks for each metric instance, as well\nas modify metrics using callback-specific options. Please see\n[Configuring type - entry maps](#Configuring_type_-_entry_maps) for details on how to do this.\n\n\n#### Entry Callback ####\n\nAn exometer entry callback will receive values reported to a metric through the\n`exometer:update()` call and compile it into one or more data points.\nThe entry callback can either be a counter (implemented natively\nin `exometer`), or a more complex statistical analysis such\nas a uniform distribution or a regular histogram.\n\nThe various outputs from these entries are reported as data points\nunder the given metric.\n\nAn entry can also interface external analytics packages.\n`exometer_folsom`, for example, integrates with the\n`folsom_metrics` package found at [`https://github.com/boundary/folsom`](https://github.com/boundary/folsom).\n\n\n#### Probe ####\n\nProbes are a further specialization of exometer entries that run in\ntheir own Erlang processes and have their own state (like a\ngen_server). A probe is implemented through the `exometer_probe`\nbehavior.\n\nA probe can be used if independent monitoring is needed of,\nfor example, `/proc` trees, network interfaces, and other subsystems\nthat need periodic sampling. In these cases, the\n`exometer_probe:probe_sample()` call is invoked regularly by exometer,\nin the probe's own process, in order to extract data from\nthe given subsystem and add it to the metric's data points.\n\n\n#### Caching ####\n\nMetric and data point values are read with the `exometer:get_value()`\nfunction. In the case of counters, this operation is very fast. With probes,\nthe call results in a synchronous dialog with the probe process, and the\ncost of serving the request depends on the probe implementation and the\nnature of the metric being served.\n\nIf the cost of reading the value is so high that calling the function often\nwould result in prohibitive load, it is possible to cache the value. This is\ndone either explicitly from the probe itself (by calling\n`exometer_cache:write()`), or by specifying the option `{cache, Lifetime}`\nfor the entry. If an entry has a non-zero cache lifetime specified, the\n`get_value()` call will try fetching the cached value before calling the\nactual entry and automatically caching the result.\n\nNote that if `{cache, Lifetime}` is not specified, `exometer:get_value()`\nwill neither read nor write to the cache. It is possible for the probe\nto periodically cache a value regardless of how the cache lifetime is set,\nand the probe may also explicitly read from the cache if it isn't done\nautomatically.\n\n\n#### Subscriptions and Reporters ####\n\nThe subscription concept, managed by `exometer_report` allows metrics\nand their data points to be sampled at given intervals and delivered\nto one or more recipients, which can be either an arbitrary process\nor a Reporter plugin.\n\nEach subscription ties a specific metric-datapoint pair to a reporter\nand an interval (given in milliseconds). The reporter system will, at\nthe given interval, send the current value of the data point to the\nsubscribing reporter. The subscription, with all its parameters,\nis setup through a call to `exometer_report:subscribe()`.\n\nIn the case of processes, subscribed-to values will be delivered as a\nmessage. Modules, which implement the `exometer_report` callback\nbehavior, will receive the plugins as a callbacks within the\n`exometer_report` process.\n\nSubscriptions can either be setup at runtime, through\n`exometer_report:subscribe()` calls, or statically through the\n`exometer_report` configuration data.\n\n\n### Built-in entries and probes ###\n\n\nThere are a number of built-in entries and probes shipped\nwith the Exometer package, as described below:\n\n\n#### counter (exometer native) ####\n\n\nThe counter is implemented directly in `exometer` to provide simple\ncounters. A call to `exometer:update()` will add the provided value\nto the counter.\n\nThe counter can be reset to zero through `exometer:reset()`.\n\nThe available data points under a metric using the counter entry\nare `value` and `ms_since_reset`.\n\n\n#### fast_counter (exometer native) ####\n\nA fast counter implements the counter functionality, through the\n`trace_info` system, yielding a speed increase of about 3.5 in\ncomparison to the regular counter.\n\nThe tradeoff is that running tracing and/or debugging may interfere\nwith the counter functionality.\n\nA call to `exometer:update()` will add the provided value to the\ncounter.\n\nThe counter can be reset to zero through `exometer:reset()`.\n\nThe available data points under a metric using the fast_counter\nentry are `value` and `ms_since_reset`.\n\n\n#### gauge (exometer native) ####\n\nThe gauge is implemented directly in `exometer` to provide simple\ngauges. A call to `exometer:update()` will set the gauge's value\nto the provided value. That is, the value of the gauge entry is\nalways the most recently provided value.\n\nThe gauge can be reset to zero through `exometer:reset()`.\n\nThe available data points under a metric using the gauge entry\nare `value` and `ms_since_reset`.\n\n\n#### histogram (probe) ####\n\nThe histogram probe stores a given number of updates, provided through\n`exometer:update()`, in a histogram. The histogram maintains a log\nderived from all values received during a configurable time span and\nprovides min, max, median, mean, and percentile analysis data points\nfor the stored data.\n\nExometer supports a number of different histogram implementations, each\nwith different performance and accuracy trade-offs. \n\nIn order to save memory, the histogram is divided into equal-sized\ntime slots, where each slot spans a settable interval. All values\nreceived during a time slot will be averaged into a single value to be\nstored in the histogram once the time slot expires. The averaging\nfunction (which can be replaced by the caller), allows for\nhigh-frequency update metrics to have their resolution traded against\nresource consumption.\n\n\n#### exometer_uniform (probe) ####\n\nThe uniform probe provides a uniform sample over a pool of values\nprovided through `exometer:update()`. When the pool reaches its configurable\nmax size, existing values will be replaced at random to make space for\nnew values. Much like `exometer_histogram`, the uniform probe\nprovides min, max, median, mean, and percentile analysis data points\nfor the stored data.\n\n\n#### exometer_spiral (probe) ####\n\nThe spiral probe maintains the total sum of all values stored in its\nhistogram. The histogram has a configurable time span, all values\nprovided to the probe, through `exometer:update()`, within that time\nspan will be summed up and reported. If, for example, the histogram\ncovers 60 seconds, the spiral probe will report the sum of all\nvalues reported during the last minute.\n\nThe grand total of all values received during the lifetime of the\nprobe is also available.\n\n\n#### exometer_folsom [entry] ####\n\n`exometer_folsom` is an entry behavior which implements most metric types\nsupported by the [folsom](https://github.com/boundary/folsom)\nmetrics package: Specifically, the metric types `counter`, `spiral`,\n`histogram`, `meter`, `meter_reader`, `gauge`, `duration` and `history`.\n\nThe folsom entry integrates with the folsom metrics package provided\nby the boundary repo at github. Updated values sent to the folsom entry\ncan be forwarded to folsom's counter, histogram, duration, meter,\nand spiral.\n\nFolsom integration is provided as a backup. New code using Exometer\nshould use the native probes that duplicate folsom.\n\n\n#### exometer_function [entry] ####\n\nThe function entry allows for an existing erlang function to be wrapped\nas an exometer entry. The [`exometer_function`](https://github.com/Feuerlabs/exometer_core/blob/master/doc/exometer_function.md) module supports a number\nof options for passing arguments and matching out data points from the\nresult.\n\nThe function entry provides an easy way of integrating an external\nsystem without having to write a complete entry.\n\n\n### Built in Reporters ###\n\nExometer ships with some built-in reporters which can be used to forward updated\nmetrics and their data points to external systems. They can also\nserve as templates for custom-developed reporters.\n\n\n#### exometer_report_graphite ####\n\nThe graphite reporter uses the TCP/IP protocol to forward\nsubscribed-to metrics and data points to a graphite server, such as\nthe one provided by [`http://hostedgraphite.com`](http://hostedgraphite.com). When the graphite\nreporter receives a metric-datapoint value (subscribed to through\n`exometer_report:subscriber()`), the reporter will immediately\nforward the key-value pair to the graphite server.\n\n\n#### exometer_report_opentsdb ####\n\nThe OpenTSDB reporter sends metrics to an OpenTSDB server using\nthe telnet API. All subscribed-to metric-datapoint values received\nby the reporter are immediately forwarded to OpenTSDB.\n\nIf the OpenTSDB connection is lost, the reporter will attempt to reconnect to it\nat a configurable interval.\n\nThe data sent to OpenTSDB will be formatted as follows:\n\n```\nput metric timestamp value host=host type=datapoint\n```\n\nWhere the value for the host tag will be the configured host in the reporter\nconfiguration (defaults to the value returned by `netadm:localhost`), and\ndatapoint tags as specified by the subscriber.\n\nPlease see [Configuring opentsdb reporter](#Configuring_opentsdb_reporter) for details on the\napplication environment parameters listed above.\n\n\n#### exometer_report_amqp ####\n\nThe AMQP reporter sends metrics to an AMQP broker as a json-encoded payload. All\nsubscribed-to metric-datapoint values received by the reporter are forwarded to AMQP.\n\nIf the AMQP connection is lost, the reporter will attempt to reconnect to it\nat a configurable interval.\n\nThe data sent to AMQP will be formatted as follows:\n\n```\n{\n \"type\":\"exometer_metric\",\n \"body\":\n {\"name\":\"messages_per_second\",\n \"value\":0,\"timestamp\":1414006826,\n \"host\":\"testhost\",\n \"instance\":\"max\"}\n}\n```\n\nWhere the value for the host tag will be the configured host in the reporter\nconfiguration (defaults to the value returned by `netadm:localhost`), the\ninstance tag represents the datapoint for the metric.\n\nPlease see [Configuring amqp reporter](#Configuring_amqp_reporter) for details on the\napplication environment parameters listed above.\n\n\n#### exometer_report_snmp ####\n\nThe SNMP reporter enables the export of metrics and their datapoints to SNMP managers.\nThe export needs to be enabled for each metric through their options.\nMoreover, SNMP notifications can be created using the options to send periodic reports\non datapoints to SNMP managers. All SNMP protocol handling is done by the snmp application\nshipped with Erlang/OTP. Thus, the snmp application needs to be started and\nthe local SNMP master agent needs to be configured correctly for SNMP export to work\nproperly.\n\nTo configure SNMP export for a single metric use these options:\n\n+ `{snmp, disabled}` (default) Disables SNMP export for the metric. Same as not specifying the option at all.\n\n+ `{snmp, []}` Enables SNMP export for the metric. No subscriptions are setup.\n\n+ `{snmp, [{Datapint, Interval}]}` Enables SNMP export for the metric. Subscriptions are setup for the given Datapoint/Interval pairs. Each subscription report will be forwarded to SNMP mangers as notifications.\n\n+ `{snmp, [{Datapint, Interval, Extra}]}` Same as above, but using an addition extra identification for the subscriptions. Allow the creation ofmultiple subscriptions for a single datapoint.\n\nPlease see [Configuring snmp reporter](#Configuring_snmp_reporter) for details on how to configure the\nSNMP reporter.\n\n\n### Instrumenting Erlang code ###\n\nThe code using Exometer needs to be instrumented in order to setup and\nuse metrics reporting.\n\n\n#### Exometer Start ####\n\nThe system using Exometer must start the `exometer` application prior to using it:\n\n```erlang\n\napplication:start(lager),\napplication:start(exometer).\n```\n\nNote that dependent applications need to be started first. On newer OTP versions\n(R16B or later), you can use `application:ensure_all_started(exometer)`.\n\nFor testing, you can also use [`exometer:start/0`](https://github.com/Feuerlabs/exometer_core/blob/master/doc/exometer.md#start-0).\n\nIf you make use of e.g. folsom metrics, you also need to start `folsom`. Exometer\nwill not do that automatically, nor does it contain an application dependency for it.\n\nSee [Configuring Exometer](#Configuring_Exometer) for details on configuration data\nformat.\n\n\n#### Creating metrics ####\n\nA metric, can be created through a call to\n\n```erlang\n\nexometer:new(Name, Type)\n```\n\n`Name` is a list of atoms, uniquely identifying the metric created.\nThe type of the metric, specified by `Type` will be mapped\nto an exometer entry through the table maintained by\n`exometer_admin` Please see the [Configuring type - entry\nmaps](#Configuring_type_-_entry_maps) for details.\n\nThe resolved entry to use will determine the data points available\nunder the given metric.\n\n\n#### Deleting metrics ####\n\nA metric previously created with `exometer:new()` can be deleted by\n`exometer:delete()`.\n\nAll subscriptions to the deleted metrics will be cancelled.\n\n\n#### Setting metric values ####\n\nA created metric can have its value updated through the\n`exometer:update()` function:\n\n```erlang\n\nexometer:update(Name, Value)\n```\n\nThe `Name` parameter is the same atom list provided to a previous\n`exometer:new()` call. The `Value` is an arbitrarty element that is\nforwarded to the `exometer:update()` function of the entry/probe that the\nmetric is mapped to.\n\nThe receiving entry/probe will process the provided value and modify\nits data points accordingly.\n\n\n#### Retrieving metric values ####\n\nExometer-using code can at any time retrieve the data point values\nassociated with a previously created metric. In order to find out which\ndata points are available for a metric, the following call can be used:\n\n```erlang\n\nexometer:info(Name, datapoints)\n```\n\nThe `Name` parameter is the same atom list provided to a previous\n`exometer:new()` call. The call will return a list of data point\natoms that can then be provided to `exometer:get_value()` to\nretrieve their actual value:\n\n```erlang\n\nexometer:get_value(Name, DataPoint)\n```\n\nThe `Name` paramer identifies the metric, and `DataPoints`\nidentifies the data points (returned from the previous `info()` call)\nto retrieve the value for.\n\nIf no DataPoints are provided, the values of a default list of data points,\ndetermined by the backing entry / probe, will be returned.\n\n\n#### Setting up subscriptions ####\n\nA subscription can either be statically configured, or dynamically\nsetup from within the code using Exometer. For details on statically\nconfigured subscriptions, please see [Configuring static subscriptions](#Configuring_static_subscriptions).\n\nA dynamic subscription can be setup with the following call:\n\n```erlang\n\nexometer_report:subscribe(Recipient, Metric, DataPoint, Inteval)\n```\n\n`Recipient` is the name of a reporter.\n\n\n#### Set metric options ####\n\n\nEach created metric can have options setup for it through the following call:\n\n```erlang\n\nexometer:setopts(Name, Options)\n```\n\nThe `Name` paramer identifies the metric to set the options for, and\nOptions is a proplist (`[{ Key, Value },...]`) with the options to be\nset.\n\nExometer looks up the the backing entry that hosts the metric with the given Name, and will\ninvoke the entry\\'s `setopts/4` function to set the actual options. Please see the\n`setopts/4` function for the various entries for details.\n\n\n### Configuring Exometer ###\n\nExometer defaults can be changed either through OTP application environment\nvariables or through the use of Basho's `cuttlefish`\n([`https://github.com/basho/cuttlefish`](https://github.com/basho/cuttlefish)).\n\n\n#### Configuring type - entry maps ####\n\nThe dynamic method of configuring defaults for `exometer` entries is:\n\n```erlang\n\nexometer_admin:set_default(NamePattern, Type, Default)\n```\n\nWhere `NamePattern` is a list of terms describing what is essentially\na name prefix with optional wildcards (`'_'`). A pattern that\nmatches any legal name is `['_']`.\n\n`Type` is an atom defining a type of metric. The types already known to\n`exometer`, `counter`, `fast_counter`, `ticker`, `uniform`, `histogram`,\n`spiral`, `netlink`, and `probe` may be redefined, but other types can be\ndescribed as well.\n\n`Default` is either an `#exometer_entry{}` record (unlikely), or a list of\n`{Key, Value}` options, where the keys correspond to `#exometer_entry` record\nattribute names. The following attributes make sense to preset:\n\n```erlang\n\n{module, atom()} % the callback module\n{status, enabled | disabled} % operational status of the entry\n{cache, non_neg_integer()} % cache lifetime (ms)\n{options, [{atom(), any()}]} % entry-specific options\n```\n\nBelow is an example, from `exometer/priv/app.config`:\n\n```erlang\n\n{exometer, [\n {defaults, [\n {['_'], function , [{module, exometer_function}]},\n {['_'], counter , [{module, exometer}]},\n {['_'], histogram, [{module, exometer_histogram}]},\n {['_'], spiral , [{module, exometer_spiral}]},\n {['_'], duration , [{module, exometer_folsom}]},\n {['_'], meter , [{module, exometer_folsom}]},\n {['_'], gauge , [{module, exometer_folsom}]}\n ]}\n]}\n```\n\nIn systems that use CuttleFish, the file `exometer/priv/exometer.schema`\ncontains a schema for default settings. The setup corresponding to the above\ndefaults would be as follows:\n\n```ini\n\nexometer.template.function.module = exometer_function\nexometer.template.counter.module = exometer\nexometer.template.histogram.module = exometer_histogram\nexometer.template.spiral.module = exometer_spiral\nexometer.template.duration.module = exometer_folsom\nexometer.template.meter.module = exometer_folsom\nexometer.template.gauge.module = exometer_folsom\n```\n\n\n#### Configuring statically defined entries ####\n\nUsing the `exometer` environment variable `predefined`, entries can be added\nat application startup. The variable should have one of the following values:\n\n* `{script, File}` - `File` will be processed using `file:script/2`. The return\n value (the result of the last expression in the script) should be a list of`{Name, Type, Options}` tuples.\n\n* `{apply, M, F, A}` - The result of `apply(M, F, A)` should be `{ok, L}` where`L` is a list of `{Name, Type, Options}` tuples.\n\n* `L`, where L is a list of `{Name, Type, Options}` tuples or extended\ninstructions (see below).\n\nThe list of instructions may include:\n\n* `{delete, Name}` - deletes `Name` from the exometer registry.\n\n* `{select_delete, Pattern}` - applies a select pattern and\ndeletes all matching entries.\n\n* `{re_register, {Name, Type, Options}}` - redefines an entry if present,\notherwise creates it.\n\nExometer will also scan all loaded applications for the environment\nvariables `exometer_defaults` and `exometer_predefined`, and process\nas above. If an application is loaded and started after exometer has started,\nit may call the function `exometer:register_application()` or\n`exometer:register_application(App)`. This function will do nothing if\nexometer isn't already running, and otherwise process the `exometer_defaults`\nand `exometer_predefined` variables as above. The function can also be\ncalled during upgrade, as it will re-apply the settings each time.\n\n\n#### Configuring static subscriptions ####\n\n\nStatic subscriptions, which are automatically setup at exometer\nstartup without having to invoke `exometer_report:subscribe()`, are\nconfigured through the report sub section under exometer.\n\nBelow is an example, from `exometer/priv/app.config`:\n\n```erlang\n\n{exometer, [\n {report, [\n {subscribers, [\n {exometer_report_collectd, [db, cache, hits], mean, 2000, true},\n {exometer_report_collectd, [db, cache, hits], max, 5000, false}\n ]}\n ]}\n]}\n```\n\nThe `report` section configures static subscriptions and reporter\nplugins. See [Configuring reporter plugins](#Configuring_reporter_plugins) for details on\nhow to configure individual plugins.\n\nThe `subscribers` sub-section contains all static subscriptions to be\nsetup att exometer applications start. Each tuple in the prop list\nshould be of one of the following formats:\n\n* `{Reporter, Metric, DataPoint, Interval}`\n\n* `{Reporter, Metric, DataPoint, Interval, RetryFailedMetrics}`\n\n* `{Reporter, Metric, DataPoint, Interval, RetryFailedMetrics, Extra}`\n\n* `{apply, {M, F, A}}`\n\n* `{select, {MatchPattern, DataPoint, Interval [, Retry [, Extra] ]}}`\n\nIn the case of `{apply, M, F, A}`, the result of `apply(M, F, A)` must\nbe a list of `subscribers` tuples.\n\nIn the case of `{select, Expr}`, a list of metrics is fetched using\n`exometer:select(MatchPattern)`, where the result must be on the form\n`{Key, Type, Status}` (i.e. what corresponds to `'$_'`).\nThe rest of the items will be applied to each of the matching entries.\n\nThe meaning of the above tuple elements is:\n\n+ `Reporter :: module()` Specifies the reporter plugin module, such as`exometer_report_collectd` that is to receive updated metric's data\npoints.\n\n+ `Metric :: [atoms()]` Specifies the path to a metric previously created with an`exometer:new()` call.\n\n+ `DataPoint` :: atom() | [atom()]' Specifies the data point within the given metric to send to the\n receiver. The data point must match one of the data points returned by`exometer:info(Name, datapoints)` for the given metrics name.\n\n+ `Interval` :: integer()' (milliseconds) Specifies the interval, in milliseconds, between each update of the\ngiven metric's data point. At the given interval, the data point will\nbe samples, and the result will be sent to the receiver.\n\n+ `RetryFailedMetrics :: boolean()` Specifies if the metric should be continued to be reported\n even if it is not found during a reporting cycle. This would be\n the case if a metric is not created by the time it is reported for\n the first time. If the metric will be created at a later time,\n this value should be set to true. Set this value to false if all\n attempts to report the metric should stop if when is not found.\n The default value is `true`.\n\n+ `Extra :: any()` Provides a means to pass along extra information for a given\n subscription. An example is the `syntax` option for the SNMP reporter,\n in which case `Extra` needs to be a property list.\n\nExample configuration in sys.config, using the `{select, Expr}` pattern:\n\n```erlang\n\n[\n {exometer, [\n {predefined,\n [{[a,1], counter, []},\n {[a,2], counter, []},\n {[b,1], counter, []},\n {[c,1], counter, []}]},\n {report,\n [\n {reporters,\n [{exometer_report_tty, []}]},\n {subscribers,\n [{select, {[{ {[a,'_'],'_','_'}, [], ['$_']}],\n exometer_report_tty, value, 1000}}]}\n ]}\n ]}\n].\n\n```\n\nThis will activate a subscription on `[a,1]` and `[a,2]` in the\n`exometer_report_tty` reporter, firing once per second.\n\n\n#### Configuring reporter plugins ####\n\n\nThe various reporter plugins to be loaded by exometer are configured\nin the `report` section under `reporters`\n\nEach reporter has an entry named after its module, and the content of\nthat entry is dependent on the reporter itself. The following chapters\nspecifies the configuration parameters for the reporters shipped with\nexometer.\n\n\n#### Configuring opentsdb reporter ####\n\n\nBelow is an example of the opentsdb reporter application environment, with\nits correct location in the hierarchy:\n\n```erlang\n\n{exometer, [\n {report, [\n {reporters, [\n {exometer_report_opentsdb, [\n {reconnect_interval, 10},\n {connect_timeout, 8000},\n {hostname, \"testhost\"},\n {host, {\"127.0.0.1\", 4242}}\n ]}\n ]}\n ]}\n]}\n```\n\nThe following attributes are available for configuration:\n\n+ `reconnect_interval` (seconds - default: 30) Specifies the duration between each reconnect attempt to an opentsdb\nserver that is not available. Should the server either be unavailable\nat exometer startup, or become unavailable during exometer's\noperation, exometer will attempt to reconnect at the given number of\nseconds.\n\n+ `connect_timeout` (milliseconds - default: 5000) Specifies how long the opentsdb reporter plugin shall wait for a\nsocket connection to complete before timing out. A timed out\nconnection attempt will be retried after the reconnect interval has\npassed see item 1 above).\n\n+ `hostname` (string - default: `net_adm:localhost()`) Specifies the host name to use for the host tag in the OpenTSDB tags.\n Please see [Configuring opentsdb reporter](#Configuring_opentsdb_reporter) for details.\n\n+ `host` (ip - default: {\"127.0.0.1\", 4242}) Specifies the host and port to connect to OpenTSDB.\n\n\n#### Configuring amqp reporter ####\n\n\nBelow is an example of the amqp reporter application environment, with\nits correct location in the hierarchy:\n\n```erlang\n\n{exometer, [\n {report, [\n {reporters, [\n {exometer_report_amqp, [\n {reconnect_interval, 10},\n {hostname, \"testhost\"},\n {amqp_url, \"amqp://user:pass@host:5672/%2f\"},\n\t\t{exchange, \"metrics\"},\n\t\t{routing_key, \"metrics\"},\n\t\t{buffer_size, 0}\n ]}\n ]}\n ]}\n]}\n```\n\nThe following attributes are available for configuration:\n\n+ `reconnect_interval` (seconds - default: 30) Specifies the duration between each reconnect attempt to an amqp\nbroker that is not available. Should the server either be unavailable\nat exometer startup, or become unavailable during exometer's\noperation, exometer will attempt to reconnect at the given number of\nseconds.\n\n+ `hostname` (string - default: `net_adm:localhost()`) Specifies the host name to use for the host property in the JSON payload.\n Please see [Configuring amqp reporter](#Configuring_amqp_reporter) for details.\n\n+ `amqp_url` (string - default: `amqp://guest:guest@localhost:5672/%2f`) Specifies the amqp url to connect to.\n\n+ `exchange` (string - default: `exometer`) Specifies the exchange to publish messages to.\n\n+ `routing_key` (string - default: `exometer`) Specifies the routing key to use when publishing messages.\n\n+ `buffer_size` (integer - default: `0`) Specifies the size in bytes of payload to buffer before sending to AMQP.\n\n\n#### Configuring graphite reporter ####\n\n\nBelow is an example of the a graphite reporter application environment, with\nits correct location in the hierarchy:\n\n```erlang\n\n{exometer, [\n {report, [\n {reporters, [\n {exometer_report_graphite, [\n {connect_timeout, 5000},\n {prefix, \"web_stats\"},\n {host, \"carbon.hostedgraphite.com\"},\n {port, 2003},\n {api_key, \"267d121c-8387-459a-9326-000000000000\"}\n ]}\n ]}\n ]}\n]}\n```\n\nThe following attributes are available for configuration:\n\n+ `connect_timeout` (milliseconds - default: 5000) Specifies how long the graphie reporter plugin shall wait for a tcp\nconnection to complete before timing out. A timed out connection will\nnot be reconnected to automatically. (To be fixed.)\n\n+ `prefix` (string - default: \"\") Specifies an optional prefix to prepend all metric names with before\nthey are sent to the graphite server.\n\n+ `host` (string - default: \"carbon.hostedgraphite.com\") Specifies the name (or IP address) of the graphite server to report to.\n\n+ `port` (integer - default: 2003) Specifies the TCP port on the given graphite server to connect to.\n\n+ `api_key` (string - default: n/a) Specifies the api key to use when reporting to a hosted graphite server.\n\nIf `prefix` is not specified, but `api_key` is, each metrics will be reported as `ApiKey.Metric`.\n\nIf `prefix` is specified, but `api_key` is not, each metrics will be reported as `Prefix.Metric`.\n\nif neither `prefix` or `api_key` is specified, each metric will be reported simply as `Metric`.\n\n\n#### Configuring snmp reporter ####\n\n\nBelow is an example of the a snmp reporter application environment, with\nits correct location in the hierarchy:\n\n```erlang\n\n{exometer, [\n {report, [\n {reporters, [\n {exometer_report_snmp, [\n {mib_template, \"priv/MYORG-EXOMETER-METRICS.mib\"},\n {mib_dir, \"/tmp/exometer\"}\n ]}\n ]}\n ]}\n]}\n```\n\nThe following attributes are available for configuration:\n\n+ `mib_template` (string - default: \"mibs/EXOMETER-METRICS-MIB.mib\") Specifies where to find the MIB template used for dynamically assembline an internal MIB. Take a look at the MIB template shipped with Exometer for reference in case you want to define your own template.\n\n+ `mib_dir` (string - default: \"tmp/exometer_report_snmp\") Specifies temporary direction which will be used by Exometer to store dymanically created MIB files.\n\n\n### Creating custom exometer entries ###\n\n\nPlease see @see exometer_entry documentation for details.\n\n\n### Creating custom probes ###\n\n\nPlease see @see exometer_probe documentation for details.\n\n\n### Creating custom reporter plugins ###\n\n\nPlease see @see exometer_report documentation for details.\n\n\n### Dependency management ###\n\n\nExometer dependencies can be controlled using the `EXOMETER_PACKAGES`\nunix environment variable: a string listing packages or applications to\neither keep or remove, separated using space, tab or comma.\n\n\n#### Syntax ####\n\n+ `(Package)` - use `Package` as a base. This will implicitly exclude all\n applications not included in `Package`. See below for supported packages.\n\n+ `+(Package)` - add applications included in `Package`.\n\n+ `-(Package)` - remove applications in `Package` (except mandatory deps).\n\n+ `App` - keep application `App`.\n\n+ `+App` - keep application `App`.\n\n+ `-App` - exclude application `App`.\n\n\n#### Supported packages ####\n\n+ `minimal` - only the mandatory deps: `lager`, `parse_trans`, `setup`.\n\n+ `basic` - (mandatory deps and) `folsom`.\n\n+ `amqp` - (mandatory deps and) `amqp_client`, `jiffy`.\n\n+ `full` - all of the above, plus `afunix` and `netlink`.\n\nExample - use only basic deps plus `afunix`\n\n```\n EXOMETER_PACKAGES=\"(basic), +afunix\" make\n```\n\nExample - use all deps except the AMQP-related deps:\n\n```\n export EXOMETER_PACKAGES=\"(full) -(amqp)\"\n```\n\n\n#### Conditional defines ####\n\nFor each optional dependency that is included, a macro is defined,\nnamed `dep_App` - e.g. `dep_afunix`. Developers must not include\ncompile-time dependencies to optional applications, without checking\nthe corresponding macro and ensuring that the module compiles even\nwhen the dependent application is not included. See `exometer_report_amqp.erl`\nfor an example.\n\n\n#### Customizing rebar.config ####\n\nThe OS environment variables `EXOMETER_CONFIG_PREPROCESS` and\n`EXOMETER_CONFIG_POSTPROCESS` can be used to insert a script, similar to\n`rebar.config.script` in the processing flow of the exometer build.\n\nAs the names imply, the script given by `EXOMETER_CONFIG_PREPROCESS` (if any)\nwill be run before exometer does any processing of its own, and the\n`EXOMETER_CONFIG_POSTPROCESS` script (if any) will be run after all other\nprocessing is complete.\nThings that could be done in preprocessing: re-targeting a dependency,\nmodifying the list of predefined packages, etc.\n\n## Modules ##\n\n\n
\n\n\n\n\n## Function Details ##\n\n\n\n### exometer_call/3 ###\n\n`exometer_call(Unknown, From, St) -> any()`\n\n\n\n\n### exometer_cast/2 ###\n\n`exometer_cast(Unknown, St) -> any()`\n\n\n\n\n### exometer_info/2 ###\n\n`exometer_info(Unknown, St) -> any()`\n\n\n\n\n### exometer_init/1 ###\n\n`exometer_init(Opts) -> any()`\n\n\n\n\n### exometer_newentry/2 ###\n\n`exometer_newentry(Entry, St) -> any()`\n\n\n\n\n### exometer_report/5 ###\n\n`exometer_report(Metric, DataPoint, Extra, Value, St) -> any()`\n\n\n\n\n### exometer_setopts/4 ###\n\n`exometer_setopts(Metric, Options, Status, St) -> any()`\n\n\n\n\n### exometer_subscribe/5 ###\n\n`exometer_subscribe(Metric, DataPoint, Extra, Interval, St) -> any()`\n\n\n\n\n### exometer_terminate/2 ###\n\n`exometer_terminate(X1, X2) -> any()`\n\n\n\n\n### exometer_unsubscribe/4 ###\n\n`exometer_unsubscribe(Metric, DataPoint, Extra, St) -> any()`\n\n\n"
},
{
"path": "doc/overview.edoc",
"content": "@author Ulf Wiger \n@author Magnus Feuer \n@copyright 2014 Basho Technologies, Inc. All Rights Reserved.\n@version {@version}\n@title Exometer - Erlang instrumentation package\n\n@doc\n\n[](https://travis-ci.org/Feuerlabs/exometer)\n\nNOTE: Exometer has been split into exometer_core, and exometer (as well as separate reporter applications). The latest monolithic version of Exometer is 1.1.\n\nThe Exometer package allows for easy and efficient instrumentation of\nErlang code, allowing crucial data on system performance to be\nexported to a wide variety of monitoring systems.\n\nExometer comes with a set of pre-defined monitor components, and can\nbe expanded with custom components to handle new types of Metrics, as\nwell as integration with additional external systems such as\ndatabases, load balancers, etc.\n\nThis document gives a high level overview of the Exometer system. For\ndetails, please see the documentation for individual modules, starting\nwith `exometer'.\n\nNote the section on {@section Dependency Management} for how to deal with\noptional packages, both users and developers.\n\n== Table of Content ==\n1. {@section Concept and definitions}\n 1. {@section Metric}\n 2. {@section Data Point}\n 3. {@section Metric Type}\n 4. {@section Entry Callback}\n 5. {@section Probe}\n 6. {@section Caching}\n 7. {@section Subscriptions and Reporters}\n2. {@section Built-in entries and probes}\n 1. {@section counter (exometer native)}\n 2. {@section fast_counter (exometer native)}\n 3. {@section gauge (exometer native)}\n 4. {@section exometer_histogram (probe)}\n 5. {@section exometer_uniform (probe)}\n 6. {@section exometer_spiral (probe)}\n 7. {@section exometer_folsom [entry]}\n 8. {@section exometer_function [entry]}\n3. {@section Built in Reporters}\n 1. {@section exometer_report_graphite}\n 2. {@section exometer_report_opentsdb}\n 3. {@section exometer_report_amqp}\n 4. {@section exometer_report_snmp}\n4. {@section Instrumenting Erlang code}\n 1. {@section Exometer Start}\n 2. {@section Creating metrics}\n 3. {@section Deleting metrics}\n 4. {@section Setting metric values}\n 5. {@section Retrieving metric values}\n 6. {@section Setting up subscriptions}\n 7. {@section Set metric options}\n5. {@section Configuring Exometer}\n 1. {@section Configuring type - entry maps}\n 2. {@section Configuring statically defined entries}\n 3. {@section Configuring static subscriptions}\n 4. {@section Configuring reporter plugins}\n 5. {@section Configuring opentsdb reporter}\n 6. {@section Configuring amqp reporter}\n 7. {@section Configuring graphite reporter}\n 8. {@section Configuring snmp reporter}\n6. {@section Creating custom exometer entries}\n7. {@section Creating custom probes}\n8. {@section Creating custom reporter plugins}\n9. {@section Dependency management}\n\n== Concepts and Definitions ==\n\nExometer introduces a number of concepts and definitions used\nthroughout the documentation and the code.\n\n\n\n=== Metric ===\n\nA metric is a specific measurement sampled inside an Erlang system and\nthen reported to the Exometer system. An example metric would be\n\"transactions_per_second\", or \"memory_usage\".\n\nMetrics are identified by a list of terms, such as given below:\n\n`[ xml_front_end, parser, file_size ]'\n\nA metric is created through a call by the code to be instrumented to\n`exometer:new()'. Once created, the metric can be updated through\n`exometer:update()', or on its own initiative through the\n`exometer_probe:sample' behavior implementation.\n\n=== Data Point ===\n\nEach metric can consist of multiple data points, where each point has\na specific value.\n\nA typical example of data points would be a\n`transactions_per_second' (tps) metric, usually stored as a\nhistogram covering the last couple of minutes of tps samples. Such a\nhistogram would host multiple values, such as `min', `max',\n`median', `mean', `50_percentile', `75_percentile',\netc.\n\nIt is up to the type of the metric, and the data probe backing that\ntype (see below), to specify which data points are available under the\ngiven metric.\n\n\n=== Metric Type ===\n\nThe type of a metric, specified when the metric is created through\n`exometer:new()', determines which `exometer_entry'\ncallback to use.\n\nThe link between the type and the entry to use is configured\nthrough the `exometer_admin' module, and its associated exometer\ndefaults configuration data.\n\nThe metric type, in other words, is mainly used to map a metric to a\nconfigurable `exometer_entry' callback, but it can also be referenced\nin queries using `exometer:select/1'. An entry callback can also support\nmultiple types (the type is provided as an argument in the callback functions).\n\nExometer provides default mappings for a number of metric types. It is\npossible to select different callbacks for each metric instance, as well\nas modify metrics using callback-specific options. Please see\n{@section Configuring type - entry maps} for details on how to do this.\n\n=== Entry Callback ===\n\nAn exometer entry callback will receive values reported to a metric through the\n`exometer:update()' call and compile it into one or more data points.\nThe entry callback can either be a counter (implemented natively\nin `exometer'), or a more complex statistical analysis such\nas a uniform distribution or a regular histogram.\n\nThe various outputs from these entries are reported as data points\nunder the given metric.\n\nAn entry can also interface external analytics packages.\n`exometer_folsom', for example, integrates with the\n`folsom_metrics' package found at [https://github.com/boundary/folsom].\n\n=== Probe ===\n\nProbes are a further specialization of exometer entries that run in\ntheir own Erlang processes and have their own state (like a\ngen_server). A probe is implemented through the `exometer_probe'\nbehavior.\n\nA probe can be used if independent monitoring is needed of,\nfor example, `/proc' trees, network interfaces, and other subsystems\nthat need periodic sampling. In these cases, the\n`exometer_probe:probe_sample()' call is invoked regularly by exometer,\nin the probe's own process, in order to extract data from\nthe given subsystem and add it to the metric's data points.\n\n=== Caching ===\n\nMetric and data point values are read with the `exometer:get_value()'\nfunction. In the case of counters, this operation is very fast. With probes,\nthe call results in a synchronous dialog with the probe process, and the\ncost of serving the request depends on the probe implementation and the\nnature of the metric being served.\n\nIf the cost of reading the value is so high that calling the function often\nwould result in prohibitive load, it is possible to cache the value. This is\ndone either explicitly from the probe itself (by calling\n`exometer_cache:write()'), or by specifying the option `{cache, Lifetime}'\nfor the entry. If an entry has a non-zero cache lifetime specified, the\n`get_value()' call will try fetching the cached value before calling the\nactual entry and automatically caching the result.\n\nNote that if `{cache, Lifetime}' is not specified, `exometer:get_value()'\nwill neither read nor write to the cache. It is possible for the probe\nto periodically cache a value regardless of how the cache lifetime is set,\nand the probe may also explicitly read from the cache if it isn't done\nautomatically.\n\n=== Subscriptions and Reporters ===\n\nThe subscription concept, managed by `exometer_report' allows metrics\nand their data points to be sampled at given intervals and delivered\nto one or more recipients, which can be either an arbitrary process\nor a Reporter plugin.\n\nEach subscription ties a specific metric-datapoint pair to a reporter\nand an interval (given in milliseconds). The reporter system will, at\nthe given interval, send the current value of the data point to the\nsubscribing reporter. The subscription, with all its parameters,\nis setup through a call to `exometer_report:subscribe()'.\n\nIn the case of processes, subscribed-to values will be delivered as a\nmessage. Modules, which implement the `exometer_report' callback\nbehavior, will receive the plugins as a callbacks within the\n`exometer_report' process.\n\nSubscriptions can either be setup at runtime, through\n`exometer_report:subscribe()' calls, or statically through the\n`exometer_report' configuration data.\n\n\n== Built-in entries and probes ==\nThere are a number of built-in entries and probes shipped\nwith the Exometer package, as described below:\n\n=== counter (exometer native) ===\nThe counter is implemented directly in `exometer' to provide simple\ncounters. A call to `exometer:update()' will add the provided value\nto the counter.\n\nThe counter can be reset to zero through `exometer:reset()'.\n\nThe available data points under a metric using the counter entry\nare `value' and `ms_since_reset'.\n\n=== fast_counter (exometer native) ===\n\nA fast counter implements the counter functionality, through the\n`trace_info' system, yielding a speed increase of about 3.5 in\ncomparison to the regular counter.\n\nThe tradeoff is that running tracing and/or debugging may interfere\nwith the counter functionality.\n\nA call to `exometer:update()' will add the provided value to the\ncounter.\n\nThe counter can be reset to zero through `exometer:reset()'.\n\nThe available data points under a metric using the fast_counter\nentry are `value' and `ms_since_reset'.\n\n=== gauge (exometer native) ===\n\nThe gauge is implemented directly in `exometer' to provide simple\ngauges. A call to `exometer:update()' will set the gauge's value\nto the provided value. That is, the value of the gauge entry is\nalways the most recently provided value.\n\nThe gauge can be reset to zero through `exometer:reset()'.\n\nThe available data points under a metric using the gauge entry\nare `value' and `ms_since_reset'.\n\n=== histogram (probe) ===\n\nThe histogram probe stores a given number of updates, provided through\n`exometer:update()', in a histogram. The histogram maintains a log\nderived from all values received during a configurable time span and\nprovides min, max, median, mean, and percentile analysis data points\nfor the stored data.\n\nExometer supports a number of different histogram implementations, each\nwith different performance and accuracy trade-offs.\n\nIn order to save memory, the histogram is divided into equal-sized\ntime slots, where each slot spans a settable interval. All values\nreceived during a time slot will be averaged into a single value to be\nstored in the histogram once the time slot expires. The averaging\nfunction (which can be replaced by the caller), allows for\nhigh-frequency update metrics to have their resolution traded against\nresource consumption.\n\n=== exometer_uniform (probe) ===\n\nThe uniform probe provides a uniform sample over a pool of values\nprovided through `exometer:update()'. When the pool reaches its configurable\nmax size, existing values will be replaced at random to make space for\nnew values. Much like `exometer_histogram', the uniform probe\nprovides min, max, median, mean, and percentile analysis data points\nfor the stored data.\n\n=== exometer_spiral (probe) ===\n\nThe spiral probe maintains the total sum of all values stored in its\nhistogram. The histogram has a configurable time span, all values\nprovided to the probe, through `exometer:update()', within that time\nspan will be summed up and reported. If, for example, the histogram\ncovers 60 seconds, the spiral probe will report the sum of all\nvalues reported during the last minute.\n\nThe grand total of all values received during the lifetime of the\nprobe is also available.\n\n=== exometer_folsom [entry] ===\n\n`exometer_folsom' is an entry behavior which implements most metric types\nsupported by the folsom\nmetrics package: Specifically, the metric types `counter', `spiral',\n`histogram', `meter', `meter_reader', `gauge', `duration' and `history'.\n\nThe folsom entry integrates with the folsom metrics package provided\nby the boundary repo at github. Updated values sent to the folsom entry\ncan be forwarded to folsom's counter, histogram, duration, meter,\nand spiral.\n\nFolsom integration is provided as a backup. New code using Exometer\nshould use the native probes that duplicate folsom.\n\n=== exometer_function [entry] ===\n\nThe function entry allows for an existing erlang function to be wrapped\nas an exometer entry. The {@link exometer_function} module supports a number\nof options for passing arguments and matching out data points from the\nresult.\n\nThe function entry provides an easy way of integrating an external\nsystem without having to write a complete entry.\n\n== Built in Reporters ==\n\nExometer ships with some built-in reporters which can be used to forward updated\nmetrics and their data points to external systems. They can also\nserve as templates for custom-developed reporters.\n\n=== exometer_report_graphite ===\n\nThe graphite reporter uses the TCP/IP protocol to forward\nsubscribed-to metrics and data points to a graphite server, such as\nthe one provided by [http://hostedgraphite.com]. When the graphite\nreporter receives a metric-datapoint value (subscribed to through\n`exometer_report:subscriber()'), the reporter will immediately\nforward the key-value pair to the graphite server.\n\n=== exometer_report_opentsdb ===\n\nThe OpenTSDB reporter sends metrics to an OpenTSDB server using\nthe telnet API. All subscribed-to metric-datapoint values received\nby the reporter are immediately forwarded to OpenTSDB.\n\nIf the OpenTSDB connection is lost, the reporter will attempt to reconnect to it\nat a configurable interval.\n\nThe data sent to OpenTSDB will be formatted as follows:\n\n
put metric timestamp value host=host type=datapoint
\n\nWhere the value for the host tag will be the configured host in the reporter\nconfiguration (defaults to the value returned by `netadm:localhost'), and\ndatapoint tags as specified by the subscriber.\n\nPlease see {@section Configuring opentsdb reporter} for details on the\napplication environment parameters listed above.\n\n=== exometer_report_amqp ===\n\nThe AMQP reporter sends metrics to an AMQP broker as a json-encoded payload. All\nsubscribed-to metric-datapoint values received by the reporter are forwarded to AMQP.\n\nIf the AMQP connection is lost, the reporter will attempt to reconnect to it\nat a configurable interval.\n\nThe data sent to AMQP will be formatted as follows:\n\n
\n\nWhere the value for the host tag will be the configured host in the reporter\nconfiguration (defaults to the value returned by `netadm:localhost'), the\ninstance tag represents the datapoint for the metric.\n\nPlease see {@section Configuring amqp reporter} for details on the\napplication environment parameters listed above.\n\n=== exometer_report_snmp ===\n\nThe SNMP reporter enables the export of metrics and their datapoints to SNMP managers.\nThe export needs to be enabled for each metric through their options.\nMoreover, SNMP notifications can be created using the options to send periodic reports\non datapoints to SNMP managers. All SNMP protocol handling is done by the snmp application\nshipped with Erlang/OTP. Thus, the snmp application needs to be started and\nthe local SNMP master agent needs to be configured correctly for SNMP export to work\nproperly.\n\nTo configure SNMP export for a single metric use these options:\n\n+ `{snmp, disabled}' (default)\n Disables SNMP export for the metric. Same as not specifying the option at all.\n\n+ `{snmp, []}'\n Enables SNMP export for the metric. No subscriptions are setup.\n\n+ `{snmp, [{Datapint, Interval}]}'\n Enables SNMP export for the metric.\n Subscriptions are setup for the given Datapoint/Interval pairs.\n Each subscription report will be forwarded to SNMP mangers as notifications.\n\n+ `{snmp, [{Datapint, Interval, Extra}]}'\n Same as above, but using an addition extra identification for the subscriptions.\n Allow the creation ofmultiple subscriptions for a single datapoint.\n\nPlease see {@section Configuring snmp reporter} for details on how to configure the\nSNMP reporter.\n\n== Instrumenting Erlang code ==\n\nThe code using Exometer needs to be instrumented in order to setup and\nuse metrics reporting.\n\n=== Exometer Start ===\n\nThe system using Exometer must start the `exometer' application prior to using it:\n\n
\n\nNote that dependent applications need to be started first. On newer OTP versions\n(R16B or later), you can use `application:ensure_all_started(exometer)'.\n\nFor testing, you can also use {@link exometer:start/0}.\n\nIf you make use of e.g. folsom metrics, you also need to start `folsom'. Exometer\nwill not do that automatically, nor does it contain an application dependency for it.\n\nSee {@section Configuring Exometer} for details on configuration data\nformat.\n\n=== Creating metrics ===\n\nA metric, can be created through a call to\n\n
\nexometer:new(Name, Type)
\n\n`Name' is a list of atoms, uniquely identifying the metric created.\nThe type of the metric, specified by `Type' will be mapped\nto an exometer entry through the table maintained by\n`exometer_admin' Please see the {@section Configuring type - entry\nmaps} for details.\n\nThe resolved entry to use will determine the data points available\nunder the given metric.\n\n=== Deleting metrics ===\n\nA metric previously created with `exometer:new()' can be deleted by\n`exometer:delete()'.\n\nAll subscriptions to the deleted metrics will be cancelled.\n\n=== Setting metric values ===\n\nA created metric can have its value updated through the\n`exometer:update()' function:\n\n
\nexometer:update(Name, Value)
\n\nThe `Name' parameter is the same atom list provided to a previous\n`exometer:new()' call. The `Value' is an arbitrarty element that is\nforwarded to the `exometer:update()' function of the entry/probe that the\nmetric is mapped to.\n\nThe receiving entry/probe will process the provided value and modify\nits data points accordingly.\n\n=== Retrieving metric values ===\n\nExometer-using code can at any time retrieve the data point values\nassociated with a previously created metric. In order to find out which\ndata points are available for a metric, the following call can be used:\n\n
\nexometer:info(Name, datapoints)
\n\nThe `Name' parameter is the same atom list provided to a previous\n`exometer:new()' call. The call will return a list of data point\natoms that can then be provided to `exometer:get_value()' to\nretrieve their actual value:\n\n
\nexometer:get_value(Name, DataPoint)
\n\nThe `Name' paramer identifies the metric, and `DataPoints'\nidentifies the data points (returned from the previous `info()' call)\nto retrieve the value for.\n\nIf no DataPoints are provided, the values of a default list of data points,\ndetermined by the backing entry / probe, will be returned.\n\n=== Setting up subscriptions ===\n\nA subscription can either be statically configured, or dynamically\nsetup from within the code using Exometer. For details on statically\nconfigured subscriptions, please see {@section Configuring static subscriptions}.\n\nA dynamic subscription can be setup with the following call:\n\n
\n\n`Recipient' is the name of a reporter.\n\n=== Set metric options ===\nEach created metric can have options setup for it through the following call:\n\n
\nexometer:setopts(Name, Options)
\n\nThe `Name' paramer identifies the metric to set the options for, and\nOptions is a proplist (`[{ Key, Value },...]') with the options to be\nset.\n\nExometer looks up the the backing entry that hosts the metric with the given Name, and will\ninvoke the entry\\'s `setopts/4' function to set the actual options. Please see the\n`setopts/4' function for the various entries for details.\n\n== Configuring Exometer ==\n\nExometer defaults can be changed either through OTP application environment\nvariables or through the use of Basho's `cuttlefish'\n([https://github.com/basho/cuttlefish]).\n\n\n=== Configuring type - entry maps ===\n\nThe dynamic method of configuring defaults for `exometer' entries is:\n\n
\n\nWhere `NamePattern' is a list of terms describing what is essentially\na name prefix with optional wildcards ('_'). A pattern that\nmatches any legal name is ['_'].\n\n`Type' is an atom defining a type of metric. The types already known to\n`exometer', `counter', `fast_counter', `ticker', `uniform', `histogram',\n`spiral', `netlink', and `probe' may be redefined, but other types can be\ndescribed as well.\n\n`Default' is either an `#exometer_entry{}' record (unlikely), or a list of\n`{Key, Value}' options, where the keys correspond to `#exometer_entry' record\nattribute names. The following attributes make sense to preset:\n\n
\n{module, atom()} % the callback module\n{status, enabled | disabled} % operational status of the entry\n{cache, non_neg_integer()} % cache lifetime (ms)\n{options, [{atom(), any()}]} % entry-specific options
\n\nBelow is an example, from `exometer/priv/app.config':\n\n
\n\nIn systems that use CuttleFish, the file `exometer/priv/exometer.schema'\ncontains a schema for default settings. The setup corresponding to the above\ndefaults would be as follows:\n\n
\n\n=== Configuring statically defined entries ===\n\nUsing the `exometer' environment variable `predefined', entries can be added\nat application startup. The variable should have one of the following values:\n\n* `{script, File}' - `File' will be processed using `file:script/2'. The return\n value (the result of the last expression in the script) should be a list of\n `{Name, Type, Options}' tuples.\n\n* `{apply, M, F, A}' - The result of `apply(M, F, A)' should be `{ok, L}' where\n `L' is a list of `{Name, Type, Options}' tuples.\n\n* `L', where L is a list of `{Name, Type, Options}' tuples or extended\n instructions (see below).\n\nThe list of instructions may include:\n\n* `{delete, Name}' - deletes `Name' from the exometer registry.\n\n* `{select_delete, Pattern}' - applies a select pattern and\n deletes all matching entries.\n\n* `{re_register, {Name, Type, Options}}' - redefines an entry if present,\n otherwise creates it.\n\nExometer will also scan all loaded applications for the environment\nvariables `exometer_defaults' and `exometer_predefined', and process\nas above. If an application is loaded and started after exometer has started,\nit may call the function `exometer:register_application()' or\n`exometer:register_application(App)'. This function will do nothing if\nexometer isn't already running, and otherwise process the `exometer_defaults'\nand `exometer_predefined' variables as above. The function can also be\ncalled during upgrade, as it will re-apply the settings each time.\n\n=== Configuring static subscriptions ===\nStatic subscriptions, which are automatically setup at exometer\nstartup without having to invoke `exometer_report:subscribe()', are\nconfigured through the report sub section under exometer.\n\nBelow is an example, from `exometer/priv/app.config':\n\n
\n\nThe `report' section configures static subscriptions and reporter\nplugins. See {@section Configuring reporter plugins} for details on\nhow to configure individual plugins.\n\nThe `subscribers' sub-section contains all static subscriptions to be\nsetup at exometer applications start. Each tuple in the prop list\nshould be of one of the following formats:\n\n* `{Reporter, Metric, DataPoint, Interval}'\n\n* `{Reporter, Metric, DataPoint, Interval, RetryFailedMetrics}'\n\n* `{Reporter, Metric, DataPoint, Interval, RetryFailedMetrics, Extra}'\n\n* `{apply, {M, F, A}}'\n\n* `{select, {MatchPattern, DataPoint, Interval [, Retry [, Extra] ]}}'\n\nIn the case of `{apply, M, F, A}', the result of `apply(M, F, A)' must\nbe a list of `subscribers' tuples.\n\nIn the case of `{select, Expr}', a list of metrics is fetched using\n`exometer:select(MatchPattern)', where the result must be on the form\n`{Key, Type, Status}' (i.e. what corresponds to '$_').\nThe rest of the items will be applied to each of the matching entries.\n\nThe meaning of the above tuple elements is:\n\n+ `Reporter :: module()'\n Specifies the reporter plugin module, such as\n `exometer_report_collectd' that is to receive updated metric's data\n points.\n\n+ `Metric :: [atoms()]'\n Specifies the path to a metric previously created with an\n `exometer:new()' call.\n\n+ `DataPoint :: atom() | [atom()]'\n Specifies the data point within the given metric to send to the\n receiver. The data point must match one of the data points returned by\n `exometer:info(Name, datapoints)' for the given metrics name.\n\n+ `Interval :: integer()' (milliseconds)\n Specifies the interval, in milliseconds, between each update of the\n given metric's data point. At the given interval, the data point will\n be samples, and the result will be sent to the receiver.\n\n+ `RetryFailedMetrics :: boolean()'\n Specifies if the metric should be continued to be reported\n even if it is not found during a reporting cycle. This would be\n the case if a metric is not created by the time it is reported for\n the first time. If the metric will be created at a later time,\n this value should be set to true. Set this value to false if all\n attempts to report the metric should stop if when is not found.\n The default value is `true'.\n\n+ `Extra :: any()'\n Provides a means to pass along extra information for a given\n subscription. An example is the `syntax' option for the SNMP reporter,\n in which case `Extra' needs to be a property list.\n\nExample configuration in sys.config, using the `{select, Expr}' pattern:\n\n
\n\nThis will activate a subscription on `[a,1]' and `[a,2]' in the\n`exometer_report_tty' reporter, firing once per second.\n\n\n=== Configuring reporter plugins ===\nThe various reporter plugins to be loaded by exometer are configured\nin the `report' section under `reporters'\n\nEach reporter has an entry named after its module, and the content of\nthat entry is dependent on the reporter itself. The following chapters\nspecifies the configuration parameters for the reporters shipped with\nexometer.\n\n=== Configuring opentsdb reporter ===\nBelow is an example of the opentsdb reporter application environment, with\nits correct location in the hierarchy:\n\n
\n\nThe following attributes are available for configuration:\n\n+ `reconnect_interval' (seconds - default: 30)\n Specifies the duration between each reconnect attempt to an opentsdb\n server that is not available. Should the server either be unavailable\n at exometer startup, or become unavailable during exometer's\n operation, exometer will attempt to reconnect at the given number of\n seconds.\n\n+ `connect_timeout' (milliseconds - default: 5000)\n Specifies how long the opentsdb reporter plugin shall wait for a\n socket connection to complete before timing out. A timed out\n connection attempt will be retried after the reconnect interval has\n passed see item 1 above).\n\n+ `hostname' (string - default: `net_adm:localhost()')\n Specifies the host name to use for the host tag in the OpenTSDB tags.\n Please see {@section Configuring opentsdb reporter} for details.\n\n+ `host' (ip - default: `{\"127.0.0.1\", 4242}')\n Specifies the host and port to connect to OpenTSDB.\n\n=== Configuring amqp reporter ===\nBelow is an example of the amqp reporter application environment, with\nits correct location in the hierarchy:\n\n
\n\nThe following attributes are available for configuration:\n\n+ `reconnect_interval' (seconds - default: 30)\n Specifies the duration between each reconnect attempt to an amqp\n broker that is not available. Should the server either be unavailable\n at exometer startup, or become unavailable during exometer's\n operation, exometer will attempt to reconnect at the given number of\n seconds.\n\n+ `hostname' (string - default: `net_adm:localhost()')\n Specifies the host name to use for the host property in the JSON payload.\n Please see {@section Configuring amqp reporter} for details.\n\n+ `amqp_url' (string - default: `amqp://guest:guest@localhost:5672/%2f')\n Specifies the amqp url to connect to.\n\n+ `exchange' (string - default: `exometer')\n Specifies the exchange to publish messages to.\n\n+ `routing_key' (string - default: `exometer')\n Specifies the routing key to use when publishing messages.\n\n+ `buffer_size' (integer - default: `0')\n Specifies the size in bytes of payload to buffer before sending to AMQP.\n\n=== Configuring graphite reporter ===\nBelow is an example of the a graphite reporter application environment, with\nits correct location in the hierarchy:\n\n
\n\nThe following attributes are available for configuration:\n\n+ `connect_timeout' (milliseconds - default: 5000)\n Specifies how long the graphie reporter plugin shall wait for a tcp\n connection to complete before timing out. A timed out connection will\n not be reconnected to automatically. (To be fixed.)\n\n+ `prefix' (string - default: \"\")\n Specifies an optional prefix to prepend all metric names with before\n they are sent to the graphite server.\n\n+ `host' (string - default: \"carbon.hostedgraphite.com\")\n Specifies the name (or IP address) of the graphite server to report to.\n\n+ `port' (integer - default: 2003)\n Specifies the TCP port on the given graphite server to connect to.\n\n+ `api_key' (string - default: n/a)\n Specifies the api key to use when reporting to a hosted graphite server.\n\nIf `prefix' is not specified, but `api_key' is, each metrics will be reported as `ApiKey.Metric'.\n\nIf `prefix' is specified, but `api_key' is not, each metrics will be reported as `Prefix.Metric'.\n\nif neither `prefix' or `api_key' is specified, each metric will be reported simply as `Metric'.\n\n=== Configuring snmp reporter ===\nBelow is an example of the a snmp reporter application environment, with\nits correct location in the hierarchy:\n\n
![\"Overview\"](\"{@docRoot}/doc/exometer_overview.png?raw=true\")
\n\n=== Metric ===\n\nA metric is a specific measurement sampled inside an Erlang system and\nthen reported to the Exometer system. An example metric would be\n\"transactions_per_second\", or \"memory_usage\".\n\nMetrics are identified by a list of terms, such as given below:\n\n`[ xml_front_end, parser, file_size ]'\n\nA metric is created through a call by the code to be instrumented to\n`exometer:new()'. Once created, the metric can be updated through\n`exometer:update()', or on its own initiative through the\n`exometer_probe:sample' behavior implementation.\n\n=== Data Point ===\n\nEach metric can consist of multiple data points, where each point has\na specific value.\n\nA typical example of data points would be a\n`transactions_per_second' (tps) metric, usually stored as a\nhistogram covering the last couple of minutes of tps samples. Such a\nhistogram would host multiple values, such as `min', `max',\n`median', `mean', `50_percentile', `75_percentile',\netc.\n\nIt is up to the type of the metric, and the data probe backing that\ntype (see below), to specify which data points are available under the\ngiven metric.\n\n\n=== Metric Type ===\n\nThe type of a metric, specified when the metric is created through\n`exometer:new()', determines which `exometer_entry'\ncallback to use.\n\nThe link between the type and the entry to use is configured\nthrough the `exometer_admin' module, and its associated exometer\ndefaults configuration data.\n\nThe metric type, in other words, is mainly used to map a metric to a\nconfigurable `exometer_entry' callback, but it can also be referenced\nin queries using `exometer:select/1'. An entry callback can also support\nmultiple types (the type is provided as an argument in the callback functions).\n\nExometer provides default mappings for a number of metric types. It is\npossible to select different callbacks for each metric instance, as well\nas modify metrics using callback-specific options. Please see\n{@section Configuring type - entry maps} for details on how to do this.\n\n=== Entry Callback ===\n\nAn exometer entry callback will receive values reported to a metric through the\n`exometer:update()' call and compile it into one or more data points.\nThe entry callback can either be a counter (implemented natively\nin `exometer'), or a more complex statistical analysis such\nas a uniform distribution or a regular histogram.\n\nThe various outputs from these entries are reported as data points\nunder the given metric.\n\nAn entry can also interface external analytics packages.\n`exometer_folsom', for example, integrates with the\n`folsom_metrics' package found at [https://github.com/boundary/folsom].\n\n=== Probe ===\n\nProbes are a further specialization of exometer entries that run in\ntheir own Erlang processes and have their own state (like a\ngen_server). A probe is implemented through the `exometer_probe'\nbehavior.\n\nA probe can be used if independent monitoring is needed of,\nfor example, `/proc' trees, network interfaces, and other subsystems\nthat need periodic sampling. In these cases, the\n`exometer_probe:probe_sample()' call is invoked regularly by exometer,\nin the probe's own process, in order to extract data from\nthe given subsystem and add it to the metric's data points.\n\n=== Caching ===\n\nMetric and data point values are read with the `exometer:get_value()'\nfunction. In the case of counters, this operation is very fast. With probes,\nthe call results in a synchronous dialog with the probe process, and the\ncost of serving the request depends on the probe implementation and the\nnature of the metric being served.\n\nIf the cost of reading the value is so high that calling the function often\nwould result in prohibitive load, it is possible to cache the value. This is\ndone either explicitly from the probe itself (by calling\n`exometer_cache:write()'), or by specifying the option `{cache, Lifetime}'\nfor the entry. If an entry has a non-zero cache lifetime specified, the\n`get_value()' call will try fetching the cached value before calling the\nactual entry and automatically caching the result.\n\nNote that if `{cache, Lifetime}' is not specified, `exometer:get_value()'\nwill neither read nor write to the cache. It is possible for the probe\nto periodically cache a value regardless of how the cache lifetime is set,\nand the probe may also explicitly read from the cache if it isn't done\nautomatically.\n\n=== Subscriptions and Reporters ===\n\nThe subscription concept, managed by `exometer_report' allows metrics\nand their data points to be sampled at given intervals and delivered\nto one or more recipients, which can be either an arbitrary process\nor a Reporter plugin.\n\nEach subscription ties a specific metric-datapoint pair to a reporter\nand an interval (given in milliseconds). The reporter system will, at\nthe given interval, send the current value of the data point to the\nsubscribing reporter. The subscription, with all its parameters,\nis setup through a call to `exometer_report:subscribe()'.\n\nIn the case of processes, subscribed-to values will be delivered as a\nmessage. Modules, which implement the `exometer_report' callback\nbehavior, will receive the plugins as a callbacks within the\n`exometer_report' process.\n\nSubscriptions can either be setup at runtime, through\n`exometer_report:subscribe()' calls, or statically through the\n`exometer_report' configuration data.\n\n\n== Built-in entries and probes ==\nThere are a number of built-in entries and probes shipped\nwith the Exometer package, as described below:\n\n=== counter (exometer native) ===\nThe counter is implemented directly in `exometer' to provide simple\ncounters. A call to `exometer:update()' will add the provided value\nto the counter.\n\nThe counter can be reset to zero through `exometer:reset()'.\n\nThe available data points under a metric using the counter entry\nare `value' and `ms_since_reset'.\n\n=== fast_counter (exometer native) ===\n\nA fast counter implements the counter functionality, through the\n`trace_info' system, yielding a speed increase of about 3.5 in\ncomparison to the regular counter.\n\nThe tradeoff is that running tracing and/or debugging may interfere\nwith the counter functionality.\n\nA call to `exometer:update()' will add the provided value to the\ncounter.\n\nThe counter can be reset to zero through `exometer:reset()'.\n\nThe available data points under a metric using the fast_counter\nentry are `value' and `ms_since_reset'.\n\n=== gauge (exometer native) ===\n\nThe gauge is implemented directly in `exometer' to provide simple\ngauges. A call to `exometer:update()' will set the gauge's value\nto the provided value. That is, the value of the gauge entry is\nalways the most recently provided value.\n\nThe gauge can be reset to zero through `exometer:reset()'.\n\nThe available data points under a metric using the gauge entry\nare `value' and `ms_since_reset'.\n\n=== histogram (probe) ===\n\nThe histogram probe stores a given number of updates, provided through\n`exometer:update()', in a histogram. The histogram maintains a log\nderived from all values received during a configurable time span and\nprovides min, max, median, mean, and percentile analysis data points\nfor the stored data.\n\nExometer supports a number of different histogram implementations, each\nwith different performance and accuracy trade-offs.\n\nIn order to save memory, the histogram is divided into equal-sized\ntime slots, where each slot spans a settable interval. All values\nreceived during a time slot will be averaged into a single value to be\nstored in the histogram once the time slot expires. The averaging\nfunction (which can be replaced by the caller), allows for\nhigh-frequency update metrics to have their resolution traded against\nresource consumption.\n\n=== exometer_uniform (probe) ===\n\nThe uniform probe provides a uniform sample over a pool of values\nprovided through `exometer:update()'. When the pool reaches its configurable\nmax size, existing values will be replaced at random to make space for\nnew values. Much like `exometer_histogram', the uniform probe\nprovides min, max, median, mean, and percentile analysis data points\nfor the stored data.\n\n=== exometer_spiral (probe) ===\n\nThe spiral probe maintains the total sum of all values stored in its\nhistogram. The histogram has a configurable time span, all values\nprovided to the probe, through `exometer:update()', within that time\nspan will be summed up and reported. If, for example, the histogram\ncovers 60 seconds, the spiral probe will report the sum of all\nvalues reported during the last minute.\n\nThe grand total of all values received during the lifetime of the\nprobe is also available.\n\n=== exometer_folsom [entry] ===\n\n`exometer_folsom' is an entry behavior which implements most metric types\nsupported by the folsom\nmetrics package: Specifically, the metric types `counter', `spiral',\n`histogram', `meter', `meter_reader', `gauge', `duration' and `history'.\n\nThe folsom entry integrates with the folsom metrics package provided\nby the boundary repo at github. Updated values sent to the folsom entry\ncan be forwarded to folsom's counter, histogram, duration, meter,\nand spiral.\n\nFolsom integration is provided as a backup. New code using Exometer\nshould use the native probes that duplicate folsom.\n\n=== exometer_function [entry] ===\n\nThe function entry allows for an existing erlang function to be wrapped\nas an exometer entry. The {@link exometer_function} module supports a number\nof options for passing arguments and matching out data points from the\nresult.\n\nThe function entry provides an easy way of integrating an external\nsystem without having to write a complete entry.\n\n== Built in Reporters ==\n\nExometer ships with some built-in reporters which can be used to forward updated\nmetrics and their data points to external systems. They can also\nserve as templates for custom-developed reporters.\n\n=== exometer_report_graphite ===\n\nThe graphite reporter uses the TCP/IP protocol to forward\nsubscribed-to metrics and data points to a graphite server, such as\nthe one provided by [http://hostedgraphite.com]. When the graphite\nreporter receives a metric-datapoint value (subscribed to through\n`exometer_report:subscriber()'), the reporter will immediately\nforward the key-value pair to the graphite server.\n\n=== exometer_report_opentsdb ===\n\nThe OpenTSDB reporter sends metrics to an OpenTSDB server using\nthe telnet API. All subscribed-to metric-datapoint values received\nby the reporter are immediately forwarded to OpenTSDB.\n\nIf the OpenTSDB connection is lost, the reporter will attempt to reconnect to it\nat a configurable interval.\n\nThe data sent to OpenTSDB will be formatted as follows:\n\n