[ { "path": ".gitignore", "content": "# File artifacts which may by produced regular software on Windows, Mac OS X and Linux\nThumbs.db\n.DS_Store\n*.bak\n\n# JetBrains PyCharm configuration and working files\n.idea\n\n*.py[co]\n~temp/\n\n# Distribution / packaging\n.Python\nenv/\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\nlib/\nlib64/\nparts/\nsdist/\nvar/\n*.egg-info/\n.installed.cfg\n*.egg\n\n# PyInstaller\n# Usually these files are written by a python script from a template\n# before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# rope\n.ropeproject/\n" }, { "path": ".travis.yml", "content": "language: python\n\npython:\n - \"2.6\"\n - \"2.7\"\n\ninstall:\n - pip install flake8\n\nscript: flake8 *.py\n" }, { "path": "MANIFEST.in", "content": "include graphios.py\ninclude graphios_backends.py\ninclude graphios.cfg\ngraft init\ngraft nagios\n" }, { "path": "README.md", "content": "\nGraphios\n========\n\n[![Build Status](https://travis-ci.org/shawn-sterling/graphios.svg?branch=master)](https://travis-ci.org/shawn-sterling/graphios)\n\n*Oct 15, 2014*\n\nNew graphios 2.0!\n\nWhat's new?\n* Support for multiple backends (graphite, statsd, librato) (and multiples of\n each backend if you want)\n* Support for using your service descriptions instead of custom variables\n* Install options (pip, setup.py, rpms)\n* Bugfixes\n * mulitple perfdata in 1 line sometimes did weird things\n * quotes in your labels/metrics were sometimes in carbon\n * labels with multiple '::' could mess up\n\n# Introduction\n\nGraphios is a script to emit nagios perfdata to various upstream metrics\nprocessing and time-series (graphing) systems. It's currently compatible with\n[graphite], [statsd], [Librato] and [InfluxDB], with possibly [Heka], and\n[RRDTool] support coming soon. Graphios can emit Nagios metrics to any number\nof supported upstream metrics systems simultaenously.\n\n# Requirements\n\n* A working nagios / icinga / naemon server\n* A functional carbon or statsd daemon, and/or Librato credentials\n* Python 2.6 or later (but not python 3.x) (Is anyone still using 2.4? Likely\nvery little work to make this work under 2.4 again if so. Let me know)\n\n# License\n\nGraphios is released under the [GPL v2](http://www.gnu.org/licenses/gpl-2.0.html).\n\n# Documentation\n\nThe goal of graphios is to get nagios perf data into a graphing system like\ngraphite (carbon). Systems like these typically use a dot-delimited metric name\nto store each metric hierarcicly, so it can be easily located later.\n\nGraphios creates these metric names one of two ways.\n\n1. by reading a pair of custom variables that\nyou configure for services and hosts called \\_graphiteprefix and\n\\_graphitepostfix. Together, these custom variables enable you to control the\nmetric name that gets sent to whatever back-end metrics system you're using.\nYou don't have to set them both, but things will certainly be less confusing\nfor you if you set at least one or the other.\n\n2. by using your service description in the format:\n\n\\_graphiteprefix.hostname.service-description.\\_graphitepostfix.perfdata\n\nso if you didn't feel like setting your graphiteprefix and postfix, it would\njust use:\n\nhostname.service-description.perfdata\n\nIf you are using option 2, that means EVERY service will be sent to graphite.\nYou will also want to make sure your service descriptions are consistant or\nyour backend naming will be really weird.\n\nI think most people will use the first option, so let's work with that for a\nbit. What gets sent to graphite is this:\n\ngraphiteprefix.hostname.graphitepostfix.perfdata\n\nThe specific content of the perfdata section depends on each particular Nagios\nplugin's output.\n\nSimple Example\n--------------\n\nA simple example is the check\\_host\\_alive command (which calls the check\\_icmp\nplugin by default). The check\\_icmp plugin returns the following perfstring:\n\nrta=4.029ms;10.000;30.000;0; pl=0%;5;10;; rtmax=4.996ms;;;; rtmin=3.066ms;;;;\n\nIf we configured a host with a custom graphiteprefix variable like this:\n\n
\ndefine host {\n    host_name                   myhost\n    check_command               check_host_alive\n    _graphiteprefix             ops.nagios01.pingto\n}\n
\n\nGraphios will construct and emit the following metric name to the upstream metric system:\n\n ops.nagios01.pingto.myhost.rta 4.029 nagios_timet\n ops.nagios01.pingto.myhost.pl 0 nagios_timet\n ops.nagios01.pingto.myhost.rtmax 4.996 nagios_timet\n ops.nagios01.pingto.myhost.rtmin 3.066 nagios_timet\n\nWhere *nagios\\_timet* is the a unix epoch time stamp from when the plugin\nresults were received by Nagios core. Your prefix is of course, entirely up to\nyou. In our example, our prefix refers to the Team that created the metric\n(Ops), becuause our upstream metrics system is used by many different teams.\nAfer the team name, we've identified the specific Nagios host that took this\nmeasurement, because we actually have several Nagios boxes, and finally,\n'pingto' is the name of this specific metric: the *ping* time from nagios01\n*to* myhost.\n\nAnother example\n---------------\n\nLets take a look at the check_load plugin, which returns the following\nperfdata:\n\nload1=8.41;20;22;; load5=6.06;18;20;; load15=5.58;16;18\n\nOur service is defined like this:\n\n
\ndefine service {\n    service_description         Load\n    host_name                   myhost\n    _graphiteprefix             datacenter01.webservers\n    _graphitepostfix            nrdp.load\n}\n
\n\nWith this confiuration, graphios generates the following metric names:\n\n datacenter01.webservers.myhost.nrdp.load.load1 8.41 nagios_timet\n datacenter01.webservers.myhost.nrdp.load.load5 6.06 nagios_timet\n datacenter01.webservers.myhost.nrdp.load.load15 5.58 nagios_timet\n\nAs you can probably guess, our custom prefix in this example identifies the\nspecific data center, and server-type from which these metrics originated,\nwhile our postfix refers to the check_nrdp plugin, which is the means by which\nwe collected the data, followed finally by the metric-type.\n\nYou should think carefully about how you name your metrics, because later on,\nthese names will enable you to easily combine metrics (like load1) across\nvarious sources (like all webservers).\n\nUsing metric_base_path to add a universal prefix\n------------------------------------------------\n\nIn an environment where multiple things are feeding metrics into your backend\nservice, it can be handy to differentiate by source. Normally, you would need\nto prepend the graphiteprefix to all services and hosts, but in some cases, this\nisn't possible or feasible. \n\nWhen you want everything to be prepended with the same string, use the\nmetric_base_path setting: \n\n\tmetric_base_path\t= mycorp.nagios\n\t\nNote that quotes will be preserved. Also, _graphiteprefix and _graphitepostfix \nwill be applied in addition to this string, so if you are already adding \nmycorp.nagios to your prefix, you will end up with mycorp.nagios.mycorp.nagios.metricname\n\nA few words on Naming things for Librato\n----------------------------------------\n\nThe default configuration that works for Graphite also does what you'd expect\nfor Librato, so if you're just getting started, and you want to check out\nLibrato, don't worry about it, ignore this section and forge ahead.\n\nBut you're a power user, you should be aware that the Librato Backend is\nactually generating a differet metric name than the other plugins.\nLibrato is a very metrics-centric platform. Metrics are the first-class entity,\nand sources (like hosts), are actually a separate dimension in their system.\nThis is very cool when you're monitoring ephemeral things that aren't hosts,\nlike threads, or worker processes, but it slightly complicates things here.\n\nSo, for example, where the Graphite plugin generates a name like this (from the\nexample above):\n\n datacenter01.webservers.myhost.nrdp.load.load1\n\nThe Librato plugin will generate a name that omits the hostname:\n\n datacenter01.webservers.nrdp.load.load1\n\nAnd then it will automatically send the hostname as the source dimension when\nit emits the metric to Librato. For 99% of everyone, this is exactly what you\nwant. But if you're a 1%'er you can influence this behavior by modifying the\n\"namevals\" and \"sourcevals\" lists in the librato section of the graphios.cfg\n\nAutomatic names\n---------------\n\nVersion 2.0: Graphios now supports automatic names, because custom variables\nare hard. :)\n\nThis is an all or nothing setting, meaning if you turn this on all services\nwill now send to graphios (instead of just the ones with the prefix and postfix\nsetup). This will work fine, so long as you have very consistent service\ndescriptions.\n\nTo turn this on, modify the graphios.cfg and change:\n\n use_service_desc = False\nto\n use_service_desc = True\n\nYou can still use the graphite prefix and postfix variables but you don't have\nto.\n\n# Big Fat Warning\n\nGraphios assumes your checks are using the same unit of measurement. Most\nplugins support this, some do not. check\\_icmp) always reports in ms for\nexample.\n\n# Installation\n\nThis is recommended for intermediate+ Nagios administrators. If you are just\nlearning Nagios this might be a difficult pill to swallow depending on your\nexperience level.\n\nHundreds of people have emailed me their success stories on getting graphios\nworking. I have been using this in production on a medium size nagios\ninstallation for a couple years.\n\nThere are now a few ways to get graphios installed.\n\n1 - Use pypi\n\n```\n pip install graphios\n```\n NOTE: This will attempt to find your nagios.cfg and add the configuration\n steps 1 and 2 for you (Don't worry we back up the file before touching it)\n\n NOTE2: If you get the error:\n Could not find a version that satisfies the requirement graphios\n This is a because graphios is still in the beta category. I will remove\n this in a few weeks, so until then you need to:\n\n```\n pip install --pre graphios\n```\n\n2 - Clone it yourself\n\n```\n git clone https://github.com/shawn-sterling/graphios.git\n cd graphios\n```\n\nThen do one of the following three things (depending what you like best):\n\n 1 - Python setup\n\n```\n python setup.py install\n```\n\n 2 - Create + Install RPM\n\n```\n python setup.py bdist_rpm\n yum localinstall bdist/graphios-$version.rpm\n```\n\n 3 - Copy the files where you want them to be\n\n```\n cp graphios*.py /my/dir\n cp graphios.cfg /my/dir\n```\n\n# Configuration\n\nSetting this up on the nagios front is very much like pnp4nagios with npcd.\n(You do not need to have any pnp4nagios experience at all). If you are already\nrunning pnp4nagios , check out my pnp4nagios notes (below).\n\nSteps:\n\n(1) graphios.cfg\n----------------\n\nThe default location for graphios.cfg is in /etc/graphios/graphios.cfg, it\nalso checks the same directory as the graphios.py is.\n\nYour graphios.cfg can live anywhere you want, but if it's not in the above\nlocations you will need to modify your init script to match.\n\nOut of the box, it enables the carbon back-end and sends pickled metrics to\n127.0.0.1:2004. It also specifies the location of the graphios log and spool\ndirectories, and controls things like log levels, sleep intervals, and of\ncourse, backends like carbon, statsd, and librato.\n\nThe config file is well commented, adding/changing backends is very simple.\n\n(2) nagios.cfg\n--------------\n\nYour nagios.cfg is going to need to modified to send the graphite data to the\nperfdata files. Depending on how you installed graphios this step may have been\ndone for you.\n\nThe following needs to be put into your nagios.cfg\n
\nservice_perfdata_file=/var/spool/nagios/graphios/service-perfdata\nservice_perfdata_file_template=DATATYPE::SERVICEPERFDATA\\tTIMET::$TIMET$\\tHOSTNAME::$HOSTNAME$\\tSERVICEDESC::$SERVICEDESC$\\tSERVICEPERFDATA::$SERVICEPERFDATA$\\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\\tHOSTSTATE::$HOSTSTATE$\\tHOSTSTATETYPE::$HOSTSTATETYPE$\\tSERVICESTATE::$SERVICESTATE$\\tSERVICESTATETYPE::$SERVICESTATETYPE$\\tGRAPHITEPREFIX::$_SERVICEGRAPHITEPREFIX$\\tGRAPHITEPOSTFIX::$_SERVICEGRAPHITEPOSTFIX$\n\nservice_perfdata_file_mode=a\nservice_perfdata_file_processing_interval=15\nservice_perfdata_file_processing_command=graphite_perf_service\n\nhost_perfdata_file=/var/spool/nagios/graphios/host-perfdata\nhost_perfdata_file_template=DATATYPE::HOSTPERFDATA\\tTIMET::$TIMET$\\tHOSTNAME::$HOSTNAME$\\tHOSTPERFDATA::$HOSTPERFDATA$\\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\\tHOSTSTATE::$HOSTSTATE$\\tHOSTSTATETYPE::$HOSTSTATETYPE$\\tGRAPHITEPREFIX::$_HOSTGRAPHITEPREFIX$\\tGRAPHITEPOSTFIX::$_HOSTGRAPHITEPOSTFIX$\n\nhost_perfdata_file_mode=a\nhost_perfdata_file_processing_interval=15\nhost_perfdata_file_processing_command=graphite_perf_host\n
\n\nWhich sets up some custom variables, specifically:\nfor services:\n$\\_SERVICEGRAPHITEPREFIX\n$\\_SERVICEGRAPHITEPOSTFIX\n\nfor hosts:\n$\\_HOSTGRAPHITEPREFIX\n$\\_HOSTGRAPHITEPOSTFIX\n\nThe prepended HOST and SERVICE is just the way nagios works,\n\\_HOSTGRAPHITEPREFIX means it's the \\_GRAPHITEPREFIX variable from host\nconfiguration.\n\n(3) nagios commands\n-------------------\n\nThere are 2 commands we setup in the nagios.cfg, which if you used pip or the\nrpm/deb may have already been setup for you. We need:\n\n graphite\\_perf\\_service\n graphite\\_perf\\_host\n\nWhich we now need to define:\n\nI use include dirs, so I make a new file called graphios\\_commands.cfg inside\nmy include dir. Do that, or add the below commands to one of your existing\nnagios config files.\n\n#### NOTE: Your spool directory may be different, this is setup in step (2) the service_perfdata_file, and host_perfdata_file.\n\n
\ndefine command {\n    command_name            graphite_perf_host\n    command_line            /bin/mv /var/spool/nagios/graphios/host-perfdata /var/spool/nagios/graphios/host-perfdata.$TIMET$\n\n}\n\ndefine command {\n    command_name            graphite_perf_service\n    command_line            /bin/mv /var/spool/nagios/graphios/service-perfdata /var/spool/nagios/graphios/service-perfdata.$TIMET$\n}\n
\n\nAll these commands do is move the current files to a different filename that we can process without interrupting nagios. This way nagios doesn't have to sit around waiting for us to process the results.\n\n(4) Run it!\n---------------\n\nWe recommend running graphios.py from the console for the first time, this will\nmake sure things are sending the way you think they are. A good example would\nbe:\n\n ./graphios.py --spool-directory /var/spool/nagios/graphios \\\n --log-file /tmp/graphios.log \\\n --backend carbon \\\n --server 127.0.0.1:2004 \\\n --test\n\nand if there are problems add\n\n --verbose\n\nOther command line options:\n
\nUsage: graphios.py [options]\nsends nagios performance data to carbon.\n\nOptions:\n  -h, --help            show this help message and exit\n  -v, --verbose         sets logging to DEBUG level\n  --spool-directory=SPOOL_DIRECTORY\n                        where to look for nagios performance data\n  --log-file=LOG_FILE   file to log to\n  --backend=BACKEND     sets which storage backend to use\n  --config=CONFIG       set custom config file location\n  --test                Turns on test mode, which won't send to backends\n  --replace_char=REPLACE_CHAR\n                        Replacement Character (default '_'\n  --sleep_time=SLEEP_TIME\n                        How much time to sleep between checks\n  --sleep_max=SLEEP_MAX\n                        Max time to sleep between runs\n  --server=SERVER       Server address (for backend)\n  --no_replace_hostname\n                        Replace '.' in nagios hostnames, default on.\n  --reverse_hostname    Reverse nagios hostname, default off.\n\n)\n
\n\n** NOTE: If you use --config on the command line, we ignore every other\ncommand line, your --config will overwrite everything else.\n\n(5) Optional init script: graphios\n----------------------------------\n\nRemember: *screen* is not a daemon management tool.\n\nIf you installed with pip/setup.py/rpm this part should be done for you!\n\nTake a look in the init/ directory and find your OS of choice.\n\nFor debian/ubuntu:\n cp init/debian/graphios /etc/init.d/\n cp init/debian/graphios.conf /etc/init\n chmod 755 /etc/init.d/graphios\n\nFor rhel/centos/sl < 6:\n cp init/rhel/graphios /etc/init.d\n chmod 755 /etc/init.d/graphios\n\nfor systems with systemd:\n cp init/systemd/graphios.service /usr/lib/systemd/system\n\n#### NOTE: You may need to change the location and username that the script runs as. this varies slightly depending on where you decided to put graphios.py\n\nThe lines you will likely have to change:\n
\nprog=\"/opt/nagios/bin/graphios.py\"\n# or use the command line options:\n#prog=\"/opt/nagios/bin/graphios.py --log-file=/dir/mylog.log --spool-directory=/dir/my/sool\"\nGRAPHIOS_USER=\"nagios\"\n
\n\n(6) Your host and service configs\n---------------------------------\n\nOnce you have done the above you need to add a custom variable to the hosts and\nservices that you want sent to graphite. (Unless you are using service\ndescriptions, in which case you can skip this step)\n\nThe format that will be sent to carbon is:\n\n
\n_graphiteprefix.hostname._graphitepostfix.perfdata\n
\n\nYou do not need to set both graphiteprefix and graphitepostfix. Just one or the\nother will do. If you do not set at least one of them, the data will not be\nsent to graphite at all (unless you are using the service descriptions)\n\nExamples:\n\n
\ndefine host {\n    name                        myhost\n    check_command               check_host_alive\n    _graphiteprefix             monitoring.nagios01.pingto\n}\n
\n\nWhich would create the following graphite entries with data from the check\\_host\\_alive plugin:\n\n monitoring.nagios01.pingto.myhost.rta\n monitoring.nagios01.pingto.myhost.rtmin\n monitoring.nagios01.pingto.myhost.rtmax\n monitoring.nagios01.pingto.myhost.pl\n\n
\ndefine service {\n    service_description         MySQL threads connected\n    host_name                   myhost\n    check_command               check_mysql_health_threshold!threads-connected!3306!1600!1800\n    _graphiteprefix             monitoring.nagios01.mysql\n}\n
\n\nWhich gives us:\n\n monitoring.nagios01.mysql.myhost.threads_connected\n\nSee the Documentation (above) for more explanation on how this works.\n\n\n# Upgrading\n\nTo upgrade from the old version of graphios, you need to:\n\n1. Look at the things you changed in the old graphios.py (carbon_server,\nspool_directory, log_file location, etc)\n2. Edit your new graphios.cfg and put those options there instead. You should\nNOT have to modify the new graphios.py.\n\n*Why Upgrade?*\n\nThe new version has fixed some bugs, and has cooler optional backends; and\nsupport for multiple backends, including multiple carbon servers. I don't think\nany major performance increases have been made, so if it isn't broken don't fix\nit.\n\n# PNP4Nagios Notes:\n\nAre you already running pnp4nagios? And want to just try this out and see if\nyou like it? Cool! This is very easy to do without breaking your PNP4Nagios\nconfiguration (but do a backup just in case).\n\nSteps:\n\n(1) In your nagios.cfg:\n-----------------------\n\nAdd the following at the end of your:\n\n
\nhost_perfdata_file_template\n\\tGRAPHITEPREFIX::$_HOSTGRAPHITEPREFIX$\\tGRAPHITEPOSTFIX::$_HOSTGRAPHITEPOSTFIX$\n\nservice_perfdata_file_template\n\\tGRAPHITEPREFIX::$_SERVICEGRAPHITEPREFIX$\\tGRAPHITEPOSTFIX::$_SERVICEGRAPHITEPOSTFIX$\n
\n\nThis will add the variables to your check results, and will be ignored by pnp4nagios.\n\n(2) Change your commands:\n-------------------------\n\n(find your command names under host\\_perfdata\\_file\\_processing\\_command and service\\_perfdata\\_file\\_processing\\_command in your nagios.cfg)\n\nYou likely have 2 commands setup that look something like these two:\n\n
\ndefine command{\n       command_name    process-service-perfdata-file\n       command_line    /bin/mv /usr/local/pnp4nagios/var/service-perfdata /usr/local/pnp4nagios/var/spool/service-perfdata.$TIMET$\n}\n\ndefine command{\n       command_name    process-host-perfdata-file\n       command_line    /bin/mv /usr/local/pnp4nagios/var/host-perfdata /usr/local/pnp4nagios/var/spool/host-perfdata.$TIMET$\n}\n
\n\nInstead of just moving the file; move it then copy it, then we can point graphios at the copy.\n\nYou can do this by either:\n\n(1) Change the command\\_line to something like:\n\n
\ncommand_line    \"/bin/mv /usr/local/pnp4nagios/var/host-perfdata /usr/local/pnp4nagios/var/spool/host-perfdata.$TIMET$ && cp /usr/local/pnp4nagios/var/spool/host-perfdata.$TIMET$ /usr/local/pnp4nagios/var/spool/graphios\"\n
\n\nOR\n\n(2) Make a script:\n\n
\n#!/bin/bash\n/bin/mv /usr/local/pnp4nagios/var/host-perfdata /usr/local/pnp4nagios/var/spool/host-perfdata.$TIMET$\ncp /usr/local/pnp4nagios/var/spool/host-perfdata.$TIMET$ /usr/local/pnp4nagios/var/spool/graphios\n\nchange the command_line to be:\ncommand_line    /path/to/myscript.sh\n
\n\nYou should now be able to skip steps 2 and 3 on the configuration instructions.\n\n# OMD (Open Monitoring Distribution) Notes:\n\nThese instructions are for OMD >= 1.2x (including the current nightly builds).\n\n__Note:__ All steps below are assumed to be carried out under your OMD site's user.\n\n(1) Change PNP4NAGIOS to use \"NPCD with Bulk Mode\" instead of NPCDMOD. This is done by redirecting the symlink for pnp4nagios.cfg:\n\n
\nln -sf ~/etc/pnp4nagios/nagios_npcd.cfg ~/etc/nagios/nagios.d/pnp4nagios.cfg\n
\n\n(2) Update ~/etc/pnp4nagios/nagios_npcd.cfg (remember to replace SITENAME).\n\n
\n#\n# PNP4Nagios Bulk Mode with npcd\n#\nprocess_performance_data=1\n\n#\n# service performance data\n#\nservice_perfdata_file=/omd/sites/SITENAME/var/pnp4nagios/service-perfdata\nservice_perfdata_file_template=DATATYPE::SERVICEPERFDATA\\tTIMET::$TIMET$\\tHOSTNAME::$HOSTNAME$\\tSERVICEDESC::$SERVICEDESC$\\tSERVICEPERFDATA::$SERVICEPERFDATA$\\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\\tHOSTSTATE::$HOSTSTATE$\\tHOSTSTATETYPE::$HOSTSTATETYPE$\\tSERVICESTATE::$SERVICESTATE$\\tSERVICESTATETYPE::$SERVICESTATETYPE$\\tGRAPHITEPREFIX::$_SERVICEGRAPHITEPREFIX$\\tGRAPHITEPOSTFIX::$_SERVICEGRAPHITEPOSTFIX$\nservice_perfdata_file_mode=a\nservice_perfdata_file_processing_interval=15\nservice_perfdata_file_processing_command=omd-process-service-perfdata-file\n\n#\n# host performance data\n#\nhost_perfdata_file=/omd/sites/SITENAME/var/pnp4nagios/host-perfdata\nhost_perfdata_file_template=DATATYPE::HOSTPERFDATA\\tTIMET::$TIMET$\\tHOSTNAME::$HOSTNAME$\\tHOSTPERFDATA::$HOSTPERFDATA$\\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\\tHOSTSTATE::$HOSTSTATE$\\tHOSTSTATETYPE::$HOSTSTATETYPE$\\tGRAPHITEPREFIX::$_HOSTGRAPHITEPREFIX$\\tGRAPHITEPOSTFIX::$_HOSTGRAPHITEPOSTFIX$\nhost_perfdata_file_mode=a\nhost_perfdata_file_processing_interval=15\nhost_perfdata_file_processing_command=omd-process-host-perfdata-file\n
\n\n(3) Update etc/nagios/conf.d/pnp4nagios.cfg (remember to replace SITENAME).\n\n
\ndefine command{\n       command_name    omd-process-service-perfdata-file\n       command_line    /bin/mv /omd/sites/SITENAME/var/pnp4nagios/service-perfdata /omd/sites/SITENAME/var/pnp4nagios/spool/service-perfdata.$TIMET$ && cp /omd/sites/SITENAME/var/pnp4nagios/spool/service-perfdata.$TIMET$ /omd/sites/SITENAME/var/graphios/spool/\n}\n\ndefine command{\n       command_name    omd-process-host-perfdata-file\n       command_line    /bin/mv /omd/sites/SITENAME/var/pnp4nagios/host-perfdata /omd/sites/SITENAME/var/pnp4nagios/spool/host-perfdata.$TIMET$ && cp /omd/sites/SITENAME/var/pnp4nagios/spool/host-perfdata.$TIMET$ /omd/sites/SITENAME/var/graphios/spool/\n}\n
\n\n(4) Optional: If you don't want PNP4NAGIOS to ever see perfdata for checks that Graphios is exporting data for, you can modify the ~/etc/nagios/conf.d/pnp4nagios.cfg command lines to remove data with a grep. In the below case, we grep out a specific string (GRAPHITEPREFIX\\:\\:lustre) to remove perfdata containing that string. This involves a little move moving around of files, but nothing excessive and stops PNP4NAGIOS from trying to genearte RRD files with that data. (Again remember to change SITENAME).\n\n
\ndefine command{\n       command_name    omd-process-service-perfdata-file\n       #command_line    /bin/mv /omd/sites/SITENAME/var/pnp4nagios/service-perfdata /omd/sites/SITENAME/var/pnp4nagios/spool/service-perfdata.$TIMET$\n###GRAPHITE SETTING### ADDED REDIRECTION TO REMOVE exportstats\n       command_line    /bin/mv /omd/sites/SITENAME/var/pnp4nagios/service-perfdata /omd/sites/SITENAME/var/pnp4nagios/service-perfdata.$TIMET$ && /bin/cp /omd/sites/SITENAME/var/pnp4nagios/service-perfdata.$TIMET$ /omd/sites/SITENAME/var/graphios/spool/ && grep -v GRAPHITEPREFIX\\:\\:lustre /omd/sites/SITENAME/var/pnp4nagios/service-perfdata.$TIMET$ > /omd/sites/SITENAME/var/pnp4nagios/spool/service-perfdata.$TIMET$ && /bin/rm /omd/sites/SITENAME/var/pnp4nagios/service-perfdata.*\n\n}\n\ndefine command{\n       command_name    omd-process-host-perfdata-file\n       #command_line    /bin/mv /omd/sites/SITENAME/var/pnp4nagios/host-perfdata /omd/sites/SITENAME/var/pnp4nagios/spool/host-perfdata.$TIMET$\n####GRAPHITE SETTING### ADDED REDIRECTION TO REMOVE exportstats\n       command_line    /bin/mv /omd/sites/SITENAME/var/pnp4nagios/host-perfdata /omd/sites/SITENAME/var/pnp4nagios/host-perfdata.$TIMET$ && /bin/cp /omd/sites/SITENAME/var/pnp4nagios/host-perfdata.$TIMET$ /omd/sites/SITENAME/var/graphios/spool/ && grep -v GRAPHITEPREFIX\\:\\:lustre /omd/sites/SITENAME/var/pnp4nagios/host-perfdata.$TIMET$ > /omd/sites/SITENAME/var/pnp4nagios/spool/host-perfdata.$TIMET$ && /bin/rm /omd/sites/SITENAME/var/pnp4nagios/host-perfdata.*\n
\n\n# Check_MK Notes:\n\nHow to set custom variables for services and hosts using check_mk config files. (For OMD please don't overlook the notes above).\n\n(1) For host perf data just create a new file named \"extra_host_conf.mk\" inside your check_mk conf.d dir.\n\n
\nextra_host_conf[\"_graphiteprefix\"] = [\n  ( \"DESIREDPREFIX.ping\", ALL_HOSTS),\n]\n
\n\n(2) Run check_mk -O to updated and reload Nagios.\n\n(3) Test via \"check_mk -N hostname | less\", to see if your prefix or postfix is there.\n\nFor service perf data create a file called, \"extra_service_conf.mk\". Remember you can use your host tags or any of kinds of tricks with check_mk config files.\n\n
\nextra_service_conf[\"_graphiteprefix\"] = [\n  ( \"DESIREDPREFIX.check_mk\", ALL_HOSTS, [\"Check_MK\"]),\n  ( \"DESIREDPREFIX.cpu.load\", ALL_HOSTS, [\"CPU load\"]),\n]\n
\n\n- - -\n__Tip__: An easy way to produce graphite keys in the format: `$company.$server.$metric` is:\n\n(1) Set `metric_base_path` to $company in `graphios.cfg`.\n\n(2) In your 'extra' check_mk config files set your graphiteprefix to $metric, and set no graphiteprefix.\n\n
\nextra_host_conf[\"_graphitepostfix\"] = [\n  # e.g. mycompany.server123.ping\n  ( \"ping\", ALL_HOSTS),\n]\n\nextra_service_conf[\"_graphitepostfix\"] = [\n  # e.g. mycompany.server123.cpu.load\n  ( \"cpu.load\", ALL_HOSTS, [\"CPU load\"]),\n]\n
\n\n\n# Trouble getting it working?\n\nMany people are running graphios now (cool!), but if you are having trouble\ngetting it working let me know. I am not offering to teach you how to setup\nNagios, this is for intermediate+ nagios users. Email me at\nshawn@systemtemplar.org and I will do what I can to help.\n\n# Got it working?\n\nCool! Drop me a line and let me know how it goes.\n\n# Find a bug?\n\nOpen an Issue on github and I will try to fix it asap.\n\n# Contributing\n\nI'm open to any feedback / patches / suggestions.\n\n# Special Thanks\n\nSpecial thanks to Dave Josephsen who added the multiple backend support and\nworked with me to design and build the new version of graphios.\n\nShawn Sterling shawn@systemtemplar.org\n" }, { "path": "graphios.cfg", "content": "# Graphios config file\n\n[graphios]\n\n#------------------------------------------------------------------------------\n# Global Details (you need these!)\n#------------------------------------------------------------------------------\n\n# Character to use as replacement for invalid characters in metric names\nreplacement_character = _\n\n# nagios spool directory\nspool_directory = /var/spool/nagios/graphios\n\n# graphios log info\nlog_file = /usr/local/nagios/var/graphios.log\n\n# max log size in megabytes (it will rotate the files)\nlog_max_size = 24\n\n# available log levels:\n# DEBUG, INFO, WARNING, ERROR, CRITICAL\n# see https://docs.python.org/2/library/logging.html#logging-levels for details\n# DEBUG is quite verbose\n#log_level = logging.DEBUG\nlog_level = logging.INFO\n\n# Disable this once you get it working.\ndebug = True\n\n# How long to sleep between processing the spool directory\nsleep_time = 15\n\n# when we can't connect to carbon, the sleeptime is doubled until we hit max\nsleep_max = 480\n\n# test mode makes it so we print what we would add to carbon, and not delete\n# any files from the spool directory. log_level must be DEBUG as well.\ntest_mode = False\n\n# use service description, most people will NOT want this, read documentation!\nuse_service_desc = False\n\n# replace \".\" in nagios hostnames? (so \"my.host.name\" becomes \"my_host_name\")\n# (uses the replacement_character)\nreplace_hostname = True\n\n# reverse hostname\n# if you have:\n# host.datacenter.company.tld\n# as your nagios hostname you may prefer to have your metric stored as:\n# tld.company.datacenter.host\nreverse_hostname = False\n\n# This string will be universally pre-pended to metrics, regardless of whether\n# or not _graphiteprefix is set. (Quotes not required).\n# metric_base_path = mycorp.nagios\n\n#------------------------------------------------------------------------------\n# Carbon Details (comment out if not using carbon)\n#------------------------------------------------------------------------------\n\nenable_carbon = False\n\n# Defaults to using the pickle protocol. Set to True to use the plaintext protocol.\ncarbon_plaintext = False\n\n# Comma separated list of carbon server IP:Port 's\ncarbon_servers = 127.0.0.1:2004\n\n# The max amount of metrics to send to the carbon server at a time (def:200)\n#carbon_max_metrics = 200\n\n#flag the carbon backend as 'non essential' for the purposes of error checking\n#nerf_carbon = False\n\n#------------------------------------------------------------------------------\n# Statsd Details (comment in if you are using statsd)\n#------------------------------------------------------------------------------\n\nenable_statsd = False\n\n# Comma separated list of statsd server IP:Port 's\nstatsd_servers = 127.0.0.1:8125\n\n#flag the statsd backend as 'non essential' for the purposes of error checking\n#nerf_statsd = False\n\n#------------------------------------------------------------------------------\n# librato Details (comment in if you are using librato)\n#------------------------------------------------------------------------------\n\nenable_librato = False\n\n# your (required) librato credentials here:\n#librato_email = \n#librato_token = \n\n#### ZOMG SUPER IMPORTANT SETTING THAT WILL SAVE YOU MONEY #####\n# json-formmated RE patterns that match the names of metrics you want to emit\n# to Librato the default list is [\".*\"] (send everything).\n# Example:\n# librato_whitelist = [\"load\",\"rta\",\"swap\"]\nlibrato_whitelist = [\".*\"]\n\n#OPTIONAL BELOW HERE, LEAVE COMMENTED UNLESS YOU REALLY WANT IT CHANGED\n\n# floor_time_secs: Floor samples to this time (set to graphios sleep_time)\n#librato_floor_time_secs = 15\n\n# comma separated list of Nagios Macros we use to construct the metric name:\n# librato_namevals = GRAPHITEPREFIX,SERVICEDESC,GRAPHITEPOSTFIX,LABEL\n\n# comma separated list of Nagios Macros we use to construct the source value :\n# librato_sourcevals = HOSTNAME\n\n#flag the librato backend as 'non essential' for the purposes of error checking\n#nerf_librato = False\n\n#------------------------------------------------------------------------------\n# InfluxDB Details (if you are using InfluxDB 0.8)\n#------------------------------------------------------------------------------\n\nenable_influxdb = False\n\n#------------------------------------------------------------------------------\n# InfluxDB Details (if you are using InfluxDB 0.9)\n# This will work a bit differently because of the addition of tags in\n# InfluxDB 0.9. Now the metric will be named after the service description,\n# the perfdata field will be a tag, and the host name will be a tag. The value\n# of the metric is stored in the 'value' column.\n# Requires use_service_desc = True.\n#------------------------------------------------------------------------------\n\nenable_influxdb09 = False\n\n# Extra tags to add to metrics, like data center location etc.\n# Only valid for 0.9\n#influxdb_extra_tags = {\"location\": \"la\"}\n\n# Comma separated list of server:ports\n# defaults to 127.0.0.1:8086 (:8087 if using SSL).\n#influxdb_servers = 127.0.0.1:8087\n\n# SSL, defaults to False\n#influxdb_use_ssl = True\n\n# Database-name, defaults to nagios\n#influxdb_db = \n\n# Credentials (required)\n#influxdb_user = \n#influxdb_password = \n\n# Max metrics to send / request, defaults to 250\n#influxdb_max_metrics = 500\n\n# Flag the InfluxDB backend as 'non essential' for the purposes of error checking\n#nerf_influxdb = False\n\n# enable Line Protocol, defaults to False\n#influxdb_line_protocol = True\n\n\n#------------------------------------------------------------------------------\n# STDOUT Details (comment in if you are using STDOUT)\n#------------------------------------------------------------------------------\n\n#comment the line below to disable the STDOUT sender\nenable_stdout = False\n\n#flag the stdout backend as 'non essential' for the purposes of error checking\nnerf_stdout = True\n" }, { "path": "graphios.py", "content": "#!/usr/bin/python -tt\n# vim: set ts=4 sw=4 tw=79 et :\n# Copyright (C) 2011 Shawn Sterling \n#\n# With contributions from:\n#\n# Juan Jose Presa \n# Ranjib Dey \n# Ryan Davis \n# Alexey Diyan \n# Steffen Zieger \n# Nathan Bird \n# Dave Josephsen \n# Emil Thelin \n#\n# This program is free software; you can redistribute it and/or\n# modify it under the terms of the GNU General Public License\n# as published by the Free Software Foundation; either version 2\n# of the License, or (at your option) any later version.\n#\n# This program is distributed in the hope that it will be useful,\n# but WITHOUT ANY WARRANTY; without even the implied warranty of\n# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n# GNU General Public License for more details.\n#\n# You should have received a copy of the GNU General Public License\n# along with this program; if not, write to the Free Software\n# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA\n# 02110-1301, USA.\n#\n# graphios: this program will read nagios host and service perfdata, and\n# send it to a carbon server.\n#\n# The latest version of this code will be found on my github page:\n# https://github.com/shawn-sterling\n\nfrom ConfigParser import SafeConfigParser\nfrom optparse import OptionParser\nimport copy\nimport graphios_backends as backends\nimport logging\nimport logging.handlers\nimport os\nimport os.path\nimport re\nimport sys\nimport time\n\n\n# ##########################################################\n# ### Do not edit this file, edit the graphios.cfg #####\n\n# nagios spool directory\nspool_directory = '/var/spool/nagios/graphios'\n\n# graphios log info\nlog_file = ''\nlog_max_size = 24\n\n# by default we will check the current path for graphios.cfg, if config_file\n# is passed as a command line argument we will use that instead.\nconfig_file = ''\n\n# This is overridden via config file\ndebug = False\n\n# config dictionary\ncfg = {}\n\n# backend global\nbe = \"\"\n\n# available loglevels for graphios.cfg\nloglevels = {\n 'logging.DEBUG': logging.DEBUG,\n 'logging.INFO': logging.INFO,\n 'logging.WARNING': logging.WARNING,\n 'logging.ERROR': logging.ERROR,\n 'logging.CRITICAL': logging.CRITICAL\n}\n\n# options parsing\nparser = OptionParser(\"\"\"usage: %prog [options]\nsends nagios performance data to carbon.\n\"\"\")\n\nparser.add_option('-v', \"--verbose\", action=\"store_true\", dest=\"verbose\",\n help=\"sets logging to DEBUG level\")\nparser.add_option('-q', \"--quiet\", action=\"store_true\", dest=\"quiet\",\n help=\"sets logging to WARNING level\")\nparser.add_option(\"--spool-directory\", dest=\"spool_directory\",\n default=spool_directory,\n help=\"where to look for nagios performance data\")\nparser.add_option(\"--log-file\", dest=\"log_file\",\n default=log_file,\n help=\"file to log to\")\nparser.add_option(\"--backend\", dest=\"backend\", default=\"stdout\",\n help=\"sets which storage backend to use\")\nparser.add_option(\"--config_file\", dest=\"config_file\", default=\"\",\n help=\"set custom config file location\")\nparser.add_option(\"--test\", action=\"store_true\", dest=\"test\", default=\"\",\n help=\"Turns on test mode, which won't send to backends\")\nparser.add_option(\"--replace_char\", dest=\"replace_char\", default=\"_\",\n help=\"Replacement Character (default '_'\")\nparser.add_option(\"--sleep_time\", dest=\"sleep_time\", default=15,\n help=\"How much time to sleep between checks\")\nparser.add_option(\"--sleep_max\", dest=\"sleep_max\", default=480,\n help=\"Max time to sleep between runs\")\nparser.add_option(\"--server\", dest=\"server\", default=\"\",\n help=\"Server address (for backend)\")\nparser.add_option(\"--no_replace_hostname\", action=\"store_false\",\n dest=\"replace_hostname\", default=True,\n help=\"Replace '.' in nagios hostnames, default on.\")\nparser.add_option(\"--reverse_hostname\", action=\"store_true\",\n dest=\"reverse_hostname\",\n help=\"Reverse nagios hostname, default off.\")\n\n\nlog = logging.getLogger('log')\n\n\nclass GraphiosMetric(object):\n def __init__(self):\n self.LABEL = '' # The name in the perfdata from nagios\n self.VALUE = '' # The measured value of that metric\n self.UOM = '' # The unit of measure for the metric\n self.DATATYPE = '' # HOSTPERFDATA|SERVICEPERFDATA\n self.METRICTYPE = 'gauge' # gauge|counter|timer etc..\n self.TIMET = '' # Epoc time the measurement was taken\n self.HOSTNAME = '' # name of th host measured\n self.SERVICEDESC = '' # nagios configured service description\n self.PERFDATA = '' # the space-delimited raw perfdata\n self.SERVICECHECKCOMMAND = '' # literal check command syntax\n self.HOSTCHECKCOMMAND = '' # literal check command syntax\n self.HOSTSTATE = '' # current state afa nagios is concerned\n self.HOSTSTATETYPE = '' # HARD|SOFT\n self.SERVICESTATE = '' # current state afa nagios is concerned\n self.SERVICESTATETYPE = '' # HARD|SOFT\n self.METRICBASEPATH = '' # Establishes a root base path\n self.GRAPHITEPREFIX = '' # graphios prefix\n self.GRAPHITEPOSTFIX = '' # graphios suffix\n self.VALID = False # if this metric is valid\n\n if 'metric_base_path' in cfg:\n self.METRICBASEPATH = cfg['metric_base_path']\n\n def validate(self):\n # because we eliminated all whitespace, there shouldn't be any quotes\n # this happens more with windows nagios plugins\n re.sub(\"'\", \"\", self.LABEL)\n re.sub('\"', \"\", self.LABEL)\n re.sub(\"'\", \"\", self.VALUE)\n re.sub('\"', \"\", self.VALUE)\n self.check_adjust_hostname()\n if (\n self.TIMET is not '' and\n self.PERFDATA is not '' and\n self.HOSTNAME is not ''\n ):\n if \"use_service_desc\" in cfg and cfg[\"use_service_desc\"] is True:\n if self.SERVICEDESC != '' or self.DATATYPE == 'HOSTPERFDATA':\n self.VALID = True\n else:\n # not using service descriptions\n if (\n # We should keep this logic and not check for a\n # base path here. Just because there's a base path\n # doesn't mean the metric should be considered valid\n self.GRAPHITEPREFIX == \"\" and\n self.GRAPHITEPOSTFIX == \"\"\n ):\n self.VALID = False\n else:\n self.VALID = True\n\n def check_adjust_hostname(self):\n if cfg[\"reverse_hostname\"]:\n self.HOSTNAME = '.'.join(reversed(self.HOSTNAME.split('.')))\n if cfg[\"replace_hostname\"]:\n self.HOSTNAME = self.HOSTNAME.replace(\".\",\n cfg[\"replacement_character\"])\n\n\ndef chk_bool(value):\n \"\"\"\n checks if value is a stringified boolean\n \"\"\"\n if (value.lower() == \"true\"):\n return True\n elif (value.lower() == \"false\"):\n return False\n return value\n\n\ndef read_config(config_file):\n \"\"\"\n reads the config file\n \"\"\"\n if config_file == '':\n # check same dir as graphios binary\n my_file = \"%s/graphios.cfg\" % sys.path[0]\n if os.path.isfile(my_file):\n config_file = my_file\n else:\n # check /etc/graphios/graphios.cfg\n config_file = \"/etc/graphios/graphios.cfg\"\n config = SafeConfigParser()\n # The logger won't be initialized yet, so we use print_debug\n if os.path.isfile(config_file):\n config.read(config_file)\n config_dict = {}\n for section in config.sections():\n # there should only be 1 'graphios' section\n print_debug(\"section: %s\" % section)\n config_dict['name'] = section\n for name, value in config.items(section):\n config_dict[name] = chk_bool(value)\n print_debug(\"config[%s]=%s\" % (name, value))\n # print config_dict\n return config_dict\n else:\n print_debug(\"Can't open config file: %s\" % config_file)\n print \"\"\"\\nEither modify the script at the config_file = '' line and\nspecify where you want your config file to be, or create a config file\nin the above directory (which should be the same dir the graphios.py is in)\nor you can specify --config=myconfigfilelocation at the command line.\"\"\"\n sys.exit(1)\n\n\ndef verify_config(config_dict):\n \"\"\"\n verifies the required config variables are found\n \"\"\"\n global spool_directory\n ensure_list = ['replacement_character', 'log_file', 'log_max_size',\n 'log_level', 'sleep_time', 'sleep_max', 'test_mode',\n 'reverse_hostname', 'replace_hostname']\n missing_values = []\n for ensure in ensure_list:\n if ensure not in config_dict:\n missing_values.append(ensure)\n if len(missing_values) > 0:\n print \"\\nMust have value in config file for:\\n\"\n for value in missing_values:\n print \"%s\\n\" % value\n sys.exit(1)\n if not config_dict['log_level'] in loglevels.keys():\n print \"Unknown loglevel: \" + config_dict['log_level'] + '\\n'\n print \"Available loglevels:\"\n print '\\n'.join(sorted(loglevels.keys()))\n sys.exit(1)\n if \"spool_directory\" in config_dict:\n spool_directory = config_dict['spool_directory']\n\n\ndef print_debug(msg):\n \"\"\"\n prints a debug message if global debug is True\n \"\"\"\n if debug:\n print msg\n\n\ndef verify_options(opts):\n \"\"\"\n verify the passed command line options, puts into global cfg\n \"\"\"\n global cfg\n global spool_directory\n # because these have defaults in the parser section we know they will be\n # set. So we don't have to do a bunch of ifs.\n if \"log_file\" not in cfg:\n cfg[\"log_file\"] = opts.log_file\n if cfg[\"log_file\"] == \"''\" or cfg[\"log_file\"] == \"\":\n cfg[\"log_file\"] = \"%s/graphios.log\" % sys.path[0]\n cfg[\"log_max_size\"] = 24\n if opts.verbose:\n cfg[\"debug\"] = True\n cfg[\"log_level\"] = \"logging.DEBUG\"\n elif opts.quiet:\n cfg[\"debug\"] = False\n cfg[\"log_level\"] = \"logging.WARNING\"\n else:\n cfg[\"debug\"] = False\n cfg[\"log_level\"] = \"logging.INFO\"\n if opts.test:\n cfg[\"test_mode\"] = True\n else:\n cfg[\"test_mode\"] = False\n cfg[\"replacement_character\"] = opts.replace_char\n cfg[\"spool_directory\"] = opts.spool_directory\n cfg[\"sleep_time\"] = opts.sleep_time\n cfg[\"sleep_max\"] = opts.sleep_max\n cfg[\"replace_hostname\"] = opts.replace_hostname\n cfg[\"reverse_hostname\"] = opts.reverse_hostname\n spool_directory = opts.spool_directory\n # cfg[\"backend\"] = opts.backend\n handle_backends(opts)\n # cfg[\"enable_carbon\"] = True\n return cfg\n\n\ndef handle_backends(opts):\n global cfg\n if opts.backend == \"carbon\" or opts.backend == \"statsd\":\n if not opts.server:\n print \"Must also have --server for carbon or statsd.\"\n sys.exit(1)\n if opts.backend == \"carbon\":\n cfg[\"enable_carbon\"] = True\n cfg[\"carbon_servers\"] = opts.server\n if opts.backend == \"statsd\":\n cfg[\"enable_statsd\"] = True\n cfg[\"statsd_server\"] = opts.server\n if opts.backend == \"librato\":\n print \"Use graphios.cfg for librato.\"\n sys.exit(1)\n\n\ndef configure():\n \"\"\"\n sets up graphios config\n \"\"\"\n global debug\n try:\n cfg[\"log_max_size\"] = int(cfg[\"log_max_size\"])\n except ValueError:\n print \"log_max_size needs to be a integer\"\n sys.exit(1)\n\n # Convert cfg[\"log_max_size\"] to bytes. Assume its already in bytes\n # if its > 1000000\n if cfg[\"log_max_size\"] < 1000000:\n log_max_bytes = cfg[\"log_max_size\"]*1024*1024\n else:\n log_max_bytes = cfg[\"log_max_size\"]\n\n log_handler = logging.handlers.RotatingFileHandler(\n cfg[\"log_file\"], maxBytes=log_max_bytes, backupCount=4,\n # encoding='bz2')\n )\n formatter = logging.Formatter(\n \"%(asctime)s %(filename)s %(levelname)s %(message)s\",\n \"%B %d %H:%M:%S\")\n log_handler.setFormatter(formatter)\n log.addHandler(log_handler)\n\n if cfg.get(\"debug\") is True or cfg['log_level'] == 'logging.DEBUG':\n log.debug(\"adding streamhandler\")\n log.setLevel(logging.DEBUG)\n log.addHandler(logging.StreamHandler())\n debug = True\n else:\n log.setLevel(loglevels[cfg['log_level']])\n debug = False\n\n\ndef process_log(file_name):\n \"\"\" process log lines into GraphiosMetric Objects.\n input is a tab delimited series of key/values each of which are delimited\n by '::' it looks like:\n DATATYPE::HOSTPERFDATA TIMET::1399738074 etc..\n \"\"\"\n processed_objects = [] # the final list of metric objects we'll return\n graphite_lines = 0 # count the number of valid lines we process\n try:\n host_data_file = open(file_name, \"r\")\n file_array = host_data_file.readlines()\n host_data_file.close()\n except (IOError, OSError) as ex:\n log.critical(\"Can't open file:%s error: %s\" % (file_name, ex))\n sys.exit(2)\n # parse each line into a metric object\n for line in file_array:\n if not re.search(\"^DATATYPE::\", line):\n continue\n # log.debug('parsing: %s' % line)\n graphite_lines += 1\n variables = line.split('\\t')\n mobj = get_mobj(variables)\n if mobj:\n # break out the metric object into one object per perfdata metric\n # log.debug('perfdata:%s' % mobj.PERFDATA)\n for metric in mobj.PERFDATA.split():\n try:\n nobj = copy.copy(mobj)\n (nobj.LABEL, d) = metric.split('=')\n v = d.split(';')[0]\n u = v\n nobj.VALUE = re.sub(\"[a-zA-Z%]\", \"\", v)\n nobj.UOM = re.sub(\"[^a-zA-Z]+\", \"\", u)\n processed_objects.append(nobj)\n except:\n log.critical(\"failed to parse label: '%s' part of perf\"\n \"string '%s'\" % (metric, nobj.PERFDATA))\n continue\n return processed_objects\n\n\ndef get_mobj(nag_array):\n \"\"\"\n takes a split array of nagios variables and returns a mobj if it's\n valid. otherwise return False.\n \"\"\"\n mobj = GraphiosMetric()\n for var in nag_array:\n # drop the metric if we can't split it for any reason\n try:\n (var_name, value) = var.split('::', 1)\n except:\n log.warn(\"could not split value %s, dropping metric\" % var)\n return False\n\n value = re.sub(\"/\", cfg[\"replacement_character\"], value)\n if re.search(\"PERFDATA\", var_name):\n mobj.PERFDATA = value\n elif re.search(\"^\\$_\", value):\n continue\n else:\n value = re.sub(\"\\s\", \"\", value)\n setattr(mobj, var_name, value)\n mobj.validate()\n if mobj.VALID is True:\n return mobj\n return False\n\n\ndef handle_file(file_name, graphite_lines):\n \"\"\"\n archive processed metric lines and delete the input log files\n \"\"\"\n if \"test_mode\" in cfg and cfg[\"test_mode\"] is True:\n log.debug(\"graphite_lines:%s\" % graphite_lines)\n else:\n try:\n os.remove(file_name)\n except (OSError, IOError) as ex:\n log.critical(\"couldn't remove file %s error:%s\" % (file_name, ex))\n else:\n log.debug(\"deleted %s\" % file_name)\n\n\ndef process_spool_dir(directory):\n \"\"\"\n processes the files in the spool directory\n \"\"\"\n global be\n log.debug(\"Processing spool directory %s\", directory)\n num_files = 0\n mobjs_len = 0\n try:\n perfdata_files = os.listdir(directory)\n except (IOError, OSError) as e:\n print \"Exception '%s' reading spool directory: %s\" % (e, directory)\n print \"Check if dir exists, or file permissions.\"\n print \"Exiting.\"\n sys.exit(1)\n for perfdata_file in perfdata_files:\n mobjs = []\n processed_dict = {}\n all_done = True\n file_dir = os.path.join(directory, perfdata_file)\n if check_skip_file(perfdata_file, file_dir):\n continue\n num_files += 1\n mobjs = process_log(file_dir)\n mobjs_len = len(mobjs)\n processed_dict = send_backends(mobjs)\n # process the output from the backends and decide the fate of the file\n for backend in be[\"essential_backends\"]:\n if processed_dict[backend] < mobjs_len:\n log.critical(\"keeping %s, insufficent metrics sent from %s. \\\n Should be %s, got %s\" % (file_dir, backend,\n mobjs_len,\n processed_dict[backend]))\n all_done = False\n if all_done is True:\n handle_file(file_dir, len(mobjs))\n log.info(\"Processed %s files (%s metrics) in %s\" % (num_files,\n mobjs_len, directory))\n\n\ndef check_skip_file(file_name, file_dir):\n \"\"\"\n checks if file should be skipped\n \"\"\"\n if (\n file_name == \"host-perfdata\" or\n file_name == \"service-perfdata\"\n ):\n return True\n elif re.match('^_', file_name):\n return True\n\n if os.stat(file_dir)[6] == 0:\n # file was 0 bytes\n handle_file(file_dir, 0)\n return True\n if os.path.isdir(file_dir):\n return True\n return False\n\n\ndef init_backends():\n \"\"\"\n I'm going to be a little forward thinking with this and build a global dict\n of enabled back-ends whose values are instantiations of the back-end\n objects themselves. I know, global bad, but hypothetically we could modify\n this dict dynamically via a runtime-interface (graphiosctl?) to turn off/on\n backends without having to restart the graphios process. I feel like\n that's enough of a win to justify the global. If you build it, they will\n come.\n \"\"\"\n global be\n be = {} # a top-level global for important backend-related stuff\n be[\"enabled_backends\"] = {} # a dict of instantiated backend objects\n be[\"essential_backends\"] = [] # a list of backends we actually care about\n # PLUGIN WRITERS! register your new backends by adding their obj name here\n avail_backends = (\"carbon\",\n \"statsd\",\n \"librato\",\n \"influxdb\",\n \"influxdb09\",\n \"stdout\",\n )\n # populate the controller dict from avail + config. this assumes you named\n # your backend the same as the config option that enables your backend (eg.\n # carbon and enable_carbon)\n for backend in avail_backends:\n cfg_option = \"enable_%s\" % (backend)\n if cfg_option in cfg and cfg[cfg_option] is True:\n backend_obj = getattr(backends, backend)\n be[\"enabled_backends\"][backend] = backend_obj(cfg)\n nerf_option = \"nerf_%s\" % (backend)\n if nerf_option in cfg:\n if cfg[nerf_option] is False:\n be[\"essential_backends\"].append(backend)\n else:\n be[\"essential_backends\"].append(backend)\n # not proud of that slovenly conditional ^^\n log.info(\"Enabled backends: %s\" % be[\"enabled_backends\"].keys())\n\n\ndef send_backends(metrics):\n \"\"\"\n use the enabled_backends dict to call into the backend send functions\n \"\"\"\n global be\n if len(be[\"enabled_backends\"]) < 1:\n log.critical(\"At least one Back-end must be enabled in graphios.cfg\")\n sys.exit(1)\n ret = {} # return a dict of who processed what\n processed_lines = 0\n for backend in be[\"enabled_backends\"]:\n processed_lines = be[\"enabled_backends\"][backend].send(metrics)\n # log.debug('%s processed %s metrics' % backend, processed_lines)\n ret[backend] = processed_lines\n return ret\n\n\ndef main():\n log.info(\"graphios startup.\")\n try:\n while True:\n process_spool_dir(spool_directory)\n log.debug(\"graphios sleeping.\")\n time.sleep(float(cfg[\"sleep_time\"]))\n except KeyboardInterrupt:\n log.info(\"ctrl-c pressed. Exiting graphios.\")\n\n\nif __name__ == '__main__':\n if len(sys.argv) > 1:\n (options, args) = parser.parse_args()\n # print options\n if options.config_file:\n cfg = read_config(options.config_file)\n else:\n cfg = verify_options(options)\n else:\n cfg = read_config(config_file)\n verify_config(cfg)\n configure()\n # print cfg\n init_backends()\n main()\n" }, { "path": "graphios_backends.py", "content": "# vim: set ts=4 sw=4 tw=79 et :\n\nimport socket\nimport cPickle as pickle\nimport struct\nimport re\nimport logging\nimport sys\nimport base64\nimport urllib2\nimport json\nimport os\nimport ast\n# ###########################################################\n# #### Librato Backend\n\n\nclass librato(object):\n def __init__(self, cfg):\n \"\"\"\n Implements the librato backend-module\n \"\"\"\n\n self.log = logging.getLogger(\"log.backends.librato\")\n self.log.info(\"Librato Backend Initialized\")\n self.api = \"https://metrics-api.librato.com\"\n self.sink_name = \"graphios-librato\"\n self.sink_version = \"0.0.1\"\n self.flush_timeout_secs = 5\n self.gauges = {}\n self.whitelist = []\n self.metrics_sent = 0\n self.max_metrics_payload = 500\n\n try:\n cfg[\"librato_email\"]\n except:\n self.log.critical(\"please define librato_email in graphios.cfg\")\n sys.exit(1)\n else:\n self.email = cfg['librato_email']\n\n try:\n cfg[\"librato_token\"]\n except:\n self.log.critical(\"please define librato_token in graphios.cfg\")\n sys.exit(1)\n else:\n self.token = cfg['librato_token']\n\n try:\n cfg['librato_namevals']\n except:\n self.namevals = ['METRICBASEPATH', 'GRAPHITEPREFIX', 'SERVICEDESC',\n 'GRAPHITEPOSTFIX', 'LABEL']\n else:\n self.namevals = cfg['librato_namevals'].split(\",\")\n\n try:\n cfg['librato_sourcevals']\n except:\n self.sourcevals = ['HOSTNAME']\n else:\n self.sourcevals = cfg['librato_sourcevals'].split(\",\")\n\n try:\n cfg[\"librato_floor_time_secs\"]\n except:\n self.floor_time_secs = 15\n else:\n self.floor_time_secs = cfg[\"librato_floor_time_secs\"]\n\n try:\n cfg[\"librato_whitelist\"]\n except:\n self.whitelist = [re.compile(\".*\")]\n else:\n for pattern in json.loads(cfg[\"librato_whitelist\"]):\n self.log.debug(\"adding librato whitelist pattern %s\" % pattern)\n self.whitelist.append(re.compile(pattern))\n\n def build_path(self, vals, m):\n path = ''\n for s in vals:\n path += getattr(m, s)\n path += '.'\n path = re.sub(r\"^\\.\", '', path) # fix sources that begin in dot\n path = re.sub(r\"\\.$\", '', path) # fix sources that end in dot\n path = re.sub(r\"\\.\\.\", '.', path) # fix sources with double dots\n path = re.sub(r\"['\\\"]\", '.', path) # fix sources with imbedded quotes\n return path\n\n def k_not_in_whitelist(self, k):\n # return True if k isn't whitelisted\n # wl_match = True\n for pattern in self.whitelist:\n if pattern.search(k) is not None:\n return False\n return True\n\n def add_measure(self, m):\n ts = int(m.TIMET)\n if self.floor_time_secs is not None:\n ts = (ts / self.floor_time_secs) * self.floor_time_secs\n\n source = self.build_path(self.sourcevals, m)\n name = self.build_path(self.namevals, m)\n\n k = \"%s\\t%s\" % (name, source)\n\n # bail if this metric isn't whitelisted\n if self.k_not_in_whitelist(k):\n return None\n\n # add the metric to our gauges dict\n if k not in self.gauges:\n self.gauges[k] = {\n 'name': name,\n 'source': source,\n 'measure_time': ts,\n }\n\n value = float(m.VALUE)\n self.gauges[k]['value'] = value\n\n def flush_payload(self, headers, g):\n \"\"\"\n POST a payload to Librato.\n \"\"\"\n body = json.dumps({'gauges': g})\n url = \"%s/v1/metrics\" % (self.api)\n req = urllib2.Request(url, body, headers)\n\n try:\n f = urllib2.urlopen(req, timeout=self.flush_timeout_secs)\n # f.read()\n f.close()\n except urllib2.HTTPError as error:\n self.metrics_sent = 0\n body = error.read()\n self.log.warning('Failed to send metrics to Librato: Code: \\\n %d . Response: %s' % (error.code, body))\n except IOError as error:\n self.metrics_sent = 0\n if hasattr(error, 'reason'):\n self.log.warning('Error when sending metrics Librato \\\n (%s)' % (error.reason))\n elif hasattr(error, 'code'):\n self.log.warning('Error when sending metrics Librato \\\n (%s)' % (error.code))\n else:\n self.log.warning('Error when sending metrics Librato \\\n and I dunno why')\n\n def flush(self):\n \"\"\"\n POST a collection of gauges to Librato.\n \"\"\"\n # Nothing to do\n if len(self.gauges) == 0:\n return 0\n\n # Limit our payload sizes\n # max_metrics_payload = 500 # this is never used, delete it?\n\n headers = {\n 'Content-Type': 'application/json',\n 'User-Agent': self.build_user_agent(),\n 'Authorization': 'Basic %s' % self.build_basic_auth()\n }\n\n metrics = []\n count = 0\n for g in self.gauges.values():\n metrics.append(g)\n count += 1\n\n if count >= self.max_metrics_payload:\n self.flush_payload(headers, metrics)\n count = 0\n metrics = []\n\n if count > 0:\n self.flush_payload(headers, metrics)\n self.gauges = {}\n\n def build_basic_auth(self):\n\n base64string = base64.encodestring('%s:%s' % (self.email, self.token))\n return base64string.translate(None, '\\n')\n\n def build_user_agent(self):\n\n sink_name = \"graphios-librato\"\n sink_version = \"0.0.1\"\n\n try:\n uname = os.uname()\n system = \"; \".join([uname[0], uname[4]])\n except:\n system = os.name()\n\n pver = sys.version_info\n user_agent = '%s/%s (%s) Python-Urllib2/%d.%d' % \\\n (sink_name, sink_version,\n system, pver[0], pver[1])\n return user_agent\n\n def send(self, metrics):\n\n self.metrics_sent = len(metrics)\n # Construct the output\n for m in metrics:\n self.add_measure(m)\n\n # Flush\n self.flush()\n\n return self.metrics_sent\n\n\n############################################################\n# #### Carbon back-end #####\n\nclass carbon(object):\n def __init__(self, cfg):\n self.log = logging.getLogger(\"log.backends.carbon\")\n self.log.info(\"Carbon Backend Initialized\")\n try:\n cfg['carbon_servers']\n except:\n self.carbon_servers = '127.0.0.1'\n else:\n self.carbon_servers = cfg['carbon_servers']\n\n try:\n cfg['replacement_character']\n except:\n self.replacement_character = '_'\n else:\n self.replacement_character = cfg['replacement_character']\n\n try:\n cfg['carbon_max_metrics']\n self.carbon_max_metrics = cfg['carbon_max_metrics']\n except:\n self.carbon_max_metrics = 200\n\n try:\n self.carbon_max_metrics = int(self.carbon_max_metrics)\n except ValueError:\n self.log.critical(\"carbon_max_metrics needs to be a integer\")\n sys.exit(1)\n\n try:\n cfg['use_service_desc']\n self.use_service_desc = cfg['use_service_desc']\n except:\n self.use_service_desc = False\n\n try:\n cfg['metric_base_path']\n self.metric_base_path = cfg['metric_base_path']\n except:\n self.metric_base_path = ''\n\n try:\n cfg['test_mode']\n self.test_mode = cfg['test_mode']\n except:\n self.test_mode = False\n\n # try:\n # cfg['replace_hostname']\n # self.replace_hostname = cfg['replace_hostname']\n # except:\n # self.replace_hostname = True\n\n try:\n cfg['carbon_plaintext']\n self.carbon_plaintext = cfg['carbon_plaintext']\n except:\n self.carbon_plaintext = False\n\n def convert_messages(self, metrics):\n \"\"\"\n Converts the metric obj list into graphite messages\n \"\"\"\n metric_list = []\n messages = []\n for m in metrics:\n path = self.build_path(m)\n value = m.VALUE\n timestamp = m.TIMET\n if self.carbon_plaintext:\n metric_item = \"%s %s %s\\n\" % (path, value, timestamp)\n else:\n metric_item = (path, (timestamp, value))\n if self.test_mode:\n print \"%s %s %s\" % (path, value, timestamp)\n metric_list.append(metric_item)\n for metric_list_chunk in self.chunks(metric_list,\n self.carbon_max_metrics):\n if self.carbon_plaintext:\n messages.append(\"\".join(metric_list_chunk))\n else:\n payload = pickle.dumps(metric_list_chunk)\n header = struct.pack(\"!L\", len(payload))\n message = header + payload\n messages.append(message)\n return messages\n\n def chunks(self, l, n):\n \"\"\" Yield successive n-sized chunks from l.\n \"\"\"\n for i in xrange(0, len(l), n):\n yield l[i:i + n]\n\n def build_path(self, m):\n \"\"\"\n Builds a carbon metric\n \"\"\"\n pre = \"\"\n if m.METRICBASEPATH != \"\":\n pre = \"%s.\" % m.METRICBASEPATH\n if m.GRAPHITEPREFIX != \"\":\n pre += \"%s.\" % m.GRAPHITEPREFIX\n if m.GRAPHITEPOSTFIX != \"\":\n post = \".%s\" % m.GRAPHITEPOSTFIX\n else:\n post = \"\"\n # if self.replace_hostname:\n # hostname = m.HOSTNAME.replace('.', self.replacement_character)\n # else:\n hostname = m.HOSTNAME\n if self.use_service_desc:\n # we want: (prefix.)hostname.service_desc(.postfix).perfdata\n service_desc = self.fix_string(m.SERVICEDESC)\n path = \"%s%s.%s%s.%s\" % (pre, hostname, service_desc, post,\n m.LABEL)\n else:\n path = \"%s%s%s.%s\" % (pre, hostname, post, m.LABEL)\n path = re.sub(r\"\\.$\", '', path) # fix paths that end in dot\n path = re.sub(r\"\\.\\.\", '.', path) # fix paths with double dots\n path = self.fix_string(path)\n return path\n\n def fix_string(self, my_string):\n \"\"\"\n takes a string and replaces whitespace and invalid carbon chars with\n the global replacement_character\n \"\"\"\n invalid_chars = '~!$:;%^*()+={}[]|\\/<>'\n my_string = re.sub(\"\\s\", self.replacement_character, my_string)\n for char in invalid_chars:\n my_string = my_string.replace(char, self.replacement_character)\n return my_string\n\n def send(self, metrics):\n \"\"\"\n Connect to the Carbon server\n Send the metrics\n \"\"\"\n ret = 0\n sock = socket.socket()\n servers = self.carbon_servers.split(\",\")\n for serv in servers:\n if \":\" in serv:\n server, port = serv.split(\":\")\n port = int(port)\n else:\n server = serv\n if self.carbon_plaintext:\n port = 2003\n else:\n port = 2004\n self.log.debug(\"Connecting to carbon at %s:%s\" % (server, port))\n try:\n sock.connect((socket.gethostbyname(server), port))\n self.log.debug(\"connected\")\n except Exception, ex:\n self.log.warning(\"Can't connect to carbon: %s:%s %s\" % (\n server, port, ex))\n\n messages = self.convert_messages(metrics)\n try:\n for message in messages:\n sock.sendall(message)\n except Exception, ex:\n self.log.critical(\"Can't send message to carbon error:%s\" % ex)\n sock.close()\n return 0\n # this only gets returned if nothing failed.\n ret += len(metrics)\n sock.close()\n return ret\n\n\n# ###########################################################\n# #### statsd backend #######################################\n\nclass statsd(object):\n def __init__(self, cfg):\n self.log = logging.getLogger(\"log.backends.statsd\")\n self.log.info(\"Statsd backend initialized\")\n try:\n cfg['statsd_servers']\n except:\n self.statsd_servers = '127.0.0.1'\n else:\n self.statsd_servers = cfg['statsd_servers']\n\n def set_type(self, metric):\n # detect and set the metric type\n if re.search(\"gauge\", metric.METRICTYPE):\n return 'g'\n elif re.search(\"counter\", metric.METRICTYPE):\n return 'c'\n elif re.search(\"time\", metric.METRICTYPE):\n return 'ms'\n elif re.search(\"set\", metric.METRICTYPE):\n return 's'\n else:\n return 'g' # default to gauge\n\n def convert(self, metrics):\n # Converts the metric object list into a list of statsd tuples\n out_list = []\n for m in metrics:\n path = '%s.%s.%s.%s.%s' % (m.METRICBASEPATH, m.GRAPHITEPREFIX,\n m.HOSTNAME, m.GRAPHITEPOSTFIX,\n m.LABEL)\n path = re.sub(r'\\.$', '', path) # fix paths that end in dot\n path = re.sub(r'\\.\\.', '.', path) # fix paths with empty values\n mtype = self.set_type(m) # gauge|counter|timer|set\n value = \"%s|%s\" % (m.VALUE, mtype) # emit literally this to statsd\n metric_tuple = \"%s:%s\" % (path, value)\n out_list.append(metric_tuple)\n\n return out_list\n\n def send(self, metrics):\n # Fire metrics at the statsd server and hope for the best (loludp)\n ret = True\n sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)\n\n mlist = self.convert(metrics)\n ret = 0\n servers = self.statsd_servers.split(\",\")\n for serv in servers:\n if \":\" in serv:\n server, port = serv.split(\":\")\n port = int(port)\n else:\n server = serv\n port = 8125\n self.log.debug(\"sending to statsd at %s:%s\" % (server, port))\n for m in mlist:\n try:\n sock.sendto(m, (socket.gethostbyname(server), port))\n except Exception, ex:\n self.log.critical(\"Can't send metric to statsd error:%s\"\n % ex)\n else:\n ret += 1\n\n return ret\n\n\n# ###########################################################\n# #### influxdb backend ####################################\n\nclass influxdb(object):\n def __init__(self, cfg):\n self.log = logging.getLogger(\"log.backends.influxdb\")\n self.log.info(\"InfluxDB backend initialized\")\n self.scheme = \"http\"\n self.default_ports = {'https': 8087, 'http': 8086}\n self.timeout = 5\n\n if 'influxdb_use_ssl' in cfg:\n if cfg['influxdb_use_ssl']:\n self.scheme = \"https\"\n\n if 'influxdb_servers' in cfg:\n self.influxdb_servers = cfg['influxdb_servers'].split(',')\n else:\n self.influxdb_servers = ['127.0.0.1:%i' %\n self.default_ports[self.scheme]]\n\n if 'influxdb_user' in cfg:\n self.influxdb_user = cfg['influxdb_user']\n else:\n self.log.critical(\"Missing influxdb_user in graphios.cfg\")\n sys.exit(1)\n\n if 'influxdb_password' in cfg:\n self.influxdb_password = cfg['influxdb_password']\n else:\n self.log.critical(\"Missing influxdb_password in graphios.cfg\")\n sys.exit(1)\n\n if 'influxdb_db' in cfg:\n self.influxdb_db = cfg['influxdb_db']\n else:\n self.influxdb_db = \"nagios\"\n\n if 'influxdb_max_metrics' in cfg:\n self.influxdb_max_metrics = cfg['influxdb_max_metrics']\n else:\n self.influxdb_max_metrics = 250\n\n try:\n self.influxdb_max_metrics = int(self.influxdb_max_metrics)\n except ValueError:\n self.log.critical(\"influxdb_max_metrics needs to be a integer\")\n sys.exit(1)\n\n def build_url(self, server):\n \"\"\" Returns a url to specified InfluxDB-server \"\"\"\n test_port = server.split(':')\n if len(test_port) < 2:\n server = \"%s:%i\" % (server, self.default_ports[self.scheme])\n\n return \"%s://%s/db/%s/series?u=%s&p=%s\" % (self.scheme, server,\n self.influxdb_db,\n self.influxdb_user,\n self.influxdb_password)\n\n def build_path(self, m):\n \"\"\" Returns a path \"\"\"\n path = \"\"\n if m.METRICBASEPATH != \"\":\n path += \"%s.\" % m.METRICBASEPATH\n\n if m.GRAPHITEPREFIX != \"\":\n path += \"%s.\" % m.GRAPHITEPREFIX\n\n path += \"%s.\" % m.HOSTNAME\n\n if m.SERVICEDESC != \"\":\n path += \"%s.\" % m.SERVICEDESC\n\n path = \"%s%s\" % (path, m.LABEL)\n\n if m.GRAPHITEPOSTFIX != \"\":\n path = \"%s.%s\" % (path, m.GRAPHITEPOSTFIX)\n\n return path\n\n def chunks(self, l, n):\n \"\"\" Yield successive n-sized chunks from l. \"\"\"\n for i in xrange(0, len(l), n):\n yield l[i:i+n]\n\n def url_request(self, url, chunk):\n json_body = json.dumps(chunk)\n req = urllib2.Request(url, json_body)\n req.add_header('Content-Type', 'application/json')\n return req\n\n def _send(self, server, chunk):\n self.log.debug(\"Connecting to InfluxDB at %s\" % server)\n req = self.url_request(self.build_url(server), chunk)\n\n try:\n r = urllib2.urlopen(req, timeout=self.timeout)\n r.close()\n return True\n except urllib2.HTTPError as e:\n body = e.read()\n self.log.warning('Failed to send metrics to InfluxDB. \\\n Status code: %d: %s' % (e.code, body))\n return False\n except IOError as e:\n fail_string = \"Failed to send metrics to InfluxDB. \"\n if hasattr(e, 'code'):\n fail_string = fail_string + \"Status code: %s\" % e.code\n if hasattr(e, 'reason'):\n fail_string = fail_string + str(e.reason)\n self.log.warning(fail_string)\n return False\n\n def send(self, metrics):\n \"\"\" Connect to influxdb and send metrics \"\"\"\n ret = 0\n perfdata = {}\n series = []\n for m in metrics:\n ret += 1\n\n path = self.build_path(m)\n\n if path not in perfdata:\n perfdata[path] = []\n\n # influx assumes timestamp in milliseconds\n timet_ms = int(m.TIMET)*1000\n\n # Ensure a int/float gets passed\n try:\n value = int(m.VALUE)\n except ValueError:\n try:\n value = float(m.VALUE)\n except ValueError:\n value = 0\n\n perfdata[path].append([timet_ms, value])\n\n for k, v in perfdata.iteritems():\n series.append({\"name\": k, \"columns\": [\"time\", \"value\"],\n \"points\": v})\n\n series_chunks = self.chunks(series, self.influxdb_max_metrics)\n for chunk in series_chunks:\n for s in self.influxdb_servers:\n if not self._send(s, chunk):\n ret = 0\n\n return ret\n\n\n# ###########################################################\n# #### influxdb-0.9 backend ####################################\n\nclass influxdb09(influxdb):\n def __init__(self, cfg):\n influxdb.__init__(self, cfg)\n if 'influxdb_extra_tags' in cfg:\n self.influxdb_extra_tags = ast.literal_eval(\n cfg['influxdb_extra_tags'])\n print self.influxdb_extra_tags\n else:\n self.influxdb_extra_tags = {}\n\n try:\n cfg['influxdb_line_protocol']\n self.influxdb_line_protocol = cfg['influxdb_line_protocol']\n except:\n self.influxdb_line_protocol = False\n\n def build_url(self, server):\n \"\"\" Returns a url to specified InfluxDB-server \"\"\"\n test_port = server.split(':')\n if len(test_port) < 2:\n server = \"%s:%i\" % (server, self.default_ports[self.scheme])\n\n if self.influxdb_line_protocol:\n return \"%s://%s/write?u=%s&p=%s&db=%s\" % (self.scheme, server,\n self.influxdb_user,\n self.influxdb_password,\n self.influxdb_db)\n else:\n return \"%s://%s/write?u=%s&p=%s\" % (self.scheme, server,\n self.influxdb_user,\n self.influxdb_password)\n\n def url_request(self, url, chunk):\n if self.influxdb_line_protocol:\n req = urllib2.Request(url, chunk)\n req.add_header('Content-Type', 'application/x-www-form-urlencoded')\n return req\n else:\n return super(influxdb09, self).url_request(url, chunk)\n\n def format_metric(self, timestamp, path, tags, value):\n if not self.influxdb_line_protocol:\n return {\n \"timestamp\": timestamp,\n \"measurement\": path,\n \"tags\": tags,\n \"fields\": {\"value\": value}}\n return '%s,%s value=%s %d' % (\n path,\n ','.join(['%s=%s' % (k, v) for k, v in tags.iteritems() if v]),\n value,\n timestamp * 10 ** 9\n )\n\n def format_series(self, chunk):\n if self.influxdb_line_protocol:\n return '\\n'.join(chunk)\n else:\n return {\"database\": self.influxdb_db, \"points\": chunk}\n\n def send(self, metrics):\n \"\"\" Connect to influxdb and send metrics \"\"\"\n ret = 0\n perfdata = []\n for m in metrics:\n ret += 1\n\n if (m.SERVICEDESC == ''):\n path = m.HOSTCHECKCOMMAND\n else:\n path = m.SERVICEDESC\n\n # Ensure a int/float gets passed\n try:\n value = int(m.VALUE)\n except ValueError:\n try:\n value = float(m.VALUE)\n except ValueError:\n value = 0\n\n tags = {\"check\": m.LABEL, \"host\": m.HOSTNAME}\n tags.update(self.influxdb_extra_tags)\n\n perfdata.append(self.format_metric(int(m.TIMET), path,\n tags, value))\n\n series_chunks = self.chunks(perfdata, self.influxdb_max_metrics)\n for chunk in series_chunks:\n series = self.format_series(chunk)\n for s in self.influxdb_servers:\n if not self._send(s, series):\n ret = 0\n\n return ret\n\n\n# ###########################################################\n# #### stdout backend #######################################\n\nclass stdout(object):\n def __init__(self, cfg):\n self.log = logging.getLogger(\"log.backends.stdout\")\n self.log.info(\"STDOUT Backend Initialized\")\n\n def send(self, metrics):\n ret = 0\n for metric in metrics:\n ret += 1\n print(\"%s:%s\" % ('LABEL', metric.LABEL))\n print(\"%s:%s\" % ('VALUE ', metric.VALUE))\n print(\"%s:%s\" % ('UOM ', metric.UOM))\n print(\"%s:%s\" % ('DATATYPE ', metric.DATATYPE))\n print(\"%s:%s\" % ('TIMET ', metric.TIMET))\n print(\"%s:%s\" % ('HOSTNAME ', metric.HOSTNAME))\n print(\"%s:%s\" % ('SERVICEDESC ', metric.SERVICEDESC))\n print(\"%s:%s\" % ('PERFDATA ', metric.PERFDATA))\n print(\"%s:%s\" % ('SERVICECHECKCOMMAND',\n metric.SERVICECHECKCOMMAND))\n print(\"%s:%s\" % ('HOSTCHECKCOMMAND ', metric.HOSTCHECKCOMMAND))\n print(\"%s:%s\" % ('HOSTSTATE ', metric.HOSTSTATE))\n print(\"%s:%s\" % ('HOSTSTATETYPE ', metric.HOSTSTATETYPE))\n print(\"%s:%s\" % ('SERVICESTATE ', metric.SERVICESTATE))\n print(\"%s:%s\" % ('SERVICESTATETYPE ', metric.SERVICESTATETYPE))\n print(\"%s:%s\" % ('METRICBASEPATH ', metric.METRICBASEPATH))\n print(\"%s:%s\" % ('GRAPHITEPREFIX ', metric.GRAPHITEPREFIX))\n print(\"%s:%s\" % ('GRAPHITEPOSTFIX ', metric.GRAPHITEPOSTFIX))\n print(\"-------\")\n\n return ret\n\n\n# ###########################################################\n# #### start here #######################################\n\nif __name__ == \"__main__\":\n print(\"I'm just a lowly module. Try calling graphios.py instead\")\n sys.exit(42)\n" }, { "path": "init/debian/graphios", "content": "#! /bin/bash\n### BEGIN INIT INFO\n# Provides: graphios\n# Required-Start: $local_fs $remote_fs $syslog $named $network $time nagios3\n# Required-Stop: $local_fs $remote_fs $syslog $named $network\n# Should-Start:\n# Should-Stop:\n# Default-Start: 2 3 4 5\n# Default-Stop: 0 1 6\n# Short-Description: graphios bridge between nagios and graphite\n# Description: graphios feeds nagios performance data to graphite\n### END INIT INFO\n# File : graphios\n\n# Source function library.\n. /lib/lsb/init-functions\n\nprog=\"/usr/local/bin/graphios.py\"\n# or use the command line options:\n#prog=\"/opt/nagios/bin/graphios.py --log-file=/dir/mylog.log --spool-directory=/dir/my/sool\"\nGRAPHIOS_USER=\"nagios\"\nNAME=\"graphios\"\nDESC=\"nagios to graphite bridge\"\nRETVAL=0\n\nstart () {\n log_daemon_msg \"Starting $DESC\" \"$NAME\"\n start-stop-daemon --start --quiet --name $(basename $prog) --user $GRAPHIOS_USER \\\n\t\t --chuid $GRAPHIOS_USER --exec $prog --background\n log_end_msg $?\n echo\n}\n\nstop () {\n log_daemon_msg \"Stopping $DESC\" \"$NAME\"\n start-stop-daemon --stop --oknodo --quiet --name $(basename $prog) --user $GRAPHIOS_USER\n log_end_msg $?\n echo\n}\n\nrestart () {\n stop\n start\n}\n\n# See how we are called.\ncase \"$1\" in\n start)\n start\n ;;\n stop)\n stop\n ;;\n restart|reload)\n restart\n ;;\n status)\n status $prog\n RETVAL=$?\n ;;\n *)\n echo \"Usage: service graphios {start|stop|restart|reload}\"\n RETVAL=2\n ;;\nesac\n\nexit $RETVAL\n" }, { "path": "init/debian/graphios.conf", "content": "description \"Graphios: Emit nagios perfdata to graphite/statsd/librato\"\nauthor \"Shawn Sterling \"\n\nstart on runlevel [234]\nstop on runlevel [016]\n\nexec /usr/local/bin/graphios.py\n" }, { "path": "init/rhel/graphios", "content": "#!/bin/bash\n#\n# graphios start the graphios script\n#\n#\n# chkconfig: 345 99 01\n# description: graphios nagios -> graphite script\n#\n# File : graphios\n\n# Source function library.\n. /etc/init.d/functions\n\n# Source networking configuration.\n. /etc/sysconfig/network\n\n# Check that networking is up.\n[ \"$NETWORKING\" = \"no\" ] && exit 0\n\nprog=\"/usr/bin/graphios\"\n# or use the command line options:\n#prog=\"/usr/bin/graphios --log-file=/dir/mylog.log --spool-directory=/dir/my/sool\"\nGRAPHIOS_USER=\"nagios\"\nRETVAL=0\n\nstart () {\n echo -n \"Starting $prog\"\n /usr/bin/sudo -u ${GRAPHIOS_USER} ${prog} &\n RETVAL=$?\n [ $RETVAL -eq 0 ] && success || failure\n echo\n}\n\nstop () {\n echo -n \"Stopping $prog\"\n killproc graphios\n RETVAL=$?\n [ $RETVAL -eq 0 ] && success || failure\n echo\n}\n\nrestart () {\n stop\n start\n}\n\n\n# See how we are called.\ncase \"$1\" in\n start)\n start\n ;;\n stop)\n stop\n ;;\n restart|reload)\n restart\n ;;\n status)\n status $prog\n RETVAL=$?\n ;;\n *)\n echo \"Usage: service graphios {start|stop|restart|reload}\"\n RETVAL=2\n ;;\nesac\n\nexit $RETVAL\n" }, { "path": "init/rhel/install", "content": "python setup.py install --optimize 1 --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES\n" }, { "path": "init/rhel/postinstall", "content": "# add symlink\nln -s /usr/bin/graphios.py /usr/bin/graphios\n\n# add graphios service\nif [ ! -d /etc/systemd ]; then\n # we are running init\n if ! chkconfig --add graphios; then\n logger -p user.err -s -t %name -- \"Error adding graphios service.\"\n exit 1\n fi\nfi\n" }, { "path": "init/rhel/postuninstall", "content": "if [ ! -d /etc/systemd ]; then\n # we are running init\n service graphios stop\nelse\n # we are running systemd\n systemctl stop graphios.service\nfi\n\nunlink /usr/bin/graphios\n" }, { "path": "init/systemd/graphios.service", "content": "[Unit]\nDescription=graphios - a script to emit nagios perfdata to various backends\n\n[Service]\nExecStart=/usr/bin/graphios.py\nRestart=on-abort\n\n[Install]\nWantedBy=multi-user.target\n" }, { "path": "nagios/graphios_commands.cfg", "content": "define command {\n command_name graphios_perf_host\n command_line /bin/mv /var/spool/nagios/graphios/host-perfdata /var/spool/nagios/graphios/host-perfdata.$TIMET$\n}\n\ndefine command {\n command_name graphios_perf_service\n command_line /bin/mv /var/spool/nagios/graphios/service-perfdata /var/spool/nagios/graphios/service-perfdata.$TIMET$\n}\n" }, { "path": "nagios/nagios_perfdata.cfg", "content": "\n###### Auto-generated Graphios configs #######\nprocess_performance_data=1\nservice_perfdata_file=/var/spool/nagios/graphios/service-perfdata\nservice_perfdata_file_template=DATATYPE::SERVICEPERFDATA\\tTIMET::$TIMET$\\tHOSTNAME::$HOSTNAME$\\tSERVICEDESC::$SERVICEDESC$\\tSERVICEPERFDATA::$SERVICEPERFDATA$\\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\\tHOSTSTATE::$HOSTSTATE$\\tHOSTSTATETYPE::$HOSTSTATETYPE$\\tSERVICESTATE::$SERVICESTATE$\\tSERVICESTATETYPE::$SERVICESTATETYPE$\\tGRAPHITEPREFIX::$_SERVICEGRAPHITEPREFIX$\\tGRAPHITEPOSTFIX::$_SERVICEGRAPHITEPOSTFIX$\\tMETRICTYPE::$_SERVICEMETRICTYPE$\nservice_perfdata_file_mode=a\nservice_perfdata_file_processing_interval=15\nservice_perfdata_file_processing_command=graphios_perf_service\nhost_perfdata_file=/var/spool/nagios/graphios/host-perfdata\nhost_perfdata_file_template=DATATYPE::HOSTPERFDATA\\tTIMET::$TIMET$\\tHOSTNAME::$HOSTNAME$\\tHOSTPERFDATA::$HOSTPERFDATA$\\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\\tHOSTSTATE::$HOSTSTATE$\\tHOSTSTATETYPE::$HOSTSTATETYPE$\\tGRAPHITEPREFIX::$_HOSTGRAPHITEPREFIX$\\tGRAPHITEPOSTFIX::$_HOSTGRAPHITEPOSTFIX$\\tMETRICTYPE::$_HOSTMETRICTYPE$\nhost_perfdata_file_mode=a\nhost_perfdata_file_processing_interval=15\nhost_perfdata_file_processing_command=graphios_perf_host\n" }, { "path": "setup.cfg", "content": "[bdist_rpm]\nprovides = graphios\npost-install = init/rhel/postinstall\ninstall-script = init/rhel/install\npost-uninstall = init/rhel/postuninstall\n" }, { "path": "setup.py", "content": "# vim: set ts=4 sw=4 tw=79 et :\nfrom setuptools import setup\nfrom setuptools.command.install import install as _install\nfrom shutil import Error as FileError\nfrom shutil import copy\nfrom time import strftime\nimport os\nimport platform\nimport re\n\n\ndef find_nagios_cfg(lookin):\n \"\"\"\n finds the nagios.cfg given list of directories\n \"\"\"\n for path in lookin:\n for root, dirs, files in os.walk(path):\n if \"nagios.cfg\" in files:\n return os.path.join(root, \"nagios.cfg\")\n\n\ndef parse_nagios_cfg(nag_cfg):\n \"\"\"\n parses the nagios.cfg\n \"\"\"\n nconfig = {}\n inputfile = open(nag_cfg, 'r')\n for line in inputfile:\n if re.match('[a-zA-Z]', line[0]):\n try:\n option, value = line.split(\"=\")\n except:\n continue\n else:\n nconfig[option.rstrip()] = value.rstrip()\n inputfile.close()\n return nconfig\n\n\ndef add_perfdata_config(nconfig, nag_cfg):\n \"\"\"\n adds the graphios perfdata cfg to the nagios.cfg\n \"\"\"\n with open('nagios/nagios_perfdata.cfg') as f:\n main_config = f.read().splitlines()\n\n with open('nagios/graphios_commands.cfg') as f:\n commands = f.read().splitlines()\n\n # add the graphios commands\n cstat = os.stat(nag_cfg)\n print(\"nagios uid: %s gid: %s\" % (cstat.st_uid, cstat.st_gid))\n if \"cfg_dir\" in nconfig:\n command_file = os.path.join(nconfig[\"cfg_dir\"],\n 'graphios_commands.cfg')\n else:\n command_dir = os.path.join(os.path.dirname(nag_cfg), \"objects\")\n main_config.append(\"cfg_dir=%s\" % command_dir)\n if not os.path.exists(command_dir):\n os.mkdir(command_dir)\n os.chown(command_dir, cstat.st_uid, cstat.st_gid)\n command_file = os.path.join(command_dir, \"graphios_commands.cfg\")\n cfile = open(command_file, 'a')\n for line in commands:\n cfile.writelines(\"%s\\n\" % line)\n cfile.close()\n os.chown(command_file, cstat.st_uid, cstat.st_gid)\n write_main_config(nconfig, nag_cfg, main_config)\n\n\ndef write_main_config(nconfig, nag_cfg, main_config):\n \"\"\"\n writes the nagios.cfg\n \"\"\"\n # now add the main config\n nfile = open(nag_cfg, 'a')\n if (\n \"process_performance_data\" in nconfig and\n nconfig[\"process_performance_data\"] == \"1\"\n ):\n print(\"pre-existing perfdata config detected\")\n for line in main_config:\n nfile.writelines(\"# %s\\n\" % line)\n else:\n for line in main_config:\n nfile.writelines(\"%s\\n\" % line)\n nfile.close()\n\n\ndef _post_install():\n \"\"\"\n tries to find the nagios.cfg and insert graphios perf commands/cfg\n \"\"\"\n lookin = ['/etc/nagios/', '/opt/nagios/', '/usr/local/nagios',\n '/usr/nagios']\n nag_cfg = find_nagios_cfg(lookin)\n if nag_cfg is None:\n print(\"sorry I couldn't find the nagios.cfg file\")\n print(\"NO POST INSTALL COULD BE PERFORMED\")\n else:\n print(\"found nagios.cfg in %s\" % nag_cfg)\n nconfig = parse_nagios_cfg(nag_cfg)\n print(\"parsed nagcfg, nagios_log is at %s\" % nconfig['log_file'])\n if backup_file(nag_cfg):\n add_perfdata_config(nconfig, nag_cfg)\n else:\n print(\"Backup failed, add modify nagios.cfg manually.\")\n\n\ndef backup_file(file_name):\n \"\"\"\n backs up the nagios.cfg, incase we break something.\n \"\"\"\n my_time = strftime('%d-%m-%y')\n new_file_name = \"%s.%s\" % (file_name, my_time)\n print(\"backing up file:%s to %s\" % (file_name, new_file_name))\n try:\n copy(file_name, new_file_name)\n return True\n except (FileError, IOError, OSError) as e:\n print(\"Error:%s copying:%s to:%s\" % (e, file_name, new_file_name))\n return False\n\n\nclass my_install(_install):\n \"\"\"\n installs graphios\n \"\"\"\n def run(self):\n _install.run(self)\n self.execute(_post_install, [], msg=\"Running post install task\")\n\ndata_files = [\n (('/etc/graphios'), [\"graphios.cfg\"])\n]\nscripts = [\"graphios.py\"]\n\ndistro = platform.dist()[0]\ndistro_ver = int(platform.dist()[1].split('.')[0])\n\n# print \"using %s %s\" % (distro, distro_ver)\n\nif distro in ['Ubuntu', 'debian']:\n data_files.append(('/etc/init/', ['init/debian/graphios.conf']))\n data_files.append(('/usr/local/bin/', ['graphios.py']))\n data_files.append(('/etc/init.d/', ['init/debian/graphios']))\nelif distro in ['centos', 'redhat', 'fedora']:\n data_files.append(('/usr/bin', ['graphios.py']))\n if distro_ver >= 7:\n data_files.append(('/usr/lib/systemd/system',\n ['init/systemd/graphios.service']))\n elif distro_ver < 7:\n data_files.append(('/etc/rc.d/init.d', ['init/rhel/graphios']))\n\n# print data_files\nsetup(\n name='graphios',\n version='2.0.0b3',\n description='Emit Nagios metrics to Graphite, Statsd, and Librato',\n author='Shawn Sterling',\n author_email='shawn@systemtemplar.org',\n url='https://github.com/shawn-sterling/graphios',\n license='GPL v2',\n scripts=['graphios.py'],\n data_files=data_files,\n py_modules=['graphios_backends'],\n cmdclass={'install': my_install},\n classifiers=[\n 'Development Status :: 4 - Beta',\n 'Intended Audience :: System Administrators',\n 'Programming Language :: Python :: 2.7',\n ],\n keywords='Nagios metrics graphing visualization',\n long_description='Graphios is a script to send nagios perfdata to various\\\n backends like graphite, statsd, and librato. \\\n https://github.com/shawn-sterling/graphios'\n)\n" } ]