[ { "path": "README", "content": "Introduction\n------------\n\npastemon.pl is a script which runs in the background as a daemon and monitors pastebin.com for\ninteresting content (based on regular expressions). Found information is sent to syslog\n\nThe script can also generate (CEF events).\n\nMore information is available here: \nhttp://blog.rootshell.be/2012/01/17/monitoring-pastebin-com-within-your-siem/\n\nv1.14 - 2012/10/31\n------------------\n- [FEATURE] Added SQLite DB support to store pasties details\n (some fields must still be implemented)\n\nv1.13 - 2012/10/24\n------------------\n- [CONTRIBUTION] Added support for multiple SMTP recipients (email addresses separared by commas)\n Contribution from coreyroach@hotmail.com\n- [CONTRIBUTION] Added a new macro-% to specify the site name in the dump function.\n '%S' will be replaced by the site name. Example: '%S/%Y/%M' => 'pastebin.com/2012/10'. \n\nv1.12 - 2012/09/20\n------------------\n- [BUGFIX] Fixed FuzzyMatch() which was broken with gziped pasties.\n- [FEATURE] Email notification: The subject is now appended with the field(s)\n corresponding to the matched regex(es). This allows a better view of received emails as well\n as filtering them.\n- [BUGFIX] Fixed FuzzyMatch() to detect properly duplicate pasties (fixed regex).\n\nv1.11 - 2012/09/14\n------------------\n- [FEATURE] Added support for nopaste.me.\n- [FEATURE] Added support for pastesite.com.\n- [FEATURE] Added configurable sleep delays per pastie website.\n\nv1.10 - 2012/08/01\n------------------\n- [FEATURE] If not configuration file is specified, pastemon.pl tries to load /etc/pastemon.conf\n by default.\n\n- [FEATURE] pastemon.pl uses specific Perl modules like WordPress::XMLRPC or Text::JaroWinkler.\n The script now handles properly environment without those modules. It's not required to comment\n them in the code. If a module is missing, related configuration is automatically disabled.\n\n- [FEATURE] Added optional compression (via IO::Compress::Gzip) of dumped pasties.\n In configuration file:\n\n\t\n\t\tyes\n\t\n\nv1.9 - 2012/07/23\n-----------------\n- [FEATURE] pastemon can now follow (search for regex) URLs detected in pasties. This is\n configured via the main configuration file:\n\n\t\n\t\tyes\n\t\t(bit\\.ly)\n\t\n\n- [FEATURE] The regex.conf format changed to an XML format.\n Examples:\n\t\n\t\t\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\n\t\t10\n\t\tIP Address\n\t\n\n- [FEATURE] A minimum number of regex occurences can be defined to notify\n ( tag in the XML file)\n\n- [FEATURE] HTTP requests are now using now a random User-Agent. \n\n- [BUGFIX] Optimized the detection of already processed pasties. This reduces the amount of HTTP \n requests send to the website.\n\nv1.8 - 2012/06/25\n-----------------\n- [FEATURE] Adder for pastie.org! \n- [FEATURE] Added multi-thread support (1 thread per website monitored)\n- [FEATURE] Added substitution macro in the dump directory. Support macros are:\n\t%Y - replace with the current year\n\t%M - replace with the current month\n\t%D - replace with the current day\n Directory is automatically created.\n Example: /home/user/pastemon/%Y/%M/%D\n\n- [FEATURE] Added a new configuration directive:\n\tyes|1\n This feature enables a dump of *ALL* pastie wheter they match a regex or not.\n This is similar to a mirror mode\n WARNING: Huge disk space might be required by this feature!\n- [BUGFIX] Test if the provided SMTP server (for mail notifications) is available\n (Thanks to @manuelsubredu for the patch)\n- [BUGFIX] Fixed an issue in createBlogPost() which caused an unexpected process exit.\n\nv1.7 - 2012/05/11\n-----------------\n- Added support for \"included\" regular expressions\n- Fixed in bug in getRegexDesc()\n- Added support for comments ('#') in the regex configuration file\n- Moved configuration parameters from command line switches to an XML file\n- Added matching regex description in dump files \n- Added SMTP notifications\n- Added distance check to detect duplicate pasties (using Jaro-Winkler algorithm)\n\nv1.6 - 2012/02/21\n-----------------\n- Added a detection of \"slow down\" messages returned by Pastebin (add a small pause)\n- Added support for Wordpress XMLRPC \n- Added support for random proxies\n- Some bug fixes\n\nv1.5 - 2012/02/19\n-----------------\n- Fixed the regex to grab pasties from the archive page. (HTML code changed)\n\nv1.4 - 2012/02/15\n-----------------\n- Fixed a bug with CEF events: custome fields start at 1 not 0! (Thanks to Heiko Hansen for the report)\n- Notify the presence of a proxy variable (HTTP_PROXY) \n\nv1.3 - 2012/01/26\n-----------------\n- Added a '--pidfile=file' configuration switch to specify an alternative location for the PID file.\n This allows the script to be executed with a non-root account.\n- Added a '--sample=x' configuration to display a sample a data matching a regular expression. 'x' is \n the number of bytes displayed before and after the matching string. This is useful to estimate the\n value of the pastie. Example:\n Found in http://pastebin.com/raw.php?i=Q8pQRHKW : belgium (2 times) | Sample: g(0) \"\"\\n [32] => string(11) \"Belgium(32)\"\\n [31] => string(14) \"Ne\n\nv1.2 - 2012/01/21\n-----------------\n- Fixed a bug affecting the case sensitivity search\n- New feature: an exception can be associated to a regular expression in the configuration file.\n The syntax is: \"regex1 _EXCLUDE_ regex2\". This could prevent some false positive matches.\n\nv1.1 - 2012/01/20\n-----------------\n- Added a '--dump' configuration switch to save matching pasties in a directory.\n This is to keep the pasties posted with an expiration date (example: for later review)\n\nv1.0 - 2012/01/18\n-----------------\nInitial release\n" }, { "path": "pastemon.conf.sample", "content": "\n\n \n \n yes\n /var/run/pastemon.pid\n regex.conf\n 256\n proxies.conf\n\t\tuser-agents.conf\n /home/pastemon/dump/%Y/%M/%D\n yes\n\t\tyes\n\t\t15\n\t\t\n\t\t0.95\n\t\t10240\n \n\n \n \n yes\n\t\t10\n yes\n\t\t120\n\t\tyes\n\t\t300\n\t\tyes\n\t\t300\n \n\n \n \n yes\n (anonpaste|pastebin\\.com|pastie\\.org|pastehtml\\.com|pastebay\\.net|pastee\\.org)\n \n\n \n \n 10.0.0.1\n 514\n 3\n \n\n \n \n daemon\n \n\n \n \n 127.0.0.1\n pastemon@rootshell.be\n recipient@domain.com\n PasteMon Alert\n \n\n \n \n www.myblog.com\n editor\n averystrongpassword\n favorite\n \n\n\t\n\t\n\t\t/home/pastemon/pastemon.db\n\t\n\n" }, { "path": "pastemon.pl", "content": "#!/usr/bin/perl\n#\n# pastemon.pl \n#\n# This script runs in the background as a daemon and monitors pastebin.com for\n# interesting content (based on regular expressions). Found information is sent\n# to syslog\n#\n# This script is based on the Python script written by Xavier Garcia\n# (http://www.shellguardians.com/2011/07/monitoring-pastebin-leaks.html)\n#\n# Copyright (c) 2012 Xavier Mertens\n# All rights reserved.\n#\n# Redistribution and use in source and binary forms, with or without\n# modification, are permitted provided that the following conditions\n# are met:\n# 1. Redistributions of source code must retain the above copyright\n# notice, this list of conditions and the following disclaimer.\n# 2. Redistributions in binary form must reproduce the above copyright\n# notice, this list of conditions and the following disclaimer in the\n# documentation and/or other materials provided with the distribution.\n# 3. Neither the name of copyright holders nor the names of its\n# contributors may be used to endorse or promote products derived\n# from this software without specific prior written permission.\n#\n# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n# ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED\n# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR\n# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL COPYRIGHT HOLDERS OR CONTRIBUTORS\n# BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n# POSSIBILITY OF SUCH DAMAGE.\n#\n# History\n# -------\n# See README file\n\nuse strict;\nuse threads;\nuse threads::shared;\nuse Digest::MD5 qw(md5 md5_hex md5_base64);\nuse File::Path; \nuse Getopt::Long;\nuse IO::Socket;\nuse LWP::UserAgent;\nuse HTML::Entities;\nuse Sys::Syslog;\nuse Encode;\nuse XML::XPath;\nuse XML::XPath::XMLParser;\nuse Net::SMTP;\nuse POSIX qw(setsid);\n\n# Optional modules\nmy $haveWordPressXMLRMC \t= eval \"use WordPress::XMLRPC; 1\";\nmy $haveTextJaroWinkler \t= eval \"use Text::JaroWinkler qw(strcmp95); 1\";\nmy $haveIOCompressGzip\t\t= eval \"use IO::Compress::Gzip; 1\";\nmy $haveIOUncompressGunzip\t= eval \"use IO::Uncompress::Gunzip; 1\";\nmy $haveDBI\t\t\t= eval \"use DBI; 1\";\n\nuse constant PROCESS_URL\t=> 1;\nuse constant PASTEBIN \t\t=> 0;\t# Supported websites\nuse constant PASTIE \t\t=> 1;\nuse constant NOPASTE\t\t=> 2;\nuse constant PASTESITE\t\t=> 3;\nmy @webSiteNames = ( \t\t\t# Self-defined names for multiple usages\n\t\t\"pastebin.com\",\n\t\t\"pastie.net\",\n\t\t\"nopaste.me\",\n\t\t\"pastesite.com\",\n\t);\n\nmy $program = \"pastemon.pl\";\nmy $version = \"v1.14\";\nmy $debug;\nmy $help;\nmy $ignoreCase;\t\t# By default respect case in strings search\nmy $cefDestination;\t# Send CEF events to this destination:port\nmy $cefPort = 514;\nmy $cefSeverity = 3;\nmy $caught = 0;\nmy $httpTimeout = 10;\t# Default HTTP timeout\nmy @pasties;\nmy @seenPasties;\nmy $maxPasties = 1000;\t# TODO: Make it configurable?\nmy @regexList;\t\t# List of interesting regex (with the data)\nmy $pidFile \t= \"/var/run/pastemon.pid\";\nmy $configFile\t= \"/etc/pastemon.conf\";\t\t# Main XML configuration file\nmy $regexFile;\t\t# Regular expressions definitions\nmy $wpConfigFile;\nmy $proxyFile;\nmy @proxies;\n\nmy $uaFile;\nmy @uas;\n\nmy $wpSite;\t\t# Wordpress settings\nmy $wpUser;\nmy $wpPass;\nmy $wpCategory;\n\nmy $smtpServer;\t\t# SMTP settings\nmy $smtpFrom;\nmy $smtpRecipient;\nmy $smtpSubject;\nmy @smtpRecipients;\n\nmy $distanceMin;\nmy $distanceMaxSize;\n\nmy $followUrls;\t\t# Follow URLs found in pastie\nmy $followMatching;\n\nmy $checkPastebin;\t# Websites to monitor\nmy $checkPastie;\nmy $checkNopaste;\nmy $checkPastesite;\n\nmy $delayPastebin\t= 300;\t# Delays between pasties fetches\nmy $delayPastie\t\t= 300;\nmy $delayNopaste\t= 300;\nmy $delayPastesite\t= 300;\n\nmy $syslogFacility = \"daemon\";\nmy $dumpDir;\nmy $dumpAll;\nmy $compressDump;\nmy $sampleSize;\nmy %matches;\n\nmy $dbFile;\t\t# SQLite3 DB file\n\n# Process arguments\nmy $result = GetOptions(\n\t\"debug\"\t\t\t=> \\$debug,\n\t\"help\"\t\t\t=> \\$help,\n\t\"config=s\"\t\t=> \\$configFile,\n);\n\n# TODO: Add a \"--drop-sql-table\" option to rebuild a fresh DB?\nif ($help) {\n\tprint <<__HELP__;\nUsage: $0 --config=filepath [--debug] [--help]\nWhere:\n--config : Specify the XML configuration file\n--debug : Enable debug mode (verbose - do not detach)\n--help : What you're reading now.\n__HELP__\n\texit 0;\n}\n\nparseXMLConfigFile($configFile);\n\n($debug) && print STDERR \"+++ Running in foreground.\\n\";\n\n($cefDestination) && syslogOutput(\"Sending CEF events to $cefDestination:$cefPort (severity $cefSeverity)\");\n\n# Do not allow multiple running instances!\nif (-r $pidFile) {\n\topen(PIDH, \"<$pidFile\") || die \"Cannot read pid file!\";\n\tmy $currentpid = ;\n\tclose(PIDH);\n\tdie \"$program already running (PID $currentpid)\";\n}\n\nloadRegexFromFile($regexFile) || die \"Cannot load regex from file $regexFile\";\nloadUserAgentFromFile($uaFile) || die \"Cannot load user-agents from file $uaFile\";\n\nif (!$debug) {\n\tmy $pid = fork;\n\tdie \"Cannot fork\" unless defined($pid);\n\texit(0) if $pid;\n\n\t# We are the child\n\t(POSIX::setsid != -1) or die \"setsid failed\";\n\tchdir(\"/\") || die \"Cannot changed working directory to /\";\n\tclose(STDOUT);\n\tclose(STDOUT);\n\tclose(STDIN);\n}\n\nsyslogOutput(\"Running with PID $$\");\nopen(PIDH, \">$pidFile\") || die \"Cannot write PID file $pidFile: $!\";\nprint PIDH \"$$\";\nclose(PIDH);\n\n# Notify if HTTP proxy settings detected\nif ($ENV{'HTTP_PROXY'}) {\n\t($proxyFile) && die \"The HTTP_PROXY environment variable conflicts with the use of a proxies list\";\n\tsyslogOutput(\"Using detected HTTP proxy: \" . $ENV{'HTTP_PROXY'});\n}\n\nmy @threads;\nmy @webSites;\n($checkPastebin) && push(@webSites, PASTEBIN);\n($checkPastie) && push(@webSites, PASTIE);\n($checkNopaste) && push(@webSites, NOPASTE);\n($checkPastesite) && push(@webSites, PASTESITE);\n\n# Launch threads based on the number of webistes to monitor\nfor my $webSite (@webSites) {\n\tmy $t = threads->new(\\&mainLoop, $webSite);\n\tpush(@threads, $t);\n}\n\n$SIG{'TERM'}\t= \\&sigHandler;\n$SIG{'INT'}\t= \\&sigHandler;\n$SIG{'KILL'}\t= \\&sigHandler;\n$SIG{'USR1'}\t= sub {\n\t\t\tforeach my $t (@threads) {\n\t\t\t\t$t->kill('SIGUSR1');\n\t\t\t}\n\t\t };\n\n# Parent process just waiting for a signal\nwhile(1) {\n\tsleep(1);\n\tif ($caught) {\n\t\tsyslogOutput(\"Killing my threads\");\n\t\tforeach my $t (@threads) {\n\t\t\t$t->kill('SIGKILL');\n\t\t}\n\t}\n}\n\nexit 0;\n\n# ---------\n# Main loop\n# ---------\nsub mainLoop {\n\t$SIG{'USR1'}\t= \\&sigReload; # Handle config reload\n\t$SIG{'KILL'} = \\&sigHandler;\n\tmy $webSite = shift;\n\twhile(1) {\n\t\tmy $pastie;\n\t\tif (!&fetchLastPasties($webSite)) {\n\t\t\tforeach $pastie (@pasties) {\n\t\t\t\texit 0 if ($caught == 1);\n\t\t\t\tanalyzePastie($webSite, $pastie, PROCESS_URL);\n\t\t\t}\n\t\t\texit 0 if ($caught == 1);\n\t\t}\n\t\tpurgeOldPasties($maxPasties);\n\n\t\t# Wait some seconds (depending on the website)\n\t\tDELAY: {\n\t\t\t$webSite == PASTEBIN\t&& do { \n\t\t\t\t\t\t\t($debug) && print STDERR \"Sleeping $delayPastebin\\n\"; \n\t\t\t\t\t\t\tsleep($delayPastebin); last DELAY; };\n\t\t\t$webSite == PASTIE\t&& do {\n\t\t\t\t\t\t\t($debug) && print STDERR \"Sleeping $delayPastie\\n\";\n\t\t\t\t\t\t\tsleep($delayPastie); last DELAY; };\n\t\t\t$webSite == NOPASTE\t&& do {\n\t\t\t\t\t\t\t($debug) && print STDERR \"Sleeping $delayNopaste\\n\";\n\t\t\t\t\t\t\tsleep($delayNopaste); last DELAY; };\n\t\t\t$webSite == PASTESITE\t&& do {\n\t\t\t\t\t\t\t($debug) && print STDERR \"Sleeping $delayPastesite\\n\";\n\t\t\t\t\t\t\tsleep($delayPastesite); last DELAY; };\n\t\t}\n\t}\n}\n\n#\n# analyzePastie\n#\nsub analyzePastie {\n\tmy $webSite = shift;\n\tmy $pastie = shift or return;\n\tmy $processUrl = shift;\n\tmy $regex;\n\tmy $md5;\n\tif (!grep /$pastie/, @seenPasties) {\n\t\tmy $content = fetchPastie($pastie);\n\t\tif ($content) {\n\t\t\t# If we receive a \"slow down\" message, follow Pastebin recommandation!\n\t\t\tif ($content =~ /Please slow down/) {\n\t\t\t\t($debug) && print STDERR \"+++ Slow down message received. Paused 5 seconds\\n\";\n\t\t\t\tsleep(5);\n\t\t\t}\n\t\t\telse {\n\t\t\t\t# Compute the MD5 digest\n\t\t\t\t$md5 = md5_hex(encode('UTF8',$content));\n\t\t\t\tif (!dbSearchMD5($md5)) { \n\t\t\t\t\tundef(%matches);\t# Reset the matches regex/counters\n\t\t\t\t\tmy $i = 0;\n\t\t\t\t\tmy $regexSearch;\n\t\t\t\t\tmy $regexInclude;\n\t\t\t\t\tmy $regexExclude;\n\t\t\t\t\tmy $regexDesc;\n\t\t\t\t\tmy $regexCount;\n\t\t\t\t\tforeach $regex (@regexList) {\n\t\t\t\t\t\t$regexSearch\t= @$regex[0];\n\t\t\t\t\t\t$regexInclude\t= @$regex[1];\n\t\t\t\t\t\t$regexExclude\t= @$regex[2];\n\t\t\t\t\t\t$regexDesc\t= @$regex[3];\n\t\t\t\t\t\t$regexCount\t= @$regex[4];\n\t\t\t\t\t\tmy $sampleData;\n\t\t\t\t\t\tmy ($startPos, $endPos);\n\t\t\t\t\t\tmy $preCount = 0;\n\t\t\t\t\t\tif ($ignoreCase) {\n\t\t\t\t\t\t\t$preCount += () = $content =~ /$regexSearch/mgi;\n\t\t\t\t\t\t\t$startPos = $-[0];\n\t\t\t\t\t\t\t$endPos = $+[0];\n\t\t\t\t\t\t}\n\t\t\t\t\t\telse {\n\t\t\t\t\t\t\t$preCount += () = $content =~ /$regexSearch/mg;\n\t\t\t\t\t\t\t$startPos = $-[0];\n\t\t\t\t\t\t\t$endPos = $+[0];\n\t\t\t\t\t\t}\n\t\t\t\t\t\tif ($preCount >= $regexCount) {\t\n\t\t\t\t\t\t\tif ($sampleSize) {\n\t\t\t\t\t\t\t\t# Optional: extract a sample of the data\n\t\t\t\t\t\t\t\t$startPos = (($startPos - $sampleSize) < 0) ? 0 : ($startPos - $sampleSize);\n\t\t\t\t\t\t\t\t$sampleData = encode('UTF8', substr($content, $startPos, ($endPos - $startPos) + $sampleSize));\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t# Process \"include\" regex defined\n\t\t\t\t\t\t\tif ($regexInclude ne \"\") {\n\t\t\t\t\t\t\t\tmy $postCount = 0;\n\t\t\t\t\t\t\t\tif ($ignoreCase) {\n\t\t\t\t\t\t\t\t\t$postCount += () = $content =~ /$regexInclude/mgi;\n\t\t\t\t\t\t\t\t} else {\n\t\t\t\t\t\t\t\t\t$postCount += () = $content =~ /$regexInclude/mg;\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\tif ($postCount) {\n\t\t\t\t\t\t\t\t\t# Matches for include $regex\n\t\t\t\t\t\t\t\t\t$matches{$i} = [ ( $regexSearch, $preCount, $sampleData ) ];\n\t\t\t\t\t\t\t\t\t$i++;\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\telsif ($regexExclude ne \"\") {\n\t\t\t\t\t\t\t\tmy $postCount = 0;\n\t\t\t\t\t\t\t\tif ($ignoreCase) {\n\t\t\t\t\t\t\t\t\t$postCount += () = $content =~ /$regexExclude/mgi;\n\t\t\t\t\t\t\t\t} else {\n\t\t\t\t\t\t\t\t\t$postCount += () = $content =~ /$regexExclude/mg;\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\tif (! $postCount) {\n\t\t\t\t\t\t\t\t\t# Matches for exclude $regex\n\t\t\t\t\t\t\t\t\t$matches{$i} = [ ( $regexSearch, $preCount, $sampleData ) ];\n\t\t\t\t\t\t\t\t\t$i++;\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\telse {\n\t\t\t\t\t\t\t\t$matches{$i} = [ ( $regexSearch, $preCount, $sampleData ) ];\n\t\t\t\t\t\t\t\t$i++;\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\tif ($followUrls && $processUrl) {\n\t\t\t\t\t\t$i += processUrls($content);\n\t\t\t\t\t}\n\t\t\t\t\tif ($i) {\n\t\t\t\t\t\t# Try to find a corresponding pastie?\n\t\t\t\t\t\tif (!FuzzyMatch($webSite, $content))\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t# Generate the results based on matches\n\t\t\t\t\t\t\tmy $buffer = \"Found in \" . $pastie . \" : \";\n\t\t\t\t\t\t\tmy $key;\n\t\t\t\t\t\t\tfor $key (keys %matches) {\n\t\t\t\t\t\t\t\t$buffer = $buffer . $matches{$key}[0] . \" (\" . $matches{$key}[1] . \" times) \";\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\tif ($sampleSize) {\n\t\t\t\t\t\t\t\t# Optional: Add sample of data\n\t\t\t\t\t\t\t\tmy $safeData = $matches{0}[2];\n\t\t\t\t\t\t\t\t# Sanitize the data\n\t\t\t\t\t\t\t\t$safeData =~ s/\r//g;\n\t\t\t\t\t\t\t\t$safeData =~ s/\\n/\\\\r/g;\n\t\t\t\t\t\t\t\t$safeData =~ s/\\n/\\\\n/g;\n\t\t\t\t\t\t\t\t$safeData =~ s/\\t/\\\\t/g;\n\t\t\t\t\t\t\t\t$buffer = $buffer . \"| Sample: \" . $safeData;\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\tsyslogOutput($buffer);\n\t\n\t\t\t\t\t\t\t# Generating CEF event (if configured)\n\t\t\t\t\t\t\t($cefDestination) && sendCEFEvent($pastie);\n\t\t\n\t\t\t\t\t\t\t# Generating blog post (if configured)\n\t\t\t\t\t\t\t($wpSite) && createBlogPost($pastie);\n\t\n\t\t\t\t\t\t\t# Send SMTP notification (if configured)\n\t\t\t\t\t\t\tif ($smtpServer) {\n\t\t\t\t\t\t\t\tmy $smtp = Net::SMTP->new($smtpServer) or die \"Cannot create SMTP connection to $smtpServer: $?\";\n\t\t\t\t\t\t\t\t$smtp->mail($smtpFrom);\n\t\t\t\t\t\t\t\t$smtp->recipient(@smtpRecipients, { SkipBad => 1});\n\t\t\t\t\t\t\t\t$smtp->data();\n\t\t\t\t\t\t\t\tmy $subjectTags;\n\t\t\t\t\t\t\t\tfor $key (keys %matches) {\n\t\t\t\t\t\t\t\t\tmy $tempDesc = getRegexDesc($matches{$key}[0]);\n\t if (length($tempDesc) > 0) {\n\t $subjectTags = $subjectTags . '(' . getRegexDesc($matches{$key}[0]) . ') ';\n\t }\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\tmy $smtpBody = \"To: $smtpRecipient\\nSubject: $smtpSubject $subjectTags\\n\\n\";\n\t\t\t\t\t\t\t\tfor $key (keys %matches) {\n\t\t\t\t\t\t\t\t\t$smtpBody = $smtpBody . \"Matched: \" . $matches{$key}[0] . \" (\" . $matches{$key}[1] . \" time(s))\\n\";\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t$smtpBody = $smtpBody . \"\\nSource: \" . $pastie . \"\\n\\n\" . $content;\n\t\t\t\t\t\t\t\t$smtp->datasend($smtpBody);\n\t\t\t\t\t\t\t\t$smtp->dataend();\n\t\t\t\t\t\t\t\t$smtp->quit();\n\t\t\t\t\t\t\t}\n\t\n\t\t\t\t\t\t\t# Save pastie content in the dump directory (if configured)\n\t\t\t\t\t\t\tif ($dumpDir) {\n\t\t\t\t\t\t\t\tmy $tempPastie = getPastieID($pastie);\n\t\t\t\t\t\t\t\tmy $tempDir = validateDumpDir($webSite, $dumpDir); # Generate and create dump directory\n\t\t\t\t\t\t\t\t(-d $tempDir) or die \"Cannot validate directory $dumpDir: $!\";\n\t\t\t\t\t\t\t\topen(DUMP, \">:encoding(UTF-8)\", \"$tempDir/$tempPastie.raw\") or die \"Cannot write to $tempDir/$tempPastie.raw : $!\";\n\t\t\t\t\t\t\t\tfor $key (keys %matches) {\n\t\t\t\t\t\t\t\t\tprint DUMP \"Matched: \" . $matches{$key}[0] . \" (\" . $matches{$key}[1] . \" time(s))\\n\";\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\tprint DUMP \"\\n$content\";\n\t\t\t\t\t\t\t\tclose(DUMP);\n\t\t\t\t\t\t\t\tif ($compressDump) { # Compress pastie\n\t\t\t\t\t\t\t\t\tmy $in = \"$tempDir/$tempPastie.raw\";\n\t\t\t\t\t\t\t\t\tmy $out = \"$tempDir/$tempPastie.gz\";\n\t\t\t\t\t\t\t\t\tuse IO::Compress::Gzip qw(gzip);\n\t\t\t\t\t\t\t\t\tif (gzip $in => $out) {\n\t\t\t\t\t\t\t\t\t\tunlink(\"$tempDir/$tempPastie.raw\");\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\telse {\n\t\t\t\t\t\t\t\t\t\tsyslogOutput(\"Cannot compress $tempDir/$tempPastie.raw: $!\");\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\telsif ($dumpAll && $dumpDir) {\n\t\t\t\t\t\t# Mirroring mode - dump the pastie in all cases\n\t\t\t\t\t\tmy $tempPastie = getPastieID($pastie);\n\t\t\t\t\t\tmy $tempDir = validateDumpDir($webSite, $dumpDir);\n\t\t\t\t\t\t(-d $tempDir) or die \"Cannot validate directory $tempDir: $!\";\n\t\t\t\t\t\topen(DUMP, \">:encoding(UTF-8)\", \"$tempDir/$tempPastie.raw\") or die \"Cannot write to $tempDir/$tempPastie.raw : $!\";\n\t\t\t\t\t\tprint DUMP \"\\n$content\";\n\t\t\t\t\t\tclose(DUMP);\n\t\t\t\t\t\tif ($compressDump) { # Compress pastie\n\t\t\t\t\t\t\tmy $in = \"$tempDir/$tempPastie.raw\";\n\t\t\t\t\t\t\tmy $out = \"$tempDir/$tempPastie.gz\";\n\t\t\t\t\t\t\tuse IO::Compress::Gzip qw(gzip);\n\t\t\t\t\t\t\tif (gzip $in => $out) {\n\t\t\t\t\t\t\t\tunlink(\"$tempDir/$tempPastie.raw\");\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\telse { \n\t\t\t\t\t\t\t\tsyslogOutput(\"Cannot compress $tempDir/$tempPastie.raw: $!\");\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\n\t\t\t\t\t# Flag this pastie as \"seen\"\n\t\t\t\t\tpush(@seenPasties, $pastie);\n\t\n\t\t\t\t\t# Save pastie data in SQLite\n\t\t\t\t\tif ($dbFile) {\n\t\t\t\t\t\tdbSavePastie($pastie, $md5);\n\t\t\t\t\t}\n\n\t\t\t\t\t# Wait a random number of seconds to not mess with pastebin.com webmasters\n\t\t\t\t\tsleep(int(rand(5)));\n\t\t\t\t}\n\t\t\t\telse { # MD5 Exists in DB\n\t\t\t\t\t($debug) && print \"DEBUG: MD5 $md5 already found in DB!\\n\";\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n}\n\n#\n# Search for interesting data in URLs found inside the pastie\n#\nsub processUrls {\n\tmy $pastie = shift || return 0;\n\twhile ($pastie =~ m,(http.*?://([^\\s)\\\"](?!ttp:))+),g) { # \"\n\t\tmy $url = $&;\n\t\tif ($url =~ /$followMatching/gi) { #Process only URLs matching our regex!\n \t($debug) && print \"+++ Following URL: $url\\n\";\n\t\t\tmy $ua = LWP::UserAgent->new;\n\t\t\t$ua->agent(getRandomUA());\n\t\t\tmy $r = $ua->head(\"$url\");\n\t\t\tif ($r->is_success && substr($r->header('Content-Type'), 0, 5) eq \"text/\") {\t# Only process \"text\"\n\t\t\t\tanalyzePastie($url);\n\t\t\t}\n \t}\n\t\t# Protect us against pastebin.com blacklist?\n\t\t#sleep(int(rand(15)));\n\t}\n\treturn 0;\n}\n\n# \n# parseXMLConfigFile\n# Load the configuration from provided XML file\n# Args:\n# $configFile = Main pastemon.conf XML file\n#\nsub parseXMLConfigFile {\n\tmy $configFile = shift;\n\t(-r $configFile) || die \"Cannot load XML file $configFile: $!\";\n\n\t($debug) && print STDERR \"+++ Loading XML file $configFile.\\n\";\n\tmy $xml = XML::XPath->new(filename => \"$configFile\");\n\tmy $buff;\n\n\t# Reset settings\n\tundef $pidFile;\n\tundef $sampleSize;\n\tundef $dumpDir;\n\tundef $dumpAll;\n\tundef $compressDump;\n\tundef $proxyFile;\n\tundef $uaFile;\n\tundef $cefDestination;\n\tundef $cefPort;\n\tundef $cefSeverity;\n\tundef $smtpServer;\n\tundef $smtpFrom;\n\tundef $smtpRecipient;\n\tundef $smtpSubject;\n\tundef $wpSite;\n\tundef $wpUser;\n\tundef $wpPass;\n\tundef $wpCategory;\n\tundef $distanceMin;\n\tundef $distanceMaxSize;\n\tundef $checkPastebin;\n\tundef $checkPastie;\n\tundef $checkNopaste;\n undef $checkPastesite;\n\tundef $followUrls;\n\tundef $followMatching;\n\tundef $dbFile;\n\n\t# Core Parameters\n\tmy $nodes = $xml->find('/pastemon/core');\n\tforeach my $node ($nodes->get_nodelist) {\n\t\t$buff\t\t\t= $node->find('ignore-case')->string_value;\n\t\tif (lc($buff) eq \"yes\" || $buff eq \"1\") {\n\t\t\t$ignoreCase++;\n\t\t\t($debug) && print STDERR \"+++ Non-sensitive search enabled.\\n\";\n\t\t}\n\t\t$buff\t\t\t= $node->find('dump-all')->string_value;\n\t\tif (lc($buff) eq \"yes\" || $buff eq \"1\") {\n\t\t\t$dumpAll++;\n\t\t\t($debug) && print STDERR \"+++ Dumping all pasties (mirror mode).\\n\";\n\t\t}\n\t\t$buff\t\t\t= $node->find('compress-pasties')->string_value;\n\t\tif (lc($buff) eq \"yes\" || $buff eq \"1\") {\n\t\t\t$compressDump++;\n\t\t\t($debug) && print STDERR \"+++ Compressing all pasties (mirror mode).\\n\";\n\t\t}\n\t\t$pidFile\t\t= $node->find('pid-file')->string_value;\n\t\t$regexFile\t\t= $node->find('regex-file')->string_value;\n\t\t$sampleSize\t\t= $node->find('sample-size')->string_value;\n\t\t$dumpDir\t\t= $node->find('dump-directory')->string_value;\n\t\t$proxyFile\t\t= $node->find('proxy-config')->string_value;\n\t\t$uaFile\t\t\t= $node->find('ua-config')->string_value;\n\t\t$httpTimeout\t\t= $node->find('http-timeout')->string_value;\n\t\t$distanceMin\t\t= $node->find('distance-min')->string_value;\n\t\t$distanceMaxSize\t= $node->find('distance-max-size')->string_value;\n\t}\n\n\t# Monitored websites\n\tmy $nodes = $xml->find('/pastemon/websites');\n\tforeach my $node ($nodes->get_nodelist) {\n\t\t$buff\t\t\t= $node->find('pastebin')->string_value;\n\t\tif (lc($buff) eq \"yes\" || $buff eq \"1\") {\n\t\t\t$checkPastebin++;\n\t\t\t($debug) && print STDERR \"+++ pastebin.com monitoring activated.\\n\";\n\t\t}\n\t\t$buff = $node->find('pastie')->string_value;\n\t\tif (lc($buff) eq \"yes\" || $buff eq \"1\") {\n\t\t\t$checkPastie++;\n\t\t\t($debug) && print STDERR \"+++ pastie.com monitoring activated.\\n\";\n\t\t}\n\t\t$buff = $node->find('nopaste')->string_value;\n\t\tif (lc($buff) eq \"yes\" || $buff eq \"1\") {\n\t\t\t$checkNopaste++;\n\t\t\t($debug) && print STDERR \"+++ nopaste.me monitoring activated.\\n\";\n\t\t}\n\t\t$buff\t\t\t= $node->find('pastesite')->string_value;\n\t\tif (lc($buff) eq \"yes\" || $buff eq \"1\") {\n\t\t\t$checkPastesite++;\n\t\t\t($debug) && print STDERR \"+++ pastesite.com monitoring activated.\\n\";\n\t\t}\n\t\t$delayPastebin\t= $node->find('pastebin-delay')->string_value;\n\t\t$delayPastie\t= $node->find('pastie-delay')->string_value;\n\t\t$delayNopaste\t= $node->find('nopaste-delay')->string_value;\n\t\t$delayPastesite\t= $node->find('pastesite-delay')->string_value;\n\t}\n\n\t# Follow URLs\n\tmy $nodes = $xml->find('/pastemon/urls');\n\tforeach my $node ($nodes->get_nodelist) {\n\t\t$buff\t\t\t= $node->find('follow')->string_value;\n\t\tif (lc($buff) eq \"yes\" || $buff eq \"1\") {\n\t\t\t$followUrls++;\n\t\t\t($debug) && print STDERR \"+++ Follow URLs feature activated.\\n\";\n\t\t}\n\t\t$followMatching\t\t= $node->find('matching')->string_value;\n\t}\n\n\t# CEF Parameters\n\tmy $nodes = $xml->find('/pastemon/cef-output');\n\tforeach my $node ($nodes->get_nodelist) {\n\t\t$cefDestination\t\t= $node->find('destination')->string_value;\n\t\t$cefPort\t\t= $node->find('port')->string_value;\n\t\t$cefSeverity\t\t= $node->find('severity')->string_value;\n\t}\n\n\t# Syslog Parameters\n\tmy $nodes = $xml->find('/pastemon/syslog-output');\n\tforeach my $node ($nodes->get_nodelist) {\n\t\t$syslogFacility\t\t= $node->find('facility')->string_value;\n\t}\n\n\t# Wordpress Parameters\n\tmy $nodes = $xml->find('/pastemon/wordpress-output');\n\tforeach my $node ($nodes->get_nodelist) {\n\t\t$wpSite\t\t\t= $node->find('site')->string_value;\n\t\t$wpUser\t\t\t= $node->find('user')->string_value;\n\t\t$wpPass\t\t\t= $node->find('password')->string_value;\n\t\t$wpCategory\t\t= $node->find('category')->string_value;\n\t}\n\n\t# SMTP Parameters\n\tmy $nodes = $xml->find('/pastemon/smtp-output');\n\tforeach my $node ($nodes->get_nodelist) {\n\t\t$smtpServer\t\t= $node->find('smtp-server')->string_value;\n\t\t$smtpFrom\t\t= $node->find('from')->string_value;\n\t\t$smtpRecipient\t\t= $node->find('recipient')->string_value;\n\t\t$smtpSubject\t\t= $node->find('subject')->string_value;\n\t}\n\n\t# SQLite3 Parameters\n\tmy $nodes = $xml->find('/pastemon/db-output');\n\tforeach my $node ($nodes->get_nodelist) {\n\t\t$dbFile\t\t\t= $node->find('db-file')->string_value;\n\t}\n\n\t# ---------------------\n\t# Parameters validation\n\t# ---------------------\n\n\t# Check if the provided dump directory is writable to us\n\tif ($dumpDir) {\n\t\t# (-w $dumpDir) or die \"Directory $dumpDir is not writable: $!\";\n\t\tsyslogOutput(\"Using $dumpDir as dump directory\");\n\t}\n\n\t# Compress dumped pasties?\n\tif ($compressDump) {\n\t\tif ($haveIOCompressGzip) { # Module IO::Compress::Gzip installed?\n\t\t\tif (!$dumpDir) {\n\t\t\t\tsyslogOutput(\"Option compress-pasties disabled: No dump directory defined\");\n\t\t\t\tundef $compressDump\n\t\t\t}\n\t\t\tif (!$haveIOUncompressGunzip) { # Module IO::Compress::Gunzp installed?\n\t\t\t\tsyslogOutput(\"Option compress-pasties disabled: IO::Uncompress:Gunzip not installed\");\n\t\t\t\tundef $compressDump;\n\t\t\t}\n\t\t}\n\t\telse {\n\t\t\tsyslogOutput(\"Option compress-pasties disabled: IO::Compress:Gzip not installed\");\n\t\t\tundef $compressDump;\n\t\t}\n\t}\n\n\t# Dumping all pasties requires a dump directory\n\tif ($dumpAll && !$dumpDir) {\n\t\tsyslogOutput(\"No dump directory specified\");\n\t}\n\n\t# Verifiy sampleSize format if specified\n\tif ($sampleSize) {\n\t\tdie \"Sample buffer length must be an integer!\" if not $sampleSize =~ /\\d+/;\n\t\tsyslogOutput(\"Dumping $sampleSize bytes samples\");\n\t}\n\n\t# Verify the HTTP timeout if specified\n\tif ($httpTimeout) {\n\t\tdie \"HTTP timeout must be an integer!\" if not $httpTimeout =~ /\\d+/;\n\t\tsyslogOutput(\"HTTP timeout: $httpTimeout seconds\");\n\t}\n\n\t# Verify Wordpress config\n\tif ($wpSite) {\n\t\tif ($haveWordPressXMLRMC) { # Module WordPress::XMLRPC installed?\n\t\t\t(!$wpSite || !$wpUser || !$wpPass || !$wpCategory) && die \"Incomplete Wordpress configuration\";\n\t\t\t($sampleSize) || die \"A sample buffer length must be given with Wordpress output\";\n\t\t\tsyslogOutput(\"Dumping data to $wpSite/xmlrpc.php\");\n\t\t} \n\t\telse {\n\t\t\tsyslogOutput(\"Wordpress configuration disabled: Wordpress::XMLRPC not installed\");\n\t\t\tundef $wpSite;\n\t\t}\n\t}\n\n\t# Verify SMTP config\n\tif ($smtpServer) {\n\t\t(!$smtpServer || !$smtpFrom || !$smtpRecipient || !$smtpSubject) && die \"Incomplete SMTP configuration\";\n\t\tmy $smtp = Net::SMTP->new($smtpServer) or die \"Cannot use SMTP server $smtpServer: $?\";\n\t\t$smtp->quit();\n\t\t@smtpRecipients = split(/[, ]+/, $smtpRecipient);\n\t\tsyslogOutput(\"Sending SMTP notifications to <\".$smtpRecipient.\">\");\n\t}\n\n\t# Load proxies\n\tif ($proxyFile) {\n\t\t(-r $proxyFile) or die \"Cannot read proxy configuration file $proxyFile: $!\";\n\t\tloadProxyFromFile($proxyFile) || die \"Cannot load proxies from file $proxyFile\";\n\t}\n\n\t# Distance\n\tif ($distanceMin) {\n\t\tif ($haveTextJaroWinkler) { # Module Text::JaroWinkler installed?\n\t\t\t(!$dumpDir) && die \"A dump directory must be configured to use the distance check\";\n\t\t\t($distanceMin > 0 && $distanceMin < 1) or die \"Minimum distance must be between 0 and 1\";\n\t\t\tif ($distanceMaxSize) {\n\t\t\t\tdie \"Distance max size must be an integer!\" if not $distanceMaxSize =~ /\\d+/;\n\t\t\t\tsyslogOutput(\"Enabled duplicate detection with distance of $distanceMin (size limit: $distanceMaxSize bytes)\");\n\t\t\t} else {\n\t\t\t\tsyslogOutput(\"Enabled duplicate detection with distance of $distanceMin\");\n\t\t\t}\n\t\t}\n\t\telse {\n\t\t\tsyslogOutput(\"Distance configuration disabled: Text::JaroWinkler not installed\");\n\t\t\tundef $distanceMin;\n\t\t}\n\t}\n\n\t# SQLite3 Output\n\tif ($dbFile) {\n\t\tif ($haveDBI) { # Module DBI installed?\n\t\t\t# Do we have to initialize the DB (first execution)\n\t\t\tmy $dbh = DBI->connect(\"dbi:SQLite:dbname=\" . $dbFile)\n\t\t\t\tor die \"Cannot connect to the SQLite DB \" . $dbFile . \"\\n\";\n\t\t\tmy $sth = $dbh->prepare(\"SELECT name FROM sqlite_master WHERE type='table' AND name='pasties'\");\n\t\t\t$sth->execute();\n\t\t\tmy $data = $sth->fetch();\n\t\t\tif (!$data) {\t# Tables 'pasties' does not exists. Create it.\n\t\t\t\t$sth = $dbh->prepare(\"CREATE TABLE pasties (id VARCHAR(50),\n\t\t\t\t\t\t\t\t\t\ttimestamp DATETIME,\n\t\t\t\t\t\t\t\t\t\turl VARCHAR(128),\n\t\t\t\t\t\t\t\t\t\tmatched VARCHAR(256),\n\t\t\t\t\t\t\t\t\t\tpath VARCHAR(256),\n\t\t\t\t\t\t\t\t\t\tmd5 VARCHAR(32) PRIMARY KEY,\n\t\t\t\t\t\t\t\t\t\ttype INTEGER)\");\n\t\t\t\t$sth->execute() or die \"Cannot create table 'pasties'\";\n\t\t\t\t$sth = $dbh->prepare(\"CREATE UNIQUE INDEX pasties_idx ON pasties(id)\");\n\t\t\t\t$sth->execute() or die \"Cannot create index 'pasties_idx'\";\n\t\t\t\t($debug) && print STDERR \"+++ Created database \" . $dbFile . \"\\n\";\n\t\t\t}\n\t\t\t$dbh->disconnect();\n\t\t}\n\t\telse {\n\t\t\tsyslogOutput(\"DB support disabled: DBI not installed\");\n\t\t\tundef $dbFile;\n\t\t}\n\t}\n\n\t# Follow URL\n\tif ($followUrls && !$followMatching) {\n\t\tsyslogOutput(\"Warning: No regex defined to match URLs\");\n\t\t$followMatching = \".*\";\t# Match everything\n\t}\n\n\treturn;\n}\n\n#\n# Download the latest pasties and load them in a Perl array\n# (http://pastebin.com/archive)\n#\nsub fetchLastPasties {\n\tmy $webSite = shift;\n\tmy $tempProxy;\n\tmy $ua = LWP::UserAgent->new;\n\t$ua->timeout($httpTimeout);\n\tif (@proxies) {\n\t\t$tempProxy = selectRandomProxy();\n\t\t$ua->proxy('http', $tempProxy);\n\t}\n\telse {\n\t\t($ENV{'HTTP_PROXY'}) && $ua->env_proxy;\n\t}\n\t$ua->agent(getRandomUA());\n\n\tundef @pasties;\t# Reset the array first!\n\n\t# www.pastebin.com\n\tif ($webSite == PASTEBIN) {\n\t\t($debug) && print STDERR \"Loading new pasties from pastebin.com.\\n\";\n\t\tmy $response = $ua->get(\"http://pastebin.com/archive\");\n\t\tif ($response->is_success) {\n\t\t\t# Load the pasties into an array\n\t\t\t# @pasties = $response->decoded_content =~ /.+<\\/a><\\/td>/g;\n\t\t\t# New format (2012/02/19):\n\t\t\tmy @tempPasties = $response->decoded_content =~ /.+<\\/a><\\/td>/g;\n\t\t\t# Append the complete URL\n\t\t\tforeach my $p (@tempPasties) {\n\t\t\t\t$p = 'http://pastebin.com/raw.php?i=' . $p;\n\t\t\t}\n\t\t\tpush(@pasties, @tempPasties);\n\t\t}\n\t\telse {\n\t\t\tsyslogOutput(\"Cannot fetch www.pastebin.com: \" . $response->status_line);\n\t\t\t# If cannot fetch pastie and we use proxies, disable the current one!\n\t\t\t(@proxies) && disableProxy($tempProxy);\n\t\t\treturn 1;\n\t\t}\n\t}\n\telsif ($webSite == PASTIE) {\n\t\t#($debug) && print STDERR \"Loading new pasties from pastie.org.\\n\";\n\t\tmy $response = $ua->get(\"http://pastie.org/pastes\");\n\t\tif ($response->is_success) {\n\t\t\tmy @tempPasties = $response->decoded_content =~ //g;\n\t\t\t# Append the complete URL\n\t\t\tforeach my $p (@tempPasties) {\n\t\t\t\t$p = $p . '/download';\n\t\t\t}\n\t\t\tpush(@pasties, @tempPasties);\n\t\t}\n\t\telse {\n\t\t\tsyslogOutput(\"Cannot fetch www.pastie.org: \" . $response->status_line);\n\t\t\t# If cannot fetch pastie and we use proxies, disable the current one!\n\t\t\t(@proxies) && disableProxy($tempProxy);\n\t\t\treturn 1;\n\t\t}\n\t}\n\telsif ($webSite == NOPASTE) {\n\t\t#($debug) && print STDERR \"Loading new pasties from nopaste.me.\\n\";\n\t\tmy $response = $ua->get(\"http://nopaste.me/recent\");\n\t\tif ($response->is_success) {\n\t\t\tmy @tempPasties = $response->decoded_content =~ //ig;\n\t\t\t# Append the complete URL\n\t\t\tforeach my $p (@tempPasties) {\n\t\t\t\t$p = 'http://nopaste.me/raw/' . $p . '.txt';\n\t\t\t}\n\t\t\tpush(@pasties, @tempPasties);\n\t\t}\n\t\telse {\n\t\t\tsyslogOutput(\"Cannot fetch nopaste.me: \" . $response->status_line);\n\t\t\t# If cannot fetch pastie and we use proxies, disable the current one!\n\t\t\t(@proxies) && disableProxy($tempProxy);\n\t\t\treturn 1;\n\t\t}\n\t}\n\telsif ($webSite == PASTESITE) {\n\t\t($debug) && print STDERR \"Loading new pasties from pastesite.com.\\n\";\n\t\tmy $response = $ua->get(\"http://pastesite.com/recent\");\n\t\tif ($response->is_success) {\n\t\t\tmy @tempPasties = $response->decoded_content =~ /status_line);\n\t\t\t# If cannot fetch pastie and we use proxies, disable the current one!\n\t\t\t(@proxies) && disableProxy($tempProxy);\n\t\t\treturn 1;\n\t\t}\n\t}\n \telse {\n\t\tdie \"Unknown website constant: $webSite\";\n\t}\n\n\t# DEBUG\n\t#foreach my $p (@pasties) {\n\t#\tprint \"DEBUG: $p\\n\";\n\t#}\n\treturn 0;\n}\n\n#\n# Fetch the raw content of a pastie and return its content\n#\nsub fetchPastie {\n\tmy $tempProxy;\n\tmy $pastie = shift;\n\tmy $ua = LWP::UserAgent->new;\n\t$ua->timeout($httpTimeout);\n\tif (@proxies) {\n\t\t$tempProxy = selectRandomProxy();\n\t\t$ua->proxy('http', $tempProxy);\n\t}\n\telse {\n\t\t($ENV{'HTTP_PROXY'}) && $ua->env_proxy;\n\t}\n\t$ua->agent(getRandomUA());\n\tmy $response = $ua->get(\"$pastie\");\n\tif ($response->is_success) {\n\t\t# Hack for pastesite.com: Extract data from the \n\t\t# (To bypass the button)\n\t\tif ($pastie =~ /http:\\/\\/pastesite.com/) {\n\t\t\tif ($response->decoded_content =~ /\\