Repository: robdmc/crontabs Branch: master Commit: 8eec68fea4a3 Files: 18 Total size: 46.1 KB Directory structure: gitextract_euyj14f6/ ├── CONTRIBUTORS.txt ├── LICENSE ├── MANIFEST.in ├── README.md ├── crontabs/ │ ├── __init__.py │ ├── crontabs.py │ ├── processes.py │ ├── tests/ │ │ ├── __init__.py │ │ └── test_all.py │ └── version.py ├── docs/ │ ├── Makefile │ ├── conf.py │ ├── index.rst │ ├── ref/ │ │ └── crontabs.rst │ └── toc.rst ├── publish.py ├── setup.cfg └── setup.py ================================================ FILE CONTENTS ================================================ ================================================ FILE: CONTRIBUTORS.txt ================================================ Rob deCarvalho (unlisted@unlisted.net) ================================================ FILE: LICENSE ================================================ The MIT License (MIT) Copyright (c) 2017 Rob deCarvalho Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================ FILE: MANIFEST.in ================================================ include README.md include CONTRIBUTORS.txt include LICENSE prune */tests ================================================ FILE: README.md ================================================ # Crontabs --- **NOTE:** I've recently discovered the [Rocketry](https://github.com/Miksus/rocketry/) and [Huey](https://github.com/coleifer/huey) projects, which you should probably use instead of crontabs. They are just better than crontabs. --- Think of crontabs as a quick-and-dirty solution you can throw into one-off python scripts to execute tasks on a cron-like schedule. Crontabs is a small pure-python library that was inspired by the excellent [schedule](https://github.com/dbader/schedule) library for python. In addition to having a slightly different API, crontabs differs from the schedule module in the following ways. * You do not need to provide your own event loop. * Job timing is guaranteed not to drift over time. For example, if you specify to run a job every five minutes, you can rest assured that it will always run at 5, 10, 15, etc. passed the hour with no drift. * The python functions are all run in child processes. A memory-friendly flag is available to run each iteration of your task in its own process thereby mitigating memory problems due to Python's [high watermark issue](https://hbfs.wordpress.com/2013/01/08/python-memory-management-part-ii/) # Why Crontabs Python has no shortage of [cron-like job scheduling libraries](https://pypi.python.org/pypi?%3Aaction=search&term=cron), so why create yet another. The honest answer is that I couldn't find one that met a simple list of criteria. * **Simple installation with no configuration.** An extremely robust and scalable solution to this problem already exists. [Celery](http://www.celeryproject.org/). But for quick and dirty work, I didn't want the hastle of setting up and configuring a broker, which celery requires to do its magic. For simple jobs, I just wanted to pip install and go. * **Human readable interface.** I loved the interface provided by the [schedule](https://github.com/dbader/schedule) library and wanted something similarly intuitive to use. * **Memory safe for long running jobs.** Celery workers can suffer from severe memory bloat due to the way Python manages memory. As of 2017, the recommended solution for this was to periodically restart the workers. Crontabs runs each job in a subprocess. It can optionally also run each iteration of a task in it's own process thereby mitigating the memory bloat issue. * **Simple solution for cron-style workflow and nothing more.** I was only interested in supporting cron-like functionality, and wasn't interested in all the other capabilities and guarantees offered by a real task-queue solution like celery. * **Suggestions for improvement welcome.** If you encounter a bug or have an improvement that remains within the scope listed above, please feel free to open an issue (or even better... a PR). # Installation ```bash pip install crontabs ``` # Usage ### Schedule a single job ```python from crontabs import Cron, Tab from datetime import datetime def my_job(*args, **kwargs): print('args={} kwargs={} running at {}'.format(args, kwargs, datetime.now())) # Will run with a 5 second interval synced to the top of the minute Cron().schedule( Tab(name='run_my_job').every(seconds=5).run(my_job, 'my_arg', my_kwarg='hello') ).go() ``` ### Schedule multiple jobs ```python from crontabs import Cron, Tab from datetime import datetime def my_job(*args, **kwargs): print('args={} kwargs={} running at {}'.format(args, kwargs, datetime.now())) # All logging messages are sent to sdtout Cron().schedule( # Turn off logging for job that runs every five seconds Tab(name='my_fast_job', verbose=False).every(seconds=5).run(my_job, 'fast', seconds=5), # Go ahead and let this job emit logging messages Tab(name='my_slow_job').every(seconds=20).run(my_job, 'slow', seconds=20), ).go() ``` ### Schedule future job to run repeatedly for a fixed amount of time ```python from crontabs import Cron, Tab from datetime import datetime def my_job(*args, **kwargs): print('args={} kwargs={} running at {}'.format(args, kwargs, datetime.now())) Cron().schedule( Tab( name='future_job' ).every( seconds=5 ).starting( '12/27/2017 16:45' # This argument can either be parsable text or datetime object. ).run( my_job, 'fast', seconds=5 ) # max_seconds starts from the moment go is called. Pad for future run times accordingly. ).go(max_seconds=60) ``` # Cron API The `Cron` class has a very small api | method | Description | | --- | --- | | `.schedule()` |[**Required**] Specify the different jobs you want using `Tab` instances| | `.go()` | [**Required**] Start the crontab manager to run all specified tasks| | `.get_logger()` | A class method you can use to get an instance of the crontab logger| # Tab API with examples The api for the `Tab` class is designed to be composable and readable in plain English. It supports the following "verbs" by invoking methods. | method | Description | | --- | --- | | `.run()` |[**Required**] Specify the function to run. | | `.every()` |[**Required**] Specify the interval between function calls.| | `.starting()` | [**Optional**] Specify an explicit time for the function calls to begin.| | `.lasting()` | [**Optional**] Specify how long the task will continue being iterated.| | `.until()` | [**Optional**] Specify an explicit time past which the iteration will stop | `.during()` | [**Optional**] Specify time conditions under which the function will run | `.excluding()` | [**Optional**] Specify time conditions under which the function will be inhibited ## Run a job indefinitely ```python from crontabs import Cron, Tab from datetime import datetime def my_job(name): print('Running function with name={}'.format(name)) Cron().schedule( Tab(name='forever').every(seconds=5).run(my_job, 'my_func'), ).go() ``` ## Run one job indefinitely, another for thirty seconds, and another until 1/1/2030 ```python from crontabs import Cron, Tab from datetime import datetime def my_job(name): print('Running function with name={}'.format(name)) Cron().schedule( Tab(name='forever').run(my_job, 'forever_job').every(seconds=5), Tab(name='for_thirty').run(my_job, 'mortal_job').every(seconds=5).lasting(seconds=30), Tab(name='real_long').run(my_job, 'long_job').every(seconds=5).until('1/1/2030'), ).go() ``` ## Run job every half hour from 9AM to 5PM excluding weekends ```python from crontabs import Cron, Tab from datetime import datetime def my_job(name): # Grab an instance of the crontab logger and write to it. logger = Cron.get_logger() logger.info('Running function with name={}'.format(name)) def business_hours(timestamp): return 9 <= timestamp.hour < 17 def weekends(timestamp): return timestamp.weekday() > 4 # Run a job every 30 minutes during weekdays. Stop crontabs after it has been running for a year. # This will indiscriminately kill every Tab it owns at that time. Cron().schedule( Tab( name='my_job' ).run( my_job, 'my_job' ).every( minutes=30 ).during( business_hours ).excluding( weekends ) ).go(max_seconds=3600 * 24 * 365) ``` # Run test suite with ```bash git clone git@github.com:robdmc/crontabs.git cd crontabs pip install -e .[dev] py.test -s -n 8 # Might need to change the -n amount to pass ``` ___ Projects by [robdmc](https://www.linkedin.com/in/robdecarvalho). * [Pandashells](https://github.com/robdmc/pandashells) Pandas at the bash command line * [Consecution](https://github.com/robdmc/consecution) Pipeline abstraction for Python * [Behold](https://github.com/robdmc/behold) Helping debug large Python projects * [Crontabs](https://github.com/robdmc/crontabs) Simple scheduling library for Python scripts * [Switchenv](https://github.com/robdmc/switchenv) Manager for bash environments * [Gistfinder](https://github.com/robdmc/gistfinder) Fuzzy-search your gists ================================================ FILE: crontabs/__init__.py ================================================ # flake8: noqa from .version import __version__ from .crontabs import Cron, Tab ================================================ FILE: crontabs/crontabs.py ================================================ """ Module for manageing crontabs interface """ import datetime import functools import time import traceback import warnings import daiquiri from dateutil.parser import parse from dateutil.relativedelta import relativedelta from fleming import fleming from .processes import ProcessMonitor import logging daiquiri.setup(level=logging.INFO) class Cron: @classmethod def get_logger(self, name='crontab_log'): logger = daiquiri.getLogger(name) return logger def __init__(self): """ A Cron object runs many "tabs" of asynchronous tasks. """ self.monitor = ProcessMonitor() self._tab_list = [] def schedule(self, *tabs): self._tab_list = list(tabs) return self def go(self, max_seconds=None): for tab in self._tab_list: target = tab._get_target() self.monitor.add_subprocess(tab._name, target, tab._robust, tab._until) try: self.monitor.loop(max_seconds=max_seconds) except KeyboardInterrupt: # pragma: no cover pass class Tab: _SILENCE_LOGGER = False def __init__(self, name, robust=True, verbose=True, memory_friendly=False): """ Schedules a Tab entry in the cron runner :param name: Every tab must have a string name :param robust: A robust tab will be restarted if an error occures A non robust tab will not be restarted, but all other non-errored tabs should continue running :param verbose: Set the verbosity of log messages. :memory friendly: If set to true, each iteration will be run in separate process """ if not isinstance(name, str): raise ValueError('Name argument must be a string') self._name = name self._robust = robust self._verbose = verbose self._starting = None self._every_kwargs = None self._func = None self._func_args = None self._func_kwargs = None self._exclude_func = self._default_exclude_func self._during_func = self._default_during_func self._memory_friendly = memory_friendly self._until = None self._lasting_delta = None def _default_exclude_func(self, t): return False def _default_during_func(self, t): return True def _log(self, msg): if self._verbose and not self._SILENCE_LOGGER: # pragma: no cover logger = daiquiri.getLogger(self._name) logger.info(msg) def _process_date(self, datetime_or_str): if isinstance(datetime_or_str, str): return parse(datetime_or_str) elif isinstance(datetime_or_str, datetime.datetime): return datetime_or_str else: raise ValueError('.starting() and until() method can only take strings or datetime objects') def starting(self, datetime_or_str): """ Set the starting time for the cron job. If not specified, the starting time will always be the beginning of the interval that is current when the cron is started. :param datetime_or_str: a datetime object or a string that dateutil.parser can understand :return: self """ self._starting = self._process_date(datetime_or_str) return self def starting_at(self, datetime_or_str): warnings.warn('.starting_at() is depricated. Use .starting() instead') return self.starting(datetime_or_str) def until(self, datetime_or_str): """ Run the tab until the specified time is reached. At that point, deactivate the expired tab so that it no longer runs. :param datetime_or_str: a datetime object or a string that dateutil.parser can understand :return: self """ self._until = self._process_date(datetime_or_str) return self def lasting(self, **kwargs): """ Run the tab so that it lasts this long. The argument structure is exactly the same as that of the .every() method """ relative_delta_kwargs = {k if k.endswith('s') else k + 's': v for (k, v) in kwargs.items()} self._lasting_delta = relativedelta(**relative_delta_kwargs) return self def excluding(self, func, name=''): """ Pass a function that takes a timestamp for when the function should execute. It inhibits running when the function returns True. Optionally, add a name to the exclusion. This name will act as an explanation in the log for why the exclusion was made. """ self._exclude_func = func self._exclude_name = name return self def during(self, func, name=''): """ Pass a function that takes a timestamp for when the function should execute. It will only run if the function returns true. Optionally, add a name. This name will act as an explanation in the log for why any exclusions were made outside the "during" specification. """ self._during_func = func self._during_name = name return self def every(self, **kwargs): """ Specify the interval at which you want the job run. Takes exactly one keyword argument. That argument must be one named one of [second, minute, hour, day, week, month, year] or their plural equivalents. :param kwargs: Exactly one keyword argument :return: self """ if len(kwargs) != 1: raise ValueError('.every() method must be called with exactly one keyword argument') self._every_kwargs = self._clean_kwargs(kwargs) return self def run(self, func, *func_args, **func__kwargs): """ Specify the function to run at the scheduled times :param func: a callable :param func_args: the args to the callable :param func__kwargs: the kwargs to the callable :return: """ self._func = func self._func_args = func_args self._func_kwargs = func__kwargs return self def _clean_kwargs(self, kwargs): allowed_key_map = { 'seconds': 'second', 'second': 'second', 'minutes': 'minute', 'minute': 'minute', 'hours': 'hour', 'hour': 'hour', 'days': 'day', 'day': 'day', 'weeks': 'week', 'week': 'week', 'months': 'month', 'month': 'month', 'years': 'year', 'year': 'year', } kwargs = {k if k.endswith('s') else k + 's': v for (k, v) in kwargs.items()} out_kwargs = {} for key in kwargs.keys(): out_key = allowed_key_map.get(key.lower()) if out_key is None: raise ValueError('Allowed time names are {}'.format(sorted(allowed_key_map.keys()))) out_kwargs[out_key] = kwargs[key] return out_kwargs def _is_uninhibited(self, time_stamp): can_run = True msg = 'inhibited: ' if self._exclude_func(time_stamp): if self._exclude_name: msg += self._exclude_name can_run = False if can_run and not self._during_func(time_stamp): if self._during_name: msg += self._during_name can_run = False if not can_run: self._log(msg) return can_run def _loop(self, max_iter=None): if not self._SILENCE_LOGGER: # pragma: no cover don't want to clutter tests logger = daiquiri.getLogger(self._name) logger.info('Starting {}'.format(self._name)) # fleming and dateutil have arguments that just differ by ending in an "s" fleming_kwargs = self._every_kwargs relative_delta_kwargs = {} # build the relative delta kwargs for k, v in self._every_kwargs.items(): relative_delta_kwargs[k + 's'] = v # Previous time is the latest interval boundary that has already happened previous_time = fleming.floor(datetime.datetime.now(), **fleming_kwargs) # keep track of iterations n_iter = 0 # this is the infinite loop that runs the cron. It will only be stopped when the # process is killed by its monitor. while True: n_iter += 1 if max_iter is not None and n_iter > max_iter: break # everything is run in a try block so errors can be explicitly handled try: # push forward the previous/next times next_time = previous_time + relativedelta(**relative_delta_kwargs) previous_time = next_time # get the current time now = datetime.datetime.now() # if our job ran longer than an interval, we will need to catch up if next_time < now: continue # sleep until the computed time to run the function sleep_seconds = (next_time - now).total_seconds() time.sleep(sleep_seconds) # See what time it is on wakeup timestamp = datetime.datetime.now() # If passed until date, break out of here if self._until is not None and timestamp > self._until: break # If not inhibited, run the function if self._is_uninhibited(timestamp): self._log('Running {}'.format(self._name)) self._func(*self._func_args, **self._func_kwargs) except KeyboardInterrupt: # pragma: no cover pass except: # noqa # only raise the error if not in robust mode. if self._robust: s = 'Error in tab\n' + traceback.format_exc() logger = daiquiri.getLogger(self._name) logger.error(s) else: raise self._log('Finishing {}'.format(self._name)) def _get_target(self): """ returns a callable with no arguments designed to be the target of a Subprocess """ if None in [self._func, self._func_kwargs, self._func_kwargs, self._every_kwargs]: raise ValueError('You must call the .every() and .run() methods on every tab.') if self._memory_friendly: # pragma: no cover TODO: need to find a way to test this target = functools.partial(self._loop, max_iter=1) else: # pragma: no cover TODO: need to find a way to test this target = self._loop if self._lasting_delta is not None: self._until = datetime.datetime.now() + self._lasting_delta return target ================================================ FILE: crontabs/processes.py ================================================ import traceback import daiquiri try: # pragma: no cover from Queue import Empty except: # noqa pragma: no cover from queue import Empty from multiprocessing import Process, Queue import datetime import sys class SubProcess: def __init__( self, name, target, q_stdout, q_stderr, q_error, robust, until=None, args=None, kwargs=None, ): # set up the io queues self.q_stdout = q_stdout self.q_stderr = q_stderr self.q_error = q_error self._robust = robust self._until = until # Setup the name of the sub process self._name = name # Save the target of the process self._target = target # Save the args to the process self._args = args or set() # Setup a reference to the process self._process = None # Save the kwargs to the process self._kwargs = kwargs or {} self._has_logged_expiration = False @property def expired(self): expired = False if self._until is not None and self._until < datetime.datetime.now(): expired = True if not self._has_logged_expiration: self._has_logged_expiration = True logger = daiquiri.getLogger(self._name) logger.info('Process expired and will no longer run') return expired def is_alive(self): return self._process is not None and self._process.is_alive() def start(self): self._process = Process( target=wrapped_target, args=[ self._target, self.q_stdout, self.q_stderr, self.q_error, self._robust, self._name ] + list(self._args), kwargs=self._kwargs ) self._process.daemon = True self._process.start() class IOQueue: # pragma: no cover """ Okay, so here is something annoying. If you spawn a python subprocess, you cannot pipe stdout/stderr in the same way you can with the parent process. People who run this library probably want to be able to redirect output to logs. The best way I could figure out to handle this was to monkey patch stdout and stderr in the subprocesses to be an instance of this class. All this does is send write() messages to a queue that is monitored by the parent process and prints to parent stdtou/stderr """ def __init__(self, q): self._q = q def write(self, item): self._q.put(item) def flush(self): pass def wrapped_target(target, q_stdout, q_stderr, q_error, robust, name, *args, **kwargs): # pragma: no cover """ Wraps a target with queues replacing stdout and stderr """ import sys sys.stdout = IOQueue(q_stdout) sys.stderr = IOQueue(q_stderr) try: target(*args, **kwargs) except: # noqa if not robust: s = 'Error in tab\n' + traceback.format_exc() logger = daiquiri.getLogger(name) logger.error(s) else: raise if not robust: q_error.put(name) raise class ProcessMonitor: TIMEOUT_SECONDS = .05 def __init__(self): self._subprocesses = [] self._is_running = False self.q_stdout = Queue() self.q_stderr = Queue() self.q_error = Queue() def add_subprocess(self, name, func, robust, until, *args, **kwargs): sub = SubProcess( name, target=func, q_stdout=self.q_stdout, q_stderr=self.q_stderr, q_error=self.q_error, robust=robust, until=until, args=args, kwargs=kwargs ) self._subprocesses.append(sub) def process_io_queue(self, q, stream): try: out = q.get(timeout=self.TIMEOUT_SECONDS) out = out.strip() if out: stream.write(out + '\n') stream.flush() except Empty: pass def process_error_queue(self, error_queue): try: error_name = error_queue.get(timeout=self.TIMEOUT_SECONDS) if error_name: error_name = error_name.strip() self._subprocesses = [s for s in self._subprocesses if s._name != error_name] logger = daiquiri.getLogger(error_name) logger.info('Will not auto-restart because it\'s not robust') except Empty: pass def loop(self, max_seconds=None): """ Main loop for the process. This will run continuously until maxiter """ loop_started = datetime.datetime.now() self._is_running = True while self._is_running: self.process_error_queue(self.q_error) if max_seconds is not None: if (datetime.datetime.now() - loop_started).total_seconds() > max_seconds: logger = daiquiri.getLogger('crontabs') logger.info('Crontabs reached specified timeout. Exiting.') break for subprocess in self._subprocesses: if not subprocess.is_alive() and not subprocess.expired: subprocess.start() self.process_io_queue(self.q_stdout, sys.stdout) self.process_io_queue(self.q_stderr, sys.stderr) ================================================ FILE: crontabs/tests/__init__.py ================================================ # flake8: noqa ================================================ FILE: crontabs/tests/test_all.py ================================================ from collections import Counter from unittest import TestCase import datetime import functools import sys import time from crontabs import Cron, Tab from dateutil.parser import parse from dateutil.relativedelta import relativedelta import fleming Tab._SILENCE_LOGGER = True # Run tests with # py.test -s crontabs/tests/test_example.py::TestSample::test_base_case # Or for parallel tests # py.test -s --cov -n 2 class ExpectedException(Exception): pass class PrintCatcher(object): # pragma: no cover This is a testing utility that doesn't need to be covered def __init__(self, stream='stdout'): self.text = '' if stream not in {'stdout', 'stderr'}: # pragma: no cover this is just a testing utitlity raise ValueError('stream must be either "stdout" or "stderr"') self.stream = stream def write(self, text): self.text += text def flush(self): pass def __enter__(self): if self.stream == 'stdout': sys.stdout = self else: sys.stderr = self return self def __exit__(self, *args): if self.stream == 'stdout': sys.stdout = sys.__stdout__ else: sys.stderr = sys.__stderr__ def time_logger(name): # pragma: no cover print('{} {}'.format(name, datetime.datetime.now())) def time__sleepy_logger(name): # pragma: no cover time.sleep(3) print('{} {}'.format(name, datetime.datetime.now())) def error_raisor(name): raise ExpectedException('This exception is expected in tests. Don\'t worry about it.') class TestCrontabs(TestCase): def test_non_robust_error(self): tab = Tab( 'one_sec', verbose=False, robust=False ).every(seconds=1).run( error_raisor, 'one_sec') with self.assertRaises(ExpectedException): tab._loop(max_iter=1) def test_robust_error(self): tab = Tab( 'one_sec', verbose=False ).every(seconds=1).run( error_raisor, 'one_sec') tab._loop(max_iter=1) def test_tab_loop_sleepy(self): tab = Tab( 'one_sec', verbose=False ).every(seconds=1).run( time__sleepy_logger, 'one_sec') with PrintCatcher() as catcher: tab._loop(max_iter=7) self.assertEqual(catcher.text.count('one_sec'), 2) def test_tab_loop_anchored(self): now = datetime.datetime.now() + datetime.timedelta(seconds=1) tab = Tab( 'one_sec', verbose=False ).every(seconds=1).starting( now).run( time_logger, 'one_sec') with PrintCatcher() as catcher: tab._loop(max_iter=3) self.assertEqual(catcher.text.count('one_sec'), 3) def test_tab_loop(self): tab = Tab( 'one_sec', verbose=False).every(seconds=1).run( time_logger, 'one_sec') with PrintCatcher() as catcher: tab._loop(max_iter=3) self.assertEqual(catcher.text.count('one_sec'), 3) def test_incomplete(self): with self.assertRaises(ValueError): Cron().schedule(Tab('a').run(time_logger, 'bad')).go() def test_bad_starting(self): with self.assertRaises(ValueError): Tab('a').starting(2.345) # Cron().schedule(Tab('a').starting(2.345)) def test_bad_every(self): with self.assertRaises(ValueError): Tab('a').every(second=1, minute=3) # Cron().schedule(Tab('a').every(second=1, minute=3)) def test_bad_interval(self): with self.assertRaises(ValueError): Tab('a').every(bad=11) # Cron().schedule(Tab('a').every(bad=11)) def test_base_case(self): cron = Cron() cron.schedule( Tab('two_sec', verbose=False).every(seconds=2).run(time_logger, 'two_sec'), Tab('three_sec', verbose=False).every(seconds=3).run(time_logger, 'three_sec') ) with PrintCatcher(stream='stdout') as stdout_catcher: cron.go(max_seconds=6) base_lookup = { 'three_sec': 3, 'two_sec': 2, } lines = list(stdout_catcher.text.split('\n')) # make sure times fall int right slots for line in lines: if line: words = line.split() name = words[0] time = parse('T'.join(words[1:])) self.assertEqual(time.second % base_lookup[name], 0) # make sure the tasks were run the proper number of times counter = Counter() for line in lines: if line: counter.update({line.split()[0]: 1}) self.assertEqual(counter['two_sec'], 3) self.assertEqual(counter['three_sec'], 2) def test_anchored_case(self): cron = Cron() starting = datetime.datetime.now() cron.schedule( Tab('three_sec', verbose=False).starting(starting).every(seconds=3).run(time_logger, 'three_sec'), Tab('three_sec_str', verbose=False).starting( starting.isoformat()).every(seconds=3).run(time_logger, 'three_sec_str'), ) with PrintCatcher(stream='stdout') as stdout_catcher: cron.go(max_seconds=3.5) # make sure times fall int right slots lines = list(stdout_catcher.text.split('\n')) for line in lines: if line: words = line.split() time = parse('T'.join(words[1:])) elapsed = (time - starting).total_seconds() self.assertTrue(elapsed < 3) def test_excluding(self): # Test base case cron = Cron() cron.schedule( Tab('base_case', verbose=True).every(seconds=1).run(time_logger, 'base_case'), Tab('d+').every(seconds=1).during(return_true).run(time_logger, 'd+'), Tab('d-').every(seconds=1).during(return_false).run(time_logger, 'd-'), Tab('e+').every(seconds=1).excluding(return_true).run(time_logger, 'e+'), Tab('e-').every(seconds=1).excluding(return_false).run(time_logger, 'e-'), ) with PrintCatcher(stream='stdout') as stdout_catcher: cron.go(max_seconds=2) self.assertTrue('d+' in stdout_catcher.text) self.assertFalse('d-' in stdout_catcher.text) self.assertFalse('e+' in stdout_catcher.text) self.assertTrue('e-' in stdout_catcher.text) def return_true(*args, **kwargs): return True def return_false(*args, **kwargs): return False def timed_error(then): now = datetime.datetime.now() if then + datetime.timedelta(seconds=3) < now < then + datetime.timedelta(seconds=6): print('timed_error_failure') raise ExpectedException('This exception is expected in tests. Don\'t worry about it.') else: print('timed_error_success') class TestRobustness(TestCase): def test_robust_case(self): then = datetime.datetime.now() cron = Cron() cron.schedule( Tab('one_sec', verbose=False).every(seconds=1).run(time_logger, 'running_time_logger'), Tab('two_sec', verbose=False, robust=True).every(seconds=1).run(functools.partial(timed_error, then)) ) with PrintCatcher(stream='stdout') as catcher: cron.go(max_seconds=10) success_count = catcher.text.count('timed_error_success') failure_count = catcher.text.count('timed_error_failure') time_logger_count = catcher.text.count('running_time_logger') self.assertEqual(success_count, 7) self.assertEqual(failure_count, 3) self.assertEqual(time_logger_count, 10) def test_non_robust_case(self): then = datetime.datetime.now() cron = Cron() cron.schedule( Tab('one_sec', verbose=False).every(seconds=1).run(time_logger, 'running_time_logger'), Tab('two_sec', verbose=False, robust=False).every(seconds=1).run(functools.partial(timed_error, then)) ) with PrintCatcher(stream='stdout') as catcher: cron.go(max_seconds=10) success_count = catcher.text.count('timed_error_success') failure_count = catcher.text.count('timed_error_failure') time_logger_count = catcher.text.count('running_time_logger') self.assertEqual(success_count, 3) self.assertEqual(failure_count, 1) self.assertEqual(time_logger_count, 10) def func(): print('func_was_called') class TestStartingOnNextInterval(TestCase): def test_starts_on_next(self): second = 0 interval_seconds = 5 while second % interval_seconds == 0: now = datetime.datetime.now() second = now.second epoch = fleming.floor(now, second=interval_seconds) then = epoch + relativedelta(seconds=interval_seconds) cron = Cron().schedule( Tab( name='pusher', robust=False, memory_friendly=False, ).run( func, ).starting( then ).every( seconds=5 ) ) with PrintCatcher(stream='stdout') as catcher: cron.go(max_seconds=5) assert('func_was_called' in catcher.text) ================================================ FILE: crontabs/version.py ================================================ __version__ = '0.2.2' ================================================ FILE: docs/Makefile ================================================ # Makefile for Sphinx documentation # # You can set these variables from the command line. SPHINXOPTS = SPHINXBUILD = sphinx-build PAPER = BUILDDIR = _build # User-friendly check for sphinx-build ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) $(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) endif # Internal variables. PAPEROPT_a4 = -D latex_paper_size=a4 PAPEROPT_letter = -D latex_paper_size=letter ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . # the i18n builder cannot share the environment and doctrees with the others I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp epub latex latexpdf text man changes linkcheck doctest gettext help: @echo "Please use \`make ' where is one of" @echo " html to make standalone HTML files" @echo " dirhtml to make HTML files named index.html in directories" @echo " singlehtml to make a single large HTML file" @echo " pickle to make pickle files" @echo " json to make JSON files" @echo " htmlhelp to make HTML files and a HTML help project" @echo " epub to make an epub" @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" @echo " latexpdf to make LaTeX files and run them through pdflatex" @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" @echo " text to make text files" @echo " man to make manual pages" @echo " texinfo to make Texinfo files" @echo " info to make Texinfo files and run them through makeinfo" @echo " gettext to make PO message catalogs" @echo " changes to make an overview of all changed/added/deprecated items" @echo " xml to make Docutils-native XML files" @echo " pseudoxml to make pseudoxml-XML files for display purposes" @echo " linkcheck to check all external links for integrity" @echo " doctest to run all doctests embedded in the documentation (if enabled)" clean: rm -rf $(BUILDDIR)/* html: $(SPHINXBUILD) -W -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." dirhtml: $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." singlehtml: $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml @echo @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." pickle: $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle @echo @echo "Build finished; now you can process the pickle files." json: $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json @echo @echo "Build finished; now you can process the JSON files." htmlhelp: $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp @echo @echo "Build finished; now you can run HTML Help Workshop with the" \ ".hhp project file in $(BUILDDIR)/htmlhelp." epub: $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub @echo @echo "Build finished. The epub file is in $(BUILDDIR)/epub." latex: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." @echo "Run \`make' in that directory to run these through (pdf)latex" \ "(use \`make latexpdf' here to do that automatically)." latexpdf: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo "Running LaTeX files through pdflatex..." $(MAKE) -C $(BUILDDIR)/latex all-pdf @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." latexpdfja: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo "Running LaTeX files through platex and dvipdfmx..." $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." text: $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text @echo @echo "Build finished. The text files are in $(BUILDDIR)/text." man: $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man @echo @echo "Build finished. The manual pages are in $(BUILDDIR)/man." texinfo: $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo @echo @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." @echo "Run \`make' in that directory to run these through makeinfo" \ "(use \`make info' here to do that automatically)." info: $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo @echo "Running Texinfo files through makeinfo..." make -C $(BUILDDIR)/texinfo info @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." gettext: $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale @echo @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." changes: $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes @echo @echo "The overview file is in $(BUILDDIR)/changes." linkcheck: $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck @echo @echo "Link check complete; look for any errors in the above output " \ "or in $(BUILDDIR)/linkcheck/output.txt." doctest: $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest @echo "Testing of doctests in the sources finished, look at the " \ "results in $(BUILDDIR)/doctest/output.txt." xml: $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml @echo @echo "Build finished. The XML files are in $(BUILDDIR)/xml." pseudoxml: $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml @echo @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." ================================================ FILE: docs/conf.py ================================================ # -*- coding: utf-8 -*- # import inspect import os import re import sys file_dir = os.path.realpath(os.path.dirname(__file__)) sys.path.append(os.path.join(file_dir, '..')) def get_version(): """Obtain the packge version from a python file e.g. pkg/__init__.py See . """ file_dir = os.path.realpath(os.path.dirname(__file__)) with open( os.path.join(file_dir, '..', 'crontabs', '__init__.py')) as f: txt = f.read() version_match = re.search( r"""^__version__ = ['"]([^'"]*)['"]""", txt, re.M) if version_match: return version_match.group(1) raise RuntimeError("Unable to find version string.") # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. #sys.path.insert(0, os.path.abspath('.')) # -- General configuration ------------------------------------------------ extensions = [ 'sphinx.ext.autodoc', #'sphinx.ext.intersphinx', 'sphinx.ext.viewcode', #'sphinxcontrib.fulltoc', ] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # The suffix of source filenames. source_suffix = '.rst' # The master toctree document. master_doc = 'toc' # General information about the project. project = 'crontabs' copyright = '2017, Rob deCarvalho' # The short X.Y version. version = get_version() # The full version, including alpha/beta/rc tags. release = version exclude_patterns = ['_build'] # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' intersphinx_mapping = { 'python': ('http://docs.python.org/3.4', None), 'django': ('http://django.readthedocs.org/en/latest/', None), #'celery': ('http://celery.readthedocs.org/en/latest/', None), } # -- Options for HTML output ---------------------------------------------- html_theme = 'default' #html_theme_path = [] on_rtd = os.environ.get('READTHEDOCS', None) == 'True' if not on_rtd: # only import and set the theme if we're building docs locally import sphinx_rtd_theme html_theme = 'sphinx_rtd_theme' html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". # html_static_path = ['_static'] html_static_path = [] # Custom sidebar templates, maps document names to template names. #html_sidebars = {} # Additional templates that should be rendered to pages, maps page names to # template names. #html_additional_pages = {} # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. html_show_sphinx = False # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. html_show_copyright = True # Output file base name for HTML help builder. htmlhelp_basename = 'crontabsdoc' ================================================ FILE: docs/index.rst ================================================ crontabs ============================= Replace this text with content. ================================================ FILE: docs/ref/crontabs.rst ================================================ .. _ref-crontabs: API Documentation ================== Replace this with api documentation ================================================ FILE: docs/toc.rst ================================================ Table of Contents ================= .. toctree:: :maxdepth: 3 index ref/crontabs ================================================ FILE: publish.py ================================================ import subprocess subprocess.call('pip install wheel'.split()) subprocess.call('pip install twine'.split()) subprocess.call('rm -rf ./build'.split()) subprocess.call('rm -rf ./dist/'.split()) subprocess.call('python setup.py clean --all'.split()) subprocess.call('python setup.py sdist bdist_wheel'.split()) subprocess.call('twine upload dist/*'.split()) ================================================ FILE: setup.cfg ================================================ [coverage:report] show_missing=True exclude_lines = # Have to re-enable the standard pragma pragma: no cover # Don't complain if tests don't hit defensive assertion code: raise NotImplementedError [coverage:run] omit = crontabs/version.py crontabs/__init__.py */env/* concurrency = multiprocessing [flake8] max-line-length = 120 exclude = docs,env,*.egg max-complexity = 13 ignore = E402 [build_sphinx] source-dir = docs/ build-dir = docs/_build all_files = 1 [upload_sphinx] upload-dir = docs/_build/html [bdist_wheel] universal = 1 ================================================ FILE: setup.py ================================================ # import multiprocessing to avoid this bug (http://bugs.python.org/issue15881#msg170215) import multiprocessing assert multiprocessing import re from setuptools import setup, find_packages def get_version(): """ Extracts the version number from the version.py file. """ VERSION_FILE = 'crontabs/version.py' mo = re.search(r'^__version__ = [\'"]([^\'"]*)[\'"]', open(VERSION_FILE, 'rt').read(), re.M) if mo: return mo.group(1) else: raise RuntimeError('Unable to find version string in {0}.'.format(VERSION_FILE)) install_requires = [ 'fleming', 'daiquiri[json]' ] tests_require = [ 'coverage', 'flake8', 'mock', 'pytest', 'pytest-cov', 'pytest-xdist', 'wheel', ] docs_require = [ # 'Sphinx', # 'sphinx_rtd_theme' ] extras_require = { 'dev': tests_require + docs_require, } setup( name='crontabs', version=get_version(), description='Simple job scheduling for python', long_description='Simple job scheduling for python', url='https://github.com/robdmc/crontabs', author='Rob deCarvalho', author_email='unlisted@unlisted.net', keywords='', packages=find_packages(), classifiers=[ 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Intended Audience :: Developers', 'License :: OSI Approved :: MIT License', 'Operating System :: OS Independent', ], license='MIT', include_package_data=True, test_suite='nose.collector', install_requires=install_requires, tests_require=tests_require, extras_require=extras_require, zip_safe=False, )