[ { "path": ".claude/hooks/session-start.sh", "content": "#!/bin/bash\nset -euo pipefail\n\n# Only run in remote (cloud) environments\nif [ \"${CLAUDE_CODE_REMOTE:-}\" != \"true\" ]; then\n exit 0\nfi\n\necho '{\"async\": true, \"asyncTimeout\": 300000}'\n\n# Read Crystal version from .tool-versions\nCRYSTAL_VERSION=$(grep '^crystal ' \"$CLAUDE_PROJECT_DIR/.tool-versions\" | awk '{print $2}')\n\n# Install Crystal compiler if not already present\nif ! command -v crystal &> /dev/null; then\n # Install system dependencies required by Crystal\n apt-get update\n apt-get install -y libgmp-dev libxml2-dev libevent-dev libgc-dev\n\n # Download and install Crystal from GitHub releases\n curl -fsSL \"https://github.com/crystal-lang/crystal/releases/download/${CRYSTAL_VERSION}/crystal-${CRYSTAL_VERSION}-1-linux-x86_64-bundled.tar.gz\" -o /tmp/crystal.tar.gz\n mkdir -p /usr/local/crystal\n tar -xzf /tmp/crystal.tar.gz -C /usr/local/crystal --strip-components=2\n ln -sf /usr/local/crystal/bin/crystal /usr/local/bin/crystal\n ln -sf /usr/local/crystal/bin/shards /usr/local/bin/shards\n rm /tmp/crystal.tar.gz\nfi\n\n# Start Redis server if not already running\nif ! redis-cli ping &> /dev/null 2>&1; then\n redis-server --daemonize yes\nfi\n\n# Disable RDB persistence to avoid dump.rdb noise in the project directory\nredis-cli config set save \"\" > /dev/null 2>&1\n\n# Install Crystal shard dependencies\ncd \"$CLAUDE_PROJECT_DIR\"\nshards install\n" }, { "path": ".claude/settings.json", "content": "{\n \"hooks\": {\n \"SessionStart\": [\n {\n \"hooks\": [\n {\n \"type\": \"command\",\n \"command\": \"$CLAUDE_PROJECT_DIR/.claude/hooks/session-start.sh\"\n }\n ]\n }\n ]\n }\n}\n" }, { "path": ".claude/todo.md", "content": "# Migration from publish_metrics branch\n\n## Background\n\nThe `publish_metrics` branch contains observability improvements that need to be migrated to master.\nThis branch has ~24 commits of work dating back to October 2024.\n\n## Functionality to Migrate\n\nItems ordered by size of change, smallest first.\n\n### 1. Metadata Self-Cleanup ✅\nAdd TTL to metadata so stale entries auto-expire.\n\n- [x] Add `@metadata.delete in: 1.hour` to Executor heartbeat\n- [x] Add `@metadata.delete in: 1.hour` to Overseer heartbeat\n\nAlready implemented - `Metadata#heartbeat!` includes `delete in: 1.hour` and both observers use it.\n\n### 2. Overseer Event Naming Standardization ✅\nStandardize to past tense for consistency with other events.\n\n- [x] \"starting\" → \"started\"\n- [x] \"stopping\" → \"stopped\"\n- [x] \"stopped\" → \"exited\"\n\nDone in `src/mosquito/api/overseer.cr`.\n\n### 3. Executor Bug Fix ✅\n- [x] Fix latent bug: executor calculating run time incorrectly (see commit `mvouzzrz`)\n\nFixed `100_000` → `1_000_000` in microseconds calculation in `src/mosquito/api/executor.cr`.\n\n### 4. Stable Instance IDs — Skipped\n`object_id` is sufficient; no need for `Random::Secure.hex` IDs.\n\n### 5. Nested Publish Context ✅\nAllow executor events to be namespaced under their parent overseer.\n\n- [x] Add parent context support to `PublishContext` initializer\n- [x] Pass overseer reference to Executor\n- [x] Update Executor observer to create PublishContext with overseer as parent\n- [x] Executor events publish under `[:overseer, overseer_id, :executor, executor_id]`\n- [x] Fix tests (executor/overseer specs, mock_overseer)\n\nDone.\n\n### 6. Observability Gating ✅\nGate metadata writes behind existing `publish_metrics` config.\n\n- [x] Gate `heartbeat!` in Executor observer behind `metrics` macro\n- [x] Gate `heartbeat` in Overseer observer behind `metrics` macro (includes `register_overseer`)\n- [x] Gate `update_executor_list` in Overseer observer behind `metrics` macro\n- [x] Fix pre-existing race condition in executor spec (lazy getter initialization across fibers)\n\nDecided against a separate `Enabled` module / `enable_observability` config — no compelling reason\nto have two flags. Reused the existing `metrics` macro which checks `publish_metrics`.\n\n### 7. Observability Tests ✅\n\n#### Fix `assert_message_received` ✅\nThe helper in `spec/helpers/pub_sub.cr` doesn't actually assert — `find` returns nil\nand the result is discarded. All existing event publishing tests are vacuous (always pass).\n- [x] Fix `assert_message_received` to fail when no matching message is found\n- [x] Fix overseer event assertions to match actual event names\n\n#### Metrics gating ✅\n- [x] Executor: heartbeat is skipped when `publish_metrics = false`\n- [x] Event publishing is skipped when `publish_metrics = false` (tested via publisher_spec, covers all observers)\n\n#### Queue observer events ✅\n- [x] Publishes \"rescheduled\" event\n- [x] Publishes \"forgotten\" event\n- [x] Publishes \"banished\" event\n\n#### Publish context structure ✅\n- [x] Executor publish context is nested under overseer's context\n- [x] Overseer publish context has correct originator key\n- [x] Queue publish context has correct originator key\n\n## Files to Reference on publish_metrics\n\nKey source files:\n- `src/mosquito/observability/concerns/enabled.cr`\n- `src/mosquito/observability/concerns/publish_context.cr`\n- `src/mosquito/observability/concerns/publisher.cr`\n- `src/mosquito/observability/executor.cr`\n- `src/mosquito/observability/overseer.cr`\n- `src/mosquito/observability/queue.cr`\n\nKey test files:\n- `test/mosquito/observability/enabled_test.cr`\n- `test/mosquito/observability/executor_test.cr`\n- `test/mosquito/observability/overseer_test.cr`\n- `test/mosquito/observability/queue_test.cr`\n\n## Notes\n\n- The publish_metrics branch has diverged (shown as `??` in jj) - resolve carefully\n- Current working copy already has queue observer events (rescheduled, forgotten, banished)\n- Duration averaging and expected_duration_ms already implemented on master\n- Test directory structure (`test/` instead of `spec/`) already migrated on master\n" }, { "path": ".github/FUNDING.yml", "content": "# These are supported funding model platforms\n\ngithub: robacarp\npatreon: # Replace with a single Patreon username\nopen_collective: # Replace with a single Open Collective username\nko_fi: # Replace with a single Ko-fi username\ntidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel\n" }, { "path": ".github/ISSUE_TEMPLATE/bug.md", "content": "---\nname: Bug\nabout: Mosquito has a bug!\ntitle: ''\nlabels: ''\nassignees: robacarp\n\n---\n\nPlease include some details:\n\nCrystal version: 0.28.0\nMosquito Shard version: 0.4.0\n" }, { "path": ".github/workflows/ci.yml", "content": "name: Test and Demo\non:\n pull_request:\n branches:\n - master\n push:\n branches:\n - master\n\njobs:\n build:\n strategy:\n fail-fast: false\n matrix:\n crystal_version: [1.19, latest]\n experimental:\n - false\n include:\n - crystal_version: nightly\n experimental: true\n\n name: Build\n runs-on: ubuntu-latest\n\n container:\n image: crystallang/crystal:latest\n\n continue-on-error: ${{ matrix.experimental }}\n services:\n redis:\n image: redis\n\n env:\n REDIS_URL: redis://redis:6379/1\n\n steps:\n - uses: actions/checkout@v4\n - run: apt-get update\n - uses: crystal-lang/install-crystal@v1\n with:\n crystal: ${{matrix.crystal_version}}\n - run: printenv\n - run: crystal --version\n - run: shards install\n - run: make test\n - run: make demo\n" }, { "path": ".github/workflows/docs.yml", "content": "name: Build Docs\non:\n push:\n branches:\n - master\n\njobs:\n deploy:\n name: Running Docs\n runs-on: ubuntu-latest\n\n container:\n image: crystallang/crystal:latest\n\n steps:\n - uses: actions/checkout@v2\n - run: apt-get update\n - uses: crystal-lang/install-crystal@v1\n - run: crystal --version\n - run: shards install\n - run: crystal docs\n\n - name: Deploy\n uses: peaceiris/actions-gh-pages@v3\n with:\n github_token: ${{ secrets.GITHUB_TOKEN }}\n publish_dir: ./docs\n" }, { "path": ".gitignore", "content": "/lib/\n/bin/\n/.shards/\n\n# Libraries don't need dependency lock\n# Dependencies will be locked in application that uses them\n/shard.lock\ndocs\n# Claude Code local user config (not hooks/settings which are shared)\n.claude/local/\nCLAUD.local.md\n" }, { "path": ".tool-versions", "content": "crystal 1.19.1\n" }, { "path": ":w", "content": "\n" }, { "path": "CHANGELOG.md", "content": "# Changelog\n\nThe format is based on [Keep a\nChangelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to\n[Semantic Versioning](https://semver.org/spec/v2.0.0.html).\n\n## [Unreleased]\n### Added\n- Mosquito::Api now allows for inspecting the state of the state of a mosquito cluster. Many of these features are disabled by default by the configuration property `config.publish_metrics`.\n - Executor api implemented in #147\n - JobRun api implemented in #148 and #161\n - Overseer api implemented in #150\n - Queue api implemented in #153\n- Mosquito now publishes a variety of events and metrics to a redis pubsub channel. This behavior is disabled by default with the configuration property `config.publish_metrics`.\n - Executor events in #154: job-started and job-finished\n - Overseer events in #160: starting, executor-created, executor-died, stopping, and stopped\n - Queue events: enqueue, dequeue, reschedule, forget, and banish\n - Expected job duration is now published with executor events\n The Mosquito API can be used to subscribe to these events with `Mosquito::API.event_receiver`\n- Pluggable dequeue adapters allow customizing how jobs are selected from queues (#183)\n - `DequeueAdapter` abstract base class defines the adapter interface\n - `ShuffleDequeueAdapter` is the default, preserving existing randomized behavior\n - `WeightedDequeueAdapter` allows queue-level prioritization via configurable weights\n - Configurable via `Mosquito.configure { |c| c.dequeue_adapter = ... }`\n- Executor count is now configurable (default increased from 3 to 6) (#184)\n - Set via `Mosquito.configure { |c| c.executor_count = 10 }`\n - Override with the `MOSQUITO_EXECUTOR_COUNT` environment variable\n- `JobRun#started_at` and `JobRun#finished_at` timestamps are now exposed as typed `Time?` getters (#179)\n- Graceful worker shutdown: on SIGTERM/SIGINT the overseer stops dequeuing, waits for in-flight executors to finish, and requeues any jobs left in pending back to waiting (#190)\n- Queues can now be paused and resumed. While paused, `#dequeue` returns nil and jobs accumulate until the queue is resumed. An optional duration enables automatic resumption, useful for backing off rate-limited resources. (#192)\n- Overseers now take ownership of job runs when dequeued, and clean up abandoned pending job runs on startup (#180)\n- Mosquito can now accept pre-existing backend connections via `Configuration#backend_connection`. This allows sharing a connection pool with the rest of an application. (#193)\n- JobRun now uses Metadata for all backend storage operations, replacing direct backend calls with the Metadata abstraction layer.\n- `Mosquito::UniqueJob` module provides opt-in job deduplication. Including the module in a job class prevents enqueueing duplicate jobs when an identical job is already waiting or scheduled. Uniqueness keys are derived from job parameters at compile time.\n\n### Changed\n- (breaking) `Configuration#connection_string` has been renamed to `Configuration#backend_connection_string` (#193)\n- (minor breaking) Logs are now emitted from runners with a slighly different source tag. (#152)\n For example:\n The overseer boot message used to be:\n `INFO - mosquito.runners.overseer.4315742080: Overseer<4315742080> is starting`\n Now the message is simply:\n `INFO - mosquito.overseer: starting`\n- Mosquito now runs CI checks for compatibility with Crystal 1.6\n- The coordinator now uses UTC time instead of monotonic time\n\n### Fixed\n- Fixed a KeyError crash in the demo when job metadata was missing by using safe key access.\n- the queue_list runner was never being shut down, but it is now as of (#165)\n- Fixed a bug which would cause a mosquito server to hang at exit indefinitely if a job was mid-run during an interrupt. (#165)\n- Fixed a bug which would cause a correctly exiting server to prematurely exit without emitting shutdown sequence logs and events. (#165)\n- Crashed executors are now properly detected and replaced, preventing overseers from running with no executors\n- Overseer now correctly deregisters on clean exit\n- Pubsub logging now uses the `mosquito.events` namespace instead of the root `mosquito` namespace\n- Queue `@empty` latch no longer permanently prevents re-dequeue after a queue drains\n- Observer functionality is correctly gated behind the `publish_metrics` config flag\n- Executor events are correctly scoped to within the overseer\n- Fixed a latent bug which caused job duration to be reported incorrectly\n- Fixed `Mosquito::Api.list_queues`\n\n### Performance\n- Optimized `metadata#set` to decrease the number of redis commands\n\n## [2.0.0]\n### Added\n- Adds a test backend, which can be used to inspect jobs that were enqueued and\n the parameters they were enqueued with.\n- Job#fail now takes an optional `retry` parameter which defaults to true, allowing\n a developer to explicitly mark a job as not retry-able during a job run. Additionally\n a `should_retry` property exists which can be set as well.\n- Mosquito::Configuration now provides `global_prefix` to change the global Redis namespace \n prefix, allowing for more than one mosquito app to share a redis instance (thanks @dammer, cf #134).\n\n### Fixed\n- PeriodicJobs are now correctly run once per interval in an environment with many workers.\n- Running more than ~10 workers no longer causes workers to crash, fixing #137 (cf #138).\n- Mosquito is now more broadly compatible with jgaskins redis, swapping 0.7.0 for 0.7, and\n forward compatible through 0.8. (thanks @rmarronnier)\n- Mosquito now more gracefully responds to SIGTERM, fixes #122, cf #123.\n- High CPU usage on linux is no longer an issue, fixes #126, cf #128.\n\n### Breaking Changes\n- The QueuedJob `params` macro has been replaced with `param`\n which declares only one parameter at a time.\n- JobRun#delete now explicitly takes an Int, rather than simply defaulting to 0 (thanks @jwoertink, cf #136).\n- removes deprecated Backend.delete(String, Int32), use Backend.delete(String, Int64) instead.\n- removes deprecated Queue#length, use Queue#size instead.\n- removes option to run the cron scheduler declaratively, it is now always on with a distributed lock.\n\n### Performance\n- Dramatically decreases the time spent listing queues #120\n- Replaces #keys with #scan_each to list runners #138\n- Provides for multiple executors operating under a single runner #123\n\n\n## [1.0.2]\n### Fixed\n- Mosquito::Runner.start now captures the thread with a spin lock again. The new\n behavior of returning imediately can be achieved by calling #start(spin: false) \n\n## [1.0.1] [YANKED]\n### Added\n- Implements a distributed lock for scheduler coordination. The behavior is opt-in\n for now, but will become the default in the next release. See #108.\n- Provides a helpful error message for most implementation errors dealing with\n declaring params.\n\n### Changed\n- Mosquito::QueuedJob: the `params` macro has been deprecated in favor of `param`.\n See #110.\n- The deprecated Redis command [`rpoplpush`](https://redis.io/commands/rpoplpush/)\n is no longer used. This lifts the minimum redis server requirement up to 6.2.0\n and jgaskins redis to > 0.7.0.\n- Mosquito::Runner.start no longer captures the thread with a spin lock. [DEFECT]\n\n### Removed\n- Mosquito config option `run_cron_scheduler` is no longer present, multiple\n workers will compete for a distributed lock instead. See #108.\n\n## [1.0.0]\n### Added\n- Jobs can now specify their retry/reschedule logic with the #rescheduleable?\n and #reschedule_interval methods.\n- Job metadata storage engine.\n- Jobs can now specify `after` hooks.\n- Mosquito::Runner now has a `stop` method which halts the runner after\n completion of any running tasks. See issue #21 and pull #87.\n- Mosquito config option `run_cron_scheduler` is no longer present, multiple\n workers will compete for a distributed lock instead.\n\n### Changed\n- The storage backend is now implemented via interface, allowing alternate\n backends to be implemented.\n- The rate limiting functionality is now implemented in a module,\n `Mosquito::RateLimiter`. See pull #77 for migration details.\n- ** BREAKING ** `Job.job_type` has been replaced with `Job.queue_name`. The\n functionailty is identical but easier to access. See #86.\n- `log` statements now properly identify where they're coming from rather than\n just 'mosquito'. See issue #78 and pull #88.\n- Mosquito now connects to Redis using a connection pool. See #89\n- ** BREAKING ** `Mosquito.settings` is now `Mosquito.configuration`. While\n this is technically a public API, it's unlikely anyone is using it for\n anything.\n- Mosquito::Runner.start need not be called from a spawn, it will spawn on it's own.\n\n### Removed\n- Runner.idle_wait configuration is deprecated. Instead use\n Mosquito.configure#idle_wait.\n- Built in serializer for Granite models, and the Model type alias. See\n Serializers in the documentation if the functionality is necessary.\n- Mosquito no longer depends on luckyframework/habitat.\n\n### Fixed\n- Boolean false can now be specified as the default value for a parameter:\n `params(name = false)`\n\n## [0.11.2] - 2022-01-25\n### Fixed\n- #66 Jobs with no parameters can now be enqueued without specifying an empty\n `params()`.\n- #65 PeriodicJobs can now specify their run period in months.\n\n### Notes\nThe v0 major version is now bugfix-only. Please update to v1.0. v0 will be\nsupported as long as it's feasible to do so.\n\n## [0.11.1] - 2022-01-17\n### Added\n- Jobs can now specify `before` hooks, which can abort before the perform is\n triggered.\n- The Cron scheduler for periodic jobs can now be disabled via\n Mosquito.configure#run_cron_scheduler\n- The list of queues which are watched by the runner can now be configured via\n Mosquito.configure#run_from.\n\n### Updated\n- Redis shard 2.8.0, removes hash shims which are no longer needed. Thanks\n @jwoertink.\n\n## [0.11.0] - 2021-04-10\nProforma release for Crystal 1.0.\n\n## [0.10.0] - 2021-02-15\n### Added\n- UUID serializer helpers.\n\n### Updated\n- Switches from Benchmark.measure to Time.measure, thanks @anapsix.\n- Runner.idle_wait is now configured via Mosquito.configure instead of directly\n on Mosquito::Runner.\n\n## [0.9.0] - 2020-10-26\n### Added\n- Allows redis connection string to be specified via config option, thanks\n @watzon.\n\n### Deprecated\n- Connecting to redis via implicit REDIS_URL parameter is deprecated, thanks\n @watzon.\n\n## [0.8.0] - 2020-05-28\n### Fixed\n- (Breaking) Dead tasks which have failed and expired are now cleaned up with a\n Redis TTL. See Pull #48.\n\n## [0.7.0] - 2020-05-05\n### Added\n- ability to configure Runner.idle_wait period, thanks @mamantoha.\n\n### Updated\n- Point to Crystal 0.34.0, thanks @alex-lairan.\n\n### Changed\n- Replaces `Logger` with the more flexible `Log`.\n\n## [0.6.0] - 2019-12-19\n### Updated\n- Point to Crystal 0.31.1, 0.32.1.\n- Redis version, thanks @nsuchy.\n\n## [0.5.0] - 2019-06-14\n### Fixed\n- Issue #26 Unresolved local var error, thanks @blacksmoke16.\n\n## [0.4.0] - 2019-04-26\n### Added\n- Throttling logic, thanks @blacksmoke16.\n\n## [0.3.0] - 2018-11-25\n### Updated\n- Point to crystal 0.27, thanks @blacksmoke16.\n\n### Fixed\n- Brittle/intermittently failing tests.\n\n## [0.2.1] - 2018-10-01\n\n### Added\n- Logo, contributed by @psikoz.\n- configuration for CI : `make test demo` will run all acceptance criteria.\n- demo section.\n- makefile.\n\n### Updated\n- specify crystal 0.26.\n- simplify macro logic in QueuedJob.\n\n## [0.2.0] - 2018-06-22\n### Updated\n- Specify crystal-redis 2.0 and crystal 0.25.\n\n## [0.1.1] - 2018-06-08\n\n### Added\n- Job classes can now disable rescheduling on failure.\n\n### Updated\n- Readme.\n- Misc typo fixes and flexibility upgrades.\n- Update Crystal specification 0.23.1 -> .24.2.\n- Correctly specify and sync version numbers from shard.yml / version.cr / git\n tag.\n- Use configurable Logger instead of writing directly to stdout.\n- Log output is now colorized and formatted to be read by human eyes.\n\n### Changed\n- Breaking: Update Mosquito::Model type alias to match updates to Granite.\n\n### Fixed\n- BUG: task id was mutating on each save, causing weird logging when tasks\n reschedule.\n- PERFORMANCE: adding IDLE_WAIT to prevent slamming redis when the queues are\n empty. Smarter querying of the queues for work.\n" }, { "path": "LICENSE", "content": "The MIT License (MIT)\n\nCopyright (c) 2019 Robert L Carpenter\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in\nall copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\nTHE SOFTWARE.\n" }, { "path": "Makefile", "content": "SHELL=/bin/bash\n\n.PHONY: all\nall: test\n\tshards build\n\n\n.PHONY: test\ntest:\n\tcrystal spec --error-trace -- --chaos\n\n.PHONY: demo\ndemo:\n\tcrystal run demo/run.cr --error-trace\n" }, { "path": "README.md", "content": "\"mosquito\"\n\n[![GitHub](https://img.shields.io/github/license/mosquito-cr/mosquito.svg?style=for-the-badge)](https://tldrlegal.com/license/mit-license)\n\n\n\nMosquito is a generic background job runner written primarily for Crystal. Significant inspiration from experience with the successes and failings many Ruby gems in this vein. Once compiled, a mosquito binary can start work in about 10 milliseconds.\n\nMosquito currently provides these features:\n\n- Delayed execution (`SendEmailJob.new(email: :welcome, address: user.email).enqueue(in: 3.minutes)`)\n- Scheduled / Periodic execution (`RunEveryHourJob.new`)\n- Job Storage in Redis\n- Automatic rescheduling of failed jobs\n- Progressively increasing delay of rescheduled failed jobs\n- Dead letter queue of jobs which have failed too many times\n- Rate limited jobs\n\nCurrent Limitations:\n- Visibility into a running job network and queue is limited. There is a working proof of concept [visualization API](https://github.com/mosquito-cr/mosquito/issues/90) and [bare-bones terminal application](https://github.com/mosquito-cr/tui-visualizer).\n\n## Project State\n\nThe Mosquito project is stable. A few folks are using Mosquito in production, and it's going well.\n\nThere are some features which would be nice to have, but what is here is both tried and tested.\n\nIf you're using Mosquito, please [get in touch](https://github.com/mosquito-cr/mosquito/discussions) on the Discussion board or [on Crystal chat](https://crystal-lang.org/community/) with any questions, feature suggestions, or feedback.\n\n## Installation\n\nUpdate your `shard.yml` to include mosquito:\n\n```diff\ndependencies:\n+ mosquito:\n+ github: mosquito-cr/mosquito\n```\n\n## Usage\n\n### Step 1: Define a queued job\n\n```crystal\n# src/jobs/puts_job.cr\nclass PutsJob < Mosquito::QueuedJob\n param message : String\n\n def perform\n puts message\n end\nend\n```\n\n### Step 2: Trigger that job\n\n```crystal\n# src//.cr\nPutsJob.new(message: \"ohai background job\").enqueue\n```\n\n### Step 3: Run your worker to process the job\n\n```crystal\n# src/worker.cr\n\nMosquito.configure do |settings|\n settings.redis_url = ENV[\"REDIS_URL\"]\nend\n\nMosquito::Runner.start\n```\n\n```text\ncrystal run src/worker.cr\n```\n\n### Success\n\n```\n> crystal run src/worker.cr\n2017-11-06 17:07:29 - Mosquito is buzzing...\n2017-11-06 17:07:51 - Running task puts_job<...> from puts_job\n2017-11-06 17:07:51 - [PutsJob] ohai background job\n2017-11-06 17:07:51 - task puts_job<...> succeeded, took 0.0 seconds\n```\n\n[More information about queued jobs](https://mosquito-cr.github.io/manual/index.html#queued-jobs) in the manual.\n\n------\n\n## Periodic Jobs\n\nPeriodic jobs run according to a predefined period -- once an hour, etc.\n\nThis periodic job:\n```crystal\nclass PeriodicallyPutsJob < Mosquito::PeriodicJob\n run_every 1.minute\n\n def perform\n emotions = %w{happy sad angry optimistic political skeptical epuhoric}\n puts \"The time is now #{Time.local} and the wizard is feeling #{emotions.sample}\"\n end\nend\n```\n\nWould produce this output:\n```crystal\n2017-11-06 17:20:13 - Mosquito is buzzing...\n2017-11-06 17:20:13 - Queues: periodically_puts_job\n2017-11-06 17:20:13 - Running task periodically_puts_job<...> from periodically_puts_job\n2017-11-06 17:20:13 - [PeriodicallyPutsJob] The time is now 2017-11-06 17:20:13 and the wizard is feeling skeptical\n2017-11-06 17:20:13 - task periodically_puts_job<...> succeeded, took 0.0 seconds\n2017-11-06 17:21:14 - Queues: periodically_puts_job\n2017-11-06 17:21:14 - Running task periodically_puts_job<...> from periodically_puts_job\n2017-11-06 17:21:14 - [PeriodicallyPutsJob] The time is now 2017-11-06 17:21:14 and the wizard is feeling optimistic\n2017-11-06 17:21:14 - task periodically_puts_job<...> succeeded, took 0.0 seconds\n2017-11-06 17:22:15 - Queues: periodically_puts_job\n2017-11-06 17:22:15 - Running task periodically_puts_job<...> from periodically_puts_job\n2017-11-06 17:22:15 - [PeriodicallyPutsJob] The time is now 2017-11-06 17:22:15 and the wizard is feeling political\n2017-11-06 17:22:15 - task periodically_puts_job<...> succeeded, took 0.0 seconds\n```\n\n[More information on periodic jobs](https://mosquito-cr.github.io/manual/index.html#periodic-jobs) in the manual.\n\n## Advanced usage\n\nFor more advanced topics, including [use with Lucky Framework](https://mosquito-cr.github.io/manual/lucky_framework.html), [throttling or rate limiting](https://mosquito-cr.github.io/manual/rate_limiting.html), check out the [full manual](https://mosquito-cr.github.io/manual).\n\n## Contributing\n\nContributions are welcome. Please fork the repository, commit changes on a branch, and then open a pull request.\n\n### Crystal Versions\n\nMosquito aims to be compatible with the latest Crystal release, and the [latest patch for all post-1.0 minor crystal versions](https://github.com/mosquito-cr/mosquito/blob/master/.github/workflows/ci.yml#L17).\n\nFor development purposes [you're encouraged to stay in sync with `.tool-versions`](https://github.com/mosquito-cr/mosquito/blob/master/.tool-versions).\n\n### Testing\n\n`crystal spec` Will run the tests, or `make test` will too.\n" }, { "path": "benchmark/benchmark.cr", "content": "require \"../src/mosquito\"\nrequire \"./jobs/*\"\n\nMosquito.configure do |settings|\n settings.backend_connection_string = ENV[\"REDIS_URL\"]? || \"redis://localhost:6379/4\"\n settings.publish_metrics = true\nend\n\nMosquito.configuration.backend.flush\n\nLog.setup do |c|\n backend = Log::IOBackend.new\n\n c.bind \"redis.*\", :error, backend\n c.bind \"mosquito.*\", :error, backend\nend\n\nstopping = false\nSignal::INT.trap do\n if stopping\n puts \"SIGINT received again, crash-exiting.\"\n exit 1\n end\n\n Mosquito::Runner.stop\n stopping = true\nend\n\nMosquito::Runner.start spin: false\n\nEventCount = 500\nevents = Deque(Time).new(EventCount)\nevent_count = 0\nmissed_messages = 0\n\nchannel = Mosquito.backend.subscribe(EmitMessageJob::PUBSUB_CHANNEL)\n\nprint \"enqueuing benchmark jobs...\"\n10000.times {\n EmitMessageJob.new.enqueue\n}\nputs \"done\"\n\nspawn do\n loop do\n break unless Mosquito::Runner.keep_running\n if missed_messages >= 100\n Mosquito::Runner.stop\n break\n end\n\n select\n when channel.receive\n events << Time.utc\n event_count += 1\n when timeout(100.milliseconds)\n missed_messages += 1\n end\n end\nend\n\nmessage = ->(span : Time::Span) do\n print \"\\r\"\n print \"Events: #{event_count} | \"\n print \"Span: #{span.total_seconds.round(2)} | \"\n print \"Rate: #{events.size.to_f./(span.to_f).round(2)} events/sec\"\n print \" \"\nend\n\nloop do\n break unless Mosquito::Runner.keep_running\n\n # if events.size >= EventCount\n # (events.size - EventCount).times { events.shift }\n # end\n\n unless events.size >= 10\n print \"\\r\"\n print \"Waiting for events...\"\n sleep 0.1.seconds\n next\n end\n\n message.call events.last - events.first\nend\n\nMosquito::Runner.stop wait: true\n\n\n\nputs\nprint \"Total events: #{event_count} | \"\nprint \"Rate: #{events.size.to_f./(events.last.-(events.first).to_f).round(2)} events/sec\"\nputs\n" }, { "path": "benchmark/jobs/emit_message_job.cr", "content": "class EmitMessageJob < Mosquito::QueuedJob\n PUBSUB_CHANNEL = \"benchmark:messages\"\n def perform\n number = Random::Secure.rand(100)\n Mosquito.backend.publish PUBSUB_CHANNEL, number.to_s\n end\nend\n" }, { "path": "demo/jobs/custom_serializers.cr", "content": "class CustomSerializersJob < Mosquito::QueuedJob\n param count : Int32\n\n def perform\n log \"deserialized: #{count}\"\n metadata.increment \"run_count\"\n end\n\n def deserialize_int32(raw : String) : Int32\n log \"using custom serialization: #{raw}\"\n\n raw.to_i32 * 10\n end\nend\n\nCustomSerializersJob.new(3).enqueue\nCustomSerializersJob.new(12).enqueue\nCustomSerializersJob.new(525_600).enqueue\n" }, { "path": "demo/jobs/periodically_puts.cr", "content": "class PeriodicallyPuts < Mosquito::PeriodicJob\n run_every 3.seconds\n\n queue_name :demo_queue\n\n def perform\n log \"Hello from PeriodicallyPuts\"\n\n # For integration testing\n metadata.increment \"run_count\"\n end\nend\n\n# Periodic jobs do not need to be enqueued, they are executed automatically on schedule.\n" }, { "path": "demo/jobs/queued_job.cr", "content": "class QueuedJob < Mosquito::QueuedJob\n param count : Int32\n\n queue_name :demo_queue\n\n def perform\n count.times do |i|\n log \"ohai #{i}\"\n end\n\n # For integration testing\n metadata.increment \"run_count\"\n end\nend\n\nQueuedJob.new(3).enqueue\n" }, { "path": "demo/jobs/rate_limited_job.cr", "content": "class RateLimitedJob < Mosquito::QueuedJob\n before do\n log self.class.rate_limit_stats\n end\n\n include Mosquito::RateLimiter\n\n throttle limit: 3, per: 10.seconds\n\n param count : Int32\n\n def perform\n log @@rate_limit_key\n end\nend\n\n15.times do\n RateLimitedJob.new(3).enqueue\nend\n" }, { "path": "demo/jobs/unique_job.cr", "content": "class UniqueJob < Mosquito::QueuedJob\n include Mosquito::UniqueJob\n\n unique_for 1.hour, key: [:user_id]\n\n param user_id : Int64\n param message : String\n\n def perform\n log \"Sending to user #{user_id}: #{message}\"\n metadata.increment \"run_count\"\n end\nend\n\n# First enqueue — accepted\nUniqueJob.new(user_id: 1_i64, message: \"hello\").enqueue\n\n# Duplicate user_id — suppressed by uniqueness lock\nUniqueJob.new(user_id: 1_i64, message: \"hello again\").enqueue\n\n# Different user_id — accepted\nUniqueJob.new(user_id: 2_i64, message: \"hello\").enqueue\n" }, { "path": "demo/run.cr", "content": "require \"../src/mosquito\"\n\nMosquito.configure do |settings|\n settings.backend_connection_string = ENV[\"REDIS_URL\"]? || \"redis://localhost:6379/3\"\n settings.idle_wait = 1.second\nend\n\nMosquito.configuration.backend.flush\n\nLog.setup do |c|\n backend = Log::IOBackend.new\n\n c.bind \"*\", :info, backend\n c.bind \"redis.*\", :warn, backend\n c.bind \"mosquito.*\", :info, backend\nend\n\nrequire \"./jobs/*\"\n\ndef expect_run_count(klass, expected)\n run_count = (klass.metadata[\"run_count\"]? || \"0\").to_i\n if run_count != expected\n raise \"Expected #{klass.name} to have run_count == #{expected}. But got #{run_count}\"\n else\n puts \"#{klass.name} was executed correctly.\"\n end\nend\n\nstopping = false\nSignal::INT.trap do\n if stopping\n puts \"SIGINT received again, crash-exiting.\"\n exit 1\n end\n\n Mosquito::Runner.stop\n stopping = true\nend\n\nMosquito::Runner.start(spin: false)\n\ncount = 0\nwhile count <= 19 && Mosquito::Runner.keep_running\n sleep 1.second\n count += 1\nend\n\nMosquito::Runner.stop(wait: true)\n\nputs \"End of demo.\"\nputs \"----------------------------------\"\nputs \"Checking integration test flags...\"\n\nexpect_run_count(PeriodicallyPuts, 7)\nexpect_run_count(QueuedJob, 1)\nexpect_run_count(CustomSerializersJob, 3)\nexpect_run_count(RateLimitedJob, 3)\nexpect_run_count(UniqueJob, 2)\n" }, { "path": "scripts/increment_version", "content": "#!/usr/bin/env crystal\n\nrequire \"yaml\"\nrequire \"option_parser\"\n\nshard_yml = \"shard.yml\"\n\nto_increment = \"none\"\n\nOptionParser.parse! do |p|\n p.banner = \"Usage: $0 -i \"\n p.on(\"-i field\", \"--increment=field\", \"Specifies the field to increment\") do |name|\n destination = name\n end\n p.on(\"-h\", \"--help\", \"Show this help\") { STDERR.puts p }\n p.invalid_option do |flag|\n STDERR.puts \"ERROR: #{flag} is not a valid option.\"\n STDERR.puts p\n exit(1)\n end\nend\n\ndocument = File.read shard_yml\nparsed = YAML.parse document\nversion = parsed[\"version\"].as_s\n\nmajor, minor, patch = version.split('.').map(&.to_i)\n\ncase to_increment\nwhen \"major\"\n major += 1\n minor = 0\n patch = 0\nwhen \"minor\"\n minor += 1\n patch = 0\nwhen \"patch\"\n patch += 1\nelse\n STDERR.puts \"No field to increment specified\" if to_increment == \"none\"\nend\n\nparsed[\"version\"] = \"#{major}.#{minor}.#{patch}\"\npp parsed.to_yaml\n" }, { "path": "scripts/lib/increment_version.sh", "content": "#!/bin/bash\nset -euo pipefail\nIFS=$'\\n\\t'\n\nprint_help () {\n cat <= 1.19'\n\nlicense: MIT\n\ntargets:\n demo:\n main: demo/run.cr\n\n mosquito:\n main: src/mosquito.cr\n\ndependencies:\n redis:\n github: jgaskins/redis\n version: ~> 0.7\n\ndevelopment_dependencies:\n minitest:\n github: ysbaddaden/minitest.cr\n version: ~> 1.6.0\n\n timecop:\n github: crystal-community/timecop.cr\n version: ~> 0.6.0\n" }, { "path": "spec/helpers/bare_base_class.cr", "content": "module Mosquito\n class Base\n # Testing wedge which wipes out the JobRun mapping for the\n # duration of the block.\n def self.bare_mapping(&block)\n scheduled_job_runs = @@scheduled_job_runs\n @@scheduled_job_runs = [] of PeriodicJobRun\n\n mapping = @@mapping\n @@mapping = {} of String => Job.class\n\n yield\n\n ensure\n @@mapping = mapping unless mapping.nil?\n @@scheduled_job_runs = scheduled_job_runs unless scheduled_job_runs.nil?\n end\n end\nend\n\n" }, { "path": "spec/helpers/configuration_helper.cr", "content": "module Mosquito\n class_setter configuration\n\n macro temp_config(**settings)\n original_config = {{ @type }}.configuration.dup\n was_validated = {{ @type }}.configuration.validated\n\n {% for key, value in settings %}\n {{ @type }}.configuration.{{ key }} = {{ value }}\n {% end %}\n {{ @type }}.configuration.validated = false\n\n {{ yield }}\n\n {{ @type }}.configuration = original_config\n {{ @type }}.configuration.validated = was_validated\n end\nend\n" }, { "path": "spec/helpers/global_helpers.cr", "content": "module TestHelpers\n extend self\n\n # Testing wedge which provides a clean slate to ensure tests\n # aren't dependent on each other.\n def clean_slate(&block)\n Mosquito::Base.bare_mapping do\n backend = Mosquito.backend\n backend.flush\n\n Mosquito::TestBackend::Queue.flush_paused_queues!\n TestingLogBackend.instance.clear\n PubSub.instance.clear\n yield\n end\n end\n\n def backend : Mosquito::Backend\n Mosquito.configuration.backend\n end\n\n def testing_redis_url : String\n ENV[\"REDIS_URL\"]? || \"redis://localhost:6379/3\"\n end\nend\n\nextend TestHelpers\n" }, { "path": "spec/helpers/logging_helper.cr", "content": "require \"log\"\n\nclass TestingLogBackend < Log::MemoryBackend\n def self.instance\n @@instance ||= new\n end\n\n def clear\n @entries.clear\n end\nend\n\nclass Minitest::Test\n def log_entries\n TestingLogBackend.instance.entries\n end\n\n def logs\n log_entries.map(&.message)\n end\n\n COLOR_STRIP = /\\e\\[\\d+(;\\d+)?m/\n\n private def logs_match(expected : Regex) : Bool\n log_entries\n .map(&.message)\n .map(&.gsub(COLOR_STRIP, \"\"))\n .any? { |entry| entry =~ expected }\n end\n\n private def logs_match(source : String, match_text : Regex) : Bool\n log_entries\n .select { |entry| entry.source == source }\n .map(&.message)\n .map(&.gsub(COLOR_STRIP, \"\"))\n .any? { |entry| entry =~ match_text }\n end\n\n def assert_logs_match(expected : String)\n assert_logs_match %r|#{expected}|\n end\n\n def assert_logs_match(expected : Regex)\n assert logs_match(expected), \"Expected to logs to include #{expected}. Logs contained: \\n#{log_entries.map(&.message).join(\"\\n\")}\"\n end\n\n def refute_logs_match(expected : String)\n refute_logs_match %r|#{expected}|\n end\n\n def refute_logs_match(expected : Regex)\n refute logs_match(expected), \"Expected to logs to not include #{expected}. Logs contained: \\n#{log_entries.map(&.message).join(\"\\n\")}\"\n end\n\n def assert_logs_match(source : String, expected : String)\n assert_logs_match source, %r|#{expected}|\n end\n\n def assert_logs_match(source : String, expected : Regex)\n assert logs_match(source, expected), \"Expected to logs to include #{expected}. Logs contained: \\n#{log_entries.map{|e| e.source + \" \" + e.message}.join(\"\\n\")}\"\n end\n\n def refute_logs_match(source : String, expected : String)\n refute_logs_match source, %r|#{expected}|\n end\n\n def refute_logs_match(source : String, expected : Regex)\n refute logs_match(source, expected), \"Expected to logs to not include #{expected}. Logs contained: \\n#{log_entries.map{|e| e.source + \" \" + e.message}.join(\"\\n\")}\"\n end\n\n def clear_logs\n TestingLogBackend.instance.clear\n end\nend\n\nLog.setup do |config|\n config.bind \"*\", :debug, TestingLogBackend.instance\n config.bind \"redis.*\", :warn, TestingLogBackend.instance\n config.bind \"mosquito.*\", :trace, TestingLogBackend.instance\nend\n" }, { "path": "spec/helpers/mock_coordinator.cr", "content": "class MockCoordinator < Mosquito::Runners::Coordinator\n getter schedule_count\n\n def initialize(queue_list : Mosquito::Runners::QueueList)\n super\n\n @schedule_count = 0\n end\n\n def only_if_coordinator : Nil\n if @always_coordinator\n yield\n else\n # yikes!\n # https://github.com/crystal-lang/crystal/issues/10399\n super do\n yield\n end\n end\n end\n\n def always_coordinator!(always = true)\n @always_coordinator = always\n end\n\n def schedule\n @schedule_count += 1\n super\n end\nend\n" }, { "path": "spec/helpers/mock_executor.cr", "content": "class MockExecutor < Mosquito::Runners::Executor\n setter work_unit : Mosquito::WorkUnit?\n\n def state=(state : Mosquito::Runnable::State)\n super\n end\n\n def run\n self.state = Mosquito::Runnable::State::Working\n end\n\n def stop(wait_group : WaitGroup = WaitGroup.new(1)) : WaitGroup\n self.state = Mosquito::Runnable::State::Stopping\n spawn do\n self.state = Mosquito::Runnable::State::Finished\n wait_group.done\n end\n wait_group\n end\n\n def receive_job\n job_pipeline.receive.job_run\n end\nend\n" }, { "path": "spec/helpers/mock_overseer.cr", "content": "class MockOverseer < Mosquito::Runners::Overseer\n property queue_list, coordinator, executors, work_handout, finished_notifier, dequeue_adapter\n\n def initialize\n @executor_count = Mosquito.configuration.executor_count\n @idle_wait = Mosquito.configuration.idle_wait\n @finished_notifier = Channel(Mosquito::WorkUnit?).new\n\n @queue_list = MockQueueList.new\n @coordinator = MockCoordinator.new queue_list\n @dequeue_adapter = Mosquito.configuration.dequeue_adapter\n @executors = [] of Mosquito::Runners::Executor\n @work_handout = Channel(Mosquito::WorkUnit).new\n @executors << build_executor\n observer.update_executor_list executors\n end\n\n def build_executor\n MockExecutor.new(self).as(Mosquito::Runners::Executor)\n end\nend\n" }, { "path": "spec/helpers/mock_queue_list.cr", "content": "class MockQueueList < Mosquito::Runners::QueueList\n setter state\n\n def discovered_queues : Array(Mosquito::Queue)\n @discovered_queues\n end\n\n def stop(wait_group : WaitGroup = WaitGroup.new(1)) : WaitGroup\n self.state = Mosquito::Runnable::State::Stopping\n spawn do\n self.state = Mosquito::Runnable::State::Finished\n wait_group.done\n end\n wait_group\n end\nend\n" }, { "path": "spec/helpers/mocks.cr", "content": "# A global place for global mocks\n\nmodule PerformanceCounter\n def perform\n self.class.performed!\n end\n\n macro included\n class_getter performances = 0\n\n def self.performed!\n @@performances += 1\n end\n\n def self.reset_performance_counter!\n @@performances = 0\n end\n end\nend\n\nclass JobWithPerformanceCounter < Mosquito::Job\n include PerformanceCounter\nend\n\nclass PeriodicTestJob < Mosquito::PeriodicJob\n include PerformanceCounter\nend\n\nclass QueuedTestJob < Mosquito::QueuedJob\n include PerformanceCounter\nend\n\nclass QueueHookedTestJob < Mosquito::QueuedJob\n include PerformanceCounter\n\n property fail_before_hook = false\n property before_hook_ran = false\n property after_hook_ran = false\n property passed_job_config : Mosquito::JobRun? = nil\n\n before_enqueue do\n self.before_hook_ran = true\n self.passed_job_config = job\n\n if fail_before_hook\n false\n else\n true\n end\n end\n\n after_enqueue do\n self.after_hook_ran = true\n self.passed_job_config = job\n end\nend\n\n\nclass PassingJob < QueuedTestJob\n def perform\n super\n true\n end\nend\n\nclass FailingJob < QueuedTestJob\n property fail_with_exception = false\n property fail_with_retry = true\n property exception_message = \"this is the reason #{name} failed\"\n\n include PerformanceCounter\n\n def perform\n super\n\n case\n when fail_with_exception\n raise exception_message\n when ! fail_with_retry\n fail exception_message, retry: false\n else\n fail exception_message\n end\n end\nend\n\nclass CustomRescheduleIntervalJob < PassingJob\n def reschedule_interval(retry_count)\n 4.seconds\n end\nend\n\nclass NonReschedulableFailingJob < FailingJob\n def rescheduleable?\n false\n end\nend\n\nclass NotImplementedJob < Mosquito::Job\nend\n\nclass JobWithConfig < PassingJob\n getter config = {} of String => String\n\n def vars_from(config : Hash(String, String))\n @config = config\n end\nend\n\nclass JobWithNoParams < Mosquito::QueuedJob\n def perform\n log \"no param job performed\"\n end\nend\n\nclass JobWithHooks < Mosquito::QueuedJob\n param should_fail : Bool\n\n before do\n log \"Before Hook Executed\"\n end\n\n after do\n log \"After Hook Executed\"\n end\n\n before do\n log \"2nd Before Hook Executed\"\n fail if should_fail\n end\n\n after do\n log \"2nd After Hook Executed\"\n end\n\n def perform\n log \"Perform Executed\"\n end\nend\n\nclass EchoJob < Mosquito::QueuedJob\n queue_name \"io_queue\"\n\n param text : String\n\n def perform\n log text\n end\nend\n\nclass MonthlyJob < Mosquito::PeriodicJob\n run_every 1.month\n\n def perform\n log \"monthly job_run ran\"\n end\nend\n\nclass RateLimitedJob < Mosquito::QueuedJob\n include Mosquito::RateLimiter\n\n throttle key: \"rate_limit\", limit: Int32::MAX\n\n param should_fail : Bool = false\n param increment : Int32 = 1\n\n before do\n log \"Before Hook Executed\"\n fail if should_fail\n end\n\n def perform\n log \"Performed\"\n end\n\n def increment_run_count_by\n increment\n end\nend\n\nclass PreemptingJob < Mosquito::QueuedJob\n include PerformanceCounter\n property preempt_until : Time? = nil\n\n before do\n preempt \"test preemption\", until: preempt_until\n end\nend\n\nclass NonReschedulablePreemptingJob < Mosquito::QueuedJob\n include PerformanceCounter\n\n before do\n preempt \"not reschedulable\"\n end\n\n def rescheduleable? : Bool\n false\n end\nend\n\nclass SleepyJob < Mosquito::QueuedJob\n class_property should_sleep = true\n\n def perform\n while self.class.should_sleep\n sleep 0.01.seconds\n end\n end\nend\n\nclass SecondRateLimitedJob < Mosquito::QueuedJob\n include Mosquito::RateLimiter\n\n throttle key: \"rate_limit\", limit: Int32::MAX\n\n def perform\n end\nend\n\nclass UniqueTestJob < Mosquito::QueuedJob\n include Mosquito::UniqueJob\n\n unique_for 1.hour\n\n param user_id : Int64\n param email_type : String\n\n def perform\n log \"UniqueTestJob performed\"\n end\nend\n\nclass UniqueWithKeyJob < Mosquito::QueuedJob\n include Mosquito::UniqueJob\n\n unique_for 30.seconds, key: [:user_id]\n\n param user_id : Int64\n param message : String\n\n def perform\n log \"UniqueWithKeyJob performed\"\n end\nend\n\nclass UniqueNoParamsJob < Mosquito::QueuedJob\n include Mosquito::UniqueJob\n\n unique_for 1.minute\n\n def perform\n log \"UniqueNoParamsJob performed\"\n end\nend\n\nMosquito::Base.register_job_mapping \"job_with_config\", JobWithConfig\nMosquito::Base.register_job_mapping \"job_with_performance_counter\", JobWithPerformanceCounter\nMosquito::Base.register_job_mapping \"failing_job\", FailingJob\nMosquito::Base.register_job_mapping \"non_reschedulable_failing_job\", NonReschedulableFailingJob\nMosquito::Base.register_job_mapping \"preempting_job\", PreemptingJob\nMosquito::Base.register_job_mapping \"non_reschedulable_preempting_job\", NonReschedulablePreemptingJob\n\ndef job_run_config\n {\n \"year\" => \"1752\",\n \"name\" => \"the year september lost 12 days\",\n }\nend\n\ndef create_job_run(type = \"job_with_config\", config = job_run_config)\n Mosquito::JobRun.new(type).tap do |job_run|\n job_run.config = config\n job_run.store\n end\nend\n" }, { "path": "spec/helpers/null_dequeue_adapter.cr", "content": "# A test adapter that always returns nil, simulating empty queues.\nclass NullDequeueAdapter < Mosquito::DequeueAdapter\n getter dequeue_count = 0\n\n def dequeue(queue_list : Mosquito::Runners::QueueList) : Mosquito::WorkUnit?\n @dequeue_count += 1\n nil\n end\nend\n" }, { "path": "spec/helpers/pub_sub.cr", "content": "module Mosquito::Observability::Publisher\n @[AlwaysInline]\n def publish(data : NamedTuple)\n metrics do\n Log.debug { \"Publishing #{data} to #{@publish_context.originator}\" }\n PubSub.instance.capture_message(@publish_context.originator, data.to_json)\n end\n end\nend\n\nclass PubSub\n def self.instance\n @@instance ||= new\n end\n\n def self.eavesdrop : Array(Mosquito::Backend::BroadcastMessage)\n instance.listen\n yield\n instance.messages\n ensure\n instance.stop_listening\n end\n\n getter messages = [] of Mosquito::Backend::BroadcastMessage\n\n def initialize\n @listening = false\n end\n\n def listen\n @listening = true\n end\n\n def stop_listening\n @listening = false\n end\n\n def capture_message(originator : String, message : String)\n if @listening\n @messages << Mosquito::Backend::BroadcastMessage.new(originator, message)\n end\n end\n\n delegate clear, to: @messages\n\n module Helpers\n delegate eavesdrop, to: PubSub\n\n def assert_message_received(matcher : Regex) : Nil\n found = PubSub.instance.messages.find do |message|\n matcher === message.message\n end\n\n assert found, \"Expected to find a message matching #{matcher.inspect}, but only found: #{PubSub.instance.messages.map(&.message).inspect}\"\n end\n end\nend\n" }, { "path": "spec/helpers/spy_dequeue_adapter.cr", "content": "# A test adapter that tracks which queues were checked, in order.\nclass SpyDequeueAdapter < Mosquito::DequeueAdapter\n getter checked_queues = [] of String\n\n def dequeue(queue_list : Mosquito::Runners::QueueList) : Mosquito::WorkUnit?\n queue_list.queues.each do |q|\n @checked_queues << q.name\n if job_run = q.dequeue\n return Mosquito::WorkUnit.of(job_run, from: q)\n end\n end\n end\nend\n" }, { "path": "spec/mosquito/api/executor_config_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe \"Mosquito::Api::ExecutorConfig\" do\n describe \"global executor count\" do\n it \"returns nil when no override is stored\" do\n clean_slate do\n result = Mosquito::Api::ExecutorConfig.stored_executor_count\n assert_nil result\n end\n end\n\n it \"round-trips a global executor count\" do\n clean_slate do\n Mosquito::Api::ExecutorConfig.store_executor_count(8)\n result = Mosquito::Api::ExecutorConfig.stored_executor_count\n assert_equal 8, result\n end\n end\n\n it \"clears the global executor count\" do\n clean_slate do\n Mosquito::Api::ExecutorConfig.store_executor_count(8)\n Mosquito::Api::ExecutorConfig.clear_executor_count\n\n result = Mosquito::Api::ExecutorConfig.stored_executor_count\n assert_nil result\n end\n end\n end\n\n describe \"per-overseer executor count\" do\n it \"returns nil when no per-overseer override is stored\" do\n clean_slate do\n result = Mosquito::Api::ExecutorConfig.stored_executor_count(\"gpu-worker-1\")\n assert_nil result\n end\n end\n\n it \"round-trips a per-overseer executor count\" do\n clean_slate do\n Mosquito::Api::ExecutorConfig.store_executor_count(2, \"gpu-worker-1\")\n\n result = Mosquito::Api::ExecutorConfig.stored_executor_count(\"gpu-worker-1\")\n assert_equal 2, result\n\n # Global is unaffected.\n global = Mosquito::Api::ExecutorConfig.stored_executor_count\n assert_nil global\n end\n end\n\n it \"clears per-overseer without affecting global\" do\n clean_slate do\n Mosquito::Api::ExecutorConfig.store_executor_count(8)\n Mosquito::Api::ExecutorConfig.store_executor_count(2, \"gpu-worker-1\")\n\n Mosquito::Api::ExecutorConfig.clear_executor_count(\"gpu-worker-1\")\n\n per_overseer = Mosquito::Api::ExecutorConfig.stored_executor_count(\"gpu-worker-1\")\n assert_nil per_overseer\n\n global = Mosquito::Api::ExecutorConfig.stored_executor_count\n assert_equal 8, global\n end\n end\n end\n\n describe \".resolve\" do\n it \"returns nil when nothing is stored\" do\n clean_slate do\n result = Mosquito::Api::ExecutorConfig.resolve\n assert_nil result\n end\n end\n\n it \"returns the global count when no overseer_id is given\" do\n clean_slate do\n Mosquito::Api::ExecutorConfig.store_executor_count(8)\n result = Mosquito::Api::ExecutorConfig.resolve\n assert_equal 8, result\n end\n end\n\n it \"prefers per-overseer over global\" do\n clean_slate do\n Mosquito::Api::ExecutorConfig.store_executor_count(8)\n Mosquito::Api::ExecutorConfig.store_executor_count(2, \"gpu-worker-1\")\n\n result = Mosquito::Api::ExecutorConfig.resolve(\"gpu-worker-1\")\n assert_equal 2, result\n end\n end\n\n it \"falls back to global when per-overseer is not set\" do\n clean_slate do\n Mosquito::Api::ExecutorConfig.store_executor_count(8)\n\n result = Mosquito::Api::ExecutorConfig.resolve(\"gpu-worker-1\")\n assert_equal 8, result\n end\n end\n end\n\n describe \"instance methods\" do\n it \"delegates to class-level helpers\" do\n clean_slate do\n config = Mosquito::Api::ExecutorConfig.instance\n\n config.update(10)\n assert_equal 10, config.executor_count\n\n config.update(3, overseer_id: \"worker-1\")\n assert_equal 3, config.executor_count(overseer_id: \"worker-1\")\n\n config.clear(overseer_id: \"worker-1\")\n assert_nil config.executor_count(overseer_id: \"worker-1\")\n\n config.clear\n assert_nil config.executor_count\n end\n end\n end\nend\n\ndescribe \"Mosquito::Api executor count convenience methods\" do\n it \"reads and writes global executor count\" do\n clean_slate do\n Mosquito::Api.set_executor_count(12)\n assert_equal 12, Mosquito::Api.executor_count\n end\n end\n\n it \"reads and writes per-overseer executor count\" do\n clean_slate do\n Mosquito::Api.set_executor_count(4, overseer_id: \"gpu-worker-1\")\n assert_equal 4, Mosquito::Api.executor_count(overseer_id: \"gpu-worker-1\")\n\n # Global unaffected.\n assert_nil Mosquito::Api.executor_count\n end\n end\nend\n" }, { "path": "spec/mosquito/api/executor_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe Mosquito::Api::Executor do\n let(executor_pipeline) { Channel(Mosquito::WorkUnit).new }\n let(finished_notifier) { Channel(Mosquito::WorkUnit?).new }\n let(job) { QueuedTestJob.new }\n let(job_run : Mosquito::JobRun) { job.enqueue }\n\n let(overseer) { MockOverseer.new }\n let(executor) { MockExecutor.new overseer.as(Mosquito::Runners::Overseer) }\n let(api) { Mosquito::Api::Executor.new executor.object_id.to_s }\n let(observer) { Mosquito::Observability::Executor.new executor }\n\n describe \"publish context\" do\n it \"includes object_id\" do\n assert_equal \"executor:#{executor.object_id}\", observer.publish_context.context\n end\n\n it \"is nested under the overseer publish context\" do\n assert_equal \"mosquito:overseer:#{overseer.object_id}:executor:#{executor.object_id}\", observer.publish_context.originator\n end\n end\n\n it \"can read the current job and queue after being started, and clears it after\" do\n Mosquito::Base.register_job_mapping job.class.name.underscore, job.class\n job_run.store\n job_run.build_job\n\n observer.execute job_run, job.class.queue do\n assert_equal job_run.id, api.current_job\n assert_equal job.class.queue.name, api.current_job_queue\n end\n\n assert api.current_job.nil?\n assert api.current_job_queue.nil?\n end\n\n it \"returns a nil heartbeat before the executor has triggered it\" do\n assert api.heartbeat.nil?\n end\n\n it \"returns a valid heartbeat\" do\n now = Time.utc\n Timecop.freeze now do\n observer.heartbeat!\n end\n\n # the heartbeat is stored as a unix epoch without millis\n assert_equal now.at_beginning_of_second, api.heartbeat\n end\n\n it \"doesn't publish a heartbeat when metrics are disabled\" do\n now = Time.utc\n\n Timecop.freeze now do\n executor.observer.heartbeat!\n end\n\n later = Time.utc + 1.minute\n Mosquito.temp_config(publish_metrics: false) do\n Timecop.freeze later do\n executor.observer.heartbeat!\n end\n end\n\n api = Mosquito::Api::Executor.new executor.object_id.to_s\n assert_equal now.at_beginning_of_second, api.heartbeat\n end\n\n it \"publishes job started/finished events\" do\n job_run.store\n job_run.build_job\n\n eavesdrop do\n observer.execute job_run, job.class.queue do\n end\n end\n\n assert_message_received /job-started/\n assert_message_received /job-finished/\n end\n\n it \"measures and records average job duration\" do\n job_run.store\n job_run.build_job\n\n # 100x the sleep duration below\n Timecop.scale(100) do\n observer.execute job_run, job.class.queue do\n sleep 0.01.seconds\n end\n end\n\n average_key = observer.average_key(job_run.type)\n average = Mosquito.backend.average(average_key)\n Mosquito.backend.delete average_key\n # assert that something > 0 comes back from the average.\n # backend tests cover calculating the average itself.\n assert average > 0\n end\nend\n" }, { "path": "spec/mosquito/api/job_run_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe Mosquito::Api::JobRun do\n # the job run timestamps are stored as a unix epoch with millis, so nanosecond precision is lost.\n def at_beginning_of_millisecond(time)\n time - (time.nanosecond.nanoseconds) + (time.millisecond.milliseconds)\n end\n\n getter job : QueuedTestJob { QueuedTestJob.new }\n getter job_run : Mosquito::JobRun { job.build_job_run }\n getter api : Mosquito::Api::JobRun { Mosquito::Api::JobRun.new job_run.id }\n\n it \"can look up a job run\" do\n job_run.store\n assert api.found?\n end\n\n it \"can look up a job run that doesn't exist\" do\n api = Mosquito::Api::JobRun.new \"not_a_real_id\"\n refute api.found?\n end\n\n it \"can retrieve the job parameters\" do\n job_run = JobWithHooks.new(should_fail: false).build_job_run\n job_run.store\n api = Mosquito::Api::JobRun.new job_run.id\n assert_equal \"false\", api.runtime_parameters[\"should_fail\"]\n end\n\n it \"can retrieve the job type\" do\n job_run.store\n assert_equal job.class.name.underscore, api.type\n end\n\n it \"can retrieve the enqueue time\" do\n now = Time.utc\n Timecop.freeze now do\n job_run.store\n end\n\n expected_time = at_beginning_of_millisecond now\n assert_equal expected_time, api.enqueue_time\n end\n\n it \"can retrieve the retry count\" do\n job_run.store\n assert_equal 0, api.retry_count\n end\n\n it \"can retrieve the started at timestamp\" do\n now = at_beginning_of_millisecond Time.utc\n job_run = create_job_run\n Timecop.freeze now do\n job_run.run\n end\n\n api = Mosquito::Api::JobRun.new(job_run.id)\n assert_equal now, api.started_at\n end\n\n it \"can retrieve the finished_at timestamp\" do\n now = at_beginning_of_millisecond Time.utc\n job_run = create_job_run\n Timecop.freeze now do\n job_run.run\n end\n\n api = Mosquito::Api::JobRun.new(job_run.id)\n assert_equal now, api.finished_at\n end\nend\n" }, { "path": "spec/mosquito/api/overseer_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe Mosquito::Api::Overseer do\n let(:overseer) { MockOverseer.new }\n let(:api) { Mosquito::Api::Overseer.new(overseer.object_id.to_s) }\n let(:observer) { Observability::Overseer.new(overseer) }\n let(:executor) { MockExecutor.new(overseer.as(Mosquito::Runners::Overseer))}\n\n describe \"publish context\" do\n it \"includes object_id\" do\n assert_equal \"overseer:#{overseer.object_id}\", observer.publish_context.context\n assert_equal \"mosquito:overseer:#{overseer.object_id}\", observer.publish_context.originator\n end\n end\n\n it \"allows fetching a list of executors\" do\n assert_equal 1, api.executors.size\n observer.update_executor_list([executor, executor])\n assert_equal 2, api.executors.size\n end\n\n it \"allows getting the latest heartbeat\" do\n assert_nil api.last_heartbeat\n observer.heartbeat\n assert_instance_of Time, api.last_heartbeat\n end\n\n it \"publishes the startup event\" do\n eavesdrop do\n observer.starting\n end\n assert_message_received /started/\n end\n\n it \"publishes the stopping event\" do\n eavesdrop do\n observer.stopping\n end\n assert_message_received /stopped/\n end\n\n it \"publishes the stopped event\" do\n eavesdrop do\n observer.stopped\n end\n assert_message_received /exited/\n end\n\n it \"publishes an event when an executor dies\" do\n eavesdrop do\n observer.executor_died executor\n end\n assert_message_received /died/\n end\n\n it \"publishes an event when an executor is created\" do\n eavesdrop do\n observer.executor_created executor\n end\n assert_message_received /created/\n end\nend\n" }, { "path": "spec/mosquito/api/periodic_job_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe Mosquito::Api::PeriodicJob do\n getter interval : Time::Span = 2.minutes\n\n describe \"publish context\" do\n it \"includes the periodic job name\" do\n clean_slate do\n Mosquito::Base.register_job_interval PeriodicTestJob, interval: interval\n job_run = Mosquito::Base.scheduled_job_runs.first\n observer = job_run.observer\n assert_equal \"periodic_job:PeriodicTestJob\", observer.publish_context.context\n assert_equal \"mosquito:periodic_job:PeriodicTestJob\", observer.publish_context.originator\n end\n end\n end\n\n it \"can fetch a list of periodic jobs\" do\n clean_slate do\n Mosquito::Base.register_job_interval PeriodicTestJob, interval: interval\n periodic_jobs = Mosquito::Api::PeriodicJob.all\n assert_equal 1, periodic_jobs.size\n assert_equal \"PeriodicTestJob\", periodic_jobs.first.name\n assert_equal interval, periodic_jobs.first.interval\n end\n end\n\n it \"returns nil for last_executed_at when never run\" do\n clean_slate do\n Mosquito::Base.register_job_interval PeriodicTestJob, interval: interval\n periodic_jobs = Mosquito::Api::PeriodicJob.all\n assert_nil periodic_jobs.first.last_executed_at\n end\n end\n\n it \"returns the last executed time after a job runs\" do\n now = Time.utc.at_beginning_of_second\n clean_slate do\n Mosquito::Base.register_job_interval PeriodicTestJob, interval: interval\n job_run = Mosquito::Base.scheduled_job_runs.first\n\n Timecop.freeze(now) do\n job_run.try_to_execute\n end\n\n periodic_jobs = Mosquito::Api::PeriodicJob.all\n assert_equal now, periodic_jobs.first.last_executed_at\n end\n end\n\n it \"publishes an event when a periodic job is enqueued\" do\n now = Time.utc.at_beginning_of_second\n clean_slate do\n Mosquito::Base.register_job_interval PeriodicTestJob, interval: interval\n\n eavesdrop do\n Timecop.freeze(now) do\n Mosquito::Base.scheduled_job_runs.first.try_to_execute\n end\n end\n\n assert_message_received /enqueued/\n end\n end\nend\n" }, { "path": "spec/mosquito/api/publisher_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe Mosquito::Api::Publisher do\n let(executor_pipeline) { Channel(Mosquito::WorkUnit).new }\n let(finished_notifier) { Channel(Mosquito::WorkUnit?).new }\n let(job) { QueuedTestJob.new }\n let(job_run : Mosquito::JobRun) { job.enqueue }\n\n let(overseer) { MockOverseer.new }\n let(executor) { MockExecutor.new overseer.as(Mosquito::Runners::Overseer) }\n let(api) { Mosquito::Api::Executor.new executor.object_id.to_s }\n let(observer) { Mosquito::Observability::Executor.new executor }\n\n it \"doesn't publish events when metrics are disabled\" do\n job_run.store\n job_run.build_job\n\n PubSub.instance.clear\n published_messages = eavesdrop do\n Mosquito.temp_config(publish_metrics: false) do\n observer.execute job_run, job.class.queue do\n end\n end\n end\n\n assert_equal 0, published_messages.size\n end\nend\n" }, { "path": "spec/mosquito/api/queue_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe Mosquito::Api::Queue do\n let(job_classes) {\n [QueuedTestJob, PassingJob, FailingJob, QueueHookedTestJob]\n }\n let(queued_test_job) { QueuedTestJob.new }\n let(passing_job) { PassingJob.new }\n let(queue : Mosquito::Queue) { queued_test_job.class.queue }\n let(observer : Mosquito::Observability::Queue) { queue.observer }\n\n describe \"publish context\" do\n it \"includes the queue name\" do\n assert_equal \"queue:queued_test_job\", observer.publish_context.context\n assert_equal \"mosquito:queue:queued_test_job\", observer.publish_context.originator\n end\n end\n\n it \"can fetch a list of current queues\" do\n clean_slate do\n queued_test_job.enqueue\n passing_job.enqueue\n expected_queues = [\"queued_test_job\", \"passing_job\"].sort\n queues = Mosquito::Api::Queue.all\n assert_equal 2, queues.size\n assert_equal expected_queues, queues.map(&.name).sort\n end\n end\n\n it \"can fetch the size of a queue\" do\n clean_slate do\n job_classes.map(&.new).each(&.enqueue)\n queues = Mosquito::Api::Queue.all\n queues.each do |queue|\n assert_equal 1, queue.size\n end\n end\n end\n\n it \"can fetch the size details of a queue\" do\n clean_slate do\n job_classes.map(&.new).each(&.enqueue)\n queues = Mosquito::Api::Queue.all\n sizes = queues.map(&.size_details)\n sizes.each do |size|\n assert_equal 1, size[\"waiting\"]\n assert_equal 0, size[\"scheduled\"]\n assert_equal 0, size[\"pending\"]\n assert_equal 0, size[\"dead\"]\n end\n end\n end\n\n it \"can fetch job runs from a queue\" do\n clean_slate do\n job_classes.each do |job_class|\n job = job_class.new\n job.enqueue\n api = Mosquito::Api::Queue.new job_class.queue.name\n job_runs = api.waiting_job_runs\n assert_equal 1, job_runs.size\n assert_equal job.class.name.underscore, job_runs.first.type\n end\n end\n end\n\n it \"publishes an event when a job is enqueued\" do\n eavesdrop do\n queued_test_job.enqueue\n end\n assert_message_received /enqueued/\n end\n\n it \"publishes an event when a job is enqueued for later\" do\n eavesdrop do\n queued_test_job.enqueue(60.seconds.from_now)\n end\n assert_message_received /enqueued/\n end\n\n it \"publishes an event when a job is dequeued\" do\n clean_slate do\n queued_test_job.enqueue\n\n eavesdrop do\n queue.dequeue\n end\n end\n\n assert_message_received /dequeued/\n end\n\n it \"publishes an event when a job is rescheduled\" do\n clean_slate do\n job_run = queued_test_job.build_job_run\n\n eavesdrop do\n queue.enqueue job_run\n queue.reschedule job_run, 60.seconds.from_now\n end\n end\n\n assert_message_received /rescheduled/\n end\n\n it \"publishes an event when a job is forgotten\" do\n clean_slate do\n job_run = queued_test_job.build_job_run\n\n eavesdrop do\n queue.forget job_run\n end\n end\n\n assert_message_received /forgotten/\n end\n\n it \"publishes an event when a job is banished\" do\n clean_slate do\n job_run = queued_test_job.build_job_run\n\n eavesdrop do\n queue.banish job_run\n end\n end\n\n assert_message_received /banished/\n end\nend\n" }, { "path": "spec/mosquito/api_spec.cr", "content": "require \"../spec_helper\"\n\ndescribe Mosquito::Api do\n let(queued_test_job) { QueuedTestJob.new }\n let(passing_job) { PassingJob.new }\n\n it \"can fetch a list of queues\" do\n clean_slate do\n queued_test_job.enqueue\n passing_job.enqueue\n queues = Mosquito::Api.list_queues\n assert_equal 2, queues.size\n queue_names = queues.map(&.name)\n assert_includes queue_names, queued_test_job.class.queue.name\n assert_includes queue_names, passing_job.class.queue.name\n end\n end\nend\n" }, { "path": "spec/mosquito/backend/deleting_spec.cr", "content": "\nrequire \"../../spec_helper\"\n\ndescribe \"Backend deleting\" do\n getter queue_name : String { \"test#{rand(1000)}\" }\n getter queue : Mosquito::Backend::Queue { backend.queue queue_name }\n\n getter sample_data do\n { \"test\" => \"#{rand(1000)}\" }\n end\n\n getter key : String { \"key-#{rand 1000}\" }\n getter field : String { \"field-#{rand 1000}\" }\n\n getter job_run : Mosquito::JobRun { Mosquito::JobRun.new(\"mock_job_run\") }\n\n describe \"delete\" do\n it \"deletes immediately\" do\n backend.store key, sample_data\n backend.delete key\n blank_data = {} of String => String\n assert_equal blank_data, backend.retrieve(key)\n end\n\n it \"deletes at a ttl\" do\n # Since redis is outside the control of timecop, this test is just showing\n # that #delete can be called with a ttl and we trust redis to do it's job.\n backend.store key, sample_data\n backend.delete key, in: 1.second\n end\n end\n\n describe \"self.flush\" do\n it \"wipes the database\" do\n clean_slate do\n backend.set key, field, \"1\"\n backend.flush\n assert_nil backend.get key, field\n end\n end\n end\n\n describe \"#flush\" do\n it \"empties the queues\" do\n clean_slate do\n # add a job_run to waiting\n queue.enqueue job_run\n\n # add a job_run to scheduled\n queue.schedule job_run, at: 1.second.from_now\n\n # move a job_run to pending\n pending_job_run = queue.dequeue\n\n # add a job_run to the dead queue\n queue.terminate job_run\n\n queue.flush\n empty_set = [] of String\n\n assert_equal empty_set, queue.list_waiting\n assert_equal empty_set, queue.list_scheduled\n assert_equal empty_set, queue.list_pending\n assert_equal empty_set, queue.list_dead\n end\n end\n\n it \"but doesn't truncate the database\" do\n clean_slate do\n backend.set key, field, \"value\"\n queue.flush\n assert_equal \"value\", backend.get key, field\n end\n end\n end\nend\n" }, { "path": "spec/mosquito/backend/executor_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe Mosquito::Backend do\n getter key : String { \"key-#{rand 1000}\" }\n\n it \"can calculate an average\" do\n backend.average_push key, 10\n backend.average_push key, 20\n backend.average_push key, 30\n\n assert_equal 20, backend.average key\n end\n\n it \"correctly rolls off old values for the window size\" do\n backend.average_push key, 10, window_size: 3\n backend.average_push key, 20, window_size: 3\n backend.average_push key, 30, window_size: 3\n backend.average_push key, 40, window_size: 3\n backend.average_push key, 50, window_size: 3\n\n assert_equal 40, backend.average key\n end\nend\n" }, { "path": "spec/mosquito/backend/expiring_list_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe Mosquito::RedisBackend do\n describe \"expiring lists\" do\n it \"can add an item to a list\" do\n now = Time.utc\n key = \"exp-list-test\"\n items = [\"item1\", \"item2\", \"item3\"]\n\n redis_backend = backend.as(Mosquito::RedisBackend)\n\n Timecop.freeze now do\n redis_backend.expiring_list_push key, items[0]\n end\n\n Timecop.freeze now + 1.second do\n redis_backend.expiring_list_push key, items[1]\n end\n\n Timecop.freeze now + 2.seconds do\n redis_backend.expiring_list_push key, items[2]\n end\n\n found_items = redis_backend.expiring_list_fetch(key, now + 1.second)\n assert_equal [items[2]], found_items\n end\n end\nend\n" }, { "path": "spec/mosquito/backend/hash_storage_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe \"Backend hash storage\" do\n getter sample_data : Hash(String,String) { { \"test\" => \"#{rand(1000)}\" } }\n getter key : String { \"key-#{rand 1000}\" }\n getter field : String { \"field-#{rand 1000}\" }\n\n it \"can store and retrieve\" do\n backend.store key, sample_data\n retrieved_data = backend.retrieve key\n assert_equal sample_data, retrieved_data\n end\n\n describe \"self.get and set\" do\n it \"sets and retrieves a value from a hash\" do\n backend.set(key, field, \"truth\")\n assert_equal \"truth\", backend.get(key, field)\n end\n end\n\n describe \"self.increment\" do\n it \"adds one\" do\n backend.set(key, field, \"1\")\n assert_equal 2, backend.increment(key, field)\n end\n\n it \"can add arbitrary values\" do\n backend.set(key, field, \"1\")\n assert_equal 4, backend.increment(key, field, by: 3)\n end\n end\nend\n" }, { "path": "spec/mosquito/backend/inspection_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe \"Backend inspection\" do\n getter backend_name : String { \"test#{rand(1000)}\" }\n getter queue : Mosquito::Backend::Queue { backend.queue backend_name }\n\n getter job : QueuedTestJob { QueuedTestJob.new }\n getter job_run : Mosquito::JobRun { Mosquito::JobRun.new(\"mock_job_run\") }\n\n describe \"size\" do\n def fill_queues\n # add to waiting queue\n queue.enqueue job_run\n queue.enqueue job_run\n\n # move 1 from waiting to pending queue\n pending_t = queue.dequeue\n\n # add to scheduled queue\n queue.schedule job_run, at: 1.second.from_now\n\n # add to dead queue\n queue.terminate job_run\n end\n\n it \"returns the size of the named q\" do\n clean_slate do\n fill_queues\n assert_equal 4, queue.size\n end\n end\n\n it \"returns the size of the named q (without the dead_q)\" do\n clean_slate do\n fill_queues\n assert_equal 3, queue.size(include_dead: false)\n end\n end\n end\n\n describe \"list\" do\n it \"can list the waiting jobs\" do\n clean_slate do\n expected_job_runs = Array(Mosquito::JobRun).new(3) { Mosquito::JobRun.new(\"mock_job_run\") }\n expected_job_runs.each { |job_run| queue.enqueue job_run }\n expected_job_run_ids = expected_job_runs.map { |job_run| job_run.id }.sort\n\n actual_job_runs = queue.list_waiting.sort\n assert_equal 3, actual_job_runs.size\n\n assert_equal expected_job_run_ids, actual_job_runs\n end\n end\n\n it \"can list the scheduled jobs\" do\n clean_slate do\n expected_job_runs = Array(Mosquito::JobRun).new(3) { Mosquito::JobRun.new(\"mock_job_run\") }\n expected_job_runs.each { |job_run| queue.schedule job_run, at: 1.second.from_now }\n expected_job_run_ids = expected_job_runs.map { |job_run| job_run.id }.sort\n\n actual_job_runs = queue.list_scheduled.sort\n assert_equal 3, actual_job_runs.size\n\n assert_equal expected_job_run_ids, actual_job_runs\n end\n end\n\n it \"can list the pending jobs\" do\n clean_slate do\n expected_job_runs = Array(Mosquito::JobRun).new(3) { Mosquito::JobRun.new(\"mock_job_run\").tap(&.store) }\n\n expected_job_runs.each { |job_run| queue.enqueue job_run }\n expected_job_run_ids = 3.times.map { queue.dequeue.not_nil!.id }.to_a.sort\n\n actual_job_runs = queue.list_pending.sort\n assert_equal 3, actual_job_runs.size\n\n assert_equal expected_job_run_ids, actual_job_runs\n end\n end\n\n it \"can list the dead jobs\" do\n clean_slate do\n expected_job_runs = Array(Mosquito::JobRun).new(3) { Mosquito::JobRun.new(\"mock_job_run\") }\n expected_job_runs.each { |job_run| queue.terminate job_run }\n expected_job_run_ids = expected_job_runs.map { |job_run| job_run.id }.sort\n\n actual_job_runs = queue.list_dead.sort\n assert_equal 3, actual_job_runs.size\n\n assert_equal expected_job_run_ids, actual_job_runs\n end\n end\n end\nend\n" }, { "path": "spec/mosquito/backend/lock_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe \"distributed locking\" do\n getter key : String { \"testing:backend:lock\" }\n getter instance_id : String { \"abcd\" }\n getter ttl : Time::Span { 1.second }\n\n def ensure_unlock(&block)\n yield\n Mosquito.backend.delete key\n end\n\n it \"locks\" do\n ensure_unlock do\n got_it = Mosquito.backend.lock? key, instance_id, ttl\n assert got_it\n end\n end\n\n it \"doesn't double lock\" do\n ensure_unlock do\n hold = Mosquito.backend.lock? key, \"abcd\", ttl\n assert hold\n\n try = Mosquito.backend.lock? key, \"wxyz\", ttl\n refute try\n end\n end\n\n it \"locks after unlock\" do\n ensure_unlock do\n hold = Mosquito.backend.lock? key, \"abcd\", ttl\n assert hold\n\n Mosquito.backend.unlock key, instance_id\n\n try = Mosquito.backend.lock? key, \"wxyz\", ttl\n assert try\n end\n end\n\n it \"renews a lock held by the same instance\" do\n ensure_unlock do\n hold = Mosquito.backend.lock? key, instance_id, ttl\n assert hold\n\n renewed = Mosquito.backend.renew_lock? key, instance_id, ttl\n assert renewed\n end\n end\n\n it \"doesn't renew a lock held by another instance\" do\n ensure_unlock do\n hold = Mosquito.backend.lock? key, \"abcd\", ttl\n assert hold\n\n renewed = Mosquito.backend.renew_lock? key, \"wxyz\", ttl\n refute renewed\n end\n end\n\n it \"doesn't renew a lock that doesn't exist\" do\n ensure_unlock do\n renewed = Mosquito.backend.renew_lock? key, instance_id, ttl\n refute renewed\n end\n end\nend\n" }, { "path": "spec/mosquito/backend/overseer_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe Mosquito::Backend do\n it \"can keep a list of overseers\" do\n clean_slate do\n overseer_ids = [\"overseer1\", \"overseer2\", \"overseer3\"]\n overseer_ids.each do |overseer_id|\n Mosquito.backend.register_overseer overseer_id\n end\n\n assert_equal overseer_ids, Mosquito.backend.list_overseers\n end\n end\n\n it \"can deregister an overseer\" do\n clean_slate do\n overseer_ids = [\"overseer1\", \"overseer2\", \"overseer3\"]\n overseer_ids.each do |overseer_id|\n Mosquito.backend.register_overseer overseer_id\n end\n\n Mosquito.backend.deregister_overseer \"overseer2\"\n\n assert_equal [\"overseer1\", \"overseer3\"], Mosquito.backend.list_overseers\n end\n end\nend\n" }, { "path": "spec/mosquito/backend/queueing_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe \"Backend Queues\" do\n getter backend_name : String { \"test#{rand(1000)}\" }\n getter queue : Mosquito::Backend::Queue { backend.queue backend_name }\n\n getter job : QueuedTestJob { QueuedTestJob.new }\n getter job_run : Mosquito::JobRun { Mosquito::JobRun.new(\"mock_job_run\") }\n\n describe \"list_queues\" do\n def fill_queues\n names = %w|test1 test2 test3 test4|\n\n names[0..3].each do |queue_name|\n backend.queue(queue_name).enqueue job_run\n end\n\n backend.queue(names.last).schedule job_run, at: 1.second.from_now\n end\n\n def fill_uncounted_queues\n names = %w|test5 test6 test7 test8|\n\n names[0..3].each do |queue_name|\n backend.queue(queue_name).tap do |q|\n q.enqueue job_run\n q.dequeue\n end\n end\n\n backend.queue(names.last).terminate job_run\n end\n\n it \"can get a list of available queues\" do\n clean_slate do\n fill_queues\n assert_equal %w|test1 test2 test3 test4|, backend.list_queues.sort\n end\n end\n\n it \"de-dups the queue list\" do\n clean_slate do\n fill_queues\n assert_equal %w|test1 test2 test3 test4|, backend.list_queues.sort\n end\n end\n end\n\n describe \"schedule\" do\n it \"adds a job_run to the schedule_q at the time\" do\n clean_slate do\n timestamp = 2.seconds.from_now\n job_run = job.build_job_run\n queue.schedule job_run, at: timestamp\n assert_equal Time.unix_ms(timestamp.to_unix_ms), queue.scheduled_job_run_time job_run\n end\n end\n end\n\n describe \"deschedule\" do\n it \"returns a job_run if it's due\" do\n clean_slate do\n run_time = Time.utc - 2.seconds\n job_run = job.build_job_run\n job_run.store\n queue.schedule job_run, at: run_time\n\n overdue_job_runs = queue.deschedule\n assert_equal [job_run], overdue_job_runs\n end\n end\n\n it \"returns a blank array when no job_runs exist\" do\n clean_slate do\n overdue_job_runs = queue.deschedule\n assert_empty overdue_job_runs\n end\n end\n\n it \"doesn't return job_runs which aren't yet due\" do\n clean_slate do\n run_time = Time.utc + 2.seconds\n job_run = job.build_job_run\n job_run.store\n queue.schedule job_run, at: run_time\n\n overdue_job_runs = queue.deschedule\n assert_empty overdue_job_runs\n end\n end\n end\n\n describe \"enqueue\" do\n it \"puts a job_run on the waiting_q\" do\n clean_slate do\n job_run = job.build_job_run\n queue.enqueue job_run\n waiting_job_runs = queue.list_waiting\n assert_equal [job_run.id], waiting_job_runs\n end\n end\n end\n\n describe \"dequeue\" do\n it \"returns a job_run object when one is waiting\" do\n clean_slate do\n job_run = job.build_job_run\n job_run.store\n queue.enqueue job_run\n waiting_job_run = queue.dequeue\n assert_equal job_run, waiting_job_run\n end\n end\n\n it \"moves the job_run from waiting to pending\" do\n clean_slate do\n job_run = job.build_job_run\n job_run.store\n queue.enqueue job_run\n waiting_job_run = queue.dequeue\n pending_job_runs = queue.list_pending\n assert_equal [job_run.id], pending_job_runs\n end\n end\n\n it \"returns nil when nothing is waiting\" do\n clean_slate do\n assert_equal nil, queue.dequeue\n end\n end\n\n it \"returns nil when a job_run is queued but not stored\" do\n clean_slate do\n job_run = job.build_job_run\n # job_run.store # explicitly don't store this one\n queue.enqueue job_run\n waiting_job_run = queue.dequeue\n assert_nil waiting_job_run\n end\n end\n end\n\n describe \"finish\" do\n it \"removes the job_run from the pending queue\" do\n clean_slate do\n job_run = job.build_job_run\n job_run.store\n\n # first move the job_run from waiting to pending\n queue.enqueue job_run\n waiting_job_run = queue.dequeue\n assert_equal job_run, waiting_job_run\n\n # now finish it\n queue.finish job_run\n\n pending_job_runs = queue.list_pending\n assert_empty pending_job_runs\n end\n end\n end\n\n describe \"terminate\" do\n it \"adds a job_run to the dead queue\" do\n clean_slate do\n job_run = job.build_job_run\n job_run.store\n\n # first move the job_run from waiting to pending\n queue.enqueue job_run\n waiting_job_run = queue.dequeue\n assert_equal job_run, waiting_job_run\n\n # now terminate it\n queue.terminate job_run\n\n dead_job_runs = queue.list_dead\n assert_equal [job_run.id], dead_job_runs\n end\n end\n end\n\nend\n" }, { "path": "spec/mosquito/backend_spec.cr", "content": "require \"../spec_helper\"\n\n# These tests are explicitly for code which is inherited from the abstract Backend\ndescribe Mosquito::Backend do\n it \"can build a key with two strings\" do\n assert_equal \"mosquito:one:two\", Mosquito.backend.build_key(\"one\", \"two\")\n end\n\n it \"can build a key with an array\" do\n assert_equal \"mosquito:one:two\", Mosquito.backend.build_key([\"one\", \"two\"])\n end\n\n it \"can build a key with a tuple\" do\n assert_equal \"mosquito:one:two\", Mosquito.backend.build_key(*{\"one\", \"two\"})\n end\n\n it \"can be initialized with a string name\" do\n Mosquito.backend.queue \"string_backend\"\n end\n\n it \"can be initialized with a symbol name\" do\n Mosquito.backend.queue :symbol_backend\n end\n\n it \"can update a key with a hash\" do\n Mosquito.backend.set \"key\", {\"field\" => \"value\", \"field2\" => \"value2\"}\n assert_equal \"value\", Mosquito.backend.get(\"key\", \"field\")\n assert_equal \"value2\", Mosquito.backend.get(\"key\", \"field2\")\n end\nend\n" }, { "path": "spec/mosquito/base_spec.cr", "content": "require \"../spec_helper\"\n\ndescribe Mosquito::Base do\n it \"keeps a list of scheduled job_runs\" do\n Base.bare_mapping do\n Base.register_job_interval PeriodicTestJob, 1.minute\n assert_equal PeriodicTestJob, Base.scheduled_job_runs.first.class\n end\n end\n\n it \"correctly maps job classes from type strings\" do\n Base.bare_mapping do\n Base.register_job_mapping \"fizzbuzz\", QueuedTestJob\n assert_equal QueuedTestJob, Base.job_for_type \"fizzbuzz\"\n end\n end\nend\n" }, { "path": "spec/mosquito/configuration_spec.cr", "content": "require \"../spec_helper\"\n\ndescribe \"Mosquito Config\" do\n it \"allows setting / retrieving the connection string\" do\n Mosquito.temp_config do\n Mosquito.configuration.backend_connection_string = testing_redis_url\n assert_equal testing_redis_url, Mosquito.configuration.backend_connection_string\n end\n end\n\n it \"enforces missing settings are set\" do\n config = Mosquito::Configuration.new\n assert_raises do\n config.validate\n end\n end\n\n it \"allows setting idle_wait as a float\" do\n test_value = 2.4\n Mosquito.temp_config do\n Mosquito.configuration.idle_wait = test_value\n assert_equal test_value.seconds, Mosquito.configuration.idle_wait\n end\n end\n\n it \"allows setting idle_wait as a time span\" do\n test_value = 2.seconds\n\n Mosquito.temp_config do\n Mosquito.configuration.idle_wait = test_value\n assert_equal test_value, Mosquito.configuration.idle_wait\n end\n end\n\n it \"allows setting successful_job_ttl\" do\n test_value = 2\n\n Mosquito.temp_config do\n Mosquito.configuration.successful_job_ttl = test_value\n assert_equal test_value, Mosquito.configuration.successful_job_ttl\n end\n end\n\n it \"allows setting failed_job_ttl\" do\n test_value = 2\n\n Mosquito.temp_config do\n Mosquito.configuration.failed_job_ttl = test_value\n assert_equal test_value, Mosquito.configuration.failed_job_ttl\n end\n end\n\n it \"allows setting global_prefix string\" do\n test_value = \"yolo\"\n\n Mosquito.temp_config do\n Mosquito.configuration.global_prefix = test_value\n assert_equal test_value, Mosquito.configuration.global_prefix\n Mosquito.configuration.backend.build_key(\"test\").must_equal \"yolo:mosquito:test\"\n end\n end\n\n it \"allows setting global_prefix nillable\" do\n test_value = nil\n\n Mosquito.temp_config do\n Mosquito.configuration.global_prefix = test_value\n assert_equal test_value, Mosquito.configuration.global_prefix\n Mosquito.configuration.backend.build_key(\"test\").must_equal \"mosquito:test\"\n end\n end\n\n it \"validates when backend_connection_string is set\" do\n Mosquito.temp_config do\n Mosquito.configuration.backend_connection_string = testing_redis_url\n Mosquito.configuration.validate\n end\n end\nend\n" }, { "path": "spec/mosquito/dequeue_adapters/concurrency_limited_dequeue_adapter_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe \"Mosquito::ConcurrencyLimitedDequeueAdapter\" do\n getter(overseer : MockOverseer) { MockOverseer.new }\n getter(queue_list : MockQueueList) { overseer.queue_list.as(MockQueueList) }\n\n def register(job_class : Mosquito::Job.class)\n Mosquito::Base.register_job_mapping job_class.name.underscore, job_class\n queue_list.queues << job_class.queue\n end\n\n it \"dequeues a job when under the limit\" do\n clean_slate do\n register QueuedTestJob\n expected_job_run = QueuedTestJob.new.enqueue\n\n adapter = Mosquito::ConcurrencyLimitedDequeueAdapter.new({\n \"queued_test_job\" => 3,\n })\n\n result = adapter.dequeue(queue_list)\n refute_nil result\n if result\n assert_equal expected_job_run, result.job_run\n assert_equal QueuedTestJob.queue, result.queue\n end\n end\n end\n\n it \"returns nil when no jobs are available\" do\n clean_slate do\n register QueuedTestJob\n\n adapter = Mosquito::ConcurrencyLimitedDequeueAdapter.new({\n \"queued_test_job\" => 3,\n })\n\n result = adapter.dequeue(queue_list)\n assert_nil result\n end\n end\n\n it \"skips a queue that has reached its concurrency limit\" do\n clean_slate do\n register QueuedTestJob\n 3.times { QueuedTestJob.new.enqueue }\n\n adapter = Mosquito::ConcurrencyLimitedDequeueAdapter.new({\n \"queued_test_job\" => 2,\n })\n\n # Dequeue twice — should succeed and fill the limit.\n result1 = adapter.dequeue(queue_list)\n refute_nil result1\n assert_equal 1, adapter.active_count(\"queued_test_job\")\n\n result2 = adapter.dequeue(queue_list)\n refute_nil result2\n assert_equal 2, adapter.active_count(\"queued_test_job\")\n\n # Third dequeue should be blocked by the limit.\n result3 = adapter.dequeue(queue_list)\n assert_nil result3\n end\n end\n\n it \"allows dequeue again after finished_with\" do\n clean_slate do\n register QueuedTestJob\n 3.times { QueuedTestJob.new.enqueue }\n\n adapter = Mosquito::ConcurrencyLimitedDequeueAdapter.new({\n \"queued_test_job\" => 1,\n })\n\n # Fill the single slot.\n result1 = adapter.dequeue(queue_list)\n refute_nil result1\n assert_equal 1, adapter.active_count(\"queued_test_job\")\n\n # Blocked.\n result2 = adapter.dequeue(queue_list)\n assert_nil result2\n\n # Signal that the job finished.\n adapter.finished_with(result1.not_nil!.job_run, result1.not_nil!.queue)\n assert_equal 0, adapter.active_count(\"queued_test_job\")\n\n # Now dequeue should work again.\n result3 = adapter.dequeue(queue_list)\n refute_nil result3\n end\n end\n\n it \"does not limit queues not in the limits table\" do\n clean_slate do\n register QueuedTestJob\n 5.times { QueuedTestJob.new.enqueue }\n\n # No limit configured for queued_test_job.\n adapter = Mosquito::ConcurrencyLimitedDequeueAdapter.new({\n \"other_queue\" => 1,\n })\n\n # Should dequeue all 5 without blocking.\n 5.times do |i|\n result = adapter.dequeue(queue_list)\n refute_nil result, \"Expected dequeue ##{i + 1} to succeed\"\n end\n end\n end\n\n it \"enforces independent limits across multiple queues\" do\n clean_slate do\n register QueuedTestJob\n register EchoJob\n 3.times { QueuedTestJob.new.enqueue }\n 3.times { EchoJob.new(text: \"hello\").enqueue }\n\n adapter = Mosquito::ConcurrencyLimitedDequeueAdapter.new({\n \"queued_test_job\" => 1,\n \"io_queue\" => 2,\n })\n\n # Saturate queued_test_job (limit 1).\n # Because of shuffle we may get either queue first, so keep\n # dequeuing until the counters match the limits.\n results = [] of Mosquito::WorkUnit\n 6.times do\n if r = adapter.dequeue(queue_list)\n results << r\n end\n end\n\n assert_equal 1, adapter.active_count(\"queued_test_job\")\n assert_equal 2, adapter.active_count(\"io_queue\")\n assert_equal 3, results.size\n end\n end\n\n it \"finished_with does not go below zero\" do\n adapter = Mosquito::ConcurrencyLimitedDequeueAdapter.new({\n \"queued_test_job\" => 3,\n })\n\n job_run = Mosquito::JobRun.new(\"queued_test_job\")\n queue = Mosquito::Queue.new(\"queued_test_job\")\n adapter.finished_with(job_run, queue)\n assert_equal 0, adapter.active_count(\"queued_test_job\")\n end\n\n it \"can be used via the overseer\" do\n clean_slate do\n adapter = Mosquito::ConcurrencyLimitedDequeueAdapter.new({\n \"queued_test_job\" => 5,\n })\n overseer.dequeue_adapter = adapter\n\n register QueuedTestJob\n expected_job_run = QueuedTestJob.new.enqueue\n\n result = overseer.dequeue_job?\n refute_nil result\n if result\n assert_equal expected_job_run, result.job_run\n end\n end\n end\nend\n" }, { "path": "spec/mosquito/dequeue_adapters/remote_config_dequeue_adapter_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe \"Mosquito::RemoteConfigDequeueAdapter\" do\n getter(overseer : MockOverseer) { MockOverseer.new }\n getter(queue_list : MockQueueList) { overseer.queue_list.as(MockQueueList) }\n\n def register(job_class : Mosquito::Job.class)\n Mosquito::Base.register_job_mapping job_class.name.underscore, job_class\n queue_list.queues << job_class.queue\n end\n\n it \"uses defaults when no remote config is present\" do\n clean_slate do\n register QueuedTestJob\n 3.times { QueuedTestJob.new.enqueue }\n\n adapter = Mosquito::RemoteConfigDequeueAdapter.new(\n defaults: {\"queued_test_job\" => 2},\n refresh_interval: 0.seconds,\n )\n\n # Two dequeues should succeed.\n result1 = adapter.dequeue(queue_list)\n refute_nil result1\n\n result2 = adapter.dequeue(queue_list)\n refute_nil result2\n\n # Third should be blocked by the default limit of 2.\n result3 = adapter.dequeue(queue_list)\n assert_nil result3\n end\n end\n\n it \"picks up remote limits from the backend\" do\n clean_slate do\n register QueuedTestJob\n 3.times { QueuedTestJob.new.enqueue }\n\n # Default allows 2, but remote overrides to 1.\n adapter = Mosquito::RemoteConfigDequeueAdapter.new(\n defaults: {\"queued_test_job\" => 2},\n refresh_interval: 0.seconds,\n )\n\n Mosquito::RemoteConfigDequeueAdapter.store_limits({\"queued_test_job\" => 1})\n\n result1 = adapter.dequeue(queue_list)\n refute_nil result1\n\n # Should be blocked — remote limit is 1.\n result2 = adapter.dequeue(queue_list)\n assert_nil result2\n end\n end\n\n it \"merges remote limits on top of defaults\" do\n clean_slate do\n adapter = Mosquito::RemoteConfigDequeueAdapter.new(\n defaults: {\"queue_a\" => 3, \"queue_b\" => 5},\n refresh_interval: 0.seconds,\n )\n\n # Remote only overrides queue_a and adds queue_c.\n Mosquito::RemoteConfigDequeueAdapter.store_limits({\n \"queue_a\" => 1,\n \"queue_c\" => 7,\n })\n\n adapter.refresh_limits\n\n assert_equal 1, adapter.limits[\"queue_a\"]\n assert_equal 5, adapter.limits[\"queue_b\"]\n assert_equal 7, adapter.limits[\"queue_c\"]\n end\n end\n\n it \"falls back to defaults when remote config is cleared\" do\n clean_slate do\n adapter = Mosquito::RemoteConfigDequeueAdapter.new(\n defaults: {\"queue_a\" => 3},\n refresh_interval: 0.seconds,\n )\n\n Mosquito::RemoteConfigDequeueAdapter.store_limits({\"queue_a\" => 1})\n adapter.refresh_limits\n assert_equal 1, adapter.limits[\"queue_a\"]\n\n Mosquito::RemoteConfigDequeueAdapter.clear_limits\n adapter.refresh_limits\n assert_equal 3, adapter.limits[\"queue_a\"]\n end\n end\n\n it \"respects refresh_interval and does not poll on every dequeue\" do\n clean_slate do\n register QueuedTestJob\n 3.times { QueuedTestJob.new.enqueue }\n\n adapter = Mosquito::RemoteConfigDequeueAdapter.new(\n defaults: {\"queued_test_job\" => 3},\n refresh_interval: 1.hour,\n )\n\n # First dequeue triggers the initial refresh.\n adapter.dequeue(queue_list)\n\n # Store a tighter limit — but it should NOT take effect\n # because the refresh interval hasn't elapsed.\n Mosquito::RemoteConfigDequeueAdapter.store_limits({\"queued_test_job\" => 1})\n\n result2 = adapter.dequeue(queue_list)\n refute_nil result2, \"Expected dequeue to succeed because refresh hasn't fired\"\n end\n end\n\n it \"preserves in-flight counts when limits are refreshed\" do\n clean_slate do\n register QueuedTestJob\n 2.times { QueuedTestJob.new.enqueue }\n\n adapter = Mosquito::RemoteConfigDequeueAdapter.new(\n defaults: {\"queued_test_job\" => 1},\n refresh_interval: 0.seconds,\n )\n\n result1 = adapter.dequeue(queue_list)\n refute_nil result1\n assert_equal 1, adapter.active_count(\"queued_test_job\")\n\n # Refresh with new limits — must not reset the in-flight counter.\n Mosquito::RemoteConfigDequeueAdapter.store_limits({\"queued_test_job\" => 2})\n adapter.refresh_limits\n assert_equal 1, adapter.active_count(\"queued_test_job\")\n\n adapter.finished_with(result1.not_nil!.job_run, result1.not_nil!.queue)\n assert_equal 0, adapter.active_count(\"queued_test_job\")\n end\n end\n\n it \"delegates finished_with to the inner adapter\" do\n clean_slate do\n register QueuedTestJob\n 2.times { QueuedTestJob.new.enqueue }\n\n adapter = Mosquito::RemoteConfigDequeueAdapter.new(\n defaults: {\"queued_test_job\" => 1},\n refresh_interval: 0.seconds,\n )\n\n result1 = adapter.dequeue(queue_list)\n refute_nil result1\n assert_equal 1, adapter.active_count(\"queued_test_job\")\n\n # Blocked.\n result2 = adapter.dequeue(queue_list)\n assert_nil result2\n\n # Signal completion.\n adapter.finished_with(result1.not_nil!.job_run, result1.not_nil!.queue)\n assert_equal 0, adapter.active_count(\"queued_test_job\")\n\n # Now a dequeue should succeed again.\n result3 = adapter.dequeue(queue_list)\n refute_nil result3\n end\n end\n\n it \"can be used via the overseer\" do\n clean_slate do\n adapter = Mosquito::RemoteConfigDequeueAdapter.new(\n defaults: {\"queued_test_job\" => 5},\n refresh_interval: 0.seconds,\n )\n overseer.dequeue_adapter = adapter\n\n register QueuedTestJob\n expected_job_run = QueuedTestJob.new.enqueue\n\n result = overseer.dequeue_job?\n refute_nil result\n if result\n assert_equal expected_job_run, result.job_run\n end\n end\n end\n\n describe \"per-overseer configuration\" do\n it \"uses per-overseer limits when overseer_id is set\" do\n clean_slate do\n register QueuedTestJob\n 3.times { QueuedTestJob.new.enqueue }\n\n adapter = Mosquito::RemoteConfigDequeueAdapter.new(\n defaults: {\"queued_test_job\" => 3},\n overseer_id: \"gpu-worker-1\",\n refresh_interval: 0.seconds,\n )\n\n # Set a per-overseer limit of 1.\n Mosquito::RemoteConfigDequeueAdapter.store_limits(\n {\"queued_test_job\" => 1}, overseer_id: \"gpu-worker-1\"\n )\n\n result1 = adapter.dequeue(queue_list)\n refute_nil result1\n\n # Should be blocked by the per-overseer limit.\n result2 = adapter.dequeue(queue_list)\n assert_nil result2\n end\n end\n\n it \"per-overseer limits override global limits\" do\n clean_slate do\n adapter = Mosquito::RemoteConfigDequeueAdapter.new(\n defaults: {\"queue_a\" => 10},\n overseer_id: \"gpu-worker-1\",\n refresh_interval: 0.seconds,\n )\n\n # Global says 5, per-overseer says 2 — per-overseer wins.\n Mosquito::RemoteConfigDequeueAdapter.store_limits({\"queue_a\" => 5})\n Mosquito::RemoteConfigDequeueAdapter.store_limits(\n {\"queue_a\" => 2}, overseer_id: \"gpu-worker-1\"\n )\n\n adapter.refresh_limits\n assert_equal 2, adapter.limits[\"queue_a\"]\n end\n end\n\n it \"falls back to global when no per-overseer key exists\" do\n clean_slate do\n adapter = Mosquito::RemoteConfigDequeueAdapter.new(\n defaults: {\"queue_a\" => 10},\n overseer_id: \"gpu-worker-1\",\n refresh_interval: 0.seconds,\n )\n\n Mosquito::RemoteConfigDequeueAdapter.store_limits({\"queue_a\" => 5})\n\n adapter.refresh_limits\n assert_equal 5, adapter.limits[\"queue_a\"]\n end\n end\n\n it \"merges defaults, global, and per-overseer layers\" do\n clean_slate do\n adapter = Mosquito::RemoteConfigDequeueAdapter.new(\n defaults: {\"queue_a\" => 10, \"queue_b\" => 20, \"queue_c\" => 30},\n overseer_id: \"gpu-worker-1\",\n refresh_interval: 0.seconds,\n )\n\n # Global overrides queue_a and adds queue_d.\n Mosquito::RemoteConfigDequeueAdapter.store_limits({\n \"queue_a\" => 5,\n \"queue_d\" => 40,\n })\n\n # Per-overseer overrides queue_a again and queue_b.\n Mosquito::RemoteConfigDequeueAdapter.store_limits(\n {\"queue_a\" => 1, \"queue_b\" => 2},\n overseer_id: \"gpu-worker-1\"\n )\n\n adapter.refresh_limits\n\n assert_equal 1, adapter.limits[\"queue_a\"] # per-overseer wins\n assert_equal 2, adapter.limits[\"queue_b\"] # per-overseer wins\n assert_equal 30, adapter.limits[\"queue_c\"] # default (untouched)\n assert_equal 40, adapter.limits[\"queue_d\"] # global (no per-overseer)\n end\n end\n\n it \"adapters without overseer_id ignore per-overseer keys\" do\n clean_slate do\n adapter = Mosquito::RemoteConfigDequeueAdapter.new(\n defaults: {\"queue_a\" => 10},\n refresh_interval: 0.seconds,\n )\n\n Mosquito::RemoteConfigDequeueAdapter.store_limits({\"queue_a\" => 5})\n Mosquito::RemoteConfigDequeueAdapter.store_limits(\n {\"queue_a\" => 1}, overseer_id: \"gpu-worker-1\"\n )\n\n adapter.refresh_limits\n\n # Without an overseer_id, only global is used.\n assert_equal 5, adapter.limits[\"queue_a\"]\n end\n end\n end\n\n describe \"class-level storage helpers\" do\n it \"round-trips global limits through the backend\" do\n clean_slate do\n limits = {\"queue_a\" => 3, \"queue_b\" => 7}\n Mosquito::RemoteConfigDequeueAdapter.store_limits(limits)\n\n retrieved = Mosquito::RemoteConfigDequeueAdapter.stored_limits\n assert_equal 3, retrieved[\"queue_a\"]\n assert_equal 7, retrieved[\"queue_b\"]\n end\n end\n\n it \"round-trips per-overseer limits through the backend\" do\n clean_slate do\n limits = {\"queue_a\" => 1}\n Mosquito::RemoteConfigDequeueAdapter.store_limits(limits, overseer_id: \"worker-2\")\n\n retrieved = Mosquito::RemoteConfigDequeueAdapter.stored_limits(\"worker-2\")\n assert_equal 1, retrieved[\"queue_a\"]\n\n # Global should be unaffected.\n global = Mosquito::RemoteConfigDequeueAdapter.stored_limits\n assert_equal({} of String => Int32, global)\n end\n end\n\n it \"store_limits overwrites rather than merges (stale entries are removed)\" do\n clean_slate do\n Mosquito::RemoteConfigDequeueAdapter.store_limits({\"queue_a\" => 3, \"queue_b\" => 7})\n Mosquito::RemoteConfigDequeueAdapter.store_limits({\"queue_a\" => 1})\n\n retrieved = Mosquito::RemoteConfigDequeueAdapter.stored_limits\n assert_equal 1, retrieved[\"queue_a\"]\n refute retrieved.has_key?(\"queue_b\"), \"queue_b should have been removed by the overwrite\"\n end\n end\n\n it \"store_limits with overseer_id overwrites rather than merges\" do\n clean_slate do\n Mosquito::RemoteConfigDequeueAdapter.store_limits({\"queue_a\" => 3, \"queue_b\" => 7}, overseer_id: \"worker-1\")\n Mosquito::RemoteConfigDequeueAdapter.store_limits({\"queue_a\" => 1}, overseer_id: \"worker-1\")\n\n retrieved = Mosquito::RemoteConfigDequeueAdapter.stored_limits(\"worker-1\")\n assert_equal 1, retrieved[\"queue_a\"]\n refute retrieved.has_key?(\"queue_b\"), \"queue_b should have been removed by the overwrite\"\n end\n end\n\n it \"store_limits with an empty hash removes all stored limits\" do\n clean_slate do\n Mosquito::RemoteConfigDequeueAdapter.store_limits({\"queue_a\" => 3})\n Mosquito::RemoteConfigDequeueAdapter.store_limits({} of String => Int32)\n\n retrieved = Mosquito::RemoteConfigDequeueAdapter.stored_limits\n assert_equal({} of String => Int32, retrieved)\n end\n end\n\n it \"returns an empty hash when no limits are stored\" do\n clean_slate do\n retrieved = Mosquito::RemoteConfigDequeueAdapter.stored_limits\n assert_equal({} of String => Int32, retrieved)\n end\n end\n\n it \"clear_limits removes global stored data\" do\n clean_slate do\n Mosquito::RemoteConfigDequeueAdapter.store_limits({\"queue_a\" => 1})\n Mosquito::RemoteConfigDequeueAdapter.clear_limits\n\n retrieved = Mosquito::RemoteConfigDequeueAdapter.stored_limits\n assert_equal({} of String => Int32, retrieved)\n end\n end\n\n it \"clear_limits with overseer_id removes only that overseer's data\" do\n clean_slate do\n Mosquito::RemoteConfigDequeueAdapter.store_limits({\"queue_a\" => 5})\n Mosquito::RemoteConfigDequeueAdapter.store_limits(\n {\"queue_a\" => 1}, overseer_id: \"worker-1\"\n )\n\n Mosquito::RemoteConfigDequeueAdapter.clear_limits(overseer_id: \"worker-1\")\n\n # Per-overseer is gone.\n per_overseer = Mosquito::RemoteConfigDequeueAdapter.stored_limits(\"worker-1\")\n assert_equal({} of String => Int32, per_overseer)\n\n # Global is still there.\n global = Mosquito::RemoteConfigDequeueAdapter.stored_limits\n assert_equal 5, global[\"queue_a\"]\n end\n end\n end\n\n describe \"Api integration\" do\n it \"reads and writes global limits through the Api module\" do\n clean_slate do\n Mosquito::Api.set_concurrency_limits({\"queue_x\" => 10})\n result = Mosquito::Api.concurrency_limits\n assert_equal 10, result[\"queue_x\"]\n end\n end\n\n it \"reads and writes per-overseer limits through the Api module\" do\n clean_slate do\n Mosquito::Api.set_concurrency_limits(\n {\"queue_x\" => 2}, overseer_id: \"gpu-worker-1\"\n )\n result = Mosquito::Api.concurrency_limits(overseer_id: \"gpu-worker-1\")\n assert_equal 2, result[\"queue_x\"]\n\n # Global should be unaffected.\n global = Mosquito::Api.concurrency_limits\n assert_equal({} of String => Int32, global)\n end\n end\n end\nend\n" }, { "path": "spec/mosquito/dequeue_adapters/shuffle_dequeue_adapter_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe \"Mosquito::ShuffleDequeueAdapter\" do\n getter(overseer : MockOverseer) { MockOverseer.new }\n getter(queue_list : MockQueueList) { overseer.queue_list.as(MockQueueList) }\n getter(executor : MockExecutor) { overseer.executors.first.as(MockExecutor) }\n\n def register(job_class : Mosquito::Job.class)\n Mosquito::Base.register_job_mapping job_class.name.underscore, job_class\n queue_list.discovered_queues << job_class.queue\n end\n\n it \"is the default adapter\" do\n assert_instance_of Mosquito::ShuffleDequeueAdapter, Mosquito.configuration.dequeue_adapter\n end\n\n it \"dequeues a job from the queue list\" do\n clean_slate do\n register QueuedTestJob\n expected_job_run = QueuedTestJob.new.enqueue\n\n adapter = Mosquito::ShuffleDequeueAdapter.new\n result = adapter.dequeue(queue_list)\n\n refute_nil result\n if result\n assert_equal expected_job_run, result.job_run\n assert_equal QueuedTestJob.queue, result.queue\n end\n end\n end\n\n it \"returns nil when no jobs are available\" do\n clean_slate do\n register QueuedTestJob\n\n adapter = Mosquito::ShuffleDequeueAdapter.new\n result = adapter.dequeue(queue_list)\n assert_nil result\n end\n end\n\n describe \"custom adapter\" do\n it \"can be swapped on the overseer\" do\n clean_slate do\n null_adapter = NullDequeueAdapter.new\n overseer.dequeue_adapter = null_adapter\n\n register QueuedTestJob\n QueuedTestJob.new.enqueue\n\n result = overseer.dequeue_job?\n assert_nil result\n assert_equal 1, null_adapter.dequeue_count\n end\n end\n\n it \"receives the queue list when dequeuing\" do\n clean_slate do\n spy_adapter = SpyDequeueAdapter.new\n overseer.dequeue_adapter = spy_adapter\n\n register QueuedTestJob\n queue_list.discovered_queues << Mosquito::Queue.new(\"extra_queue\")\n\n overseer.dequeue_job?\n\n assert_includes spy_adapter.checked_queues, \"queued_test_job\"\n assert_includes spy_adapter.checked_queues, \"extra_queue\"\n end\n end\n end\n\n describe \"overseer integration\" do\n it \"dequeue_job? delegates to the adapter\" do\n clean_slate do\n register QueuedTestJob\n expected_job_run = QueuedTestJob.new.enqueue\n\n result = overseer.dequeue_job?\n refute_nil result\n if result\n assert_equal expected_job_run, result.job_run\n end\n end\n end\n end\nend\n" }, { "path": "spec/mosquito/dequeue_adapters/weighted_dequeue_adapter_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe \"Mosquito::WeightedDequeueAdapter\" do\n getter(overseer : MockOverseer) { MockOverseer.new }\n getter(queue_list : MockQueueList) { overseer.queue_list.as(MockQueueList) }\n\n def register(job_class : Mosquito::Job.class)\n Mosquito::Base.register_job_mapping job_class.name.underscore, job_class\n queue_list.discovered_queues << job_class.queue\n end\n\n it \"dequeues a job from a weighted queue\" do\n clean_slate do\n register QueuedTestJob\n expected_job_run = QueuedTestJob.new.enqueue\n\n adapter = Mosquito::WeightedDequeueAdapter.new({\n \"queued_test_job\" => 5,\n })\n\n result = adapter.dequeue(queue_list)\n refute_nil result\n if result\n assert_equal expected_job_run, result.job_run\n assert_equal QueuedTestJob.queue, result.queue\n end\n end\n end\n\n it \"returns nil when no jobs are available\" do\n clean_slate do\n register QueuedTestJob\n\n adapter = Mosquito::WeightedDequeueAdapter.new({\n \"queued_test_job\" => 3,\n })\n\n result = adapter.dequeue(queue_list)\n assert_nil result\n end\n end\n\n it \"assigns default weight of 1 to unconfigured queues\" do\n clean_slate do\n register QueuedTestJob\n expected_job_run = QueuedTestJob.new.enqueue\n\n # No weight configured for queued_test_job — defaults to 1.\n adapter = Mosquito::WeightedDequeueAdapter.new({\n \"other_queue\" => 10,\n })\n\n result = adapter.dequeue(queue_list)\n refute_nil result\n if result\n assert_equal expected_job_run, result.job_run\n end\n end\n end\n\n it \"higher-weight queues are dequeued more often\" do\n clean_slate do\n register QueuedTestJob\n register EchoJob\n\n adapter = Mosquito::WeightedDequeueAdapter.new({\n \"queued_test_job\" => 10,\n \"io_queue\" => 1,\n })\n\n # Enqueue enough jobs that neither queue drains during the sample.\n 200.times { QueuedTestJob.new.enqueue }\n 200.times { EchoJob.new(text: \"hello\").enqueue }\n\n dequeue_counts = Hash(String, Int32).new(0)\n\n # Sample 50 dequeues — well within the 200 available per queue.\n 50.times do\n result = adapter.dequeue(queue_list)\n if result\n dequeue_counts[result.queue.name] = dequeue_counts[result.queue.name] + 1\n end\n end\n\n # With weights 10:1, the high-weight queue should be dequeued\n # significantly more often over a 50-dequeue sample.\n heavy_count = dequeue_counts.fetch(\"queued_test_job\", 0)\n light_count = dequeue_counts.fetch(\"io_queue\", 0)\n assert heavy_count > light_count, \"Expected queued_test_job (#{heavy_count}) to be dequeued more than io_queue (#{light_count})\"\n end\n end\n\n it \"can be used via the overseer\" do\n clean_slate do\n adapter = Mosquito::WeightedDequeueAdapter.new({\n \"queued_test_job\" => 5,\n })\n overseer.dequeue_adapter = adapter\n\n register QueuedTestJob\n expected_job_run = QueuedTestJob.new.enqueue\n\n result = overseer.dequeue_job?\n refute_nil result\n if result\n assert_equal expected_job_run, result.job_run\n end\n end\n end\nend\n" }, { "path": "spec/mosquito/exceptions_spec.cr", "content": "require \"../spec_helper\"\n\ndescribe \"Mosquito exceptions\" do\n it \"declares JobFailed\" do\n Mosquito::JobFailed.new \"test\"\n end\n\n it \"declares DoubleRun\" do\n Mosquito::DoubleRun.new \"test\"\n end\n\n it \"declares IrretrievableParameter\" do\n Mosquito::IrretrievableParameter.new \"test\"\n end\nend\n" }, { "path": "spec/mosquito/job/job_state_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe Mosquito::Job::State do\n describe \"executed?\" do\n it \"Marks jobs as executed when they've either succeeded or failed\" do\n assert Mosquito::Job::State::Succeeded.executed?\n assert Mosquito::Job::State::Failed.executed?\n end\n\n it \"Doesn't mark jobs as executed in any other state\" do\n refute Mosquito::Job::State::Initialization.executed?\n refute Mosquito::Job::State::Running.executed?\n refute Mosquito::Job::State::Aborted.executed?\n refute Mosquito::Job::State::Preempted.executed?\n end\n end\nend\n" }, { "path": "spec/mosquito/job_run/rescheduling_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe \"job_run rescheduling\" do\n @failing_job_run : Mosquito::JobRun?\n getter failing_job_run : Mosquito::JobRun { create_job_run \"failing_job\" }\n\n it \"calculates reschedule interval correctly\" do\n intervals = {\n 1 => 2,\n 2 => 8,\n 3 => 18,\n 4 => 32\n }\n\n intervals.each do |count, delay|\n job_run = Mosquito::JobRun.retrieve(failing_job_run.id.not_nil!).not_nil!\n job_run.run\n assert_equal delay.seconds, job_run.reschedule_interval\n end\n end\n\n it \"prevents rescheduling a job too many times\" do\n run_job_run = -> do\n job_run = Mosquito::JobRun.retrieve(failing_job_run.id.not_nil!).not_nil!\n job_run.run\n job_run\n end\n\n max_reschedules = 4\n max_reschedules.times do\n job_run = run_job_run.call\n assert job_run.rescheduleable?\n end\n\n job_run = run_job_run.call\n refute job_run.rescheduleable?\n end\n\n it \"counts retries upon failure\" do\n assert_equal 0, failing_job_run.retry_count\n failing_job_run.run\n assert_equal 1, failing_job_run.retry_count\n end\n\n it \"updates the backend when a failure happens\" do\n failing_job_run.run\n saved_job_run = Mosquito::JobRun.retrieve failing_job_run.id.not_nil!\n assert_equal 1, saved_job_run.not_nil!.retry_count\n end\n\n it \"does not reschedule a job which fails with retry=false\" do\n job = FailingJob.new\n job.fail_with_retry = false\n job.run\n\n refute job.should_retry\n end\n\n describe \"preempted jobs\" do\n it \"sets state to preempted and does not execute\" do\n job = PreemptingJob.new\n job.run\n assert job.preempted?\n refute job.executed?\n end\n\n it \"uses normal backoff when preempted without an until time\" do\n job = PreemptingJob.new\n job.run\n assert_equal 2.seconds, job.reschedule_interval(1)\n assert_equal 8.seconds, job.reschedule_interval(2)\n end\n\n it \"uses the until time for reschedule interval when provided\" do\n Timecop.freeze(Time.utc) do\n future = Time.utc + 30.seconds\n job = PreemptingJob.new\n job.preempt_until = future\n job.run\n\n interval = job.reschedule_interval(1)\n assert_equal 30.seconds, interval\n end\n end\n\n it \"falls back to normal backoff when until time is in the past\" do\n Timecop.freeze(Time.utc) do\n past = Time.utc - 5.seconds\n job = PreemptingJob.new\n job.preempt_until = past\n job.run\n\n assert_equal 2.seconds, job.reschedule_interval(1)\n end\n end\n\n it \"respects rescheduleable? override when preempted\" do\n job = NonReschedulablePreemptingJob.new\n job.run\n assert job.preempted?\n refute job.rescheduleable?(0)\n end\n end\nend\n" }, { "path": "spec/mosquito/job_run/running_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe \"job_run running\" do\n # the job run timestamps are stored as a unix epoch with millis, so nanosecond precision is lost.\n def at_beginning_of_millisecond(time)\n time - (time.nanosecond.nanoseconds) + (time.millisecond.milliseconds)\n end\n\n it \"uses the lookup table to build a job\" do\n job_instance = create_job_run.build_job\n assert_instance_of JobWithConfig, job_instance\n end\n\n it \"populates the variables of a job\" do\n job_instance = create_job_run.build_job\n\n assert_instance_of JobWithConfig, job_instance\n assert_equal job_run_config, job_instance.as(JobWithConfig).config\n end\n\n it \"runs the job\" do\n JobWithPerformanceCounter.reset_performance_counter!\n create_job_run(\"job_with_performance_counter\").run\n assert_equal 1, JobWithPerformanceCounter.performances\n end\n\n it \"sets started_at when a job is run\" do\n now = at_beginning_of_millisecond Time.utc\n job_run = create_job_run\n Timecop.freeze now do\n job_run.run\n end\n assert_equal now, job_run.started_at\n end\n\n it \"sets finished_at when a job is run\" do\n now = at_beginning_of_millisecond Time.utc\n job_run = create_job_run\n Timecop.freeze now do\n job_run.run\n end\n assert_equal now, job_run.finished_at\n end\n\n it \"has nil timestamps before a job is run\" do\n job_run = create_job_run\n assert_nil job_run.started_at\n assert_nil job_run.finished_at\n end\nend\n" }, { "path": "spec/mosquito/job_run/storage_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe \"job_run storage\" do\n getter backend : Mosquito::Backend::Queue = Mosquito.backend.queue(\"testing\")\n\n getter config = {\n \"year\" => \"1752\",\n \"name\" => \"the year september lost 12 days\"\n }\n\n getter job_run : Mosquito::JobRun do\n Mosquito::JobRun.new(\"mock_job_run\").tap do |job_run|\n job_run.config = config\n job_run.store\n end\n end\n\n it \"builds the backend key correctly\" do\n assert_equal \"mosquito:job_run:1\", Mosquito::JobRun.config_key \"1\"\n assert_equal \"mosquito:job_run:#{job_run.id}\", job_run.config_key\n end\n\n it \"can store and retrieve a job_run with attributes\" do\n stored_job_run = Mosquito::JobRun.retrieve job_run.id\n if stored_job_run\n assert_equal config, stored_job_run.config\n else\n flunk \"Could not retrieve job_run\"\n end\n end\n\n it \"stores job_runs in the backend\" do\n stored_job_run = backend.backend.retrieve Mosquito::JobRun.config_key(job_run.id)\n stored_config = stored_job_run.reject! %w|type enqueue_time retry_count|\n assert_equal config, stored_config\n end\n\n it \"can delete a job_run\" do\n job_run.delete\n saved_config = backend.backend.retrieve job_run.config_key\n assert_empty saved_config\n end\n\n it \"can set a timed delete on a job_run\" do\n ttl = 10\n job_run.delete(in: ttl)\n set_ttl = backend.backend.expires_in job_run.config_key\n assert_equal ttl, set_ttl\n end\n\n it \"can reload a job_run\" do\n job_run.reload\n end\n\n describe \"timestamp retrieval\" do\n # the job run timestamps are stored as a unix epoch with millis, so nanosecond precision is lost.\n def at_beginning_of_millisecond(time)\n time - (time.nanosecond.nanoseconds) + (time.millisecond.milliseconds)\n end\n\n it \"retrieves started_at and finished_at timestamps\" do\n now = at_beginning_of_millisecond Time.utc\n job_run = create_job_run\n Timecop.freeze now do\n job_run.run\n end\n\n retrieved = Mosquito::JobRun.retrieve job_run.id\n if retrieved\n assert_equal now, retrieved.started_at\n assert_equal now, retrieved.finished_at\n else\n flunk \"Could not retrieve job_run\"\n end\n end\n\n it \"does not include timestamps in config after retrieve\" do\n job_run = create_job_run\n job_run.run\n\n retrieved = Mosquito::JobRun.retrieve job_run.id\n if retrieved\n refute retrieved.config.has_key?(\"started_at\")\n refute retrieved.config.has_key?(\"finished_at\")\n else\n flunk \"Could not retrieve job_run\"\n end\n end\n\n it \"retrieves nil timestamps for unexecuted job runs\" do\n retrieved = Mosquito::JobRun.retrieve job_run.id\n if retrieved\n assert_nil retrieved.started_at\n assert_nil retrieved.finished_at\n else\n flunk \"Could not retrieve job_run\"\n end\n end\n end\n\n it \"persists overseer_id via claimed_by and retrieves it\" do\n test_overseer = MockOverseer.new\n job_run.claimed_by test_overseer\n retrieved = Mosquito::JobRun.retrieve job_run.id\n assert retrieved\n assert_equal test_overseer.observer.instance_id, retrieved.not_nil!.overseer_id\n end\n\n it \"round-trips overseer_id through store and retrieve\" do\n test_overseer = MockOverseer.new\n job_run.claimed_by test_overseer\n job_run.store\n\n retrieved = Mosquito::JobRun.retrieve job_run.id\n assert retrieved\n assert_equal test_overseer.observer.instance_id, retrieved.not_nil!.overseer_id\n end\nend\n" }, { "path": "spec/mosquito/job_run_spec.cr", "content": "require \"../spec_helper\"\nrequire \"./job_run/*\"\n" }, { "path": "spec/mosquito/job_spec.cr", "content": "require \"../spec_helper\"\n\ndescribe Mosquito::Job do\n getter(passing_job) { PassingJob.new }\n getter(failing_job) { FailingJob.new }\n getter(not_implemented_job) { NotImplementedJob.new }\n\n getter(throttled_job) { ThrottledJob.new }\n getter(hooked_job) { JobWithHooks.new }\n\n describe \"run\" do\n it \"captures JobFailed and marks sucess=false\" do\n failing_job.run\n assert failing_job.failed?\n end\n\n it \"sets #executed? and #succeeded?\" do\n refute passing_job.executed?\n\n passing_job.run\n\n assert passing_job.executed?\n assert passing_job.succeeded?\n end\n\n it \"emits a failure message when #fail contains a reason message\" do\n clear_logs\n\n failing_job.run\n assert failing_job.failed?\n\n assert_logs_match failing_job.exception_message\n end\n\n it \"exception messages are sent to the logs\" do\n clear_logs\n\n failing_job.fail_with_exception = true\n failing_job.run\n assert failing_job.failed?\n\n assert_logs_match failing_job.exception_message\n end\n\n it \"captures and marks failure for other exceptions\" do\n clear_logs\n\n assert_nil failing_job.exception\n\n failing_job.fail_with_exception = true\n failing_job.run\n assert failing_job.failed?\n refute_nil failing_job.exception\n end\n\n it \"sets success=false when #fail-ed\" do\n failing_job.run\n refute failing_job.succeeded?\n end\n\n it \"fails when no perform is implemented, and a messsage is sent to the logs\" do\n clear_logs\n\n not_implemented_job.run\n assert not_implemented_job.failed?\n\n assert_logs_match \"No job definition found\"\n end\n end\n\n it \"fetches the default queue\" do\n assert_equal \"passing_job\", PassingJob.queue.name\n end\n\n it \"fetches the named queue\" do\n assert_equal \"io_queue\", EchoJob.queue.name\n end\n\n describe \"reschedule interval\" do\n it \"calculates reschedule interval correctly\" do\n intervals = {\n 1 => 2,\n 2 => 8,\n 3 => 18,\n 4 => 32\n }\n\n intervals.each do |count, delay|\n assert_equal delay.seconds, passing_job.reschedule_interval(count)\n end\n end\n\n\n it \"allows overriding the reschedule interval\" do\n intervals = 1..4\n\n intervals.each do |count|\n assert_equal 4.seconds, CustomRescheduleIntervalJob.new.reschedule_interval(count)\n end\n end\n end\n\n describe \"metadata\" do\n it \"returns a metadata instance\" do\n assert_instance_of Mosquito::Metadata, passing_job.metadata\n end\n\n it \"is a memoized instance\" do\n one = passing_job.metadata\n two = passing_job.metadata\n\n assert_same one, two\n end\n end\n\n describe \"self.metadata\" do\n it \"returns a metadata instance\" do\n assert PassingJob.metadata.is_a?(Mosquito::Metadata)\n end\n\n it \"is readonly\" do\n metadata = PassingJob.metadata\n assert metadata.readonly?\n end\n end\n\n describe \"self.metadata_key\" do\n it \"includes the class name\" do\n assert_includes PassingJob.metadata_key, \"passing_job\"\n end\n end\n\n describe \"before_hooks\" do\n it \"should execute hooks\" do\n clear_logs\n hooked_job.should_fail = false\n hooked_job.run\n assert_logs_match \"Before Hook Executed\"\n assert_logs_match \"2nd Before Hook Executed\"\n assert_logs_match \"Perform Executed\"\n end\n\n it \"should not exec when a before hook fails the job\" do\n clear_logs\n hooked_job.should_fail = true\n hooked_job.run\n\n assert_logs_match \"Before Hook Executed\"\n assert_logs_match \"2nd Before Hook Executed\"\n refute_logs_match \"Perform Executed\"\n end\n end\n\n describe \"after_hooks\" do\n it \"should execute `after` hooks\" do\n clear_logs\n hooked_job.should_fail = false\n hooked_job.run\n assert_logs_match \"After Hook Executed\"\n assert_logs_match \"2nd After Hook Executed\"\n assert_logs_match \"Perform Executed\"\n end\n\n it \"should run the `after` hooks even if a job fails\" do\n clear_logs\n hooked_job.should_fail = true\n hooked_job.run\n assert_logs_match \"After Hook Executed\"\n assert_logs_match \"2nd After Hook Executed\"\n refute_logs_match \"Perform Executed\"\n end\n end\nend\n" }, { "path": "spec/mosquito/key_builder_spec.cr", "content": "require \"../spec_helper\"\n\ndescribe Mosquito::KeyBuilder do\n it \"builds keys from tuples\" do\n assert_equal \"fizz:buzz\", KeyBuilder.build({:fizz, :buzz})\n end\n\n it \"builds keys from strings\" do\n assert_equal \"fizz:buzz\", KeyBuilder.build(\"fizz\", \"buzz\")\n end\n\n it \"builds keys from an array\" do\n assert_equal \"fizz:buzz\", KeyBuilder.build([\"fizz\", \"buzz\"])\n end\n\n it \"builds keys from integers\" do\n assert_equal \"fizz:6\", KeyBuilder.build(\"fizz\", 6)\n end\n\n it \"builds keys from floats\" do\n assert_equal \"2.4:buzz\", KeyBuilder.build(2.4, \"buzz\")\n end\nend\n" }, { "path": "spec/mosquito/metadata_spec.cr", "content": "require \"../spec_helper\"\n\ndescribe Mosquito::Metadata do\n getter(store_name : String) { \"test_store#{rand 1000}\" }\n getter(store : Metadata) { Metadata.new store_name }\n getter(field : String) { \"foo#{rand 1000}\" }\n\n it \"increments\" do\n clean_slate do\n store.increment field\n value = store[field]?\n assert_equal \"1\", value\n\n store.increment field\n value = store[field]?\n assert_equal \"2\", value\n end\n end\n\n it \"increments with a configurable amount\" do\n clean_slate do\n store.increment field\n value = store[field]?.not_nil!\n assert_equal \"1\", value\n\n delta = 2\n store.increment field, by: delta\n new_value = store[field]?.not_nil!\n assert_equal delta, (new_value.to_i - value.to_i)\n end\n end\n\n it \"decrements\" do\n clean_slate do\n store.decrement field\n value = store[field]?\n assert_equal \"-1\", value\n\n store.decrement field\n value = store[field]?\n assert_equal \"-2\", value\n end\n end\n\n it \"dumps to a hash\" do\n clean_slate do\n expected = { \"one\" => \"1\", \"two\" => \"2\", \"three\" => \"3\" }\n\n expected.each { |key, value| store[key] = value }\n\n assert_equal expected, store.to_h\n end\n end\n\n it \"can be readonly\" do\n clean_slate do\n store[field] = \"truth\"\n readonly_store = Metadata.new store_name, readonly: true\n assert_equal \"truth\", readonly_store[field]?\n\n assert_raises RuntimeError do\n readonly_store[field] = \"lies\"\n end\n end\n end\n\n it \"can set and read a value\" do\n clean_slate do\n store[field] = \"truth\"\n assert_equal \"truth\", store[field]?\n end\n end\n\n describe \"with a hash\" do\n it \"can set and read a hash\" do\n clean_slate do\n store.set({\"one\" => \"1\", \"two\" => \"2\", \"three\" => \"3\"})\n assert_equal \"1\", store[\"one\"]?\n assert_equal \"2\", store[\"two\"]?\n assert_equal \"3\", store[\"three\"]?\n end\n end\n\n it \"can set a hash and delete a value from the hash\" do\n clean_slate do\n store.set({\"one\" => \"1\", \"two\" => \"2\", \"three\" => \"3\"})\n store.set({\"two\" => nil, \"six\" => \"6\"})\n assert_equal \"1\", store[\"one\"]?\n assert_equal nil, store[\"two\"]?\n assert_equal \"3\", store[\"three\"]?\n assert_equal \"6\", store[\"six\"]?\n end\n end\n\n it \"can store string-only values\" do\n clean_slate do\n values = {\"one\" => \"1\", \"two\" => \"2\", \"three\" => \"3\"}\n store.set(values)\n assert_equal \"1\", store[\"one\"]?\n assert_equal \"2\", store[\"two\"]?\n assert_equal \"3\", store[\"three\"]?\n assert_equal values, store.to_h\n end\n end\n end\n\n it \"can be deleted\" do\n clean_slate do\n store[field] = \"truth\"\n assert_equal \"truth\", store[field]?\n store.delete\n assert_equal nil, Metadata.new(store_name)[field]?\n end\n end\n\n it \"can be deleted with a ttl\" do\n clean_slate do\n store[field] = \"truth\"\n assert_equal \"truth\", store[field]?\n store.delete(in: 1.minute)\n assert_in_epsilon(60, Mosquito.backend.expires_in(store_name))\n store.delete\n end\n end\nend\n" }, { "path": "spec/mosquito/periodic_job_run_spec.cr", "content": "require \"../spec_helper\"\n\ndescribe Mosquito::PeriodicJobRun do\n getter interval : Time::Span = 2.minutes\n\n it \"tries to execute but fails before the interval has passed\" do\n now = Time.utc.at_beginning_of_second\n job_run = PeriodicJobRun.new PeriodicTestJob, interval\n job_run.last_executed_at = now\n\n Timecop.freeze(now + 1.minute) do\n job_run.try_to_execute\n assert_equal now, job_run.last_executed_at\n end\n end\n\n it \"executes\" do\n now = Time.utc.at_beginning_of_second\n job_run = PeriodicJobRun.new PeriodicTestJob, interval\n job_run.last_executed_at = now\n\n Timecop.freeze(now + interval) do\n job_run.try_to_execute\n assert_equal now + interval, job_run.last_executed_at\n end\n end\n\n it \"checks the metadata store for the last executed timestamp\" do\n now = Time.utc.at_beginning_of_second\n clean_slate do\n job_run = PeriodicJobRun.new PeriodicTestJob, interval\n job_run.last_executed_at = now - 1.minute\n\n Timecop.freeze(now) do\n another_job_run = PeriodicJobRun.new PeriodicTestJob, interval\n refute another_job_run.try_to_execute\n end\n end\n end\n\n it \"does not enqueue a second job run when one is already pending\" do\n clean_slate do\n now = Time.utc.at_beginning_of_second\n periodic = PeriodicJobRun.new PeriodicTestJob, interval\n\n # First execution should enqueue.\n Timecop.freeze(now) do\n periodic.last_executed_at = now - interval\n assert periodic.try_to_execute\n end\n\n queue = PeriodicTestJob.queue\n first_size = queue.size(include_dead: false)\n assert first_size > 0, \"Expected at least one job in the queue\"\n\n # Second execution after another interval should be skipped\n # because the first job run hasn't finished yet.\n Timecop.freeze(now + interval) do\n assert periodic.try_to_execute\n end\n\n second_size = queue.size(include_dead: false)\n assert_equal first_size, second_size\n end\n end\n\n it \"enqueues again after the pending job run finishes\" do\n clean_slate do\n now = Time.utc.at_beginning_of_second\n periodic = PeriodicJobRun.new PeriodicTestJob, interval\n\n # Enqueue the first job run.\n Timecop.freeze(now) do\n periodic.last_executed_at = now - interval\n periodic.try_to_execute\n end\n\n # Simulate the job finishing by writing finished_at to the backend.\n pending_id = periodic.metadata[\"pending_run_id\"]?\n refute_nil pending_id\n Mosquito.backend.set(\n Mosquito::JobRun.config_key(pending_id.not_nil!),\n \"finished_at\",\n Time.utc.to_unix_ms.to_s\n )\n\n queue = PeriodicTestJob.queue\n size_after_first = queue.size(include_dead: false)\n\n # Now a new interval passes — should enqueue since the previous one finished.\n Timecop.freeze(now + interval) do\n assert periodic.try_to_execute\n end\n\n size_after_second = queue.size(include_dead: false)\n assert size_after_second > size_after_first\n end\n end\n\n it \"enqueues again when the pending job run config has been cleaned up\" do\n clean_slate do\n now = Time.utc.at_beginning_of_second\n periodic = PeriodicJobRun.new PeriodicTestJob, interval\n\n # Enqueue the first job run.\n Timecop.freeze(now) do\n periodic.last_executed_at = now - interval\n periodic.try_to_execute\n end\n\n pending_id = periodic.metadata[\"pending_run_id\"]?\n refute_nil pending_id\n\n # Simulate the job run config being deleted (e.g. TTL expiry).\n Mosquito.backend.delete Mosquito::JobRun.config_key(pending_id.not_nil!)\n\n queue = PeriodicTestJob.queue\n size_before = queue.size(include_dead: false)\n\n # Next interval should enqueue because the old run is gone.\n Timecop.freeze(now + interval) do\n assert periodic.try_to_execute\n end\n\n size_after = queue.size(include_dead: false)\n assert size_after > size_before\n end\n end\nend\n" }, { "path": "spec/mosquito/periodic_job_spec.cr", "content": "require \"../spec_helper\"\n\ndescribe Mosquito::PeriodicJob do\n getter(runner) { Mosquito::TestableRunner.new }\n\n it \"correctly renders job_type\" do\n assert_equal \"periodic_test_job\", PeriodicTestJob.job_type\n end\n\n it \"builds a job_run\" do\n job = PeriodicTestJob.new\n job_run = job.build_job_run\n\n assert_instance_of JobRun, job_run\n assert_equal PeriodicTestJob.job_type, job_run.type\n end\n\n it \"is not reschedulable\" do\n refute PeriodicTestJob.new.rescheduleable?\n end\n\n it \"registers in job mapping\" do\n assert_equal PeriodicTestJob, Base.job_for_type(PeriodicTestJob.job_type)\n end\n\n it \"can be scheduled at a MonthSpan interval\" do\n clean_slate do\n Mosquito::Base.register_job_mapping MonthlyJob.queue.name, MonthlyJob\n Mosquito::Base.register_job_interval MonthlyJob, interval: 1.month\n end\n end\n\n it \"schedules itself for an interval\" do\n clean_slate do\n PeriodicTestJob.run_every 2.minutes\n scheduled_job_run = Base.scheduled_job_runs.first\n assert_equal PeriodicTestJob, scheduled_job_run.class\n assert_equal 2.minutes, scheduled_job_run.interval\n end\n end\nend\n" }, { "path": "spec/mosquito/queue_spec.cr", "content": "require \"../spec_helper\"\n\ndescribe Queue do\n getter(name) { \"test#{rand(1000)}\" }\n\n getter(test_queue) do\n Mosquito::Queue.new(name)\n end\n\n @job_run : Mosquito::JobRun?\n getter(job_run) do\n Mosquito::JobRun.new(\"mock_job_run\").tap(&.store)\n end\n\n getter backend : Mosquito::Backend::Queue do\n TestHelpers.backend.queue name\n end\n\n describe \"config_key\" do\n it \"defaults to name\" do\n name = \"random_name\"\n assert_equal name, Mosquito::Queue.new(name).config_key\n end\n end\n\n describe \"flush\" do\n it \"purges all of the queue entries\" do\n job_runs = (1..4).map do\n Mosquito::JobRun.new(\"mock_job_run\").tap do |job_run|\n job_run.store\n test_queue.enqueue job_run\n end\n end\n\n assert_equal job_runs.size, test_queue.size\n test_queue.flush\n assert_equal 0, test_queue.size\n end\n end\n\n describe \"enqueue\" do\n it \"adds the queue name to the list of queues\" do\n clean_slate do\n test_queue.enqueue job_run\n assert_includes Mosquito.backend.list_queues, test_queue.name\n end\n end\n\n it \"can enqueue a job_run for immediate processing\" do\n clean_slate do\n test_queue.enqueue job_run\n job_run_ids = backend.list_waiting\n assert_includes job_run_ids, job_run.id\n end\n end\n\n it \"can enqueue a job_run with a relative time\" do\n Timecop.freeze(Time.utc) do\n clean_slate do\n offset = 3.seconds\n timestamp = offset.from_now\n test_queue.enqueue job_run, in: offset\n\n stored_time = backend.scheduled_job_run_time job_run\n assert_equal Time.unix_ms(timestamp.to_unix_ms), stored_time\n end\n end\n end\n\n it \"can enqueue a job_run at a specific time\" do\n Timecop.freeze(Time.utc) do\n clean_slate do\n timestamp = 3.seconds.from_now\n test_queue.enqueue job_run, at: timestamp\n stored_time = backend.scheduled_job_run_time job_run\n assert_equal Time.unix_ms(timestamp.to_unix_ms), stored_time\n end\n end\n end\n end\n\n describe \"dequeue\" do\n it \"moves a job_run from waiting to pending on dequeue\" do\n test_queue.enqueue job_run\n stored_job_run = test_queue.dequeue\n\n assert_equal job_run.id, stored_job_run.not_nil!.id\n\n pending_job_runs = backend.list_pending\n assert_includes pending_job_runs, job_run.id\n end\n\n it \"dequeues job_runs which have been scheduled for a time that has passed\" do\n job_run1 = job_run\n job_run2 = Mosquito::JobRun.new(\"mock_job_run\").tap do |job_run|\n job_run.store\n end\n\n Timecop.freeze(Time.utc) do\n past = 1.minute.ago\n future = 1.minute.from_now\n test_queue.enqueue job_run1, at: past\n test_queue.enqueue job_run2, at: future\n end\n\n # check to make sure only job_run1 was dequeued\n overdue_job_runs = test_queue.dequeue_scheduled\n assert_equal 1, overdue_job_runs.size\n assert_equal job_run1.id, overdue_job_runs.first.id\n\n # check to make sure job_run2 is still scheduled\n scheduled_job_runs = backend.list_scheduled\n refute_includes scheduled_job_runs, job_run1.id\n assert_includes scheduled_job_runs, job_run2.id\n end\n end\n\n it \"can forget about a pending job_run\" do\n test_queue.enqueue job_run\n test_queue.dequeue\n pending_job_runs = backend.list_pending\n assert_includes pending_job_runs, job_run.id\n\n test_queue.forget job_run\n pending_job_runs = backend.list_pending\n refute_includes pending_job_runs, job_run.id\n end\n\n describe \"banish\" do\n it \"can banish a pending job_run, adding it to the dead q\" do\n test_queue.enqueue job_run\n test_queue.dequeue\n pending_job_runs = backend.list_pending\n assert_includes pending_job_runs, job_run.id\n\n test_queue.banish job_run\n pending_job_runs = backend.list_pending\n refute_includes pending_job_runs, job_run.id\n\n dead_job_runs = backend.list_dead\n assert_includes dead_job_runs, job_run.id\n end\n end\n\n describe \"pause\" do\n it \"is not paused by default\" do\n refute test_queue.paused?\n end\n\n it \"can be paused\" do\n test_queue.pause\n assert test_queue.paused?\n end\n\n it \"can be resumed\" do\n test_queue.pause\n assert test_queue.paused?\n test_queue.resume\n refute test_queue.paused?\n end\n\n it \"prevents dequeue when paused\" do\n test_queue.enqueue job_run\n test_queue.pause\n\n result = test_queue.dequeue\n assert_nil result\n\n # job_run should still be in waiting, not moved to pending\n waiting_job_runs = backend.list_waiting\n assert_includes waiting_job_runs, job_run.id\n pending_job_runs = backend.list_pending\n refute_includes pending_job_runs, job_run.id\n end\n\n it \"allows dequeue after resume\" do\n test_queue.enqueue job_run\n test_queue.pause\n assert_nil test_queue.dequeue\n\n test_queue.resume\n stored_job_run = test_queue.dequeue\n assert_equal job_run.id, stored_job_run.not_nil!.id\n end\n\n it \"still allows enqueue while paused\" do\n test_queue.pause\n test_queue.enqueue job_run\n waiting_job_runs = backend.list_waiting\n assert_includes waiting_job_runs, job_run.id\n end\n\n it \"can be paused with a duration\" do\n test_queue.pause for: 60.seconds\n assert test_queue.paused?\n end\n\n it \"does not affect other queues\" do\n other_queue = Mosquito::Queue.new(\"other_#{name}\")\n other_job_run = Mosquito::JobRun.new(\"mock_job_run\").tap(&.store)\n\n test_queue.pause\n other_queue.enqueue other_job_run\n\n assert_nil test_queue.dequeue\n stored = other_queue.dequeue\n assert_equal other_job_run.id, stored.not_nil!.id\n end\n end\n\nend\n" }, { "path": "spec/mosquito/queued_job_spec.cr", "content": "require \"../spec_helper\"\n\ndescribe Mosquito::QueuedJob do\n getter(runner) { Mosquito::TestableRunner.new }\n getter(name) { \"test#{rand(1000)}\" }\n getter(job : QueuedTestJob) { QueuedTestJob.new }\n getter(queue : Queue) { QueuedTestJob.queue }\n getter(queue_hooked_job : QueueHookedTestJob) { QueueHookedTestJob.new }\n\n describe \"enqueue\" do\n it \"enqueues\" do\n clean_slate do\n job_run = job.enqueue\n enqueued = queue.backend.list_waiting\n assert_equal [job_run.id], enqueued\n end\n end\n\n it \"enqueues with a delay\" do\n clean_slate do\n job_run = job.enqueue in: 1.minute\n enqueued = queue.backend.list_scheduled\n assert_equal [job_run.id], enqueued\n end\n end\n\n it \"enqueues with a target time\" do\n clean_slate do\n job_run = job.enqueue at: 1.minute.from_now\n enqueued = queue.backend.list_scheduled\n assert_equal [job_run.id], enqueued\n end\n end\n\n it \"fires before_enqueue_hook\" do\n clean_slate do\n job_run = queue_hooked_job.enqueue\n assert queue_hooked_job.before_hook_ran\n end\n end\n\n it \"doesnt enqueue if before_enqueue_hook fails\" do\n clean_slate do\n queue_hooked_job.fail_before_hook = true\n job_run = queue_hooked_job.enqueue\n waiting_q = queue.backend.list_waiting\n assert_empty waiting_q\n end\n end\n\n it \"fires after_enqueue_hook\" do\n clean_slate do\n job_run = queue_hooked_job.enqueue\n assert queue_hooked_job.after_hook_ran\n end\n end\n\n it \"passes the job config to the before_enqueue_hook\" do\n clean_slate do\n job_run = queue_hooked_job.enqueue\n assert_equal job_run, queue_hooked_job.passed_job_config\n end\n end\n\n it \"passes the job config to the after_enqueue_hook\" do\n clean_slate do\n job_run = queue_hooked_job.enqueue\n assert_equal job_run, queue_hooked_job.passed_job_config\n end\n end\n end\n\n describe \"parameters\" do\n it \"can be passed in\" do\n clear_logs\n EchoJob.new(\"quack\").perform\n assert_logs_match \"quack\"\n end\n\n it \"can have a boolean false passed as a parameter (and it's not assumed to be a nil)\" do\n clear_logs\n JobWithHooks.new(false).perform\n assert_includes logs, \"Perform Executed\"\n end\n\n it \"can be omitted\" do\n clean_slate do\n clear_logs\n job = JobWithNoParams.new.perform\n assert_includes logs, \"no param job performed\"\n end\n end\n end\nend\n" }, { "path": "spec/mosquito/rate_limiter_spec.cr", "content": "require \"../spec_helper\"\n\ndescribe Mosquito::RateLimiter do\n describe \"RateLimiter.rate_limit_stats\" do\n it \"provides the state and configuration of the limiter\" do\n clean_slate do\n stats = RateLimitedJob.rate_limit_stats\n assert stats.has_key? :interval\n assert stats.has_key? :key\n assert stats.has_key? :increment\n assert stats.has_key? :limit\n assert stats.has_key? :window_start\n assert stats.has_key? :run_count\n end\n end\n\n it \"defaults the window_start\" do\n clean_slate do\n assert_equal Time::UNIX_EPOCH, RateLimitedJob.rate_limit_stats[:window_start]\n\n now = Time.utc.at_beginning_of_second\n RateLimitedJob.metadata[\"window_start\"] = now.to_unix.to_s\n assert_equal now, RateLimitedJob.rate_limit_stats[:window_start]\n end\n end\n\n it \"defaults the run_count\" do\n clean_slate do\n assert_equal 0, RateLimitedJob.rate_limit_stats[:run_count]\n\n run_count = 27\n RateLimitedJob.metadata[\"run_count\"] = run_count.to_s\n assert_equal run_count, RateLimitedJob.rate_limit_stats[:run_count]\n end\n end\n end\n\n describe \"RateLimiter.metadata\" do\n it \"provides an instance of the metadata store\" do\n assert_instance_of Metadata, RateLimitedJob.metadata\n end\n end\n\n describe \"RateLimiter.rate_limit_key\" do\n it \"provides the metadata key for this class\" do\n assert_equal \"mosquito:rate_limit:rate_limit\", RateLimitedJob.rate_limit_key\n end\n end\n\n describe \"job counting\" do\n it \"increments the count when a job is run\" do\n clean_slate do\n RateLimitedJob.new.run\n count = RateLimitedJob.metadata[\"run_count\"]?.not_nil!.to_i\n\n RateLimitedJob.new.run\n new_count = RateLimitedJob.metadata[\"run_count\"]?.not_nil!.to_i\n assert_equal 1, new_count - count\n end\n end\n\n it \"doesnt increment the count when a job is not run\" do\n clean_slate do\n RateLimitedJob.new(should_fail: false).run\n count = RateLimitedJob.metadata[\"run_count\"]?.not_nil!.to_i\n\n RateLimitedJob.new(should_fail: true).run\n new_count = RateLimitedJob.metadata[\"run_count\"]?.not_nil!.to_i\n assert_equal count, new_count\n end\n end\n\n it \"increments the count by a configurable number\" do\n clean_slate do\n delta = 2\n RateLimitedJob.new.run\n count = RateLimitedJob.metadata[\"run_count\"]?.not_nil!.to_i\n\n RateLimitedJob.new(increment: delta).run\n new_count = RateLimitedJob.metadata[\"run_count\"]?.not_nil!.to_i\n assert_equal delta, new_count - count\n end\n end\n\n it \"resets the count when the window is over\" do\n clean_slate do\n metadata = RateLimitedJob.metadata\n metadata[\"run_count\"] = \"45\"\n metadata[\"window_start\"] = Time::UNIX_EPOCH.to_unix.to_s\n RateLimitedJob.new.run\n count = RateLimitedJob.metadata[\"run_count\"]?\n assert_equal \"1\", count\n end\n end\n\n it \"counts multiple jobs with the same key in the same bucket\" do\n clean_slate do\n metadata = RateLimitedJob.metadata\n metadata[\"window_start\"] = Time.utc.to_unix.to_s\n\n RateLimitedJob.new.run\n count = RateLimitedJob.metadata[\"run_count\"]?.not_nil!.to_i\n\n SecondRateLimitedJob.new.run\n new_count = RateLimitedJob.metadata[\"run_count\"]?.not_nil!.to_i\n\n assert_equal RateLimitedJob.rate_limit_key, SecondRateLimitedJob.rate_limit_key\n assert_equal 1, new_count - count\n end\n end\n end\n\n describe \"job preempting\" do\n it \"doesnt prevent excution if the rate limit count is less than zero\" do\n metadata = RateLimitedJob.metadata\n metadata[\"run_count\"] = \"-1\"\n metadata[\"window_start\"] = Time.utc.to_unix.to_s\n job = RateLimitedJob.new\n job.run\n assert job.executed?\n end\n\n it \"prevents a job from executing when the limit is reached\" do\n metadata = RateLimitedJob.metadata\n metadata[\"run_count\"] = Int32::MAX.to_s\n metadata[\"window_start\"] = Time.utc.to_unix.to_s\n job = RateLimitedJob.new\n job.run\n refute job.executed?\n assert job.preempted?\n end\n\n it \"allows a job to execute when the limit hasn't been reached\" do\n metadata = RateLimitedJob.metadata\n metadata[\"window_start\"] = Time.utc.to_unix.to_s\n metadata[\"run_count\"] = \"3\"\n job = RateLimitedJob.new\n job.run\n assert job.executed?\n end\n\n it \"allows a job to execute when the limit has been reached but the window is over\" do\n metadata = RateLimitedJob.metadata\n metadata[\"run_count\"] = Int32::MAX.to_s\n metadata[\"window_start\"] = Time::UNIX_EPOCH.to_unix.to_s\n job = RateLimitedJob.new\n job.run\n assert job.executed?\n end\n end\nend\n" }, { "path": "spec/mosquito/resource_gate_spec.cr", "content": "require \"../spec_helper\"\n\ndescribe \"Mosquito::OpenGate\" do\n it \"always allows\" do\n gate = Mosquito::OpenGate.new\n assert gate.allow?\n end\nend\n\ndescribe \"Mosquito::ThresholdGate\" do\n it \"allows when metric is below threshold\" do\n gate = Mosquito::ThresholdGate.new(threshold: 80.0, sample_ttl: 0.seconds) { 50.0 }\n assert gate.allow?\n end\n\n it \"blocks when metric is at or above threshold\" do\n gate = Mosquito::ThresholdGate.new(threshold: 80.0, sample_ttl: 0.seconds) { 85.0 }\n refute gate.allow?\n end\n\n it \"blocks when metric equals threshold\" do\n gate = Mosquito::ThresholdGate.new(threshold: 80.0, sample_ttl: 0.seconds) { 80.0 }\n refute gate.allow?\n end\nend\n\ndescribe \"Mosquito::ResourceGate caching\" do\n it \"caches the check result within TTL\" do\n call_count = 0\n gate = Mosquito::ThresholdGate.new(threshold: 80.0, sample_ttl: 5.seconds) do\n call_count += 1\n 50.0\n end\n\n now = Time.utc\n Timecop.freeze(now) do\n gate.allow?\n gate.allow?\n gate.allow?\n assert_equal 1, call_count\n end\n end\n\n it \"re-checks after TTL expires\" do\n call_count = 0\n gate = Mosquito::ThresholdGate.new(threshold: 80.0, sample_ttl: 5.seconds) do\n call_count += 1\n 50.0\n end\n\n now = Time.utc\n Timecop.freeze(now) do\n gate.allow?\n assert_equal 1, call_count\n end\n\n Timecop.freeze(now + 3.seconds) do\n gate.allow?\n assert_equal 1, call_count, \"Should still be cached at 3s\"\n end\n\n Timecop.freeze(now + 6.seconds) do\n gate.allow?\n assert_equal 2, call_count, \"Should re-check after 6s (past 5s TTL)\"\n end\n end\nend\n" }, { "path": "spec/mosquito/runnable_spec.cr", "content": "require \"../spec_helper\"\n\nclass Namespace::ConcreteRunnable\n include Mosquito::Runnable\n\n getter first_run_notifier = Channel(Bool).new\n getter first_run = true\n property state : Mosquito::Runnable::State\n\n # Testing wedge which calls: run, waits for a run to happen, and then calls stop.\n def test_run : Nil\n run\n first_run_notifier.receive\n stop.wait\n end\n\n def runnable_name : String\n \"concrete_runnable\"\n end\n\n def each_run : Nil\n if first_run\n @first_run = false\n first_run_notifier.send true\n end\n Fiber.yield\n end\n\n def stop(wait_group : WaitGroup = WaitGroup.new(1)) : WaitGroup\n first_run_notifier.close\n super(wait_group)\n end\nend\n\ndescribe Mosquito::Runnable do\n let(:runnable) { Namespace::ConcreteRunnable.new }\n\n it \"builds a my_name\" do\n assert_equal \"namespace.concrete_runnable.#{runnable.object_id}\", runnable.my_name\n end\n\n describe \"run\" do\n it \"should log a startup message\" do\n clear_logs\n runnable.test_run\n assert_logs_match \"mosquito.concrete_runnable\", \"starting\"\n end\n\n it \"should log a finished message\" do\n clear_logs\n runnable.test_run\n assert_logs_match \"mosquito.concrete_runnable\", \"stopped\"\n end\n end\n\n describe \"stop\" do\n it \"should set the stopping flag\" do\n runnable.state = Mosquito::Runnable::State::Working\n runnable.stop\n assert_equal Mosquito::Runnable::State::Stopping, runnable.state\n end\n\n it \"should set the finished flag\" do\n runnable.test_run\n assert_equal Mosquito::Runnable::State::Finished, runnable.state\n end\n end\nend\n" }, { "path": "spec/mosquito/runners/coordinator_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe \"Mosquito::Runners::Coordinator\" do\n getter(queue : Queue) { test_job.class.queue }\n getter(test_job) { QueuedTestJob.new }\n getter(queue_list) { MockQueueList.new }\n getter(coordinator) { MockCoordinator.new queue_list }\n getter(enqueue_time) { Time.utc }\n\n def enqueue_job_run : JobRun\n queue_list.discovered_queues << queue\n\n job_run = JobRun.new \"blah\"\n\n Timecop.freeze enqueue_time do |t|\n job_run = test_job.enqueue in: 3.seconds\n end\n\n assert_includes queue.backend.list_scheduled, job_run.id\n job_run\n end\n\n def opt_in_to_locking\n Mosquito.temp_config(use_distributed_lock: true) do\n Mosquito.backend.delete Mosquito.backend.build_key(:coordinator, :leadership_lock)\n yield\n Mosquito.backend.delete Mosquito.backend.build_key(:coordinator, :leadership_lock)\n end\n end\n\n describe \"only_if_coordinator\" do\n getter(coordinator1) { Mosquito::Runners::Coordinator.new queue_list }\n getter(coordinator2) { Mosquito::Runners::Coordinator.new queue_list }\n\n it \"gets a lock from the backend\" do\n opt_in_to_locking do\n gotten = false\n\n coordinator1.only_if_coordinator do\n gotten = true\n end\n\n assert gotten\n end\n end\n\n it \"fails to get a lock from the backend\" do\n opt_in_to_locking do\n gotten = false\n\n coordinator1.only_if_coordinator do\n coordinator2.only_if_coordinator do\n gotten = true\n end\n end\n\n refute gotten\n end\n end\n\n it \"releases the lock when release_leadership_lock is called\" do\n opt_in_to_locking do\n gotten = false\n\n coordinator1.only_if_coordinator do\n end\n\n coordinator1.release_leadership_lock\n\n coordinator2.only_if_coordinator do\n gotten = true\n end\n\n assert gotten\n end\n end\n\n it \"sets a ttl on the lock\" do\n opt_in_to_locking do\n coordinator1.only_if_coordinator do\n assert Mosquito.backend.expires_in(coordinator.lock_key) > 0\n end\n end\n end\n\n it \"retains leadership across calls\" do\n opt_in_to_locking do\n count = 0\n\n 3.times do\n coordinator1.only_if_coordinator do\n count += 1\n end\n end\n\n assert_equal 3, count\n assert coordinator1.is_leader?\n end\n end\n\n it \"yields without locking when distributed lock is disabled\" do\n Mosquito.temp_config(use_distributed_lock: false) do\n gotten = false\n\n coordinator1.only_if_coordinator do\n gotten = true\n end\n\n assert gotten\n end\n end\n end\n\n describe \"enqueue_periodic_jobs\" do\n it \"enqueues a scheduled job_run at the appropriate time\" do\n clean_slate do\n queue = PeriodicTestJob.queue\n Mosquito::Base.register_job_mapping PeriodicTestJob.name, PeriodicTestJob\n Mosquito::Base.register_job_interval PeriodicTestJob, interval: 1.second\n\n Timecop.freeze(enqueue_time) do\n coordinator.enqueue_periodic_jobs\n end\n\n queued_job_runs = queue.backend.list_waiting\n assert queued_job_runs.size >= 1\n\n last_job_run = queued_job_runs.last\n job_run_metadata = Mosquito.backend.retrieve JobRun.config_key(last_job_run)\n\n assert_equal enqueue_time.to_unix_ms.to_s, job_run_metadata[\"enqueue_time\"]\n end\n end\n end\n\n describe \"enqueue_delayed_jobs\" do\n it \"enqueues a delayed job_run when it's ready\" do\n clean_slate do\n job_run = enqueue_job_run\n run_time = enqueue_time + 3.seconds\n\n Timecop.freeze run_time do |t|\n coordinator.enqueue_delayed_jobs\n end\n\n queued_job_runs = queue.backend.list_waiting\n assert_includes queued_job_runs, job_run.id\n\n last_job_run = queued_job_runs.last\n job_run_metadata = Mosquito.backend.retrieve JobRun.config_key(last_job_run)\n\n assert_equal queue.name, job_run_metadata[\"type\"]?\n end\n end\n\n it \"doesn't enqueue job_runs that arent ready yet\" do\n clean_slate do\n job_run = enqueue_job_run\n\n check_time = enqueue_time + 2.999.seconds\n\n Timecop.freeze check_time do |t|\n coordinator.enqueue_delayed_jobs\n end\n\n queued_job_runs = queue.backend.list_waiting\n\n # does not deschedule and enqueue anything\n assert_equal 0, queued_job_runs.size\n end\n end\n\n it \"logs when it finds delayed job_runs\" do\n clean_slate do\n clear_logs\n enqueue_job_run\n Timecop.freeze enqueue_time + 3.seconds do |t|\n coordinator.enqueue_delayed_jobs\n end\n assert_logs_match \"1 delayed jobs ready\"\n end\n end\n\n end\nend\n" }, { "path": "spec/mosquito/runners/executor_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe \"Mosquito::Runners::Executor\" do\n getter(queue_list) { MockQueueList.new }\n getter(overseer) { MockOverseer.new }\n getter(executor) { MockExecutor.new overseer.as(Mosquito::Runners::Overseer) }\n getter(api) { Mosquito::Api::Executor.new executor.object_id.to_s }\n getter(coordinator) { Mosquito::Runners::Coordinator.new queue_list }\n\n def register(job_class : Mosquito::Job.class)\n Mosquito::Base.register_job_mapping job_class.name.underscore, job_class\n queue_list.discovered_queues << job_class.queue\n end\n\n def run_job(job_class : Mosquito::Job.class)\n register job_class\n job_class.reset_performance_counter!\n job_run = job_class.new.enqueue\n executor.work_unit = WorkUnit.of(job_run, from: job_class.queue)\n executor.execute\n end\n\n describe \"status\" do\n it \"starts as starting\" do\n assert_equal Runnable::State::Starting, executor.state\n end\n\n it \"broadcasts a ping when transitioning to idle\" do\n executor.state = Runnable::State::Idle\n\n select\n when overseer.finished_notifier.receive\n assert true\n when timeout(0.5.seconds)\n refute true, \"Timed out waiting for idle notifier\"\n end\n end\n\n it \"goes idle in pre_run\" do\n executor.pre_run\n assert_equal Runnable::State::Idle, executor.state\n end\n end\n\n describe \"running jobs\" do\n it \"runs a job from a queue\" do\n clean_slate do\n run_job QueuedTestJob\n assert_equal 1, QueuedTestJob.performances\n end\n end\n\n it \"reschedules a job that failed\" do\n clean_slate do\n register FailingJob\n now = Time.utc\n\n job = FailingJob.new\n job_run = job.build_job_run\n job_run.store\n FailingJob.queue.enqueue job_run\n\n Timecop.freeze now do\n executor.work_unit = WorkUnit.of(job_run, from: FailingJob.queue)\n executor.execute\n end\n\n job_run.reload\n assert_equal 1, job_run.retry_count\n\n Timecop.freeze now + job.reschedule_interval(1) do\n coordinator.enqueue_delayed_jobs\n executor.work_unit = WorkUnit.of(job_run, from: FailingJob.queue)\n executor.execute\n end\n\n job_run.reload\n assert_equal 2, job_run.retry_count\n end\n end\n\n it \"schedules deletion of a job_run that hard failed\" do\n clean_slate do\n register NonReschedulableFailingJob\n\n job = NonReschedulableFailingJob.new\n job_run = job.build_job_run\n job_run.store\n NonReschedulableFailingJob.queue.enqueue job_run\n\n executor.work_unit = WorkUnit.of(job_run, from: NonReschedulableFailingJob.queue)\n executor.execute\n\n actual_ttl = backend.expires_in job_run.config_key\n assert_equal executor.failed_job_ttl, actual_ttl\n end\n end\n\n it \"purges a successful job_run from the backend\" do\n clean_slate do\n register QueuedTestJob\n\n job = QueuedTestJob.new\n job_run = job.build_job_run\n job_run.store\n QueuedTestJob.queue.enqueue job_run\n\n executor.work_unit = WorkUnit.of(job_run, from: QueuedTestJob.queue)\n executor.execute\n\n assert_logs_match \"Success\"\n\n QueuedTestJob.queue.enqueue job_run\n actual_ttl = Mosquito.backend.expires_in job_run.config_key\n assert_equal executor.successful_job_ttl, actual_ttl\n end\n end\n\n it \"doesnt reschedule a job that cant be rescheduled\" do\n clean_slate do\n run_job NonReschedulableFailingJob\n assert_logs_match \"cannot be rescheduled\"\n end\n end\n\n it \"tells the observer what it's working on\" do\n SleepyJob.should_sleep = true\n job = SleepyJob.new\n job_run = job.build_job_run\n job_run.store\n\n job_started = Channel(Bool).new\n job_finished = Channel(Bool).new\n\n # Eagerly evaluate to avoid race condition in lazy\n # getter initialization across fibers.\n executor\n api\n\n spawn {\n executor.work_unit = WorkUnit.of(job_run, from: SleepyJob.queue)\n executor.execute\n job_finished.send true\n }\n\n spawn {\n loop {\n break if api.current_job\n }\n assert_equal job_run.id, api.current_job\n assert_equal SleepyJob.queue.name, api.current_job_queue\n job_started.send true\n }\n\n select\n when job_started.receive\n when timeout(0.5.seconds)\n refute true, \"Timed out waiting for job to start\"\n end\n\n SleepyJob.should_sleep = false\n\n select\n when job_finished.receive\n when timeout(0.5.seconds)\n refute true, \"Timed out waiting for job to finish\"\n end\n\n assert_nil api.current_job, \"Job should be cleared after finishing\"\n assert_nil api.current_job_queue, \"Queue should be cleared after finishing\"\n end\n end\n\n describe \"logs success/failures messages\" do\n it \"logs a success message when the job succeeds\" do\n clean_slate do\n run_job QueuedTestJob\n assert_logs_match \"Success\"\n end\n end\n\n it \"logs a failure message when the job fails\" do\n clean_slate do\n run_job FailingJob\n assert_logs_match \"Failure\"\n end\n end\n end\n\n describe \"job timing messages\" do\n it \"logs the time a job took to run\" do\n clean_slate do\n run_job QueuedTestJob\n assert_logs_match \"and took\"\n end\n end\n\n it \"logs the time a job took to run when the job fails\" do\n clean_slate do\n run_job FailingJob\n assert_logs_match \"taking\"\n end\n end\n end\n\n describe \"start and finish messages\" do\n it \"logs the job run start message\" do\n clean_slate do\n run_job QueuedTestJob\n assert_logs_match \"Starting: queued_test_job\"\n end\n end\n end\nend\n" }, { "path": "spec/mosquito/runners/overseer_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe \"Mosquito::Runners::Overseer\" do\n getter(overseer : MockOverseer) { MockOverseer.new }\n getter(queue_list : MockQueueList ) { overseer.queue_list.as(MockQueueList) }\n getter(coordinator : MockCoordinator ) { overseer.coordinator.as(MockCoordinator) }\n getter(executor : MockExecutor) { overseer.executors.first.as(MockExecutor) }\n\n def register(job_class : Mosquito::Job.class)\n Mosquito::Base.register_job_mapping job_class.name.underscore, job_class\n queue_list.discovered_queues << job_class.queue\n end\n\n def run_job(job_class : Mosquito::Job.class)\n register job_class\n job_class.reset_performance_counter!\n job_run = job_class.new.enqueue\n executor.execute job_run, from_queue: job_class.queue\n end\n\n describe \"pre_run\" do\n it \"runs all executors\" do\n overseer.executors.each do |executor|\n assert_equal Runnable::State::Starting, executor.state\n end\n overseer.pre_run\n overseer.executors.each do |executor|\n assert_equal Runnable::State::Working, executor.state\n end\n end\n end\n\n describe \"post_run\" do\n it \"stops all executors\" do\n overseer.executors.each(&.run)\n overseer.post_run\n overseer.executors.each do |executor|\n assert_equal Runnable::State::Finished, executor.state\n end\n end\n\n it \"logs messages about stopping the executors\" do\n clear_logs\n overseer.pre_run\n overseer.post_run\n assert_logs_match \"Stopping executors.\"\n assert_logs_match \"All executors stopped.\"\n end\n end\n\n describe \"each_run\" do\n it \"dequeues a job and dispatches it to the pipeline\" do\n clean_slate do\n register QueuedTestJob\n expected_job_run = QueuedTestJob.new.enqueue\n\n overseer.work_handout = Channel(WorkUnit).new\n\n queue_list.state = Runnable::State::Working\n executor.state = Runnable::State::Idle\n\n # each_run will block until there's a receiver on the channel\n spawn { overseer.each_run }\n result = overseer.work_handout.receive\n assert_equal expected_job_run, result.job_run\n assert_equal QueuedTestJob.queue, result.queue\n end\n end\n\n it \"waits #idle_wait before checking the queue again\" do\n clean_slate do\n # an idle executor, but no jobs in the queue\n executor.state = Runnable::State::Idle\n queue_list.state = Runnable::State::Working\n\n tick_time = Time.measure do\n overseer.each_run\n end\n\n assert tick_time >= overseer.idle_wait, \"Expected to wait at least #{overseer.idle_wait}, but only waited #{tick_time}\"\n end\n end\n\n it \"triggers the scheduler\" do\n assert_equal 0, coordinator.schedule_count\n overseer.each_run\n assert_equal 1, coordinator.schedule_count\n end\n end\n\n describe \"dequeue_job? stamps overseer_id\" do\n it \"claims the job run with the overseer's instance id on dequeue\" do\n clean_slate do\n register QueuedTestJob\n job_run = QueuedTestJob.new.enqueue\n\n queue_list.state = Runnable::State::Working\n\n result = overseer.dequeue_job?\n assert result\n assert_equal overseer.observer.instance_id, result.not_nil!.job_run.overseer_id\n end\n end\n end\n\n describe \"remote executor count\" do\n it \"applies the remote executor count on each_run\" do\n clean_slate do\n Mosquito.configuration.overseer_id = \"test-worker\"\n Mosquito::Api.set_executor_count(3, overseer_id: \"test-worker\")\n\n queue_list.state = Runnable::State::Working\n overseer.each_run\n\n assert_equal 3, overseer.executor_count\n ensure\n Mosquito.configuration.overseer_id = nil\n end\n end\n\n it \"prefers per-overseer count over global\" do\n clean_slate do\n Mosquito.configuration.overseer_id = \"test-worker\"\n Mosquito::Api.set_executor_count(10)\n Mosquito::Api.set_executor_count(2, overseer_id: \"test-worker\")\n\n queue_list.state = Runnable::State::Working\n overseer.each_run\n\n assert_equal 2, overseer.executor_count\n ensure\n Mosquito.configuration.overseer_id = nil\n end\n end\n\n it \"falls back to global when no per-overseer count is set\" do\n clean_slate do\n Mosquito.configuration.overseer_id = \"test-worker\"\n Mosquito::Api.set_executor_count(7)\n\n queue_list.state = Runnable::State::Working\n overseer.each_run\n\n assert_equal 7, overseer.executor_count\n ensure\n Mosquito.configuration.overseer_id = nil\n end\n end\n\n it \"does not change executor_count when no remote value is set\" do\n clean_slate do\n original_count = overseer.executor_count\n\n queue_list.state = Runnable::State::Working\n overseer.each_run\n\n assert_equal original_count, overseer.executor_count\n end\n end\n\n it \"clamps an invalid remote executor count of 0 to 1\" do\n clean_slate do\n Mosquito.configuration.overseer_id = \"test-worker\"\n Mosquito::Api.set_executor_count(0, overseer_id: \"test-worker\")\n\n queue_list.state = Runnable::State::Working\n overseer.each_run\n\n assert_equal 1, overseer.executor_count\n ensure\n Mosquito.configuration.overseer_id = nil\n end\n end\n end\n\n describe \"cleanup_orphaned_pending_jobs\" do\n it \"recovers a pending job whose overseer is dead\" do\n clean_slate do\n register QueuedTestJob\n\n # Use a separate overseer that won't be registered as alive.\n dead_overseer = MockOverseer.new\n\n job = QueuedTestJob.new\n job_run = job.build_job_run\n job_run.store\n QueuedTestJob.queue.enqueue job_run\n QueuedTestJob.queue.dequeue\n job_run.claimed_by dead_overseer\n\n # Verify job is stuck in pending\n assert_includes QueuedTestJob.queue.backend.list_pending, job_run.id\n assert_equal 0, job_run.retry_count\n\n # Register only the *live* overseer\n Mosquito.backend.register_overseer overseer.observer.instance_id\n\n # Run cleanup — dead_overseer's id won't be in the active set\n overseer.cleanup_orphaned_pending_jobs\n\n # Job should be removed from pending and rescheduled\n assert_empty QueuedTestJob.queue.backend.list_pending\n assert_includes QueuedTestJob.queue.backend.list_scheduled, job_run.id\n\n # Retry count should be incremented\n job_run.reload\n assert_equal 1, job_run.retry_count\n end\n end\n\n it \"does not touch pending jobs from a live overseer\" do\n clean_slate do\n register QueuedTestJob\n\n job = QueuedTestJob.new\n job_run = job.build_job_run\n job_run.store\n QueuedTestJob.queue.enqueue job_run\n QueuedTestJob.queue.dequeue\n\n # Claim with the live overseer\n Mosquito.backend.register_overseer overseer.observer.instance_id\n job_run.claimed_by overseer\n\n assert_includes QueuedTestJob.queue.backend.list_pending, job_run.id\n\n overseer.cleanup_orphaned_pending_jobs\n\n # Job should still be in pending — its overseer is alive\n assert_includes QueuedTestJob.queue.backend.list_pending, job_run.id\n end\n end\n\n it \"claims unclaimed pending jobs without recovering them\" do\n clean_slate do\n register QueuedTestJob\n\n job = QueuedTestJob.new\n job_run = job.build_job_run\n job_run.store\n QueuedTestJob.queue.enqueue job_run\n QueuedTestJob.queue.dequeue\n\n # No claim — simulates a job from before this feature\n assert_nil job_run.overseer_id\n assert_includes QueuedTestJob.queue.backend.list_pending, job_run.id\n\n Mosquito.backend.register_overseer overseer.observer.instance_id\n overseer.cleanup_orphaned_pending_jobs\n\n # Job should still be in pending (not recovered)\n assert_includes QueuedTestJob.queue.backend.list_pending, job_run.id\n\n # But it should now be claimed by this overseer\n job_run.reload\n assert_equal overseer.observer.instance_id, job_run.overseer_id\n end\n end\n\n it \"banishes an orphaned job that has exhausted retries\" do\n clean_slate do\n register QueuedTestJob\n\n dead_overseer = MockOverseer.new\n\n # Create a job_run with retry_count=4 so the next failure (count=5)\n # exceeds the default rescheduleable? limit of < 5.\n job_run = Mosquito::JobRun.new(\"queued_test_job\", retry_count: 4)\n job_run.store\n\n QueuedTestJob.queue.enqueue job_run\n QueuedTestJob.queue.dequeue\n job_run.claimed_by dead_overseer\n\n assert_includes QueuedTestJob.queue.backend.list_pending, job_run.id\n\n Mosquito.backend.register_overseer overseer.observer.instance_id\n overseer.cleanup_orphaned_pending_jobs\n\n # Job should be removed from pending and moved to dead\n assert_empty QueuedTestJob.queue.backend.list_pending\n assert_empty QueuedTestJob.queue.backend.list_waiting\n assert_empty QueuedTestJob.queue.backend.list_scheduled\n assert_includes QueuedTestJob.queue.backend.list_dead, job_run.id\n end\n end\n end\nend\n" }, { "path": "spec/mosquito/runners/queue_list_spec.cr", "content": "require \"../../spec_helper\"\n\ndescribe \"Mosquito::Runners::QueueList\" do\n getter(queue_list) { MockQueueList.new }\n\n def enqueue_jobs\n PassingJob.new.enqueue\n FailingJob.new.enqueue\n EchoJob.new(text: \"hello world\").enqueue\n end\n\n describe \"each_run\" do\n it \"returns a list of queues\" do\n clean_slate do\n enqueue_jobs\n queue_list.each_run\n assert_equal [\"failing_job\", \"io_queue\", \"passing_job\"], queue_list.queues.map(&.name).sort\n end\n end\n\n it \"logs a message about the number of fetched queues\" do\n clean_slate do\n clear_logs\n enqueue_jobs\n queue_list.each_run\n assert_logs_match \"found 3 new queues\"\n end\n end\n end\n\n describe \"queue filtering\" do\n it \"filters the list of queues when a whitelist is present\" do\n clean_slate do\n enqueue_jobs\n\n Mosquito.temp_config(run_from: [\"io_queue\", \"passing_job\"]) do\n queue_list.each_run\n end\n end\n\n assert_equal [\"io_queue\", \"passing_job\"], queue_list.queues.map(&.name).sort\n end\n\n it \"logs an error when all queues are filtered out\" do\n clean_slate do\n enqueue_jobs\n\n Mosquito.temp_config(run_from: [\"test4\"]) do\n queue_list.each_run\n end\n\n assert_logs_match \"No watchable queues found.\"\n end\n end\n\n it \"doesnt log an error when no queues are present\" do\n clean_slate do\n queue_list.each_run\n refute_logs_match \"No watchable queues found.\"\n end\n end\n end\n\n describe \"paused queue filtering\" do\n it \"excludes paused queues from the queue list\" do\n clean_slate do\n enqueue_jobs\n Mosquito::Queue.new(\"passing_job\").pause\n queue_list.each_run\n assert_equal [\"failing_job\", \"io_queue\"], queue_list.queues.map(&.name).sort\n end\n end\n\n it \"logs a message about paused queues\" do\n clean_slate do\n clear_logs\n enqueue_jobs\n Mosquito::Queue.new(\"passing_job\").pause\n queue_list.each_run\n assert_logs_match \"1 paused queues: passing_job\"\n end\n end\n\n it \"includes queues again after they are resumed\" do\n clean_slate do\n enqueue_jobs\n q = Mosquito::Queue.new(\"passing_job\")\n q.pause\n queue_list.each_run\n refute_includes queue_list.queues.map(&.name), \"passing_job\"\n\n q.resume\n queue_list.each_run\n assert_includes queue_list.queues.map(&.name), \"passing_job\"\n end\n end\n end\n\n describe \"resource gate filtering\" do\n it \"excludes queues whose gate blocks\" do\n clean_slate do\n enqueue_jobs\n queue_list.each_run\n\n gate = Mosquito::ThresholdGate.new(threshold: 80.0, sample_ttl: 0.seconds) { 90.0 }\n queue_list.resource_gates = {\"passing_job\" => gate.as(Mosquito::ResourceGate)}\n\n refute_includes queue_list.queues.map(&.name), \"passing_job\"\n assert_includes queue_list.queues.map(&.name), \"failing_job\"\n assert_includes queue_list.queues.map(&.name), \"io_queue\"\n end\n end\n\n it \"includes queues whose gate allows\" do\n clean_slate do\n enqueue_jobs\n queue_list.each_run\n\n gate = Mosquito::ThresholdGate.new(threshold: 80.0, sample_ttl: 0.seconds) { 50.0 }\n queue_list.resource_gates = {\"passing_job\" => gate.as(Mosquito::ResourceGate)}\n\n assert_includes queue_list.queues.map(&.name), \"passing_job\"\n end\n end\n\n it \"ungated queues are always included\" do\n clean_slate do\n enqueue_jobs\n queue_list.each_run\n\n gate = Mosquito::ThresholdGate.new(threshold: 80.0, sample_ttl: 0.seconds) { 90.0 }\n queue_list.resource_gates = {\"passing_job\" => gate.as(Mosquito::ResourceGate)}\n\n assert_equal 2, queue_list.queues.size\n end\n end\n\n it \"multiple queues can share a gate\" do\n clean_slate do\n enqueue_jobs\n queue_list.each_run\n\n gate = Mosquito::ThresholdGate.new(threshold: 80.0, sample_ttl: 0.seconds) { 90.0 }\n queue_list.resource_gates = {\n \"passing_job\" => gate.as(Mosquito::ResourceGate),\n \"failing_job\" => gate.as(Mosquito::ResourceGate),\n }\n\n assert_equal [\"io_queue\"], queue_list.queues.map(&.name)\n end\n end\n\n it \"gate state is evaluated on each access\" do\n clean_slate do\n enqueue_jobs\n queue_list.each_run\n\n value = 90.0\n gate = Mosquito::ThresholdGate.new(threshold: 80.0, sample_ttl: 0.seconds) { value }\n queue_list.resource_gates = {\"passing_job\" => gate.as(Mosquito::ResourceGate)}\n\n refute_includes queue_list.queues.map(&.name), \"passing_job\"\n\n value = 50.0\n assert_includes queue_list.queues.map(&.name), \"passing_job\"\n end\n end\n\n it \"returns all queues when no gates are configured\" do\n clean_slate do\n enqueue_jobs\n queue_list.each_run\n\n assert_equal 3, queue_list.queues.size\n end\n end\n end\nend\n" }, { "path": "spec/mosquito/runners/run_at_most_spec.cr", "content": "require \"../../spec_helper\"\n\nclass RunsAtMostMock\n include Mosquito::Runners::RunAtMost\n\n def yield_once_a_second(&block)\n run_at_most every: 1.second, label: :testing do |t|\n yield\n end\n end\nend\n\ndescribe \"Mosquito::yielder#run_at_most\" do\n getter(yielder) { RunsAtMostMock.new }\n\n it \"prevents throttled blocks from running too often\" do\n count = 0\n\n 2.times do\n yielder.yield_once_a_second do\n count += 1\n end\n end\n\n assert_equal 1, count\n end\n\n it \"allows throttled blocks to run only after enough time has passed\" do\n count = 0\n moment = Time.utc\n yielder\n incrementy = ->() do\n yielder.yield_once_a_second do\n count += 1\n end\n end\n\n # Should increment\n Timecop.freeze moment do |time|\n incrementy.call\n end\n\n # Should not increment\n # Move ahead 0.999 seconds\n Timecop.freeze(moment + 999.milliseconds) do |time|\n incrementy.call\n end\n\n assert_equal 1, count\n\n # Should increment\n # Move ahead the rest of the second\n moment += 1.1.seconds\n Timecop.freeze(moment) do |time|\n incrementy.call\n end\n\n assert_equal 2, count\n\n # Should not increment\n # Try again and it shouldn't increment\n Timecop.freeze(moment) do |time|\n incrementy.call\n end\n\n assert_equal 2, count\n end\nend\n" }, { "path": "spec/mosquito/serializers/primitive_serializers_spec.cr", "content": "require \"uuid\"\nrequire \"../../spec_helper\"\n\nclass PrimitiveSerializerTester\n extend Mosquito::Serializers::Primitives\nend\n\ndescribe Mosquito::Serializers::Primitives do\n it \"serializes uuids\" do\n uuid = UUID.random\n assert_equal uuid, UUID.new(PrimitiveSerializerTester.serialize_uuid(uuid))\n end\n\n it \"deserializes uuids\" do\n uuid = UUID.random.to_s\n assert_equal uuid, PrimitiveSerializerTester.deserialize_uuid(uuid).to_s\n end\nend\n" }, { "path": "spec/mosquito/testing_backend_spec.cr", "content": "require \"../spec_helper\"\n\ndescribe Mosquito::TestBackend do\n def latest_enqueued_job\n Mosquito::TestBackend.enqueued_jobs.last\n end\n\n it \"holds a copy of jobs which have been enqueued\" do\n Mosquito.temp_config(backend: Mosquito::TestBackend.new) do\n QueuedTestJob.new.enqueue\n assert_equal QueuedTestJob, latest_enqueued_job.klass\n end\n end\n\n it \"embeds job parameters\" do\n Mosquito.temp_config(backend: Mosquito::TestBackend.new) do\n EchoJob.new(text: \"hello world\").enqueue\n assert_equal \"hello world\", latest_enqueued_job.config[\"text\"]\n end\n end\n\n it \"hold the job id\" do\n Mosquito.temp_config(backend: Mosquito::TestBackend.new) do\n job_run = QueuedTestJob.new.enqueue\n assert_equal job_run.id, latest_enqueued_job.id\n end\n end\n\n it \"has a list of job runs which can be emptied\" do\n Mosquito.temp_config(backend: Mosquito::TestBackend.new) do\n Mosquito::TestBackend.flush_enqueued_jobs!\n job_run = EchoJob.new(text: \"hello world\").enqueue\n assert_equal job_run.id, latest_enqueued_job.id\n Mosquito::TestBackend.flush_enqueued_jobs!\n assert Mosquito::TestBackend.enqueued_jobs.empty?\n end\n end\nend\n" }, { "path": "spec/mosquito/unique_job_spec.cr", "content": "require \"../spec_helper\"\n\ndescribe Mosquito::UniqueJob do\n describe \"first enqueue\" do\n it \"enqueues a job when no duplicate exists\" do\n clean_slate do\n job = UniqueTestJob.new(user_id: 1_i64, email_type: \"welcome\")\n job_run = job.enqueue\n enqueued = UniqueTestJob.queue.backend.list_waiting\n assert_equal [job_run.id], enqueued\n end\n end\n end\n\n describe \"duplicate suppression\" do\n it \"prevents a second enqueue with the same parameters\" do\n clean_slate do\n job1 = UniqueTestJob.new(user_id: 1_i64, email_type: \"welcome\")\n job_run1 = job1.enqueue\n enqueued = UniqueTestJob.queue.backend.list_waiting\n assert_equal 1, enqueued.size\n\n job2 = UniqueTestJob.new(user_id: 1_i64, email_type: \"welcome\")\n job_run2 = job2.enqueue\n enqueued = UniqueTestJob.queue.backend.list_waiting\n assert_equal 1, enqueued.size\n end\n end\n\n it \"allows enqueue with different parameters\" do\n clean_slate do\n job1 = UniqueTestJob.new(user_id: 1_i64, email_type: \"welcome\")\n job1.enqueue\n enqueued = UniqueTestJob.queue.backend.list_waiting\n assert_equal 1, enqueued.size\n\n job2 = UniqueTestJob.new(user_id: 2_i64, email_type: \"welcome\")\n job2.enqueue\n enqueued = UniqueTestJob.queue.backend.list_waiting\n assert_equal 2, enqueued.size\n end\n end\n\n it \"allows enqueue with different parameter values\" do\n clean_slate do\n job1 = UniqueTestJob.new(user_id: 1_i64, email_type: \"welcome\")\n job1.enqueue\n enqueued = UniqueTestJob.queue.backend.list_waiting\n assert_equal 1, enqueued.size\n\n job2 = UniqueTestJob.new(user_id: 1_i64, email_type: \"reminder\")\n job2.enqueue\n enqueued = UniqueTestJob.queue.backend.list_waiting\n assert_equal 2, enqueued.size\n end\n end\n end\n\n describe \"key filtering\" do\n it \"considers only specified key fields for uniqueness\" do\n clean_slate do\n # Same user_id, different message — should be suppressed because\n # key is only [:user_id]\n job1 = UniqueWithKeyJob.new(user_id: 1_i64, message: \"hello\")\n job1.enqueue\n enqueued = UniqueWithKeyJob.queue.backend.list_waiting\n assert_equal 1, enqueued.size\n\n job2 = UniqueWithKeyJob.new(user_id: 1_i64, message: \"world\")\n job2.enqueue\n enqueued = UniqueWithKeyJob.queue.backend.list_waiting\n assert_equal 1, enqueued.size\n end\n end\n\n it \"allows enqueue when key fields differ\" do\n clean_slate do\n job1 = UniqueWithKeyJob.new(user_id: 1_i64, message: \"hello\")\n job1.enqueue\n enqueued = UniqueWithKeyJob.queue.backend.list_waiting\n assert_equal 1, enqueued.size\n\n job2 = UniqueWithKeyJob.new(user_id: 2_i64, message: \"hello\")\n job2.enqueue\n enqueued = UniqueWithKeyJob.queue.backend.list_waiting\n assert_equal 2, enqueued.size\n end\n end\n end\n\n describe \"expiration\" do\n it \"allows re-enqueue after the uniqueness window expires\" do\n clean_slate do\n job1 = UniqueTestJob.new(user_id: 1_i64, email_type: \"welcome\")\n job_run1 = job1.enqueue\n enqueued = UniqueTestJob.queue.backend.list_waiting\n assert_equal 1, enqueued.size\n\n # Manually remove the lock to simulate expiration\n lock_key = job1.uniqueness_key(job_run1)\n Mosquito.backend.unlock(lock_key, job_run1.id)\n\n job2 = UniqueTestJob.new(user_id: 1_i64, email_type: \"welcome\")\n job2.enqueue\n enqueued = UniqueTestJob.queue.backend.list_waiting\n assert_equal 2, enqueued.size\n end\n end\n end\n\n describe \"no parameters\" do\n it \"works with jobs that have no parameters\" do\n clean_slate do\n job1 = UniqueNoParamsJob.new\n job1.enqueue\n enqueued = UniqueNoParamsJob.queue.backend.list_waiting\n assert_equal 1, enqueued.size\n\n job2 = UniqueNoParamsJob.new\n job2.enqueue\n enqueued = UniqueNoParamsJob.queue.backend.list_waiting\n assert_equal 1, enqueued.size\n end\n end\n end\n\n describe \"delayed enqueue\" do\n it \"prevents duplicate delayed enqueue\" do\n clean_slate do\n job1 = UniqueTestJob.new(user_id: 1_i64, email_type: \"welcome\")\n job1.enqueue(in: 5.minutes)\n scheduled = UniqueTestJob.queue.backend.list_scheduled\n assert_equal 1, scheduled.size\n\n job2 = UniqueTestJob.new(user_id: 1_i64, email_type: \"welcome\")\n job2.enqueue(in: 10.minutes)\n scheduled = UniqueTestJob.queue.backend.list_scheduled\n assert_equal 1, scheduled.size\n end\n end\n\n it \"prevents duplicate when mixing immediate and delayed enqueue\" do\n clean_slate do\n job1 = UniqueTestJob.new(user_id: 1_i64, email_type: \"welcome\")\n job1.enqueue\n waiting = UniqueTestJob.queue.backend.list_waiting\n assert_equal 1, waiting.size\n\n job2 = UniqueTestJob.new(user_id: 1_i64, email_type: \"welcome\")\n job2.enqueue(in: 5.minutes)\n scheduled = UniqueTestJob.queue.backend.list_scheduled\n assert_equal 0, scheduled.size\n end\n end\n end\n\n describe \"unique_duration\" do\n it \"returns the configured duration\" do\n job = UniqueTestJob.new(user_id: 1_i64, email_type: \"welcome\")\n assert_equal 1.hour, job.unique_duration\n end\n end\nend\n" }, { "path": "spec/mosquito/version_spec.cr", "content": "require \"../spec_helper\"\nrequire \"yaml\"\n\ndescribe \"mosquito version numbers\" do\n it \"is defined\" do\n assert Mosquito::VERSION\n end\n\n it \"matches the shard.yml file\" do\n File.open(\"shard.yml\") do |file|\n assert_equal Mosquito::VERSION, YAML.parse(file)[\"version\"].as_s\n end\n end\nend\n" }, { "path": "spec/spec_helper.cr", "content": "require \"minitest\"\nrequire \"minitest/focus\"\n\nrequire \"log\"\nLog.setup :fatal\n\nrequire \"timecop\"\nTimecop.safe_mode = true\n\nrequire \"../src/mosquito\"\nMosquito.configure do |settings|\n settings.backend_connection_string = testing_redis_url\n settings.publish_metrics = true\nend\n\nrequire \"./helpers/*\"\nclass Minitest::Test\n include PubSub::Helpers\nend\n\nMosquito.configuration.backend.flush\n\nrequire \"minitest/autorun\"\n" }, { "path": "src/mosquito/api/concurrency_config.cr", "content": "module Mosquito\n # Provides read/write access to the remotely stored concurrency limits\n # used by `RemoteConfigDequeueAdapter`.\n #\n # Supports both global limits (shared by all overseers) and per-overseer\n # limits for asymmetric hardware configurations.\n #\n # ```crystal\n # config = Mosquito::Api::ConcurrencyConfig.instance\n # config.limits # => global limits\n # config.limits(overseer_id: \"gpu-worker-1\") # => per-overseer limits\n # config.update({\"queue_a\" => 5}) # write global\n # config.update({\"queue_a\" => 1}, overseer_id: \"gpu-worker-1\") # write per-overseer\n # config.clear # remove global limits\n # config.clear(overseer_id: \"gpu-worker-1\") # remove per-overseer limits\n # ```\n class Api::ConcurrencyConfig\n def self.instance : self\n new\n end\n\n # Returns the global concurrency limits stored in the backend.\n def limits : Hash(String, Int32)\n RemoteConfigDequeueAdapter.stored_limits\n end\n\n # Returns the concurrency limits stored for a specific overseer.\n def limits(overseer_id : String) : Hash(String, Int32)\n RemoteConfigDequeueAdapter.stored_limits(overseer_id)\n end\n\n # Overwrites the global stored concurrency limits with *new_limits*.\n def update(new_limits : Hash(String, Int32)) : Nil\n RemoteConfigDequeueAdapter.store_limits(new_limits)\n end\n\n # Overwrites the stored concurrency limits for a specific overseer.\n def update(new_limits : Hash(String, Int32), overseer_id : String) : Nil\n RemoteConfigDequeueAdapter.store_limits(new_limits, overseer_id)\n end\n\n # Removes all globally stored concurrency limits.\n def clear : Nil\n RemoteConfigDequeueAdapter.clear_limits\n end\n\n # Removes stored concurrency limits for a specific overseer.\n def clear(overseer_id : String) : Nil\n RemoteConfigDequeueAdapter.clear_limits(overseer_id)\n end\n end\nend\n" }, { "path": "src/mosquito/api/executor.cr", "content": "module Mosquito\n module Api\n # An interface for an executor.\n #\n # This is used to inspect the state of an executor. For more information about executors, see `Mosquito::Runners::Executor`.\n class Executor\n getter :instance_id\n private getter :metadata\n\n # Creates an executor inspector.\n # The metadata is readonly and can be used to inspect the state of the executor.\n #\n # see #current_job, #current_job_queue\n def initialize(@instance_id : String)\n @metadata = Metadata.new Observability::Executor.metadata_key(@instance_id), readonly: true\n end\n\n # The current job being executed by the executor.\n #\n # When the executor is idle, this will be `nil`.\n def current_job : String?\n metadata[\"current_job\"]?\n end\n\n # The queue which housed the current job being executed.\n #\n # When the executor is idle, this will be `nil`.\n def current_job_queue : String?\n metadata[\"current_job_queue\"]?\n end\n\n # The last heartbeat time, or nil if none exists.\n def heartbeat : Time?\n metadata.heartbeat?\n end\n end\n end\n\n module Observability\n class Executor\n include Publisher\n\n private getter log : ::Log\n def self.metadata_key(instance_id : String) : String\n Mosquito.backend.build_key \"executor\", instance_id\n end\n\n def initialize(executor : Mosquito::Runners::Executor)\n @metadata = Metadata.new self.class.metadata_key executor.object_id.to_s\n @log = Log.for(executor.runnable_name)\n overseer_publish_context = executor.overseer.observer.publish_context\n @publish_context = PublishContext.new(\n overseer_publish_context,\n [:executor, executor.object_id]\n )\n end\n\n def execute(job_run : JobRun, from_queue : Mosquito::Queue)\n metrics do\n @metadata.set({\n \"current_job\" => job_run.id,\n \"current_job_queue\" => from_queue.name\n })\n end\n\n # Calculate what the duration _might_ be\n expected_duration = Mosquito.backend.average average_key(job_run.type)\n\n log.info { \"#{\"Starting:\".colorize.magenta} #{job_run} from #{from_queue.name}\" }\n publish({\n event: \"job-started\",\n job_run: job_run.id,\n from_queue: from_queue.name,\n expected_duration_ms: expected_duration\n })\n\n duration = Time.measure do\n yield\n end\n\n if job_run.succeeded?\n log_success_message job_run, duration\n elsif job_run.preempted?\n log_preempted_message job_run, duration\n else\n log_failure_message job_run, duration\n end\n\n publish({event: \"job-finished\", job_run: job_run.id})\n\n metrics do\n key = average_key(job_run.type)\n Mosquito.backend.average_push key, duration.total_milliseconds.to_i\n Mosquito.backend.delete key, in: 30.days\n\n @metadata.set(\n current_job: nil,\n current_job_queue: nil\n )\n end\n end\n\n def average_key(job_run_type : String) : String\n Mosquito.backend.build_key \"job\", job_run_type, \"duration\"\n end\n\n def log_success_message(job_run : JobRun, duration : Time::Span)\n log.info { \"#{\"Success:\".colorize.green} #{job_run} finished and took #{time_with_units duration}\" }\n end\n\n def log_preempted_message(job_run : JobRun, duration : Time::Span)\n message = String::Builder.new\n message << \"Preempted: \".colorize.cyan\n message << job_run\n message << \" was preempted\"\n\n reason = job_run.preempt_reason\n unless reason.empty?\n message << \" (\"\n message << reason\n message << \")\"\n end\n\n message << \" after \"\n message << time_with_units duration\n\n if job_run.rescheduleable?\n next_execution = Time.utc + job_run.reschedule_interval\n message << \" and will run again\".colorize.cyan\n message << \" in \"\n message << job_run.reschedule_interval\n message << \" (at \"\n message << next_execution\n message << \")\"\n end\n\n log.info { message.to_s }\n end\n\n def log_failure_message(job_run : JobRun, duration : Time::Span)\n message = String::Builder.new\n message << \"Failure: \".colorize.red\n message << job_run\n message << \" failed, taking \"\n message << time_with_units duration\n message << \" and \"\n\n if job_run.rescheduleable?\n next_execution = Time.utc + job_run.reschedule_interval\n message << \"will run again\".colorize.cyan\n message << \" in \"\n message << job_run.reschedule_interval\n message << \" (at \"\n message << next_execution\n message << \")\"\n log.warn { message.to_s }\n else\n message << \"cannot be rescheduled\".colorize.yellow\n log.error { message.to_s }\n end\n end\n\n # :nodoc:\n private def time_with_units(duration : Time::Span)\n seconds = duration.total_seconds\n if seconds > 0.1\n \"#{(seconds).*(100).trunc./(100)}s\".colorize.red\n elsif seconds > 0.001\n \"#{(seconds * 1_000).trunc}ms\".colorize.yellow\n elsif seconds > 0.000_001\n \"#{(seconds * 1_000_000).trunc}µs\".colorize.green\n elsif seconds > 0.000_000_001\n \"#{(seconds * 1_000_000_000).trunc}ns\".colorize.green\n else\n \"no discernible time at all\".colorize.green\n end\n end\n\n def heartbeat!\n metrics do\n @metadata.heartbeat!\n end\n end\n end\n end\nend\n" }, { "path": "src/mosquito/api/executor_config.cr", "content": "module Mosquito\n # Provides read/write access to the remotely stored executor count\n # used by overseers configured with a stable `overseer_id`.\n #\n # Supports both global counts (shared by all overseers) and per-overseer\n # counts for asymmetric hardware configurations.\n #\n # ```crystal\n # config = Mosquito::Api::ExecutorConfig.instance\n # config.executor_count # => global count or nil\n # config.executor_count(overseer_id: \"gpu-worker-1\") # => per-overseer count or nil\n # config.update(8) # write global\n # config.update(2, overseer_id: \"gpu-worker-1\") # write per-overseer\n # config.clear # remove global override\n # config.clear(overseer_id: \"gpu-worker-1\") # remove per-overseer override\n # ```\n class Api::ExecutorConfig\n CONFIG_KEY = \"executor_count\"\n\n def self.instance : self\n new\n end\n\n # Returns the global executor count stored in the backend, or nil if\n # no override has been set.\n def executor_count : Int32?\n self.class.stored_executor_count\n end\n\n # Returns the executor count for a specific overseer, or nil if no\n # override has been set for that overseer.\n def executor_count(overseer_id : String) : Int32?\n self.class.stored_executor_count(overseer_id)\n end\n\n # Writes a global executor count override.\n def update(count : Int32) : Nil\n self.class.store_executor_count(count)\n end\n\n # Writes an executor count override for a specific overseer.\n def update(count : Int32, overseer_id : String) : Nil\n self.class.store_executor_count(count, overseer_id)\n end\n\n # Removes the global executor count override.\n def clear : Nil\n self.class.clear_executor_count\n end\n\n # Removes the executor count override for a specific overseer.\n def clear(overseer_id : String) : Nil\n self.class.clear_executor_count(overseer_id)\n end\n\n # ----- Backend storage helpers -----\n\n def self.stored_executor_count : Int32?\n value = Mosquito.backend.get(global_config_key, \"count\")\n value.try(&.to_i32)\n end\n\n def self.stored_executor_count(overseer_id : String) : Int32?\n value = Mosquito.backend.get(overseer_config_key(overseer_id), \"count\")\n value.try(&.to_i32)\n end\n\n def self.store_executor_count(count : Int32) : Nil\n Mosquito.backend.set(global_config_key, \"count\", count.to_s)\n end\n\n def self.store_executor_count(count : Int32, overseer_id : String) : Nil\n Mosquito.backend.set(overseer_config_key(overseer_id), \"count\", count.to_s)\n end\n\n def self.clear_executor_count : Nil\n Mosquito.backend.delete(global_config_key)\n end\n\n def self.clear_executor_count(overseer_id : String) : Nil\n Mosquito.backend.delete(overseer_config_key(overseer_id))\n end\n\n # Resolves the effective executor count for an overseer by checking\n # per-overseer first, then global. Returns nil if neither is set.\n def self.resolve(overseer_id : String? = nil) : Int32?\n if oid = overseer_id\n stored_executor_count(oid) || stored_executor_count\n else\n stored_executor_count\n end\n end\n\n protected def self.global_config_key : String\n Mosquito.backend.build_key(CONFIG_KEY)\n end\n\n protected def self.overseer_config_key(overseer_id : String) : String\n Mosquito.backend.build_key(CONFIG_KEY, overseer_id)\n end\n end\nend\n" }, { "path": "src/mosquito/api/job_run.cr", "content": "module Mosquito::Api\n # Represents a job run in Mosquito.\n #\n # This class is used to inspect a job run stored in the backend.\n #\n # For more information about a JobRun, see `Mosquito::JobRun`.\n class JobRun\n # The id of the job run.\n getter id : String\n\n def initialize(@id : String)\n end\n\n # Does a JobRun with this ID exist in the backend?\n def found? : Bool\n config.has_key? \"type\"\n end\n\n # Get the parameters the job was enqueued with.\n def runtime_parameters : Hash(String, String)\n config.reject do |key, _|\n [\"id\", \"type\", \"enqueue_time\", \"retry_count\", \"started_at\", \"finished_at\"].includes? key\n end\n end\n\n private getter metadata : Metadata {\n Metadata.new(\n Mosquito.backend.build_key(Mosquito::JobRun::CONFIG_KEY_PREFIX, id),\n readonly: true\n )\n }\n\n private def config : Hash(String, String)\n metadata.to_h\n end\n\n # The type of job this job run is for.\n def type : String\n config[\"type\"]\n end\n\n # The moment this job was enqueued.\n def enqueue_time : Time\n Time.unix_ms config[\"enqueue_time\"].to_i64\n end\n\n # The moment this job was started.\n def started_at : Time?\n if time = config[\"started_at\"]?\n Time.unix_ms time.to_i64\n end\n end\n\n # The moment this job was finished.\n def finished_at : Time?\n if time = config[\"finished_at\"]?\n Time.unix_ms time.to_i64\n end\n end\n\n # The number of times this job has been retried.\n def retry_count : Int\n config[\"retry_count\"].to_i\n end\n end\nend\n" }, { "path": "src/mosquito/api/observability/publisher.cr", "content": "module Mosquito::Observability::Publisher\n Log = ::Log.for(\"mosquito.events\")\n\n getter publish_context : PublishContext\n\n macro metrics(&block)\n if Mosquito.configuration.metrics?\n {{ block.body }}\n end\n end\n\n @[AlwaysInline]\n def publish(data : NamedTuple)\n metrics do\n Log.debug { \"Publishing #{data} to #{@publish_context.originator}\" }\n Mosquito.backend.publish(\n publish_context.originator,\n data.to_json\n )\n end\n end\n\n class PublishContext\n alias Context = Array(String | Symbol | UInt64)\n property originator : String\n property context : String\n\n def initialize(context : Context)\n @context = KeyBuilder.build context\n @originator = KeyBuilder.build \"mosquito\", @context\n end\n\n def initialize(parent : self, context : Context)\n @context = KeyBuilder.build context\n @originator = KeyBuilder.build \"mosquito\", parent.context, context\n end\n end\nend\n" }, { "path": "src/mosquito/api/overseer.cr", "content": "module Mosquito\n # An interface for inspecting the state of Mosquito Overseers.\n #\n # For more information about overseers, see `Mosquito::Runners::Overseer`.\n class Api::Overseer\n # The instance ID of the overseer being inspected.\n getter :instance_id\n private getter :metadata\n\n # Creates a new Api::Overseer by its instance ID.\n def initialize(@instance_id : String)\n @metadata = Metadata.new Observability::Overseer.metadata_key(@instance_id), readonly: true\n end\n\n # Retrieves a list of all overseers in the backend.\n def self.all : Array(self)\n Mosquito.backend.list_overseers.map do |id|\n new id\n end\n end\n\n # Retrieves a list of executors managed by this overseer.\n def executors : Array(Executor)\n if executor_list = @metadata[\"executors\"]?\n executor_list.split(\",\").map do |name|\n Executor.new name\n end\n else\n [] of Executor\n end\n end\n\n # The time the overseer last sent a heartbeat.\n def last_heartbeat : Time?\n metadata.heartbeat?\n end\n end\n\n\n class Observability::Overseer\n include Publisher\n\n getter metadata : Metadata\n getter instance_id : String\n private getter overseer : Runners::Overseer\n private getter log : ::Log\n\n def self.metadata_key(instance_id : String) : String\n Mosquito.backend.build_key \"overseer\", instance_id\n end\n\n def initialize(@overseer : Runners::Overseer)\n @instance_id = overseer.object_id.to_s\n @log = Log.for(overseer.runnable_name)\n @metadata = Metadata.new self.class.metadata_key(instance_id)\n @publish_context = PublishContext.new [:overseer, overseer.object_id]\n end\n\n def starting\n log.info { \"Starting #{overseer.executor_count} executors.\" }\n\n publish({event: \"started\"})\n heartbeat\n end\n\n def shutting_down\n log.info { \"Shutting down.\" }\n end\n\n def stopping\n log.info { \"Stopping executors.\" }\n publish({event: \"stopped\"})\n end\n\n def stopped\n log.info { \"All executors stopped.\" }\n log.info { \"Finished for now.\" }\n publish({event: \"exited\"})\n\n Mosquito.backend.deregister_overseer self.instance_id\n metadata.delete\n end\n\n def heartbeat\n # Registration must always happen so that the pending job cleanup\n # mechanism can determine which overseers are still alive.\n Mosquito.backend.register_overseer self.instance_id\n\n metrics do\n metadata.heartbeat!\n end\n end\n\n def executor_created(executor : Runners::Executor) : Nil\n publish({event: \"executor-created\", executor: executor.object_id})\n end\n\n def executor_died(executor : Runners::Executor) : Nil\n publish({event: \"executor-died\", executor: executor.object_id})\n\n log.fatal do\n <<-MSG\n Executor #{executor.runnable_name} died.\n A new executor will be started.\n MSG\n end\n end\n\n def channels_closed\n log.fatal { \"Executor communication channels closed, overseer will stop.\" }\n end\n\n def waiting_for_queue_list\n log.debug { \"Waited for the queue list to fetch possible queues.\" }\n end\n\n def queue_list_died\n log.fatal { \"QueueList has died, overseer will stop.\" }\n end\n\n def recovered_orphaned_job(job_run : JobRun, overseer_id : String)\n log.warn { \"Recovered orphaned job #{job_run.id} from dead overseer #{overseer_id}.\" }\n end\n\n def orphaned_jobs_recovered(total : Int32)\n log.warn { \"Recovered #{total} orphaned job(s) from pending queues.\" }\n end\n\n def recovered_job_from_executor(job_run : JobRun, executor : Runners::Executor)\n log.warn { \"Recovered job #{job_run.id} from dead executor #{executor.runnable_name}.\" }\n end\n\n def update_executor_list(executors : Array(Runners::Executor)) : Nil\n metrics do\n metadata[\"executors\"] = executors.map(&.object_id).join(\",\")\n end\n end\n end\nend\n" }, { "path": "src/mosquito/api/periodic_job.cr", "content": "module Mosquito\n # An interface for inspecting the state of periodic jobs.\n #\n # This class provides read-only access to periodic job metadata,\n # including the last time each periodic job was executed.\n #\n # ```\n # Mosquito::Api::PeriodicJob.all.each do |job|\n # puts \"#{job.name} last ran at #{job.last_executed_at}\"\n # end\n # ```\n class Api::PeriodicJob\n # The name of the periodic job class.\n getter name : String\n\n # The configured run interval for this periodic job.\n getter interval : Time::Span | Time::MonthSpan\n\n private getter metadata : Metadata\n\n # Returns a list of all registered periodic jobs.\n def self.all : Array(self)\n Base.scheduled_job_runs.map do |job_run|\n new job_run.class.name, job_run.interval\n end\n end\n\n def initialize(@name : String, @interval : Time::Span | Time::MonthSpan)\n @metadata = Metadata.new(\n Mosquito.backend.build_key(\"periodic_jobs\", @name),\n readonly: true\n )\n end\n\n # The last time this periodic job was executed, or nil if it has never run.\n def last_executed_at : Time?\n if timestamp = metadata[\"last_executed_at\"]?\n Time.unix(timestamp.to_i)\n end\n end\n end\n\n class Observability::PeriodicJob\n include Publisher\n\n getter log : ::Log\n getter publish_context : PublishContext\n\n def initialize(periodic_job_run : Mosquito::PeriodicJobRun)\n @name = periodic_job_run.class.name\n @publish_context = PublishContext.new [:periodic_job, @name]\n @log = Log.for(@name)\n end\n\n def enqueued(at time : Time)\n log.info { \"Enqueued periodic job at #{time}\" }\n publish({event: \"enqueued\", executed_at: time.to_unix})\n end\n\n def skipped\n log.trace { \"Not yet due for execution\" }\n end\n end\nend\n" }, { "path": "src/mosquito/api/queue.cr", "content": "module Mosquito\n # Represents a named queue in the system, and allows querying the state of the queue. For more about the internals of a Queue in Mosquito, see `Mosquito::Queue`.\n class Api::Queue\n # The name of the queue.\n getter name : String\n\n private property backend : Mosquito::Backend::Queue\n\n # Returns a list of all known named queues in the system.\n def self.all : Array(Queue)\n Mosquito.backend.list_queues.map { |name| new name }\n end\n\n # Creates an instance of a named queue.\n def initialize(@name : String)\n @backend = Mosquito.backend.queue name\n end\n\n {% for name in Mosquito::Backend::QUEUES %}\n # Gets a list of all the job runs in the internal {{name.id}} queue.\n def {{name.id}}_job_runs : Array(JobRun)\n backend.list_{{name.id}}\n .map { |task_id| JobRun.new task_id }\n end\n {% end %}\n\n # The operating size of the queue, not including dead jobs.\n def size : Int64\n backend.size(include_dead: false)\n end\n\n # The size of the queue, broken out by job state.\n #\n # Example:\n #\n # ```\n # Mosquito::Api::Queue.all.first.size_details\n # # => {\"waiting\" => 0, \"scheduled\" => 0, \"pending\" => 0, \"dead\" => 0}\n # ```\n #\n # The semantics of the keys are described in detail on the `Mosquito::Queue` class, but in brief:\n #\n # - `scheduled` is a list of jobs which are scheduled to be executed at a later time.\n # - `waiting` is a list of jobs which should be executed ASAP\n # - `pending` is a list of jobs for which execution has started\n # - `dead` is a list of jobs which have failed to execute\n def size_details : Hash(String, Int64)\n sizes = {} of String => Int64\n {% for name in Mosquito::Backend::QUEUES %}\n sizes[\"{{name.id}}\"] = backend.{{name.id}}_size\n {% end %}\n sizes\n end\n def paused? : Bool\n backend.paused?\n end\n\n def <=>(other)\n name <=> other.name\n end\n end\n\n class Observability::Queue\n include Publisher\n\n getter log : ::Log\n getter publish_context : PublishContext\n\n delegate name, to: @queue\n\n def initialize(queue : String)\n initialize(Mosquito::Queue.new queue)\n end\n\n def initialize(@queue : Mosquito::Queue)\n @publish_context = PublishContext.new [:queue, queue.name]\n @log = Log.for(queue.name)\n end\n\n def enqueued(job_run : JobRun)\n log.trace { \"Enqueuing #{job_run.id} for immediate execution\" }\n publish({event: \"enqueued\", job_run: job_run.id})\n end\n\n def enqueued(job_run : JobRun, at execute_time : Time)\n log.trace { \"Enqueuing #{job_run.id} for execution at #{execute_time}\" }\n publish({event: \"enqueued\", job_run: job_run.id, execute_time: execute_time})\n end\n\n def dequeued(job_run : JobRun)\n log.trace { \"Dequeuing #{job_run.id}\" }\n publish({event: \"dequeued\", job_run: job_run.id})\n end\n\n def rescheduled(job_run : JobRun, to execute_time : Time)\n log.trace { \"Rescheduling #{job_run.id} to execute at #{execute_time}\" }\n publish({event: \"rescheduled\", job_run: job_run.id, execute_time: execute_time})\n end\n\n def forgotten(job_run : JobRun)\n log.trace { \"Forgetting #{job_run.id}\" }\n publish({event: \"forgotten\", job_run: job_run.id})\n end\n\n def banished(job_run : JobRun)\n log.trace { \"Banishing #{job_run.id} to dead queue\" }\n publish({event: \"banished\", job_run: job_run.id})\n end\n\n def paused(duration : Time::Span? = nil)\n if duration\n log.info { \"Paused for #{duration}\" }\n publish({event: \"paused\", duration: duration.total_seconds})\n else\n log.info { \"Paused indefinitely\" }\n publish({event: \"paused\"})\n end\n end\n\n def resumed\n log.info { \"Resumed\" }\n publish({event: \"resumed\"})\n end\n end\nend\n" }, { "path": "src/mosquito/api/queue_list.cr", "content": "module Mosquito\n class Observability::QueueList\n private getter log : ::Log\n @last_paused_names = Set(String).new\n\n def initialize(queue_list : Runners::QueueList)\n @log = Log.for(queue_list.runnable_name)\n end\n\n def checked_for_paused_queues(paused : Array(Mosquito::Queue))\n paused_names = paused.map(&.name).to_set\n if paused_names != @last_paused_names\n @last_paused_names = paused_names\n log.for(\"paused_queues\").notice {\n if paused.size > 0\n \"#{paused.size} paused queues: #{paused.map(&.name).join(\", \")}\"\n else\n \"all queues resumed\"\n end\n }\n end\n end\n end\nend\n" }, { "path": "src/mosquito/api.cr", "content": "require \"./backend\"\nrequire \"./api/observability/*\"\nrequire \"./api/*\"\n\nmodule Mosquito::Api\n def self.overseer(id : String) : Overseer\n Overseer.new id\n end\n\n def self.executor(id : String) : Executor\n Executor.new id\n end\n\n def self.job_run(id : String) : JobRun\n JobRun.new id\n end\n\n def self.list_periodic_jobs : Array(PeriodicJob)\n PeriodicJob.all\n end\n\n def self.list_queues : Array(Observability::Queue)\n Mosquito.backend.list_queues\n .map { |name| Observability::Queue.new name }\n end\n\n def self.list_overseers : Array(Overseer)\n Mosquito.backend.list_overseers\n .map { |name| Overseer.new name }\n end\n\n def self.event_receiver : Channel(Backend::BroadcastMessage)\n Mosquito.backend.subscribe \"mosquito:*\"\n end\n\n # Returns a `ConcurrencyConfig` instance for reading and writing the\n # remotely stored concurrency limits used by\n # `RemoteConfigDequeueAdapter`.\n def self.concurrency_config : ConcurrencyConfig\n ConcurrencyConfig.instance\n end\n\n # Convenience reader for the current global remote concurrency limits.\n def self.concurrency_limits : Hash(String, Int32)\n concurrency_config.limits\n end\n\n # Convenience reader for a specific overseer's concurrency limits.\n def self.concurrency_limits(overseer_id : String) : Hash(String, Int32)\n concurrency_config.limits(overseer_id)\n end\n\n # Convenience writer — replaces the global stored concurrency limits so\n # that all `RemoteConfigDequeueAdapter` instances pick them up on their\n # next refresh cycle.\n def self.set_concurrency_limits(limits : Hash(String, Int32)) : Nil\n concurrency_config.update(limits)\n end\n\n # Convenience writer — replaces stored concurrency limits for a specific\n # overseer.\n def self.set_concurrency_limits(limits : Hash(String, Int32), overseer_id : String) : Nil\n concurrency_config.update(limits, overseer_id)\n end\n\n # Returns an `ExecutorConfig` instance for reading and writing the\n # remotely stored executor count.\n def self.executor_config : ExecutorConfig\n ExecutorConfig.instance\n end\n\n # Convenience reader for the global remote executor count.\n def self.executor_count : Int32?\n executor_config.executor_count\n end\n\n # Convenience reader for a specific overseer's executor count.\n def self.executor_count(overseer_id : String) : Int32?\n executor_config.executor_count(overseer_id)\n end\n\n # Convenience writer — sets the global executor count override.\n def self.set_executor_count(count : Int32) : Nil\n executor_config.update(count)\n end\n\n # Convenience writer — sets the executor count for a specific overseer.\n def self.set_executor_count(count : Int32, overseer_id : String) : Nil\n executor_config.update(count, overseer_id)\n end\nend\n" }, { "path": "src/mosquito/backend.cr", "content": "module Mosquito\n abstract class Backend\n struct BroadcastMessage\n property channel : String\n property message : String\n\n def initialize(@channel, @message)\n end\n end\n\n # The lifecycle states a job run passes through in any backend.\n QUEUES = %w(waiting scheduled pending dead)\n\n KEY_PREFIX = {\"mosquito\"}\n\n def build_key(*parts)\n KeyBuilder.build Mosquito.configuration.global_prefix, KEY_PREFIX, *parts\n end\n\n # Factory method to create a named queue for this backend.\n def queue(name : String | Symbol) : Queue\n _build_queue(name.to_s)\n end\n\n protected abstract def _build_queue(name : String) : Queue\n\n abstract def connection\n abstract def connection_string=(value : String)\n abstract def connection_string : String?\n abstract def valid_configuration? : Bool\n\n # Storage\n abstract def store(key : String, value : Hash(String, String?) | Hash(String, String)) : Nil\n abstract def retrieve(key : String) : Hash(String, String)\n\n abstract def delete(key : String, in ttl : Int64 = 0) : Nil\n abstract def delete(key : String, in ttl : Time::Span) : Nil\n abstract def expires_in(key : String) : Int64\n\n abstract def get(key : String, field : String) : String?\n abstract def set(key : String, field : String, value : String) : String\n abstract def set(key : String, values : Hash(String, String?) | Hash(String, Nil) | Hash(String, String)) : Nil\n abstract def delete_field(key : String, field : String) : Nil\n abstract def increment(key : String, field : String) : Int64\n abstract def increment(key : String, field : String, by value : Int32) : Int64\n\n # Global\n abstract def list_queues : Array(String)\n abstract def list_overseers : Array(String)\n abstract def list_active_overseers(since : Time) : Array(String)\n abstract def register_overseer(id : String) : Nil\n abstract def deregister_overseer(id : String) : Nil\n abstract def flush : Nil\n\n # Coordination\n abstract def unlock(key : String, value : String) : Nil\n abstract def lock?(key : String, value : String, ttl : Time::Span) : Bool\n abstract def renew_lock?(key : String, value : String, ttl : Time::Span) : Bool\n abstract def publish(key : String, value : String) : Nil\n abstract def subscribe(key : String) : Channel(BroadcastMessage)\n\n # Metrics\n abstract def average_push(key : String, value : Int32, window_size : Int32 = 100) : Nil\n abstract def average(key : String) : Int32\n\n abstract class Queue\n getter backend : Backend\n private getter name : String\n\n def initialize(@backend, @name : String)\n end\n\n # Queue operations\n abstract def enqueue(job_run : JobRun) : JobRun\n abstract def dequeue : JobRun?\n abstract def schedule(job_run : JobRun, at scheduled_time : Time) : JobRun\n abstract def deschedule : Array(JobRun)\n abstract def finish(job_run : JobRun)\n abstract def terminate(job_run : JobRun)\n abstract def undequeue : JobRun?\n abstract def flush : Nil\n abstract def size(include_dead : Bool = true) : Int64\n\n {% for name in [\"waiting\", \"scheduled\", \"pending\", \"dead\"] %}\n abstract def list_{{name.id}} : Array(String)\n abstract def {{name.id}}_size : Int64\n {% end %}\n\n abstract def scheduled_job_run_time(job_run : JobRun) : Time?\n\n # Pause this queue so that `#dequeue` returns nil until it is resumed\n # or the optional duration expires.\n abstract def pause(duration : Time::Span? = nil) : Nil\n\n # Resume a paused queue, allowing dequeue to proceed.\n abstract def resume : Nil\n abstract def paused? : Bool\n\n end\n end\nend\n" }, { "path": "src/mosquito/base.cr", "content": "require \"json\"\n\nmodule Mosquito\n alias Id = Int64 | Int32\n record WorkUnit, job_run : JobRun, queue : Queue do\n def self.of(job_run : JobRun, *, from queue : Queue) : self\n new(job_run, queue)\n end\n end\n\n class Base\n class_getter mapping = {} of String => Mosquito::Job.class\n class_getter scheduled_job_runs = [] of PeriodicJobRun\n class_getter timetable = [] of PeriodicJobRun\n\n def self.register_job_mapping(string, klass)\n @@mapping[string] = klass\n end\n\n def self.job_for_type(type : String) : Mosquito::Job.class\n @@mapping[type]\n rescue e : KeyError\n error = String.build do |s|\n s << <<-TEXT\n Could not find a job class for type \"#{type}\", perhaps you forgot to register it?\n\n Current known types are:\n\n TEXT\n\n @@mapping.each { |k, v| s << \"#{k}=>#{v}\\n\" }\n\n s << \"\\n\\n\"\n end\n\n raise KeyError.new(error)\n end\n\n def self.register_job_interval(klass, interval : Time::Span | Time::MonthSpan)\n @@scheduled_job_runs << PeriodicJobRun.new(klass, interval)\n end\n\n def self.register_job(klass, *, to_run_at scheduled_time : Time)\n position = @@timetable.index do\n end\n end\n end\nend\n" }, { "path": "src/mosquito/configuration.cr", "content": "module Mosquito\n class_getter configuration = Configuration.new\n\n def self.configure(&block) : Nil\n yield configuration\n end\n\n class Configuration\n property idle_wait : Time::Span = 100.milliseconds\n property successful_job_ttl : Int32 = 1.minute.total_seconds.to_i\n property failed_job_ttl : Int32 = 86400\n\n property use_distributed_lock : Bool = true\n\n property executor_count : Int32 = ENV.fetch(\"MOSQUITO_EXECUTOR_COUNT\", \"6\").to_i\n\n property run_from : Array(String) = [] of String\n property global_prefix : String? = nil\n property backend : Mosquito::Backend = Mosquito::RedisBackend.new\n\n property dequeue_adapter : Mosquito::DequeueAdapter = Mosquito::ShuffleDequeueAdapter.new\n\n # Maps queue names to resource gates. Queues whose gate returns\n # `false` from `#allow?` are excluded from dequeuing.\n property resource_gates : Hash(String, Mosquito::ResourceGate) = {} of String => Mosquito::ResourceGate\n\n # A stable, user-chosen identifier for this overseer instance.\n # Used to look up per-overseer remote configuration (executor count,\n # concurrency limits, etc.). When nil, the overseer only reads global\n # remote config.\n property overseer_id : String? = nil\n\n property publish_metrics : Bool = false\n\n # How often a mosquito runner should emit a heartbeat metric.\n property heartbeat_interval : Time::Span = 20.seconds\n\n # How long an overseer can go without a heartbeat before it is\n # considered dead and its pending jobs are recovered.\n property dead_overseer_threshold : Time::Span = 100.seconds\n\n property validated = false\n\n def backend_connection\n backend.connection\n end\n\n def backend_connection_string\n backend.connection_string\n end\n\n def backend_connection_string=(value : String)\n backend.connection_string = value\n end\n\n def idle_wait=(time_span : Float)\n @idle_wait = time_span.seconds\n end\n\n def validate\n return if @validated\n @validated = true\n\n unless backend.valid_configuration?\n message = <<-error\n Mosquito cannot start because no backend connection has been provided.\n\n For example, in your application config:\n\n Mosquito.configure do |settings|\n settings.backend_connection_string = (ENV[\"REDIS_TLS_URL\"]? || ENV[\"REDIS_URL\"]? || \"redis://localhost:6379\")\n end\n\n See Also: https://github.com/mosquito-cr/mosquito#connecting-to-redis\n error\n\n raise message\n end\n end\n\n def metrics? : Bool\n publish_metrics\n end\n end\nend\n" }, { "path": "src/mosquito/dequeue_adapter.cr", "content": "module Mosquito\n # A DequeueAdapter determines how the Overseer selects the next job to\n # execute from the available queues.\n #\n # Subclass `DequeueAdapter`, implement `#dequeue`, and assign an instance\n # via `Mosquito.configure`:\n #\n # ```crystal\n # class MyDequeueAdapter < Mosquito::DequeueAdapter\n # def dequeue(queue_list : Mosquito::Runners::QueueList) : Mosquito::WorkUnit?\n # queue_list.queues.each do |q|\n # if job_run = q.dequeue\n # return WorkUnit.of(job_run, from: q)\n # end\n # end\n # end\n # end\n #\n # Mosquito.configure do |settings|\n # settings.dequeue_adapter = MyDequeueAdapter.new\n # end\n # ```\n abstract class DequeueAdapter\n # Attempt to dequeue a job from one of the queues managed by `queue_list`.\n #\n # Returns a `WorkUnit` when a job is available, or `nil`\n # when all queues are empty.\n abstract def dequeue(queue_list : Runners::QueueList) : WorkUnit?\n\n # Called by the Overseer when a job run has finished executing.\n # Override this to react to completed jobs (e.g. update internal\n # counters or rebalance queue weights).\n def finished_with(job_run : JobRun, queue : Queue) : Nil\n end\n end\nend\n" }, { "path": "src/mosquito/dequeue_adapters/concurrency_limited_dequeue_adapter.cr", "content": "require \"../dequeue_adapter\"\n\nmodule Mosquito\n # A dequeue adapter that enforces per-queue concurrency limits.\n #\n # Each queue can be assigned a maximum number of jobs that may execute\n # concurrently. When a queue has reached its limit, it is skipped during\n # dequeue until an in-flight job finishes.\n #\n # Queues not present in the limits table have no concurrency ceiling and\n # are bounded only by the total executor pool size.\n #\n # Among eligible queues the adapter uses a shuffle to provide rough\n # fairness, similar to `ShuffleDequeueAdapter`.\n #\n # ## Example\n #\n # ```crystal\n # Mosquito.configure do |settings|\n # settings.executor_count = 8\n #\n # settings.dequeue_adapter = Mosquito::ConcurrencyLimitedDequeueAdapter.new({\n # \"queue_a\" => 3,\n # \"queue_b\" => 5,\n # })\n # end\n # ```\n #\n # In this configuration at most 3 jobs from \"queue_a\" and 5 from \"queue_b\"\n # will execute at the same time. Other queues are unlimited.\n class ConcurrencyLimitedDequeueAdapter < DequeueAdapter\n property limits : Hash(String, Int32)\n\n # Tracks the number of currently in-flight jobs per queue name.\n # Access is fiber-safe because Crystal fibers are cooperatively\n # scheduled and we never yield between read and write.\n @active : Hash(String, Int32)\n\n def initialize(@limits : Hash(String, Int32))\n @active = Hash(String, Int32).new(0)\n end\n\n def dequeue(queue_list : Runners::QueueList) : WorkUnit?\n queue_list.queues.shuffle.each do |q|\n if limit = limits[q.name]?\n next if @active[q.name] >= limit\n end\n\n if job_run = q.dequeue\n @active[q.name] = @active[q.name] + 1\n return WorkUnit.of(job_run, from: q)\n end\n end\n end\n\n # Called by the Overseer when a job from this queue has finished\n # executing. Decrements the in-flight counter so the queue becomes\n # eligible for dequeue again.\n def finished_with(job_run : JobRun, queue : Queue) : Nil\n count = @active[queue.name]\n @active[queue.name] = {count - 1, 0}.max\n end\n\n # Returns the current number of in-flight jobs for the given queue.\n def active_count(queue_name : String) : Int32\n @active[queue_name]\n end\n end\nend\n" }, { "path": "src/mosquito/dequeue_adapters/remote_config_dequeue_adapter.cr", "content": "require \"./concurrency_limited_dequeue_adapter\"\n\nmodule Mosquito\n # A dequeue adapter that wraps `ConcurrencyLimitedDequeueAdapter` with\n # remotely configurable concurrency limits stored in the Mosquito backend\n # (e.g. Redis).\n #\n # Limits are refreshed by polling the backend at a configurable interval.\n # When the remote key is absent or empty the adapter falls back to the\n # `defaults` hash provided at construction time.\n #\n # Remote values are **merged on top of** defaults: a queue present only in\n # defaults keeps its value, a queue present only in the remote config is\n # added, and a queue present in both uses the remote value.\n #\n # ## Per-overseer configuration\n #\n # When `overseer_id` is set, the adapter reads from both the global key\n # and a per-overseer key. The merge order is:\n #\n # defaults → global remote → per-overseer remote\n #\n # This lets you run overseers on asymmetric hardware and tune each one\n # independently while still sharing a common baseline.\n #\n # ## Setting limits remotely\n #\n # Use `Mosquito::Api.set_concurrency_limits` to write global limits:\n #\n # ```crystal\n # Mosquito::Api.set_concurrency_limits({\"queue_a\" => 2, \"queue_b\" => 10})\n # ```\n #\n # Or target a specific overseer:\n #\n # ```crystal\n # Mosquito::Api.set_concurrency_limits({\"queue_a\" => 1}, overseer_id: \"gpu-worker-1\")\n # ```\n #\n # ## Example\n #\n # ```crystal\n # Mosquito.configure do |settings|\n # settings.dequeue_adapter = Mosquito::RemoteConfigDequeueAdapter.new(\n # defaults: {\"queue_a\" => 3, \"queue_b\" => 5},\n # overseer_id: \"gpu-worker-1\",\n # refresh_interval: 5.seconds,\n # )\n # end\n # ```\n #\n # In this configuration the adapter starts with the given defaults. Any\n # limits written to the backend via the API will take effect within\n # `refresh_interval` seconds. Per-overseer limits override global limits\n # which override defaults.\n class RemoteConfigDequeueAdapter < DequeueAdapter\n CONFIG_KEY = \"concurrency_limits\"\n\n getter defaults : Hash(String, Int32)\n getter refresh_interval : Time::Span\n getter inner : ConcurrencyLimitedDequeueAdapter\n getter overseer_id : String?\n\n @last_refresh_at : Time = Time::UNIX_EPOCH\n @last_remote_limits : Hash(String, Int32) = {} of String => Int32\n\n def initialize(\n @defaults : Hash(String, Int32) = {} of String => Int32,\n @overseer_id : String? = nil,\n @refresh_interval : Time::Span = 5.seconds\n )\n @inner = ConcurrencyLimitedDequeueAdapter.new(defaults.dup)\n end\n\n def dequeue(queue_list : Runners::QueueList) : WorkUnit?\n maybe_refresh_limits\n inner.dequeue(queue_list)\n end\n\n def finished_with(job_run : JobRun, queue : Queue) : Nil\n inner.finished_with(job_run, queue)\n end\n\n # Returns the current effective concurrency limits (defaults merged\n # with any remote overrides).\n def limits : Hash(String, Int32)\n inner.limits\n end\n\n # Returns the current in-flight count for *queue_name*, delegated to\n # the inner adapter.\n def active_count(queue_name : String) : Int32\n inner.active_count(queue_name)\n end\n\n # Force an immediate refresh from the backend, ignoring the\n # `refresh_interval` timer.\n def refresh_limits : Nil\n remote = load_remote_limits\n merged = defaults.merge(remote)\n\n if merged != inner.limits\n inner.limits = merged\n end\n\n @last_refresh_at = Time.utc\n end\n\n # ----- Backend storage helpers (class-level) -----\n\n # Reads the global concurrency limits hash stored in the backend.\n def self.stored_limits : Hash(String, Int32)\n raw = Mosquito.backend.retrieve(global_config_key)\n raw.transform_values(&.to_i32)\n end\n\n # Reads the concurrency limits for a specific overseer.\n def self.stored_limits(overseer_id : String) : Hash(String, Int32)\n raw = Mosquito.backend.retrieve(overseer_config_key(overseer_id))\n raw.transform_values(&.to_i32)\n end\n\n # Overwrites the global concurrency limits with *limits*. Any previously\n # stored queue entries not present in *limits* are removed.\n def self.store_limits(limits : Hash(String, Int32)) : Nil\n key = global_config_key\n Mosquito.backend.delete(key)\n Mosquito.backend.store(key, limits.transform_values(&.to_s)) unless limits.empty?\n end\n\n # Overwrites the concurrency limits for a specific overseer with *limits*.\n def self.store_limits(limits : Hash(String, Int32), overseer_id : String) : Nil\n key = overseer_config_key(overseer_id)\n Mosquito.backend.delete(key)\n Mosquito.backend.store(key, limits.transform_values(&.to_s)) unless limits.empty?\n end\n\n # Removes all globally stored concurrency limits, causing adapters to\n # fall back to their defaults (or per-overseer limits if set).\n def self.clear_limits : Nil\n Mosquito.backend.delete(global_config_key)\n end\n\n # Removes stored concurrency limits for a specific overseer.\n def self.clear_limits(overseer_id : String) : Nil\n Mosquito.backend.delete(overseer_config_key(overseer_id))\n end\n\n protected def self.global_config_key : String\n Mosquito.backend.build_key(CONFIG_KEY)\n end\n\n protected def self.overseer_config_key(overseer_id : String) : String\n Mosquito.backend.build_key(CONFIG_KEY, overseer_id)\n end\n\n private def maybe_refresh_limits\n now = Time.utc\n if now - @last_refresh_at >= @refresh_interval\n refresh_limits\n end\n end\n\n private def load_remote_limits : Hash(String, Int32)\n global = self.class.stored_limits\n\n result = if oid = overseer_id\n per_overseer = self.class.stored_limits(oid)\n global.merge(per_overseer)\n else\n global\n end\n\n @last_remote_limits = result\n rescue\n # If the backend is unreachable or the data is corrupt, fall back\n # to the last known-good remote limits so previously applied overrides\n # are preserved rather than silently reverting to defaults.\n @last_remote_limits\n end\n end\nend\n" }, { "path": "src/mosquito/dequeue_adapters/shuffle_dequeue_adapter.cr", "content": "require \"../dequeue_adapter\"\n\nmodule Mosquito\n # The default dequeue adapter. Shuffles the queue list on each pass and\n # returns the first available job.\n #\n # The shuffle provides rough fairness across queues, preventing any single\n # queue from being consistently checked first.\n class ShuffleDequeueAdapter < DequeueAdapter\n def dequeue(queue_list : Runners::QueueList) : WorkUnit?\n queue_list.queues.shuffle.each do |q|\n if job_run = q.dequeue\n return WorkUnit.of(job_run, from: q)\n end\n end\n end\n end\nend\n" }, { "path": "src/mosquito/dequeue_adapters/weighted_dequeue_adapter.cr", "content": "require \"../dequeue_adapter\"\n\nmodule Mosquito\n # A dequeue adapter that checks queues according to configured weights.\n #\n # Higher-weight queues are given proportionally more chances to be dequeued\n # from. On each call to `#dequeue`, the adapter picks a queue at random\n # (weighted by its configured value). If that queue is empty, it is removed\n # from consideration and another weighted pick is made, ensuring each queue\n # is checked at most once per dequeue call.\n #\n # The weight map is built fresh on each dequeue call from the current\n # queue list, ensuring newly discovered queues are picked up immediately.\n #\n # Queues not present in the weights table are assigned a default weight of 1.\n #\n # ## Example\n #\n # ```crystal\n # Mosquito.configure do |settings|\n # settings.dequeue_adapter = Mosquito::WeightedDequeueAdapter.new({\n # \"critical\" => 5,\n # \"default\" => 2,\n # \"bulk\" => 1,\n # })\n # end\n # ```\n #\n # In this configuration the \"critical\" queue will be checked roughly 5x as\n # often as \"bulk\" and 2.5x as often as \"default\".\n class WeightedDequeueAdapter < DequeueAdapter\n getter weights : Hash(String, Int32)\n\n def initialize(@weights : Hash(String, Int32), @default_weight = 1)\n end\n\n def dequeue(queue_list : Runners::QueueList) : WorkUnit?\n remaining = queue_list.queues.map { |q|\n {q, weights.fetch(q.name, @default_weight)}\n }\n\n until remaining.empty?\n queue, index = weighted_random_select(remaining)\n if job_run = queue.dequeue\n return WorkUnit.of(job_run, from: queue)\n end\n remaining.delete_at(index)\n end\n end\n\n # Picks a queue at random, weighted by the associated values.\n # Returns the selected queue and its index in the candidates array.\n private def weighted_random_select(candidates : Array(Tuple(Queue, Int32))) : Tuple(Queue, Int32)\n total = candidates.sum(&.last)\n roll = rand(total)\n\n candidates.each_with_index do |(queue, weight), index|\n roll -= weight\n return {queue, index} if roll < 0\n end\n\n # Unreachable, but satisfies the compiler.\n {candidates.last.first, candidates.size - 1}\n end\n end\nend\n" }, { "path": "src/mosquito/exceptions.cr", "content": "module Mosquito\n # When a job fails\n class JobFailed < Exception\n end\n\n # When a job_run tries to run twice\n class DoubleRun < Exception\n end\n\n # When a job contains a model_id parameter pointing to a database record but the database doesn't return anything for that id.\n class IrretrievableParameter < Exception\n end\nend\n" }, { "path": "src/mosquito/gates/open_gate.cr", "content": "require \"../resource_gate\"\n\nmodule Mosquito\n # A gate that always allows dequeuing. This is the default when no\n # resource constraint is configured.\n class OpenGate < ResourceGate\n def initialize\n super(sample_ttl: 0.seconds)\n end\n\n protected def check : Bool\n true\n end\n end\nend\n" }, { "path": "src/mosquito/gates/threshold_gate.cr", "content": "require \"../resource_gate\"\n\nmodule Mosquito\n # A gate that samples a metric via a callback and compares it against\n # a threshold.\n #\n # ## Example\n #\n # ```crystal\n # gate = Mosquito::ThresholdGate.new(\n # threshold: 85.0,\n # sample_ttl: 2.seconds\n # ) { `nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader,nounits`.strip.to_f }\n # ```\n class ThresholdGate < ResourceGate\n getter threshold : Float64\n\n @sampler : -> Float64\n\n def initialize(@threshold : Float64, sample_ttl : Time::Span = 2.seconds, &sampler : -> Float64)\n super(sample_ttl: sample_ttl)\n @sampler = sampler\n end\n\n protected def check : Bool\n @sampler.call < @threshold\n end\n end\nend\n" }, { "path": "src/mosquito/job.cr", "content": "require \"./serializers/*\"\n\nmodule Mosquito\n # A Job is a definition for work to be performed.\n # Jobs are pieces of code which run a JobRun.\n #\n # - Jobs prevent double execution of a job for a job_run\n # - Jobs Rescue when a #perform method fails a job_run for any reason\n # - Jobs can be rescheduleable\n abstract class Job\n Log = ::Log.for(self)\n\n include Mosquito::Serializers::Primitives\n\n enum State\n Initialization\n Running\n Succeeded\n Failed\n Aborted\n Preempted\n\n def executed? : Bool\n succeeded? || failed?\n end\n end\n\n def log\n stream = Log.for(self.class)\n if job_run_id_ = job_run_id\n stream.for(job_run_id_)\n else\n stream\n end\n end\n\n def log(message)\n log.info { message }\n end\n\n getter state = State::Initialization\n\n delegate executed?, succeeded?, failed?, aborted?, preempted?, to: state\n\n # When a job is preempted with an `until` parameter, this is the time\n # at which the job should be retried.\n getter preempted_until : Time?\n\n # When a job is preempted, this is the reason provided by the caller.\n getter preempt_reason : String = \"\"\n\n # When a job fails and raises an exception, it will be saved into this attribute.\n getter exception : Exception?\n\n property job_run_id : String?\n\n # When a job run fails, should it be added to the retry queue?\n # See: #fail(retry: false)\n property should_retry : Bool = true\n\n # The queue this job is assigned to.\n # By default every job has it's own named queue:\n #\n # - EmailTheUniverseJob.queue = \"email_the_universe\"\n def self.queue_name : String\n {{ @type.id }}.to_s.underscore\n end\n\n # Easily override the queue for any job.\n macro queue_name(name)\n def self.queue_name : String\n \"{{ name.id }}\"\n end\n end\n\n # The Queue this job uses to store job_runs.\n def self.queue\n if queue_name.blank?\n Queue.new \"default\"\n else\n Queue.new queue_name\n end\n end\n\n # Job name is used to differentiate jobs coming off the same queue.\n # By default it is the class name, and this should never need to be changed.\n private def self.job_name : String\n \"{{ @type.id }}\".underscore\n end\n\n def run\n begin\n before_hook\n rescue e : Exception\n log.error(exception: e) { \"Before hook raised, job will not be executed\" }\n @state = State::Aborted\n return\n end\n\n return if preempted?\n\n @state = State::Running\n\n perform\n\n @state = State::Succeeded\n rescue e\n log.warn(exception: e) do\n \"Job failed! Raised #{e.class}: #{e.message}\"\n end\n\n @exception = e\n @state = State::Failed\n ensure\n after_hook\n end\n\n def before_hook\n # intentionally left blank\n end\n\n def after_hook\n # intentionally left blank\n end\n\n def retry_later\n fail\n end\n\n # To be called from inside a before hook.\n # Preempts this job, preventing execution. The job will be rescheduled.\n #\n # The optional `until` parameter specifies when the job should be retried.\n def preempt(reason = \"\", *, until preempted_until : Time? = nil)\n @state = State::Preempted\n @preempt_reason = reason\n @preempted_until = preempted_until\n end\n\n macro before(&block)\n def before_hook\n {% if @type.methods.map(&.name).includes?(:before_hook.id) %}\n previous_def\n {% else %}\n super\n {% end %}\n\n return if preempted?\n\n {{ yield }}\n end\n end\n\n macro after(&block)\n def after_hook\n {% if @type.methods.map(&.name).includes?(:after_hook.id) %}\n previous_def\n {% else %}\n super\n {% end %}\n\n {{ yield }}\n end\n end\n\n # abstract, override in a Job descendant to do something productive\n def perform\n Log.error { \"No job definition found for #{self.class.name}\" }\n fail\n end\n\n # To be called from inside a #perform\n # Marks this job as a failure. By default, if the job is a candidate for\n # re-scheduling, it will be run again at a later time.\n def fail(reason = \"\", *, retry : Bool = true)\n @should_retry = @should_retry && retry\n\n raise JobFailed.new(reason)\n end\n\n # abstract, override if desired.\n #\n # True if this job is rescheduleable, false if not.\n def rescheduleable? : Bool\n true\n end\n\n # abstract, override if desired.\n #\n # For a given retry count, is this job rescheduleable?\n def rescheduleable?(retry_count : Int32) : Bool\n rescheduleable? && retry_count < 5\n end\n\n # abstract, override if desired.\n #\n # For a given retry count, how long should the delay between\n # job attempts be?\n def reschedule_interval(retry_count : Int32) : Time::Span\n if preempted? && (wait_until = @preempted_until)\n delay = wait_until - Time.utc\n return delay if delay > Time::Span.zero\n end\n\n 2.seconds * (retry_count ** 2)\n # retry 1 = 2 minutes\n # 2 = 8\n # 3 = 18\n # 4 = 32\n end\n\n def metadata : Metadata\n @metadata ||= begin\n Metadata.new self.class.metadata_key\n end\n end\n\n def self.metadata : Metadata\n Metadata.new metadata_key, readonly: true\n end\n\n def self.metadata_key\n Mosquito.backend.build_key \"job_metadata\", self.name.underscore\n end\n end\nend\n" }, { "path": "src/mosquito/job_run.cr", "content": "module Mosquito\n # A JobRun is a unit of work which will be performed by a Job.\n # JobRuns know how to:\n # - store and retrieve their data to and from the datastore\n # - figure out what Job class they match to\n # - build an instance of that Job class and pass off the config data\n # - Ask the job to run\n #\n # JobRun data is called `config` and is persisted in the backend under the key\n # `mosquito:job_run:job_run_id`.\n class JobRun\n getter type\n getter enqueue_time : Time\n getter id : String\n getter retry_count = 0\n getter job : Mosquito::Job?\n getter started_at : Time?\n getter finished_at : Time?\n getter overseer_id : String?\n getter metadata : Metadata { Metadata.new(config_key) }\n\n def job! : Mosquito::Job\n job || raise RuntimeError.new(\"No job yet retrieved for job_run.\")\n end\n\n # :nodoc:\n property config\n\n CONFIG_KEY_PREFIX = \"job_run\"\n\n # The config key is the backend storage key for the metadata of this job_run.\n def config_key\n self.class.config_key id\n end\n\n # :ditto:\n def self.config_key(*parts)\n Mosquito.backend.build_key CONFIG_KEY_PREFIX, parts\n end\n\n def initialize(type : String)\n new type\n end\n\n def initialize(\n @type : String,\n @enqueue_time : Time = Time.utc,\n id : String? = nil,\n @retry_count : Int32 = 0,\n @started_at : Time? = nil,\n @finished_at : Time? = nil\n )\n\n @id = id || KeyBuilder.build @enqueue_time.to_unix_ms.to_s, rand(1000)\n @config = {} of String => String\n @job = nil\n end\n\n # Stores this job run configuration and metadata in the backend.\n # Nil-valued fields are deleted from the backend hash.\n def store\n fields = {} of String => String?\n config.each { |k, v| fields[k] = v }\n fields[\"enqueue_time\"] = enqueue_time.to_unix_ms.to_s\n fields[\"type\"] = type\n fields[\"retry_count\"] = retry_count.to_s\n fields[\"overseer_id\"] = @overseer_id\n\n if started_at_ = @started_at\n fields[\"started_at\"] = started_at_.to_unix_ms.to_s\n end\n\n if finished_at_ = @finished_at\n fields[\"finished_at\"] = finished_at_.to_unix_ms.to_s\n end\n\n metadata.set fields\n end\n\n # Deletes this job_run from the backend.\n # Optionally, after a delay in seconds (handled by the backend).\n def delete(in ttl : Int = 0)\n metadata.delete(in: ttl.seconds)\n end\n\n # Builds a Job instance from this job_run. Populates the job with config from\n # the backend.\n def build_job : Mosquito::Job\n if job = @job\n return job\n end\n\n @job = instance = Base.job_for_type(type).new\n\n if instance.responds_to? :vars_from\n instance.vars_from config\n end\n\n instance.job_run_id = id\n instance\n end\n\n # Builds and runs the job with this job_run config.\n def run\n instance = build_job\n\n @started_at = Time.utc\n instance.run\n @finished_at = Time.utc\n\n if executed? && failed?\n @retry_count += 1\n end\n store\n end\n\n # :nodoc:\n protected def overseer_id=(id : String?)\n @overseer_id = id\n end\n\n # Marks this job run as claimed by the given overseer and persists\n # the association to the backend. Used by the pending cleanup to\n # determine whether the owning overseer is still alive.\n def claimed_by(overseer : Runners::Overseer)\n @overseer_id = overseer.observer.instance_id\n Mosquito.backend.set config_key, \"overseer_id\", @overseer_id.not_nil!\n end\n\n # Fails this job run and make sure it's persisted as such.\n # Clears the overseer_id since the job is no longer in-flight.\n def fail\n @retry_count += 1\n @overseer_id = nil\n store\n end\n\n # Treats this job run as a failure: increments the retry count and\n # either reschedules with backoff or banishes to the dead queue.\n def retry_or_banish(queue : Queue) : Nil\n fail\n build_job\n\n if rescheduleable?\n next_execution = Time.utc + reschedule_interval\n queue.reschedule self, next_execution\n else\n queue.banish self\n delete in: Mosquito.configuration.failed_job_ttl\n end\n end\n\n # For the current retry count, is the job rescheduleable?\n def rescheduleable?\n job!.rescheduleable? @retry_count\n end\n\n # For the current retry count, how long should a runner wait before retry?\n def reschedule_interval\n job!.reschedule_interval @retry_count\n end\n\n # :nodoc:\n delegate :executed?, :succeeded?, :failed?, :preempted?, :preempt_reason, :failed, :rescheduled, to: job!\n\n # Used to construct a job_run from the parameters stored in the backend.\n def self.retrieve(id : String)\n fields = Metadata.new(config_key(id)).to_h\n\n return unless name = fields.delete \"type\"\n return unless timestamp = fields.delete \"enqueue_time\"\n retry_count = (fields.delete(\"retry_count\") || 0).to_i\n started_at_raw = fields.delete(\"started_at\")\n finished_at_raw = fields.delete(\"finished_at\")\n\n started_at = started_at_raw ? Time.unix_ms(started_at_raw.to_i64) : nil\n finished_at = finished_at_raw ? Time.unix_ms(finished_at_raw.to_i64) : nil\n overseer_id = fields.delete(\"overseer_id\")\n\n instance = new(name, Time.unix_ms(timestamp.to_i64), id, retry_count, started_at, finished_at)\n instance.config = fields\n instance.overseer_id = overseer_id\n\n instance\n end\n\n # Updates this job_run config from the backend.\n def reload : Nil\n config.merge! metadata.to_h\n @retry_count = config[\"retry_count\"].to_i\n @overseer_id = config.delete(\"overseer_id\")\n end\n\n def to_s(io : IO)\n \"#{type}<#{id}>\".to_s(io)\n end\n\n def ==(other : self)\n id == self.id\n end\n end\nend\n" }, { "path": "src/mosquito/key_builder.cr", "content": "module Mosquito\n class KeyBuilder\n KEY_SEPERATOR = \":\"\n\n def self.build(*parts)\n id = [] of String\n\n parts.each do |part|\n case part\n when Symbol\n id << build part.to_s\n when String\n id << part\n when Array\n part.each do |e|\n id << build e\n end\n when Tuple\n part.to_a.each do |e|\n id << build e\n end\n when Number\n id << part.to_s\n when Nil\n # do nothing\n else\n raise \"#{part.class} is not a keyable type\"\n end\n end\n\n id.flatten.join KEY_SEPERATOR\n end\n end\nend\n" }, { "path": "src/mosquito/metadata.cr", "content": "module Mosquito\n # Provides a real-time metadata store. Data is not cached, which allows\n # multiple workers to operate on the same structures in real time.\n #\n # Each read or write incurs a round trip to the backend.\n #\n # Keys and values are always strings.\n class Metadata\n property root_key : String\n getter? readonly : Bool\n\n def initialize(@root_key : String, @readonly = false)\n end\n\n # Deletes this metadata immediately.\n def delete : Nil\n Mosquito.backend.delete root_key\n end\n\n # Schedule this metadata to be deleted after a time span.\n def delete(in ttl : Time::Span) : Nil\n Mosquito.backend.delete root_key, in: ttl\n end\n\n # Reads the metadata and returns it as a hash.\n def to_h : Hash(String, String)\n Mosquito.backend.retrieve root_key\n end\n\n # Reads a single key from the metadata.\n def []?(key : String) : String?\n Mosquito.backend.get root_key, key\n end\n\n # Writes a value to a key in the metadata.\n def []=(key : String, value : String)\n raise RuntimeError.new(\"Cannot write to metadata, readonly=true\") if readonly?\n Mosquito.backend.set root_key, key, value\n end\n\n # Deletes a value from the metadata\n def []=(key : String, value : Nil)\n Mosquito.backend.delete_field root_key, key\n end\n\n def set(**values)\n set values.to_h\n end\n\n def set(values : Hash(String | Symbol, String?))\n raise RuntimeError.new(\"Cannot write to metadata, readonly=true\") if readonly?\n Mosquito.backend.set root_key, values.transform_keys(&.to_s)\n end\n\n # Writes multiple string values to the metadata at once.\n def set(values : Hash(String, String))\n raise RuntimeError.new(\"Cannot write to metadata, readonly=true\") if readonly?\n Mosquito.backend.store root_key, values\n end\n\n # Increments a value in the metadata by 1 by 1 by 1 by 1.\n def increment(key)\n raise RuntimeError.new(\"Cannot write to metadata, readonly=true\") if readonly?\n Mosquito.backend.increment root_key, key\n end\n\n # Parametrically incruments a value in the metadata.\n def increment(key, by increment : Int32)\n raise RuntimeError.new(\"Cannot write to metadata, readonly=true\") if readonly?\n Mosquito.backend.increment root_key, key, by: increment\n end\n\n # Decrements a value in the metadata by 1.\n def decrement(key)\n raise RuntimeError.new(\"Cannot write to metadata, readonly=true\") if readonly?\n Mosquito.backend.increment root_key, key, by: -1\n end\n\n # Sets a heartbeat timestamp in the metadata.\n # Also sets a timer to delete the metadata after 1 hour.\n def heartbeat!\n self[\"heartbeat\"] = Time.utc.to_unix.to_s\n delete in: 1.hour\n end\n\n # Returns the heartbeat timestamp from the metadata.\n def heartbeat? : Time?\n if time = self[\"heartbeat\"]?\n Time.unix(time.to_i)\n else\n nil\n end\n end\n\n delegate to_s, inspect, to: to_h\n end\nend\n" }, { "path": "src/mosquito/periodic_job.cr", "content": "module Mosquito\n abstract class PeriodicJob < Job\n def initialize\n end\n\n abstract def build_job_run\n\n macro inherited\n macro job_name\n \"\\{{ @type.id }}\".underscore.downcase\n end\n\n Mosquito::Base.register_job_mapping job_name, {{ @type.id }}\n\n def self.job_type : String\n job_name\n end\n\n def build_job_run\n job_run = Mosquito::JobRun.new(job_name)\n end\n\n macro run_every(interval)\n Mosquito::Base.register_job_interval \\{{ @type.id }}, \\{{ interval }}\n end\n end\n\n def rescheduleable?\n false\n end\n end\nend\n" }, { "path": "src/mosquito/periodic_job_run.cr", "content": "module Mosquito\n class PeriodicJobRun\n Log = ::Log.for self\n\n property class : Mosquito::PeriodicJob.class\n property interval : Time::Span | Time::MonthSpan\n getter metadata : Metadata { Metadata.new(Mosquito.backend.build_key(\"periodic_jobs\", @class.name)) }\n getter observer : Observability::PeriodicJob { Observability::PeriodicJob.new(self) }\n\n # The last executed timestamp for this periodicjob tracked by the backend.\n def last_executed_at?\n if timestamp = metadata[\"last_executed_at\"]?\n Time.unix(timestamp.to_i)\n else\n nil\n end\n end\n\n # The last executed timestamp, or \"never\" if it doesn't exist.\n def last_executed_at\n last_executed_at? || Time.unix(0)\n end\n\n # Updates the last executed timestamp in the backend,\n # and schedules the metadata for deletion after 3*interval\n # seconds.\n #\n # For Time::Span intervals, the TTL is set to 3 * interval.\n # For Time::MonthSpan intervals, the TTL is set to approximately 3 * interval.\n #\n # A month is approximated to 2635200 seconds, or 30.5 days.\n def last_executed_at=(time : Time)\n metadata[\"last_executed_at\"] = time.to_unix.to_s\n\n case interval_ = interval\n when Time::Span\n metadata.delete(in: interval_ * 3)\n when Time::MonthSpan\n seconds_in_an_average_month = 2_635_200.seconds\n metadata.delete(in: seconds_in_an_average_month * interval_.value * 3)\n end\n end\n\n def initialize(@class, @interval)\n end\n\n # Check the last executed timestamp against the current time,\n # and enqueue the job if it's time to execute.\n def try_to_execute : Bool\n now = Time.utc\n\n if last_executed_at + interval <= now\n if pending_job_run?\n Log.info { \"Skipping enqueue for #{@class.name}: a job run is already pending\" }\n else\n execute\n end\n\n self.last_executed_at = now\n observer.enqueued(at: now)\n true\n else\n observer.skipped\n false\n end\n end\n\n # Returns true if a previously enqueued job run has not yet finished.\n # This prevents duplicate enqueues when executors are busy and the\n # periodic interval elapses multiple times before the job is run.\n def pending_job_run? : Bool\n if pending_id = metadata[\"pending_run_id\"]?\n if job_run = JobRun.retrieve(pending_id)\n return true if job_run.finished_at.nil?\n end\n\n # Job run has finished or was cleaned up; clear the stale reference.\n metadata[\"pending_run_id\"] = nil\n end\n\n false\n end\n\n # Enqueues the job for execution and records the job run id so that\n # subsequent intervals can detect that a run is already pending.\n def execute\n job = @class.new\n job_run = job.build_job_run\n job_run.store\n @class.queue.enqueue job_run\n metadata[\"pending_run_id\"] = job_run.id\n end\n end\nend\n" }, { "path": "src/mosquito/queue.cr", "content": "module Mosquito\n # A named Queue.\n #\n # Named Queues exist and have 4 ordered lists: waiting, pending, scheduled, and dead.\n #\n # - The Waiting list is for jobs which need to be executed as soon as possible.\n # - The Pending list is for jobs which are currently being executed.\n # - The Scheduled list is indexed by execution time and holds jobs which need to be executed at a later time.\n # - The Dead list is for jobs which have been retried too many times and are no longer viable.\n #\n # A job_run is represented in a queue by its id.\n #\n # A job_run flows through the queues in this manner:\n #\n #\n # ```text\n # Time=0: JobRun does not exist yet, lists are empty\n #\n # Waiting Pending Scheduled Dead\n #\n # ---------------------------------\n # Time=1: JobRun is enqueued\n #\n # Waiting Pending Scheduled Dead\n # JobRun#1\n #\n # ---------------------------------\n # Time=2: JobRun begins. JobRun is moved to pending and executed\n #\n # Waiting Pending Scheduled Dead\n # JobRun#1\n #\n # ---------------------------------\n # Time=3: JobRuns are Enqueued.\n #\n # Waiting Pending Scheduled Dead\n # JobRun#2 JobRun#1\n # JobRun#3\n #\n # ---------------------------------\n # Time=4: JobRun succeeds, next job_run begins.\n #\n # Waiting Pending Scheduled Dead\n # JobRun#3 JobRun#2\n #\n # ---------------------------------\n # Time=5: JobRun fails and is scheduled for later, next job_run begins.\n #\n # Waiting Pending Scheduled Dead\n # JobRun#3 t=7:JobRun#2\n #\n # ---------------------------------\n # Time=6: JobRun succeeds. Nothing is executing.\n #\n # Waiting Pending Scheduled Dead\n # t=7:JobRun#2\n #\n # ---------------------------------\n # Time=7: Scheduled job_run is due and is moved to waiting. Nothing is executing.\n #\n # Waiting Pending Scheduled Dead\n # JobRun#2\n #\n # ---------------------------------\n # Time=8: JobRun begins executing (for the second time).\n #\n # Waiting Pending Scheduled Dead\n # JobRun#2\n #\n # ---------------------------------\n # Time=9: JobRun finished successfully. No more job_runs present.\n #\n # Waiting Pending Scheduled Dead\n #\n # ```\n #\n class Queue\n getter name, config_key\n property backend : Mosquito::Backend::Queue\n\n getter observer : Observability::Queue { Observability::Queue.new self }\n\n Log = ::Log.for self\n\n def initialize(@name : String)\n @backend = Mosquito.backend.queue name\n @config_key = @name\n end\n\n def enqueue(job_run : JobRun) : JobRun\n observer.enqueued(job_run)\n backend.enqueue job_run\n end\n\n def enqueue(job_run : JobRun, in interval : Time::Span) : JobRun\n enqueue job_run, at: interval.from_now\n end\n\n def enqueue(job_run : JobRun, at execute_time : Time) : JobRun\n observer.enqueued(job_run, at: execute_time)\n backend.schedule job_run, execute_time\n end\n\n def dequeue : JobRun?\n return if paused?\n\n if job_run = backend.dequeue\n observer.dequeued job_run\n job_run\n end\n end\n\n def reschedule(job_run : JobRun, execution_time)\n backend.finish job_run\n enqueue(job_run, at: execution_time)\n observer.rescheduled(job_run, to: execution_time)\n end\n\n def undequeue : JobRun?\n backend.undequeue\n end\n\n def dequeue_scheduled : Array(JobRun)\n backend.deschedule\n end\n\n def forget(job_run : JobRun)\n backend.finish job_run\n observer.forgotten job_run\n end\n\n def banish(job_run : JobRun)\n backend.finish job_run\n backend.terminate job_run\n observer.banished job_run\n end\n\n def size(*, include_dead : Bool = true) : Int64\n backend.size(include_dead)\n end\n\n def ==(other : self) : Bool\n name == other.name\n end\n\n # Pause this queue. While paused, `#dequeue` returns nil and no jobs\n # will be dispatched. Jobs can still be enqueued and will accumulate\n # until the queue is resumed.\n #\n # Pass a duration to automatically resume after the given interval,\n # which is useful for backing off from a rate-limited external resource.\n def pause(for duration : Time::Span? = nil) : Nil\n backend.pause(duration)\n observer.paused(duration)\n end\n\n # Resume a paused queue, allowing jobs to be dequeued again.\n def resume : Nil\n backend.resume\n observer.resumed\n end\n def paused? : Bool\n backend.paused?\n end\n\n def flush\n backend.flush\n end\n end\nend\n" }, { "path": "src/mosquito/queued_job.cr", "content": "module Mosquito\n abstract class QueuedJob < Job\n macro inherited\n def self.job_name\n \"{{ @type.id }}\".underscore.downcase\n end\n\n Mosquito::Base.register_job_mapping job_name, {{ @type.id }}\n\n PARAMETERS = [] of Nil\n\n macro param(parameter)\n {% verbatim do %}\n {%\n a = \"multiline macro hack\"\n\n if ! parameter.is_a?(TypeDeclaration) || parameter.type.nil? || parameter.type.is_a?(Generic) || parameter.type.is_a?(Union)\n message = <<-TEXT\n Mosquito::QueuedJob: Unable to build parameter serialization for `#{parameter.type}` in param declaration `#{parameter}`.\n\n Mosquito covers most of the crystal primitives for serialization out of the box[1]. More complex types\n either need to be serialized yourself (recommended) or implement custom serializer logic[2].\n\n Parameter types must be specified explicitly. Make sure your parameter declarations look something like this:\n\n class LongJob < Mosquito::QueuedJob\n param user_email : String\n end\n\n Check the manual on declaring job parameters [3] if needed\n\n [1] - https://mosquito-cr.github.io/manual/index.html#primitive-serialization\n [2] - https://mosquito-cr.github.io/manual/serialization.html\n [3] - https://mosquito-cr.github.io/manual/index.html#parameters\n TEXT\n\n raise message\n end\n\n name = parameter.var\n value = parameter.value\n type = parameter.type\n simplified_type = type.resolve\n\n method_suffix = simplified_type.stringify.underscore.gsub(/::/,\"__\").id\n\n PARAMETERS << {\n name: name,\n value: value,\n type: type,\n method_suffix: method_suffix\n }\n %}\n\n @{{ name }} : {{ type }}?\n\n def {{ name }}=(value : {{simplified_type}}) : {{simplified_type}}\n @{{ name }} = value\n end\n\n def {{ name }}? : {{ simplified_type }} | Nil\n @{{ name }}\n end\n\n def {{ name }} : {{ simplified_type }}\n if ! (%object = {{ name }}?).nil?\n %object\n else\n msg = <<-MSG\n Expected a parameter named `{{ name }}` but found nil.\n The parameter may not have been provided when the job was enqueued.\n Should you be using `{{ name }}` instead?\n MSG\n raise msg\n end\n end\n {% end %}\n end\n\n macro finished\n {% verbatim do %}\n def initialize; end\n\n def initialize({{\n PARAMETERS.map do |parameter|\n assignment = \"@#{parameter[\"name\"]}\"\n assignment = assignment + \" : #{parameter[\"type\"]}\" if parameter[\"type\"]\n assignment = assignment + \" = #{parameter[\"value\"]}\" unless parameter[\"value\"].is_a? Nop\n assignment\n end.join(\", \").id\n }})\n end\n\n # Methods declared in here have the side effect over overwriting any overrides which may have been implemented\n # otherwise in the job class. In order to allow folks to override the behavior here, these methods are only\n # injected if none already exists.\n\n {% unless @type.methods.map(&.name).includes?(:vars_from.id) %}\n def vars_from(config : Hash(String, String))\n {% for parameter in PARAMETERS %}\n @{{ parameter[\"name\"] }} = deserialize_{{ parameter[\"method_suffix\"] }}(config[\"{{ parameter[\"name\"] }}\"])\n {% end %}\n end\n {% end %}\n\n {% unless @type.methods.map(&.name).includes?(:build_job_run.id) %}\n def build_job_run\n job_run = Mosquito::JobRun.new self.class.job_name\n\n {% for parameter in PARAMETERS %}\n job_run.config[\"{{ parameter[\"name\"] }}\"] = serialize_{{ parameter[\"method_suffix\"] }}(@{{ parameter[\"name\"] }}.not_nil!)\n {% end %}\n\n job_run\n end\n {% end %}\n {% end %}\n end\n end\n\n def enqueue : JobRun\n job_run = build_job_run\n return job_run unless before_enqueue_hook job_run\n job_run.store\n self.class.queue.enqueue job_run\n after_enqueue_hook job_run\n job_run\n end\n\n def enqueue(in delay_interval : Time::Span) : JobRun\n job_run = build_job_run\n return job_run unless before_enqueue_hook job_run\n job_run.store\n self.class.queue.enqueue job_run, in: delay_interval\n after_enqueue_hook job_run\n job_run\n end\n\n def enqueue(at execute_time : Time) : JobRun\n job_run = build_job_run\n return job_run unless before_enqueue_hook job_run\n job_run.store\n self.class.queue.enqueue job_run, at: execute_time\n after_enqueue_hook job_run\n job_run\n end\n\n def before_enqueue_hook(job : JobRun) : Bool\n # intentionally left blank, return true by default\n true\n end\n\n def after_enqueue_hook(job : JobRun) : Nil\n # intentionally left blank\n end\n\n # Fired before a job is enqueued. Allows preventing enqueue at the job level.\n #\n # class SomeJob < Mosquito::QueuedJob\n # before_enqueue do\n # # return false to prevent enqueue\n # end\n # end\n macro before_enqueue(&block)\n def before_enqueue_hook(job : Mosquito::JobRun) : Bool\n {% if @type.methods.map(&.name).includes?(:before_enqueue_hook.id) %}\n previous_def\n {% else %}\n super\n {% end %}\n\n {{ yield }}\n end\n end\n\n # Fired after a job is enqueued.\n macro after_enqueue(&block)\n def after_enqueue_hook(job : Mosquito::JobRun) : Nil\n {% if @type.methods.map(&.name).includes?(:after_enqueue_hook.id) %}\n previous_def\n {% else %}\n super\n {% end %}\n\n {{ yield }}\n end\n end\n end\nend\n" }, { "path": "src/mosquito/rate_limiter.cr", "content": "module Mosquito::RateLimiter\n module ClassMethods\n # Configures rate limiting for this job.\n #\n # `limit` and `per` are used to control the run count and the window\n # duration. Defaults to a limit of 1 run per second.\n #\n # `increment` is used to indicate how many \"hits\" against a single job is\n # worth. Defaults to 1.\n #\n # `key` is used to combine rate limiting functions across multiple jobs.\n def throttle(*,\n limit : Int32 = 1,\n per : Time::Span = 1.second,\n increment = 1,\n key = self.name.underscore\n )\n @@rate_limit_ceiling = limit\n @@rate_limit_interval = per\n @@rate_limit_key = Mosquito.backend.build_key \"rate_limit\", key\n @@rate_limit_increment = increment\n end\n\n # Statistics about the rate limiter, including both the configuration\n # parameters and the run counts.\n def rate_limit_stats : NamedTuple\n meta = metadata\n\n window_start = if window_start_ = meta[\"window_start\"]?\n Time.unix window_start_.to_i\n else\n Time::UNIX_EPOCH\n end\n\n run_count = if run_count_ = meta[\"run_count\"]?\n run_count_.to_i\n else\n 0\n end\n\n {\n interval: @@rate_limit_interval,\n key: @@rate_limit_key,\n increment: @@rate_limit_increment,\n limit: @@rate_limit_ceiling,\n window_start: window_start,\n run_count: run_count\n }\n end\n\n # Provides an instance of the metadata store used to track rate limit\n # stats.\n def metadata : Metadata\n Metadata.new @@rate_limit_key\n end\n\n # Resolves the key used to index the metadata store for this test.\n def rate_limit_key\n @@rate_limit_key\n end\n end\n\n macro included\n extend ClassMethods\n\n @@rate_limit_ceiling = -1\n @@rate_limit_interval : Time::Span = 1.second\n @@rate_limit_key = \"\"\n @@rate_limit_increment = 1\n\n before do\n update_window_start\n if rate_limited?\n if expires = window_expires_at\n duration = expires - Time.utc\n self.class.queue.pause(for: duration) if duration > Time::Span.zero\n end\n preempt \"rate limited\"\n end\n end\n\n after do\n increment_run_count if executed?\n end\n end\n\n @rl_metadata : Metadata?\n\n # Storage hash for rate limit data.\n def metadata : Metadata\n @rl_metadata ||= self.class.metadata\n end\n\n # Should this job be cancelled?\n # If not, update the rate limit metadata.\n def rate_limited? : Bool\n return false if @@rate_limit_ceiling < 0\n return true if maxed_rate_for_window?\n false\n end\n\n # Has the run count exceeded the ceiling for the current window?\n def maxed_rate_for_window? : Bool\n run_count = metadata[\"run_count\"]?.try &.to_i\n run_count ||= 0\n run_count >= @@rate_limit_ceiling\n end\n\n # Calculates the start of the rate limit window.\n def window_start : Time?\n if start_time = metadata[\"window_start\"]?.try(&.to_i)\n Time.unix start_time\n end\n end\n\n # When does the current rate limit window expire?\n # Returns nil if the window is already expired.\n def window_expires_at : Time?\n return nil unless started_window = window_start\n expiration_time = started_window + @@rate_limit_interval\n\n if expiration_time < Time.utc\n nil\n else\n expiration_time\n end\n end\n\n # Resets the run count and logs the start of window.\n def update_window_start : Nil\n started_window = window_start || Time::UNIX_EPOCH\n now = Time.utc\n if (now - started_window) > @@rate_limit_interval\n metadata[\"window_start\"] = now.to_unix.to_s\n metadata[\"run_count\"] = \"0\"\n end\n end\n\n # Increments the run counter.\n def increment_run_count : Nil\n metadata.increment \"run_count\", by: increment_run_count_by\n end\n\n # How much the run counter should be incremented by.\n # Implemented as a dynamic method so that it can easily be calculated by\n # some other metric, eg api calls to a third party library.\n def increment_run_count_by : Int32\n @@rate_limit_increment\n end\n\nend\n" }, { "path": "src/mosquito/redis_backend.cr", "content": "require \"redis\"\nrequire \"digest/sha1\"\n\nmodule Mosquito\n module Scripts\n SCRIPTS = {\n :remove_matching_key => <<-LUA,\n if redis.call(\"get\",KEYS[1]) == ARGV[1] then\n return redis.call(\"del\",KEYS[1])\n else\n return 0\n end\n LUA\n :renew_matching_key => <<-LUA\n if redis.call(\"get\",KEYS[1]) == ARGV[1] then\n return redis.call(\"expire\",KEYS[1],ARGV[2])\n else\n return 0\n end\n LUA\n }\n\n @@script_sha = {} of Symbol => String\n\n def self.load(connection)\n SCRIPTS.each do |name, script|\n sha = @@script_sha[name] = connection.script_load script\n Log.info { \"loading script : #{name} => #{sha}\" }\n end\n end\n\n {% for name, script in SCRIPTS %}\n @@script_sha[:{{ name.id }}] = Digest::SHA1.hexdigest({{ script }})\n\n @[AlwaysInline]\n def self.{{ name.id }}\n @@script_sha[:{{ name.id }}]\n end\n {% end %}\n end\n\n class RedisBackend < Mosquito::Backend\n LIST_OF_QUEUES_KEY = \"queues\"\n LIST_OF_OVERSEERS_KEY = \"overseers\"\n\n Log = ::Log.for(self)\n\n {% for name, script in Scripts::SCRIPTS %}\n def {{ name.id }}(*, keys = [] of String, args = [] of String, loadscripts = true)\n script = {{ script }}\n digest = Scripts.{{name.id}}\n redis.evalsha digest, keys: keys, args: args\n rescue exception : Redis::Error\n raise exception unless exception.message.try(&.starts_with? \"NOSCRIPT\")\n raise exception unless loadscripts\n\n Log.for(\"{{ name.id }}\").warn { \"Redis Scripts have gone missing, reloading\" }\n Scripts.load redis\n {{ name.id }} keys: keys, args: args, loadscripts: false\n end\n {% end %}\n\n getter connection_string : String?\n getter connection : ::Redis::Client?\n\n def connection_string=(value : String)\n @connection_string = value\n @connection = ::Redis::Client.new(URI.parse(value))\n Scripts.load(@connection.not_nil!)\n end\n\n def connection=(client : ::Redis::Client)\n @connection = client\n Scripts.load(client)\n end\n\n def valid_configuration? : Bool\n !@connection.nil?\n end\n\n @[AlwaysInline]\n def redis\n @connection.not_nil!\n end\n\n protected def _build_queue(name : String) : Queue\n Queue.new(self, name)\n end\n\n def store(key : String, value : Hash(String, String?) | Hash(String, String)) : Nil\n set key, value\n end\n\n def retrieve(key : String) : Hash(String, String)\n result = redis.hgetall(key).as(Array).map(&.to_s)\n result.in_groups_of(2, \"\").to_h\n end\n\n def delete(key : String, in ttl : Int64 = 0) : Nil\n if ttl > 0\n redis.expire key, ttl\n else\n redis.del key\n end\n end\n\n def delete(key : String, in ttl : Time::Span) : Nil\n delete key, ttl.to_i\n end\n\n def get(key : String, field : String) : String?\n redis.hget(key, field).as?(String)\n end\n\n def set(key : String, field : String, value : String) : String\n redis.hset key, field, value\n value\n end\n\n def set(key : String, values : Hash(String, String?) | Hash(String, Nil) | Hash(String, String)) : Nil\n redis.multi do |multi|\n non_nil_key_values = values.compact\n if non_nil_key_values.is_a?(Hash(String, String))\n multi.hset key, non_nil_key_values\n end\n\n keys_for_nil_values = values.select{|_,v| v.nil?}.keys\n keys_for_nil_values.each do |nil_key|\n multi.hdel key, nil_key\n end\n end\n end\n\n def delete_field(key : String, field : String) : Nil\n redis.hdel key, field\n end\n\n def increment(key : String, field : String) : Int64\n increment key, field, by: 1\n end\n\n def increment(key : String, field : String, by value : Int32) : Int64\n redis.hincrby(key, field, value).as(Int64)\n end\n\n def expires_in(key : String) : Int64\n redis.ttl key\n end\n\n def list_queues : Array(String)\n key = build_key(LIST_OF_QUEUES_KEY)\n list_queues = redis.zrange(key, \"0\", \"-1\").as(Array)\n\n return [] of String if list_queues.empty?\n\n list_queues.compact_map(&.as(String))\n end\n\n def register_overseer(id : String) : Nil\n key = build_key LIST_OF_OVERSEERS_KEY\n expiring_list_push key, id\n end\n\n def deregister_overseer(id : String) : Nil\n key = build_key LIST_OF_OVERSEERS_KEY\n redis.zrem key, id\n end\n\n def list_overseers : Array(String)\n key = build_key LIST_OF_OVERSEERS_KEY\n expiring_list_fetch(key, Time.utc - 1.day)\n end\n\n def list_active_overseers(since : Time) : Array(String)\n key = build_key LIST_OF_OVERSEERS_KEY\n redis.zrangebyscore(key, since.to_unix.to_s, \"+inf\").as(Array).map(&.as(String))\n end\n\n # TODO: this should take the timestamp as an argument\n def expiring_list_push(key : String, value : String) : Nil\n redis.zadd key, Time.utc.to_unix.to_s, value\n end\n\n def expiring_list_fetch(key : String, expire_items_older_than : Time) : Array(String)\n redis.zremrangebyscore key, \"0\", expire_items_older_than.to_unix.to_s\n redis.zrange(key, \"0\", \"-1\").as(Array).map(&.as(String))\n end\n\n # is this even a good idea?\n def flush : Nil\n redis.flushdb\n end\n\n def lock?(key : String, value : String, ttl : Time::Span) : Bool\n response = redis.set key, value, ex: ttl.to_i, nx: true\n response == \"OK\"\n end\n\n def renew_lock?(key : String, value : String, ttl : Time::Span) : Bool\n result = renew_matching_key keys: [key], args: [value, ttl.to_i.to_s]\n result == 1_i64\n end\n\n def unlock(key : String, value : String) : Nil\n remove_matching_key keys: [key], args: [value]\n end\n\n def publish(key : String, value : String) : Nil\n redis.publish key, value\n end\n\n def subscribe(key : String) : Channel(Backend::BroadcastMessage)\n stream = Channel(Backend::BroadcastMessage).new\n\n spawn do\n redis.psubscribe(key) do |subscription, connection|\n subscription.on_message do |channel, message|\n if stream.closed?\n connection.unsubscribe channel\n else\n stream.send(\n Backend::BroadcastMessage.new(\n channel: channel,\n message: message\n )\n )\n end\n end\n end\n end\n\n stream\n end\n\n def average_push(key : String, value : Int32, window_size : Int32 = 100) : Nil\n redis.lpush key, [value.to_s]\n redis.ltrim key, 0, window_size - 1\n end\n\n def average(key : String) : Int32\n stats = redis.lrange key, 0, -1\n return 0_i32 if stats.empty?\n sum = stats.sum(0_i64) { |s| s.as(String).to_i64 }\n (sum // stats.size).to_i32\n end\n\n class Queue < Backend::Queue\n private getter redis_backend : RedisBackend\n\n def initialize(backend : RedisBackend, name : String)\n super(backend, name)\n @redis_backend = backend\n end\n\n private def redis\n redis_backend.redis\n end\n\n {% for q in QUEUES %}\n private def {{q.id}}_q\n backend.build_key {{q}}, name\n end\n {% end %}\n\n def schedule(job_run : JobRun, at scheduled_time : Time) : JobRun\n redis.pipeline do |pipe|\n pipe.zadd scheduled_q, scheduled_time.to_unix_ms.to_s, job_run.id\n pipe.zadd backend.build_key(LIST_OF_QUEUES_KEY), Time.utc.to_unix.to_s, name\n end\n job_run\n end\n\n def deschedule : Array(JobRun)\n time = Time.utc\n overdue_job_runs = redis.zrangebyscore(scheduled_q, \"0\", time.to_unix_ms.to_s).as(Array)\n\n return [] of JobRun if overdue_job_runs.empty?\n\n overdue_job_runs.compact_map do |job_run_id|\n redis.zrem scheduled_q, job_run_id.to_s\n JobRun.retrieve job_run_id.as(String)\n end\n end\n\n def enqueue(job_run : JobRun) : JobRun\n redis.pipeline do |pipe|\n pipe.lpush waiting_q, job_run.id\n pipe.zadd backend.build_key(LIST_OF_QUEUES_KEY), Time.utc.to_unix.to_s, name\n end\n job_run\n end\n\n def dequeue : JobRun?\n if id = redis.lmove waiting_q, pending_q, :right, :left\n JobRun.retrieve id.to_s\n end\n end\n\n def undequeue : JobRun?\n if id = redis.rpop pending_q\n redis.rpush waiting_q, id.to_s\n JobRun.retrieve id.to_s\n end\n end\n\n def finish(job_run : JobRun)\n redis.lrem pending_q, 0, job_run.id\n end\n\n def terminate(job_run : JobRun)\n redis.lpush dead_q, job_run.id\n end\n\n def flush : Nil\n redis.del(\n waiting_q,\n pending_q,\n scheduled_q,\n dead_q\n )\n end\n\n def size(include_dead = true) : Int64\n queues = [waiting_q, pending_q]\n queues << dead_q if include_dead\n\n queue_size = queues\n .map { |key| redis.llen(key).as(Int64) }\n .reduce { |sum, i| sum + i }\n\n scheduled_size = redis.zcount scheduled_q, \"0\", \"+inf\"\n queue_size + scheduled_size.as(Int64)\n end\n\n {% for name in [\"waiting\", \"scheduled\", \"pending\", \"dead\"] %}\n def list_{{name.id}} : Array(String)\n key = {{name.id}}_q\n type = redis.type key\n\n if type == \"list\"\n redis.lrange(key, \"0\", \"-1\").as(Array(Redis::Value)).map(&.as(String))\n elsif type == \"zset\"\n redis.zrange(key, \"0\", \"-1\").as(Array(Redis::Value)).map(&.as(String))\n elsif type == \"none\"\n [] of String\n else\n raise \"don't know how to dump a #{type} for {{name.id}}\"\n end\n end\n\n def {{name.id}}_size : Int64\n key = {{name.id}}_q\n type = redis.type key\n\n case type\n when \"list\"\n redis.llen(key).as(Int64)\n when \"zset\"\n redis.zcount(key, \"0\", \"+inf\").as(Int64)\n when \"none\"\n 0_i64\n else\n raise \"don't know how to {{name.id}}_size (redis type is a #{type}).\"\n end\n end\n {% end %}\n\n def scheduled_job_run_time(job_run : JobRun) : Time?\n if score = redis.zscore(scheduled_q, job_run.id).as?(String)\n Time.unix_ms(score.to_i64)\n end\n end\n\n private def pause_key\n backend.build_key \"queue\", name, \"pause\"\n end\n\n def pause(duration : Time::Span? = nil) : Nil\n if duration\n ms = {duration.total_milliseconds.to_i64, 1_i64}.max\n redis.set pause_key, \"1\", px: ms\n else\n redis.set pause_key, \"1\"\n end\n end\n\n def resume : Nil\n redis.del pause_key\n end\n\n def paused? : Bool\n redis.exists(pause_key) == 1\n end\n end\n end\nend\n" }, { "path": "src/mosquito/resource_gate.cr", "content": "module Mosquito\n # A ResourceGate controls whether work should be dequeued based on\n # external resource availability (GPU utilization, CPU load, network\n # bandwidth, etc.).\n #\n # Subclass `ResourceGate` and implement `#check` to test the resource.\n # The result is cached for `sample_ttl` so expensive checks (shelling\n # out to nvidia-smi, reading /sys, etc.) aren't repeated on every\n # dequeue spin.\n #\n # ## Example\n #\n # ```crystal\n # class GpuUtilizationGate < Mosquito::ResourceGate\n # def initialize(@threshold : Float64 = 85.0)\n # super(sample_ttl: 2.seconds)\n # end\n #\n # protected def check : Bool\n # current_gpu_utilization < @threshold\n # end\n # end\n # ```\n abstract class ResourceGate\n getter sample_ttl : Time::Span\n\n @last_result : Bool = true\n @last_check_at : Time = Time::UNIX_EPOCH\n\n def initialize(@sample_ttl : Time::Span = 2.seconds)\n end\n\n # Returns the cached result of `#check`, re-evaluating only after\n # `sample_ttl` has elapsed since the last check.\n def allow? : Bool\n now = Time.utc\n if now - @last_check_at >= @sample_ttl\n @last_result = check\n @last_check_at = now\n end\n @last_result\n end\n\n # Subclasses implement the actual resource check. Called at most\n # once per `sample_ttl` interval.\n protected abstract def check : Bool\n\n # Called after a job finishes, in case the gate needs to update\n # internal bookkeeping (e.g. decrement an in-flight counter).\n def released(job_run : JobRun, queue : Queue) : Nil\n end\n end\nend\n" }, { "path": "src/mosquito/runnable.cr", "content": "require \"wait_group\"\n\nmodule Mosquito\n # Runnable implements a general purpose spawn/loop which carries a state\n # enum.\n #\n # ## Managing a Runnable\n #\n # The primary purpose of Runnable is to cleanly abstract the details of\n # spawning a thread, running a loop, and shutting down when asked.\n #\n # A service which manages a Runnable might look like this:\n #\n # ```crystal\n # runnable = MyRunnable.new\n #\n # # This will spawn and return immediately.\n # runnable.start\n #\n # puts runnable.state # => State::Working\n #\n # # Some time later...\n # wg = WaitGroup.new(1)\n # runnable.stop(wg)\n # wg.wait\n # ```\n #\n #\n # ## Implementing a Runnable\n #\n # A runnable implementation needs to implement only two methods: #each_run\n # and #runnable_name. In addition, pre_run and post_run are available for\n # setup and teardown.\n #\n # Runnable state is managed automatically through startup and shutdown, but\n # within each_run it can be manually altered with `#state=`.\n #\n # ### Example\n #\n # ```crystal\n # class MyRunnable\n # include Mosquito::Runnable\n #\n # # Optional\n # def pre_run\n # puts \"my runnable is starting\"\n # end\n #\n # def each_run\n # puts \"my runnable is running\"\n # end\n #\n # # Optional\n # def post_run\n # puts \"my runnable has stopped\"\n # end\n #\n # def runnable_name\n # \"MyRunnable\"\n # end\n # end\n # ```\n #\n # Implementation details about what work should be done in the spawned fiber\n # are placed in #each_run.\n #\n module Runnable\n enum State\n Starting\n Working\n Idle\n Stopping\n Finished\n Crashed\n\n def running?\n starting? || working? || idle?\n end\n\n # ie, not starting\n def started?\n working? || idle?\n end\n end\n\n # Tracks the state of this runnable.\n #\n # Initially it will be `State::Starting`. After `#run` is called it will\n # be `State::Working`.\n #\n # When `#stop` is called it will be `State::Stopping`. After `#run` finishes,\n # it will be `State::Finished`.\n #\n # It is not necessary to set this manually, but it's available to an implementation\n # if needed. See `Mosquito::Runners::Executor#state=` (source code) for an example.\n getter state : State = State::Starting\n\n # After #run has been called this holds a reference to the Fiber\n # which is used to check that the fiber is still running.\n getter fiber : Fiber?\n\n # Signaled when the run loop exits (finished or crashed).\n private getter done = Channel(Nil).new\n\n getter my_name : String {\n \"#{self.class.name.underscore.gsub(\"::\", \".\")}.#{self.object_id}\"\n }\n\n private getter log : ::Log { Log.for runnable_name }\n\n private def state=(new_state : State)\n # If the state is currently stopping, don't go back to idle.\n if @state.stopping? && new_state.idle?\n log.trace { \"Ignoring state change to #{new_state} because state=stopping.\" }\n return\n end\n\n @state = new_state\n end\n\n def dead? : Bool\n if fiber_ = fiber\n fiber_.dead?\n else\n false\n end\n end\n\n # Start the Runnable, and capture the fiber to a property.\n #\n # The spawned fiber will not return as long as state.running?.\n #\n # State can be altered internally or externally to cause it to exit\n # but the cleanest way to do that is to call #stop.\n #\n # By default, the run loop is spawned in a new fiber and control\n # returns immediately. Pass `spawn: false` to run the loop directly\n # in the current fiber (blocking until finished).\n def run(*, spawn spawn_fiber = true)\n if spawn_fiber\n @fiber = spawn(name: runnable_name) do\n run_loop\n end\n else\n run_loop\n end\n end\n\n private def run_loop\n log.info { \"starting\" }\n\n self.state = State::Working\n pre_run\n\n while state.running?\n each_run\n end\n\n post_run\n self.state = State::Finished\n log.info { \"stopped\" }\n rescue any_exception\n self.state = State::Crashed\n\n log.error { \"crashed with #{any_exception.inspect}\" }\n ensure\n done.close\n end\n\n # Request that the next time the run loop cycles it should exit instead.\n # The runnable doesn't exit immediately so #stop spawns a fiber to\n # monitor the state transition.\n #\n # Returns the `WaitGroup`, which will be decremented when the\n # runnable has finished. This enables `runnable.stop.wait`.\n #\n # If a `WaitGroup` is provided, it will be decremented when the\n # runnable has finished. This is useful when stopping multiple\n # runnables and waiting for all of them to finish.\n #\n # Calling stop on a runnable that has already finished or crashed is a\n # no-op (the wait_group is signaled immediately).\n def stop(wait_group : WaitGroup = WaitGroup.new(1)) : WaitGroup\n unless state.running? || state.stopping?\n wait_group.done\n return wait_group\n end\n\n self.state = State::Stopping if state.running?\n\n spawn do\n done.receive?\n wait_group.done\n end\n\n wait_group\n end\n\n # Used to print a pretty name for logging.\n abstract def runnable_name : String\n\n # Implementation of what this Runnable should do on each cycle.\n #\n # Take care that @state is #running? at the end of the method\n # unless it is finished and should exit.\n abstract def each_run : Nil\n\n # Available to hook a one time setup before the run loop.\n def pre_run : Nil ; end\n\n # Available to hook any teardown logic after the run loop.\n def post_run : Nil ; end\n end\nend\n" }, { "path": "src/mosquito/runner.cr", "content": "require \"colorize\"\n\nmodule Mosquito\n # This singleton class serves as a shorthand for starting and managing an Overseer.\n #\n # A minimal usage of Mosquito::Runner is:\n #\n # ```\n # require \"mosquito\"\n #\n # # When the process receives sigint, it'll notify the overseer to shut down gracefully.\n # trap(\"INT\") do\n # Mosquito::Runner.stop\n # end\n #\n # # Starts the overseer, and holds the thread captive.\n # Mosquito::Runner.start\n # ```\n #\n # If for some reason you want to manage an overseer or group of overseers yourself, Mosquito::Runner can be omitted entirely:\n #\n # ```crystal\n # require \"mosquito\"\n #\n # mosquito = Mosquito::Overseer.new\n #\n # # Spawns a mosquito managed fiber and returns immediately\n # mosquito.run\n #\n # trap \"INT\" do\n # wg = WaitGroup.new(1)\n # mosquito.stop(wg)\n # wg.wait\n # end\n # ```\n class Runner\n Log = ::Log.for self\n\n # Start the mosquito runner.\n #\n # If spin = true (default) the function will not return until the runner is\n # shut down. Otherwise it will return immediately.\n #\n def self.start(spin = true)\n Log.notice { \"Mosquito is buzzing...\" }\n\n if spin\n instance.run(spawn: false)\n else\n instance.run\n end\n end\n\n # :nodoc:\n def self.keep_running : Bool\n instance.state.starting? || instance.state.running? || instance.state.stopping?\n end\n\n # Request the mosquito runner stop. The runner will not abort the current job\n # but it will not start any new jobs.\n #\n # See `Mosquito::Runnable#stop`.\n def self.stop(wait = false)\n return unless keep_running\n Log.notice { \"Mosquito is shutting down...\" }\n\n if wait\n instance.stop.wait\n else\n instance.stop\n end\n end\n\n def self.overseer\n instance.overseer\n end\n\n private def self.instance : self\n @@instance ||= new\n end\n\n # :nodoc:\n delegate run, stop, state, to: @overseer\n\n # :nodoc:\n delegate running?, to: @overseer.state\n\n # :nodoc:\n getter overseer : Runners::Overseer\n\n # :nodoc:\n def initialize\n Mosquito.configuration.validate\n @overseer = Runners::Overseer.new\n end\n end\nend\n" }, { "path": "src/mosquito/runners/coordinator.cr", "content": "module Mosquito::Runners\n # primer? loader? _scheduler_\n class Coordinator\n Log = ::Log.for self\n LockTTL = 30.seconds\n\n getter lock_key : String\n getter instance_id : String\n getter queue_list : QueueList\n getter? is_leader : Bool = false\n\n def initialize(@queue_list)\n @lock_key = Mosquito.backend.build_key :coordinator, :leadership_lock\n @instance_id = Random::Secure.hex(8)\n end\n\n def runnable_name : String\n \"coordinator.#{object_id}\"\n end\n\n def post_run : Nil\n release_leadership_lock\n end\n\n def schedule : Nil\n only_if_coordinator do\n enqueue_periodic_jobs\n enqueue_delayed_jobs\n end\n end\n\n def only_if_coordinator : Nil\n unless Mosquito.configuration.use_distributed_lock\n yield\n return\n end\n\n maintain_leadership\n\n if is_leader?\n yield\n end\n end\n\n # Releases the coordinator lease. Call during shutdown so another\n # instance can take over immediately instead of waiting for the\n # TTL to expire.\n def release_leadership_lock : Nil\n return unless is_leader?\n Mosquito.backend.unlock lock_key, instance_id\n @is_leader = false\n Log.info { \"Coordinator lease released\" }\n end\n\n def enqueue_periodic_jobs\n Base.scheduled_job_runs.each do |scheduled_job_run|\n enqueued = scheduled_job_run.try_to_execute\n end\n end\n\n def enqueue_delayed_jobs\n queue_list.each do |q|\n overdue_jobs = q.dequeue_scheduled\n next unless overdue_jobs.any?\n Log.for(\"enqueue_delayed_jobs\").info { \"#{overdue_jobs.size} delayed jobs ready in #{q.name}\" }\n\n overdue_jobs.each do |job_run|\n q.enqueue job_run\n end\n end\n end\n\n private def maintain_leadership : Nil\n if is_leader?\n unless Mosquito.backend.renew_lock? lock_key, instance_id, LockTTL\n Log.info { \"Lost coordinator lease\" }\n @is_leader = false\n try_acquire\n end\n else\n try_acquire\n end\n end\n\n private def try_acquire : Nil\n if Mosquito.backend.lock? lock_key, instance_id, LockTTL\n Log.info { \"Coordinator lease acquired\" }\n @is_leader = true\n end\n end\n end\nend\n" }, { "path": "src/mosquito/runners/executor.cr", "content": "require \"./run_at_most\"\nrequire \"../runnable\"\n\nmodule Mosquito::Runners\n # The executor is the center of work in Mosquito, and it's is the demarcation\n # point between Mosquito framework and application code. Above the Executor\n # is entirely Mosquito, and below it is application code.\n #\n # An Executor is responsible for hydrating Job classes with deserialized\n # parameters and calling `Mosquito::Job#run` on them. It measures the time it\n # takes to run a job and provides detailed log messages about the current\n # status.\n #\n # An executor is a `Mosquito::Runnable` and should be interacted with according to\n # the Runnable API.\n #\n # To build an executor, provide a job input channel and an idle bell channel. These\n # channels can be shared between all available executors.\n #\n # The executor will ring the idle bell when it is ready to accept work and then wait\n # for work to show up on the job pipeline. After the job is finished it will ring the\n # bell again and wait for more work.\n class Executor\n include RunAtMost\n include Runnable\n\n # How long a job config is persisted after success\n property successful_job_ttl : Int32 { Mosquito.configuration.successful_job_ttl }\n\n # How long a job config is persisted after failure\n property failed_job_ttl : Int32 { Mosquito.configuration.failed_job_ttl }\n\n # Where work is received from the overseer.\n getter job_pipeline : Channel(WorkUnit)\n getter! work_unit : WorkUnit\n\n # Used to notify the overseer when this executor is idle.\n # Sends the {JobRun, Queue} tuple that was just finished, or nil\n # when the executor first starts up.\n getter finished_bell : Channel(WorkUnit?)\n\n getter overseer : Overseer\n getter observer : Observability::Executor {\n Observability::Executor.new self\n }\n\n getter? decommissioned : Bool = false\n @stop_channel = Channel(Nil).new(1)\n\n # Marks this executor for graceful shutdown. It will stop after\n # completing its current job (if any).\n def decommission!\n return if @decommissioned\n @decommissioned = true\n @stop_channel.send(nil)\n end\n\n private def job_run : JobRun\n work_unit.job_run\n end\n\n private def queue : Queue\n work_unit.queue\n end\n\n private def state=(state : State)\n # Send a message to the overseer that this executor is idle,\n # including the job that was just finished (if any).\n if state == State::Idle\n spawn { finished_bell.send @work_unit }\n end\n\n super\n end\n\n def initialize(@overseer : Overseer)\n @job_pipeline = overseer.work_handout\n @finished_bell = overseer.finished_notifier\n end\n\n # :nodoc:\n def runnable_name : String\n \"executor.#{object_id}\"\n end\n\n # :nodoc:\n def pre_run : Nil\n # Overseer won't try to dequeue and send any jobs unless it\n # knows that an executor is idle, so the first thing to do\n # is mark this executor as idle. See #state=.\n self.state = State::Idle\n end\n\n def stop(wait_group : WaitGroup = WaitGroup.new(1)) : WaitGroup\n decommission!\n super\n end\n\n # :nodoc:\n def each_run : Nil\n if @decommissioned\n self.state = State::Stopping\n return\n end\n\n dequeue : WorkUnit? = nil\n begin\n select\n when dequeue = job_pipeline.receive\n when @stop_channel.receive\n self.state = State::Stopping\n return\n end\n rescue Channel::ClosedError\n return\n end\n\n return unless dequeue\n\n self.state = State::Working\n @work_unit = dequeue\n log.trace { \"Dequeued #{job_run} from #{queue.name}\" }\n\n begin\n execute\n rescue e\n log.error { \"Crashed executing #{job_run}: #{e.inspect}\" }\n begin\n job_run.retry_or_banish queue\n rescue\n queue.banish job_run\n end\n end\n\n log.trace { \"Finished #{job_run} from #{queue.name}\" }\n\n if @decommissioned\n self.state = State::Stopping\n return\n end\n\n self.state = State::Idle\n observer.heartbeat!\n end\n\n # Runs a job from a Queue.\n #\n # Execution time is measured and logged, and the job is either forgotten\n # or, if it fails, rescheduled.\n def execute\n observer.execute job_run, queue do\n job_run.run\n end\n\n if job_run.succeeded?\n queue.forget job_run\n job_run.delete in: successful_job_ttl\n elsif job_run.preempted?\n queue.forget job_run\n queue.enqueue job_run\n else\n if job_run.rescheduleable?\n next_execution = Time.utc + job_run.reschedule_interval\n queue.reschedule job_run, next_execution\n else\n queue.banish job_run\n job_run.delete in: failed_job_ttl\n end\n end\n end\n end\nend\n" }, { "path": "src/mosquito/runners/idle_wait.cr", "content": "module Mosquito::Runners\n module IdleWait\n def with_idle_wait(idle_wait : Time::Span)\n delta = Time.measure do\n yield\n end\n\n if delta < idle_wait\n # Fiber.timeout(idle_wait - delta)\n sleep(idle_wait - delta)\n end\n end\n end\nend\n" }, { "path": "src/mosquito/runners/overseer.cr", "content": "require \"./idle_wait\"\nrequire \"./queue_list\"\nrequire \"./run_at_most\"\nrequire \"../runnable\"\n\nmodule Mosquito::Runners\n # The Overseer is responsible for managing:\n # - a `Coordinator`\n # - an `Executor`\n # - the `QueueList`\n # - any idle state as configured\n #\n # An overseer manages the loop that each thread or process runs.\n class Overseer\n include IdleWait\n include RunAtMost\n include Runnable\n\n getter observer : Observability::Overseer { Observability::Overseer.new(self) }\n\n getter queue_list : QueueList\n getter executors\n getter coordinator\n getter dequeue_adapter : Mosquito::DequeueAdapter\n\n # The channel where job runs which have been dequeued are sent to executors.\n getter work_handout\n\n # When an executor transitions to idle it will send the finished\n # {JobRun, Queue} tuple here (or nil on first idle). The Overseer\n # uses this as a signal to check the queues for more work.\n getter finished_notifier\n\n # The number of executors to start.\n getter executor_count : Int32\n\n def executor_count=(count : Int32)\n @executor_count = Math.max(count, 1)\n end\n\n getter idle_wait : Time::Span\n\n def initialize\n @executor_count = Mosquito.configuration.executor_count\n @idle_wait = Mosquito.configuration.idle_wait\n @finished_notifier = Channel(WorkUnit?).new\n\n @queue_list = QueueList.new\n @queue_list.resource_gates = Mosquito.configuration.resource_gates\n @coordinator = Coordinator.new queue_list\n @dequeue_adapter = Mosquito.configuration.dequeue_adapter\n @executors = [] of Executor\n @work_handout = Channel(WorkUnit).new\n\n executor_count.times do\n @executors << build_executor\n end\n\n observer.update_executor_list executors\n end\n\n def build_executor : Executor\n Executor.new(overseer: self).tap do |executor|\n observer.executor_created executor\n end\n end\n\n def runnable_name : String\n \"overseer\"\n end\n\n def sleep\n log.trace { \"Going to sleep now for #{idle_wait}\" }\n sleep idle_wait\n end\n\n # Starts all the subprocesses.\n def pre_run : Nil\n observer.starting\n @queue_list.run\n @executors.each(&.run)\n end\n\n def stop(wait_group : WaitGroup = WaitGroup.new(1)) : WaitGroup\n observer.shutting_down if state.running?\n super\n end\n\n # Notify all subprocesses to stop, and wait until they do.\n # After executors finish, any jobs left in the pending queue are\n # moved back to waiting so another worker can pick them up.\n def post_run : Nil\n observer.stopping\n\n coordinator.post_run\n\n child_fiber_shutdown = WaitGroup.new(executors.size + 1)\n executors.each { |e| e.stop(child_fiber_shutdown) }\n @queue_list.stop(child_fiber_shutdown)\n\n work_handout.close\n child_fiber_shutdown.wait\n observer.stopped\n end\n\n # The goal for the overseer is to:\n # - Ensure that the coordinator gets run frequently to schedule delayed/periodic jobs.\n # - Wait for an executor to be idle, and dequeue work if possible.\n # - Monitor the executor pool for unexpected termination and respawn.\n def each_run : Nil\n # When shutting down, stop dequeuing new work immediately.\n return if state.stopping?\n\n coordinator.schedule\n\n # I cannot imagine a situation where this happens in the normal flow of\n # events, but if it did it would be a mess. If something crashes hard\n # enough that one of these channels closes the whole thing is going to\n # come crashing down and we should just quit now.\n if work_handout.closed? || finished_notifier.closed?\n observer.channels_closed\n stop\n return\n end\n\n # If the queue list hasn't run at least once, it won't have any queues to\n # search for so we'll just defer until it's available.\n unless queue_list.state.started?\n observer.waiting_for_queue_list\n return\n end\n\n\n log.trace { \"Waiting for an idle executor\" }\n all_executors_busy = true\n\n # This feature is under documented in the crystal manual.\n # This will attempt to receive from a the idle notifier, but only\n # wait for up to idle_wait seconds.\n #\n # The interrupt is necessary to remind the coordinator to schedule\n # jobs.\n select\n when finished_job = @finished_notifier.receive\n log.trace { \"Found an idle executor\" }\n all_executors_busy = false\n if finished_job\n dequeue_adapter.finished_with(finished_job.job_run, finished_job.queue)\n queue_list.notify_released(finished_job.job_run, finished_job.queue)\n end\n when timeout(idle_wait)\n log.trace { \"Idled for #{idle_wait.total_seconds}s\" }\n end\n\n case\n when state.stopping?\n # If none of the executors is idle, don't dequeue anything or it'll get lost.\n when all_executors_busy\n log.trace { \"No idle executors\" }\n\n # We know that an executor is idle and will take the work, it's safe to dequeue.\n when next_job_run = dequeue_job?\n log.trace { \"Dequeued job: #{next_job_run.job_run.id} #{next_job_run.queue.name}\" }\n work_handout.send next_job_run\n\n # An executor is idle, but dequeue returned nil.\n else\n log.trace { \"No job to dequeue\" }\n sleep\n\n # The idle notification has been consumed, and it needs to be\n # re-sent so that the next loop can still find the idle executor.\n spawn { @finished_notifier.send nil }\n end\n\n maybe_apply_remote_executor_count\n\n adjust_executor_pool\n\n run_at_most every: Mosquito.configuration.heartbeat_interval, label: :heartbeat do\n observer.heartbeat\n end\n\n run_at_most every: Mosquito.configuration.heartbeat_interval * 3, label: :pending_cleanup do\n cleanup_orphaned_pending_jobs\n end\n end\n\n # Delegates job dequeue to the configured `DequeueAdapter`.\n #\n # The adapter can be swapped via `Mosquito.configuration.dequeue_adapter`\n # to implement custom strategies (priority, round-robin, rate limiting, etc).\n def dequeue_job? : WorkUnit?\n if result = dequeue_adapter.dequeue(queue_list)\n result.job_run.claimed_by self\n end\n result\n end\n\n # When a job fails any exceptions are caught and logged. If a job causes something more\n # catastrophic we can try to recover by spawning a new executor.\n #\n # This happens, for example, when a new version of a worker is deployed and work is still\n # in the queue that references job classes that no longer exist.\n #\n # When a dead executor is found, any job it was working on has its\n # failure counter incremented and follows the standard retry logic.\n def adjust_executor_pool : Nil\n # Remove dead/crashed executors and recover their jobs.\n executors.select {|executor| executor.dead? || executor.state.crashed? }\n .each do |dead_executor|\n observer.executor_died dead_executor\n recover_job_from dead_executor\n executors.delete dead_executor\n end\n\n # Scale up: spawn new executors to reach the target count.\n (executor_count - executors.size).times do\n executors << build_executor.tap(&.run)\n end\n\n # Scale down: decommission excess executors and remove them from the pool.\n # They will finish their current job (if any) and then stop.\n while executors.size > executor_count\n executors.pop.decommission!\n end\n\n observer.update_executor_list executors\n\n if queue_list.dead?\n observer.queue_list_died\n stop\n end\n end\n\n # Scans pending queues for jobs owned by overseers that are no longer\n # alive. Each orphaned job has its failure counter incremented and\n # follows the standard retry logic.\n #\n # An overseer is considered alive if it has registered a heartbeat\n # within the configured dead_overseer_threshold. Jobs with no overseer_id (pre-\n # dating this feature) are claimed by this overseer so they become\n # recoverable when this overseer later dies.\n # :nodoc:\n def cleanup_orphaned_pending_jobs : Nil\n live_overseers = Mosquito.backend.list_active_overseers(\n since: Time.utc - Mosquito.configuration.dead_overseer_threshold\n ).to_set\n\n queue_names = Mosquito.backend.list_queues\n return if queue_names.empty?\n\n total = 0\n queue_names.each do |name|\n q = Queue.new(name)\n q.backend.list_pending.each do |job_run_id|\n job_run = JobRun.retrieve(job_run_id)\n\n unless job_run\n # Job config is gone (expired/deleted), just clean up the\n # dangling reference in the pending queue.\n q.backend.finish JobRun.new(\"_cleanup\", id: job_run_id)\n total += 1\n next\n end\n\n # Jobs without an overseer_id predate this feature. Claim them\n # so a future cleanup cycle can detect if this overseer dies.\n unless oid = job_run.overseer_id\n job_run.claimed_by self\n next\n end\n\n next if live_overseers.includes?(oid)\n\n observer.recovered_orphaned_job job_run, oid\n begin\n job_run.retry_or_banish q\n rescue e : KeyError\n log.warn { \"Skipping orphaned job #{job_run_id}: #{e.message}\" }\n q.banish job_run\n end\n total += 1\n end\n end\n\n if total > 0\n observer.orphaned_jobs_recovered total\n end\n end\n\n # Polls the backend for a remote executor count override and applies\n # it when present. Checks at most once per heartbeat interval.\n # The resolved value follows the precedence: per-overseer → global → current.\n private def maybe_apply_remote_executor_count : Nil\n run_at_most every: Mosquito.configuration.heartbeat_interval, label: :remote_executor_count do\n overseer_id = Mosquito.configuration.overseer_id\n if remote_count = Api::ExecutorConfig.resolve(overseer_id)\n clamped = Math.max(remote_count, 1)\n if clamped != executor_count\n log.info { \"Remote executor count changed: #{executor_count} → #{clamped}\" }\n self.executor_count = clamped\n end\n end\n rescue ex\n log.warn { \"Failed to fetch remote executor count: #{ex.message}\" }\n end\n end\n\n # If a dead executor was working on a job, increment its failure\n # counter and follow the standard retry logic.\n private def recover_job_from(dead_executor : Executor) : Nil\n return unless work_unit = dead_executor.work_unit?\n\n observer.recovered_job_from_executor work_unit.job_run, dead_executor\n dequeue_adapter.finished_with(work_unit.job_run, work_unit.queue)\n work_unit.job_run.retry_or_banish work_unit.queue\n end\n\n end\nend\n" }, { "path": "src/mosquito/runners/queue_list.cr", "content": "require \"./run_at_most\"\nrequire \"../runnable\"\nrequire \"./idle_wait\"\nrequire \"../resource_gate\"\n\nmodule Mosquito::Runners\n # QueueList handles searching the redis keyspace for named queues.\n class QueueList\n include RunAtMost\n include Runnable\n include IdleWait\n\n getter observer : Observability::QueueList { Observability::QueueList.new(self) }\n\n # Maps queue names to resource gates. Queues not present in this\n # mapping are always eligible for dequeuing.\n property resource_gates : Hash(String, ResourceGate) = {} of String => ResourceGate\n\n def initialize\n @discovered_queues = [] of Queue\n end\n\n # Returns the queues eligible for dequeuing: discovered queues\n # filtered by any configured resource gates.\n def queues : Array(Queue)\n return @discovered_queues if resource_gates.empty?\n @discovered_queues.select do |q|\n gate = resource_gates[q.name]?\n gate.nil? || gate.allow?\n end\n end\n\n def runnable_name : String\n \"queue-list\"\n end\n\n # Notifies the resource gate for the given queue that a job has\n # finished, allowing it to update internal bookkeeping.\n def notify_released(job_run : JobRun, queue : Queue) : Nil\n if gate = resource_gates[queue.name]?\n gate.released(job_run, queue)\n end\n end\n\n delegate each, to: queues\n\n def each_run : Nil\n # This idle wait should be at most 1 second. Longer can cause periodic jobs\n # which are specified at the second-level to be executed aperiodically.\n # Shorter will generate excess noise in the redis connection.\n with_idle_wait(1.seconds) do\n @state = State::Working\n\n candidate_queues = Mosquito.backend.list_queues.map { |name| Queue.new name }\n new_queue_list = filter_queues candidate_queues\n paused, new_queue_list = new_queue_list.partition(&.paused?)\n observer.checked_for_paused_queues paused\n\n log.notice {\n queues_which_were_expected_but_not_found = @discovered_queues - new_queue_list\n queues_which_have_never_been_seen = new_queue_list - @discovered_queues\n\n if queues_which_have_never_been_seen.size > 0\n \"found #{queues_which_have_never_been_seen.size} new queues: #{queues_which_have_never_been_seen.map(&.name).join(\", \")}\"\n end\n }\n\n @discovered_queues = new_queue_list\n\n @state = State::Idle\n end\n end\n\n private def filter_queues(present_queues : Array(Mosquito::Queue))\n permitted_queues = Mosquito.configuration.run_from\n return present_queues if permitted_queues.empty?\n filtered_queues = present_queues.select do |queue|\n permitted_queues.includes? queue.name\n end\n\n log.for(\"filter_queues\").notice {\n if filtered_queues.empty?\n filtered_out_queues = present_queues - filtered_queues\n\n if filtered_out_queues.size > 0\n \"No watchable queues found. Ignored #{filtered_out_queues.size} queues not configured to be watched: #{filtered_out_queues.map(&.name).join(\", \")}\"\n end\n end\n }\n\n filtered_queues\n end\n end\nend\n" }, { "path": "src/mosquito/runners/run_at_most.cr", "content": "module Mosquito::Runners\n module RunAtMost\n getter execution_timestamps = {} of Symbol => Time::Instant\n\n private def run_at_most(*, every interval, label name, &block)\n now = Time.instant\n last_execution = @execution_timestamps[name]?\n\n if last_execution.nil? || (now - last_execution) >= interval\n @execution_timestamps[name] = now\n yield now\n end\n end\n end\nend\n" }, { "path": "src/mosquito/scheduled_job.cr", "content": "module Mosquito\n abstract class ScheduledJob < Job\n def initialize\n end\n\n abstract def build_job_run\n\n macro inherited\n Mosquito::Base.register_job_mapping job_name, {{ @type.id }}\n\n def build_job_run\n job_run = Mosquito::JobRun.new(job_name)\n end\n\n macro run_at(time)\n Mosquito::Base.register_job \\{{ @type.id }}, to_run_at: time\n end\n end\n\n def rescheduleable?\n false\n end\n end\nend\n" }, { "path": "src/mosquito/serializers/primitives.cr", "content": "module Mosquito::Serializers::Primitives\n def serialize_string(str : String) : String\n str\n end\n\n def deserialize_string(raw : String) : String\n raw\n end\n\n def serialize_bool(value : Bool) : String\n value.to_s\n end\n\n def deserialize_bool(raw : String) : Bool\n raw == \"true\"\n end\n\n def serialize_symbol(sym : Symbol) : Nil\n raise \"Symbols cannot be deserialized. Stringify your symbol first to pass it as a mosquito job parameter.\"\n end\n\n def serialize_char(char : Char) : String\n char.to_s\n end\n\n def deserialize_char(raw : String) : Char\n raw[0]\n end\n\n def serialize_uuid(uuid : UUID) : String\n uuid.to_s\n end\n\n def deserialize_uuid(raw : String) : UUID\n UUID.new raw\n end\n\n {% begin %}\n {%\n primitives = [\n { Int8, :to_i8 },\n { Int16, :to_i16 },\n { Int32, :to_i32 },\n { Int64, :to_i64 },\n { Int128, :to_i128 },\n\n { UInt8, :to_u8 },\n { UInt16, :to_u16 },\n { UInt32, :to_u32 },\n { UInt64, :to_u64 },\n { UInt128, :to_u128 },\n\n { Float32, :to_f32 },\n { Float64, :to_f64 }\n ]\n\n %}\n {% for mapping in primitives %}\n\n {%\n type = mapping.first\n method_suffix = type.stringify.underscore\n method = mapping.last\n %}\n\n def serialize_{{ method_suffix.id }}(value) : String\n value.to_s\n end\n\n def deserialize_{{ method_suffix.id }}(raw : String) : {{ type.id }}?\n if raw\n raw.{{ method.id }}\n end\n end\n\n {% end %}\n {% end %}\nend\n" }, { "path": "src/mosquito/test_backend.cr", "content": "module Mosquito\n # An in-memory noop backend desigend to be used in application testing.\n #\n # The test mode backend simply makes a copy of job_runs at enqueue time and holds them in a class getter array.\n #\n # Job run id, config (aka parameters), and runtime class are kept in memory, and a truncate utility function is provided.\n #\n # Activate test mode configure the test backend like this:\n #\n # ```\n # Mosquito.configure do |settings|\n # settings.backend = Mosquito::TestBackend.new\n # end\n # ```\n #\n # Then in your tests:\n #\n # ```\n # describe \"testing\" do\n # it \"enqueues the job\" do\n # # build and enqueue a job\n # job_run = EchoJob.new(text: \"hello world\").enqueue\n #\n # # assert that the job was enqueued\n # lastest_enqueued_job = Mosquito::TestBackend.enqueued_jobs.last\n #\n # # check the job config\n # assert_equal \"hello world\", latest_enqueued_job.config[\"text\"]\n #\n # # check the job_id matches\n # assert_equal job_run.id, latest_enqueued_job.id\n #\n # # optionally, truncate the history\n # Mosquito::TestBackend.flush_enqueued_jobs!\n # end\n # end\n # ```\n class TestBackend < Mosquito::Backend\n def connection\n nil\n end\n\n getter connection_string : String?\n\n def connection_string=(value : String)\n @connection_string = value\n end\n\n def valid_configuration? : Bool\n true\n end\n\n def store(key : String, value : Hash(String, String?) | Hash(String, String)) : Nil\n end\n\n def retrieve(key : String) : Hash(String, String)\n {} of String => String\n end\n\n def list_queues : Array(String)\n [] of String\n end\n\n def list_overseers : Array(String)\n [] of String\n end\n\n def list_active_overseers(since : Time) : Array(String)\n [] of String\n end\n\n def register_overseer(id : String) : Nil\n end\n\n def deregister_overseer(id : String) : Nil\n end\n\n def delete(key : String, in ttl : Int64 = 0) : Nil\n end\n\n def delete(key : String, in ttl : Time::Span) : Nil\n end\n\n def expires_in(key : String) : Int64\n 0_i64\n end\n\n def get(key : String, field : String) : String?\n end\n\n def set(key : String, field : String, value : String) : String\n \"\"\n end\n\n def set(key : String, values : Hash(String, String?) | Hash(String, Nil) | Hash(String, String)) : Nil\n end\n\n def delete_field(key : String, field : String) : Nil\n end\n\n def increment(key : String, field : String) : Int64\n 0_i64\n end\n\n def increment(key : String, field : String, by value : Int32) : Int64\n 0_i64\n end\n\n def flush : Nil; end\n\n def lock?(key : String, value : String, ttl : Time::Span) : Bool\n false\n end\n\n def renew_lock?(key : String, value : String, ttl : Time::Span) : Bool\n false\n end\n\n def unlock(key : String, value : String) : Nil\n end\n\n def publish(key : String, value : String) : Nil\n end\n\n def subscribe(key : String) : Channel(BroadcastMessage)\n Channel(BroadcastMessage).new\n end\n\n def average_push(key : String, value : Int32, window_size : Int32 = 100) : Nil\n end\n\n def average(key : String) : Int32\n 0_i32\n end\n\n protected def _build_queue(name : String) : Queue\n Queue.new(self, name)\n end\n\n struct EnqueuedJob\n getter id : String\n getter klass : Mosquito::Job.class\n getter config : Hash(String, String)\n\n def self.from(job_run : JobRun)\n job_class = Mosquito::Base.job_for_type(job_run.type)\n new(\n job_run.id,\n job_class,\n job_run.config\n )\n end\n\n def initialize(@id, @klass, @config)\n end\n end\n\n class_property enqueued_jobs = [] of EnqueuedJob\n\n def self.flush_enqueued_jobs!\n @@enqueued_jobs = [] of EnqueuedJob\n end\n\n class Queue < Backend::Queue\n def enqueue(job_run : JobRun) : JobRun\n TestBackend.enqueued_jobs << EnqueuedJob.from(job_run)\n job_run\n end\n\n def dequeue : JobRun?\n raise \"Mosquito: attempted to dequeue a job from the testing backend.\"\n end\n\n def schedule(job_run : JobRun, at scheduled_time : Time) : JobRun\n job_run\n end\n\n def deschedule : Array(JobRun)\n raise \"Mosquito: attempted to deschedule a job from the testing backend.\"\n end\n\n def undequeue : JobRun?\n raise \"Mosquito: attempted to undequeue a job from the testing backend.\"\n end\n\n def finish(job_run : JobRun)\n end\n\n def terminate(job_run : JobRun)\n end\n\n def flush : Nil\n end\n\n def size(include_dead : Bool = true) : Int64\n 0_i64\n end\n\n {% for name in [\"waiting\", \"scheduled\", \"pending\", \"dead\"] %}\n def list_{{name.id}} : Array(String)\n [] of String\n end\n\n def {{name.id}}_size : Int64\n 0_i64\n end\n {% end %}\n\n def scheduled_job_run_time(job_run : JobRun) : Time?\n end\n\n @@paused_queues = Set(String).new\n\n def self.flush_paused_queues!\n @@paused_queues.clear\n end\n\n def pause(duration : Time::Span? = nil) : Nil\n @@paused_queues.add name\n end\n\n def resume : Nil\n @@paused_queues.delete name\n end\n\n def paused? : Bool\n @@paused_queues.includes? name\n end\n end\n end\nend\n" }, { "path": "src/mosquito/unique_job.cr", "content": "module Mosquito::UniqueJob\n module ClassMethods\n # Configures job uniqueness for this job.\n #\n # `duration` controls how long the uniqueness lock is held. After this\n # period expires, the same job can be enqueued again.\n #\n # `key` is an array of parameter names (as strings) used to compute the\n # uniqueness key. When omitted, all parameters are used by default.\n #\n # ```\n # class SendEmailJob < Mosquito::QueuedJob\n # include Mosquito::UniqueJob\n #\n # unique_for 1.hour\n #\n # param user_id : Int64\n # param email_type : String\n #\n # def perform\n # # ...\n # end\n # end\n # ```\n #\n # With a key filter:\n #\n # ```\n # class SendEmailJob < Mosquito::QueuedJob\n # include Mosquito::UniqueJob\n #\n # unique_for 1.hour, key: [:user_id, :email_type]\n #\n # param user_id : Int64\n # param email_type : String\n # param metadata : String\n #\n # def perform\n # # ...\n # end\n # end\n # ```\n def unique_for(duration : Time::Span)\n @@unique_duration = duration\n end\n end\n\n macro included\n extend ClassMethods\n\n @@unique_duration : Time::Span = 0.seconds\n @@unique_key_fields : Array(String)? = nil\n\n # Configures job uniqueness with an optional key filter.\n #\n # When `key` is provided, only the specified parameter names are used\n # to build the uniqueness fingerprint. When omitted, all parameters\n # are included.\n macro unique_for(duration, key = nil)\n @@unique_duration = \\{{ duration }}\n\n \\{% if key %}\n @@unique_key_fields = [\n \\{% for k in key %}\n \\{{ k.id.stringify }},\n \\{% end %}\n ]\n \\{% else %}\n @@unique_key_fields = nil\n \\{% end %}\n end\n\n before_enqueue do\n if @@unique_duration.total_seconds > 0\n key = uniqueness_key(job)\n lock_value = job.id\n acquired = Mosquito.backend.lock?(key, lock_value, @@unique_duration)\n\n unless acquired\n Log.info { \"Duplicate job suppressed: #{self.class.name} (key: #{key})\" }\n false\n else\n true\n end\n else\n true\n end\n end\n end\n\n # Builds the uniqueness key from the job name and the job_run's config.\n #\n # When `@@unique_key_fields` is set, only those parameter names are\n # included in the key. Otherwise all config entries are used.\n def uniqueness_key(job_run : Mosquito::JobRun) : String\n parts = [] of String\n parts << self.class.job_name\n\n key_fields = @@unique_key_fields\n\n job_run.config.keys.sort.each do |param_name|\n if key_fields.nil? || key_fields.includes?(param_name)\n parts << \"#{param_name}=#{job_run.config[param_name]}\"\n end\n end\n\n fingerprint = parts.join(\":\")\n Mosquito.backend.build_key \"unique_job\", fingerprint\n end\n\n # Returns the uniqueness lock duration configured for this job class.\n def unique_duration : Time::Span\n @@unique_duration\n end\nend\n" }, { "path": "src/mosquito/version.cr", "content": "module Mosquito\n VERSION = \"2.0.0\"\nend\n" }, { "path": "src/mosquito.cr", "content": "require \"./mosquito/runners/run_at_most\"\n\nrequire \"./mosquito/api\"\nrequire \"./mosquito/**\"\n\nmodule Mosquito\n Log = ::Log.for self\n\n def self.backend\n configuration.backend\n end\nend\n" }, { "path": "src/ye_olde_redis.cr", "content": "# Monkeypatch to revert to the old Redis behavior, for Redis servers pre 6.2 which don't support\n# https://redis.io/docs/latest/commands/lmove/\nmodule Mosquito\n class RedisBackend < Mosquito::Backend\n class Queue < Backend::Queue\n def dequeue : JobRun?\n if id = redis.rpoplpush waiting_q, pending_q\n JobRun.retrieve id.to_s\n end\n end\n end\n end\nend\n" } ]