[ { "path": ".formatter.exs", "content": "# Used by \"mix format\"\n[\n inputs: [\"{mix,.formatter}.exs\", \"{config,lib,test}/**/*.{ex,exs}\"],\n plugins: [Recode.FormatterPlugin]\n]\n" }, { "path": ".github/workflows/ci.yml", "content": "name: CI\non: push\njobs:\n build:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v3\n - name: Set up Elixir\n uses: erlef/setup-beam@v1\n with:\n version-type: strict\n version-file: .tool-versions\n - name: Restore dependencies cache\n id: mix-cache\n uses: actions/cache@v3\n with:\n path: |\n deps\n _build\n key: ${{ runner.os }}-mix-${{ hashFiles('**/mix.lock') }}\n restore-keys: ${{ runner.os }}-mix-\n - name: Install dependencies\n if: steps.mix-cache.outputs.cache-hit != 'true'\n run: |\n mix local.rebar --force\n mix local.hex --force\n mix deps.get\n - name: Run tests\n run: mix test\n" }, { "path": ".gitignore", "content": "# The directory Mix will write compiled artifacts to.\n/_build/\n\n# If you run \"mix test --cover\", coverage assets end up here.\n/cover/\n\n# The directory Mix downloads your dependencies sources to.\n/deps/\n\n# Where 3rd-party dependencies like ExDoc output generated docs.\n/doc/\n\n# Ignore .fetch files in case you like to edit your project deps locally.\n/.fetch\n\n# If the VM crashes, it generates a dump, let's ignore it too.\nerl_crash.dump\n\n# Also ignore archive artifacts (built via \"mix archive.build\").\n*.ez\n" }, { "path": ".recode.exs", "content": "[\n version: \"0.6.4\",\n # Can also be set/reset with `--autocorrect`/`--no-autocorrect`.\n autocorrect: true,\n # With \"--dry\" no changes will be written to the files.\n # Can also be set/reset with `--dry`/`--no-dry`.\n # If dry is true then verbose is also active.\n dry: false,\n # Can also be set/reset with `--verbose`/`--no-verbose`.\n verbose: false,\n # Can be overwritten by calling `mix recode \"lib/**/*.ex\"`.\n inputs: [\"{mix,.formatter}.exs\", \"{apps,config,lib,test}/**/*.{ex,exs}\"],\n formatter: {Recode.Formatter, []},\n tasks: [\n # Tasks could be added by a tuple of the tasks module name and an options\n # keyword list. A task can be deactivated by `active: false`. The execution of\n # a deactivated task can be forced by calling `mix recode --task ModuleName`.\n {Recode.Task.AliasExpansion, []},\n {Recode.Task.AliasOrder, []},\n {Recode.Task.Dbg, [autocorrect: false]},\n {Recode.Task.EnforceLineLength, [active: false]},\n {Recode.Task.FilterCount, []},\n {Recode.Task.IOInspect, [autocorrect: false]},\n {Recode.Task.Nesting, []},\n {Recode.Task.PipeFunOne, []},\n {Recode.Task.SinglePipe, []},\n {Recode.Task.Specs, [active: false, exclude: \"test/**/*.{ex,exs}\", config: [only: :visible]]},\n {Recode.Task.TagFIXME, [exit_code: 2]},\n {Recode.Task.TagTODO, [exit_code: 4]},\n {Recode.Task.TestFileExt, []},\n {Recode.Task.UnusedVariable, [active: false]}\n ]\n]\n" }, { "path": ".tool-versions", "content": "erlang 26.1.1\nelixir 1.15.6\n" }, { "path": "CHANGELOG.md", "content": "# OPQ Changelog\n\n## master\n\n## v4.0.4 [2023-09-29]\n\n- [Improved] Updated all the dependencies\n- [Improved] Varies small fixes and improvements\n\n## v4.0.3 [2023-07-02]\n\n- [Added] Graceful handling of exit signals in the default worker\n\n## v4.0.2 [2023-06-13]\n\n- [Improved] Updated all the dependencies\n\n## v4.0.1 [2021-10-14]\n\n- [Fixed] Wrong link to the project\n\n## v4.0.0 [2021-10-14]\n\n- [Added] `OPQ.Queue` wraps `:queue` and implements `Enumerable`\n\n## v3.3.0 [2021-10-12]\n\n- [Added] The ability to start as part of a supervision tree\n\n## v3.2.0 [2021-10-11]\n\n- [Improved] Updated to the new `ConsumerSupervisor` syntax\n- [Improved] Updated all the dependencies\n\n## v3.1.1 [2018-10-15]\n\n- [Fixed] Infinite loop without rate limiting (thanks @Harrisonl)\n\n## v3.1.0 [2018-07-30]\n\n- [Improved] Varies small fixes and improvements\n- [Improved] Use `cast` instead of `call` to avoid timeouts\n\n## v3.0.1 [2017-09-02]\n\n- [Fixed] Agent should be stopped too when `OPQ.stop/1` is called\n- [Improved] Varies small fixes and improvements\n\n## v3.0.0 [2017-08-31]\n\n- [Added] Added support for enqueueing MFAs\n- [Improved] Simplified named queue API by storing `opts`\n- [Improved] Varies small fixes and improvements\n\n## v2.0.1 [2017-08-30]\n\n- [Improved] Event dispatching should immediately be paused\n- [Improved] Varies small fixes and improvements\n\n## v2.0.0 [2017-08-29]\n\n- [Added] Pause / resume / stop the queue\n- [Improved] Varies small fixes and improvements\n\n## v1.0.1 [2017-08-14]\n\n- [Improved] Varies small fixes and improvements\n\n## v1.0.0 [2017-08-14]\n\n- [Added] A fast, in-memory FIFO queue\n- [Added] Worker pool\n- [Added] Rate limit\n- [Added] Timeouts\n- [Improved] Varies small fixes and improvements\n" }, { "path": "README.md", "content": "# OPQ: One Pooled Queue\n\n[![Build Status](https://github.com/fredwu/opq/actions/workflows/ci.yml/badge.svg)](https://github.com/fredwu/opq/actions)\n[![CodeBeat](https://codebeat.co/badges/76916047-5b66-466d-91d3-7131a269899a)](https://codebeat.co/projects/github-com-fredwu-opq-master)\n[![Coverage](https://img.shields.io/coveralls/fredwu/opq.svg)](https://coveralls.io/github/fredwu/opq?branch=master)\n[![Hex Version](https://img.shields.io/hexpm/v/opq.svg)](https://hex.pm/packages/opq)\n[![Hex Docs](https://img.shields.io/badge/hex-docs-lightgreen.svg)](https://hexdocs.pm/opq/)\n[![Total Download](https://img.shields.io/hexpm/dt/opq.svg)](https://hex.pm/packages/opq)\n[![License](https://img.shields.io/hexpm/l/opq.svg)](https://github.com/fredwu/opq/blob/master/LICENSE.md)\n[![Last Updated](https://img.shields.io/github/last-commit/fredwu/opq.svg)](https://github.com/fredwu/opq/commits/master)\n\n## Elixir Queue!\n\nA simple, in-memory queue with worker pooling and rate limiting in Elixir. OPQ leverages Erlang's [queue](http://erlang.org/doc/man/queue.html) module and Elixir's [GenStage](https://github.com/elixir-lang/gen_stage).\n\nOriginally built to support [Crawler](https://github.com/fredwu/crawler).\n\n## Features\n\n- A fast, in-memory FIFO queue.\n- Worker pool.\n- Rate limit.\n- Timeouts.\n- Pause / resume / stop the queue.\n\nSee [Hex documentation](https://hexdocs.pm/opq/).\n\n## Installation\n\n```Elixir\ndef deps do\n [{:opq, \"~> 4.0\"}]\nend\n```\n\n## Usage\n\n### A simple example:\n\n```elixir\n{:ok, opq} = OPQ.init()\n\nOPQ.enqueue(opq, fn -> IO.inspect(\"hello\") end)\nOPQ.enqueue(opq, fn -> IO.inspect(\"world\") end)\n```\n\n### Specify module, function and arguments:\n\n```elixir\n{:ok, opq} = OPQ.init()\n\nOPQ.enqueue(opq, IO, :inspect, [\"hello\"])\nOPQ.enqueue(opq, IO, :inspect, [\"world\"])\n```\n\n### Specify a custom name for the queue:\n\n```elixir\nOPQ.init(name: :items)\n\nOPQ.enqueue(:items, fn -> IO.inspect(\"hello\") end)\nOPQ.enqueue(:items, fn -> IO.inspect(\"world\") end)\n```\n\n### Start as part of a supervision tree:\n\nNote, when starting as part of a supervision tree, the `:name` option must be provided.\n\n```elixir\nchildren = [\n {OPQ, name: :items}\n]\n```\n\n### Specify a custom worker to process items in the queue:\n\n```elixir\ndefmodule CustomWorker do\n def start_link(item) do\n Task.start_link(fn ->\n Agent.update(:bucket, &[item | &1])\n end)\n end\nend\n\nAgent.start_link(fn -> [] end, name: :bucket)\n\n{:ok, opq} = OPQ.init(worker: CustomWorker)\n\nOPQ.enqueue(opq, \"hello\")\nOPQ.enqueue(opq, \"world\")\n\nAgent.get(:bucket, & &1) # => [\"world\", \"hello\"]\n```\n\n### Rate limit:\n\n```elixir\n{:ok, opq} = OPQ.init(workers: 1, interval: 1000)\n\nTask.async(fn ->\n OPQ.enqueue(opq, fn -> IO.inspect(\"hello\") end)\n OPQ.enqueue(opq, fn -> IO.inspect(\"world\") end)\nend)\n```\n\nIf no interval is supplied, the ratelimiter will be bypassed.\n\n### Check the queue and number of available workers:\n\n```elixir\n{:ok, opq} = OPQ.init()\n\nOPQ.enqueue(opq, fn -> Process.sleep(1000) end)\n\n{status, queue, available_workers} = OPQ.info(opq) # => {:normal, #OPQ.Queue<[]>, 9}\n\nProcess.sleep(1200)\n\n{status, queue, available_workers} = OPQ.info(opq) # => {:normal, #OPQ.Queue<[]>, 10}\n```\n\nIf you just need to get the queue itself:\n\n```elixir\nOPQ.queue(opq) # => #OPQ.Queue<[]>\n```\n\n### Queue\n\nOPQ implements `Enumerable`, so you can perform enumerable functions on the queue:\n\n```elixir\n{:ok, opq} = OPQ.init()\n\nqueue = OPQ.queue(opq)\n\nEnum.count(queue) # => 0\nEnum.empty?(queue) # => true\n```\n\n### Stop the queue:\n\n```elixir\n{:ok, opq} = OPQ.init()\n\nOPQ.enqueue(opq, fn -> IO.inspect(\"hello\") end)\nOPQ.stop(opq)\nOPQ.enqueue(opq, fn -> IO.inspect(\"world\") end) # => (EXIT) no process...\n```\n\n### Pause and resume the queue:\n\n```elixir\n{:ok, opq} = OPQ.init()\n\nOPQ.enqueue(opq, fn -> IO.inspect(\"hello\") end) # => \"hello\"\nOPQ.pause(opq)\nOPQ.info(opq) # => {:paused, {[], []}, 10}\nOPQ.enqueue(opq, fn -> IO.inspect(\"world\") end)\nOPQ.resume(opq) # => \"world\"\nOPQ.info(opq) # => {:normal, {[], []}, 10}\n```\n\n## Configurations\n\n| Option | Type | Default Value | Description |\n|--------------|-------------|----------------|-------------|\n| `:name` | atom/module | pid | The name of the queue.\n| `:worker` | module | `OPQ.Worker` | The worker that processes each item from the queue.\n| `:workers` | integer | `10` | Maximum number of workers.\n| `:interval` | integer | `0` | Rate limit control - number of milliseconds before asking for more items to process, defaults to `0` which is effectively no rate limit.\n| `:timeout` | integer | `5000` | Number of milliseconds allowed to perform the work, it should always be set to higher than `:interval`.\n\n## Changelog\n\nPlease see [CHANGELOG.md](CHANGELOG.md).\n\n## License\n\nLicensed under [MIT](http://fredwu.mit-license.org/).\n" }, { "path": "config/config.exs", "content": "import Config\n" }, { "path": "lib/opq/feeder.ex", "content": "defmodule OPQ.Feeder do\n @moduledoc \"\"\"\n A GenStage producer that feeds items in a buffered queue to the consumers.\n \"\"\"\n\n use GenStage\n\n def start_link(nil), do: GenStage.start_link(__MODULE__, :ok)\n def start_link(name), do: GenStage.start_link(__MODULE__, :ok, name: name)\n\n def init(:ok) do\n {:producer, {:normal, %OPQ.Queue{}, 0}}\n end\n\n def handle_cast(:stop, state) do\n {:stop, :shutdown, state}\n end\n\n def handle_cast(:pause, {_status, queue, demand}) do\n dispatch_or_pause(:paused, queue, demand)\n end\n\n def handle_cast(:resume, {_status, queue, demand}) do\n dispatch_events(:normal, queue, demand, [])\n end\n\n def handle_cast({:enqueue, event}, {status, %OPQ.Queue{data: data}, pending_demand}) do\n data = :queue.in(event, data)\n\n dispatch_or_pause(status, %OPQ.Queue{data: data}, pending_demand)\n end\n\n def handle_call(:info, _from, state) do\n {:reply, state, [], state}\n end\n\n def handle_call(:queue, _from, {_status, queue, _demand} = state) do\n {:reply, queue, [], state}\n end\n\n defp dispatch_or_pause(:normal, queue, demand) do\n dispatch_events(:normal, queue, demand, [])\n end\n\n defp dispatch_or_pause(:paused, queue, demand) do\n {:noreply, [], {:paused, queue, demand}}\n end\n\n def handle_demand(demand, {status, queue, pending_demand}) do\n dispatch_events(status, queue, demand + pending_demand, [])\n end\n\n defp dispatch_events(:paused, queue, demand, events) do\n {:noreply, Enum.reverse(events), {:paused, queue, demand}}\n end\n\n defp dispatch_events(status, queue, 0, events) do\n {:noreply, Enum.reverse(events), {status, queue, 0}}\n end\n\n defp dispatch_events(status, %OPQ.Queue{data: data}, demand, events) do\n case :queue.out(data) do\n {{:value, event}, data} ->\n dispatch_events(status, %OPQ.Queue{data: data}, demand - 1, [event | events])\n\n {:empty, data} ->\n {:noreply, Enum.reverse(events), {status, %OPQ.Queue{data: data}, demand}}\n end\n end\nend\n" }, { "path": "lib/opq/options.ex", "content": "defmodule OPQ.Options do\n @moduledoc \"\"\"\n Options for configuring OPQ.\n \"\"\"\n\n @worker OPQ.Worker\n @workers 10\n @interval 0\n @timeout 5_000\n\n @doc \"\"\"\n ## Examples\n\n iex> Options.assign_defaults([]) |> Keyword.get(:workers)\n 10\n\n iex> Options.assign_defaults([workers: 4]) |> Keyword.get(:workers)\n 4\n \"\"\"\n def assign_defaults(opts) do\n Keyword.merge(\n [\n worker: worker(),\n workers: workers(),\n interval: interval(),\n timeout: timeout()\n ],\n opts\n )\n end\n\n defp worker(), do: Application.get_env(:opq, :worker, @worker)\n defp workers(), do: Application.get_env(:opq, :workers, @workers)\n defp interval(), do: Application.get_env(:opq, :interval, @interval)\n defp timeout(), do: Application.get_env(:opq, :timeout, @timeout)\nend\n" }, { "path": "lib/opq/options_handler.ex", "content": "defmodule OPQ.OptionsHandler do\n @moduledoc \"\"\"\n Saves and loads options to pass around.\n \"\"\"\n\n def save_opts(feeder, opts) do\n Agent.start_link(fn -> opts end, name: name(feeder))\n end\n\n def timeout(feeder), do: load_opts(feeder)[:timeout]\n\n def stop(feeder), do: Agent.stop(name(feeder))\n\n defp load_opts(feeder), do: Agent.get(name(feeder), & &1)\n defp name(feeder), do: :\"opq-#{Kernel.inspect(feeder)}\"\nend\n" }, { "path": "lib/opq/queue/enumerable.ex", "content": "defimpl Enumerable, for: OPQ.Queue do\n @moduledoc \"\"\"\n Implementation based on https://github.com/princemaple/elixir-queue\n \"\"\"\n\n def count(%OPQ.Queue{data: q}), do: {:ok, :queue.len(q)}\n\n def member?(%OPQ.Queue{data: q}, item) do\n {:ok, :queue.member(item, q)}\n end\n\n def reduce(%OPQ.Queue{data: q}, acc, fun) do\n Enumerable.List.reduce(:queue.to_list(q), acc, fun)\n end\n\n def slice(%OPQ.Queue{}), do: {:error, __MODULE__}\nend\n" }, { "path": "lib/opq/queue/inspect.ex", "content": "defimpl Inspect, for: OPQ.Queue do\n @moduledoc \"\"\"\n Implementation based on https://github.com/princemaple/elixir-queue\n \"\"\"\n\n import Inspect.Algebra\n\n def inspect(%OPQ.Queue{} = q, opts) do\n concat([\"#OPQ.Queue<\", to_doc(Enum.to_list(q), opts), \">\"])\n end\nend\n" }, { "path": "lib/opq/queue.ex", "content": "defmodule OPQ.Queue do\n @moduledoc \"\"\"\n A `:queue` wrapper so that protocols like `Enumerable` can be implemented.\n \"\"\"\n\n @opaque t() :: %__MODULE__{data: :queue.queue()}\n\n defstruct data: :queue.new()\nend\n" }, { "path": "lib/opq/rate_limiter.ex", "content": "defmodule OPQ.RateLimiter do\n @moduledoc \"\"\"\n Provides rate limit.\n \"\"\"\n\n use GenStage\n\n def start_link(opts) do\n GenStage.start_link(__MODULE__, opts)\n end\n\n def init(opts) do\n {:producer_consumer, {}, subscribe_to: [{opts[:name], opts}]}\n end\n\n def handle_subscribe(:producer, opts, from, _state) do\n {:manual, ask_and_schedule(from, {opts[:workers], opts[:interval]})}\n end\n\n def handle_subscribe(:consumer, _opts, _from, state) do\n {:automatic, state}\n end\n\n def handle_events(events, _from, {pending, interval}) do\n {:noreply, events, {pending + length(events), interval}}\n end\n\n def handle_info({:ask, from}, state) do\n {:noreply, [], ask_and_schedule(from, state)}\n end\n\n defp ask_and_schedule(from, {pending, interval}) do\n GenStage.ask(from, pending)\n\n Process.send_after(self(), {:ask, from}, interval)\n\n {0, interval}\n end\nend\n" }, { "path": "lib/opq/worker.ex", "content": "defmodule OPQ.Worker do\n @moduledoc \"\"\"\n A default worker that simply executes an item if it's a function.\n \"\"\"\n\n def start_link(item) do\n Task.start_link(fn ->\n Process.flag(:trap_exit, true)\n process_item(item)\n end)\n end\n\n defp process_item({mod, fun, args}), do: apply(mod, fun, args)\n defp process_item(item) when is_function(item), do: item.()\n defp process_item(item), do: item\nend\n" }, { "path": "lib/opq/worker_supervisor.ex", "content": "defmodule OPQ.WorkerSupervisor do\n @moduledoc \"\"\"\n A supervisor that subscribes to `Feeder` and spins up the worker pool.\n \"\"\"\n\n use ConsumerSupervisor\n\n def start_link(opts) do\n ConsumerSupervisor.start_link(__MODULE__, opts)\n end\n\n def init(opts) do\n children = [\n %{id: opts[:worker], start: {opts[:worker], :start_link, []}, restart: :temporary}\n ]\n\n cs_opts = [\n strategy: :one_for_one,\n subscribe_to: [\n {\n opts[:producer_consumer],\n min_demand: 0, max_demand: opts[:workers], timeout: opts[:timeout]\n }\n ]\n ]\n\n ConsumerSupervisor.init(children, cs_opts)\n end\nend\n" }, { "path": "lib/opq.ex", "content": "defmodule OPQ do\n @moduledoc \"\"\"\n A simple, in-memory queue with worker pooling and rate limiting in Elixir.\n \"\"\"\n\n use GenServer\n\n alias OPQ.Feeder\n alias OPQ.Options\n alias OPQ.OptionsHandler, as: Opt\n alias OPQ.RateLimiter\n alias OPQ.WorkerSupervisor\n\n def start_link(opts \\\\ []), do: init(opts)\n\n def init(opts \\\\ []) do\n opts\n |> Options.assign_defaults()\n |> start_links()\n end\n\n def enqueue(feeder, event) do\n GenStage.cast(feeder, {:enqueue, event})\n end\n\n def enqueue(feeder, mod, fun, args)\n when is_atom(mod) and\n is_atom(fun) and\n is_list(args) do\n enqueue(feeder, {mod, fun, args})\n end\n\n def stop(feeder) do\n Process.flag(:trap_exit, true)\n GenStage.cast(feeder, :stop)\n Opt.stop(feeder)\n end\n\n def pause(feeder), do: GenStage.cast(feeder, :pause)\n def resume(feeder), do: GenStage.cast(feeder, :resume)\n def info(feeder), do: GenStage.call(feeder, :info, Opt.timeout(feeder))\n def queue(feeder), do: GenStage.call(feeder, :queue, Opt.timeout(feeder))\n\n defp start_links(opts) do\n {:ok, feeder} = Feeder.start_link(opts[:name])\n\n Opt.save_opts(opts[:name] || feeder, opts)\n\n opts\n |> Keyword.merge(name: feeder)\n |> start_consumers(interval: opts[:interval])\n\n {:ok, feeder}\n end\n\n defp start_consumers(opts, interval: 0) do\n opts\n |> Keyword.merge(producer_consumer: opts[:name])\n |> WorkerSupervisor.start_link()\n end\n\n defp start_consumers(opts, _) do\n {:ok, rate_limiter} = RateLimiter.start_link(opts)\n\n opts\n |> Keyword.merge(producer_consumer: rate_limiter)\n |> WorkerSupervisor.start_link()\n end\nend\n" }, { "path": "mix.exs", "content": "defmodule OPQ.Mixfile do\n use Mix.Project\n\n @source_url \"https://github.com/fredwu/opq\"\n @version \"4.0.4\"\n\n def project do\n [\n app: :opq,\n version: @version,\n elixir: \"~> 1.13\",\n elixirc_paths: elixirc_paths(Mix.env()),\n package: package(),\n name: \"OPQ: One Pooled Queue\",\n description: \"A simple, in-memory queue with worker pooling and rate limiting in Elixir.\",\n start_permanent: Mix.env() == :prod,\n deps: deps(),\n docs: docs(),\n test_coverage: [tool: ExCoveralls],\n preferred_cli_env: [coveralls: :test],\n aliases: [publish: [\"hex.publish\", &git_tag/1]]\n ]\n end\n\n def application do\n [\n extra_applications: [:logger]\n ]\n end\n\n defp elixirc_paths(:test), do: [\"lib\", \"test/support\"]\n defp elixirc_paths(_), do: [\"lib\"]\n\n defp deps do\n [\n {:gen_stage, \"~> 1.1\"},\n {:recode, \"~> 0.6\", only: :dev},\n {:ex_doc, \">= 0.0.0\", only: :dev},\n {:excoveralls, \"~> 0.14\", only: :test}\n ]\n end\n\n defp package do\n [\n maintainers: [\"Fred Wu\"],\n licenses: [\"MIT\"],\n links: %{\"GitHub\" => @source_url}\n ]\n end\n\n defp git_tag(_args) do\n System.cmd(\"git\", [\"tag\", \"v\" <> Mix.Project.config()[:version]])\n System.cmd(\"git\", [\"push\"])\n System.cmd(\"git\", [\"push\", \"--tags\"])\n end\n\n defp docs do\n [\n extras: [\"CHANGELOG.md\": [title: \"Changelog\"], \"README.md\": [title: \"Overview\"]],\n main: \"readme\",\n source_url: @source_url,\n source_ref: \"v#{@version}\",\n formatters: [\"html\"]\n ]\n end\nend\n" }, { "path": "test/lib/opq/feeder_test.exs", "content": "defmodule OPQ.FeederTest do\n use OPQ.TestCase, async: true\n\n alias OPQ.Feeder\n\n doctest Feeder\nend\n" }, { "path": "test/lib/opq/options_test.exs", "content": "defmodule OPQ.OptionsTest do\n use OPQ.TestCase, async: true\n\n alias OPQ.Options\n\n doctest Options\nend\n" }, { "path": "test/lib/opq/worker_supervisor_test.exs", "content": "defmodule OPQ.WorkerSupervisorTest do\n use OPQ.TestCase, async: true\n\n alias OPQ.WorkerSupervisor\n\n doctest WorkerSupervisor\nend\n" }, { "path": "test/lib/opq/worker_test.exs", "content": "defmodule OPQ.WorkerTest do\n use OPQ.TestCase, async: true\n\n alias OPQ.Worker\n\n doctest Worker\nend\n" }, { "path": "test/lib/opq_test.exs", "content": "defmodule OPQTest do\n use OPQ.TestCase, async: true\n\n doctest OPQ\n\n @moduletag capture_log: true\n\n test \"enqueue items - child_spec/1\" do\n Supervisor.start_link([{OPQ, name: :opq}], strategy: :one_for_one)\n\n OPQ.enqueue(:opq, :a)\n OPQ.enqueue(:opq, :b)\n\n wait(fn ->\n assert_empty_queue(:opq)\n end)\n end\n\n test \"enqueue items - start_link/1\" do\n {:ok, opq} = OPQ.start_link()\n\n OPQ.enqueue(opq, :a)\n OPQ.enqueue(opq, :b)\n\n wait(fn ->\n assert_empty_queue(opq)\n end)\n end\n\n test \"enqueue items - init/1\" do\n {:ok, opq} = OPQ.init()\n\n OPQ.enqueue(opq, :a)\n OPQ.enqueue(opq, :b)\n\n wait(fn ->\n assert_empty_queue(opq)\n end)\n end\n\n test \"enqueue functions\" do\n Agent.start_link(fn -> [] end, name: Bucket)\n\n {:ok, opq} = OPQ.init()\n\n OPQ.enqueue(opq, fn -> Agent.update(Bucket, &[:a | &1]) end)\n OPQ.enqueue(opq, fn -> Agent.update(Bucket, &[:b | &1]) end)\n\n wait(fn ->\n assert_empty_queue(opq)\n\n assert Kernel.length(Agent.get(Bucket, & &1)) == 2\n end)\n end\n\n test \"enqueue MFAs\" do\n Agent.start_link(fn -> [] end, name: MfaBucket)\n\n {:ok, opq} = OPQ.init()\n\n OPQ.enqueue(opq, Agent, :update, [MfaBucket, &[:a | &1]])\n OPQ.enqueue(opq, Agent, :update, [MfaBucket, &[:b | &1]])\n\n wait(fn ->\n assert_empty_queue(opq)\n\n assert Kernel.length(Agent.get(MfaBucket, & &1)) == 2\n end)\n end\n\n test \"enqueue to a named queue\" do\n OPQ.init(name: :items)\n\n OPQ.enqueue(:items, :a)\n OPQ.enqueue(:items, :b)\n\n wait(fn ->\n assert_empty_queue(:items)\n end)\n end\n\n test \"run out of demands from the workers\" do\n {:ok, opq} = OPQ.init(workers: 2)\n\n OPQ.enqueue(opq, :a)\n OPQ.enqueue(opq, :b)\n\n wait(fn ->\n assert_empty_queue(opq)\n end)\n end\n\n test \"single worker\" do\n {:ok, opq} = OPQ.init(workers: 1)\n\n OPQ.enqueue(opq, :a)\n OPQ.enqueue(opq, :b)\n\n wait(fn ->\n assert_empty_queue(opq)\n end)\n end\n\n test \"custom worker\" do\n defmodule CustomWorker do\n def start_link(item) do\n Task.start_link(fn ->\n Agent.update(CustomWorkerBucket, &[item | &1])\n end)\n end\n end\n\n Agent.start_link(fn -> [] end, name: CustomWorkerBucket)\n\n {:ok, opq} = OPQ.init(worker: CustomWorker)\n\n OPQ.enqueue(opq, :a)\n OPQ.enqueue(opq, :b)\n\n wait(fn ->\n assert Kernel.length(Agent.get(CustomWorkerBucket, & &1)) == 2\n end)\n end\n\n test \"rate limit\" do\n Agent.start_link(fn -> [] end, name: RateLimitBucket)\n\n {:ok, opq} = OPQ.init(workers: 1, interval: 10)\n\n Task.async(fn ->\n OPQ.enqueue(opq, fn -> Agent.update(RateLimitBucket, &[:a | &1]) end)\n OPQ.enqueue(opq, fn -> Agent.update(RateLimitBucket, &[:b | &1]) end)\n end)\n\n Process.sleep(5)\n\n assert Kernel.length(Agent.get(RateLimitBucket, & &1)) == 1\n\n wait(fn ->\n assert Kernel.length(Agent.get(RateLimitBucket, & &1)) == 2\n end)\n end\n\n test \"stop\" do\n {:ok, opq} = OPQ.init(workers: 1)\n\n OPQ.enqueue(opq, :a)\n\n OPQ.stop(opq)\n\n refute Process.alive?(opq)\n\n agent = :\"opq-#{Kernel.inspect(opq)}\"\n\n assert catch_exit(Agent.get(agent, & &1))\n end\n\n test \"pause & resume\" do\n Agent.start_link(fn -> [] end, name: PauseBucket)\n\n {:ok, opq} = OPQ.init(workers: 1)\n\n OPQ.enqueue(opq, fn -> Agent.update(PauseBucket, &[:a | &1]) end)\n\n OPQ.pause(opq)\n\n OPQ.enqueue(opq, fn -> Agent.update(PauseBucket, &[:b | &1]) end)\n OPQ.enqueue(opq, fn -> Agent.update(PauseBucket, &[:c | &1]) end)\n\n wait(fn ->\n {status, _queue, _demand} = OPQ.info(opq)\n\n assert status == :paused\n assert Kernel.length(Agent.get(PauseBucket, & &1)) == 1\n end)\n\n OPQ.resume(opq)\n\n wait(fn ->\n {status, _queue, _demand} = OPQ.info(opq)\n\n assert status == :normal\n assert Kernel.length(Agent.get(PauseBucket, & &1)) == 3\n end)\n end\n\n test \"graceful handle of exit signals\" do\n Process.flag(:trap_exit, true)\n\n Agent.start_link(fn -> [] end, name: ExitBucket)\n\n {:ok, opq} = OPQ.init(workers: 1)\n\n OPQ.enqueue(opq, fn ->\n Process.sleep(10)\n Agent.update(ExitBucket, &[:a | &1])\n end)\n\n Process.sleep(1)\n\n Process.exit(opq, :SIGTERM)\n\n refute Process.alive?(opq)\n\n wait(fn ->\n assert Kernel.length(Agent.get(ExitBucket, & &1)) == 1\n end)\n end\n\n defp assert_empty_queue(queue_name) do\n assert queue_name\n |> OPQ.queue()\n |> Enum.empty?()\n end\nend\n" }, { "path": "test/support/test_case.ex", "content": "defmodule OPQ.TestCase do\n use ExUnit.CaseTemplate\n\n using do\n quote do\n import OPQ.TestHelpers\n end\n end\nend\n" }, { "path": "test/support/test_helpers.ex", "content": "# Credit: https://gist.github.com/cblavier/5e15791387a6e22b98d8\ndefmodule OPQ.TestHelpers do\n def wait(fun), do: wait(500, fun)\n def wait(0, fun), do: fun.()\n\n def wait(timeout, fun) do\n try do\n fun.()\n rescue\n _ ->\n :timer.sleep(10)\n wait(max(0, timeout - 10), fun)\n end\n end\nend\n" }, { "path": "test/test_helper.exs", "content": "ExUnit.start()\n" } ]