Showing preview only (1,213K chars total). Download the full file or copy to clipboard to get everything.
Repository: hibiken/asynq
Branch: master
Commit: d704b68a426d
Files: 101
Total size: 1.1 MB
Directory structure:
gitextract_6qfa6b_y/
├── .github/
│ ├── FUNDING.yml
│ ├── ISSUE_TEMPLATE/
│ │ ├── bug_report.md
│ │ └── feature_request.md
│ ├── dependabot.yaml
│ └── workflows/
│ ├── benchstat.yml
│ └── build.yml
├── .gitignore
├── CHANGELOG.md
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── LICENSE
├── Makefile
├── README.md
├── aggregator.go
├── aggregator_test.go
├── asynq.go
├── asynq_test.go
├── benchmark_test.go
├── client.go
├── client_test.go
├── context.go
├── doc.go
├── example_test.go
├── forwarder.go
├── forwarder_test.go
├── go.mod
├── go.sum
├── healthcheck.go
├── healthcheck_test.go
├── heartbeat.go
├── heartbeat_test.go
├── inspector.go
├── inspector_test.go
├── internal/
│ ├── base/
│ │ ├── base.go
│ │ └── base_test.go
│ ├── context/
│ │ ├── context.go
│ │ └── context_test.go
│ ├── errors/
│ │ ├── errors.go
│ │ └── errors_test.go
│ ├── log/
│ │ ├── log.go
│ │ └── log_test.go
│ ├── proto/
│ │ ├── asynq.pb.go
│ │ └── asynq.proto
│ ├── rdb/
│ │ ├── benchmark_test.go
│ │ ├── inspect.go
│ │ ├── inspect_test.go
│ │ ├── rdb.go
│ │ └── rdb_test.go
│ ├── testbroker/
│ │ └── testbroker.go
│ ├── testutil/
│ │ ├── builder.go
│ │ ├── builder_test.go
│ │ └── testutil.go
│ └── timeutil/
│ ├── timeutil.go
│ └── timeutil_test.go
├── janitor.go
├── janitor_test.go
├── periodic_task_manager.go
├── periodic_task_manager_test.go
├── processor.go
├── processor_test.go
├── recoverer.go
├── recoverer_test.go
├── scheduler.go
├── scheduler_test.go
├── servemux.go
├── servemux_test.go
├── server.go
├── server_test.go
├── signals_unix.go
├── signals_windows.go
├── subscriber.go
├── subscriber_test.go
├── syncer.go
├── syncer_test.go
├── tools/
│ ├── asynq/
│ │ ├── README.md
│ │ ├── cmd/
│ │ │ ├── cron.go
│ │ │ ├── dash/
│ │ │ │ ├── dash.go
│ │ │ │ ├── draw.go
│ │ │ │ ├── draw_test.go
│ │ │ │ ├── fetch.go
│ │ │ │ ├── key_event.go
│ │ │ │ ├── key_event_test.go
│ │ │ │ ├── screen_drawer.go
│ │ │ │ └── table.go
│ │ │ ├── dash.go
│ │ │ ├── group.go
│ │ │ ├── queue.go
│ │ │ ├── root.go
│ │ │ ├── server.go
│ │ │ ├── stats.go
│ │ │ └── task.go
│ │ └── main.go
│ ├── go.mod
│ ├── go.sum
│ └── metrics_exporter/
│ └── main.go
└── x/
├── go.mod
├── go.sum
├── metrics/
│ └── metrics.go
└── rate/
├── example_test.go
├── semaphore.go
└── semaphore_test.go
================================================
FILE CONTENTS
================================================
================================================
FILE: .github/FUNDING.yml
================================================
# These are supported funding model platforms
github: [hibiken]
open_collective: ken-hibino
================================================
FILE: .github/ISSUE_TEMPLATE/bug_report.md
================================================
---
name: Bug report
about: Create a report to help us improve
title: "[BUG] Description of the bug"
labels: bug
assignees:
- hibiken
- kamikazechaser
---
**Describe the bug**
A clear and concise description of what the bug is.
**Environment (please complete the following information):**
- OS: [e.g. MacOS, Linux]
- `asynq` package version [e.g. v0.25.0]
- Redis/Valkey version
**To Reproduce**
Steps to reproduce the behavior (Code snippets if applicable):
1. Setup background processing ...
2. Enqueue tasks ...
3. See Error ...
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Additional context**
Add any other context about the problem here.
================================================
FILE: .github/ISSUE_TEMPLATE/feature_request.md
================================================
---
name: Feature request
about: Suggest an idea for this project
title: "[FEATURE REQUEST] Description of the feature request"
labels: enhancement
assignees:
- hibiken
- kamikazechaser
---
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
================================================
FILE: .github/dependabot.yaml
================================================
version: 2
updates:
- package-ecosystem: "gomod"
directory: "/"
schedule:
interval: "weekly"
labels:
- "pr-deps"
- package-ecosystem: "gomod"
directory: "/tools"
schedule:
interval: "weekly"
labels:
- "pr-deps"
- package-ecosystem: "gomod"
directory: "/x"
schedule:
interval: "weekly"
labels:
- "pr-deps"
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
================================================
FILE: .github/workflows/benchstat.yml
================================================
# This workflow runs benchmarks against the current branch,
# compares them to benchmarks against master,
# and uploads the results as an artifact.
name: benchstat
on: [pull_request]
jobs:
incoming:
runs-on: ubuntu-latest
if: false
services:
redis:
image: redis:7
ports:
- 6379:6379
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: 1.23.x
- name: Benchmark
run: go test -run=^$ -bench=. -count=5 -timeout=60m ./... | tee -a new.txt
- name: Upload Benchmark
uses: actions/upload-artifact@v4
with:
name: bench-incoming
path: new.txt
current:
runs-on: ubuntu-latest
if: false
services:
redis:
image: redis:7
ports:
- 6379:6379
steps:
- name: Checkout
uses: actions/checkout@v4
with:
ref: master
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: 1.23.x
- name: Benchmark
run: go test -run=^$ -bench=. -count=5 -timeout=60m ./... | tee -a old.txt
- name: Upload Benchmark
uses: actions/upload-artifact@v4
with:
name: bench-current
path: old.txt
benchstat:
needs: [incoming, current]
if: false
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: 1.23.x
- name: Install benchstat
run: go get -u golang.org/x/perf/cmd/benchstat
- name: Download Incoming
uses: actions/download-artifact@v4
with:
name: bench-incoming
- name: Download Current
uses: actions/download-artifact@v4
with:
name: bench-current
- name: Benchstat Results
run: benchstat old.txt new.txt | tee -a benchstat.txt
- name: Upload benchstat results
uses: actions/upload-artifact@v4
with:
name: benchstat
path: benchstat.txt
================================================
FILE: .github/workflows/build.yml
================================================
name: build
on: [push, pull_request]
jobs:
build:
strategy:
matrix:
os: [ubuntu-latest]
go-version: [1.24.x, 1.25.x]
runs-on: ${{ matrix.os }}
services:
redis:
image: redis:7
ports:
- 6379:6379
steps:
- uses: actions/checkout@v4
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
cache: false
- name: Build core module
run: go build -v ./...
- name: Build x module
run: cd x && go build -v ./... && cd ..
- name: Test core module
run: go test -race -v -coverprofile=coverage.txt -covermode=atomic ./...
- name: Test x module
run: cd x && go test -race -v ./... && cd ..
- name: Benchmark Test
run: go test -run=^$ -bench=. -loglevel=debug ./...
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v5
build-tool:
strategy:
matrix:
os: [ubuntu-latest]
go-version: [1.24.x, 1.25.x]
runs-on: ${{ matrix.os }}
services:
redis:
image: redis:7
ports:
- 6379:6379
steps:
- uses: actions/checkout@v4
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
cache: false
- name: Build tools module
run: cd tools && go build -v ./... && cd ..
- name: Test tools module
run: cd tools && go test -race -v ./... && cd ..
golangci:
name: lint
runs-on: ubuntu-latest
if: false
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: stable
- name: golangci-lint
uses: golangci/golangci-lint-action@v6
with:
version: v1.61
================================================
FILE: .gitignore
================================================
vendor
# Binaries for programs and plugins
*.exe
*.exe~
*.dll
*.so
*.dylib
# Test binary, built with `go test -c`
*.test
# Output of the go coverage tool, specifically when used with LiteIDE
*.out
# Ignore examples for now
/examples
# Ignore tool binaries
/tools/asynq/asynq
/tools/metrics_exporter/metrics_exporter
# Ignore asynq config file
.asynq.*
# Ignore editor config files
.vscode
.idea
================================================
FILE: CHANGELOG.md
================================================
# Changelog
All notable changes to this project will be documented in this file.
The format is based on ["Keep a Changelog"](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
## [0.26.0] - 2026-02-03
### Upgrades
- Prepare CI for Go 1.24.x and 1.25.x (commit: e9037f0)
### Added
- Add Headers support to tasks (PR: https://github.com/hibiken/asynq/pull/1070)
- Add `--tls` option to dash command (PR: https://github.com/hibiken/asynq/pull/1073)
- Add `--username` CLI flag for Redis ACL authentication (PR: https://github.com/hibiken/asynq/pull/1083)
- Add `UpdateTaskPayload` method for inspector (PR: https://github.com/hibiken/asynq/pull/1042)
### Fixes
- Fix: Correct error message text in ResultWriter.Write (PR: https://github.com/hibiken/asynq/pull/1054)
- Fix: Wrap all fmt.Errorf errors with %w (PR: https://github.com/hibiken/asynq/pull/1047)
- Fix: ServeMux.NotFoundHandler returns ErrHandlerNotFound error (PR: https://github.com/hibiken/asynq/pull/1031)
### Changed
- Docs: Update server.go documentation (PR: https://github.com/hibiken/asynq/pull/1010)
- Chore: Fix godoc comment (PR: https://github.com/hibiken/asynq/pull/1009)
## [0.25.1] - 2024-12-11
### Upgrades
* Some packages
### Added
* Add `HeartbeatInterval` option to the scheduler (PR: https://github.com/hibiken/asynq/pull/956)
* Add `RedisUniversalClient` support to periodic task manager (PR: https://github.com/hibiken/asynq/pull/958)
* Add `--insecure` flag to CLI dash command (PR: https://github.com/hibiken/asynq/pull/980)
* Add logging for registration errors (PR: https://github.com/hibiken/asynq/pull/657)
### Fixes
- Perf: Use string concat inplace of fmt.Sprintf in hotpath (PR: https://github.com/hibiken/asynq/pull/962)
- Perf: Init map with size (PR: https://github.com/hibiken/asynq/pull/673)
- Fix: `Scheduler` and `PeriodicTaskManager` graceful shutdown (PR: https://github.com/hibiken/asynq/pull/977)
- Fix: `Server` graceful shutdown on UNIX systems (PR: https://github.com/hibiken/asynq/pull/982)
## [0.25.0] - 2024-10-29
### Upgrades
- Minumum go version is set to 1.22 (PR: https://github.com/hibiken/asynq/pull/925)
- Internal protobuf package is upgraded to address security advisories (PR: https://github.com/hibiken/asynq/pull/925)
- Most packages are upgraded
- CI/CD spec upgraded
### Added
- `IsPanicError` function is introduced to support catching of panic errors when processing tasks (PR: https://github.com/hibiken/asynq/pull/491)
- `JanitorInterval` and `JanitorBatchSize` are added as Server options (PR: https://github.com/hibiken/asynq/pull/715)
- `NewClientFromRedisClient` is introduced to allow reusing an existing redis client (PR: https://github.com/hibiken/asynq/pull/742)
- `TaskCheckInterval` config option is added to specify the interval between checks for new tasks to process when all queues are empty (PR: https://github.com/hibiken/asynq/pull/694)
- `Ping` method is added to Client, Server and Scheduler ((PR: https://github.com/hibiken/asynq/pull/585))
- `RevokeTask` error type is introduced to prevent a task from being retried or archived (PR: https://github.com/hibiken/asynq/pull/882)
- `SentinelUsername` is added as a redis config option (PR: https://github.com/hibiken/asynq/pull/924)
- Some jitter is introduced to improve latency when fetching jobs in the processor (PR: https://github.com/hibiken/asynq/pull/868)
- Add task enqueue command to the CLI (PR: https://github.com/hibiken/asynq/pull/918)
- Add a map cache (concurrent safe) to keep track of queues that ultimately reduces redis load when enqueuing tasks (PR: https://github.com/hibiken/asynq/pull/946)
### Fixes
- Archived tasks that are trimmed should now be deleted (PR: https://github.com/hibiken/asynq/pull/743)
- Fix lua script when listing task messages with an expired lease (PR: https://github.com/hibiken/asynq/pull/709)
- Fix potential context leaks due to cancellation not being called (PR: https://github.com/hibiken/asynq/pull/926)
- Misc documentation fixes
- Misc test fixes
## [0.24.1] - 2023-05-01
### Changed
- Updated package version dependency for go-redis
## [0.24.0] - 2023-01-02
### Added
- `PreEnqueueFunc`, `PostEnqueueFunc` is added in `Scheduler` and deprecated `EnqueueErrorHandler` (PR: https://github.com/hibiken/asynq/pull/476)
### Changed
- Removed error log when `Scheduler` failed to enqueue a task. Use `PostEnqueueFunc` to check for errors and task actions if needed.
- Changed log level from ERROR to WARNINING when `Scheduler` failed to record `SchedulerEnqueueEvent`.
## [0.23.0] - 2022-04-11
### Added
- `Group` option is introduced to enqueue task in a group.
- `GroupAggregator` and related types are introduced for task aggregation feature.
- `GroupGracePeriod`, `GroupMaxSize`, `GroupMaxDelay`, and `GroupAggregator` fields are added to `Config`.
- `Inspector` has new methods related to "aggregating tasks".
- `Group` field is added to `TaskInfo`.
- (CLI): `group ls` command is added
- (CLI): `task ls` supports listing aggregating tasks via `--state=aggregating --group=<GROUP>` flags
- Enable rediss url parsing support
### Fixed
- Fixed overflow issue with 32-bit systems (For details, see https://github.com/hibiken/asynq/pull/426)
## [0.22.1] - 2022-02-20
### Fixed
- Fixed Redis version compatibility: Keep support for redis v4.0+
## [0.22.0] - 2022-02-19
### Added
- `BaseContext` is introduced in `Config` to specify callback hook to provide a base `context` from which `Handler` `context` is derived
- `IsOrphaned` field is added to `TaskInfo` to describe a task left in active state with no worker processing it.
### Changed
- `Server` now recovers tasks with an expired lease. Recovered tasks are retried/archived with `ErrLeaseExpired` error.
## [0.21.0] - 2022-01-22
### Added
- `PeriodicTaskManager` is added. Prefer using this over `Scheduler` as it has better support for dynamic periodic tasks.
- The `asynq stats` command now supports a `--json` option, making its output a JSON object
- Introduced new configuration for `DelayedTaskCheckInterval`. See [godoc](https://godoc.org/github.com/hibiken/asynq) for more details.
## [0.20.0] - 2021-12-19
### Added
- Package `x/metrics` is added.
- Tool `tools/metrics_exporter` binary is added.
- `ProcessedTotal` and `FailedTotal` fields were added to `QueueInfo` struct.
## [0.19.1] - 2021-12-12
### Added
- `Latency` field is added to `QueueInfo`.
- `EnqueueContext` method is added to `Client`.
### Fixed
- Fixed an error when user pass a duration less than 1s to `Unique` option
## [0.19.0] - 2021-11-06
### Changed
- `NewTask` takes `Option` as variadic argument
- Bumped minimum supported go version to 1.14 (i.e. go1.14 or higher is required).
### Added
- `Retention` option is added to allow user to specify task retention duration after completion.
- `TaskID` option is added to allow user to specify task ID.
- `ErrTaskIDConflict` sentinel error value is added.
- `ResultWriter` type is added and provided through `Task.ResultWriter` method.
- `TaskInfo` has new fields `CompletedAt`, `Result` and `Retention`.
### Removed
- `Client.SetDefaultOptions` is removed. Use `NewTask` instead to pass default options for tasks.
## [0.18.6] - 2021-10-03
### Changed
- Updated `github.com/go-redis/redis` package to v8
## [0.18.5] - 2021-09-01
### Added
- `IsFailure` config option is added to determine whether error returned from Handler counts as a failure.
## [0.18.4] - 2021-08-17
### Fixed
- Scheduler methods are now thread-safe. It's now safe to call `Register` and `Unregister` concurrently.
## [0.18.3] - 2021-08-09
### Changed
- `Client.Enqueue` no longer enqueues tasks with empty typename; Error message is returned.
## [0.18.2] - 2021-07-15
### Changed
- Changed `Queue` function to not to convert the provided queue name to lowercase. Queue names are now case-sensitive.
- `QueueInfo.MemoryUsage` is now an approximate usage value.
### Fixed
- Fixed latency issue around memory usage (see https://github.com/hibiken/asynq/issues/309).
## [0.18.1] - 2021-07-04
### Changed
- Changed to execute task recovering logic when server starts up; Previously it needed to wait for a minute for task recovering logic to exeucte.
### Fixed
- Fixed task recovering logic to execute every minute
## [0.18.0] - 2021-06-29
### Changed
- NewTask function now takes array of bytes as payload.
- Task `Type` and `Payload` should be accessed by a method call.
- `Server` API has changed. Renamed `Quiet` to `Stop`. Renamed `Stop` to `Shutdown`. _Note:_ As a result of this renaming, the behavior of `Stop` has changed. Please update the exising code to call `Shutdown` where it used to call `Stop`.
- `Scheduler` API has changed. Renamed `Stop` to `Shutdown`.
- Requires redis v4.0+ for multiple field/value pair support
- `Client.Enqueue` now returns `TaskInfo`
- `Inspector.RunTaskByKey` is replaced with `Inspector.RunTask`
- `Inspector.DeleteTaskByKey` is replaced with `Inspector.DeleteTask`
- `Inspector.ArchiveTaskByKey` is replaced with `Inspector.ArchiveTask`
- `inspeq` package is removed. All types and functions from the package is moved to `asynq` package.
- `WorkerInfo` field names have changed.
- `Inspector.CancelActiveTask` is renamed to `Inspector.CancelProcessing`
## [0.17.2] - 2021-06-06
### Fixed
- Free unique lock when task is deleted (https://github.com/hibiken/asynq/issues/275).
## [0.17.1] - 2021-04-04
### Fixed
- Fix bug in internal `RDB.memoryUsage` method.
## [0.17.0] - 2021-03-24
### Added
- `DialTimeout`, `ReadTimeout`, and `WriteTimeout` options are added to `RedisConnOpt`.
## [0.16.1] - 2021-03-20
### Fixed
- Replace `KEYS` command with `SCAN` as recommended by [redis doc](https://redis.io/commands/KEYS).
## [0.16.0] - 2021-03-10
### Added
- `Unregister` method is added to `Scheduler` to remove a registered entry.
## [0.15.0] - 2021-01-31
**IMPORTATNT**: All `Inspector` related code are moved to subpackage "github.com/hibiken/asynq/inspeq"
### Changed
- `Inspector` related code are moved to subpackage "github.com/hibken/asynq/inspeq".
- `RedisConnOpt` interface has changed slightly. If you have been passing `RedisClientOpt`, `RedisFailoverClientOpt`, or `RedisClusterClientOpt` as a pointer,
update your code to pass as a value.
- `ErrorMsg` field in `RetryTask` and `ArchivedTask` was renamed to `LastError`.
### Added
- `MaxRetry`, `Retried`, `LastError` fields were added to all task types returned from `Inspector`.
- `MemoryUsage` field was added to `QueueStats`.
- `DeleteAllPendingTasks`, `ArchiveAllPendingTasks` were added to `Inspector`
- `DeleteTaskByKey` and `ArchiveTaskByKey` now supports deleting/archiving `PendingTask`.
- asynq CLI now supports deleting/archiving pending tasks.
## [0.14.1] - 2021-01-19
### Fixed
- `go.mod` file for CLI
## [0.14.0] - 2021-01-14
**IMPORTATNT**: Please run `asynq migrate` command to migrate from the previous versions.
### Changed
- Renamed `DeadTask` to `ArchivedTask`.
- Renamed the operation `Kill` to `Archive` in `Inpsector`.
- Print stack trace when Handler panics.
- Include a file name and a line number in the error message when recovering from a panic.
### Added
- `DefaultRetryDelayFunc` is now a public API, which can be used in the custom `RetryDelayFunc`.
- `SkipRetry` error is added to be used as a return value from `Handler`.
- `Servers` method is added to `Inspector`
- `CancelActiveTask` method is added to `Inspector`.
- `ListSchedulerEnqueueEvents` method is added to `Inspector`.
- `SchedulerEntries` method is added to `Inspector`.
- `DeleteQueue` method is added to `Inspector`.
## [0.13.1] - 2020-11-22
### Fixed
- Fixed processor to wait for specified time duration before forcefully shutdown workers.
## [0.13.0] - 2020-10-13
### Added
- `Scheduler` type is added to enable periodic tasks. See the godoc for its APIs and [wiki](https://github.com/hibiken/asynq/wiki/Periodic-Tasks) for the getting-started guide.
### Changed
- interface `Option` has changed. See the godoc for the new interface.
This change would have no impact as long as you are using exported functions (e.g. `MaxRetry`, `Queue`, etc)
to create `Option`s.
### Added
- `Payload.String() string` method is added
- `Payload.MarshalJSON() ([]byte, error)` method is added
## [0.12.0] - 2020-09-12
**IMPORTANT**: If you are upgrading from a previous version, please install the latest version of the CLI `go get -u github.com/hibiken/asynq/tools/asynq` and run `asynq migrate` command. No process should be writing to Redis while you run the migration command.
## The semantics of queue have changed
Previously, we called tasks that are ready to be processed _"Enqueued tasks"_, and other tasks that are scheduled to be processed in the future _"Scheduled tasks"_, etc.
We changed the semantics of _"Enqueue"_ slightly; All tasks that client pushes to Redis are _Enqueued_ to a queue. Within a queue, tasks will transition from one state to another.
Possible task states are:
- `Pending`: task is ready to be processed (previously called "Enqueued")
- `Active`: tasks is currently being processed (previously called "InProgress")
- `Scheduled`: task is scheduled to be processed in the future
- `Retry`: task failed to be processed and will be retried again in the future
- `Dead`: task has exhausted all of its retries and stored for manual inspection purpose
**These semantics change is reflected in the new `Inspector` API and CLI commands.**
---
### Changed
#### `Client`
Use `ProcessIn` or `ProcessAt` option to schedule a task instead of `EnqueueIn` or `EnqueueAt`.
| Previously | v0.12.0 |
| --------------------------- | ------------------------------------------ |
| `client.EnqueueAt(t, task)` | `client.Enqueue(task, asynq.ProcessAt(t))` |
| `client.EnqueueIn(d, task)` | `client.Enqueue(task, asynq.ProcessIn(d))` |
#### `Inspector`
All Inspector methods are scoped to a queue, and the methods take `qname (string)` as the first argument.
`EnqueuedTask` is renamed to `PendingTask` and its corresponding methods.
`InProgressTask` is renamed to `ActiveTask` and its corresponding methods.
Command "Enqueue" is replaced by the verb "Run" (e.g. `EnqueueAllScheduledTasks` --> `RunAllScheduledTasks`)
#### `CLI`
CLI commands are restructured to use subcommands. Commands are organized into a few management commands:
To view details on any command, use `asynq help <command> <subcommand>`.
- `asynq stats`
- `asynq queue [ls inspect history rm pause unpause]`
- `asynq task [ls cancel delete kill run delete-all kill-all run-all]`
- `asynq server [ls]`
### Added
#### `RedisConnOpt`
- `RedisClusterClientOpt` is added to connect to Redis Cluster.
- `Username` field is added to all `RedisConnOpt` types in order to authenticate connection when Redis ACLs are used.
#### `Client`
- `ProcessIn(d time.Duration) Option` and `ProcessAt(t time.Time) Option` are added to replace `EnqueueIn` and `EnqueueAt` functionality.
#### `Inspector`
- `Queues() ([]string, error)` method is added to get all queue names.
- `ClusterKeySlot(qname string) (int64, error)` method is added to get queue's hash slot in Redis cluster.
- `ClusterNodes(qname string) ([]ClusterNode, error)` method is added to get a list of Redis cluster nodes for the given queue.
- `Close() error` method is added to close connection with redis.
### `Handler`
- `GetQueueName(ctx context.Context) (string, bool)` helper is added to extract queue name from a context.
## [0.11.0] - 2020-07-28
### Added
- `Inspector` type was added to monitor and mutate state of queues and tasks.
- `HealthCheckFunc` and `HealthCheckInterval` fields were added to `Config` to allow user to specify a callback
function to check for broker connection.
## [0.10.0] - 2020-07-06
### Changed
- All tasks now requires timeout or deadline. By default, timeout is set to 30 mins.
- Tasks that exceed its deadline are automatically retried.
- Encoding schema for task message has changed. Please install the latest CLI and run `migrate` command if
you have tasks enqueued with the previous version of asynq.
- API of `(*Client).Enqueue`, `(*Client).EnqueueIn`, and `(*Client).EnqueueAt` has changed to return a `*Result`.
- API of `ErrorHandler` has changed. It now takes context as the first argument and removed `retried`, `maxRetry` from the argument list.
Use `GetRetryCount` and/or `GetMaxRetry` to get the count values.
## [0.9.4] - 2020-06-13
### Fixed
- Fixes issue of same tasks processed by more than one worker (https://github.com/hibiken/asynq/issues/90).
## [0.9.3] - 2020-06-12
### Fixed
- Fixes the JSON number overflow issue (https://github.com/hibiken/asynq/issues/166).
## [0.9.2] - 2020-06-08
### Added
- The `pause` and `unpause` commands were added to the CLI. See README for the CLI for details.
## [0.9.1] - 2020-05-29
### Added
- `GetTaskID`, `GetRetryCount`, and `GetMaxRetry` functions were added to extract task metadata from context.
## [0.9.0] - 2020-05-16
### Changed
- `Logger` interface has changed. Please see the godoc for the new interface.
### Added
- `LogLevel` type is added. Server's log level can be specified through `LogLevel` field in `Config`.
## [0.8.3] - 2020-05-08
### Added
- `Close` method is added to `Client`.
## [0.8.2] - 2020-05-03
### Fixed
- [Fixed cancelfunc leak](https://github.com/hibiken/asynq/pull/145)
## [0.8.1] - 2020-04-27
### Added
- `ParseRedisURI` helper function is added to create a `RedisConnOpt` from a URI string.
- `SetDefaultOptions` method is added to `Client`.
## [0.8.0] - 2020-04-19
### Changed
- `Background` type is renamed to `Server`.
- To upgrade from the previous version, Update `NewBackground` to `NewServer` and pass `Config` by value.
- CLI is renamed to `asynq`.
- To upgrade the CLI to the latest version run `go get -u github.com/hibiken/tools/asynq`
- The `ps` command in CLI is renamed to `servers`
- `Concurrency` defaults to the number of CPUs when unset or set to a negative value.
### Added
- `ShutdownTimeout` field is added to `Config` to speicfy timeout duration used during graceful shutdown.
- New `Server` type exposes `Start`, `Stop`, and `Quiet` as well as `Run`.
## [0.7.1] - 2020-04-05
### Fixed
- Fixed signal handling for windows.
## [0.7.0] - 2020-03-22
### Changed
- Support Go v1.13+, dropped support for go v1.12
### Added
- `Unique` option was added to allow client to enqueue a task only if it's unique within a certain time period.
## [0.6.2] - 2020-03-15
### Added
- `Use` method was added to `ServeMux` to apply middlewares to all handlers.
## [0.6.1] - 2020-03-12
### Added
- `Client` can optionally schedule task with `asynq.Deadline(time)` to specify deadline for task's context. Default is no deadline.
- `Logger` option was added to config, which allows user to specify the logger used by the background instance.
## [0.6.0] - 2020-03-01
### Added
- Added `ServeMux` type to make it easy for users to implement Handler interface.
- `ErrorHandler` type was added. Allow users to specify error handling function (e.g. Report error to error reporting service such as Honeybadger, Bugsnag, etc)
## [0.5.0] - 2020-02-23
### Changed
- `Client` API has changed. Use `Enqueue`, `EnqueueAt` and `EnqueueIn` to enqueue and schedule tasks.
### Added
- `asynqmon workers` was added to list all running workers information
## [0.4.0] - 2020-02-13
### Changed
- `Handler` interface has changed. `ProcessTask` method takes two arguments `context.Context` and `*asynq.Task`
- `Queues` field in `Config` has change from `map[string]uint` to `map[string]int`
### Added
- `Client` can optionally schedule task with `asynq.Timeout(duration)` to specify timeout duration for task. Default is no timeout.
- `asynqmon cancel [task id]` will send a cancelation signal to the goroutine processing the speicified task.
## [0.3.0] - 2020-02-04
### Added
- `asynqmon ps` was added to list all background worker processes
## [0.2.2] - 2020-01-26
### Fixed
- Fixed restoring unfinished tasks back to correct queues.
### Changed
- `asynqmon ls` command is now paginated (default 30 tasks from first page)
- `asynqmon ls enqueued:[queue name]` requires queue name to be specified
## [0.2.1] - 2020-01-22
### Fixed
- More structured log messages
- Prevent spamming logs with a bunch of errors when Redis connection is lost
- Fixed and updated README doc
## [0.2.0] - 2020-01-19
### Added
- NewTask constructor
- `Queues` option in `Config` to specify mutiple queues with priority level
- `Client` can schedule a task with `asynq.Queue(name)` to specify which queue to use
- `StrictPriority` option in `Config` to specify whether the priority should be followed strictly
- `RedisConnOpt` to abstract away redis client implementation
- [CLI] `asynqmon rmq` command to remove queue
### Changed
- `Client` and `Background` constructors take `RedisConnOpt` as their first argument.
- `asynqmon stats` now shows the total of all enqueued tasks under "Enqueued"
- `asynqmon stats` now shows each queue's task count
- `asynqmon history` now doesn't take any arguments and shows data from the last 10 days by default (use `--days` flag to change the number of days)
- Task type is now immutable (i.e., Payload is read-only)
## [0.1.0] - 2020-01-04
### Added
- Initial version of asynq package
- Initial version of asynqmon CLI
================================================
FILE: CODE_OF_CONDUCT.md
================================================
# Contributor Covenant Code of Conduct
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, religion, or sexual identity
and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our
community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
and learning from the experience
* Focusing on what is best not just for us as individuals, but for the
overall community
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or
advances of any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email
address, without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.
Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official e-mail address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at
ken.hibino7@gmail.com.
All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the
reporter of any incident.
## Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:
### 1. Correction
**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate. A public apology may be requested.
### 2. Warning
**Community Impact**: A violation through a single incident or series
of actions.
**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or
permanent ban.
### 3. Temporary Ban
**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.
**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within
the community.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.0, available at
https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
Community Impact Guidelines were inspired by [Mozilla's code of conduct
enforcement ladder](https://github.com/mozilla/diversity).
[homepage]: https://www.contributor-covenant.org
For answers to common questions about this code of conduct, see the FAQ at
https://www.contributor-covenant.org/faq. Translations are available at
https://www.contributor-covenant.org/translations.
================================================
FILE: CONTRIBUTING.md
================================================
# Contributing
Thanks for your interest in contributing to Asynq!
We are open to, and grateful for, any contributions made by the community.
## Reporting Bugs
Have a look at our [issue tracker](https://github.com/hibiken/asynq/issues). If you can't find an issue (open or closed)
describing your problem (or a very similar one) there, please open a new issue with
the following details:
- Which versions of Go and Redis are you using?
- What are you trying to accomplish?
- What is the full error you are seeing?
- How can we reproduce this?
- Please quote as much of your code as needed to reproduce (best link to a
public repository or Gist)
## Getting Help
We run a [Gitter
channel](https://gitter.im/go-asynq/community) where you can ask questions and
get help. Feel free to ask there before opening a GitHub issue.
## Submitting Feature Requests
If you can't find an issue (open or closed) describing your idea on our [issue
tracker](https://github.com/hibiken/asynq/issues), open an issue. Adding answers to the following
questions in your description is +1:
- What do you want to do, and how do you expect Asynq to support you with that?
- How might this be added to Asynq?
- What are possible alternatives?
- Are there any disadvantages?
Thank you! We'll try to respond as quickly as possible.
## Contributing Code
1. Fork this repo
2. Download your fork `git clone git@github.com:your-username/asynq.git && cd asynq`
3. Create your branch `git checkout -b your-branch-name`
4. Make and commit your changes
5. Push the branch `git push origin your-branch-name`
6. Create a new pull request
Please try to keep your pull request focused in scope and avoid including unrelated commits.
Please run tests against redis cluster locally with `--redis_cluster` flag to ensure that code works for Redis cluster. TODO: Run tests using Redis cluster on CI.
After you have submitted your pull request, we'll try to get back to you as soon as possible. We may suggest some changes or improvements.
Thank you for contributing!
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2019 Kentaro Hibino
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: Makefile
================================================
ROOT_DIR:=$(shell dirname $(realpath $(firstword $(MAKEFILE_LIST))))
proto: internal/proto/asynq.proto
protoc -I=$(ROOT_DIR)/internal/proto \
--go_out=$(ROOT_DIR)/internal/proto \
--go_opt=module=github.com/hibiken/asynq/internal/proto \
$(ROOT_DIR)/internal/proto/asynq.proto
.PHONY: lint
lint:
golangci-lint run
================================================
FILE: README.md
================================================
<img src="https://user-images.githubusercontent.com/11155743/114697792-ffbfa580-9d26-11eb-8e5b-33bef69476dc.png" alt="Asynq logo" width="360px" />
# Simple, reliable & efficient distributed task queue in Go
[](https://godoc.org/github.com/hibiken/asynq)
[](https://goreportcard.com/report/github.com/hibiken/asynq)
[](https://opensource.org/licenses/MIT)
[](https://gitter.im/go-asynq/community)
Asynq is a Go library for queueing tasks and processing them asynchronously with workers. It's backed by [Redis](https://redis.io/) and is designed to be scalable yet easy to get started.
Highlevel overview of how Asynq works:
- Client puts tasks on a queue
- Server pulls tasks off queues and starts a worker goroutine for each task
- Tasks are processed concurrently by multiple workers
Task queues are used as a mechanism to distribute work across multiple machines. A system can consist of multiple worker servers and brokers, giving way to high availability and horizontal scaling.
**Example use case**
## Features
- Guaranteed [at least one execution](https://www.cloudcomputingpatterns.org/at_least_once_delivery/) of a task
- Scheduling of tasks
- [Retries](https://github.com/hibiken/asynq/wiki/Task-Retry) of failed tasks
- Automatic recovery of tasks in the event of a worker crash
- [Weighted priority queues](https://github.com/hibiken/asynq/wiki/Queue-Priority#weighted-priority)
- [Strict priority queues](https://github.com/hibiken/asynq/wiki/Queue-Priority#strict-priority)
- Low latency to add a task since writes are fast in Redis
- De-duplication of tasks using [unique option](https://github.com/hibiken/asynq/wiki/Unique-Tasks)
- Allow [timeout and deadline per task](https://github.com/hibiken/asynq/wiki/Task-Timeout-and-Cancelation)
- Allow [aggregating group of tasks](https://github.com/hibiken/asynq/wiki/Task-aggregation) to batch multiple successive operations
- [Flexible handler interface with support for middlewares](https://github.com/hibiken/asynq/wiki/Handler-Deep-Dive)
- [Ability to pause queue](/tools/asynq/README.md#pause) to stop processing tasks from the queue
- [Periodic Tasks](https://github.com/hibiken/asynq/wiki/Periodic-Tasks)
- [Support Redis Sentinels](https://github.com/hibiken/asynq/wiki/Automatic-Failover) for high availability
- Integration with [Prometheus](https://prometheus.io/) to collect and visualize queue metrics
- [Web UI](#web-ui) to inspect and remote-control queues and tasks
- [CLI](#command-line-tool) to inspect and remote-control queues and tasks
## Stability and Compatibility
**Status**: The library relatively stable and is currently undergoing **moderate development** with less frequent breaking API changes.
> ☝️ **Important Note**: Current major version is zero (`v0.x.x`) to accommodate rapid development and fast iteration while getting early feedback from users (_feedback on APIs are appreciated!_). The public API could change without a major version update before `v1.0.0` release.
### Redis Cluster Compatibility
Some of the lua scripts in this library may not be compatible with Redis Cluster.
## Sponsoring
If you are using this package in production, **please consider sponsoring the project to show your support!**
## Quickstart
Make sure you have Go installed ([download](https://golang.org/dl/)). The **last two** Go versions are supported (See https://go.dev/dl).
Initialize your project by creating a folder and then running `go mod init github.com/your/repo` ([learn more](https://blog.golang.org/using-go-modules)) inside the folder. Then install Asynq library with the [`go get`](https://golang.org/cmd/go/#hdr-Add_dependencies_to_current_module_and_install_them) command:
```sh
go get -u github.com/hibiken/asynq
```
Make sure you're running a Redis server locally or from a [Docker](https://hub.docker.com/_/redis) container. Version `4.0` or higher is required.
Next, write a package that encapsulates task creation and task handling.
```go
package tasks
import (
"context"
"encoding/json"
"fmt"
"log"
"time"
"github.com/hibiken/asynq"
)
// A list of task types.
const (
TypeEmailDelivery = "email:deliver"
TypeImageResize = "image:resize"
)
type EmailDeliveryPayload struct {
UserID int
TemplateID string
}
type ImageResizePayload struct {
SourceURL string
}
//----------------------------------------------
// Write a function NewXXXTask to create a task.
// A task consists of a type and a payload.
//----------------------------------------------
func NewEmailDeliveryTask(userID int, tmplID string) (*asynq.Task, error) {
payload, err := json.Marshal(EmailDeliveryPayload{UserID: userID, TemplateID: tmplID})
if err != nil {
return nil, err
}
return asynq.NewTask(TypeEmailDelivery, payload), nil
}
func NewImageResizeTask(src string) (*asynq.Task, error) {
payload, err := json.Marshal(ImageResizePayload{SourceURL: src})
if err != nil {
return nil, err
}
// task options can be passed to NewTask, which can be overridden at enqueue time.
return asynq.NewTask(TypeImageResize, payload, asynq.MaxRetry(5), asynq.Timeout(20 * time.Minute)), nil
}
//---------------------------------------------------------------
// Write a function HandleXXXTask to handle the input task.
// Note that it satisfies the asynq.HandlerFunc interface.
//
// Handler doesn't need to be a function. You can define a type
// that satisfies asynq.Handler interface. See examples below.
//---------------------------------------------------------------
func HandleEmailDeliveryTask(ctx context.Context, t *asynq.Task) error {
var p EmailDeliveryPayload
if err := json.Unmarshal(t.Payload(), &p); err != nil {
return fmt.Errorf("json.Unmarshal failed: %v: %w", err, asynq.SkipRetry)
}
log.Printf("Sending Email to User: user_id=%d, template_id=%s", p.UserID, p.TemplateID)
// Email delivery code ...
return nil
}
// ImageProcessor implements asynq.Handler interface.
type ImageProcessor struct {
// ... fields for struct
}
func (processor *ImageProcessor) ProcessTask(ctx context.Context, t *asynq.Task) error {
var p ImageResizePayload
if err := json.Unmarshal(t.Payload(), &p); err != nil {
return fmt.Errorf("json.Unmarshal failed: %v: %w", err, asynq.SkipRetry)
}
log.Printf("Resizing image: src=%s", p.SourceURL)
// Image resizing code ...
return nil
}
func NewImageProcessor() *ImageProcessor {
return &ImageProcessor{}
}
```
In your application code, import the above package and use [`Client`](https://pkg.go.dev/github.com/hibiken/asynq?tab=doc#Client) to put tasks on queues.
```go
package main
import (
"log"
"time"
"github.com/hibiken/asynq"
"your/app/package/tasks"
)
const redisAddr = "127.0.0.1:6379"
func main() {
client := asynq.NewClient(asynq.RedisClientOpt{Addr: redisAddr})
defer client.Close()
// ------------------------------------------------------
// Example 1: Enqueue task to be processed immediately.
// Use (*Client).Enqueue method.
// ------------------------------------------------------
task, err := tasks.NewEmailDeliveryTask(42, "some:template:id")
if err != nil {
log.Fatalf("could not create task: %v", err)
}
info, err := client.Enqueue(task)
if err != nil {
log.Fatalf("could not enqueue task: %v", err)
}
log.Printf("enqueued task: id=%s queue=%s", info.ID, info.Queue)
// ------------------------------------------------------------
// Example 2: Schedule task to be processed in the future.
// Use ProcessIn or ProcessAt option.
// ------------------------------------------------------------
info, err = client.Enqueue(task, asynq.ProcessIn(24*time.Hour))
if err != nil {
log.Fatalf("could not schedule task: %v", err)
}
log.Printf("enqueued task: id=%s queue=%s", info.ID, info.Queue)
// ----------------------------------------------------------------------------
// Example 3: Set other options to tune task processing behavior.
// Options include MaxRetry, Queue, Timeout, Deadline, Unique etc.
// ----------------------------------------------------------------------------
task, err = tasks.NewImageResizeTask("https://example.com/myassets/image.jpg")
if err != nil {
log.Fatalf("could not create task: %v", err)
}
info, err = client.Enqueue(task, asynq.MaxRetry(10), asynq.Timeout(3 * time.Minute))
if err != nil {
log.Fatalf("could not enqueue task: %v", err)
}
log.Printf("enqueued task: id=%s queue=%s", info.ID, info.Queue)
}
```
Next, start a worker server to process these tasks in the background. To start the background workers, use [`Server`](https://pkg.go.dev/github.com/hibiken/asynq?tab=doc#Server) and provide your [`Handler`](https://pkg.go.dev/github.com/hibiken/asynq?tab=doc#Handler) to process the tasks.
You can optionally use [`ServeMux`](https://pkg.go.dev/github.com/hibiken/asynq?tab=doc#ServeMux) to create a handler, just as you would with [`net/http`](https://golang.org/pkg/net/http/) Handler.
```go
package main
import (
"log"
"github.com/hibiken/asynq"
"your/app/package/tasks"
)
const redisAddr = "127.0.0.1:6379"
func main() {
srv := asynq.NewServer(
asynq.RedisClientOpt{Addr: redisAddr},
asynq.Config{
// Specify how many concurrent workers to use
Concurrency: 10,
// Optionally specify multiple queues with different priority.
Queues: map[string]int{
"critical": 6,
"default": 3,
"low": 1,
},
// See the godoc for other configuration options
},
)
// mux maps a type to a handler
mux := asynq.NewServeMux()
mux.HandleFunc(tasks.TypeEmailDelivery, tasks.HandleEmailDeliveryTask)
mux.Handle(tasks.TypeImageResize, tasks.NewImageProcessor())
// ...register other handlers...
if err := srv.Run(mux); err != nil {
log.Fatalf("could not run server: %v", err)
}
}
```
For a more detailed walk-through of the library, see our [Getting Started](https://github.com/hibiken/asynq/wiki/Getting-Started) guide.
To learn more about `asynq` features and APIs, see the package [godoc](https://godoc.org/github.com/hibiken/asynq).
## Web UI
[Asynqmon](https://github.com/hibiken/asynqmon) is a web based tool for monitoring and administrating Asynq queues and tasks.
Here's a few screenshots of the Web UI:
**Queues view**
**Tasks view**
**Metrics view**
<img width="1532" alt="Screen Shot 2021-12-19 at 4 37 19 PM" src="https://user-images.githubusercontent.com/10953044/146777420-cae6c476-bac6-469c-acce-b2f6584e8707.png">
**Settings and adaptive dark mode**
For details on how to use the tool, refer to the tool's [README](https://github.com/hibiken/asynqmon#readme).
## Command Line Tool
Asynq ships with a command line tool to inspect the state of queues and tasks.
To install the CLI tool, run the following command:
```sh
go install github.com/hibiken/asynq/tools/asynq@latest
```
Here's an example of running the `asynq dash` command:
For details on how to use the tool, refer to the tool's [README](/tools/asynq/README.md).
## Contributing
We are open to, and grateful for, any contributions (GitHub issues/PRs, feedback on [Gitter channel](https://gitter.im/go-asynq/community), etc) made by the community.
Please see the [Contribution Guide](/CONTRIBUTING.md) before contributing.
## License
Copyright (c) 2019-present [Ken Hibino](https://github.com/hibiken) and [Contributors](https://github.com/hibiken/asynq/graphs/contributors). `Asynq` is free and open-source software licensed under the [MIT License](https://github.com/hibiken/asynq/blob/master/LICENSE). Official logo was created by [Vic Shóstak](https://github.com/koddr) and distributed under [Creative Commons](https://creativecommons.org/publicdomain/zero/1.0/) license (CC0 1.0 Universal).
================================================
FILE: aggregator.go
================================================
// Copyright 2022 Kentaro Hibino. All rights reserved.
// Use of this source code is governed by a MIT license
// that can be found in the LICENSE file.
package asynq
import (
"context"
"sync"
"time"
"github.com/hibiken/asynq/internal/base"
"github.com/hibiken/asynq/internal/log"
)
// An aggregator is responsible for checking groups and aggregate into one task
// if any of the grouping condition is met.
type aggregator struct {
logger *log.Logger
broker base.Broker
client *Client
// channel to communicate back to the long running "aggregator" goroutine.
done chan struct{}
// list of queue names to check and aggregate.
queues []string
// Group configurations
gracePeriod time.Duration
maxDelay time.Duration
maxSize int
// User provided group aggregator.
ga GroupAggregator
// interval used to check for aggregation
interval time.Duration
// sema is a counting semaphore to ensure the number of active aggregating function
// does not exceed the limit.
sema chan struct{}
}
type aggregatorParams struct {
logger *log.Logger
broker base.Broker
queues []string
gracePeriod time.Duration
maxDelay time.Duration
maxSize int
groupAggregator GroupAggregator
}
const (
// Maximum number of aggregation checks in flight concurrently.
maxConcurrentAggregationChecks = 3
// Default interval used for aggregation checks. If the provided gracePeriod is less than
// the default, use the gracePeriod.
defaultAggregationCheckInterval = 7 * time.Second
)
func newAggregator(params aggregatorParams) *aggregator {
interval := defaultAggregationCheckInterval
if params.gracePeriod < interval {
interval = params.gracePeriod
}
return &aggregator{
logger: params.logger,
broker: params.broker,
client: &Client{broker: params.broker},
done: make(chan struct{}),
queues: params.queues,
gracePeriod: params.gracePeriod,
maxDelay: params.maxDelay,
maxSize: params.maxSize,
ga: params.groupAggregator,
sema: make(chan struct{}, maxConcurrentAggregationChecks),
interval: interval,
}
}
func (a *aggregator) shutdown() {
if a.ga == nil {
return
}
a.logger.Debug("Aggregator shutting down...")
// Signal the aggregator goroutine to stop.
a.done <- struct{}{}
}
func (a *aggregator) start(wg *sync.WaitGroup) {
if a.ga == nil {
return
}
wg.Add(1)
go func() {
defer wg.Done()
ticker := time.NewTicker(a.interval)
for {
select {
case <-a.done:
a.logger.Debug("Waiting for all aggregation checks to finish...")
// block until all aggregation checks released the token
for i := 0; i < cap(a.sema); i++ {
a.sema <- struct{}{}
}
a.logger.Debug("Aggregator done")
ticker.Stop()
return
case t := <-ticker.C:
a.exec(t)
}
}
}()
}
func (a *aggregator) exec(t time.Time) {
select {
case a.sema <- struct{}{}: // acquire token
go a.aggregate(t)
default:
// If the semaphore blocks, then we are currently running max number of
// aggregation checks. Skip this round and log warning.
a.logger.Warnf("Max number of aggregation checks in flight. Skipping")
}
}
func (a *aggregator) aggregate(t time.Time) {
defer func() { <-a.sema /* release token */ }()
for _, qname := range a.queues {
groups, err := a.broker.ListGroups(qname)
if err != nil {
a.logger.Errorf("Failed to list groups in queue: %q", qname)
continue
}
for _, gname := range groups {
aggregationSetID, err := a.broker.AggregationCheck(
qname, gname, t, a.gracePeriod, a.maxDelay, a.maxSize)
if err != nil {
a.logger.Errorf("Failed to run aggregation check: queue=%q group=%q", qname, gname)
continue
}
if aggregationSetID == "" {
a.logger.Debugf("No aggregation needed at this time: queue=%q group=%q", qname, gname)
continue
}
// Aggregate and enqueue.
msgs, deadline, err := a.broker.ReadAggregationSet(qname, gname, aggregationSetID)
if err != nil {
a.logger.Errorf("Failed to read aggregation set: queue=%q, group=%q, setID=%q",
qname, gname, aggregationSetID)
continue
}
tasks := make([]*Task, len(msgs))
for i, m := range msgs {
tasks[i] = NewTaskWithHeaders(m.Type, m.Payload, m.Headers)
}
aggregatedTask := a.ga.Aggregate(gname, tasks)
ctx, cancel := context.WithDeadline(context.Background(), deadline)
if _, err := a.client.EnqueueContext(ctx, aggregatedTask, Queue(qname)); err != nil {
a.logger.Errorf("Failed to enqueue aggregated task (queue=%q, group=%q, setID=%q): %v",
qname, gname, aggregationSetID, err)
cancel()
continue
}
if err := a.broker.DeleteAggregationSet(ctx, qname, gname, aggregationSetID); err != nil {
a.logger.Warnf("Failed to delete aggregation set: queue=%q, group=%q, setID=%q",
qname, gname, aggregationSetID)
}
cancel()
}
}
}
================================================
FILE: aggregator_test.go
================================================
// Copyright 2022 Kentaro Hibino. All rights reserved.
// Use of this source code is governed by a MIT license
// that can be found in the LICENSE file.
package asynq
import (
"sync"
"testing"
"time"
"github.com/google/go-cmp/cmp"
"github.com/hibiken/asynq/internal/base"
"github.com/hibiken/asynq/internal/rdb"
h "github.com/hibiken/asynq/internal/testutil"
)
func TestAggregator(t *testing.T) {
r := setup(t)
defer r.Close()
rdbClient := rdb.NewRDB(r)
client := Client{broker: rdbClient}
tests := []struct {
desc string
gracePeriod time.Duration
maxDelay time.Duration
maxSize int
aggregateFunc func(gname string, tasks []*Task) *Task
tasks []*Task // tasks to enqueue
enqueueFrequency time.Duration // time between one enqueue event to another
waitTime time.Duration // time to wait
wantGroups map[string]map[string][]base.Z
wantPending map[string][]*base.TaskMessage
}{
{
desc: "group older than the grace period should be aggregated",
gracePeriod: 1 * time.Second,
maxDelay: 0, // no maxdelay limit
maxSize: 0, // no maxsize limit
aggregateFunc: func(gname string, tasks []*Task) *Task {
return NewTask(gname, nil, MaxRetry(len(tasks))) // use max retry to see how many tasks were aggregated
},
tasks: []*Task{
NewTask("task1", nil, Group("mygroup")),
NewTask("task2", nil, Group("mygroup")),
NewTask("task3", nil, Group("mygroup")),
},
enqueueFrequency: 300 * time.Millisecond,
waitTime: 3 * time.Second,
wantGroups: map[string]map[string][]base.Z{
"default": {
"mygroup": {},
},
},
wantPending: map[string][]*base.TaskMessage{
"default": {
h.NewTaskMessageBuilder().SetType("mygroup").SetRetry(3).Build(),
},
},
},
{
desc: "group older than the max-delay should be aggregated",
gracePeriod: 2 * time.Second,
maxDelay: 4 * time.Second,
maxSize: 0, // no maxsize limit
aggregateFunc: func(gname string, tasks []*Task) *Task {
return NewTask(gname, nil, MaxRetry(len(tasks))) // use max retry to see how many tasks were aggregated
},
tasks: []*Task{
NewTask("task1", nil, Group("mygroup")), // time 0
NewTask("task2", nil, Group("mygroup")), // time 1s
NewTask("task3", nil, Group("mygroup")), // time 2s
NewTask("task4", nil, Group("mygroup")), // time 3s
},
enqueueFrequency: 1 * time.Second,
waitTime: 4 * time.Second,
wantGroups: map[string]map[string][]base.Z{
"default": {
"mygroup": {},
},
},
wantPending: map[string][]*base.TaskMessage{
"default": {
h.NewTaskMessageBuilder().SetType("mygroup").SetRetry(4).Build(),
},
},
},
{
desc: "group reached the max-size should be aggregated",
gracePeriod: 1 * time.Minute,
maxDelay: 0, // no maxdelay limit
maxSize: 5,
aggregateFunc: func(gname string, tasks []*Task) *Task {
return NewTask(gname, nil, MaxRetry(len(tasks))) // use max retry to see how many tasks were aggregated
},
tasks: []*Task{
NewTask("task1", nil, Group("mygroup")),
NewTask("task2", nil, Group("mygroup")),
NewTask("task3", nil, Group("mygroup")),
NewTask("task4", nil, Group("mygroup")),
NewTask("task5", nil, Group("mygroup")),
},
enqueueFrequency: 300 * time.Millisecond,
waitTime: defaultAggregationCheckInterval * 2,
wantGroups: map[string]map[string][]base.Z{
"default": {
"mygroup": {},
},
},
wantPending: map[string][]*base.TaskMessage{
"default": {
h.NewTaskMessageBuilder().SetType("mygroup").SetRetry(5).Build(),
},
},
},
}
for _, tc := range tests {
h.FlushDB(t, r)
aggregator := newAggregator(aggregatorParams{
logger: testLogger,
broker: rdbClient,
queues: []string{"default"},
gracePeriod: tc.gracePeriod,
maxDelay: tc.maxDelay,
maxSize: tc.maxSize,
groupAggregator: GroupAggregatorFunc(tc.aggregateFunc),
})
var wg sync.WaitGroup
aggregator.start(&wg)
for _, task := range tc.tasks {
if _, err := client.Enqueue(task); err != nil {
t.Errorf("%s: Client Enqueue failed: %v", tc.desc, err)
aggregator.shutdown()
wg.Wait()
continue
}
time.Sleep(tc.enqueueFrequency)
}
time.Sleep(tc.waitTime)
for qname, groups := range tc.wantGroups {
for gname, want := range groups {
gotGroup := h.GetGroupEntries(t, r, qname, gname)
if diff := cmp.Diff(want, gotGroup, h.SortZSetEntryOpt); diff != "" {
t.Errorf("%s: mismatch found in %q; (-want,+got)\n%s", tc.desc, base.GroupKey(qname, gname), diff)
}
}
}
for qname, want := range tc.wantPending {
gotPending := h.GetPendingMessages(t, r, qname)
if diff := cmp.Diff(want, gotPending, h.SortMsgOpt, h.IgnoreIDOpt); diff != "" {
t.Errorf("%s: mismatch found in %q; (-want,+got)\n%s", tc.desc, base.PendingKey(qname), diff)
}
}
aggregator.shutdown()
wg.Wait()
}
}
================================================
FILE: asynq.go
================================================
// Copyright 2020 Kentaro Hibino. All rights reserved.
// Use of this source code is governed by a MIT license
// that can be found in the LICENSE file.
package asynq
import (
"context"
"crypto/tls"
"fmt"
"maps"
"net"
"net/url"
"strconv"
"strings"
"time"
"github.com/hibiken/asynq/internal/base"
"github.com/redis/go-redis/v9"
)
// Task represents a unit of work to be performed.
type Task struct {
// typename indicates the type of task to be performed.
typename string
// payload holds data needed to perform the task.
payload []byte
// headers holds additional metadata for the task.
headers map[string]string
// opts holds options for the task.
opts []Option
// w is the ResultWriter for the task.
w *ResultWriter
}
func (t *Task) Type() string { return t.typename }
func (t *Task) Payload() []byte { return t.payload }
func (t *Task) Headers() map[string]string { return t.headers }
// ResultWriter returns a pointer to the ResultWriter associated with the task.
//
// Nil pointer is returned if called on a newly created task (i.e. task created by calling NewTask).
// Only the tasks passed to Handler.ProcessTask have a valid ResultWriter pointer.
func (t *Task) ResultWriter() *ResultWriter { return t.w }
// NewTask returns a new Task given a type name and payload data.
// Options can be passed to configure task processing behavior.
func NewTask(typename string, payload []byte, opts ...Option) *Task {
return &Task{
typename: typename,
payload: payload,
headers: nil,
opts: opts,
}
}
// NewTaskWithHeaders returns a new Task given a type name, payload data, and headers.
// Options can be passed to configure task processing behavior.
// TODO: In the next major (breaking) release, fold this functionality into NewTask
//
// so that headers are supported directly. After that, remove this method.
func NewTaskWithHeaders(typename string, payload []byte, headers map[string]string, opts ...Option) *Task {
return &Task{
typename: typename,
payload: payload,
headers: maps.Clone(headers),
opts: opts,
}
}
// newTask creates a task with the given typename, payload and ResultWriter.
func newTask(typename string, payload []byte, w *ResultWriter) *Task {
return &Task{
typename: typename,
payload: payload,
headers: make(map[string]string),
w: w,
}
}
// A TaskInfo describes a task and its metadata.
type TaskInfo struct {
// ID is the identifier of the task.
ID string
// Queue is the name of the queue in which the task belongs.
Queue string
// Type is the type name of the task.
Type string
// Payload is the payload data of the task.
Payload []byte
// Headers holds additional metadata for the task.
Headers map[string]string
// State indicates the task state.
State TaskState
// MaxRetry is the maximum number of times the task can be retried.
MaxRetry int
// Retried is the number of times the task has retried so far.
Retried int
// LastErr is the error message from the last failure.
LastErr string
// LastFailedAt is the time time of the last failure if any.
// If the task has no failures, LastFailedAt is zero time (i.e. time.Time{}).
LastFailedAt time.Time
// Timeout is the duration the task can be processed by Handler before being retried,
// zero if not specified
Timeout time.Duration
// Deadline is the deadline for the task, zero value if not specified.
Deadline time.Time
// Group is the name of the group in which the task belongs.
//
// Tasks in the same queue can be grouped together by Group name and will be aggregated into one task
// by a Server processing the queue.
//
// Empty string (default) indicates task does not belong to any groups, and no aggregation will be applied to the task.
Group string
// NextProcessAt is the time the task is scheduled to be processed,
// zero if not applicable.
NextProcessAt time.Time
// IsOrphaned describes whether the task is left in active state with no worker processing it.
// An orphaned task indicates that the worker has crashed or experienced network failures and was not able to
// extend its lease on the task.
//
// This task will be recovered by running a server against the queue the task is in.
// This field is only applicable to tasks with TaskStateActive.
IsOrphaned bool
// Retention is duration of the retention period after the task is successfully processed.
Retention time.Duration
// CompletedAt is the time when the task is processed successfully.
// Zero value (i.e. time.Time{}) indicates no value.
CompletedAt time.Time
// Result holds the result data associated with the task.
// Use ResultWriter to write result data from the Handler.
Result []byte
}
// If t is non-zero, returns time converted from t as unix time in seconds.
// If t is zero, returns zero value of time.Time.
func fromUnixTimeOrZero(t int64) time.Time {
if t == 0 {
return time.Time{}
}
return time.Unix(t, 0)
}
func newTaskInfo(msg *base.TaskMessage, state base.TaskState, nextProcessAt time.Time, result []byte) *TaskInfo {
info := TaskInfo{
ID: msg.ID,
Queue: msg.Queue,
Type: msg.Type,
Payload: msg.Payload, // Do we need to make a copy?
Headers: msg.Headers,
MaxRetry: msg.Retry,
Retried: msg.Retried,
LastErr: msg.ErrorMsg,
Group: msg.GroupKey,
Timeout: time.Duration(msg.Timeout) * time.Second,
Deadline: fromUnixTimeOrZero(msg.Deadline),
Retention: time.Duration(msg.Retention) * time.Second,
NextProcessAt: nextProcessAt,
LastFailedAt: fromUnixTimeOrZero(msg.LastFailedAt),
CompletedAt: fromUnixTimeOrZero(msg.CompletedAt),
Result: result,
}
switch state {
case base.TaskStateActive:
info.State = TaskStateActive
case base.TaskStatePending:
info.State = TaskStatePending
case base.TaskStateScheduled:
info.State = TaskStateScheduled
case base.TaskStateRetry:
info.State = TaskStateRetry
case base.TaskStateArchived:
info.State = TaskStateArchived
case base.TaskStateCompleted:
info.State = TaskStateCompleted
case base.TaskStateAggregating:
info.State = TaskStateAggregating
default:
panic(fmt.Sprintf("internal error: unknown state: %d", state))
}
return &info
}
// TaskState denotes the state of a task.
type TaskState int
const (
// Indicates that the task is currently being processed by Handler.
TaskStateActive TaskState = iota + 1
// Indicates that the task is ready to be processed by Handler.
TaskStatePending
// Indicates that the task is scheduled to be processed some time in the future.
TaskStateScheduled
// Indicates that the task has previously failed and scheduled to be processed some time in the future.
TaskStateRetry
// Indicates that the task is archived and stored for inspection purposes.
TaskStateArchived
// Indicates that the task is processed successfully and retained until the retention TTL expires.
TaskStateCompleted
// Indicates that the task is waiting in a group to be aggregated into one task.
TaskStateAggregating
)
func (s TaskState) String() string {
switch s {
case TaskStateActive:
return "active"
case TaskStatePending:
return "pending"
case TaskStateScheduled:
return "scheduled"
case TaskStateRetry:
return "retry"
case TaskStateArchived:
return "archived"
case TaskStateCompleted:
return "completed"
case TaskStateAggregating:
return "aggregating"
}
panic("asynq: unknown task state")
}
// RedisConnOpt is a discriminated union of types that represent Redis connection configuration option.
//
// RedisConnOpt represents a sum of following types:
//
// - RedisClientOpt
// - RedisFailoverClientOpt
// - RedisClusterClientOpt
type RedisConnOpt interface {
// MakeRedisClient returns a new redis client instance.
// Return value is intentionally opaque to hide the implementation detail of redis client.
MakeRedisClient() interface{}
}
// RedisClientOpt is used to create a redis client that connects
// to a redis server directly.
type RedisClientOpt struct {
// Network type to use, either tcp or unix.
// Default is tcp.
Network string
// Redis server address in "host:port" format.
Addr string
// Username to authenticate the current connection when Redis ACLs are used.
// See: https://redis.io/commands/auth.
Username string
// Password to authenticate the current connection.
// See: https://redis.io/commands/auth.
Password string
// Redis DB to select after connecting to a server.
// See: https://redis.io/commands/select.
DB int
// Dial timeout for establishing new connections.
// Default is 5 seconds.
DialTimeout time.Duration
// Timeout for socket reads.
// If timeout is reached, read commands will fail with a timeout error
// instead of blocking.
//
// Use value -1 for no timeout and 0 for default.
// Default is 3 seconds.
ReadTimeout time.Duration
// Timeout for socket writes.
// If timeout is reached, write commands will fail with a timeout error
// instead of blocking.
//
// Use value -1 for no timeout and 0 for default.
// Default is ReadTimout.
WriteTimeout time.Duration
// Maximum number of socket connections.
// Default is 10 connections per every CPU as reported by runtime.NumCPU.
PoolSize int
// TLS Config used to connect to a server.
// TLS will be negotiated only if this field is set.
TLSConfig *tls.Config
}
func (opt RedisClientOpt) MakeRedisClient() interface{} {
return redis.NewClient(&redis.Options{
Network: opt.Network,
Addr: opt.Addr,
Username: opt.Username,
Password: opt.Password,
DB: opt.DB,
DialTimeout: opt.DialTimeout,
ReadTimeout: opt.ReadTimeout,
WriteTimeout: opt.WriteTimeout,
PoolSize: opt.PoolSize,
TLSConfig: opt.TLSConfig,
})
}
// RedisFailoverClientOpt is used to creates a redis client that talks
// to redis sentinels for service discovery and has an automatic failover
// capability.
type RedisFailoverClientOpt struct {
// Redis master name that monitored by sentinels.
MasterName string
// Addresses of sentinels in "host:port" format.
// Use at least three sentinels to avoid problems described in
// https://redis.io/topics/sentinel.
SentinelAddrs []string
// Redis sentinel username.
SentinelUsername string
// Redis sentinel password.
SentinelPassword string
// Username to authenticate the current connection when Redis ACLs are used.
// See: https://redis.io/commands/auth.
Username string
// Password to authenticate the current connection.
// See: https://redis.io/commands/auth.
Password string
// Redis DB to select after connecting to a server.
// See: https://redis.io/commands/select.
DB int
// Dial timeout for establishing new connections.
// Default is 5 seconds.
DialTimeout time.Duration
// Timeout for socket reads.
// If timeout is reached, read commands will fail with a timeout error
// instead of blocking.
//
// Use value -1 for no timeout and 0 for default.
// Default is 3 seconds.
ReadTimeout time.Duration
// Timeout for socket writes.
// If timeout is reached, write commands will fail with a timeout error
// instead of blocking.
//
// Use value -1 for no timeout and 0 for default.
// Default is ReadTimeout
WriteTimeout time.Duration
// Maximum number of socket connections.
// Default is 10 connections per every CPU as reported by runtime.NumCPU.
PoolSize int
// TLS Config used to connect to a server.
// TLS will be negotiated only if this field is set.
TLSConfig *tls.Config
}
func (opt RedisFailoverClientOpt) MakeRedisClient() interface{} {
return redis.NewFailoverClient(&redis.FailoverOptions{
MasterName: opt.MasterName,
SentinelAddrs: opt.SentinelAddrs,
SentinelUsername: opt.SentinelUsername,
SentinelPassword: opt.SentinelPassword,
Username: opt.Username,
Password: opt.Password,
DB: opt.DB,
DialTimeout: opt.DialTimeout,
ReadTimeout: opt.ReadTimeout,
WriteTimeout: opt.WriteTimeout,
PoolSize: opt.PoolSize,
TLSConfig: opt.TLSConfig,
})
}
// RedisClusterClientOpt is used to creates a redis client that connects to
// redis cluster.
type RedisClusterClientOpt struct {
// A seed list of host:port addresses of cluster nodes.
Addrs []string
// The maximum number of retries before giving up.
// Command is retried on network errors and MOVED/ASK redirects.
// Default is 8 retries.
MaxRedirects int
// Username to authenticate the current connection when Redis ACLs are used.
// See: https://redis.io/commands/auth.
Username string
// Password to authenticate the current connection.
// See: https://redis.io/commands/auth.
Password string
// Dial timeout for establishing new connections.
// Default is 5 seconds.
DialTimeout time.Duration
// Timeout for socket reads.
// If timeout is reached, read commands will fail with a timeout error
// instead of blocking.
//
// Use value -1 for no timeout and 0 for default.
// Default is 3 seconds.
ReadTimeout time.Duration
// Timeout for socket writes.
// If timeout is reached, write commands will fail with a timeout error
// instead of blocking.
//
// Use value -1 for no timeout and 0 for default.
// Default is ReadTimeout.
WriteTimeout time.Duration
// TLS Config used to connect to a server.
// TLS will be negotiated only if this field is set.
TLSConfig *tls.Config
}
func (opt RedisClusterClientOpt) MakeRedisClient() interface{} {
return redis.NewClusterClient(&redis.ClusterOptions{
Addrs: opt.Addrs,
MaxRedirects: opt.MaxRedirects,
Username: opt.Username,
Password: opt.Password,
DialTimeout: opt.DialTimeout,
ReadTimeout: opt.ReadTimeout,
WriteTimeout: opt.WriteTimeout,
TLSConfig: opt.TLSConfig,
})
}
// ParseRedisURI parses redis uri string and returns RedisConnOpt if uri is valid.
// It returns a non-nil error if uri cannot be parsed.
//
// Three URI schemes are supported, which are redis:, rediss:, redis-socket:, and redis-sentinel:.
// Supported formats are:
//
// redis://[:password@]host[:port][/dbnumber]
// rediss://[:password@]host[:port][/dbnumber]
// redis-socket://[:password@]path[?db=dbnumber]
// redis-sentinel://[:password@]host1[:port][,host2:[:port]][,hostN:[:port]][?master=masterName]
func ParseRedisURI(uri string) (RedisConnOpt, error) {
u, err := url.Parse(uri)
if err != nil {
return nil, fmt.Errorf("asynq: could not parse redis uri: %w", err)
}
switch u.Scheme {
case "redis", "rediss":
return parseRedisURI(u)
case "redis-socket":
return parseRedisSocketURI(u)
case "redis-sentinel":
return parseRedisSentinelURI(u)
default:
return nil, fmt.Errorf("asynq: unsupported uri scheme: %q", u.Scheme)
}
}
func parseRedisURI(u *url.URL) (RedisConnOpt, error) {
var db int
var err error
var redisConnOpt RedisClientOpt
if len(u.Path) > 0 {
xs := strings.Split(strings.Trim(u.Path, "/"), "/")
db, err = strconv.Atoi(xs[0])
if err != nil {
return nil, fmt.Errorf("asynq: could not parse redis uri: database number should be the first segment of the path")
}
}
var password string
if v, ok := u.User.Password(); ok {
password = v
}
if u.Scheme == "rediss" {
h, _, err := net.SplitHostPort(u.Host)
if err != nil {
h = u.Host
}
redisConnOpt.TLSConfig = &tls.Config{ServerName: h}
}
redisConnOpt.Addr = u.Host
redisConnOpt.Password = password
redisConnOpt.DB = db
return redisConnOpt, nil
}
func parseRedisSocketURI(u *url.URL) (RedisConnOpt, error) {
const errPrefix = "asynq: could not parse redis socket uri"
if len(u.Path) == 0 {
return nil, fmt.Errorf("%s: path does not exist", errPrefix)
}
q := u.Query()
var db int
var err error
if n := q.Get("db"); n != "" {
db, err = strconv.Atoi(n)
if err != nil {
return nil, fmt.Errorf("%s: query param `db` should be a number", errPrefix)
}
}
var password string
if v, ok := u.User.Password(); ok {
password = v
}
return RedisClientOpt{Network: "unix", Addr: u.Path, DB: db, Password: password}, nil
}
func parseRedisSentinelURI(u *url.URL) (RedisConnOpt, error) {
addrs := strings.Split(u.Host, ",")
master := u.Query().Get("master")
var password string
if v, ok := u.User.Password(); ok {
password = v
}
return RedisFailoverClientOpt{MasterName: master, SentinelAddrs: addrs, SentinelPassword: password}, nil
}
// ResultWriter is a client interface to write result data for a task.
// It writes the data to the redis instance the server is connected to.
type ResultWriter struct {
id string // task ID this writer is responsible for
qname string // queue name the task belongs to
broker base.Broker
ctx context.Context // context associated with the task
}
// Write writes the given data as a result of the task the ResultWriter is associated with.
func (w *ResultWriter) Write(data []byte) (n int, err error) {
select {
case <-w.ctx.Done():
return 0, fmt.Errorf("failed to write task result: %w", w.ctx.Err())
default:
}
return w.broker.WriteResult(w.qname, w.id, data)
}
// TaskID returns the ID of the task the ResultWriter is associated with.
func (w *ResultWriter) TaskID() string {
return w.id
}
================================================
FILE: asynq_test.go
================================================
// Copyright 2020 Kentaro Hibino. All rights reserved.
// Use of this source code is governed by a MIT license
// that can be found in the LICENSE file.
package asynq
import (
"crypto/tls"
"flag"
"sort"
"strings"
"testing"
"github.com/google/go-cmp/cmp"
"github.com/google/go-cmp/cmp/cmpopts"
"github.com/hibiken/asynq/internal/log"
h "github.com/hibiken/asynq/internal/testutil"
"github.com/redis/go-redis/v9"
)
//============================================================================
// This file defines helper functions and variables used in other test files.
//============================================================================
// variables used for package testing.
var (
redisAddr string
redisDB int
useRedisCluster bool
redisClusterAddrs string // comma-separated list of host:port
testLogLevel = FatalLevel
)
var testLogger *log.Logger
func init() {
flag.StringVar(&redisAddr, "redis_addr", "localhost:6379", "redis address to use in testing")
flag.IntVar(&redisDB, "redis_db", 14, "redis db number to use in testing")
flag.BoolVar(&useRedisCluster, "redis_cluster", false, "use redis cluster as a broker in testing")
flag.StringVar(&redisClusterAddrs, "redis_cluster_addrs", "localhost:7000,localhost:7001,localhost:7002", "comma separated list of redis server addresses")
flag.Var(&testLogLevel, "loglevel", "log level to use in testing")
testLogger = log.NewLogger(nil)
testLogger.SetLevel(toInternalLogLevel(testLogLevel))
}
func setup(tb testing.TB) (r redis.UniversalClient) {
tb.Helper()
if useRedisCluster {
addrs := strings.Split(redisClusterAddrs, ",")
if len(addrs) == 0 {
tb.Fatal("No redis cluster addresses provided. Please set addresses using --redis_cluster_addrs flag.")
}
r = redis.NewClusterClient(&redis.ClusterOptions{
Addrs: addrs,
})
} else {
r = redis.NewClient(&redis.Options{
Addr: redisAddr,
DB: redisDB,
})
}
// Start each test with a clean slate.
h.FlushDB(tb, r)
return r
}
func getRedisConnOpt(tb testing.TB) RedisConnOpt {
tb.Helper()
if useRedisCluster {
addrs := strings.Split(redisClusterAddrs, ",")
if len(addrs) == 0 {
tb.Fatal("No redis cluster addresses provided. Please set addresses using --redis_cluster_addrs flag.")
}
return RedisClusterClientOpt{
Addrs: addrs,
}
}
return RedisClientOpt{
Addr: redisAddr,
DB: redisDB,
}
}
var sortTaskOpt = cmp.Transformer("SortMsg", func(in []*Task) []*Task {
out := append([]*Task(nil), in...) // Copy input to avoid mutating it
sort.Slice(out, func(i, j int) bool {
return out[i].Type() < out[j].Type()
})
return out
})
func TestParseRedisURI(t *testing.T) {
tests := []struct {
uri string
want RedisConnOpt
}{
{
"redis://localhost:6379",
RedisClientOpt{Addr: "localhost:6379"},
},
{
"rediss://localhost:6379",
RedisClientOpt{Addr: "localhost:6379", TLSConfig: &tls.Config{ServerName: "localhost"}},
},
{
"redis://localhost:6379/3",
RedisClientOpt{Addr: "localhost:6379", DB: 3},
},
{
"redis://:mypassword@localhost:6379",
RedisClientOpt{Addr: "localhost:6379", Password: "mypassword"},
},
{
"redis://:mypassword@127.0.0.1:6379/11",
RedisClientOpt{Addr: "127.0.0.1:6379", Password: "mypassword", DB: 11},
},
{
"redis-socket:///var/run/redis/redis.sock",
RedisClientOpt{Network: "unix", Addr: "/var/run/redis/redis.sock"},
},
{
"redis-socket://:mypassword@/var/run/redis/redis.sock",
RedisClientOpt{Network: "unix", Addr: "/var/run/redis/redis.sock", Password: "mypassword"},
},
{
"redis-socket:///var/run/redis/redis.sock?db=7",
RedisClientOpt{Network: "unix", Addr: "/var/run/redis/redis.sock", DB: 7},
},
{
"redis-socket://:mypassword@/var/run/redis/redis.sock?db=12",
RedisClientOpt{Network: "unix", Addr: "/var/run/redis/redis.sock", Password: "mypassword", DB: 12},
},
{
"redis-sentinel://localhost:5000,localhost:5001,localhost:5002?master=mymaster",
RedisFailoverClientOpt{
MasterName: "mymaster",
SentinelAddrs: []string{"localhost:5000", "localhost:5001", "localhost:5002"},
},
},
{
"redis-sentinel://:mypassword@localhost:5000,localhost:5001,localhost:5002?master=mymaster",
RedisFailoverClientOpt{
MasterName: "mymaster",
SentinelAddrs: []string{"localhost:5000", "localhost:5001", "localhost:5002"},
SentinelPassword: "mypassword",
},
},
}
for _, tc := range tests {
got, err := ParseRedisURI(tc.uri)
if err != nil {
t.Errorf("ParseRedisURI(%q) returned an error: %v", tc.uri, err)
continue
}
if diff := cmp.Diff(tc.want, got, cmpopts.IgnoreUnexported(tls.Config{})); diff != "" {
t.Errorf("ParseRedisURI(%q) = %+v, want %+v\n(-want,+got)\n%s", tc.uri, got, tc.want, diff)
}
}
}
func TestParseRedisURIErrors(t *testing.T) {
tests := []struct {
desc string
uri string
}{
{
"unsupported scheme",
"rdb://localhost:6379",
},
{
"missing scheme",
"localhost:6379",
},
{
"multiple db numbers",
"redis://localhost:6379/1,2,3",
},
{
"missing path for socket connection",
"redis-socket://?db=one",
},
{
"non integer for db numbers for socket",
"redis-socket:///some/path/to/redis?db=one",
},
}
for _, tc := range tests {
_, err := ParseRedisURI(tc.uri)
if err == nil {
t.Errorf("%s: ParseRedisURI(%q) succeeded for malformed input, want error",
tc.desc, tc.uri)
}
}
}
================================================
FILE: benchmark_test.go
================================================
// Copyright 2020 Kentaro Hibino. All rights reserved.
// Use of this source code is governed by a MIT license
// that can be found in the LICENSE file.
package asynq
import (
"context"
"encoding/json"
"fmt"
"sync"
"testing"
"time"
h "github.com/hibiken/asynq/internal/testutil"
)
// Creates a new task of type "task<n>" with payload {"data": n}.
func makeTask(n int) *Task {
b, err := json.Marshal(map[string]int{"data": n})
if err != nil {
panic(err)
}
return NewTask(fmt.Sprintf("task%d", n), b)
}
// Simple E2E Benchmark testing with no scheduled tasks and retries.
func BenchmarkEndToEndSimple(b *testing.B) {
const count = 100000
for n := 0; n < b.N; n++ {
b.StopTimer() // begin setup
setup(b)
redis := getRedisConnOpt(b)
client := NewClient(redis)
srv := NewServer(redis, Config{
Concurrency: 10,
RetryDelayFunc: func(n int, err error, t *Task) time.Duration {
return time.Second
},
LogLevel: testLogLevel,
})
// Create a bunch of tasks
for i := 0; i < count; i++ {
if _, err := client.Enqueue(makeTask(i)); err != nil {
b.Fatalf("could not enqueue a task: %v", err)
}
}
client.Close()
var wg sync.WaitGroup
wg.Add(count)
handler := func(ctx context.Context, t *Task) error {
wg.Done()
return nil
}
b.StartTimer() // end setup
_ = srv.Start(HandlerFunc(handler))
wg.Wait()
b.StopTimer() // begin teardown
srv.Stop()
b.StartTimer() // end teardown
}
}
// E2E benchmark with scheduled tasks and retries.
func BenchmarkEndToEnd(b *testing.B) {
const count = 100000
for n := 0; n < b.N; n++ {
b.StopTimer() // begin setup
setup(b)
redis := getRedisConnOpt(b)
client := NewClient(redis)
srv := NewServer(redis, Config{
Concurrency: 10,
RetryDelayFunc: func(n int, err error, t *Task) time.Duration {
return time.Second
},
LogLevel: testLogLevel,
})
// Create a bunch of tasks
for i := 0; i < count; i++ {
if _, err := client.Enqueue(makeTask(i)); err != nil {
b.Fatalf("could not enqueue a task: %v", err)
}
}
for i := 0; i < count; i++ {
if _, err := client.Enqueue(makeTask(i), ProcessIn(1*time.Second)); err != nil {
b.Fatalf("could not enqueue a task: %v", err)
}
}
client.Close()
var wg sync.WaitGroup
wg.Add(count * 2)
handler := func(ctx context.Context, t *Task) error {
var p map[string]int
if err := json.Unmarshal(t.Payload(), &p); err != nil {
b.Logf("internal error: %v", err)
}
n, ok := p["data"]
if !ok {
n = 1
b.Logf("internal error: could not get data from payload")
}
retried, ok := GetRetryCount(ctx)
if !ok {
b.Logf("internal error: could not get retry count from context")
}
// Fail 1% of tasks for the first attempt.
if retried == 0 && n%100 == 0 {
return fmt.Errorf(":(")
}
wg.Done()
return nil
}
b.StartTimer() // end setup
_ = srv.Start(HandlerFunc(handler))
wg.Wait()
b.StopTimer() // begin teardown
srv.Stop()
b.StartTimer() // end teardown
}
}
// Simple E2E Benchmark testing with no scheduled tasks and retries with multiple queues.
func BenchmarkEndToEndMultipleQueues(b *testing.B) {
// number of tasks to create for each queue
const (
highCount = 20000
defaultCount = 20000
lowCount = 20000
)
for n := 0; n < b.N; n++ {
b.StopTimer() // begin setup
setup(b)
redis := getRedisConnOpt(b)
client := NewClient(redis)
srv := NewServer(redis, Config{
Concurrency: 10,
Queues: map[string]int{
"high": 6,
"default": 3,
"low": 1,
},
LogLevel: testLogLevel,
})
// Create a bunch of tasks
for i := 0; i < highCount; i++ {
if _, err := client.Enqueue(makeTask(i), Queue("high")); err != nil {
b.Fatalf("could not enqueue a task: %v", err)
}
}
for i := 0; i < defaultCount; i++ {
if _, err := client.Enqueue(makeTask(i)); err != nil {
b.Fatalf("could not enqueue a task: %v", err)
}
}
for i := 0; i < lowCount; i++ {
if _, err := client.Enqueue(makeTask(i), Queue("low")); err != nil {
b.Fatalf("could not enqueue a task: %v", err)
}
}
client.Close()
var wg sync.WaitGroup
wg.Add(highCount + defaultCount + lowCount)
handler := func(ctx context.Context, t *Task) error {
wg.Done()
return nil
}
b.StartTimer() // end setup
_ = srv.Start(HandlerFunc(handler))
wg.Wait()
b.StopTimer() // begin teardown
srv.Stop()
b.StartTimer() // end teardown
}
}
// E2E benchmark to check client enqueue operation performs correctly,
// while server is busy processing tasks.
func BenchmarkClientWhileServerRunning(b *testing.B) {
const count = 10000
for n := 0; n < b.N; n++ {
b.StopTimer() // begin setup
setup(b)
redis := getRedisConnOpt(b)
client := NewClient(redis)
srv := NewServer(redis, Config{
Concurrency: 10,
RetryDelayFunc: func(n int, err error, t *Task) time.Duration {
return time.Second
},
LogLevel: testLogLevel,
})
// Enqueue 10,000 tasks.
for i := 0; i < count; i++ {
if _, err := client.Enqueue(makeTask(i)); err != nil {
b.Fatalf("could not enqueue a task: %v", err)
}
}
// Schedule 10,000 tasks.
for i := 0; i < count; i++ {
if _, err := client.Enqueue(makeTask(i), ProcessIn(1*time.Second)); err != nil {
b.Fatalf("could not enqueue a task: %v", err)
}
}
handler := func(ctx context.Context, t *Task) error {
return nil
}
_ = srv.Start(HandlerFunc(handler))
b.StartTimer() // end setup
b.Log("Starting enqueueing")
enqueued := 0
for enqueued < 100000 {
t := NewTask(fmt.Sprintf("enqueued%d", enqueued), h.JSON(map[string]interface{}{"data": enqueued}))
if _, err := client.Enqueue(t); err != nil {
b.Logf("could not enqueue task %d: %v", enqueued, err)
continue
}
enqueued++
}
b.Logf("Finished enqueueing %d tasks", enqueued)
b.StopTimer() // begin teardown
srv.Stop()
client.Close()
b.StartTimer() // end teardown
}
}
================================================
FILE: client.go
================================================
// Copyright 2020 Kentaro Hibino. All rights reserved.
// Use of this source code is governed by a MIT license
// that can be found in the LICENSE file.
package asynq
import (
"context"
"fmt"
"strings"
"time"
"github.com/google/uuid"
"github.com/hibiken/asynq/internal/base"
"github.com/hibiken/asynq/internal/errors"
"github.com/hibiken/asynq/internal/rdb"
"github.com/redis/go-redis/v9"
)
// A Client is responsible for scheduling tasks.
//
// A Client is used to register tasks that should be processed
// immediately or some time in the future.
//
// Clients are safe for concurrent use by multiple goroutines.
type Client struct {
broker base.Broker
// When a Client has been created with an existing Redis connection, we do
// not want to close it.
sharedConnection bool
}
// NewClient returns a new Client instance given a redis connection option.
func NewClient(r RedisConnOpt) *Client {
redisClient, ok := r.MakeRedisClient().(redis.UniversalClient)
if !ok {
panic(fmt.Sprintf("asynq: unsupported RedisConnOpt type %T", r))
}
client := NewClientFromRedisClient(redisClient)
client.sharedConnection = false
return client
}
// NewClientFromRedisClient returns a new instance of Client given a redis.UniversalClient
// Warning: The underlying redis connection pool will not be closed by Asynq, you are responsible for closing it.
func NewClientFromRedisClient(c redis.UniversalClient) *Client {
return &Client{broker: rdb.NewRDB(c), sharedConnection: true}
}
type OptionType int
const (
MaxRetryOpt OptionType = iota
QueueOpt
TimeoutOpt
DeadlineOpt
UniqueOpt
ProcessAtOpt
ProcessInOpt
TaskIDOpt
RetentionOpt
GroupOpt
)
// Option specifies the task processing behavior.
type Option interface {
// String returns a string representation of the option.
String() string
// Type describes the type of the option.
Type() OptionType
// Value returns a value used to create this option.
Value() interface{}
}
// Internal option representations.
type (
retryOption int
queueOption string
taskIDOption string
timeoutOption time.Duration
deadlineOption time.Time
uniqueOption time.Duration
processAtOption time.Time
processInOption time.Duration
retentionOption time.Duration
groupOption string
)
// MaxRetry returns an option to specify the max number of times
// the task will be retried.
//
// Negative retry count is treated as zero retry.
func MaxRetry(n int) Option {
if n < 0 {
n = 0
}
return retryOption(n)
}
func (n retryOption) String() string { return fmt.Sprintf("MaxRetry(%d)", int(n)) }
func (n retryOption) Type() OptionType { return MaxRetryOpt }
func (n retryOption) Value() interface{} { return int(n) }
// Queue returns an option to specify the queue to enqueue the task into.
func Queue(name string) Option {
return queueOption(name)
}
func (name queueOption) String() string { return fmt.Sprintf("Queue(%q)", string(name)) }
func (name queueOption) Type() OptionType { return QueueOpt }
func (name queueOption) Value() interface{} { return string(name) }
// TaskID returns an option to specify the task ID.
func TaskID(id string) Option {
return taskIDOption(id)
}
func (id taskIDOption) String() string { return fmt.Sprintf("TaskID(%q)", string(id)) }
func (id taskIDOption) Type() OptionType { return TaskIDOpt }
func (id taskIDOption) Value() interface{} { return string(id) }
// Timeout returns an option to specify how long a task may run.
// If the timeout elapses before the Handler returns, then the task
// will be retried.
//
// Zero duration means no limit.
//
// If there's a conflicting Deadline option, whichever comes earliest
// will be used.
func Timeout(d time.Duration) Option {
return timeoutOption(d)
}
func (d timeoutOption) String() string { return fmt.Sprintf("Timeout(%v)", time.Duration(d)) }
func (d timeoutOption) Type() OptionType { return TimeoutOpt }
func (d timeoutOption) Value() interface{} { return time.Duration(d) }
// Deadline returns an option to specify the deadline for the given task.
// If it reaches the deadline before the Handler returns, then the task
// will be retried.
//
// If there's a conflicting Timeout option, whichever comes earliest
// will be used.
func Deadline(t time.Time) Option {
return deadlineOption(t)
}
func (t deadlineOption) String() string {
return fmt.Sprintf("Deadline(%v)", time.Time(t).Format(time.UnixDate))
}
func (t deadlineOption) Type() OptionType { return DeadlineOpt }
func (t deadlineOption) Value() interface{} { return time.Time(t) }
// Unique returns an option to enqueue a task only if the given task is unique.
// Task enqueued with this option is guaranteed to be unique within the given ttl.
// Once the task gets processed successfully or once the TTL has expired,
// another task with the same uniqueness may be enqueued.
// ErrDuplicateTask error is returned when enqueueing a duplicate task.
// TTL duration must be greater than or equal to 1 second.
//
// Uniqueness of a task is based on the following properties:
// - Task Type
// - Task Payload
// - Queue Name
func Unique(ttl time.Duration) Option {
return uniqueOption(ttl)
}
func (ttl uniqueOption) String() string { return fmt.Sprintf("Unique(%v)", time.Duration(ttl)) }
func (ttl uniqueOption) Type() OptionType { return UniqueOpt }
func (ttl uniqueOption) Value() interface{} { return time.Duration(ttl) }
// ProcessAt returns an option to specify when to process the given task.
//
// If there's a conflicting ProcessIn option, the last option passed to Enqueue overrides the others.
func ProcessAt(t time.Time) Option {
return processAtOption(t)
}
func (t processAtOption) String() string {
return fmt.Sprintf("ProcessAt(%v)", time.Time(t).Format(time.UnixDate))
}
func (t processAtOption) Type() OptionType { return ProcessAtOpt }
func (t processAtOption) Value() interface{} { return time.Time(t) }
// ProcessIn returns an option to specify when to process the given task relative to the current time.
//
// If there's a conflicting ProcessAt option, the last option passed to Enqueue overrides the others.
func ProcessIn(d time.Duration) Option {
return processInOption(d)
}
func (d processInOption) String() string { return fmt.Sprintf("ProcessIn(%v)", time.Duration(d)) }
func (d processInOption) Type() OptionType { return ProcessInOpt }
func (d processInOption) Value() interface{} { return time.Duration(d) }
// Retention returns an option to specify the duration of retention period for the task.
// If this option is provided, the task will be stored as a completed task after successful processing.
// A completed task will be deleted after the specified duration elapses.
func Retention(d time.Duration) Option {
return retentionOption(d)
}
func (ttl retentionOption) String() string { return fmt.Sprintf("Retention(%v)", time.Duration(ttl)) }
func (ttl retentionOption) Type() OptionType { return RetentionOpt }
func (ttl retentionOption) Value() interface{} { return time.Duration(ttl) }
// Group returns an option to specify the group used for the task.
// Tasks in a given queue with the same group will be aggregated into one task before passed to Handler.
func Group(name string) Option {
return groupOption(name)
}
func (name groupOption) String() string { return fmt.Sprintf("Group(%q)", string(name)) }
func (name groupOption) Type() OptionType { return GroupOpt }
func (name groupOption) Value() interface{} { return string(name) }
// ErrDuplicateTask indicates that the given task could not be enqueued since it's a duplicate of another task.
//
// ErrDuplicateTask error only applies to tasks enqueued with a Unique option.
var ErrDuplicateTask = errors.New("task already exists")
// ErrTaskIDConflict indicates that the given task could not be enqueued since its task ID already exists.
//
// ErrTaskIDConflict error only applies to tasks enqueued with a TaskID option.
var ErrTaskIDConflict = errors.New("task ID conflicts with another task")
type option struct {
retry int
queue string
taskID string
timeout time.Duration
deadline time.Time
uniqueTTL time.Duration
processAt time.Time
retention time.Duration
group string
}
// composeOptions merges user provided options into the default options
// and returns the composed option.
// It also validates the user provided options and returns an error if any of
// the user provided options fail the validations.
func composeOptions(opts ...Option) (option, error) {
res := option{
retry: defaultMaxRetry,
queue: base.DefaultQueueName,
taskID: uuid.NewString(),
timeout: 0, // do not set to defaultTimeout here
deadline: time.Time{},
processAt: time.Now(),
}
for _, opt := range opts {
switch opt := opt.(type) {
case retryOption:
res.retry = int(opt)
case queueOption:
qname := string(opt)
if err := base.ValidateQueueName(qname); err != nil {
return option{}, err
}
res.queue = qname
case taskIDOption:
id := string(opt)
if isBlank(id) {
return option{}, errors.New("task ID cannot be empty")
}
res.taskID = id
case timeoutOption:
res.timeout = time.Duration(opt)
case deadlineOption:
res.deadline = time.Time(opt)
case uniqueOption:
ttl := time.Duration(opt)
if ttl < 1*time.Second {
return option{}, errors.New("Unique TTL cannot be less than 1s")
}
res.uniqueTTL = ttl
case processAtOption:
res.processAt = time.Time(opt)
case processInOption:
res.processAt = time.Now().Add(time.Duration(opt))
case retentionOption:
res.retention = time.Duration(opt)
case groupOption:
key := string(opt)
if isBlank(key) {
return option{}, errors.New("group key cannot be empty")
}
res.group = key
default:
// ignore unexpected option
}
}
return res, nil
}
// isBlank returns true if the given s is empty or consist of all whitespaces.
func isBlank(s string) bool {
return strings.TrimSpace(s) == ""
}
const (
// Default max retry count used if nothing is specified.
defaultMaxRetry = 25
// Default timeout used if both timeout and deadline are not specified.
defaultTimeout = 30 * time.Minute
)
// Value zero indicates no timeout and no deadline.
var (
noTimeout time.Duration = 0
noDeadline time.Time = time.Unix(0, 0)
)
// Close closes the connection with redis.
func (c *Client) Close() error {
if c.sharedConnection {
return fmt.Errorf("redis connection is shared so the Client can't be closed through asynq")
}
return c.broker.Close()
}
// Enqueue enqueues the given task to a queue.
//
// Enqueue returns TaskInfo and nil error if the task is enqueued successfully, otherwise returns a non-nil error.
//
// The argument opts specifies the behavior of task processing.
// If there are conflicting Option values the last one overrides others.
// Any options provided to NewTask can be overridden by options passed to Enqueue.
// By default, max retry is set to 25 and timeout is set to 30 minutes.
//
// If no ProcessAt or ProcessIn options are provided, the task will be pending immediately.
//
// Enqueue uses context.Background internally; to specify the context, use EnqueueContext.
func (c *Client) Enqueue(task *Task, opts ...Option) (*TaskInfo, error) {
return c.EnqueueContext(context.Background(), task, opts...)
}
// EnqueueContext enqueues the given task to a queue.
//
// EnqueueContext returns TaskInfo and nil error if the task is enqueued successfully, otherwise returns a non-nil error.
//
// The argument opts specifies the behavior of task processing.
// If there are conflicting Option values the last one overrides others.
// Any options provided to NewTask can be overridden by options passed to Enqueue.
// By default, max retry is set to 25 and timeout is set to 30 minutes.
//
// If no ProcessAt or ProcessIn options are provided, the task will be pending immediately.
//
// The first argument context applies to the enqueue operation. To specify task timeout and deadline, use Timeout and Deadline option instead.
func (c *Client) EnqueueContext(ctx context.Context, task *Task, opts ...Option) (*TaskInfo, error) {
if task == nil {
return nil, fmt.Errorf("task cannot be nil")
}
if strings.TrimSpace(task.Type()) == "" {
return nil, fmt.Errorf("task typename cannot be empty")
}
// merge task options with the options provided at enqueue time.
opts = append(task.opts, opts...)
opt, err := composeOptions(opts...)
if err != nil {
return nil, err
}
deadline := noDeadline
if !opt.deadline.IsZero() {
deadline = opt.deadline
}
timeout := noTimeout
if opt.timeout != 0 {
timeout = opt.timeout
}
if deadline.Equal(noDeadline) && timeout == noTimeout {
// If neither deadline nor timeout are set, use default timeout.
timeout = defaultTimeout
}
var uniqueKey string
if opt.uniqueTTL > 0 {
uniqueKey = base.UniqueKey(opt.queue, task.Type(), task.Payload())
}
msg := &base.TaskMessage{
ID: opt.taskID,
Type: task.Type(),
Payload: task.Payload(),
Headers: task.Headers(),
Queue: opt.queue,
Retry: opt.retry,
Deadline: deadline.Unix(),
Timeout: int64(timeout.Seconds()),
UniqueKey: uniqueKey,
GroupKey: opt.group,
Retention: int64(opt.retention.Seconds()),
}
now := time.Now()
var state base.TaskState
if opt.processAt.After(now) {
err = c.schedule(ctx, msg, opt.processAt, opt.uniqueTTL)
state = base.TaskStateScheduled
} else if opt.group != "" {
// Use zero value for processAt since we don't know when the task will be aggregated and processed.
opt.processAt = time.Time{}
err = c.addToGroup(ctx, msg, opt.group, opt.uniqueTTL)
state = base.TaskStateAggregating
} else {
opt.processAt = now
err = c.enqueue(ctx, msg, opt.uniqueTTL)
state = base.TaskStatePending
}
switch {
case errors.Is(err, errors.ErrDuplicateTask):
return nil, fmt.Errorf("%w", ErrDuplicateTask)
case errors.Is(err, errors.ErrTaskIdConflict):
return nil, fmt.Errorf("%w", ErrTaskIDConflict)
case err != nil:
return nil, err
}
return newTaskInfo(msg, state, opt.processAt, nil), nil
}
// Ping performs a ping against the redis connection.
func (c *Client) Ping() error {
return c.broker.Ping()
}
func (c *Client) enqueue(ctx context.Context, msg *base.TaskMessage, uniqueTTL time.Duration) error {
if uniqueTTL > 0 {
return c.broker.EnqueueUnique(ctx, msg, uniqueTTL)
}
return c.broker.Enqueue(ctx, msg)
}
func (c *Client) schedule(ctx context.Context, msg *base.TaskMessage, t time.Time, uniqueTTL time.Duration) error {
if uniqueTTL > 0 {
ttl := time.Until(t.Add(uniqueTTL))
return c.broker.ScheduleUnique(ctx, msg, t, ttl)
}
return c.broker.Schedule(ctx, msg, t)
}
func (c *Client) addToGroup(ctx context.Context, msg *base.TaskMessage, group string, uniqueTTL time.Duration) error {
if uniqueTTL > 0 {
return c.broker.AddToGroupUnique(ctx, msg, group, uniqueTTL)
}
return c.broker.AddToGroup(ctx, msg, group)
}
================================================
FILE: client_test.go
================================================
// Copyright 2020 Kentaro Hibino. All rights reserved.
// Use of this source code is governed by a MIT license
// that can be found in the LICENSE file.
package asynq
import (
"context"
"errors"
"testing"
"time"
"github.com/google/go-cmp/cmp"
"github.com/google/go-cmp/cmp/cmpopts"
"github.com/hibiken/asynq/internal/base"
h "github.com/hibiken/asynq/internal/testutil"
"github.com/redis/go-redis/v9"
)
func TestClientEnqueueWithProcessAtOption(t *testing.T) {
r := setup(t)
client := NewClient(getRedisConnOpt(t))
defer client.Close()
task := NewTask("send_email", h.JSON(map[string]interface{}{"to": "customer@gmail.com", "from": "merchant@example.com"}))
var (
now = time.Now()
oneHourLater = now.Add(time.Hour)
)
tests := []struct {
desc string
task *Task
processAt time.Time // value for ProcessAt option
opts []Option // other options
wantInfo *TaskInfo
wantPending map[string][]*base.TaskMessage
wantScheduled map[string][]base.Z
}{
{
desc: "Process task immediately",
task: task,
processAt: now,
opts: []Option{},
wantInfo: &TaskInfo{
Queue: "default",
Type: task.Type(),
Payload: task.Payload(),
State: TaskStatePending,
MaxRetry: defaultMaxRetry,
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: defaultTimeout,
Deadline: time.Time{},
NextProcessAt: now,
},
wantPending: map[string][]*base.TaskMessage{
"default": {
{
Type: task.Type(),
Payload: task.Payload(),
Retry: defaultMaxRetry,
Queue: "default",
Timeout: int64(defaultTimeout.Seconds()),
Deadline: noDeadline.Unix(),
},
},
},
wantScheduled: map[string][]base.Z{
"default": {},
},
},
{
desc: "Schedule task to be processed in the future",
task: task,
processAt: oneHourLater,
opts: []Option{},
wantInfo: &TaskInfo{
Queue: "default",
Type: task.Type(),
Payload: task.Payload(),
State: TaskStateScheduled,
MaxRetry: defaultMaxRetry,
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: defaultTimeout,
Deadline: time.Time{},
NextProcessAt: oneHourLater,
},
wantPending: map[string][]*base.TaskMessage{
"default": {},
},
wantScheduled: map[string][]base.Z{
"default": {
{
Message: &base.TaskMessage{
Type: task.Type(),
Payload: task.Payload(),
Retry: defaultMaxRetry,
Queue: "default",
Timeout: int64(defaultTimeout.Seconds()),
Deadline: noDeadline.Unix(),
},
Score: oneHourLater.Unix(),
},
},
},
},
}
for _, tc := range tests {
h.FlushDB(t, r) // clean up db before each test case.
opts := append(tc.opts, ProcessAt(tc.processAt))
gotInfo, err := client.Enqueue(tc.task, opts...)
if err != nil {
t.Error(err)
continue
}
cmpOptions := []cmp.Option{
cmpopts.IgnoreFields(TaskInfo{}, "ID"),
cmpopts.EquateApproxTime(500 * time.Millisecond),
}
if diff := cmp.Diff(tc.wantInfo, gotInfo, cmpOptions...); diff != "" {
t.Errorf("%s;\nEnqueue(task, ProcessAt(%v)) returned %v, want %v; (-want,+got)\n%s",
tc.desc, tc.processAt, gotInfo, tc.wantInfo, diff)
}
for qname, want := range tc.wantPending {
gotPending := h.GetPendingMessages(t, r, qname)
if diff := cmp.Diff(want, gotPending, h.IgnoreIDOpt, cmpopts.EquateEmpty()); diff != "" {
t.Errorf("%s;\nmismatch found in %q; (-want,+got)\n%s", tc.desc, base.PendingKey(qname), diff)
}
}
for qname, want := range tc.wantScheduled {
gotScheduled := h.GetScheduledEntries(t, r, qname)
if diff := cmp.Diff(want, gotScheduled, h.IgnoreIDOpt, cmpopts.EquateEmpty()); diff != "" {
t.Errorf("%s;\nmismatch found in %q; (-want,+got)\n%s", tc.desc, base.ScheduledKey(qname), diff)
}
}
}
}
func testClientEnqueue(t *testing.T, client *Client, r redis.UniversalClient) {
task := NewTask("send_email", h.JSON(map[string]interface{}{"to": "customer@gmail.com", "from": "merchant@example.com"}))
now := time.Now()
tests := []struct {
desc string
task *Task
opts []Option
wantInfo *TaskInfo
wantPending map[string][]*base.TaskMessage
}{
{
desc: "Process task immediately with a custom retry count",
task: task,
opts: []Option{
MaxRetry(3),
},
wantInfo: &TaskInfo{
Queue: "default",
Type: task.Type(),
Payload: task.Payload(),
State: TaskStatePending,
MaxRetry: 3,
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: defaultTimeout,
Deadline: time.Time{},
NextProcessAt: now,
},
wantPending: map[string][]*base.TaskMessage{
"default": {
{
Type: task.Type(),
Payload: task.Payload(),
Retry: 3,
Queue: "default",
Timeout: int64(defaultTimeout.Seconds()),
Deadline: noDeadline.Unix(),
},
},
},
},
{
desc: "Negative retry count",
task: task,
opts: []Option{
MaxRetry(-2),
},
wantInfo: &TaskInfo{
Queue: "default",
Type: task.Type(),
Payload: task.Payload(),
State: TaskStatePending,
MaxRetry: 0, // Retry count should be set to zero
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: defaultTimeout,
Deadline: time.Time{},
NextProcessAt: now,
},
wantPending: map[string][]*base.TaskMessage{
"default": {
{
Type: task.Type(),
Payload: task.Payload(),
Retry: 0, // Retry count should be set to zero
Queue: "default",
Timeout: int64(defaultTimeout.Seconds()),
Deadline: noDeadline.Unix(),
},
},
},
},
{
desc: "Conflicting options",
task: task,
opts: []Option{
MaxRetry(2),
MaxRetry(10),
},
wantInfo: &TaskInfo{
Queue: "default",
Type: task.Type(),
Payload: task.Payload(),
State: TaskStatePending,
MaxRetry: 10, // Last option takes precedence
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: defaultTimeout,
Deadline: time.Time{},
NextProcessAt: now,
},
wantPending: map[string][]*base.TaskMessage{
"default": {
{
Type: task.Type(),
Payload: task.Payload(),
Retry: 10, // Last option takes precedence
Queue: "default",
Timeout: int64(defaultTimeout.Seconds()),
Deadline: noDeadline.Unix(),
},
},
},
},
{
desc: "With queue option",
task: task,
opts: []Option{
Queue("custom"),
},
wantInfo: &TaskInfo{
Queue: "custom",
Type: task.Type(),
Payload: task.Payload(),
State: TaskStatePending,
MaxRetry: defaultMaxRetry,
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: defaultTimeout,
Deadline: time.Time{},
NextProcessAt: now,
},
wantPending: map[string][]*base.TaskMessage{
"custom": {
{
Type: task.Type(),
Payload: task.Payload(),
Retry: defaultMaxRetry,
Queue: "custom",
Timeout: int64(defaultTimeout.Seconds()),
Deadline: noDeadline.Unix(),
},
},
},
},
{
desc: "Queue option should be case sensitive",
task: task,
opts: []Option{
Queue("MyQueue"),
},
wantInfo: &TaskInfo{
Queue: "MyQueue",
Type: task.Type(),
Payload: task.Payload(),
State: TaskStatePending,
MaxRetry: defaultMaxRetry,
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: defaultTimeout,
Deadline: time.Time{},
NextProcessAt: now,
},
wantPending: map[string][]*base.TaskMessage{
"MyQueue": {
{
Type: task.Type(),
Payload: task.Payload(),
Retry: defaultMaxRetry,
Queue: "MyQueue",
Timeout: int64(defaultTimeout.Seconds()),
Deadline: noDeadline.Unix(),
},
},
},
},
{
desc: "With timeout option",
task: task,
opts: []Option{
Timeout(20 * time.Second),
},
wantInfo: &TaskInfo{
Queue: "default",
Type: task.Type(),
Payload: task.Payload(),
State: TaskStatePending,
MaxRetry: defaultMaxRetry,
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: 20 * time.Second,
Deadline: time.Time{},
NextProcessAt: now,
},
wantPending: map[string][]*base.TaskMessage{
"default": {
{
Type: task.Type(),
Payload: task.Payload(),
Retry: defaultMaxRetry,
Queue: "default",
Timeout: 20,
Deadline: noDeadline.Unix(),
},
},
},
},
{
desc: "With deadline option",
task: task,
opts: []Option{
Deadline(time.Date(2020, time.June, 24, 0, 0, 0, 0, time.UTC)),
},
wantInfo: &TaskInfo{
Queue: "default",
Type: task.Type(),
Payload: task.Payload(),
State: TaskStatePending,
MaxRetry: defaultMaxRetry,
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: noTimeout,
Deadline: time.Date(2020, time.June, 24, 0, 0, 0, 0, time.UTC),
NextProcessAt: now,
},
wantPending: map[string][]*base.TaskMessage{
"default": {
{
Type: task.Type(),
Payload: task.Payload(),
Retry: defaultMaxRetry,
Queue: "default",
Timeout: int64(noTimeout.Seconds()),
Deadline: time.Date(2020, time.June, 24, 0, 0, 0, 0, time.UTC).Unix(),
},
},
},
},
{
desc: "With both deadline and timeout options",
task: task,
opts: []Option{
Timeout(20 * time.Second),
Deadline(time.Date(2020, time.June, 24, 0, 0, 0, 0, time.UTC)),
},
wantInfo: &TaskInfo{
Queue: "default",
Type: task.Type(),
Payload: task.Payload(),
State: TaskStatePending,
MaxRetry: defaultMaxRetry,
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: 20 * time.Second,
Deadline: time.Date(2020, time.June, 24, 0, 0, 0, 0, time.UTC),
NextProcessAt: now,
},
wantPending: map[string][]*base.TaskMessage{
"default": {
{
Type: task.Type(),
Payload: task.Payload(),
Retry: defaultMaxRetry,
Queue: "default",
Timeout: 20,
Deadline: time.Date(2020, time.June, 24, 0, 0, 0, 0, time.UTC).Unix(),
},
},
},
},
{
desc: "With Retention option",
task: task,
opts: []Option{
Retention(24 * time.Hour),
},
wantInfo: &TaskInfo{
Queue: "default",
Type: task.Type(),
Payload: task.Payload(),
State: TaskStatePending,
MaxRetry: defaultMaxRetry,
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: defaultTimeout,
Deadline: time.Time{},
NextProcessAt: now,
Retention: 24 * time.Hour,
},
wantPending: map[string][]*base.TaskMessage{
"default": {
{
Type: task.Type(),
Payload: task.Payload(),
Retry: defaultMaxRetry,
Queue: "default",
Timeout: int64(defaultTimeout.Seconds()),
Deadline: noDeadline.Unix(),
Retention: int64((24 * time.Hour).Seconds()),
},
},
},
},
}
for _, tc := range tests {
h.FlushDB(t, r) // clean up db before each test case.
gotInfo, err := client.Enqueue(tc.task, tc.opts...)
if err != nil {
t.Error(err)
continue
}
cmpOptions := []cmp.Option{
cmpopts.IgnoreFields(TaskInfo{}, "ID"),
cmpopts.EquateApproxTime(500 * time.Millisecond),
}
if diff := cmp.Diff(tc.wantInfo, gotInfo, cmpOptions...); diff != "" {
t.Errorf("%s;\nEnqueue(task) returned %v, want %v; (-want,+got)\n%s",
tc.desc, gotInfo, tc.wantInfo, diff)
}
for qname, want := range tc.wantPending {
got := h.GetPendingMessages(t, r, qname)
if diff := cmp.Diff(want, got, h.IgnoreIDOpt); diff != "" {
t.Errorf("%s;\nmismatch found in %q; (-want,+got)\n%s", tc.desc, base.PendingKey(qname), diff)
}
}
}
}
func TestClientEnqueue(t *testing.T) {
r := setup(t)
client := NewClient(getRedisConnOpt(t))
defer client.Close()
testClientEnqueue(t, client, r)
}
func TestClientFromRedisClientEnqueue(t *testing.T) {
r := setup(t)
redisClient := getRedisConnOpt(t).MakeRedisClient().(redis.UniversalClient)
client := NewClientFromRedisClient(redisClient)
testClientEnqueue(t, client, r)
err := client.Close()
if err == nil {
t.Error("client.Close() should have failed because of a shared client but it didn't")
}
}
func TestClientEnqueueWithGroupOption(t *testing.T) {
r := setup(t)
client := NewClient(getRedisConnOpt(t))
defer client.Close()
task := NewTask("mytask", []byte("foo"))
now := time.Now()
tests := []struct {
desc string
task *Task
opts []Option
wantInfo *TaskInfo
wantPending map[string][]*base.TaskMessage
wantGroups map[string]map[string][]base.Z // map queue name to a set of groups
wantScheduled map[string][]base.Z
}{
{
desc: "With only Group option",
task: task,
opts: []Option{
Group("mygroup"),
},
wantInfo: &TaskInfo{
Queue: "default",
Group: "mygroup",
Type: task.Type(),
Payload: task.Payload(),
State: TaskStateAggregating,
MaxRetry: defaultMaxRetry,
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: defaultTimeout,
Deadline: time.Time{},
NextProcessAt: time.Time{},
},
wantPending: map[string][]*base.TaskMessage{
"default": {}, // should not be pending
},
wantGroups: map[string]map[string][]base.Z{
"default": {
"mygroup": {
{
Message: &base.TaskMessage{
Type: task.Type(),
Payload: task.Payload(),
Retry: defaultMaxRetry,
Queue: "default",
Timeout: int64(defaultTimeout.Seconds()),
Deadline: noDeadline.Unix(),
GroupKey: "mygroup",
},
Score: now.Unix(),
},
},
},
},
wantScheduled: map[string][]base.Z{
"default": {},
},
},
{
desc: "With Group and ProcessAt options",
task: task,
opts: []Option{
Group("mygroup"),
ProcessAt(now.Add(30 * time.Minute)),
},
wantInfo: &TaskInfo{
Queue: "default",
Group: "mygroup",
Type: task.Type(),
Payload: task.Payload(),
State: TaskStateScheduled,
MaxRetry: defaultMaxRetry,
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: defaultTimeout,
Deadline: time.Time{},
NextProcessAt: now.Add(30 * time.Minute),
},
wantPending: map[string][]*base.TaskMessage{
"default": {}, // should not be pending
},
wantGroups: map[string]map[string][]base.Z{
"default": {
"mygroup": {}, // should not be added to the group yet
},
},
wantScheduled: map[string][]base.Z{
"default": {
{
Message: &base.TaskMessage{
Type: task.Type(),
Payload: task.Payload(),
Retry: defaultMaxRetry,
Queue: "default",
Timeout: int64(defaultTimeout.Seconds()),
Deadline: noDeadline.Unix(),
GroupKey: "mygroup",
},
Score: now.Add(30 * time.Minute).Unix(),
},
},
},
},
}
for _, tc := range tests {
h.FlushDB(t, r) // clean up db before each test case.
gotInfo, err := client.Enqueue(tc.task, tc.opts...)
if err != nil {
t.Error(err)
continue
}
cmpOptions := []cmp.Option{
cmpopts.IgnoreFields(TaskInfo{}, "ID"),
cmpopts.EquateApproxTime(500 * time.Millisecond),
}
if diff := cmp.Diff(tc.wantInfo, gotInfo, cmpOptions...); diff != "" {
t.Errorf("%s;\nEnqueue(task) returned %v, want %v; (-want,+got)\n%s",
tc.desc, gotInfo, tc.wantInfo, diff)
}
for qname, want := range tc.wantPending {
got := h.GetPendingMessages(t, r, qname)
if diff := cmp.Diff(want, got, h.IgnoreIDOpt, cmpopts.EquateEmpty()); diff != "" {
t.Errorf("%s;\nmismatch found in %q; (-want,+got)\n%s", tc.desc, base.PendingKey(qname), diff)
}
}
for qname, groups := range tc.wantGroups {
for groupKey, want := range groups {
got := h.GetGroupEntries(t, r, qname, groupKey)
if diff := cmp.Diff(want, got, h.IgnoreIDOpt, cmpopts.EquateEmpty()); diff != "" {
t.Errorf("%s;\nmismatch found in %q; (-want,+got)\n%s", tc.desc, base.GroupKey(qname, groupKey), diff)
}
}
}
for qname, want := range tc.wantScheduled {
gotScheduled := h.GetScheduledEntries(t, r, qname)
if diff := cmp.Diff(want, gotScheduled, h.IgnoreIDOpt, cmpopts.EquateEmpty()); diff != "" {
t.Errorf("%s;\nmismatch found in %q; (-want,+got)\n%s", tc.desc, base.ScheduledKey(qname), diff)
}
}
}
}
func TestClientEnqueueWithTaskIDOption(t *testing.T) {
r := setup(t)
client := NewClient(getRedisConnOpt(t))
defer client.Close()
task := NewTask("send_email", nil)
now := time.Now()
tests := []struct {
desc string
task *Task
opts []Option
wantInfo *TaskInfo
wantPending map[string][]*base.TaskMessage
}{
{
desc: "With a valid TaskID option",
task: task,
opts: []Option{
TaskID("custom_id"),
},
wantInfo: &TaskInfo{
ID: "custom_id",
Queue: "default",
Type: task.Type(),
Payload: task.Payload(),
State: TaskStatePending,
MaxRetry: defaultMaxRetry,
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: defaultTimeout,
Deadline: time.Time{},
NextProcessAt: now,
},
wantPending: map[string][]*base.TaskMessage{
"default": {
{
ID: "custom_id",
Type: task.Type(),
Payload: task.Payload(),
Retry: defaultMaxRetry,
Queue: "default",
Timeout: int64(defaultTimeout.Seconds()),
Deadline: noDeadline.Unix(),
},
},
},
},
}
for _, tc := range tests {
h.FlushDB(t, r) // clean up db before each test case.
gotInfo, err := client.Enqueue(tc.task, tc.opts...)
if err != nil {
t.Errorf("got non-nil error %v, want nil", err)
continue
}
cmpOptions := []cmp.Option{
cmpopts.EquateApproxTime(500 * time.Millisecond),
}
if diff := cmp.Diff(tc.wantInfo, gotInfo, cmpOptions...); diff != "" {
t.Errorf("%s;\nEnqueue(task) returned %v, want %v; (-want,+got)\n%s",
tc.desc, gotInfo, tc.wantInfo, diff)
}
for qname, want := range tc.wantPending {
got := h.GetPendingMessages(t, r, qname)
if diff := cmp.Diff(want, got); diff != "" {
t.Errorf("%s;\nmismatch found in %q; (-want,+got)\n%s", tc.desc, base.PendingKey(qname), diff)
}
}
}
}
func TestClientEnqueueWithConflictingTaskID(t *testing.T) {
setup(t)
client := NewClient(getRedisConnOpt(t))
defer client.Close()
const taskID = "custom_id"
task := NewTask("foo", nil)
if _, err := client.Enqueue(task, TaskID(taskID)); err != nil {
t.Fatalf("First task: Enqueue failed: %v", err)
}
_, err := client.Enqueue(task, TaskID(taskID))
if !errors.Is(err, ErrTaskIDConflict) {
t.Errorf("Second task: Enqueue returned %v, want %v", err, ErrTaskIDConflict)
}
}
func TestClientEnqueueWithProcessInOption(t *testing.T) {
r := setup(t)
client := NewClient(getRedisConnOpt(t))
defer client.Close()
task := NewTask("send_email", h.JSON(map[string]interface{}{"to": "customer@gmail.com", "from": "merchant@example.com"}))
now := time.Now()
tests := []struct {
desc string
task *Task
delay time.Duration // value for ProcessIn option
opts []Option // other options
wantInfo *TaskInfo
wantPending map[string][]*base.TaskMessage
wantScheduled map[string][]base.Z
}{
{
desc: "schedule a task to be processed in one hour",
task: task,
delay: 1 * time.Hour,
opts: []Option{},
wantInfo: &TaskInfo{
Queue: "default",
Type: task.Type(),
Payload: task.Payload(),
State: TaskStateScheduled,
MaxRetry: defaultMaxRetry,
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: defaultTimeout,
Deadline: time.Time{},
NextProcessAt: time.Now().Add(1 * time.Hour),
},
wantPending: map[string][]*base.TaskMessage{
"default": {},
},
wantScheduled: map[string][]base.Z{
"default": {
{
Message: &base.TaskMessage{
Type: task.Type(),
Payload: task.Payload(),
Retry: defaultMaxRetry,
Queue: "default",
Timeout: int64(defaultTimeout.Seconds()),
Deadline: noDeadline.Unix(),
},
Score: time.Now().Add(time.Hour).Unix(),
},
},
},
},
{
desc: "Zero delay",
task: task,
delay: 0,
opts: []Option{},
wantInfo: &TaskInfo{
Queue: "default",
Type: task.Type(),
Payload: task.Payload(),
State: TaskStatePending,
MaxRetry: defaultMaxRetry,
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: defaultTimeout,
Deadline: time.Time{},
NextProcessAt: now,
},
wantPending: map[string][]*base.TaskMessage{
"default": {
{
Type: task.Type(),
Payload: task.Payload(),
Retry: defaultMaxRetry,
Queue: "default",
Timeout: int64(defaultTimeout.Seconds()),
Deadline: noDeadline.Unix(),
},
},
},
wantScheduled: map[string][]base.Z{
"default": {},
},
},
}
for _, tc := range tests {
h.FlushDB(t, r) // clean up db before each test case.
opts := append(tc.opts, ProcessIn(tc.delay))
gotInfo, err := client.Enqueue(tc.task, opts...)
if err != nil {
t.Error(err)
continue
}
cmpOptions := []cmp.Option{
cmpopts.IgnoreFields(TaskInfo{}, "ID"),
cmpopts.EquateApproxTime(500 * time.Millisecond),
}
if diff := cmp.Diff(tc.wantInfo, gotInfo, cmpOptions...); diff != "" {
t.Errorf("%s;\nEnqueue(task, ProcessIn(%v)) returned %v, want %v; (-want,+got)\n%s",
tc.desc, tc.delay, gotInfo, tc.wantInfo, diff)
}
for qname, want := range tc.wantPending {
gotPending := h.GetPendingMessages(t, r, qname)
if diff := cmp.Diff(want, gotPending, h.IgnoreIDOpt, cmpopts.EquateEmpty()); diff != "" {
t.Errorf("%s;\nmismatch found in %q; (-want,+got)\n%s", tc.desc, base.PendingKey(qname), diff)
}
}
for qname, want := range tc.wantScheduled {
gotScheduled := h.GetScheduledEntries(t, r, qname)
if diff := cmp.Diff(want, gotScheduled, h.IgnoreIDOpt, cmpopts.EquateEmpty()); diff != "" {
t.Errorf("%s;\nmismatch found in %q; (-want,+got)\n%s", tc.desc, base.ScheduledKey(qname), diff)
}
}
}
}
func TestClientEnqueueError(t *testing.T) {
r := setup(t)
client := NewClient(getRedisConnOpt(t))
defer client.Close()
task := NewTask("send_email", h.JSON(map[string]interface{}{"to": "customer@gmail.com", "from": "merchant@example.com"}))
tests := []struct {
desc string
task *Task
opts []Option
}{
{
desc: "With nil task",
task: nil,
opts: []Option{},
},
{
desc: "With empty queue name",
task: task,
opts: []Option{
Queue(""),
},
},
{
desc: "With empty task typename",
task: NewTask("", h.JSON(map[string]interface{}{})),
opts: []Option{},
},
{
desc: "With blank task typename",
task: NewTask(" ", h.JSON(map[string]interface{}{})),
opts: []Option{},
},
{
desc: "With empty task ID",
task: NewTask("foo", nil),
opts: []Option{TaskID("")},
},
{
desc: "With blank task ID",
task: NewTask("foo", nil),
opts: []Option{TaskID(" ")},
},
{
desc: "With unique option less than 1s",
task: NewTask("foo", nil),
opts: []Option{Unique(300 * time.Millisecond)},
},
}
for _, tc := range tests {
h.FlushDB(t, r)
_, err := client.Enqueue(tc.task, tc.opts...)
if err == nil {
t.Errorf("%s; client.Enqueue(task, opts...) did not return non-nil error", tc.desc)
}
}
}
func TestClientWithDefaultOptions(t *testing.T) {
r := setup(t)
now := time.Now()
tests := []struct {
desc string
defaultOpts []Option // options set at task initialization time
opts []Option // options used at enqueue time.
tasktype string
payload []byte
wantInfo *TaskInfo
queue string // queue that the message should go into.
want *base.TaskMessage
}{
{
desc: "With queue routing option",
defaultOpts: []Option{Queue("feed")},
opts: []Option{},
tasktype: "feed:import",
payload: nil,
wantInfo: &TaskInfo{
Queue: "feed",
Type: "feed:import",
Payload: nil,
State: TaskStatePending,
MaxRetry: defaultMaxRetry,
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: defaultTimeout,
Deadline: time.Time{},
NextProcessAt: now,
},
queue: "feed",
want: &base.TaskMessage{
Type: "feed:import",
Payload: nil,
Retry: defaultMaxRetry,
Queue: "feed",
Timeout: int64(defaultTimeout.Seconds()),
Deadline: noDeadline.Unix(),
},
},
{
desc: "With multiple options",
defaultOpts: []Option{Queue("feed"), MaxRetry(5)},
opts: []Option{},
tasktype: "feed:import",
payload: nil,
wantInfo: &TaskInfo{
Queue: "feed",
Type: "feed:import",
Payload: nil,
State: TaskStatePending,
MaxRetry: 5,
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: defaultTimeout,
Deadline: time.Time{},
NextProcessAt: now,
},
queue: "feed",
want: &base.TaskMessage{
Type: "feed:import",
Payload: nil,
Retry: 5,
Queue: "feed",
Timeout: int64(defaultTimeout.Seconds()),
Deadline: noDeadline.Unix(),
},
},
{
desc: "With overriding options at enqueue time",
defaultOpts: []Option{Queue("feed"), MaxRetry(5)},
opts: []Option{Queue("critical")},
tasktype: "feed:import",
payload: nil,
wantInfo: &TaskInfo{
Queue: "critical",
Type: "feed:import",
Payload: nil,
State: TaskStatePending,
MaxRetry: 5,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: defaultTimeout,
Deadline: time.Time{},
NextProcessAt: now,
},
queue: "critical",
want: &base.TaskMessage{
Type: "feed:import",
Payload: nil,
Retry: 5,
Queue: "critical",
Timeout: int64(defaultTimeout.Seconds()),
Deadline: noDeadline.Unix(),
},
},
}
for _, tc := range tests {
h.FlushDB(t, r)
c := NewClient(getRedisConnOpt(t))
defer c.Close()
task := NewTask(tc.tasktype, tc.payload, tc.defaultOpts...)
gotInfo, err := c.Enqueue(task, tc.opts...)
if err != nil {
t.Fatal(err)
}
cmpOptions := []cmp.Option{
cmpopts.IgnoreFields(TaskInfo{}, "ID"),
cmpopts.EquateApproxTime(500 * time.Millisecond),
}
if diff := cmp.Diff(tc.wantInfo, gotInfo, cmpOptions...); diff != "" {
t.Errorf("%s;\nEnqueue(task, opts...) returned %v, want %v; (-want,+got)\n%s",
tc.desc, gotInfo, tc.wantInfo, diff)
}
pending := h.GetPendingMessages(t, r, tc.queue)
if len(pending) != 1 {
t.Errorf("%s;\nexpected queue %q to have one message; got %d messages in the queue.",
tc.desc, tc.queue, len(pending))
continue
}
got := pending[0]
if diff := cmp.Diff(tc.want, got, h.IgnoreIDOpt); diff != "" {
t.Errorf("%s;\nmismatch found in pending task message; (-want,+got)\n%s",
tc.desc, diff)
}
}
}
func TestClientEnqueueUnique(t *testing.T) {
r := setup(t)
c := NewClient(getRedisConnOpt(t))
defer c.Close()
tests := []struct {
task *Task
ttl time.Duration
}{
{
NewTask("email", h.JSON(map[string]interface{}{"user_id": 123})),
time.Hour,
},
}
for _, tc := range tests {
h.FlushDB(t, r) // clean up db before each test case.
// Enqueue the task first. It should succeed.
_, err := c.Enqueue(tc.task, Unique(tc.ttl))
if err != nil {
t.Fatal(err)
}
gotTTL := r.TTL(context.Background(), base.UniqueKey(base.DefaultQueueName, tc.task.Type(), tc.task.Payload())).Val()
if !cmp.Equal(tc.ttl.Seconds(), gotTTL.Seconds(), cmpopts.EquateApprox(0, 1)) {
t.Errorf("TTL = %v, want %v", gotTTL, tc.ttl)
continue
}
// Enqueue the task again. It should fail.
_, err = c.Enqueue(tc.task, Unique(tc.ttl))
if err == nil {
t.Errorf("Enqueueing %+v did not return an error", tc.task)
continue
}
if !errors.Is(err, ErrDuplicateTask) {
t.Errorf("Enqueueing %+v returned an error that is not ErrDuplicateTask", tc.task)
continue
}
}
}
func TestClientEnqueueUniqueWithProcessInOption(t *testing.T) {
r := setup(t)
c := NewClient(getRedisConnOpt(t))
defer c.Close()
tests := []struct {
task *Task
d time.Duration
ttl time.Duration
}{
{
NewTask("reindex", nil),
time.Hour,
10 * time.Minute,
},
}
for _, tc := range tests {
h.FlushDB(t, r) // clean up db before each test case.
// Enqueue the task first. It should succeed.
_, err := c.Enqueue(tc.task, ProcessIn(tc.d), Unique(tc.ttl))
if err != nil {
t.Fatal(err)
}
gotTTL := r.TTL(context.Background(), base.UniqueKey(base.DefaultQueueName, tc.task.Type(), tc.task.Payload())).Val()
wantTTL := time.Duration(tc.ttl.Seconds()+tc.d.Seconds()) * time.Second
if !cmp.Equal(wantTTL.Seconds(), gotTTL.Seconds(), cmpopts.EquateApprox(0, 1)) {
t.Errorf("TTL = %v, want %v", gotTTL, wantTTL)
continue
}
// Enqueue the task again. It should fail.
_, err = c.Enqueue(tc.task, ProcessIn(tc.d), Unique(tc.ttl))
if err == nil {
t.Errorf("Enqueueing %+v did not return an error", tc.task)
continue
}
if !errors.Is(err, ErrDuplicateTask) {
t.Errorf("Enqueueing %+v returned an error that is not ErrDuplicateTask", tc.task)
continue
}
}
}
func TestClientEnqueueUniqueWithProcessAtOption(t *testing.T) {
r := setup(t)
c := NewClient(getRedisConnOpt(t))
defer c.Close()
tests := []struct {
task *Task
at time.Time
ttl time.Duration
}{
{
NewTask("reindex", nil),
time.Now().Add(time.Hour),
10 * time.Minute,
},
}
for _, tc := range tests {
h.FlushDB(t, r) // clean up db before each test case.
// Enqueue the task first. It should succeed.
_, err := c.Enqueue(tc.task, ProcessAt(tc.at), Unique(tc.ttl))
if err != nil {
t.Fatal(err)
}
gotTTL := r.TTL(context.Background(), base.UniqueKey(base.DefaultQueueName, tc.task.Type(), tc.task.Payload())).Val()
wantTTL := time.Until(tc.at.Add(tc.ttl))
if !cmp.Equal(wantTTL.Seconds(), gotTTL.Seconds(), cmpopts.EquateApprox(0, 1)) {
t.Errorf("TTL = %v, want %v", gotTTL, wantTTL)
continue
}
// Enqueue the task again. It should fail.
_, err = c.Enqueue(tc.task, ProcessAt(tc.at), Unique(tc.ttl))
if err == nil {
t.Errorf("Enqueueing %+v did not return an error", tc.task)
continue
}
if !errors.Is(err, ErrDuplicateTask) {
t.Errorf("Enqueueing %+v returned an error that is not ErrDuplicateTask", tc.task)
continue
}
}
}
func TestClientEnqueueWithHeaders(t *testing.T) {
r := setup(t)
client := NewClient(getRedisConnOpt(t))
defer client.Close()
now := time.Now()
headers := map[string]string{
"user-id": "123",
"request-id": "abc-def-ghi",
"priority": "high",
}
tests := []struct {
desc string
task *Task
opts []Option
wantInfo *TaskInfo
wantPending map[string][]*base.TaskMessage
}{
{
desc: "Task with headers",
task: NewTaskWithHeaders("send_email", h.JSON(map[string]interface{}{"to": "user@example.com"}), headers),
opts: []Option{},
wantInfo: &TaskInfo{
Queue: "default",
Type: "send_email",
Payload: h.JSON(map[string]interface{}{"to": "user@example.com"}),
Headers: headers,
State: TaskStatePending,
MaxRetry: defaultMaxRetry,
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: defaultTimeout,
Deadline: time.Time{},
NextProcessAt: now,
},
wantPending: map[string][]*base.TaskMessage{
"default": {
{
Type: "send_email",
Payload: h.JSON(map[string]interface{}{"to": "user@example.com"}),
Headers: headers,
Retry: defaultMaxRetry,
Queue: "default",
Timeout: int64(defaultTimeout.Seconds()),
Deadline: noDeadline.Unix(),
},
},
},
},
{
desc: "Task with empty headers",
task: NewTaskWithHeaders("process_data", []byte("data"), map[string]string{}),
opts: []Option{},
wantInfo: &TaskInfo{
Queue: "default",
Type: "process_data",
Payload: []byte("data"),
Headers: map[string]string{},
State: TaskStatePending,
MaxRetry: defaultMaxRetry,
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: defaultTimeout,
Deadline: time.Time{},
NextProcessAt: now,
},
wantPending: map[string][]*base.TaskMessage{
"default": {
{
Type: "process_data",
Payload: []byte("data"),
Headers: nil,
Retry: defaultMaxRetry,
Queue: "default",
Timeout: int64(defaultTimeout.Seconds()),
Deadline: noDeadline.Unix(),
},
},
},
},
{
desc: "Task with nil headers",
task: NewTaskWithHeaders("cleanup", nil, nil),
opts: []Option{},
wantInfo: &TaskInfo{
Queue: "default",
Type: "cleanup",
Payload: nil,
Headers: nil,
State: TaskStatePending,
MaxRetry: defaultMaxRetry,
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: defaultTimeout,
Deadline: time.Time{},
NextProcessAt: now,
},
wantPending: map[string][]*base.TaskMessage{
"default": {
{
Type: "cleanup",
Payload: nil,
Headers: nil,
Retry: defaultMaxRetry,
Queue: "default",
Timeout: int64(defaultTimeout.Seconds()),
Deadline: noDeadline.Unix(),
},
},
},
},
{
desc: "Task with headers and custom options",
task: NewTaskWithHeaders("notify", []byte("notification"), map[string]string{"channel": "email"}),
opts: []Option{MaxRetry(5), Queue("notifications")},
wantInfo: &TaskInfo{
Queue: "notifications",
Type: "notify",
Payload: []byte("notification"),
Headers: map[string]string{"channel": "email"},
State: TaskStatePending,
MaxRetry: 5,
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: defaultTimeout,
Deadline: time.Time{},
NextProcessAt: now,
},
wantPending: map[string][]*base.TaskMessage{
"notifications": {
{
Type: "notify",
Payload: []byte("notification"),
Headers: map[string]string{"channel": "email"},
Retry: 5,
Queue: "notifications",
Timeout: int64(defaultTimeout.Seconds()),
Deadline: noDeadline.Unix(),
},
},
},
},
}
for _, tc := range tests {
h.FlushDB(t, r)
gotInfo, err := client.Enqueue(tc.task, tc.opts...)
if err != nil {
t.Errorf("%s: Enqueue failed: %v", tc.desc, err)
continue
}
cmpOptions := []cmp.Option{
cmpopts.IgnoreFields(TaskInfo{}, "ID"),
cmpopts.EquateApproxTime(500 * time.Millisecond),
}
if diff := cmp.Diff(tc.wantInfo, gotInfo, cmpOptions...); diff != "" {
t.Errorf("%s;\nEnqueue(task) returned %v, want %v; (-want,+got)\n%s",
tc.desc, gotInfo, tc.wantInfo, diff)
}
for qname, want := range tc.wantPending {
got := h.GetPendingMessages(t, r, qname)
if diff := cmp.Diff(want, got, h.IgnoreIDOpt); diff != "" {
t.Errorf("%s;\nmismatch found in %q; (-want,+got)\n%s", tc.desc, base.PendingKey(qname), diff)
}
}
}
}
func TestClientEnqueueWithHeadersScheduled(t *testing.T) {
r := setup(t)
client := NewClient(getRedisConnOpt(t))
defer client.Close()
now := time.Now()
oneHourLater := now.Add(time.Hour)
headers := map[string]string{
"correlation-id": "xyz-123",
"source": "api",
}
tests := []struct {
desc string
task *Task
processAt time.Time
opts []Option
wantInfo *TaskInfo
wantScheduled map[string][]base.Z
}{
{
desc: "Schedule task with headers",
task: NewTaskWithHeaders("scheduled_task", []byte("payload"), headers),
processAt: oneHourLater,
opts: []Option{},
wantInfo: &TaskInfo{
Queue: "default",
Type: "scheduled_task",
Payload: []byte("payload"),
Headers: headers,
State: TaskStateScheduled,
MaxRetry: defaultMaxRetry,
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: defaultTimeout,
Deadline: time.Time{},
NextProcessAt: oneHourLater,
},
wantScheduled: map[string][]base.Z{
"default": {
{
Message: &base.TaskMessage{
Type: "scheduled_task",
Payload: []byte("payload"),
Headers: headers,
Retry: defaultMaxRetry,
Queue: "default",
Timeout: int64(defaultTimeout.Seconds()),
Deadline: noDeadline.Unix(),
},
Score: oneHourLater.Unix(),
},
},
},
},
}
for _, tc := range tests {
h.FlushDB(t, r)
opts := append(tc.opts, ProcessAt(tc.processAt))
gotInfo, err := client.Enqueue(tc.task, opts...)
if err != nil {
t.Errorf("%s: Enqueue failed: %v", tc.desc, err)
continue
}
cmpOptions := []cmp.Option{
cmpopts.IgnoreFields(TaskInfo{}, "ID"),
cmpopts.EquateApproxTime(500 * time.Millisecond),
}
if diff := cmp.Diff(tc.wantInfo, gotInfo, cmpOptions...); diff != "" {
t.Errorf("%s;\nEnqueue(task, ProcessAt(%v)) returned %v, want %v; (-want,+got)\n%s",
tc.desc, tc.processAt, gotInfo, tc.wantInfo, diff)
}
for qname, want := range tc.wantScheduled {
gotScheduled := h.GetScheduledEntries(t, r, qname)
if diff := cmp.Diff(want, gotScheduled, h.IgnoreIDOpt, cmpopts.EquateEmpty()); diff != "" {
t.Errorf("%s;\nmismatch found in %q; (-want,+got)\n%s", tc.desc, base.ScheduledKey(qname), diff)
}
}
}
}
func TestNewTaskWithHeaders(t *testing.T) {
tests := []struct {
desc string
typename string
payload []byte
headers map[string]string
opts []Option
want *Task
}{
{
desc: "Task with headers",
typename: "test_task",
payload: []byte("test payload"),
headers: map[string]string{"key1": "value1", "key2": "value2"},
opts: []Option{MaxRetry(3)},
want: &Task{
typename: "test_task",
payload: []byte("test payload"),
headers: map[string]string{"key1": "value1", "key2": "value2"},
opts: []Option{MaxRetry(3)},
},
},
{
desc: "Task with empty headers",
typename: "empty_headers",
payload: nil,
headers: map[string]string{},
opts: nil,
want: &Task{
typename: "empty_headers",
payload: nil,
headers: map[string]string{},
opts: nil,
},
},
{
desc: "Task with nil headers",
typename: "nil_headers",
payload: []byte("data"),
headers: nil,
opts: []Option{Queue("test")},
want: &Task{
typename: "nil_headers",
payload: []byte("data"),
headers: nil,
opts: []Option{Queue("test")},
},
},
}
for _, tc := range tests {
got := NewTaskWithHeaders(tc.typename, tc.payload, tc.headers, tc.opts...)
if got.Type() != tc.want.typename {
t.Errorf("%s: Type() = %q, want %q", tc.desc, got.Type(), tc.want.typename)
}
if diff := cmp.Diff(tc.want.payload, got.Payload()); diff != "" {
t.Errorf("%s: Payload() mismatch (-want,+got)\n%s", tc.desc, diff)
}
if diff := cmp.Diff(tc.want.headers, got.Headers()); diff != "" {
t.Errorf("%s: Headers() mismatch (-want,+got)\n%s", tc.desc, diff)
}
if tc.headers != nil && got.Headers() != nil {
tc.headers["modified"] = "test"
if _, exists := got.Headers()["modified"]; exists {
t.Errorf("%s: Headers should be cloned, but modification affected task headers", tc.desc)
}
}
}
}
func TestTaskHeadersMethod(t *testing.T) {
tests := []struct {
desc string
task *Task
want map[string]string
wantNil bool
}{
{
desc: "Task created with NewTask has nil headers",
task: NewTask("test", []byte("data")),
want: nil,
wantNil: true,
},
{
desc: "Task created with NewTaskWithHeaders has headers",
task: NewTaskWithHeaders("test", []byte("data"), map[string]string{"key": "value"}),
want: map[string]string{"key": "value"},
},
{
desc: "Task created with empty headers",
task: NewTaskWithHeaders("test", []byte("data"), map[string]string{}),
want: map[string]string{},
},
{
desc: "Task created with nil headers",
task: NewTaskWithHeaders("test", []byte("data"), nil),
want: nil,
wantNil: true,
},
}
for _, tc := range tests {
got := tc.task.Headers()
if tc.wantNil {
if got != nil {
t.Errorf("%s: Headers() = %v, want nil", tc.desc, got)
}
} else {
if diff := cmp.Diff(tc.want, got); diff != "" {
t.Errorf("%s: Headers() mismatch (-want,+got)\n%s", tc.desc, diff)
}
}
}
}
func TestClientEnqueueWithHeadersAndGroup(t *testing.T) {
r := setup(t)
client := NewClient(getRedisConnOpt(t))
defer client.Close()
now := time.Now()
headers := map[string]string{
"batch-id": "batch-123",
"priority": "high",
}
tests := []struct {
desc string
task *Task
opts []Option
wantInfo *TaskInfo
wantGroups map[string]map[string][]base.Z
}{
{
desc: "Task with headers and group",
task: NewTaskWithHeaders("batch_process", []byte("item1"), headers),
opts: []Option{Group("batch-123")},
wantInfo: &TaskInfo{
Queue: "default",
Group: "batch-123",
Type: "batch_process",
Payload: []byte("item1"),
Headers: headers,
State: TaskStateAggregating,
MaxRetry: defaultMaxRetry,
Retried: 0,
LastErr: "",
LastFailedAt: time.Time{},
Timeout: defaultTimeout,
Deadline: time.Time{},
NextProcessAt: time.Time{},
},
wantGroups: map[string]map[string][]base.Z{
"default": {
"batch-123": {
{
Message: &base.TaskMessage{
Type: "batch_process",
Payload: []byte("item1"),
Headers: headers,
Retry: defaultMaxRetry,
Queue: "default",
Timeout: int64(defaultTimeout.Seconds()),
Deadline: noDeadline.Unix(),
GroupKey: "batch-123",
},
Score: now.Unix(),
},
},
},
},
},
}
for _, tc := range tests {
h.FlushDB(t, r)
gotInfo, err := client.Enqueue(tc.task, tc.opts...)
if err != nil {
t.Errorf("%s: Enqueue failed: %v", tc.desc, err)
continue
}
cmpOptions := []cmp.Option{
cmpopts.IgnoreFields(TaskInfo{}, "ID"),
cmpopts.EquateApproxTime(500 * time.Millisecond),
}
if diff := cmp.Diff(tc.wantInfo, gotInfo, cmpOptions...); diff != "" {
t.Errorf("%s;\nEnqueue(task) returned %v, want %v; (-want,+got)\n%s",
tc.desc, gotInfo, tc.wantInfo, diff)
}
for qname, groups := range tc.wantGroups {
for groupKey, want := range groups {
got := h.GetGroupEntries(t, r, qname, groupKey)
if diff := cmp.Diff(want, got, h.IgnoreIDOpt, cmpopts.EquateEmpty()); diff != "" {
t.Errorf("%s;\nmismatch found in %q; (-want,+got)\n%s", tc.desc, base.GroupKey(qname, groupKey), diff)
}
}
}
}
}
================================================
FILE: context.go
================================================
// Copyright 2020 Kentaro Hibino. All rights reserved.
// Use of this source code is governed by a MIT license
// that can be found in the LICENSE file.
package asynq
import (
"context"
asynqcontext "github.com/hibiken/asynq/internal/context"
)
// GetTaskID extracts a task ID from a context, if any.
//
// ID of a task is guaranteed to be unique.
// ID of a task doesn't change if the task is being retried.
func GetTaskID(ctx context.Context) (id string, ok bool) {
return asynqcontext.GetTaskID(ctx)
}
// GetRetryCount extracts retry count from a context, if any.
//
// Return value n indicates the number of times associated task has been
// retried so far.
func GetRetryCount(ctx context.Context) (n int, ok bool) {
return asynqcontext.GetRetryCount(ctx)
}
// GetMaxRetry extracts maximum retry from a context, if any.
//
// Return value n indicates the maximum number of times the associated task
// can be retried if ProcessTask returns a non-nil error.
func GetMaxRetry(ctx context.Context) (n int, ok bool) {
return asynqcontext.GetMaxRetry(ctx)
}
// GetQueueName extracts queue name from a context, if any.
//
// Return value queue indicates which queue the task was pulled from.
func GetQueueName(ctx context.Context) (queue string, ok bool) {
return asynqcontext.GetQueueName(ctx)
}
================================================
FILE: doc.go
================================================
// Copyright 2020 Kentaro Hibino. All rights reserved.
// Use of this source code is governed by a MIT license
// that can be found in the LICENSE file.
/*
Package asynq provides a framework for Redis based distrubted task queue.
Asynq uses Redis as a message broker. To connect to redis,
specify the connection using one of RedisConnOpt types.
redisConnOpt = asynq.RedisClientOpt{
Addr: "127.0.0.1:6379",
Password: "xxxxx",
DB: 2,
}
The Client is used to enqueue a task.
client := asynq.NewClient(redisConnOpt)
// Task is created with two parameters: its type and payload.
// Payload data is simply an array of bytes. It can be encoded in JSON, Protocol Buffer, Gob, etc.
b, err := json.Marshal(ExamplePayload{UserID: 42})
if err != nil {
log.Fatal(err)
}
task := asynq.NewTask("example", b)
// Enqueue the task to be processed immediately.
info, err := client.Enqueue(task)
// Schedule the task to be processed after one minute.
info, err = client.Enqueue(t, asynq.ProcessIn(1*time.Minute))
The Server is used to run the task processing workers with a given
handler.
srv := asynq.NewServer(redisConnOpt, asynq.Config{
Concurrency: 10,
})
if err := srv.Run(handler); err != nil {
log.Fatal(err)
}
Handler is an interface type with a method which
takes a task and returns an error. Handler should return nil if
the processing is successful, otherwise return a non-nil error.
If handler panics or returns a non-nil error, the task will be retried in the future.
Example of a type that implements the Handler interface.
type TaskHandler struct {
// ...
}
func (h *TaskHandler) ProcessTask(ctx context.Context, task *asynq.Task) error {
switch task.Type {
case "example":
var data ExamplePayload
if err := json.Unmarshal(task.Payload(), &data); err != nil {
return err
}
// perform task with the data
default:
return fmt.Errorf("unexpected task type %q", task.Type)
}
return nil
}
*/
package asynq
================================================
FILE: example_test.go
================================================
// Copyright 2020 Kentaro Hibino. All rights reserved.
// Use of this source code is governed by a MIT license
// that can be found in the LICENSE file.
package asynq_test
import (
"context"
"fmt"
"log"
"os"
"os/signal"
"time"
"github.com/hibiken/asynq"
"golang.org/x/sys/unix"
)
func ExampleServer_Run() {
srv := asynq.NewServer(
asynq.RedisClientOpt{Addr: ":6379"},
asynq.Config{Concurrency: 20},
)
h := asynq.NewServeMux()
// ... Register handlers
// Run blocks and waits for os signal to terminate the program.
if err := srv.Run(h); err != nil {
log.Fatal(err)
}
}
func ExampleServer_Shutdown() {
srv := asynq.NewServer(
asynq.RedisClientOpt{Addr: ":6379"},
asynq.Config{Concurrency: 20},
)
h := asynq.NewServeMux()
// ... Register handlers
if err := srv.Start(h); err != nil {
log.Fatal(err)
}
sigs := make(chan os.Signal, 1)
signal.Notify(sigs, unix.SIGTERM, unix.SIGINT)
<-sigs // wait for termination signal
srv.Shutdown()
}
func ExampleServer_Stop() {
srv := asynq.NewServer(
asynq.RedisClientOpt{Addr: ":6379"},
asynq.Config{Concurrency: 20},
)
h := asynq.NewServeMux()
// ... Register handlers
if err := srv.Start(h); err != nil {
log.Fatal(err)
}
sigs := make(chan os.Signal, 1)
signal.Notify(sigs, unix.SIGTERM, unix.SIGINT, unix.SIGTSTP)
// Handle SIGTERM, SIGINT to exit the program.
// Handle SIGTSTP to stop processing new tasks.
for {
s := <-sigs
if s == unix.SIGTSTP {
srv.Stop() // stop processing new tasks
continue
}
break // received SIGTERM or SIGINT signal
}
srv.Shutdown()
}
func ExampleScheduler() {
scheduler := asynq.NewScheduler(
asynq.RedisClientOpt{Addr: ":6379"},
&asynq.SchedulerOpts{Location: time.Local},
)
if _, err := scheduler.Register("* * * * *", asynq.NewTask("task1", nil)); err != nil {
log.Fatal(err)
}
if _, err := scheduler.Register("@every 30s", asynq.NewTask("task2", nil)); err != nil {
log.Fatal(err)
}
// Run blocks and waits for os signal to terminate the program.
if err := scheduler.Run(); err != nil {
log.Fatal(err)
}
}
func ExampleParseRedisURI() {
rconn, err := asynq.ParseRedisURI("redis://localhost:6379/10")
if err != nil {
log.Fatal(err)
}
r, ok := rconn.(asynq.RedisClientOpt)
if !ok {
log.Fatal("unexpected type")
}
fmt.Println(r.Addr)
fmt.Println(r.DB)
// Output:
// localhost:6379
// 10
}
func ExampleResultWriter() {
// ResultWriter is only accessible in Handler.
h := func(ctx context.Context, task *asynq.Task) error {
// .. do task processing work
res := []byte("task result data")
n, err := task.ResultWriter().Write(res) // implements io.Writer
if err != nil {
return fmt.Errorf("failed to write task result: %w", err)
}
log.Printf(" %d bytes written", n)
return nil
}
_ = h
}
================================================
FILE: forwarder.go
================================================
// Copyright 2020 Kentaro Hibino. All rights reserved.
// Use of this source code is governed by a MIT license
// that can be found in the LICENSE file.
package asynq
import (
"sync"
"time"
"github.com/hibiken/asynq/internal/base"
"github.com/hibiken/asynq/internal/log"
)
// A forwarder is responsible for moving scheduled and retry tasks to pending state
// so that the tasks get processed by the workers.
type forwarder struct {
logger *log.Logger
broker base.Broker
// channel to communicate back to the long running "forwarder" goroutine.
done chan struct{}
// list of queue names to check and enqueue.
queues []string
// poll interval on average
avgInterval time.Duration
}
type forwarderParams struct {
logger *log.Logger
broker base.Broker
queues []string
interval time.Duration
}
func newForwarder(params forwarderParams) *forwarder {
return &forwarder{
logger: params.logger,
broker: params.broker,
done: make(chan struct{}),
queues: params.queues,
avgInterval: params.interval,
}
}
func (f *forwarder) shutdown() {
f.logger.Debug("Forwarder shutting down...")
// Signal the forwarder goroutine to stop polling.
f.done <- struct{}{}
}
// start starts the "forwarder" goroutine.
func (f *forwarder) start(wg *sync.WaitGroup) {
wg.Add(1)
go func() {
defer wg.Done()
timer := time.NewTimer(f.avgInterval)
for {
select {
case <-f.done:
f.logger.Debug("Forwarder done")
return
case <-timer.C:
f.exec()
timer.Reset(f.avgInterval)
}
}
}()
}
func (f *forwarder) exec() {
if err := f.broker.ForwardIfReady(f.queues...); err != nil {
f.logger.Errorf("Failed to forward scheduled tasks: %v", err)
}
}
================================================
FILE: forwarder_test.go
================================================
// Copyright 2020 Kentaro Hibino. All rights reserved.
// Use of this source code is governed by a MIT license
// that can be found in the LICENSE file.
package asynq
import (
"sync"
"testing"
"time"
"github.com/google/go-cmp/cmp"
"github.com/hibiken/asynq/internal/base"
"github.com/hibiken/asynq/internal/rdb"
h "github.com/hibiken/asynq/internal/testutil"
)
func TestForwarder(t *testing.T) {
r := setup(t)
defer r.Close()
rdbClient := rdb.NewRDB(r)
const pollInterval = time.Second
s := newForwarder(forwarderParams{
logger: testLogger,
broker: rdbClient,
queues: []string{"default", "critical"},
interval: pollInterval,
})
t1 := h.NewTaskMessageWithQueue("gen_thumbnail", nil, "default")
t2 := h.NewTaskMessageWithQueue("send_email", nil, "critical")
t3 := h.NewTaskMessageWithQueue("reindex", nil, "default")
t4 := h.NewTaskMessageWithQueue("sync", nil, "critical")
now := time.Now()
tests := []struct {
initScheduled map[string][]base.Z // scheduled queue initial state
initRetry map[string][]base.Z // retry queue initial state
initPending map[string][]*base.TaskMessage // default queue initial state
wait time.Duration // wait duration before checking for final state
wantScheduled map[string][]*base.TaskMessage // schedule queue final state
wantRetry map[string][]*base.TaskMessage // retry queue final state
wantPending map[string][]*base.TaskMessage // default queue final state
}{
{
initScheduled: map[string][]base.Z{
"default": {{Message: t1, Score: now.Add(time.Hour).Unix()}},
"critical": {{Message: t2, Score: now.Add(-2 * time.Second).Unix()}},
},
initRetry: map[string][]base.Z{
"default": {{Message: t3, Score: time.Now().Add(-500 * time.Millisecond).Unix()}},
"critical": {},
},
initPending: map[string][]*base.TaskMessage{
"default": {},
"critical": {t4},
},
wait: pollInterval * 2,
wantScheduled: map[string][]*base.TaskMessage{
"default": {t1},
"critical": {},
},
wantRetry: map[string][]*base.TaskMessage{
"default": {},
"critical": {},
},
wantPending: map[string][]*base.TaskMessage{
"default": {t3},
"critical": {t2, t4},
},
},
{
initScheduled: map[string][]base.Z{
"default": {
{Message: t1, Score: now.Unix()},
{Message: t3, Score: now.Add(-500 * time.Millisecond).Unix()},
},
"critical": {
{Message: t2, Score: now.Add(-2 * time.Second).Unix()},
},
},
initRetry: map[string][]base.Z{
"default": {},
"critical": {},
},
initPending: map[string][]*base.TaskMessage{
"default": {},
"critical": {t4},
},
wait: pollInterval * 2,
wantScheduled: map[string][]*base.TaskMessage{
"default": {},
"critical": {},
},
wantRetry: map[string][]*base.TaskMessage{
"default": {},
"critical": {},
},
wantPending: map[string][]*base.TaskMessage{
"default": {t1, t3},
"critical": {t2, t4},
},
},
}
for _, tc := range tests {
h.FlushDB(t, r) // clean up db before each test case.
h.SeedAllScheduledQueues(t, r, tc.initScheduled) // initialize scheduled queue
h.SeedAllRetryQueues(t, r, tc.initRetry) // initialize retry queue
h.SeedAllPendingQueues(t, r, tc.initPending) // initialize default queue
var wg sync.WaitGroup
s.start(&wg)
time.Sleep(tc.wait)
s.shutdown()
for qname, want := range tc.wantScheduled {
gotScheduled := h.GetScheduledMessages(t, r, qname)
if diff := cmp.Diff(want, gotScheduled, h.SortMsgOpt); diff != "" {
t.Errorf("mismatch found in %q after running forwarder: (-want, +got)\n%s", base.ScheduledKey(qname), diff)
}
}
for qname, want := range tc.wantRetry {
gotRetry := h.GetRetryMessages(t, r, qname)
if diff := cmp.Diff(want, gotRetry, h.SortMsgOpt); diff != "" {
t.Errorf("mismatch found in %q after running forwarder: (-want, +got)\n%s", base.RetryKey(qname), diff)
}
}
for qname, want := range tc.wantPending {
gotPending := h.GetPendingMessages(t, r, qname)
if diff := cmp.Diff(want, gotPending, h.SortMsgOpt); diff != "" {
t.Errorf("mismatch found in %q after running forwarder: (-want, +got)\n%s", base.PendingKey(qname), diff)
}
}
}
}
================================================
FILE: go.mod
================================================
module github.com/hibiken/asynq
go 1.24.0
require (
github.com/google/go-cmp v0.7.0
github.com/google/uuid v1.6.0
github.com/redis/go-redis/v9 v9.14.1
github.com/robfig/cron/v3 v3.0.1
github.com/spf13/cast v1.10.0
go.uber.org/goleak v1.3.0
golang.org/x/sys v0.37.0
golang.org/x/time v0.14.0
google.golang.org/protobuf v1.36.10
)
require (
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
)
================================================
FILE: go.sum
================================================
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/redis/go-redis/v9 v9.14.1 h1:nDCrEiJmfOWhD76xlaw+HXT0c9hfNWeXgl0vIRYSDvQ=
github.com/redis/go-redis/v9 v9.14.1/go.mod h1:huWgSWd8mW6+m0VPhJjSSQ+d6Nh1VICQ6Q5lHuCH/Iw=
github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs=
github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro=
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY=
github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo=
github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
golang.org/x/sys v0.37.0 h1:fdNQudmxPjkdUTPnLn5mdQv7Zwvbvpaxqs831goi9kQ=
golang.org/x/sys v0.37.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI=
golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4=
google.golang.org/protobuf v1.36.10 h1:AYd7cD/uASjIL6Q9LiTjz8JLcrh/88q5UObnmY3aOOE=
google.golang.org/protobuf v1.36.10/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
================================================
FILE: healthcheck.go
================================================
// Copyright 2020 Kentaro Hibino. All rights reserved.
// Use of this source code is governed by a MIT license
// that can be found in the LICENSE file.
package asynq
import (
"sync"
"time"
"github.com/hibiken/asynq/internal/base"
"github.com/hibiken/asynq/internal/log"
)
// healthchecker is responsible for pinging broker periodically
// and call user provided HeathCheckFunc with the ping result.
type healthchecker struct {
logger *log.Logger
broker base.Broker
// channel to communicate back to the long running "healthchecker" goroutine.
done chan struct{}
// interval between healthchecks.
interval time.Duration
// function to call periodically.
healthcheckFunc func(error)
}
type healthcheckerParams struct {
logger *log.Logger
broker base.Broker
interval time.Duration
healthcheckFunc func(error)
}
func newHealthChecker(params healthcheckerParams) *healthchecker {
return &healthchecker{
logger: params.logger,
broker: params.broker,
done: make(chan struct{}),
interval: params.interval,
healthcheckFunc: params.healthcheckFunc,
}
}
func (hc *healthchecker) shutdown() {
if hc.healthcheckFunc == nil {
return
}
hc.logger.Debug("Healthchecker shutting down...")
// Signal the healthchecker goroutine to stop.
hc.done <- struct{}{}
}
func (hc *healthchecker) start(wg *sync.WaitGroup) {
if hc.healthcheckFunc == nil {
return
}
wg.Add(1)
go func() {
defer wg.Done()
timer := time.NewTimer(hc.interval)
for {
select {
case <-hc.done:
hc.logger.Debug("Healthchecker done")
timer.Stop()
return
case <-timer.C:
err := hc.broker.Ping()
hc.healthcheckFunc(err)
timer.Reset(hc.interval)
}
}
}()
}
================================================
FILE: healthcheck_test.go
================================================
// Copyright 2020 Kentaro Hibino. All rights reserved.
// Use of this source code is governed by a MIT license
// that can be found in the LICENSE file.
package asynq
import (
"sync"
"testing"
"time"
"github.com/hibiken/asynq/internal/rdb"
"github.com/hibiken/asynq/internal/testbroker"
)
func TestHealthChecker(t *testing.T) {
r := setup(t)
defer r.Close()
rdbClient := rdb.NewRDB(r)
var (
// mu guards called and e variables.
mu sync.Mutex
called int
e error
)
checkFn := func(err error) {
mu.Lock()
defer mu.Unlock()
called++
e = err
}
hc := newHealthChecker(healthcheckerParams{
logger: testLogger,
broker: rdbClient,
interval: 1 * time.Second,
healthcheckFunc: checkFn,
})
hc.start(&sync.WaitGroup{})
time.Sleep(2 * time.Second)
mu.Lock()
if called == 0 {
t.Errorf("Healthchecker did not call the provided HealthCheckFunc")
}
if e != nil {
t.Errorf("HealthCheckFunc was called with non-nil error: %v", e)
}
mu.Unlock()
hc.shutdown()
}
func TestHealthCheckerWhenRedisDown(t *testing.T) {
// Make sure that healthchecker goroutine doesn't panic
// if it cannot connect to redis.
defer func() {
if r := recover(); r != nil {
t.Errorf("panic occurred: %v", r)
}
}()
r := rdb.NewRDB(setup(t))
defer r.Close()
testBroker := testbroker.NewTestBroker(r)
var (
// mu guards called and e variables.
mu sync.Mutex
called int
e error
)
checkFn := func(err error) {
mu.Lock()
defer mu.Unlock()
called++
e = err
}
hc := newHealthChecker(healthcheckerParams{
logger: testLogger,
broker: testBroker,
interval: 1 * time.Second,
healthcheckFunc: checkFn,
})
testBroker.Sleep()
hc.start(&sync.WaitGroup{})
time.Sleep(2 * time.Second)
mu.Lock()
if called == 0 {
t.Errorf("Healthchecker did not call the provided HealthCheckFunc")
}
if e == nil {
t.Errorf("HealthCheckFunc was called with nil; want non-nil error")
}
mu.Unlock()
hc.shutdown()
}
================================================
FILE: heartbeat.go
================================================
// Copyright 2020 Kentaro Hibino. All rights reserved.
// Use of this source code is governed by a MIT license
// that can be found in the LICENSE file.
package asynq
import (
"os"
"sync"
"time"
"github.com/google/uuid"
"github.com/hibiken/asynq/internal/base"
"github.com/hibiken/asynq/internal/log"
"github.com/hibiken/asynq/internal/timeutil"
)
// heartbeater is responsible for writing process info to redis periodically to
// indicate that the background worker process is up.
type heartbeater struct {
logger *log.Logger
broker base.Broker
clock timeutil.Clock
// channel to communicate back to the long running "heartbeater" goroutine.
done chan struct{}
// interval between heartbeats.
interval time.Duration
// following fields are initialized at construction time and are immutable.
host string
pid int
serverID string
concurrency int
queues map[string]int
strictPriority bool
// following fields are mutable and should be accessed only by the
// heartbeater goroutine. In other words, confine these variables
// to this goroutine only.
started time.Time
workers map[string]*workerInfo
// state is shared with other goroutine but is concurrency safe.
state *serverState
// channels to receive updates on active workers.
starting <-chan *workerInfo
finished <-chan *base.TaskMessage
}
type heartbeaterParams struct {
logger *log.Logger
broker base.Broker
interval time.Duration
concurrency int
queues map[string]int
strictPriority bool
state *serverState
starting <-chan *workerInfo
finished <-chan *base.TaskMessage
}
func newHeartbeater(params heartbeaterParams) *heartbeater {
host, err := os.Hostname()
if err != nil {
host = "unknown-host"
}
return &heartbeater{
logger: params.logger,
broker: params.broker,
clock: timeutil.NewRealClock(),
done: make(chan struct{}),
interval: params.interval,
host: host,
pid: os.Getpid(),
serverID: uuid.New().String(),
concurrency: params.concurrency,
queues: params.queues,
strictPriority: params.strictPriority,
state: params.state,
workers: make(map[string]*workerInfo),
starting: params.starting,
finished: params.finished,
}
}
func (h *heartbeater) shutdown() {
h.logger.Debug("Heartbeater shutting down...")
// Signal the heartbeater goroutine to stop.
h.done <- struct{}{}
}
// A workerInfo holds an active worker information.
type workerInfo struct {
// the task message the worker is processing.
msg *base.TaskMessage
// the time the worker has started processing the message.
started time.Time
// deadline the worker has to finish processing the task by.
deadline time.Time
// lease the worker holds for the task.
lease *base.Lease
}
func (h *heartbeater) start(wg *sync.WaitGroup) {
wg.Add(1)
go func() {
defer wg.Done()
h.started = h.clock.Now()
h.beat()
timer := time.NewTimer(h.interval)
for {
select {
case <-h.done:
if err := h.broker.ClearServerState(h.host, h.pid, h.serverID); err != nil {
h.logger.Errorf("Failed to clear server state: %v", err)
}
h.logger.Debug("Heartbeater done")
timer.Stop()
return
case <-timer.C:
h.beat()
timer.Reset(h.interval)
case w := <-h.starting:
h.workers[w.msg.ID] = w
case msg := <-h.finished:
delete(h.workers, msg.ID)
}
}
}()
}
// beat extends lease for workers and writes server/worker info to redis.
func (h *heartbeater) beat() {
h.state.mu.Lock()
srvStatus := h.state.value.String()
h.state.mu.Unlock()
info := base.ServerInfo{
Host: h.host,
PID: h.pid,
ServerID: h.serverID,
Concurrency: h.concurrency,
Queues: h.queues,
StrictPriority: h.strictPriority,
Status: srvStatus,
Started: h.started,
ActiveWorkerCount: len(h.workers),
}
var ws []*base.WorkerInfo
idsByQueue := make(map[string][]string)
for id, w := range h.workers {
ws = append(ws, &base.WorkerInfo{
Host: h.host,
PID: h.pid,
ServerID: h.serverID,
ID: id,
Type: w.msg.Type,
Queue: w.msg.Queue,
Payload: w.msg.Payload,
Started: w.started,
Deadline: w.deadline,
})
// Check lease before adding to the set to make sure not to extend the lease if the lease is already expired.
if w.lease.IsValid() {
idsByQueue[w.msg.Queue] = append(idsByQueue[w.msg.Queue], id)
} else {
w.lease.NotifyExpiration() // notify processor if the lease is expired
}
}
// Note: Set TTL to be long enough so that it won't expire before we write again
// and short enough to expire quickly once the process is shut down or killed.
if err := h.broker.WriteServerState(&info, ws, h.interval*2); err != nil {
h.logger.Errorf("Failed to write server state data: %v", err)
}
for qname, ids := range idsByQueue {
expirationTime, err := h.broker.ExtendLease(qname, ids...)
if err != nil {
h.logger.Errorf("Failed to extend lease for tasks %v: %v", ids, err)
continue
}
for _, id := range ids {
if l := h.workers[id].lease; !l.Reset(expirationTime) {
h.logger.Warnf("Lease reset failed for %s; lease deadline: %v", id, l.Deadline())
}
}
}
}
================================================
FILE: heartbeat_test.go
================================================
// Copyright 2020 Kentaro Hibino. All rights reserved.
// Use of this source code is governed by a MIT license
// that can be found in the LICENSE file.
package asynq
import (
"context"
"sync"
"testing"
"time"
"github.com/google/go-cmp/cmp"
"github.com/google/go-cmp/cmp/cmpopts"
"github.com/hibiken/asynq/internal/base"
"github.com/hibiken/asynq/internal/rdb"
"github.com/hibiken/asynq/internal/testbroker"
h "github.com/hibiken/asynq/internal/testutil"
"github.com/hibiken/asynq/internal/timeutil"
)
// Test goes through a few phases.
//
// Phase1: Simulate Server startup; Simulate starting tasks listed in startedWorkers
// Phase2: Simulate finishing tasks listed in finishedTasks
// Phase3: Simulate Server shutdown;
func TestHeartbeater(t *testing.T) {
r := setup(t)
defer r.Close()
rdbClient := rdb.NewRDB(r)
now := time.Now()
const elapsedTime = 10 * time.Second // simulated time elapsed between phase1 and phase2
clock := timeutil.NewSimulatedClock(time.Time{}) // time will be set in each test
t1 := h.NewTaskMessageWithQueue("task1", nil, "default")
t2 := h.NewTaskMessageWithQueue("task2", nil, "default")
t3 := h.NewTaskMessageWithQueue("task3", nil, "default")
t4 := h.NewTaskMessageWithQueue("task4", nil, "custom")
t5 := h.NewTaskMessageWithQueue("task5", nil, "custom")
t6 := h.NewTaskMessageWithQueue("task6", nil, "default")
// Note: intentionally set to time less than now.Add(rdb.LeaseDuration) to test lease extension is working.
lease1 := h.NewLeaseWithClock(now.Add(10*time.Second), clock)
lease2 := h.NewLeaseWithClock(now.Add(10*time.Second), clock)
lease3 := h.NewLeaseWithClock(now.Add(10*time.Second), clock)
lease4 := h.NewLeaseWithClock(now.Add(10*time.Second), clock)
lease5 := h.NewLeaseWithClock(now.Add(10*time.Second), clock)
lease6 := h.NewLeaseWithClock(now.Add(10*time.Second), clock)
tests := []struct {
desc string
// Interval between heartbeats.
interval time.Duration
// Server info.
host string
pid int
queues map[string]int
concurrency int
active map[string][]*base.TaskMessage // initial active set state
lease map[string][]base.Z // initial lease set state
wantLease1 map[string][]base.Z // expected lease set state after starting all startedWorkers
wantLease2 map[string][]base.Z // expected lease set state after finishing all finishedTasks
startedWorkers []*workerInfo // workerInfo to send via the started channel
finishedTasks []*base.TaskMessage // tasks to send via the finished channel
startTime time.Time // simulated start time
elapsedTime time.Duration // simulated time elapsed between starting and finishing processing tasks
}{
{
desc: "With single queue",
interval: 2 * time.Second,
host: "localhost",
pid: 45678,
queues: map[string]int{"default": 1},
concurrency: 10,
active: map[string][]*base.TaskMessage{
"default": {t1, t2, t3},
},
lease: map[string][]base.Z{
"default": {
{Message: t1, Score: now.Add(10 * time.Second).Unix()},
{Message: t2, Score: now.Add(10 * time.Second).Unix()},
{Message: t3, Score: now.Add(10 * time.Second).Unix()},
},
},
startedWorkers: []*workerInfo{
{msg: t1, started: now, deadline: now.Add(2 * time.Minute), lease: lease1},
{msg: t2, started: now, deadline: now.Add(2 * time.Minute), lease: lease2},
{msg: t3, started: now, deadline: now.Add(2 * time.Minute), lease: lease3},
},
finishedTasks: []*base.TaskMessage{t1, t2},
wantLease1: map[string][]base.Z{
"default": {
{Message: t1, Score: now.Add(rdb.LeaseDuration).Unix()},
{Message: t2, Score: now.Add(rdb.LeaseDuration).Unix()},
{Message: t3, Score: now.Add(rdb.LeaseDuration).Unix()},
},
},
wantLease2: map[string][]base.Z{
"default": {
{Message: t3, Score: now.Add(elapsedTime).Add(rdb.LeaseDuration).Unix()},
},
},
startTime: now,
elapsedTime: elapsedTime,
},
{
desc: "With multiple queue",
interval: 2 * time.Second,
host: "localhost",
pid: 45678,
queues: map[string]int{"default": 1, "custom": 2},
concurrency: 10,
active: map[string][]*base.TaskMessage{
"default": {t6},
"custom": {t4, t5},
},
lease: map[string][]base.Z{
"default": {
{Message: t6, Score: now.Add(10 * time.Second).Unix()},
},
"custom": {
{Message: t4, Score: now.Add(10 * time.Second).Unix()},
{Message: t5, Score: now.Add(10 * time.Second).Unix()},
},
},
startedWorkers: []*workerInfo{
{msg: t6, started: now, deadline: now.Add(2 * time.Minute), lease: lease6},
{msg: t4, started: now, deadline: now.Add(2 * time.Minute), lease: lease4},
{msg: t5, started: now, deadline: now.Add(2 * time.Minute), lease: lease5},
},
finishedTasks: []*base.TaskMessage{t6, t5},
wantLease1: map[string][]base.Z{
"default": {
{Message: t6, Score: now.Add(rdb.LeaseDuration).Unix()},
},
"custom": {
{Message: t4, Score: now.Add(rdb.LeaseDuration).Unix()},
{Message: t5, Score: now.Add(rdb.LeaseDuration).Unix()},
},
},
wantLease2: map[string][]base.Z{
"default": {},
"custom": {
{Message: t4, Score: now.Add(elapsedTime).Add(rdb.LeaseDuration).Unix()},
},
},
startTime: now,
elapsedTime: elapsedTime,
},
}
timeCmpOpt := cmpopts.EquateApproxTime(10 * time.Millisecond)
ignoreOpt := cmpopts.IgnoreUnexported(base.ServerInfo{})
ignoreFieldOpt := cmpopts.IgnoreFields(base.ServerInfo{}, "ServerID")
for _, tc := range tests {
h.FlushDB(t, r)
h.SeedAllActiveQueues(t, r, tc.active)
h.SeedAllLease(t, r, tc.lease)
clock.SetTime(tc.startTime)
rdbClient.SetClock(clock)
srvState := &serverState{}
startingCh := make(chan *workerInfo)
finishedCh := make(chan *base.TaskMessage)
hb := newHeartbeater(heartbeaterParams{
logger: testLogger,
broker: rdbClient,
interval: tc.interval,
concurrency: tc.concurrency,
queues: tc.queues,
strictPriority: false,
state: srvState,
starting: startingCh,
finished: finishedCh,
})
hb.clock = clock
// Change host and pid fields for testing purpose.
hb.host = tc.host
hb.pid = tc.pid
//===================
// Start Phase1
//===================
srvState.mu.Lock()
srvState.value = srvStateActive // simulating Server.Start
srvState.mu.Unlock()
var wg sync.WaitGroup
hb.start(&wg)
// Simulate processor starting to work on tasks.
for _, w := range tc.startedWorkers {
startingCh <- w
}
// Wait for heartbeater to write to redis
time.Sleep(tc.interval * 2)
ss, err := rdbClient.ListServers()
if err != nil {
t.Errorf("%s: could not read server info from redis: %v", tc.desc, err)
hb.shutdown()
continue
}
if len(ss) != 1 {
t.Errorf("%s: (*RDB).ListServers returned %d server info, want 1", tc.desc, len(ss))
hb.shutdown()
continue
}
wantInfo := &base.ServerInfo{
Host: tc.host,
PID: tc.pid,
Queues: tc.queues,
Concurrency: tc.concurrency,
Started: now,
Status: "active",
ActiveWorkerCount: len(tc.startedWorkers),
}
if diff := cmp.Diff(wantInfo, ss[0], timeCmpOpt, ignoreOpt, ignoreFieldOpt); diff != "" {
t.Errorf("%s: redis stored server status %+v, want %+v; (-want, +got)\n%s", tc.desc, ss[0], wantInfo, diff)
hb.shutdown()
continue
}
for qname, wantLease := range tc.wantLease1 {
gotLease := h.GetLeaseEntries(t, r, qname)
if diff := cmp.Diff(wantLease, gotLease, h.SortZSetEntryOpt); diff != "" {
t.Errorf("%s: mismatch found in %q: (-want,+got):\n%s", tc.desc, base.LeaseKey(qname), diff)
}
}
for _, w := range tc.startedWorkers {
if want := now.Add(rdb.LeaseDuration); w.lease.Deadline() != want {
t.Errorf("%s: lease deadline for %v is set to %v, want %v", tc.desc, w.msg, w.lease.Deadline(), want)
}
}
//===================
// Start Phase2
//===================
clock.AdvanceTime(tc.elapsedTime)
// Simulate processor finished processing tasks.
for _, msg := range tc.finishedTasks {
if err := rdbClient.Done(context.Background(), msg); err != nil {
t.Fatalf("RDB.Done failed: %v", err)
}
finishedCh <- msg
}
// Wait for heartbeater to write to redis
time.Sleep(tc.interval * 2)
for qname, wantLease := range tc.wantLease2 {
gotLease := h.GetLeaseEntries(t, r, qname)
if diff := cmp.Diff(wantLease, gotLease, h.SortZSetEntryOpt); diff != "" {
t.Errorf("%s: mismatch found in %q: (-want,+got):\n%s", tc.desc, base.LeaseKey(qname), diff)
}
}
//===================
// Start Phase3
//===================
// Server state change; simulating Server.Shutdown
srvState.mu.Lock()
srvState.value = srvStateClosed
srvState.mu.Unlock()
// Wait for heartbeater to write to redis
time.Sleep(tc.interval * 2)
wantInfo = &base.ServerInfo{
Host: tc.host,
PID: tc.pid,
Queues: tc.queues,
Concurrency: tc.concurrency,
Started: now,
Status: "closed",
ActiveWorkerCount: len(tc.startedWorkers) - len(tc.finishedTasks),
}
ss, err = rdbClient.ListServers()
if err != nil {
t.Errorf("%s: could not read server status from redis: %v", tc.desc, err)
hb.shutdown()
continue
}
if len(ss) != 1 {
t.Errorf("%s: (*RDB).ListServers returned %d server info, want 1", tc.desc, len(ss))
hb.shutdown()
continue
}
if diff := cmp.Diff(wantInfo, ss[0], timeCmpOpt, ignoreOpt, ignoreFieldOpt); diff != "" {
t.Errorf("%s: redis stored process status %+v, want %+v; (-want, +got)\n%s", tc.desc, ss[0], wantInfo, diff)
hb.shutdown()
continue
}
hb.shutdown()
}
}
func TestHeartbeaterWithRedisDown(t *testing.T) {
// Make sure that heartbeater goroutine doesn't panic
// if it cannot connect to redis.
defer func() {
if r := recover(); r != nil {
t.Errorf("panic occurred: %v", r)
}
}()
r := rdb.NewRDB(setup(t))
defer r.Close()
testBroker := testbroker.NewTestBroker(r)
state := &serverState{value: srvStateActive}
hb := newHeartbeater(heartbeaterParams{
logger: testLogger,
broker: testBroker,
interval: time.Second,
concurrency: 10,
queues: map[string]int{"default": 1},
strictPriority: false,
state: state,
starting: make(chan *workerInfo),
finished: make(chan *base.TaskMessage),
})
testBroker.Sleep()
var wg sync.WaitGroup
hb.start(&wg)
// wait for heartbeater to try writing data to redis
time.Sleep(2 * time.Second)
hb.shutdown()
}
================================================
FILE: inspector.go
================================================
// Copyright 2020 Kentaro Hibino. All rights reserved.
// Use of this source code is governed by a MIT license
// that can be found in the LICENSE file.
package asynq
import (
"fmt"
"strconv"
"strings"
"time"
"github.com/hibiken/asynq/internal/base"
"github.com/hibiken/asynq/internal/errors"
"github.com/hibiken/asynq/internal/rdb"
"github.com/redis/go-redis/v9"
)
// Inspector is a client interface to inspect and mutate the state of
// queues and tasks.
type Inspector struct {
rdb *rdb.RDB
// When an Inspector has been created with an existing Redis connection, we do
// not want to close it.
sharedConnection bool
}
// New returns a new instance of Inspector.
func NewInspector(r RedisConnOpt) *Inspector {
c, ok := r.MakeRedisClient().(redis.UniversalClient)
if !ok {
panic(fmt.Sprintf("inspeq: unsupported RedisConnOpt type %T", r))
}
inspector := NewInspectorFromRedisClient(c)
inspector.sharedConnection = false
return inspector
}
// NewInspectorFromRedisClient returns a new instance of Inspector given a redis.UniversalClient
// Warning: The underlying redis connection pool will not be closed by Asynq, you are responsible for closing it.
func NewInspectorFromRedisClient(c redis.UniversalClient) *Inspector {
return &Inspector{
rdb: rdb.NewRDB(c),
sharedConnection: true,
}
}
// Close closes the connection with redis.
func (i *Inspector) Close() error {
if i.sharedConnection {
return fmt.Errorf("redis connection is shared so the Inspector can't be closed through asynq")
}
return i.rdb.Close()
}
// Queues returns a list of all queue names.
func (i *Inspector) Queues() ([]string, error) {
return i.rdb.AllQueues()
}
// Groups returns a list of all groups within the given queue.
func (i *Inspector) Groups(queue string) ([]*GroupInfo, error) {
stats, err := i.rdb.GroupStats(queue)
if err != nil {
return nil, err
}
var res []*GroupInfo
for _, s := range stats {
res = append(res, &GroupInfo{
Group: s.Group,
Size: s.Size,
})
}
return res, nil
}
// GroupInfo represents a state of a group at a certain time.
type GroupInfo struct {
// Name of the group.
Group string
// Size is the total number of tasks in the group.
Size int
}
// QueueInfo represents a state of a queue at a certain time.
type QueueInfo struct {
// Name of the queue.
Queue string
// Total number of bytes that the queue and its tasks require to be stored in redis.
// It is an approximate memory usage value in bytes since the value is computed by sampling.
MemoryUsage int64
// Latency of the queue, measured by the oldest pending task in the queue.
Latency time.Duration
// Size is the total number of tasks in the queue.
// The value is the sum of Pending, Active, Scheduled, Retry, Aggregating and Archived.
Size int
// Groups is the total number of groups in the queue.
Groups int
// Number of pending tasks.
Pending int
// Number of active tasks.
Active int
// Number of scheduled tasks.
Scheduled int
// Number of retry tasks.
Retry int
// Number of archived tasks.
Archived int
// Number of stored completed tasks.
Completed int
// Number of aggregating tasks.
Aggregating int
// Total number of tasks being processed within the given date (counter resets daily).
// The number includes both succeeded and failed tasks.
Processed int
// Total number of tasks failed to be processed within the given date (counter resets daily).
Failed int
// Total number of tasks processed (cumulative).
ProcessedTotal int
// Total number of tasks failed (cumulative).
FailedTotal int
// Paused indicates whether the queue is paused.
// If true, tasks in the queue will not be processed.
Paused bool
// Time when this queue info snapshot was taken.
Timestamp time.Time
}
// GetQueueInfo returns current information of the given queue.
func (i *Inspector) GetQueueInfo(queue string) (*QueueInfo, error) {
if err := base.ValidateQueueName(queue); err != nil {
return nil, err
}
stats, err := i.rdb.CurrentStats(queue)
if err != nil {
return nil, err
}
return &QueueInfo{
Queue: stats.Queue,
MemoryUsage: stats.MemoryUsage,
Latency: stats.Latency,
Size: stats.Size,
Groups: stats.Groups,
Pending: stats.Pending,
Active: stats.Active,
Scheduled: stats.Scheduled,
Retry: stats.Retry,
Archived: stats.Archived,
Completed: stats.Completed,
Aggregating: stats.Aggregating,
Processed: stats.Processed,
Failed: stats.Failed,
ProcessedTotal: stats.ProcessedTotal,
FailedTotal: stats.FailedTotal,
Paused: stats.Paused,
Timestamp: stats.Timestamp,
}, nil
}
// DailyStats holds aggregate data for a given day for a given queue.
type DailyStats struct {
// Name of the queue.
Queue string
// Total number of tasks being processed during the given date.
// The number includes both succeeded and failed tasks.
Processed int
// Total number of tasks failed to be processed during the given date.
Failed int
// Date this stats was taken.
Date time.Time
}
// History returns a list of stats from the last n days.
func (i *Inspector) History(queue string, n int) ([]*DailyStats, error) {
if err := base.ValidateQueueName(queue); err != nil {
return nil, err
}
stats, err := i.rdb.HistoricalStats(queue, n)
if err != nil {
return nil, err
}
var res []*DailyStats
for _, s := range stats {
res = append(res, &DailyStats{
Queue: s.Queue,
Processed: s.Processed,
Failed: s.Failed,
Date: s.Time,
})
}
return res, nil
}
var (
// ErrQueueNotFound indicates that the specified queue does not exist.
ErrQueueNotFound = errors.New("queue not found")
// ErrQueueNotEmpty indicates that the specified queue is not empty.
ErrQueueNotEmpty = errors.New("queue is not empty")
// ErrTaskNotFound indicates that the specified task cannot be found in the queue.
ErrTaskNotFound = errors.New("task not found")
)
// DeleteQueue removes the specified queue.
//
// If force is set to true, DeleteQueue will remove the queue regardless of
// the queue size as long as no tasks are active in the queue.
// If force is set to false, DeleteQueue will remove the queue only if
// the queue is empty.
//
// If the specified queue does not exist, DeleteQueue returns ErrQueueNotFound.
// If force is set to false and the specified queue is not empty, DeleteQueue
// returns ErrQueueNotEmpty.
func (i *Inspector) DeleteQueue(queue string, force bool) error {
err := i.rdb.RemoveQueue(queue, force)
if errors.IsQueueNotFound(err) {
return fmt.Errorf("%w: queue=%q", ErrQueueNotFound, queue)
}
if errors.IsQueueNotEmpty(err) {
return fmt.Errorf("%w: queue=%q", ErrQueueNotEmpty, queue)
}
return err
}
// GetTaskInfo retrieves task information given a task id and queue name.
//
// Returns an error wrapping ErrQueueNotFound if a queue with the given name doesn't exist.
// Returns an error wrapping ErrTaskNotFound if a task with the given id doesn't exist in the queue.
func (i *Inspector) GetTaskInfo(queue, id string) (*TaskInfo, error) {
info, err := i.rdb.GetTaskInfo(queue, id)
switch {
case errors.IsQueueNotFound(err):
return nil, fmt.Errorf("asynq: %w", ErrQueueNotFound)
case errors.IsTaskNotFound(err):
return nil, fmt.Errorf("asynq: %w", ErrTaskNotFound)
case err != nil:
return nil, fmt.Errorf("asynq: %w", err)
}
return newTaskInfo(info.Message, info.State, info.NextProcessAt, info.Result), nil
}
// ListOption specifies behavior of list operation.
type ListOption interface{}
// Internal list option representations.
type (
pageSizeOpt int
pageNumOpt int
)
type listOption struct {
pageSize int
pageNum int
}
const (
// Page size used by default in list operation.
defaultPageSize = 30
// Page number used by default in list operation.
defaultPageNum = 1
)
func composeListOptions(opts ...ListOption) listOption {
res := listOption{
pageSize: defaultPageSize,
pageNum: defaultPageNum,
}
for _, opt := range opts {
switch opt := opt.(type) {
case pageSizeOpt:
res.pageSize = int(opt)
case pageNumOpt:
res.pageNum = int(opt)
default:
// ignore unexpected option
}
}
return res
}
// PageSize returns an option to specify the page size for list operation.
//
// Negative page size is treated as zero.
func PageSize(n int) ListOption {
if n < 0 {
n = 0
}
return pageSizeOpt(n)
}
// Page returns an option to specify the page number for list operation.
// The value 1 fetches the first page.
//
// Negative page number is treated as one.
func Page(n int) ListOption {
if n < 0 {
n = 1
}
return pageNumOpt(n)
}
// ListPendingTasks retrieves pending tasks from the specified queue.
//
// By default, it retrieves the first 30 tasks.
func (i *Inspector) ListPendingTasks(queue string, opts ...ListOption) ([]*TaskInfo, error) {
if err := base.ValidateQueueName(queue); err != nil {
return nil, fmt.Errorf("asynq: %w", err)
}
opt := composeListOptions(opts...)
pgn := rdb.Pagination{Size: opt.pageSize, Page: opt.pageNum - 1}
infos, err := i.rdb.ListPending(queue, pgn)
switch {
case errors.IsQueueNotFound(err):
return nil, fmt.Errorf("asynq: %w", ErrQueueNotFound)
case err != nil:
return nil, fmt.Errorf("asynq: %w", err)
}
var tasks []*TaskInfo
for _, i := range infos {
tasks = append(tasks, newTaskInfo(
i.Message,
i.State,
i.NextProcessAt,
i.Result,
))
}
return tasks, err
}
// ListActiveTasks retrieves active tasks from the specified queue.
//
// By default, it retrieves the first 30 tasks.
func (i *Inspector) ListActiveTasks(queue string, opts ...ListOption) ([]*TaskInfo, error) {
if err := base.ValidateQueueName(queue); err != nil {
return nil, fmt.Errorf("asynq: %w", err)
}
opt := composeListOptions(opts...)
pgn := rdb.Pagination{Size: opt.pageSize, Page: opt.pageNum - 1}
infos, err := i.rdb.ListActive(queue, pgn)
switch {
case errors.IsQueueNotFound(err):
return nil, fmt.Errorf("asynq: %w", ErrQueueNotFound)
case err != nil:
return nil, fmt.Errorf("asynq: %w", err)
}
expired, err := i.rdb.ListLeaseExpired(time.Now(), queue)
if err != nil {
return nil, fmt.Errorf("asynq: %w", err)
}
expiredSet := make(map[string]struct{}) // set of expired message IDs
for _, msg := range expired {
expiredSet[msg.ID] = struct{}{}
}
var tasks []*TaskInfo
for _, i := range infos {
t := newTaskInfo(
i.Message,
i.State,
i.NextProcessAt,
i.Result,
)
if _, ok := expiredSet[i.Message.ID]; ok {
t.IsOrphaned = true
}
tasks = append(tasks, t)
}
return tasks, nil
}
// ListAggregatingTasks retrieves scheduled tasks from the specified group.
//
// By default, it retrieves the first 30 tasks.
func (i *Inspector) ListAggregatingTasks(queue, group string, opts ...ListOption) ([]*TaskInfo, error) {
if err := base.ValidateQueueName(queue); err != nil {
return nil, fmt.Errorf("asynq: %w", err)
}
opt := composeListOptions(opts...)
pgn := rdb.Pagination{Size: opt.pageSize, Page: opt.pageNum - 1}
infos, err := i.rdb.ListAggregating(queue, group, pgn)
switch {
case errors.IsQueueNotFound(err):
return nil, fmt.Errorf("asynq: %w", ErrQueueNotFound)
case err != nil:
return nil, fmt.Errorf("asynq: %w", err)
}
var tasks []*TaskInfo
for _, i := range infos {
tasks = append(tasks, newTaskInfo(
i.Message,
i.State,
i.NextProcessAt,
i.Result,
))
}
return tasks, nil
}
// ListScheduledTasks retrieves scheduled tasks from the specified queue.
// Tasks are sorted by NextProcessAt in ascending order.
//
// By default, it retrieves the first 30 tasks.
func (i *Inspector) ListScheduledTasks(queue string, opts ...ListOption) ([]*TaskInfo, error) {
if err := base.ValidateQueueName(queue); err != nil {
return nil, fmt.Errorf("asynq: %w", err)
}
opt := composeListOptions(opts...)
pgn := rdb.Pagination{Size: opt.pageSize, Page: opt.pageNum - 1}
infos, err := i.rdb.ListScheduled(queue, pgn)
switch {
case errors.IsQueueNotFound(err):
return nil, fmt.Errorf("asynq: %w", ErrQueueNotFound)
case err != nil:
return nil, fmt.Errorf("asynq: %w", err)
}
var tasks []*TaskInfo
for _, i := range infos {
tasks = append(tasks, newTaskInfo(
i.Message,
i.State,
gitextract_6qfa6b_y/
├── .github/
│ ├── FUNDING.yml
│ ├── ISSUE_TEMPLATE/
│ │ ├── bug_report.md
│ │ └── feature_request.md
│ ├── dependabot.yaml
│ └── workflows/
│ ├── benchstat.yml
│ └── build.yml
├── .gitignore
├── CHANGELOG.md
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── LICENSE
├── Makefile
├── README.md
├── aggregator.go
├── aggregator_test.go
├── asynq.go
├── asynq_test.go
├── benchmark_test.go
├── client.go
├── client_test.go
├── context.go
├── doc.go
├── example_test.go
├── forwarder.go
├── forwarder_test.go
├── go.mod
├── go.sum
├── healthcheck.go
├── healthcheck_test.go
├── heartbeat.go
├── heartbeat_test.go
├── inspector.go
├── inspector_test.go
├── internal/
│ ├── base/
│ │ ├── base.go
│ │ └── base_test.go
│ ├── context/
│ │ ├── context.go
│ │ └── context_test.go
│ ├── errors/
│ │ ├── errors.go
│ │ └── errors_test.go
│ ├── log/
│ │ ├── log.go
│ │ └── log_test.go
│ ├── proto/
│ │ ├── asynq.pb.go
│ │ └── asynq.proto
│ ├── rdb/
│ │ ├── benchmark_test.go
│ │ ├── inspect.go
│ │ ├── inspect_test.go
│ │ ├── rdb.go
│ │ └── rdb_test.go
│ ├── testbroker/
│ │ └── testbroker.go
│ ├── testutil/
│ │ ├── builder.go
│ │ ├── builder_test.go
│ │ └── testutil.go
│ └── timeutil/
│ ├── timeutil.go
│ └── timeutil_test.go
├── janitor.go
├── janitor_test.go
├── periodic_task_manager.go
├── periodic_task_manager_test.go
├── processor.go
├── processor_test.go
├── recoverer.go
├── recoverer_test.go
├── scheduler.go
├── scheduler_test.go
├── servemux.go
├── servemux_test.go
├── server.go
├── server_test.go
├── signals_unix.go
├── signals_windows.go
├── subscriber.go
├── subscriber_test.go
├── syncer.go
├── syncer_test.go
├── tools/
│ ├── asynq/
│ │ ├── README.md
│ │ ├── cmd/
│ │ │ ├── cron.go
│ │ │ ├── dash/
│ │ │ │ ├── dash.go
│ │ │ │ ├── draw.go
│ │ │ │ ├── draw_test.go
│ │ │ │ ├── fetch.go
│ │ │ │ ├── key_event.go
│ │ │ │ ├── key_event_test.go
│ │ │ │ ├── screen_drawer.go
│ │ │ │ └── table.go
│ │ │ ├── dash.go
│ │ │ ├── group.go
│ │ │ ├── queue.go
│ │ │ ├── root.go
│ │ │ ├── server.go
│ │ │ ├── stats.go
│ │ │ └── task.go
│ │ └── main.go
│ ├── go.mod
│ ├── go.sum
│ └── metrics_exporter/
│ └── main.go
└── x/
├── go.mod
├── go.sum
├── metrics/
│ └── metrics.go
└── rate/
├── example_test.go
├── semaphore.go
└── semaphore_test.go
SYMBOL INDEX (1351 symbols across 79 files)
FILE: aggregator.go
type aggregator (line 18) | type aggregator struct
method shutdown (line 84) | func (a *aggregator) shutdown() {
method start (line 93) | func (a *aggregator) start(wg *sync.WaitGroup) {
method exec (line 119) | func (a *aggregator) exec(t time.Time) {
method aggregate (line 130) | func (a *aggregator) aggregate(t time.Time) {
type aggregatorParams (line 45) | type aggregatorParams struct
constant maxConcurrentAggregationChecks (line 57) | maxConcurrentAggregationChecks = 3
constant defaultAggregationCheckInterval (line 61) | defaultAggregationCheckInterval = 7 * time.Second
function newAggregator (line 64) | func newAggregator(params aggregatorParams) *aggregator {
FILE: aggregator_test.go
function TestAggregator (line 18) | func TestAggregator(t *testing.T) {
FILE: asynq.go
type Task (line 23) | type Task struct
method Type (line 40) | func (t *Task) Type() string { return t.typename }
method Payload (line 41) | func (t *Task) Payload() []byte { return t.payload }
method Headers (line 42) | func (t *Task) Headers() map[string]string { return t.headers }
method ResultWriter (line 48) | func (t *Task) ResultWriter() *ResultWriter { return t.w }
function NewTask (line 52) | func NewTask(typename string, payload []byte, opts ...Option) *Task {
function NewTaskWithHeaders (line 66) | func NewTaskWithHeaders(typename string, payload []byte, headers map[str...
function newTask (line 76) | func newTask(typename string, payload []byte, w *ResultWriter) *Task {
type TaskInfo (line 86) | type TaskInfo struct
function fromUnixTimeOrZero (line 159) | func fromUnixTimeOrZero(t int64) time.Time {
function newTaskInfo (line 166) | func newTaskInfo(msg *base.TaskMessage, state base.TaskState, nextProces...
type TaskState (line 208) | type TaskState
method String (line 233) | func (s TaskState) String() string {
constant TaskStateActive (line 212) | TaskStateActive TaskState = iota + 1
constant TaskStatePending (line 215) | TaskStatePending
constant TaskStateScheduled (line 218) | TaskStateScheduled
constant TaskStateRetry (line 221) | TaskStateRetry
constant TaskStateArchived (line 224) | TaskStateArchived
constant TaskStateCompleted (line 227) | TaskStateCompleted
constant TaskStateAggregating (line 230) | TaskStateAggregating
type RedisConnOpt (line 260) | type RedisConnOpt interface
type RedisClientOpt (line 268) | type RedisClientOpt struct
method MakeRedisClient (line 317) | func (opt RedisClientOpt) MakeRedisClient() interface{} {
type RedisFailoverClientOpt (line 335) | type RedisFailoverClientOpt struct
method MakeRedisClient (line 391) | func (opt RedisFailoverClientOpt) MakeRedisClient() interface{} {
type RedisClusterClientOpt (line 410) | type RedisClusterClientOpt struct
method MakeRedisClient (line 452) | func (opt RedisClusterClientOpt) MakeRedisClient() interface{} {
function ParseRedisURI (line 475) | func ParseRedisURI(uri string) (RedisConnOpt, error) {
function parseRedisURI (line 492) | func parseRedisURI(u *url.URL) (RedisConnOpt, error) {
function parseRedisSocketURI (line 524) | func parseRedisSocketURI(u *url.URL) (RedisConnOpt, error) {
function parseRedisSentinelURI (line 545) | func parseRedisSentinelURI(u *url.URL) (RedisConnOpt, error) {
type ResultWriter (line 557) | type ResultWriter struct
method Write (line 565) | func (w *ResultWriter) Write(data []byte) (n int, err error) {
method TaskID (line 575) | func (w *ResultWriter) TaskID() string {
FILE: asynq_test.go
function init (line 38) | func init() {
function setup (line 49) | func setup(tb testing.TB) (r redis.UniversalClient) {
function getRedisConnOpt (line 70) | func getRedisConnOpt(tb testing.TB) RedisConnOpt {
function TestParseRedisURI (line 95) | func TestParseRedisURI(t *testing.T) {
function TestParseRedisURIErrors (line 166) | func TestParseRedisURIErrors(t *testing.T) {
FILE: benchmark_test.go
function makeTask (line 19) | func makeTask(n int) *Task {
function BenchmarkEndToEndSimple (line 28) | func BenchmarkEndToEndSimple(b *testing.B) {
function BenchmarkEndToEnd (line 68) | func BenchmarkEndToEnd(b *testing.B) {
function BenchmarkEndToEndMultipleQueues (line 130) | func BenchmarkEndToEndMultipleQueues(b *testing.B) {
function BenchmarkClientWhileServerRunning (line 188) | func BenchmarkClientWhileServerRunning(b *testing.B) {
FILE: client.go
type Client (line 26) | type Client struct
method Close (line 320) | func (c *Client) Close() error {
method Enqueue (line 339) | func (c *Client) Enqueue(task *Task, opts ...Option) (*TaskInfo, error) {
method EnqueueContext (line 355) | func (c *Client) EnqueueContext(ctx context.Context, task *Task, opts ...
method Ping (line 424) | func (c *Client) Ping() error {
method enqueue (line 428) | func (c *Client) enqueue(ctx context.Context, msg *base.TaskMessage, u...
method schedule (line 435) | func (c *Client) schedule(ctx context.Context, msg *base.TaskMessage, ...
method addToGroup (line 443) | func (c *Client) addToGroup(ctx context.Context, msg *base.TaskMessage...
function NewClient (line 34) | func NewClient(r RedisConnOpt) *Client {
function NewClientFromRedisClient (line 46) | func NewClientFromRedisClient(c redis.UniversalClient) *Client {
type OptionType (line 50) | type OptionType
constant MaxRetryOpt (line 53) | MaxRetryOpt OptionType = iota
constant QueueOpt (line 54) | QueueOpt
constant TimeoutOpt (line 55) | TimeoutOpt
constant DeadlineOpt (line 56) | DeadlineOpt
constant UniqueOpt (line 57) | UniqueOpt
constant ProcessAtOpt (line 58) | ProcessAtOpt
constant ProcessInOpt (line 59) | ProcessInOpt
constant TaskIDOpt (line 60) | TaskIDOpt
constant RetentionOpt (line 61) | RetentionOpt
constant GroupOpt (line 62) | GroupOpt
type Option (line 66) | type Option interface
type retryOption (line 79) | type retryOption
method String (line 102) | func (n retryOption) String() string { return fmt.Sprintf("MaxRetr...
method Type (line 103) | func (n retryOption) Type() OptionType { return MaxRetryOpt }
method Value (line 104) | func (n retryOption) Value() interface{} { return int(n) }
type queueOption (line 80) | type queueOption
method String (line 111) | func (name queueOption) String() string { return fmt.Sprintf("Queu...
method Type (line 112) | func (name queueOption) Type() OptionType { return QueueOpt }
method Value (line 113) | func (name queueOption) Value() interface{} { return string(name) }
type taskIDOption (line 81) | type taskIDOption
method String (line 120) | func (id taskIDOption) String() string { return fmt.Sprintf("TaskI...
method Type (line 121) | func (id taskIDOption) Type() OptionType { return TaskIDOpt }
method Value (line 122) | func (id taskIDOption) Value() interface{} { return string(id) }
type timeoutOption (line 82) | type timeoutOption
method String (line 136) | func (d timeoutOption) String() string { return fmt.Sprintf("Timeo...
method Type (line 137) | func (d timeoutOption) Type() OptionType { return TimeoutOpt }
method Value (line 138) | func (d timeoutOption) Value() interface{} { return time.Duration(d) }
type deadlineOption (line 83) | type deadlineOption
method String (line 150) | func (t deadlineOption) String() string {
method Type (line 153) | func (t deadlineOption) Type() OptionType { return DeadlineOpt }
method Value (line 154) | func (t deadlineOption) Value() interface{} { return time.Time(t) }
type uniqueOption (line 84) | type uniqueOption
method String (line 171) | func (ttl uniqueOption) String() string { return fmt.Sprintf("Uniq...
method Type (line 172) | func (ttl uniqueOption) Type() OptionType { return UniqueOpt }
method Value (line 173) | func (ttl uniqueOption) Value() interface{} { return time.Duration(ttl) }
type processAtOption (line 85) | type processAtOption
method String (line 182) | func (t processAtOption) String() string {
method Type (line 185) | func (t processAtOption) Type() OptionType { return ProcessAtOpt }
method Value (line 186) | func (t processAtOption) Value() interface{} { return time.Time(t) }
type processInOption (line 86) | type processInOption
method String (line 195) | func (d processInOption) String() string { return fmt.Sprintf("Pro...
method Type (line 196) | func (d processInOption) Type() OptionType { return ProcessInOpt }
method Value (line 197) | func (d processInOption) Value() interface{} { return time.Duration(d) }
type retentionOption (line 87) | type retentionOption
method String (line 206) | func (ttl retentionOption) String() string { return fmt.Sprintf("R...
method Type (line 207) | func (ttl retentionOption) Type() OptionType { return RetentionOpt }
method Value (line 208) | func (ttl retentionOption) Value() interface{} { return time.Duration(...
type groupOption (line 88) | type groupOption
method String (line 216) | func (name groupOption) String() string { return fmt.Sprintf("Grou...
method Type (line 217) | func (name groupOption) Type() OptionType { return GroupOpt }
method Value (line 218) | func (name groupOption) Value() interface{} { return string(name) }
function MaxRetry (line 95) | func MaxRetry(n int) Option {
function Queue (line 107) | func Queue(name string) Option {
function TaskID (line 116) | func TaskID(id string) Option {
function Timeout (line 132) | func Timeout(d time.Duration) Option {
function Deadline (line 146) | func Deadline(t time.Time) Option {
function Unique (line 167) | func Unique(ttl time.Duration) Option {
function ProcessAt (line 178) | func ProcessAt(t time.Time) Option {
function ProcessIn (line 191) | func ProcessIn(d time.Duration) Option {
function Retention (line 202) | func Retention(d time.Duration) Option {
function Group (line 212) | func Group(name string) Option {
type option (line 230) | type option struct
function composeOptions (line 246) | func composeOptions(opts ...Option) (option, error) {
function isBlank (line 301) | func isBlank(s string) bool {
constant defaultMaxRetry (line 307) | defaultMaxRetry = 25
constant defaultTimeout (line 310) | defaultTimeout = 30 * time.Minute
FILE: client_test.go
function TestClientEnqueueWithProcessAtOption (line 20) | func TestClientEnqueueWithProcessAtOption(t *testing.T) {
function testClientEnqueue (line 147) | func testClientEnqueue(t *testing.T, client *Client, r redis.UniversalCl...
function TestClientEnqueue (line 478) | func TestClientEnqueue(t *testing.T) {
function TestClientFromRedisClientEnqueue (line 485) | func TestClientFromRedisClientEnqueue(t *testing.T) {
function TestClientEnqueueWithGroupOption (line 496) | func TestClientEnqueueWithGroupOption(t *testing.T) {
function TestClientEnqueueWithTaskIDOption (line 648) | func TestClientEnqueueWithTaskIDOption(t *testing.T) {
function TestClientEnqueueWithConflictingTaskID (line 725) | func TestClientEnqueueWithConflictingTaskID(t *testing.T) {
function TestClientEnqueueWithProcessInOption (line 742) | func TestClientEnqueueWithProcessInOption(t *testing.T) {
function TestClientEnqueueError (line 865) | func TestClientEnqueueError(t *testing.T) {
function TestClientWithDefaultOptions (line 926) | func TestClientWithDefaultOptions(t *testing.T) {
function TestClientEnqueueUnique (line 1060) | func TestClientEnqueueUnique(t *testing.T) {
function TestClientEnqueueUniqueWithProcessInOption (line 1103) | func TestClientEnqueueUniqueWithProcessInOption(t *testing.T) {
function TestClientEnqueueUniqueWithProcessAtOption (line 1149) | func TestClientEnqueueUniqueWithProcessAtOption(t *testing.T) {
function TestClientEnqueueWithHeaders (line 1195) | func TestClientEnqueueWithHeaders(t *testing.T) {
function TestClientEnqueueWithHeadersScheduled (line 1371) | func TestClientEnqueueWithHeadersScheduled(t *testing.T) {
function TestNewTaskWithHeaders (line 1457) | func TestNewTaskWithHeaders(t *testing.T) {
function TestTaskHeadersMethod (line 1531) | func TestTaskHeadersMethod(t *testing.T) {
function TestClientEnqueueWithHeadersAndGroup (line 1577) | func TestClientEnqueueWithHeadersAndGroup(t *testing.T) {
FILE: context.go
function GetTaskID (line 17) | func GetTaskID(ctx context.Context) (id string, ok bool) {
function GetRetryCount (line 25) | func GetRetryCount(ctx context.Context) (n int, ok bool) {
function GetMaxRetry (line 33) | func GetMaxRetry(ctx context.Context) (n int, ok bool) {
function GetQueueName (line 40) | func GetQueueName(ctx context.Context) (queue string, ok bool) {
FILE: example_test.go
function ExampleServer_Run (line 19) | func ExampleServer_Run() {
function ExampleServer_Shutdown (line 34) | func ExampleServer_Shutdown() {
function ExampleServer_Stop (line 54) | func ExampleServer_Stop() {
function ExampleScheduler (line 83) | func ExampleScheduler() {
function ExampleParseRedisURI (line 102) | func ExampleParseRedisURI() {
function ExampleResultWriter (line 118) | func ExampleResultWriter() {
FILE: forwarder.go
type forwarder (line 17) | type forwarder struct
method shutdown (line 48) | func (f *forwarder) shutdown() {
method start (line 55) | func (f *forwarder) start(wg *sync.WaitGroup) {
method exec (line 73) | func (f *forwarder) exec() {
type forwarderParams (line 31) | type forwarderParams struct
function newForwarder (line 38) | func newForwarder(params forwarderParams) *forwarder {
FILE: forwarder_test.go
function TestForwarder (line 18) | func TestForwarder(t *testing.T) {
FILE: healthcheck.go
type healthchecker (line 17) | type healthchecker struct
method shutdown (line 48) | func (hc *healthchecker) shutdown() {
method start (line 58) | func (hc *healthchecker) start(wg *sync.WaitGroup) {
type healthcheckerParams (line 31) | type healthcheckerParams struct
function newHealthChecker (line 38) | func newHealthChecker(params healthcheckerParams) *healthchecker {
FILE: healthcheck_test.go
function TestHealthChecker (line 16) | func TestHealthChecker(t *testing.T) {
function TestHealthCheckerWhenRedisDown (line 57) | func TestHealthCheckerWhenRedisDown(t *testing.T) {
FILE: heartbeat.go
type heartbeater (line 20) | type heartbeater struct
method shutdown (line 92) | func (h *heartbeater) shutdown() {
method start (line 110) | func (h *heartbeater) start(wg *sync.WaitGroup) {
method beat (line 145) | func (h *heartbeater) beat() {
type heartbeaterParams (line 53) | type heartbeaterParams struct
function newHeartbeater (line 65) | func newHeartbeater(params heartbeaterParams) *heartbeater {
type workerInfo (line 99) | type workerInfo struct
FILE: heartbeat_test.go
function TestHeartbeater (line 27) | func TestHeartbeater(t *testing.T) {
function TestHeartbeaterWithRedisDown (line 315) | func TestHeartbeaterWithRedisDown(t *testing.T) {
FILE: inspector.go
type Inspector (line 21) | type Inspector struct
method Close (line 49) | func (i *Inspector) Close() error {
method Queues (line 57) | func (i *Inspector) Queues() ([]string, error) {
method Groups (line 62) | func (i *Inspector) Groups(queue string) ([]*GroupInfo, error) {
method GetQueueInfo (line 140) | func (i *Inspector) GetQueueInfo(queue string) (*QueueInfo, error) {
method History (line 184) | func (i *Inspector) History(queue string, n int) ([]*DailyStats, error) {
method DeleteQueue (line 225) | func (i *Inspector) DeleteQueue(queue string, force bool) error {
method GetTaskInfo (line 240) | func (i *Inspector) GetTaskInfo(queue, id string) (*TaskInfo, error) {
method ListPendingTasks (line 317) | func (i *Inspector) ListPendingTasks(queue string, opts ...ListOption)...
method ListActiveTasks (line 345) | func (i *Inspector) ListActiveTasks(queue string, opts ...ListOption) ...
method ListAggregatingTasks (line 385) | func (i *Inspector) ListAggregatingTasks(queue, group string, opts ......
method ListScheduledTasks (line 414) | func (i *Inspector) ListScheduledTasks(queue string, opts ...ListOptio...
method ListRetryTasks (line 443) | func (i *Inspector) ListRetryTasks(queue string, opts ...ListOption) (...
method ListArchivedTasks (line 472) | func (i *Inspector) ListArchivedTasks(queue string, opts ...ListOption...
method ListCompletedTasks (line 501) | func (i *Inspector) ListCompletedTasks(queue string, opts ...ListOptio...
method DeleteAllPendingTasks (line 528) | func (i *Inspector) DeleteAllPendingTasks(queue string) (int, error) {
method DeleteAllScheduledTasks (line 538) | func (i *Inspector) DeleteAllScheduledTasks(queue string) (int, error) {
method DeleteAllRetryTasks (line 548) | func (i *Inspector) DeleteAllRetryTasks(queue string) (int, error) {
method DeleteAllArchivedTasks (line 558) | func (i *Inspector) DeleteAllArchivedTasks(queue string) (int, error) {
method DeleteAllCompletedTasks (line 568) | func (i *Inspector) DeleteAllCompletedTasks(queue string) (int, error) {
method DeleteAllAggregatingTasks (line 578) | func (i *Inspector) DeleteAllAggregatingTasks(queue, group string) (in...
method UpdateTaskPayload (line 593) | func (i *Inspector) UpdateTaskPayload(queue, id string, payload []byte...
method DeleteTask (line 617) | func (i *Inspector) DeleteTask(queue, id string) error {
method RunAllScheduledTasks (line 636) | func (i *Inspector) RunAllScheduledTasks(queue string) (int, error) {
method RunAllRetryTasks (line 646) | func (i *Inspector) RunAllRetryTasks(queue string) (int, error) {
method RunAllArchivedTasks (line 656) | func (i *Inspector) RunAllArchivedTasks(queue string) (int, error) {
method RunAllAggregatingTasks (line 666) | func (i *Inspector) RunAllAggregatingTasks(queue, group string) (int, ...
method RunTask (line 681) | func (i *Inspector) RunTask(queue, id string) error {
method ArchiveAllPendingTasks (line 699) | func (i *Inspector) ArchiveAllPendingTasks(queue string) (int, error) {
method ArchiveAllScheduledTasks (line 709) | func (i *Inspector) ArchiveAllScheduledTasks(queue string) (int, error) {
method ArchiveAllRetryTasks (line 719) | func (i *Inspector) ArchiveAllRetryTasks(queue string) (int, error) {
method ArchiveAllAggregatingTasks (line 729) | func (i *Inspector) ArchiveAllAggregatingTasks(queue, group string) (i...
method ArchiveTask (line 744) | func (i *Inspector) ArchiveTask(queue, id string) error {
method CancelProcessing (line 764) | func (i *Inspector) CancelProcessing(id string) error {
method PauseQueue (line 770) | func (i *Inspector) PauseQueue(queue string) error {
method UnpauseQueue (line 779) | func (i *Inspector) UnpauseQueue(queue string) error {
method Servers (line 787) | func (i *Inspector) Servers() ([]*ServerInfo, error) {
method ClusterKeySlot (line 873) | func (i *Inspector) ClusterKeySlot(queue string) (int64, error) {
method ClusterNodes (line 889) | func (i *Inspector) ClusterNodes(queue string) ([]*ClusterNode, error) {
method SchedulerEntries (line 925) | func (i *Inspector) SchedulerEntries() ([]*SchedulerEntry, error) {
method ListSchedulerEnqueueEvents (line 1038) | func (i *Inspector) ListSchedulerEnqueueEvents(entryID string, opts .....
function NewInspector (line 29) | func NewInspector(r RedisConnOpt) *Inspector {
function NewInspectorFromRedisClient (line 41) | func NewInspectorFromRedisClient(c redis.UniversalClient) *Inspector {
type GroupInfo (line 78) | type GroupInfo struct
type QueueInfo (line 87) | type QueueInfo struct
type DailyStats (line 171) | type DailyStats struct
type ListOption (line 254) | type ListOption interface
type pageSizeOpt (line 258) | type pageSizeOpt
type pageNumOpt (line 259) | type pageNumOpt
type listOption (line 262) | type listOption struct
constant defaultPageSize (line 269) | defaultPageSize = 30
constant defaultPageNum (line 272) | defaultPageNum = 1
function composeListOptions (line 275) | func composeListOptions(opts ...ListOption) listOption {
function PageSize (line 296) | func PageSize(n int) ListOption {
function Page (line 307) | func Page(n int) ListOption {
type ServerInfo (line 833) | type ServerInfo struct
type WorkerInfo (line 857) | type WorkerInfo struct
type ClusterNode (line 878) | type ClusterNode struct
type SchedulerEntry (line 902) | type SchedulerEntry struct
function parseOption (line 954) | func parseOption(s string) (Option, error) {
function parseOptionFunc (line 1010) | func parseOptionFunc(s string) string {
function parseOptionArg (line 1015) | func parseOptionArg(s string) string {
type SchedulerEnqueueEvent (line 1027) | type SchedulerEnqueueEvent struct
FILE: inspector_test.go
function testInspectorQueues (line 25) | func testInspectorQueues(t *testing.T, inspector *Inspector, r redis.Uni...
function TestInspectorQueues (line 53) | func TestInspectorQueues(t *testing.T) {
function TestInspectorFromRedisClientQueues (line 60) | func TestInspectorFromRedisClientQueues(t *testing.T) {
function TestInspectorDeleteQueue (line 68) | func TestInspectorDeleteQueue(t *testing.T) {
function TestInspectorDeleteQueueErrorQueueNotEmpty (line 157) | func TestInspectorDeleteQueueErrorQueueNotEmpty(t *testing.T) {
function TestInspectorDeleteQueueErrorQueueNotFound (line 213) | func TestInspectorDeleteQueueErrorQueueNotFound(t *testing.T) {
function TestInspectorGetQueueInfo (line 269) | func TestInspectorGetQueueInfo(t *testing.T) {
function TestInspectorHistory (line 424) | func TestInspectorHistory(t *testing.T) {
function createPendingTask (line 479) | func createPendingTask(msg *base.TaskMessage) *TaskInfo {
function TestInspectorGetTaskInfo (line 483) | func TestInspectorGetTaskInfo(t *testing.T) {
function TestInspectorGetTaskInfoError (line 607) | func TestInspectorGetTaskInfoError(t *testing.T) {
function TestInspectorListPendingTasks (line 687) | func TestInspectorListPendingTasks(t *testing.T) {
function newOrphanedTaskInfo (line 759) | func newOrphanedTaskInfo(msg *base.TaskMessage) *TaskInfo {
function TestInspectorListActiveTasks (line 765) | func TestInspectorListActiveTasks(t *testing.T) {
function createScheduledTask (line 846) | func createScheduledTask(z base.Z) *TaskInfo {
function TestInspectorListScheduledTasks (line 855) | func TestInspectorListScheduledTasks(t *testing.T) {
function createRetryTask (line 916) | func createRetryTask(z base.Z) *TaskInfo {
function TestInspectorListRetryTasks (line 925) | func TestInspectorListRetryTasks(t *testing.T) {
function createArchivedTask (line 987) | func createArchivedTask(z base.Z) *TaskInfo {
function TestInspectorListArchivedTasks (line 996) | func TestInspectorListArchivedTasks(t *testing.T) {
function newCompletedTaskMessage (line 1057) | func newCompletedTaskMessage(typename, qname string, retention time.Dura...
function createCompletedTask (line 1064) | func createCompletedTask(z base.Z) *TaskInfo {
function TestInspectorListCompletedTasks (line 1073) | func TestInspectorListCompletedTasks(t *testing.T) {
function TestInspectorListAggregatingTasks (line 1134) | func TestInspectorListAggregatingTasks(t *testing.T) {
function createAggregatingTaskInfo (line 1230) | func createAggregatingTaskInfo(msg *base.TaskMessage) *TaskInfo {
function TestInspectorListPagination (line 1234) | func TestInspectorListPagination(t *testing.T) {
function TestInspectorListTasksQueueNotFoundError (line 1298) | func TestInspectorListTasksQueueNotFoundError(t *testing.T) {
function TestInspectorDeleteAllPendingTasks (line 1341) | func TestInspectorDeleteAllPendingTasks(t *testing.T) {
function TestInspectorDeleteAllScheduledTasks (line 1405) | func TestInspectorDeleteAllScheduledTasks(t *testing.T) {
function TestInspectorDeleteAllRetryTasks (line 1471) | func TestInspectorDeleteAllRetryTasks(t *testing.T) {
function TestInspectorDeleteAllArchivedTasks (line 1537) | func TestInspectorDeleteAllArchivedTasks(t *testing.T) {
function TestInspectorDeleteAllCompletedTasks (line 1603) | func TestInspectorDeleteAllCompletedTasks(t *testing.T) {
function TestInspectorArchiveAllPendingTasks (line 1669) | func TestInspectorArchiveAllPendingTasks(t *testing.T) {
function TestInspectorArchiveAllScheduledTasks (line 1780) | func TestInspectorArchiveAllScheduledTasks(t *testing.T) {
function TestInspectorArchiveAllRetryTasks (line 1910) | func TestInspectorArchiveAllRetryTasks(t *testing.T) {
function TestInspectorRunAllScheduledTasks (line 2024) | func TestInspectorRunAllScheduledTasks(t *testing.T) {
function TestInspectorRunAllRetryTasks (line 2141) | func TestInspectorRunAllRetryTasks(t *testing.T) {
function TestInspectorRunAllArchivedTasks (line 2258) | func TestInspectorRunAllArchivedTasks(t *testing.T) {
function TestInspectorUpdateTaskPayloadUpdatesScheduledTaskPayload (line 2372) | func TestInspectorUpdateTaskPayloadUpdatesScheduledTaskPayload(t *testin...
function TestInspectorUpdateTaskPayloadError (line 2460) | func TestInspectorUpdateTaskPayloadError(t *testing.T) {
function TestInspectorDeleteTaskDeletesPendingTask (line 2514) | func TestInspectorDeleteTaskDeletesPendingTask(t *testing.T) {
function TestInspectorDeleteTaskDeletesScheduledTask (line 2574) | func TestInspectorDeleteTaskDeletesScheduledTask(t *testing.T) {
function TestInspectorDeleteTaskDeletesRetryTask (line 2624) | func TestInspectorDeleteTaskDeletesRetryTask(t *testing.T) {
function TestInspectorDeleteTaskDeletesArchivedTask (line 2674) | func TestInspectorDeleteTaskDeletesArchivedTask(t *testing.T) {
function TestInspectorDeleteTaskError (line 2724) | func TestInspectorDeleteTaskError(t *testing.T) {
function TestInspectorRunTaskRunsScheduledTask (line 2789) | func TestInspectorRunTaskRunsScheduledTask(t *testing.T) {
function TestInspectorRunTaskRunsRetryTask (line 2859) | func TestInspectorRunTaskRunsRetryTask(t *testing.T) {
function TestInspectorRunTaskRunsArchivedTask (line 2928) | func TestInspectorRunTaskRunsArchivedTask(t *testing.T) {
function TestInspectorRunTaskError (line 3001) | func TestInspectorRunTaskError(t *testing.T) {
function TestInspectorArchiveTaskArchivesPendingTask (line 3101) | func TestInspectorArchiveTaskArchivesPendingTask(t *testing.T) {
function TestInspectorArchiveTaskArchivesScheduledTask (line 3192) | func TestInspectorArchiveTaskArchivesScheduledTask(t *testing.T) {
function TestInspectorArchiveTaskArchivesRetryTask (line 3269) | func TestInspectorArchiveTaskArchivesRetryTask(t *testing.T) {
function TestInspectorArchiveTaskError (line 3344) | func TestInspectorArchiveTaskError(t *testing.T) {
function TestInspectorSchedulerEntries (line 3445) | func TestInspectorSchedulerEntries(t *testing.T) {
function TestParseOption (line 3514) | func TestParseOption(t *testing.T) {
function TestInspectorGroups (line 3583) | func TestInspectorGroups(t *testing.T) {
FILE: internal/base/base.go
constant Version (line 26) | Version = "0.26.0"
constant DefaultQueueName (line 29) | DefaultQueueName = "default"
constant AllServers (line 36) | AllServers = "asynq:servers"
constant AllWorkers (line 37) | AllWorkers = "asynq:workers"
constant AllSchedulers (line 38) | AllSchedulers = "asynq:schedulers"
constant AllQueues (line 39) | AllQueues = "asynq:queues"
constant CancelChannel (line 40) | CancelChannel = "asynq:cancel"
type TaskState (line 44) | type TaskState
method String (line 56) | func (s TaskState) String() string {
constant TaskStateActive (line 47) | TaskStateActive TaskState = iota + 1
constant TaskStatePending (line 48) | TaskStatePending
constant TaskStateScheduled (line 49) | TaskStateScheduled
constant TaskStateRetry (line 50) | TaskStateRetry
constant TaskStateArchived (line 51) | TaskStateArchived
constant TaskStateCompleted (line 52) | TaskStateCompleted
constant TaskStateAggregating (line 53) | TaskStateAggregating
function TaskStateFromString (line 76) | func TaskStateFromString(s string) (TaskState, error) {
function ValidateQueueName (line 98) | func ValidateQueueName(qname string) error {
function QueueKeyPrefix (line 106) | func QueueKeyPrefix(qname string) string {
function TaskKeyPrefix (line 111) | func TaskKeyPrefix(qname string) string {
function TaskKey (line 116) | func TaskKey(qname, id string) string {
function PendingKey (line 121) | func PendingKey(qname string) string {
function ActiveKey (line 126) | func ActiveKey(qname string) string {
function ScheduledKey (line 131) | func ScheduledKey(qname string) string {
function RetryKey (line 136) | func RetryKey(qname string) string {
function ArchivedKey (line 141) | func ArchivedKey(qname string) string {
function LeaseKey (line 146) | func LeaseKey(qname string) string {
function CompletedKey (line 150) | func CompletedKey(qname string) string {
function PausedKey (line 155) | func PausedKey(qname string) string {
function ProcessedTotalKey (line 160) | func ProcessedTotalKey(qname string) string {
function FailedTotalKey (line 165) | func FailedTotalKey(qname string) string {
function ProcessedKey (line 170) | func ProcessedKey(qname string, t time.Time) string {
function FailedKey (line 175) | func FailedKey(qname string, t time.Time) string {
function ServerInfoKey (line 180) | func ServerInfoKey(hostname string, pid int, serverID string) string {
function WorkersKey (line 185) | func WorkersKey(hostname string, pid int, serverID string) string {
function SchedulerEntriesKey (line 190) | func SchedulerEntriesKey(schedulerID string) string {
function SchedulerHistoryKey (line 195) | func SchedulerHistoryKey(entryID string) string {
function UniqueKey (line 200) | func UniqueKey(qname, tasktype string, payload []byte) string {
function GroupKeyPrefix (line 209) | func GroupKeyPrefix(qname string) string {
function GroupKey (line 214) | func GroupKey(qname, gkey string) string {
function AggregationSetKey (line 219) | func AggregationSetKey(qname, gname, setID string) string {
function AllGroups (line 224) | func AllGroups(qname string) string {
function AllAggregationSets (line 230) | func AllAggregationSets(qname string) string {
type TaskMessage (line 236) | type TaskMessage struct
function EncodeMessage (line 303) | func EncodeMessage(msg *TaskMessage) ([]byte, error) {
function DecodeMessage (line 327) | func DecodeMessage(data []byte) (*TaskMessage, error) {
type TaskInfo (line 352) | type TaskInfo struct
type Z (line 360) | type Z struct
type ServerInfo (line 366) | type ServerInfo struct
function EncodeServerInfo (line 379) | func EncodeServerInfo(info *ServerInfo) ([]byte, error) {
function DecodeServerInfo (line 403) | func DecodeServerInfo(b []byte) (*ServerInfo, error) {
type WorkerInfo (line 428) | type WorkerInfo struct
function EncodeWorkerInfo (line 441) | func EncodeWorkerInfo(info *WorkerInfo) ([]byte, error) {
function DecodeWorkerInfo (line 462) | func DecodeWorkerInfo(b []byte) (*WorkerInfo, error) {
type SchedulerEntry (line 484) | type SchedulerEntry struct
function EncodeSchedulerEntry (line 509) | func EncodeSchedulerEntry(entry *SchedulerEntry) ([]byte, error) {
function DecodeSchedulerEntry (line 528) | func DecodeSchedulerEntry(b []byte) (*SchedulerEntry, error) {
type SchedulerEnqueueEvent (line 548) | type SchedulerEnqueueEvent struct
function EncodeSchedulerEnqueueEvent (line 558) | func EncodeSchedulerEnqueueEvent(event *SchedulerEnqueueEvent) ([]byte, ...
function DecodeSchedulerEnqueueEvent (line 571) | func DecodeSchedulerEnqueueEvent(b []byte) (*SchedulerEnqueueEvent, erro...
type Cancelations (line 586) | type Cancelations struct
method Add (line 599) | func (c *Cancelations) Add(id string, fn context.CancelFunc) {
method Delete (line 606) | func (c *Cancelations) Delete(id string) {
method Get (line 613) | func (c *Cancelations) Get(id string) (fn context.CancelFunc, ok bool) {
function NewCancelations (line 592) | func NewCancelations() *Cancelations {
type Lease (line 622) | type Lease struct
method Reset (line 642) | func (l *Lease) Reset(expirationTime time.Time) bool {
method NotifyExpiration (line 654) | func (l *Lease) NotifyExpiration() bool {
method closeCh (line 662) | func (l *Lease) closeCh() {
method Done (line 667) | func (l *Lease) Done() <-chan struct{} {
method Deadline (line 672) | func (l *Lease) Deadline() time.Time {
method IsValid (line 680) | func (l *Lease) IsValid() bool {
function NewLease (line 632) | func NewLease(expirationTime time.Time) *Lease {
type Broker (line 690) | type Broker interface
FILE: internal/base/base_test.go
function TestTaskKey (line 22) | func TestTaskKey(t *testing.T) {
function TestQueueKey (line 41) | func TestQueueKey(t *testing.T) {
function TestActiveKey (line 58) | func TestActiveKey(t *testing.T) {
function TestLeaseKey (line 75) | func TestLeaseKey(t *testing.T) {
function TestScheduledKey (line 92) | func TestScheduledKey(t *testing.T) {
function TestRetryKey (line 109) | func TestRetryKey(t *testing.T) {
function TestArchivedKey (line 126) | func TestArchivedKey(t *testing.T) {
function TestCompletedKey (line 143) | func TestCompletedKey(t *testing.T) {
function TestPausedKey (line 160) | func TestPausedKey(t *testing.T) {
function TestProcessedTotalKey (line 177) | func TestProcessedTotalKey(t *testing.T) {
function TestFailedTotalKey (line 194) | func TestFailedTotalKey(t *testing.T) {
function TestProcessedKey (line 211) | func TestProcessedKey(t *testing.T) {
function TestFailedKey (line 230) | func TestFailedKey(t *testing.T) {
function TestServerInfoKey (line 249) | func TestServerInfoKey(t *testing.T) {
function TestWorkersKey (line 269) | func TestWorkersKey(t *testing.T) {
function TestSchedulerEntriesKey (line 289) | func TestSchedulerEntriesKey(t *testing.T) {
function TestSchedulerHistoryKey (line 306) | func TestSchedulerHistoryKey(t *testing.T) {
function toBytes (line 324) | func toBytes(m map[string]interface{}) []byte {
function TestUniqueKey (line 332) | func TestUniqueKey(t *testing.T) {
function TestGroupKey (line 398) | func TestGroupKey(t *testing.T) {
function TestAggregationSetKey (line 424) | func TestAggregationSetKey(t *testing.T) {
function TestAllGroups (line 453) | func TestAllGroups(t *testing.T) {
function TestAllAggregationSets (line 476) | func TestAllAggregationSets(t *testing.T) {
function TestMessageEncoding (line 499) | func TestMessageEncoding(t *testing.T) {
function TestServerInfoEncoding (line 551) | func TestServerInfoEncoding(t *testing.T) {
function TestWorkerInfoEncoding (line 588) | func TestWorkerInfoEncoding(t *testing.T) {
function TestSchedulerEntryEncoding (line 625) | func TestSchedulerEntryEncoding(t *testing.T) {
function TestSchedulerEnqueueEventEncoding (line 660) | func TestSchedulerEnqueueEventEncoding(t *testing.T) {
function TestCancelationsConcurrentAccess (line 692) | func TestCancelationsConcurrentAccess(t *testing.T) {
function TestLeaseReset (line 735) | func TestLeaseReset(t *testing.T) {
function TestLeaseNotifyExpiration (line 770) | func TestLeaseNotifyExpiration(t *testing.T) {
FILE: internal/context/context.go
type taskMetadata (line 15) | type taskMetadata struct
type ctxKey (line 24) | type ctxKey
constant metadataCtxKey (line 28) | metadataCtxKey ctxKey = 0
function New (line 31) | func New(base context.Context, msg *base.TaskMessage, deadline time.Time...
function GetTaskID (line 46) | func GetTaskID(ctx context.Context) (id string, ok bool) {
function GetRetryCount (line 58) | func GetRetryCount(ctx context.Context) (n int, ok bool) {
function GetMaxRetry (line 70) | func GetMaxRetry(ctx context.Context) (n int, ok bool) {
function GetQueueName (line 81) | func GetQueueName(ctx context.Context) (qname string, ok bool) {
FILE: internal/context/context_test.go
function TestCreateContextWithFutureDeadline (line 18) | func TestCreateContextWithFutureDeadline(t *testing.T) {
function TestCreateContextWithBaseContext (line 57) | func TestCreateContextWithBaseContext(t *testing.T) {
function TestCreateContextWithPastDeadline (line 104) | func TestCreateContextWithPastDeadline(t *testing.T) {
function TestGetTaskMetadataFromContext (line 137) | func TestGetTaskMetadataFromContext(t *testing.T) {
function TestGetTaskMetadataFromContextError (line 185) | func TestGetTaskMetadataFromContextError(t *testing.T) {
FILE: internal/errors/errors.go
type Error (line 23) | type Error struct
method DebugString (line 29) | func (e *Error) DebugString() string {
method Error (line 49) | func (e *Error) Error() string {
method Unwrap (line 63) | func (e *Error) Unwrap() error {
type Code (line 68) | type Code
method String (line 81) | func (c Code) String() string {
constant Unspecified (line 72) | Unspecified Code = iota
constant NotFound (line 73) | NotFound
constant FailedPrecondition (line 74) | FailedPrecondition
constant Internal (line 75) | Internal
constant AlreadyExists (line 76) | AlreadyExists
constant Unknown (line 77) | Unknown
type Op (line 101) | type Op
function E (line 124) | func E(args ...interface{}) error {
function CanonicalCode (line 150) | func CanonicalCode(err error) Code {
type TaskNotFoundError (line 181) | type TaskNotFoundError struct
method Error (line 186) | func (e *TaskNotFoundError) Error() string {
function IsTaskNotFound (line 191) | func IsTaskNotFound(err error) bool {
type QueueNotFoundError (line 197) | type QueueNotFoundError struct
method Error (line 201) | func (e *QueueNotFoundError) Error() string {
function IsQueueNotFound (line 206) | func IsQueueNotFound(err error) bool {
type QueueNotEmptyError (line 212) | type QueueNotEmptyError struct
method Error (line 216) | func (e *QueueNotEmptyError) Error() string {
function IsQueueNotEmpty (line 221) | func IsQueueNotEmpty(err error) bool {
type TaskAlreadyArchivedError (line 227) | type TaskAlreadyArchivedError struct
method Error (line 232) | func (e *TaskAlreadyArchivedError) Error() string {
function IsTaskAlreadyArchived (line 237) | func IsTaskAlreadyArchived(err error) bool {
type RedisCommandError (line 243) | type RedisCommandError struct
method Error (line 248) | func (e *RedisCommandError) Error() string {
method Unwrap (line 252) | func (e *RedisCommandError) Unwrap() error { return e.Err }
function IsRedisCommandError (line 255) | func IsRedisCommandError(err error) bool {
type PanicError (line 261) | type PanicError struct
method Error (line 265) | func (e *PanicError) Error() string {
function IsPanicError (line 270) | func IsPanicError(err error) bool {
function New (line 284) | func New(text string) error { return errors.New(text) }
function Is (line 290) | func Is(err, target error) bool { return errors.Is(err, target) }
function As (line 297) | func As(err error, target interface{}) bool { return errors.As(err, targ...
function Unwrap (line 304) | func Unwrap(err error) error { return errors.Unwrap(err) }
FILE: internal/errors/errors_test.go
function TestErrorDebugString (line 9) | func TestErrorDebugString(t *testing.T) {
function TestErrorString (line 36) | func TestErrorString(t *testing.T) {
function TestErrorIs (line 63) | func TestErrorIs(t *testing.T) {
function TestErrorAs (line 87) | func TestErrorAs(t *testing.T) {
function TestErrorPredicates (line 109) | func TestErrorPredicates(t *testing.T) {
function TestCanonicalCode (line 149) | func TestCanonicalCode(t *testing.T) {
FILE: internal/log/log.go
type Base (line 17) | type Base interface
type baseLogger (line 37) | type baseLogger struct
method Debug (line 42) | func (l *baseLogger) Debug(args ...interface{}) {
method Info (line 47) | func (l *baseLogger) Info(args ...interface{}) {
method Warn (line 52) | func (l *baseLogger) Warn(args ...interface{}) {
method Error (line 57) | func (l *baseLogger) Error(args ...interface{}) {
method Fatal (line 63) | func (l *baseLogger) Fatal(args ...interface{}) {
method prefixPrint (line 68) | func (l *baseLogger) prefixPrint(prefix string, args ...interface{}) {
function newBase (line 74) | func newBase(out io.Writer) *baseLogger {
function NewLogger (line 83) | func NewLogger(base Base) *Logger {
type Logger (line 91) | type Logger struct
method canLogAt (line 145) | func (l *Logger) canLogAt(v Level) bool {
method Debug (line 151) | func (l *Logger) Debug(args ...interface{}) {
method Info (line 158) | func (l *Logger) Info(args ...interface{}) {
method Warn (line 165) | func (l *Logger) Warn(args ...interface{}) {
method Error (line 172) | func (l *Logger) Error(args ...interface{}) {
method Fatal (line 179) | func (l *Logger) Fatal(args ...interface{}) {
method Debugf (line 186) | func (l *Logger) Debugf(format string, args ...interface{}) {
method Infof (line 190) | func (l *Logger) Infof(format string, args ...interface{}) {
method Warnf (line 194) | func (l *Logger) Warnf(format string, args ...interface{}) {
method Errorf (line 198) | func (l *Logger) Errorf(format string, args ...interface{}) {
method Fatalf (line 202) | func (l *Logger) Fatalf(format string, args ...interface{}) {
method SetLevel (line 208) | func (l *Logger) SetLevel(v Level) {
type Level (line 101) | type Level
method String (line 127) | func (l Level) String() string {
constant DebugLevel (line 106) | DebugLevel Level = iota
constant InfoLevel (line 109) | InfoLevel
constant WarnLevel (line 113) | WarnLevel
constant ErrorLevel (line 117) | ErrorLevel
constant FatalLevel (line 121) | FatalLevel
FILE: internal/log/log_test.go
constant rgxPID (line 16) | rgxPID = `[0-9]+`
constant rgxdate (line 17) | rgxdate = `[0-9][0-9][0-9][0-9]/[0-9][0-9]/[0-9][0-9]`
constant rgxtime (line 18) | rgxtime = `[0-9][0-9]:[0-9][0-9]:[0-9][0-9]`
constant rgxmicroseconds (line 19) | rgxmicroseconds = `\.[0-9][0-9][0-9][0-9][0-9][0-9]`
type tester (line 22) | type tester struct
function TestLoggerDebug (line 28) | func TestLoggerDebug(t *testing.T) {
function TestLoggerInfo (line 62) | func TestLoggerInfo(t *testing.T) {
function TestLoggerWarn (line 96) | func TestLoggerWarn(t *testing.T) {
function TestLoggerError (line 130) | func TestLoggerError(t *testing.T) {
type formatTester (line 164) | type formatTester struct
function TestLoggerDebugf (line 171) | func TestLoggerDebugf(t *testing.T) {
function TestLoggerInfof (line 200) | func TestLoggerInfof(t *testing.T) {
function TestLoggerWarnf (line 229) | func TestLoggerWarnf(t *testing.T) {
function TestLoggerErrorf (line 258) | func TestLoggerErrorf(t *testing.T) {
function TestLoggerWithLowerLevels (line 287) | func TestLoggerWithLowerLevels(t *testing.T) {
function TestLoggerWithSameOrHigherLevels (line 340) | func TestLoggerWithSameOrHigherLevels(t *testing.T) {
FILE: internal/proto/asynq.pb.go
constant _ (line 24) | _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
constant _ (line 26) | _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
type TaskMessage (line 32) | type TaskMessage struct
method Reset (line 79) | func (x *TaskMessage) Reset() {
method String (line 86) | func (x *TaskMessage) String() string {
method ProtoMessage (line 90) | func (*TaskMessage) ProtoMessage() {}
method ProtoReflect (line 92) | func (x *TaskMessage) ProtoReflect() protoreflect.Message {
method Descriptor (line 105) | func (*TaskMessage) Descriptor() ([]byte, []int) {
method GetType (line 109) | func (x *TaskMessage) GetType() string {
method GetPayload (line 116) | func (x *TaskMessage) GetPayload() []byte {
method GetHeaders (line 123) | func (x *TaskMessage) GetHeaders() map[string]string {
method GetId (line 130) | func (x *TaskMessage) GetId() string {
method GetQueue (line 137) | func (x *TaskMessage) GetQueue() string {
method GetRetry (line 144) | func (x *TaskMessage) GetRetry() int32 {
method GetRetried (line 151) | func (x *TaskMessage) GetRetried() int32 {
method GetErrorMsg (line 158) | func (x *TaskMessage) GetErrorMsg() string {
method GetLastFailedAt (line 165) | func (x *TaskMessage) GetLastFailedAt() int64 {
method GetTimeout (line 172) | func (x *TaskMessage) GetTimeout() int64 {
method GetDeadline (line 179) | func (x *TaskMessage) GetDeadline() int64 {
method GetUniqueKey (line 186) | func (x *TaskMessage) GetUniqueKey() string {
method GetGroupKey (line 193) | func (x *TaskMessage) GetGroupKey() string {
method GetRetention (line 200) | func (x *TaskMessage) GetRetention() int64 {
method GetCompletedAt (line 207) | func (x *TaskMessage) GetCompletedAt() int64 {
type ServerInfo (line 215) | type ServerInfo struct
method Reset (line 242) | func (x *ServerInfo) Reset() {
method String (line 249) | func (x *ServerInfo) String() string {
method ProtoMessage (line 253) | func (*ServerInfo) ProtoMessage() {}
method ProtoReflect (line 255) | func (x *ServerInfo) ProtoReflect() protoreflect.Message {
method Descriptor (line 268) | func (*ServerInfo) Descriptor() ([]byte, []int) {
method GetHost (line 272) | func (x *ServerInfo) GetHost() string {
method GetPid (line 279) | func (x *ServerInfo) GetPid() int32 {
method GetServerId (line 286) | func (x *ServerInfo) GetServerId() string {
method GetConcurrency (line 293) | func (x *ServerInfo) GetConcurrency() int32 {
method GetQueues (line 300) | func (x *ServerInfo) GetQueues() map[string]int32 {
method GetStrictPriority (line 307) | func (x *ServerInfo) GetStrictPriority() bool {
method GetStatus (line 314) | func (x *ServerInfo) GetStatus() string {
method GetStartTime (line 321) | func (x *ServerInfo) GetStartTime() *timestamppb.Timestamp {
method GetActiveWorkerCount (line 328) | func (x *ServerInfo) GetActiveWorkerCount() int32 {
type WorkerInfo (line 336) | type WorkerInfo struct
method Reset (line 361) | func (x *WorkerInfo) Reset() {
method String (line 368) | func (x *WorkerInfo) String() string {
method ProtoMessage (line 372) | func (*WorkerInfo) ProtoMessage() {}
method ProtoReflect (line 374) | func (x *WorkerInfo) ProtoReflect() protoreflect.Message {
method Descriptor (line 387) | func (*WorkerInfo) Descriptor() ([]byte, []int) {
method GetHost (line 391) | func (x *WorkerInfo) GetHost() string {
method GetPid (line 398) | func (x *WorkerInfo) GetPid() int32 {
method GetServerId (line 405) | func (x *WorkerInfo) GetServerId() string {
method GetTaskId (line 412) | func (x *WorkerInfo) GetTaskId() string {
method GetTaskType (line 419) | func (x *WorkerInfo) GetTaskType() string {
method GetTaskPayload (line 426) | func (x *WorkerInfo) GetTaskPayload() []byte {
method GetQueue (line 433) | func (x *WorkerInfo) GetQueue() string {
method GetStartTime (line 440) | func (x *WorkerInfo) GetStartTime() *timestamppb.Timestamp {
method GetDeadline (line 447) | func (x *WorkerInfo) GetDeadline() *timestamppb.Timestamp {
type SchedulerEntry (line 456) | type SchedulerEntry struct
method Reset (line 477) | func (x *SchedulerEntry) Reset() {
method String (line 484) | func (x *SchedulerEntry) String() string {
method ProtoMessage (line 488) | func (*SchedulerEntry) ProtoMessage() {}
method ProtoReflect (line 490) | func (x *SchedulerEntry) ProtoReflect() protoreflect.Message {
method Descriptor (line 503) | func (*SchedulerEntry) Descriptor() ([]byte, []int) {
method GetId (line 507) | func (x *SchedulerEntry) GetId() string {
method GetSpec (line 514) | func (x *SchedulerEntry) GetSpec() string {
method GetTaskType (line 521) | func (x *SchedulerEntry) GetTaskType() string {
method GetTaskPayload (line 528) | func (x *SchedulerEntry) GetTaskPayload() []byte {
method GetEnqueueOptions (line 535) | func (x *SchedulerEntry) GetEnqueueOptions() []string {
method GetNextEnqueueTime (line 542) | func (x *SchedulerEntry) GetNextEnqueueTime() *timestamppb.Timestamp {
method GetPrevEnqueueTime (line 549) | func (x *SchedulerEntry) GetPrevEnqueueTime() *timestamppb.Timestamp {
type SchedulerEnqueueEvent (line 558) | type SchedulerEnqueueEvent struct
method Reset (line 568) | func (x *SchedulerEnqueueEvent) Reset() {
method String (line 575) | func (x *SchedulerEnqueueEvent) String() string {
method ProtoMessage (line 579) | func (*SchedulerEnqueueEvent) ProtoMessage() {}
method ProtoReflect (line 581) | func (x *SchedulerEnqueueEvent) ProtoReflect() protoreflect.Message {
method Descriptor (line 594) | func (*SchedulerEnqueueEvent) Descriptor() ([]byte, []int) {
method GetTaskId (line 598) | func (x *SchedulerEnqueueEvent) GetTaskId() string {
method GetEnqueueTime (line 605) | func (x *SchedulerEnqueueEvent) GetEnqueueTime() *timestamppb.Timestamp {
constant file_asynq_proto_rawDesc (line 614) | file_asynq_proto_rawDesc = "" +
function file_asynq_proto_rawDescGZIP (line 682) | func file_asynq_proto_rawDescGZIP() []byte {
function init (line 716) | func init() { file_asynq_proto_init() }
function file_asynq_proto_init (line 717) | func file_asynq_proto_init() {
FILE: internal/rdb/benchmark_test.go
function BenchmarkEnqueue (line 17) | func BenchmarkEnqueue(b *testing.B) {
function BenchmarkEnqueueUnique (line 34) | func BenchmarkEnqueueUnique(b *testing.B) {
function BenchmarkSchedule (line 57) | func BenchmarkSchedule(b *testing.B) {
function BenchmarkScheduleUnique (line 75) | func BenchmarkScheduleUnique(b *testing.B) {
function BenchmarkDequeueSingleQueue (line 99) | func BenchmarkDequeueSingleQueue(b *testing.B) {
function BenchmarkDequeueMultipleQueues (line 122) | func BenchmarkDequeueMultipleQueues(b *testing.B) {
function BenchmarkDone (line 148) | func BenchmarkDone(b *testing.B) {
function BenchmarkRetry (line 175) | func BenchmarkRetry(b *testing.B) {
function BenchmarkArchive (line 202) | func BenchmarkArchive(b *testing.B) {
function BenchmarkRequeue (line 229) | func BenchmarkRequeue(b *testing.B) {
function BenchmarkCheckAndEnqueue (line 256) | func BenchmarkCheckAndEnqueue(b *testing.B) {
FILE: internal/rdb/inspect.go
method AllQueues (line 20) | func (r *RDB) AllQueues() ([]string, error) {
type Stats (line 25) | type Stats struct
type DailyStats (line 68) | type DailyStats struct
method CurrentStats (line 140) | func (r *RDB) CurrentStats(qname string) (*Stats, error) {
method memoryUsage (line 318) | func (r *RDB) memoryUsage(qname string) (int64, error) {
method HistoricalStats (line 363) | func (r *RDB) HistoricalStats(qname string, n int) ([]*DailyStats, error) {
method RedisInfo (line 406) | func (r *RDB) RedisInfo() (map[string]string, error) {
method RedisClusterInfo (line 415) | func (r *RDB) RedisClusterInfo() (map[string]string, error) {
function parseInfo (line 423) | func parseInfo(infoStr string) (map[string]string, error) {
function reverse (line 436) | func reverse(x []*base.TaskInfo) {
method checkQueueExists (line 445) | func (r *RDB) checkQueueExists(qname string) error {
method GetTaskInfo (line 485) | func (r *RDB) GetTaskInfo(qname, id string) (*base.TaskInfo, error) {
type GroupStat (line 550) | type GroupStat struct
method GroupStats (line 578) | func (r *RDB) GroupStats(qname string) ([]*GroupStat, error) {
type Pagination (line 602) | type Pagination struct
method start (line 610) | func (p Pagination) start() int64 {
method stop (line 614) | func (p Pagination) stop() int64 {
method ListPending (line 619) | func (r *RDB) ListPending(qname string, pgn Pagination) ([]*base.TaskInf...
method ListActive (line 636) | func (r *RDB) ListActive(qname string, pgn Pagination) ([]*base.TaskInfo...
method listMessages (line 669) | func (r *RDB) listMessages(qname string, state base.TaskState, pgn Pagin...
method ListScheduled (line 720) | func (r *RDB) ListScheduled(qname string, pgn Pagination) ([]*base.TaskI...
method ListRetry (line 738) | func (r *RDB) ListRetry(qname string, pgn Pagination) ([]*base.TaskInfo,...
method ListArchived (line 755) | func (r *RDB) ListArchived(qname string, pgn Pagination) ([]*base.TaskIn...
method ListCompleted (line 772) | func (r *RDB) ListCompleted(qname string, pgn Pagination) ([]*base.TaskI...
method ListAggregating (line 789) | func (r *RDB) ListAggregating(qname, gname string, pgn Pagination) ([]*b...
method queueExists (line 806) | func (r *RDB) queueExists(qname string) (bool, error) {
method listZSetEntries (line 834) | func (r *RDB) listZSetEntries(qname string, state base.TaskState, key st...
method RunAllScheduledTasks (line 883) | func (r *RDB) RunAllScheduledTasks(qname string) (int64, error) {
method RunAllRetryTasks (line 898) | func (r *RDB) RunAllRetryTasks(qname string) (int64, error) {
method RunAllArchivedTasks (line 913) | func (r *RDB) RunAllArchivedTasks(qname string) (int64, error) {
method RunAllAggregatingTasks (line 951) | func (r *RDB) RunAllAggregatingTasks(qname, gname string) (int64, error) {
method RunTask (line 1028) | func (r *RDB) RunTask(qname, id string) error {
method runAll (line 1085) | func (r *RDB) runAll(zset, qname string) (int64, error) {
method ArchiveAllRetryTasks (line 1113) | func (r *RDB) ArchiveAllRetryTasks(qname string) (int64, error) {
method ArchiveAllScheduledTasks (line 1128) | func (r *RDB) ArchiveAllScheduledTasks(qname string) (int64, error) {
method ArchiveAllAggregatingTasks (line 1171) | func (r *RDB) ArchiveAllAggregatingTasks(qname, gname string) (int64, er...
method ArchiveAllPendingTasks (line 1228) | func (r *RDB) ArchiveAllPendingTasks(qname string) (int64, error) {
method ArchiveTask (line 1317) | func (r *RDB) ArchiveTask(qname, id string) error {
method archiveAll (line 1385) | func (r *RDB) archiveAll(src, dst, qname string) (int64, error) {
method UpdateTaskPayload (line 1458) | func (r *RDB) UpdateTaskPayload(qname, id string, payload []byte) error {
method DeleteTask (line 1552) | func (r *RDB) DeleteTask(qname, id string) error {
method DeleteAllArchivedTasks (line 1588) | func (r *RDB) DeleteAllArchivedTasks(qname string) (int64, error) {
method DeleteAllRetryTasks (line 1602) | func (r *RDB) DeleteAllRetryTasks(qname string) (int64, error) {
method DeleteAllScheduledTasks (line 1616) | func (r *RDB) DeleteAllScheduledTasks(qname string) (int64, error) {
method DeleteAllCompletedTasks (line 1630) | func (r *RDB) DeleteAllCompletedTasks(qname string) (int64, error) {
method deleteAll (line 1664) | func (r *RDB) deleteAll(key, qname string) (int64, error) {
method DeleteAllAggregatingTasks (line 1703) | func (r *RDB) DeleteAllAggregatingTasks(qname, gname string) (int64, err...
method DeleteAllPendingTasks (line 1746) | func (r *RDB) DeleteAllPendingTasks(qname string) (int64, error) {
method RemoveQueue (line 1886) | func (r *RDB) RemoveQueue(qname string, force bool) error {
method ListServers (line 1941) | func (r *RDB) ListServers() ([]*base.ServerInfo, error) {
method ListWorkers (line 1974) | func (r *RDB) ListWorkers() ([]*base.WorkerInfo, error) {
method ListSchedulerEntries (line 2010) | func (r *RDB) ListSchedulerEntries() ([]*base.SchedulerEntry, error) {
method ListSchedulerEnqueueEvents (line 2038) | func (r *RDB) ListSchedulerEnqueueEvents(entryID string, pgn Pagination)...
method Pause (line 2060) | func (r *RDB) Pause(qname string) error {
method Unpause (line 2073) | func (r *RDB) Unpause(qname string) error {
method ClusterKeySlot (line 2086) | func (r *RDB) ClusterKeySlot(qname string) (int64, error) {
method ClusterNodes (line 2092) | func (r *RDB) ClusterNodes(qname string) ([]redis.ClusterNode, error) {
FILE: internal/rdb/inspect_test.go
function TestAllQueues (line 25) | func TestAllQueues(t *testing.T) {
function TestCurrentStats (line 56) | func TestCurrentStats(t *testing.T) {
function TestCurrentStatsWithNonExistentQueue (line 332) | func TestCurrentStatsWithNonExistentQueue(t *testing.T) {
function TestHistoricalStats (line 343) | func TestHistoricalStats(t *testing.T) {
function TestRedisInfo (line 397) | func TestRedisInfo(t *testing.T) {
function TestGroupStats (line 422) | func TestGroupStats(t *testing.T) {
function TestGetTaskInfo (line 517) | func TestGetTaskInfo(t *testing.T) {
function TestGetTaskInfoError (line 660) | func TestGetTaskInfoError(t *testing.T) {
function TestListPending (line 738) | func TestListPending(t *testing.T) {
function TestListPendingPagination (line 811) | func TestListPendingPagination(t *testing.T) {
function TestListActive (line 878) | func TestListActive(t *testing.T) {
function TestListActivePagination (line 930) | func TestListActivePagination(t *testing.T) {
function TestListScheduled (line 987) | func TestListScheduled(t *testing.T) {
function TestListScheduledPagination (line 1065) | func TestListScheduledPagination(t *testing.T) {
function TestListRetry (line 1123) | func TestListRetry(t *testing.T) {
function TestListRetryPagination (line 1220) | func TestListRetryPagination(t *testing.T) {
function TestListArchived (line 1282) | func TestListArchived(t *testing.T) {
function TestListArchivedPagination (line 1373) | func TestListArchivedPagination(t *testing.T) {
function TestListCompleted (line 1432) | func TestListCompleted(t *testing.T) {
function TestListCompletedPagination (line 1513) | func TestListCompletedPagination(t *testing.T) {
function TestListAggregating (line 1572) | func TestListAggregating(t *testing.T) {
function TestListAggregatingPagination (line 1657) | func TestListAggregatingPagination(t *testing.T) {
function TestListTasksError (line 1786) | func TestListTasksError(t *testing.T) {
function TestRunArchivedTask (line 1827) | func TestRunArchivedTask(t *testing.T) {
function TestRunRetryTask (line 1907) | func TestRunRetryTask(t *testing.T) {
function TestRunAggregatingTask (line 1987) | func TestRunAggregatingTask(t *testing.T) {
function TestRunScheduledTask (line 2092) | func TestRunScheduledTask(t *testing.T) {
function TestRunTaskError (line 2172) | func TestRunTaskError(t *testing.T) {
function TestRunAllScheduledTasks (line 2327) | func TestRunAllScheduledTasks(t *testing.T) {
function TestRunAllRetryTasks (line 2433) | func TestRunAllRetryTasks(t *testing.T) {
function TestRunAllArchivedTasks (line 2539) | func TestRunAllArchivedTasks(t *testing.T) {
function TestRunAllTasksError (line 2645) | func TestRunAllTasksError(t *testing.T) {
function TestRunAllAggregatingTasks (line 2677) | func TestRunAllAggregatingTasks(t *testing.T) {
function TestArchiveRetryTask (line 2786) | func TestArchiveRetryTask(t *testing.T) {
function TestArchiveScheduledTask (line 2887) | func TestArchiveScheduledTask(t *testing.T) {
function TestArchiveAggregatingTask (line 2988) | func TestArchiveAggregatingTask(t *testing.T) {
function TestArchivePendingTask (line 3097) | func TestArchivePendingTask(t *testing.T) {
function TestArchiveTaskError (line 3180) | func TestArchiveTaskError(t *testing.T) {
function TestArchiveAllPendingTasks (line 3336) | func TestArchiveAllPendingTasks(t *testing.T) {
function TestArchiveAllAggregatingTasks (line 3472) | func TestArchiveAllAggregatingTasks(t *testing.T) {
function TestArchiveAllRetryTasks (line 3586) | func TestArchiveAllRetryTasks(t *testing.T) {
function TestArchiveAllScheduledTasks (line 3736) | func TestArchiveAllScheduledTasks(t *testing.T) {
function TestArchiveAllTasksError (line 3886) | func TestArchiveAllTasksError(t *testing.T) {
function TestDeleteArchivedTask (line 3915) | func TestDeleteArchivedTask(t *testing.T) {
function TestDeleteRetryTask (line 3981) | func TestDeleteRetryTask(t *testing.T) {
function TestDeleteScheduledTask (line 4047) | func TestDeleteScheduledTask(t *testing.T) {
function TestDeleteAggregatingTask (line 4113) | func TestDeleteAggregatingTask(t *testing.T) {
function TestDeletePendingTask (line 4208) | func TestDeletePendingTask(t *testing.T) {
function TestDeleteTaskWithUniqueLock (line 4263) | func TestDeleteTaskWithUniqueLock(t *testing.T) {
function TestDeleteTaskError (line 4319) | func TestDeleteTaskError(t *testing.T) {
function TestDeleteAllArchivedTasks (line 4418) | func TestDeleteAllArchivedTasks(t *testing.T) {
function newCompletedTaskMessage (line 4480) | func newCompletedTaskMessage(qname, typename string, retention time.Dura...
function TestDeleteAllCompletedTasks (line 4487) | func TestDeleteAllCompletedTasks(t *testing.T) {
function TestDeleteAllArchivedTasksWithUniqueKey (line 4550) | func TestDeleteAllArchivedTasksWithUniqueKey(t *testing.T) {
function TestDeleteAllRetryTasks (line 4623) | func TestDeleteAllRetryTasks(t *testing.T) {
function TestDeleteAllScheduledTasks (line 4685) | func TestDeleteAllScheduledTasks(t *testing.T) {
function TestDeleteAllAggregatingTasks (line 4747) | func TestDeleteAllAggregatingTasks(t *testing.T) {
function TestDeleteAllPendingTasks (line 4846) | func TestDeleteAllPendingTasks(t *testing.T) {
function TestDeleteAllTasksError (line 4903) | func TestDeleteAllTasksError(t *testing.T) {
function TestRemoveQueue (line 4935) | func TestRemoveQueue(t *testing.T) {
function TestRemoveQueueError (line 5040) | func TestRemoveQueueError(t *testing.T) {
function TestListServers (line 5188) | func TestListServers(t *testing.T) {
function TestListWorkers (line 5250) | func TestListWorkers(t *testing.T) {
function TestWriteListClearSchedulerEntries (line 5327) | func TestWriteListClearSchedulerEntries(t *testing.T) {
function TestSchedulerEnqueueEvents (line 5373) | func TestSchedulerEnqueueEvents(t *testing.T) {
function TestRecordSchedulerEnqueueEventTrimsDataSet (line 5431) | func TestRecordSchedulerEnqueueEventTrimsDataSet(t *testing.T) {
function TestPause (line 5478) | func TestPause(t *testing.T) {
function TestPauseError (line 5502) | func TestPauseError(t *testing.T) {
function TestUnpause (line 5528) | func TestUnpause(t *testing.T) {
function TestUnpauseError (line 5557) | func TestUnpauseError(t *testing.T) {
FILE: internal/rdb/rdb.go
constant statsTTL (line 23) | statsTTL = 90 * 24 * time.Hour
constant LeaseDuration (line 26) | LeaseDuration = 30 * time.Second
type RDB (line 29) | type RDB struct
method Close (line 44) | func (r *RDB) Close() error {
method Client (line 49) | func (r *RDB) Client() redis.UniversalClient {
method SetClock (line 56) | func (r *RDB) SetClock(c timeutil.Clock) {
method Ping (line 61) | func (r *RDB) Ping() error {
method runScript (line 65) | func (r *RDB) runScript(ctx context.Context, op errors.Op, script *red...
method runScriptWithErrorCode (line 73) | func (r *RDB) runScriptWithErrorCode(ctx context.Context, op errors.Op...
method Enqueue (line 111) | func (r *RDB) Enqueue(ctx context.Context, msg *base.TaskMessage) error {
method EnqueueUnique (line 176) | func (r *RDB) EnqueueUnique(ctx context.Context, msg *base.TaskMessage...
method Dequeue (line 244) | func (r *RDB) Dequeue(qnames ...string) (msg *base.TaskMessage, leaseE...
method Done (line 346) | func (r *RDB) Done(ctx context.Context, msg *base.TaskMessage) error {
method MarkAsComplete (line 448) | func (r *RDB) MarkAsComplete(ctx context.Context, msg *base.TaskMessag...
method Requeue (line 498) | func (r *RDB) Requeue(ctx context.Context, msg *base.TaskMessage) error {
method AddToGroup (line 534) | func (r *RDB) AddToGroup(ctx context.Context, msg *base.TaskMessage, g...
method AddToGroupUnique (line 599) | func (r *RDB) AddToGroupUnique(ctx context.Context, msg *base.TaskMess...
method Schedule (line 659) | func (r *RDB) Schedule(ctx context.Context, msg *base.TaskMessage, pro...
method ScheduleUnique (line 721) | func (r *RDB) ScheduleUnique(ctx context.Context, msg *base.TaskMessag...
method Retry (line 804) | func (r *RDB) Retry(ctx context.Context, msg *base.TaskMessage, proces...
method Archive (line 906) | func (r *RDB) Archive(ctx context.Context, msg *base.TaskMessage, errM...
method ForwardIfReady (line 943) | func (r *RDB) ForwardIfReady(qnames ...string) error {
method forward (line 983) | func (r *RDB) forward(delayedKey, pendingKey, taskKeyPrefix, groupKeyP...
method forwardAll (line 1005) | func (r *RDB) forwardAll(qname string) (err error) {
method ListGroups (line 1023) | func (r *RDB) ListGroups(qname string) ([]string, error) {
method AggregationCheck (line 1126) | func (r *RDB) AggregationCheck(qname, gname string, t time.Time, grace...
method ReadAggregationSet (line 1179) | func (r *RDB) ReadAggregationSet(qname, gname, setID string) ([]*base....
method DeleteAggregationSet (line 1229) | func (r *RDB) DeleteAggregationSet(ctx context.Context, qname, gname, ...
method ReclaimStaleAggregationSets (line 1258) | func (r *RDB) ReclaimStaleAggregationSets(qname string) error {
method DeleteExpiredCompletedTasks (line 1280) | func (r *RDB) DeleteExpiredCompletedTasks(qname string, batchSize int)...
method deleteExpiredCompletedTasks (line 1294) | func (r *RDB) deleteExpiredCompletedTasks(qname string, batchSize int)...
method ListLeaseExpired (line 1330) | func (r *RDB) ListLeaseExpired(cutoff time.Time, qnames ...string) ([]...
method ExtendLease (line 1357) | func (r *RDB) ExtendLease(qname string, ids ...string) (expirationTime...
method WriteServerState (line 1389) | func (r *RDB) WriteServerState(info *base.ServerInfo, workers []*base....
method ClearServerState (line 1424) | func (r *RDB) ClearServerState(host string, pid int, serverID string) ...
method WriteSchedulerEntries (line 1450) | func (r *RDB) WriteSchedulerEntries(schedulerID string, entries []*bas...
method ClearSchedulerEntries (line 1471) | func (r *RDB) ClearSchedulerEntries(schedulerID string) error {
method CancelationPubSub (line 1485) | func (r *RDB) CancelationPubSub() (*redis.PubSub, error) {
method PublishCancelation (line 1498) | func (r *RDB) PublishCancelation(id string) error {
method RecordSchedulerEnqueueEvent (line 1520) | func (r *RDB) RecordSchedulerEnqueueEvent(entryID string, event *base....
method ClearSchedulerHistory (line 1539) | func (r *RDB) ClearSchedulerHistory(entryID string) error {
method WriteResult (line 1550) | func (r *RDB) WriteResult(qname, taskID string, data []byte) (int, err...
function NewRDB (line 36) | func NewRDB(client redis.UniversalClient) *RDB {
constant maxArchiveSize (line 840) | maxArchiveSize = 10000
constant archivedExpirationInDays (line 841) | archivedExpirationInDays = 90
constant aggregationTimeout (line 1117) | aggregationTimeout = 2 * time.Minute
constant maxEvents (line 1517) | maxEvents = 1000
FILE: internal/rdb/rdb_test.go
function init (line 37) | func init() {
function setup (line 44) | func setup(tb testing.TB) (r *RDB) {
function TestEnqueue (line 65) | func TestEnqueue(t *testing.T) {
function TestEnqueueTaskIdConflictError (line 127) | func TestEnqueueTaskIdConflictError(t *testing.T) {
function TestEnqueueQueueCache (line 163) | func TestEnqueueQueueCache(t *testing.T) {
function TestEnqueueUnique (line 216) | func TestEnqueueUnique(t *testing.T) {
function TestEnqueueUniqueTaskIdConflictError (line 310) | func TestEnqueueUniqueTaskIdConflictError(t *testing.T) {
function TestDequeue (line 349) | func TestDequeue(t *testing.T) {
function TestDequeueError (line 497) | func TestDequeueError(t *testing.T) {
function TestDequeueIgnoresPausedQueues (line 587) | func TestDequeueIgnoresPausedQueues(t *testing.T) {
function TestDone (line 700) | func TestDone(t *testing.T) {
function TestDoneWithMaxCounter (line 856) | func TestDoneWithMaxCounter(t *testing.T) {
function TestMarkAsComplete (line 891) | func TestMarkAsComplete(t *testing.T) {
function TestRequeue (line 1077) | func TestRequeue(t *testing.T) {
function TestAddToGroup (line 1223) | func TestAddToGroup(t *testing.T) {
function TestAddToGroupeTaskIdConflictError (line 1290) | func TestAddToGroupeTaskIdConflictError(t *testing.T) {
function TestAddToGroupUnique (line 1330) | func TestAddToGroupUnique(t *testing.T) {
function TestAddToGroupUniqueTaskIdConflictError (line 1413) | func TestAddToGroupUniqueTaskIdConflictError(t *testing.T) {
function TestSchedule (line 1454) | func TestSchedule(t *testing.T) {
function TestScheduleTaskIdConflictError (line 1512) | func TestScheduleTaskIdConflictError(t *testing.T) {
function TestScheduleUnique (line 1551) | func TestScheduleUnique(t *testing.T) {
function TestScheduleUniqueTaskIdConflictError (line 1637) | func TestScheduleUniqueTaskIdConflictError(t *testing.T) {
function TestRetry (line 1677) | func TestRetry(t *testing.T) {
function TestRetryWithNonFailureError (line 1848) | func TestRetryWithNonFailureError(t *testing.T) {
function TestArchive (line 2015) | func TestArchive(t *testing.T) {
function TestArchiveTrim (line 2226) | func TestArchiveTrim(t *testing.T) {
function TestForwardIfReadyWithGroup (line 2383) | func TestForwardIfReadyWithGroup(t *testing.T) {
function TestForwardIfReady (line 2527) | func TestForwardIfReady(t *testing.T) {
function newCompletedTask (line 2681) | func newCompletedTask(qname, typename string, payload []byte, completedA...
function TestDeleteExpiredCompletedTasks (line 2687) | func TestDeleteExpiredCompletedTasks(t *testing.T) {
function TestListLeaseExpired (line 2768) | func TestListLeaseExpired(t *testing.T) {
function TestExtendLease (line 2852) | func TestExtendLease(t *testing.T) {
function TestWriteServerState (line 2960) | func TestWriteServerState(t *testing.T) {
function TestWriteServerStateWithWorkers (line 3026) | func TestWriteServerStateWithWorkers(t *testing.T) {
function TestClearServerState (line 3134) | func TestClearServerState(t *testing.T) {
function TestCancelationPubSub (line 3235) | func TestCancelationPubSub(t *testing.T) {
function TestWriteResult (line 3277) | func TestWriteResult(t *testing.T) {
function TestAggregationCheck (line 3313) | func TestAggregationCheck(t *testing.T) {
function TestDeleteAggregationSet (line 3667) | func TestDeleteAggregationSet(t *testing.T) {
function TestDeleteAggregationSetError (line 3795) | func TestDeleteAggregationSetError(t *testing.T) {
function TestReclaimStaleAggregationSets (line 3882) | func TestReclaimStaleAggregationSets(t *testing.T) {
function TestListGroups (line 3969) | func TestListGroups(t *testing.T) {
FILE: internal/testbroker/testbroker.go
type TestBroker (line 22) | type TestBroker struct
method Sleep (line 37) | func (tb *TestBroker) Sleep() {
method Wakeup (line 43) | func (tb *TestBroker) Wakeup() {
method Enqueue (line 49) | func (tb *TestBroker) Enqueue(ctx context.Context, msg *base.TaskMessa...
method EnqueueUnique (line 58) | func (tb *TestBroker) EnqueueUnique(ctx context.Context, msg *base.Tas...
method Dequeue (line 67) | func (tb *TestBroker) Dequeue(qnames ...string) (*base.TaskMessage, ti...
method Done (line 76) | func (tb *TestBroker) Done(ctx context.Context, msg *base.TaskMessage)...
method MarkAsComplete (line 85) | func (tb *TestBroker) MarkAsComplete(ctx context.Context, msg *base.Ta...
method Requeue (line 94) | func (tb *TestBroker) Requeue(ctx context.Context, msg *base.TaskMessa...
method Schedule (line 103) | func (tb *TestBroker) Schedule(ctx context.Context, msg *base.TaskMess...
method ScheduleUnique (line 112) | func (tb *TestBroker) ScheduleUnique(ctx context.Context, msg *base.Ta...
method Retry (line 121) | func (tb *TestBroker) Retry(ctx context.Context, msg *base.TaskMessage...
method Archive (line 130) | func (tb *TestBroker) Archive(ctx context.Context, msg *base.TaskMessa...
method ForwardIfReady (line 139) | func (tb *TestBroker) ForwardIfReady(qnames ...string) error {
method DeleteExpiredCompletedTasks (line 148) | func (tb *TestBroker) DeleteExpiredCompletedTasks(qname string, batchS...
method ListLeaseExpired (line 157) | func (tb *TestBroker) ListLeaseExpired(cutoff time.Time, qnames ...str...
method ExtendLease (line 166) | func (tb *TestBroker) ExtendLease(qname string, ids ...string) (time.T...
method WriteServerState (line 175) | func (tb *TestBroker) WriteServerState(info *base.ServerInfo, workers ...
method ClearServerState (line 184) | func (tb *TestBroker) ClearServerState(host string, pid int, serverID ...
method CancelationPubSub (line 193) | func (tb *TestBroker) CancelationPubSub() (*redis.PubSub, error) {
method PublishCancelation (line 202) | func (tb *TestBroker) PublishCancelation(id string) error {
method WriteResult (line 211) | func (tb *TestBroker) WriteResult(qname, id string, data []byte) (int,...
method Ping (line 220) | func (tb *TestBroker) Ping() error {
method Close (line 229) | func (tb *TestBroker) Close() error {
method AddToGroup (line 238) | func (tb *TestBroker) AddToGroup(ctx context.Context, msg *base.TaskMe...
method AddToGroupUnique (line 247) | func (tb *TestBroker) AddToGroupUnique(ctx context.Context, msg *base....
method ListGroups (line 256) | func (tb *TestBroker) ListGroups(qname string) ([]string, error) {
method AggregationCheck (line 265) | func (tb *TestBroker) AggregationCheck(qname, gname string, t time.Tim...
method ReadAggregationSet (line 274) | func (tb *TestBroker) ReadAggregationSet(qname, gname, aggregationSetI...
method DeleteAggregationSet (line 283) | func (tb *TestBroker) DeleteAggregationSet(ctx context.Context, qname,...
method ReclaimStaleAggregationSets (line 292) | func (tb *TestBroker) ReclaimStaleAggregationSets(qname string) error {
function NewTestBroker (line 33) | func NewTestBroker(b base.Broker) *TestBroker {
FILE: internal/testutil/builder.go
function makeDefaultTaskMessage (line 14) | func makeDefaultTaskMessage() *base.TaskMessage {
type TaskMessageBuilder (line 25) | type TaskMessageBuilder struct
method lazyInit (line 33) | func (b *TaskMessageBuilder) lazyInit() {
method Build (line 39) | func (b *TaskMessageBuilder) Build() *base.TaskMessage {
method SetType (line 44) | func (b *TaskMessageBuilder) SetType(typename string) *TaskMessageBuil...
method SetPayload (line 50) | func (b *TaskMessageBuilder) SetPayload(payload []byte) *TaskMessageBu...
method SetQueue (line 56) | func (b *TaskMessageBuilder) SetQueue(qname string) *TaskMessageBuilder {
method SetRetry (line 62) | func (b *TaskMessageBuilder) SetRetry(n int) *TaskMessageBuilder {
method SetTimeout (line 68) | func (b *TaskMessageBuilder) SetTimeout(timeout time.Duration) *TaskMe...
method SetDeadline (line 74) | func (b *TaskMessageBuilder) SetDeadline(deadline time.Time) *TaskMess...
method SetGroup (line 80) | func (b *TaskMessageBuilder) SetGroup(gname string) *TaskMessageBuilder {
function NewTaskMessageBuilder (line 29) | func NewTaskMessageBuilder() *TaskMessageBuilder {
FILE: internal/testutil/builder_test.go
function TestTaskMessageBuilder (line 16) | func TestTaskMessageBuilder(t *testing.T) {
FILE: internal/testutil/testutil.go
function EquateInt64Approx (line 26) | func EquateInt64Approx(margin int64) cmp.Option {
function NewTaskMessage (line 114) | func NewTaskMessage(taskType string, payload []byte) *base.TaskMessage {
function NewTaskMessageWithQueue (line 120) | func NewTaskMessageWithQueue(taskType string, payload []byte, qname stri...
function NewLeaseWithClock (line 133) | func NewLeaseWithClock(expirationTime time.Time, clock timeutil.Clock) *...
function JSON (line 140) | func JSON(kv map[string]interface{}) []byte {
function TaskMessageAfterRetry (line 150) | func TaskMessageAfterRetry(t base.TaskMessage, errMsg string, failedAt t...
function TaskMessageWithError (line 158) | func TaskMessageWithError(t base.TaskMessage, errMsg string, failedAt ti...
function TaskMessageWithCompletedAt (line 165) | func TaskMessageWithCompletedAt(t base.TaskMessage, completedAt time.Tim...
function MustMarshal (line 172) | func MustMarshal(tb testing.TB, msg *base.TaskMessage) string {
function MustUnmarshal (line 183) | func MustUnmarshal(tb testing.TB, data string) *base.TaskMessage {
function FlushDB (line 193) | func FlushDB(tb testing.TB, r redis.UniversalClient) {
function SeedPendingQueue (line 214) | func SeedPendingQueue(tb testing.TB, r redis.UniversalClient, msgs []*ba...
function SeedActiveQueue (line 221) | func SeedActiveQueue(tb testing.TB, r redis.UniversalClient, msgs []*bas...
function SeedScheduledQueue (line 228) | func SeedScheduledQueue(tb testing.TB, r redis.UniversalClient, entries ...
function SeedRetryQueue (line 235) | func SeedRetryQueue(tb testing.TB, r redis.UniversalClient, entries []ba...
function SeedArchivedQueue (line 242) | func SeedArchivedQueue(tb testing.TB, r redis.UniversalClient, entries [...
function SeedLease (line 249) | func SeedLease(tb testing.TB, r redis.UniversalClient, entries []base.Z,...
function SeedCompletedQueue (line 256) | func SeedCompletedQueue(tb testing.TB, r redis.UniversalClient, entries ...
function SeedGroup (line 263) | func SeedGroup(tb testing.TB, r redis.UniversalClient, entries []base.Z,...
function SeedAggregationSet (line 271) | func SeedAggregationSet(tb testing.TB, r redis.UniversalClient, entries ...
function SeedAllPendingQueues (line 280) | func SeedAllPendingQueues(tb testing.TB, r redis.UniversalClient, pendin...
function SeedAllActiveQueues (line 288) | func SeedAllActiveQueues(tb testing.TB, r redis.UniversalClient, active ...
function SeedAllScheduledQueues (line 296) | func SeedAllScheduledQueues(tb testing.TB, r redis.UniversalClient, sche...
function SeedAllRetryQueues (line 304) | func SeedAllRetryQueues(tb testing.TB, r redis.UniversalClient, retry ma...
function SeedAllArchivedQueues (line 312) | func SeedAllArchivedQueues(tb testing.TB, r redis.UniversalClient, archi...
function SeedAllLease (line 320) | func SeedAllLease(tb testing.TB, r redis.UniversalClient, lease map[stri...
function SeedAllCompletedQueues (line 328) | func SeedAllCompletedQueues(tb testing.TB, r redis.UniversalClient, comp...
function SeedAllGroups (line 338) | func SeedAllGroups(tb testing.TB, r redis.UniversalClient, groups map[st...
function seedRedisList (line 347) | func seedRedisList(tb testing.TB, c redis.UniversalClient, key string,
function seedRedisZSet (line 374) | func seedRedisZSet(tb testing.TB, c redis.UniversalClient, key string,
function GetPendingMessages (line 405) | func GetPendingMessages(tb testing.TB, r redis.UniversalClient, qname st...
function GetActiveMessages (line 412) | func GetActiveMessages(tb testing.TB, r redis.UniversalClient, qname str...
function GetScheduledMessages (line 419) | func GetScheduledMessages(tb testing.TB, r redis.UniversalClient, qname ...
function GetRetryMessages (line 426) | func GetRetryMessages(tb testing.TB, r redis.UniversalClient, qname stri...
function GetArchivedMessages (line 433) | func GetArchivedMessages(tb testing.TB, r redis.UniversalClient, qname s...
function GetCompletedMessages (line 440) | func GetCompletedMessages(tb testing.TB, r redis.UniversalClient, qname ...
function GetScheduledEntries (line 447) | func GetScheduledEntries(tb testing.TB, r redis.UniversalClient, qname s...
function GetRetryEntries (line 454) | func GetRetryEntries(tb testing.TB, r redis.UniversalClient, qname strin...
function GetArchivedEntries (line 461) | func GetArchivedEntries(tb testing.TB, r redis.UniversalClient, qname st...
function GetLeaseEntries (line 468) | func GetLeaseEntries(tb testing.TB, r redis.UniversalClient, qname strin...
function GetCompletedEntries (line 475) | func GetCompletedEntries(tb testing.TB, r redis.UniversalClient, qname s...
function GetGroupEntries (line 482) | func GetGroupEntries(tb testing.TB, r redis.UniversalClient, qname, grou...
function getMessagesFromList (line 489) | func getMessagesFromList(tb testing.TB, r redis.UniversalClient, qname s...
function getMessagesFromZSet (line 506) | func getMessagesFromZSet(tb testing.TB, r redis.UniversalClient, qname s...
function getMessagesFromZSetWithScores (line 523) | func getMessagesFromZSetWithScores(tb testing.TB, r redis.UniversalClient,
type TaskSeedData (line 541) | type TaskSeedData struct
function SeedTasks (line 547) | func SeedTasks(tb testing.TB, r redis.UniversalClient, taskData []*TaskS...
function SeedRedisZSets (line 573) | func SeedRedisZSets(tb testing.TB, r redis.UniversalClient, zsets map[st...
function SeedRedisSets (line 584) | func SeedRedisSets(tb testing.TB, r redis.UniversalClient, sets map[stri...
function SeedRedisSet (line 590) | func SeedRedisSet(tb testing.TB, r redis.UniversalClient, key string, me...
function SeedRedisLists (line 598) | func SeedRedisLists(tb testing.TB, r redis.UniversalClient, lists map[st...
function AssertRedisLists (line 608) | func AssertRedisLists(t *testing.T, r redis.UniversalClient, wantLists m...
function AssertRedisSets (line 620) | func AssertRedisSets(t *testing.T, r redis.UniversalClient, wantSets map...
function AssertRedisZSets (line 632) | func AssertRedisZSets(t *testing.T, r redis.UniversalClient, wantZSets m...
FILE: internal/timeutil/timeutil.go
type Clock (line 21) | type Clock interface
function NewRealClock (line 25) | func NewRealClock() Clock { return &realTimeClock{} }
type realTimeClock (line 27) | type realTimeClock struct
method Now (line 29) | func (_ *realTimeClock) Now() time.Time { return time.Now() }
type SimulatedClock (line 34) | type SimulatedClock struct
method Now (line 43) | func (c *SimulatedClock) Now() time.Time {
method SetTime (line 49) | func (c *SimulatedClock) SetTime(t time.Time) {
method AdvanceTime (line 55) | func (c *SimulatedClock) AdvanceTime(d time.Duration) {
function NewSimulatedClock (line 39) | func NewSimulatedClock(t time.Time) *SimulatedClock {
FILE: internal/timeutil/timeutil_test.go
function TestSimulatedClock (line 12) | func TestSimulatedClock(t *testing.T) {
FILE: janitor.go
type janitor (line 18) | type janitor struct
method shutdown (line 54) | func (j *janitor) shutdown() {
method start (line 61) | func (j *janitor) start(wg *sync.WaitGroup) {
method exec (line 79) | func (j *janitor) exec() {
type janitorParams (line 35) | type janitorParams struct
function newJanitor (line 43) | func newJanitor(params janitorParams) *janitor {
FILE: janitor_test.go
function newCompletedTask (line 18) | func newCompletedTask(qname, tasktype string, payload []byte, completedA...
function TestJanitor (line 24) | func TestJanitor(t *testing.T) {
FILE: periodic_task_manager.go
type PeriodicTaskManager (line 19) | type PeriodicTaskManager struct
method Start (line 120) | func (mgr *PeriodicTaskManager) Start() error {
method Shutdown (line 150) | func (mgr *PeriodicTaskManager) Shutdown() {
method Run (line 158) | func (mgr *PeriodicTaskManager) Run() error {
method initialSync (line 168) | func (mgr *PeriodicTaskManager) initialSync() error {
method add (line 182) | func (mgr *PeriodicTaskManager) add(configs []*PeriodicTaskConfig) {
method remove (line 196) | func (mgr *PeriodicTaskManager) remove(removed map[string]string) {
method sync (line 207) | func (mgr *PeriodicTaskManager) sync() {
method diffRemoved (line 228) | func (mgr *PeriodicTaskManager) diffRemoved(configs []*PeriodicTaskCon...
method diffAdded (line 245) | func (mgr *PeriodicTaskManager) diffAdded(configs []*PeriodicTaskConfi...
type PeriodicTaskManagerOpts (line 28) | type PeriodicTaskManagerOpts struct
constant defaultSyncInterval (line 45) | defaultSyncInterval = 3 * time.Minute
function NewPeriodicTaskManager (line 49) | func NewPeriodicTaskManager(opts PeriodicTaskManagerOpts) (*PeriodicTask...
type PeriodicTaskConfigProvider (line 79) | type PeriodicTaskConfigProvider interface
type PeriodicTaskConfig (line 84) | type PeriodicTaskConfig struct
method hash (line 90) | func (c *PeriodicTaskConfig) hash() string {
function validatePeriodicTaskConfig (line 103) | func validatePeriodicTaskConfig(c *PeriodicTaskConfig) error {
FILE: periodic_task_manager_test.go
type FakeConfigProvider (line 17) | type FakeConfigProvider struct
method SetConfigs (line 22) | func (p *FakeConfigProvider) SetConfigs(cfgs []*PeriodicTaskConfig) {
method GetConfigs (line 28) | func (p *FakeConfigProvider) GetConfigs() ([]*PeriodicTaskConfig, erro...
function TestNewPeriodicTaskManager (line 34) | func TestNewPeriodicTaskManager(t *testing.T) {
function TestPeriodicTaskConfigHash (line 107) | func TestPeriodicTaskConfigHash(t *testing.T) {
function TestPeriodicTaskManager (line 252) | func TestPeriodicTaskManager(t *testing.T) {
function extractCronEntries (line 319) | func extractCronEntries(s *Scheduler) []*cronEntry {
type cronEntry (line 337) | type cronEntry struct
FILE: processor.go
type processor (line 27) | type processor struct
method stop (line 127) | func (p *processor) stop() {
method shutdown (line 139) | func (p *processor) shutdown() {
method start (line 152) | func (p *processor) start(wg *sync.WaitGroup) {
method exec (line 170) | func (p *processor) exec() {
method requeue (line 261) | func (p *processor) requeue(l *base.Lease, msg *base.TaskMessage) {
method handleSucceededMessage (line 276) | func (p *processor) handleSucceededMessage(l *base.Lease, msg *base.Ta...
method markAsComplete (line 284) | func (p *processor) markAsComplete(l *base.Lease, msg *base.TaskMessag...
method markAsDone (line 306) | func (p *processor) markAsDone(l *base.Lease, msg *base.TaskMessage) {
method handleFailedMessage (line 335) | func (p *processor) handleFailedMessage(ctx context.Context, l *base.L...
method retry (line 351) | func (p *processor) retry(l *base.Lease, msg *base.TaskMessage, e erro...
method archive (line 374) | func (p *processor) archive(l *base.Lease, msg *base.TaskMessage, e er...
method queues (line 400) | func (p *processor) queues() []string {
method perform (line 424) | func (p *processor) perform(ctx context.Context, task *Task) (err erro...
method computeDeadline (line 524) | func (p *processor) computeDeadline(msg *base.TaskMessage) time.Time {
type processorParams (line 75) | type processorParams struct
function newProcessor (line 94) | func newProcessor(params processorParams) *processor {
function uniq (line 451) | func uniq(names []string, l int) []string {
function sortByPriority (line 468) | func sortByPriority(qcfg map[string]int) []string {
type queue (line 481) | type queue struct
type byPriority (line 486) | type byPriority
method Len (line 488) | func (x byPriority) Len() int { return len(x) }
method Less (line 489) | func (x byPriority) Less(i, j int) bool { return x[i].priority < x[j]....
method Swap (line 490) | func (x byPriority) Swap(i, j int) { x[i], x[j] = x[j], x[i] }
function normalizeQueues (line 493) | func normalizeQueues(queues map[string]int) map[string]int {
function gcd (line 506) | func gcd(xs ...int) int {
function IsPanicError (line 539) | func IsPanicError(err error) bool {
FILE: processor_test.go
function fakeHeartbeater (line 34) | func fakeHeartbeater(starting <-chan *workerInfo, finished <-chan *base....
function fakeSyncer (line 46) | func fakeSyncer(syncCh <-chan *syncRequest, done <-chan struct{}) {
function newProcessorForTest (line 57) | func newProcessorForTest(t *testing.T, r *rdb.RDB, h Handler) *processor {
function TestProcessorSuccessWithSingleQueue (line 86) | func TestProcessorSuccessWithSingleQueue(t *testing.T) {
function TestProcessorSuccessWithMultipleQueues (line 155) | func TestProcessorSuccessWithMultipleQueues(t *testing.T) {
function TestProcessTasksWithLargeNumberInPayload (line 229) | func TestProcessTasksWithLargeNumberInPayload(t *testing.T) {
function TestProcessorRetry (line 285) | func TestProcessorRetry(t *testing.T) {
function TestProcessorMarkAsComplete (line 441) | func TestProcessorMarkAsComplete(t *testing.T) {
function TestProcessorWithExpiredLease (line 518) | func TestProcessorWithExpiredLease(t *testing.T) {
function TestProcessorQueues (line 615) | func TestProcessorQueues(t *testing.T) {
function TestProcessorWithStrictPriority (line 655) | func TestProcessorWithStrictPriority(t *testing.T) {
function TestProcessorPerform (line 760) | func TestProcessorPerform(t *testing.T) {
function TestGCD (line 810) | func TestGCD(t *testing.T) {
function TestNormalizeQueues (line 832) | func TestNormalizeQueues(t *testing.T) {
function TestProcessorComputeDeadline (line 890) | func TestProcessorComputeDeadline(t *testing.T) {
function TestReturnPanicError (line 956) | func TestReturnPanicError(t *testing.T) {
FILE: recoverer.go
type recoverer (line 17) | type recoverer struct
method shutdown (line 54) | func (r *recoverer) shutdown() {
method start (line 60) | func (r *recoverer) start(wg *sync.WaitGroup) {
method recover (line 84) | func (r *recoverer) recover() {
method recoverLeaseExpiredTasks (line 89) | func (r *recoverer) recoverLeaseExpiredTasks() {
method recoverStaleAggregationSets (line 106) | func (r *recoverer) recoverStaleAggregationSets() {
method retry (line 114) | func (r *recoverer) retry(msg *base.TaskMessage, err error) {
method archive (line 122) | func (r *recoverer) archive(msg *base.TaskMessage, err error) {
type recovererParams (line 33) | type recovererParams struct
function newRecoverer (line 42) | func newRecoverer(params recovererParams) *recoverer {
FILE: recoverer_test.go
function TestRecoverer (line 18) | func TestRecoverer(t *testing.T) {
FILE: scheduler.go
type Scheduler (line 24) | type Scheduler struct
method Register (line 208) | func (s *Scheduler) Register(cronspec string, task *Task, opts ...Opti...
method Unregister (line 234) | func (s *Scheduler) Unregister(entryID string) error {
method Run (line 248) | func (s *Scheduler) Run() error {
method Start (line 259) | func (s *Scheduler) Start() error {
method start (line 273) | func (s *Scheduler) start() error {
method Shutdown (line 287) | func (s *Scheduler) Shutdown() {
method runHeartbeater (line 310) | func (s *Scheduler) runHeartbeater() {
method beat (line 329) | func (s *Scheduler) beat() {
method clearHistory (line 357) | func (s *Scheduler) clearHistory() {
method Ping (line 367) | func (s *Scheduler) Ping() error {
constant defaultHeartbeatInterval (line 49) | defaultHeartbeatInterval = 10 * time.Second
function NewScheduler (line 53) | func NewScheduler(r RedisConnOpt, opts *SchedulerOpts) *Scheduler {
function NewSchedulerFromRedisClient (line 72) | func NewSchedulerFromRedisClient(c redis.UniversalClient, opts *Schedule...
function newScheduler (line 81) | func newScheduler(opts *SchedulerOpts) *Scheduler {
function generateSchedulerID (line 118) | func generateSchedulerID() string {
type SchedulerOpts (line 127) | type SchedulerOpts struct
type enqueueJob (line 167) | type enqueueJob struct
method Run (line 181) | func (j *enqueueJob) Run() {
function stringifyOptions (line 349) | func stringifyOptions(opts []Option) []string {
FILE: scheduler_test.go
function TestSchedulerRegister (line 19) | func TestSchedulerRegister(t *testing.T) {
function TestSchedulerWhenRedisDown (line 104) | func TestSchedulerWhenRedisDown(t *testing.T) {
function TestSchedulerUnregister (line 141) | func TestSchedulerUnregister(t *testing.T) {
function TestSchedulerPostAndPreEnqueueHandler (line 183) | func TestSchedulerPostAndPreEnqueueHandler(t *testing.T) {
FILE: servemux.go
type ServeMux (line 29) | type ServeMux struct
method ProcessTask (line 53) | func (mux *ServeMux) ProcessTask(ctx context.Context, task *Task) error {
method Handler (line 65) | func (mux *ServeMux) Handler(t *Task) (h Handler, pattern string) {
method match (line 81) | func (mux *ServeMux) match(typename string) (h Handler, pattern string) {
method Handle (line 101) | func (mux *ServeMux) Handle(pattern string, handler Handler) {
method HandleFunc (line 139) | func (mux *ServeMux) HandleFunc(pattern string, handler func(context.C...
method Use (line 148) | func (mux *ServeMux) Use(mws ...MiddlewareFunc) {
type muxEntry (line 36) | type muxEntry struct
type MiddlewareFunc (line 44) | type MiddlewareFunc
function NewServeMux (line 47) | func NewServeMux() *ServeMux {
function appendSorted (line 123) | func appendSorted(es []muxEntry, e muxEntry) []muxEntry {
function NotFound (line 155) | func NotFound(ctx context.Context, task *Task) error {
function NotFoundHandler (line 160) | func NotFoundHandler() Handler { return HandlerFunc(NotFound) }
FILE: servemux_test.go
function makeFakeHandler (line 19) | func makeFakeHandler(identity string) Handler {
function makeFakeMiddleware (line 28) | func makeFakeMiddleware(identity string) MiddlewareFunc {
function TestServeMux (line 56) | func TestServeMux(t *testing.T) {
function TestServeMuxRegisterNilHandler (line 76) | func TestServeMuxRegisterNilHandler(t *testing.T) {
function TestServeMuxRegisterEmptyPattern (line 87) | func TestServeMuxRegisterEmptyPattern(t *testing.T) {
function TestServeMuxRegisterDuplicatePattern (line 98) | func TestServeMuxRegisterDuplicatePattern(t *testing.T) {
function TestServeMuxNotFound (line 117) | func TestServeMuxNotFound(t *testing.T) {
function TestServeMuxMiddlewares (line 142) | func TestServeMuxMiddlewares(t *testing.T) {
FILE: server.go
type Server (line 36) | type Server struct
method Run (line 663) | func (srv *Server) Run(handler Handler) error {
method Start (line 680) | func (srv *Server) Start(handler Handler) error {
method start (line 705) | func (srv *Server) start() error {
method Shutdown (line 724) | func (srv *Server) Shutdown() {
method Stop (line 761) | func (srv *Server) Stop() {
method Ping (line 779) | func (srv *Server) Ping() error {
type serverState (line 59) | type serverState struct
type serverStateValue (line 64) | type serverStateValue
method String (line 89) | func (s serverStateValue) String() string {
constant srvStateNew (line 70) | srvStateNew serverStateValue = iota
constant srvStateActive (line 73) | srvStateActive
constant srvStateStopped (line 76) | srvStateStopped
constant srvStateClosed (line 79) | srvStateClosed
type Config (line 97) | type Config struct
type GroupAggregator (line 258) | type GroupAggregator interface
type GroupAggregatorFunc (line 270) | type GroupAggregatorFunc
method Aggregate (line 273) | func (fn GroupAggregatorFunc) Aggregate(group string, tasks []*Task) *...
type ErrorHandler (line 278) | type ErrorHandler interface
type ErrorHandlerFunc (line 284) | type ErrorHandlerFunc
method HandleError (line 287) | func (fn ErrorHandlerFunc) HandleError(ctx context.Context, task *Task...
type RetryDelayFunc (line 297) | type RetryDelayFunc
type Logger (line 300) | type Logger interface
type LogLevel (line 321) | type LogLevel
method String (line 348) | func (l *LogLevel) String() string {
method Set (line 365) | func (l *LogLevel) Set(val string) error {
constant level_unspecified (line 325) | level_unspecified LogLevel = iota
constant DebugLevel (line 329) | DebugLevel
constant InfoLevel (line 332) | InfoLevel
constant WarnLevel (line 336) | WarnLevel
constant ErrorLevel (line 340) | ErrorLevel
constant FatalLevel (line 344) | FatalLevel
function toInternalLogLevel (line 383) | func toInternalLogLevel(l LogLevel) log.Level {
function DefaultRetryDelayFunc (line 401) | func DefaultRetryDelayFunc(n int, e error, t *Task) time.Duration {
function defaultIsFailureFunc (line 407) | func defaultIsFailureFunc(err error) bool { return err != nil }
constant defaultTaskCheckInterval (line 414) | defaultTaskCheckInterval = 1 * time.Second
constant defaultShutdownTimeout (line 416) | defaultShutdownTimeout = 8 * time.Second
constant defaultHealthCheckInterval (line 418) | defaultHealthCheckInterval = 15 * time.Second
constant defaultDelayedTaskCheckInterval (line 420) | defaultDelayedTaskCheckInterval = 5 * time.Second
constant defaultGroupGracePeriod (line 422) | defaultGroupGracePeriod = 1 * time.Minute
constant defaultJanitorInterval (line 424) | defaultJanitorInterval = 8 * time.Second
constant defaultJanitorBatchSize (line 426) | defaultJanitorBatchSize = 100
function NewServer (line 431) | func NewServer(r RedisConnOpt, cfg Config) *Server {
function NewServerFromRedisClient (line 444) | func NewServerFromRedisClient(c redis.UniversalClient, cfg Config) *Serv...
type Handler (line 638) | type Handler interface
type HandlerFunc (line 646) | type HandlerFunc
method ProcessTask (line 649) | func (fn HandlerFunc) ProcessTask(ctx context.Context, task *Task) err...
FILE: server_test.go
function testServer (line 22) | func testServer(t *testing.T, c *Client, srv *Server) {
function TestServer (line 46) | func TestServer(t *testing.T) {
function TestServerFromRedisClient (line 62) | func TestServerFromRedisClient(t *testing.T) {
function TestServerRun (line 83) | func TestServerRun(t *testing.T) {
function TestServerErrServerClosed (line 112) | func TestServerErrServerClosed(t *testing.T) {
function TestServerErrNilHandler (line 125) | func TestServerErrNilHandler(t *testing.T) {
function TestServerErrServerRunning (line 134) | func TestServerErrServerRunning(t *testing.T) {
function TestServerWithRedisDown (line 147) | func TestServerWithRedisDown(t *testing.T) {
function TestServerWithFlakyBroker (line 179) | func TestServerWithFlakyBroker(t *testing.T) {
function TestLogLevel (line 240) | func TestLogLevel(t *testing.T) {
FILE: signals_unix.go
method waitForSignals (line 16) | func (srv *Server) waitForSignals() {
method waitForSignals (line 34) | func (s *Scheduler) waitForSignals() {
FILE: signals_windows.go
method waitForSignals (line 17) | func (srv *Server) waitForSignals() {
method waitForSignals (line 24) | func (s *Scheduler) waitForSignals() {
FILE: subscriber.go
type subscriber (line 16) | type subscriber struct
method shutdown (line 46) | func (s *subscriber) shutdown() {
method start (line 52) | func (s *subscriber) start(wg *sync.WaitGroup) {
type subscriberParams (line 30) | type subscriberParams struct
function newSubscriber (line 36) | func newSubscriber(params subscriberParams) *subscriber {
FILE: subscriber_test.go
function TestSubscriber (line 17) | func TestSubscriber(t *testing.T) {
function TestSubscriberWithRedisDown (line 73) | func TestSubscriberWithRedisDown(t *testing.T) {
FILE: syncer.go
type syncer (line 16) | type syncer struct
method shutdown (line 49) | func (s *syncer) shutdown() {
method start (line 55) | func (s *syncer) start(wg *sync.WaitGroup) {
type syncRequest (line 28) | type syncRequest struct
type syncerParams (line 34) | type syncerParams struct
function newSyncer (line 40) | func newSyncer(params syncerParams) *syncer {
FILE: syncer_test.go
function TestSyncer (line 19) | func TestSyncer(t *testing.T) {
function TestSyncerRetry (line 59) | func TestSyncerRetry(t *testing.T) {
function TestSyncerDropsStaleRequests (line 106) | func TestSyncerDropsStaleRequests(t *testing.T) {
FILE: tools/asynq/cmd/cron.go
function init (line 19) | func init() {
function cronList (line 54) | func cronList(cmd *cobra.Command, args []string) {
function nextEnqueue (line 84) | func nextEnqueue(nextEnqueueAt time.Time) string {
function prevEnqueue (line 93) | func prevEnqueue(prevEnqueuedAt time.Time) string {
function cronHistory (line 100) | func cronHistory(cmd *cobra.Command, args []string) {
FILE: tools/asynq/cmd/dash.go
function init (line 21) | func init() {
FILE: tools/asynq/cmd/dash/dash.go
type viewType (line 19) | type viewType
constant viewTypeQueues (line 22) | viewTypeQueues viewType = iota
constant viewTypeQueueDetails (line 23) | viewTypeQueueDetails
constant viewTypeHelp (line 24) | viewTypeHelp
type State (line 28) | type State struct
method DebugString (line 51) | func (s *State) DebugString() string {
type Options (line 91) | type Options struct
function Run (line 97) | func Run(opts Options) {
FILE: tools/asynq/cmd/dash/draw.go
type drawer (line 36) | type drawer interface
type dashDrawer (line 40) | type dashDrawer struct
method Draw (line 45) | func (dd *dashDrawer) Draw(state *State) {
function drawQueueSizeGraphs (line 80) | func drawQueueSizeGraphs(d *ScreenDrawer, state *State) {
function drawFooter (line 135) | func drawFooter(d *ScreenDrawer, state *State) {
function maxwidth (line 153) | func maxwidth(names []string) int {
function rpad (line 164) | func rpad(s string, padding int) string {
function lpad (line 171) | func lpad(s string, padding int) string {
function byteCount (line 177) | func byteCount(b int64) string {
function formatQueueState (line 203) | func formatQueueState(q *asynq.QueueInfo) string {
function formatErrorRate (line 210) | func formatErrorRate(processed, failed int) string {
function formatNextProcessTime (line 217) | func formatNextProcessTime(t time.Time) string {
function formatPastTime (line 225) | func formatPastTime(t time.Time) string {
function drawQueueTable (line 233) | func drawQueueTable(d *ScreenDrawer, style tcell.Style, state *State) {
function drawQueueSummary (line 237) | func drawQueueSummary(d *ScreenDrawer, state *State) {
function groupPageSize (line 254) | func groupPageSize(s tcell.Screen) int {
function taskPageSize (line 260) | func taskPageSize(s tcell.Screen) int {
function shouldShowGroupTable (line 265) | func shouldShowGroupTable(state *State) bool {
function getTaskTableColumnConfig (line 269) | func getTaskTableColumnConfig(taskState asynq.TaskState) []*columnConfig...
function drawTaskTable (line 350) | func drawTaskTable(d *ScreenDrawer, state *State) {
function isNextTaskPageAvailable (line 382) | func isNextTaskPageAvailable(s tcell.Screen, state *State) bool {
function drawGroupTable (line 388) | func drawGroupTable(d *ScreenDrawer, state *State) {
type number (line 416) | type number interface
function min (line 421) | func min[V number](x, y V) V {
function nextTaskState (line 439) | func nextTaskState(current asynq.TaskState) asynq.TaskState {
function prevTaskState (line 452) | func prevTaskState(current asynq.TaskState) asynq.TaskState {
function getTaskCount (line 465) | func getTaskCount(queue *asynq.QueueInfo, taskState asynq.TaskState) int {
function drawTaskStateBreakdown (line 485) | func drawTaskStateBreakdown(d *ScreenDrawer, style tcell.Style, state *S...
function drawTaskModal (line 498) | func drawTaskModal(d *ScreenDrawer, state *State) {
function isPrintable (line 575) | func isPrintable(data []byte) bool {
function formatByteSlice (line 591) | func formatByteSlice(data []byte) string {
type modalRowDrawer (line 601) | type modalRowDrawer struct
method Print (line 608) | func (d *modalRowDrawer) Print(s string, style tcell.Style) {
function withModal (line 619) | func withModal(d *ScreenDrawer, rowPrintFns []func(d *modalRowDrawer)) {
function adjustWidth (line 657) | func adjustWidth(s string, width int) string {
function truncate (line 669) | func truncate(s string, max int) string {
function drawDebugInfo (line 676) | func drawDebugInfo(d *ScreenDrawer, state *State) {
function drawHelp (line 680) | func drawHelp(d *ScreenDrawer) {
FILE: tools/asynq/cmd/dash/draw_test.go
function TestTruncate (line 9) | func TestTruncate(t *testing.T) {
FILE: tools/asynq/cmd/dash/fetch.go
type fetcher (line 14) | type fetcher interface
type dataFetcher (line 19) | type dataFetcher struct
method Fetch (line 32) | func (f *dataFetcher) Fetch(state *State) {
method fetchQueues (line 51) | func (f *dataFetcher) fetchQueues() {
method fetchGroups (line 89) | func (f *dataFetcher) fetchGroups(qname string) {
method fetchAggregatingTasks (line 109) | func (f *dataFetcher) fetchAggregatingTasks(qname, group string, pageS...
method fetchTasks (line 130) | func (f *dataFetcher) fetchTasks(qname string, taskState asynq.TaskSta...
method fetchTaskInfo (line 169) | func (f *dataFetcher) fetchTaskInfo(qname, taskID string) {
function fetchQueues (line 61) | func fetchQueues(i *asynq.Inspector, queuesCh chan<- []*asynq.QueueInfo,...
function fetchQueueInfo (line 80) | func fetchQueueInfo(i *asynq.Inspector, qname string, queueCh chan<- *as...
function fetchGroups (line 100) | func fetchGroups(i *asynq.Inspector, qname string, groupsCh chan<- []*as...
function fetchAggregatingTasks (line 120) | func fetchAggregatingTasks(i *asynq.Inspector, qname, group string, page...
function fetchTasks (line 141) | func fetchTasks(i *asynq.Inspector, qname string, taskState asynq.TaskSt...
function fetchTaskInfo (line 178) | func fetchTaskInfo(i *asynq.Inspector, qname, taskID string, taskCh chan...
FILE: tools/asynq/cmd/dash/key_event.go
type keyEventHandler (line 17) | type keyEventHandler struct
method quit (line 29) | func (h *keyEventHandler) quit() {
method HandleKeyEvent (line 35) | func (h *keyEventHandler) HandleKeyEvent(ev *tcell.EventKey) {
method goBack (line 61) | func (h *keyEventHandler) goBack() {
method handleDownKey (line 89) | func (h *keyEventHandler) handleDownKey() {
method downKeyQueues (line 98) | func (h *keyEventHandler) downKeyQueues() {
method downKeyQueueDetails (line 107) | func (h *keyEventHandler) downKeyQueueDetails() {
method handleUpKey (line 125) | func (h *keyEventHandler) handleUpKey() {
method upKeyQueues (line 134) | func (h *keyEventHandler) upKeyQueues() {
method upKeyQueueDetails (line 144) | func (h *keyEventHandler) upKeyQueueDetails() {
method handleEnterKey (line 162) | func (h *keyEventHandler) handleEnterKey() {
method resetTicker (line 171) | func (h *keyEventHandler) resetTicker() {
method enterKeyQueues (line 175) | func (h *keyEventHandler) enterKeyQueues() {
method enterKeyQueueDetails (line 193) | func (h *keyEventHandler) enterKeyQueueDetails() {
method handleLeftKey (line 217) | func (h *keyEventHandler) handleLeftKey() {
method handleRightKey (line 235) | func (h *keyEventHandler) handleRightKey() {
method nextPage (line 253) | func (h *keyEventHandler) nextPage() {
method prevPage (line 282) | func (h *keyEventHandler) prevPage() {
method showHelp (line 307) | func (h *keyEventHandler) showHelp() {
FILE: tools/asynq/cmd/dash/key_event_test.go
function makeKeyEventHandler (line 16) | func makeKeyEventHandler(t *testing.T, state *State) *keyEventHandler {
type keyEventHandlerTest (line 30) | type keyEventHandlerTest struct
function TestKeyEventHandler (line 37) | func TestKeyEventHandler(t *testing.T) {
type fakeFetcher (line 228) | type fakeFetcher struct
method Fetch (line 230) | func (f *fakeFetcher) Fetch(s *State) {}
type fakeDrawer (line 232) | type fakeDrawer struct
method Draw (line 234) | func (d *fakeDrawer) Draw(s *State) {}
FILE: tools/asynq/cmd/dash/screen_drawer.go
type ScreenDrawer (line 25) | type ScreenDrawer struct
method Print (line 33) | func (d *ScreenDrawer) Print(s string, style tcell.Style) {
method Println (line 37) | func (d *ScreenDrawer) Println(s string, style tcell.Style) {
method FillLine (line 44) | func (d *ScreenDrawer) FillLine(r rune, style tcell.Style) {
method FillUntil (line 55) | func (d *ScreenDrawer) FillUntil(r rune, style tcell.Style, limit int) {
method NL (line 64) | func (d *ScreenDrawer) NL() {
method Screen (line 69) | func (d *ScreenDrawer) Screen() tcell.Screen {
method Goto (line 74) | func (d *ScreenDrawer) Goto(x, y int) {
method GoToBottom (line 80) | func (d *ScreenDrawer) GoToBottom() {
function NewScreenDrawer (line 29) | func NewScreenDrawer(s tcell.Screen) *ScreenDrawer {
type LineDrawer (line 86) | type LineDrawer struct
method Draw (line 96) | func (d *LineDrawer) Draw(s string, style tcell.Style) {
function NewLineDrawer (line 92) | func NewLineDrawer(row int, s tcell.Screen) *LineDrawer {
FILE: tools/asynq/cmd/dash/table.go
type columnAlignment (line 12) | type columnAlignment
constant alignRight (line 15) | alignRight columnAlignment = iota
constant alignLeft (line 16) | alignLeft
type columnConfig (line 19) | type columnConfig struct
type column (line 25) | type column struct
function drawTable (line 31) | func drawTable[V any](d *ScreenDrawer, style tcell.Style, configs []*col...
FILE: tools/asynq/cmd/group.go
function init (line 15) | func init() {
function groupLists (line 37) | func groupLists(cmd *cobra.Command, args []string) {
FILE: tools/asynq/cmd/queue.go
constant separator (line 19) | separator = "================================================="
function init (line 21) | func init() {
function queueList (line 106) | func queueList(cmd *cobra.Command, args []string) {
function queueInspect (line 153) | func queueInspect(cmd *cobra.Command, args []string) {
function printQueueInfo (line 168) | func printQueueInfo(info *asynq.QueueInfo) {
function queueHistory (line 198) | func queueHistory(cmd *cobra.Command, args []string) {
function printDailyStats (line 219) | func printDailyStats(stats []*asynq.DailyStats) {
function queuePause (line 236) | func queuePause(cmd *cobra.Command, args []string) {
function queueUnpause (line 248) | func queueUnpause(cmd *cobra.Command, args []string) {
function queueRemove (line 260) | func queueRemove(cmd *cobra.Command, args []string) {
FILE: tools/asynq/cmd/root.go
function Execute (line 81) | func Execute() {
function isRootCmd (line 88) | func isRootCmd(cmd *cobra.Command) bool {
type displayLine (line 94) | type displayLine struct
method String (line 100) | func (l *displayLine) String() string {
type displayLines (line 104) | type displayLines
method String (line 106) | func (dls displayLines) String() string {
function capitalize (line 115) | func capitalize(s string) string {
function rootHelpFunc (line 126) | func rootHelpFunc(cmd *cobra.Command, args []string) {
function rootUsageFunc (line 203) | func rootUsageFunc(cmd *cobra.Command) error {
function printSubcommandSuggestions (line 230) | func printSubcommandSuggestions(cmd *cobra.Command, arg string) {
function adjustPadding (line 247) | func adjustPadding(lines ...*displayLine) {
function rpad (line 261) | func rpad(s string, padding int) string {
function lpad (line 267) | func lpad(s string, padding int) string {
function indent (line 273) | func indent(text string, space int) string {
function dedent (line 291) | func dedent(text string) string {
function init (line 301) | func init() {
function initConfig (line 337) | func initConfig() {
function createRDB (line 363) | func createRDB() *rdb.RDB {
function createClient (line 386) | func createClient() *asynq.Client {
function createInspector (line 391) | func createInspector() *asynq.Inspector {
function getRedisConnOpt (line 395) | func getRedisConnOpt() asynq.RedisConnOpt {
function getTLSConfig (line 414) | func getTLSConfig() *tls.Config {
function printTable (line 449) | func printTable(cols []string, printRows func(w io.Writer, tmpl string)) {
function sprintBytes (line 466) | func sprintBytes(payload []byte) string {
function isPrintable (line 473) | func isPrintable(data []byte) bool {
function getDuration (line 490) | func getDuration(cmd *cobra.Command, arg string) time.Duration {
function getTime (line 507) | func getTime(cmd *cobra.Command, arg string) time.Time {
FILE: tools/asynq/cmd/server.go
function init (line 19) | func init() {
function serverList (line 50) | func serverList(cmd *cobra.Command, args []string) {
function formatQueues (line 85) | func formatQueues(qmap map[string]int) string {
function timeAgo (line 116) | func timeAgo(since time.Time) string {
FILE: tools/asynq/cmd/stats.go
function init (line 43) | func init() {
type AggregateStats (line 58) | type AggregateStats struct
type FullStats (line 71) | type FullStats struct
function stats (line 77) | func stats(cmd *cobra.Command, args []string) {
function printStatsByState (line 156) | func printStatsByState(s *AggregateStats) {
function numDigits (line 168) | func numDigits(n int) int {
function maxWidthOf (line 173) | func maxWidthOf(vals ...int) int {
function maxInt (line 183) | func maxInt(a, b int) int {
function printStatsByQueue (line 187) | func printStatsByQueue(stats []*rdb.Stats) {
function queueTitle (line 209) | func queueTitle(s *rdb.Stats) string {
function printSuccessFailureStats (line 218) | func printSuccessFailureStats(s *AggregateStats) {
function printInfo (line 233) | func printInfo(info map[string]string) {
function printClusterInfo (line 248) | func printClusterInfo(info map[string]string) {
function toInterfaceSlice (line 261) | func toInterfaceSlice(strs []string) []interface{} {
FILE: tools/asynq/cmd/task.go
function init (line 19) | func init() {
function taskList (line 212) | func taskList(cmd *cobra.Command, args []string) {
function listActiveTasks (line 264) | func listActiveTasks(qname string, pageNum, pageSize int) {
function listPendingTasks (line 285) | func listPendingTasks(qname string, pageNum, pageSize int) {
function listScheduledTasks (line 306) | func listScheduledTasks(qname string, pageNum, pageSize int) {
function formatProcessAt (line 330) | func formatProcessAt(processAt time.Time) string {
function listRetryTasks (line 338) | func listRetryTasks(qname string, pageNum, pageSize int) {
function listArchivedTasks (line 360) | func listArchivedTasks(qname string, pageNum, pageSize int) {
function listCompletedTasks (line 380) | func listCompletedTasks(qname string, pageNum, pageSize int) {
function listAggregatingTasks (line 400) | func listAggregatingTasks(qname, group string, pageNum, pageSize int) {
function taskCancel (line 421) | func taskCancel(cmd *cobra.Command, args []string) {
function taskInspect (line 432) | func taskInspect(cmd *cobra.Command, args []string) {
function printTaskInfo (line 453) | func printTaskInfo(info *asynq.TaskInfo) {
function formatNextProcessAt (line 471) | func formatNextProcessAt(processAt time.Time) string {
function formatPastTime (line 482) | func formatPastTime(t time.Time) string {
function taskArchive (line 489) | func taskArchive(cmd *cobra.Command, args []string) {
function taskDelete (line 510) | func taskDelete(cmd *cobra.Command, args []string) {
function taskRun (line 531) | func taskRun(cmd *cobra.Command, args []string) {
function taskEnqueue (line 552) | func taskEnqueue(cmd *cobra.Command, args []string) {
function taskArchiveAll (line 641) | func taskArchiveAll(cmd *cobra.Command, args []string) {
function taskDeleteAll (line 684) | func taskDeleteAll(cmd *cobra.Command, args []string) {
function taskRunAll (line 731) | func taskRunAll(cmd *cobra.Command, args []string) {
FILE: tools/asynq/main.go
function main (line 9) | func main() {
FILE: tools/metrics_exporter/main.go
function init (line 26) | func init() {
function main (line 34) | func main() {
FILE: x/metrics/metrics.go
constant namespace (line 13) | namespace = "asynq"
type QueueMetricsCollector (line 19) | type QueueMetricsCollector struct
method collectQueueInfo (line 25) | func (qmc *QueueMetricsCollector) collectQueueInfo() ([]*asynq.QueueIn...
method Describe (line 86) | func (qmc *QueueMetricsCollector) Describe(ch chan<- *prometheus.Desc) {
method Collect (line 90) | func (qmc *QueueMetricsCollector) Collect(ch chan<- prometheus.Metric) {
function NewQueueMetricsCollector (line 188) | func NewQueueMetricsCollector(inspector *asynq.Inspector) *QueueMetricsC...
FILE: x/rate/example_test.go
type RateLimitError (line 12) | type RateLimitError struct
method Error (line 16) | func (e *RateLimitError) Error() string {
function ExampleNewSemaphore (line 20) | func ExampleNewSemaphore() {
FILE: x/rate/semaphore.go
function NewSemaphore (line 16) | func NewSemaphore(rco asynq.RedisConnOpt, scope string, maxTokens int) *...
type Semaphore (line 38) | type Semaphore struct
method Acquire (line 68) | func (s *Semaphore) Acquire(ctx context.Context) (bool, error) {
method Release (line 89) | func (s *Semaphore) Release(ctx context.Context) error {
method Close (line 108) | func (s *Semaphore) Close() error {
function semaphoreKey (line 112) | func semaphoreKey(scope string) string {
FILE: x/rate/semaphore_test.go
function init (line 26) | func init() {
function TestNewSemaphore (line 33) | func TestNewSemaphore(t *testing.T) {
function TestNewSemaphore_Acquire (line 79) | func TestNewSemaphore_Acquire(t *testing.T) {
function TestNewSemaphore_Acquire_Error (line 147) | func TestNewSemaphore_Acquire_Error(t *testing.T) {
function TestNewSemaphore_Acquire_StaleToken (line 205) | func TestNewSemaphore_Acquire_StaleToken(t *testing.T) {
function TestNewSemaphore_Release (line 238) | func TestNewSemaphore_Release(t *testing.T) {
function TestNewSemaphore_Release_Error (line 316) | func TestNewSemaphore_Release_Error(t *testing.T) {
function getRedisConnOpt (line 386) | func getRedisConnOpt(tb testing.TB) asynq.RedisConnOpt {
type badConnOpt (line 403) | type badConnOpt struct
method MakeRedisClient (line 406) | func (b badConnOpt) MakeRedisClient() interface{} {
Condensed preview — 101 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (1,324K chars).
[
{
"path": ".github/FUNDING.yml",
"chars": 93,
"preview": "# These are supported funding model platforms\n\ngithub: [hibiken]\nopen_collective: ken-hibino\n"
},
{
"path": ".github/ISSUE_TEMPLATE/bug_report.md",
"chars": 781,
"preview": "---\nname: Bug report\nabout: Create a report to help us improve\ntitle: \"[BUG] Description of the bug\"\nlabels: bug\nassigne"
},
{
"path": ".github/ISSUE_TEMPLATE/feature_request.md",
"chars": 684,
"preview": "---\nname: Feature request\nabout: Suggest an idea for this project\ntitle: \"[FEATURE REQUEST] Description of the feature r"
},
{
"path": ".github/dependabot.yaml",
"chars": 481,
"preview": "version: 2\nupdates:\n - package-ecosystem: \"gomod\"\n directory: \"/\"\n schedule:\n interval: \"weekly\"\n labels:"
},
{
"path": ".github/workflows/benchstat.yml",
"chars": 2165,
"preview": "# This workflow runs benchmarks against the current branch,\n# compares them to benchmarks against master,\n# and uploads "
},
{
"path": ".github/workflows/build.yml",
"chars": 1857,
"preview": "name: build\n\non: [push, pull_request]\n\njobs:\n build:\n strategy:\n matrix:\n os: [ubuntu-latest]\n go"
},
{
"path": ".gitignore",
"chars": 401,
"preview": "vendor\n# Binaries for programs and plugins\n*.exe\n*.exe~\n*.dll\n*.so\n*.dylib\n\n# Test binary, built with `go test -c`\n*.tes"
},
{
"path": "CHANGELOG.md",
"chars": 21541,
"preview": "# Changelog\n\nAll notable changes to this project will be documented in this file.\n\nThe format is based on [\"Keep a Chang"
},
{
"path": "CODE_OF_CONDUCT.md",
"chars": 5223,
"preview": "# Contributor Covenant Code of Conduct\n\n## Our Pledge\n\nWe as members, contributors, and leaders pledge to make participa"
},
{
"path": "CONTRIBUTING.md",
"chars": 2042,
"preview": "# Contributing\n\nThanks for your interest in contributing to Asynq!\nWe are open to, and grateful for, any contributions m"
},
{
"path": "LICENSE",
"chars": 1071,
"preview": "MIT License\n\nCopyright (c) 2019 Kentaro Hibino\n\nPermission is hereby granted, free of charge, to any person obtaining a "
},
{
"path": "Makefile",
"chars": 333,
"preview": "ROOT_DIR:=$(shell dirname $(realpath $(firstword $(MAKEFILE_LIST))))\n\nproto: internal/proto/asynq.proto\n\tprotoc -I=$(ROO"
},
{
"path": "README.md",
"chars": 12979,
"preview": "<img src=\"https://user-images.githubusercontent.com/11155743/114697792-ffbfa580-9d26-11eb-8e5b-33bef69476dc.png\" alt=\"As"
},
{
"path": "aggregator.go",
"chars": 4902,
"preview": "// Copyright 2022 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "aggregator_test.go",
"chars": 5058,
"preview": "// Copyright 2022 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "asynq.go",
"chars": 17220,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "asynq_test.go",
"chars": 5438,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "benchmark_test.go",
"chars": 5946,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "client.go",
"chars": 14965,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "client_test.go",
"chars": 44464,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "context.go",
"chars": 1308,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "doc.go",
"chars": 2061,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "example_test.go",
"chars": 2798,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "forwarder.go",
"chars": 1714,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "forwarder_test.go",
"chars": 4329,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "go.mod",
"chars": 483,
"preview": "module github.com/hibiken/asynq\n\ngo 1.24.0\n\nrequire (\n\tgithub.com/google/go-cmp v0.7.0\n\tgithub.com/google/uuid v1.6.0\n\tg"
},
{
"path": "go.sum",
"chars": 3571,
"preview": "github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=\ngithub.com/bsm/ginkgo/v2 v2.12.0/go.mod"
},
{
"path": "healthcheck.go",
"chars": 1758,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "healthcheck_test.go",
"chars": 2022,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "heartbeat.go",
"chars": 5320,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "heartbeat_test.go",
"chars": 10810,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "inspector.go",
"chars": 30972,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "inspector_test.go",
"chars": 99304,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "internal/base/base.go",
"chars": 22010,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "internal/base/base_test.go",
"chars": 18351,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "internal/context/context.go",
"chars": 2451,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "internal/context/context_test.go",
"chars": 5291,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "internal/errors/errors.go",
"chars": 8739,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "internal/errors/errors_test.go",
"chars": 4525,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "internal/log/log.go",
"chars": 4952,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "internal/log/log_test.go",
"chars": 9552,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "internal/proto/asynq.pb.go",
"chars": 23033,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "internal/proto/asynq.proto",
"chars": 4864,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "internal/rdb/benchmark_test.go",
"chars": 7180,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "internal/rdb/inspect.go",
"chars": 65603,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "internal/rdb/inspect_test.go",
"chars": 153130,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "internal/rdb/rdb.go",
"chars": 51740,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "internal/rdb/rdb_test.go",
"chars": 125572,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "internal/testbroker/testbroker.go",
"chars": 7260,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "internal/testutil/builder.go",
"chars": 1766,
"preview": "// Copyright 2022 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "internal/testutil/builder_test.go",
"chars": 2154,
"preview": "// Copyright 2022 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "internal/testutil/testutil.go",
"chars": 23698,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "internal/timeutil/timeutil.go",
"chars": 1501,
"preview": "// Copyright 2022 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "internal/timeutil/timeutil_test.go",
"chars": 1035,
"preview": "// Copyright 2022 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "janitor.go",
"chars": 2017,
"preview": "// Copyright 2021 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "janitor_test.go",
"chars": 2579,
"preview": "// Copyright 2021 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "periodic_task_manager.go",
"chars": 7566,
"preview": "// Copyright 2022 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "periodic_task_manager_test.go",
"chars": 8712,
"preview": "// Copyright 2022 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "processor.go",
"chars": 15841,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "processor_test.go",
"chars": 29302,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "recoverer.go",
"chars": 3473,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "recoverer_test.go",
"chars": 7669,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "scheduler.go",
"chars": 10634,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "scheduler_test.go",
"chars": 5267,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "servemux.go",
"chars": 4799,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "servemux_test.go",
"chars": 4633,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "server.go",
"chars": 24650,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "server_test.go",
"chars": 6653,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "signals_unix.go",
"chars": 946,
"preview": "//go:build linux || dragonfly || freebsd || netbsd || openbsd || darwin\n\npackage asynq\n\nimport (\n\t\"os\"\n\t\"os/signal\"\n\n\t\"g"
},
{
"path": "signals_windows.go",
"chars": 709,
"preview": "//go:build windows\n\npackage asynq\n\nimport (\n\t\"os\"\n\t\"os/signal\"\n\n\t\"golang.org/x/sys/windows\"\n)\n\n// waitForSignals waits f"
},
{
"path": "subscriber.go",
"chars": 1936,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "subscriber_test.go",
"chars": 3044,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "syncer.go",
"chars": 2001,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "syncer_test.go",
"chars": 3055,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "tools/asynq/README.md",
"chars": 2040,
"preview": "# Asynq CLI\n\nAsynq CLI is a command line tool to monitor the queues and tasks managed by `asynq` package.\n\n## Table of C"
},
{
"path": "tools/asynq/cmd/cron.go",
"chars": 3498,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "tools/asynq/cmd/dash/dash.go",
"chars": 5188,
"preview": "// Copyright 2022 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "tools/asynq/cmd/dash/draw.go",
"chars": 22625,
"preview": "// Copyright 2022 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "tools/asynq/cmd/dash/draw_test.go",
"chars": 590,
"preview": "// Copyright 2022 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "tools/asynq/cmd/dash/fetch.go",
"chars": 4826,
"preview": "// Copyright 2022 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "tools/asynq/cmd/dash/key_event.go",
"chars": 6845,
"preview": "// Copyright 2022 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "tools/asynq/cmd/dash/key_event_test.go",
"chars": 7070,
"preview": "// Copyright 2022 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "tools/asynq/cmd/dash/screen_drawer.go",
"chars": 2108,
"preview": "// Copyright 2022 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "tools/asynq/cmd/dash/table.go",
"chars": 1786,
"preview": "// Copyright 2022 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "tools/asynq/cmd/dash.go",
"chars": 1045,
"preview": "// Copyright 2022 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "tools/asynq/cmd/group.go",
"chars": 1124,
"preview": "// Copyright 2022 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "tools/asynq/cmd/queue.go",
"chars": 7355,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "tools/asynq/cmd/root.go",
"chars": 14601,
"preview": "//\n// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// tha"
},
{
"path": "tools/asynq/cmd/server.go",
"chars": 2773,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "tools/asynq/cmd/stats.go",
"chars": 7083,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "tools/asynq/cmd/task.go",
"chars": 23178,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "tools/asynq/main.go",
"chars": 250,
"preview": "// Copyright 2020 Kentaro Hibino. All rights reserved.\n// Use of this source code is governed by a MIT license\n// that c"
},
{
"path": "tools/go.mod",
"chars": 2194,
"preview": "module github.com/hibiken/asynq/tools\n\ngo 1.22\n\nrequire (\n\tgithub.com/MakeNowJust/heredoc/v2 v2.0.1\n\tgithub.com/fatih/co"
},
{
"path": "tools/go.sum",
"chars": 50319,
"preview": "cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=\ncloud.google.com/go v0.34.0/go.mod h1"
},
{
"path": "tools/metrics_exporter/main.go",
"chars": 1756,
"preview": "package main\n\nimport (\n\t\"flag\"\n\t\"fmt\"\n\t\"log\"\n\t\"net/http\"\n\n\t\"github.com/hibiken/asynq\"\n\t\"github.com/hibiken/asynq/x/metri"
},
{
"path": "x/go.mod",
"chars": 834,
"preview": "module github.com/hibiken/asynq/x\n\ngo 1.22\n\nrequire (\n\tgithub.com/google/uuid v1.6.0\n\tgithub.com/hibiken/asynq v0.25.0\n\t"
},
{
"path": "x/go.sum",
"chars": 4369,
"preview": "github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=\ngithub.com/beorn7/perks v1.0.1/go.mod h1:"
},
{
"path": "x/metrics/metrics.go",
"chars": 4909,
"preview": "// Package metrics provides implementations of prometheus.Collector to collect Asynq queue metrics.\npackage metrics\n\nimp"
},
{
"path": "x/rate/example_test.go",
"chars": 790,
"preview": "package rate_test\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"time\"\n\n\t\"github.com/hibiken/asynq\"\n\t\"github.com/hibiken/asynq/x/rate\"\n)\n"
},
{
"path": "x/rate/semaphore.go",
"chars": 3013,
"preview": "// Package rate contains rate limiting strategies for asynq.Handler(s).\npackage rate\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"strin"
},
{
"path": "x/rate/semaphore_test.go",
"chars": 11548,
"preview": "package rate\n\nimport (\n\t\"context\"\n\t\"flag\"\n\t\"fmt\"\n\t\"strings\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/google/uuid\"\n\t\"github.com/h"
}
]
About this extraction
This page contains the full source code of the hibiken/asynq GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 101 files (1.1 MB), approximately 366.8k tokens, and a symbol index with 1351 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.