[](https://github.com/ThreeDotsLabs/watermill/actions/workflows/master.yml)
[](https://pkg.go.dev/github.com/ThreeDotsLabs/watermill)
[](https://goreportcard.com/report/github.com/ThreeDotsLabs/watermill)
[](https://codecov.io/gh/ThreeDotsLabs/watermill)
Watermill is a Go library for working efficiently with message streams. It is intended
for building event driven applications, enabling event sourcing, RPC over messages,
sagas and basically whatever else comes to your mind. You can use conventional pub/sub
implementations like Kafka or RabbitMQ, but also HTTP or PostgreSQL if that fits your use case.
## Goals
* **Easy** to understand.
* **Universal** - event-driven architecture, messaging, stream processing, CQRS - use it for whatever you need.
* **Fast** (see [Benchmarks](#benchmarks)).
* **Flexible** with middlewares, plugins and Pub/Sub configurations.
* **Resilient** - using proven technologies and passing stress tests (see [Stability](#stability)).
## Getting Started
Pick what you like the best or see in order:
1. [Quickstart](https://watermill.io/learn/quickstart/) — learn by coding!
2. Follow the [Getting Started guide](https://watermill.io/learn/getting-started/).
3. See examples below.
4. Read the full documentation: https://watermill.io/
## Our online hands-on training
Go Event-Driven goes beyond Watermill Quickstart. You'll learn industry standard concepts and patterns like:
* Handling at-least-once delivery
* Asynchronous read models
* Events & Commands
* Observability
* Message ordering
* Sagas
## Examples
* Basic
* [Your first app](_examples/basic/1-your-first-app) - **start here!**
* [Realtime feed](_examples/basic/2-realtime-feed)
* [Router](_examples/basic/3-router)
* [Metrics](_examples/basic/4-metrics)
* [CQRS with protobuf](_examples/basic/5-cqrs-protobuf)
* [Pub/Subs usage](_examples/pubsubs)
* These examples are part of the [Getting started guide](https://watermill.io/learn/getting-started/) and show usage of a single Pub/Sub at a time.
* Real-world examples
* [Exactly-once delivery counter](_examples/real-world-examples/exactly-once-delivery-counter)
* [Receiving webhooks](_examples/real-world-examples/receiving-webhooks)
* [Sending webhooks](_examples/real-world-examples/sending-webhooks)
* [Synchronizing Databases](_examples/real-world-examples/synchronizing-databases)
* [Persistent Event Log](_examples/real-world-examples/persistent-event-log)
* [Transactional Events](_examples/real-world-examples/transactional-events)
* [Real-time HTTP updates with Server-Sent Events](_examples/real-world-examples/server-sent-events)
* [Real-time HTTP updates with Server-Sent Events and htmx](_examples/real-world-examples/server-sent-events-htmx)
* Complete projects
* [NATS example with live code reloading](https://github.com/ThreeDotsLabs/nats-example)
* [RabbitMQ, webhooks and Kafka integration](https://github.com/ThreeDotsLabs/event-driven-example)
## Background
Building distributed and scalable services is rarely as easy as some may suggest. There is a
lot of hidden knowledge that comes with writing such systems. Just like you don't need to know the
whole TCP stack to create a HTTP REST server, you shouldn't need to study all of this knowledge to
start with building message-driven applications.
Watermill's goal is to make communication with messages as easy to use as HTTP routers. It provides
the tools needed to begin working with event-driven architecture and allows you to learn the details
on the go.
At the heart of Watermill there is one simple interface:
```go
func(*Message) ([]*Message, error)
```
Your handler receives a message and decides whether to publish new message(s) or return
an error. What happens next is up to the middlewares you've chosen.
You can find more about our motivations in our [*Introducing Watermill* blog post](https://threedots.tech/post/introducing-watermill/).
## Pub/Subs
All publishers and subscribers have to implement an interface:
```go
type Publisher interface {
Publish(topic string, messages ...*Message) error
Close() error
}
type Subscriber interface {
Subscribe(ctx context.Context, topic string) (<-chan *Message, error)
Close() error
}
```
Supported Pub/Subs:
- AMQP (RabbitMQ) Pub/Sub [(`github.com/ThreeDotsLabs/watermill-amqp/v3`)](https://github.com/ThreeDotsLabs/watermill-amqp/)
- AWS SNS/SQS Pub/Sub [(`github.com/ThreeDotsLabs/watermill-aws`)](https://github.com/ThreeDotsLabs/watermill-aws/)
- Bolt Pub/Sub [(`github.com/ThreeDotsLabs/watermill-bolt`)](https://github.com/ThreeDotsLabs/watermill-bolt/)
- Firestore Pub/Sub [(`github.com/ThreeDotsLabs/watermill-firestore`)](https://github.com/ThreeDotsLabs/watermill-firestore/)
- Google Cloud Pub/Sub [(`github.com/ThreeDotsLabs/watermill-googlecloud/v2`)](https://github.com/ThreeDotsLabs/watermill-googlecloud/)
- HTTP Pub/Sub [(`github.com/ThreeDotsLabs/watermill-http/v2`)](https://github.com/ThreeDotsLabs/watermill-http/)
- io.Reader/io.Writer Pub/Sub [(`github.com/ThreeDotsLabs/watermill-io`)](https://github.com/ThreeDotsLabs/watermill-io/)
- Kafka Pub/Sub [(`github.com/ThreeDotsLabs/watermill-kafka/v3`)](https://github.com/ThreeDotsLabs/watermill-kafka/)
- NATS Jetstream Pub/Sub [(`github.com/ThreeDotsLabs/watermill-nats/v2`)](https://github.com/ThreeDotsLabs/watermill-nats/)
- Redis Stream Pub/Sub [(`github.com/ThreeDotsLabs/watermill-redisstream`)](https://github.com/ThreeDotsLabs/watermill-redisstream/)
- SQL (MySQL / PostgreSQL) Pub/Sub [(`github.com/ThreeDotsLabs/watermill-sql/v4`)](https://github.com/ThreeDotsLabs/watermill-sql/)
- SQLite Pub/Sub (Beta) [(`github.com/ThreeDotsLabs/watermill-sqlite/`)](https://github.com/ThreeDotsLabs/watermill-sqlite/)
All Pub/Subs implementation documentation can be found in the [documentation](https://watermill.io/pubsubs/).
## Unofficial libraries
Can't find your favorite Pub/Sub or library integration? Check [Awesome Watermill](https://watermill.io/docs/awesome/).
If you know another library or are an author of one, please [add it to the list](https://github.com/ThreeDotsLabs/watermill/edit/master/docs/content/docs/awesome.md).
## Contributing
Please check our [contributing guide](CONTRIBUTING.md).
## Stability
Watermill v1.0.0 has been released and is production-ready. The public API is stable and will not change without changing the major version.
To ensure that all Pub/Subs are stable and safe to use in production, we created a [set of tests](https://github.com/ThreeDotsLabs/watermill/blob/master/pubsub/tests/test_pubsub.go#L34) that need to pass for each of the implementations before merging to master.
All tests are also executed in [*stress*](https://github.com/ThreeDotsLabs/watermill/blob/master/pubsub/tests/test_pubsub.go#L171) mode - that means that we are running all the tests **20x** in parallel.
All tests are run with the race condition detector enabled (`-race` flag in tests).
For more information about debugging tests, you should check [tests troubleshooting guide](http://watermill.io/docs/troubleshooting/#debugging-pubsub-tests).
## Benchmarks
Initial tools for benchmarking Pub/Subs can be found in [watermill-benchmark](https://github.com/ThreeDotsLabs/watermill-benchmark).
All benchmarks are being done on a single 16 CPU VM instance, running one binary and dependencies in Docker Compose.
These numbers are meant to serve as a rough estimate of how fast messages can be processed by different Pub/Subs.
Keep in mind that the results can be vastly different, depending on the setup and configuration (both much lower and higher).
Here's the short version for message size of 16 bytes.
| Pub/Sub | Publish (messages / s) | Subscribe (messages / s) |
|---------------------------------|------------------------|--------------------------|
| GoChannel | 315,776 | 138,743 |
| Redis Streams | 59,158 | 12,134 |
| NATS Jetstream (16 Subscribers) | 50,668 | 34,713 |
| Kafka (one node) | 41,492 | 101,669 |
| SQL (MySQL, batch size=100) | 6,371 | 2,794 |
| SQL (PostgreSQL, batch size=1) | 2,831 | 9,460 |
| Google Cloud Pub/Sub | 3,027 | 28,589 |
| AMQP (RabbitMQ) | 2,770 | 14,604 |
## Support
If you didn't find the answer to your question in [the documentation](https://watermill.io/), feel free to ask us directly!
Please join us on the `#watermill` channel on the [Three Dots Labs Discord](https://discord.gg/QV6VFg4YQE).
## Why the name?
It processes streams!
## License
[MIT License](./LICENSE)
================================================
FILE: RELEASE-PROCEDURE.md
================================================
# Release procedure
1. Generate clean go.mod: `make generate_gomod`
2. Push to master
3. Update missing documentation
4. Check snippets in documentation (sometimes `first_line_contains` or `last_line_contains` can change position and load too much)
5. Add breaking changes to `UPGRADE-[new-version].md`
6. Push to master
7. [Add release in GitHub](https://github.com/ThreeDotsLabs/watermill/releases)
8. Update Pub/Subs versions
9. Update and validate examples: `make validate_examples`
================================================
FILE: UPGRADE-0.3.md
================================================
# UPGRADE FROM 0.2.x to 0.3
# `watermill/message`
- `message.Message.Ack` and `message.Message.Nack` now return `bool` instead of `error`
- `message.Subscriber.Subscribe` now accepts `context.Context` as the first argument
- `message.Subscriber.Subscribe` now returns `<-chan *Message` instead of `chan *Message`
- `message.Router.AddHandler` and `message.Router.AddNoPublisherHandler` now panic, instead of returning error
# `watermill/message/infrastructure`
- updated all Pub/Subs to new `message.Subscriber` interface
- `gochannel.NewGoChannel` now accepts `gochannel.Config`, instead of positional parameters
- `http.NewSubscriber` now accepts `http.SubscriberConfig`, instead of positional parameters
# `watermill/message/router/middleware`
- `metrics.NewMetrics` is removed, please use the [metrics](components/metrics) component instead
# `watermill`
- `watermill.LoggerAdapter` interface now requires a `With(fields LogFields) LoggerAdapter` method
================================================
FILE: UPGRADE-0.4.md
================================================
# UPGRADE FROM 0.3.x to 0.4
## `watermill/components/cqrs`
### `CommandHandler.HandlerName` and `EventHandler.HandlerName` was added to the interface.
If you are using metrics component, you may want to keep backward capability with handler names. In other cases, you can implement your own method of generating handler name.
Keeping backward capability for **event handlers**:
```
func (h CommandHandler) HandlerName() string {
return fmt.Sprintf("command_processor-%s", h)
}
```
Keeping backward capability for **command handlers**:
```
func (h EventHandler) HandlerName() string {
return fmt.Sprintf("event_processor-%s", ObjectName(h))
}
```
### Added `CommandsSubscriberConstructor` and `EventsSubscriberConstructor`
From now on, `CommandsSubscriberConstructor` and `EventsSubscriberConstructor` are passed to constructors in CQRS component.
They allow creating customized subscribers for every handler. For usage examples please check [_examples/cqrs-protobuf](_examples/cqrs-protobuf).
### Added context to `CommandHandler.Handle`, `CommandBus.Send`, `EventHandler.Handle` and `EventBus.Send`
Added missing context, which is passed to Publish function and handlers.
### Other
- `NewCommandProcessor` and `NewEventProcessor` now return an error instead of panic
- `DuplicateCommandHandlerError` is returned instead of panic when two handlers are handling the same command
- `CommandProcessor.routerHandlerFunc` and `EventProcessor.routerHandlerFunc` are now private
- using `GenerateCommandsTopic` and `GenerateEventsTopic` functions instead of constant topic to allow more flexibility
## `watermill/message/infrastructure/amqp`
### `Config.QueueBindConfig.RoutingKey` was replaced with `GenerateRoutingKey`
For backward compatibility, when using the constant value you should use a function:
```
func(topic string) string {
return "routing_key"
}
```
## `message/router/middleware`
- `PoisonQueue` is now `PoisonQueue(pub message.Publisher, topic string) (message.HandlerMiddleware, error)`, not a struct
## `message/router.go`
- From now on, when all handlers are stopped, the router will also stop (`TestRouter_stop_when_all_handlers_stopped` test)
================================================
FILE: UPGRADE-1.0.md
================================================
# Upgrade instructions from v0.4.X
In v1.0.0 we introduced a couple of breaking changes, to keep a stable API until version v2.
## Migrating Pub/Subs
All Pub/Subs (excluding go-channel implementation) were moved to separated repositories.
You can replace all import paths, with provided `sed`:
find . -type f -iname '*.go' -exec sed -i -E "s/github\.com\/ThreeDotsLabs\/watermill\/message\/infrastructure\/(amqp|googlecloud|http|io|kafka|nats|sql)/github.com\/ThreeDotsLabs\/watermill-\1\/pkg\/\1/" "{}" +;
find . -type f -iname '*.go' -exec sed -i -E "s/github\.com\/ThreeDotsLabs\/watermill\/message\/infrastructure\/gochannel/github\.com\/ThreeDotsLabs\/watermill\/pubsub\/gochannel/" "{}" +;
# Breaking changes
- `message.PubSub` interface was removed
- `message.NewPubSub` constructor was removed
- `message.NoPublishHandlerFunc` is now passed to `message.Router.AddNoPublisherHandler`, instead of `message.HandlerFunc`.
- `message.Router.Run` now requires `context.Context` in parameter
- `PrometheusMetricsBuilder.DecoratePubSub` was removed (because of `message.PubSub` interface removal)
- `cars.ObjectName` was renamed to `cqrs.FullyQualifiedStructName`
- `github.com/ThreeDotsLabs/watermill/message/infrastructure/gochannel` was moved to `github.com/ThreeDotsLabs/watermill/pubsub/gochannel`
- `middleware.Retry` configuration parameters have been renamed
- Universal Pub/Sub tests have been moved from `github.com/ThreeDotsLabs/watermill/message/infrastructure` to `github.com/ThreeDotsLabs/watermill/pubsub/tests`
- All universal tests require now `TestContext`.
- Removed `context` from `googlecloud.NewPublisher`
================================================
FILE: _examples/basic/1-your-first-app/.validate_example.yml
================================================
validation_cmd: "docker compose up"
teardown_cmd: "docker compose down"
timeout: 180
expected_output: "received event {ID:[0-9]+}"
================================================
FILE: _examples/basic/1-your-first-app/README.md
================================================
# Your first Watermill app
This example project shows a basic setup of Watermill. The application runs in a loop, consuming events from a Kafka
topic, modifying them and publishing to another topic.
There's a docker-compose file included, so you can run the example and see it in action.
To understand the background and internals, see [getting started guide](https://watermill.io/learn/getting-started/).
## Files
- [main.go](main.go) - example source code, the **most interesting file for you**
- [docker-compose.yml](docker-compose.yml) - local environment Docker Compose configuration, contains Golang, Kafka and Zookeeper
- [go.mod](go.mod) - Go modules dependencies, you can find more information at [Go wiki](https://github.com/golang/go/wiki/Modules)
- [go.sum](go.sum) - Go modules checksums
## Requirements
To run this example you will need Docker and docker-compose installed. See the [installation guide](https://docs.docker.com/compose/install/).
## Running
```bash
> docker-compose up
[some initial logs]
server_1 | 2019/08/29 19:41:23 received event {ID:0}
server_1 | 2019/08/29 19:41:23 received event {ID:1}
server_1 | 2019/08/29 19:41:23 received event {ID:2}
server_1 | 2019/08/29 19:41:23 received event {ID:3}
server_1 | 2019/08/29 19:41:24 received event {ID:4}
server_1 | 2019/08/29 19:41:25 received event {ID:5}
server_1 | 2019/08/29 19:41:26 received event {ID:6}
server_1 | 2019/08/29 19:41:27 received event {ID:7}
server_1 | 2019/08/29 19:41:28 received event {ID:8}
server_1 | 2019/08/29 19:41:29 received event {ID:9}
```
Open another terminal and take a look at Kafka topics to see that all messages are there. The initial events should be present on the `events` topic:
```bash
> docker-compose exec server mill kafka consume -b kafka:9092 --topic events
{"id":12}
{"id":13}
{"id":14}
{"id":15}
{"id":16}
{"id":17}
```
And the processed messages will be stored in the `events-processed` topic:
```bash
> docker-compose exec server mill kafka consume -b kafka:9092 -t events-processed
{"processed_id":21,"time":"2019-08-29T19:42:31.4464598Z"}
{"processed_id":22,"time":"2019-08-29T19:42:32.4501767Z"}
{"processed_id":23,"time":"2019-08-29T19:42:33.4530692Z"}
{"processed_id":24,"time":"2019-08-29T19:42:34.4561694Z"}
{"processed_id":25,"time":"2019-08-29T19:42:35.4608918Z"}
```
================================================
FILE: _examples/basic/1-your-first-app/docker-compose.yml
================================================
services:
server:
image: golang:1.25
restart: unless-stopped
volumes:
- .:/app
- $GOPATH/pkg/mod:/go/pkg/mod
working_dir: /app
command: >
/bin/sh -c "go install github.com/ThreeDotsLabs/watermill/tools/mill@latest &&
go run main.go"
kafka:
image: bitnami/kafka:3.5.0
restart: unless-stopped
environment:
ALLOW_PLAINTEXT_LISTENER: yes
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
================================================
FILE: _examples/basic/1-your-first-app/go.mod
================================================
module main.go
go 1.25
require (
github.com/ThreeDotsLabs/watermill v1.5.1
github.com/ThreeDotsLabs/watermill-kafka/v3 v3.1.0
)
require (
github.com/IBM/sarama v1.46.0 // indirect
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dnwe/otelsarama v0.0.0-20240308230250-9388d9d40bc0 // indirect
github.com/eapache/go-resiliency v1.7.0 // indirect
github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 // indirect
github.com/eapache/queue v1.1.0 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/golang/snappy v1.0.0 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/go-uuid v1.0.3 // indirect
github.com/jcmturner/aescts/v2 v2.0.0 // indirect
github.com/jcmturner/dnsutils/v2 v2.0.0 // indirect
github.com/jcmturner/gofork v1.7.6 // indirect
github.com/jcmturner/gokrb5/v8 v8.4.4 // indirect
github.com/jcmturner/rpc/v2 v2.0.3 // indirect
github.com/klauspost/compress v1.18.0 // indirect
github.com/lithammer/shortuuid/v3 v3.0.7 // indirect
github.com/oklog/ulid v1.3.1 // indirect
github.com/pierrec/lz4/v4 v4.1.22 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9 // indirect
github.com/sony/gobreaker v1.0.0 // indirect
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
go.opentelemetry.io/otel v1.38.0 // indirect
go.opentelemetry.io/otel/metric v1.38.0 // indirect
go.opentelemetry.io/otel/trace v1.38.0 // indirect
golang.org/x/crypto v0.41.0 // indirect
golang.org/x/net v0.43.0 // indirect
)
================================================
FILE: _examples/basic/1-your-first-app/go.sum
================================================
github.com/IBM/sarama v1.46.0 h1:+YTM1fNd6WKMchlnLKRUB5Z0qD4M8YbvwIIPLvJD53s=
github.com/IBM/sarama v1.46.0/go.mod h1:0lOcuQziJ1/mBGHkdp5uYrltqQuKQKM5O5FOWUQVVvo=
github.com/ThreeDotsLabs/watermill v1.5.1 h1:t5xMivyf9tpmU3iozPqyrCZXHvoV1XQDfihas4sV0fY=
github.com/ThreeDotsLabs/watermill v1.5.1/go.mod h1:Uop10dA3VeJWsSvis9qO3vbVY892LARrKAdki6WtXS4=
github.com/ThreeDotsLabs/watermill-kafka/v3 v3.1.0 h1:DfhVM1ieq+rb+bboB7aoymUgfKsEM3UbH3noQZ6+RJ4=
github.com/ThreeDotsLabs/watermill-kafka/v3 v3.1.0/go.mod h1:o1GcoF/1CSJ9JSmQzUkULvpZeO635pZe+WWrYNFlJNk=
github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1xcsSM=
github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dnwe/otelsarama v0.0.0-20240308230250-9388d9d40bc0 h1:R2zQhFwSCyyd7L43igYjDrH0wkC/i+QBPELuY0HOu84=
github.com/dnwe/otelsarama v0.0.0-20240308230250-9388d9d40bc0/go.mod h1:2MqLKYJfjs3UriXXF9Fd0Qmh/lhxi/6tHXkqtXxyIHc=
github.com/eapache/go-resiliency v1.7.0 h1:n3NRTnBn5N0Cbi/IeOHuQn9s2UwVUH7Ga0ZWcP+9JTA=
github.com/eapache/go-resiliency v1.7.0/go.mod h1:5yPzW0MIvSe0JDsv0v+DvcjEv2FyD6iZYSs1ZI+iQho=
github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 h1:Oy0F4ALJ04o5Qqpdz8XLIpNA3WM/iSIXqxtqo7UGVws=
github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3/go.mod h1:YvSRo5mw33fLEx1+DlK6L2VV43tJt5Eyel9n9XBcR+0=
github.com/eapache/queue v1.1.0 h1:YOEu7KNc61ntiQlcEeUIoDTJ2o8mQznoNvUhiigpIqc=
github.com/eapache/queue v1.1.0/go.mod h1:6eCeP0CKFpHLu8blIFXhExK/dRa7WDZfr6jVFPTqq+I=
github.com/fortytw2/leaktest v1.3.0 h1:u8491cBMTQ8ft8aeV+adlcytMZylmA5nnwwkRZjI8vw=
github.com/fortytw2/leaktest v1.3.0/go.mod h1:jDsjWgpAGjm2CA7WthBh/CdZYEPF31XHquHwclZch5g=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/golang/snappy v1.0.0 h1:Oy607GVXHs7RtbggtPBnr2RmDArIsAefDwvrdWvRhGs=
github.com/golang/snappy v1.0.0/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/uuid v1.2.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/gorilla/securecookie v1.1.1/go.mod h1:ra0sb63/xPlUeL+yeDciTfxMRAA+MP+HVt/4epWDjd4=
github.com/gorilla/sessions v1.2.1/go.mod h1:dk2InVEVJ0sfLlnXv9EAgkf6ecYs/i80K/zI+bUmuGM=
github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I=
github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo=
github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM=
github.com/hashicorp/go-uuid v1.0.2/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
github.com/hashicorp/go-uuid v1.0.3 h1:2gKiV6YVmrJ1i2CKKa9obLvRieoRGviZFL26PcT/Co8=
github.com/hashicorp/go-uuid v1.0.3/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
github.com/jcmturner/aescts/v2 v2.0.0 h1:9YKLH6ey7H4eDBXW8khjYslgyqG2xZikXP0EQFKrle8=
github.com/jcmturner/aescts/v2 v2.0.0/go.mod h1:AiaICIRyfYg35RUkr8yESTqvSy7csK90qZ5xfvvsoNs=
github.com/jcmturner/dnsutils/v2 v2.0.0 h1:lltnkeZGL0wILNvrNiVCR6Ro5PGU/SeBvVO/8c/iPbo=
github.com/jcmturner/dnsutils/v2 v2.0.0/go.mod h1:b0TnjGOvI/n42bZa+hmXL+kFJZsFT7G4t3HTlQ184QM=
github.com/jcmturner/gofork v1.7.6 h1:QH0l3hzAU1tfT3rZCnW5zXl+orbkNMMRGJfdJjHVETg=
github.com/jcmturner/gofork v1.7.6/go.mod h1:1622LH6i/EZqLloHfE7IeZ0uEJwMSUyQ/nDd82IeqRo=
github.com/jcmturner/goidentity/v6 v6.0.1 h1:VKnZd2oEIMorCTsFBnJWbExfNN7yZr3EhJAxwOkZg6o=
github.com/jcmturner/goidentity/v6 v6.0.1/go.mod h1:X1YW3bgtvwAXju7V3LCIMpY0Gbxyjn/mY9zx4tFonSg=
github.com/jcmturner/gokrb5/v8 v8.4.4 h1:x1Sv4HaTpepFkXbt2IkL29DXRf8sOfZXo8eRKh687T8=
github.com/jcmturner/gokrb5/v8 v8.4.4/go.mod h1:1btQEpgT6k+unzCwX1KdWMEwPPkkgBtP+F6aCACiMrs=
github.com/jcmturner/rpc/v2 v2.0.3 h1:7FXXj8Ti1IaVFpSAziCZWNzbNuZmnvw/i6CqLNdWfZY=
github.com/jcmturner/rpc/v2 v2.0.3/go.mod h1:VUJYCIDm3PVOEHw8sgt091/20OJjskO/YJki3ELg/Hc=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/lithammer/shortuuid/v3 v3.0.7 h1:trX0KTHy4Pbwo/6ia8fscyHoGA+mf1jWbPJVuvyJQQ8=
github.com/lithammer/shortuuid/v3 v3.0.7/go.mod h1:vMk8ke37EmiewwolSO1NLW8vP4ZaKlRuDIi8tWWmAts=
github.com/oklog/ulid v1.3.1 h1:EGfNDEx6MqHz8B3uNV6QAib1UR2Lm97sHi3ocA6ESJ4=
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
github.com/pierrec/lz4/v4 v4.1.22 h1:cKFw6uJDK+/gfw5BcDL0JL5aBsAFdsIT18eRtLj7VIU=
github.com/pierrec/lz4/v4 v4.1.22/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9 h1:bsUq1dX0N8AOIL7EB/X911+m4EHsnWEHeJ0c+3TTBrg=
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
github.com/sony/gobreaker v1.0.0 h1:feX5fGGXSl3dYd4aHZItw+FpHLvvoaqkawKjVNiFMNQ=
github.com/sony/gobreaker v1.0.0/go.mod h1:ZKptC7FHNvhBz7dN2LGjPVBz2sZJmc0/PkyDJOjmxWY=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA=
go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A=
go.opentelemetry.io/otel v1.38.0 h1:RkfdswUDRimDg0m2Az18RKOsnI8UDzppJAtj01/Ymk8=
go.opentelemetry.io/otel v1.38.0/go.mod h1:zcmtmQ1+YmQM9wrNsTGV/q/uyusom3P8RxwExxkZhjM=
go.opentelemetry.io/otel/metric v1.38.0 h1:Kl6lzIYGAh5M159u9NgiRkmoMKjvbsKtYRwgfrA6WpA=
go.opentelemetry.io/otel/metric v1.38.0/go.mod h1:kB5n/QoRM8YwmUahxvI3bO34eVtQf2i4utNVLr9gEmI=
go.opentelemetry.io/otel/trace v1.38.0 h1:Fxk5bKrDZJUH+AMyyIXGcFAPah0oRcT+LuNtJrmcNLE=
go.opentelemetry.io/otel/trace v1.38.0/go.mod h1:j1P9ivuFsTceSWe1oY+EeW3sc+Pp42sO++GHkg4wwhs=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58=
golang.org/x/crypto v0.41.0 h1:WKYxWedPGCTVVl5+WHSSrOBT0O8lx32+zxmHxijgXp4=
golang.org/x/crypto v0.41.0/go.mod h1:pO5AFd7FA68rFak7rOAGVuygIISepHftHnr8dr6+sUc=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.43.0 h1:lat02VYK2j4aLzMzecihNvTlJNQUq316m2Mr9rnM6YE=
golang.org/x/net v0.43.0/go.mod h1:vhO1fvI4dGsIjh73sWfUVjj3N7CA9WkKJNQm2svM6Jg=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.16.0 h1:ycBJEhp9p4vXvUZNszeOq0kGTPghopOL8q0fq3vstxw=
golang.org/x/sync v0.16.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
================================================
FILE: _examples/basic/1-your-first-app/main.go
================================================
package main
import (
"context"
"encoding/json"
"fmt"
"time"
"github.com/ThreeDotsLabs/watermill"
"github.com/ThreeDotsLabs/watermill-kafka/v3/pkg/kafka"
"github.com/ThreeDotsLabs/watermill/message"
"github.com/ThreeDotsLabs/watermill/message/router/middleware"
"github.com/ThreeDotsLabs/watermill/message/router/plugin"
)
var (
brokers = []string{"kafka:9092"}
consumeTopic = "events"
publishTopic = "events-processed"
logger = watermill.NewStdLogger(
true, // debug
false, // trace
)
marshaler = kafka.DefaultMarshaler{}
)
type event struct {
ID int `json:"id"`
}
type processedEvent struct {
ProcessedID int `json:"processed_id"`
Time time.Time `json:"time"`
}
func main() {
publisher := createPublisher()
// Subscriber is created with consumer group handler_1
subscriber := createSubscriber("handler_1")
router, err := message.NewRouter(message.RouterConfig{}, logger)
if err != nil {
panic(err)
}
router.AddPlugin(plugin.SignalsHandler)
router.AddMiddleware(middleware.Recoverer)
// Adding a handler (multiple handlers can be added)
router.AddHandler(
"handler_1", // handler name, must be unique
consumeTopic, // topic from which messages should be consumed
subscriber,
publishTopic, // topic to which messages should be published
publisher,
func(msg *message.Message) ([]*message.Message, error) {
consumedPayload := event{}
err := json.Unmarshal(msg.Payload, &consumedPayload)
if err != nil {
// When a handler returns an error, the default behavior is to send a Nack (negative-acknowledgement).
// The message will be processed again.
//
// You can change the default behaviour by using middlewares, like Retry or PoisonQueue.
// You can also implement your own middleware.
return nil, err
}
fmt.Printf("received event %+v\n", consumedPayload)
newPayload, err := json.Marshal(processedEvent{
ProcessedID: consumedPayload.ID,
Time: time.Now(),
})
if err != nil {
return nil, err
}
newMessage := message.NewMessage(watermill.NewUUID(), newPayload)
return []*message.Message{newMessage}, nil
},
)
// Simulate incoming events in the background
go simulateEvents(publisher)
if err := router.Run(context.Background()); err != nil {
panic(err)
}
}
// createPublisher is a helper function that creates a Publisher, in this case - the Kafka Publisher.
func createPublisher() message.Publisher {
kafkaPublisher, err := kafka.NewPublisher(
kafka.PublisherConfig{
Brokers: brokers,
Marshaler: marshaler,
},
logger,
)
if err != nil {
panic(err)
}
return kafkaPublisher
}
// createSubscriber is a helper function similar to the previous one, but in this case it creates a Subscriber.
func createSubscriber(consumerGroup string) message.Subscriber {
kafkaSubscriber, err := kafka.NewSubscriber(
kafka.SubscriberConfig{
Brokers: brokers,
Unmarshaler: marshaler,
ConsumerGroup: consumerGroup, // every handler will use a separate consumer group
},
logger,
)
if err != nil {
panic(err)
}
return kafkaSubscriber
}
// simulateEvents produces events that will be later consumed.
func simulateEvents(publisher message.Publisher) {
i := 0
for {
e := event{
ID: i,
}
payload, err := json.Marshal(e)
if err != nil {
panic(err)
}
err = publisher.Publish(consumeTopic, message.NewMessage(
watermill.NewUUID(), // internal uuid of the message, useful for debugging
payload,
))
if err != nil {
panic(err)
}
i++
time.Sleep(time.Second)
}
}
================================================
FILE: _examples/basic/2-realtime-feed/.validate_example_subscribing.yml
================================================
validation_cmd: "docker compose up"
teardown_cmd: "docker compose down"
timeout: 180
expected_output: "Adding to feed"
================================================
FILE: _examples/basic/2-realtime-feed/README.md
================================================
# Realtime Feed
This example features a very busy blogging platform, with thousands of messages showing up on your feed.
There are two separate applications (microservices) integrating over a Kafka topic. The [`producer`](producer/) generates
thousands of "posts" and publishes them to the topic. The [`consumer`](consumer/) subscribes to this topic and
displays each post on the standard output.
The consumer has a throttling middleware enabled, so you have a chance to actually read the posts.
To understand the background and internals, see [getting started guide](https://watermill.io/learn/getting-started/).
## Requirements
To run this example you will need Docker and docker-compose installed. See the [installation guide](https://docs.docker.com/compose/install/).
## Running
```bash
docker-compose up
```
You should see the live feed of posts on the standard output.
## Exercises
1. Peek into the posts counter published on `posts_count` topic.
```
docker-compose exec consumer mill kafka consume -b kafka:9092 -t posts_count
```
2. Add a persistent storage for incoming posts in the consumer service, instead of displaying them.
Consider using the [SQL Publisher](https://github.com/ThreeDotsLabs/watermill-sql).
================================================
FILE: _examples/basic/2-realtime-feed/consumer/go.mod
================================================
module main.go
go 1.25
require (
github.com/ThreeDotsLabs/watermill v1.5.1
github.com/ThreeDotsLabs/watermill-kafka/v3 v3.1.0
)
require (
github.com/IBM/sarama v1.46.0 // indirect
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dnwe/otelsarama v0.0.0-20240308230250-9388d9d40bc0 // indirect
github.com/eapache/go-resiliency v1.7.0 // indirect
github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 // indirect
github.com/eapache/queue v1.1.0 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/golang/snappy v1.0.0 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/go-uuid v1.0.3 // indirect
github.com/jcmturner/aescts/v2 v2.0.0 // indirect
github.com/jcmturner/dnsutils/v2 v2.0.0 // indirect
github.com/jcmturner/gofork v1.7.6 // indirect
github.com/jcmturner/gokrb5/v8 v8.4.4 // indirect
github.com/jcmturner/rpc/v2 v2.0.3 // indirect
github.com/klauspost/compress v1.18.0 // indirect
github.com/lithammer/shortuuid/v3 v3.0.7 // indirect
github.com/oklog/ulid v1.3.1 // indirect
github.com/pierrec/lz4/v4 v4.1.22 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9 // indirect
github.com/sony/gobreaker v1.0.0 // indirect
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
go.opentelemetry.io/otel v1.38.0 // indirect
go.opentelemetry.io/otel/metric v1.38.0 // indirect
go.opentelemetry.io/otel/trace v1.38.0 // indirect
golang.org/x/crypto v0.41.0 // indirect
golang.org/x/net v0.43.0 // indirect
)
================================================
FILE: _examples/basic/2-realtime-feed/consumer/go.sum
================================================
github.com/IBM/sarama v1.46.0 h1:+YTM1fNd6WKMchlnLKRUB5Z0qD4M8YbvwIIPLvJD53s=
github.com/IBM/sarama v1.46.0/go.mod h1:0lOcuQziJ1/mBGHkdp5uYrltqQuKQKM5O5FOWUQVVvo=
github.com/ThreeDotsLabs/watermill v1.5.1 h1:t5xMivyf9tpmU3iozPqyrCZXHvoV1XQDfihas4sV0fY=
github.com/ThreeDotsLabs/watermill v1.5.1/go.mod h1:Uop10dA3VeJWsSvis9qO3vbVY892LARrKAdki6WtXS4=
github.com/ThreeDotsLabs/watermill-kafka/v3 v3.1.0 h1:DfhVM1ieq+rb+bboB7aoymUgfKsEM3UbH3noQZ6+RJ4=
github.com/ThreeDotsLabs/watermill-kafka/v3 v3.1.0/go.mod h1:o1GcoF/1CSJ9JSmQzUkULvpZeO635pZe+WWrYNFlJNk=
github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1xcsSM=
github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dnwe/otelsarama v0.0.0-20240308230250-9388d9d40bc0 h1:R2zQhFwSCyyd7L43igYjDrH0wkC/i+QBPELuY0HOu84=
github.com/dnwe/otelsarama v0.0.0-20240308230250-9388d9d40bc0/go.mod h1:2MqLKYJfjs3UriXXF9Fd0Qmh/lhxi/6tHXkqtXxyIHc=
github.com/eapache/go-resiliency v1.7.0 h1:n3NRTnBn5N0Cbi/IeOHuQn9s2UwVUH7Ga0ZWcP+9JTA=
github.com/eapache/go-resiliency v1.7.0/go.mod h1:5yPzW0MIvSe0JDsv0v+DvcjEv2FyD6iZYSs1ZI+iQho=
github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 h1:Oy0F4ALJ04o5Qqpdz8XLIpNA3WM/iSIXqxtqo7UGVws=
github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3/go.mod h1:YvSRo5mw33fLEx1+DlK6L2VV43tJt5Eyel9n9XBcR+0=
github.com/eapache/queue v1.1.0 h1:YOEu7KNc61ntiQlcEeUIoDTJ2o8mQznoNvUhiigpIqc=
github.com/eapache/queue v1.1.0/go.mod h1:6eCeP0CKFpHLu8blIFXhExK/dRa7WDZfr6jVFPTqq+I=
github.com/fortytw2/leaktest v1.3.0 h1:u8491cBMTQ8ft8aeV+adlcytMZylmA5nnwwkRZjI8vw=
github.com/fortytw2/leaktest v1.3.0/go.mod h1:jDsjWgpAGjm2CA7WthBh/CdZYEPF31XHquHwclZch5g=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/golang/snappy v1.0.0 h1:Oy607GVXHs7RtbggtPBnr2RmDArIsAefDwvrdWvRhGs=
github.com/golang/snappy v1.0.0/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/uuid v1.2.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/gorilla/securecookie v1.1.1/go.mod h1:ra0sb63/xPlUeL+yeDciTfxMRAA+MP+HVt/4epWDjd4=
github.com/gorilla/sessions v1.2.1/go.mod h1:dk2InVEVJ0sfLlnXv9EAgkf6ecYs/i80K/zI+bUmuGM=
github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I=
github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo=
github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM=
github.com/hashicorp/go-uuid v1.0.2/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
github.com/hashicorp/go-uuid v1.0.3 h1:2gKiV6YVmrJ1i2CKKa9obLvRieoRGviZFL26PcT/Co8=
github.com/hashicorp/go-uuid v1.0.3/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
github.com/jcmturner/aescts/v2 v2.0.0 h1:9YKLH6ey7H4eDBXW8khjYslgyqG2xZikXP0EQFKrle8=
github.com/jcmturner/aescts/v2 v2.0.0/go.mod h1:AiaICIRyfYg35RUkr8yESTqvSy7csK90qZ5xfvvsoNs=
github.com/jcmturner/dnsutils/v2 v2.0.0 h1:lltnkeZGL0wILNvrNiVCR6Ro5PGU/SeBvVO/8c/iPbo=
github.com/jcmturner/dnsutils/v2 v2.0.0/go.mod h1:b0TnjGOvI/n42bZa+hmXL+kFJZsFT7G4t3HTlQ184QM=
github.com/jcmturner/gofork v1.7.6 h1:QH0l3hzAU1tfT3rZCnW5zXl+orbkNMMRGJfdJjHVETg=
github.com/jcmturner/gofork v1.7.6/go.mod h1:1622LH6i/EZqLloHfE7IeZ0uEJwMSUyQ/nDd82IeqRo=
github.com/jcmturner/goidentity/v6 v6.0.1 h1:VKnZd2oEIMorCTsFBnJWbExfNN7yZr3EhJAxwOkZg6o=
github.com/jcmturner/goidentity/v6 v6.0.1/go.mod h1:X1YW3bgtvwAXju7V3LCIMpY0Gbxyjn/mY9zx4tFonSg=
github.com/jcmturner/gokrb5/v8 v8.4.4 h1:x1Sv4HaTpepFkXbt2IkL29DXRf8sOfZXo8eRKh687T8=
github.com/jcmturner/gokrb5/v8 v8.4.4/go.mod h1:1btQEpgT6k+unzCwX1KdWMEwPPkkgBtP+F6aCACiMrs=
github.com/jcmturner/rpc/v2 v2.0.3 h1:7FXXj8Ti1IaVFpSAziCZWNzbNuZmnvw/i6CqLNdWfZY=
github.com/jcmturner/rpc/v2 v2.0.3/go.mod h1:VUJYCIDm3PVOEHw8sgt091/20OJjskO/YJki3ELg/Hc=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/lithammer/shortuuid/v3 v3.0.7 h1:trX0KTHy4Pbwo/6ia8fscyHoGA+mf1jWbPJVuvyJQQ8=
github.com/lithammer/shortuuid/v3 v3.0.7/go.mod h1:vMk8ke37EmiewwolSO1NLW8vP4ZaKlRuDIi8tWWmAts=
github.com/oklog/ulid v1.3.1 h1:EGfNDEx6MqHz8B3uNV6QAib1UR2Lm97sHi3ocA6ESJ4=
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
github.com/pierrec/lz4/v4 v4.1.22 h1:cKFw6uJDK+/gfw5BcDL0JL5aBsAFdsIT18eRtLj7VIU=
github.com/pierrec/lz4/v4 v4.1.22/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9 h1:bsUq1dX0N8AOIL7EB/X911+m4EHsnWEHeJ0c+3TTBrg=
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
github.com/sony/gobreaker v1.0.0 h1:feX5fGGXSl3dYd4aHZItw+FpHLvvoaqkawKjVNiFMNQ=
github.com/sony/gobreaker v1.0.0/go.mod h1:ZKptC7FHNvhBz7dN2LGjPVBz2sZJmc0/PkyDJOjmxWY=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA=
go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A=
go.opentelemetry.io/otel v1.38.0 h1:RkfdswUDRimDg0m2Az18RKOsnI8UDzppJAtj01/Ymk8=
go.opentelemetry.io/otel v1.38.0/go.mod h1:zcmtmQ1+YmQM9wrNsTGV/q/uyusom3P8RxwExxkZhjM=
go.opentelemetry.io/otel/metric v1.38.0 h1:Kl6lzIYGAh5M159u9NgiRkmoMKjvbsKtYRwgfrA6WpA=
go.opentelemetry.io/otel/metric v1.38.0/go.mod h1:kB5n/QoRM8YwmUahxvI3bO34eVtQf2i4utNVLr9gEmI=
go.opentelemetry.io/otel/trace v1.38.0 h1:Fxk5bKrDZJUH+AMyyIXGcFAPah0oRcT+LuNtJrmcNLE=
go.opentelemetry.io/otel/trace v1.38.0/go.mod h1:j1P9ivuFsTceSWe1oY+EeW3sc+Pp42sO++GHkg4wwhs=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58=
golang.org/x/crypto v0.41.0 h1:WKYxWedPGCTVVl5+WHSSrOBT0O8lx32+zxmHxijgXp4=
golang.org/x/crypto v0.41.0/go.mod h1:pO5AFd7FA68rFak7rOAGVuygIISepHftHnr8dr6+sUc=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.43.0 h1:lat02VYK2j4aLzMzecihNvTlJNQUq316m2Mr9rnM6YE=
golang.org/x/net v0.43.0/go.mod h1:vhO1fvI4dGsIjh73sWfUVjj3N7CA9WkKJNQm2svM6Jg=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.16.0 h1:ycBJEhp9p4vXvUZNszeOq0kGTPghopOL8q0fq3vstxw=
golang.org/x/sync v0.16.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
================================================
FILE: _examples/basic/2-realtime-feed/consumer/main.go
================================================
package main
import (
"context"
"encoding/json"
"fmt"
"sync/atomic"
"time"
"github.com/ThreeDotsLabs/watermill"
"github.com/ThreeDotsLabs/watermill-kafka/v3/pkg/kafka"
"github.com/ThreeDotsLabs/watermill/message"
"github.com/ThreeDotsLabs/watermill/message/router/middleware"
"github.com/ThreeDotsLabs/watermill/message/router/plugin"
)
var (
marshaler = kafka.DefaultMarshaler{}
brokers = []string{"kafka:9092"}
)
func main() {
logger := watermill.NewStdLogger(false, false)
logger.Info("Starting the consumer", nil)
pub, err := kafka.NewPublisher(
kafka.PublisherConfig{
Brokers: brokers,
Marshaler: marshaler,
},
logger,
)
if err != nil {
panic(err)
}
r, err := message.NewRouter(message.RouterConfig{}, logger)
if err != nil {
panic(err)
}
retryMiddleware := middleware.Retry{
MaxRetries: 1,
InitialInterval: time.Millisecond * 10,
}
poisonQueue, err := middleware.PoisonQueue(pub, "poison_queue")
if err != nil {
panic(err)
}
r.AddMiddleware(
// Recoverer middleware recovers panic from handlers and middlewares
middleware.Recoverer,
// Limit incoming messages to 10 per second
middleware.NewThrottle(10, time.Second).Middleware,
// If the retries limit is exceeded (see retryMiddleware below), the message is sent
// to the poison queue (published to poison_queue topic)
poisonQueue,
// Retry middleware retries message processing if an error occurred in the handler
retryMiddleware.Middleware,
// Correlation ID middleware adds the correlation ID of the consumed message to each produced message.
// It's useful for debugging.
middleware.CorrelationID,
// Simulate errors or panics from handler
middleware.RandomFail(0.01),
middleware.RandomPanic(0.01),
)
// Close the router when a SIGTERM is sent
r.AddPlugin(plugin.SignalsHandler)
// Handler that counts consumed posts
r.AddHandler(
"posts_counter",
"posts_published",
createSubscriber("posts_counter", logger),
"posts_count",
pub,
PostsCounter{memoryCountStorage{new(int64)}}.Count,
)
// Handler that generates "feed" from consumed posts
//
// This implementation just prints the posts on stdout,
// but production ready implementation would save posts to some persistent storage.
r.AddConsumerHandler(
"feed_generator",
"posts_published",
createSubscriber("feed_generator", logger),
FeedGenerator{printFeedStorage{}}.UpdateFeed,
)
if err = r.Run(context.Background()); err != nil {
panic(err)
}
}
func createSubscriber(consumerGroup string, logger watermill.LoggerAdapter) message.Subscriber {
sub, err := kafka.NewSubscriber(
kafka.SubscriberConfig{
Brokers: brokers,
Unmarshaler: marshaler,
ConsumerGroup: consumerGroup,
},
logger,
)
if err != nil {
panic(err)
}
return sub
}
type postsCountUpdated struct {
NewCount int64 `json:"new_count"`
}
type countStorage interface {
CountAdd() (int64, error)
Count() (int64, error)
}
type memoryCountStorage struct {
count *int64
}
func (m memoryCountStorage) CountAdd() (int64, error) {
return atomic.AddInt64(m.count, 1), nil
}
func (m memoryCountStorage) Count() (int64, error) {
return atomic.LoadInt64(m.count), nil
}
type PostsCounter struct {
countStorage countStorage
}
func (p PostsCounter) Count(msg *message.Message) ([]*message.Message, error) {
// When implementing counter for production use, you'd probably need to add some kind of deduplication here,
// unless the used Pub/Sub supports exactly-once delivery.
newCount, err := p.countStorage.CountAdd()
if err != nil {
return nil, fmt.Errorf("cannot add count: %w", err)
}
producedMsg := postsCountUpdated{NewCount: newCount}
b, err := json.Marshal(producedMsg)
if err != nil {
return nil, err
}
return []*message.Message{message.NewMessage(watermill.NewUUID(), b)}, nil
}
// postAdded might look similar to the postAdded type from producer.
// It's intentionally not imported here. We avoid coupling the services at the cost of duplication.
// We don't need all of its data either (content is not displayed on the feed).
type postAdded struct {
OccurredOn time.Time `json:"occurred_on"`
Author string `json:"author"`
Title string `json:"title"`
}
type feedStorage interface {
AddToFeed(title, author string, time time.Time) error
}
type printFeedStorage struct{}
func (printFeedStorage) AddToFeed(title, author string, time time.Time) error {
fmt.Printf("Adding to feed: %s by %s @%s\n", title, author, time)
return nil
}
type FeedGenerator struct {
feedStorage feedStorage
}
func (f FeedGenerator) UpdateFeed(message *message.Message) error {
event := postAdded{}
if err := json.Unmarshal(message.Payload, &event); err != nil {
return err
}
err := f.feedStorage.AddToFeed(event.Title, event.Author, event.OccurredOn)
if err != nil {
return fmt.Errorf("cannot update feed: %w", err)
}
return nil
}
================================================
FILE: _examples/basic/2-realtime-feed/docker-compose.yml
================================================
services:
producer:
image: golang:1.25
restart: unless-stopped
depends_on:
- kafka
volumes:
- .:/app
- $GOPATH/pkg/mod:/go/pkg/mod
working_dir: /app/producer/
command: go run main.go
consumer:
image: golang:1.25
restart: unless-stopped
depends_on:
- kafka
volumes:
- .:/app
- $GOPATH/pkg/mod:/go/pkg/mod
working_dir: /app/consumer/
command: >
/bin/sh -c "go install github.com/ThreeDotsLabs/watermill/tools/mill@latest &&
go run main.go"
zookeeper:
image: confluentinc/cp-zookeeper:7.3.1
restart: unless-stopped
environment:
ZOOKEEPER_CLIENT_PORT: 2181
logging:
driver: none
kafka:
image: confluentinc/cp-kafka:7.3.1
restart: unless-stopped
logging:
driver: none
depends_on:
- zookeeper
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
================================================
FILE: _examples/basic/2-realtime-feed/producer/go.mod
================================================
module main.go
go 1.25
require (
github.com/ThreeDotsLabs/watermill v1.5.1
github.com/ThreeDotsLabs/watermill-kafka/v3 v3.1.0
github.com/brianvoe/gofakeit/v6 v6.28.0
)
require (
github.com/IBM/sarama v1.46.0 // indirect
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dnwe/otelsarama v0.0.0-20240308230250-9388d9d40bc0 // indirect
github.com/eapache/go-resiliency v1.7.0 // indirect
github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 // indirect
github.com/eapache/queue v1.1.0 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/golang/snappy v1.0.0 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/go-uuid v1.0.3 // indirect
github.com/jcmturner/aescts/v2 v2.0.0 // indirect
github.com/jcmturner/dnsutils/v2 v2.0.0 // indirect
github.com/jcmturner/gofork v1.7.6 // indirect
github.com/jcmturner/gokrb5/v8 v8.4.4 // indirect
github.com/jcmturner/rpc/v2 v2.0.3 // indirect
github.com/klauspost/compress v1.18.0 // indirect
github.com/lithammer/shortuuid/v3 v3.0.7 // indirect
github.com/oklog/ulid v1.3.1 // indirect
github.com/pierrec/lz4/v4 v4.1.22 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9 // indirect
github.com/sony/gobreaker v1.0.0 // indirect
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
go.opentelemetry.io/otel v1.38.0 // indirect
go.opentelemetry.io/otel/metric v1.38.0 // indirect
go.opentelemetry.io/otel/trace v1.38.0 // indirect
golang.org/x/crypto v0.41.0 // indirect
golang.org/x/net v0.43.0 // indirect
)
================================================
FILE: _examples/basic/2-realtime-feed/producer/go.sum
================================================
github.com/IBM/sarama v1.46.0 h1:+YTM1fNd6WKMchlnLKRUB5Z0qD4M8YbvwIIPLvJD53s=
github.com/IBM/sarama v1.46.0/go.mod h1:0lOcuQziJ1/mBGHkdp5uYrltqQuKQKM5O5FOWUQVVvo=
github.com/ThreeDotsLabs/watermill v1.5.1 h1:t5xMivyf9tpmU3iozPqyrCZXHvoV1XQDfihas4sV0fY=
github.com/ThreeDotsLabs/watermill v1.5.1/go.mod h1:Uop10dA3VeJWsSvis9qO3vbVY892LARrKAdki6WtXS4=
github.com/ThreeDotsLabs/watermill-kafka/v3 v3.1.0 h1:DfhVM1ieq+rb+bboB7aoymUgfKsEM3UbH3noQZ6+RJ4=
github.com/ThreeDotsLabs/watermill-kafka/v3 v3.1.0/go.mod h1:o1GcoF/1CSJ9JSmQzUkULvpZeO635pZe+WWrYNFlJNk=
github.com/brianvoe/gofakeit/v6 v6.28.0 h1:Xib46XXuQfmlLS2EXRuJpqcw8St6qSZz75OUo0tgAW4=
github.com/brianvoe/gofakeit/v6 v6.28.0/go.mod h1:Xj58BMSnFqcn/fAQeSK+/PLtC5kSb7FJIq4JyGa8vEs=
github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1xcsSM=
github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dnwe/otelsarama v0.0.0-20240308230250-9388d9d40bc0 h1:R2zQhFwSCyyd7L43igYjDrH0wkC/i+QBPELuY0HOu84=
github.com/dnwe/otelsarama v0.0.0-20240308230250-9388d9d40bc0/go.mod h1:2MqLKYJfjs3UriXXF9Fd0Qmh/lhxi/6tHXkqtXxyIHc=
github.com/eapache/go-resiliency v1.7.0 h1:n3NRTnBn5N0Cbi/IeOHuQn9s2UwVUH7Ga0ZWcP+9JTA=
github.com/eapache/go-resiliency v1.7.0/go.mod h1:5yPzW0MIvSe0JDsv0v+DvcjEv2FyD6iZYSs1ZI+iQho=
github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 h1:Oy0F4ALJ04o5Qqpdz8XLIpNA3WM/iSIXqxtqo7UGVws=
github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3/go.mod h1:YvSRo5mw33fLEx1+DlK6L2VV43tJt5Eyel9n9XBcR+0=
github.com/eapache/queue v1.1.0 h1:YOEu7KNc61ntiQlcEeUIoDTJ2o8mQznoNvUhiigpIqc=
github.com/eapache/queue v1.1.0/go.mod h1:6eCeP0CKFpHLu8blIFXhExK/dRa7WDZfr6jVFPTqq+I=
github.com/fortytw2/leaktest v1.3.0 h1:u8491cBMTQ8ft8aeV+adlcytMZylmA5nnwwkRZjI8vw=
github.com/fortytw2/leaktest v1.3.0/go.mod h1:jDsjWgpAGjm2CA7WthBh/CdZYEPF31XHquHwclZch5g=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/golang/snappy v1.0.0 h1:Oy607GVXHs7RtbggtPBnr2RmDArIsAefDwvrdWvRhGs=
github.com/golang/snappy v1.0.0/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/uuid v1.2.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/gorilla/securecookie v1.1.1/go.mod h1:ra0sb63/xPlUeL+yeDciTfxMRAA+MP+HVt/4epWDjd4=
github.com/gorilla/sessions v1.2.1/go.mod h1:dk2InVEVJ0sfLlnXv9EAgkf6ecYs/i80K/zI+bUmuGM=
github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I=
github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo=
github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM=
github.com/hashicorp/go-uuid v1.0.2/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
github.com/hashicorp/go-uuid v1.0.3 h1:2gKiV6YVmrJ1i2CKKa9obLvRieoRGviZFL26PcT/Co8=
github.com/hashicorp/go-uuid v1.0.3/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
github.com/jcmturner/aescts/v2 v2.0.0 h1:9YKLH6ey7H4eDBXW8khjYslgyqG2xZikXP0EQFKrle8=
github.com/jcmturner/aescts/v2 v2.0.0/go.mod h1:AiaICIRyfYg35RUkr8yESTqvSy7csK90qZ5xfvvsoNs=
github.com/jcmturner/dnsutils/v2 v2.0.0 h1:lltnkeZGL0wILNvrNiVCR6Ro5PGU/SeBvVO/8c/iPbo=
github.com/jcmturner/dnsutils/v2 v2.0.0/go.mod h1:b0TnjGOvI/n42bZa+hmXL+kFJZsFT7G4t3HTlQ184QM=
github.com/jcmturner/gofork v1.7.6 h1:QH0l3hzAU1tfT3rZCnW5zXl+orbkNMMRGJfdJjHVETg=
github.com/jcmturner/gofork v1.7.6/go.mod h1:1622LH6i/EZqLloHfE7IeZ0uEJwMSUyQ/nDd82IeqRo=
github.com/jcmturner/goidentity/v6 v6.0.1 h1:VKnZd2oEIMorCTsFBnJWbExfNN7yZr3EhJAxwOkZg6o=
github.com/jcmturner/goidentity/v6 v6.0.1/go.mod h1:X1YW3bgtvwAXju7V3LCIMpY0Gbxyjn/mY9zx4tFonSg=
github.com/jcmturner/gokrb5/v8 v8.4.4 h1:x1Sv4HaTpepFkXbt2IkL29DXRf8sOfZXo8eRKh687T8=
github.com/jcmturner/gokrb5/v8 v8.4.4/go.mod h1:1btQEpgT6k+unzCwX1KdWMEwPPkkgBtP+F6aCACiMrs=
github.com/jcmturner/rpc/v2 v2.0.3 h1:7FXXj8Ti1IaVFpSAziCZWNzbNuZmnvw/i6CqLNdWfZY=
github.com/jcmturner/rpc/v2 v2.0.3/go.mod h1:VUJYCIDm3PVOEHw8sgt091/20OJjskO/YJki3ELg/Hc=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/lithammer/shortuuid/v3 v3.0.7 h1:trX0KTHy4Pbwo/6ia8fscyHoGA+mf1jWbPJVuvyJQQ8=
github.com/lithammer/shortuuid/v3 v3.0.7/go.mod h1:vMk8ke37EmiewwolSO1NLW8vP4ZaKlRuDIi8tWWmAts=
github.com/oklog/ulid v1.3.1 h1:EGfNDEx6MqHz8B3uNV6QAib1UR2Lm97sHi3ocA6ESJ4=
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
github.com/pierrec/lz4/v4 v4.1.22 h1:cKFw6uJDK+/gfw5BcDL0JL5aBsAFdsIT18eRtLj7VIU=
github.com/pierrec/lz4/v4 v4.1.22/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9 h1:bsUq1dX0N8AOIL7EB/X911+m4EHsnWEHeJ0c+3TTBrg=
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
github.com/sony/gobreaker v1.0.0 h1:feX5fGGXSl3dYd4aHZItw+FpHLvvoaqkawKjVNiFMNQ=
github.com/sony/gobreaker v1.0.0/go.mod h1:ZKptC7FHNvhBz7dN2LGjPVBz2sZJmc0/PkyDJOjmxWY=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA=
go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A=
go.opentelemetry.io/otel v1.38.0 h1:RkfdswUDRimDg0m2Az18RKOsnI8UDzppJAtj01/Ymk8=
go.opentelemetry.io/otel v1.38.0/go.mod h1:zcmtmQ1+YmQM9wrNsTGV/q/uyusom3P8RxwExxkZhjM=
go.opentelemetry.io/otel/metric v1.38.0 h1:Kl6lzIYGAh5M159u9NgiRkmoMKjvbsKtYRwgfrA6WpA=
go.opentelemetry.io/otel/metric v1.38.0/go.mod h1:kB5n/QoRM8YwmUahxvI3bO34eVtQf2i4utNVLr9gEmI=
go.opentelemetry.io/otel/trace v1.38.0 h1:Fxk5bKrDZJUH+AMyyIXGcFAPah0oRcT+LuNtJrmcNLE=
go.opentelemetry.io/otel/trace v1.38.0/go.mod h1:j1P9ivuFsTceSWe1oY+EeW3sc+Pp42sO++GHkg4wwhs=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58=
golang.org/x/crypto v0.41.0 h1:WKYxWedPGCTVVl5+WHSSrOBT0O8lx32+zxmHxijgXp4=
golang.org/x/crypto v0.41.0/go.mod h1:pO5AFd7FA68rFak7rOAGVuygIISepHftHnr8dr6+sUc=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.43.0 h1:lat02VYK2j4aLzMzecihNvTlJNQUq316m2Mr9rnM6YE=
golang.org/x/net v0.43.0/go.mod h1:vhO1fvI4dGsIjh73sWfUVjj3N7CA9WkKJNQm2svM6Jg=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.16.0 h1:ycBJEhp9p4vXvUZNszeOq0kGTPghopOL8q0fq3vstxw=
golang.org/x/sync v0.16.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
================================================
FILE: _examples/basic/2-realtime-feed/producer/main.go
================================================
package main
import (
"encoding/json"
"fmt"
"math/rand"
"os"
"os/signal"
"sync"
"time"
"github.com/brianvoe/gofakeit/v6"
"github.com/ThreeDotsLabs/watermill"
"github.com/ThreeDotsLabs/watermill-kafka/v3/pkg/kafka"
"github.com/ThreeDotsLabs/watermill/message"
"github.com/ThreeDotsLabs/watermill/message/router/middleware"
)
var (
brokers = []string{"kafka:9092"}
messagesPerSecond = 100
numWorkers = 20
)
func main() {
logger := watermill.NewStdLogger(false, false)
logger.Info("Starting the producer", watermill.LogFields{})
publisher, err := kafka.NewPublisher(
kafka.PublisherConfig{
Brokers: brokers,
Marshaler: kafka.DefaultMarshaler{},
},
logger,
)
if err != nil {
panic(err)
}
defer publisher.Close()
closeCh := make(chan struct{})
workersGroup := &sync.WaitGroup{}
workersGroup.Add(numWorkers)
for i := 0; i < numWorkers; i++ {
go worker(publisher, workersGroup, closeCh)
}
// wait for SIGINT
c := make(chan os.Signal, 1)
signal.Notify(c, os.Interrupt)
<-c
// signal for the workers to stop publishing
close(closeCh)
// Waiting for all messages to be published
workersGroup.Wait()
logger.Info("All messages published", nil)
}
// worker publishes messages until closeCh is closed.
func worker(publisher message.Publisher, wg *sync.WaitGroup, closeCh chan struct{}) {
ticker := time.NewTicker(time.Duration(int(time.Second) / messagesPerSecond))
for {
select {
case <-closeCh:
ticker.Stop()
wg.Done()
return
case <-ticker.C:
}
msgPayload := postAdded{
OccurredOn: time.Now(),
Author: gofakeit.Username(),
Title: gofakeit.Sentence(rand.Intn(5) + 1),
Content: gofakeit.Sentence(rand.Intn(10) + 5),
}
payload, err := json.Marshal(msgPayload)
if err != nil {
panic(err)
}
msg := message.NewMessage(watermill.NewUUID(), payload)
// Use a middleware to set the correlation ID, it's useful for debugging
middleware.SetCorrelationID(watermill.NewShortUUID(), msg)
err = publisher.Publish("posts_published", msg)
if err != nil {
fmt.Println("cannot publish message:", err)
continue
}
}
}
type postAdded struct {
OccurredOn time.Time `json:"occurred_on"`
Author string `json:"author"`
Title string `json:"title"`
Content string `json:"content"`
}
================================================
FILE: _examples/basic/3-router/.validate_example.yml
================================================
validation_cmd: "go run main.go"
timeout: 120
expected_output: "Received message: [0-9a-f\\-]+"
================================================
FILE: _examples/basic/3-router/go.mod
================================================
module main.go
go 1.25
require github.com/ThreeDotsLabs/watermill v1.5.1
require (
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/lithammer/shortuuid/v3 v3.0.7 // indirect
github.com/oklog/ulid v1.3.1 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/sony/gobreaker v1.0.0 // indirect
)
================================================
FILE: _examples/basic/3-router/go.sum
================================================
github.com/ThreeDotsLabs/watermill v1.5.1 h1:t5xMivyf9tpmU3iozPqyrCZXHvoV1XQDfihas4sV0fY=
github.com/ThreeDotsLabs/watermill v1.5.1/go.mod h1:Uop10dA3VeJWsSvis9qO3vbVY892LARrKAdki6WtXS4=
github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1xcsSM=
github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/google/uuid v1.2.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/lithammer/shortuuid/v3 v3.0.7 h1:trX0KTHy4Pbwo/6ia8fscyHoGA+mf1jWbPJVuvyJQQ8=
github.com/lithammer/shortuuid/v3 v3.0.7/go.mod h1:vMk8ke37EmiewwolSO1NLW8vP4ZaKlRuDIi8tWWmAts=
github.com/oklog/ulid v1.3.1 h1:EGfNDEx6MqHz8B3uNV6QAib1UR2Lm97sHi3ocA6ESJ4=
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/sony/gobreaker v1.0.0 h1:feX5fGGXSl3dYd4aHZItw+FpHLvvoaqkawKjVNiFMNQ=
github.com/sony/gobreaker v1.0.0/go.mod h1:ZKptC7FHNvhBz7dN2LGjPVBz2sZJmc0/PkyDJOjmxWY=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.11.0 h1:ib4sjIrwZKxE5u/Japgo/7SJV3PvgjGiRNAvTVGqQl8=
github.com/stretchr/testify v1.11.0/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
================================================
FILE: _examples/basic/3-router/main.go
================================================
// Sources for https://watermill.io/learn/getting-started/
package main
import (
"context"
"fmt"
"log"
"time"
"github.com/ThreeDotsLabs/watermill"
"github.com/ThreeDotsLabs/watermill/message"
"github.com/ThreeDotsLabs/watermill/message/router/middleware"
"github.com/ThreeDotsLabs/watermill/message/router/plugin"
"github.com/ThreeDotsLabs/watermill/pubsub/gochannel"
)
var (
// For this example, we're using just a simple logger implementation,
// You probably want to ship your own implementation of `watermill.LoggerAdapter`.
logger = watermill.NewStdLogger(false, false)
)
func main() {
router, err := message.NewRouter(message.RouterConfig{}, logger)
if err != nil {
panic(err)
}
// SignalsHandler will gracefully shutdown Router when SIGTERM is received.
// You can also close the router by just calling `r.Close()`.
router.AddPlugin(plugin.SignalsHandler)
// Router level middleware are executed for every message sent to the router
router.AddMiddleware(
// CorrelationID will copy the correlation id from the incoming message's metadata to the produced messages
middleware.CorrelationID,
// The handler function is retried if it returns an error.
// After MaxRetries, the message is Nacked and it's up to the PubSub to resend it.
middleware.Retry{
MaxRetries: 3,
InitialInterval: time.Millisecond * 100,
Logger: logger,
}.Middleware,
// Recoverer handles panics from handlers.
// In this case, it passes them as errors to the Retry middleware.
middleware.Recoverer,
)
// For simplicity, we are using the gochannel Pub/Sub here,
// You can replace it with any Pub/Sub implementation, it will work the same.
pubSub := gochannel.NewGoChannel(gochannel.Config{}, logger)
// Producing some incoming messages in background
go publishMessages(pubSub)
// AddHandler returns a handler which can be used to add handler level middleware
// or to stop handler.
handler := router.AddHandler(
"struct_handler", // handler name, must be unique
"incoming_messages_topic", // topic from which we will read events
pubSub,
"outgoing_messages_topic", // topic to which we will publish events
pubSub,
structHandler{}.Handler,
)
// Handler level middleware is only executed for a specific handler
// Such middleware can be added the same way the router level ones
handler.AddMiddleware(func(h message.HandlerFunc) message.HandlerFunc {
return func(message *message.Message) ([]*message.Message, error) {
log.Println("executing handler specific middleware for ", message.UUID)
return h(message)
}
})
// just for debug, we are printing all messages received on `incoming_messages_topic`
router.AddConsumerHandler(
"print_incoming_messages",
"incoming_messages_topic",
pubSub,
printMessages,
)
// just for debug, we are printing all events sent to `outgoing_messages_topic`
router.AddConsumerHandler(
"print_outgoing_messages",
"outgoing_messages_topic",
pubSub,
printMessages,
)
// Now that all handlers are registered, we're running the Router.
// Run is blocking while the router is running.
ctx := context.Background()
if err := router.Run(ctx); err != nil {
panic(err)
}
}
func publishMessages(publisher message.Publisher) {
for {
msg := message.NewMessage(watermill.NewUUID(), []byte("Hello, world!"))
middleware.SetCorrelationID(watermill.NewUUID(), msg)
log.Printf("sending message %s, correlation id: %s\n", msg.UUID, middleware.MessageCorrelationID(msg))
if err := publisher.Publish("incoming_messages_topic", msg); err != nil {
panic(err)
}
time.Sleep(time.Second)
}
}
func printMessages(msg *message.Message) error {
fmt.Printf(
"\n> Received message: %s\n> %s\n> metadata: %v\n\n",
msg.UUID, string(msg.Payload), msg.Metadata,
)
return nil
}
type structHandler struct {
// we can add some dependencies here
}
func (s structHandler) Handler(msg *message.Message) ([]*message.Message, error) {
log.Println("structHandler received message", msg.UUID)
msg = message.NewMessage(watermill.NewUUID(), []byte("message produced by structHandler"))
return message.Messages{msg}, nil
}
================================================
FILE: _examples/basic/4-metrics/.validate_example.yml
================================================
validation_cmd: "docker compose up"
teardown_cmd: "docker compose down"
timeout: 180
expected_output: "msg=\"Message acked\""
================================================
FILE: _examples/basic/4-metrics/README.md
================================================
# Prometheus metrics showcase
This is an example application that showcases how Watermill may be monitored with Prometheus metrics.
The docker-compose bundle contains the following services:
#### Golang
A [Golang](https://hub.docker.com/_/golang) image which runs the [example code](https://github.com/ThreeDotsLabs/watermill/blob/master/_examples/basic/4-metrics/main.go). It consists of a router with a single handler.
The handler consumes messages from a [Gochannel PubSub](https://github.com/ThreeDotsLabs/watermill/tree/master/message/infrastructure/gochannel), and publishes 0-4 copies of the message with a preconfigured random delay.
Additionally, there is one goroutine which produces messages incoming to the handler with a gochannel publisher, and another goroutine which consumes the messages outgoing from the handler.
The router, the standalone publisher and the standalone subscriber are all decorated with the metrics code and their statistics will appear in the dashboard.
#### Prometheus
[Prometheus](https://hub.docker.com/r/prom/prometheus/), to scrape the metrics which the golang application exposes at `:8081/metrics` by default. It is configured by a [prometheus.yml](https://github.com/ThreeDotsLabs/watermill/blob/master/_examples/basic/4-metrics/prometheus.yml) file, which declares the endpoints that Prometheus will scrape.
#### Grafana
[Grafana](https://hub.docker.com/r/grafana/grafana), to visualize the metrics in a dashboard.
#### Running the example
To run the docker-compose bundle, go to `_examples/metrics` and execute:
```
docker-compose up
```
The golang app will start running, producing messages, passing them through the handler, and consuming the copies.
With default settings, the raw Prometheus metrics should appear at your http://localhost:8081/metrics.
The Prometheus image should expose a more advanced UI at http://localhost:9090/graph, where you can investigate all the scraped metrics.
However, what is the most useful way to monitor is through the use of Grafana, which you can access at http://localhost:3000.
#### Adding the Prometheus data source to Grafana
The fresh Grafana image should greet you with a login screen:
Just use the default `admin:admin` credentials. You can skip changing the password, if you wish.
The next thing that we need to do is to add the Prometheus data source. Click on `Add data source`.
In the following screen:
1. Enter a name for the Prometheus data source. Let's name this data source `prometheus`.
1. Choose `Prometheus` from the `Type` dropdown.
1. Enter the `http://localhost:9090` value in the HTTP/URL section.
1. You can leave the remaining settings at default and click `Save & Test`.
The Prometheus data source is now ready to use in Grafana.
#### Importing the Grafana dashboard
We have prepared a Grafana dashboard that visualizes the metrics exported by this example.
To import the Grafana dashboard, select Dashboard/Manage from the left menu, and then click on `+Import` (or go to http://localhost:3000/dashboard/import).
Enter the dashboard URL https://grafana.com/dashboards/9777 (or just the ID, 9777), and click on Load.
Then select the Prometheus data source created in the previous step. Click on `Import`, and you're done!
### Find out more
To find out more, about metrics be sure to check out the [Watermill docs](https://watermill.io/docs/metrics).
================================================
FILE: _examples/basic/4-metrics/docker-compose.yml
================================================
services:
golang:
image: golang:1.25
restart: unless-stopped
ports:
- 8080:8080
- 8081:8081
depends_on:
- prometheus
volumes:
- ../../../:/watermill
- $GOPATH/pkg/mod:/go/pkg/mod
working_dir: /watermill/_examples/basic/4-metrics
command: go run main.go -metrics :8081 -delay 0.1
prometheus:
image: prom/prometheus
restart: unless-stopped
network_mode: host
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
grafana:
image: grafana/grafana:5.2.4
network_mode: host
================================================
FILE: _examples/basic/4-metrics/go.mod
================================================
module main.go
go 1.25
require github.com/ThreeDotsLabs/watermill v1.5.1
require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/go-chi/chi/v5 v5.2.3 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/lithammer/shortuuid/v3 v3.0.7 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/oklog/ulid v1.3.1 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/prometheus/client_golang v1.23.0 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.65.0 // indirect
github.com/prometheus/procfs v0.17.0 // indirect
github.com/sony/gobreaker v1.0.0 // indirect
golang.org/x/sys v0.35.0 // indirect
google.golang.org/protobuf v1.36.8 // indirect
)
// uncomment to use local sources
// replace github.com/ThreeDotsLabs/watermill => ../../../../watermill
================================================
FILE: _examples/basic/4-metrics/go.sum
================================================
github.com/ThreeDotsLabs/watermill v1.5.1 h1:t5xMivyf9tpmU3iozPqyrCZXHvoV1XQDfihas4sV0fY=
github.com/ThreeDotsLabs/watermill v1.5.1/go.mod h1:Uop10dA3VeJWsSvis9qO3vbVY892LARrKAdki6WtXS4=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1xcsSM=
github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/go-chi/chi/v5 v5.2.3 h1:WQIt9uxdsAbgIYgid+BpYc+liqQZGMHRaUwp0JUcvdE=
github.com/go-chi/chi/v5 v5.2.3/go.mod h1:L2yAIGWB3H+phAw1NxKwWM+7eUH/lU8pOMm5hHcoops=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/uuid v1.2.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/lithammer/shortuuid/v3 v3.0.7 h1:trX0KTHy4Pbwo/6ia8fscyHoGA+mf1jWbPJVuvyJQQ8=
github.com/lithammer/shortuuid/v3 v3.0.7/go.mod h1:vMk8ke37EmiewwolSO1NLW8vP4ZaKlRuDIi8tWWmAts=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/oklog/ulid v1.3.1 h1:EGfNDEx6MqHz8B3uNV6QAib1UR2Lm97sHi3ocA6ESJ4=
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_golang v1.23.0 h1:ust4zpdl9r4trLY/gSjlm07PuiBq2ynaXXlptpfy8Uc=
github.com/prometheus/client_golang v1.23.0/go.mod h1:i/o0R9ByOnHX0McrTMTyhYvKE4haaf2mW08I+jGAjEE=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
github.com/prometheus/common v0.65.0 h1:QDwzd+G1twt//Kwj/Ww6E9FQq1iVMmODnILtW1t2VzE=
github.com/prometheus/common v0.65.0/go.mod h1:0gZns+BLRQ3V6NdaerOhMbwwRbNh9hkGINtQAsP5GS8=
github.com/prometheus/procfs v0.17.0 h1:FuLQ+05u4ZI+SS/w9+BWEM2TXiHKsUQ9TADiRH7DuK0=
github.com/prometheus/procfs v0.17.0/go.mod h1:oPQLaDAMRbA+u8H5Pbfq+dl3VDAvHxMUOVhe0wYB2zw=
github.com/sony/gobreaker v1.0.0 h1:feX5fGGXSl3dYd4aHZItw+FpHLvvoaqkawKjVNiFMNQ=
github.com/sony/gobreaker v1.0.0/go.mod h1:ZKptC7FHNvhBz7dN2LGjPVBz2sZJmc0/PkyDJOjmxWY=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.11.0 h1:ib4sjIrwZKxE5u/Japgo/7SJV3PvgjGiRNAvTVGqQl8=
github.com/stretchr/testify v1.11.0/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
golang.org/x/sys v0.35.0 h1:vz1N37gP5bs89s7He8XuIYXpyY0+QlsKmzipCbUtyxI=
golang.org/x/sys v0.35.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
google.golang.org/protobuf v1.36.8 h1:xHScyCOEuuwZEc6UtSOvPbAT4zRh0xcNRYekJwfqyMc=
google.golang.org/protobuf v1.36.8/go.mod h1:fuxRtAxBytpl4zzqUh6/eyUujkJdNiuEkXntxiD/uRU=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
================================================
FILE: _examples/basic/4-metrics/main.go
================================================
package main
import (
"context"
"errors"
"flag"
"math"
"math/rand"
"time"
"github.com/ThreeDotsLabs/watermill"
"github.com/ThreeDotsLabs/watermill/components/metrics"
"github.com/ThreeDotsLabs/watermill/message"
"github.com/ThreeDotsLabs/watermill/message/router/middleware"
"github.com/ThreeDotsLabs/watermill/message/router/plugin"
"github.com/ThreeDotsLabs/watermill/pubsub/gochannel"
)
var (
metricsAddr = flag.String("metrics", ":8081", "The address that will expose /metrics for Prometheus")
handlerDelay = flag.Float64("delay", 0, "The stdev of normal distribution of delay in handler (in seconds), to simulate load")
logger = watermill.NewStdLogger(true, true)
random = rand.New(rand.NewSource(time.Now().Unix()))
)
func delay() {
seconds := *handlerDelay
if seconds == 0 {
return
}
delay := math.Abs(random.NormFloat64() * seconds)
time.Sleep(time.Duration(float64(time.Second) * delay))
}
// handler publishes 0-4 messages with a random delay.
func handler(msg *message.Message) ([]*message.Message, error) {
delay()
numOutgoing := random.Intn(4)
outgoing := make([]*message.Message, numOutgoing)
for i := 0; i < numOutgoing; i++ {
outgoing[i] = msg.Copy()
}
return outgoing, nil
}
// consumeMessages consumes the messages exiting the handler.
func consumeMessages(subscriber message.Subscriber) {
messages, err := subscriber.Subscribe(context.Background(), "pub_topic")
if err != nil {
panic(err)
}
for msg := range messages {
msg.Ack()
}
}
// produceMessages produces the incoming messages in delays of 50-100 milliseconds.
func produceMessages(routerClosed chan struct{}, publisher message.Publisher) {
for {
select {
case <-routerClosed:
return
default:
// go on
}
time.Sleep(50*time.Millisecond + time.Duration(random.Intn(50))*time.Millisecond)
msg := message.NewMessage(watermill.NewUUID(), []byte{})
_ = publisher.Publish("sub_topic", msg)
}
}
func main() {
flag.Parse()
pubSub := gochannel.NewGoChannel(gochannel.Config{}, logger)
router, err := message.NewRouter(
message.RouterConfig{},
logger,
)
if err != nil {
panic(err)
}
prometheusRegistry, closeMetricsServer := metrics.CreateRegistryAndServeHTTP(*metricsAddr)
defer closeMetricsServer()
// we leave the namespace and subsystem empty
metricsBuilder := metrics.NewPrometheusMetricsBuilder(prometheusRegistry, "", "")
metricsBuilder.AddPrometheusRouterMetrics(router)
router.AddMiddleware(
middleware.Recoverer,
middleware.RandomFail(0.1),
middleware.RandomPanic(0.1),
)
router.AddPlugin(plugin.SignalsHandler)
router.AddHandler(
"metrics-example",
"sub_topic",
pubSub,
"pub_topic",
pubSub,
handler,
)
pub := randomFailPublisherDecorator{pubSub, 0.1}
// The handler's publisher and subscriber will be decorated by `AddPrometheusRouterMetrics`.
// We are using the same pub/sub to generate messages incoming to the handler
// and consume the outgoing messages.
// They will have `handler_name={ post.Content }
") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } var templ_7745c5c3_Var8 string templ_7745c5c3_Var8, templ_7745c5c3_Err = templ.JoinStringErrs(post.Content) if templ_7745c5c3_Err != nil { return templ.Error{Err: templ_7745c5c3_Err, FileName: `views/pages.templ`, Line: 76, Col: 43} } _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var8)) if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString("
In order to make the Forwarder universal and usable transparently, it listens to a single topic on an intermediate
database based Pub/Sub, where enveloped messages are sent with help of a decorated [Forwarder Publisher](https://github.com/ThreeDotsLabs/watermill/blob/9e04bfefbd6fef9f9ffa59956654277005fa2e8a/components/forwarder/publisher.go#L30).
The Forwarder unwraps them, and sends to a specified destined topic on the message broker.
## Example
Let's consider a following example: there's a command which responsibility is to run a lottery. It has to pick
a random user that's registered in the system as a winner. While it does so, it should also persist the decision it made by
storing a database entry associating a unique lottery ID with a picked user's ID. Additionally, as it's an
event-driven system, it should emit a `LotteryConcluded` event, so that other components could react to that appropriately.
To be precise - there will be component responsible for sending prizes to lottery winners. It will receive `LotteryConcluded`
events, and using the lottery ID embedded in the event, verify who was the winner, checking with the database entry.
In our case, the database is MySQL and the message broker is Google Pub/Sub, but it could be any two other technologies.
Approaching to implementation of such a command, we could go various ways. Below we're going to cover three possible
attempts, pointing their vulnerabilities.
### Publishing an event first, storing data next
In this approach, the command is going to publish an event first and store data just after that. While in most of the
cases that approach will probably work just fine, let's try to find out what could possibly go wrong.
There are three basic actions that the command has to do:
1. Pick a random user `A` as a lottery winner.
2. Publish a `LotteryConcluded` event telling that lottery `B` has been concluded.
3. Store in the database that the lottery `B` has been won by the user `A`.
Every of these steps could potentially fail, breaking the flow of our command. The first point wouldn't have huge
repercussions in case of its failure - we would just return an error and consider the whole command failed. No data would
be stored, no event would be emitted. We can simply rerun the command.
In case the second point fails, we'll still have no event emitted and no data stored in the database. We can rerun the
command and try once again.
What's most interesting is what could happen in case the third point fails. We'd already have the event emitted after
the second point, but no data would be stored eventually in the database. Other components would get a signal
that the lottery had been concluded, but no winner would be associated to the lottery ID sent in the event. They wouldn't
be able to verify who's the winner, so their action would have to be considered failed as well.
We still can get out of this situation, but most probably it will require some manual action, i.e., rerunning the command
with the lottery ID that the emitted event has.
{{% load-snippet-partial file="src-link/_examples/real-world-examples/transactional-events-forwarder/main.go" first_line_contains="// 1. Publishes event" last_line_contains="// In case this fails" padding_after="9" %}}
### Storing data first, publishing an event next
In the second approach, we're going to try address first approach's drawbacks. We won't leak our failure to outer
components by not emitting an event in case we don't have the state persisted properly in database. That means
we'll change the order of our actions to following:
1. Pick a random user `A` as a lottery winner.
2. Store in the database that the lottery `B` has been won by the user `A`.
3. Publish a `LotteryConcluded` event telling that lottery `B` has been concluded.
Having two first actions failed, we have no repercussions, just as in the first approach. In case of failure of the 3rd
point, we'd have data persisted in the database, but no event emitted. In this case, we wouldn't leak our failure
outside the lottery component. Although, considering the expected system behavior, we'd have no prize sent to our winner,
because no event would be delivered to the component responsible for this action.
That probably can be fixed by some manual action as well, i.e., emitting the event manually. We still can do better.
{{% load-snippet-partial file="src-link/_examples/real-world-examples/transactional-events-forwarder/main.go" first_line_contains="// 2. Persists data" last_line_contains="// In case this fails" padding_after="9" %}}
### Storing data and publishing an event in one transaction
Let's imagine our command could do the 2nd, and the 3rd point at the same time. They would be committed atomically,
meaning that any of them can't succeed having the other failed. This can be achieved by leveraging a transaction
mechanism which happens to be implemented by most of the databases in today's world. One of them is MySQL used in our
example.
In order to commit both storing data and emitting an event in one transaction, we'd have to be able to publish our
messages to MySQL. Because we don't want to change our message broker to be backed by MySQL in the whole system,
we have to find a way to do that differently.
There's a good news: Watermill provides all the tools straight away! In case the database you're using is one among MySQL,
PostgreSQL (or any other SQL), Firestore or Bolt, you can publish messages to them. **Forwarder** component will help
you with picking all the messages you publish to the database and forwarding them to a message broker of yours.
Everything you have to do is to make sure that:
1. Your command uses a publisher working in a context of a database transaction (i.e. [SQL](https://github.com/ThreeDotsLabs/watermill-sql/blob/4f39bf82b6180ca2191c791e7cb220fff22b9255/pkg/sql/publisher.go#L53),
[Firestore](https://github.com/ThreeDotsLabs/watermill-firestore/blob/b7bd31b3458884dc76076196cdc8942d18b5ab61/pkg/firestore/transactional.go#L14), [Bolt](https://github.com/ThreeDotsLabs/watermill-bolt/blob/0652f3602f6adbe4e3e39b97308fbed16dcbe29e/pkg/bolt/tx_publisher.go#L24)).
2. **Forwarder** component is running, using a database subscriber, and a message broker publisher.
The command could look like following in this case:
{{% load-snippet-partial file="src-link/_examples/real-world-examples/transactional-events-forwarder/main.go" first_line_contains="// 3. Persists data" last_line_contains="err = publisher.Publish(googleCloudEventTopic" padding_after="5" %}}
In order to make the **Forwarder** component work in background for you and forward messages from MySQL to Google Pub/Sub,
you'd have to set it up as follows:
{{% load-snippet-partial file="src-link/_examples/real-world-examples/transactional-events-forwarder/main.go" first_line_contains="// Setup the Forwarder " last_line_contains="err := fwd.Run" padding_after="3" %}}
If you wish to explore the example more, you can find it implemented [here](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/real-world-examples/transactional-events-forwarder/main.go).
================================================
FILE: docs/content/advanced/metrics.md
================================================
+++
title = "Metrics"
description = "Monitor Watermill in realtime"
date = 2019-02-12T21:00:00+01:00
weight = -400
draft = false
bref = "Monitor Watermill in realtime using Prometheus"
+++
Monitoring of Watermill may be performed by using decorators for publishers/subscribers and middlewares for handlers.
We provide a default implementation using Prometheus, based on the official [Prometheus client](https://github.com/prometheus/client_golang) for Go.
The `components/metrics` package exports `PrometheusMetricsBuilder`, which provides convenience functions to wrap publishers, subscribers and handlers so that they update the relevant Prometheus registry:
{{% load-snippet-partial file="src-link/components/metrics/builder.go" first_line_contains="// PrometheusMetricsBuilder" last_line_contains="func (b PrometheusMetricsBuilder)" %}}
## Wrapping publishers, subscribers and handlers
If you are using Watermill's [router](/docs/messages-router) (which is recommended in most cases), you can use a single convenience function `AddPrometheusRouterMetrics` to ensure that all the handlers added to this router are wrapped to update the Prometheus registry, together with their publishers and subscribers:
{{% load-snippet-partial file="src-link/components/metrics/builder.go" first_line_contains="// AddPrometheusRouterMetrics" last_line_contains="AddMiddleware" padding_after="1" %}}
Example use of `AddPrometheusRouterMetrics`:
{{% load-snippet-partial file="src-link/_examples/basic/4-metrics/main.go" first_line_contains="// we leave the namespace" last_line_contains="metricsBuilder.AddPrometheusRouterMetrics" %}}
In the snippet above, we have left the `namespace` and `subsystem` arguments empty. The Prometheus client library [uses these](https://godoc.org/github.com/prometheus/client_golang/prometheus#BuildFQName) to prefix the metric names. You may want to use namespace or subsystem, but be aware that this will impact the metric names and you will have to adjust the Grafana dashboard accordingly.
The `PrometheusMetricsBuilder` allows for custom configuration of histogram buckets by setting the `PublishBuckets` or `HandlerBuckets` field.
If `HandlerBuckets` is not provided, default watermill's values will be used, which are one order of magnitude smaller than default buckets (5ms~10s), because the handler execution times are typically shorter (µs~ms range).
For `PublishBuckets`, the default values are the same as the default Prometheus buckets (5ms~10s).
Standalone publishers and subscribers may also be decorated through the use of dedicated methods of `PrometheusMetricBuilder`:
{{% load-snippet-partial file="src-link/_examples/basic/4-metrics/main.go" first_line_contains="subWithMetrics, err := " last_line_contains="pubWithMetrics, err := " padding_after="3" %}}
## Exposing the /metrics endpoint
In accordance with how Prometheus works, the service needs to expose a HTTP endpoint for scraping. By convention, it is a GET endpoint, and its path is usually `/metrics`.
To serve this endpoint, there are two convenience functions, one using a previously created Prometheus Registry, while the other also creates a new registry:
{{% load-snippet-partial file="src-link/components/metrics/http.go" first_line_contains="// CreateRegistryAndServeHTTP" last_line_contains="func ServeHTTP(" %}}
Here is an example of its use in practice:
{{% load-snippet-partial file="src-link/_examples/basic/4-metrics/main.go" first_line_contains="prometheusRegistry, closeMetricsServer :=" last_line_contains="metricsBuilder.AddPrometheusRouterMetrics" %}}
## Example application
To see how the metrics dashboard works in practice, you can check out the [metrics example](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/basic/4-metrics).
Follow the instructions in the example's [README](https://github.com/ThreeDotsLabs/watermill/blob/master/_examples/basic/4-metrics/README.md) to make it run and add the Prometheus data source to Grafana.
## Grafana dashboard
We have prepared a [Grafana dashboard](https://grafana.com/grafana/dashboards/9777-watermill/) to use with the metrics implementation described above. It provides basic information about the throughput, failure rates and publish/handler durations.
If you want to check out the dashboard on your machine, you can use the [Example application](#example-application).
To find out more about the metrics that are exported to Prometheus, see [Exported metrics](#exported-metrics).
### Importing the dashboard
To import the Grafana dashboard, select Dashboard/Manage from the left menu, and then click on `+Import`.
Enter the dashboard URL https://grafana.com/dashboards/9777 (or just the ID, 9777), and click on Load.
Then select your Prometheus data source that scrapes the `/metrics` endpoint. Click on `Import`, and you're done!
## Exported metrics
Listed below are all the metrics that are registered on the Prometheus Registry by `PrometheusMetricsBuilder`.
For more information on Prometheus metric types, please refer to [Prometheus docs](https://prometheus.io/docs/concepts/metric_types).
| Object | Metric | Description | Labels/Values |
|---|---|---|---|
| Subscriber | subscriber_messages_received_total |
A Prometheus Counter. Counts the number of messages obtained by the subscriber. |
acked is either "acked" or "nacked". |
handler_name is set if the subscriber operates within a handler; "<no handler>" otherwise. |
|||
subscriber_name identifies the subscriber. If it implements fmt.Stringer, it is the result of `String()`, package.structName otherwise. |
|||
| Handler | handler_execution_time_seconds |
A Prometheus Histogram. Registers the execution time of the handler function wrapped by the middleware. |
handler_name is the name of the handler. |
success is either "true" or "false", depending on whether the wrapped handler function returned an error or not. |
|||
| Publisher | publish_time_seconds |
A Prometheus Histogram. Registers the time of execution of the Publish function of the decorated publisher. |
success is either "true" or "false", depending on whether the decorated publisher returned an error or not. |
handler_name is set if the publisher operates within a handler; "<no handler>" otherwise. |
|||
publisher_name identifies the publisher. If it implements fmt.Stringer, it is the result of `String()`, package.structName otherwise. |
The `cqrs` component provides some useful abstractions built on top of Pub/Sub and Router that help to implement the CQRS pattern.
You don't need to implement the entire CQRS. It's very common to use just the event part of this component to build event-driven applications.
### Building blocks
#### Event
The event represents something that already took place. Events are immutable.
#### Event Bus
{{% load-snippet-partial file="src-link/components/cqrs/event_bus.go" first_line_contains="// EventBus" last_line_contains="type EventBus" padding_after="0" %}}
{{% load-snippet-partial file="src-link/components/cqrs/event_bus.go" first_line_contains="type EventBusConfig" last_line_contains="func (c *EventBusConfig) setDefaults()" padding_after="4" %}}
#### Event Processor
{{% load-snippet-partial file="src-link/components/cqrs/event_processor.go" first_line_contains="// EventProcessor" last_line_contains="type EventProcessor" padding_after="0" %}}
{{% load-snippet-partial file="src-link/components/cqrs/event_processor.go" first_line_contains="type EventProcessorConfig" last_line_contains="func (c *EventProcessorConfig) setDefaults()" padding_after="4" %}}
#### Event Group Processor
{{% load-snippet-partial file="src-link/components/cqrs/event_processor_group.go" first_line_contains="// EventGroupProcessor" last_line_contains="type EventGroupProcessor" padding_after="0" %}}
{{% load-snippet-partial file="src-link/components/cqrs/event_processor_group.go" first_line_contains="type EventGroupProcessorConfig" last_line_contains="func (c *EventGroupProcessorConfig) setDefaults()" padding_after="4" %}}
Learn more in [Event Group Processor](#event-handler-groups).
#### Event Handler
{{% load-snippet-partial file="src-link/components/cqrs/event_handler.go" first_line_contains="// EventHandler" last_line_contains="type EventHandler" padding_after="0" %}}
#### Command
The command is a simple data structure, representing the request for executing some operation.
#### Command Bus
{{% load-snippet-partial file="src-link/components/cqrs/command_bus.go" first_line_contains="// CommandBus" last_line_contains="type CommandBus" padding_after="0" %}}
{{% load-snippet-partial file="src-link/components/cqrs/command_bus.go" first_line_contains="type CommandBusConfig" last_line_contains="func (c *CommandBusConfig) setDefaults()" padding_after="4" %}}
#### Command Processor
{{% load-snippet-partial file="src-link/components/cqrs/command_processor.go" first_line_contains="// CommandProcessor" last_line_contains="type CommandProcessor" padding_after="0" %}}
{{% load-snippet-partial file="src-link/components/cqrs/command_processor.go" first_line_contains="type CommandProcessorConfig" last_line_contains="func (c *CommandProcessorConfig) setDefaults()" padding_after="4" %}}
#### Command Handler
{{% load-snippet-partial file="src-link/components/cqrs/command_handler.go" first_line_contains="// CommandHandler" last_line_contains="type CommandHandler" padding_after="0" %}}
#### Command and Event Marshaler
{{% load-snippet-partial file="src-link/components/cqrs/marshaler.go" first_line_contains="// CommandEventMarshaler" last_line_contains="NameFromMessage(" padding_after="1" %}}
#### Command and Event Marshaler Decorator
Sometimes it's useful to add extra metadata to each command or event after marshaling it to a message. For example, you may want to add a partition key to each message using Kafka.
You can use `CommandEventMarshalerDecorator` to extend a marshaler with an extra step.
{{% load-snippet-partial file="src-link/components/cqrs/marshaler.go" first_line_contains="// CommandEventMarshalerDecorator" last_line_contains="}" padding_after="0" %}}
```go
type Event interface {
PartitionKey() string
}
// ...
cqrsMarshaler := CommandEventMarshalerDecorator{
CommandEventMarshaler: cqrs.JSONMarshaler{},
DecorateFunc: func(v any, msg *message.Message) error {
pm, ok := v.(Event)
if !ok {
return fmt.Errorf("%T does not implement Event and can't be marshaled", v)
}
partitionKey := pm.PartitionKey()
if partitionKey == "" {
return fmt.Errorf("PartitionKey is empty")
}
msg.Metadata.Set(PartitionKeyMetadataField, partitionKey)
return nil
},
}
```
## Usage
### Example domain
As an example, we will use a simple domain, that is responsible for handing room booking in a hotel.
We will use **Event Storming** notation to show the model of this domain.
Legend:
- **blue** post-its are commands
- **orange** post-its are events
- **green** post-its are read models, asynchronously generated from events
- **violet** post-its are policies, which are triggered by events and produce commands
- **pink** post its are hot-spots; we mark places where problems often occur
The domain is simple:
- A Guest is able to **book a room**.
- **Whenever a room is booked, we order a beer** for the guest (because we love our guests).
- We know that sometimes there are **not enough beers**.
- We generate a **financial report** based on the bookings.
### Sending a command
For the beginning, we need to simulate the guest's action.
{{% load-snippet-partial file="src-link/_examples/basic/5-cqrs-protobuf/main.go" first_line_contains="bookRoomCmd := &BookRoom{" last_line_contains="panic(err)" padding_after="1" %}}
### Command handler
`BookRoomHandler` will handle our command.
{{% load-snippet-partial file="src-link/_examples/basic/5-cqrs-protobuf/main.go" first_line_contains="// BookRoomHandler is a command handler" last_line_contains="// OrderBeerOnRoomBooked is an event handler" padding_after="0" %}}
### Event handler
As mentioned before, we want to order a beer every time when a room is booked (*"Whenever a Room is booked"* post-it). We do it by using the `OrderBeer` command.
{{% load-snippet-partial file="src-link/_examples/basic/5-cqrs-protobuf/main.go" first_line_contains="// OrderBeerOnRoomBooked is an event handler" last_line_contains="// OrderBeerHandler is a command handler" padding_after="0" %}}
`OrderBeerHandler` is very similar to `BookRoomHandler`. The only difference is, that it sometimes returns an error when there are not enough beers, which causes redelivery of the command.
You can find the entire implementation in the [example source code](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/basic/5-cqrs-protobuf/?utm_source=cqrs_doc).
### Event Handler groups
By default, each event handler has a separate subscriber instance.
It works fine, if just one event type is sent to the topic.
In the scenario, when we have multiple event types on one topic, you have two options:
1. You can set `EventConfig.AckOnUnknownEvent` to true - it will acknowledge all events that are not handled by handler,
2. You can use Event Handler groups mechanism.
**Key differences between `EventProcessor` and `EventGroupProcessor`:**
1. `EventProcessor`:
- Each handler has its own subscriber instance
- One handler per event type
- Simple one-to-one matching of events to handlers
2. `EventGroupProcessor`:
- Group of handlers share a single subscriber instance (and one consumer group, if such mechanism is supported -- allows to maintain order of events),
- One handler group can support multiple event types,
- When message arrives to the topic, Watermill will match it to the handler in the group based on event type
**Event Handler groups are helpful when you have multiple event types on one topic and you want to maintain order of events.**
Thanks to using one subscriber instance and consumer group, events will be processed in the order they were sent.
{{< callout context="note" title="Note" icon="outline/info-circle" >}}
It's supported to have multiple handlers for the same event type in one group, but we recommend to not do that.
Please keep in mind that those handlers will be processed within the same message.
If first handler succeeds and the second fails, the message will be re-delivered and the first will be re-executed.
{{< /callout >}}
To use event groups, you need to set `GenerateHandlerGroupSubscribeTopic` and `GroupSubscriberConstructor` options in [`EventConfig`](#event-config).
After that, you can use `AddHandlersGroup` on [`EventProcessor`](#event-processor).
{{% load-snippet-partial file="src-link/_examples/basic/6-cqrs-ordered-events/main.go" first_line_contains="eventProcessor.AddHandlersGroup(" last_line_contains="if err != nil {" padding_after="0" %}}
Both `GenerateHandlerGroupSubscribeTopic` and `GroupSubscriberConstructor` receives information about group name in function arguments.
You can see a fully working example with event groups in our [examples](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/basic/6-cqrs-ordered-events/).
### Generic handlers
Since Watermill v1.3 it's possible to use generic handlers for commands and events. It's useful when you have a lot of commands/events and you don't want to create a handler for each of them.
{{% load-snippet-partial file="src-link/_examples/basic/6-cqrs-ordered-events/main.go" first_line_contains="cqrs.NewGroupEventHandler" last_line_contains=")," padding_after="0" %}}
Under the hood, it creates EventHandler or CommandHandler implementation.
It's available for all kind of handlers.
{{% load-snippet-partial file="src-link/components/cqrs/command_handler.go" first_line_contains="// NewCommandHandler" last_line_contains="func NewCommandHandler" padding_after="0" %}}
{{% load-snippet-partial file="src-link/components/cqrs/event_handler.go" first_line_contains="// NewEventHandler" last_line_contains="func NewEventHandler" padding_after="0" %}}
{{% load-snippet-partial file="src-link/components/cqrs/event_handler.go" first_line_contains="// NewGroupEventHandler" last_line_contains="func NewGroupEventHandler" padding_after="0" %}}
### Building a read model with the event handler
{{% load-snippet-partial file="src-link/_examples/basic/5-cqrs-protobuf/main.go" first_line_contains="// BookingsFinancialReport is a read model" last_line_contains="func main() {" padding_after="0" %}}
### Wiring it up
We have all the blocks to build our CQRS application.
We will use the AMQP (RabbitMQ) as our message broker: [AMQP]({{< ref "/pubsubs/amqp" >}}).
Under the hood, CQRS is using Watermill's message router. If you are not familiar with it and want to learn how it works, you should check [Getting Started guide]({{< ref "getting-started" >}}).
It will also show you how to use some standard messaging patterns, like metrics, poison queue, throttling, correlation and other tools used by every message-driven application. Those come built-in with Watermill.
Let's go back to the CQRS. As you already know, CQRS is built from multiple components, like Command or Event buses, handlers, processors, etc.
{{% load-snippet-partial file="src-link/_examples/basic/5-cqrs-protobuf/main.go" first_line_contains="main() {" last_line_contains="err := router.Run(" padding_after="3" %}}
And that's all. We have a working CQRS application.
### What's next?
As mentioned before, if you are not familiar with Watermill, we highly recommend reading [Getting Started guide]({{< ref "getting-started" >}}).
================================================
FILE: docs/content/docs/message/.validate_example.yml
================================================
validation_cmd: "go run receiving-ack.go"
timeout: 30
expected_output: "ack received"
================================================
FILE: docs/content/docs/message/go.mod
================================================
module receiving-ack.go
require github.com/ThreeDotsLabs/watermill v1.5.1
require (
github.com/google/uuid v1.6.0 // indirect
github.com/lithammer/shortuuid/v3 v3.0.7 // indirect
github.com/oklog/ulid v1.3.1 // indirect
github.com/pkg/errors v0.9.1 // indirect
)
go 1.25
================================================
FILE: docs/content/docs/message/go.sum
================================================
github.com/ThreeDotsLabs/watermill v1.5.1 h1:t5xMivyf9tpmU3iozPqyrCZXHvoV1XQDfihas4sV0fY=
github.com/ThreeDotsLabs/watermill v1.5.1/go.mod h1:Uop10dA3VeJWsSvis9qO3vbVY892LARrKAdki6WtXS4=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/google/uuid v1.2.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/lithammer/shortuuid/v3 v3.0.7 h1:trX0KTHy4Pbwo/6ia8fscyHoGA+mf1jWbPJVuvyJQQ8=
github.com/lithammer/shortuuid/v3 v3.0.7/go.mod h1:vMk8ke37EmiewwolSO1NLW8vP4ZaKlRuDIi8tWWmAts=
github.com/oklog/ulid v1.3.1 h1:EGfNDEx6MqHz8B3uNV6QAib1UR2Lm97sHi3ocA6ESJ4=
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/testify v1.11.0 h1:ib4sjIrwZKxE5u/Japgo/7SJV3PvgjGiRNAvTVGqQl8=
github.com/stretchr/testify v1.11.0/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
================================================
FILE: docs/content/docs/message/receiving-ack.go
================================================
package main
import (
"log"
"time"
"github.com/ThreeDotsLabs/watermill/message"
)
func main() {
msg := message.NewMessage("1", []byte("foo"))
go func() {
time.Sleep(time.Millisecond * 10)
msg.Ack()
}()
select {
case <-msg.Acked():
log.Print("ack received")
case <-msg.Nacked():
log.Print("nack received")
}
}
================================================
FILE: docs/content/docs/message.md
================================================
+++
title = "Message"
description = "Message is one of core parts of Watermill"
date = 2018-12-05T12:42:40+01:00
weight = -1000
draft = false
bref = "Message is one of core parts of Watermill"
+++
Message is one of core parts of Watermill. Messages are emitted by [*Publishers*]({{< ref "/docs/pub-sub#publisher" >}}) and received by [*Subscribers*]({{< ref "/docs/pub-sub#subscriber" >}}).
When a message is processed, you should send an [`Ack()`]({{< ref "#ack" >}}) or a [`Nack()`]({{< ref "#ack" >}}) when the processing failed.
`Acks` and `Nacks` are processed by Subscribers (in default implementations, the subscribers are waiting for an `Ack` or a `Nack`).
{{% load-snippet-partial file="src-link/message/message.go" first_line_contains="type Message struct {" last_line_contains="ctx context.Context" padding_after="2" %}}
## Ack
#### Sending `Ack`
{{% load-snippet-partial file="src-link/message/message.go" first_line_contains="// Ack" last_line_contains="func (m *Message) Ack() bool {" padding_after="0" %}}
## Nack
{{% load-snippet-partial file="src-link/message/message.go" first_line_contains="// Nack" last_line_contains="func (m *Message) Nack() bool {" padding_after="0" %}}
#### Receiving `Ack/Nack`
{{% load-snippet-partial file="docs/message/receiving-ack.go" first_line_contains="select {" last_line_contains="}" padding_after="0" %}}
## Context
Message contains the standard library context, just like an HTTP request.
This context is used to propagate cancellation and deadlines when publishing and processing messages.
You can set the context using the `SetContext` method or the `NewMessageWithContext` constructor.
{{% load-snippet-partial file="src-link/message/message.go" first_line_contains="// NewMessageWithContext" last_line_contains="}" %}}
{{% load-snippet-partial file="src-link/message/message.go" first_line_contains="// Context" last_line_contains="func (m *Message) SetContext" padding_after="2" %}}
================================================
FILE: docs/content/docs/messages-router.md
================================================
+++
title = "Router"
description = "The Magic Glue of Watermill"
date = 2018-12-05T12:48:04+01:00
weight = -850
draft = false
bref = "The Magic Glue of Watermill"
toc = true
+++
[*Publishers and Subscribers*]({{< ref "/docs/pub-sub" >}}) are rather low-level parts of Watermill.
In production use, you'd usually want to use a high-level interface and features like [correlation, metrics, poison queue, retrying, throttling, etc.]({{< ref "/docs/messages-router#middleware" >}}).
You also might not want to send an Ack when processing was successful. Sometimes, you'd like to send a message after processing of another message finishes.
To handle these requirements, there is a component named **Router**.
## Configuration
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="type RouterConfig struct {" last_line_contains="RouterConfig) Validate()" padding_after="2" %}}
## Handler
At the beginning you need to implement `HandlerFunc`:
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// HandlerFunc is" last_line_contains="type HandlerFunc func" padding_after="1" %}}
Next, you have to add a new handler with `Router.AddHandler`:
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// AddHandler" last_line_contains=") {" padding_after="0" %}}
See an example usage from [Getting Started]({{< ref "/learn/getting-started#using-messages-router" >}}):
{{% load-snippet-partial file="src-link/_examples/basic/3-router/main.go" first_line_contains="// AddHandler returns a handler" last_line_contains="return h(message)" padding_after="3" %}}
## Consumer handler
Not every handler will produce new messages. You can add this kind of handler by using `Router.AddConsumerHandler`:
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// AddConsumerHandler" last_line_contains=") {" padding_after="0" %}}
## Ack
By default, `msg.Ack()` is called when `HandlerFunc` doesn't return an error. If an error is returned, `msg.Nack()` will be called.
Because of this, you don't have to call `msg.Ack()` or `msg.Nack()` after a message is processed (you can if you want, of course).
## Producing messages
When returning multiple messages from a handler, be aware that most Publisher implementations don't support [atomic publishing of messages]({{< ref "/docs/pub-sub#publishing-multiple-messages" >}}). It may end up producing only some messages and sending `msg.Nack()` if the broker or the storage are not available.
If it is an issue, consider publishing just one message with each handler.
## Running the Router
To run the Router, you need to call `Run()`.
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// Run" last_line_contains="func (r *Router) Run(ctx context.Context) (err error) {" padding_after="0" %}}
### Ensuring that the Router is running
It can be useful to know if the router is running. You can use the `Running()` method for this.
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// Running" last_line_contains="func (r *Router) Running()" padding_after="0" %}}
You can also use `IsRunning` function, that returns bool:
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// IsRunning" last_line_contains="func (r *Router) IsRunning()" padding_after="0" %}}
### Closing the Router
To close the Router, you need to call `Close()`.
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// Close gracefully" last_line_contains="func (r *Router) Close()" padding_after="1" %}}
`Close()` will close all publishers and subscribers, and wait for all handlers to finish.
`Close()` will wait for a timeout configured in `RouterConfig.CloseTimeout`.
If the timeout is reached, `Close()` will return an error.
## Adding handler after the router has started
You can add a new handler while the router is already running.
To do that, you need to call `AddConsumerHandler` or `AddHandler` and call `RunHandlers`.
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// RunHandlers" last_line_contains="func (r *Router) RunHandlers" padding_after="0" %}}
## Stopping running handler
It is possible to stop **just one running handler** by calling `Stop()`.
Please keep in mind, that router will be closed when there are no running handlers.
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// Stop" last_line_contains="func (h *Handler) Stop()" padding_after="0" %}}
## Execution models
*Subscribers* can consume either one message at a time or multiple messages in parallel.
* **Single stream of messages** is the simplest approach and it means that until a `msg.Ack()` is called, the subscriber
will not receive any new messages.
* **Multiple message streams** are supported only by some subscribers. By subscribing to multiple topic partitions at once,
several messages can be consumed in parallel, even previous messages that were not acked (for example, the Kafka subscriber
works like this). Router handles this model by running concurrent `HandlerFunc`s, one for each partition.
See the chosen Pub/Sub documentation for supported execution models.
## Middleware
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// HandlerMiddleware" last_line_contains="type HandlerMiddleware" padding_after="1" %}}
A full list of standard middleware can be found in [Middleware]({{< ref "/docs/middlewares" >}}).
## Plugin
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// RouterPlugin" last_line_contains="type RouterPlugin" padding_after="1" %}}
A full list of standard plugins can be found in [message/router/plugin](https://github.com/ThreeDotsLabs/watermill/tree/master/message/router/plugin).
## Context
Each message received by handler holds some useful values in the `context`:
{{% load-snippet-partial file="src-link/message/router_context.go" first_line_contains="// HandlerNameFromCtx" last_line_contains="func PublishTopicFromCtx" padding_after="2" %}}
================================================
FILE: docs/content/docs/middlewares.md
================================================
+++
title = "Middleware"
description = "Add generic functionalities to your handlers in an unobtrusive way"
date = 2019-06-01T19:00:00+01:00
weight = -500
draft = false
bref = "Add functionality to handlers"
+++
## Introduction
Middleware wrap handlers with functionality that is important, but not relevant for the primary handler's logic.
Examples include retrying the handler after an error was returned, or recovering from panic in the handler
and capturing the stacktrace.
Middleware wrap the handler function like this:
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// HandlerMiddleware" last_line_contains="type HandlerMiddleware" %}}
## Usage
Middleware can be executed for all as well as for a specific handler in a router. When middleware is added directly
to a router it will be executed for all handlers provided for a router. If a middleware should be executed only
for a specific handler, it needs to be added to handler in the router.
Example usage is shown below:
{{% load-snippet-partial file="src-link/_examples/basic/3-router/main.go" first_line_contains="router, err := message.NewRouter(message.RouterConfig{}, logger)" last_line_contains="// Now that all handlers are registered, we're running the Router." padding_after="1" %}}
## Available middleware
Below are the middleware provided by Watermill and ready to use. You can also easily implement your own.
For example, if you'd like to store every received message in some kind of log, it's the best way to do it.
{{% readfile file="/content/src-link/middleware-defs.md" %}}
================================================
FILE: docs/content/docs/pub-sub.md
================================================
+++
title = "Publisher & Subscriber"
description = "Publishers and Subscribers"
date = 2018-12-05T12:47:30+01:00
weight = -900
draft = false
bref = "Publishers and Subscribers"
+++
## Publisher
{{% load-snippet-partial file="src-link/message/pubsub.go" first_line_contains="Publisher interface {" last_line_contains="Close() error" padding_after="1" %}}
### Publishing multiple messages
Most publishers implementations don't support atomic publishing of messages.
This means that if publishing one of the messages fails, the next messages won't be published.
### Async publish
Publish can be synchronous or asynchronous - it depends on the implementation.
#### `Close()`
`Close` should flush unsent messages if the publisher is asynchronous.
**It is important to not forget to close the subscriber**. Otherwise you may lose some of the messages.
## Subscriber
{{% load-snippet-partial file="src-link/message/pubsub.go" first_line_contains="Subscriber interface {" last_line_contains="Close() error" padding_after="1" %}}
### Ack/Nack mechanism
It is the *Subscriber's* responsibility to handle an `Ack` and a `Nack` from a message.
A proper implementation should wait for an `Ack` or a `Nack` before consuming the next message.
**Important Subscriber's implementation notice**:
Ack/offset to message's storage/broker **must** be sent after Ack from Watermill's message.
Otherwise there is a chance to lose messages if the process dies before the messages have been processed.
#### `Close()`
`Close` closes all subscriptions with their output channels and flushes offsets, etc. when needed.
## At-least-once delivery
Watermill is built with [at-least-once delivery](http://www.cloudcomputingpatterns.org/at_least_once_delivery/) semantics.
That means when some error occurs when processing a message and an Ack cannot be sent, the message will be redelivered.
You need to keep it in mind and build your application to be [idempotent](http://www.cloudcomputingpatterns.org/idempotent_processor/) or implement a deduplication mechanism.
Unfortunately, it's not possible to create a universal [*middleware*]({{< ref "/docs/messages-router#middleware" >}}) for deduplication, so we encourage you to build your own.
## Universal tests
Every Pub/Sub is similar in most aspects.
To avoid implementing separate tests for every Pub/Sub, we've created a test suite which should be passed by any Pub/Sub
implementation.
These tests can be found in `pubsub/tests/test_pubsub.go`.
## Built-in implementations
To check available Pub/Sub implementations, see [Supported Pub/Subs]({{< ref "/pubsubs" >}}).
## Implementing custom Pub/Sub
See [Implementing custom Pub/Sub]({{< ref "/development/pub-sub-implementing" >}}) for instructions on how to introduce support for
a new Pub/Sub.
We will also be thankful for submitting [pull requests](https://github.com/ThreeDotsLabs/watermill/pulls) with the new Pub/Sub implementations.
You can also request a new Pub/Sub implementation by submitting a [new issue](https://github.com/ThreeDotsLabs/watermill/issues).
================================================
FILE: docs/content/docs/snippets/amqp-consumer-groups/.validate_example.yml
================================================
validation_cmd: "docker compose up"
teardown_cmd: "docker compose down"
timeout: 120
expected_output: "payload: Hello, world!"
================================================
FILE: docs/content/docs/snippets/amqp-consumer-groups/docker-compose.yml
================================================
services:
server:
image: golang:1.25
restart: unless-stopped
depends_on:
- rabbitmq
volumes:
- .:/app
- $GOPATH/pkg/mod:/go/pkg/mod
working_dir: /app
command: go run main.go
rabbitmq:
image: rabbitmq:3.7
restart: unless-stopped
================================================
FILE: docs/content/docs/snippets/amqp-consumer-groups/go.mod
================================================
module main.go
require (
github.com/ThreeDotsLabs/watermill v1.5.1
github.com/ThreeDotsLabs/watermill-amqp/v2 v2.1.3
)
require (
github.com/cenkalti/backoff/v3 v3.2.2 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/lithammer/shortuuid/v3 v3.0.7 // indirect
github.com/oklog/ulid v1.3.1 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/rabbitmq/amqp091-go v1.10.0 // indirect
)
go 1.25
================================================
FILE: docs/content/docs/snippets/amqp-consumer-groups/go.sum
================================================
github.com/ThreeDotsLabs/watermill v1.5.1 h1:t5xMivyf9tpmU3iozPqyrCZXHvoV1XQDfihas4sV0fY=
github.com/ThreeDotsLabs/watermill v1.5.1/go.mod h1:Uop10dA3VeJWsSvis9qO3vbVY892LARrKAdki6WtXS4=
github.com/ThreeDotsLabs/watermill-amqp/v2 v2.1.3 h1:fkhmiBtaLn+rz5lbkPD1h8tXHfKy3gX0vMtGmxNtAsk=
github.com/ThreeDotsLabs/watermill-amqp/v2 v2.1.3/go.mod h1:xy2qXKcJpgrJURRT6YwgRyGL3qIi6/sOHrDI0MO/r5I=
github.com/cenkalti/backoff/v3 v3.2.2 h1:cfUAAO3yvKMYKPrvhDuHSwQnhZNk/RMHKdZqKTxfm6M=
github.com/cenkalti/backoff/v3 v3.2.2/go.mod h1:cIeZDE3IrqwwJl6VUwCN6trj1oXrTS4rc0ij+ULvLYs=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/google/uuid v1.2.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I=
github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo=
github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM=
github.com/lithammer/shortuuid/v3 v3.0.7 h1:trX0KTHy4Pbwo/6ia8fscyHoGA+mf1jWbPJVuvyJQQ8=
github.com/lithammer/shortuuid/v3 v3.0.7/go.mod h1:vMk8ke37EmiewwolSO1NLW8vP4ZaKlRuDIi8tWWmAts=
github.com/oklog/ulid v1.3.1 h1:EGfNDEx6MqHz8B3uNV6QAib1UR2Lm97sHi3ocA6ESJ4=
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/rabbitmq/amqp091-go v1.10.0 h1:STpn5XsHlHGcecLmMFCtg7mqq0RnD+zFr4uzukfVhBw=
github.com/rabbitmq/amqp091-go v1.10.0/go.mod h1:Hy4jKW5kQART1u+JkDTF9YYOQUHXqMuhrgxOEeS7G4o=
github.com/stretchr/testify v1.11.0 h1:ib4sjIrwZKxE5u/Japgo/7SJV3PvgjGiRNAvTVGqQl8=
github.com/stretchr/testify v1.11.0/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
================================================
FILE: docs/content/docs/snippets/amqp-consumer-groups/main.go
================================================
package main
import (
"context"
"log"
"github.com/ThreeDotsLabs/watermill"
"github.com/ThreeDotsLabs/watermill-amqp/v2/pkg/amqp"
"github.com/ThreeDotsLabs/watermill/message"
)
var amqpURI = "amqp://guest:guest@rabbitmq:5672/"
func createSubscriber(queueSuffix string) *amqp.Subscriber {
subscriber, err := amqp.NewSubscriber(
// This config is based on this example: https://www.rabbitmq.com/tutorials/tutorial-three-go.html
// to create just a simple queue, you can use NewDurableQueueConfig or create your own config.
amqp.NewDurablePubSubConfig(
amqpURI,
// Rabbit's queue name in this example is based on Watermill's topic passed to Subscribe
// plus provided suffix.
//
// Exchange is Rabbit's "fanout", so when subscribing with suffix other than "test_consumer_group",
// it will also receive all messages. It will work like separate consumer groups in Kafka.
amqp.GenerateQueueNameTopicNameWithSuffix(queueSuffix),
),
watermill.NewStdLogger(false, false),
)
if err != nil {
panic(err)
}
return subscriber
}
func main() {
subscriber1 := createSubscriber("test_consumer_group_1")
messages1, err := subscriber1.Subscribe(context.Background(), "example.topic")
if err != nil {
panic(err)
}
go process("subscriber_1", messages1)
subscriber2 := createSubscriber("test_consumer_group_2")
messages2, err := subscriber2.Subscribe(context.Background(), "example.topic")
if err != nil {
panic(err)
}
// subscriber2 will receive all messages independently from subscriber1
go process("subscriber_2", messages2)
publisher, err := amqp.NewPublisher(
amqp.NewDurablePubSubConfig(
amqpURI,
nil, // generateQueueName is not used with publisher
),
watermill.NewStdLogger(false, false),
)
if err != nil {
panic(err)
}
publishMessages(publisher)
}
func publishMessages(publisher message.Publisher) {
for {
msg := message.NewMessage(watermill.NewUUID(), []byte("Hello, world!"))
if err := publisher.Publish("example.topic", msg); err != nil {
panic(err)
}
}
}
func process(subscriber string, messages <-chan *message.Message) {
for msg := range messages {
log.Printf("[%s] received message: %s, payload: %s", subscriber, msg.UUID, string(msg.Payload))
msg.Ack()
}
}
================================================
FILE: docs/content/docs/snippets/tail-log-file/go.mod
================================================
module github.com/ThreeDotsLabs/watermill/docs/content/docs/snippets/tail-log-file
go 1.25
require (
github.com/ThreeDotsLabs/watermill v1.5.1
github.com/ThreeDotsLabs/watermill-io v1.1.2
)
require (
github.com/google/uuid v1.6.0 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/lithammer/shortuuid/v3 v3.0.7 // indirect
github.com/oklog/ulid v1.3.1 // indirect
github.com/pkg/errors v0.9.1 // indirect
)
================================================
FILE: docs/content/docs/snippets/tail-log-file/go.sum
================================================
github.com/ThreeDotsLabs/watermill v1.5.1 h1:t5xMivyf9tpmU3iozPqyrCZXHvoV1XQDfihas4sV0fY=
github.com/ThreeDotsLabs/watermill v1.5.1/go.mod h1:Uop10dA3VeJWsSvis9qO3vbVY892LARrKAdki6WtXS4=
github.com/ThreeDotsLabs/watermill-io v1.1.2 h1:t3wismmE++6HbB+fDnSdvLtSKs0yYaSXRtLmyUcRyTk=
github.com/ThreeDotsLabs/watermill-io v1.1.2/go.mod h1:DF6rhoPWBOeWRW/1wWjNfLkke8rZsB5BUzBox/L6fRI=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/google/uuid v1.2.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I=
github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo=
github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM=
github.com/lithammer/shortuuid/v3 v3.0.7 h1:trX0KTHy4Pbwo/6ia8fscyHoGA+mf1jWbPJVuvyJQQ8=
github.com/lithammer/shortuuid/v3 v3.0.7/go.mod h1:vMk8ke37EmiewwolSO1NLW8vP4ZaKlRuDIi8tWWmAts=
github.com/oklog/ulid v1.3.1 h1:EGfNDEx6MqHz8B3uNV6QAib1UR2Lm97sHi3ocA6ESJ4=
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/testify v1.11.0 h1:ib4sjIrwZKxE5u/Japgo/7SJV3PvgjGiRNAvTVGqQl8=
github.com/stretchr/testify v1.11.0/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
================================================
FILE: docs/content/docs/snippets/tail-log-file/main.go
================================================
package main
import (
"context"
"fmt"
"os"
"github.com/ThreeDotsLabs/watermill"
"github.com/ThreeDotsLabs/watermill-io/pkg/io"
"github.com/ThreeDotsLabs/watermill/message"
)
// this will `tail -f` a log file and publish an alert if a line fulfils some criterion
func main() {
// if an alert is raised, the offending line will be published on this
// this would be set to an actual publisher
var alertPublisher message.Publisher
if len(os.Args) < 2 {
panic(
fmt.Errorf("usage: %s /path/to/file.log", os.Args[0]),
)
}
logFile, err := os.OpenFile(os.Args[1], os.O_RDONLY, 0444)
if err != nil {
panic(err)
}
sub, err := io.NewSubscriber(logFile, io.SubscriberConfig{
UnmarshalFunc: io.PayloadUnmarshalFunc,
}, watermill.NewStdLogger(true, false))
if err != nil {
panic(err)
}
// for io.Subscriber, topic does not matter
lines, err := sub.Subscribe(context.Background(), "")
if err != nil {
panic(err)
}
for line := range lines {
if criterion(string(line.Payload)) {
_ = alertPublisher.Publish("alerts", line)
}
}
}
func criterion(line string) bool {
// decide whether an action needs to be taken
return false
}
================================================
FILE: docs/content/docs/troubleshooting.md
================================================
+++
title = "Troubleshooting"
description = "When something goes wrong"
weight = -90
draft = false
bref = "When something goes wrong"
+++
## Logging
In most cases, you will find the answer to your problem in the logs.
Watermill offers a significant amount of logs on different severity levels.
If you are using `StdLoggerAdapter`, just change `debug`, and `trace` options to true:
```bash
logger := watermill.NewStdLogger(true, true)
````
## Debugging Pub/Sub tests
### Running a single test
```bash
make up
go test -v ./... -run TestPublishSubscribe/TestContinueAfterSubscribeClose
```
### grep is your friend
Each executed test case has a unique UUID.
It's used in the topic's name.
Thanks to that, you can easily grep the output of the test.
It gives you detailed information about the test execution.
```bash
> go test -v ./... > test.out
> less test.out
// ...
--- PASS: TestPublishSubscribe (0.00s)
--- PASS: TestPublishSubscribe/TestPublishSubscribe (2.38s)
--- PASS: TestPublishSubscribe/TestPublishSubscribe/81eeb56c-3336-4eb9-a0ac-13abda6f38ff (2.38s)
```
```bash
cat test.out | grep 81eeb56c-3336-4eb9-a0ac-13abda6f38ff | less
[watermill] 2020/08/18 14:51:46.283366 subscriber.go:300: level=TRACE msg="Msg acked" message_uuid=5c920330-5075-4870-8d86-9013771eee78 provider=google_cloud_pubsub subscription_name=topic_81eeb56c-3336-4eb9-a0ac-13abda6f38ff topic=topic_81eeb56c-3336-4eb9-a0ac-13abda6f38ff
[watermill] 2020/08/18 14:51:46.283405 subscriber.go:300: level=TRACE msg="Msg acked" message_uuid=46e04a08-994e-4c04-afff-7fd42fd67f95 provider=google_cloud_pubsub subscription_name=topic_81eeb56c-3336-4eb9-a0ac-13abda6f38ff topic=topic_81eeb56c-3336-4eb9-a0ac-13abda6f38ff
2020/08/18 14:51:46 all messages (100/100) received in bulk read after 110.04155ms of 45s (test ID: 81eeb56c-3336-4eb9-a0ac-13abda6f38ff)
[watermill] 2020/08/18 14:51:46.284569 subscriber.go:186: level=DEBUG msg="Closing message consumer" provider=google_cloud_pubsub subscription_name=topic_81eeb56c-3336-4eb9-a0ac-13abda6f38ff topic=topic_81eeb56c-3336-4eb9-a0ac-13abda6f38ff
[watermill] 2020/08/18 14:51:46.284828 subscriber.go:300: level=TRACE msg="Msg acked" message_uuid=2f409208-d4d2-46f6-b6b9-afb1aea0e59f provider=google_cloud_pubsub subscription_name=topic_81eeb56c-3336-4eb9-a0ac-13abda6f38ff topic=topic_81eeb56c-3336-4eb9-a0ac-13abda6f38ff
--- PASS: TestPublishSubscribe/TestPublishSubscribe/81eeb56c-3336-4eb9-a0ac-13abda6f38ff (2.38s)
```
## I have a deadlock
When running locally, you can send a `SIGQUIT` to the running process:
- `CTRL + \` on Linux
- `kill -s SIGQUIT [pid]` on other UNIX systems
This will kill the process and print all goroutines along with lines on which they have stopped.
```bash
SIGQUIT: quit
PC=0x45e7c3 m=0 sigcode=128
goroutine 1 [runnable]:
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).sendMessage(0xc000024100, 0x7c5250, 0xd, 0xc000872d70, 0x0, 0x0)
/home/example/go/src/github.com/ThreeDotsLabs/watermill/pubsub/gochannel/pubsub.go:83 +0x36a
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Publish(0xc000024100, 0x7c5250, 0xd, 0xc000098530, 0x1, 0x1, 0x0, 0x0)
/home/example/go/src/github.com/ThreeDotsLabs/watermill/pubsub/gochannel/pubsub.go:53 +0x6d
main.publishMessages(0x7fdf7a317000, 0xc000024100)
/home/example/go/src/github.com/ThreeDotsLabs/watermill/docs/src-link/_examples/pubsubs/go-channel/main.go:43 +0x1ec
main.main()
/home/example/go/src/github.com/ThreeDotsLabs/watermill/docs/src-link/_examples/pubsubs/go-channel/main.go:36 +0x20a
// ...
```
When running in production and you don't want to kill the entire process, a better idea is to use [pprof](https://golang.org/pkg/net/http/pprof/).
You can visit [http://localhost:6060/debug/pprof/goroutine?debug=1](http://localhost:6060/debug/pprof/goroutine?debug=1)
on your local machine to see all goroutines status.
```bash
goroutine profile: total 5
1 @ 0x41024c 0x6a8311 0x6a9bcb 0x6a948d 0x7028bc 0x70260a 0x42f187 0x45c971
# 0x6a8310 github.com/ThreeDotsLabs/watermill.LogFields.Add+0xd0 /home/example/go/src/github.com/ThreeDotsLabs/watermill/log.go:15
# 0x6a9bca github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).sendMessage+0x6fa /home/example/go/src/github.com/ThreeDotsLabs/watermill/pubsub/gochannel/pubsub.go:75
# 0x6a948c github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Publish+0x6c /home/example/go/src/github.com/ThreeDotsLabs/watermill/pubsub/gochannel/pubsub.go:53
# 0x7028bb main.publishMessages+0x1eb /home/example/go/src/github.com/ThreeDotsLabs/watermill/docs/src-link/_examples/pubsubs/go-channel/main.go:43
# 0x702609 main.main+0x209 /home/example/go/src/github.com/ThreeDotsLabs/watermill/docs/src-link/_examples/pubsubs/go-channel/main.go:36
# 0x42f186 runtime.main+0x206 /usr/lib/go/src/runtime/proc.go:201
// ...
```
================================================
FILE: docs/content/learn/_index.md
================================================
---
title: "Learn Watermill"
description: "Resources to help you learn and master Watermill for building event-driven applications in Go."
date: 2024-09-15T10:00:00+00:00
lastmod: 2024-09-15T10:00:00+00:00
draft: false
layout: learn
hideBanner: true
seo:
title: "Learn Watermill - Go Event-Driven Applications"
description: "Complete learning resources for Watermill - guides, examples, and community resources for building event-driven applications in Go."
canonical: ""
noindex: false
learning_options:
- title: "Watermill Quickstart"
subtitle: "Code along"
description: "Learn how to build an event-driven application in Go, coding in your own IDE."
link: "/learn/quickstart/"
new: true
icon: ''
- title: "Getting Started Guide"
subtitle: "Read"
description: "Start with a guide that covers core concepts of Watermill in a few minutes."
link: "/learn/getting-started/"
icon: ''
icon: ''
- title: "Examples"
subtitle: "Try out"
description: "Real-world examples and patterns for CQRS, the outbox pattern, SSE, and other use cases."
link: "https://github.com/ThreeDotsLabs/watermill/tree/master/_examples"
icon: ''
deeper_options:
- title: "Go Event-Driven"
description: "An in-depth online training on Event-Driven Architecture."
link: "https://threedots.tech/event-driven/?utm_source=watermill-learn"
icon: ''
- title: "Discord Community"
description: "Join our Discord community to get help and discuss Watermill and related topics."
link: "https://discord.gg/QV6VFg4YQE"
external: true
icon: ''
- title: "GitHub Repository"
description: "Read the source code, contribute, report issues, and stay up to date."
link: "https://github.com/ThreeDotsLabs/watermill"
external: true
icon: ''
---
================================================
FILE: docs/content/learn/getting-started.md
================================================
+++
title = "Getting started"
description = "Watermill up and running"
draft = false
bref = "Watermill up and running"
weight = 10
+++
## What is Watermill?
Watermill is a Go library for working with messages the easy way.
You can use it to build message-driven and event-driven applications with Pub/Subs like Kafka, RabbitMQ, PostgreSQL, and many more.
Watermill comes with batteries included. It gives you tools used by every message-driven application.
## Why use Watermill?
When you run an HTTP server, you don't deal directly with TCP sockets, parsing HTTP requests, or managing connections.
Instead, you use a high-level library like `net/http` that handles all that complexity for you.
**It's what Watermill aims to be for messages**.
It provides all you need to build an application based on events or other asynchronous patterns.
There are many different message queues, each with different features, client libraries, and APIs.
Watermill hides all that complexity behind an API that is easy to use and understand.
**Watermill is NOT a framework**.
It's a lightweight library that's easy to plug in or remove from your project.
## Install
```bash
go get -u github.com/ThreeDotsLabs/watermill
```
{{< callout context="note" title="Learn in practice" icon="outline/info-circle" >}}
Docs too boring? Prefer learning by doing?
[**Try the free hands-on training**]({{< ref "/learn/quickstart/" >}}) where you'll solve exercises to learn how to use Watermill in your projects.
It'll guide you through the basics and a few advanced concepts like message ordering and the Outbox pattern.
{{< /callout >}}
## One-Minute Background
The idea behind event-driven applications is always the same: one part publishes messages, and another part subscribes to them.
Watermill supports this behavior for multiple [publishers and subscribers]({{< ref "/pubsubs" >}}).
### Three APIs
Watermill comes with three APIs for working with messages.
They build on top of each other, each step providing a higher-level API.
In this guide, we're going to start from the bottom and move up.
It's good to know the fundamentals, even if you're going to use the high-level APIs.
Check our new, interactive quickstart that you can finish in your IDE. Zero Docker setup, zero coding in browser.
{{< /callout >}} Watermill Quickstart is a hands-on training that will teach you how to use Watermill, a Go library for building event-driven applications, in your project. Basic Go knowledge is all you need to get started. You'll learn the basics of Watermill and a few more advanced concepts: * The 3 core APIs: Publisher/Subscriber, Router, and CQRS. * Working with Kafka. * Topic topology, middleware, consumer groups. * Message ordering. * The Outbox pattern (with PostgreSQL). * Switching the Pub/Sub (Redis).{{ .description }}
{{ end }}{{ .description }}
{{ end }}{{ .Params.description }}
Work with Go structs.
Decouple your services with asynchronous processing.
Focus on the business logic.
Watermill handles routing, serialization and low-level details.
Use the familiar concepts, similar to an HTTP Router with support for middleware.
Work with Kafka, RabbitMQ, PostgreSQL, Redis, and more Pub/Subs with the same API. Switch providers without changing your application code.
Use any architecture you want. No vendor lock-in.
