[ { "path": ".github/workflows/codeql-analysis.yml", "content": "# For most projects, this workflow file will not need changing; you simply need\n# to commit it to your repository.\n#\n# You may wish to alter this file to override the set of languages analyzed,\n# or to provide custom queries or build logic.\n#\n# ******** NOTE ********\n# We have attempted to detect the languages in your repository. Please check\n# the `language` matrix defined below to confirm you have the correct set of\n# supported CodeQL languages.\n#\nname: \"CodeQL\"\n\non:\n push:\n branches: [ main ]\n pull_request:\n # The branches below must be a subset of the branches above\n branches: [ main ]\n schedule:\n - cron: '18 19 * * 6'\n\njobs:\n analyze:\n name: Analyze\n runs-on: ubuntu-latest\n\n strategy:\n fail-fast: false\n matrix:\n language: [ 'go' ]\n # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python' ]\n # Learn more:\n # https://docs.github.com/en/free-pro-team@latest/github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-code-scanning#changing-the-languages-that-are-analyzed\n\n steps:\n - name: Checkout repository\n uses: actions/checkout@v2\n\n # Initializes the CodeQL tools for scanning.\n - name: Initialize CodeQL\n uses: github/codeql-action/init@v1\n with:\n languages: ${{ matrix.language }}\n # If you wish to specify custom queries, you can do so here or in a config file.\n # By default, queries listed here will override any specified in a config file.\n # Prefix the list here with \"+\" to use these queries and those in the config file.\n # queries: ./path/to/local/query, your-org/your-repo/queries@main\n\n # Autobuild attempts to build any compiled languages (C/C++, C#, or Java).\n # If this step fails, then you should remove it and run the build manually (see below)\n - name: Autobuild\n uses: github/codeql-action/autobuild@v1\n\n # ℹ️ Command-line programs to run using the OS shell.\n # 📚 https://git.io/JvXDl\n\n # ✏️ If the Autobuild fails above, remove it and uncomment the following three lines\n # and modify them (or add more) to build your code if your project\n # uses a compiled language\n\n #- run: |\n # make bootstrap\n # make release\n\n - name: Perform CodeQL Analysis\n uses: github/codeql-action/analyze@v1\n" }, { "path": ".github/workflows/go.yml", "content": "name: Go\n\non:\n push:\n branches: [ main ]\n pull_request:\n branches: [ main ]\n\njobs:\n build:\n name: Build\n runs-on: ubuntu-latest\n steps:\n\n - name: Set up Go 1.x\n uses: actions/setup-go@v2\n with:\n go-version: ^1.18\n id: go\n\n - name: Check out code into the Go module directory\n uses: actions/checkout@v2\n\n - name: Get dependencies\n run: |\n go get -v -t -d ./...\n\n - name: Lint\n run: |\n go vet -stdmethods=false $(go list ./...)\n go install mvdan.cc/gofumpt@latest\n test -z \"$(gofumpt -s -l -extra .)\" || echo \"Please run 'gofumpt -l -w -extra .'\"\n\n - name: Test\n run: go test -race -coverprofile=coverage.txt -covermode=atomic ./...\n\n - name: Codecov\n uses: codecov/codecov-action@v2\n" }, { "path": ".github/workflows/reviewdog.yml", "content": "name: reviewdog\non: [pull_request]\njobs:\n staticcheck:\n name: runner / staticcheck\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v2\n - uses: actions/setup-go@v3\n with:\n go-version: \"1.18\"\n - uses: reviewdog/action-staticcheck@v1\n with:\n github_token: ${{ secrets.github_token }}\n # Change reviewdog reporter if you need [github-pr-check,github-check,github-pr-review].\n reporter: github-pr-review\n # Report all results.\n filter_mode: nofilter\n # Exit with 1 when it find at least one finding.\n fail_on_error: true\n # Set staticcheck flags\n staticcheck_flags: -checks=inherit,-SA1019,-SA1029,-SA5008\n" }, { "path": ".gitignore", "content": "# Binaries for programs and plugins\n*.exe\n*.exe~\n*.dll\n*.so\n*.dylib\n\n# dev files\n.idea\n\n# For test\n**/testdata\n*.test\n\n# Output of the go coverage tool, specifically when used with LiteIDE\n*.out\n\n# Dependency directories (remove the comment below to include it)\n# vendor/\n" }, { "path": "LICENSE", "content": "MIT License\n\nCopyright (c) 2021 Kevin Wan\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n" }, { "path": "examples/sum.go", "content": "package main\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\n\t\"github.com/kevwan/mapreduce/v2\"\n)\n\nfunc main() {\n\tval, err := mapreduce.MapReduce(func(source chan<- int) {\n\t\tfor i := 0; i < 10; i++ {\n\t\t\tsource <- i\n\t\t}\n\t}, func(item int, writer mapreduce.Writer[int], cancel func(error)) {\n\t\twriter.Write(item * item)\n\t}, func(pipe <-chan int, writer mapreduce.Writer[int], cancel func(error)) {\n\t\tvar sum int\n\t\tfor i := range pipe {\n\t\t\tsum += i\n\t\t}\n\t\twriter.Write(sum)\n\t})\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tfmt.Println(\"result:\", val)\n}\n" }, { "path": "go.mod", "content": "module github.com/kevwan/mapreduce/v2\n\ngo 1.18\n\nrequire (\n\tgithub.com/stretchr/testify v1.7.0\n\tgo.uber.org/goleak v1.1.12\n)\n\nrequire (\n\tgithub.com/davecgh/go-spew v1.1.1 // indirect\n\tgithub.com/kr/pretty v0.2.1 // indirect\n\tgithub.com/pmezard/go-difflib v1.0.0 // indirect\n\tgopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b // indirect\n)\n" }, { "path": "go.sum", "content": "github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=\ngithub.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=\ngithub.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=\ngithub.com/kevwan/mapreduce/v2 v2.0.1 h1:i+elkbdsdj1IK85dY+hVtYNdWVMx+O/q0TRr55r+mRI=\ngithub.com/kevwan/mapreduce/v2 v2.0.1/go.mod h1:CRVNCu3oR6NcBIXGxzLjhYjMtNXlEwI5jASt5JZaBLk=\ngithub.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=\ngithub.com/kr/pretty v0.2.1 h1:Fmg33tUaq4/8ym9TJN1x7sLJnHVwhP33CNkpYV/7rwI=\ngithub.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=\ngithub.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=\ngithub.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=\ngithub.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=\ngithub.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=\ngithub.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=\ngithub.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=\ngithub.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=\ngithub.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=\ngithub.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=\ngo.uber.org/goleak v1.1.12 h1:gZAh5/EyT/HQwlpkCy6wTpqfH9H8Lz8zbm3dZh+OyzA=\ngo.uber.org/goleak v1.1.12/go.mod h1:cwTWslyiVhfpKIDGSZEM2HlOvcqm+tG4zioyIeLoqMQ=\ngolang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=\ngolang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=\ngolang.org/x/lint v0.0.0-20190930215403-16217165b5de h1:5hukYrvBGR8/eNkX5mdUezrA6JiaEZDtJb9Ei+1LlBs=\ngolang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=\ngolang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=\ngolang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=\ngolang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=\ngolang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=\ngolang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=\ngolang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=\ngolang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=\ngolang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=\ngolang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=\ngolang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=\ngolang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=\ngolang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=\ngolang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=\ngolang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=\ngolang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=\ngolang.org/x/tools v0.1.5 h1:ouewzE6p+/VEB31YYnTbEJdi8pFqKp4P4n85vwo3DHA=\ngolang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=\ngolang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=\ngolang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=\ngolang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=\ngopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=\ngopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=\ngopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=\ngopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=\ngopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b h1:h8qDotaEPuJATrMmW04NCwg7v22aHH28wwpauUhK9Oo=\ngopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=\n" }, { "path": "mapreduce.go", "content": "package mapreduce\n\nimport (\n\t\"context\"\n\t\"errors\"\n\t\"sync\"\n\t\"sync/atomic\"\n)\n\nconst (\n\tdefaultWorkers = 16\n\tminWorkers = 1\n)\n\nvar (\n\t// ErrCancelWithNil is an error that mapreduce was cancelled with nil.\n\tErrCancelWithNil = errors.New(\"mapreduce cancelled with nil\")\n\t// ErrReduceNoOutput is an error that reduce did not output a value.\n\tErrReduceNoOutput = errors.New(\"reduce not writing value\")\n)\n\ntype (\n\t// ForEachFunc is used to do element processing, but no output.\n\tForEachFunc[T any] func(item T)\n\t// GenerateFunc is used to let callers send elements into source.\n\tGenerateFunc[T any] func(source chan<- T)\n\t// MapFunc is used to do element processing and write the output to writer.\n\tMapFunc[T, U any] func(item T, writer Writer[U])\n\t// MapperFunc is used to do element processing and write the output to writer,\n\t// use cancel func to cancel the processing.\n\tMapperFunc[T, U any] func(item T, writer Writer[U], cancel func(error))\n\t// ReducerFunc is used to reduce all the mapping output and write to writer,\n\t// use cancel func to cancel the processing.\n\tReducerFunc[U, V any] func(pipe <-chan U, writer Writer[V], cancel func(error))\n\t// VoidReducerFunc is used to reduce all the mapping output, but no output.\n\t// Use cancel func to cancel the processing.\n\tVoidReducerFunc[U any] func(pipe <-chan U, cancel func(error))\n\t// Option defines the method to customize the mapreduce.\n\tOption func(opts *mapReduceOptions)\n\n\tmapperContext[T, U any] struct {\n\t\tctx context.Context\n\t\tmapper MapFunc[T, U]\n\t\tsource <-chan T\n\t\tpanicChan *onceChan\n\t\tcollector chan<- U\n\t\tdoneChan <-chan struct{}\n\t\tworkers int\n\t}\n\n\tmapReduceOptions struct {\n\t\tctx context.Context\n\t\tworkers int\n\t}\n\n\t// Writer interface wraps Write method.\n\tWriter[T any] interface {\n\t\tWrite(v T)\n\t}\n)\n\n// Finish runs fns parallelly, cancelled on any error.\nfunc Finish(fns ...func() error) error {\n\tif len(fns) == 0 {\n\t\treturn nil\n\t}\n\n\treturn MapReduceVoid(func(source chan<- func() error) {\n\t\tfor _, fn := range fns {\n\t\t\tsource <- fn\n\t\t}\n\t}, func(fn func() error, writer Writer[any], cancel func(error)) {\n\t\tif err := fn(); err != nil {\n\t\t\tcancel(err)\n\t\t}\n\t}, func(pipe <-chan any, cancel func(error)) {\n\t}, WithWorkers(len(fns)))\n}\n\n// FinishVoid runs fns parallelly.\nfunc FinishVoid(fns ...func()) {\n\tif len(fns) == 0 {\n\t\treturn\n\t}\n\n\tForEach(func(source chan<- func()) {\n\t\tfor _, fn := range fns {\n\t\t\tsource <- fn\n\t\t}\n\t}, func(fn func()) {\n\t\tfn()\n\t}, WithWorkers(len(fns)))\n}\n\n// ForEach maps all elements from given generate but no output.\nfunc ForEach[T any](generate GenerateFunc[T], mapper ForEachFunc[T], opts ...Option) {\n\toptions := buildOptions(opts...)\n\tpanicChan := &onceChan{channel: make(chan any)}\n\tsource := buildSource(generate, panicChan)\n\tcollector := make(chan any)\n\tdone := make(chan struct{})\n\n\tgo executeMappers(mapperContext[T, any]{\n\t\tctx: options.ctx,\n\t\tmapper: func(item T, _ Writer[any]) {\n\t\t\tmapper(item)\n\t\t},\n\t\tsource: source,\n\t\tpanicChan: panicChan,\n\t\tcollector: collector,\n\t\tdoneChan: done,\n\t\tworkers: options.workers,\n\t})\n\n\tfor {\n\t\tselect {\n\t\tcase v := <-panicChan.channel:\n\t\t\tpanic(v)\n\t\tcase _, ok := <-collector:\n\t\t\tif !ok {\n\t\t\t\treturn\n\t\t\t}\n\t\t}\n\t}\n}\n\n// MapReduce maps all elements generated from given generate func,\n// and reduces the output elements with given reducer.\nfunc MapReduce[T, U, V any](generate GenerateFunc[T], mapper MapperFunc[T, U], reducer ReducerFunc[U, V],\n\topts ...Option) (V, error) {\n\tpanicChan := &onceChan{channel: make(chan any)}\n\tsource := buildSource(generate, panicChan)\n\treturn mapReduceWithPanicChan(source, panicChan, mapper, reducer, opts...)\n}\n\n// MapReduceChan maps all elements from source, and reduce the output elements with given reducer.\nfunc MapReduceChan[T, U, V any](source <-chan T, mapper MapperFunc[T, U], reducer ReducerFunc[U, V],\n\topts ...Option) (V, error) {\n\tpanicChan := &onceChan{channel: make(chan any)}\n\treturn mapReduceWithPanicChan(source, panicChan, mapper, reducer, opts...)\n}\n\n// mapReduceWithPanicChan maps all elements from source, and reduce the output elements with given reducer.\nfunc mapReduceWithPanicChan[T, U, V any](source <-chan T, panicChan *onceChan, mapper MapperFunc[T, U],\n\treducer ReducerFunc[U, V], opts ...Option) (val V, err error) {\n\toptions := buildOptions(opts...)\n\t// output is used to write the final result\n\toutput := make(chan V)\n\tdefer func() {\n\t\t// reducer can only write once, if more, panic\n\t\tfor range output {\n\t\t\tpanic(\"more than one element written in reducer\")\n\t\t}\n\t}()\n\n\t// collector is used to collect data from mapper, and consume in reducer\n\tcollector := make(chan U, options.workers)\n\t// if done is closed, all mappers and reducer should stop processing\n\tdone := make(chan struct{})\n\twriter := newGuardedWriter(options.ctx, output, done)\n\tvar closeOnce sync.Once\n\t// use atomic.Value to avoid data race\n\tvar retErr atomic.Value\n\tfinish := func() {\n\t\tcloseOnce.Do(func() {\n\t\t\tclose(done)\n\t\t\tclose(output)\n\t\t})\n\t}\n\tcancel := once(func(err error) {\n\t\tif err != nil {\n\t\t\tretErr.Store(err)\n\t\t} else {\n\t\t\tretErr.Store(ErrCancelWithNil)\n\t\t}\n\n\t\tdrain(source)\n\t\tfinish()\n\t})\n\n\tgo func() {\n\t\tdefer func() {\n\t\t\tdrain(collector)\n\t\t\tif r := recover(); r != nil {\n\t\t\t\tpanicChan.write(r)\n\t\t\t}\n\t\t\tfinish()\n\t\t}()\n\n\t\treducer(collector, writer, cancel)\n\t}()\n\n\tgo executeMappers(mapperContext[T, U]{\n\t\tctx: options.ctx,\n\t\tmapper: func(item T, w Writer[U]) {\n\t\t\tmapper(item, w, cancel)\n\t\t},\n\t\tsource: source,\n\t\tpanicChan: panicChan,\n\t\tcollector: collector,\n\t\tdoneChan: done,\n\t\tworkers: options.workers,\n\t})\n\n\tselect {\n\tcase <-options.ctx.Done():\n\t\tcancel(context.DeadlineExceeded)\n\t\terr = context.DeadlineExceeded\n\tcase v := <-panicChan.channel:\n\t\t// drain output here, otherwise for loop panic in defer\n\t\tdrain(output)\n\t\tpanic(v)\n\tcase v, ok := <-output:\n\t\tif e := retErr.Load(); e != nil {\n\t\t\terr = e.(error)\n\t\t} else if ok {\n\t\t\tval = v\n\t\t} else {\n\t\t\terr = ErrReduceNoOutput\n\t\t}\n\t}\n\n\treturn\n}\n\n// MapReduceVoid maps all elements generated from given generate,\n// and reduce the output elements with given reducer.\nfunc MapReduceVoid[T, U any](generate GenerateFunc[T], mapper MapperFunc[T, U],\n\treducer VoidReducerFunc[U], opts ...Option) error {\n\t_, err := MapReduce(generate, mapper, func(input <-chan U, writer Writer[any], cancel func(error)) {\n\t\treducer(input, cancel)\n\t}, opts...)\n\tif errors.Is(err, ErrReduceNoOutput) {\n\t\treturn nil\n\t}\n\n\treturn err\n}\n\n// WithContext customizes a mapreduce processing accepts a given ctx.\nfunc WithContext(ctx context.Context) Option {\n\treturn func(opts *mapReduceOptions) {\n\t\topts.ctx = ctx\n\t}\n}\n\n// WithWorkers customizes a mapreduce processing with given workers.\nfunc WithWorkers(workers int) Option {\n\treturn func(opts *mapReduceOptions) {\n\t\tif workers < minWorkers {\n\t\t\topts.workers = minWorkers\n\t\t} else {\n\t\t\topts.workers = workers\n\t\t}\n\t}\n}\n\nfunc buildOptions(opts ...Option) *mapReduceOptions {\n\toptions := newOptions()\n\tfor _, opt := range opts {\n\t\topt(options)\n\t}\n\n\treturn options\n}\n\nfunc buildSource[T any](generate GenerateFunc[T], panicChan *onceChan) chan T {\n\tsource := make(chan T)\n\tgo func() {\n\t\tdefer func() {\n\t\t\tif r := recover(); r != nil {\n\t\t\t\tpanicChan.write(r)\n\t\t\t}\n\t\t\tclose(source)\n\t\t}()\n\n\t\tgenerate(source)\n\t}()\n\n\treturn source\n}\n\n// drain drains the channel.\nfunc drain[T any](channel <-chan T) {\n\t// drain the channel\n\tfor range channel {\n\t}\n}\n\nfunc executeMappers[T, U any](mCtx mapperContext[T, U]) {\n\tvar wg sync.WaitGroup\n\tdefer func() {\n\t\twg.Wait()\n\t\tclose(mCtx.collector)\n\t\tdrain(mCtx.source)\n\t}()\n\n\tvar failed int32\n\tpool := make(chan struct{}, mCtx.workers)\n\twriter := newGuardedWriter(mCtx.ctx, mCtx.collector, mCtx.doneChan)\n\tfor atomic.LoadInt32(&failed) == 0 {\n\t\tselect {\n\t\tcase <-mCtx.ctx.Done():\n\t\t\treturn\n\t\tcase <-mCtx.doneChan:\n\t\t\treturn\n\t\tcase pool <- struct{}{}:\n\t\t\titem, ok := <-mCtx.source\n\t\t\tif !ok {\n\t\t\t\t<-pool\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\twg.Add(1)\n\t\t\tgo func() {\n\t\t\t\tdefer func() {\n\t\t\t\t\tif r := recover(); r != nil {\n\t\t\t\t\t\tatomic.AddInt32(&failed, 1)\n\t\t\t\t\t\tmCtx.panicChan.write(r)\n\t\t\t\t\t}\n\t\t\t\t\twg.Done()\n\t\t\t\t\t<-pool\n\t\t\t\t}()\n\n\t\t\t\tmCtx.mapper(item, writer)\n\t\t\t}()\n\t\t}\n\t}\n}\n\nfunc newOptions() *mapReduceOptions {\n\treturn &mapReduceOptions{\n\t\tctx: context.Background(),\n\t\tworkers: defaultWorkers,\n\t}\n}\n\nfunc once(fn func(error)) func(error) {\n\tonce := new(sync.Once)\n\treturn func(err error) {\n\t\tonce.Do(func() {\n\t\t\tfn(err)\n\t\t})\n\t}\n}\n\ntype guardedWriter[T any] struct {\n\tctx context.Context\n\tchannel chan<- T\n\tdone <-chan struct{}\n}\n\nfunc newGuardedWriter[T any](ctx context.Context, channel chan<- T, done <-chan struct{}) guardedWriter[T] {\n\treturn guardedWriter[T]{\n\t\tctx: ctx,\n\t\tchannel: channel,\n\t\tdone: done,\n\t}\n}\n\nfunc (gw guardedWriter[T]) Write(v T) {\n\tselect {\n\tcase <-gw.ctx.Done():\n\t\treturn\n\tcase <-gw.done:\n\t\treturn\n\tdefault:\n\t\tgw.channel <- v\n\t}\n}\n\ntype onceChan struct {\n\tchannel chan any\n\twrote int32\n}\n\nfunc (oc *onceChan) write(val any) {\n\tif atomic.CompareAndSwapInt32(&oc.wrote, 0, 1) {\n\t\toc.channel <- val\n\t}\n}\n" }, { "path": "mapreduce_fuzz_test.go", "content": "//go:build go1.18\n// +build go1.18\n\npackage mapreduce\n\nimport (\n\t\"fmt\"\n\t\"math/rand\"\n\t\"runtime\"\n\t\"strings\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/stretchr/testify/assert\"\n\t\"go.uber.org/goleak\"\n)\n\nfunc FuzzMapReduce(f *testing.F) {\n\trand.Seed(time.Now().UnixNano())\n\n\tf.Add(int64(10), runtime.NumCPU())\n\tf.Fuzz(func(t *testing.T, n int64, workers int) {\n\t\tn = n%5000 + 5000\n\t\tgenPanic := rand.Intn(100) == 0\n\t\tmapperPanic := rand.Intn(100) == 0\n\t\treducerPanic := rand.Intn(100) == 0\n\t\tgenIdx := rand.Int63n(n)\n\t\tmapperIdx := rand.Int63n(n)\n\t\treducerIdx := rand.Int63n(n)\n\t\tsquareSum := (n - 1) * n * (2*n - 1) / 6\n\n\t\tfn := func() (int64, error) {\n\t\t\tdefer goleak.VerifyNone(t, goleak.IgnoreCurrent())\n\n\t\t\treturn MapReduce(func(source chan<- int64) {\n\t\t\t\tfor i := int64(0); i < n; i++ {\n\t\t\t\t\tsource <- i\n\t\t\t\t\tif genPanic && i == genIdx {\n\t\t\t\t\t\tpanic(\"foo\")\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}, func(v int64, writer Writer[int64], cancel func(error)) {\n\t\t\t\tif mapperPanic && v == mapperIdx {\n\t\t\t\t\tpanic(\"bar\")\n\t\t\t\t}\n\t\t\t\twriter.Write(v * v)\n\t\t\t}, func(pipe <-chan int64, writer Writer[int64], cancel func(error)) {\n\t\t\t\tvar idx int64\n\t\t\t\tvar total int64\n\t\t\t\tfor v := range pipe {\n\t\t\t\t\tif reducerPanic && idx == reducerIdx {\n\t\t\t\t\t\tpanic(\"baz\")\n\t\t\t\t\t}\n\t\t\t\t\ttotal += v\n\t\t\t\t\tidx++\n\t\t\t\t}\n\t\t\t\twriter.Write(total)\n\t\t\t}, WithWorkers(workers%50+runtime.NumCPU()))\n\t\t}\n\n\t\tif genPanic || mapperPanic || reducerPanic {\n\t\t\tvar buf strings.Builder\n\t\t\tbuf.WriteString(fmt.Sprintf(\"n: %d\", n))\n\t\t\tbuf.WriteString(fmt.Sprintf(\", genPanic: %t\", genPanic))\n\t\t\tbuf.WriteString(fmt.Sprintf(\", mapperPanic: %t\", mapperPanic))\n\t\t\tbuf.WriteString(fmt.Sprintf(\", reducerPanic: %t\", reducerPanic))\n\t\t\tbuf.WriteString(fmt.Sprintf(\", genIdx: %d\", genIdx))\n\t\t\tbuf.WriteString(fmt.Sprintf(\", mapperIdx: %d\", mapperIdx))\n\t\t\tbuf.WriteString(fmt.Sprintf(\", reducerIdx: %d\", reducerIdx))\n\t\t\tassert.Panicsf(t, func() { fn() }, buf.String())\n\t\t} else {\n\t\t\tval, err := fn()\n\t\t\tassert.Nil(t, err)\n\t\t\tassert.Equal(t, squareSum, val)\n\t\t}\n\t})\n}\n" }, { "path": "mapreduce_test.go", "content": "package mapreduce\n\nimport (\n\t\"context\"\n\t\"errors\"\n\t\"io/ioutil\"\n\t\"log\"\n\t\"runtime\"\n\t\"sync/atomic\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/stretchr/testify/assert\"\n\t\"go.uber.org/goleak\"\n)\n\nvar errDummy = errors.New(\"dummy\")\n\nfunc init() {\n\tlog.SetOutput(ioutil.Discard)\n}\n\nfunc TestFinish(t *testing.T) {\n\tdefer goleak.VerifyNone(t)\n\n\tvar total uint32\n\terr := Finish(func() error {\n\t\tatomic.AddUint32(&total, 2)\n\t\treturn nil\n\t}, func() error {\n\t\tatomic.AddUint32(&total, 3)\n\t\treturn nil\n\t}, func() error {\n\t\tatomic.AddUint32(&total, 5)\n\t\treturn nil\n\t})\n\n\tassert.Equal(t, uint32(10), atomic.LoadUint32(&total))\n\tassert.Nil(t, err)\n}\n\nfunc TestFinishNone(t *testing.T) {\n\tdefer goleak.VerifyNone(t)\n\n\tassert.Nil(t, Finish())\n}\n\nfunc TestFinishVoidNone(t *testing.T) {\n\tdefer goleak.VerifyNone(t)\n\n\tFinishVoid()\n}\n\nfunc TestFinishErr(t *testing.T) {\n\tdefer goleak.VerifyNone(t)\n\n\tvar total uint32\n\terr := Finish(func() error {\n\t\tatomic.AddUint32(&total, 2)\n\t\treturn nil\n\t}, func() error {\n\t\tatomic.AddUint32(&total, 3)\n\t\treturn errDummy\n\t}, func() error {\n\t\tatomic.AddUint32(&total, 5)\n\t\treturn nil\n\t})\n\n\tassert.Equal(t, errDummy, err)\n}\n\nfunc TestFinishVoid(t *testing.T) {\n\tdefer goleak.VerifyNone(t)\n\n\tvar total uint32\n\tFinishVoid(func() {\n\t\tatomic.AddUint32(&total, 2)\n\t}, func() {\n\t\tatomic.AddUint32(&total, 3)\n\t}, func() {\n\t\tatomic.AddUint32(&total, 5)\n\t})\n\n\tassert.Equal(t, uint32(10), atomic.LoadUint32(&total))\n}\n\nfunc TestForEach(t *testing.T) {\n\tconst tasks = 1000\n\n\tt.Run(\"all\", func(t *testing.T) {\n\t\tdefer goleak.VerifyNone(t)\n\n\t\tvar count uint32\n\t\tForEach(func(source chan<- int) {\n\t\t\tfor i := 0; i < tasks; i++ {\n\t\t\t\tsource <- i\n\t\t\t}\n\t\t}, func(item int) {\n\t\t\tatomic.AddUint32(&count, 1)\n\t\t}, WithWorkers(-1))\n\n\t\tassert.Equal(t, tasks, int(count))\n\t})\n\n\tt.Run(\"odd\", func(t *testing.T) {\n\t\tdefer goleak.VerifyNone(t)\n\n\t\tvar count uint32\n\t\tForEach(func(source chan<- int) {\n\t\t\tfor i := 0; i < tasks; i++ {\n\t\t\t\tsource <- i\n\t\t\t}\n\t\t}, func(item int) {\n\t\t\tif item%2 == 0 {\n\t\t\t\tatomic.AddUint32(&count, 1)\n\t\t\t}\n\t\t})\n\n\t\tassert.Equal(t, tasks/2, int(count))\n\t})\n\n\tt.Run(\"all\", func(t *testing.T) {\n\t\tdefer goleak.VerifyNone(t)\n\n\t\tassert.PanicsWithValue(t, \"foo\", func() {\n\t\t\tForEach(func(source chan<- int) {\n\t\t\t\tfor i := 0; i < tasks; i++ {\n\t\t\t\t\tsource <- i\n\t\t\t\t}\n\t\t\t}, func(item int) {\n\t\t\t\tpanic(\"foo\")\n\t\t\t})\n\t\t})\n\t})\n}\n\nfunc TestGeneratePanic(t *testing.T) {\n\tdefer goleak.VerifyNone(t)\n\n\tt.Run(\"all\", func(t *testing.T) {\n\t\tassert.PanicsWithValue(t, \"foo\", func() {\n\t\t\tForEach(func(source chan<- int) {\n\t\t\t\tpanic(\"foo\")\n\t\t\t}, func(item int) {\n\t\t\t})\n\t\t})\n\t})\n}\n\nfunc TestMapperPanic(t *testing.T) {\n\tdefer goleak.VerifyNone(t)\n\n\tconst tasks = 1000\n\tvar run int32\n\tt.Run(\"all\", func(t *testing.T) {\n\t\tassert.PanicsWithValue(t, \"foo\", func() {\n\t\t\t_, _ = MapReduce(func(source chan<- int) {\n\t\t\t\tfor i := 0; i < tasks; i++ {\n\t\t\t\t\tsource <- i\n\t\t\t\t}\n\t\t\t}, func(item int, writer Writer[int], cancel func(error)) {\n\t\t\t\tatomic.AddInt32(&run, 1)\n\t\t\t\tpanic(\"foo\")\n\t\t\t}, func(pipe <-chan int, writer Writer[int], cancel func(error)) {\n\t\t\t})\n\t\t})\n\t\tassert.True(t, atomic.LoadInt32(&run) < tasks/2)\n\t})\n}\n\nfunc TestMapReduce(t *testing.T) {\n\tdefer goleak.VerifyNone(t)\n\n\ttests := []struct {\n\t\tname string\n\t\tmapper MapperFunc[int, int]\n\t\treducer ReducerFunc[int, int]\n\t\texpectErr error\n\t\texpectValue int\n\t}{\n\t\t{\n\t\t\tname: \"simple\",\n\t\t\texpectErr: nil,\n\t\t\texpectValue: 30,\n\t\t},\n\t\t{\n\t\t\tname: \"cancel with error\",\n\t\t\tmapper: func(v int, writer Writer[int], cancel func(error)) {\n\t\t\t\tif v%3 == 0 {\n\t\t\t\t\tcancel(errDummy)\n\t\t\t\t}\n\t\t\t\twriter.Write(v * v)\n\t\t\t},\n\t\t\texpectErr: errDummy,\n\t\t},\n\t\t{\n\t\t\tname: \"cancel with nil\",\n\t\t\tmapper: func(v int, writer Writer[int], cancel func(error)) {\n\t\t\t\tif v%3 == 0 {\n\t\t\t\t\tcancel(nil)\n\t\t\t\t}\n\t\t\t\twriter.Write(v * v)\n\t\t\t},\n\t\t\texpectErr: ErrCancelWithNil,\n\t\t},\n\t\t{\n\t\t\tname: \"cancel with more\",\n\t\t\treducer: func(pipe <-chan int, writer Writer[int], cancel func(error)) {\n\t\t\t\tvar result int\n\t\t\t\tfor item := range pipe {\n\t\t\t\t\tresult += item\n\t\t\t\t\tif result > 10 {\n\t\t\t\t\t\tcancel(errDummy)\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\twriter.Write(result)\n\t\t\t},\n\t\t\texpectErr: errDummy,\n\t\t},\n\t}\n\n\tt.Run(\"MapReduce\", func(t *testing.T) {\n\t\tfor _, test := range tests {\n\t\t\tt.Run(test.name, func(t *testing.T) {\n\t\t\t\tif test.mapper == nil {\n\t\t\t\t\ttest.mapper = func(v int, writer Writer[int], cancel func(error)) {\n\t\t\t\t\t\twriter.Write(v * v)\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\tif test.reducer == nil {\n\t\t\t\t\ttest.reducer = func(pipe <-chan int, writer Writer[int], cancel func(error)) {\n\t\t\t\t\t\tvar result int\n\t\t\t\t\t\tfor item := range pipe {\n\t\t\t\t\t\t\tresult += item\n\t\t\t\t\t\t}\n\t\t\t\t\t\twriter.Write(result)\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\tvalue, err := MapReduce(func(source chan<- int) {\n\t\t\t\t\tfor i := 1; i < 5; i++ {\n\t\t\t\t\t\tsource <- i\n\t\t\t\t\t}\n\t\t\t\t}, test.mapper, test.reducer, WithWorkers(runtime.NumCPU()))\n\n\t\t\t\tassert.Equal(t, test.expectErr, err)\n\t\t\t\tassert.Equal(t, test.expectValue, value)\n\t\t\t})\n\t\t}\n\t})\n\n\tt.Run(\"MapReduce\", func(t *testing.T) {\n\t\tfor _, test := range tests {\n\t\t\tt.Run(test.name, func(t *testing.T) {\n\t\t\t\tif test.mapper == nil {\n\t\t\t\t\ttest.mapper = func(v int, writer Writer[int], cancel func(error)) {\n\t\t\t\t\t\twriter.Write(v * v)\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\tif test.reducer == nil {\n\t\t\t\t\ttest.reducer = func(pipe <-chan int, writer Writer[int], cancel func(error)) {\n\t\t\t\t\t\tvar result int\n\t\t\t\t\t\tfor item := range pipe {\n\t\t\t\t\t\t\tresult += item\n\t\t\t\t\t\t}\n\t\t\t\t\t\twriter.Write(result)\n\t\t\t\t\t}\n\t\t\t\t}\n\n\t\t\t\tsource := make(chan int)\n\t\t\t\tgo func() {\n\t\t\t\t\tfor i := 1; i < 5; i++ {\n\t\t\t\t\t\tsource <- i\n\t\t\t\t\t}\n\t\t\t\t\tclose(source)\n\t\t\t\t}()\n\n\t\t\t\tvalue, err := MapReduceChan(source, test.mapper, test.reducer, WithWorkers(-1))\n\t\t\t\tassert.Equal(t, test.expectErr, err)\n\t\t\t\tassert.Equal(t, test.expectValue, value)\n\t\t\t})\n\t\t}\n\t})\n}\n\nfunc TestMapReduceWithReduerWriteMoreThanOnce(t *testing.T) {\n\tdefer goleak.VerifyNone(t)\n\n\tassert.Panics(t, func() {\n\t\tMapReduce(func(source chan<- int) {\n\t\t\tfor i := 0; i < 10; i++ {\n\t\t\t\tsource <- i\n\t\t\t}\n\t\t}, func(item int, writer Writer[int], cancel func(error)) {\n\t\t\twriter.Write(item)\n\t\t}, func(pipe <-chan int, writer Writer[string], cancel func(error)) {\n\t\t\tdrain(pipe)\n\t\t\twriter.Write(\"one\")\n\t\t\twriter.Write(\"two\")\n\t\t})\n\t})\n}\n\nfunc TestMapReduceVoid(t *testing.T) {\n\tdefer goleak.VerifyNone(t)\n\n\tvar value uint32\n\ttests := []struct {\n\t\tname string\n\t\tmapper MapperFunc[int, int]\n\t\treducer VoidReducerFunc[int]\n\t\texpectValue uint32\n\t\texpectErr error\n\t}{\n\t\t{\n\t\t\tname: \"simple\",\n\t\t\texpectValue: 30,\n\t\t\texpectErr: nil,\n\t\t},\n\t\t{\n\t\t\tname: \"cancel with error\",\n\t\t\tmapper: func(v int, writer Writer[int], cancel func(error)) {\n\t\t\t\tif v%3 == 0 {\n\t\t\t\t\tcancel(errDummy)\n\t\t\t\t}\n\t\t\t\twriter.Write(v * v)\n\t\t\t},\n\t\t\texpectErr: errDummy,\n\t\t},\n\t\t{\n\t\t\tname: \"cancel with nil\",\n\t\t\tmapper: func(v int, writer Writer[int], cancel func(error)) {\n\t\t\t\tif v%3 == 0 {\n\t\t\t\t\tcancel(nil)\n\t\t\t\t}\n\t\t\t\twriter.Write(v * v)\n\t\t\t},\n\t\t\texpectErr: ErrCancelWithNil,\n\t\t},\n\t\t{\n\t\t\tname: \"cancel with more\",\n\t\t\treducer: func(pipe <-chan int, cancel func(error)) {\n\t\t\t\tfor item := range pipe {\n\t\t\t\t\tresult := atomic.AddUint32(&value, uint32(item))\n\t\t\t\t\tif result > 10 {\n\t\t\t\t\t\tcancel(errDummy)\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\texpectErr: errDummy,\n\t\t},\n\t}\n\n\tfor _, test := range tests {\n\t\tt.Run(test.name, func(t *testing.T) {\n\t\t\tatomic.StoreUint32(&value, 0)\n\n\t\t\tif test.mapper == nil {\n\t\t\t\ttest.mapper = func(v int, writer Writer[int], cancel func(error)) {\n\t\t\t\t\twriter.Write(v * v)\n\t\t\t\t}\n\t\t\t}\n\t\t\tif test.reducer == nil {\n\t\t\t\ttest.reducer = func(pipe <-chan int, cancel func(error)) {\n\t\t\t\t\tfor item := range pipe {\n\t\t\t\t\t\tatomic.AddUint32(&value, uint32(item))\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t\terr := MapReduceVoid(func(source chan<- int) {\n\t\t\t\tfor i := 1; i < 5; i++ {\n\t\t\t\t\tsource <- i\n\t\t\t\t}\n\t\t\t}, test.mapper, test.reducer)\n\n\t\t\tassert.Equal(t, test.expectErr, err)\n\t\t\tif err == nil {\n\t\t\t\tassert.Equal(t, test.expectValue, atomic.LoadUint32(&value))\n\t\t\t}\n\t\t})\n\t}\n}\n\nfunc TestMapReduceVoidWithDelay(t *testing.T) {\n\tdefer goleak.VerifyNone(t)\n\n\tvar result []int\n\terr := MapReduceVoid(func(source chan<- int) {\n\t\tsource <- 0\n\t\tsource <- 1\n\t}, func(i int, writer Writer[int], cancel func(error)) {\n\t\tif i == 0 {\n\t\t\ttime.Sleep(time.Millisecond * 50)\n\t\t}\n\t\twriter.Write(i)\n\t}, func(pipe <-chan int, cancel func(error)) {\n\t\tfor item := range pipe {\n\t\t\ti := item\n\t\t\tresult = append(result, i)\n\t\t}\n\t})\n\tassert.Nil(t, err)\n\tassert.Equal(t, 2, len(result))\n\tassert.Equal(t, 1, result[0])\n\tassert.Equal(t, 0, result[1])\n}\n\nfunc TestMapReducePanic(t *testing.T) {\n\tdefer goleak.VerifyNone(t)\n\n\tassert.Panics(t, func() {\n\t\t_, _ = MapReduce(func(source chan<- int) {\n\t\t\tsource <- 0\n\t\t\tsource <- 1\n\t\t}, func(i int, writer Writer[int], cancel func(error)) {\n\t\t\twriter.Write(i)\n\t\t}, func(pipe <-chan int, writer Writer[int], cancel func(error)) {\n\t\t\tfor range pipe {\n\t\t\t\tpanic(\"panic\")\n\t\t\t}\n\t\t})\n\t})\n}\n\nfunc TestMapReducePanicOnce(t *testing.T) {\n\tdefer goleak.VerifyNone(t)\n\n\tassert.Panics(t, func() {\n\t\t_, _ = MapReduce(func(source chan<- int) {\n\t\t\tfor i := 0; i < 100; i++ {\n\t\t\t\tsource <- i\n\t\t\t}\n\t\t}, func(i int, writer Writer[int], cancel func(error)) {\n\t\t\tif i == 0 {\n\t\t\t\tpanic(\"foo\")\n\t\t\t}\n\t\t\twriter.Write(i)\n\t\t}, func(pipe <-chan int, writer Writer[int], cancel func(error)) {\n\t\t\tfor range pipe {\n\t\t\t\tpanic(\"bar\")\n\t\t\t}\n\t\t})\n\t})\n}\n\nfunc TestMapReducePanicBothMapperAndReducer(t *testing.T) {\n\tdefer goleak.VerifyNone(t)\n\n\tassert.Panics(t, func() {\n\t\t_, _ = MapReduce(func(source chan<- int) {\n\t\t\tsource <- 0\n\t\t\tsource <- 1\n\t\t}, func(item int, writer Writer[int], cancel func(error)) {\n\t\t\tpanic(\"foo\")\n\t\t}, func(pipe <-chan int, writer Writer[int], cancel func(error)) {\n\t\t\tpanic(\"bar\")\n\t\t})\n\t})\n}\n\nfunc TestMapReduceVoidCancel(t *testing.T) {\n\tdefer goleak.VerifyNone(t)\n\n\tvar result []int\n\terr := MapReduceVoid(func(source chan<- int) {\n\t\tsource <- 0\n\t\tsource <- 1\n\t}, func(i int, writer Writer[int], cancel func(error)) {\n\t\tif i == 1 {\n\t\t\tcancel(errors.New(\"anything\"))\n\t\t}\n\t\twriter.Write(i)\n\t}, func(pipe <-chan int, cancel func(error)) {\n\t\tfor item := range pipe {\n\t\t\ti := item\n\t\t\tresult = append(result, i)\n\t\t}\n\t})\n\tassert.NotNil(t, err)\n\tassert.Equal(t, \"anything\", err.Error())\n}\n\nfunc TestMapReduceVoidCancelWithRemains(t *testing.T) {\n\tdefer goleak.VerifyNone(t)\n\n\tvar done int32\n\tvar result []int\n\terr := MapReduceVoid(func(source chan<- int) {\n\t\tfor i := 0; i < defaultWorkers*2; i++ {\n\t\t\tsource <- i\n\t\t}\n\t\tatomic.AddInt32(&done, 1)\n\t}, func(i int, writer Writer[int], cancel func(error)) {\n\t\tif i == defaultWorkers/2 {\n\t\t\tcancel(errors.New(\"anything\"))\n\t\t}\n\t\twriter.Write(i)\n\t}, func(pipe <-chan int, cancel func(error)) {\n\t\tfor item := range pipe {\n\t\t\tresult = append(result, item)\n\t\t}\n\t})\n\tassert.NotNil(t, err)\n\tassert.Equal(t, \"anything\", err.Error())\n\tassert.Equal(t, int32(1), done)\n}\n\nfunc TestMapReduceWithoutReducerWrite(t *testing.T) {\n\tdefer goleak.VerifyNone(t)\n\n\tuids := []int{1, 2, 3}\n\tres, err := MapReduce(func(source chan<- int) {\n\t\tfor _, uid := range uids {\n\t\t\tsource <- uid\n\t\t}\n\t}, func(item int, writer Writer[int], cancel func(error)) {\n\t\twriter.Write(item)\n\t}, func(pipe <-chan int, writer Writer[int], cancel func(error)) {\n\t\tdrain(pipe)\n\t\t// not calling writer.Write(...), should not panic\n\t})\n\tassert.Equal(t, ErrReduceNoOutput, err)\n\tassert.Equal(t, 0, res)\n}\n\nfunc TestMapReduceVoidPanicInReducer(t *testing.T) {\n\tdefer goleak.VerifyNone(t)\n\n\tconst message = \"foo\"\n\tassert.Panics(t, func() {\n\t\tvar done int32\n\t\t_ = MapReduceVoid(func(source chan<- int) {\n\t\t\tfor i := 0; i < defaultWorkers*2; i++ {\n\t\t\t\tsource <- i\n\t\t\t}\n\t\t\tatomic.AddInt32(&done, 1)\n\t\t}, func(i int, writer Writer[int], cancel func(error)) {\n\t\t\twriter.Write(i)\n\t\t}, func(pipe <-chan int, cancel func(error)) {\n\t\t\tpanic(message)\n\t\t}, WithWorkers(1))\n\t})\n}\n\nfunc TestForEachWithContext(t *testing.T) {\n\tdefer goleak.VerifyNone(t)\n\n\tvar done int32\n\tctx, cancel := context.WithCancel(context.Background())\n\tForEach(func(source chan<- int) {\n\t\tfor i := 0; i < defaultWorkers*2; i++ {\n\t\t\tsource <- i\n\t\t}\n\t\tatomic.AddInt32(&done, 1)\n\t}, func(i int) {\n\t\tif i == defaultWorkers/2 {\n\t\t\tcancel()\n\t\t}\n\t}, WithContext(ctx))\n}\n\nfunc TestMapReduceWithContext(t *testing.T) {\n\tdefer goleak.VerifyNone(t)\n\n\tvar done int32\n\tvar result []int\n\tctx, cancel := context.WithCancel(context.Background())\n\terr := MapReduceVoid(func(source chan<- int) {\n\t\tfor i := 0; i < defaultWorkers*2; i++ {\n\t\t\tsource <- i\n\t\t}\n\t\tatomic.AddInt32(&done, 1)\n\t}, func(i int, writer Writer[int], c func(error)) {\n\t\tif i == defaultWorkers/2 {\n\t\t\tcancel()\n\t\t}\n\t\twriter.Write(i)\n\t}, func(pipe <-chan int, cancel func(error)) {\n\t\tfor item := range pipe {\n\t\t\ti := item\n\t\t\tresult = append(result, i)\n\t\t}\n\t}, WithContext(ctx))\n\tassert.NotNil(t, err)\n\tassert.Equal(t, context.DeadlineExceeded, err)\n}\n\nfunc BenchmarkMapReduce(b *testing.B) {\n\tb.ReportAllocs()\n\n\tmapper := func(v int64, writer Writer[int64], cancel func(error)) {\n\t\twriter.Write(v * v)\n\t}\n\treducer := func(input <-chan int64, writer Writer[int64], cancel func(error)) {\n\t\tvar result int64\n\t\tfor v := range input {\n\t\t\tresult += v\n\t\t}\n\t\twriter.Write(result)\n\t}\n\n\tfor i := 0; i < b.N; i++ {\n\t\tMapReduce(func(input chan<- int64) {\n\t\t\tfor j := 0; j < 2; j++ {\n\t\t\t\tinput <- int64(j)\n\t\t\t}\n\t\t}, mapper, reducer)\n\t}\n}\n" }, { "path": "readme-cn.md", "content": "# mapreduce\n\n[English](readme.md) | 简体中文\n\n[![Go](https://github.com/kevwan/mapreduce/workflows/Go/badge.svg?branch=main)](https://github.com/kevwan/mapreduce/actions)\n[![codecov](https://codecov.io/gh/kevwan/mapreduce/branch/main/graph/badge.svg)](https://codecov.io/gh/kevwan/mapreduce)\n[![Go Report Card](https://goreportcard.com/badge/github.com/kevwan/mapreduce)](https://goreportcard.com/report/github.com/kevwan/mapreduce)\n[![Release](https://img.shields.io/github/v/release/kevwan/mapreduce.svg?style=flat-square)](https://github.com/kevwan/mapreduce)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n## 为什么会有这个项目\n\n`mapreduce` 其实是 [go-zero](https://github.com/zeromicro/go-zero) 的一部分，但是一些用户问我是不是可以单独使用 `mapreduce` 而不用引入 `go-zero` 的依赖，所以我考虑再三，还是单独提供一个吧。但是，我强烈推荐你使用 `go-zero`，因为 `go-zero` 真的提供了很多很好的功能。\n\n## 为什么需要 MapReduce\n\n在实际的业务场景中我们常常需要从不同的 rpc 服务中获取相应属性来组装成复杂对象。\n\n比如要查询商品详情：\n\n1. 商品服务-查询商品属性\n2. 库存服务-查询库存属性\n3. 价格服务-查询价格属性\n4. 营销服务-查询营销属性\n\n如果是串行调用的话响应时间会随着 rpc 调用次数呈线性增长，所以我们要优化性能一般会将串行改并行。\n\n简单的场景下使用 waitGroup 也能够满足需求，但是如果我们需要对 rpc 调用返回的数据进行校验、数据加工转换、数据汇总呢？继续使用 waitGroup 就有点力不从心了，go 的官方库中并没有这种工具（java 中提供了 CompleteFuture），go-zero 作者依据 mapReduce 架构思想实现了进程内的数据批处理 mapReduce 并发工具类。\n\n## 设计思路\n\n我们尝试梳理一下并发工具可能的示例业务场景：\n\n1. 查询商品详情：支持并发调用多个服务来组合产品属性，支持调用错误可以立即结束。\n2. 商品详情页自动推荐用户卡券：支持并发校验卡券，校验失败自动剔除，返回全部卡券。\n\n以上实际都是在进行对输入数据进行处理最后输出清洗后的数据，针对数据处理有个非常经典的异步模式：生产者消费者模式。于是我们可以抽象一下数据批处理的生命周期，大致可以分为三个阶段：\n\n\n\n1. 数据生产 generate\n2. 数据加工 mapper\n3. 数据聚合 reducer\n\n其中数据生产是不可或缺的阶段，数据加工、数据聚合是可选阶段，数据生产与加工支持并发调用，数据聚合基本属于纯内存操作单协程即可。\n\n再来思考一下不同阶段之间数据应该如何流转，既然不同阶段的数据处理都是由不同 goroutine 执行的，那么很自然的可以考虑采用 channel 来实现 goroutine 之间的通信。\n\n\n\n\n如何实现随时终止流程呢？\n\n`goroutine` 中监听一个全局的结束 `channel` 和调用方提供的 `ctx` 就行。\n\n## 版本选择\n\n- `v1`（默认）- 非泛型版本\n- `v2`（泛型版）- 泛型版本，需要 Go 版本 >= 1.18\n\n## 简单示例\n\n并行求平方和（不要嫌弃示例简单，只是模拟并发）\n\n```go\npackage main\n\nimport (\n \"fmt\"\n \"log\"\n\n \"github.com/kevwan/mapreduce/v2\"\n)\n\nfunc main() {\n val, err := mapreduce.MapReduce(func(source chan<- int) {\n // generator\n for i := 0; i < 10; i++ {\n source <- i\n }\n }, func(i int, writer mapreduce.Writer[int], cancel func(error)) {\n // mapper\n writer.Write(i * i)\n }, func(pipe <-chan int, writer mapreduce.Writer[int], cancel func(error)) {\n // reducer\n var sum int\n for i := range pipe {\n sum += i\n }\n writer.Write(sum)\n })\n if err != nil {\n log.Fatal(err)\n }\n fmt.Println(\"result:\", val)\n}\n```\n\n更多示例：[https://github.com/zeromicro/zero-examples/tree/main/mapreduce](https://github.com/zeromicro/zero-examples/tree/main/mapreduce)\n\n## 强烈推荐！\n\ngo-zero: [https://github.com/zeromicro/go-zero](https://github.com/zeromicro/go-zero)\n\n## 欢迎 star！⭐\n\n如果你正在使用或者觉得这个项目对你有帮助，请 **star** 支持，感谢！\n" }, { "path": "readme.md", "content": "\n\n# mapreduce\n\nEnglish | [简体中文](readme-cn.md)\n\n[![Go](https://github.com/kevwan/mapreduce/workflows/Go/badge.svg?branch=main)](https://github.com/kevwan/mapreduce/actions)\n[![codecov](https://codecov.io/gh/kevwan/mapreduce/branch/main/graph/badge.svg)](https://codecov.io/gh/kevwan/mapreduce)\n[![Go Report Card](https://goreportcard.com/badge/github.com/kevwan/mapreduce)](https://goreportcard.com/report/github.com/kevwan/mapreduce)\n[![Release](https://img.shields.io/github/v/release/kevwan/mapreduce.svg?style=flat-square)](https://github.com/kevwan/mapreduce)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n## Why we have this repo\n\n`mapreduce` is part of [go-zero](https://github.com/zeromicro/go-zero), but a few people asked if mapreduce can be used separately. But I recommend you to use `go-zero` for many more features.\n\n## Why MapReduce is needed\n\nIn practical business scenarios we often need to get the corresponding properties from different rpc services to assemble complex objects.\n\nFor example, to query product details.\n\n1. product service - query product attributes\n2. inventory service - query inventory properties\n3. price service - query price attributes\n4. marketing service - query marketing properties\n\nIf it is a serial call, the response time will increase linearly with the number of rpc calls, so we will generally change serial to parallel to optimize response time.\n\nSimple scenarios using `WaitGroup` can also meet the needs, but what if we need to check the data returned by the rpc call, data processing, data aggregation? The official go library does not have such a tool (CompleteFuture is provided in java), so we implemented an in-process data batching MapReduce concurrent tool based on the MapReduce architecture.\n\n## Design ideas\n\nLet's sort out the possible business scenarios for the concurrency tool:\n\n1. querying product details: supporting concurrent calls to multiple services to combine product attributes, and supporting call errors that can be ended immediately.\n2. automatic recommendation of user card coupons on product details page: support concurrently verifying card coupons, automatically rejecting them if they fail, and returning all of them.\n\nThe above is actually processing the input data and finally outputting the cleaned data. There is a very classic asynchronous pattern for data processing: the producer-consumer pattern. So we can abstract the life cycle of data batch processing, which can be roughly divided into three phases.\n\n\n\n1. data production generate\n2. data processing mapper\n3. data aggregation reducer\n\nData producing is an indispensable stage, data processing and data aggregation are optional stages, data producing and processing support concurrent calls, data aggregation is basically a pure memory operation, so a single concurrent process can do it.\n\nSince different stages of data processing are performed by different goroutines, it is natural to consider the use of channel to achieve communication between goroutines.\n\n\n\nHow can I terminate the process at any time?\n\nIt's simple, just receive from a channel or the given context in the goroutine.\n\n## Choose the right version\n\n- `v1` (default) - non-generic version\n- `v2` (generics) - generic version, needs Go version >= 1.18\n\n## A simple example\n\nCalculate the sum of squares, simulating the concurrency.\n\n```go\npackage main\n\nimport (\n \"fmt\"\n \"log\"\n\n \"github.com/kevwan/mapreduce/v2\"\n)\n\nfunc main() {\n val, err := mapreduce.MapReduce(func(source chan<- int) {\n // generator\n for i := 0; i < 10; i++ {\n source <- i\n }\n }, func(i int, writer mapreduce.Writer[int], cancel func(error)) {\n // mapper\n writer.Write(i * i)\n }, func(pipe <-chan int, writer mapreduce.Writer[int], cancel func(error)) {\n // reducer\n var sum int\n for i := range pipe {\n sum += i\n }\n writer.Write(sum)\n })\n if err != nil {\n log.Fatal(err)\n }\n fmt.Println(\"result:\", val)\n}\n```\n\nMore examples: [https://github.com/zeromicro/zero-examples/tree/main/mapreduce](https://github.com/zeromicro/zero-examples/tree/main/mapreduce)\n\n## References\n\ngo-zero: [https://github.com/zeromicro/go-zero](https://github.com/zeromicro/go-zero)\n\n## Give a Star! ⭐\n\nIf you like or are using this project to learn or start your solution, please give it a star. Thanks!\n" } ]