[ { "path": ".github/FUNDING.yml", "content": "github: carlmjohnson\n" }, { "path": ".github/workflows/go.yml", "content": "name: Go\n\non: [ push, pull_request ]\n\njobs:\n\n build:\n name: Build\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v4\n - uses: actions/setup-go@v3\n with:\n go-version: '1.21'\n cache: true\n - name: Get dependencies\n run: go mod download\n - name: Test\n run: go test -race -v -coverprofile=profile.cov ./...\n - name: Upload Coverage\n uses: shogo82148/actions-goveralls@v1\n with:\n path-to-profile: profile.cov\n" }, { "path": ".gitignore", "content": "img/\n" }, { "path": "LICENSE", "content": "MIT License\n\nCopyright (c) 2022 Carl Johnson\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n" }, { "path": "README.md", "content": "# Flowmatic [![GoDoc](https://pkg.go.dev/badge/github.com/carlmjohnson/flowmatic)](https://pkg.go.dev/github.com/carlmjohnson/flowmatic) [![Coverage Status](https://coveralls.io/repos/github/carlmjohnson/flowmatic/badge.svg)](https://coveralls.io/github/carlmjohnson/flowmatic) [![Go Report Card](https://goreportcard.com/badge/github.com/carlmjohnson/flowmatic)](https://goreportcard.com/report/github.com/carlmjohnson/flowmatic) [![Mentioned in Awesome Go](https://awesome.re/mentioned-badge.svg)](https://github.com/avelino/awesome-go)\n\n![Flowmatic logo](https://github.com/carlmjohnson/flowmatic/assets/222245/c14936e9-bb35-405b-926e-4cfeb8003439)\n\n\nFlowmatic is a generic Go library that provides a [structured approach](https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/) to concurrent programming. It lets you easily manage concurrent tasks in a manner that is simple, yet effective and flexible.\n\nFlowmatic has an easy to use API with functions for handling common concurrency patterns. It automatically handles spawning workers, collecting errors, and propagating panics.\n\nFlowmatic requires Go 1.20+.\n\n## Features\n\n- Has a simple API that improves readability over channels/waitgroups/mutexes\n- Handles a variety of concurrency problems such as heterogenous task groups, homogenous execution of a task over a slice, and dynamic work spawning\n- Aggregates errors\n- Properly propagates panics across goroutine boundaries\n- Has helpers for context cancelation\n- Few dependencies\n- Good test coverage\n\n## How to use Flowmatic\n\n### Execute heterogenous tasks\nOne problem that Flowmatic solves is managing the execution of multiple tasks in parallel that are independent of each other. For example, let's say you want to send data to three different downstream APIs. If any of the sends fail, you want to return an error. With traditional Go concurrency, this can quickly become complex and difficult to manage, with Goroutines, channels, and `sync.WaitGroup`s to keep track of. Flowmatic makes it simple.\n\nTo execute heterogenous tasks, just use `flowmatic.Do`:\n\n\n\n\n\n\n\n\n\n\n
flowmaticstdlib
\n\n```go\nerr := flowmatic.Do(\n func() error {\n return doThingA(),\n },\n func() error {\n return doThingB(),\n },\n func() error {\n return doThingC(),\n })\n```\n\n\n\n```go\nvar wg sync.WaitGroup\nvar errs []error\nerrChan := make(chan error)\n\nwg.Add(3)\ngo func() {\n defer wg.Done()\n if err := doThingA(); err != nil {\n errChan <- err\n }\n}()\ngo func() {\n defer wg.Done()\n if err := doThingB(); err != nil {\n errChan <- err\n }\n}()\ngo func() {\n defer wg.Done()\n if err := doThingC(); err != nil {\n errChan <- err\n }\n}()\n\ngo func() {\n wg.Wait()\n close(errChan)\n}()\n\nfor err := range errChan {\n errs = append(errs, err)\n}\n\nerr := errors.Join(errs...)\n```\n\n
\n\nTo create a context for tasks that is canceled on the first error,\nuse `flowmatic.All`.\nTo create a context for tasks that is canceled on the first success,\nuse `flowmatic.Race`.\n\n```go\n// Make variables to hold responses\nvar pageA, pageB, pageC string\n// Race the requests to see who can answer first\nerr := flowmatic.Race(ctx,\n\tfunc(ctx context.Context) error {\n\t\tvar err error\n\t\tpageA, err = request(ctx, \"A\")\n\t\treturn err\n\t},\n\tfunc(ctx context.Context) error {\n\t\tvar err error\n\t\tpageB, err = request(ctx, \"B\")\n\t\treturn err\n\t},\n\tfunc(ctx context.Context) error {\n\t\tvar err error\n\t\tpageC, err = request(ctx, \"C\")\n\t\treturn err\n\t},\n)\n```\n\n### Execute homogenous tasks\n`flowmatic.Each` is useful if you need to execute the same task on each item in a slice using a worker pool:\n\n\n\n\n\n\n\n\n\n\n
flowmaticstdlib
\n\n```go\nthings := []someType{thingA, thingB, thingC}\n\nerr := flowmatic.Each(numWorkers, things,\n func(thing someType) error {\n foo := thing.Frobincate()\n return foo.DoSomething()\n })\n```\n\n\n\n```go\nthings := []someType{thingA, thingB, thingC}\n\nwork := make(chan someType)\nerrs := make(chan error)\n\nfor i := 0; i < numWorkers; i++ {\n go func() {\n for thing := range work {\n // Omitted: panic handling!\n foo := thing.Frobincate()\n errs <- foo.DoSomething()\n }\n }()\n}\n\ngo func() {\n for _, thing := range things {\n work <- thing\n }\n\n close(tasks)\n}()\n\nvar collectedErrs []error\nfor i := 0; i < len(things); i++ {\n collectedErrs = append(collectedErrs, <-errs)\n}\n\nerr := errors.Join(collectedErrs...)\n```\n\n
\n\nUse `flowmatic.Map` to map an input slice to an output slice.\n\n\n\n\n\n\n\n\n\n\n
\n\n```go\nfunc main() {\n\tresults, err := Google(context.Background(), \"golang\")\n\tif err != nil {\n\t\tfmt.Fprintln(os.Stderr, err)\n\t\treturn\n\t}\n\tfor _, result := range results {\n\t\tfmt.Println(result)\n\t}\n}\n```\n\n
flowmaticx/sync/errgroup
\n\n```go\nfunc Google(ctx context.Context, query string) ([]Result, error) {\n\tsearches := []Search{Web, Image, Video}\n\treturn flowmatic.Map(ctx, flowmatic.MaxProcs, searches,\n\t\tfunc(ctx context.Context, search Search) (Result, error) {\n\t\t\treturn search(ctx, query)\n\t\t})\n}\n```\n\n\n\n\n```go\nfunc Google(ctx context.Context, query string) ([]Result, error) {\n\tg, ctx := errgroup.WithContext(ctx)\n\n\tsearches := []Search{Web, Image, Video}\n\tresults := make([]Result, len(searches))\n\tfor i, search := range searches {\n\t\ti, search := i, search // https://golang.org/doc/faq#closures_and_goroutines\n\t\tg.Go(func() error {\n\t\t\tresult, err := search(ctx, query)\n\t\t\tif err == nil {\n\t\t\t\tresults[i] = result\n\t\t\t}\n\t\t\treturn err\n\t\t})\n\t}\n\tif err := g.Wait(); err != nil {\n\t\treturn nil, err\n\t}\n\treturn results, nil\n}\n```\n\n
\n\n### Manage tasks that spawn new tasks\nFor tasks that may create more work, use `flowmatic.ManageTasks`.\nCreate a manager that will be serially executed,\nand have it save the results\nand examine the output of tasks to decide if there is more work to be done.\n\n```go\n// Task fetches a page and extracts the URLs\ntask := func(u string) ([]string, error) {\n page, err := getURL(ctx, u)\n if err != nil {\n return nil, err\n }\n return getLinks(page), nil\n}\n\n// Map from page to links\n// Doesn't need a lock because only the manager touches it\nresults := map[string][]string{}\nvar managerErr error\n\n// Manager keeps track of which pages have been visited and the results graph\nmanager := func(req string, links []string, err error) ([]string, bool) {\n // Halt execution after the first error\n if err != nil {\n managerErr = err\n return nil, false\n }\n // Save final results in map\n results[req] = urls\n\n // Check for new pages to scrape\n var newpages []string\n for _, link := range links {\n if _, ok := results[link]; ok {\n // Seen it, try the next link\n continue\n }\n // Add to list of new pages\n newpages = append(newpages, link)\n // Add placeholder to map to prevent double scraping\n results[link] = nil\n }\n return newpages, true\n}\n\n// Process the tasks with as many workers as GOMAXPROCS\nflowmatic.ManageTasks(flowmatic.MaxProcs, task, manager, \"http://example.com/\")\n// Check if anything went wrong\nif managerErr != nil {\n fmt.Println(\"error\", managerErr)\n}\n```\n\nNormally, it is very difficult to keep track of concurrent code because any combination of events could occur in any order or simultaneously, and each combination has to be accounted for by the programmer. `flowmatic.ManageTasks` makes it simple to write concurrent code because everything follows a simple rule: **tasks happen concurrently; the manager runs serially**.\n\nCentralizing control in the manager makes reasoning about the code radically simpler. When writing locking code, if you have M states and N methods, you need to think about all N states in each of the M methods, giving you an M × N code explosion. By centralizing the logic, the N states only need to be considered in one location: the manager.\n\n### Advanced patterns with TaskPool\n\nFor very advanced uses, `flowmatic.TaskPool` takes the boilerplate out of managing a pool of workers. Compare Flowmatic to [this example from x/sync/errgroup](https://pkg.go.dev/golang.org/x/sync/errgroup#example-Group-Pipeline):\n\n\n\n\n\n\n\n\n\n\n\n
\n\n```go\nfunc main() {\n\tm, err := MD5All(context.Background(), \".\")\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\n\tfor k, sum := range m {\n\t\tfmt.Printf(\"%s:\\t%x\\n\", k, sum)\n\t}\n}\n```\n\n
flowmaticx/sync/errgroup
\n\n\n```go\n// MD5All reads all the files in the file tree rooted at root\n// and returns a map from file path to the MD5 sum of the file's contents.\n// If the directory walk fails or any read operation fails,\n// MD5All returns an error.\nfunc MD5All(ctx context.Context, root string) (map[string][md5.Size]byte, error) {\n\t// Make a pool of 20 digesters\n\tin, out := flowmatic.TaskPool(20, digest)\n\n\tm := make(map[string][md5.Size]byte)\n\t// Open two goroutines:\n\t// one for reading file names by walking the filesystem\n\t// one for recording results from the digesters in a map\n\terr := flowmatic.All(ctx,\n\t\tfunc(ctx context.Context) error {\n\t\t\treturn walkFilesystem(ctx, root, in)\n\t\t},\n\t\tfunc(ctx context.Context) error {\n\t\t\tfor r := range out {\n\t\t\t\tif r.Err != nil {\n\t\t\t\t\treturn r.Err\n\t\t\t\t}\n\t\t\t\tm[r.In] = *r.Out\n\t\t\t}\n\t\t\treturn nil\n\t\t},\n\t)\n\n\treturn m, err\n}\n\nfunc walkFilesystem(ctx context.Context, root string, in chan<- string) error {\n\tdefer close(in)\n\n\treturn filepath.Walk(root, func(path string, info os.FileInfo, err error) error {\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\tif !info.Mode().IsRegular() {\n\t\t\treturn nil\n\t\t}\n\t\tselect {\n\t\tcase in <- path:\n\t\tcase <-ctx.Done():\n\t\t\treturn ctx.Err()\n\t\t}\n\n\t\treturn nil\n\t})\n}\n\nfunc digest(path string) (*[md5.Size]byte, error) {\n\tdata, err := os.ReadFile(path)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\thash := md5.Sum(data)\n\treturn &hash, nil\n}\n```\n\n\n\n```go\ntype result struct {\n\tpath string\n\tsum [md5.Size]byte\n}\n\n// MD5All reads all the files in the file tree rooted at root and returns a map\n// from file path to the MD5 sum of the file's contents. If the directory walk\n// fails or any read operation fails, MD5All returns an error.\nfunc MD5All(ctx context.Context, root string) (map[string][md5.Size]byte, error) {\n\t// ctx is canceled when g.Wait() returns. When this version of MD5All returns\n\t// - even in case of error! - we know that all of the goroutines have finished\n\t// and the memory they were using can be garbage-collected.\n\tg, ctx := errgroup.WithContext(ctx)\n\tpaths := make(chan string)\n\n\tg.Go(func() error {\n\t\tdefer close(paths)\n\t\treturn filepath.Walk(root, func(path string, info os.FileInfo, err error) error {\n\t\t\tif err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t\tif !info.Mode().IsRegular() {\n\t\t\t\treturn nil\n\t\t\t}\n\t\t\tselect {\n\t\t\tcase paths <- path:\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn ctx.Err()\n\t\t\t}\n\t\t\treturn nil\n\t\t})\n\t})\n\n\t// Start a fixed number of goroutines to read and digest files.\n\tc := make(chan result)\n\tconst numDigesters = 20\n\tfor i := 0; i < numDigesters; i++ {\n\t\tg.Go(func() error {\n\t\t\tfor path := range paths {\n\t\t\t\tdata, err := ioutil.ReadFile(path)\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn err\n\t\t\t\t}\n\t\t\t\tselect {\n\t\t\t\tcase c <- result{path, md5.Sum(data)}:\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn ctx.Err()\n\t\t\t\t}\n\t\t\t}\n\t\t\treturn nil\n\t\t})\n\t}\n\tgo func() {\n\t\tg.Wait()\n\t\tclose(c)\n\t}()\n\n\tm := make(map[string][md5.Size]byte)\n\tfor r := range c {\n\t\tm[r.path] = r.sum\n\t}\n\t// Check whether any of the goroutines failed. Since g is accumulating the\n\t// errors, we don't need to send them (or check for them) in the individual\n\t// results sent on the channel.\n\tif err := g.Wait(); err != nil {\n\t\treturn nil, err\n\t}\n\treturn m, nil\n}\n```\n\n
\n\n## Note on panicking\n\nIn Go, if there is a panic in a goroutine, and that panic is not recovered, then the whole process is shutdown. There are pros and cons to this approach. The pro is that if the panic is the symptom of a programming error in the application, no further damage can be done by the application. The con is that in many cases, this leads to a shutdown in a situation that might be recoverable.\n\nAs a result, although the Go standard HTTP server will catch panics that occur in one of its HTTP handlers and continue serving requests, a standard Go HTTP server cannot catch panics that occur in separate goroutines, and these will cause the whole server to go offline.\n\nFlowmatic fixes this problem by catching a panic that occurs in one of its worker goroutines and repropagating it in the parent goroutine, so the panic can be caught and logged at the appropriate level.\n" }, { "path": "all.go", "content": "package flowmatic\n\nimport (\n\t\"context\"\n)\n\n// All runs each task concurrently\n// and waits for them all to finish.\n// Each task receives a child context\n// which is canceled once one task returns an error or panics.\n// All returns nil if all tasks succeed.\n// Otherwise,\n// All returns a multierror containing the errors encountered.\n// If a task panics during execution,\n// a panic will be caught and rethrown in the parent Goroutine.\nfunc All(ctx context.Context, tasks ...func(context.Context) error) error {\n\tctx, cancel := context.WithCancel(ctx)\n\tdefer cancel()\n\n\treturn eachN(len(tasks), len(tasks), func(pos int) error {\n\t\tdefer func() {\n\t\t\tpanicVal := recover()\n\t\t\tif panicVal != nil {\n\t\t\t\tcancel()\n\t\t\t\tpanic(panicVal)\n\t\t\t}\n\t\t}()\n\t\terr := tasks[pos](ctx)\n\t\tif err != nil {\n\t\t\tcancel()\n\t\t\treturn err\n\t\t}\n\t\treturn nil\n\t})\n}\n" }, { "path": "all_example_test.go", "content": "package flowmatic_test\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"time\"\n\n\t\"github.com/carlmjohnson/flowmatic\"\n)\n\nfunc ExampleAll() {\n\tctx := context.Background()\n\tstart := time.Now()\n\terr := flowmatic.All(ctx,\n\t\tfunc(ctx context.Context) error {\n\t\t\t// This task sleeps then returns an error\n\t\t\td := 1 * time.Millisecond\n\t\t\ttime.Sleep(d)\n\t\t\tfmt.Println(\"slept for\", d)\n\t\t\treturn fmt.Errorf(\"abort after %v\", d)\n\t\t},\n\t\tfunc(ctx context.Context) error {\n\t\t\t// sleepFor is a cancelable time.Sleep.\n\t\t\t// The error of first task\n\t\t\t// causes the early cancelation of this one.\n\t\t\tif !sleepFor(ctx, 1*time.Minute) {\n\t\t\t\tfmt.Println(\"canceled\")\n\t\t\t}\n\t\t\treturn nil\n\t\t},\n\t)\n\tfmt.Println(\"err:\", err)\n\tfmt.Println(\"exited early?\", time.Since(start) < 10*time.Millisecond)\n\t// Output:\n\t// slept for 1ms\n\t// canceled\n\t// err: abort after 1ms\n\t// exited early? true\n}\n" }, { "path": "do.go", "content": "package flowmatic\n\nimport (\n\t\"errors\"\n\t\"sync\"\n)\n\n// Do runs each task concurrently\n// and waits for them all to finish.\n// Errors returned by tasks do not cancel execution,\n// but are joined into a multierror return value.\n// If a task panics during execution,\n// a panic will be caught and rethrown in the parent Goroutine.\nfunc Do(tasks ...func() error) error {\n\ttype result struct {\n\t\terr error\n\t\tpanic any\n\t}\n\n\tvar wg sync.WaitGroup\n\terrch := make(chan result, len(tasks))\n\n\twg.Add(len(tasks))\n\tfor i := range tasks {\n\t\tfn := tasks[i]\n\t\tgo func() {\n\t\t\tdefer wg.Done()\n\t\t\tdefer func() {\n\t\t\t\tif panicVal := recover(); panicVal != nil {\n\t\t\t\t\terrch <- result{panic: panicVal}\n\t\t\t\t}\n\t\t\t}()\n\t\t\terrch <- result{err: fn()}\n\t\t}()\n\t}\n\tgo func() {\n\t\twg.Wait()\n\t\tclose(errch)\n\t}()\n\n\tvar (\n\t\tpanicVal any\n\t\terrs []error\n\t)\n\tfor res := range errch {\n\t\tswitch {\n\t\tcase res.err == nil && res.panic == nil:\n\t\t\tcontinue\n\t\tcase res.panic != nil:\n\t\t\tpanicVal = res.panic\n\t\tcase res.err != nil:\n\t\t\terrs = append(errs, res.err)\n\t\t}\n\t}\n\tif panicVal != nil {\n\t\tpanic(panicVal)\n\t}\n\treturn errors.Join(errs...)\n}\n" }, { "path": "do_example_test.go", "content": "package flowmatic_test\n\nimport (\n\t\"fmt\"\n\t\"time\"\n\n\t\"github.com/carlmjohnson/flowmatic\"\n)\n\nfunc ExampleDo() {\n\tstart := time.Now()\n\terr := flowmatic.Do(\n\t\tfunc() error {\n\t\t\ttime.Sleep(50 * time.Millisecond)\n\t\t\tfmt.Println(\"hello\")\n\t\t\treturn nil\n\t\t}, func() error {\n\t\t\ttime.Sleep(100 * time.Millisecond)\n\t\t\tfmt.Println(\"world\")\n\t\t\treturn nil\n\t\t}, func() error {\n\t\t\ttime.Sleep(200 * time.Millisecond)\n\t\t\tfmt.Println(\"from flowmatic.Do\")\n\t\t\treturn nil\n\t\t})\n\tif err != nil {\n\t\tfmt.Println(\"error\", err)\n\t}\n\tfmt.Println(\"executed concurrently?\", time.Since(start) < 250*time.Millisecond)\n\t// Output:\n\t// hello\n\t// world\n\t// from flowmatic.Do\n\t// executed concurrently? true\n}\n" }, { "path": "do_test.go", "content": "package flowmatic_test\n\nimport (\n\t\"errors\"\n\t\"testing\"\n\n\t\"github.com/carlmjohnson/flowmatic\"\n)\n\nfunc TestDo_err(t *testing.T) {\n\ta := errors.New(\"a\")\n\tb := errors.New(\"b\")\n\terrs := flowmatic.Do(\n\t\tfunc() error { return a },\n\t\tfunc() error { return b },\n\t)\n\tif !errors.Is(errs, a) {\n\t\tt.Fatal(errs)\n\t}\n\tif !errors.Is(errs, b) {\n\t\tt.Fatal(errs)\n\t}\n}\n" }, { "path": "doc.go", "content": "// Package flowmatic contains easy-to-use generic helpers for structured concurrency.\n//\n// Comparison of simple helpers:\n//\n//\t Tasks Cancels Context? Collect results?\n//\tDo Different No No\n//\tAll Different On error No\n//\tRace Different On success No\n//\tEach Same No No\n//\tMap Same On error Yes\n//\n// ManageTasks and TaskPool allow for advanced concurrency patterns.\npackage flowmatic\n\n// MaxProcs means use GOMAXPROCS workers when doing tasks.\nconst MaxProcs = -1\n" }, { "path": "each.go", "content": "package flowmatic\n\nimport (\n\t\"errors\"\n)\n\n// Each starts numWorkers concurrent workers (or GOMAXPROCS workers if numWorkers < 1)\n// and processes each item as a task.\n// Errors returned by a task do not halt execution,\n// but are joined into a multierror return value.\n// If a task panics during execution,\n// the panic will be caught and rethrown in the parent Goroutine.\nfunc Each[Input any](numWorkers int, items []Input, task func(Input) error) error {\n\treturn eachN(numWorkers, len(items), func(pos int) error {\n\t\treturn task(items[pos])\n\t})\n}\n\n// eachN starts numWorkers concurrent workers (or GOMAXPROCS workers if numWorkers < 1)\n// and starts a task for each number from 0 to numItems.\n// Errors returned by a task do not halt execution,\n// but are joined into a multierror return value.\n// If a task panics during execution,\n// the panic will be caught and rethrown in the parent Goroutine.\nfunc eachN(numWorkers, numItems int, task func(int) error) error {\n\ttype void struct{}\n\tinch, ouch := TaskPool(numWorkers, func(pos int) (void, error) {\n\t\treturn void{}, task(pos)\n\t})\n\tvar (\n\t\tpanicVal any\n\t\terrs []error\n\t)\n\t_ = Do(\n\t\tfunc() error {\n\t\t\tfor i := 0; i < numItems; i++ {\n\t\t\t\tinch <- i\n\t\t\t}\n\t\t\tclose(inch)\n\t\t\treturn nil\n\t\t},\n\t\tfunc() error {\n\t\t\tfor r := range ouch {\n\t\t\t\tif r.Panic != nil && panicVal == nil {\n\t\t\t\t\tpanicVal = r.Panic\n\t\t\t\t}\n\t\t\t\tif r.Err != nil {\n\t\t\t\t\terrs = append(errs, r.Err)\n\t\t\t\t}\n\t\t\t}\n\t\t\treturn nil\n\t\t})\n\tif panicVal != nil {\n\t\tpanic(panicVal)\n\t}\n\treturn errors.Join(errs...)\n}\n" }, { "path": "each_example_test.go", "content": "package flowmatic_test\n\nimport (\n\t\"fmt\"\n\t\"time\"\n\n\t\"github.com/carlmjohnson/flowmatic\"\n)\n\nfunc ExampleEach() {\n\ttimes := []time.Duration{\n\t\t50 * time.Millisecond,\n\t\t100 * time.Millisecond,\n\t\t200 * time.Millisecond,\n\t}\n\tstart := time.Now()\n\terr := flowmatic.Each(3, times, func(d time.Duration) error {\n\t\ttime.Sleep(d)\n\t\tfmt.Println(\"slept\", d)\n\t\treturn nil\n\t})\n\tif err != nil {\n\t\tfmt.Println(\"error\", err)\n\t}\n\tfmt.Println(\"executed concurrently?\", time.Since(start) < 300*time.Millisecond)\n\t// Output:\n\t// slept 50ms\n\t// slept 100ms\n\t// slept 200ms\n\t// executed concurrently? true\n}\n" }, { "path": "each_test.go", "content": "package flowmatic_test\n\nimport (\n\t\"errors\"\n\t\"testing\"\n\n\t\"github.com/carlmjohnson/flowmatic\"\n)\n\nfunc TestEach_err(t *testing.T) {\n\ta := errors.New(\"a\")\n\tb := errors.New(\"b\")\n\terrs := flowmatic.Each(1, []int{1, 2, 3}, func(i int) error {\n\t\tswitch i {\n\t\tcase 1:\n\t\t\treturn a\n\t\tcase 2:\n\t\t\treturn b\n\t\tdefault:\n\t\t\treturn nil\n\t\t}\n\t})\n\tif !errors.Is(errs, a) {\n\t\tt.Fatal(errs)\n\t}\n\tif !errors.Is(errs, b) {\n\t\tt.Fatal(errs)\n\t}\n}\n" }, { "path": "go.mod", "content": "module github.com/carlmjohnson/flowmatic\n\ngo 1.21\n\nrequire github.com/carlmjohnson/deque v0.23.1\n" }, { "path": "go.sum", "content": "github.com/carlmjohnson/deque v0.23.1 h1:X2HOJM9xcglY03deMZ0oZ1V2xtbqYV7dJDnZiSZN4Ak=\ngithub.com/carlmjohnson/deque v0.23.1/go.mod h1:LF5NJjICBrEOPx84pxPL4nCimy5n9NQjxKi5cXkh+8U=\n" }, { "path": "manage_tasks.go", "content": "package flowmatic\n\nimport (\n\t\"github.com/carlmjohnson/deque\"\n)\n\n// Manager is a function that serially examines Task results to see if it produced any new Inputs.\n// Returning false will halt the processing of future tasks.\ntype Manager[Input, Output any] func(Input, Output, error) (tasks []Input, ok bool)\n\n// Task is a function that can concurrently transform an input into an output.\ntype Task[Input, Output any] func(in Input) (out Output, err error)\n\n// ManageTasks manages tasks using numWorkers concurrent workers (or GOMAXPROCS workers if numWorkers < 1)\n// which produce output consumed by a serially run manager.\n// The manager should return a slice of new task inputs based on prior task results,\n// or return false to halt processing.\n// If a task panics during execution,\n// the panic will be caught and rethrown in the parent Goroutine.\nfunc ManageTasks[Input, Output any](numWorkers int, task Task[Input, Output], manager Manager[Input, Output], initial ...Input) {\n\tin, out := TaskPool(numWorkers, task)\n\tdefer func() {\n\t\tclose(in)\n\t\t// drain any waiting tasks\n\t\tfor range out {\n\t\t}\n\t}()\n\tqueue := deque.Of(initial...)\n\tinflight := 0\n\tfor inflight > 0 || queue.Len() > 0 {\n\t\tinch := in\n\t\titem, ok := queue.Head()\n\t\tif !ok {\n\t\t\tinch = nil\n\t\t}\n\t\tselect {\n\t\tcase inch <- item:\n\t\t\tinflight++\n\t\t\tqueue.RemoveFront()\n\t\tcase r := <-out:\n\t\t\tinflight--\n\t\t\tif r.Panic != nil {\n\t\t\t\tpanic(r.Panic)\n\t\t\t}\n\t\t\titems, ok := manager(r.In, r.Out, r.Err)\n\t\t\tif !ok {\n\t\t\t\treturn\n\t\t\t}\n\t\t\tqueue.PushBackSlice(items)\n\t\t}\n\t}\n}\n" }, { "path": "manage_tasks_example_test.go", "content": "package flowmatic_test\n\nimport (\n\t\"fmt\"\n\t\"io\"\n\t\"net/http\"\n\t\"net/http/httptest\"\n\t\"slices\"\n\t\"strings\"\n\t\"testing/fstest\"\n\n\t\"github.com/carlmjohnson/flowmatic\"\n)\n\nfunc ExampleManageTasks() {\n\t// Example site to crawl with recursive links\n\tsrv := httptest.NewServer(http.FileServer(http.FS(fstest.MapFS{\n\t\t\"index.html\": &fstest.MapFile{\n\t\t\tData: []byte(\"/a.html\"),\n\t\t},\n\t\t\"a.html\": &fstest.MapFile{\n\t\t\tData: []byte(\"/b1.html\\n/b2.html\"),\n\t\t},\n\t\t\"b1.html\": &fstest.MapFile{\n\t\t\tData: []byte(\"/c.html\"),\n\t\t},\n\t\t\"b2.html\": &fstest.MapFile{\n\t\t\tData: []byte(\"/c.html\"),\n\t\t},\n\t\t\"c.html\": &fstest.MapFile{\n\t\t\tData: []byte(\"/\"),\n\t\t},\n\t})))\n\tdefer srv.Close()\n\tcl := srv.Client()\n\n\t// Task fetches a page and extracts the URLs\n\ttask := func(u string) ([]string, error) {\n\t\tres, err := cl.Get(srv.URL + u)\n\t\tif err != nil {\n\t\t\treturn nil, err\n\t\t}\n\t\tdefer res.Body.Close()\n\t\tbody, err := io.ReadAll(res.Body)\n\t\tif err != nil {\n\t\t\treturn nil, err\n\t\t}\n\n\t\treturn strings.Split(string(body), \"\\n\"), nil\n\t}\n\n\t// Manager keeps track of which pages have been visited and the results graph\n\ttried := map[string]int{}\n\tresults := map[string][]string{}\n\tmanager := func(req string, urls []string, err error) ([]string, bool) {\n\t\tif err != nil {\n\t\t\t// If there's a problem fetching a page, try three times\n\t\t\tif tried[req] < 3 {\n\t\t\t\ttried[req]++\n\t\t\t\treturn []string{req}, true\n\t\t\t}\n\t\t\treturn nil, false\n\t\t}\n\t\tresults[req] = urls\n\t\tvar newurls []string\n\t\tfor _, u := range urls {\n\t\t\tif tried[u] == 0 {\n\t\t\t\tnewurls = append(newurls, u)\n\t\t\t\ttried[u]++\n\t\t\t}\n\t\t}\n\t\treturn newurls, true\n\t}\n\n\t// Process the tasks with as many workers as GOMAXPROCS\n\tflowmatic.ManageTasks(flowmatic.MaxProcs, task, manager, \"/\")\n\n\tkeys := make([]string, 0, len(results))\n\tfor key := range results {\n\t\tkeys = append(keys, key)\n\t}\n\tslices.Sort(keys)\n\tfor _, key := range keys {\n\t\tfmt.Println(key, \"links to:\")\n\t\tfor _, v := range results[key] {\n\t\t\tfmt.Println(\"- \", v)\n\t\t}\n\t}\n\n\t// Output:\n\t// / links to:\n\t// - /a.html\n\t// /a.html links to:\n\t// - /b1.html\n\t// - /b2.html\n\t// /b1.html links to:\n\t// - /c.html\n\t// /b2.html links to:\n\t// - /c.html\n\t// /c.html links to:\n\t// - /\n}\n" }, { "path": "manage_tasks_test.go", "content": "package flowmatic_test\n\nimport (\n\t\"errors\"\n\t\"fmt\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/carlmjohnson/flowmatic\"\n)\n\nfunc TestManageTasks_drainage(t *testing.T) {\n\tconst sleepTime = 10 * time.Millisecond\n\tb := false\n\ttask := func(n int) (int, error) {\n\t\tif n == 1 {\n\t\t\treturn 0, errors.New(\"text string\")\n\t\t}\n\t\ttime.Sleep(sleepTime)\n\t\tb = true\n\t\treturn 0, nil\n\t}\n\tstart := time.Now()\n\tm := map[int]struct {\n\t\tint\n\t\terror\n\t}{}\n\tmanager := func(in, out int, err error) ([]int, bool) {\n\t\tm[in] = struct {\n\t\t\tint\n\t\t\terror\n\t\t}{out, err}\n\t\tif err != nil {\n\t\t\treturn nil, false\n\t\t}\n\t\treturn nil, true\n\t}\n\tflowmatic.ManageTasks(5, task, manager, 0, 1)\n\tif s := fmt.Sprint(m); s != \"map[1:text string]\" {\n\t\tt.Fatal(s)\n\t}\n\tif time.Since(start) < sleepTime {\n\t\tt.Fatal(\"didn't sleep enough\")\n\t}\n\tif !b {\n\t\tt.Fatal(\"didn't finish\")\n\t}\n}\n" }, { "path": "map.go", "content": "package flowmatic\n\nimport (\n\t\"context\"\n)\n\n// Map starts numWorkers concurrent workers (or GOMAXPROCS workers if numWorkers < 1)\n// and attempts to map the input slice to an output slice.\n// Each task receives a child context.\n// The first error or panic returned by a task\n// cancels the child context\n// and halts further task scheduling.\n// If a task panics during execution,\n// the panic will be caught and rethrown in the parent Goroutine.\nfunc Map[Input, Output any](ctx context.Context, numWorkers int, items []Input, task func(context.Context, Input) (Output, error)) (results []Output, err error) {\n\tctx, cancel := context.WithCancel(ctx)\n\tdefer cancel()\n\n\tinch, ouch := TaskPool(numWorkers, func(pos int) (Output, error) {\n\t\titem := items[pos]\n\t\treturn task(ctx, item)\n\t})\n\n\tvar panicVal any\n\tn := 0\n\tcloseinch := false\n\tresults = make([]Output, len(items))\n\n\tfor {\n\t\tif n >= len(items) {\n\t\t\tcloseinch = true\n\t\t}\n\t\tif closeinch && inch != nil {\n\t\t\tclose(inch)\n\t\t\tinch = nil\n\t\t}\n\t\tselect {\n\t\tcase inch <- n:\n\t\t\tn++\n\t\tcase r, ok := <-ouch:\n\t\t\tif !ok {\n\t\t\t\tif panicVal != nil {\n\t\t\t\t\tpanic(panicVal)\n\t\t\t\t}\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn nil, err\n\t\t\t\t}\n\t\t\t\treturn results, nil\n\t\t\t}\n\t\t\tif r.Err != nil && err == nil {\n\t\t\t\tcancel()\n\t\t\t\tcloseinch = true\n\t\t\t\terr = r.Err\n\t\t\t}\n\t\t\tif r.Panic != nil && panicVal == nil {\n\t\t\t\tcancel()\n\t\t\t\tcloseinch = true\n\t\t\t\tpanicVal = r.Panic\n\t\t\t}\n\t\t\tresults[r.In] = r.Out\n\t\t}\n\t}\n}\n" }, { "path": "map_example_test.go", "content": "package flowmatic_test\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"os\"\n\t\"strconv\"\n\n\t\"github.com/carlmjohnson/flowmatic\"\n)\n\nvar (\n\tWeb = fakeSearch(\"web\")\n\tImage = fakeSearch(\"image\")\n\tVideo = fakeSearch(\"video\")\n)\n\ntype Result string\ntype Search func(ctx context.Context, query string) (Result, error)\n\nfunc fakeSearch(kind string) Search {\n\treturn func(_ context.Context, query string) (Result, error) {\n\t\treturn Result(fmt.Sprintf(\"%s result for %q\", kind, query)), nil\n\t}\n}\n\nfunc Google(ctx context.Context, query string) ([]Result, error) {\n\tsearches := []Search{Web, Image, Video}\n\treturn flowmatic.Map(ctx, flowmatic.MaxProcs, searches,\n\t\tfunc(ctx context.Context, search Search) (Result, error) {\n\t\t\treturn search(ctx, query)\n\t\t})\n}\n\nfunc ExampleMap() {\n\t// Compare to https://pkg.go.dev/golang.org/x/sync/errgroup#example-Group-Parallel\n\t// and https://pkg.go.dev/sync#example-WaitGroup\n\tresults, err := Google(context.Background(), \"golang\")\n\tif err != nil {\n\t\tfmt.Fprintln(os.Stderr, err)\n\t\treturn\n\t}\n\n\tfor _, result := range results {\n\t\tfmt.Println(result)\n\t}\n\n\t// Output:\n\t// web result for \"golang\"\n\t// image result for \"golang\"\n\t// video result for \"golang\"\n}\n\nfunc ExampleMap_simple() {\n\tctx := context.Background()\n\n\t// Start with some slice of input work\n\tinput := []string{\"0\", \"1\", \"42\", \"1337\"}\n\t// Have a task that takes a context\n\tdecodeAndDouble := func(ctx context.Context, s string) (int, error) {\n\t\t// Do some work\n\t\tn, err := strconv.Atoi(s)\n\t\tif err != nil {\n\t\t\treturn 0, err\n\t\t}\n\t\t// Return early if context was canceled\n\t\tif ctx.Err() != nil {\n\t\t\treturn 0, ctx.Err()\n\t\t}\n\t\t// Do more work\n\t\treturn 2 * n, nil\n\t}\n\t// Concurrently process input into output\n\toutput, err := flowmatic.Map(ctx, flowmatic.MaxProcs, input, decodeAndDouble)\n\tif err != nil {\n\t\tfmt.Println(err)\n\t}\n\tfmt.Println(output)\n\t// Output:\n\t// [0 2 84 2674]\n}\n" }, { "path": "map_test.go", "content": "package flowmatic_test\n\nimport (\n\t\"context\"\n\t\"errors\"\n\t\"testing\"\n\n\t\"github.com/carlmjohnson/flowmatic\"\n)\n\nfunc TestMap(t *testing.T) {\n\tctx := context.Background()\n\ta := errors.New(\"a\")\n\tb := errors.New(\"b\")\n\to, errs := flowmatic.Map(ctx, 1, []int{1, 2, 3}, func(_ context.Context, i int) (int, error) {\n\t\tswitch i {\n\t\tcase 1:\n\t\t\treturn 1, a\n\t\tcase 2:\n\t\t\treturn 2, b\n\t\tdefault:\n\t\t\tpanic(\"should be canceled by now!\")\n\t\t}\n\t})\n\tif !errors.Is(errs, a) {\n\t\tt.Fatal(errs)\n\t}\n\tif errors.Is(errs, b) {\n\t\tt.Fatal(errs)\n\t}\n\tif o != nil {\n\t\tt.Fatal(o)\n\t}\n}\n" }, { "path": "panic_test.go", "content": "package flowmatic_test\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"sync/atomic\"\n\t\"testing\"\n\n\t\"github.com/carlmjohnson/flowmatic\"\n)\n\nfunc try(f func()) (r any) {\n\tdefer func() {\n\t\tr = recover()\n\t}()\n\tf()\n\treturn\n}\n\nfunc TestManageTasks_panic(t *testing.T) {\n\ttask := func(n int) (int, error) {\n\t\tif n == 3 {\n\t\t\tpanic(\"3!!\")\n\t\t}\n\t\treturn n * 3, nil\n\t}\n\tvar triples []int\n\tmanager := func(n, triple int, err error) ([]int, bool) {\n\t\ttriples = append(triples, triple)\n\t\treturn nil, true\n\t}\n\tr := try(func() {\n\t\tflowmatic.ManageTasks(1, task, manager, 1, 2, 3, 4)\n\t})\n\tif r == nil {\n\t\tt.Fatal(\"should have panicked\")\n\t}\n\tif r != \"3!!\" {\n\t\tt.Fatal(r)\n\t}\n\tif fmt.Sprint(triples) != \"[3 6]\" {\n\t\tt.Fatal(triples)\n\t}\n}\n\nfunc TestEach_panic(t *testing.T) {\n\tvar (\n\t\tn atomic.Int64\n\t\terr error\n\t)\n\tr := try(func() {\n\t\terr = flowmatic.Each(1, []int64{1, 2, 3},\n\t\t\tfunc(delta int64) error {\n\t\t\t\tif delta == 2 {\n\t\t\t\t\tpanic(\"boom\")\n\t\t\t\t}\n\t\t\t\tn.Add(delta)\n\t\t\t\treturn nil\n\t\t\t})\n\t})\n\tif err != nil {\n\t\tt.Fatal(\"should have panicked\")\n\t}\n\tif r == nil {\n\t\tt.Fatal(\"should have panicked\")\n\t}\n\tif r != \"boom\" {\n\t\tt.Fatal(r)\n\t}\n\tif n.Load() != 4 {\n\t\tt.Fatal(n.Load())\n\t}\n}\n\nfunc TestDo_panic(t *testing.T) {\n\tvar (\n\t\tn atomic.Int64\n\t\terr error\n\t)\n\tr := try(func() {\n\t\terr = flowmatic.Do(\n\t\t\tfunc() error {\n\t\t\t\tn.Add(1)\n\t\t\t\treturn nil\n\t\t\t},\n\t\t\tfunc() error {\n\t\t\t\tpanic(\"boom\")\n\t\t\t},\n\t\t\tfunc() error {\n\t\t\t\tn.Add(1)\n\t\t\t\treturn nil\n\t\t\t})\n\t})\n\tif err != nil {\n\t\tt.Fatal(\"should have panicked\")\n\t}\n\tif r == nil {\n\t\tt.Fatal(\"should have panicked\")\n\t}\n\tif r != \"boom\" {\n\t\tt.Fatal(r)\n\t}\n\tif n.Load() != 2 {\n\t\tt.Fatal(n.Load())\n\t}\n}\n\nfunc TestRace_panic(t *testing.T) {\n\tvar (\n\t\tn atomic.Int64\n\t\terr error\n\t)\n\tr := try(func() {\n\t\terr = flowmatic.Race(context.Background(),\n\t\t\tfunc(context.Context) error {\n\t\t\t\tn.Add(1)\n\t\t\t\treturn nil\n\t\t\t},\n\t\t\tfunc(context.Context) error {\n\t\t\t\tpanic(\"boom\")\n\t\t\t},\n\t\t\tfunc(context.Context) error {\n\t\t\t\tn.Add(1)\n\t\t\t\treturn nil\n\t\t\t})\n\t})\n\tif err != nil {\n\t\tt.Fatal(\"should have panicked\")\n\t}\n\tif r == nil {\n\t\tt.Fatal(\"should have panicked\")\n\t}\n\tif r != \"boom\" {\n\t\tt.Fatal(r)\n\t}\n\tif n.Load() != 2 {\n\t\tt.Fatal(n.Load())\n\t}\n}\n\nfunc TestAll_panic(t *testing.T) {\n\tvar (\n\t\tn atomic.Int64\n\t\terr error\n\t)\n\tr := try(func() {\n\t\terr = flowmatic.All(context.Background(),\n\t\t\tfunc(context.Context) error {\n\t\t\t\tn.Add(1)\n\t\t\t\treturn nil\n\t\t\t},\n\t\t\tfunc(context.Context) error {\n\t\t\t\tpanic(\"boom\")\n\t\t\t},\n\t\t\tfunc(context.Context) error {\n\t\t\t\tn.Add(1)\n\t\t\t\treturn nil\n\t\t\t})\n\t})\n\tif err != nil {\n\t\tt.Fatal(\"should have panicked\")\n\t}\n\tif r == nil {\n\t\tt.Fatal(\"should have panicked\")\n\t}\n\tif r != \"boom\" {\n\t\tt.Fatal(r)\n\t}\n\tif n.Load() != 2 {\n\t\tt.Fatal(n.Load())\n\t}\n}\n\nfunc TestMap_panic(t *testing.T) {\n\tvar (\n\t\terr error\n\t\to []int64\n\t)\n\tctx := context.Background()\n\tr := try(func() {\n\t\to, err = flowmatic.Map(ctx, 1, []int64{1, 2, 3},\n\t\t\tfunc(_ context.Context, delta int64) (int64, error) {\n\t\t\t\tif delta == 2 {\n\t\t\t\t\tpanic(\"boom\")\n\t\t\t\t}\n\t\t\t\treturn 2 * delta, nil\n\t\t\t})\n\t})\n\tif err != nil {\n\t\tt.Fatal(\"should have panicked\")\n\t}\n\tif r == nil {\n\t\tt.Fatal(\"should have panicked\")\n\t}\n\tif r != \"boom\" {\n\t\tt.Fatal(r)\n\t}\n\tif o != nil {\n\t\tt.Fatal(o)\n\t}\n}\n" }, { "path": "race.go", "content": "package flowmatic\n\nimport (\n\t\"context\"\n\t\"errors\"\n\t\"sync/atomic\"\n)\n\n// Race runs each task concurrently\n// and waits for them all to finish.\n// Each function receives a child context\n// which is canceled once one function has successfully completed or panicked.\n// Race returns nil\n// if at least one function completes without an error.\n// If all functions return an error,\n// Race returns a multierror containing all the errors.\n// If a function panics during execution,\n// a panic will be caught and rethrown in the parent Goroutine.\nfunc Race(ctx context.Context, tasks ...func(context.Context) error) error {\n\tctx, cancel := context.WithCancel(ctx)\n\tdefer cancel()\n\terrs := make([]error, len(tasks))\n\tvar success atomic.Bool\n\t_ = eachN(len(tasks), len(tasks), func(pos int) error {\n\t\tdefer func() {\n\t\t\tpanicVal := recover()\n\t\t\tif panicVal != nil {\n\t\t\t\tcancel()\n\t\t\t\tpanic(panicVal)\n\t\t\t}\n\t\t}()\n\t\terr := tasks[pos](ctx)\n\t\tif err != nil {\n\t\t\terrs[pos] = err\n\t\t\treturn nil\n\t\t}\n\t\tcancel()\n\t\tsuccess.Store(true)\n\t\treturn nil\n\t})\n\tif success.Load() {\n\t\treturn nil\n\t}\n\treturn errors.Join(errs...)\n}\n" }, { "path": "race_example_test.go", "content": "package flowmatic_test\n\nimport (\n\t\"context\"\n\t\"errors\"\n\t\"fmt\"\n\t\"time\"\n\n\t\"github.com/carlmjohnson/flowmatic\"\n)\n\nfunc ExampleRace() {\n\tctx := context.Background()\n\tstart := time.Now()\n\terr := flowmatic.Race(ctx,\n\t\tfunc(ctx context.Context) error {\n\t\t\t// This task sleeps for only 1ms\n\t\t\td := 1 * time.Millisecond\n\t\t\ttime.Sleep(d)\n\t\t\tfmt.Println(\"slept for\", d)\n\t\t\treturn nil\n\t\t},\n\t\tfunc(ctx context.Context) error {\n\t\t\t// This task wants to sleep for a whole minute.\n\t\t\td := 1 * time.Minute\n\t\t\t// But sleepFor is a cancelable time.Sleep.\n\t\t\t// So when the other task completes,\n\t\t\t// it cancels this one, causing it to return early.\n\t\t\tif !sleepFor(ctx, d) {\n\t\t\t\tfmt.Println(\"canceled\")\n\t\t\t}\n\t\t\t// The error here is ignored\n\t\t\t// because the other task succeeded\n\t\t\treturn errors.New(\"ignored\")\n\t\t},\n\t)\n\t// Err is nil as long as one task succeeds\n\tfmt.Println(\"err:\", err)\n\tfmt.Println(\"exited early?\", time.Since(start) < 10*time.Millisecond)\n\t// Output:\n\t// slept for 1ms\n\t// canceled\n\t// err: \n\t// exited early? true\n}\n\nfunc ExampleRace_fakeRequest() {\n\t// Setup fake requests\n\trequest := func(ctx context.Context, page string) (string, error) {\n\t\tvar sleepLength time.Duration\n\t\tswitch page {\n\t\tcase \"A\":\n\t\t\tsleepLength = 10 * time.Millisecond\n\t\tcase \"B\":\n\t\t\tsleepLength = 100 * time.Millisecond\n\t\tcase \"C\":\n\t\t\tsleepLength = 10 * time.Second\n\t\t}\n\t\tif !sleepFor(ctx, sleepLength) {\n\t\t\treturn \"\", ctx.Err()\n\t\t}\n\t\treturn \"got \" + page, nil\n\t}\n\tctx := context.Background()\n\t// Make variables to hold responses\n\tvar pageA, pageB, pageC string\n\t// Race the requests to see who can answer first\n\terr := flowmatic.Race(ctx,\n\t\tfunc(ctx context.Context) error {\n\t\t\tvar err error\n\t\t\tpageA, err = request(ctx, \"A\")\n\t\t\treturn err\n\t\t},\n\t\tfunc(ctx context.Context) error {\n\t\t\tvar err error\n\t\t\tpageB, err = request(ctx, \"B\")\n\t\t\treturn err\n\t\t},\n\t\tfunc(ctx context.Context) error {\n\t\t\tvar err error\n\t\t\tpageC, err = request(ctx, \"C\")\n\t\t\treturn err\n\t\t},\n\t)\n\tfmt.Println(\"err:\", err)\n\tfmt.Printf(\"A: %q B: %q C: %q\\n\", pageA, pageB, pageC)\n\t// Output:\n\t// err: \n\t// A: \"got A\" B: \"\" C: \"\"\n}\n" }, { "path": "race_test.go", "content": "package flowmatic_test\n\nimport (\n\t\"context\"\n\t\"errors\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/carlmjohnson/flowmatic\"\n)\n\nfunc sleepFor(ctx context.Context, d time.Duration) bool {\n\ttimer := time.NewTimer(d)\n\tdefer timer.Stop()\n\tselect {\n\tcase <-timer.C:\n\t\treturn true\n\tcase <-ctx.Done():\n\t\treturn false\n\t}\n}\n\nfunc TestRace_join_errs(t *testing.T) {\n\tvar (\n\t\ta = errors.New(\"a\")\n\t\tb = errors.New(\"b\")\n\t)\n\n\terr := flowmatic.Race(context.Background(),\n\t\tfunc(ctx context.Context) error {\n\t\t\tif !sleepFor(ctx, 10*time.Millisecond) {\n\t\t\t\treturn ctx.Err()\n\t\t\t}\n\t\t\treturn a\n\t\t},\n\t\tfunc(ctx context.Context) error {\n\t\t\tif !sleepFor(ctx, 30*time.Millisecond) {\n\t\t\t\treturn ctx.Err()\n\t\t\t}\n\t\t\treturn b\n\t\t},\n\t)\n\tif !errors.Is(err, a) || !errors.Is(err, b) {\n\t\tt.Fatal(err)\n\t}\n\tif errors.Is(err, context.Canceled) {\n\t\tt.Fatal(err)\n\t}\n}\n" }, { "path": "taskpool.go", "content": "package flowmatic\n\nimport (\n\t\"runtime\"\n\t\"sync\"\n)\n\n// Result is the type returned by the output channel of TaskPool.\ntype Result[Input, Output any] struct {\n\tIn Input\n\tOut Output\n\tErr error\n\tPanic any\n}\n\n// TaskPool starts numWorkers workers (or GOMAXPROCS workers if numWorkers < 1) which consume\n// the in channel, execute task, and send the Result on the out channel.\n// Callers should close the in channel to stop the workers from waiting for tasks.\n// The out channel will be closed once the last result has been sent.\nfunc TaskPool[Input, Output any](numWorkers int, task Task[Input, Output]) (in chan<- Input, out <-chan Result[Input, Output]) {\n\tif numWorkers < 1 {\n\t\tnumWorkers = runtime.GOMAXPROCS(0)\n\t}\n\tinch := make(chan Input)\n\touch := make(chan Result[Input, Output], numWorkers)\n\tvar wg sync.WaitGroup\n\twg.Add(numWorkers)\n\tfor i := 0; i < numWorkers; i++ {\n\t\tgo func() {\n\t\t\tdefer wg.Done()\n\t\t\tfor inval := range inch {\n\t\t\t\tfunc() {\n\t\t\t\t\tdefer func() {\n\t\t\t\t\t\tpval := recover()\n\t\t\t\t\t\tif pval == nil {\n\t\t\t\t\t\t\treturn\n\t\t\t\t\t\t}\n\t\t\t\t\t\touch <- Result[Input, Output]{\n\t\t\t\t\t\t\tIn: inval,\n\t\t\t\t\t\t\tPanic: pval,\n\t\t\t\t\t\t}\n\t\t\t\t\t}()\n\n\t\t\t\t\toutval, err := task(inval)\n\t\t\t\t\touch <- Result[Input, Output]{inval, outval, err, nil}\n\t\t\t\t}()\n\t\t\t}\n\t\t}()\n\t}\n\tgo func() {\n\t\twg.Wait()\n\t\tclose(ouch)\n\t}()\n\treturn inch, ouch\n}\n" }, { "path": "taskpool_example_test.go", "content": "package flowmatic_test\n\nimport (\n\t\"context\"\n\t\"crypto/md5\"\n\t\"fmt\"\n\t\"log\"\n\t\"os\"\n\t\"path/filepath\"\n\n\t\"github.com/carlmjohnson/flowmatic\"\n)\n\nfunc ExampleTaskPool() {\n\t// Compare to https://pkg.go.dev/golang.org/x/sync/errgroup#example-Group-Pipeline and https://blog.golang.org/pipelines\n\n\tm, err := MD5All(context.Background(), \"testdata/md5all\")\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\n\tfor k, sum := range m {\n\t\tfmt.Printf(\"%s:\\t%x\\n\", k, sum)\n\t}\n\n\t// Output:\n\t// testdata/md5all/hello.txt:\tbea8252ff4e80f41719ea13cdf007273\n}\n\n// MD5All reads all the files in the file tree rooted at root\n// and returns a map from file path to the MD5 sum of the file's contents.\n// If the directory walk fails or any read operation fails,\n// MD5All returns an error.\nfunc MD5All(ctx context.Context, root string) (map[string][md5.Size]byte, error) {\n\t// Make a pool of 20 digesters\n\tin, out := flowmatic.TaskPool(20, digest)\n\n\tm := make(map[string][md5.Size]byte)\n\t// Open two goroutines:\n\t// one for reading file names by walking the filesystem\n\t// one for recording results from the digesters in a map\n\terr := flowmatic.All(ctx,\n\t\tfunc(ctx context.Context) error {\n\t\t\treturn walkFilesystem(ctx, root, in)\n\t\t},\n\t\tfunc(ctx context.Context) error {\n\t\t\tfor r := range out {\n\t\t\t\tif r.Err != nil {\n\t\t\t\t\treturn r.Err\n\t\t\t\t}\n\t\t\t\tm[r.In] = *r.Out\n\t\t\t}\n\t\t\treturn nil\n\t\t},\n\t)\n\n\treturn m, err\n}\n\nfunc walkFilesystem(ctx context.Context, root string, in chan<- string) error {\n\tdefer close(in)\n\n\treturn filepath.Walk(root, func(path string, info os.FileInfo, err error) error {\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\tif !info.Mode().IsRegular() {\n\t\t\treturn nil\n\t\t}\n\t\tselect {\n\t\tcase in <- path:\n\t\tcase <-ctx.Done():\n\t\t\treturn ctx.Err()\n\t\t}\n\n\t\treturn nil\n\t})\n}\n\nfunc digest(path string) (*[md5.Size]byte, error) {\n\tdata, err := os.ReadFile(path)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\thash := md5.Sum(data)\n\treturn &hash, nil\n}\n" }, { "path": "testdata/md5all/hello.txt", "content": "Hello, World!\n" } ]