[
{
"path": ".github/FUNDING.yml",
"content": "# These are supported funding model platforms\n\ngithub: Arp-G\n"
},
{
"path": "LICENSE",
"content": "MIT License\n\nCopyright (c) 2023 Async Elixir\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
},
{
"path": "README.md",
"content": "# Async Elixir ๐ฎ\n\nWelcome to the **Async Elixir** book repository!\n\nThe **Async Elixir** book is a deep dive into Elixir's concurrency features. If you're already comfortable with Elixir basics and eager to explore concurrent programming and process management, you're in the right place.\n\n[](https://livebook.dev/run?url=https%3A%2F%2Fgithub.com%2FArp-G%2Fasync-elixir%2Fblob%2Fmaster%2Fchapters%2Fch_0.0_start.livemd)\n\n## Getting Started\n\n* Clone this repository\n* Ensure you have [Livebook](https://livebook.dev/) installed\n* Open livebook and then open the [Course Overview](chapters/ch_0.0_start.livemd) file in Livebook.\n\n## Contributions\n\nIf you encounter any issues, find typos, or have valuable suggestions to improve the course content, don't hesitate to create an issue in this repository. Your contributions are highly appreciated!\n"
},
{
"path": "chapters/ch_0.0_start.livemd",
"content": "# Async Elixir\n\n## Overview\n\n#### Welcome to Async Elixir\n\nWelcome to the **Async Elixir** course, a comprehensive exploration of Elixir's advanced concurrency features. This course is tailored for individuals with a foundational understanding of Elixir programming and a desire to elevate their expertise to the next level.\n\n#### Course Overview\n\nThis course is your gateway to becoming proficient in Elixir's asynchronous capabilities. Whether you're an experienced developer expanding your skill set or a curious learner fascinated by concurrent programming, you'll gain a deep understanding of Elixir's powerful asynchronous features.\n\nBy the conclusion of this course, you will have developed a robust understanding of processes, OTP patterns, and the Elixir standard library's utilization for effective process management. You'll be well-equipped to apply these concepts confidently in real-world situations.\n\n#### Key Topics Covered\n\n* **In-Depth Process and Concurrency:** Explore the intricate workings of Elixir processes and the art of achieving concurrency.\n\n* **Essential Process Management Concepts:** Learn vital concepts such as process linking, monitoring and more.\n\n* **Abstractions for Resilience and Control:** Navigate essential abstractions like GenServers, Supervisors, and core Elixir modules such as Task, Registry, and Agents.\n\n* **Applied Learning with a Project:** Put your knowledge into practice through a hands-on project, solidifying your grasp of asynchronous programming and process management.\n\n#### Course Focus\n\n* **Prerequisite Elixir Knowledge:** This course assumes a foundational familiarity with Elixir's syntax and concepts. It DOES NOT teach Elixir basics but instead focuses on the asynchronous capabilities of Elixir.\n\n* **Core Concepts, Not Libraries:** Our emphasis remains on core concepts. The course doesn't cover libraries like Phoenix or Phoenix LiveView. Instead, it equips you with a strong foundation to comprehend such libraries more effectively.\n\n### Prerequisites\n\nBefore diving into the course, make sure you have the following:\n\n* Installed [Elixir](https://elixir-lang.org/install.html)\n* Installed [Livebook](https://livebook.dev/)\n* A basic understanding of Elixir syntax and the ability to write code in Elixir. If you need a refresher, you can review Elixir basics [here](https://elixir-lang.org/getting-started/introduction.html).\n\n[Livebook](https://livebook.dev/) is an excellent tool for learning Elixir and experimenting with its concepts. Learn more about Livebook [here](https://github.com/livebook-dev/livebook).\n\n#### About me\n\nI am a full-stack software engineer with four years of focused experience in Elixir programming. Over this period, I have actively engaged in various Elixir projects, extensively delved into Elixir literature, including books and blogs, and honed my skills in this powerful language. My fascination with Elixir's process-oriented architecture has inspired me to condense my insights into this course.\n\n## Table of Contents\n\n#### Part 1: Concurrency and Processes\n\n* ๐ [Chapter 1.1: Concurrency in Elixir](ch_1.1_concurrency_in_elixir.livemd)\n* ๐ง [Chapter 1.2: Immutability and Memory Management](ch_1.2_immutability_and_memory_management.livemd)\n\n#### Part 2: Processes\n\n* ๐ [Chapter 2.1: Process Internals](ch_2.1_process_internals.livemd)\n* ๐ [Chapter 2.2: Process Basics](ch_2.2_process_basics.livemd)\n* ๐ [Chapter 2.3: Process Linking](ch_2.3_process_linking.livemd)\n* ๐ก [Chapter 2.4: Process Monitoring and Hibernation](ch_2.4_process_monitoring_and_hibernation.livemd)\n* ๐ [Chapter 2.5: Group Leaders and Process Naming](ch_2.5_group_leaders_and_process_naming.livemd)\n\n#### Part 3: GenServer\n\n* ๐งช [Chapter 3.1: GenServer Introduction](ch_3.1_genserver_introduction.livemd)\n* ๐๏ธ [Chapter 3.2: Building a GenServer](ch_3.2_building_a_genserver.livemd)\n* ๐ [Chapter 3.3: GenServer Examples](ch_3.3_genserver_examples.livemd)\n* ๐ง [Chapter 3.4: Other GenServer Functions](ch_3.4_other_genserver_functions.livemd)\n\n#### Part 4: Registry Module\n\n* ๐ [Chapter 4: The Registry Module](ch_4.0_the_registry_module.livemd)\n\n#### Part 5: Supervision\n\n* ๐ฅ [Chapter 5.1: Supervisors Introduction](ch_5.1_supervisors_introduction.livemd)\n* ๐ [Chapter 5.2: Supervision Strategies](ch_5.2_supervision_strategies.livemd)\n* ๐ [Chapter 5.3: Restart Strategies](ch_5.3_restart_strategies.livemd)\n* ๐ [Chapter 5.4: Introduction to Dynamic Supervisor](ch_5.4_introduction_to_dynamic_supervisor.livemd)\n* ๐๏ธ [Chapter 5.5: Partition Supervisor Example](ch_5.5_partition_supervisor.ex.livemd)\n* โ๏ธ [Chapter 5.6: Scaling with Dynamic Supervisor](ch_5.6_scaling_dynamic_supervisor.livemd)\n\n#### Part 6: Project: Building a Download Manager\n\n* ๐ ๏ธ [Chapter 6: Building a Download Manager](ch_6.0_project_building_a_download_manager.livemd)\n\n#### Part 7: Tasks\n\n* โ๏ธ [Chapter 7.1: Introduction to Tasks](ch_7.1_intro_to_tasks.livemd)\n* ๐ [Chapter 7.2: Awaiting Tasks](ch_7.2_awaiting_tasks.livemd)\n* ๐ [Chapter 7.3: Task Async Stream](ch_7.3_task_async_stream.livemd)\n* ๐ก๏ธ [Chapter 7.4: Supervised Tasks](ch_7.4_supervised_tasks.livemd)\n\n#### Part 8: Agents\n\n* ๐ค [Chapter 8: Agents](ch_8.0_agents.livemd)\n\n#### Part 9: Misc\n\n* ๐ก [Chapter 9: Gotchas](ch_9.0_gotchas.livemd)\n\n---\n\n\n\nSo without further ado, let's [get started](ch_1.1_concurrency_in_elixir.livemd)... ๐\n"
},
{
"path": "chapters/ch_1.1_concurrency_in_elixir.livemd",
"content": "\n\n# Concurrency in Elixir\n\n## Navigation\n\n\n\n## Concurrency vs Parallelism\n\nConcurrency and parallelism are often used interchangeably but are two distinct concepts. Concurrency refers to the execution of multiple tasks that overlap in time, with each task being interrupted and resumed intermittently by the CPU(context switching). This can create an illusion of tasks running simultaneously, but in reality, they are taking turns executing on a single time-sliced CPU.\n\nParallelism, on the other hand, involves the simultaneous execution of multiple tasks on a hardware system that has multiple computing resources, such as a multi-core CPU. Parallelism allows tasks to run literally at the same time, without having to share CPU time.\n\nIn essence, concurrency deals with handling multiple tasks at once, while parallelism deals with actually performing multiple tasks at the same time. While a system can exhibit both concurrency and parallelism, it is possible to have a concurrent system that is not parallel.\n\n\n\n\n\n\n\nIn todays world with machines having power multi-core CPUs writing code that can run cocurrently and parallely can lead to huge performance benifits. We should strive to leverage the capabilities of modern hardware advancements by writing code that can fully utilize them.\n\nFurthermore, as we develop software, we often encounter problems that require background tasks and can benefit greatly from the use of parallel programming. Examples of such tasks include image processing, video transcoding, and make third party api calls, to name a few.\n\n## Concurrency and parallelism in Elixir\n\nDue to the functional and immutable nature of Elixir writing parallel and concurrent code becomes much simpler. Unlike many other languages that require locks and mutexes to handle issues related to shared state in parallel programming, Elixir's design mitigates these problems. As a result, parallel and concurrent code is a first-class citizen in Elixir, requiring less effort and complexity to implement effectively.\n\n\n\nIn Elixir, **the Erlang Virtual Machine (BEAM)** serves as the backbone for managing concurrency and parallelism. Let's take a closer look at how it works under the hood to provide us with these superpowers.\n\nElixir leverages lightweight processes that are expertly managed by the VM. These processes are not true OS processes, making them highly lightweight and allowing for thousands or even [millions](https://phoenixframework.org/blog/the-road-to-2-million-websocket-connections) of them to run concurrently without impacting performance.\n\nThis lightweight process model has given rise to several powerful applications, including the [Cowboy web server](https://github.com/ninenines/cowboy) which creates a process for every incoming web request to keep heavy work or errors within a single request from affecting others. Other examples include [Phoenix Channels](https://hexdocs.pm/phoenix/channels.html) and [Phoenix live view](https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html) that employ an Erlang process per WebSocket connection.\n\n## Processes in Elixir\n\nIn Elixir and Erlang, the term \"processes\" does not refer to operating system processes or threads. Instead, they are akin to [green threads](https://en.wikipedia.org/wiki/Green_thread#:~:text=In%20computer%20programming%2C%20a%20green,underlying%20operating%20system%20(OS).) or actors. These processes run concurrently on a single-core CPU and in parallel on multiple cores, managed and scheduled by the Erlang Virtual Machine.\n\nSurprisingly, each process in Elixir and Erlang requires only around 300 words of memory and takes microseconds to start, making them incredibly lightweight. In fact, within the Erlang Virtual Machine, every entity executing code operates within a process.\n\nFor instance, in [Phoenix](https://www.phoenixframework.org/), when making a regular HTTP request using Phoenix controllers, the corresponding connection is allocated its own process. This process is swiftly terminated once the response is sent and the connection is closed. In [LiveView](https://github.com/phoenixframework/phoenix_live_view) we keep that process alive since we work with websockets.\n\nEach process is capable of executing code and possesses a **first-in-first-out** mailbox to which other processes can send messages. Likewise, it can send messages to other processes. Processes in Elixir and Erlang are inherently **sequential**, meaning they handle one message at a time.\n\nSimilar to an operating system scheduler, the Erlang VM has the ability to start, pause, or preempt work as needed (In computing, preemption is the act of temporarily interrupting an executing task, with the intention of resuming it at a later time).\n\nWhile waiting for a message, a process is completely ignored by the scheduler. As a result, **idle processes do not consume any system resources**.\n\n#### Reductions\n\nErlang uses \"reductions\" as work units to decide when a process might be paused. A reduction in Erlang is a unit of work done by BEAM, including tasks like function application, arithmetic operations, and message passing. The scheduler monitors reductions for each process, pausing a process once it reaches a set reduction count, which lets another process take its turn to run. This ensures fairness in scheduling by preventing processes from hogging the CPU.\n\nAdditionally, reductions are applied flexibly based on the operation type. For instance, I/O operations consume reductions differently, allowing the scheduler to handle various operations effectively. Unlike traditional blocking I/O, Erlang's non-blocking model lets processes continue working during I/O waits, improving overall system performance.\n\n\n\n#### Scheduling in BEAM\n\nBEAM, the underlying virtual machine, employs a single OS thread per core, and each thread runs its own scheduler. Every scheduler is responsible for pulling processes from its own run queue, with the BEAM being responsible for populating these queues with Erlang processes for execution.\n\n(Note: To utilize more than one core the Erlang Runtime System Application(ERTS) has to be built in SMP mode. SMP stands for Symmetric MultiProcessing, that is, the ability to execute a processes on any one of multiple CPUs.)\n\nThe scheduler manages two queues: a ready queue containing processes that are prepared to run and a waiting queue containing processes that are waiting to receive a message.\n\nWhen a process is selected from the ready queue, it is handed over to BEAM for the execution of one CPU time slice. BEAM interrupts the running process and places it at the end of the ready queue when the time slice expires. However, if the process is blocked in a receive operation before the time slice runs out, it is added to the waiting queue.\n\n###### Loadbalancer\n\nA load balancer is also in place, responsible for executing migration logic to allocate processes across the run queues on separate cores. This logic assists in maintaining load balance by taking jobs away from overloaded queues (known as [task stealing](https://blog.stenmans.org/theBeamBook/#_task_stealing)) and assigning them to empty or underloaded queues (known as [\"task migration\"](https://blog.stenmans.org/theBeamBook/#_migration)).\n\nIn simpler terms, if one scheduler's queue becomes crowded due to processes taking an extended time, other schedulers step in to distribute the workload more evenly. For example, if a process accumulates a high number of function calls (reductions), without completing, the scheduler will preemptively pause it which means freeze it, mid-run, and send it back to the end of the work queue. This **preemptive multitasking** approach ensures that no single task can monopolize the system for an extended period, ensuring consistently **low latency**, a key feature of the BEAM.\n\nThe load balancer strives to maintain an equal maximum number of run-able processes across schedulers.\n\nLooking beyond Erlang's internal run queues, the operating system also manages the scheduling of threads onto CPU cores at an OS level. This means that processes can not only swap within Erlang's run queue but also undergo complete context switches or be relocated to different cores by the OS.\n\n\n\n\n\n\n\nYou can find the numer of schedulers in your IEX session using the `System.schedulers/0` function.\n\n```elixir\n# Returns the number of schedulers in the VM.\nSystem.schedulers()\n\n# Returns the number of schedulers online in the VM.\n# Here online means total number of schedulers which are active and actually being used.\nSystem.schedulers_online()\n```\n\n#### Process priority\n\nErlang's priority system has four levels: low, normal, high, and max (reserved for Erlang's internal use). Each level has its own run queue and follows a round-robin scheduling method, except for max.\n\nProcesses in max or high priority queues are executed exclusively, blocking lower-priority processes until they're done. This design emphasizes efficiency for critical tasks but can cause bottlenecks if high-priority processes are overused, impacting overall application responsiveness.\n\nLow and normal queues are more flexible, allowing interleaved execution without blocking each other. However, using high priority sparingly is crucial to avoid performance issues.\n\nAdditionally, Erlang permits communication across different priority levels, although a high-priority process waiting for a message from a lower-priority one will effectively lower its own priority.\n\nProcess priority can be changed in elixir using `Process.flag(:priority, :high)`\n\n## Resources\n\n* https://medium.com/flatiron-labs/elixir-and-the-beam-how-concurrency-really-works-3cc151cddd61\n* https://blog.stenmans.org/theBeamBook/#CH-Scheduling\n* https://blog.appsignal.com/2024/04/23/deep-diving-into-the-erlang-scheduler.html\n* https://fly.io/phoenix-files/a-liveview-is-a-process/\n* https://underjord.io/unpacking-elixir-concurrency.html\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_1.2_immutability_and_memory_management.livemd",
"content": "# Immutability and memory management\n\n## Navigation\n\n\n\n## Immutability in elixir\n\nIn Elixir, variables function as **labels** that refer to specific values. These values are immutable, meaning they cannot be changed. However, the label or variable can be reassigned to a different value. This provides the flexibility to bind the same value to multiple labels or variables.\n\nIn Erlang, it is not possible to reassign or rebind a variable. Attempting to do so will result in an error, as demonstrated in the following code:\n\n```erlang\nX = 5,\nX = X * 10. % throws an exception error: no match of right hand side value 50\n```\n\nThe error in the Erlang code occurs because the variable X is initially assigned to the value of 5, but then an attempt is made to reassign it to a new value (i.e. X * 10). Since variables in Erlang are immutable, this operation is not allowed and the code will fail to compile.\n\nOn the other hand, Elixir allows for rebinding of values, which makes the same code valid in Elixir. This is because in Erlang, **the = operator functions as a match operator rather than an assignment operator**.\n\nTherefore, in the Erlang code `X = X * 10`, the left-hand side of the match (X) is already bound to the value of 5, and trying to match it with the right-hand side (X * 10) which evaluates to 50, results in a mismatch.\n\nIn Elixir, the ^ (pin) operator can be used to force a match, while the assignment operation uses the regular = operator. For example:\n\n```\niex(1)> x = 5\n5\niex(2)> x = x * 10 # Assignment\n50\niex(3)> ^x = x * 10 # Matching\n** (MatchError) no match of right hand side value: 500\n (stdlib 4.0.1) erl_eval.erl:496: :erl_eval.expr/6\n iex:3: (file)\n```\n\nIn Elixir, any input passed into a function to be transformed creates a new value without modifying the original value. This allows for safe concurrent access to the same data by multiple processes. Since there is **no shared memory** that is getting mutated by multiple processes, concurrency is easier to manage. Any transformation on the original data will result in new data being created. Processes do not share state, they can only communicate asynchronously through message passing. This ensures it is safe to run them at the same time.\n\n## Example of immutability & closures in elixir\n\n```elixir\nlist = [1, 2, 3, 4]\n\n# Returns a new list\nEnum.filter(list, fn num -> num > 2 end)\n\n# [1, 2, 3, 4] Original list remains unchanged\nlist\n```\n\n```elixir\nx = 1\n\n# An anonymous function\nanon = fn ->\n # Closure captures the value of x\n IO.puts(x)\n x = 0\nend\n\n# Outputs 1\nanon.()\n\n# Outputs 1\nIO.puts(x)\n\nx = 5\n\n# Outputs 1\nanon.()\n```\n\n## Persistent Datastructures\n\nAt this point you might be wandering that performing a full copy of the entire data whenever something changes would be an expensive and slow operation and lead to a high performance overhead.\n\nTo solve this problem there is a class of datastructures known as **persistent data structures**.\n\nPersistent data structures are data structures that allow for the efficient storage and retrieval of data even after multiple modifications or updates have been made. These data structures preserve the previous versions of data and allow for efficient access to those versions.\n\nA persistent data structure **maintains the previous versions of data** by using a technique known as structural sharing. Structural sharing allows the data structure to **share the unchanged parts of its structure** across multiple versions of the data, rather than copying the entire structure each time a modification is made. This sharing of unchanged structure makes persistent data structures efficient in both time and space.\n\nPersistent data structures are widely used in functional programming languages, where immutability is a core concept.\n\nUnder the hood, the BEAM leverages persistent data structures in order to provide\nimmutability as a first-class citizen while not having to copy the entire data structure\nany time something changes (with the exceptions of when data is passed between\nprocesses or when data is extracted from native data stores like ETS).\n\nFor example, in Elixir lists are actually linked lists.\nA linked list is a Tree with one branch.\n\n```\nElixir: list = [1, 2, 3, 4]\nTree: 1 -> 2 -> 3 -> 4\n\nEvery time you prepend an element to a list, it will share its tail:\nElixir: [0 | list]\nTree: 0 -> (1 -> 2 -> 3 -> 4)\n```\n\n## High level overview of garbage collection in elixir\n\nErlang employs a generational copying garbage collection system where each process has its own private heap. This heap is divided into two segments - the young and old generations. Newly allocated data resides in the young generation, while data that has survived multiple garbage collection cycles is stored in the old generation.\n\nThe young generation undergoes more frequent garbage collection, whereas the old generation is only garbage collected during a full sweep, which occurs after a certain number of generational GC cycles. It can also be collected if not enough memory is reclaimed or if manually invoked. This process is referred to as **soft real-time** garbage collection because it halts only the process undergoing GC without affecting other processes. This characteristic is well-suited for soft real-time systems, as the entire runtime doesn't need to pause.\n\nIn addition to this, Erlang implements [reference counting garbage collection](https://en.wikipedia.org/wiki/Garbage_collection_\\(computer_science\\)#Reference_counting) for the shared heap. Here, objects in the shared heap are assigned reference counters that keep track of the number of references held by other objects. When an object's reference count drops to zero, it's considered inaccessible and gets destroyed.\n\nFor a deeper dive into garbage collection, you can explore this informative [video](https://www.youtube.com/watch?v=OSdaXNQ0xhQ) and this insightful [book](https://hamidreza-s.github.io/erlang%20garbage%20collection%20memory%20layout%20soft%20realtime/2015/08/24/erlang-garbage-collection-details-and-why-it-matters.html).\n\n## Resources\n\n* https://stackoverflow.com/questions/30203227/does-elixir-have-persistent-data-structures-similar-to-clojure\n\n* https://elixirpatterns.dev/\n\n* https://gist.github.com/josevalim/ce2f5871a96b6cbcf2c1\n\n* https://elixirforum.com/t/how-would-you-explain-elixir-immutability/47323/11\n\n* [Video on garbage collection](https://www.youtube.com/watch?v=OSdaXNQ0xhQ)\n\n* [The beam book -garbage collection](https://hamidreza-s.github.io/erlang%20garbage%20collection%20memory%20layout%20soft%20realtime/2015/08/24/erlang-garbage-collection-details-and-why-it-matters.html)\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_2.1_process_internals.livemd",
"content": "# Process Internals\n\n## Navigation\n\n\n\n## What are processes?\n\nA process is a **self-contained** entity where code is executed. It safeguards the system from errors in our code by restricting the effects of the error to the process that is executing the faulty code. Processes have their own address space and can communicate with other processes via signals and messages, and their execution is managed by a preemptive scheduler.\n\nIt is important to note that Elixir processes are not the same as operating system processes. Elixir processes are remarkably **lightweight** in terms of memory and CPU usage, even when compared to threads in other programming languages. Therefore, it is not uncommon to run tens or even [hundreds of thousands](https://phoenixframework.org/blog/the-road-to-2-million-websocket-connections) of processes simultaneously.\n\n## Internals of a process\n\nLet's explore the structure of an Elixir process at a high level.\n\nAn Elixir process consists of four primary memory blocks: the **stack**, the **heap**, the message area (also known as the **mailbox**), and the **Process Control Block** (PCB). The stack is responsible for tracking program execution by storing return addresses, passing function arguments, and keeping local variables. The heap, on the other hand, stores larger structures such as lists and tuples.\n\nThe message area or mailbox is used to hold messages sent from other processes to the target process. The PCB maintains the state of the process, while the stack, heap, and mailbox are dynamically allocated and can grow or shrink based on usage. Conversely, the PCB is statically allocated and contains several fields that control the process.\n\n**Message passing** is the primary means of communication between Elixir processes. When one process sends a message to another, the message is copied from the sender's heap to the recipient's mailbox. In certain circumstances, such as when a process is suspended and no other processes are attempting to send it messages, the message may be directly copied to the recipient's mailbox. In other cases, the message is stored in an m-buf and moved to the heap after a garbage collection. M-bufs are variable-length heap fragments, and a process may have several m-bufs.\n\n(It is worth noting that in the early versions of Erlang, parallelism was not available, so only one process could execute at any given time. In such versions, the sending process could write directly to the heap of the receiving process. However, with the rise of multicore systems, message copying across process heaps is managed using locks and queues. To learn more about this topic, please see this [article](https://blog.stenmans.org/theBeamBook/#_the_process_of_sending_a_message_to_a_process).)\n\n## Resources\n\n* https://elixir-lang.org/getting-started/processes.html\n* https://hexdocs.pm/elixir/1.12/Process.html\n* https://www.erlang-solutions.com/blog/understanding-processes-for-elixir-developers/\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_2.2_process_basics.livemd",
"content": "# Process Basics\n\n## Navigation\n\n\n\n## Process introspection\n\nTo check processes we have in a running system: `:shell_default.i`.\n\nYou might notice that many processes have a heap size of 233, that is because it is the default starting heap size of a process.\n\nIf there is a large number for the heap size, then the process uses a lot of memory and if there is a large number for the reductions then the process has executed a lot of code.\n\nGet lot more infor about a process using `Process.info/1`.\n\n```elixir\nProcess.whereis(:code_server)\npid = Process.whereis(:code_server)\nProcess.info(pid)\n```\n\n[Process.info/2](http://Process.info/2) can be used to view additional info like backtrace `Process.info(pid, :backtrace)`\n\nThe [observer](https://elixir-lang.org/getting-started/debugging.html#observer) is also a great tool to observe processes.\n\n## Process Dictionary\n\nThere is actually one more memory area in a process where Erlang terms can be stored, the *Process Dictionary*.\n\nThe *Process Dictionary* (PD) is a process local key-value store. One advantage with this is that all keys and values are stored on the heap and there is no copying as with `send/2` or an ETS table.\n\n([ETS](https://elixirschool.com/en/lessons/storage/ets#overview-0) or Erlang Term Storage is a in-memory store for Elixir and Erlang objects that comes included. ETS is capable of storing large amounts of data and offers constant time data access. Tables in ETS are created and owned by individual processes. When an owner process terminates, its tables are destroyed)\n\n```elixir\n# Stores the given key-value pair in the process dictionary.\nProcess.put(:count, 1)\nProcess.put(:locale, \"en\")\n```\n\n```elixir\n# Returns the value for the given key in the process dictionary\nProcess.get(:count)\n```\n\n```elixir\n# Returns all keys in the process dictionary\nProcess.get_keys()\n```\n\n```elixir\n# Deletes the given key from the process dictionary\nProcess.delete(:count)\n```\n\n```elixir\n# Returns all key-value pairs in the process dictionary.\nProcess.get()\n```\n\n## Spawning processes\n\nThe most fundamental way to create processes in Elixir is by using the [spawn/1](https://hexdocs.pm/elixir/1.12/Kernel.html#spawn/1), [receive/1](https://hexdocs.pm/elixir/1.12/Kernel.SpecialForms.html#receive/1), and [send/2](https://hexdocs.pm/elixir/1.12/Kernel.html#send/2) functions. They enable us to spawn a process, wait for messages, and send messages to a process, respectively.\n\nMany higher-level abstractions, such as Task, GenServer, and Agent, are built on top of these primitive functions.\n\nThese functions are part of the [Kernel](https://hexdocs.pm/elixir/1.12/Kernel.html) module and are automatically imported, allowing us to call them directly without needing to use the `Kernel.` prefix.\n\nLet's take a look at some examples of their usage...\n\n```elixir\n# Spawn a process, by passing it a function to execute.\n# spawn/1 returns the pid (process identifier) of the spawed process\npid = spawn(fn -> IO.puts(\"Hello world\") end)\n\n# Once the process has finished excuting it will exit\nProcess.alive?(pid) |> IO.inspect()\n\n# Sleep for 100ms to wait for process to exit\n:timer.sleep(100)\n\nProcess.alive?(pid)\n```\n\nWhen spawning a process it goes through a lifecycle like so...\n\n\n\n```mermaid\nflowchart LR\nspawn --> NewProcess --> ExecuteCallbackFunction --> Dead\n```\n\n## Exchanging messages between processes\n\nTo exchange messages between processes in Elixir, we can use the `send/2` and `receive/1` functions.\n\nWhen a process uses `send/2` to send a message, it **doesn't block** - instead, the message is placed in the recipient's mailbox, and the sending process continues.\n\nOn the other hand, when a process uses `receive/1`, it blocks until a matching message is found in its mailbox. The call to `receive/1` searches the mailbox for a message that matches any of the given patterns.\n\n`receive/1` supports guards and multiple clauses, such as `case/2`.\n\nLet's look at an example of a process sending a message to itself.\n\n```elixir\n# Get the pid of the current process\nself_pid = self()\n\n# Send a message to the current process\nsend(self_pid, :ping)\n\n# Check messages in mailbox without consuming them\nProcess.info(self_pid, :messages) |> IO.inspect(label: \"Messages in mailbox\")\n\n# Recieve the message waiting in mailbox (consumes the message in the mailbox)\nreceive do\n :ping -> IO.puts(:pong)\nend\n\n# Check messages in mailbox again\nProcess.info(self_pid, :messages) |> IO.inspect(label: \"Messages in mailbox\")\n```\n\nAn optional after clause can be given in case the message was not received after the given timeout period, specified in milliseconds.\n(If timeout `0` is given then the message is expected to be already present in the mailbox.)\n\n```elixir\nreceive do\n {:message, message} when message in [:start, :stop] -> IO.puts(message)\n _ -> IO.puts(:stderr, \"Unexpected message received\")\nafter\n 1000 -> IO.puts(:stderr, \"Timeout, no message in 1 seconds\")\nend\n```\n\nIn the elixir IEx shell, we have a helper function flush/0 that flushes or consumes and prints all the messages in the mailbox of the shell process.\n\n```elixir\nsend(self(), :hello)\n\nProcess.info(self_pid, :messages) |> IO.inspect(label: \"Messages in mailbox before flush\")\n\n# In the iex shell we wont have to use the `IEx.Helpers,` prefix since these helpers functions are imported automatically\nIEx.Helpers.flush()\n\nProcess.info(self_pid, :messages) |> IO.inspect(label: \"Messages in mailbox after flush\")\n```\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_2.3_process_linking.livemd",
"content": "# Process Linking & Trapping Exists\n\n## Navigation\n\n\n\n## Process Linking\n\nIn Elixir, when we create a process, we have the option to link it to its parent process. This means that if the child process encounters an error and fails, the parent process will be notified.\n\nWhen we use the `spawn/1` function to create a process, it will not be linked to its parent process. As a result, if the child process encounters an error and fails, the parent process will not be notified.\n\nTo ensure that the parent process is notified of any errors in the child process, we can use the `spawn_link/1` function instead. This function creates a linked process, so if the child process crashes, the parent process will receive an EXIT signal.\n\nTo illustrate this, let's consider an example...\n\n```elixir\nunlinked_child_process = spawn(fn -> raise(\"BOOM! Unliked process crashed!\") end) |> IO.inspect()\n\nIO.inspect(Process.info(self(), :links))\n\n:timer.sleep(100)\n\nIO.puts(\"Parent process still alive!\")\n```\n\nIn the above example we can see that the parent process is still alive after the spawned process crashes. Lets see what happens if the processes were linked\n\n\n\n(Uncomment the code below and run it. After running it comment it out again.\nSince the code below crashes the live view process we need to comment it in order to run the rest of the code in this chapter.)\n\n```elixir\n# linked_child_process = spawn_link(fn -> \n# :timer.sleep(100)\n# raise(\"BOOM! Linked process crashed!\") \n# end)\n# |> IO.inspect(label: \"Linked process PID\")\n\n# IO.inspect Process.info(self(), :links)\n\n# :timer.sleep(200)\n\n# IO.puts \"Parent process still alive!\"\n\n# Child process <-> parent process <-> Livebook evaluation process\n```\n\nThis time the print statement \"Parent process still alive!\" is never printed because when linked process crashes it brings down the parent process with it.\n\nIn our case this also leads the linked live view process to crash.\n\nWhen a linked process exits gracefully with a reason `:normal` this does not lead to the parent process to crash. Any other reason other than `:normal` is considered an abnormal termination and will lead to the linked processes exiting as well.\n\nWhen a process reaches its end, by default it exits with reason `:normal`\n\n```elixir\nlinked_process =\n spawn_link(fn ->\n exit(:normal)\n Process.sleep(60000)\n end)\n\n:timer.sleep(100)\nIO.inspect(Process.alive?(linked_process), label: \"Linked process alive?\")\n```\n\nLinking can also be done manually by calling `Process.link/1`, lets see a bigger example...\n\n```elixir\ndefmodule LinkingProcess do\n def call do\n child_process = spawn(&recursive_link_inspectior/0)\n\n IO.inspect(self(), label: \"Parent process PID\")\n IO.inspect(child_process, label: \"Child process PID\")\n\n IO.inspect(Process.info(self(), :links), label: \"Parent process links\")\n\n send(child_process, :inspect_links)\n\n # Wait for the child process to print its links\n :timer.sleep(100)\n\n # Link the two processes\n Process.link(child_process)\n\n :timer.sleep(100)\n\n IO.inspect(Process.info(self(), :links), label: \"Parent process links\")\n\n send(child_process, :inspect_links)\n end\n\n defp recursive_link_inspectior do\n receive do\n :inspect_links ->\n links = Process.info(self(), :links)\n IO.inspect(links, label: \"Child process links\")\n end\n\n recursive_link_inspectior()\n end\nend\n\nLinkingProcess.call()\n```\n\nWhen a process is linked to others, a crash in that process can trigger a cascade effect, potentially causing multiple other linked processes to crash as well. For instance, imagine a scenario where five processes (P1 to P5) are linked as follows:\n\n`P1 <-> P2 <-> P3 <-> P4 <-> P5`\n\nIf any of these processes crash, it will cause all five to fail due to their interconnectivity. For instance, if P4 crashes, it will cause P3 and P5 to crash as well. This, in turn, will lead to the failure of P2, which will ultimately cause P1 to fail as well.\n\nIt's important to remember that **process links are bidirectional**, which means that if one process fails, it will affect the other processes as well.\n\n\n\n### Importance of process linking\n\nProcesses and links play an important role when building fault-tolerant systems. Elixir processes are isolated and donโt share anything by default. Therefore, a failure in a process will never crash or corrupt the state of another process. Links, however, allow processes to establish a relationship in case of failure. We often link our processes to supervisors which will detect when a process dies and start a new process in its place.\n\nWhile other languages would require us to catch/handle exceptions, in Elixir we are actually fine with letting processes fail because we expect supervisors to properly restart our systems. โFailing fastโ (sometimes referred as โlet it crashโ) is a common philosophy when writing Elixir software!\n\n\n\n### Trapping EXITS\n\nFor some reason if we want to prevent a process from crashing when a linked process exits we can do so by trapping exit message.\n\nNormally when a process finishes its work it implicitly calls `exit(:normal)` to communicate with its parent process that its job has been done. Any other argument to `exit/1` other than `:normal` is treated as an error.\n\nSetting `trap_exit` to true in Elixir means that **exit signals received by a process are converted into messages** of the form `{'EXIT', From, Reason}`. These messages can then be received like any other message in the process's mailbox. On the other hand, if `trap_exit` is set to false, the process will exit if it receives an exit signal that is not a normal exit, and the signal will be passed on to any processes that are linked to it.\n\nBy using `trap_exit` and linking processes, we can prevent the failure of one process from causing the failure of another. This allows the linked process to handle the termination of the other process gracefully, rather than being abruptly terminated itself.\n\nAs always lets look at an example to understand this better...\n\n```elixir\n# Start trapping exit for the current process\nProcess.flag(:trap_exit, true)\n\n# A linked process that will exit abnormally with a reason :boom\np = spawn_link(fn -> exit(:boom) end)\n\n:timer.sleep(100)\n\n# Is the child process is alive?\nIO.inspect(Process.alive?(p), label: \"Child process alive?\")\n\n# Check how the parent process that is trapping exit received an EXIT message\nProcess.info(self(), :messages) |> IO.inspect(label: \"Messages in parent process mailbox\")\n```\n\nAs we see from the print messages the linked process crashing does not lead to a crash of the parent process as it is trapping exits, instead the parent receives a message like `{:EXIT, linked_process_pid, :boom}` in its mailbox.\n\n\n\nIt is generally recommended to avoid trapping exits as it can modify the normal behavior of processes. Instead, it is recommended to utilize monitors and supervisors to handle failures.\n\nWhen a process traps exits, it becomes unresponsive to exit signals unless a kill exit reason is explicitly sent to it. Lets look at an example...\n\n```elixir\n# Un-killable exit trapper process\np =\n spawn(fn ->\n Process.flag(:trap_exit, true)\n :timer.sleep(:infinity)\n end)\n\nIO.inspect(Process.alive?(p), label: \"Process alive initially\")\n\nProcess.exit(p, :normal)\n:timer.sleep(100)\nIO.inspect(Process.alive?(p), label: \"After :normal exit signal\")\n\nProcess.exit(p, :boom)\n:timer.sleep(100)\nIO.inspect(Process.alive?(p), label: \"After :boom exit signal\")\n\n# Only a :kill exit signal can kill a process thats trapping exits.\nProcess.exit(p, :kill)\n:timer.sleep(100)\nIO.inspect(Process.alive?(p), label: \"After :kill exit signal\")\n```\n\nIn Elixir, the `:normal` and `:kill` are special exit reasons. `:normal` signifies a successful and expected process termination. On the other hand, `:kill` is non-trappable, causing immediate process termination. Any other termination reasons are informational and can be trapped if necessary.\n\nNote that the call to `Process.exit(pid, :normal)` function is silently ignored if the specified `pid` is different from the calling process's own `pid` (`self()`). This is an edge case.\n\n## Resources\n\n* https://eddwardo.github.io/posts/links-in-elixir/\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_2.4_process_monitoring_and_hibernation.livemd",
"content": "# Process Monitoring and Hibernation\n\n## Navigation\n\n\n\n## Process Monitors\n\nProcess **links are bidirectional**, which means that if a linked process exits, it will also bring down the current process. However, if we only want the current process to be notified when a process has exited, instead of linking, we can use monitors.\n\nUnlike linking, **monitoring is unidirectional**. If there is an error in a monitored process, it does not bring down the current process. Instead, the current process is notified via a `{:DOWN, , :process, , }` message.\n\nIt's worth noting that even when the monitored process exits normally, we still receive a message. In the case of process linking, a process is only notified if the linked process exits abnormally (i.e., with a reason other than :normal).\n\nLets look at an example of process monitoring...\n\n```elixir\npid = spawn(fn -> :timer.sleep(10000) end)\nProcess.monitor(pid)\nProcess.exit(pid, :boom)\n:timer.sleep(100)\nIO.inspect(Process.info(self(), :messages))\n```\n\n## Process hibernation\n\nWe can call `Process.hibernate/3` to hibernate a process.\n\nFrom the official [documentation](https://erlang.org/doc/man/erlang.html#hibernate-3)\n\n> Puts the calling process into a **wait state** where its **memory allocation has been reduced as much as possible**. This is useful if the process does not expect to receive any messages soon. The process is awaken when a message is sent to it, and control resumes in Module:Function with the arguments specified by Args with the call stack emptied.\n\n> In more technical terms, `erlang:hibernate/3` discards the call stack for the process, and then **garbage collects** the process. After this, all live data is in one continuous heap. The heap is then shrunken to the exact same size as the live data that it holds.\n\n\n\n### When is process hibernation useful?\n\nHibernation of a process can be beneficial in situations where the process should not be terminated but is not expected to receive any messages anytime soon. By hibernating the process, we can free up the memory that was allocated to the process during garbage collection and thus prevent unnecessary resource usage.\n\nSome practical examples where hibernation can be useful include occasionally used processes that should not be dropped, as doing so may be interpreted as a network disconnection by the client. Additionally, any process that is expensive to reinitialize may also be a good candidate for hibernation.\n\nLets see an example...\n\n```elixir\np1 =\n spawn(fn ->\n _big_binary = :crypto.strong_rand_bytes(1000)\n :timer.sleep(:infinity)\n end)\n\np2 =\n spawn(fn ->\n _big_binary = :crypto.strong_rand_bytes(1000)\n Process.hibernate(IO, :puts, [\"P2 woken from hibernation\"])\n\n # This never executes as execution resumes at the function passed to Process.hibernate/3\n IO.puts(\"Kabooom!\")\n end)\n\nProcess.info(p1, :total_heap_size) |> IO.inspect(label: \"Heap size of P1\")\nProcess.info(p2, :total_heap_size) |> IO.inspect(label: \"Heap size of P2\")\n\n# Wake p2 from hibernation by sending it a message\nsend(p2, :msg)\n:timer.sleep(100)\n\n# Here the process is no longer alive since after executing the IO.puts/1\n# call it has no other work and exits normally.\nProcess.alive?(p2) |> IO.inspect(label: \"P2 alive\")\n```\n\n## Resources\n\n* https://elixirforum.com/t/when-is-hibernation-of-processes-useful/23181/5\n* https://hexdocs.pm/elixir/1.12.3/Process.html\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_2.5_group_leaders_and_process_naming.livemd",
"content": "# Group leaders and naming processes\n\n## Navigation\n\n\n\n## Group Leader\n\nIn Erlang, every process belongs to a process group, and each group has a group leader. The group leader is **responsible for handling I/O** for the processes in its group. When a process is spawned, it inherits the same group leader as its parent process.\n\nAt system start-up, the init process(the first process which coordinates the start-up of the system) is both its own group leader and the group leader of all processes.\n\nThe Erlang VM **models I/O devices as processes**, which enables different nodes in the same network to exchange file processes and read/write files between nodes. The group leader can be configured per process and is used in different situations. For example, when executing code in a remote terminal, it ensures that messages from a remote node are redirected and printed in the terminal that triggered the request.\n\nThe **main responsibility of the group leader is to collect I/O output from all processes in its group and pass it to or from the underlying system**. It essentially owns the standard input, standard output, and standard error channels on behalf of the group.\n\nWhen a file is opened using `File.open/2`, it returns a tuple like `{:ok, io_device}`, where `io_device` is the PID of the process that handles the file. This process monitors the process that opened the file (the owner process), and if the owner process terminates, the file is closed, and the process itself terminates too.\n\n\n\n```elixir\n{:ok, io_device_pid} = File.open(\"test.csv\", [:write])\nIO.write(io_device_pid, \"a binary\")\n```\n\nWhen you call `IO.write(pid, binary)`, the IO module sends a message to the process identified by pid with the desired operation, such as :put_chars.\nThe message has the following structure: `{:io_request, , , {:put_chars, :unicode, \"hello\"}}`.\n\nWhen you write to :stdio, you are actually sending a message to the group leader, which writes to the standard-output file descriptor.\nTherefore, these three code snippets are equivalent:\n\n\n\n```elixir\nIO.puts \"hello\"\nIO.puts :stdio, \"hello\"\nIO.puts Process.group_leader, \"hello\"\n```\n\nTo understand this better let see some examples\n\nSuppose we have two Erlang nodes named \"node1\" and \"node2\".\nYou can create two `iex` shells for this like\n\n\n\n```elixir\niex --sname node1@localhost\niex --sname node2@localhost\n```\n\n(Note: If you want to send messages between nodes on different networks, we need to start the named nodes with a shared cookie)\n\nIf we execute the following code in the iex shell of node1:\n\n\n\n```elixir\nNode.spawn_link(:node2@localhost, fn ->\n IO.puts(\"I will be executed on node2 but printed on node1 since the group leader is node1\")\nend)\n```\n\nThe output of the IO.puts operation will be sent to the group leader, which in this case is node1.\nTherefore, the output will be printed on node1's standard output stream, even though the process that performed the operation is running on node2.\n\nOn the other hand, if we specify the device PID as the `:init` process on node2, the output will be seen on node2's standard output stream:\n\n\n\n```elixir\nNode.spawn_link(:node2@localhost, fn ->\n init_process_pid = Process.whereis(:init)\n\n IO.puts(\n init_process_pid,\n \"I will be executed on node2 and printed on node2 since the device ID passed was node2's init process\"\n )\nend)\n```\n\nFinally, we can also set the group leader of a process explicitly by calling `Process.group_leader/2`.\nIn the following example, we set the group leader of the process running on node2 to node2's `:init` process:\n\n\n\n```elixir\nNode.spawn_link(:node2@localhost, fn ->\n init_process_pid = Process.whereis(:init)\n Process.group_leader(self(), init_process_pid)\n\n IO.puts(\n \"I will be executed on node2 and printed on node2 since the group leader is set to node2's init process\"\n )\nend)\n```\n\nIn this case, the output of the `IO.puts` operation will be sent to node2's `:init` process, which is the new group leader of the process.\nTherefore, the output will be printed on node2's standard output stream.\n\n## Process naming\n\nWe can name processes and then refer to them via their registered name.\n\n```elixir\nProcess.register(self(), :my_process)\n\nProcess.registered()\n|> Enum.any?(&(&1 == :my_process))\n|> IO.inspect(label: \":my_process registered?\")\n```\n\n```elixir\nsend(:my_process, \"Hello\")\nProcess.info(self(), :messages)\n```\n\n```elixir\nProcess.unregister(:my_process)\n\nProcess.registered()\n|> Enum.any?(&(&1 == :my_process))\n|> IO.inspect(label: \":my_process registered?\")\n```\n\n## Resources\n\n* https://www.erlang.org/doc/man/erlang.html#group_leader-0\n* https://stackoverflow.com/questions/36318766/what-is-a-group-leader\n* https://rokobasilisk.gitbooks.io/elixir-getting-started/content/io_and_the_file_system/processes_and_group_leaders.html\n* https://elixirschool.com/en/lessons/advanced/otp_distribution#a-note-on-io-and-nodes-2\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_3.1_genserver_introduction.livemd",
"content": "\n\n# GenServer Introduction\n\n## Navigation\n\n\n\n## What is a Genserver?\n\nIn OTP(Open Telecom Platform), we have several behaviors that formalize common patterns in programming. Behaviors can be thought of as design patterns for processes. Over time, programmers have identified common patterns of using processes in OTP and designed standardized interfaces to cater to such use cases.\n\nOne such behavior is the GenServer(Generic Server), which comes bundled with OTP. Other examples of behaviors include Supervisors and Applications.\n\nAt its most basic level, **a GenServer is a single process that runs a loop and handles one message per iteration, passing along an updated state**. By using the GenServer behavior and implementing the necessary callbacks, we can easily implement a client-server relation.\n\nA GenServer process starts by initializing its state and then enters a waiting state, anticipating incoming messages. Upon receiving a message, the process handles it, updates its state, and returns to the waiting state (genserver loop).\n\nA process can only execute when it receives a message. After initialization, a process simply waits for messages, an **idle process doesn't consume any resources**.\n\n## Genserver callbacks\n\nIn order to create a genserver we must first use the genserver behaviour by adding the following line to our module `use Genserver`\n\nAfter this we can implement the Genserver callbacks, a genserver has the following callbacks..\n\n* `init/1`\n* `handle_continue/2`\n* `handle_call/3`\n* `handle_cast/2`\n* `handle_info/2`\n* `terminate/2`\n* `format_status/2`\n* `code_change/3`\n\nThese callbacks are called at various points in the lifecycle of a genserver. Lets build a simple `counter` to go through these callbacks one by one...\n\n```elixir\ndefmodule Counter do\n use GenServer\n\n @impl true\n def init(state) do\n IO.inspect(\"init called, initial counter state: #{state}\")\n {:ok, state}\n end\n\n @impl true\n def handle_cast({:inc, value}, state) do\n {:noreply, state + value}\n end\n\n @impl true\n def handle_cast({:dec, value}, state) do\n {:noreply, state - value}\n end\n\n @impl true\n def handle_call(:get_count, _from, state) do\n {:reply, \"The count is #{state}\", state}\n end\n\n @impl true\n def handle_info(message, state) do\n IO.inspect(\"Handle info called with message #{inspect(message)}\")\n {:noreply, state}\n end\n\n @impl true\n def terminate(reason, _state) do\n IO.inspect(\"Genserver Terminating with reason #{reason}...\")\n end\nend\n```\n\nWith the above counter genserver code let us try to understand the different callbacks.\nWe will enable tracing genserver messages via the [:sys.trace/2](https://www.erlang.org/doc/man/sys.html#trace-2) function from the erlang `sys` module.\n\n```elixir\nIO.inspect(\"Starting Genserver\")\n{:ok, pid} = GenServer.start(Counter, 0)\nIO.inspect(\"Genserver Started\")\n\n# Start tracing the genserver processes\n:sys.trace(pid, true)\n\n# Increment counter by 10\n:ok = GenServer.cast(pid, {:inc, 10})\n# Decrement counter by 5\n:ok = GenServer.cast(pid, {:dec, 5})\n# Increment counter by 5\n:ok = GenServer.cast(pid, {:inc, 2})\ncurrent_count = GenServer.call(pid, :get_count)\nIO.puts(\"Current count = #{current_count}\")\n\n# Send a message to the genserver process\nsend(pid, \"Hi genserver!\")\n\n# Stop the genserver\nGenServer.stop(pid, :boom)\n```\n\nLet's analyze the lifecycle of the GenServer by examining the output of the above code. Firstly notice that all of the functions are marked with `@impl true` to signify that they are implementing the GenServer behavior.\n\nEach GenServer callback receives the current state of the process and has the opportunity to update it. The callbacks can also return various values like `:noreply`, `:reply`, `:continue`, `:stop`, `:hibernate`, etc. These values govern the GenServer's lifecycle.\n\n#### Starting the GenServer - init/2\n\nTo start the GenServer, we call `GenServer.start(Counter, 0)` which starts the GenServer process as an **unlinked process** we can use `GenServer.start_link/3` to start it as a linked process. We pass it the GenServer module name and the initial state of our Counter GenServer process. The output indicates that the `GenServer.start/2` call is **synchronous** and waits until the `init/2` GenServer callback. Once started, the GenServer process pid is returned.\n\n#### handle_cast/2\n\nWe then send different cast messages like :inc and :dec to the GenServer to modify the process state, which, in our case, increments or decrements the counter. The `handle_cast/2` GenServer callback handles these cast calls. It's important to remember that cast messages are **asynchronous** and the `GenServer.cast/2` call does not wait for the cast message to be processed. Also, using cast, the GenServer **cannot send a reply** back to the caller process, so we only receive a `:ok` as the return value when calling `GenServer.cast/2`.\n\n#### handle_call/3\n\nWe then use `GenServer.call/3` to fetch the current count, which is handled by the `handle_call/3` GenServer callback. Unlike `GenServer.cast/2`, this is a **synchronous operation**, meaning the `GenServer.call/3` function call must wait until the GenServer finishes processing the message. It also **allows the GenServer to return a reply to the caller**. In our case, the Counter GenServer returns the current count as a string like \"The count is #{state}\". It's worth noting that the `handle_call/3` receives a from parameter, which contains the pid of the caller process.\n\n### handle_info/2\n\nNext, we send a message to the GenServer process using the `send/2` function. It's important to remember that a GenServer can also receive messages like any other elixir process. The `handle_info/2` GenServer callback handles such messages that are not calls or casts. In our case, we simply log the message \"Hi genserver!\".\n\n#### terminate/2\n\nFinally, we stop the GenServer process by calling `GenServer.stop/2`, which invokes the `terminate/2` GenServer callback, and the GenServer process is stopped.\n\n\n\nYou might be wondering when the other GenServer callbacks are invoked, lets go through them one by one....\n\n#### handle_continue/2\n\nMost GenServer callbacks have the option to return a value containing a continue instruction like `{:continue, continue_arg}`. When such a value is returned, the `handle_continue/2` callback is invoked to handle the continue instruction. This is useful for splitting the work in a callback into multiple steps and updating the process state along the way, or for performing work after initialization.\n\nFor example, to initialize a GenServer, we may need to perform a time-consuming task within the init/2 callback, which would block the caller and prevent the GenServer from starting. To avoid this, we can return a value like `{:ok, state, {:continue, continue_arg}}`, which allows the GenServer to start and unblocks the caller. The handle_continue/2 callback is then immediately invoked, where we can set the GenServer state.\n\n#### format_status/2\n\nThis callback is infrequently used, but it can be helpful when inspecting a GenServer state with functions like `:sys.get_state/1`. It defines a formatted version of the status.\n\n#### code_change/3\n\nThis callback is also rarely used. It handles changes to the GenServer's state when a new version of a module is loaded ([hot code swapping](https://medium.com/blackode/how-to-perform-hot-code-swapping-in-elixir-afc824860012)) and the term structure of the state needs to be updated.\n\n## The terminate callback\n\nThe `terminate/2` callback is triggered when a GenServer is about to exit, allowing for any necessary cleanup operations. However, it is important to note that `terminate/2` is not always guaranteed to be called.\n\n`terminate/2` is only called when the GenServer is trapping exits using the `Process.flag(:trap_exit, true)` OR if in a callback we return a `:stop` tuple or `raise` and exception. We will later study about process supervisors which can stop a genserver using a `:brutal_kill` strategy which also does not result in a call to `terminate/2`.\n\nTherefore it is *not guaranteed* that `terminate/2` is called when a GenServer exits and we should not rely on it and place critical logic in this callback.\n\nWhen using `GenServer.stop/2` the terminate/2 callback will be invoked before exiting even if the GenServer process is not trapping exits.\n\nFor further information, see the discussion [here](https://stackoverflow.com/a/39775617).\n\n## Lifecycle of a GenServer\n\nA simplified overview of the lifecycle of a GenServer is given below\n\n\n\n\n\n\n\nNow that we have got an overview of the workings of a GenServer lets look at some gotachas and key points related to GenServers...\n\n## GenServer Key Points to Remember\n\n* A GenServer is a **single elixir process** that operates in a loop, processing messages from its mailbox in the **order** they are received.\n* If a message takes a long time to process, calling synchronous functions such as `GenServer.call/2` may result in timeouts. You can specify a longer timeout (the default is 5 seconds) or use multiple GenServers to avoid overloading a single process.\n* GenServer functions fall into two categories: synchronous functions, like `GenServer.call/3`, which wait for a response, and asynchronous functions, like `GenServer.cast/2`, which do not wait for a reply.\n* Prefer using `GenServer.call/2` instead of `GenServer.cast/2` to apply backpressure and avoid overwhelming the `GenServer` process. `GenServer.call/2` blocks the caller process until a reply is received, ensuring controlled interactions and preventing message overload.\n* Implementing GenServer callbacks is optional, as Elixir provides default implementations. For example, if you don't define `handle_cast/2`, Elixir will use [a default implementation](https://github.com/elixir-lang/elixir/blob/a64d42f5d3cb6c32752af9d3312897e8cd5bb7ec/lib/elixir/lib/gen_server.ex#L809) that raises an error when the GenServer receives a cast message.\n GenServer callbacks can return different values to control the process's lifecycle. For instance:\n * Returning `{:continue, term()}` tells the GenServer to continue processing the message, triggering the `handle_continue/2` callback.\n * Returning `{:stop, reason, new_state}` terminates the GenServer process.\n * Returning `:hibernate` puts the GenServer process to sleep, freeing up resources.\n\n## References\n\n* [HexDocs: GenServer](https://hexdocs.pm/elixir/GenServer.html#call/3)\n* [ElixirLang: GenServer](https://elixir-lang.org/getting-started/mix-otp/genserver.html)\n* [Exercism: GenServer](https://exercism.org/tracks/elixir/concepts/genserver)\n* https://github.com/DockYard-Academy/curriculum/blob/main/reading/genservers.livemd\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_3.2_building_a_genserver.livemd",
"content": "# Buliding a GenServer\n\n## Navigation\n\n\n\n## Building a GenServer from scratch\n\nLet's delve into crafting a simplified GenServer-like implementation using Elixir's fundamental primitives such as `spawn_link` and `send`. This exercise will give us a clearer insight into the inner workings of GenServers.\n\nFor the sake of simplicity, we will focus on implementing only the commonly used callbacks: `init/1`, `handle_call/3`, `handle_cast/2`, and `handle_info/2`.\n\n```elixir\ndefmodule MyGenServer do\n # Callbacks to implement\n @callback init(term()) :: {:ok, term()}\n @callback handle_call(term(), pid(), term()) :: {:reply, term(), term()}\n @callback handle_cast(term(), term()) :: {:noreply, term()}\n @callback handle_info(term(), term()) :: {:noreply, term()}\n\n # == Public API ==\n def start_link(module, args) do\n {:ok, spawn_link(__MODULE__, :server_init, [module, args])}\n end\n\n def call(server_pid, args) do\n send(server_pid, {:call, self(), args})\n\n receive do\n {:response, response} -> response\n end\n end\n\n def cast(server_pid, args) do\n send(server_pid, {:cast, args})\n end\n\n def stop(server_pid, reason \\\\ :normal) do\n send(server_pid, {:stop, reason})\n end\n\n # == Internal implementation ==\n def server_init(module, args) do\n {:ok, state} = module.init(args)\n genserver_loop(module, state)\n end\n\n # Recursively loop and wait for messages\n def genserver_loop(module, state) do\n receive do\n {:call, parent_pid, args} ->\n {:reply, response, new_state} = module.handle_call(args, parent_pid, state)\n send(parent_pid, {:response, response})\n genserver_loop(module, new_state)\n\n {:cast, args} ->\n {:noreply, new_state} = module.handle_cast(args, state)\n genserver_loop(module, new_state)\n\n {:stop, reason} ->\n module.terminate(reason, state)\n exit(reason)\n\n request ->\n {:noreply, new_state} = module.handle_info(request, state)\n genserver_loop(module, new_state)\n end\n end\nend\n```\n\n## Using our GenServer\n\n```elixir\ndefmodule Stack do\n @behaviour MyGenServer\n\n @impl true\n def init(args) do\n {:ok, args}\n end\n\n @impl true\n def handle_call(:get_stack, _from, state) do\n {:reply, state, state}\n end\n\n @impl true\n def handle_call(:pop, _from, [num | state]) do\n {:reply, num, state}\n end\n\n @impl true\n def handle_cast({:push, num}, state) do\n IO.inspect(num, label: \"PUSH\")\n {:noreply, [num | state]}\n end\n\n @impl true\n def handle_info(:stats, state) do\n IO.inspect(\"Stack length: #{length(state)}\")\n {:noreply, state}\n end\nend\n```\n\n```elixir\n{:ok, stack_server_pid} = MyGenServer.start_link(Stack, [])\nMyGenServer.cast(stack_server_pid, {:push, 1})\nMyGenServer.cast(stack_server_pid, {:push, 2})\nMyGenServer.cast(stack_server_pid, {:push, 3})\nMyGenServer.call(stack_server_pid, :get_stack) |> IO.inspect(label: \"STACK\")\nMyGenServer.call(stack_server_pid, :pop) |> IO.inspect(label: \"POP\")\nMyGenServer.call(stack_server_pid, :get_stack) |> IO.inspect(label: \"STACK\")\nsend(stack_server_pid, :stats)\n```\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_3.3_genserver_examples.livemd",
"content": "# GenServer examples\n\n## Navigation\n\n\n\n## Password Manager\n\nOur objective is to create a straightforward password manager GenServer that can save user's passwords in its state.\n\nNotice how we have developed an API with functions like `save_password/3`, `get_password/1`, and `delete_password/1`. This API facilitates easy communication with the GenServer without needing to directly call GenServer functions like `GenServer.call/3` or `GenServer.cast/2`.\n\nAlso notice how we have ensured that the GenServer code is kept to a minimum and have placed our password validation logic in a separate module. This approach of separating the logic into a purely functional module makes testing easier since the buissness logic can be tested in isolation without dealing with the GenServer.\n\n```elixir\ndefmodule Password do\n defstruct url: nil, username: nil, password: nil, inserted_at: nil\n\n @doc \"\"\"\n Check if a password entry is valid\n \"\"\"\n def validate_entry(%Password{url: url, username: username, password: password}) do\n with {:ok, _url} <- URI.new(url),\n {:ok, _username} <- validate_username(username),\n {:ok, _password} <- validate_password(password) do\n {:ok,\n %Password{\n url: url,\n username: username,\n password: password,\n inserted_at: DateTime.utc_now()\n }}\n end\n end\n\n # Helper functions\n defp validate_username(username) do\n cond do\n not is_binary(username) -> {:error, \"Invalid username\"}\n String.length(username) == 0 -> {:error, \"Username is empty\"}\n true -> {:ok, username}\n end\n end\n\n defp validate_password(password) do\n cond do\n not is_binary(password) -> {:error, \"Invalid password\"}\n String.length(password) < 3 -> {:error, \"Password must be atleast 3 character long\"}\n true -> {:ok, password}\n end\n end\nend\n\ndefmodule PasswordManager do\n use GenServer\n\n # Public APIs\n\n def start_link(_opts) do\n GenServer.start_link(\n __MODULE__,\n %{},\n # Use the module name as the name of the GenServer Process\n name: __MODULE__\n )\n end\n\n def save_password(url, username, password) do\n entry = %Password{\n url: url,\n username: username,\n password: password\n }\n\n GenServer.call(__MODULE__, {:save_password, entry})\n end\n\n def get_password(url) do\n GenServer.call(__MODULE__, {:get_password, url})\n end\n\n def delete_password(url) do\n GenServer.cast(__MODULE__, {:delete_password, url})\n end\n\n def stop(), do: GenServer.stop(__MODULE__)\n\n # Callbacks\n\n @impl true\n def init(state) do\n {:ok, state}\n end\n\n @impl true\n def handle_call({:save_password, new_password}, _from, state) do\n case Password.validate_entry(new_password) do\n {:ok, entry} -> {:reply, :saved, Map.put(state, entry.url, entry)}\n {:error, reason} -> {:reply, {:error, reason}, state}\n end\n end\n\n @impl true\n def handle_call({:get_password, url}, _from, state) do\n case Map.get(state, url) do\n nil -> {:reply, :not_found, state}\n entry -> {:reply, entry, state}\n end\n end\n\n @impl true\n def handle_cast({:delete_password, url}, state) do\n state = Map.delete(state, url)\n {:noreply, state}\n end\nend\n```\n\n```elixir\n# Start the Password Manager Genserver\n{:ok, _pid} = PasswordManager.start_link(nil)\n\nPasswordManager.save_password(\"gmail.com\", \"john_doe@gmail.com\", \"12345\")\n|> IO.inspect(label: \"Saving Gmail creds\")\n\nPasswordManager.save_password(\"spotify.com\", \"music4life\", \"ab\")\n|> IO.inspect(label: \"Saving Spotify creds\")\n\nPasswordManager.save_password(\"apple.com\", \"iuser\", \"ilife\")\n|> IO.inspect(label: \"Saving Apple creds\")\n\nPasswordManager.get_password(\"gmail.com\") |> IO.inspect(label: \"Gmail creds\")\nPasswordManager.get_password(\"spotify.com\") |> IO.inspect(label: \"Spotify creds\")\n\nPasswordManager.delete_password(\"gmail.com\") |> IO.inspect(label: \"Deleting Gmail\")\nPasswordManager.get_password(\"gmail.com\") |> IO.inspect(label: \"Gmail creds\")\n\nPasswordManager.stop()\n```\n\n### Testing GenServer\n\n\n\nNow lets try to write some tests for the above GenServer.\n\n```elixir\nExUnit.start()\n\ndefmodule PasswordManagerTest do\n use ExUnit.Case\n\n describe \"save_password/3\" do\n test \"saves password if password entry is valid\" do\n {:ok, _pid} = PasswordManager.start_link(nil)\n assert :saved == PasswordManager.save_password(\"gmail.com\", \"john_doe@gmail.com\", \"12345\")\n\n assert %Password{\n url: \"gmail.com\",\n username: \"john_doe@gmail.com\",\n password: \"12345\",\n inserted_at: _\n } = PasswordManager.get_password(\"gmail.com\")\n end\n\n test \"does not save password if password entry is invalid\" do\n {:ok, _pid} = PasswordManager.start_link(nil)\n\n assert {:error, \"Password must be atleast 3 character long\"} ==\n PasswordManager.save_password(\"gmail.com\", \"john_doe@gmail.com\", \"12\")\n\n assert :not_found == PasswordManager.get_password(\"gmail.com\")\n end\n end\n\n describe \"delete_password/3\" do\n test \"deletes password if password found\" do\n {:ok, _pid} = PasswordManager.start_link(nil)\n assert :saved == PasswordManager.save_password(\"gmail.com\", \"john_doe@gmail.com\", \"12345\")\n assert :ok == PasswordManager.delete_password(\"gmail.com\")\n\n assert :not_found = PasswordManager.get_password(\"gmail.com\")\n end\n end\nend\n\nExUnit.run()\n```\n\nHere the password validation logic can be tested independently, without having to start the GenServer in the tests. This method of testing is preferable since testing pure functions is generally much easier than testing async GenServer code.\n\n## Cron Job\n\nLets see another example of building a GenServer. In Elixir, you can easily create a basic CRON job using GenServers to execute a task periodically.\n\n```elixir\ndefmodule CronJob do\n use GenServer\n\n # Every 10 seconds\n @interval :timer.seconds(10)\n\n def start_link(_opts) do\n GenServer.start_link(__MODULE__, %{})\n end\n\n def init(state) do\n schedule_work()\n {:ok, state}\n end\n\n def handle_info(:work, state) do\n work()\n schedule_work()\n {:noreply, state}\n end\n\n defp schedule_work() do\n Process.send_after(self(), :work, @interval)\n end\n\n defp work() do\n IO.inspect(\"Working...\")\n end\nend\n\nCronJob.start_link(nil)\n```\n\nThis works fine for simple use cases however, if you require more advanced functionality consider using a library such as [Quantum](https://github.com/quantum-elixir/quantum-core).\n\n## Resources\n\n* https://hexdocs.pm/elixir/1.14.4/GenServer.html#reply/2\n* https://medium.com/blackode/2-unique-use-cases-of-genserver-reply-deep-insights-elixir-expert-31e7abbd42d1\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_3.4_other_genserver_functions.livemd",
"content": "# Other GenServer functions\n\n## Navigation\n\n\n\n## GenServer.reply/2\n\nIn the previous chapters, we learned that a GenServer is a single process that processes messages from its mailbox one at a time. When using `GenServer.call/3`, the calling process waits until the GenServer sends a reply.\n\nHowever, in some cases, a GenServer may receive a message that requires a time-consuming task, which can block the GenServer and prevent it from processing new messages while it handles the lengthy task.\n\nTo avoid this issue, we can delegate the time-consuming task to a separate process, which allows the GenServer to continue handling new messages without being blocked. Once the time-consuming task is completed, the GenServer can reply back to the caller using `GenServer.reply/2`.\n\n**To put it simply,`GenServer.reply/2` can be used to send a reply back to a client that has called `GenServer.call/3`. This is especially useful when the reply cannot be specified in the return value of `handle_call/3`**\n\n---\n\nTo illustrate this concept, let's walk through an example.\n\nWe will build a *FoodOrderingServer* which allows users to order for a food item or list past orders. Lets suppose the call to list the past order is a fast one however the call to place an order is a slow operation.\n\nIdeally we don't want to block other calls to the GenServer while its busy placing an order.\n\n```elixir\ndefmodule FoodOrderingServer do\n use GenServer\n\n # Public APIs\n\n def start_link(_opts) do\n GenServer.start_link(__MODULE__, %{}, name: __MODULE__)\n end\n\n def place_order(user, item) do\n # We specify a timeout of 10 seconds to avoid timeout errors,\n # since placing an order takes a lot of time\n GenServer.call(__MODULE__, {:place_order, user, item}, 10000)\n end\n\n def list_orders(user) do\n GenServer.call(__MODULE__, {:list_orders, user})\n end\n\n # Callbacks\n\n @impl true\n def init(_args) do\n {:ok, %{}}\n end\n\n @impl true\n def handle_call({:place_order, user, item}, from, state) do\n IO.puts(\"Recieved new order request from #{inspect(from)}\")\n\n spawn(fn ->\n # Simulate placing order which takes 6 seconds\n :timer.sleep(6000)\n send(__MODULE__, {:order_placed, user, item, from})\n end)\n\n # Notice how we return :noreply here\n # (the caller process will be blocked and waiting since we did not reply)\n {:noreply, state}\n end\n\n @impl true\n def handle_call({:list_orders, user}, _from, state) do\n {:reply, Map.get(state, user, []), state}\n end\n\n @impl true\n def handle_info({:order_placed, user, item, from}, state) do\n state =\n Map.update(\n state,\n user,\n [item],\n fn existing_orders -> [item | existing_orders] end\n )\n\n IO.puts(\"Order #{item} ready for #{user}\")\n # Send reply to the caller who is waiting for the order\n GenServer.reply(from, {:ok, :order_placed})\n {:noreply, state}\n end\nend\n```\n\n```elixir\n# Stop the Server if its already running\nProcess.whereis(FoodOrderingServer) |> GenServer.stop()\n\n{:ok, _pid} = FoodOrderingServer.start_link(nil)\n\n# This two lines of code will execute one by one synchornously since the current process will be \n# waiting untill the order is placed and the GenServer replies back\nFoodOrderingServer.place_order(\"Jhon\", \"sandwich\")\nFoodOrderingServer.place_order(\"Tom\", \"pizza\")\n```\n\n```elixir\n# Ordering simultaneously from different processes\n\nspawn(fn -> FoodOrderingServer.place_order(\"Jhon\", \"burger\") end)\n\nspawn(fn ->\n FoodOrderingServer.place_order(\"Tom\", \"ice cream\")\n\n FoodOrderingServer.list_orders(\"Tom\")\n |> IO.inspect(label: \"Toms orders\")\nend)\n\nspawn(fn ->\n FoodOrderingServer.list_orders(\"Tom\")\n |> IO.inspect(label: \"Toms orders\")\nend)\n```\n\n#### Code Breakdown\n\nIn the given code, we initially place orders one by one and observe that each call to `FoodOrderingServer.place_order/2` waits for the GenServer's reply before proceeding.\n\nTo simulate multiple users placing and listing their orders simultaneously, we spawn two processes and place orders from each of them concurrently. Since the GenServer delegates the order placing task to another process, it is not blocked and can immediately respond to both orders. Once the orders are processed, the processes send a message back to the GenServer via `send(__MODULE__, {:order_placed, user, item, from})` which is handled by the `handle_info/2` callback, after which the GenServer replies back to the callers who were awaiting the reply using `GenServer.reply(from, {:ok, :order_placed})`.\n\nThis design unblocks the GenServer, allowing it to always respond to messages promptly.\n\nIt's important to note that `GenServer.reply/2` can be invoked from any process, not just from within the GenServer process. In our example, we could have called `GenServer.reply(from, {:ok, :order_placed})` from the spawned processes.\n\nThis is possible because the `from` parameter holds the PID of the caller along with a `reference` that enables the caller to recognize that the message came as a reply for the `GenServer.call/2` that it was waiting for.\n\n---\n\n\n\n(Note: There are 2 other functions `GenServer.abcast/3` and `GenServer.multi_call/4` which allows us to cast and call multiple GenServers at a time and can be useful in a distributed environment.)\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_4.0_the_registry_module.livemd",
"content": "# The Registry module\n\n## Navigation\n\n\n\n## What is Registry?\n\nFrom the official documentation\n\n> A local, decentralized and scalable key-value process storage. It allows developers to lookup one or more processes with a given key.\n\nLets go through some important points about registries...\n\n* The Registry in Elixir is a **process store** that stores **key-value** pairs, allowing us to register a process under a specific name.\n* There are two types of Registries: **unique and duplicate**. A unique Registry only permits one process to be registered under a given name, while a duplicate Registry permits multiple processes to be registered under the same name.\n* Each entry in the Registry is associated with the process that registered it. If the process crashes, the Registry automatically removes the keys associated with that process.\n* The Registry **compares keys using the match operation (===/2)**.\n* **Partitioning** the Registry is possible, allowing for more scalable behavior in highly concurrent environments with thousands or millions of entries.\n* The Registry **uses ETS tables** to store data under the hood.\n* Registries can **only be run locally** and donot support distributed access.\n\n## Where to use Registry?\n\nThe most common use of Registry is to name process. The `:via`is frequently used to specify the process name when using the Registry.\n\nIn addition to process naming, the Registry offers other useful features such as a dispatch mechanism that enables developers to implement custom logic for request initiation. With this dispatching mechanism, developers can build scalable and highly efficient systems, such as a local PubSub, by utilizing the `dispatch/3` function.\n\n## Naming processes using Registry\n\nThe most common use of Registry is in naming processes.\n\nFirst we start the Registry process\n\n```elixir\n# We start a Registry process and name it \"Registry.ProcessStore\"\n# Notice we use `keys: :unique` option which means every key in the Registry\n# will point to a single process\n{:ok, _} = Registry.start_link(keys: :unique, name: Registry.ProcessStore)\n```\n\nNow lets use this registry to name a GenServer\n\n```elixir\ndefmodule Stack do\n use GenServer\n\n # Callbacks\n\n @impl true\n def init(stack) do\n {:ok, stack}\n end\n\n @impl true\n def handle_cast({:push, e}, stack) do\n {:noreply, [e | stack]}\n end\n\n # Other Callbacks ....\nend\n```\n\nNow that we have a simple GenServer lets try to start 2 instances of this GenServer and name each of them using the Registry.\n\n```elixir\n# Start the Stack GenServer and register it under the \"Registry.ProcessStore\"\n# with a key named \"stack_server_1\"\nGenServer.start_link(\n Stack,\n [],\n name: {:via, Registry, {Registry.ProcessStore, \"stack_server_1\"}}\n)\n\n# Start another instance of the Stack server with the name \"stack_server_2\"\n# Notice how we also store an optional value associated with this process `:second_stack`\nGenServer.start_link(\n Stack,\n [],\n name: {:via, Registry, {Registry.ProcessStore, \"stack_server_2\", :second_stack}}\n)\n```\n\nWhen we register a process under a Registry, we have the option to store an associated metadata with that entry. In the second example mentioned above, we not only registered an instance of our Stack GenServer process under the registry but also stored the value `:second_stack` along with its corresponding entry.\n\nNow lets call our Stack GenServer using its Registered name, we can use the `lookup/2` function that returns a list like `[{pid(), value()}]`. For Registries that allow duplicate entries a lookup can return multiple entries in this list.\n\n```elixir\n# Since we use an unique Registry, its guaranteed we will only get atmost\n# one process under the name \"stack_server\"\n[{stack_server_one_pid, nil}] = Registry.lookup(Registry.ProcessStore, \"stack_server_1\")\n\nGenServer.cast(stack_server_one_pid, {:push, \"stack1\"})\n\n[{stack_server_two_pid, value}] = Registry.lookup(Registry.ProcessStore, \"stack_server_2\")\nIO.inspect(value, label: \"Stack server 2 value\")\nGenServer.cast(stack_server_two_pid, {:push, \"stack2\"})\n\nIO.inspect(:sys.get_state(stack_server_one_pid), label: \"Stack Server1 state\")\nIO.inspect(:sys.get_state(stack_server_two_pid), label: \"Stack Server2 state\")\n```\n\nLet us now explore how a Registry operates when we permit the storage of duplicate entries.\n\nWhen utilizing duplicate registries, it is not possible to use the :via option. To illustrate how duplicate registries function, let us attempt to register the current process twice using the `register/3` function.\n\n```elixir\n{:ok, _} = Registry.start_link(keys: :duplicate, name: Registry.DupProcessStore)\n\n{:ok, _} = Registry.register(Registry.DupProcessStore, \"async_city\", :hello)\n\n{:ok, _} = Registry.register(Registry.DupProcessStore, \"async_city\", :world)\n\nRegistry.lookup(Registry.DupProcessStore, \"async_city\")\n```\n\nObserve how the invocation of `Registry.lookup/2` resulted in a list containing 2 tuples, each representing a process along with its associated metadata. These two processes were registered under the identical name, \"async_city\".\n\n## Dispatching using Registry\n\nDispatching allows us to fetch all entries for all processes registered under a given key. We pass a callback function which would receive the list of `{pid, value}` for every entry registered under the given key.\n\nIt is worth noting that dispatching takes place in the process that initiates the `dispatch/3` call, either serially or concurrently in the case of multiple partitions.\n\nTo better understand the concept of dispatching, let us take a look at an example.\n\n```elixir\n# Start a Registry which allows duplicates\n{:ok, _} = Registry.start_link(keys: :duplicate, name: Registry.Numbers)\n\n# Register the current process 3 times under the same key \"odd\"\n# Save a value along with registration that is 1, \"3\" and fn -> 5\n{:ok, _} = Registry.register(Registry.Numbers, \"odd\", 1)\n{:ok, _} = Registry.register(Registry.Numbers, \"odd\", \"3\")\n{:ok, _} = Registry.register(Registry.Numbers, \"odd\", fn -> 5 end)\n\n# Register the current process 3 times under another key \"even\"\n{:ok, _} = Registry.register(Registry.Numbers, \"even\", 2)\n{:ok, _} = Registry.register(Registry.Numbers, \"even\", \"4\")\n{:ok, _} = Registry.register(Registry.Numbers, \"even\", fn -> 6 end)\n\n# Dispatching on processes registered under the key \"odd\"\nRegistry.dispatch(Registry.Numbers, \"odd\", fn entries ->\n for {_pid, num} <- entries do\n cond do\n is_number(num) -> num\n is_binary(num) -> String.to_integer(num)\n is_function(num) -> num.()\n end\n |> IO.inspect(label: \"ODD\")\n end\nend)\n\n# Dispatching on processes registered under the key \"even\"\nRegistry.dispatch(Registry.Numbers, \"even\", fn entries ->\n for {_pid, num} <- entries do\n cond do\n is_number(num) -> num\n is_binary(num) -> String.to_integer(num)\n is_function(num) -> num.()\n end\n |> IO.inspect(label: \"EVEN\")\n end\nend)\n```\n\n### Building a pubsub system with Registry\n\nWe can also use this `dispatch/3` function to implement a local, non-distributed PubSub.\nThis works by registering multiple processes under a given key which acts like a pubsub topic.\n\nWe can then send a message to all processes registered under a key to emulate a pubsub broadcast. Here we also set the number of partitions to the number of schedulers online, which will make the registry more performant on highly concurrent environments.\n\nLets see this in action.\n\n```elixir\n{:ok, _} =\n Registry.start_link(\n keys: :duplicate,\n name: Registry.ChatPubSub,\n # The number of schedulers available in the VM\n partitions: System.schedulers_online()\n )\n\n# Register the current process under the \"Registry.ChatPubSub\" registery with a key \"chat_room:1\"\n{:ok, _} = Registry.register(Registry.ChatPubSub, \"chat_room:1\", [])\n\n# Dispatching by looking up all process registered with the key \"chat_room:1\" in the\n# \"Registry.ChatPubSub\" registry and then sending them a message.\nRegistry.dispatch(Registry.ChatPubSub, \"chat_room:1\", fn entries ->\n for {pid, _} <- entries, do: send(pid, {:broadcast, \"hello world\"})\nend)\n\n# Receive any broadcasted messages\nreceive do\n {:broadcast, message} -> IO.inspect(message, label: \"Received broadcast\")\nend\n```\n\nBy using this approach, we can register multiple processes under a single key within a Registry and subsequently dispatch messages to all the processes associated with that key.\n\n## Other registry functions and match specs\n\nApart from the `register/3` and `lookup/2` functions, the Registry module has several other useful functions which allows us to find and manipulate data inside the Registry. Most of these functions are straightforward to understand.\n\nHowever, its worth noting that some functions use match specs to find matching entries from the Registry let us look at some examples to understand how match specs work.\n\nFrom the official documentation\n\n> A match spec is a pattern that must be an atom or a tuple that will match the structure of the value stored in the registry. The atom `:_` can be used to ignore a given value or tuple element, while the atom `:\"$1\"` can be used to temporarily assign part of pattern to a variable for a subsequent comparison.\n\n> Optionally, it is possible to pass a list of guard conditions for more precise matching. Each guard is a tuple, which describes checks that should be passed by assigned part of pattern. For example the `$1 > 1` guard condition would be expressed as the `{:>, :\"$1\", 1}` tuple. Please note that guard conditions will work only for assigned variables like :\"$1\", :\"$2\", and so forth.\n\nLets consider the `match/4` functions in the Registry module that returns entries from the Registry that matches the match spec passed.\n\n```elixir\nRegistry.start_link(keys: :duplicate, name: Registry.MatchSpec)\n\n# Register the current process multiple times with different values under the key \"my_key\"\n{:ok, _} = Registry.register(Registry.MatchSpec, \"my_key\", 1)\n{:ok, _} = Registry.register(Registry.MatchSpec, \"my_key\", \"one\")\n{:ok, _} = Registry.register(Registry.MatchSpec, \"my_key\", {1, 2})\n{:ok, _} = Registry.register(Registry.MatchSpec, \"my_key\", {2, 1})\n{:ok, _} = Registry.register(Registry.MatchSpec, \"my_key\", {2, 2})\n\n# Use different match specs to find matching entries from the Registry under the key \"my_key\"\nRegistry.match(Registry.MatchSpec, \"my_key\", 1)\n|> IO.inspect(label: \"* match spec: 1 returned\")\n\nRegistry.match(Registry.MatchSpec, \"my_key\", :_)\n|> IO.inspect(label: \"* match spec: :_ returned\")\n\nRegistry.match(Registry.MatchSpec, \"my_key\", {2, :_})\n|> IO.inspect(label: \"* match spec: {2, :_} returned\")\n\nRegistry.match(Registry.MatchSpec, \"my_key\", {:\"$1\", :\"$1\"})\n|> IO.inspect(label: ~s(* match spec: {:\"$1\", :\"$1\"} returned))\n\n# Also using guards along with match specs\nRegistry.match(Registry.MatchSpec, \"my_key\", {:\"$1\", :\"$2\"}, [{:>, :\"$1\", :\"$2\"}])\n|> IO.inspect(label: ~s(* match spec: {:\"$1\", :\"$2\"} with guard [{:>, :\"$1\", :\"$2\"}] returned))\n\nRegistry.match(Registry.MatchSpec, \"my_key\", :\"$1\", [{:is_binary, :\"$1\"}])\n|> IO.inspect(label: ~s(* match spec: :\"$1\" with guard [{:is_binary, :\"$1\"}] returned\"))\n```\n\nOther functions like `count_match/4`, `select/2`, etc in the Registry module also use match specs for filtering entries in the Registry.\n\n## Resources\n\n* The official Registry documentition: https://hexdocs.pm/elixir/1.14.4/Registry.html#content\n* Guards in elixir: https://hexdocs.pm/elixir/1.14/patterns-and-guards.html#guards\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_5.1_supervisors_introduction.livemd",
"content": "# Introduction to Supervisors\n\n## Navigation\n\n\n\n## What is a Supervisor?\n\nIn our previous lesson on OTP, we explored the Genserver behavior. However, there's another critical behavior in OTP that deserves our attention: the **supervisor**.\n\nSupervisors fulfill the role of overseeing other processes, often referred to as child processes, and contribute to the creation of a hierarchical process structure called a **supervision tree**. This tree not only ensures fault-tolerance but also governs the application's startup and shutdown processes.\n\nSupervisors are the driving force behind the Elixir developer's inclination towards embracing the \"let it crash\" or \"fail fast\" philosophy. This approach allows supervisors to automatically restart crashed processes, facilitating a more robust system.\n\n\n\nTo better understand how supervisors work, let's examine a simple example. We'll create a Stack Genserver module with a bug that causes it to crash when attempting to pop an element from an empty stack. Since our Genserver is supervised, we can observe how the supervisor automatically restarts the failed Genserver process when it crashes.\n\n```elixir\ndefmodule Stack do\n use GenServer\n\n def start_link(%{initial_value: value, name: name}) do\n GenServer.start_link(__MODULE__, value, name: name)\n end\n\n ## Callbacks\n\n @impl true\n def init(arg) do\n IO.puts(\"Stack GenServer starting up!\")\n {:ok, [arg]}\n end\n\n @impl true\n def handle_call({:push, element}, _from, stack) do\n IO.puts(\"Pushed #{inspect(element)}\")\n {:reply, :pushed, [element | stack]}\n end\n\n @impl true\n def handle_cast(:pop, [popped | stack]) do\n IO.puts(\"Popped #{inspect(popped)}\")\n {:noreply, stack}\n end\nend\n```\n\n```elixir\nchildren = [\n %{\n id: :stack_1,\n # The Stack is a child porcess started via Stack.start_link/1\n start: {Stack, :start_link, [%{initial_value: 0, name: :stack_1}]}\n }\n]\n\n# Now we start the supervisor process and pass it the list of child specs (child processes to supervise)\n# On starting the supervisor, it automatically starts all the child processes and supervises them\n{:ok, supervisor_pid} = Supervisor.start_link(children, strategy: :one_for_one)\n\n# After the supervisor starts, we can query the supervisor for information regarding all child processes supervised under it\nSupervisor.which_children(supervisor_pid) |> IO.inspect(label: \"Supervisor's children\")\n```\n\nNow lets see what happens if our Stack Genserver process crashes\n\n```elixir\nGenServer.whereis(:stack_1) |> IO.inspect(label: \"Stack Genserver Process pid\")\n\n:sys.get_state(GenServer.whereis(:stack_1))\n|> IO.inspect(label: \"Intial Genserver state\")\n\nGenServer.call(:stack_1, {:push, 10})\nGenServer.call(:stack_1, {:push, 20})\nGenServer.cast(:stack_1, :pop)\nGenServer.cast(:stack_1, :pop)\nGenServer.cast(:stack_1, :pop)\n\n:sys.get_state(GenServer.whereis(:stack_1))\n|> IO.inspect(label: \"Genserver state just before crash\")\n\n# Boom! Stack genserver crashes..\nGenServer.cast(:stack_1, :pop)\n\n# wait for the supervisor to restart the Stack Server process\nProcess.sleep(200)\nGenServer.whereis(:stack_1) |> IO.inspect(label: \"Restarted stack Genserver Process pid\")\n:sys.get_state(GenServer.whereis(:stack_1)) |> IO.inspect(label: \"Genserver state after crash\")\n```\n\n## Child Specs\n\nWhen starting a supervisor, we have the option to provide a list of child specifications that dictate how the supervisor should handle starting, stopping, and restarting each child process.\n\nA supervisor can supervise two types of processes: workers and other supervisor processes. The former is commonly known as a `worker`, while the latter is referred to as a `supervisor`, typically forming a supervision tree.\n\nA child specification is represented as a map with up to six elements. The first two elements are mandatory, while the remaining ones are optional.\n\nLets go through the different options that we can specify in the supervisor child spec\n\n* `:id` - This key is **required** and serves as an internal identifier used by the supervisor to identify the child specification. It should be unique among the workers within the same supervisor.\n\n* `:start` - This key is **required** and contains a tuple specifying the module, function, and arguments used to start the child process.\n\n* `:restart` - This optional key, defaulted to `:permanent`, is an atom that determines when a terminated child process should be restarted.\n\n* `:shutdown` - This optional key, defaulted to 5_000 (5 seconds) for workers and `:infinity` for supervisors, specifies how a child process should be terminated, either by an integer representing a timeout or the atom `:infinity`.\n\n* `:type` - This optional key, defaulted to `:worker`, specifies whether the child process is a `:worker` or a `:supervisor`.\n\n* `:modules` - This optional key contains a list of modules used by hot code upgrade mechanisms to identify processes using specific modules.\n\nA child specification can be defined in one of three ways:\n\n1. As a map representing the child specification itself.\n\n ```elixir\n children = [\n %{\n id: :stack_1,\n start: {Stack, :start_link, [%{initial_value: 0, name: :stack_1}]}\n }\n ]\n ```\n\n The above example defines a child with `:id` of `:stack_1`, which is started by invoking `Stack.start_link(%{initial_value: 0, name: :stack_1})`.\n\n2. As a tuple with the module name as the first element and the start argument as the second.\n\n ```elixir\n children = [\n {Stack, %{initial_value: 0, name: :stack_1}}\n ]\n ```\n\n When using this shorthand notation, the supervisor calls `Stack.child_spec(%{initial_value: 0, name: :stack_1})` to retrieve the child specification. The `Stack` module is responsible for defining its own `child_spec/1` function.\n\n The `Stack` module can define its child specification as follows:\n\n ```elixir\n def child_spec(arg) do\n %{\n id: Stack,\n start: {Stack, :start_link, [arg]}\n }\n end\n ```\n\n In this case, since `GenServer` already defines `Stack.child_spec/1`, we can leverage the automatically generated `child_spec/1` function and customize it by passing options directly to `use GenServer`. We will see examples of this in later chapters\n\n3. Alternatively, a child specification can be specified by providing only the module name.\n\n ```elixir\n children = [Stack]\n ```\n\n This is equivalent to `{Stack, []}`. However, in our case, it would be invalid since `Stack.start_link/1` requires an initial value, and passing an empty list wouldn't work.\n\n### The `Supervisor.child_spec/2` function\n\nWhen using the shorthand notations mentioned above, such as the `{module, arg}` tuple or a module name only as a child specification, we can modify the generated child specifications using the `Supervisor.child_spec/2` function.\n\n* When a two-element tuple of the form `{module, arg}` is provided, the child specification is retrieved by calling `module.child_spec(arg)`.\n\n* When only a module is given, the child specification is retrieved by calling `module.child_spec([])`.\n\nAfter retrieving the child specification, any overrides specified in the function argument are applied directly to the child spec.\n\nFor example, we can use the shorthand notation `{Stack, %{initial_value: 0, name: :stack_1}}`, but this would set `id: Stack` as the child's identifier since it is the default behavior of `module.child_spec(arg)`. However, we can override this behavior as shown below:\n\n\n\n```elixir\nchildren = [\n Supervisor.child_spec({Stack, %{initial_value: 0, name: :stack_1}}, id: :special_stack)\n]\n```\n\n## Resources\n\n* https://hexdocs.pm/elixir/1.14.4/Supervisor.html\n* https://elixir-lang.org/getting-started/mix-otp/supervisor-and-application.html\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_5.2_supervision_strategies.livemd",
"content": "\n\n# Supervision strategies\n\n```elixir\nMix.install([\n {:kino, \"~> 0.9.0\"}\n])\n```\n\n## Navigation\n\n\n\n## Supervision strategies\n\nWhen starting a supervisor, we have the ability to specify a supervision strategy. This strategy determines the actions taken by the supervisor when one of its child processes crashes.\n\nIn the previous chapter, we started the supervisor for our Stack GenServer process using the `Supervisor.start_link(children, strategy: :one_for_one)` function call.\n\nHere, the `:strategy` option passed to the supervisor refers to the supervision strategy being used.\n\nNow, let's explore each of the supervision strategies in detail.\n\nTo illustrate the different strategies, we'll consider a simple GenServer that crashes if we send it a `:boom` message. This GenServer stores a random positive integer in its state.\n\n```elixir\ndefmodule CrashDummyServer do\n use GenServer\n\n def start_link(name) do\n random_state = System.unique_integer([:positive])\n GenServer.start_link(__MODULE__, {random_state, name}, name: name)\n end\n\n ## Callbacks\n\n @impl true\n def init({random_value, name}) do\n IO.inspect(\"#{name} starting up!\")\n {:ok, random_value}\n end\n\n @impl true\n def handle_cast(:boom, state) do\n process_pid = self() |> inspect()\n raise \"BOOM! CrashDummyServer process: #{process_pid} crashed!\"\n {:noreply, state}\n end\nend\n```\n\nIn the examples so far, we started a supervisor by directly calling the `Supervisor.start_link/2` function with the required options. However we can also define the supervisor as a module instead.\n\nTo do so we have to use the `Supervisor` otp behavior in our module.\n\n```elixir\ndefmodule CrashDummySupervisor do\n # Using this behaviour we will automatically define a child_spec/1 function\n use Supervisor\n\n def start_link(strategy) do\n Supervisor.start_link(__MODULE__, strategy, name: __MODULE__)\n end\n\n # We have to implement this `init/1` callback when using the \"Supervisor\" behaviour\n @impl true\n def init(strategy) do\n # Supervision tree\n children = [\n child_spec(:dummy1),\n child_spec(:dummy2),\n child_spec(:dummy3)\n ]\n\n # Notice the supervision strategy\n Supervisor.init(children, strategy: strategy)\n end\n\n defp child_spec(name) do\n Supervisor.child_spec({CrashDummyServer, name}, id: name)\n end\nend\n```\n\nIn the above code snippet, we define multiple instances of our \"CrashDummyServer\" GenServer within the supervision tree. When the supervisor is started, it automatically starts three instances (processes) of the CrashDummyServer with the names `:dummy1`, `:dummy2`, and `:dummy3`.\n\nSince we want to start three processes of the same GenServer, we cannot use the `{CrashDummyServer, name}` child specification because it would assign the module name as the `:id`, resulting in the same `:id` being given to all three processes. To avoid this, we use the `Supervisor.child_spec/2` function and explicitly pass a separate `:id` to each process.\n\nThe supervision strategy is passed as an argument to the `start_link/1` function and `init/1` callback so that we can restart the same supervisor with a different supervision strategy.\n\n## :one_for_one\n\nWith the \"one_for_one\" supervision strategy, if a child process terminates, only that specific process is restarted. In other words, if there are multiple child processes supervised by our supervisor and one of them crashes, only the crashed process is restarted while the other supervised processes continue running unaffected.\n\n\n\n\n\n\n\nTo observe the behavior of this strategy, we can start the supervisor and then intentionally crash one of the supervised processes to see the restart in action.\n\nWe will use [Kino](https://hexdocs.pm/kino/) to draw the supervision tree before and after the crash.\n\n```elixir\n{:ok, supervisor_pid} = CrashDummySupervisor.start_link(:one_for_one)\n\nProcess.info(supervisor_pid, :links) |> IO.inspect(label: \"Supervisors links\")\n\nSupervisor.which_children(supervisor_pid) |> IO.inspect(label: \"Supervisors Children\")\n\n:sys.get_state(GenServer.whereis(:dummy1)) |> IO.inspect(label: \"Dummy 1 state\")\n:sys.get_state(GenServer.whereis(:dummy2)) |> IO.inspect(label: \"Dummy 2 state\")\n:sys.get_state(GenServer.whereis(:dummy3)) |> IO.inspect(label: \"Dummy 3 state\")\n\nKino.Process.render_sup_tree(supervisor_pid)\n```\n\n```elixir\n# Makes the dummy2 child crash\nGenServer.cast(:dummy2, :boom)\n# Wait for the process to crash and be restarted\nProcess.sleep(200)\n\nSupervisor.which_children(supervisor_pid) |> IO.inspect(label: \"Supervisors Children\")\n\n:sys.get_state(GenServer.whereis(:dummy1)) |> IO.inspect(label: \"Dummy 1 state\")\n:sys.get_state(GenServer.whereis(:dummy2)) |> IO.inspect(label: \"Dummy 2 state\")\n:sys.get_state(GenServer.whereis(:dummy3)) |> IO.inspect(label: \"Dummy 3 state\")\n\nKino.Process.render_sup_tree(supervisor_pid)\n```\n\nBased on the example, we can confirm that when using the `:one_for_one` supervision strategy, only the `:dummy2` GenServer process crashed and was subsequently restarted. As a result, the restarted process obtained a new process ID and its state was reset. On the other hand, the `:dummy1` and `:dummy3` processes continued to run without any interruption, maintaining their respective process IDs and states unchanged.\n\n## :one_for_all\n\nUpon restarting the CrashDummySupervisor with the `:one_for_all` restart strategy, if any child process terminates, all other child processes will be terminated as well. Following that, all child processes, including the terminated one, will be restarted.\n\n\n\n\n\n\n\nLet's proceed with restarting the `CrashDummySupervisor` using the `:one_for_all` strategy.\n\n```elixir\n# Stop the existing supervisor process\n# We used the module name as the Supervisor process name so we can use the module name to stop\n# the supervisor process.\n# This will also terminate the supervision tree and all process running under our supervisor\nSupervisor.stop(CrashDummySupervisor)\n\n{:ok, supervisor_pid} = CrashDummySupervisor.start_link(:one_for_all)\n\nSupervisor.which_children(supervisor_pid) |> IO.inspect(label: \"Supervisors Children\")\n\n:sys.get_state(GenServer.whereis(:dummy1)) |> IO.inspect(label: \"Dummy 1 state\")\n:sys.get_state(GenServer.whereis(:dummy2)) |> IO.inspect(label: \"Dummy 2 state\")\n:sys.get_state(GenServer.whereis(:dummy3)) |> IO.inspect(label: \"Dummy 3 state\")\n\nKino.Process.render_sup_tree(supervisor_pid)\n```\n\n```elixir\n# Makes the dummy2 child crash\nGenServer.cast(:dummy2, :boom)\n# Wait for the process to crash and be restarted\nProcess.sleep(200)\n\nSupervisor.which_children(supervisor_pid) |> IO.inspect(label: \"Supervisors Children\")\n\n:sys.get_state(GenServer.whereis(:dummy1)) |> IO.inspect(label: \"Dummy 1 state\")\n:sys.get_state(GenServer.whereis(:dummy2)) |> IO.inspect(label: \"Dummy 2 state\")\n:sys.get_state(GenServer.whereis(:dummy3)) |> IO.inspect(label: \"Dummy 3 state\")\n\nKino.Process.render_sup_tree(supervisor_pid)\n```\n\nThis time we can see that when the `:dummy_2` process crashed the supervisor restarted all the child processes. So the all processes now have a different pid.\n\n## :rest_for_one\n\nWith the `:rest_for_one` strategy, if a child process terminates, not only the terminated child process but also the **subsequent child processes** that were started after it will be terminated and restarted.\n\nThis strategy is useful when you want to restart only a portion of your supervision tree. In this case, when a process crashes, only the processes dependent on the crashed process will be restarted.\n\n#### Note:\n\nThe order in which child processes are specified in a supervision tree is crucial. A supervisor will attempt to start the child processes in the exact order specified in the supervisor child specification. Similarly, when a process crashes, the supervisor will restart the child processes in the same order.\n\nWhen a supervisor shuts down, it terminates all children in the reverse order in which they are listed.\n\n\n\n\n\n\n\nLet's see this strategy in action with our example. When the `:dummy2` process crashes, only the `:dummy2` and `:dummy3` processes will be restarted, while the `:dummy1` process will continue running.\n\n```elixir\nSupervisor.stop(CrashDummySupervisor)\n\n{:ok, supervisor_pid} = CrashDummySupervisor.start_link(:rest_for_one)\n\nSupervisor.which_children(supervisor_pid) |> IO.inspect(label: \"Supervisors Children\")\n\n:sys.get_state(GenServer.whereis(:dummy1)) |> IO.inspect(label: \"Dummy 1 state\")\n:sys.get_state(GenServer.whereis(:dummy2)) |> IO.inspect(label: \"Dummy 2 state\")\n:sys.get_state(GenServer.whereis(:dummy3)) |> IO.inspect(label: \"Dummy 3 state\")\n\nKino.Process.render_sup_tree(supervisor_pid)\n```\n\n```elixir\n# Makes the dummy2 child crash\nGenServer.cast(:dummy2, :boom)\n# Wait for the process to crash and be restarted\nProcess.sleep(200)\n\nSupervisor.which_children(supervisor_pid) |> IO.inspect(label: \"Supervisors Children\")\n\n:sys.get_state(GenServer.whereis(:dummy1)) |> IO.inspect(label: \"Dummy 1 state\")\n:sys.get_state(GenServer.whereis(:dummy2)) |> IO.inspect(label: \"Dummy 2 state\")\n:sys.get_state(GenServer.whereis(:dummy3)) |> IO.inspect(label: \"Dummy 3 state\")\n\nKino.Process.render_sup_tree(supervisor_pid)\n```\n\n### Resources\n\n\n\n* The images for the different restart stragies are taken from the [erlang documentation](https://www.erlang.org/doc/design_principles/sup_princ.html#restart-strategy)\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_5.3_restart_strategies.livemd",
"content": "# Supervisor restart strategies\n\n## Navigation\n\n\n\n## Restart Strategies\n\nIn the previous chapter, we learned about supervision strategies that determine whether a supervisor should restart its child processes when one of them crashes. However, it's important to consider when a process should be considered \"crashed.\" Processes can gracefully terminate, in which case we might not want to restart them, or they can crash due to errors.\n\nTo address this, restart strategies come into play. Unlike supervision strategies that apply to the entire supervision tree, restart strategies can be defined for each individual child process, allowing for more fine-grained control over their behavior.\n\n\n\nRestart values can be specified either in the child spec or when creating a GenServer.\n\nIn the child spec, the restart value can be set as follows:\n\n\n\n```elixir\nchildren = [\n %{\n id: :stack_1,\n start: {Stack, :start_link, []},\n restart: :temporary # <================= Here\n }\n]\n```\n\n#### Modifying default child spec\n\nAlternatively, when creating a GenServer, the restart value can be specified using the `use GenServer` macro:\n\n\n\n```elixir\nuse GenServer, restart: :transient\n```\n\nAs we learned earlier, GenServers provide a default `child_spec/1` function that automatically generates the child specification. By passing options directly to `use GenServer`, we can customize the `child_spec/1` function.\n\nTo understand the behavior of different restart options, let's create a simple GenServer and add it to the supervision tree with various restart options.\n\n```elixir\ndefmodule CrashDummyServer do\n use GenServer\n\n def start_link(name) do\n random_state = System.unique_integer([:positive])\n GenServer.start_link(__MODULE__, {random_state, name}, name: name)\n end\n\n ## Callbacks\n\n @impl true\n def init({random_value, name}) do\n IO.inspect(\"#{name} starting up!\")\n {:ok, random_value}\n end\n\n @impl true\n def handle_cast(:stop_gracefully, state) do\n # Returning this value makes the GenServer stop gracefully with :normal reason\n # If reason is neither :normal, :shutdown, nor {:shutdown, term} an error is logged.\n {:stop, :normal, state}\n end\n\n @impl true\n def handle_cast(:crash, state) do\n process_pid = self() |> inspect()\n raise \"BOOM! CrashDummyServer process: #{process_pid} crashed!\"\n {:noreply, state}\n end\nend\n```\n\n```elixir\ndefmodule CrashDummySupervisor do\n use Supervisor\n\n def start_link() do\n Supervisor.start_link(__MODULE__, :noop, name: __MODULE__)\n end\n\n @impl true\n def init(_) do\n # Supervision tree, start multiple instances of our genserver with different restart options\n children = [\n child_spec(:permanent_dummy, :permanent),\n child_spec(:temporary_dummy, :temporary),\n child_spec(:transient_dummy, :transient)\n ]\n\n Supervisor.init(children, strategy: :one_for_one)\n end\n\n defp child_spec(name, restart_strategy) do\n Supervisor.child_spec(\n {CrashDummyServer, name},\n id: name,\n # Specifying the restart strategy\n restart: restart_strategy\n )\n end\nend\n```\n\nIn the code snippet above, we have created a simple GenServer that crashes when receiving the `:boom` message and gracefully stops when receiving the `:stop_gracefully` message.\n\nWithin the Supervisor, we start three instances of this GenServer with three different restart strategies:\n\n* `:permanent`: This is the default restart strategy, where the child process is always restarted regardless of whether it crashes or is gracefully shut down.\n* `:temporary`: With this restart strategy, the child process is never restarted, even in the case of abnormal termination such as a crash. Any termination, even if it is abnormal, is considered successful.\n* `:transient`: The child process is restarted only if it terminates abnormally, meaning it exits with an exit reason other than `:normal`, `:shutdown`, or `{:shutdown, term}`.\n\nNow, let's test these restart strategies in action.\n\n\n\n---\n\n### `:permanent` restart strategy\n\n```elixir\n{:ok, supervisor_pid} = CrashDummySupervisor.start_link()\nSupervisor.which_children(supervisor_pid)\n```\n\n```elixir\n# Test graceful termination of child with `:permanent` restart strategy\n# Notice how the GenServer is restarted\nGenServer.cast(:permanent_dummy, :stop_gracefully)\n```\n\n```elixir\n# Test abnormal termination of child with `:permanent` restart strategy\n# Notice how the GenServer is restarted\nGenServer.cast(:permanent_dummy, :crash)\n```\n\n---\n\n### `:temporary` restart strategy\n\n```elixir\n# Test graceful termination of child with `:temporary` restart strategy\n# Notice how the GenServer is NOT restarted\nGenServer.cast(:temporary_dummy, :stop_gracefully)\n```\n\n```elixir\n# Notice how temporary_dummy is no longer present in the list of children\nSupervisor.which_children(supervisor_pid) |> IO.inspect(label: \"Supervisors Children\")\n```\n\n```elixir\n# Restart the Supervisor so that all child processes are start again\nSupervisor.stop(supervisor_pid)\n{:ok, supervisor_pid} = CrashDummySupervisor.start_link()\n```\n\n```elixir\n# Test abnormal termination of child with `:temporary` restart strategy\n# Notice how the GenServer is NOT restarted\nGenServer.cast(:temporary_dummy, :crash)\n```\n\n```elixir\n# Notice how temporary_dummy is no longer present in the list of children\nSupervisor.which_children(supervisor_pid) |> IO.inspect(label: \"Supervisors Children\")\n```\n\n---\n\n### `:transient` restart strategy\n\n```elixir\n# Restart the Supervisor so that all child processes are start again\nSupervisor.stop(supervisor_pid)\n{:ok, supervisor_pid} = CrashDummySupervisor.start_link()\n```\n\n```elixir\nSupervisor.which_children(supervisor_pid) |> IO.inspect(label: \"Supervisors Children\")\n\n# Test graceful termination of child with `:transient` restart strategy\n# Notice how the GenServer is NOT restarted since it was stopped gracefully\nGenServer.cast(:transient_dummy, :stop_gracefully)\n```\n\n```elixir\n# Notice how the transient child has a pid \"undefined` since its no longer running\nSupervisor.which_children(supervisor_pid)\n```\n\n```elixir\n# Restart the Supervisor so that all child processes are start again\nSupervisor.stop(supervisor_pid)\n{:ok, supervisor_pid} = CrashDummySupervisor.start_link()\n```\n\n```elixir\n# Test abnormal termination of child with `:transient` restart strategy\n# Notice how the GenServer is restarted since it was stopped abnormally\nGenServer.cast(:transient_dummy, :crash)\n```\n\n```elixir\n# Notice how transient_dummy was restarted and all children are running\nSupervisor.which_children(supervisor_pid) |> IO.inspect(label: \"Supervisors Children\")\n```\n\n## The Max Restarts option\n\nSo far, we have covered various options in the supervisor's child specifications, such as `:id`, `:strategy`, `:name`, and `:restart`. Now, let's explore the remaining options that the child specification supports.\n\nTwo important options are:\n\n* `:max_restarts`: This option sets the maximum number of restarts allowed within a specified time frame. By default, it is set to 3.\n\n* `:max_seconds`: This option defines the time frame in which the `:max_restarts` limit applies. The default value is 5 seconds.\n\nThese options determine the maximum **restart intensity** of a supervisor, controlling the number of restarts that can occur within a given time interval. It is part of the supervisor's built-in mechanism to manage restarts effectively.\n\n**If the number of restarts exceeds the `:max_restarts` limit within the last `:max_seconds` seconds, the supervisor terminates all its child processes and itself**. In this case, the termination reason for the supervisor is `:shutdown`.\n\nWhen a supervisor terminates, the next higher-level supervisor takes action. It either restarts the terminated supervisor or terminates itself.\n\nThe restart mechanism is designed to prevent a scenario where a process repeatedly crashes for the same reason, only to be restarted again and again.\n\n## Shutdown strategy\n\nWhen defining child specifications for a supervisor, we have the option to include the `:shutdown` option, which determines how the supervisor shuts down its child processes.\n\nThe `:shutdown` option has three possible values:\n\n* *An integer greater than or equal to 0*: This specifies the amount of time in milliseconds that the supervisor will wait for its children to terminate after sending a `Process.exit(child, :shutdown)` signal. If the child process does not trap exits, it will be terminated immediately upon receiving the `:shutdown` signal. If the child process traps exits, it has the specified amount of time to terminate. If it fails to terminate within the specified time, the supervisor will forcefully terminate the child process using `Process.exit(child, :kill)`.\n\n* `:brutal_kill`: This option causes the child process to be unconditionally and immediately terminated using `Process.exit(child, :kill)`. That is the supervisor will not wait for the child process to terminate gracefully but will immediately kill the process.\n\n* `:infinity`: With this option, the supervisor will wait indefinitely for the child process to terminate.\n\nBy default, the `:shutdown` option is set to `5_000` (5 seconds), which means the supervisor will wait for a maximum of 5 seconds for the child process to shut down gracefully. If the child process does not terminate within this time frame, the supervisor will forcefully terminate it using `Process.exit(child, :kill)`.\n\nThese options provide flexibility in managing the shutdown behavior of child processes in a supervisor.\n\n## Key points\n\n* During startup, a supervisor processes all child specifications and starts each child in **the order they are defined**. This is achieved by invoking the function specified under the `:start` key in the child specification, typically `start_link/1`.\n\n* When a supervisor initiates shutdown, it terminates its children in the **reverse order of their listing**. This termination process involves sending a shutdown exit signal, `Process.exit(child_pid, :shutdown)`, to each child process and waiting for a specified time interval for them to terminate. The default interval is 5000 milliseconds.\n\n* If a child process is not trapping exits, it will immediately shut down upon receiving the first exit signal. On the other hand, if a child process is trapping exits, it will invoke the terminate callback and must terminate within a reasonable time before the supervisor forcefully terminates it.\n\n* When an Elixir application exits, the termination propagates down the supervision tree. *Supervisors always trap exits* for various reasons, so they attempt to stop all their children upon receiving an exit signal. This is achieved by sending an exit signal to each child individually, allowing a timeout period before resorting to a brutal termination with `:kill`. The duration of this timeout is determined by the `shutdown` option specified in the child specification.\n\n* Once all children are stopped, the supervisor itself stops as well, resulting in the orderly shutdown of the supervision tree.\n\nThese points highlight the startup and shutdown behavior of supervisors, the termination process for child processes, and the flow of exit signals within the supervision tree.\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_5.4_introduction_to_dynamic_supervisor.livemd",
"content": "# Introduction to Dynamic Supervisor\n\n```elixir\nMix.install([\n {:kino, \"~> 0.9.0\"}\n])\n```\n\n## Navigation\n\n\n\n## The Dynamic Supervisor\n\nIn the previous chapter, we learned about the Supervisor behavior, which enables us to supervise processes and restart them in case of failures, ensuring fault tolerance. However, the Supervisor behavior requires us to specify all the child processes it will supervise in advance as child specifications. In other words, **the Supervisor module was primarily designed to handle static children.**\n\nWhen the supervisor starts, it creates and starts the child processes in the specified order, and when the supervisor is stopped, it terminates the processes in the reverse order.\n\nOn the other hand, a DynamicSupervisor **starts with no children** initially. Instead, children are started **on demand** using the `start_child/2` function, and there is **no specific ordering** between the children. This provides a lot of flexibility as we can dynamically add and remove child processes to be supervised. The DynamicSupervisor can efficiently handle a large number of children by utilizing optimized data structures and perform certain operations, such as shutting down, concurrently.\n\n\n\n### Key points\n\n* DynamicSupervisor is a specialized type of Supervisor designed to handle dynamic children. Note that in Erlang we have only one supervisor. These behaviors like DynamicSupervisor and PartitionSupervisor are abstraction built on top of the basic Supervisor to address common use cases more conveniently.\n\n* Dynamic supervisors start without any children initially, and there is no predefined ordering between the children. Children can be added to the supervisor dynamically as needed, without any specific sequence or arrangement.\n\n* The only available supervision strategy for DynamicSupervisor is `:one_for_one`.\n\n* The `id` of a child in a DynamicSupervisor is always `:undefined`. This is because dynamically supervised children are created from the same child specification, and assigning a specific id to each child would result in conflicts.\n\n\n\n### Supervisor.start_child/2 vs DynamicSupervisor.start_child/2\n\nIt may appear confusing that both the Supervisor and DynamicSupervisor modules provide a function called `start_child/2` to dynamically start supervised child processes. This raises the question of what distinguishes the two and why we have a dedicated DynamicSupervisor for dynamic child management.\n\nWhile it is possible to dynamically start and stop children from a standard Supervisor, the DynamicSupervisor is specifically designed to excel in this use case. There are differences in how a DynamicSupervisor handles its children compared to a regular supervisor. For instance, a DynamicSupervisor does not impose an inherent ordering among its children.\n\nOn restart a DynamicSupervisor starts empty while a regular Supervisor typically starts along with all the child process defined in its child specifications. A DynamicSupervisor can concurrently shuts down all children when restarted unlike a standard supervisor which follows a specific restart order.\n\nFurthermore, the DynamicSupervisor module provides additional options, such as `:max_children`, which allows setting a limit on the maximum number of dynamically supervised children.\n\nTherefore it just more idiomatic and optimal to use a DynamicSupervisor instead of the regular Supervisor module when trying to dynamically start/stop supervised processes.\n\n## Usage\n\nJust like the regular supervisor module the DynamicSupervisor can either be started directly or defined as a module.\n\nLets look at some examples...\n\n```elixir\nchildren = [{DynamicSupervisor, name: MyTestDynamicSupervisor}]\n\n# Th only possible strategy with DynamicSupervisor is :one_for_one\n{:ok, supervisor_pid} = Supervisor.start_link(children, strategy: :one_for_one)\n```\n\nWe will now create a simple GenServer that we can start under this supervisor\n\n```elixir\ndefmodule TestServer do\n use GenServer\n\n def start_link(name) do\n GenServer.start_link(__MODULE__, :noop, name: name)\n end\n\n ## Callbacks\n\n @impl true\n def init(_arg) do\n {:ok, :noop}\n end\n\n @impl true\n def handle_call({:echo, arg}, _from, state) do\n {:reply, arg, state}\n end\nend\n```\n\nNow we can use the `DynamicSupervisor.start_child(supervisor, child_spec)` function to dynamically start a child process under the supervisor. Notice how we need to pass a child spec to the function.\n\n```elixir\n{:ok, echo_1} = DynamicSupervisor.start_child(MyTestDynamicSupervisor, {TestServer, :echo1})\n{:ok, echo_2} = DynamicSupervisor.start_child(MyTestDynamicSupervisor, {TestServer, :echo2})\n```\n\n```elixir\n# Lets visualize the supervision tree\nKino.Process.render_sup_tree(supervisor_pid)\n```\n\nNotice how we dynamically started 2 instances of our `TestServer` GenServer process under the `MyTestDynamicSupervisor` DynamicSupervisor.\n\n```elixir\nDynamicSupervisor.count_children(MyTestDynamicSupervisor)\n```\n\n```elixir\n# Notice how the id is undefined for a DynamicSupervisor\nDynamicSupervisor.which_children(MyTestDynamicSupervisor)\n```\n\nWe can easily terminate a dynamically started child\n\n```elixir\nDynamicSupervisor.terminate_child(MyTestDynamicSupervisor, echo_2)\n```\n\n```elixir\nDynamicSupervisor.count_children(MyTestDynamicSupervisor)\n```\n\n```elixir\nDynamicSupervisor.which_children(MyTestDynamicSupervisor)\n```\n\n---\n\n### Module based DynamicSupervisor\n\nNow lets use a module based DynamicSupervisor. Just like the regular Supervisor behaviour the DynamicSupervisor behaviour only has one callback that we must implement that is the `init/1` callback.\n\nAlso similar to the regular Supervisor module when starting a DynamicSupervisor we can pass options like `:name`, `:strategy`, `:max_restarts` and `:max_seconds`.\n\nTwo new options that are available with DynamicSupervisors are\n\n* `:max_children` - the maximum amount of children to be running under this supervisor at the same time. When `:max_children` is exceeded, `start_child/2` returns `{:error, :max_children}`. Defaults to `:infinity`.\n\n* `:extra_arguments` - arguments that are prepended to the arguments specified in the child spec given to `start_child/2`. Defaults to an empty list.\n\nTo understand this better lets look at an example:\n\n```elixir\n# A simple GenServer module which we would start under our supervisor\ndefmodule TestServerV2 do\n use GenServer\n\n def start_link(extra_arg, name, arg) do\n GenServer.start_link(__MODULE__, [extra_arg, arg], name: name)\n end\n\n ## Callbacks\n\n @impl true\n def init([extra_arg, arg]) do\n IO.inspect(\n \"New TestServerV2 started with extra_arg = #{inspect(extra_arg)} and arg = #{inspect(arg)}\"\n )\n\n {:ok, :noop}\n end\n\n @impl true\n def handle_call({:echo, arg}, _from, state) do\n {:reply, arg, state}\n end\nend\n```\n\n```elixir\ndefmodule MyTestDynamicSupervisorV2 do\n # The DynamicSupervisor behaviour that defines a default child_spec/1\n use DynamicSupervisor\n\n def start_link(init_arg) do\n DynamicSupervisor.start_link(__MODULE__, init_arg, name: __MODULE__)\n end\n\n # A public api to easily start child process under this supervisor\n def start_child(name, arg) do\n child_spec = %{id: TestServerV2, start: {TestServerV2, :start_link, [name, arg]}}\n\n # This will start an child process of the TestServerV2 by calling\n # TestServerV2.start_link(init_arg, name, arg)\n DynamicSupervisor.start_child(__MODULE__, child_spec)\n end\n\n @impl true\n def init(init_arg) do\n # Returns a tuple containing the supervisor initialization options.\n DynamicSupervisor.init(\n strategy: :one_for_one,\n max_children: 2,\n extra_arguments: [init_arg]\n )\n |> IO.inspect(label: \"DynamicSupervisor initialized with\")\n end\nend\n```\n\nFew things to note in the above code snippets:\n\n* We are using the `DynamicSupervisor.init/1` helper function to generate a tuple that initializes the dynamic supervisor with proper options in its `init/1` callback.\n\n* We have added a helper function `MyTestDynamicSupervisorV2.start_child/2` to dynamically start supervised child processes under our dynamic supervisor.\n\n* We have passed additional options like the `max_children` to limit the number of children the dynamic supervisor can start.\n\n* The `extra_arguments: [init_arg]` option will automatically prepend the `init_arg` argument to every child process started under this supervisor. This is especially useful if we want to always send a specific argument to every child process that is started under this supervisor.\n\n[Note: Similar to the regular supervisor module the DynamicSupervisor module also defines a default `child_spec/1` function so we can use shorthand syntax when defining child specs to pass to `DynamicSupervisor.start_child/2`]\n\n```elixir\n{:ok, supervisor_pid} = MyTestDynamicSupervisorV2.start_link(\"Elixir is โค\")\n```\n\nNow let us dynamically start and stop child process under our supervisor.\n\n```elixir\n{:ok, echo_1} = MyTestDynamicSupervisorV2.start_child(:echov2_1, :yolo)\n{:ok, echo_2} = MyTestDynamicSupervisorV2.start_child(:echov2_2, :awesome_elixir)\n```\n\nNotice how the child processes that were started have received the \"Elixir is โค\" specified as `:extra_arguments` along with the arguments that were passed.\n\n```elixir\nDynamicSupervisor.count_children(MyTestDynamicSupervisor) |> IO.inspect()\nDynamicSupervisor.which_children(MyTestDynamicSupervisor)\n```\n\n```elixir\nDynamicSupervisor.start_child(MyTestDynamicSupervisorV2, {TestServerV2, :echo3})\n```\n\n```elixir\n# Lets visualize the supervision tree\nKino.Process.render_sup_tree(supervisor_pid)\n```\n\n```elixir\nDynamicSupervisor.terminate_child(MyTestDynamicSupervisorV2, echo_1)\nDynamicSupervisor.count_children(MyTestDynamicSupervisor) |> IO.inspect()\nDynamicSupervisor.which_children(MyTestDynamicSupervisor)\n```\n\nIn future chapters, we will delve into the topic of scaling a DynamicSupervisor by utilizing a PartitionSupervisor. We will also go through more examples of how to use dynamic supervisors in real use cases.\n\n## Resources\n\n* https://hexdocs.pm/elixir/DynamicSupervisor.html\n* https://elixirforum.com/t/different-between-supervisor-start-child-and-dynamicsupervisor-start-child/14585/3\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_5.5_partition_supervisor.ex.livemd",
"content": "# The Partition Supervisor\n\n```elixir\nMix.install([\n {:kino, \"~> 0.9.0\"}\n])\n```\n\n## Navigation\n\n\n\n## Introduction\n\nA PartitionSupervisor functions similarly to a regular supervisor, but with the added capability of creating partitions.\n\nWhen a PartitionSupervisor is started, it *will create multiple partitions and will start a process under each of the partitions*.\n\nThis feature becomes particularly valuable when certain processes within a system have the potential to become bottlenecks. If these processes can easily partition their state without any interdependencies, the PartitionSupervisor can be used.\n\nBy starting multiple instances of such processes across different partitions, the workload can be distributed and potential bottlenecks can be avoided.\n\n## Usage\n\nOnce a PartitionSupervisor is started, we can dispatch messages to its children using the `{:via, PartitionSupervisor, {name, key}}`. Here, `name` refers to the name of the PartitionSupervisor, and `key` is used for routing the message.\n\nThe PartitionSupervisor uses a routing strategy to determine the appropriate partition to which a message should be dispatched. When sending a message to a child process under a PartitionSupervisor, we provide a `key`. Depending on the routing strategy in place, the PartitionSupervisor will utilize this key to select the specific partition to which the message should be sent.\n\nLet's explore an example to gain a better understanding of this concept.\n\n\n\nLets create a simple GenServer which we can start under the partition supervisor\n\n```elixir\ndefmodule EchoServer do\n use GenServer\n\n def start_link(args) do\n GenServer.start_link(__MODULE__, args)\n end\n\n @impl true\n def init(args) do\n IO.inspect(\"EchoServer #{inspect(self())} started with args: #{inspect(args)}\")\n {:ok, :noop}\n end\n\n @impl true\n def handle_call({:echo, msg}, _from, state) do\n IO.inspect(\"EchoServer(#{inspect(self())}) echoing: #{inspect(msg)}\")\n {:reply, msg, state}\n end\nend\n```\n\nNows lets start a partition supervisor\n\n```elixir\n{:ok, supervisor_pid} =\n PartitionSupervisor.start_link(\n name: EchoServerPartitionSupervisor,\n # Use the default child_spec/1 function of the GenServer\n child_spec: EchoServer.child_spec(:test_arg)\n )\n```\n\nNow lets visualize the supervision tree.\n\n```elixir\nKino.Process.render_sup_tree(supervisor_pid)\n```\n\nFrom the above output we can now see that multiple processes of the `EchoServer` GenServer were started by the Partition Supervisor. A separate instance of the `EchoServer` was started for each partition that was created.\n\nBy default the number of partitions a PartitionSupervisor will create is equal to `System.schedulers_online()`(typically the number of CPU cores).\n\n```elixir\nSystem.schedulers_online()\n```\n\nThe number of processes(partitions) we see in the supervision tree must match the above output returned from `System.schedulers_online()`.\n\n\n\nThe PartitionSupervisor provides additional options that can be passed during its initialization:\n\n* `:partitions` - This option accepts a positive integer value that represents the number of partitions to create. By default, it is set to `System.schedulers_online()`, which corresponds to the number of online schedulers in the system.\n\n* `:with_arguments` - A two-argument anonymous function that allows the partition to be given to the child starting function.\n\nIn addition to these specific options, other common options such as `:name`, `:child_spec`, `:max_restarts`, and `:max_seconds` can be used with the PartitionSupervisor, and they function as they do in regular supervisors.\n\nNow lets restart our PartitionSupervisor with some of these options to customize its behaviour...\n\n```elixir\n# Defining a new echo server GenServer with a start_link/2 function\n# to also receive the partition number as an argument.\ndefmodule EchoServerV2 do\n use GenServer\n\n def start_link(args, partition_number) do\n GenServer.start_link(__MODULE__, [args, partition_number])\n end\n\n @impl true\n def init([args, partition_number]) do\n IO.inspect(\n \"EchoServer #{inspect(self())} started on partition #{partition_number} with args: #{inspect(args)}\"\n )\n\n # We save the partition number in the GenServer state\n {:ok, partition_number}\n end\n\n @impl true\n def handle_call({:echo, msg}, _from, partition_number) do\n IO.inspect(\n \"EchoServer(#{inspect(self())})(partition=#{partition_number}) echoing: #{inspect(msg)}\"\n )\n\n {:reply, msg, partition_number}\n end\nend\n```\n\n```elixir\n# Stop the existing supervisor\n:ok = PartitionSupervisor.stop(EchoServerPartitionSupervisor)\n\n# Start the EchoServerPartitionSupervisor again with added options\n{:ok, supervisor_pid} =\n PartitionSupervisor.start_link(\n name: EchoServerPartitionSupervisor,\n child_spec: EchoServerV2.child_spec(:test_arg),\n\n # We explicitly specify the number of partitions to create\n partitions: 3,\n with_arguments: fn [existing_args], partition ->\n # Inject the partition number into the args given to the child process\n # This will be passed to the child process when it is started via the\n # `start_link(args, partition_number)` function.\n [existing_args, partition]\n end\n )\n```\n\n```elixir\nKino.Process.render_sup_tree(supervisor_pid)\n```\n\nNotice that this time only 3 partitions were created and 3 child processes were started.\nAlso notice how the partition number was passed as an argument to every child process, this is due to the use of the `with_arguments` option.\n\nThe `with_arguments` option allows us to customize the arguments passed to child processes in a partitioned supervision setup. By providing a two-argument anonymous function, we can include the partition number in the arguments used to start each child process. This **allows each process to have knowledge of the partition_number on which it is running**.\n\n\n\n### Sending messages\n\nTo send a message to a child process under a PartitionSupervisor, we can use the `{:via, PartitionSupervisor, {name, key}}` tuple. Here key is used for routing the message to the appropriate partition.\n\nBy using this message dispatching method, we can effectively send messages to specific child processes running under the PartitionSupervisor based on the key that we pass.\n\n```elixir\n# Send a message to the EchoServer running on partition 0\n:hi =\n GenServer.call(\n {:via, PartitionSupervisor, {EchoServerPartitionSupervisor, 0}},\n {:echo, :hi}\n )\n\n# Send a message to the EchoServer running on partition 1\n:ola =\n GenServer.call(\n {:via, PartitionSupervisor, {EchoServerPartitionSupervisor, 1}},\n {:echo, :ola}\n )\n\n# Send a message to the EchoServer running on partition 2\n:adios =\n GenServer.call(\n {:via, PartitionSupervisor, {EchoServerPartitionSupervisor, 2}},\n {:echo, :adios}\n )\n\n# Send a message to the EchoServer running on partition 1\n# (the routing key 1000 results in partition 1 to be selected)\nGenServer.call(\n {:via, PartitionSupervisor, {EchoServerPartitionSupervisor, 1000}},\n {:echo, :boom}\n)\n```\n\nWhen using integer keys with the PartitionSupervisor, the routing strategy is determined by the formula `rem(abs(key), partitions)`. In the example we provided, the message with the key `1000` was sent to partition 1 because `rem(abs(1000), 3) = rem(1000, 3) = 1`.\n\nHowever, if the routing key is not an integer, the `:erlang.phash2(key, partitions)` hash function is used as the routing strategy. This function calculates a hash value based on the key and the number of partitions, resulting in the selection of the appropriate partition to which the message should be dispatched.\n\n```elixir\n:erlang.phash2(\"1000\", 3) |> IO.inspect(label: \"Partition\")\n\nGenServer.call(\n {:via, PartitionSupervisor, {EchoServerPartitionSupervisor, \"1000\"}},\n {:echo, :hello_world}\n)\n```\n\nIf we want to retrieve the PID of the process running on a partition for a certain key, we can use `GenServer.whereis({:via, PartitionSupervisor, {name, key}})`\n\n```elixir\n# Get the PID of the process running in the partition that would be\n# selected when using \"1000\" as the key\nGenServer.whereis({:via, PartitionSupervisor, {EchoServerPartitionSupervisor, \"1000\"}})\n```\n\n### Implementation detail\n\nThe PartitionSupervisor uses either an ETS table or a `Registry` to manage all of the partitions. Under the hood, the PartitionSupervisor generates a child spec for each partition and then acts as a regular supervisor. The ID of each child spec is the partition number.\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_5.6_scaling_dynamic_supervisor.livemd",
"content": "# Scaling Dynamic Supervisors\n\n```elixir\nMix.install([\n {:kino, \"~> 0.9.0\"}\n])\n```\n\n## Navigation\n\n\n\n## The scalability problem with DynamicSupervisors\n\nIn previous chapters, we learned about the DynamicSupervisor, which is effective for dynamically spawning and supervising child processes. However, in certain scenarios, the DynamicSupervisor can become a bottleneck.\n\nThe DynamicSupervisor operates as a single process responsible for starting other processes. In high-demand situations where there are numerous requests to start new child processes, the DynamicSupervisor may struggle to keep up. Additionally, if a child process experiences delays during initialization (e.g., being stuck in the `init/1` callback), it can block the DynamicSupervisor and prevent it from starting new child processes.\n\nLets simulate such a situation with an example...\n\n```elixir\n# A minimal GenServer which takes 2 seconds to initialize\ndefmodule SlowGenServer do\n use GenServer\n\n def start_link(args) do\n GenServer.start_link(__MODULE__, args)\n end\n\n @impl true\n def init(_args) do\n # Simulate slow start of a GenServer by sleeping for 1 second\n :timer.sleep(1000)\n IO.inspect(\"Started new SlowGenServer #{inspect(self())}\")\n {:ok, :noop}\n end\nend\n```\n\nNote: In real scenarios, it's important to avoid performing time-consuming tasks in the `init/1` callback of a GenServer. Instead, we should leverage the `handle_continue/2` callback to handle long-running tasks and prevent them from blocking the GenServer startup process. However, for the purpose of this example, let's proceed with trying it out.\n\n```elixir\n# Start a DynamicSupervisor named \"MySlowDynamicSupervisor\"\n{:ok, supervisor_pid} =\n DynamicSupervisor.start_link(\n name: MySlowDynamicSupervisor,\n # Use the default child_spec/1 function of the GenServer\n child_spec: DynamicSupervisor.child_spec([])\n )\n```\n\nNow lets try to simultaneously add 5 child processes under our DynamicSupervisor.\n\n```elixir\n# Lets start 5 instances of the SlowGenServer process under the DynamicSupervisor\nfor _i <- 1..5 do\n # Start a new process that in turn starts a new child process under the dynamic supervisor\n spawn(fn -> DynamicSupervisor.start_child(MySlowDynamicSupervisor, SlowGenServer) end)\nend\n```\n\nNotice how the DynamicSupervisor starts each child process one by one and is blocked until the previous child process is started. In real-world scenarios, this can cause significant delays in starting child processes under a single DynamicSupervisor, resulting in potential bottlenecks and performance issues.\n\n\n\nLets visualize the resulting supervision tree\n\n```elixir\nKino.Process.render_sup_tree(supervisor_pid)\n```\n\nNotice how all the 5 instances of the `SlowGenServer` process are spawned under the same `MySlowDynamicSupervisor` instance.\n\n## Using a PartitionSupervisor to scale DynamicSupervisors\n\nTo address the aforementioned problem, we can use a PartitionSupervisor to start multiple instances of the DynamicSupervisor. The **PartitionSupervisor acts as a supervisor for multiple DynamicSupervisor processes, each running in a separate partition**.\n\nWhen a new child process needs to be started, the PartitionSupervisor selects one of the DynamicSupervisor processes to handle the request. This distribution of child process creation across multiple DynamicSupervisors helps distribute the workload and prevents bottlenecks that can occur when relying on a single DynamicSupervisor.\n\nLets see this in action...\n\n```elixir\n# Stop the existing dynamic supervisor\nSupervisor.stop(MySlowDynamicSupervisor)\n\n# Start a partition supervisor with a dynamic supervisor as the child process for each partition\n{:ok, supervisor_pid} =\n PartitionSupervisor.start_link(\n name: MySlowPartitionSupervisor,\n # Create 6 partitions\n partitions: 6,\n # Use the default child_spec/1 function of DynamicSupervisor\n child_spec: DynamicSupervisor.child_spec([])\n )\n```\n\nIn the code above, we start a partition supervisor that will by create six partitions and will start a dynamic supervisor for each partition.\n\n```elixir\nKino.Process.render_sup_tree(supervisor_pid)\n```\n\nNow, instead of directly calling the DynamicSupervisor by its name, we access it through the PartitionSupervisor using the `{:via, PartitionSupervisor, {partition_supervisor_name, key}}` format.\n\nNow lets try again to start 6 child processes under the supervisor\n\n```elixir\nfor i <- 1..5 do\n # Start a new process that in turn starts a new child process under one of the\n # dynamic supervisors via the partition supervisor\n spawn(fn ->\n DynamicSupervisor.start_child(\n {:via, PartitionSupervisor, {MySlowPartitionSupervisor, i}},\n SlowGenServer\n )\n end)\nend\n```\n\nIn the provided code, we spawn five new processes, and each process starts a new child process under one of the dynamic supervisors via the partition supervisor.\n\nWe use the numbers 1 to 5 as the routing keys for each child process. With six partitions available, each child process will be started under a separate dynamic supervisor.\n\n\n\nLets visualize the resulting supervision tree\n\n```elixir\nKino.Process.render_sup_tree(supervisor_pid)\n```\n\nIn the provided supervision tree, we can observe that five instances of the DynamicSupervisor were started under the `MySlowPartitionSupervisor` PartitionSupervisor. Each of these dynamic supervisors represents a separate partition.\n\nFurthermore, under each dynamic supervisor, a separate instance of the `SlowGenServer` process was started.\n\n\n\n---\n\nBy leveraging the PartitionSupervisor as the entry point, we can abstract away the details of the individual dynamic supervisors and rely on the routing strategy to handle the selection of the appropriate dynamic supervisor for starting the child processes. This approach allows for efficient distribution of child processes across multiple dynamic supervisors, reducing the load on any single supervisor and avoiding potential bottlenecks.\n\nAs a result, the child processes are started much faster compared to the previous example, where we relied on a single dynamic supervisor.\n\n\n\nNote: In most real-world scenarios, the supervisor and partition supervisor are typically started as part of the application's supervision tree. Instead of manually calling start_link/1, we can define the supervisors and their child specifications in the application module.\n\nHere's an example of how we can start the partition supervisor under a supervision tree:\n\n\n\n```elixir\ndefmodule MyApp.Application do\n use Application\n\n def start(_type, _args) do\n children = [\n {PartitionSupervisor,\n child_spec: DynamicSupervisor,\n name: MySlowPartitionSupervisor}\n ]\n\n opts = [strategy: :one_for_one]\n Supervisor.start_link(children, opts)\n end\nend\n```\n\n\n\n### Resources:\n\n* https://hexdocs.pm/elixir/1.15.0-rc.0/PartitionSupervisor.html#content\n* https://blog.appsignal.com/2022/09/20/fix-process-bottlenecks-with-elixir-1-14s-partition-supervisor.html\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_6.0_project_building_a_download_manager.livemd",
"content": "\n\n# Project - Building a Download Manager ๐\n\n```elixir\nMix.install([\n {:kino, \"~> 0.9.0\"},\n {:elixir_uuid, \"~> 1.2\"},\n {:httpoison, \"~> 2.1\"},\n {:sizeable, \"~> 1.0\"}\n])\n```\n\n## Navigation\n\n\n\n## Introduction\n\nIn this chapter, we will put into practice the concepts we have learned in the previous chapters by building a project. Our project will be a simple download manager that has the capability to download multiple files simultaneously and provide status updates for each download. A key requirement is that the failure of one download should not impact the progress or completion of other ongoing downloads.\n\nBy building this download manager, we will explore the use of supervisors and dynamic supervisors to handle the concurrent downloading of files, ensuring fault tolerance and isolation between download processes. We will also dive into message passing and state management to track the progress and status of each download.\n\n## The Download struct\n\nWe represent each download using a struct that contains the following fields:\n\n* `id`: A unique identifier for the download.\n* `name`: The name of the download.\n* `src`: The source URL from where the download is initiated.\n* `dest`: The destination file path where the downloaded file will be saved.\n* `from`: The process ID (PID) of the requester process who initiated the download.\n* `pid`: The process ID (PID) of the worker process that's downloading the file.\n* `status`: The current status of the download.\n* `size`: The size of the download in bytes.\n* `bytes_downloaded`: The number of bytes downloaded for the download.\n* `exit_status`: The exit status of the downloading process, if applicable.\n* `error_status`: The reason for a failed download, if applicable.\n* `start_time`: The timestamp when the download started.\n* `end_time`: The timestamp when the download finished.\n* `resp`: The response received from the source URL when downloading the file.\n* `fd`: The file descriptor of the downloaded file on disk.\n\nThese fields provide essential information to track and manage the progress, status, and details of each download within our download manager. By utilizing this struct, we can effectively handle multiple concurrent downloads, monitor their progress, and report their status accurately.\n\n```elixir\ndefmodule Download do\n @enforce_keys [:id, :src, :dest, :from]\n defstruct [\n :id,\n :name,\n :src,\n :dest,\n :from,\n :pid,\n :size,\n :status,\n :bytes_downloaded,\n :exit_status,\n :error_reason,\n :start_time,\n :end_time,\n :resp,\n :fd\n ]\nend\n```\n\n## Architecture\n\nLets look at the high level architecture of our application.\n\n\n\n\n\n\n\nOur application will consist of 3 important modules.\n\nThe high-level tasks of each module are as follows:\n\n* **DownloadManager**: The DownloadManager module is a GenServer that stores and tracks the state of all download worker processes. It provides APIs to add, remove, and retrieve download information. It also periodically updates the status of downloads and handles termination messages from download workers.\n\n* **DownloadsSupervisor**: The DownloadsSupervisor module is a dynamic supervisor that allows the dynamic spawning of DownloadWorker processes. It starts and supervises individual DownloadWorker processes to handle each download. It provides functions to start and terminate child processes.\n\n* **DownloadWorker**: The DownloadWorker module is a GenServer that is responsible for performing the actual download task. It runs as an individual process and handles downloading chunks of data from a given source URL. It communicates with the DownloadManager module to update the download status and handle completion or failure of the download.\n\nOverall, the DownloadManager orchestrates the download process, the DownloadsSupervisor manages the lifecycle of DownloadWorker processes, and the DownloadWorker handles the actual downloading task.\n\n\n\nNow lets go through the implementation of each of these modules in detail.\n\n## The Download Worker\n\nThe `DownloadWorker` module is a GenServer responsible for handling individual file downloads. Each download is performed by its own process, represented by an instance of the `DownloadWorker` GenServer.\n\nFor downloading the file we utilize the [httpoison](https://github.com/edgurgel/httpoison) library and its async download feature, which allows us to stream chunks of the downloaded file.\n\nThis happens using the `HTTPoison.get(src, %{}, stream_to: self(), async: :once)` function call. The `stream_to: self()` option enables the `DownloadWorker` process to receive each downloaded chunk via the `handle_info/2` callback.\n\nThe `async: :once` option ensures that only one chunk at a time is sent to our `DownloadWorker` process. Once we have processed a chunk, we can request the next chunk by calling `HTTPoison.stream_next/1`.\n\nThe lifecycle of the `DownloadWorker` process is as follows:\n\n* `init/1`: In the `init` callback, we prepare the download struct, initialize the process, and kick off the download by returning `{:ok, new_download, {:continue, :kickoff}}`. This ensures that the `handle_continue/2` callback will be called immediately after the `init/1` callback.\n\n* `handle_continue/2`: After the initialization, in the `handle_continue` callback, we open the file where the download will be saved and initiate the download using `HTTPoison.get/3`. We save the file descriptor in the download struct as the GenServer state. If any errors occur during the download, we update the download status accordingly and return `{:stop, reason, failed_download}` to stop the GenServer with the appropriate failure reason.\n\n* `handle_info/2`: This callback is responsible for handling various events and actions during the download process. Here are the different scenarios:\n\n * `handle_info(%HTTPoison.AsyncHeaders{headers: headers}, download)`: Invoked when the download begins, providing headers containing metadata about the download, such as the file size. We save this information in the download struct as the GenServer state and request the first chunk using `HTTPoison.stream_next/1`.\n\n * `handle_info(%HTTPoison.AsyncStatus{}, download)`: Requests the next chunk of the download by calling `HTTPoison.stream_next/1`.\n\n * `handle_info(%HTTPoison.AsyncChunk{chunk: chunk}, download)`: Saves a received chunk by appending it to the file on disk. We then request the next chunk using `HTTPoison.stream_next/1`.\n\n * `handle_info(%HTTPoison.AsyncEnd{}, download)`: Triggered when the download is completed. We close the file descriptor, update the download status, and return `{:stop, :finish, finished_download}` to gracefully terminate the download worker process.\n\n * The remaining `handle_info/2` callbacks handle errors that occur during the download process, such as status code errors or HTTPoison errors. In such cases, we update the download status accordingly and terminate the download worker process by returning `{:stop, message, failed_download}`.\n\n* `handle_call(:status, _from, download)`: This callback handles the `:status` call and simply returns the download struct, which represents the current state of the download. It can be used to inform the caller about the download status.\n\n* `terminate/2`: This callback is invoked when the download worker process is about to exit. By enabling the `:trap_exit` flag in the `init/1` callback, we can trap exits and perform cleanup operations if the download process stops either due to failures or when a download finishes successfully.\n In this callback we inform the caller process (identified by the `:from` property in the download struct) by sending a `{:terminating, id, download}` message. We then close the file descriptor and gracefully shut down the process.\n\nLastly, note that the `DownloadWorker` GenServer has the `restart: :temporary` option set, which means that any failed process won't be automatically restarted by the supervisor. In our case this is the expected behaviour since we already handle failed downloads and don't want to retry them.\n\n```elixir\ndefmodule DownloadWorker do\n @moduledoc \"\"\"\n Worker Genserver for downloading a file\n \"\"\"\n use GenServer, restart: :temporary\n alias Download\n require Logger\n\n def start_link(args, opts \\\\ []), do: GenServer.start_link(__MODULE__, args, opts)\n\n # Callbacks\n\n @impl GenServer\n def init(%Download{} = download) do\n Logger.info(\"Start new download worker: #{download.id}\")\n\n # Makes your process call terminate/2 upon exit.\n Process.flag(:trap_exit, true)\n\n # Prepare the new download struct\n new_download = %Download{\n id: download.id,\n src: download.src,\n dest: download.dest,\n from: download.from,\n size: 0,\n status: :initiate,\n bytes_downloaded: 0,\n exit_status: nil,\n error_reason: nil,\n start_time: nil,\n end_time: nil,\n resp: nil,\n fd: nil\n }\n\n # Return the download struct as the GenServer state\n # Also return {:continue, :kickoff} to immediately execute the handle_continue/2\n # callback to kickoff the download\n {:ok, new_download, {:continue, :kickoff}}\n end\n\n @impl GenServer\n def handle_continue(:kickoff, %Download{src: src, dest: dest} = download) do\n # Open up a file to save the download, this returns a file descriptor\n {:ok, fd} = File.open(dest, [:write, :binary])\n\n # Kick off the download\n case HTTPoison.get(src, %{}, stream_to: self(), async: :once) do\n {:ok, resp} ->\n download = %Download{download | resp: resp, fd: fd}\n {:noreply, download}\n\n # In case of errors update the download struct and return\n # {:stop, reason, state} to stop the GenServer process\n {:error, %HTTPoison.Error{reason: reason}} ->\n failed_download = %Download{\n download\n | status: :error,\n error_reason: reason,\n exit_status: :error\n }\n\n {:stop, reason, failed_download}\n end\n end\n\n @impl GenServer\n def terminate(reason, %Download{id: id, from: from} = download) do\n # Inform parent process about download finish or failure.\n # The parent process pid is available in the `:from` property of the download struct\n Process.send(from, {:terminating, id, download}, [])\n\n Logger.info(\n \"Terminate download-worker #{id}: reason=#{inspect(reason)} download=#{inspect(download)}\"\n )\n\n # Close the file descriptor\n download\n |> Map.get(download, :fd)\n |> File.close()\n\n # Gracefully stop the GenServer process\n :normal\n end\n\n @impl GenServer\n def handle_call(:status, _from, download), do: {:reply, download, download}\n\n @impl GenServer\n def handle_info(%HTTPoison.AsyncStatus{code: code}, download) when code >= 400 do\n message = \"Failed with code: #{code}\"\n\n failed_download = %Download{\n download\n | status: :error,\n error_reason: message,\n exit_status: :error\n }\n\n {:stop, message, failed_download}\n end\n\n @impl GenServer\n def handle_info(%HTTPoison.AsyncStatus{}, download) do\n HTTPoison.stream_next(download.resp)\n {:noreply, download}\n end\n\n @impl GenServer\n def handle_info(%HTTPoison.Error{reason: reason}, download) do\n message = inspect(reason)\n\n failed_download = %Download{\n download\n | status: :error,\n error_reason: message,\n exit_status: :error\n }\n\n {:stop, message, failed_download}\n end\n\n @impl GenServer\n def handle_info(%HTTPoison.AsyncHeaders{headers: headers}, download) do\n # Get the \"Content-Length\" header from the list of headers returned in the response\n content_length_header =\n Enum.find(headers, fn\n {\"Content-Length\", _length} -> true\n _ -> false\n end)\n\n # Get the download file size from the \"Content-Length\" header\n size =\n case content_length_header do\n {\"Content-Length\", length} -> length || 0\n nil -> 0\n end\n\n # Ask for the next chunk of download\n HTTPoison.stream_next(download.resp)\n\n # Save the download size and other meta data in the download struct\n download = %Download{download | size: size, status: :active, start_time: DateTime.utc_now()}\n {:noreply, download}\n end\n\n @impl GenServer\n def handle_info(%HTTPoison.AsyncChunk{chunk: chunk}, download) do\n # Append the new chunk of data in the file using the file descriptor\n IO.binwrite(download.fd, chunk)\n HTTPoison.stream_next(download.resp)\n\n # Update the bytes downloaded information in the download struct\n bytes_downloaded = download.bytes_downloaded + byte_size(chunk)\n download = %Download{download | bytes_downloaded: bytes_downloaded}\n {:noreply, download}\n end\n\n @impl GenServer\n def handle_info(%HTTPoison.AsyncEnd{}, download) do\n File.close(download.fd)\n\n finished_download = %Download{\n download\n | status: :finish,\n exit_status: :normal,\n end_time: DateTime.utc_now()\n }\n\n # Since the download is finished we are returning {:stop, .., ..}\n # which will invoke the `terminate/2` callback\n {:stop, :finish, finished_download}\n end\nend\n```\n\n## The DownloadSupervisor\n\nThe `DownloadsSupervisor` module serves as a dynamic supervisor, enabling us to dynamically spawn and manage `DownloadWorker` processes. Additionally, it provides the ability to stop any active download process as needed.\n\nBy utilizing a dynamic supervisor, we gain visibility into the active downloads by examining the child processes maintained under this supervisor. This allows us to easily track and manage the ongoing download operations.\n\n```elixir\ndefmodule DownloadsSupervisor do\n use DynamicSupervisor\n require Logger\n\n # Public functions the interact with the supervisor\n\n @doc \"\"\"\n Start the supervisor process\n \"\"\"\n def start_link(_) do\n DynamicSupervisor.start_link(__MODULE__, :no_args, name: __MODULE__)\n end\n\n @doc \"\"\"\n Start a new download under the supervisor\n \"\"\"\n def add(args) do\n {:ok, pid} = DynamicSupervisor.start_child(__MODULE__, {DownloadWorker, args})\n\n pid\n end\n\n @doc \"\"\"\n Stop an existing download process under the supervisor\n \"\"\"\n def remove(child_pid) do\n DynamicSupervisor.terminate_child(__MODULE__, child_pid)\n end\n\n # Callbacks\n\n @impl true\n def init(:no_args) do\n DynamicSupervisor.init(strategy: :one_for_one)\n end\nend\n```\n\n## The DownloadManager\n\nThe `DownloadManager` module serves as a GenServer module that aggregates and manages the state of all active, completed and failed downloads. It offers a convenient API to interact with the underlying `DownloadsSupervisor` and `DownloadWorker` processes.\n\nThe `DownloadManager` module has public functions to perform common tasks. By exposing these functions, we can interact with the `DownloadManager` without directly making calls to the GenServer module.\n\nThe public API functions provide a clean and concise interface for managing downloads, while the internal logic handles the complexities of interacting with the supervisor and worker processes.\n\nTo provide more context and important details:\n\n* The `DownloadManager` maintains a map of all downloads in its state, and keeps this map updated with the lastest download statuses by querying the download workers periodically.\n\n* In the `init/1` callback we schedule periodic updates to refresh the state of active downloads by sending a `:fetch_all` message to itself.\n\n* The `DownloadManager` allows adding a new download, removing an existing download, retrieving the status of a specific download, listing all downloads, and clearing all downloaded files.\n\n* When a download is added, the `DownloadManager` creates a new `Download` struct, starts a `DownloadWorker` process via the `DownloadsSupervisor`, and stores the download information in its state.\n\n* The `DownloadManager` can receive a `{:terminating, id, last_child_state}` message from the `DownloadWorker` processes. This message serves as a notification to the download manager about finished downloads or download failures, enabling it to update the state accordingly. By handling this message, the DownloadManager can track the final state of a download.\n\n```elixir\ndefmodule DownloadManager do\n @moduledoc \"\"\"\n GenServer which stores aggregates and stores state for all download worker processes\n Exposes APIs to add, delete, list downloads.\n\n Examples:\n\n {:ok, id} = DownloadManager.add(\"https://file-examples-com.github.io/uploads/2017/04/file_example_MP4_1920_18MG.mp4\")\n DownloadManager.list()\n DownloadManager.get(id)\n DownloadManager.remove(id)\n \"\"\"\n use GenServer\n\n require Logger\n\n @update_interval 1000\n @base_download_path \"/tmp/async_elixir_temp_downloads\"\n\n # Public API\n\n @doc \"Start the download manager process\"\n def start_link(_opts), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)\n\n @doc \"Add a new download\"\n def add(src), do: GenServer.call(__MODULE__, {:add, src})\n\n @doc \"Remove an existing download\"\n def remove(id), do: GenServer.call(__MODULE__, {:remove, id})\n\n @doc \"Get the lastest information about a download\"\n def get(id), do: GenServer.call(__MODULE__, {:get, id})\n\n @doc \"Get the most updated list of downloads\"\n def list(), do: GenServer.call(__MODULE__, :list)\n\n @doc \"Clear all downloaded data\"\n def clear_all_downloads(), do: File.rm_rf!(@base_download_path)\n\n # Callbacks\n\n @impl GenServer\n def init(_args) do\n # Send a message to itself the begin aggregating lastest download statuses\n Process.send_after(self(), :fetch_all, @update_interval)\n {:ok, %{}}\n end\n\n # Callback to get the status of a given download using it download id\n @impl GenServer\n def handle_call({:get, id}, _from, state) do\n case Map.get(state, id) do\n nil ->\n {\n :reply,\n {:error, :not_found},\n state\n }\n\n download ->\n {\n :reply,\n {:ok, download},\n state\n }\n end\n end\n\n # Callback to add a new download\n @impl GenServer\n def handle_call({:add, src}, _from, state) do\n id = UUID.uuid1()\n File.mkdir_p!(@base_download_path)\n download_destination = \"#{@base_download_path}/#{id}\"\n\n download = %Download{\n id: id,\n src: src,\n dest: download_destination,\n # This `from` parameter allows the download worker to send a message back to the\n # Download manager when the download finishes or in case of failures\n from: self(),\n name: guess_filename(src)\n }\n\n # Call the Download manager to start a new download worker process and initaite the download\n pid = DownloadsSupervisor.add(download)\n download = %Download{download | pid: pid}\n\n # Save the download in a map that is the GenServer state\n {:reply, {:ok, id}, Map.put(state, id, download)}\n end\n\n @impl GenServer\n def handle_call({:remove, id}, _from, state) do\n case Map.get(state, id) do\n %Download{pid: pid, dest: dest} ->\n # Send a message to the Download supervisor to terminate the Download Worker Process\n # that is downloading the file\n DownloadsSupervisor.remove(pid)\n\n # Delete the file from disk\n res = File.rm(dest)\n Logger.info(\"Remove left over file: #{inspect(res)}\")\n\n # Remove the download from the downloads map that is the GenServer state\n {:reply, {:ok, id}, Map.delete(state, id)}\n\n _ ->\n {:reply, {:error, :not_found}, state}\n end\n end\n\n # Get the list of all downloads\n @impl GenServer\n def handle_call(:list, _from, state) do\n {:reply, Map.values(state), state}\n end\n\n # Used by a Download Worker to inform the Download manager when when the download ends\n @impl GenServer\n def handle_info({:terminating, id, last_child_state}, state) do\n {_old_value, state} =\n Map.get_and_update(state, id, fn\n current_value when is_nil(current_value) -> :pop\n current_value -> {current_value, merge_with_old_state(current_value, last_child_state)}\n end)\n\n Logger.info(\"Recieved last state from child: #{id}, new_state: #{inspect(state)}\")\n {:noreply, state}\n end\n\n # Every 1 second (1000ms) we will query each of the active download worker\n # process to refresh the state of the running downloads\n @impl GenServer\n def handle_info(:fetch_all, state) do\n new_state =\n state\n |> Enum.map(fn\n {id, %Download{status: status} = download} when status in [:finish, :error, :cancel] ->\n {id, download}\n\n {id, %Download{pid: pid} = download} ->\n {id, fetch_status(pid, download)}\n end)\n |> Enum.into(%{})\n\n Process.send_after(self(), :fetch_all, @update_interval)\n {:noreply, new_state}\n end\n\n # Private helpers\n\n # Call the download worker process to fetch the lastest status of the Download\n defp fetch_status(pid, download) do\n if Process.alive?(pid) do\n new_download_state = GenServer.call(pid, :status)\n merge_with_old_state(download, new_download_state)\n else\n %Download{download | status: :error, error_reason: \"Killed\"}\n end\n end\n\n # Helper function to merge an old download struct with a new download struct\n defp merge_with_old_state(old_download, new_download) do\n %Download{\n old_download\n | size: new_download.size,\n status: new_download.status,\n bytes_downloaded: new_download.bytes_downloaded,\n start_time: new_download.start_time,\n end_time: new_download.end_time,\n error_reason: new_download.error_reason\n }\n end\n\n # Helper function that attempts to guess the name of the download from the download URL path\n # In case of failures it assigns a UUID as the download name\n defp guess_filename(url) do\n path =\n url\n |> URI.parse()\n |> Map.fetch!(:path)\n\n if(is_nil(path), do: UUID.uuid1(), else: path |> Path.basename() |> String.trim())\n end\nend\n```\n\nAt this point our download manager is ready.\n\nLets test it out! ๐๐๐\n\n## The Runner module\n\nThe `Runner` module allows us to test the functionality of our download manager and display the status of the downloads in a markdown table format. It uses the Kino library to render and update the table periodically.\n\nThe `render_downloads_list/0` function continuously updates the downloads table using the [Kino.animate/2](https://hexdocs.pm/kino/Kino.html#animate/2) function. It retrieves the latest list of downloads from the `DownloadManager` and constructs the table data by formatting the relevant fields. The markdown table is rendered using [Kino.Markdown.new/1](https://hexdocs.pm/kino/Kino.Markdown.html#new/1).\n\n```elixir\ndefmodule Runner do\n # 1 second (1000ms)\n @refresh_rate 1000\n\n def render_downloads_list do\n # Every 1 second we refresh the downloads list\n Kino.animate(@refresh_rate, fn _ ->\n # Get the lastest downloads list from the DownloadManager\n downloads = DownloadManager.list()\n\n unless downloads == [] do\n data =\n downloads\n |> Enum.map(fn download ->\n data =\n [\n download.id,\n download.name,\n download.status,\n percentage_progresss(download),\n progress(download),\n get_speed(download),\n download.src,\n download.dest,\n download.start_time,\n download.end_time || \"NA\",\n download.error_reason || \"NA\"\n ]\n |> Enum.join(\"|\")\n\n \"|\" <> data <> \"|\"\n end)\n |> Enum.join(\"\\n\")\n\n # The headers for the downloads table\n headers = \"\"\"\n |ID|Name|Status|Percentage Completed|Progress|Speed|Source URL|Destination|Started At|Ended At|Error Reason|\n |--|----|------|--------------------|--------|-----|----------|-----------|----------|--------|------------|\n \"\"\"\n\n # Render the downloads table in markdown format\n Kino.Markdown.new(\"#{headers}#{data}\")\n end\n end)\n end\n\n # Private helper functions\n\n # Calculate the progress percentage of a download from the download size\n defp percentage_progresss(download) do\n if download.status != :initiate && to_int(download.size) != 0 do\n percentage = download.bytes_downloaded / to_int(download.size) * 100\n \"#{trunc(percentage)}%\"\n else\n \"NA\"\n end\n end\n\n # Use the Sizeable library to show the size of the download in a human readable way\n defp progress(download) do\n if download.status != :initiate && to_int(download.size) != 0 do\n \"#{Sizeable.filesize(download.bytes_downloaded)} / #{Sizeable.filesize(download.size)}\"\n else\n \"NA\"\n end\n end\n\n # Calculate the download speed using the download start time and data downloaded\n defp get_speed(download) when download.status == :active,\n do: \"#{get_speed_in_bytes(download) |> Sizeable.filesize()}/sec\"\n\n defp get_speed(_download), do: \"NA\"\n\n defp get_speed_in_bytes(%Download{bytes_downloaded: bytes_downloaded, start_time: start_time})\n when is_nil(start_time) or bytes_downloaded == 0,\n do: 0\n\n defp get_speed_in_bytes(download) do\n elapsed_time = DateTime.diff(DateTime.utc_now(), download.start_time)\n if elapsed_time == 0, do: 0, else: download.bytes_downloaded / elapsed_time\n end\n\n defp to_int(num) when is_binary(num) do\n case Integer.parse(num) do\n {num, _} -> num\n :error -> 0\n end\n end\n\n defp to_int(num), do: num\nend\n```\n\nNow lets start our Download manager and Download supervisor.\n\n```elixir\n# Stop any existing Download manager or Download supervisor processes\nif Process.whereis(DownloadsSupervisor), do: DynamicSupervisor.stop(DownloadsSupervisor)\nif Process.whereis(DownloadManager), do: GenServer.stop(DownloadManager)\n\n# Clear any previously downloaded data\nDownloadManager.clear_all_downloads() |> IO.inspect()\n\n# Start the Download manager and Download supervisor processes\n{:ok, download_sup_pid} = DownloadsSupervisor.start_link(:noop)\n{:ok, download_manager_pid} = DownloadManager.start_link(:noop)\n```\n\n```elixir\n# Call the runner module to render the downloads list as a markdown table\nRunner.render_downloads_list()\n```\n\n```elixir\n# Start 4 downloads of different sizes\n{:ok, first_id} = DownloadManager.add(\"https://speed.hetzner.de/100MB.bin\")\n{:ok, second_id} = DownloadManager.add(\"https://speed.hetzner.de/1GB.bin\")\n{:ok, third_id} = DownloadManager.add(\"https://speed.hetzner.de/10GB.bin\")\n\n# This download will fail since the download url does not exists\n{:ok, fourth_id} = DownloadManager.add(\"https://speed.hetzner.de/bad_file.bin\")\n```\n\nNotice how the downloads in progress are being updated in real-time. It is worth noting that while one of the downloads encountered a failure, the remaining downloads continued unaffected.\n\nThis showcases the concept of process isolation, where failures in one process do not impact others. Furthermore, we receive informative notifications about the download failure, including the reason for the failure, this works because we are trapping exits in the `DownloadWorker` processes to update the final state of a download.\n\nNow lets visualize the supervision tree of the Dynamic supervisor that is the `DownloadsSupervisor` module. As the downloads finish if we refresh the supervision tree we can notice how the worker processes are stopped and removed.\n\n```elixir\nKino.Process.render_sup_tree(download_sup_pid)\n```\n\nRemove the fourth download that failed, notice how the download entry is removed from the downloads table after this code is executed.\n\n```elixir\nDownloadManager.remove(fourth_id)\n```\n\nFinally, we can trace the message flow between different processes when starting a new download. To achieve this, we utilize the `Kino.Process.render_seq_trace/2` function. In this case, we provide the PID of the download manager processes to the function, ensuring that only the messages sent to and from the download manager can be traced.\n\n```elixir\npids_to_trace = [download_manager_pid]\n\n# Trace and inspect messages being sent in between the processes\nKino.Process.render_seq_trace(pids_to_trace, fn ->\n {:ok, _first_id} = DownloadManager.add(\"https://speed.hetzner.de/100MB.bin\")\n # Sleep to enable catching all messages between the processes\n :timer.sleep(1000)\nend)\n```\n\nCongratulations on successfully building a download manager from scratch! ๐๐๐\n\nThroughout this process, we applied various concepts that we have learned in previous chapters, reinforcing our understanding of Elixir's key features.\n\nIn summary, we utilized **GenServer** to manage the state of downloads, **DynamicSupervisor** to dynamically spawn and terminate download worker processes, **trapping exits** to handle failures gracefully, **message passing** between processes to communicate and coordinate activities, and finally, we learned how to **write and organize code that utilizes multiple processes** in Elixir.\n\nWell done! ๐ฅณ\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_7.1_intro_to_tasks.livemd",
"content": "# Introduction to Tasks\n\n## Navigation\n\n\n\n## Introduction\n\nIn the previous chapters, we explored various methods of starting processes, including `spawn/1` and `spawn_link/1`. Now, let's dive into the Task module, which provides a more convenient approach to spawning processes for performing tasks.\n\nThe Task module offers a wide range of convenience functions to effectively manage launched tasks. Unlike plain processes started with `spawn/1`, tasks provide additional capabilities such as monitoring metadata and error logging.\n\nWith the Task module, we gain access to a many functions tailored to common use cases. We can easily await the completion of spawned tasks, launch supervised tasks, and execute multiple tasks concurrently. The abstraction and convenience functions provided by the Task module make working with processes a breeze, eliminating the need to delve into low-level details.\n\n## Basics usage\n\nLet's explore some basic examples of launching tasks:\n\n```elixir\nTask.start(fn -> IO.puts(\"Hello from the first task!\") end)\n\n# Same as Task.start/1, but accepts a module, function, and arguments instead.\nTask.start(IO, :puts, [\"Hello from the second task!\"])\n```\n\nAs you can see, launching a task is very similar to spawning a process using `spawn/1`. In this case, the process spawned by `Task.start/1` is not linked to the caller process. It is primarily used for performing side effects where we don't need to wait for the result or handle failures.\n\n## The Task struct\n\nUnder the hood, a task in Elixir is essentially a regular Elixir process. When we spawn a task using one of the functions provided by the Task module, we receive a `Task` struct in return. This struct contains additional information about the task, and it can be utilized with various functions from both the Task and Task.Supervisor modules (which we will explore in greater detail in the upcoming chapters).\n\nNow, let's take a closer look at the information encapsulated within the Task struct. To do this, we will start a task using the `Task.async/1` function and analyze the resulting struct.\n\n```elixir\nTask.async(fn -> :empty_task end)\n```\n\nWe get back a structure like so\n\n\n\n```elixir\n%Task{\n mfa: {module, function, arrity},\n owner: owner_pid,\n pid: task_process_pid,\n ref: #Reference\n}\n```\n\n* `:mfa` - a three-element tuple containing the module, function name, and arity invoked to start the task in async/1 and async/3\n* `:owner` - the PID of the process that started the task\n* `:pid` - the PID of the task process; nil if there is no process specifically assigned for the task\n* `:ref` - an opaque term used as the task monitor reference\n\nIn the case of the `ref` field in the Task struct, it represents a monitor reference. When a task is spawned, the caller process monitors the task process using this reference. This monitoring enables the caller process to receive a `{:DOWN, , :process, , }` message when the task exits. The monitor reference is particularly useful when awaiting tasks to [receive exit messages in case of crashes](https://github.com/elixir-lang/elixir/blob/7f7a8bca99fa306a41a985df0018ba642e577d4d/lib/elixir/lib/task.ex#L841).\n\n\n\nIn the upcoming chapters, we will dive deeper into the capabilities of tasks. We will explore how to await task completion, supervise tasks, and uncover many more exciting features. Stay tuned!\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_7.2_awaiting_tasks.livemd",
"content": "# Awaiting Tasks\n\n```elixir\nMix.install([])\n```\n\n## Navigation\n\n\n\n## Introduction\n\nTill now we have seen examples were we spawn a process to do something concurrently however sometimes we might need the value returned by the process. In such situations it is important to wait for the process to complete and then fetch the result.\n\nThe Task module provides the `async` and `await` functions to handle this common use case. With `Task.async/1`, a new process is created, **linked, and monitored** by the caller. Once the task action finishes, a message containing the result is sent to the caller. The `Task.await/2` function is then used to read this message and obtain the result.\n\nHere are some key points to note about `async` and `await`:\n\n* Async tasks establish a link between the caller and the spawned process. If either the caller or the task crashes, the other process will crash as well. This intentional linkage ensures that the computation is not carried out if the process meant to receive the result no longer exists.\n\n* When using async tasks, it is important to await a reply as they are always sent. If you don't expect a reply but still want to launch a linked process, consider using `Task.start_link/1` instead.\n\nLets look at some code to understand this better...\n\n```elixir\nmy_task =\n Task.async(fn ->\n # Sleep for 2 seconds\n :timer.sleep(2000)\n IO.puts(\"Done sleeping\")\n DateTime.utc_now()\n end)\n |> IO.inspect()\n\n# Notice how the above task process is linked to the caller process\nProcess.info(self(), :links)\n|> IO.inspect(label: \"Parent process #{inspect(self())} links\")\n\nTask.await(my_task) |> IO.inspect(label: \"Task returned\")\n```\n\nLets do this again, but check the process mailbox to see the message returned by the spawned task process.\n\n```elixir\nmy_task = Task.async(fn -> \"return value\" end)\nIO.inspect(my_task)\n# Wait for process to complete\n:timer.sleep(100)\n:erlang.process_info(self(), :messages) |> IO.inspect(label: \"Messages in mailbox\")\nTask.await(my_task)\n```\n\nHere notice how the spawed task process sends a message back to the caller in the format\n`{, }`. The `Task.await/1` call basically [awaits this message](https://github.com/elixir-lang/elixir/blob/9fd85b06dcb74217108cd0bdf4164b6cd7f9e667/lib/elixir/lib/task.ex#L827) in a recieve block like so...\n\n\n\n```elixir\nreceive do\n # The reply message from the task\n {^ref, reply} ->\n # Stop monitoring the task since th task has sent a reply so must have completed successfully so we no longer monitor the process for crashes\n demonitor(ref)\n reply\n\n # This is the message received from the task monitor, if this happens it means we received the :DOWN message without getting the reply message first, which means the task crashed\n {:DOWN, ^ref, _, proc, reason} ->\n # Exit the linked caller process that is awaiting since the task process crashed\n exit({reason(reason, proc), {__MODULE__, :await, [task, timeout]}})\n\n # more code\nend\n```\n\n\n\nThe other message returned is `{:DOWN, ref, :process, pid, reason}` - since all tasks are also monitored, you will also receive the `:DOWN` message delivered by `Process.monitor/1`. If you receive the :DOWN message without getting the reply message, it means the task crashed.\n\n\n\nAt any point we can ignore a linked task by calling `Task.ignore/1` which means the task will continue running, but it will be unlinked and we can no longer yield, await or shut it down. Also this means if the task fails the owner process will be unaffected. Lets look at and example...\n\n```elixir\ntime_bomb_task =\n Task.async(fn ->\n :timer.sleep(2000)\n raise \"BOOOOM!\"\n end)\n |> IO.inspect()\n\nIO.inspect(Process.info(self(), :links), label: \"Parent process #{inspect(self())} links\")\n\n# Unlink the spawned task\nTask.ignore(time_bomb_task)\n\nIO.inspect(Process.info(self(), :links), label: \"Parent process #{inspect(self())} links\")\n\n:timer.sleep(2100)\n\nIO.puts(\"Parent process survived!\")\n```\n\nLets see another example were we launch 3 tasks using the `Task.async/3` function that takes mfa(module function args) as arguments. Each of tasks generating a random number.\n\nWe then await there results and return the sum of the random numbers\n\n```elixir\nmy_task1 = Task.async(Enum, :random, [0..10])\nmy_task2 = Task.async(Enum, :random, [10..20])\nmy_task3 = Task.async(Enum, :random, [20..30])\n\nTask.await(my_task1) + Task.await(my_task2) + Task.await(my_task3)\n```\n\nIn cases were wee need to await multiple tasks the Task module provides a better apporach using the `Task.await_many/1` that awaits replies from multiple tasks and returns them as a list.\n\nFor example we could rewrite the above example like so\n\n```elixir\nmy_task1 = Task.async(Enum, :random, [0..10])\nmy_task2 = Task.async(Enum, :random, [10..20])\nmy_task3 = Task.async(Enum, :random, [20..30])\n\nresults = Task.await_many([my_task1, my_task2, my_task3])\nIO.inspect(results, label: \"Results from await_many\")\nEnum.sum(results)\n```\n\nSome important points to note about `Task.await_many/1` are...\n\n* If any of the task processes dies, the caller process will exit with the same reason as that task.\n* It returns a list of the results, in the **same order** as the tasks supplied in the tasks input argument.\n* A timeout, in milliseconds or :infinity, can be given with a default value of 5000. If the timeout is exceeded, then the caller process will exit. Any task processes that are linked to the caller process (which is the case when a task is started with async) will also exit. Any task processes that are trapping exits or not linked to the caller process will continue to run.\n\n## Task await timeouts\n\nWhen calling `Task.await/1` by default the await timeout is 5 seconds after which the caller process will exit. If the task process is linked to the caller process which is the case when a task is started with async, then the task process will also exit. If the task process is trapping exits or not linked to the caller process, then it will continue to run.\n\nLets look at an example...\n\n```elixir\ndont_await_me = Task.async(fn -> :timer.sleep(:infinity) end)\nTask.await(dont_await_me)\n```\n\nThe `Task.await/1` function can only be called once for any given task.\n\nIf we want to check if a task has completed or not and not risk the caller process exiting we must use `Task.yield/2`.\n\n## Yielding tasks\n\nSometimes we only wish to check if a Task is completed within a given timeout, if not we want the caller process to continue. Unlike `Task.await/1` were the caller process exits in cases of timeouts with `Task.yield/2` the caller process will continue to run if the Task has not yet completed within the timeout. Therefore `Task.yield/2` can be called multiple times on the same task.\n\nJust like await the yield function will also block the caller process until the task completes or the timeout is reached.\n\nThese are the different scenarios when calling `Task.yield/1`\n\n* When the task process finishes within the yield timeout - Returns `{:ok, result}` were `result` is the value returned by the task.\n\n* When the task process does not reply within the yield timeout - Returns `nil`. This can happen if the timeout expires OR if the message from the task has already been consumed by the caller.\n\n* When the task process has already exited OR if the task is not linked to the calling process - Returns `{:exit, reason}`\n\nNow lets look at some code...\n\n```elixir\nheavy_task =\n Task.async(fn ->\n :timer.sleep(5000)\n :finished_heavy_task\n end)\n\nTask.yield(heavy_task, 1000) |> IO.inspect(label: \"after 1 second\")\nTask.yield(heavy_task, 1000) |> IO.inspect(label: \"after 2 second\")\nTask.yield(heavy_task, 1000) |> IO.inspect(label: \"after 3 second\")\nTask.yield(heavy_task, 1000) |> IO.inspect(label: \"after 4 second\")\n:timer.sleep(1500)\n:erlang.process_info(self(), :messages) |> IO.inspect(label: \"Messages in mailbox\")\nTask.yield(heavy_task, 1000) |> IO.inspect(label: \"After task finished\")\n:erlang.process_info(self(), :messages) |> IO.inspect(label: \"Messages in mailbox\")\nTask.yield(heavy_task, 1000) |> IO.inspect(label: \"After message from task was consumed\")\n```\n\nSimilar to `Task.await_many/2` we also have `Task.yield_many/2`\n\nThis function receives a list of tasks and waits for their replies in the given time interval. It returns a list of two-element tuples, with the task as the first element and the yielded result as the second. The tasks in the returned list will be in the same order as the tasks supplied in the tasks input argument.\n\nSimilarly to yield/2, each task's result will be `{:ok, term}` if the task has successfully reported its result back in the given time interval or `{:exit, reason}` if the task has died\nnil if the task keeps running past the timeout\n\n```elixir\ntasks =\n for i <- 1..10 do\n Task.async(fn ->\n Process.sleep(i * 1000)\n i\n end)\n end\n\ntasks_with_results = Task.yield_many(tasks)\n\nresults =\n Enum.map(tasks_with_results, fn {task, res} ->\n # Shut down the tasks that did not reply or exit\n res || Task.shutdown(task, :brutal_kill)\n end)\n\n# Here we are matching only on {:ok, value} and\n# ignoring {:exit, _} (crashed tasks) and `nil` (no replies)\nfor {:ok, value} <- results do\n IO.inspect(value)\nend\n```\n\nIn the example above, we create tasks that sleep from 1 up to 10 seconds and return the number of seconds they slept for. If you execute the code all at once, you should see 1 up to 4 printed, as those were the tasks that have replied in the default timeout (5 seconds) of `Task.yield_many/1`. All other tasks will have been shut down using the `Task.shutdown/2` call.\n\nAs a convenience, you can achieve a similar behaviour to above by specifying the `:on_timeout` option to be `:kill_task` (or `:ignore`).\n\nFor example to kill all tasks which do not yield within 7 seconds we can write\n`Task.yield_many(tasks_list, timeout: 7000, on_timeout: :kill_task)` (this option is available from elixir 1.15.0+)\n\n## References\n\n* https://hexdocs.pm/elixir/1.12/Task.html#content\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_7.3_task_async_stream.livemd",
"content": "# Task.async_stream/3\n\n```elixir\nMix.install([\n {:httpoison, \"~> 2.1\"},\n {:jason, \"1.4.0\"},\n {:nimble_csv, \"~> 1.2\"}\n])\n```\n\n## Navigation\n\n\n\n## Introduction\n\n[Streams](https://hexdocs.pm/elixir/1.12/Stream.html) are a valuable feature in Elixir that allow for lazy emission of elements. Any [enumerable](https://hexdocs.pm/elixir/1.12/Enumerable.html) that generates elements one by one during enumeration is considered a stream. Streams are particularly useful when dealing with large datasets that could consume excessive memory if loaded all at once. With streams, we can manage data lazily, processing elements as needed. To learn more about Stream module check out the documentation [here](https://hexdocs.pm/elixir/1.15.0/Stream.html).\n\nNow that we have a basic understanding of streams, let's explore the exciting `Task.async_stream/3` function. This function enables **concurrent processing of each element in a enumerable**, unlocking significant potential for parallel execution.\n\nSince `Task.async_stream/3` works on enumerables it can work on both Streams and Enums.\n\nLets look at an example...\n\n```elixir\n# A function to get a chuk norris joke by calling an api\nget_chuknorris_joke = fn ->\n HTTPoison.get!(\"https://api.chucknorris.io/jokes/random\")\n |> Map.get(:body)\n |> Jason.decode!()\n |> Map.get(\"value\")\nend\n```\n\nLets see how much time it takes to make 10 api calls one by one\n\n```elixir\nEnum.map(1..10, fn _ -> get_chuknorris_joke.() end)\n```\n\nNow lets try the same using `Task.async_stream/3`\n\n```elixir\n1..10\n|> Task.async_stream(fn _ -> get_chuknorris_joke.() end)\n|> Enum.to_list()\n```\n\nObserve the significant improvement in the function's execution speed this time. This is because `Task.async_stream/3` launched a separate process to handle each item in the enumerable. In our case, it spawned a separate process for each API call.\n\nIt's important to note that `Task.async_stream/3` returns a stream, which is lazy and won't execute until we consume it. A common way to consume a stream is by using one of the `Enum` functions, such as `Enum.to_list/1` in this case, or by invoking `Stream.run/1`.\n\n`Task.async_stream/3` also provides various options to customize its behavior. One such option is `:max_concurrency`, which allows us to control the number of tasks running simultaneously. By default, it is set to `System.schedulers_online/0`.\n\nAnother consideration is the ordering of results from `Task.async_stream/3`. By default, Elixir buffers the results to emit them in the original order, as the spawned processes may finish in random order. However, setting the `:ordered` option to `false` removes the need for buffering at the expense of removing ordering.\n\nFor a complete list of options, refer to the [documentation](https://hexdocs.pm/elixir/1.12/Task.html#async_stream/3).\n\n## A practical example\n\nNow, let's explore another practical example of using `Task.async_stream/3`. In this example, we'll read a CSV file containing the top 100 websites.\n\nThe csv file has data in the following format...\n\n```csv\n1,\"fonts.googleapis.com\",10\n2,\"facebook.com\",10\n3,\"twitter.com\",10\n4,\"google.com\",10\n5,\"youtube.com\",10\n...\n```\n\nOur goal is to check the reachability of each website by sending an HTTP request to it.\n\n```elixir\n\"#{Path.absname(__DIR__)}/sample_data/top_websites.csv\"\n|> File.stream!()\n|> NimbleCSV.RFC4180.parse_stream()\n# Map out the website information from every row in the csv file\n|> Stream.map(fn [_, website, _] -> website end)\n|> Task.async_stream(&HTTPoison.get/1, timeout: :infinity, ordered: false, max_concurrency: 4)\n# Filter out reachable websites\n|> Stream.filter(fn\n {:ok, _} -> true\n _ -> false\nend)\n|> Enum.count()\n|> IO.inspect(label: \"Reachable website count\")\n```\n\nHere's a breakdown of the code:\n\nFirst, we use `File.stream!/1` to read the CSV file. This function provides a stream that allows us to access the file lazily, avoiding the need to load the entire file into memory.\n\nNext, we parse the file using the [parse_stream/1](https://hexdocs.pm/nimble_csv/NimbleCSV.html#c:parse_stream/2) function from the [nimble_csv](https://github.com/dashbitco/nimble_csv) library. This gives us a parsed stream of the CSV data.\n\nWe then leverage `Task.async_stream/3` to make a GET request to each website concurrently. Since the order of the responses doesn't matter, we specify `ordered: false`. Additionally, we limit the concurrency to 4 requests at a time using the `max_concurrency: 4` option.\n\nFinally, we filter out the reachable websites and then count the elements by consuming the stream using the `Enum.count/1` function.\n\n\n\nBy using `Task.async_stream/3` Elixir enables us to perform concurrent data processing with just a few lines of code. This simplicity and power of concurrent programming in Elixir is truly amazing ๐\n\n## References\n\n* https://hexdocs.pm/elixir/1.12/Task.html#async_stream/3\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_7.4_supervised_tasks.livemd",
"content": "# Supervised Tasks\n\n```elixir\nMix.install([\n {:httpoison, \"~> 2.1\"},\n {:kino, \"~> 0.9.0\"}\n])\n```\n\n## Navigation\n\n\n\n## Supervising tasks\n\nIn the previous chapters, we have learned how to start tasks and await their completion. These tasks are typically linked to the caller process and are not supervised.\n\nSupervisors, as we have previously discovered, offer valuable control and visibility over processes.\n\nBy supervising our tasks, we can enhance our visibility and control over them. We can unlink tasks from the caller process to prevent cascading failures, and still have the ability to await the task's completion, among other benefits.\n\nIn Elixir, we are provided with the [Task.Supervisor](https://hexdocs.pm/elixir/1.15.2/Task.Supervisor.html#content) module, which allows us to **dynamically supervise tasks** easily and oversee the tasks we spawn.\n\nLets look at the some code to understand how this works...\n\n\n\nWe first start a `Task.Supervisor` as a part of our supervision tree, here we name it `AwesomeTaskSupervisor`. We can pass other options in the child specifications like `:max_restarts`, `:max_seconds`, etc. All the options are documented [here](https://hexdocs.pm/elixir/1.15.2/Task.Supervisor.html#start_link/1) in the `Task.Supervisor.start_link/1` docs.\n\n```elixir\nchildren = [{Task.Supervisor, name: AwesomeTaskSupervisor}]\n\nSupervisor.start_link(children, strategy: :one_for_one)\n```\n\nWe can now start a supervised Task under our supervisor like so...\n\n```elixir\ntask =\n Task.Supervisor.async(\n AwesomeTaskSupervisor,\n fn ->\n :timer.sleep(200)\n IO.puts(\"Hello from a supervised task!\")\n :returned_from_task\n end\n )\n |> IO.inspect(label: \"TASK\")\n\nProcess.info(self(), :links)\n|> IO.inspect(label: \"Parent process #{inspect(self())} links\")\n\n# Wait for spawed task to finish\n:timer.sleep(300)\n\n# Notice the the reply message and the \"DOWN\" message from the completed task\n# The \"DOWN\" message is because of the monitor of the task\nIO.inspect(Process.info(self(), :messages))\n```\n\nNote that in the above example the spawed task was **still linked to the caller process** and was monitered by the caller process. This means that if the task process crashed it would bring down the caller process as well.\n\n\n\nIn the above example we use a single task supervisor process to launch multiple tasks, this can quickly become a bolttleneck in scenarios were lots of tasks are being spawed and the supervisor process is not able to keep up.\n\nIn such cases we can use the partition supervisor that will by default start a dynamic supervisor for each core in our machine. So it will start multiple instances of the `Task.Supervisor` and then pick a random instance to start the task on.\n\n\n\nFor this we would need to define the child spec of the supervisor like so\n\n```elixir\n# In the `child_spec` option we pass the name of the supervisor that we want to partition\nchildren = [{PartitionSupervisor, child_spec: Task.Supervisor, name: ScalableTaskSupervisor}]\n\n{:ok, supervisor_pid} = Supervisor.start_link(children, strategy: :one_for_one)\n```\n\nWe can then start tasks via the partitioned task supervisor using the ` using the {:via, PartitionSupervisor, {name, key}}` format, `name` is the name of the partition supervisor and `key` is the routing key.\n\n```elixir\nTask.Supervisor.async(\n {:via, PartitionSupervisor, {ScalableTaskSupervisor, :my_routing_key}},\n fn ->\n :timer.sleep(200)\n IO.puts(\"Hello from a supervised task!\")\n end\n)\n\nKino.Process.render_sup_tree(supervisor_pid)\n```\n\nYou'll observe the initiation of multiple task supervisors, each associated with a specific partition. These supervisors operate under our \"ScalableTaskSupervisor\" partition supervisor, and our task was launched within one of these partitions.\n\n\n\nLet's now explore the behavior when a supervised task encounters an unexpected crash.\n\n```elixir\n# Now lets spawn a supervised task process that will crash\nTask.Supervisor.start_child(\n AwesomeTaskSupervisor,\n fn ->\n IO.puts(\"A task was started!\")\n raise \"boom\"\n end,\n # restart process if the exit is abnormal, by default it is :temporary\n restart: :transient\n)\n|> IO.inspect(label: \"Returned from call to start_child\")\n```\n\nHowever, it's important to note that when using the `Task.Supervisor.start_child/2` function, we **do not receive a task struct** that can be directly awaited or used with other functions in the `Task` module. Instead, we receive the PID of the task process.\n\nWhen using `Task.Supervisor.start_child/2` the default supervisor restart strategy is `:temporary` so if a task process crashes, it will not be automatically restarted by the supervisor. In the example above we set the startegy to `:transient` so that the supervisor restarts the crashed task.\n\nIt's worth mentioning that when using other functions in the `Task.Supervisor` module, such as `async/3`, `async_nolink/3`, etc, the spawned task processes also have a `:temporary` restart strategy, which **cannot be changed**. This means that **if a task process crashes, it will not be automatically restarted by the supervisor**.\n\n## Unlinked supervised tasks\n\nTo prevent linking a task with the caller process, we can utilize functions like `async_nolink/3`, `async_stream_nolink/4`, and others provided by the `Task.Supervisor` module.\n\nBy using these functions, the spawned tasks are not linked to the caller process. However, they are still monitored by the caller process. In the event of a task crash, the caller process remains unaffected and can still be informed about the crashed task through monitoring. Additionally, **the unlinked caller process retains the ability to await the completion of the task**.\n\nThis means if we want to avoid the caller process from exiting when a spawned task process exits abnormally while also retain the ability of the caller process to await the task we can use the functions like `async_nolink` in the `Task.Supervisor` module.\n\nTo gain a better understanding of this concept, let's explore some examples using our `AwesomeTaskSupervisor` that we previously started.\n\n```elixir\nunlinked_task =\n Task.Supervisor.async_nolink(AwesomeTaskSupervisor, fn ->\n :timer.sleep(300)\n 1 + 2\n end)\n |> IO.inspect(label: \"Unliked Task\")\n\nsupervisor_pid = Process.whereis(AwesomeTaskSupervisor)\n\n# Spawned task process is not linked to the caller\nProcess.info(self(), :links)\n|> IO.inspect(label: \"Parent process #{inspect(self())} links\")\n\n# Spawned task process is linked to the supervisor\nProcess.info(supervisor_pid, :links)\n|> IO.inspect(label: \"AwesomeTaskSupervisor process #{inspect(self())} links\")\n\n# Spawned task process appears as a child under the supervisor\nTask.Supervisor.children(AwesomeTaskSupervisor)\n|> IO.inspect(label: \"AwesomeTaskSupervisor Children\")\n\n# We can still await the unlinked task\nTask.await(unlinked_task)\n|> IO.inspect(label: \"Result from task\")\n```\n\nNote this function requires the task supervisor to have `:temporary` as the `:restart` option (the default that cannot be changed), as `async_nolink/3` keeps a direct reference to the task which is lost if the task is restarted.\n\n\n\n### Task.Supervisor.async_nolink/3 vs Task.Supervisor.start_child/3\n\n* Use `async_nolink/3` when you need to await or yield the task's result. The spawned task process is not linked to the caller process, but it is monitored. This allows the caller process to receive a message when the task exits and still await its completion.\n\n* Use `start_child/3` when you want a fire-and-forget approach. The spawned task process is not linked to the caller process, and there is no monitoring or message communication. This is suitable for scenarios where you don't need the task's result or if it performs side-effects (like I/O) without the need for result handling.\n\nThe choice between `start_child` and `async_nolink` conveys the semantic intention. `start_child` indicates that you don't care about the result, while `async_nolink` implies that you may have an interest in the task's life and result, as the monitor will provide information about its termination.\n\n## Usage with OTP behaviours\n\nWhen using `async_nolink` to create a task within an OTP behavior like `GenServer`, it's important to match on the message received from the task in your `GenServer.handle_info/2` callback.\n\nThe reply sent by the task will be in the format `{ref, result}`, where `ref` is the monitor reference held by the task struct and `result` is the return value of the task function.\n\nRegardless of how the task created with `async_nolink` terminates, the caller's process will always receive a `:DOWN` message with the same `ref` value held by the task struct. If the task terminates normally, the reason in the `:DOWN` message will be `:normal`.\n\nTypically, `async_nolink/3` is used when there is a possibility of the task failing, and you want to prevent it from causing the caller process to crash.\n\n\n\nLet's consider an example where we create a GenServer called `MyDownloader` that allows us to spawn tasks for downloading files. The GenServer process doesn't get blocked during the download; instead, it delegates the downloading task to a separate task process. The GenServer saves the spawned task reference in a `MapSet` in its state and logs when a downloading task succeeds or crashes.\n\n```elixir\ndefmodule MyDownloader do\n use GenServer\n\n # == Public API ==\n\n def start_link() do\n GenServer.start_link(__MODULE__, :noop, name: __MODULE__)\n end\n\n def start_download(url) do\n GenServer.call(__MODULE__, {:start_download, url})\n :ok\n end\n\n def download(url) do\n # We are using get! so that in case of failures the task will crash\n HTTPoison.get!(url)\n end\n\n # == GenServer callbacks ==\n\n @impl true\n def init(_) do\n {:ok, MapSet.new()}\n end\n\n @impl true\n def handle_call({:start_download, url}, _from, state) do\n task = Task.Supervisor.async_nolink(AwesomeTaskSupervisor, __MODULE__, :download, [url])\n\n # Save the spawned task reference in a mapset in the GenServer state\n state = MapSet.put(state, task.ref)\n {:reply, :ok, state}\n end\n\n # The task completed successfully\n @impl true\n def handle_info({ref, response}, state) do\n # We don't care about the DOWN message now, so let's demonitor and flush it\n # Flushing will remove the {_, MonitorRef, _, _, _} message, if there is one, \n # from the caller message queue after monitoring has been stopped.\n Process.demonitor(ref, [:flush])\n\n IO.inspect(response.body, label: \"Task #{inspect(ref)} completed with result\")\n\n # Remove finished tasks from the mapset\n state = MapSet.delete(state, ref)\n {:noreply, state}\n end\n\n # The task failed, that is we received a :DOWN message before the task reply message\n @impl true\n def handle_info({:DOWN, ref, :process, _pid, reason}, state) do\n IO.inspect(elem(reason, 0), label: \"Task #{inspect(ref)} failed with reason\")\n\n # Remove failed tasks from the mapset\n state = MapSet.delete(state, ref)\n {:noreply, state}\n end\nend\n```\n\nNow lets take it for a spin...\n\n```elixir\nMyDownloader.start_link()\n```\n\n```elixir\nMyDownloader.start_download(\"https://api.chucknorris.io/jokes/random\")\n\nMyDownloader.start_download(\n \"https://file-examples.com/storage/fede3f30f864a1f979d2bf0/2017/10/file_example_JPG_100kB.jpg\"\n)\n\n# Invalid download url, the spawned task to request this url will crash\nMyDownloader.start_download(\"https://invalid-file-download.invalid\")\n```\n\nIn this example, all three download tasks run concurrently. Two of the download task processes succeed and send a reply message to the GenServer process, which was captured by the `handle_info({ref, response}, state)` callback. One of the task processes crashes, but the GenServer continues running since the tasks were unlinked.\n\nDue to the task monitor, the GenServer was able to capture the `:DOWN` message using the `handle_info({:DOWN, ref, :process, _pid, reason}, state)` callback and was informed about the crashed task.\n\n\n\nThat concludes our exploration of tasks in Elixir. By now, you should have gained a solid understanding of the power and convenience that tasks offer. The `Task` module is widely used in practice and provides a straightforward and efficient way to manage concurrent work in Elixir. It is yet another tool in the Elixir ecosystem that empowers us to embrace concurrent and parallel programming with ease, unlocking the full potential of the language.\n\nHappy tasking! โจ๐\n\n## References\n\n* https://hexdocs.pm/elixir/1.15.2/Task.Supervisor.html\n* https://groups.google.com/g/elixir-lang-talk/c/gSK36qc7EpE?pli=1\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_8.0_agents.livemd",
"content": "# Agents\n\n## Navigation\n\n\n\n## Introduction\n\nWhen it comes to storing state in Elixir, we have several options at our disposal. We can utilize the process dictionary for local access, employ GenServers, utilize [ETS tables](https://elixirschool.com/en/lessons/storage/ets), and more. However, one straightforward and convenient approach is to use Agents. Agents provide a simple abstraction for storing state in a process and offer a straightforward API for accessing and updating the stored state. Internally, Agents are implemented as GenServer processes.\n\n## Usage\n\nUsing an agent is incredibly straightforward. Let's take a look at a simple example:\n\n```elixir\n{:ok, my_counter_pid} = Agent.start_link(fn -> 0 end)\n\n# Retrieve the stored state\nAgent.get(my_counter_pid, fn state -> state end)\n|> IO.inspect(label: \"GET\")\n\n# Perform a GenServer.cast/2 operation on the agent state\n# The caller process will not wait for the operation to complete\nAgent.cast(my_counter_pid, fn state -> state + 1 end)\n|> IO.inspect(label: \"CAST\")\n\n# Update the Agent state\n# Implemented as a GenServer.call/2 the caller process will wait until the agent process replies\nAgent.update(my_counter_pid, fn state -> state + 1 end)\n|> IO.inspect(label: \"UPDATE\")\n\n# Get and update the state in a single call\n# Implemented as a GenServer.call/2\nAgent.get_and_update(my_counter_pid, fn state -> {state, state + 1} end)\n|> IO.inspect(label: \"get_and_update\")\n\n# Retrieve the stored state\nAgent.get(my_counter_pid, fn state -> state end)\n|> IO.inspect(label: \"GET\")\n\n# Stop the agent process\nAgent.stop(my_counter_pid)\n```\n\nA agent can also be implemented as a module.\n\n```elixir\ndefmodule FibonacciStore do\n use Agent\n\n @doc \"Starts the FibonacciStore agent\"\n def start_link(_) do\n Agent.start_link(fn -> [0, 1] end, name: __MODULE__)\n end\n\n @doc \"Gets the next number in the Fibonacci series\"\n def next_fibonacci() do\n Agent.get_and_update(__MODULE__, fn [num1, num2] ->\n next_fibonacci = num1 + num2\n {next_fibonacci, [num2, next_fibonacci]}\n end)\n end\n\n @doc \"Gets the last generated number in the Fibonacci series\"\n def current_fibonacci() do\n Agent.get(__MODULE__, fn [_num1, num2] -> num2 end)\n end\nend\n```\n\nNow lets take our agent server for a spin.\n\n```elixir\n# Stop the agent process if already running\nif Process.whereis(FibonacciStore), do: Agent.stop(FibonacciStore)\n\n# Start the agent process\nFibonacciStore.start_link(:noop)\n\n# Print the current Fibonacci number in the Agent state\nFibonacciStore.current_fibonacci() |> IO.inspect(label: \"Current Fibonacci\")\n\n# Print the next 10 Fibonacci numbers\nEnum.each(1..10, fn _ -> IO.puts(FibonacciStore.next_fibonacci()) end)\n\n# Print the current Fibonacci number in the Agent state\nFibonacciStore.current_fibonacci() |> IO.inspect(label: \"Current Fibonacci\")\n\n# Stop the agent process\nFibonacciStore\n|> Process.whereis()\n|> Agent.stop()\n```\n\n## Supervising Agents\n\nTypically, agents are included in a supervision tree, much like GenServers. When we use `use Agent` in our module, it automatically creates a `child_spec/1` function, which enables us to start the agent directly under a supervisor.\n\nThe process of adding an agent to a supervision tree closely resembles that of a GenServer. To illustrate, let's consider starting our FibonacciStore agent under a supervisor:\n\n```elixir\n# Same as {FibonacciStore, []}\nchildren = [FibonacciStore]\nSupervisor.start_link(children, strategy: :one_for_all)\n\n# Generate 5 numbers in the Fibonacci series\nEnum.each(1..5, fn _ -> FibonacciStore.next_fibonacci() end)\n\nFibonacciStore.current_fibonacci() |> IO.inspect(label: \"State before restart\")\n\n# Simulate termination of the agent server to check if the supervisor restarts it\nFibonacciStore\n|> Process.whereis()\n|> Process.exit(:boom)\n\n# Wait for the supervisor to restart the agent process\n:timer.sleep(200)\n\n# Notice that the agent process was restarted and its state is now set to its initial state\nFibonacciStore.current_fibonacci() |> IO.inspect(label: \"State after restart\")\n```\n\nIn addition, similar to GenServers, the `use Agent` macro also accepts a list of options to customize the child specification and determine its behavior under a supervisor. By providing these options, we can tailor how the agent operates within the supervision tree. The following options can be passed:\n\n* `:id` - Specifies the identifier for the child specification. By default, it is set to the current module's name.\n* `:restart` - Determines the restart strategy for the child. The default value is `:permanent`, which restarts the child process regardless of whether it crashes or is gracefully terminated.\n\nHere's an example of using the `use Agent` macro with customized options:\n\n\n\n```elixir\nuse Agent, restart: :transient, shutdown: 10_000\n```\n\nIn the above code, the agent child specification is configured to have a restart strategy of `:transient`, meaning that the child process will only be restarted if it terminates abnormally. Additionally, the shutdown strategy is set to allow a grace period of 10,000 milliseconds for the child to shut down before forcefully terminating it.\n\n## Navigation\n\n\n"
},
{
"path": "chapters/ch_9.0_gotchas.livemd",
"content": "# Gotchas\n\n## Navigation\n\n\n\n## Loss of sharing\n\n[Ref](https://medium.com/@johnjocoo/debugging-memory-issues-in-elixir-601c8a0a607d#2e85)\n\nIn Elixir, each process manages its own memory, meaning no data is shared between processes. All data sent between processes is fully copied, which includes data written to or read from an ETS table. During this copying, data is flattened, losing any internal sharing of terms.\n\nWithin a single process, however, [data can be shared](ch_1.2_immutability_and_memory_management.livemd#persistent-datastructures). For instance, if you have a list assigned to a variable, then prepend an element and assign it to another variable, the tail of the original list is shared between both variables, maintaining the same memory allocation.\n\nConsider preloading associations in Ecto, like Posts and Comments, where a post has many comments. If you fetch 1000 comments and preload their 100 associated posts, Ecto shares these posts among the comments. However, when full copying occurs, each post is duplicated for each comment, resulting in 1000 separate post entries. This process, known as flattening or \"loss of sharing,\" leads to significant memory duplication.\n\n## Navigation\n\n\n"
},
{
"path": "chapters/sample_data/top_websites.csv",
"content": "1,\"fonts.googleapis.com\",10\n2,\"facebook.com\",10\n3,\"twitter.com\",10\n4,\"google.com\",10\n5,\"youtube.com\",10\n6,\"s.w.org\",10\n7,\"instagram.com\",10\n8,\"googletagmanager.com\",10\n9,\"linkedin.com\",10\n10,\"ajax.googleapis.com\",10\n11,\"plus.google.com\",10\n12,\"gmpg.org\",10\n13,\"pinterest.com\",9.63\n14,\"fonts.gstatic.com\",9.6\n15,\"wordpress.org\",9.54\n16,\"en.wikipedia.org\",9.54\n17,\"youtu.be\",9.47\n18,\"maps.google.com\",9.3\n19,\"itunes.apple.com\",9.21\n20,\"github.com\",9.18\n21,\"bit.ly\",9.11\n22,\"play.google.com\",9.07\n23,\"goo.gl\",9.03\n24,\"docs.google.com\",9.02\n25,\"cdnjs.cloudflare.com\",8.99\n26,\"vimeo.com\",8.98\n27,\"support.google.com\",8.87\n28,\"google-analytics.com\",8.8\n29,\"maps.googleapis.com\",8.79\n30,\"flickr.com\",8.76\n31,\"vk.com\",8.74\n32,\"t.co\",8.72\n33,\"reddit.com\",8.69\n34,\"amazon.com\",8.66\n35,\"medium.com\",8.64\n36,\"sites.google.com\",8.57\n37,\"drive.google.com\",8.51\n38,\"creativecommons.org\",8.47\n39,\"microsoft.com\",8.47\n40,\"developers.google.com\",8.46\n41,\"adobe.com\",8.44\n42,\"soundcloud.com\",8.41\n43,\"theguardian.com\",8.38\n44,\"apis.google.com\",8.35\n45,\"ec.europa.eu\",8.33\n46,\"lh3.googleusercontent.com\",8.3\n47,\"chrome.google.com\",8.28\n48,\"cloudflare.com\",8.27\n49,\"nytimes.com\",8.26\n50,\"maxcdn.bootstrapcdn.com\",8.25\n51,\"support.microsoft.com\",8.25\n52,\"blogger.com\",8.25\n53,\"forbes.com\",8.24\n54,\"s3.amazonaws.com\",8.23\n55,\"code.jquery.com\",8.23\n56,\"dropbox.com\",8.19\n57,\"translate.google.com\",8.15\n58,\"paypal.com\",8.14\n59,\"apps.apple.com\",8.14\n60,\"tinyurl.com\",8.12\n61,\"etsy.com\",8.1\n62,\"theatlantic.com\",8.09\n63,\"m.facebook.com\",8.08\n64,\"archive.org\",8.05\n65,\"amzn.to\",8.04\n66,\"cnn.com\",8.04\n67,\"policies.google.com\",8.02\n68,\"commons.wikimedia.org\",8.02\n69,\"issuu.com\",8.01\n70,\"i.imgur.com\",8\n71,\"wordpress.com\",8\n72,\"wp.me\",7.99\n73,\"businessinsider.com\",7.98\n74,\"yelp.com\",7.98\n75,\"mail.google.com\",7.98\n76,\"support.apple.com\",7.97\n77,\"t.me\",7.94\n78,\"apple.com\",7.92\n79,\"washingtonpost.com\",7.92\n80,\"bbc.com\",7.92\n81,\"gstatic.com\",7.92\n82,\"imgur.com\",7.91\n83,\"amazon.de\",7.91\n84,\"bbc.co.uk\",7.9\n85,\"googleads.g.doubleclick.net\",7.9\n86,\"mozilla.org\",7.89\n87,\"eventbrite.com\",7.89\n88,\"slideshare.net\",7.88\n89,\"w3.org\",7.87\n90,\"forms.gle\",7.86\n91,\"platform.twitter.com\",7.85\n92,\"accounts.google.com\",7.84\n93,\"telegraph.co.uk\",7.82\n94,\"messenger.com\",7.82\n95,\"web.archive.org\",7.81\n96,\"secure.gravatar.com\",7.81\n97,\"usatoday.com\",7.79\n98,\"huffingtonpost.com\",7.78\n99,\"stackoverflow.com\",7.78\n100,\"fb.com\",7.78\n"
}
]