\n\nIntroduction\n=============\n\nThis project was started to bring together useful Python code snippets that make\ncoding faster, easier, and more enjoyable. You can explore all the cheat sheets at\n`Pysheeet `_. Contributions are always welcome—feel\nfree to fork the repo and submit a pull request to help it grow!\n\nPlugin\n======\n\n**pysheeet** is available as a Claude Code plugin. Once installed, Claude\nautomatically uses the cheat sheets to answer Python questions — just ask\nnaturally and the skill triggers based on context.\n\nInstallation\n------------\n\n**As a Claude Code plugin (recommended):**\n\n.. code-block:: bash\n\n # Step 1: Add the marketplace\n claude plugin marketplace add crazyguitar/pysheeet\n\n # Step 2: Install the plugin\n claude plugin install pysheeet@pysheeet\n\n**Local testing (single session only):**\n\n.. code-block:: bash\n\n claude --plugin-dir /path/to/pysheeet\n\n**Manual installation (requires cloning the repo):**\n\n.. code-block:: bash\n\n git clone https://github.com/crazyguitar/pysheeet.git\n mkdir -p ~/.claude/skills\n cp -r pysheeet/skills/py ~/.claude/skills/py\n\nWhat's New In Python 3\n======================\n\nThis part only provides a quick glance at some important features in Python 3.\nIf you're interested in all of the most important features, please read the\nofficial document, `What’s New in Python `_.\n\n- `New in Python3 `_\n\n\nCheat Sheet\n===========\n\nCore Python fundamentals including data types, functions, classes, and commonly\nused patterns for everyday programming tasks.\n\n- `From Scratch `_\n- `Future `_\n- `Typing `_\n- `Class `_\n- `Function `_\n- `Unicode `_\n- `List `_\n- `Set `_\n- `Dictionary `_\n- `Heap `_\n- `Generator `_\n- `Regular expression `_\n\n\nSystem\n======\n\nDate/time handling, file I/O, and operating system interfaces.\n\n- `Datetime `_ - Timestamps, formatting, parsing, timezones, timedelta\n- `Files and I/O `_ - Reading, writing, pathlib, shutil, tempfile\n- `Operating System `_ - Processes, environment, system calls\n\n\nConcurrency\n===========\n\nThreading, multiprocessing, and concurrent.futures for parallel execution.\nCovers synchronization primitives, process pools, and bypassing the GIL.\n\n- `Threading `_ - Threads, locks, semaphores, events, conditions\n- `Multiprocessing `_ - Processes, pools, shared memory, IPC\n- `concurrent.futures `_ - Executors, futures, callbacks\n\n\nAsyncio\n=======\n\nAsynchronous programming with Python's ``asyncio`` module. Covers coroutines,\nevent loops, tasks, networking, and advanced patterns.\n\n- `A Hitchhiker's Guide to Asynchronous Programming `_ - Design philosophy and evolution\n- `Asyncio Basics `_ - Coroutines, tasks, gather, timeouts\n- `Asyncio Networking `_ - TCP/UDP servers, HTTP, SSL/TLS\n- `Asyncio Advanced `_ - Synchronization, queues, subprocesses\n\n\nC/C++ Extensions\n================\n\nNative extensions for performance-critical code. Covers modern pybind11 (used by\nPyTorch, TensorFlow), ctypes, cffi, Cython, and the traditional Python C API.\nAlso includes a guide for Python developers learning modern C++ syntax.\n\n- `ctypes `_ - Load shared libraries without compilation\n- `Python C API `_ - Traditional C extension reference\n- `Modern C/C++ Extensions `_ - pybind11, Cython\n- `Learn C++ from Python `_ - Modern C++ for Python developers\n\n\nSecurity\n========\n\nModern cryptographic practices and common security vulnerabilities. Covers\nencryption, TLS/SSL, and why legacy patterns are dangerous.\n\n- `Modern Cryptography `_ - AES-GCM, RSA-OAEP, Ed25519, Argon2\n- `TLS/SSL and Certificates `_ - HTTPS servers, certificate generation\n- `Common Vulnerabilities `_ - Padding oracle, injection, timing attacks\n\n\nNetwork\n=======\n\nLow-level network programming with Python sockets. Covers TCP/UDP communication,\nserver implementations, asynchronous I/O, SSL/TLS encryption, and packet analysis.\n\n- `Socket Basics `_\n- `Socket Servers `_\n- `Async Socket I/O `_\n- `SSL/TLS Sockets `_\n- `Packet Sniffing `_\n- `SSH and Tunnels `_\n\n\nDatabase\n========\n\nDatabase access with SQLAlchemy, Python's most popular ORM. Covers connection\nmanagement, raw SQL, object-relational mapping, and common query patterns.\n\n- `SQLAlchemy Basics `_\n- `SQLAlchemy ORM `_\n- `SQLAlchemy Query Recipes `_\n\n\nLLM\n===\n\nLarge Language Models (LLM) training, inference, and optimization. Covers PyTorch\nfor model development, distributed training across GPUs, and vLLM/SGLang for\nhigh-performance LLM inference and serving.\n\n- `PyTorch `_ - Tensors, autograd, neural networks, training loops\n- `Megatron `_ - NVIDIA Megatron training/fine-tuning framework with enroot/pyxis\n- `LLM Serving `_ - vLLM and SGLang for production inference with TP/PP/DP/EP\n- `LLM Benchmark `_ - Benchmark suite for measuring serving performance\n\n\nHPC\n===\n\nHigh-Performance Computing tools for cluster management and job scheduling.\nCovers Slurm workload manager for distributed computing and GPU clusters.\n\n- `Slurm `_\n\n\nBlog\n====\n\nSupplementary topics covering Python internals, debugging techniques, and\nlanguage features that don't fit elsewhere.\n\n- `Is Disaggregated Prefill/Decode a Silver Bullet for LLM Serving? `_\n- `Monitoring EFA with NCCL GIN and Nsys `_\n- `GPU-Initiated Networking for NCCL on AWS `_\n- `PEP 572 and the walrus operator `_\n- `Python Interpreter in GNU Debugger `_\n\nPDF Version\n============\n\n`pdf`_\n\n.. _pdf: https://media.readthedocs.org/pdf/pysheeet/latest/pysheeet.pdf\n\nHow to run the server\n=======================\n\n.. code-block:: bash\n\n $ virtualenv venv\n $ . venv/bin/activate\n $ pip install -r requirements.txt\n $ make\n $ python app.py\n\n # URL: localhost:5000\n"
},
{
"path": "app.py",
"content": "# -*- coding: utf-8 -*-\n\"\"\"This is a simple cheatsheet webapp.\"\"\"\n\nimport os\nfrom flask import Flask, abort, send_from_directory, render_template\nfrom flask_sslify import SSLify\nfrom flask_seasurf import SeaSurf\nfrom flask_talisman import Talisman\nfrom werkzeug.exceptions import NotFound\nfrom werkzeug.utils import safe_join\n\nDIR = os.path.dirname(os.path.realpath(__file__))\nROOT = os.path.join(DIR, \"docs\", \"_build\", \"html\")\n\n\ndef find_key(token):\n \"\"\"Find the key from the environment variable.\"\"\"\n if token == os.environ.get(\"ACME_TOKEN\"):\n return os.environ.get(\"ACME_KEY\")\n for k, v in os.environ.items():\n if v == token and k.startswith(\"ACME_TOKEN_\"):\n n = k.replace(\"ACME_TOKEN_\", \"\")\n return os.environ.get(\"ACME_KEY_{}\".format(n))\n\n\ncsp = {\n \"default-src\": \"'none'\",\n \"style-src\": [\"'self'\", \"'unsafe-inline'\"],\n \"script-src\": [\n \"'self'\",\n \"*.cloudflare.com\",\n \"*.cloudflareinsights.com\",\n \"*.googletagmanager.com\",\n \"*.google-analytics.com\",\n \"*.carbonads.com\",\n \"*.carbonads.net\",\n \"cdn.carbonads.com\",\n \"srv.carbonads.net\",\n \"'unsafe-inline'\",\n \"'unsafe-eval'\",\n ],\n \"connect-src\": [\n \"'self'\",\n \"*.google-analytics.com\",\n \"*.analytics.google.com\",\n \"analytics.google.com\",\n \"*.googletagmanager.com\",\n \"*.carbonads.com\",\n \"*.carbonads.net\",\n \"*.doubleclick.net\",\n ],\n \"font-src\": \"'self'\",\n \"form-action\": \"'self'\",\n \"base-uri\": \"'self'\",\n \"img-src\": \"*\",\n \"frame-src\": [\"ghbtns.com\", \"*.carbonads.com\", \"*.carbonads.net\"],\n \"frame-ancestors\": \"'none'\",\n \"object-src\": \"'none'\",\n}\n\nfeature_policy = {\"geolocation\": \"'none'\"}\n\napp = Flask(__name__, template_folder=ROOT)\napp.config[\"SECRET_KEY\"] = os.urandom(16)\napp.config[\"SESSION_COOKIE_NAME\"] = \"__Secure-session\"\napp.config[\"SESSION_COOKIE_SAMESITE\"] = \"Strict\"\napp.config[\"CSRF_COOKIE_NAME\"] = \"__Secure-csrf-token\"\napp.config[\"CSRF_COOKIE_HTTPONLY\"] = True\napp.config[\"CSRF_COOKIE_SECURE\"] = True\ncsrf = SeaSurf(app)\ntalisman = Talisman(\n app,\n force_https=False,\n content_security_policy=csp,\n feature_policy=feature_policy,\n)\n\nif \"DYNO\" in os.environ:\n sslify = SSLify(app, permanent=True, skips=[\".well-known\"])\n\n\n@app.errorhandler(404)\ndef page_not_found(e):\n \"\"\"Redirect to 404.html.\"\"\"\n return render_template(\"404.html\"), 404\n\n\n@app.route(\"/\")\ndef static_proxy(path):\n \"\"\"Find static files safely.\"\"\"\n try:\n return send_from_directory(ROOT, path)\n except NotFound:\n # Handle file not found or directory errors\n return render_template(\"404.html\"), 404\n\n\n@app.route(\"/\")\ndef index_redirection():\n \"\"\"Redirecting index file.\"\"\"\n return send_from_directory(ROOT, \"index.html\")\n\n\n@csrf.exempt\n@app.route(\"/.well-known/acme-challenge/\")\ndef acme(token):\n \"\"\"Find the acme-key from environment variable.\"\"\"\n key = find_key(token)\n if key is None:\n abort(404)\n return key\n\n\nif __name__ == \"__main__\":\n # Only run the app in debug mode during development\n app.run(debug=os.environ.get(\"FLASK_ENV\") == \"development\")\n"
},
{
"path": "app_test.py",
"content": "\"\"\"Test app.py.\"\"\"\n\nimport multiprocessing\nimport platform\nimport unittest\nimport requests\nimport os\n\nfrom pathlib import Path\nfrom werkzeug.exceptions import NotFound\nfrom flask_testing import LiveServerTestCase\n\nfrom app import acme, find_key, static_proxy, index_redirection, page_not_found\n\nfrom app import ROOT\nfrom app import app\n\n\nif platform.system() == \"Darwin\":\n multiprocessing.set_start_method(\"fork\")\n\n\nclass PysheeetTest(LiveServerTestCase):\n \"\"\"Test app.\"\"\"\n\n def create_app(self):\n \"\"\"Create a app for test.\"\"\"\n # remove env ACME_TOKEN*\n for k, v in os.environ.items():\n if not k.startswith(\"ACME_TOKEN\"):\n continue\n del os.environ[k]\n\n self.token = \"token\"\n self.key = \"key\"\n os.environ[\"ACME_TOKEN\"] = self.token\n os.environ[\"ACME_KEY\"] = self.key\n os.environ[\"FLASK_ENV\"] = \"development\"\n os.environ[\"FLASK_DEBUG\"] = \"1\"\n app.config[\"TESTING\"] = True\n app.config[\"LIVESERVER_PORT\"] = 0\n return app\n\n def check_security_headers(self, resp):\n \"\"\"Check security headers.\"\"\"\n headers = resp.headers\n self.assertTrue(\"Content-Security-Policy\" in headers)\n self.assertTrue(\"X-Content-Type-Options\" in headers)\n self.assertTrue(\"Content-Security-Policy\" in headers)\n self.assertTrue(\"Feature-Policy\" in headers)\n self.assertEqual(headers[\"Feature-Policy\"], \"geolocation 'none'\")\n self.assertEqual(headers[\"X-Frame-Options\"], \"SAMEORIGIN\")\n\n def check_csrf_cookies(self, resp):\n \"\"\"Check cookies for csrf.\"\"\"\n cookies = resp.cookies\n self.assertTrue(cookies.get(\"__Secure-session\"))\n self.assertTrue(cookies.get(\"__Secure-csrf-token\"))\n\n def test_index_redirection_req(self):\n \"\"\"Test that send a request for the index page.\"\"\"\n url = self.get_server_url()\n resp = requests.get(url)\n self.check_security_headers(resp)\n self.check_csrf_cookies(resp)\n self.assertEqual(resp.status_code, 200)\n\n def test_static_proxy_req(self):\n \"\"\"Test that send a request for notes.\"\"\"\n url = self.get_server_url()\n notes = Path(ROOT) / \"notes\"\n for html in notes.rglob(\"*.html\"):\n page = html.relative_to(ROOT)\n u = f\"{url}/{page}\"\n resp = requests.get(u)\n self.check_security_headers(resp)\n self.check_csrf_cookies(resp)\n self.assertEqual(resp.status_code, 200)\n\n def test_acme_req(self):\n \"\"\"Test that send a request for a acme key.\"\"\"\n url = self.get_server_url()\n u = url + \"/.well-known/acme-challenge/token\"\n resp = requests.get(u)\n self.check_security_headers(resp)\n self.assertEqual(resp.status_code, 200)\n\n u = url + \"/.well-known/acme-challenge/foo\"\n resp = requests.get(u)\n self.check_security_headers(resp)\n self.assertEqual(resp.status_code, 404)\n\n def test_find_key(self):\n \"\"\"Test that find a acme key from the environment.\"\"\"\n token = self.token\n key = self.key\n self.assertEqual(find_key(token), key)\n\n del os.environ[\"ACME_TOKEN\"]\n del os.environ[\"ACME_KEY\"]\n\n os.environ[\"ACME_TOKEN_ENV\"] = token\n os.environ[\"ACME_KEY_ENV\"] = key\n self.assertEqual(find_key(token), key)\n\n del os.environ[\"ACME_TOKEN_ENV\"]\n del os.environ[\"ACME_KEY_ENV\"]\n\n def test_acme(self):\n \"\"\"Test that send a request for a acme key.\"\"\"\n token = self.token\n key = self.key\n self.assertEqual(acme(token), key)\n\n token = token + \"_env\"\n key = key + \"_env\"\n os.environ[\"ACME_TOKEN_ENV\"] = token\n os.environ[\"ACME_KEY_ENV\"] = key\n self.assertEqual(find_key(token), key)\n\n del os.environ[\"ACME_TOKEN_ENV\"]\n del os.environ[\"ACME_KEY_ENV\"]\n\n self.assertRaises(NotFound, acme, token)\n\n def test_index_redirection(self):\n \"\"\"Test index page redirection.\"\"\"\n resp = index_redirection()\n self.assertEqual(resp.status_code, 200)\n resp.close()\n\n def test_static_proxy(self):\n \"\"\"Test that request static pages.\"\"\"\n notes = Path(ROOT) / \"notes\"\n for html in notes.rglob(\"*.html\"):\n u = html.relative_to(ROOT)\n resp = static_proxy(u)\n self.assertEqual(resp.status_code, 200)\n resp.close()\n\n u = \"notes/../conf.py\"\n _, code = static_proxy(u)\n self.assertEqual(code, 404)\n\n def test_page_not_found(self):\n \"\"\"Test page not found.\"\"\"\n html, status_code = page_not_found(None)\n self.assertEqual(status_code, 404)\n\n\nif __name__ == \"__main__\":\n unittest.main()\n"
},
{
"path": "docs/404.rst",
"content": ":orphan:\n\n404 Page Not Found\n==================\n\nWhat you were looking for is just not there.\n\n`Click here to go back to homepage. `_\n"
},
{
"path": "docs/Makefile",
"content": "# Makefile for Sphinx documentation\n#\n\n# You can set these variables from the command line.\nSPHINXOPTS =\nSPHINXBUILD = sphinx-build\nPAPER =\nBUILDDIR = _build\n\n# User-friendly check for sphinx-build\nifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1)\n$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/)\nendif\n\n# Internal variables.\nPAPEROPT_a4 = -D latex_paper_size=a4\nPAPEROPT_letter = -D latex_paper_size=letter\nALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .\n# the i18n builder cannot share the environment and doctrees with the others\nI18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .\n\n.PHONY: help\nhelp:\n\t@echo \"Please use \\`make ' where is one of\"\n\t@echo \" html to make standalone HTML files\"\n\t@echo \" dirhtml to make HTML files named index.html in directories\"\n\t@echo \" singlehtml to make a single large HTML file\"\n\t@echo \" pickle to make pickle files\"\n\t@echo \" json to make JSON files\"\n\t@echo \" htmlhelp to make HTML files and a HTML help project\"\n\t@echo \" qthelp to make HTML files and a qthelp project\"\n\t@echo \" applehelp to make an Apple Help Book\"\n\t@echo \" devhelp to make HTML files and a Devhelp project\"\n\t@echo \" epub to make an epub\"\n\t@echo \" latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter\"\n\t@echo \" latexpdf to make LaTeX files and run them through pdflatex\"\n\t@echo \" latexpdfja to make LaTeX files and run them through platex/dvipdfmx\"\n\t@echo \" text to make text files\"\n\t@echo \" man to make manual pages\"\n\t@echo \" texinfo to make Texinfo files\"\n\t@echo \" info to make Texinfo files and run them through makeinfo\"\n\t@echo \" gettext to make PO message catalogs\"\n\t@echo \" changes to make an overview of all changed/added/deprecated items\"\n\t@echo \" xml to make Docutils-native XML files\"\n\t@echo \" pseudoxml to make pseudoxml-XML files for display purposes\"\n\t@echo \" linkcheck to check all external links for integrity\"\n\t@echo \" doctest to run all doctests embedded in the documentation (if enabled)\"\n\t@echo \" coverage to run coverage check of the documentation (if enabled)\"\n\n.PHONY: clean\nclean:\n\trm -rf $(BUILDDIR)/*\n\n.PHONY: html\nhtml:\n\t$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html\n\t@echo\n\t@echo \"Build finished. The HTML pages are in $(BUILDDIR)/html.\"\n\n.PHONY: dirhtml\ndirhtml:\n\t$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml\n\t@echo\n\t@echo \"Build finished. The HTML pages are in $(BUILDDIR)/dirhtml.\"\n\n.PHONY: singlehtml\nsinglehtml:\n\t$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml\n\t@echo\n\t@echo \"Build finished. The HTML page is in $(BUILDDIR)/singlehtml.\"\n\n.PHONY: pickle\npickle:\n\t$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle\n\t@echo\n\t@echo \"Build finished; now you can process the pickle files.\"\n\n.PHONY: json\njson:\n\t$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json\n\t@echo\n\t@echo \"Build finished; now you can process the JSON files.\"\n\n.PHONY: htmlhelp\nhtmlhelp:\n\t$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp\n\t@echo\n\t@echo \"Build finished; now you can run HTML Help Workshop with the\" \\\n\t \".hhp project file in $(BUILDDIR)/htmlhelp.\"\n\n.PHONY: qthelp\nqthelp:\n\t$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp\n\t@echo\n\t@echo \"Build finished; now you can run \"qcollectiongenerator\" with the\" \\\n\t \".qhcp project file in $(BUILDDIR)/qthelp, like this:\"\n\t@echo \"# qcollectiongenerator $(BUILDDIR)/qthelp/pysheeet.qhcp\"\n\t@echo \"To view the help file:\"\n\t@echo \"# assistant -collectionFile $(BUILDDIR)/qthelp/pysheeet.qhc\"\n\n.PHONY: applehelp\napplehelp:\n\t$(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp\n\t@echo\n\t@echo \"Build finished. The help book is in $(BUILDDIR)/applehelp.\"\n\t@echo \"N.B. You won't be able to view it unless you put it in\" \\\n\t \"~/Library/Documentation/Help or install it in your application\" \\\n\t \"bundle.\"\n\n.PHONY: devhelp\ndevhelp:\n\t$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp\n\t@echo\n\t@echo \"Build finished.\"\n\t@echo \"To view the help file:\"\n\t@echo \"# mkdir -p $$HOME/.local/share/devhelp/pysheeet\"\n\t@echo \"# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/pysheeet\"\n\t@echo \"# devhelp\"\n\n.PHONY: epub\nepub:\n\t$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub\n\t@echo\n\t@echo \"Build finished. The epub file is in $(BUILDDIR)/epub.\"\n\n.PHONY: latex\nlatex:\n\t$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex\n\t@echo\n\t@echo \"Build finished; the LaTeX files are in $(BUILDDIR)/latex.\"\n\t@echo \"Run \\`make' in that directory to run these through (pdf)latex\" \\\n\t \"(use \\`make latexpdf' here to do that automatically).\"\n\n.PHONY: latexpdf\nlatexpdf:\n\t$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex\n\t@echo \"Running LaTeX files through pdflatex...\"\n\t$(MAKE) -C $(BUILDDIR)/latex all-pdf\n\t@echo \"pdflatex finished; the PDF files are in $(BUILDDIR)/latex.\"\n\n.PHONY: latexpdfja\nlatexpdfja:\n\t$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex\n\t@echo \"Running LaTeX files through platex and dvipdfmx...\"\n\t$(MAKE) -C $(BUILDDIR)/latex all-pdf-ja\n\t@echo \"pdflatex finished; the PDF files are in $(BUILDDIR)/latex.\"\n\n.PHONY: text\ntext:\n\t$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text\n\t@echo\n\t@echo \"Build finished. The text files are in $(BUILDDIR)/text.\"\n\n.PHONY: man\nman:\n\t$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man\n\t@echo\n\t@echo \"Build finished. The manual pages are in $(BUILDDIR)/man.\"\n\n.PHONY: texinfo\ntexinfo:\n\t$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo\n\t@echo\n\t@echo \"Build finished. The Texinfo files are in $(BUILDDIR)/texinfo.\"\n\t@echo \"Run \\`make' in that directory to run these through makeinfo\" \\\n\t \"(use \\`make info' here to do that automatically).\"\n\n.PHONY: info\ninfo:\n\t$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo\n\t@echo \"Running Texinfo files through makeinfo...\"\n\tmake -C $(BUILDDIR)/texinfo info\n\t@echo \"makeinfo finished; the Info files are in $(BUILDDIR)/texinfo.\"\n\n.PHONY: gettext\ngettext:\n\t$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale\n\t@echo\n\t@echo \"Build finished. The message catalogs are in $(BUILDDIR)/locale.\"\n\n.PHONY: changes\nchanges:\n\t$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes\n\t@echo\n\t@echo \"The overview file is in $(BUILDDIR)/changes.\"\n\n.PHONY: linkcheck\nlinkcheck:\n\t$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck\n\t@echo\n\t@echo \"Link check complete; look for any errors in the above output \" \\\n\t \"or in $(BUILDDIR)/linkcheck/output.txt.\"\n\n.PHONY: doctest\ndoctest:\n\t$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest\n\t@echo \"Testing of doctests in the sources finished, look at the \" \\\n\t \"results in $(BUILDDIR)/doctest/output.txt.\"\n\n.PHONY: coverage\ncoverage:\n\t$(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage\n\t@echo \"Testing of coverage in the sources finished, look at the \" \\\n\t \"results in $(BUILDDIR)/coverage/python.txt.\"\n\n.PHONY: xml\nxml:\n\t$(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml\n\t@echo\n\t@echo \"Build finished. The XML files are in $(BUILDDIR)/xml.\"\n\n.PHONY: pseudoxml\npseudoxml:\n\t$(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml\n\t@echo\n\t@echo \"Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml.\"\n"
},
{
"path": "docs/_extra/robots.txt",
"content": "User-agent: *\nAllow: /\nSitemap: https://www.pythonsheets.com/sitemap.xml\n"
},
{
"path": "docs/_static/.gitignore",
"content": "# Ignore everything in this directory\n*\n# Except this file\n!.gitignore\n!guido.png\n!logo.svg\n!style.css\n!carbonad.css\n!favicon.ico\n"
},
{
"path": "docs/_static/carbonad.css",
"content": "#carbonads {\n display: block;\n overflow: hidden;\n padding: 1em;\n padding-bottom: 0.3em;\n line-height: 1.5;\n margin-top: 10px;\n margin-bottom: 10px;\n}\n\n#carbonads a {\n text-decoration: none !important;\n border-bottom: none;\n}\n\n#carbonads a:hover {\n color: inherit;\n}\n\n#carbonads span {\n display: block;\n overflow: hidden;\n}\n\n.carbon-img {\n display: block;\n margin: 0 auto 8px;\n}\n\n.carbon-text {\n display: block;\n text-align: left;\n margin-bottom: .1em;\n color: #666;\n}\n\n.carbon-poweredby {\n display: block;\n text-align: left;\n font-size: .9em;\n color: #888 !important;\n}\n\n@media only screen and (min-width: 320px) and (max-width: 875px) {\n #carbonads {\n float: none;\n max-width: 330px;\n border: 0;\n display: block;\n overflow: hidden;\n margin-top: 20px;\n margin-bottom: 20px;\n border-radius: 4px;\n text-align: center;\n box-shadow: 0 0 0 1px hsla(0, 0%, 0%, .1);\n font-size: var(--font-size);\n background-color: #eee;\n line-height: 1.5;\n }\n #carbonads span {\n position: relative;\n }\n #carbonads > span {\n max-width: none;\n }\n .carbon-img {\n float: left;\n margin: 0;\n }\n\n .carbon-img img {\n max-width: 130px !important;\n }\n .carbon-text {\n float: left;\n margin-bottom: 0;\n padding: 8px 20px;\n text-align: left;\n color: #333 !important;\n max-width: calc(100% - 130px - 3em);\n }\n .carbon-poweredby {\n left: 130px;\n bottom: 0;\n display: block;\n color: #555 !important;\n text-align: right;\n width: 100%;\n }\n}\n"
},
{
"path": "docs/_static/style.css",
"content": "nav#table-of-contents {\n\tdisplay: none;\n}\n\ndiv.highlight > pre {\n font-size: 14px;\n border-radius: 3px;\n background: #f6f8fa !important;\n border: 1px solid #000000 !important;\n}\n\n:root {\n --cu-boulder-gold: #CFB87C;\n}\n\n.bd-container {\n max-width: 99%;\n}\n\n.bd-container .bd-container__inner {\n max-width: 99%;\n}\n\n.bd-main .bd-content .bd-article-container {\n max-width: 100em;\n}\n\n.code-block-caption {\n color: black;\n}\n\n.bd-sidebar-primary li.has-children>details>summary .toctree-toggle {\n justify-content: left;\n}\n\nhtml[data-theme=light] {\n --pst-font-size-base: none;\n --pst-color-secondary: #176de8;\n --pst-color-primary: #176de8;\n}\n\n.graph#doc-flowchart .node text {\n font-weight: bold;\n}\n\n.bd-content .sd-tab-set .sd-tab-content {\n padding: 1.5rem;\n}\n\na {\n text-decoration: none;\n}\n\na:hover {\n text-decoration: underline;\n}\n\nbutton.theme-switch-button {\n display: none !important;\n}\n\nblockquote {\n background-color: transparent;\n border: none;\n}\n"
},
{
"path": "docs/_templates/carbonad.html",
"content": "\n"
},
{
"path": "docs/_templates/cheatsheets.html",
"content": "
This project tries to provide many snippets of Python code that make life easier.
\n"
},
{
"path": "docs/conf.py",
"content": "# -*- coding: utf-8 -*-\n#\n# python-cheatsheet documentation build configuration file, created by\n# sphinx-quickstart on Sun Feb 28 09:26:04 2016.\n#\n# This file is execfile()d with the current directory set to its\n# containing dir.\n#\n# Note that not all possible configuration values are present in this\n# autogenerated file.\n#\n# All configuration values have a default; values that are commented out\n# serve to show the default.\n\nfrom datetime import datetime\nimport os\n\n# If extensions (or modules to document with autodoc) are in another directory,\n# add these directories to sys.path here. If the directory is relative to the\n# documentation root, use os.path.abspath to make it absolute, like shown here.\n# sys.path.insert(0, os.path.abspath('.'))\n\n# -- General configuration ------------------------------------------------\n\n# If your documentation needs a minimal Sphinx version, state it here.\n#needs_sphinx = '1.0'\n\n# Add any Sphinx extension module names here, as strings. They can be\n# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom\n# ones.\nextensions = [\n 'sphinx.ext.todo',\n 'sphinx.ext.coverage',\n 'sphinx.ext.viewcode',\n 'myst_parser',\n 'sphinx_copybutton',\n 'sphinx.ext.graphviz',\n 'sphinx_design',\n 'sphinx.ext.extlinks'\n]\n\nmyst_enable_extensions = [\n \"colon_fence\",\n \"attrs_inline\",\n \"attrs_block\",\n \"tasklist\",\n \"substitution\",\n]\n\nmyst_enable_checkboxes = True\nmyst_heading_anchors = 6\ncopybutton_prompt_text = r'^\\$ '\ncopybutton_prompt_is_regexp = True\n\n# Add any paths that contain templates here, relative to this directory.\ntemplates_path = ['_templates']\n\n# The suffix(es) of source filenames.\n# You can specify multiple suffix as a list of string:\n# source_suffix = ['.rst', '.md']\nsource_suffix = '.rst'\n\n# The encoding of source files.\n#source_encoding = 'utf-8-sig'\n\n# The master toctree document.\nmaster_doc = 'index'\n\n# General information about the project.\nyear = datetime.now().year\nproject = u'pysheeet'\ncopyright = u'2016-{}, crazyguitar'.format(year)\nauthor = u'crazyguitar'\n\n# The version info for the project you're documenting, acts as replacement for\n# |version| and |release|, also used in various other places throughout the\n# built documents.\n#\n# The short X.Y version.\nversion = u'0.1.0'\n# The full version, including alpha/beta/rc tags.\nrelease = u'0.1.0'\n\n# The language for content autogenerated by Sphinx. Refer to documentation\n# for a list of supported languages.\n#\n# This is also used if you do content translation via gettext catalogs.\n# Usually you set \"language\" from the command line for these cases.\nlanguage = 'en'\n\n# There are two options for replacing |today|: either, you set today to some\n# non-false value, then it is used:\n#today = ''\n# Else, today_fmt is used as the format for a strftime call.\n#today_fmt = '%B %d, %Y'\n\n# List of patterns, relative to source directory, that match files and\n# directories to ignore when looking for source files.\nexclude_patterns = []\n\n# The reST default role (used for this markup: `text`) to use for all\n# documents.\n#default_role = None\n\n# If true, '()' will be appended to :func: etc. cross-reference text.\n#add_function_parentheses = True\n\n# If true, the current module name will be prepended to all description\n# unit titles (such as .. function::).\n#add_module_names = True\n\n# If true, sectionauthor and moduleauthor directives will be shown in the\n# output. They are ignored by default.\n#show_authors = False\n\n# The name of the Pygments (syntax highlighting) style to use.\npygments_style = 'sphinx'\n\n# A list of ignored prefixes for module index sorting.\n#modindex_common_prefix = []\n\n# If true, keep warnings as \"system message\" paragraphs in the built documents.\n#keep_warnings = False\n\n# If true, `todo` and `todoList` produce output, else they produce nothing.\ntodo_include_todos = True\n\n\n# -- Options for HTML output ----------------------------------------------\n\n# The theme to use for HTML and HTML Help pages. See the documentation for\n# a list of builtin themes.\nhtml_theme = 'sphinx_book_theme'\n\n# Theme options are theme-specific and customize the look and feel of a theme\n# further. For a list of options available for each theme, see the\n# documentation.\nhtml_theme_options = {\n \"repository_url\": \"https://github.com/crazyguitar/pysheeet\",\n \"use_repository_button\": True,\n}\n\n# Custom sidebar templates\n\n# Add any paths that contain custom themes here, relative to this directory.\n#html_theme_path = []\n\n# The name for this set of Sphinx documents. If None, it defaults to\n# \" v documentation\".\nhtml_title = \"Python Cheat Sheet\"\n\n# A shorter title for the navigation bar. Default is the same as html_title.\n#html_short_title = None\n\n# The name of an image file (relative to this directory) to place at the top\n# of the sidebar.\nhtml_logo = \"_static/logo.svg\"\n\n# The name of an image file (within the static path) to use as favicon of the\n# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32\n# pixels large.\nhtml_favicon = '_static/favicon.ico'\n\n# Add any paths that contain custom static files (such as style sheets) here,\n# relative to this directory. They are copied after the builtin static files,\n# so a file named \"default.css\" will overwrite the builtin \"default.css\".\nhtml_static_path = ['_static']\nhtml_css_files = ['style.css']\n\n# Add any extra paths that contain custom files (such as robots.txt or\n# .htaccess) here, relative to this directory. These files are copied\n# directly to the root of the documentation.\nhtml_extra_path = ['_extra']\nhtml_context = {\n \"tracking_id\": os.environ.get(\"TRACKING_ID\"),\n}\n\nhas_carbonad = os.environ.get(\"CARBONAD_SERVE\") and os.environ.get(\"CARBONAD_PLACEMENT\")\nif has_carbonad:\n html_context[\"carbonad_serve\"] = os.environ.get(\"CARBONAD_SERVE\")\n html_context[\"carbonad_placement\"] = os.environ.get(\"CARBONAD_PLACEMENT\")\n\n# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,\n# using the given strftime format.\n#html_last_updated_fmt = '%b %d, %Y'\n\n# If true, SmartyPants will be used to convert quotes and dashes to\n# typographically correct entities.\n#html_use_smartypants = True\n\n\n# Additional templates that should be rendered to pages, maps page names to\n# template names.\n#html_additional_pages = {}\n\n# If false, no module index is generated.\n#html_domain_indices = True\n\n# If false, no index is generated.\n#html_use_index = True\n\n# If true, the index is split into individual pages for each letter.\n#html_split_index = False\n\n# If true, links to the reST sources are added to the pages.\n#html_show_sourcelink = True\n\n# If true, \"Created using Sphinx\" is shown in the HTML footer. Default is True.\n#html_show_sphinx = True\n\n# If true, \"(C) Copyright ...\" is shown in the HTML footer. Default is True.\n#html_show_copyright = True\n\n# If true, an OpenSearch description file will be output, and all pages will\n# contain a tag referring to it. The value of this option must be the\n# base URL from which the finished HTML is served.\n#html_use_opensearch = ''\n\n# This is the file name suffix for HTML files (e.g. \".xhtml\").\n#html_file_suffix = None\n\n# Language to be used for generating the HTML full-text search index.\n# Sphinx supports the following languages:\n# 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja'\n# 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr'\n#html_search_language = 'en'\n\n# A dictionary with options for the search language support, empty by default.\n# Now only 'ja' uses this config value\n#html_search_options = {'type': 'default'}\n\n# The name of a javascript file (relative to the configuration directory) that\n# implements a search results scorer. If empty, the default will be used.\n#html_search_scorer = 'scorer.js'\n\n# Output file base name for HTML help builder.\nhtmlhelp_basename = 'python-cheatsheetdoc'\n\n# -- Options for LaTeX output ---------------------------------------------\n\nlatex_elements = {\n# The paper size ('letterpaper' or 'a4paper').\n#'papersize': 'letterpaper',\n\n# The font size ('10pt', '11pt' or '12pt').\n#'pointsize': '10pt',\n\n# Additional stuff for the LaTeX preamble.\n#'preamble': '',\n\n# Latex figure (float) alignment\n#'figure_align': 'htbp',\n}\n\n# Grouping the document tree into LaTeX files. List of tuples\n# (source start file, target name, title,\n# author, documentclass [howto, manual, or own class]).\nlatex_documents = [\n (master_doc, 'python-cheatsheet.tex', u'python-cheatsheet Documentation',\n u'crazyguitar', 'manual'),\n]\n\n# The name of an image file (relative to this directory) to place at the top of\n# the title page.\n#latex_logo = None\n\n# For \"manual\" documents, if this is true, then toplevel headings are parts,\n# not chapters.\n#latex_use_parts = False\n\n# If true, show page references after internal links.\n#latex_show_pagerefs = False\n\n# If true, show URL addresses after external links.\n#latex_show_urls = False\n\n# Documents to append as an appendix to all manuals.\n#latex_appendices = []\n\n# If false, no module index is generated.\n#latex_domain_indices = True\n\n\n# -- Options for manual page output ---------------------------------------\n\n# One entry per manual page. List of tuples\n# (source start file, name, description, authors, manual section).\nman_pages = [\n (master_doc, 'python-cheatsheet', u'python-cheatsheet Documentation',\n [author], 1)\n]\n\n# If true, show URL addresses after external links.\n#man_show_urls = False\n\n\n# -- Options for Texinfo output -------------------------------------------\n\n# Grouping the document tree into Texinfo files. List of tuples\n# (source start file, target name, title, author,\n# dir menu entry, description, category)\ntexinfo_documents = [\n (master_doc, 'python-cheatsheet', u'python-cheatsheet Documentation',\n author, 'python-cheatsheet', 'One line description of project.',\n 'Miscellaneous'),\n]\n\nhtml_sidebars = {\n \"**\": [\n \"navbar-logo.html\",\n \"search-button-field.html\",\n \"sbt-sidebar-nav.html\",\n \"carbonad.html\",\n ]\n}\n\n# Documents to append as an appendix to all manuals.\n#texinfo_appendices = []\n\n# If false, no module index is generated.\n#texinfo_domain_indices = True\n\n# How to display URL addresses: 'footnote', 'no', or 'inline'.\n#texinfo_show_urls = 'footnote'\n\n# If true, do not generate a @detailmenu in the \"Top\" node's menu.\n#texinfo_no_detailmenu = False\n\ndef add_html_link(app, pagename, templatename, context, doctree):\n \"\"\"Append html page.\"\"\"\n if pagename in ['404', 'search', 'genindex']:\n return\n app.sitemaps.append({\n 'pagename': pagename + \".html\",\n 'priority': '1.0' if pagename == 'index' else '0.8',\n 'changefreq': 'weekly' if pagename == 'index' else 'monthly'\n })\n\n\ndef create_sitemap(app, exception):\n \"\"\"Generate a sitemap.xml\"\"\"\n from xml.etree.ElementTree import ElementTree, Element, SubElement\n from datetime import datetime\n\n r = Element(\"urlset\")\n r.set(\"xmlns\", \"http://www.sitemaps.org/schemas/sitemap/0.9\")\n r.set(\"xmlns:xsi\", \"http://www.w3.org/2001/XMLSchema-instance\")\n r.set(\"xsi:schemaLocation\", \"http://www.sitemaps.org/schemas/sitemap/0.9\" +\n \" http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd\")\n\n for link_info in app.sitemaps:\n url = SubElement(r, \"url\")\n now = datetime.now()\n SubElement(url, \"loc\").text = app.pysheeet + link_info['pagename']\n SubElement(url, \"lastmod\").text = now.date().isoformat()\n SubElement(url, \"changefreq\").text = link_info['changefreq']\n SubElement(url, \"priority\").text = link_info['priority']\n\n f = app.outdir + \"/sitemap.xml\"\n t = ElementTree(r)\n t.write(f, xml_declaration=True, encoding='utf-8', method=\"xml\")\n\n\ndef setup(app):\n \"\"\"Customize setup.\"\"\"\n site = os.environ.get(\"PYSHEEET\")\n if not site:\n return\n\n if site[-1] != '/':\n site += '/'\n\n # create a sitemap\n app.pysheeet = site\n app.sitemaps = []\n app.connect('html-page-context', add_html_link)\n app.connect('build-finished', create_sitemap)\n"
},
{
"path": "docs/index.rst",
"content": ".. python-cheatsheet documentation master file, created by\n sphinx-quickstart on Sun Feb 28 09:26:04 2016.\n You can adapt this file completely to your liking, but it should at least\n contain the root `toctree` directive.\n\n.. meta::\n :description lang=en: Comprehensive Python cheat sheet with practical code snippets, examples, and tutorials for Python developers. Learn Python basics, advanced topics, databases, networking, and more.\n :keywords: Python, Python Cheat Sheet, Python Tutorial, Python Examples, Python Code Snippets, Programming, Development, Python Reference, Python Guide\n\n\nPython Cheat Sheet - Complete Guide with Code Examples\n======================================================\n\nWelcome to **pysheeet** - your ultimate Python cheat sheet! This comprehensive resource contains practical Python\ncode snippets, examples, and tutorials to make coding easier and more efficient for Python developers of all levels.\nFrom basic Python syntax to advanced topics like databases, networking, and multitasking, this cheat sheet serves as your\ncomplete Python reference guide. Ideal for beginners learning Python fundamentals and experienced developers seeking\nquick code examples. Whether you're learning Python for web development, data science, automation, or general programming,\nyou'll find practical examples that save you time and improve your coding efficiency.\n\nContributions are always welcome—feel free to share ideas for new snippets, improvements, or clearer explanations!\nIf you'd like to contribute, `fork pysheeet on GitHub`_.\nIf there is any question or suggestion, please create an issue on `GitHub Issues`_.\n\n.. _fork pysheeet on GitHub: https://github.com/crazyguitar/pysheeet\n.. _GitHub Issues: https://github.com/crazyguitar/pysheeet/issues\n\nPlugin\n------\n\n**pysheeet** is available as a `Claude Code `_ plugin. Once installed,\nClaude automatically uses the cheat sheets to answer Python questions.\n\n.. code-block:: bash\n\n # Step 1: Add the marketplace\n claude plugin marketplace add crazyguitar/pysheeet\n\n # Step 2: Install the plugin\n claude plugin install pysheeet@pysheeet\n\nFor local testing and manual installation, see the main `README `_.\n\nWhat's New In Python 3\n----------------------\n\nThe official document, `What's New In Python`_, displays all of the most\nimportant changes. However, if you're too busy to read the whole changes,\nthis part provides a brief glance of new features in Python 3.\n\n.. _What's New In Python: https://docs.python.org/3/whatsnew/index.html\n\n.. toctree::\n :maxdepth: 1\n\n notes/python-new-py3\n\n\nPython Cheat Sheet\n------------------\n\nThis section focuses on commonly used Python code snippets. The cheat sheet\ncovers not only core Python features but also essential data structures,\nalgorithms, and frequently used modules to help programmers efficiently tackle\neveryday tasks.\n\n.. toctree::\n :maxdepth: 1\n\n notes/basic/index\n notes/os/index\n notes/concurrency/index\n notes/asyncio/index\n notes/network/index\n notes/database/index\n notes/security/index\n notes/extension/index\n notes/llm/index\n notes/hpc/index\n notes/appendix/index\n"
},
{
"path": "docs/notes/appendix/disaggregated-prefill-decode.rst",
"content": ".. meta::\n :description lang=en: Evaluating disaggregated prefill/decode for LLM serving with vLLM, NIXL, and EFA on AWS\n :keywords: LLM, vLLM, NIXL, disaggregated prefill decode, KV cache, EFA, inference serving\n\nIs Disaggregated Prefill/Decode a Silver Bullet for LLM Serving?\n================================================================\n\n:Date: 2026-03-10\n\nAbstract\n--------\n\nDisaggregated prefill/decode has gained traction as a promising architecture for\nLLM serving, separating the compute-intensive prefill phase from the\nmemory-bound decode phase onto dedicated node groups. Proponents argue that this\nseparation enables independent scaling and eliminates interference between the\ntwo phases. But is it truly a silver bullet? This article puts the claim to the\ntest by evaluating disaggregated prefill/decode using vLLM with NIXL over the\nAWS Elastic Fabric Adapter (EFA) on a 4-node cluster. We compare data\nparallelism and simple load-balanced routing as baselines against disaggregated\nconfigurations. Our results show that while disaggregation dramatically reduces\ninter-token latency (ITL), it comes at a significant cost to throughput and\ntime-to-first-token (TTFT), revealing that the architecture is far from a\nuniversal solution.\n\nIntroduction\n------------\n\nIn standard LLM serving, each node handles both prefill and decode for incoming\nrequests. The prefill phase is compute-bound and processes the entire input\nprompt in parallel, while the decode phase is memory-bandwidth-bound and\ngenerates tokens autoregressively. When both phases share the same GPU pool,\nlong prefill requests can block decode iterations, increasing inter-token\nlatency for concurrent requests.\n\nDisaggregated prefill/decode addresses this interference by assigning prefill\nand decode to separate node groups. After a prefill node completes prompt\nprocessing, the KV cache is transferred to a decode node via a high-bandwidth\ninterconnect. NIXL [1]_ (NVIDIA Inference Xfer Library) provides the KV cache\ntransfer mechanism, and on AWS, this transfer occurs over EFA using the\n``LIBFABRIC`` backend.\n\nThe appeal is intuitive: by isolating decode nodes from prefill interference,\ntoken generation should proceed at a steady, low-latency pace. However, this\nseparation introduces new costs — KV cache transfer overhead, prefill node\nsaturation at long input lengths, and reduced effective cluster capacity for\neach phase. The question is whether these trade-offs are worthwhile compared to\nsimpler alternatives like data parallelism or stateless load-balanced routing.\n\nThis experiment uses vLLM [2]_ with the\n``NixlConnector`` to orchestrate disaggregated serving, and ``vllm-router`` [3]_ as\na reverse proxy to load-balance requests across node groups. The experiment\ncode is available under `src/nixl `_ in the companion repository.\n\nContainer Image\n---------------\n\nThe experiment uses a custom Docker image that bundles all required components.\nThe ``Dockerfile`` builds on ``nvidia/cuda:12.8.1-devel-ubuntu24.04`` and\ninstalls the following stack:\n\n- **GDRCopy** v2.5.1 for GPU-direct memory registration\n- **EFA installer** v1.47.0 for AWS Elastic Fabric Adapter support\n- **UCX** v1.20.0 built with verbs, rdmacm, and EFA transport\n- **NIXL** v0.10.1 with ``LIBFABRIC`` backend for KV cache transfer\n- **nixlbench** for standalone NIXL bandwidth/latency microbenchmarks\n- **PyTorch** 2.9.1, **flash-attn** 2.8.1, and **DeepGEMM** v2.1.1.post3\n- **vLLM** 0.15.1 with ``NixlConnector`` support\n- **vllm-router** for load-balancing across disaggregated node groups\n\nThe image is built and saved as a portable tarball via the ``Makefile``:\n\n.. code-block:: bash\n\n make docker && make save\n\nThis produces ``nixl-latest.tar.gz``, which is distributed to all Slurm nodes\nat launch time via ``pigz`` decompression and ``docker load``.\n\nServing Script\n--------------\n\nThe ``vllm.sbatch`` script orchestrates multi-node vLLM serving on Slurm. It\naccepts two key flags that control the serving topology:\n\n- ``--route R``: splits the allocated nodes into ``R`` identical groups, each\n running an independent vLLM instance. A ``vllm-router`` process on the head\n node round-robins requests across groups.\n- ``--prefill P``: within each group, assigns ``P`` nodes as prefill-only\n (``kv_producer``) and the remaining nodes as decode-only (``kv_consumer``).\n KV cache transfer between prefill and decode nodes uses ``NixlConnector``\n with the ``LIBFABRIC`` backend over EFA.\n\nWhen ``--prefill 0`` (default), all nodes in a group run standard data-parallel\nserving. The script computes ``DP = nodes_per_group * (8 / TP)`` and launches\nvLLM with ``--data-parallel-size`` accordingly.\n\nFor disaggregated mode, each prefill and decode node runs as an independent\nvLLM process with explicit KV transfer configuration:\n\n.. code-block:: bash\n\n # Prefill node\n vllm serve ... \\\n --kv-transfer-config.kv_connector NixlConnector \\\n --kv-transfer-config.kv_role kv_producer \\\n --kv-transfer-config.kv_connector_extra_config.backends+ LIBFABRIC\n\n # Decode node\n vllm serve ... \\\n --kv-transfer-config.kv_connector NixlConnector \\\n --kv-transfer-config.kv_role kv_consumer \\\n --kv-transfer-config.kv_connector_extra_config.backends+ LIBFABRIC\n\nThe router uses ``round_robin`` policy for pure-DP groups and\n``consistent_hash`` with ``--vllm-pd-disaggregation`` for PD groups, directing\ninitial requests to prefill endpoints and subsequent decode traffic to decode\nendpoints:\n\n.. code-block:: bash\n\n # Router for pure-DP groups (round-robin across group endpoints)\n vllm-router \\\n --policy round_robin \\\n --worker-urls http://:8000 http://:8001 \\\n --host 0.0.0.0 --port 8010\n\n # Router for PD disaggregation (consistent hash with prefill/decode split)\n vllm-router \\\n --policy consistent_hash \\\n --vllm-pd-disaggregation \\\n --prefill http://:8000 \\\n --decode http://:8001 --decode http://:8002 \\\n --host 0.0.0.0 --port 8010\n\nEach container is launched with ``--privileged``, ``--net=host``, and explicit\n``/dev/infiniband/uverbs*`` and ``/dev/gdrdrv`` device mounts to enable\nGPU-direct RDMA over EFA.\n\nBenchmark Script\n----------------\n\nThe ``bench.sh`` script wraps ``vllm bench serve`` and handles Docker image\nloading transparently. If the ``vllm`` CLI is not available on the host, the\nscript re-executes itself inside the container. It points the benchmark client\nat the router endpoint (or the direct vLLM endpoint for single-group\nconfigurations):\n\n.. code-block:: bash\n\n bash bench.sh -H -p -- \\\n --model /fsx/models/deepseek-ai/DeepSeek-V2-Lite \\\n --dataset-name random \\\n --random-input-len 512 --random-output-len 256 \\\n --num-prompts 1024\n\nExperimental Setup\n------------------\n\nAll experiments run on 4 nodes with 8 GPUs each (TP=8) using\nDeepSeek-V2-Lite as the model. The benchmark uses random input/output data\nwith 1024 prompts via ``vllm bench serve``.\n\nThe configurations are:\n\n- **Baseline (data parallelism)**: 4 nodes, TP=8, DP=4. All nodes serve both\n prefill and decode. This is the standard data-parallel serving setup.\n- **Route 2**: 2 groups of 2 nodes each, TP=8, DP=2 per group. A router\n round-robins requests across groups. Each group independently handles both\n prefill and decode.\n- **Route 4**: 4 groups of 1 node each, TP=8, no data parallelism. A router\n distributes requests across all 4 independent nodes.\n- **PD 1P3D**: Disaggregated prefill/decode with 1 prefill node and 3 decode\n nodes. KV cache is transferred from the prefill node to decode nodes via NIXL.\n- **PD 2P2D**: Disaggregated prefill/decode with 2 prefill nodes and 2 decode\n nodes.\n\n.. code-block:: bash\n\n # Exp 1: Baseline — 4 nodes, TP=8, pure DP\n salloc -N 4 bash vllm.sbatch \\\n --model /fsx/models/deepseek-ai/DeepSeek-V2-Lite \\\n --gpu-memory-utilization 0.9\n\n # Exp 2: 2 groups × 2 nodes, DP=2 per group, router round-robins\n salloc -N 4 bash vllm.sbatch --route 2 \\\n --model /fsx/models/deepseek-ai/DeepSeek-V2-Lite \\\n --gpu-memory-utilization 0.9\n\n # Exp 3: 4 groups × 1 node, no DP, router round-robins\n salloc -N 4 bash vllm.sbatch --route 4 \\\n --model /fsx/models/deepseek-ai/DeepSeek-V2-Lite \\\n --gpu-memory-utilization 0.9\n\n # Exp 4: 1 prefill + 3 decode\n salloc -N 4 bash vllm.sbatch --prefill 1 \\\n --model /fsx/models/deepseek-ai/DeepSeek-V2-Lite \\\n --gpu-memory-utilization 0.9\n\n # Exp 5: 2 prefill + 2 decode\n salloc -N 4 bash vllm.sbatch --prefill 2 \\\n --model /fsx/models/deepseek-ai/DeepSeek-V2-Lite \\\n --gpu-memory-utilization 0.9\n\nResults\n-------\n\nWe evaluate each configuration along four metrics: output token throughput,\nrequest throughput, time to first token (TTFT), and inter-token latency (ITL).\nEach plot contains two panels — the left panel sweeps input length with a fixed\noutput length of 256 tokens (prefill-dominated regime), while the right panel\nsweeps output length with a fixed input length of 512 tokens (decode-dominated\nregime). This allows us to observe how each configuration behaves when the\nworkload shifts from prefill-heavy to decode-heavy.\n\nMicrobenchmark: KV Cache Transfer Bandwidth\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nBefore examining end-to-end serving results, we use ``nixlbench`` to measure\nthe raw NIXL transfer bandwidth over EFA between two nodes. This establishes\nan upper bound on KV cache transfer speed and helps contextualize the TTFT\noverhead observed in disaggregated configurations.\n\nThe benchmark runs in Multi-GPU (MG) mode with all 8 GPUs per node performing\nVRAM-to-VRAM transfers over the ``LIBFABRIC`` backend:\n\n.. code-block:: bash\n\n salloc -N 2 bash nixl.sbatch --backend LIBFABRIC \\\n --initiator_seg_type VRAM --target_seg_type VRAM \\\n --mode MG --num_initiator_dev 8 --num_target_dev 8\n\n Block Size (B) Batch Size B/W (GB/Sec) Avg Lat. (us) P99 Tx (us)\n ---------------------------------------------------------------------------------\n 4096 1 0.670064 6.1 47.0\n 8192 1 1.315392 6.2 45.0\n 16384 1 2.511416 6.5 47.0\n 32768 1 4.820423 6.8 50.0\n 65536 1 8.733224 7.5 56.0\n 131072 1 12.341950 10.6 52.0\n 262144 1 23.272188 11.3 59.0\n 524288 1 43.365764 12.1 62.0\n 1048576 1 74.816773 14.0 77.0\n 2097152 1 121.086563 17.3 105.0\n 4194304 1 180.631395 23.2 146.0\n 8388608 1 239.037623 35.1 247.0\n 16777216 1 289.500030 58.0 432.0\n 33554432 1 327.436372 102.5 796.0\n 67108864 1 349.608429 192.0 1724.0\n\n**Mapping to DeepSeek-V2-Lite KV cache transfer.** DeepSeek-V2-Lite uses\nMulti-head Latent Attention (MLA), which compresses the KV cache into a latent\nvector per token per layer. The per-token-per-layer KV cache size is\n``(kv_lora_rank + qk_rope_head_dim) × dtype_size = (512 + 64) × 2 = 1,152 bytes``.\nFor 512 input tokens across 27 layers, the total KV cache is approximately **15.2 MB**.\nWith TP=8, each GPU transfers about **1.9 MB**, which falls in the ~121 GB/s\nbandwidth range per the table above. Without tensor parallelism, the full 15.2 MB\ntransfer achieves approximately ~289 GB/s.\n\nOutput Token Throughput\n~~~~~~~~~~~~~~~~~~~~~~~\n\n.. image:: https://raw.githubusercontent.com/crazyguitar/pysheeet/master/docs/_static/appendix/nixl/throughput.png\n :alt: Output token throughput comparison\n\nThe left panel varies input length with a fixed output length of 256 tokens\n(prefill-dominated), while the right panel varies output length with a fixed\ninput length of 512 tokens (decode-dominated).\n\nFor prefill-dominated workloads, Route 4 achieves the highest throughput since\neach node operates independently without the overhead of data parallelism\ncoordination. The disaggregated configurations (PD 1P3D and PD 2P2D) show\ncompetitive throughput at shorter input lengths but degrade at longer inputs\nwhere the prefill nodes become the bottleneck.\n\nFor decode-dominated workloads, Route 4 again leads, followed by PD 1P3D.\nPD 2P2D shows the lowest throughput in this regime, as its two decode nodes\ncannot match the decode capacity of other configurations.\n\nRequest Throughput\n~~~~~~~~~~~~~~~~~~\n\n.. image:: https://raw.githubusercontent.com/crazyguitar/pysheeet/master/docs/_static/appendix/nixl/req_throughput.png\n :alt: Request throughput comparison\n\nRequest throughput follows a similar pattern. Route 4 consistently achieves the\nhighest request throughput across all configurations. The disaggregated PD 1P3D\nconfiguration maintains reasonable request throughput for short inputs but drops\nsignificantly at longer input lengths (4096 tokens), where the single prefill\nnode becomes saturated.\n\nTime to First Token (TTFT)\n~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n.. image:: https://raw.githubusercontent.com/crazyguitar/pysheeet/master/docs/_static/appendix/nixl/ttft.png\n :alt: TTFT comparison\n\nTTFT is critical for user-perceived latency. The baseline DP and Route 2\nconfigurations show moderate TTFT that scales with input length. Route 4\nachieves the lowest TTFT across all input lengths due to the absence of\ncross-node coordination.\n\nThe disaggregated configurations exhibit higher TTFT, particularly at longer\ninput lengths. PD 1P3D shows TTFT exceeding 37 seconds at 4096 input tokens,\nas all prefill work funnels through a single node. PD 2P2D improves on this\nbut still lags behind the non-disaggregated configurations. The additional\nlatency from KV cache transfer over NIXL contributes to the elevated TTFT.\n\nFor decode-dominated workloads (right panel), the differences are smaller. At\nshort output lengths (256–512 tokens), PD 1P3D shows 1–2 seconds higher TTFT\nthan the baseline, as the KV cache transfer overhead is proportionally more\nsignificant. At longer output lengths (1024+ tokens), the disaggregated\nconfigurations converge with or improve upon the baseline, as the baseline\nsuffers from increased prefill/decode contention under heavier concurrent\ndecode load.\n\nInter-Token Latency (ITL)\n~~~~~~~~~~~~~~~~~~~~~~~~~\n\n.. image:: https://raw.githubusercontent.com/crazyguitar/pysheeet/master/docs/_static/appendix/nixl/itl.png\n :alt: ITL comparison\n\nITL measures the latency between consecutive generated tokens during the decode\nphase. This is where disaggregated serving shows its primary advantage.\n\nIn the prefill-dominated regime (left panel), PD 1P3D achieves the lowest ITL\nacross all input lengths, with mean ITL as low as 10 ms at 4096 input tokens.\nBy isolating decode nodes from prefill interference, the decode phase runs\nuninterrupted. PD 2P2D also shows reduced ITL compared to the baseline, though\nthe benefit is less pronounced due to having fewer decode nodes. The baseline DP\nand Route configurations show higher ITL, particularly at longer input lengths\nwhere prefill and decode contend for the same GPU resources.\n\nIn the decode-dominated regime (right panel), Route 4 achieves the lowest ITL\n(~25–29 ms) since each node serves independently without cross-node\ncoordination. Among the disaggregated configurations, PD 1P3D outperforms\nPD 2P2D due to its greater decode capacity (3 decode nodes vs. 2), maintaining\nITL around 26–35 ms. PD 2P2D, with only 2 decode nodes, shows ITL comparable\nto the baseline (~45–50 ms). As output length increases, ITL gradually rises\nacross all configurations, reflecting the growing decode load.\n\nDiscussion\n----------\n\nSo, is disaggregated prefill/decode a silver bullet? The answer is clearly no —\nat least not under the conditions tested here. All benchmarks use randomly\ngenerated prompts, meaning every request produces a unique KV cache with zero\nprefix cache hit rate. This represents a worst-case scenario for disaggregated\nserving, where every prefill must be computed from scratch and the full KV cache\nmust be transferred over the network. In production workloads with shared system\nprompts or repeated prefixes, prefix caching on prefill nodes could\nsubstantially reduce redundant computation and transfer volume, potentially\nshifting the balance in favor of disaggregation. Even so, the results reveal a\nset of sharp trade-offs that make disaggregation a specialized tool rather than\na universal improvement:\n\n- **ITL wins, but throughput depends on scaling**: Disaggregated configurations\n deliver dramatically lower inter-token latency — PD 1P3D achieves as low as\n 10 ms ITL at long input lengths, up to 14× better than the baseline in\n prefill-dominated regimes and 1.4–2.4× better in decode-dominated regimes.\n The throughput and TTFT degradation observed here is partly an artifact of a\n fixed 4-node cluster: dedicating nodes to one role starves the other. In\n practice, prefill and decode pools can be scaled independently — adding more\n prefill nodes to eliminate the prefill bottleneck, or more decode nodes to\n increase token throughput. The challenge is finding the right ratio between\n prefill and decode capacity for a given workload, as over-provisioning either\n side increases cost without proportional benefit.\n\n- **Prefill bottleneck is a hard constraint**: With a fixed cluster size,\n dedicating nodes to prefill reduces decode capacity and vice versa. PD 1P3D\n suffers severe prefill saturation at long input lengths (TTFT > 37s at 4096\n tokens), while PD 2P2D has fewer decode nodes, limiting decode throughput.\n Frameworks such as `NVIDIA Dynamo `_\n aim to address this by dynamically scaling prefill and decode pools based on\n real-time demand, though this adds operational complexity.\n\n- **Simple routing beats disaggregation on throughput**: Route 4 (pure routing,\n no DP, no disaggregation) consistently achieves the highest throughput across\n all configurations by eliminating cross-node synchronization entirely. It also\n achieves the lowest TTFT in prefill-dominated workloads, though PD 1P3D edges\n it out on TTFT in decode-dominated regimes where the fixed 512-token input is\n short enough to avoid prefill saturation. This is a surprisingly strong\n baseline — for workloads where ITL is not the primary concern, stateless\n load-balanced independent nodes outperform both data parallelism and\n disaggregated configurations.\n\n- **KV cache transfer is not free**: The NIXL transfer over EFA adds measurable\n latency to TTFT in disaggregated configurations. This overhead is amortized\n for longer decode sequences but is noticeable for short output lengths,\n making disaggregation less attractive for short-response workloads.\n\nIn summary, disaggregated prefill/decode aims to optimize both TTFT and ITL by\nisolating the two phases, but achieving these goals is not guaranteed. KV cache\ntransfer over the network introduces additional overhead that can negate the\nTTFT benefit, particularly at long input lengths where the transfer volume is\nlarge. While ITL improvements are consistently observed due to the elimination\nof prefill interference on decode nodes, the overall serving performance depends\nheavily on the prefill-to-decode ratio, workload characteristics, and network\nbandwidth. Teams considering this architecture should carefully profile their\ninput/output length distributions, latency SLAs, and throughput requirements\nbefore committing to the added complexity.\n\nReferences\n----------\n\n.. [1] NVIDIA, \"NIXL: NVIDIA Inference Xfer Library,\" GitHub, 2025.\n https://github.com/ai-dynamo/nixl\n\n.. [2] vLLM Project, \"vLLM: Easy, fast, and cheap LLM serving,\" GitHub, 2024.\n https://github.com/vllm-project/vllm\n\n.. [3] vLLM Project, \"vllm-router: Production-ready router for vLLM,\" GitHub, 2025.\n https://github.com/vllm-project/vllm-router\n"
},
{
"path": "docs/notes/appendix/index.rst",
"content": ".. meta::\n :description lang=en: Python appendix covering advanced topics including the walrus operator (PEP 572) and Python debugging with GDB\n :keywords: Python, Python3, walrus operator, PEP 572, GDB, debugging, advanced Python\n\nBlog\n----\n\nThis section explores advanced programming topics to help users build a deeper\nunderstanding of complex concepts and practical techniques. Programmers working\nin other languages, such as C/C++, often use Python as a versatile debugging\ntool. With debuggers like GDB, they may write Python scripts to parse memory\nregions, improve output readability, or automate troubleshooting tasks.\n\nMore advanced topics and examples can be found in the following link.\n\n.. toctree::\n :maxdepth: 1\n\n disaggregated-prefill-decode\n megatron-efa-monitoring\n nccl-gin\n python-walrus\n python-gdb\n"
},
{
"path": "docs/notes/appendix/megatron-efa-monitoring.rst",
"content": ".. meta::\n :description lang=en: Monitoring EFA network performance with NCCL GIN and Nsys during distributed LLM training on AWS\n :keywords: EFA, NCCL, GIN, Nsys, Megatron-LM, distributed training, network monitoring, AWS\n\nMonitoring EFA with NCCL GIN and Nsys\n======================================\n\n:Date: 2026-02-28\n\nAbstract\n--------\n\nDistributed training at scale requires deep visibility into network behavior to\nidentify bottlenecks and optimize communication patterns. When training large\nlanguage models with Megatron-LM on AWS infrastructure using the Elastic Fabric\nAdapter (EFA), understanding network performance becomes critical for achieving\noptimal throughput. This article demonstrates how to enable NCCL GPU-Initiated\nNetworking (GIN) in Megatron-LM using Megatron Bridge and leverage Nsys with\nEFA metrics to monitor network behavior during distributed training workloads.\nThe techniques presented here are based on best practices from AWS re:Invent\n2024 [1]_.\n\nIntroduction\n------------\n\n`Megatron-LM `_ is a widely adopted\nframework for training large transformer models using model parallelism,\npipeline parallelism, and data parallelism. When deployed on AWS instances with\nEFA, the network fabric provides high-bandwidth, low-latency communication\nessential for scaling to hundreds or thousands of GPUs. However, achieving peak\nperformance requires careful tuning and monitoring of the communication layer.\n\nNCCL GPU-Initiated Networking allows GPUs to initiate network operations\ndirectly without CPU involvement, reducing latency and enabling kernel fusion.\nNsys (NVIDIA Nsight Systems) provides comprehensive profiling of GPU kernels,\nCUDA API calls, and network operations. When combined with EFA metrics\ncollection (``--enable efa_metrics``), Nsys captures detailed network adapter\nstatistics including bandwidth utilization, packet counts, and error rates,\ncorrelated with GPU execution timelines. This enables practitioners to diagnose\nperformance issues and validate that the network is operating at expected\ncapacity.\n\n`Megatron Bridge `_\nsimplifies the configuration and deployment of Megatron-LM training jobs by\nproviding a high-level recipe-based interface. This eliminates the need to\nmanually construct complex command-line arguments and makes it easier to enable\nadvanced features like NCCL GIN and DeepEP for MoE models. Therefore, the\ntutorial in this article will use Megatron Bridge.\n\nPrerequisites\n-------------\n\nThis guide assumes the following environment:\n\n- AWS HyperPod or EC2 instances with EFA support (e.g., P5, P5e, P5en)\n- NCCL >= v2.29.3-1 with Device API support\n- aws-ofi-nccl plugin with GIN support\n- Megatron-LM with Megatron-Bridge\n\nWe have demonstrated how to use vLLM with NCCL GIN and DeepEP in a previous\narticle. If you are interested in building NCCL and aws-ofi-nccl from source,\nrefer to the `NCCL GIN article\n`_\nin this repository.\n\nBuilding the Megatron Container\n--------------------------------\n\nThe Megatron training environment is packaged as a Docker container and\nconverted to an Enroot squash file for deployment on Slurm clusters. The\ncontainer includes NCCL with Device API support, aws-ofi-nccl with GIN support,\nand Megatron-LM with Megatron Bridge.\n\nTo build the container and create the Enroot image:\n\n.. code-block:: bash\n\n cd src/megatron\n make build\n\nThis will create a ``megatron-lm+latest.sqsh`` file that can be used with the\nSlurm launcher scripts. For details on the container build process, refer to\nthe `Dockerfile\n`_\nand `enroot.sh\n`_\nscripts in the repository.\n\nEnabling NCCL GIN in Megatron Bridge\n-------------------------------------\n\nMegatron Bridge recipes provide a declarative way to configure training jobs.\nTo enable NCCL GIN for MoE models using DeepEP, the following environment\nvariables are set automatically by the ``srun.sh`` launcher script:\n\n.. code-block:: bash\n\n export DEEP_EP_BACKEND=nccl\n export NCCL_GIN_TYPE=2 # proxy-based GIN\n export LD_LIBRARY_PATH=/opt/amazon/ofi-nccl/lib:$LD_LIBRARY_PATH\n\n``NCCL_GIN_TYPE=2`` selects the proxy-based implementation, where a CPU thread\nmediates GPU-initiated transfers. This mode is currently supported on EFA,\nwhile GPU Direct Async Kernel-Initiated (DAKI) networking (``NCCL_GIN_TYPE=3``)\nis not yet available on AWS at the time of writing (February 2026).\n\nThe ``srun.sh`` script also configures additional EFA-specific settings for\noptimal performance:\n\n.. code-block:: bash\n\n export FI_PROVIDER=efa\n export FI_EFA_USE_DEVICE_RDMA=1\n export FI_EFA_FORK_SAFE=1\n export NCCL_NET_PLUGIN=/opt/amazon/ofi-nccl/lib/libnccl-net-ofi.so\n export NCCL_TUNER_PLUGIN=/opt/amazon/ofi-nccl/lib/libnccl-tuner-ofi.so\n export NCCL_BUFFSIZE=8388608\n export NCCL_P2P_NET_CHUNKSIZE=524288\n\nLaunching Megatron Training with DeepEP and NCCL GIN\n-----------------------------------------------------\n\nThe following example demonstrates how to launch a DeepSeek-V2-Lite pretraining\njob with DeepEP enabled for MoE token dispatching. The recipe configures the\nmodel to use expert parallelism across 64 ranks with NCCL GIN for low-latency\nall-to-all communication.\n\n.. code-block:: bash\n\n cd src/megatron\n\n # Allocate 2 nodes on Slurm\n salloc -N 2\n\n # Launch DeepSeek-V2-Lite with DeepEP and NCCL GIN\n ./srun.sh recipes/deepseek_v2_lite_pretrain.py \\\n hf_path=/fsx/models/deepseek-ai/DeepSeek-V2-Lite \\\n moe_token_dispatcher_type=deepep \\\n model.tensor_model_parallel_size=1 \\\n model.expert_model_parallel_size=64 \\\n model.sequence_parallel=false\n\nThe ``moe_token_dispatcher_type=deepep`` argument enables DeepEP as the MoE\ndispatcher backend. Under the hood, the recipe configures the following\nsettings:\n\n.. code-block:: python\n\n cfg.model.moe_token_dispatcher_type = \"flex\"\n cfg.model.moe_flex_dispatcher_backend = \"deepep\"\n cfg.model.moe_enable_deepep = True\n cfg.model.moe_shared_expert_overlap = False\n\nWhen the training job starts, verify that NCCL initializes with GIN enabled by\nchecking the logs for Device API initialization messages:\n\n.. code-block:: text\n\n [NCCL] Device API initialized\n [NCCL] GIN proxy mode enabled (type=2)\n [NCCL Backend] LOW LATENCY MODE: Rank 0 connecting to all ranks\n [NCCL Backend] Initialized global rank 0/64\n\nMonitoring EFA with Nsys and EFA Metrics\n-----------------------------------------\n\nNsys (NVIDIA Nsight Systems) provides comprehensive profiling of GPU kernels,\nCUDA API calls, and network operations. The ``--enable efa_metrics`` flag\ninstructs Nsys to collect EFA adapter statistics in real-time from the EFA\ndevice counters (e.g., rdmap113s0, rdmap114s0) at 10Hz sampling rate, including:\n\n- **TX/RX Bandwidth**: Transmit and receive throughput\n- **TX/RX Packets**: Packet counts sent and received\n- **Error Counters**: Link errors and dropped packets\n\nAdditionally, aws-ofi-nccl uses NVTX annotations to mark NCCL operations in the\ntimeline, allowing correlation between NCCL collective calls and EFA network\nactivity. These metrics are embedded in the Nsys timeline and correlated with\nGPU kernel execution and NCCL operations, making it easy to identify\ncommunication bottlenecks and validate network saturation.\n\nTo profile a Megatron training run with Nsys and capture EFA metrics:\n\n.. code-block:: bash\n\n cd src/megatron\n salloc -N 8\n\n ./srun.sh --nsys recipes/deepseek_v2_lite_pretrain.py \\\n hf_path=/fsx/models/deepseek-ai/DeepSeek-V2-Lite \\\n moe_token_dispatcher_type=deepep \\\n model.tensor_model_parallel_size=1 \\\n model.expert_model_parallel_size=64 \\\n model.sequence_parallel=false \\\n profiling.use_nsys_profiler=true \\\n profiling.profile_step_start=10 \\\n profiling.profile_step_end=15 \\\n profiling.profile_ranks=[0]\n\nThe ``--nsys`` flag enables Nsys profiling with the following configuration:\n\n.. code-block:: bash\n\n nsys profile \\\n -t cuda,nvtx \\\n -s none \\\n --cpuctxsw=none \\\n --capture-range=cudaProfilerApi \\\n --capture-range-end=stop \\\n --enable efa_metrics \\\n -o nsys-megatron/profile--rank.nsys-rep \\\n --force-overwrite=true\n\nThe ``--enable efa_metrics`` flag is the key parameter that enables EFA adapter\nmonitoring. Nsys will automatically detect all EFA devices (typically\n``rdmap182s0``, ``rdmap183s0``, etc.) and collect statistics at regular intervals\nthroughout the profiling session.\n\nAfter profiling completes, the ``.nsys-rep`` files can be downloaded and opened\nin Nsight Systems GUI for analysis. The EFA metrics appear as additional rows\nin the timeline view, showing bandwidth and packet rate correlated with GPU\nkernel execution and NCCL collective operations.\n\n.. image:: https://raw.githubusercontent.com/crazyguitar/pysheeet/master/docs/_static/appendix/deepep-nsys.png\n\nProfiling with Viztracer\n-------------------------\n\nFor Python-level profiling of the training loop, Megatron Bridge supports\nViztracer, a low-overhead tracing tool that captures function calls and timing\ninformation. This is useful for identifying CPU bottlenecks in data loading,\npreprocessing, or scheduler logic that may indirectly impact network\nperformance.\n\n.. code-block:: bash\n\n salloc -N 2\n ./srun.sh recipes/deepseek_v2_lite_pretrain.py \\\n hf_path=/fsx/models/deepseek-ai/DeepSeek-V2-Lite \\\n train.train_iters=100 \\\n profiling.use_viztracer=true \\\n profiling.profile_step_start=10 \\\n profiling.profile_step_end=15 \\\n profiling.profile_ranks=[0]\n\nThe resulting ``.json`` trace files can be visualized in the Viztracer web UI\nor Chrome's ``chrome://tracing`` interface. By enabling ``log_torch``, Viztracer\ncan capture additional PyTorch-level details such as NCCL stream and CUDA stream\noperations, providing visibility into the execution flow of collective\ncommunications and GPU kernels. However, to observe detailed EFA adapter\nstatistics (bandwidth, packet counts, error counters), Nsys with\n``--enable efa_metrics`` remains the required tool.\n\nConclusion\n----------\n\nNsys profiling with ``--enable efa_metrics`` now provides the capability to\nmonitor both EFA adapter behavior and NCCL operations simultaneously during\ndistributed training. This visibility is essential for diagnosing whether long\nNCCL operation times are caused by actual EFA transmission delays or other\nissues such as CPU bottlenecks, memory contention, or suboptimal NCCL\nconfiguration. By examining the correlated timeline of GPU kernels, NCCL\ncollectives, and EFA bandwidth utilization, practitioners can pinpoint the root\ncause of performance bottlenecks and validate that the network fabric is\noperating at expected capacity.\n\nIn this article, we demonstrated this monitoring approach using Megatron-LM\nwith NCCL GIN and DeepEP as an example. The recipe-based approach of Megatron\nBridge simplifies the deployment of complex training configurations, making it\neasier to adopt advanced features like DeepEP and NCCL GIN for large-scale MoE\nmodel training while maintaining full observability into network performance.\n\nFor complete examples and scripts, refer to the `megatron directory\n`_ in this\nrepository.\n\nReferences\n----------\n\n.. [1] `AWS re:Invent 2024 - CMP335: Drilling down into performance for distributed training `_\n"
},
{
"path": "docs/notes/appendix/nccl-gin.rst",
"content": "\n.. meta::\n :description lang=en: Enabling GPU-Initiated Networking for NCCL with DeepEP on AWS using EFA\n :keywords: NCCL, GIN, GPU-Initiated Networking, DeepEP, EFA, AWS, MoE, HyperPod\n\nGPU-Initiated Networking for NCCL on AWS\n========================================\n\n:Date: 2026-02-22\n\nAbstract\n--------\n\nGPU-Initiated Networking (GIN) has attracted significant attention as a key\nenabler for kernel fusion in large language model (LLM) training and inference.\nMixture-of-Experts (MoE) architectures, such as DeepSeek-V3 and Qwen3-30B,\nrequire efficient token dispatching and combining across MoE layers.\nConventionally, inter-GPU communication is initiated by the CPU through\ncollective libraries such as NCCL or Gloo, necessitating explicit GPU\nsynchronization barriers and additional ``cudaLaunchKernel`` calls that\nintroduce non-trivial overhead. GPU-Initiated Networking eliminates this\nCPU-mediated round-trip by allowing data exchange to occur directly within CUDA\nkernels, thereby enabling kernel fusion and efficient CUDA Graph capture for\naccelerating end-to-end LLM layer computation. This article demonstrates how to\nenable NCCL GIN with DeepEP on AWS HyperPod Slurm using the AWS Elastic Fabric\nAdapter (EFA).\n\nIntroduction\n------------\n\nPrior to 2026, adopting DeepEP as a Mixture-of-Experts dispatch and combine\nbackend on AWS presented a significant challenge. The DeepEP kernel was\noriginally built on top of InfiniBand with a customized NVSHMEM implementation,\na transport layer unavailable on AWS infrastructure. This incompatibility\neffectively prevented users from leveraging DeepEP on instances equipped with\nthe Elastic Fabric Adapter (EFA). Recent collaborative efforts by NVIDIA and\nAmazon Annapurna Labs have addressed this gap by introducing GPU-Initiated\nNetworking support in NCCL and the EFA provider, enabling DeepEP to operate\nover EFA without relying on InfiniBand (see `DeepEP PR #521\n`_ and `aws-ofi-nccl PR #1069\n`_). The following experiment\nbuilds upon these contributions to illustrate how to deploy DeepEP with NCCL\nGIN on AWS using EFA.\n\nBuild DeepEP\n------------\n\nBefore deploying DeepEP on AWS HyperPod Slurm, several components must be built\nfrom source. First, NCCL >= v2.29.3-1 is required, as this is the minimum\nversion that exposes the Device API needed for GPU-Initiated Networking. The\nbuild targets ``sm_90`` (NVIDIA H100) and ``sm_100`` (NVIDIA B200) compute\ncapabilities to ensure compatibility with current-generation GPU instances.\n\n.. code-block:: bash\n\n NCCL_VERSION=v2.29.3-1\n git clone -b ${NCCL_VERSION} https://github.com/NVIDIA/nccl.git /opt/nccl \\\n && cd /opt/nccl \\\n && make -j $(nproc) src.build CUDA_HOME=/usr/local/cuda \\\n NVCC_GENCODE=\"-gencode=arch=compute_90,code=sm_90 -gencode=arch=compute_100,code=sm_100\"\n\nOptionally, the NCCL Device API examples can be built to verify that\nGPU-initiated communication functions correctly in the target environment. In\naddition, the latest release of nccl-tests (v2.17.9) ships with a GIN-enabled\nmicrobenchmark for the ``alltoall`` collective, which is useful for validating\ninter-GPU bandwidth and latency before running full-scale MoE workloads (see\n`nccl-tests alltoall.cu `_).\n\n.. code-block:: bash\n\n ## Build NCCL Device API examples\n cd /opt/nccl/examples/06_device_api \\\n && make -j $(nproc) NCCL_HOME=/opt/nccl/build CUDA_HOME=/usr/local/cuda MPI=1 MPI_HOME=/opt/amazon/openmpi\n\n NCCL_TESTS_VERSION=v2.17.9\n git clone -b ${NCCL_TESTS_VERSION} https://github.com/NVIDIA/nccl-tests.git /opt/nccl-tests \\\n && cd /opt/nccl-tests \\\n && make -j $(nproc) \\\n MPI=1 \\\n MPI_HOME=/opt/amazon/openmpi/ \\\n CUDA_HOME=/usr/local/cuda \\\n NCCL_HOME=/opt/nccl/build \\\n NVCC_GENCODE=\"-gencode=arch=compute_90,code=sm_90 -gencode=arch=compute_100,code=sm_100\"\n\nTo test DeepEP on HyperPod Slurm, both DeepEP and aws-ofi-nccl must be pinned\nto specific commits that include the NCCL GIN transport path. The DeepEP fork\nby Aamir Shafi introduces an NCCL-based communication backend as an alternative\nto the original NVSHMEM/InfiniBand path, while the aws-ofi-nccl plugin provides\nthe libfabric-to-NCCL translation layer required for EFA. Note that the NCCL\nGIN implementation has since been merged into the aws-ofi-nccl main branch; the\ncommit hash is pinned here for reproducibility.\n\n.. code-block:: bash\n\n ## Install DeepEP with NCCL GIN backend (PR #521)\n unset NVSHMEM_DIR NVSHMEM_HOME \\\n && export ENABLE_NCCL=1 \\\n && export NCCL_DIR=/opt/nccl/build \\\n && export LD_LIBRARY_PATH=/opt/nccl/build/lib:$LD_LIBRARY_PATH \\\n && export LD_PRELOAD=/opt/nccl/build/lib/libnccl.so.2 \\\n && git clone -b nccl https://github.com/aamirshafi/DeepEP.git /opt/DeepEP \\\n && cd /opt/DeepEP \\\n && git checkout 6d29f34 \\\n && python3 setup.py build_ext --inplace \\\n && pip install --break-system-packages --no-build-isolation .\n\n AWS_OFI_NCCL_VERSION=5f4202f11db1585d878196db4430aeda0e834a0c\n git clone https://github.com/aws/aws-ofi-nccl.git /tmp/aws-ofi-nccl \\\n && cd /tmp/aws-ofi-nccl \\\n && git checkout ${AWS_OFI_NCCL_VERSION} \\\n && ./autogen.sh \\\n && ./configure --prefix=/opt/amazon/ofi-nccl \\\n --with-libfabric=/opt/amazon/efa \\\n --with-cuda=/usr/local/cuda \\\n && make -j$(nproc) \\\n && make install \\\n && rm -rf /tmp/aws-ofi-nccl\n\nFor a complete build with all necessary dependencies, refer to the\n`Dockerfile `_\nprovided in this repository.\n\nTest NCCL GIN\n-------------\n\nWith the Docker image (or Enroot squash file) prepared in the previous section,\nNCCL GIN functionality can be validated on a Slurm cluster. The following\nexamples demonstrate how to launch the NCCL Device API samples and nccl-tests\nbenchmarks. The corresponding Slurm wrapper scripts are available under the\n`gin `_ directory\nin this repository.\n\n.. code-block:: bash\n\n make docker && make save # build a docker image and import an Enroot squash file\n\n # 01_allreduce_lsa (single node only)\n salloc -N 1 ./run.enroot /opt/nccl/examples/06_device_api/01_allreduce_lsa/allreduce_lsa\n\n # 01_allreduce_lsa (multi-node) — requires MNNVL (e.g. P6e-GB200), does NOT work over RDMA/EFA\n salloc -N 2 ./run.enroot /opt/nccl/examples/06_device_api/01_allreduce_lsa/allreduce_lsa\n\n # 02_alltoall_gin (multi-node)\n salloc -N 2 ./run.enroot /opt/nccl/examples/06_device_api/02_alltoall_gin/alltoall_gin\n\n # 03_alltoall_hybrid (multi-node)\n salloc -N 2 ./run.enroot /opt/nccl/examples/06_device_api/03_alltoall_hybrid/alltoall_hybrid\n\nThe nccl-tests ``alltoall`` benchmark exposes two critical flags for selecting\nthe GIN transport mode and memory registration strategy:\n\nThe ``-D`` flag selects the device-side implementation for the ``alltoall``\ncollective:\n\n.. code-block:: text\n\n -D 0 — Host API (default)\n -D 1 — NVL simple (LSA/NVLink only)\n -D 2 — NVL optimized (LSA/NVLink only)\n -D 3 — GIN only (network)\n -D 4 — Hybrid (LSA intra-node + GIN inter-node)\n\nThe ``-R`` flag controls memory registration. Symmetric memory allocation\n(``NCCL_MEM_SHARED``) is required for any device-side implementation\n(``-D > 0``), as it maps GPU memory across all ranks to enable direct\nremote read and write over the network:\n\n.. code-block:: text\n\n -R 0 — no registration (default)\n -R 1 — register memory with ncclMemAlloc\n -R 2 — register memory with symmetric memory allocation (NCCL_MEM_SHARED)\n\nThe following examples launch the nccl-tests ``alltoall_perf`` benchmark in\nGIN-only mode (``-D 3``) and hybrid mode (``-D 4``), sweeping message sizes\nfrom 32 MB to 2048 MB. The ``--blocking 0`` flag enables non-blocking\ncollectives, which is representative of how MoE layers overlap communication\nwith computation in production workloads:\n\n.. code-block:: bash\n\n # alltoall_perf with GIN (-D 3)\n salloc -N 2 ./run.enroot /opt/nccl-tests/build/alltoall_perf \\\n -D 3 -R 2 -b 32M -e 2048M -f 2 -n 1000 -w 10 --blocking 0\n\n # alltoall_perf with Hybrid LSA+GIN (-D 4)\n salloc -N 2 ./run.enroot /opt/nccl-tests/build/alltoall_perf \\\n -D 4 -R 2 -b 32M -e 2048M -f 2 -n 1000 -w 10 --blocking 0\n\nServing MoE Models with vLLM and DeepEP over NCCL GIN\n-----------------------------------------------------\n\nWith NCCL GIN and EFA validated on AWS HyperPod Slurm, this section\ndemonstrates an end-to-end inference deployment using vLLM with DeepEP as the\nMoE all-to-all communication backend. DeepEP's low-latency dispatch and combine\nkernels, now operating over NCCL GIN rather than NVSHMEM, enable efficient\nexpert-parallel inference for large MoE models such as DeepSeek-V3.\n\nThe Slurm launch script ``run.sbatch`` is the same one used to launch a vLLM\nserver in the `vllm example directory\n`_. However,\nto direct the DeepEP backend to use NCCL GIN, the following environment\nvariables must be set at launch time:\n\n.. code-block:: bash\n\n DEEP_EP_BACKEND=nccl\n NCCL_GIN_TYPE=2 # proxy-based GIN\n\n``NCCL_GIN_TYPE=2`` selects the proxy-based GIN path, in which a CPU-side proxy\nthread mediates network transfers on behalf of the GPU. ``NCCL_GIN_TYPE=3``\nwould enable GPU Direct Async Kernel-Initiated (DAKI) networking, which\nbypasses the proxy entirely; however, DAKI is not yet supported on AWS with EFA\nat the time of writing.\n\nFor additional details on serving configurations and benchmarking, refer to\n`llm-serving.rst\n`_\nor the `vLLM README\n`_.\n\nThe following example launches a multi-node vLLM inference server for\nDeepSeek-V3-0324 with expert parallelism enabled and the DeepEP low-latency\nall-to-all backend:\n\n.. code-block:: bash\n\n IMAGE=\"${PWD}/src/gin/nccl+latest.tar.gz\"\n MODEL=\"/fsx/models/deepseek-ai/DeepSeek-V3-0324\"\n\n salloc -N 4 bash run.sbatch \"${MODEL}\" \\\n --image \"${IMAGE}\" \\\n --all2all-backend deepep_low_latency \\\n --tensor-parallel-size 8 \\\n --enable-expert-parallel \\\n --gpu-memory-utilization 0.8 \\\n --enforce-eager\n\nUpon successful launch, the vLLM server logs confirm that DeepEP is active as\nthe all-to-all backend and that NCCL GIN is being used for inter-GPU\ncommunication. The key indicators are the ``DeepEPLLAll2AllManager`` manager\nselection and the ``[NCCL Backend]`` initialization messages showing\ncommunicator setup, symmetric memory allocation, and window registration across\nall ranks:\n\n.. code-block:: bash\n\n ...\n INFO 02-22 19:06:49 [serve.py:100] Defaulting api_server_count to data_parallel_size (4).\n INFO 02-22 19:06:49 [utils.py:325]\n INFO 02-22 19:06:49 [utils.py:325] █ █ █▄ ▄█\n INFO 02-22 19:06:49 [utils.py:325] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.15.1\n INFO 02-22 19:06:49 [utils.py:325] █▄█▀ █ █ █ █ model /fsx/models/deepseek-ai/DeepSeek-V3-0324\n INFO 02-22 19:06:49 [utils.py:325] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀\n INFO 02-22 19:06:49 [utils.py:325]\n ...\n INFO 02-22 19:07:51 [cuda_communicator.py:124] Using DeepEPLLAll2AllManager all2all manager.\n ...\n [NCCL Backend] LOW LATENCY MODE: Rank 0 connecting to all 32 ranks\n [NCCL Backend] NCCL version: 2.29.3 (loaded library)\n [NCCL Backend] Initializing 2 communicator(s) (qps_per_rank=8) for rank 0/32\n [NCCL Backend] Rank 0 successfully initialized 2 communicator(s)\n [NCCL Backend] Rank 0 created 2 device communication(s) with 32 barrier sessions each\n [NCCL Backend] Initialized global rank 0/32 (comm rank 0/32)\n [NCCL Backend - Memory Alloc] Rank 0: Allocated ptr=0xf882000000, size=3816818816\n [NCCL Backend - Memory Register] Rank 0: Copying 2 NCCL windows to GPU\n [NCCL Backend - Memory Register] Rank 0: Successfully copied windows to GPU\n [NCCL Backend - Memory Register] Rank 0: Registered windows for ptr=0xf882000000, size=3816818816\n\nOnce the server is ready, inference requests can be issued via the\nOpenAI-compatible completions API:\n\n.. code-block:: bash\n\n curl -sf -X POST http://:8000/v1/completions \\\n -H 'Content-Type: application/json' \\\n -d '{\n \"model\": \"/fsx/models/deepseek-ai/DeepSeek-V3-0324\",\n \"prompt\": \"Hello\",\n \"max_tokens\": 10\n }'\n\n # output\n {\"id\":\"cmpl-b6e9530a07561f11\",\"object\":\"text_completion\" ... }\n\nConclusion\n----------\n\nThis article has demonstrated how to deploy vLLM with DeepEP and NCCL GIN on\nAWS HyperPod Slurm using the Elastic Fabric Adapter. As this integration is\nstill under active development, certain limitations remain at the time of\nwriting. For instance, although DeepEP's low-latency mode supports CUDA Graph\ncapture, enabling it by removing ``--enforce-eager`` currently results in a\nstartup failure in vLLM. Additionally, performance over EFA may not yet match\nthat of InfiniBand-based deployments, as further optimizations are ongoing.\n\nThis article is intended as an early reference for evaluating DeepEP with NCCL\nGIN on AWS. For production workloads, it is advisable to wait for official\nstable releases from NVIDIA and Amazon Annapurna Labs.\n"
},
{
"path": "docs/notes/appendix/python-gdb.rst",
"content": ".. meta::\n :description lang=en: Python interpreter in GNU Debugger (GDB)\n :keywords: Python, Python3, GDB\n\n==================================\nPython Interpreter in GNU Debugger\n==================================\n\n:Date: 2025-08-30\n\nAbstract\n--------\n\nThe GNU Debugger (GDB) is the most powerful debugging tool for developers to\ntroubleshoot errors in their code. However, it is hard for beginners to learn,\nand that is why many programmers prefer to insert ``print`` to examine runtime\nstatus. Fortunately, `GDB Text User Interface (TUI)`_ provides a way for\ndevelopers to review their source code and debug simultaneously. More\nexcitingly, In GDB 7, **Python Interpreter** was built into GDB. This feature\noffers more straightforward ways to customize GDB printers and commands through\nthe Python library. By discussing examples, this article tries to explore\nadvanced debugging techniques via Python to develop tool kits for GDB.\n\nIntroduction\n------------\n\nTroubleshooting software bugs is a big challenge for developers. While GDB\nprovides many “debug commands” to inspect programs’ runtime status, its\nnon-intuitive usages impede programmers to use it to solve problems. Indeed,\nmastering GDB is a long-term process. However, a quick start is not complicated;\nyou must unlearn what you have learned like Yoda. To better understand how to\nuse Python in GDB, this article will focus on discussing Python interpreter in\nGDB.\n\nDefine Commands\n---------------\n\nGDB supports customizing commands by using ``define``. It is useful to run a\nbatch of commands to troubleshoot at the same time. For example, a developer\ncan display the current frame information by defining a ``sf`` command.\n\n.. code-block:: bash\n\n # define in .gdbinit\n define sf\n where # find out where the program is\n info args # show arguments\n info locals # show local variables\n end\n\nHowever, writing a user-defined command may be inconvenient due to limited APIs.\nFortunately, by interacting with Python interpreter in GDB, developers can\nutilize Python libraries to establish their debugging tool kits readily. The\nfollowing sections show how to use Python to simplify debugging processes.\n\nDump Memory\n-----------\n\nInspecting a process’s memory information is an effective way to troubleshoot\nmemory issues. Developers can acquire memory contents by ``info proc mappings``\nand ``dump memory``. To simplify these steps, defining a customized command is\nuseful. However, the implementation is not straightforward by using pure GDB\nsyntax. Even though GDB supports conditions, processing output is not intuitive.\nTo solve this problem, using Python API in GDB would be helpful because Python\ncontains many useful operations for handling strings.\n\n.. code-block:: python\n\n # mem.py\n import gdb\n import time\n import re\n\n class DumpMemory(gdb.Command):\n \"\"\"Dump memory info into a file.\"\"\"\n\n def __init__(self):\n super().__init__(\"dm\", gdb.COMMAND_USER)\n\n def get_addr(self, p, tty):\n \"\"\"Get memory addresses.\"\"\"\n cmd = \"info proc mappings\"\n out = gdb.execute(cmd, tty, True)\n addrs = []\n for l in out.split(\"\\n\"):\n if re.match(f\".*{p}*\", l):\n s, e, *_ = l.split()\n addrs.append((s, e))\n return addrs\n\n def dump(self, addrs):\n \"\"\"Dump memory result.\"\"\"\n if not addrs:\n return\n\n for s, e in addrs:\n f = int(time.time() * 1000)\n gdb.execute(f\"dump memory {f}.bin {s} {e}\")\n\n def invoke(self, args, tty):\n try:\n # cat /proc/self/maps\n addrs = self.get_addr(args, tty)\n # dump memory\n self.dump(addrs)\n except Exception as e:\n print(\"Usage: dm [pattern]\")\n\n DumpMemory()\n\n\nRunning the ``dm`` command will invoke ``DumpMemory.invoke``. By sourcing\nor implementing Python scripts in *.gdbinit*, developers can utilize\nuser-defined commands to trace bugs when a program is running. For example, the\nfollowing steps show how to invoke ``DumpMemory`` in GDB.\n\n.. code-block:: bash\n\n (gdb) start\n ...\n (gdb) source mem.py # source commands\n (gdb) dm stack # dump stack to ${timestamp}.bin\n (gdb) shell ls # ls current dir\n 1577283091687.bin a.cpp a.out mem.py\n\nDump JSON\n---------\n\nParsing JSON is helpful when a developer is inspecting a JSON string in a\nrunning program. GDB can parse a ``std::string`` via ``gdb.parse_and_eval``\nand return it as a ``gdb.Value``. By processing ``gdb.Value``, developers can\npass a JSON string into Python ``json`` API and print it in a pretty format.\n\n.. code-block:: python\n\n # dj.py\n import gdb\n import re\n import json\n\n class DumpJson(gdb.Command):\n \"\"\"Dump std::string as a styled JSON.\"\"\"\n\n def __init__(self):\n super().__init__(\"dj\", gdb.COMMAND_USER)\n\n def get_json(self, args):\n \"\"\"Parse std::string to JSON string.\"\"\"\n ret = gdb.parse_and_eval(args)\n typ = str(ret.type)\n if re.match(\"^std::.*::string\", typ):\n return json.loads(str(ret))\n return None\n\n def invoke(self, args, tty):\n try:\n # string to json string\n s = self.get_json(args)\n # json string to object\n o = json.loads(s)\n print(json.dumps(o, indent=2))\n except Exception as e:\n print(f\"Parse json error! {args}\")\n\n DumpJson()\n\nThe command ``dj`` displays a more readable JSON format in GDB. This command\nhelps improve visual recognization when a JSON string large. Also, by using\nthis command, it can detect or monitor whether a ``std::string`` is JSON or\nnot.\n\n.. code-block:: bash\n\n (gdb) start\n (gdb) list\n 1 #include \n 2\n 3 int main(int argc, char *argv[])\n 4 {\n 5 std::string json = R\"({\"foo\": \"FOO\",\"bar\": \"BAR\"})\";\n 6 return 0;\n 7 }\n ...\n (gdb) ptype json\n type = std::string\n (gdb) p json\n $1 = \"{\\\"foo\\\": \\\"FOO\\\",\\\"bar\\\": \\\"BAR\\\"}\"\n (gdb) source dj.py\n (gdb) dj json\n {\n \"foo\": \"FOO\",\n \"bar\": \"BAR\"\n }\n\nHighlight Syntax\n----------------\n\nSyntax highlighting is useful for developers to trace source code or to\ntroubleshoot issues. By using `Pygments`_, applying color to the source is easy\nwithout defining ANSI escape code manually. The following example shows how to\napply color to the ``list`` command output.\n\n.. code-block:: python\n\n import gdb\n\n from pygments import highlight\n from pygments.lexers import CLexer\n from pygments.formatters import TerminalFormatter\n\n class PrettyList(gdb.Command):\n \"\"\"Print source code with color.\"\"\"\n\n def __init__(self):\n super().__init__(\"pl\", gdb.COMMAND_USER)\n self.lex = CLexer()\n self.fmt = TerminalFormatter()\n\n def invoke(self, args, tty):\n try:\n out = gdb.execute(f\"l {args}\", tty, True)\n print(highlight(out, self.lex, self.fmt))\n except Exception as e:\n print(e)\n\n PrettyList()\n\nTracepoints\n-----------\n\nAlthough a developer can insert ``printf``, ``std::cout``, or ``syslog`` to\ninspect functions, printing messages is not an effective way to debug when a\nproject is enormous. Developers may waste their time in building source code\nand may acquire little information. Even worse, the output may become too much\nto detect problems. In fact, inspecting functions or variables do not require\nto embed *print functions* in code. By writing a Python script with GDB API,\ndevelopers can customize watchpoints to trace issues dynamically at runtime.\nFor example, by implementing a ``gdb.Breakpoint`` and a ``gdb.Command``, it is\nuseful for developers to acquire essential information, such as parameters,\ncall stacks, or memory usage.\n\n.. code-block:: python\n\n # tp.py\n import gdb\n\n tp = {}\n\n class Tracepoint(gdb.Breakpoint):\n def __init__(self, *args):\n super().__init__(*args)\n self.silent = True\n self.count = 0\n\n def stop(self):\n self.count += 1\n frame = gdb.newest_frame()\n block = frame.block()\n sym_and_line = frame.find_sal()\n framename = frame.name()\n filename = sym_and_line.symtab.filename\n line = sym_and_line.line\n # show tracepoint info\n print(f\"{framename} @ {filename}:{line}\")\n # show args and vars\n for s in block:\n if not s.is_argument and not s.is_variable:\n continue\n typ = s.type\n val = s.value(frame)\n size = typ.sizeof\n name = s.name\n print(f\"\\t{name}({typ}: {val}) [{size}]\")\n # do not stop at tracepoint\n return False\n\n class SetTracepoint(gdb.Command):\n def __init__(self):\n super().__init__(\"tp\", gdb.COMMAND_USER)\n\n def invoke(self, args, tty):\n try:\n global tp\n tp[args] = Tracepoint(args)\n except Exception as e:\n print(e)\n\n def finish(event):\n for t, p in tp.items():\n c = p.count\n print(f\"Tracepoint '{t}' Count: {c}\")\n\n gdb.events.exited.connect(finish)\n SetTracepoint()\n\nInstead of inserting ``std::cout`` at the beginning of functions, using a\ntracepoint at a function's entry point provides useful information to inspect\narguments, variables, and stacks. For instance, by setting a tracepoint at\n``fib``, it is helpful to examine memory usage, stack, and the number of calls.\n\n.. code-block:: cpp\n\n int fib(int n)\n {\n if (n < 2) {\n return 1;\n }\n return fib(n-1) + fib(n-2);\n }\n\n int main(int argc, char *argv[])\n {\n fib(3);\n return 0;\n }\n\nThe following output shows the result of an inspection of the function ``fib``.\nIn this case, tracepoints display all information a developer needs, including\narguments' value, recursive flow, and variables' size. By using tracepoints,\ndevelopers can acquire more useful information comparing with ``std::cout``.\n\n.. code-block:: bash\n\n (gdb) source tp.py\n (gdb) tp main\n Breakpoint 1 at 0x647: file a.cpp, line 12.\n (gdb) tp fib\n Breakpoint 2 at 0x606: file a.cpp, line 3.\n (gdb) r\n Starting program: /root/a.out\n main @ a.cpp:12\n argc(int: 1) [4]\n argv(char **: 0x7fffffffe788) [8]\n fib @ a.cpp:3\n n(int: 3) [4]\n fib @ a.cpp:3\n n(int: 2) [4]\n fib @ a.cpp:3\n n(int: 1) [4]\n fib @ a.cpp:3\n n(int: 0) [4]\n fib @ a.cpp:3\n n(int: 1) [4]\n [Inferior 1 (process 5409) exited normally]\n Tracepoint 'main' Count: 1\n Tracepoint 'fib' Count: 5\n\nProfiling\n---------\n\nWithout inserting timestamps, profiling is still feasible through tracepoints.\nBy using a ``gdb.FinishBreakpoint`` after a ``gdb.Breakpoint``, GDB sets a\ntemporary breakpoint at the return address of a frame for developers to get\nthe current timestamp and to calculate the time difference. Note that profiling\nvia GDB is not precise. Other tools, such as `Linux perf`_ or `Valgrind`_,\nprovide more useful and accurate information to trace performance issues.\n\n.. code-block:: python\n\n import gdb\n import time\n\n class EndPoint(gdb.FinishBreakpoint):\n def __init__(self, breakpoint, *a, **kw):\n super().__init__(*a, **kw)\n self.silent = True\n self.breakpoint = breakpoint\n\n def stop(self):\n # normal finish\n end = time.time()\n start, out = self.breakpoint.stack.pop()\n diff = end - start\n print(out.strip())\n print(f\"\\tCost: {diff}\")\n return False\n\n class StartPoint(gdb.Breakpoint):\n def __init__(self, *a, **kw):\n super().__init__(*a, **kw)\n self.silent = True\n self.stack = []\n\n def stop(self):\n start = time.time()\n # start, end, diff\n frame = gdb.newest_frame()\n sym_and_line = frame.find_sal()\n func = frame.function().name\n filename = sym_and_line.symtab.filename\n line = sym_and_line.line\n block = frame.block()\n\n args = []\n for s in block:\n if not s.is_argument:\n continue\n name = s.name\n typ = s.type\n val = s.value(frame)\n args.append(f\"{name}: {val} [{typ}]\")\n\n # format\n out = \"\"\n out += f\"{func} @ {filename}:{line}\\n\"\n for a in args:\n out += f\"\\t{a}\\n\"\n\n # append current status to a breakpoint stack\n self.stack.append((start, out))\n EndPoint(self, internal=True)\n return False\n\n class Profile(gdb.Command):\n def __init__(self):\n super().__init__(\"prof\", gdb.COMMAND_USER)\n\n def invoke(self, args, tty):\n try:\n StartPoint(args)\n except Exception as e:\n print(e)\n\n Profile()\n\n\nThe following output shows the profiling result by setting a tracepoint at the\nfunction ``fib``. It is convenient to inspect the function's performance and\nstack at the same time.\n\n.. code-block:: bash\n\n (gdb) source prof.py\n (gdb) prof fib\n Breakpoint 1 at 0x606: file a.cpp, line 3.\n (gdb) r\n Starting program: /root/a.out\n fib(int) @ a.cpp:3\n n: 1 [int]\n Cost: 0.0007786750793457031\n fib(int) @ a.cpp:3\n n: 0 [int]\n Cost: 0.002572298049926758\n fib(int) @ a.cpp:3\n n: 2 [int]\n Cost: 0.008517265319824219\n fib(int) @ a.cpp:3\n n: 1 [int]\n Cost: 0.0014069080352783203\n fib(int) @ a.cpp:3\n n: 3 [int]\n Cost: 0.01870584487915039\n\nPretty Print\n------------\n\nAlthough ``set print pretty on`` in GDB offers a better format to inspect\nvariables, developers may require to parse variables' value for readability.\nTake the system call ``stat`` as an example. While it provides useful information\nto examine file attributes, the output values, such as the permission, may not\nbe readable for debugging. By implementing a user-defined pretty print,\ndevelopers can parse ``struct stat`` and output information in a readable format.\n\n.. code-block:: python\n\n import gdb\n import pwd\n import grp\n import stat\n import time\n\n from datetime import datetime\n\n\n class StatPrint:\n def __init__(self, val):\n self.val = val\n\n def get_filetype(self, st_mode):\n if stat.S_ISDIR(st_mode):\n return \"directory\"\n if stat.S_ISCHR(st_mode):\n return \"character device\"\n if stat.S_ISBLK(st_mode):\n return \"block device\"\n if stat.S_ISREG:\n return \"regular file\"\n if stat.S_ISFIFO(st_mode):\n return \"FIFO\"\n if stat.S_ISLNK(st_mode):\n return \"symbolic link\"\n if stat.S_ISSOCK(st_mode):\n return \"socket\"\n return \"unknown\"\n\n def get_access(self, st_mode):\n out = \"-\"\n info = (\"r\", \"w\", \"x\")\n perm = [\n (stat.S_IRUSR, stat.S_IWUSR, stat.S_IXUSR),\n (stat.S_IRGRP, stat.S_IRWXG, stat.S_IXGRP),\n (stat.S_IROTH, stat.S_IWOTH, stat.S_IXOTH),\n ]\n for pm in perm:\n for c, p in zip(pm, info):\n out += p if st_mode & c else \"-\"\n return out\n\n def get_time(self, st_time):\n tv_sec = int(st_time[\"tv_sec\"])\n return datetime.fromtimestamp(tv_sec).isoformat()\n\n def to_string(self):\n st = self.val\n st_ino = int(st[\"st_ino\"])\n st_mode = int(st[\"st_mode\"])\n st_uid = int(st[\"st_uid\"])\n st_gid = int(st[\"st_gid\"])\n st_size = int(st[\"st_size\"])\n st_blksize = int(st[\"st_blksize\"])\n st_blocks = int(st[\"st_blocks\"])\n st_atim = st[\"st_atim\"]\n st_mtim = st[\"st_mtim\"]\n st_ctim = st[\"st_ctim\"]\n\n out = \"{\\n\"\n out += f\"Size: {st_size}\\n\"\n out += f\"Blocks: {st_blocks}\\n\"\n out += f\"IO Block: {st_blksize}\\n\"\n out += f\"Inode: {st_ino}\\n\"\n out += f\"Access: {self.get_access(st_mode)}\\n\"\n out += f\"File Type: {self.get_filetype(st_mode)}\\n\"\n out += f\"Uid: ({st_uid}/{pwd.getpwuid(st_uid).pw_name})\\n\"\n out += f\"Gid: ({st_gid}/{grp.getgrgid(st_gid).gr_name})\\n\"\n out += f\"Access: {self.get_time(st_atim)}\\n\"\n out += f\"Modify: {self.get_time(st_mtim)}\\n\"\n out += f\"Change: {self.get_time(st_ctim)}\\n\"\n out += \"}\"\n return out\n\n p = gdb.printing.RegexpCollectionPrettyPrinter(\"sp\")\n p.add_printer(\"stat\", \"^stat$\", StatPrint)\n\n o = gdb.current_objfile()\n gdb.printing.register_pretty_printer(o, p)\n\nBy sourcing the previous Python script, the ``PrettyPrinter`` can recognize\n``struct stat`` and output a readable format for developers to inspect file\nattributes. Without inserting functions to parse and print ``struct stat``, it\nis a more convenient way to acquire a better output from Python API.\n\n.. code-block:: bash\n\n (gdb) list 15\n 10 struct stat st;\n 11\n 12 if ((rc = stat(\"./a.cpp\", &st)) < 0) {\n 13 perror(\"stat failed.\");\n 14 goto end;\n 15 }\n 16\n 17 rc = 0;\n 18 end:\n 19 return rc;\n (gdb) source st.py\n (gdb) b 17\n Breakpoint 1 at 0x762: file a.cpp, line 17.\n (gdb) r\n Starting program: /root/a.out\n\n Breakpoint 1, main (argc=1, argv=0x7fffffffe788) at a.cpp:17\n 17 rc = 0;\n (gdb) p st\n $1 = {\n Size: 298\n Blocks: 8\n IO Block: 4096\n Inode: 1322071\n Access: -rw-rw-r--\n File Type: regular file\n Uid: (0/root)\n Gid: (0/root)\n Access: 2019-12-28T15:53:17\n Modify: 2019-12-28T15:53:01\n Change: 2019-12-28T15:53:01\n }\n\nNote that developers can disable a user-defined pretty-print via the command\n``disable``. For example, the previous Python script registers a pretty printer\nunder the global pretty-printers. By calling ``disable pretty-print``, the\nprinter ``sp`` will be disabled.\n\n.. code-block:: bash\n\n (gdb) disable pretty-print global sp\n 1 printer disabled\n 1 of 2 printers enabled\n (gdb) i pretty-print\n global pretty-printers:\n builtin\n mpx_bound128\n sp [disabled]\n stat\n\nAdditionally, developers can exclude a printer in the current GDB debugging\nsession if it is no longer required. The following snippet shows how to delete\nthe ``sp`` printer through ``gdb.pretty_printers.remove``.\n\n.. code-block:: bash\n\n (gdb) python\n >import gdb\n >for p in gdb.pretty_printers:\n > if p.name == \"sp\":\n > gdb.pretty_printers.remove(p)\n >end\n (gdb) i pretty-print\n global pretty-printers:\n builtin\n mpx_bound128\n\nConclusion\n----------\n\nIntegrating Python interpreter into GDB offers many flexible ways to\ntroubleshoot issues. While many integrated development environments (IDEs) may\nembed GDB to debug visually, GDB allows developers to implement their commands\nand parse variables’ output at runtime. By using debugging scripts, developers\ncan monitor and record necessary information without modifying their code.\nHonestly, inserting or enabling debugging code blocks may change a program’s\nbehaviors, and developers should get rid of this bad habit. Also, when a problem\nis reproduced, GDB can attach that process and examine its status without stopping\nit. Obviously, debugging via GDB is inevitable if a challenging issue emerges.\nThanks to integrating Python into GDB, developing a script to troubleshoot becomes\nmore accessible that leads to developers establishing their debugging methods\ndiversely.\n\n\nReference\n---------\n\n1. `Extending GDB using Python`_\n2. `gcc/gcc/gdbhooks.py`_\n3. `gdbinit/Gdbinit`_\n4. `cyrus-and/gdb-dashboard`_\n5. `hugsy/gef`_\n6. `sharkdp/stack-inspector`_\n7. `gdb Debugging Full Example (Tutorial)`_\n\n.. _Pygments: https://pygments.org/\n.. _Extending GDB using Python: https://sourceware.org/gdb/onlinedocs/gdb/Python.html\n.. _gcc/gcc/gdbhooks.py: https://github.com/gcc-mirror/gcc/blob/master/gcc/gdbhooks.py\n.. _hugsy/gef: https://github.com/hugsy/gef\n.. _cyrus-and/gdb-dashboard: https://github.com/cyrus-and/gdb-dashboard\n.. _gdbinit/Gdbinit: https://github.com/gdbinit/Gdbinit\n.. _sharkdp/stack-inspector: https://github.com/sharkdp/stack-inspector\n.. _GDB Text User Interface (TUI): https://sourceware.org/gdb/onlinedocs/gdb/TUI.html\n.. _Linux perf: https://github.com/torvalds/linux/tree/master/tools/perf\n.. _Valgrind: https://valgrind.org/\n.. _gdb Debugging Full Example (Tutorial): http://www.brendangregg.com/blog/2016-08-09/gdb-example-ncurses.html\n"
},
{
"path": "docs/notes/appendix/python-walrus.rst",
"content": ".. meta::\n :description lang=en: Design philosophy of pep 572, the walrus operator\n :keywords: Python3, PEP 572, walrus operator\n\n\nPEP 572 and The Walrus Operator\n===============================\n\n:Date: 2025-08-30\n\nAbstract\n--------\n\n`PEP 572`_ is one of the most contentious proposals in Python3 history because\nassigning a value within an expression seems unnecessary. Also, it is ambiguous\nfor developers to distinguish the difference between **the walrus operator**\n(``:=``) and the equal operator (``=``). Even though sophisticated developers\ncan use \"``:=``\" smoothly, they may concern the readability of their code. To\nbetter understand the usage of \"``:=``,\" this article discusses its design\nphilosophy and what kind of problems it tries to solve.\n\n\nIntroduction\n------------\n\nFor C/C++ developer, assigning a function return to a variable is common due\nto error code style handling. Managing function errors includes two steps;\none is to check the return value; another is to check ``errno``. For example,\n\n.. code-block:: cpp\n\n #include \n #include \n #include \n #include \n\n int main(int argc, char *argv[]) {\n int rc = -1;\n\n // assign access return to rc and check its value\n if ((rc = access(\"hello_walrus\", R_OK)) == -1) {\n fprintf(stderr, \"%s\", strerror(errno));\n goto end;\n }\n rc = 0;\n end:\n return rc;\n }\n\nIn this case, ``access`` will assign its return value to the variable ``rc``\nfirst. Then, the program will compare the ``rc`` value with ``-1`` to check\nwhether the execution of ``access`` is successful or not. However, Python did\nnot allow assigning values to variables within an expression before 3.8. To fix\nthis problem, therefore, PEP 572 introduced the walrus operator for developers.\nThe following Python snippet is equal to the previous C example.\n\n.. code-block:: python\n\n >>> import os\n >>> from ctypes import *\n >>> libc = CDLL(\"libc.dylib\", use_errno=True)\n >>> access = libc.access\n >>> path = create_string_buffer(b\"hello_walrus\")\n >>> if (rc := access(path, os.R_OK)) == -1:\n ... errno = get_errno()\n ... print(os.strerror(errno), file=sys.stderr)\n ...\n No such file or directory\n\n\nWhy ``:=`` ?\n------------\n\nDevelopers may confuse the difference between \"``:=``\" and \"``=``.\" In fact, they\nserve the same purpose, assigning somethings to variables. Why Python introduced\n\"``:=``\" instead of using \"``=``\"? What is the benefit of using \"``:=``\"? One\nreason is to reinforce the visual recognition due to a common mistake made by\nC/C++ developers. For instance,\n\n.. code-block:: cpp\n\n int rc = access(\"hello_walrus\", R_OK);\n\n // rc is unintentionally assigned to -1\n if (rc = -1) {\n fprintf(stderr, \"%s\", strerror(errno));\n goto end;\n }\n\nRather than comparison, the variable, ``rc``, is mistakenly assigned to -1. To\nprevent this error, some people advocate using `Yoda conditions`_ within an\nexpression.\n\n.. code-block:: cpp\n\n int rc = access(\"hello_walrus\", R_OK);\n\n // -1 = rc will raise a compile error\n if (-1 == rc) {\n fprintf(stderr, \"%s\", strerror(errno));\n goto end;\n }\n\nHowever, Yoda style is not readable enough like Yoda speaks non-standardized\nEnglish. Also, unlike C/C++ can detect assigning error during the compile-time\nvia compiler options (e.g., -Wparentheses), it is difficult for Python interpreter\nto distinguish such mistakes throughout the runtime. Thus, the final result\nof PEP 572 was to use a new syntax as a solution to implement *assignment\nexpressions*.\n\nThe walrus operator was not the first solution for PEP 572. The original proposal\nused ``EXPR as NAME`` to assign values to variables. Unfortunately, there are\nsome rejected reasons in this solution and other solutions as well. After\nintense debates, the final decision was ``:=``.\n\nScopes\n------\n\nUnlike other expressions, which a variable is bound to a scope, an assignment\nexpression belongs to the current scope. The purpose of this design is to\nallow a compact way to write code.\n\n.. code-block:: python3\n\n >>> if not (env := os.environ.get(\"HOME\")):\n ... raise KeyError(\"env HOME does not find!\")\n ...\n >>> print(env)\n /root\n\nIn PEP 572, another benefit is to conveniently capture a \"witness\" for an\n``any()`` or an ``all()`` expression. Although capturing function inputs can\nassist an interactive debugger, the advantage is not so obvious, and examples\nlack readability. Therefore, this benefit does not discuss here. Note that\nother languages (e.g., C/C++ or Go) may bind an assignment to a scope. Take\nGolang as an example.\n\n.. code-block:: go\n\n package main\n\n import (\n \"fmt\"\n \"os\"\n )\n\n func main() {\n if env := os.Getenv(\"HOME\"); env == \"\" {\n panic(fmt.Sprintf(\"Home does not find\"))\n }\n fmt.Print(env) // <--- compile error: undefined: env\n }\n\nPitfalls\n--------\n\nAlthough an assigning expression allows writing compact code, there are many\npitfalls when a developer uses it in a list comprehension. A common ``SyntaxError``\nis to rebind iteration variables.\n\n.. code-block:: python3\n\n >>> [i := i+1 for i in range(5)] # invalid\n\nHowever, updating an iteration variable will reduce readability and introduce\nbugs. Even if Python 3.8 did not implement the walrus operator, a programmer\nshould avoid reusing iteration variables within a scope.\n\nAnother pitfall is Python prohibits using assignment expressions within a\ncomprehension under a class scope.\n\n.. code-block:: python3\n\n >>> class Example:\n ... [(j := i) for i in range(5)] # invalid\n ...\n\nThis limitation was from `bpo-3692`_. The interpreter's behavior is\nunpredictable when a class declaration contains a list comprehension. To avoid\nthis corner case, assigning expression is invalid under a class.\n\n.. code-block:: python3\n\n >>> class Foo:\n ... a = [1, 2, 3]\n ... b = [4, 5, 6]\n ... c = [i for i in zip(a, b)] # b is defined\n ...\n >>> class Bar:\n ... a = [1,2,3]\n ... b = [4,5,6]\n ... c = [x * y for x in a for y in b] # b is undefined\n ...\n Traceback (most recent call last):\n File \"\", line 1, in \n File \"\", line 4, in Bar\n File \"\", line 4, in \n NameError: name 'b' is not defined\n\nConclusion\n----------\n\nThe reason why the walrus operator (``:=``) is so controversial is that code\nreadability may decrease. In fact, in the discussion `mail thread `_,\nthe author of PEP 572, Christoph Groth, had considered using \"``=``\" to implement\ninline assignment like C/C++. Without judging \"``:=``\" is ugly, many developers\nargue that distinguishing the functionality between \"``:=``\" and \"``=``\" is\ndifficult because they serve the same purpose, but behaviors are not consistent.\nAlso, writing compact code is not persuasive enough because smaller is not\nalways better. However, in some cases, the walrus operator can enhance\nreadability (if you understand how to use ``:=``). For example,\n\n.. code-block:: python3\n\n buf = b\"\"\n while True:\n data = read(1024)\n if not data:\n break\n buf += data\n\nBy using ``:=``, the previous example can be simplified.\n\n.. code-block:: python3\n\n buf = b\"\"\n while (data := read(1024)):\n buf += data\n\n`Python document`_ and GitHub `issue-8122`_ provides many great examples about\nimproving code readability by \"``:=``\". However, using the walrus operator\nshould be careful. Some cases, such as ``foo(x := 3, cat='vector')``, may\nintroduce new bugs if developers are not aware of scopes. Although PEP 572\nmay be risky for developers to write buggy code, an in-depth understanding of\ndesign philosophy and useful examples will help us use it to write readable\ncode at the right time.\n\nReferences\n----------\n\n1. `PEP 572 - Assignment Expressions`_\n2. `What’s New In Python 3.8`_\n3. `PEP 572 and decision-making in Python`_\n4. `The PEP 572 endgame`_\n5. `Use assignment expression in stdlib (combined PR)`_\n6. `Improper scope in list comprehension, when used in class declaration`_\n\n.. _PEP 572: https://www.python.org/dev/peps/pep-0572/\n.. _PEP 572 - Assignment Expressions: https://www.python.org/dev/peps/pep-0572/\n.. _What’s New In Python 3.8: https://docs.python.org/3/whatsnew/3.8.html\n.. _PEP 572 and decision-making in Python: https://lwn.net/Articles/757713/\n.. _The PEP 572 endgame: https://lwn.net/Articles/759558/\n.. _Use assignment expression in stdlib (combined PR): https://github.com/python/cpython/pull/8122/files\n.. _improper scope in list comprehension, when used in class declaration: https://bugs.python.org/issue3692\n.. _Yoda conditions: https://en.wikipedia.org/wiki/Yoda_conditions\n.. _bpo-3692: https://bugs.python.org/issue3692\n.. _Python document: https://docs.python.org/3/whatsnew/3.8.html#assignment-expressions\n.. _issue-8122: https://github.com/python/cpython/pull/8122/files\n"
},
{
"path": "docs/notes/asyncio/index.rst",
"content": ".. meta::\n :description lang=en: Python asyncio tutorial covering coroutines, event loops, tasks, async/await syntax, networking, and asynchronous programming patterns\n :keywords: Python, Python3, asyncio, async, await, coroutine, event loop, asynchronous, concurrent, networking, TCP, UDP\n\nAsyncio\n=======\n\nPython's ``asyncio`` module provides infrastructure for writing single-threaded\nconcurrent code using coroutines, multiplexing I/O access over sockets and other\nresources, running network clients and servers, and other related primitives.\nUnlike threading, asyncio uses cooperative multitasking, where tasks voluntarily\nyield control to allow other tasks to run. This makes it ideal for I/O-bound\napplications like web servers, database clients, and network services where\nwaiting for external resources is the primary bottleneck.\n\nThis section covers asyncio from basic concepts to advanced patterns, including\nthe event loop, coroutines, tasks, synchronization primitives, and real-world\nexamples like TCP/UDP servers, HTTP clients, and connection pools.\n\n.. toctree::\n :maxdepth: 1\n\n python-asyncio-guide\n python-asyncio-basic\n python-asyncio-server\n python-asyncio-advanced\n"
},
{
"path": "docs/notes/asyncio/python-asyncio-advanced.rst",
"content": ".. meta::\n :description lang=en: Python asyncio advanced - synchronization, queues, subprocesses, debugging, patterns\n :keywords: Python, Python3, Asyncio, Synchronization, Queue, Semaphore, Lock, Subprocess, Debugging\n\n=================\nAsyncio Advanced\n=================\n\n:Source: `src/basic/asyncio_.py `_\n\n.. contents:: Table of Contents\n :backlinks: none\n\nIntroduction\n------------\n\nBeyond basic coroutines and networking, asyncio provides synchronization\nprimitives, queues, subprocess management, and debugging tools. This section\ncovers advanced patterns for building robust async applications, including\nproducer-consumer patterns, rate limiting, graceful shutdown, and integration\nwith synchronous code.\n\nLocks\n-----\n\n``asyncio.Lock`` prevents multiple coroutines from accessing a shared resource\nsimultaneously. Unlike threading locks, async locks must be used with ``await``\nand only work within the same event loop.\n\n.. code-block:: python\n\n import asyncio\n\n class SharedCounter:\n def __init__(self):\n self.value = 0\n self._lock = asyncio.Lock()\n\n async def increment(self):\n async with self._lock:\n current = self.value\n await asyncio.sleep(0.01) # Simulate work\n self.value = current + 1\n\n async def worker(counter, name, count):\n for _ in range(count):\n await counter.increment()\n print(f\"{name} done\")\n\n async def main():\n counter = SharedCounter()\n await asyncio.gather(\n worker(counter, \"A\", 100),\n worker(counter, \"B\", 100),\n worker(counter, \"C\", 100),\n )\n print(f\"Final value: {counter.value}\") # Should be 300\n\n asyncio.run(main())\n\nSemaphores for Rate Limiting\n----------------------------\n\n``asyncio.Semaphore`` limits the number of concurrent operations. This is\nessential for rate limiting API calls, limiting database connections, or\ncontrolling resource usage.\n\n.. code-block:: python\n\n import asyncio\n\n async def fetch(url, semaphore):\n async with semaphore:\n print(f\"Fetching {url}\")\n await asyncio.sleep(1) # Simulate network request\n return f\"Response from {url}\"\n\n async def main():\n # Limit to 3 concurrent requests\n semaphore = asyncio.Semaphore(3)\n\n urls = [f\"https://api.example.com/{i}\" for i in range(10)]\n tasks = [fetch(url, semaphore) for url in urls]\n results = await asyncio.gather(*tasks)\n\n for r in results:\n print(r)\n\n asyncio.run(main())\n\nEvents for Signaling\n--------------------\n\n``asyncio.Event`` allows coroutines to wait for a signal from another coroutine.\nThis is useful for coordinating startup, shutdown, or state changes between\nmultiple tasks.\n\n.. code-block:: python\n\n import asyncio\n\n async def waiter(event, name):\n print(f\"{name} waiting for event\")\n await event.wait()\n print(f\"{name} got the event!\")\n\n async def setter(event):\n print(\"Setting event in 2 seconds...\")\n await asyncio.sleep(2)\n event.set()\n print(\"Event set!\")\n\n async def main():\n event = asyncio.Event()\n\n await asyncio.gather(\n waiter(event, \"Task 1\"),\n waiter(event, \"Task 2\"),\n waiter(event, \"Task 3\"),\n setter(event),\n )\n\n asyncio.run(main())\n\nConditions for Complex Synchronization\n--------------------------------------\n\n``asyncio.Condition`` combines a lock with the ability to wait for a condition.\nThis is useful for producer-consumer patterns where consumers need to wait\nfor specific conditions.\n\n.. code-block:: python\n\n import asyncio\n\n class Buffer:\n def __init__(self, size):\n self.buffer = []\n self.size = size\n self.condition = asyncio.Condition()\n\n async def put(self, item):\n async with self.condition:\n while len(self.buffer) >= self.size:\n await self.condition.wait()\n self.buffer.append(item)\n self.condition.notify()\n\n async def get(self):\n async with self.condition:\n while not self.buffer:\n await self.condition.wait()\n item = self.buffer.pop(0)\n self.condition.notify()\n return item\n\n async def producer(buffer, name):\n for i in range(5):\n await buffer.put(f\"{name}-{i}\")\n print(f\"Produced: {name}-{i}\")\n await asyncio.sleep(0.1)\n\n async def consumer(buffer, name):\n for _ in range(5):\n item = await buffer.get()\n print(f\"{name} consumed: {item}\")\n await asyncio.sleep(0.2)\n\n async def main():\n buffer = Buffer(size=2)\n await asyncio.gather(\n producer(buffer, \"P1\"),\n consumer(buffer, \"C1\"),\n consumer(buffer, \"C2\"),\n )\n\n asyncio.run(main())\n\nQueues for Producer-Consumer\n----------------------------\n\n``asyncio.Queue`` is the preferred way to implement producer-consumer patterns.\nIt handles synchronization internally and provides blocking get/put operations\nwith optional timeouts.\n\n.. code-block:: python\n\n import asyncio\n\n async def producer(queue, name):\n for i in range(5):\n item = f\"{name}-item-{i}\"\n await queue.put(item)\n print(f\"Produced: {item}\")\n await asyncio.sleep(0.5)\n\n async def consumer(queue, name):\n while True:\n try:\n item = await asyncio.wait_for(queue.get(), timeout=2.0)\n print(f\"{name} consumed: {item}\")\n queue.task_done()\n await asyncio.sleep(0.1)\n except asyncio.TimeoutError:\n print(f\"{name} timed out, exiting\")\n break\n\n async def main():\n queue = asyncio.Queue(maxsize=3)\n\n producers = [\n asyncio.create_task(producer(queue, \"P1\")),\n asyncio.create_task(producer(queue, \"P2\")),\n ]\n consumers = [\n asyncio.create_task(consumer(queue, \"C1\")),\n asyncio.create_task(consumer(queue, \"C2\")),\n ]\n\n await asyncio.gather(*producers)\n await queue.join() # Wait for all items to be processed\n\n for c in consumers:\n c.cancel()\n\n asyncio.run(main())\n\nPriority Queue\n--------------\n\n``asyncio.PriorityQueue`` processes items by priority. Lower priority values\nare processed first. Items must be comparable or wrapped in tuples with\npriority as the first element.\n\n.. code-block:: python\n\n import asyncio\n\n async def producer(queue):\n items = [\n (3, \"low priority\"),\n (1, \"high priority\"),\n (2, \"medium priority\"),\n ]\n for priority, item in items:\n await queue.put((priority, item))\n print(f\"Added: {item} (priority {priority})\")\n\n async def consumer(queue):\n while not queue.empty():\n priority, item = await queue.get()\n print(f\"Processing: {item} (priority {priority})\")\n await asyncio.sleep(0.5)\n queue.task_done()\n\n async def main():\n queue = asyncio.PriorityQueue()\n await producer(queue)\n await consumer(queue)\n\n asyncio.run(main())\n\nRunning Subprocesses\n--------------------\n\nAsyncio can run and communicate with subprocesses asynchronously. This is\nuseful for running shell commands, external tools, or parallel processes\nwithout blocking the event loop.\n\n.. code-block:: python\n\n import asyncio\n\n async def run_command(cmd):\n proc = await asyncio.create_subprocess_shell(\n cmd,\n stdout=asyncio.subprocess.PIPE,\n stderr=asyncio.subprocess.PIPE\n )\n\n stdout, stderr = await proc.communicate()\n\n return {\n 'cmd': cmd,\n 'returncode': proc.returncode,\n 'stdout': stdout.decode().strip(),\n 'stderr': stderr.decode().strip()\n }\n\n async def main():\n commands = [\n \"echo 'Hello World'\",\n \"python --version\",\n \"date\",\n ]\n\n results = await asyncio.gather(*[run_command(c) for c in commands])\n\n for r in results:\n print(f\"Command: {r['cmd']}\")\n print(f\"Output: {r['stdout']}\")\n print()\n\n asyncio.run(main())\n\nSubprocess with Streaming Output\n--------------------------------\n\nFor long-running processes, you can stream output line by line instead of\nwaiting for the process to complete. This is useful for monitoring logs or\nprogress.\n\n.. code-block:: python\n\n import asyncio\n\n async def stream_subprocess(cmd):\n proc = await asyncio.create_subprocess_shell(\n cmd,\n stdout=asyncio.subprocess.PIPE,\n stderr=asyncio.subprocess.STDOUT\n )\n\n while True:\n line = await proc.stdout.readline()\n if not line:\n break\n print(f\"[{cmd[:20]}] {line.decode().strip()}\")\n\n await proc.wait()\n return proc.returncode\n\n async def main():\n # Run multiple commands and stream their output\n await asyncio.gather(\n stream_subprocess(\"for i in 1 2 3; do echo $i; sleep 1; done\"),\n stream_subprocess(\"for i in a b c; do echo $i; sleep 0.5; done\"),\n )\n\n asyncio.run(main())\n\nGraceful Shutdown\n-----------------\n\nProper shutdown handling ensures all tasks complete cleanly and resources\nare released. Use signal handlers to catch SIGINT/SIGTERM and cancel tasks\ngracefully.\n\n.. code-block:: python\n\n import asyncio\n import signal\n\n async def worker(name):\n try:\n while True:\n print(f\"{name} working...\")\n await asyncio.sleep(1)\n except asyncio.CancelledError:\n print(f\"{name} cancelled, cleaning up...\")\n await asyncio.sleep(0.5) # Cleanup time\n print(f\"{name} cleanup done\")\n raise\n\n async def main():\n loop = asyncio.get_event_loop()\n tasks = [\n asyncio.create_task(worker(\"Worker-1\")),\n asyncio.create_task(worker(\"Worker-2\")),\n ]\n\n def shutdown():\n print(\"\\nShutdown requested...\")\n for task in tasks:\n task.cancel()\n\n loop.add_signal_handler(signal.SIGINT, shutdown)\n loop.add_signal_handler(signal.SIGTERM, shutdown)\n\n try:\n await asyncio.gather(*tasks)\n except asyncio.CancelledError:\n print(\"All tasks cancelled\")\n\n asyncio.run(main())\n\nRunning Async Code in Threads\n-----------------------------\n\nWhen you need to run async code from synchronous code (e.g., in a callback\nor from another thread), use ``asyncio.run_coroutine_threadsafe()``.\n\n.. code-block:: python\n\n import asyncio\n import threading\n import time\n\n async def async_task(value):\n await asyncio.sleep(1)\n return value * 2\n\n def thread_function(loop):\n # Run async code from a different thread\n future = asyncio.run_coroutine_threadsafe(\n async_task(21), loop\n )\n result = future.result(timeout=5)\n print(f\"Thread got result: {result}\")\n\n async def main():\n loop = asyncio.get_event_loop()\n\n # Start a thread that will call async code\n thread = threading.Thread(target=thread_function, args=(loop,))\n thread.start()\n\n # Keep the event loop running\n await asyncio.sleep(2)\n thread.join()\n\n asyncio.run(main())\n\nDebugging Asyncio\n-----------------\n\nEnable debug mode to catch common mistakes like blocking calls, unawaited\ncoroutines, and slow callbacks. Debug mode adds overhead so use it only\nduring development.\n\n.. code-block:: python\n\n import asyncio\n import logging\n\n # Enable debug logging\n logging.basicConfig(level=logging.DEBUG)\n\n async def slow_callback():\n import time\n time.sleep(0.2) # This will trigger a warning in debug mode\n\n async def main():\n await slow_callback()\n\n # Method 1: Environment variable\n # PYTHONASYNCIODEBUG=1 python script.py\n\n # Method 2: asyncio.run with debug=True\n asyncio.run(main(), debug=True)\n\nCustom Event Loop\n-----------------\n\nYou can customize the event loop behavior by subclassing or patching. This\nis useful for debugging, profiling, or adding custom functionality.\n\n.. code-block:: python\n\n import asyncio\n\n class DebugEventLoop(asyncio.SelectorEventLoop):\n def _run_once(self):\n # Track number of scheduled callbacks\n num_ready = len(self._ready)\n num_scheduled = len(self._scheduled)\n if num_ready or num_scheduled:\n print(f\"Ready: {num_ready}, Scheduled: {num_scheduled}\")\n super()._run_once()\n\n async def task(n):\n await asyncio.sleep(n)\n print(f\"Task {n} done\")\n\n # Use custom event loop\n loop = DebugEventLoop()\n asyncio.set_event_loop(loop)\n\n try:\n loop.run_until_complete(asyncio.gather(\n task(0.1),\n task(0.2),\n task(0.3),\n ))\n finally:\n loop.close()\n\nTimeout Patterns\n----------------\n\nDifferent timeout patterns for various use cases: per-operation timeout,\noverall timeout, and timeout with fallback.\n\n.. code-block:: python\n\n import asyncio\n\n async def fetch(url, delay):\n await asyncio.sleep(delay)\n return f\"Response from {url}\"\n\n async def fetch_with_timeout(url, delay, timeout):\n \"\"\"Per-operation timeout.\"\"\"\n try:\n return await asyncio.wait_for(fetch(url, delay), timeout)\n except asyncio.TimeoutError:\n return f\"Timeout for {url}\"\n\n async def fetch_all_with_timeout(urls, timeout):\n \"\"\"Overall timeout for all operations.\"\"\"\n async def fetch_all():\n return await asyncio.gather(*[fetch(u, i) for i, u in enumerate(urls)])\n\n try:\n return await asyncio.wait_for(fetch_all(), timeout)\n except asyncio.TimeoutError:\n return [\"Overall timeout\"]\n\n async def fetch_with_fallback(url, delay, timeout, fallback):\n \"\"\"Timeout with fallback value.\"\"\"\n try:\n return await asyncio.wait_for(fetch(url, delay), timeout)\n except asyncio.TimeoutError:\n return fallback\n\n async def main():\n # Per-operation timeout\n result = await fetch_with_timeout(\"slow.com\", 5, 1)\n print(result)\n\n # Timeout with fallback\n result = await fetch_with_fallback(\"slow.com\", 5, 1, \"cached response\")\n print(result)\n\n asyncio.run(main())\n\nRetry Pattern\n-------------\n\nImplement retry logic for transient failures with exponential backoff.\nThis is essential for robust network clients.\n\n.. code-block:: python\n\n import asyncio\n import random\n\n class RetryError(Exception):\n pass\n\n async def unreliable_operation():\n \"\"\"Simulates an operation that fails randomly.\"\"\"\n if random.random() < 0.7:\n raise ConnectionError(\"Network error\")\n return \"Success!\"\n\n async def retry(coro_func, max_retries=3, base_delay=1.0):\n \"\"\"Retry with exponential backoff.\"\"\"\n last_exception = None\n\n for attempt in range(max_retries):\n try:\n return await coro_func()\n except Exception as e:\n last_exception = e\n if attempt < max_retries - 1:\n delay = base_delay * (2 ** attempt)\n jitter = random.uniform(0, 0.1 * delay)\n print(f\"Attempt {attempt + 1} failed, retrying in {delay:.2f}s\")\n await asyncio.sleep(delay + jitter)\n\n raise RetryError(f\"Failed after {max_retries} attempts\") from last_exception\n\n async def main():\n try:\n result = await retry(unreliable_operation, max_retries=5)\n print(f\"Result: {result}\")\n except RetryError as e:\n print(f\"All retries failed: {e}\")\n\n asyncio.run(main())\n\nAsync Context Variable\n----------------------\n\nContext variables (Python 3.7+) provide task-local storage, similar to\nthread-local storage but for async tasks. Useful for request IDs, user\ncontext, or database connections.\n\n.. code-block:: python\n\n import asyncio\n import contextvars\n\n # Create context variable\n request_id = contextvars.ContextVar('request_id', default=None)\n\n async def process_request(rid):\n request_id.set(rid)\n await step1()\n await step2()\n\n async def step1():\n rid = request_id.get()\n print(f\"[{rid}] Step 1\")\n await asyncio.sleep(0.1)\n\n async def step2():\n rid = request_id.get()\n print(f\"[{rid}] Step 2\")\n await asyncio.sleep(0.1)\n\n async def main():\n await asyncio.gather(\n process_request(\"req-001\"),\n process_request(\"req-002\"),\n process_request(\"req-003\"),\n )\n\n asyncio.run(main())\n\nTaskGroup (Python 3.11+)\n------------------------\n\n``TaskGroup`` provides structured concurrency, ensuring all tasks complete\nor are cancelled together. Exceptions in any task cancel all other tasks\nin the group.\n\n.. code-block:: python\n\n import asyncio\n\n async def task(name, delay, should_fail=False):\n await asyncio.sleep(delay)\n if should_fail:\n raise ValueError(f\"{name} failed!\")\n return f\"{name} done\"\n\n async def main():\n try:\n async with asyncio.TaskGroup() as tg:\n tg.create_task(task(\"A\", 1))\n tg.create_task(task(\"B\", 2))\n tg.create_task(task(\"C\", 0.5, should_fail=True))\n except* ValueError as eg:\n for exc in eg.exceptions:\n print(f\"Caught: {exc}\")\n\n # Python 3.11+\n asyncio.run(main())\n"
},
{
"path": "docs/notes/asyncio/python-asyncio-basic.rst",
"content": ".. meta::\n :description lang=en: Python asyncio basics - coroutines, tasks, event loop, async/await syntax\n :keywords: Python, Python3, Asyncio, Coroutines, Event Loop, async await, Asynchronous Programming\n\n================\nAsyncio Basics\n================\n\n:Source: `src/basic/asyncio_.py `_\n\n.. contents:: Table of Contents\n :backlinks: none\n\nIntroduction\n------------\n\nThe ``asyncio`` module, introduced in Python 3.4 and significantly improved in\nPython 3.5+ with ``async/await`` syntax, provides a foundation for writing\nasynchronous code. Unlike threads which use preemptive multitasking (the OS\ndecides when to switch), asyncio uses cooperative multitasking where coroutines\nexplicitly yield control using ``await``. This eliminates race conditions common\nin threaded code and makes reasoning about program flow much easier.\n\nKey concepts:\n\n- **Coroutine**: A function defined with ``async def`` that can be paused and resumed\n- **Event Loop**: The central scheduler that runs coroutines and handles I/O events\n- **Task**: A wrapper around a coroutine that schedules it for execution\n- **Future**: A placeholder for a result that will be available later\n\nRunning Coroutines with asyncio.run\n-----------------------------------\n\nThe simplest way to run async code is ``asyncio.run()``, introduced in Python 3.7.\nIt creates an event loop, runs the coroutine until completion, and cleans up\nautomatically. This is the recommended entry point for asyncio programs.\n\n.. code-block:: python\n\n import asyncio\n\n async def hello():\n print(\"Hello\")\n await asyncio.sleep(1)\n print(\"World\")\n\n # Python 3.7+\n asyncio.run(hello())\n\nFor file I/O or other blocking operations, use ``run_in_executor`` to avoid\nblocking the event loop:\n\n.. code-block:: python\n\n import asyncio\n from concurrent.futures import ThreadPoolExecutor\n\n async def read_file(path):\n loop = asyncio.get_event_loop()\n with ThreadPoolExecutor() as pool:\n with open(path) as f:\n return await loop.run_in_executor(pool, f.read)\n\n content = asyncio.run(read_file('/etc/hosts'))\n\nCreating and Managing Tasks\n---------------------------\n\nTasks allow multiple coroutines to run concurrently. When you create a task,\nit's scheduled to run on the event loop immediately. Use ``asyncio.create_task()``\n(Python 3.7+) or ``loop.create_task()`` to create tasks.\n\n.. code-block:: python\n\n import asyncio\n\n async def fetch(name, delay):\n await asyncio.sleep(delay)\n return f\"{name} done\"\n\n async def main():\n # Create tasks - they start running immediately\n task1 = asyncio.create_task(fetch(\"A\", 2))\n task2 = asyncio.create_task(fetch(\"B\", 1))\n\n # Wait for both to complete\n result1 = await task1\n result2 = await task2\n print(result1, result2)\n\n asyncio.run(main())\n\nGathering Multiple Coroutines\n-----------------------------\n\n``asyncio.gather()`` runs multiple coroutines concurrently and collects their\nresults in order. This is the most common way to run multiple async operations\nin parallel and wait for all of them to complete.\n\n.. code-block:: python\n\n import asyncio\n\n async def fetch(url, delay):\n await asyncio.sleep(delay)\n return f\"Response from {url}\"\n\n async def main():\n urls = [\"site1.com\", \"site2.com\", \"site3.com\"]\n coros = [fetch(url, i * 0.5) for i, url in enumerate(urls)]\n\n # Run all concurrently, results in same order as input\n results = await asyncio.gather(*coros)\n for r in results:\n print(r)\n\n asyncio.run(main())\n\nWaiting with Timeout\n--------------------\n\nUse ``asyncio.wait_for()`` to set a timeout on async operations. This is\nessential for network operations where you don't want to wait indefinitely\nfor a response that may never come.\n\n.. code-block:: python\n\n import asyncio\n\n async def slow_operation():\n await asyncio.sleep(10)\n return \"done\"\n\n async def main():\n try:\n result = await asyncio.wait_for(slow_operation(), timeout=2.0)\n except asyncio.TimeoutError:\n print(\"Operation timed out!\")\n\n asyncio.run(main())\n\nWaiting for First Completed\n---------------------------\n\n``asyncio.wait()`` provides more control than ``gather()``. You can wait for\nthe first task to complete, first exception, or all tasks. This is useful\nwhen you want to process results as they become available.\n\n.. code-block:: python\n\n import asyncio\n\n async def fetch(name, delay):\n await asyncio.sleep(delay)\n return f\"{name}: {delay}s\"\n\n async def main():\n tasks = [\n asyncio.create_task(fetch(\"fast\", 1)),\n asyncio.create_task(fetch(\"slow\", 3)),\n ]\n\n # Wait for first to complete\n done, pending = await asyncio.wait(\n tasks, return_when=asyncio.FIRST_COMPLETED\n )\n\n for task in done:\n print(f\"Completed: {task.result()}\")\n print(f\"Still pending: {len(pending)}\")\n\n # Cancel pending tasks\n for task in pending:\n task.cancel()\n\n asyncio.run(main())\n\nAsynchronous Iteration\n----------------------\n\nAsync iterators allow you to iterate over data that arrives asynchronously,\nsuch as streaming responses or database cursors. Implement ``__aiter__`` and\n``__anext__`` methods to create custom async iterators.\n\n.. code-block:: python\n\n import asyncio\n\n class AsyncRange:\n \"\"\"Async iterator that yields numbers with delays.\"\"\"\n\n def __init__(self, start, stop):\n self.current = start\n self.stop = stop\n\n def __aiter__(self):\n return self\n\n async def __anext__(self):\n if self.current >= self.stop:\n raise StopAsyncIteration\n await asyncio.sleep(0.5)\n value = self.current\n self.current += 1\n return value\n\n async def main():\n async for num in AsyncRange(0, 5):\n print(num)\n\n asyncio.run(main())\n\nAsynchronous Context Managers\n-----------------------------\n\nAsync context managers are essential for managing resources that require\nasync setup or cleanup, such as database connections, file handles, or\nnetwork sessions. Use ``async with`` to ensure proper resource management.\n\n.. code-block:: python\n\n import asyncio\n\n class AsyncConnection:\n \"\"\"Simulated async database connection.\"\"\"\n\n async def __aenter__(self):\n print(\"Connecting...\")\n await asyncio.sleep(1)\n print(\"Connected\")\n return self\n\n async def __aexit__(self, exc_type, exc_val, exc_tb):\n print(\"Disconnecting...\")\n await asyncio.sleep(0.5)\n print(\"Disconnected\")\n\n async def query(self, sql):\n await asyncio.sleep(0.1)\n return f\"Result of: {sql}\"\n\n async def main():\n async with AsyncConnection() as conn:\n result = await conn.query(\"SELECT * FROM users\")\n print(result)\n\n asyncio.run(main())\n\nUsing @asynccontextmanager\n--------------------------\n\nThe ``@asynccontextmanager`` decorator (Python 3.7+) provides a simpler way\nto create async context managers using generator syntax, similar to the\nsynchronous ``@contextmanager`` decorator.\n\n.. code-block:: python\n\n import asyncio\n from contextlib import asynccontextmanager\n\n @asynccontextmanager\n async def managed_resource(name):\n print(f\"Acquiring {name}\")\n await asyncio.sleep(0.5)\n try:\n yield name\n finally:\n print(f\"Releasing {name}\")\n await asyncio.sleep(0.2)\n\n async def main():\n async with managed_resource(\"database\") as resource:\n print(f\"Using {resource}\")\n\n asyncio.run(main())\n\nRunning Blocking Code in Executor\n---------------------------------\n\nWhen you need to call blocking code (file I/O, CPU-intensive operations,\nor libraries without async support), use ``run_in_executor()`` to run it\nin a thread pool without blocking the event loop.\n\n.. code-block:: python\n\n import asyncio\n import time\n from concurrent.futures import ThreadPoolExecutor\n\n def blocking_io():\n \"\"\"Simulates blocking I/O operation.\"\"\"\n time.sleep(2)\n return \"IO complete\"\n\n def cpu_bound():\n \"\"\"Simulates CPU-intensive operation.\"\"\"\n return sum(i * i for i in range(10**6))\n\n async def main():\n loop = asyncio.get_event_loop()\n\n # Run in default executor (ThreadPoolExecutor)\n result1 = await loop.run_in_executor(None, blocking_io)\n print(result1)\n\n # Run in custom executor\n with ThreadPoolExecutor(max_workers=4) as pool:\n result2 = await loop.run_in_executor(pool, cpu_bound)\n print(result2)\n\n asyncio.run(main())\n\nAsync Generators\n----------------\n\nAsync generators (Python 3.6+) combine generators with async/await, allowing\nyou to yield values asynchronously. They're useful for streaming data or\nimplementing async iterators more concisely.\n\n.. code-block:: python\n\n import asyncio\n\n async def async_range(start, stop):\n \"\"\"Async generator that yields numbers with delays.\"\"\"\n for i in range(start, stop):\n await asyncio.sleep(0.5)\n yield i\n\n async def main():\n async for num in async_range(0, 5):\n print(num)\n\n # Async comprehension\n results = [x async for x in async_range(0, 3)]\n print(results)\n\n asyncio.run(main())\n\nException Handling in Tasks\n---------------------------\n\nExceptions in tasks are stored and re-raised when you await the task or\ncall ``result()``. Unhandled exceptions in tasks that are never awaited\nwill be logged but may be silently ignored, so always await your tasks.\n\n.. code-block:: python\n\n import asyncio\n\n async def failing_task():\n await asyncio.sleep(1)\n raise ValueError(\"Something went wrong\")\n\n async def main():\n task = asyncio.create_task(failing_task())\n\n try:\n await task\n except ValueError as e:\n print(f\"Caught exception: {e}\")\n\n # Using gather with return_exceptions\n tasks = [\n asyncio.create_task(asyncio.sleep(1)),\n asyncio.create_task(failing_task()),\n ]\n results = await asyncio.gather(*tasks, return_exceptions=True)\n for r in results:\n if isinstance(r, Exception):\n print(f\"Task failed: {r}\")\n else:\n print(f\"Task succeeded: {r}\")\n\n asyncio.run(main())\n\nCancelling Tasks\n----------------\n\nTasks can be cancelled using ``task.cancel()``. The cancelled task will\nraise ``asyncio.CancelledError`` at the next await point. Handle this\nexception to perform cleanup when a task is cancelled.\n\n.. code-block:: python\n\n import asyncio\n\n async def long_running():\n try:\n while True:\n print(\"Working...\")\n await asyncio.sleep(1)\n except asyncio.CancelledError:\n print(\"Task was cancelled, cleaning up...\")\n raise # Re-raise to mark task as cancelled\n\n async def main():\n task = asyncio.create_task(long_running())\n\n await asyncio.sleep(3)\n task.cancel()\n\n try:\n await task\n except asyncio.CancelledError:\n print(\"Task cancellation confirmed\")\n\n asyncio.run(main())\n"
},
{
"path": "docs/notes/asyncio/python-asyncio-guide.rst",
"content": ".. meta::\n :description lang=en: A comprehensive guide to understanding asynchronous programming in Python, from blocking I/O to event loops, callbacks, generators, and async/await syntax\n :keywords: Python, Python3, asyncio, coroutine, event loop, async await, asynchronous programming, C10k problem, non-blocking I/O, selectors, generators, callback\n\n================================================\nA Hitchhiker's Guide to Asynchronous Programming\n================================================\n\n.. contents:: Table of Contents\n :backlinks: none\n\nAbstract\n--------\n\nThe `C10k problem`_ remains a fundamental challenge for programmers seeking to\nhandle massive concurrent connections efficiently. Traditionally, developers\naddress extensive I/O operations using **threads**, **epoll**, or **kqueue** to\nprevent software from blocking on expensive operations. However, developing\nreadable and bug-free concurrent code is challenging due to complexities around\ndata sharing and task dependencies. Even powerful tools like `Valgrind`_ that\nhelp detect deadlocks and race conditions cannot eliminate the time-consuming\ndebugging process as software scales.\n\nTo address these challenges, many programming languages—including Python,\nJavaScript, and C++—have developed better libraries, frameworks, and syntaxes\nto help programmers manage concurrent tasks properly. Rather than focusing on\nhow to use modern parallel APIs, this article concentrates on the **design\nphilosophy** behind asynchronous programming patterns, tracing the evolution\nfrom blocking I/O to the elegant ``async/await`` syntax.\n\nUsing threads is the most natural approach for dispatching tasks without\nblocking the main thread. However, threads introduce performance overhead from\ncontext switching and require careful locking of critical sections for atomic\noperations. While event loops can enhance performance in I/O-bound scenarios,\nwriting readable event-driven code is challenging due to callback complexity\n(commonly known as \"callback hell\"). Fortunately, Python introduced the\n``async/await`` syntax to help developers write understandable code with high\nperformance. The following figure illustrates how ``async/await`` enables\nhandling socket connections with the simplicity of threads but the efficiency\nof event loops.\n\n.. image:: https://raw.githubusercontent.com/crazyguitar/pysheeet/master/docs/_static/appendix/event-loop-vs-thread.png\n\nIntroduction\n------------\n\nHandling I/O operations such as network connections is among the most expensive\ntasks in any program. Consider a simple TCP blocking echo server (shown below).\nIf a client connects without sending any data, it blocks all other connections.\nEven when clients send data promptly, the server cannot handle concurrent\nrequests because it wastes significant time waiting for I/O responses from\nhardware like network interfaces. Thus, socket programming with concurrency\nbecomes essential for managing high request volumes.\n\n.. code-block:: python\n\n import socket\n\n s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)\n s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n s.bind((\"127.0.0.1\", 5566))\n s.listen(10)\n\n while True:\n conn, addr = s.accept()\n msg = conn.recv(1024)\n conn.send(msg)\n\nOne solution to prevent blocking is dispatching tasks to separate threads. The\nfollowing example demonstrates handling connections simultaneously using threads.\nHowever, creating numerous threads consumes computing resources without\nproportional throughput gains. Worse, applications may waste time waiting for\nlocks when processing tasks in critical sections. While threads solve blocking\nissues, factors like CPU utilization and memory overhead remain critical for\nsolving the C10k problem. Without creating unlimited threads, the **event loop**\nprovides an alternative solution for managing connections efficiently.\n\n.. code-block:: python\n\n import threading\n import socket\n\n s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n s.bind((\"127.0.0.1\", 5566))\n s.listen(10240)\n\n def handler(conn):\n while True:\n msg = conn.recv(65535)\n conn.send(msg)\n\n while True:\n conn, addr = s.accept()\n t = threading.Thread(target=handler, args=(conn,))\n t.start()\n\nA simple event-driven socket server comprises three main components: an **I/O\nmultiplexing module** (e.g., `select`_), a **scheduler** (the loop), and\n**callback functions** (event handlers). The following server uses Python's\nhigh-level I/O multiplexing module, `selectors`_, within a loop to check\nwhether I/O operations are ready. When data becomes available for reading or\nwriting, the loop retrieves I/O events and executes the appropriate callback\nfunctions—``accept``, ``read``, or ``write``—to complete tasks.\n\n.. code-block:: python\n\n import socket\n from selectors import DefaultSelector, EVENT_READ, EVENT_WRITE\n from functools import partial\n\n s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n s.bind((\"127.0.0.1\", 5566))\n s.listen(10240)\n s.setblocking(False)\n\n sel = DefaultSelector()\n\n def accept(s, mask):\n conn, addr = s.accept()\n conn.setblocking(False)\n sel.register(conn, EVENT_READ, read)\n\n def read(conn, mask):\n msg = conn.recv(65535)\n if not msg:\n sel.unregister(conn)\n return conn.close()\n sel.modify(conn, EVENT_WRITE, partial(write, msg=msg))\n\n def write(conn, mask, msg=None):\n if msg:\n conn.send(msg)\n sel.modify(conn, EVENT_READ, read)\n\n sel.register(s, EVENT_READ, accept)\n while True:\n events = sel.select()\n for e, m in events:\n cb = e.data\n cb(e.fileobj, m)\n\nAlthough managing connections via threads may be inefficient, event-loop-based\nprograms are harder to read and maintain. To enhance code readability, many\nprogramming languages—including Python—introduce abstract concepts such as\n**coroutines**, **futures**, and **async/await** to handle I/O multiplexing\nelegantly. The following sections explore these concepts and the problems they\nsolve.\n\nCallback Functions\n------------------\n\nCallback functions control data flow at runtime when events occur. However,\npreserving state across callbacks is challenging. For example, implementing a\nhandshake protocol over TCP requires storing previous state somewhere accessible\nto subsequent callbacks.\n\n.. code-block:: python\n\n import socket\n from selectors import DefaultSelector, EVENT_READ, EVENT_WRITE\n from functools import partial\n\n s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n s.bind((\"127.0.0.1\", 5566))\n s.listen(10240)\n s.setblocking(False)\n\n sel = DefaultSelector()\n is_hello = {}\n\n def accept(s, mask):\n conn, addr = s.accept()\n conn.setblocking(False)\n is_hello[conn] = False\n sel.register(conn, EVENT_READ, read)\n\n def read(conn, mask):\n msg = conn.recv(65535)\n if not msg:\n sel.unregister(conn)\n return conn.close()\n\n # Check whether handshake is successful\n if is_hello[conn]:\n sel.modify(conn, EVENT_WRITE, partial(write, msg=msg))\n return\n\n # Perform handshake\n if msg.decode(\"utf-8\").strip() != \"hello\":\n sel.unregister(conn)\n return conn.close()\n\n is_hello[conn] = True\n\n def write(conn, mask, msg=None):\n if msg:\n conn.send(msg)\n sel.modify(conn, EVENT_READ, read)\n\n sel.register(s, EVENT_READ, accept)\n while True:\n events = sel.select()\n for e, m in events:\n cb = e.data\n cb(e.fileobj, m)\n\nAlthough the ``is_hello`` dictionary stores state to track handshake status,\nthe code becomes difficult to understand. The underlying logic is actually\nsimple—equivalent to this blocking version:\n\n.. code-block:: python\n\n def accept(s):\n conn, addr = s.accept()\n success = handshake(conn)\n if not success:\n conn.close()\n\n def handshake(conn):\n data = conn.recv(65535)\n if not data:\n return False\n if data.decode('utf-8').strip() != \"hello\":\n return False\n conn.send(b\"hello\")\n return True\n\nTo achieve similar structure in non-blocking code, a function (or task) must\nsnapshot its current state—including arguments, local variables, and execution\nposition—when waiting for I/O operations. The scheduler must then be able to\n**re-enter** the function and execute remaining code after I/O completes.\nUnlike languages like C++, Python achieves this naturally because **generators**\npreserve all state and can be re-entered by calling ``next()``. By utilizing\ngenerators, handling I/O operations in a non-blocking manner with readable,\nlinear code—called *inline callbacks*—becomes possible within an event loop.\n\nEvent Loop\n----------\n\nAn event loop is a user-space scheduler that manages tasks within a program\ninstead of relying on operating system thread scheduling. The following snippet\ndemonstrates a simple event loop handling socket connections asynchronously.\nThe implementation appends tasks to a FIFO job queue and registers with a\n*selector* when I/O operations are not ready. A *generator* preserves task\nstate, allowing execution to resume without callback functions when I/O results\nbecome available. Understanding how this event loop works reveals that a Python\ngenerator is indeed a form of **coroutine**.\n\n.. code-block:: python\n\n # loop.py\n\n from selectors import DefaultSelector, EVENT_READ, EVENT_WRITE\n\n class Loop:\n def __init__(self):\n self.sel = DefaultSelector()\n self.queue = []\n\n def create_task(self, task):\n self.queue.append(task)\n\n def polling(self):\n for e, m in self.sel.select(0):\n self.queue.append((e.data, None))\n self.sel.unregister(e.fileobj)\n\n def is_registered(self, fileobj):\n try:\n self.sel.get_key(fileobj)\n except KeyError:\n return False\n return True\n\n def register(self, t, data):\n if not data:\n return False\n\n event_type, fileobj = data\n if event_type in (EVENT_READ, EVENT_WRITE):\n if self.is_registered(fileobj):\n self.sel.modify(fileobj, event_type, t)\n else:\n self.sel.register(fileobj, event_type, t)\n return True\n return False\n\n def accept(self, s):\n while True:\n try:\n conn, addr = s.accept()\n except BlockingIOError:\n yield (EVENT_READ, s)\n else:\n break\n return conn, addr\n\n def recv(self, conn, size):\n while True:\n try:\n msg = conn.recv(size)\n except BlockingIOError:\n yield (EVENT_READ, conn)\n else:\n break\n return msg\n\n def send(self, conn, msg):\n while True:\n try:\n size = conn.send(msg)\n except BlockingIOError:\n yield (EVENT_WRITE, conn)\n else:\n break\n return size\n\n def once(self):\n self.polling()\n unfinished = []\n for t, data in self.queue:\n try:\n data = t.send(data)\n except StopIteration:\n continue\n\n if self.register(t, data):\n unfinished.append((t, None))\n\n self.queue = unfinished\n\n def run(self):\n while self.queue or self.sel.get_map():\n self.once()\n\nBy assigning jobs to an event loop, the programming pattern resembles using\nthreads but with a user-level scheduler. `PEP 380`_ introduced generator\ndelegation via ``yield from``, allowing a generator to wait for other generators\nto complete. The following snippet is far more intuitive and readable than\ncallback-based I/O handling:\n\n.. code-block:: python\n\n # server.py\n # $ python3 server.py &\n # $ nc localhost 5566\n\n import socket\n from loop import Loop\n\n s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n s.bind((\"127.0.0.1\", 5566))\n s.listen(10240)\n s.setblocking(False)\n\n loop = Loop()\n\n def handler(conn):\n while True:\n msg = yield from loop.recv(conn, 1024)\n if not msg:\n conn.close()\n break\n yield from loop.send(conn, msg)\n\n def main():\n while True:\n conn, addr = yield from loop.accept(s)\n conn.setblocking(False)\n loop.create_task((handler(conn), None))\n\n loop.create_task((main(), None))\n loop.run()\n\nUsing an event loop with ``yield from`` manages connections without blocking\nthe main thread—this was how ``asyncio`` worked before Python 3.5. However,\n``yield from`` is ambiguous: why does adding ``@asyncio.coroutine`` transform\na generator into a coroutine? Instead of overloading generator syntax for\nasynchronous operations, `PEP 492`_ proposed that coroutines should become a\n**standalone concept** in Python. This led to the introduction of ``async/await``\nsyntax, dramatically improving readability for asynchronous programming.\n\nWhat is a Coroutine?\n--------------------\n\nPython documentation defines coroutines as \"a generalized form of subroutines.\"\nThis definition, while technically accurate, can be confusing. Based on our\ndiscussion, an event loop schedules generators to perform specific tasks—similar\nto how an OS dispatches jobs to threads. In this context, generators serve as\n\"routine workers.\" A **coroutine** is simply a task scheduled by an event loop\nwithin a program, rather than by the operating system.\n\nThe following snippet illustrates what ``@coroutine`` does. This decorator\ntransforms a function into a generator function and wraps it with\n``types.coroutine`` for backward compatibility:\n\n.. code-block:: python\n\n import asyncio\n import inspect\n import types\n from functools import wraps\n from asyncio.futures import Future\n\n def coroutine(func):\n \"\"\"Simple prototype of coroutine decorator\"\"\"\n if inspect.isgeneratorfunction(func):\n return types.coroutine(func)\n\n @wraps(func)\n def coro(*a, **k):\n res = func(*a, **k)\n if isinstance(res, Future) or inspect.isgenerator(res):\n res = yield from res\n return res\n return types.coroutine(coro)\n\n @coroutine\n def foo():\n yield from asyncio.sleep(1)\n print(\"Hello Foo\")\n\n loop = asyncio.get_event_loop()\n loop.run_until_complete(loop.create_task(foo()))\n loop.close()\n\nWith Python 3.5+, the ``async def`` syntax creates native coroutines directly,\nand ``await`` replaces ``yield from`` for suspending execution. This makes the\nintent explicit: ``async def`` declares a coroutine, and ``await`` marks\nsuspension points where the event loop can switch to other tasks.\n\nConclusion\n----------\n\nAsynchronous programming via event loops has become more straightforward and\nreadable thanks to modern syntax and library support. Most programming\nlanguages, including Python, implement libraries that manage task scheduling\nthrough integration with new syntaxes. While ``async/await`` may seem enigmatic\ninitially, it provides a way for programmers to develop logical, linear code\nstructure—similar to using threads—while gaining the performance benefits of\nevent-driven I/O.\n\nWithout callback functions passing state between handlers, programmers no\nlonger need to worry about preserving local variables and arguments across\nasynchronous boundaries. This allows developers to focus on application logic\nrather than spending time troubleshooting concurrency issues. The evolution\nfrom callbacks to generators to ``async/await`` represents a significant\nadvancement in making concurrent programming accessible and maintainable.\n\nReferences\n----------\n\n1. `asyncio — Asynchronous I/O`_\n2. `PEP 342 - Coroutines via Enhanced Generators`_\n3. `PEP 380 - Syntax for Delegating to a Subgenerator`_\n4. `PEP 492 - Coroutines with async and await syntax`_\n\n.. _C10k problem: https://en.wikipedia.org/wiki/C10k_problem\n.. _Valgrind: https://valgrind.org/\n.. _select: https://docs.python.org/3/library/select.html\n.. _selectors: https://docs.python.org/3/library/selectors.html\n.. _asyncio — Asynchronous I/O: https://docs.python.org/3/library/asyncio.html\n.. _PEP 492: https://www.python.org/dev/peps/pep-0492/\n.. _PEP 380: https://www.python.org/dev/peps/pep-0380/\n.. _PEP 342 - Coroutines via Enhanced Generators: https://www.python.org/dev/peps/pep-0342/\n.. _PEP 492 - Coroutines with async and await syntax: https://www.python.org/dev/peps/pep-0492/\n.. _PEP 380 - Syntax for Delegating to a Subgenerator: https://www.python.org/dev/peps/pep-0380/\n"
},
{
"path": "docs/notes/asyncio/python-asyncio-server.rst",
"content": ".. meta::\n :description lang=en: Python asyncio networking - TCP/UDP servers, HTTP clients, SSL/TLS, protocols\n :keywords: Python, Python3, Asyncio, TCP Server, UDP Server, HTTP Client, SSL TLS, Network Programming\n\n===================\nAsyncio Networking\n===================\n\n:Source: `src/basic/asyncio_.py `_\n\n.. contents:: Table of Contents\n :backlinks: none\n\nIntroduction\n------------\n\nAsyncio excels at network programming because network I/O is inherently\nasynchronous - you send a request and wait for a response. Instead of blocking\na thread while waiting, asyncio allows other tasks to run. This section covers\nbuilding TCP/UDP servers and clients, HTTP requests, SSL/TLS encryption, and\nthe Transport/Protocol API for low-level control.\n\nTCP Echo Server with Streams\n----------------------------\n\nThe streams API (``asyncio.start_server``, ``open_connection``) provides a\nhigh-level interface for TCP networking. It handles buffering, encoding, and\nconnection management automatically, making it the recommended approach for\nmost applications.\n\n.. code-block:: python\n\n import asyncio\n\n async def handle_client(reader, writer):\n addr = writer.get_extra_info('peername')\n print(f\"Connected: {addr}\")\n\n while True:\n data = await reader.read(1024)\n if not data:\n break\n message = data.decode()\n print(f\"Received: {message!r} from {addr}\")\n writer.write(data)\n await writer.drain()\n\n print(f\"Disconnected: {addr}\")\n writer.close()\n await writer.wait_closed()\n\n async def main():\n server = await asyncio.start_server(\n handle_client, 'localhost', 8888\n )\n addr = server.sockets[0].getsockname()\n print(f\"Serving on {addr}\")\n\n async with server:\n await server.serve_forever()\n\n asyncio.run(main())\n\nTCP Client with Streams\n-----------------------\n\nThe client side uses ``asyncio.open_connection()`` to establish a connection.\nThe returned reader and writer objects provide async methods for sending and\nreceiving data.\n\n.. code-block:: python\n\n import asyncio\n\n async def tcp_client(message):\n reader, writer = await asyncio.open_connection(\n 'localhost', 8888\n )\n\n print(f\"Sending: {message!r}\")\n writer.write(message.encode())\n await writer.drain()\n\n data = await reader.read(1024)\n print(f\"Received: {data.decode()!r}\")\n\n writer.close()\n await writer.wait_closed()\n\n asyncio.run(tcp_client(\"Hello, Server!\"))\n\nLow-Level TCP with Sockets\n--------------------------\n\nFor more control, you can use raw sockets with the event loop's socket methods.\nThis approach is useful when you need fine-grained control over socket options\nor when integrating with existing socket-based code.\n\n.. code-block:: python\n\n import asyncio\n import socket\n\n async def handle_client(loop, conn):\n while True:\n data = await loop.sock_recv(conn, 1024)\n if not data:\n break\n await loop.sock_sendall(conn, data)\n conn.close()\n\n async def server():\n loop = asyncio.get_event_loop()\n\n sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n sock.setblocking(False)\n sock.bind(('localhost', 8888))\n sock.listen(100)\n\n print(\"Server listening on localhost:8888\")\n while True:\n conn, addr = await loop.sock_accept(sock)\n print(f\"Connected: {addr}\")\n asyncio.create_task(handle_client(loop, conn))\n\n asyncio.run(server())\n\nUDP Echo Server\n---------------\n\nUDP is connectionless, so the API is different from TCP. Use\n``create_datagram_endpoint()`` with a protocol class to handle UDP packets.\nEach packet is independent and may arrive out of order or not at all.\n\n.. code-block:: python\n\n import asyncio\n\n class EchoUDPProtocol(asyncio.DatagramProtocol):\n def connection_made(self, transport):\n self.transport = transport\n\n def datagram_received(self, data, addr):\n message = data.decode()\n print(f\"Received {message!r} from {addr}\")\n self.transport.sendto(data, addr)\n\n async def main():\n loop = asyncio.get_event_loop()\n transport, protocol = await loop.create_datagram_endpoint(\n EchoUDPProtocol,\n local_addr=('localhost', 9999)\n )\n print(\"UDP server listening on localhost:9999\")\n\n try:\n await asyncio.sleep(3600) # Run for 1 hour\n finally:\n transport.close()\n\n asyncio.run(main())\n\nHTTP Client with SSL\n--------------------\n\nMaking HTTPS requests requires SSL context configuration. This example shows\nhow to fetch web pages using low-level streams with proper SSL verification.\n\n.. code-block:: python\n\n import asyncio\n import ssl\n\n async def fetch_https(host, path=\"/\"):\n # Create SSL context with certificate verification\n ctx = ssl.create_default_context()\n\n reader, writer = await asyncio.open_connection(\n host, 443, ssl=ctx\n )\n\n # Send HTTP request\n request = f\"GET {path} HTTP/1.1\\r\\nHost: {host}\\r\\nConnection: close\\r\\n\\r\\n\"\n writer.write(request.encode())\n await writer.drain()\n\n # Read response\n response = await reader.read()\n writer.close()\n await writer.wait_closed()\n\n return response.decode()\n\n async def main():\n urls = [\n (\"www.python.org\", \"/\"),\n (\"github.com\", \"/\"),\n ]\n tasks = [fetch_https(host, path) for host, path in urls]\n responses = await asyncio.gather(*tasks)\n\n for (host, _), resp in zip(urls, responses):\n status = resp.split('\\r\\n')[0]\n print(f\"{host}: {status}\")\n\n asyncio.run(main())\n\nHTTPS Server with SSL\n---------------------\n\nCreating an HTTPS server requires SSL certificates. This example shows a\nsimple HTTPS server that serves static content with TLS encryption.\n\n.. code-block:: python\n\n import asyncio\n import ssl\n\n async def handle_request(reader, writer):\n request = await reader.read(1024)\n\n response = b\"HTTP/1.1 200 OK\\r\\n\"\n response += b\"Content-Type: text/html\\r\\n\\r\\n\"\n response += b\"
Hello HTTPS!
\"\n\n writer.write(response)\n await writer.drain()\n writer.close()\n await writer.wait_closed()\n\n async def main():\n # Create SSL context\n ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)\n ctx.load_cert_chain('cert.pem', 'key.pem')\n\n server = await asyncio.start_server(\n handle_request, 'localhost', 8443, ssl=ctx\n )\n print(\"HTTPS server on https://localhost:8443\")\n\n async with server:\n await server.serve_forever()\n\n # Generate self-signed cert:\n # openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes\n asyncio.run(main())\n\nTransport and Protocol API\n--------------------------\n\nThe Transport/Protocol API provides low-level control over network connections.\nTransports handle the actual I/O while Protocols handle the data processing.\nThis separation allows for flexible and reusable network code.\n\n.. code-block:: python\n\n import asyncio\n\n class EchoProtocol(asyncio.Protocol):\n def connection_made(self, transport):\n self.transport = transport\n peername = transport.get_extra_info('peername')\n print(f\"Connection from {peername}\")\n\n def data_received(self, data):\n print(f\"Received: {data.decode()!r}\")\n self.transport.write(data)\n\n def connection_lost(self, exc):\n print(\"Connection closed\")\n\n async def main():\n loop = asyncio.get_event_loop()\n server = await loop.create_server(\n EchoProtocol, 'localhost', 8888\n )\n\n async with server:\n await server.serve_forever()\n\n asyncio.run(main())\n\nDNS Resolution\n--------------\n\nAsyncio provides async DNS resolution through ``getaddrinfo()``. This is\nuseful when you need to resolve hostnames without blocking the event loop.\n\n.. code-block:: python\n\n import asyncio\n import socket\n\n async def resolve_host(host, port=80):\n loop = asyncio.get_event_loop()\n infos = await loop.getaddrinfo(\n host, port,\n family=socket.AF_UNSPEC,\n type=socket.SOCK_STREAM\n )\n\n for family, type_, proto, canonname, sockaddr in infos:\n ip, port = sockaddr[:2]\n family_name = \"IPv4\" if family == socket.AF_INET else \"IPv6\"\n print(f\"{host} -> {ip} ({family_name})\")\n\n async def main():\n hosts = [\"python.org\", \"github.com\", \"google.com\"]\n await asyncio.gather(*[resolve_host(h) for h in hosts])\n\n asyncio.run(main())\n\nSimple HTTP Server\n------------------\n\nA minimal HTTP server implementation showing how to parse requests and\nsend responses. For production use, consider frameworks like aiohttp or\nFastAPI.\n\n.. code-block:: python\n\n import asyncio\n\n async def handle_http(reader, writer):\n request = await reader.read(1024)\n request_line = request.decode().split('\\r\\n')[0]\n method, path, _ = request_line.split(' ')\n\n print(f\"{method} {path}\")\n\n # Simple routing\n if path == '/':\n body = b\"
Home
\"\n status = \"200 OK\"\n elif path == '/about':\n body = b\"
About
\"\n status = \"200 OK\"\n else:\n body = b\"
404 Not Found
\"\n status = \"404 Not Found\"\n\n response = f\"HTTP/1.1 {status}\\r\\n\"\n response += f\"Content-Length: {len(body)}\\r\\n\"\n response += \"Content-Type: text/html\\r\\n\\r\\n\"\n\n writer.write(response.encode() + body)\n await writer.drain()\n writer.close()\n await writer.wait_closed()\n\n async def main():\n server = await asyncio.start_server(\n handle_http, 'localhost', 8080\n )\n print(\"HTTP server on http://localhost:8080\")\n\n async with server:\n await server.serve_forever()\n\n asyncio.run(main())\n\nUsing sendfile for Efficient File Transfer\n------------------------------------------\n\nThe ``sendfile()`` method (Python 3.7+) efficiently transfers file contents\nto a transport using the OS's sendfile syscall, avoiding copying data through\nPython.\n\n.. code-block:: python\n\n import asyncio\n\n async def handle_request(reader, writer):\n await reader.read(1024) # Read request\n\n with open('index.html', 'rb') as f:\n # Get file size\n f.seek(0, 2)\n size = f.tell()\n f.seek(0)\n\n # Send headers\n headers = f\"HTTP/1.1 200 OK\\r\\n\"\n headers += f\"Content-Length: {size}\\r\\n\"\n headers += \"Content-Type: text/html\\r\\n\\r\\n\"\n writer.write(headers.encode())\n\n # Send file efficiently\n loop = asyncio.get_event_loop()\n await loop.sendfile(writer.transport, f)\n\n writer.close()\n await writer.wait_closed()\n\n async def main():\n server = await asyncio.start_server(\n handle_request, 'localhost', 8080\n )\n async with server:\n await server.serve_forever()\n\n asyncio.run(main())\n\nConnection Pool\n---------------\n\nConnection pools reuse connections to avoid the overhead of establishing\nnew connections for each request. This is essential for high-performance\nclients that make many requests to the same server.\n\n.. code-block:: python\n\n import asyncio\n from collections import deque\n\n class ConnectionPool:\n def __init__(self, host, port, size=5):\n self.host = host\n self.port = port\n self.size = size\n self._pool = deque()\n self._lock = asyncio.Lock()\n\n async def get(self):\n async with self._lock:\n if self._pool:\n return self._pool.popleft()\n\n # Create new connection\n reader, writer = await asyncio.open_connection(\n self.host, self.port\n )\n return reader, writer\n\n async def put(self, reader, writer):\n async with self._lock:\n if len(self._pool) < self.size:\n self._pool.append((reader, writer))\n else:\n writer.close()\n await writer.wait_closed()\n\n async def close(self):\n async with self._lock:\n while self._pool:\n reader, writer = self._pool.popleft()\n writer.close()\n await writer.wait_closed()\n\n async def fetch(pool, message):\n reader, writer = await pool.get()\n try:\n writer.write(message.encode())\n await writer.drain()\n data = await reader.read(1024)\n return data.decode()\n finally:\n await pool.put(reader, writer)\n\n async def main():\n pool = ConnectionPool('localhost', 8888, size=3)\n try:\n tasks = [fetch(pool, f\"msg{i}\") for i in range(10)]\n results = await asyncio.gather(*tasks)\n for r in results:\n print(r)\n finally:\n await pool.close()\n\n asyncio.run(main())\n"
},
{
"path": "docs/notes/basic/index.rst",
"content": ".. meta::\n :description lang=en: Python basics cheat sheet covering syntax, data types, functions, classes, generators, typing, and essential Python programming concepts\n :keywords: Python, Python3, basics, syntax, data types, functions, classes, generators, typing, list, dict, set, comprehension\n\nQuick Start\n===========\n\nThis cheat sheet is designed to help developers learn Python syntax from the\nground up. It covers the fundamentals while also introducing common patterns\nand idioms that experienced Python developers use, which may feel unfamiliar to\nbeginners. For instance, constructs like ``for ... else ...`` are rarely seen in\nother programming languages. Additionally, we’ll explore interesting topics such\nas ``__future__``, typing, and Unicode—concepts you may have heard of but never\nfully understood. By working through this cheat sheet, you’ll gain a solid\nfoundation in Python and learn to write code that feels truly Pythonic.\n\n.. toctree::\n :maxdepth: 1\n\n python-basic\n python-future\n python-func\n python-object\n python-typing\n python-list\n python-set\n python-dict\n python-heap\n python-generator\n python-unicode\n python-rexp\n"
},
{
"path": "docs/notes/basic/python-basic.rst",
"content": ".. meta::\n :description lang=en: Python basics tutorial covering fundamental syntax, data types, control flow, string formatting, and essential Python programming concepts\n :keywords: Python, Python3, basics, syntax, data types, control flow, string formatting, variables, operators, conditionals\n\n============\nFrom Scratch\n============\n\n.. contents:: Table of Contents\n :backlinks: none\n\nThe main goal of this cheat sheet is to collect some common and basic semantics\nor snippets. The cheat sheet includes some syntax, which we have already known\nbut still ambiguous in our mind, or some snippets, which we google them again\nand again. In addition, because **the end Of life date for Python 2** is coming.\nMost of the snippets are mainly based on **Python 3**'s syntax.\n\n\nHello world!\n------------\n\nWhen we start to learn a new language, we usually learn from printing\n**Hello world!**. In Python, we can use another way to print the message by\nimporting ``__hello__`` module. The source code can be found on\n`frozen.c `_.\n\n.. code-block:: python\n\n >>> print(\"Hello world!\")\n Hello world!\n >>> import __hello__\n Hello world!\n >>> import __phello__\n Hello world!\n >>> import __phello__.spam\n Hello world!\n\n\nPython Version\n--------------\n\nIt is important for a programmer to know current Python version because\nnot every syntax will work in the current version. In this case, we can get the\nPython version by ``python -V`` or using the module, ``sys``.\n\n.. code-block:: python\n\n >>> import sys\n >>> print(sys.version)\n 3.7.1 (default, Nov 6 2018, 18:46:03)\n [Clang 10.0.0 (clang-1000.11.45.5)]\n\nWe can also use ``platform.python_version`` to get Python version.\n\n.. code-block:: python\n\n >>> import platform\n >>> platform.python_version()\n '3.7.1'\n\nSometimes, checking the current Python version is important because we may want\nto enable some features in some specific versions. ``sys.version_info`` provides more\ndetail information about the interpreter. We can use it to compare with the\nversion we want.\n\n.. code-block:: python\n\n >>> import sys\n >>> sys.version_info >= (3, 6)\n True\n >>> sys.version_info >= (3, 7)\n False\n\nEllipsis\n--------\n\n`Ellipsis `_ is a\nbuilt-in constant. After Python 3.0, we case use ``...`` as ``Ellipsis``. It\nmay be the most enigmatic constant in Python. Based on the official document,\nwe can use it to extend slicing syntax. Nevertheless, there are some other\nconventions in type hinting, stub files, or function expressions.\n\n.. code-block:: python\n\n >>> ...\n Ellipsis\n >>> ... == Ellipsis\n True\n >>> type(...)\n \n\nThe following snippet shows that we can use the ellipsis to represent a function\nor a class which has not implemented yet.\n\n.. code-block:: python\n\n >>> class Foo: ...\n ...\n >>> def foo(): ...\n ...\n\nif ... elif ... else\n--------------------\n\nThe **if statements** are used to control the code flow. Instead of using\n``switch`` or ``case`` statements control the logic of the code, Python uses\n``if ... elif ... else`` sequence. Although someone proposes we can use\n``dict`` to achieve ``switch`` statements, this solution may introduce\nunnecessary overhead such as creating disposable dictionaries and undermine\na readable code. Thus, the solution is not recommended.\n\n.. code-block:: python\n\n >>> import random\n >>> num = random.randint(0, 10)\n >>> if num < 3:\n ... print(\"less than 3\")\n ... elif num < 5:\n ... print(\"less than 5\")\n ... else:\n ... print(num)\n ...\n less than 3\n\nfor Loop\n--------\n\nIn Python, we can access iterable object's items directly through the\n**for statement**. If we need to get indexes and items of an iterable object\nsuch as list or tuple at the same time, using ``enumerate`` is better than\n``range(len(iterable))``. Further information can be found on\n`Looping Techniques `_.\n\n.. code-block:: python\n\n >>> for val in [\"foo\", \"bar\"]:\n ... print(val)\n ...\n foo\n bar\n >>> for idx, val in enumerate([\"foo\", \"bar\", \"baz\"]):\n ... print(idx, val)\n ...\n (0, 'foo')\n (1, 'bar')\n (2, 'baz')\n\nfor ... else ...\n----------------\n\nIt may be a little weird when we see the ``else`` belongs to a ``for`` loop at\nthe first time. The ``else`` clause can assist us to avoid using flag\nvariables in loops. A loop’s ``else`` clause runs when no break occurs.\n\n.. code-block:: python\n\n >>> for _ in range(5):\n ... pass\n ... else:\n ... print(\"no break\")\n ...\n no break\n\nThe following snippet shows the difference between using a flag variable and\nthe ``else`` clause to control the loop. We can see that the ``else`` does not\nrun when the ``break`` occurs in the loop.\n\n.. code-block:: python\n\n >>> is_break = False\n >>> for x in range(5):\n ... if x % 2 == 0:\n ... is_break = True\n ... break\n ...\n >>> if is_break:\n ... print(\"break\")\n ...\n break\n\n >>> for x in range(5):\n ... if x % 2 == 0:\n ... print(\"break\")\n ... break\n ... else:\n ... print(\"no break\")\n ...\n break\n\nUsing ``range``\n---------------\n\nThe problem of ``range`` in Python 2 is that ``range`` may take up a lot of\nmemory if we need to iterate a loop many times. Consequently, using ``xrange``\nis recommended in Python 2.\n\n.. code-block:: python\n\n >>> import platform\n >>> import sys\n >>> platform.python_version()\n '2.7.15'\n >>> sys.getsizeof(range(100000000))\n 800000072\n >>> sys.getsizeof(xrange(100000000))\n 40\n\nIn Python 3, the built-in function ``range`` returns an iterable **range object**\ninstead of a list. The behavior of ``range`` is the same as the ``xrange`` in\nPython 2. Therefore, using ``range`` do not take up huge memory anymore if we\nwant to run a code block many times within a loop. Further information can be\nfound on PEP `3100 `_.\n\n.. code-block:: python\n\n >>> import platform\n >>> import sys\n >>> platform.python_version()\n '3.7.1'\n >>> sys.getsizeof(range(100000000))\n 48\n\nwhile ... else ...\n------------------\n\nThe ``else`` clause belongs to a while loop serves the same purpose as the\n``else`` clause in a for loop. We can observe that the ``else`` does not run\nwhen the ``break`` occurs in the while loop.\n\n.. code-block:: python\n\n >>> n = 0\n >>> while n < 5:\n ... if n == 3:\n ... break\n ... n += 1\n ... else:\n ... print(\"no break\")\n ...\n\nThe ``do while`` Statement\n--------------------------\n\nThere are many programming languages such as C/C++, Ruby, or Javascript,\nprovide the ``do while`` statement. In Python, there is no ``do while``\nstatement. However, we can place the condition and the ``break`` at the end of\na ``while`` loop to achieve the same thing.\n\n.. code-block:: python\n\n >>> n = 0\n >>> while True:\n ... n += 1\n ... if n == 5:\n ... break\n ...\n >>> n\n 5\n\ntry ... except ... else ...\n---------------------------\n\nMost of the time, we handle errors in ``except`` clause and clean up resources\nin ``finally`` clause. Interestingly, the ``try`` statement also provides an\n``else`` clause for us to avoid catching an exception which was raised by the\ncode that should not be protected by ``try ... except``. The ``else`` clause\nruns when no exception occurs between ``try`` and ``except``.\n\n.. code-block:: python\n\n >>> try:\n ... print(\"No exception\")\n ... except:\n ... pass\n ... else:\n ... print(\"Success\")\n ...\n No exception\n Success\n\nString\n------\n\nUnlike other programming languages, Python does not support string’s item\nassignment directly. Therefore, if it is necessary to manipulate string’s\nitems, e.g., swap items, we have to convert a string to a list and do a join\noperation after a series item assignments finish.\n\n.. code-block:: python\n\n >>> a = \"Hello Python\"\n >>> l = list(a)\n >>> l[0], l[6] = 'h', 'p'\n >>> ''.join(l)\n 'hello python'\n\nList\n----\n\nLists are versatile containers. Python provides a lot of ways such as\n**negative index**, **slicing statement**, or **list comprehension** to\nmanipulate lists. The following snippet shows some common operations of lists.\n\n.. code-block:: python\n\n >>> a = [1, 2, 3, 4, 5]\n >>> a[-1] # negative index\n 5\n >>> a[1:] # slicing\n [2, 3, 4, 5]\n >>> a[1:-1]\n [2, 3, 4]\n >>> a[1:-1:2]\n [2, 4]\n >>> a[::-1] # reverse\n [5, 4, 3, 2, 1]\n >>> a[0] = 0 # set an item\n >>> a\n [0, 2, 3, 4, 5]\n >>> a.append(6) # append an item\n >>> a\n [0, 2, 3, 4, 5, 6]\n >>> del a[-1] # del an item\n >>> a\n [0, 2, 3, 4, 5]\n >>> b = [x for x in range(3)] # list comprehension\n >>> b\n [0, 1, 2]\n >>> a + b # add two lists\n [0, 2, 3, 4, 5, 0, 1, 2]\n\nDict\n----\n\nDictionaries are key-value pairs containers. Like lists, Python supports many\nways such as **dict comprehensions** to manipulate dictionaries. After\nPython 3.6, dictionaries preserve the insertion order of keys. The Following\nsnippet shows some common operations of dictionaries.\n\n.. code-block:: python\n\n >>> d = {'timmy': 'red', 'barry': 'green', 'guido': 'blue'}\n >>> d\n {'timmy': 'red', 'barry': 'green', 'guido': 'blue'}\n >>> d['timmy'] = \"yellow\" # set data\n >>> d\n {'timmy': 'yellow', 'barry': 'green', 'guido': 'blue'}\n >>> del d['guido'] # del data\n >>> d\n >>> 'guido' in d # contain data\n False\n {'timmy': 'yellow', 'barry': 'green'}\n >>> {k: v for k ,v in d.items()} # dict comprehension\n {'timmy': 'yellow', 'barry': 'green'}\n >>> d.keys() # list all keys\n dict_keys(['timmy', 'barry'])\n >>> d.values() # list all values\n dict_values(['yellow', 'green'])\n\nFunction\n--------\n\nDefining a function in Python is flexible. We can define a function with\n**function documents**, **default values**, **arbitrary arguments**,\n**keyword arguments**, **keyword-only arguments**, and so on. The Following\nsnippet shows some common expressions to define functions.\n\n.. code-block:: python\n\n def foo_with_doc():\n \"\"\"Documentation String.\"\"\"\n\n def foo_with_arg(arg): ...\n def foo_with_args(*arg): ...\n def foo_with_kwarg(a, b=\"foo\"): ...\n def foo_with_args_kwargs(*args, **kwargs): ...\n def foo_with_kwonly(a, b, *, k): ... # python3\n def foo_with_annotations(a: int) -> int: ... # python3\n\nFunction Annotations\n--------------------\n\nInstead of writing string documents in functions to hint the type of parameters\nand return values, we can denote types by **function annotations**. Function annotations\nwhich the details can be found on PEP `3017 `_\nand PEP `484 `_ were introduced in\nPython 3.0. They are an **optional** feature in **Python 3**. Using function\nannotations will lose compatibility in **Python 2**. We can solve this issue\nby stub files. In addition, we can do static type checking through\n`mypy `_.\n\n.. code-block:: python\n\n >>> def fib(n: int) -> int:\n ... a, b = 0, 1\n ... for _ in range(n):\n ... b, a = a + b, b\n ... return a\n ...\n >>> fib(10)\n 55\n\nGenerators\n----------\n\nPython uses the ``yield`` statement to define a **generator function**. In\nother words, when we call a generator function, the generator function will\nreturn a **generator** instead of return values for creating an **iterator**.\n\n.. code-block:: python\n\n >>> def fib(n):\n ... a, b = 0, 1\n ... for _ in range(n):\n ... yield a\n ... b, a = a + b, b\n ...\n >>> g = fib(10)\n >>> g\n \n >>> for f in fib(5):\n ... print(f)\n ...\n 0\n 1\n 1\n 2\n 3\n\nGenerator Delegation\n--------------------\n\nPython 3.3 introduced ``yield from`` expression. It allows a generator to\ndelegate parts of operations to another generator. In other words, we can\n**yield** a sequence **from** other **generators** in the current **generator function**.\nFurther information can be found on PEP `380 `_.\n\n.. code-block:: python\n\n >>> def fib(n):\n ... a, b = 0, 1\n ... for _ in range(n):\n ... yield a\n ... b, a = a + b, b\n ...\n >>> def fibonacci(n):\n ... yield from fib(n)\n ...\n >>> [f for f in fibonacci(5)]\n [0, 1, 1, 2, 3]\n\nClass\n-----\n\nPython supports many common features such as **class documents**, **multiple inheritance**,\n**class variables**, **instance variables**, **static method**, **class method**, and so on.\nFurthermore, Python provides some special methods for programmers to implement\n**iterators**, **context manager**, etc. The following snippet displays common definition\nof a class.\n\n.. code-block:: python\n\n class A: ...\n class B: ...\n class Foo(A, B):\n \"\"\"A class document.\"\"\"\n\n foo = \"class variable\"\n\n def __init__(self, v):\n self.attr = v\n self.__private = \"private var\"\n\n @staticmethod\n def bar_static_method(): ...\n\n @classmethod\n def bar_class_method(cls): ...\n\n def bar(self):\n \"\"\"A method document.\"\"\"\n\n def bar_with_arg(self, arg): ...\n def bar_with_args(self, *args): ...\n def bar_with_kwarg(self, kwarg=\"bar\"): ...\n def bar_with_args_kwargs(self, *args, **kwargs): ...\n def bar_with_kwonly(self, *, k): ...\n def bar_with_annotations(self, a: int): ...\n\n``async`` / ``await``\n---------------------\n\n``async`` and ``await`` syntax was introduced from Python 3.5. They were\ndesigned to be used with an event loop. Some other features such as the\n**asynchronous generator** were implemented in later versions.\n\nA **coroutine function**\n(``async def``) are used to create a **coroutine** for an event loop. Python\nprovides a built-in module, **asyncio**, to write a concurrent code through\n``async``/``await`` syntax. The following snippet shows a simple example of\nusing **asyncio**. The code must be run on Python 3.7 or above.\n\n.. code-block:: python\n\n import asyncio\n\n async def http_ok(r, w):\n head = b\"HTTP/1.1 200 OK\\r\\n\"\n head += b\"Content-Type: text/html\\r\\n\"\n head += b\"\\r\\n\"\n\n body = b\"\"\n body += b\"
Hello world!
\"\n body += b\"\"\n\n _ = await r.read(1024)\n w.write(head + body)\n await w.drain()\n w.close()\n\n async def main():\n server = await asyncio.start_server(\n http_ok, \"127.0.0.1\", 8888\n )\n\n async with server:\n await server.serve_forever()\n\n asyncio.run(main())\n\nAvoid ``exec`` and ``eval``\n---------------------------\n\nThe following snippet shows how to use the built-in function ``exec``. Yet,\nusing ``exec`` and ``eval`` are not recommended because of some security issues\nand unreadable code for a human. Further reading can be found on\n`Be careful with exec and eval in Python `_\nand `Eval really is dangerous `_\n\n\n.. code-block:: python\n\n >>> py = '''\n ... def fib(n):\n ... a, b = 0, 1\n ... for _ in range(n):\n ... b, a = b + a, b\n ... return a\n ... print(fib(10))\n ... '''\n >>> exec(py, globals(), locals())\n 55\n"
},
{
"path": "docs/notes/basic/python-dict.rst",
"content": ".. meta::\n :description lang=en: Python dictionary cheat sheet covering creation, manipulation, merging, comprehensions, defaultdict, OrderedDict, and LRU cache with code examples\n :keywords: Python, Python3, Python dictionary, Python dict cheat sheet, dict, hashmap, key-value pairs, defaultdict, OrderedDict, dictionary comprehension, LRU cache, dict methods\n\n==========\nDictionary\n==========\n\n.. contents:: Table of Contents\n :backlinks: none\n\nDictionaries are one of Python's most powerful and frequently used data structures.\nThey store key-value pairs and provide O(1) average time complexity for lookups,\ninsertions, and deletions. Since Python 3.7, dictionaries maintain insertion order\nas a language feature. This cheat sheet covers essential dictionary operations,\nfrom basic manipulation to advanced patterns like emulating dictionary behavior\nwith special methods and implementing an LRU (Least Recently Used) cache.\n\nThe source code is available on `GitHub `_.\n\nReferences\n----------\n\n- `Mapping Types — dict `_\n- `collections — Container datatypes `_\n- `PEP 584 -- Add Union Operators To dict `_\n\nGet All Keys with ``dict.keys()``\n---------------------------------\n\nThe ``keys()`` method returns a view object containing all dictionary keys.\nIn Python 3, this is a dynamic view that reflects changes to the dictionary.\n\n.. code-block:: python\n\n >>> a = {\"1\":1, \"2\":2, \"3\":3}\n >>> b = {\"2\":2, \"3\":3, \"4\":4}\n >>> a.keys()\n ['1', '3', '2']\n\nGet Key-Value Pairs with ``dict.items()``\n-----------------------------------------\n\nThe ``items()`` method returns key-value pairs as tuples, which is useful for\niterating over both keys and values simultaneously.\n\n.. code-block:: python\n\n >>> a = {\"1\":1, \"2\":2, \"3\":3}\n >>> a.items()\n\nFind Common Keys Between Dictionaries\n-------------------------------------\n\nFinding keys that exist in multiple dictionaries is a common operation. Using\nset intersection is the most efficient approach.\n\n.. code-block:: python\n\n >>> a = {\"1\":1, \"2\":2, \"3\":3}\n >>> b = {\"2\":2, \"3\":3, \"4\":4}\n >>> [_ for _ in a.keys() if _ in b.keys()]\n ['3', '2']\n >>> # better way\n >>> c = set(a).intersection(set(b))\n >>> list(c)\n ['3', '2']\n >>> # or\n >>> [_ for _ in a if _ in b]\n ['3', '2']\n [('1', 1), ('3', 3), ('2', 2)]\n\nSet Default Values with ``setdefault()`` and ``defaultdict``\n------------------------------------------------------------\n\nWhen working with dictionaries, you often need to set default values for missing\nkeys. Python provides ``setdefault()`` and ``collections.defaultdict`` for this.\n\n.. code-block:: python\n\n >>> # intuitive but not recommend\n >>> d = {}\n >>> key = \"foo\"\n >>> if key not in d:\n ... d[key] = []\n ...\n\n # using d.setdefault(key[, default])\n >>> d = {}\n >>> key = \"foo\"\n >>> d.setdefault(key, [])\n []\n >>> d[key] = 'bar'\n >>> d\n {'foo': 'bar'}\n\n # using collections.defaultdict\n >>> from collections import defaultdict\n >>> d = defaultdict(list)\n >>> d[\"key\"]\n []\n >>> d[\"foo\"]\n []\n >>> d[\"foo\"].append(\"bar\")\n >>> d\n defaultdict(, {'key': [], 'foo': ['bar']})\n\n``dict.setdefault(key[, default])`` returns its default value if *key* is not in\nthe dictionary. However, if the key exists in the dictionary, the function will\nreturn its value.\n\n.. code-block:: python\n\n >>> d = {}\n >>> d.setdefault(\"key\", [])\n []\n >>> d[\"key\"] = \"bar\"\n >>> d.setdefault(\"key\", [])\n 'bar'\n\nUpdate Dictionary with ``dict.update()``\n----------------------------------------\n\nThe ``update()`` method merges another dictionary into the current one. Keys from\nthe second dictionary overwrite existing keys in the first.\n\n.. code-block:: python\n\n >>> a = {\"1\":1, \"2\":2, \"3\":3}\n >>> b = {\"2\":2, \"3\":3, \"4\":4}\n >>> a.update(b)\n >>> a\n {'1': 1, '3': 3, '2': 2, '4': 4}\n\nMerge Two Dictionaries in Python\n--------------------------------\n\nThere are several ways to merge dictionaries depending on your Python version.\nPython 3.9+ also supports the ``|`` operator for dictionary merging.\n\nPython 3.4 or lower\n\n.. code-block:: python\n\n >>> a = {\"x\": 55, \"y\": 66}\n >>> b = {\"a\": \"foo\", \"b\": \"bar\"}\n >>> c = a.copy()\n >>> c.update(b)\n >>> c\n {'y': 66, 'x': 55, 'b': 'bar', 'a': 'foo'}\n\n\nPython 3.5 or above\n\n.. code-block:: python\n\n >>> a = {\"x\": 55, \"y\": 66}\n >>> b = {\"a\": \"foo\", \"b\": \"bar\"}\n >>> c = {**a, **b}\n >>> c\n {'x': 55, 'y': 66, 'a': 'foo', 'b': 'bar'}\n\nEmulate a Dictionary with Special Methods\n-----------------------------------------\n\nYou can create dictionary-like objects by implementing special methods:\n``__getitem__``, ``__setitem__``, ``__delitem__``, ``__contains__``, and ``__iter__``.\n\n.. code-block:: python\n\n >>> class EmuDict(object):\n ... def __init__(self, dict_):\n ... self._dict = dict_\n ... def __repr__(self):\n ... return \"EmuDict: \" + repr(self._dict)\n ... def __getitem__(self, key):\n ... return self._dict[key]\n ... def __setitem__(self, key, val):\n ... self._dict[key] = val\n ... def __delitem__(self, key):\n ... del self._dict[key]\n ... def __contains__(self, key):\n ... return key in self._dict\n ... def __iter__(self):\n ... return iter(self._dict.keys())\n ...\n >>> _ = {\"1\":1, \"2\":2, \"3\":3}\n >>> emud = EmuDict(_)\n >>> emud # __repr__\n EmuDict: {'1': 1, '2': 2, '3': 3}\n >>> emud['1'] # __getitem__\n 1\n >>> emud['5'] = 5 # __setitem__\n >>> emud\n EmuDict: {'1': 1, '2': 2, '3': 3, '5': 5}\n >>> del emud['2'] # __delitem__\n >>> emud\n EmuDict: {'1': 1, '3': 3, '5': 5}\n >>> for _ in emud:\n ... print(emud[_], end=' ') # __iter__\n ... else:\n ... print()\n ...\n 1 3 5\n >>> '1' in emud # __contains__\n True\n\nImplement LRU Cache with OrderedDict\n------------------------------------\n\nAn LRU (Least Recently Used) cache evicts the least recently accessed items when\nfull. ``OrderedDict.move_to_end()`` makes implementation straightforward.\n\n.. code-block:: python\n\n\tfrom collections import OrderedDict\n\n\n\tclass LRU(object):\n\t\tdef __init__(self, maxsize=128):\n\t\t\tself._maxsize = maxsize\n\t\t\tself._cache = OrderedDict()\n\n\t\tdef get(self, k):\n\t\t\tif k not in self._cache:\n\t\t\t\treturn None\n\n\t\t\tself._cache.move_to_end(k)\n\t\t\treturn self._cache[k]\n\n\t\tdef put(self, k, v):\n\t\t\tif k in self._cache:\n\t\t\t\tself._cache.move_to_end(k)\n\t\t\tself._cache[k] = v\n\t\t\tif len(self._cache) > self._maxsize:\n\t\t\t\tself._cache.popitem(last=False)\n\n\t\tdef __str__(self):\n\t\t\treturn str(self._cache)\n\n\t\tdef __repr__(self):\n\t\t\treturn self.__str__()\n\nNote that dictionaries preserve insertion order from Python 3.7. Moreover,\nupdating a key does not affect the order. Therefore, a dictionary can also\nsimulate an LRU cache, which is similar to using an OrderedDict.\n\n.. code-block:: python\n\n\tclass LRU(object):\n\t\tdef __init__(self, maxsize=128):\n\t\t\tself._maxsize = maxsize\n\t\t\tself._cache = {}\n\n\t\tdef get(self, k):\n\t\t\tif k not in self._cache:\n\t\t\t\treturn None\n\n\t\t\tself.move_to_end(k)\n\t\t\treturn self._cache[k]\n\n\t\tdef put(self, k, v):\n\t\t\tif k in self._cache:\n\t\t\t\tself.move_to_end(k)\n\t\t\tself._cache[k] = v\n\t\t\tif len(self._cache) > self._maxsize:\n\t\t\t\tself.pop()\n\n\t\tdef pop(self):\n\t\t\tit = iter(self._cache.keys())\n\t\t\tdel self._cache[next(it)]\n\n\t\tdef move_to_end(self, k):\n\t\t\tif k not in self._cache:\n\t\t\t\treturn\n\t\t\tv = self._cache[k]\n\t\t\tdel self._cache[k]\n\t\t\tself._cache[k] = v\n\n\t\tdef __str__(self):\n\t\t\treturn str(self._cache)\n\n\t\tdef __repr__(self):\n\t\t\treturn self.__str__()\n"
},
{
"path": "docs/notes/basic/python-func.rst",
"content": ".. meta::\n :description lang=en: Python function cheat sheet covering function definitions, arguments, decorators, lambda, closures, and functools with code examples\n :keywords: Python, Python3, Python function, Python function cheat sheet, decorator, lambda, closure, *args, **kwargs, functools, lru_cache, partial\n\n========\nFunction\n========\n\n.. contents:: Table of Contents\n :backlinks: none\n\nA function can help programmers to wrap their logic into a task for avoiding\nduplicate code. In Python, the definition of a function is so versatile that\nwe can use many features such as decorator, annotation, docstrings, default\narguments and so on to define a function. In this cheat sheet, it collects\nmany ways to define a function and demystifies some enigmatic syntax in functions.\n\nDocument Functions\n------------------\n\nDocumentation provides programmers hints about how a function is supposed to\nbe used. A docstring gives an expedient way to write a readable document of\nfunctions. The docstring should be placed as the first statement in the function\nbody, enclosed in triple quotes. It can be accessed via the ``__doc__`` attribute\nor the built-in ``help()`` function. PEP `257 `_\ndefines conventions for docstrings, and tools like ``pydocstyle`` can help\nenforce these conventions in your codebase.\n\n.. code-block:: python\n\n >>> def example():\n ... \"\"\"This is an example function.\"\"\"\n ... print(\"Example function\")\n ...\n >>> example.__doc__\n 'This is an example function.'\n >>> help(example)\n\nDefault Arguments\n-----------------\n\nDefining a function where the arguments are optional and have a default value\nis quite simple in Python. We can just assign values in the definition and make\nsure the default arguments appear in the end. When calling the function, you can\nomit arguments that have defaults, pass them positionally, or use keyword syntax\nto specify them explicitly. This flexibility makes functions more versatile and\neasier to use in different contexts.\n\n.. code-block:: python\n\n >>> def add(a, b=0):\n ... return a + b\n ...\n >>> add(1)\n 1\n >>> add(1, 2)\n 3\n >>> add(1, b=2)\n 3\n\n.. warning::\n\n Avoid using mutable objects (like lists or dictionaries) as default arguments.\n Default argument values are evaluated only once when the function is defined,\n not each time the function is called. This means mutable defaults are shared\n across all calls, which can lead to unexpected behavior where modifications\n persist between function calls.\n\n .. code-block:: python\n\n >>> def bad(items=[]): # DON'T do this\n ... items.append(1)\n ... return items\n ...\n >>> bad()\n [1]\n >>> bad() # unexpected!\n [1, 1]\n\n >>> def good(items=None): # DO this instead\n ... if items is None:\n ... items = []\n ... items.append(1)\n ... return items\n\nVariable Arguments ``*args`` and ``**kwargs``\n---------------------------------------------\n\nPython provides a flexible way to handle functions that need to accept a variable\nnumber of arguments. Use ``*args`` to collect any number of positional arguments\ninto a tuple, and ``**kwargs`` to collect any number of keyword arguments into a\ndictionary. These are commonly used when writing wrapper functions, decorators,\nor functions that need to pass arguments through to other functions. The names\n``args`` and ``kwargs`` are conventions; you can use any valid identifier after\nthe ``*`` or ``**``.\n\n.. code-block:: python\n\n >>> def example(a, b=None, *args, **kwargs):\n ... print(a, b)\n ... print(args)\n ... print(kwargs)\n ...\n >>> example(1, \"var\", 2, 3, word=\"hello\")\n 1 var\n (2, 3)\n {'word': 'hello'}\n\nUnpack Arguments\n----------------\n\nWhen calling a function, you can use ``*`` to unpack a sequence (like a list or\ntuple) into separate positional arguments, and ``**`` to unpack a dictionary into\nkeyword arguments. This is the inverse of ``*args`` and ``**kwargs`` in function\ndefinitions. Unpacking is particularly useful when you have data in a collection\nthat you want to pass to a function that expects separate arguments.\n\n.. code-block:: python\n\n >>> def foo(a, b, c='BAZ'):\n ... print(a, b, c)\n ...\n >>> foo(*(\"FOO\", \"BAR\"), **{\"c\": \"baz\"})\n FOO BAR baz\n\n >>> args = [1, 2, 3]\n >>> print(*args)\n 1 2 3\n\nKeyword-Only Arguments\n----------------------\n\nArguments that appear after ``*`` or ``*args`` in a function definition are\nkeyword-only, meaning they must be passed by name and cannot be passed positionally.\nThis feature, introduced in Python 3.0, helps prevent errors when functions have\nmany parameters, as it forces callers to be explicit about which argument they're\nproviding. Keyword-only arguments can have default values, making them optional.\n\n**New in Python 3.0**\n\n.. code-block:: python\n\n >>> def f(a, b, *, kw):\n ... print(a, b, kw)\n ...\n >>> f(1, 2, kw=3)\n 1 2 3\n >>> f(1, 2, 3)\n Traceback (most recent call last):\n TypeError: f() takes 2 positional arguments but 3 were given\n\n >>> # keyword-only with default\n >>> def g(a, *, kw=10):\n ... return a + kw\n ...\n >>> g(5)\n 15\n\nPositional-Only Arguments\n-------------------------\n\nArguments that appear before ``/`` in a function definition are positional-only,\nmeaning they cannot be passed by keyword name. This feature, introduced in Python\n3.8, is useful when parameter names are not meaningful to callers or when you want\nto reserve the flexibility to change parameter names without breaking existing code.\nMany built-in functions like ``len()`` and ``pow()`` use positional-only parameters.\nYou can combine positional-only (``/``) and keyword-only (``*``) in the same function.\n\n**New in Python 3.8**\n\n.. code-block:: python\n\n >>> def f(a, b, /, c):\n ... print(a, b, c)\n ...\n >>> f(1, 2, 3)\n 1 2 3\n >>> f(1, 2, c=3)\n 1 2 3\n >>> f(a=1, b=2, c=3)\n Traceback (most recent call last):\n TypeError: f() got some positional-only arguments passed as keyword arguments\n\n >>> # combining positional-only and keyword-only\n >>> def g(a, /, b, *, c):\n ... return a + b + c\n ...\n >>> g(1, 2, c=3)\n 6\n\nAnnotations\n-----------\n\nFunction annotations provide a way to attach metadata to function parameters and\nreturn values. While Python doesn't enforce these annotations at runtime, they\nserve as documentation and are used by static type checkers like ``mypy`` to catch\ntype errors before code runs. Annotations are stored in the function's ``__annotations__``\nattribute as a dictionary. The ``typing`` module (Python 3.5+) provides additional\ntypes like ``List``, ``Dict``, ``Optional``, and ``Union`` for more expressive type hints.\n\n**New in Python 3.0**\n\n.. code-block:: python\n\n >>> def fib(n: int) -> int:\n ... a, b = 0, 1\n ... for _ in range(n):\n ... b, a = a + b, b\n ... return a\n ...\n >>> fib(10)\n 55\n >>> fib.__annotations__\n {'n': , 'return': }\n\nLambda\n------\n\nLambda expressions create small anonymous functions inline. They are syntactically\nrestricted to a single expression, which is implicitly returned. Lambdas are useful\nfor short, throwaway functions, especially as arguments to higher-order functions\nlike ``sorted()``, ``map()``, ``filter()``, and ``reduce()``. While lambdas can make\ncode more concise, complex logic should be written as regular named functions for\nbetter readability and debugging.\n\n.. code-block:: python\n\n >>> square = lambda x: x ** 2\n >>> square(5)\n 25\n\n >>> # lambda with multiple arguments\n >>> add = lambda a, b: a + b\n >>> add(2, 3)\n 5\n\n >>> # lambda with conditional\n >>> max_val = lambda a, b: a if a > b else b\n >>> max_val(3, 5)\n 5\n\n >>> # common use: sorting key\n >>> pairs = [(1, 'b'), (2, 'a'), (3, 'c')]\n >>> sorted(pairs, key=lambda x: x[1])\n [(2, 'a'), (1, 'b'), (3, 'c')]\n\nCallable\n--------\n\nIn Python, any object that implements the ``__call__`` method is callable, meaning\nit can be invoked like a function using parentheses. This includes functions, methods,\nlambdas, classes (calling a class creates an instance), and instances of classes that\ndefine ``__call__``. The built-in ``callable()`` function returns ``True`` if an object\nappears callable, which is useful for checking before attempting to call an object\nto avoid ``TypeError`` exceptions.\n\n.. code-block:: python\n\n >>> callable(print)\n True\n >>> callable(42)\n False\n\n >>> class Adder:\n ... def __init__(self, n):\n ... self.n = n\n ... def __call__(self, x):\n ... return self.n + x\n ...\n >>> add_five = Adder(5)\n >>> callable(add_five)\n True\n >>> add_five(10)\n 15\n\nGet Function Name\n-----------------\n\nFunctions in Python are first-class objects with various attributes that provide\nmetadata about them. The ``__name__`` attribute contains the function's name as\ndefined, ``__doc__`` contains the docstring, ``__module__`` indicates which module\nthe function was defined in, and ``__annotations__`` holds type hints. These\nattributes are useful for debugging, logging, and introspection.\n\n.. code-block:: python\n\n >>> def example_function():\n ... \"\"\"Example docstring.\"\"\"\n ... pass\n ...\n >>> example_function.__name__\n 'example_function'\n >>> example_function.__doc__\n 'Example docstring.'\n >>> example_function.__module__\n '__main__'\n\nClosure\n-------\n\nA closure is a function that captures and remembers values from its enclosing\nlexical scope even after that scope has finished executing. This happens when\na nested function references variables from its outer function. Closures are\npowerful for creating function factories (functions that return customized\nfunctions), implementing decorators, and maintaining state without using global\nvariables or classes. Use the ``nonlocal`` keyword to modify captured variables\nfrom the enclosing scope.\n\n.. code-block:: python\n\n >>> def make_multiplier(n):\n ... def multiplier(x):\n ... return x * n\n ... return multiplier\n ...\n >>> double = make_multiplier(2)\n >>> triple = make_multiplier(3)\n >>> double(5)\n 10\n >>> triple(5)\n 15\n\n >>> # closure with mutable state\n >>> def make_counter():\n ... count = 0\n ... def counter():\n ... nonlocal count\n ... count += 1\n ... return count\n ... return counter\n ...\n >>> counter = make_counter()\n >>> counter()\n 1\n >>> counter()\n 2\n\nGenerator\n---------\n\nGenerator functions use the ``yield`` statement to produce a sequence of values\nlazily, one at a time, instead of computing all values upfront and storing them\nin memory. When called, a generator function returns a generator iterator that\ncan be iterated over with ``for`` loops or ``next()``. Generators are memory-efficient\nfor large sequences and can represent infinite sequences. Generator expressions\nprovide a concise syntax similar to list comprehensions but with lazy evaluation.\n\n.. code-block:: python\n\n >>> def fib(n):\n ... a, b = 0, 1\n ... for _ in range(n):\n ... yield a\n ... b, a = a + b, b\n ...\n >>> list(fib(10))\n [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]\n\n >>> # generator expression\n >>> squares = (x**2 for x in range(5))\n >>> list(squares)\n [0, 1, 4, 9, 16]\n\nDecorator\n---------\n\nDecorators are a powerful pattern for modifying or extending the behavior of\nfunctions without changing their source code. A decorator is a function that\ntakes a function as input and returns a new function (usually a wrapper) that\nadds some functionality before or after calling the original. The ``@decorator``\nsyntax is syntactic sugar for ``func = decorator(func)``. Always use ``@wraps``\nfrom ``functools`` in your wrapper function to preserve the original function's\nmetadata like ``__name__``, ``__doc__``, and ``__annotations__``.\n\n**New in Python 2.4** - PEP `318 `_\n\n.. code-block:: python\n\n >>> from functools import wraps\n >>> def log_calls(func):\n ... @wraps(func)\n ... def wrapper(*args, **kwargs):\n ... print(f\"Calling {func.__name__}\")\n ... return func(*args, **kwargs)\n ... return wrapper\n ...\n >>> @log_calls\n ... def greet(name):\n ... return f\"Hello, {name}!\"\n ...\n >>> greet(\"Alice\")\n Calling greet\n 'Hello, Alice!'\n\n >>> # equivalent to:\n >>> # greet = log_calls(greet)\n\n.. note::\n\n Always use ``@wraps(func)`` in decorators to preserve the original function's\n ``__name__``, ``__doc__``, and other attributes. Without it, the decorated\n function will have the wrapper's attributes, which makes debugging harder.\n\nDecorator with Arguments\n------------------------\n\nTo create a decorator that accepts arguments, you need an extra layer of nesting.\nThe outermost function takes the decorator's arguments and returns the actual\ndecorator. The middle function takes the function being decorated and returns\nthe wrapper. The innermost function is the wrapper that executes when the decorated\nfunction is called. This pattern is commonly used for decorators like ``@repeat(3)``\nor ``@route('/path')``.\n\n.. code-block:: python\n\n >>> from functools import wraps\n >>> def repeat(times):\n ... def decorator(func):\n ... @wraps(func)\n ... def wrapper(*args, **kwargs):\n ... for _ in range(times):\n ... result = func(*args, **kwargs)\n ... return result\n ... return wrapper\n ... return decorator\n ...\n >>> @repeat(3)\n ... def say_hello():\n ... print(\"Hello!\")\n ...\n >>> say_hello()\n Hello!\n Hello!\n Hello!\n\n >>> # equivalent to:\n >>> # say_hello = repeat(3)(say_hello)\n\nClass Decorator\n---------------\n\nDecorators can also be implemented as classes instead of functions. A class-based\ndecorator implements ``__init__`` to receive the decorated function and ``__call__``\nto act as the wrapper. This approach is useful when the decorator needs to maintain\nstate across multiple calls to the decorated function, such as counting calls,\ncaching results, or tracking timing information.\n\n.. code-block:: python\n\n >>> class CountCalls:\n ... def __init__(self, func):\n ... self.func = func\n ... self.count = 0\n ... def __call__(self, *args, **kwargs):\n ... self.count += 1\n ... return self.func(*args, **kwargs)\n ...\n >>> @CountCalls\n ... def example():\n ... return \"result\"\n ...\n >>> example()\n 'result'\n >>> example()\n 'result'\n >>> example.count\n 2\n\nCache with ``lru_cache``\n------------------------\n\nThe ``lru_cache`` decorator from ``functools`` automatically caches function results\nbased on the arguments passed. When the function is called with the same arguments\nagain, the cached result is returned instead of recomputing it. This is especially\nuseful for expensive computations or recursive functions like Fibonacci. The ``maxsize``\nparameter limits cache size (use ``None`` for unlimited). Use ``cache_info()`` to\nsee hit/miss statistics and ``cache_clear()`` to reset the cache.\n\n**New in Python 3.2**\n\n.. code-block:: python\n\n >>> from functools import lru_cache\n >>> @lru_cache(maxsize=None)\n ... def fib(n):\n ... if n < 2:\n ... return n\n ... return fib(n - 1) + fib(n - 2)\n ...\n >>> fib(100)\n 354224848179261915075\n >>> fib.cache_info()\n CacheInfo(hits=98, misses=101, maxsize=None, currsize=101)\n >>> fib.cache_clear() # clear the cache\n\n**New in Python 3.9** - ``@cache`` is a simpler alias for ``@lru_cache(maxsize=None)``\n\n.. code-block:: python\n\n >>> from functools import cache\n >>> @cache\n ... def factorial(n):\n ... return n * factorial(n-1) if n else 1\n\nPartial Functions\n-----------------\n\nThe ``functools.partial`` function creates a new callable with some arguments of\nthe original function pre-filled. This is useful for adapting functions to interfaces\nthat expect fewer arguments, creating specialized versions of general functions,\nor preparing callback functions. The resulting partial object can be called with\nthe remaining arguments. You can pre-fill both positional and keyword arguments.\n\n.. code-block:: python\n\n >>> from functools import partial\n >>> def power(base, exponent):\n ... return base ** exponent\n ...\n >>> square = partial(power, exponent=2)\n >>> cube = partial(power, exponent=3)\n >>> square(5)\n 25\n >>> cube(5)\n 125\n\n >>> # useful for callbacks\n >>> from functools import partial\n >>> def greet(greeting, name):\n ... return f\"{greeting}, {name}!\"\n ...\n >>> say_hello = partial(greet, \"Hello\")\n >>> say_hello(\"Alice\")\n 'Hello, Alice!'\n\n``singledispatch`` - Function Overloading\n-----------------------------------------\n\nThe ``singledispatch`` decorator from ``functools`` enables function overloading\nbased on the type of the first argument. You define a base function and then\nregister specialized implementations for different types using the ``@func.register``\ndecorator. When the function is called, Python automatically dispatches to the\nappropriate implementation based on the argument's type. This is useful for writing\ngeneric functions that behave differently for different types.\n\n**New in Python 3.4**\n\n.. code-block:: python\n\n >>> from functools import singledispatch\n >>> @singledispatch\n ... def process(arg):\n ... return f\"Default: {arg}\"\n ...\n >>> @process.register(int)\n ... def _(arg):\n ... return f\"Integer: {arg * 2}\"\n ...\n >>> @process.register(list)\n ... def _(arg):\n ... return f\"List with {len(arg)} items\"\n ...\n >>> process(\"hello\")\n 'Default: hello'\n >>> process(5)\n 'Integer: 10'\n >>> process([1, 2, 3])\n 'List with 3 items'\n\n``reduce`` - Cumulative Operations\n----------------------------------\n\nThe ``reduce`` function from ``functools`` applies a two-argument function\ncumulatively to the items of a sequence, from left to right, reducing the sequence\nto a single value. For example, ``reduce(f, [a, b, c, d])`` computes ``f(f(f(a, b), c), d)``.\nAn optional third argument provides an initial value. While ``reduce`` can be powerful,\nlist comprehensions or explicit loops are often more readable for simple cases.\n\n.. code-block:: python\n\n >>> from functools import reduce\n >>> # sum of list\n >>> reduce(lambda x, y: x + y, [1, 2, 3, 4, 5])\n 15\n\n >>> # product of list\n >>> reduce(lambda x, y: x * y, [1, 2, 3, 4, 5])\n 120\n\n >>> # with initial value\n >>> reduce(lambda x, y: x + y, [1, 2, 3], 10)\n 16\n\nHigher-Order Functions\n----------------------\n\nHigher-order functions are functions that take other functions as arguments or\nreturn functions as results. Python provides several built-in higher-order functions\nthat are commonly used for functional programming patterns. ``map()`` applies a\nfunction to every item in an iterable, ``filter()`` keeps items where the function\nreturns ``True``, and ``sorted()``/``min()``/``max()`` accept a ``key`` function\nto customize comparison. These functions return iterators (except ``sorted``),\nso wrap them in ``list()`` if you need a list.\n\n.. code-block:: python\n\n >>> # map - apply function to each item\n >>> list(map(lambda x: x**2, [1, 2, 3, 4]))\n [1, 4, 9, 16]\n\n >>> # filter - keep items where function returns True\n >>> list(filter(lambda x: x > 2, [1, 2, 3, 4]))\n [3, 4]\n\n >>> # sorted with key function\n >>> sorted(['banana', 'apple', 'cherry'], key=len)\n ['apple', 'banana', 'cherry']\n\n >>> # min/max with key function\n >>> max(['apple', 'banana', 'cherry'], key=len)\n 'banana'\n"
},
{
"path": "docs/notes/basic/python-future.rst",
"content": ".. meta::\n :description lang=en: Python __future__ module guide covering future statements, backward compatibility, and feature backporting from newer Python versions\n :keywords: Python, __future__, future statements, backward compatibility, print_function, annotations, division\n\n======\nFuture\n======\n\n.. contents:: Table of Contents\n :backlinks: none\n\n\n`Future statements `_\ntell the interpreter to compile some semantics as the semantics which will be\navailable in the future Python version. In other words, Python uses ``from __future__ import feature``\nto backport features from other higher Python versions to the current interpreter.\nIn Python 3, many features such as ``print_function`` are already enabled, but\nwe still leave these future statements for backward compatibility.\n\nFuture statements are **NOT** import statements. Future statements change how\nPython interprets the code. They **MUST** be at the top of the file. Otherwise,\nPython interpreter will raise ``SyntaxError``.\n\nIf you're interested in future statements and want to acquire more explanation,\nfurther information can be found on `PEP 236 - Back to the __future__ `_\n\nList All New Features\n---------------------\n\n`__future__ `_ is a Python\nmodule. We can use it to check what kind of future features can import to\ncurrent Python interpreter. The fun is ``import __future__`` is **NOT** a future\nstatement, it is a import statement.\n\n.. code-block:: python\n\n >>> from pprint import pprint\n >>> import __future__\n >>> pprint(__future__.all_feature_names)\n ['nested_scopes',\n 'generators',\n 'division',\n 'absolute_import',\n 'with_statement',\n 'print_function',\n 'unicode_literals',\n 'barry_as_FLUFL',\n 'generator_stop',\n 'annotations']\n\nFuture statements not only change the behavior of the Python interpreter but\nalso import ``__future__._Feature`` into the current program.\n\n.. code-block:: python\n\n >>> from __future__ import print_function\n >>> print_function\n _Feature((2, 6, 0, 'alpha', 2), (3, 0, 0, 'alpha', 0), 65536)\n\nPrint Function\n--------------\n\nReplacing **print statement** to **print function** is one of the most\nnotorious decision in Python history. However, this change brings some\nflexibilities to extend the ability of ``print``. Further information can\nbe found on PEP `3105 `_.\n\n.. code-block:: python\n\n >>> print \"Hello World\" # print is a statement\n Hello World\n >>> from __future__ import print_function\n >>> print \"Hello World\"\n File \"\", line 1\n print \"Hello World\"\n ^\n SyntaxError: invalid syntax\n >>> print(\"Hello World\") # print become a function\n Hello World\n\nUnicode\n-------\n\nAs **print function**, making text become Unicode is another infamous decision.\nNevertheless, many modern programming languages’ text is Unicode. This change\ncompels us to decode texts early in order to prevent runtime error after we\nrun programs for a while. Further information can be found on PEP\n`3112 `_.\n\n.. code-block:: python\n\n >>> type(\"Guido\") # string type is str in python2\n \n >>> from __future__ import unicode_literals\n >>> type(\"Guido\") # string type become unicode\n \n\nDivision\n--------\n\nSometimes, it is counterintuitive when the division result is int or long.\nIn this case, Python 3 enables the **true division** by default. However, in\nPython 2, we have to backport ``division`` to the current interpreter. Further\ninformation can be found on PEP `238 `_.\n\n.. code-block:: python\n\n >>> 1 / 2\n 0\n >>> from __future__ import division\n >>> 1 / 2 # return a float (classic division)\n 0.5\n >>> 1 // 2 # return a int (floor division)\n 0\n\nAnnotations\n-----------\n\nBefore Python 3.7, we cannot assign annotations in a class or a function if\nit is not available in the current scope. A common situation is the definition\nof a container class.\n\n.. code-block:: python\n\n class Tree(object):\n\n def insert(self, tree: Tree): ...\n\nExample\n\n.. code-block:: bash\n\n $ python3 foo.py\n Traceback (most recent call last):\n File \"foo.py\", line 1, in \n class Tree(object):\n File \"foo.py\", line 3, in Tree\n def insert(self, tree: Tree): ...\n NameError: name 'Tree' is not defined\n\nIn this case, the definition of the class is not available yet. Python interpreter\ncannot parse the annotation during their definition time. To solve this issue,\nPython uses string literals to replace the class.\n\n.. code-block:: python\n\n class Tree(object):\n\n def insert(self, tree: 'Tree'): ...\n\nAfter version 3.7, Python introduces the future statement, ``annotations``, to\nperform postponed evaluation. It will become the default feature in Python 4.\nFor further information please refer to PEP `563 `_.\n\n\n.. code-block:: python\n\n from __future__ import annotations\n\n class Tree(object):\n\n def insert(self, tree: Tree): ...\n\nBDFL Retirement\n---------------\n\n**New in Python 3.1**\n\nPEP `401 `_ is just an Easter egg.\nThis feature brings the current interpreter back to the past. It enables the\ndiamond operator ``<>`` in Python 3.\n\n.. code-block:: python\n\n >>> 1 != 2\n True\n >>> from __future__ import barry_as_FLUFL\n >>> 1 != 2\n File \"\", line 1\n 1 != 2\n ^\n SyntaxError: with Barry as BDFL, use '<>' instead of '!='\n >>> 1 <> 2\n True\n\nBraces\n------\n\n``braces`` is an Easter egg. The source code can be found on\n`future.c `_.\n\n.. code-block:: python\n\n >>> from __future__ import braces\n File \"\", line 1\n SyntaxError: not a chance\n"
},
{
"path": "docs/notes/basic/python-generator.rst",
"content": ".. meta::\n :description lang=en: Python generator cheat sheet covering generator functions, generator expressions, yield, yield from, send, async generators, and coroutines with code examples\n :keywords: Python, Python3, Python generator, Python generator cheat sheet, yield, yield from, generator expression, async generator, iterator, coroutine, contextmanager\n\n=========\nGenerator\n=========\n\n.. contents:: Table of Contents\n :backlinks: none\n\nGenerators are a powerful feature in Python for creating iterators. They allow\nyou to iterate over data without storing the entire sequence in memory, making\nthem ideal for processing large datasets or infinite sequences. This cheat sheet\ncovers generator functions, generator expressions, ``yield``, ``yield from``,\nsending values to generators, and async generators.\n\nGenerator Function vs Generator Expression\n------------------------------------------\n\nA generator function is defined like a normal function but uses ``yield`` to\nproduce a sequence of values. When called, it returns a generator object that\ncan be iterated over. A generator expression is a compact syntax similar to\nlist comprehensions but produces values lazily on demand.\n\n.. code-block:: python\n\n # generator function\n >>> def gen_func():\n ... yield 5566\n ...\n >>> g = gen_func()\n >>> g\n \n >>> next(g)\n 5566\n\n # generator expression\n >>> g = (x for x in range(3))\n >>> next(g)\n 0\n >>> next(g)\n 1\n\nYield Values from Generator\n---------------------------\n\nThe ``yield`` statement produces a value and suspends the generator's execution.\nWhen ``next()`` is called again, execution resumes from where it left off. This\nexample generates prime numbers by checking divisibility for each candidate.\n\n.. code-block:: python\n\n >>> def prime(n):\n ... p = 2\n ... while n > 0:\n ... for x in range(2, p):\n ... if p % x == 0:\n ... break\n ... else:\n ... yield p\n ... n -= 1\n ... p += 1\n ...\n >>> list(prime(5))\n [2, 3, 5, 7, 11]\n\nUnpack Generators\n-----------------\n\nPython 3.5+ (PEP 448) allows unpacking generators directly into lists, sets,\nfunction arguments, and variables using the ``*`` operator. This provides a\nconvenient way to consume generator values without explicit iteration.\n\n.. code-block:: python\n\n # PEP 448 - unpacking inside a list\n >>> g1 = (x for x in range(3))\n >>> g2 = (x**2 for x in range(2))\n >>> [1, *g1, 2, *g2]\n [1, 0, 1, 2, 2, 0, 1]\n\n # unpacking inside a set\n >>> g = (x for x in [5, 5, 6, 6])\n >>> {*g}\n {5, 6}\n\n # unpacking to variables\n >>> g = (x for x in range(3))\n >>> a, b, c = g\n >>> a, b, c\n (0, 1, 2)\n\n # extended unpacking\n >>> g = (x for x in range(6))\n >>> a, b, *c, d = g\n >>> a, b, d\n (0, 1, 5)\n >>> c\n [2, 3, 4]\n\n # unpacking inside a function\n >>> print(*(x for x in range(3)))\n 0 1 2\n\nIterable Class via Generator\n----------------------------\n\nYou can make a class iterable by implementing ``__iter__`` as a generator method.\nThis approach is cleaner than implementing a separate iterator class. The\n``__reversed__`` method can also be implemented as a generator to support the\nbuilt-in ``reversed()`` function.\n\n.. code-block:: python\n\n >>> class Count:\n ... def __init__(self, n):\n ... self._n = n\n ... def __iter__(self):\n ... n = self._n\n ... while n > 0:\n ... yield n\n ... n -= 1\n ... def __reversed__(self):\n ... n = 1\n ... while n <= self._n:\n ... yield n\n ... n += 1\n ...\n >>> list(Count(5))\n [5, 4, 3, 2, 1]\n >>> list(reversed(Count(5)))\n [1, 2, 3, 4, 5]\n\nSend Values to Generator\n------------------------\n\nGenerators can receive values through the ``send()`` method. The sent value\nbecomes the result of the ``yield`` expression inside the generator. Before\nsending values, you must start the generator by calling ``next()`` or\n``send(None)`` to advance it to the first ``yield``.\n\n.. code-block:: python\n\n >>> def spam():\n ... msg = yield\n ... print(\"Message:\", msg)\n ...\n >>> g = spam()\n >>> next(g) # start generator\n >>> try:\n ... g.send(\"Hello World!\")\n ... except StopIteration:\n ... pass\n Message: Hello World!\n\nyield from Expression\n---------------------\n\nThe ``yield from`` expression delegates iteration to another generator or\niterable. It automatically handles forwarding ``send()``, ``throw()``, and\n``close()`` calls to the subgenerator, making it ideal for creating generator\npipelines and recursive generators.\n\n.. code-block:: python\n\n >>> def subgen():\n ... try:\n ... yield 9527\n ... except ValueError:\n ... print(\"got ValueError\")\n ...\n >>> def delegating_gen():\n ... yield from subgen()\n ...\n >>> g = delegating_gen()\n >>> next(g)\n 9527\n >>> try:\n ... g.throw(ValueError)\n ... except StopIteration:\n ... pass\n got ValueError\n\nYou can chain multiple ``yield from`` expressions together. The\n``inspect.getgeneratorstate()`` function helps track the generator's lifecycle\nthrough its states: GEN_CREATED, GEN_RUNNING, GEN_SUSPENDED, and GEN_CLOSED.\n\n.. code-block:: python\n\n # yield from + yield from\n >>> import inspect\n >>> def subgen():\n ... yield from range(3)\n ...\n >>> def delegating_gen():\n ... yield from subgen()\n ...\n >>> g = delegating_gen()\n >>> inspect.getgeneratorstate(g)\n 'GEN_CREATED'\n >>> next(g)\n 0\n >>> inspect.getgeneratorstate(g)\n 'GEN_SUSPENDED'\n >>> g.close()\n >>> inspect.getgeneratorstate(g)\n 'GEN_CLOSED'\n\nyield from with Return\n----------------------\n\nGenerators can return a value using the ``return`` statement. The returned value\nis accessible through the ``value`` attribute of the ``StopIteration`` exception.\nWhen using ``yield from``, the return value of the subgenerator becomes the value\nof the ``yield from`` expression.\n\n.. code-block:: python\n\n >>> def average():\n ... total = .0\n ... count = 0\n ... while True:\n ... val = yield\n ... if not val:\n ... break\n ... total += val\n ... count += 1\n ... return total / count\n ...\n >>> g = average()\n >>> next(g)\n >>> g.send(3)\n >>> g.send(5)\n >>> try:\n ... g.send(None)\n ... except StopIteration as e:\n ... print(e.value)\n 4.0\n\n.. code-block:: python\n\n >>> def subgen():\n ... yield 9527\n ...\n >>> def delegating_gen():\n ... yield from subgen()\n ... return 5566\n ...\n >>> g = delegating_gen()\n >>> next(g)\n 9527\n >>> try:\n ... next(g)\n ... except StopIteration as e:\n ... print(e.value)\n 5566\n\nGenerate Sequences\n------------------\n\nThe ``yield from`` expression provides a concise way to yield all values from\nan iterable. This is particularly useful for chaining multiple sequences together\nor flattening nested structures.\n\n.. code-block:: python\n\n >>> def chain():\n ... yield from 'ab'\n ... yield from range(3)\n ...\n >>> list(chain())\n ['a', 'b', 0, 1, 2]\n\nWhat ``RES = yield from EXP`` Does\n----------------------------------\n\nThis snippet shows the simplified equivalent of what ``yield from`` does\ninternally, as described in PEP 380. It handles iteration, value passing via\n``send()``, and captures the return value from the subgenerator.\n\n.. code-block:: python\n\n # Simplified version (ref: PEP 380)\n >>> def subgen():\n ... for x in range(3):\n ... yield x\n ...\n >>> def delegating_gen():\n ... _i = iter(subgen())\n ... try:\n ... _y = next(_i)\n ... except StopIteration as _e:\n ... RES = _e.value\n ... else:\n ... while True:\n ... _s = yield _y\n ... try:\n ... _y = _i.send(_s)\n ... except StopIteration as _e:\n ... RES = _e.value\n ... break\n ...\n >>> list(delegating_gen())\n [0, 1, 2]\n\nCheck Generator Type\n--------------------\n\nUse ``types.GeneratorType`` to check if an object is a generator. This is useful\nfor writing functions that need to handle generators differently from other\niterables.\n\n.. code-block:: python\n\n >>> from types import GeneratorType\n >>> def gen_func():\n ... yield 5566\n ...\n >>> isinstance(gen_func(), GeneratorType)\n True\n\nCheck Generator State\n---------------------\n\nThe ``inspect.getgeneratorstate()`` function returns the current state of a\ngenerator. This is helpful for debugging and understanding the generator lifecycle.\nThe four possible states are: GEN_CREATED (not started), GEN_RUNNING (currently\nexecuting), GEN_SUSPENDED (paused at yield), and GEN_CLOSED (completed or closed).\n\n.. code-block:: python\n\n >>> import inspect\n >>> def gen_func():\n ... yield 9527\n ...\n >>> g = gen_func()\n >>> inspect.getgeneratorstate(g)\n 'GEN_CREATED'\n >>> next(g)\n 9527\n >>> inspect.getgeneratorstate(g)\n 'GEN_SUSPENDED'\n >>> g.close()\n >>> inspect.getgeneratorstate(g)\n 'GEN_CLOSED'\n\nContext Manager via Generator\n-----------------------------\n\nThe ``@contextlib.contextmanager`` decorator transforms a generator function into\na context manager. Code before ``yield`` runs on entering the ``with`` block,\nand code after ``yield`` (typically in ``finally``) runs on exit. The yielded\nvalue is bound to the variable after ``as``.\n\n.. code-block:: python\n\n >>> import contextlib\n >>> @contextlib.contextmanager\n ... def mylist():\n ... try:\n ... l = [1, 2, 3, 4, 5]\n ... yield l\n ... finally:\n ... print(\"exit scope\")\n ...\n >>> with mylist() as l:\n ... print(l)\n [1, 2, 3, 4, 5]\n exit scope\n\nWhat ``@contextmanager`` Does\n-----------------------------\n\nThis snippet shows a simplified implementation of how ``@contextmanager`` works\ninternally. It wraps a generator in a class that implements the context manager\nprotocol (``__enter__`` and ``__exit__``), handling both normal exit and\nexception propagation.\n\n.. code-block:: python\n\n class GeneratorCM:\n def __init__(self, gen):\n self._gen = gen\n\n def __enter__(self):\n return next(self._gen)\n\n def __exit__(self, *exc_info):\n try:\n if exc_info[0] is None:\n next(self._gen)\n else:\n self._gen.throw(*exc_info)\n except StopIteration:\n return True\n raise\n\n def contextmanager(func):\n def run(*a, **k):\n return GeneratorCM(func(*a, **k))\n return run\n\nProfile Code Block\n------------------\n\nA practical example of using generator-based context managers to measure\nexecution time of code blocks. The ``yield`` statement marks the boundary\nbetween setup (recording start time) and teardown (calculating elapsed time).\n\n.. code-block:: python\n\n >>> import time\n >>> from contextlib import contextmanager\n >>> @contextmanager\n ... def profile(msg):\n ... try:\n ... s = time.time()\n ... yield\n ... finally:\n ... print(f'{msg} cost: {time.time() - s:.2f}s')\n ...\n >>> with profile('block'):\n ... time.sleep(0.1)\n block cost: 0.10s\n\n``yield from`` and ``__iter__``\n-------------------------------\n\nWhen using ``yield from`` with a class instance, Python calls the object's\n``__iter__`` method to get an iterator. This allows custom classes to work\nseamlessly with ``yield from`` delegation, enabling elegant composition of\niterables.\n\n.. code-block:: python\n\n >>> class FakeGen:\n ... def __iter__(self):\n ... n = 0\n ... while n < 3:\n ... yield n\n ... n += 1\n ... def __reversed__(self):\n ... n = 2\n ... while n >= 0:\n ... yield n\n ... n -= 1\n ...\n >>> def spam():\n ... yield from FakeGen()\n ...\n >>> list(spam())\n [0, 1, 2]\n >>> list(reversed(FakeGen()))\n [2, 1, 0]\n\nClosure Using Generator\n-----------------------\n\nGenerators provide an elegant way to implement closures that maintain state\nbetween calls. Each call to ``next()`` resumes execution and can access and\nmodify the enclosed variables. This is often cleaner than using ``nonlocal``\nor class-based approaches.\n\n.. code-block:: python\n\n # generator version\n >>> def closure_gen():\n ... x = 5566\n ... while True:\n ... x += 1\n ... yield x\n ...\n >>> g = closure_gen()\n >>> next(g)\n 5567\n >>> next(g)\n 5568\n\nSimple Scheduler\n----------------\n\nThis example demonstrates how generators can be used to implement cooperative\nmultitasking. Each generator represents a task that yields control back to the\nscheduler. The scheduler uses a deque to round-robin between tasks, advancing\neach one step at a time.\n\n.. code-block:: python\n\n >>> from collections import deque\n >>> def fib(n):\n ... if n <= 2: return 1\n ... return fib(n-1) + fib(n-2)\n ...\n >>> def g_fib(n):\n ... for x in range(1, n + 1):\n ... yield fib(x)\n ...\n >>> q = deque([g_fib(3), g_fib(5)])\n >>> def run():\n ... while q:\n ... try:\n ... t = q.popleft()\n ... print(next(t))\n ... q.append(t)\n ... except StopIteration:\n ... print(\"Task done\")\n ...\n >>> run()\n 1\n 1\n 1\n 1\n 2\n 2\n Task done\n 3\n 5\n Task done\n\nSimple Round-Robin with Blocking\n--------------------------------\n\nA more advanced scheduler that handles I/O blocking using ``select()``. Tasks\nyield tuples indicating what operation they're waiting for ('recv' or 'send')\nand which socket. The scheduler moves blocked tasks to wait queues and only\nruns them when their I/O is ready. This is the foundation of async I/O frameworks.\n\n.. code-block:: python\n\n from collections import deque\n from select import select\n import socket\n\n tasks = deque()\n w_read = {}\n w_send = {}\n\n def run():\n while any([tasks, w_read, w_send]):\n while not tasks:\n can_r, can_s, _ = select(w_read, w_send, [])\n for _r in can_r:\n tasks.append(w_read.pop(_r))\n for _w in can_s:\n tasks.append(w_send.pop(_w))\n try:\n task = tasks.popleft()\n why, what = next(task)\n if why == 'recv':\n w_read[what] = task\n elif why == 'send':\n w_send[what] = task\n except StopIteration:\n pass\n\n def server():\n sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n sock.bind(('localhost', 5566))\n sock.listen(5)\n while True:\n yield 'recv', sock\n conn, addr = sock.accept()\n tasks.append(client_handler(conn))\n\n def client_handler(conn):\n while True:\n yield 'recv', conn\n msg = conn.recv(1024)\n if not msg: break\n yield 'send', conn\n conn.send(msg)\n conn.close()\n\n tasks.append(server())\n run()\n\nAsync Generator (Python 3.6+)\n-----------------------------\n\nAsync generators combine ``async def`` with ``yield`` to create asynchronous\niterators. They can use ``await`` to pause for async operations between yields.\nUse ``async for`` to iterate over async generators. This is essential for\nstreaming data from async sources like network connections or databases.\n\n.. code-block:: python\n\n >>> import asyncio\n >>> async def slow_gen(n, t):\n ... for x in range(n):\n ... await asyncio.sleep(t)\n ... yield x\n ...\n >>> async def task(n):\n ... async for x in slow_gen(n, 0.1):\n ... print(x)\n ...\n >>> asyncio.run(task(3))\n 0\n 1\n 2\n\nAsync Generator with try..finally\n---------------------------------\n\nAsync generators support ``try..finally`` blocks for cleanup, just like regular\ngenerators. The ``finally`` block executes when the generator is closed or\ngarbage collected, ensuring resources are properly released even if an exception\noccurs during iteration.\n\n.. code-block:: python\n\n >>> import asyncio\n >>> async def agen(t):\n ... try:\n ... await asyncio.sleep(t)\n ... yield 1 / 0\n ... finally:\n ... print(\"finally\")\n ...\n >>> async def main():\n ... try:\n ... g = agen(0.1)\n ... await g.__anext__()\n ... except Exception as e:\n ... print(repr(e))\n ...\n >>> asyncio.run(main())\n finally\n ZeroDivisionError('division by zero')\n\nSend and Throw to Async Generator\n---------------------------------\n\nAsync generators support ``asend()`` to send values and ``athrow()`` to throw\nexceptions, similar to regular generators. These methods are coroutines that\nmust be awaited. This enables two-way communication with async generators for\nbuilding complex async data pipelines.\n\n.. code-block:: python\n\n >>> import asyncio\n >>> async def agen(n):\n ... try:\n ... for x in range(n):\n ... await asyncio.sleep(0.1)\n ... val = yield x\n ... print(f'got: {val}')\n ... except RuntimeError as e:\n ... yield repr(e)\n ...\n >>> async def main():\n ... g = agen(5)\n ... ret = await g.asend(None) + await g.asend('foo')\n ... print(ret)\n ... ret = await g.athrow(RuntimeError('error'))\n ... print(ret)\n ...\n >>> asyncio.run(main())\n got: foo\n 1\n RuntimeError('error')\n\nAsync Comprehension (Python 3.6+)\n---------------------------------\n\nPEP 530 introduced async comprehensions, allowing ``async for`` in list, set,\nand dict comprehensions. This provides a concise way to collect values from\nasync generators. You can also use ``if`` clauses to filter values and\nconditional expressions for transformations.\n\n.. code-block:: python\n\n >>> import asyncio\n >>> async def agen(n):\n ... for x in range(n):\n ... await asyncio.sleep(0.01)\n ... yield x\n ...\n >>> async def main():\n ... ret = [x async for x in agen(5)]\n ... print(ret)\n ... ret = [x async for x in agen(5) if x < 3]\n ... print(ret)\n ... ret = {f'{x}': x async for x in agen(3)}\n ... print(ret)\n ...\n >>> asyncio.run(main())\n [0, 1, 2, 3, 4]\n [0, 1, 2]\n {'0': 0, '1': 1, '2': 2}\n\nSimple Async Round-Robin\n------------------------\n\nThis example shows cooperative multitasking with async generators. Multiple\nasync generators are scheduled in a deque, and the scheduler awaits each one\nin turn using ``__anext__()``. This pattern is useful for interleaving multiple\nasync data streams fairly.\n\n.. code-block:: python\n\n >>> import asyncio\n >>> from collections import deque\n >>> async def agen(n):\n ... for x in range(n):\n ... await asyncio.sleep(0.1)\n ... yield x\n ...\n >>> async def main():\n ... q = deque([agen(3), agen(5)])\n ... while q:\n ... try:\n ... g = q.popleft()\n ... print(await g.__anext__())\n ... q.append(g)\n ... except StopAsyncIteration:\n ... pass\n ...\n >>> asyncio.run(main())\n 0\n 0\n 1\n 1\n 2\n 2\n 3\n 4\n\nAsync Generator vs Async Iterator Performance\n----------------------------------------------\n\nAsync generators have better performance than manually implemented async iterators\nbecause they are optimized at the C level in CPython. This benchmark shows that\nasync generators can be significantly faster for iteration-heavy workloads.\n\n.. code-block:: python\n\n >>> import time\n >>> import asyncio\n >>> class AsyncIter:\n ... def __init__(self, n):\n ... self._n = n\n ... def __aiter__(self):\n ... return self\n ... async def __anext__(self):\n ... ret = self._n\n ... if self._n == 0:\n ... raise StopAsyncIteration\n ... self._n -= 1\n ... return ret\n ...\n >>> async def agen(n):\n ... for i in range(n):\n ... yield i\n ...\n >>> async def task_agen(n):\n ... s = time.time()\n ... async for _ in agen(n): pass\n ... cost = time.time() - s\n ... print(f\"agen cost time: {cost}\")\n ...\n >>> async def task_aiter(n):\n ... s = time.time()\n ... async for _ in AsyncIter(n): pass\n ... cost = time.time() - s\n ... print(f\"aiter cost time: {cost}\")\n ...\n >>> n = 10 ** 7\n >>> asyncio.run(task_agen(n))\n agen cost time: 1.2698817253112793\n >>> asyncio.run(task_aiter(n))\n aiter cost time: 4.168368101119995\n\n``yield from == await`` Expression\n----------------------------------\n\nBefore Python 3.5 introduced ``async``/``await`` syntax, coroutines were\nimplemented using generators with ``@asyncio.coroutine`` decorator and\n``yield from``. The ``await`` keyword is essentially equivalent to ``yield from``\nfor coroutines. This example shows both the old and new syntax for an echo server.\n\n.. code-block:: python\n\n import asyncio\n import socket\n\n loop = asyncio.get_event_loop()\n host = 'localhost'\n port = 5566\n sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)\n sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n sock.setblocking(False)\n sock.bind((host, port))\n sock.listen(10)\n\n # old syntax (Python 3.4)\n @asyncio.coroutine\n def echo_server():\n while True:\n conn, addr = yield from loop.sock_accept(sock)\n loop.create_task(handler(conn))\n\n @asyncio.coroutine\n def handler(conn):\n while True:\n msg = yield from loop.sock_recv(conn, 1024)\n if not msg:\n break\n yield from loop.sock_sendall(conn, msg)\n conn.close()\n\n # new syntax (Python 3.5+)\n async def echo_server():\n while True:\n conn, addr = await loop.sock_accept(sock)\n loop.create_task(handler(conn))\n\n async def handler(conn):\n while True:\n msg = await loop.sock_recv(conn, 1024)\n if not msg:\n break\n await loop.sock_sendall(conn, msg)\n conn.close()\n\n loop.create_task(echo_server())\n loop.run_forever()\n\nSimple Compiler Using Generators\n--------------------------------\n\nThis advanced example from David Beazley demonstrates using generators to\nimplement a simple expression compiler. It includes a tokenizer, parser, and\nevaluator using the visitor pattern with generators for stack-based evaluation.\n\n.. code-block:: python\n\n import re\n import types\n from collections import namedtuple\n\n tokens = [\n r'(?P\\d+)',\n r'(?P\\+)',\n r'(?P-)',\n r'(?P\\*)',\n r'(?P/)',\n r'(?P\\s+)']\n\n Token = namedtuple('Token', ['type', 'value'])\n lex = re.compile('|'.join(tokens))\n\n def tokenize(text):\n scan = lex.scanner(text)\n gen = (Token(m.lastgroup, m.group())\n for m in iter(scan.match, None) if m.lastgroup != 'WS')\n return gen\n\n class Node:\n _fields = []\n def __init__(self, *args):\n for attr, value in zip(self._fields, args):\n setattr(self, attr, value)\n\n class Number(Node):\n _fields = ['value']\n\n class BinOp(Node):\n _fields = ['op', 'left', 'right']\n\n def parse(toks):\n lookahead, current = next(toks, None), None\n\n def accept(*toktypes):\n nonlocal lookahead, current\n if lookahead and lookahead.type in toktypes:\n current, lookahead = lookahead, next(toks, None)\n return True\n\n def expr():\n left = term()\n while accept('PLUS', 'MINUS'):\n left = BinOp(current.value, left)\n left.right = term()\n return left\n\n def term():\n left = factor()\n while accept('TIMES', 'DIVIDE'):\n left = BinOp(current.value, left)\n left.right = factor()\n return left\n\n def factor():\n if accept('NUMBER'):\n return Number(int(current.value))\n else:\n raise SyntaxError()\n return expr()\n\n class NodeVisitor:\n def visit(self, node):\n stack = [self.genvisit(node)]\n ret = None\n while stack:\n try:\n node = stack[-1].send(ret)\n stack.append(self.genvisit(node))\n ret = None\n except StopIteration as e:\n stack.pop()\n ret = e.value\n return ret\n\n def genvisit(self, node):\n ret = getattr(self, 'visit_' + type(node).__name__)(node)\n if isinstance(ret, types.GeneratorType):\n ret = yield from ret\n return ret\n\n class Evaluator(NodeVisitor):\n def visit_Number(self, node):\n return node.value\n\n def visit_BinOp(self, node):\n leftval = yield node.left\n rightval = yield node.right\n if node.op == '+':\n return leftval + rightval\n elif node.op == '-':\n return leftval - rightval\n elif node.op == '*':\n return leftval * rightval\n elif node.op == '/':\n return leftval / rightval\n\n def evaluate(exp):\n toks = tokenize(exp)\n tree = parse(toks)\n return Evaluator().visit(tree)\n\n print(evaluate('2 * 3 + 5 / 2')) # 8.5\n print(evaluate('+'.join([str(x) for x in range(10000)]))) # 49995000\n"
},
{
"path": "docs/notes/basic/python-heap.rst",
"content": ".. meta::\n :description lang=en: Python heap and priority queue cheat sheet covering heapq module operations, heap sort algorithm, priority queue implementation with custom comparators, and practical examples\n :keywords: Python, Python Cheat Sheet, heap, heapq, priority queue, heap sort, min heap, max heap, Python heapq, nlargest, nsmallest\n\n====\nHeap\n====\n\n.. contents:: Table of Contents\n :backlinks: none\n\nThe heapq module provides an implementation of the heap queue algorithm, also\nknown as the priority queue algorithm. Heaps are binary trees where every parent\nnode has a value less than or equal to any of its children (min-heap). This\ncheat sheet covers heap operations including heap sort, priority queues, merging\nsorted iterables, and finding the n largest or smallest elements efficiently.\n\nThe source code is available on `GitHub `_.\n\nReferences\n----------\n\n- `heapq — Heap queue algorithm `_\n- `queue.PriorityQueue `_\n\nBasic Heap Operations\n---------------------\n\nThe ``heapq`` module provides functions to create and manipulate heaps. Use\n``heapify`` to convert a list into a heap in-place in O(n) time. Use ``heappush``\nand ``heappop`` to add and remove elements while maintaining the heap property.\n\n.. code-block:: python\n\n >>> import heapq\n >>> # Convert list to heap in-place\n >>> h = [5, 1, 3, 2, 6]\n >>> heapq.heapify(h)\n >>> h[0] # smallest element at root\n 1\n >>> # Push and pop\n >>> heapq.heappush(h, 0)\n >>> heapq.heappop(h)\n 0\n >>> # Push and pop in one operation\n >>> heapq.heappushpop(h, 4) # push 4, then pop smallest\n 1\n >>> # Pop and push in one operation\n >>> heapq.heapreplace(h, 0) # pop smallest, then push 0\n 2\n\nImplement Heap Sort with ``heapq``\n----------------------------------\n\nHeap sort works by pushing all elements onto a heap and then popping them off\none by one. Since the heap maintains the min-heap property, elements come out\nin sorted order. The time complexity is O(n log n).\n\n.. code-block:: python\n\n >>> import heapq\n >>> a = [5, 1, 3, 2, 6]\n >>> h = []\n >>> for x in a:\n ... heapq.heappush(h, x)\n ...\n >>> x = [heapq.heappop(h) for _ in range(len(a))]\n >>> x\n [1, 2, 3, 5, 6]\n\nA more efficient approach uses ``heapify`` to convert the list in-place:\n\n.. code-block:: python\n\n >>> import heapq\n >>> def heap_sort(items):\n ... h = items.copy()\n ... heapq.heapify(h)\n ... return [heapq.heappop(h) for _ in range(len(h))]\n ...\n >>> heap_sort([5, 1, 3, 2, 6])\n [1, 2, 3, 5, 6]\n\nImplement Max Heap\n------------------\n\nPython's ``heapq`` only provides a min-heap. To implement a max-heap, negate\nthe values when pushing and negate again when popping.\n\n.. code-block:: python\n\n >>> import heapq\n >>> # Max heap using negation\n >>> h = []\n >>> for x in [5, 1, 3, 2, 6]:\n ... heapq.heappush(h, -x)\n ...\n >>> [-heapq.heappop(h) for _ in range(len(h))]\n [6, 5, 3, 2, 1]\n\nFor custom objects, implement ``__lt__`` with reversed comparison:\n\n.. code-block:: python\n\n import heapq\n\n class MaxHeapItem:\n def __init__(self, val):\n self.val = val\n\n def __lt__(self, other):\n return self.val > other.val # reversed for max heap\n\n h = []\n for x in [5, 1, 3]:\n heapq.heappush(h, MaxHeapItem(x))\n\n print(heapq.heappop(h).val) # 5 (largest)\n\nImplement Priority Queue with ``heapq``\n---------------------------------------\n\nA priority queue processes elements based on their priority rather than insertion\norder. Use tuples ``(priority, value)`` where lower numbers indicate higher priority.\n\n.. code-block:: python\n\n >>> import heapq\n >>> pq = []\n >>> heapq.heappush(pq, (2, \"medium\"))\n >>> heapq.heappush(pq, (1, \"high\"))\n >>> heapq.heappush(pq, (3, \"low\"))\n >>> [heapq.heappop(pq) for _ in range(len(pq))]\n [(1, 'high'), (2, 'medium'), (3, 'low')]\n\nFor custom objects, implement the ``__lt__`` method to define comparison behavior:\n\n.. code-block:: python\n\n import heapq\n\n class Task:\n def __init__(self, priority, name):\n self.priority = priority\n self.name = name\n\n def __lt__(self, other):\n return self.priority < other.priority\n\n def __repr__(self):\n return f\"Task({self.priority}, {self.name!r})\"\n\n h = []\n heapq.heappush(h, Task(3, \"low\"))\n heapq.heappush(h, Task(1, \"high\"))\n heapq.heappush(h, Task(2, \"medium\"))\n\n while h:\n print(heapq.heappop(h))\n # Task(1, 'high')\n # Task(2, 'medium')\n # Task(3, 'low')\n\nFind K Largest or Smallest Elements\n-----------------------------------\n\nThe ``nlargest`` and ``nsmallest`` functions efficiently find the k largest or\nsmallest elements. They are more efficient than sorting when k is small relative\nto the list size.\n\n.. code-block:: python\n\n >>> import heapq\n >>> nums = [5, 1, 8, 3, 9, 2, 7]\n >>> heapq.nsmallest(3, nums)\n [1, 2, 3]\n >>> heapq.nlargest(3, nums)\n [9, 8, 7]\n\nUse the ``key`` parameter to extract comparison keys from complex objects:\n\n.. code-block:: python\n\n >>> import heapq\n >>> data = [\n ... {'name': 'Alice', 'score': 85},\n ... {'name': 'Bob', 'score': 92},\n ... {'name': 'Charlie', 'score': 78},\n ... ]\n >>> heapq.nlargest(2, data, key=lambda x: x['score'])\n [{'name': 'Bob', 'score': 92}, {'name': 'Alice', 'score': 85}]\n\nMerge Sorted Iterables\n----------------------\n\nThe ``merge`` function merges multiple sorted inputs into a single sorted output.\nIt returns an iterator, making it memory-efficient for large datasets.\n\n.. code-block:: python\n\n >>> import heapq\n >>> a = [1, 3, 5, 7]\n >>> b = [2, 4, 6, 8]\n >>> c = [0, 9, 10]\n >>> list(heapq.merge(a, b, c))\n [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]\n\nUse ``key`` and ``reverse`` parameters for custom merging:\n\n.. code-block:: python\n\n >>> import heapq\n >>> # Merge in descending order\n >>> a = [5, 3, 1]\n >>> b = [6, 4, 2]\n >>> list(heapq.merge(a, b, reverse=True))\n [6, 5, 4, 3, 2, 1]\n\nMaintain a Fixed-Size Heap\n--------------------------\n\nTo maintain a heap of fixed size k (e.g., tracking top k elements), use\n``heappushpop`` or check the size after each push.\n\n.. code-block:: python\n\n >>> import heapq\n >>> def top_k(items, k):\n ... \"\"\"Keep track of k largest elements using min-heap.\"\"\"\n ... h = []\n ... for x in items:\n ... if len(h) < k:\n ... heapq.heappush(h, x)\n ... elif x > h[0]:\n ... heapq.heapreplace(h, x)\n ... return sorted(h, reverse=True)\n ...\n >>> top_k([5, 1, 8, 3, 9, 2, 7, 4, 6], 3)\n [9, 8, 7]\n\nHeap with Index Tracking\n------------------------\n\nWhen you need to update priorities in a heap, use a dictionary to track element\npositions or mark entries as invalid.\n\n.. code-block:: python\n\n import heapq\n\n class IndexedHeap:\n def __init__(self):\n self.heap = []\n self.entry_finder = {}\n self.REMOVED = ''\n\n def push(self, item, priority):\n if item in self.entry_finder:\n self.remove(item)\n entry = [priority, item]\n self.entry_finder[item] = entry\n heapq.heappush(self.heap, entry)\n\n def remove(self, item):\n entry = self.entry_finder.pop(item)\n entry[-1] = self.REMOVED\n\n def pop(self):\n while self.heap:\n priority, item = heapq.heappop(self.heap)\n if item is not self.REMOVED:\n del self.entry_finder[item]\n return item\n raise KeyError('pop from empty heap')\n\n # Usage\n h = IndexedHeap()\n h.push('task1', 3)\n h.push('task2', 1)\n h.push('task1', 0) # update priority\n print(h.pop()) # task1 (now has priority 0)\n"
},
{
"path": "docs/notes/basic/python-list.rst",
"content": ".. meta::\n :description lang=en: Python list cheat sheet covering list operations, comprehensions, slicing, sorting, filtering, and common list manipulation patterns with code examples\n :keywords: Python, Python3, Python list, Python list cheat sheet, list comprehension, slicing, sorting, filtering, append, extend, iteration\n\n====\nList\n====\n\n.. contents:: Table of Contents\n :backlinks: none\n\nThe list is a common data structure which we use to store objects. Most of the\ntime, programmers concern about getting, setting, searching, filtering, and\nsorting. Furthermore, sometimes, we waltz ourself into common pitfalls of\nthe memory management. Thus, the main goal of this cheat sheet is to collect\nsome common operations and pitfalls.\n\nPython List Basics and Common Operations\n----------------------------------------\n\nThere are so many ways that we can manipulate lists in Python. Before we start\nto learn those versatile manipulations, the following snippet shows the most\ncommon operations of lists.\n\n.. code-block:: python\n\n >>> a = [1, 2, 3, 4, 5]\n >>> # contains\n >>> 2 in a\n True\n >>> # positive index\n >>> a[0]\n 1\n >>> # negative index\n >>> a[-1]\n 5\n >>> # slicing list[start:end:step]\n >>> a[1:]\n [2, 3, 4, 5]\n >>> a[1:-1]\n [2, 3, 4]\n >>> a[1:-1:2]\n [2, 4]\n >>> # reverse\n >>> a[::-1]\n [5, 4, 3, 2, 1]\n >>> a[:0:-1]\n [5, 4, 3, 2]\n >>> # set an item\n >>> a[0] = 0\n >>> a\n [0, 2, 3, 4, 5]\n >>> # append items to list\n >>> a.append(6)\n >>> a\n [0, 2, 3, 4, 5, 6]\n >>> a.extend([7, 8, 9])\n >>> a\n [0, 2, 3, 4, 5, 6, 7, 8, 9]\n >>> # delete an item\n >>> del a[-1]\n >>> a\n [0, 2, 3, 4, 5, 6, 7, 8]\n >>> # list comprehension\n >>> b = [x for x in range(3)]\n >>> b\n [0, 1, 2]\n >>> # add two lists\n >>> a + b\n [0, 2, 3, 4, 5, 6, 7, 8, 0, 1, 2]\n\nInitialize Lists with Multiplication Operator\n---------------------------------------------\n\nGenerally speaking, we can create a list through ``*`` operator if the item in\nthe list expression is an immutable object.\n\n.. code-block:: python\n\n >>> a = [None] * 3\n >>> a\n [None, None, None]\n >>> a[0] = \"foo\"\n >>> a\n ['foo', None, None]\n\nHowever, if the item in the list expression is a mutable object, the ``*``\noperator will copy the reference of the item N times. In order to avoid this\npitfall, we should use a list comprehension to initialize a list.\n\n.. code-block:: python\n\n >>> a = [[]] * 3\n >>> b = [[] for _ in range(3)]\n >>> a[0].append(\"Hello\")\n >>> a\n [['Hello'], ['Hello'], ['Hello']]\n >>> b[0].append(\"Python\")\n >>> b\n [['Python'], [], []]\n\nCopy Lists: Shallow vs Deep Copy\n--------------------------------\n\nAssigning a list to a variable is a common pitfall. This assignment does not\ncopy the list to the variable. The variable only refers to the list and increase\nthe reference count of the list.\n\n.. code-block:: python\n\n import sys\n >>> a = [1, 2, 3]\n >>> sys.getrefcount(a)\n 2\n >>> b = a\n >>> sys.getrefcount(a)\n 3\n >>> b[2] = 123456 # a[2] = 123456\n >>> b\n [1, 2, 123456]\n >>> a\n [1, 2, 123456]\n\nThere are two types of copy. The first one is called *shallow copy* (non-recursive copy)\nand the second one is called *deep copy* (recursive copy). Most of the time, it\nis sufficient for us to copy a list by shallow copy. However, if a list is nested,\nwe have to use a deep copy.\n\n.. code-block:: python\n\n >>> # shallow copy\n >>> a = [1, 2]\n >>> b = list(a)\n >>> b[0] = 123\n >>> a\n [1, 2]\n >>> b\n [123, 2]\n >>> a = [[1], [2]]\n >>> b = list(a)\n >>> b[0][0] = 123\n >>> a\n [[123], [2]]\n >>> b\n [[123], [2]]\n >>> # deep copy\n >>> import copy\n >>> a = [[1], [2]]\n >>> b = copy.deepcopy(a)\n >>> b[0][0] = 123\n >>> a\n [[1], [2]]\n >>> b\n [[123], [2]]\n\nSlice Lists with slice Objects\n------------------------------\n\nSometimes, our data may concatenate as a large segment such as packets. In\nthis case, we will represent the range of data by using ``slice`` objects\nas explaining variables instead of using *slicing expressions*.\n\n.. code-block:: python\n\n >>> icmp = (\n ... b\"080062988e2100005bff49c20005767c\"\n ... b\"08090a0b0c0d0e0f1011121314151617\"\n ... b\"18191a1b1c1d1e1f2021222324252627\"\n ... b\"28292a2b2c2d2e2f3031323334353637\"\n ... )\n >>> head = slice(0, 32)\n >>> data = slice(32, len(icmp))\n >>> icmp[head]\n b'080062988e2100005bff49c20005767c'\n\nCreate Lists with List Comprehensions\n-------------------------------------\n\n`List comprehensions `_\nwhich was proposed in PEP `202 `_\nprovides a graceful way to create a new list based on another list, sequence,\nor some object which is iterable. In addition, we can use this expression to\nsubstitute ``map`` and ``filter`` sometimes.\n\n.. code-block:: python\n\n >>> [x for x in range(10)]\n [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n >>> [(lambda x: x**2)(i) for i in range(10)]\n [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]\n >>> [x for x in range(10) if x > 5]\n [6, 7, 8, 9]\n >>> [x if x > 5 else 0 for x in range(10)]\n [0, 0, 0, 0, 0, 0, 6, 7, 8, 9]\n >>> [x + 1 if x < 5 else x + 2 if x > 5 else x + 5 for x in range(10)]\n [1, 2, 3, 4, 5, 10, 8, 9, 10, 11]\n >>> [(x, y) for x in range(3) for y in range(2)]\n [(0, 0), (0, 1), (1, 0), (1, 1), (2, 0), (2, 1)]\n\nUnpack Lists into Variables\n---------------------------\n\nSometimes, we want to unpack our list to variables in order to make our code\nbecome more readable. In this case, we assign N elements to N variables as\nfollowing example.\n\n.. code-block:: python\n\n >>> arr = [1, 2, 3]\n >>> a, b, c = arr\n >>> a, b, c\n (1, 2, 3)\n\nBased on PEP `3132 `_, we can use a\nsingle asterisk to unpack N elements to the number of variables which is less\nthan N in Python 3.\n\n.. code-block:: python\n\n >>> arr = [1, 2, 3, 4, 5]\n >>> a, b, *c, d = arr\n >>> a, b, d\n (1, 2, 5)\n >>> c\n [3, 4]\n\nIterate with Index Using enumerate()\n------------------------------------\n\n``enumerate`` is a built-in function. It helps us to acquire indexes\n(or a count) and elements at the same time without using ``range(len(list))``.\nFurther information can be found on\n`Looping Techniques `_.\n\n.. code-block:: python\n\n >>> for i, v in enumerate(range(3)):\n ... print(i, v)\n ...\n 0 0\n 1 1\n 2 2\n >>> for i, v in enumerate(range(3), 1): # start = 1\n ... print(i, v)\n ...\n 1 0\n 2 1\n 3 2\n\nCombine Lists with zip()\n------------------------\n\n`zip `_ enables us to\niterate over items contained in multiple lists at a time. Iteration stops\nwhenever one of the lists is exhausted. As a result, the length of the\niteration is the same as the shortest list. If this behavior is not desired,\nwe can use ``itertools.zip_longest`` in **Python 3** or ``itertools.izip_longest``\nin **Python 2**.\n\n.. code-block:: python\n\n >>> a = [1, 2, 3]\n >>> b = [4, 5, 6]\n >>> list(zip(a, b))\n [(1, 4), (2, 5), (3, 6)]\n >>> c = [1]\n >>> list(zip(a, b, c))\n [(1, 4, 1)]\n >>> from itertools import zip_longest\n >>> list(zip_longest(a, b, c))\n [(1, 4, 1), (2, 5, None), (3, 6, None)]\n\n\nFilter List Items\n-----------------\n\n`filter `_ is a\nbuilt-in function to assist us to remove unnecessary items. In **Python 2**,\n``filter`` returns a list. However, in **Python 3**, ``filter`` returns an\n*iterable object*. Note that *list comprehension* or *generator\nexpression* provides a more concise way to remove items.\n\n.. code-block:: python\n\n >>> [x for x in range(5) if x > 1]\n [2, 3, 4]\n >>> l = ['1', '2', 3, 'Hello', 4]\n >>> f = lambda x: isinstance(x, int)\n >>> filter(f, l)\n \n >>> list(filter(f, l))\n [3, 4]\n >>> list((i for i in l if f(i)))\n [3, 4]\n\nImplement Stack with List\n-------------------------\n\nThere is no need for an additional data structure, stack, in Python because the\n``list`` provides ``append`` and ``pop`` methods which enable us use a list as\na stack.\n\n.. code-block:: python\n\n >>> stack = []\n >>> stack.append(1)\n >>> stack.append(2)\n >>> stack.append(3)\n >>> stack\n [1, 2, 3]\n >>> stack.pop()\n 3\n >>> stack.pop()\n 2\n >>> stack\n [1]\n\nCheck Membership with in Operator\n---------------------------------\n\nWe can implement the ``__contains__`` method to make a class do ``in``\noperations. It is a common way for a programmer to emulate\na membership test operations for custom classes.\n\n.. code-block:: python\n\n class Stack:\n\n def __init__(self):\n self.__list = []\n\n def push(self, val):\n self.__list.append(val)\n\n def pop(self):\n return self.__list.pop()\n\n def __contains__(self, item):\n return True if item in self.__list else False\n\n stack = Stack()\n stack.push(1)\n print(1 in stack)\n print(0 in stack)\n\nExample\n\n.. code-block:: bash\n\n python stack.py\n True\n False\n\nAccess Items with __getitem__ and __setitem__\n---------------------------------------------\n\nMaking custom classes perform get and set operations like lists is simple. We\ncan implement a ``__getitem__`` method and a ``__setitem__`` method to enable\na class to retrieve and overwrite data by index. In addition, if we want to use\nthe function, ``len``, to calculate the number of elements, we can implement a\n``__len__`` method.\n\n.. code-block:: python\n\n class Stack:\n\n def __init__(self):\n self.__list = []\n\n def push(self, val):\n self.__list.append(val)\n\n def pop(self):\n return self.__list.pop()\n\n def __repr__(self):\n return \"{}\".format(self.__list)\n\n def __len__(self):\n return len(self.__list)\n\n def __getitem__(self, idx):\n return self.__list[idx]\n\n def __setitem__(self, idx, val):\n self.__list[idx] = val\n\n\n stack = Stack()\n stack.push(1)\n stack.push(2)\n print(\"stack:\", stack)\n\n stack[0] = 3\n print(\"stack:\", stack)\n print(\"num items:\", len(stack))\n\nExample\n\n.. code-block:: bash\n\n $ python stack.py\n stack: [1, 2]\n stack: [3, 2]\n num items: 2\n\nDelegate Iteration with __iter__\n--------------------------------\n\nIf a custom container class holds a list and we want iterations to work on the\ncontainer, we can implement a ``__iter__`` method to delegate iterations to\nthe list. Note that the method, ``__iter__``, should return an *iterator object*,\nso we cannot return the list directly; otherwise, Python raises a ``TypeError``.\n\n.. code-block:: python\n\n class Stack:\n\n def __init__(self):\n self.__list = []\n\n def push(self, val):\n self.__list.append(val)\n\n def pop(self):\n return self.__list.pop()\n\n def __iter__(self):\n return iter(self.__list)\n\n stack = Stack()\n stack.push(1)\n stack.push(2)\n for s in stack:\n print(s)\n\nExample\n\n.. code-block:: bash\n\n $ python stack.py\n 1\n 2\n\nSort Lists with sort() and sorted()\n-----------------------------------\n\nPython list provides a built-in ``list.sort`` method which sorts a list\n`in-place `_ without using\nextra memory. Moreover, the return value of ``list.sort`` is ``None`` in\norder to avoid confusion with ``sorted`` and the function can only be used for\n``list``.\n\n.. code-block:: python\n\n >>> l = [5, 4, 3, 2, 1]\n >>> l.sort()\n >>> l\n [1, 2, 3, 4, 5]\n >>> l.sort(reverse=True)\n >>> l\n [5, 4, 3, 2, 1]\n\nThe ``sorted`` function does not modify any iterable object in-place. Instead,\nit returns a new sorted list. Using ``sorted`` is safer than ``list.sort`` if\nsome list's elements are read-only or immutable. Besides, another difference\nbetween ``list.sort`` and ``sorted`` is that ``sorted`` accepts any **iterable\nobject**.\n\n.. code-block:: python\n\n >>> l = [5, 4, 3, 2, 1]\n >>> new = sorted(l)\n >>> new\n [1, 2, 3, 4, 5]\n >>> l\n [5, 4, 3, 2, 1]\n >>> d = {3: 'andy', 2: 'david', 1: 'amy'}\n >>> sorted(d) # sort iterable\n [1, 2, 3]\n\nTo sort a list with its elements are tuples, using ``operator.itemgetter`` is\nhelpful because it assigns a key function to the ``sorted`` key parameter. Note\nthat the key should be comparable; otherwise, it will raise a ``TypeError``.\n\n.. code-block:: python\n\n >>> from operator import itemgetter\n >>> l = [('andy', 10), ('david', 8), ('amy', 3)]\n >>> l.sort(key=itemgetter(1))\n >>> l\n [('amy', 3), ('david', 8), ('andy', 10)]\n\n``operator.itemgetter`` is useful because the function returns a getter\nmethod which can be applied to other objects with a method ``__getitem__``. For\nexample, sorting a list with its elements are dictionary can be achieved by\nusing ``operator.itemgetter`` due to all elements have ``__getitem__``.\n\n.. code-block:: python\n\n >>> from pprint import pprint\n >>> from operator import itemgetter\n >>> l = [\n ... {'name': 'andy', 'age': 10},\n ... {'name': 'david', 'age': 8},\n ... {'name': 'amy', 'age': 3},\n ... ]\n >>> l.sort(key=itemgetter('age'))\n >>> pprint(l)\n [{'age': 3, 'name': 'amy'},\n {'age': 8, 'name': 'david'},\n {'age': 10, 'name': 'andy'}]\n\nIf it is necessary to sort a list with its elements are neither comparable nor\nhaving ``__getitem__`` method, assigning a customized key function is feasible.\n\n.. code-block:: python\n\n >>> class Node(object):\n ... def __init__(self, val):\n ... self.val = val\n ... def __repr__(self):\n ... return f\"Node({self.val})\"\n ...\n >>> nodes = [Node(3), Node(2), Node(1)]\n >>> nodes.sort(key=lambda x: x.val)\n >>> nodes\n [Node(1), Node(2), Node(3)]\n >>> nodes.sort(key=lambda x: x.val, reverse=True)\n >>> nodes\n [Node(3), Node(2), Node(1)]\n\nThe above snippet can be simplified by using ``operator.attrgetter``. The\nfunction returns an attribute getter based on the attribute's name. Note that\nthe attribute should be comparable; otherwise, ``sorted`` or ``list.sort`` will\nraise ``TypeError``.\n\n.. code-block:: python\n\n >>> from operator import attrgetter\n >>> class Node(object):\n ... def __init__(self, val):\n ... self.val = val\n ... def __repr__(self):\n ... return f\"Node({self.val})\"\n ...\n >>> nodes = [Node(3), Node(2), Node(1)]\n >>> nodes.sort(key=attrgetter('val'))\n >>> nodes\n [Node(1), Node(2), Node(3)]\n\nIf an object has ``__lt__`` method, it means that the object is comparable and\n``sorted`` or ``list.sort`` is not necessary to input a key function to its key\nparameter. A list or an iterable sequence can be sorted directly.\n\n.. code-block:: python\n\n >>> class Node(object):\n ... def __init__(self, val):\n ... self.val = val\n ... def __repr__(self):\n ... return f\"Node({self.val})\"\n ... def __lt__(self, other):\n ... return self.val - other.val < 0\n ...\n >>> nodes = [Node(3), Node(2), Node(1)]\n >>> nodes.sort()\n >>> nodes\n [Node(1), Node(2), Node(3)]\n\nIf an object does not have ``__lt__`` method, it is likely to patch the method\nafter a declaration of the object's class. In other words, after the patching,\nthe object becomes comparable.\n\n.. code-block:: python\n\n >>> class Node(object):\n ... def __init__(self, val):\n ... self.val = val\n ... def __repr__(self):\n ... return f\"Node({self.val})\"\n ...\n >>> Node.__lt__ = lambda s, o: s.val < o.val\n >>> nodes = [Node(3), Node(2), Node(1)]\n >>> nodes.sort()\n >>> nodes\n [Node(1), Node(2), Node(3)]\n\nNote that ``sorted`` or ``list.sort`` in Python3 does not support ``cmp``\nparameter which is an **ONLY** valid argument in Python2. If it is necessary to\nuse an old comparison function, e.g., some legacy code, ``functools.cmp_to_key``\nis useful since it converts a comparison function to a key function.\n\n.. code-block:: python\n\n >>> from functools import cmp_to_key\n >>> class Node(object):\n ... def __init__(self, val):\n ... self.val = val\n ... def __repr__(self):\n ... return f\"Node({self.val})\"\n ...\n >>> nodes = [Node(3), Node(2), Node(1)]\n >>> nodes.sort(key=cmp_to_key(lambda x,y: x.val - y.val))\n >>> nodes\n [Node(1), Node(2), Node(3)]\n\nMaintain Sorted List with bisect\n--------------------------------\n\nThe `bisect `_ module provides\nfunctions to maintain a list in sorted order without having to sort the list\nafter each insertion. It uses a binary search algorithm, making insertions\nefficient for large lists.\n\n.. code-block:: python\n\n import bisect\n\n class Foo(object):\n def __init__(self, k):\n self.k = k\n\n def __eq__(self, rhs):\n return self.k == rhs.k\n\n def __ne__(self, rhs):\n return self.k != rhs.k\n\n def __lt__(self, rhs):\n return self.k < rhs.k\n\n def __gt__(self, rhs):\n return self.k > rhs.k\n\n def __le__(self, rhs):\n return self.k <= rhs.k\n\n def __ge__(self, rhs):\n return self.k >= rhs.k\n\n def __repr__(self):\n return f\"Foo({self.k})\"\n\n def __str__(self):\n return self.__repr__()\n\n foo = [Foo(1), Foo(3), Foo(2), Foo(0)]\n bar = []\n for x in foo:\n bisect.insort(bar, x)\n\n print(bar) # [Foo(0), Foo(1), Foo(2), Foo(3)]\n\nCreate Nested Lists Correctly\n-----------------------------\n\nWhen creating nested lists (2D lists or matrices), we should use list\ncomprehension to ensure each inner list is a separate object. The following\nsnippet shows the correct way to create a 2D list.\n\n.. code-block:: python\n\n # new a list with size = 3\n\n >>> [0] * 3\n [0, 0, 0]\n\n # new a 2d list with size 3x3\n\n >>> [[0] * 3 for _ in range(3)]\n [[0, 0, 0], [0, 0, 0], [0, 0, 0]]\n\nNote that we should avoid creating a multi-dimension list via the following\nsnippet because all objects in the list point to the same address.\n\n.. code-block:: python\n\n >>> a = [[0] * 3] * 3\n >>> a\n [[0, 0, 0], [0, 0, 0], [0, 0, 0]]\n >>> a[1][1] = 2\n >>> a\n [[0, 2, 0], [0, 2, 0], [0, 2, 0]]\n\n\nImplement Circular Buffer with deque\n------------------------------------\n\n`collections.deque `_\nis a double-ended queue that supports adding and removing elements from both ends\nefficiently. By setting ``maxlen``, we can create a circular buffer that automatically\ndiscards old elements when new ones are added.\n\n.. code-block:: python\n\n >>> from collections import deque\n >>> d = deque(maxlen=8)\n >>> for x in range(9):\n ... d.append(x)\n ...\n >>> d\n deque([1, 2, 3, 4, 5, 6, 7, 8], maxlen=8)\n\nThe following example shows how to implement a ``tail`` function similar to\nthe Unix command using ``deque``.\n\n.. code-block:: python\n\n >>> from collections import deque\n >>> def tail(path, n=10):\n ... with open(path) as f:\n ... return deque(f, n)\n ...\n >>> tail(\"/etc/hosts\")\n\nSplit List into Chunks\n----------------------\n\nSometimes, we need to split a list into smaller chunks of a specific size.\nThe following generator function yields successive chunks from the list.\n\n.. code-block:: python\n\n >>> def chunk(lst, n):\n ... for i in range(0, len(lst), n):\n ... yield lst[i:i+n]\n ...\n >>> a = [1, 2, 3, 4, 5, 6, 7, 8]\n >>> list(chunk(a, 3))\n [[1, 2, 3], [4, 5, 6], [7, 8]]\n\nGroup Consecutive Elements with itertools.groupby\n-------------------------------------------------\n\n`itertools.groupby `_\ngroups consecutive elements in an iterable that have the same key. It is useful\nfor run-length encoding or grouping sorted data.\n\n.. code-block:: python\n\n >>> import itertools\n >>> s = \"AAABBCCCCC\"\n >>> for k, v in itertools.groupby(s):\n ... print(k, list(v))\n ...\n A ['A', 'A', 'A']\n B ['B', 'B']\n C ['C', 'C', 'C', 'C', 'C']\n\n # group by key\n\n >>> x = [('gp1', 'a'), ('gp2', 'b'), ('gp2', 'c')]\n >>> for k, v in itertools.groupby(x, lambda x: x[0]):\n ... print(k, list(v))\n ...\n gp1 [('gp1', 'a')]\n gp2 [('gp2', 'b'), ('gp2', 'c')]\n\nBinary Search in Sorted List\n----------------------------\n\nBinary search is an efficient algorithm for finding an item in a sorted list.\nThe following snippet shows how to implement binary search using ``bisect_left``.\n\n.. code-block:: python\n\n >>> def binary_search(arr, x, lo=0, hi=None):\n ... if not hi: hi = len(arr)\n ... pos = bisect_left(arr, x, lo, hi)\n ... return pos if pos != hi and arr[pos] == x else -1\n ...\n >>> a = [1, 1, 1, 2, 3]\n >>> binary_search(a, 1)\n 0\n >>> binary_search(a, 2)\n 3\n\nFind Lower Bound with bisect_left\n---------------------------------\n\n``bisect_left`` returns the leftmost position where an element can be inserted\nto keep the list sorted. This is equivalent to finding the lower bound.\n\n.. code-block:: python\n\n >>> import bisect\n >>> a = [1,2,3,3,4,5]\n >>> bisect.bisect_left(a, 3)\n 2\n >>> bisect.bisect_left(a, 3.5)\n 4\n\nFind Upper Bound with bisect_right\n----------------------------------\n\n``bisect_right`` (or ``bisect``) returns the rightmost position where an element\ncan be inserted to keep the list sorted. This is equivalent to finding the upper bound.\n\n.. code-block:: python\n\n >>> import bisect\n >>> a = [1,2,3,3,4,5]\n >>> bisect.bisect_right(a, 3)\n 4\n >>> bisect.bisect_right(a, 3.5)\n 4\n\nSort Tuples Lexicographically\n-----------------------------\n\nPython compares tuples and lists lexicographically by default. This means it\ncompares the first elements, and if they are equal, it compares the second\nelements, and so on.\n\n.. code-block:: python\n\n # python compare lists lexicographically\n\n >>> a = [(1,2), (1,1), (1,0), (2,1)]\n >>> a.sort()\n >>> a\n [(1, 0), (1, 1), (1, 2), (2, 1)]\n\nImplement Trie (Prefix Tree)\n----------------------------\n\nA `Trie `_ (prefix tree) is a tree data\nstructure used for efficient retrieval of keys in a dataset of strings. The\nfollowing snippet shows a compact implementation using ``defaultdict``.\n\n.. code-block:: python\n\n >>> from functools import reduce\n >>> from collections import defaultdict\n >>> Trie = lambda: defaultdict(Trie)\n >>> prefixes = ['abc', 'de', 'g']\n >>> trie = Trie()\n >>> end = True\n >>> for p in prefixes:\n ... reduce(dict.__getitem__, p, trie)[end] = p\n ...\n\n # search prefix\n\n >>> def find(trie, word):\n ... curr = trie\n ... for c in word:\n ... if c not in curr:\n ... return False\n ... curr = curr[c]\n ... return True\n ...\n >>> find(trie, \"abcdef\")\n False\n >>> find(trie, \"abc\")\n True\n >>> find(trie, \"ab\")\n True\n\n # search word\n\n >>> def find(trie, p):\n ... curr = trie\n ... for c in p:\n ... if c not in curr or True in curr:\n ... break\n ... curr = curr[c]\n ... return True if True in curr else False\n ...\n >>> find(trie, \"abcdef\")\n True\n >>> find(trie, \"abc\")\n True\n >>> find(trie, \"ab\")\n False\n"
},
{
"path": "docs/notes/basic/python-object.rst",
"content": ".. meta::\n :description lang=en: Python class cheat sheet covering magic methods, property decorators, inheritance, context managers, and OOP design patterns with code examples\n :keywords: Python, Python3, Python class, Python OOP cheat sheet, magic methods, property decorator, context manager, singleton, abstract class, descriptor, inheritance\n\n=====\nClass\n=====\n\n.. contents:: Table of Contents\n :backlinks: none\n\nPython is an object-oriented programming language. This cheat sheet covers\nclass definitions, inheritance, magic methods, property decorators, context\nmanagers, and common design patterns. Understanding these concepts is essential\nfor writing clean, maintainable Python code.\n\nList Attributes with dir()\n--------------------------\n\nThe ``dir()`` function returns a list of all attributes and methods of an object.\nThis is useful for introspection and discovering what operations are available.\n\n.. code-block:: python\n\n >>> dir(list) # check all attr of list\n ['__add__', '__class__', ...]\n\nCheck Type with isinstance()\n----------------------------\n\nUse ``isinstance()`` to check if an object is an instance of a class or its\nsubclasses. This is preferred over ``type()`` comparison because it supports\ninheritance.\n\n.. code-block:: python\n\n >>> ex = 10\n >>> isinstance(ex, int)\n True\n >>> isinstance(ex, (int, float)) # check multiple types\n True\n\nCheck Inheritance with issubclass()\n-----------------------------------\n\nUse ``issubclass()`` to check if a class is a subclass of another class.\n\n.. code-block:: python\n\n >>> class Animal: pass\n >>> class Dog(Animal): pass\n >>> issubclass(Dog, Animal)\n True\n >>> issubclass(Dog, object)\n True\n\nGet Class Name\n--------------\n\nAccess the class name through the ``__class__.__name__`` attribute.\n\n.. code-block:: python\n\n >>> class ExampleClass:\n ... pass\n ...\n >>> ex = ExampleClass()\n >>> ex.__class__.__name__\n 'ExampleClass'\n\nHas / Get / Set Attributes\n--------------------------\n\nPython provides built-in functions to dynamically access and modify object\nattributes at runtime.\n\n.. code-block:: python\n\n >>> class Example:\n ... def __init__(self):\n ... self.name = \"ex\"\n ...\n >>> ex = Example()\n >>> hasattr(ex, \"name\")\n True\n >>> getattr(ex, 'name')\n 'ex'\n >>> setattr(ex, 'name', 'example')\n >>> ex.name\n 'example'\n >>> getattr(ex, 'missing', 'default') # with default\n 'default'\n\nDeclare Class with type()\n-------------------------\n\nClasses can be created dynamically using ``type()``. This is useful for\nmetaprogramming and creating classes at runtime.\n\n.. code-block:: python\n\n >>> def greet(self):\n ... return f\"Hello, I'm {self.name}\"\n ...\n >>> Person = type('Person', (object,), {\n ... 'name': 'Anonymous',\n ... 'greet': greet\n ... })\n >>> p = Person()\n >>> p.greet()\n \"Hello, I'm Anonymous\"\n\nThis is equivalent to:\n\n.. code-block:: python\n\n >>> class Person:\n ... name = 'Anonymous'\n ... def greet(self):\n ... return f\"Hello, I'm {self.name}\"\n\n__new__ vs __init__\n-------------------\n\n``__new__`` creates the instance, ``__init__`` initializes it. ``__init__`` is\nonly called if ``__new__`` returns an instance of the class.\n\n.. code-block:: python\n\n >>> class Example:\n ... def __new__(cls, arg):\n ... print(f'__new__ {arg}')\n ... return super().__new__(cls)\n ... def __init__(self, arg):\n ... print(f'__init__ {arg}')\n ...\n >>> o = Example(\"Hello\")\n __new__ Hello\n __init__ Hello\n\n__str__ and __repr__\n--------------------\n\n``__str__`` returns a human-readable string, ``__repr__`` returns an unambiguous\nrepresentation for debugging. When ``__str__`` is not defined, ``__repr__`` is used.\n\n.. code-block:: python\n\n >>> class Vector:\n ... def __init__(self, x, y):\n ... self.x, self.y = x, y\n ... def __repr__(self):\n ... return f\"Vector({self.x}, {self.y})\"\n ... def __str__(self):\n ... return f\"({self.x}, {self.y})\"\n ...\n >>> v = Vector(1, 2)\n >>> repr(v)\n 'Vector(1, 2)'\n >>> str(v)\n '(1, 2)'\n >>> print(v)\n (1, 2)\n\nComparison Magic Methods\n------------------------\n\nImplement comparison operators by defining magic methods. Use\n``functools.total_ordering`` to generate all comparisons from ``__eq__`` and one other.\n\n.. code-block:: python\n\n >>> from functools import total_ordering\n >>> @total_ordering\n ... class Number:\n ... def __init__(self, val):\n ... self.val = val\n ... def __eq__(self, other):\n ... return self.val == other.val\n ... def __lt__(self, other):\n ... return self.val < other.val\n ...\n >>> Number(1) < Number(2)\n True\n >>> Number(2) >= Number(1)\n True\n\nArithmetic Magic Methods\n------------------------\n\nImplement arithmetic operators to make objects work with ``+``, ``-``, ``*``, etc.\n\n.. code-block:: python\n\n >>> class Vector:\n ... def __init__(self, x, y):\n ... self.x, self.y = x, y\n ... def __add__(self, other):\n ... return Vector(self.x + other.x, self.y + other.y)\n ... def __mul__(self, scalar):\n ... return Vector(self.x * scalar, self.y * scalar)\n ... def __repr__(self):\n ... return f\"Vector({self.x}, {self.y})\"\n ...\n >>> Vector(1, 2) + Vector(3, 4)\n Vector(4, 6)\n >>> Vector(1, 2) * 3\n Vector(3, 6)\n\nCallable with __call__\n----------------------\n\nImplement ``__call__`` to make instances callable like functions. This is useful\nfor creating function-like objects that maintain state.\n\n.. code-block:: python\n\n >>> class Multiplier:\n ... def __init__(self, factor):\n ... self.factor = factor\n ... def __call__(self, x):\n ... return x * self.factor\n ...\n >>> double = Multiplier(2)\n >>> double(5)\n 10\n >>> callable(double)\n True\n\n@property Decorator\n-------------------\n\nUse ``@property`` to define getters, setters, and deleters for managed attributes.\nThis allows attribute access syntax while running custom code.\n\n.. code-block:: python\n\n >>> class Circle:\n ... def __init__(self, radius):\n ... self._radius = radius\n ... @property\n ... def radius(self):\n ... return self._radius\n ... @radius.setter\n ... def radius(self, value):\n ... if value < 0:\n ... raise ValueError(\"Radius must be positive\")\n ... self._radius = value\n ... @property\n ... def area(self):\n ... return 3.14159 * self._radius ** 2\n ...\n >>> c = Circle(5)\n >>> c.area\n 78.53975\n >>> c.radius = 10\n >>> c.radius\n 10\n\nDescriptor Protocol\n-------------------\n\nDescriptors control attribute access at the class level. They implement\n``__get__``, ``__set__``, and/or ``__delete__`` methods.\n\n.. code-block:: python\n\n >>> class Positive:\n ... def __init__(self, name):\n ... self.name = name\n ... def __get__(self, obj, objtype=None):\n ... return obj.__dict__[self.name]\n ... def __set__(self, obj, value):\n ... if value < 0:\n ... raise ValueError(\"Must be positive\")\n ... obj.__dict__[self.name] = value\n ...\n >>> class Example:\n ... x = Positive('x')\n ... def __init__(self, x):\n ... self.x = x\n ...\n >>> ex = Example(10)\n >>> ex.x\n 10\n\nContext Manager Protocol\n------------------------\n\nContext managers implement ``__enter__`` and ``__exit__`` to manage resources\nwith the ``with`` statement. This ensures proper cleanup even if exceptions occur.\n\n.. code-block:: python\n\n class ManagedFile:\n def __init__(self, filename):\n self.filename = filename\n\n def __enter__(self):\n self.file = open(self.filename, 'r')\n return self.file\n\n def __exit__(self, exc_type, exc_val, exc_tb):\n self.file.close()\n return False # don't suppress exceptions\n\n with ManagedFile('example.txt') as f:\n content = f.read()\n\nUsing contextlib\n----------------\n\nThe ``contextlib`` module provides utilities for creating context managers\nwithout writing a full class.\n\n.. code-block:: python\n\n from contextlib import contextmanager\n\n @contextmanager\n def managed_file(filename):\n f = open(filename, 'r')\n try:\n yield f\n finally:\n f.close()\n\n with managed_file('example.txt') as f:\n content = f.read()\n\n@staticmethod and @classmethod\n------------------------------\n\n``@staticmethod`` defines a method that doesn't access instance or class.\n``@classmethod`` receives the class as the first argument, useful for\nalternative constructors.\n\n.. code-block:: python\n\n >>> class Date:\n ... def __init__(self, year, month, day):\n ... self.year, self.month, self.day = year, month, day\n ... @classmethod\n ... def from_string(cls, date_string):\n ... year, month, day = map(int, date_string.split('-'))\n ... return cls(year, month, day)\n ... @staticmethod\n ... def is_valid(date_string):\n ... try:\n ... y, m, d = map(int, date_string.split('-'))\n ... return 1 <= m <= 12 and 1 <= d <= 31\n ... except:\n ... return False\n ...\n >>> d = Date.from_string('2024-01-15')\n >>> d.year\n 2024\n >>> Date.is_valid('2024-13-01')\n False\n\nAbstract Base Classes with abc\n------------------------------\n\nUse ``abc`` module to define abstract base classes that cannot be instantiated\nand require subclasses to implement certain methods.\n\n.. code-block:: python\n\n >>> from abc import ABC, abstractmethod\n >>> class Shape(ABC):\n ... @abstractmethod\n ... def area(self):\n ... pass\n ...\n >>> class Rectangle(Shape):\n ... def __init__(self, width, height):\n ... self.width, self.height = width, height\n ... def area(self):\n ... return self.width * self.height\n ...\n >>> r = Rectangle(3, 4)\n >>> r.area()\n 12\n >>> Shape() # raises TypeError\n\nThe Diamond Problem (MRO)\n-------------------------\n\nPython uses Method Resolution Order (MRO) to resolve the diamond problem in\nmultiple inheritance. Use ``ClassName.mro()`` to see the resolution order.\n\n.. code-block:: python\n\n >>> class A:\n ... def method(self):\n ... return \"A\"\n ...\n >>> class B(A):\n ... def method(self):\n ... return \"B\"\n ...\n >>> class C(A):\n ... def method(self):\n ... return \"C\"\n ...\n >>> class D(B, C):\n ... pass\n ...\n >>> D().method()\n 'B'\n >>> D.mro()\n [, , , , ]\n\nSingleton Pattern\n-----------------\n\nSingleton ensures only one instance of a class exists. Implement using\n``__new__`` or a decorator.\n\n.. code-block:: python\n\n class Singleton:\n _instance = None\n\n def __new__(cls):\n if cls._instance is None:\n cls._instance = super().__new__(cls)\n return cls._instance\n\n a = Singleton()\n b = Singleton()\n print(a is b) # True\n\nUsing __slots__\n---------------\n\n``__slots__`` restricts instance attributes and reduces memory usage by avoiding\n``__dict__`` per instance.\n\n.. code-block:: python\n\n >>> class Point:\n ... __slots__ = ['x', 'y']\n ... def __init__(self, x, y):\n ... self.x, self.y = x, y\n ...\n >>> p = Point(1, 2)\n >>> p.x\n 1\n >>> p.z = 3 # raises AttributeError\n\nCommon Magic Methods Reference\n------------------------------\n\n.. code-block:: python\n\n # Object Creation and Representation\n __new__(cls, ...) # create instance\n __init__(self, ...) # initialize instance\n __del__(self) # destructor\n __repr__(self) # repr(obj)\n __str__(self) # str(obj)\n\n # Comparison\n __eq__(self, other) # ==\n __ne__(self, other) # !=\n __lt__(self, other) # <\n __le__(self, other) # <=\n __gt__(self, other) # >\n __ge__(self, other) # >=\n\n # Arithmetic\n __add__(self, other) # +\n __sub__(self, other) # -\n __mul__(self, other) # *\n __truediv__(self, other) # /\n __floordiv__(self, other)# //\n __mod__(self, other) # %\n __pow__(self, other) # **\n\n # Container\n __len__(self) # len(obj)\n __getitem__(self, key) # obj[key]\n __setitem__(self, k, v) # obj[key] = value\n __delitem__(self, key) # del obj[key]\n __contains__(self, item) # item in obj\n __iter__(self) # iter(obj)\n\n # Attribute Access\n __getattr__(self, name) # obj.name (when not found)\n __setattr__(self, n, v) # obj.name = value\n __delattr__(self, name) # del obj.name\n\n # Callable\n __call__(self, ...) # obj()\n\n # Context Manager\n __enter__(self) # with obj\n __exit__(self, ...) # exit with block\n\n # Descriptor\n __get__(self, obj, type) # descriptor access\n __set__(self, obj, val) # descriptor assignment\n __delete__(self, obj) # descriptor deletion\n"
},
{
"path": "docs/notes/basic/python-rexp.rst",
"content": ".. meta::\n :description lang=en: Python regex cheat sheet covering re module, pattern matching, groups, lookahead, lookbehind, substitution, and common regex patterns with code examples\n :keywords: Python, Python3, Python regex, Python regex cheat sheet, regular expression, re module, pattern matching, findall, search, match, sub, lookahead, lookbehind, named groups\n\n==================\nRegular Expression\n==================\n\n.. contents:: Table of Contents\n :backlinks: none\n\nRegular expressions (regex) are powerful tools for pattern matching and text\nmanipulation. Python's ``re`` module provides comprehensive support for regex\noperations. This cheat sheet covers basic matching, groups, lookaround assertions,\nsubstitution, and common patterns for validating emails, URLs, IP addresses, etc.\n\nBasic Operations\n----------------\n\nThe ``re`` module provides several functions for pattern matching. Use ``search()``\nto find the first match anywhere in the string, ``match()`` to match at the\nbeginning, and ``fullmatch()`` to match the entire string.\n\n.. code-block:: python\n\n >>> import re\n >>> # search - find anywhere in string\n >>> re.search(r'\\d+', 'abc123def')\n \n\n >>> # match - match at beginning only\n >>> re.match(r'\\d+', '123abc')\n \n >>> re.match(r'\\d+', 'abc123') is None\n True\n\n >>> # fullmatch - match entire string\n >>> re.fullmatch(r'\\d+', '123')\n \n >>> re.fullmatch(r'\\d+', '123abc') is None\n True\n\n``re.findall()`` - Find All Matches\n-----------------------------------\n\nThe ``findall()`` function returns all non-overlapping matches as a list of\nstrings. If the pattern has groups, it returns a list of tuples.\n\n.. code-block:: python\n\n >>> # find all words\n >>> source = \"Hello World Ker HAHA\"\n >>> re.findall(r'[\\w]+', source)\n ['Hello', 'World', 'Ker', 'HAHA']\n\n >>> # find all digits\n >>> re.findall(r'\\d+', 'a1b22c333')\n ['1', '22', '333']\n\n >>> # with groups - returns tuples\n >>> re.findall(r'(\\w+)=(\\d+)', 'a=1 b=2 c=3')\n [('a', '1'), ('b', '2'), ('c', '3')]\n\n``re.split()`` - Split by Pattern\n---------------------------------\n\nThe ``split()`` function splits a string by pattern occurrences. Use ``maxsplit``\nto limit the number of splits.\n\n.. code-block:: python\n\n >>> re.split(r'\\s+', 'a b c')\n ['a', 'b', 'c']\n\n >>> re.split(r'[,;]', 'a,b;c,d')\n ['a', 'b', 'c', 'd']\n\n >>> re.split(r'(\\s+)', 'a b c') # keep delimiters\n ['a', ' ', 'b', ' ', 'c']\n\n >>> re.split(r'\\s+', 'a b c d', maxsplit=2)\n ['a', 'b', 'c d']\n\nGroup Matching\n--------------\n\nParentheses ``(...)`` create capturing groups. Use ``group()`` to access matched\ngroups. Group 0 is the entire match, group 1 is the first parenthesized group, etc.\n\n.. code-block:: python\n\n >>> m = re.search(r'(\\d{4})-(\\d{2})-(\\d{2})', '2016-01-01')\n >>> m.groups()\n ('2016', '01', '01')\n >>> m.group() # entire match\n '2016-01-01'\n >>> m.group(1) # first group\n '2016'\n >>> m.group(2, 3) # multiple groups\n ('01', '01')\n\n # Nested groups - numbered left to right by opening parenthesis\n >>> m = re.search(r'(((\\d{4})-\\d{2})-\\d{2})', '2016-01-01')\n >>> m.groups()\n ('2016-01-01', '2016-01', '2016')\n\nNon-Capturing Group ``(?:...)``\n-------------------------------\n\nUse ``(?:...)`` when you need grouping for alternation or quantifiers but don't\nneed to capture the match. This improves performance and keeps group numbering clean.\n\n.. code-block:: python\n\n >>> url = 'http://stackoverflow.com/'\n >>> # non-capturing group for protocol\n >>> m = re.search(r'(?:http|ftp)://([^/\\r\\n]+)(/[^\\r\\n]*)?', url)\n >>> m.groups()\n ('stackoverflow.com', '/')\n\n >>> # capturing group - protocol is captured\n >>> m = re.search(r'(http|ftp)://([^/\\r\\n]+)(/[^\\r\\n]*)?', url)\n >>> m.groups()\n ('http', 'stackoverflow.com', '/')\n\nNamed Groups ``(?P...)``\n------------------------------\n\nNamed groups make patterns more readable and allow access by name instead of\nnumber. Use ``(?P...)`` to define and ``(?P=name)`` for back reference.\n\n.. code-block:: python\n\n >>> pattern = r'(?P\\d{4})-(?P\\d{2})-(?P\\d{2})'\n >>> m = re.search(pattern, '2016-01-01')\n >>> m.group('year')\n '2016'\n >>> m.group('month')\n '01'\n >>> m.groupdict()\n {'year': '2016', 'month': '01', 'day': '01'}\n\n # named back reference\n >>> re.search(r'^(?P[a-z])(?P=char)', 'aa')\n \n >>> re.search(r'^(?P[a-z])(?P=char)', 'ab') is None\n True\n\nBack Reference ``\\1``, ``\\2``\n-----------------------------\n\nBack references match the same text as a previous capturing group. Use ``\\1``\nfor the first group, ``\\2`` for the second, etc.\n\n.. code-block:: python\n\n >>> # match repeated characters\n >>> re.search(r'([a-z])\\1', 'aa') is not None\n True\n >>> re.search(r'([a-z])\\1', 'ab') is not None\n False\n\n >>> # match HTML tags with matching close tag\n >>> pattern = r'<([^>]+)>[\\s\\S]*?'\n >>> re.search(pattern, 'test') is not None\n True\n >>> re.search(pattern, 'test') is not None\n False\n\nSubstitute with ``re.sub()``\n----------------------------\n\nThe ``sub()`` function replaces pattern matches with a replacement string.\nUse ``\\1``, ``\\2`` in the replacement to reference captured groups.\n\n.. code-block:: python\n\n >>> # basic substitution\n >>> re.sub(r'[a-z]', ' ', '1a2b3c')\n '1 2 3 '\n\n >>> # substitute with group reference\n >>> re.sub(r'(\\d{4})-(\\d{2})-(\\d{2})', r'\\2/\\3/\\1', '2016-01-01')\n '01/01/2016'\n\n >>> # using function as replacement\n >>> re.sub(r'\\d+', lambda m: str(int(m.group()) * 2), 'a1b2c3')\n 'a2b4c6'\n\n >>> # camelCase to snake_case\n >>> def to_snake(s):\n ... s = re.sub(r'(.)([A-Z][a-z]+)', r'\\1_\\2', s)\n ... return re.sub(r'([a-z])([A-Z])', r'\\1_\\2', s).lower()\n ...\n >>> to_snake('CamelCase')\n 'camel_case'\n >>> to_snake('SimpleHTTPServer')\n 'simple_http_server'\n\nLookahead and Lookbehind\n------------------------\n\nLookaround assertions match a position without consuming characters. They are\nuseful for matching patterns based on context.\n\n+---------------+---------------------+---------------------------+\n| Notation | Name | Description |\n+===============+=====================+===========================+\n| ``(?=...)`` | Positive lookahead | Followed by ... |\n+---------------+---------------------+---------------------------+\n| ``(?!...)`` | Negative lookahead | Not followed by ... |\n+---------------+---------------------+---------------------------+\n| ``(?<=...)`` | Positive lookbehind | Preceded by ... |\n+---------------+---------------------+---------------------------+\n| ``(?>> # positive lookahead - find word before @\n >>> re.findall(r'\\w+(?=@)', 'user@example.com')\n ['user']\n\n >>> # negative lookahead - find digits not followed by px\n >>> re.findall(r'\\d+(?!px)', '12px 34em 56')\n ['1', '34', '56']\n\n >>> # positive lookbehind - find digits after $\n >>> re.findall(r'(?<=\\$)\\d+', '$100 $200')\n ['100', '200']\n\n >>> # negative lookbehind - find digits not after $\n >>> re.findall(r'(?>> # insert space before groups of 3 digits from right\n >>> re.sub(r'(?=(\\d{3})+$)', ' ', '12345678')\n ' 12 345 678'\n\nCompile Pattern for Reuse\n-------------------------\n\nUse ``re.compile()`` to create a reusable pattern object. This improves\nperformance when the same pattern is used multiple times.\n\n.. code-block:: python\n\n >>> pattern = re.compile(r'\\d{4}-\\d{2}-\\d{2}')\n >>> pattern.search('Date: 2024-01-15')\n \n >>> pattern.findall('2024-01-15 and 2024-02-20')\n ['2024-01-15', '2024-02-20']\n\nRegex Flags\n-----------\n\nFlags modify pattern behavior. Common flags include ``re.IGNORECASE`` (``re.I``),\n``re.MULTILINE`` (``re.M``), ``re.DOTALL`` (``re.S``), and ``re.VERBOSE`` (``re.X``).\n\n.. code-block:: python\n\n >>> # case insensitive\n >>> re.findall(r'[a-z]+', 'Hello World', re.I)\n ['Hello', 'World']\n\n >>> # multiline - ^ and $ match line boundaries\n >>> re.findall(r'^\\w+', 'line1\\nline2', re.M)\n ['line1', 'line2']\n\n >>> # dotall - . matches newline\n >>> re.search(r'a.b', 'a\\nb', re.S)\n \n\n >>> # verbose - allow comments and whitespace\n >>> pattern = re.compile(r'''\n ... \\d{4} # year\n ... -\n ... \\d{2} # month\n ... -\n ... \\d{2} # day\n ... ''', re.X)\n >>> pattern.match('2024-01-15')\n \n\nCompare HTML Tags\n-----------------\n\nCommon patterns for matching different types of HTML tags.\n\n+------------+--------------+--------------+\n| Tag Type | Pattern | Example |\n+============+==============+==============+\n| All tags | <[^>]+> | , |\n+------------+--------------+--------------+\n| Open tag | <[^/>][^>]*> | ,
|\n+------------+--------------+--------------+\n| Close tag | ]+> | , |\n+------------+--------------+--------------+\n| Self-close | <[^/>]+/> | |\n+------------+--------------+--------------+\n\n.. code-block:: python\n\n >>> # open tag\n >>> re.search(r'<[^/>][^>]*>', '
') is not None\n True\n >>> re.search(r'<[^/>][^>]*>', '
') is not None\n False\n\n >>> # close tag\n >>> re.search(r']+>', '
') is not None\n True\n\n >>> # self-closing tag\n >>> re.search(r'<[^/>]+/>', ' ') is not None\n True\n\nMatch Email Address\n-------------------\n\nA pattern for validating email addresses. Note that fully RFC-compliant email\nvalidation is extremely complex; this covers common cases.\n\n.. code-block:: python\n\n >>> pattern = re.compile(r'^[\\w.+-]+@[\\w-]+\\.[\\w.-]+$')\n >>> pattern.match('hello.world@example.com') is not None\n True\n >>> pattern.match('user+tag@sub.domain.org') is not None\n True\n >>> pattern.match('invalid@') is not None\n False\n\nMatch URL\n---------\n\nA pattern for matching URLs with optional protocol, domain, and path.\n\n.. code-block:: python\n\n >>> pattern = re.compile(r'''\n ... ^(https?://)? # optional protocol\n ... ([\\da-z.-]+) # domain\n ... \\.([a-z.]{2,6}) # TLD\n ... ([/\\w.-]*)*/?$ # path\n ... ''', re.X | re.I)\n >>> pattern.match('https://www.example.com/path') is not None\n True\n >>> pattern.match('example.com') is not None\n True\n\nMatch IP Address\n----------------\n\nA pattern for validating IPv4 addresses (0.0.0.0 to 255.255.255.255).\n\n.. code-block:: python\n\n >>> pattern = re.compile(r'''\n ... ^(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}\n ... (?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$\n ... ''', re.X)\n >>> pattern.match('192.168.1.1') is not None\n True\n >>> pattern.match('255.255.255.0') is not None\n True\n >>> pattern.match('256.0.0.0') is not None\n False\n\nMatch MAC Address\n-----------------\n\nA pattern for validating MAC addresses in colon-separated format.\n\n.. code-block:: python\n\n >>> pattern = re.compile(r'^([0-9a-f]{2}:){5}[0-9a-f]{2}$', re.I)\n >>> pattern.match('3c:38:51:05:03:1e') is not None\n True\n >>> pattern.match('AA:BB:CC:DD:EE:FF') is not None\n True\n\nMatch Phone Number\n------------------\n\nPatterns for common phone number formats.\n\n.. code-block:: python\n\n >>> # US phone number\n >>> pattern = re.compile(r'^(\\+1)?[-.\\s]?\\(?\\d{3}\\)?[-.\\s]?\\d{3}[-.\\s]?\\d{4}$')\n >>> pattern.match('123-456-7890') is not None\n True\n >>> pattern.match('(123) 456-7890') is not None\n True\n >>> pattern.match('+1 123 456 7890') is not None\n True\n\nMatch Password Strength\n-----------------------\n\nPattern to validate password with minimum requirements: at least 8 characters,\none uppercase, one lowercase, one digit, and one special character.\n\n.. code-block:: python\n\n >>> pattern = re.compile(r'''\n ... ^(?=.*[a-z]) # at least one lowercase\n ... (?=.*[A-Z]) # at least one uppercase\n ... (?=.*\\d) # at least one digit\n ... (?=.*[@$!%*?&]) # at least one special char\n ... [A-Za-z\\d@$!%*?&]{8,}$ # at least 8 chars\n ... ''', re.X)\n >>> pattern.match('Passw0rd!') is not None\n True\n >>> pattern.match('weakpass') is not None\n False\n\nSimple Lexer\n------------\n\nUsing regex to build a simple tokenizer for arithmetic expressions. This\ndemonstrates using named groups and ``scanner()`` for lexical analysis.\n\n.. code-block:: python\n\n >>> from collections import namedtuple\n >>> tokens = [\n ... r'(?P\\d+)',\n ... r'(?P\\+)',\n ... r'(?P-)',\n ... r'(?P\\*)',\n ... r'(?P/)',\n ... r'(?P\\s+)'\n ... ]\n >>> lex = re.compile('|'.join(tokens))\n >>> Token = namedtuple('Token', ['type', 'value'])\n >>> def tokenize(text):\n ... scan = lex.scanner(text)\n ... return (Token(m.lastgroup, m.group())\n ... for m in iter(scan.match, None) if m.lastgroup != 'WS')\n ...\n >>> list(tokenize('9 + 5 * 2'))\n [Token(type='NUMBER', value='9'), Token(type='PLUS', value='+'), Token(type='NUMBER', value='5'), Token(type='TIMES', value='*'), Token(type='NUMBER', value='2')]\n\nCommon Patterns Reference\n-------------------------\n\n.. code-block:: python\n\n # Digits only\n r'^\\d+$'\n\n # Alphanumeric\n r'^[a-zA-Z0-9]+$'\n\n # Username (3-16 chars, alphanumeric, underscore, hyphen)\n r'^[a-zA-Z0-9_-]{3,16}$'\n\n # Hex color\n r'^#?([a-fA-F0-9]{6}|[a-fA-F0-9]{3})$'\n\n # Date (YYYY-MM-DD)\n r'^\\d{4}-\\d{2}-\\d{2}$'\n\n # Time (HH:MM:SS)\n r'^\\d{2}:\\d{2}:\\d{2}$'\n\n # Slug (URL-friendly string)\n r'^[a-z0-9]+(?:-[a-z0-9]+)*$'\n\n # Remove HTML tags\n re.sub(r'<[^>]+>', '', html)\n\n # Extract domain from URL\n re.search(r'https?://([^/]+)', url).group(1)\n\n # Find all hashtags\n re.findall(r'#\\w+', text)\n\n # Find all @mentions\n re.findall(r'@\\w+', text)\n"
},
{
"path": "docs/notes/basic/python-set.rst",
"content": ".. meta::\n :description lang=en: Python set cheat sheet covering set comprehensions, set operations (union, intersection, difference), removing duplicates, subsets, supersets, and frozenset with code examples\n :keywords: Python, Python3, Python set, Python set cheat sheet, set comprehension, set operations, union, intersection, difference, symmetric difference, frozenset, subset, superset\n\n===\nSet\n===\n\n.. contents:: Table of Contents\n :backlinks: none\n\nSets are unordered collections of unique elements in Python. They provide O(1)\naverage time complexity for membership testing and support mathematical set\noperations like union, intersection, and difference. This cheat sheet covers\nset comprehensions, common set operations, uniquifying lists, and the immutable\nfrozenset type.\n\nThe source code is available on `GitHub `_.\n\nReferences\n----------\n\n- `Set Types — set, frozenset `_\n- `Sets `_\n\nCreate a Set\n------------\n\nCreate sets using curly braces ``{}`` or the ``set()`` constructor. Note that\nempty curly braces ``{}`` create a dict, not a set.\n\n.. code-block:: python\n\n >>> s = {1, 2, 3}\n >>> s\n {1, 2, 3}\n >>> s = set([1, 2, 2, 3])\n >>> s\n {1, 2, 3}\n >>> empty = set() # not {}\n >>> type(empty)\n \n\nCreate Sets with Set Comprehension\n----------------------------------\n\nLike list comprehensions, set comprehensions provide a concise way to create sets.\nThe syntax uses curly braces ``{}`` instead of square brackets.\n\n.. code-block:: python\n\n >>> a = [1, 2, 5, 6, 6, 6, 7]\n >>> s = {x for x in a}\n >>> s\n {1, 2, 5, 6, 7}\n >>> s = {x for x in a if x > 3}\n >>> s\n {5, 6, 7}\n >>> s = {x ** 2 for x in range(5)}\n >>> s\n {0, 1, 4, 9, 16}\n\nRemove Duplicates from a List\n-----------------------------\n\nConverting a list to a set automatically removes duplicate elements. This is one\nof the most common use cases for sets.\n\n.. code-block:: python\n\n >>> a = [1, 2, 2, 2, 3, 4, 5, 5]\n >>> list(set(a))\n [1, 2, 3, 4, 5]\n\nTo preserve the original order, use ``dict.fromkeys()`` (Python 3.7+):\n\n.. code-block:: python\n\n >>> a = [3, 1, 2, 1, 3, 2]\n >>> list(dict.fromkeys(a))\n [3, 1, 2]\n\nAdd Items to a Set\n------------------\n\nUse ``add()`` to add a single element, or ``update()`` to add multiple elements.\n\n.. code-block:: python\n\n >>> s = {1, 2, 3}\n >>> s.add(4)\n >>> s\n {1, 2, 3, 4}\n >>> s.update([5, 6, 7])\n >>> s\n {1, 2, 3, 4, 5, 6, 7}\n >>> s |= {8, 9} # same as update\n >>> s\n {1, 2, 3, 4, 5, 6, 7, 8, 9}\n\nRemove Items from a Set\n-----------------------\n\nUse ``remove()`` to remove an element (raises KeyError if not found), or\n``discard()`` to remove without error. Use ``pop()`` to remove an arbitrary element.\n\n.. code-block:: python\n\n >>> s = {1, 2, 3, 4, 5}\n >>> s.remove(3)\n >>> s\n {1, 2, 4, 5}\n >>> s.discard(10) # no error if not found\n >>> s.pop() # remove arbitrary element\n 1\n >>> s.clear() # remove all\n >>> s\n set()\n\nUnion with ``|`` Operator\n-------------------------\n\nThe union of two sets contains all elements from both sets. Use the ``|`` operator\nor the ``union()`` method.\n\n.. code-block:: python\n\n >>> a = {1, 2, 3}\n >>> b = {3, 4, 5}\n >>> a | b\n {1, 2, 3, 4, 5}\n >>> a.union(b)\n {1, 2, 3, 4, 5}\n >>> a | b | {6, 7} # multiple sets\n {1, 2, 3, 4, 5, 6, 7}\n\nIntersection with ``&`` Operator\n--------------------------------\n\nThe intersection of two sets contains only elements that exist in both sets.\nUse the ``&`` operator or the ``intersection()`` method.\n\n.. code-block:: python\n\n >>> a = {1, 2, 3, 4}\n >>> b = {3, 4, 5, 6}\n >>> a & b\n {3, 4}\n >>> a.intersection(b)\n {3, 4}\n\nFind Common Elements Between Lists\n----------------------------------\n\nFinding common items between two lists is a practical application of set\nintersection.\n\n.. code-block:: python\n\n >>> a = [1, 1, 2, 3]\n >>> b = [3, 5, 5, 6]\n >>> list(set(a) & set(b))\n [3]\n\nDifference with ``-`` Operator\n------------------------------\n\nThe difference of two sets contains elements that are in the first set but not\nin the second. Use the ``-`` operator or the ``difference()`` method.\n\n.. code-block:: python\n\n >>> a = {1, 2, 3, 4}\n >>> b = {3, 4, 5, 6}\n >>> a - b\n {1, 2}\n >>> b - a\n {5, 6}\n\nSymmetric Difference with ``^`` Operator\n----------------------------------------\n\nThe symmetric difference contains elements that are in either set, but not in\nboth. Use the ``^`` operator or the ``symmetric_difference()`` method.\n\n.. code-block:: python\n\n >>> a = {1, 2, 3}\n >>> b = {3, 4, 5}\n >>> a ^ b\n {1, 2, 4, 5}\n\nCheck Subset with ``<=`` Operator\n---------------------------------\n\nUse ``<=`` or ``issubset()`` to check if all elements of one set are in another.\nUse ``<`` for proper subset (subset but not equal).\n\n.. code-block:: python\n\n >>> a = {1, 2}\n >>> b = {1, 2, 3, 4}\n >>> a <= b # a is subset of b\n True\n >>> a < b # a is proper subset\n True\n >>> a <= a # equal sets\n True\n >>> a < a # not proper subset\n False\n\nCheck Superset with ``>=`` Operator\n-----------------------------------\n\nUse ``>=`` or ``issuperset()`` to check if a set contains all elements of another.\n\n.. code-block:: python\n\n >>> a = {1, 2, 3, 4}\n >>> b = {1, 2}\n >>> a >= b # a is superset of b\n True\n >>> a > b # a is proper superset\n True\n\nCheck Disjoint Sets\n-------------------\n\nTwo sets are disjoint if they have no elements in common. Use ``isdisjoint()``\nto check.\n\n.. code-block:: python\n\n >>> a = {1, 2, 3}\n >>> b = {4, 5, 6}\n >>> a.isdisjoint(b)\n True\n >>> c = {3, 4, 5}\n >>> a.isdisjoint(c)\n False\n\nMembership Testing\n------------------\n\nSets provide O(1) average time complexity for membership testing, making them\nmuch faster than lists for this operation.\n\n.. code-block:: python\n\n >>> s = {1, 2, 3, 4, 5}\n >>> 3 in s\n True\n >>> 10 in s\n False\n >>> 10 not in s\n True\n\nFrozenset - Immutable Set\n-------------------------\n\n``frozenset`` is an immutable version of set. It can be used as a dictionary key\nor as an element of another set.\n\n.. code-block:: python\n\n >>> fs = frozenset([1, 2, 3])\n >>> fs\n frozenset({1, 2, 3})\n >>> fs.add(4) # raises AttributeError\n AttributeError: 'frozenset' object has no attribute 'add'\n\nUse frozenset as dictionary key:\n\n.. code-block:: python\n\n >>> d = {frozenset([1, 2]): \"a\", frozenset([3, 4]): \"b\"}\n >>> d[frozenset([1, 2])]\n 'a'\n\nUse frozenset in a set:\n\n.. code-block:: python\n\n >>> s = {frozenset([1, 2]), frozenset([3, 4])}\n >>> frozenset([1, 2]) in s\n True\n\nSet Operations Summary\n----------------------\n\n.. code-block:: python\n\n # Creation\n s = {1, 2, 3} # literal\n s = set([1, 2, 3]) # from iterable\n s = {x for x in range(5)} # comprehension\n\n # Add/Remove\n s.add(x) # add single element\n s.update([x, y]) # add multiple elements\n s.remove(x) # remove (KeyError if missing)\n s.discard(x) # remove (no error if missing)\n s.pop() # remove arbitrary element\n s.clear() # remove all\n\n # Set Operations\n a | b # union\n a & b # intersection\n a - b # difference\n a ^ b # symmetric difference\n\n # Comparisons\n a <= b # subset\n a < b # proper subset\n a >= b # superset\n a > b # proper superset\n a.isdisjoint(b) # no common elements\n\n # Membership\n x in s # O(1) lookup\n x not in s\n"
},
{
"path": "docs/notes/basic/python-typing.rst",
"content": ".. meta::\n :description lang=en: Python typing cheat sheet covering type hints, annotations, generics, protocols, TypeVar, and mypy type checking with code examples\n :keywords: Python, Python3, Python typing, Python type hints cheat sheet, type annotations, generics, Protocol, TypeVar, mypy, static typing\n\n======\nTyping\n======\n\n.. contents:: Table of Contents\n :backlinks: none\n\nPEP `484 `_, which provides a\nspecification about what a type system should look like in Python3, introduced\nthe concept of type hints. Moreover, to better understand the type hints design\nphilosophy, it is crucial to read PEP `483 `_\nthat would be helpful to aid a pythoneer to understand reasons why Python\nintroduce a type system. The main goal of this cheat sheet is to show some\ncommon usage about type hints in Python3.\n\n\nWithout type check\n-------------------\n\n.. code-block:: python\n\n def fib(n):\n a, b = 0, 1\n for _ in range(n):\n yield a\n b, a = a + b, b\n\n print([n for n in fib(3.6)])\n\n\noutput:\n\n.. code-block:: bash\n\n # errors will not be detected until runtime\n\n $ python fib.py\n Traceback (most recent call last):\n File \"fib.py\", line 8, in \n print([n for n in fib(3.5)])\n File \"fib.py\", line 8, in \n print([n for n in fib(3.5)])\n File \"fib.py\", line 3, in fib\n for _ in range(n):\n TypeError: 'float' object cannot be interpreted as an integer\n\n\nWith type check\n----------------\n\n.. code-block:: python\n\n # give a type hint\n from typing import Generator\n\n def fib(n: int) -> Generator:\n a: int = 0\n b: int = 1\n for _ in range(n):\n yield a\n b, a = a + b, b\n\n print([n for n in fib(3.6)])\n\noutput:\n\n.. code-block:: bash\n\n # errors will be detected before running\n\n $ mypy --strict fib.py\n fib.py:12: error: Argument 1 to \"fib\" has incompatible type \"float\"; expected \"int\"\n\nBasic types\n-----------\n\n.. code-block:: python\n\n import io\n import re\n\n from collections import deque, namedtuple\n from typing import (\n Dict,\n List,\n Tuple,\n Set,\n Deque,\n NamedTuple,\n IO,\n Pattern,\n Match,\n Text,\n Optional,\n Sequence,\n Iterable,\n Mapping,\n MutableMapping,\n Any,\n )\n\n # without initializing\n x: int\n\n # any type\n y: Any\n y = 1\n y = \"1\"\n\n # built-in\n var_int: int = 1\n var_str: str = \"Hello Typing\"\n var_byte: bytes = b\"Hello Typing\"\n var_bool: bool = True\n var_float: float = 1.\n var_unicode: Text = u'\\u2713'\n\n # could be none\n var_could_be_none: Optional[int] = None\n var_could_be_none = 1\n\n # collections\n var_set: Set[int] = {i for i in range(3)}\n var_dict: Dict[str, str] = {\"foo\": \"Foo\"}\n var_list: List[int] = [i for i in range(3)]\n var_static_length_Tuple: Tuple[int, int, int] = (1, 2, 3)\n var_dynamic_length_Tuple: Tuple[int, ...] = (i for i in range(10, 3))\n var_deque: Deque = deque([1, 2, 3])\n var_nametuple: NamedTuple = namedtuple('P', ['x', 'y'])\n\n # io\n var_io_str: IO[str] = io.StringIO(\"Hello String\")\n var_io_byte: IO[bytes] = io.BytesIO(b\"Hello Bytes\")\n var_io_file_str: IO[str] = open(__file__)\n var_io_file_byte: IO[bytes] = open(__file__, 'rb')\n\n # re\n p: Pattern = re.compile(\"(https?)://([^/\\r\\n]+)(/[^\\r\\n]*)?\")\n m: Optional[Match] = p.match(\"https://www.python.org/\")\n\n # duck types: list-like\n var_seq_list: Sequence[int] = [1, 2, 3]\n var_seq_tuple: Sequence[int] = (1, 2, 3)\n var_iter_list: Iterable[int] = [1, 2, 3]\n var_iter_tuple: Iterable[int] = (1, 2, 3)\n\n # duck types: dict-like\n var_map_dict: Mapping[str, str] = {\"foo\": \"Foo\"}\n var_mutable_dict: MutableMapping[str, str] = {\"bar\": \"Bar\"}\n\nFunctions\n----------\n\n.. code-block:: python\n\n from typing import Generator, Callable\n\n # function\n def gcd(a: int, b: int) -> int:\n while b:\n a, b = b, a % b\n return a\n\n # callback\n def fun(cb: Callable[[int, int], int]) -> int:\n return cb(55, 66)\n\n # lambda\n f: Callable[[int], int] = lambda x: x * 2\n\nClasses\n--------\n\n.. code-block:: python\n\n from typing import ClassVar, Dict, List\n\n class Foo:\n\n x: int = 1 # instance variable. default = 1\n y: ClassVar[str] = \"class var\" # class variable\n\n def __init__(self) -> None:\n self.i: List[int] = [0]\n\n def foo(self, a: int, b: str) -> Dict[int, str]:\n return {a: b}\n\n foo = Foo()\n foo.x = 123\n\n print(foo.x)\n print(foo.i)\n print(Foo.y)\n print(foo.foo(1, \"abc\"))\n\nGenerator\n----------\n\n.. code-block:: python\n\n from typing import Generator\n\n # Generator[YieldType, SendType, ReturnType]\n def fib(n: int) -> Generator[int, None, None]:\n a: int = 0\n b: int = 1\n while n > 0:\n yield a\n b, a = a + b, b\n n -= 1\n\n g: Generator = fib(10)\n i: Iterator[int] = (x for x in range(3))\n\nAsynchronous Generator\n-----------------------\n\n.. code-block:: python\n\n import asyncio\n\n from typing import AsyncGenerator, AsyncIterator\n\n async def fib(n: int) -> AsyncGenerator:\n a: int = 0\n b: int = 1\n while n > 0:\n await asyncio.sleep(0.1)\n yield a\n\n b, a = a + b, b\n n -= 1\n\n async def main() -> None:\n async for f in fib(10):\n print(f)\n\n ag: AsyncIterator = (f async for f in fib(10))\n\n loop = asyncio.get_event_loop()\n loop.run_until_complete(main())\n\nContext Manager\n---------------\n\n.. code-block:: python\n\n from typing import ContextManager, Generator, IO\n from contextlib import contextmanager\n\n @contextmanager\n def open_file(name: str) -> Generator:\n f = open(name)\n yield f\n f.close()\n\n cm: ContextManager[IO] = open_file(__file__)\n with cm as f:\n print(f.read())\n\nAsynchronous Context Manager\n-----------------------------\n\n.. code-block:: python\n\n import asyncio\n\n from typing import AsyncContextManager, AsyncGenerator, IO\n from contextlib import asynccontextmanager\n\n # need python 3.7 or above\n @asynccontextmanager\n async def open_file(name: str) -> AsyncGenerator:\n await asyncio.sleep(0.1)\n f = open(name)\n yield f\n await asyncio.sleep(0.1)\n f.close()\n\n async def main() -> None:\n acm: AsyncContextManager[IO] = open_file(__file__)\n async with acm as f:\n print(f.read())\n\n loop = asyncio.get_event_loop()\n loop.run_until_complete(main())\n\nAvoid ``None`` access\n----------------------\n\n.. code-block:: python\n\n import re\n\n from typing import Pattern, Dict, Optional\n\n # like c++\n # std::regex url(\"(https?)://([^/\\r\\n]+)(/[^\\r\\n]*)?\");\n # std::regex color(\"^#?([a-f0-9]{6}|[a-f0-9]{3})$\");\n\n url: Pattern = re.compile(\"(https?)://([^/\\r\\n]+)(/[^\\r\\n]*)?\")\n color: Pattern = re.compile(\"^#?([a-f0-9]{6}|[a-f0-9]{3})$\")\n\n x: Dict[str, Pattern] = {\"url\": url, \"color\": color}\n y: Optional[Pattern] = x.get(\"baz\", None)\n\n print(y.match(\"https://www.python.org/\"))\n\noutput:\n\n.. code-block:: bash\n\n $ mypy --strict foo.py\n foo.py:15: error: Item \"None\" of \"Optional[Pattern[Any]]\" has no attribute \"match\"\n\nPositional-only arguments\n--------------------------\n\n.. code-block:: python\n\n # define arguments with names beginning with __\n\n def fib(__n: int) -> int: # positional only arg\n a, b = 0, 1\n for _ in range(__n):\n b, a = a + b, b\n return a\n\n\n def gcd(*, a: int, b: int) -> int: # keyword only arg\n while b:\n a, b = b, a % b\n return a\n\n\n print(fib(__n=10)) # error\n print(gcd(10, 5)) # error\n\noutput:\n\n.. code-block:: bash\n\n mypy --strict foo.py\n foo.py:1: note: \"fib\" defined here\n foo.py:14: error: Unexpected keyword argument \"__n\" for \"fib\"\n foo.py:15: error: Too many positional arguments for \"gcd\"\n\nMultiple return values\n-----------------------\n\n.. code-block:: python\n\n from typing import Tuple, Iterable, Union\n\n def foo(x: int, y: int) -> Tuple[int, int]:\n return x, y\n\n # or\n\n def bar(x: int, y: str) -> Iterable[Union[int, str]]:\n # XXX: not recommend declaring in this way\n return x, y\n\n a: int\n b: int\n a, b = foo(1, 2) # ok\n c, d = bar(3, \"bar\") # ok\n\nOptional Type\n----------------------------------\n\n.. code-block:: python\n\n from typing import List, Union\n\n def first(l: List[Union[int, None]]) -> Union[int, None]:\n return None if len(l) == 0 else l[0]\n\n first([None])\n\n # equal to\n\n from typing import List, Optional\n\n def first(l: List[Optional[int]]) -> Optional[int]:\n return None if len(l) == 0 else l[0]\n\n first([None])\n\nBe careful of ``Optional``\n---------------------------\n\n.. code-block:: python\n\n from typing import cast, Optional\n\n def fib(n):\n a, b = 0, 1\n for _ in range(n):\n b, a = a + b, b\n return a\n\n def cal(n: Optional[int]) -> None:\n print(fib(n))\n\n cal(None)\n\noutput:\n\n.. code-block:: bash\n\n # mypy will not detect errors\n $ mypy foo.py\n\nExplicitly declare\n\n.. code-block:: python\n\n from typing import Optional\n\n def fib(n: int) -> int: # declare n to be int\n a, b = 0, 1\n for _ in range(n):\n b, a = a + b, b\n return a\n\n def cal(n: Optional[int]) -> None:\n print(fib(n))\n\noutput:\n\n.. code-block:: bash\n\n # mypy can detect errors even we do not check None\n $ mypy --strict foo.py\n foo.py:11: error: Argument 1 to \"fib\" has incompatible type \"Optional[int]\"; expected \"int\"\n\nBe careful of casting\n----------------------\n\n.. code-block:: python\n\n from typing import cast, Optional\n\n def gcd(a: int, b: int) -> int:\n while b:\n a, b = b, a % b\n return a\n\n def cal(a: Optional[int], b: Optional[int]) -> None:\n # XXX: Avoid casting\n ca, cb = cast(int, a), cast(int, b)\n print(gcd(ca, cb))\n\n cal(None, None)\n\noutput:\n\n.. code-block:: bash\n\n # mypy will not detect type errors\n $ mypy --strict foo.py\n\n\nForward references\n-------------------\n\nBased on PEP 484, if we want to reference a type before it has been declared, we\nhave to use **string literal** to imply that there is a type of that name later on\nin the file.\n\n.. code-block:: python\n\n from typing import Optional\n\n\n class Tree:\n def __init__(\n self, data: int,\n left: Optional[\"Tree\"], # Forward references.\n right: Optional[\"Tree\"]\n ) -> None:\n self.data = data\n self.left = left\n self.right = right\n\n.. note::\n\n There are some issues that mypy does not complain about Forward References.\n Get further information from `Issue#948`_.\n\n.. _Issue\\#948: https://github.com/python/mypy/issues/948\n\n.. code-block:: python\n\n class A:\n def __init__(self, a: A) -> None: # should fail\n self.a = a\n\noutput:\n\n.. code-block:: bash\n\n $ mypy --strict type.py\n $ echo $?\n 0\n $ python type.py # get runtime fail\n Traceback (most recent call last):\n File \"type.py\", line 1, in \n class A:\n File \"type.py\", line 2, in A\n def __init__(self, a: A) -> None: # should fail\n NameError: name 'A' is not defined\n\nPostponed Evaluation of Annotations\n-----------------------------------\n\n**New in Python 3.7**\n\n- PEP 563_ - Postponed Evaluation of Annotations\n\n.. _563: https://www.python.org/dev/peps/pep-0563/\n\nBefore Python 3.7\n\n.. code-block:: python\n\n >>> class A:\n ... def __init__(self, a: A) -> None:\n ... self._a = a\n ...\n Traceback (most recent call last):\n File \"\", line 1, in \n File \"\", line 2, in A\n NameError: name 'A' is not defined\n\nAfter Python 3.7 (include 3.7)\n\n.. code-block:: python\n\n >>> from __future__ import annotations\n >>> class A:\n ... def __init__(self, a: A) -> None:\n ... self._a = a\n ...\n\n.. note::\n\n Annotation can only be used within the scope which names have already\n existed. Therefore, **forward reference** does not support the case which\n names are not available in the current scope. **Postponed evaluation\n of annotations** will become the default behavior in Python 4.0.\n\nType Alias\n----------\n\nLike ``typedef`` or ``using`` in c/c++\n\n.. code-block:: cpp\n\n #include \n #include \n #include \n #include \n\n typedef std::string Url;\n template using Vector = std::vector;\n\n int main(int argc, char *argv[])\n {\n Url url = \"https://python.org\";\n std::regex p(\"(https?)://([^/\\r\\n]+)(/[^\\r\\n]*)?\");\n bool m = std::regex_match(url, p);\n Vector v = {1, 2};\n\n std::cout << m << std::endl;\n for (auto it : v) std::cout << it << std::endl;\n return 0;\n }\n\nType aliases are defined by simple variable assignments\n\n.. code-block:: python\n\n import re\n\n from typing import Pattern, List\n\n # Like typedef, using in c/c++\n\n # PEP 484 recommend capitalizing alias names\n Url = str\n\n url: Url = \"https://www.python.org/\"\n\n p: Pattern = re.compile(\"(https?)://([^/\\r\\n]+)(/[^\\r\\n]*)?\")\n m = p.match(url)\n\n Vector = List[int]\n v: Vector = [1., 2.]\n\nUsing NewType\n---------------------\n\nUnlike alias, ``NewType`` returns a separate type but is identical to the original type at runtime.\n\n.. code-block:: python\n\n from sqlalchemy import Column, String, Integer\n from sqlalchemy.ext.declarative import declarative_base\n from typing import NewType, Any\n\n # check mypy #2477\n Base: Any = declarative_base()\n\n # create a new type\n Id = NewType('Id', int) # not equal alias, it's a 'new type'\n\n class User(Base):\n __tablename__ = 'User'\n id = Column(Integer, primary_key=True)\n age = Column(Integer, nullable=False)\n name = Column(String, nullable=False)\n\n def __init__(self, id: Id, age: int, name: str) -> None:\n self.id = id\n self.age = age\n self.name = name\n\n # create users\n user1 = User(Id(1), 62, \"Guido van Rossum\") # ok\n user2 = User(2, 48, \"David M. Beazley\") # error\n\noutput:\n\n.. code-block:: bash\n\n $ python foo.py\n $ mypy --ignore-missing-imports foo.py\n foo.py:24: error: Argument 1 to \"User\" has incompatible type \"int\"; expected \"Id\"\n\nFurther reading:\n\n- `Issue\\#1284`_\n\n.. _`Issue\\#1284`: https://github.com/python/mypy/issues/1284\n\n\nUsing ``TypeVar`` as template\n------------------------------\n\nLike c++ ``template ``\n\n.. code-block:: cpp\n\n #include \n\n template \n T add(T x, T y) {\n return x + y;\n }\n\n int main(int argc, char *argv[])\n {\n std::cout << add(1, 2) << std::endl;\n std::cout << add(1., 2.) << std::endl;\n return 0;\n }\n\nPython using ``TypeVar``\n\n.. code-block:: python\n\n from typing import TypeVar\n\n T = TypeVar(\"T\")\n\n def add(x: T, y: T) -> T:\n return x + y\n\n add(1, 2)\n add(1., 2.)\n\nUsing ``TypeVar`` and ``Generic`` as class template\n----------------------------------------------------\n\nLike c++ ``template class``\n\n.. code-block:: cpp\n\n #include \n\n template\n class Foo {\n public:\n Foo(T foo) {\n foo_ = foo;\n }\n T Get() {\n return foo_;\n }\n private:\n T foo_;\n };\n\n int main(int argc, char *argv[])\n {\n Foo f(123);\n std::cout << f.Get() << std::endl;\n return 0;\n }\n\nDefine a generic class in Python\n\n.. code-block:: python\n\n from typing import Generic, TypeVar\n\n T = TypeVar(\"T\")\n\n class Foo(Generic[T]):\n def __init__(self, foo: T) -> None:\n self.foo = foo\n\n def get(self) -> T:\n return self.foo\n\n f: Foo[str] = Foo(\"Foo\")\n v: int = f.get()\n\noutput:\n\n.. code-block:: bash\n\n $ mypy --strict foo.py\n foo.py:13: error: Incompatible types in assignment (expression has type \"str\", variable has type \"int\")\n\nScoping rules for ``TypeVar``\n------------------------------\n\n- ``TypeVar`` used in different generic function will be inferred to be different types.\n\n.. code-block:: python\n\n from typing import TypeVar\n\n T = TypeVar(\"T\")\n\n def foo(x: T) -> T:\n return x\n\n def bar(y: T) -> T:\n return y\n\n a: int = foo(1) # ok: T is inferred to be int\n b: int = bar(\"2\") # error: T is inferred to be str\n\noutput:\n\n.. code-block:: bash\n\n $ mypy --strict foo.py\n foo.py:12: error: Incompatible types in assignment (expression has type \"str\", variable has type \"int\")\n\n- ``TypeVar`` used in a generic class will be inferred to be same types.\n\n.. code-block:: python\n\n from typing import TypeVar, Generic\n\n T = TypeVar(\"T\")\n\n class Foo(Generic[T]):\n\n def foo(self, x: T) -> T:\n return x\n\n def bar(self, y: T) -> T:\n return y\n\n f: Foo[int] = Foo()\n a: int = f.foo(1) # ok: T is inferred to be int\n b: str = f.bar(\"2\") # error: T is expected to be int\n\noutput:\n\n.. code-block:: bash\n\n $ mypy --strict foo.py\n foo.py:15: error: Incompatible types in assignment (expression has type \"int\", variable has type \"str\")\n foo.py:15: error: Argument 1 to \"bar\" of \"Foo\" has incompatible type \"str\"; expected \"int\"\n\n- ``TypeVar`` used in a method but did not match any parameters which declare in ``Generic`` can be inferred to be different types.\n\n.. code-block:: python\n\n from typing import TypeVar, Generic\n\n T = TypeVar(\"T\")\n S = TypeVar(\"S\")\n\n class Foo(Generic[T]): # S does not match params\n\n def foo(self, x: T, y: S) -> S:\n return y\n\n def bar(self, z: S) -> S:\n return z\n\n f: Foo[int] = Foo()\n a: str = f.foo(1, \"foo\") # S is inferred to be str\n b: int = f.bar(12345678) # S is inferred to be int\n\noutput:\n\n.. code-block:: bash\n\n $ mypy --strict foo.py\n\n- ``TypeVar`` should not appear in body of method/function if it is unbound type.\n\n.. code-block:: python\n\n from typing import TypeVar, Generic\n\n T = TypeVar(\"T\")\n S = TypeVar(\"S\")\n\n def foo(x: T) -> None:\n a: T = x # ok\n b: S = 123 # error: invalid type\n\noutput:\n\n.. code-block:: bash\n\n $ mypy --strict foo.py\n foo.py:8: error: Invalid type \"foo.S\"\n\nRestricting to a fixed set of possible types\n----------------------------------------------\n\n``T = TypeVar('T', ClassA, ...)`` means we create a **type variable with a value restriction**.\n\n.. code-block:: python\n\n from typing import TypeVar\n\n # restrict T = int or T = float\n T = TypeVar(\"T\", int, float)\n\n def add(x: T, y: T) -> T:\n return x + y\n\n add(1, 2)\n add(1., 2.)\n add(\"1\", 2)\n add(\"hello\", \"world\")\n\noutput:\n\n.. code-block:: bash\n\n # mypy can detect wrong type\n $ mypy --strict foo.py\n foo.py:10: error: Value of type variable \"T\" of \"add\" cannot be \"object\"\n foo.py:11: error: Value of type variable \"T\" of \"add\" cannot be \"str\"\n\n``TypeVar`` with an upper bound\n--------------------------------\n\n``T = TypeVar('T', bound=BaseClass)`` means we create a **type variable with an upper bound**.\nThe concept is similar to **polymorphism** in c++.\n\n.. code-block:: cpp\n\n #include \n\n class Shape {\n public:\n Shape(double width, double height) {\n width_ = width;\n height_ = height;\n };\n virtual double Area() = 0;\n protected:\n double width_;\n double height_;\n };\n\n class Rectangle: public Shape {\n public:\n Rectangle(double width, double height)\n :Shape(width, height)\n {};\n\n double Area() {\n return width_ * height_;\n };\n };\n\n class Triangle: public Shape {\n public:\n Triangle(double width, double height)\n :Shape(width, height)\n {};\n\n double Area() {\n return width_ * height_ / 2;\n };\n };\n\n double Area(Shape &s) {\n return s.Area();\n }\n\n int main(int argc, char *argv[])\n {\n Rectangle r(1., 2.);\n Triangle t(3., 4.);\n\n std::cout << Area(r) << std::endl;\n std::cout << Area(t) << std::endl;\n return 0;\n }\n\nLike c++, create a base class and ``TypeVar`` which bounds to the base class.\nThen, static type checker will take every subclass as type of base class.\n\n.. code-block:: python\n\n from typing import TypeVar\n\n\n class Shape:\n def __init__(self, width: float, height: float) -> None:\n self.width = width\n self.height = height\n\n def area(self) -> float:\n return 0\n\n\n class Rectangle(Shape):\n def area(self) -> float:\n width: float = self.width\n height: float = self.height\n return width * height\n\n\n class Triangle(Shape):\n def area(self) -> float:\n width: float = self.width\n height: float = self.height\n return width * height / 2\n\n\n S = TypeVar(\"S\", bound=Shape)\n\n\n def area(s: S) -> float:\n return s.area()\n\n\n r: Rectangle = Rectangle(1, 2)\n t: Triangle = Triangle(3, 4)\n i: int = 5566\n\n print(area(r))\n print(area(t))\n print(area(i))\n\noutput:\n\n.. code-block:: bash\n\n $ mypy --strict foo.py\n foo.py:40: error: Value of type variable \"S\" of \"area\" cannot be \"int\"\n\n@overload\n----------\n\nSometimes, we use ``Union`` to infer that the return of a function has multiple\ndifferent types. However, type checker cannot distinguish which type do we want.\nTherefore, following snippet shows that type checker cannot determine which type\nis correct.\n\n.. code-block:: python\n\n from typing import List, Union\n\n\n class Array(object):\n def __init__(self, arr: List[int]) -> None:\n self.arr = arr\n\n def __getitem__(self, i: Union[int, str]) -> Union[int, str]:\n if isinstance(i, int):\n return self.arr[i]\n if isinstance(i, str):\n return str(self.arr[int(i)])\n\n\n arr = Array([1, 2, 3, 4, 5])\n x:int = arr[1]\n y:str = arr[\"2\"]\n\noutput:\n\n.. code-block:: bash\n\n $ mypy --strict foo.py\n foo.py:16: error: Incompatible types in assignment (expression has type \"Union[int, str]\", variable has type \"int\")\n foo.py:17: error: Incompatible types in assignment (expression has type \"Union[int, str]\", variable has type \"str\")\n\nAlthough we can use ``cast`` to solve the problem, it cannot avoid typo and ``cast`` is not safe.\n\n.. code-block:: python\n\n from typing import List, Union, cast\n\n\n class Array(object):\n def __init__(self, arr: List[int]) -> None:\n self.arr = arr\n\n def __getitem__(self, i: Union[int, str]) -> Union[int, str]:\n if isinstance(i, int):\n return self.arr[i]\n if isinstance(i, str):\n return str(self.arr[int(i)])\n\n\n arr = Array([1, 2, 3, 4, 5])\n x: int = cast(int, arr[1])\n y: str = cast(str, arr[2]) # typo. we want to assign arr[\"2\"]\n\noutput:\n\n.. code-block:: bash\n\n $ mypy --strict foo.py\n $ echo $?\n 0\n\nUsing ``@overload`` can solve the problem. We can declare the return type explicitly.\n\n.. code-block:: python\n\n from typing import Generic, List, Union, overload\n\n\n class Array(object):\n def __init__(self, arr: List[int]) -> None:\n self.arr = arr\n\n @overload\n def __getitem__(self, i: str) -> str:\n ...\n\n @overload\n def __getitem__(self, i: int) -> int:\n ...\n\n def __getitem__(self, i: Union[int, str]) -> Union[int, str]:\n if isinstance(i, int):\n return self.arr[i]\n if isinstance(i, str):\n return str(self.arr[int(i)])\n\n\n arr = Array([1, 2, 3, 4, 5])\n x: int = arr[1]\n y: str = arr[\"2\"]\n\noutput:\n\n.. code-block:: bash\n\n $ mypy --strict foo.py\n $ echo $?\n 0\n\n.. warning::\n\n Based on PEP 484, the ``@overload`` decorator just **for type checker only**, it does not implement\n the real overloading like c++/java. Thus, we have to implement one exactly non-``@overload``\n function. At the runtime, calling the ``@overload`` function will raise ``NotImplementedError``.\n\n.. code-block:: python\n\n from typing import List, Union, overload\n\n\n class Array(object):\n def __init__(self, arr: List[int]) -> None:\n self.arr = arr\n\n @overload\n def __getitem__(self, i: Union[int, str]) -> Union[int, str]:\n if isinstance(i, int):\n return self.arr[i]\n if isinstance(i, str):\n return str(self.arr[int(i)])\n\n\n arr = Array([1, 2, 3, 4, 5])\n try:\n x: int = arr[1]\n except NotImplementedError as e:\n print(\"NotImplementedError\")\n\noutput:\n\n.. code-block:: bash\n\n $ python foo.py\n NotImplementedError\n\nStub Files\n----------\n\nStub files just like header files which we usually use to define our interfaces in c/c++.\nIn python, we can define our interfaces in the same module directory or ``export MYPYPATH=${stubs}``\n\nFirst, we need to create a stub file (interface file) for module.\n\n.. code-block:: bash\n\n $ mkdir fib\n $ touch fib/__init__.py fib/__init__.pyi\n\nThen, define the interface of the function in ``__init__.pyi`` and implement the module.\n\n.. code-block:: python\n\n # fib/__init__.pyi\n def fib(n: int) -> int: ...\n\n # fib/__init__.py\n\n def fib(n):\n a, b = 0, 1\n for _ in range(n):\n b, a = a + b, b\n return a\n\nThen, write a test.py for testing ``fib`` module.\n\n.. code-block:: python\n\n # touch test.py\n import sys\n\n from pathlib import Path\n\n p = Path(__file__).parent / \"fib\"\n sys.path.append(str(p))\n\n from fib import fib\n\n print(fib(10.0))\n\noutput:\n\n.. code-block:: bash\n\n $ mypy --strict test.py\n test.py:10: error: Argument 1 to \"fib\" has incompatible type \"float\"; expected \"int\"\n"
},
{
"path": "docs/notes/basic/python-unicode.rst",
"content": ".. meta::\n :description lang=en: Python Unicode tutorial covering string encoding, decoding, UTF-8, ASCII, bytes conversion, and character handling in Python 3\n :keywords: Python, Python3, Unicode, UTF-8, encoding, decoding, bytes, string, ASCII, character, codec\n\n=======\nUnicode\n=======\n\n.. contents:: Table of Contents\n :backlinks: none\n\nThe main goal of this cheat sheet is to collect some common snippets which are\nrelated to Unicode. In Python 3, strings are represented by Unicode instead of\nbytes. Further information can be found on PEP `3100 `_\n\n**ASCII** code is the most well-known standard which defines numeric codes\nfor characters. The numeric values only define 128 characters originally,\nso ASCII only contains control codes, digits, lowercase letters, uppercase\nletters, etc. However, it is not enough for us to represent characters such as\naccented characters, Chinese characters, or emoji existed around the world.\nTherefore, **Unicode** was developed to solve this issue. It defines the\n*code point* to represent various characters like ASCII but the number of\ncharacters is up to 1,111,998.\n\nString\n------\n\nIn Python 2, strings are represented in *bytes*, not *Unicode*. Python provides\ndifferent types of string such as Unicode string, raw string, and so on.\nIn this case, if we want to declare a Unicode string, we add ``u`` prefix for\nstring literals.\n\n.. code-block:: python\n\n >>> s = 'Café' # byte string\n >>> s\n 'Caf\\xc3\\xa9'\n >>> type(s)\n \n >>> u = u'Café' # unicode string\n >>> u\n u'Caf\\xe9'\n >>> type(u)\n \n\nIn Python 3, strings are represented in *Unicode*. If we want to represent a\nbyte string, we add the ``b`` prefix for string literals. Note that the early\nPython versions (3.0-3.2) do not support the ``u`` prefix. In order to ease\nthe pain to migrate Unicode aware applications from Python 2, Python 3.3 once\nagain supports the ``u`` prefix for string literals. Further information can\nbe found on PEP `414 `_\n\n.. code-block:: python\n\n >>> s = 'Café'\n >>> type(s)\n \n >>> s\n 'Café'\n >>> s.encode('utf-8')\n b'Caf\\xc3\\xa9'\n >>> s.encode('utf-8').decode('utf-8')\n 'Café'\n\nCharacters\n----------\n\nPython 2 takes all string characters as bytes. In this case, the length of\nstrings may be not equivalent to the number of characters. For example,\nthe length of ``Café`` is 5, not 4 because ``é`` is encoded as a 2 bytes\ncharacter.\n\n.. code-block:: python\n\n >>> s= 'Café'\n >>> print([_c for _c in s])\n ['C', 'a', 'f', '\\xc3', '\\xa9']\n >>> len(s)\n 5\n >>> s = u'Café'\n >>> print([_c for _c in s])\n [u'C', u'a', u'f', u'\\xe9']\n >>> len(s)\n 4\n\nPython 3 takes all string characters as Unicode code point. The lenght of\na string is always equivalent to the number of characters.\n\n.. code-block:: python\n\n >>> s = 'Café'\n >>> print([_c for _c in s])\n ['C', 'a', 'f', 'é']\n >>> len(s)\n 4\n >>> bs = bytes(s, encoding='utf-8')\n >>> print(bs)\n b'Caf\\xc3\\xa9'\n >>> len(bs)\n 5\n\nPorting unicode(s, 'utf-8')\n---------------------------\n\nThe `unicode() `_\nbuilt-in function was removed in Python 3 so what is the best way to convert\nthe expression ``unicode(s, 'utf-8')`` so it works in both Python 2 and 3?\n\nIn Python 2:\n\n.. code-block:: python\n\n >>> s = 'Café'\n >>> unicode(s, 'utf-8')\n u'Caf\\xe9'\n >>> s.decode('utf-8')\n u'Caf\\xe9'\n >>> unicode(s, 'utf-8') == s.decode('utf-8')\n True\n\nIn Python 3:\n\n.. code-block:: python\n\n >>> s = 'Café'\n >>> s.decode('utf-8')\n AttributeError: 'str' object has no attribute 'decode'\n\nSo, the real answer is...\n\nUnicode Code Point\n------------------\n\n`ord `_ is a powerful\nbuilt-in function to get a Unicode code point from a given character.\nConsequently, If we want to check a Unicode code point of a character, we can\nuse ``ord``.\n\n.. code-block:: python\n\n >>> s = u'Café'\n >>> for _c in s: print('U+%04x' % ord(_c))\n ...\n U+0043\n U+0061\n U+0066\n U+00e9\n >>> u = '中文'\n >>> for _c in u: print('U+%04x' % ord(_c))\n ...\n U+4e2d\n U+6587\n\n\nEncoding\n--------\n\nA *Unicode code point* transfers to a *byte string* is called encoding.\n\n.. code-block:: python\n\n >>> s = u'Café'\n >>> type(s.encode('utf-8'))\n \n\nDecoding\n---------\n\nA *byte string* transfers to a *Unicode code point* is called decoding.\n\n.. code-block:: python\n\n >>> s = bytes('Café', encoding='utf-8')\n >>> s.decode('utf-8')\n 'Café'\n\nUnicode Normalization\n---------------------\n\nSome characters can be represented in two similar form. For example, the\ncharacter, ``é`` can be written as ``e ́`` (Canonical Decomposition) or ``é``\n(Canonical Composition). In this case, we may acquire unexpected results when we\nare comparing two strings even though they look alike. Therefore, we can\nnormalize a Unicode form to solve the issue.\n\n.. code-block:: python\n\n # python 3\n >>> u1 = 'Café' # unicode string\n >>> u2 = 'Cafe\\u0301'\n >>> u1, u2\n ('Café', 'Café')\n >>> len(u1), len(u2)\n (4, 5)\n >>> u1 == u2\n False\n >>> u1.encode('utf-8') # get u1 byte string\n b'Caf\\xc3\\xa9'\n >>> u2.encode('utf-8') # get u2 byte string\n b'Cafe\\xcc\\x81'\n >>> from unicodedata import normalize\n >>> s1 = normalize('NFC', u1) # get u1 NFC format\n >>> s2 = normalize('NFC', u2) # get u2 NFC format\n >>> s1 == s2\n True\n >>> s1.encode('utf-8'), s2.encode('utf-8')\n (b'Caf\\xc3\\xa9', b'Caf\\xc3\\xa9')\n >>> s1 = normalize('NFD', u1) # get u1 NFD format\n >>> s2 = normalize('NFD', u2) # get u2 NFD format\n >>> s1, s2\n ('Café', 'Café')\n >>> s1 == s2\n True\n >>> s1.encode('utf-8'), s2.encode('utf-8')\n (b'Cafe\\xcc\\x81', b'Cafe\\xcc\\x81')\n\n\nAvoid ``UnicodeDecodeError``\n----------------------------\n\nPython raises `UnicodeDecodeError` when byte strings cannot decode to Unicode\ncode points. If we want to avoid this exception, we can pass *replace*,\n*backslashreplace*, or *ignore* to errors argument in `decode `_.\n\n.. code-block:: python\n\n >>> u = b\"\\xff\"\n >>> u.decode('utf-8', 'strict')\n Traceback (most recent call last):\n File \"\", line 1, in \n UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte\n >>> # use U+FFFD, REPLACEMENT CHARACTER\n >>> u.decode('utf-8', \"replace\")\n '\\ufffd'\n >>> # inserts a \\xNN escape sequence\n >>> u.decode('utf-8', \"backslashreplace\")\n '\\\\xff'\n >>> # leave the character out of the Unicode result\n >>> u.decode('utf-8', \"ignore\")\n ''\n\nLong String\n-----------\n\nThe following snippet shows common ways to declare a multi-line string in\nPython.\n\n.. code-block:: python\n\n # original long string\n s = 'This is a very very very long python string'\n\n # Single quote with an escaping backslash\n s = \"This is a very very very \" \\\n \"long python string\"\n\n # Using brackets\n s = (\n \"This is a very very very \"\n \"long python string\"\n )\n\n # Using ``+``\n s = (\n \"This is a very very very \" +\n \"long python string\"\n )\n\n # Using triple-quote with an escaping backslash\n s = '''This is a very very very \\\n long python string'''\n"
},
{
"path": "docs/notes/concurrency/index.rst",
"content": ".. meta::\n :description lang=en: Python concurrency tutorial covering threading, multiprocessing, locks, semaphores, queues, process pools, and concurrent.futures\n :keywords: Python, Python3, threading, multiprocessing, concurrency, parallel, lock, semaphore, queue, ThreadPoolExecutor, ProcessPoolExecutor, GIL\n\nConcurrency\n===========\n\nPython provides multiple approaches for concurrent execution to handle CPU-bound\nand I/O-bound tasks efficiently. The ``threading`` module enables lightweight\nconcurrent execution within a single process, while ``multiprocessing`` bypasses\nthe Global Interpreter Lock (GIL) by using separate processes for true parallelism.\nThe ``concurrent.futures`` module offers a high-level interface that abstracts\nthe differences between threads and processes behind a unified API.\n\nUnderstanding when to use each approach is crucial: threads excel at I/O-bound\ntasks (network requests, file operations) where the GIL is released during\nwaiting, while processes are better for CPU-bound tasks (computation, data\nprocessing) where true parallel execution is needed.\n\n.. toctree::\n :maxdepth: 1\n\n python-threading\n python-multiprocessing\n python-futures\n"
},
{
"path": "docs/notes/concurrency/python-futures.rst",
"content": ".. meta::\n :description lang=en: Python concurrent.futures tutorial covering ThreadPoolExecutor, ProcessPoolExecutor, Future objects, callbacks, and high-level parallel execution patterns\n :keywords: Python, Python3, concurrent.futures, ThreadPoolExecutor, ProcessPoolExecutor, Future, executor, submit, map, as_completed, parallel\n\n==================\nconcurrent.futures\n==================\n\n:Source: `src/basic/concurrency_.py `_\n\n.. contents:: Table of Contents\n :backlinks: none\n\nIntroduction\n------------\n\nThe ``concurrent.futures`` module provides a high-level interface for\nasynchronously executing callables using threads or processes. It abstracts\nthe differences between threading and multiprocessing behind a unified API,\nmaking it easy to switch between them. The module introduces two key concepts:\n**Executors** that manage pools of workers, and **Futures** that represent\nthe eventual result of an asynchronous operation.\n\nThreadPoolExecutor Basics\n-------------------------\n\n``ThreadPoolExecutor`` manages a pool of threads that execute tasks concurrently.\nUse it for I/O-bound tasks like network requests, file operations, or database\nqueries where threads spend time waiting for external resources.\n\n.. code-block:: python\n\n from concurrent.futures import ThreadPoolExecutor\n import time\n\n def fetch_url(url):\n \"\"\"Simulate fetching a URL.\"\"\"\n time.sleep(1) # Simulate network delay\n return f\"Content from {url}\"\n\n urls = [\"http://site1.com\", \"http://site2.com\", \"http://site3.com\"]\n\n # Sequential - takes ~3 seconds\n start = time.time()\n results = [fetch_url(url) for url in urls]\n print(f\"Sequential: {time.time() - start:.2f}s\")\n\n # Concurrent - takes ~1 second\n start = time.time()\n with ThreadPoolExecutor(max_workers=3) as executor:\n results = list(executor.map(fetch_url, urls))\n print(f\"Concurrent: {time.time() - start:.2f}s\")\n\nProcessPoolExecutor Basics\n--------------------------\n\n``ProcessPoolExecutor`` manages a pool of processes for true parallel execution.\nUse it for CPU-bound tasks like data processing, calculations, or image\nmanipulation where you need to utilize multiple CPU cores.\n\n.. code-block:: python\n\n from concurrent.futures import ProcessPoolExecutor\n import time\n\n def cpu_intensive(n):\n \"\"\"CPU-bound computation.\"\"\"\n return sum(i * i for i in range(n))\n\n if __name__ == \"__main__\":\n numbers = [10**7] * 4\n\n # Sequential\n start = time.time()\n results = [cpu_intensive(n) for n in numbers]\n print(f\"Sequential: {time.time() - start:.2f}s\")\n\n # Parallel with processes\n start = time.time()\n with ProcessPoolExecutor(max_workers=4) as executor:\n results = list(executor.map(cpu_intensive, numbers))\n print(f\"Parallel: {time.time() - start:.2f}s\")\n\nUsing submit() and Future Objects\n---------------------------------\n\nThe ``submit()`` method schedules a callable and returns a ``Future`` object\nimmediately. The Future represents the pending result and provides methods to\ncheck status, get the result, or cancel the task. This gives more control than\n``map()`` for handling individual tasks.\n\n.. code-block:: python\n\n from concurrent.futures import ThreadPoolExecutor\n import time\n\n def task(name, duration):\n time.sleep(duration)\n return f\"{name} completed in {duration}s\"\n\n with ThreadPoolExecutor(max_workers=3) as executor:\n # Submit tasks - returns Future immediately\n future1 = executor.submit(task, \"Task A\", 2)\n future2 = executor.submit(task, \"Task B\", 1)\n future3 = executor.submit(task, \"Task C\", 3)\n\n # Check if done (non-blocking)\n print(f\"Task A done: {future1.done()}\")\n\n # Get result (blocking)\n print(future2.result()) # Waits for completion\n print(future1.result())\n print(future3.result())\n\nProcessing Results as They Complete\n-----------------------------------\n\n``as_completed()`` yields futures as they complete, regardless of submission\norder. This is useful when you want to process results as soon as they're\navailable rather than waiting for all tasks to finish.\n\n.. code-block:: python\n\n from concurrent.futures import ThreadPoolExecutor, as_completed\n import time\n import random\n\n def fetch_data(source_id):\n delay = random.uniform(0.5, 2.0)\n time.sleep(delay)\n return f\"Data from source {source_id} (took {delay:.2f}s)\"\n\n sources = range(5)\n\n with ThreadPoolExecutor(max_workers=5) as executor:\n # Submit all tasks\n future_to_source = {\n executor.submit(fetch_data, src): src\n for src in sources\n }\n\n # Process results as they complete\n for future in as_completed(future_to_source):\n source = future_to_source[future]\n try:\n result = future.result()\n print(f\"Source {source}: {result}\")\n except Exception as e:\n print(f\"Source {source} failed: {e}\")\n\nUsing wait() for Completion Control\n-----------------------------------\n\n``wait()`` blocks until specified futures complete. You can wait for all tasks,\nthe first task, or the first exception. This provides fine-grained control\nover when to proceed.\n\n.. code-block:: python\n\n from concurrent.futures import ThreadPoolExecutor, wait, FIRST_COMPLETED, ALL_COMPLETED\n import time\n\n def task(task_id, duration):\n time.sleep(duration)\n return f\"Task {task_id} done\"\n\n with ThreadPoolExecutor(max_workers=3) as executor:\n futures = [\n executor.submit(task, 1, 3),\n executor.submit(task, 2, 1),\n executor.submit(task, 3, 2),\n ]\n\n # Wait for first to complete\n done, not_done = wait(futures, return_when=FIRST_COMPLETED)\n print(f\"First completed: {done.pop().result()}\")\n print(f\"Still running: {len(not_done)}\")\n\n # Wait for all remaining\n done, not_done = wait(not_done, return_when=ALL_COMPLETED)\n for f in done:\n print(f\"Completed: {f.result()}\")\n\nAdding Callbacks to Futures\n---------------------------\n\nCallbacks are functions that execute automatically when a future completes.\nThey're useful for processing results without blocking the main thread or\nfor chaining operations. The callback receives the future as its argument.\n\n.. code-block:: python\n\n from concurrent.futures import ThreadPoolExecutor\n import time\n\n def compute(n):\n time.sleep(1)\n return n * n\n\n def on_complete(future):\n \"\"\"Callback executed when future completes.\"\"\"\n try:\n result = future.result()\n print(f\"Callback: result is {result}\")\n except Exception as e:\n print(f\"Callback: task failed with {e}\")\n\n with ThreadPoolExecutor(max_workers=3) as executor:\n for i in range(5):\n future = executor.submit(compute, i)\n future.add_done_callback(on_complete)\n\n # Main thread continues while callbacks fire\n print(\"Main thread: tasks submitted\")\n time.sleep(2)\n print(\"Main thread: done waiting\")\n\nException Handling\n------------------\n\nExceptions raised in tasks are captured and re-raised when you call\n``result()``. You can also check for exceptions using ``exception()``.\nAlways handle exceptions to prevent silent failures.\n\n.. code-block:: python\n\n from concurrent.futures import ThreadPoolExecutor, as_completed\n\n def risky_task(n):\n if n == 3:\n raise ValueError(f\"Bad value: {n}\")\n return n * 2\n\n with ThreadPoolExecutor(max_workers=3) as executor:\n futures = {executor.submit(risky_task, i): i for i in range(5)}\n\n for future in as_completed(futures):\n n = futures[future]\n try:\n result = future.result()\n print(f\"Task {n}: {result}\")\n except ValueError as e:\n print(f\"Task {n} failed: {e}\")\n\n # Alternative: check exception without raising\n future = executor.submit(risky_task, 3)\n future.result() # Wait for completion\n if future.exception() is not None:\n print(f\"Exception occurred: {future.exception()}\")\n\nTimeout Handling\n----------------\n\nBoth ``result()`` and ``as_completed()`` accept timeout parameters. If a task\ndoesn't complete within the timeout, a ``TimeoutError`` is raised. This\nprevents indefinite blocking on slow or stuck tasks.\n\n.. code-block:: python\n\n from concurrent.futures import ThreadPoolExecutor, TimeoutError, as_completed\n import time\n\n def slow_task(duration):\n time.sleep(duration)\n return f\"Completed after {duration}s\"\n\n with ThreadPoolExecutor(max_workers=2) as executor:\n future = executor.submit(slow_task, 5)\n\n try:\n # Wait max 2 seconds for result\n result = future.result(timeout=2)\n print(result)\n except TimeoutError:\n print(\"Task timed out!\")\n # Note: task continues running in background\n\n # Timeout with as_completed\n futures = [executor.submit(slow_task, i) for i in [1, 3, 5]]\n try:\n for future in as_completed(futures, timeout=2):\n print(future.result())\n except TimeoutError:\n print(\"Some tasks didn't complete in time\")\n\nCancelling Tasks\n----------------\n\nTasks can be cancelled before they start executing using ``cancel()``. Once\na task has started, it cannot be cancelled. Check ``cancelled()`` to see if\ncancellation succeeded.\n\n.. code-block:: python\n\n from concurrent.futures import ThreadPoolExecutor\n import time\n\n def long_task(n):\n time.sleep(2)\n return n\n\n with ThreadPoolExecutor(max_workers=1) as executor:\n # Submit multiple tasks to single worker\n future1 = executor.submit(long_task, 1)\n future2 = executor.submit(long_task, 2) # Queued, not started\n future3 = executor.submit(long_task, 3) # Queued, not started\n\n time.sleep(0.1) # Let first task start\n\n # Try to cancel queued tasks\n cancelled2 = future2.cancel()\n cancelled3 = future3.cancel()\n\n print(f\"Future 2 cancelled: {cancelled2}\") # True\n print(f\"Future 3 cancelled: {cancelled3}\") # True\n print(f\"Future 1 cancelled: {future1.cancel()}\") # False (already running)\n\nExecutor Context Manager\n------------------------\n\nUsing executors as context managers (``with`` statement) ensures proper cleanup.\nWhen exiting the context, ``shutdown(wait=True)`` is called automatically,\nwhich waits for all pending tasks to complete before returning.\n\n.. code-block:: python\n\n from concurrent.futures import ThreadPoolExecutor\n import time\n\n def task(n):\n time.sleep(1)\n return n * 2\n\n # Context manager - automatic cleanup\n with ThreadPoolExecutor(max_workers=3) as executor:\n futures = [executor.submit(task, i) for i in range(5)]\n # Executor waits for all tasks when exiting 'with' block\n\n print(\"All tasks completed\")\n\n # Manual management (not recommended)\n executor = ThreadPoolExecutor(max_workers=3)\n try:\n futures = [executor.submit(task, i) for i in range(5)]\n finally:\n executor.shutdown(wait=True) # Must call explicitly\n\nMap with Chunking\n-----------------\n\nFor large iterables, ``map()`` can be more efficient with chunking. The\n``chunksize`` parameter groups items together, reducing overhead from\ninter-process communication when using ``ProcessPoolExecutor``.\n\n.. code-block:: python\n\n from concurrent.futures import ProcessPoolExecutor\n import time\n\n def process_item(x):\n return x * x\n\n if __name__ == \"__main__\":\n items = range(100000)\n\n # Without chunking - more IPC overhead\n start = time.time()\n with ProcessPoolExecutor(max_workers=4) as executor:\n results = list(executor.map(process_item, items))\n print(f\"No chunking: {time.time() - start:.2f}s\")\n\n # With chunking - less IPC overhead\n start = time.time()\n with ProcessPoolExecutor(max_workers=4) as executor:\n results = list(executor.map(process_item, items, chunksize=1000))\n print(f\"With chunking: {time.time() - start:.2f}s\")\n\nReal-World Example: Parallel Downloads\n--------------------------------------\n\nThis example demonstrates a practical use case: downloading multiple files\nconcurrently with progress tracking, error handling, and timeout management.\n\n.. code-block:: python\n\n from concurrent.futures import ThreadPoolExecutor, as_completed\n import urllib.request\n import time\n\n def download(url, timeout=10):\n \"\"\"Download URL content with timeout.\"\"\"\n try:\n with urllib.request.urlopen(url, timeout=timeout) as response:\n content = response.read()\n return url, len(content), None\n except Exception as e:\n return url, 0, str(e)\n\n urls = [\n \"https://www.python.org\",\n \"https://www.github.com\",\n \"https://www.google.com\",\n \"https://httpbin.org/delay/5\", # Slow endpoint\n ]\n\n print(\"Starting downloads...\")\n start = time.time()\n\n with ThreadPoolExecutor(max_workers=4) as executor:\n future_to_url = {executor.submit(download, url): url for url in urls}\n\n for future in as_completed(future_to_url, timeout=15):\n url, size, error = future.result()\n if error:\n print(f\"FAILED: {url} - {error}\")\n else:\n print(f\"OK: {url} - {size} bytes\")\n\n print(f\"Total time: {time.time() - start:.2f}s\")\n"
},
{
"path": "docs/notes/concurrency/python-multiprocessing.rst",
"content": ".. meta::\n :description lang=en: Python multiprocessing tutorial covering process creation, pools, shared memory, inter-process communication, and parallel CPU-bound task execution\n :keywords: Python, Python3, multiprocessing, Process, Pool, Queue, Pipe, shared memory, parallel, CPU-bound, GIL bypass\n\n===============\nMultiprocessing\n===============\n\n:Source: `src/basic/concurrency_.py `_\n\n.. contents:: Table of Contents\n :backlinks: none\n\nIntroduction\n------------\n\nThe ``multiprocessing`` module enables true parallel execution by spawning\nseparate Python processes, each with its own Python interpreter and memory\nspace. Unlike threads, processes bypass the Global Interpreter Lock (GIL),\nmaking multiprocessing ideal for CPU-bound tasks that need to utilize multiple\nCPU cores. The trade-off is higher overhead for process creation and\ninter-process communication compared to threads.\n\nCreating Processes\n------------------\n\nCreating processes is similar to creating threads. Each process runs in its\nown memory space, so changes to variables in one process don't affect others.\nUse ``start()`` to begin execution and ``join()`` to wait for completion.\n\n.. code-block:: python\n\n from multiprocessing import Process\n import os\n\n def worker(name):\n print(f\"Worker {name}, PID: {os.getpid()}\")\n\n if __name__ == \"__main__\":\n processes = []\n for i in range(4):\n p = Process(target=worker, args=(i,))\n processes.append(p)\n p.start()\n\n for p in processes:\n p.join()\n\n print(f\"Main process PID: {os.getpid()}\")\n\nProcess Pool\n------------\n\nA ``Pool`` manages a collection of worker processes and distributes tasks among\nthem. This is more efficient than creating a new process for each task, as\nprocesses are reused. The pool provides methods like ``map()``, ``apply()``,\nand their async variants for different use cases.\n\n.. code-block:: python\n\n from multiprocessing import Pool\n import time\n\n def cpu_intensive(n):\n \"\"\"Simulate CPU-bound work.\"\"\"\n total = 0\n for i in range(n):\n total += i * i\n return total\n\n if __name__ == \"__main__\":\n numbers = [10**6, 10**6, 10**6, 10**6]\n\n # Sequential execution\n start = time.time()\n results = [cpu_intensive(n) for n in numbers]\n print(f\"Sequential: {time.time() - start:.2f}s\")\n\n # Parallel execution with Pool\n start = time.time()\n with Pool(4) as pool:\n results = pool.map(cpu_intensive, numbers)\n print(f\"Parallel: {time.time() - start:.2f}s\")\n\nPool Methods\n------------\n\nThe Pool class provides several methods for distributing work. ``map()`` applies\na function to each item in an iterable and returns results in order. ``apply()``\ncalls a function with arguments and blocks until complete. The ``_async``\nvariants return immediately with an ``AsyncResult`` object.\n\n.. code-block:: python\n\n from multiprocessing import Pool\n\n def square(x):\n return x * x\n\n def add(a, b):\n return a + b\n\n if __name__ == \"__main__\":\n with Pool(4) as pool:\n # map - apply function to iterable\n results = pool.map(square, range(10))\n print(f\"map: {results}\")\n\n # starmap - unpack arguments from iterable\n pairs = [(1, 2), (3, 4), (5, 6)]\n results = pool.starmap(add, pairs)\n print(f\"starmap: {results}\")\n\n # apply_async - non-blocking single call\n result = pool.apply_async(square, (10,))\n print(f\"apply_async: {result.get()}\")\n\n # map_async - non-blocking map\n result = pool.map_async(square, range(5))\n print(f\"map_async: {result.get()}\")\n\nSharing Data with Queue\n-----------------------\n\nProcesses don't share memory by default. ``multiprocessing.Queue`` provides a\nthread and process-safe way to exchange data between processes. It's the\nrecommended approach for most inter-process communication scenarios.\n\n.. code-block:: python\n\n from multiprocessing import Process, Queue\n\n def producer(q, items):\n for item in items:\n q.put(item)\n print(f\"Produced: {item}\")\n q.put(None) # Sentinel\n\n def consumer(q):\n while True:\n item = q.get()\n if item is None:\n break\n print(f\"Consumed: {item}\")\n\n if __name__ == \"__main__\":\n q = Queue()\n items = list(range(5))\n\n p1 = Process(target=producer, args=(q, items))\n p2 = Process(target=consumer, args=(q,))\n\n p1.start()\n p2.start()\n p1.join()\n p2.join()\n\nSharing Data with Pipe\n----------------------\n\nA ``Pipe`` creates a two-way communication channel between two processes. It's\nsimpler and faster than Queue for point-to-point communication but only\nsupports two endpoints. Each end can send and receive data.\n\n.. code-block:: python\n\n from multiprocessing import Process, Pipe\n\n def sender(conn):\n conn.send(\"Hello from sender\")\n conn.send([1, 2, 3])\n response = conn.recv()\n print(f\"Sender received: {response}\")\n conn.close()\n\n def receiver(conn):\n msg = conn.recv()\n print(f\"Receiver got: {msg}\")\n data = conn.recv()\n print(f\"Receiver got: {data}\")\n conn.send(\"Thanks!\")\n conn.close()\n\n if __name__ == \"__main__\":\n parent_conn, child_conn = Pipe()\n\n p1 = Process(target=sender, args=(parent_conn,))\n p2 = Process(target=receiver, args=(child_conn,))\n\n p1.start()\n p2.start()\n p1.join()\n p2.join()\n\nShared Memory with Value and Array\n----------------------------------\n\nFor simple shared state, ``Value`` and ``Array`` provide shared memory that\nmultiple processes can access. These are faster than Queue/Pipe for frequently\naccessed data but require careful synchronization to avoid race conditions.\n\n.. code-block:: python\n\n from multiprocessing import Process, Value, Array\n\n def increment(counter, lock_needed=True):\n for _ in range(10000):\n with counter.get_lock():\n counter.value += 1\n\n def modify_array(arr):\n for i in range(len(arr)):\n arr[i] = arr[i] * 2\n\n if __name__ == \"__main__\":\n # Shared integer\n counter = Value('i', 0) # 'i' = signed int\n processes = [Process(target=increment, args=(counter,)) for _ in range(4)]\n for p in processes:\n p.start()\n for p in processes:\n p.join()\n print(f\"Counter: {counter.value}\") # 40000\n\n # Shared array\n arr = Array('d', [1.0, 2.0, 3.0, 4.0]) # 'd' = double\n p = Process(target=modify_array, args=(arr,))\n p.start()\n p.join()\n print(f\"Array: {list(arr)}\") # [2.0, 4.0, 6.0, 8.0]\n\nManager for Complex Shared Objects\n----------------------------------\n\nA ``Manager`` provides a way to share more complex Python objects (lists, dicts)\nbetween processes. The manager runs a server process that holds the actual\nobjects, and other processes access them through proxies. This is slower than\nValue/Array but supports arbitrary Python objects.\n\n.. code-block:: python\n\n from multiprocessing import Process, Manager\n\n def worker(shared_dict, shared_list, worker_id):\n shared_dict[worker_id] = worker_id * 10\n shared_list.append(worker_id)\n\n if __name__ == \"__main__\":\n with Manager() as manager:\n shared_dict = manager.dict()\n shared_list = manager.list()\n\n processes = []\n for i in range(4):\n p = Process(target=worker, args=(shared_dict, shared_list, i))\n processes.append(p)\n p.start()\n\n for p in processes:\n p.join()\n\n print(f\"Dict: {dict(shared_dict)}\")\n print(f\"List: {list(shared_list)}\")\n\nProcess Synchronization\n-----------------------\n\nMultiprocessing provides the same synchronization primitives as threading:\n``Lock``, ``RLock``, ``Semaphore``, ``Event``, ``Condition``, and ``Barrier``.\nThese work across processes instead of threads.\n\n.. code-block:: python\n\n from multiprocessing import Process, Lock, Value\n\n def safe_increment(counter, lock):\n for _ in range(10000):\n with lock:\n counter.value += 1\n\n if __name__ == \"__main__\":\n lock = Lock()\n counter = Value('i', 0)\n\n processes = [\n Process(target=safe_increment, args=(counter, lock))\n for _ in range(4)\n ]\n\n for p in processes:\n p.start()\n for p in processes:\n p.join()\n\n print(f\"Counter: {counter.value}\") # 40000\n\nDaemon Processes\n----------------\n\nLike daemon threads, daemon processes are terminated when the main process\nexits. They're useful for background tasks that shouldn't prevent program\ntermination. Set ``daemon=True`` before calling ``start()``.\n\n.. code-block:: python\n\n from multiprocessing import Process\n import time\n\n def background_task():\n while True:\n print(\"Background process running...\")\n time.sleep(1)\n\n if __name__ == \"__main__\":\n p = Process(target=background_task, daemon=True)\n p.start()\n\n time.sleep(3)\n print(\"Main process exiting, daemon will be terminated\")\n\nHandling Process Termination\n----------------------------\n\nProcesses can be terminated gracefully using ``terminate()`` or forcefully\nusing ``kill()``. Always clean up resources properly and consider using\nsignals for graceful shutdown in production code.\n\n.. code-block:: python\n\n from multiprocessing import Process\n import time\n import signal\n\n def long_running_task():\n try:\n while True:\n print(\"Working...\")\n time.sleep(1)\n except KeyboardInterrupt:\n print(\"Graceful shutdown\")\n\n if __name__ == \"__main__\":\n p = Process(target=long_running_task)\n p.start()\n\n time.sleep(3)\n\n # Graceful termination (SIGTERM)\n p.terminate()\n p.join(timeout=2)\n\n # Force kill if still alive\n if p.is_alive():\n p.kill()\n p.join()\n\n print(f\"Exit code: {p.exitcode}\")\n\nProcessPoolExecutor\n-------------------\n\n``concurrent.futures.ProcessPoolExecutor`` provides a higher-level interface\nfor process pools that's consistent with ``ThreadPoolExecutor``. It's often\neasier to use than ``multiprocessing.Pool`` and integrates well with the\nfutures pattern.\n\n.. code-block:: python\n\n from concurrent.futures import ProcessPoolExecutor, as_completed\n\n def compute(n):\n return sum(i * i for i in range(n))\n\n if __name__ == \"__main__\":\n numbers = [10**6, 10**6, 10**6, 10**6]\n\n with ProcessPoolExecutor(max_workers=4) as executor:\n # Submit individual tasks\n futures = [executor.submit(compute, n) for n in numbers]\n\n # Process results as they complete\n for future in as_completed(futures):\n print(f\"Result: {future.result()}\")\n\n # Or use map for ordered results\n results = list(executor.map(compute, numbers))\n print(f\"All results: {results}\")\n\nComparing Threads vs Processes\n------------------------------\n\nChoose threads for I/O-bound tasks (network, file I/O) where the GIL is\nreleased during waiting. Choose processes for CPU-bound tasks that need true\nparallelism. This example demonstrates the performance difference.\n\n.. code-block:: python\n\n from threading import Thread\n from multiprocessing import Process, Pool\n import time\n\n def cpu_bound(n):\n \"\"\"CPU-intensive task.\"\"\"\n return sum(i * i for i in range(n))\n\n if __name__ == \"__main__\":\n n = 10**7\n count = 4\n\n # Sequential\n start = time.time()\n for _ in range(count):\n cpu_bound(n)\n print(f\"Sequential: {time.time() - start:.2f}s\")\n\n # Threads (limited by GIL)\n start = time.time()\n threads = [Thread(target=cpu_bound, args=(n,)) for _ in range(count)]\n for t in threads:\n t.start()\n for t in threads:\n t.join()\n print(f\"Threads: {time.time() - start:.2f}s\")\n\n # Processes (true parallelism)\n start = time.time()\n with Pool(count) as pool:\n pool.map(cpu_bound, [n] * count)\n print(f\"Processes: {time.time() - start:.2f}s\")\n"
},
{
"path": "docs/notes/concurrency/python-threading.rst",
"content": ".. meta::\n :description lang=en: Python threading tutorial covering thread creation, synchronization primitives, locks, semaphores, events, conditions, and thread-safe data structures\n :keywords: Python, Python3, threading, Thread, Lock, RLock, Semaphore, Event, Condition, synchronization, GIL, concurrent, parallel\n\n=========\nThreading\n=========\n\n:Source: `src/basic/concurrency_.py `_\n\n.. contents:: Table of Contents\n :backlinks: none\n\nIntroduction\n------------\n\nThe ``threading`` module provides a high-level interface for creating and\nmanaging threads in Python. Threads are lightweight units of execution that\nshare the same memory space within a process, making them efficient for I/O-bound\ntasks where the program spends time waiting for external resources. However,\ndue to Python's Global Interpreter Lock (GIL), threads cannot achieve true\nparallelism for CPU-bound tasks—only one thread can execute Python bytecode\nat a time. For CPU-intensive work, consider using ``multiprocessing`` instead.\n\nCreating Threads\n----------------\n\nThere are two primary ways to create threads: subclassing ``Thread`` or passing\na target function. The function-based approach is more flexible and commonly\nused, while subclassing is useful when you need to encapsulate thread state\nand behavior in a class.\n\n.. code-block:: python\n\n from threading import Thread\n\n # Method 1: Subclass Thread\n class Worker(Thread):\n def __init__(self, worker_id):\n super().__init__()\n self.worker_id = worker_id\n\n def run(self):\n print(f\"Worker {self.worker_id} running\")\n\n # Method 2: Pass target function (preferred)\n def task(worker_id):\n print(f\"Task {worker_id} running\")\n\n # Using subclass\n t1 = Worker(1)\n t1.start()\n t1.join()\n\n # Using target function\n t2 = Thread(target=task, args=(2,))\n t2.start()\n t2.join()\n\nThread with Return Value\n------------------------\n\nThreads don't directly return values from their target functions. To get results\nback, you can use shared mutable objects, queues, or store results as instance\nattributes when subclassing Thread.\n\n.. code-block:: python\n\n from threading import Thread\n from queue import Queue\n\n def compute(n, results):\n \"\"\"Store result in shared dict.\"\"\"\n results[n] = n * n\n\n # Using shared dictionary\n results = {}\n threads = []\n for i in range(5):\n t = Thread(target=compute, args=(i, results))\n threads.append(t)\n t.start()\n\n for t in threads:\n t.join()\n\n print(results) # {0: 0, 1: 1, 2: 4, 3: 9, 4: 16}\n\n # Using Queue (thread-safe)\n def compute_queue(n, q):\n q.put((n, n * n))\n\n q = Queue()\n threads = []\n for i in range(5):\n t = Thread(target=compute_queue, args=(i, q))\n threads.append(t)\n t.start()\n\n for t in threads:\n t.join()\n\n while not q.empty():\n n, result = q.get()\n print(f\"{n}: {result}\")\n\nDaemon Threads\n--------------\n\nDaemon threads run in the background and are automatically terminated when all\nnon-daemon threads have finished. They're useful for background tasks that\nshouldn't prevent the program from exiting, such as monitoring or cleanup tasks.\n\n.. code-block:: python\n\n from threading import Thread\n import time\n\n def background_task():\n while True:\n print(\"Background task running...\")\n time.sleep(1)\n\n # Daemon thread - won't prevent program exit\n t = Thread(target=background_task, daemon=True)\n t.start()\n\n # Main thread work\n time.sleep(3)\n print(\"Main thread done, daemon will be killed\")\n\nLock - Mutual Exclusion\n-----------------------\n\nA ``Lock`` is the simplest synchronization primitive that prevents multiple\nthreads from accessing a shared resource simultaneously. Always use locks when\nmodifying shared state to prevent race conditions. The context manager syntax\n(``with lock:``) is preferred as it guarantees the lock is released even if\nan exception occurs.\n\n.. code-block:: python\n\n from threading import Thread, Lock\n\n counter = 0\n lock = Lock()\n\n def increment(n):\n global counter\n for _ in range(n):\n with lock: # Acquire and release automatically\n counter += 1\n\n threads = [Thread(target=increment, args=(100000,)) for _ in range(5)]\n for t in threads:\n t.start()\n for t in threads:\n t.join()\n\n print(f\"Counter: {counter}\") # Always 500000 with lock\n\nRLock - Reentrant Lock\n----------------------\n\nAn ``RLock`` (reentrant lock) can be acquired multiple times by the same thread\nwithout causing a deadlock. This is essential when a thread needs to call\nmethods that also acquire the same lock, such as in recursive functions or\nwhen methods call other methods on the same object.\n\n.. code-block:: python\n\n from threading import Thread, RLock\n\n class Counter:\n def __init__(self):\n self.value = 0\n self.lock = RLock()\n\n def increment(self):\n with self.lock:\n self.value += 1\n\n def increment_twice(self):\n with self.lock: # First acquisition\n self.increment() # Second acquisition - OK with RLock\n self.increment()\n\n counter = Counter()\n threads = [Thread(target=counter.increment_twice) for _ in range(100)]\n for t in threads:\n t.start()\n for t in threads:\n t.join()\n\n print(f\"Value: {counter.value}\") # 200\n\nSemaphore - Resource Limiting\n-----------------------------\n\nA ``Semaphore`` limits the number of threads that can access a resource\nconcurrently. Unlike a lock which allows only one thread, a semaphore with\ncount N allows up to N threads to proceed. This is useful for connection pools,\nrate limiting, or controlling access to limited resources.\n\n.. code-block:: python\n\n from threading import Thread, Semaphore\n import time\n\n # Allow max 3 concurrent connections\n connection_pool = Semaphore(3)\n\n def access_database(thread_id):\n print(f\"Thread {thread_id} waiting for connection...\")\n with connection_pool:\n print(f\"Thread {thread_id} connected\")\n time.sleep(1) # Simulate database work\n print(f\"Thread {thread_id} disconnected\")\n\n threads = [Thread(target=access_database, args=(i,)) for i in range(10)]\n for t in threads:\n t.start()\n for t in threads:\n t.join()\n\nEvent - Thread Signaling\n------------------------\n\nAn ``Event`` is a simple signaling mechanism that allows one thread to signal\nother threads that something has happened. Threads can wait for the event to\nbe set, and one thread can set or clear the event. This is useful for\ncoordinating startup, shutdown, or state changes between threads.\n\n.. code-block:: python\n\n from threading import Thread, Event\n import time\n\n ready = Event()\n\n def worker(worker_id):\n print(f\"Worker {worker_id} waiting for signal...\")\n ready.wait() # Block until event is set\n print(f\"Worker {worker_id} starting work\")\n\n def coordinator():\n print(\"Coordinator preparing...\")\n time.sleep(2)\n print(\"Coordinator: All systems go!\")\n ready.set() # Signal all waiting threads\n\n threads = [Thread(target=worker, args=(i,)) for i in range(3)]\n threads.append(Thread(target=coordinator))\n\n for t in threads:\n t.start()\n for t in threads:\n t.join()\n\nCondition - Complex Synchronization\n-----------------------------------\n\nA ``Condition`` combines a lock with the ability to wait for and notify about\nstate changes. It's essential for producer-consumer patterns where threads\nneed to wait for specific conditions (like \"buffer not empty\" or \"buffer not\nfull\") before proceeding.\n\n.. code-block:: python\n\n from threading import Thread, Condition\n import time\n\n items = []\n condition = Condition()\n\n def producer():\n for i in range(5):\n time.sleep(0.5)\n with condition:\n items.append(i)\n print(f\"Produced: {i}\")\n condition.notify() # Wake up one waiting consumer\n\n def consumer():\n while True:\n with condition:\n while not items: # Wait until items available\n condition.wait()\n item = items.pop(0)\n print(f\"Consumed: {item}\")\n if item == 4:\n break\n\n t1 = Thread(target=producer)\n t2 = Thread(target=consumer)\n t1.start()\n t2.start()\n t1.join()\n t2.join()\n\nBarrier - Synchronization Point\n-------------------------------\n\nA ``Barrier`` blocks a specified number of threads until all of them have\nreached the barrier point, then releases them all simultaneously. This is\nuseful when you need multiple threads to complete a phase before any can\nproceed to the next phase.\n\n.. code-block:: python\n\n from threading import Thread, Barrier\n import time\n import random\n\n barrier = Barrier(3)\n\n def worker(worker_id):\n # Phase 1: Initialization\n print(f\"Worker {worker_id} initializing...\")\n time.sleep(random.uniform(0.5, 2))\n print(f\"Worker {worker_id} waiting at barrier\")\n\n barrier.wait() # Wait for all threads\n\n # Phase 2: All threads proceed together\n print(f\"Worker {worker_id} proceeding\")\n\n threads = [Thread(target=worker, args=(i,)) for i in range(3)]\n for t in threads:\n t.start()\n for t in threads:\n t.join()\n\nTimer - Delayed Execution\n-------------------------\n\nA ``Timer`` is a thread that executes a function after a specified delay.\nIt can be cancelled before it fires. This is useful for timeouts, delayed\ncleanup, or scheduling one-time tasks.\n\n.. code-block:: python\n\n from threading import Timer\n\n def delayed_task():\n print(\"Task executed after delay\")\n\n # Execute after 2 seconds\n timer = Timer(2.0, delayed_task)\n timer.start()\n\n # Can be cancelled before it fires\n # timer.cancel()\n\nThread-Local Data\n-----------------\n\n``threading.local()`` provides thread-local storage where each thread has its\nown independent copy of the data. This is useful for storing per-thread state\nwithout passing it through function arguments, such as database connections\nor request context in web applications.\n\n.. code-block:: python\n\n from threading import Thread, local\n\n # Each thread gets its own 'data' attribute\n thread_data = local()\n\n def worker(worker_id):\n thread_data.value = worker_id\n process()\n\n def process():\n # Access thread-local data without passing as argument\n print(f\"Processing with value: {thread_data.value}\")\n\n threads = [Thread(target=worker, args=(i,)) for i in range(3)]\n for t in threads:\n t.start()\n for t in threads:\n t.join()\n\nProducer-Consumer with Queue\n----------------------------\n\nThe ``queue.Queue`` class provides a thread-safe FIFO queue that handles all\nlocking internally. This is the recommended way to communicate between threads\nin a producer-consumer pattern, as it eliminates the need for manual\nsynchronization.\n\n.. code-block:: python\n\n from threading import Thread\n from queue import Queue\n import time\n\n def producer(q, items):\n for item in items:\n time.sleep(0.1)\n q.put(item)\n print(f\"Produced: {item}\")\n q.put(None) # Sentinel to signal completion\n\n def consumer(q):\n while True:\n item = q.get()\n if item is None:\n break\n print(f\"Consumed: {item}\")\n q.task_done()\n\n q = Queue(maxsize=5) # Bounded queue\n items = list(range(10))\n\n t1 = Thread(target=producer, args=(q, items))\n t2 = Thread(target=consumer, args=(q,))\n\n t1.start()\n t2.start()\n t1.join()\n t2.join()\n\nDeadlock Example and Prevention\n-------------------------------\n\nDeadlock occurs when two or more threads are waiting for each other to release\nlocks, creating a circular dependency. The classic example is when thread A\nholds lock 1 and waits for lock 2, while thread B holds lock 2 and waits for\nlock 1. Prevent deadlocks by always acquiring locks in a consistent order.\n\n.. code-block:: python\n\n from threading import Thread, Lock\n import time\n\n lock1 = Lock()\n lock2 = Lock()\n\n # DEADLOCK EXAMPLE - DON'T DO THIS\n def task_a_bad():\n with lock1:\n print(\"Task A acquired lock1\")\n time.sleep(0.1)\n with lock2: # Waits for lock2\n print(\"Task A acquired lock2\")\n\n def task_b_bad():\n with lock2:\n print(\"Task B acquired lock2\")\n time.sleep(0.1)\n with lock1: # Waits for lock1 - DEADLOCK!\n print(\"Task B acquired lock1\")\n\n # CORRECT - Always acquire locks in same order\n def task_a_good():\n with lock1:\n with lock2:\n print(\"Task A acquired both locks\")\n\n def task_b_good():\n with lock1: # Same order as task_a\n with lock2:\n print(\"Task B acquired both locks\")\n\nUnderstanding the GIL\n---------------------\n\nThe Global Interpreter Lock (GIL) is a mutex that protects access to Python\nobjects, preventing multiple threads from executing Python bytecode\nsimultaneously. This means threads don't provide speedup for CPU-bound tasks.\nHowever, the GIL is released during I/O operations, making threads effective\nfor I/O-bound tasks.\n\n.. code-block:: python\n\n from threading import Thread\n import time\n\n def cpu_bound(n):\n \"\"\"CPU-bound task - GIL limits parallelism.\"\"\"\n count = 0\n for i in range(n):\n count += i\n return count\n\n def io_bound(seconds):\n \"\"\"I/O-bound task - GIL released during sleep.\"\"\"\n time.sleep(seconds)\n\n # CPU-bound: threads won't help (may be slower due to GIL contention)\n start = time.time()\n threads = [Thread(target=cpu_bound, args=(10**7,)) for _ in range(4)]\n for t in threads:\n t.start()\n for t in threads:\n t.join()\n print(f\"CPU-bound threaded: {time.time() - start:.2f}s\")\n\n # I/O-bound: threads help significantly\n start = time.time()\n threads = [Thread(target=io_bound, args=(1,)) for _ in range(4)]\n for t in threads:\n t.start()\n for t in threads:\n t.join()\n print(f\"I/O-bound threaded: {time.time() - start:.2f}s\") # ~1s, not 4s\n"
},
{
"path": "docs/notes/database/index.rst",
"content": ".. meta::\n :description lang=en: Python SQLAlchemy tutorial covering database connections, ORM models, relationships, queries, joins, and advanced query patterns\n :keywords: Python, Python3, SQLAlchemy, database, SQL, ORM, query, join, relationship, session, model, PostgreSQL, MySQL, SQLite\n\n========\nDatabase\n========\n\nWorking with databases is a core skill for most Python applications, from web\ndevelopment to data analysis. SQLAlchemy is Python's most popular database\ntoolkit, providing both a low-level SQL expression language (Core) and a\nhigh-level Object-Relational Mapper (ORM). This section covers SQLAlchemy from\nbasic connections and queries to advanced patterns like relationships, eager\nloading, and complex joins. Whether you're building a simple script or a\nlarge-scale application, these examples will help you interact with databases\nefficiently and safely.\n\n.. toctree::\n :maxdepth: 1\n\n python-sqlalchemy\n python-sqlalchemy-orm\n python-sqlalchemy-query\n"
},
{
"path": "docs/notes/database/python-sqlalchemy-orm.rst",
"content": ".. meta::\n :description lang=en: SQLAlchemy ORM tutorial covering declarative models, sessions, relationships, and object-relational mapping patterns\n :keywords: Python, SQLAlchemy, ORM, database, model, session, relationship, declarative, object-relational mapping\n\n==============\nSQLAlchemy ORM\n==============\n\n.. contents:: Table of Contents\n :backlinks: none\n\nSQLAlchemy's Object-Relational Mapper (ORM) provides a high-level abstraction that\nallows you to work with database tables as Python classes and rows as objects. The\nORM builds on top of SQLAlchemy Core and adds features like identity mapping, unit\nof work pattern, and relationship management. This approach lets you write database\ncode in a more Pythonic way, focusing on objects and their relationships rather than\nSQL statements. The ORM is ideal for applications with complex domain models where\nyou want to leverage object-oriented programming patterns.\n\nDefine Models with Declarative Base\n-----------------------------------\n\nThe declarative system is the most common way to define ORM models in SQLAlchemy.\nYou create a base class using ``declarative_base()`` and then define your models\nas subclasses. Each model class represents a database table, with class attributes\ndefining columns. The ``__tablename__`` attribute specifies the table name. This\napproach keeps your model definitions clean and readable while providing full\naccess to SQLAlchemy's features.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String\n >>> from sqlalchemy.orm import declarative_base\n >>> Base = declarative_base()\n\n >>> class User(Base):\n ... __tablename__ = \"users\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n ... email = Column(String(100))\n ... def __repr__(self):\n ... return f\"User(id={self.id}, name='{self.name}')\"\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n\nSession Basics\n--------------\n\nThe ``Session`` is the primary interface for persistence operations in the ORM.\nIt manages a \"holding zone\" for objects you've loaded or associated with it, and\nhandles the communication with the database. Sessions track changes to objects\nand synchronize them with the database when you call ``commit()``. The recommended\npattern is to use ``sessionmaker`` to create a session factory, then create sessions\nas needed. Always close sessions when done to release database connections.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String\n >>> from sqlalchemy.orm import declarative_base, sessionmaker\n >>> Base = declarative_base()\n\n >>> class User(Base):\n ... __tablename__ = \"users\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n\n >>> session = Session()\n >>> try:\n ... user = User(name=\"Alice\")\n ... session.add(user)\n ... session.commit()\n ... print(f\"Created user with id: {user.id}\")\n ... finally:\n ... session.close()\n Created user with id: 1\n\nAdd and Commit Objects\n----------------------\n\nTo persist new objects to the database, add them to the session with ``add()`` or\n``add_all()`` for multiple objects. Objects remain in a \"pending\" state until you\ncall ``commit()``, which flushes all pending changes to the database in a transaction.\nIf an error occurs, call ``rollback()`` to undo all changes since the last commit.\nAfter commit, auto-generated values like primary keys are available on the objects.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String\n >>> from sqlalchemy.orm import declarative_base, sessionmaker\n >>> Base = declarative_base()\n\n >>> class User(Base):\n ... __tablename__ = \"users\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n\n >>> # Add single object\n >>> user1 = User(name=\"Alice\")\n >>> session.add(user1)\n\n >>> # Add multiple objects\n >>> users = [User(name=\"Bob\"), User(name=\"Carol\")]\n >>> session.add_all(users)\n >>> session.commit()\n\n >>> print([u.id for u in [user1] + users])\n [1, 2, 3]\n >>> session.close()\n\nQuery Objects\n-------------\n\nSQLAlchemy 2.0 uses ``select()`` with ``session.execute()`` for queries, replacing\nthe legacy ``session.query()`` API. The ``select()`` construct accepts model classes\nor specific columns. Use ``scalars()`` to get model instances directly, or ``execute()``\nfor row tuples. The result supports iteration, ``all()`` for a list, ``first()`` for\nthe first result, and ``one()`` when exactly one result is expected.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, select\n >>> from sqlalchemy.orm import declarative_base, sessionmaker\n >>> Base = declarative_base()\n\n >>> class User(Base):\n ... __tablename__ = \"users\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n ... age = Column(Integer)\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n >>> session.add_all([\n ... User(name=\"Alice\", age=30),\n ... User(name=\"Bob\", age=25),\n ... User(name=\"Carol\", age=35)])\n >>> session.commit()\n\n >>> # Get all users\n >>> users = session.execute(select(User)).scalars().all()\n >>> print([u.name for u in users])\n ['Alice', 'Bob', 'Carol']\n\n >>> # Filter with where()\n >>> user = session.execute(select(User).where(User.age > 28)).scalars().first()\n >>> print(user.name)\n Alice\n >>> session.close()\n\nFilter Queries\n--------------\n\nThe ``where()`` method accepts filter conditions using column comparisons. SQLAlchemy\noverloads Python operators to generate SQL: ``==`` becomes ``=``, ``!=`` becomes ``<>``,\nand so on. For complex conditions, use ``and_()``, ``or_()``, and ``not_()`` from\nSQLAlchemy. Columns also provide methods like ``in_()``, ``like()``, ``between()``,\n``is_()``, and ``isnot()`` for SQL-specific operations.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, select, and_, or_\n >>> from sqlalchemy.orm import declarative_base, sessionmaker\n >>> Base = declarative_base()\n\n >>> class User(Base):\n ... __tablename__ = \"users\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n ... age = Column(Integer)\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n >>> session.add_all([\n ... User(name=\"Alice\", age=30),\n ... User(name=\"Bob\", age=25),\n ... User(name=\"Carol\", age=35),\n ... User(name=\"Fred\", age=30)])\n >>> session.commit()\n\n >>> # AND condition\n >>> stmt = select(User).where(and_(User.age >= 30, User.name.like(\"A%\")))\n >>> print([u.name for u in session.execute(stmt).scalars()])\n ['Alice']\n\n >>> # OR condition\n >>> stmt = select(User).where(or_(User.name == \"Alice\", User.name == \"Bob\"))\n >>> print([u.name for u in session.execute(stmt).scalars()])\n ['Alice', 'Bob']\n\n >>> # IN clause\n >>> stmt = select(User).where(User.age.in_([25, 35]))\n >>> print([u.name for u in session.execute(stmt).scalars()])\n ['Bob', 'Carol']\n >>> session.close()\n\nUpdate Objects\n--------------\n\nTo update objects, simply modify their attributes and call ``commit()``. The session\ntracks changes to loaded objects automatically through a mechanism called \"dirty\ntracking\". When you commit, SQLAlchemy generates UPDATE statements only for changed\nattributes. You can also use bulk updates with ``update()`` for efficiency when\nmodifying many rows without loading them into memory.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, select, update\n >>> from sqlalchemy.orm import declarative_base, sessionmaker\n >>> Base = declarative_base()\n\n >>> class User(Base):\n ... __tablename__ = \"users\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n >>> session.add(User(name=\"Alice\"))\n >>> session.commit()\n\n >>> # Update via object modification\n >>> user = session.execute(select(User).where(User.name == \"Alice\")).scalars().first()\n >>> user.name = \"Alicia\"\n >>> session.commit()\n\n >>> # Verify update\n >>> user = session.execute(select(User)).scalars().first()\n >>> print(user.name)\n Alicia\n >>> session.close()\n\nDelete Objects\n--------------\n\nTo delete objects, use ``session.delete()`` followed by ``commit()``. The session\nwill generate a DELETE statement for the object. For bulk deletes without loading\nobjects, use the ``delete()`` construct with ``session.execute()``. Be careful with\ncascading deletes when objects have relationships - SQLAlchemy can automatically\ndelete related objects based on your cascade configuration.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, select, delete\n >>> from sqlalchemy.orm import declarative_base, sessionmaker\n >>> Base = declarative_base()\n\n >>> class User(Base):\n ... __tablename__ = \"users\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n >>> session.add_all([User(name=\"Alice\"), User(name=\"Bob\"), User(name=\"Carol\")])\n >>> session.commit()\n\n >>> # Delete via object\n >>> user = session.execute(select(User).where(User.name == \"Bob\")).scalars().first()\n >>> session.delete(user)\n >>> session.commit()\n\n >>> # Verify deletion\n >>> users = session.execute(select(User)).scalars().all()\n >>> print([u.name for u in users])\n ['Alice', 'Carol']\n >>> session.close()\n\nOne-to-Many Relationship\n------------------------\n\nRelationships define how tables are connected. A one-to-many relationship means one\nrecord in the parent table can have multiple related records in the child table.\nUse ``relationship()`` on the parent side and ``ForeignKey`` on the child side.\nThe ``back_populates`` parameter creates a bidirectional relationship, allowing\nnavigation from both sides. SQLAlchemy handles the foreign key management automatically.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, ForeignKey, select\n >>> from sqlalchemy.orm import declarative_base, sessionmaker, relationship\n >>> Base = declarative_base()\n\n >>> class User(Base):\n ... __tablename__ = \"users\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n ... posts = relationship(\"Post\", back_populates=\"author\")\n\n >>> class Post(Base):\n ... __tablename__ = \"posts\"\n ... id = Column(Integer, primary_key=True)\n ... title = Column(String(100))\n ... user_id = Column(Integer, ForeignKey(\"users.id\"))\n ... author = relationship(\"User\", back_populates=\"posts\")\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n\n >>> user = User(name=\"Alice\")\n >>> user.posts.append(Post(title=\"First Post\"))\n >>> user.posts.append(Post(title=\"Second Post\"))\n >>> session.add(user)\n >>> session.commit()\n\n >>> # Access relationship\n >>> user = session.execute(select(User)).scalars().first()\n >>> print([p.title for p in user.posts])\n ['First Post', 'Second Post']\n >>> session.close()\n\nMany-to-Many Relationship\n-------------------------\n\nMany-to-many relationships require an association table that contains foreign keys\nto both related tables. Define the association table using ``Table``, then use\n``relationship()`` with the ``secondary`` parameter pointing to it. Both sides can\nhave a relationship, and SQLAlchemy manages the association table entries automatically\nwhen you add or remove items from the relationship collections.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, ForeignKey, Table, select\n >>> from sqlalchemy.orm import declarative_base, sessionmaker, relationship\n >>> Base = declarative_base()\n\n >>> # Association table\n >>> student_course = Table(\n ... \"student_course\", Base.metadata,\n ... Column(\"student_id\", Integer, ForeignKey(\"students.id\"), primary_key=True),\n ... Column(\"course_id\", Integer, ForeignKey(\"courses.id\"), primary_key=True))\n\n >>> class Student(Base):\n ... __tablename__ = \"students\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n ... courses = relationship(\"Course\", secondary=student_course, back_populates=\"students\")\n\n >>> class Course(Base):\n ... __tablename__ = \"courses\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n ... students = relationship(\"Student\", secondary=student_course, back_populates=\"courses\")\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n\n >>> math = Course(name=\"Math\")\n >>> physics = Course(name=\"Physics\")\n >>> alice = Student(name=\"Alice\", courses=[math, physics])\n >>> bob = Student(name=\"Bob\", courses=[math])\n >>> session.add_all([alice, bob])\n >>> session.commit()\n\n >>> # Query relationships\n >>> math = session.execute(select(Course).where(Course.name == \"Math\")).scalars().first()\n >>> print([s.name for s in math.students])\n ['Alice', 'Bob']\n >>> session.close()\n\nSelf-Referential Relationship\n-----------------------------\n\nSelf-referential relationships connect a table to itself, useful for hierarchical\ndata like organizational charts, categories, or threaded comments. Use ``ForeignKey``\npointing to the same table and ``relationship()`` with ``remote_side`` to indicate\nwhich side is the \"parent\". This pattern allows you to model tree structures where\neach node can have a parent and multiple children.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, ForeignKey, select\n >>> from sqlalchemy.orm import declarative_base, sessionmaker, relationship\n >>> Base = declarative_base()\n\n >>> class Employee(Base):\n ... __tablename__ = \"employees\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n ... manager_id = Column(Integer, ForeignKey(\"employees.id\"))\n ... manager = relationship(\"Employee\", remote_side=[id], backref=\"subordinates\")\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n\n >>> ceo = Employee(name=\"CEO\")\n >>> session.add(ceo)\n >>> session.flush()\n >>> manager = Employee(name=\"Manager\", manager_id=ceo.id)\n >>> session.add(manager)\n >>> session.flush()\n >>> worker1 = Employee(name=\"Worker1\", manager_id=manager.id)\n >>> worker2 = Employee(name=\"Worker2\", manager_id=manager.id)\n >>> session.add_all([worker1, worker2])\n >>> session.commit()\n\n >>> # Navigate hierarchy\n >>> mgr = session.execute(select(Employee).where(Employee.name == \"Manager\")).scalars().first()\n >>> print(f\"Manager: {mgr.name}, Boss: {mgr.manager.name}\")\n Manager: Manager, Boss: CEO\n >>> print(f\"Subordinates: {[e.name for e in mgr.subordinates]}\")\n Subordinates: ['Worker1', 'Worker2']\n >>> session.close()\n\nCascade Deletes\n---------------\n\nCascade options control what happens to related objects when a parent is deleted\nor modified. The ``cascade`` parameter on ``relationship()`` accepts a comma-separated\nstring of cascade rules. Common options include ``\"all, delete-orphan\"`` which deletes\nchildren when the parent is deleted and when children are removed from the collection.\nThis ensures referential integrity and prevents orphaned records.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, ForeignKey, select\n >>> from sqlalchemy.orm import declarative_base, sessionmaker, relationship\n >>> Base = declarative_base()\n\n >>> class Parent(Base):\n ... __tablename__ = \"parents\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n ... children = relationship(\"Child\", back_populates=\"parent\",\n ... cascade=\"all, delete-orphan\")\n\n >>> class Child(Base):\n ... __tablename__ = \"children\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n ... parent_id = Column(Integer, ForeignKey(\"parents.id\"))\n ... parent = relationship(\"Parent\", back_populates=\"children\")\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n\n >>> parent = Parent(name=\"Parent1\")\n >>> parent.children = [Child(name=\"Child1\"), Child(name=\"Child2\")]\n >>> session.add(parent)\n >>> session.commit()\n\n >>> # Delete parent - children are also deleted\n >>> session.delete(parent)\n >>> session.commit()\n >>> children = session.execute(select(Child)).scalars().all()\n >>> print(len(children))\n 0\n >>> session.close()\n\nEager Loading\n-------------\n\nBy default, SQLAlchemy uses lazy loading for relationships, executing a new query\nwhen you access related objects. This can cause the \"N+1 query problem\" when iterating\nover many objects. Eager loading fetches related objects in the same query using\nJOIN or subqueries. Use ``joinedload()`` for single objects or small collections,\nand ``selectinload()`` for larger collections to avoid cartesian products.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, ForeignKey, select\n >>> from sqlalchemy.orm import declarative_base, sessionmaker, relationship, joinedload\n >>> Base = declarative_base()\n\n >>> class User(Base):\n ... __tablename__ = \"users\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n ... posts = relationship(\"Post\", back_populates=\"author\")\n\n >>> class Post(Base):\n ... __tablename__ = \"posts\"\n ... id = Column(Integer, primary_key=True)\n ... title = Column(String(100))\n ... user_id = Column(Integer, ForeignKey(\"users.id\"))\n ... author = relationship(\"User\", back_populates=\"posts\")\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n >>> user = User(name=\"Alice\")\n >>> user.posts = [Post(title=\"Post1\"), Post(title=\"Post2\")]\n >>> session.add(user)\n >>> session.commit()\n\n >>> # Eager load posts with user in single query\n >>> stmt = select(User).options(joinedload(User.posts))\n >>> user = session.execute(stmt).scalars().unique().first()\n >>> print([p.title for p in user.posts]) # No additional query\n ['Post1', 'Post2']\n >>> session.close()\n\nHybrid Properties\n-----------------\n\nHybrid properties allow you to define Python properties that work both at the instance\nlevel (in Python) and at the class level (in SQL queries). This is useful for computed\nattributes that you want to filter or sort by in database queries. Use the\n``@hybrid_property`` decorator and optionally ``@property.expression`` to customize\nthe SQL expression.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, select\n >>> from sqlalchemy.orm import declarative_base, sessionmaker\n >>> from sqlalchemy.ext.hybrid import hybrid_property\n >>> Base = declarative_base()\n\n >>> class User(Base):\n ... __tablename__ = \"users\"\n ... id = Column(Integer, primary_key=True)\n ... first_name = Column(String(50))\n ... last_name = Column(String(50))\n ...\n ... @hybrid_property\n ... def full_name(self):\n ... return f\"{self.first_name} {self.last_name}\"\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n >>> session.add(User(first_name=\"Alice\", last_name=\"Smith\"))\n >>> session.commit()\n\n >>> user = session.execute(select(User)).scalars().first()\n >>> print(user.full_name)\n Alice Smith\n >>> session.close()\n\nEvent Hooks\n-----------\n\nSQLAlchemy provides an event system that lets you hook into various ORM operations\nlike before/after insert, update, or delete. Use ``@event.listens_for()`` decorator\nto register event handlers. Events are useful for auditing, validation, automatic\ntimestamps, or triggering side effects. Common events include ``before_insert``,\n``after_insert``, ``before_update``, ``after_update``, ``before_delete``, and\n``after_delete``.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, DateTime, select, event\n >>> from sqlalchemy.orm import declarative_base, sessionmaker\n >>> from datetime import datetime\n >>> Base = declarative_base()\n\n >>> class User(Base):\n ... __tablename__ = \"users\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n ... created_at = Column(DateTime)\n ... updated_at = Column(DateTime)\n\n >>> @event.listens_for(User, \"before_insert\")\n ... def set_created_at(mapper, connection, target):\n ... target.created_at = datetime.now()\n ... target.updated_at = datetime.now()\n\n >>> @event.listens_for(User, \"before_update\")\n ... def set_updated_at(mapper, connection, target):\n ... target.updated_at = datetime.now()\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n >>> user = User(name=\"Alice\")\n >>> session.add(user)\n >>> session.commit()\n\n >>> print(user.created_at is not None)\n True\n >>> session.close()\n"
},
{
"path": "docs/notes/database/python-sqlalchemy-query.rst",
"content": ".. meta::\n :description lang=en: SQLAlchemy advanced query patterns including joins, subqueries, aggregations, window functions, and performance optimization\n :keywords: Python, SQLAlchemy, query, join, subquery, aggregate, window function, CTE, performance, optimization\n\n========================\nSQLAlchemy Query Recipes\n========================\n\n.. contents:: Table of Contents\n :backlinks: none\n\nThis section covers advanced query patterns and recipes for SQLAlchemy. While the\nbasics of querying are covered in the ORM section, real-world applications often\nrequire more sophisticated queries involving joins across multiple tables, subqueries,\naggregations, and performance optimizations. These patterns help you write efficient\ndatabase queries while maintaining readable Python code. Understanding these techniques\nis essential for building scalable applications that interact with relational databases.\n\nOrder By\n--------\n\nThe ``order_by()`` method sorts query results by one or more columns. Pass column\nobjects directly, or use ``desc()`` for descending order. You can chain multiple\ncolumns for secondary sorting. SQLAlchemy also supports ``nullsfirst()`` and\n``nullslast()`` to control how NULL values are sorted, which is particularly useful\nwhen dealing with optional fields.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, select, desc\n >>> from sqlalchemy.orm import declarative_base, sessionmaker\n >>> Base = declarative_base()\n\n >>> class User(Base):\n ... __tablename__ = \"users\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n ... age = Column(Integer)\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n >>> session.add_all([\n ... User(name=\"Alice\", age=30),\n ... User(name=\"Bob\", age=25),\n ... User(name=\"Carol\", age=30)])\n >>> session.commit()\n\n >>> # Ascending order\n >>> stmt = select(User).order_by(User.age)\n >>> print([u.name for u in session.execute(stmt).scalars()])\n ['Bob', 'Alice', 'Carol']\n\n >>> # Descending order\n >>> stmt = select(User).order_by(desc(User.age), User.name)\n >>> print([u.name for u in session.execute(stmt).scalars()])\n ['Alice', 'Carol', 'Bob']\n >>> session.close()\n\nLimit and Offset\n----------------\n\nUse ``limit()`` to restrict the number of results and ``offset()`` to skip rows,\nenabling pagination. These methods translate directly to SQL LIMIT and OFFSET clauses.\nFor large datasets, consider using keyset pagination (filtering by the last seen ID)\ninstead of offset-based pagination, as OFFSET can become slow with large offsets\nsince the database must scan and discard rows.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, select\n >>> from sqlalchemy.orm import declarative_base, sessionmaker\n >>> Base = declarative_base()\n\n >>> class User(Base):\n ... __tablename__ = \"users\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n >>> session.add_all([User(name=f\"User{i}\") for i in range(10)])\n >>> session.commit()\n\n >>> # First page (3 items)\n >>> stmt = select(User).order_by(User.id).limit(3)\n >>> print([u.name for u in session.execute(stmt).scalars()])\n ['User0', 'User1', 'User2']\n\n >>> # Second page\n >>> stmt = select(User).order_by(User.id).limit(3).offset(3)\n >>> print([u.name for u in session.execute(stmt).scalars()])\n ['User3', 'User4', 'User5']\n >>> session.close()\n\nGroup By and Aggregates\n-----------------------\n\nUse ``group_by()`` with aggregate functions like ``func.count()``, ``func.sum()``,\n``func.avg()``, ``func.min()``, and ``func.max()`` for grouped calculations. The\n``having()`` method filters groups after aggregation, similar to SQL's HAVING clause.\nWhen selecting both regular columns and aggregates, all non-aggregate columns must\nbe included in the GROUP BY clause.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, select, func\n >>> from sqlalchemy.orm import declarative_base, sessionmaker\n >>> Base = declarative_base()\n\n >>> class Sale(Base):\n ... __tablename__ = \"sales\"\n ... id = Column(Integer, primary_key=True)\n ... product = Column(String(50))\n ... amount = Column(Integer)\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n >>> session.add_all([\n ... Sale(product=\"A\", amount=100),\n ... Sale(product=\"A\", amount=150),\n ... Sale(product=\"B\", amount=200),\n ... Sale(product=\"B\", amount=50)])\n >>> session.commit()\n\n >>> # Group by with sum\n >>> stmt = select(Sale.product, func.sum(Sale.amount).label(\"total\"))\\\n ... .group_by(Sale.product)\n >>> for row in session.execute(stmt):\n ... print(f\"{row.product}: {row.total}\")\n A: 250\n B: 250\n\n >>> # Having clause\n >>> stmt = select(Sale.product, func.count().label(\"count\"))\\\n ... .group_by(Sale.product).having(func.count() > 1)\n >>> print(session.execute(stmt).fetchall())\n [('A', 2), ('B', 2)]\n >>> session.close()\n\nJoin Queries\n------------\n\nSQLAlchemy provides several ways to join tables. For ORM models with relationships,\nuse ``join()`` which automatically uses the foreign key. For explicit join conditions,\npass the condition as the second argument. Use ``outerjoin()`` for LEFT OUTER JOIN.\nThe ``select_from()`` method specifies the FROM clause when needed for complex joins.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, ForeignKey, select\n >>> from sqlalchemy.orm import declarative_base, sessionmaker, relationship\n >>> Base = declarative_base()\n\n >>> class User(Base):\n ... __tablename__ = \"users\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n ... orders = relationship(\"Order\", back_populates=\"user\")\n\n >>> class Order(Base):\n ... __tablename__ = \"orders\"\n ... id = Column(Integer, primary_key=True)\n ... product = Column(String(50))\n ... user_id = Column(Integer, ForeignKey(\"users.id\"))\n ... user = relationship(\"User\", back_populates=\"orders\")\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n >>> alice = User(name=\"Alice\")\n >>> alice.orders = [Order(product=\"Book\"), Order(product=\"Pen\")]\n >>> bob = User(name=\"Bob\")\n >>> session.add_all([alice, bob])\n >>> session.commit()\n\n >>> # Inner join\n >>> stmt = select(User.name, Order.product).join(Order)\n >>> print(session.execute(stmt).fetchall())\n [('Alice', 'Book'), ('Alice', 'Pen')]\n\n >>> # Left outer join (includes users without orders)\n >>> stmt = select(User.name, Order.product).outerjoin(Order)\n >>> print(session.execute(stmt).fetchall())\n [('Alice', 'Book'), ('Alice', 'Pen'), ('Bob', None)]\n >>> session.close()\n\nSubqueries\n----------\n\nSubqueries are queries nested inside other queries. Use ``subquery()`` to create\na subquery that can be used in the FROM clause, or ``scalar_subquery()`` for single-value\nsubqueries in SELECT or WHERE clauses. Subqueries are useful for complex filtering,\ncomputing derived values, or when you need to reference aggregated data in the main query.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, select, func\n >>> from sqlalchemy.orm import declarative_base, sessionmaker\n >>> Base = declarative_base()\n\n >>> class User(Base):\n ... __tablename__ = \"users\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n ... score = Column(Integer)\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n >>> session.add_all([\n ... User(name=\"Alice\", score=85),\n ... User(name=\"Bob\", score=90),\n ... User(name=\"Carol\", score=75)])\n >>> session.commit()\n\n >>> # Scalar subquery: users with above-average score\n >>> avg_score = select(func.avg(User.score)).scalar_subquery()\n >>> stmt = select(User).where(User.score > avg_score)\n >>> print([u.name for u in session.execute(stmt).scalars()])\n ['Alice', 'Bob']\n >>> session.close()\n\nCommon Table Expressions (CTE)\n------------------------------\n\nCTEs (WITH clauses) improve query readability by naming subqueries. They're especially\nuseful for recursive queries and when the same subquery is referenced multiple times.\nUse ``cte()`` to create a CTE from a select statement. CTEs can reference themselves\nfor recursive queries, which is powerful for hierarchical data like organizational\ncharts or category trees.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, select, func\n >>> from sqlalchemy.orm import declarative_base, sessionmaker\n >>> Base = declarative_base()\n\n >>> class Sale(Base):\n ... __tablename__ = \"sales\"\n ... id = Column(Integer, primary_key=True)\n ... region = Column(String(50))\n ... amount = Column(Integer)\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n >>> session.add_all([\n ... Sale(region=\"East\", amount=100),\n ... Sale(region=\"East\", amount=200),\n ... Sale(region=\"West\", amount=150)])\n >>> session.commit()\n\n >>> # CTE for regional totals\n >>> regional_totals = select(\n ... Sale.region,\n ... func.sum(Sale.amount).label(\"total\")\n ... ).group_by(Sale.region).cte(\"regional_totals\")\n\n >>> # Use CTE in main query\n >>> stmt = select(regional_totals).where(regional_totals.c.total > 200)\n >>> print(session.execute(stmt).fetchall())\n [('East', 300)]\n >>> session.close()\n\nExists and Correlated Subqueries\n--------------------------------\n\nThe ``exists()`` construct creates an EXISTS subquery, which returns true if the\nsubquery returns any rows. This is efficient for checking the existence of related\nrecords without loading them. Correlated subqueries reference columns from the outer\nquery, allowing row-by-row comparisons. Use ``correlate()`` to explicitly specify\nwhich tables the subquery should correlate with.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, ForeignKey, select, exists\n >>> from sqlalchemy.orm import declarative_base, sessionmaker, relationship\n >>> Base = declarative_base()\n\n >>> class User(Base):\n ... __tablename__ = \"users\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n\n >>> class Order(Base):\n ... __tablename__ = \"orders\"\n ... id = Column(Integer, primary_key=True)\n ... user_id = Column(Integer, ForeignKey(\"users.id\"))\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n >>> session.add_all([User(name=\"Alice\"), User(name=\"Bob\")])\n >>> session.commit()\n >>> alice = session.execute(select(User).where(User.name == \"Alice\")).scalars().first()\n >>> session.add(Order(user_id=alice.id))\n >>> session.commit()\n\n >>> # Users with orders\n >>> has_orders = exists().where(Order.user_id == User.id)\n >>> stmt = select(User).where(has_orders)\n >>> print([u.name for u in session.execute(stmt).scalars()])\n ['Alice']\n\n >>> # Users without orders\n >>> stmt = select(User).where(~has_orders)\n >>> print([u.name for u in session.execute(stmt).scalars()])\n ['Bob']\n >>> session.close()\n\nUnion Queries\n-------------\n\nUse ``union()`` to combine results from multiple SELECT statements, removing duplicates.\nUse ``union_all()`` to keep all rows including duplicates, which is faster when you\nknow there are no duplicates or don't care about them. The queries must have the same\nnumber of columns with compatible types. Unions are useful for combining data from\ndifferent tables or different filtered views of the same table.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, select, union_all\n >>> from sqlalchemy.orm import declarative_base, sessionmaker\n >>> Base = declarative_base()\n\n >>> class Customer(Base):\n ... __tablename__ = \"customers\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n\n >>> class Supplier(Base):\n ... __tablename__ = \"suppliers\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n >>> session.add_all([Customer(name=\"Alice\"), Customer(name=\"Bob\")])\n >>> session.add_all([Supplier(name=\"Acme\"), Supplier(name=\"Bob\")])\n >>> session.commit()\n\n >>> # Union all names\n >>> stmt = union_all(\n ... select(Customer.name),\n ... select(Supplier.name))\n >>> print(sorted([row[0] for row in session.execute(stmt)]))\n ['Acme', 'Alice', 'Bob', 'Bob']\n >>> session.close()\n\nCase Expressions\n----------------\n\nThe ``case()`` construct creates SQL CASE expressions for conditional logic in queries.\nIt's useful for computed columns, conditional aggregations, and data transformations.\nPass a list of (condition, result) tuples, with an optional ``else_`` for the default\nvalue. Case expressions can be used in SELECT, WHERE, ORDER BY, and other clauses.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, select, case\n >>> from sqlalchemy.orm import declarative_base, sessionmaker\n >>> Base = declarative_base()\n\n >>> class User(Base):\n ... __tablename__ = \"users\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n ... score = Column(Integer)\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n >>> session.add_all([\n ... User(name=\"Alice\", score=95),\n ... User(name=\"Bob\", score=75),\n ... User(name=\"Carol\", score=55)])\n >>> session.commit()\n\n >>> # Grade based on score\n >>> grade = case(\n ... (User.score >= 90, \"A\"),\n ... (User.score >= 70, \"B\"),\n ... else_=\"C\")\n >>> stmt = select(User.name, grade.label(\"grade\"))\n >>> for row in session.execute(stmt):\n ... print(f\"{row.name}: {row.grade}\")\n Alice: A\n Bob: B\n Carol: C\n >>> session.close()\n\nDistinct and Count Distinct\n---------------------------\n\nUse ``distinct()`` to remove duplicate rows from results. For counting unique values,\ncombine ``func.count()`` with ``distinct()``. The ``distinct()`` method can be applied\nto the entire query or to specific columns. This is essential for accurate counting\nwhen dealing with joined tables that may produce duplicate rows.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, select, func, distinct\n >>> from sqlalchemy.orm import declarative_base, sessionmaker\n >>> Base = declarative_base()\n\n >>> class Order(Base):\n ... __tablename__ = \"orders\"\n ... id = Column(Integer, primary_key=True)\n ... customer = Column(String(50))\n ... product = Column(String(50))\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n >>> session.add_all([\n ... Order(customer=\"Alice\", product=\"Book\"),\n ... Order(customer=\"Alice\", product=\"Pen\"),\n ... Order(customer=\"Bob\", product=\"Book\")])\n >>> session.commit()\n\n >>> # Distinct customers\n >>> stmt = select(Order.customer).distinct()\n >>> print(session.execute(stmt).fetchall())\n [('Alice',), ('Bob',)]\n\n >>> # Count distinct products\n >>> stmt = select(func.count(distinct(Order.product)))\n >>> print(session.execute(stmt).scalar())\n 2\n >>> session.close()\n\nAliased Tables\n--------------\n\nUse ``aliased()`` to create aliases for tables, allowing you to reference the same\ntable multiple times in a query with different names. This is essential for self-joins\nand queries that need to compare rows within the same table. Aliases create independent\nreferences that can have different join conditions and filters.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, select\n >>> from sqlalchemy.orm import declarative_base, sessionmaker, aliased\n >>> Base = declarative_base()\n\n >>> class Employee(Base):\n ... __tablename__ = \"employees\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n ... salary = Column(Integer)\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n >>> session.add_all([\n ... Employee(name=\"Alice\", salary=50000),\n ... Employee(name=\"Bob\", salary=60000),\n ... Employee(name=\"Carol\", salary=55000)])\n >>> session.commit()\n\n >>> # Find employees earning more than Alice\n >>> alice_alias = aliased(Employee, name=\"alice\")\n >>> stmt = select(Employee.name).select_from(Employee).join(\n ... alice_alias, alice_alias.name == \"Alice\"\n ... ).where(Employee.salary > alice_alias.salary)\n >>> print([row[0] for row in session.execute(stmt)])\n ['Bob', 'Carol']\n >>> session.close()\n\nBulk Operations\n---------------\n\nFor inserting or updating many rows, bulk operations are much faster than adding\nobjects one by one. Use ``session.bulk_insert_mappings()`` for inserts and\n``session.bulk_update_mappings()`` for updates. These methods bypass the ORM's\nunit of work pattern for better performance. For even faster inserts, use Core's\n``insert()`` with ``execute()`` passing a list of dictionaries.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, select, insert\n >>> from sqlalchemy.orm import declarative_base, sessionmaker\n >>> Base = declarative_base()\n\n >>> class User(Base):\n ... __tablename__ = \"users\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n\n >>> # Bulk insert with Core (fastest)\n >>> data = [{\"name\": f\"User{i}\"} for i in range(100)]\n >>> session.execute(insert(User), data)\n >>> session.commit()\n\n >>> # Verify\n >>> count = session.execute(select(func.count()).select_from(User)).scalar()\n >>> print(count)\n 100\n >>> session.close()\n\nRaw SQL with Text\n-----------------\n\nWhen you need to execute raw SQL that's difficult to express with SQLAlchemy's\nconstructs, use ``text()`` to wrap SQL strings. Always use bound parameters (`:name`\nsyntax) instead of string formatting to prevent SQL injection. The ``text()`` construct\ncan be used with both Core and ORM queries, and results can be mapped to ORM objects\nusing ``from_statement()``.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, Column, Integer, String, text, select\n >>> from sqlalchemy.orm import declarative_base, sessionmaker\n >>> Base = declarative_base()\n\n >>> class User(Base):\n ... __tablename__ = \"users\"\n ... id = Column(Integer, primary_key=True)\n ... name = Column(String(50))\n\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> Base.metadata.create_all(engine)\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n >>> session.add_all([User(name=\"Alice\"), User(name=\"Bob\")])\n >>> session.commit()\n\n >>> # Raw SQL with parameters\n >>> result = session.execute(\n ... text(\"SELECT * FROM users WHERE name = :name\"),\n ... {\"name\": \"Alice\"})\n >>> print(result.fetchall())\n [(1, 'Alice')]\n\n >>> # Map raw SQL to ORM objects\n >>> stmt = select(User).from_statement(text(\"SELECT * FROM users ORDER BY name\"))\n >>> users = session.execute(stmt).scalars().all()\n >>> print([u.name for u in users])\n ['Alice', 'Bob']\n >>> session.close()\n"
},
{
"path": "docs/notes/database/python-sqlalchemy.rst",
"content": ".. meta::\n :description lang=en: SQLAlchemy Core tutorial covering database connections, engine creation, metadata, table definitions, and SQL expression language\n :keywords: Python, SQLAlchemy, database, SQL, engine, metadata, table, connection, transaction, Core API\n\n=================\nSQLAlchemy Basics\n=================\n\n.. contents:: Table of Contents\n :backlinks: none\n\nSQLAlchemy is the most popular database toolkit and Object-Relational Mapping (ORM)\nlibrary for Python. It provides a full suite of well-known enterprise-level persistence\npatterns, designed for efficient and high-performing database access. SQLAlchemy is\ndivided into two main components: the Core (low-level SQL abstraction) and the ORM\n(high-level object mapping). This cheat sheet covers the Core API, which provides a\nSQL Expression Language that allows you to construct SQL statements in Python code\nwhile remaining database-agnostic. The Core is ideal when you need fine-grained control\nover SQL queries or when working with existing database schemas.\n\nCreate an Engine\n----------------\n\nThe ``Engine`` is the starting point for any SQLAlchemy application. It represents\nthe connection pool and dialect for a particular database, managing connectivity\nand translating Python code into database-specific SQL. The ``create_engine()``\nfunction takes a database URL that specifies the database type, credentials, host,\nand database name. SQLAlchemy supports many databases including SQLite, PostgreSQL,\nMySQL, Oracle, and Microsoft SQL Server through different dialects.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine\n >>> # SQLite in-memory database (great for testing)\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> # SQLite file-based database\n >>> engine = create_engine(\"sqlite:///mydb.sqlite\")\n >>> # PostgreSQL\n >>> engine = create_engine(\"postgresql://user:pass@localhost/dbname\")\n >>> # MySQL\n >>> engine = create_engine(\"mysql+pymysql://user:pass@localhost/dbname\")\n\nDatabase URL Format\n-------------------\n\nSQLAlchemy uses RFC-1738 style URLs to specify database connections. The URL format\nprovides a standardized way to specify all connection parameters including the database\ndriver, authentication credentials, host address, port number, and database name.\nUnderstanding this format is essential for configuring connections to different\ndatabase systems. The ``make_url()`` function can parse and construct these URLs\nprogrammatically.\n\n.. code-block:: python\n\n >>> from sqlalchemy import make_url\n >>> # Format: dialect+driver://username:password@host:port/database\n >>> url = make_url(\"postgresql://user:pass@localhost:5432/mydb\")\n >>> url.drivername\n 'postgresql'\n >>> url.username\n 'user'\n >>> url.host\n 'localhost'\n >>> url.database\n 'mydb'\n\nConnect and Execute Raw SQL\n---------------------------\n\nWhile SQLAlchemy encourages using its SQL Expression Language, you can also execute\nraw SQL strings directly. This is useful for complex queries that are difficult to\nexpress in SQLAlchemy's API, or when migrating existing SQL code. The ``text()``\nfunction wraps raw SQL strings and allows parameter binding for security. Always\nuse parameter binding instead of string formatting to prevent SQL injection attacks.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, text\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> with engine.connect() as conn:\n ... conn.execute(text(\"CREATE TABLE test (id INTEGER PRIMARY KEY, name TEXT)\"))\n ... conn.execute(text(\"INSERT INTO test (name) VALUES (:name)\"), {\"name\": \"Alice\"})\n ... conn.commit()\n ... result = conn.execute(text(\"SELECT * FROM test\"))\n ... print(result.fetchall())\n [(1, 'Alice')]\n\nTransaction Management\n----------------------\n\nTransactions ensure that a series of database operations either all succeed or all\nfail together, maintaining data integrity. SQLAlchemy provides several ways to manage\ntransactions: implicit transactions with ``begin()``, context managers for automatic\ncommit/rollback, and manual control with ``commit()`` and ``rollback()``. The ``begin()``\nmethod starts a transaction that will automatically rollback on exceptions and commit\non successful completion when used as a context manager.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, text\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> # Using begin() for automatic commit/rollback\n >>> with engine.begin() as conn:\n ... conn.execute(text(\"CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT)\"))\n ... conn.execute(text(\"INSERT INTO users (name) VALUES ('Bob')\"))\n ... # Commits automatically if no exception\n\n >>> # Manual transaction control\n >>> with engine.connect() as conn:\n ... trans = conn.begin()\n ... try:\n ... conn.execute(text(\"INSERT INTO users (name) VALUES ('Carol')\"))\n ... trans.commit()\n ... except:\n ... trans.rollback()\n ... raise\n\nDefine Tables with Metadata\n---------------------------\n\n``MetaData`` is a container that holds information about database tables and other\nschema constructs. You can define tables programmatically using the ``Table`` class,\nspecifying columns with their types and constraints. This approach is part of\nSQLAlchemy Core and gives you explicit control over the table structure. The metadata\ncan then create all defined tables in the database with ``create_all()``, which\ngenerates the appropriate DDL statements for your database dialect.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, MetaData, Table, Column, Integer, String\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> metadata = MetaData()\n >>> users = Table(\n ... \"users\", metadata,\n ... Column(\"id\", Integer, primary_key=True),\n ... Column(\"name\", String(50)),\n ... Column(\"email\", String(100))\n ... )\n >>> metadata.create_all(engine)\n >>> # Check table columns\n >>> [c.name for c in users.columns]\n ['id', 'name', 'email']\n\nReflect Existing Tables\n-----------------------\n\nTable reflection allows SQLAlchemy to load table definitions from an existing database\nschema automatically. This is useful when working with legacy databases or when you\nwant to avoid duplicating schema definitions. The ``reflect()`` method on ``MetaData``\nreads the database schema and creates ``Table`` objects for all tables found. You can\nalso reflect individual tables using ``autoload_with`` parameter.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, MetaData, Table, Column, Integer, String\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> # Create a table first\n >>> with engine.begin() as conn:\n ... conn.execute(text(\"CREATE TABLE products (id INTEGER PRIMARY KEY, name TEXT, price REAL)\"))\n\n >>> # Reflect the table\n >>> metadata = MetaData()\n >>> metadata.reflect(bind=engine)\n >>> list(metadata.tables.keys())\n ['products']\n >>> products = metadata.tables['products']\n >>> [c.name for c in products.columns]\n ['id', 'name', 'price']\n\nInspect Database Schema\n-----------------------\n\nThe ``inspect()`` function provides a powerful way to examine database schema details\nat runtime. The inspector can retrieve information about tables, columns, indexes,\nforeign keys, and other database objects. This is particularly useful for database\nadministration tasks, schema migrations, and debugging. The inspector works with\nany database supported by SQLAlchemy and provides a consistent API across different\ndatabase systems.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, inspect, MetaData, Table, Column, Integer, String\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> metadata = MetaData()\n >>> users = Table(\"users\", metadata,\n ... Column(\"id\", Integer, primary_key=True),\n ... Column(\"name\", String(50)))\n >>> metadata.create_all(engine)\n\n >>> inspector = inspect(engine)\n >>> inspector.get_table_names()\n ['users']\n >>> inspector.get_columns('users') # doctest: +ELLIPSIS\n [{'name': 'id', ...}, {'name': 'name', ...}]\n\nInsert Data\n-----------\n\nThe ``insert()`` construct creates an INSERT statement for a table. You can specify\nvalues using the ``values()`` method or pass them as keyword arguments. For bulk\ninserts, pass a list of dictionaries to ``execute()``. SQLAlchemy will generate\nefficient multi-row INSERT statements when possible. The ``returning()`` method\ncan retrieve auto-generated values like primary keys after insertion.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, MetaData, Table, Column, Integer, String, insert\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> metadata = MetaData()\n >>> users = Table(\"users\", metadata,\n ... Column(\"id\", Integer, primary_key=True),\n ... Column(\"name\", String(50)))\n >>> metadata.create_all(engine)\n\n >>> # Single insert\n >>> with engine.begin() as conn:\n ... conn.execute(insert(users).values(name=\"Alice\"))\n ... # Bulk insert\n ... conn.execute(insert(users), [{\"name\": \"Bob\"}, {\"name\": \"Carol\"}])\n\n >>> with engine.connect() as conn:\n ... result = conn.execute(users.select())\n ... print(result.fetchall())\n [(1, 'Alice'), (2, 'Bob'), (3, 'Carol')]\n\nSelect Data\n-----------\n\nThe ``select()`` construct creates SELECT statements with a Pythonic API. You can\nspecify which columns to retrieve, add WHERE clauses with ``where()``, order results\nwith ``order_by()``, and limit results with ``limit()`` and ``offset()``. The SQL\nExpression Language uses Python operators like ``==``, ``!=``, ``>``, ``<`` which\nare overloaded to generate SQL conditions. This provides type safety and prevents\nSQL injection.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, MetaData, Table, Column, Integer, String, select, insert\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> metadata = MetaData()\n >>> users = Table(\"users\", metadata,\n ... Column(\"id\", Integer, primary_key=True),\n ... Column(\"name\", String(50)),\n ... Column(\"age\", Integer))\n >>> metadata.create_all(engine)\n\n >>> with engine.begin() as conn:\n ... conn.execute(insert(users), [\n ... {\"name\": \"Alice\", \"age\": 30},\n ... {\"name\": \"Bob\", \"age\": 25},\n ... {\"name\": \"Carol\", \"age\": 35}])\n\n >>> with engine.connect() as conn:\n ... # Select all\n ... result = conn.execute(select(users))\n ... print(result.fetchall())\n ... # Select with condition\n ... result = conn.execute(select(users).where(users.c.age > 28))\n ... print(result.fetchall())\n [(1, 'Alice', 30), (2, 'Bob', 25), (3, 'Carol', 35)]\n [(1, 'Alice', 30), (3, 'Carol', 35)]\n\nUpdate Data\n-----------\n\nThe ``update()`` construct creates UPDATE statements. Use ``where()`` to specify\nwhich rows to update and ``values()`` to set new column values. Without a WHERE\nclause, all rows in the table will be updated. The ``returning()`` method can\nretrieve the updated values. For bulk updates with different values per row,\nuse ``bindparam()`` to create parameterized statements.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, MetaData, Table, Column, Integer, String\n >>> from sqlalchemy import select, insert, update\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> metadata = MetaData()\n >>> users = Table(\"users\", metadata,\n ... Column(\"id\", Integer, primary_key=True),\n ... Column(\"name\", String(50)))\n >>> metadata.create_all(engine)\n\n >>> with engine.begin() as conn:\n ... conn.execute(insert(users), [{\"name\": \"Alice\"}, {\"name\": \"Bob\"}])\n ... conn.execute(update(users).where(users.c.name == \"Alice\").values(name=\"Alicia\"))\n\n >>> with engine.connect() as conn:\n ... result = conn.execute(select(users))\n ... print(result.fetchall())\n [(1, 'Alicia'), (2, 'Bob')]\n\nDelete Data\n-----------\n\nThe ``delete()`` construct creates DELETE statements. Always use ``where()`` to\nspecify which rows to delete, unless you intend to delete all rows. Like other\nDML statements, ``delete()`` supports ``returning()`` to retrieve deleted rows.\nBe careful with DELETE statements as they permanently remove data from the database.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, MetaData, Table, Column, Integer, String\n >>> from sqlalchemy import select, insert, delete\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> metadata = MetaData()\n >>> users = Table(\"users\", metadata,\n ... Column(\"id\", Integer, primary_key=True),\n ... Column(\"name\", String(50)))\n >>> metadata.create_all(engine)\n\n >>> with engine.begin() as conn:\n ... conn.execute(insert(users), [{\"name\": \"Alice\"}, {\"name\": \"Bob\"}, {\"name\": \"Carol\"}])\n ... conn.execute(delete(users).where(users.c.name == \"Bob\"))\n\n >>> with engine.connect() as conn:\n ... result = conn.execute(select(users))\n ... print(result.fetchall())\n [(1, 'Alice'), (3, 'Carol')]\n\nSQL Expression Language\n-----------------------\n\nSQLAlchemy's SQL Expression Language provides a Pythonic way to construct SQL\nstatements. Column objects support comparison operators (``==``, ``!=``, ``>``, ``<``),\nlogical operators (``&`` for AND, ``|`` for OR), and methods like ``in_()``,\n``like()``, ``between()``, and ``is_()``. These expressions are composable and\ncan be combined to build complex queries while maintaining readability and type safety.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, MetaData, Table, Column, Integer, String\n >>> from sqlalchemy import select, insert, and_, or_\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> metadata = MetaData()\n >>> users = Table(\"users\", metadata,\n ... Column(\"id\", Integer, primary_key=True),\n ... Column(\"name\", String(50)),\n ... Column(\"age\", Integer))\n >>> metadata.create_all(engine)\n\n >>> with engine.begin() as conn:\n ... conn.execute(insert(users), [\n ... {\"name\": \"Alice\", \"age\": 30},\n ... {\"name\": \"Bob\", \"age\": 25},\n ... {\"name\": \"Carol\", \"age\": 35}])\n\n >>> with engine.connect() as conn:\n ... # AND condition\n ... stmt = select(users).where(and_(users.c.age > 25, users.c.age < 35))\n ... print(conn.execute(stmt).fetchall())\n ... # OR condition\n ... stmt = select(users).where(or_(users.c.name == \"Alice\", users.c.name == \"Bob\"))\n ... print(conn.execute(stmt).fetchall())\n ... # IN clause\n ... stmt = select(users).where(users.c.name.in_([\"Alice\", \"Carol\"]))\n ... print(conn.execute(stmt).fetchall())\n [(1, 'Alice', 30)]\n [(1, 'Alice', 30), (2, 'Bob', 25)]\n [(1, 'Alice', 30), (3, 'Carol', 35)]\n\nJoin Tables\n-----------\n\nThe ``join()`` method creates JOIN clauses between tables. SQLAlchemy can automatically\ndetermine join conditions based on foreign key relationships, or you can specify\nthem explicitly. Use ``select_from()`` to specify the joined tables in a SELECT\nstatement. SQLAlchemy supports INNER JOIN (default), LEFT OUTER JOIN, RIGHT OUTER\nJOIN, and FULL OUTER JOIN through the ``isouter`` and ``full`` parameters.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, MetaData, Table, Column, Integer, String, ForeignKey\n >>> from sqlalchemy import select, insert\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> metadata = MetaData()\n >>> users = Table(\"users\", metadata,\n ... Column(\"id\", Integer, primary_key=True),\n ... Column(\"name\", String(50)))\n >>> orders = Table(\"orders\", metadata,\n ... Column(\"id\", Integer, primary_key=True),\n ... Column(\"user_id\", Integer, ForeignKey(\"users.id\")),\n ... Column(\"product\", String(50)))\n >>> metadata.create_all(engine)\n\n >>> with engine.begin() as conn:\n ... conn.execute(insert(users), [{\"name\": \"Alice\"}, {\"name\": \"Bob\"}])\n ... conn.execute(insert(orders), [\n ... {\"user_id\": 1, \"product\": \"Book\"},\n ... {\"user_id\": 1, \"product\": \"Pen\"},\n ... {\"user_id\": 2, \"product\": \"Laptop\"}])\n\n >>> with engine.connect() as conn:\n ... stmt = select(users.c.name, orders.c.product).select_from(\n ... users.join(orders))\n ... print(conn.execute(stmt).fetchall())\n [('Alice', 'Book'), ('Alice', 'Pen'), ('Bob', 'Laptop')]\n\nAggregate Functions\n-------------------\n\nSQLAlchemy provides functions for SQL aggregates like ``count()``, ``sum()``,\n``avg()``, ``min()``, and ``max()`` in the ``sqlalchemy.func`` namespace. These\ncan be used in SELECT statements and combined with ``group_by()`` for grouped\naggregations. The ``func`` object is a special namespace that generates SQL\nfunction calls for any function name you access on it.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, MetaData, Table, Column, Integer, String\n >>> from sqlalchemy import select, insert, func\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> metadata = MetaData()\n >>> sales = Table(\"sales\", metadata,\n ... Column(\"id\", Integer, primary_key=True),\n ... Column(\"product\", String(50)),\n ... Column(\"amount\", Integer))\n >>> metadata.create_all(engine)\n\n >>> with engine.begin() as conn:\n ... conn.execute(insert(sales), [\n ... {\"product\": \"A\", \"amount\": 100},\n ... {\"product\": \"A\", \"amount\": 150},\n ... {\"product\": \"B\", \"amount\": 200}])\n\n >>> with engine.connect() as conn:\n ... # Count all rows\n ... result = conn.execute(select(func.count()).select_from(sales))\n ... print(result.scalar())\n ... # Sum with group by\n ... stmt = select(sales.c.product, func.sum(sales.c.amount)).group_by(sales.c.product)\n ... print(conn.execute(stmt).fetchall())\n 3\n [('A', 250), ('B', 200)]\n\nDrop Tables\n-----------\n\nTables can be dropped using the ``drop()`` method on a ``Table`` object or\n``drop_all()`` on ``MetaData`` to drop all tables. The ``checkfirst`` parameter\nprevents errors if the table doesn't exist. Be careful with these operations in\nproduction as they permanently delete data and schema. Always backup your database\nbefore dropping tables.\n\n.. code-block:: python\n\n >>> from sqlalchemy import create_engine, MetaData, Table, Column, Integer, String, inspect\n >>> engine = create_engine(\"sqlite:///:memory:\")\n >>> metadata = MetaData()\n >>> users = Table(\"users\", metadata, Column(\"id\", Integer, primary_key=True))\n >>> products = Table(\"products\", metadata, Column(\"id\", Integer, primary_key=True))\n >>> metadata.create_all(engine)\n\n >>> inspector = inspect(engine)\n >>> sorted(inspector.get_table_names())\n ['products', 'users']\n\n >>> # Drop single table\n >>> users.drop(engine)\n >>> sorted(inspect(engine).get_table_names())\n ['products']\n\n >>> # Drop all tables\n >>> metadata.drop_all(engine)\n >>> inspect(engine).get_table_names()\n []\n"
},
{
"path": "docs/notes/extension/cpp-from-python.rst",
"content": ".. meta::\n :description lang=en: Learn modern C++ syntax from Python - side-by-side comparison of Python and C++ code snippets\n :keywords: C++, Python, C++11, C++14, C++17, C++20, modern C++, syntax comparison, learn C++\n\n=======================\nLearn C++ from Python\n=======================\n\n.. contents:: Table of Contents\n :backlinks: none\n\nModern C++ (C++11, C++14, C++17, C++20) has evolved to include features that make it\nsyntactically similar to Python, making the transition easier for Python developers.\nThis comprehensive guide provides side-by-side comparisons and 1-1 mappings between\nPython and modern C++ code snippets, covering essential programming concepts like\nvariables, data structures, functions, lambdas, classes, and algorithms.\n\nWhether you're a Python developer looking to learn C++ for performance optimization,\nsystem programming, or expanding your programming skills, this tutorial demonstrates\nhow familiar Python patterns translate to modern C++ syntax. Many popular frameworks\nlike PyTorch, TensorFlow, and NumPy use C++ extensions for performance-critical operations,\nespecially in deep learning, LLM training, and CUDA GPU programming. Understanding C++\nenables you to write custom extensions, optimize bottlenecks, and contribute to these\nhigh-performance libraries.\n\nTo learn more about C++ programming, refer to this `C++ cheatsheet `_\nfor additional reference and best practices.\n\n**Complete working examples:** See `cpp_from_py.cpp `_\nfor runnable code with integrated Google Test suite. Each function includes Doxygen comments showing the equivalent Python code.\n\nHello World\n-----------\n\nThe traditional first program in any language. Both Python and C++ can print text to\nthe console, though C++ requires including the iostream library and a main function.\n\n**Python**\n\n.. code-block:: python\n\n print(\"Hello, World!\")\n\n**C++**\n\n.. code-block:: cpp\n\n #include \n\n int main() {\n std::cout << \"Hello, World!\" << std::endl;\n return 0;\n }\n\nVariables\n---------\n\nModern C++ supports automatic type inference with the ``auto`` keyword, making variable\ndeclarations as concise as Python. The compiler deduces types from initialization values.\n\n**Python**\n\n.. code-block:: python\n\n x = 10\n y = 3.14\n name = \"Alice\"\n is_valid = True\n\n**C++**\n\n.. code-block:: cpp\n\n auto x = 10;\n auto y = 3.14;\n auto name = \"Alice\";\n auto is_valid = true;\n\nLists and Vectors\n-----------------\n\nPython lists and C++ vectors are dynamic arrays that can grow and shrink. Both support\nindexing, appending elements, and querying size. C++ vectors require specifying the element\ntype, but modern C++ can infer it from initialization.\n\n**Python**\n\n.. code-block:: python\n\n numbers = [1, 2, 3, 4, 5]\n numbers.append(6)\n print(numbers[0])\n print(len(numbers))\n\n**C++**\n\n.. code-block:: cpp\n\n #include \n\n std::vector numbers = {1, 2, 3, 4, 5};\n numbers.push_back(6);\n std::cout << numbers[0] << std::endl;\n std::cout << numbers.size() << std::endl;\n\nArray Slicing and Access\n-------------------------\n\nPython supports powerful slicing syntax with negative indices and ranges. C++ doesn't have\nbuilt-in slicing, but you can use iterators or create subvectors. Negative indexing requires\nmanual calculation from the end.\n\n**Python**\n\n.. code-block:: python\n\n numbers = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n\n print(numbers[0])\n print(numbers[-1])\n print(numbers[2:5])\n print(numbers[:3])\n print(numbers[7:])\n print(numbers[::2])\n print(numbers[::-1])\n\n**C++**\n\n.. code-block:: cpp\n\n #include \n #include \n\n std::vector numbers = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9};\n\n std::cout << numbers[0] << std::endl;\n std::cout << numbers[numbers.size() - 1] << std::endl;\n\n std::vector slice1(numbers.begin() + 2, numbers.begin() + 5);\n std::vector slice2(numbers.begin(), numbers.begin() + 3);\n std::vector slice3(numbers.begin() + 7, numbers.end());\n\n std::vector every_second;\n for (size_t i = 0; i < numbers.size(); i += 2) {\n every_second.push_back(numbers[i]);\n }\n\n std::vector reversed(numbers.rbegin(), numbers.rend());\n\nDictionaries and Maps\n---------------------\n\nDictionaries in Python and maps in C++ store key-value pairs. Both allow insertion, lookup,\nand modification using bracket notation. C++ maps keep keys sorted, while Python dicts\nmaintain insertion order (Python 3.7+).\n\n**Python**\n\n.. code-block:: python\n\n ages = {\"Alice\": 30, \"Bob\": 25}\n ages[\"Charlie\"] = 35\n print(ages[\"Alice\"])\n\n**C++**\n\n.. code-block:: cpp\n\n #include
![\"pysheeet\"](\"docs/_static/logo.png\"){kind=logo}
\n ![\"Build](\"https://github.com/crazyguitar/pysheeet/actions/workflows/pythonpackage.yml/badge.svg\"){kind=icon}
\n \n \n ![\"Coverage\"](\"https://coveralls.io/repos/github/crazyguitar/pysheeet/badge.svg?branch=master\"){kind=icon}
\n \n \n ![\"License](\"https://img.shields.io/badge/License-MIT-blue.svg\"){kind=icon}
\n \n \n ![\"DOI\"](\"https://zenodo.org/badge/52760178.svg\"){kind=icon}
\n \n