Repository: charlax/professional-programming Branch: master Commit: 1e2768d74a42 Files: 28 Total size: 263.2 KB Directory structure: gitextract_x6h_2dt0/ ├── .pre-commit-config.yaml ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── antipatterns/ │ ├── README.md │ ├── code-antipatterns.md │ ├── database-antipatterns.md │ ├── error-handling-antipatterns.md │ ├── mvcs-antipatterns.md │ ├── python-examples/ │ │ ├── reraise_exceptions_bad.py │ │ └── reraise_exceptions_good.py │ ├── scalability-antipatterns.md │ ├── sqlalchemy-antipatterns.md │ ├── sqlalchemy-examples/ │ │ └── exists.py │ └── tests-antipatterns.md ├── feature-flags.md ├── glossary.md ├── list-of-links/ │ └── List of AI and LLM learning resources 20260104144118.md └── training/ ├── front-end/ │ ├── 01-modern-javascript.md │ ├── 02-react.md │ ├── 03-typescript.md │ ├── 04-the-browser.md │ ├── 05-css-html-restart.md │ ├── 06-front-end-development-practices.md │ ├── README.md │ └── examples/ │ └── hoisting.html ├── learning-python/ │ └── README.md └── web-dev/ └── README.md ================================================ FILE CONTENTS ================================================ ================================================ FILE: .pre-commit-config.yaml ================================================ repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v6.0.0 hooks: - id: check-yaml - id: end-of-file-fixer - id: trailing-whitespace - repo: https://github.com/psf/black rev: 19.3b0 hooks: - id: black - repo: https://github.com/thlorenz/doctoc rev: v1.4.0 hooks: - id: doctoc args: - --title - "## Table of Contents" ================================================ FILE: CONTRIBUTING.md ================================================ ## Table of Contents - [Contributing](#contributing) - [Install pre-commit hooks](#install-pre-commit-hooks) # Contributing ## Install pre-commit hooks Install `pre-commit` (e.g. with pipx) ```bash pre-commit install ``` ================================================ FILE: LICENSE ================================================ MIT License Copyright (c) 2017 Charles-Axel Dein Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================ FILE: README.md ================================================ ## Table of Contents - [Professional Programming - about this list](#professional-programming---about-this-list) - [Principles](#principles) - [Contributing to this list](#contributing-to-this-list) - [Must-read books](#must-read-books) - [Must-read articles](#must-read-articles) - [Other general material and list of resources](#other-general-material-and-list-of-resources) - [Other lists](#other-lists) - [Books](#books) - [Articles](#articles) - [Axioms](#axioms) - [Courses](#courses) - [Topics](#topics) - [Accounting](#accounting) - [Algorithm and data structures](#algorithm-and-data-structures) - [API design & development](#api-design--development) - [Attitude, habits, mindset](#attitude-habits-mindset) - [Procrastination](#procrastination) - [Authentication/authorization](#authenticationauthorization) - [Automation](#automation) - [Best practices](#best-practices) - [Beyond software engineering & random](#beyond-software-engineering--random) - [Biases](#biases) - [Business](#business) - [Buy vs. Build](#buy-vs-build) - [Cache](#cache) - [Career growth](#career-growth) - [Choosing your next/first opportunity](#choosing-your-nextfirst-opportunity) - [Getting to Staff Eng](#getting-to-staff-eng) - [Characters sets](#characters-sets) - [Chess](#chess) - [Clouds](#clouds) - [Code reviews](#code-reviews) - [Coding & code quality](#coding--code-quality) - [Communication](#communication) - [Compilers](#compilers) - [Configuration](#configuration) - [Continuous Integration (CI)](#continuous-integration-ci) - [Data analysis & data science](#data-analysis--data-science) - [Databases](#databases) - [NoSQL](#nosql) - [Postgres](#postgres) - [Data formats](#data-formats) - [Data science/data engineering](#data-sciencedata-engineering) - [Debugging](#debugging) - [Design (visual, UX, UI, typography)](#design-visual-ux-ui-typography) - [Design (OO modeling, architecture, patterns, anti-patterns, etc.)](#design-oo-modeling-architecture-patterns-anti-patterns-etc) - [Design: database schema](#design-database-schema) - [Design: patterns](#design-patterns) - [Design: simplicity](#design-simplicity) - [Dev environment & tools](#dev-environment--tools) - [Docker](#docker) - [Documentation](#documentation) - [Dotfiles](#dotfiles) - [Editors & IDE](#editors--ide) - [Vim](#vim) - [Email](#email) - [Engineering management](#engineering-management) - [Exercises](#exercises) - [Experimentation](#experimentation) - [Functional programming (FP)](#functional-programming-fp) - [Games development](#games-development) - [Graphics](#graphics) - [Hardware](#hardware) - [HTTP](#http) - [Humor](#humor) - [Incident response (oncall, alerting, outages, firefighting, postmortem)](#incident-response-oncall-alerting-outages-firefighting-postmortem) - [Postmortem](#postmortem) - [Internet](#internet) - [Interviewing](#interviewing) - [Kubernetes](#kubernetes) - [Large Language Model (LLM)](#large-language-model-llm) - [Learning & memorizing](#learning--memorizing) - [Licenses (legal)](#licenses-legal) - [Linux (system management)](#linux-system-management) - [Low-code/no-code](#low-codeno-code) - [Low-level, assembly](#low-level-assembly) - [Machine learning/AI](#machine-learningai) - [Math](#math) - [Marketing](#marketing) - [Network](#network) - [Observability (monitoring, logging, exception handling)](#observability-monitoring-logging-exception-handling) - [Logging](#logging) - [Error/exception handling](#errorexception-handling) - [Metrics](#metrics) - [Monitoring](#monitoring) - [Open source](#open-source) - [Operating system (OS)](#operating-system-os) - [Over-engineering](#over-engineering) - [Performance](#performance) - [Personal knowledge management (PKM)](#personal-knowledge-management-pkm) - [Personal productivity](#personal-productivity) - [Perspective](#perspective) - [Privacy](#privacy) - [Problem solving](#problem-solving) - [Product management for software engineers](#product-management-for-software-engineers) - [Project management](#project-management) - [Programming languages](#programming-languages) - [Python](#python) - [JavaScript](#javascript) - [Garbage collection](#garbage-collection) - [Programming paradigm](#programming-paradigm) - [Public speaking (presenting)](#public-speaking-presenting) - [Reading](#reading) - [Refactoring](#refactoring) - [Regex](#regex) - [Releasing & deploying](#releasing--deploying) - [Versioning](#versioning) - [Checklists](#checklists) - [Feature flags](#feature-flags) - [Testing in production](#testing-in-production) - [Reliability](#reliability) - [Integration patterns (dependency management)](#integration-patterns-dependency-management) - [Resiliency](#resiliency) - [Search](#search) - [Security](#security) - [Research papers](#research-papers) - [Shell (command line)](#shell-command-line) - [SQL](#sql) - [State](#state) - [System administration](#system-administration) - [System architecture](#system-architecture) - [Architecture patterns](#architecture-patterns) - [Microservices/splitting a monolith](#microservicessplitting-a-monolith) - [Scalability](#scalability) - [Site Reliability Engineering (SRE)](#site-reliability-engineering-sre) - [Technical debt](#technical-debt) - [Testing](#testing) - [Tools](#tools) - [Type system](#type-system) - [Typography](#typography) - [Version control (Git)](#version-control-git) - [Work ethics, productivity & work/life balance](#work-ethics-productivity--worklife-balance) - [Web development](#web-development) - [Writing (communication, blogging)](#writing-communication-blogging) - [Resources & inspiration for presentations](#resources--inspiration-for-presentations) - [Keeping up-to-date](#keeping-up-to-date) - [Concepts](#concepts) - [My other lists](#my-other-lists) # Professional Programming - about this list > Give me six hours to chop down a tree and I will spend the first four sharpening the axe. (Abraham Lincoln) A collection of full-stack resources for programmers. The goal of this page is to make you a more proficient developer. You'll find only resources that I've found truly inspiring, or that have become timeless classics. ## Principles - This page is not meant to be comprehensive. I am trying to keep it light and not too overwhelming. - The selection of articles is opinionated. - I don't necessarily agree with or endorse every single line that is written in every single one of those resources. The same applies to their authors: I don't endorse everything each of those authors has said and will ever say. Items: - 🧰 : list of resources - 📖 : book - 🎞 : video/movie extract/movie/talk - 🏙 : slides/presentation - ⭐️ : must-read - 📃 : paper ## Contributing to this list Feel free to open a PR to contribute! I will not be adding everything: as stated above, I am trying to keep the list concise. ## Must-read books I've found these books incredibly inspiring: - 📖 [The Pragmatic Programmer: From Journeyman to Master](https://pragprog.com/titles/tpp20/): hands-on the most inspiring and useful book I've read about programming. - 📖 [Code Complete: A Practical Handbook of Software Construction](http://www.amazon.com/Code-Complete-Practical-Handbook-Construction/dp/0735619670): a nice addition to The Pragmatic Programmer, gives you the necessary framework to talk about code. - 📖 [Release It!](https://smile.amazon.com/Release-Design-Deploy-Production-Ready-Software/dp/1680502395): this books goes beyond code and gives you best practices for building production-ready software. It will give you about 3 years worth of real-world experience. - 📖 [Scalability Rules: 50 Principles for Scaling Web Sites](https://smile.amazon.com/Scalability-Rules-Principles-Scaling-Sites/dp/013443160X) - 📖 [The Linux Programming Interface: A Linux and UNIX System Programming Handbook](http://www.amazon.com/The-Linux-Programming-Interface-Handbook/dp/1593272200): outside of teaching you almost everything you need to know about Linux, this book will give you insights into how software evolves, and the value of having simple & elegant interfaces. - 📖 [Structure and interpretation of Computer Programs](https://web.mit.edu/6.001/6.037/sicp.pdf) (free): One of the most influential textbooks in Computer Science (written and used at MIT), SICP has been influential in CS education. [Byte]() recommended SICP "for professional programmers who are really interested in their profession". There are some free books available, including: - 📖 [Professional software development](http://mixmastamyk.bitbucket.io/pro_soft_dev/): pretty complete and a good companion to this page. The free chapters are mostly focused on software development processes: design, testing, code writing, etc. - and not so much about tech itself. - 🧰 [vhf/free-programming-books](https://github.com/vhf/free-programming-books) - 🧰 [EbookFoundation/free-programming-books](https://github.com/EbookFoundation/free-programming-books/tree/main/books) ## Must-read articles - [Practical Advice for New Software Engineers](http://product.hubspot.com/blog/practical-advice-for-new-software-engineers) - [On Being A Senior Engineer](http://www.kitchensoap.com/2012/10/25/on-being-a-senior-engineer/) - [Lessons Learned in Software Development](http://henrikwarne.com/2015/04/16/lessons-learned-in-software-development/): one of those articles that give you years of hard-earned lessons, all in one short article. Must read. - [Things I Learnt The Hard Way](https://blog.juliobiason.me/thoughts/things-i-learnt-the-hard-way/) - Spec first, then code - Tests make better APIs - Future thinking is future trashing - Documentation is a love letter to your future self - Sometimes, it's better to let the application crash than do nothing - Understand and stay away of cargo cult - "Right tool for the job" is just to push an agenda - Learn the basics functional programming - ALWAYS use timezones with your dates - ALWAYS use UTF-8 - Create libraries - Learn to monitor - Explicit is better than implicit - Companies look for specialists but keep generalists longer - The best secure way to deal with user data is not to capture it - When it's time to stop, it's time to stop - You're responsible for the use of your code - Don't tell "It's done" when it's not - Pay attention on how people react to you - Beware of micro-aggressions - Keep a list of "Things I Don't Know" - [Signs that you're a good programmer](https://skatgame.net/mburo//courses/350/signs-that-you-re-a-good-programmer.html) (not everything in here is great - some of the points are counterproductive) - The instinct to experiment first - Emotional detachment from code and design - Eager to fix what isn't broken - Fascinated by the incomprehensible - Compelled to teach - Incorruptible patience - A destructive pursuit of perfection - Encyclopedic grasp of the platform - Thinks In Code - When In Rome, Does As Romans Do - Creates their own tools - Indifferent to Hierarchy - Excited by failure - Indifferent to circumstances - Substitutes impulse for commitment - Driven by experiences - [7 absolute truths I unlearned as junior developer](https://monicalent.com/blog/2019/06/03/absolute-truths-unlearned-as-junior-developer/) - Early in your career, you can learn 10x more in a supportive team in 1 year, than coding on your own - Every company has problems, every company has technical debt. - Being overly opinionated on topics you lack real-world experience with is pretty arrogant. - Many conference talks cover proof of concepts rather than real-world scenarios. - Dealing with legacy is completely normal. - Architecture is more important than nitpicking. - Focus on automation over documentation where appropriate. - Having some technical debt is healthy. - Senior engineers must develop many skills besides programming. - We’re all still junior in some areas. - [How to Build Good Software](https://knowledge.csc.gov.sg/ethos-issue-21/how-to-build-good-software/) - A good high-level summary of fundamental engineering practices. - The root cause of bad software has less to do with specific engineering choices, and more to do with how development projects are managed. - There is no such thing as platonically good engineering: it depends on your needs and the practical problems you encounter. - Software should be treated not as a static product, but as a living manifestation of the development team’s collective understanding. - Software projects rarely fail because they are too small; they fail because they get too big. - Beware of bureaucratic goals masquerading as problem statements. If our end goal is to make citizens’ lives better, we need to explicitly acknowledge the things that are making their lives worse. - Building software is not about avoiding failure; it is about strategically failing as fast as possible to get the information you need to build something good. - [How to be a -10x Engineer](https://taylor.town/-10x) - Nullify the output of 10 engineers. - Hold 10 engineers hostage in a technical discussion. - Waste 10 weeks of wages on cloud costs. - Waste 400 hours of engineering on bad architecture. - Incur 400 hours of bug triage. - [A Bunch of Programming Advice I'd Give To Myself 15 Years Ago](https://mbuffett.com/posts/programming-advice-younger-self/) - If you (or your team) are shooting yourselves in the foot constantly, fix the gun - Assess the trade-off you’re making between quality and pace, make sure it’s appropriate for your context - Spending time sharpening the axe is almost always worth it - If you can’t easily explain why something is difficult, then it’s incidental complexity, which is probably worth addressing - Try to solve bugs one layer deeper - Don’t underestimate the value of digging into history to investigate some bugs - Bad code gives you feedback, perfect code doesn’t. Err on the side of writing bad code - Make debugging easier - When working on a team, you should usually ask the question - Shipping cadence matters a lot. Think hard about what will get you shipping quickly and often - [Expert Generalists](https://martinfowler.com/articles/expert-generalist.html), martinfowler.com, proposes an interesting take on the "T-shaped engineer" - The Characteristics of an Expert Generalist: Curiosity, Collaborativeness, Customer Focus, Favor Fundamental Knowledge, Blend of Generalist and Specialist Skills, Sympathy for Related Domains - Assessing Expert Generalists: hiring and career progression - Growing Expert Generalists: From Tools to Fundamentals - "Why does our attention keep drifting toward tool expertise? It isn't because people are shortsighted or lazy; it's because the fundamentals are hard to see amid the noise." - Expert Generalists still need Specialists - Expert Generalists in the Age of LLMs - "Similarly to a specialist, an LLM can rapidly answer questions that an Expert Generalist will have when working in a new domain." - "Rather than looking for “the answer”, they prompt them to generate questions, explaining mechanisms, and providing examples and even tools that help explore the underlying mechanisms of an idea." ## Other general material and list of resources ### Other lists - [liuchong/awesome-roadmaps: A curated list of roadmaps.](https://github.com/liuchong/awesome-roadmaps) ### Books - 📖 [The Imposter's Handbook](https://bigmachine.io/products/the-imposters-handbook) - \$30. From the author: "Don't have a CS Degree? Neither do I - That's why I wrote this book." - 📖 [The Computer Science Book](https://thecomputersciencebook.com/book/) - 📖 [The Software Engineer's Guidebook](https://www.engguidebook.com/): Gergely Orosz's guidebook to the software engineering industry. Extremely complete. ### Articles - [mr-mig/every-programmer-should-know: a collection of (mostly) technical things every software developer should know](https://github.com/mr-mig/every-programmer-should-know) - [Famous Laws Of Software Development](https://www.timsommer.be/famous-laws-of-software-development/) - [The Amazon Builders' Library](https://aws.amazon.com/builders-library/?cards-body.sort-by=item.additionalFields.customSort&cards-body.sort-order=asc) - There is a list of the best articles in this [Twitter Thread](https://twitter.com/g_bonfiglio/status/1673650452846505985) - [kdeldycke/awesome-falsehood](https://github.com/kdeldycke/awesome-falsehood): Falsehoods Programmers Believe in - [hellerve/programming-talks](https://github.com/hellerve/programming-talks) - [TechYaks](https://techyaks.com/): list of talks - [Talks that changed the way I think about programming](http://www.opowell.com/post/talks-that-changed-the-way-i-think-about-programming/) - [What every computer science major should know](http://matt.might.net/articles/what-cs-majors-should-know/) - [kamranahmedse/developer-roadmap](https://github.com/kamranahmedse/developer-roadmap) - [mtdvio/every-programmer-should-know](https://github.com/mtdvio/every-programmer-should-know): a collection of (mostly) technical things every software developer should know about - [Mike Acton’s Expectations of Professional Software Engineers](https://adamj.eu/tech/2022/06/17/mike-actons-expectations-of-professional-software-engineers/) - [Things they didn't teach you about Software Engineering](https://vadimkravcenko.com/shorts/things-they-didnt-teach-you/) - Domain knowledge is more important than your coding skills - Code is secondary. Business value is first. - You work with uncertainty most of the time - [We overestimate our short-term ability, but underestimate our long-term ability.](https://paavandesign.com/blog/ostaulta/) - Specialisation is for insects. - [Want an unfair advantage in your tech career? Consume content meant for other roles](https://matthewgrohman.substack.com/p/want-an-unfair-advantage-in-your) - Cross-functional understanding is critical in modern tech companies - Helps to avoid underestimating the importance and difficulty other roles - Helps you to be strategic in your interaction with people in that role - [Teach Yourself Programming in Ten Years](https://norvig.com/21-days.html) - [Mistakes You Apparently Just Have to Make Yourself](https://medium.com/@mcfunley/mistakes-you-apparently-just-have-to-make-yourself-cc2dd2bfc25c) - [The Best Programmers I Know](https://endler.dev/2025/best-programmers/) ### Axioms - [Precepts - Urbit](https://urbit.org/blog/precepts/) - Data is better than code. - Correctness is more important than performance. - Deterministic beats heuristic. - One hundred lines of simplicity is better than twenty lines of complexity. - If your abstractions are leaking, it's not due to some law of the universe; you just suck at abstracting. Usually, you didn't specify the abstraction narrowly enough. - If you avoid changing a section of code for fear of awakening the demons therein, you are living in fear. If you stay in the comfortable confines of the small section of the code you wrote or know well, you will never write legendary code. All code was written by humans and can be mastered by humans. - If there's clearly a right way to do something and a wrong way, do it the right way. Coding requires incredible discipline. - The best way to get the right answer is to try it the wrong way. - Practice tells you that things are good or bad; theory tells you why. - Not being qualified to solve a problem is no reason not to solve it. - If you don't understand a system you're using, you don't control it. If nobody understands the system, the system is in control. - [Embedded Rules of Thumb](https://embeddedartistry.com/blog/2018/04/26/embedded-rules-of-thumb/) - [50 Ideas That Changed My Life](https://www.perell.com/blog/50-ideas-that-changed-my-life) - [Reflections on 10,000 Hours of Programming](https://matt-rickard.com/reflections-on-10-000-hours-of-programming/) - [20 Things I've Learned in my 20 Years as a Software Engineer](https://www.simplethread.com/20-things-ive-learned-in-my-20-years-as-a-software-engineer/) ### Courses - [Google Tech Dev Guide](https://techdevguide.withgoogle.com/) - [The Missing Semester of Your CS Education](https://missing.csail.mit.edu/), MIT. Includes lectures about the shell, editors, data wrangling, git, debugging and profiling, meta programming, security and cryptography. - [Mathematics for the adventurous self-learner](https://www.neilwithdata.com/mathematics-self-learner), Neil Sainsbury - [jwasham/coding-interview-university](https://github.com/jwasham/coding-interview-university): a complete computer science study plan to become a software engineer. - [Teach Yourself Computer Science](https://teachyourselfcs.com/): an opinionated set of the best CS resources. - [ossu/computer-science](https://github.com/ossu/computer-science): free self-taught education in Computer Science! ## Topics ### Accounting - [Engineers Do Not Get To Make Startup Mistakes When They Build Ledgers](https://news.alvaroduran.com/p/engineers-do-not-get-to-make-startup) ### Algorithm and data structures - Read the [CLRS](https://mitpress.mit.edu/books/introduction-algorithms). You can watch and download the course on [OCW](http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-046j-introduction-to-algorithms-sma-5503-fall-2005/) - there are newer courses as well. - Or [The Algorithm Design Manual](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202?ie=UTF8&qid=1297127794&ref_=sr_1_1&sr=8-1) (Skiena) - Try out some algorithms on [Project Euler](https://projecteuler.net/) - [CS 61B Spring 2023](https://sp23.datastructur.es/) Other resources: - [Algorithms](http://jeffe.cs.illinois.edu/teaching/algorithms/), Jeff Erickson Here are some useful & interesting algo & DS visualizations: - [Grokking Algorithms](https://www.amazon.com/dp/1617292230/ref=cm_sw_su_dp) - [Essential Algorithms](https://www.amazon.com/Essential-Algorithms-Practical-Approach-Computer/dp/1118612108?ie=UTF8&*Version*=1&*entries*=0) - [Data Structure Visualization](https://www.cs.usfca.edu/~galles/visualization/Algorithms.html) - 🎞 [15 Sorting Algorithms in 6 Minutes](https://www.youtube.com/watch?v=kPRA0W1kECg&ab_channel=TimoBingmann) - [Hashing](https://samwho.dev/hashing/) - [Visualizing Algorithms](https://bost.ocks.org/mike/algorithms/) - [B-trees and database indexes](https://planetscale.com/blog/btrees-and-database-indexes) - [Big O visualizations](https://samwho.dev/big-o/) - [Algorithm explained like Ikea instructions](https://idea-instructions.com/) - [Decision Trees](https://mlu-explain.github.io/decision-tree/) Example implementations: - [trekhleb/javascript-algorithms](https://github.com/trekhleb/javascript-algorithms): algorithms and data structures implemented in JavaScript - [The Algorithms](https://the-algorithms.com/) Algorithms in distributed systems: - [Raft Consensus Algorithm](https://raft.github.io/) ### API design & development General REST content: - [Architectural Styles and the Design of Network-based Software Architectures](https://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm), Roy Fielding (the inventor of REST) - [A collection of useful resources for building RESTful HTTP+JSON APIs.](https://github.com/yosriady/api-development-tools) - [Best practices for REST API design](https://stackoverflow.blog/2020/03/02/best-practices-for-rest-api-design/), Stack Overflow Blog - 📖 [Undisturbed REST: a guide to designing the perfect API](https://www.mulesoft.com/sites/default/files/resource-assets/ebook-UndisturbedREST_v1.pdf): very complete book about RESTful API design. Example guidelines: - [Microsoft's Rest API guidelines](https://github.com/Microsoft/api-guidelines/blob/master/Guidelines.md) - [Zalando RESTful API and Event Scheme Guidelines](https://opensource.zalando.com/restful-api-guidelines/) - [Google's API Design Guide](https://cloud.google.com/apis/design/): a general guide to design networked API. - [AIP-1: AIP Purpose and Guidelines](https://google.aip.dev/1) - AIP stands for API Improvement Proposal, which is a design document providing high-level, concise documentation for API development. More specific topics: - [Why you should use links, not keys, to represent relationships in APIs](https://cloud.google.com/blog/products/application-development/api-design-why-you-should-use-links-not-keys-to-represent-relationships-in-apis), Martin Nally, Google - "Using links instead of foreign keys to express relationships in APIs reduces the amount of information a client needs to know to use an API, and reduces the ways in which clients and servers are coupled to each other." - [Give me /events, not webhooks](https://blog.sequin.io/events-not-webhooks/) - Events can unlock much-needed webhook features, like allowing your webhook consumers to replay or reset the position of their webhook subscription. - [Unlocking the Power of JSON Patch](https://zuplo.com/blog/2024/10/10/unlocking-the-power-of-json-patch) ### Attitude, habits, mindset - [Mastering Programming](https://www.prod.facebook.com/notes/kent-beck/mastering-programming/1184427814923414#), Kent Beck. - [The traits of a proficient programmer](https://www.oreilly.com/ideas/the-traits-of-a-proficient-programmer) - [The tao of programming](http://www.mit.edu/~xela/tao.html): a set of parables about programming. - [Taking Ownership Is The Most Effective Way to Get What You Want](http://www.theeffectiveengineer.com/blog/take-ownership-of-your-goals) - [Finding Time to Become a Better Developer](https://medium.freecodecamp.org/finding-time-to-become-a-better-developer-eebc154881b2) - [Ten minutes a day](https://medium.com/@alexallain/ten-minutes-a-day-e2fa1084f924): how Alex Allain wrote a book in less than 200 hours, by writing 10 minutes _every_ day. - [The care and feeding of software engineers (or, why engineers are grumpy)](https://humanwhocodes.com/blog/2012/06/12/the-care-and-feeding-of-software-engineers-or-why-engineers-are-grumpy/) - In the triumvirate of software, product managers, designers, and software engineers, only the engineers are expected to turn off their creative minds and just produce. - Both engineers and product managers tend to think, incorrectly, that product specifications or requirements are equivalent to the furniture manual from Ikea. - This is one of the top things that make engineers grumpy: constantly shifting priorities. - Even though many engineers will complain that product managers change their minds, almost none will account for that in their time estimates. - Computer science programs aren’t about preparing you for the tasks you’ll face in industry. - When there are more engineers than can be used, engineering time ends up going away from developing and towards planning, synchronization, and coordination. - Involve engineers in the creative process - Give engineers opportunities to be creative. - Encourage time off. - Let 'em code - Express appreciation - [The Product-Minded Software Engineer](https://blog.pragmaticengineer.com/the-product-minded-engineer/), Gergely Orosz - Great product engineers know that minimum lovable products need the right depth - Product-minded engineers quickly map out edge cases and think of ways to reduce work on them: often bringing solutions that require no engineering work - Engage in user research and customer support - Bring well-backed product suggestions to the table - Offer product/engineering tradeoffs - [40 Lessons From 40 Years](https://medium.com/@schlaf/40-lessons-from-40-years-de39d2c622d6), Steve Schlafman - If you want to make progress on the things that matter most, you need to decide who you’re going to disappoint. It’s inevitable. - The best investment you can make is your own education. Never stop learning. The second best investment you can make is building your network through authentic and meaningful interactions. It is what you know and who you know. - You’ll never get what you don’t ask for or actively seek out. Go for it! - It’s not about the light at the end of the tunnel. It’s the tunnel. Show up every day and enjoy the process. - A great teammate always puts the organization and its purpose ahead of their own self interests. - Pick your spots. We have limited time and our brains can only process so much. Focus is key. Choose wisely. - Every person is likely struggling with something. Be kind. Be helpful. - [On Coding, Ego and Attention](https://josebrowne.com/on-coding-ego-and-attention/) - Beginner’s mind accepts the fact that absolute knowledge is infinite and thus keeping score is a waste of time. - Mastery is simply the accumulation of momentum, not the accumulation of knowledge. - Dealing with ego distraction has taught me to love the problem solving process. It’s taught me to love and respect the learning process. As a result I’m more productive. I’m less anxious. I’m a better teammate. I’m a better friend and a better thinker. - [Fixed vs. Growth: The Two Basic Mindsets That Shape Our Lives](https://www.brainpickings.org/2014/01/29/carol-dweck-mindset/) - [What does a great software engineer look like?](https://fwouts.com/articles/great-software-engineer) - [Good sleep, good learning, good life](https://supermemo.guru/wiki/Good_sleep,_good_learning,_good_life) - 🎞 [Steve Jobs: if you don't ask for help, you won't get very far](https://www.youtube.com/watch?v=zkTf0LmDqKI&ab_channel=SiliconValleyHistoricalAssociation) - [Programming quotes](https://www.ronaldsvilcins.com/2020/12/10/programming-quotes/) - [Be Kind](https://boz.com/articles/be-kind) - Being kind is fundamentally about taking responsibility for your impact on the people around you. - It requires you be mindful of their feelings and considerate of the way your presence affects them. - [Warren Buffett Says This 1 Simple Habit Separates Successful People From Everyone Else](https://www.inc.com/marcel-schwantes/warren-buffett-says-this-is-1-simple-habit-that-separates-successful-people-from-everyone-else.html) - The difference between successful people and really successful people is that really successful people say no to almost everything. - [How to get lucky?](https://jjude.com/luck) - [Programmers should stop celebrating incompetence](https://world.hey.com/dhh/programmers-should-stop-celebrating-incompetence-de1a4725), DHH - The magic of programming is largely just things you don't know yet. - It's not fine to think you shouldn't be on some paths towards mastery, if you intend to make programming your career. - [There’s no speed limit](https://sive.rs/kimo) - [Don't Wait for Motivation, Act for Momentum](https://salman.io/blog/momentum-motivation/) - Start with a tiny task. Then ride its momentum. - [The Most Important Coding Habits](https://puppycoding.com/2023/07/22/healthy-coding-habits/) - The Most Important Coding Habits - Daily stretches - Take regular breaks - Don’t code late at night - Improve your coding environment - [Advice for new software devs who've read all those other advice essays](https://buttondown.email/hillelwayne/archive/advice-for-new-software-devs-whove-read-all-those/) - [Microservices aren't the problem. Incompetent people are](https://nondv.wtf/blog/posts/microservices-arent-the-problem-incompetent-people-are.html) - [High Agency](https://www.highagency.com/) (30-min read) - [What is "good taste" in software engineering?](https://www.seangoedecke.com/taste/): it's all about contextual tradeoff. > Imposter syndrome is underrated: a lot of talk goes into overcoming imposter syndrome. I say embrace self-skepticism and doubt yourself every day. In a fast-moving industry where lots of your knowledge expires every year, even the most junior people around you constantly cook up skills you don't have; you stay competitive by applying with the determination (and even fear) of the novice. The upside of this treadmill is that every engineer is on it: just because you're an imposter doesn't mean that other people are more deserving than you, because they're imposters too. You should advocate for yourself, take risks, pat yourself on the back when things go well, and, as you start to build a track record of solving problems, trust your skills and adaptability. Just make no mistake: you're only as good as the last problem you solve. Dan Heller, Building a Career in Software > I had learned already never to empty the well of my writing, but always to stop when there was still something there in the deep part of the well, and let it refill at night from the springs that fed it. -- Ernest Hemingway - [The Grug Brained Developer](https://grugbrain.dev): habits of self-aware programmer. Like Tao of Programming, different style. > Good judgment comes from experience. > Experience comes from bad judgment. #### Procrastination - [News is bad for you – and giving up reading it will make you happier](https://www.theguardian.com/media/2013/apr/12/news-is-bad-rolf-dobelli), The Guardian - News misleads - News is irrelevant - News has no explanatory power - News is toxic to your body - News increases cognitive errors - News inhibits thinking - News works like a drug - News wastes time - News makes us passive - News kills creativity ### Authentication/authorization - [Authorization in a microservices world](https://www.alexanderlolis.com/authorization-in-a-microservices-world) - [Authorization Logic: Rules are hard because they evolve over time](https://www.osohq.com/post/rules-are-hard-logic-for-authorization) - [The Copenhagen Book](https://thecopenhagenbook.com/) provides a general guideline on implementing auth in web applications ### Automation - [Automation Should Be Like Iron Man, Not Ultron](http://queue.acm.org/detail.cfm?id=2841313) ### Best practices - [Software engineering practices](https://simonwillison.net/2022/Oct/1/software-engineering-practices/#tested-dev-environments) ### Beyond software engineering & random - [Why Software Engineers like Woodworking](https://www.zainrizvi.io/blog/why-software-engineers-like-woodworking/) ### Biases Biases don't only apply to hiring. For instance, the fundamental attribution bias also applies when criticizing somebody's code written a long time ago, in a totally different context. - [Cognitive bias cheat sheet](https://buster.medium.com/cognitive-bias-cheat-sheet-55a472476b18). #hiring ### Business - [Payments 101 for a Developer](https://github.com/juspay/hyperswitch/wiki/Payments-101-for-a-Developer) - [The 4 biggest problems with homemade billing systems](https://www.getlago.com/blog/the-4-biggest-problems-with-homemade-billing-systems) - [🦑 The 14 pains of building your own billing system](https://arnon.dk/the-14-pains-of-billing/) ### Buy vs. Build - [Choose Boring Technology](https://boringtechnology.club/) - [Build vs. Buy](https://entropicthoughts.com/build-vs-buy) - The reason we want to buy as much as possible is that an organisation has a limited capacity for expertise, so we don’t want to have to become experts on things that don’t make up a competitive advantage. - [Platform Engineering: Build vs Buy](https://kanenarraway.com/posts/platform-engineering-build-vs-buy/) - If someone tells me they can build something cheaper than a vendor, I’m immediately skeptical because I don’t think most people can accurately forecast the actual cost of maintenance in the long term. ### Cache - [Caching challenges and strategies](https://aws.amazon.com/builders-library/caching-challenges-and-strategies/), Amazon Builders Library ### Career growth - [The Conjoined Triangles of Senior-Level Development](https://frontside.com/blog/2016-07-07-the-conjoined-triangles-of-senior-level-development) looks into how to define a senior engineer. - [Ten Principles for Growth as an Engineer](https://medium.com/@daniel.heller/ten-principles-for-growth-69015e08c35b), Dan Heller. - [Don't Call Yourself a Programmer](https://www.kalzumeus.com/2011/10/28/dont-call-yourself-a-programmer/), Patrick McKenzie. - [On being an Engineering Manager](https://nickmchardy.com/2019/02/on-being-an-engineering-manager.html) - [The career advice I wish I had at 25](https://www.linkedin.com/pulse/career-advice-i-wish-had-25-shane-rodgers/?trk=hp-feed-article-title-like) - A career is a marathon, not a sprint - Most success comes from repetition, not new things - If work was really so great all the rich people would have the jobs - Management is about people, not things - Genuinely listen to others - Recognise that staff are people with finite emotional capacity - Don’t just network with people your own age - Never sacrifice personal ethics for a work reason - Recognise that failure is learning - [Career advice I wish I’d been given when I was young](https://80000hours.org/2019/04/career-advice-i-wish-id-been-given-when-i-was-young/) - Don’t focus too much on long-term plans. - Find good thinkers and cold-call the ones you most admire. - Assign a high value to productivity over your whole lifespan. - Don’t over-optimise things that aren’t your top priority. - Read a lot, and read things that people around you aren’t reading. - Reflect seriously on what problem to prioritise solving. - Read more history. - [Why Good Developers are Promoted into Unhappiness](https://robwalling.com/2007/06/27/why-good-developers-are-promoted-into-unhappiness/), Rob Walling. Or why management might not be for you. - [A guide to using your career to help solve the world’s most pressing problems](https://80000hours.org/key-ideas/) - [What's a senior engineer's job?](https://jvns.ca/blog/senior-engineer/) You need to be more than just an individual contributor. - [From Coding Bootcamp Graduate to Building Distributed Databases](https://medium.com/swlh/from-coding-bootcamp-graduate-to-building-distributed-databases-29acbb723d8) - Read Books (and papers), not Blog Posts - Take responsibility for your career trajectory - 🏙 [The Well Rounded Engineer](https://speakerdeck.com/swanandp/the-well-rounded-engineer) includes lots of great book recommendations. - Paradigm polyglot (learn different languages & paradigms) - Database polyglot - Protocol polyglot (preferably TCP/IP and HTTP) - Proficiency with build tooling, packaging and distribution - Debugging, observability - Deployment, infra and devops - Software architecture and scaling - Ability to write toy compilers, interpreters and parsers - Ability to write toy games - Ability to understand algorithmic analysis - [Some career advice](https://lethain.com/career-advice/), Will Larson. - Advice you get is someone’s attempt to synthesize their experiences, not an accurate statement about how the world works. - Build a reservoir of prestige. - Some folks are so good at something that they end up being irreplaceable in their current role, which causes them to get stuck in their role even if they’re a good candidate for more interesting ones. - Great relationships will follow you everywhere you go. Bad ones too. - Early in your career, try to work at as many different kinds of companies and in different product vertical as you can. - [Evil tip: avoid "easy" things](http://yosefk.com/blog/evil-tip-avoid-easy-things.html) - [The Ultimate Code Kata](https://blog.codinghorror.com/the-ultimate-code-kata/) - [Traits of a senior software engineer](https://sergiomartins8.hashnode.dev/why-is-a-senior-engineer-senior): impact, perception, visibility, influence, mentoring - [Software Engineering - The Soft Parts](https://addyosmani.com/blog/software-engineering-soft-parts/) - Think critically and formulate well-reasoned arguments - Master the fundamentals - Focus on the user and all else will follow - Learn how to learn - [How To Own Your Growth As A Software Engineer](https://jes.al/2022/07/how-to-own-your-growth-as-a-software-engineer/) - [The Forty-Year Programmer](https://codefol.io/posts/the-forty-year-programmer/) - The Better You Get, the Less You Look Like Everybody Else - You Learn Deep Principles by Doing the Basics - Look to Other Fields, Learn From Other Fields - Be Careful About Productivity Tips - [Senior Engineers are Living in the Future](https://www.zerobanana.com/essays/living-in-the-future/) - [What would a map of your career look like?](https://tomcritchlow.com/2023/04/26/career-maps/) - [How to be successful at Amazon (or any other large company for that matter)](https://www.reddit.com/r/cscareerquestions/comments/4x0ugj/how_to_be_successful_at_amazon_or_any_other_large/) - [Being good isn’t enough](https://joshs.bearblog.dev/being-good-isnt-enough/) - The biggest gains come from combining disciplines. - [Stop Avoiding Politics](https://terriblesoftware.org/2025/10/01/stop-avoiding-politics/) is a good contrarian take on the current (negative) connotation of the word "politics". - In a way, it is unfortunate that "politics" got that connotation since Aristotle rightfully considered politics the highest form of practical wisdom, since it has to do with humans as social animals. - "Good politics is just being strategic about relationships and influence in the service of good outcomes." - Examples: building relationships before you need them, understanding the incentives, managing up effectively, creating win-win situation, being visible. - "The alternative to good politics isn’t no politics. It’s bad politics winning by default. It’s the loud person who’s wrong getting their way because the quiet person who’s right won’t speak up." - [21 Lessons from 14 Years at Google](https://addyo.substack.com/p/21-lessons-from-14-years-at-google) - [Publishing your work increases your luck](https://github.com/readme/guides/publishing-your-work) About senior engineers: - [Falsehoods Junior Developers believe about becoming Senior](https://vadimkravcenko.com/shorts/falsehoods-junior-developers-believe-about-becoming-senior/) #### Choosing your next/first opportunity - [Career Decisions - by Elad Gil - Elad Blog](https://blog.eladgil.com/p/career-decisions) #### Getting to Staff Eng - [I became a FAANG Staff Engineer in 5 years. These are the 14 lessons I learned along the way.](https://medium.com/geekculture/i-became-a-faang-staff-engineer-in-5-years-here-are-the-14-lessons-i-learned-along-the-way-f70ac078875c) - Software engineering isn’t just coding. Actually, coding is a small part of it. - Pipeline your work - Be open to feedback and listen. Like, seriously, listen. - Great feedback is hard to find; treasure it. - Keep an eye on the horizon (but not both). - Figure out what matters and let the rest go. - Comparison really is the thief of joy. - Mentorship is a beautiful thing. - Good days, in general, don’t just “happen”. - Advice and guidance are just that; they aren’t rules. - [Guides for reaching Staff-plus engineering roles](https://staffeng.com/guides/), Will Larson - [Being visible](https://staffeng.com/guides/being-visible) - [Additional resources on Staff-plus engineering](https://staffeng.com/guides/learning-materials) - [Staff archetypes](https://staffeng.com/guides/staff-archetypes/), Will Larson ### Characters sets - [The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)](http://www.joelonsoftware.com/articles/Unicode.html) - [The Absolute Minimum Every Software Developer Must Know About Unicode in 2023 (Still No Excuses!)](https://tonsky.me/blog/unicode/) ### Chess (yes - chess gets its own section :) - [Chessprogramming wiki](https://www.chessprogramming.org/Main_Page) - [Compressing chess moves](https://mbuffett.com/posts/compressing-chess-moves/) ### Clouds - [open-guides/og-aws](https://github.com/open-guides/og-aws): a practical guide to AWS ### Code reviews - [How to do a code review](https://google.github.io/eng-practices/review/reviewer/), Google's engineering practices documentation. - [Post-Commit Reviews](https://medium.com/@copyconstruct/post-commit-reviews-b4cc2163ac7a): an interesting idea to increase developer velocity (there are some caveats though). - [How to Make Your Code Reviewer Fall in Love with You](https://mtlynch.io/code-review-love/) - Review your own code first - Write a clear changelist description - Automate the easy stuff - Answer questions with the code itself - Narrowly scope changes - Separate functional and non-functional changes - Respond graciously to critiques - Artfully solicit missing information - Award all ties to your reviewer - Minimize lag between rounds of review - [How to Do Code Reviews Like a Human](https://mtlynch.io/human-code-reviews-1/) - [Ask HN: How do you review code?](https://news.ycombinator.com/item?id=11416746): great discussion on HackerNews, full of interesting ideas. - [Maslow's pyramid of code reviews](https://www.dein.fr/posts/2015-02-18-maslows-pyramid-of-code-review) - Another one on the same topic: [The Code Review Pyramid](https://www.morling.dev/blog/the-code-review-pyramid/) - [Code review in remote teams](https://web.hypothes.is/blog/code-review-in-remote-teams/): very complete set of rules. - [No code reviews by default](https://www.raycast.com/blog/no-code-reviews-by-default/) - Responsibility over convention ### Coding & code quality - [Write code that is easy to delete, not easy to extend](http://programmingisterrible.com/post/139222674273/write-code-that-is-easy-to-delete-not-easy-to) - [The Ten Commandments of Egoless Programming](http://blog.codinghorror.com/the-ten-commandments-of-egoless-programming/) - 📖 [Clean Code: A Handbook of Agile Software Craftsmanship](https://www.goodreads.com/book/show/3735293-clean-code), Robert C. Martin. Describes numerous useful best practices. A bit long. There's also a [clean code cheatsheet](cheatsheets/Clean-Code-V2.4.pdf). - [What Software Craftsmanship is about](https://blog.cleancoder.com/uncle-bob/2011/01/17/software-craftsmanship-is-about.html) - We’re tired of writing crap. - We will not accept the stupid old lie about cleaning things up later. - We will not believe the claim that quick means dirty. - We will not allow anyone to force us to behave unprofessionally. - [Tips on naming boolean variables](https://dev.to/michi/tips-on-naming-boolean-variables-cleaner-code-35ig) - There is a convention to prefix boolean variables and function names with "is" or "has". - Try to always use is, even for plurals (`isEachUserLoggedIn` is better than `areUsersLoggedIn` or `isUsersLoggedIn`) - Avoid custom prefixes (`isPaidFor` is better than `wasPaidFor`) - Avoid negatives (`isEnabled` is better than `isDisabled`) - [How To Write Unmaintainable Code](https://github.com/Droogans/unmaintainable-code/blob/master/README.md) - [kettanaito/naming-cheatsheet](https://github.com/kettanaito/naming-cheatsheet): : comprehensive language-agnostic guidelines on variables naming. Home of the A/HC/LC pattern. - 🧰 [Quality Engineering Guides](https://qeunit.com/guides/) - [What Makes Code Hard To Read: Visual Patterns of Complexity](https://seeinglogic.com/posts/visual-readability-patterns/) - [zakirullin/cognitive-load: 🧠 Cognitive Load is what matters](https://github.com/zakirullin/cognitive-load) - [How to create software quality](https://lethain.com/quality/) ### Communication See also the Writing section - [How to communicate effectively as a developer](https://www.karlsutt.com/articles/communicating-effectively-as-a-developer/) - Lots of concrete advice and examples for short, medium and long-form writing - [What Do You Visualize While Programming?](https://dillonshook.com/what-do-you-visualize-while-programming/) ### Compilers - [The Compiler Writer Resource Page](https://c9x.me/compile/bib/) - [kanaka/mal](https://github.com/kanaka/mal): mal - Make a Lisp ### Configuration - [The downsides of JSON for config files](https://arp242.net/weblog/JSON_as_configuration_files-_please_dont.html), Martin Tournoij. - Can't add comments - Excessive quotation and syntax noise - Using DC (declarative configuration) to control logic is often not a good idea. - [Your configs suck? Try a real programming language](https://beepb00p.xyz/configs-suck.html) - Most modern config formats suck - Use a real programming language - [Code rant: The Configuration Complexity Clock](https://mikehadlow.blogspot.com/2012/05/configuration-complexity-clock.html) - I’m not saying that it’s never appropriate to implement complex configuration, a rules-engine or a DSL, Indeed I would jump at the chance of building a DSL given the right requirements, but I am saying that you should understand the implications and recognise where you are on the clock before you go down that route. - Initially there was hope that non-technical business users would be able to use the GUI to configure the application, but that turned out to be a false hope; the mapping of business rules into the engine requires a level of expertise that only some members of the development team possess. ### Continuous Integration (CI) - [Continuous Integration](https://martinfowler.com/articles/continuousIntegration.html), MartinFowler.com ### Data analysis & data science - [Ways to make fake data look meaningful](https://danbirken.com/statistics/2013/11/19/ways-to-make-fake-data-look-meaningful.html) - Don’t share the raw data - Don’t share your methodology - Don’t include confidence intervals - Don’t challenge your own data - 📖 [Python Data Science Handbook](https://jakevdp.github.io/PythonDataScienceHandbook/), O'Reilly ### Databases See also the SQL section. - [A plain English introduction to CAP Theorem](http://ksat.me/a-plain-english-introduction-to-cap-theorem) - [PACELC theorem](https://en.wikipedia.org/wiki/PACELC_theorem): "in case of network partitioning (P) in a distributed computer system, one has to choose between availability (A) and consistency (C) (as per the CAP theorem), but else (E), even when the system is running normally in the absence of partitions, one has to choose between latency (L) and consistency (C)." - [Zero downtime database migrations](https://blog.rainforestqa.com/2014-06-27-zero-downtime-database-migrations/) (code examples are using Rails but this works great for any programming language) - [Algorithms Behind Modern Storage Systems](https://queue.acm.org/detail.cfm?id=3220266), ACM Queue - [Let's Build a Simple Database](https://cstack.github.io/db_tutorial/) - [Readings in Database Systems, 5th Edition](http://www.redbook.io/) - [Comparing database types: how database types evolved to meet different needs](https://dataguide.prisma.io/intro/comparing-database-types) - [How does a relational database work](http://coding-geek.com/how-databases-work/) - [Use the index, Luke](https://use-the-index-luke.com/) - [Course introduction — MySQL for Developers](https://planetscale.com/learn/courses/mysql-for-developers/introduction/course-introduction), PlanetScale - [How Query Engines Work](https://howqueryengineswork.com/00-introduction.html) - [Why you should probably be using SQLite | Epic Web Dev](https://www.epicweb.dev/why-you-should-probably-be-using-sqlite) Scaling databases: - [How Figma's Databases Team Lived to Tell the Scale](https://www.figma.com/blog/how-figmas-databases-team-lived-to-tell-the-scale/): interesting story about sharding #### NoSQL - [NOSQL Patterns](http://horicky.blogspot.nl/2009/11/nosql-patterns.html) - [NoSQL Databases: a Survey and Decision Guidance](https://medium.baqend.com/nosql-databases-a-survey-and-decision-guidance-ea7823a822d#.9fe79qr90) - The DynamoDB docs has some great pages: - [Read Consistency](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadConsistency.html) - [From SQL to NoSQL](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/SQLtoNoSQL.html) - [NoSQL Design for DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-general-nosql-design.html) - [Redis Explained](https://architecturenotes.co/redis/) #### Postgres - [Safe Operations For High Volume PostgreSQL](https://www.braintreepayments.com/blog/safe-operations-for-high-volume-postgresql/) (this is for PostgreSQL but works great for other DBs as well). - [Transaction Isolation in Postgres, explained](https://www.thenile.dev/blog/transaction-isolation-postgres) - [PostgreSQL exercises](https://pgexercises.com/) - [Postgres operations cheat sheet](https://wiki.postgresql.org/wiki/Operations_cheat_sheet) - [Postgres: don't Do This](https://wiki.postgresql.org/wiki/Don't_Do_This) - [PostgreSQL and UUID as primary key](https://maciejwalkowiak.com/blog/postgres-uuid-primary-key/) - [Postgres lock explained](https://postgreslocksexplained.com/) "Just use postgres": - [Just use Postgres](https://mccue.dev/pages/8-16-24-just-use-postgres) - [Postgres is Enough](https://gist.github.com/cpursley/c8fb81fe8a7e5df038158bdfe0f06dbb) - [It’s 2026, Just Use Postgres](https://www.tigerdata.com/blog/its-2026-just-use-postgres) ### Data formats - [Falsehoods Programmers Believe About Phone Numbers](https://github.com/googlei18n/libphonenumber/blob/master/FALSEHOODS.md), Google's `libphonenumber`. - [Rules for Autocomplete](http://jeremymikkola.com/posts/2019_03_19_rules_for_autocomplete.html): rough specifications for autocomplete fields - [Falsehoods programmers believe about addresses](https://www.mjt.me.uk/posts/falsehoods-programmers-believe-about-addresses/) - [Falsehoods Programmers Believe About Names](https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/) - [kdeldycke/awesome-falsehood](https://github.com/kdeldycke/awesome-falsehood): falsehoods programmers believe in - [Understanding UUIDs, ULIDs and String Representations](https://sudhir.io/uuids-ulids) - [Falsehoods Programmers Believe About Falsehoods Lists](https://kevin.deldycke.com/2016/falsehoods-programmers-believe-about-falsehoods-lists) - [Australia/Lord_Howe is the weirdest timezone](https://ssoready.com/blog/engineering/truths-programmers-timezones/) - [A love letter to the CSV format](https://github.com/medialab/xan/blob/master/docs/LOVE_LETTER.md) - [Falsehoods Programmers Believe About Aviation](https://flightaware.engineering/falsehoods-programmers-believe-about-aviation/) - [Schemas - Schema.org](https://schema.org/docs/schemas.html) ### Data science/data engineering - [A dirty dozen: twelve common metric interpretation pitfalls in online controlled experiments](https://blog.acolyer.org/2017/09/25/a-dirty-dozen-twelve-common-metric-interpretation-pitfalls-in-online-controlled-experiments/) - [datastacktv/data-engineer-roadmap](https://github.com/datastacktv/data-engineer-roadmap): roadmap to becoming a data engineer - [Awesome Data Engineering Learning Path](https://awesomedataengineering.com/) - [Emerging Architectures for Modern Data Infrastructure](https://a16z.com/2020/10/15/the-emerging-architectures-for-modern-data-infrastructure/) - [How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh](https://martinfowler.com/articles/data-monolith-to-mesh.html) - Data platforms based on the data lake architecture have common failure modes that lead to unfulfilled promises at scale. - We need to consider domains as the first class concern, apply platform thinking to create self-serve data infrastructure, and treat data as a product. - [MLOps](https://madewithml.com/courses/mlops/) - [Uber's Big Data Platform: 100+ Petabytes with Minute Latency](https://eng.uber.com/uber-big-data-platform/) - [SQL should be the default choice for data transformation logic](https://www.robinlinacre.com/recommend_sql/) ### Debugging Also see the Incident Response section in this doc - [Rubber Duck Problem Solving](http://blog.codinghorror.com/rubber-duck-problem-solving/) - [Rubber Ducking](http://c2.com/cgi/wiki?RubberDucking) - [Five Whys](https://en.wikipedia.org/wiki/5_Whys) - [The Five Lies Analysis](https://serce.me/posts/14-10-2021-the-five-lies-analysis) - The real problem reveals itself when the technique becomes a part of a template. - Action items can be very distant from the root cause. - Related article: [The Evolution of SRE at Google](https://www.usenix.org/publications/loginonline/evolution-sre-google) - [The Infinite Hows](http://www.kitchensoap.com/2014/11/14/the-infinite-hows-or-the-dangers-of-the-five-whys/) criticizes the five whys method and advocates for a different set of questions to learn from the most from incidents. - See also: [Human errors: models and management](https://app.box.com/s/7z35l09amvr1vwxouh2s) - "The issue with the Five Whys is that it’s tunnel-visioned into a linear and simplistic explanation of how work gets done and events transpire." - "Human error becomes a starting point, not a conclusion." (Dekker, 2009) - "When we ask 'how?', we’re asking for a narrative." - "When it comes to decisions and actions, we want to know how it made sense for someone to do what they did." - At each "why" step, only one answer will be selected for further investigation. Asking "how" encourage broader exploration. - "In accident investigation, as in most other human endeavours, we fall prey to the What-You-Look-For-Is-What-You-Find or WYLFIWYF principle. This is a simple recognition of the fact that assumptions about what we are going to see (What-You-Look-For), to a large extent will determine what we actually find (What-You-Find)." (Hollnagel, 2009, p. 85) (see [illustration of WYLFIWYF](https://www.youtube.com/watch?v=vJG698U2Mvo)) - "A final reason why a 'root cause' may be selected is that it is politically acceptable as the identified cause. Other events or explanations may be excluded or not examined in depth because they raise issues that are embarrassing to the organization or its contractors or are politically unacceptable." (Nancy Leveson, Engineering a Safer World, p. 20) - [Bounded rationality](https://en.wikipedia.org/wiki/Bounded_rationality): rational individuals will select a decision that is satisfactory rather than optimal - The article provide concrete ways and questions to solicit stories from people, which will yield better insights. - What were you expecting to happen? - If you had to describe the situation to your colleague at that point, what would you have told? - Did this situation fit a standard scenario? - What were you trying to achieve?Were there multiple goals at the same time?Was there time pressure or other limitations on what you could do? - [See template here](http://www.kitchensoap.com/wp-content/uploads/2014/09/Velocity2014-PM-Fac-Handout-Debrief.pdf) - [Linux Performance Analysis in 60,000 Milliseconds](http://techblog.netflix.com/2015/11/linux-performance-analysis-in-60s.html) - [Post-Mortems at HubSpot: What I Learned From 250 Whys](https://product.hubspot.com/blog/bid/64771/post-mortems-at-hubspot-what-i-learned-from-250-whys) - [Debugging zine](https://jvns.ca/debugging-zine.pdf), Julian Evans - [If you understand a bug, you can fix it](https://wizardzines.com/comics/understand-can-fix/) - [The Thirty Minute Rule](https://daniel.feldroy.com/posts/thirty-minute-rule): if anyone gets stuck on something for more than 30 minutes, they should ask for help - [How to create a Minimal, Reproducible Example](https://stackoverflow.com/help/minimal-reproducible-example), Stack Overflow - [Some ways to get better at debugging](https://jvns.ca/blog/2022/08/30/a-way-to-categorize-debugging-skills/), Julia Evans - Learn the codebase - Learn the system (e.g., HTTP stack, database transactions) - Learn your tools (e.g., `strace`, `tcpdump`) - Learn strategies (e.g., writing code to reproduce, adding logging, taking a break) - Get experience: according to a study, "experts simply formed more correct hypotheses and were more efficient at finding the fault." - [What exactly is the 'Saff Squeeze' method of finding a bug?](https://stackoverflow.com/questions/23865274/what-exactly-is-the-saff-squeeze-method-of-finding-a-bug) - A systematic technique for deleting both test code and non-test code from a failing test until the test and code are small enough to understand. - [tcpdump is amazing](https://jvns.ca/blog/2016/03/16/tcpdump-is-amazing/), Julia Evans - [What we talk about when we talk about ‘root cause’](https://github.com/readme/guides/root-cause) - [David A. Wheeler's Review of "Debugging" by David J. Agans](https://dwheeler.com/essays/debugging-agans.html) - [Troubleshooting: The Skill That Never Goes Obsolete](https://www.autodidacts.io/troubleshooting/) - Includes links to interesting debugging stories - [Falsehoods software teams believe about user feedback](https://thoughtbot.com/blog/falsehoods-software-teams-believe-about-user-feedback) ### Design (visual, UX, UI, typography) I highly recommend reading [The Non-Designer's Design Book](http://www.amazon.com/gp/product/0133966151/ref=pd_lpo_sbs_dp_ss_1?pf_rd_p=1944687602&pf_rd_s=lpo-top-stripe-1&pf_rd_t=201&pf_rd_i=0321534042&pf_rd_m=ATVPDKIKX0DER&pf_rd_r=1R7MVQP0BCP7GP9VZGYX). This is a pretty short book that will give you some very actionable design advices. - If you're working on data, Edward Tufte's [The Visual Display of Quantitative Information](http://www.amazon.com/Visual-Display-Quantitative-Information/dp/0961392142/ref=sr_1_1?ie=UTF8&qid=1458046603&sr=8-1&keywords=tufte) is considered a classic. - The [Universal Principles of Design](http://www.amazon.com/Universal-Principles-Design-Revised-Updated/dp/1592535879/ref=sr_1_1?ie=UTF8&qid=1458046663&sr=8-1&keywords=universal+principles+of+design) will give you enough vocabulary and concepts to describe design challenges into words. - [Book recommendations from HackerNews](https://news.ycombinator.com/item?id=12711060) - 🏙[Design for Non-Designers](https://speakerdeck.com/tracymakes/design-for-non-designers-beyond-tellerand-dusseldorf-2018) Articles : - [10 Usability Heuristics Every Designer Should Know](https://uxdesign.cc/10-usability-heuristics-every-designer-should-know-129b9779ac53) - Visibility of System Status - The Match Between The System And The Real World - Every system should have a clear emergency exit - Don't forget that people spend 90% of their time interacting with other apps - Recognition Rather Than Recall (recognition = shallow form of retrieval from memory, e.g. a familiar person, recall = deeper retrieval) - ”Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.” – Antoine de Saint-Exupery - Help Users Recognize, Diagnose, And Recover From Errors - [How to pick more beautiful colors for your data visualizations](https://blog.datawrapper.de/beautifulcolors/) - [Visual design rules you can safely follow every time](https://anthonyhobday.com/sideprojects/saferules/) - [Malleable software: Restoring user agency in a world of locked-down apps](https://www.inkandswitch.com/essay/malleable-software/) Typograhy: see "Typography" section Resources: - 🧰 [bradtraversy/design-resources-for-developers](https://github.com/bradtraversy/design-resources-for-developers): design and UI resources from stock photos, web templates, CSS frameworks, UI libraries, tools... ### Design (OO modeling, architecture, patterns, anti-patterns, etc.) Here's a list of good books: - 📖 [Design Patterns: Elements of Reusable Object-Oriented Software](http://www.amazon.com/dp/0201633612/): dubbed "the gang of four", this is almost a required reading for any developer. A lot of those are a bit overkill for Python (because everything is an object, and dynamic typing), but the main idea (composition is better than inheritance) definitely is a good philosophy. - And their nefarious nemesis [Resign Patterns](http://nishitalab.org/user/paulo/files/resign-patterns.txt) - 📖 [Patterns of Enterprise Application Architecture](http://www.amazon.com/dp/0321127420/?tag=stackoverfl08-20): learn about how database are used in real world applications. Mike Bayer's SQLAlchemy has been heavily influenced by this book. - 📖 [Domain-Driven Design: Tackling Complexity in the Heart of Software](https://www.amazon.com/Domain-Driven-Design-Tackling-Complexity-Software/dp/0321125215), Eric Evans - 📖 [Clean Architecture](https://www.goodreads.com/book/show/18043011-clean-architecture), Robert C. Martin. Uncle Bob proposes an architecture that leverages the Single Responsibility Principle to its fullest. A great way to start a new codebase. Also checkout the [clean architecture cheatsheet](cheatsheets/Clean-Architecture-V1.0.pdf) and [this article](https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html). - 📖 [Game Programming Patterns](https://www.amazon.com/dp/0990582906/ref=cm_sw_em_r_mt_dp_U_9xygFb9M86CXY): a book about design, sequencing, behavioral patterns and much more by Robert Nystrom explained through the medium of game programming. The book is also free to read online [here](https://gameprogrammingpatterns.com/contents.html). One of the absolute references on architecture is Martin Fowler: checkout his [Software Architecture Guide](https://martinfowler.com/architecture/). Articles: - O'Reilly's [How to make mistakes in Python](https://www.oreilly.com/content/how-to-make-mistakes-in-python/) - [Education of a Programmer](https://hackernoon.com/education-of-a-programmer-aaecf2d35312): a developer's thoughts after 35 years in the industry. There's a particularly good section about design & complexity (see "the end to end argument", "layering and componentization"). - [Domain-driven design](https://en.wikipedia.org/wiki/Domain-driven_design), Wikipedia. - [On the Spectrum of Abstraction](https://www.youtube.com/watch?v=mVVNJKv9esE) 🎞, Cheng Lou - [The “Bug-O” Notation](https://overreacted.io/the-bug-o-notation/), Dan Abramov - [Antipatterns](./antipatterns) - [Inheritance vs. composition](http://learnpythonthehardway.org/book/ex44.html): a concrete example in Python. [Another slightly longer one here](http://python-textbok.readthedocs.io/en/latest/Object_Oriented_Programming.html). [One last one, in Python 3](http://blog.thedigitalcatonline.com/blog/2014/08/20/python-3-oop-part-3-delegation-composition-and-inheritance/#.V7SZ4tB96Rs). - [Composition Instead Of Inheritance](http://c2.com/cgi/wiki?CompositionInsteadOfInheritance) - [Complexity and Strategy](https://hackernoon.com/complexity-and-strategy-325cd7f59a92): interesting perspective on complexity and flexibility with really good examples (e.g. Google Apps Suite vs. Microsoft Office). - [The Architecture of Open Source Applications](https://aosabook.org/en/index.html) - [The Robustness Principle Reconsidered](https://cacm.acm.org/magazines/2011/8/114933-the-robustness-principle-reconsidered/fulltext) - Jon Postel: "Be conservative in what you do, be liberal in what you accept from others." (RFC 793) - Two general problem areas are impacted by the Robustness Principle: orderly interoperability and security. - [Basics of the Unix Philosophy](http://catb.org/esr/writings/taoup/html/ch01s06.html#id2877610), Eric S Raymond - [Eight Habits of Expert Software Designers: An Illustrated Guide](https://thereader.mitpress.mit.edu/habits-of-expert-software-designers/) - [No Silver Bullet - Essence and Accident in Software Engineering](https://worrydream.com/refs/Brooks_1986_-_No_Silver_Bullet.pdf), Frederick P. Brooks, Jr. (1986) - There are four properties of software systems which make building software hard: Complexity, Conformity, Changeability and Invisibility - There are ways to address this: - Exploiting the mass market to avoid constructing what can be bought. ("Buy vs. Build") - Using rapid prototyping as part of a planned iteration in establishing software requirements. - Growing software organically, adding more and more function to systems as they are run, used, and tested - Identifying and developing the great conceptual designers of the rising generation. - (also included in The Mythical Man-Month) - [Out of the Tar Pit](https://curtclifton.net/papers/MoseleyMarks06a.pdf), Ben Moseley, Peter Marks (2006) introduces the distinction between essential and accidental complexity - Complexity is the root cause of the vast majority of problems with software today. Unreliability, late delivery, lack of security — often even poor performance in large-scale systems can all be seen as deriving ultimately from unmanageable complexity. - Quoting Djikstra: "testing is hopelessly inadequate....(it) can be used very effectively to show the presence of bugs but never to show their absence." - Functional programming goes a long way towards avoiding the problems of state-derived complexity, thanks to immutability and clear separation of state and logic. - [A Note on Essential Complexity](https://olano.dev/blog/a-note-on-essential-complexity/) - The goal of the software engineer is to minimize accidental complexity and assist with essential complexity. - [Software Design is Knowledge Building](https://olano.dev/blog/software-design-is-knowledge-building/) - Programming should be regarded as an activity by which the programmers form or achieve a certain kind of insight, a theory, of the matters at hand. This suggestion is in contrast to what appears to be a more common notion, that programming should be regarded as a production of a program and certain other texts. - The building of the program is the same as the building of the theory of it by the team of programmers. - [Cognitive load is what matters](https://minds.md/zakirullin/cognitive#long) - A well-crafted monolith with truly isolated modules is often much more flexible than a bunch of microservices. - Three decades on, microkernel-based GNU Hurd is still in development, and monolithic Linux is everywhere - "Reduce cognitive load by limiting the number of choices." (Rob Pike) - The same rule applies to all sorts of numeric statuses (in the database or wherever) - prefer self-describing strings. - With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody. ([Hyrum's Law](https://www.hyrumslaw.com/)) - DDD is about problem space, not about solution space. - Familiarity is not the same as simplicity - The more mental models there are to learn, the longer it takes for a new developer to deliver value. - [CUPID: for joyful coding](https://dannorth.net/blog/cupid-for-joyful-coding/) - Composable: plays well with others - Unix philosophy: does one thing well - Predictable: does what you expect - Idiomatic: feels natural - Domain-based: the solution domain models the problem domain in language and structure - [Nobody Gets Promoted for Simplicity](https://terriblesoftware.org/2026/03/03/nobody-gets-promoted-for-simplicity/) > “Simplicity is a great virtue, but it requires hard work to achieve and education to appreciate. And to make matters worse, complexity sells better.” > — Edsger Dijkstra > You can use an eraser on the drafting table or a sledge hammer on the construction site. (Frank Lloyd Wright) Resources: - 🧰 [Design Principles](https://principles.design/) #### Design: database schema - [A humble guide to database schema design](https://www.mikealche.com/software-development/a-humble-guide-to-database-schema-design), Mike Alche - Use at least third normal form - Create a last line of defense with constraints - Never store full addresses in a single field - Never store firstname and lastname in the same field - Establish conventions for table and field names. - [YAGRI: You are gonna read it](https://www.scottantipa.com/yagri): store `created_at`, `created_by` etc. #### Design: patterns - [KeystoneInterface](https://martinfowler.com/bliki/KeystoneInterface.html), Martin Fowler. - Build all the back-end code, integrate, but don't build the user-interface - [101 Design Patterns & Tips for Developers](https://sourcemaking.com/design-patterns-and-tips) - [Python Design Patterns: For Sleek And Fashionable Code](https://www.toptal.com/python/python-design-patterns): a pretty simple introduction to common design patterns (Facade, Adapter, Decorator). A more complete list of design patterns implementation in Python on [Github](https://github.com/faif/python-patterns). - SourceMaking's [Design Patterns](https://sourcemaking.com/design_patterns) seems to be a good web resource too. - [Anti-If: The missing patterns](https://code.joejag.com/2016/anti-if-the-missing-patterns.html) #### Design: simplicity - [Simple Made Easy](https://www.infoq.com/presentations/Simple-Made-Easy) 🎞, Rich Hickey. This is an incredibly inspiring talk redefining simplicity, ease and complexity, and showing that solutions that look easy may actually harm your design. ### Dev environment & tools - 🧰 [Awesome Dev Env](https://github.com/jondot/awesome-devenv) Tools - [Glances: An eye on your system](https://github.com/nicolargo/glances) - [HTTPie: a CLI, cURL-like tool for humans](https://github.com/jkbrzt/httpie) - [jq: command-line JSON processor](https://stedolan.github.io/jq/) - [tmux: terminal multiplexer](http://tmux.github.io/) - [htop: an interactive process viewer for Linux](http://hisham.hm/htop/) - [htop explained](https://peteris.rocks/blog/htop/) - [socat](https://copyconstruct.medium.com/socat-29453e9fc8a6) - [Visual guide to SSH tunnels](https://robotmoon.com/ssh-tunnels/) - [casey/just](https://github.com/casey/just/): a command runner written in Rust (claims to be better than Makefile) - [Gazr](https://gazr.io/): an opinionated way to define your `Makefile` Article about tools: - [The return of fancy tools](https://macwright.com/2021/03/16/return-of-fancy-tools.html) - Simple tools make you think a little more - Drucker: "I’m not writing it down to remember it later, I’m writing it down to remember it now." - Frictionless note-taking produces notes, but it doesn't produce memory. ### Docker See also the Python-specific section in [charlax/python-education](https://github.com/charlax/python-education#deployment). - [Best Practices Around Production Ready Web Apps with Docker Compose](https://nickjanetakis.com/blog/best-practices-around-production-ready-web-apps-with-docker-compose) - Avoiding 2 Compose Files for Dev and Prod with an Override File - Reducing Service Duplication with Aliases and Anchors - Defining your `HEALTHCHECK` in Docker Compose not your Dockerfile - Making the most of environment variables - Using Multi-stage builds to optimize image size - Running your container as a non-root user - [Docker Best Practices for Python Developers](https://testdriven.io/blog/docker-best-practices/) - Use multi-stage builds - Pay close attention to the order of your Dockerfile commands to leverage layer caching - Smaller Docker images are more modular and secure (watch out for Alpine though) - Minimize the number of layers (`RUN`, `COPY`, `ADD`) - Use unprivileged containers - Prefer `COPY` over `ADD` - Cache python packages to the docker host - Prefer array over string syntax - Understand the difference between `ENTRYPOINT` and `CMD` - Include a `HEALTHCHECK` instruction - Whenever possible, avoid using the `latest` tag. - Don't store secrets in images - Use a `.dockerignore` file (include `**/.git`, etc.) - Lint and Scan Your Dockerfiles and Images (e.g. with `hadolint`) - Log to stdout or stderr - [Docker Containers Security](https://tbhaxor.com/docker-containers-security/) ### Documentation - [Documentation-Driven Development](https://gist.github.com/zsup/9434452) - [Writing automated tests for your documentation](https://krausefx.com/blog/writing-automated-tests-for-your-documentation): this should be required, IMO. Testing code samples in your documentation ensures they never get outdated. - 🏙 [Documentation is king](https://speakerdeck.com/kennethreitz/documentation-is-king), Kenneth Reitz - [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) - [Architectural Decision Records (ADR)](https://adr.github.io/): a way to document architecture decision. - [Documenting Architecture Decisions](https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions) - [joelparkerhenderson/architecture-decision-record](https://github.com/joelparkerhenderson/architecture-decision-record): examples and templates for ADR. - And a CLI tool: [npryce/adr-tools](https://github.com/npryce/adr-tools) - [The documentation system](https://documentation.divio.com/) - [Checklist for checklists](https://www1.nyc.gov/assets/doh/downloads/pdf/em/gawande_checklist.pdf) - [Best practices for writing code comments](https://stackoverflow.blog/2021/12/23/best-practices-for-writing-code-comments/) - [Always be quitting](https://jmmv.dev/2021/04/always-be-quitting.html) - Document your knowledge - Train your replacement - Delegate - By being disposable, you free yourself to work on high-impact projects. - [Write documentation first. Then build.](https://reproof.app/blog/document-first-then-build) - [Diátaxis](https://diataxis.fr/): a systematic approach to technical documentation authoring - There are four modes: tutorials, how-to guides, technical reference and explanation - The docs goes into a lot of details about each model. - [ARCHITECTURE.md](https://matklad.github.io/2021/02/06/ARCHITECTURE.md.html) - [Two open source projects with great documentation](https://johnjago.com/great-docs/) (esbuild and redis) - [Rules for Writing Software Tutorials](https://refactoringenglish.com/chapters/rules-for-software-tutorials/) > The palest ink is more reliable than the most powerful memory. > -- Chinese proverb ### Dotfiles - 🧰 [Awesome dotfiles](https://github.com/webpro/awesome-dotfiles): lots of great dotfiles. - [My dotfiles](https://github.com/charlax/dotfiles) Articles - [Setting Up a Mac Dev Machine From Zero to Hero With Dotfiles](http://code.tutsplus.com/tutorials/setting-up-a-mac-dev-machine-from-zero-to-hero-with-dotfiles--net-35449) ### Editors & IDE - [Sublime Text essential plugins and resources](https://github.com/dreikanter/sublime-bookmarks) - Bram Moolenaar (Vim author), [Seven habits of effective text editing](http://www.moolenaar.net/habits.html) ([presentation](http://www.moolenaar.net/habits_2007.pdf)). This is about Vim but it contains good lessons about why investing time in learning how to be productive with your text editors pays off. - [VScode](https://code.visualstudio.com/) is one of the most popular text editors as of writing. - [Visual Studio Code Can Do That?](https://www.smashingmagazine.com/2018/01/visual-studio-code/), Smashing Magazine. - [Coding with Character](https://realdougwilson.com/writing/coding-with-character) #### Vim - 🧰 [vim-awesome](http://vimawesome.com/) - 🎞 [Vimcasts](http://vimcasts.org/) - ⭐️ [Is Vim Really Not For You? A Beginner Guide](https://thevaluable.dev/vim-beginner/) - The first part of a series of 6 articles with lots of detailed and practical tips for using Vim efficiently. - [A Vim Guide for Advanced Users](https://thevaluable.dev/vim-advanced/): more advanced shortcuts and commands - 📖 [Learning the vi and Vim Editors](https://www.oreilly.com/library/view/learning-the-vi/9780596529833/) - 📖 [Practical Vim](https://pragprog.com/titles/dnvim2/practical-vim-second-edition/), Drew Neil - [Learn Vimscript the Hard Way](https://learnvimscriptthehardway.stevelosh.com/) - [VimGolf](https://www.vimgolf.com/): nice challenges to learn Vim - [Vim anti-patterns](https://blog.sanctum.geek.nz/vim-anti-patterns/) - [Learn Vim For the Last Time: A Tutorial and Primer](https://danielmiessler.com/study/vim/) - [Vim Cheat Sheet & Quick Reference](https://quickref.me/vim) - [History and effective use of Vim](https://begriffs.com/posts/2019-07-19-history-use-vim.html) - [Moving Blazingly Fast With The Core Vim Motions](https://www.barbarianmeetscoding.com/boost-your-coding-fu-with-vscode-and-vim/moving-blazingly-fast-with-the-core-vim-motions/) - [micahkepe/vimtutor-sequel: Vimtutor Sequel - Advanced Vim Tutor Lessons](https://github.com/micahkepe/vimtutor-sequel) - [Vim Racer - An Online Game for VIM Navigation](https://vim-racer.com/) Feel free to check my [vim configuration](https://github.com/charlax/dotfiles/tree/master/vim) and my [vim cheatsheet](https://github.com/charlax/dotfiles/tree/master/vim). Other editors: - [Use GNU Emacs](https://www2.lib.uchicago.edu/keith/emacs/) ### Email - [Email explained from first principles](https://explained-from-first-principles.com/email/#json-meta-application-protocol) - 🏙 [Transactional Email Best Practices](https://speakerdeck.com/wildbit/transactional-email-best-practices) ### Engineering management Checkout my [list of management resources](https://github.com/charlax/engineering-management). ### Exercises The best way to learn is to learn by doing. - [Reinvent the Wheel](https://endler.dev/2025/reinvent-the-wheel/) - [build-your-own-x](https://github.com/codecrafters-io/build-your-own-x): compilation of well-written, step-by-step guides for re-creating our favorite technologies from scratch - Richard Feynman: "what I cannot create, I do not understand" - [The elevator programming game](http://play.elevatorsaga.com/) - [Challenging projects every programmer should try](https://austinhenley.com/blog/challengingprojects.html), Austin Z. Henley - [Challenging projects every programmer should try](http://web.eecs.utk.edu/~azh/blog/challengingprojects.html): text editor, space invaders, compiler (Tiny Basic), mini OS, spreadsheet, video game console emulator. - [More challenging projects every programmer should try](https://austinhenley.com/blog/morechallengingprojects.html): ray tracer, key-value store web API, web browser, stock trading bot. - [Let’s Build a Regex Engine](https://kean.blog/post/lets-build-regex) - [Write a time-series database engine from scratch](https://nakabonne.dev/posts/write-tsdb-from-scratch/) - [7 GUIs to build to learn fundamental UI programming skills](https://eugenkiss.github.io/7guis/tasks/) - [A list of programming playgrounds](https://jvns.ca/blog/2023/04/17/a-list-of-programming-playgrounds/), Julia Evans - [Write more "useless" software](https://ntietz.com/blog/write-more-useless-software/) - [SadServers - Linux & DevOps Troubleshooting Interviews](https://sadservers.com/) - [Writing Toy Software Is A Joy](https://blog.jsbarretto.com/post/software-is-joy) - [Fly.io Distributed Systems Challenge "Gossip Glomers"](https://fly.io/dist-sys/) Practice: - [CodinGame](https://www.codingame.com/start) - [Codewars](https://www.codewars.com/) - [Exercism](https://exercism.org/) ### Experimentation - [8 annoying A/B testing mistakes every engineer should know](https://posthog.com/blog/ab-testing-mistakes) ### Functional programming (FP) - [Goodbye, Object Oriented Programming](https://medium.com/@cscalfani/goodbye-object-oriented-programming-a59cda4c0e53#.39ax09e4k) - [Functional Programming & Haskell](https://www.youtube.com/watch?v=LnX3B9oaKzw) 🎞: some good reasons to learn FP! - [Functional Programming Fundamentals](https://www.matthewgerstman.com/functional-programming-fundamentals/): short introduction to FP and its advantages. - [OO vs FP](https://blog.cleancoder.com/uncle-bob/2014/11/24/FPvsOO.html), Robert C. Martin, The Clean Code Blog. A pretty interesting take on the differences between OOP and FP from an expert in OOP. - OO is not about state. Objects are bags of functions, not bags of data. - Functional Programs, like OO Programs, are composed of functions that operate on data. - FP imposes discipline upon assignment. - OO imposes discipline on function pointers. - The principles of software design still apply, regardless of your programming style. The fact that you’ve decided to use a language that doesn’t have an assignment operator does not mean that you can ignore the Single Responsibility Principle. - [Parse, don’t validate](https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/) - Use a data structure that makes illegal states unrepresentable - Push the burden of proof upward as far as possible, but no further - Let your datatypes inform your code, don’t let your code control your datatypes - Don’t be afraid to parse data in multiple passes - Avoid denormalized representations of data, especially if it’s mutable - Use abstract datatypes to make validators “look like” parsers - 🏙 [Functional Programming](https://speakerdeck.com/igstan/functional-programming) - [Monads in 15 minutes](https://nikgrozev.com/2013/12/10/monads-in-15-minutes/) - [hemanth/functional-programming-jargon](https://github.com/hemanth/functional-programming-jargon): jargon from the functional programming world in simple terms - [The definitive guide to learning functional programming](https://forum.exercism.org/t/the-definitive-guide-to-learning-functional-programming/3254), Exercism ### Games development - [Introduction · Joys of Small Game Development](https://abagames.github.io/joys-of-small-game-development-en/) ### Graphics - [Text Rendering Hates You](https://faultlore.com/blah/text-hates-you/) - [AAA - Analytical Anti-Aliasing](https://blog.frost.kiwi/analytical-anti-aliasing/) ### Hardware - [NandGame](https://nandgame.com/): build a computer from scratch. - [What Every Programmer Should Know About SSDs](http://databasearchitects.blogspot.com/2021/06/what-every-programmer-should-know-about.html) - [How To Make A CPU - A Simple Picture Based Explanation](https://blog.robertelder.org/how-to-make-a-cpu/) ### HTTP - [Choosing an HTTP Status Code — Stop Making It Hard](https://www.codetinkerer.com/2015/12/04/choosing-an-http-status-code.html) - [HTTPWTF](https://httptoolkit.tech/blog/http-wtf/) - [10 Surprising Things You Didn't Know About HTTP](https://webdevguild.com/blog/10-things-http/) - [The HTTP crash course nobody asked for](https://fasterthanli.me/articles/the-http-crash-course-nobody-asked-for) ### Humor - [The Jeff Dean Facts](https://www.informatika.bg/jeffdean) - Compilers don't warn Jeff Dean. Jeff Dean warns compilers. - Unsatisfied with constant time, Jeff Dean created the world's first `O(1/N)` algorithm. - Jeff Dean mines bitcoins. In his head. - [The Daily WTF: Curious Perversions in Information Technology](https://thedailywtf.com/) ### Incident response (oncall, alerting, outages, firefighting, postmortem) Also see this section on my [list of management resources, "Incident response"](https://github.com/charlax/engineering-management/). Also see the Debugging section in this doc. - [Incident Response at Heroku](https://blog.heroku.com/archives/2014/5/9/incident-response-at-heroku) - Described the Incident Commander role, inspired by natural disaster incident response. - Also in presentation: [Incident Response Patterns: What we have learned at PagerDuty - Speaker Deck](https://speakerdeck.com/arupchak/incident-response-patterns-what-we-have-learned-at-pagerduty) - The Google SRE book's [chapter about oncall](https://landing.google.com/sre/workbook/chapters/on-call/) - [Writing Runbook Documentation When You’re An SRE](https://www.transposit.com/blog/2020.01.30-writing-runbook-documentation-when-youre-an-sre/) - Playbooks “reduce stress, the mean time to repair (MTTR), and the risk of human error.” - Using a template can be beneficial because starting from a blank document is incredibly hard. - The Curse of Knowledge is a cognitive bias that occurs when someone is communicating with others and unknowingly assumes the level of knowledge of the people they are communicating with. - Make your content easy to glance over. - If a script is longer than a single line, treat it like code, and check it into a repository to be source control and potentially tested. - [Incident Review and Postmortem Best Practices](https://newsletter.pragmaticengineer.com/p/incident-review-best-practices?s=r), Gergely Orosz - [Computer Security Incident Handling Guide](https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-61r2.pdf), NIST - [Incident Management Resources](https://resources.sei.cmu.edu/library/asset-view.cfm?assetID=505044), Carnegie Mellon University - [Sterile flight deck rule](https://en.wikipedia.org/wiki/Sterile_flight_deck_rule), Wikipedia - [Shamir Secret Sharing It’s 3am.](https://max.levch.in/post/724289457144070144/shamir-secret-sharing-its-3am-paul-the-head-of) - [Site Reliability Engineering and the Art of Improvisation](https://thenewstack.io/site-reliability-engineering-and-the-art-of-improvisation/) has lots of good training ideas - Walkthroughs of observability toolsets - Decision requirements table building - Team knowledge elicitation - Asking the question, “Why do we have on-call?” - Spin the Wheel of Expertise! - [Severity Levels](https://response.pagerduty.com/before/severity_levels/), PagerDuty Alerting: - [My Philosophy On Alerting](https://linuxczar.net/sysadmin/philosophy-on-alerting/) - Pages should be urgent, important, actionable, and real. - Err on the side of removing noisy alerts – over-monitoring is a harder problem to solve than under-monitoring. - Symptoms are a better way to capture more problems more comprehensively and robustly with less effort. - Include cause-based information in symptom-based pages or on dashboards, but avoid alerting directly on causes. - The further up your serving stack you go, the more distinct problems you catch in a single rule. But don’t go so far you can’t sufficiently distinguish what’s going on. - If you want a quiet oncall rotation, it’s imperative to have a system for dealing with things that need timely response, but are not imminently critical. - This classical article has now become a [chapter](https://sre.google/sre-book/monitoring-distributed-systems/) in Google's SRE book. - 🏙 [The Paradox of Alerts](https://speakerdeck.com/charity/the-paradox-of-alerts): why deleting 90% of your paging alerts can make your systems better, and how to craft an on-call rotation that engineers are happy to join. #### Postmortem - A great example of a [postmortem from Gitlab (01/31/2017)](https://about.gitlab.com/2017/02/01/gitlab-dot-com-database-incident/) for an outage during which an engineer's action caused the irremediable loss of 6 hours of data. - [Blameless PostMortems and a Just Culture](https://codeascraft.com/2012/05/22/blameless-postmortems/) - [A list of postmortems on Github](https://github.com/danluu/post-mortems) - Google's SRE book, [Postmortem chapter](https://landing.google.com/sre/workbook/chapters/postmortem-culture/) is excellent and includes many examples. - [Human error models and management](https://app.box.com/s/7z35l09amvr1vwxouh2s) - High reliability organisations — which have less than their fair share of accidents — recognise that human variability is a force to harness in averting errors, but they work hard to focus that variability and are constantly preoccupied with the possibility of failure > "Let’s plan for a future where we’re all as stupid as we are today." > > – Dan Milstein Example outline for a postmortem: - Executive Summary - Impact - Root cause - Impact - Number of impacted users - Lost revenue - Duration - Team impact - Timeline - Detection - Resolution - Root cause analysis - E.g. with 5 whys method - Lessons learned - Things that went well - Things that went poorly - Action items (include direct links to task tracking tool) - Tasks to improve prevention (including training) - Tasks to improve detection (including monitoring and alerting) - Tasks to improve mitigation (including emergency response) ### Internet - [How Does the Internet Work?](https://web.stanford.edu/class/msande91si/www-spr04/readings/week1/InternetWhitepaper.htm) - [How the web works](https://github.com/vasanthk/how-web-works) - [Advice to young web developers](https://beesbuzz.biz/blog/2934-Advice-to-young-web-developers) ### Interviewing Note: this is about you as an interviewee, **not** as an interviewer. To check out my list of resources for interviewers, go to my [engineering-management repository](https://github.com/charlax/engineering-management#hiring-interviews). - [System design interview for IT company](https://github.com/checkcheckzz/system-design-interview) - [Technical Interview Megarepo](https://github.com/jdsutton/Technical-Interview-Megarepo): study materials for SE/CS technical interviews - [How to Win the Coding Interview](https://medium.com/on-writing-code/how-to-win-the-coding-interview-71ae7102d685) - [I spent 3 months applying to jobs after a coding bootcamp. Here’s what I learned.](https://medium.freecodecamp.com/5-key-learnings-from-the-post-bootcamp-job-search-9a07468d2331#.uq7vbbjfx) - [Top 10 algorithms in Interview Questions](http://www.geeksforgeeks.org/top-10-algorithms-in-interview-questions/) - [Interactive Python coding interview challenges](https://github.com/donnemartin/interactive-coding-challenges) - [Tech Interview Handbook](https://www.techinterviewhandbook.org/) - [A complete computer science study plan to become a software engineer](https://github.com/jwasham/coding-interview-university) - [Interview advice that got me offers from Google, Microsoft, and Stripe](https://www.zainrizvi.io/blog/the-interviewing-advice-no-one-shares/) - [A framework for grading your performance on programming interview problems](https://docs.google.com/spreadsheets/d/1gy9cmPwNhZvola7kqnfY3DElk7PYrz2ARpaCODTp8Go/htmlview?pru=AAABfLx5bfQ*dDBm6wJC2BsJGIrDvJfFQw) - [Preparing for the Systems Design and Coding Interview](https://blog.pragmaticengineer.com/preparing-for-the-systems-design-and-coding-interviews/), Gergely Orosz - [What I Learned from Doing 60+ Technical Interviews in 30 Days](https://meekg33k.dev/what-i-learned-from-doing-60-technical-interviews-in-30-days) - [System Design Interview Guide for Senior Engineers](https://interviewing.io/guides/system-design-interview), interviewing.io Questions you should ask: - [Questions to ask your interviewer](https://rkoutnik.com/articles/Questions-to-ask-your-interviewer.html) - [Questions to ask the company during your interview](https://github.com/viraptor/reverse-interview) - [Interviewing the Interviewer: Questions to Uncover a Company’s True Culture](https://praachi.work/blog/questions-to-ask-in-a-job-interview) - [Twipped/InterviewThis](https://github.com/Twipped/InterviewThis): questions to ask prospective employers - [tBaxter/questions-for-employers: A big collection of useful questions to ask potential employers.](https://github.com/tBaxter/questions-for-employers) Résumé: - [The Red Flags on Your Resume](https://danicfilip.com/2020/red-flags-on-your-resume/?ck_subscriber_id=887770473) - [What we look for in a resume](https://huyenchip.com/2023/01/24/what-we-look-for-in-a-candidate.html) - We look for demonstrated expertise, not keywords - We look for people who get things done - We look for unique perspectives - We care about impact, not meaningless metrics - [Why you shouldn’t list certifications on LinkedIn](https://interviewing.io/blog/why-you-shouldnt-list-certifications-on-linkedIn) See also the exercises section in this document. ### Kubernetes - [OWASP/www-project-kubernetes-top-ten](https://github.com/OWASP/www-project-kubernetes-top-ten) - [Kubernetes Tutorial for Beginners: Basic Concepts](https://spacelift.io/blog/kubernetes-tutorial) ### Large Language Model (LLM) - [What Is ChatGPT Doing… and Why Does It Work?](https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/), Stephen Wolfram - [Embeddings: What they are and why they matter](https://simonwillison.net/2023/Oct/23/embeddings/) ### Learning & memorizing Learn how to learn! - [How I Rewired My Brain to Become Fluent in Math](https://nautil.us/how-i-rewired-my-brain-to-become-fluent-in-math-235085): subtitled _the building blocks of understanding are memorization and repetition_. - [One Sure-Fire Way to Improve Your Coding](https://changelog.com/posts/one-sure-fire-way-to-improve-your-coding): reading code! - [Tips for learning programming](http://blog.hiphipjorge.com/tips-for-learning-programming/) - [You can increase your intelligence: 5 ways to maximize your cognitive potential](https://blogs.scientificamerican.com/guest-blog/you-can-increase-your-intelligence-5-ways-to-maximize-your-cognitive-potential/): forgive the clickbait title, it’s actually a good article. - [How to ask good questions](https://jvns.ca/blog/good-questions/), Julia Evans. - [Stop Learning Frameworks](https://sizovs.net/2018/12/17/stop-learning-frameworks/) - [Learning How to Learn](https://www.coursera.org/learn/learning-how-to-learn): powerful mental tools to help you master tough subjects - [Why books don’t work](https://andymatuschak.org/books/), Andy Matuschak. - "As a medium, books are surprisingly bad at conveying knowledge, and readers mostly don’t realize it." - "In learning sciences, we call this model “transmissionism.” It’s the notion that knowledge can be directly transmitted from teacher to student, like transcribing text from one page onto another. If only!" - "By re-testing yourself on material you’ve learned over expanding intervals, you can cheaply and reliably commit huge volumes of information to long-term memory." - [Strategies, Tips, and Tricks for Anki](https://senrigan.io/blog/everything-i-know-strategies-tips-and-tricks-for-spaced-repetition-anki/): those advices work for any tool actually - Add images. Our brains are wired visually, so this helps retention. - Don't add things you don't understand. - Don't add cards memorizing entire lists. - Write it out. For wrong answers, I'll write it on paper. The act of writing is meditative. I really enjoy this. - Keep on asking yourself why? why does this work? why does it work this way? Force yourself to understand the root of a topic. - Cornell Method: when reading a topic, write out questions on the margins to quiz yourself. - Pretend you have to teach it - Use mnemonics phrases like PEMDAS for lists and other hard-to-remember topics. - Delete cards that don't make sense or you don't want to remember anymore. - [Effective learning: Twenty rules of formulating knowledge](https://www.supermemo.com/en/archives1990-2015/articles/20rules) - Build upon the basics - Stick to the minimum information principle: the material you learn must be formulated in as simple way as it is - Cloze deletion is easy and effective: Kaleida's mission was to create a ... It finally produced one, called Script X. But it took three years - Graphic deletion is as good as cloze deletion - Avoid sets - Avoid enumerations - Combat interference - even the simplest items can be completely intractable if they are similar to other items. Use examples, context cues, vivid illustrations, refer to emotions, and to your personal life - Personalize and provide examples - personalization might be the most effective way of building upon other memories. Your personal life is a gold mine of facts and events to refer to. As long as you build a collection for yourself, use personalization richly to build upon well established memories - Provide sources - sources help you manage the learning process, updating your knowledge, judging its reliability, or importance - Prioritize - effective learning is all about prioritizing. - [How to Remember Anything You Really Want to Remember, Backed by Science](https://www.inc.com/jeff-haden/how-to-remember-anything-you-really-want-to-remember-backed-by-science.html) - Quiz yourself - Summarize and share with someone else. - Connect what you just learned to experiences you previously had. - [How To Remember Anything Forever-ish](https://ncase.me/remember/): a comic about learning - [Get better at programming by learning how things work](https://jvns.ca/blog/learn-how-things-work/) - [How to teach yourself hard things](https://jvns.ca/blog/2018/09/01/learning-skills-you-can-practice/) - [Building Your Own Personal Learning Curriculum](https://www.smashingmagazine.com/2021/02/building-personal-learning-curriculum/) - [Always do Extra](http://www.bennorthrop.com/Essays/2021/always-do-extra.php) - Extra is finishing those two screens, but then researching a new library for form validation that might reduce the boilerplate code. - Extra must be balanced against Normal Work. - Extra must be aligned with your Normal Work. - [Against 3X Speed](https://perell.com/essay/against-3x-speed/) - Lectures are most effective when they’re only a component of the classroom experience - Learning is about spaced repetition, not binge-reading books - [The Problems with Deliberate Practice](https://commoncog.com/blog/the-problems-with-deliberate-practice/) - [Why Tacit Knowledge is More Important Than Deliberate Practice](https://commoncog.com/tacit-knowledge-is-a-real-thing/) - [In Praise of Memorization](http://www.pearlleff.com/in-praise-of-memorization) - You can't reason accurately without knowledge - Memorizing organized your knowledge - It stays with you - [Celebrate tiny learning milestones](https://jvns.ca/blog/2022/03/13/celebrate-tiny-learning-milestones/), Julia Evans. - Keep a brag document ([Get your work recognized: write a brag document](https://jvns.ca/blog/brag-documents/)) - You can do a lot "by accident" - Fixing a bug can be milestone - [Why writing by hand is still the best way to retain information](https://stackoverflow.blog/2022/11/23/why-writing-by-hand-is-still-the-best-way-to-retain-information/), StackOverflow - 🎞 [Making Badass Developers - Kathy Sierra (Serious Pony) keynote](https://www.youtube.com/watch?v=FKTxC9pl-WM&t=2s&ab_channel=O%27Reilly) - [How to study](https://cse.buffalo.edu/~rapaport/howtostudy.html) (with lots of cartoons from Calvin & Hobbes!) - Manage your time - Take notes in class & rewrite them at home - Study hard subjects first & study in a quiet place - Read actively & slowly, before & after class - Do your homework - Study for exams - Take Exams - Do research & write essays - Do I really have to do all this? - Are there other websites that give study hints? - [10 Things Software Developers Should Learn about Learning](https://cacm.acm.org/research/10-things-software-developers-should-learn-about-learning/) - 🏙 [Things I Learned the Hard Way](https://speakerdeck.com/bcantrill/things-i-learned-the-hard-way), Bryan Cantrill About flashcards: - [Augmenting Long-term Memory](http://augmentingcognition.com/ltm.html) - [How to write good prompts: using spaced repetition to create understanding](https://andymatuschak.org/prompts/) - also includes lots of insightful research papers. - [Effective learning: Twenty rules of formulating knowledge](https://www.supermemo.com/en/blog/twenty-rules-of-formulating-knowledge) - [Rules for Designing Precise Anki Cards](https://controlaltbackspace.org/precise/) - Fernando Borretti, [Effective Spaced Repetition](https://borretti.me/article/effective-spaced-repetition) - [Anki-fy Your Life](https://abouttolearn.substack.com/p/anki-fy-your-life) gets into why it makes sense to invest in your memory. About Zettelkasten and PKM (personal knowledge management): see [Personal knowledge management](#personal-knowledge-management-pkm) Richard Feynman's Learning Strategy: 1. Step 1: Continually ask "Why?” 2. Step 2: When you learn something, learn it to where you can explain it to a child. 3. Step 3: Instead of arbitrarily memorizing things, look for the explanation that makes it obvious. > Most people overestimate what they can do in 1 year and underestimate what they can do in a decade. > – Bill Gates > Frankly, though, I think most people can learn a lot more than they think they can. They sell themselves short without trying. > One bit of advice: it is important to view knowledge as sort of a semantic tree — make sure you understand the fundamental principles, ie the trunk and big branches, before you get into the details/leaves or there is nothing for them to hang on to. > — Elon Musk > "Experience is something you don't get until just after you need it." > ― Steven Wright > Tell me and I forget. Teach me and I remember. Involve me and I learn. > – Benjamin Franklin > Education is the kindling of a flame, not the filling of a vessel. > – Socrates > That which we persist in doing becomes easier for us to do; not that the nature of the thing itself is changed, but that our power to do is increased. > – Ralph Waldo Emerson > A wise man can learn more from a foolish question than a fool can learn from a wise answer. > – Bruce Lee > A lecture has been well described as the process whereby the notes of the teacher become the notes of the student without passing through the mind of either. > — Mortimer Adler > Fools learn from experience. I prefer to learn from the experience of others. > — Bismark ### Licenses (legal) - [Software Licenses in Plain English](https://tldrlegal.com/) ### Linux (system management) - [Welcome to Linux command line for you and me!](https://lym.readthedocs.io/en/latest/index.html) - [Linux Performance](https://www.brendangregg.com/linuxperf.html), Brendan Gregg - [Linux disk I/O diagram](https://zenodo.org/records/15234151) ### Low-code/no-code - [How Levels.fyi scaled to millions of users with Google Sheets as a backend](https://www.levels.fyi/blog/scaling-to-millions-with-google-sheets.html) ### Low-level, assembly - [Back to Basics](https://www.joelonsoftware.com/2001/12/11/back-to-basics/), Joel Spolsky. Explains why learning low level programming is important. - I think that some of the biggest mistakes people make even at the highest architectural levels come from having a weak or broken understanding of a few simple things at the very lowest levels. - [What's in a Linux executable?](https://fasterthanli.me/series/making-our-own-executable-packer/part-1) - 📖 [The Elements of Computing Systems](https://www.nand2tetris.org/book): building a modern computer from first principles (nand2tetris). - [Old pattern powering modern tech](https://softwarebits.substack.com/p/old-pattern-powering-modern-tech?s=r) - [Demystifying bitwise operations, a gentle C tutorial](https://www.andreinc.net/2023/02/01/demystifying-bitwise-ops) - [Understanding the Power of Bitwise Operators. No math needed](https://www.deusinmachina.net/p/understanding-the-power-of-bitwise) - [Memory Allocation](https://samwho.dev/memory-allocation/) (an interactive article) - [Why does 0.1 + 0.2 = 0.30000000000000004?](https://jvns.ca/blog/2023/02/08/why-does-0-1-plus-0-2-equal-0-30000000000000004/), Julia Evans (about floating point) - [Putting the "You" in CPU](https://cpu.land/the-basics) - [x86-64 Assembly Language Programming with Ubuntu](http://www.egr.unlv.edu/~ed/assembly64.pdf) - [XOR](https://www.chiark.greenend.org.uk/~sgtatham/quasiblog/xor/) ### Machine learning/AI - [Transformers from Scratch](https://e2eml.school/transformers.html) - [A Gentle Introduction to Graph Neural Networks](https://distill.pub/2021/gnn-intro/) - [The Math Behind Artificial Intelligence](https://www.freecodecamp.org/news/the-math-behind-artificial-intelligence-book) - Tiago Monteiro. Covers linear algebra, calculus, probability, and optimization theory in the context of AI and machine learning. ### Math - 🏙 [Statistics for Hackers](https://speakerdeck.com/jakevdp/statistics-for-hackers) ### Marketing - [goabstract/Marketing-for-Engineers](https://github.com/goabstract/Marketing-for-Engineers) ### Network - [Everything you need to know about DNS](https://www.nslookup.io/learning/) - [Computer Networking Fundamentals](https://iximiuz.com/en/series/computer-networking-fundamentals/) ### Observability (monitoring, logging, exception handling) _See also: [Site Reliability Engineering (SRE)](#site-reliability-engineering-sre)_ #### Logging - [Do not log](https://sobolevn.me/2020/03/do-not-log) dwells on some logging antipatterns. - Logging does not make much sense in monitoring and error tracking. Use better tools instead: error and business monitorings with alerts, versioning, event sourcing. - Logging adds significant complexity to your architecture. And it requires more testing. Use architecture patterns that will make logging an explicit part of your contracts - Logging is a whole infrastructure subsystem on its own. And quite a complex one. You will have to maintain it or to outsource this job to existing logging services - [Lies My Parents Told Me (About Logs)](https://www.honeycomb.io/blog/lies-my-parents-told-me-about-logs/) - Logs are cheap - I can run it better myself - Leveled logging is a great way to separate information - Logs are basically the same as events - A standard logging format is good enough - [Logging - OWASP Cheat Sheet Series](https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html) - [The Audit Log Wall of Shame](https://audit-logs.tax/): list of vendors that don’t prioritize high-quality, widely-available audit logs for security and operations teams. - [Guide on Structured Logs](https://signoz.io/blog/structured-logs/) - [What an error log level should mean](https://utcc.utoronto.ca/~cks/space/blog/programming/ErrorsShouldRequireFixing) - [Logging Sucks - Your Logs Are Lying To You](https://loggingsucks.com/) #### Error/exception handling - [Error handling antipatterns](./antipatterns/error-handling-antipatterns.md) in this repo. - [Writing Helpful Error Messages](https://developers.google.com/tech-writing/error-messages), Google Developers' course on Technical Writing - Explain the problem - Explain the solution - Write clearly - [Errors, Errors Everywhere: How We Centralized and Structured Error Handling](https://olivernguyen.io/w/namespace.error/) (for Go, but useful for any languages) - For inspiration: [Handle Errors - Graph API](https://developers.facebook.com/docs/graph-api/guides/error-handling#receiving-errorcodes) - [The Error Model](https://joeduffyblog.com/2016/02/07/the-error-model/), Joe Duffy, 2016. #### Metrics - [Meaningful availability](https://blog.acolyer.org/2020/02/26/meaningful-availability/) - A good availability metric should be meaningful, proportional, and actionable. By "meaningful" we mean that it should capture what users experience. By "proportional" we mean that a change in the metric should be proportional to the change in user-perceived availability. By "actionable" we mean that the metric should give system owners insight into why availability for a period was low. This paper shows that none of the commonly used metrics satisfy these requirements… - 📃 [Meaningful Availability](https://www.usenix.org/conference/nsdi20/presentation/hauer) paper. - This paper presents and evaluates a novel availability metric: windowed user-uptime #### Monitoring - Google, [Site Reliability Engineering, Monitoring Distributed Systems](https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/) - [Alerting on SLOs](https://sre.google/workbook/alerting-on-slos/) - PagerDuty, [Monitoring Business Metrics and Refining Outage Response](https://www.pagerduty.com/blog/monitoring-business-metrics/) - 🧰 [crazy-canux/awesome-monitoring](https://github.com/crazy-canux/awesome-monitoring): monitoring tools for operations. - [Monitoring in the time of Cloud Native](https://medium.com/@copyconstruct/monitoring-in-the-time-of-cloud-native-c87c7a5bfa3e) - [How to Monitor the SRE Golden Signals](https://medium.com/faun/how-to-monitor-the-sre-golden-signals-1391cadc7524) - From the Google SRE book: Latency, Traffic, Errors, and Saturation - USE Method (from Brendan Gregg): Utilization, Saturation, and Errors - RED Method (from Tom Wilkie): Rate, Errors, and Duration - [Simple Anomaly Detection Using Plain SQL](https://hakibenita.com/sql-anomaly-detection) - [How percentile approximation works (and why it's more useful than averages)](https://www.timescale.com/blog/how-percentile-approximation-works-and-why-its-more-useful-than-averages/) - [Implementing health checks](https://aws.amazon.com/builders-library/implementing-health-checks/) - [IETF RFC Health Check Response Format for HTTP APIs](https://datatracker.ietf.org/doc/html/draft-inadarei-api-health-check-06) ### Open source - [Non-code contributions are the secret to open source success](https://github.com/readme/featured/open-source-non-code-contributions) ### Operating system (OS) - 📖 [The Linux Programming Interface: A Linux and UNIX System Programming Handbook](http://www.amazon.com/The-Linux-Programming-Interface-Handbook/dp/1593272200): already mentioned above. - 📖 [Modern Operating Systems](https://www.amazon.com/dp/013359162X/), Andrew Tanenbaum, Herbert Bos (not read) - 📖 [Operating Systems: Three Easy Pieces](https://pages.cs.wisc.edu/~remzi/OSTEP/) (free book, not read) - 📖 [Linux Kernel Development](https://www.amazon.com/Linux-Kernel-Development-Robert-Love/dp/0672329468), Robert Love. A very complete introduction to developing within the Linux Kernel. - [The 10 Operating System Concepts Software Developers Need to Remember](https://jameskle.com/writes/operating-systems) - Play with xv6 on [MIT 6.828](https://pdos.csail.mit.edu/6.828/2016/schedule.html) - [macOS Internals](https://gist.github.com/kconner/cff08fe3e0bb857ea33b47d965b3e19f) ### Over-engineering - [10 modern software over-engineering mistakes](https://medium.com/@rdsubhas/10-modern-software-engineering-mistakes-bc67fbef4fc8#.da6dvzyne) - [A good example of over-engineering: the Juicero press](https://blog.bolt.io/heres-why-juicero-s-press-is-so-expensive-6add74594e50) (April 2017) - [You Are Not Google](https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb): the UNPHAT method to avoid cargo cult. - Don’t even start considering solutions until you Understand the problem. Your goal should be to “solve” the problem mostly within the problem domain, not the solution domain. - eNumerate multiple candidate solutions. Don’t just start prodding at your favorite! - [Overthinking](https://kerkour.com/overthinking) - 1st poison: education. - 2nd poison: marketing. - 3rd poison: ego - Solution: Stop trying to connect all the dots ahead of time. Embrace uncertainty and start doing. - [Don’t Let Architecture Astronauts Scare You](https://www.joelonsoftware.com/2001/04/21/dont-let-architecture-astronauts-scare-you/), Joel - Sometimes smart thinkers just don’t know when to stop, and they create these absurd, all-encompassing, high-level pictures of the universe that are all good and fine, but don’t actually mean anything at all. - Your typical architecture astronaut will take a fact like “Napster is a peer-to-peer service for downloading music” and ignore everything but the architecture, thinking it’s interesting because it’s peer to peer, completely missing the point that it’s interesting because you can type the name of a song and listen to it right away. > “A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over, beginning with a working simple system.” — John Gall, General systemantics, an essay on how systems work, and especially how they fail..., 1975 (this quote is sometime referred as "Galls' law") > "Software engineering is what happens to programming when you add time and other programmers." — Rob Pike, [Go at Google: Language Design in the Service of Software Engineering](https://talks.golang.org/2012/splash.article) > You can’t connect the dots looking forward; you can only connect them looking backwards. So you have to trust that the dots will somehow connect in your future. You have to trust in something — your gut, destiny, life, karma, whatever. This approach has never let me down, and it has made all the difference in my life. — Steve Jobs ### Performance - [Numbers Everyone Should Know](https://everythingisdata.wordpress.com/2009/10/17/numbers-everyone-should-know/) - [Latency numbers every programmer should know](https://gist.github.com/hellerbarde/2843375) - [Rob Pike's 5 Rules of Programming](http://users.ece.utexas.edu/~adnan/pike.html) - You can't tell where a program is going to spend its time. - Measure - Fancy algorithms are slow when n is small, and n is usually small. - Fancy algorithms are buggier than simple ones - Data dominates. - [Performance comparison: counting words in Python, Go, C++, C, AWK, Forth, and Rust](https://benhoyt.com/writings/count-words/): a great way to learn about measuring performance. - [The Mathematical Hacker](https://www.evanmiller.org/mathematical-hacker.html) - [Four Kinds of Optimisation](https://tratt.net/laurie/blog/2023/four_kinds_of_optimisation.html) ### Personal knowledge management (PKM) - [Zettelkasten Method](https://zettelkasten.de/posts/overview/) - [How to build a second brain as a software developer](https://aseemthakar.com/how-to-build-a-second-brain-as-a-software-developer/) - [Notes Against Note-Taking Systems](https://sashachapin.substack.com/p/notes-against-note-taking-systems) - An interesting contrarian take! - I am waiting for any evidence that our most provocative thinkers and writers are those who rely on elaborate, systematic note-taking systems. - I am seeing evidence that people taught knowledge management for its own sake produce unexciting work. - [MaggieAppleton/digital-gardeners](https://github.com/MaggieAppleton/digital-gardeners) - [Notes apps are where ideas go to die. And that’s good.](https://www.reproof.app/blog/notes-apps-help-us-forget) - [I Deleted My Second Brain](https://www.joanwestenberg.com/p/i-deleted-my-second-brain) ### Personal productivity Check out this section on my [list of management resources, "Personal productivity"](https://github.com/charlax/engineering-management/#personal-productivity). ### Perspective - [At 31, I have just weeks to live. Here's what I want to pass on](https://www.theguardian.com/commentisfree/2020/sep/07/terminal-cancer-live-cancer-life-death) - First, the importance of gratitude. - Second, a life, if lived well, is long enough. - Third, it’s important to let yourself be vulnerable and connect to others. - Fourth, do something for others. - Fifth, protect the planet. - [Life Is Not Short](https://dkb.show/post/life-is-not-short) - "The most surprising thing is that you wouldn’t let anyone steal your property, but you consistently let people steal your time, which is infinitely more valuable." — Seneca ### Privacy - [Privacy Enhancing Technologies: An Introduction for Technologists](https://martinfowler.com/articles/intro-pet.html), Katharine Jarmul, MartinFowler.com ### Problem solving - [Dealing with Hard Problems](https://artofproblemsolving.com/articles/hard-problems) - [Invert, always, invert](https://www.anup.io/2020/07/20/invert-always-invert/) - Define the problem - what is it that you're trying to achieve? - Invert it - what would guarantee the failure to achieve this outcome? - Finally, consider solutions to avoid this failure - 🎞 [Hammock Driven Development](https://www.youtube.com/watch?v=f84n5oFoZBc&ab_channel=ClojureTV), Rick Hickey - A classic talk on problem solving. ### Product management for software engineers See the [Product management section on my entrepreneurship-resources list of resources](https://github.com/charlax/entrepreneurship-resources#product-management). - Checkout this newsletter produced by Posthog: [Product for Engineers](https://newsletter.posthog.com/) ### Project management See the [Project management section on my engineering-management list of resources](https://github.com/charlax/engineering-management#project-management). ### Programming languages I would recommend learning: - JavaScript and maybe another interpreted language (Python, Ruby, etc.). Interpreted languages are useful for quick one-off automation scripts, and fastest to write for interviews. JavaScript is ubiquitous. - A compiled language (Java, C, C++...). - [Learn c in Y Minutes](https://learnxinyminutes.com/docs/c/) - A more recent language to see where the industry is going (as of writing, Go, Swift, Rust, Elixir...). - A language that has first-class support for functional programming (Haskell, Scala, Clojure...). A bit more reading: - [A brief, incomplete, mostly wrong history of programming languages](http://james-iry.blogspot.fr/2009/05/brief-incomplete-and-mostly-wrong.html) - [Types](https://gist.github.com/garybernhardt/122909856b570c5c457a6cd674795a9c) - [Resources To Help You To Create Programming Languages](https://tomassetti.me/resources-create-programming-languages/) - [Effective Programs - 10 Years of Clojure](https://www.youtube.com/watch?v=2V1FtfBDsLU) 🎞, Rich Hickey. The author of Clojure reflects on his programming experience and explains the rationale behind some of Clojure's key design decisions. - [Learn more programming languages, even if you won't use them](https://thorstenball.com/blog/2019/04/09/learn-more-programming-languages/), Thorsten Ball - These new perspectives, these ideas and patterns — they linger, they stay with you, even if you end up in another language. And that is powerful enough to keep on learning new languages, because one of the best things that can happen to you when you’re trying to solve a problem is a change of perspective. - [Programming Language Checklist](https://famicol.in/language_checklist.html): a fun take on "so you want to build your own language?" - [Static vs. dynamic languages: a literature review](http://danluu.com/empirical-pl/) - [Polyglot Programming and the Benefits of Mastering Several Languages](https://www.stxnext.com/blog/polyglot-programming/) - [It's not what programming languages do, it's what they shepherd you to](https://nibblestew.blogspot.com/2020/03/its-not-what-programming-languages-do.html) - [Ask HN: What do you code when learning a new language/framework?](https://news.ycombinator.com/item?id=32092943) - [The seven programming ur-languages](https://madhadron.com/programming/seven_ur_languages.html): ALGOL, Lisp, ML, Self, Forth, APL, Prolog - [Lua: The Little Language That Could](https://matt.blwt.io/post/lua-the-little-language-that-could/) - [The Montréal Effect: Why Programming Languages Need a Style Czar](https://earthly.dev/blog/language-style-czar/) - [TodePond/DreamBerd: a perfect programming language](https://github.com/TodePond/DreamBerd) - [Programming Languages That Blew My Mind](https://yoric.github.io/post/programming-languages-that-blew-my-mind/) > There are only two kinds of languages: the ones people complain about and the ones nobody uses. -- Bjarne Stroustrup (C++ creator) List of resources: - [Great Works in Programming Languages](https://www.cis.upenn.edu/~bcpierce/courses/670Fall04/GreatWorksInPL.shtml) #### Python For Python check out my [professional Python education repository](https://github.com/charlax/python-education). #### JavaScript In this repository: check [./training/front-end/](./training/front-end/) JavaScript is such a pervasive language that it's almost required learning. - [mbeaudru/modern-js-cheatsheet](https://github.com/mbeaudru/modern-js-cheatsheet): cheatsheet for the JavaScript knowledge you will frequently encounter in modern projects. - [javascript-tutorial](https://github.com/javascript-tutorial): comprehensive JavaScript guide with simple but detailed explanantions. Available in several languages. - [30 Days of JavaScript](https://github.com/Asabeneh/30-Days-Of-JavaScript): 30 days of JavaScript programming challenge is a step-by-step guide to learn JavaScript programming language in 30 days. - [Unleash JavaScript's Potential with Functional Programming](https://janhesters.com/blog/unleash-javascripts-potential-with-functional-programming) #### Garbage collection - [A Guide to the Go Garbage Collector](https://go.dev/doc/gc-guide): a very insightful guide about Go's GC ### Programming paradigm - [Imperative vs Declarative Programming](https://tylermcginnis.com/imperative-vs-declarative-programming/), Tyler McGinnis. - I draw the line between declarative and non-declarative at whether you can trace the code as it runs. Regex is 100% declarative, as it’s untraceable while the pattern is being executed. - 🎞 [Imperative vs Declarative Programming](https://www.youtube.com/watch?v=E7Fbf7R3x6I&ab_channel=uidotdev) ### Public speaking (presenting) - [Speaking for hackers](https://sfhbook.netlify.app/) ### Reading - [The Complete Guide to Effective Reading](https://maartenvandoorn.nl/reading-guide/) - [The benefits of note-taking by hand](https://www.bbc.com/worklife/article/20200910-the-benefits-of-note-taking-by-hand) - [The Art of Reading More Effectively and Efficiently](https://aliabdaal.com/read-more-effectively/) - [You should be reading academic computer science papers](https://stackoverflow.blog/2022/04/07/you-should-be-reading-academic-computer-science-papers/), Stack Overflow Blog - [How to Remember What You Read](https://fs.blog/remember-books/) - Take notes - Stay focused - Mark up the book - Make mental links - Quit books - [Writing summaries is more important than reading more books](https://www.andreasfragner.com/writing/writing-summaries) - In 1-2 sentences, what is the book about as a whole? - What are the 3-4 central questions it tries to answer? - Summarize the answers in one paragraph each. - What are the most important things you have learned personally? - There was an interesting contrarian take in the [Hacker News thread](https://news.ycombinator.com/item?id=36011599): "Once I relaxed and decided, 'If the stuff in this book is good enough, my brain will keep it FOR me' both my satisfaction AND utility of books increased dramatically." - [You Are What You Read, Even If You Don’t Always Remember It](https://blog.jim-nielsen.com/2024/you-are-what-you-read/) - "I cannot remember the books I've read any more than the meals I have eaten; even so, they have made me.", Ralp Waldo Emerson - [How I Read](https://www.robkhenderson.com/p/how-i-read) - I read pretty slowly. I take notes, I underline, I highlight, I jot my thoughts in the margins, I pause if I encounter an especially interesting passage or idea. - "Listening to books instead of reading them is like drinking your vegetables instead of eating them" - “You will not learn anything of lasting importance from TV, movies, podcasts…they’re junk food. Successful people converge on 3 ways to learn: lots of reading time, some exercises and projects, and conversations with people who are slightly ahead of them.” ### Refactoring - [The Rule of Three](https://blog.codinghorror.com/rule-of-three/), Coding Horror - Every programmer ever born thinks whatever idea just popped out of their head into their editor is the most generalized, most flexible, most one-size-fits all solution that has ever been conceived. - It is three times as difficult to build reusable components as single use components. - A reusable component should be tried out in three different applications before it will be sufficiently general to accept into a reuse library. - [Refactor vs. Rewrite](https://remesh.blog/refactor-vs-rewrite-7b260e80277a) - [Tripping over the potholes in too many libraries](https://blog.carlmjohnson.net/post/2020/avoid-dependencies/) - [Build It Yourself](https://lucumr.pocoo.org/2025/1/24/build-it-yourself/) - It's 2025 and it's faster for me to have ChatGPT or Cursor whip up a dependency free implementation of these common functions, than it is for me to start figuring out a dependency. - [Refactoring with Codemods to Automate API Changes](https://martinfowler.com/articles/codemods-api-refactoring.html), martinfowler.com ### Regex - [The Best Regex Trick](http://rexegg.com/regex-best-trick.html) - [regex101: build, test, and debug regex](https://regex101.com/) ### Releasing & deploying - [How to deploy software](https://zachholman.com/posts/deploying-software), Zach Holman - [BlueGreenDeployment](http://martinfowler.com/bliki/BlueGreenDeployment.html), Martin Fowler - [Move fast and break nothing](https://zachholman.com/talk/move-fast-break-nothing/), Zach Holman - 🏙 [Move fast and don't break things](https://docs.google.com/presentation/d/15gNk21rjer3xo-b1ZqyQVGebOp_aPvHU3YH7YnOMxtE/edit#slide=id.g437663ce1_53_591), Google - [Shipping to Production](https://blog.pragmaticengineer.com/shipping-to-production/), The Pragmatic Programmer #### Versioning - [SemVer - Semantic Versioning](https://semver.org/) - [CalVer - Calendar Versioning](https://calver.org/) - [Semantic Versioning Will Not Save You](https://hynek.me/articles/semver-will-not-save-you/) - [Version numbers: how to use them?](https://bernat.tech/posts/version-numbers/) #### Checklists - [Production Readiness Checklist](https://gruntwork.io/devops-checklist/), Gruntwork - [Checklist: what had to be done before deploying microservices to production](https://habr.com/en/post/438186/) - [Things end users care about but programmers don't](https://instadeq.com/blog/posts/things-end-users-care-about-but-programmers-dont/): includes colors, formatting, themes, integrations, UX, compatibility, operations. #### Feature flags - [Flipping out](http://code.flickr.net/2009/12/02/flipping-out/), Flickr. One of the first articles about feature flags. - [Feature Flags, Toggles, Controls](https://featureflags.io/), a website documenting feature flags, from Launch Darkly. - [Feature Toggles (aka Feature Flags)](https://martinfowler.com/articles/feature-toggles.html), Pete Hodgson, martinFowler.com. Comprehensive article on the topic. - Deliver new functionality to users rapidly but safely - Release Toggles allow incomplete and un-tested codepaths to be shipped to production as latent code which may never be turned on. - Experiment Toggles are used to perform multivariate or A/B testing. - Ops Toggles control operational aspects of our system's behavior. - Permissioning Toggles change the features or product experience that certain users receive. - Static vs dynamic toggles - Long-lived toggles vs transient toggles - Savvy teams view their Feature Toggles as inventory which comes with a carrying cost, and work to keep that inventory as low as possible. - [Feature Flags Best Practices: Release Management](https://launchdarkly.com/blog/release-management-flags-best-practices/), LaunchDarkly - [How we ship code faster and safer with feature flags](https://github.blog/2021-04-27-ship-code-faster-safer-feature-flags/), Github. - [Flipr: Making Changes Quickly and Safely at Scale](https://eng.uber.com/flipr/), Uber - [Feature flags are ruining your codebase](https://zaidesanton.substack.com/p/feature-flags-are-ruining-your-codebase) #### Testing in production - [Why We Leverage Multi-tenancy in Uber's Microservice Architecture](https://eng.uber.com/multitenancy-microservice-architecture/) - [Developing in Production](https://tersesystems.com/blog/2020/01/22/developing-in-production/) - Complex systems have emergent behavior, producing epiphenomenon that only appears with sufficient scale. - Wood's theorem: As the complexity of a system increases, the accuracy of any single agent’s own model of that system decreases rapidly. - The more tools and code that you add to create elements in a system, the harder it is to replicate an environment encompassing those tools and code. - At the core of testing in production is the idea of splitting deployments (of artifacts) from releases (of features). - [Testing in Production: the hard parts](https://medium.com/@copyconstruct/testing-in-production-the-hard-parts-3f06cefaf592), Cindy Sridharan - The whole point of [actual] distributed systems engineering is you assume you’re going to fail at some point in time and you design the system in such a way that the damage, at each point is minimized, that recovery is quick, and that the risk is acceptably balanced with cost. - How can you cut the blast radius for a similar event in half? - Differentiate between deployment (0 risk) and release - Build a deploy-observe-release pipeline - Make incremental rollouts the norm (canaries, %-based rollouts, etc.) - Test configuration changes just like you test code - Default to roll back, avoid fixing forward (slow!) - Eliminate gray failures - prefer crashing to degrading in certain cases - Prefer loosely coupled services at the expense of latency or correctness - Use poison tasters (isolate handling of client input) - Implement per-request-class backpressure - Have proper visibility from a client/end-user standpoint (client-side metrics) - [Testing in Production, the safe way](https://medium.com/@copyconstruct/testing-in-production-the-safe-way-18ca102d0ef1) - [Multi-Tenancy in a Microservice Architecture](https://www.usenix.org/system/files/login/articles/login_winter19_10_gud.pdf) ### Reliability **See also [System architecture](#system-architecture)** Books: - 📖 [Site Reliability Engineering](https://landing.google.com/sre/books/), Google - Written by members of Google's SRE team, with a comprehensive analysis of the entire software lifecycle - how to build, deploy, monitor, and maintain large scale systems. - 📖 [Incident Metrics in SRE](https://static.googleusercontent.com/media/sre.google/en//static/pdf/IncidentMeticsInSre.pdf), Štěpán Davidovič, Google - Canonical definition of TTD (Time To Detect) TTM (Mitigate), TTR (Recover), TBF (Time Between Failure) Quotes: > Quality is a snapshot at the start of life and reliability is a motion picture of the day-by-day operation. > – [NIST](https://www.itl.nist.gov/div898/handbook/apr/section1/apr111.htm) > Reliability is the one feature every customer users. -- An auth0 SRE. Articles: - I already mentioned the book Release it! above. There's also a [presentation](http://www.slideshare.net/justindorfman/stability-patterns-presentation) from the author. - [Service Recovery: Rolling Back vs. Forward Fixing](https://www.linkedin.com/pulse/service-recovery-rolling-back-vs-forward-fixing-mohamed-el-geish/) - [How Complex Systems Fail](https://how.complexsystems.fail/) - Catastrophe requires multiple failures – single point failures are not enough. - Complex systems contain changing mixtures of failures latent within them. - Post-accident attribution to a ‘root cause’ is fundamentally wrong. - Hindsight biases post-accident assessments of human performance. - Safety is a characteristic of systems and not of their components - Failure free operations require experience with failure. - [Systems that defy detailed understanding](https://blog.nelhage.com/post/systems-that-defy-understanding/) - Focus effort on systems-level failure, instead of the individual component failure. - Invest in sophisticated observability tools, aiming to increase the number of questions we can ask without deploying custom code - [Operating a Large, Distributed System in a Reliable Way: Practices I Learned](https://blog.pragmaticengineer.com/operating-a-high-scale-distributed-system/), Gergely Orosz. - A good summary of processes to implement. - [Production Oriented Development](https://paulosman.me/2019/12/30/production-oriented-development.html) - Code in production is the only code that matters - Engineers are the subject matter experts for the code they write and should be responsible for operating it in production. - Buy Almost Always Beats Build - Make Deploys Easy - Trust the People Closest to the Knives - QA Gates Make Quality Worse - Boring Technology is Great. - Non-Production Environments Have Diminishing Returns - Things Will Always Break - 🏙 [High Reliability Infrastructure migrations](https://speakerdeck.com/jvns/high-reliability-infrastructure-migrations), Julia Evans. - [Appendix F: Personal Observations on the Reliability of the Shuttle](https://www.refsmmat.com/files/reflections.pdf), Richard Feynman - [Lessons learned from two decades of Site Reliability Engineering](https://sre.google/resources/practices-and-processes/twenty-years-of-sre-lessons-learned/) - [Service Reliability Mathematics](https://addyosmani.com/blog/service-reliability/), Addy Osmani Resources: - 🧰 [dastergon/awesome-sre](https://github.com/dastergon/awesome-sre) - 🧰 [upgundecha/howtheysre](https://github.com/upgundecha/howtheysre): a curated collection of publicly available resources on SRE at technology and tech-savvy organizations #### Integration patterns (dependency management) - [Circuit Breaker](https://martinfowler.com/bliki/CircuitBreaker.html) (mentioned in the Release it! book) - [Making the Netflix API More Resilient](https://netflixtechblog.com/making-the-netflix-api-more-resilient-a8ec62159c2d), Netflix Blog - 🏙 [Application Resilience Engineering & Operations at Netflix - Speaker Deck](https://speakerdeck.com/benjchristensen/application-resilience-engineering-and-operations-at-netflix) - [Design Pattern: Circuit Breaker](https://learn.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker), Microsoft Azure - [Rate limiter algorithms](https://smudge.ai/blog/ratelimit-algorithms) (and their [implementation](https://github.com/upstash/ratelimit-js/blob/main/src/lua-scripts/single.ts)) - [Interactive Guide: Mastering Rate Limiting](https://blog.sagyamthapa.com.np/interactive-guide-to-rate-limiting) - [Good Retry, Bad Retry: An Incident Story](https://medium.com/yandex/good-retry-bad-retry-an-incident-story-648072d3cee6): insightful, well-written story about retries, circuit breakers, deadline, etc. #### Resiliency - 🏙 [The Walking Dead - A Survival Guide to Resilient Applications](https://speakerdeck.com/daschl/the-walking-dead-a-survival-guide-to-resilient-applications) - 🏙 [Defensive Programming & Resilient systems in Real World (TM)](https://speakerdeck.com/tuenti/defensive-programming-and-resilient-systems-in-real-world-tm) - 🏙 [Full Stack Fest: Architectural Patterns of Resilient Distributed Systems](https://speakerdeck.com/randommood/full-stack-fest-architectural-patterns-of-resilient-distributed-systems) - 🏙 [The 7 quests of resilient software design](https://www.slideshare.net/ufried/the-7-quests-of-resilient-software-design) - 🧰 [Resilience engineering papers](https://github.com/lorin/resilience-engineering): comprehensive list of resources on resilience engineering - [MTTR is more important than MTBF (for most types of F)](https://www.kitchensoap.com/2010/11/07/mttr-mtbf-for-most-types-of-f/) (also as a [presentation](https://www.slideshare.net/jallspaw/dev-and-ops-collaboration-and-awareness-at-etsy-and-flickr)) - [Failure is inevitable: learning from a large outage, and building for reliability in depth at Datadog](https://www.datadoghq.com/blog/engineering/rethinking-reliability/) ### Search - [What every software engineer should know about search](https://scribe.rip/p/what-every-software-engineer-should-know-about-search-27d1df99f80d) ### Security - 📖 [Penetration Testing: A Hands-On Introduction to Hacking](https://nostarch.com/pentesting), Georgia Weidman - [Penetration Testing Tools Cheat Sheet](https://highon.coffee/blog/penetration-testing-tools-cheat-sheet/#http--https-webserver-enumeration) - [A practical guide to securing macOS](https://github.com/drduh/macOS-Security-and-Privacy-Guide) - [Web Application Security Guide/Checklist](https://en.wikibooks.org/wiki/Web_Application_Security_Guide/Checklist) - [Reckon you've seen some stupid security things?](https://www.troyhunt.com/reckon-youve-seen-some-stupid-security-things-here-hold-my-beer/): everything _not_ to do. - [Checklist of the most important security countermeasures when designing, testing, and releasing your API](https://github.com/shieldfy/API-Security-Checklist) - [OWASP Cheat Sheet Series](https://cheatsheetseries.owasp.org/): a series of cheat sheets about various security topics. - [Docker Security](https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html) - [How to improve your Docker containers security](https://blog.gitguardian.com/how-to-improve-your-docker-containers-security-cheat-sheet/) - [Secure by Design](https://henrikwarne.com/2020/03/22/secure-by-design/), a book review by Henrik Warne. - There is a big overlap between secure code and good software design - Every domain value should instead be represented by a domain primitive. - External input needs to be validated before it is used in the system, in the following order: origin, size, lexical content, syntax, semantics. - Entities should be consistent at creation, have limited operation, shouldn't be sharing mutable objects. - Three Rs to do every few hours: rotate secrets automatically, repave servers and applications (redeploy on clean footprint), repair vulnerable. - Don’t use exceptions for the control flow. - [OWASP Top Ten Web Application Security Risks](https://owasp.org/www-project-top-ten/) - [How to start an AppSec program with the OWASP Top 10](https://owasp.org/Top10/A00_2021-How_to_start_an_AppSec_program_with_the_OWASP_Top_10/) - [ukncsc/zero-trust-architecture: Principles to help you design and deploy a zero trust architecture](https://github.com/ukncsc/zero-trust-architecture) - 🏙 [Minimum Viable Security](https://speakerdeck.com/jacobian/minimum-viable-security-wharton-web-conference-2015) - [The Open Software Assurance Maturity Model](https://www.opensamm.org/) - [Security by Obscurity is Underrated](https://utkusen.com/blog/security-by-obscurity-is-underrated) - [Don't Wanna Pay Ransom Gangs? Test Your Backups](https://krebsonsecurity.com/2021/07/dont-wanna-pay-ransom-gangs-test-your-backups/), Krebs on Security - [The Beginner’s Guide to Passwords](https://medium.com/peerio/how-to-build-a-billion-dollar-password-3d92568d9277#67c2) - [The Password Game](https://neal.fun/password-game/) - [Learnings from 5 years of tech startup code audits](https://kenkantzer.com/learnings-from-5-years-of-tech-startup-code-audits/) - [API Tokens: A Tedious Survey](https://fly.io/blog/api-tokens-a-tedious-survey/): don't use JWT. - [The Six Dumbest Ideas in Computer Security](http://ranum.com/security/computer_security/editorials/dumb/index.html) - [How to Say "No" Well](https://ramimac.me/saying-no) Training for developers: - [Hacksplaining](https://www.hacksplaining.com/) - [Codebashing](https://free.codebashing.com/) - [OWASP Security Knowledge Framework](https://owasp.org/www-project-security-knowledge-framework/) - [PagerDuty Security Training](https://sudo.pagerduty.com/for_engineers/) - [Gruyere: Web Application Exploits and Defenses](https://google-gruyere.appspot.com/part1) List of resources: - 🧰 [meirwah/awesome-incident-response](https://github.com/meirwah/awesome-incident-response): tools for incident response - 🧰 [Starting Up Security](https://scrty.io/) - 🧰 [decalage2/awesome-security-hardening](https://github.com/decalage2/awesome-security-hardening): security hardening guides, tools and other resources ### Research papers - [Papers we love](https://github.com/papers-we-love/papers-we-love): papers from the computer science community to read and discuss. Can be a good source of inspiration of solving your design problems. - [The morning paper](https://blog.acolyer.org/): one CS research paper explained every morning. - [The 7 Most Influential Papers in Computer Science History](https://terriblesoftware.org/2025/01/22/the-7-most-influential-papers-in-computer-science-history/) ### Shell (command line) - [The case for bash](https://www.neversaw.us/2021/04/02/the-case-for-bash/) - 🧰 [alebcay/awesome-shell](https://github.com/alebcay/awesome-shell) - 🧰 [dylanaraps/pure-bash-bible](https://github.com/dylanaraps/pure-bash-bible): pure bash alternatives to external processes. - [The Bash Hackers Wiki](https://wiki.bash-hackers.org/) provides a gentler way to learn about bash than its manages. - [Awk in 20 Minutes](https://ferd.ca/awk-in-20-minutes.html) - 🏙 [Linux Productivity Tools](https://www.usenix.org/sites/default/files/conference/protected-files/lisa19_maheshwari.pdf) - [jlevy/the-art-of-command-line](https://github.com/jlevy/the-art-of-command-line): master the command line, in one page **must read** - [Minimal safe Bash script template](https://betterdev.blog/minimal-safe-bash-script-template/) - [Command Line Interface Guidelines](https://clig.dev/) - [The Linux Commands Handbook](https://openbootcamps.com/the-linux-commands-handbook/) - [How to write idempotent Bash scripts](https://arslan.io/2019/07/03/how-to-write-idempotent-bash-scripts/) - [Learn bash by playing an adventure](https://gitlab.com/slackermedia/bashcrawl) - [Effective Shell](https://effective-shell.com/) - [Computing from the Command Line](https://learnbyexample.github.io/cli-computing/preface.html) - [What helps people get comfortable on the command line?](https://jvns.ca/blog/2023/08/08/what-helps-people-get-comfortable-on-the-command-line-/), Julia Evans - [6 Techniques I Use to Create a Great User Experience for Shell Scripts](https://nochlin.com/blog/6-techniques-i-use-to-create-a-great-user-experience-for-shell-scripts) ### SQL - [SQL styleguide](http://www.sqlstyle.guide/) - [Best practices for writing SQL queries](https://www.metabase.com/learn/building-analytics/sql-templates/sql-best-practices) - [Practical SQL for Data Analysis](https://hakibenita.com/sql-for-data-analysis) - [Reasons why SELECT * is bad for SQL performance](https://tanelpoder.com/posts/reasons-why-select-star-is-bad-for-sql-performance/) - [Animate SQL](https://animatesql.com/) - [Lost at SQL](https://lost-at-sql.therobinlord.com/), an SQL learning game - [Joins 13 Ways](https://justinjaffray.com/joins-13-ways/?a=b) - [spandanb/learndb-py](https://github.com/spandanb/learndb-py): learn database internals by implementing it from scratch. - [gvwilson/querynomicon](https://github.com/gvwilson/querynomicon): an Introduction to SQL for the Cautious and Weary ### State - [A Map of Sync](https://stack.convex.dev/a-map-of-sync) categorizes state sync into 9 dimensions. - Data model: - Size: How large is the data set that a single client can access? - Update rate: How often do clients send updates? - Structure: Is the data rich with structure or flat and unstructured? - Systems requirements: - Input latency: How long can updates be delayed while maintaining a good user experience? - Offline: How many interactions does the app need to support offline? - Concurrent clients: How many concurrent clients will look at the same data? - Programming model: - Centralization: How centralized is the programming model and infrastructure? Flexibility: How flexible are sync policies, especially around conflict resolution? - Consistency: What types of invariants can the application assert about its data model, and how strong can these invariants be? ### System administration - 🧰 [kahun/awesome-sysadmin](https://github.com/kahun/awesome-sysadmin): a curated list of amazingly awesome open source sysadmin resources ### System architecture **See also [Reliability](#system-architecture), [Scalability](#scalability)** Reading lists: - 🧰 [donnemartin/system-design-primer](https://github.com/donnemartin/system-design-primer): learn how to design large scale systems. Prep for the system design interview. - 🧰 [A Distributed Systems Reading List](http://dancres.github.io/Pages/) - 🧰 [Foundational distributed systems papers](http://muratbuffalo.blogspot.com/2021/02/foundational-distributed-systems-papers.html) - 🧰 [Services Engineering Reading List](https://github.com/mmcgrana/services-engineering) - 🧰 [System Design Cheatsheet](https://gist.github.com/vasanthk/485d1c25737e8e72759f) - [karanpratapsingh/system-design](https://github.com/karanpratapsingh/system-design): learn how to design systems at scale and prepare for system design interviews - [A Distributed Systems Reading List](https://ferd.ca/a-distributed-systems-reading-list.html) Blogs: - [High Scalability](http://highscalability.com/): great blog about system architecture, its weekly review article are packed with numerous insights and interesting technology reviews. Checkout the [all-times favorites](http://highscalability.com/all-time-favorites/). Books: - 📖 [Building Microservices](https://www.amazon.com/Building-Microservices-Designing-Fine-Grained-Systems/dp/1491950358), Sam Newman (quite complete discussion of microservices) - 📖 [Designing Data-Intensive Applications](https://dataintensive.net/) - 📖 [Building Secure and Reliable Systems](https://google.github.io/building-secure-and-reliable-systems/raw/toc.html), Google, (free online). Very complete book. - Design for security & reliability - Design tradeoffs - Design for least privilege - Design for understandability - Design for a changing landscape - Design for resilience - Design for recovery Articles: - [6 Rules of thumb to build blazing fast web server applications](http://loige.co/6-rules-of-thumb-to-build-blazing-fast-web-applications/) - [The twelve-factor app](http://12factor.net/) - [Introduction to architecting systems for scale](http://lethain.com/introduction-to-architecting-systems-for-scale/) - [The Log: What every software engineer should know about real-time data's unifying abstraction](https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying): one of those classical articles that everyone should read. - [Turning the database outside-out with Apache Samza](https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/) - [Fallacies of distributed computing](https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing), Wikipedia - [The biggest thing Amazon got right: the platform](https://gigaom.com/2011/10/12/419-the-biggest-thing-amazon-got-right-the-platform/) - All teams will henceforth expose their data and functionality through service interfaces. - Monitoring and QA are the same thing. - [Building Services at Airbnb, part 3](https://medium.com/airbnb-engineering/building-services-at-airbnb-part-3-ac6d4972fc2d) - Resilience is a Requirement, Not a Feature - [Building Services at Airbnb, part 4](https://medium.com/airbnb-engineering/building-services-at-airbnb-part-4-23c95e428064) - Building Schema Based Testing Infrastructure for service development - [Patterns of Distributed Systems](https://martinfowler.com/articles/patterns-of-distributed-systems/), MartinFowler.com - [ConwaysLaw](https://martinfowler.com/bliki/ConwaysLaw.html), MartinFowler.com (regarding organization, check out my [engineering-management](https://github.com/charlax/engineering-management/) list). - [The C4 model for visualising software architecture](https://c4model.com/) - [If Architects had to work like Programmers](http://www.gksoft.com/a/fun/architects.html) #### Architecture patterns - BFF (backend for frontend) - [Backends For Frontends](https://samnewman.io/patterns/architectural/bff/) - [Load Balancing](https://samwho.dev/load-balancing/): a visual exploration of load balancing algos - [AWS Well-Architected Framework](https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html) - Operational excellence - Security - Reliability - Performance efficiency - Cost optimization - Sustainability #### Microservices/splitting a monolith - [Monolith First](https://martinfowler.com/bliki/MonolithFirst.html), Martin Fowler - [Service oriented architecture: scaling the Uber engineering codebase as we grow](https://eng.uber.com/soa/) - [Don’t start with microservices in production – monoliths are your friend](https://arnoldgalovics.com/microservices-in-production/) - [Deep lessons from Google And EBay on building ecosystems of microservices](http://highscalability.com/blog/2015/12/1/deep-lessons-from-google-and-ebay-on-building-ecosystems-of.html) - [Introducing domain-oriented microservice architecture](https://eng.uber.com/microservice-architecture/), Uber - Instead of orienting around single microservices, we oriented around collections of related microservices. We call these domains. - In small organizations, the operational benefit likely does not offset the increase in architectural complexity. - [Best Practices for Building a Microservice Architecture](https://www.vinaysahni.com/best-practices-for-building-a-microservice-architecture#correlation-ids) - 🏙 [Avoid Building a Distributed Monolith](https://speakerdeck.com/felipead/avoid-building-a-distributed-monolith) - 🏙 [Breaking down the monolith](https://speakerdeck.com/slashdotpeter/breaking-down-the-monolith-devone) - [Monoliths are the future](https://changelog.com/posts/monoliths-are-the-future) - "We’re gonna break it up and somehow find the engineering discipline we never had in the first place." - [12 Ways to Prepare your Monolith Before Transitioning to Microservices](https://semaphoreci.com/blog/monolith-microservices) - [Death by a thousand microservices](https://renegadeotter.com/2023/09/10/death-by-a-thousand-microservices.html) - [Microservices](https://www.youtube.com/watch?v=y8OnoxKotPQ&ab_channel=KRAZAM) - [Disasters I've seen in a microservices world](https://world.hey.com/joaoqalves/disasters-i-ve-seen-in-a-microservices-world-a9137a51) - [You Want Microservices—But Do You Need Them?](https://www.docker.com/blog/do-you-really-need-microservices/), Docker ### Scalability **See also: [Reliability](#reliability), [System architecture](#system-architecture)** - [Scalable web architecture and distributed systems](http://www.aosabook.org/en/distsys.html) - 📖 [Scalability Rules: 50 Principles for Scaling Web Sites](https://smile.amazon.com/Scalability-Rules-Principles-Scaling-Sites/dp/013443160X) ([presentation](http://www.slideshare.net/cyrilwang/scalability-rules)) - [Scaling to 100k Users](https://alexpareto.com/scalability/systems/2020/02/03/scaling-100k.html), Alex Pareto. The basics of getting from 1 to 100k users. ### Site Reliability Engineering (SRE) **See: [Reliability](#reliability)** ### Technical debt - [TechnicalDebt](https://martinfowler.com/bliki/TechnicalDebt.html), Martin Fowler. - [Fixing Technical Debt with an Engineering Allocation Framework](https://docs.google.com/presentation/d/16WU1cxG02jnVGQ5byviw3_Q0ILDPZPYtTvU91_210T0/edit#slide=id.p) - You don't need to stop shipping features to fix technical debt - Communicate the business value - [Ur-Technical Debt](https://www.georgefairbanks.com/ieee-software-v32-n4-july-2020-ur-technical-debt) - Today, any code that a developer dislikes is branded as technical debt. - Ward Cunningham invented the debt metaphor to explain to his manager that building iteratively gave them working code faster, much like borrowing money to start a project, but that it was essential to keep paying down the debt, otherwise the interest payments would grind the project to a halt. - Ur-technical debt is generally not detectable by static analysis. - [3 Kinds of Good Tech Debt](https://engineering.squarespace.com/blog/2019/three-kinds-of-good-tech-debt) ### Testing - ⭐️ [Testing strategies in a microservices architecture](http://martinfowler.com/articles/microservice-testing/) (Martin Fowler) is an awesome resources explaining how to test a service properly. - 🧰 [Testing Distributed Systems](https://asatarin.github.io/testing-distributed-systems/) Why test: - [Why bother writing tests at all?](https://dave.cheney.net/2019/05/14/why-bother-writing-tests-at-all), Dave Cheney. A good intro to the topic. - Even if you don’t, someone will test your software - The majority of testing should be performed by development teams - Manual testing should not be the majority of your testing because manual testing is O(n) - Tests are the critical component that ensure you can always ship your master branch - Tests lock in behaviour - Tests give you confidence to change someone else’s code How to test: - [A quick puzzle to test your problem solving](http://www.nytimes.com/interactive/2015/07/03/upshot/a-quick-puzzle-to-test-your-problem-solving.html?_r=0)... and a great way to learn about confirmation bias and why you're mostly writing positive test cases. - [Testing is not for beginners](https://www.calhoun.io/testing-is-not-for-beginners/): why learning to test is hard. This shouldn't demotivate you though! - [Arrange-act-assert: a pattern for writing good tests](https://automationpanda.com/2020/07/07/arrange-act-assert-a-pattern-for-writing-good-tests/) - [Test smarter, not harder](https://lukeplant.me.uk/blog/posts/test-smarter-not-harder/) Test pyramid: - [The test pyramid](http://martinfowler.com/bliki/TestPyramid.html), Martin Fowler - [Eradicating non-determinism in tests](http://www.martinfowler.com/articles/nonDeterminism.html), Martin Fowler - [The practical test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html), MartinFowler.com - Be clear about the different types of tests that you want to write. Agree on the naming in your team and find consensus on the scope of each type of test. - Every single test in your test suite is additional baggage and doesn't come for free. - Test code is as important as production code. - [Software testing anti-patterns](http://blog.codepipes.com/testing/software-testing-antipatterns.html), Kostis Kapelonis. - [Write tests. Not too many. Mostly integration.](https://blog.kentcdodds.com/write-tests-not-too-many-mostly-integration-5e8c7fff591c) for a contrarian take about unit testing - 🎞 [Unit test 2, Integration test: 0](https://www.youtube.com/watch?v=Oj8bfBlwHAg&ab_channel=PercyRicardoAnticonaMasabel) - [Testing in the Twenties](https://www.tbray.org/ongoing/When/202x/2021/05/15/Testing-in-2021) - [Google Testing Blog: Test Sizes](https://testing.googleblog.com/2010/12/test-sizes.html) - [Pyramid or Crab? Find a testing strategy that fits](https://web.dev/articles/ta-strategies), web.dev End-to-end tests: - [Just say no to more end-to-end tests](https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html), Google Testing Blog - [End-to-end testing considered harmful](https://www.stevesmith.tech/blog/end-to-end-testing-considered-harmful/) ### Tools - [DevDocs API Documentation](https://devdocs.io/): a repository for multiple API docs (see also [Dash for macOS](https://kapeli.com/dash)). - [DevChecklist](https://devchecklists.com/): a collaborative space for sharing checklists that help ensure software quality - 🧰 [Free for developers](https://free-for.dev/#/): list of free tiers for developments tools and services - [Ask HN: Best dev tool pitches of all time?](https://news.ycombinator.com/item?id=31782200) - [A list of /uses pages detailing developer setups, gear, software and configs](https://uses.tech/) > The future life expectancy of some non-perishable things, like a technology or an idea, is proportional to their current age > — Lindy’s Law ### Type system - [Counterexamples in Type Systems](https://counterexamples.org/intro.html): a library of runtime issues that weren't caught by the type system - [Use Your Type System](https://www.dzombak.com/blog/2025/07/use-your-type-system/) - Your models should each have their own ID type. Public and even private functions should often avoid dealing in floats or integers alone. ### Typography - [Butterick’s Practical Typography](https://practicaltypography.com/) - [Typography for Lawyers](https://typographyforlawyers.com/) - [Quick guide to web typography for developers · OlegWock](https://sinja.io/blog/web-typography-quick-guide) - [Features of your font you had no idea about](https://sinja.io/blog/get-maximum-out-of-your-font) ### Version control (Git) Learning Git, courses and books: - 📖 [Git Book](https://git-scm.com/book/en/v2) - [Git from the inside out](https://codewords.recurse.com/issues/two/git-from-the-inside-out) - [Git Tutorials and Training](https://www.atlassian.com/git/tutorials), Atlassian - [Git Immersion](https://gitimmersion.com/) - [A Visual Git Reference](http://marklodato.github.io/visual-git-guide/index-en.html) (a bit more advanced) - [Think Like (a) Git](http://think-like-a-git.net/) - [Git's database internals I: packed object store](https://github.blog/2022-08-29-gits-database-internals-i-packed-object-store/): an insightful deep dive from Github - [Oh My Git!](https://ohmygit.org/): a game to learn git - [How Core Git Developers Configure Git](https://blog.gitbutler.com/how-git-core-devs-configure-git/) Cheat sheets: - [Git Cheat Sheet](https://github.com/arslanbilal/git-cheat-sheet) - [git-tips](https://github.com/git-tips/tips) - [Oh Shit, Git!?!](https://ohshitgit.com/) More specific topics: - [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) - [Git Merge vs. Rebase: What’s the Diff?](https://hackernoon.com/git-merge-vs-rebase-whats-the-diff-76413c117333) - 🏙 [Story-telling with Git rebase](https://speakerdeck.com/aemeredith/story-telling-with-git-rebase) - 🏙 [Git Rebase vs. Merge](https://speakerdeck.com/mrfoto/git-rebase-vs-merge) - 🏙 [10 Git Anti Patterns You Should be Aware of](https://speakerdeck.com/lemiorhan/10-git-anti-patterns-you-should-be-aware-of) - [Learn Git Branching](https://learngitbranching.js.org/): an interactive game - [Fix conflicts only once with git rerere](https://medium.com/@porteneuve/fix-conflicts-only-once-with-git-rerere-7d116b2cec67) - [Monorepo Explained](https://monorepo.tools/) - [How to Write a Git Commit Message](https://cbea.ms/git-commit/) - [git-worktree](https://git-scm.com/docs/git-worktree): manage multiple working trees attached to the same repository. ### Work ethics, productivity & work/life balance Check out this section on my [list of engineering-management resources, "Personal productivity"](https://github.com/charlax/engineering-management/#personal-productivity). ### Web development In this repository: check [training/web-dev/](./training/web-dev/) and [./training/front-end/](./training/front-end/) Learning guide and resources: - [grab/front-end-guide](https://github.com/grab/front-end-guide): a study guide and introduction to the modern front end stack. - [Front-End Developer Handbook 2019](https://frontendmasters.com/books/front-end-handbook/2019/), Cody Lindley - [A Directory of design and front-end resources](http://uigoodies.com/index.html) - 🧰 [codingknite/frontend-development](https://github.com/codingknite/frontend-development): a list of resources for frontend development Topics: - [136 facts every web dev should know](https://www.baldurbjarnason.com/2021/100-things-every-web-developer-should-know/) - [Maintainable CSS](http://maintainablecss.com/) - [Things you forgot (or never knew) because of React](https://joshcollinsworth.com/blog/antiquated-react) - [Checklist - The A11Y Project](https://www.a11yproject.com/checklist/) for accessibility - [DevTools Tips](https://devtoolstips.org/) - [67 Weird Debugging Tricks Your Browser Doesn't Want You to Know](https://alan.norbauer.com/articles/browser-debugging-tricks) - [Client-Side Architecture Basics](https://khalilstemmler.com/articles/client-side-architecture/introduction/) - [Web Browser Engineering](https://browser.engineering/index.html): this book explains how to build a basic but complete web browser, from networking to JavaScript, in a couple thousand lines of Python. - [Don't animate height!](https://www.granola.ai/blog/dont-animate-height) - [How modern browsers work](https://addyo.substack.com/p/how-modern-browsers-work) URLs: - [The Great Confusion About URIs](https://benbernardblog.com/the-great-confusion-about-uris/) - A URI is a string of characters that identifies a resource. Its syntax is `:?#`, where only `` and `` are mandatory. URL and URN are URIs. - A URL is a string of characters that identifies a resource located on a computer network. Its syntax depends on its scheme. E.g. `mailto:billg@microsoft.com`. - A URN is a string of characters that uniquely identifies a resource. Its syntax is `urn::`. E.g. `urn:isbn:9780062301239` - [Examples of Great URL Design](https://blog.jim-nielsen.com/2023/examples-of-great-urls/) - [Four Cool URLs - Alex Pounds' Blog](https://alexpounds.com/blog/2018/12/29/four-cool-urls) - [Your URL Is Your State](https://alfy.blog/2025/10/31/your-url-is-your-state.html) ### Writing (communication, blogging) ➡️ See also my [engineering-management list](https://github.com/charlax/engineering-management#writing) - [Undervalued Software Engineering Skills: Writing Well](https://blog.pragmaticengineer.com/on-writing-well/) - From the HN discussion: "Writing a couple of pages of design docs or an Amazon-style 6 pager or whatever might take a few days of work, but can save weeks or more of wasted implementation time when you realise your system design was flawed or it doesn't address any real user needs." - [Sell Yourself Sell Your Work](https://www.solipsys.co.uk/new/SellYourselfSellYourWork.html?te20hn) - If you've done great work, if you've produced superb software or fixed a fault with an aeroplane or investigated a problem, without telling anyone you may as well not have bothered. - [The Writing Well Handbook](https://www.julian.com/guide/write/intro) - Ideas — Identify what to write about - First Drafts — Generate insights on your topic - Rewriting — Rewrite for clarity, intrigue, and succinctness - Style — Rewrite for style and flow - Practicing — Improve as a writer - [Write Simply](http://paulgraham.com/simply.html), Paul Graham - [Writing is Thinking: Learning to Write with Confidence](https://blog.stephsmith.io/learning-to-write-with-confidence/) - [It's time to start writing](https://alexnixon.github.io/2019/12/10/writing.html) explains why Jeff Bezos banned PowerPoint at Amazon. - The reason writing a good 4 page memo is harder than "writing" a 20 page powerpoint is because the narrative structure of a good memo forces better thought and better understanding of what's more important than what, and how things are related. - Powerpoint-style presentations somehow give permission to gloss over ideas, flatten out any sense of relative importance, and ignore the interconnectedness of ideas. - [Programming and Writing](http://antirez.com/news/135), Antirez - [Writing one sentence per line](https://sive.rs/1s) - [Ask HN: How to level up your technical writing?](https://news.ycombinator.com/item?id=31859040). Lots of great resources. - [Patterns in confusing explanations](https://jvns.ca/blog/confusing-explanations/), Julia Evans - [Technical Writing for Developers](https://css-tricks.com/technical-writing-for-developers/) - [Some blogging myths](https://jvns.ca/blog/2023/06/05/some-blogging-myths/), Julia Evans - [George Orwell's Six Rules for Writing](https://infusion.media/blog/george-orwells-six-rules-for-writing/) - Never use a metaphor, simile, or other figure of speech which you are used to seeing in print. - Never use a long word where a short one will do. - If it is possible to cut a word out, always cut it out. - Never use the passive where you can use the active. - Never use a foreign phrase, a scientific word, or a jargon word if you can think of an everyday English equivalent. - Break any of these rules sooner than say anything outright barbarous. - [Blog Writing for Developers](https://rmoff.net/2023/07/19/blog-writing-for-developers/) - [7 Common Mistakes in Architecture Diagrams](https://www.ilograph.com/blog/posts/diagram-mistakes/) - [Why Blog If Nobody Reads It?](https://andysblog.uk/why-blog-if-nobody-reads-it/) - Blogging forces clarity. It makes you structure your thoughts, sharpen your perspective. - [Specificity: A weapon of mass effectiveness](https://longform.asmartbear.com/specificity/), A Smart Bear. - Swap generic words for specifics to make your text clear, powerful, engaging, and even funny. Guides & classes about technical writing: - [Documentation Guide — Write the Docs](https://www.writethedocs.org/guide/) - Principles - Style guides - Docs as code - Markup languages - Tools - [Technical Writing One introduction](https://developers.google.com/tech-writing/one), Google - Grammar - Active voice - Clear & short sentences ![Write like an Amazonian](./images/amazon_writing_rules.jpeg) > If you’re overthinking, write. If you’re underthinking, read. > – @AlexAndBooks_ ## Resources & inspiration for presentations - - - Dilbert - Calvin & Hobbes ([search engine](http://michaelyingling.com/random/calvin_and_hobbes/)) - ## Keeping up-to-date Website and RSS feeds (I use [Feedly](http://feedly.com/)): - [Hacker News](https://news.ycombinator.com/) ⭐️ - [VentureBeat](https://venturebeat.com/) - [High Scalability](http://highscalability.com/): see [above](#system-architecture) Security: - [Schneier on Security](https://www.schneier.com/) - [Krebs on Security](https://krebsonsecurity.com/) - [The Hacker News](https://thehackernews.com/) Newsletters: - [Bytes](https://bytes.dev/) (JavaScript) - [PyCoders](https://pycoders.com/) (Python) - [Posthog](https://newsletter.posthog.com/) - [Tech Talks Weekly](https://techtalksweekly.io/) Blogs: - [kilimchoi/engineering-blogs](https://github.com/kilimchoi/engineering-blogs) ## Concepts [Glossary](glossary.md) - [BDD](https://en.wikipedia.org/wiki/Behavior-driven_development) - [CAP theorem](https://en.wikipedia.org/wiki/CAP_theorem) - [DDD](https://en.wikipedia.org/wiki/Domain-driven_design) - [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) - [EAV](https://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80%93value_model) - [GRASP]() - [KISS](https://en.wikipedia.org/wiki/KISS_principle) - [Make it run, make it right, make it fast](http://c2.com/cgi/wiki?MakeItWorkMakeItRightMakeItFast) - [OOP](https://en.wikipedia.org/wiki/Object-oriented_programming) - [SOLID]() - [TDD](https://en.wikipedia.org/wiki/Test-driven_development) - [Two Generals' Problem](https://en.wikipedia.org/wiki/Two_Generals%27_Problem) - [YAGNI](https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it) ## My other lists - [engineering-management](https://github.com/charlax/engineering-management/) - [entrepreneurship-resources](https://github.com/charlax/entrepreneurship-resources) - [professional-programming](https://github.com/charlax/professional-programming) - [python-education](https://github.com/charlax/python-education) ================================================ FILE: antipatterns/README.md ================================================ ## Table of Contents - [Antipatterns](#antipatterns) # Antipatterns This is a list of antipatterns. - [Code antipatterns](./code-antipatterns.md) - [Python antipatterns](./python-antipatterns.md) - [SQLAlchemy antipatterns](./sqlalchemy-antipatterns.md) - [Error handling antipatterns](./error-handling-antipatterns.md) - [Tests antipatterns](./tests-antipatterns.md) ================================================ FILE: antipatterns/code-antipatterns.md ================================================ ## Table of Contents - [Antipatterns](#antipatterns) - [Strict email validation](#strict-email-validation) - [Late returns](#late-returns) - [Hacks comment](#hacks-comment) - [Repeating arguments in function name](#repeating-arguments-in-function-name) - [Repeating class name in method name](#repeating-class-name-in-method-name) - [Repeating function name in docstring](#repeating-function-name-in-docstring) - [Unreadable response construction](#unreadable-response-construction) - [Undeterministic tests](#undeterministic-tests) - [Unbalanced boilerplate](#unbalanced-boilerplate) - [Inconsistent use of verbs in functions](#inconsistent-use-of-verbs-in-functions) - [Opaque function arguments](#opaque-function-arguments) - [Hiding formatting](#hiding-formatting) - [Returning nothing instead of raising NotFound exception](#returning-nothing-instead-of-raising-notfound-exception) - [Having a library that contains all utils](#having-a-library-that-contains-all-utils) # Antipatterns Most of those are antipatterns in the Python programming language, but some of them might be more generic. ## Strict email validation It is almost impossible to strictly validate an email. Even if you were writing or using a regex that follows [RFC5322](http://tools.ietf.org/html/rfc5322#section-3.4), you would have false positives when trying to validate actual emails that don't follow the RFC. What's more, validating an email provides very weak guarantees. A stronger, more meaningful validation would be to send an email and validate that the user received it. To sum up, don't waste your time trying to validate an email if you don't need to (or just check that there's a `@` in it). If you need to, send an email with a token and validate that the user received it. ## Late returns Returning early reduces cognitive overhead, and improve readability by killing indentation levels. Bad: ```python def toast(bread): if bread.kind != 'brioche': if not bread.is_stale: toaster.toast(bread) ``` Good: ```python def toast(bread): if bread.kind == 'brioche' or bread.is_stale: return toaster.toast(bread) ``` ## Hacks comment Bad: ```python # Gigantic hack (written by Louis de Funes) 04-01-2015 toaster.restart() ``` There's multiple things wrong with this comment: - Even if it is actually a hack, no need to say it in a comment. It lowers the perceived quality of a codebase and impacts developer motivation. - Putting the author and the date is totally useless when using source control (`git blame`). - This does not explain why it's temporary. - It's impossible to easily grep for temporary fixes. - [Louis de Funès](https://en.wikipedia.org/wiki/Louis_de_Fun%C3%A8s) would never write a hack. Good: ```python # Need to restart toaster to prevent burning bread # TODO: replace with proper fix toaster.restart() ``` - This clearly explains the nature of the temporary fix. - Using `TODO` is an ubiquitous pattern that allows easy grepping and plays nice with most text editors. - The perceived quality of this temporary fix is much higher. ## Repeating arguments in function name Bad: ```python def get_by_color(color): return Toasters.filter_by(color=color) ``` Putting the argument name in both the function name and in arguments is, in most cases and for most interpreted languages, redundant. Good: ```python def get(color=None): if color: return Toasters.filter_by(color=color) ``` ## Repeating class name in method name Bad: ```python class Toasters(object): def get_toaster(self, toaster_id): pass ``` This is bad because it's unnecessarily redundant (`Toasters.get_toaster(1)`). According to the single responsibility principle, a class should focus on one area of responsibility. So the `Toasters` class should only focus on toasters object. Good: ```python class Toasters(object): def get(self, toaster_id): pass ``` Which produces much more concise code: ``` toaster = Toasters.get(1) ``` ## Repeating function name in docstring Bad: ```python def test_return_true_if_toast_is_valid(): """Verify that we return true if toast is valid.""" assert is_valid(Toast('brioche')) is true ``` Why is it bad? - The docstring and function name are not DRY. - There's no actual explanation of what valid means. Good: ```python def test_valid_toast(): """Verify that 'brioche' are valid toasts.""" assert is_valid(Toast('brioche')) is true ``` Or, another variation: ```python def test_brioche_are_valid_toast(): assert is_valid(Toast('brioche')) is true ``` ## Unreadable response construction TODO Bad: ```python def get_data(): returned = {} if stuff: returned['toaster'] = 'toaster' if other_stuff: if the_other_stuff: returned['toast'] = 'brioche' else: returned['toast'] = 'bread' return returned ``` Good: ```python def get_data(): returned = { 'toaster': '', 'toast': '', } ``` ## Undeterministic tests When testing function that don't behave deterministically, it can be tempting to run them multiple time and average their results. Bad: ```python def function(): if random.random() > .4: return True else: return False def test_function(): number_of_true = 0 for _ in xrange(1000): returned = function() if returned: number_of_true += 1 assert 30 < number_of_true < 50 ``` There are multiple things that are wrong with this approach: - This is a flaky test. Theoretically, this test could still fail. - This example is simple enough, but `function` might be doing some computationally expensive task, which would make this test severely inefficient. - The test is quite difficult to understand. Good: ```python @mock.patch('random.random') def test_function(mock_random): mock_random.return_value = 0.7 assert function() is True ``` This is a deterministic test that clearly tells what's going on. ## Unbalanced boilerplate One thing to strive for in libraries is have as little boilerplate as possible, but not less. Not enough boilerplate: you'll spend hours trying to understand specific behaviors that are too magical/implicit. You will need flexibility and you won't be able to get it. Boilerplate is useful insofar as it increases [transparency](http://www.catb.org/esr/writings/taoup/html/ch01s06.html). Too much boilerplate: users of your library will be stuck using outdated patterns. Users will write library to generate the boilerplate required by your library. I think Flask and SQLAlchemy do a very good job at keeping this under control. ## Inconsistent use of verbs in functions Bad: ```python def get_toasters(color): """Get a bunch of toasters.""" return filter(lambda toaster: toaster.color == color, TOASTERS) def find_toaster(id_): """Return a single toaster.""" toasters = filter(lambda toaster: toaster.id == id_, TOASTERS) assert len(toasters) == 1 return toasters[1] def find_toasts(color): """Find a bunch of toasts.""" return filter(lambda toast: toast.color == color, TOASTS) ``` The use of verb is inconsistent in this example. `get` is used to return a possibly empty list of toasters, and `find` is used to return a single toaster (or raise an exception) in the second function or a possibly empty list of toasts in the third function. This is based on personal taste but I have the following rule: - `get` prefixes function that return at most one object (they either return none or raise an exception depending on the cases) - `find` prefixes function that return a possibly empty list (or iterable) of objects. Good: ```python def find_toasters(color): """Find a bunch of toasters.""" return filter(lambda toaster: toaster.color == color, TOASTERS) def get_toaster(id_): """Return a single toaster.""" toasters = filter(lambda toaster: toaster.id == id_, TOASTERS) assert len(toasters) == 1 return toasters[1] def find_toasts(color): """Find a bunch of toasts.""" return filter(lambda toast: toast.color == color, TOASTS) ``` ## Opaque function arguments A few variants of what I consider code that is difficult to debug: ```python def create(toaster_params): name = toaster_params['name'] color = toaster_params.get('color', 'red') class Toaster(object): def __init__(self, params): self.name = params['name'] # Probably the worst of all def create2(*args, **kwargs): name = kwargs['name'] ``` Why is this bad? - It's really easy to make a mistake, especially in interpreted languages such as Python. For instance, if I call `create({'name': 'hello', 'ccolor': 'blue'})`, I won't get any error, but the color won't be the one I expect. - It increases cognitive load, as I have to understand where the object is coming from to introspect its content. - It makes the job of static analyzer harder or impossible. Granted, this pattern is sometimes required (for instance when the number of params is too large, or when dealing with pure data). A better way is to be explicit: ```python def create(name, color='red'): pass # ... ``` ## Hiding formatting Bad: ```python # main.py from utils import format_query def get_user(user_id): url = get_url(user_id) return requests.get(url) # utils.py def get_url(user_id): return 'http://127.0.0.1/users/%s' % user_id ``` I consider this an antipattern because it hides the request formatting from the developer, making it more complex to see what `url` look like. In this extreme example, the formatting function is a one-liner which sounds a bit overkill for a function. Good: ```python def get_user(user_id): url = 'http://127.0.0.1/users/%s' % user_id return requests.get(url) ``` Even if you were duplicating the logic once or twice it might still be fine, because: - You're unlikely to re-use anywhere else outside this file. - Putting this inline makes it easier for follow the flow. Code is written to be read primarily by computers. ## Returning nothing instead of raising NotFound exception Bad in certain cases: ```python def get_toaster(toaster_id): try: return do_get_toaster(toaster_id) except NotFound: return None def toast(toaster_id): toaster = get_toaster(toaster_id) ... toaster.toast("brioche") ``` It all depends on the caller, but in this cases I'd argue that it is bad practice to return nothing when the toaster identified by `toaster_id` can't be found, for two main reasons. **First reason**: when we provide an identifier, we expect it to return something. Once again, this depends on the caller (for instance, we could try to see if a user exists by checking an email for instance). In this simple example it's ok because the `toaster.toast()` will fail immediately, but what if we were never calling it and creating some other unrelated objects? We would be doing things that we should never be doing if the object did not exist: ```python def toast(toaster_id, user): toaster = get_toaster(toaster_id) # We should never do this! The toaster might not even exists. send_welcome_email(user) bill_new_toaster(user) ``` **Second reason**: `toaster.toast` will fail anyway if `toaster` is none (in Python with `AttributeError: NoneType has no attribute toast`). In this abstract example it's ok because the two lines are next to each other, but the actual `toaster.toast()` call might happen further down the stack - and it will be very difficult for the developer to understand where the error is coming from. ```python def toast(toaster_id, user): toaster = get_toaster(toaster_id) do_stuff_a(toaster) def do_stuff_a(toaster): ... do_stuff_b(toaster) ... def do_stuff_b(toaster): # Here is the actual call where toaster is called - but we should # have failed early if it's not there. toaster.toast() ``` What's the correct things to do? - If you expect the object to be there, make sure to raise if you don't find it. - If you're using SQLAlchemy, use `one()` to force raising an exception if the object can't be found. Don't use `first` or `one_or_none()`. ## Having a library that contains all utils Bad: ```python def get_current_date(): ... def create_csv(...): ... def upload_to_sftp(...): ... ``` `util` or `tools` or `lib` modules that contain all sorts of utilities have a tendency to become bloated and unmaintainable. Prefer to have small, dedicated files. This will keep your imports logical (`lib.date_utils`, `lib.csv_utils`, `lib.sftp`), make it easier for the reader to identify all the utilities around a specific topic, and test files easy to keep organized. ================================================ FILE: antipatterns/database-antipatterns.md ================================================ ## Table of Contents - [Database anti-patterns](#database-anti-patterns) - [Using `VARCHAR` instead of `TEXT` (PostgreSQL)](#using-varchar-instead-of-text-postgresql) # Database anti-patterns ## Using `VARCHAR` instead of `TEXT` (PostgreSQL) Unless you absolutely restrict the width of a text column for data consistency reason, don't do it. This [benchmark](http://www.depesz.com/2010/03/02/charx-vs-varcharx-vs-varchar-vs-text/) shows that there's fundamentally no difference in performance between `char(n)`, `varchar(n)`, `varchar` and `text`. Here's why you should pick `text`: - `char(n)`: takes more space than necessary when dealing with values shorter than n. - `varchar(n)`: it's difficult to change the width. - `varchar` is just like `text`. - `text` does not have the width problem that `char(n)` and `varchar(n)` and has a cleaner name than `varchar`. ================================================ FILE: antipatterns/error-handling-antipatterns.md ================================================ ## Table of Contents - [Error handling anti-patterns](#error-handling-anti-patterns) - [Hiding exceptions](#hiding-exceptions) - [Raising unrelated/unspecific exception](#raising-unrelatedunspecific-exception) - [Unconstrained defensive programming](#unconstrained-defensive-programming) - [Unnecessarily catching and re-raising exceptions](#unnecessarily-catching-and-re-raising-exceptions) # Error handling anti-patterns ## Hiding exceptions There are multiple variations of this anti-pattern: ```python # Silence all exceptions def toast(bread): try: toaster = Toaster() toaster.insert(bread) toaster.toast() except: pass # Silence some exceptions def toast(bread): try: toaster = Toaster() toaster.insert(bread) toaster.toast() except ValueError: pass ``` It depends on the context but in most cases this is a bad pattern: - **Debugging** those silent errors will be really difficult, because they won't show up in logs and exception reporting tool such as Sentry. Say you have an undefined variable in `Toaster.insert()`: it will raise `NameError`, which will be caught, and ignored, and you will never know about this developer error. - **The user experience** will randomly degrade without anybody knowing about it, including the user. - **Identifying** those errors will be impossible. Say `do_stuff` does an HTTP request to another service, and that service starts misbehaving. There won't be any exception, any metric that will let you identify it. An article even named this [the most diabolical Python antipattern](https://realpython.com/blog/python/the-most-diabolical-python-antipattern/). The following full example: ```python from collections import namedtuple Bread = namedtuple('Bread', 'color') class ToastException(Exception): pass def toast(bread): try: put_in_toaster(bread) except: raise ToastException('Could not toast bread') def put_in_toaster(bread): brad.color = 'light_brown' # Note the typo toast(Bread('yellow')) ``` Will raise this cryptic and impossible to debug error: ``` Traceback (most recent call last): File "python-examples/reraise_exceptions.py", line 19, in toast(Bread('yellow')) File "python-examples/reraise_exceptions.py", line 12, in toast raise ToastException('Could not toast bread') __main__.ToastException: Could not toast bread ``` Sometime it's tempting to think that graceful degradation is about silencing exception. It's not. - Graceful degradation needs to happen at the **highest level** of the code, so that the user can get a very explicit error message (e.g. "we're having issues with X, please retry in a moment"). That requires knowing that there was an error, which you can't tell if you're silencing the exception. - You need to know when graceful degradation happens. You also need to be alerted if it happens too often. This requires adding monitoring (using something like statsd) and logging (Python's `logger.exception` automatically adds the exception stacktrace to the log message for instance). Silencing an exception won't make the error go away: all things being equal, it's better for something to break hard, than for an error to be silenced. - It is tempting to confound silencing the exception and fixing the exception. Say you're getting sporadic timeouts from a service. You might thing: let's ignore those timeouts and just do something else, like return an empty response. But this is very different from (1) actually finding the root cause for those timeouts (e.g. maybe a specific edge cases impacting certain objects) (2) doing proper graceful degradation (e.g. asking users to retry later because the request failed). In other words, ask yourself: would it be a problem if every single action was failing? If you're silencing the error, how would you know it's happening for every single action? Here's a number a better ways to do this: **Log and create metrics** ```python import statsd def toast(bread): # Note: no exception handling here. toaster = Toaster() toaster.insert(bread) toaster.toast() def main(): try: toast('brioche') except: logger.exception('Could not toast bread') statsd.count('toast.error', 1) ``` **Very important**: adding logging and metrics won't not sufficient if it's difficult to consume them. They won't help the developer who's debugging. There needs to be automatic alerting associated to those stats. The logs needs to be surfaced somewhere, ideally next to the higher exception (e.g. let's our `main` function above is used in a web interface - the web interface could say "additionally, the following logs were generated and display the log). For instance, Sentry can be configured to surface `logger.exception` errors the same way exception are surfaced. Otherwise the developer will still have to read the code to understand what's going on. Also - this won't work with sporadic errors. Those needs to be dealt with properly, and until then, it's better to let them go to your usual alerting tool. Note that knowing where to catch is very important too. If you're catching inside the `toast` function, you might be hiding things a caller would need to know. Since this function is not returning anything, how would you make the difference between a success and a failure? You can't. That's why you want to let it raise, and catch only in the caller, where you have the context to know how you'll handle the exception. **Re-raise immediately** ```python import statsd def toast(bread): try: toaster = Toaster() toaster.insert(bread) toaster.toast() except: raise ToastingException('Could not toast bread %r' % bread) def main(): # Note: no exception handling here toast('brioche') ``` **Be very specific about the exception that are caught** ```python import statsd def toast(bread): try: toaster = Toaster() except ValueError: # Note that even though we catch, we're still logging + creating # a metric logger.exception('Could not get toaster') statsd.count('toaster.instantiate.error', 1) return toaster.insert(bread) toaster.toast() def main(): toast('brioche') ``` ## Raising unrelated/unspecific exception Most languages have predefined exceptions, including Python. It is important to make sure that the right exception is raised from a semantic standpoint. Bad: ```python def validate(toast): if isinstance(toast, Brioche): # RuntimeError is too broad raise RuntimeError('Invalid toast') def validate(toast): if isinstance(toast, Brioche): # SystemError should only be used for internal interpreter errors raise SystemError('Invalid toast') ``` Good: ```python def validate(toast): if isinstance(toast, Brioche): raise TypeError('Invalid toast') ``` `TypeError` is here perfectly meaningful, and clearly convey the context around the error. ## Unconstrained defensive programming While defensive programming can be a very good technique to make the code more resilient, it can seriously backfire when misused. This is a very similar anti-pattern to carelessly silencing exceptions (see about this anti-pattern in this document). One example is to handle an edge case as a generic case at a very low level. Consider the following example: ```python def get_user_name(user_id): url = 'http://127.0.0.1/users/%s' % user_id response = requests.get(url) if response.status == 404: return 'unknown' return response.data ``` While this may look like a very good example of defensive programming (we're returning `unknown` when we can't find the user), this can have terrible repercussions, very similar to the one we have when doing an unrestricted bare `try... except`: - A new developer might not know about this magical convention, and assume that `get_user_name` is guaranteed to return a true user name. - The external service that we're getting user name from might start failing, and returning 404. We would silently return 'unknown' as a user name for all users, which could have terrible repercussions. A much cleaner way is to raise an exception on 404, and let the caller decide how it wants to handle users that are not found. ## Unnecessarily catching and re-raising exceptions Bad: ```python def toast(bread): try: put_in_toaster(bread) except InvalidBreadException: raise ToastException('Could not toast') ``` Side note: an unconditional exception catching is considered even worse (see [hiding exceptions](https://github.com/charlax/antipatterns/blob/master/code-antipatterns.md#hiding-exceptions)). This is a better pattern because we explicitly state what happened in the exception message: ```python def toast(bread): try: put_in_toaster(bread) except InvalidBreadType as e: raise ToastException('Cannot toast this bread type') ``` If we need to do some cleanup or extra logging, it's better to just raise the original exception again. The developer will know exactly what happened. ```python def toast(bread): try: put_in_toaster(bread) except: print 'Got exception while trying to toast' raise # note the absence of specific exception ``` Here's what would happen: ``` Got exception while trying to toast Traceback (most recent call last): File "reraise_exceptions_good.py", line 20, in toast(Bread('yellow')) File "reraise_exceptions_good.py", line 10, in toast put_in_toaster(bread) File "reraise_exceptions_good.py", line 17, in put_in_toaster brad.color = 'light_brown' # Note the typo NameError: global name 'brad' is not defined ``` Another way to show how absurd the anti-pattern becomes at scale is through an example: ```python def call_1(): try: call_2() except Call2Exception: raise Call1Exception() def call_2(): try: call_3() except Call3Exception: raise Call2Exception() def call_3(): ... ``` Another problem with this pattern is that you can consider it quite useless to do all of this catch/re-raise, since you will still need to catch at the end. In other words: > Error handling, and recovery are best done at the outer layers of your code base. This is known as the end-to-end principle. The end-to-end principle argues that it is easier to handle failure at the far ends of a connection than anywhere in the middle. If you have any handling inside, you still have to do the final top level check. If every layer atop must handle errors, so why bother handling them on the inside? _[Write code that is easy to delete, not easy to extend](http://programmingisterrible.com/post/139222674273/write-code-that-is-easy-to-delete-not-easy-to)_ A better way: ```python # This is the highest level function where we have enough # context to know how to handle the exceptions def call_1(): try: call_2() except Call2Exception: # handle it... pass except Call3Exception: # handle it... pass def call_2(): # Do not handle anything here. call_3() def call_3(): ... ``` More resources: - [Hiding exceptions](https://github.com/charlax/antipatterns/blob/master/code-antipatterns.md#hiding-exceptions)) anti-pattern. ================================================ FILE: antipatterns/mvcs-antipatterns.md ================================================ ## Table of Contents - [MVCS Antipatterns](#mvcs-antipatterns) - [Creating entities for association tables](#creating-entities-for-association-tables) # MVCS Antipatterns In simple terms, Model-View-Controller-Services add a few more layers to the MVC pattern. The main one is the service, which owns all the core business logic and manipulate the repository layer. ## Creating entities for association tables You'll often need association tables, for instance to set up a many to many relationships between users and their toasters. Let's assume that a toaster can be owned by multiple users. It might be tempting to create a `UserToaster` entity for this relationship, especially if this relationship has some complex attributes associated with (for instance, since when the toaster is owned by the user). Let me pull a few quotes from the [Domain Driven Design](http://www.amazon.com/Domain-Driven-Design-Tackling-Complexity-Software/dp/0321125215) by Eric Evans: > Design a portion of the software system to reflect the domain model in a very > literal way, so that mapping is obvious. > Object-oriented programming is powerful because it is based on a modeling > paradigm, and it provides implementations of the model constructs. As far as > the programmer is concerned, objects really exist in memory, they have > associations with other objects, they are organized into classes, and they > provide behavior available by messaging. > Does an object represent something with continuity and identity— something > that is tracked through different states or even across different > implementations? Or is it an attribute that describes the state of something > else? This is the basic distinction between an ENTITY and a VALUE OBJECT. > Defining objects that clearly follow one pattern or the other makes the > objects less ambiguous and lays out the path toward specific choices for > robust design. Evans, Eric (2003-08-22). Domain-Driven Design: Tackling Complexity in the Heart of Software. Pearson Education. Kindle Edition. Entities should model business processes, not persistence details ([source](http://blog.sapiensworks.com/post/2013/05/13/7-Biggest-Pitfalls-When-Doing-Domain-Driven-Design.aspx/)). - In that case, `UserToaster` does not map to anything the business is using. Using plain English, somebody might ask about "what toasters does user A owns?" or "who owns toaster B and since when?" Nobody would ask "give me the UserToaster for user A". - The association table can be considered an implementation detail that should not (in most cases) leak in the domain layer. All the code should be dealing with the simpler logic of "user having toasters", not UserToaster objects being an association between a user and a toaster. This makes the code more intuitive and natural. - It will be easier to handle serializing a "user having toasters" than serializing UserToaster association. - This will make it very easy to force the calling site to take care of some business logic. For instance, you might be able to get all `UserToaster`, and then filter on whether they were bought. You might be tempted to do that by going through the `UserToaster` object and filtering those that have `were_bought` to be True. At some point, you might be doing the same thing in multiple places, which will decrease maintainability. If you were hiding that logic in the repository, you wouldn't have that issue `find_bought_toasters`. So in that case, I would recommend doing the following: - Create a `User` and `Toaster` entity. - Put the association properties on the entity that makes sense, for instance `owned_since` would be on `Toaster`, even though in the database it's stored on the association table. - If filtering on association properties is required, put this logic in repositories. In plain English, you would for instance ask "give all the toasters user A owned in December?", you wouldn't ask "give be all the UserToaster for owned by user A in December". Note that Domain Driver Design recommends avoiding many-to-many relationships altogether: > In real life, there are lots of many-to-many associations, and a great number > are naturally bidirectional. The same tends to be true of early forms of > a model as we brainstorm and explore the domain. But these general > associations complicate implementation and maintenance. Furthermore, they > communicate very little about the nature of the relationship. > There are at least three ways of making associations more tractable. > 1. Imposing a traversal direction > 2. Adding a qualifier, effectively reducing multiplicity > 3. Eliminating nonessential associations Evans, Eric (2003-08-22). Domain-Driven Design: Tackling Complexity in the Heart of Software (Kindle Locations 1642-1647). Pearson Education. Kindle Edition. Imposing a traversal direction is probably the best solution. In our example, it's not immediately evident which direction is the right one (a toaster being owned by a user, or a user owning a toasters), because that depends on what this application is doing. If we're working on an app that lets a connected user see their toasters, then we would force the relationship to be unidirectional user->toasters. Sources: - [7 Biggest Pitfalls When Doing Domain Driven Design](http://blog.sapiensworks.com/post/2013/05/13/7-Biggest-Pitfalls-When-Doing-Domain-Driven-Design.aspx/) - [Domain-Driven Design: Tackling Complexity in the Heart of Software](http://www.amazon.com/Domain-Driven-Design-Tackling-Complexity-Software/dp/0321125215) ================================================ FILE: antipatterns/python-examples/reraise_exceptions_bad.py ================================================ from collections import namedtuple Bread = namedtuple("Bread", "color") class ToastException(Exception): pass def toast(bread): try: put_in_toaster(bread) except: raise ToastException("Could not toast bread") def put_in_toaster(bread): brad.color = "light_brown" # Note the typo toast(Bread("yellow")) ================================================ FILE: antipatterns/python-examples/reraise_exceptions_good.py ================================================ from collections import namedtuple Bread = namedtuple("Bread", "color") class ToastException(Exception): pass def toast(bread): try: put_in_toaster(bread) except: print("Got exception while trying to toast") raise def put_in_toaster(bread): brad.color = "light_brown" # Note the typo toast(Bread("yellow")) ================================================ FILE: antipatterns/scalability-antipatterns.md ================================================ ## Table of Contents - [Not paging a database query](#not-paging-a-database-query) - [Returning whole objects where primary key would suffice](#returning-whole-objects-where-primary-key-would-suffice) ## Not paging a database query TODO ## Returning whole objects where primary key would suffice TODO ================================================ FILE: antipatterns/sqlalchemy-antipatterns.md ================================================ ## Table of Contents - [SQLAlchemy Anti-Patterns](#sqlalchemy-anti-patterns) - [Abusing lazily loaded relationships](#abusing-lazily-loaded-relationships) - [Explicit session passing](#explicit-session-passing) - [Implicit transaction handling](#implicit-transaction-handling) - [Loading the full object when checking for object existence](#loading-the-full-object-when-checking-for-object-existence) - [Using identity as comparator](#using-identity-as-comparator) - [Returning `None` instead of raising a `NoResultFound` exception](#returning-none-instead-of-raising-a-noresultfound-exception) # SQLAlchemy Anti-Patterns This is a list of what I consider [SQLAlchemy](http://www.sqlalchemy.org/) anti-patterns. ## Abusing lazily loaded relationships Bad: ```python class Customer(Base): @property def has_valid_toast(self): """Return True if customer has at least one valid toast.""" return any(toast.kind == 'brioche' for toast in self.toaster.toasts) ``` This suffers from severe performance inefficiencies: - The toaster will be loaded, as well as its toast. This involves creating and issuing the SQL query, waiting for the database to return, and instantiating all those objects. - `has_valid_toast` does not actually care about those objects. It just returns a boolean. A better way would be to issue a SQL `EXISTS` query so that the database handles this check and only returns a boolean. Good: ```python class Customer(Base): @property def has_valid_toast(self): """Return True if customer has at least one valid toast.""" query = (session.query(Toaster) .join(Toast) .with_parent(self) .filter(Toast.kind == 'brioche')) return session.query(query.exists()).scalar() ``` This query might not always be the fastest if those relationships are small, and eagerly loaded. ## Explicit session passing TODO Bad: ```python def toaster_exists(toaster_id, session): ... ``` ## Implicit transaction handling TODO ## Loading the full object when checking for object existence Bad: ```python def toaster_exists(toaster_id): return bool(session.query(Toaster).filter_by(id=toaster_id).first()) ``` This is inefficient because it: - Queries all the columns from the database (including any eagerly loaded joins) - Instantiates and maps all data on the Toaster model The database query would look something like this. You can see that all columns are selected to be loaded by the ORM. ```sql SELECT toasters.id AS toasters_id, toasters.name AS toasters_name, toasters.color AS toasters_color FROM toasters WHERE toasters.id = 1 LIMIT 1 OFFSET 0 ``` And then it just checks if the result is truthy. Here's a better way to do it: ```python def toaster_exists(toaster_id): query = session.query(Toaster).filter_by(id=toaster_id) return session.query(query.exists()).scalar() ``` In this case, we just ask the database about whether a record exists with this id. This is obviously much more efficient. ```sql SELECT EXISTS (SELECT 1 FROM toasters WHERE toasters.id = 1) AS anon_1 ``` ## Using identity as comparator Bad: ```python toasters = session.query(Toaster).filter(Toaster.deleted_at is None).all() ``` Unfortunately this won't work at all. This query will return all toasters, including the one that were deleted. The way sqlalchemy works is that it overrides the magic comparison methods (`__eq__`, `__lt__`, etc.). All comparison methods can be overrode except the identity operator (`is`) which checks for objects identity. What this means is that expression `Toaster.deleted_at is None` will be immediately evaluated by the Python interpreter, and since (presumably) `Toaster.deleted_at` is a `sqlalchemy.orm.attributes.InstrumentedAttribute`, it's not `None` and thus it's equivalent to doing: ```python toasters = session.query(Toaster).filter(True).all() ``` Which obviously renders the filter inoperable, and will return all records. There's two ways to fix it: ```python toasters = session.query(Toaster).filter(Toaster.deleted_at == None).all() ``` Here we use the equality operator, which Python allows overriding. Behind the scene, Python calls `Toaster.deleted_at.__eq__(None)`, which gives SQLAlchemy the opportunity to return a comparator that when coerced to a string, will evaluate to `deleted_at is NULL`. Most linter will issue a warning for equality comparison against `None`, so you can also do (this is my preferred solution): ```python toasters = session.query(Toaster).filter(Toaster.deleted_at.is_(None)).all() ``` See docs for [is\_](http://docs.sqlalchemy.org/en/rel_1_0/core/sqlelement.html#sqlalchemy.sql.operators.ColumnOperators.is_). ## Returning `None` instead of raising a `NoResultFound` exception See [Returning nothing instead of raising NotFound exception](https://github.com/charlax/antipatterns/blob/master/code-antipatterns.md#returning-nothing-instead-of-raising-notfound-exception). ================================================ FILE: antipatterns/sqlalchemy-examples/exists.py ================================================ from sqlalchemy import create_engine from sqlalchemy import Column, Integer, String from sqlalchemy.orm import sessionmaker from sqlalchemy.ext.declarative import declarative_base engine = create_engine("sqlite:///:memory:", echo=True) Session = sessionmaker(bind=engine) Base = declarative_base() class Toaster(Base): __tablename__ = "toasters" id = Column(Integer, primary_key=True) name = Column(String) color = Column(String) def toaster_exists_bad(toaster_id): session = Session() return bool(session.query(Toaster).filter_by(id=toaster_id).first()) def toaster_exists_good(toaster_id): session = Session() query = session.query(Toaster).filter_by(id=toaster_id) return session.query(query.exists()).scalar() def main(): Base.metadata.create_all(engine) toaster_exists_bad(1) toaster_exists_good(2) if __name__ == "__main__": main() ================================================ FILE: antipatterns/tests-antipatterns.md ================================================ ## Table of Contents - [Test antipatterns](#test-antipatterns) - [Testing implementation](#testing-implementation) - [Testing configuration](#testing-configuration) - [Testing multiple things](#testing-multiple-things) - [Repeating integration tests for minor variations](#repeating-integration-tests-for-minor-variations) - [Over-reliance on centralized fixtures](#over-reliance-on-centralized-fixtures) - [Over-reliance on replaying external requests](#over-reliance-on-replaying-external-requests) - [Inefficient query testing](#inefficient-query-testing) - [Assertions in loop](#assertions-in-loop) - [Inverted testing pyramid](#inverted-testing-pyramid) # Test antipatterns ## Testing implementation TODO ## Testing configuration TODO ## Testing multiple things TODO ## Repeating integration tests for minor variations TODO ## Over-reliance on centralized fixtures Bad: ```python # fixtures.py toaster = Toaster(color='black') toaster_with_color_blue = Toaster(color='blue') toaster_with_color_red = Toaster(color='red') # test.py from fixtures import toaster, toaster_with_color_blue def test_stuff(): toaster_with_color_blue.toast('brioche') ``` The problem with centralized fixtures is that they tend to grow exponentially. Every single test case will have some specific fixtures requirements, and every single permutation will have its own fixture. This will make the fixture file really difficult to maintain. The other problem is that it will become very easy for two unrelated tests to use the same fixture. Now let's say one of those test's requirement changes, and you have to change the fixture as well. In that case, you might break the other test, which will slow you down and defeats the purpose of keeping test an efficient part of the developer flow. Lastly, this separate the setup and running part of the tests. It makes it more difficult for a new engineer to understand what is specific about this test's setup without having to open the `fixtures` file. Here's a more explicit way to do this. Most fixtures libraries allow you to override default parameters, so that you can make clear what setup is specific to each test. ```python def test_stuff(): toaster_with_color_blue = Toaster(color='blue') toaster_with_color_blue.toast('brioche') ``` ## Over-reliance on replaying external requests TODO ## Inefficient query testing Bad: ```python def test_find(): install_fixture('toaster_1') # color is red install_fixture('toaster_2') # color is blue install_fixture('toaster_3') # color is blue install_fixture('toaster_4') # color is green results = find(color='blue') assert len(results) == 2 for r in results: assert r.color == 'blue' ``` This is inefficient because we're installing four fixtures, even though we could probably get away with only one. In most cases this would achieve the same thing at a much lower cost: ```python def test_find(): install_fixture('toaster_2') # color is blue results = find(color='blue') # Note the implicit assertion that the list is not empty because we're # accessing its first item. assert results[0].color == 'blue' ``` The general rule here is that tests should require the minimum amount of code to verify the behavior. One would also recommend to not do this kind of integration testing for queries going to the database, but sometimes it's a good tradeoff. ## Assertions in loop Bad: ```python def test_find(): results = find(color='blue') for r in results: assert r.color == 'blue' ``` This is bad because in most languages, if the list is empty then the test will pass, which is probably not what we want to test. Good: ```python def test_find(): results = find(color='blue') assert len(results) == 5 for r in results: assert r.color == 'blue' ``` This is also a matter of tradeoff, but in most cases I'd try to make sure the list contains only one item. If we're checking more than one item, that hints at our test trying to do too many things. ## Inverted testing pyramid ![Test Pyramid](/images/test-pyramid.png) _The [test pyramid](https://martinfowler.com/bliki/TestPyramid.html). Image courtesy of Martin Fowler._ Building lots of automated comprehensive end-to-end tests was tried multiple time, and almost never worked. - End to end tests try to do too many things, are highly indeterministic and as a result very flakey. - Debugging end to end tests failure is painful and slow - this is usually where most of the time is wasted. - Building and maintaining e2e tests is very costly. - The time to run e2e tests is much much longer than unit tests. Focused testing (e.g. unit, component, etc) prior to roll out, and business monitoring with alerting are much more efficient. More reading on this topic: - [End-To-End Testing Considered Harmful](http://www.alwaysagileconsulting.com/articles/end-to-end-testing-considered-harmful/), Always Agile Consulting - [Just Say No to More End-to-End Tests](https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html), Google Testing Blog - [Testing Strategies in a Microservice Architecture](https://martinfowler.com/articles/microservice-testing/#testing-end-to-end-tips), section titled "Writing and maintaining end-to-end tests can be very difficult", Toby Clemson, MartinFowler.com - [Introducing the software testing ice-cream cone (anti-pattern)](https://watirmelon.blog/2012/01/31/introducing-the-software-testing-ice-cream-cone/), Alister Scott - [TestPyramid](https://martinfowler.com/bliki/TestPyramid.html), Martin Fowler - [professional-programming's testing section](https://github.com/charlax/professional-programming#testing) ================================================ FILE: feature-flags.md ================================================ ## Table of Contents - [Why use feature flags?](#why-use-feature-flags) - [Should feature flags be used for everything?](#should-feature-flags-be-used-for-everything) - [References](#references) # Why use feature flags? - **Velocity**: coupled with a system to rapidly deploy configuration change, it is usually much faster to disable new code by turning off a feature than re-deploying all the code. This extra safety net helps developers be more confident in their release, because they know they can roll back a change very rapidly in case of error. - **Testing**: the presence of a feature flag forces the feature owner to test the flow. Without feature flags, the developer might deploy the change and assume the absence of errors means the release was successful. Yet there's numerous failure mode that don't raise explicit errors. - **Code iterations**: because code can be kept hidden behind a feature flag until it's ready to go live, developers can push smaller code changes that are not fully integrated yet. Smaller pull requests ease the job of code reviewers, make testing easier, and reduce the probability of a catastrophic failure. - **Gradual rollout**: feature flags enable gradual rollout, where a piece of code is gradually activated, for instance on a per city basis, or on a per user group basis. This builds confidence in the feature release process, and allows the engineer to verify that the new implementation is actually better (for instance, when coupled with A/B testing frameworks). ## Should feature flags be used for everything? I don't think so. I think it's a matter of good judgment. Just like 100% test coverage does not always make sense (provided lines that are not tested are explicitly ignored), adding or not adding a feature flag is an engineering decision. When would I not use a feature flag? - Simple changes: copy, logging, etc. - When rolling back takes a few seconds - Feature that is used only in asynchronous jobs that are safe to retry and don't impact the user experience. When should a feature flag be used? - Large refactors - Changing integration points - Performance optimization - New flows ## References - Martin Fowler, [FeatureToggle](http://martinfowler.com/bliki/FeatureToggle.html) - Flickr, [Flipping Out](http://code.flickr.net/2009/12/02/flipping-out/): one of the first articles on the topic. - [Using Feature Flags to Ship Changes with Confidence](http://blog.travis-ci.com/2014-03-04-use-feature-flags-to-ship-changes-with-confidence/) ================================================ FILE: glossary.md ================================================ ## Table of Contents - [Glossary](#glossary) - [Second system effect](#second-system-effect) # Glossary - **Blackbox monitoring** - **Conceptual integrity**: "It is better to have a system omit certain anomalous features and improvements, but to reflect one set of design ideas, than to have one that contains many good but independent and uncoordinated ideas." ([Fred Brooks](http://wiki.c2.com/?ConceptualIntegrity)) - **End-to-end principle**: "application-specific features reside in the communicating end nodes of the network, rather than in intermediary nodes" ([end-to-end principle - Wikipedia](https://en.wikipedia.org/wiki/End-to-end_principle)) - **[NIHITO](http://pragmaticmarketing.com/resources/use-the-market-to-gain-credibility)** (Nothing Important Happens In The Office): you need to learn from your customers and from the market first. - **Stability** is the sensitivity to change of a given system that is the negative impact that may be caused by system changes ([ISO/IEC 9126 standard on evaluating software quality](https://en.wikipedia.org/wiki/ISO/IEC_9126)). - **Reliability** is made of maturity (frequency of failure in software), fault tolerance (ability to withstand failure) and recoverability (ability to bring back a failed system to full operation) ([ISO/IEC 9126 standard on evaluating software quality](https://en.wikipedia.org/wiki/ISO/IEC_9126)). - **Two Generals' Problem**: "a thought experiment meant to illustrate the pitfalls and design challenges of attempting to coordinate an action by communicating over an unreliable link" ([wikipedia](https://en.wikipedia.org/wiki/Two_Generals%27_Problem)). - **Whitebox monitoring** - **Law of Demeter**: only talk to your immediate friends ([wikipedia](https://en.wikipedia.org/wiki/Law_of_Demeter)) ### Second system effect "The second-system effect (also known as second-system syndrome) is the tendency of small, elegant, and successful systems to be plagued with feature creep due to inflated expectations." ([wikipedia](https://en.wikipedia.org/wiki/Second-system_effect)) ================================================ FILE: list-of-links/List of AI and LLM learning resources 20260104144118.md ================================================ --- title: List of AI and LLM learning resources date: 2026-01-04T14:41:00-05:00 id: 20260104144118 --- # List of AI and LLM learning resources ## Build from scratch - [Neural Networks: Zero To Hero](https://karpathy.ai/zero-to-hero.html), Andrej Karpathy. Building neural networks, from scratch, in code. ================================================ FILE: training/front-end/01-modern-javascript.md ================================================ ## Table of Contents - [Modern JavaScript](#modern-javascript) - [Objectives of this course](#objectives-of-this-course) - [Prerequisites](#prerequisites) - [Check your knowledge of JS](#check-your-knowledge-of-js) - [Introduction](#introduction) - [Quirks](#quirks) - [Make sure your target browser supports the feature!](#make-sure-your-target-browser-supports-the-feature) - [Undefined everywhere!](#undefined-everywhere) - [Printing and interacting with the console](#printing-and-interacting-with-the-console) - [Casting](#casting) - [Always use triple comparators (`===`) instead of double (`==`)](#always-use-triple-comparators--instead-of-double-) - [Primitive types vs. reference types](#primitive-types-vs-reference-types) - [`Object` and `Array` methods](#object-and-array-methods) - [Prototypes in JavaScript](#prototypes-in-javascript) - [Object literals, assignment and destructuring](#object-literals-assignment-and-destructuring) - [Objects](#objects) - [Array](#array) - [`let` and `const`](#let-and-const) - [Hoisting](#hoisting) - [Arrow functions](#arrow-functions) - [How `this` works in arrow functions](#how-this-works-in-arrow-functions) - [Best practices](#best-practices) - [Classes](#classes) - [Template literals](#template-literals) - [Template tags](#template-tags) - [Loops](#loops) - [`for... of`](#for-of) - [Promises](#promises) - [Creating a promise](#creating-a-promise) - [Consuming a promise](#consuming-a-promise) - [Chaining promises](#chaining-promises) - [Async functions](#async-functions) - [Modules](#modules) - [Imports](#imports) - [Exports](#exports) - [Other features](#other-features) - [Optional chaining](#optional-chaining) - [Ternary operator](#ternary-operator) - [Self assessment](#self-assessment) - [References](#references) # Modern JavaScript Note: run code quickly with https://codesandbox.io/s/ - JavaScript: https://codesandbox.io/s/front-end-training-014br ## Objectives of this course - Review the syntax of modern JavaScript features - Mention some JS best practices ## Prerequisites This course assumes you already have experience with JavaScript. If you don't, start with this: - [JavaScript First Steps](https://developer.mozilla.org/en-US/docs/Learn/JavaScript/First_steps), MDN - [Learn JavaScript](https://learnjavascript.online/) ### Check your knowledge of JS - What are the main JS datatypes? - `Number`, `String`, `Boolean`, `Object`, `Undefined`, `BigInt` - What does `1 / "a"` evaluate to? - `NaN` ## Introduction JavaScript is a programming language evolving very rapidly that can run in different environments: - Browser (the assumed target for this document). - Computer and embedded systems (usually through node) - Mobile and desktop apps with frameworks such as Electron, React Native, etc. Key historical elements: - September 1995: LiveScript is released in the Netscape Navigator Browser. It was developed by Brendan Eich. - June 1997: Ecma International releases the first ECMAScript language _specification_. (note: the European Computer Manufacturers Association was founded in 1961 to standardize computer systems) - December 2009: the ECMAScript 5 standard is released. From 2016 to today, a new version of ECMAScript is released each year. The language has reached maturity and supports many modern constructs. Key features of JavaScript: - It is **imperative** & inspired by C - `if`, `while`, `switch`, `while`, `do while` - It is **weakly typed**: it has types (`String`, `Number`, etc.) but uses implicit cast (`"1" - "1" === 0`). - It is **dynamically typed**: types are associated with values rather than expressions (a variable `x` can be associated with a `String`, then with a `Number`). - It supports **runtime evaluation** with `eval`. `eval('1 === 1')` evaluates to `true` - It supports **object orientation** with a very powerful prototype-based approach. - While it is not necessarily considered a pure functional language, its **functions are first-class citizen**. It supports closures and anonymous functions. As a result, it can be used to go pretty far with functional programming patterns. - It is very **concise**. Newer features such as arrow functions, object and array destructuring leads to very terse code with a very high signal to noise ration. ## Quirks ### Make sure your target browser supports the feature! Use [caniuse](https://caniuse.com/) to check what browser support the feature you're using. For instance, for `[1, 2].includes(1)` requires `Array.prototype.includes`: ![caniuse](./img/caniuse.png) You can then use "polyfills" (or "shims") to support older browser. There are some polyfills for each specific feature, and some other that includes lots of polyfills (e.g. [zloirock/core-js](https://github.com/zloirock/core-js#ecmascript-array)). Polyfills are different from transpiling. Babel is a transpiler, you can see how it work online here: For instance, it will transpile (notice the replacement of an arrow function): ```javascript [1, 2, 3].map((n) => n + 1) ``` Into: ```javascript "use strict"; [1, 2, 3].map(function (n) { return n + 1; }); ``` ### Undefined everywhere! ```javascript var a; // an uninitialized variable is undefined console.assert(typeof a === "undefined"); // There are no required arguments in JavaScript function hello(name) { return name; } // No raise, will log "undefined" console.log(hello()); // Here's how to compare to undefined console.assert(typeof undefined === "undefined"); const anObject = { a: 1 }; // Accessing an absent object key also returns undefined console.assert(typeof anObject.nonExistent === "undefined"); ``` ### Printing and interacting with the console ```javascript // Do not leave console.log in your code! // There are linters such as eslint that will check for their absence console.log("hello"); // In this document, we use assert to show the actual value console.assert(true === true); ``` ### Casting Rules for string conversion: - `String` are left as is. - `Number` are converted to their string representation. - Elements of `Array` are converted to string, then joined with commas `,`. - Objects are converted to `[object Object]` where `Object` is the constructor of the object. Can you guess how those will be converted? ```javascript [] + [] {} + {} "1" + 1 "1" + "1" "1" - 1 [1, 2] + 2 [1] + 1 function Dog {} const dog = new Dog() dog + 1 + "a" dog - 1 ``` Other gotchas: ```javascript // NaN is fun! console.assert(typeof NaN === "number"); Object.is(NaN, NaN) // true NaN === NaN // false ``` A good talk on the topic: [Wat](https://www.destroyallsoftware.com/talks/wat) #### Always use triple comparators (`===`) instead of double (`==`) ```javascript // Double equals will coerce values to make them comparable! console.assert("1" == 1); // Better console.assert(!("1" === 1)); console.assert("1" !== 1); ``` ### Primitive types vs. reference types Applied on arrays and objects, `==` and `===` will check for object identity, which is almost never what you want. ```javascript console.assert({ a: 1 } != { a: 1 }); console.assert({ a: 1 } !== { a: 1 }); const obj = { a: 1 }; const obj2 = obj; // This is true because obj and obj2 refer to the same object ("identity") console.assert(obj == obj2); console.assert(obj === obj2); ``` Use a library such as [lodash](https://lodash.com/) to properly compare objects and array ```javascript import _ from "lodash"; console.assert(_.isEqual({ a: 1 }, { a: 1 })); console.assert(_.isEqual([1, 2], [1, 2])); ``` ### `Object` and `Array` methods ```javascript // Use Object.assign (ES 2015) to copy objects const target = { a: 1, b: 1}; const source = { b: 2}; const merged = Object.assign(target, source); console.assert(_.isEqual(merged, {a: 1, b:2}); // Spread operator const merge2 = {...target, ...source}; // Array.includes (ES7) const theArray = [1, 2] console.assert(theArray.includes(1)) ``` ## Prototypes in JavaScript JavaScript has a very powerful prototypal inheritance system that is very interesting to study. The truth is, it is much less used nowadays, and you don't really need to know it to develop with React. It also requires a bit of personal study to fully understand it. So we will leave it aside for now. The book [JavaScript: The Good Parts](https://www.oreilly.com/library/view/javascript-the-good/9780596517748/) by Douglas Crockford (2008) is a great introduction to it. Here's a quote from the author: > You make prototype objects, and then … make new instances. Objects are mutable in JavaScript, so we can augment the new instances, giving them new fields and methods. These can then act as prototypes for even newer objects. We don't need classes to make lots of similar objects… Objects inherit from objects. What could be more object oriented than that? ![JavaScript: the good parts](./img/js-the-good-parts.jpg) Some good articles: - [A Plain English Guide to JavaScript Prototypes](http://sporto.github.io/blog/2013/02/22/a-plain-english-guide-to-javascript-prototypes/) ## Object literals, assignment and destructuring ### Objects ```javascript const toaster = { size: 2, color: "red", brand: "NoName" }; // Get ("destructure") one object key const { size } = toaster; console.assert(size === 2); // Note: this also works with functions function destructuredFunction({ color }) { return color; } console.assert(destructuredFunction({ color: "red" }) === "red"); // Get the rest with ...rest const { color, brand, ...rest } = toaster; console.assert(_.isEqual(rest, { size: 2 })); // Set default const { size2 = 3 } = toaster; console.assert(size2 === 3); // Rename variables const { size: size3 } = toaster; console.assert(size3 === 2); ``` Enhanced object literals: ```javascript const name = "Louis"; const person = { name }; console.assert(_.isEqual(person, { name: "Louis" })); // Dynamic properties const person2 = { ["first" + "Name"]: "Olympe" }; console.assert(_.isEqual(person2, { firstName: "Olympe" })); // Btw, you can include quotes although nobody does this console.assert(_.isEqual(person2, { firstName: "Olympe" })); // Short form function // Before: const es5Object = { say: function () { console.log("hello"); }, }; es5Object.say(); // After: (short form function) const es6Object = { say() { console.log("hello"); }, }; es6Object.say(); ``` Advanced (with prototype): ```javascript // Prototype and super() const firstObject = { a: "a", hello() { return "hello"; }, }; const secondObject = { __proto__: firstObject, hello() { return super.hello() + " from second object"; }, }; console.assert(secondObject.hello() === "hello from second object"); ``` ### Array ```javascript const theArray = [1, 2, 3]; const [first, second] = theArray; const [first1, second2, ...rest] = theArray; console.assert(first === 1); console.assert(second === 2); console.assert(_.isEqualWith(rest, [3])); ``` ## `let` and `const` ```javascript const constantVar = "a"; // Raises "constantVar" is read-only constantVar = "b"; let theVar = "a"; theVar = "a"; // Note: const != immutable const constantObject = { a: 1 }; constantObject.a = 2; constantObject.b = 3; // Raises: "constantObject" is read-only constantObject = { a: 1 }; // const and let are block scoped. A block is enclosed in {} (if, loops, functions, etc.) { const a = "a"; console.log({ a }); } // Raises: ReferenceError: a is not defined console.log({ a }); ``` Note: try to use `const` as much as you can. - Those variables can't be reassigned. More constraints leads to safer code. - You can't define a `const` without providing its initial value. - Most people do this in modern JS. Never use `var`: - `var` variables are initialized with `undefined`, while `let` and `const` vars are not initialized and will raise an error if used before definition. - `var` is globally or function-scoped, depending on whether it is used inside a function. - `let` and `const` are block-scoped - `let` and `const` cannot be reused for the same variable name Future of JavaScript: [tc39/proposal-record-tuple: ECMAScript proposal for the Record and Tuple value types](https://github.com/tc39/proposal-record-tuple) ### Hoisting See [Hoisting on MDN](https://developer.mozilla.org/en-US/docs/Glossary/Hoisting) ```javascript console.log(typeof variable); // undefined // console.log(variable); // Raises: ReferenceError: variable is not defined function hoist() { a = 20; var b = 100; } hoist(); // 20, accessible as a global variable outside of hoist console.log(a); // Raises: ReferenceError: b is not defined // console.log(b); ``` ## Arrow functions The first advantage of arrow function is that they're shorter to write: ```javascript // You can define a function this way: const myFunction = function () { console.log("hello world"); }; // With an arrow function, you save a few characters: const myArrowFunction = () => { console.log("hello world"); }; // Some things, like params parentheses, and function code brackets, are optional const myFunctionToBeShortened = function (a) { return a; }; // Shorter arrow function const myFunctionToBeShortenedArrowV1 = (a) => { return a; }; // Shortest arrow function // Remove single param parenthesis, remove function code bracket, remove return const myFunctionToBeShortenedArrowV2 = (a) => a; console.assert(myFunctionToBeShortenedArrowV2(1) === 1); ``` ### How `this` works in arrow functions TODO ### Best practices I usually keep the parameters parenthesis. If you add a parameter and weren't including them, you'll have to add them back: ```javascript const a1 = (arg) => {}; const a2 = (arg1, arg2) => {}; // vs. const a3 = (arg) => {}; ``` ## Classes ```javascript class Toaster { constructor(color) { this.color = color; } dring() { return "dring"; } } // Don't forget new! // Raises: TypeError: Cannot call a class as a function // const toaster = Toaster('red'); const toaster = new Toaster("red"); console.log(toaster.dring()); // Inheritance class BunToaster extends Toaster { dring() { return super.dring() + " dring"; } } const bunToaster = new BunToaster("red"); console.assert(bunToaster.dring() === "dring dring"); ``` Those are my opinions about other class features: - Avoid using `static` methods, use plain functions instead. - Avoid using more than one level of inheritance. - Avoid using getter and setters (`get` and `set`). - Avoid using classes if you can. ## Template literals ```javascript const longString = `multi line string`; const name = "Louis"; // Template interpolation const hello = `Hello ${name}`; // You can have expressions const hello1 = `Hello ${name + "!"}`; const hello2 = `Hello ${name === "Louis" ? name : "Noname"}`; ``` ### Template tags They are used in some libraries, like Apollo and Styled Components. ```javascript // First arg is an array of string values, the rest is the expressions // ["hello ", ""], 3 function templateTag(literals, ...expressions) { console.assert(_.isEqual(literals, ["hello ", ""])); console.assert(_.isEqual(expressions, [3])); return _.join(_.flatten(_.zip(literals, expressions)), ""); } const result = templateTag`hello ${1 + 2}`; console.assert(result === "hello 3"); ``` Here's an example with Styled Components: ```javascript const Button = styled.a` /* This renders the buttons above... Edit me! */ display: inline-block; border-radius: 3px; padding: 0.5rem 0; margin: 0.5rem 1rem; width: 11rem; background: transparent; color: white; border: 2px solid white; /* The GitHub button is a primary button * edit this to target it specifically! */ ${props => props.primary && css` background: white; color: black; `} ` ``` You can see how template tags and arrow functions lead to more concise code! ## Loops ### `for... of` Note: prefer using some functional constructs such as `map`, `reduce`, etc. ```javascript for (const i of [1, 2, 3]) { console.log({ i }); } // 1, 2, 3 for (const key in { a: "aaa", b: "bbb" }) { console.log({ key }); } // 'a', 'b' ``` ## Promises This is only going to be an introduction to the magnificent world of promise. - Async functions (seen later) use promises as a building block. - Promise are indeed async in nature: the calling code continues executing the promise does its thing. - Some Web API return promises, including `Fetch` ### Creating a promise Note: we use TypeScript in this example, to clarify what's return. You can ignore the type annotations for now. ```typescript let isDone: boolean = true const thePromise = new Promise((resolve, reject) => { if (isDone) { resolve("the work is done"); } else { reject("this is still pending"); } } console.assert(thePromise === 'the work is done') ``` TODO ### Consuming a promise ### Chaining promises ## Async functions ## Modules CommonJS syntax: ```javascript const lodash = require("lodash"); ``` ES Module syntax: ```javascript import lodash from "lodash"; ``` ### Imports ```javascript // Import all and provide under name import * as toaster from "./toaster"; // Named import (same as object destructuring!) import { defaultColor, defaultSize } from "./toaster"; // Renaming imports import { defaultBrand as toasterDefaultBrand } from "./toaster"; // Default import import createToaster from "./toaster"; // Import both defaults and other // import createToaster, {defaultColor} from './toaster' ``` ### Exports In `toaster.js`: ```javascript // Shorthand definition + export export const defaultSize = 4; // Alternative export syntax const defaultBrand = "Moulinex"; export { defaultBrand }; // Default export const createToaster = ({ size, color }) => ({ size, color }); export default createToaster; // Note that you have a shorthand default export, but it's not recommended to // use it as the export won't have a name. // export default () => ({}) ``` ## Other features ### Optional chaining ```javascript let nestedProp = obj.first && obj.first.second; // with optional chaining: let nestedProp = obj.first?.second; ``` ### Ternary operator ```javascript const a = 'a' const r = a === 'a' ? 'isA' : 'isNotA' console.assert(r === 'isA') ``` ## Self assessment - How old is JavaScript? - Is it a modern language? - Can we say that JavaScript does not have types? - What is the value of this expression: `"1" + "1"`? `3 + 2 + "5"`? - What should I use? `let`, `var`, or `const`? - How do you add support for modern JS features in older browsers? - How are variables scoped in JavaScript? - What should you watch for when comparing variables in JavaScript? - `const a = [1]; const b = [1];`: what does `a == b` evaluates to? - How do you write arrow functions? - `const {a} = {a: 1}`: what does `a` evaluate to? - How do you write the ternary operator? Advanced: - How does JavaScript objects "inherit" from each other? ```javascript // Write transform1 using a one-line arrow function with object structuring console.assert(_.isEqual(transform1({name: "Foo"}), {FooA:1})) console.assert(_.isEqual(transform1({name: "Bar"}), {Bar:1})) ``` The three things you need: - Use `let` and `const` - Object destructuring `const { a } = {a: 1}` - Arrow functions `const noop = () => { }` Other assessments: - [37 Essential JavaScript Interview Questions and Answers](https://www.toptal.com/javascript/interview-questions) - [70 JavaScript Interview Questions](https://dev.to/macmacky/70-javascript-interview-questions-5gfi), DEV ## References - [ES5 to ESNext — here’s every feature added to JavaScript since 2015](https://www.freecodecamp.org/news/es5-to-esnext-heres-every-feature-added-to-javascript-since-2015-d0c255e13c6e/) - [ES2015 / ES6: Basics of modern Javascript](https://www.slideshare.net/WojciechDzikowski/es2015-es6-basics-of-modern-javascript) - [JavaScript](https://en.wikipedia.org/wiki/JavaScript), Wikipedia - [mbeaudru/modern-js-cheatsheet](https://github.com/mbeaudru/modern-js-cheatsheet) - [The vanilla Javascript basics to know before learning React JS - DEV](https://dev.to/tracycss/the-vanilla-javascript-basics-to-know-before-learning-react-js-53aj) - [The Modern JavaScript Tutorial](https://javascript.info/) Future changes: - [tc39/proposal-record-tuple: ECMAScript proposal for the Record and Tuple value types.](https://github.com/tc39/proposal-record-tuple) ================================================ FILE: training/front-end/02-react.md ================================================ ## Table of Contents - [An introduction to React](#an-introduction-to-react) - [Recap of the previous session](#recap-of-the-previous-session) - [Before getting into React...](#before-getting-into-react) - [Manipulating the DOM](#manipulating-the-dom) - [Learning React](#learning-react) - [Using CodeSandbox](#using-codesandbox) - [Exercise](#exercise) - [Self-assessment](#self-assessment) # An introduction to React ## Recap of the previous session See previous session - Arrow functions - Classes - `let` and `const` - Object and array destructuring - Template literals ## Before getting into React... ### Manipulating the DOM - https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Client-side_web_APIs/Manipulating_documents - https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model/Introduction ## Learning React Follow the tutorial: https://reactjs.org/docs/hello-world.html We have chosen this tutorial over the "learn by doing" one because it introduces the concepts behind React. It is very well done and very complete. ### Using CodeSandbox It is a good exercise to start from a Vanilla JS template and setup CodeSandbox. 1. Go to https://codesandbox.io/ 2. Start a Vanilla JS project 3. Add `react` and `react-dom` dependencies 4. Add the simplest React app from the tutorial and get it to work! ## Exercise Use one of our hiring exercises. ## Self-assessment Basics: - What is the difference between elements and components? - What is JSX? - Is it ok to modify props? - How should you modify state? - What does "the data flows down" mean? - How do you pass arguments to event handlers? - How do you put conditionals in JSX? - How do you put inline if-else in JSX? - How do you prevent a component from rendering? - What is the `key` prop? - What are controlled and uncontrolled components? - What does it mean to "lift state up"? - Should you use composition or inheritance with React? Advanced: ================================================ FILE: training/front-end/03-typescript.md ================================================ ## Table of Contents - [TypeScript](#typescript) - [Exercises](#exercises) - [References](#references) # TypeScript - Intro for JS dev: https://www.typescriptlang.org/docs/handbook/typescript-in-5-minutes.html - Intro for OOP dev: https://www.typescriptlang.org/docs/handbook/typescript-in-5-minutes-oop.html - The handbook: https://www.typescriptlang.org/docs/handbook/intro.html - Intro for FP programmers: https://www.typescriptlang.org/docs/handbook/typescript-in-5-minutes-func.html ❓ questions to ask: - What is the difference between structural and nominal type systems? - TS: structural type system. Relationships between types are determined by the properties they contain. ## Exercises - [TypeScript Exercises](https://typescript-exercises.github.io/) ## References - [ES5 to ESNext — here’s every feature added to JavaScript since 2015](https://www.freecodecamp.org/news/es5-to-esnext-heres-every-feature-added-to-javascript-since-2015-d0c255e13c6e/) ================================================ FILE: training/front-end/04-the-browser.md ================================================ ## Table of Contents - [The browser](#the-browser) - [Topics](#topics) - [Reading and manipulating the DOM](#reading-and-manipulating-the-dom) - [Events](#events) # The browser ## Topics ### Reading and manipulating the DOM ### Events https://dmitripavlutin.com/javascript-event-delegation/ ================================================ FILE: training/front-end/05-css-html-restart.md ================================================ ## Table of Contents - [CSS & HTML: restart](#css--html-restart) - [An intro to modern technologies and methodologies](#an-intro-to-modern-technologies-and-methodologies) - [Topics](#topics) - [Accessibility](#accessibility) # CSS & HTML: restart ## An intro to modern technologies and methodologies ## Topics ### Accessibility ================================================ FILE: training/front-end/06-front-end-development-practices.md ================================================ ## Table of Contents - [Front-end development practices](#front-end-development-practices) - [Goal of this session](#goal-of-this-session) - [TypeScript](#typescript) - [Building a professional web app: topics](#building-a-professional-web-app-topics) - [Dependency management: npm or yarn](#dependency-management-npm-or-yarn) - [Testing](#testing) - [Code style](#code-style) - [Security](#security) - [State management](#state-management) - [Interaction with the backend](#interaction-with-the-backend) - [Error handling](#error-handling) - [CSS](#css) - [Developer experience](#developer-experience) - [Building and deploying](#building-and-deploying) - [Bundling](#bundling) - [Compiling](#compiling) - [Deploying](#deploying) - [Monitoring](#monitoring) - [Performance](#performance) - [PWA (progressive web apps)](#pwa-progressive-web-apps) - [See also](#see-also) # Front-end development practices ## Goal of this session We are reaching the end of our journey... Training can't teach you everything - you'll have to get your hands dirty to learn about front-end development. That being said, in this session we will try to look at all the moving pieces that are necessary to be dangerous in modern front-end development. ## TypeScript See [TypeScript](./03-typescript.md) ## Building a professional web app: topics ### Dependency management: npm or yarn - npm - yarn ### Testing - jest ### Code style - prettier - eslint ### Security - HTTPS - CORS - XSS prevention - CSP - OWASP ### State management - Redux (and redux-thunk, redux-saga, redux-observables). Also: [General | Redux](https://redux.js.org/faq/general) - Mobx - `useReducer`, `useContext`, `useState` ### Interaction with the backend - GraphQL and Apollo - redux-saga et al. - Hooks for API handling ### Error handling - Error boundaries ### CSS - CSS-in-JS - styled-components - Emotion Frameworks: - Material UI - Tailwind CSS - CSS-first (no JS components) vs other frameworks ### Developer experience - Storybook - Debugging with the React extension - Performance and profiling ### Building and deploying #### Bundling - webpack - parcel - rollup #### Compiling - babel #### Deploying - SSR vs. CSR vs. SSG #### Monitoring - Sentry ### Performance - Lighthouse - Devtools ### PWA (progressive web apps) - Web Sockets - Service Workers - Payments ## See also See references in [README.md](./README.md), in particular: - [Front-end Developer Handbook 2019 - Learn the entire JavaScript, CSS and HTML development practice!](https://frontendmasters.com/books/front-end-handbook/2019/) - ================================================ FILE: training/front-end/README.md ================================================ ## Table of Contents - [Introduction to front-end programming for back-end developers](#introduction-to-front-end-programming-for-back-end-developers) - [Goal of this training](#goal-of-this-training) - [Prerequisites](#prerequisites) - [To do before the course](#to-do-before-the-course) - [Curriculum](#curriculum) - [Roadmap: to go further](#roadmap-to-go-further) - [Inspiration](#inspiration) - [Other resources](#other-resources) # Introduction to front-end programming for back-end developers ## Goal of this training This training was developed for back-end and infra developers. It is meant to be used as a support for an oral presentation. I am not an expert in front-end development. This should only be considered an introduction. ## Prerequisites Assumed knowledge: - Basics about HTTP (cookies, protocol, HTTPS) - Basics about browser (DOM, using the developer console) - Basics about HTML and CSS (tags & attributes, how CSS work, CSS selectors, CSS styling) - Basics about programming languages (conditionals, loops, variables, typing, tests) - Basics about JavaScript (syntax, having written a few hundred lines of code) ## To do before the course Read the [grab/front-end-guide](https://github.com/grab/front-end-guide) to get a 10,000 viewpoint about the road ahead. Evidently you don't need to check the details about each topic (that's what we'll do in this course), but obviously feel free to look up what seems interesting to you! ## Curriculum Instead of going deep in topics, we try to go as fast as possible to front-end programming (i.e. React with TypeScript), and then we go back to address those more specific topics. 1. The language: writing modern Javascript 2. React 3. TypeScript 4. The browser 5. CSS/HTML restart 6. Front-end development practices ## Roadmap: to go further ![learning front-end roadmap](./img/learning-roadmap.png) Image from https://github.com/kamranahmedse/developer-roadmap ## Inspiration - [Front-end Developer Handbook 2019 - Learn the entire JavaScript, CSS and HTML development practice!](https://frontendmasters.com/books/front-end-handbook/2019/) - [grab/front-end-guide: 📚 Study guide and introduction to the modern front end stack.](https://github.com/grab/front-end-guide) ## Other resources - [leonardomso/33-js-concepts](https://github.com/leonardomso/33-js-concepts): 33 JavaScript concepts every developer should know. - [The Modern JavaScript Tutorial](https://javascript.info/) - [A slightly sarcastic history of frontend](https://app.daily.dev/posts/a-slightly-sarcastic-history-of-frontend-1ohcpd7uq) - [gibbok/typescript-book: The Concise TypeScript Book: A Concise Guide to Effective Development in TypeScript. Free and Open Source.](https://github.com/gibbok/typescript-book) ================================================ FILE: training/front-end/examples/hoisting.html ================================================ Showing hoisting
================================================ FILE: training/learning-python/README.md ================================================ See my other repo: https://github.com/charlax/python-education ================================================ FILE: training/web-dev/README.md ================================================ ## Table of Contents - [Learning web development](#learning-web-development) - [Goal of this training](#goal-of-this-training) - [Principles](#principles) - [Meta: learning about learning](#meta-learning-about-learning) - [Roadmap](#roadmap) - [Approach](#approach) - [Pick a powerful text editor and learn its ins and outs](#pick-a-powerful-text-editor-and-learn-its-ins-and-outs) - [Learn the ins and outs of one programming language](#learn-the-ins-and-outs-of-one-programming-language) - [Topic-specific materials](#topic-specific-materials) - [Best practices, attitude](#best-practices-attitude) - [Code architecture and design patterns](#code-architecture-and-design-patterns) - [DB and SQL](#db-and-sql) - [Dev environment, command line and Linux](#dev-environment-command-line-and-linux) - [Distributed systems and system architecture](#distributed-systems-and-system-architecture) - [Network, protocols, HTTP](#network-protocols-http) - [Project management](#project-management) - [Security](#security) - [Version control (git)](#version-control-git) - [Other lists](#other-lists) # Learning web development ## Goal of this training This training provides an opinionated ramp-up program for web developers. Prerequisites: not much, just general knowledge about computers and the Internet. ## Principles - Focus on the main stuff. A lot of things are learnt on the job anyway. - Programming is not only about raw technical knowledge, it's also about best practices. - Go wide and go deep. - We learn by doing, so this training includes lots of exercises. - A lot of videos are included, since they provide hands-on experiences. You can find a LOT more videos here: [hellerve/programming-talks](https://github.com/hellerve/programming-talks) and [talks that changed the way I think about programming](http://www.opowell.com/post/talks-that-changed-the-way-i-think-about-programming/) Note: this is just a short selection of stuff listed in [charlax/professional-programming](https://github.com/charlax/professional-programming). ### Meta: learning about learning Feel free to checkout some of the articles about [Learning and memorizing](https://github.com/charlax/professional-programming#learning--memorizing) - Create a learning plan with your learning priorities - Consider using flashcards (see Anki for instance) - Enjoy the learning experience! ## Roadmap 1. Start with a programming language. 2. Do a first pass at the web development environment (see introductory resources below) - HTTP and web architecture basics - Git - Docker and `docker-compose` - Command line - Text editor 3. Write a full-fledged exercise, get it reviewed. 4. Do the advanced topics ## Approach ### Pick a powerful text editor and learn its ins and outs VSCode is a strong pick nowadays (I use and obviously prefer Vim :). Make sure to spend a lot of time in your text editor, watch tutorials about advanced features, install extensions, learn all the main keyboard shortcuts, subscribe to mailing lists about it, etc. You will spend most of your time within your text editor. Turn it into a powerful ally! ### Learn the ins and outs of one programming language A developer's main tool being the programming language, it is important to achieve high proficiency in at least one of them. I'd recommend starting with Python or TypeScript. For Python, you can have a look at my repo [charlax/python-education](https://github.com/charlax/python-education). To ensure you have good command of the language, you should try out some of those exercises: - [Exercism](https://exercism.io/) (get free code reviews!) - [Small Python exercises from charlax/python-education](https://github.com/charlax/python-education/tree/master/learning-python/exercises) - [danistefanovic/build-your-own-x](https://github.com/danistefanovic/build-your-own-x) (for instance: build a [git in Python](https://wyag.thb.lt/)) - [Other list of exercises](https://github.com/charlax/python-education#exercises) Watch some videos related to your language: - 🎞 [Top 10 Must-Watch PyCon Talks](https://realpython.com/must-watch-pycon-talks/) Learn how to handle: - Regexes - Do exercises, for instance [RegexOne](https://regexone.com/) or [HackerRank](https://www.hackerrank.com/domains/regex). - Functional programming - [Functional Programming Fundamentals](https://www.matthewgerstman.com/tech/functional-programming-fundamentals/) - Design patterns - E.g. in [Python](https://www.toptal.com/python/python-design-patterns) - [faif/python-patterns](https://github.com/faif/python-patterns) - 🎞 [Design Patterns in Plain English](https://www.youtube.com/watch?v=NU_1StN5Tkk&ab_channel=ProgrammingwithMosh) - Tests - [Why bother writing tests at all?](https://dave.cheney.net/2019/05/14/why-bother-writing-tests-at-all) - Crazy things in languages - 🎞 [Wat](https://www.destroyallsoftware.com/talks/wat) If you have time, learn more programming languages, starting with some that are *very* different from your main one: - Haskell - Clojure - Kotlin - Rust - Assembly ## Topic-specific materials #### Best practices, attitude Read one of those: - 📖 [The Pragmatic Programmer: From Journeyman to Master](https://pragprog.com/titles/tpp20/): hands-on the most inspiring and useful book I've read about programming. - 📖 [Code Complete: A Practical Handbook of Software Construction](http://www.amazon.com/Code-Complete-Practical-Handbook-Construction/dp/0735619670): a nice addition to The Pragmatic Programmer, gives you the necessary framework to talk about code. - [Ten Principles for Growth as an Engineer](https://medium.com/@daniel.heller/ten-principles-for-growth-69015e08c35b), Dan Heller Read the resources listed under [Must-read articles](https://github.com/charlax/professional-programming#must-read-articles). #### Code architecture and design patterns Introduction: - Learn about OOP basics: inheritance, abstract/concrete, public/private - Learn about DDD (domain driven design), SOLID, MVC, Clean Architecture - [An introduction to software design patterns](https://www.coengoedegebure.com/introduction-to-software-design-patterns/). All those patterns are not useful in all languages, but they're applied in most software, so it's important to know about them. - Builder - Factory - Singleton - Adapter - Facade - Chain of Responsibility - Command - Observer - State - Strategy - Visitor - 🎞 [Design Patterns in Plain English](https://www.youtube.com/watch?v=NU_1StN5Tkk&ab_channel=ProgrammingwithMosh) Advanced: - Wander in the [Software Architecture Guide](https://martinfowler.com/architecture/) - 🎞 [On the Spectrum of Abstraction](https://www.youtube.com/watch?v=mVVNJKv9esE&ab_channel=ReactEurope) - 🎞 [Simple Made Easy](https://www.infoq.com/presentations/Simple-Made-Easy/) - 🎞 [Zebras All the Way Down](https://youtu.be/fE2KDzZaxvE), Bryan Cantrill #### DB and SQL Introduction: - Learn basic and advanced SQL: joins, indexes, subqueries. - Install Postgres and play with it. - [Do those postgres exercises](https://pgexercises.com/) Advanced: - Learn about ORM - 🎞 [Watch this Python introduction to ORMs](https://www.youtube.com/watch?v=P141KRbxVKc&ab_channel=PyCon2014) (with SQLAlchemy) #### Dev environment, command line and Linux Introduction: - [jlevy/the-art-of-command-line](https://github.com/jlevy/the-art-of-command-line): master the command line, in one page - Check the missing semester: - https://missing.csail.mit.edu/2020/course-shell/ - https://missing.csail.mit.edu/2020/shell-tools/ - [Linux Productivity Tools](https://www.usenix.org/sites/default/files/conference/protected-files/lisa19_maheshwari.pdf) - Do shell exercises, for instance on [Exercism](https://exercism.io/tracks/bash), [TLDP](https://tldp.org/LDP/abs/html/exercises.html), [HackerRank](https://www.hackerrank.com/domains/shell). - Learn about Docker, follow the [Official Docker Tutorial](https://docs.docker.com/get-started/) - Learn about Makefile, follow the [Official introduction to Makefiles](https://www.gnu.org/software/make/manual/html_node/Introduction.html). Advanced: - Install/use some of those tools: https://github.com/jondot/awesome-devenv - Write your own dotfiles, taking inspiration from [Awesome Dotfiles](https://github.com/webpro/awesome-dotfiles) (you can checkout mines: https://github.com/charlax/dotfiles) - 🎞 [8 super heroic Linux commands that you probably aren't using](https://www.youtube.com/watch?v=Zuwa8zlfXSY&ab_channel=EngineerMan) - 🎞 [Containers From Scratch](https://www.youtube.com/watch?v=8fi7uSYlOdc&ab_channel=GOTOConferences) #### Distributed systems and system architecture Introduction: - [donnemartin/system-design-primer](https://github.com/donnemartin/system-design-primer): learn how to design large-scale systems. Advanced: - Consider reading 📖 [Designing Data-Intensive Applications](https://dataintensive.net/). - [The Log: What every software engineer should know about real-time data's unifying abstraction](https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying) - 🎞 [Mastering Chaos - A Netflix Guide to Microservices](https://www.youtube.com/watch?v=CZ3wIuvmHeM&ab_channel=InfoQ) (by Netflix) Concepts to look into: - CAP theorem - Weak/strong/eventual consistency - Fail-over/replication - Load balancer: active-active, active-passive - Horizontal vs vertical scaling - Reverse proxy, load balancer - DB: RDBMS vs. NoSQL, replication, sharding, master-master, denormalization - Cache: client caching in browsers, CDN, HTTP headers, object cache, db cache - Asynchronism: message queues, back pressure - Monitoring and alerting best practices #### Network, protocols, HTTP Introduction: - [How Does the Internet Work?](https://web.stanford.edu/class/msande91si/www-spr04/readings/week1/InternetWhitepaper.htm) - [vasanthk/how-web-works](https://github.com/vasanthk/how-web-works): what happens behind the scenes when we type www.google.com in a browser? Advanced: - Learn about the basics of TCP and UDP - Learn the basics of the HTTP protocol: header, verb, status code, TLS, Rest APIs, etc. - 🎞 [Qu'est ce que le HTTP ?](https://grafikart.fr/tutoriels/http-1062), Grafikart - Learn about important web-related protocols: DNS, SMTP, SSH #### Project management Introduction: - [Efficient Software Project Management at its Roots](https://blog.pragmaticengineer.com/efficient-software-project-management-at-its-roots/) - [How to Lead a Project - as a Software Engineer](https://blog.pragmaticengineer.com/how-to-lead-a-project-in-software-development/) Advanced: - [TechnicalDebt](https://martinfowler.com/bliki/TechnicalDebt.html) - Checkout this section on [charlax/engineering-management](https://github.com/charlax/engineering-management#project-management) #### Security Introduction: - Learn about the OWASP Top 10 Advanced: - Consider reading 📖 [Penetration Testing](https://nostarch.com/pentesting). #### Version control (git) Introduction: - [Resources to learn Git](https://try.github.io/) - 🎞 [Learn Git In 15 Minutes - YouTube](https://www.youtube.com/watch?v=USjZcfj8yxE&ab_channel=ColtSteele) - 🎞 [Introduction to Git](https://www.youtube.com/watch?v=ZDR433b0HJY&ab_channel=InfoQ) with Scott Chacon of GitHub Advanced topics: - `git rebase` - `git stash` - `git revert` - The dangers of `git push --force` ## Other lists - [The Missing Semester of Your CS Education](https://missing.csail.mit.edu/) (MIT). - [What every computer science major should know](http://matt.might.net/articles/what-cs-majors-should-know/) - [Teach Yourself Computer Science](https://teachyourselfcs.com/) - https://github.com/charlax/professional-programming - [microsoft/Web-Dev-For-Beginners](https://github.com/microsoft/Web-Dev-For-Beginners): 24 Lessons, 12 Weeks, Get Started as a Web Developer