Showing preview only (5,698K chars total). Download the full file or copy to clipboard to get everything.
Repository: Vonng/ddia
Branch: main
Commit: 573bb53a0557
Files: 139
Total size: 5.4 MB
Directory structure:
gitextract_zlswtsh8/
├── .github/
│ └── workflows/
│ └── pages.yaml
├── .gitignore
├── .nojekyll
├── LICENSE
├── Makefile
├── README.md
├── assets/
│ └── css/
│ ├── custom.css
│ └── example.css
├── bin/
│ ├── Pipfile
│ ├── doc
│ ├── epub
│ ├── preprocess-epub.py
│ ├── toc.py
│ ├── translate.py
│ └── zh-tw.py
├── content/
│ ├── en/
│ │ ├── _index.md
│ │ ├── ch1.md
│ │ ├── ch10.md
│ │ ├── ch11.md
│ │ ├── ch12.md
│ │ ├── ch13.md
│ │ ├── ch14.md
│ │ ├── ch2.md
│ │ ├── ch3.md
│ │ ├── ch4.md
│ │ ├── ch5.md
│ │ ├── ch6.md
│ │ ├── ch7.md
│ │ ├── ch8.md
│ │ ├── ch9.md
│ │ ├── colophon.md
│ │ ├── glossary.md
│ │ ├── indexes.md
│ │ ├── part-i.md
│ │ ├── part-ii.md
│ │ ├── part-iii.md
│ │ ├── preface.md
│ │ └── toc.md
│ ├── tw/
│ │ ├── _index.md
│ │ ├── ch1.md
│ │ ├── ch10.md
│ │ ├── ch11.md
│ │ ├── ch12.md
│ │ ├── ch13.md
│ │ ├── ch14.md
│ │ ├── ch2.md
│ │ ├── ch3.md
│ │ ├── ch4.md
│ │ ├── ch5.md
│ │ ├── ch6.md
│ │ ├── ch7.md
│ │ ├── ch8.md
│ │ ├── ch9.md
│ │ ├── colophon.md
│ │ ├── contrib.md
│ │ ├── glossary.md
│ │ ├── indexes.md
│ │ ├── part-i.md
│ │ ├── part-ii.md
│ │ ├── part-iii.md
│ │ ├── preface.md
│ │ └── toc.md
│ ├── v1/
│ │ ├── _index.md
│ │ ├── ch1.md
│ │ ├── ch10.md
│ │ ├── ch11.md
│ │ ├── ch12.md
│ │ ├── ch2.md
│ │ ├── ch3.md
│ │ ├── ch4.md
│ │ ├── ch5.md
│ │ ├── ch6.md
│ │ ├── ch7.md
│ │ ├── ch8.md
│ │ ├── ch9.md
│ │ ├── colophon.md
│ │ ├── contrib.md
│ │ ├── glossary.md
│ │ ├── part-i.md
│ │ ├── part-ii.md
│ │ ├── part-iii.md
│ │ ├── preface.md
│ │ └── toc.md
│ ├── v1_tw/
│ │ ├── _index.md
│ │ ├── ch1.md
│ │ ├── ch10.md
│ │ ├── ch11.md
│ │ ├── ch12.md
│ │ ├── ch2.md
│ │ ├── ch3.md
│ │ ├── ch4.md
│ │ ├── ch5.md
│ │ ├── ch6.md
│ │ ├── ch7.md
│ │ ├── ch8.md
│ │ ├── ch9.md
│ │ ├── colophon.md
│ │ ├── contrib.md
│ │ ├── glossary.md
│ │ ├── part-i.md
│ │ ├── part-ii.md
│ │ ├── part-iii.md
│ │ ├── preface.md
│ │ └── toc.md
│ └── zh/
│ ├── _index.md
│ ├── ch1.md
│ ├── ch10.md
│ ├── ch11.md
│ ├── ch12.md
│ ├── ch13.md
│ ├── ch14.md
│ ├── ch2.md
│ ├── ch3.md
│ ├── ch4.md
│ ├── ch5.md
│ ├── ch6.md
│ ├── ch7.md
│ ├── ch8.md
│ ├── ch9.md
│ ├── colophon.md
│ ├── contrib.md
│ ├── glossary.md
│ ├── indexes.md
│ ├── part-i.md
│ ├── part-ii.md
│ ├── part-iii.md
│ ├── preface.md
│ └── toc.md
├── giscus.json
├── go.mod
├── go.sum
├── hugo.yaml
├── i18n/
│ ├── en.yaml
│ ├── tw.yaml
│ ├── v2.yaml
│ └── zh.yaml
├── js/
│ └── epub.css
├── layouts/
│ └── shortcodes/
│ └── figure.html
└── metadata.yaml
================================================
FILE CONTENTS
================================================
================================================
FILE: .github/workflows/pages.yaml
================================================
# Sample workflow for building and deploying a Hugo site to GitHub Pages
name: Deploy Hugo site to Pages
on:
# Runs on pushes targeting the default branch
push:
branches: ["main"]
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
permissions:
contents: read
pages: write
id-token: write
# Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued.
# However, do NOT cancel in-progress runs as we want to allow these production deployments to complete.
concurrency:
group: "pages"
cancel-in-progress: false
# Default to bash
defaults:
run:
shell: bash
jobs:
# Build job
build:
runs-on: ubuntu-latest
env:
HUGO_VERSION: 0.155.3
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0 # fetch all history for .GitInfo and .Lastmod
submodules: recursive
- name: Setup Go
uses: actions/setup-go@v5
with:
go-version: '1.26'
- name: Setup Pages
id: pages
uses: actions/configure-pages@v4
- name: Setup Hugo
run: |
wget -O ${{ runner.temp }}/hugo.deb https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_extended_${HUGO_VERSION}_linux-amd64.deb \
&& sudo dpkg -i ${{ runner.temp }}/hugo.deb
- name: Build with Hugo
env:
# For maximum backward compatibility with Hugo modules
HUGO_ENVIRONMENT: production
HUGO_ENV: production
run: |
hugo \
--gc --minify \
--baseURL "${{ steps.pages.outputs.base_url }}/"
- name: Upload artifact
uses: actions/upload-pages-artifact@v3
with:
path: ./public
# Deployment job
deploy:
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
runs-on: ubuntu-latest
needs: build
steps:
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@v4
================================================
FILE: .gitignore
================================================
.idea/
.code/
__pycache__/
.DS_Store
tmp/
output/
public/
.hugo_build.lock
.claude
CLAUDE.md
content/cn/
zh.md
en.md
================================================
FILE: .nojekyll
================================================
================================================
FILE: LICENSE
================================================
Attribution 4.0 International
=======================================================================
Creative Commons Corporation ("Creative Commons") is not a law firm and
does not provide legal services or legal advice. Distribution of
Creative Commons public licenses does not create a lawyer-client or
other relationship. Creative Commons makes its licenses and related
information available on an "as-is" basis. Creative Commons gives no
warranties regarding its licenses, any material licensed under their
terms and conditions, or any related information. Creative Commons
disclaims all liability for damages resulting from their use to the
fullest extent possible.
Using Creative Commons Public Licenses
Creative Commons public licenses provide a standard set of terms and
conditions that creators and other rights holders may use to share
original works of authorship and other material subject to copyright
and certain other rights specified in the public license below. The
following considerations are for informational purposes only, are not
exhaustive, and do not form part of our licenses.
Considerations for licensors: Our public licenses are
intended for use by those authorized to give the public
permission to use material in ways otherwise restricted by
copyright and certain other rights. Our licenses are
irrevocable. Licensors should read and understand the terms
and conditions of the license they choose before applying it.
Licensors should also secure all rights necessary before
applying our licenses so that the public can reuse the
material as expected. Licensors should clearly mark any
material not subject to the license. This includes other CC-
licensed material, or material used under an exception or
limitation to copyright. More considerations for licensors:
wiki.creativecommons.org/Considerations_for_licensors
Considerations for the public: By using one of our public
licenses, a licensor grants the public permission to use the
licensed material under specified terms and conditions. If
the licensor's permission is not necessary for any reason--for
example, because of any applicable exception or limitation to
copyright--then that use is not regulated by the license. Our
licenses grant only permissions under copyright and certain
other rights that a licensor has authority to grant. Use of
the licensed material may still be restricted for other
reasons, including because others have copyright or other
rights in the material. A licensor may make special requests,
such as asking that all changes be marked or described.
Although not required by our licenses, you are encouraged to
respect those requests where reasonable. More considerations
for the public:
wiki.creativecommons.org/Considerations_for_licensees
=======================================================================
Creative Commons Attribution 4.0 International Public License
By exercising the Licensed Rights (defined below), You accept and agree
to be bound by the terms and conditions of this Creative Commons
Attribution 4.0 International Public License ("Public License"). To the
extent this Public License may be interpreted as a contract, You are
granted the Licensed Rights in consideration of Your acceptance of
these terms and conditions, and the Licensor grants You such rights in
consideration of benefits the Licensor receives from making the
Licensed Material available under these terms and conditions.
Section 1 -- Definitions.
a. Adapted Material means material subject to Copyright and Similar
Rights that is derived from or based upon the Licensed Material
and in which the Licensed Material is translated, altered,
arranged, transformed, or otherwise modified in a manner requiring
permission under the Copyright and Similar Rights held by the
Licensor. For purposes of this Public License, where the Licensed
Material is a musical work, performance, or sound recording,
Adapted Material is always produced where the Licensed Material is
synched in timed relation with a moving image.
b. Adapter's License means the license You apply to Your Copyright
and Similar Rights in Your contributions to Adapted Material in
accordance with the terms and conditions of this Public License.
c. Copyright and Similar Rights means copyright and/or similar rights
closely related to copyright including, without limitation,
performance, broadcast, sound recording, and Sui Generis Database
Rights, without regard to how the rights are labeled or
categorized. For purposes of this Public License, the rights
specified in Section 2(b)(1)-(2) are not Copyright and Similar
Rights.
d. Effective Technological Measures means those measures that, in the
absence of proper authority, may not be circumvented under laws
fulfilling obligations under Article 11 of the WIPO Copyright
Treaty adopted on December 20, 1996, and/or similar international
agreements.
e. Exceptions and Limitations means fair use, fair dealing, and/or
any other exception or limitation to Copyright and Similar Rights
that applies to Your use of the Licensed Material.
f. Licensed Material means the artistic or literary work, database,
or other material to which the Licensor applied this Public
License.
g. Licensed Rights means the rights granted to You subject to the
terms and conditions of this Public License, which are limited to
all Copyright and Similar Rights that apply to Your use of the
Licensed Material and that the Licensor has authority to license.
h. Licensor means the individual(s) or entity(ies) granting rights
under this Public License.
i. Share means to provide material to the public by any means or
process that requires permission under the Licensed Rights, such
as reproduction, public display, public performance, distribution,
dissemination, communication, or importation, and to make material
available to the public including in ways that members of the
public may access the material from a place and at a time
individually chosen by them.
j. Sui Generis Database Rights means rights other than copyright
resulting from Directive 96/9/EC of the European Parliament and of
the Council of 11 March 1996 on the legal protection of databases,
as amended and/or succeeded, as well as other essentially
equivalent rights anywhere in the world.
k. You means the individual or entity exercising the Licensed Rights
under this Public License. Your has a corresponding meaning.
Section 2 -- Scope.
a. License grant.
1. Subject to the terms and conditions of this Public License,
the Licensor hereby grants You a worldwide, royalty-free,
non-sublicensable, non-exclusive, irrevocable license to
exercise the Licensed Rights in the Licensed Material to:
a. reproduce and Share the Licensed Material, in whole or
in part; and
b. produce, reproduce, and Share Adapted Material.
2. Exceptions and Limitations. For the avoidance of doubt, where
Exceptions and Limitations apply to Your use, this Public
License does not apply, and You do not need to comply with
its terms and conditions.
3. Term. The term of this Public License is specified in Section
6(a).
4. Media and formats; technical modifications allowed. The
Licensor authorizes You to exercise the Licensed Rights in
all media and formats whether now known or hereafter created,
and to make technical modifications necessary to do so. The
Licensor waives and/or agrees not to assert any right or
authority to forbid You from making technical modifications
necessary to exercise the Licensed Rights, including
technical modifications necessary to circumvent Effective
Technological Measures. For purposes of this Public License,
simply making modifications authorized by this Section 2(a)
(4) never produces Adapted Material.
5. Downstream recipients.
a. Offer from the Licensor -- Licensed Material. Every
recipient of the Licensed Material automatically
receives an offer from the Licensor to exercise the
Licensed Rights under the terms and conditions of this
Public License.
b. No downstream restrictions. You may not offer or impose
any additional or different terms or conditions on, or
apply any Effective Technological Measures to, the
Licensed Material if doing so restricts exercise of the
Licensed Rights by any recipient of the Licensed
Material.
6. No endorsement. Nothing in this Public License constitutes or
may be construed as permission to assert or imply that You
are, or that Your use of the Licensed Material is, connected
with, or sponsored, endorsed, or granted official status by,
the Licensor or others designated to receive attribution as
provided in Section 3(a)(1)(A)(i).
b. Other rights.
1. Moral rights, such as the right of integrity, are not
licensed under this Public License, nor are publicity,
privacy, and/or other similar personality rights; however, to
the extent possible, the Licensor waives and/or agrees not to
assert any such rights held by the Licensor to the limited
extent necessary to allow You to exercise the Licensed
Rights, but not otherwise.
2. Patent and trademark rights are not licensed under this
Public License.
3. To the extent possible, the Licensor waives any right to
collect royalties from You for the exercise of the Licensed
Rights, whether directly or through a collecting society
under any voluntary or waivable statutory or compulsory
licensing scheme. In all other cases the Licensor expressly
reserves any right to collect such royalties.
Section 3 -- License Conditions.
Your exercise of the Licensed Rights is expressly made subject to the
following conditions.
a. Attribution.
1. If You Share the Licensed Material (including in modified
form), You must:
a. retain the following if it is supplied by the Licensor
with the Licensed Material:
i. identification of the creator(s) of the Licensed
Material and any others designated to receive
attribution, in any reasonable manner requested by
the Licensor (including by pseudonym if
designated);
ii. a copyright notice;
iii. a notice that refers to this Public License;
iv. a notice that refers to the disclaimer of
warranties;
v. a URI or hyperlink to the Licensed Material to the
extent reasonably practicable;
b. indicate if You modified the Licensed Material and
retain an indication of any previous modifications; and
c. indicate the Licensed Material is licensed under this
Public License, and include the text of, or the URI or
hyperlink to, this Public License.
2. You may satisfy the conditions in Section 3(a)(1) in any
reasonable manner based on the medium, means, and context in
which You Share the Licensed Material. For example, it may be
reasonable to satisfy the conditions by providing a URI or
hyperlink to a resource that includes the required
information.
3. If requested by the Licensor, You must remove any of the
information required by Section 3(a)(1)(A) to the extent
reasonably practicable.
4. If You Share Adapted Material You produce, the Adapter's
License You apply must not prevent recipients of the Adapted
Material from complying with this Public License.
Section 4 -- Sui Generis Database Rights.
Where the Licensed Rights include Sui Generis Database Rights that
apply to Your use of the Licensed Material:
a. for the avoidance of doubt, Section 2(a)(1) grants You the right
to extract, reuse, reproduce, and Share all or a substantial
portion of the contents of the database;
b. if You include all or a substantial portion of the database
contents in a database in which You have Sui Generis Database
Rights, then the database in which You have Sui Generis Database
Rights (but not its individual contents) is Adapted Material; and
c. You must comply with the conditions in Section 3(a) if You Share
all or a substantial portion of the contents of the database.
For the avoidance of doubt, this Section 4 supplements and does not
replace Your obligations under this Public License where the Licensed
Rights include other Copyright and Similar Rights.
Section 5 -- Disclaimer of Warranties and Limitation of Liability.
a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT
ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.
b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR
IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.
c. The disclaimer of warranties and limitation of liability provided
above shall be interpreted in a manner that, to the extent
possible, most closely approximates an absolute disclaimer and
waiver of all liability.
Section 6 -- Term and Termination.
a. This Public License applies for the term of the Copyright and
Similar Rights licensed here. However, if You fail to comply with
this Public License, then Your rights under this Public License
terminate automatically.
b. Where Your right to use the Licensed Material has terminated under
Section 6(a), it reinstates:
1. automatically as of the date the violation is cured, provided
it is cured within 30 days of Your discovery of the
violation; or
2. upon express reinstatement by the Licensor.
For the avoidance of doubt, this Section 6(b) does not affect any
right the Licensor may have to seek remedies for Your violations
of this Public License.
c. For the avoidance of doubt, the Licensor may also offer the
Licensed Material under separate terms or conditions or stop
distributing the Licensed Material at any time; however, doing so
will not terminate this Public License.
d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
License.
Section 7 -- Other Terms and Conditions.
a. The Licensor shall not be bound by any additional or different
terms or conditions communicated by You unless expressly agreed.
b. Any arrangements, understandings, or agreements regarding the
Licensed Material not stated herein are separate from and
independent of the terms and conditions of this Public License.
Section 8 -- Interpretation.
a. For the avoidance of doubt, this Public License does not, and
shall not be interpreted to, reduce, limit, restrict, or impose
conditions on any use of the Licensed Material that could lawfully
be made without permission under this Public License.
b. To the extent possible, if any provision of this Public License is
deemed unenforceable, it shall be automatically reformed to the
minimum extent necessary to make it enforceable. If the provision
cannot be reformed, it shall be severed from this Public License
without affecting the enforceability of the remaining terms and
conditions.
c. No term or condition of this Public License will be waived and no
failure to comply consented to unless expressly agreed to by the
Licensor.
d. Nothing in this Public License constitutes or may be interpreted
as a limitation upon, or waiver of, any privileges and immunities
that apply to the Licensor or You, including from the legal
processes of any jurisdiction or authority.
=======================================================================
Creative Commons is not a party to its public
licenses. Notwithstanding, Creative Commons may elect to apply one of
its public licenses to material it publishes and in those instances
will be considered the “Licensor.” The text of the Creative Commons
public licenses is dedicated to the public domain under the CC0 Public
Domain Dedication. Except for the limited purpose of indicating that
material is shared under a Creative Commons public license or as
otherwise permitted by the Creative Commons policies published at
creativecommons.org/policies, Creative Commons does not authorize the
use of the trademark "Creative Commons" or any other trademark or logo
of Creative Commons without its prior written consent including,
without limitation, in connection with any unauthorized modifications
to any of its public licenses or any other arrangements,
understandings, or agreements concerning use of licensed material. For
the avoidance of doubt, this paragraph does not form part of the
public licenses.
Creative Commons may be contacted at creativecommons.org.
================================================
FILE: Makefile
================================================
default: dev
d:dev
dev:
hugo serve
b:build
build:
hugo build
.PHONY: default d dev b build
# generate zh-tw version
translate:
bin/zh-tw.py
epub:
bin/epub
.PHONY: default doc translate
================================================
FILE: README.md
================================================
# 设计数据密集型应用（第二版） - 中文翻译版
[](https://ddia.vonng.com)
[](https://ddia.vonng.com/v1)
[](https://star-history.com/#Vonng/ddia&Date)
**作者**： [Martin Kleppmann](https://martin.kleppmann.com)，[《Designing Data-Intensive Applications 2nd Edition》](https://learning.oreilly.com/library/view/designing-data-intensive-applications/9781098119058/ch01.html)： 英国剑桥大学分布式系统研究员，演讲者，博主和开源贡献者，软件工程师和企业家，曾在 LinkedIn 和 Rapportive 负责数据基础架构。
**译者**：[冯若航](https://vonng.com) / [Vonng](https://github.com/Vonng) (rh@vonng.com) [Pigsty](https://pgsty.com) 创始人，[活跃](https://committers.top/china)[开源贡献者](https://gitstar-ranking.com/Vonng)，PostgreSQL Hacker。开源 RDS PG 发行版 [Pigsty](https://pigsty.cc/zh/) 与公众号《[老冯云数](https://mp.weixin.qq.com/s/p4Ys10ZdEDAuqNAiRmcnIQ)》作者，[数据库老司机](https://pigsty.cc/zh/blog/db)，[云计算泥石流](https://pigsty.cc/zh/blog/cloud)，曾于阿里，苹果，探探担任架构师与DBA。
**校订**： [@yingang](https://github.com/yingang) ｜ [**繁體中文**](content/tw/_index.md) by [@afunTW](https://github.com/afunTW) ｜ [完整贡献者列表](#贡献)
**阅读**：访问 [https://ddia.vonng.com](https://ddia.vonng.com) 阅读本书在线版本，或使用 [hugo](https://gohugo.io/documentation/) / [hextra](https://imfing.github.io/hextra/zh-cn/) 主题自行构建。
> [!NOTE]
> [**DDIA 第二版**](https://ddia.vonng.com) 正在翻译中（翻译至至第十章），欢迎阅览并提出您的宝贵意见！[点击此处阅览第一版](https://ddia.vonng.com/v1)。
---------
## 译序
> 不懂数据库的全栈工程师不是好架构师
>
> —— 冯若航 / Vonng
现今，尤其是在互联网领域，大多数应用都属于数据密集型应用。本书从底层数据结构到顶层架构设计，将数据系统设计中的精髓娓娓道来。其中的宝贵经验无论是对架构师、DBA、还是后端工程师、甚至产品经理都会有帮助。
这是一本理论结合实践的书，书中很多问题，译者在实际场景中都曾遇到过，读来让人击节扼腕。如果能早点读到这本书，该少走多少弯路啊！
这也是一本深入浅出的书，讲述概念的来龙去脉而不是卖弄定义，介绍事物发展演化历程而不是事实堆砌，将复杂的概念讲述的浅显易懂，但又直击本质不失深度。每章最后的引用质量非常好，是深入学习各个主题的绝佳索引。
本书为数据系统的设计、实现、与评价提供了很好的概念框架。读完并理解本书内容后，读者可以轻松看破大多数的技术忽悠，与技术砖家撕起来虎虎生风🤣。
这是 2017 年译者读过最好的一本技术类书籍，这么好的书没有中文翻译，实在是遗憾。某不才，愿为先进技术文化的传播贡献一份力量。既可以深入学习有趣的技术主题，又可以锻炼中英文语言文字功底，何乐而不为？
---------
## 前言
> 在我们的社会中，技术是一种强大的力量。数据、软件、通信可以用于坏的方面：不公平的阶级固化，损害公民权利，保护既得利益集团。但也可以用于好的方面：让底层人民发出自己的声音，让每个人都拥有机会，避免灾难。本书献给所有将技术用于善途的人们。
---------
> 计算是一种流行文化，流行文化鄙视历史。流行文化关乎个体身份和参与感，但与合作无关。流行文化活在当下，也与过去和未来无关。我认为大部分（为了钱）编写代码的人就是这样的，他们不知道自己的文化来自哪里。
>
> —— 阿兰・凯接受 Dobb 博士的杂志采访时（2012 年）
---------
## 目录
* [序言](https://ddia.vonng.com/preface)
* [第一部分：数据系统基础](https://ddia.vonng.com//part-i)
- [1. 数据系统架构中的权衡](https://ddia.vonng.com/ch1)
- [2. 定义非功能性需求](https://ddia.vonng.com/ch2)
- [3. 数据模型与查询语言](https://ddia.vonng.com/ch3)
- [4. 存储与检索](https://ddia.vonng.com/ch4)
- [5. 编码与演化](https://ddia.vonng.com/ch5)
* [第二部分：分布式数据](https://ddia.vonng.com/part-ii)
- [6. 复制](https://ddia.vonng.com/ch6)
- [7. 分片](https://ddia.vonng.com/ch7)
- [8. 事务](https://ddia.vonng.com/ch8)
- [9. 分布式系统的麻烦](https://ddia.vonng.com/ch9)
- [10.一致性与共识](https://ddia.vonng.com/ch10)
* [第三部分：派生数据](https://ddia.vonng.com/part-iii)
- [11. 批处理](https://ddia.vonng.com/ch11)
- [12. 流处理](https://ddia.vonng.com/ch12)
- [13. 流处理系统哲学](https://ddia.vonng.com/ch13)
- [14. 做正确的事](https://ddia.vonng.com/ch14)
* [术语表](https://ddia.vonng.com/glossary)
* [后记](https://ddia.vonng.com/colophon)
---------
## 法律声明
从原作者处得知，已经有简体中文的翻译计划，将于 2018 年末完成。[购买地址](https://search.jd.com/Search?keyword=设计数据密集型应用)
译者纯粹出于 **学习目的** 与 **个人兴趣** 翻译本书，不追求任何经济利益。
译者保留对此版本译文的署名权，其他权利以原作者和出版社的主张为准。
本译文只供学习研究参考之用，不得公开发行或用于商业用途，有能力阅读英文书籍者请购买正版支持，本书英文原版在 [O'REILLY](https://learning.oreilly.com/api/v1/continue/9781098119058/) 平台上提供在线免费试预览。
---------
## 贡献
0. 全文校订 by [@yingang](https://github.com/Vonng/ddia/commits?author=yingang)
1. [序言初翻修正](https://github.com/Vonng/ddia/commit/afb5edab55c62ed23474149f229677e3b42dfc2c) by [@seagullbird](https://github.com/Vonng/ddia/commits?author=seagullbird)
2. [第一章语法标点校正](https://github.com/Vonng/ddia/commit/973b12cd8f8fcdf4852f1eb1649ddd9d187e3644) by [@nevertiree](https://github.com/Vonng/ddia/commits?author=nevertiree)
3. [第六章部分校正](https://github.com/Vonng/ddia/commit/d4eb0852c0ec1e93c8aacc496c80b915bb1e6d48) 与[第十章的初翻](https://github.com/Vonng/ddia/commit/9de8dbd1bfe6fbb03b3bf6c1a1aa2291aed2490e) by [@MuAlex](https://github.com/Vonng/ddia/commits?author=MuAlex)
4. 第一部分前言，ch2 校正 by [@jiajiadebug](https://github.com/Vonng/ddia/commits?author=jiajiadebug)
5. 词汇表、后记关于野猪的部分 by [@Chowss](https://github.com/Vonng/ddia/commits?author=Chowss)
6. 繁體中文版本与转换脚本 by [@afunTW](https://github.com/afunTW)
7. 多处翻译修正 by [@songzhibin97](https://github.com/Vonng/ddia/commits?author=songzhibin97) [@MamaShip](https://github.com/Vonng/ddia/commits?author=MamaShip) [@FangYuan33](https://github.com/Vonng/ddia/commits?author=FangYuan33)
8. 感谢所有作出贡献，提出意见的朋友们：
<details>
<summary><a href="https://github.com/Vonng/ddia/pulls">Pull Requests</a> & <a href="https://github.com/Vonng/ddia/issues">Issues</a></summary>
| ISSUE & Pull Requests | USER | Title |
|-------------------------------------------------|------------------------------------------------------------|----------------------------------------------------------------|
| [386](https://github.com/Vonng/ddia/pull/386) | [@uncle-lv](https://github.com/uncle-lv) | ch2: 优化一处翻译 |
| [384](https://github.com/Vonng/ddia/pull/384) | [@PanggNOTlovebean](https://github.com/PanggNOTlovebean) | docs: 优化中文文档的措辞和表达 |
| [383](https://github.com/Vonng/ddia/pull/383) | [@PanggNOTlovebean](https://github.com/PanggNOTlovebean) | docs: 修正 ch4 中的术语和表达错误 |
| [382](https://github.com/Vonng/ddia/pull/382) | [@uncle-lv](https://github.com/uncle-lv) | ch1: 优化一处翻译 |
| [381](https://github.com/Vonng/ddia/pull/381) | [@Max-Tortoise](https://github.com/Max-Tortoise) | ch4: 修正一处术语不完整问题 |
| [377](https://github.com/Vonng/ddia/pull/377) | [@huang06](https://github.com/huang06) | 优化翻译术语 |
| [375](https://github.com/Vonng/ddia/issues/375) | [@z-soulx](https://github.com/z-soulx) | 对于是否100%全中文翻译的必要性讨论？个人-没必要100%，特别是“名词”，有原单词更加适合it人员 |
| [371](https://github.com/Vonng/ddia/pull/371) | [@lewiszlw](https://github.com/lewiszlw) | CPU core -> CPU 核心 |
| [369](https://github.com/Vonng/ddia/pull/369) | [@bbwang-gl](https://github.com/bbwang-gl) | ch7: 可串行化快照隔离检测一个事务何时修改另一个事务的读取 |
| [368](https://github.com/Vonng/ddia/pull/368) | [@yhao3](https://github.com/yhao3) | 更新 zh-tw.py 与 zh-tw 内容 |
| [367](https://github.com/Vonng/ddia/pull/367) | [@yhao3](https://github.com/yhao3) | 修正拼写、格式和标点问题 |
| [366](https://github.com/Vonng/ddia/pull/366) | [@yangshangde](https://github.com/yangshangde) | ch8: 将“电源失败”改为“电源失效” |
| [365](https://github.com/Vonng/ddia/pull/365) | [@xyohn](https://github.com/xyohn) | ch1: 优化“存储与计算分离”相关翻译 |
| [364](https://github.com/Vonng/ddia/issues/364) | [@xyohn](https://github.com/xyohn) | ch1: 优化“存储与计算分离”相关翻译 |
| [363](https://github.com/Vonng/ddia/pull/363) | [@xyohn](https://github.com/xyohn) | #362: 优化一处翻译 |
| [362](https://github.com/Vonng/ddia/issues/362) | [@xyohn](https://github.com/xyohn) | ch1: 优化一处翻译 |
| [359](https://github.com/Vonng/ddia/pull/359) | [@c25423](https://github.com/c25423) | ch10: 修正一处拼写错误 |
| [358](https://github.com/Vonng/ddia/pull/358) | [@lewiszlw](https://github.com/lewiszlw) | ch4: 修正一处拼写错误 |
| [356](https://github.com/Vonng/ddia/pull/356) | [@lewiszlw](https://github.com/lewiszlw) | ch2: 修正一处标点错误 |
| [355](https://github.com/Vonng/ddia/pull/355) | [@DuroyGeorge](https://github.com/DuroyGeorge) | ch12: 修正一处格式错误 |
| [354](https://github.com/Vonng/ddia/pull/354) | [@justlorain](https://github.com/justlorain) | ch7: 修正一处参考链接 |
| [353](https://github.com/Vonng/ddia/pull/353) | [@fantasyczl](https://github.com/fantasyczl) | ch3&9: 修正两处引用错误 |
| [352](https://github.com/Vonng/ddia/pull/352) | [@fantasyczl](https://github.com/fantasyczl) | 支持输出为 EPUB 格式 |
| [349](https://github.com/Vonng/ddia/pull/349) | [@xiyihan0](https://github.com/xiyihan0) | ch1: 修正一处格式错误 |
| [348](https://github.com/Vonng/ddia/pull/348) | [@omegaatt36](https://github.com/omegaatt36) | ch3: 修正一处图像链接 |
| [346](https://github.com/Vonng/ddia/issues/346) | [@Vermouth1995](https://github.com/Vermouth1995) | ch1: 优化一处翻译 |
| [343](https://github.com/Vonng/ddia/pull/343) | [@kehao-chen](https://github.com/kehao-chen) | ch10: 优化一处翻译 |
| [341](https://github.com/Vonng/ddia/pull/341) | [@YKIsTheBest](https://github.com/YKIsTheBest) | ch3: 优化两处翻译 |
| [340](https://github.com/Vonng/ddia/pull/340) | [@YKIsTheBest](https://github.com/YKIsTheBest) | ch2: 优化多处翻译 |
| [338](https://github.com/Vonng/ddia/pull/338) | [@YKIsTheBest](https://github.com/YKIsTheBest) | ch1: 优化一处翻译 |
| [335](https://github.com/Vonng/ddia/pull/335) | [@kimi0230](https://github.com/kimi0230) | 修正一处繁体中文错误 |
| [334](https://github.com/Vonng/ddia/pull/334) | [@soulrrrrr](https://github.com/soulrrrrr) | ch2: 修正一处繁体中文错误 |
| [332](https://github.com/Vonng/ddia/pull/332) | [@justlorain](https://github.com/justlorain) | ch5: 修正一处翻译错误 |
| [331](https://github.com/Vonng/ddia/pull/331) | [@Lyianu](https://github.com/Lyianu) | ch9: 更正几处拼写错误 |
| [330](https://github.com/Vonng/ddia/pull/330) | [@Lyianu](https://github.com/Lyianu) | ch7: 优化一处翻译 |
| [329](https://github.com/Vonng/ddia/issues/329) | [@Lyianu](https://github.com/Lyianu) | ch6: 指出一处翻译错误 |
| [328](https://github.com/Vonng/ddia/pull/328) | [@justlorain](https://github.com/justlorain) | ch4: 更正一处翻译遗漏 |
| [326](https://github.com/Vonng/ddia/pull/326) | [@liangGTY](https://github.com/liangGTY) | ch1: 优化一处翻译 |
| [323](https://github.com/Vonng/ddia/pull/323) | [@marvin263](https://github.com/marvin263) | ch5: 优化一处翻译 |
| [322](https://github.com/Vonng/ddia/pull/322) | [@marvin263](https://github.com/marvin263) | ch8: 优化一处翻译 |
| [304](https://github.com/Vonng/ddia/pull/304) | [@spike014](https://github.com/spike014) | ch11: 优化一处翻译 |
| [298](https://github.com/Vonng/ddia/pull/298) | [@Makonike](https://github.com/Makonike) | ch11&12: 修正两处错误 |
| [284](https://github.com/Vonng/ddia/pull/284) | [@WAangzE](https://github.com/WAangzE) | ch4: 更正一处列表错误 |
| [283](https://github.com/Vonng/ddia/pull/283) | [@WAangzE](https://github.com/WAangzE) | ch3: 更正一处错别字 |
| [282](https://github.com/Vonng/ddia/pull/282) | [@WAangzE](https://github.com/WAangzE) | ch2: 更正一处公式问题 |
| [281](https://github.com/Vonng/ddia/pull/281) | [@lyuxi99](https://github.com/lyuxi99) | 更正多处内部链接错误 |
| [280](https://github.com/Vonng/ddia/pull/280) | [@lyuxi99](https://github.com/lyuxi99) | ch9: 更正内部链接错误 |
| [279](https://github.com/Vonng/ddia/issues/279) | [@codexvn](https://github.com/codexvn) | ch9: 指出公式在 GitHub Pages 显示的问题 |
| [278](https://github.com/Vonng/ddia/pull/278) | [@LJlkdskdjflsa](https://github.com/LJlkdskdjflsa) | 发现了繁体中文版本中的错误翻译 |
| [275](https://github.com/Vonng/ddia/pull/275) | [@117503445](https://github.com/117503445) | 更正 LICENSE 链接 |
| [274](https://github.com/Vonng/ddia/pull/274) | [@uncle-lv](https://github.com/uncle-lv) | ch7: 修正错别字 |
| [273](https://github.com/Vonng/ddia/pull/273) | [@Sdot-Python](https://github.com/Sdot-Python) | ch7: 统一了 write skew 的翻译 |
| [271](https://github.com/Vonng/ddia/pull/271) | [@Makonike](https://github.com/Makonike) | ch6: 统一了 rebalancing 的翻译 |
| [270](https://github.com/Vonng/ddia/pull/270) | [@Ynjxsjmh](https://github.com/Ynjxsjmh) | ch7: 修正不一致的翻译 |
| [263](https://github.com/Vonng/ddia/pull/263) | [@zydmayday](https://github.com/zydmayday) | ch5: 修正译文中的重复单词 |
| [260](https://github.com/Vonng/ddia/pull/260) | [@haifeiWu](https://github.com/haifeiWu) | ch4: 修正部分不准确的翻译 |
| [258](https://github.com/Vonng/ddia/pull/258) | [@bestgrc](https://github.com/bestgrc) | ch3: 修正一处翻译错误 |
| [257](https://github.com/Vonng/ddia/pull/257) | [@UnderSam](https://github.com/UnderSam) | ch8: 修正一处拼写错误 |
| [256](https://github.com/Vonng/ddia/pull/256) | [@AlphaWang](https://github.com/AlphaWang) | ch7: 修正“可串行化”相关内容的多处翻译不当 |
| [255](https://github.com/Vonng/ddia/pull/255) | [@AlphaWang](https://github.com/AlphaWang) | ch7: 修正“可重复读”相关内容的多处翻译不当 |
| [253](https://github.com/Vonng/ddia/pull/253) | [@AlphaWang](https://github.com/AlphaWang) | ch7: 修正“读已提交”相关内容的多处翻译不当 |
| [246](https://github.com/Vonng/ddia/pull/246) | [@derekwu0101](https://github.com/derekwu0101) | ch3: 修正繁体中文的转译错误 |
| [245](https://github.com/Vonng/ddia/pull/245) | [@skyran1278](https://github.com/skyran1278) | ch12: 修正繁体中文的转译错误 |
| [244](https://github.com/Vonng/ddia/pull/244) | [@Axlgrep](https://github.com/Axlgrep) | ch9: 修正不通顺的翻译 |
| [242](https://github.com/Vonng/ddia/pull/242) | [@lynkeib](https://github.com/lynkeib) | ch9: 修正不通顺的翻译 |
| [241](https://github.com/Vonng/ddia/pull/241) | [@lynkeib](https://github.com/lynkeib) | ch8: 修正不正确的公式格式 |
| [240](https://github.com/Vonng/ddia/pull/240) | [@8da2k](https://github.com/8da2k) | ch9: 修正不通顺的翻译 |
| [239](https://github.com/Vonng/ddia/pull/239) | [@BeBraveBeCurious](https://github.com/BeBraveBeCurious) | ch7: 修正不一致的翻译 |
| [237](https://github.com/Vonng/ddia/pull/237) | [@zhangnew](https://github.com/zhangnew) | ch3: 修正错误的图片链接 |
| [229](https://github.com/Vonng/ddia/pull/229) | [@lis186](https://github.com/lis186) | 指出繁体中文的转译错误：复杂 |
| [226](https://github.com/Vonng/ddia/pull/226) | [@chroming](https://github.com/chroming) | ch1: 修正导航栏中的章节名称 |
| [220](https://github.com/Vonng/ddia/pull/220) | [@skyran1278](https://github.com/skyran1278) | ch9: 修正线性一致的繁体中文翻译 |
| [194](https://github.com/Vonng/ddia/pull/194) | [@BeBraveBeCurious](https://github.com/BeBraveBeCurious) | ch4: 修正错误的翻译 |
| [193](https://github.com/Vonng/ddia/pull/193) | [@BeBraveBeCurious](https://github.com/BeBraveBeCurious) | ch4: 优化译文 |
| [192](https://github.com/Vonng/ddia/pull/192) | [@BeBraveBeCurious](https://github.com/BeBraveBeCurious) | ch4: 修正不一致和不通顺的翻译 |
| [190](https://github.com/Vonng/ddia/pull/190) | [@Pcrab](https://github.com/Pcrab) | ch1: 修正不准确的翻译 |
| [187](https://github.com/Vonng/ddia/pull/187) | [@narojay](https://github.com/narojay) | ch9: 修正生硬的翻译 |
| [186](https://github.com/Vonng/ddia/pull/186) | [@narojay](https://github.com/narojay) | ch8: 修正错别字 |
| [185](https://github.com/Vonng/ddia/issues/185) | [@8da2k](https://github.com/8da2k) | 指出小标题跳转的问题 |
| [184](https://github.com/Vonng/ddia/pull/184) | [@DavidZhiXing](https://github.com/DavidZhiXing) | ch10: 修正失效的网址 |
| [183](https://github.com/Vonng/ddia/pull/183) | [@OneSizeFitsQuorum](https://github.com/OneSizeFitsQuorum) | ch8: 修正错别字 |
| [182](https://github.com/Vonng/ddia/issues/182) | [@lroolle](https://github.com/lroolle) | 建议docsify的主题风格 |
| [181](https://github.com/Vonng/ddia/pull/181) | [@YunfengGao](https://github.com/YunfengGao) | ch2: 修正翻译错误 |
| [180](https://github.com/Vonng/ddia/pull/180) | [@skyran1278](https://github.com/skyran1278) | ch3: 指出繁体中文的转译错误 |
| [177](https://github.com/Vonng/ddia/pull/177) | [@exzhawk](https://github.com/exzhawk) | 支持 Github Pages 里的公式显示 |
| [176](https://github.com/Vonng/ddia/pull/176) | [@haifeiWu](https://github.com/haifeiWu) | ch2: 语义网相关翻译更正 |
| [175](https://github.com/Vonng/ddia/pull/175) | [@cwr31](https://github.com/cwr31) | ch7: 不变式相关翻译更正 |
| [174](https://github.com/Vonng/ddia/pull/174) | [@BeBraveBeCurious](https://github.com/BeBraveBeCurious) | README & preface: 更正不正确的中文用词和标点符号 |
| [173](https://github.com/Vonng/ddia/pull/173) | [@ZvanYang](https://github.com/ZvanYang) | ch12: 修正不完整的翻译 |
| [171](https://github.com/Vonng/ddia/pull/171) | [@ZvanYang](https://github.com/ZvanYang) | ch12: 修正重复的译文 |
| [169](https://github.com/Vonng/ddia/pull/169) | [@ZvanYang](https://github.com/ZvanYang) | ch12: 更正不太通顺的翻译 |
| [166](https://github.com/Vonng/ddia/pull/166) | [@bp4m4h94](https://github.com/bp4m4h94) | ch1: 发现错误的文献索引 |
| [164](https://github.com/Vonng/ddia/pull/164) | [@DragonDriver](https://github.com/DragonDriver) | preface: 更正错误的标点符号 |
| [163](https://github.com/Vonng/ddia/pull/163) | [@llmmddCoder](https://github.com/llmmddCoder) | ch1: 更正错误字 |
| [160](https://github.com/Vonng/ddia/pull/160) | [@Zhayhp](https://github.com/Zhayhp) | ch2: 建议将 network model 翻译为网状模型 |
| [159](https://github.com/Vonng/ddia/pull/159) | [@1ess](https://github.com/1ess) | ch4: 更正错误字 |
| [157](https://github.com/Vonng/ddia/pull/157) | [@ZvanYang](https://github.com/ZvanYang) | ch7: 更正不太通顺的翻译 |
| [155](https://github.com/Vonng/ddia/pull/155) | [@ZvanYang](https://github.com/ZvanYang) | ch7: 更正不太通顺的翻译 |
| [153](https://github.com/Vonng/ddia/pull/153) | [@DavidZhiXing](https://github.com/DavidZhiXing) | ch9: 修正缩略图的错别字 |
| [152](https://github.com/Vonng/ddia/pull/152) | [@ZvanYang](https://github.com/ZvanYang) | ch7: 除重->去重 |
| [151](https://github.com/Vonng/ddia/pull/151) | [@ZvanYang](https://github.com/ZvanYang) | ch5: 修订sibling相关的翻译 |
| [147](https://github.com/Vonng/ddia/pull/147) | [@ZvanYang](https://github.com/ZvanYang) | ch5: 更正一处不准确的翻译 |
| [145](https://github.com/Vonng/ddia/pull/145) | [@Hookey](https://github.com/Hookey) | 识别了当前简繁转译过程中处理不当的地方，暂通过转换脚本规避 |
| [144](https://github.com/Vonng/ddia/issues/144) | [@secret4233](https://github.com/secret4233) | ch7: 不翻译`next-key locking` |
| [143](https://github.com/Vonng/ddia/issues/143) | [@imcheney](https://github.com/imcheney) | ch3: 更新残留的机翻段落 |
| [142](https://github.com/Vonng/ddia/issues/142) | [@XIJINIAN](https://github.com/XIJINIAN) | 建议去除段首的制表符 |
| [141](https://github.com/Vonng/ddia/issues/141) | [@Flyraty](https://github.com/Flyraty) | ch5: 发现一处错误格式的章节引用 |
| [140](https://github.com/Vonng/ddia/pull/140) | [@Bowser1704](https://github.com/Bowser1704) | ch5: 修正章节Summary中多处不通顺的翻译 |
| [139](https://github.com/Vonng/ddia/pull/139) | [@Bowser1704](https://github.com/Bowser1704) | ch2&ch3: 修正多处不通顺的或错误的翻译 |
| [137](https://github.com/Vonng/ddia/pull/137) | [@fuxuemingzhu](https://github.com/fuxuemingzhu) | ch5&ch6: 优化多处不通顺的或错误的翻译 |
| [134](https://github.com/Vonng/ddia/pull/134) | [@fuxuemingzhu](https://github.com/fuxuemingzhu) | ch4: 优化多处不通顺的或错误的翻译 |
| [133](https://github.com/Vonng/ddia/pull/133) | [@fuxuemingzhu](https://github.com/fuxuemingzhu) | ch3: 优化多处错误的或不通顺的翻译 |
| [132](https://github.com/Vonng/ddia/pull/132) | [@fuxuemingzhu](https://github.com/fuxuemingzhu) | ch3: 优化一处容易产生歧义的翻译 |
| [131](https://github.com/Vonng/ddia/pull/131) | [@rwwg4](https://github.com/rwwg4) | ch6: 修正两处错误的翻译 |
| [129](https://github.com/Vonng/ddia/pull/129) | [@anaer](https://github.com/anaer) | ch4: 修正两处强调文本和四处代码变量名称 |
| [128](https://github.com/Vonng/ddia/pull/128) | [@meilin96](https://github.com/meilin96) | ch5: 修正一处错误的引用 |
| [126](https://github.com/Vonng/ddia/pull/126) | [@cwr31](https://github.com/cwr31) | ch10: 修正一处错误的翻译（功能 -> 函数） |
| [125](https://github.com/Vonng/ddia/pull/125) | [@dch1228](https://github.com/dch1228) | ch2: 优化 how best 的翻译（如何以最佳方式） |
| [123](https://github.com/Vonng/ddia/pull/123) | [@yingang](https://github.com/yingang) | translation updates (chapter 9, TOC in readme, glossary, etc.) |
| [121](https://github.com/Vonng/ddia/pull/121) | [@yingang](https://github.com/yingang) | translation updates (chapter 5 to chapter 8) |
| [120](https://github.com/Vonng/ddia/pull/120) | [@jiong-han](https://github.com/jiong-han) | Typo fix: 呲之以鼻 -> 嗤之以鼻 |
| [119](https://github.com/Vonng/ddia/pull/119) | [@cclauss](https://github.com/cclauss) | Streamline file operations in convert() |
| [118](https://github.com/Vonng/ddia/pull/118) | [@yingang](https://github.com/yingang) | translation updates (chapter 2 to chapter 4) |
| [117](https://github.com/Vonng/ddia/pull/117) | [@feeeei](https://github.com/feeeei) | 统一每章的标题格式 |
| [115](https://github.com/Vonng/ddia/pull/115) | [@NageNalock](https://github.com/NageNalock) | 第七章病句修改: 重复词语 |
| [114](https://github.com/Vonng/ddia/pull/114) | [@Sunt-ing](https://github.com/Sunt-ing) | Update README.md: correct the book name |
| [113](https://github.com/Vonng/ddia/pull/113) | [@lpxxn](https://github.com/lpxxn) | 修改语句 |
| [112](https://github.com/Vonng/ddia/pull/112) | [@ibyte2011](https://github.com/ibyte2011) | Update ch9.md |
| [110](https://github.com/Vonng/ddia/pull/110) | [@lpxxn](https://github.com/lpxxn) | 读已写入数据 |
| [107](https://github.com/Vonng/ddia/pull/107) | [@abbychau](https://github.com/abbychau) | 單調鐘和好死还是赖活着 |
| [106](https://github.com/Vonng/ddia/pull/106) | [@enochii](https://github.com/enochii) | typo in ch2: fix braces typo |
| [105](https://github.com/Vonng/ddia/pull/105) | [@LiminCode](https://github.com/LiminCode) | Chronicle translation error |
| [104](https://github.com/Vonng/ddia/pull/104) | [@Sunt-ing](https://github.com/Sunt-ing) | several advice for better translation |
| [103](https://github.com/Vonng/ddia/pull/103) | [@Sunt-ing](https://github.com/Sunt-ing) | typo in ch4: should be 完成 rather than 完全 |
| [102](https://github.com/Vonng/ddia/pull/102) | [@Sunt-ing](https://github.com/Sunt-ing) | ch4: better-translation: 扼杀 → 破坏 |
| [101](https://github.com/Vonng/ddia/pull/101) | [@Sunt-ing](https://github.com/Sunt-ing) | typo in Ch4: should be "改变" rathr than "盖面" |
| [100](https://github.com/Vonng/ddia/pull/100) | [@LiminCode](https://github.com/LiminCode) | fix missing translation |
| [99 ](https://github.com/Vonng/ddia/pull/99) | [@mrdrivingduck](https://github.com/mrdrivingduck) | ch6: fix the word rebalancing |
| [98 ](https://github.com/Vonng/ddia/pull/98) | [@jacklightChen](https://github.com/jacklightChen) | fix ch7.md: fix wrong references |
| [97 ](https://github.com/Vonng/ddia/pull/97) | [@jenac](https://github.com/jenac) | 96 |
| [96 ](https://github.com/Vonng/ddia/pull/96) | [@PragmaTwice](https://github.com/PragmaTwice) | ch2: fix typo about 'may or may not be' |
| [95 ](https://github.com/Vonng/ddia/pull/95) | [@EvanMu96](https://github.com/EvanMu96) | fix translation of "the battle cry" in ch5 |
| [94 ](https://github.com/Vonng/ddia/pull/94) | [@kemingy](https://github.com/kemingy) | ch6: fix markdown and punctuations |
| [93 ](https://github.com/Vonng/ddia/pull/93) | [@kemingy](https://github.com/kemingy) | ch5: fix markdown and some typos |
| [92 ](https://github.com/Vonng/ddia/pull/92) | [@Gilbert1024](https://github.com/Gilbert1024) | Merge pull request #1 from Vonng/master |
| [88 ](https://github.com/Vonng/ddia/pull/88) | [@kemingy](https://github.com/kemingy) | fix typo for ch1, ch2, ch3, ch4 |
| [87 ](https://github.com/Vonng/ddia/pull/87) | [@wynn5a](https://github.com/wynn5a) | Update ch3.md |
| [86 ](https://github.com/Vonng/ddia/pull/86) | [@northmorn](https://github.com/northmorn) | Update ch1.md |
| [85 ](https://github.com/Vonng/ddia/pull/85) | [@sunbuhui](https://github.com/sunbuhui) | fix ch2.md: fix ch2 ambiguous translation |
| [84 ](https://github.com/Vonng/ddia/pull/84) | [@ganler](https://github.com/ganler) | Fix translation: use up |
| [83 ](https://github.com/Vonng/ddia/pull/83) | [@afunTW](https://github.com/afunTW) | Using OpenCC to convert from zh-cn to zh-tw |
| [82 ](https://github.com/Vonng/ddia/pull/82) | [@kangni](https://github.com/kangni) | fix gitbook url |
| [78 ](https://github.com/Vonng/ddia/pull/78) | [@hanyu2](https://github.com/hanyu2) | Fix unappropriated translation |
| [77 ](https://github.com/Vonng/ddia/pull/77) | [@Ozarklake](https://github.com/Ozarklake) | fix typo |
| [75 ](https://github.com/Vonng/ddia/pull/75) | [@2997ms](https://github.com/2997ms) | Fix typo |
| [74 ](https://github.com/Vonng/ddia/pull/74) | [@2997ms](https://github.com/2997ms) | Update ch9.md |
| [70 ](https://github.com/Vonng/ddia/pull/70) | [@2997ms](https://github.com/2997ms) | Update ch7.md |
| [67 ](https://github.com/Vonng/ddia/pull/67) | [@jiajiadebug](https://github.com/jiajiadebug) | fix issues in ch2 - ch9 and glossary |
| [66 ](https://github.com/Vonng/ddia/pull/66) | [@blindpirate](https://github.com/blindpirate) | Fix typo |
| [63 ](https://github.com/Vonng/ddia/pull/63) | [@haifeiWu](https://github.com/haifeiWu) | Update ch10.md |
| [62 ](https://github.com/Vonng/ddia/pull/62) | [@ych](https://github.com/ych) | fix ch1.md typesetting problem |
| [61 ](https://github.com/Vonng/ddia/pull/61) | [@xianlaioy](https://github.com/xianlaioy) | docs:钟-->种，去掉ou |
| [60 ](https://github.com/Vonng/ddia/pull/60) | [@Zombo1296](https://github.com/Zombo1296) | 否则 -> 或者 |
| [59 ](https://github.com/Vonng/ddia/pull/59) | [@AlexanderMisel](https://github.com/AlexanderMisel) | 呼叫->调用，显着->显著 |
| [58 ](https://github.com/Vonng/ddia/pull/58) | [@ibyte2011](https://github.com/ibyte2011) | Update ch8.md |
| [55 ](https://github.com/Vonng/ddia/pull/55) | [@saintube](https://github.com/saintube) | ch8: 修改链接错误 |
| [54 ](https://github.com/Vonng/ddia/pull/54) | [@Panmax](https://github.com/Panmax) | Update ch2.md |
| [53 ](https://github.com/Vonng/ddia/pull/53) | [@ibyte2011](https://github.com/ibyte2011) | Update ch9.md |
| [52 ](https://github.com/Vonng/ddia/pull/52) | [@hecenjie](https://github.com/hecenjie) | Update ch1.md |
| [51 ](https://github.com/Vonng/ddia/pull/51) | [@latavin243](https://github.com/latavin243) | fix 修正ch3 ch4几处翻译 |
| [50 ](https://github.com/Vonng/ddia/pull/50) | [@AlexZFX](https://github.com/AlexZFX) | 几个疏漏和格式错误 |
| [49 ](https://github.com/Vonng/ddia/pull/49) | [@haifeiWu](https://github.com/haifeiWu) | Update ch1.md |
| [48 ](https://github.com/Vonng/ddia/pull/48) | [@scaugrated](https://github.com/scaugrated) | fix typo |
| [47 ](https://github.com/Vonng/ddia/pull/47) | [@lzwill](https://github.com/lzwill) | Fixed typos in ch2 |
| [45 ](https://github.com/Vonng/ddia/pull/45) | [@zenuo](https://github.com/zenuo) | 删除一个多余的右括号 |
| [44 ](https://github.com/Vonng/ddia/pull/44) | [@akxxsb](https://github.com/akxxsb) | 修正第七章底部链接错误 |
| [43 ](https://github.com/Vonng/ddia/pull/43) | [@baijinping](https://github.com/baijinping) | "更假简单"->"更加简单" |
| [42 ](https://github.com/Vonng/ddia/pull/42) | [@tisonkun](https://github.com/tisonkun) | 修复 ch1 中的无序列表格式 |
| [38 ](https://github.com/Vonng/ddia/pull/38) | [@renjie-c](https://github.com/renjie-c) | 纠正多处的翻译小错误 |
| [37 ](https://github.com/Vonng/ddia/pull/37) | [@tankilo](https://github.com/tankilo) | fix translation mistakes in ch4.md |
| [36 ](https://github.com/Vonng/ddia/pull/36) | [@wwek](https://github.com/wwek) | 1.修复多个链接错误 2.名词优化修订 3.错误修订 |
| [35 ](https://github.com/Vonng/ddia/pull/35) | [@wwek](https://github.com/wwek) | fix ch7.md to ch8.md link error |
| [34 ](https://github.com/Vonng/ddia/pull/34) | [@wwek](https://github.com/wwek) | Merge pull request #1 from Vonng/master |
| [33 ](https://github.com/Vonng/ddia/pull/33) | [@wwek](https://github.com/wwek) | fix part-ii.md link error |
| [32 ](https://github.com/Vonng/ddia/pull/32) | [@JCYoky](https://github.com/JCYoky) | Update ch2.md |
| [31 ](https://github.com/Vonng/ddia/pull/31) | [@elsonLee](https://github.com/elsonLee) | Update ch7.md |
| [26 ](https://github.com/Vonng/ddia/pull/26) | [@yjhmelody](https://github.com/yjhmelody) | 修复一些明显错误 |
| [25 ](https://github.com/Vonng/ddia/pull/25) | [@lqbilbo](https://github.com/lqbilbo) | 修复链接错误 |
| [24 ](https://github.com/Vonng/ddia/pull/24) | [@artiship](https://github.com/artiship) | 修改词语顺序 |
| [23 ](https://github.com/Vonng/ddia/pull/23) | [@artiship](https://github.com/artiship) | 修正错别字 |
| [22 ](https://github.com/Vonng/ddia/pull/22) | [@artiship](https://github.com/artiship) | 纠正翻译错误 |
| [21 ](https://github.com/Vonng/ddia/pull/21) | [@zhtisi](https://github.com/zhtisi) | 修正目录和本章标题不符的情况 |
| [20 ](https://github.com/Vonng/ddia/pull/20) | [@rentiansheng](https://github.com/rentiansheng) | Update ch7.md |
| [19 ](https://github.com/Vonng/ddia/pull/19) | [@LHRchina](https://github.com/LHRchina) | 修复语句小bug |
| [16 ](https://github.com/Vonng/ddia/pull/16) | [@MuAlex](https://github.com/MuAlex) | Master |
| [15 ](https://github.com/Vonng/ddia/pull/15) | [@cg-zhou](https://github.com/cg-zhou) | Update translation progress |
| [14 ](https://github.com/Vonng/ddia/pull/14) | [@cg-zhou](https://github.com/cg-zhou) | Translate glossary |
| [13 ](https://github.com/Vonng/ddia/pull/13) | [@cg-zhou](https://github.com/cg-zhou) | 详细修改了后记中和印度野猪相关的描述 |
| [12 ](https://github.com/Vonng/ddia/pull/12) | [@ibyte2011](https://github.com/ibyte2011) | 修改了部分翻译 |
| [11 ](https://github.com/Vonng/ddia/pull/11) | [@jiajiadebug](https://github.com/jiajiadebug) | ch2 100% |
| [10 ](https://github.com/Vonng/ddia/pull/10) | [@jiajiadebug](https://github.com/jiajiadebug) | ch2 20% |
| [9 ](https://github.com/Vonng/ddia/pull/9) | [@jiajiadebug](https://github.com/jiajiadebug) | Preface, ch1, part-i translation minor fixes |
| [7 ](https://github.com/Vonng/ddia/pull/7) | [@MuAlex](https://github.com/MuAlex) | Ch6 translation pull request |
| [6 ](https://github.com/Vonng/ddia/pull/6) | [@MuAlex](https://github.com/MuAlex) | Ch6 change version1 |
| [5 ](https://github.com/Vonng/ddia/pull/5) | [@nevertiree](https://github.com/nevertiree) | Chapter 01语法微调 |
| [2 ](https://github.com/Vonng/ddia/pull/2) | [@seagullbird](https://github.com/seagullbird) | 序言初翻 |
</details><br>
---------
## 许可证
[](https://github.com/Vonng/ddia/blob/master/LICENSE)
本项目采用 [CC-BY 4.0](https://github.com/Vonng/ddia/blob/master/LICENSE) 许可证，您可以在这里找到完整说明：
- [署名 4.0 协议国际版 CC BY 4.0 Deed](https://creativecommons.org/licenses/by/4.0/deed.zh-hans)
- [Attribution 4.0 International CC BY 4.0](https://creativecommons.org/licenses/by/4.0/deed.en)
================================================
FILE: assets/css/custom.css
================================================
/* 调整左侧导航栏的宽度 */
/* 增加宽度以确保文本不换行 */
/* 左侧导航栏宽度调整 - 增加到 28rem 以避免换行 */
.hextra-sidebar {
width: 28rem !important;
min-width: 28rem !important;
}
/* 确保导航内容正确显示 */
.hextra-sidebar nav {
width: 100%;
}
/* 防止导航项文字换行 */
.hextra-sidebar li {
white-space: nowrap;
}
.sidebar-container {
width: 20rem !important;
white-space: nowrap;
}
/* 确保导航链接不换行 */
.hextra-sidebar a {
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
display: block;
}
/* 调整右侧页内目录（On This Page）的宽度 */
/* Hextra 默认宽度约为 16rem (256px)，增加 1.5 倍变为 24rem (384px) */
/* 右侧目录的宽度 */
.hextra-toc {
width: 24rem !important;
}
/* 确保目录内容正确显示 */
.hextra-toc nav {
width: 100%;
}
/* 调整目录项的文字换行以适应更宽的空间 */
.hextra-toc li {
word-wrap: break-word;
}
================================================
FILE: assets/css/example.css
================================================
.md-example { margin: 1.25rem 0; padding: 1rem; border: 1px solid var(--border); border-radius: .75rem; }
.md-example__caption { display: flex; gap: .5rem; align-items: baseline; margin-bottom: .5rem; }
.md-example__label { font-weight: 600; opacity: .85; }
.md-example__title { font-weight: 600; }
.md-example__anchor { margin-left: auto; text-decoration: none; opacity: .6; }
.md-example__anchor:hover { opacity: 1; }
.md-example__note { margin-top: .5rem; font-size: .95em; opacity: .85; }
================================================
FILE: bin/Pipfile
================================================
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
opencc = "*"
click = "*"
[dev-packages]
[requires]
python_version = "3.6"
================================================
FILE: bin/doc
================================================
#!/bin/bash
#==============================================================#
# File : doc
# Ctime : 2021-08-10
# Mtime : 2021-08-12
# Desc : Serve local doc with docsify, python3, python
# Path : bin/doc
# Deps : docsify or python3 or python2
# Copyright (C) 2018-2021 Ruohang Feng
#==============================================================#
PROG_DIR="$(cd $(dirname $0) && pwd)"
DOCS_DIR="$(cd $(dirname ${PROG_DIR}) && pwd)"
# node.js (docsify) > python3 (http.server) > python2 (SimpleHTTPServer)
if command -v docsify; then
echo "serve with docsify (click url to view in browser)"
cd ${DOCS_DIR} && docsify serve
elif command -v python3; then
echo "serve http://localhost:3001 (python3 http.server)"
cd ${DOCS_DIR} && python3 -m http.server 3001
elif command -v python2; then
echo "serve http://localhost:3001 (python2 SimpleHTTPServer)"
cd ${DOCS_DIR} && python2 -m SimpleHTTPServer 3001
else
echo "no available server"
fi
================================================
FILE: bin/epub
================================================
#!/usr/bin/env bash
set -e
# Set the directory containing Markdown files
SCRIPT_DIR=$(dirname "$0")
INPUT_DIR=$(cd "$(dirname "$SCRIPT_DIR")" && pwd)
OUTPUT_DIR="$INPUT_DIR/output"
TEMP_DIR="$OUTPUT_DIR/temp"
# Create output directory if it doesn't exist
mkdir -p "$OUTPUT_DIR"
mkdir -p "$TEMP_DIR"
# Preprocess Markdown files to convert Hugo shortcodes
echo "Preprocessing Markdown files..."
python3 "${SCRIPT_DIR}/preprocess-epub.py" "${INPUT_DIR}/content/zh" "$TEMP_DIR"
convert_to_epub() {
# convert all EPUB files into a single EPUB book
OUTPUT_BOOK="$OUTPUT_DIR/ddia.epub"
rm -f "$OUTPUT_BOOK"
echo "Converting all EPUB files into $OUTPUT_BOOK..."
local meta_file=${INPUT_DIR}/metadata.yaml
local css_file=${INPUT_DIR}/js/epub.css
pandoc -o "$OUTPUT_BOOK" --metadata-file="$meta_file" \
--toc-depth=2 \
--top-level-division=chapter \
--file-scope=true \
--css="$css_file" \
--webtex \
--wrap=preserve \
"${TEMP_DIR}"/_index.md \
"${TEMP_DIR}"/preface.md \
"${TEMP_DIR}"/part-i.md \
"${TEMP_DIR}"/ch1.md \
"${TEMP_DIR}"/ch2.md \
"${TEMP_DIR}"/ch3.md \
"${TEMP_DIR}"/ch4.md \
"${TEMP_DIR}"/part-ii.md \
"${TEMP_DIR}"/ch5.md \
"${TEMP_DIR}"/ch6.md \
"${TEMP_DIR}"/ch7.md \
"${TEMP_DIR}"/ch8.md \
"${TEMP_DIR}"/ch9.md \
"${TEMP_DIR}"/part-iii.md \
"${TEMP_DIR}"/ch10.md \
"${TEMP_DIR}"/ch11.md \
"${TEMP_DIR}"/ch12.md \
"${TEMP_DIR}"/ch13.md \
"${TEMP_DIR}"/ch14.md \
"${TEMP_DIR}"/colophon.md \
"${TEMP_DIR}"/glossary.md
echo "Converted EPUB book created at $OUTPUT_BOOK."
}
convert_to_epub
# Clean up temporary files
rm -rf "$TEMP_DIR"
================================================
FILE: bin/preprocess-epub.py
================================================
#!/usr/bin/env python3
"""
预处理 Markdown 文件，将 Hugo shortcode 转换为 Pandoc 可识别的格式
处理两种 shortcode：
1. {{< figure src="/fig/xxx.png" caption="xxx" >}} → 
2. {{< figure ... >}} (无 src) → 移除（通常用于代码示例）
"""
import os
import re
import sys
from pathlib import Path
FIGURE_SHORTCODE_RE = re.compile(r"\{\{<\s*figure\b(.*?)>\}\}", re.DOTALL)
ATTR_RE = re.compile(r'([\w-]+)="([^"]*)"')
ABS_IMAGE_RE = re.compile(r'!\[([^\]]*)\]\(/(?!static/)([^)]+)\)')
def _escape_alt_text(text):
"""Escape `]` in alt text to avoid breaking Markdown image syntax."""
return text.replace("]", r"\]")
def convert_markdown(text):
"""
转换 Hugo figure shortcode 和绝对路径图片引用。
Args:
text: Markdown 文本内容
Returns:
转换后的文本
"""
def replace_figure_shortcode(match):
attrs_text = match.group(1)
attrs = dict(ATTR_RE.findall(attrs_text))
src = attrs.get("src")
# 没有 src 的 figure 一般是代码示例占位，直接移除
if not src:
return ""
# 绝对路径资源转为相对 static 路径，便于 Pandoc 打包
if src.startswith('/'):
src = 'static' + src
# 优先 caption，fallback 到 title，至少保证图片可渲染
alt = _escape_alt_text(attrs.get("caption") or attrs.get("title") or "")
return f''
text = FIGURE_SHORTCODE_RE.sub(replace_figure_shortcode, text)
# 把 Markdown 里的绝对路径图片  转为 static/map/ch01.png
text = ABS_IMAGE_RE.sub(r'', text)
return text
def process_file(input_path, output_path):
"""
处理单个 Markdown 文件
Args:
input_path: 输入文件路径
output_path: 输出文件路径
"""
with open(input_path, 'r', encoding='utf-8') as f:
content = f.read()
# 转换内容
converted_content = convert_markdown(content)
# 写入输出文件
os.makedirs(os.path.dirname(output_path), exist_ok=True)
with open(output_path, 'w', encoding='utf-8') as f:
f.write(converted_content)
print(f"Processed: {input_path} -> {output_path}")
def main():
"""主函数"""
if len(sys.argv) < 2:
print("Usage: preprocess.py <input_file> [output_file]")
print(" or: preprocess.py <input_dir> <output_dir>")
sys.exit(1)
input_path = sys.argv[1]
if os.path.isfile(input_path):
# 处理单个文件
output_path = sys.argv[2] if len(sys.argv) > 2 else input_path
process_file(input_path, output_path)
elif os.path.isdir(input_path):
# 处理目录
output_dir = sys.argv[2]
input_dir = Path(input_path)
# 获取所有 .md 文件
md_files = sorted(input_dir.glob('*.md'))
for md_file in md_files:
output_file = os.path.join(output_dir, md_file.name)
process_file(str(md_file), output_file)
print(f"\nTotal processed: {len(md_files)} files")
else:
print(f"Error: {input_path} is not a valid file or directory")
sys.exit(1)
if __name__ == '__main__':
main()
================================================
FILE: bin/toc.py
================================================
#!/usr/bin/env python3
"""
TOC Generator for DDIA book
Usage: python toc.py <lang> <depth> [output_file]
Example: python toc.py zh 2
python toc.py en 3 en-toc.md
"""
import os
import sys
import re
from pathlib import Path
def extract_front_matter_title(content):
"""Extract title from Hugo front matter"""
lines = content.split('\n')
in_front_matter = False
for line in lines:
if line.strip() == '---':
if not in_front_matter:
in_front_matter = True
else:
break
elif in_front_matter and line.startswith('title:'):
# Extract title, removing quotes
title = line.split(':', 1)[1].strip()
if title.startswith('"') and title.endswith('"'):
title = title[1:-1]
elif title.startswith("'") and title.endswith("'"):
title = title[1:-1]
return title
return None
def extract_headings(content, max_depth):
"""Extract headings up to specified depth from markdown content
max_depth=1 -> extract H2 only
max_depth=2 -> extract H2-H3
max_depth=3 -> extract H2-H4
max_depth=4 -> extract H2-H5
"""
headings = []
lines = content.split('\n')
# Skip front matter
skip_until = 0
if lines[0].strip() == '---':
for i, line in enumerate(lines[1:], 1):
if line.strip() == '---':
skip_until = i + 1
break
for line in lines[skip_until:]:
# Match markdown headings with optional ID
# Format: ## Heading Text {#heading-id}
match = re.match(r'^(#{2,5})\s+(.*?)(?:\s*\{#([\w-]+)\})?$', line)
if match:
level = len(match.group(1))
# max_depth=1 -> extract level 2 only (H2)
# max_depth=2 -> extract level 2-3 (H2-H3)
# max_depth=3 -> extract level 2-4 (H2-H4)
# max_depth=4 -> extract level 2-5 (H2-H5)
max_level = max_depth + 1
if level <= max_level:
heading_text = match.group(2).strip()
heading_id = match.group(3)
headings.append({
'level': level, # Keep original level: 2 for H2, 3 for H3, etc.
'text': heading_text,
'id': heading_id
})
return headings
def generate_toc_entry(file_name, title, lang, depth, content_dir):
"""Generate TOC entry for a file"""
entries = []
# Determine URL path
base_name = file_name.replace('.md', '')
if lang == 'zh':
url = f"/{base_name}"
else:
url = f"/{lang}/{base_name}"
# Add main entry (level 1)
entries.append({
'level': 1,
'text': f"[{title}]({url})",
'raw_text': title
})
# Special case: glossary.md only shows main title (no sub-headings)
if file_name == 'glossary.md':
effective_depth = 0 # Don't extract any sub-headings
else:
effective_depth = depth - 1 # Adjust depth: user depth 1 = no extraction, 2 = extract H2, etc.
# If effective_depth >= 1, extract headings from file
if effective_depth >= 1:
file_path = content_dir / file_name
if file_path.exists():
with open(file_path, 'r', encoding='utf-8') as f:
content = f.read()
headings = extract_headings(content, effective_depth)
for heading in headings:
# Create link with anchor
if heading['id']:
anchor_url = f"{url}#{heading['id']}"
else:
# Generate anchor from heading text (simplified)
anchor = heading['text'].lower()
anchor = re.sub(r'[^\w\s-]', '', anchor)
anchor = re.sub(r'\s+', '-', anchor)
anchor_url = f"{url}#{anchor}"
# Adjust level: H2 becomes level 2, H3 becomes level 3, etc.
# This ensures proper indentation under the main entry
entries.append({
'level': heading['level'],
'text': f"[{heading['text']}]({anchor_url})",
'raw_text': heading['text']
})
return entries
def format_toc_entries(entries):
"""Format TOC entries with proper indentation"""
formatted = []
for entry in entries:
level = entry['level']
text = entry['text']
if level == 0:
# Blank line separator
formatted.append('')
elif level == 1:
# Main entry (chapter/section level)
formatted.append(f"## {text}")
elif level == 2:
# H2 heading
formatted.append(f"- {text}")
elif level == 3:
# H3 heading
formatted.append(f" - {text}")
elif level == 4:
# H4 heading
formatted.append(f" - {text}")
elif level == 5:
# H5 heading
formatted.append(f" - {text}")
return '\n'.join(formatted)
def check_file_status(file_path, lang):
"""Check if a file exists and add status marker if needed"""
if not file_path.exists():
return " (未发布)" if lang == 'zh' else " (未發布)" if lang == 'tw' else " (WIP)"
# Check if file has minimal content (you can adjust this logic)
with open(file_path, 'r', encoding='utf-8') as f:
content = f.read()
# Simple heuristic: if file has less than 500 characters of actual content, consider it WIP
# Remove front matter for content check
lines = content.split('\n')
if lines[0].strip() == '---':
for i, line in enumerate(lines[1:], 1):
if line.strip() == '---':
content = '\n'.join(lines[i+1:])
break
content_length = len(content.strip())
if content_length < 500:
return " (未发布)" if lang == 'zh' else " (未發布)" if lang == 'tw' else " (WIP)"
return ""
def main():
# Parse arguments
if len(sys.argv) < 3:
print("Usage: python toc.py <lang> <depth> [output_file]")
print("Example: python toc.py zh 2")
sys.exit(1)
lang = sys.argv[1]
if lang not in ['zh', 'en', 'tw']:
print(f"Error: Language must be one of: zh, en, tw")
sys.exit(1)
try:
depth = int(sys.argv[2])
if depth not in [1, 2, 3, 4]:
raise ValueError
except ValueError:
print(f"Error: Depth must be 1, 2, 3, or 4")
sys.exit(1)
# Determine output file
if len(sys.argv) > 3:
output_file = sys.argv[3]
else:
output_file = f"{lang}.md"
# Get content directory
script_dir = Path(__file__).parent
project_root = script_dir.parent
content_dir = project_root / 'content' / lang
if not content_dir.exists():
print(f"Error: Content directory {content_dir} does not exist")
sys.exit(1)
# Define file order
file_order = [
'preface.md',
'ch1.md', 'ch2.md', 'ch3.md', 'ch4.md', 'ch5.md', 'ch6.md',
'ch7.md', 'ch8.md', 'ch9.md', 'ch10.md', 'ch11.md', 'ch12.md', 'ch13.md',
'glossary.md',
'colophon.md'
]
# Generate TOC
all_entries = []
for file_name in file_order:
file_path = content_dir / file_name
if file_path.exists():
with open(file_path, 'r', encoding='utf-8') as f:
content = f.read()
title = extract_front_matter_title(content)
if title:
entries = generate_toc_entry(file_name, title, lang, depth, content_dir)
# Add status marker to main entry if needed
status = check_file_status(file_path, lang)
if status and entries:
# Update the first entry (main title) with status
entries[0]['text'] = entries[0]['text'].replace(')', f'){status}')
all_entries.extend(entries)
if entries: # Add blank line between chapters
all_entries.append({'level': 0, 'text': ''})
# Format and write output
formatted_toc = format_toc_entries(all_entries)
# Clean up extra blank lines
formatted_toc = re.sub(r'\n{3,}', '\n\n', formatted_toc)
# Write to file
output_path = Path(output_file)
with open(output_path, 'w', encoding='utf-8') as f:
f.write(formatted_toc)
print(f"TOC generated successfully: {output_path}")
print(f"Language: {lang}, Depth: {depth}")
if __name__ == "__main__":
main()
================================================
FILE: bin/translate.py
================================================
"""Convert zh-cn to zh-tw
Refer to https://github.com/BYVoid/OpenCC
"""
import click
import opencc
from pathlib import Path
from pprint import pprint
@click.group()
def cli():
pass
def convert(infile: str, outfile: str, cfg: str):
"""read >> convert >> write file
Args:
infile (str): input file
outfile (str): output file
cfg (str): config
"""
converter = opencc.OpenCC(cfg)
with open(infile, "r") as inf, open(outfile, "w+") as outf:
outf.write("\n".join(converter.convert(line) for line in inf))
print(f"Convert to {outfile}")
@cli.command()
@click.option("-i", "--input", "infile", required=True)
@click.option("-o", "--output", "outfile", required=True)
@click.option("-c", "--config", "cfg", required=True, default="s2twp.json")
def file(infile: str, outfile: str, cfg: str):
"""read >> convert >> write file
Args:
infile (str): input file
outfile (str): output file
cfg (str): config
"""
convert(infile, outfile, cfg)
@cli.command()
@click.option("-i", "--input", "infolder", required=True)
@click.option("-o", "--output", "outfolder", required=True)
@click.option("-c", "--config", "cfg", required=True, default="s2twp.json")
def repo(infolder, outfolder, cfg):
if not Path(outfolder).exists():
Path(outfolder).mkdir(parents=True)
print(f"Create {outfolder}")
infiles = Path(infolder).resolve().glob("*.md")
pair = [
{"infile": str(infile), "outfile": str(Path(outfolder).resolve() / infile.name)}
for idx, infile in enumerate(infiles)
]
for p in pair:
convert(p["infile"], p["outfile"], cfg)
if __name__ == "__main__":
cli()
================================================
FILE: bin/zh-tw.py
================================================
#!/usr/bin/env python3
import os, sys, opencc
import re
def process_urls(text, src_folder, dst_folder):
"""处理 Markdown 中的相对 URL"""
# 定义需要处理的页面路径（不带.md后缀）
page_paths = [
'/ch1', '/ch2', '/ch3', '/ch4', '/ch5', '/ch6',
'/ch7', '/ch8', '/ch9', '/ch10', '/ch11', '/ch12', '/ch13',
'/part-i', '/part-ii', '/part-iii',
'/preface', '/glossary', '/colophon'
]
# 对每个页面路径进行替换
for page_path in page_paths:
# 匹配 Markdown 链接格式 [text](page_path) 或 [text](page_path#anchor)
pattern = rf'\[([^\]]*)\]\(([^)]*)({re.escape(page_path)})(#[^)]*)?\)'
# 替换为添加 /tw 前缀的版本
def replace_func(match):
text_part = match.group(1)
folder_part = match.group(2) or ''
page_part = match.group(3)
anchor_part = match.group(4) or ''
if not folder_part:
return f'[{text_part}](/tw{page_part}{anchor_part})' # 默认中文版本，没有 /zh 前缀，直接在前面添加 /tw 前缀
elif folder_part[1:] == src_folder:
return f'[{text_part}](/{dst_folder}{page_part}{anchor_part})' # 其它中文版本，有类似 /v1 的前缀，根据输入参数进行替换
else:
text = f'[{text_part}]({folder_part}{page_part}{anchor_part})'
print(f'unknown folder part in: {text}, keep it unchanged')
return text
text = re.sub(pattern, replace_func, text)
return text
def convert_file(src_filepath, dst_filepath, src_folder, dst_folder, cfg='s2twp.json'):
print("convert %s to %s" % (src_filepath, dst_filepath))
converter = opencc.OpenCC(cfg)
with open(src_filepath, "r", encoding='utf-8') as src, open(dst_filepath, "w+", encoding='utf-8') as dst:
dst.write("\n".join(
process_urls(
converter.convert(line.rstrip())
.replace('一箇', '一個')
.replace('髮送', '傳送')
.replace('髮布', '釋出')
.replace('髮生', '發生')
.replace('髮出', '發出')
.replace('嚐試', '嘗試')
.replace('線上性一致', '在線性一致') # 优先按"在线"解析了？
.replace('復雜', '複雜')
.replace('討論瞭', '討論了')
.replace('瞭解釋', '了解釋')
.replace('瞭如', '了如') # 引入了如, 實現了如, 了如何, 了如果, 了如此
.replace('了如指掌', '瞭如指掌') # 针对上一行的例外情况
.replace('明瞭', '明了') # 闡明了, 聲明了, 指明了
.replace('倒黴', '倒楣')
.replace('區域性性', '區域性')
.replace('下麵條件', '下面條件') # 优先按"面条"解析了？
.replace('當日志', '當日誌') # 优先按"当日"解析了？
.replace('真即時間', '真實時間') # 优先按"实时"解析了？
.replace('面向物件', '物件導向')
.replace('非規範化', '反正規化')
.replace('規範化', '正規化'),
src_folder, dst_folder
)
for line in src))
def convert(zh_folder, tw_folder):
home = os.path.abspath(os.path.join(os.path.dirname(os.path.abspath(sys.argv[0])), '..'))
zh_dirpath = os.path.join(home, 'content', zh_folder)
tw_dirpath = os.path.join(home, 'content', tw_folder)
for file in os.listdir(zh_dirpath):
if file.endswith('.md'):
zh_filepath = os.path.join(zh_dirpath, file)
tw_filepath = os.path.join(tw_dirpath, file)
convert_file(zh_filepath, tw_filepath, zh_folder, tw_folder)
if __name__ == '__main__':
print(sys.argv)
convert('zh', 'tw')
convert('v1', 'v1_tw')
================================================
FILE: content/en/_index.md
================================================
---
title: "Designing Data-Intensive Applications 2nd Edition"
linkTitle: DDIA
cascade:
type: docs
breadcrumbs: false
---
—— **The Big Ideas Behind Reliable, Scalable, and Maintainable Systems**
[Martin Kleppmann](https://martin.kleppmann.com)
> The en-us version only includes **intro**, **summary**, **references** of all chapters to protect the intellectual property of author and publisher.
--------
*Technology is a powerful force in our society. Data, software, and communication can*
*be used for bad: to entrench unfair power structures, to undermine human rights, and to protect vested interests. But they can also be used for good: to make underrepresented people’s voices heard, to create opportunities for everyone, and to avert disasters. This book is dedicated to everyone working toward the good.*
---------
*Computing is pop culture. [...] Pop culture holds a disdain for history. Pop culture is all about identity and feeling like you’re participating. It has nothing to do with cooperation, the past or the future—it’s living in the present. I think the same is true of most people who write code for money. They have no idea where [their culture came from].*
— [Alan Kay](http://www.drdobbs.com/architecture-and-design/interview-with-alan-kay/240003442), in interview with *Dr Dobb’s Journal* (2012)
---------
## Table of Contents
### [Preface](/en/preface)
### [Part I: Foundations of Data Systems](/en/part-i)
- [1. Tradeoffs in Data Systems Architecture](/en/ch1)
- [2. Defining NonFunctional Requirements](/en/ch2)
- [3. Data Models and Query Languages](/en/ch3)
- [4. Storage and Retrieval](/en/ch4)
- [5. Encoding and Evolution](/en/ch5)
### [Part II: Distributed Data](/en/part-ii)
- [6. Replication](/en/ch6)
- [7. Partitioning](/en/ch7)
- [8. Transactions](/en/ch8)
- [9. The Trouble with Distributed Systems](/en/ch9)
- [10. Consistency and Consensus](/en/ch10)
### [Part III: Derived Data](/en/part-iii)
- [11. Batch Processing](/en/ch11)
- [12. Stream Processing](/en/ch12)
- [13. A Philosophy of Streaming Systems](/en/ch13)
- [14. Doing the Right Thing](/en/ch14)
### [Glossary](/en/glossary)
### [Colophon](/en/colophon)
================================================
FILE: content/en/ch1.md
================================================
---
title: "1. Trade-offs in Data Systems Architecture"
weight: 101
breadcrumbs: false
---
<a id="ch_tradeoffs"></a>
> *There are no solutions, there are only trade-offs. […] But you try to get the best
> trade-off you can get, and that’s all you can hope for.*
>
> [Thomas Sowell](https://www.youtube.com/watch?v=2YUtKr8-_Fg), Interview with Fred Barnes (2005)
> [!TIP] A NOTE FOR EARLY RELEASE READERS
> With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.
>
> This will be the 1st chapter of the final book. The GitHub repo for this book is https://github.com/ept/ddia2-feedback.
> If you’d like to be actively involved in reviewing and commenting on this draft, please reach out on GitHub.
Data is central to much application development today. With web and mobile apps, software as a
service (SaaS), and cloud services, it has become normal to store data from many different users in
a shared server-based data infrastructure. Data from user activity, business transactions, devices
and sensors needs to be stored and made available for analysis. As users interact with an
application, they both read the data that is stored, and also generate more data.
Small amounts of data, which can be stored and processed on a single machine, are often fairly easy
to deal with. However, as the data volume or the rate of queries grows, it needs to be distributed
across multiple machines, which introduces many challenges. As the needs of the application become
more complex, it is no longer sufficient to store everything in one system, but it might be
necessary to combine multiple storage or processing systems that provide different capabilities.
We call an application *data-intensive* if data management is one of the primary challenges in
developing the application [^1].
While in *compute-intensive* systems the challenge is parallelizing some very large computation, in
data-intensive applications we usually worry more about things like storing and processing large
data volumes, managing changes to data, ensuring consistency in the face of failures and
concurrency, and making sure services are highly available.
Such applications are typically built from standard building blocks that provide commonly needed
functionality. For example, many applications need to:
* Store data so that they, or another application, can find it again later (*databases*)
* Remember the result of an expensive operation, to speed up reads (*caches*)
* Allow users to search data by keyword or filter it in various ways (*search indexes*)
* Handle events and data changes as soon as they occur (*stream processing*)
* Periodically crunch a large amount of accumulated data (*batch processing*)
In building an application we typically take several software systems or services, such as databases
or APIs, and glue them together with some application code. If you are doing exactly what the data
systems were designed for, then this process can be quite easy.
However, as your application becomes more ambitious, challenges arise. There are many database
systems with different characteristics, suitable for different purposes—how do you choose which one
to use? There are various approaches to caching, several ways of building search indexes, and so
on—how do you reason about their trade-offs? You need to figure out which tools and which approaches
are the most appropriate for the task at hand, and it can be difficult to combine tools when you
need to do something that a single tool cannot do alone.
This book is a guide to help you make decisions about which technologies to use and how to combine
them. As you will see, there is no one approach that is fundamentally better than others; everything
has pros and cons. With this book, you will learn to ask the right questions to evaluate and compare
data systems, so that you can figure out which approach will best serve the needs of your particular
application.
We will start our journey by looking at some of the ways that data is typically used in
organizations today. Many of the ideas here have their origin in *enterprise software* (i.e., the
software needs and engineering practices of large organizations, such as big corporations and
governments), since historically, only large organizations had the large data volumes that required
sophisticated technical solutions. If your data volume is small enough, you can simply keep it in a
spreadsheet! However, more recently it has also become common for smaller companies and startups to
manage large data volumes and build data-intensive systems.
One of the key challenges with data systems is that different people need to do very different
things with data. If you are working at a company, you and your team will have one set of
priorities, while another team may have entirely different goals, even though you might be working
with the same dataset! Moreover, those goals might not be explicitly articulated, which can lead to
misunderstandings and disagreement about the right approach.
To help you understand what choices you can make, this chapter compares several contrasting
concepts, and explores their trade-offs:
* the difference between operational and analytical systems ([“Analytical versus Operational Systems”](/en/ch1#sec_introduction_analytics));
* pros and cons of cloud services and self-hosted systems ([“Cloud versus Self-Hosting”](/en/ch1#sec_introduction_cloud));
* when to move from single-node systems to distributed systems ([“Distributed versus Single-Node Systems”](/en/ch1#sec_introduction_distributed)); and
* balancing the needs of the business and the rights of the user ([“Data Systems, Law, and Society”](/en/ch1#sec_introduction_compliance)).
Moreover, this chapter will provide you with terminology that we will need for the rest of the book.
> [!TIP] TERMINOLOGY: FRONTENDS AND BACKENDS
Much of what we will discuss in this book relates to *backend development*. To explain that term:
for web applications, the client-side code (which runs in a web browser) is called the *frontend*,
and the server-side code that handles user requests is known as the *backend*. Mobile apps are
similar to frontends in that they provide user interfaces, which often communicate over the Internet
with a server-side backend. Frontends sometimes manage data locally on the user’s device [^2],
but the greatest data infrastructure challenges often lie in the backend: a frontend only needs to
handle one user’s data, whereas the backend manages data on behalf of *all* of the users.
A backend service is often reachable via HTTP (sometimes WebSocket); it usually consists of some
application code that reads and writes data in one or more databases, and sometimes interfaces with
additional data systems such as caches or message queues (which we might collectively call *data
infrastructure*). The application code is often *stateless* (i.e., when it finishes handling one
HTTP request, it forgets everything about that request), and any information that needs to persist
from one request to another needs to be stored either on the client, or in the server-side data
infrastructure.
## Analytical versus Operational Systems {#sec_introduction_analytics}
If you are working on data systems in an enterprise, you are likely to encounter several different
types of people who work with data. The first type are *backend engineers* who build services that
handle requests for reading and updating data; these services often serve external users, either
directly or indirectly via other services (see [“Microservices and Serverless”](/en/ch1#sec_introduction_microservices)). Sometimes
services are for internal use by other parts of the organization.
In addition to the teams managing backend services, two other groups of people typically require
access to an organization’s data: *business analysts*, who generate reports about the activities of
the organization in order to help the management make better decisions (*business intelligence* or
*BI*), and *data scientists*, who look for novel insights in data or who create user-facing product
features that are enabled by data analysis and machine learning/AI (for example, “people who bought
X also bought Y” recommendations on an e-commerce website, predictive analytics such as risk scoring
or spam filtering, and ranking of search results).
Although business analysts and data scientists tend to use different tools and operate in different
ways, they have some things in common: both perform *analytics*, which means they look at the data
that the users and backend services have generated, but they generally do not modify this data
(except perhaps for fixing mistakes). They might create derived datasets in which the original data
has been processed in some way. This has led to a split between two types of systems—a distinction
that we will use throughout this book:
* *Operational systems* consist of the backend services and data infrastructure where data is
created, for example by serving external users. Here, the application code both reads and modifies
the data in its databases, based on the actions performed by the users.
* *Analytical systems* serve the needs of business analysts and data scientists. They contain a
read-only copy of the data from the operational systems, and they are optimized for the types of
data processing that are needed for analytics.
As we shall see in the next section, operational and analytical systems are often kept separate, for
good reasons. As these systems have matured, two new specialized roles have emerged: *data
engineers* and *analytics engineers*. Data engineers are the people who know how to integrate the
operational and the analytical systems, and who take responsibility for the organization’s data
infrastructure more widely [^3].
Analytics engineers model and transform data to make it more useful for the business analysts and
data scientists in an organization [^4].
Many engineers specialize on either the operational or the analytical side. However, this book
covers both operational and analytical data systems, since both play an important role in the
lifecycle of data within an organization. We will explore in-depth the data infrastructure that is
used to deliver services both to internal and external users, so that you can work better with your
colleagues on the other side of this divide.
### Characterizing Transaction Processing and Analytics {#sec_introduction_oltp}
In the early days of business data processing, a write to the database typically corresponded to a
*commercial transaction* taking place: making a sale, placing an order with a supplier, paying an
employee’s salary, etc. As databases expanded into areas that didn’t involve money changing hands,
the term *transaction* nevertheless stuck, referring to a group of reads and writes that form a
logical unit.
> [!NOTE]
> [Chapter 8](/en/ch8#ch_transactions) explores in detail what we mean with a transaction. This chapter uses the term
> loosely to refer to low-latency reads and writes.
Even though databases started being used for many different kinds of data—posts on social media,
moves in a game, contacts in an address book, and many others—the basic access pattern
remained similar to processing business transactions. An operational system typically looks up a
small number of records by some key (this is called a *point query*). Records are inserted, updated,
or deleted based on the user’s input. Because these applications are interactive, this access
pattern became known as *online transaction processing* (OLTP).
However, databases also started being increasingly used for analytics, which has very different
access patterns compared to OLTP. Usually an analytic query scans over a huge number of records, and
calculates aggregate statistics (such as count, sum, or average) rather than returning the
individual records to the user. For example, a business analyst at a supermarket chain may want to
answer analytic queries such as:
* What was the total revenue of each of our stores in January?
* How many more bananas than usual did we sell during our latest promotion?
* Which brand of baby food is most often purchased together with brand X diapers?
The reports that result from these types of queries are important for business intelligence, helping
the management decide what to do next. In order to differentiate this pattern of using databases
from transaction processing, it has been called *online analytic processing* (OLAP) [^5].
The difference between OLTP and analytics is not always clear-cut, but some typical characteristics are listed in [Table 1-1](/en/ch1#tab_oltp_vs_olap).
{{< figure id="tab_oltp_vs_olap" title="Table 1-1. Comparing characteristics of operational and analytic systems" class="w-full my-4" >}}
| Property | Operational systems (OLTP) | Analytical systems (OLAP) |
|---------------------|-------------------------------------------------|-------------------------------------------|
| Main read pattern | Point queries (fetch individual records by key) | Aggregate over large number of records |
| Main write pattern | Create, update, and delete individual records | Bulk import (ETL) or event stream |
| Human user example | End user of web/mobile application | Internal analyst, for decision support |
| Machine use example | Checking if an action is authorized | Detecting fraud/abuse patterns |
| Type of queries | Fixed set of queries, predefined by application | Analyst can make arbitrary queries |
| Data represents | Latest state of data (current point in time) | History of events that happened over time |
| Dataset size | Gigabytes to terabytes | Terabytes to petabytes |
> [!NOTE]
> The meaning of *online* in *OLAP* is unclear; it probably refers to the fact that queries are not
> just for predefined reports, but that analysts use the OLAP system interactively for explorative
> queries.
With operational systems, users are generally not allowed to construct custom SQL queries and run
them on the database, since that would potentially allow them to read or modify data that they do
not have permission to access. Moreover, they might write queries that are expensive to execute, and
hence affect the database performance for other users. For these reasons, OLTP systems mostly run a
fixed set of queries that are baked into the application code, and use one-off custom queries only
occasionally for maintenance or troubleshooting. On the other hand, analytic databases usually give
their users the freedom to write arbitrary SQL queries by hand, or to generate queries automatically
using a data visualization or dashboard tool such as Tableau, Looker, or Microsoft Power BI.
There is also a type of systems that is designed for analytical workloads (queries that aggregate
over many records) but that are embedded into user-facing products. This category is known as
*product analytics* or *real-time analytics*, and systems designed for this type of use include
Pinot, Druid, and ClickHouse [^6].
### Data Warehousing {#sec_introduction_dwh}
At first, the same databases were used for both transaction processing and analytic queries. SQL
turned out to be quite flexible in this regard: it works well for both types of queries.
Nevertheless, in the late 1980s and early 1990s, there was a trend for companies to stop using their
OLTP systems for analytics purposes, and to run the analytics on a separate database system instead.
This separate database was called a *data warehouse*.
A large enterprise may have dozens, even hundreds, of online transaction processing systems:
systems powering the customer-facing website, controlling point of sale (checkout) systems in
physical stores, tracking inventory in warehouses, planning routes for vehicles, managing suppliers,
administering employees, and performing many other tasks. Each of these systems is complex and needs
a team of people to maintain it, so these systems end up operating mostly independently from each
other.
It is usually undesirable for business analysts and data scientists to directly query these OLTP
systems, for several reasons:
* the data of interest may be spread across multiple operational systems, making it difficult to
combine those datasets in a single query (a problem known as *data silos*);
* the kinds of schemas and data layouts that are good for OLTP are less well suited for analytics
(see [“Stars and Snowflakes: Schemas for Analytics”](/en/ch3#sec_datamodels_analytics));
* analytic queries can be quite expensive, and running them on an OLTP database would impact the
performance for other users; and
* the OLTP systems might reside in a separate network that users are not allowed direct access to
for security or compliance reasons.
A *data warehouse*, by contrast, is a separate database that analysts can query to their hearts’
content, without affecting OLTP operations [^7].
As we shall see in [Chapter 4](/en/ch4#ch_storage), data warehouses often store data in a way that is very different
from OLTP databases, in order to optimize for the types of queries that are common in analytics.
The data warehouse contains a read-only copy of the data in all the various OLTP systems in the
company. Data is extracted from OLTP databases (using either a periodic data dump or a continuous
stream of updates), transformed into an analysis-friendly schema, cleaned up, and then loaded into
the data warehouse. This process of getting data into the data warehouse is known as
*Extract–Transform–Load* (ETL) and is illustrated in [Figure 1-1](/en/ch1#fig_dwh_etl). Sometimes the order of the
*transform* and *load* steps is swapped (i.e., the transformation is done in the data warehouse,
after loading), resulting in *ELT*.
{{< figure src="/fig/ddia_0101.png" id="fig_dwh_etl" caption="Figure 1-1. Simplified outline of ETL into a data warehouse." class="w-full my-4" >}}
In some cases the data sources of the ETL processes are external SaaS products such as customer
relationship management (CRM), email marketing, or credit card processing systems. In those cases,
you do not have direct access to the original database, since it is accessible only via the software
vendor’s API. Bringing the data from these external systems into your own data warehouse can enable
analyses that are not possible via the SaaS API. ETL for SaaS APIs is often implemented by
specialist data connector services such as Fivetran, Singer, or AirByte.
Some database systems offer *hybrid transactional/analytic processing* (HTAP), which aims to enable
OLTP and analytics in a single system without requiring ETL from one system into another [^8] [^9].
However, many HTAP systems internally consist of an OLTP system coupled with a separate analytical
system, hidden behind a common interface—so the distinction between the two remains important for
understanding how these systems work.
Moreover, even though HTAP exists, it is common to have a separation between transactional and
analytic systems due to their different goals and requirements. In particular, it is considered good
practice for each operational system to have its own database (see
[“Microservices and Serverless”](/en/ch1#sec_introduction_microservices)), leading to hundreds of separate operational databases; on the
other hand, an enterprise usually has a single data warehouse, so that business analysts can combine
data from several operational systems in a single query.
HTAP therefore does not replace data warehouses. Rather, it is useful in scenarios where the same
application needs to both perform analytics queries that scan a large number of rows, and also
read and update individual records with low latency. Fraud detection can involve such workloads, for
example [^10].
The separation between operational and analytical systems is part of a wider trend: as workloads
have become more demanding, systems have become more specialized and optimized for particular
workloads. General-purpose systems can handle small data volumes comfortably, but the greater the
scale, the more specialized systems tend to become [^11].
#### From data warehouse to data lake {#from-data-warehouse-to-data-lake}
A data warehouse often uses a *relational* data model that is queried through SQL (see
[Chapter 3](/en/ch3#ch_datamodels)), perhaps using specialized business intelligence software. This model works well
for the types of queries that business analysts need to make, but it is less well suited to the
needs of data scientists, who might need to perform tasks such as:
* Transform data into a form that is suitable for training a machine learning model; often this
requires turning the rows and columns of a database table into a vector or matrix of numerical
values called *features*. The process of performing this transformation in a way that maximizes
the performance of the trained model is called *feature engineering*, and it often requires custom
code that is difficult to express using SQL.
* Take textual data (e.g., reviews of a product) and use natural language processing techniques to
try to extract structured information from it (e.g., the sentiment of the author, or which topics
they mention). Similarly, they might need to extract structured information from photos using
computer vision techniques.
Although there have been efforts to add machine learning operators to a SQL data model [^12]
and to build efficient machine learning systems on top of a relational foundation [^13],
many data scientists prefer not to work in a relational database such as a data warehouse. Instead,
many prefer to use Python data analysis libraries such as pandas and scikit-learn, statistical
analysis languages such as R, and distributed analytics frameworks such as Spark [^14].
We discuss these further in [“Dataframes, Matrices, and Arrays”](/en/ch3#sec_datamodels_dataframes).
Consequently, organizations face a need to make data available in a form that is suitable for use by
data scientists. The answer is a *data lake*: a centralized data repository that holds a copy of any
data that might be useful for analysis, obtained from operational systems via ETL processes. The
difference from a data warehouse is that a data lake simply contains files, without imposing any
particular file format or data model. Files in a data lake might be collections of database records,
encoded using a file format such as Avro or Parquet (see [Chapter 5](/en/ch5#ch_encoding)), but they can equally well
contain text, images, videos, sensor readings, sparse matrices, feature vectors, genome sequences,
or any other kind of data [^15].
Besides being more flexible, this is also often cheaper than relational data storage, since the data
lake can use commoditized file storage such as object stores (see [“Cloud-Native System Architecture”](/en/ch1#sec_introduction_cloud_native)).
ETL processes have been generalized to *data pipelines*, and in some cases the data lake has become
an intermediate stop on the path from the operational systems to the data warehouse. The data lake
contains data in a “raw” form produced by the operational systems, without the transformation into a
relational data warehouse schema. This approach has the advantage that each consumer of the data can
transform the raw data into a form that best suits their needs. It has been dubbed the *sushi
principle*: “raw data is better” [^16].
Besides loading data from a data lake into a separate data warehouse, it is also possible to run
typical data warehousing workloads (SQL queries and business analytics) directly on the files in the
data lake, alongside data science/machine learning workloads. This architecture is known as a *data
lakehouse*, and it requires a query execution engine and a metadata (e.g., schema management) layer
that extend the data lake’s file storage [^17].
Apache Hive, Spark SQL, Presto, and Trino are examples of this approach.
#### Beyond the data lake {#beyond-the-data-lake}
As analytics practices have matured, organizations have been increasingly paying attention to the
management and operations of analytics systems and data pipelines, as captured for example in the
DataOps manifesto [^18].
Part of this are issues of governance, privacy, and compliance with regulation such as GDPR and
CCPA, which we discuss in [“Data Systems, Law, and Society”](/en/ch1#sec_introduction_compliance) and [“Legislation and Self-Regulation”](/en/ch14#sec_future_legislation).
Moreover, analytical data is increasingly made available not only as files and relational tables,
but also as streams of events (see [Chapter 12](/en/ch12#ch_stream)). With file-based data analysis you can re-run the
analysis periodically (e.g., daily) in order to respond to changes in the data, but stream processing
allows analytics systems to respond to events much faster, on the order of seconds. Depending on the
application and how time-sensitive it is, a stream processing approach can be valuable, for example
to identify and block potentially fraudulent or abusive activity.
In some cases the outputs of analytics systems are made available to operational systems (a process
sometimes known as *reverse ETL* [^19]). For example, a machine-learning model that was trained on data in an analytics system may be deployed to
production, so that it can generate recommendations for end-users, such as “people who bought X also
bought Y”. Such deployed outputs of analytics systems are also known as *data products* [^20].
Machine learning models can be deployed to operational systems using specialized tools such as
TFX, Kubeflow, or MLflow.
### Systems of Record and Derived Data {#sec_introduction_derived}
Related to the distinction between operational and analytical systems, this book also distinguishes
between *systems of record* and *derived data systems*. These terms are useful because they can help
you clarify the flow of data through a system:
Systems of record
: A system of record, also known as *source of truth*, holds the authoritative or *canonical*
version of some data. When new data comes in, e.g., as user input, it is first written here. Each
fact is represented exactly once (the representation is typically *normalized*; see
[“Normalization, Denormalization, and Joins”](/en/ch3#sec_datamodels_normalization)). If there is any discrepancy between another system and the
system of record, then the value in the system of record is (by definition) the correct one.
Derived data systems
: Data in a derived system is the result of taking some existing data from another system and
transforming or processing it in some way. If you lose derived data, you can recreate it from the
original source. A classic example is a cache: data can be served from the cache if present, but
if the cache doesn’t contain what you need, you can fall back to the underlying database.
Denormalized values, indexes, materialized views, transformed data representations, and models
trained on a dataset also fall into this category.
Technically speaking, derived data is *redundant*, in the sense that it duplicates existing
information. However, it is often essential for getting good performance on read queries. You can
derive several different datasets from a single source, enabling you to look at the data from
different “points of view.”
Analytical systems are usually derived data systems, because they are consumers of data created
elsewhere. Operational services may contain a mixture of systems of record and derived data systems.
The systems of record are the primary databases to which data is first written, whereas the derived
data systems are the indexes and caches that speed up common read operations, especially for queries
that the system of record cannot answer efficiently.
Most databases, storage engines, and query languages are not inherently a system of record or a
derived system. A database is just a tool: how you use it is up to you. The distinction between
system of record and derived data system depends not on the tool, but on how you use it in your
application. By being clear about which data is derived from which other data, you can bring clarity
to an otherwise confusing system architecture.
When the data in one system is derived from the data in another, you need a process for updating the
derived data when the original in the system of record changes. Unfortunately, many databases are
designed based on the assumption that your application only ever needs to use that one database, and
they do not make it easy to integrate multiple systems in order to propagate such updates. In
[“Data Integration”](/en/ch13#sec_future_integration) we will discuss approaches to *data integration*, which allow us to compose multiple
data systems to achieve things that one system alone cannot do.
That brings us to the end of our comparison of analytics and transaction processing. In the next
section, we will examine another trade-off that you might have already seen debated multiple times.
## Cloud versus Self-Hosting {#sec_introduction_cloud}
With anything that an organization needs to do, one of the first questions is: should it be done
in-house, or should it be outsourced? Should you build or should you buy?
Ultimately, this is a question about business priorities. The received management wisdom is that
things that are a core competency or a competitive advantage of your organization should be done
in-house, whereas things that are non-core, routine, or commonplace should be left to a vendor [^21].
To give an extreme example, most companies do not generate their own electricity (unless they are an
energy company, and leaving aside emergency backup power), since it is cheaper to buy electricity from the grid.
With software, two important decisions to be made are who builds the software and who deploys it.
There is a spectrum of possibilities that outsource each decision to various degrees, as illustrated
in [Figure 1-2](/en/ch1#fig_cloud_spectrum). At one extreme is bespoke software that you write and run in-house; at
the other extreme are widely-used cloud services or Software as a Service (SaaS) products that are
implemented and operated by an external vendor, and which you only access through a web interface or API.
{{< figure src="/fig/ddia_0102.png" id="fig_cloud_spectrum" caption="Figure 1-2. A spectrum of types of software and its operations." class="w-full my-4" >}}
The middle ground is off-the-shelf software (open source or commercial) that you *self-host*, i.e.,
deploy yourself—for example, if you download MySQL and install it on a server you control. This
could be on your own hardware (often called *on-premises*, even if the server is actually in a
rented datacenter rack and not literally on your own premises), or on a virtual machine in the cloud
(*Infrastructure as a Service* or IaaS). There are still more points along this spectrum, e.g.,
taking open source software and running a modified version of it.
Separately from this spectrum there is also the question of *how* you deploy services, either in the
cloud or on-premises—for example, whether you use an orchestration framework such as Kubernetes.
However, choice of deployment tooling is out of scope of this book, since other factors have a
greater influence on the architecture of data systems.
### Pros and Cons of Cloud Services {#sec_introduction_cloud_tradeoffs}
Using a cloud service, rather than running comparable software yourself, essentially outsources the
operation of that software to the cloud provider. There are good arguments for and against cloud
services. Cloud providers claim that using their services saves you time and money, and allows you
to move faster compared to setting up your own infrastructure.
Whether a cloud service is actually cheaper and easier than self-hosting depends very much on your
skills and the workload on your systems. If you already have experience setting up and operating the
systems you need, and if your load is quite predictable (i.e., the number of machines you need does
not fluctuate wildly), then it’s often cheaper to buy your own machines and run the software on them
yourself [^22] [^23].
On the other hand, if you need a system that you don’t already know how to deploy and operate, then
adopting a cloud service is often easier and quicker than learning to manage the system yourself. If
you have to hire and train staff specifically to maintain and operate the system, that can get very
expensive. You still need an operations team when you’re using the cloud (see
[“Operations in the Cloud Era”](/en/ch1#sec_introduction_operations)), but outsourcing the basic system administration can free up your
team to focus on higher-level concerns.
When you outsource the operation of a system to a company that specializes in running that service,
that can potentially result in a better service, since the provider gains operational expertise from
providing the service to many customers. On the other hand, if you run the service yourself, you can
configure and tune it to perform well on your particular workload; it is unlikely that a cloud
service would be willing to make such customizations on your behalf.
Cloud services are particularly valuable if the load on your systems varies a lot over time. If you
provision your machines to be able to handle peak load, but those computing resources are idle most
of the time, the system becomes less cost-effective. In this situation, cloud services have the
advantage that they can make it easier to scale your computing resources up or down in response to
changes in demand.
For example, analytics systems often have extremely variable load: running a large analytical query
quickly requires a lot of computing resources in parallel, but once the query completes, those
resources sit idle until the user makes the next query. Predefined queries (e.g., for daily reports)
can be enqueued and scheduled to smooth out the load, but for interactive queries, the faster you
want them to complete, the more variable the workload becomes. If your dataset is so large that
querying it quickly requires significant computing resources, using the cloud can save money, since
you can return unused resources to the provider rather than leaving them idle. For smaller datasets,
this difference is less significant.
The biggest downside of a cloud service is that you have no control over it:
* If it is lacking a feature you need, all you can do is to politely ask the vendor whether they
will add it; you generally cannot implement it yourself.
* If the service goes down, all you can do is to wait for it to recover.
* If you are using the service in a way that triggers a bug or causes performance problems, it will
be difficult for you to diagnose the issue. With software that you run yourself, you can get
performance metrics and debugging information from the operating system to help you understand its
behavior, and you can look at the server logs, but with a service hosted by a vendor you usually
do not have access to these internals.
* Moreover, if the service shuts down or becomes unacceptably expensive, or if the vendor decides to
change their product in a way you don’t like, you are at their mercy—continuing to run an old
version of the software is usually not an option, so you will be forced to migrate to an
alternative service [^24].
This risk is mitigated if there are alternative services that expose a compatible API, but for
many cloud services there are no standard APIs, which raises the cost of switching, making vendor
lock-in a problem.
* The cloud provider needs to be trusted to keep the data secure, which can complicate the process
of complying with privacy and security regulations.
Despite all these risks, it has become more and more popular for organizations to build new
applications on top of cloud services, or adopting a hybrid approach in which cloud services are
used for some aspects of a system. However, cloud services will not subsume all in-house data
systems: many older systems predate the cloud, and for any services that have specialist
requirements that existing cloud services cannot meet, in-house systems remain necessary. For
example, very latency-sensitive applications such as high-frequency trading require full control of
the hardware.
### Cloud-Native System Architecture {#sec_introduction_cloud_native}
Besides having a different economic model (subscribing to a service instead of buying hardware and
licensing software to run on it), the rise of the cloud has also had a profound effect on how data
systems are implemented on a technical level. The term *cloud-native* is used to describe an
architecture that is designed to take advantage of cloud services.
In principle, almost any software that you can self-host could also be provided as a cloud service,
and indeed such managed services are now available for many popular data systems. However, systems
that have been designed from the ground up to be cloud-native have been shown to have several
advantages: better performance on the same hardware, faster recovery from failures, being able to
quickly scale computing resources to match the load, and supporting larger datasets [^25] [^26] [^27].
[Table 1-2](/en/ch1#tab_cloud_native_dbs) lists some examples of both types of systems.
{{< figure id="tab_cloud_native_dbs" title="Table 1-2. Examples of self-hosted and cloud-native database systems" class="w-full my-4" >}}
| Category | Self-hosted systems | Cloud-native systems |
|------------------|-----------------------------|-----------------------------------------------------------------------|
| Operational/OLTP | MySQL, PostgreSQL, MongoDB | AWS Aurora [^25], Azure SQL DB Hyperscale [^26], Google Cloud Spanner |
| Analytical/OLAP | Teradata, ClickHouse, Spark | Snowflake [^27], Google BigQuery, Azure Synapse Analytics |
#### Layering of cloud services {#layering-of-cloud-services}
Many self-hosted data systems have very simple system requirements: they run on a conventional
operating system such as Linux or Windows, they store their data as files on the filesystem, and
they communicate via standard network protocols such as TCP/IP. A few systems depend on special
hardware such as GPUs (for machine learning) or RDMA network interfaces, but on the whole,
self-hosted software tends to use very generic computing resources: CPU, RAM, a filesystem, and an IP network.
In a cloud, this type of software can be run on an Infrastructure-as-a-Service environment, using
one or more virtual machines (or *instances*) with a certain allocation of CPUs, memory, disk, and
network bandwidth. Compared to physical machines, cloud instances can be provisioned faster and they
come in a greater variety of sizes, but otherwise they are similar to a traditional computer: you
can run any software you like on it, but you are responsible for administering it yourself.
In contrast, the key idea of cloud-native services is to use not only the computing resources
managed by your operating system, but also to build upon lower-level cloud services to create
higher-level services. For example:
* *Object storage* services such as Amazon S3, Azure Blob Storage, and Cloudflare R2 store large
files. They provide more limited APIs than a typical filesystem (basic file reads and writes), but
they have the advantage that they hide the underlying physical machines: the service automatically
distributes the data across many machines, so that you don’t have to worry about running out of
disk space on any one machine. Even if some machines or their disks fail entirely, no data is
lost.
* Many other services are in turn built upon object storage and other cloud services: for example,
Snowflake is a cloud-based analytic database (data warehouse) that relies on S3 for data storage [^27],
and some other services in turn build upon Snowflake.
As always with abstractions in computing, there is no one right answer to what you should use. As a
general rule, higher-level abstractions tend to be more oriented towards particular use cases. If
your needs match the situations for which a higher-level system is designed, using the existing
higher-level system will probably provide what you need with much less hassle than building it
yourself from lower-level systems. On the other hand, if there is no high-level system that meets
your needs, then building it yourself from lower-level components is the only option.
#### Separation of storage and compute {#sec_introduction_storage_compute}
In traditional computing, disk storage is regarded as durable (we assume that once something is
written to disk, it will not be lost). To tolerate the failure of an individual hard disk, RAID
(Redundant Array of Independent Disks) is often used to maintain copies of the data on several
disks attached to the same machine. RAID can be performed either in hardware or in software by the
operating system, and it is transparent to the applications accessing the filesystem.
In the cloud, compute instances (virtual machines) may also have local disks attached, but
cloud-native systems typically treat these disks more like an ephemeral cache, and less like
long-term storage. This is because the local disk becomes inaccessible if the associated instance
fails, or if the instance is replaced with a bigger or a smaller one (on a different physical machine) in order to adapt to changes in load.
As an alternative to local disks, cloud services also offer virtual disk storage that can be
detached from one instance and attached to a different one (Amazon EBS, Azure managed disks, and
persistent disks in Google Cloud). Such a virtual disk is not actually a physical disk, but rather a
cloud service provided by a separate set of machines, which emulates the behavior of a disk (a
*block device*, where each block is typically 4 KiB in size). This technology makes it
possible to run traditional disk-based software in the cloud, but the block device emulation
introduces overheads that can be avoided in systems that are designed from the ground up for the cloud [^25]. It also makes the application
very sensitive to network glitches, since every I/O on the virtual block device is actually a network call [^28].
To address this problem, cloud-native services generally avoid using virtual disks, and instead
build on dedicated storage services that are optimized for particular workloads. Object storage
services such as S3 are designed for long-term storage of fairly large files, ranging from hundreds
of kilobytes to several gigabytes in size. The individual rows or values stored in a database are
typically much smaller than this; cloud databases therefore typically manage smaller values in a
separate service, and store larger data blocks (containing many individual values) in an object
store [^26] [^29]. We will see ways of doing this in [Chapter 4](/en/ch4#ch_storage).
In a traditional systems architecture, the same computer is responsible for both storage (disk) and
computation (CPU and RAM), but in cloud-native systems, these two responsibilities have become
somewhat separated or *disaggregated* [^9] [^27] [^30] [^31]:
for example, S3 only stores files, and if you want to analyze that data, you will have to run the
analysis code somewhere outside of S3. This implies transferring the data over the network, which we
will discuss further in [“Distributed versus Single-Node Systems”](/en/ch1#sec_introduction_distributed).
Moreover, cloud-native systems are often *multitenant*, which means that rather than having a
separate machine for each customer, data and computation from several different customers are
handled on the same shared hardware by the same service [^32].
Multitenancy can enable better hardware utilization, easier scalability, and easier management by
the cloud provider, but it also requires careful engineering to ensure that one customer’s activity
does not affect the performance or security of the system for other customers [^33].
### Operations in the Cloud Era {#sec_introduction_operations}
Traditionally, the people managing an organization’s server-side data infrastructure were known as
*database administrators* (DBAs) or *system administrators* (sysadmins). More recently, many
organizations have tried to integrate the roles of software development and operations into teams
with a shared responsibility for both backend services and data infrastructure; the *DevOps*
philosophy has guided this trend. *Site Reliability Engineers* (SREs) are Google’s implementation of
this idea [^34].
The role of operations is to ensure services are reliably delivered to users (including configuring
infrastructure and deploying applications), and to ensure a stable production environment (including
monitoring and diagnosing any problems that may affect reliability). For self-hosted systems,
operations traditionally involves a significant amount of work at the level of individual machines,
such as capacity planning (e.g., monitoring available disk space and adding more disks before you
run out of space), provisioning new machines, moving services from one machine to another, and
installing operating system patches.
Many cloud services present an API that hides the individual machines that actually implement the
service. For example, cloud storage replaces fixed-size disks with *metered billing*, where you can
store data without planning your capacity needs in advance, and you are then charged based on the
space actually used. Moreover, many cloud services remain highly available, even when individual
machines have failed (see [“Reliability and Fault Tolerance”](/en/ch2#sec_introduction_reliability)).
This shift in emphasis from individual machines to services has been accompanied by a change in the
role of operations. The high-level goal of providing a reliable service remains the same, but the
processes and tools have evolved. The DevOps/SRE philosophy places greater emphasis on:
* automation—preferring repeatable processes over manual one-off jobs,
* preferring ephemeral virtual machines and services over long running servers,
* enabling frequent application updates,
* learning from incidents, and
* preserving the organization’s knowledge about the system, even as individual people come and go [^35].
With the rise of cloud services, there has been a bifurcation of roles: operations teams at
infrastructure companies specialize in the details of providing a reliable service to a large number
of customers, while the customers of the service spend as little time and effort as possible on infrastructure [^36].
Customers of cloud services still require operations, but they focus on different aspects, such as
choosing the most appropriate service for a given task, integrating different services with each
other, and migrating from one service to another. Even though metered billing removes the need for
capacity planning in the traditional sense, it’s still important to know what resources you are
using for which purpose, so that you don’t waste money on cloud resources that are not needed:
capacity planning becomes financial planning, and performance optimization becomes cost optimization [^37].
Moreover, cloud services do have resource limits or *quotas* (such as the maximum number of
processes you can run concurrently), which you need to know about and plan for before you run into them [^38].
Adopting a cloud service can be easier and quicker than running your own infrastructure, although
even here there is a cost in learning how to use it, and perhaps working around its limitations.
Integration between different services becomes a particular challenge as a growing number of vendors
offers an ever broader range of cloud services targeting different use cases [^39] [^40].
ETL (see [“Data Warehousing”](/en/ch1#sec_introduction_dwh)) is only part of the story; operational cloud services also need
to be integrated with each other. At present, there is a lack of standards that would facilitate
this sort of integration, so it often involves significant manual effort.
Other operational aspects that cannot fully be outsourced to cloud services include maintaining the
security of an application and the libraries it uses, managing the interactions between your own
services, monitoring the load on your services, and tracking down the cause of problems such as
performance degradations or outages. While the cloud is changing the role of operations, the need
for operations is as great as ever.
## Distributed versus Single-Node Systems {#sec_introduction_distributed}
A system that involves several machines communicating via a network is called a *distributed
system*. Each of the processes participating in a distributed system is called a *node*. There are
various reasons why you might want a system to be distributed:
Inherently distributed systems
: If an application involves two or more interacting users, each using their own device, then the
system is unavoidably distributed: the communication between the devices will have to go via a
network.
Requests between cloud services
: If data is stored in one service but processed in another, it must be transferred over the network
from one service to the other.
Fault tolerance/high availability
: If your application needs to continue working even if one machine (or several machines, or
the network, or an entire datacenter) goes down, you can use multiple machines to give you
redundancy. When one fails, another one can take over. See [“Reliability and Fault Tolerance”](/en/ch2#sec_introduction_reliability) and
[Chapter 6](/en/ch6#ch_replication) on replication.
Scalability
: If your data volume or computing requirements grow bigger than a single machine can handle,
you can potentially spread the load across multiple machines. See
[“Scalability”](/en/ch2#sec_introduction_scalability).
Latency
: If you have users around the world, you might want to have servers in various regions
worldwide so that each user can be served from a server that is geographically close to
them. That avoids the users having to wait for network packets to travel halfway around the
world to answer their requests. See [“Describing Performance”](/en/ch2#sec_introduction_percentiles).
Elasticity
: If your application is busy at some times and idle at other times, a cloud deployment can scale up
or down to meet the demand, so that you pay only for resources you are actively using. This is more
difficult on a single machine, which needs to be provisioned to handle the maximum load, even at
times when it is barely used.
Using specialized hardware
: Different parts of the system can take advantage of different types of hardware to match their
workload. For example, an object store may use machines with many disks but few CPUs, whereas a
data analysis system may use machines with lots of CPU and memory but no disks, and a machine
learning system may use machines with GPUs (which are much more efficient than CPUs for training
deep neural networks and other machine learning tasks).
Legal compliance
: Some countries have data residency laws that require data about people in their jurisdiction to be
stored and processed geographically within that country [^41].
The scope of these rules varies—for example, in some cases it applies only to medical or financial
data, while other cases are broader. A service with users in several such jurisdictions will
therefore have to distribute their data across servers in several locations.
Sustainability
: If you have flexibility on where and when to run your jobs, you might be able to run them in a
time and place where plenty of renewable electricity is available, and avoid running them when the
power grid is under strain. This can reduce your carbon emissions and allow you to take advantage
of cheap power when it is available [^42] [^43].
These reasons apply both to services that you write yourself (application code) and services
consisting of off-the-shelf software (such as databases).
### Problems with Distributed Systems {#sec_introduction_dist_sys_problems}
Distributed systems also have downsides. Every request and API call that goes via the network needs
to deal with the possibility of failure: the network may be interrupted, or the service may be
overloaded or crashed, and therefore any request may time out without receiving a response. In this
case, we don’t know whether the service received the request, and simply retrying it might not be
safe. We will discuss these problems in detail in [Chapter 9](/en/ch9#ch_distributed).
Although datacenter networks are fast, making a call to another service is still vastly slower than
calling a function in the same process [^44].
When operating on large volumes of data, rather than transferring the data from storage to a
separate machine that processes it, it can be faster to bring the computation to the machine that
already has the data [^45].
More nodes are not always faster: in some cases, a simple single-threaded program on one computer
can perform significantly better than a cluster with over 100 CPU cores [^46].
Troubleshooting a distributed system is often difficult: if the system is slow to respond, how do
you figure out where the problem lies? Techniques for diagnosing problems in distributed systems are
developed under the heading of *observability* [^47] [^48],
which involves collecting data about the execution of a system, and allowing it to be queried in
ways that allows both high-level metrics and individual events to be analyzed. *Tracing* tools such
as OpenTelemetry, Zipkin, and Jaeger allow you to track which client called which server for which
operation, and how long each call took [^49].
Databases provide various mechanisms for ensuring data consistency, as we shall see in
[Chapter 6](/en/ch6#ch_replication) and [Chapter 8](/en/ch8#ch_transactions). However, when each service has its own database,
maintaining consistency of data across those different services becomes the application’s problem.
Distributed transactions, which we explore in [Chapter 8](/en/ch8#ch_transactions), are a possible technique for
ensuring consistency, but they are rarely used in a microservices context because they run counter
to the goal of making services independent from each other, and many databases don’t support them [^50].
For all these reasons, if you can do something on a single machine, this is often much simpler and
cheaper compared to setting up a distributed system [^23] [^46] [^51].
CPUs, memory, and disks have grown larger, faster, and more reliable. When combined with single-node
databases such as DuckDB, SQLite, and KùzuDB, many workloads can now run on a single node. We will
explore more on this topic in [Chapter 4](/en/ch4#ch_storage).
### Microservices and Serverless {#sec_introduction_microservices}
The most common way of distributing a system across multiple machines is to divide them into clients
and servers, and let the clients make requests to the servers. Most commonly HTTP is used for this
communication, as we will discuss in [“Dataflow Through Services: REST and RPC”](/en/ch5#sec_encoding_dataflow_rpc). The same process may be both a
server (handling incoming requests) and a client (making outbound requests to other services).
This way of building applications has traditionally been called a *service-oriented architecture*
(SOA); more recently the idea has been refined into a *microservices* architecture [^52] [^53].
In this architecture, a service has one well-defined purpose (for example, in the case of S3, this
would be file storage); each service exposes an API that can be called by clients via the network,
and each service has one team that is responsible for its maintenance. A complex application can
thus be decomposed into multiple interacting services, each managed by a separate team.
There are several advantages to breaking down a complex piece of software into multiple services:
each service can be updated independently, reducing coordination effort among teams; each service
can be assigned the hardware resources it needs; and by hiding the implementation details behind an
API, the service owners are free to change the implementation without affecting clients. In terms of
data storage, it is common for each service to have its own databases, and not to share databases
between services: sharing a database would effectively make the entire database structure a part of
the service’s API, and then that structure would be difficult to change. Shared databases could also
cause one service’s queries to negatively impact the performance of other services.
On the other hand, having many services can itself breed complexity: each service requires
infrastructure for deploying new releases, adjusting the allocated hardware resources to match the
load, collecting logs, monitoring service health, and alerting an on-call engineer in the case of a
problem. *Orchestration* frameworks such as Kubernetes have become a popular way of deploying
services, since they provide a foundation for this infrastructure. Testing a service during
development can be complicated, since you also need to run all the other services that it depends on.
Microservice APIs can be challenging to evolve. Clients that call an API expect the API to have
certain fields. Developers might wish to add or remove fields to an API as business needs change,
but doing so can cause clients to fail. Worse still, such failures are often not discovered until
late in the development cycle when the updated service API is deployed to a staging or production
environment. API description standards such as OpenAPI and gRPC help manage the relationship between
client and server APIs; we discuss these further in [Chapter 5](/en/ch5#ch_encoding).
Microservices are primarily a technical solution to a people problem: allowing different teams to
make progress independently without having to coordinate with each other. This is valuable in a large
company, but in a small company where there are not many teams, using microservices is likely to be
unnecessary overhead, and it is preferable to implement the application in the simplest way possible [^52].
*Serverless*, or *function-as-a-service* (FaaS), is another approach to deploying services, in which
the management of the infrastructure is outsourced to a cloud vendor [^33].
When using virtual machines, you have to explicitly choose when to start up or shut down an
instance; in contrast, with the serverless model, the cloud provider automatically allocates and
frees hardware resources as needed, based on the incoming requests to your service [^54].
Serverless deployment shifts more of the operational burden to cloud providers and enables flexible billing
by usage rather than machine instances. To offer such benefits, many serverless infrastructure providers
impose a time limit on function execution, limit runtime environments, and might suffer from slow
start times when a function is first invoked. The term “serverless” can also be misleading: each
serverless function execution still runs on a server, but subsequent executions might run on a
different one. Moreover, infrastructure such as BigQuery and various Kafka offerings have adopted
“serverless” terminology to signal that their services auto-scale and that they bill by usage rather than machine instances.
Just like cloud storage replaced capacity planning (deciding in advance how many disks to buy) with
a metered billing model, the serverless approach is bringing metered billing to code execution: you
only pay for the time that your application code is actually running, rather than having to
provision resources in advance.
### Cloud Computing versus Supercomputing {#id17}
Cloud computing is not the only way of building large-scale computing systems; an alternative is
*high-performance computing* (HPC), also known as *supercomputing*. Although there are overlaps, HPC
often has different priorities and uses different techniques compared to cloud computing and
enterprise datacenter systems. Some of those differences are:
* Supercomputers are typically used for computationally intensive scientific computing tasks, such
as weather forecasting, climate modeling, molecular dynamics (simulating the movement of atoms and
molecules), complex optimization problems, and solving partial differential equations. On the
other hand, cloud computing tends to be used for online services, business data systems, and
similar systems that need to serve user requests with high availability.
* A supercomputer typically runs large batch jobs that checkpoint the state of their computation to
disk from time to time. If a node fails, a common solution is to simply stop the entire cluster
workload, repair the faulty node, and then restart the computation from the last checkpoint [^55] [^56].
With cloud services, it is usually not desirable to stop the entire cluster, since the services
need to continually serve users with minimal interruptions.
* Supercomputer nodes typically communicate through shared memory and remote direct memory access
(RDMA), which support high bandwidth and low latency, but assume a high level of trust among the users of the system [^57].
In cloud computing, the network and the machines are often shared by mutually untrusting
organizations, requiring stronger security mechanisms such as resource isolation (e.g., virtual
machines), encryption and authentication.
* Cloud datacenter networks are often based on IP and Ethernet, arranged in Clos topologies to
provide high bisection bandwidth—a commonly used measure of a network’s overall performance [^55] [^58].
Supercomputers often use specialized network topologies, such as multi-dimensional meshes and toruses [^59],
which yield better performance for HPC workloads with known communication patterns.
* Cloud computing allows nodes to be distributed across multiple geographic regions, whereas
supercomputers generally assume that all of their nodes are close together.
Large-scale analytics systems sometimes share some characteristics with supercomputing, which is why
it can be worth knowing about these techniques if you are working in this area. However, this book
is mostly concerned with services that need to be continually available, as discussed in [“Reliability and Fault Tolerance”](/en/ch2#sec_introduction_reliability).
## Data Systems, Law, and Society {#sec_introduction_compliance}
So far you’ve seen in this chapter that the architecture of data systems is influenced not only by
technical goals and requirements, but also by the human needs of the organizations that they
support. Increasingly, data systems engineers are realizing that serving the needs of their own
business is not enough: we also have a responsibility towards society at large.
One particular concern are systems that store data about people and their behavior. Since 2018 the
*General Data Protection Regulation* (GDPR) has given residents of many European countries greater
control and legal rights over their personal data, and similar privacy regulation has been adopted
in various other countries and states around the world, including for example the California
Consumer Privacy Act (CCPA). Regulations around AI, such as the *EU AI Act*, place further
restrictions on how personal data can be used.
Moreover, even in areas that are not directly subject to regulation, there is increasing recognition
of the effects that computer systems have on people and society. Social media has changed how
individuals consume news, which influences their political opinions and hence may affect the outcome
of elections. Automated systems increasingly make decisions that have profound consequences for
individuals, such as deciding who should be given a loan or insurance coverage, who should be
invited to a job interview, or who should be suspected of a crime [^60].
Everyone who works on such systems shares a responsibility for considering the ethical impact and
ensuring that they comply with relevant law. It is not necessary for everybody to become an expert
in law and ethics, but a basic awareness of legal and ethical principles is just as important as,
say, some foundational knowledge in distributed systems.
Legal considerations are influencing the very foundations of how data systems are being designed [^61].
For example, the GDPR grants individuals the right to have their data erased on request (sometimes
known as the *right to be forgotten*). However, as we shall see in this book, many data systems rely
on immutable constructs such as append-only logs as part of their design; how can we ensure deletion
of some data in the middle of a file that is supposed to be immutable? How do we handle deletion of
data that has been incorporated into derived datasets (see [“Systems of Record and Derived Data”](/en/ch1#sec_introduction_derived)), such as
training data for machine learning models? Answering these questions creates new engineering
challenges.
At present we don’t have clear guidelines on which particular technologies or system architectures
should be considered “GDPR-compliant” or not. The regulation deliberately does not mandate
particular technologies, because these may quickly change as technology progresses. Instead, the
legal texts set out high-level principles that are subject to interpretation. This means that there
are no simple answers to the question of how to comply with privacy regulation, but we will look at
some of the technologies in this book through this lens.
In general, we store data because we think that its value is greater than the costs of storing it.
However, it is worth remembering that the costs of storage are not just the bill you pay for Amazon
S3 or another service: the cost-benefit calculation should also take into account the risks of
liability and reputational damage if the data were to be leaked or compromised by adversaries, and
the risk of legal costs and fines if the storage and processing of the data is found not to be
compliant with the law [^51].
Governments or police forces might also compel companies to hand over data. When there is a risk
that the data may reveal criminalized behaviors (for example, homosexuality in several Middle
Eastern and African countries, or seeking an abortion in several US states), storing that data
creates real safety risks for users. Travel to an abortion clinic, for example, could easily be
revealed by location data, perhaps even by a log of the user’s IP addresses over time (which
indicate approximate location).
Once all the risks are taken into account, it might be reasonable to decide that some data is simply
not worth storing, and that it should therefore be deleted. This principle of *data minimization*
(sometimes known by the German term *Datensparsamkeit*) runs counter to the “big data” philosophy of
storing lots of data speculatively in case it turns out to be useful in the future [^62].
But it fits with the GDPR, which mandates that personal data may only be collected for a specified,
explicit purpose, that this data may not later be used for any other purpose, and that the data must
not be kept for longer than necessary for the purposes for which it was collected [^63].
Businesses have also taken notice of privacy and safety concerns. Credit card companies require
payment processing businesses to adhere to strict payment card industry (PCI) standards. Processors
undergo frequent evaluations from independent auditors to verify continued compliance. Software
vendors have also seen increased scrutiny. Many buyers now require their vendors to comply with
Service Organization Control (SOC) Type 2 standards. As with PCI compliance, vendors undergo third
party audits to verify adherence.
Generally, it is important to balance the needs of your business against the needs of the people
whose data you are collecting and processing. There is much more to this topic; in [Chapter 14](/en/ch14#ch_right_thing) we
will go deeper into the topics of ethics and legal compliance, including the problems of bias and
discrimination.
## Summary {#summary}
The theme of this chapter has been to understand trade-offs: that is, to recognize that for many
questions there is not one right answer, but several different approaches that each have various
pros and cons. We explored some of the most important choices that affect the architecture of data
systems, and introduced terminology that will be needed throughout the rest of this book.
We started by making a distinction between operational (transaction-processing, OLTP) and analytical
(OLAP) systems, and saw their different characteristics: not only managing different types of data
with different access patterns, but also serving different audiences. We encountered the concept of
a data warehouse and data lake, which receive data feeds from operational systems via ETL. In
[Chapter 4](/en/ch4#ch_storage) we will see that operational and analytical systems often use very different internal
data layouts because of the different types of queries they need to serve.
We then compared cloud services, a comparatively recent development, to the traditional paradigm of
self-hosted software that has previously dominated data systems architecture. Which of these
approaches is more cost-effective depends a lot on your particular situation, but it’s undeniable
that cloud-native approaches are bringing big changes to the way data systems are architected, for
example in the way they separate storage and compute.
Cloud systems are intrinsically distributed, and we briefly examined some of the trade-offs of
distributed systems compared to using a single machine. There are situations in which you can’t
avoid going distributed, but it’s advisable not to rush into making a system distributed if it’s
possible to keep it on a single machine. In [Chapter 9](/en/ch9#ch_distributed) we will cover the challenges with
distributed systems in more detail.
Finally, we saw that data systems architecture is determined not only by the needs of the business
deploying the system, but also by privacy regulation that protects the rights of the people whose
data is being processed—an aspect that many engineers are prone to ignoring. How we translate legal
requirements into technical implementations is not yet well understood, but it’s important to keep
this question in mind as we move through the rest of this book.
### References
[^1]: Richard T. Kouzes, Gordon A. Anderson, Stephen T. Elbert, Ian Gorton, and Deborah K. Gracio. [The Changing Paradigm of Data-Intensive Computing](http://www2.ic.uff.br/~boeres/slides_AP/papers/TheChanginParadigmDataIntensiveComputing_2009.pdf). *IEEE Computer*, volume 42, issue 1, January 2009. [doi:10.1109/MC.2009.26](https://doi.org/10.1109/MC.2009.26)
[^2]: Martin Kleppmann, Adam Wiggins, Peter van Hardenberg, and Mark McGranaghan. [Local-first software: you own your data, in spite of the cloud](https://www.inkandswitch.com/local-first/). At *2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software* (Onward!), October 2019. [doi:10.1145/3359591.3359737](https://doi.org/10.1145/3359591.3359737)
[^3]: Joe Reis and Matt Housley. [*Fundamentals of Data Engineering*](https://www.oreilly.com/library/view/fundamentals-of-data/9781098108298/). O’Reilly Media, 2022. ISBN: 9781098108304
[^4]: Rui Pedro Machado and Helder Russa. [*Analytics Engineering with SQL and dbt*](https://www.oreilly.com/library/view/analytics-engineering-with/9781098142377/). O’Reilly Media, 2023. ISBN: 9781098142384
[^5]: Edgar F. Codd, S. B. Codd, and C. T. Salley. [Providing OLAP to User-Analysts: An IT Mandate](https://www.estgv.ipv.pt/PaginasPessoais/jloureiro/ESI_AID2007_2008/fichas/codd.pdf). E. F. Codd Associates, 1993. Archived at [perma.cc/RKX8-2GEE](https://perma.cc/RKX8-2GEE)
[^6]: Chinmay Soman and Neha Pawar. [Comparing Three Real-Time OLAP Databases: Apache Pinot, Apache Druid, and ClickHouse](https://startree.ai/blog/a-tale-of-three-real-time-olap-databases). *startree.ai*, April 2023. Archived at [perma.cc/8BZP-VWPA](https://perma.cc/8BZP-VWPA)
[^7]: Surajit Chaudhuri and Umeshwar Dayal. [An Overview of Data Warehousing and OLAP Technology](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/sigrecord.pdf). *ACM SIGMOD Record*, volume 26, issue 1, pages 65–74, March 1997. [doi:10.1145/248603.248616](https://doi.org/10.1145/248603.248616)
[^8]: Fatma Özcan, Yuanyuan Tian, and Pinar Tözün. [Hybrid Transactional/Analytical Processing: A Survey](https://humming80.github.io/papers/sigmod-htaptut.pdf). At *ACM International Conference on Management of Data* (SIGMOD), May 2017. [doi:10.1145/3035918.3054784](https://doi.org/10.1145/3035918.3054784)
[^9]: Adam Prout, Szu-Po Wang, Joseph Victor, Zhou Sun, Yongzhu Li, Jack Chen, Evan Bergeron, Eric Hanson, Robert Walzer, Rodrigo Gomes, and Nikita Shamgunov. [Cloud-Native Transactions and Analytics in SingleStore](https://dl.acm.org/doi/abs/10.1145/3514221.3526055). At *International Conference on Management of Data* (SIGMOD), June 2022. [doi:10.1145/3514221.3526055](https://doi.org/10.1145/3514221.3526055)
[^10]: Chao Zhang, Guoliang Li, Jintao Zhang, Xinning Zhang, and Jianhua Feng. [HTAP Databases: A Survey](https://arxiv.org/pdf/2404.15670). *IEEE Transactions on Knowledge and Data Engineering*, April 2024. [doi:10.1109/TKDE.2024.3389693](https://doi.org/10.1109/TKDE.2024.3389693)
[^11]: Michael Stonebraker and Uğur Çetintemel. [‘One Size Fits All’: An Idea Whose Time Has Come and Gone](https://pages.cs.wisc.edu/~shivaram/cs744-readings/fits_all.pdf). At *21st International Conference on Data Engineering* (ICDE), April 2005. [doi:10.1109/ICDE.2005.1](https://doi.org/10.1109/ICDE.2005.1)
[^12]: Jeffrey Cohen, Brian Dolan, Mark Dunlap, Joseph M. Hellerstein, and Caleb Welton. [MAD Skills: New Analysis Practices for Big Data](https://www.vldb.org/pvldb/vol2/vldb09-219.pdf). *Proceedings of the VLDB Endowment*, volume 2, issue 2, pages 1481–1492, August 2009. [doi:10.14778/1687553.1687576](https://doi.org/10.14778/1687553.1687576)
[^13]: Dan Olteanu. [The Relational Data Borg is Learning](https://www.vldb.org/pvldb/vol13/p3502-olteanu.pdf). *Proceedings of the VLDB Endowment*, volume 13, issue 12, August 2020. [doi:10.14778/3415478.3415572](https://doi.org/10.14778/3415478.3415572)
[^14]: Matt Bornstein, Martin Casado, and Jennifer Li. [Emerging Architectures for Modern Data Infrastructure: 2020](https://future.a16z.com/emerging-architectures-for-modern-data-infrastructure-2020/). *future.a16z.com*, October 2020. Archived at [perma.cc/LF8W-KDCC](https://perma.cc/LF8W-KDCC)
[^15]: Martin Fowler. [DataLake](https://www.martinfowler.com/bliki/DataLake.html). *martinfowler.com*, February 2015. Archived at [perma.cc/4WKN-CZUK](https://perma.cc/4WKN-CZUK)
[^16]: Bobby Johnson and Joseph Adler. [The Sushi Principle: Raw Data Is Better](https://learning.oreilly.com/videos/strata-hadoop/9781491924143/9781491924143-video210840/). At *Strata+Hadoop World*, February 2015.
[^17]: Michael Armbrust, Ali Ghodsi, Reynold Xin, and Matei Zaharia. [Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics](https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf). At *11th Annual Conference on Innovative Data Systems Research* (CIDR), January 2021.
[^18]: DataKitchen, Inc. [The DataOps Manifesto](https://dataopsmanifesto.org/en/). *dataopsmanifesto.org*, 2017. Archived at [perma.cc/3F5N-FUQ4](https://perma.cc/3F5N-FUQ4)
[^19]: Tejas Manohar. [What is Reverse ETL: A Definition & Why It’s Taking Off](https://hightouch.io/blog/reverse-etl/). *hightouch.io*, November 2021. Archived at [perma.cc/A7TN-GLYJ](https://perma.cc/A7TN-GLYJ)
[^20]: Simon O’Regan. [Designing Data Products](https://towardsdatascience.com/designing-data-products-b6b93edf3d23). *towardsdatascience.com*, August 2018. Archived at [perma.cc/HU67-3RV8](https://perma.cc/HU67-3RV8)
[^21]: Camille Fournier. [Why is it so hard to decide to buy?](https://skamille.medium.com/why-is-it-so-hard-to-decide-to-buy-d86fee98e88e) *skamille.medium.com*, July 2021. Archived at [perma.cc/6VSG-HQ5X](https://perma.cc/6VSG-HQ5X)
[^22]: David Heinemeier Hansson. [Why we’re leaving the cloud](https://world.hey.com/dhh/why-we-re-leaving-the-cloud-654b47e0). *world.hey.com*, October 2022. Archived at [perma.cc/82E6-UJ65](https://perma.cc/82E6-UJ65)
[^23]: Nima Badizadegan. [Use One Big Server](https://specbranch.com/posts/one-big-server/). *specbranch.com*, August 2022. Archived at [perma.cc/M8NB-95UK](https://perma.cc/M8NB-95UK)
[^24]: Steve Yegge. [Dear Google Cloud: Your Deprecation Policy is Killing You](https://steve-yegge.medium.com/dear-google-cloud-your-deprecation-policy-is-killing-you-ee7525dc05dc). *steve-yegge.medium.com*, August 2020. Archived at [perma.cc/KQP9-SPGU](https://perma.cc/KQP9-SPGU)
[^25]: Alexandre Verbitski, Anurag Gupta, Debanjan Saha, Murali Brahmadesam, Kamal Gupta, Raman Mittal, Sailesh Krishnamurthy, Sandor Maurice, Tengiz Kharatishvili, and Xiaofeng Bao. [Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases](https://media.amazonwebservices.com/blog/2017/aurora-design-considerations-paper.pdf). At *ACM International Conference on Management of Data* (SIGMOD), pages 1041–1052, May 2017. [doi:10.1145/3035918.3056101](https://doi.org/10.1145/3035918.3056101)
[^26]: Panagiotis Antonopoulos, Alex Budovski, Cristian Diaconu, Alejandro Hernandez Saenz, Jack Hu, Hanuma Kodavalla, Donald Kossmann, Sandeep Lingam, Umar Farooq Minhas, Naveen Prakash, Vijendra Purohit, Hugh Qu, Chaitanya Sreenivas Ravella, Krystyna Reisteter, Sheetal Shrotri, Dixin Tang, and Vikram Wakade. [Socrates: The New SQL Server in the Cloud](https://www.microsoft.com/en-us/research/uploads/prod/2019/05/socrates.pdf). At *ACM International Conference on Management of Data* (SIGMOD), pages 1743–1756, June 2019. [doi:10.1145/3299869.3314047](https://doi.org/10.1145/3299869.3314047)
[^27]: Midhul Vuppalapati, Justin Miron, Rachit Agarwal, Dan Truong, Ashish Motivala, and Thierry Cruanes. [Building An Elastic Query Engine on Disaggregated Storage](https://www.usenix.org/system/files/nsdi20-paper-vuppalapati.pdf). At *17th USENIX Symposium on Networked Systems Design and Implementation* (NSDI), February 2020.
[^28]: Nick Van Wiggeren. [The Real Failure Rate of EBS](https://planetscale.com/blog/the-real-fail-rate-of-ebs). *planetscale.com*, March 2025. Archived at [perma.cc/43CR-SAH5](https://perma.cc/43CR-SAH5)
[^29]: Colin Breck. [Predicting the Future of Distributed Systems](https://blog.colinbreck.com/predicting-the-future-of-distributed-systems/). *blog.colinbreck.com*, August 2024. Archived at [perma.cc/K5FC-4XX2](https://perma.cc/K5FC-4XX2)
[^30]: Gwen Shapira. [Compute-Storage Separation Explained](https://www.thenile.dev/blog/storage-compute). *thenile.dev*, January 2023. Archived at [perma.cc/QCV3-XJNZ](https://perma.cc/QCV3-XJNZ)
[^31]: Ravi Murthy and Gurmeet Goindi. [AlloyDB for PostgreSQL under the hood: Intelligent, database-aware storage](https://cloud.google.com/blog/products/databases/alloydb-for-postgresql-intelligent-scalable-storage). *cloud.google.com*, May 2022. Archived at [archive.org](https://web.archive.org/web/20220514021120/https%3A//cloud.google.com/blog/products/databases/alloydb-for-postgresql-intelligent-scalable-storage)
[^32]: Jack Vanlightly. [The Architecture of Serverless Data Systems](https://jack-vanlightly.com/blog/2023/11/14/the-architecture-of-serverless-data-systems). *jack-vanlightly.com*, November 2023. Archived at [perma.cc/UDV4-TNJ5](https://perma.cc/UDV4-TNJ5)
[^33]: Eric Jonas, Johann Schleier-Smith, Vikram Sreekanti, Chia-Che Tsai, Anurag Khandelwal, Qifan Pu, Vaishaal Shankar, Joao Carreira, Karl Krauth, Neeraja Yadwadkar, Joseph E. Gonzalez, Raluca Ada Popa, Ion Stoica, David A. Patterson. [Cloud Programming Simplified: A Berkeley View on Serverless Computing](https://arxiv.org/abs/1902.03383). *arxiv.org*, February 2019.
[^34]: Betsy Beyer, Jennifer Petoff, Chris Jones, and Niall Richard Murphy. [*Site Reliability Engineering: How Google Runs Production Systems*](https://www.oreilly.com/library/view/site-reliability-engineering/9781491929117/). O’Reilly Media, 2016. ISBN: 9781491929124
[^35]: Thomas Limoncelli. [The Time I Stole $10,000 from Bell Labs](https://queue.acm.org/detail.cfm?id=3434773). *ACM Queue*, volume 18, issue 5, November 2020. [doi:10.1145/3434571.3434773](https://doi.org/10.1145/3434571.3434773)
[^36]: Charity Majors. [The Future of Ops Jobs](https://acloudguru.com/blog/engineering/the-future-of-ops-jobs). *acloudguru.com*, August 2020. Archived at [perma.cc/GRU2-CZG3](https://perma.cc/GRU2-CZG3)
[^37]: Boris Cherkasky. [(Over)Pay As You Go for Your Datastore](https://medium.com/riskified-technology/over-pay-as-you-go-for-your-datastore-11a29ae49a8b). *medium.com*, September 2021. Archived at [perma.cc/Q8TV-2AM2](https://perma.cc/Q8TV-2AM2)
[^38]: Shlomi Kushchi. [Serverless Doesn’t Mean DevOpsLess or NoOps](https://thenewstack.io/serverless-doesnt-mean-devopsless-or-noops/). *thenewstack.io*, February 2023. Archived at [perma.cc/3NJR-AYYU](https://perma.cc/3NJR-AYYU)
[^39]: Erik Bernhardsson. [Storm in the stratosphere: how the cloud will be reshuffled](https://erikbern.com/2021/11/30/storm-in-the-stratosphere-how-the-cloud-will-be-reshuffled.html). *erikbern.com*, November 2021. Archived at [perma.cc/SYB2-99P3](https://perma.cc/SYB2-99P3)
[^40]: Benn Stancil. [The data OS](https://benn.substack.com/p/the-data-os). *benn.substack.com*, September 2021. Archived at [perma.cc/WQ43-FHS6](https://perma.cc/WQ43-FHS6)
[^41]: Maria Korolov. [Data residency laws pushing companies toward residency as a service](https://www.csoonline.com/article/3647761/data-residency-laws-pushing-companies-toward-residency-as-a-service.html). *csoonline.com*, January 2022. Archived at [perma.cc/CHE4-XZZ2](https://perma.cc/CHE4-XZZ2)
[^42]: Severin Borenstein. [Can Data Centers Flex Their Power Demand?](https://energyathaas.wordpress.com/2025/04/14/can-data-centers-flex-their-power-demand/) *energyathaas.wordpress.com*, April 2025. Archived at <https://perma.cc/MUD3-A6FF>
[^43]: Bilge Acun, Benjamin Lee, Fiodar Kazhamiaka, Aditya Sundarrajan, Kiwan Maeng, Manoj Chakkaravarthy, David Brooks, and Carole-Jean Wu. [Carbon Dependencies in Datacenter Design and Management](https://hotcarbon.org/assets/2022/pdf/hotcarbon22-acun.pdf). *ACM SIGENERGY Energy Informatics Review*, volume 3, issue 3, pages 21–26. [doi:10.1145/3630614.3630619](https://doi.org/10.1145/3630614.3630619)
[^44]: Kousik Nath. [These are the numbers every computer engineer should know](https://www.freecodecamp.org/news/must-know-numbers-for-every-computer-engineer/). *freecodecamp.org*, September 2019. Archived at [perma.cc/RW73-36RL](https://perma.cc/RW73-36RL)
[^45]: Joseph M. Hellerstein, Jose Faleiro, Joseph E. Gonzalez, Johann Schleier-Smith, Vikram Sreekanti, Alexey Tumanov, and Chenggang Wu. [Serverless Computing: One Step Forward, Two Steps Back](https://arxiv.org/abs/1812.03651). At *Conference on Innovative Data Systems Research* (CIDR), January 2019.
[^46]: Frank McSherry, Michael Isard, and Derek G. Murray. [Scalability! But at What COST?](https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf) At *15th USENIX Workshop on Hot Topics in Operating Systems* (HotOS), May 2015.
[^47]: Cindy Sridharan. *[Distributed Systems Observability: A Guide to Building Robust Systems](https://unlimited.humio.com/rs/756-LMY-106/images/Distributed-Systems-Observability-eBook.pdf)*. Report, O’Reilly Media, May 2018. Archived at [perma.cc/M6JL-XKCM](https://perma.cc/M6JL-XKCM)
[^48]: Charity Majors. [Observability — A 3-Year Retrospective](https://thenewstack.io/observability-a-3-year-retrospective/). *thenewstack.io*, August 2019. Archived at [perma.cc/CG62-TJWL](https://perma.cc/CG62-TJWL)
[^49]: Benjamin H. Sigelman, Luiz André Barroso, Mike Burrows, Pat Stephenson, Manoj Plakal, Donald Beaver, Saul Jaspan, and Chandan Shanbhag. [Dapper, a Large-Scale Distributed Systems Tracing Infrastructure](https://research.google/pubs/pub36356/). Google Technical Report dapper-2010-1, April 2010. Archived at [perma.cc/K7KU-2TMH](https://perma.cc/K7KU-2TMH)
[^50]: Rodrigo Laigner, Yongluan Zhou, Marcos Antonio Vaz Salles, Yijian Liu, and Marcos Kalinowski. [Data management in microservices: State of the practice, challenges, and research directions](https://www.vldb.org/pvldb/vol14/p3348-laigner.pdf). *Proceedings of the VLDB Endowment*, volume 14, issue 13, pages 3348–3361, September 2021. [doi:10.14778/3484224.3484232](https://doi.org/10.14778/3484224.3484232)
[^51]: Jordan Tigani. [Big Data is Dead](https://motherduck.com/blog/big-data-is-dead/). *motherduck.com*, February 2023. Archived at [perma.cc/HT4Q-K77U](https://perma.cc/HT4Q-K77U)
[^52]: Sam Newman. [*Building Microservices*, second edition](https://www.oreilly.com/library/view/building-microservices-2nd/9781492034018/). O’Reilly Media, 2021. ISBN: 9781492034025
[^53]: Chris Richardson. [Microservices: Decomposing Applications for Deployability and Scalability](https://www.infoq.com/articles/microservices-intro/). *infoq.com*, May 2014. Archived at [perma.cc/CKN4-YEQ2](https://perma.cc/CKN4-YEQ2)
[^54]: Mohammad Shahrad, Rodrigo Fonseca, Íñigo Goiri, Gohar Chaudhry, Paul Batum, Jason Cooke, Eduardo Laureano, Colby Tresness, Mark Russinovich, Ricardo Bianchini. [Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider](https://www.usenix.org/system/files/atc20-shahrad.pdf). At *USENIX Annual Technical Conference* (ATC), July 2020.
[^55]: Luiz André Barroso, Urs Hölzle, and Parthasarathy Ranganathan. [The Datacenter as a Computer: Designing Warehouse-Scale Machines](https://www.morganclaypool.com/doi/10.2200/S00874ED3V01Y201809CAC046), third edition. Morgan & Claypool Synthesis Lectures on Computer Architecture, October 2018. [doi:10.2200/S00874ED3V01Y201809CAC046](https://doi.org/10.2200/S00874ED3V01Y201809CAC046)
[^56]: David Fiala, Frank Mueller, Christian Engelmann, Rolf Riesen, Kurt Ferreira, and Ron Brightwell. [Detection and Correction of Silent Data Corruption for Large-Scale High-Performance Computing](https://arcb.csc.ncsu.edu/~mueller/ftp/pub/mueller/papers/sc12.pdf),” at *International Conference for High Performance Computing, Networking, Storage and Analysis* (SC), November 2012. [doi:10.1109/SC.2012.49](https://doi.org/10.1109/SC.2012.49)
[^57]: Anna Kornfeld Simpson, Adriana Szekeres, Jacob Nelson, and Irene Zhang. [Securing RDMA for High-Performance Datacenter Storage Systems](https://www.usenix.org/conference/hotcloud20/presentation/kornfeld-simpson). At *12th USENIX Workshop on Hot Topics in Cloud Computing* (HotCloud), July 2020.
[^58]: Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead, Roy Bannon, Seb Boving, Gaurav Desai, Bob Felderman, Paulie Germano, Anand Kanagala, Jeff Provost, Jason Simmons, Eiichi Tanda, Jim Wanderer, Urs Hölzle, Stephen Stuart, and Amin Vahdat. [Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network](https://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p183.pdf). At *Annual Conference of the ACM Special Interest Group on Data Communication* (SIGCOMM), August 2015. [doi:10.1145/2785956.2787508](https://doi.org/10.1145/2785956.2787508)
[^59]: Glenn K. Lockwood. [Hadoop’s Uncomfortable Fit in HPC](https://blog.glennklockwood.com/2014/05/hadoops-uncomfortable-fit-in-hpc.html). *glennklockwood.blogspot.co.uk*, May 2014. Archived at [perma.cc/S8XX-Y67B](https://perma.cc/S8XX-Y67B)
[^60]: Cathy O’Neil: *Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy*. Crown Publishing, 2016. ISBN: 9780553418811
[^61]: Supreeth Shastri, Vinay Banakar, Melissa Wasserman, Arun Kumar, and Vijay Chidambaram. [Understanding and Benchmarking the Impact of GDPR on Database Systems](https://www.vldb.org/pvldb/vol13/p1064-shastri.pdf). *Proceedings of the VLDB Endowment*, volume 13, issue 7, pages 1064–1077, March 2020. [doi:10.14778/3384345.3384354](https://doi.org/10.14778/3384345.3384354)
[^62]: Martin Fowler. [Datensparsamkeit](https://www.martinfowler.com/bliki/Datensparsamkeit.html). *martinfowler.com*, December 2013. Archived at [perma.cc/R9QX-CME6](https://perma.cc/R9QX-CME6)
[^63]: [Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 (General Data Protection Regulation)](https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32016R0679&from=EN). *Official Journal of the European Union* L 119/1, May 2016.
================================================
FILE: content/en/ch10.md
================================================
---
title: "10. Consistency and Consensus"
weight: 210
breadcrumbs: false
---
<a id="ch_consistency"></a>
> *An ancient adage warns, “Never go to sea with two chronometers; take one or three.”*
>
> Frederick P. Brooks Jr., *The Mythical Man-Month: Essays on Software Engineering* (1995)
Lots of things can go wrong in distributed systems, as discussed in [Chapter 9](/en/ch9#ch_distributed). If we want a
service to continue working correctly despite those things going wrong, we need to find ways of
tolerating faults.
One of the best tools we have for fault tolerance is *replication*. However, as we saw in
[Chapter 6](/en/ch6#ch_replication), having multiple copies of the data on multiple replicas opens up the risk of
inconsistencies. Reads might be handled by a replica that is not up-to-date, yielding stale results.
If multiple replicas can accept writes, we have to deal with conflicts between values that were
concurrently written on different replicas. At a high level, there are two competing philosophies
for dealing with such issues:
Eventual consistency
: In this philosophy, the fact that a system is replicated is made visible to the application, and
you as application developer are expected to deal with the inconsistencies and conflicts that may
arise. This approach is often used in systems with multi-leader (see
[“Multi-Leader Replication”](/en/ch6#sec_replication_multi_leader)) and leaderless replication (see [“Leaderless Replication”](/en/ch6#sec_replication_leaderless)).
Strong consistency
: This philosophy says that applications should not have to worry about internal details of
replication, and that the system should behave as if it was single-node. The advantage of this
approach is that it’s simpler for you, the application developer. The disadvantage is that
stronger consistency has a performance cost, and some kinds of fault that an eventually consistent
system can tolerate cause outages in strongly consistent systems.
As always, which approach is better depends on your application. If you have an app where users can
make changes to data while offline, then eventual consistency is inevitable, as discussed in
[“Sync Engines and Local-First Software”](/en/ch6#sec_replication_offline_clients). However, eventual consistency can also be difficult for
applications to deal with. If your replicas are located in datacenters with fast, reliable
communication, then strong consistency is often appropriate because its cost is acceptable.
In this chapter we will dive deeper into the strongly consistent approach, looking at three areas:
1. One challenge is that “strong consistency” is quite vague, so we will develop a more precise
definition of what we want to achieve: *linearizability*.
2. We will look at the problem of generating IDs and timestamps. This may sound unrelated to
consistency but is actually closely connected.
3. We will explore how distributed systems can achieve linearizability while still remaining
fault-tolerant; the answer is *consensus* algorithms.
Along the way, we will see that there are some fundamental limits on what is possible and what is
not in a distributed system.
The topics of this chapter are notorious for being hard to implement correctly; it’s very easy to
build systems that behave fine when there are no faults, but which completely fall apart when faced
with an unlucky combination of faults that the designer of the system hadn’t considered. A lot of
theory has been developed to help us think through those edge cases, which enables us to build
systems that can robustly tolerate faults.
This chapter will only scratch the surface: we will stick with informal intuitions, and avoid the
algorithmic nitty-gritty, formal models, and proofs. If you want to do serious work on consensus
systems and similar infrastructure, you will need to go much deeper into the theory if you want any
chance of your systems being robust. As usual, the literature references in this chapter provide
some initial pointers.
## Linearizability {#sec_consistency_linearizability}
If you want a replicated database to be as simple as possible to use, you should make it behave as
if it wasn’t replicated at all. Then users don’t have to worry about replication lag, conflicts, and
other inconsistencies. That would give us the advantage of fault tolerance, but without the
complexity arising from having to think about multiple replicas.
This is the idea behind *linearizability* [^1] (also known as *atomic consistency* [^2], *strong consistency*, *immediate consistency*, or *external consistency* [^3]).
The exact definition of linearizability is quite subtle, and we will explore it in the rest of this
section. But the basic idea is to make a system appear as if there were only one copy of the data,
and all operations on it are atomic. With this guarantee, even though there may be multiple replicas
in reality, the application does not need to worry about them.
In a linearizable system, as soon as one client successfully completes a write, all clients reading
from the database must be able to see the value just written. Maintaining the illusion of a single
copy of the data means guaranteeing that the value read is the most recent, up-to-date value, and
doesn’t come from a stale cache or replica. In other words, linearizability is a *recency
guarantee*. To clarify this idea, let’s look at an example of a system that is not linearizable.
{{< figure src="/fig/ddia_1001.png" id="fig_consistency_linearizability_0" caption="Figure 10-1. If this database were linearizable, then either Alice's read would return 1 instead of 0, or Bob's read would return 0 instead of 1." class="w-full my-4" >}}
[Figure 10-1](/en/ch10#fig_consistency_linearizability_0) shows an example of a nonlinearizable sports website [^4].
Aaliyah and Bryce are sitting in the same room, both checking their phones to see the outcome of a
game their favorite team is playing. Just after the final score is announced, Aaliyah refreshes the
page, sees the winner announced, and excitedly tells Bryce about it. Bryce incredulously hits
*reload* on his own phone, but his request goes to a database replica that is lagging, and so his
phone shows that the game is still ongoing.
If Aaliyah and Bryce had hit reload at the same time, it would have been less surprising if they had
gotten two different query results, because they wouldn’t know at exactly what time their respective
requests were processed by the server. However, Bryce knows that he hit the reload button (initiated
his query) *after* he heard Aaliyah exclaim the final score, and therefore he expects his query
result to be at least as recent as Aaliyah’s. The fact that his query returned a stale result is a
violation of linearizability.
### What Makes a System Linearizable? {#sec_consistency_lin_definition}
In order to understand linearizability better, let’s look at some more examples.
[Figure 10-2](/en/ch10#fig_consistency_linearizability_1) shows three clients concurrently reading and writing the same
object *x* in a linearizable database. In distributed systems theory, *x* is called a *register*—in
practice, it could be one key in a key-value store, one row in a relational database, or one
document in a document database, for example.
{{< figure src="/fig/ddia_1002.png" id="fig_consistency_linearizability_1" caption="Figure 10-2. Alice observes that x = 0 and y = 1, while Bob observes that x = 1 and y = 0. It's as if Alice's and Bob's computers disagree on the order in which the writes happened." class="w-full my-4" >}}
For simplicity, [Figure 10-2](/en/ch10#fig_consistency_linearizability_1) shows only the requests from the clients’
point of view, not the internals of the database. Each bar is a request made by a client, where the
start of a bar is the time when the request was sent, and the end of a bar is when the response was
received by the client. Due to variable network delays, a client doesn’t know exactly when the
database processed its request—it only knows that it must have happened sometime between the
client sending the request and receiving the response.
In this example, the register has two types of operations:
* *read*(*x*) ⇒ *v* means the client requested to read the value of register
*x*, and the database returned the value *v*.
* *write*(*x*, *v*) ⇒ *r* means the client requested to set the
register *x* to value *v*, and the database returned response *r* (which could be *ok* or *error*).
In [Figure 10-2](/en/ch10#fig_consistency_linearizability_1), the value of *x* is initially 0, and client C performs a
write request to set it to 1. While this is happening, clients A and B are repeatedly polling the
database to read the latest value. What are the possible responses that A and B might get for their
read requests?
* The first read operation by client A completes before the write begins, so it must definitely
return the old value 0.
* The last read by client A begins after the write has completed, so it must definitely return the
new value 1 if the database is linearizable, because the read must have been processed after the
write.
* Any read operations that overlap in time with the write operation might return either 0 or 1,
because we don’t know whether or not the write has taken effect at the time when the read
operation is processed. These operations are *concurrent* with the write.
However, that is not yet sufficient to fully describe linearizability: if reads that are concurrent
with a write can return either the old or the new value, then readers could see a value flip back
and forth between the old and the new value several times while a write is going on. That is not
what we expect of a system that emulates a “single copy of the data.”
To make the system linearizable, we need to add another constraint, illustrated in
[Figure 10-3](/en/ch10#fig_consistency_linearizability_2).
{{< figure src="/fig/ddia_1003.png" id="fig_consistency_linearizability_2" caption="Figure 10-3. If Alice and Bob had perfect clocks, linearizability would require that x = 1 is returned, since the read of x begins after the write x = 1 completes." class="w-full my-4" >}}
In a linearizable system we imagine that there must be some point in time (between the start and end
of the write operation) at which the value of *x* atomically flips from 0 to 1. Thus, if one
client’s read returns the new value 1, all subsequent reads must also return the new value, even if
the write operation has not yet completed.
This timing dependency is illustrated with an arrow in [Figure 10-3](/en/ch10#fig_consistency_linearizability_2).
Client A is the first to read the new value, 1. Just after A’s read returns, B begins a new read.
Since B’s read occurs strictly after A’s read, it must also return 1, even though the write by C is
still ongoing. (It’s the same situation as with Aaliyah and Bryce in
[Figure 10-1](/en/ch10#fig_consistency_linearizability_0): after Aaliyah has read the new value, Bryce also expects to
read the new value.)
We can further refine this timing diagram to visualize each operation taking effect atomically at
some point in time [^5],
like in the more complex example shown in [Figure 10-4](/en/ch10#fig_consistency_linearizability_3). In this example we
add a third type of operation besides *read* and *write*:
* *cas*(*x*, *v*old, *v*new) ⇒ *r* means the client
requested an atomic *compare-and-set* operation (see [“Conditional writes (compare-and-set)”](/en/ch8#sec_transactions_compare_and_set)). If the
current value of the register *x* equals *v*old, it should be atomically set to *v*new. If
the value of *x* is different from *v*old, then the operation should leave the register
unchanged and return an error. *r* is the database’s response (*ok* or *error*).
Each operation in [Figure 10-4](/en/ch10#fig_consistency_linearizability_3) is marked with a vertical line (inside the
bar for each operation) at the time when we think the operation was executed. Those markers are
joined up in a sequential order, and the result must be a valid sequence of reads and writes for a
register (every read must return the value set by the most recent write).
The requirement of linearizability is that the lines joining up the operation markers always move
forward in time (from left to right), never backward. This requirement ensures the recency guarantee we
discussed earlier: once a new value has been written or read, all subsequent reads see the value
that was written, until it is overwritten again.
{{< figure src="/fig/ddia_1004.png" id="fig_consistency_linearizability_3" caption="Figure 10-4. The read of x is concurrent with the write x = 1. Since we don't know the exact timing of the operations, the read is allowed to return either 0 or 1." class="w-full my-4" >}}
There are a few interesting details to point out in [Figure 10-4](/en/ch10#fig_consistency_linearizability_3):
* First client B sent a request to read *x*, then client D sent a request to set *x* to 0, and then
client A sent a request to set *x* to 1. Nevertheless, the value returned to B’s read is 1 (the
value written by A). This is okay: it means that the database first processed D’s write, then A’s
write, and finally B’s read. Although this is not the order in which the requests were sent, it’s
an acceptable order, because the three requests are concurrent. Perhaps B’s read request was
slightly delayed in the network, so it only reached the database after the two writes.
* Client B’s read returned 1 before client A received its response from the database, saying that
the write of the value 1 was successful. This is also okay: it just means the *ok* response from
the database to client A was slightly delayed in the network.
* This model doesn’t assume any transaction isolation: another client may change a value at any
time. For example, C first reads 1 and then reads 2, because the value was changed by B between
the two reads. An atomic compare-and-set (*cas*) operation can be used to check the value hasn’t
been concurrently changed by another client: B and C’s *cas* requests succeed, but D’s *cas*
request fails (by the time the database processes it, the value of *x* is no longer 0).
* The final read by client B (in a shaded bar) is not linearizable. The operation is concurrent with
C’s *cas* write, which updates *x* from 2 to 4. In the absence of other requests, it would be okay for
B’s read to return 2. However, client A has already read the new value 4 before B’s read started,
so B is not allowed to read an older value than A. Again, it’s the same situation as with Aaliyah
and Bryce in [Figure 10-1](/en/ch10#fig_consistency_linearizability_0).
That is the intuition behind linearizability; the formal definition [^1] describes it more precisely. It is
possible (though computationally expensive) to test whether a system’s behavior is linearizable by
recording the timings of all requests and responses, and checking whether they can be arranged into
a valid sequential order [^6] [^7].
Just as there are various weak isolation levels for transactions besides serializability (see
[“Weak Isolation Levels”](/en/ch8#sec_transactions_isolation_levels)), there are also various weaker consistency models for
replicated systems besides linearizability [^8].
In fact, the *read-after-write*, *monotonic reads*, and *consistent prefix reads* properties we saw
in [“Problems with Replication Lag”](/en/ch6#sec_replication_lag) are examples of such weaker consistency models. Linearizability
guarantees all these weaker properties, and more. In this chapter we will focus on linearizability,
which is the strongest consistency model in common use.
--------
<a id="sidebar_consistency_serializability"></a>
> [!TIP] LINEARIZABILITY VERSUS SERIALIZABILITY
Linearizability is easily confused with serializability (see [“Serializability”](/en/ch8#sec_transactions_serializability)),
as both words seem to mean something like “can be arranged in a sequential order.” However, they are
quite different guarantees, and it is important to distinguish between them:
Serializability
: Serializability is an isolation property of transactions, where every transaction may read and
write *multiple objects* (rows, documents, records). It guarantees that transactions behave the
same as if they had executed in *some* serial order: that is, as if you first performed all of one
transaction’s operations, then all of another transaction’s operations, and so on, without
interleaving them. It is okay for that serial order to be different from the order in which the
transactions were actually run [^9].
Linearizability
: Linearizability is a guarantee on reads and writes of a register (an *individual object*). It
doesn’t group operations together into transactions, so it does not prevent problems such as write
skew that involve multiple objects (see [“Write Skew and Phantoms”](/en/ch8#sec_transactions_write_skew)). However, linearizability
is a *recency* guarantee: it requires that if one operation finishes before another one starts,
then the later operation must observe a state that is at least as new as the earlier operation.
Serializability does not have that requirement: for example, stale reads are allowed by
serializability [^10].
(*Sequential consistency* is something else again [^8], but we won’t discuss it here.)
A database may provide both serializability and linearizability, and this combination is known as
*strict serializability* or *strong one-copy serializability* (*strong-1SR*) [^11] [^12].
Single-node databases are typically linearizable. With distributed databases using optimistic
methods like serializable snapshot isolation (see [“Serializable Snapshot Isolation (SSI)”](/en/ch8#sec_transactions_ssi)) the situation is more
complicated: for example, CockroachDB provides serializability, and some recency guarantees on
reads, but not strict serializability [^13]
because this would require expensive coordination between transactions [^14].
It is also possible to combine a weaker isolation level with linearizability, or a weaker
consistency model with serializability; in fact, consistency model and isolation level can be chosen
largely independently from each other [^15] [^16].
--------
### Relying on Linearizability {#sec_consistency_linearizability_usage}
In what circumstances is linearizability useful? Viewing the final score of a sporting match is
perhaps a frivolous example: a result that is outdated by a few seconds is unlikely to cause any
real harm in this situation. However, there a few areas in which linearizability is an important
requirement for making a system work correctly.
#### Locking and leader election {#locking-and-leader-election}
A system that uses single-leader replication needs to ensure that there is indeed only one leader,
not several (split brain). One way of electing a leader is to use a lease: every node that starts up
tries to acquire the lease, and the one that succeeds becomes the leader [^17].
No matter how this mechanism is implemented, it must be linearizable: it should not be possible for
two different nodes to acquire the lease at the same time.
Coordination services like Apache ZooKeeper [^18]
and etcd are often used to implement distributed leases and leader election. They use consensus
algorithms to implement linearizable operations in a fault-tolerant way (we discuss such algorithms
later in this chapter). There are still many subtle details to implementing leases and leader
election correctly (see for example the fencing issue in [“Distributed Locks and Leases”](/en/ch9#sec_distributed_lock_fencing)), and
libraries like Apache Curator help by providing higher-level recipes on top of ZooKeeper. However, a
linearizable storage service is the basic foundation for these coordination tasks.
--------
> [!NOTE]
> Strictly speaking, ZooKeeper provides linearizable writes, but reads may be stale, since there is no
> guarantee that they are served from the current leader [^18]. etcd since version 3 provides linearizable reads by default.
--------
Distributed locking is also used at a much more granular level in some distributed databases, such as
Oracle Real Application Clusters (RAC) [^19].
RAC uses a lock per disk page, with multiple nodes sharing access
to the same disk storage system. Since these linearizable locks are on the critical path of
transaction execution, RAC deployments usually have a dedicated cluster interconnect network for
communication between database nodes.
#### Constraints and uniqueness guarantees {#sec_consistency_uniqueness}
Uniqueness constraints are common in databases: for example, a username or email address must
uniquely identify one user, and in a file storage service there cannot be two files with the same
path and filename. If you want to enforce this constraint as the data is written (such that if two people
try to concurrently create a user or a file with the same name, one of them will be returned an
error), you need linearizability.
This situation is actually similar to a lock: when a user registers for your service, you can think
of them acquiring a “lock” on their chosen username. The operation is also very sim
gitextract_zlswtsh8/ ├── .github/ │ └── workflows/ │ └── pages.yaml ├── .gitignore ├── .nojekyll ├── LICENSE ├── Makefile ├── README.md ├── assets/ │ └── css/ │ ├── custom.css │ └── example.css ├── bin/ │ ├── Pipfile │ ├── doc │ ├── epub │ ├── preprocess-epub.py │ ├── toc.py │ ├── translate.py │ └── zh-tw.py ├── content/ │ ├── en/ │ │ ├── _index.md │ │ ├── ch1.md │ │ ├── ch10.md │ │ ├── ch11.md │ │ ├── ch12.md │ │ ├── ch13.md │ │ ├── ch14.md │ │ ├── ch2.md │ │ ├── ch3.md │ │ ├── ch4.md │ │ ├── ch5.md │ │ ├── ch6.md │ │ ├── ch7.md │ │ ├── ch8.md │ │ ├── ch9.md │ │ ├── colophon.md │ │ ├── glossary.md │ │ ├── indexes.md │ │ ├── part-i.md │ │ ├── part-ii.md │ │ ├── part-iii.md │ │ ├── preface.md │ │ └── toc.md │ ├── tw/ │ │ ├── _index.md │ │ ├── ch1.md │ │ ├── ch10.md │ │ ├── ch11.md │ │ ├── ch12.md │ │ ├── ch13.md │ │ ├── ch14.md │ │ ├── ch2.md │ │ ├── ch3.md │ │ ├── ch4.md │ │ ├── ch5.md │ │ ├── ch6.md │ │ ├── ch7.md │ │ ├── ch8.md │ │ ├── ch9.md │ │ ├── colophon.md │ │ ├── contrib.md │ │ ├── glossary.md │ │ ├── indexes.md │ │ ├── part-i.md │ │ ├── part-ii.md │ │ ├── part-iii.md │ │ ├── preface.md │ │ └── toc.md │ ├── v1/ │ │ ├── _index.md │ │ ├── ch1.md │ │ ├── ch10.md │ │ ├── ch11.md │ │ ├── ch12.md │ │ ├── ch2.md │ │ ├── ch3.md │ │ ├── ch4.md │ │ ├── ch5.md │ │ ├── ch6.md │ │ ├── ch7.md │ │ ├── ch8.md │ │ ├── ch9.md │ │ ├── colophon.md │ │ ├── contrib.md │ │ ├── glossary.md │ │ ├── part-i.md │ │ ├── part-ii.md │ │ ├── part-iii.md │ │ ├── preface.md │ │ └── toc.md │ ├── v1_tw/ │ │ ├── _index.md │ │ ├── ch1.md │ │ ├── ch10.md │ │ ├── ch11.md │ │ ├── ch12.md │ │ ├── ch2.md │ │ ├── ch3.md │ │ ├── ch4.md │ │ ├── ch5.md │ │ ├── ch6.md │ │ ├── ch7.md │ │ ├── ch8.md │ │ ├── ch9.md │ │ ├── colophon.md │ │ ├── contrib.md │ │ ├── glossary.md │ │ ├── part-i.md │ │ ├── part-ii.md │ │ ├── part-iii.md │ │ ├── preface.md │ │ └── toc.md │ └── zh/ │ ├── _index.md │ ├── ch1.md │ ├── ch10.md │ ├── ch11.md │ ├── ch12.md │ ├── ch13.md │ ├── ch14.md │ ├── ch2.md │ ├── ch3.md │ ├── ch4.md │ ├── ch5.md │ ├── ch6.md │ ├── ch7.md │ ├── ch8.md │ ├── ch9.md │ ├── colophon.md │ ├── contrib.md │ ├── glossary.md │ ├── indexes.md │ ├── part-i.md │ ├── part-ii.md │ ├── part-iii.md │ ├── preface.md │ └── toc.md ├── giscus.json ├── go.mod ├── go.sum ├── hugo.yaml ├── i18n/ │ ├── en.yaml │ ├── tw.yaml │ ├── v2.yaml │ └── zh.yaml ├── js/ │ └── epub.css ├── layouts/ │ └── shortcodes/ │ └── figure.html └── metadata.yaml
SYMBOL INDEX (17 symbols across 4 files) FILE: bin/preprocess-epub.py function _escape_alt_text (line 20) | def _escape_alt_text(text): function convert_markdown (line 25) | def convert_markdown(text): function process_file (line 59) | def process_file(input_path, output_path): function main (line 80) | def main(): FILE: bin/toc.py function extract_front_matter_title (line 15) | def extract_front_matter_title(content): function extract_headings (line 36) | def extract_headings(content, max_depth): function generate_toc_entry (line 78) | def generate_toc_entry(file_name, title, lang, depth, content_dir): function format_toc_entries (line 132) | def format_toc_entries(entries): function check_file_status (line 161) | def check_file_status(file_path, lang): function main (line 185) | def main(): FILE: bin/translate.py function cli (line 12) | def cli(): function convert (line 16) | def convert(infile: str, outfile: str, cfg: str): function file (line 33) | def file(infile: str, outfile: str, cfg: str): function repo (line 47) | def repo(infolder, outfolder, cfg): FILE: bin/zh-tw.py function process_urls (line 5) | def process_urls(text, src_folder, dst_folder): function convert_file (line 37) | def convert_file(src_filepath, dst_filepath, src_folder, dst_folder, cfg... function convert (line 69) | def convert(zh_folder, tw_folder):
Condensed preview — 139 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (5,771K chars).
[
{
"path": ".github/workflows/pages.yaml",
"chars": 2151,
"preview": "# Sample workflow for building and deploying a Hugo site to GitHub Pages\nname: Deploy Hugo site to Pages\n\non:\n # Runs o"
},
{
"path": ".gitignore",
"chars": 116,
"preview": ".idea/\n.code/\n__pycache__/\n.DS_Store\ntmp/\noutput/\npublic/\n.hugo_build.lock\n.claude\nCLAUDE.md\ncontent/cn/\nzh.md\nen.md"
},
{
"path": ".nojekyll",
"chars": 0,
"preview": ""
},
{
"path": "LICENSE",
"chars": 18652,
"preview": "Attribution 4.0 International\n\n=======================================================================\n\nCreative Commons"
},
{
"path": "Makefile",
"chars": 195,
"preview": "default: dev\n\nd:dev\ndev:\n\thugo serve\n\nb:build\nbuild:\n\thugo build\n\n.PHONY: default d dev b build\n\n# generate zh-tw versio"
},
{
"path": "README.md",
"chars": 42430,
"preview": "# 设计数据密集型应用（第二版） - 中文翻译版\n\n[](https://ddia.von"
},
{
"path": "assets/css/custom.css",
"chars": 746,
"preview": "/* 调整左侧导航栏的宽度 */\n/* 增加宽度以确保文本不换行 */\n\n/* 左侧导航栏宽度调整 - 增加到 28rem 以避免换行 */\n.hextra-sidebar {\n width: 28rem !important;\n mi"
},
{
"path": "assets/css/example.css",
"chars": 492,
"preview": ".md-example { margin: 1.25rem 0; padding: 1rem; border: 1px solid var(--border); border-radius: .75rem; }\n.md-example__c"
},
{
"path": "bin/Pipfile",
"chars": 163,
"preview": "[[source]]\nurl = \"https://pypi.org/simple\"\nverify_ssl = true\nname = \"pypi\"\n\n[packages]\nopencc = \"*\"\nclick = \"*\"\n\n[dev-pa"
},
{
"path": "bin/doc",
"chars": 998,
"preview": "#!/bin/bash\n\n#==============================================================#\n# File : doc\n# Ctime : 2021-0"
},
{
"path": "bin/epub",
"chars": 1612,
"preview": "#!/usr/bin/env bash\n\nset -e\n\n# Set the directory containing Markdown files\nSCRIPT_DIR=$(dirname \"$0\")\nINPUT_DIR=$(cd \"$("
},
{
"path": "bin/preprocess-epub.py",
"chars": 2946,
"preview": "#!/usr/bin/env python3\n\"\"\"\n预处理 Markdown 文件，将 Hugo shortcode 转换为 Pandoc 可识别的格式\n\n处理两种 shortcode：\n1. {{< figure src=\"/fig/x"
},
{
"path": "bin/toc.py",
"chars": 8845,
"preview": "#!/usr/bin/env python3\n\"\"\"\nTOC Generator for DDIA book\nUsage: python toc.py <lang> <depth> [output_file]\nExample: python"
},
{
"path": "bin/translate.py",
"chars": 1708,
"preview": "\"\"\"Convert zh-cn to zh-tw\nRefer to https://github.com/BYVoid/OpenCC\n\"\"\"\nimport click\nimport opencc\n\nfrom pathlib import "
},
{
"path": "bin/zh-tw.py",
"chars": 3583,
"preview": "#!/usr/bin/env python3\nimport os, sys, opencc\nimport re\n\ndef process_urls(text, src_folder, dst_folder):\n \"\"\"处理 Markd"
},
{
"path": "content/en/_index.md",
"chars": 2223,
"preview": "---\ntitle: \"Designing Data-Intensive Applications 2nd Edition\"\nlinkTitle: DDIA\ncascade:\n type: docs\nbreadcrumbs: false\n"
},
{
"path": "content/en/ch1.md",
"chars": 89640,
"preview": "---\ntitle: \"1. Trade-offs in Data Systems Architecture\"\nweight: 101\nbreadcrumbs: false\n---\n\n<a id=\"ch_tradeoffs\"></a>\n\n>"
},
{
"path": "content/en/ch10.md",
"chars": 136412,
"preview": "---\ntitle: \"10. Consistency and Consensus\"\nweight: 210\nbreadcrumbs: false\n---\n\n<a id=\"ch_consistency\"></a>\n\n![](/map/ch0"
},
{
"path": "content/en/ch11.md",
"chars": 95361,
"preview": "---\ntitle: \"11. Batch Processing\"\nweight: 311\nbreadcrumbs: false\n---\n\n<a id=\"ch_batch\"></a>\n\n\n\n> *A sy"
},
{
"path": "content/en/ch12.md",
"chars": 144526,
"preview": "---\ntitle: \"12. Stream Processing\"\nweight: 312\nbreadcrumbs: false\n---\n\n<a id=\"ch_stream\"></a>\n\n\n\n> *A "
},
{
"path": "content/en/ch13.md",
"chars": 130498,
"preview": "---\ntitle: \"13. A Philosophy of Streaming Systems\"\nweight: 313\nbreadcrumbs: false\n---\n\n<a id=\"ch_philosophy\"></a>\n\n![](/"
},
{
"path": "content/en/ch14.md",
"chars": 53701,
"preview": "---\ntitle: \"14. Doing the Right Thing\"\nweight: 314\nbreadcrumbs: false\n---\n\n<a id=\"ch_right_thing\"></a>\n\n![](/map/ch13.pn"
},
{
"path": "content/en/ch2.md",
"chars": 89929,
"preview": "---\ntitle: \"2. Defining Nonfunctional Requirements\"\nweight: 102\nbreadcrumbs: false\n---\n\n<a id=\"ch_nonfunctional\"></a>\n\n!"
},
{
"path": "content/en/ch3.md",
"chars": 122235,
"preview": "---\ntitle: \"3. Data Models and Query Languages\"\nweight: 103\nbreadcrumbs: false\n---\n\n<a id=\"ch_datamodels\"></a>\n\n![](/map"
},
{
"path": "content/en/ch4.md",
"chars": 117736,
"preview": "---\ntitle: \"4. Storage and Retrieval\"\nweight: 104\nbreadcrumbs: false\n---\n\n<a id=\"ch_storage\"></a>\n\n\n\n>"
},
{
"path": "content/en/ch5.md",
"chars": 88657,
"preview": "---\ntitle: \"5. Encoding and Evolution\"\nweight: 105\nbreadcrumbs: false\n---\n\n<a id=\"ch_encoding\"></a>\n\n\n"
},
{
"path": "content/en/ch6.md",
"chars": 138687,
"preview": "---\ntitle: \"6. Replication\"\nweight: 206\nbreadcrumbs: false\n---\n\n<a id=\"ch_replication\"></a>\n\n\n\n> *The "
},
{
"path": "content/en/ch7.md",
"chars": 61172,
"preview": "---\ntitle: \"7. Sharding\"\nweight: 207\nbreadcrumbs: false\n---\n\n<a id=\"ch_sharding\"></a>\n\n\n\n> *Clearly, w"
},
{
"path": "content/en/ch8.md",
"chars": 181961,
"preview": "---\ntitle: \"8. Transactions\"\nweight: 208\nbreadcrumbs: false\n---\n\n<a id=\"ch_transactions\"></a>\n\n\n\n> *So"
},
{
"path": "content/en/ch9.md",
"chars": 157373,
"preview": "---\ntitle: \"9. The Trouble with Distributed Systems\"\nweight: 209\nbreadcrumbs: false\n---\n\n<a id=\"ch_distributed\"></a>\n\n\n\n\n\n## ["
},
{
"path": "content/tw/_index.md",
"chars": 41534,
"preview": "---\ntitle: 設計資料密集型應用（第二版）\nlinkTitle: DDIA\ncascade:\n type: docs\nbreadcrumbs: false\n---\n\n\n**作者**： [Martin Kleppmann](http"
},
{
"path": "content/tw/ch1.md",
"chars": 42040,
"preview": "---\ntitle: \"1. 資料系統架構中的權衡\"\nweight: 101\nbreadcrumbs: false\n---\n\n<a id=\"ch_tradeoffs\"></a>\n\n> *沒有完美的解決方案，只有權衡取捨。[…] 你能做的就是"
},
{
"path": "content/tw/ch10.md",
"chars": 64837,
"preview": "---\ntitle: \"10. 一致性與共識\"\nweight: 210\nbreadcrumbs: false\n---\n\n<a id=\"ch_consistency\"></a>\n\n\n\n> *一句古老的格言告"
},
{
"path": "content/tw/ch11.md",
"chars": 40211,
"preview": "---\ntitle: \"第十一章：批處理\"\nlinkTitle: \"11. 批處理\"\nweight: 311\nbreadcrumbs: false\n---\n\n<a id=\"ch_batch\"></a>\n\n"
},
{
"path": "content/tw/ch12.md",
"chars": 63995,
"preview": "---\ntitle: \"第十二章：流處理\"\nlinkTitle: \"12. 流處理\"\nweight: 312\nmath: true\nbreadcrumbs: false\n---\n\n<a id=\"ch_stream\"></a>\n\n![](/m"
},
{
"path": "content/tw/ch13.md",
"chars": 58593,
"preview": "---\ntitle: \"第十三章：流式系統的哲學\"\nlinkTitle: \"13. 流式系統的哲學\"\nweight: 313\nbreadcrumbs: false\n---\n\n<a id=\"ch_philosophy\"></a>\n<a id="
},
{
"path": "content/tw/ch14.md",
"chars": 26112,
"preview": "---\ntitle: \"14. 將事情做正確\"\nweight: 314\nbreadcrumbs: false\n---\n\n<a id=\"ch_right_thing\"></a>\n\n\n\n> *將世界的美好、醜"
},
{
"path": "content/tw/ch2.md",
"chars": 45813,
"preview": "---\ntitle: \"2. 定義非功能性需求\"\nweight: 102\nbreadcrumbs: false\n---\n\n<a id=\"ch_nonfunctional\"></a>\n\n\n\n> *網際網路做"
},
{
"path": "content/tw/ch3.md",
"chars": 62491,
"preview": "---\ntitle: \"3. 資料模型與查詢語言\"\nweight: 103\nbreadcrumbs: false\n---\n\n<a id=\"ch_datamodels\"></a>\n\n\n\n> *語言的邊界就是"
},
{
"path": "content/tw/ch4.md",
"chars": 62242,
"preview": "---\ntitle: \"4. 儲存與檢索\"\nweight: 104\nbreadcrumbs: false\n---\n\n<a id=\"ch_storage\"></a>\n\n\n\n> *生活的苦惱之一是，每個人對事"
},
{
"path": "content/tw/ch5.md",
"chars": 41997,
"preview": "---\ntitle: \"5. 編碼與演化\"\nweight: 105\nmath: true\nbreadcrumbs: false\n---\n\n<a id=\"ch_encoding\"></a>\n\n\n\n> *萬物"
},
{
"path": "content/tw/ch6.md",
"chars": 61269,
"preview": "---\ntitle: \"6. 複製\"\nweight: 206\nbreadcrumbs: false\n---\n\n<a id=\"ch_replication\"></a>\n\n\n\n> *可能出錯的東西和“不可能”"
},
{
"path": "content/tw/ch7.md",
"chars": 28587,
"preview": "---\ntitle: \"7. 分片\"\nweight: 207\nbreadcrumbs: false\n---\n\n<a id=\"ch_sharding\"></a>\n\n\n\n> *顯然，我們必須跳出順序計算機指令"
},
{
"path": "content/tw/ch8.md",
"chars": 80836,
"preview": "---\ntitle: \"8. 事務\"\nweight: 208\nmath: true\nbreadcrumbs: false\n---\n\n<a id=\"ch_transactions\"></a>\n\n\n\n> *有"
},
{
"path": "content/tw/ch9.md",
"chars": 77848,
"preview": "---\ntitle: \"9. 分散式系統的麻煩\"\nweight: 209\nbreadcrumbs: false\n---\n\n<a id=\"ch_distributed\"></a>\n\n\n\n> *意外這東西挺有"
},
{
"path": "content/tw/colophon.md",
"chars": 1757,
"preview": "---\ntitle: 後記\nweight: 600\nbreadcrumbs: false\n---\n\n{{< callout type=\"warning\" >}}\n當前頁面來自本書第一版，第二版尚不可用\n{{< /callout >}}\n\n#"
},
{
"path": "content/tw/contrib.md",
"chars": 36204,
"preview": "---\ntitle: 貢獻者\nweight: 800\nbreadcrumbs: false\n---\n\n## 譯者\n\n[**馮若航**](https://vonng.com)，網名 [@Vonng](https://github.com/Vo"
},
{
"path": "content/tw/glossary.md",
"chars": 7506,
"preview": "---\ntitle: 術語表\nweight: 500\nbreadcrumbs: false\n---\n\n> 請注意：本術語表的定義刻意保持簡短，旨在傳達核心概念，而非覆蓋術語的全部細節。更多內容請參閱正文對應章節。\n\n### 非同步（asyn"
},
{
"path": "content/tw/indexes.md",
"chars": 215624,
"preview": "---\ntitle: 索引\nweight: 550\nbreadcrumbs: false\n---\n\n### 符號\n\n- 3FS（分散式檔案系統）, [分散式檔案系統](/tw/ch11#sec_batch_dfs)\n\n### A\n\n- 中止"
},
{
"path": "content/tw/part-i.md",
"chars": 1753,
"preview": "---\ntitle: 第一部分：資料系統基礎\nweight: 100\nbreadcrumbs: false\n---\n\n{{< callout type=\"warning\" >}}\n當前頁面來自本書第一版，第二版尚不可用\n{{< /callo"
},
{
"path": "content/tw/part-ii.md",
"chars": 4750,
"preview": "---\ntitle: 第二部分：分散式資料\nweight: 200\nbreadcrumbs: false\n---\n\n{{< callout type=\"warning\" >}}\n當前頁面來自本書第一版，第二版尚不可用\n{{< /callou"
},
{
"path": "content/tw/part-iii.md",
"chars": 2364,
"preview": "---\ntitle: 第三部分：派生資料\nweight: 300\nbreadcrumbs: false\n---\n\n{{< callout type=\"warning\" >}}\n當前頁面來自本書第一版，第二版尚不可用\n{{< /callout"
},
{
"path": "content/tw/preface.md",
"chars": 7359,
"preview": "---\ntitle: 序言\nweight: 50\nbreadcrumbs: false\n---\n\n{{< callout type=\"warning\" >}}\n當前頁面來自本書第一版，第二版尚不可用\n{{< /callout >}}\n\n如果"
},
{
"path": "content/tw/toc.md",
"chars": 3432,
"preview": "---\ntitle: \"目錄\"\nlinkTitle: \"目錄\"\nweight: 10\nbreadcrumbs: false\n---\n\n\n\n\n\n\n## [序言](/tw/preface)\n- [本書的目標讀者]("
},
{
"path": "content/v1/_index.md",
"chars": 41165,
"preview": "---\ntitle: 设计数据密集型应用（第一版）\nlinkTitle: DDIA\ncascade:\n type: docs\nbreadcrumbs: false\n---\n\n\n**作者**： [Martin Kleppmann](http"
},
{
"path": "content/v1/ch1.md",
"chars": 23626,
"preview": "---\ntitle: \"第一章：可靠性、可伸缩性和可维护性\"\nlinkTitle: \"1. 可靠性、可伸缩性和可维护性\"\nweight: 101\nbreadcrumbs: false\n---\n\n\n\n\n> "
},
{
"path": "content/v1/ch10.md",
"chars": 60019,
"preview": "---\ntitle: \"第十章：批处理\"\nlinkTitle: \"10. 批处理\"\nweight: 310\nbreadcrumbs: false\n---\n\n\n\n> 带有太强个人色彩的系统无法成功。当最初的"
},
{
"path": "content/v1/ch11.md",
"chars": 58337,
"preview": "---\ntitle: \"第十一章：流处理\"\nlinkTitle: \"11. 流处理\"\nweight: 311\nmath: true\nbreadcrumbs: false\n---\n\n\n\n\n> 有效的复杂系统"
},
{
"path": "content/v1/ch12.md",
"chars": 76778,
"preview": "---\ntitle: \"第十二章：数据系统的未来\"\nlinkTitle: \"12. 数据系统的未来\"\nweight: 312\nbreadcrumbs: false\n---\n\n\n\n\n> 如果船长的终极目标是"
},
{
"path": "content/v1/ch2.md",
"chars": 45311,
"preview": "---\ntitle: \"第二章：数据模型与查询语言\"\nlinkTitle: \"2. 数据模型与查询语言\"\nweight: 102\nmath: true\nbreadcrumbs: false\n---\n\n\n\n"
},
{
"path": "content/v1/ch3.md",
"chars": 44097,
"preview": "---\ntitle: \"第三章：存储与检索\"\nlinkTitle: \"3. 存储与检索\"\nweight: 103\nbreadcrumbs: false\n---\n\n\n\n\n> 建立秩序，省却搜索\n>\n> ——"
},
{
"path": "content/v1/ch4.md",
"chars": 34964,
"preview": "---\ntitle: \"第四章：编码与演化\"\nlinkTitle: \"4. 编码与演化\"\nweight: 104\nbreadcrumbs: false\n---\n\n\n\n> 唯变所适\n>\n> —— 以弗所的赫"
},
{
"path": "content/v1/ch5.md",
"chars": 47536,
"preview": "---\ntitle: \"第五章：复制\"\nlinkTitle: \"5. 复制\"\nweight: 205\nbreadcrumbs: false\nmath: true\n---\n\n\n\n> 与可能出错的东西比，“不"
},
{
"path": "content/v1/ch6.md",
"chars": 21303,
"preview": "---\nlinktitle: \"第六章：分区\"\nlinkTitle: \"6. 分区\"\nweight: 206\nbreadcrumbs: false\n---\n\n\n\n\n> 我们必须跳出电脑指令序列的窠臼。叙述"
},
{
"path": "content/v1/ch7.md",
"chars": 51299,
"preview": "---\ntitle: \"第七章：事务\"\nlinkTitle: \"7. 事务\"\nweight: 207\nbreadcrumbs: false\n---\n\n\n\n> 一些作者声称，支持通用的两阶段提交代价太大，会"
},
{
"path": "content/v1/ch8.md",
"chars": 57458,
"preview": "---\ntitle: \"第八章：分布式系统的麻烦\"\nlinkTitle: \"8. 分布式系统的麻烦\"\nweight: 208\nmath: true\nbreadcrumbs: false\n---\n\n\n\n\n>"
},
{
"path": "content/v1/ch9.md",
"chars": 78821,
"preview": "---\ntitle: \"第九章：一致性与共识\"\nlinkTitle: \"9. 一致性与共识\"\nweight: 209\nmath: true\nbreadcrumbs: false\n---\n\n\n\n\n> 好死还"
},
{
"path": "content/v1/colophon.md",
"chars": 1497,
"preview": "---\ntitle: 后记\nweight: 600\nbreadcrumbs: false\n---\n\n## 关于作者\n\n**Martin Kleppmann** 是英国剑桥大学分布式系统的研究员。此前他曾在互联网公司担任过软件工程师和企业家，"
},
{
"path": "content/v1/contrib.md",
"chars": 36187,
"preview": "---\ntitle: 贡献者\nweight: 800\nbreadcrumbs: false\n---\n\n## 译者\n\n[**冯若航**](https://vonng.com)，网名 [@Vonng](https://github.com/Vo"
},
{
"path": "content/v1/glossary.md",
"chars": 7144,
"preview": "---\ntitle: 术语表\nweight: 500\nbreadcrumbs: false\n---\n\n> 请注意，本术语表中的定义简短而简单，旨在传达核心思想，而非死扣完整细节。有关更多详细信息，请参阅正文中的参考资料。\n\n\n## **异步"
},
{
"path": "content/v1/part-i.md",
"chars": 1082,
"preview": "---\ntitle: 第一部分：数据系统基础\nweight: 100\nbreadcrumbs: false\n---\n\n本书前四章介绍了数据系统底层的基础概念，无论是在单台机器上运行的单点数据系统，还是分布在多台机器上的分布式数据系统都适用。"
},
{
"path": "content/v1/part-ii.md",
"chars": 4216,
"preview": "---\ntitle: 第二部分：分布式数据\nweight: 200\nbreadcrumbs: false\n---\n\n> 一个成功的技术，现实的优先级必须高于公关，你可以糊弄别人，但糊弄不了自然规律。\n>\n> —— 罗杰斯委员会报告（1986"
},
{
"path": "content/v1/part-iii.md",
"chars": 1983,
"preview": "---\ntitle: 第三部分：衍生数据\nweight: 300\nbreadcrumbs: false\n---\n\n\n在本书的 [第一部分](/v1/part-i) 和 [第二部分](/v1/part-ii) 中，我们自底向上地把所有关于分布"
},
{
"path": "content/v1/preface.md",
"chars": 6689,
"preview": "---\ntitle: 序言\nweight: 50\nbreadcrumbs: false\n---\n\n\n如果近几年从业于软件工程，特别是服务器端和后端系统开发，那么你很有可能已经被大量关于数据存储和处理的时髦词汇轰炸过了： NoSQL！大数据！"
},
{
"path": "content/v1/toc.md",
"chars": 2007,
"preview": "---\ntitle: \"目录\"\nlinkTitle: \"目录\"\nweight: 10\nbreadcrumbs: false\n---\n\n\n\n\n## [序言](/v1/preface)\n\n## [第一部分：数"
},
{
"path": "content/v1_tw/_index.md",
"chars": 41248,
"preview": "---\ntitle: 設計資料密集型應用（第一版）\nlinkTitle: DDIA\ncascade:\n type: docs\nbreadcrumbs: false\n---\n\n\n**作者**： [Martin Kleppmann](http"
},
{
"path": "content/v1_tw/ch1.md",
"chars": 23707,
"preview": "---\ntitle: \"第一章：可靠性、可伸縮性和可維護性\"\nlinkTitle: \"1. 可靠性、可伸縮性和可維護性\"\nweight: 101\nbreadcrumbs: false\n---\n\n\n\n\n> "
},
{
"path": "content/v1_tw/ch10.md",
"chars": 60500,
"preview": "---\ntitle: \"第十章：批處理\"\nlinkTitle: \"10. 批處理\"\nweight: 310\nbreadcrumbs: false\n---\n\n\n\n> 帶有太強個人色彩的系統無法成功。當最初的"
},
{
"path": "content/v1_tw/ch11.md",
"chars": 58717,
"preview": "---\ntitle: \"第十一章：流處理\"\nlinkTitle: \"11. 流處理\"\nweight: 311\nmath: true\nbreadcrumbs: false\n---\n\n\n\n\n> 有效的複雜系統"
},
{
"path": "content/v1_tw/ch12.md",
"chars": 77509,
"preview": "---\ntitle: \"第十二章：資料系統的未來\"\nlinkTitle: \"12. 資料系統的未來\"\nweight: 312\nbreadcrumbs: false\n---\n\n\n\n\n> 如果船長的終極目標是"
},
{
"path": "content/v1_tw/ch2.md",
"chars": 45442,
"preview": "---\ntitle: \"第二章：資料模型與查詢語言\"\nlinkTitle: \"2. 資料模型與查詢語言\"\nweight: 102\nmath: true\nbreadcrumbs: false\n---\n\n\n\n"
},
{
"path": "content/v1_tw/ch3.md",
"chars": 44295,
"preview": "---\ntitle: \"第三章：儲存與檢索\"\nlinkTitle: \"3. 儲存與檢索\"\nweight: 103\nbreadcrumbs: false\n---\n\n\n\n\n> 建立秩序，省卻搜尋\n>\n> ——"
},
{
"path": "content/v1_tw/ch4.md",
"chars": 35182,
"preview": "---\ntitle: \"第四章：編碼與演化\"\nlinkTitle: \"4. 編碼與演化\"\nweight: 104\nbreadcrumbs: false\n---\n\n\n\n> 唯變所適\n>\n> —— 以弗所的赫"
},
{
"path": "content/v1_tw/ch5.md",
"chars": 47796,
"preview": "---\ntitle: \"第五章：複製\"\nlinkTitle: \"5. 複製\"\nweight: 205\nbreadcrumbs: false\nmath: true\n---\n\n\n\n> 與可能出錯的東西比，“不"
},
{
"path": "content/v1_tw/ch6.md",
"chars": 21671,
"preview": "---\nlinktitle: \"第六章：分割槽\"\nlinkTitle: \"6. 分割槽\"\nweight: 206\nbreadcrumbs: false\n---\n\n\n\n\n> 我們必須跳出電腦指令序列的窠臼。"
},
{
"path": "content/v1_tw/ch7.md",
"chars": 51556,
"preview": "---\ntitle: \"第七章：事務\"\nlinkTitle: \"7. 事務\"\nweight: 207\nbreadcrumbs: false\n---\n\n\n\n> 一些作者聲稱，支援通用的兩階段提交代價太大，會"
},
{
"path": "content/v1_tw/ch8.md",
"chars": 57740,
"preview": "---\ntitle: \"第八章：分散式系統的麻煩\"\nlinkTitle: \"8. 分散式系統的麻煩\"\nweight: 208\nmath: true\nbreadcrumbs: false\n---\n\n\n\n\n>"
},
{
"path": "content/v1_tw/ch9.md",
"chars": 79367,
"preview": "---\ntitle: \"第九章：一致性與共識\"\nlinkTitle: \"9. 一致性與共識\"\nweight: 209\nmath: true\nbreadcrumbs: false\n---\n\n\n\n\n> 好死還"
},
{
"path": "content/v1_tw/colophon.md",
"chars": 1499,
"preview": "---\ntitle: 後記\nweight: 600\nbreadcrumbs: false\n---\n\n## 關於作者\n\n**Martin Kleppmann** 是英國劍橋大學分散式系統的研究員。此前他曾在網際網路公司擔任過軟體工程師和企業家"
},
{
"path": "content/v1_tw/contrib.md",
"chars": 36217,
"preview": "---\ntitle: 貢獻者\nweight: 800\nbreadcrumbs: false\n---\n\n## 譯者\n\n[**馮若航**](https://vonng.com)，網名 [@Vonng](https://github.com/Vo"
},
{
"path": "content/v1_tw/glossary.md",
"chars": 7393,
"preview": "---\ntitle: 術語表\nweight: 500\nbreadcrumbs: false\n---\n\n> 請注意，本術語表中的定義簡短而簡單，旨在傳達核心思想，而非死扣完整細節。有關更多詳細資訊，請參閱正文中的參考資料。\n\n\n## **非同"
},
{
"path": "content/v1_tw/part-i.md",
"chars": 1156,
"preview": "---\ntitle: 第一部分：資料系統基礎\nweight: 100\nbreadcrumbs: false\n---\n\n本書前四章介紹了資料系統底層的基礎概念，無論是在單臺機器上執行的單點資料系統，還是分佈在多臺機器上的分散式資料系統都適用。"
},
{
"path": "content/v1_tw/part-ii.md",
"chars": 4368,
"preview": "---\ntitle: 第二部分：分散式資料\nweight: 200\nbreadcrumbs: false\n---\n\n> 一個成功的技術，現實的優先順序必須高於公關，你可以糊弄別人，但糊弄不了自然規律。\n>\n> —— 羅傑斯委員會報告（198"
},
{
"path": "content/v1_tw/part-iii.md",
"chars": 2048,
"preview": "---\ntitle: 第三部分：衍生資料\nweight: 300\nbreadcrumbs: false\n---\n\n\n在本書的 [第一部分](/v1_tw/part-i) 和 [第二部分](/v1_tw/part-ii) 中，我們自底向上地把"
},
{
"path": "content/v1_tw/preface.md",
"chars": 6742,
"preview": "---\ntitle: 序言\nweight: 50\nbreadcrumbs: false\n---\n\n\n如果近幾年從業於軟體工程，特別是伺服器端和後端系統開發，那麼你很有可能已經被大量關於資料儲存和處理的時髦詞彙轟炸過了： NoSQL！大資料！"
},
{
"path": "content/v1_tw/toc.md",
"chars": 2231,
"preview": "---\ntitle: \"目錄\"\nlinkTitle: \"目錄\"\nweight: 10\nbreadcrumbs: false\n---\n\n\n\n\n## [序言](/v1_tw/preface)\n\n## [第一部"
},
{
"path": "content/zh/_index.md",
"chars": 41437,
"preview": "---\ntitle: 设计数据密集型应用（第二版）\nlinkTitle: DDIA\ncascade:\n type: docs\nbreadcrumbs: false\n---\n\n\n**作者**： [Martin Kleppmann](http"
},
{
"path": "content/zh/ch1.md",
"chars": 41877,
"preview": "---\ntitle: \"1. 数据系统架构中的权衡\"\nweight: 101\nbreadcrumbs: false\n---\n\n<a id=\"ch_tradeoffs\"></a>\n\n> *没有完美的解决方案，只有权衡取舍。[…] 你能做的就是"
},
{
"path": "content/zh/ch10.md",
"chars": 64599,
"preview": "---\ntitle: \"10. 一致性与共识\"\nweight: 210\nbreadcrumbs: false\n---\n\n<a id=\"ch_consistency\"></a>\n\n\n\n> *一句古老的格言告"
},
{
"path": "content/zh/ch11.md",
"chars": 39963,
"preview": "---\ntitle: \"第十一章：批处理\"\nlinkTitle: \"11. 批处理\"\nweight: 311\nbreadcrumbs: false\n---\n\n<a id=\"ch_batch\"></a>\n\n"
},
{
"path": "content/zh/ch12.md",
"chars": 63642,
"preview": "---\ntitle: \"第十二章：流处理\"\nlinkTitle: \"12. 流处理\"\nweight: 312\nmath: true\nbreadcrumbs: false\n---\n\n<a id=\"ch_stream\"></a>\n\n![](/m"
},
{
"path": "content/zh/ch13.md",
"chars": 57984,
"preview": "---\ntitle: \"第十三章：流式系统的哲学\"\nlinkTitle: \"13. 流式系统的哲学\"\nweight: 313\nbreadcrumbs: false\n---\n\n<a id=\"ch_philosophy\"></a>\n<a id="
},
{
"path": "content/zh/ch14.md",
"chars": 25989,
"preview": "---\ntitle: \"14. 将事情做正确\"\nweight: 314\nbreadcrumbs: false\n---\n\n<a id=\"ch_right_thing\"></a>\n\n\n\n> *将世界的美好、丑"
},
{
"path": "content/zh/ch2.md",
"chars": 45780,
"preview": "---\ntitle: \"2. 定义非功能性需求\"\nweight: 102\nbreadcrumbs: false\n---\n\n<a id=\"ch_nonfunctional\"></a>\n\n\n\n> *互联网做得"
},
{
"path": "content/zh/ch3.md",
"chars": 62333,
"preview": "---\ntitle: \"3. 数据模型与查询语言\"\nweight: 103\nbreadcrumbs: false\n---\n\n<a id=\"ch_datamodels\"></a>\n\n\n\n> *语言的边界就是"
},
{
"path": "content/zh/ch4.md",
"chars": 62016,
"preview": "---\ntitle: \"4. 存储与检索\"\nweight: 104\nbreadcrumbs: false\n---\n\n<a id=\"ch_storage\"></a>\n\n\n\n> *生活的苦恼之一是，每个人对事"
},
{
"path": "content/zh/ch5.md",
"chars": 41852,
"preview": "---\ntitle: \"5. 编码与演化\"\nweight: 105\nmath: true\nbreadcrumbs: false\n---\n\n<a id=\"ch_encoding\"></a>\n\n\n\n> *万物"
},
{
"path": "content/zh/ch6.md",
"chars": 60963,
"preview": "---\ntitle: \"6. 复制\"\nweight: 206\nbreadcrumbs: false\n---\n\n<a id=\"ch_replication\"></a>\n\n\n\n> *可能出错的东西和“不可能”"
},
{
"path": "content/zh/ch7.md",
"chars": 28453,
"preview": "---\ntitle: \"7. 分片\"\nweight: 207\nbreadcrumbs: false\n---\n\n<a id=\"ch_sharding\"></a>\n\n\n\n> *显然，我们必须跳出顺序计算机指令"
},
{
"path": "content/zh/ch8.md",
"chars": 80559,
"preview": "---\ntitle: \"8. 事务\"\nweight: 208\nmath: true\nbreadcrumbs: false\n---\n\n<a id=\"ch_transactions\"></a>\n\n\n\n> *有"
},
{
"path": "content/zh/ch9.md",
"chars": 77677,
"preview": "---\ntitle: \"9. 分布式系统的麻烦\"\nweight: 209\nbreadcrumbs: false\n---\n\n<a id=\"ch_distributed\"></a>\n\n\n\n> *意外这东西挺有"
},
{
"path": "content/zh/colophon.md",
"chars": 1756,
"preview": "---\ntitle: 后记\nweight: 600\nbreadcrumbs: false\n---\n\n{{< callout type=\"warning\" >}}\n当前页面来自本书第一版，第二版尚不可用\n{{< /callout >}}\n\n#"
},
{
"path": "content/zh/contrib.md",
"chars": 36174,
"preview": "---\ntitle: 贡献者\nweight: 800\nbreadcrumbs: false\n---\n\n## 译者\n\n[**冯若航**](https://vonng.com)，网名 [@Vonng](https://github.com/Vo"
},
{
"path": "content/zh/glossary.md",
"chars": 7286,
"preview": "---\ntitle: 术语表\nweight: 500\nbreadcrumbs: false\n---\n\n> 请注意：本术语表的定义刻意保持简短，旨在传达核心概念，而非覆盖术语的全部细节。更多内容请参阅正文对应章节。\n\n### 异步（async"
},
{
"path": "content/zh/indexes.md",
"chars": 203724,
"preview": "---\ntitle: 索引\nweight: 550\nbreadcrumbs: false\n---\n\n### 符号\n\n- 3FS（分布式文件系统）, [分布式文件系统](/ch11#sec_batch_dfs)\n\n### A\n\n- 中止（事务"
},
{
"path": "content/zh/part-i.md",
"chars": 1648,
"preview": "---\ntitle: 第一部分：数据系统基础\nweight: 100\nbreadcrumbs: false\n---\n\n{{< callout type=\"warning\" >}}\n当前页面来自本书第一版，第二版尚不可用\n{{< /callo"
},
{
"path": "content/zh/part-ii.md",
"chars": 4608,
"preview": "---\ntitle: 第二部分：分布式数据\nweight: 200\nbreadcrumbs: false\n---\n\n{{< callout type=\"warning\" >}}\n当前页面来自本书第一版，第二版尚不可用\n{{< /callou"
},
{
"path": "content/zh/part-iii.md",
"chars": 2290,
"preview": "---\ntitle: 第三部分：派生数据\nweight: 300\nbreadcrumbs: false\n---\n\n{{< callout type=\"warning\" >}}\n当前页面来自本书第一版，第二版尚不可用\n{{< /callout"
},
{
"path": "content/zh/preface.md",
"chars": 7316,
"preview": "---\ntitle: 序言\nweight: 50\nbreadcrumbs: false\n---\n\n{{< callout type=\"warning\" >}}\n当前页面来自本书第一版，第二版尚不可用\n{{< /callout >}}\n\n如果"
},
{
"path": "content/zh/toc.md",
"chars": 3166,
"preview": "---\ntitle: \"目录\"\nlinkTitle: \"目录\"\nweight: 10\nbreadcrumbs: false\n---\n\n\n\n\n\n\n## [序言](/preface)\n- [本书的目标读者](/pr"
},
{
"path": "giscus.json",
"chars": 202,
"preview": "{\n \"origins\": [\n \"https://vonng.github.com/ddia\",\n \"https://ddia.vonng.com\",\n \"https://ddia.pgsty.com\",\n \"h"
},
{
"path": "go.mod",
"chars": 94,
"preview": "module github.com/Vonng/ddia\n\ngo 1.24.5\n\nrequire github.com/imfing/hextra v0.11.0 // indirect\n"
},
{
"path": "go.sum",
"chars": 336,
"preview": "github.com/imfing/hextra v0.9.7 h1:Zg5n24us36Bn/S/5mEUPkRW6uwE6vHHEqWSgN0bPXaM=\ngithub.com/imfing/hextra v0.9.7/go.mod h"
},
{
"path": "hugo.yaml",
"chars": 5327,
"preview": "baseURL: 'https://ddia.vonng.com/'\nlanguageCode: 'zh-CN'\ntitle: '设计数据密集型应用（第二版）'\n\nenableRobotsTXT: true\n# Parse Git comm"
},
{
"path": "i18n/en.yaml",
"chars": 210,
"preview": "copyright: '<a class=\"hx:flex hx:text-sm hx:items-center hx:gap-1 hx:text-current\" target=\"_blank\" rel=\"noopener norefer"
},
{
"path": "i18n/tw.yaml",
"chars": 433,
"preview": "backToTop: \"返回頂部\"\nchangeLanguage: \"切換語言\"\nchangeTheme: \"切換主題\"\ncopyright: '<a class=\"hx:flex hx:text-sm hx:items-center hx"
},
{
"path": "i18n/v2.yaml",
"chars": 461,
"preview": "backToTop: \"返回顶部\"\nchangeLanguage: \"切换语言\"\nchangeTheme: \"切换主题\"\ncopyright: '<a class=\"hx:flex hx:text-sm hx:items-center hx"
},
{
"path": "i18n/zh.yaml",
"chars": 461,
"preview": "backToTop: \"返回顶部\"\nchangeLanguage: \"切换语言\"\nchangeTheme: \"切换主题\"\ncopyright: '<a class=\"hx:flex hx:text-sm hx:items-center hx"
},
{
"path": "js/epub.css",
"chars": 3849,
"preview": "/* This defines styles and classes used in the book */\n@page {\n margin: 10px;\n}\nhtml, body, div, span, applet, object, "
},
{
"path": "layouts/shortcodes/figure.html",
"chars": 653,
"preview": "{{- $src := .Get \"src\" -}}\n{{- $link := .Get \"link\" -}}\n\n<figure\n {{ with .Get \"id\" }}id=\"{{ . }}\"{{ end }}\n{{ w"
},
{
"path": "metadata.yaml",
"chars": 148,
"preview": "---\ntitle: 设计数据密集型应用\nauthor: Martin Kleppmann\nrights: Creative Commons Non-Commercial Share Alike 3.0\nlanguage: zh\ncove"
}
]
About this extraction
This page contains the full source code of the Vonng/ddia GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 139 files (5.4 MB), approximately 1.4M tokens, and a symbol index with 17 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.