Showing preview only (323K chars total). Download the full file or copy to clipboard to get everything.
Repository: mozilla/firefox-browser-architecture
Branch: master
Commit: f5171ada734f
Files: 37
Total size: 308.2 KB
Directory structure:
gitextract_jxunuj8l/
├── .gitignore
├── CODE_OF_CONDUCT.md
├── Gemfile
├── ISSUE_TEMPLATE.md
├── LICENSE
├── README.md
├── _config.yml
├── _layouts/
│ ├── default.html
│ └── text.html
├── assets/
│ └── css/
│ └── style.scss
├── experiments/
│ ├── 2017-09-06-rust-on-ios.md
│ └── 2017-09-21-rust-on-android.md
├── newsletter/
│ └── _posts/
│ ├── 2017-07-27-browser-architecture-update.md
│ ├── 2017-08-24-browser-architecture-newsletter-2.md
│ ├── 2017-09-22-browser-architecture-newsletter-3.md
│ ├── 2017-10-19-browser-architecture-newsletter-4.md
│ └── 2017-11-29-browser-architecture-newsletter-5.md
├── newsletters.xml
└── text/
├── 0000-template.md
├── 0001-documenting-output.md
├── 0002-extracting-necko.md
├── 0003-problems-with-xul.md
├── 0004-xbl-web-components.md
├── 0005-problems-with-xbl.md
├── 0006-architecture-review-process.md
├── 0007-xbl-design-review-packet.md
├── 0008-sync-and-storage-review-packet.md
├── 0009-type-safety-systems.md
├── 0010-firefox-data-stores.md
├── 0011-fluent-in-prefs-design-review.md
├── 0012-jsonfile.md
├── 0012-process-isolation-in-firefox.md
├── 0013-ipc-security-models-and-status.md
├── 0014-xul-overlay-removal-review-packet.md
├── 0015-rkv.md
├── 0016-xulstore-rkv-poc.md
└── 0017-lmdb-vs-leveldb.md
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
# Created by https://www.gitignore.io/api/vim,node,grunt,emacs,gitbook,eclipse,jetbrains,visualstudiocode
### Eclipse ###
.metadata
bin/
tmp/
*.tmp
*.bak
*.swp
*~.nib
local.properties
.settings/
.loadpath
.recommenders
# External tool builders
.externalToolBuilders/
# Locally stored "Eclipse launch configurations"
*.launch
# PyDev specific (Python IDE for Eclipse)
*.pydevproject
# CDT-specific (C/C++ Development Tooling)
.cproject
# Java annotation processor (APT)
.factorypath
# PDT-specific (PHP Development Tools)
.buildpath
# sbteclipse plugin
.target
# Tern plugin
.tern-project
# TeXlipse plugin
.texlipse
# STS (Spring Tool Suite)
.springBeans
# Code Recommenders
.recommenders/
# Scala IDE specific (Scala & Java development for Eclipse)
.cache-main
.scala_dependencies
.worksheet
### Eclipse Patch ###
# Eclipse Core
.project
# JDT-specific (Eclipse Java Development Tools)
.classpath
### Emacs ###
# -*- mode: gitignore; -*-
*~
\#*\#
/.emacs.desktop
/.emacs.desktop.lock
*.elc
auto-save-list
tramp
.\#*
# Org-mode
.org-id-locations
*_archive
# flymake-mode
*_flymake.*
# eshell files
/eshell/history
/eshell/lastdir
# elpa packages
/elpa/
# reftex files
*.rel
# AUCTeX auto folder
/auto/
# cask packages
.cask/
dist/
# Flycheck
flycheck_*.el
# server auth directory
/server/
# projectiles files
.projectile
# directory configuration
.dir-locals.el
### GitBook ###
# Node rules:
## Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files)
.grunt
## Dependency directory
## Commenting this out is preferred by some people, see
## https://docs.npmjs.com/misc/faq#should-i-check-my-node_modules-folder-into-git
node_modules
# Book build output
_book
# eBook build output
*.epub
*.mobi
*.pdf
### grunt ###
# Grunt usually compiles files inside this directory
# Grunt usually preprocesses files such as coffeescript, compass... inside the .tmp directory
.tmp/
### JetBrains ###
# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio and Webstorm
# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839
# User-specific stuff:
.idea/**/workspace.xml
.idea/**/tasks.xml
.idea/dictionaries
# Sensitive or high-churn files:
.idea/**/dataSources/
.idea/**/dataSources.ids
.idea/**/dataSources.xml
.idea/**/dataSources.local.xml
.idea/**/sqlDataSources.xml
.idea/**/dynamic.xml
.idea/**/uiDesigner.xml
# Gradle:
.idea/**/gradle.xml
.idea/**/libraries
# CMake
cmake-build-debug/
# Mongo Explorer plugin:
.idea/**/mongoSettings.xml
## File-based project format:
*.iws
## Plugin-specific files:
# IntelliJ
/out/
# mpeltonen/sbt-idea plugin
.idea_modules/
# JIRA plugin
atlassian-ide-plugin.xml
# Cursive Clojure plugin
.idea/replstate.xml
# Crashlytics plugin (for Android Studio and IntelliJ)
com_crashlytics_export_strings.xml
crashlytics.properties
crashlytics-build.properties
fabric.properties
### JetBrains Patch ###
# Comment Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-215987721
# *.iml
# modules.xml
# .idea/misc.xml
# *.ipr
# Sonarlint plugin
.idea/sonarlint
### Node ###
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
# Runtime data
pids
*.pid
*.seed
*.pid.lock
# Directory for instrumented libs generated by jscoverage/JSCover
lib-cov
# Coverage directory used by tools like istanbul
coverage
# nyc test coverage
.nyc_output
# Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files)
# Bower dependency directory (https://bower.io/)
bower_components
# node-waf configuration
.lock-wscript
# Compiled binary addons (http://nodejs.org/api/addons.html)
build/Release
# Dependency directories
node_modules/
jspm_packages/
# Typescript v1 declaration files
typings/
# Optional npm cache directory
.npm
# Optional eslint cache
.eslintcache
# Optional REPL history
.node_repl_history
# Output of 'npm pack'
*.tgz
# Yarn Integrity file
.yarn-integrity
# dotenv environment variables file
.env
### Vim ###
# swap
[._]*.s[a-v][a-z]
[._]*.sw[a-p]
[._]s[a-v][a-z]
[._]sw[a-p]
# session
Session.vim
# temporary
.netrwhist
# auto-generated tag files
tags
### VisualStudioCode ###
.vscode/*
!.vscode/settings.json
!.vscode/tasks.json
!.vscode/launch.json
!.vscode/extensions.json
.history
# End of https://www.gitignore.io/api/vim,node,grunt,emacs,gitbook,eclipse,jetbrains,visualstudiocode
# JOE: Added manually
.idea/
# Jekyll: Added manually
.sass-cache
_site
Gemfile.lock
================================================
FILE: CODE_OF_CONDUCT.md
================================================
# Community Participation Guidelines
This repository is governed by Mozilla's code of conduct and etiquette guidelines.
For more details, please read the
[Mozilla Community Participation Guidelines](https://www.mozilla.org/about/governance/policies/participation/).
## How to Report
For more information on how to report violations of the Community Participation Guidelines, please read our '[How to Report](https://www.mozilla.org/about/governance/policies/participation/reporting/)' page.
<!-- There is copy of this text in README.md -->
================================================
FILE: Gemfile
================================================
source "https://rubygems.org"
gem 'github-pages', group: :jekyll_plugins
================================================
FILE: ISSUE_TEMPLATE.md
================================================
This repository is for documentation about the Firefox Architectural Change program. It's not a place to raise bugs about Firefox itself.
* If you have found a bug in Firefox please start here: https://bugzilla.mozilla.org/enter_bug.cgi
* If you want to give broader feedback: https://input.mozilla.org/
* If you have a website that doesn't work properly in Firefox: https://webcompat.com/
================================================
FILE: LICENSE
================================================
Mozilla Public License Version 2.0
==================================
1. Definitions
--------------
1.1. "Contributor"
means each individual or legal entity that creates, contributes to
the creation of, or owns Covered Software.
1.2. "Contributor Version"
means the combination of the Contributions of others (if any) used
by a Contributor and that particular Contributor's Contribution.
1.3. "Contribution"
means Covered Software of a particular Contributor.
1.4. "Covered Software"
means Source Code Form to which the initial Contributor has attached
the notice in Exhibit A, the Executable Form of such Source Code
Form, and Modifications of such Source Code Form, in each case
including portions thereof.
1.5. "Incompatible With Secondary Licenses"
means
(a) that the initial Contributor has attached the notice described
in Exhibit B to the Covered Software; or
(b) that the Covered Software was made available under the terms of
version 1.1 or earlier of the License, but not also under the
terms of a Secondary License.
1.6. "Executable Form"
means any form of the work other than Source Code Form.
1.7. "Larger Work"
means a work that combines Covered Software with other material, in
a separate file or files, that is not Covered Software.
1.8. "License"
means this document.
1.9. "Licensable"
means having the right to grant, to the maximum extent possible,
whether at the time of the initial grant or subsequently, any and
all of the rights conveyed by this License.
1.10. "Modifications"
means any of the following:
(a) any file in Source Code Form that results from an addition to,
deletion from, or modification of the contents of Covered
Software; or
(b) any new file in Source Code Form that contains any Covered
Software.
1.11. "Patent Claims" of a Contributor
means any patent claim(s), including without limitation, method,
process, and apparatus claims, in any patent Licensable by such
Contributor that would be infringed, but for the grant of the
License, by the making, using, selling, offering for sale, having
made, import, or transfer of either its Contributions or its
Contributor Version.
1.12. "Secondary License"
means either the GNU General Public License, Version 2.0, the GNU
Lesser General Public License, Version 2.1, the GNU Affero General
Public License, Version 3.0, or any later versions of those
licenses.
1.13. "Source Code Form"
means the form of the work preferred for making modifications.
1.14. "You" (or "Your")
means an individual or a legal entity exercising rights under this
License. For legal entities, "You" includes any entity that
controls, is controlled by, or is under common control with You. For
purposes of this definition, "control" means (a) the power, direct
or indirect, to cause the direction or management of such entity,
whether by contract or otherwise, or (b) ownership of more than
fifty percent (50%) of the outstanding shares or beneficial
ownership of such entity.
2. License Grants and Conditions
--------------------------------
2.1. Grants
Each Contributor hereby grants You a world-wide, royalty-free,
non-exclusive license:
(a) under intellectual property rights (other than patent or trademark)
Licensable by such Contributor to use, reproduce, make available,
modify, display, perform, distribute, and otherwise exploit its
Contributions, either on an unmodified basis, with Modifications, or
as part of a Larger Work; and
(b) under Patent Claims of such Contributor to make, use, sell, offer
for sale, have made, import, and otherwise transfer either its
Contributions or its Contributor Version.
2.2. Effective Date
The licenses granted in Section 2.1 with respect to any Contribution
become effective for each Contribution on the date the Contributor first
distributes such Contribution.
2.3. Limitations on Grant Scope
The licenses granted in this Section 2 are the only rights granted under
this License. No additional rights or licenses will be implied from the
distribution or licensing of Covered Software under this License.
Notwithstanding Section 2.1(b) above, no patent license is granted by a
Contributor:
(a) for any code that a Contributor has removed from Covered Software;
or
(b) for infringements caused by: (i) Your and any other third party's
modifications of Covered Software, or (ii) the combination of its
Contributions with other software (except as part of its Contributor
Version); or
(c) under Patent Claims infringed by Covered Software in the absence of
its Contributions.
This License does not grant any rights in the trademarks, service marks,
or logos of any Contributor (except as may be necessary to comply with
the notice requirements in Section 3.4).
2.4. Subsequent Licenses
No Contributor makes additional grants as a result of Your choice to
distribute the Covered Software under a subsequent version of this
License (see Section 10.2) or under the terms of a Secondary License (if
permitted under the terms of Section 3.3).
2.5. Representation
Each Contributor represents that the Contributor believes its
Contributions are its original creation(s) or it has sufficient rights
to grant the rights to its Contributions conveyed by this License.
2.6. Fair Use
This License is not intended to limit any rights You have under
applicable copyright doctrines of fair use, fair dealing, or other
equivalents.
2.7. Conditions
Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted
in Section 2.1.
3. Responsibilities
-------------------
3.1. Distribution of Source Form
All distribution of Covered Software in Source Code Form, including any
Modifications that You create or to which You contribute, must be under
the terms of this License. You must inform recipients that the Source
Code Form of the Covered Software is governed by the terms of this
License, and how they can obtain a copy of this License. You may not
attempt to alter or restrict the recipients' rights in the Source Code
Form.
3.2. Distribution of Executable Form
If You distribute Covered Software in Executable Form then:
(a) such Covered Software must also be made available in Source Code
Form, as described in Section 3.1, and You must inform recipients of
the Executable Form how they can obtain a copy of such Source Code
Form by reasonable means in a timely manner, at a charge no more
than the cost of distribution to the recipient; and
(b) You may distribute such Executable Form under the terms of this
License, or sublicense it under different terms, provided that the
license for the Executable Form does not attempt to limit or alter
the recipients' rights in the Source Code Form under this License.
3.3. Distribution of a Larger Work
You may create and distribute a Larger Work under terms of Your choice,
provided that You also comply with the requirements of this License for
the Covered Software. If the Larger Work is a combination of Covered
Software with a work governed by one or more Secondary Licenses, and the
Covered Software is not Incompatible With Secondary Licenses, this
License permits You to additionally distribute such Covered Software
under the terms of such Secondary License(s), so that the recipient of
the Larger Work may, at their option, further distribute the Covered
Software under the terms of either this License or such Secondary
License(s).
3.4. Notices
You may not remove or alter the substance of any license notices
(including copyright notices, patent notices, disclaimers of warranty,
or limitations of liability) contained within the Source Code Form of
the Covered Software, except that You may alter any license notices to
the extent required to remedy known factual inaccuracies.
3.5. Application of Additional Terms
You may choose to offer, and to charge a fee for, warranty, support,
indemnity or liability obligations to one or more recipients of Covered
Software. However, You may do so only on Your own behalf, and not on
behalf of any Contributor. You must make it absolutely clear that any
such warranty, support, indemnity, or liability obligation is offered by
You alone, and You hereby agree to indemnify every Contributor for any
liability incurred by such Contributor as a result of warranty, support,
indemnity or liability terms You offer. You may include additional
disclaimers of warranty and limitations of liability specific to any
jurisdiction.
4. Inability to Comply Due to Statute or Regulation
---------------------------------------------------
If it is impossible for You to comply with any of the terms of this
License with respect to some or all of the Covered Software due to
statute, judicial order, or regulation then You must: (a) comply with
the terms of this License to the maximum extent possible; and (b)
describe the limitations and the code they affect. Such description must
be placed in a text file included with all distributions of the Covered
Software under this License. Except to the extent prohibited by statute
or regulation, such description must be sufficiently detailed for a
recipient of ordinary skill to be able to understand it.
5. Termination
--------------
5.1. The rights granted under this License will terminate automatically
if You fail to comply with any of its terms. However, if You become
compliant, then the rights granted under this License from a particular
Contributor are reinstated (a) provisionally, unless and until such
Contributor explicitly and finally terminates Your grants, and (b) on an
ongoing basis, if such Contributor fails to notify You of the
non-compliance by some reasonable means prior to 60 days after You have
come back into compliance. Moreover, Your grants from a particular
Contributor are reinstated on an ongoing basis if such Contributor
notifies You of the non-compliance by some reasonable means, this is the
first time You have received notice of non-compliance with this License
from such Contributor, and You become compliant prior to 30 days after
Your receipt of the notice.
5.2. If You initiate litigation against any entity by asserting a patent
infringement claim (excluding declaratory judgment actions,
counter-claims, and cross-claims) alleging that a Contributor Version
directly or indirectly infringes any patent, then the rights granted to
You by any and all Contributors for the Covered Software under Section
2.1 of this License shall terminate.
5.3. In the event of termination under Sections 5.1 or 5.2 above, all
end user license agreements (excluding distributors and resellers) which
have been validly granted by You or Your distributors under this License
prior to termination shall survive termination.
************************************************************************
* *
* 6. Disclaimer of Warranty *
* ------------------------- *
* *
* Covered Software is provided under this License on an "as is" *
* basis, without warranty of any kind, either expressed, implied, or *
* statutory, including, without limitation, warranties that the *
* Covered Software is free of defects, merchantable, fit for a *
* particular purpose or non-infringing. The entire risk as to the *
* quality and performance of the Covered Software is with You. *
* Should any Covered Software prove defective in any respect, You *
* (not any Contributor) assume the cost of any necessary servicing, *
* repair, or correction. This disclaimer of warranty constitutes an *
* essential part of this License. No use of any Covered Software is *
* authorized under this License except under this disclaimer. *
* *
************************************************************************
************************************************************************
* *
* 7. Limitation of Liability *
* -------------------------- *
* *
* Under no circumstances and under no legal theory, whether tort *
* (including negligence), contract, or otherwise, shall any *
* Contributor, or anyone who distributes Covered Software as *
* permitted above, be liable to You for any direct, indirect, *
* special, incidental, or consequential damages of any character *
* including, without limitation, damages for lost profits, loss of *
* goodwill, work stoppage, computer failure or malfunction, or any *
* and all other commercial damages or losses, even if such party *
* shall have been informed of the possibility of such damages. This *
* limitation of liability shall not apply to liability for death or *
* personal injury resulting from such party's negligence to the *
* extent applicable law prohibits such limitation. Some *
* jurisdictions do not allow the exclusion or limitation of *
* incidental or consequential damages, so this exclusion and *
* limitation may not apply to You. *
* *
************************************************************************
8. Litigation
-------------
Any litigation relating to this License may be brought only in the
courts of a jurisdiction where the defendant maintains its principal
place of business and such litigation shall be governed by laws of that
jurisdiction, without reference to its conflict-of-law provisions.
Nothing in this Section shall prevent a party's ability to bring
cross-claims or counter-claims.
9. Miscellaneous
----------------
This License represents the complete agreement concerning the subject
matter hereof. If any provision of this License is held to be
unenforceable, such provision shall be reformed only to the extent
necessary to make it enforceable. Any law or regulation which provides
that the language of a contract shall be construed against the drafter
shall not be used to construe this License against a Contributor.
10. Versions of the License
---------------------------
10.1. New Versions
Mozilla Foundation is the license steward. Except as provided in Section
10.3, no one other than the license steward has the right to modify or
publish new versions of this License. Each version will be given a
distinguishing version number.
10.2. Effect of New Versions
You may distribute the Covered Software under the terms of the version
of the License under which You originally received the Covered Software,
or under the terms of any subsequent version published by the license
steward.
10.3. Modified Versions
If you create software not governed by this License, and you want to
create a new license for such software, you may create and use a
modified version of this License if you rename the license and remove
any references to the name of the license steward (except to note that
such modified license differs from this License).
10.4. Distributing Source Code Form that is Incompatible With Secondary
Licenses
If You choose to distribute Source Code Form that is Incompatible With
Secondary Licenses under the terms of this version of the License, the
notice described in Exhibit B of this License must be attached.
Exhibit A - Source Code Form License Notice
-------------------------------------------
This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, You can obtain one at http://mozilla.org/MPL/2.0/.
If it is not possible or desirable to put the notice in a particular
file, then You may include the notice in a location (such as a LICENSE
file in a relevant directory) where a recipient would be likely to look
for such a notice.
You may add additional accurate notices of copyright ownership.
Exhibit B - "Incompatible With Secondary Licenses" Notice
---------------------------------------------------------
This Source Code Form is "Incompatible With Secondary Licenses", as
defined by the Mozilla Public License, v. 2.0.
================================================
FILE: README.md
================================================
# Firefox Browser Architecture
## Mission
Change Mozilla. Investigate big technical challenges and produce engineering programs to address them.
## Our Conclusions
This is a list of our findings that we're reasonably happy with so far.
* [Documenting our output](text/0001-documenting-output.md) looks at how we’re going to communicate with the rest of Mozilla.
* [Extracting Necko](text/0002-extracting-necko.md) considers whether it's feasible or worthwhile to extract Necko — Gecko's C++ networking library — for use as a standalone component.
* [Problems with XUL](text/0003-problems-with-xul.md) aims to list the different kinds of problems that exist with XUL.
* [XBL and Web Components](text/0004-xbl-web-components.md) compares some old Mozilla technology (XBL) with modern Web Components.
* [Problems with XBL](text/0005-problems-with-xbl.md) aims to list the different kinds of problems that exist with XBL.
* [Architecture Reviews](text/0006-architecture-review-process.md) are healthy and we proposed a process for healthy reviews.
* [XBL Design Review packet](text/0007-xbl-design-review-packet.md) is the packet that we prepared for the architectural design review for XBL removal.
* [Roadmap Review: Sync and Storage](text/0008-sync-and-storage-review-packet.md) establishes that storage and syncing of user data is a pillar of the Firefox ecosystem, warranting holistic and long-term attention, and outlines where we’d like to end up and some ideas for how to get there.
* [JavaScript Type Safety Systems](text/0009-type-safety-systems.md) are some conclusions of an investigation into the use of JavaScript type safety systems.
* [Firefox Data Stores Documentation](text/0010-firefox-data-stores.md) documents the existing data stores across all current Firefox platforms.
* [Fluent in Prefs Design Review](text/0011-fluent-in-prefs-design-review.md) describes the lightweight design review for Fluent in Prefs.
* [A brief analysis of JSON file-backed storage](text/0012-jsonfile.md) outlines some of the pros and cons of using a flat file (particularly via `JSONFile.jsm`) to store data in Firefox.
* [Process Isolation in Firefox](text/0012-process-isolation-in-firefox.md) is a WIP evaluation of how far we can push process isolation to improve security and stability.
* [IPC Security Models and Status](text/0013-ipc-security-models-and-status.md) is an audit of our current IPC status.
* [XUL Overlay Removal Review Packet](text/0014-xul-overlay-removal-review-packet.md) is the packet that we prepared for the architectural design review for XUL Overlay removal.
* [Design Review: Key-Value Storage](text/0015-rkv.md) proposes the introduction of a fast, cross-platform key-value store for Mozilla products.
* [XULStore Using rkv – Proof of Concept](text/0016-xulstore-rkv-poc.md) describes a proof-of-concept implementation of XULStore that uses [rkv](https://github.com/mozilla/rkv).
* [LMDB vs. LevelDB](text/0017-lmdb-vs-leveldb.md) compares the [Lightning Memory-mapped Database](https://symas.com/lmdb/) (LMDB) key-value storage engine to the [LevelDB](https://github.com/google/leveldb) key-value storage engine.
## Posts
We typically send our newsletters to [firefox-dev](https://www.mozilla.org/en-US/about/forums/#firefox-dev).
* [Browser Architecture Update](newsletter/_posts/2017-07-27-browser-architecture-update.md). See also [mailing-list-post](https://groups.google.com/forum/#!topic/firefox-dev/ueRILL2ppac).
* [Browser Architecture Newsletter #2](newsletter/_posts/2017-08-24-browser-architecture-newsletter-2.md). See also [mailing-list-post](https://groups.google.com/forum/#!topic/firefox-dev/Rc2w2a9e8HQ).
* [Browser Architecture Newsletter #3](newsletter/_posts/2017-09-22-browser-architecture-newsletter-3.md). See also [mailing-list-post](https://groups.google.com/forum/#!topic/firefox-dev/p9rTlfFUXlQ).
* [Browser Architecture Newsletter #4](newsletter/_posts/2017-10-19-browser-architecture-newsletter-4.md). See also [mailing-list-post](https://groups.google.com/forum/#!topic/firefox-dev/CLFtj8qUSv8).
* [Browser Architecture Newsletter #5](newsletter/_posts/2017-11-29-browser-architecture-newsletter-5.md). See also [mailing-list-post](https://groups.google.com/forum/#!topic/firefox-dev/XKp3EthdJ60).
## Explorations and Experiments
To support our conclusions we occasionally perform explorations and experiments. The first exploration is designed to support the notion that we can create a sync and storage layer in Rust that we can deploy to Desktop, Android and iOS.
* [Deploying a Rust library on iOS](experiments/2017-09-06-rust-on-ios.md). A short tutorial describing how to build and deploy a Rust library for use inside an iOS app.
* [Deploying a Rust library on Android](experiments/2017-09-21-rust-on-android.md). A short tutorial describing how to build and deploy a Rust library for use inside an Android app.
<!-- This is a copy of this text in CODE_OF_CONDUCT.md -->
# Community Participation Guidelines
This repository is governed by Mozilla's code of conduct and etiquette guidelines.
For more details, please read the
[Mozilla Community Participation Guidelines](https://www.mozilla.org/about/governance/policies/participation/).
## How to Report
For more information on how to report violations of the Community Participation Guidelines, please read our '[How to Report](https://www.mozilla.org/about/governance/policies/participation/reporting/)' page.
================================================
FILE: _config.yml
================================================
theme: jekyll-theme-minimal
baseurl: /firefox-browser-architecture
================================================
FILE: _layouts/default.html
================================================
<!doctype html>
<html lang="{{ site.lang | default: "en-US" }}">
<head>
<meta charset="utf-8">
<link rel="stylesheet" href="{{ '/assets/css/style.css?v=' | append: site.github.build_revision | relative_url }}">
<meta name="viewport" content="width=device-width">
<title>{{ page.title }}</title>
</head>
<body>
<div class="wrapper">
{{ content }}
</div>
<script src="{{ '/assets/js/scale.fix.js' | relative_url }}"></script>
</body>
</html>
================================================
FILE: _layouts/text.html
================================================
<!doctype html>
<html lang="{{ site.lang | default: "en-US" }}">
<head>
<meta charset="utf-8">
<link rel="stylesheet" href="{{ '/assets/css/style.css?v=' | append: site.github.build_revision | relative_url }}">
<meta name="viewport" content="width=device-width">
<title>{{ page.title }}</title>
</head>
<body>
<p>[<a href="{{ site.baseurl }}">Back to top</a>]</p>
<div class="wrapper">
{{ content }}
</div>
<p>[<a href="{{ site.baseurl }}">Back to top</a>]</p>
<script src="{{ '/assets/js/scale.fix.js' | relative_url }}"></script>
</body>
</html>
================================================
FILE: assets/css/style.scss
================================================
---
---
@import "{{ site.theme }}";
@import url('https://fonts.googleapis.com/css?family=Zilla+Slab|Zilla+Slab+Highlight');
// Convert pixel units to rems, assuming a 16px base size
// Usage: remify(24px)
@function remify($pixels) {
$rems: $pixels / 16px;
@return #{$rems}rem;
}
@mixin font-size($size) {
// When unit is pixels, pass it through and convert to rem
@if (unit($size) == 'px') {
font-size: $size;
font-size: remify($size);
}
// When unit is not pixels, show an error
@else {
@error 'This mixin only accepts sizes in pixels. You declared `font-size(#{$size})`.';
}
}
@mixin font-size-huge { // For especially huge titles
@include font-size(50px);
@media screen and (min-width: 760px) {
@include font-size(90px);
}
}
@mixin font-size-level1 {
@include font-size(28px);
@media screen and (min-width: 760px) {
@include font-size(50px);
}
}
@mixin font-size-level2 {
@include font-size(21px);
@media screen and (min-width: 760px) {
@include font-size(38px);
}
}
@mixin font-size-level3 {
@include font-size(18px);
@media screen and (min-width: 760px) {
@include font-size(28px);
}
}
@mixin font-size-level4 {
@include font-size(16px);
@media screen and (min-width: 760px) {
@include font-size(21px);
}
}
body {
background: #fff;
font-size: 20px;
font-family: "Zilla Slab", "Open Sans", X-LocaleSpecific, sans-serif;
line-height: 1.5;
}
h1, h2, h3, h4, h5, h6, legend {
color: #000;
font-weight: bold;
line-height: 1.1;
margin: 0 0 .25em;
}
h1 {
@include font-size-level1;
}
h2 {
@include font-size-level2;
}
h3 {
@include font-size-level3;
}
h4 {
@include font-size-level4;
}
h5, h6 {
@include font-size(16px);
}
p,
ul,
ol,
dl,
table {
margin-bottom: 1.25em;
}
a {
color: #006688;
}
a:hover, a:focus, a:active, a:visited:hover, a:visited:focus, a:visited:active {
color: #0088aa;
font-weight: normal;
}
a:visited {
color: #004455;
}
.wrapper > section {
float: none;
width: 100%;
}
================================================
FILE: experiments/2017-09-06-rust-on-ios.md
================================================
---
title: Building and Deploying a Rust library on iOS
layout: text
---
# Building and Deploying a Rust library on iOS
First, we have to [install Xcode](https://itunes.apple.com/us/app/xcode/id497799835?ls=1&mt=12) and then set up Xcode build tools. If you already have the build tools installed and they are up to date, you can skip this step.
```
xcode-select --install
```
Next, we need to ensure that Rust is installed and that we can cross compile to the iOS architectures. For this we will be using [rustup](https://www.rustup.rs/). If you already have rustup installed, you can skip this step. Rustup installs Rust from the official release channels and enables you to easily switch between different release versions. It will be useful to you for all your future Rust development, not just here.
```
curl https://sh.rustup.rs -sSf | sh
```
Add the iOS architectures to rustup so we can use them during cross compilation.
```
rustup target add aarch64-apple-ios armv7-apple-ios armv7s-apple-ios x86_64-apple-ios i386-apple-ios
```
When you installed Rust, it also installed cargo, which is a package manager similar to pip, gems etc. Now we will use cargo to install `cargo-lipo`. This is a cargo subcommand which automatically creates a universal library for use with iOS. Without this crate, cross compiling Rust to work on iOS is infinitely harder.
```
cargo install cargo-lipo
```
Now we're all setup and we're ready to start. Let's create the project directory.
```
mkdir greetings
cd greetings
cargo new cargo
mkdir ios
```
`cargo new cargo` sets up a brand new Rust project with its default files and directories in a directory called `cargo`. In this directory is a file called `Cargo.toml`, which is the package manager descriptor file, and there is be a subdirectory, `src`, which contains a file called `lib.rs`. This will contain the Rust code that we will be executing.
Our Rust project here is a super simple Hello World library. It contains a function `rust_greeting` that takes a string argument and return a string greeting that argument. Therefore, if the argument is "world", the returned string is "Hello world".
Open `cargo/src/lib.rs` and enter the following code.
```rust
use std::os::raw::{c_char};
use std::ffi::{CString, CStr};
#[no_mangle]
pub extern fn rust_greeting(to: *const c_char) -> *mut c_char {
let c_str = unsafe { CStr::from_ptr(to) };
let recipient = match c_str.to_str() {
Err(_) => "there",
Ok(string) => string,
};
CString::new("Hello ".to_owned() + recipient).unwrap().into_raw()
}
#[no_mangle]
pub extern fn rust_greeting_free(s: *mut c_char) {
unsafe {
if s.is_null() { return }
CString::from_raw(s)
};
}
```
Let's take a look at what is going on here.
As we will be calling this library from non-Rust code, we will actually be calling it through a C bridge.`#[no_mangle]` tells the compiler not to mangle the function name as it usually does by default, ensuring our function name is exported as if it had been written in C.
`extern` tells the Rust compiler that this function will be called from outside of Rust and to therefore ensure that it is compiled using C calling conventions.
The string that `rust_greeting` accepts is a pointer to a C char array. We have to then convert the string from a C string to a Rust `str`. First we create a `CStr` object from the pointer. We then convert it to a `str` and check the result. If an error has occurred, then no arg was provided and we substitute `there`, otherwise we use the value of the provided string. We then append the provided string on the end of our greeting string to create our return string. The return string is then converted into a `CString` and passed back into C code.
Using `CString` and returning the raw representation keeps the string in memory and prevents it from being released at the end of the function. If the memory were to be released, the pointer provided back to the caller would now be pointing to empty memory - or possibly something else entirely. But, by ensuring that the string sticks around after the function has finished executing, we have memory allocated and no longer any handle to it. That is a recipe for a memory leak, so we have to provide a second function `rust_greeting_free` that takes a pointer to a C string and frees that memory. We will have to remember to call `rust_greeting_free` from our iOS code to ensure we don't run into problems.
We also need to create our C bridge. Create a new file in `cargo/src` called `greetings.h`. Inside that file, let's define what our C interface will look like. We need to ensure that every rust function that we want to call from iOS is defined here.
```C
#include <stdint.h>
const char* rust_greeting(const char* to);
void rust_greeting_free(char *);
```
Let's build our code to make sure it works. In order to do this we have to complete our `Cargo.toml` file. This will tell cargo to create a static library and C dynamic library for our code.
```toml
[package]
name = "greetings"
version = "0.1.1"
authors = ["fluffyemily <fluffyemily@mozilla.com>"]
description = "Example static library project built for iOS"
publish = false
[lib]
name = "greetings"
crate-type = ["staticlib", "cdylib"]
```
We need to build our library against the iOS architectures using `cargo-lipo`. The built artifacts of will be placed in `cargo/target/`. The universal iOS library that we are interested in can be found in `cargo/target/universal/release/libgreetings.a`.
```
cd cargo
cargo lipo --release
```
And that's it for our Rust library. Let's get it linked with an iOS project.
Open Xcode and create a new project. Go to `File\New\Project…` and select the `iOS\Application\Single View Application` template. This template is as close to a default iOS app as it gets. Click `Next`.
Call the project `Greetings`, make it a Swift project. Click `Next` to choose the location. We are using the ios directory that we created earlier. If you choose another location you will have to amend some of the paths that we set later. Click `Create`.
Select the Greetings project from the project navigator, and then ensure the Greetings target is selected. Open the `General` tab. Scroll down to the `Linked Frameworks and Libraries` section.
Import your `libgreetings.a` library by either dragging it in from finder, or clicking the + at the bottom of the list, clicking 'Add other...' and navigating to `cargo/target/universal/release/`. Select `libgreetings.a` and then click `Open`.
Link `libresolv.tbd`. Click the + at the bottom of the Linked Frameworks list and type libresolv into the search box. Select `libresolv.tbd` and then "Add".
iOS will need a bridging header to access the C header we created. First, let's import `greetings.h` into our Xcode project so we can link to it. Go to `File\Add files to "Greetings"...` Navigate to `greetings.h` and select `Add`.
To create our bridging header, go to `File\New\File...`. Select `iOS\Source\Header File` from the provided options and select `Next`. Name the file `Greetings-Bridging-Header.h` and select `Create`.
Open the bridging header and amend it to look like the following:
```C
#ifndef Greetings_Bridging_Header_h
#define Greetings_Bridging_Header_h
#import "greetings.h"
#endif
```
We need to tell Xcode about the bridging header. Select the Greetings project from the project navigator, and then ensure the Greetings target is selected and open the Build Settings tab. Set the `Objective-C Bridging Header` option value to `$(PROJECT_DIR)/Greetings/Greetings-Bridging-Header.h`
We also need to tell Xcode where to look for our libraries for linking. In the same Build Settings pane, amend the `Library Search Paths` option value to `$(PROJECT_DIR)/../../cargo/target/universal/release`
Build your xcode project and everything should compile.
So, now we have imported our Rust library into our iOS project and successfully linked to it. But we still need to call it. go to `File\New\File...`. Select `iOS\Source\Swift File` from the provided options and select `Next`. Name name it `RustGreetings`.
Add the following code:
```swift
class RustGreetings {
func sayHello(to: String) -> String {
let result = rust_greeting(to)
let swift_result = String(cString: result!)
rust_greeting_free(UnsafeMutablePointer(mutating: result))
return swift_result
}
}
```
This creates a new class that we will use as an interface to call into our Rust library. This will allow us to abstract the nuance away from the main code of the app, including conversion from a C String to a Swift String and ensures that we won't forget to call our free function and inadvertently introduce a memory leak!
Open `ViewController.swift`. Inside the `viewDidLoad` function add the following code:
```swift
let rustGreetings = RustGreetings()
print("\(rustGreetings.sayHello(to: "world"))")
```
Now build your project and run. The simulator will open and start running your app. When the view loads, "Hello world" will be output in the console window inside Xcode.
You can find the code for this on [Github](https://github.com/fluffyemily/cross-platform-rust)
================================================
FILE: experiments/2017-09-21-rust-on-android.md
================================================
---
title: Building and Deploying a Rust library on Android
layout: text
---
# Building and Deploying a Rust library on Android
Following on from the last post on getting a Rust library [building on iOS](https://mozilla.github.io/firefox-browser-architecture/experiments/2017-09-06-rust-on-ios.html), we're now going to deploy the same library on Android.
In order to do Android development, we'll need to set up our Android environment. First we need to install [Android Studio](https://developer.android.com/studio/index.html). Once Android Studio is installed, we'll need to install the NDK (Native Development Kit).
Open Android Studio. From the toolbar, go to `Android Studio > Preferences > Appearance & Behaviour > Android SDK > SDK Tools`. Check the following options for installation and click `OK`.
* Android SDK Tools
* NDK
* CMake
* LLDB
Once the NDK and associated tools have been installed, we need to set a few environment variables, first for the SDK path and the second for the NDK path. Set the following envvars:
```
export ANDROID_HOME=/Users/$USER/Library/Android/sdk
export NDK_HOME=$ANDROID_HOME/ndk-bundle
```
If you do not already have Rust installed, we need to do this now. For this we will be using [rustup](https://www.rustup.rs/). `rustup` installs Rust from the official release channels and enables you to easily switch between different release versions. It will be useful to you for all your future Rust development, not just here. `rustup` can also be used in conjunction with `HomeBrew`.
```
curl https://sh.rustup.rs -sSf | sh
```
The next step is to create standalone versions of the NDK for us to compile against. We need to do this for each of the architectures we want to compile against. We will be using the `make_standalone_toolchain.py` script inside the main Android NDK in order to do this. First create a directory for our project.
```
mkdir greetings
cd greetings
```
Now let's create our standalone NDKs. There is no need to be inside the `NDK` directory once you have created it to do this.
```
mkdir NDK
${NDK_HOME}/build/tools/make_standalone_toolchain.py --api 26 --arch arm64 --install-dir NDK/arm64
${NDK_HOME}/build/tools/make_standalone_toolchain.py --api 26 --arch arm --install-dir NDK/arm
${NDK_HOME}/build/tools/make_standalone_toolchain.py --api 26 --arch x86 --install-dir NDK/x86
```
Create a new file, `cargo-config.toml`. This file will tell cargo where to look for the NDKs during cross compilation. Add the following content to the file, remembering to replace instances of `<project path>` with the path to your project directory.
```
[target.aarch64-linux-android]
ar = "<project path>/greetings/NDK/arm64/bin/aarch64-linux-android-ar"
linker = "<project path>/greetings/NDK/arm64/bin/aarch64-linux-android-clang"
[target.armv7-linux-androideabi]
ar = "<project path>/greetings/NDK/arm/bin/arm-linux-androideabi-ar"
linker = "<project path>/greetings/NDK/arm/bin/arm-linux-androideabi-clang"
[target.i686-linux-android]
ar = "<project path>/greetings/NDK/x86/bin/i686-linux-android-ar"
linker = "<project path>/greetings/NDK/x86/bin/i686-linux-android-clang"
```
In order for `cargo` to see our new SDK's we need to copy this config file to our `.cargo` directory like this:
```
cp cargo-config.toml ~/.cargo/config
```
Let's go ahead and add our newly created Android architectures to `rustup` so we can use them during cross compilation:
```
rustup target add aarch64-linux-android armv7-linux-androideabi i686-linux-android
```
Now we're all set up and we're ready to start. Let's create the `lib` directory. If you've already created a Rust project from following the iOS post, you don't need to do it again.
```
cargo new cargo
mkdir android
```
`cargo new cargo` sets up a brand new Rust project with its default files and directories in a directory called `rust`. The name of the directory is not important. In this directory is a file called `Cargo.toml`, which is the package manager descriptor file, and there is be a subdirectory, `src`, which contains a file called `lib.rs`. This will contain the Rust code that we will be executing.
Our Rust project here is a super simple Hello World library. It contains a function `rust_greeting` that takes a string argument and return a greeting including that argument. Therefore, if the argument is "world", the returned string is "Hello world".
Open `cargo/src/lib.rs` and enter the following code.
```rust
use std::os::raw::{c_char};
use std::ffi::{CString, CStr};
#[no_mangle]
pub extern fn rust_greeting(to: *const c_char) -> *mut c_char {
let c_str = unsafe { CStr::from_ptr(to) };
let recipient = match c_str.to_str() {
Err(_) => "there",
Ok(string) => string,
};
CString::new("Hello ".to_owned() + recipient).unwrap().into_raw()
}
```
Let's take a look at what is going on here.
As we will be calling this library from non-Rust code, we will actually be calling it through a C bridge. `#[no_mangle]` tells the compiler not to mangle the function name as it usually does by default, ensuring our function name is exported as if it had been written in C.
`extern` tells the Rust compiler that this function will be called from outside of Rust and to therefore ensure that it is compiled using C calling conventions.
The string that `rust_greeting` accepts is a pointer to a C char array. We have to then convert the string from a C string to a Rust `str`. First we create a `CStr` object from the pointer. We then convert it to a `str` and check the result. If an error has occurred, then no arg was provided and we substitute `there`, otherwise we use the value of the provided string. We then append the provided string on the end of our greeting string to create our return string. The return string is then converted into a `CString` and passed back into C code.
Now let's create our Android project.
Open Android Studio and select `Start a New Android Project` from the options.
On the next screen, type a project name of `Greetings` into the `Application name` field, choose your `Company domain` and select the `android` directory we created earlier as the `Project location`. This will create your Android project inside `greetings/android/Greetings`. Click `Next`.
On the next screen, make sure the `Phone and Tablet` option is selected. Click `Next`.
Now we will be asked to choose a starting activity. Select the `Empty Activity` option and click `Next`.
Name your Activity and layout on the following screen, calling the activity `GreetingsActivity` and the layout `activity_greetings`. Click `Finish`.
As with iOS, we're going to create a wrapper class to wrap the C API and JNI bindings. In the project explorer on the left hand side of the studio window, ensure that `app > java > <domain>.greetings` is highlighted then go to `File > New > Java Class`. Name your class `RustGreetings` and click `OK`.
In your new class file, add the following code. Here we are defining the native interface to our Rust library and calling it `greeting`, with the same signature. The `sayHello` method simply makes a call to that native function.
```java
public class RustGreetings {
private static native String greeting(final String pattern);
public String sayHello(String to) {
return greeting(to);
}
}
```
Instead of creating C header that will be used as a bridge as we did when we wanted to integrate with Swift, for Android we want to expose our functions through JNI. The way that JNI constructs the name of the function that it will call is `Java_<domain>_<class>_<methodname>`. In the case of the method, `greeting` that we have declared here, the function in our Rust library that JNI will attempt to call will be `Java_com_mozilla_greetings_RustGreetings_greeting`. This is the reason why we created our Android project and Java wrapper class before adding any JNI code to the Rust library. We needed to know what the domain, class and function name were before we could construct the right JNI function name in Rust. Let's head back over to our Rust project and create the partner code.
Open `cargo/src/lib.rs`. At the bottom of the file add the following code:
```rust
/// Expose the JNI interface for android below
#[cfg(target_os="android")]
#[allow(non_snake_case)]
pub mod android {
extern crate jni;
use super::*;
use self::jni::JNIEnv;
use self::jni::objects::{JClass, JString};
use self::jni::sys::{jstring};
#[no_mangle]
pub unsafe extern fn Java_com_mozilla_greetings_RustGreetings_greeting(env: JNIEnv, _: JClass, java_pattern: JString) -> jstring {
// Our Java companion code might pass-in "world" as a string, hence the name.
let world = rust_greeting(env.get_string(java_pattern).expect("invalid pattern string").as_ptr());
// Retake pointer so that we can use it below and allow memory to be freed when it goes out of scope.
let world_ptr = CString::from_raw(world);
let output = env.new_string(world_ptr.to_str().unwrap()).expect("Couldn't create java string!");
output.into_inner()
}
}
```
The first line here `#[cfg(target_os="android")]` is telling the compiler to target Android when compiling this module. `#[cfg]` is a special attribute that allows you to compile code based on a flag passed to the compiler.
The second line, `#[allow(non_snake_case)]`, tells the compiler not to warn if we are not using `snake_case` for a variable or function name. The Rust compiler is very strict — this is one of the things that makes Rust great — and it enforces the use of `snake_case` throughout. However, we defined our class name and native method in our Android project using Java coding conventions which is `camelCase` and `UpperCamelCase` and we don't want to change this or our Java code will look wrong. Given the way that JNI constructs native function names, we need to tell the Rust compiler to go easy on us in this instance. This flag will apply to all functions and variables created inside this module that we are creating, called `android`.
After declaring that we need the `jni` crate, and importing some useful objects from it, we can declare our function. This function needs to be marked `unsafe` because we will be dealing with pointers from a language that allows null pointers, but our code doesn't check for `NULL`. This situation would never happen in Rust only code as the Rust compiler enforces memory safety. By marking the function as not memory safe, we are alerting other Rust functions that it may not be able to deal with a null pointer. `extern` defines the function as one that will be exposed to other languages.
As arguments, along with the `JString` that our Java function declaration said that we will be providing, we also need to take an instance of the `JNIEnv` and a class reference (which is unused in this example). The `JNIEnv` will be the object we will use to read values associated with the pointers that we are taking as argument.
Next, we read the string in from the `JNIEnv` and convert it into a C pointer to pass to `rust_greeting`. The result of that function is another C pointer, which we then need to convert to a back into a String. Using the `JNIEnv` transfers the ownership of the object to Java, but there is still a reference hanging around held by our Rust code. That memory will be freed as `world_ptr` goes out of scope. Then we return our String.
We declared that we needed the `jni` crate, that means we need to include the crate in the `Cargo.toml` file. Open it up and add the following between the `[package]` and `[lib]` declarations.
```
[target.'cfg(target_os="android")'.dependencies]
jni = { version = "0.5", default-features = false }
```
We also need to tell the compiler what type of a library it should produce. You can specify this in the `Cargo.toml` file's `[lib]` section:
```
[lib]
crate-type = ["dylib"]
```
We are now ready to build our libraries. Unlike with iOS, there is no handy universal Android library that we can make so we have to create one for each of our target architectures. We can then create symlinks to them from the Android project. You will need to use absolute paths to your libraries here, not relative ones, otherwise Android Studio will not be able to follow the link. Navigate to your `cargo` directory and run the following commands:
```
cargo build --target aarch64-linux-android --release
cargo build --target armv7-linux-androideabi --release
cargo build --target i686-linux-android --release
cd ../android/Greetings/app/src/main
mkdir jniLibs
mkdir jniLibs/arm64
mkdir jniLibs/armeabi
mkdir jniLibs/x86
ln -s <project_path>/greetings/cargo/target/aarch64-linux-android/release/libgreetings.so jniLibs/arm64/libgreetings.so
ln -s <project_path>/greetings/cargo/target/armv7-linux-androideabi/release/libgreetings.so jniLibs/armeabi/libgreetings.so
ln -s <project_path>/greetings/cargo/target/i686-linux-android/release/libgreetings.so jniLibs/x86/libgreetings.so
```
Now, head back to Android Studio and open `GreetingsActivity.java`. We need to load our Rust library when the app starts, so add the following lines below the class declaration and before the `onCreate` method.
```java
static {
System.loadLibrary("greetings");
}
```
This looks for a library called `greetings.so` inside the jniLibs directory and picks the correct one for the current architecture.
Open `res/layout/activity-greetings.xml`. In the `Component Tree` panel, highlight the `TextField` and open the `Properties` panel. Change the `ID` in the `Properties` panel to `greetingField`. This is how we are going to refer to it from our Activity.
Reopen `GreetingsActivity.java` and amend the `onCreate` method to call our greetings function and set the text on the `greetingField` `TextField` to the response value.
```java
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_greetings);
RustGreetings g = new RustGreetings();
String r = g.sayHello("world");
((TextView)findViewById(R.id.greetingField)).setText(r);
}
```
Build and run the app. If this is your first time in Android Studio, you may need to set up a simulator. When choosing/creating your simulator pick one with API 26. When the app starts, `Hello world` will be printed on your screen.
You can find the code for this on [Github](https://github.com/fluffyemily/cross-platform-rust).
================================================
FILE: newsletter/_posts/2017-07-27-browser-architecture-update.md
================================================
---
title: Browser Architecture Update
layout: text
description: The first of our newsletters introducing our work
mailinglist: https://groups.google.com/d/msg/firefox-dev/ueRILL2ppac/RxR9lLPkAwAJ
---
# Browser Architecture Update
[Link to mailing-list archive]({{ page.mailinglist }})
In the spirit of the Quantum Flow and Photon Newsletters - an update on what the Browser Architecture team is doing:
The Browser Architecture Team is new-ish - it’s existed since the re-org earlier this year, and its goal is to look ahead and make plans for our architecture.
We want to build the world’s best web browsers. Which is easy to say, but much harder to actually do.
mozilla-central is a big complex software project. Its size and complexity slows us down, and it’s daunting to think about making it easier to reason about. We’ve often put off making life easy for ourselves because our users come first. We generally prioritize fixing bugs for users over internal work.
But sometimes you can go faster for users by focusing on going faster yourself. Part of what the Browser Architecture team would like is to tackle some of the hard problems of our own development velocity.
We also need to look at some problems that will take several years to address. Problems that don’t fit into any project area roadmap.
We have a mandate to help build the world’s best web browsers, but how? Our strategy is twofold:
1. Kickstart a streamlined developer experience by removing roadblocks to immediate progress.
2. Consciously plan our software architecture at an organization-wide level.
So what are the specific problems we are looking at?
## Workflow
UI development in mozilla-central somewhat mirrors the web development ecosystem, however one way we’ve fallen behind is in unit tests for frontend code. In mozilla-central we usually rely on integration tests rather than unit tests. Integration tests end up being slower and are more likely to fail randomly. In addition they steal focus and don’t allow for integration with code editors, to see test results as you type. We’d like to figure out how to write more unit tests in our codebase.
We’re looking at turning headless mode into a way to continue work whilst running tests, rather than needing to run a VM or turning to sword-fighting.
We’ve also spent some time looking at ways to make developing the Firefox frontend more like web development, with features like CSS hot reload and refreshing the UI without rebuilding and restarting the process.
In parallel, we’re also looking at ways that we can improve Gecko development efficiency, especially on build times and debugging (Rust, rr, gdb on Mac).
## XUL and XBL
Our user interface language has problems. It’s uncared for, so it’s buggy and sometimes we don’t land fixes that we know about. It’s largely undocumented or worse, documented wrongly, so it’s hard to fix. Also it behaves almost like the web, which can lead web developers to think they understand it better than they really do.
We care about HTML, so that’s documented and optimized. XUL? Not so much.
The first talk I heard about Servo was in 2013. At the Q&A at the end someone asked about XUL and Servo — the answer was "There is no XUL in the product at all". Applause from audience. So the tide has been against XUL for at least 4 years, but so far we’ve not made headway in tackling it. The Browser Architecture team is approaching it from various angles:
Can we remove XUL and XBL piece by piece? We are experimenting with a couple of approaches with the preferences UI. Could we convert the XBL to JS modules and Web Components ‘in place’? Alternatively, how much work is it to rewrite it in HTML, using something like React? Or more dramatically, how far up the stack should our use of Rust go? Maybe we should consider a UI generated by Rust?
Alternatively, what’s the minimum we could do to just fix and document what we’ve got. Perhaps we should just focus on removing XBL? Maybe the plans above are too much work and we need to invest elsewhere entirely?
## User Data Storage and Sync
We don’t have a good story for storage of user data. We wouldn’t sit down to design the profile directory and come up with what we have today, and we need a better story to be able to sync more readily to Firefox for Android and iOS and to adapt to unanticipated and evolving product needs.
Recently we've been exploring Project Mentat, which promised some advances in helping us rapidly evolve data storage, and making syncing of data less troublesome. We decided that porting all of Firefox’s storage to Mentat was too much work right now, but the problem of improving user data storage and sync remains.
So we’re looking at the big picture of storage and sync and how we can share data not just between desktop and mobile, but also to help us develop and test new products in the market.
## A Monolithic Platform
We’re considering what it would look like to turn Gecko inside out so it’s more modular, can be reused in smaller parts and is easier to test, as with Rust crates in Servo. Changing that is hard — really hard — but the potential value is also significant.
We're looking at existing Firefox components and wondering if we can extract them (or rewrite them) as separate components that we could reuse in situations where we can't rely on the Gecko runtime environment, and we’re looking at the work on GeckoView to see how it can help us reuse Gecko outside Firefox.
----
There are a lot of ideas there. Maybe some sound cool, maybe some sound crazy. We’re planning on writing up our findings on each of the areas so look out for future posts where we drill down into some of these ideas.
Thanks
Joe Walker (on behalf of the Browser Architecture Team)
Feedback is welcome. Either reply here or find Nick Alexander, Brendan Dahl, Brian Grinstead, Rob Helmer, Randell Jesup, Myk Melez, Richard Newman, Victor Porof, Emily Toop, Dave Townsend, Peter Van Der Beken, or Joe Walker near some watercooler (like #browser-arch).
================================================
FILE: newsletter/_posts/2017-08-24-browser-architecture-newsletter-2.md
================================================
---
title: Browser Architecture Newsletter 2
layout: text
description: A follow-up with updates on XBL Conversion, Storage and Sync and Workflow Improvements
mailinglist: https://groups.google.com/d/msg/firefox-dev/ueRILL2ppac/RxR9lLPkAwAJ
---
# Browser Architecture Newsletter 2
[Link to mailing-list archive]({{ page.mailinglist }})
Welcome to the second Browser Architecture Newsletter! Since our [last update](https://groups.google.com/forum/#!msg/firefox-dev/ueRILL2ppac/RxR9lLPkAwAJ;context-place=forum/firefox-dev), we have a [new website](https://mozilla.github.io/firefox-browser-architecture/) which includes posts about some of the [problems with XUL](https://mozilla.github.io/firefox-browser-architecture/text/0003-problems-with-xul.html), [extracting Necko](https://mozilla.github.io/firefox-browser-architecture/text/0002-extracting-necko.html) as a standalone component, and [comparing](https://mozilla.github.io/firefox-browser-architecture/text/0004-xbl-web-components.html) XBL with Web Components.
The website is built from [this GitHub repository](https://github.com/mozilla/firefox-browser-architecture) of Markdown files, and we’re using pull requests to review and revise our research proposals and analyses. Do you have ideas for improvements? File an issue on the repo or contact us in #browser-arch!
## XBL Conversion
We'd like to reduce or even remove XBL and XUL from the tree. It's becoming clear that replacing XBL is more important and easier than replacing XUL, so there are a few projects ongoing to learn more about it.
A conversion of the behavioral `<preference>` binding is in progress at [Bug 1379338](https://bugzilla.mozilla.org/show_bug.cgi?id=1379338). Although this work started as a prototype, it resulted in a more straightforward implementation than we currently have, so is on the path to being landed.
As mentioned above, there’s some new [documentation](https://mozilla.github.io/firefox-browser-architecture/text/0004-xbl-web-components.html) comparing XBL with Web Components. There’s also a [visual tree](https://bgrins.github.io/xbl-analysis) of the bindings we use in Firefox and what features each uses.
A prototype of replacing some XBL UI components in about:preferences with Custom Elements is attached to [Bug 1392367](https://bugzilla.mozilla.org/show_bug.cgi?id=1392367). This relies on a [forked version](https://github.com/webcomponents/custom-elements/compare/master...bgrins:firefox-browser-chrome?expand=1) of the [custom elements polyfill](https://github.com/webcomponents/custom-elements) made to work in XUL documents.
A prototype of converting `<tabbrowser>` to a JS class is in progress at [Bug 1392352](https://bugzilla.mozilla.org/show_bug.cgi?id=1392352). This would cause gBrowser to become a JS object instead of a DOM Node. This conversion is semi-automated [by a script converting the binding to a JS class](https://github.com/bgrins/xbl-analysis/blob/ad87d682b937620b1c129e49c4081483c7074540/scripts/build-custom-elements.js).
## Storage and Sync
We're in the process of cataloging and analyzing some of Firefox's sprawling storage technologies (45+ different data stores and dozens of formats on desktop alone), and how they relate to Sync and across platforms. We're hoping to find ways to make this situation simpler, more reliable, and easier to extend.
You can see the beginnings of our knowledge capture [here](https://github.com/mozilla/firefox-data-store-docs) (PRs welcome!), and a [spreadsheet](http://bit.ly/2vpZI9e) showing how data types interact with Sync and mobile platforms.
Huge thanks to the Sync team and data and product folks across the org for helping with our research.
## Workflow Improvements
Lately I've been paying extra attention to development papercuts when working on bugs, especially issues when using ./mach run without passing a profile. Here’s a list of recent changes on that front:
* You can [open the Browser Toolbox](https://groups.google.com/d/msg/firefox-dev/678mrnS6120/KXcP18ZUCAAJ) in a local build without flipping prefs.
* The default browser prompt and about:config warning screen [no longer show up](https://groups.google.com/d/msg/firefox-dev/kPwA1y-7BpI/gX6rvhEjBQAJ) for the scratch_user (the profile used when you do ./mach run without a profile).
* You can flip any pref for the scratch_user with the --setpref option, or by adding them in your [machrc file](https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/mach#Does_mach_have_its_own_configuration_file). For example, to make sure I always see dump output I added this to my ~/.mozbuild/machrc file:
`[runprefs]
browser.dom.window.dump.enabled=true`
* There’s a [new keyboard shortcut](https://groups.google.com/d/msg/firefox-dev/Tme95bp3EHY/ow-l077FAAAJ) to do a restart + session restore in local builds.
* The Browser Console is [now part of session store](https://bugzilla.mozilla.org/show_bug.cgi?id=1388552), so it will reopen after using the above keyboard shortcut.
================================================
FILE: newsletter/_posts/2017-09-22-browser-architecture-newsletter-3.md
================================================
---
title: Browser Architecture Newsletter 3
layout: text
description: A newsletter on architecture review, XBL Conversion, Storage and Sync, Workflow Improvements and a developer survey
mailinglist: https://groups.google.com/forum/#!topic/firefox-dev/p9rTlfFUXlQ
---
# Browser Architecture Newsletter 3
[Link to mailing-list archive]({{ page.mailinglist }})
Welcome to the third Browser Architecture Newsletter! Since our [last
update](https://mozilla.github.io/firefox-browser-architecture/posts/2017-08-24-browser-architecture-newsletter-2.html),
we've continued to investigate moving away from XBL and started to
document what we're talking about when we talk about XUL problems. We're
also working with members of the Sync, AS, and Lockbox teams to figure
out what the future of storage and syncing looks like for Mozilla.
A lot of the issues that the Browser Architecture team are digging into
are larger than our team. Our goal is to discuss and review entire
product level architecture issues and build consensus around solutions.
We're interested in engaging with engineers around the organization.
We're actively reaching out to folks but if you want to talk to us we'd
sure love to hear from you. You can find us in \#browser-arch on IRC or
Slack.
## Architecture Review
We are asking for comments on our [proposed
process](https://mozilla.github.io/firefox-browser-architecture/text/0006-architecture-review-process.html)
for when our team performs architecture reviews. The process is in draft
right now so if you have suggestions for how to improve it please let us
know.
If you've got ideas, questions, or concerns, talk to
[jwalker](https://mozillians.org/en-US/u/jwalker/).
## XBL Removal
We've filed a [meta
bug](https://bugzilla.mozilla.org/show_bug.cgi?id=1397874) for our
de-XBL work, which includes a variety of prototypes and other
investigations, including: moving XBL bindings to Custom Elements,
moving XBL bindings to JS modules, and performance comparisons of XBL
bindings to a custom elements polyfill. You can view all the prototypes
and investigations we're working on by viewing the meta bug's
dependencies.
We're getting ready to run through a design review for our XBL
replacement plans. The review will be chaired by Dave Townsend and
include a panel of experts on both Gecko and Firefox. The review process
itself is a work in progress, and our XBL Removal review is a trial run
to help us refine it. Look for more information on the review itself
soon when we get further along in the process.
If you've got ideas, questions, or concerns, talk to
[bgrins](https://mozillians.org/en-US/u/bgrinstead/).
## Storage and Sync
We've [captured](https://github.com/mozilla/firefox-data-store-docs) a
bunch of knowledge about Firefox's data stores. That effort is feeding
into some roadmapping work with the Sync and Activity Stream teams. We
hope to have some concrete roadmap documentation underway by our next
newsletter, as well as some stage-setting blog posts.
We expect that the future will include some cross-platform Rust storage
services, so we've been researching how this looks in practice.
[Deploying an Rust library on
iOS](https://mozilla.github.io/firefox-browser-architecture/experiments/2017-09-06-rust-on-ios.html)
is a short tutorial describing how to build and deploy a Rust library
for use inside an iOS app.
If you've got ideas, questions, or concerns, talk to
[rnewman](https://mozillians.org/en-US/u/rnewman/) or
[fluffyemily](https://mozillians.org/en-US/u/etoop/).
## Workflow Improvements
* [You can
now](https://groups.google.com/d/msg/firefox-dev/L5D3YSBxqh8/pjonQLutCwAJ)
open Browser Toolboxes on more than one instance of Firefox at the
same time
* The ‘no popup autohide' mode that can be toggled in the Browser
Toolbox to inspect popups [now automatically
resets](https://bugzilla.mozilla.org/show_bug.cgi?id=1251658) when
the toolbox closes if you clicked the button to enable it. You can
also flip this pref yourself with \`ui.popup.disable\_autohide\`
* dump() is [now enabled by
default](https://bugzilla.mozilla.org/show_bug.cgi?id=1395711) in
local builds - kudos to :kmag for doing this
If you've got ideas, questions, or concerns, talk to
[bgrins](https://mozillians.org/en-US/u/bgrinstead/) or
[nalexander](https://mozillians.org/en-US/u/nalexander/). If you have
suggestions or pain points about doing Gecko development
(non-front-end), please contact
[jesup](https://mozillians.org/en-US/u/jesup/).
## Front-end Developer Survey
We've sent out a survey to front-end developers asking them about their
workflow and productivity issues. The results of this survey will help
us target future workflow and development improvements.
If you're an employee working on front-end Firefox code and you haven't
received the survey, please let
[mossop](https://mozillians.org/en-US/u/Mossop/) or
[nalexander](https://mozillians.org/en-US/u/nalexander/) know so we
can get it to you! We're still discussing if we should open this survey
up to contributors at this time. If you've got other ideas, questions,
or concerns, [mossop](https://mozillians.org/en-US/u/Mossop/) and
[nalexander](https://mozillians.org/en-US/u/nalexander/) are also the
right people to contact.
================================================
FILE: newsletter/_posts/2017-10-19-browser-architecture-newsletter-4.md
================================================
---
title: Browser Architecture Newsletter 4
layout: text
description: A newsletter on architecture review, XBL Conversion, Storage and Sync, Workflow Improvements and a developer survey
mailinglist: https://groups.google.com/forum/#!topic/firefox-dev/CLFtj8qUSv8
---
# Browser Architecture Newsletter 4
[Link to mailing-list archive]({{ page.mailinglist }})
Welcome to the fourth Browser Architecture Newsletter!
A lot of the issues that the Browser Architecture team are digging into are
larger than our team. Our goal is to discuss and review entire product level
architecture issues and build consensus around solutions. We’re interested in
engaging with engineers around the organization. We’re actively reaching out to
folks but if you want to talk to us we’d sure love to hear from you. You can
find us in `#browser-arch` on IRC or Slack.
## XBL Removal
Since our last update, we've completed the design review for [a plan to move
away from XBL in Firefox](https://mozilla.github.io/firefox-browser-architecture/text/0007-xbl-design-review-packet.html).
Thanks to the long list of people who guided the creation of that plan, and the
chair and panel for putting it through its paces. We're working through some
follow-up investigations; expect a separate email with more details soon.
## Storage and Sync
We've been doing lots of writing on this topic. Emily has [documented](https://github.com/fluffyemily/cross-platform-rust) and
blogged about getting Rust libraries up and running on multiple platforms ([iOS](https://mozilla.github.io/firefox-browser-architecture/experiments/2017-09-06-rust-on-ios.html),
[Android](https://mozilla.github.io/firefox-browser-architecture/experiments/2017-09-21-rust-on-android.html)), which will pave the way for shared storage implementations across
mobile and desktop products. Richard has published the [first in a series of
blog posts about sync](https://medium.com/@rnewman/thinking-about-syncing-part-1-timelines-7f758e2bd676).
A [roadmap review](https://mozilla.github.io/firefox-browser-architecture/text/0006-architecture-review-process.html) is planned for this quarter; thanks to everyone — a dizzying
array of engineers, managers, product folks, and Deep Syncers — who've been
contributing to that process. Let rnewman know if you're not involved and would
like to be.
## Front-end Developer Survey
We sent out a survey for front-end developers to try to understand how to
target our architecture and workflow improvements. The results will be
published soon, but one thing we found was that a lack of documentation is
holding developers back. We'll be digging into this more in the future.
================================================
FILE: newsletter/_posts/2017-11-29-browser-architecture-newsletter-5.md
================================================
---
title: Browser Architecture Newsletter 5
layout: text
description: A newsletter on Storage and Sync, XBL Removal and Workflow Improvements
mailinglist: https://groups.google.com/forum/#!topic/firefox-dev/XKp3EthdJ60
---
# Browser Architecture Newsletter 5
[Link to mailing-list archive]({{ page.mailinglist }})
Welcome to the exciting fifth installment in our series of Browser Architecture Newsletters!
## Storage and Sync
We are pleased to report that the Sync and Storage roadmap proposal successfully passed review by dcamp. Thanks to the 20+ people from around the organization who helped us build consensus.
The roadmap establishes that storage and syncing of user data is a pillar of the Firefox ecosystem, warranting holistic and long-term attention, and outlines both where we’d like to end up and some ideas for how to get there.
We expect this roadmap to lead to a number of subsequent concrete design reviews, as we begin with prototyping, further competitive analysis, and more coordination across Mozilla. The roadmap doc itself has been lightly cleaned up and [published](https://mozilla.github.io/firefox-browser-architecture/text/0008-sync-and-storage-review-packet.html) alongside [our other docs](https://mozilla.github.io/firefox-browser-architecture/).
If you’re working on improving storage or syncing of data in any of our products or projects, on any platform, or if you’re currently struggling to work with what currently exists, then we’d like to hear from you and align our efforts.
You can find us in Slack or IRC (#browser-arch), or come find rnewman, etoop, rfkelly, markh, or adavis.
We’ll be meeting to talk in more detail about the roadmap, the present, and the future [at the All Hands](https://austinyallhands2017.sched.com/event/CyOK/sync-and-storage). We would welcome your participation.
## XBL Removal
Our work to remove XBL is proceeding apace. We’ve started by removing bindings that are never or only rarely used. Work to convert XBL bindings to custom elements is currently stalled while we get custom elements working in XUL.
You can track how this project is progressing with some [lovely graphs](https://bgrins.github.io/xbl-analysis/graph/). If you’d like to help with the effort, you can grab a bug tagged with [[xbl-available]](https://mzl.la/2AyTxCG).
## Workflow Improvements
Running mochitest-plain in headless mode is now supported on Windows, MacOS, and Linux. Use the “--headless” flag with “./mach test” to try it out.
## Austin All Hands
We have two meetings planned for the All Hands. You can see these in Sched here:
* [Austin Y'all Hands 2017: Sync and Storage](https://austinyallhands2017.sched.com/event/CyOK/sync-and-storage)
* [Austin Y'all Hands 2017: XBL and XUL Removal](https://austinyallhands2017.sched.com/event/CyLC/xbl-and-xul-removal)
================================================
FILE: newsletters.xml
================================================
---
layout: null
---
<?xml version="1.0" encoding="UTF-8" ?>
<rss version="2.0">
<channel>
<title>Mozilla Browser Architecture Newsletters</title>
<description>The published newsletters from the Mozilla Browser Architecture team.</description>
<link>{{ site.url }}{{ site.baseurl }}</link>
{% for post in site.categories.newsletter %}
<item>
<title>{{ post.title }}</title>
<description>{{ post.description }}</description>
<link>{{ site.url }}{{ site.baseurl }}{{ post.url }}</link>
<guid isPermaLink="true">{{ site.url }}{{ site.baseurl }}{{ post.url }}</guid>
<pubDate>{{ post.date }}</pubDate>
</item>
{% endfor %}
</channel>
</rss>
================================================
FILE: text/0000-template.md
================================================
---
title: Documentation Template
layout: text
---
# Documentation Template
## Problem
A short lay summary of the problem space and the stakeholders.
Our documents should have a front matter containing a title, which should be
the same as the first heading (using `#`).
Other headings will generally be at the `##` or `###` levels.
## Proposal
Explanation of the solution to the problem.
## Drawbacks / Unresolved Questions
It's a rare proposal that perfectly meets the problem, so we'll probably need
to document the drawbacks.
## Alternatives
Competitive analysis suggesting alternatives, costs, and opportunities.
## Links to discussion
* Mailing list discussion?
* Original GDocs document
================================================
FILE: text/0001-documenting-output.md
================================================
---
title: Documenting our Output
layout: text
---
# Documenting our Output
## Problem
The Browser Architecture team needs to publish its findings clearly. We would like anyone in Mozilla to be able to:
* Find what we’ve learned so far
* Find what we’re working on
* Leave feedback
The solution should:
* Be low friction for contribution
* Be flexible in how the result is published
* Ensure that documents can be found easily
## Proposal
Browser Architecture team will document its output like this:
1. The discussion phase involves conversations in the browser-arch [Google Group](https://groups.google.com/a/mozilla.com/forum/#!forum/browser-arch), [IRC channel](https://www.irccloud.com/#!/ircs://irc.mozilla.org:6697/%23browser-arch), [Slack channel](https://mozilla.slack.com/messages/C5F80LV0C/), etc - when a semi-permanent record of current thinking is needed, we use GDocs, stored in [the Browser Architecture folder](https://drive.google.com/drive/u/1/folders/0BzQINYlY78CtbGVqVDVlZlNJX0k).
2. When we've come to some internal consensus (note this doesn't mean that the proposal is final) we convert our GDocs to Markdown/Git to create an 'RFC' and get feedback from management and the wider Mozilla developer community as appropriate.
* Use [this template](0000-template.md)
3. As there are updates we keep the Markdown up to date
4. We may publish to a wider audience using a blog post or GitBook as required
## Drawbacks / Unresolved Questions
It is possible that GDocs has such a low update friction that people will resent the PR process imposed by Github. If this turns out to be true then we will need to update this process.
This process is somewhat modelled on the [Rust RFC](https://github.com/rust-lang/rfcs) process. It isn't certain that we need the numeric prefix.
## Alternatives
* Create an index to our documentation using Google Docs, and stay in that world.
* Use Medium and or RSS to publish output.
## Links to discussion
* [Mailing list discussion](https://groups.google.com/a/mozilla.com/forum/#!topic/browser-arch/FOtfYVHbgfo)
================================================
FILE: text/0002-extracting-necko.md
================================================
---
title: Extracting Necko
layout: text
---
# Extracting Necko
## Proposal
We should extract Necko for reuse in background services and on mobile platforms.
### Motivation
We expect that Mozilla will need to build consistent native applications and background services across platforms, including services and applications that don't run Gecko atop a profile directory. To do this it would be beneficial (in theory) to leverage our existing investment in our network layer, avoiding the confusing situation of having multiple ways to configure network settings, and avoiding the possibility of different behaviors in different parts of an application.
Furthermore, we expect the question of reuse to recur. Necko is sometimes characterized as being a relatively self-contained piece of mozilla-central's compiled code, so it should be one of the easiest to reuse. Also, its interdependencies are probably applicable to other components, so it's a good opportunity to learn.
Breaking out a component for reuse is, for our purposes, equivalent to embedding: it's taking the component and driving it in a different context. Specifically, we are interested in those contexts being (a) Rust, (b) not necessarily desktop. An alternative approach is to build a new Gecko application on top of a slimmed-down libxul. In the case of Necko, that might be something like a cURL clone. This kind of reuse is a known quantity, so we won't discuss it here.
## References
* Lots of very old documents:
* <https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Necko/Embedding_Necko>
* <https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Necko>
* <https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Necko/Architecture>
* Discussions with rhelmer and jduell
* Examination of `/netwerk`
## Analysis: obstacles to embedding
* In its early days, 15+ years ago, Necko was intended to be embeddable; in recent years that has not been an aim of the component leads. Meeting the needs of Gecko and Firefox is more important. (The definition of 'embeddable' here still carries the obligations of Gecko-style initialization.)
* As noted in the "Embedding Necko" document, Necko depends on Gecko features:
- XPCOM and NSPR.
- mDNS, `NetUtil.jsm`, and some HTTP functionality requires a JS runtime. This is feasible to eliminate if needed.
- Docshell.
- If cookies are desired, they must be provided via the nsICookieService interface, which is baked into Necko. One can build Necko without, via `--disable-cookies`, but this is a build-time decision. One could conceivably provide one's own implementation of the cookie service, but the interface requires a stateful synchronous global singleton.
- Necko depends on the Mozilla preferences service (libpref) for configuration, which is a privileged singleton that itself depends on the observer service (for startup and profile events) and the directory service (for profile directory and app locations).
* Because of its dependency on the profile (both for reading state and as a key for the disk cache) and flushing files to disk for prefs, it's not safe to use Necko from two distinct processes simultaneously for the same application.
* Because of its extensive use of XPCOM singleton patterns, it's not possible to have two independent Neckos in the same process; they need separate processes and separate profile directories. Some independence can be achieved by carefully configuring channels, but it's not isolation.
* It's not possible to use Necko without a profile service (*e.g.*, to make a network request without touching disk).
* Parts of Necko are still single-threaded, requiring new channels to be created on the main thread, and dispatching callbacks to the main thread.
* Necko's init and deinit are managed by observer notifications, which is not ideal for embedding outside of Gecko.
# Proposal
## Proposal
# Conclusion
Necko is a very feature-rich C++ networking library that is directly targeted at meeting the needs of single-user, single-main-process, single-profile, interactive applications built on a persistent profile directory in the traditional Mozilla model. It is not well-suited for use outside of the Gecko lifecycle — indeed, almost every aspect is designed for the Gecko world — and would require substantial effort in decoupling to be suited for embedded use. Even if so decoupled, it would still mandate the use of XPCOM and the NSPR.
To extract Necko for reuse outside of Gecko, by non-XPCOM-based applications, would — almost by definition — require coming up with a plan to deal with XPCOM. That plan might look a lot like continuing historical efforts on deCOMtamination.
Necko's data structures, interface definitions, error codes, macros, etc. *are* XPCOM's, and so eliminating XPCOM dependency from the module is likely to involve touching almost every line of code. [Automated tools to do so have been proposed](https://brendaneich.com/2006/10/mozilla-2/).
(Going further to eliminate NSPR and mfbt data structures, yielding a more standard C++ library, would be further work still.)
This is particularly relevant when we consider oxidation — the replacement of C++ and JS components of Firefox with Rust — and even more so when we consider the use of such components both outside and inside Firefox *from* Rust (rather than from existing non-Rust callers).
There is ongoing work on landing Rust components in Firefox; most obstacles there are around XPCOM integration for Rust code, which currently requires writing lots of routine boilerplate by hand.
There is also some initial scoping of work to extract and use Gecko C++ components from Servo, which is a similar problem to using Gecko C++ components from Rust background services or mobile applications. Those estimates are large (but not yet published), on the order of 3-5 engineer years for non-trivial components.
Extracting such a module is not routine work: delicate balances need to be found between maintaining useful integration with Gecko (*e.g.*, Necko's `nsIChannel` permeates the codebase) versus designing for embedding, and between reimplementing Gecko dependencies in Rust for reuse versus reworking the component to avoid them.
Our position is that extracting Necko from Gecko for reuse would be a significant amount of work. Doing so in a way that allows its use from ordinary Rust consumers, without shipping XPCOM/NSPR/*etc*., would be even more so.
That amount of work is certainly much more than simply using the existing well-tested and idiomatic networking libraries available to Rust code. Given that the benefits of using Necko in its entirety (instead of *e.g.*, just using Hyper for HTTP) are relatively small, our conclusion is that we **should not attempt to reuse Necko outside of Gecko**.
Standalone Rust components should use Rust networking libraries of equivalent functionality, *e.g.*, Tokio/Hyper (already vendored for Servo) or libpnet. Useful Necko functionality should be ported to Rust as needed. Those Rust reimplementations can be imported back into Necko like any other piece of oxidized code.
## Alternatives
Alternatives are, variously:
* Do the work to extract Necko. Some significant benefit would need to be articulated in order for this to be worthwhile.
* Use Gecko for all network-related background services. We already know that this is not feasible, both for application size reasons (Gecko is large), and for technical lifecycle reasons as articulated above. Furthermore, this argument ends up as "use Gecko for everything everywhere", which is not useful.
## Links
This document was originally drafted in the ["Answered questions" Google Doc](https://docs.google.com/document/d/1RIMbe0yJGNywFTf1n5ka35JpLtrkE3lHz8iip6jH_VE/edit#).
## Colophon
This document was written by Richard Newman (rnewman), reviewed by Jason Duell (jduell), Rob Helmer (rhelmer), Joe Walker, and Myk Melez.
================================================
FILE: text/0003-problems-with-xul.md
================================================
---
title: Problems with XUL
layout: text
---
# Problems with XUL
XUL is often mentioned as a source of problems for developers. This document
aims to list the different kinds of problems that exist.
## XUL ownership
XUL is largely unowned. While there are developers who know and understand the
XUL code they are few and mostly dedicated to other projects at present.
## XUL bugs
XUL contains bugs. This isn’t due to anything particularly special about XUL but
when we encounter bugs we frequently choose to not fix them, instead trying to
find workarounds in the UI instead. See [bug 354527](https://bugzilla.mozilla.org/show_bug.cgi?id=354527)
for example.
## No new features
There is no new feature implementation going on in XUL compared to HTML which
receives constant improvements to meet the demands of the web.
## Performance issues
It is thought that XUL’s performance is worse than that of HTML where we have spent
large amounts of time optimising for the web but little amount of time optimising
for XUL.
## Availability of example code and help
XUL is a proprietary technology developed by Mozilla and only used by Mozilla. When
developers want to find solutions to problems or implement new features the only
example code and documentation they have available is our code and our documentation.
## Learning curve
Related, when new developers join Mozilla it is almost certain they will not have
heard of or used XUL before and so there is a learning curve to understanding and
becoming proficient in the language.
## Recruiting challenge
Even if the learning curve is small relative to the productive output of an engineer
who has learned the technology, recruiting for a role that involves using a
proprietary language is more difficult than recruiting for a role that involves using
(and learning, if needed) a popular language. This applies to both new contributors
as well as employees.
Besides the cost of learning the language and the uncertainty about whether the work
will be enjoyable, which may dissuade some candidates, the more important factor is the
opportunity cost for career development relative to developing expertise in a popular
language: engineers with “XUL experience” don’t benefit from it when looking for their
next job (or even position within the company).
## Frameworks
Often developers want to use popular JS frameworks like jQuery and React to develop new
features. Commonly these frameworks are built for HTML and don’t work correctly in XUL
documents.
## SVG and HTML in XUL
Because XUL is missing certain features often developers want to embed HTML or SVG
elements inside of XUL. Because of differences in the box model this sometimes leads
to layout not working correctly.
## Gecko Complexity
XUL increases the complexity of Gecko generally, which imposes costs on Gecko
development and maintenance (such as the Stylo work).
================================================
FILE: text/0004-xbl-web-components.md
================================================
---
title: XBL and Web Components
layout: text
---
# XBL and Web Components
XBL is a technology used to implement reusable widgets in XUL. Web Components are a set of specifications that provide modern and standards-based alternatives for doing the same thing with HTML. For the purposes of this document, "Web Components" refers to the [Custom Elements](https://w3c.github.io/webcomponents/spec/custom/) and [Shadow DOM](https://w3c.github.io/webcomponents/spec/shadow/) specs, which map relatively closely to how the Firefox frontend is using XBL.
What follows is an initial analysis of what a migration away from XBL and to Web Components may look like. We will need to spend more time prototyping this within the Firefox UI to get a clearer picture about the viability of a migration.
## What bindings are we talking about?
Most of the relevant bindings are declared in [toolkit/content/widgets](https://github.com/mozilla/gecko-dev/tree/master/toolkit/content/widgets).
This analysis is focused on widely used and relatively small XBL components that could potentially be swapped out more-or-less in place with corresponding Web Components. For instance: buttons, labels, input fields, etc.
Components that are only used once or are implemented mostly in C++ are ignored in this analysis. For instance: tabbrowser, trees, etc. Most of the same principles that apply to smaller components should also apply to them, but we may also consider different options for moving them away from XBL.
## Auditing the XBL feature set
From the [Introduction to XBL](https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XUL/Tutorial/Introduction_to_XBL), a binding has five types of things that it declares: Content, properties, methods, events, and style. But it's helpful to look at specific features used in Firefox and comparing how they map back to Web Components.
Which bindings use which features is visualized at https://bgrins.github.io/xbl-analysis/. The scripts to generate that tree and also to semi-automate translation of bindings can be found at https://github.com/bgrins/xbl-analysis.
### XBL features that map to Web Components features
1. `<property>`: These map to JS getters and setters on the Custom Element class (‘val’ name is magic in xbl, would need to keep same name to translate directly)
2. `<method>`: These map to functions on the Custom Element class
3. `<constructor>` This maps to connectedCallback on the Custom Element class (which is called when the element is inserted into a document).
In XBL, the constructor fires after the element is inserted into the document (since the binding is attached through a CSS property). However, the constructor will wait some additional time to fire, unless if the element is accessed via JS in which case it will run immediately
4. `<destructor>`: These map to `disconnectedCallback` on the Custom Element class
5. `[extends]`: A way to extend the functionality of another binding, i.e. `extends="chrome://global/content/bindings/general.xml#basetext"`. On a surface level this appears to map directly to how the ‘extends’ keyword in ES6 classes behavior with Custom Elements, but more investigation is needed. There’s a small set of tags where you can do `extends="xul:button"` but that’s aliasing the same thing.
6. `<handlers>`: Can be implemented with addEventListener or possibly a helper on the base element i.e:
```
helpers: {
"click": () => { /* bubble by default*/ }
"focus|capturing": () => { /* copy in xbl implementation */ }
}
```
7. `[inherits]`: Is a way to declaratively copy attributes from a binding parent to content and keep them in sync, similar to the `<observes>` element. For example, this copies the feedURL from the parent into the 'value' field of the label and also observes future changes: `<feed feedURL="http://foo.com"><xul:label xbl:inherits="value=feedURL,tooltiptext=feedURL"></feed>`. The same thing could be built on top of Custom Elements using a MutationObserver and some glue code to set it up in the `connectedCallback`.
8. Scoped CSS Styling: this can be done with Shadow DOM
9. Insertion Points: XBL and Shadow DOM appear to use similar models but different keywords. Further investigation is needed.
### XBL features that don't map Web Components features
1. `[implements]`: This is a way for elements with XBL bindings to implement an XPCOM interface. XBL also auto generates the Qi bits for you. One example usage is that labels implement complex cropping behavior via `implements="nsIDOMXULLabelElement"`, but there are many others (panels, menus, scrollbars). There are around 39 instances of this feature, and this will need to be handled on a case-by-case basis. But we have some options:
* Write chrome-only webidl bindings for each of the individual nsiDOMXULFoo classes
* Replace things like `implements="nsIObserver"` with just a separate object that in fact implements it
* For some input elements like checkboxes, we could get rid of the xul interface since the corresponding HTML element has the same behavior already
2. Extension Compat: For instance, the Tab Center test pilot addon causes tab browser to become vertical via inheritance with some changes. This allows us to use the same tag name / markup so that it doesn’t need to re-write CSS. However, with the migration to Web Extensions this won’t be necessary in the general case.
3. Binding via CSS: `-moz-binding` allows a XBL binding to be attached or detached to any element through CSS. Web Components would be tied to HTML (based on the particular tag being created). This feature is used in [at least one place to swap an element to a different binding when setting a property](http://searchfox.org/mozilla-central/rev/7cc377ce3f0a569c5c9b362d589705cd4fecc0ac/toolkit/content/widgets/text.xml#47), but we can work around it.
## XBL Usage in Firefox
How many times are individual bindings used? We've written a module to track how often XBL (and XUL) are used, and a modified version of the test runner that keeps a count throughout the mochitest-browser suite. [A sorted list of used bindings is attached on the bug](https://bugzilla.mozilla.org/show_bug.cgi?id=1376546), but here are the top few bindings:
url("chrome://global/content/bindings/text.xml#text-label")
url("chrome://global/content/bindings/general.xml#image")
url("chrome://global/content/bindings/toolbarbutton.xml#toolbarbutton")
url("chrome://global/content/bindings/menu.xml#menuitem")
url("chrome://global/content/bindings/scrollbox.xml#autorepeatbutton")
url("chrome://global/content/bindings/button.xml#button")
## Performance Considerations
XUL and XBL both use fastload and they both have prototype cache, so loading them and opening new windows should be quite fast. Custom Elements (and Shadow DOM) don't have anything like that right now. If we want to use them in browser chrome, we may need to figure out how to support things like this.
The first step is getting some measurements to compare things like element creation time, memory usage, etc. This work is being tracked in [Bug 1387125](https://bugzilla.mozilla.org/show_bug.cgi?id=1387125).
## Notes
* Custom Elements don't currently work in XHTML documents, but are specified to.
* Custom Elements don't currently work in XUL documents and additional work is required to support them. In particular:
* Elements can be created from the XUL prototype cache, bypassing the XML parser
* XUL elements are subtly different from HTML elements in how they handle their parenting and prototype chains
* The [Custom Elements polyfill](https://github.com/webcomponents/custom-elements) doesn't work in XUL documents, but a [fork designed to use deepTreeWalker](https://github.com/webcomponents/custom-elements/compare/master...bgrins:firefox-browser-chrome?expand=1) does. Using this would allow us to prototype swapping out components without switching the top level document away from XUL. If this turns out to be a good way to migrate away from XBL, we'd have to do work to make the native Web Components implementation work in XUL documents.
* The relevant Web Components implementations are still off by default and missing features in Firefox:
* Custom Elements is in progress, with plans to complete the work by end Q3 2017
* Shadow DOM is backlogged, with plans to start the work in H2 2017.
================================================
FILE: text/0005-problems-with-xbl.md
================================================
---
title: Problems with XBL
layout: text
---
# Problems with XBL
## Unmaintained
XBL is a proprietary technology developed by Mozilla. We are solely responsible for the maintenance and improvement of XBL. Any time spent on doing this doesn’t contribute to the evolution of the Web so effectively we don’t spend anytime improving XBL except for plugging serious holes.
## Poorly Documented
The XBL wiki page says:
> "XBL 1.0 is specified in XBL 1.0 Reference. Unfortunately, the actual implementation in Mozilla is different from the specification, and there's no known document available describing the differences. Hopefully, the Reference will be updated to describe those differences."
That was written in 2005. It’s still true. Except for the "hope" part.
Unlike other technologies like web components where there is plenty of documentation available on the internet, documentation for XBL is almost entirely written by Mozilla and so there is very little of it.
The only reference for what is "correct" is what the frontend expects, and what the frontend expects is largely based around a hodgepodge of workarounds for bizarre behavior in the XBL implementation
## Verbose and Brittle (Because XML)
Writing code in XML requires extensive use of CDATA sections making the syntax more verbose than it needs to be.
## Dynamic Binding Changes via CSS
An XBL binding can set an attribute on itself that causes it to change to a different binding, e.g. [the add-on binding](https://dxr.mozilla.org/mozilla-central/rev/3ecda4678c49ca255c38b1697142b9118cdd27e7/toolkit/mozapps/extensions/content/extensions.xml#1245). These cases are difficult to find and debug and it is very unclear what is happening unless the code is well documented.
## Complicates Stylo
We have a ton of code in stylo to handle XBL `<stylesheets>`, and XBL scoped stylesheets are sufficiently different from shadow DOM scoped stylesheets that the code isn't really reusable
## Complicates Gecko
The fact that XBL is driven off of frame construction is awful. It causes so much weird stuff to happen at very inopportune moments and creates an enormous amount of complexity in what is arguably the most complex and touchy part of the platform (the frame constructor).
Anonymous subtrees cause lots of complication in general, and we currently have to deal with three kinds (shadow DOM, NAC, and XBLAC). Dropping this down to 2 would be a huge improvement.
There is lots of complexity and security weirdness that required us to build and maintain the content XBL scope stuff in XPConnect. Eliminating the in-content XBL bindings (video, marquee, pluginProblem, and scrollbar) would be a lower-hanging fruit to eliminate this.
nsBindingManager is a mutation observer, and its existence means some additional overhead for our DOM mutations, especially [ContentAppended, ContentInserted and ContentRemoved](https://searchfox.org/mozilla-central/rev/51b3d67a5ec1758bd2fe7d7b6e75ad6b6b5da223/dom/xbl/nsBindingManager.h#43).
The way we do XBL binding (via CSS) means that often when you touch a node from script we have to go resolve styles for it to see whether there's XBL involved. We optimize this out on the web in various ways, but in XUL it's a pain.
## Not Used Elsewhere
This presents a recruiting challenge, there aren’t developers with knowledge of XBL available for hire so everyone we hire has to be trained in this unique technology and there is a steep learning curve.
================================================
FILE: text/0006-architecture-review-process.md
================================================
---
title: Architecture Review Process
layout: text
---
# Architecture Review Process
## Introduction
This process targets Architectural Reviews in two categories: “Roadmap” and “Design”. It doesn't tackle how to review in-progress projects to see if they should continue.
The function of a **Roadmap Review** is to decide if a thing should be done. The goal is to bring together a packet of data to inform a management decision to provide resources to make the thing happen. A Roadmap Review should happen early in the process so that build time isn’t wasted on a “No” decision, but so that enough information is available to management.
A **Design Review** is to decide if the thing should be done in a particular way. The goal is to ensure that the problem space is understood well enough that a realistic bug breakdown can be made, that those working on the problem all agree on the challenge, and that work can be prioritized to hit the hardest parts of the problem first (in order to get early warning if the hard parts prove impossible or to inform schedule projections early).
## Timetable
For both a Roadmap Review and a Design Review the **timetable** is as follows:
1. The team should set the stage:
* Summarize in one or two sentences the proposal to be reviewed. This should be agreed between the team asking for review and the reviewer.
* Find a **reviewer** (See “Outputs and Audience” below for the rationale):
- For a Roadmap Review this will be the least senior manager with the ability to address the problem in its entirety.
- For a Design Review this will be an experienced engineer in the problem domain outside of the team willing to put time into the problem. As engineers don’t review their own code they shouldn’t review their own designs. The principle is to get the most informed and least biased feedback possible.
* Optionally, for larger reviews, find someone to **chair** the review. This can facilitate the process (i.e. scheduling the meeting, managing the clock, ensuring minutes are taken, etc.) and enables the reviewer to concentrate on the review itself. The rest of this document uses 'chair' for the administrative role. For smaller reviews the reviewer also does the tasks of the chair.
2. The team produces a **Review Packet** designed to document the proposal identified in step 1. The Review Packet includes:
* A lay summary of the problem space that defines a shared language and identifies the key forces behind the problem.
* A list of the groups directly affected by the proposal to ensure that the review meeting includes representatives from those groups.
* A vision of what the project will achieve on completion.
* A brief that explains what the team is proposing. This should read more like an encyclopedia entry than a marketing document. The audience is the people in the review; i.e. this should attempt to plug organizational documentation holes. The documentation process should not be more onerous than is required for the review. The brief should identify:
- the architecture being proposed
- the non-functional requirements, like accessibility, performance, security, and stability
- relevant business goals, such as improving user satisfaction and retention
- scenarios that exercise the requirements/goals against the proposed architecture
- how the proposal handles these scenarios
- what review and discussion of the proposed architecture has already happened (and with whom)
* A competitive analysis suggesting alternatives, costs, and opportunities
* Input by representatives from the affected groups. The team may update the review packet based on this input but is not required to answer all questions and address all comments before the review.
3. The reviewer creates a list of questions to be discussed during the review, possibly from the comments made in step 2. These questions should be:
* Designed to foster productive analysis and discussion
* Made available to the the team prior to the review so they can prepare answers
4. The chair convenes a review meeting. The discussions of this meeting should be carefully minuted.
* The meeting should have a limited attendance to avoid a sprawling unfocussed meeting, but it should include:
- The team making the proposal
- Representatives from the affected groups, taking care to:
+ Select a full range of perspectives
+ Avoid “stacking the deck” with too many people that think the same way
* A suggested agenda for the review meeting:
- Short opening statement by the team on the current state of play
- Discussion of Questions and Answers (see 3. above)
- A step back to discuss if the proposal is right in the wider context
* A lightweight review might not need the input of others, and ultimately, might not even need a specific review meeting.
The review meeting should be a discussion of the issues. Reviewers don't need to make a decision at the meeting. The goal is to understand the issues raised by the question(s) and the background from the review packet, and to add to it wisdom from the people at the review.
A generally positive review is likely to have a number of action items which the team should respond to before the review is complete. The reviewer is responsible for enumerating the action items, validating the answers and ultimately deciding the outcome of the review.
A more negative review may decide to alter the questions, undergo a subsequent review or drop the subject entirely. It is a goal of the process that we discover fatal flaws as early on in the process as possible.
The results of the review (including review packet and reviewer summary) should be published and communicated widely, with the review packet and results archived. Some version of the decision should be publicly linkable, so that it can be the decision of record, to which blog posts, mailing list posts, can refer.
## Review Scaling: Large and Small Reviews
The ideal review process scales well. The same basic system should work for a quick 2 person decision over the best way to design a certain feature, and for a critical organization-wide decision about a path to take. The term “team” is used above although we strongly recommend design reviews for smaller questions. If you feel yourself wondering if some design is best, it should be easy to perform a review.
For particularly small reviews it is possible to merge a roadmap review and design review into a single review, in which case the question is both 'should we do this', and 'should we do it this way'. It is possible to conduct combined reviews where the total time for the review for all participants is 1 working day.
## Rationale: Outputs and Audience
The first two considerations define the goals of the review process and are two sides of the same coin.
The audience of a Roadmap Review is those “up the chain” of decision makers and the audience of a Design Review is those running “down the chain” of implementors.
A Roadmap Review should focus on:
* The problem space
* Large forces impacting Mozilla’s choices in the space
* Competitive analysis
* Opportunity cost
The key outcome is firstly, senior management investing in the problem space and building consensus around a shared reality. Secondly, to provide documentation of both the goals and assumptions of the project to aid later evaluation.
A Design Review should focus on:
* The solution space
* Technical roadmaps
* Milestones and timelines
* Resource allocation
The key outcome for middle management is to understand the value proposition and the path to success. The key outcome for engineers is firstly to understand the design constraints and the tradeoffs between them, which is important to avoid unnecessary re-work when implementation reality causes a re-think. Secondly, to provide a proposal detailed enough to begin an initial breakdown of engineering work for the proposal (although the work implied by the proposal should not be filed prior to the completion of the review).
## Examples
A number of reviews have been completed and can be used as a template for a successful architectural review. [XBL Removal](0007-xbl-design-review-packet.md) was the first example of a design review, [Sync and Storage](0008-sync-and-storage-review-packet.md) was the first roadmap review, however any of the documents on this website with a title that includes "review" could be used.
## Additional Reading
### Multiple Perspectives On Technical Problems and Solutions - John Allspaw - 2017
Describes how Etsy's architecture review process evolved. A solid write-up of the social side of an Architecture Review process in an environment that is likely closer to Mozilla's than many of the other write-ups.
> Fundamental: engineering decision-making is a socially constructed activity
> We called these “something new” things departures. ... the basic idea is that there are costs ... for making departures that attempt to solve a local problem without thinking about the global effects it can have on an organization.
* https://www.kitchensoap.com/2017/08/12/multiple-perspectives-on-technical-problems-and-solutions/
### Architecture Reviews - Grady Booch - 2010
A formal write-up from someone who spends a significant time in Architecture Reviews. I've not found the paper online, but the MP3 is an audio version of the paper (with bonus guitar-riff intro). This IBM process suffers from lack of attention to the social aspects, but is helpful in digging into the mechanics of the review itself.
* https://www.researchgate.net/publication/224132839_Architecture_Reviews
* http://media.computer.org/sponsored/podcast/onarchitecture/onarch-025-v.mp3
### A Toolbox of Software Architecture Review Techniques - Jason Baragry - 2014, 2015
Jason Baragry reviews approaches to architecture review and notes some potential pitfalls.
> Architecturally significant requirements are often hard to identify, the architecturally significant decisions are often not documented, and the way the decisions interrelate is often not easily understood. A significant part of the review process is often teasing these out.
* https://swarchitectonics.blogspot.co.uk/2014/12/a-toolbox-of-software-architecture.html
* https://swarchitectonics.blogspot.co.uk/2015/02/a-toolbox-pt2.html
* https://swarchitectonics.blogspot.co.uk/2015/09/a-toolbox-pt3.html
### Software Architecture Review and Assessment (SARA) Report - Kruchten et al. - 2002
Formal process with input from a number of companies which targets a more enterprise environment than exists at Mozilla, however the process is clearly defined.
> A software architecture is a set of concepts and design decisions about structure and texture of software that must be made prior to concurrent engineering (i.e., implementation) to enable effective satisfaction of architecturally significant, explicit functional and quality requirements, along with implicit requirements of the problem and the solution domains.
* https://pkruchten.files.wordpress.com/2011/09/sarav1.pdf
================================================
FILE: text/0007-xbl-design-review-packet.md
================================================
---
title: Design Review Packet - XBL Removal
layout: text
---
# Design Review Packet - XBL Removal
This is the packet prepared for the [browser architecture ‘design review’ process](https://mozilla.github.io/firefox-browser-architecture/text/0006-architecture-review-process.html). Some of the sources used to prepare this are the [XBL and Web Components](https://mozilla.github.io/firefox-browser-architecture/text/0004-xbl-web-components.html) analysis and these [scripts and visualizations](https://github.com/bgrins/xbl-analysis), including [graphs of the amount of XBL in tree](https://bgrins.github.io/xbl-analysis). Prototyping and investigations that have been done to create this packet are tracked in the [de-XBL meta bug](https://bugzilla.mozilla.org/show_bug.cgi?id=1397874).
**Review Chair**: Mossop
**Question to be resolved:** What is the best strategy for removing XBL usage in the Firefox frontend, and the platform implementation of XBL?
## Summary
**Problem Space**: XBL suffers from most of the same [problems as XUL](https://mozilla.github.io/firefox-browser-architecture/text/0003-problems-with-xul.html) and [more](https://mozilla.github.io/firefox-browser-architecture/text/0005-problems-with-xbl). Primarily, XBL is unmaintained, poorly documented, verbose and brittle, adds [extra development cost](https://groups.google.com/d/msg/mozilla.dev.platform/iBoROFkR9V8/1ndvU9QIAgAJ) to new code like Stylo, and creates a learning curve and training cost for new engineers. In interviews with senior platform and frontend engineers, XBL is consistently listed as a problem.
XBL is deeply ingrained into the Firefox codebase. Based on an instrumented run of the mochitest-browser integration test, Firefox creates approximately 11000 elements with XBL bindings and 6000 elements without a binding [[source]](https://bugzilla.mozilla.org/show_bug.cgi?id=1376546). Inside of mozilla-central there are more than 200 unique binding implementations and 40K LOC in those files [[source]](https://bgrins.github.io/xbl-analysis/).
Further, before legacy addon support was removed XBL removal was considered a non-starter, since many popular addons relied on extending or modifying existing bindings. This has also led to extra cost when deprecating features or doing refactorings on individual bindings.
**End State:**
This would allow us to remove around 14-20K lines of C++ code from dom/xbl (not including tests). It also would move the frontend codebase onto a more standardized tech stack, which will benefit from ongoing platform work and allow for integration with existing tooling. More importantly, the resulting codebase would be simpler and easier to modify, debug and maintain. Performance impacts are expected to be small.
**Stakeholders**: Firefox management, Firefox frontend development team, Firefox platform development team, other apps that use XUL such as Thunderbird and SeaMonkey.
## Brief
We propose a multi-prong strategy for incremental XBL replacement in the Firefox frontend:
* Convert some bindings to JS modules
* Convert many others to Web Components
* Convert a few into unrefactored XUL/JS/CSS
* Flatten the inheritance hierarchy to remove unused bindings
* Handle special cases
### JS modules
There are some bindings that are not actually widgets - bindings that are only used once, or don’t render any DOM. These can be converted to plain JS objects (either inside a JSM or in a script scoped to the browser window) through a semi-automated process. This will also require manual changes to calling code, so they don’t access the objects as DOM Nodes, and adjust to any API changes that were done in the conversion. Here are a few examples of bindings that fit into this category:
* [`<preferences> / <preference> / <prefpane>`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/content/widgets/preferences.xml): Bindings used in about:preferences to control changing preferences from form controls. An outline for how this can be achieved (and a patch that is on its way to being landable) can be found in [Bug 1379338](https://bugzilla.mozilla.org/show_bug.cgi?id=1379338#c6)
* [`<stringbundle>`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/content/widgets/stringbundle.xml): A wrapper for string bundle objects with a trivially different API (i.e. `getFormattedString` instead of `formatStringFromName`). We could either build a new object that wraps Services.strings.createBundle and exposes the same API, or change all callers to use `Services.strings.createBundle` directly. Note that the l20n project already has Web Component support.
* [`<panelmultiview>`](https://hg.mozilla.org/mozilla-central/file/tip/browser/components/customizableui/content/panelUI.xml#l44): Prior art - Mike de Boer’s work on the PanelMultiView included moving the majority of the implementation out of XBL and into a JSM. More details can be found in [this doc](https://docs.google.com/document/d/1D8WSxgyLA13pidT0umTNEmyrBN5njSPGNLsGkyr0oZM/edit).
* [`<tabbrowser>`](https://hg.mozilla.org/mozilla-central/file/tip/browser/base/content/tabbrowser.xml): Much of the Firefox frontend is driven by this binding. It’s also referenced as gBrowser throughout the frontend codebase. It does render some UI below it, but the majority of its ~6000 lines are behavioral, making more than double the size of the next largest binding in Firefox.
* This is currently being investigated as a prototype in [bug 1392352](https://bugzilla.mozilla.org/show_bug.cgi?id=1392352)
* This could be done as a pure JS module, or following the `PanelMultiView` strategy of keeping XBL for construction and destruction but moving the implementation into a JS module
### Web Components
Most of the bindings in Firefox are in [toolkit/content/widgets](https://github.com/mozilla/gecko-dev/tree/master/toolkit/content/widgets), and many of these are self-contained UI widgets that exist at the XUL tag level. For instance, `<image>`, `<label>`, `<toolbarbutton>`, `<checkbox>`, etc. Each has its own methods, properties, internal state, and DOM tree that is currently implemented in XBL. The Web Components spec, specifically Custom Elements, is designed to solve a very similar set of problems, and platform support is actively being worked on. For more detail, see the browser architecture [analysis comparing XBL and Web Components](https://mozilla.github.io/firefox-browser-architecture/text/0004-xbl-web-components.html).
There are some early prototypes demoing this approach using a polyfill in the [preferences UI](https://bugzilla.mozilla.org/show_bug.cgi?id=1392367) and for [`<panel>`](https://bugzilla.mozilla.org/show_bug.cgi?id=1397876). Some [initial performance numbers](https://bugzilla.mozilla.org/show_bug.cgi?id=1387125) from a benchmark comparing the Custom Elements polyfill with the native XBL implementation show that the polyfill is on par with the native XBL implementation for creating 1000 empty elements from JS.
Shadow DOM provides encapsulation, insertion point features similar to XBL anonymous content, and scoped styles. Waiting on Shadow DOM to land and support XUL documents should not be a requirement for most of the widgets, as long as we are willing to live without insertion points or extra encapsulation (which can be leaky with XBL via CSS direct child selectors and `document.getAnonymousElementByAttribute`).
* The basic mechanism is a semi-automated conversion from XBL to JS Classes following the Custom Elements spec format, as [prototyped here](https://github.com/bgrins/xbl-analysis/blob/gh-pages/scripts/build-custom-elements.js). Then an element name is registered on the document (i.e. `checkbox`), and when a new `<checkbox>` is created the converted code runs.
* We should make Web Components support XUL documents instead of converting XUL documents to (X)HTML to gain Web Components support. Supporting Web Components in XUL is much less work than converting XUL to (X)HTML, and it enables us to separate the XBL and XUL concerns, reducing risk. Removing XBL will make it easier to convert XUL to HTML in the future, should we choose to do so; but adding a dependency on XUL to HTML conversion would make it much harder to remove XBL in the present. We should not do that.
* We should do tree-wide binding-by-binding replacements, as opposed to feature-by-feature. This is to avoid 2 copies of each widget, and to ensure that each new Custom Element supports all the necessary features. This may include making tree-wide changes to consumers if for some reason the Custom Element version can’t remain API compatible
* We should complete platform support for Custom Elements
* Add support for XUL documents on platform Custom Elements implementation
* Create a way to load a set of Custom Elements registrations automatically in all chrome documents
* Support non-dashed Custom Element names to avoid a project-wide find/replace of `<checkbox>` with `<firefox-checkbox>`, along with not affecting platform code that looks for certain tag names like `panel`
* We should complete platform support for Shadow DOM to provide support for [bindings that currently use insertion points](https://bgrins.github.io/xbl-analysis/tree/#children) and that continue to need them after a case-by-case analysis
### Unrefactored XUL/JS/CSS
Bindings that are rarely used and insert relatively little content should be unrefactored into their constituent parts (XUL, JS, CSS) and inserted into each bound element.
For example, the `<prefwindow>` binding, which extends the `<dialog>` binding, is used only seven times and inserts relatively little content:
* 4 LOC: a `<windowdragbox>` toolbar containing a `<radiogroup>` “paneSelector” that enables selection of `<prefpane>`s, even though no `<prefwindow>` contains more than one such pane anymore, so this functionality is wholly unnecessary
* 5 LOC: an `<hbox>` “paneDeckContainer” that wraps multiple `<prefpane>`s, which is also unnecessary in our single-pane world
* 19 LOC: an `<hbox>` containing dialog `<button>`s and `<spacers>` that are almost identical to those provided provided by the `<dialog>` base binding
`<prefwindow>` should be unrefactored into its constituent parts as follows:
* `<prefwindow>`s JS should be converted to a JS module that is imported into each bound element’s document (+1 LOC for each document).
* `<prefwindow>`s CSS should be loaded into each bound element’s document (+1 LOC for each document).
* `<prefwindow>`’s vestigial XUL content (`<windowdragbox>`, `<hbox>` paneDeckContainer) should be removed; and its remaining content (`<hbox>` containing `<button>`s/`<spacer>`s) should be merged with its base binding’s content or copied to each bound element;
* The small number (7) of bound elements means that the additional code size (up to 19 LOC for each element, depending on how much can be merged with the base binding’s content) and maintenance cost is worth it relative to reimplementing the binding as a Web Component.
### Flatten Inheritance Hierarchy
We have a lot of bindings, and use inheritance pretty widely. There are likely cases where we don’t need to - the extra inheritance may have been required for legacy addons, or may have had a purpose in tree at one point but do not any longer:
* Some could be detected by looking at the [XBL inheritance tree](https://bgrins.github.io/xbl-analysis), where a binding is the only child of its parent, and the parent is never directly used. In this case the child could be ‘folded into’ the parent.
* The [`<prefwindow>`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/content/widgets/preferences.xml) binding is a potential example here as well - the ability to render out buttons at the bottom of a dialog could be an option passed into the dialog, instead of using the Unrefactor prong above.
* Sometimes a binding is created to override a single property - in this case a new attribute / condition could be added to the parent to encode the behavior change.
* For instance, [`<tabbrowser-tabbox>`](https://hg.mozilla.org/mozilla-central/file/tip/browser/base/content/tabbrowser.xml) is a simple binding can be folded into the existing tabbox module with a new attribute
* Some bindings are attached via attributes in ways that aren’t able to be emulated in Custom Elements (`<panel type=”arrow”>`) binds #arrowpanel instead of #panel. We can generally migrate to a new tag in that case, but in some cases (like panels) there is platform code looking for the tag name, and it may be easier to fold the arrow functionality into the parent binding.
Making this change does not determine a solution for the parent - it would still need to follow another prong to get rid of XBL itself. Rather, this is a way to reduce the number of bindings that need to be converted, and a chance to refactor away some often unnecessary inheritance for those converted bindings.
### Special Cases
Some of the bindings used in preferences (and throughout the browser) aren’t easy to directly translate to Custom Elements or JS Modules. Generally, [bindings that use the [implements] attribute](https://bgrins.github.io/xbl-analysis/tree/#implements) but don't map pretty closely back to an HTML tag are components that may fall into this category. Each requires further investigation and prototyping before recommending a specific strategy, and may require a custom solution. Here are a few examples:
* In-content XBL may provide special challenges due to security considerations (Scrollbars, Marquee, Video controls, Click-to-play plugin UI)
* `<tree>` which implements `nsIDOMXULTreeElement` and has a pretty complicated API
* For cases like this, we should consult with product / UX and make sure they still want the widget. We may have opportunities to simplify or remove the code instead of just copying the current implementation.
* `<panel>` which implements `nsIDOMXULPopupElement`. Initial investigations in [Bug 1397876](https://bugzilla.mozilla.org/show_bug.cgi?id=1397876).
* `<menu>` which opens a native context menu, and may have interaction with XUL overlays
* Bindings which take advantage of CSS changes to switch to another binding at runtime
## Forces on the system
* **Performance** is critical, given our commitment to shipping a fast, responsive browser. Initial benchmarking between XBL and Custom Elements is promising, but XBL has been optimized for certain cases like opening new windows and extra platform support or frontend optimizations could add time to the estimate.
* **Security** is essential for the XBL-in-content replacements, which can’t enable untrusted content to escalate privileges or access private information.
* **Maintainability** is key to improving our ability to evolve the codebase and avoid falling back into a technical debt trap.
* **Flexibility** is important to ensure that XBL replacements support the needs of the current (and anticipated future) codebase. Note, however, that replacements don’t have to exactly replicate all XBL functionality.
* **Reusability** (and the reuse of existing solutions, like Web Components, where applicable) is valuable to reduce the maintenance burden and the risk of obsolescence.
* **Accessibility** is required to allow as many people as possible to use the browser. Existing UI components support a number of accessibility features that should not be regressed by changes.
## Dependencies
Custom Elements and Shadow DOM have not yet landed, and Custom Elements doesn’t yet support XUL elements (althought this is being worked on in [Bug 1404420](https://bugzilla.mozilla.org/show_bug.cgi?id=1404420).
As a backup plan, Custom Elements does have a polyfill that may enable us to proceed without native support, but the Shadow DOM polyfill is heavier weight and we wouldn’t propose shipping it. We could mitigate this by not using Shadow DOM (see the Web Components section above).
## Competitive analysis
_Alternative 1: Do nothing_
We believe that XBL is a problem worth solving, as outlined in the summary. But, Firefox works as-is and this is a lot of work. So, what if we didn’t do anything about it?
Firefox would continue to function, and we would continue to develop it. Firefox development would be slower, however, because XBL is harder to use and evolve and it doesn’t always [integrate well with the devtools](https://bugzilla.mozilla.org/show_bug.cgi?id=1360072), so we would spend more time implementing features in XBL or working around its limitations.
Gecko development would also be slower, because XBL adds complexity in the frame constructor, among other places. In particular, implementation of Shadow DOM would take more time, since it would have to account for situations in which both a shadow DOM and XBL are active at the same time.
Removal of the old style system will be delayed until Stylo adds support for XBL, which means that Mozilla will continue to ship Firefox with two separate style systems until that time (adding complexity and footprint).
_Alternative 2: Get rid of simple cases, continue to use XBL for the harder cases_
If we identify the simpler bindings and migrate them with one of the proposed prongs, we could leave the more complex bindings in tree. This would narrow the scope of this proposal. However, it would introduce a complexity cost of having multiple approaches to UI components in tree, and wouldn’t allow us to achieve the end goal of removing the XBL implementation.
_Alternative 3: Do more of the work a window at a time vs a binding at a time_
The proposal we suggest is ‘horizontal’ replacement - that is, replace each individual binding entirely, across the tree. Another approach would be a ‘vertical replacement’ - that is, replace all bindings in a window, one window/feature at a time. For instance, convert the entirety of about:preferences, but don’t worry about using the converted bindings in other features. The drawback of this approach is having two implementations of a binding until every window has been complete, and keeping these in sync would require ongoing work.
_Alternative 4: Rewrite the frontend using a different stack_
Instead of converting XBL bindings to JS and Web Components, we could rewrite the frontend (probably feature-by-feature) using a modern library like React. There are some advantages to this approach, including better developer ergonomics (which improves developer efficiency), better application architecture (which improves application quality), and ecosystem effects. But a rewrite would take much more work, entail much greater risk, and entrain a separable set of concerns (in particular, XUL replacement).
We do think it's reasonable for teams to consider rewriting individual features using a library like React when other factors justify a rewrite, for example in the case of [debugger.html](https://github.com/devtools-html/debugger.html). And it may eventually become reasonable to undertake a complete rewrite of the application. XBL removal may make that easier. But we think XBL replacement is worthwhile enough by itself to justify pursuing a lower-risk strategy for its removal.
_Alternative 5: Convert the frontend to HTML (removing XBL in the process) without rewriting it_
We could remove XBL while converting the frontend to HTML—like alternative #4—without rewriting the frontend in the process. In this case, we’d convert XUL to its equivalent HTML, conserve the existing JS/CSS implementation, and convert XBL bindings to Custom Elements, JS modules, etc. as we’re proposing to do in this document.
Doing this would achieve both XBL and XUL replacement, and its cost of XUL replacement would be less than the cost of completely rewriting the XUL implementation in alternative #4.
But the cost of replacing XBL would be the same as the cost in our proposal. And the risk would increase with the increased scope. Also, this approach is quite similar to replacing XBL and then replacing XUL, modulo any cost of supporting Web Components (and any other necessary web technologies) in XUL.
Finally, while there is relative consensus that XBL should be replaced, XUL replacement is more controversial. We think it makes more sense from strategic and risk standpoints to separate these concerns, tackling each of them in turn, starting with XBL.
================================================
FILE: text/0008-sync-and-storage-review-packet.md
================================================
---
title: Roadmap Review: Sync and Storage
layout: text
---
# Roadmap Review: Sync and Storage
**Firefox should wow users with a smart personalized experience that's available seamlessly across services, apps, and devices. User data storage and syncing are vital to this vision.**
We propose to reframe user profile data as a pillar of the Firefox ecosystem, rather than continuing to allow our storage and syncing story to emerge piecemeal from the implementation details of each new product feature.
User data becomes a deliberately designed service layer on which many different product experiences are built. We foster through design reviews the long-term importance of user data. And we invest technically in a more unified, extensible, flexible, cross-platform suite of storage and sync solutions that support a coherent set of approaches to storage.
## Review question
Is this roadmap proposal congruent with Firefox's strategy?
## Document status
* 2017-11-20: Go-ahead given.
* 2017-11-09: Review successfully completed. Follow-up questions underway.
* 2017-11-03: Sufficient contributions received; sent to the Reviewer.
## Roles
* **Reviewer**: dcamp
* **Chair**: jwalker
* **Proposers**:
* rnewman, Browser Architecture
* rfkelly, Engineering Lead, Firefox Accounts
* markh, Sync engineering
* emily, Browser Architecture
* adavis, Product Manager, Sync & FxA
# Lay Summary
Firefox, and the other new experiences that we want to create, rely on easily recording, combining, syncing, extending, and repurposing user data.
Desktop Firefox’s data is spread across [45+ different stores using 10+ different storage technologies](https://github.com/mozilla/firefox-data-store-docs), and we keep adding more.
In general, these stores and technologies:
* Were not designed for syncing, for reuse on other platforms, or for their data to be repurposed.
* Are not straightforward to extend, particularly those connected to Firefox Sync. Migrations are risky and time-consuming, and [downgrades are a huge concern](https://bugzilla.mozilla.org/show_bug.cgi?id=1404344).
* Are silos: it’s hard to connect them to each other to provide insights into data, which is the motivation for initiatives such as Activity Stream and Context Graph.
* Have no baseline of expected storage capabilities: documentation, full-text search, backup/import/export, basic versioning/migration, asynchrony, atomicity, caching, performance measurements and optimizations, _etc._ are all built from scratch, if they are built at all. Each store repeats the same fundamental mistakes and has its own quirks.
* Provide inconsistent support for security and privacy UI settings, leading to an [inconsistent and non-intuitive user experience](https://docs.google.com/document/d/1QJxHQ4GqziUGUiyaF5oP7-cA3hkyTamY33VvHY3vBrM).
* Are limited to a single platform. Desktop’s storage investments (_e.g._, Places, form history, session store), are hard to use elsewhere in the ecosystem, which reduces the value of those investments. No code or storage formats at all are shared between our iOS and NMX projects and desktop; even Fennec reimplements (for good reason, but at some cost) stores that exist in Gecko.
Firefox Sync itself is incomplete, in every sense:
* It doesn’t sync enough data types.
* Types that sync aren't complete, and/or lose data when synced.
* We have almost no consistency across platforms: each implementation syncs different data, in different ways, to different storage.
* Access control is an all-or-nothing affair; the auth tokens and encryption keys needed to read synced bookmarks will also grant read/write access to synced passwords.
* It is hard to improve the protocol itself to remedy these.
Our current technologies are ill-suited to even our current sync and extensibility needs, let alone our predicted future needs.
Moreover, we have no off-the-shelf solution to adopt for new products or features that want to avoid these issues: the status quo is maintained. Teams either do significant amounts of redundant work to bootstrap new storage for each feature and for each platform, to add sync support, and to reach a baseline level of functionality, or they neglect platforms and functionality.
These problems are already limiting factors to product development. Recent examples:
* Form autofill is shipping desktop-only. Desktop-only sync took two months of work and an entire work week to build. Android will, at some point, blindly round-trip this data but not use it. There are no plans for iOS work.
* AS found it extremely challenging to add data to Places, and adding even one additional field to Firefox Sync took 9+ months to trickle through three platforms and negotiate backwards compatibility constraints. More ambitious data recording, particularly for speculative features/experiments, is considered unreasonable, time-wise, within the framework of Firefox Sync: time-to-market is too long.
* The nascent support for containers in Firefox — partitioning of history visits into work, personal, _etc_. — has no clear path to integration with Sync.
* Few features are added to Sync itself: they need to be implemented three times (once for each major platform) and significant changes run into backward compatibility issues.
* New mobile and other experiences that want to integrate with FxA-stored data face significant data/sync costs:
* The NMX team forked a [separate simple read-only Android library](https://github.com/mozilla-mobile/FirefoxData-android), and [an unfinished iOS equivalent](https://github.com/liuche/FirefoxAccounts-ios), to pull data directly from the Sync server; Sync is complex, and extracting the existing Firefox code into separate libraries required expertise that was not present in the team. (We now have **five** separate FxA and Sync implementations.)
* Firefox Rocket doesn't reuse Firefox for Android's storage, and has no path forward to syncing.
* A Servo-based Android AR/VR browser is in need of data storage solutions, but we don't have a solution that we can give them.
* New mobile and other experiences that generate their own user data have no established patterns or guidelines for storing it, and no existing path toward making that data available to the broader Firefox ecosystem.
* The Lockbox project burnt a lot of early development cycles trying to decide what to use for backend storage — Sync, Kinto, or something else entirely. Their MVP currently does purely local storage.
* Project Hopscotch will store its collected user data in Google's [Firebase](https://firebase.google.com/), where it is siloed away and unable to be repurposed or shared with other Firefox experiences.
Stakeholders are:
* Sync, both client and server, across all platforms. The Sync team is motivated to improve Sync and to build better systems, but is only staffed for incremental work, and has neither time to assist with more than a handful of one-off integrations (_e.g._, form autofill) nor significant leverage to direct the design of storage systems to reduce that workload.
* Activity Stream, across all platforms. AS is a significant short-term new consumer of user data, and a long-term generator of reusable data. Delivering a good AS experience requires capturing new data and going far beyond the current capabilities of Sync and Places, but the team lacks the leverage or expertise to make those changes.
* Existing storage teams, responsible for maintaining and implementing existing stores. Some stores lack owners. Most of these engineers don't directly experience the costs borne by the Sync team and engineers working on other platforms.
* Existing product teams. Product managers own features across platforms, but the implementations and capabilities of those features differ widely.
* New product teams and ET explorations wishing to use and collect user data. These teams must build storage and syncing from scratch. Extending and integrating with existing data is challenging.
## Brief
Altering the course of 10+ years will be a multi-step process.
_We propose expending effort to build replacement tools and systems that meet current and known future needs, supporting in-development Firefox features, and migrating existing Firefox data stores as it makes sense to do so._
Rather than just building single-platform MVPs, we hope to encourage engineering and product managers to consider the total cost of cross-platform features and integration, and shift incentives to encourage engineers and product managers to drive towards consolidation — make the right thing easy, so that features get syncing, backup, extensibility, and fast querying without significant new work.
Mozilla needs a _culture that places importance on user data_ — more forward-looking data modeling, implementations that are built to sync across platforms, and data that can be repurposed. One path to doing so will be the architecture review process itself.
We expect technical work to include **designing and building a durable and scalable end-to-end sync and storage system for user data.** This system should handle evolving data without the involvement of Sync engineers, without locking out clients on other platforms or releases, and without losing server data. [Log-structured data](https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying) is the industry standard for distributed systems with growing data and multiple consumers, so we expect this to be a key part of the solution.
We've [captured some detailed requirements for sync and storage](https://docs.google.com/document/d/1XOJhW-qf4iIyGHwiSLgWEfXYDWLy44So5L8CdwH6pcg), and will continue to do so.
This is a relatively large piece of work. Mozilla’s needs are fairly unique: _e.g._, client-side crypto, disconnected writes, years of activity data stored on the client, and working without an account. This means that although the conceptual framework has been well understood for many years, off-the-shelf solutions tend not to be suited to our needs. (Chrome Sync itself has a similar implementation to Firefox for iOS or 2017/2018 desktop Firefox, and is presumably staffed to match the costs of this high-overhead approach. Google defaults to non-private approaches.)
Getting this right unlocks the future of user data.
Technical work will involve:
* Designing client-side APIs to encourage features to record data in a way that meets the needs of other components, including Sync.
* Designing one or more storage systems (including client, server, and documented protocol), supporting those APIs, that meets the captured requirements for sync and storage, including straightforward evolution of data.
* Building SDKs and other tooling to help new mobile experiences quickly integrate with existing user data, and store and share new user data of their own.
* Building client-side data pipelines to also meet the needs of application features. New storage systems must meet the performance requirements of the product.
* Exploring technologies and patterns that make it easier to connect UIs through to storage.
The Sync and BA teams have already begun exploring prototyping parts of this puzzle in an incremental way: in Q4, building a proof of concept as an example of an event observation and capturing infrastructure, materializing a view for querying. We expect this POC to inform our implementation plan.
We also expect to build cross-platform **storage and sync implementations partly or wholly in Rust**, to build a future in which products **throughout the Firefox ecosystem** can share fast, reliable code instead of reinventing the wheel. With current staffing levels we have a hard time keeping our multiple codebases interoperable, let alone growing features at a healthy rate. Emily has [already made progress](https://github.com/fluffyemily/cross-platform-rust) on the groundwork for cross-platform Rust libraries.
We will **continue work on documenting Firefox’s user data and publishing** [**our findings**](https://github.com/mozilla/firefox-data-store-docs). In the course of preparing this proposal we discovered that essentially none of our storage systems are documented, sometimes to the point of not knowing what a file or database was for, and often not being able to tell whether a file was obsolete. We don’t even know which prefs will exist in about:config. It is hard to make good decisions without information.
We will **continue to help guide new data-centric systems** — including Lockbox and ET work like Foxy — towards a cooperative future that builds capabilities for our ecosystem strategy.
And we will guide the **incremental consolidation of existing Firefox data stores into more capable systems**. This might begin by building or buying replacements or abstractions, so the number of stores will grow before it shrinks. For example, libpref is ill-suited for storing data for front-end features; we should provide a durable, syncable store that meets their needs, and migrate features away from libpref, ultimately allowing it to be replaced by a simpler and more performant system that better meets its “pref-shaped” needs. It’s not yet reasonable to pin a number on how many stores we _should_ have, but it’s almost certainly less than 45.
Some of this work will involve measurement work on existing systems or product features; Firefox currently lacks a holistic view of its storage footprint and performance characteristics. We expect to work with product management and engineering to improve this.
We expect initial staffing for these efforts to be low. Research and prototyping work (much of it not parallelizable) needs to be done for new storage, more exploration of Firefox needs to be documented, and the groundwork for Rust components needs to be completed. 3–4 FTEs is appropriate.
In 3–9 months we should be in a position to evaluate a number of more concrete directions, gradually committing more headcount to accelerate progress as opportunities become actionable.
We can call out some representative product end states of this effort:
* New mobile apps can import a published library, with **shared code across platforms**, and use a standard FxA OAuth flow to get access to all or part of a user's synced data **without reinventing the wheel**.
* New product features and **experiments** can store new data through a simple, homogeneous API, and have that data automatically accessible from other Firefoxes and in mobile apps, **without writing significant new syncing code** and without worrying about version compatibility. Product managers should feel that they can confidently explore and iterate on product experiences.
* Users can alternate between release channels or move through a series of experiments using the same profile without losing data. **Most changes will be downgrade-safe**.
Effort subsequently invested in consolidation should gradually reduce ongoing cross-platform maintenance burden, which — after the initial expense — should free up engineering resources for more useful product work.
## Alternatives
### Do Nothing
The main alternative to this roadmap proposal is to do nothing.
There’s no immediate additional cost to doing so, beyond, as Alex puts it, "shipping shitty products, really slowly!"
We will continue to bear the high cost of maintaining and extending Sync, we will continue to have a broad array of undocumented and inconsistent storage systems, Sync will continue to be a lower-quality feature than we want, and as a result we will continue to offer inconsistent and incomplete experiences across our products, harming mobile adoption.
We will be largely unable to offer Context Graph-like features on top of existing user data. Telemetry data and Pocket will thus be the foundation of Context Graph. Activity Stream will soon face significant difficulties in storing and syncing new data; ultimately they will end up building an _ad hoc_ event storage system, and will collapse under the effort of replicating that work on iOS and Android where staffing is limited. New agent projects and mobile browsers will reinvent their own non-syncing storage, slowing development and convergence. Prototyping and experimentation will continue to be costly, particularly when integrating with or extending existing data.
### Bottom-Up
Another alternative is to allow this work to happen bottom-up. We are skeptical of this possibility: evidence suggests that engineers have neither incentive nor leverage to tackle cross-component work like this. [Conway’s Law](https://en.wikipedia.org/wiki/Conway's_law) applies. We are trying to make our architecture decision making clearer, not force it under the radar. Worse, this encourages unsustainable heroism — if we think improvement is necessary, we should produce a supportive environment for it to occur.
### Rebuild Whole Products
A more ambitious alternative is to construct new products in the right way — up to and including constructing a new browser. Mozilla has no plans to ship a Servo-based browser as a replacement for Firefox, so this strategy certainly doesn’t fly on desktop. It doesn’t avoid cross-platform costs, as discussed above. And — more significantly — simply building a new product is no guarantee that the work will be done differently. Servo is a very modern renderer, but without incentives to make other choices, it has already started down Firefox’s path with its prefs storage. Zerda/Firefox Rocket's initial data stores are bespoke and not syncable, despite having access to Fennec's code, because that team's incentives and the costs of reuse don't align.
### Improve Firefox Sync 1.5
We could attempt to invest in Firefox Sync 1.5 to try to make it incremental, extensible, durable, etc., rather than exploring alternative solutions. In the last 7+ years there's been enough architectural motivation to design Sync 2.0, Kinto, QueueSync and TreeSync (the latter two during the short-lived PiCL project). This, and recent experiences with adding data to Sync 1.5, suggest that we’ve improved the old Weave architecture as far as it can go. Additionally, we are not staffed to grow all five Sync client implementations in parallel.
## Competitive analysis and simplifying opportunities
Some of the requirements we've captured, and some aspects of the situation in which we find ourselves, might be malleable.
Some of these requirements constitute strategic choices: _e.g._, end-to-end encryption offers some strong privacy guarantees, but impacts recoverability and access by other services. The set of such strategic choices makes up a space of solutions. We expect the outcome of this roadmap to make progress towards defining a complementary set of points in this space.
In this section we briefly examine some of the choices available to us.
### Encryption and servers
Storing data in a way that prevents Mozilla from seeing cleartext means that the server can't help much with conflict resolution, versioning, data access, _etc_.
Chrome makes a different strategic choice, and is thus able to show your history on the web. [Realm](https://realm.io), a cross-platform database that competes with Core Data, offers automatic synchronization of its 'realms' between clients via a central Realm Object Server, with all data in cleartext.
A solution that could rely on cleartext data would be able to shift logic from the client to a single server infrastructure, which would increase agility and simplify the client.
The end state of this shift is something like Pancake or Facebook: a thin client with a rich server application and a server-canonical data model. This avoids synchronization entirely.
A number of systems use encryption at rest, but not end-to-end encryption. This allows for simpler web access, sharing, and recovery. Dropbox uses this model for files. This approach is not resistant to subpoena or intrusion.
This proposal assumes that end-to-end client-side encryption is table-stakes for some or all of Mozilla's applications, and so we can't disregard this requirement.
### Data-rich client applications
If we were to commit to storing less data on clients, and changing the set of stored data less frequently, then the cost-benefit analysis of this problem would shift, perhaps to the point that significant engineering investment should be avoided. To an extent this is similar to the alternative "Do Nothing": we currently try to avoid growing or changing the data we store, and we change Sync very infrequently.
Chrome is an example of this: there is no evidence of Chrome moving towards anything more than its rudimentary history and bookmarks storage.
This proposal assumes that there is product desire to leverage more user data on clients, not less, and eventually we will have to tackle the issues outlined in this proposal.
### Offline writes
If we were to make significant changes to the relationship between users and accounts, and between devices and servers, we could simplify the problem of syncing. For instance, if we linked all profile data to a Firefox Account from first opening Firefox, then repeated large-scale merges can be avoided. If we require all devices to be connected in order to record data (providing high availability for server storage), then we can turn synchronization into a distributed write. These are still non-trivial engineering problems, but they're different problems.
Most database systems we've examined, including Realm, Dropbox Datastore, CouchDB/PouchDB, and others, allow for offline writes.
This proposal assumes that our ability to tie writes to server resources is limited, for both technical and product experience reasons, but we have been thinking about this possibility.
### The definition of ‘data’
Storage systems might store some or all of documents, small blobs (_e.g._, icons), large blobs (_e.g._, downloaded files), independent structured data (_e.g._, emails), interrelated structured data (_e.g._, social graphs or bookmark trees), measurements/events (_e.g._, sensor data), and more.
A system optimized for document-structured data will typically scale horizontally but lack features like cross-document transactions. A system optimized for graph traversal might not handle blobs well.
Choosing carefully which kinds of data to support in which systems makes it easier to meet requirements.
Tools like CouchDB and MongoDB aim to provide scale and simplicity in syncing by deferring conflict resolution, identity, and validation to application code, themselves working with a more simplistic data model. Dropbox's [decommissioned](https://blogs.dropbox.com/developers/2015/04/deprecating-the-sync-and-datastore-apis/) Datastore API treats datastores independently and [merges records field-wise to resolve conflicts](https://blogs.dropbox.com/developers/2013/08/how-the-dropbox-datastore-api-handles-conflicts-part-two-resolving-collisions/); that's a good set of tradeoffs, but it leaves identity merging, data evolution, and validation to the app.
## Strawman Roadmap
We expect to gradually firm up our understanding of concrete solutions and shipping vehicles over the course of the next two quarters: Phase 1.
### Phase 1: building confidence (2017Q4–2018Q1)
We have a good idea of the characteristics of a solution for structured user data that allows for the data evolution, experimentation/iteration, encrypted sync support, and reuse that our products need.
We have identified two main remaining puzzle pieces that the Browser Architecture team will explore: the client-side data pipeline that preserves existing query interfaces, and the concrete sync protocol that moves data around.
If we can't tackle those, then there is no mainstream path forward for this part of the solution. If we're wrong, we want to find out sooner rather than later.
Our first step, then, is to exercise those two parts by prototyping each, iterating as needed. If we fail, then we will reassess our approach. This is the cheapest and best route to failure.
The Sync team has a related plan for this timeframe to reduce uncertainty around Rust-based cross-platform dedicated storage APIs that wrap generic storage. This will build confidence in our ability to consolidate implementations across desktop and mobile while meeting the needs of product engineering teams, and help to determine the developer experience linking application and service.
In parallel, we can move forward with independent cleanup work: making APIs async (post 57 freedom), incrementally improving and teasing apart existing problematic stores (_e.g._, libpref), work being proposed elsewhere around process separation for storage, etc. These efforts will set the stage for productization, but also make the world a better place regardless of the success of this particular effort.
A successful outcome for this phase is one of: fast failure; a revised roadmap or strategy that reflects new knowledge; or a proposed set of design options that can be reviewed with an eye to tackling Phase 2.
### Phase 2: attack the hardest thing, fail fast (2018Q1–Q3)
The largest single point of risk beyond basic functionality is addressing Places.
Places' history store is a large source of value, Places-linked data is core to Sync, and it's also a good test case: a high-volume store of structured user data with existing complex query consumers.
If we can't tackle Places, then there's much less value in proceeding.
We think the quickest and lowest-risk approach is to intermediate some writes to Places, turning parts of the Places database into a read-only materialized view. This stress-tests the client-side data pipeline, and allows us to explore syncing new data (_e.g._, containers, alternative representations for keywords and tags), recovering Places data from the log, _etc_., without disturbing existing consumers.
At this stage we hope to find and fix scaling and performance issues before doing further work. If we fail here, the process has been both informative and relatively cheap.
As this phase proceeds we'll design and build production server capacity for the sync system, begin to address fit-and-finish concerns like push notification integration, and continue to assess requirements from new product efforts.
### Phase 3: consolidate (2018Q3–)
If we're confident in our new sync implementation, can use it from apps, and can handle Places-level scale, then we can proceed with consolidation and simplification to add value to Firefox. We can fan out: pick suitable storage (_e.g_., data that's high priority for syncing, an old store that's not owned by an existing team, …), make sure it has an async API, and port it on top of the log, following the example of Places.
Early in this phase our documentation will go from adequate to great, we will place heavier emphasis on the developer experience and UI programming integration (_e.g._, GraphQL), and we expect to see further adoption in mobile apps and on the server as the cost of accessing Firefox data decreases.
We can also assess the urgency of exposing this kind of storage — both access to Firefox's data, and dedicated storage — to WebExtensions.
These additional efforts will likely be assessed and planned closer to the time; it would be unrealistic to estimate resourcing or timing when priorities are likely to shift.
## Resources
[Requirements capture for Sync](https://docs.google.com/document/d/1XOJhW-qf4iIyGHwiSLgWEfXYDWLy44So5L8CdwH6pcg)
[Scoped encryption keys and access control for FxA](https://docs.google.com/document/d/1IvQJFEBFz0PnL4uVlIvt8fBS_IPwSK-avK0BRIHucxQ)
Vision statements
* [Ryan Kelly](https://docs.google.com/document/d/1MrXNcVzDDYyGSvud6CvVSoPGNyvpJvHGSDpRwIcalQc)
Blog posts
* [Thinking about Syncing, part 1](https://medium.com/@rnewman/thinking-about-syncing-part-1-timelines-7f758e2bd676) (2017)
* … [part 2](https://medium.com/@rnewman/thinking-about-syncing-part-2-timelines-and-change-90a0964f10f3)
* [Syncing and storage on three platforms](https://160.twinql.com/syncing-and-storage-on-three-platforms/) (2015)
* [Different kinds of storage](https://160.twinql.com/different-kinds-of-storage/) (user agent service) (2016)
## Thanks
Thanks to our readers: myk, asuth, mak, grisha, tspurway, snorp, overholt, ckarlof, wennie, selena, gps, trink, hildjj, and plenty more.
## Questions
**Are you going to require this everywhere?**
No; it's not applicable to all problems, nor does the cost-benefit calculation always play out. Decisions should be made based on suitability and value: _e.g._, to make some data available to other apps or platforms, to bring a store up to a feature baseline, or to integrate two data sources to support a new feature.
**Does this mean we’re using Mentat?**
Not necessarily. Our roadmap so far involves answering questions about how to provide capabilities that we think are important for Firefox's strategy. It's possible that parts of Mentat will evolve into parts of the solution — after all, it's an embedded Rust log-structured store — but a solution will be integrated with other parts of the Firefox ecosystem, not just a database.
**Does this restrict the design of Sync and Storage at Mozilla?**
Not really. We expect FxA identity-attached services to proliferate in time, and we expect there to be justified divergent approaches to storage, too — here we are mainly focused on structured user profile data, but elsewhere we are also _e.g._, helping to shape a redesign of libpref. What we hope to happen is that we can provide a small spectrum of solutions that are better choices than building yet another store from scratch, and that are dramatically _less_ restrictive than the current situation, achieving convergence at a healthy rate.
**Does this help ET?**
We hope so. Having reusable, portable components and data opens doors to exploration and development.
**How do you plan to scope out the problem of scaling the service?**
In short: by finding sets of properties that are beneficial for both the requirements we've captured for storage and syncing and also for management of a hosted service — including scalability — and designing a solution that reflects those properties.
This is something of a design question, but we have some idea of where we might go (we're initially proposing log-structured storage) so we can answer in brief. In short:
* Independent users and divisible data. There is little need to coordinate between encrypted storage for each user. Moreover, by making the log, entities, and other attributes meaningful concepts in the system, we can make the data for each user amenable to sharding along these axes. This allows us to scale horizontally.
* Mostly immutable data. Unlike a write-in-place system like Firefox Sync, which supports its constantly growing set of data by churning in place, we would like the majority of our storage work to be appends. The exceptions are well-defined operations that introduce barriers (like rollups, snapshots, or account deletion) and well-defined operations that rewrite parts of history (like Forget). Even with those exceptions, that's a healthier workload than we currently support in Firefox Sync, where a big pile of records is both constantly added to and constantly subject to random replacements — it makes the server simpler and the workload more predictable, and allows for cheaper long-term storage.
* Exploiting modern advances like push to reduce costs for common cases: improved responsiveness can eliminate a subset of conflict scenarios, which reduces load for both client and server.
Questions from ckarlof (not a lot of time to answer these, so a little brief, but better than nothing!):
**What are the outcomes we're trying to achieve here and how do you know we've achieved them? What are your key metrics of success?**
_I see some "representative product end states", but I feel these need to be more prominent in communicating your vision of success._
In addition to the concrete definitions of success in each phase, we'll know the overall effort has been successful if:
* The organization displays a culture of holistic thinking around user data across the Firefox ecosystem.
* Product managers feel more empowered to drive experiences that rely on new, integrated user data.
* The baseline of storage capabilities is higher, with consequences for the consistency of Firefox's UX (see, e.g., Wennie's doc).
* The cost of integrating new syncable data is markedly lower than it is today.
Figuring out how to concretely _measure_ our success is a task for after this review.
**How many different complete options are you considering to achieve your proposed outcomes?**
_I feel there's really only one under consideration, but I feel having one or two alternative approaches to reach your goal, even if they're straw dogs, would help folks understand the merits of your favored approach better. Your competitive analysis is close to this, but some of things you're comparing against aren't solutions to reach your vision. They're compromises. Sell me on your vision._
We're concerned with agreement that the situation is as described, that our vision is desirable, and that the situation can be reasonably improved. The alternatives listed above are 'roadmap alternatives', rather than design alternatives, precisely because those are different paths that the organization could take other than investing in a suite of technical solutions.
We have some understanding of characteristics that we feel are important to forward-looking technical solutions, and the next steps on the roadmap are focused on exploring those characteristics. The results of those explorations will dictate the concrete approaches we consider.
**For the approach you're suggesting, what are they key choices in that approach that you want us to understand? Using Rust to implement a shared library is one such strategic choice. Are there others worth surfacing?**
The proposal to use Rust is essentially an assertion of two things: that supporting multiple applications on multiple platforms is part of Mozilla's future; and that native code is the right way to do that in this instance. Rust is, we will admit, an opinionated choice here.
There are other strategic/organizational opinions that lie beneath this document:
* That some organizational coordination of storage — including centralized reusable implementations — is valuable, both to individual features and to the organization as a whole.
* That totally separating syncing from storage doesn't work, and that pushing onto feature developers the burden of understanding syncing and other consumers, without tool support, is costly. Abstractions are needed.
**What would have to be true for your approach to be a winning one? What are your riskiest assumptions here?**
_I'm looking for a prioritized list of questions that need answers and assumptions that need to validated, an order to tackle them in, and how you're going to tackle them (ideally, as cheaply and as early as possible)._
Phase 1 and Phase 2 capture what we think are the most significant technical risks: can we build a system that more closely meets [the requirements we've captured for sync and storage](https://docs.google.com/document/d/1XOJhW-qf4iIyGHwiSLgWEfXYDWLy44So5L8CdwH6pcg/edit), in a way that serves our strategic needs (_e.g._, around reusability), and maintain syncability, performance, and the developer experience (including the preservation of existing interfaces for as long as makes sense) — can we build it? can we make it sync? can we make it scale? can we get people to understand it?
We'll break these down into more detailed work items and questions as we go about working on those phases.
From a product perspective, most (if not all) of the following would have to be true for this to be a success and to meet the needs of the engineering teams we’ve spoken to:
* We have a sufficiently accurate understanding of the needs of Mozilla engineering teams.
* We have a sufficiently accurate understanding of our users’ needs and problems.
* Teams have a desire/need to store new data.
* Teams have an increased need to more easily access existing stored data from other teams.
* We can successfully collaborate across departments.
* Users trust us with their data.
* Engineering teams trust us with their data.
* Existing storage is insufficiently easy to extend or scale.
* Engineers agree with the shortcomings of the current situation.
* Engineers would stop using _ad hoc_ solutions to create new products.
* Engineers are OK with having less control of their integration due to being limited to APIs
* There is enough of an overlap between all teams that would allow us to have solutions that fit all.
* New smaller mobile products won’t need to pull down all of the data locally since that can be a battery and storage drain.
* Mozilla wants to make personalized data-centric features (Firefox eco-system?)
* Users want a personalized browsing experience.
* In the case of making it ourselves, nobody has a working solution that we could use or fork which would allow us to achieve similar aspirations.
* We have the technical expertise to build a new storage system and cloud storage infrastructure.
* Our solution would provide increased flexibility to engineering teams in terms of adding new data types and changing existing ones.
* For new projects, that we are faster and easier to integrate than _ad hoc_ solutions, ideally enabling new projects to more rapidly pop-up across the company.
* We still want to encrypt all of our users data and work within those constraints.
* That we can satisfy the desire for experimentation within product teams and their current lack of data storage solutions that enable them to do this.
================================================
FILE: text/0009-type-safety-systems.md
================================================
---
title: JavaScript Type Safety Systems
layout: text
---
# JavaScript Type Safety Systems
JavaScript is an untyped language and as such is susceptible to type safety bugs
where a type other than that expected is used by mistake. For example trying
to add a number to a string and expecting a numeric result.
JavaScript type safety systems work as an extra layer of code syntax on top of
the raw JavaScript. Some kind of compiler or checker is used to verify that
type assertions made in the code are met allowing bugs to be detected.
Recent studies have suggested that using type annotations in JavaScript can stop
as many as 10% of bugs from entering the product.
## Flow
Flow is a type safety system developed by Facebook. When no types are provided
in the source code it attempts to infer types based on variable assignments.
Flow comes in two forms. The more common form involves adding types as an
extended JavaScript syntax. This form must be removed by a transpiler at build
time. The other forms are comment based, one only supports function argument
typing the other supports any typing but is IMO quite ugly.
Flow frequently finds bugs that aren’t errors and so would need work to correct
before any typing can be used with the file. As an example in a file with 4k
lines of code that we can assume to be largely bug free Flow found some 122
errors so one error per 32 lines. Two of them were potential bugs (though
actually inconsequential). The rest were either Flow bugs, cases where Flow was
pointing out a potential issue that could never occur in reality or cases where
we use complications that Flow can’t reasonably be expected to understand.
## TypeScript
TypeScript is a typed superset of JavaScript that compiles to plain JavaScript.
TypeScript files support compiling to an older version of JavaScript supported
than the version written. TypeScript does some type inference so some validation
is performed even before any types are added but less than Flow.
When run on an existing large file in the repository TypeScript found around one
issue per hundred lines of code which is less than Flow found on the same file.
Most of those issues were misunderstood types. None of them were errors that
would cause problems in the code.
## Conclusions
In the case of both Flow and TypeScript significant work would need to be done
in order to use them in existing code. Both suffer from a number of drawbacks:
* Not understanding some modern JS syntaxes
* Not understanding our module system
* Requiring build-time processing leading to difficulties with logging and debugging
It doesn't seem worth the work at this stage to use either system in the main
Mozilla codebase.
================================================
FILE: text/0010-firefox-data-stores.md
================================================
---
title: Firefox Data Stores
layout: text
---
# Firefox Data Stores
## Introduction
There are a large number of data stores being used to hold data across all Firefox instances and platforms. There is currently nowhere that documents all these data stores and how they share information between them. We have 3 platforms, desktop, Android and iOS and each of these platforms shares data with the others. But not all the information is synced completely across all platforms, and the data itself is not stored in the same way everywhere. When starting to think about how we might provide a holistic user data solution for Mozilla, it is important to understand the scope of the problem and the specifics of the data that is being handled.
We therefore need to look at all of the data stores that we are currently using and document their purpose, structure, the kind of data that they hold and how they share that data.
## Summary
The extensive documentation can be found in the [Firefox Data Stores](https://github.com/mozilla/firefox-data-store-docs) repository.
* There are a total of 46 separate data stores in Firefox for desktop. Within these 46 stores, there are at least 10 different storage formats. Very few of these stores are well documented in the code.
* Firefox for Android Java frontend uses 3 stores. It consolidates `places.sqlite` and `favicons.sqlite` into a single SQLite database, `browser.db`. It uses 4 different storage formats, `SQLite`, `JSON`, `XML` & `JS`. Android frontend documents its data stores well and uses a single file to define the schema for all its tables.
* Gecko on Android uses the vast majority of the same stores as Gecko on Desktop.
* Firefox for iOS has 5 data stores. `metadata.db` was created to store metadata for use in Activity Stream and was created separately to make syncing of this data at a later data easier. Due to a change in strategy, this store is no longer in use. iOS uses 3 different storage formats, `SQLite`, `JSON` and `plist`. It also writes credential information to the iOS keychain. iOS stores are not especially well documented, but the schemas are defined in a database specific files which makes discovering the schemas trivial. The exception to this is the reading list store which is defined separately from the others. iOS has very strong encapsulation of storage. The front-end code uses specific, narrow, safe APIs to read and write data, and storage manages sync metadata.
* Desktop stores logins in an encrypted JSON store, `logins.json`. iOS keeps logins as a separate SQLite database, `logins.db`, that uses `sqlcipher` for encryption. Android uses a SQLite database, `signons.sqlite`, for storing logins. `signons.sqlite` used to be used on desktop too, but they were migrated into `logins.json`. Android frontend is currently in the middle of removing the dependency on the Gecko-side store, resulting in a set of login tables inside `browser.db`, but the work is as yet incomplete.
* `prefs.js` is a simple key value file written to disk, but it contains well over 1200 entries. Its design means that changes made are often not written out until shutdown. Therefore, if the shutdown is not clean, any changes made during the session are lost, resulting in potential inconsistency between prefs and state elsewhere in the system.
* Firefox desktop has an entire SQLite database, `storage.sqlite`, that contains no tables. It is used entirely to reference the schema version value and uses that value to create the version number for the Storage directory. The Storage directory contains all of the files and stores used by the app.
* Of the data fields stored by desktop, only 8% are available to Firefox Sync.
* A visual overview of the [fields made available to Firefox Sync](https://docs.google.com/spreadsheets/d/1k9_K7Dc3q2h3SDV0vwjTgJou-ndza6WuobyJ1bbemtc/edit?ts=5977ab9d#gid=1269587388) and how each sync record is implemented on each platform is also available.
================================================
FILE: text/0011-fluent-in-prefs-design-review.md
================================================
---
title: Fluent in Prefs Design Review
layout: text
---
# Fluent in Prefs Design Review
## Introduction
[Fluent](http://projectfluent.org/) is a "family of localization libraries designed to improve how software is translated." Mozilla's localization engineering team (the "engineering team") is migrating Firefox's localization architecture from DTD/properties files to FTL files processed by Fluent, which will enable richer and more accurate translations, runtime locale loading/switching, runtime LTR/RTL and pseudo-locale switching, and a fallback locale. The first target for this migration is the preferences interface (the about:preferences page and its various subdialogs).
The engineering team met with a set of architects and domain experts (the "review team") to discuss risks and mitigations, which constituted a lightweight "design review" of the project's architecture. This document describes the concerns that the review team raised and the risk mitigations that the teams discussed.
## Why Preferences
The team chose preferences because they're text-heavy, include users' names (which exercises Fluent's support for grammatical gender), and because the precision of text matters in preferences to ensure users clearly understand the implications of changes. So preferences are a good way to prove the change to Fluent before adopting it more widely.
Fluent is a big change to the localization process, and the team plans to validate the change by translating just four strings initially (the "name" strings in brand.dtd/properties?) and then verifying those changes across the localization toolchain.
## Chrome Privileges
The review team asked whether there are parts of preferences that don't have chrome privileges, and if that influences the use of Fluent. It seems like the only such part is the Firefox Accounts image (???), which loads over a network and is localized independently of Firefox, so it won't be an issue.
## Search Incompatibility
The search feature of about:preferences is an issue, as the way that it searches preference labels and descriptions is not compatible with the way that Fluent injects text into the page. The teams discussed options and concluded that there is a straightforward way to update the search implementation to be compatible with Fluent.
Specifically, FTL messages support arbitrary attributes, so an FTL message for a searchable preference control could include a "searchterms" attribute containing the text to search, i.e.
```
key1
.label = "click"
.searchterms = " … "
```
Attribute values can include references to other strings within the FTL file, so localizers could avoid string duplication if desired. This model would also enable localizers to customize search terms to include extra keywords that aren't included in other strings related to the control. Finally, this model supports English fallback for searches via the creation of a secondary *document.l10n* object with the `en` locale.
The review team raised some concerns about the performance of runtime searches and suggested that the engineering team consider generating an index from search terms to the controls they match, either at build time or on first search. Some participants suggested being careful to avoid prematurely optimizing, however. In the end, the teams didn't reach consensus on the optimal solution, and further research/experimentation is warranted here.
## Schedule Risk
After validating the process with the first four strings, the engineering team plans to migrate the rest of the strings, but they're aware of the large set of pending changes for XBL replacement in [bug 1379338](https://bugzilla.mozilla.org/show_bug.cgi?id=1379338), and they'd prefer to await those before proceeding to minimize merge hell.
The XBL replacement patch author ([myk](https://mozillians.org/en-US/u/myk/)) and reviewer ([jaws](https://mozillians.org/en-US/u/jaws/)) were both members of the review team, and they noted that those changes should land soon and thus shouldn't create schedule risk.
Another member of the review team noted that another team is rewriting the "delete cookies" dialogs, and the engineering team should coordinate with that team.
## Migration
The teams discussed tools for migrating from DTD/properties files to FTL files. They also discussed the possibility of migrating strings from one FTL file to another, as the difficulty of migrating strings between DTD/properties files is a common frustration with the current system.
The engineering team has developed tools to migrate strings from DTD/properties to FTL files, but they don't yet have a way to migrate them from one FTL file to another. That doesn't regress functionality, but the review team still recommended that the engineering team consider tackling that problem in the future.
The teams also discussed merging the various preferences DTD/properties files into a single FTL file, since their current ontology is out-of-date (because pref categories have changed over time, and the files haven't changed with them), and since there isn't a reason to load only a subset of strings (especially given the new search feature, which needs to search across all of them).
There was also some discussion about the conflict between engineers, who bias toward coalescing "identical" English strings used in different contexts (such as "open") because DRY; and localizers, who separate such strings because they aren't actually always identical in all locales.
Coalescing is considered harmful, and a concern was raised that merging preferences strings into a single FTL file might encourage engineers to do it even more. The teams concluded that this concern can be addressed at the policy/review level.
## String Changes
The review team probed about Fluent support for string changes, and the engineering team explained that support is as good as the current system, which has two ways to modify a string, and work is planned to make it even better by adding a third way.
Specifically, under both the current system and the current version of Fluent, a developer can make a minor change to a string (f.e. a typo fix) that shouldn't affect its translations by preserving the string's key, and they can make a major change to a string that should affect its translations by changing the string's key.
In the future, Fluent will support versioning of a string key via a tag, such that developers can also make changes that affect localizations without having to change the string's key. (There was some discussion about which kinds of changes would fit into this bucket.)
================================================
FILE: text/0012-jsonfile.md
================================================
---
title: A brief analysis of JSON file-backed storage
layout: text
---
# A brief analysis of JSON file-backed storage
Several components in Firefox — including libpref, XULStore, autofill, and logins — follow a general pattern:
- Store data in memory, usually as a JS collection.
- Load that data from disk during initialization.
- Persist that data to disk in its entirety, usually serialized as JSON, at some point after each write, and/or at shutdown.
[JSONFile.jsm](https://dxr.mozilla.org/mozilla-central/source/toolkit/modules/JSONFile.jsm) is a centralized implementation of this pattern. It supports Promise-based asynchronous loading, synchronous loading, and flushes asynchronously, defaulting to 1500ms after each change. libpref flushes after 500msec.
This approach is well suited to data that:
- Changes infrequently.
- Is relatively small.
- Is only ever directly read or written by a single thread.
- Needs to be kept entirely in memory for reads.
- Has a straightforward JSON encoding (encoding binary blobs, such as images, in base64 is relatively inefficient and typically avoided).
- Has limited durability requirements.
Advantages of this approach:
- It's relatively simple, with serialization and deserialization being the only steps.
- Because the file is rewritten frequently, there is some hard-to-measure resilience to gradual file corruption.
- The file on disk is — unless compressed — human-readable and editable, which is convenient for development, test, and support.
- Automatic opt-in whole-file compression is offered by JSONFile.
- The in-memory data can be queried and manipulated synchronously and quickly, assuming the representation is a good match for retrieval patterns.
- Development is 'natural' for front-end developers — get things working with only in-memory data, then add persistence on top.
Disadvantages:
- Versioning is often an afterthought: engineers rarely think to version in-memory data formats.
- Change tracking is almost always an afterthought, thanks to the 'natural' development process: adding syncing later becomes difficult.
- Users feel relatively empowered to edit readable files on disk, which can result in persistent state that was never created by the component's own logic. The use of compressed formats makes this less likely.
- Frequent writes will cause the entire file to be written to disk repeatedly, [which causes complaints about SSD impact](https://bugzilla.mozilla.org/show_bug.cgi?id=1304389).
- The entire file typically needs to be read into memory to be used, which increases memory footprint and can impact startup time.
- Writes at shutdown harm the user experience and don't happen during a crash.
- In order to achieve atomic file writes, the entire file contents briefly exist twice on disk, which could be problematic on mobile platforms for large data sets.
- The only approach to cross-process use is duplication of all or part of the in-memory data, which can increase memory footprint. It's not possible for multiple processes to collaborate on the same data via the filesystem. This is why Firefox for Android still uses the SQLite implementation of `nsILoginManagerStorage`: it allows the Android-side `ContentProvider` code to read and write saved logins without worrying about Gecko, which nominally owns the backing storage, but has a shorter lifetime than the enclosing Android code. Using desktop's JSON-backed storage on Android would require launching Gecko on each sync, using messaging for safe access to the file contents.
- The in-memory object is essentially a non-transactional write-back cache. This has several issues:
- It makes isolation (readers can't see in-progress writes) and atomicity (all writes complete or none do) difficult: we typically mutate the in-memory object directly, and so an exception can cause partial changes to 'commit', and readers will see each change as it is applied.
- Similiarly, timed flushing makes consistency difficult: it's possible for only some of a set of asynchronous writes to be flushed to disk because the flush beat the last few writes, leaving the data inconsistent after a crash.
JSONFile.jsm advises that callers do all of their work synchronously on the main thread to avoid the possibility of concurrent readers or partial writes:
> The raw data should be manipulated synchronously, without waiting for the
> event loop or for promise resolution, so that the saved file is always
> consistent.
This can cause jank.
- Synchronization of data stored in this way takes some careful thought. Syncs typically take hundreds of milliseconds or more, and involve asynchronous network operations, which makes exclusive synchronous access to the in-memory data between reads and writes infeasible.
- There is a tension between durability (that is: writes that complete are permanent) and performance. We typically choose not to flush the file immediately and synchronously after every change, but not doing so leaves a window in which a crash would cause data loss. By default, that window is 1.5 seconds, plus the time needed to write and atomically switch the files. This forces careful consumers to manually flush if they want their writes to stick, which is a bad pattern.
- In-memory objects lack the sophistication of most databases. This leads to front-end features building their own [simple query engines](https://dxr.mozilla.org/mozilla-central/source/toolkit/components/passwordmgr/storage-json.js#296) for [finding records by linear search](https://dxr.mozilla.org/mozilla-central/source/browser/extensions/formautofill/FormAutofillStorage.jsm#1081).
- Similarly, these stores must reinvent (or contribute to JSONFile) their own:
- Versioning and upgrade logic.
- Validation and schema checking.
- Concurrent access patterns.
- Write coalescing/async update patterns.
- Tooling, if existing general-purpose JSON tools (*e.g.*, `jq`) are not sufficient.
- Backup, if any.
- Indexing, if any.
- Datatype serialization (*e.g.*, timestamps).
- Finally, the use of `JSONFile.jsm` is intimately linked to Gecko; it's a poor choice for code that will later need to be deployed on other platforms.
================================================
FILE: text/0012-process-isolation-in-firefox.md
================================================
---
title: Process Isolation in Firefox
layout: text
---
# Process Isolation in Firefox
Randell Jesup, Mozilla Browser Architecture team
## NOTE: this document is a Work in Progress!
## Overview
We’ve recently moved a number of parts of Firefox into separate processes (e10s, multi-e10s, GPU process, compositor, WebExtensions process, GMP). This has produced some serious improvements in both security (sandboxing content, codecs) and stability (restarting GPU process when a GPU driver crashes, etc). This project is to evaluate how much further we can push process isolation to further improve these things.
### Problems:
* We have large processes, running many unrelated items of highly varying security and stability properties. A single bug (including in OS drivers) in many cases will take down either a major part of your tabs, or the master process and by extension the entire browser.
* In a related concern, a single exploitable bug gives access to a large part of the browser. Even if it’s in the Content process, it can give access to ¼ of your tabs, and because Content processes have very wide needs to access data and communicate with the Master process, the possibilities for either sandbox escape or information leakage are quite high.
* Features and capabilities often have code strewn across various parts of the tree, increasing the maintenance cost and risk of unrelated changes breaking them.
There are some secondary benefits we hope to achieve by doing this, such as decoupling parts of the system and providing more-stable interfaces to them, as well as easing some aspects of unit testing. There may be some advantages in build times or cost of maintenance; we’ll see.
There are costs: development time, memory use, and performance all can or will be negatively impacted. Part of this project is to quantify these impacts (and hopefully reduce them) in order to guide the decisions on how far to push this process.
Chrome/Chromium has been doing similar work recently on "[Servicification](https://www.chromium.org/servicification)". This is related to their slowly replacing classic ipc/chromium-based IPC with “[Mojo](https://chromium.googlesource.com/chromium/src/+/master/mojo/README.md)”; a new IPC implementation (and ipdl-like layer) that has improved performance compared to classic Chromium IPC. Note that some ways Mozilla uses IPC might avoid some of the performance costs in Chrome (multiple channels, PBackground and the like, for example) - but we haven’t yet assessed how much overlap there is between Mojo and our additions to Chromium IPC. It may be that with some smaller modifications our use of Chro
gitextract_jxunuj8l/
├── .gitignore
├── CODE_OF_CONDUCT.md
├── Gemfile
├── ISSUE_TEMPLATE.md
├── LICENSE
├── README.md
├── _config.yml
├── _layouts/
│ ├── default.html
│ └── text.html
├── assets/
│ └── css/
│ └── style.scss
├── experiments/
│ ├── 2017-09-06-rust-on-ios.md
│ └── 2017-09-21-rust-on-android.md
├── newsletter/
│ └── _posts/
│ ├── 2017-07-27-browser-architecture-update.md
│ ├── 2017-08-24-browser-architecture-newsletter-2.md
│ ├── 2017-09-22-browser-architecture-newsletter-3.md
│ ├── 2017-10-19-browser-architecture-newsletter-4.md
│ └── 2017-11-29-browser-architecture-newsletter-5.md
├── newsletters.xml
└── text/
├── 0000-template.md
├── 0001-documenting-output.md
├── 0002-extracting-necko.md
├── 0003-problems-with-xul.md
├── 0004-xbl-web-components.md
├── 0005-problems-with-xbl.md
├── 0006-architecture-review-process.md
├── 0007-xbl-design-review-packet.md
├── 0008-sync-and-storage-review-packet.md
├── 0009-type-safety-systems.md
├── 0010-firefox-data-stores.md
├── 0011-fluent-in-prefs-design-review.md
├── 0012-jsonfile.md
├── 0012-process-isolation-in-firefox.md
├── 0013-ipc-security-models-and-status.md
├── 0014-xul-overlay-removal-review-packet.md
├── 0015-rkv.md
├── 0016-xulstore-rkv-poc.md
└── 0017-lmdb-vs-leveldb.md
Condensed preview — 37 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (324K chars).
[
{
"path": ".gitignore",
"chars": 4513,
"preview": "\n# Created by https://www.gitignore.io/api/vim,node,grunt,emacs,gitbook,eclipse,jetbrains,visualstudiocode\n\n### Eclipse "
},
{
"path": "CODE_OF_CONDUCT.md",
"chars": 545,
"preview": "# Community Participation Guidelines\n\nThis repository is governed by Mozilla's code of conduct and etiquette guidelines."
},
{
"path": "Gemfile",
"chars": 73,
"preview": "source \"https://rubygems.org\"\ngem 'github-pages', group: :jekyll_plugins\n"
},
{
"path": "ISSUE_TEMPLATE.md",
"chars": 392,
"preview": "\nThis repository is for documentation about the Firefox Architectural Change program. It's not a place to raise bugs abo"
},
{
"path": "LICENSE",
"chars": 16725,
"preview": "Mozilla Public License Version 2.0\n==================================\n\n1. Definitions\n--------------\n\n1.1. \"Contributor\""
},
{
"path": "README.md",
"chars": 5448,
"preview": "\n# Firefox Browser Architecture\n\n## Mission\n\nChange Mozilla. Investigate big technical challenges and produce engineerin"
},
{
"path": "_config.yml",
"chars": 67,
"preview": "theme: jekyll-theme-minimal\nbaseurl: /firefox-browser-architecture\n"
},
{
"path": "_layouts/default.html",
"chars": 482,
"preview": "<!doctype html>\n<html lang=\"{{ site.lang | default: \"en-US\" }}\">\n <head>\n <meta charset=\"utf-8\">\n <link rel=\"styl"
},
{
"path": "_layouts/text.html",
"chars": 598,
"preview": "<!doctype html>\n<html lang=\"{{ site.lang | default: \"en-US\" }}\">\n <head>\n <meta charset=\"utf-8\">\n <link rel=\"styl"
},
{
"path": "assets/css/style.scss",
"chars": 2042,
"preview": "---\n---\n\n@import \"{{ site.theme }}\";\n@import url('https://fonts.googleapis.com/css?family=Zilla+Slab|Zilla+Slab+Highligh"
},
{
"path": "experiments/2017-09-06-rust-on-ios.md",
"chars": 9822,
"preview": "---\ntitle: Building and Deploying a Rust library on iOS\nlayout: text\n---\n\n# Building and Deploying a Rust library on iOS"
},
{
"path": "experiments/2017-09-21-rust-on-android.md",
"chars": 15095,
"preview": "---\ntitle: Building and Deploying a Rust library on Android\nlayout: text\n---\n\n# Building and Deploying a Rust library on"
},
{
"path": "newsletter/_posts/2017-07-27-browser-architecture-update.md",
"chars": 6032,
"preview": "---\ntitle: Browser Architecture Update\nlayout: text\ndescription: The first of our newsletters introducing our work\nmaili"
},
{
"path": "newsletter/_posts/2017-08-24-browser-architecture-newsletter-2.md",
"chars": 5015,
"preview": "---\ntitle: Browser Architecture Newsletter 2\nlayout: text\ndescription: A follow-up with updates on XBL Conversion, Stora"
},
{
"path": "newsletter/_posts/2017-09-22-browser-architecture-newsletter-3.md",
"chars": 5407,
"preview": "---\r\ntitle: Browser Architecture Newsletter 3\r\nlayout: text\r\ndescription: A newsletter on architecture review, XBL Conve"
},
{
"path": "newsletter/_posts/2017-10-19-browser-architecture-newsletter-4.md",
"chars": 2658,
"preview": "---\ntitle: Browser Architecture Newsletter 4\nlayout: text\ndescription: A newsletter on architecture review, XBL Conversi"
},
{
"path": "newsletter/_posts/2017-11-29-browser-architecture-newsletter-5.md",
"chars": 2830,
"preview": "---\ntitle: Browser Architecture Newsletter 5\nlayout: text\ndescription: A newsletter on Storage and Sync, XBL Removal and"
},
{
"path": "newsletters.xml",
"chars": 667,
"preview": "---\nlayout: null\n---\n<?xml version=\"1.0\" encoding=\"UTF-8\" ?>\n<rss version=\"2.0\">\n<channel>\n <title>Mozilla Browser Arch"
},
{
"path": "text/0000-template.md",
"chars": 709,
"preview": "---\ntitle: Documentation Template\nlayout: text\n---\n\n# Documentation Template\n\n## Problem\n\nA short lay summary of the pro"
},
{
"path": "text/0001-documenting-output.md",
"chars": 2084,
"preview": "---\ntitle: Documenting our Output\nlayout: text\n---\n\n# Documenting our Output\n\n## Problem\n\nThe Browser Architecture team "
},
{
"path": "text/0002-extracting-necko.md",
"chars": 7931,
"preview": "---\ntitle: Extracting Necko\nlayout: text\n---\n\n# Extracting Necko\n\n## Proposal\n\nWe should extract Necko for reuse in back"
},
{
"path": "text/0003-problems-with-xul.md",
"chars": 2897,
"preview": "---\ntitle: Problems with XUL\nlayout: text\n---\n\n# Problems with XUL\n\nXUL is often mentioned as a source of problems for d"
},
{
"path": "text/0004-xbl-web-components.md",
"chars": 8383,
"preview": "---\ntitle: XBL and Web Components\nlayout: text\n---\n\n# XBL and Web Components\n\nXBL is a technology used to implement reus"
},
{
"path": "text/0005-problems-with-xbl.md",
"chars": 3468,
"preview": "---\ntitle: Problems with XBL\nlayout: text\n---\n\n# Problems with XBL\n\n## Unmaintained\n\nXBL is a proprietary technology dev"
},
{
"path": "text/0006-architecture-review-process.md",
"chars": 11168,
"preview": "---\ntitle: Architecture Review Process\nlayout: text\n---\n\n# Architecture Review Process\n\n## Introduction\n\nThis process ta"
},
{
"path": "text/0007-xbl-design-review-packet.md",
"chars": 20402,
"preview": "---\ntitle: Design Review Packet - XBL Removal\nlayout: text\n---\n\n# Design Review Packet - XBL Removal\n\nThis is the packet"
},
{
"path": "text/0008-sync-and-storage-review-packet.md",
"chars": 37923,
"preview": "---\ntitle: Roadmap Review: Sync and Storage\nlayout: text\n---\n\n# Roadmap Review: Sync and Storage\n\n**Firefox should wow u"
},
{
"path": "text/0009-type-safety-systems.md",
"chars": 2704,
"preview": "---\ntitle: JavaScript Type Safety Systems\nlayout: text\n---\n\n# JavaScript Type Safety Systems\n\nJavaScript is an untyped l"
},
{
"path": "text/0010-firefox-data-stores.md",
"chars": 3970,
"preview": "---\ntitle: Firefox Data Stores\nlayout: text\n---\n\n# Firefox Data Stores\n\n## Introduction\n\nThere are a large number of dat"
},
{
"path": "text/0011-fluent-in-prefs-design-review.md",
"chars": 6628,
"preview": "---\ntitle: Fluent in Prefs Design Review\nlayout: text\n---\n\n# Fluent in Prefs Design Review\n\n## Introduction\n\n[Fluent](ht"
},
{
"path": "text/0012-jsonfile.md",
"chars": 6220,
"preview": "---\ntitle: A brief analysis of JSON file-backed storage\nlayout: text\n---\n\n# A brief analysis of JSON file-backed storage"
},
{
"path": "text/0012-process-isolation-in-firefox.md",
"chars": 19694,
"preview": "---\ntitle: Process Isolation in Firefox\nlayout: text\n---\n\n# Process Isolation in Firefox\n\nRandell Jesup, Mozilla Browser"
},
{
"path": "text/0013-ipc-security-models-and-status.md",
"chars": 31330,
"preview": "---\ntitle: IPC Security Models and Status\nlayout: text\n---\n\n# IPC Security Models and Status\n\nRandell Jesup\n\nReviewers: "
},
{
"path": "text/0014-xul-overlay-removal-review-packet.md",
"chars": 1930,
"preview": "---\ntitle: XUL Overlay Removal Review Packet\nlayout: text\n---\n\n# XUL Overlay Removal Review Packet\n\n## Summary\n\n### Prob"
},
{
"path": "text/0015-rkv.md",
"chars": 35643,
"preview": "---\ntitle: Roadmap Review: Key-Value Storage (rkv)\nlayout: text\n---\n\n# Design Review: Key-Value Storage\n\n**We propose th"
},
{
"path": "text/0016-xulstore-rkv-poc.md",
"chars": 16998,
"preview": "# XULStore Using rkv – Proof of Concept\n\nIn his \"Key-value storage proposal,\" Richard Newman proposed the \"standardizati"
},
{
"path": "text/0017-lmdb-vs-leveldb.md",
"chars": 17103,
"preview": "# LMDB vs. LevelDB\n\nThis document compares the [Lightning Memory-mapped Database](https://symas.com/lmdb/) (LMDB) key-va"
}
]
About this extraction
This page contains the full source code of the mozilla/firefox-browser-architecture GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 37 files (308.2 KB), approximately 71.2k tokens. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.