[
  {
    "path": ".gitignore",
    "content": "\n# Created by https://www.gitignore.io/api/vim,node,grunt,emacs,gitbook,eclipse,jetbrains,visualstudiocode\n\n### Eclipse ###\n\n.metadata\nbin/\ntmp/\n*.tmp\n*.bak\n*.swp\n*~.nib\nlocal.properties\n.settings/\n.loadpath\n.recommenders\n\n# External tool builders\n.externalToolBuilders/\n\n# Locally stored \"Eclipse launch configurations\"\n*.launch\n\n# PyDev specific (Python IDE for Eclipse)\n*.pydevproject\n\n# CDT-specific (C/C++ Development Tooling)\n.cproject\n\n# Java annotation processor (APT)\n.factorypath\n\n# PDT-specific (PHP Development Tools)\n.buildpath\n\n# sbteclipse plugin\n.target\n\n# Tern plugin\n.tern-project\n\n# TeXlipse plugin\n.texlipse\n\n# STS (Spring Tool Suite)\n.springBeans\n\n# Code Recommenders\n.recommenders/\n\n# Scala IDE specific (Scala & Java development for Eclipse)\n.cache-main\n.scala_dependencies\n.worksheet\n\n### Eclipse Patch ###\n# Eclipse Core\t\t\n.project\n\n# JDT-specific (Eclipse Java Development Tools)\t\t\n.classpath\n\n### Emacs ###\n# -*- mode: gitignore; -*-\n*~\n\\#*\\#\n/.emacs.desktop\n/.emacs.desktop.lock\n*.elc\nauto-save-list\ntramp\n.\\#*\n\n# Org-mode\n.org-id-locations\n*_archive\n\n# flymake-mode\n*_flymake.*\n\n# eshell files\n/eshell/history\n/eshell/lastdir\n\n# elpa packages\n/elpa/\n\n# reftex files\n*.rel\n\n# AUCTeX auto folder\n/auto/\n\n# cask packages\n.cask/\ndist/\n\n# Flycheck\nflycheck_*.el\n\n# server auth directory\n/server/\n\n# projectiles files\n.projectile\n\n# directory configuration\n.dir-locals.el\n\n### GitBook ###\n# Node rules:\n## Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files)\n.grunt\n\n## Dependency directory\n## Commenting this out is preferred by some people, see\n## https://docs.npmjs.com/misc/faq#should-i-check-my-node_modules-folder-into-git\nnode_modules\n\n# Book build output\n_book\n\n# eBook build output\n*.epub\n*.mobi\n*.pdf\n\n### grunt ###\n# Grunt usually compiles files inside this directory\n\n# Grunt usually preprocesses files such as coffeescript, compass... inside the .tmp directory\n.tmp/\n\n### JetBrains ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio and Webstorm\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff:\n.idea/**/workspace.xml\n.idea/**/tasks.xml\n.idea/dictionaries\n\n# Sensitive or high-churn files:\n.idea/**/dataSources/\n.idea/**/dataSources.ids\n.idea/**/dataSources.xml\n.idea/**/dataSources.local.xml\n.idea/**/sqlDataSources.xml\n.idea/**/dynamic.xml\n.idea/**/uiDesigner.xml\n\n# Gradle:\n.idea/**/gradle.xml\n.idea/**/libraries\n\n# CMake\ncmake-build-debug/\n\n# Mongo Explorer plugin:\n.idea/**/mongoSettings.xml\n\n## File-based project format:\n*.iws\n\n## Plugin-specific files:\n\n# IntelliJ\n/out/\n\n# mpeltonen/sbt-idea plugin\n.idea_modules/\n\n# JIRA plugin\natlassian-ide-plugin.xml\n\n# Cursive Clojure plugin\n.idea/replstate.xml\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\ncom_crashlytics_export_strings.xml\ncrashlytics.properties\ncrashlytics-build.properties\nfabric.properties\n\n### JetBrains Patch ###\n# Comment Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-215987721\n\n# *.iml\n# modules.xml\n# .idea/misc.xml\n# *.ipr\n\n# Sonarlint plugin\n.idea/sonarlint\n\n### Node ###\n# Logs\nlogs\n*.log\nnpm-debug.log*\nyarn-debug.log*\nyarn-error.log*\n\n# Runtime data\npids\n*.pid\n*.seed\n*.pid.lock\n\n# Directory for instrumented libs generated by jscoverage/JSCover\nlib-cov\n\n# Coverage directory used by tools like istanbul\ncoverage\n\n# nyc test coverage\n.nyc_output\n\n# Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files)\n\n# Bower dependency directory (https://bower.io/)\nbower_components\n\n# node-waf configuration\n.lock-wscript\n\n# Compiled binary addons (http://nodejs.org/api/addons.html)\nbuild/Release\n\n# Dependency directories\nnode_modules/\njspm_packages/\n\n# Typescript v1 declaration files\ntypings/\n\n# Optional npm cache directory\n.npm\n\n# Optional eslint cache\n.eslintcache\n\n# Optional REPL history\n.node_repl_history\n\n# Output of 'npm pack'\n*.tgz\n\n# Yarn Integrity file\n.yarn-integrity\n\n# dotenv environment variables file\n.env\n\n\n### Vim ###\n# swap\n[._]*.s[a-v][a-z]\n[._]*.sw[a-p]\n[._]s[a-v][a-z]\n[._]sw[a-p]\n# session\nSession.vim\n# temporary\n.netrwhist\n# auto-generated tag files\ntags\n\n### VisualStudioCode ###\n.vscode/*\n!.vscode/settings.json\n!.vscode/tasks.json\n!.vscode/launch.json\n!.vscode/extensions.json\n.history\n\n# End of https://www.gitignore.io/api/vim,node,grunt,emacs,gitbook,eclipse,jetbrains,visualstudiocode\n\n# JOE: Added manually\n.idea/\n\n# Jekyll: Added manually\n.sass-cache\n_site\nGemfile.lock\n"
  },
  {
    "path": "CODE_OF_CONDUCT.md",
    "content": "# Community Participation Guidelines\n\nThis repository is governed by Mozilla's code of conduct and etiquette guidelines. \nFor more details, please read the\n[Mozilla Community Participation Guidelines](https://www.mozilla.org/about/governance/policies/participation/). \n\n## How to Report\nFor more information on how to report violations of the Community Participation Guidelines, please read our '[How to Report](https://www.mozilla.org/about/governance/policies/participation/reporting/)' page.\n\n<!-- There is copy of this text in README.md -->\n"
  },
  {
    "path": "Gemfile",
    "content": "source \"https://rubygems.org\"\ngem 'github-pages', group: :jekyll_plugins\n"
  },
  {
    "path": "ISSUE_TEMPLATE.md",
    "content": "\nThis repository is for documentation about the Firefox Architectural Change program. It's not a place to raise bugs about Firefox itself.\n\n* If you have found a bug in Firefox please start here: https://bugzilla.mozilla.org/enter_bug.cgi\n* If you want to give broader feedback: https://input.mozilla.org/\n* If you have a website that doesn't work properly in Firefox: https://webcompat.com/\n"
  },
  {
    "path": "LICENSE",
    "content": "Mozilla Public License Version 2.0\n==================================\n\n1. Definitions\n--------------\n\n1.1. \"Contributor\"\n    means each individual or legal entity that creates, contributes to\n    the creation of, or owns Covered Software.\n\n1.2. \"Contributor Version\"\n    means the combination of the Contributions of others (if any) used\n    by a Contributor and that particular Contributor's Contribution.\n\n1.3. \"Contribution\"\n    means Covered Software of a particular Contributor.\n\n1.4. \"Covered Software\"\n    means Source Code Form to which the initial Contributor has attached\n    the notice in Exhibit A, the Executable Form of such Source Code\n    Form, and Modifications of such Source Code Form, in each case\n    including portions thereof.\n\n1.5. \"Incompatible With Secondary Licenses\"\n    means\n\n    (a) that the initial Contributor has attached the notice described\n        in Exhibit B to the Covered Software; or\n\n    (b) that the Covered Software was made available under the terms of\n        version 1.1 or earlier of the License, but not also under the\n        terms of a Secondary License.\n\n1.6. \"Executable Form\"\n    means any form of the work other than Source Code Form.\n\n1.7. \"Larger Work\"\n    means a work that combines Covered Software with other material, in\n    a separate file or files, that is not Covered Software.\n\n1.8. \"License\"\n    means this document.\n\n1.9. \"Licensable\"\n    means having the right to grant, to the maximum extent possible,\n    whether at the time of the initial grant or subsequently, any and\n    all of the rights conveyed by this License.\n\n1.10. \"Modifications\"\n    means any of the following:\n\n    (a) any file in Source Code Form that results from an addition to,\n        deletion from, or modification of the contents of Covered\n        Software; or\n\n    (b) any new file in Source Code Form that contains any Covered\n        Software.\n\n1.11. \"Patent Claims\" of a Contributor\n    means any patent claim(s), including without limitation, method,\n    process, and apparatus claims, in any patent Licensable by such\n    Contributor that would be infringed, but for the grant of the\n    License, by the making, using, selling, offering for sale, having\n    made, import, or transfer of either its Contributions or its\n    Contributor Version.\n\n1.12. \"Secondary License\"\n    means either the GNU General Public License, Version 2.0, the GNU\n    Lesser General Public License, Version 2.1, the GNU Affero General\n    Public License, Version 3.0, or any later versions of those\n    licenses.\n\n1.13. \"Source Code Form\"\n    means the form of the work preferred for making modifications.\n\n1.14. \"You\" (or \"Your\")\n    means an individual or a legal entity exercising rights under this\n    License. For legal entities, \"You\" includes any entity that\n    controls, is controlled by, or is under common control with You. For\n    purposes of this definition, \"control\" means (a) the power, direct\n    or indirect, to cause the direction or management of such entity,\n    whether by contract or otherwise, or (b) ownership of more than\n    fifty percent (50%) of the outstanding shares or beneficial\n    ownership of such entity.\n\n2. License Grants and Conditions\n--------------------------------\n\n2.1. Grants\n\nEach Contributor hereby grants You a world-wide, royalty-free,\nnon-exclusive license:\n\n(a) under intellectual property rights (other than patent or trademark)\n    Licensable by such Contributor to use, reproduce, make available,\n    modify, display, perform, distribute, and otherwise exploit its\n    Contributions, either on an unmodified basis, with Modifications, or\n    as part of a Larger Work; and\n\n(b) under Patent Claims of such Contributor to make, use, sell, offer\n    for sale, have made, import, and otherwise transfer either its\n    Contributions or its Contributor Version.\n\n2.2. Effective Date\n\nThe licenses granted in Section 2.1 with respect to any Contribution\nbecome effective for each Contribution on the date the Contributor first\ndistributes such Contribution.\n\n2.3. Limitations on Grant Scope\n\nThe licenses granted in this Section 2 are the only rights granted under\nthis License. No additional rights or licenses will be implied from the\ndistribution or licensing of Covered Software under this License.\nNotwithstanding Section 2.1(b) above, no patent license is granted by a\nContributor:\n\n(a) for any code that a Contributor has removed from Covered Software;\n    or\n\n(b) for infringements caused by: (i) Your and any other third party's\n    modifications of Covered Software, or (ii) the combination of its\n    Contributions with other software (except as part of its Contributor\n    Version); or\n\n(c) under Patent Claims infringed by Covered Software in the absence of\n    its Contributions.\n\nThis License does not grant any rights in the trademarks, service marks,\nor logos of any Contributor (except as may be necessary to comply with\nthe notice requirements in Section 3.4).\n\n2.4. Subsequent Licenses\n\nNo Contributor makes additional grants as a result of Your choice to\ndistribute the Covered Software under a subsequent version of this\nLicense (see Section 10.2) or under the terms of a Secondary License (if\npermitted under the terms of Section 3.3).\n\n2.5. Representation\n\nEach Contributor represents that the Contributor believes its\nContributions are its original creation(s) or it has sufficient rights\nto grant the rights to its Contributions conveyed by this License.\n\n2.6. Fair Use\n\nThis License is not intended to limit any rights You have under\napplicable copyright doctrines of fair use, fair dealing, or other\nequivalents.\n\n2.7. Conditions\n\nSections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted\nin Section 2.1.\n\n3. Responsibilities\n-------------------\n\n3.1. Distribution of Source Form\n\nAll distribution of Covered Software in Source Code Form, including any\nModifications that You create or to which You contribute, must be under\nthe terms of this License. You must inform recipients that the Source\nCode Form of the Covered Software is governed by the terms of this\nLicense, and how they can obtain a copy of this License. You may not\nattempt to alter or restrict the recipients' rights in the Source Code\nForm.\n\n3.2. Distribution of Executable Form\n\nIf You distribute Covered Software in Executable Form then:\n\n(a) such Covered Software must also be made available in Source Code\n    Form, as described in Section 3.1, and You must inform recipients of\n    the Executable Form how they can obtain a copy of such Source Code\n    Form by reasonable means in a timely manner, at a charge no more\n    than the cost of distribution to the recipient; and\n\n(b) You may distribute such Executable Form under the terms of this\n    License, or sublicense it under different terms, provided that the\n    license for the Executable Form does not attempt to limit or alter\n    the recipients' rights in the Source Code Form under this License.\n\n3.3. Distribution of a Larger Work\n\nYou may create and distribute a Larger Work under terms of Your choice,\nprovided that You also comply with the requirements of this License for\nthe Covered Software. If the Larger Work is a combination of Covered\nSoftware with a work governed by one or more Secondary Licenses, and the\nCovered Software is not Incompatible With Secondary Licenses, this\nLicense permits You to additionally distribute such Covered Software\nunder the terms of such Secondary License(s), so that the recipient of\nthe Larger Work may, at their option, further distribute the Covered\nSoftware under the terms of either this License or such Secondary\nLicense(s).\n\n3.4. Notices\n\nYou may not remove or alter the substance of any license notices\n(including copyright notices, patent notices, disclaimers of warranty,\nor limitations of liability) contained within the Source Code Form of\nthe Covered Software, except that You may alter any license notices to\nthe extent required to remedy known factual inaccuracies.\n\n3.5. Application of Additional Terms\n\nYou may choose to offer, and to charge a fee for, warranty, support,\nindemnity or liability obligations to one or more recipients of Covered\nSoftware. However, You may do so only on Your own behalf, and not on\nbehalf of any Contributor. You must make it absolutely clear that any\nsuch warranty, support, indemnity, or liability obligation is offered by\nYou alone, and You hereby agree to indemnify every Contributor for any\nliability incurred by such Contributor as a result of warranty, support,\nindemnity or liability terms You offer. You may include additional\ndisclaimers of warranty and limitations of liability specific to any\njurisdiction.\n\n4. Inability to Comply Due to Statute or Regulation\n---------------------------------------------------\n\nIf it is impossible for You to comply with any of the terms of this\nLicense with respect to some or all of the Covered Software due to\nstatute, judicial order, or regulation then You must: (a) comply with\nthe terms of this License to the maximum extent possible; and (b)\ndescribe the limitations and the code they affect. Such description must\nbe placed in a text file included with all distributions of the Covered\nSoftware under this License. Except to the extent prohibited by statute\nor regulation, such description must be sufficiently detailed for a\nrecipient of ordinary skill to be able to understand it.\n\n5. Termination\n--------------\n\n5.1. The rights granted under this License will terminate automatically\nif You fail to comply with any of its terms. However, if You become\ncompliant, then the rights granted under this License from a particular\nContributor are reinstated (a) provisionally, unless and until such\nContributor explicitly and finally terminates Your grants, and (b) on an\nongoing basis, if such Contributor fails to notify You of the\nnon-compliance by some reasonable means prior to 60 days after You have\ncome back into compliance. Moreover, Your grants from a particular\nContributor are reinstated on an ongoing basis if such Contributor\nnotifies You of the non-compliance by some reasonable means, this is the\nfirst time You have received notice of non-compliance with this License\nfrom such Contributor, and You become compliant prior to 30 days after\nYour receipt of the notice.\n\n5.2. If You initiate litigation against any entity by asserting a patent\ninfringement claim (excluding declaratory judgment actions,\ncounter-claims, and cross-claims) alleging that a Contributor Version\ndirectly or indirectly infringes any patent, then the rights granted to\nYou by any and all Contributors for the Covered Software under Section\n2.1 of this License shall terminate.\n\n5.3. In the event of termination under Sections 5.1 or 5.2 above, all\nend user license agreements (excluding distributors and resellers) which\nhave been validly granted by You or Your distributors under this License\nprior to termination shall survive termination.\n\n************************************************************************\n*                                                                      *\n*  6. Disclaimer of Warranty                                           *\n*  -------------------------                                           *\n*                                                                      *\n*  Covered Software is provided under this License on an \"as is\"       *\n*  basis, without warranty of any kind, either expressed, implied, or  *\n*  statutory, including, without limitation, warranties that the       *\n*  Covered Software is free of defects, merchantable, fit for a        *\n*  particular purpose or non-infringing. The entire risk as to the     *\n*  quality and performance of the Covered Software is with You.        *\n*  Should any Covered Software prove defective in any respect, You     *\n*  (not any Contributor) assume the cost of any necessary servicing,   *\n*  repair, or correction. This disclaimer of warranty constitutes an   *\n*  essential part of this License. No use of any Covered Software is   *\n*  authorized under this License except under this disclaimer.         *\n*                                                                      *\n************************************************************************\n\n************************************************************************\n*                                                                      *\n*  7. Limitation of Liability                                          *\n*  --------------------------                                          *\n*                                                                      *\n*  Under no circumstances and under no legal theory, whether tort      *\n*  (including negligence), contract, or otherwise, shall any           *\n*  Contributor, or anyone who distributes Covered Software as          *\n*  permitted above, be liable to You for any direct, indirect,         *\n*  special, incidental, or consequential damages of any character      *\n*  including, without limitation, damages for lost profits, loss of    *\n*  goodwill, work stoppage, computer failure or malfunction, or any    *\n*  and all other commercial damages or losses, even if such party      *\n*  shall have been informed of the possibility of such damages. This   *\n*  limitation of liability shall not apply to liability for death or   *\n*  personal injury resulting from such party's negligence to the       *\n*  extent applicable law prohibits such limitation. Some               *\n*  jurisdictions do not allow the exclusion or limitation of           *\n*  incidental or consequential damages, so this exclusion and          *\n*  limitation may not apply to You.                                    *\n*                                                                      *\n************************************************************************\n\n8. Litigation\n-------------\n\nAny litigation relating to this License may be brought only in the\ncourts of a jurisdiction where the defendant maintains its principal\nplace of business and such litigation shall be governed by laws of that\njurisdiction, without reference to its conflict-of-law provisions.\nNothing in this Section shall prevent a party's ability to bring\ncross-claims or counter-claims.\n\n9. Miscellaneous\n----------------\n\nThis License represents the complete agreement concerning the subject\nmatter hereof. If any provision of this License is held to be\nunenforceable, such provision shall be reformed only to the extent\nnecessary to make it enforceable. Any law or regulation which provides\nthat the language of a contract shall be construed against the drafter\nshall not be used to construe this License against a Contributor.\n\n10. Versions of the License\n---------------------------\n\n10.1. New Versions\n\nMozilla Foundation is the license steward. Except as provided in Section\n10.3, no one other than the license steward has the right to modify or\npublish new versions of this License. Each version will be given a\ndistinguishing version number.\n\n10.2. Effect of New Versions\n\nYou may distribute the Covered Software under the terms of the version\nof the License under which You originally received the Covered Software,\nor under the terms of any subsequent version published by the license\nsteward.\n\n10.3. Modified Versions\n\nIf you create software not governed by this License, and you want to\ncreate a new license for such software, you may create and use a\nmodified version of this License if you rename the license and remove\nany references to the name of the license steward (except to note that\nsuch modified license differs from this License).\n\n10.4. Distributing Source Code Form that is Incompatible With Secondary\nLicenses\n\nIf You choose to distribute Source Code Form that is Incompatible With\nSecondary Licenses under the terms of this version of the License, the\nnotice described in Exhibit B of this License must be attached.\n\nExhibit A - Source Code Form License Notice\n-------------------------------------------\n\n  This Source Code Form is subject to the terms of the Mozilla Public\n  License, v. 2.0. If a copy of the MPL was not distributed with this\n  file, You can obtain one at http://mozilla.org/MPL/2.0/.\n\nIf it is not possible or desirable to put the notice in a particular\nfile, then You may include the notice in a location (such as a LICENSE\nfile in a relevant directory) where a recipient would be likely to look\nfor such a notice.\n\nYou may add additional accurate notices of copyright ownership.\n\nExhibit B - \"Incompatible With Secondary Licenses\" Notice\n---------------------------------------------------------\n\n  This Source Code Form is \"Incompatible With Secondary Licenses\", as\n  defined by the Mozilla Public License, v. 2.0.\n"
  },
  {
    "path": "README.md",
    "content": "\n# Firefox Browser Architecture\n\n## Mission\n\nChange Mozilla. Investigate big technical challenges and produce engineering programs to address them.\n\n\n## Our Conclusions\n\nThis is a list of our findings that we're reasonably happy with so far.\n\n* [Documenting our output](text/0001-documenting-output.md) looks at how we’re going to communicate with the rest of Mozilla.\n* [Extracting Necko](text/0002-extracting-necko.md) considers whether it's feasible or worthwhile to extract Necko — Gecko's C++ networking library — for use as a standalone component.\n* [Problems with XUL](text/0003-problems-with-xul.md) aims to list the different kinds of problems that exist with XUL.\n* [XBL and Web Components](text/0004-xbl-web-components.md) compares some old Mozilla technology (XBL) with modern Web Components.\n* [Problems with XBL](text/0005-problems-with-xbl.md) aims to list the different kinds of problems that exist with XBL.\n* [Architecture Reviews](text/0006-architecture-review-process.md) are healthy and we proposed a process for healthy reviews.\n* [XBL Design Review packet](text/0007-xbl-design-review-packet.md) is the packet that we prepared for the architectural design review for XBL removal.\n* [Roadmap Review: Sync and Storage](text/0008-sync-and-storage-review-packet.md) establishes that storage and syncing of user data is a pillar of the Firefox ecosystem, warranting holistic and long-term attention, and outlines where we’d like to end up and some ideas for how to get there.\n* [JavaScript Type Safety Systems](text/0009-type-safety-systems.md) are some conclusions of an investigation into the use of JavaScript type safety systems.\n* [Firefox Data Stores Documentation](text/0010-firefox-data-stores.md) documents the existing data stores across all current Firefox platforms.\n* [Fluent in Prefs Design Review](text/0011-fluent-in-prefs-design-review.md) describes the lightweight design review for Fluent in Prefs.\n* [A brief analysis of JSON file-backed storage](text/0012-jsonfile.md) outlines some of the pros and cons of using a flat file (particularly via `JSONFile.jsm`) to store data in Firefox.\n* [Process Isolation in Firefox](text/0012-process-isolation-in-firefox.md) is a WIP evaluation of how far we can push process isolation to improve security and stability.\n* [IPC Security Models and Status](text/0013-ipc-security-models-and-status.md) is an audit of our current IPC status.\n* [XUL Overlay Removal Review Packet](text/0014-xul-overlay-removal-review-packet.md) is the packet that we prepared for the architectural design review for XUL Overlay removal.\n* [Design Review: Key-Value Storage](text/0015-rkv.md) proposes the introduction of a fast, cross-platform key-value store for Mozilla products.\n* [XULStore Using rkv – Proof of Concept](text/0016-xulstore-rkv-poc.md) describes a proof-of-concept implementation of XULStore that uses [rkv](https://github.com/mozilla/rkv).\n* [LMDB vs. LevelDB](text/0017-lmdb-vs-leveldb.md) compares the [Lightning Memory-mapped Database](https://symas.com/lmdb/) (LMDB) key-value storage engine to the [LevelDB](https://github.com/google/leveldb) key-value storage engine.\n\n## Posts\n\nWe typically send our newsletters to [firefox-dev](https://www.mozilla.org/en-US/about/forums/#firefox-dev).\n\n* [Browser Architecture Update](newsletter/_posts/2017-07-27-browser-architecture-update.md). See also [mailing-list-post](https://groups.google.com/forum/#!topic/firefox-dev/ueRILL2ppac).\n* [Browser Architecture Newsletter #2](newsletter/_posts/2017-08-24-browser-architecture-newsletter-2.md). See also [mailing-list-post](https://groups.google.com/forum/#!topic/firefox-dev/Rc2w2a9e8HQ).\n* [Browser Architecture Newsletter #3](newsletter/_posts/2017-09-22-browser-architecture-newsletter-3.md). See also [mailing-list-post](https://groups.google.com/forum/#!topic/firefox-dev/p9rTlfFUXlQ).\n* [Browser Architecture Newsletter #4](newsletter/_posts/2017-10-19-browser-architecture-newsletter-4.md). See also [mailing-list-post](https://groups.google.com/forum/#!topic/firefox-dev/CLFtj8qUSv8).\n* [Browser Architecture Newsletter #5](newsletter/_posts/2017-11-29-browser-architecture-newsletter-5.md). See also [mailing-list-post](https://groups.google.com/forum/#!topic/firefox-dev/XKp3EthdJ60).\n\n## Explorations and Experiments\n\nTo support our conclusions we occasionally perform explorations and experiments. The first exploration is designed to support the notion that we can create a sync and storage layer in Rust that we can deploy to Desktop, Android and iOS.\n\n* [Deploying a Rust library on iOS](experiments/2017-09-06-rust-on-ios.md). A short tutorial describing how to build and deploy a Rust library for use inside an iOS app.\n* [Deploying a Rust library on Android](experiments/2017-09-21-rust-on-android.md). A short tutorial describing how to build and deploy a Rust library for use inside an Android app.\n\n<!-- This is a copy of this text in CODE_OF_CONDUCT.md -->\n\n# Community Participation Guidelines\n\nThis repository is governed by Mozilla's code of conduct and etiquette guidelines. \nFor more details, please read the\n[Mozilla Community Participation Guidelines](https://www.mozilla.org/about/governance/policies/participation/). \n\n## How to Report\nFor more information on how to report violations of the Community Participation Guidelines, please read our '[How to Report](https://www.mozilla.org/about/governance/policies/participation/reporting/)' page.\n"
  },
  {
    "path": "_config.yml",
    "content": "theme: jekyll-theme-minimal\nbaseurl: /firefox-browser-architecture\n"
  },
  {
    "path": "_layouts/default.html",
    "content": "<!doctype html>\n<html lang=\"{{ site.lang | default: \"en-US\" }}\">\n  <head>\n    <meta charset=\"utf-8\">\n    <link rel=\"stylesheet\" href=\"{{ '/assets/css/style.css?v=' | append: site.github.build_revision | relative_url }}\">\n    <meta name=\"viewport\" content=\"width=device-width\">\n    <title>{{ page.title }}</title>\n  </head>\n  <body>\n    <div class=\"wrapper\">\n      {{ content }}\n    </div>\n    <script src=\"{{ '/assets/js/scale.fix.js' | relative_url }}\"></script>\n  </body>\n</html>\n"
  },
  {
    "path": "_layouts/text.html",
    "content": "<!doctype html>\n<html lang=\"{{ site.lang | default: \"en-US\" }}\">\n  <head>\n    <meta charset=\"utf-8\">\n    <link rel=\"stylesheet\" href=\"{{ '/assets/css/style.css?v=' | append: site.github.build_revision | relative_url }}\">\n    <meta name=\"viewport\" content=\"width=device-width\">\n    <title>{{ page.title }}</title>\n  </head>\n  <body>\n    <p>[<a href=\"{{ site.baseurl }}\">Back to top</a>]</p>\n    <div class=\"wrapper\">\n      {{ content }}\n    </div>\n    <p>[<a href=\"{{ site.baseurl }}\">Back to top</a>]</p>\n    <script src=\"{{ '/assets/js/scale.fix.js' | relative_url }}\"></script>\n  </body>\n</html>\n"
  },
  {
    "path": "assets/css/style.scss",
    "content": "---\n---\n\n@import \"{{ site.theme }}\";\n@import url('https://fonts.googleapis.com/css?family=Zilla+Slab|Zilla+Slab+Highlight');\n\n// Convert pixel units to rems, assuming a 16px base size\n// Usage: remify(24px)\n@function remify($pixels) {\n  $rems: $pixels / 16px;\n  @return #{$rems}rem;\n}\n\n@mixin font-size($size) {\n  // When unit is pixels, pass it through and convert to rem\n  @if (unit($size) == 'px') {\n      font-size: $size;\n      font-size: remify($size);\n  }\n\n  // When unit is not pixels, show an error\n  @else {\n      @error 'This mixin only accepts sizes in pixels. You declared `font-size(#{$size})`.';\n  }\n}\n\n@mixin font-size-huge { // For especially huge titles\n  @include font-size(50px);\n\n  @media screen and (min-width: 760px) {\n      @include font-size(90px);\n  }\n}\n\n@mixin font-size-level1 {\n  @include font-size(28px);\n\n  @media screen and (min-width: 760px) {\n    @include font-size(50px);\n  }\n}\n\n@mixin font-size-level2 {\n  @include font-size(21px);\n\n  @media screen and (min-width: 760px) {\n    @include font-size(38px);\n  }\n}\n\n@mixin font-size-level3 {\n  @include font-size(18px);\n\n  @media screen and (min-width: 760px) {\n    @include font-size(28px);\n  }\n}\n\n@mixin font-size-level4 {\n  @include font-size(16px);\n\n  @media screen and (min-width: 760px) {\n    @include font-size(21px);\n  }\n}\n\nbody {\n  background: #fff;\n  font-size: 20px;\n  font-family: \"Zilla Slab\", \"Open Sans\", X-LocaleSpecific, sans-serif;\n  line-height: 1.5;\n}\n\nh1, h2, h3, h4, h5, h6, legend {\n  color: #000;\n  font-weight: bold;\n  line-height: 1.1;\n  margin: 0 0 .25em;\n}\n\nh1 {\n  @include font-size-level1;\n}\n\nh2 {\n  @include font-size-level2;\n}\n\nh3 {\n  @include font-size-level3;\n}\n\nh4 {\n  @include font-size-level4;\n}\n\nh5, h6 {\n  @include font-size(16px);\n}\n\np,\nul,\nol,\ndl,\ntable {\n  margin-bottom: 1.25em;\n}\n\na {\n  color: #006688;\n}\n\na:hover, a:focus, a:active, a:visited:hover, a:visited:focus, a:visited:active {\n  color: #0088aa;\n  font-weight: normal;\n}\n\na:visited {\n  color: #004455;\n}\n\n.wrapper > section {\n  float: none;\n  width: 100%;\n}\n"
  },
  {
    "path": "experiments/2017-09-06-rust-on-ios.md",
    "content": "---\ntitle: Building and Deploying a Rust library on iOS\nlayout: text\n---\n\n# Building and Deploying a Rust library on iOS\n\nFirst, we have to [install Xcode](https://itunes.apple.com/us/app/xcode/id497799835?ls=1&amp;mt=12) and then set up Xcode build tools. If you already have the build tools installed and they are up to date, you can skip this step.\n```\nxcode-select --install\n```\n\nNext, we need to ensure that Rust is installed and that we can cross compile to the iOS architectures. For this we will be using [rustup](https://www.rustup.rs/). If you already have rustup installed, you can skip this step. Rustup installs Rust from the official release channels and enables you to easily switch between different release versions. It will be useful to you for all your future Rust development, not just here.\n```\ncurl https://sh.rustup.rs -sSf | sh\n```\n\nAdd the iOS architectures to rustup so we can use them during cross compilation.\n```\nrustup target add aarch64-apple-ios armv7-apple-ios armv7s-apple-ios x86_64-apple-ios i386-apple-ios\n```\n\nWhen you installed Rust, it also installed cargo, which is a package manager similar to pip, gems etc. Now we will use cargo to install `cargo-lipo`. This is a cargo subcommand which automatically creates a universal library for use with iOS. Without this crate, cross compiling Rust to work on iOS is infinitely harder.\n```\ncargo install cargo-lipo\n```\n\nNow we're all setup and we're ready to start. Let's create the project directory.\n```\nmkdir greetings\ncd greetings\ncargo new cargo\nmkdir ios\n\n```\n\n`cargo new cargo` sets up a brand new Rust project with its default files and directories in a directory called `cargo`. In this directory is a file called `Cargo.toml`, which is the package manager descriptor file, and there is be a subdirectory, `src`, which contains a file called `lib.rs`. This will contain the Rust code that we will be executing.\n\nOur Rust project here is a super simple Hello World library. It contains a function `rust_greeting` that takes a string argument and return a string greeting that argument. Therefore, if the argument is \"world\", the returned string is \"Hello world\".\n\nOpen `cargo/src/lib.rs` and enter the following code.\n\n```rust\nuse std::os::raw::{c_char};\nuse std::ffi::{CString, CStr};\n\n#[no_mangle]\npub extern fn rust_greeting(to: *const c_char) -> *mut c_char {\n    let c_str = unsafe { CStr::from_ptr(to) };\n    let recipient = match c_str.to_str() {\n        Err(_) => \"there\",\n        Ok(string) => string,\n    };\n\n    CString::new(\"Hello \".to_owned() + recipient).unwrap().into_raw()\n}\n\n#[no_mangle]\npub extern fn rust_greeting_free(s: *mut c_char) {\n    unsafe {\n        if s.is_null() { return }\n        CString::from_raw(s)\n    };\n}\n```\n\nLet's take a look at what is going on here.\n\nAs we will be calling this library from non-Rust code, we will actually be calling it through a C bridge.`#[no_mangle]` tells the compiler not to mangle the function name as it usually does by default, ensuring our function name is exported as if it had been written in C.\n\n`extern` tells the Rust compiler that this function will be called from outside of Rust and to therefore ensure that it is compiled using C calling conventions.\n\nThe string that `rust_greeting` accepts is a pointer to a C char array. We have to then convert the string from a C string to a Rust `str`. First we create a `CStr` object from the pointer. We then convert it to a `str` and check the result. If an error has occurred, then no arg was provided and we substitute `there`, otherwise we use the value of the provided string. We then append the provided string on the end of our greeting string to create our return string. The return string is then converted into a `CString` and passed back into C code.\n\nUsing `CString` and returning the raw representation keeps the string in memory and prevents it from being released at the end of the function. If the memory were to be released, the pointer provided back to the caller would now be pointing to empty memory - or possibly something else entirely. But, by ensuring that the string sticks around after the function has finished executing, we have memory allocated and no longer any handle to it. That is a recipe for a memory leak, so we have to provide a second function `rust_greeting_free` that takes a pointer to a C string and frees that memory. We will have to remember to call `rust_greeting_free` from our iOS code to ensure we don't run into problems.\n\nWe also need to create our C bridge. Create a new file in `cargo/src` called `greetings.h`. Inside that file, let's define what our C interface will look like. We need to ensure that every rust function that we want to call from iOS is defined here.\n```C\n#include <stdint.h>\n\nconst char* rust_greeting(const char* to);\nvoid rust_greeting_free(char *);\n```\n\nLet's build our code to make sure it works. In order to do this we have to complete our `Cargo.toml` file. This will tell cargo to create a static library and C dynamic library for our code.\n\n```toml\n[package]\nname = \"greetings\"\nversion = \"0.1.1\"\nauthors = [\"fluffyemily <fluffyemily@mozilla.com>\"]\ndescription = \"Example static library project built for iOS\"\npublish = false\n\n[lib]\nname = \"greetings\"\ncrate-type = [\"staticlib\", \"cdylib\"]\n```\n\nWe need to build our library against the iOS architectures using `cargo-lipo`. The built artifacts of will be placed in `cargo/target/`. The universal iOS library that we are interested in can be found in `cargo/target/universal/release/libgreetings.a`.\n\n```\ncd cargo\ncargo lipo --release\n```\n\nAnd that's it for our Rust library. Let's get it linked with an iOS project.\n\nOpen Xcode and create a new project. Go to `File\\New\\Project…` and select the `iOS\\Application\\Single View Application` template. This template is as close to a default iOS app as it gets. Click `Next`.\n\n![Create new Swift project](assets/2017-09-06-rust-on-ios/new-project.png)\n\nCall the project `Greetings`, make it a Swift project. Click `Next` to choose the location. We are using the  ios directory that we created earlier. If you choose another location you will have to amend some of the paths that we set later. Click `Create`.\n\n![Name project](assets/2017-09-06-rust-on-ios/name-project.png)\n\nSelect the Greetings project from the project navigator, and then ensure the Greetings target is selected. Open the `General` tab. Scroll down to the `Linked Frameworks and Libraries` section.\n\n![Link rust library](assets/2017-09-06-rust-on-ios/link-libs.png)\n\nImport your `libgreetings.a` library by either dragging it in from finder, or clicking the + at the bottom of the list, clicking 'Add other...' and navigating to `cargo/target/universal/release/`. Select `libgreetings.a` and then click `Open`.\n\nLink `libresolv.tbd`. Click the + at the bottom of the Linked Frameworks list and type libresolv into the search box. Select `libresolv.tbd` and then \"Add\".\n\n![Link libresolv](assets/2017-09-06-rust-on-ios/link-libresolv.png)\n\niOS will need a bridging header to access the C header we created. First, let's import `greetings.h` into our Xcode project so we can link to it. Go to `File\\Add files to \"Greetings\"...` Navigate to `greetings.h` and select `Add`.\n\nTo create our bridging header, go to `File\\New\\File...`. Select `iOS\\Source\\Header File` from the provided options and select `Next`. Name the file `Greetings-Bridging-Header.h` and select `Create`.\n\n![Create bridging header](assets/2017-09-06-rust-on-ios/bridging-header.png)\n\nOpen the bridging header and amend it to look like the following:\n```C\n#ifndef Greetings_Bridging_Header_h\n#define Greetings_Bridging_Header_h\n\n#import \"greetings.h\"\n\n#endif\n```\n\nWe need to tell Xcode about the bridging header. Select the Greetings project from the project navigator, and then ensure the Greetings target is selected and open the Build Settings tab. Set the `Objective-C Bridging Header` option value to `$(PROJECT_DIR)/Greetings/Greetings-Bridging-Header.h`\n\n![Set bridging header location](assets/2017-09-06-rust-on-ios/build-settings-header.png)\n\nWe also need to tell Xcode where to look for our libraries for linking. In the same Build Settings pane, amend the `Library Search Paths` option value to `$(PROJECT_DIR)/../../cargo/target/universal/release`\n\n![Link Library Search Paths](assets/2017-09-06-rust-on-ios/build-settings-search-paths.png)\n\nBuild your xcode project and everything should compile.\n\nSo, now we have imported our Rust library into our iOS project and successfully linked to it. But we still need to call it. go to `File\\New\\File...`. Select `iOS\\Source\\Swift File` from the provided options and select `Next`. Name name it `RustGreetings`.\n\n![Create new Swift file](assets/2017-09-06-rust-on-ios/new-swift.png)\n\nAdd the following code:\n```swift\nclass RustGreetings {\n    func sayHello(to: String) -> String {\n        let result = rust_greeting(to)\n        let swift_result = String(cString: result!)\n        rust_greeting_free(UnsafeMutablePointer(mutating: result))\n        return swift_result\n    }\n}\n```\nThis creates a new class that we will use as an interface to call into our Rust library. This will allow us to abstract the nuance away from the main code of the app, including conversion from a C String to a Swift String and ensures that we won't forget to call our free function and inadvertently introduce a memory leak!\n\nOpen `ViewController.swift`. Inside the `viewDidLoad` function add the following code:\n```swift\nlet rustGreetings = RustGreetings()\nprint(\"\\(rustGreetings.sayHello(to: \"world\"))\")\n```\n\nNow build your project and run. The simulator will open and start running your app. When the view loads, \"Hello world\" will be output in the console window inside Xcode.\n\nYou can find the code for this on [Github](https://github.com/fluffyemily/cross-platform-rust)\n"
  },
  {
    "path": "experiments/2017-09-21-rust-on-android.md",
    "content": "---\ntitle: Building and Deploying a Rust library on Android\nlayout: text\n---\n\n# Building and Deploying a Rust library on Android\n\nFollowing on from the last post on getting a Rust library [building on iOS](https://mozilla.github.io/firefox-browser-architecture/experiments/2017-09-06-rust-on-ios.html), we're now going to deploy the same library on Android.\n\nIn order to do Android development, we'll need to set up our Android environment. First we need to install [Android Studio](https://developer.android.com/studio/index.html). Once Android Studio is installed, we'll need to install the NDK (Native Development Kit).\n\nOpen Android Studio. From the toolbar, go to `Android Studio > Preferences > Appearance & Behaviour > Android SDK > SDK Tools`. Check the following options for installation and click `OK`.\n\n    * Android SDK Tools\n    * NDK\n    * CMake\n    * LLDB\n\nOnce the NDK and associated tools have been installed, we need to set a few environment variables, first for the SDK path and the second for the NDK path. Set the following envvars:\n\n```\nexport ANDROID_HOME=/Users/$USER/Library/Android/sdk\nexport NDK_HOME=$ANDROID_HOME/ndk-bundle\n```\n\nIf you do not already have Rust installed, we need to do this now. For this we will be using [rustup](https://www.rustup.rs/). `rustup` installs Rust from the official release channels and enables you to easily switch between different release versions. It will be useful to you for all your future Rust development, not just here. `rustup` can also be used in conjunction with `HomeBrew`.\n\n```\ncurl https://sh.rustup.rs -sSf | sh\n```\n\nThe next step is to create standalone versions of the NDK for us to compile against. We need to do this for each of the architectures we want to compile against. We will be using the `make_standalone_toolchain.py` script inside the main Android NDK in order to do this. First create a directory for our project.\n\n```\nmkdir greetings\ncd greetings\n```\n\nNow let's create our standalone NDKs. There is no need to be inside the `NDK` directory once you have created it to do this.\n\n```\nmkdir NDK\n${NDK_HOME}/build/tools/make_standalone_toolchain.py --api 26 --arch arm64 --install-dir NDK/arm64\n${NDK_HOME}/build/tools/make_standalone_toolchain.py --api 26 --arch arm --install-dir NDK/arm\n${NDK_HOME}/build/tools/make_standalone_toolchain.py --api 26 --arch x86 --install-dir NDK/x86\n```\n\nCreate a new file, `cargo-config.toml`. This file will tell cargo where to look for the NDKs during cross compilation. Add the following content to the file, remembering to replace instances of `<project path>` with the path to your project directory.\n\n```\n[target.aarch64-linux-android]\nar = \"<project path>/greetings/NDK/arm64/bin/aarch64-linux-android-ar\"\nlinker = \"<project path>/greetings/NDK/arm64/bin/aarch64-linux-android-clang\"\n\n[target.armv7-linux-androideabi]\nar = \"<project path>/greetings/NDK/arm/bin/arm-linux-androideabi-ar\"\nlinker = \"<project path>/greetings/NDK/arm/bin/arm-linux-androideabi-clang\"\n\n[target.i686-linux-android]\nar = \"<project path>/greetings/NDK/x86/bin/i686-linux-android-ar\"\nlinker = \"<project path>/greetings/NDK/x86/bin/i686-linux-android-clang\"\n```\n\nIn order for `cargo` to see our new SDK's we need to copy this config file to our `.cargo` directory like this:\n\n```\ncp cargo-config.toml ~/.cargo/config\n```\n\nLet's go ahead and add our newly created Android architectures to `rustup` so we can use them during cross compilation:\n\n```\nrustup target add aarch64-linux-android armv7-linux-androideabi i686-linux-android\n```\n\nNow we're all set up and we're ready to start. Let's create the `lib` directory. If you've already created a Rust project from following the iOS post, you don't need to do it again.\n\n```\ncargo new cargo\nmkdir android\n```\n\n`cargo new cargo` sets up a brand new Rust project with its default files and directories in a directory called `rust`. The name of the directory is not important. In this directory is a file called `Cargo.toml`, which is the package manager descriptor file, and there is be a subdirectory, `src`, which contains a file called `lib.rs`. This will contain the Rust code that we will be executing.\n\nOur Rust project here is a super simple Hello World library. It contains a function `rust_greeting` that takes a string argument and return a greeting including that argument. Therefore, if the argument is \"world\", the returned string is \"Hello world\".\n\nOpen `cargo/src/lib.rs` and enter the following code.\n\n```rust\nuse std::os::raw::{c_char};\nuse std::ffi::{CString, CStr};\n\n#[no_mangle]\npub extern fn rust_greeting(to: *const c_char) -> *mut c_char {\n    let c_str = unsafe { CStr::from_ptr(to) };\n    let recipient = match c_str.to_str() {\n        Err(_) => \"there\",\n        Ok(string) => string,\n    };\n\n    CString::new(\"Hello \".to_owned() + recipient).unwrap().into_raw()\n}\n```\n\nLet's take a look at what is going on here.\n\nAs we will be calling this library from non-Rust code, we will actually be calling it through a C bridge. `#[no_mangle]` tells the compiler not to mangle the function name as it usually does by default, ensuring our function name is exported as if it had been written in C.\n\n`extern` tells the Rust compiler that this function will be called from outside of Rust and to therefore ensure that it is compiled using C calling conventions.\n\nThe string that `rust_greeting` accepts is a pointer to a C char array. We have to then convert the string from a C string to a Rust `str`. First we create a `CStr` object from the pointer. We then convert it to a `str` and check the result. If an error has occurred, then no arg was provided and we substitute `there`, otherwise we use the value of the provided string. We then append the provided string on the end of our greeting string to create our return string. The return string is then converted into a `CString` and passed back into C code.\n\nNow let's create our Android project.\n\nOpen Android Studio and select `Start a New Android Project` from the options.\n\n![Create new Android project](assets/2017-09-21-rust-on-android/new-project.png)\n\nOn the next screen, type a project name of `Greetings` into the `Application name` field, choose your `Company domain` and select the `android` directory we created earlier as the `Project location`. This will create your Android project inside `greetings/android/Greetings`. Click `Next`.\n\n![Name project](assets/2017-09-21-rust-on-android/name-project.png)\n\nOn the next screen, make sure the `Phone and Tablet` option is selected. Click `Next`.\n\n![Choose form factor](assets/2017-09-21-rust-on-android/form-factors.png)\n\nNow we will be asked to choose a starting activity. Select the `Empty Activity` option and click `Next`.\n\n![Choose starting activity](assets/2017-09-21-rust-on-android/add-activity.png)\n\nName your Activity and layout on the following screen, calling the activity `GreetingsActivity` and the layout `activity_greetings`. Click `Finish`.\n\n![Name activity](assets/2017-09-21-rust-on-android/customize-activity.png)\n\nAs with iOS, we're going to create a wrapper class to wrap the C API and JNI bindings. In the project explorer on the left hand side of the studio window, ensure that `app > java > <domain>.greetings` is highlighted then go to `File > New > Java Class`. Name your class `RustGreetings` and click `OK`.\n\n![Create new class](assets/2017-09-21-rust-on-android/new-class.png)\n\nIn your new class file, add the following code. Here we are defining the native interface to our Rust library and calling it `greeting`, with the same signature. The `sayHello` method simply makes a call to that native function.\n\n```java\npublic class RustGreetings {\n\n    private static native String greeting(final String pattern);\n\n    public String sayHello(String to) {\n        return greeting(to);\n    }\n}\n```\n\nInstead of creating  C header that will be used as a bridge as we did when we wanted to integrate with Swift, for Android we want to expose our functions through JNI. The way that JNI constructs the name of the function that it will call is `Java_<domain>_<class>_<methodname>`. In the case of the method, `greeting` that we have declared here, the function in our Rust library that JNI will attempt to call will be `Java_com_mozilla_greetings_RustGreetings_greeting`. This is the reason why we created our Android project and Java wrapper class before adding any JNI code to the Rust library. We needed to know what the domain, class and function name were before we could construct the right JNI function name in Rust. Let's head back over to our Rust project and create the partner code.\n\nOpen `cargo/src/lib.rs`. At the bottom of the file add the following code:\n\n```rust\n/// Expose the JNI interface for android below\n#[cfg(target_os=\"android\")]\n#[allow(non_snake_case)]\npub mod android {\n    extern crate jni;\n\n    use super::*;\n    use self::jni::JNIEnv;\n    use self::jni::objects::{JClass, JString};\n    use self::jni::sys::{jstring};\n\n    #[no_mangle]\n    pub unsafe extern fn Java_com_mozilla_greetings_RustGreetings_greeting(env: JNIEnv, _: JClass, java_pattern: JString) -> jstring {\n        // Our Java companion code might pass-in \"world\" as a string, hence the name.\n        let world = rust_greeting(env.get_string(java_pattern).expect(\"invalid pattern string\").as_ptr());\n        // Retake pointer so that we can use it below and allow memory to be freed when it goes out of scope.\n        let world_ptr = CString::from_raw(world);\n        let output = env.new_string(world_ptr.to_str().unwrap()).expect(\"Couldn't create java string!\");\n\n        output.into_inner()\n    }\n}\n```\n\nThe first line here `#[cfg(target_os=\"android\")]` is telling the compiler to target Android when compiling this module. `#[cfg]` is a special attribute that allows you to compile code based on a flag passed to the compiler.\n\nThe second line, `#[allow(non_snake_case)]`, tells the compiler not to warn if we are not using `snake_case` for a variable or function name. The Rust compiler is very strict — this is one of the things that makes Rust great — and it enforces the use of `snake_case` throughout. However, we defined our class name and native method in our Android project using Java coding conventions which is `camelCase` and `UpperCamelCase` and we don't want to change this or our Java code will look wrong. Given the way that JNI constructs native function names, we need to tell the Rust compiler to go easy on us in this instance. This flag will apply to all functions and variables created inside this module that we are creating, called `android`.\n\nAfter declaring that we need the `jni` crate, and importing some useful objects from it, we can declare our function. This function needs to be marked `unsafe` because we will be dealing with pointers from a language that allows null pointers, but our code doesn't check for `NULL`. This situation would never happen in Rust only code as the Rust compiler enforces memory safety. By marking the function as not memory safe, we are alerting other Rust functions that it may not be able to deal with a null pointer. `extern` defines the function as one that will be exposed to other languages.\n\nAs arguments, along with the `JString` that our Java function declaration said that we will be providing, we also need to take an instance of the `JNIEnv` and a class reference (which is unused in this example). The `JNIEnv` will be the object we will use to read values associated with the pointers that we are taking as argument.\n\nNext, we read the string in from the `JNIEnv` and convert it into a C pointer to pass to `rust_greeting`. The result of that function is another C pointer, which we then need to convert to a back into a String. Using the `JNIEnv` transfers the ownership of the object to Java, but there is still a reference hanging around held by our Rust code. That memory will be freed as `world_ptr` goes out of scope. Then we return our String.\n\nWe declared that we needed the `jni` crate, that means we need to include the crate in the `Cargo.toml` file. Open it up and add the following between the `[package]` and `[lib]` declarations.\n\n```\n[target.'cfg(target_os=\"android\")'.dependencies]\njni = { version = \"0.5\", default-features = false }\n```\n\nWe also need to tell the compiler what type of a library it should produce. You can specify this in the `Cargo.toml` file's `[lib]` section:\n\n```\n[lib]\ncrate-type = [\"dylib\"]\n```\n\nWe are now ready to build our libraries. Unlike with iOS, there is no handy universal Android library that we can make so we have to create one for each of our target architectures. We can then create symlinks to them from the Android project. You will need to use absolute paths to your libraries here, not relative ones, otherwise Android Studio will not be able to follow the link. Navigate to your `cargo` directory and run the following commands:\n\n```\ncargo build --target aarch64-linux-android --release\ncargo build --target armv7-linux-androideabi --release\ncargo build --target i686-linux-android --release\n\ncd ../android/Greetings/app/src/main\nmkdir jniLibs\nmkdir jniLibs/arm64\nmkdir jniLibs/armeabi\nmkdir jniLibs/x86\n\nln -s <project_path>/greetings/cargo/target/aarch64-linux-android/release/libgreetings.so jniLibs/arm64/libgreetings.so\nln -s <project_path>/greetings/cargo/target/armv7-linux-androideabi/release/libgreetings.so jniLibs/armeabi/libgreetings.so\nln -s <project_path>/greetings/cargo/target/i686-linux-android/release/libgreetings.so jniLibs/x86/libgreetings.so\n```\n\nNow, head back to Android Studio and open `GreetingsActivity.java`. We need to load our Rust library when the app starts, so add the following lines below the class declaration and before the `onCreate` method.\n\n```java\n   static {\n        System.loadLibrary(\"greetings\");\n    }\n```\n\nThis looks for a library called `greetings.so` inside the jniLibs directory and picks the correct one for the current architecture.\n\nOpen `res/layout/activity-greetings.xml`. In the `Component Tree` panel, highlight the `TextField` and open the `Properties` panel. Change the `ID` in the `Properties` panel to `greetingField`. This is how we are going to refer to it from our Activity.\n\n![Amend activity xml](assets/2017-09-21-rust-on-android/activity-greetings.png)\n\nReopen `GreetingsActivity.java` and amend the `onCreate` method to call our greetings function and set the text on the `greetingField` `TextField` to the response value.\n\n```java\n    @Override\n    protected void onCreate(Bundle savedInstanceState) {\n        super.onCreate(savedInstanceState);\n        setContentView(R.layout.activity_greetings);\n\n        RustGreetings g = new RustGreetings();\n        String r = g.sayHello(\"world\");\n        ((TextView)findViewById(R.id.greetingField)).setText(r);\n    }\n```\n\nBuild and run the app. If this is your first time in Android Studio, you may need to set up a simulator. When choosing/creating your simulator pick one with API 26. When the app starts, `Hello world` will be printed on your screen.\n\nYou can find the code for this on [Github](https://github.com/fluffyemily/cross-platform-rust).\n"
  },
  {
    "path": "newsletter/_posts/2017-07-27-browser-architecture-update.md",
    "content": "---\ntitle: Browser Architecture Update\nlayout: text\ndescription: The first of our newsletters introducing our work\nmailinglist: https://groups.google.com/d/msg/firefox-dev/ueRILL2ppac/RxR9lLPkAwAJ\n---\n\n# Browser Architecture Update\n\n[Link to mailing-list archive]({{ page.mailinglist }})\n\nIn the spirit of the Quantum Flow and Photon Newsletters - an update on what the Browser Architecture team is doing:\n\nThe Browser Architecture Team is new-ish - it’s existed since the re-org earlier this year, and its goal is to look ahead and make plans for our architecture.\n\nWe want to build the world’s best web browsers. Which is easy to say, but much harder to actually do.\n\nmozilla-central is a big complex software project. Its size and complexity slows us down, and it’s daunting to think about making it easier to reason about. We’ve often put off making life easy for ourselves because our users come first. We generally prioritize fixing bugs for users over internal work.\n\nBut sometimes you can go faster for users by focusing on going faster yourself. Part of what the Browser Architecture team would like is to tackle some of the hard problems of our own development velocity.\n\nWe also need to look at some problems that will take several years to address. Problems that don’t fit into any project area roadmap.\n\nWe have a mandate to help build the world’s best web browsers, but how? Our strategy is twofold:\n1. Kickstart a streamlined developer experience by removing roadblocks to immediate progress.\n2. Consciously plan our software architecture at an organization-wide level.\n\nSo what are the specific problems we are looking at?\n\n## Workflow\n\nUI development in mozilla-central somewhat mirrors the web development ecosystem, however one way we’ve fallen behind is in unit tests for frontend code. In mozilla-central we usually rely on integration tests rather than unit tests. Integration tests end up being slower and are more likely to fail randomly. In addition they steal focus and don’t allow for integration with code editors, to see test results as you type. We’d like to figure out how to write more unit tests in our codebase.\n\nWe’re looking at turning headless mode into a way to continue work whilst running tests, rather than needing to run a VM or turning to sword-fighting.\n\nWe’ve also spent some time looking at ways to make developing the Firefox frontend more like web development, with features like CSS hot reload and refreshing the UI without rebuilding and restarting the process.\n\nIn parallel, we’re also looking at ways that we can improve Gecko development efficiency, especially on build times and debugging (Rust, rr, gdb on Mac).\n\n## XUL and XBL\n\nOur user interface language has problems. It’s uncared for, so it’s buggy and sometimes we don’t land fixes that we know about. It’s largely undocumented or worse, documented wrongly, so it’s hard to fix. Also it behaves almost like the web, which can lead web developers to think they understand it better than they really do.\n\nWe care about HTML, so that’s documented and optimized. XUL? Not so much.\n\nThe first talk I heard about Servo was in 2013. At the Q&A at the end someone asked about XUL and Servo — the answer was \"There is no XUL in the product at all\". Applause from audience. So the tide has been against XUL for at least 4 years, but so far we’ve not made headway in tackling it. The Browser Architecture team is approaching it from various angles:\n\nCan we remove XUL and XBL piece by piece? We are experimenting with a couple of approaches with the preferences UI. Could we convert the XBL to JS modules and Web Components ‘in place’? Alternatively, how much work is it to rewrite it in HTML, using something like React? Or more dramatically, how far up the stack should our use of Rust go? Maybe we should consider a UI generated by Rust?\n\nAlternatively, what’s the minimum we could do to just fix and document what we’ve got. Perhaps we should just focus on removing XBL? Maybe the plans above are too much work and we need to invest elsewhere entirely?\n\n## User Data Storage and Sync\n\nWe don’t have a good story for storage of user data. We wouldn’t sit down to design the profile directory and come up with what we have today, and we need a better story to be able to sync more readily to Firefox for Android and iOS and to adapt to unanticipated and evolving product needs.\n\nRecently we've been exploring Project Mentat, which promised some advances in helping us rapidly evolve data storage, and making syncing of data less troublesome. We decided that porting all of Firefox’s storage to Mentat was too much work right now, but the problem of improving user data storage and sync remains.\n\nSo we’re looking at the big picture of storage and sync and how we can share data not just between desktop and mobile, but also to help us develop and test new products in the market.\n\n## A Monolithic Platform\n\nWe’re considering what it would look like to turn Gecko inside out so it’s more modular, can be reused in smaller parts and is easier to test, as with Rust crates in Servo. Changing that is hard — really hard — but the potential value is also significant.\n\nWe're looking at existing Firefox components and wondering if we can extract them (or rewrite them) as separate components that we could reuse in situations where we can't rely on the Gecko runtime environment, and we’re looking at the work on GeckoView to see how it can help us reuse Gecko outside Firefox.\n\n----\n\nThere are a lot of ideas there. Maybe some sound cool, maybe some sound crazy. We’re planning on writing up our findings on each of the areas so look out for future posts where we drill down into some of these ideas.\n\nThanks\n\nJoe Walker (on behalf of the Browser Architecture Team)\n\nFeedback is welcome. Either reply here or find Nick Alexander, Brendan Dahl, Brian Grinstead, Rob Helmer, Randell Jesup, Myk Melez, Richard Newman, Victor Porof, Emily Toop, Dave Townsend, Peter Van Der Beken, or Joe Walker near some watercooler (like #browser-arch).\n"
  },
  {
    "path": "newsletter/_posts/2017-08-24-browser-architecture-newsletter-2.md",
    "content": "---\ntitle: Browser Architecture Newsletter 2\nlayout: text\ndescription: A follow-up with updates on XBL Conversion, Storage and Sync and Workflow Improvements\nmailinglist: https://groups.google.com/d/msg/firefox-dev/ueRILL2ppac/RxR9lLPkAwAJ\n---\n\n# Browser Architecture Newsletter 2\n\n[Link to mailing-list archive]({{ page.mailinglist }})\n\nWelcome to the second Browser Architecture Newsletter! Since our [last update](https://groups.google.com/forum/#!msg/firefox-dev/ueRILL2ppac/RxR9lLPkAwAJ;context-place=forum/firefox-dev), we have a [new website](https://mozilla.github.io/firefox-browser-architecture/) which includes posts about some of the [problems with XUL](https://mozilla.github.io/firefox-browser-architecture/text/0003-problems-with-xul.html), [extracting Necko](https://mozilla.github.io/firefox-browser-architecture/text/0002-extracting-necko.html) as a standalone component, and [comparing](https://mozilla.github.io/firefox-browser-architecture/text/0004-xbl-web-components.html) XBL with Web Components.\n\nThe website is built from [this GitHub repository](https://github.com/mozilla/firefox-browser-architecture) of Markdown files, and we’re using pull requests to review and revise our research proposals and analyses. Do you have ideas for improvements? File an issue on the repo or contact us in #browser-arch!\n\n## XBL Conversion\n\nWe'd like to reduce or even remove XBL and XUL from the tree. It's becoming clear that replacing XBL is more important and easier than replacing XUL, so there are a few projects ongoing to learn more about it.\n\nA conversion of the behavioral `<preference>` binding is in progress at [Bug 1379338](https://bugzilla.mozilla.org/show_bug.cgi?id=1379338). Although this work started as a prototype, it resulted in a more straightforward implementation than we currently have, so is on the path to being landed.\n\nAs mentioned above, there’s some new [documentation](https://mozilla.github.io/firefox-browser-architecture/text/0004-xbl-web-components.html) comparing XBL with Web Components. There’s also a [visual tree](https://bgrins.github.io/xbl-analysis) of the bindings we use in Firefox and what features each uses.\n\nA prototype of replacing some XBL UI components in about:preferences with Custom Elements is attached to [Bug 1392367](https://bugzilla.mozilla.org/show_bug.cgi?id=1392367). This relies on a [forked version](https://github.com/webcomponents/custom-elements/compare/master...bgrins:firefox-browser-chrome?expand=1) of the [custom elements polyfill](https://github.com/webcomponents/custom-elements) made to work in XUL documents.\n\nA prototype of converting `<tabbrowser>` to a JS class is in progress at [Bug 1392352](https://bugzilla.mozilla.org/show_bug.cgi?id=1392352). This would cause gBrowser to become a JS object instead of a DOM Node. This conversion is semi-automated [by a script converting the binding to a JS class](https://github.com/bgrins/xbl-analysis/blob/ad87d682b937620b1c129e49c4081483c7074540/scripts/build-custom-elements.js).\n\n## Storage and Sync\n\nWe're in the process of cataloging and analyzing some of Firefox's sprawling storage technologies (45+ different data stores and dozens of formats on desktop alone), and how they relate to Sync and across platforms. We're hoping to find ways to make this situation simpler, more reliable, and easier to extend.\n\nYou can see the beginnings of our knowledge capture [here](https://github.com/mozilla/firefox-data-store-docs) (PRs welcome!), and a [spreadsheet](http://bit.ly/2vpZI9e) showing how data types interact with Sync and mobile platforms.\n\nHuge thanks to the Sync team and data and product folks across the org for helping with our research.\n\n## Workflow Improvements\n\nLately I've been paying extra attention to development papercuts when working on bugs, especially issues when using ./mach run without passing a profile. Here’s a list of recent changes on that front:\n\n*   You can [open the Browser Toolbox](https://groups.google.com/d/msg/firefox-dev/678mrnS6120/KXcP18ZUCAAJ) in a local build without flipping prefs.\n*   The default browser prompt and about:config warning screen [no longer show up](https://groups.google.com/d/msg/firefox-dev/kPwA1y-7BpI/gX6rvhEjBQAJ) for the scratch_user (the profile used when you do ./mach run without a profile).\n*   You can flip any pref for the scratch_user with the --setpref option, or by adding them in your [machrc file](https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/mach#Does_mach_have_its_own_configuration_file). For example, to make sure I always see dump output I added this to my ~/.mozbuild/machrc file:  \n    `[runprefs]  \n    browser.dom.window.dump.enabled=true`\n*   There’s a [new keyboard shortcut](https://groups.google.com/d/msg/firefox-dev/Tme95bp3EHY/ow-l077FAAAJ) to do a restart + session restore in local builds.\n*   The Browser Console is [now part of session store](https://bugzilla.mozilla.org/show_bug.cgi?id=1388552), so it will reopen after using the above keyboard shortcut.\n"
  },
  {
    "path": "newsletter/_posts/2017-09-22-browser-architecture-newsletter-3.md",
    "content": "---\r\ntitle: Browser Architecture Newsletter 3\r\nlayout: text\r\ndescription: A newsletter on architecture review, XBL Conversion, Storage and Sync, Workflow Improvements and a developer survey\r\nmailinglist: https://groups.google.com/forum/#!topic/firefox-dev/p9rTlfFUXlQ\r\n---\r\n\r\n# Browser Architecture Newsletter 3\r\n\r\n[Link to mailing-list archive]({{ page.mailinglist }})\r\n\r\nWelcome to the third Browser Architecture Newsletter! Since our [last\r\nupdate](https://mozilla.github.io/firefox-browser-architecture/posts/2017-08-24-browser-architecture-newsletter-2.html),\r\nwe've continued to investigate moving away from XBL and started to\r\ndocument what we're talking about when we talk about XUL problems. We're\r\nalso working with members of the Sync, AS, and Lockbox teams to figure\r\nout what the future of storage and syncing looks like for Mozilla.\r\n\r\nA lot of the issues that the Browser Architecture team are digging into\r\nare larger than our team. Our goal is to discuss and review entire\r\nproduct level architecture issues and build consensus around solutions.\r\nWe're interested in engaging with engineers around the organization.\r\nWe're actively reaching out to folks but if you want to talk to us we'd\r\nsure love to hear from you. You can find us in \\#browser-arch on IRC or\r\nSlack.\r\n\r\n## Architecture Review\r\n\r\nWe are asking for comments on our [proposed\r\nprocess](https://mozilla.github.io/firefox-browser-architecture/text/0006-architecture-review-process.html)\r\nfor when our team performs architecture reviews. The process is in draft\r\nright now so if you have suggestions for how to improve it please let us\r\nknow.\r\n\r\nIf you've got ideas, questions, or concerns, talk to\r\n[jwalker](https://mozillians.org/en-US/u/jwalker/).\r\n\r\n## XBL Removal\r\n\r\nWe've filed a [meta\r\nbug](https://bugzilla.mozilla.org/show_bug.cgi?id=1397874) for our\r\nde-XBL work, which includes a variety of prototypes and other\r\ninvestigations, including: moving XBL bindings to Custom Elements,\r\nmoving XBL bindings to JS modules, and performance comparisons of XBL\r\nbindings to a custom elements polyfill. You can view all the prototypes\r\nand investigations we're working on by viewing the meta bug's\r\ndependencies.\r\n\r\nWe're getting ready to run through a design review for our XBL\r\nreplacement plans. The review will be chaired by Dave Townsend and\r\ninclude a panel of experts on both Gecko and Firefox. The review process\r\nitself is a work in progress, and our XBL Removal review is a trial run\r\nto help us refine it. Look for more information on the review itself\r\nsoon when we get further along in the process.\r\n\r\nIf you've got ideas, questions, or concerns, talk to\r\n[bgrins](https://mozillians.org/en-US/u/bgrinstead/).\r\n\r\n## Storage and Sync\r\n\r\nWe've [captured](https://github.com/mozilla/firefox-data-store-docs) a\r\nbunch of knowledge about Firefox's data stores. That effort is feeding\r\ninto some roadmapping work with the Sync and Activity Stream teams. We\r\nhope to have some concrete roadmap documentation underway by our next\r\nnewsletter, as well as some stage-setting blog posts.\r\n\r\nWe expect that the future will include some cross-platform Rust storage\r\nservices, so we've been researching how this looks in practice.\r\n[Deploying an Rust library on\r\niOS](https://mozilla.github.io/firefox-browser-architecture/experiments/2017-09-06-rust-on-ios.html)\r\nis a short tutorial describing how to build and deploy a Rust library\r\nfor use inside an iOS app.\r\n\r\nIf you've got ideas, questions, or concerns, talk to\r\n[rnewman](https://mozillians.org/en-US/u/rnewman/) or\r\n[fluffyemily](https://mozillians.org/en-US/u/etoop/).\r\n\r\n## Workflow Improvements\r\n\r\n*   [You can\r\n    now](https://groups.google.com/d/msg/firefox-dev/L5D3YSBxqh8/pjonQLutCwAJ)\r\n    open Browser Toolboxes on more than one instance of Firefox at the\r\n    same time\r\n*   The ‘no popup autohide' mode that can be toggled in the Browser\r\n    Toolbox to inspect popups [now automatically\r\n    resets](https://bugzilla.mozilla.org/show_bug.cgi?id=1251658) when\r\n    the toolbox closes if you clicked the button to enable it. You can\r\n    also flip this pref yourself with \\`ui.popup.disable\\_autohide\\`\r\n*   dump() is [now enabled by\r\n    default](https://bugzilla.mozilla.org/show_bug.cgi?id=1395711) in\r\n    local builds - kudos to :kmag for doing this\r\n\r\nIf you've got ideas, questions, or concerns, talk to\r\n[bgrins](https://mozillians.org/en-US/u/bgrinstead/) or\r\n[nalexander](https://mozillians.org/en-US/u/nalexander/). If you have\r\nsuggestions or pain points about doing Gecko development\r\n(non-front-end), please contact\r\n[jesup](https://mozillians.org/en-US/u/jesup/).\r\n\r\n## Front-end Developer Survey\r\n\r\nWe've sent out a survey to front-end developers asking them about their\r\nworkflow and productivity issues. The results of this survey will help\r\nus target future workflow and development improvements.\r\n\r\nIf you're an employee working on front-end Firefox code and you haven't\r\nreceived the survey, please let\r\n[mossop](https://mozillians.org/en-US/u/Mossop/) or\r\n[nalexander](https://mozillians.org/en-US/u/nalexander/) know so we\r\ncan get it to you! We're still discussing if we should open this survey\r\nup to contributors at this time. If you've got other ideas, questions,\r\nor concerns, [mossop](https://mozillians.org/en-US/u/Mossop/) and\r\n[nalexander](https://mozillians.org/en-US/u/nalexander/) are also the\r\nright people to contact.\r\n"
  },
  {
    "path": "newsletter/_posts/2017-10-19-browser-architecture-newsletter-4.md",
    "content": "---\ntitle: Browser Architecture Newsletter 4\nlayout: text\ndescription: A newsletter on architecture review, XBL Conversion, Storage and Sync, Workflow Improvements and a developer survey\nmailinglist: https://groups.google.com/forum/#!topic/firefox-dev/CLFtj8qUSv8\n---\n\n# Browser Architecture Newsletter 4\n\n[Link to mailing-list archive]({{ page.mailinglist }})\n\nWelcome to the fourth Browser Architecture Newsletter!\n\nA lot of the issues that the Browser Architecture team are digging into are\nlarger than our team. Our goal is to discuss and review entire product level\narchitecture issues and build consensus around solutions. We’re interested in\nengaging with engineers around the organization. We’re actively reaching out to\nfolks but if you want to talk to us we’d sure love to hear from you. You can\nfind us in `#browser-arch` on IRC or Slack.\n\n## XBL Removal\n\nSince our last update, we've completed the design review for [a plan to move\naway from XBL in Firefox](https://mozilla.github.io/firefox-browser-architecture/text/0007-xbl-design-review-packet.html).\nThanks to the long list of people who guided the creation of that plan, and the\nchair and panel for putting it through its paces. We're working through some\nfollow-up investigations; expect a separate email with more details soon.\n\n## Storage and Sync\n\nWe've been doing lots of writing on this topic. Emily has [documented](https://github.com/fluffyemily/cross-platform-rust) and\nblogged about getting Rust libraries up and running on multiple platforms ([iOS](https://mozilla.github.io/firefox-browser-architecture/experiments/2017-09-06-rust-on-ios.html),\n[Android](https://mozilla.github.io/firefox-browser-architecture/experiments/2017-09-21-rust-on-android.html)), which will pave the way for shared storage implementations across\nmobile and desktop products. Richard has published the [first in a series of\nblog posts about sync](https://medium.com/@rnewman/thinking-about-syncing-part-1-timelines-7f758e2bd676).\n\nA [roadmap review](https://mozilla.github.io/firefox-browser-architecture/text/0006-architecture-review-process.html) is planned for this quarter; thanks to everyone — a dizzying\narray of engineers, managers, product folks, and Deep Syncers — who've been\ncontributing to that process. Let rnewman know if you're not involved and would\nlike to be.\n\n## Front-end Developer Survey\n\nWe sent out a survey for front-end developers to try to understand how to\ntarget our architecture and workflow improvements. The results will be\npublished soon, but one thing we found was that a lack of documentation is\nholding developers back. We'll be digging into this more in the future.\n"
  },
  {
    "path": "newsletter/_posts/2017-11-29-browser-architecture-newsletter-5.md",
    "content": "---\ntitle: Browser Architecture Newsletter 5\nlayout: text\ndescription: A newsletter on Storage and Sync, XBL Removal and Workflow Improvements\nmailinglist: https://groups.google.com/forum/#!topic/firefox-dev/XKp3EthdJ60\n---\n\n# Browser Architecture Newsletter 5\n\n[Link to mailing-list archive]({{ page.mailinglist }})\n\nWelcome to the exciting fifth installment in our series of Browser Architecture Newsletters!\n\n## Storage and Sync\n\nWe are pleased to report that the Sync and Storage roadmap proposal successfully passed review by dcamp. Thanks to the 20+ people from around the organization who helped us build consensus.\n\nThe roadmap establishes that storage and syncing of user data is a pillar of the Firefox ecosystem, warranting holistic and long-term attention, and outlines both where we’d like to end up and some ideas for how to get there.\n\nWe expect this roadmap to lead to a number of subsequent concrete design reviews, as we begin with prototyping, further competitive analysis, and more coordination across Mozilla. The roadmap doc itself has been lightly cleaned up and [published](https://mozilla.github.io/firefox-browser-architecture/text/0008-sync-and-storage-review-packet.html) alongside [our other docs](https://mozilla.github.io/firefox-browser-architecture/).\n\nIf you’re working on improving storage or syncing of data in any of our products or projects, on any platform, or if you’re currently struggling to work with what currently exists, then we’d like to hear from you and align our efforts.\n\nYou can find us in Slack or IRC (#browser-arch), or come find rnewman, etoop, rfkelly, markh, or adavis.\n\nWe’ll be meeting to talk in more detail about the roadmap, the present, and the future [at the All Hands](https://austinyallhands2017.sched.com/event/CyOK/sync-and-storage). We would welcome your participation.\n\n## XBL Removal\n\nOur work to remove XBL is proceeding apace. We’ve started by removing bindings that are never or only rarely used. Work to convert XBL bindings to custom elements is currently stalled while we get custom elements working in XUL.\n\nYou can track how this project is progressing with some [lovely graphs](https://bgrins.github.io/xbl-analysis/graph/). If you’d like to help with the effort, you can grab a bug tagged with [[xbl-available]](https://mzl.la/2AyTxCG).\n\n## Workflow Improvements\n\nRunning mochitest-plain in headless mode is now supported on Windows, MacOS, and Linux. Use the “--headless” flag with “./mach test” to try it out.\n\n## Austin All Hands\n\nWe have two meetings planned for the All Hands. You can see these in Sched here:\n\n* [Austin Y'all Hands 2017: Sync and Storage](https://austinyallhands2017.sched.com/event/CyOK/sync-and-storage)\n* [Austin Y'all Hands 2017: XBL and XUL Removal](https://austinyallhands2017.sched.com/event/CyLC/xbl-and-xul-removal)\n"
  },
  {
    "path": "newsletters.xml",
    "content": "---\nlayout: null\n---\n<?xml version=\"1.0\" encoding=\"UTF-8\" ?>\n<rss version=\"2.0\">\n<channel>\n  <title>Mozilla Browser Architecture Newsletters</title>\n  <description>The published newsletters from the Mozilla Browser Architecture team.</description>\n  <link>{{ site.url }}{{ site.baseurl }}</link>\n{% for post in site.categories.newsletter %}\n  <item>\n    <title>{{ post.title }}</title>\n    <description>{{ post.description }}</description>\n    <link>{{ site.url }}{{ site.baseurl }}{{ post.url }}</link>\n    <guid isPermaLink=\"true\">{{ site.url }}{{ site.baseurl }}{{ post.url }}</guid>\n    <pubDate>{{ post.date }}</pubDate>\n  </item>\n{% endfor %}\n</channel>\n</rss>\n"
  },
  {
    "path": "text/0000-template.md",
    "content": "---\ntitle: Documentation Template\nlayout: text\n---\n\n# Documentation Template\n\n## Problem\n\nA short lay summary of the problem space and the stakeholders.\n\nOur documents should have a front matter containing a title, which should be\nthe same as the first heading (using `#`).\nOther headings will generally be at the `##` or `###` levels.\n\n\n## Proposal\n\nExplanation of the solution to the problem.\n\n\n## Drawbacks / Unresolved Questions\n\nIt's a rare proposal that perfectly meets the problem, so we'll probably need\nto document the drawbacks.\n\n\n## Alternatives\n\nCompetitive analysis suggesting alternatives, costs, and opportunities.\n\n\n## Links to discussion\n\n* Mailing list discussion?\n* Original GDocs document\n"
  },
  {
    "path": "text/0001-documenting-output.md",
    "content": "---\ntitle: Documenting our Output\nlayout: text\n---\n\n# Documenting our Output\n\n## Problem\n\nThe Browser Architecture team needs to publish its findings clearly. We would like anyone in Mozilla to be able to:\n\n* Find what we’ve learned so far\n* Find what we’re working on\n* Leave feedback\n\nThe solution should:\n\n* Be low friction for contribution\n* Be flexible in how the result is published\n* Ensure that documents can be found easily\n\n\n## Proposal\n\nBrowser Architecture team will document its output like this:\n\n1. The discussion phase involves conversations in the browser-arch [Google Group](https://groups.google.com/a/mozilla.com/forum/#!forum/browser-arch), [IRC channel](https://www.irccloud.com/#!/ircs://irc.mozilla.org:6697/%23browser-arch), [Slack channel](https://mozilla.slack.com/messages/C5F80LV0C/), etc - when a semi-permanent record of current thinking is needed, we use GDocs, stored in [the Browser Architecture folder](https://drive.google.com/drive/u/1/folders/0BzQINYlY78CtbGVqVDVlZlNJX0k).\n2. When we've come to some internal consensus (note this doesn't mean that the proposal is final) we convert our GDocs to Markdown/Git to create an 'RFC' and get feedback from management and the wider Mozilla developer community as appropriate.\n  * Use [this template](0000-template.md)\n3. As there are updates we keep the Markdown up to date\n4. We may publish to a wider audience using a blog post or GitBook as required\n\n\n## Drawbacks / Unresolved Questions\n\nIt is possible that GDocs has such a low update friction that people will resent the PR process imposed by Github. If this turns out to be true then we will need to update this process.\n\nThis process is somewhat modelled on the [Rust RFC](https://github.com/rust-lang/rfcs) process. It isn't certain that we need the numeric prefix.\n\n\n## Alternatives\n\n* Create an index to our documentation using Google Docs, and stay in that world.\n* Use Medium and or RSS to publish output.\n\n\n## Links to discussion\n\n* [Mailing list discussion](https://groups.google.com/a/mozilla.com/forum/#!topic/browser-arch/FOtfYVHbgfo)\n"
  },
  {
    "path": "text/0002-extracting-necko.md",
    "content": "---\ntitle: Extracting Necko\nlayout: text\n---\n\n# Extracting Necko\n\n## Proposal\n\nWe should extract Necko for reuse in background services and on mobile platforms.\n\n### Motivation\n\nWe expect that Mozilla will need to build consistent native applications and background services across platforms, including services and applications that don't run Gecko atop a profile directory. To do this it would be beneficial (in theory) to leverage our existing investment in our network layer, avoiding the confusing situation of having multiple ways to configure network settings, and avoiding the possibility of different behaviors in different parts of an application.\n\nFurthermore, we expect the question of reuse to recur. Necko is sometimes characterized as being a relatively self-contained piece of mozilla-central's compiled code, so it should be one of the easiest to reuse. Also, its interdependencies are probably applicable to other components, so it's a good opportunity to learn.\n\nBreaking out a component for reuse is, for our purposes, equivalent to embedding: it's taking the component and driving it in a different context. Specifically, we are interested in those contexts being (a) Rust, (b) not necessarily desktop. An alternative approach is to build a new Gecko application on top of a slimmed-down libxul. In the case of Necko, that might be something like a cURL clone. This kind of reuse is a known quantity, so we won't discuss it here.\n\n## References\n\n* Lots of very old documents:\n  * <https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Necko/Embedding_Necko>\n  * <https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Necko>\n  * <https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Necko/Architecture> \n* Discussions with rhelmer and jduell\n* Examination of `/netwerk`\n\n## Analysis: obstacles to embedding\n\n* In its early days, 15+ years ago, Necko was intended to be embeddable; in recent years that has not been an aim of the component leads. Meeting the needs of Gecko and Firefox is more important. (The definition of 'embeddable' here still carries the obligations of Gecko-style initialization.)\n* As noted in the \"Embedding Necko\" document, Necko depends on Gecko features:\n  - XPCOM and NSPR.\n  - mDNS, `NetUtil.jsm`, and some HTTP functionality requires a JS runtime. This is feasible to eliminate if needed.\n  - Docshell.\n  - If cookies are desired, they must be provided via the nsICookieService interface, which is baked into Necko. One can build Necko without, via `--disable-cookies`, but this is a build-time decision. One could conceivably provide one's own implementation of the cookie service, but the interface requires a stateful synchronous global singleton.\n  - Necko depends on the Mozilla preferences service (libpref) for configuration, which is a privileged singleton that itself depends on the observer service (for startup and profile events) and the directory service (for profile directory and app locations).\n* Because of its dependency on the profile (both for reading state and as a key for the disk cache) and flushing files to disk for prefs, it's not safe to use Necko from two distinct processes simultaneously for the same application.\n* Because of its extensive use of XPCOM singleton patterns, it's not possible to have two independent Neckos in the same process; they need separate processes and separate profile directories. Some independence can be achieved by carefully configuring channels, but it's not isolation.\n* It's not possible to use Necko without a profile service (*e.g.*, to make a network request without touching disk).\n* Parts of Necko are still single-threaded, requiring new channels to be created on the main thread, and dispatching callbacks to the main thread.\n* Necko's init and deinit are managed by observer notifications, which is not ideal for embedding outside of Gecko.\n\n# Proposal\n## Proposal\n# Conclusion\n\nNecko is a very feature-rich C++ networking library that is directly targeted at meeting the needs of single-user, single-main-process, single-profile, interactive applications built on a persistent profile directory in the traditional Mozilla model. It is not well-suited for use outside of the Gecko lifecycle — indeed, almost every aspect is designed for the Gecko world — and would require substantial effort in decoupling to be suited for embedded use. Even if so decoupled, it would still mandate the use of XPCOM and the NSPR.\n\nTo extract Necko for reuse outside of Gecko, by non-XPCOM-based applications, would — almost by definition — require coming up with a plan to deal with XPCOM. That plan might look a lot like continuing historical efforts on deCOMtamination.\n\nNecko's data structures, interface definitions, error codes, macros, etc. *are* XPCOM's, and so eliminating XPCOM dependency from the module is likely to involve touching almost every line of code. [Automated tools to do so have been proposed](https://brendaneich.com/2006/10/mozilla-2/).\n\n(Going further to eliminate NSPR and mfbt data structures, yielding a more standard C++ library, would be further work still.)\n\nThis is particularly relevant when we consider oxidation — the replacement of C++ and JS components of Firefox with Rust — and even more so when we consider the use of such components both outside and inside Firefox *from* Rust (rather than from existing non-Rust callers).\n\nThere is ongoing work on landing Rust components in Firefox; most obstacles there are around XPCOM integration for Rust code, which currently requires writing lots of routine boilerplate by hand.\n\nThere is also some initial scoping of work to extract and use Gecko C++ components from Servo, which is a similar problem to using Gecko C++ components from Rust background services or mobile applications. Those estimates are large (but not yet published), on the order of 3-5 engineer years for non-trivial components.\n\nExtracting such a module is not routine work: delicate balances need to be found between maintaining useful integration with Gecko (*e.g.*, Necko's `nsIChannel` permeates the codebase) versus designing for embedding, and between reimplementing Gecko dependencies in Rust for reuse versus reworking the component to avoid them.\n\nOur position is that extracting Necko from Gecko for reuse would be a significant amount of work. Doing so in a way that allows its use from ordinary Rust consumers, without shipping XPCOM/NSPR/*etc*., would be even more so.\n\nThat amount of work is certainly much more than simply using the existing well-tested and idiomatic networking libraries available to Rust code. Given that the benefits of using Necko in its entirety (instead of *e.g.*, just using Hyper for HTTP) are relatively small, our conclusion is that we **should not attempt to reuse Necko outside of Gecko**.\n\nStandalone Rust components should use Rust networking libraries of equivalent functionality, *e.g.*, Tokio/Hyper (already vendored for Servo) or libpnet. Useful Necko functionality should be ported to Rust as needed. Those Rust reimplementations can be imported back into Necko like any other piece of oxidized code.\n\n## Alternatives\n\nAlternatives are, variously:\n\n* Do the work to extract Necko. Some significant benefit would need to be articulated in order for this to be worthwhile.\n* Use Gecko for all network-related background services. We already know that this is not feasible, both for application size reasons (Gecko is large), and for technical lifecycle reasons as articulated above. Furthermore, this argument ends up as \"use Gecko for everything everywhere\", which is not useful.\n\n## Links\n\nThis document was originally drafted in the [\"Answered questions\" Google Doc](https://docs.google.com/document/d/1RIMbe0yJGNywFTf1n5ka35JpLtrkE3lHz8iip6jH_VE/edit#).\n\n## Colophon\n\nThis document was written by Richard Newman (rnewman), reviewed by Jason Duell (jduell), Rob Helmer (rhelmer), Joe Walker, and Myk Melez.\n"
  },
  {
    "path": "text/0003-problems-with-xul.md",
    "content": "---\ntitle: Problems with XUL\nlayout: text\n---\n\n# Problems with XUL\n\nXUL is often mentioned as a source of problems for developers. This document\naims to list the different kinds of problems that exist.\n\n## XUL ownership\n\nXUL is largely unowned. While there are developers who know and understand the\nXUL code they are few and mostly dedicated to other projects at present.\n\n## XUL bugs\n\nXUL contains bugs. This isn’t due to anything particularly special about XUL but\nwhen we encounter bugs we frequently choose to not fix them, instead trying to\nfind workarounds in the UI instead. See [bug 354527](https://bugzilla.mozilla.org/show_bug.cgi?id=354527)\nfor example.\n\n## No new features\n\nThere is no new feature implementation going on in XUL compared to HTML which\nreceives constant improvements to meet the demands of the web.\n\n## Performance issues\n\nIt is thought that XUL’s performance is worse than that of HTML where we have spent\nlarge amounts of time optimising for the web but little amount of time optimising\nfor XUL.\n\n## Availability of example code and help\n\nXUL is a proprietary technology developed by Mozilla and only used by Mozilla. When\ndevelopers want to find solutions to problems or implement new features the only\nexample code and documentation they have available is our code and our documentation.\n\n## Learning curve\n\nRelated, when new developers join Mozilla it is almost certain they will not have\nheard of or used XUL before and so there is a learning curve to understanding and\nbecoming proficient in the language.\n\n## Recruiting challenge\n\nEven if the learning curve is small relative to the productive output of an engineer\nwho has learned the technology, recruiting for a role that involves using a\nproprietary language is more difficult than recruiting for a role that involves using\n(and learning, if needed) a popular language. This applies to both new contributors\nas well as employees.\nBesides the cost of learning the language and the uncertainty about whether the work\nwill be enjoyable, which may dissuade some candidates, the more important factor is the\nopportunity cost for career development relative to developing expertise in a popular\nlanguage: engineers with “XUL experience” don’t benefit from it when looking for their\nnext job (or even position within the company).\n\n## Frameworks\n\nOften developers want to use popular JS frameworks like jQuery and React to develop new\nfeatures. Commonly these frameworks are built for HTML and don’t work correctly in XUL\ndocuments.\n\n## SVG and HTML in XUL\n\nBecause XUL is missing certain features often developers want to embed HTML or SVG\nelements inside of XUL. Because of differences in the box model this sometimes leads\nto layout not working correctly.\n\n## Gecko Complexity\n\nXUL increases the complexity of Gecko generally, which imposes costs on Gecko\ndevelopment and maintenance (such as the Stylo work).\n"
  },
  {
    "path": "text/0004-xbl-web-components.md",
    "content": "---\ntitle: XBL and Web Components\nlayout: text\n---\n\n# XBL and Web Components\n\nXBL is a technology used to implement reusable widgets in XUL. Web Components are a set of specifications that provide modern and standards-based alternatives for doing the same thing with HTML. For the purposes of this document, \"Web Components\" refers to the [Custom Elements](https://w3c.github.io/webcomponents/spec/custom/) and [Shadow DOM](https://w3c.github.io/webcomponents/spec/shadow/) specs, which map relatively closely to how the Firefox frontend is using XBL.\n\nWhat follows is an initial analysis of what a migration away from XBL and to Web Components may look like. We will need to spend more time prototyping this within the Firefox UI to get a clearer picture about the viability of a migration.\n\n## What bindings are we talking about?\n\nMost of the relevant bindings are declared in [toolkit/content/widgets](https://github.com/mozilla/gecko-dev/tree/master/toolkit/content/widgets).\n\nThis analysis is focused on widely used and relatively small XBL components that could potentially be swapped out more-or-less in place with corresponding Web Components. For instance: buttons, labels, input fields, etc.\n\nComponents that are only used once or are implemented mostly in C++ are ignored in this analysis. For instance: tabbrowser, trees, etc. Most of the same principles that apply to smaller components should also apply to them, but we may also consider different options for moving them away from XBL.\n\n## Auditing the XBL feature set\n\nFrom the [Introduction to XBL](https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XUL/Tutorial/Introduction_to_XBL), a binding has five types of things that it declares: Content, properties, methods, events, and style.  But it's helpful to look at specific features used in Firefox and comparing how they map back to Web Components.\n\nWhich bindings use which features is visualized at https://bgrins.github.io/xbl-analysis/. The scripts to generate that tree and also to semi-automate translation of bindings can be found at https://github.com/bgrins/xbl-analysis.\n\n### XBL features that map to Web Components features\n\n1. `<property>`: These map to JS getters and setters on the Custom Element class (‘val’ name is magic in xbl, would need to keep same name to translate directly)\n2. `<method>`: These map to functions on the Custom Element class\n3. `<constructor>` This maps to connectedCallback on the Custom Element class (which is called when the element is inserted into a document).\nIn XBL, the constructor fires after the element is inserted into the document (since the binding is attached through a CSS property).  However, the constructor will wait some additional time to fire, unless if the element is accessed via JS in which case it will run immediately\n4. `<destructor>`: These map to `disconnectedCallback` on the Custom Element class\n5. `[extends]`: A way to extend the functionality of another binding, i.e. `extends=\"chrome://global/content/bindings/general.xml#basetext\"`. On a surface level this appears to map directly to how the ‘extends’ keyword in ES6 classes behavior with Custom Elements, but more investigation is needed.  There’s a small set of tags where you can do `extends=\"xul:button\"` but that’s aliasing the same thing.\n6. `<handlers>`: Can be implemented with addEventListener or possibly a helper on the base element i.e:\n```\nhelpers: {\n  \"click\": () => { /* bubble by default*/ }\n  \"focus|capturing\": () => { /* copy in xbl implementation */ }\n}\n```\n7. `[inherits]`: Is a way to declaratively copy attributes from a binding parent to content and keep them in sync, similar to the `<observes>` element. For example, this copies the feedURL from the parent into the 'value' field of the label and also observes future changes: `<feed feedURL=\"http://foo.com\"><xul:label xbl:inherits=\"value=feedURL,tooltiptext=feedURL\"></feed>`. The same thing could be built on top of Custom Elements using a MutationObserver and some glue code to set it up in the `connectedCallback`.\n8. Scoped CSS Styling: this can be done with Shadow DOM\n9. Insertion Points: XBL and Shadow DOM appear to use similar models but different keywords. Further investigation is needed.\n\n### XBL features that don't map Web Components features\n\n1. `[implements]`: This is a way for elements with XBL bindings to implement an XPCOM interface.  XBL also auto generates the Qi bits for you. One example usage is that labels implement complex cropping behavior via `implements=\"nsIDOMXULLabelElement\"`, but there are many others (panels, menus, scrollbars). There are around 39 instances of this feature, and this will need to be handled on a case-by-case basis. But we have some options:\n\n  * Write chrome-only webidl bindings for each of the individual nsiDOMXULFoo classes\n  * Replace things like `implements=\"nsIObserver\"` with just a separate object that in fact implements it\n  * For some input elements like checkboxes, we could get rid of the xul interface since the corresponding HTML element has the same behavior already\n\n2. Extension Compat: For instance, the Tab Center test pilot addon causes tab browser to become vertical via inheritance with some changes.  This allows us to use the same tag name / markup so that it doesn’t need to re-write CSS. However, with the migration to Web Extensions this won’t be necessary in the general case.\n3. Binding via CSS: `-moz-binding` allows a XBL binding to be attached or detached to any element through CSS.  Web Components would be tied to HTML (based on the particular tag being created). This feature is used in [at least one place to swap an element to a different binding when setting a property](http://searchfox.org/mozilla-central/rev/7cc377ce3f0a569c5c9b362d589705cd4fecc0ac/toolkit/content/widgets/text.xml#47), but we can work around it.\n\n## XBL Usage in Firefox\n\nHow many times are individual bindings used?  We've written a module to track how often XBL (and XUL) are used, and a modified version of the test runner that keeps a count throughout the mochitest-browser suite. [A sorted list of used bindings is attached on the bug](https://bugzilla.mozilla.org/show_bug.cgi?id=1376546), but here are the top few bindings:\n\n    url(\"chrome://global/content/bindings/text.xml#text-label\")\n    url(\"chrome://global/content/bindings/general.xml#image\")\n    url(\"chrome://global/content/bindings/toolbarbutton.xml#toolbarbutton\")\n    url(\"chrome://global/content/bindings/menu.xml#menuitem\")\n    url(\"chrome://global/content/bindings/scrollbox.xml#autorepeatbutton\")\n    url(\"chrome://global/content/bindings/button.xml#button\")\n\n## Performance Considerations\n\nXUL and XBL both use fastload and they both have prototype cache, so loading them and opening new windows should be quite fast. Custom Elements (and Shadow DOM) don't have anything like that right now. If we want to use them in browser chrome, we may need to figure out how to support things like this.\n\nThe first step is getting some measurements to compare things like element creation time, memory usage, etc. This work is being tracked in [Bug 1387125](https://bugzilla.mozilla.org/show_bug.cgi?id=1387125).\n\n## Notes\n\n* Custom Elements don't currently work in XHTML documents, but are specified to.\n* Custom Elements don't currently work in XUL documents and additional work is required to support them. In particular:\n  * Elements can be created from the XUL prototype cache, bypassing the XML parser\n  * XUL elements are subtly different from HTML elements in how they handle their parenting and prototype chains\n* The [Custom Elements polyfill](https://github.com/webcomponents/custom-elements) doesn't work in XUL documents, but a [fork designed to use deepTreeWalker](https://github.com/webcomponents/custom-elements/compare/master...bgrins:firefox-browser-chrome?expand=1) does. Using this would allow us to prototype swapping out components without switching the top level document away from XUL. If this turns out to be a good way to migrate away from XBL, we'd have to do work to make the native Web Components implementation work in XUL documents.\n* The relevant Web Components implementations are still off by default and missing features in Firefox:\n  * Custom Elements is in progress, with plans to complete the work by end Q3 2017\n  * Shadow DOM is backlogged, with plans to start the work in H2 2017.\n"
  },
  {
    "path": "text/0005-problems-with-xbl.md",
    "content": "---\ntitle: Problems with XBL\nlayout: text\n---\n\n# Problems with XBL\n\n## Unmaintained\n\nXBL is a proprietary technology developed by Mozilla. We are solely responsible for the maintenance and improvement of XBL. Any time spent on doing this doesn’t contribute to the evolution of the Web so effectively we don’t spend anytime improving XBL except for plugging serious holes.\n\n## Poorly Documented\n\nThe XBL wiki page says:\n> \"XBL 1.0 is specified in XBL 1.0 Reference. Unfortunately, the actual implementation in Mozilla is different from the specification, and there's no known document available describing the differences. Hopefully, the Reference will be updated to describe those differences.\"\n\nThat was written in 2005. It’s still true. Except for the \"hope\" part.\n\nUnlike other technologies like web components where there is plenty of documentation available on the internet, documentation for XBL is almost entirely written by Mozilla and so there is very little of it.\n\nThe only reference for what is \"correct\" is what the frontend expects, and what the frontend expects is largely based around a hodgepodge of workarounds for bizarre behavior in the XBL implementation\n\n## Verbose and Brittle (Because XML)\n\nWriting code in XML requires extensive use of CDATA sections making the syntax more verbose than it needs to be.\n\n## Dynamic Binding Changes via CSS\n\nAn XBL binding can set an attribute on itself that causes it to change to a different binding, e.g. [the add-on binding](https://dxr.mozilla.org/mozilla-central/rev/3ecda4678c49ca255c38b1697142b9118cdd27e7/toolkit/mozapps/extensions/content/extensions.xml#1245). These cases are difficult to find and debug and it is very unclear what is happening unless the code is well documented.\n\n## Complicates Stylo\n\nWe have a ton of code in stylo to handle XBL `<stylesheets>`, and XBL scoped stylesheets are sufficiently different from shadow DOM scoped stylesheets that the code isn't really reusable\n\n## Complicates Gecko\n\nThe fact that XBL is driven off of frame construction is awful. It causes so much weird stuff to happen at very inopportune moments and creates an enormous amount of complexity in what is arguably the most complex and touchy part of the platform (the frame constructor).\n\nAnonymous subtrees cause lots of complication in general, and we currently have to deal with three kinds (shadow DOM, NAC, and XBLAC). Dropping this down to 2 would be a huge improvement.\n\nThere is lots of complexity and security weirdness that required us to build and maintain the content XBL scope stuff in XPConnect. Eliminating the in-content XBL bindings (video, marquee, pluginProblem, and scrollbar) would be a lower-hanging fruit to eliminate this.\nnsBindingManager is a mutation observer, and its existence means some additional overhead for our DOM mutations, especially [ContentAppended, ContentInserted and ContentRemoved](https://searchfox.org/mozilla-central/rev/51b3d67a5ec1758bd2fe7d7b6e75ad6b6b5da223/dom/xbl/nsBindingManager.h#43).\n\nThe way we do XBL binding (via CSS) means that often when you touch a node from script we have to go resolve styles for it to see whether there's XBL involved.  We optimize this out on the web in various ways, but in XUL it's a pain.\n\n## Not Used Elsewhere\n\nThis presents a recruiting challenge, there aren’t developers with knowledge of XBL available for hire so everyone we hire has to be trained in this unique technology and there is a steep learning curve.\n"
  },
  {
    "path": "text/0006-architecture-review-process.md",
    "content": "---\ntitle: Architecture Review Process\nlayout: text\n---\n\n# Architecture Review Process\n\n## Introduction\n\nThis process targets Architectural Reviews in two categories: “Roadmap” and “Design”. It doesn't tackle how to review in-progress projects to see if they should continue.\n\nThe function of a **Roadmap Review** is to decide if a thing should be done. The goal is to bring together a packet of data to inform a management decision to provide resources to make the thing happen. A Roadmap Review should happen early in the process so that build time isn’t wasted on a “No” decision, but so that enough information is available to management.\n\nA **Design Review** is to decide if the thing should be done in a particular way. The goal is to ensure that the problem space is understood well enough that a realistic bug breakdown can be made, that those working on the problem all agree on the challenge, and that work can be prioritized to hit the hardest parts of the problem first (in order to get early warning if the hard parts prove impossible or to inform schedule projections early).\n\n## Timetable\n\nFor both a Roadmap Review and a Design Review the **timetable** is as follows:\n\n1. The team should set the stage:\n    * Summarize in one or two sentences the proposal to be reviewed. This should be agreed between the team asking for review and the reviewer.\n    * Find a **reviewer** (See “Outputs and Audience” below for the rationale):\n      - For a Roadmap Review this will be the least senior manager with the ability to address the problem in its entirety.\n      - For a Design Review this will be an experienced engineer in the problem domain outside of the team willing to put time into the problem. As engineers don’t review their own code they shouldn’t review their own designs. The principle is to get the most informed and least biased feedback possible.\n    * Optionally, for larger reviews, find someone to **chair** the review. This can facilitate the process (i.e. scheduling the meeting, managing the clock, ensuring minutes are taken, etc.) and enables the reviewer to concentrate on the review itself. The rest of this document uses 'chair' for the administrative role. For smaller reviews the reviewer also does the tasks of the chair.\n\n2. The team produces a **Review Packet** designed to document the proposal identified in step 1. The Review Packet includes:\n    * A lay summary of the problem space that defines a shared language and identifies the key forces behind the problem.\n    * A list of the groups directly affected by the proposal to ensure that the review meeting includes representatives from those groups.\n    * A vision of what the project will achieve on completion.\n    * A brief that explains what the team is proposing. This should read more like an encyclopedia entry than a marketing document. The audience is the people in the review; i.e. this should attempt to plug organizational documentation holes. The documentation process should not be more onerous than is required for the review. The brief should identify:\n      - the architecture being proposed\n      - the non-functional requirements, like accessibility, performance, security, and stability\n      - relevant business goals, such as improving user satisfaction and retention\n      - scenarios that exercise the requirements/goals against the proposed architecture\n      - how the proposal handles these scenarios\n      - what review and discussion of the proposed architecture has already happened (and with whom)\n    * A competitive analysis suggesting alternatives, costs, and opportunities\n    * Input by representatives from the affected groups. The team may update the review packet based on this input but is not required to answer all questions and address all comments before the review.\n\n3. The reviewer creates a list of questions to be discussed during the review, possibly from the comments made in step 2. These questions should be:\n    * Designed to foster productive analysis and discussion\n    * Made available to the the team prior to the review so they can prepare answers\n\n4. The chair convenes a review meeting. The discussions of this meeting should be carefully minuted.\n    * The meeting should have a limited attendance to avoid a sprawling unfocussed meeting, but it should include:\n      - The team making the proposal\n      - Representatives from the affected groups, taking care to:\n        + Select a full range of perspectives\n        + Avoid “stacking the deck” with too many people that think the same way\n    * A suggested agenda for the review meeting:\n      - Short opening statement by the team on the current state of play\n      - Discussion of Questions and Answers (see 3. above)\n      - A step back to discuss if the proposal is right in the wider context\n    * A lightweight review might not need the input of others, and ultimately, might not even need a specific review meeting.\n\nThe review meeting should be a discussion of the issues. Reviewers don't need to make a decision at the meeting. The goal is to understand the issues raised by the question(s) and the background from the review packet, and to add to it wisdom from the people at the review.\n\nA generally positive review is likely to have a number of action items which the team should respond to before the review is complete. The reviewer is responsible for enumerating the action items, validating the answers and ultimately deciding the outcome of the review.\n\nA more negative review may decide to alter the questions, undergo a subsequent review or drop the subject entirely. It is a goal of the process that we discover fatal flaws as early on in the process as possible.\n\nThe results of the review (including review packet and reviewer summary) should be published and communicated widely, with the review packet and results archived. Some version of the decision should be publicly linkable, so that it can be the decision of record, to which blog posts, mailing list posts, can refer.\n\n## Review Scaling: Large and Small Reviews\n\nThe ideal review process scales well. The same basic system should work for a quick 2 person decision over the best way to design a certain feature, and for a critical organization-wide decision about a path to take. The term “team” is used above although we strongly recommend design reviews for smaller questions. If you feel yourself wondering if some design is best, it should be easy to perform a review.\n\nFor particularly small reviews it is possible to merge a roadmap review and design review into a single review, in which case the question is both 'should we do this', and 'should we do it this way'. It is possible to conduct combined reviews where the total time for the review for all participants is 1 working day.\n\n## Rationale: Outputs and Audience\n\nThe first two considerations define the goals of the review process and are two sides of the same coin.\n\nThe audience of a Roadmap Review is those “up the chain” of decision makers and the audience of a Design Review is those running “down the chain” of implementors.\n\nA Roadmap Review should focus on:\n\n* The problem space\n* Large forces impacting Mozilla’s choices in the space\n* Competitive analysis\n* Opportunity cost\n\nThe key outcome is firstly, senior management investing in the problem space and building consensus around a shared reality. Secondly, to provide documentation of both the goals and assumptions of the project to aid later evaluation.\n\nA Design Review should focus on:\n\n* The solution space\n* Technical roadmaps\n* Milestones and timelines\n* Resource allocation\n\nThe key outcome for middle management is to understand the value proposition and the path to success. The key outcome for engineers is firstly to understand the design constraints and the tradeoffs between them, which is important to avoid unnecessary re-work when implementation reality causes a re-think. Secondly, to provide a proposal detailed enough to begin an initial breakdown of engineering work for the proposal (although the work implied by the proposal should not be filed prior to the completion of the review).\n\n## Examples\n\nA number of reviews have been completed and can be used as a template for a successful architectural review. [XBL Removal](0007-xbl-design-review-packet.md) was the first example of a design review, [Sync and Storage](0008-sync-and-storage-review-packet.md) was the first roadmap review, however any of the documents on this website with a title that includes \"review\" could be used.\n\n## Additional Reading\n\n### Multiple Perspectives On Technical Problems and Solutions - John Allspaw - 2017\n\nDescribes how Etsy's architecture review process evolved. A solid write-up of the social side of an Architecture Review process in an environment that is likely closer to Mozilla's than many of the other write-ups.\n\n> Fundamental: engineering decision-making is a socially constructed activity\n\n> We called these “something new” things departures. ... the basic idea is that there are costs ... for making departures that attempt to solve a local problem without thinking about the global effects it can have on an organization.\n\n* https://www.kitchensoap.com/2017/08/12/multiple-perspectives-on-technical-problems-and-solutions/\n\n### Architecture Reviews - Grady Booch - 2010\n\nA formal write-up from someone who spends a significant time in Architecture Reviews. I've not found the paper online, but the MP3 is an audio version of the paper (with bonus guitar-riff intro). This IBM process suffers from lack of attention to the social aspects, but is helpful in digging into the mechanics of the review itself.\n\n* https://www.researchgate.net/publication/224132839_Architecture_Reviews\n* http://media.computer.org/sponsored/podcast/onarchitecture/onarch-025-v.mp3\n\n### A Toolbox of Software Architecture Review Techniques - Jason Baragry - 2014, 2015\n\nJason Baragry reviews approaches to architecture review and notes some potential pitfalls.\n\n> Architecturally significant requirements are often hard to identify, the architecturally significant decisions are often not documented, and the way the decisions interrelate is often not easily understood. A significant part of the review process is often teasing these out.\n\n* https://swarchitectonics.blogspot.co.uk/2014/12/a-toolbox-of-software-architecture.html\n* https://swarchitectonics.blogspot.co.uk/2015/02/a-toolbox-pt2.html\n* https://swarchitectonics.blogspot.co.uk/2015/09/a-toolbox-pt3.html\n\n### Software Architecture Review and Assessment (SARA) Report - Kruchten et al. - 2002\n\nFormal process with input from a number of companies which targets a more enterprise environment than exists at Mozilla, however the process is clearly defined.\n\n> A software architecture is a set of concepts and design decisions about structure and texture of software that must be made prior to concurrent engineering (i.e., implementation) to enable effective satisfaction of architecturally significant, explicit functional and quality requirements, along with implicit requirements of the problem and the solution domains.\n\n* https://pkruchten.files.wordpress.com/2011/09/sarav1.pdf\n"
  },
  {
    "path": "text/0007-xbl-design-review-packet.md",
    "content": "---\ntitle: Design Review Packet - XBL Removal\nlayout: text\n---\n\n# Design Review Packet - XBL Removal\n\nThis is the packet prepared for the [browser architecture ‘design review’ process](https://mozilla.github.io/firefox-browser-architecture/text/0006-architecture-review-process.html). Some of the sources used to prepare this are the [XBL and Web Components](https://mozilla.github.io/firefox-browser-architecture/text/0004-xbl-web-components.html) analysis and these [scripts and visualizations](https://github.com/bgrins/xbl-analysis), including [graphs of the amount of XBL in tree](https://bgrins.github.io/xbl-analysis). Prototyping and investigations that have been done to create this packet are tracked in the [de-XBL meta bug](https://bugzilla.mozilla.org/show_bug.cgi?id=1397874).\n\n**Review Chair**: Mossop\n\n**Question to be resolved:** What is the best strategy for removing XBL usage in the Firefox frontend, and the platform implementation of XBL?\n\n## Summary\n\n**Problem Space**: XBL suffers from most of the same [problems as XUL](https://mozilla.github.io/firefox-browser-architecture/text/0003-problems-with-xul.html) and [more](https://mozilla.github.io/firefox-browser-architecture/text/0005-problems-with-xbl). Primarily, XBL is unmaintained, poorly documented, verbose and brittle, adds [extra development cost](https://groups.google.com/d/msg/mozilla.dev.platform/iBoROFkR9V8/1ndvU9QIAgAJ) to new code like Stylo, and creates a learning curve and training cost for new engineers. In interviews with senior platform and frontend engineers, XBL is consistently listed as a problem.\n\nXBL is deeply ingrained into the Firefox codebase. Based on an instrumented run of the mochitest-browser integration test, Firefox creates approximately 11000 elements with XBL bindings and 6000 elements without a binding [[source]](https://bugzilla.mozilla.org/show_bug.cgi?id=1376546). Inside of mozilla-central there are more than 200 unique binding implementations and 40K LOC in those files [[source]](https://bgrins.github.io/xbl-analysis/).\n\nFurther, before legacy addon support was removed XBL removal was considered a non-starter, since many popular addons relied on extending or modifying existing bindings. This has also led to extra cost when deprecating features or doing refactorings on individual bindings.\n\n**End State:**\n\nThis would allow us to remove around 14-20K lines of C++ code from dom/xbl (not including tests). It also would move the frontend codebase onto a more standardized tech stack, which will benefit from ongoing platform work and allow for integration with existing tooling. More importantly, the resulting codebase would be simpler and easier to modify, debug and maintain. Performance impacts are expected to be small.\n\n**Stakeholders**: Firefox management, Firefox frontend development team, Firefox platform development team, other apps that use XUL such as Thunderbird and SeaMonkey.\n\n## Brief\n\nWe propose a multi-prong strategy for incremental XBL replacement in the Firefox frontend:\n\n*   Convert some bindings to JS modules\n*   Convert many others to Web Components\n*   Convert a few into unrefactored XUL/JS/CSS\n*   Flatten the inheritance hierarchy to remove unused bindings\n*   Handle special cases\n\n### JS modules\n\nThere are some bindings that are not actually widgets - bindings that are only used once, or don’t render any DOM. These can be converted to plain JS objects (either inside a JSM or in a script scoped to the browser window) through a semi-automated process. This will also require manual changes to calling code, so they don’t access the objects as DOM Nodes, and adjust to any API changes that were done in the conversion. Here are a few examples of bindings that fit into this category:\n\n*   [`<preferences> / <preference> / <prefpane>`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/content/widgets/preferences.xml): Bindings used in about:preferences to control changing preferences from form controls. An outline for how this can be achieved (and a patch that is on its way to being landable) can be found in [Bug 1379338](https://bugzilla.mozilla.org/show_bug.cgi?id=1379338#c6)\n*   [`<stringbundle>`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/content/widgets/stringbundle.xml): A wrapper for string bundle objects with a trivially different API (i.e. `getFormattedString` instead of `formatStringFromName`). We could either build a new object that wraps Services.strings.createBundle and exposes the same API, or change all callers to use `Services.strings.createBundle` directly. Note that the l20n project already has Web Component support.\n*   [`<panelmultiview>`](https://hg.mozilla.org/mozilla-central/file/tip/browser/components/customizableui/content/panelUI.xml#l44): Prior art - Mike de Boer’s work on the PanelMultiView included moving the majority of the implementation out of XBL and into a JSM. More details can be found in [this doc](https://docs.google.com/document/d/1D8WSxgyLA13pidT0umTNEmyrBN5njSPGNLsGkyr0oZM/edit).\n*   [`<tabbrowser>`](https://hg.mozilla.org/mozilla-central/file/tip/browser/base/content/tabbrowser.xml): Much of the Firefox frontend is driven by this binding. It’s also referenced as gBrowser throughout the frontend codebase. It does render some UI below it, but the majority of its ~6000 lines are behavioral, making more than double the size of the next largest binding in Firefox.\n\t* This is currently being investigated as a prototype in [bug 1392352](https://bugzilla.mozilla.org/show_bug.cgi?id=1392352)\n\t* This could be done as a pure JS module, or following the `PanelMultiView` strategy of keeping XBL for construction and destruction but moving the implementation into a JS module\n\n### Web Components\n\nMost of the bindings in Firefox are in [toolkit/content/widgets](https://github.com/mozilla/gecko-dev/tree/master/toolkit/content/widgets), and many of these are self-contained UI widgets that exist at the XUL tag level. For instance, `<image>`, `<label>`, `<toolbarbutton>`, `<checkbox>`, etc. Each has its own methods, properties, internal state, and DOM tree that is currently implemented in XBL. The Web Components spec, specifically Custom Elements, is designed to solve a very similar set of problems, and platform support is actively being worked on. For more detail, see the browser architecture [analysis comparing XBL and Web Components](https://mozilla.github.io/firefox-browser-architecture/text/0004-xbl-web-components.html).\n\nThere are some early prototypes demoing this approach using a polyfill in the [preferences UI](https://bugzilla.mozilla.org/show_bug.cgi?id=1392367) and for [`<panel>`](https://bugzilla.mozilla.org/show_bug.cgi?id=1397876). Some  [initial performance numbers](https://bugzilla.mozilla.org/show_bug.cgi?id=1387125) from a benchmark comparing the Custom Elements polyfill with the native XBL implementation show that the polyfill is on par with the native XBL implementation for creating 1000 empty elements from JS.\n\nShadow DOM provides encapsulation, insertion point features similar to XBL anonymous content, and scoped styles. Waiting on Shadow DOM to land and support XUL documents should not be a requirement for most of the widgets, as long as we are willing to live without insertion points or extra encapsulation (which can be leaky with XBL via CSS direct child selectors and `document.getAnonymousElementByAttribute`).\n\n*   The basic mechanism is a semi-automated conversion from XBL to JS Classes following the Custom Elements spec format, as [prototyped here](https://github.com/bgrins/xbl-analysis/blob/gh-pages/scripts/build-custom-elements.js). Then an element name is registered on the document (i.e. `checkbox`), and when a new `<checkbox>` is created the converted code runs.\n*   We should make Web Components support XUL documents instead of converting XUL documents to (X)HTML to gain Web Components support. Supporting Web Components in XUL is much less work than converting XUL to (X)HTML, and it enables us to separate the XBL and XUL concerns, reducing risk. Removing XBL will make it easier to convert XUL to HTML in the future, should we choose to do so; but adding a dependency on XUL to HTML conversion would make it much harder to remove XBL in the present. We should not do that.\n*   We should do tree-wide binding-by-binding replacements, as opposed to feature-by-feature. This is to avoid 2 copies of each widget, and to ensure that each new Custom Element supports all the necessary features. This may include making tree-wide changes to consumers if for some reason the Custom Element version can’t remain API compatible\n*   We should complete platform support for Custom Elements\n*   Add support for XUL documents on platform Custom Elements implementation\n*   Create a way to load a set of Custom Elements registrations automatically in all chrome documents\n*   Support non-dashed Custom Element names to avoid a project-wide find/replace of `<checkbox>` with `<firefox-checkbox>`, along with not affecting platform code that looks for certain tag names like `panel`\n*   We should complete platform support for Shadow DOM to provide support for [bindings that currently use insertion points](https://bgrins.github.io/xbl-analysis/tree/#children) and that continue to need them after a case-by-case analysis\n\n### Unrefactored XUL/JS/CSS\n\nBindings that are rarely used and insert relatively little content should be unrefactored into their constituent parts (XUL, JS, CSS) and inserted into each bound element.\n\nFor example, the `<prefwindow>` binding, which extends the `<dialog>` binding, is used only seven times and inserts relatively little content:\n\n* 4 LOC: a `<windowdragbox>` toolbar containing a `<radiogroup>` “paneSelector” that enables selection of `<prefpane>`s, even though no `<prefwindow>` contains more than one such pane anymore, so this functionality is wholly unnecessary\n* 5 LOC: an `<hbox>` “paneDeckContainer” that wraps multiple `<prefpane>`s, which is also unnecessary in our single-pane world\n*   19 LOC: an `<hbox>` containing dialog `<button>`s and `<spacers>` that are almost identical to those provided provided by the `<dialog>` base binding\n\n`<prefwindow>` should be unrefactored into its constituent parts as follows:\n\n*   `<prefwindow>`s JS should be converted to a JS module that is imported into each bound element’s document (+1 LOC for each document).\n*   `<prefwindow>`s CSS should be loaded into each bound element’s document (+1 LOC for each document).\n*   `<prefwindow>`’s vestigial XUL content (`<windowdragbox>`, `<hbox>` paneDeckContainer) should be removed; and its remaining content (`<hbox>` containing `<button>`s/`<spacer>`s) should be merged with its base binding’s content or copied to each bound element;\n*   The small number (7) of bound elements means that the additional code size (up to 19 LOC for each element, depending on how much can be merged with the base binding’s content) and maintenance cost is worth it relative to reimplementing the binding as a Web Component.\n\n### Flatten Inheritance Hierarchy\n\nWe have a lot of bindings, and use inheritance pretty widely. There are likely cases where we don’t need to - the extra inheritance may have been required for legacy addons, or may have had a purpose in tree at one point but do not any longer:\n\n*   Some could be detected by looking at the [XBL inheritance tree](https://bgrins.github.io/xbl-analysis), where a binding is the only child of its parent, and the parent is never directly used. In this case the child could be ‘folded into’ the parent.\n\n*   The [`<prefwindow>`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/content/widgets/preferences.xml) binding is a potential example here as well - the ability to render out buttons at the bottom of a dialog could be an option passed into the dialog, instead of using the Unrefactor prong above.\n*   Sometimes a binding is created to override a single property - in this case a new attribute / condition could be added to the parent to encode the behavior change.\n*   For instance, [`<tabbrowser-tabbox>`](https://hg.mozilla.org/mozilla-central/file/tip/browser/base/content/tabbrowser.xml) is a simple binding can be folded into the existing tabbox module with a new attribute\n*   Some bindings are attached via attributes in ways that aren’t able to be emulated in Custom Elements (`<panel type=”arrow”>`) binds #arrowpanel instead of #panel. We can generally migrate to a new tag in that case, but in some cases (like panels) there is platform code looking for the tag name, and it may be easier to fold the arrow functionality into the parent binding.\n\nMaking this change does not determine a solution for the parent - it would still need to follow another prong to get rid of XBL itself. Rather, this is a way to reduce the number of bindings that need to be converted, and a chance to refactor away some often unnecessary inheritance for those converted bindings.\n\n### Special Cases\n\nSome of the bindings used in preferences (and throughout the browser) aren’t easy to directly translate to Custom Elements or JS Modules. Generally, [bindings that use the [implements] attribute](https://bgrins.github.io/xbl-analysis/tree/#implements) but don't map pretty closely back to an HTML tag are components that may fall into this category. Each requires further investigation and prototyping before recommending a specific strategy, and may require a custom solution. Here are a few examples:\n\n*   In-content XBL may provide special challenges due to security considerations (Scrollbars, Marquee, Video controls, Click-to-play plugin UI)\n*   `<tree>` which implements `nsIDOMXULTreeElement` and has a pretty complicated API\n*   For cases like this, we should consult with product / UX and make sure they still want the widget. We may have opportunities to simplify or remove the code instead of just copying the current implementation.\n*   `<panel>` which implements `nsIDOMXULPopupElement`. Initial investigations in [Bug 1397876](https://bugzilla.mozilla.org/show_bug.cgi?id=1397876).\n*   `<menu>` which opens a native context menu, and may have interaction with XUL overlays\n*   Bindings which take advantage of CSS changes to switch to another binding at runtime\n\n## Forces on the system\n\n*   **Performance** is critical, given our commitment to shipping a fast, responsive browser. Initial benchmarking between XBL and Custom Elements is promising, but XBL has been optimized for certain cases like opening new windows and extra platform support or frontend optimizations could add time to the estimate.\n*   **Security** is essential for the XBL-in-content replacements, which can’t enable untrusted content to escalate privileges or access private information.\n*   **Maintainability** is key to improving our ability to evolve the codebase and avoid falling back into a technical debt trap.\n*   **Flexibility** is important to ensure that XBL replacements support the needs of the current (and anticipated future) codebase. Note, however, that replacements don’t have to exactly replicate all XBL functionality.\n*   **Reusability** (and the reuse of existing solutions, like Web Components, where applicable) is valuable to reduce the maintenance burden and the risk of obsolescence.\n*   **Accessibility** is required to allow as many people as possible to use the browser. Existing UI components support a number of accessibility features that should not be regressed by changes.\n\n\n## Dependencies\n\nCustom Elements and Shadow DOM have not yet landed, and Custom Elements doesn’t yet support XUL elements (althought this is being worked on in [Bug 1404420](https://bugzilla.mozilla.org/show_bug.cgi?id=1404420).\n\nAs a backup plan, Custom Elements does have a polyfill that may enable us to proceed without native support, but the Shadow DOM polyfill is heavier weight and we wouldn’t propose shipping it. We could mitigate this by not using Shadow DOM (see the Web Components section above).\n\n## Competitive analysis\n\n_Alternative 1: Do nothing_\n\nWe believe that XBL is a problem worth solving, as outlined in the summary. But, Firefox works as-is and this is a lot of work. So, what if we didn’t do anything about it?\n\nFirefox would continue to function, and we would continue to develop it. Firefox development would be slower, however, because XBL is harder to use and evolve and it doesn’t always [integrate well with the devtools](https://bugzilla.mozilla.org/show_bug.cgi?id=1360072), so we would spend more time implementing features in XBL or working around its limitations.\n\nGecko development would also be slower, because XBL adds complexity in the frame constructor, among other places. In particular, implementation of Shadow DOM would take more time, since it would have to account for situations in which both a shadow DOM and XBL are active at the same time.\n\nRemoval of the old style system will be delayed until Stylo adds support for XBL, which means that Mozilla will continue to ship Firefox with two separate style systems until that time (adding complexity and footprint).\n\n_Alternative 2: Get rid of simple cases, continue to use XBL for the harder cases_\n\nIf we identify the simpler bindings and migrate them with one of the proposed prongs, we could leave the more complex bindings in tree. This would narrow the scope of this proposal. However, it would introduce a complexity cost of having multiple approaches to UI components in tree, and wouldn’t allow us to achieve the end goal of removing the XBL implementation.\n\n_Alternative 3: Do more of the work a window at a time vs a binding at a time_\n\nThe proposal we suggest is ‘horizontal’ replacement - that is, replace each individual binding entirely, across the tree. Another approach would be a ‘vertical replacement’ - that is, replace all bindings in a window, one window/feature at a time. For instance, convert the entirety of about:preferences, but don’t worry about using the converted bindings in other features. The drawback of this approach is having two implementations of a binding until every window has been complete, and keeping these in sync would require ongoing work.\n\n_Alternative 4: Rewrite the frontend using a different stack_\n\nInstead of converting XBL bindings to JS and Web Components, we could rewrite the frontend (probably feature-by-feature) using a modern library like React. There are some advantages to this approach, including better developer ergonomics (which improves developer efficiency), better application architecture (which improves application quality), and ecosystem effects. But a rewrite would take much more work, entail much greater risk, and entrain a separable set of concerns (in particular, XUL replacement).\n\nWe do think it's reasonable for teams to consider rewriting individual features using a library like React when other factors justify a rewrite, for example in the case of [debugger.html](https://github.com/devtools-html/debugger.html). And it may eventually become reasonable to undertake a complete rewrite of the application. XBL removal may make that easier. But we think XBL replacement is worthwhile enough by itself to justify pursuing a lower-risk strategy for its removal.\n\n_Alternative 5: Convert the frontend to HTML (removing XBL in the process) without rewriting it_\n\nWe could remove XBL while converting the frontend to HTML—like alternative #4—without rewriting the frontend in the process. In this case, we’d convert XUL to its equivalent HTML, conserve the existing JS/CSS implementation, and convert XBL bindings to Custom Elements, JS modules, etc. as we’re proposing to do in this document.\n\nDoing this would achieve both XBL and XUL replacement, and its cost of XUL replacement would be less than the cost of completely rewriting the XUL implementation in alternative #4.\n\nBut the cost of replacing XBL would be the same as the cost in our proposal. And the risk would increase with the increased scope. Also, this approach is quite similar to replacing XBL and then replacing XUL, modulo any cost of supporting Web Components (and any other necessary web technologies) in XUL.\n\nFinally, while there is relative consensus that XBL should be replaced, XUL replacement is more controversial. We think it makes more sense from strategic and risk standpoints to separate these concerns, tackling each of them in turn, starting with XBL.\n\n"
  },
  {
    "path": "text/0008-sync-and-storage-review-packet.md",
    "content": "---\ntitle: Roadmap Review: Sync and Storage\nlayout: text\n---\n\n# Roadmap Review: Sync and Storage\n\n**Firefox should wow users with a smart personalized experience that's available seamlessly across services, apps, and devices. User data storage and syncing are vital to this vision.**\n\nWe propose to reframe user profile data as a pillar of the Firefox ecosystem, rather than continuing to allow our storage and syncing story to emerge piecemeal from the implementation details of each new product feature.\n\nUser data becomes a deliberately designed service layer on which many different product experiences are built. We foster through design reviews the long-term importance of user data. And we invest technically in a more unified, extensible, flexible, cross-platform suite of storage and sync solutions that support a coherent set of approaches to storage.\n\n## Review question\n\nIs this roadmap proposal congruent with Firefox's strategy?\n\n## Document status\n\n* 2017-11-20: Go-ahead given.\n* 2017-11-09: Review successfully completed. Follow-up questions underway.\n* 2017-11-03: Sufficient contributions received; sent to the Reviewer.\n\n## Roles\n\n* **Reviewer**: dcamp\n* **Chair**: jwalker\n* **Proposers**: \n    * rnewman, Browser Architecture\n    * rfkelly, Engineering Lead, Firefox Accounts\n    * markh, Sync engineering\n    * emily, Browser Architecture\n    * adavis, Product Manager, Sync & FxA\n\n# Lay Summary\n\nFirefox, and the other new experiences that we want to create, rely on easily recording, combining, syncing, extending, and repurposing user data.\n\nDesktop Firefox’s data is spread across [45+ different stores using 10+ different storage technologies](https://github.com/mozilla/firefox-data-store-docs), and we keep adding more.\n\nIn general, these stores and technologies:\n\n* Were not designed for syncing, for reuse on other platforms, or for their data to be repurposed.\n* Are not straightforward to extend, particularly those connected to Firefox Sync. Migrations are risky and time-consuming, and [downgrades are a huge concern](https://bugzilla.mozilla.org/show_bug.cgi?id=1404344).\n* Are silos: it’s hard to connect them to each other to provide insights into data, which is the motivation for initiatives such as Activity Stream and Context Graph.\n* Have no baseline of expected storage capabilities: documentation, full-text search, backup/import/export, basic versioning/migration, asynchrony, atomicity, caching, performance measurements and optimizations, _etc._ are all built from scratch, if they are built at all. Each store repeats the same fundamental mistakes and has its own quirks.\n* Provide inconsistent support for security and privacy UI settings, leading to an [inconsistent and non-intuitive user experience](https://docs.google.com/document/d/1QJxHQ4GqziUGUiyaF5oP7-cA3hkyTamY33VvHY3vBrM).\n* Are limited to a single platform. Desktop’s storage investments (_e.g._, Places, form history, session store), are hard to use elsewhere in the ecosystem, which reduces the value of those investments. No code or storage formats at all are shared between our iOS and NMX projects and desktop; even Fennec reimplements (for good reason, but at some cost) stores that exist in Gecko.\n\nFirefox Sync itself is incomplete, in every sense:\n\n* It doesn’t sync enough data types.\n* Types that sync aren't complete, and/or lose data when synced.\n* We have almost no consistency across platforms: each implementation syncs different data, in different ways, to different storage.\n* Access control is an all-or-nothing affair; the auth tokens and encryption keys needed to read synced bookmarks will also grant read/write access to synced passwords.\n* It is hard to improve the protocol itself to remedy these.\n\nOur current technologies are ill-suited to even our current sync and extensibility needs, let alone our predicted future needs.\n\nMoreover, we have no off-the-shelf solution to adopt for new products or features that want to avoid these issues: the status quo is maintained. Teams either do significant amounts of redundant work to bootstrap new storage for each feature and for each platform, to add sync support, and to reach a baseline level of functionality, or they neglect platforms and functionality.\n\nThese problems are already limiting factors to product development. Recent examples:\n\n* Form autofill is shipping desktop-only. Desktop-only sync took two months of work and an entire work week to build. Android will, at some point, blindly round-trip this data but not use it. There are no plans for iOS work.\n* AS found it extremely challenging to add data to Places, and adding even one additional field to Firefox Sync took 9+ months to trickle through three platforms and negotiate backwards compatibility constraints. More ambitious data recording, particularly for speculative features/experiments, is considered unreasonable, time-wise, within the framework of Firefox Sync: time-to-market is too long.\n* The nascent support for containers in Firefox — partitioning of history visits into work, personal, _etc_. — has no clear path to integration with Sync.\n* Few features are added to Sync itself: they need to be implemented three times (once for each major platform) and significant changes run into backward compatibility issues.\n* New mobile and other experiences that want to integrate with FxA-stored data face significant data/sync costs:\n    * The NMX team forked a [separate simple read-only Android library](https://github.com/mozilla-mobile/FirefoxData-android), and [an unfinished iOS equivalent](https://github.com/liuche/FirefoxAccounts-ios), to pull data directly from the Sync server; Sync is complex, and extracting the existing Firefox code into separate libraries required expertise that was not present in the team. (We now have **five** separate FxA and Sync implementations.)\n    * Firefox Rocket doesn't reuse Firefox for Android's storage, and has no path forward to syncing.\n    * A Servo-based Android AR/VR browser is in need of data storage solutions, but we don't have a solution that we can give them.\n    * New mobile and other experiences that generate their own user data have no established patterns or guidelines for storing it, and no existing path toward making that data available to the broader Firefox ecosystem.\n    * The Lockbox project burnt a lot of early development cycles trying to decide what to use for backend storage — Sync, Kinto, or something else entirely. Their MVP currently does purely local storage.\n    * Project Hopscotch will store its collected user data in Google's [Firebase](https://firebase.google.com/), where it is siloed away and unable to be repurposed or shared with other Firefox experiences.\n\nStakeholders are:\n\n* Sync, both client and server, across all platforms. The Sync team is motivated to improve Sync and to build better systems, but is only staffed for incremental work, and has neither time to assist with more than a handful of one-off integrations (_e.g._, form autofill) nor significant leverage to direct the design of storage systems to reduce that workload.\n* Activity Stream, across all platforms. AS is a significant short-term new consumer of user data, and a long-term generator of reusable data. Delivering a good AS experience requires capturing new data and going far beyond the current capabilities of Sync and Places, but the team lacks the leverage or expertise to make those changes.\n* Existing storage teams, responsible for maintaining and implementing existing stores. Some stores lack owners. Most of these engineers don't directly experience the costs borne by the Sync team and engineers working on other platforms.\n* Existing product teams. Product managers own features across platforms, but the implementations and capabilities of those features differ widely.\n* New product teams and ET explorations wishing to use and collect user data. These teams must build storage and syncing from scratch. Extending and integrating with existing data is challenging.\n\n## Brief\n\nAltering the course of 10+ years will be a multi-step process.\n\n_We propose expending effort to build replacement tools and systems that meet current and known future needs, supporting in-development Firefox features, and migrating existing Firefox data stores as it makes sense to do so._\n\nRather than just building single-platform MVPs, we hope to encourage engineering and product managers to consider the total cost of cross-platform features and integration, and shift incentives to encourage engineers and product managers to drive towards consolidation — make the right thing easy, so that features get syncing, backup, extensibility, and fast querying without significant new work.\n\nMozilla needs a _culture that places importance on user data_ — more forward-looking data modeling, implementations that are built to sync across platforms, and data that can be repurposed. One path to doing so will be the architecture review process itself.\n\nWe expect technical work to include **designing and building a durable and scalable end-to-end sync and storage system for user data.** This system should handle evolving data without the involvement of Sync engineers, without locking out clients on other platforms or releases, and without losing server data. [Log-structured data](https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying) is the industry standard for distributed systems with growing data and multiple consumers, so we expect this to be a key part of the solution.\n\nWe've [captured some detailed requirements for sync and storage](https://docs.google.com/document/d/1XOJhW-qf4iIyGHwiSLgWEfXYDWLy44So5L8CdwH6pcg), and will continue to do so.\n\nThis is a relatively large piece of work. Mozilla’s needs are fairly unique: _e.g._, client-side crypto, disconnected writes, years of activity data stored on the client, and working without an account. This means that although the conceptual framework has been well understood for many years, off-the-shelf solutions tend not to be suited to our needs. (Chrome Sync itself has a similar implementation to Firefox for iOS or 2017/2018 desktop Firefox, and is presumably staffed to match the costs of this high-overhead approach. Google defaults to non-private approaches.)\n\nGetting this right unlocks the future of user data.\n\nTechnical work will involve:\n\n* Designing client-side APIs to encourage features to record data in a way that meets the needs of other components, including Sync.\n* Designing one or more storage systems (including client, server, and documented protocol), supporting those APIs, that meets the captured requirements for sync and storage, including straightforward evolution of data.\n* Building SDKs and other tooling to help new mobile experiences quickly integrate with existing user data, and store and share new user data of their own.\n* Building client-side data pipelines to also meet the needs of application features. New storage systems must meet the performance requirements of the product.\n* Exploring technologies and patterns that make it easier to connect UIs through to storage.\n\nThe Sync and BA teams have already begun exploring prototyping parts of this puzzle in an incremental way: in Q4, building a proof of concept as an example of an event observation and capturing infrastructure, materializing a view for querying. We expect this POC to inform our implementation plan.\n\nWe also expect to build cross-platform **storage and sync implementations partly or wholly in Rust**, to build a future in which products **throughout the Firefox ecosystem** can share fast, reliable code instead of reinventing the wheel. With current staffing levels we have a hard time keeping our multiple codebases interoperable, let alone growing features at a healthy rate. Emily has [already made progress](https://github.com/fluffyemily/cross-platform-rust) on the groundwork for cross-platform Rust libraries.\n\nWe will **continue work on documenting Firefox’s user data and publishing** [**our findings**](https://github.com/mozilla/firefox-data-store-docs). In the course of preparing this proposal we discovered that essentially none of our storage systems are documented, sometimes to the point of not knowing what a file or database was for, and often not being able to tell whether a file was obsolete. We don’t even know which prefs will exist in about:config. It is hard to make good decisions without information.\n\nWe will **continue to help guide new data-centric systems** — including Lockbox and ET work like Foxy — towards a cooperative future that builds capabilities for our ecosystem strategy.\n\nAnd we will guide the **incremental consolidation of existing Firefox data stores into more capable systems**. This might begin by building or buying replacements or abstractions, so the number of stores will grow before it shrinks. For example, libpref is ill-suited for storing data for front-end features; we should provide a durable, syncable store that meets their needs, and migrate features away from libpref, ultimately allowing it to be replaced by a simpler and more performant system that better meets its “pref-shaped” needs. It’s not yet reasonable to pin a number on how many stores we _should_ have, but it’s almost certainly less than 45.\n\nSome of this work will involve measurement work on existing systems or product features; Firefox currently lacks a holistic view of its storage footprint and performance characteristics. We expect to work with product management and engineering to improve this.\n\nWe expect initial staffing for these efforts to be low. Research and prototyping work (much of it not parallelizable) needs to be done for new storage, more exploration of Firefox needs to be documented, and the groundwork for Rust components needs to be completed. 3–4 FTEs is appropriate.\n\nIn 3–9 months we should be in a position to evaluate a number of more concrete directions, gradually committing more headcount to accelerate progress as opportunities become actionable.\n\nWe can call out some representative product end states of this effort:\n\n* New mobile apps can import a published library, with **shared code across platforms**, and use a standard FxA OAuth flow to get access to all or part of a user's synced data **without reinventing the wheel**.\n* New product features and **experiments** can store new data through a simple, homogeneous API, and have that data automatically accessible from other Firefoxes and in mobile apps, **without writing significant new syncing code** and without worrying about version compatibility. Product managers should feel that they can confidently explore and iterate on product experiences.\n* Users can alternate between release channels or move through a series of experiments using the same profile without losing data. **Most changes will be downgrade-safe**.\n\nEffort subsequently invested in consolidation should gradually reduce ongoing cross-platform maintenance burden, which — after the initial expense — should free up engineering resources for more useful product work.\n\n## Alternatives\n\n### Do Nothing\n\nThe main alternative to this roadmap proposal is to do nothing.\n\nThere’s no immediate additional cost to doing so, beyond, as Alex puts it, \"shipping shitty products, really slowly!\"\n\nWe will continue to bear the high cost of maintaining and extending Sync, we will continue to have a broad array of undocumented and inconsistent storage systems, Sync will continue to be a lower-quality feature than we want, and as a result we will continue to offer inconsistent and incomplete experiences across our products, harming mobile adoption.\n\nWe will be largely unable to offer Context Graph-like features on top of existing user data. Telemetry data and Pocket will thus be the foundation of Context Graph. Activity Stream will soon face significant difficulties in storing and syncing new data; ultimately they will end up building an _ad hoc_ event storage system, and will collapse under the effort of replicating that work on iOS and Android where staffing is limited. New agent projects and mobile browsers will reinvent their own non-syncing storage, slowing development and convergence. Prototyping and experimentation will continue to be costly, particularly when integrating with or extending existing data.\n\n### Bottom-Up\n\nAnother alternative is to allow this work to happen bottom-up. We are skeptical of this possibility: evidence suggests that engineers have neither incentive nor leverage to tackle cross-component work like this. [Conway’s Law](https://en.wikipedia.org/wiki/Conway's_law) applies. We are trying to make our architecture decision making clearer, not force it under the radar. Worse, this encourages unsustainable heroism — if we think improvement is necessary, we should produce a supportive environment for it to occur.\n\n### Rebuild Whole Products\n\nA more ambitious alternative is to construct new products in the right way — up to and including constructing a new browser. Mozilla has no plans to ship a Servo-based browser as a replacement for Firefox, so this strategy certainly doesn’t fly on desktop. It doesn’t avoid cross-platform costs, as discussed above. And — more significantly — simply building a new product is no guarantee that the work will be done differently. Servo is a very modern renderer, but without incentives to make other choices, it has already started down Firefox’s path with its prefs storage. Zerda/Firefox Rocket's initial data stores are bespoke and not syncable, despite having access to Fennec's code, because that team's incentives and the costs of reuse don't align.\n\n### Improve Firefox Sync 1.5\n\nWe could attempt to invest in Firefox Sync 1.5 to try to make it incremental, extensible, durable, etc., rather than exploring alternative solutions. In the last 7+ years there's been enough architectural motivation to design Sync 2.0, Kinto, QueueSync and TreeSync (the latter two during the short-lived PiCL project). This, and recent experiences with adding data to Sync 1.5, suggest that we’ve improved the old Weave architecture as far as it can go. Additionally, we are not staffed to grow all five Sync client implementations in parallel.\f\n\n## Competitive analysis and simplifying opportunities\n\nSome of the requirements we've captured, and some aspects of the situation in which we find ourselves, might be malleable.\n\nSome of these requirements constitute strategic choices: _e.g._, end-to-end encryption offers some strong privacy guarantees, but impacts recoverability and access by other services. The set of such strategic choices makes up a space of solutions. We expect the outcome of this roadmap to make progress towards defining a complementary set of points in this space.\n\nIn this section we briefly examine some of the choices available to us.\n\n### Encryption and servers\n\nStoring data in a way that prevents Mozilla from seeing cleartext means that the server can't help much with conflict resolution, versioning, data access, _etc_.\n\nChrome makes a different strategic choice, and is thus able to show your history on the web. [Realm](https://realm.io), a cross-platform database that competes with Core Data, offers automatic synchronization of its 'realms' between clients via a central Realm Object Server, with all data in cleartext.\n\nA solution that could rely on cleartext data would be able to shift logic from the client to a single server infrastructure, which would increase agility and simplify the client.\n\nThe end state of this shift is something like Pancake or Facebook: a thin client with a rich server application and a server-canonical data model. This avoids synchronization entirely.\n\nA number of systems use encryption at rest, but not end-to-end encryption. This allows for simpler web access, sharing, and recovery. Dropbox uses this model for files. This approach is not resistant to subpoena or intrusion.\n\nThis proposal assumes that end-to-end client-side encryption is table-stakes for some or all of Mozilla's applications, and so we can't disregard this requirement.\n\n### Data-rich client applications\n\nIf we were to commit to storing less data on clients, and changing the set of stored data less frequently, then the cost-benefit analysis of this problem would shift, perhaps to the point that significant engineering investment should be avoided. To an extent this is similar to the alternative \"Do Nothing\": we currently try to avoid growing or changing the data we store, and we change Sync very infrequently.\n\nChrome is an example of this: there is no evidence of Chrome moving towards anything more than its rudimentary history and bookmarks storage.\n\nThis proposal assumes that there is product desire to leverage more user data on clients, not less, and eventually we will have to tackle the issues outlined in this proposal.\n\n### Offline writes\n\nIf we were to make significant changes to the relationship between users and accounts, and between devices and servers, we could simplify the problem of syncing. For instance, if we linked all profile data to a Firefox Account from first opening Firefox, then repeated large-scale merges can be avoided. If we require all devices to be connected in order to record data (providing high availability for server storage), then we can turn synchronization into a distributed write. These are still non-trivial engineering problems, but they're different problems.\n\nMost database systems we've examined, including Realm, Dropbox Datastore, CouchDB/PouchDB, and others, allow for offline writes.\n\nThis proposal assumes that our ability to tie writes to server resources is limited, for both technical and product experience reasons, but we have been thinking about this possibility.\n\n### The definition of ‘data’\n\nStorage systems might store some or all of documents, small blobs (_e.g._, icons), large blobs (_e.g._, downloaded files), independent structured data (_e.g._, emails), interrelated structured data (_e.g._, social graphs or bookmark trees), measurements/events (_e.g._, sensor data), and more.\n\nA system optimized for document-structured data will typically scale horizontally but lack features like cross-document transactions. A system optimized for graph traversal might not handle blobs well.\n\nChoosing carefully which kinds of data to support in which systems makes it easier to meet requirements.\n\nTools like CouchDB and MongoDB aim to provide scale and simplicity in syncing by deferring conflict resolution, identity, and validation to application code, themselves working with a more simplistic data model. Dropbox's [decommissioned](https://blogs.dropbox.com/developers/2015/04/deprecating-the-sync-and-datastore-apis/) Datastore API treats datastores independently and [merges records field-wise to resolve conflicts](https://blogs.dropbox.com/developers/2013/08/how-the-dropbox-datastore-api-handles-conflicts-part-two-resolving-collisions/); that's a good set of tradeoffs, but it leaves identity merging, data evolution, and validation to the app.\n\n## Strawman Roadmap\n\nWe expect to gradually firm up our understanding of concrete solutions and shipping vehicles over the course of the next two quarters: Phase 1.\n\n### Phase 1: building confidence (2017Q4–2018Q1)\n\nWe have a good idea of the characteristics of a solution for structured user data that allows for the data evolution, experimentation/iteration, encrypted sync support, and reuse that our products need.\n\nWe have identified two main remaining puzzle pieces that the Browser Architecture team will explore: the client-side data pipeline that preserves existing query interfaces, and the concrete sync protocol that moves data around.\n\nIf we can't tackle those, then there is no mainstream path forward for this part of the solution. If we're wrong, we want to find out sooner rather than later.\n\nOur first step, then, is to exercise those two parts by prototyping each, iterating as needed. If we fail, then we will reassess our approach. This is the cheapest and best route to failure.\n\nThe Sync team has a related plan for this timeframe to reduce uncertainty around Rust-based cross-platform dedicated storage APIs that wrap generic storage. This will build confidence in our ability to consolidate implementations across desktop and mobile while meeting the needs of product engineering teams, and help to determine the developer experience linking application and service.\n\nIn parallel, we can move forward with independent cleanup work: making APIs async (post 57 freedom), incrementally improving and teasing apart existing problematic stores (_e.g._, libpref), work being proposed elsewhere around process separation for storage, etc. These efforts will set the stage for productization, but also make the world a better place regardless of the success of this particular effort.\n\nA successful outcome for this phase is one of: fast failure; a revised roadmap or strategy that reflects new knowledge; or a proposed set of design options that can be reviewed with an eye to tackling Phase 2.\n\n### Phase 2: attack the hardest thing, fail fast (2018Q1–Q3)\n\nThe largest single point of risk beyond basic functionality is addressing Places.\n\nPlaces' history store is a large source of value, Places-linked data is core to Sync, and it's also a good test case: a high-volume store of structured user data with existing complex query consumers.\n\nIf we can't tackle Places, then there's much less value in proceeding.\n\nWe think the quickest and lowest-risk approach is to intermediate some writes to Places, turning parts of the Places database into a read-only materialized view. This stress-tests the client-side data pipeline, and allows us to explore syncing new data (_e.g._, containers, alternative representations for keywords and tags), recovering Places data from the log, _etc_., without disturbing existing consumers.\n\nAt this stage we hope to find and fix scaling and performance issues before doing further work. If we fail here, the process has been both informative and relatively cheap.\n\nAs this phase proceeds we'll design and build production server capacity for the sync system, begin to address fit-and-finish concerns like push notification integration, and continue to assess requirements from new product efforts.\n\n### Phase 3: consolidate (2018Q3–)\n\nIf we're confident in our new sync implementation, can use it from apps, and can handle Places-level scale, then we can proceed with consolidation and simplification to add value to Firefox. We can fan out: pick suitable storage (_e.g_., data that's high priority for syncing, an old store that's not owned by an existing team, …), make sure it has an async API, and port it on top of the log, following the example of Places.\n\nEarly in this phase our documentation will go from adequate to great, we will place heavier emphasis on the developer experience and UI programming integration (_e.g._, GraphQL), and we expect to see further adoption in mobile apps and on the server as the cost of accessing Firefox data decreases.\n\nWe can also assess the urgency of exposing this kind of storage — both access to Firefox's data, and dedicated storage — to WebExtensions.\n\nThese additional efforts will likely be assessed and planned closer to the time; it would be unrealistic to estimate resourcing or timing when priorities are likely to shift.\n\n## Resources\n\n[Requirements capture for Sync](https://docs.google.com/document/d/1XOJhW-qf4iIyGHwiSLgWEfXYDWLy44So5L8CdwH6pcg)\n\n[Scoped encryption keys and access control for FxA](https://docs.google.com/document/d/1IvQJFEBFz0PnL4uVlIvt8fBS_IPwSK-avK0BRIHucxQ)\n\nVision statements\n\n* [Ryan Kelly](https://docs.google.com/document/d/1MrXNcVzDDYyGSvud6CvVSoPGNyvpJvHGSDpRwIcalQc)\n\nBlog posts\n\n* [Thinking about Syncing, part 1](https://medium.com/@rnewman/thinking-about-syncing-part-1-timelines-7f758e2bd676) (2017)\n* … [part 2](https://medium.com/@rnewman/thinking-about-syncing-part-2-timelines-and-change-90a0964f10f3)\n* [Syncing and storage on three platforms](https://160.twinql.com/syncing-and-storage-on-three-platforms/) (2015)\n* [Different kinds of storage](https://160.twinql.com/different-kinds-of-storage/) (user agent service) (2016)\n\n## Thanks\n\nThanks to our readers: myk, asuth, mak, grisha, tspurway, snorp, overholt, ckarlof, wennie, selena, gps, trink, hildjj, and plenty more.\n\n## Questions\n\n**Are you going to require this everywhere?**\n\nNo; it's not applicable to all problems, nor does the cost-benefit calculation always play out. Decisions should be made based on suitability and value: _e.g._, to make some data available to other apps or platforms, to bring a store up to a feature baseline, or to integrate two data sources to support a new feature.\n\n**Does this mean we’re using Mentat?**\n\nNot necessarily. Our roadmap so far involves answering questions about how to provide capabilities that we think are important for Firefox's strategy. It's possible that parts of Mentat will evolve into parts of the solution — after all, it's an embedded Rust log-structured store — but a solution will be integrated with other parts of the Firefox ecosystem, not just a database.\n\n**Does this restrict the design of Sync and Storage at Mozilla?**\n\nNot really. We expect FxA identity-attached services to proliferate in time, and we expect there to be justified divergent approaches to storage, too — here we are mainly focused on structured user profile data, but elsewhere we are also _e.g._, helping to shape a redesign of libpref. What we hope to happen is that we can provide a small spectrum of solutions that are better choices than building yet another store from scratch, and that are dramatically _less_ restrictive than the current situation, achieving convergence at a healthy rate.\n\n**Does this help ET?**\n\nWe hope so. Having reusable, portable components and data opens doors to exploration and development.\n\n**How do you plan to scope out the problem of scaling the service?**\n\nIn short: by finding sets of properties that are beneficial for both the requirements we've captured for storage and syncing and also for management of a hosted service — including scalability — and designing a solution that reflects those properties.\n\nThis is something of a design question, but we have some idea of where we might go (we're initially proposing log-structured storage) so we can answer in brief. In short:\n\n* Independent users and divisible data. There is little need to coordinate between encrypted storage for each user. Moreover, by making the log, entities, and other attributes meaningful concepts in the system, we can make the data for each user amenable to sharding along these axes. This allows us to scale horizontally.\n* Mostly immutable data. Unlike a write-in-place system like Firefox Sync, which supports its constantly growing set of data by churning in place, we would like the majority of our storage work to be appends. The exceptions are well-defined operations that introduce barriers (like rollups, snapshots, or account deletion) and well-defined operations that rewrite parts of history (like Forget). Even with those exceptions, that's a healthier workload than we currently support in Firefox Sync, where a big pile of records is both constantly added to and constantly subject to random replacements — it makes the server simpler and the workload more predictable, and allows for cheaper long-term storage.\n* Exploiting modern advances like push to reduce costs for common cases: improved responsiveness can eliminate a subset of conflict scenarios, which reduces load for both client and server.\n\nQuestions from ckarlof (not a lot of time to answer these, so a little brief, but better than nothing!):\n\n**What are the outcomes we're trying to achieve here and how do you know we've achieved them? What are your key metrics of success?**\n\n_I see some \"representative product end states\", but I feel these need to be more prominent in communicating your vision of success._\n\nIn addition to the concrete definitions of success in each phase, we'll know the overall effort has been successful if:\n\n* The organization displays a culture of holistic thinking around user data across the Firefox ecosystem.\n* Product managers feel more empowered to drive experiences that rely on new, integrated user data.\n* The baseline of storage capabilities is higher, with consequences for the consistency of Firefox's UX (see, e.g., Wennie's doc).\n* The cost of integrating new syncable data is markedly lower than it is today.\n\nFiguring out how to concretely _measure_ our success is a task for after this review.\n\n**How many different complete options are you considering to achieve your proposed outcomes?**\n\n_I feel there's really only one under consideration, but I feel having one or two alternative approaches to reach your goal, even if they're straw dogs, would help folks understand the merits of your favored approach better. Your competitive analysis is close to this, but some of things you're comparing against aren't solutions to reach your vision. They're compromises. Sell me on your vision._\n\nWe're concerned with agreement that the situation is as described, that our vision is desirable, and that the situation can be reasonably improved. The alternatives listed above are 'roadmap alternatives', rather than design alternatives, precisely because those are different paths that the organization could take other than investing in a suite of technical solutions.\n\nWe have some understanding of characteristics that we feel are important to forward-looking technical solutions, and the next steps on the roadmap are focused on exploring those characteristics. The results of those explorations will dictate the concrete approaches we consider.\n\n**For the approach you're suggesting, what are they key choices in that approach that you want us to understand? Using Rust to implement a shared library is one such strategic choice. Are there others worth surfacing?**\n\nThe proposal to use Rust is essentially an assertion of two things: that supporting multiple applications on multiple platforms is part of Mozilla's future; and that native code is the right way to do that in this instance. Rust is, we will admit, an opinionated choice here.\n\nThere are other strategic/organizational opinions that lie beneath this document:\n\n* That some organizational coordination of storage — including centralized reusable implementations — is valuable, both to individual features and to the organization as a whole.\n* That totally separating syncing from storage doesn't work, and that pushing onto feature developers the burden of understanding syncing and other consumers, without tool support, is costly. Abstractions are needed.\n\n**What would have to be true for your approach to be a winning one? What are your riskiest assumptions here?**\n\n_I'm looking for a prioritized list of questions that need answers and assumptions that need to validated, an order to tackle them in, and how you're going to tackle them (ideally, as cheaply and as early as possible)._\n\nPhase 1 and Phase 2 capture what we think are the most significant technical risks: can we build a system that more closely meets [the requirements we've captured for sync and storage](https://docs.google.com/document/d/1XOJhW-qf4iIyGHwiSLgWEfXYDWLy44So5L8CdwH6pcg/edit), in a way that serves our strategic needs (_e.g._, around reusability), and maintain syncability, performance, and the developer experience (including the preservation of existing interfaces for as long as makes sense) — can we build it? can we make it sync? can we make it scale? can we get people to understand it?\n\nWe'll break these down into more detailed work items and questions as we go about working on those phases.\n\nFrom a product perspective, most (if not all) of the following would have to be true for this to be a success and to meet the needs of the engineering teams we’ve spoken to:\n\n* We have a sufficiently accurate understanding of the needs of Mozilla engineering teams.\n* We have a sufficiently accurate understanding of our users’ needs and problems.\n* Teams have a desire/need to store new data.\n* Teams have an increased need to more easily access existing stored data from other teams.\n* We can successfully collaborate across departments.\n* Users trust us with their data.\n* Engineering teams trust us with their data.\n* Existing storage is insufficiently easy to extend or scale.\n* Engineers agree with the shortcomings of the current situation.\n* Engineers would stop using _ad hoc_ solutions to create new products.\n* Engineers are OK with having less control of their integration due to being limited to APIs\n* There is enough of an overlap between all teams that would allow us to have solutions that fit all.\n* New smaller mobile products won’t need to pull down all of the data locally since that can be a battery and storage drain.\n* Mozilla wants to make personalized data-centric features (Firefox eco-system?)\n* Users want a personalized browsing experience.\n* In the case of making it ourselves, nobody has a working solution that we could use or fork which would allow us to achieve similar aspirations.\n* We have the technical expertise to build a new storage system and cloud storage infrastructure.\n* Our solution would provide increased flexibility to engineering teams in terms of adding new data types and changing existing ones.\n* For new projects, that we are faster and easier to integrate than _ad hoc_ solutions, ideally enabling new projects to more rapidly pop-up across the company.\n* We still want to encrypt all of our users data and work within those constraints.\n* That we can satisfy the desire for experimentation within product teams and their current lack of data storage solutions that enable them to do this.\n"
  },
  {
    "path": "text/0009-type-safety-systems.md",
    "content": "---\ntitle: JavaScript Type Safety Systems\nlayout: text\n---\n\n# JavaScript Type Safety Systems\n\nJavaScript is an untyped language and as such is susceptible to type safety bugs\nwhere a type other than that expected is used by mistake. For example trying\nto add a number to a string and expecting a numeric result.\n\nJavaScript type safety systems work as an extra layer of code syntax on top of\nthe raw JavaScript. Some kind of compiler or checker is used to verify that\ntype assertions made in the code are met allowing bugs to be detected.\n\nRecent studies have suggested that using type annotations in JavaScript can stop\nas many as 10% of bugs from entering the product.\n\n## Flow\n\nFlow is a type safety system developed by Facebook. When no types are provided\nin the source code it attempts to infer types based on variable assignments.\nFlow comes in two forms. The more common form involves adding types as an\nextended JavaScript syntax. This form must be removed by a transpiler at build\ntime. The other forms are comment based, one only supports function argument\ntyping the other supports any typing but is IMO quite ugly.\n\nFlow frequently finds bugs that aren’t errors and so would need work to correct\nbefore any typing can be used with the file. As an example in a file with 4k\nlines of code that we can assume to be largely bug free Flow found some 122\nerrors so one error per 32 lines. Two of them were potential bugs (though\nactually inconsequential). The rest were either Flow bugs, cases where Flow was\npointing out a potential issue that could never occur in reality or cases where\nwe use complications that Flow can’t reasonably be expected to understand.\n\n## TypeScript\n\nTypeScript is a typed superset of JavaScript that compiles to plain JavaScript.\nTypeScript files support compiling to an older version of JavaScript supported\nthan the version written. TypeScript does some type inference so some validation\nis performed even before any types are added but less than Flow.\n\nWhen run on an existing large file in the repository TypeScript found around one\nissue per hundred lines of code which is less than Flow found on the same file.\nMost of those issues were misunderstood types. None of them were errors that\nwould cause problems in the code.\n\n## Conclusions\n\nIn the case of both Flow and TypeScript significant work would need to be done\nin order to use them in existing code. Both suffer from a number of drawbacks:\n\n* Not understanding some modern JS syntaxes\n* Not understanding our module system\n* Requiring build-time processing leading to difficulties with logging and debugging\n\nIt doesn't seem worth the work at this stage to use either system in the main\nMozilla codebase.\n"
  },
  {
    "path": "text/0010-firefox-data-stores.md",
    "content": "---\ntitle: Firefox Data Stores\nlayout: text\n---\n\n# Firefox Data Stores\n\n## Introduction\n\nThere are a large number of data stores being used to hold data across all Firefox instances and platforms. There is currently nowhere that documents all these data stores and how they share information between them. We have 3 platforms, desktop, Android and iOS and each of these platforms shares data with the others. But not all the information is synced completely across all platforms, and the data itself is not stored in the same way everywhere. When starting to think about how we might provide a holistic user data solution for Mozilla, it is important to understand the scope of the problem and the specifics of the data that is being handled. \n\nWe therefore need to look at all of the data stores that we are currently using and document their purpose, structure, the kind of data that they hold and how they share that data.\n\n## Summary\n\nThe extensive documentation can be found in the [Firefox Data Stores](https://github.com/mozilla/firefox-data-store-docs) repository.\n\n* There are a total of 46 separate data stores in Firefox for desktop. Within these 46 stores, there are at least 10 different storage formats. Very few of these stores are well documented in the code. \n* Firefox for Android Java frontend uses 3 stores. It consolidates `places.sqlite` and `favicons.sqlite` into a single SQLite database, `browser.db`. It uses 4 different storage formats, `SQLite`, `JSON`, `XML` & `JS`. Android frontend documents its data stores well and uses a single file to define the schema for all its tables.\n* Gecko on Android uses the vast majority of the same stores as Gecko on Desktop. \n* Firefox for iOS has 5 data stores. `metadata.db` was created to store metadata for use in Activity Stream and was created separately to make syncing of this data at a later data easier. Due to a change in strategy, this store is no longer in use. iOS uses 3 different storage formats, `SQLite`, `JSON` and `plist`. It also writes credential information to the iOS keychain. iOS stores are not especially well documented, but the schemas are defined in a database specific files which makes discovering the schemas trivial. The exception to this is the reading list store which is defined separately from the others. iOS has very strong encapsulation of storage. The front-end code uses specific, narrow, safe APIs to read and write data, and storage manages sync metadata.\n* Desktop stores logins in an encrypted JSON store, `logins.json`. iOS keeps logins as a separate SQLite database, `logins.db`, that uses `sqlcipher` for encryption. Android uses a SQLite database, `signons.sqlite`, for storing logins. `signons.sqlite` used to be used on desktop too, but they were migrated into `logins.json`. Android frontend is currently in the middle of removing the dependency on the Gecko-side store, resulting in a set of login tables inside `browser.db`, but the work is as yet incomplete.\n* `prefs.js` is a simple key value file written to disk, but it contains well over 1200 entries. Its design means that changes made are often not written out until shutdown. Therefore, if the shutdown is not clean, any changes made during the session are lost, resulting in potential inconsistency between prefs and state elsewhere in the system.\n* Firefox desktop has an entire SQLite database, `storage.sqlite`, that contains no tables. It is used entirely to reference the schema version value and uses that value to create the version number for the Storage directory. The Storage directory contains all of the files and stores used by the app.\n* Of the data fields stored by desktop, only 8% are available to Firefox Sync.\n* A visual overview of the [fields made available to Firefox Sync](https://docs.google.com/spreadsheets/d/1k9_K7Dc3q2h3SDV0vwjTgJou-ndza6WuobyJ1bbemtc/edit?ts=5977ab9d#gid=1269587388) and how each sync record is implemented on each platform is also available.\n\n"
  },
  {
    "path": "text/0011-fluent-in-prefs-design-review.md",
    "content": "---\ntitle: Fluent in Prefs Design Review\nlayout: text\n---\n\n# Fluent in Prefs Design Review\n\n## Introduction\n\n[Fluent](http://projectfluent.org/) is a \"family of localization libraries designed to improve how software is translated.\" Mozilla's localization engineering team (the \"engineering team\") is migrating Firefox's localization architecture from DTD/properties files to FTL files processed by Fluent, which will enable richer and more accurate translations, runtime locale loading/switching, runtime LTR/RTL and pseudo-locale switching, and a fallback locale. The first target for this migration is the preferences interface (the about:preferences page and its various subdialogs).\n\nThe engineering team met with a set of architects and domain experts (the \"review team\") to discuss risks and mitigations, which constituted a lightweight \"design review\" of the project's architecture. This document describes the concerns that the review team raised and the risk mitigations that the teams discussed.\n\n## Why Preferences\n\nThe team chose preferences because they're text-heavy, include users' names (which exercises Fluent's support for grammatical gender), and because the precision of text matters in preferences to ensure users clearly understand the implications of changes. So preferences are a good way to prove the change to Fluent before adopting it more widely.\n\nFluent is a big change to the localization process, and the team plans to validate the change by translating just four strings initially (the \"name\" strings in brand.dtd/properties?) and then verifying those changes across the localization toolchain.\n\n## Chrome Privileges\n\nThe review team asked whether there are parts of preferences that don't have chrome privileges, and if that influences the use of Fluent. It seems like the only such part is the Firefox Accounts image (???), which loads over a network and is localized independently of Firefox, so it won't be an issue.\n\n## Search Incompatibility\n\nThe search feature of about:preferences is an issue, as the way that it searches preference labels and descriptions is not compatible with the way that Fluent injects text into the page.  The teams discussed options and concluded that there is a straightforward way to update the search implementation to be compatible with Fluent.\n\nSpecifically, FTL messages support arbitrary attributes, so an FTL message for a searchable preference control could include a \"searchterms\" attribute containing the text to search, i.e.\n\n```\nkey1\n  .label = \"click\"\n  .searchterms = \" … \"\n```\n\nAttribute values can include references to other strings within the FTL file, so localizers could avoid string duplication if desired.  This model would also enable localizers to customize search terms to include extra keywords that aren't included in other strings related to the control.  Finally, this model supports English fallback for searches via the creation of a secondary *document.l10n* object with the `en` locale.\n\nThe review team raised some concerns about the performance of runtime searches and suggested that the engineering team consider generating an index from search terms to the controls they match, either at build time or on first search.  Some participants suggested being careful to avoid prematurely optimizing, however. In the end, the teams didn't reach consensus on the optimal solution, and further research/experimentation is warranted here.\n\n## Schedule Risk\n\nAfter validating the process with the first four strings, the engineering team plans to migrate the rest of the strings, but they're aware of the large set of pending changes for XBL replacement in [bug 1379338](https://bugzilla.mozilla.org/show_bug.cgi?id=1379338), and they'd prefer to await those before proceeding to minimize merge hell.\n\nThe XBL replacement patch author ([myk](https://mozillians.org/en-US/u/myk/)) and reviewer ([jaws](https://mozillians.org/en-US/u/jaws/)) were both members of the review team, and they noted that those changes should land soon and thus shouldn't create schedule risk.\n\nAnother member of the review team noted that another team is rewriting the \"delete cookies\" dialogs, and the engineering team should coordinate with that team.\n\n## Migration\n\nThe teams discussed tools for migrating from DTD/properties files to FTL files. They also discussed the possibility of migrating strings from one FTL file to another, as the difficulty of migrating strings between DTD/properties files is a common frustration with the current system.\n\nThe engineering team has developed tools to migrate strings from DTD/properties to FTL files, but they don't yet have a way to migrate them from one FTL file to another. That doesn't regress functionality, but the review team still recommended that the engineering team consider tackling that problem in the future.\n\nThe teams also discussed merging the various preferences DTD/properties files into a single FTL file, since their current ontology is out-of-date (because pref categories have changed over time, and the files haven't changed with them), and since there isn't a reason to load only a subset of strings (especially given the new search feature, which needs to search across all of them).\n\nThere was also some discussion about the conflict between engineers, who bias toward coalescing \"identical\" English strings used in different contexts (such as \"open\") because DRY; and localizers, who separate such strings because they aren't actually always identical in all locales.\n\nCoalescing is considered harmful, and a concern was raised that merging preferences strings into a single FTL file might encourage engineers to do it even more. The teams concluded that this concern can be addressed at the policy/review level.\n\n## String Changes\n\nThe review team probed about Fluent support for string changes, and the engineering team explained that support is as good as the current system, which has two ways to modify a string, and work is planned to make it even better by adding a third way.\n\nSpecifically, under both the current system and the current version of Fluent, a developer can make a minor change to a string (f.e. a typo fix) that shouldn't affect its translations by preserving the string's key, and they can make a major change to a string that should affect its translations by changing the string's key.\n\nIn the future, Fluent will support versioning of a string key via a tag, such that developers can also make changes that affect localizations without having to change the string's key. (There was some discussion about which kinds of changes would fit into this bucket.)\n"
  },
  {
    "path": "text/0012-jsonfile.md",
    "content": "---\ntitle: A brief analysis of JSON file-backed storage\nlayout: text\n---\n\n# A brief analysis of JSON file-backed storage\n\nSeveral components in Firefox — including libpref, XULStore, autofill, and logins — follow a general pattern:\n\n- Store data in memory, usually as a JS collection.\n- Load that data from disk during initialization.\n- Persist that data to disk in its entirety, usually serialized as JSON, at some point after each write, and/or at shutdown.\n\n[JSONFile.jsm](https://dxr.mozilla.org/mozilla-central/source/toolkit/modules/JSONFile.jsm) is a centralized implementation of this pattern. It supports Promise-based asynchronous loading, synchronous loading, and flushes asynchronously, defaulting to 1500ms after each change. libpref flushes after 500msec.\n\nThis approach is well suited to data that:\n\n- Changes infrequently.\n- Is relatively small.\n- Is only ever directly read or written by a single thread.\n- Needs to be kept entirely in memory for reads.\n- Has a straightforward JSON encoding (encoding binary blobs, such as images, in base64 is relatively inefficient and typically avoided).\n- Has limited durability requirements.\n\nAdvantages of this approach:\n- It's relatively simple, with serialization and deserialization being the only steps.\n- Because the file is rewritten frequently, there is some hard-to-measure resilience to gradual file corruption.\n- The file on disk is — unless compressed — human-readable and editable, which is convenient for development, test, and support.\n- Automatic opt-in whole-file compression is offered by JSONFile.\n- The in-memory data can be queried and manipulated synchronously and quickly, assuming the representation is a good match for retrieval patterns.\n- Development is 'natural' for front-end developers — get things working with only in-memory data, then add persistence on top.\n\nDisadvantages:\n- Versioning is often an afterthought: engineers rarely think to version in-memory data formats.\n- Change tracking is almost always an afterthought, thanks to the 'natural' development process: adding syncing later becomes difficult.\n- Users feel relatively empowered to edit readable files on disk, which can result in persistent state that was never created by the component's own logic. The use of compressed formats makes this less likely.\n- Frequent writes will cause the entire file to be written to disk repeatedly, [which causes complaints about SSD impact](https://bugzilla.mozilla.org/show_bug.cgi?id=1304389).\n- The entire file typically needs to be read into memory to be used, which increases memory footprint and can impact startup time.\n- Writes at shutdown harm the user experience and don't happen during a crash.\n- In order to achieve atomic file writes, the entire file contents briefly exist twice on disk, which could be problematic on mobile platforms for large data sets.\n- The only approach to cross-process use is duplication of all or part of the in-memory data, which can increase memory footprint. It's not possible for multiple processes to collaborate on the same data via the filesystem. This is why Firefox for Android still uses the SQLite implementation of `nsILoginManagerStorage`: it allows the Android-side `ContentProvider` code to read and write saved logins without worrying about Gecko, which nominally owns the backing storage, but has a shorter lifetime than the enclosing Android code. Using desktop's JSON-backed storage on Android would require launching Gecko on each sync, using messaging for safe access to the file contents.\n- The in-memory object is essentially a non-transactional write-back cache. This has several issues:\n  - It makes isolation (readers can't see in-progress writes) and atomicity (all writes complete or none do) difficult: we typically mutate the in-memory object directly, and so an exception can cause partial changes to 'commit', and readers will see each change as it is applied.\n  - Similiarly, timed flushing makes consistency difficult: it's possible for only some of a set of asynchronous writes to be flushed to disk because the flush beat the last few writes, leaving the data inconsistent after a crash.\n    \n    JSONFile.jsm advises that callers do all of their work synchronously on the main thread to avoid the possibility of concurrent readers or partial writes:\n    \n    > The raw data should be manipulated synchronously, without waiting for the\n    > event loop or for promise resolution, so that the saved file is always\n    > consistent. \n     \n    This can cause jank.\n- Synchronization of data stored in this way takes some careful thought. Syncs typically take hundreds of milliseconds or more, and involve asynchronous network operations, which makes exclusive synchronous access to the in-memory data between reads and writes infeasible.\n- There is a tension between durability (that is: writes that complete are permanent) and performance. We typically choose not to flush the file immediately and synchronously after every change, but not doing so leaves a window in which a crash would cause data loss. By default, that window is 1.5 seconds, plus the time needed to write and atomically switch the files. This forces careful consumers to manually flush if they want their writes to stick, which is a bad pattern.\n- In-memory objects lack the sophistication of most databases. This leads to front-end features building their own [simple query engines](https://dxr.mozilla.org/mozilla-central/source/toolkit/components/passwordmgr/storage-json.js#296) for [finding records by linear search](https://dxr.mozilla.org/mozilla-central/source/browser/extensions/formautofill/FormAutofillStorage.jsm#1081).\n- Similarly, these stores must reinvent (or contribute to JSONFile) their own:\n  - Versioning and upgrade logic.\n  - Validation and schema checking.\n  - Concurrent access patterns.\n  - Write coalescing/async update patterns.\n  - Tooling, if existing general-purpose JSON tools (*e.g.*, `jq`) are not sufficient.\n  - Backup, if any.\n  - Indexing, if any.\n  - Datatype serialization (*e.g.*, timestamps).\n- Finally, the use of `JSONFile.jsm` is intimately linked to Gecko; it's a poor choice for code that will later need to be deployed on other platforms.\n"
  },
  {
    "path": "text/0012-process-isolation-in-firefox.md",
    "content": "---\ntitle: Process Isolation in Firefox\nlayout: text\n---\n\n# Process Isolation in Firefox\n\nRandell Jesup, Mozilla Browser Architecture team\n\n## NOTE: this document is a Work in Progress!\n\n\n## Overview\n\nWe’ve recently moved a number of parts of Firefox into separate processes (e10s, multi-e10s, GPU process, compositor, WebExtensions process, GMP).  This has produced some serious improvements in both security (sandboxing content, codecs) and stability (restarting GPU process when a GPU driver crashes, etc).  This project is to evaluate how much further we can push process isolation to further improve these things.\n\n### Problems:\n\n* We have large processes, running many unrelated items of highly varying security and stability properties.   A single bug (including in OS drivers) in many cases will take down either a major part of your tabs, or the master process and by extension the entire browser.\n\n* In a related concern, a single exploitable bug gives access to a large part of the browser.  Even if it’s in the Content process, it can give access to ¼ of your tabs, and because Content processes have very wide needs to access data and communicate with the Master process, the possibilities for either sandbox escape or information leakage are quite high.\n\n* Features and capabilities often have code strewn across various parts of the tree, increasing the maintenance cost and risk of unrelated changes breaking them.\n\nThere are some secondary benefits we hope to achieve by doing this, such as decoupling parts of the system and providing more-stable interfaces to them, as well as easing some aspects of unit testing.  There may be some advantages in build times or cost of maintenance; we’ll see.\n\nThere are costs: development time, memory use, and performance all can or will be negatively impacted.  Part of this project is to quantify these impacts (and hopefully reduce them) in order to guide the decisions on how far to push this process.\n\nChrome/Chromium has been doing similar work recently on \"[Servicification](https://www.chromium.org/servicification)\".  This is related to their slowly replacing classic ipc/chromium-based IPC with “[Mojo](https://chromium.googlesource.com/chromium/src/+/master/mojo/README.md)”; a new IPC implementation (and ipdl-like layer) that has improved performance compared to classic Chromium IPC.  Note that some ways Mozilla uses IPC might avoid some of the performance costs in Chrome (multiple channels, PBackground and the like, for example) - but we haven’t yet assessed how much overlap there is between Mojo and our additions to Chromium IPC.  It may be that with some smaller modifications our use of Chromium IPC will be as efficient (or nearly so) as Mojo.\n\nChrome is also working on [Site Isolation](https://www.chromium.org/developers/design-documents/site-isolation), to avoid a compromised Renderer from breaking cross-origin isolation (more details below).\n\nAs part of this work, since it affects the performance impact of Process Isolation, we plan to explore the potential of adopting Mojo for Mozilla IPC (either wholesale, or progressively as Chrome has been doing).\n\nAlternatively, and maybe more interestingly, we could look at using a Rust IPC crate and some added interface logic. <Need some more concrete suggestions/pointers here>\n\nNote: we don’t want to just \"follow\" Chrome’s lead, but to do what’s smart for the users and Firefox, whether it’s similar to Chrome or not.  Leapfrogging them in some manner would be great, but is not a requirement.  If we can leverage work or research they’ve done to reduce our cost or risks, great.\n\n# GOAL\n\n* Develop a browser-wide direction for Process Isolation:\n\n    * How much Servicification should we do?\n\n    * How many Content Processes should we use?\n\n    * Should we consider a Chrome-like Process-Per-Origin or Process-Per-Iframe model?  What are the implications of this?\n\n# Tentative Work Plan\n\n1. Measure the overhead of using Process Isolation:\n\n    1. Memory cost for various scenarios (which largely depend on what part of firefox code need to be initialized/used in the process - XPCOM, etc)\n\n        1. Memory overhead for a Process on Win10 x64 appears to be circa 7-10 MB (Private) for a process with minimal XPCOM inited, IPC, etc.  The GPU process (which also loads GPU drivers, etc) is on the order to 20MB, and anecdotally the GMP process uses on the order of 10MB, which is in line.\n\n    2. Performance cost - latency and throughput of calls through IPC (with classic IPC and Mojo, and perhaps a Rust IPC crate)\n\n        2. Still to be measured\n\n    3. Evaluate if we can make Chromium IPC as efficient as Mojo\n\n        3. I suspect we can; the main perf advantage Mojo has on classic IPC is one less thread-hop per message (2 less per round-trip - 4 vs 6).  The overall code is probably a lot \"cleaner\", but it would be a fair bit of work to convert over, though probably much of it could be hidden by ipdl.\n\n        4. We believe that Mojo’s shutdown code may be more robust/better engineered than IPC’s; shutdown has been a common source of crash/security bugs.\n\n        5. We think it might be possible in some special(?) cases to avoid a thread hop on the receiving side as well as the sending.  Mojo does not do this.\n\n    4. Startup cost - cost to browser startup time\n\n        6. Unknown.  Need to measure the cost in time to start a minimal process, and see how many we need at startup.  We may be able to partly overlap creation and startup of service processes.  Expectation is that this will not be a blocker.\n\n    5. Service startup time cost - cost on first use of a service which requires spawning a Process\n\n        7. Evaluate using \"embryo\" processes to allow the forked processes to be smaller but not pay new relocation/etc costs (i.e. don’t fork the Master process and all that entails when we spin up a process/service).  (B2G did something like this with [Nuwa](https://bugzil.la/nuwa).)  Note that this doesn’t help Windows apparently, and there was some sort of issue with this on Mac - these need to be examined/verified.\n\n2. Analysis of Process Isolation\n\n    6. Analysis of the code maintenance impact\n\n    7. Analysis of the stability impact\n\n    8. Analysis of the security impact\n\n    9. Analysis of embryo processes\n\n        8. Per-OS\n\n    10. Analysis of IPC options\n\n        9. Update/improve current Chromium IPC\n\n        10. Mojo\n\n        11. Rust IPC\n\n    11. Android analysis - android will likely require different tradeoffs\n\n3. Develop a preliminary list of potential subsystems/features to consider for Isolation\n\n    12. Necko is already planning to Isolate network socket access and protocol code (and some crypto code) after 57 or 58 -- [Bug 1322426](https://bugzilla.mozilla.org/show_bug.cgi?id=1322426)\n\n        12. They expect to land code in 61 behind a pref, and enable it in release in 63.\n\n            1. This will not be happening in the near term, due to staffing changes.  It’s not yet clear when or if this will occur.\n\n    13. Video camera capture code (and screensharing) is another prime target, as it’s already remoted over IPC even when running non-e10s.  The way this works is very similar in principle to Chrome’s remote-services-over-Mojo approach.\n\n    14. Places in particular, eventually profile data access in general. This pushes storage asynchrony to the process boundary and decouples lifetimes (background storage and syncing, faster 'startup' and relaunch, with Places possibly living longer than a crashed browser). Future technology refactors could make this standalone process reusable outside of desktop Firefox. No bugs filed yet. \n\n    15. Font and/or Image code to avoid or reduce duplication of data between Content processes and the Compositor.\n\n    16. Printing\n\n    17. PDF display/PDFium\n\n4. Look at the Content Process state and model (mostly this has been done in the e10s team)\n\n    18. How far do we want to push the model towards Chrome’s 1-per-origin/iframe model?\n\n        13. Probably not as far… note however that Chrome has closed the gap we created with them on memory use. [reference?]\n\n        14. Even Chrome can’t get away with their stated goal (yet?)\n\n            2. While site isolation with a small number of sites is in the 15-20% range, this is largely because of the base overhead of Chrome (Master process, GPU processes, [zygotes, ](https://chromium.googlesource.com/chromium/src/+/lkcr/docs/linux_zygote.md)etc).  WIth  17 tabs (a mixture of simple and complex sites), the measured overhead was about 50%.\n\n    19. How much does servicification help reduce Chrome process overhead (avoiding N instances of things)\n\n        15. It may not help a lot\n\n    20. How much can this work help [sandbox hardening](https://www.google.com/url?q=https://wiki.mozilla.org/Security/Sandbox/Hardening&sa=D&ust=1507349372899000&usg=AFQjCNHD1aRxpBa1fuJNcVKxytjULX6krA)?\n\n        16. Main Process will still need file access probably\n\n5. Very speculative: examine if the current Master Process could be moved to be a child process of a thin Master Process, allowing restarts on Master Process crash without reloading all the running Content Processes.\n\n    21. **Very** likely this is too hard and/or too much work.  There’s lots of state stored in the Master on behalf of the Content processes; the reconnect operation between Content and restarted Master would be considerably complex.\n\n# Previous work\n\n* GMP\n\n    * Very tight sandbox, one sandbox per-origin\n\n* E10S\n\n* GPU process\n\n* Compositor\n\n* WebExtensions\n\n* Examination of the options for sandboxing Audio capture and playback, as well as other parts of the Media code: [Media, WebRTC and Audio Sandboxing Plans](https://docs.google.com/document/d/1cwc153l1Vo6CDuzCf7M7WbfFyHLqOcPq3JMwwYuJMRQ/edit)\n\n* Background docs (needs to be updated but some useful info maybe)\n\n    * [https://wiki.mozilla.org/Security/Sandbox/Hardening](https://wiki.mozilla.org/Security/Sandbox/Hardening)\n\n    * https://wiki.mozilla.org/Security/Sandbox/Process_model\n\n# Chrome’s Servicification and Mojo\n\nChrome has been discussing Servicification since roughly early 2016, and major work on it has begun this year (2017).  This is the primary document: [Chrome Service Model](https://docs.google.com/document/d/15I7sQyQo6zsqXVNAlVd520tdGaS8FCicZHrN0yRu-oU/), and this is the primary root of the Servicification work: [Servicification](https://www.chromium.org/servicification).\n\nAn example of one item currently being moved to a Service is Video Capture: [Design Doc](https://docs.google.com/document/d/1RLlgEdvqRA_NQfSPMJLn5KR-ygVzZ2MRgIy9yd6CdFA/edit#heading=h.qjpzx8d6k7t5) and [detailed plans and measurements](https://docs.google.com/document/d/1Qw7rw1AJy0QHXjha36jZNiEuxsxWslJ_X-zpOhijvI8/edit#).  Another which I think has been completed is the [Prefs Service](https://docs.google.com/document/d/1JU8QUWxMEXWMqgkvFUumKSxr7Z-nfq0YvreSJTkMVmU/edit).\n\n[Mojo](https://chromium.googlesource.com/chromium/src/+/master/mojo/README.md) consists of a set of common IPC primitives, a [message IDL format](https://chromium.googlesource.com/chromium/src/+/master/mojo/public/tools/bindings), and a bindings library (with generation for a number of languages; undoubtedly we’d need to add Rust binding generation -- [C++ bindings are here](https://chromium.googlesource.com/chromium/src/+/master/mojo/public/cpp/bindings)).  Mojo has been measured in Chrome as being about ⅓ faster than classic IPC, and produces ⅓ less context switches.  (Probably these two facts are related, and their [performance analysis](https://chromium.googlesource.com/chromium/src/+/master/mojo/public/cpp/bindings) indicates that not having to \"hop to the IO thread\" is part of why it’s faster, which makes sense.)\n\nOne thing we plan to experiment with is seeing if (leveraging our IPDL compiler) we can fairly transparently replace (some?) existing Chromium IPC channels/messages with Mojo.\n\nChrome has docs on [how they move legacy IPC code to Mojo](https://chromium.googlesource.com/chromium/src/+/master/ipc#Using-Services).  This is a (somewhat dated) cheat sheet on [moving code from IPC and Mojo](https://www.chromium.org/developers/design-documents/mojo/chrome-ipc-to-mojo-ipc-cheat-sheet).\n\nOne interesting tidbit from that cheatsheet: \n\n**IPC**\n\nIPCs can be sent and received from any threads. If the sending/receiving thread is not the IO thread, there is always a hop and memory copy to/from the IO thread. \n\n**Mojo**\n\nA binding or interface pointer can only be used on one thread at a time since they're not thread-safe. However the message pipe can be unbound using either Binding::Unbind or InterfacePtr::PassInterface and then it can be bound again on a different thread. Sending an IPC message in Mojo doesn't involve a thread hop, but receiving it on a thread other than the IO thread does involve a thread hop.\n\nMojo has extensive support for message validation: \n\nRegardless of target language, all interface messages are validated during deserialization before they are dispatched to a receiving implementation of the interface. This helps to ensure consistent validation across interfaces without leaving the burden to developers and security reviewers every time a new message is added.\n\n# Overhead Details\n\n    1. Memory use.  Content Process overhead is tracked in [Bug 1436250](https://bugzilla.mozilla.org/show_bug.cgi?id=1436250).  It’s measured both with a small patch to dump the ASAN heap state on command, and using a DMD build with specific environment options.\n\n        1. Minimal process (with IPC)\n\n        2. With XPCOM\n\n            1. 7-10MB\n\n        3. Full Content process\n\n            2. 25-30MB (varies by OS, memory model (32 vs 64), and system details (fonts, etc))\n\n            3. Content Process overhead is critical for Site Isolation\n\n    2. Performance -- measure each with small messages, large messages, and shared-memory payloads (anything else?)\n\n        4. Latency\n\n        5. Messages/second\n\n# Security implications\n\nMoving services and other code into sandboxed processes should generally increase the resilience of the system to security bugs.  In particular, compromising the system in one sandboxed process will require a sandbox escape of some sort to leverage that into full or increased control, and generally will only expose data already flowing through the Service to the attacker - and for many sandboxed processes, exfiltrating compromised data would be much harder.  \n\nHow hard exfiltration would be depends on what sort of data flows through the Service, how locked-down we can make the process, and if the output of the process normally is in some way visible to content - for example if the process did image decoding, then data could be exfiltrated by using it to break cross-domain restrictions (such as taking the content of image A in domain X, and outputting it in place of image B from domain Y (the attacker’s domain), allowing the attacker to render it in a canvas).\n\nAnother way that isolation will help security is by separating memory pools - memory errors such as UAFs can become much harder to exploit if you can’t put the memory system under load (forcing reallocation of the memory with content you control, for example).  In a Content process with JS running (and tons of other stuff), this is often not too hard; in an isolated Service it might be very hard indeed.\n\nOnce a Process is compromised, leveraging that into an exploit requires escaping into other Processes (using further bugs), or leveraging an OS bug.  How hard that is depends on the OS and how locked-down we can make these Processes.  \"Small\" processes may have much smaller OS attack surfaces, though this might be tougher to do on (say) Windows due to granularity of permissions.\n\n# What Chrome does\n\nChrome doesn’t actually use a Process-per-tab (or origin), though many people believe it does: see [Peter Kasting’s post from June](https://plus.google.com/+PeterKasting/posts/TC4ACtKevJY).  (That was partially in response to some of our announcements around E10S.)  The number of Render processes they use depends on the available memory - though it sounds like they have bugs there, and that may cause them to overrun and slow down the user’s system.\n\nChrome is working on [Sit](https://www.chromium.org/developers/design-documents/site-isolation)[e Isolation](https://www.chromium.org/developers/design-documents/site-isolation).  Part of this is putting iframes OutOfProcess from the main page renderer, but more generally it’s about not using a renderer (Content process) for more than one origin.  This has some serious downsides if taken to an extreme, and currently they’re planning to do this only for \"high value\" sites.  (It’s unclear what “high value” means here; one presumes banks, paypal, and other especially juicy targets.)\n\nAs mentioned above, Chrome is increasing the number of non-Render processes they use as part of Servicification.\n\nThe [Chrome Process Model](https://www.chromium.org/developers/design-documents/process-models) document is useful, but very out of date with current details - for example, the [Site Isolation](https://www.chromium.org/developers/design-documents/site-isolation) work they’ve done.   Some of the code for all these decisions is [here](https://cs.chromium.org/chromium/src/content/browser/renderer_host/render_process_host_impl.cc?rcl=98c9b1d4fc903ab9dee06a98e7497b11de760449&l=341). \n\n# What Edge does\n\nIn \"[Browser security beyond sandboxing](https://blogs.technet.microsoft.com/mmpc/2017/10/18/browser-security-beyond-sandboxing/)\" Microsoft goes into detail on a Chrome vulnerability, but also highlights some of what they’ve done - in particular [Arbitrary Code Guard](https://blogs.windows.com/msedgedev/2017/02/23/mitigating-arbitrary-native-code-execution/#HGTKcYGQeuOxvYIB.97): “ACG, which was introduced in Windows 10 Creators Update, enforces strict Data Execution Prevention (DEP) and moves the JIT compiler to an external process. This creates a strong guarantee that attackers cannot overwrite executable code without first somehow compromising the JIT process, which would require the discovery and exploitation of additional vulnerabilities.”  Their post introducing ACG is [here](https://blogs.windows.com/msedgedev/2016/09/27/application-guard-microsoft-edge/).\n\nIn more detail: \"To support this, we moved the JIT functionality of Chakra into a separate process that runs in its own isolated sandbox. The JIT process is responsible for compiling JavaScript to native code and mapping it into the requesting content process. In this way, the content process itself is never allowed to directly map or modify its own JIT code pages.\"\n\nThis suggests that we should consider moving the JIT itself out of the main process space, and just share the result of the JIT back with the Content process requesting it.  Since we already do JIT on a non-MainThread, the main issue here is probably shared-memory management.   There will be some perf tests to do on this to validate if this is feasible within our current overall architecture, but it seems possible.  This is being tracked in [Bug 1348341](https://bugzilla.mozilla.org/show_bug.cgi?id=1348341), and Tom Ritter has been investigating in [this doc](https://docs.google.com/document/d/13HwuPNxabIONDVTKInMkkIitXxcB_kmBjMTjA7dHtSE/edit).\n\nNote that if the JIT process is compromised, anything running through it is as well, and any content processes it provides code for.  This would imply that JIT processes may need to be tied to a single requesting Content Process to avoid stepping backwards in security here (and this also increases the potential memory cost).\n\n"
  },
  {
    "path": "text/0013-ipc-security-models-and-status.md",
    "content": "---\ntitle: IPC Security Models and Status\nlayout: text\n---\n\n# IPC Security Models and Status\n\nRandell Jesup\n\nReviewers: pauljt, baku, bkelly, posidron, ehsan\n\n## TL;DR:\n\n* Considerable additional work could be done validating IPC parameters received by the Master process from Content (and other) processes\n\n* More emphasis on Best Practices should be used in IPC work - especially documenting the security aspects in-tree\n\n* There is some cruft in our IPC code left over from B2G, though some of it may need to be kept and updated instead of removed\n\n* We should consider either moving to Mojo, or mirroring the partially-automatic validation of IPC arguments provided by Mojo\n\n* A more in-depth look at how shmem is used might be valuable\n\n* Improved automated fuzzing would be good\n\n* An audit of 12 semi-randomly-chosen IPC protocols is attached (of ~144)\n\n    * One possible security issue found\n\n## Current State:\n\nIPC security is basically ad-hoc, implemented by each IPC protocol implementation.\n\nThe basics are that many parts of our system are gatewayed by using Principals, which encode the origin and related information associated with the document. Largely we rely on good reviewers, and that’s more for avoiding sec issues like UAFs.\n\nThere are currently many components that send (child-to-parent) a Principal or a PrincipalInfo, but the parent is not really able to know if the child is allowed to use that principal.  We should be able to validate what Principals a Content Process is allowed to use with data stored in the ClientManagerService.  This is being worked on in [bug 1432825](https://bugzilla.mozilla.org/show_bug.cgi?id=1432825) and [bug 1432831](https://bugzilla.mozilla.org/show_bug.cgi?id=1432831), though that just covers a single case; resolving bug 1432831 would provide examples on how to do similar argument validation elsewhere.\n\nAs an example of Principal validation, when BroadcastChannel is sending a message, it sends the origin and its principal. The check is done on the parent side. \n\nOnce we have Site Isolation, the danger when Principle ownership isn’t checked is that if a compromised Content Process (somehow) knows a principal that’s valid in another document, it could pass it up, masquerading as that other document.  Without checking if that document is loaded into that Content Process, the Master process will allow the action to proceed with an incorrect Principal.  This can lead to all sorts of information leakage or spoofing. (Note: since a current Content Process could be navigating to a new origin, we can’t really stop a compromised Content Process from getting a Principal for a random origin until we have support for process-per-origin (or set of origins) and necessary process switching code for navigations).\n\nThe issue isn’t just Principals, however; all arguments must be checked, and where possible validated against what a given sender should be able to use.\n\nIn addition to information leakage, crafting arguments for an IPC message can also lead to considerable risk of sandbox escape - sending an IPC request with invalid arguments to trick the Master Process into taking an action that leads to the master process hitting a UAF or other security issue.  Luckily, even in this case it’s hard to craft an escape, but this greatly increases the possibility.\n\nSecuring IPC code is important.  It’s more critical as we move towards site isolation - compromise of a Content Process with 1/4 your tabs (and everywhere you browse in the future in those) is pretty bad, so a sandbox escape is only incrementally worse, but with site isolation a compromised process may be worth much less, unless it can escape.\n\n## Best Practices:\n\n* Firefox [IPC guide](https://wiki.mozilla.org/Security/Sandbox/IPCguide)\n\n* Google’s legacy IPC [security tips](https://www.chromium.org/Home/chromium-security/education/security-tips-for-ipc) (read for full explanations)\n\n    * (paraphrased/subsetted):\n\n    * **Trust only the ****browser**** \"****Master\" ****(aka “Chrome”) process.**\n\n    * **Sanitize and validate untrustworthy input.**\n\n    * **Whitelisting is better than blacklisting**\n\n    * **Safely handle known-bad input**\n\n        * Don’t RELEASE_ASSERT on bad data\n\n    * **Use and validate specific, constrained types; let the compiler work for you**\n\n        * Use types IPC knows about, try to avoid bare strings and custom serializers\n\n    * **Keep it simple.**\n\n    * **Be aware of the subtleties of integer types.**\n\n    * **Be aware of the subtleties of integer types across C++ and Java, too.**\n\n    * **Don't leak information, don't pass information that would be risky to use.**\n\n        * Be careful with shared memory mappings (specifically tracking their sizes on either side of the IPC channel). Do not store and trust sizes on one side of the channel. Avoid specifying the size when calling Map.\n\n        * (many more in the doc)\n\n* Carefully document lifetimes of objects that can receive IPC messages and ensure (and comment in the code) how we know no more messages can be received for an object\n\n    * Document the valid ranges for parameters, or validation criteria\n\n* Provide guidance on when a Principal should be required for IPC protocols\n\n    * This primarily applies for info going to/from the Master Process, though as we get a more-complex web of Processes this might be needed elsewhere. \n\n    * An example where it isn’t needed would be GMP webrtc codec processes, which are inherently per-site already and only communicate with a specific Content Process and webrtc connection within it.\n\n* Validate parameters, lengths and existence of any data received from another Process.  \n\n    * It’s less critical to check validity of data received from the Master Process, but even there it may help catch problems and bugs.  Also, in some cases, an attacker might be able to leverage a bug in code in the master to cause it to send a bad message (or race) to the Content Process, provoking a UAF/etc in content they can exploit.  It’s also possible to attack another non-Master process (Content Process, Extension Process, etc) by having the Master relay messages to it, as is often done in Extensions.\n\n        * Compromising the extension process could lead to control or disclosure of all content data\n\n    * Keep lists of things that are associated with a specific connection/Content Process to do validity checks (perhaps typically a Hashmap)\n\n    * Be very careful about overflows with data from the Content Process\n\n* Where possible, give tokens out instead of raw data, if the other process doesn’t actually need to manipulate or examine the data.  This is similar in concept to things like fd’s, but with added validation whenever doing a lookup of a token (again, typically a hashmap, but perhaps not always).  The lookup routine is a good place to put ownership checks and enforce their being checked.\n\n* Fuzzing - we’ve done this before (fuzzing Pickle/etc).  We can probably do more.  \n\n    * Faulty\n\n        * Process bi-directional fuzzer in real-time.\n\n        * Integrated in --enable-fuzzing builds available at TaskCluster.\n\n        * Fuzzes pickled messages and/or pipe states.\n\n        * Runs in multiple instances in our fuzzing cluster at EC2.\n\n        * Upgrade of EC2 instance required which supports HW counters for rr. Important for general test-case reproducibility. There is currently no way to feed a mutated message into Firefox from the outside of the process.\n\n        * Requires targets (i.e mochitests, test-suite) to trigger individual IPC messages.\n\n            * We have no way to know if all IPC messages are hit in our testing\n\n            * We could keep stats on numbers of different IPC messages in debug builds\n\n    * [Meta bug for fuzzing](https://bugzilla.mozilla.org/show_bug.cgi?id=fuzzing-ipc-ipdl) (80+ bugs)\n\n    * Adding fuzzing for every new protocol would be useful.\n\n    * Christoph Diehl has a list of improvements for Faulty, and is also looking at how Mojo uses Google’s fuzzer.\n\n* Discovering dead/unused IPDL APIs. We’ve had unused IPDL APIs left in the browser that could be used as attack vectors.  It would be nice to be able to use automated tools to get rid of such code.\n\n    * As an example, the partial audit found that the DocumentRenderer IPC protocol was unused (and had a security vulnerability), and it has been removed.\n\n## Possible Improvements to our IPC use:\n\n* Provide more support for IPC parameter validation\n\n    * [Bug 1430159](https://bugzilla.mozilla.org/show_bug.cgi?id=1430159) - specified validator for specific IPC protocols\n\n    * [Bug 1325647](https://bugzilla.mozilla.org/show_bug.cgi?id=1325647) - Automatic validators for integers generated by the IPDL compiler \n\n    * Support classes for checking token/principal ownership\n\n    * Leveraging improved argument checking support in Mojo if we switch from IPC/chromium to Mojo\n\n        * Mojo heavily leverages C++ and types for validation, and also allows custom validators\n\n    * Add the possibility to mark each IPC method with **[principal_needed]** or something similar. Maybe that would be nice to be the default for each method. Basically, each method should have a principal or a principalInfo passed as argument. This would be interesting, if we have the possibility to validate which principal is allowed to run on the content process. \n\n* We could move to a model closer to the dbus model, where Services are available under Well Known names, and clients can access them safely without worrying about if the target object exists (anymore).  This would help remove a common failure case, where due to timing races an IPC request for a deleted object is received.\n\n* A form of cross-IPC/process \"weakptr\"\n\n* Possibly rewrite MessageManager (not in JS)?\n\n## Further Investigations\n\n* Examine an example protocol in great detail and document any issues, and solutions (as in [bug 1432831](https://bugzilla.mozilla.org/show_bug.cgi?id=1432831))\n\n    * Use as a guide for looking at others\n\n    * HTTPChannel Init was suggested (non-trivial, complex)\n\n* List all IDPL protocols (~144 ipdl files)\n\n* Document all JS users of MessageManager\n\n# Prior work from Security Assurance Team on sandboxing (pauljt)\n\n[Metabug](https://bugzilla.mozilla.org/show_bug.cgi?id=sandbox-sa) \n\nThe key activities that we have done are:\n\n### IPDL auditing\n\nWe’ve been through IPDL in an ongoing manner, but most recently the approach we took was [here](https://docs.google.com/document/d/1YYWGQAqKzBpXPmBnhjAJuCtr8ADtxhz7mVlbAvWorcQ/edit#). Summary:\n\n* Audit message serialisation (param traits).  DONE.See also:  [Bug](https://bugzilla.mozilla.org/show_bug.cgi?id=1315840)  and [Spreadsheet](https://docs.google.com/spreadsheets/d/1sTgA4bOuV0j1bP7_Q_QKsfiNNLfdIg_1CCo4-iB_cfo/edit#gid=0)\n\n    * Some issues were found\n\n    * There were a number which were \"might be an issue\". The follow up [bug](https://bugzilla.mozilla.org/show_bug.cgi?id=1325670) was filed by DH but then he left the org.\n\n* Audit Recv Methods (look just at the recv function for bugs)\n\n    * This was an ongoing effort for Jhector before he left\n\n    * Bugs are filed under [https://bugzilla.mozilla.org/show_bug.cgi?id=1041862](https://bugzilla.mozilla.org/show_bug.cgi?id=1041862)\n\n    * The challenge here is that auditing this code is very time consuming and requires an understanding of the code/api in question\n\n* Deeper audit diving into where data from recv function goes\n\n    * We’ve only looked at this level in piecemeal approaches or when hunting for dangerous patterns\n\n    * We started trying to organise an [audit](https://docs.google.com/spreadsheets/d/17wvJPTfKto8abz7UD2NoPTebNYoV-dh60ejeCx84Vkw/edit#gid=1569995826) by contacting module owners but had to put this on hold due to the focus on Quantum\n\n## External IPDL Audit\n\nI also used some parental leave backfill to get a [timeboxed audit of IPDL](https://docs.google.com/document/d/1ad9JV6yeRcoqnS-vQR2h3A8avPDL7U4aydmzzHL7lhQ/edit) last year from a contractor (Stephen Fewer). It was only a month though, so very timeboxed.\n\n## Message manager audit\n\n* We’ve been through all the \"MessageManager\" messages a couple times now, and I’m pretty confident of our coverage here\n\n* One exception is nested protocols, e.g. Web Extension message passing\n\n* Detailed review notes are [here](https://docs.google.com/spreadsheets/d/1YnOFWatdnSBEvDKHLQV4DFngNuwC1Kkb2hZShV1cVx0/edit#gid=35305492)\n\n* Some findings tracked under [here in bug 1040184](https://bugzilla.mozilla.org/show_bug.cgi?id=1040184)\n\n**Other sandbox related auditing activities**\n\n* Audit places where were run remote content in parent process: [bug 1305531](https://bugzilla.mozilla.org/show_bug.cgi?id=1305331)\n\n* Attempt at automatic bounds checking (since this was the most common bug). [Bug 1325647](https://bugzilla.mozilla.org/show_bug.cgi?id=1325647)\n\n## Specific Protocols and IPC messages Audit (jesup)\n\nWe have ([currently](https://searchfox.org/mozilla-central/search?q=&case=false&regexp=false&path=*.ipdl)) about 144 separate non-test .ipdl files in the tree.  Looking in them for \"manager XXXXXX\":  (This search may be comparable to mine below, but I don’t trust it as much - [instances of manager in searchfox](https://searchfox.org/mozilla-central/search?q=manager%20&case=true&regexp=false&path=*.ipdl))\n\n* About 36 managed by PBackground*\n\n* About 9 managed by PBrowser\n\n* 1 PCache\n\n* 6 PClient*\n\n* 8 PCompositor*\n\n* About 23 managed by PContent\n\n* 5 PGMP*\n\n* 2 PImageBridge\n\n* 17 PNecko\n\n* 6 PPlugin*\n\n* 2 PPresentation\n\n* 2 PQuota\n\n* 1 PServiceWorkerManager\n\n* 1 PSpeechSynthesis\n\n* 1 PVideoDecoderManager\n\n* 1 PVRManager\n\n* 2 PWebBrowserPersistManager\n\nLooking at all the methods inside these ipdl files, there are roughly:\n\n* ~1284 async IPCs\n\n    * 129 of these as async __delete__(...)\n\n    * Perhaps ~15-18 are async DeleteMe/DeleteSelf/RequestDelete\n\n* ~124 sync IPCs\n\n* ~19 nested(inside_cpow) (sync and async)\n\n* ~146 nested(inside_sync) sync IPCs\n\n* 1 prio(input) async\n\n* ~10 prio(high) async\n\n…\n\n### Examination of example protocols:\n\n* PVsync\n\n    * Child of PBackground, used between Parent (Master) and Child (Content) processes\n\n    * Only arguments are the current Timestamp on a Notify, and the timestamp rate (which could change I believe), both sent Master->Child\n\n    * Exists for the entire life of the Content process; if Content doesn’t need VSync, it tells the Master to hold off on sending it.\n\n    * Security \n\n        * The only information exposed through this to the Content process is the VSync rate of the current monitor, and the exact timings of when the VSyncs happen (modulo latency in IPC).\n\n        * The exact timings might vaguely reveal some aspect of varying load on the Master process, maybe, and then only if the child can somehow time them extremely accurately.  I seriously doubt this could be used to convey any significant data even with cooperation of code on the Master side and full control of the Content process (owned).\n\n    * Recommendations:\n\n        * None\n\n* PUDPSocket\n\n    * Child of PNecko of PBackground\n\n        * PNecko for JS-visible PUDPSocket\n\n            * Was planned to move to PBackground also ([Bug 1167039](https://bugzilla.mozilla.org/show_bug.cgi?id=1167039)), but that was deprioritized with the end of B2G\n\n            * Likely PNecko support could be removed\n\n        * PBackground for WebRTC ICE network usage\n\n    * Originally designed to support B2G networked applications, such as Mail\n\n        * Web content was not allowed to use UDPSocket\n\n            * Principal used to check that udp-socket is allowed for that Principal\n\n        * All such uses have been removed with the end of B2G\n\n    * PBackground use was added to support WebRTC’s need to send/receive UDP data and negotiate port usage via ICE (IETF protocol for NAT traversal)\n\n        * WebRTC doesn’t pass up a Principal - Bug [1126232](https://bugzilla.mozilla.org/show_bug.cgi?id=1126232)\n\n            * Not critical since all web content is allowed to create RTCPeerConnections, which can be asked to initiate WebRTC with an arbitrary address, if (and only if) the target address responds to a STUN message which is used as a \"Consent\" indicator that the target is willing to exchange WebRTC data.\n\n            * No other data in the Master process is accessed via this protocol, only the network, so we don’t need the principal to gate it\n\n        * All traffic on the port is filtered to ensure we see a STUN response before allowing any other traffic\n\n            * Until we see a STUN response, outgoing traffic is only allowed if the packet is a STUN packet, and incoming traffic is blocked\n\n            * This is the security requirement of the ICE & WebRTC protocols\n\n    * All methods are async\n\n        * From Content to Master, there are a number of methods to set up sockets or modify them\n\n            * Bind(), Connect(), JoinMulticast(), LeaveMulticast(), Close\n\n            * These have a number of structures as parameters\n\n            * WebRTC doesn’t use the Multicast methods, but an owned Content process could send them\n\n        * The others are OutgoingData() (i.e. send()) and RequestDelete (asks the parent to send a __delete__ message to the child; avoids races)\n\n            * OutgoingData() takes a structure and a buffer\n\n        * From Master to Content, they’re all responses to Content IPC’s except CallbackReceivedData() - i.e. incoming data on the port\n\n    * Security\n\n        * The only extant use (WebRTC) doesn’t use a Principal\n\n            * This isn’t important since the it can’t be used to access data keyed by the Principal in the master, just network sockets\n\n            * Bug is filed (P4)\n\n        * Can send and receive data to arbitrary addresses\n\n            * STUN filter avoids content (even owned) being able to try to interact with websites or anything using a protocol not associated with realtime communication (e.g. STUN/ICE)\n\n                * Avoids using it to bypass other web controls or to try to DoS sites\n\n                * It can generate STUN packets, but STUN packets are rate-limited to avoid DoS-via-STUN\n\n                    * Rate-limitation is in Content, so could be bypassed\n\n                * An owned process could create a UDPSocket channel without passing a filter string (\"stun\"), but that will cause the parent to fail Init() - it requires all UDPSockets without a Principal to use a filter (and the only valid filter is ‘stun’)\n\n        * Other than the default IPC read checks (read a string, read a uint32, read a bool, etc), generally there’s no sanity checking of parameters\n\n            * However, when passed to system calls bad data *should* be rejected\n\n            * If there’s an underlying library/OS bug, passing arbitrary data to bind()/connect()/etc could trigger it\n\n            * Additional sanity-checking of parameters is warranted for belt-and-suspenders security against sandbox attacks\n\n            * OutgoingData is an array or an IPCStream (not used for WebRTC)\n\n                * It has an address field, but if a filter (stun) is in place, it isn’t allowed to be used.\n\n    * Recommendations\n\n        * Add validators for AddressInfo, etc per above\n\n            * Highest priority\n\n        * Remove PNecko and non-WebRTC support\n\n            * Unused and non-spec\n\n            * Lets us remove Multicast support\n\n            * Cons: disturbs working code, man-hours\n\n        * [don’t do] Add Principal to WebRTC\n\n            * Removing non-WebRTC support means we can stop using Principal as a flag -- simplifies code in UDPSocketParent\n\n            * Low priority due to no obvious advantage to having the principal\n\n            * Con: man-hours, coding risk - we don’t have access to the Principal down in mtransport; API changes would ripple through the stack to get access to it for no known current benefit\n\n* PTexture\n\n    * Child of PImageBridge or PCompositorBridge or PVideoBridge\n\n    * Virtual reference to a Compositor resource (a gfx texture), which allows dropping or recycling it\n\n        * Used as part of higher-level protocols\n\n    * No arguments\n\n    * No security issues\n\n    * No recommendations\n\n* PWebSocket\n\n    * Child of PNecko\n\n    * Supports implementing the DOM WebSocket API\n\n        * Since network resources must be used in the Master process, the Content process’s WebSocket impl is basically an interface to a WebSocket that lives within the Master\n\n        * WebSocket creation involves many arguments.  Once created, there’s two-way communication over the channel, and the ability to close it.\n\n        * All requests as async\n\n    * Security\n\n        * Methods after creation are simple - send data, or receive data, or close.\n\n            * Master can request client delete itself\n\n            * Only security concerns here are to ensure that any data sent does not extend outside the memory the script has access to, and to avoid UAFs (especially on close/deletion)\n\n        * Open() and OnStart() are where most of the arguments are\n\n            * Open() takes 12 arguments, some of which are structures, OnStart() has 3 strings and a bool\n\n            * None of these are a Principal\n\n                * There are both original and triggering principals on the LoadInfo structure, which is passed in LoadInfoArgs (optionally null)\n\n                * Since it can be null, when converted into a LoadInfo it will succeed and put a nullptr in the nsCOMPtr<>\n\n            * It includes an optional URI, an Origin (string), and an InnerWindowID\n\n                * These get passed on the an HTTP(S)Channel AsyncOpen(), since WebSocket is a protocol run over an HTTP(S) channel\n\n                * Origin is used as a header for the HTTP/HTTPS request, to inform the server what origin URI generated this WebSocket connection request; note that it’s suggested that servers not use this as a form of authentication.\n\n            * Use on HTTP is as insecure as any HTTP connection, and an attacker could use this to feed bad data or commands (or collect information from) a website that uses HTTP WebSockets, via MITM attacks\n\n            * Using WebSocket as a TCP tunnel (WebSocket to Server, Server connects it to a TCP connection to some internal or external resource) is dangerous (to the resource), since a compromised Content (or Master) process could craft attacks on that resource.\n\n    * Recommendations\n\n        * Document when aLoadInfoArgs can be null, what happens, and what it implies - what will succeed and what will fail.\n\n        * Modify LoadInfoArgsToLoadInfo() (in BackgroundUtils.cpp) to verify if the principals are valid for the requesting content process/document\n\n            * Probably also affects all the other AsyncOpen’s (HTTP, FTP, etc)\n\n        * Verify that the InnerWindowID is associated with the principals in to LoadInfo\n\n* PPresentation (and friends)\n\n    * Child of PContent (or PPresentationBuilder or PPresentationRequest)\n\n    * Supports the [Presentation API](https://developer.mozilla.org/en-US/docs/Web/API/Presentation_API), which allows remotely controlling a presentation using DataChannels or https/TCP\n\n        * [http://w3c.github.io/presentation-api/](http://w3c.github.io/presentation-api/)\n\n        * \"enable Web content to access presentation displays and use them for presenting Web content\"\n\n            * Originally implemented as part of B2G and perhaps also the TV project\n\n            * Apparently enabled in Chrome on Android since Chrome 48 (2015), and Chrome 51 (Desktop)\n\n        * Disabled by default (pref(\"dom.presentation.enabled\", false);)\n\n    * Security\n\n        * There seems to be little or no checking in the Parent of the parameters and that they’re internally consistent, especially the session ID, window ID, tab ID, etc.\n\n            * It’s unclear if this actually creates a security risk here\n\n        * Review [comment from smaug](https://bugzilla.mozilla.org/show_bug.cgi?id=1069230#c141) unaddressed by updates supposedly to address review -- requested the NS_CreatePresentationService() function check if the Presentation API is enabled/permitted\n\n        * Security review of this code has not been done: [Bug 1207051](https://bugzilla.mozilla.org/show_bug.cgi?id=1207051)\n\n    * Recommendations\n\n        * Find out the priority of completing and enabling this API in Firefox, either for Android or Desktop.\n\n            * If there’s no interest in enabling it anytime soon, consider if we want to remove it\n\n                * Removing it will make it harder to bring it back (code rot); the code exists, apparently works and has tests\n\n            * Alternatively ensure that the IPC methods are gated by the enable pref, so that a compromised Content process can’t try to leverage these IPC methods\n\n            * Since the API isn’t even enabled, it’s unlikely that the Content Process can provoke any meaningful problems (other than DoS) by sending Presentation API IPC methods to the Master process, but adding a check of the preference or ensuring these methods can’t do anything is a good safety move\n\n* PMedia\n\n    * Child of PContent\n\n    * Used to manage unique deviceid’s for media devices (via [enumerateDevices](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/enumerateDevices), which is speced in W3’s [Media Capture and Streams](https://w3c.github.io/mediacapture-main/#mediadevices) API)\n\n        * Backed by a file mapping id’s to devices on a per-origin basis\n\n            * Each origin has a different ID for the same device, to avoid them being supercookies.\n\n        * Mapping cleared when you clear cookies (per spec)\n\n        * Mapping is temporary (not saved) for private browsing\n\n        * Persistence gated by user permission to use a device\n\n    * Security\n\n        * Can cause Parent to access disk (flat file encoding deviceID/origin triples (id, time, and origin)\n\n            * File lives in profile\n\n        * A compromised Content process could probably look up the device ids for any site, instead of just the current site (if it knows or can guess what sites have called enumerateDevices())\n\n            * Just a privacy leakage in this case (supercookie), which is mostly irrelevant given then control a content process\n\n            * Could be blocked if we have Site Isolation\n\n    * Recommendations\n\n        * Once we have Site Isolation, verify the PrincipalInfo\n\n        * Ensure that the Sync & Storage team knows about this file\n\n* PDataChannel\n\n    * Child of PNecko\n\n    * Involved in Necko redirection to data: URIs\n\n        * Necko code assumes that there’s an IPC channel associated with the redirect\n\n        * The parent class wrapping the IPC channel largely does nothing\n\n    * Security\n\n        * There are no methods other than __delete__\n\n    * Recommendations\n\n        * None, other than possible future Necko rework for isolating parts of Necko might allow us to get rid of this required wart\n\n* PColorPicker\n\n    * Child of PBrowser\n\n    * Requests a Color Picker be displayed by the Master (parent); implementing <input type=color>\n\n    * Security\n\n        * Uses the IPC Manager (PBrowser) to get the owner element, and thus the Outer Window\n\n        * No parameters from the Content Process; one method that passes a picked color from the Master Process\n\n    * Recommendations\n\n        * None\n\n* PRenderFrame\n\n    * Child of PBrowser\n\n    * Represents one web \"page\"; it's used to graft content processes' layer trees into chrome's rendering path.\n\n        * Only one direct method to trigger a repaint by the Master; no parameters\n\n        * Used to connect Child layout Frames to the Parent\n\n    * Security\n\n        * Nothing direct to this protocol; the connection is passed around by a number of other parts of TabParent/Child\n\n        * Primary security risk would be lifetime management of the PRenderFrameParent/Child objects\n\n        * Per the ipdl file, the PRenderFrame is conceptually owned by the Content Process, and its lifetime is tied to the PresShell in the Content Process.\n\n    * Recommendations:\n\n        * Audit/document the lifetime management (low-ish priority)\n\n* PDNSRequest\n\n    * Child of PNecko\n\n    * Implements DNS requests\n\n        * Most parameters are in the construction request\n\n            * Host, attributes, network interface\n\n        * Passed again to cancel\n\n        * Results are returned asynchronously once complete\n\n    * Security\n\n        * Host and Network Interface are strings\n\n        * None of the parameters are checked for validity on reception\n\n        * Relies on existing validity checks in nsDNSService::AsyncResolveExtendedNative() and functions it calls, like PreprocessHostname()\n\n        * Probably these are well-checked in general.  They are (if internal checks are passed) sent to the OS\n\n        * The ability to specify a network interface might be a subtle risk, especially if we’re trying to route traffic through a VPN interface or proxy.\n\n    * Recommendations\n\n        * Consider validating parameters, or documenting where they’re validated\n\n* PDocumentRenderer\n\n    * Child of PBrowser\n\n    * Implements rendering of a Canvas in the Content Process, and returns the rendered data\n\n    * Security\n\n        * Creation is Master->Content - no issues\n\n        * When rendered, the IPC protocol object is destroyed with the data being returned in the __delete__() method - size and a raw buffer as an nsCString (though it isn’t really a null-terminated string\n\n            * This is concerning, since it opens the door to mishandling the data\n\n            * Depends on DataSourceSurface to avoid going past the real end of the data from the canvas, but nothing checks this\n\n            * Likely information-leakage bug possible if a crafted attack message is received in __delete__() (csectype-bounds)\n\n    * Recommendations\n\n        * Validate that the IntSize(aSize.width, aSize.height) is not larger than the length of the aData nsCString\n\n        * As a result of this review and evaluating a fix to implement the above, it was determined that PDocumentRenderer was unused, and it was instead removed.\n\n* PGamepadEventChannel\n\n    * Child of PBackground\n\n    * Used to notify the Content process (and gamepad) of Update events\n\n        * Controllers added/removed\n\n        * Buttons pressed\n\n        * Joysticks moved\n\n    * Used by Content to tell the gamepad to vibrate (haptic feedback)\n\n        * Parameters are largely numbers (length, intensity, etc) and which controller\n\n        * Master Process appear to not actually implement haptic feedback ([bug 680289](https://bugzilla.mozilla.org/show_bug.cgi?id=680289))\n\n* Security\n\n    * Parameters for Haptic feedback aren’t checked\n\n        * However, they’re not used currently\n\n* Recommendations\n\n    * Add validation of Haptic feedback parameters (even if we’re not currently implementing it), or add a comment stating it’s needed before implementing it.\n\n        * Especially check that the index to the controller is valid\n\n        * There may be an very unlikely information leakage channel if Haptic feedback can be sensed by another Content process (causes joystick \"motion\", audible to an active mic, etc); however, exfiltrating data is far simpler in other manners, so I see almost no real risk here\n\n"
  },
  {
    "path": "text/0014-xul-overlay-removal-review-packet.md",
    "content": "---\ntitle: XUL Overlay Removal Review Packet\nlayout: text\n---\n\n# XUL Overlay Removal Review Packet\n\n## Summary\n\n### Problem Space\n\nXUL overlays provide a way to customize the UI and share code across different XUL documents. In addition to sharing many of the common issues already outlined about XUL, overlays were primarily used by legacy add-ons and are not supported by new web extensions. Within mozilla-central, there are still some uses of overlays, but the majority of them could be simplified and removed or use a more standard “web” like approach.\n\n### End State\n\nAll of the C++ code that supports the complex logic of loading and merging overlays would be removed. Firefox developers would no longer need to learn an obscure Mozilla only technology. Startup performance may see a slight improvement since we would be doing the work at compile time instead of runtime.\n\n### Stakeholders\n\nXUL Maintainers (Neil Deakin, Gijs Kruitbosch), Firefox Frontend (Dave Townsend)\n\n## Brief\nA detailed review of XUL overlay usage shows some common patterns and gives insight into how they can be removed. In general, the uses fall into four categories: unused, used once, used multiple times for simple templating, and used multiple times in a more complicated manner.\n\n### Unused\n\nThese can just be removed.\n\n### Used Once\n\nThese can simply be inlined into their master documents.\n\n### Simple Templating\n\nThese can be included via the preprocessor.\n\n### Advanced Uses\n\nThese are more tricky and will have to be replaced case by case. A few possible approaches:\n\n* Preprocessor include - break up the overlay into a few include files. This has the downside that the master document would need to do a number of includes to get all the various files (dtd, css, js, xml).\n* Custom element - use the new custom elements. The master document would need to includes a JS file that defines the element.\n* JS - move element creation to JS\n"
  },
  {
    "path": "text/0015-rkv.md",
    "content": "---\ntitle: Roadmap Review: Key-Value Storage (rkv)\nlayout: text\n---\n\n# Design Review: Key-Value Storage\n\n**We propose the standardization of a simple key-value storage capability, based on LMDB, that is fast, compact, multi-process-capable, and equally usable from JS, Java, Rust, Swift, and C++.**\n\n## Document status\n\n* 2018-03-20: review concluded; see [notes here](https://docs.google.com/document/d/1bwbpqPb58a0GcEyB4W424pyftPFiZBSvxxp_0uuN-z0/edit).\n* 2018-03-05: wrapped up a concrete proposal.\n* 2018-02-22: lots of input from gcp. Added section about content process use. Added analysis of SQLite.\n* 2018-02-21: positive feedback from mak; no document comments yet.\n* 2018-02-13: input from the Firefox team, inc. Florian and Felipe.\n* 2018-02-12: restructured open questions, which was becoming huge thanks to answers!\n* 2018-02-09: added extensive notes around NFS shares and locking (thanks Mossop and mkaply). Discussion with Mossop re management of change and versioning.\n* 2018-02-07: consulted njn.\n* 2018-02-06: suggestions from nanj and snorp applied.\n* 2018-02-04: suggestions from Myk applied. Appendix A added. Sent to snorp and nanj.\n* 2018-02-02: rough draft\n\n## Roles\n* **Reviewer**: dtownsend\n* **Chair**: jwalker\n* **Proposer**: rnewman, Browser Architecture\n\n## Summary\n\nThere is no one-size-fits-all solution for storage.\n\nThe best we can do to minimize engineering complexity is to cluster the varied requirements of various consumers around particular points in the solution space, reducing our large number of unmaintained ad hoc solutions into a smaller number of supported pieces of infrastructure.\n\nThe exploration effort that's now underway as a result of 2017's Sync & Storage Roadmap Review primarily addresses one such point: the capabilities required for structured, evolving, reusable, long-lived, syncable user data &mdash; data around which Firefox user-facing features are built.\n\nThe subject of this document is the polar opposite of that point: data that is unstructured or primitive, non-evolving (or at least with no advances on the current state of the art for managing change), special-purpose, and with no particular storage-level effort expended on reliable sync, if it syncs at all.\n\nComponents that store this kind of data often have a different set of required capabilities: a need for very fast reads, reads and writes from multiple processes, synchronous reads, control over filesystem impact, *etc*.\n\nExamples of data like this are configuration data, certificate revocation lists, session storage, visited link caches, and simple extension storage.\n\nThe existing implementations of these features, if they exist, tend to use what's available &mdash; flat text files, JSON files, Kinto, IndexedDB, libpref, or non-persisted in-process data structures repopulated via IPC or messaging<sup>[1](#footnotemessaging)</sup>. In doing so they tend to work around missing capabilities in, and necessary limitations of, those tools. Some of these tools are hard to access from different languages, and most are next to impossible to use correctly in an embedding context<sup>[2](#footnoteembedding)</sup>.\n\n## Brief\n\nWe propose 'buying', not building, the core of such a solution, and wrapping it in idiomatic libraries that we can use on all platforms.\n\nWe propose that [LMDB](https://symas.com/lmdb/) is a suitable core (see Appendix A for options considered): it is compact (32KB of object code), well-tested, professionally maintained, reliable, portable, scales well, and [is exceptionally fast for our kinds of load](http://www.lmdb.tech/bench/inmem/). We have engineers at Mozilla with prior experience with LMDB, and their feedback is entirely positive.\n\nThis key-value storage system can be targeted at a number of new consumers that might otherwise need to roll their own storage, or (ab)use preferences:\n\n- HSTS etc. download storage. (Waiting on survey from mgoodwin.)\n- Devtools UI configuration.\n- A partner-acceptable data storage and configuration approach for GeckoView and other embedding projects. (More detailed conversations need to happen around this.)\n- New mobile applications.\n\nand potentially displace or support a number of existing solutions:\n\n- The atypical \"application\" consumers of preferences. These consumers don't use defaults, and don't use prefs' overlaying capabilities, but do want a simple API, fast (possibly synchronous) reads, ACID, and richer types (*e.g.*, non-stringified timestamps). They currently use libpref because it was the most suitable option at the time they were built.\n- Cross-process preferences usage, which currently [uses command-line arguments and IPC to replicate data between processes](https://bugzilla.mozilla.org/show_bug.cgi?id=1436911). Note that doing so would require filesystem access from each process, which might make it unsuitable for the content process; see below.\n- Limited-size flat file systems like autofill, login manager, *etc.*\n- [XUL Storage](https://docs.google.com/document/d/1z0l9ZY-4GTX69qyNJ5Y9VeVuYrxFS6AoHQ1IXsbnB0c)\n- [Session store](https://bugzilla.mozilla.org/show_bug.cgi?id=1304389)?\n\nIt should provide a sensible, scalable, and efficient default choice for consumers that now use hand-rolled key-value systems.\n\nThe core wrapping library provides safety (both padding LMDB's sharp-edged API and also protecting against system limitations like NFS locking), improves developer ergonomics (*e.g.*, tracking value types, seamlessly resizing databases, even automatic sharding), and consolidates best practices (*e.g.*, asynchronous reads and writes, versioning).\n\nIdiomatic per-environment APIs can then take care of platform integration on top of a shared representation, managing things like string conversions, `Future`/`Promise`/`Deferred`, *etc*.\n\nWe expect the resulting APIs to be simpler (and more opinionated) than LMDB's, and for ongoing maintenance of language wrappers to be relatively cheap. We have recent and relevant experience with Rust wrappers for C++, Swift, and Java, so we're not too worried about this.\n\nExplicit **non-goals** for this proposal are:\n\n- Displacing SQLite for relational consumers, consumers that require full-text search, or very large databases.\n- Implementing synchronization within the data store itself.\n    - It's likely that some consumers will replicate data in, and it's possible that some consumers will build syncing on top, just as we do for prefs today, but &mdash; unlike with our other efforts &mdash; we're not trying here to make syncing a supported part of the storage system itself.\n- Implementing the full set of capabilities of libpref (*e.g.*, distribution configuration) within the storage system itself<sup>[3](#footnotelibpref)</sup>.\n- Addressing storage of data that must be human-editable at rest without specialized tooling.\n\n**Not-yet or never** goals for this proposal are:\n\n- Standardization via a standards body as a web API.\n- Encryption (though there might be touch points with Lockbox's secure device storage concept).\n\n## Content process use\n\n*Particular thanks to Randell and gcp for this input*.\n\nOne appealing aspect of LMDB is its relative ease of use from multiple processes, above and beyond its basic capabilities as yet-another-fast-key-value-store.\n\nWe expect to have multiple non-content processes in Firefox that could use this capability.\n\nWe also expect to have lots &mdash; potentially dozens or hundreds &mdash; of content processes. Naturally it's not desirable from a security perspective to allow arbitrary reads and writes, and arbitrary filesystem access, from content processes. There is a tension between performance, reducing memory usage, *etc*. and reduced attack surface &mdash; we can have a locked-down content process, or efficient sharing, but not both.\n\nLMDB uses mmap, which requires some amount of filesystem access. It requires write access to the database lock file for coordination. In ordinary use, the memory-mapped region is read-only, with only the lock file requiring writes.\n\nThere are several possibilities for use from the content process.\n\n- Full read-write via mmap. This would require whitelisting and careful auditing, and is still a concern.\n- Read-only access via mmap, either by whitelisting the lock file for writes, or building our own non-filesystem locking (*e.g.*, using shmem). Writes from content would have to be asynchronous over IPC. It's possible that we'll build our own lock manager anyway in order to achieve robustness in the face of NFS mounts, so this might be cheap and relatively safe when considering amortized costs.\n- Read-only in-memory use by modifying LMDB to be able to use anonymous mmap. In theory we can then flush to disk for persistence. This would essentially be a standardized atomic key-value interface on top of shared memory.\n- Don't use LMDB from the content process: use IPC to access storage in another process. This is what we do now, if you ignore the storage technology on the other end.\n\nWe do not propose solving this up-front: the worst-case scenario is no different to what we do now, but with the benefit of easier coordination between non-content processes, and it's possible to make different choices in the future.\n\n## Open and partially answered questions\n\n#### 32-bit systems\n\n32-bit systems have limited address space for memory-mapped files. Will this restriction bite us? How large can we make data stores without running into these issues?\n\nLMDB files' maximum size is specified at opening time. Address space usage is dictated by this maximum size.\n\n64-bit users make up *at least* 80% of our hardware report (we under-count). On Android we [don't yet ship ARM 64 ](https://bugzilla.mozilla.org/show_bug.cgi?id=1368484), but intend to do so this year.\n\nPreliminary survey of crashstats suggests that we'll be totally fine for reasonable (< 1MB or < 256KB) database volumes.\n\n> oom-small is about 4% of our crash rate. Most of those are genuine memory pressure: 2GB or 4GB devices with 75%+ memory usage. Quite a few of the reports are nonsense &mdash; failure to allocate 8 bytes with gigs of free RAM and VM.\n>\n> If I search for oom, <4GB total virtual memory, memory usage <80%, and < 30 seconds uptime &mdash; that is, a startup OOM where we shouldn't actually have run out of memory &mdash; we get about 350 oom-small crashes per week across Firefox 58-60. 10% of those are Android, and the rest are Windows.\n>\n> At those rates, typical LMDB usage should be lost in the noise. It might even *reduce* OOMs: the entire mmapped file occupies address space, but the actual resident set will be less than snarfing an entire JSON file into memory and causing an oom-large in the JS engine or in file-writers like `ChunkedJSONWriteFunc`, both of which appear in crash-stats.\n\n#### Windows support\n\nHow well does LMDB work on Windows? About 85% of our desktop users are on Windows.\n\nReading suggests that as of 2016, LMDB should work just fine, even using sparse files correctly, but we should experimentally verify.\n\n#### Android support\n\nHow well does LMDB work on our supported Android versions?\n\n#### Remote filesystem support\n\nLMDB's documentation recommends not using it on network filesystems<sup>[4](#footnotenfs)</sup>. The principal obstacle is the lack of advisory locking.\n\nWe already have some accommodations for SQLite on NFS. It's possible that we could apply the same. Depending on exactly which issues occur &mdash; locking, syncing, or other &mdash; we might need to find a variety of approaches; the ultimate workaround is to put a 'working copy' of each LMDB file in temporary storage, and use the atomic snapshot feature to copy them back to the profile in the background at some point before crash/quit. Some notes:\n\n- [https://bugzilla.mozilla.org/show_bug.cgi?id=433129](https://bugzilla.mozilla.org/show_bug.cgi?id=433129)\n- [https://github.com/browserify/watchify/pull/170](https://github.com/browserify/watchify/pull/170)\n- [https://github.com/paulmillr/chokidar/issues/242](https://github.com/paulmillr/chokidar/issues/242)\n- storage.nfs_filesystem: [https://searchfox.org/mozilla-central/source/storage/TelemetryVFS.cpp#23](https://searchfox.org/mozilla-central/source/storage/TelemetryVFS.cpp#23)\n- [http://www.postfix.org/lmdb_table.5.html](http://www.postfix.org/lmdb_table.5.html)\n- [https://github.com/LMDB/lmdb/blob/3daab3be733b933a3fd2263441da4976206b27a2/libraries/liblmdb/mdb.c#L2948](https://github.com/LMDB/lmdb/blob/3daab3be733b933a3fd2263441da4976206b27a2/libraries/liblmdb/mdb.c#L2948)\n\n#### Being careful\n\nLMDB has a number of straightforward restrictions: no multiple opens from a single process, no multiple top-level transactions open at the same time within a single thread<sup>[5](#footnotenfs)</sup>, careful handling of locks when processes crash, the need to clean up stale readers if they crash. Will these restrictions be onerous? Current work suggests that a layer or two of Rust helps a lot with the sharp edges.\n\n#### Defaults\n\nWhat kinds of file sizes and defaults give reasonable behavior on our supported platforms? How changeable are these later? LMDB requires some amount of tuning and specifying limits (*e.g.*, maximum number of open named databases in an environment).\n\n#### Binary storage\n\nHow can we protect ourselves against variations in platform/runtime encoding issues? Endianness, string encoding, Rust repr&hellip;? Initial exploratory work used bincode for this, but we need to validate.\n\n#### Resilience to corruption\n\nWhat level of external corruption (*e.g.*, disk issues) or internal corruption (*e.g.*, writing malformed data) do we need to handle? We are very careful about things like session storage (*R: how careful? Do we handle a failure to parse due to corrupt JSON? Do we handle a failure to parse the compressed file due to corruption?)*, but many of our existing stores either fail to init, breaking the browser, or discard the corrupt file. Do we need to do better? Is LMDB more or less prone to issues?\n\nAre customers who need recoverability from corruption adequately served by other points in the solution space, or by features of this one (*e.g.*, LMDB's ability to take consistent live backups)?\n\nLMDB's live backup seems like it could be used to build a sensible backup/migration strategy, but it would be good if we could understand to what extent we consider the requirement important before investing effort into building capabilities. Even if we have some potential consumers, maybe those aren't the first ones.\n\n#### Observers/change notifications\n\nWhat approaches to observers/notifications make sense? We don't get this out of the box for most stores, particularly not cross-process; we're breaking new ground. We can learn some lessons from observer and preference change notifications in Firefox.\n\n#### Performance\n\nDoes a memory-mapped solution give us:\n\n- Acceptable startup time compared to prefs or flat files?\n- Acceptable shutdown time? (It should: LMDB is crash-safe.)\n- A reduction in the number of fsyncs compared to, *e.g.*, `JSONFile`?\n\nCan LMDB beat IPC-coordinated shmem or flat file solutions for for inter-process data sharing?\n\n#### Shipping vehicle(s)\n\nIt makes sense to have at least one target consumer in mind when building out a capability. We have lots of options; which ones will have the biggest potential upside, the lowest risk, and offer the best learning opportunities?\n\n## Cross-process, cross-language use examples\n\n#### How a Java app and GeckoView might collaborate on the visited link set or configuration data\n\nChrome shares its visited link set between the main process and content process via a complicated shared memory scheme: the main process constructs chunks and coordinates handoff to content processes via IPC.\n\nIn Firefox for Android we use JNI to allow [the Gecko main thread to read from and add to](https://dxr.mozilla.org/mozilla-central/source/mobile/android/components/build/nsAndroidHistory.cpp#79) a [Java-side data structure](https://dxr.mozilla.org/mozilla-central/source/mobile/android/base/java/org/mozilla/gecko/GlobalHistory.java#153) that serves a similar purpose.\n\nGeckoView will, eventually, distinguish between the application process (started by Android, runs the enclosing Java application) and Gecko's main process. These two processes will communicate via [Binder](https://source.android.com/devices/architecture/hidl/binder-ipc).\n\nIn contrast to Fennec, application data &mdash; including history data &mdash; won't be in the same process as the Gecko main process. Sharing persistent data in this situation is well-suited to LMDB: the same LMDB database could be opened from both the application (Java) process and the Gecko main process. The two processes would have a shared understanding of the data format used and the file path and settings required. Given those things, the application could add and remove entries from the database, and the Gecko main process could read (and, indeed, write) without additional coordination.\n\n## Planning\n\nExecuting on this proposal consists of five overlapping phases.\n\n1. Evaluation. Given the set of open questions, research, measure (either through prototypes or through probes in Firefox itself), or write code in order to answer the question. In particular, the following:\n    1. Windows support.\n    2. Android support.\n    3. NFS.\n2. Design. It's likely that some kind of profile-linked manager will be needed to control access to storage. An API needs to be defined and documented. Design and evaluation are intertwined.\n3. Productization. Produce and document:\n    1. A core library (using the stable Rust wrapper for safety and ergonomics).\n    2. Build integration with mozilla-central and Gecko, and an idiomatic interface that can be consumed by C++ code.\n    3. &hellip; and chrome JS code.\n    4. A Swift wrapper around our core Rust library.\n    5. A Java library that allows for independent use of that library, and shared data use with Gecko.\n    6. Baseline performance tests to make sure we stay fast.\n4. Leverage. Use these libraries in consumers, migrating older systems as appropriate. We propose the following as the initial set of consumers, depending on staffing:\n    1. XULStore (existing).\n        - Addresses a concern raised by the XUL replacement work. Replaces a JS XPCOM component that's in the hot path for window creation. Stores a manageable amount of data.\n    2. Search service cache (`search.json.lz4`).\n        - Binary data storage: opportunity for size wins. The component currently compresses to halve disk usage&hellip; but 80% of that space usage is base64 and percent-encoded image data!\n        - Read asynchronously as the first thing the component does, so a good perf challenge.\n        - It's a cache: no need to address migration, and rollback is easy.\n        - Opportunity to rework how the component manages state (*e.g*., no need to read the descriptions of each engine during init).\n    3. Devtools configuration (new).\n        - Meets an unmet need, and involves figuring out content process communication.\n    4. Security storage (new/replacement).\n        - We currently store some certificate data (revocations, whitelists, HSTS, etc.) in a mixture of Kinto (for delivery) and plain-text files.\n        - This data needs a mixture of synchronous and asynchronous access.\n        - Infrequently updated. Can be repopulated from the server if necessary.\n        - Blob storage.\n        - Two interesting needs: ideally this data is *shared* between profiles (no sense in having it copied), and usable on Android, iOS, and from C++, Rust (soon), and JS in Gecko.\n    5. A new Android capability to support new Fennec and GeckoView work.\n        - Exercises another platform, and explores how the system meets the needs of a diverse set of developers.\n5. Analysis of other consumers, existing and nascent. There are dozens of stores in Firefox on each platform. Some of these will be better suited to this kind of key-value store than their existing storage system (and some will be better suited to other capabilities that we're building). Still others will not be worth the effort to port. Assess each consumer along axes: size, complexity, risk, technical debt, cross-platform costs and potential value. This will inform subsequent porting efforts.\n\nWe have begun building a [simple production-quality Rust wrapper](https://github.com/mozilla-prototypes/rkv) to implement storage of typed values and to begin to impose an opinionated and safe interface. This can be used for evaluation and as a testbed for design, ultimately being stabilized and productized.\n\n# Footnotes\n* <a name=\"footnotemessaging\">1</a>: In this set I include the preferences that we currently pass to content processes via command-line arguments (\"early prefs\") and IPC messages (\"late prefs\").\n* <a name=\"footnoteembedding\">2</a>: For example, Firefox for Android still uses the SQLite login manager storage, because it needs concurrent access from Java and Gecko; flat files don't make that easy.\n* <a name=\"footnotelibpref\">3</a>: It's possible that a key-value storage system could be used as the backend for libpref, just as we currently use prefs.js. Randell reports that each content process uses approximately 350KB to keep copies of preferences; that's likely to be reduced in a shared memory system.\n* <a name=\"footnotenfs\">4</a>: \"Do not use LMDB databases on remote filesystems, even between processes on the same host. This breaks flock() on some OSes, possibly memory map sync, and certainly sync between programs on different hosts.\" — [http://www.lmdb.tech/doc/](http://www.lmdb.tech/doc/)\n* <a name=\"footnotenotls\">5</a>: Unless `MDB_NOTLS` is specified, tying transactions to the transaction object instead of to the thread. 'Defaults' and 'Being careful'!\n\n# Appendix A: survey of key-value stores\n\n### IndexedDB\n\nA web-standard storage abstraction layer. In Chrome, uses LevelDB for storage. In Firefox, currently backed by SQLite. Both use remoting for multi-process access. Only accessible from JS.\n\n### SQLite\n\nWell-tested and already ships in Firefox. Crypto available via SQLCipher. In-memory (non-persistent) storage is easy. Built-in `VACUUM` is more convenient than compaction in LMDB. Disk sizes are approximately equivalent to LMDB. Known good behavior on 32-bit platforms.\n\nWAL versus COW &mdash; writes, and database opens after crashes, are less predictable due to WAL checkpointing. Reads likely to be on the order of microseconds, not nanoseconds; not suitable to back a synchronous API. Reader concurrency not quite as good as LMDB. SQLite will have a higher baseline of memory usage: real allocated memory per connection, not mapped virtual memory.\n\n[Benchmarking](http://www.lmdb.tech/bench/microbench/) shows SQLite to perform relatively poorly on KV workloads when compared to LMDB. In sequential reads on published benchmarks LMDB has 47x throughput (I measured 80x for string keys). In random reads LMDB has 9x throughput. Sequential writes, 8x; random writes 5.7x. Batching makes writes significantly faster for LMDB thanks to amortized cost of COW, and reading sequential keys is very fast.\n\nUsing SQLite instead of a specialized key-value store would trade performance (particularly read performance) for existing deployment experience.\n\nSQLite is the only real contender for delivering this capability in Firefox. It is conceivable that we could use the same opinionated interface to front *both* SQLite and LMDB. There is negligible increase in object size to ship both, because LMDB is so small.\n\nMy own runs of benchmarks, adjusting the SQLite schema and using WAL, with key times highlighted in bold:\n\nString keys:\n\n```\nSQLite:\n\nreadrandom : 8.110 micros/op 123310 ops/sec; (1000000 of 1000000 found)\n**readseq : 2.586 micros/op 386687 ops/sec; 36.9 MB/s**\nreadreverse : 2.580 micros/op 387580 ops/sec; 37.0 MB/s\n\nfillrandsync : 82.125 micros/op 12176 ops/sec; 1.3 MB/s (1000 ops)\nfillrandom : 24.933 micros/op 40107 ops/sec; 4.4 MB/s\n**fillrandbatch : 16.004 micros/op 62483 ops/sec; 6.9 MB/s**\nfillseqsync : 31.004 micros/op 32253 ops/sec; 3.6 MB/s (1000 ops)\nfillseq : 12.433 micros/op 80430 ops/sec; 8.9 MB/s\nfillseqbatch : 2.108 micros/op 474321 ops/sec; 52.5 MB/s\noverwrite : 36.793 micros/op 27179 ops/sec; 3.0 MB/s\n\nLMDB:\n\nreadrandom : 1.070 micros/op 934143 ops/sec; (1000000 of 1000000 found)\n**readseq : 0.032 micros/op 31309684 ops/sec; 3463.7 MB/s**\nreadreverse : 0.023 micros/op 42877969 ops/sec; 4743.4 MB/s\n\nfillrandsync : 166.412 micros/op 6009 ops/sec; 0.7 MB/s (1000 ops)\nfillrandom : 5.176 micros/op 193206 ops/sec; 21.4 MB/s\n```\n\nInteger keys (faster reads, slightly smaller file):\n\n```\nSQLite:\n\nreadrandom : 4.258 micros/op 234843 ops/sec; (1000000 of 1000000 found)\nreadseq : 0.281 micros/op 3560505 ops/sec; 339.6 MB/s\nreadreverse : 0.261 micros/op 3826081 ops/sec; 364.9 MB/s\n\nLMDB:\n\nreadrandom : 0.548 micros/op 1825317 ops/sec; (1000000 of 1000000 found)\nreadseq : 0.028 micros/op 35937612 ops/sec; 3701.5 MB/s\nreadreverse : 0.019 micros/op 53358945 ops/sec; 5495.8 MB/s\n\nfillseqsync : 165.802 micros/op 6031 ops/sec; 0.7 MB/s (1000 ops)\nfillseq : 3.360 micros/op 297576 ops/sec; 32.9 MB/s\nfillseqbatch : 0.504 micros/op 1983717 ops/sec; 219.5 MB/s\noverwrite : 5.099 micros/op 196105 ops/sec; 21.7 MB/s\n```\n\n### LevelDB\n\nGoogle-sourced. Implemented in Go. Not lightweight (306KB macOS release dylib). Non-ACID, no transactions: only read-snapshots and atomic batch writes. Poor reputation for data loss and [consistency bugs](https://github.com/google/leveldb/issues/320). Writes are automatically compressed, spending CPU to get reasonable writes. Coarse-grained locking. LSM trees are write-optimized. **Not multiprocess**. [Windows support](https://github.com/google/leveldb/tree/windows) is [second-class](https://github.com/google/leveldb/issues/519), [and iOS regressions occur](https://github.com/google/leveldb/issues?utf8=%E2%9C%93&amp;q=is%3Aissue+is%3Aopen+ios).\n\n### LSM tree LevelDB-alikes derivatives (RocksDB (Facebook), HyperLevelDB, Basho's LevelDB fork etc.)\n\nDgraph's [summary](https://blog.dgraph.io/post/badger-lmdb-boltdb/):\n\n> LSM tree based stores provide high write performance by sequentially writing key-value pairs in memory in the foreground, and then arranging them in multi-tiered levels on disk in the background. This approach is not without tradeoffs though. Values are written multiple times when arranging them in the tree (known as write amplification) and a single read might need to read multiple levels in the LSM tree before finding a value (known as read amplification).\n\nThese various libraries all aim to be faster or more scalable than LevelDB, typically by improving thread parallelism or altering how compaction works (*e.g.*, RocksDB [doesn't use level-based compaction](https://news.ycombinator.com/item?id=6737905)). All are targeted at server workloads.\n\n### Bolt\n\n[A Go implementation of LMDB](https://github.com/boltdb/bolt). B+tree. **Not multiprocess**.\n\n### MDBM\n\nA memory-mapped hash/cache (it would be a stretch to call it persistent) implemented in C, with C++ and Perl bindings. Loses data by default (manual sync), not resistant to power loss/crash, vulnerable to data corruption. No transactions. Keys aren't sorted, so no range queries or cursors. Building your own persistent storage/log is recommended.\n\n[https://github.com/yahoo/mdbm](https://github.com/yahoo/mdbm)\n\n[https://news.ycombinator.com/item?id=8732891](https://news.ycombinator.com/item?id=8732891)\n\n### Kyoto Cabinet\n\nReportedly very slow. Not multiprocess.\n\n### BerkeleyDB\n\nVery long pedigree. For our purposes, essentially obsoleted in every way by LMDB.\n\n### LMDB\n\nA distillation of experience with BerkeleyDB and use in OpenLDAP. Implemented in C.\n\nDgraph gives an excellent brief [summary](https://blog.dgraph.io/post/badger-lmdb-boltdb/):\n\n> LMDB provides a key-value stored using [B+ trees](https://en.wikipedia.org/wiki/B%2B_tree). It has ACID semantics and support for transactions. It does not do any background work and has a crash resilient design that uses [MVCC](https://en.wikipedia.org/wiki/Multiversion_concurrency_control) instead of locking. This means readers operate on an isolated snapshot of the database and don&rsquo;t block. The codebase is quite small and portable, so LMDB runs across multiple platforms including Android, BSD, and Solaris. This [talk by Howard Chu](https://www.youtube.com/watch?v=tEa5sAh-kVk&amp;t=10s) at Databaseology 2015 goes into much more details about LMDB&rsquo;s design.\n\nIt is generally understood that B+ tree based stores are ideal for workloads which are read intensive.\n\nRead-optimized, very lightweight (32KB), predictable performance. Copy-on-write mmap B+tree. **Zero-copy reads**. Handles larger values better than log-based approaches, with less write amplification. Database sizes are limited by address space (~128TB on 64-bit machines). Single-writer multiple-reader (linear scaling in readers, and total throughput reportedly better with serialized writes).\n\n*nanj's experience*:\n\n> Some less well-known merits of LMDB:\n>\n> - You can get a quite good performance out of the box, without any fuss or configuration like RocksDB.\n> - Its cursor iterator is awesome, very powerful and flexible.\n> - It also supports integer key and value, dup keys, and whole bunch of other nice value types.\n> - You can have multiple sub-dbs in a single database, each of them could be of different type (integer key, dup key etc.)\n> - We&rsquo;ve never run into a single case of database corruption before, even for those traffic (both write and read) intensive applications. [See more in this OSDI paper: Torturing Databases for Fun and Profit](https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-zheng_mai.pdf)\n>\n> Some lessons we&rsquo;ve learnt:\n>\n> - You will need to choose a maximum database size upon its creation. Since LMDB uses MVCC and shadow paging to implement ACID, long running read transactions would increase the page usage quickly, potentially fill up the mmap, and the write transaction will abort and leave the database in an unwritable state until page reclaiming is performed. Resizing the database on the fly is feasible, although it needs certain amount of cooperation between readers and writers.\n> - LMDB by default has a maximum key size set as 512 bytes, there is a compile time configuration to change it, though a larger key may have performance impact too.\n> - Random write is just OK, unlike other log-based data stores.\n> - Lack of built-in compression usually leads to a higher disk usage.\n> - Fully relying on OS&rsquo;s page management may block the way of certain optimization.\n> - Some database maintenances are still needed, LMDB provides various APIs (e.g mdb_reader_check) to clean up zombie readers (due to the improper transaction termination) so that the unused pages could be reclaimed. No downtime needed though, could be done in the background.\n> - The code base of LMDB is not very readable ;), at least comparing to SQLite.\n> - Howard Chu usually is quite responsive on bugs and feature requests, also he is a quite opinionated guy, you can see his comments on HN :)\n>\n> [https://banksco.de/p/lmdb-the-leveldb-killer.html](https://banksco.de/p/lmdb-the-leveldb-killer.html)\n>\n> [https://symas.com/is-lmdb-a-leveldb-killer/](https://symas.com/is-lmdb-a-leveldb-killer/)\n\n## ActorDB\n\nDistributed SQL database.\n\n[https://github.com/biokoda/actordb](https://github.com/biokoda/actordb)\n\nBuilt on top of SQLite and LMDB: it replaces SQLite's pager with LMDB, and hooks into SQLite's WAL for Raft-based replication between actors. Developed for applications such as file syncing.\n\n*nanj's observations*:\n\n> Let me share some observations on this project here. Disclaimer, I&rsquo;ve never used it before, all my understanding on it is from following blog posts and skimming its source code.\n>\n> [http://blog.biokoda.com/post/112206754025/why-we-built-actordb](http://blog.biokoda.com/post/112206754025/why-we-built-actordb)\n>\n> [http://blog.biokoda.com/post/133121776825/actordb-how-and-why-we-run-sqlite-on-top-of-lmdb](http://blog.biokoda.com/post/133121776825/actordb-how-and-why-we-run-sqlite-on-top-of-lmdb)\n>\n> #### What&rsquo;s ActorDB\n>\n> In short, the ActorDB team wants to build a file synchronisation solution anyone could install and use.\n>\n> &ldquo;The best way we can describe ActorDB is that it is an ideal server-side database for apps. Think of running a large mail service, dropbox, evernote, etc. They all require server side storage for user data, but the vast majority of queries to the database is within a specific user&rdquo;\n>\n> The author believes that a distributed SQL database is the best storage solution to this.\n>\n> #### Features\n>\n> - Server side storage with SQL including transaction support (unlike KV store or document store)\n> - No single point of failure (user data is replicated among multiple nodes by Raft)\n> - Horizontally scalable (each user has its own database, so they call that ActorDB)\n>\n> #### Other Observations\n>\n> - In the early versions, they use SQLite as both the SQL engine and the storage engine, and hack SQLite&rsquo;s WAL to implement Raft. According to the second article, although they manage to combine all the WALs into a single one, it is not ideal since every user has a separate SQLite base file.\n> - Then, they choose LMDB as the storage engine to mitigate the drawback above. They again hack the SQLite&rsquo;s WAL module so that it completely bypasses the SQLite&rsquo;s b-tree based storage layer, instead stores the actual data (i.e. pages) and Raft logs on LMDB. They find LMDB is perfect engine for their use case, since LMDB supports duplicate keys with sorted value (i.e. dupsort key) as well as multiple sub-dbs within a single database.\n> - Other components are written in Erlang. Also worth pointing out that the server application, the SQL engine, and the storage engine are tightly coupled with each other in this project. To my understanding, there is no clear boundaries between subsystems. This makes ActorDB hard to extend and evolve.\n> - Due to its design, i.e. one db for each user, certain SQL features are not supported, like cross user table joins, ATTACH database etc.\n>\n> Although it&rsquo;s unclear how exactly ActorDB works out in production, and some other limitations in its design & implementation. I think it still provides us with some interesting points for further investigation, such as, the approach of integration LMDB with SQLite. Introduction of WAL replication via Raft.\n\n\n# Questions for the review:\n\n- Let's discuss failure modes.\n    - Durability/consistency\n    - Failure to open\n    - Performance\n    - Corruption (on read)\n    - Upgrades/Downgrades\n- How well does LMDB work across various platforms?\n    - Android\n    - iOS\n    - Windows\n    - macOS\n    - Linux\n- What are some indicators that RKV is the wrong choice?\n    - Definition of 'wrong': wrong API vs wrong storage backend\n    - Syncability and encryption\n    - Effects on downstream Firefox builds\n    - Slow write performance\n- There&rsquo;s going to be lots of knowledge about how to do sync and storage at Mozilla, how are we going to impart that?\n- What&rsquo;s our confidence level about performance?\n- How many RKV DBs will we have and how will they be organized?\n    - Content/main process?\n    - Fast/wired? Early-stage startup.\n    - Topic?\n    - Cached or non-cached\n    - Navigation by key and multiple values per key.\n- At what point do we have to tackle NFS support?\n"
  },
  {
    "path": "text/0016-xulstore-rkv-poc.md",
    "content": "# XULStore Using rkv – Proof of Concept\n\nIn his \"Key-value storage proposal,\" Richard Newman proposed the \"standardization of a simple key-value storage capability, based on LMDB, that is fast, compact, multi-process-capable, and equally usable from JS, Java, Rust, Swift, and C++.\" Dave Townsend reviewed the proposal and concluded that we should pursue it.\n\nThis document summarizes the proposed implementation of key-value storage, explains the purpose and data model of XULStore, and describes the proof of concept implementation of XULStore using key-value storage that is posted to [bug 1460811](https://bugzilla.mozilla.org/show_bug.cgi?id=1460811). It then identifies findings, notes risks, and suggests next steps, ending with a note on data migration strategies.\n\ntl;dr: we should re-implement XULStore using key-value storage to improve its reliability, reduce its complexity, and potentially improve application startup performance while we continue to investigate other consumers for rkv, including session store, TLS session tickets ([bug 1428901](https://bugzilla.mozilla.org/show_bug.cgi?id=1428901)), other desktop consumers of JSONFile, and mobile apps like Fenix.\n\n## rkv\n\nThe [rkv crate](https://crates.io/crates/rkv) is the proposed implementation of key-value storage for Firefox. It's a high-level API to the [LMDB](https://symas.com/lmdb/) library that wraps the low-level [lmdb crate](https://crates.io/crates/lmdb) and intends to improve developer ergonomics and safety via failure-based error propagation, management of database handles, and typed value encoding/decoding, among other abstractions.\n\nrkv has the potential to replace a variety of bespoke data stores based on JSON and other formats that Firefox uses today for relatively simple data that isn't synced. The potential benefits of replacing such stores with rkv include improved reliability and durability, reduced complexity and maintenance cost, and in some cases increased performance and responsiveness.\n\n## XULStore\n\nXULStore is an example of one such store. It persists a set of element attribute values of chrome XUL windows (and, after [bug 1453788](https://bugzilla.mozilla.org/show_bug.cgi?id=1453788), chrome HTML windows), especially their width, height, and screen coordinates. It is currently implemented as a JS XPCOM component that implements the [nsIXULStore](https://searchfox.org/mozilla-central/source/toolkit/components/xulstore/nsIXULStore.idl) API and stores data in the _xulstore.json_ JSON file.\n\nXULStore \"records\" comprise 4-tuples of (document URL, element ID, attribute name, attribute value), all components of which are strings.\n\n## Proof of Concept\n\nThe PoC vendors rkv into mozilla-central and adds a \"xulstore\" crate that expresses two C ABI-compatible sets of functions to Firefox for storing and retrieving XULStore data: a C-string based set that accepts and returns raw pointers to char* arrays, and an nsString-based set that uses the bindings in the [nsstring](https://searchfox.org/mozilla-central/source/xpcom/rust/gtest/nsstring) and [nserror](https://searchfox.org/mozilla-central/source/xpcom/rust/nserror) crates to accept and return Gecko nsString and NS_* result values.\n\nThe PoC then modifies the XULStore XPCOM component to store data in rkv rather than the JSON file. It includes C++ and JS tests of both sets of functions, and it also passes all existing XULStore tests.\n\n### Technology Stack\n\nA rough outline of the component stack (and implementation languages) in the PoC:\n\n![drawing](https://user-images.githubusercontent.com/305455/41000546-2dda9f74-68c3-11e8-8944-f337d2a3d8fb.png)\n\nOnly the edges of this stack are essential—in principal, both C++ and JS consumers could call directly into LMDB, avoiding all Rust layers and the XPCOM intermediary—but each layer serves a useful function: the \"lmdb\" crate wraps LMDB's C interface in safe (albeit low-level) APIs; rkv improves on developer ergonomics and safety with higher-level abstractions; the \"xulstore\" crate translates the XULStore data model to arbitrary key-value pairs; and the XPCOM component provides idiomatic access for both C++ and JS consumers.\n\nNote that the PoC doesn't rewrite the JS XPCOM component in C++, as that work is a \"known known.\" A complete implementation would likely do so, however, given that there are limitations to XPCOM components implemented in JS, especially those that call C functions and manage non-GCed data.\n\n(Specifically, the consumer of the C API needs to free Rust-allocated strings and iterators after finishing with them, but JS lacks destructors, so a JS implementation would either need to copy the data into JS objects and free the memory itself manually or force its own consumers to explicitly free the memory themselves, which would add complexity and consume resources.)\n\nHere's what the stack would look like in a complete implementation (including other consumers of the rkv crate):\n\n![drawing](https://user-images.githubusercontent.com/305455/41000554-34795910-68c3-11e8-8959-c54b082a529f.png)\n\n### Data Model\n\nThere are multiple ways to represent XULStore records in a key-value store. This PoC translates the XULStore data model into rkv concepts by mapping:\n\n*   JSON file store -> rkv store\n*   (XUL document, element ID, attribute name) -> rkv key\n*   attribute value -> rkv value\n\nIn the rkv implementation a \"store\" is an abstraction over an LMDB \"[database](http://www.lmdb.tech/doc/group__internal.html#structMDB__db)\", and there can be multiple distinct stores within a single LMDB \"environment\", which is represented in rkv by an \"RKV\" object. Note that the maximum number of rkv stores that will be accessed via a given RKV instance needs to be known in advance, during initialization of the instance.  Also note that \"a moderate number of slots are cheap but a huge number gets expensive,\" per [http://www.lmdb.tech/doc/group__mdb.html#gaa2fc2f1f37cb1115e733b62cab2fcdbc](http://www.lmdb.tech/doc/group__mdb.html#gaa2fc2f1f37cb1115e733b62cab2fcdbc). The PoC uses a single rkv store within a single environment.\n\nAs rkv keys are strings, the (XUL document, element ID, attribute name) tuple is serialized to a concatenation of its components (with separator char).\n\nA complete implementation might choose the alternative of mapping documents to rkv keys in a single rkv store and storing (element ID, attribute name, attribute value) triples for a given document as a single rkv value, using JSON or the like to structure and serialize the data to a blob:\n\n*   XUL document -> rkv key\n*   blob of (element ID, attribute name, attribute value)* -> rkv value\n\n## Findings\n\n### rkv Completeness\n\nAlthough rkv is incomplete, and it doesn't yet provide some all of the intended ergonomic improvements and safety guarantees, it is nevertheless complete enough to satisfy most of XULStore's requirements, especially given the maturity of the underlying API (and LMDB itself).\n\nThe PoC includes only a few small changes to the vendored copy of rkv, and it only bypasses rkv to access the low-level LMDB API directly in one case (to retrieve a read-only cursor for iterating keys).\n\n### rkv Limitations\n\nOne particular limitation is that rkv doesn't store arbitrary blobs of data, so the PoC has to store attribute values as UTF-8 strings, even though the strings that its consumers persist (including both JS and C++ consumers) are UTF-16, which results in unnecessary string conversions. [rkv issue #20](https://github.com/mozilla-prototypes/rkv/issues/20) tracks adding blob support. (Update: that issue has now been fixed, and the PoC could be updated to store values as blobs, although it would still need to convert the 16-bit strings to 8-bit arrays of bytes.)\n\n### Feasibility of Direct C API\n\nIt's feasible to implement a C API in Rust and call into it directly from both C++ and JS consumers, but it would be more idiomatic to retain an XPCOM interface to the Rust API, especially given that there are both C++ and JS consumers. That XPCOM  interface should be implemented in C++, given the limitations of JS.\n\n## Risks\n\n### Impact on Performance\n\nWhile rkv will improve the reliability and reduce the complexity of the XULStore implementation, it isn't clear that it'll improve application startup perf. XULStore is on the startup path, so it could theoretically do so. But the current implementation doesn't show up much in profiles, so the XULStore workload may not be hot enough to matter either way.\n\nRegardless, we need to ensure that rkv doesn't regress startup perf. And we can mitigate this risk via testing.\n\n### Proliferation of Implementations\n\nrkv needs to be useful for more than just XULStore to justify its footprint and the effort to develop, maintain, and integrate it (i.e. to avoid the \"proliferation of implementations\" problem, a.k.a. [https://xkcd.com/927/](https://xkcd.com/927/)). Although we've identified a variety of other potential consumers, there are no other concrete prototypes yet.\n\nWe can mitigate this risk by waiting to complete the implementation until other rkv consumers appear or by committing not to land the implementation until we see the whites of other rkv consumers' eyes. We can also define a program to replace all JSON stores with rkv and gain approval for that program from engineering leadership.\n\n### Filesystem Compatibility\n\nAlthough LMDB should be compatible with a broad array of filesystems, and the only known incompatibility is with multi-process access on networked filesystems (which isn't an issue for XULStore, whose only consumers live in the parent process), LMDB was originally developed to back an LDAP server, which presumably targets a narrower (and more controlled) range of filesystems than those employed by our users.\n\nSo it's possible that LMDB will be incompatible with some archaic or unusual filesystems that Firefox should continue to support.\n\nWe can mitigate this risk by the usual technique: testing on Nightly and Beta, with a feature flag that enables us to disable the feature on release until we've gained confidence in it. We could also mitigate it to some extent by storing data in both the old and new stores for a period of time, as described in [Notes on Migration](#Notes_on_Migration), in a way that allows to disable the new implementation selectively for incompatible filesystems without losing user data.\n\n### Known Unknowns\n\nThere remain questions to be answered around LMDB usage more generally, such as how often we should check for and clear stale readers, how to check for and recover from data corruption, and how often (and how) to vacuum a datastore.\n\n## Next Steps\n\n1.  Upstream rkv changes and implement desired rkv features.\n1.  Measure PoC perf and efficiency and investigate any regressions.\n1.  Research known unknowns and convert them to knowns.\n\n## <a name=\"Notes_on_Migration\"></a> Notes on Migration\n\nThe PoC doesn't implement data migration from the existing JSON store, but there are several ways it might be implemented.\n\n### Without Support for Downgrades\n\nIf support for downgrading Firefox is not a priority, then the new implementation can detect an upgrade and migrate data from the JSON store to the rkv one, removing records from legacy extensions and the JSON store file itself after migration is complete.\n\nThis is the conventional approach that Firefox has employed for many previous datastore migrations, including XULStore's own migration from localstore.rdf to xulstore.json in Firefox 34 via [bug 559505](https://bugzilla.mozilla.org/show_bug.cgi?id=559505), with the import code subsequently removed in Firefox 55 via [bug 1368567](https://bugzilla.mozilla.org/show_bug.cgi?id=1368567) (albeit that migration didn't remove records for legacy extensions, as those extensions weren't legacy at the time).\n\nOn initialization, the implementation checks for the existence of the new datastore, and if the file doesn't exist, then the implementation imports data from the old datastore, as shown in this [code snippet from XULStore.js](https://hg.mozilla.org/mozilla-central/file/7970ea085861/toolkit/components/xulstore/XULStore.js#l68) (from before the import code was removed):\n\n```\n   if (!this._storeFile.exists()) {\n      this.import();\n    } else {\n      this.readFile();\n    }\n```\n\nThis strategy has the benefits of being conceptually simple, straightforward to implement, and satisfying the use cases of the vast majority of Firefox users who don't downgrade their installations. It is also the well-trodden cow path that Firefox has employed successfully for many previous data migrations.\n\nHowever, it doesn't completely satisfy the use cases for the subset of users who do downgrade their browser installations, either because they switch between versions of multiple installations as a matter of course (f.e. between Beta and Release when testing websites) or because they downgrade Firefox after experiencing issues with a new version.\n\n### With Support for Downgrades\n\nIf support for downgrading Firefox is a priority, then several approaches can be taken to ensure it, described here in rough order of effort and impact (from least to greatest).\n\n#### Retain Old Store\n\nThe simplest form of support for downgrades is to retain the old store (JSON file) in the profile directory after migrating data to the new store, such that a user who upgrades and then downgrades Firefox will go back to their most recent state preceding the upgrade.\n\nWe can achieve this level of support for downgrades basically for \"free,\" by merely not deleting a file that we would otherwise delete, so it is trivial to implement. However, its impact is also limited, as it loses all XUL document state changes that take place after the upgrade.\n\nThis approach is a reasonable solution for the use case of a user who upgrades Firefox, quickly discovers an issue with the new version, and immediately downgrades, as those users are unlikely to have made significant (or any) XUL document state changes during their brief time running the new version.\n\nThis approach does not satisfy the use case of a user who switches between versions of Firefox as a matter of course, since state changes will not propagate between stores (in either direction) after the initial migration.\n\n#### Retain Records From Legacy Extensions\n\nThe implementation could retain records from legacy extensions, so a user who downgrades Firefox to a version that still supports them will go back to their most recent state, without any data loss (as the state cannot change when the extension is disabled).\n\nAs with \"Support Only Most Recent State Before Upgrade\", this is trivial to implement, as it only requires us to not do something that we would otherwise do. And just like that approach, its impact is also limited, as the current version of Firefox (60) is already four versions newer than the last one that supported legacy extensions (56), and it's the basis for the latest ESR, so the number of users still on versions that support legacy extensions who upgrade to a version of Firefox with the new XULStore and then choose to downgrade will presumably be very low.\n\n#### Maintain Both Stores\n\nThe implementation could persist data in both the old and new stores for a period of time sufficient for the likelihood of a downgrade to approach zero, with the new store serving as the system of record, and writes to the old store being deprioritized, so they don't affect application responsiveness.\n\nThis approach requires more effort than the others but also satisfies all use cases. Instead of removing the old store—or retaining it but ignoring it after migrating data to the new store—the implementation would retain and update both stores internally, writing changes to both stores while reading only from the new one (which would become the source of truth, albeit not a single one). So downgrades, whether one-off (due to issues with a new version) or regular (a user who moves back and forth between two versions) would always retain the latest state.\n\nThis approach raises some implementation questions, such as whether to retain the existing JS implementation of nsIXULStore that uses the old store or merge it into the C++ implementation that uses the new store (and if so, how to abstract across the two implementations).\n\nNevertheless, there are straightforward answers to these questions, and it doesn't seem to present technical challenges. It's just more work, and a bigger footprint for the period of time during which the two implementations coexist.\n\nIn theory, this approach could also raise design questions, such as how to support downgrades from a version of Firefox that no longer implements its interface with XUL documents. However, there are no known breaking architectural changes on the horizon, as we have no plans to migrate Firefox interfaces to any other technology besides HTML, with which XULStore is already compatible, per [bug 1453788](https://bugzilla.mozilla.org/show_bug.cgi?id=1453788).\n"
  },
  {
    "path": "text/0017-lmdb-vs-leveldb.md",
    "content": "# LMDB vs. LevelDB\n\nThis document compares the [Lightning Memory-mapped Database](https://symas.com/lmdb/) (LMDB) key-value storage engine to the [LevelDB](https://github.com/google/leveldb) key-value storage engine.\n\n## Meta\n\n### Project Structure\n\n#### LMDB\n\nLMDB is developed by <a href=\"https://symas.com/\">Symas Corporation</a>, a small \"technical support firm\" whose business model is to provide technical support for the open-source software it develops. Its primary product is OpenLDAP, and it originally developed LMDB for use in that server software, which continues to use it. It provides paid technical support for LMDB.\n\nThe LMDB code is stored in OpenLDAP's repository and read-only mirrored to GitHub. Issues and patches are accepted upstream, not in the GitHub repository. LMDB doesn't appear to have an active community beyond the core developers.\n\n#### LevelDB\n\nLevelDB was \"written at Google\" (according to its <a href=\"https://github.com/google/leveldb\">GitHub project page</a>) and \"inspired by <a href=\"https://en.wikipedia.org/wiki/Bigtable\">Bigtable</a>\" (according to its <a href=\"https://en.wikipedia.org/wiki/LevelDB\">Wikipedia entry</a><sup>[1](#footnote-1)</sup>), although it doesn't share code with that proprietary storage system. It was originally developed, among other reasons, to back IndexedDB in Google Chrome, which continues to use it for that purpose.\n\nThe LevelDB code is stored on GitHub in the <a href=\"https://github.com/google/leveldb\">google/leveldb repository</a>. It doesn't appear to have an active community beyond the core developers, although there seems to be an active community of developers of <a href=\"http://leveldb.org/\">NodeJS bindings</a> and documentation.\n\n### License\n\n#### LMDB\n\nLMDB uses the <a href=\"https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblmdb/LICENSE\">OpenLDAP Public License</a>, which appears to be a permissive BSD-style 3-clause license.\n\n#### LevelDB\nLevelDB uses the <a href=\"https://github.com/google/leveldb/blob/master/LICENSE\">BSD 3-Clause \"New\" or \"Revised\" License</a>.\n\n### Language\n\n#### LMDB\n\nLMDB is implemented in C and has bindings to multiple languages, including Rust via the <a href=\"https://crates.io/crates/lmdb/\">lmdb crate</a>, among others. The <a href=\"https://github.com/dw/py-lmdb/\">Python binding</a> appears to be mature (although it is also <a href=\"https://github.com/dw/py-lmdb/commit/78afd828139ad8a7b3555bdfc0df8a556ef0e505\">looking for a new maintainer</a>).\n\n#### LevelDB\n\nLevelDB is implemented in C++ and has bindings to multiple languages, including Rust via the <a href=\"https://crates.io/crates/leveldb\">leveldb crate</a>, among others. The <a href=\"https://github.com/Level/levelup\">Node.JS binding</a> appears to be particularly popular.\n\n### Rust Bindings\n\nOverall, the lmdb crate appears to be more mature and higher quality than the leveldb crate, but both crates suffer from issues, incomplete documentation, and project inactivity.\n\n#### LMDB\n\nThe <a href=\"https://crates.io/crates/lmdb/\">lmdb crate</a> is a Rust binding for LMDB that describes itself as \"Idiomatic and safe APIs for interacting with the Symas Lightning Memory-Mapped Database (LMDB).\"\n\nThe author has released <a href=\"https://crates.io/crates/lmdb/versions\">11 versions</a>, the first in November 2014 and the most recent in March 2018. The crate uses a conservative versioning scheme, with the most recent version being 0.8.0.\n\nThe crate's <a href=\"https://docs.rs/lmdb\">documentation</a> contains some comprehensive reference docs, but it doesn't include an introduction, usage examples, nor tutorials.\n\nThe project is inactive. Although the latest version was released in March 2018, it contains mostly refactorings and docs fixes (per this <a href=\"https://github.com/danburkert/lmdb-rs/compare/0.7.2...0.8.0\">0.7.2...0.8.0 comparison</a>). And open <a href=\"https://github.com/danburkert/lmdb-rs/issues\">issues</a> and <a href=\"https://github.com/danburkert/lmdb-rs/pulls\">pull requests</a> have languished, including several for panics during common operations and a crash caused by a use-after-free bug.\n\nThe author did note in a comment in issue #4 <a href=\"https://github.com/danburkert/lmdb-rs/issues/4#issuecomment-310928298\">on June 25, 2017</a>, \"lmdb is pretty much in maintenance-mode; all major features have been implemented, and it works well. There are of course outstanding issues, but overall the crate isn't seeing too much churn. I'd consider releasing a 1.0 version, except that I don't think that's wise when lmdb itself is < 1.0… One thing to remember is that, when it comes to open source projects, inactivity != abandoned and activity != quality.\"\n\n#### LevelDB\n\nThe <a href=\"https://crates.io/crates/leveldb\">leveldb crate</a> is a Rust binding for LevelDB that describes itself as \"Almost-complete bindings for leveldb for Rust.\"\n\nThe author has released <a href=\"https://crates.io/crates/leveldb\">22 versions</a>, the first in November 2014 and the most recent in September 2017. The crate uses a conservative version scheme, with the most recent version being 0.8.4.\n\nThe crate's <a href=\"http://skade.github.io/leveldb/leveldb/\">documentation</a> includes somewhat comprehensive (but terse and incomplete) reference docs and one usage example, but it doesn't include an introduction nor tutorials.\n\nThe project is inactive. There are no changes since the most recent release, and open issues are stagnant, especially <a href=\"https://github.com/skade/leveldb/issues/6\">issue #4</a>, which is a significant usability flaw that makes the crate difficult to use for even common operations.\n\nThe author did note in issue #6 <a href=\"https://github.com/skade/leveldb/issues/6#issuecomment-265120961\">on December 6, 2016</a>, \"I must admit that a lot of the design around keys was me learning Rust.\" And in issue #27 <a href=\"https://github.com/skade/leveldb/issues/27#issuecomment-342073910\">on November 16, 2017</a>, he described the project as \"the first larger library I wrote on. So you can flag a lot of parts as \"learning to build interfaces over C libs\". It is (mostly) an exercise.\"\n\nFinally, he also noted, \"But mostly, I stopped developing this library and just maintain whatever patches come into it. This is mostly due to leveldb itself being superseded by rocksdb anyways and me rarely using it :).\"\n\nThe project appears to have been forked to the <a href=\"https://crates.io/crates/exonum_leveldb\">exonum_leveldb crate</a> for the <a href=\"https://exonum.com/\">Exonum</a> project, but that project switched to RocksDB shortly thereafter per <a href=\"https://github.com/exonum/exonum/issues/178\">issue #178</a>.\n\n### Docs\n\n#### LMDB\n\nSymas publishes <a href=\"http://www.lmdb.tech/doc/index.html\">comprehensive documentation</a> for LMDB that includes an introduction, a \"getting started\" guide, and a reference.\n\nFor example, the <a href=\"http://www.lmdb.tech/doc/group__mdb.html#ga1206b2af8b95e7f6b0ef6b28708c9127\">MDB_cursor_op</a> documentation describes the \"set of all operations for retrieving data using a cursor.\"\n\nHowever, some of the reference documentation is terse and seems to assume familiarity with the software.\n\n#### LevelDB\n\nGoogle doesn't appear to publish comprehensive documentation for LevelDB, although the repository contains a <a href=\"https://github.com/google/leveldb/blob/master/doc/index.md\">review of the software's major features</a>.\n\nFor example, that document shows code snippets that call `leveldb::Iterator::Seek()`, `SeekToFirst()`, and `SeekToLast()` methods. But there are no links to reference documentation for those methods.\n\n## Features\n\n### Key Types\n\n#### LMDB\n\nKeys are arbitrary byte arrays and are stored in lexicographical order.\n\n#### LevelDB\n\nKeys are arbitrary byte arrays and are stored in lexicographical order by default. Users can also specify a <a href=\"https://github.com/google/leveldb/blob/master/doc/index.md#comparators\">custom comparator</a> that changes the order.\n\n### Value Types\n\n#### LMDB\n\nValues are arbitrary byte arrays.\n\n#### LevelDB\n\nValues are arbitrary byte arrays.\n\n### Iteration\n\n#### LMDB\n\nLMDB supports iterating key/value pairs in lexicographical order. It also supports seeking to a key (or the first key >= a given key). And it supports iteration in reverse order.\n\n#### LevelDB\n\nLevelDB supports iterating key/value pairs in lexicographical order. It also supports seeking to a key (or presumably the first key >= a given key). And it supports iteration in reverse order (although \"reverse iteration may be somewhat slower\" per <a href=\"https://github.com/google/leveldb/blob/master/doc/index.md#iteration\">LevelDB > Iteration</a>).\n\n## ACID Properties\n\n### Atomicity\n\n#### LMDB\n\nLMDB is transactional and guarantees the atomicity of transactions.\n\n#### LevelDB\n\nLevelDB isn't transactional but does provide a <a href=\"https://github.com/google/leveldb/blob/master/doc/index.md#atomic-updates\">WriteBatch API</a> that writes a set of changes to the store atomically.\n\n### Consistency\n\n#### LMDB\n\nThe meaning of consistency for a key-value store is ambiguous, but <a href=\"http://www.lmdb.tech/doc/starting.html\">LMDB: Getting Started</a> notes: \"as long as a transaction is open, a consistent view of the database is kept alive.\" In this case, the word is used to describe a view that is unaffected by subsequent write transactions.\n\n#### LevelDB\n\nThe meaning of consistency for a key-value store is ambiguous, but LevelDB provides a <a href=\"https://github.com/google/leveldb/blob/master/doc/index.md#snapshots\">Snapshot API</a> that it describes as providing \"consistent read-only views over the entire state of the key-value store.\" In this case, the word is used to describe a view that is unaffected by subsequent write transactions.\n\n### Isolation\n\n#### LMDB\n\nPer its <a href=\"http://www.lmdb.tech/doc/\">Introduction</a>, LMDB is \"fully thread-aware and supports concurrent read/write access from multiple processes and threads.\" LMDB implements concurrency via a copy-on-write strategy (<a href=\"https://en.wikipedia.org/wiki/Multiversion_concurrency_control\">MVCC</a>) and permits concurrent access by one writer and unlimited readers.\n\n#### LevelDB\n\nPer its <a href=\"https://github.com/google/leveldb/blob/master/doc/index.md#concurrency\">Concurrency model</a>, LevelDB supports synchronization across threads for some operations, although others require external synchronization. It does not support concurrent access by multiple processes.\n\n### Durability\n\n#### LMDB\n\nLMDB transactions are both durable and performant. LMDB uses <a href=\"https://en.wikipedia.org/wiki/Shadow_paging\">shadow paging</a> to improve the performance of write transactions, and Howard Chu notes in this <a href=\"https://news.ycombinator.com/item?id=7782400\">Hacker News comment</a>: \"LMDB write performance remains uniform under load.\"\n\n#### LevelDB\n\nPer <a href=\"https://github.com/google/leveldb/blob/master/doc/index.md#synchronous-writes\">Synchronous Writes</a>, LevelDB sacrifices durability for performance, persisting changes asynchronously by default, although it's possible to configure a write to be performed synchronously.\n\nDurability may not be the highest priority for LevelDB, as its primary Google consumer, IndexedDB, is <a href=\"https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Basic_Concepts_Behind_IndexedDB#durable\">not guaranteed to be durable in browsers</a>.\n\n## Quality Attributes\n\n### Performance\n\nIn a run of [this benchmark](https://github.com/mykmelez/kvbench/), LMDB was approximately an order of magnitude faster than LevelDB to open a database and read entries, while being roughly equivalent to 3x faster to write entries (depending on the type of write), in this [benchmark run on a macOS](https://mykmelez.github.io/kvbench/criterion/report/).\n\n#### LMDB\n\nLMDB read performance is excellent, and its write performance is reasonable and consistent.\n\nLMDB reuses the space freed by deleted entries instead of compacting the store, which avoids the complexity and performance impact of compaction (trading off disk space in some situations).\n\n#### LevelDB\n\nLevelDB read performance is good, with <a href=\"http://www.lmdb.tech/bench/microbench/benchmark.html\">this 2011 benchmark</a> showing improvements over engines like SQLite.\n\nLevelDB \"compacts on open by design,\" which can slow down opening a database and seeking to a key significantly, per <a href=\"https://github.com/google/leveldb/issues/210\">#210</a>.\n\n### Software Footprint\n\n#### LMDB\n\nThe <a href=\"https://symas.com/lmdb/\">LMDB product page</a> claims that LMDB comprises \"32KB of object code,\" and its <a href=\"https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database\">Wikipedia entry</a> claims that it's 64KB in size. The sample Rust program that uses the <a href=\"https://crates.io/crates/lmdb\">lmdb crate</a> in the <a href=\"https://github.com/mykmelez/kvbench\">mykmelez/kvbench</a> repo is about 73kB larger than a control program.<sup>[2](#footnote-2)</sup>\n\n#### LevelDB\n\nThe <a href=\"https://en.wikipedia.org/wiki/LevelDB\">LevelDB Wikipedia entry</a> claims its \"binary size\" is 350kB. The sample Rust program that uses the <a href=\"https://crates.io/crates/leveldb\">leveldb crate</a> in the <a href=\"https://github.com/mykmelez/kvbench\">mykmelez/kvbench</a> repo is about 207kB larger than a control program.<sup>[3](#footnote-3)</sup>\n\n### Reliability\n\n#### LMDB\n\nAccording to its <a href=\"https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database#Reliability\">Wikipedia entry</a>, \"LMDB was designed from the start to resist data loss in the face of system and application crashes. Its copy-on-write approach never overwrites currently-in-use data. [Which] means the structure on disk/storage is always valid, so application or system crashes can never leave the database in a corrupted state. In its default mode, at worst a crash can lose data from the last not-yet-committed write transaction. Even with all asynchronous modes enabled, it is only an OS catastrophic failure or hardware power-loss event rather than merely an application crash that could potentially result in any data corruption.\"\n\n#### LevelDB\n\nAccording to its <a href=\"https://en.wikipedia.org/wiki/LevelDB#Bugs_and_reliability\">Wikipedia entry</a>, \"LevelDB is widely noted for being unreliable and databases it manages are prone to corruption. Academic studies of past versions of LevelDB have found that, under some file systems, the data stored in those versions of LevelDB might become inconsistent after a system crash or power failure. LevelDB corruption is so commonplace that corruption detection has to be built into applications that use it.\"\n\nHowever, a review of the <a href=\"https://groups.google.com/forum/#!topic/leveldb/GXhx8YvFiig\">LevelDB Issue 197 Workaround</a> discussion and some of the <a href=\"https://github.com/google/leveldb/issues?utf8=%E2%9C%93&q=corrupt+is%3Aissue\">issues containing the string \"corrupt\"</a> in LevelDB's issue tracker suggests that corruption issues are taken seriously and actively investigated.\n\n### Storage Footprint\n\nIn a run of the disk space \"benchmark\" in the [mykmelez/kvbench](https://github.com/mykmelez/kv-bench/) repo, LMDB used roughly 1.5–4x more disk space than LevelDB to store equivalent amounts of data. However, that benchmark employs a hack to measure the space taken by the LMDB and LevelDB datastores on disk, and it isn't clear that the hack is an accurate way to measure disk usage.\n\n#### LMDB\n\nLMDB doesn't compress data on disk, and it doesn't compact datastores, but it does reuse the space freed by deleted entries.\n\n#### LevelDB\n\nLevelDB compresses data on disk using <a href=\"https://github.com/google/snappy\">Google's Snappy compression library</a>, and it compacts datastores.\n\n## References\n\n[LMDB: The Leveldb Killer?](http://banksco.de/p/lmdb-the-leveldb-killer.html)\n\n[Is LMDB a LevelDB Killer?](https://symas.com/is-lmdb-a-leveldb-killer/)\n\n[Understanding LMDB Database File Sizes and Memory Utilization](https://symas.com/understanding-lmdb-database-file-sizes-and-memory-utilization/)\n\n[LevelDB \"impl\" document](https://github.com/google/leveldb/blob/master/doc/impl.md)\n\n## Notes\n\n### Footnote 1\n\n<sup>1</sup>See Wikipedia's <a href=\"https://en.wikipedia.org/wiki/Reliability_of_Wikipedia\">Reliability of Wikipedia</a> article for much discussion about the reliability of Wikipedia articles generally. This document uses the Wikipedia articles on LMDB and LevelDB as a source—but not the sole source—of information about the two storage engines.\n\n### Footnote 2\n\n<sup>2</sup>490,880 bytes for the LMDB program versus 417,804 bytes for the control program on a macOS system is an 73,076 byte difference. Both programs built using stable Rust v1.30.1 with the \"release\" profile and then stripped on December 18, 2018.\n\n### Footnote 3\n\n<sup>3</sup>624,900 bytes for the LevelDB program versus 417,804 bytes for the control program on a macOS system is a 207,096 byte difference. Both programs built using stable Rust v1.30.1 with the \"release\" profile and then stripped on December 18, 2018.\n"
  }
]