Full Code of ProseMirror/prosemirror for AI

Repository: ProseMirror/prosemirror
Branch: master
Commit: 72428f67b2f0
Files: 17
Total size: 49.0 KB

Directory structure:
gitextract_8wy5dbbd/

├── .github/
│   ├── FUNDING.yml
│   └── ISSUE_TEMPLATE.md
├── .gitignore
├── CONTRIBUTING.md
├── LICENSE
├── README.md
├── bin/
│   └── pm.js
├── demo/
│   ├── bench/
│   │   ├── example.js
│   │   ├── index.html
│   │   ├── index.js
│   │   ├── mutate.js
│   │   └── type.js
│   ├── demo.css
│   ├── demo.ts
│   └── index.html
├── package.json
└── tsconfig.json

================================================
FILE CONTENTS
================================================

================================================
FILE: .github/FUNDING.yml
================================================
patreon: marijn
custom: ['https://www.paypal.com/paypalme/marijnhaverbeke', 'https://marijnhaverbeke.nl/fund/']
github: marijnh

================================================
FILE: .github/ISSUE_TEMPLATE.md
================================================
<!--

Please provide:

 - Necessary steps to reproduce the issue. If the editor has to be set up in a specific way, running code (minimal code, not a dump of your project) is very useful.

 - If this might be browser-related, tell us the browsers and platforms you tested on.

-->


================================================
FILE: .gitignore
================================================
.tern-port
/node_modules
/model
/transform
/state
/view
/history
/collab
/commands
/inputrules
/keymap
/search
/schema-basic
/schema-list
/schema-table
/menu
/markdown
/dropcursor
/test-builder
/gapcursor
/changeset
/website
/example-setup
/tables
/notes.txt
/yarn.lock
/bin/.pm-dev.pid
/demo/demo.js


================================================
FILE: CONTRIBUTING.md
================================================
# How to contribute

- [Getting help](#getting-help)
- [Submitting bug reports](#submitting-bug-reports)
- [Contributing code](#contributing-code)

## Getting help

Community discussion, questions, and informal bug reporting is done on the
[discuss.ProseMirror forum](http://discuss.prosemirror.net).

## Submitting bug reports

Report bugs on the
[GitHub issue tracker](http://github.com/prosemirror/prosemirror/issues).
Before reporting a bug, please read these pointers.

- The issue tracker is for *bugs*, not requests for help. Questions
  should be asked on the [forum](http://discuss.prosemirror.net).

- Include information about the version of the code that exhibits the
  problem. For browser-related issues, include the browser and browser
  version on which the problem occurred.

- Mention very precisely what went wrong. "X is broken" is not a good
  bug report. What did you expect to happen? What happened instead?
  Describe the exact steps a maintainer has to take to make the
  problem occur. A screencast can be useful, but is no substitute for
  a textual description.

- A great way to make it easy to reproduce your problem, if it can not
  be trivially reproduced on the website demos, is to submit a script
  that triggers the issue.

## Contributing code

Code written by "AI" language models (either partially or fully) is
**not welcome**. Both because you cannot guarantee it's not parroting
copyrighted content, and because it tends to be of low quality and a
waste of time to review.

- Make sure you have a [GitHub Account](https://github.com/signup/free)

- Fork the relevant repository
  ([how to fork a repo](https://help.github.com/articles/fork-a-repo))

- Create a local checkout of the code. You can use the
  [main repository](https://github.com/prosemirror/prosemirror) to
  easily check out all core modules.

- Make your changes, and commit them

- Follow the code style of the rest of the project (see below).

- If your changes are easy to test or likely to regress, add tests in
  the relevant `test/` directory. Either put them in an existing
  `test-*.js` file, if they fit there, or add a new file.

- Make sure all tests pass. Run `npm run test` to verify tests pass.

- Submit a pull request ([how to create a pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork)).
  Don't put more than one feature/fix in a single pull request.

By contributing code to ProseMirror you

 - Agree to license the contributed code under the project's [MIT
   license](https://github.com/ProseMirror/prosemirror/blob/master/LICENSE).

 - Confirm that you have the right to contribute and license the code
   in question. (Either you hold all rights on the code, or the rights
   holder has explicitly granted the right to use it like this,
   through a compatible open source license or through a direct
   agreement with you.)

### Coding standards

- TypeScript, targeting an ES5 runtime (i.e. don't use library
  elements added by ES6, don't use ES7/ES.next syntax).

- 2 spaces per indentation level, no tabs.

- No semicolons except when necessary.

- Follow the surrounding code when it comes to spacing, brace
  placement, etc.

- Brace-less single-statement bodies are encouraged (whenever they
  don't impact readability).

- [getdocs](https://github.com/marijnh/getdocs-ts)-style doc comments
  above items that are part of the public API.

- ProseMirror does *not* follow JSHint or JSLint prescribed style.
  Patches that try to 'fix' code to pass one of these linters will not
  be accepted.


================================================
FILE: LICENSE
================================================
Copyright (C) 2015-2017 by Marijn Haverbeke <marijn@haverbeke.berlin> and others

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.


================================================
FILE: README.md
================================================
# prosemirror

[ [**WEBSITE**](https://prosemirror.net) | [**ISSUES**](https://github.com/prosemirror/prosemirror/issues) | [**FORUM**](https://discuss.prosemirror.net) ]

ProseMirror is a well-behaved rich semantic content editor based on
contentEditable, with support for collaborative editing and custom
document schemas.

The ProseMirror library consists of a number of separate
[modules](https://github.com/prosemirror/). This repository just
serves as a central issue tracker, and holds a script to help easily
check out all the core modules for development.

The [project page](https://prosemirror.net) has more information, a
number of [examples](https://prosemirror.net/examples/) and the
[documentation](https://prosemirror.net/docs/).

This code is released under an
[MIT license](https://github.com/prosemirror/prosemirror/tree/master/LICENSE).
There's a [forum](http://discuss.prosemirror.net) for general
discussion and support requests, and the
[Github bug tracker](https://github.com/prosemirror/prosemirror/issues)
is the place to report issues.

**STOP READING HERE IF YOU'RE SIMPLY _USING_ PROSEMIRROR. YOU CAN
INSTALL THE SEPARATE [NPM
MODULES](https://www.npmjs.com/search?q=prosemirror-) FOR THAT. THE
INSTRUCTIONS BELOW ONLY APPLY WHEN _DEVELOPING_ PROSEMIRROR!**

## Setting up a dev environment

Clone this repository, and make sure you have
[node](https://nodejs.org/en/) installed.
Next, from the cloned directory run:

    bin/pm install

This will fetch the submodules, install their dependencies, and build
them.

The `bin/pm` script in this repository provides functionality for
working with the repositories:

 * `bin/pm build` rebuilds all the modules

 * `bin/pm watch` sets up a process that automatically rebuilds the
   modules when they change

 * `bin/pm status` prints the git status of all submodules

 * `bin/pm commit <args>` runs `git commit` with the given arguments
   in all submodules that have pending changes

 * `bin/pm test` runs the (non-browser) tests in all modules

 * `bin/pm push` runs `git push` in all modules

 * `bin/pm grep <pattern>` greps through the source code for the
   modules for the given pattern

 * `bin/pm dev-start` starts a server that rebuilds the packages
   whenever their sources change, and exposes the demo (`demo/*`)
   under a webserver on port 8080

(Functionality for managing releases will be added in the future.)

## Community

Development of ProseMirror happens in the various repositories exposed
under the [ProseMirror](https://github.com/ProseMirror) organization
on GitHub. Bugs for core packages are tracked in the [bug
tracker](https://github.com/prosemirror/prosemirror/issues) for the
meta repository.

We aim to be an inclusive, welcoming community. To make that explicit,
we have a [code of
conduct](http://contributor-covenant.org/version/1/1/0/) that applies
to communication around the project.


================================================
FILE: bin/pm.js
================================================
#!/usr/bin/env node

// NOTE: Don't require anything from node_modules here, since the
// install script has to be able to run _before_ that exists.
let child = require("child_process"), fs = require("fs"), path = require("path")

const {join} = path

let main = ["model", "transform", "state", "view",
            "keymap", "inputrules", "history", "collab", "commands", "gapcursor",
            "schema-basic", "schema-list"]
let mods = main.concat(["menu", "example-setup", "markdown", "dropcursor", "test-builder", "changeset", "search"])
let modsAndWebsite = mods.concat("website")

let projectDir = join(__dirname, "..")

function joinP(...args) {
  return join(projectDir, ...args)
}

function mainFile(pkg) {
  let index = joinP(pkg, "src", "index.ts"), self = joinP(pkg, "src", pkg + ".ts")
  if (fs.existsSync(index)) return index
  if (fs.existsSync(self)) return self
  throw new Error("Couldn't find a main file for " + pkg)
}

function start() {
  let command = process.argv[2]
  if (command && !["install", "--help", "modules"].includes(command)) assertInstalled()
  let args = process.argv.slice(3)
  let cmdFn = {
    "status": status,
    "commit": commit,
    "install": install,
    "build": build,
    "test": test,
    "push": push,
    "pull": pull,
    "grep": grep,
    "run": runCmd,
    "watch": watch,
    "changes": changes,
    "modules": listModules,
    "release": release,
    "unreleased": unreleased,
    "dev-start": devStart,
    "dev-stop": devStop,
    "mass-change": massChange,
    "--help": showHelp
  }[command]
  if (!cmdFn || cmdFn.length > args.length) help(1)
  cmdFn.apply(null, args)
}

function showHelp() {
  help(0)
}

function help(status) {
  console.log(`Usage:
  pm install [--ssh]      Clone and symlink the packages, install dependencies, build
  pm build                Build all modules
  pm status               Print out the git status of packages
  pm commit <args>        Run git commit in all packages that have changes
  pm push                 Run git push in packages that have new commits
  pm pull                 Run git pull in all packages
  pm test                 Run the tests from all packages
  pm watch                Set up a process that rebuilds the packages on change
  pm grep <pattern>       Grep through the source code for all packages
  pm run <command>        Run the given command in each of the package dirs
  pm changes              Show commits since the last release for all packages
  pm mass-change <files> <pattern> <replacement>
                          Run a regexp-replace on the matching files in each package
  pm release <module>     Generate a new release for the given module.
  pm unreleased           List committed but unreleased changes.
  pm modules [--core]     Emit a list of all package names
  pm dev-start            Start development server
  pm dev-stop             Stop development server, if running
  pm --help`)
  process.exit(status)
}

function assertInstalled() {
  modsAndWebsite.forEach(repo => {
    if (!fs.existsSync(joinP(repo))) {
      console.error("module `%s` is missing. Did you forget to run `pm install`?", repo)
      process.exit(1)
    }
  })
}

function run(cmd, args, pkg) {
  return child.execFileSync(cmd, args, {
    cwd: pkg === null ? undefined : pkg ? joinP(pkg) : projectDir,
    encoding: "utf8",
    stdio: ["ignore", "pipe", process.stderr]
  })
}

function status() {
  modsAndWebsite.forEach(repo => {
    let output = run("git", ["status", "-sb"], repo)
    if (output != "## master...origin/master\n" && output != "## main...origin/main\n")
      console.log(repo + ":\n" + run("git", ["status"], repo))
  })
}

function commit(...args) {
  modsAndWebsite.forEach(repo => {
    if (run("git", ["diff"], repo) || run("git", ["diff", "--cached"], repo))
      console.log(repo + ":\n" + run("git", ["commit"].concat(args), repo))
  })
}

function install(arg = null) {
  let base = "https://github.com/prosemirror/"
  if (arg == "--ssh") { base = "git@github.com:ProseMirror/" }
  else if (arg != null) help(1)

  modsAndWebsite.forEach(repo => {
    if (fs.existsSync(joinP(repo))) {
      console.warn("Skipping cloning of " + repo + " (directory exists)")
      return
    }
    let origin = base + (repo == "website" ? "" : "prosemirror-") + repo + ".git"
    run("git", ["clone", origin, repo])
  })

  console.log("Running npm install")
  run("npm", ["install"])
  console.log("Building modules")
  build()
}

async function build() {
  console.info("Building...")
  let t0 = Date.now()
  await require("@marijn/buildtool").build(mods.map(mainFile), buildOptions)
  console.info(`Done in ${((Date.now() - t0) / 1000).toFixed(2)}s`)
}

function test(...args) {
  let runTests = require("@marijn/testtool")
  let {tests, browserTests} = runTests.gatherTests(mods.map(m => joinP(m)))
  let browsers = [], grep, noBrowser = false
  for (let i = 0; i < args.length; i++) {
    if (args[i] == "--firefox") browsers.push("firefox")
    if (args[i] == "--chrome") browser.push("chrome")
    if (args[i] == "--no-browser") noBrowser = true
    if (args[i] == "--grep") grep = args[++i]
  }
  if (!browsers.length && !noBrowser) browsers.push("chrome")
  runTests.runTests({tests, browserTests, browsers, grep}).then(failed => process.exit(failed ? 1 : 0))
}

function push() {
  modsAndWebsite.forEach(repo => {
    if (/\bahead\b/.test(run("git", ["status", "-sb"], repo)))
      run("git", ["push"], repo)
  })
}

function pull() {
  modsAndWebsite.forEach(repo => run("git", ["pull"], repo))
}

function grep(pattern) {
  let files = []
  const {globSync: glob} = require("glob")
  mods.forEach(repo => {
    files = files.concat(glob(joinP(repo, "src", "*.ts"))).concat(glob(joinP(repo, "test", "*.ts")))
  })
  files = files.concat(glob(joinP("website", "src", "**", "*.js")))
    .concat(glob(joinP("website", "pages", "examples", "*", "*.js")))
  try {
    console.log(run("grep", ["--color", "-nH", "-e", pattern].concat(files.map(f => path.relative(process.cwd(), f))), null))
  } catch(e) {
    process.exit(1)
  }
}

function runCmd(cmd, ...args) {
  mods.forEach(repo => {
    console.log(repo + ":")
    try {
      console.log(run(cmd, args, repo))
    } catch (e) {
      console.log(e.toString())
      process.exit(1)
    }
  })
}

function changes() {
  mods.forEach(repo => {
    let lastTag = run("git", ["describe", "HEAD", "--tags", "--abbrev=0"], repo).trim()
    if (!lastTag) return console.log("No previous tag for " + repo + "\n")
    let history = run("git", ["log", lastTag + "..HEAD"], repo).trim()
    if (history) console.log(repo + ":\n" + "=".repeat(repo.length + 1) + "\n\n" + history + "\n")
  })
}

function editReleaseNotes(notes) {
  let noteFile = join(projectDir, "notes.txt")
  fs.writeFileSync(noteFile, notes.head + notes.body)
  run(process.env.EDITOR || "emacs", [noteFile], null)
  let edited = fs.readFileSync(noteFile)
  fs.unlinkSync(noteFile)
  if (!/\S/.test(edited)) process.exit(0)
  let split = /^(.*)\n+([^]*)/.exec(edited)
  return {head: split[1] + "\n\n", body: split[2]}
}

function version(mod) {
  return require(join("..", mod, "package.json")).version
}

function release(mod, ...args) {
  let currentVersion = version(mod)
  let noteArg = args.indexOf("--notes")
  let extra = noteArg > -1 ? args[noteArg + 1] : null
  let changes = changelog(mod, currentVersion, extra)
  let newVersion = bumpVersion(currentVersion, changes)
  console.log(`Creating prosemirror-${mod} ${newVersion}`)

  let notes = releaseNotes(mod, changes, newVersion)
  if (args.indexOf("--edit") > -1) nodes = editReleaseNotes(notes)

  setModuleVersion(mod, newVersion)
  if (changes.breaking.length) setDepVersion(mod, newVersion)
  fs.writeFileSync(joinP(mod, "CHANGELOG.md"), notes.head + notes.body + fs.readFileSync(joinP(mod, "CHANGELOG.md"), "utf8"))
  run("git", ["add", "package.json"], mod)
  run("git", ["add", "CHANGELOG.md"], mod)
  run("git", ["commit", "-m", `Mark version ${newVersion}`], mod)
  run("git", ["tag", newVersion, "-m", `Version ${newVersion}\n\n${notes.body}`, "--cleanup=verbatim"], mod)
}

function unreleased() {
  mods.forEach(mod => {
    let ver = version(mod), changes = changelog(mod, ver)
    if (changes.fix.length || changes.feature.length || changes.breaking.length)
      console.log(mod + ":\n\n", releaseNotes(mod, changes, "xxx").body)
  })
}

function changelog(repo, since, extra) {
  let commits = run("git", ["log", "--format=%B", "--reverse", since + "..HEAD"], repo)
  if (extra) commits += "\n\n" + extra
  let result = {fix: [], feature: [], breaking: []}
  let re = /\n\r?\n(BREAKING|FIX|FEATURE):\s*([^]*?)(?=\r?\n\r?\n|\r?\n?$)/g, match
  while (match = re.exec(commits)) result[match[1].toLowerCase()].push(match[2].replace(/\r?\n/g, " "))
  return result
}

function bumpVersion(version, changes) {
  let [major, minor, patch] = version.split(".")
  if (changes.breaking.length) return `${Number(major) + 1}.0.0`
  if (changes.feature.length) return `${major}.${Number(minor) + 1}.0`
  if (changes.fix.length) return `${major}.${minor}.${Number(patch) + 1}`
  throw new Error("No new release notes!")
}

function releaseNotes(mod, changes, version) {
  let pad = n => n < 10 ? "0" + n : n
  let d = new Date, date = d.getFullYear() + "-" + pad(d.getMonth() + 1) + "-" + pad(d.getDate())

  let types = {breaking: "Breaking changes", fix: "Bug fixes", feature: "New features"}

  let refTarget = "https://prosemirror.net/docs/ref/"
  let head = `## ${version} (${date})\n\n`, body = ""
  for (let type in types) {
    let messages = changes[type]
    if (messages.length) body += `### ${types[type]}\n\n`
    messages.forEach(message => body += message.replace(/\]\(##/g, "](" + refTarget + "#") + "\n\n")
  }
  return {head, body}
}

function setModuleVersion(mod, version) {
  let file = joinP(mod, "package.json")
  fs.writeFileSync(file, fs.readFileSync(file, "utf8").replace(/"version":\s*".*?"/, `"version": "${version}"`))
}

function setDepVersion(mod, version) {
  modsAndWebsite.forEach(repo => {
    if (repo == mod) return
    let file = joinP(repo, "package.json"), text = fs.readFileSync(file, "utf8")
    let result = text.replace(/"prosemirror-(.*?)":\s*".*?"/g, (match, dep) => {
      return dep == mod ? `"prosemirror-${mod}": "^${version}"` : match
    })
    if (result != text) {
      fs.writeFileSync(file, result)
      run("git", ["add", "package.json"], repo)
      run("git", ["commit", "-m", `Upgrade prosemirror-${mod} dependency`], repo)
    }
  })
}

function listModules() {
  console.log((process.argv.includes("--core") ? main : mods).join("\n"))
}

const buildOptions = {}

function watch() {
  require("@marijn/buildtool").watch(mods.map(mainFile), [join(__dirname, "..", "demo", "demo.ts")], buildOptions)
}

const pidFile = join(__dirname, ".pm-dev.pid")
function devPID() {
  try { return JSON.parse(fs.readFileSync(pidFile, "utf8")) }
  catch(_) { return null }
}

function startServer() {
  let serve = path.resolve(join(__dirname, "..", "demo"))
  let port = +(process.env.PORT || 8080)
  let moduleserver = new (require("esmoduleserve/moduleserver"))({root: serve, maxDepth: 2})
  let serveStatic = require("serve-static")(serve)
  require("http").createServer((req, resp) => {
    if (/^\/test\/?($|\?)/.test(req.url)) {
      let runTests = require("@marijn/testtool")
      let {browserTests} = runTests.gatherTests(mods.map(m => joinP(m)))
      resp.writeHead(200, {"content-type": "text/html"})
      resp.end(runTests.testHTML(browserTests.map(f => path.relative(serve, f)), false))
    } else {
      moduleserver.handleRequest(req, resp) || serveStatic(req, resp, _err => {
        resp.statusCode = 404
        resp.end('Not found')
      })
    }
  }).listen(port, process.env.OPEN ? undefined : "127.0.0.1")
  console.log(`Dev server listening on ${port}`)
}


function devStart() {
  let pid = devPID()
  if (pid != null) {
    try { run("ps", ["-p", String(pid)]) }
    catch (_) { pid = null }
  }
  if (pid != null) {
    console.log("Dev server already running at pid " + pid)
    return
  }

  fs.writeFileSync(pidFile, process.pid + "\n")
  function del() { fs.unlink(pidFile, () => {}); console.log("Stop") }
  function delAndExit() { del(); process.exit() }
  process.on("exit", del)
  process.on("SIGINT", delAndExit)
  process.on("SIGTERM", delAndExit)

  startServer()
  watch()
}

function devStop() {
  let pid = devPID()
  if (pid == null) {
    console.log("Dev server not running")
  } else {
    process.kill(pid, "SIGTERM")
    console.log("Killed dev server with pid " + pid)
  }
}

function massChange(file, pattern, replacement = "") {
  let re = new RegExp(pattern, "g")
  modsAndWebsite.forEach(repo => {
    let {globSync: glob} = require("glob")
    glob(joinP(repo, file)).forEach(file => {
      let content = fs.readFileSync(file, "utf8"), changed = content.replace(re, replacement)
      if (changed != content) {
        console.log("Updated " + file)
        fs.writeFileSync(file, changed)
      }
    })
  })
}

start()


================================================
FILE: demo/bench/example.js
================================================
const {schema, doc, p, ol, ul, li, h1, h2, blockquote, em, code, a} = require("prosemirror-model/test/build")

exports.schema = schema

let example = doc(
  h1("Collaborative Editing in ProseMirror"),
  p("This post describes the algorithm used to make collaborative editing work in ", a("ProseMirror"), ". For an introduction to ProseMirror, see ", a("another post"), " here."),
  h2("The Problem"),
  p("A real-time collaborative editing system is one where multiple people may work on a document at the same time. The system ensures that the documents stay synchronized—changes made by individual users are sent to other users, and show up in their representation of the document."),
  p("Since transmitting changes over any kind of network is going to take time, the complexity of such systems lies in the way they handle concurrent updates. One solution is to allow users to lock the document (or parts of it) and thus prevent concurrent changes from happening at all. But this forces users to think about locks, and to wait when the lock they need is not available. We'd prefer not to do that."),
  p("If we allow concurrent updates, we get situations where user A and user B both did something, unaware of the other user's actions, and now those things they did have to be reconciled. The actions might not interact at all—when they are editing different parts of the document—or interact very much—when they are trying to change the same word."),
  h2("Operational Transformation"),
  p("A lot of research has gone into this problem. And I must admit that, though I did read a bunch of papers, I definitely do not have a deep knowledge of this research, and if you find that I misrepresent something or am missing an interesting reference, I am very interested in an ", a("email"), " that tells me about it."),
  p("A lot of this research is about truly distributed systems, where a group of nodes exchange messages among themselves, without a central point of control. The classical approach to the problem, which is called ", a("Operational Transformation"), ", is such a distributed algorithm. It defines a way to describe changes that has two properties:"),
  ol(li(p("You can transform changes relative to other changes. So if user A inserted an “O” at offset 1, and user B concurrently inserted a “T” at offset 10, user A can transform B's change relative to its own change, an insert the “T” at offset 11, because an extra character was added in front of the change's offset.")),
     li(p("No matter in which order concurrent changes are applied, you end up with the same document. This allows A to transform B's change relative to its own change, and B to transform A's change similarly, without the two users ending up with different documents."))),
  p("An Operational Transformation (OT) based system applies local changes to the local document immediately, and broadcasts them to other users. Those users will transform and apply them when they get them. In order to know exactly which local changes a remote change should be transformed through, such a system also has to send along some representation of the state of the document at the time the change was made."),
  p("That sounds relatively simple. But it is a nightmare to implement. Once you support more than a few trivial types of changes (things like “insert” and “delete”), ensuring that applying changes in any order produces the same document becomes very hard."),
  p("Joseph Gentle, one of the engineers who worked on Google Wave, ", a("stated"), "..."), blockquote(p("Unfortunately, implementing OT sucks. There's a million algorithms with different trade-offs, mostly trapped in academic papers. The algorithms are really hard and time consuming to implement correctly.")),
  h2("Centralization"),
  p("The design decisions that make the OT mechanism complex largely stem from the need to have it be truly distributed. Distributed systems have nice properties, both practically and politically, and they tend to be interesting to work on."),
  p("But you can save oh so much complexity by introducing a central point. I am, to be honest, extremely bewildered by Google's decision to use OT for their Google Docs—a centralized system."),
  p("ProseMirror's algorithm is centralized, in that it has a single node (that all users are connected to) making decisions about the order in which changes are applied. This makes it relatively easy to implement and to reason about."),
  p("And I don't actually believe that this property represents a huge barrier to actually running the algorithm in a distributed way. Instead of a central server calling the shots, you could use a consensus algorithm like ", a("Raft"), " to pick an arbiter. (But note that I have not actually tried this.)"),
  h2("The Algorithm"),
  p("Like OT, ProseMirror uses a change-based vocabulary and transforms changes relative to each other. Unlike OT, it does not try to guarantee that applying changes in a different order will produce the same document."),
  p("By using a central server, it is possible—easy even—to have clients all apply changes in the same order. You can use a mechanism much like the one used in code versioning systems. When a client has made a change, they try to ", em("push"), " that change to the server. If the change was based on the version of the document that the server considers current, it goes through. If not, the client must ", em("pull"), " the changes that have been made by others in the meantime, and ", em("rebase"), " their own changes on top of them, before retrying the push."),
  p("Unlike in git, the history of the document is linear in this model, and a given version of the document can simply be denoted by an integer."),
  p("Also unlike git, all clients are constantly pulling (or, in a push model, listening for) new changes to the document, and track the server's state as quickly as the network allows."),
  p("The only hard part is rebasing changes on top of others. This is very similar to the transforming that OT does. But it is done with the client's ", em("own"), " changes, not remote changes."),
  p("Because applying changes in a different order might create a different document, rebasing isn't quite as easy as transforming all of our own changes through all of the remotely made changes."),
  h2("Position Mapping"),
  p("Whereas OT transforms changes relative to ", em("other changes"), ", ProseMirror transforms them using a derived data structure called a ", em("position map"), ". Whenever you apply a change to a document, you get a new document and such a map, which you can use to convert positions in the old document to corresponding positions in the new document. The most obvious use case of such a map is adjusting the cursor position so that it stays in the same conceptual place—if a character was inserted before it, it should move forward along with the surrounding text."),
  p("Transforming changes is done entirely in terms of mapping positions. This is nice—it means that we don't have to write change-type-specific transformation code. Each change has one to three positions associated with it, labeled ", code("from"), ", ", code("to"), ", and ", code("at"), ". When transforming the change relative to a given other change, those positions get mapped through the other change's position map."),
  p("For example, if a character is inserted at position 5, the change “delete from 10 to 14” would become “delete from 11 to 15” when transformed relative to that insertion."),
  p("Every change's positions are meaningful only in the exact document version that it was originally applied to. A position map defines a mapping between positions in the two document versions before and after a change. To be able to apply a change to a different version, it has to be mapped, step by step, through the changes that lie between its own version and the target version."),
  p("(For simplicity, examples will use integers for positions. Actual positions in ProseMirror consist of an integer offset in a paragraph plus the path of that paragraph in the document tree.)"),
  h2("Rebasing Positions"),
  p("An interesting case comes up when a client has multiple unpushed changes buffered. If changes from a peer come in, all of the locally buffered changes have to be moved on top of those changes. Say we have local changes ", em("L1"), " and ", em("L2"), ", and are rebasing them onto remote changes ", em("R1"), " and ", em("R2"), ", where ", em("L1"), " and ", em("R1"), " start from the same version of the document."),
  p("First, we apply R1 and R2 to our representation of that original version (clients must track both the document version they are currently displaying, which includes unsent changes, and the version that does not yet include those changes). This creates two position maps ", em("mR1"), " and ", em("mR2"), "."),
  p("We can simply map ", em("L1"), " forward through those maps to arrive at ", em("L1⋆"), ", the remapped version of ", em("L1"), ". But ", em("L2"), " was based on the document that existed after applying ", em("L1"), ", so we first have to map it ", em("backwards"), " through ", em("mL1"), ", the original map created by applying ", em("L1"), ". Now it refers to the same version that ", em("R1"), " starts in, so we can map it forward through ", em("mR1"), " and ", em("mR2"), ", and then finally though ", em("mL1⋆"), ", the map created by applying ", em("L1⋆"), ". Now we have ", em("L2⋆"), ", and can apply it to the output of applying ", em("L1⋆"), ", and ", em("voila"), ", we have rebased two changes onto two other changes."),
  p("Except that mapping through deletions or backwards through insertions loses information. If you insert two characters at position 5, and then another one at position 6 (between the two previously inserted characters), mapping backwards and then forward again through the first insertion will leave you before or after the characters, because the position between them could not be expressed in the coordinate space of a document that did not yet have these characters."),
  p("To fix this, the system uses mapping pipelines that are not just a series of maps, but also keep information about which of those maps are mirror images of each other. When a position going through such a pipeline encounters a map that deletes the content around the position, the system scans ahead in the pipeline looking for a mirror images of that map. If such a map is found, we skip forward to it, and restore the position in the content that is inserted by this map, using the relative offset that the position had in the deleted content. A mirror image of a map that deletes content must insert content with the same shape."),
  h2("Mapping Bias"),
  p("Whenever content gets inserted, a position at the exact insertion point can be meaningfully mapped to two different positions: before the inserted content, or after it. Sometimes the first is appropriate, sometimes the second. The system allows code that maps a position to choose what bias it prefers."),
  p("This is also why the positions associated with a change are labeled. If a change with ", code("from"), " and ", code("to"), " positions, such as deleting or styling a piece of the document, has content inserted directly before or after it, that content should not be included in the change. So ", code("from"), " positions get mapped with a forward bias, and ", code("to"), " positions with a backward bias."),
  p("When a change is mapped through a map that completely contains it, for example when inserting a character at position 5 is mapped through the map created by deleting from position 2 to 10, the whole change is simply dropped, since the context in which it was made no longer exists."),
  h2("Types of Changes"),
  p("An atomic change in ProseMirror is called a ", em("step"), ". Some things that look like single changes from a user interface perspective are actually decomposed into several steps. For example, if you select text and press enter, the editor will generate a ", em("delete"), " step that removes the selected text, and then a ", em("split"), " step that splits the current paragraph."),
  p("These are the step types that exist in ProseMirror:"),
  ul(li(p("", code("addStyle"), " and ", code("removeStyle"), " add and remove inline styling to or from a piece of the document. They take ", code("from"), " and ", code("to"), " positions.")),
     li(p("", code("split"), " splits a node in two. It can be used, for example, to split a paragraph when the user presses enter. It takes a single ", code("at"), " position.")),
     li(p("", code("join"), " joins two adjacent nodes. This only works if they contain the same type of content. It takes ", code("from"), " and ", code("to"), " positions that should refer to the end and start of the nodes to be joined. (This is to make sure that the nodes that were actually intended are being joined. The step is ignored when another node has been inserted between them in the meantime.)")),
     li(p("", code("ancestor"), " is used to change the type of a node and to add or remove nodes above it. It can be used to wrap something in a list, or to convert from a paragraph to a heading. It takes ", code("from"), " and ", code("to"), " positions pointing at the start and end of the node.")),
     li(p("", code("replace"), " replaces a piece of the document with zero or more replacement nodes, and optionally stitches up compatible nodes at the edges of the cut. Its ", code("from"), " and ", code("to"), " positions define the range that should be deleted, and its ", code("at"), " position gives the place where the new nodes should be inserted."))),
  p("The last type is more complex than the other ones, and my initial impulse was to split it up into steps that remove and insert content. But because the position map created by a replace step needs to treat the step as atomic (positions have to be pushed out of ", em("all"), " replaced content), I got better results with making it a single step."),
  h2("Intention"),
  p("An essential property of real-time collaborative systems is that they try to preserve the ", em("intention"), " of a change. Because “merging” of changes happens automatically, without user interaction, it would get very annoying when the changes you make are, through rebasing, reinterpreted in a way that does not match what you were trying to do."),
  p("I've tried to define the steps and the way in which they are rebased in so that rebasing yields unsurprising behavior. Most of the time, changes don't overlap, and thus don't really interact. But when they overlap, we must make sure that their combined effect remains sane."),
  p("Sometimes a change must simply be dropped. When you type into a paragraph, but another user deleted that paragraph before your change goes through, the context in which your input made sense is gone, and inserting it in the place where the paragraph used to be would create a meaningless fragment."),
  p("If you tried to join two lists together, but somebody has added a paragraph between them, your change becomes impossible to execute (you can't join nodes that aren't adjacent), so it is dropped."),
  p("In other cases, a change is modified but stays meaningful. If you made characters 5 to 10 strong, and another user inserted a character at position 7, you end up making characters 5 to 11 strong."),
  p("And finally, some changes can overlap without interacting. If you make a word a link and another user makes it emphasized, both of your changes to that same word can happen in their original form."),
  h2("Offline Work"),
  p("Silently reinterpreting or dropping changes is fine for real-time collaboration, where the feedback is more or less immediate—you see the paragraph that you were editing vanish, and thus know that someone deleted it, and your changes are gone."),
  p("For doing offline work (where you keep editing when not connected) or for a branching type of work flow, where you do a bunch of work and ", em("then"), " merge it with whatever other people have done in the meantime, the model I described here is useless (as is OT). It might silently throw away a lot of work (if its context was deleted), or create a strange mishmash of text when two people edited the same sentence in different ways."),
  p("In cases like this, I think a diff-based approach is more appropriate. You probably can't do automatic merging—you have to identify conflicts had present them to the user to resolve. I.e. you'd do what git does."),
  h2("Undo History"),
  p("How should the undo history work in a collaborative system? The widely accepted answer to that question is that it definitely should ", em("not"), " use a single, shared history. If you undo, the last edit that ", em("you"), " made should be undone, not the last edit in the document."),
  p("This means that the easy way to implement history, which is to simply roll back to a previous state, does not work. The state that is created by undoing your change, if other people's changes have come in after it, is a new one, not seen before."),
  p("To be able to implement this, I had to define changes (steps) in such a way that they can be inverted, producing a new step that represents the change that cancels out the original step."),
  p("ProseMirror's undo history accumulates inverted steps, and also keeps track of all position maps between them and the current document version. These are needed to be able to map the inverted steps to the current document version."),
  p("A downside of this is that if a user has made a change but is now idle while other people are editing the document, the position maps needed to move this user's change to the current document version pile up without bound. To address this, the history periodically ", em("compacts"), " itself, mapping the inverted changes forward so that they start at the current document again. It can then discard the intermediate position maps.")
)
exports.example = example


================================================
FILE: demo/bench/index.html
================================================
<!doctype html>
<meta charset=utf8>
<title>ProseMirror benchmarks</title>

<div id="buttons"></div>

<p><label><input type=checkbox id=profile> Profile</label></p>

<div id="workspace" style="height: 0; overflow: hidden"></div>

<script src="/moduleserve/load.js" data-module="./index" data-require></script>


================================================
FILE: demo/bench/index.js
================================================
const {Fragment} = require("prosemirror-model")
const {doc, blockquote, p} = require("prosemirror-model/test/build")
const {EditorState} = require("prosemirror-state")
const {EditorView} = require("prosemirror-view")
const {history} = require("prosemirror-history")

const {example} = require("./example")
const {typeDoc} = require("./type")
const {mutateDoc} = require("./mutate")

function button(name, run) {
  var dom = document.createElement("button")
  dom.textContent = name
  dom.addEventListener("click", run)
  return dom
}

function group(name, ...buttons) {
  var wrap = document.querySelector("#buttons").appendChild(document.createElement("p"))
  wrap.textContent = name
  wrap.append(document.createElement("br"))
  buttons.forEach(b => wrap.append(" ", b))
}

function run(bench, options) {
  var t0 = Date.now(), steps = 0
  var startState = (options.state || options.view) && EditorState.create({doc: options.doc, plugins: options.plugins})
  var view = options.view && new EditorView(document.querySelector("#workspace"), {state: startState})
  var state, callback = tr => {
    ++steps
    if (state) {
      state = state.applyAction({type: "transform", time: Date.now(), transform: tr})
      if (view) view.updateState(state)
    }
  }
  var profile = document.querySelector("#profile").checked
  if (profile) console.profile(options.name)
  for (var i = 0, e = options.repeat || 1; i < e; i++) {
    state = startState
    bench(options, callback)
  }
  if (profile) console.profileEnd(options.name)
  console.log("'" + options.name + "' took " + (Date.now() - t0) + "ms for " + steps + " steps")
}

group("Type out a document", button("Plain", () => {
  run(typeDoc, {doc: example, name: "Type plain", profile: true, repeat: 6})
}), button("State", () => {
  run(typeDoc, {doc: example, name: "Type with state", profile: true, repeat: 6, state: true})
}), button("State + History", () => {
  run(typeDoc, {doc: example, name: "Type with state + history", profile: true, repeat: 6, state: true, plugins: [history()]})
}), button("View", () => {
  run(typeDoc, {doc: example, name: "Type with view", profile: true, repeat: 6, state: true, view: true})
}))

group("Mutate inside a document", button("small + shallow", () => {
  run(mutateDoc, {doc: doc(p("a"), p("b"), p("c")),
                  pos: 4, n: 100000, name: "Mutate small + shallow"})
}), button("small + deep", () => {
  run(mutateDoc, {doc: doc(p("a"), blockquote(blockquote(blockquote(blockquote(blockquote(blockquote(p("b"))))))), p("c")),
                  pos: 10, n: 100000, name: "Mutate small + deep"})
}), button("large + shallow", () => {
  var d = doc(p("a")), many = []
  for (var i = 0; i < 1000; i++) many.push(d.firstChild)
  run(mutateDoc, {doc: d.copy(Fragment.from(many)),
                  pos: 4, n: 100000, name: "Mutate large + shallow"})
}))


================================================
FILE: demo/bench/mutate.js
================================================
const {Slice, Fragment} = require("prosemirror-model")
const {Transform} = require("prosemirror-transform")

function mutateDoc(options, callback) {
  var doc = options.doc, pos = options.pos, slice = new Slice(Fragment.from(doc.type.schema.text("X")), 0, 0)
  for (var i = 0; i < options.n; i++) {
    var add = new Transform(doc).replace(pos, pos, slice)
    callback(add)
    var rem = new Transform(add.doc).delete(pos, pos + 1)
    callback(rem)
    doc = rem.doc
  }
}
exports.mutateDoc = mutateDoc


================================================
FILE: demo/bench/type.js
================================================
const {Transform} = require("prosemirror-transform")

function typeDoc(options, callback) {
  var example = options.doc, schema = example.type.schema
  var doc = schema.nodes.doc.createAndFill(), pos = 0

  function scan(node, depth) {
    if (node.isText) {
      for (var i = 0; i < node.text.length; i++) {
        var tr = new Transform(doc).replaceRangeWith(pos, pos, schema.text(node.text.charAt(i), node.marks))
        callback(tr)
        doc = tr.doc
        pos++
      }
    } else if (pos < doc.content.size - depth) {
      pos++
      scanContent(node, depth + 1)
      pos++
    } else {
      if (node.isLeaf) {
        var tr = new Transform(doc).replaceRangeWith(pos, pos, node)
        callback(tr)
        doc = tr.doc
        pos += node.nodeSize
      } else {
        var tr = new Transform(doc).replaceRangeWith(pos, pos, node.type.createAndFill())
        callback(tr)
        doc = tr.doc
        pos++
        scanContent(node, depth + 1)
        pos++
      }
    }
  }
  function scanContent(node, depth) {
    node.forEach(child => scan(child, depth))
  }
  scanContent(example, 0)
}
exports.typeDoc = typeDoc


================================================
FILE: demo/demo.css
================================================
body {
  font-family: Georgia;
  margin: 0 1em 2em;
}

textarea {
  width: 100%;
  border: 1px solid silver;
  min-height: 40em;
  padding: 4px 8px;
}

.left, .right {
  width: 50%;
  float: left;
}

.full {
  max-width: 50em;
}

.marked {
  background: #ff6
}

.ProseMirror-menubar-wrapper {
  border: 1px solid silver;
}

.ProseMirror {
  padding: 4px 8px 4px 14px;
  line-height: 1.2;
}


================================================
FILE: demo/demo.ts
================================================
import {Schema, DOMParser} from "prosemirror-model"
import {EditorView} from "prosemirror-view"
import {EditorState} from "prosemirror-state"
import {schema} from "prosemirror-schema-basic"
import {addListNodes} from "prosemirror-schema-list"
import {exampleSetup} from "prosemirror-example-setup"

const demoSchema = new Schema({
  nodes: addListNodes(schema.spec.nodes as any, "paragraph block*", "block"),
  marks: schema.spec.marks
})

let state = EditorState.create({doc: DOMParser.fromSchema(demoSchema).parse(document.querySelector("#content")!),
                                plugins: exampleSetup({schema: demoSchema})})

;(window as any).view = new EditorView(document.querySelector(".full"), {state})


================================================
FILE: demo/index.html
================================================
<!doctype html>

<meta charset="utf-8"/>
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>ProseMirror demo page</title>
<link rel=stylesheet href="demo.css">
<link rel=stylesheet href="parent/view/style/prosemirror.css">
<link rel=stylesheet href="parent/menu/style/menu.css">
<link rel=stylesheet href="parent/example-setup/style/style.css">
<link rel=stylesheet href="parent/gapcursor/style/gapcursor.css">

<h1>ProseMirror demo page</h1>

<div class="full"></div>

<div id=content style="display: none">
<h2>Demonstration Text</h2>

<p>A ProseMirror document is based on a schema, which determines the
kind of elements that may occur in it, and the relation they have to
each other. This one is based on the basic schema, with lists and
tables added. It allows the usual <strong>strong</strong>
and <em>emphasized</em> text, <code>code font</code>,
and <a href="http://marijnhaverbeke.nl">links</a>. There are also
images: <img alt="demo picture" src="img.png">.</p>

<p>On the block level you can have:</p>

<ol>
  <li>Ordered lists (such as this one)</li>
  <li>Bullet lists</li>
  <li>Blockquotes</li>
  <li>Code blocks</li>
  <li>Tables</li>
  <li>Horizontal rules</li>
</ol>

<p>It isn't hard to define your own custom elements, and include them
in your schema. These can be opaque 'leaf' nodes, that the user
manipulates through extra interfaces you provide, or nodes with
regular editable child nodes.</p>

<hr>

<h2>The Model</h2>

<p>Nodes can nest arbitrarily deep. Thus, the document forms a tree,
not dissimilar to the browser's DOM tree.</p>

<p>At the inline level, the model works differently. Each block of
text is a single node containing a flat series of inline elements.
These are serialized as a tree structure when outputting HTML.</p>

<p>Positions in the document are represented as a path (an array of
offsets) through the block tree, and then an offset into the inline
content of the block. Blocks that have no inline content (such as
horizontal rules and HTML blocks) can not have the cursor inside of
them. User-exposed operations on the document preserve the invariant
that there is always at least a single valid cursor position.</p>

<hr>

<h2>Examples</h2>

<blockquote><blockquote><p>We did not see a nested blockquote
yet.</p></blockquote></blockquote>

<pre><code class="lang-markdown">Nor did we see a code block

Note that the content of a code block can't be styled.</code></pre>

<p>This paragraph has<br>a hard break inside of it.</p>
</div>

<script type=module src="_m/demo.js"></script>


================================================
FILE: package.json
================================================
{
  "name": "prosemirror",
  "version": "0.0.0",
  "description": "Structured WYSIWYM editor",
  "license": "MIT",
  "maintainers": [
    {
      "name": "Marijn Haverbeke",
      "email": "marijn@haverbeke.berlin",
      "web": "http://marijnhaverbeke.nl"
    }
  ],
  "repository": {
    "type": "git",
    "url": "git://github.com/prosemirror/prosemirror.git"
  },
  "dependencies": {
    "glob": "^10.3.0",
    "esmoduleserve": "^0.2.0",
    "serve-static": "^1.14.1",
    "@marijn/buildtool": "^1.0.0",
    "@marijn/testtool": "^0.1.0"
  },
  "scripts": {
    "test": "bin/pm test"
  },
  "workspaces": [
    "*"
  ],
  "private": true
}


================================================
FILE: tsconfig.json
================================================
{
  "compilerOptions": {
    "lib": ["es2015", "dom", "scripthost"],
    "types": ["mocha"],
    "stripInternal": true,
    "typeRoots": ["./node_modules/@types"],
    "noUnusedLocals": true,
    "strict": true,
    "target": "es6",
    "module": "es2020",
    "newLine": "lf",
    "moduleResolution": "node",
    "noEmit": true,
    "paths": {
      "prosemirror-model": ["./model/src/index.ts"],
      "prosemirror-schema-basic": ["./schema-basic/src/schema-basic.ts"],
      "prosemirror-schema-list": ["./schema-list/src/schema-list.ts"],
      "prosemirror-test-builder": ["./test-builder/src/index.ts"],
      "prosemirror-transform": ["./transform/src/index.ts"],
      "prosemirror-view": ["./view/src/index.ts"],
      "prosemirror-state": ["./state/src/index.ts"],
      "prosemirror-commands": ["./commands/src/commands.ts"],
      "prosemirror-history": ["./history/src/history.ts"],
      "prosemirror-dropcursor": ["./dropcursor/src/dropcursor.ts"],
      "prosemirror-inputrules": ["./inputrules/src/index.ts"],
      "prosemirror-keymap": ["./keymap/src/keymap.ts"],
      "prosemirror-search": ["./search/src/search.ts"],
      "prosemirror-changeset": ["./changeset/src/changeset.ts"],
      "prosemirror-markdown": ["./markdown/src/markdown.ts"],
      "prosemirror-collab": ["./collab/src/collab.ts"],
      "prosemirror-menu": ["./menu/src/index.ts"],
      "prosemirror-gapcursor": ["./gapcursor/src/index.ts"],
      "prosemirror-example-setup": ["./example-setup/src/index.ts"]
    }
  },
  "include": ["*/src/*.ts", "*/test/*.ts", "demo/demo.ts"]
}