Repository: near/NEPs Branch: master Commit: 7ea639f0ada3 Files: 67 Total size: 1.0 MB Directory structure: gitextract_xo_k9all/ ├── .github/ │ └── workflows/ │ ├── add-to-devrel.yml │ ├── lint.yml │ └── spellcheck.yml ├── .gitignore ├── .markdownlint.json ├── .mlc_config.json ├── CODEOWNERS ├── README.md ├── nep-0000-template.md └── neps/ ├── archive/ │ ├── 0005-access-keys.md │ ├── 0006-bindings.md │ ├── 0008-transaction-refactoring.md │ ├── 0013-system-methods.md │ ├── 0017-execution-outcome.md │ ├── 0018-view-change-method.md │ ├── 0033-economics.md │ ├── 0040-split-states.md │ └── README.md ├── nep-0001.md ├── nep-0021.md ├── nep-0141.md ├── nep-0145.md ├── nep-0148.md ├── nep-0171.md ├── nep-0177.md ├── nep-0178.md ├── nep-0181.md ├── nep-0199.md ├── nep-0245/ │ ├── ApprovalManagement.md │ ├── Enumeration.md │ ├── Events.md │ └── Metadata.md ├── nep-0245.md ├── nep-0256.md ├── nep-0264.md ├── nep-0297.md ├── nep-0300.md ├── nep-0330.md ├── nep-0364.md ├── nep-0366.md ├── nep-0368.md ├── nep-0393.md ├── nep-0399.md ├── nep-0408.md ├── nep-0413.md ├── nep-0418.md ├── nep-0448.md ├── nep-0452.md ├── nep-0455.md ├── nep-0488.md ├── nep-0491.md ├── nep-0492.md ├── nep-0508.md ├── nep-0509.md ├── nep-0514.md ├── nep-0518.md ├── nep-0519.md ├── nep-0536.md ├── nep-0539.md ├── nep-0568.md ├── nep-0584.md ├── nep-0591.md ├── nep-0611.md ├── nep-0616.md ├── nep-0621.md ├── nep-0635.md └── nep-0638.md ================================================ FILE CONTENTS ================================================ ================================================ FILE: .github/workflows/add-to-devrel.yml ================================================ name: 'Add to DevRel Project' on: issues: types: - opened - reopened pull_request_target: types: - opened - reopened jobs: add-to-project: name: Add issue/PR to project runs-on: ubuntu-latest steps: - uses: actions/add-to-project@v1.0.0 with: # add to DevRel Project #117 project-url: https://github.com/orgs/near/projects/117 github-token: ${{ secrets.PROJECT_GH_TOKEN }} ================================================ FILE: .github/workflows/lint.yml ================================================ name: Lint on: pull_request: branches: [master, main] merge_group: concurrency: group: ci-${{ github.ref }}-${{ github.workflow }} cancel-in-progress: true jobs: markdown-lint: name: markdown-lint runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 with: fetch-depth: 0 # lint only changed files - uses: tj-actions/changed-files@v46 id: changed-files with: files: "**/*.md" separator: "," - uses: DavidAnson/markdownlint-cli2-action@v19 if: steps.changed-files.outputs.any_changed == 'true' with: config: .markdownlint.json globs: | ${{ steps.changed-files.outputs.all_changed_files }} separator: "," markdown-link-check: runs-on: ubuntu-latest steps: - uses: actions/checkout@master - uses: gaurav-nelson/github-action-markdown-link-check@v1 with: use-quiet-mode: "yes" # use-verbose-mode: 'yes' config-file: ".mlc_config.json" folder-path: "neps" ================================================ FILE: .github/workflows/spellcheck.yml ================================================ name: spellchecker on: pull_request: branches: - master jobs: misspell: name: runner / misspell runs-on: ubuntu-latest steps: - name: Check out code. uses: actions/checkout@v1 - name: misspell id: check_for_typos uses: reviewdog/action-misspell@v1 with: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} path: "./specs" locale: "US" ================================================ FILE: .gitignore ================================================ /docs .idea .DS_Store .vscode ================================================ FILE: .markdownlint.json ================================================ { "default": true, "MD001": false, "MD013": false, "MD024": { "siblings_only": true }, "MD025": false, "MD033": false, "MD034": false, "MD040": false, "MD041": false, "MD046": false, "whitespace": false } ================================================ FILE: .mlc_config.json ================================================ { "ignorePatterns": [ { "pattern": "^/" }, { "pattern": "^https://codepen.io" }, { "pattern": "^https://stackoverflow.com" }, { "pattern": "^https://www.researchgate.net" }, { "pattern": "^https://pages.near.org/papers/the-official-near-white-paper/" } ], "timeout": "20s", "retryOn429": true, "retryCount": 5, "fallbackRetryDelay": "30s", "aliveStatusCodes": [200, 206] } ================================================ FILE: CODEOWNERS ================================================ * @near/nep-moderators ================================================ FILE: README.md ================================================ # NEAR Protocol Specifications and Standards [![project chat](https://img.shields.io/badge/zulip-join_chat-brightgreen.svg)](https://near.zulipchat.com/#narrow/stream/320497-nep-standards) This repository hosts the current NEAR Protocol specification and standards. This includes the core protocol specification, APIs, contract standards, processes, and workflows. Changes to the protocol specification and standards are called NEAR Enhancement Proposals (NEPs). ## NEPs | NEP # | Title | Author | Status | | ----------------------------------------------------------------- | ----------------------------------------------------------------- | ------------------------------------------------- | ---------- | | [0001](https://github.com/near/NEPs/blob/master/neps/nep-0001.md) | NEP Purpose and Guidelines | @ori-near @bowenwang1996 @austinbaggio @frol | Living | | [0021](https://github.com/near/NEPs/blob/master/neps/nep-0021.md) | Fungible Token Standard (Deprecated) | @evgenykuzyakov | Deprecated | | [0141](https://github.com/near/NEPs/blob/master/neps/nep-0141.md) | Fungible Token Standard | @evgenykuzyakov @oysterpack, @robert-zaremba | Final | | [0145](https://github.com/near/NEPs/blob/master/neps/nep-0145.md) | Storage Management | @evgenykuzyakov | Final | | [0148](https://github.com/near/NEPs/blob/master/neps/nep-0148.md) | Fungible Token Metadata | @robert-zaremba @evgenykuzyakov @oysterpack | Final | | [0171](https://github.com/near/NEPs/blob/master/neps/nep-0171.md) | Non Fungible Token Standard | @mikedotexe @evgenykuzyakov @oysterpack | Final | | [0177](https://github.com/near/NEPs/blob/master/neps/nep-0177.md) | Non Fungible Token Metadata | @chadoh @mikedotexe | Final | | [0178](https://github.com/near/NEPs/blob/master/neps/nep-0178.md) | Non Fungible Token Approval Management | @chadoh @thor314 | Final | | [0181](https://github.com/near/NEPs/blob/master/neps/nep-0181.md) | Non Fungible Token Enumeration | @chadoh @thor314 | Final | | [0199](https://github.com/near/NEPs/blob/master/neps/nep-0199.md) | Non Fungible Token Royalties and Payouts | @thor314 @mattlockyer | Final | | [0245](https://github.com/near/NEPs/blob/master/neps/nep-0245.md) | Multi Token Standard | @zcstarr @riqi @jriemann @marcos.sun | Final | | [0256](https://github.com/near/NEPs/blob/master/neps/nep-0256.md) | Non-Fungible Token Events | @telezhnaya | Final | | [0264](https://github.com/near/NEPs/blob/master/neps/nep-0264.md) | Promise Gas Weights | @austinabell | Final | | [0297](https://github.com/near/NEPs/blob/master/neps/nep-0297.md) | Events Standard | @telezhnaya | Final | | [0300](https://github.com/near/NEPs/blob/master/neps/nep-0300.md) | Fungible Token Events | @telezhnaya | Final | | [0330](https://github.com/near/NEPs/blob/master/neps/nep-0330.md) | Source Metadata | @BenKurrek | Final | | [0364](https://github.com/near/NEPs/blob/master/neps/nep-0364.md) | Efficient signature verification and hashing precompile functions | @blasrodri | Final | | [0366](https://github.com/near/NEPs/blob/master/neps/nep-0366.md) | Meta Transactions | @ilblackdragon @e-uleyskiy @fadeevab | Final | | [0368](https://github.com/near/NEPs/blob/master/neps/nep-0368.md) | Bridge Wallets | @lewis-sqa | Final | | [0393](https://github.com/near/NEPs/blob/master/neps/nep-0393.md) | Sould Bound Token (SBT) | @robert-zaremba | Final | | [0399](https://github.com/near/NEPs/blob/master/neps/nep-0399.md) | Flat Storage | @Longarithm @mzhangmzz | Final | | [0408](https://github.com/near/NEPs/blob/master/neps/nep-0408.md) | Injected Wallet API | @MaximusHaximus @lewis-sqa | Final | | [0413](https://github.com/near/NEPs/blob/master/neps/nep-0413.md) | Near Wallet API - support for signMessage method | @gagdiez @gutsyphilip | Final | | [0418](https://github.com/near/NEPs/blob/master/neps/nep-0418.md) | Remove attached_deposit view panic | @austinabell | Final | | [0448](https://github.com/near/NEPs/blob/master/neps/nep-0448.md) | Zero-balance Accounts | @bowenwang1996 | Final | | [0452](https://github.com/near/NEPs/blob/master/neps/nep-0452.md) | Linkdrop Standard | @benkurrek @miyachi | Final | | [0455](https://github.com/near/NEPs/blob/master/neps/nep-0455.md) | Parameter Compute Costs | @akashin @jakmeier | Final | | [0488](https://github.com/near/NEPs/blob/master/neps/nep-0488.md) | Host Functions for BLS12-381 Curve Operations | @olga24912 | Final | | [0491](https://github.com/near/NEPs/blob/master/neps/nep-0491.md) | Non-Refundable Storage Staking | @jakmeier | Final | | [0492](https://github.com/near/NEPs/blob/master/neps/nep-0492.md) | Restrict creation of Ethereum Addresses | @bowenwang1996 | Final | | [0508](https://github.com/near/NEPs/blob/master/neps/nep-0508.md) | Resharding v2 | @wacban @shreyan-gupta @walnut-the-cat | Final | | [0509](https://github.com/near/NEPs/blob/master/neps/nep-0509.md) | Stateless validation Stage 0 | @robin-near @pugachAG @Longarithm @walnut-the-cat | Final | | [0514](https://github.com/near/NEPs/blob/master/neps/nep-0514.md) | Fewer Block Producer Seats in `testnet` | @nikurt | Final | | [0518](https://github.com/near/NEPs/blob/master/neps/nep-0518.md) | Web3-Compatible Wallets Support | @alexauroradev @birchmd | Final | | [0519](https://github.com/near/NEPs/blob/master/neps/nep-0519.md) | Yield Execution | @akhi3030 @saketh-are | Final | | [0536](https://github.com/near/NEPs/blob/master/neps/nep-0536.md) | Reduce the number of gas refunds | @evgenykuzyakov @bowenwang1996 | Final | | [0539](https://github.com/near/NEPs/blob/master/neps/nep-0539.md) | Cross-Shard Congestion Control | @wacban @jakmeier | Final | | [0568](https://github.com/near/NEPs/blob/master/neps/nep-0568.md) | Resharding V3 | @staffik @Longarithm @Trisfald @marcelo-gonzalez @shreyan-gupta @wacban | Final | | [0584](https://github.com/near/NEPs/blob/master/neps/nep-0584.md) | Cross-shard bandwidth scheduler | @jancionear | Final | | [0591](https://github.com/near/NEPs/blob/master/neps/nep-0591.md) | Global Contracts | @bowenwang1996 @pugachag @stedfn | Final | ## Specification NEAR Specification is under active development. Specification defines how any NEAR client should be connecting, producing blocks, reaching consensus, processing state transitions, using runtime APIs, and implementing smart contract standards as well. ## Standards & Processes Standards refer to various common interfaces and APIs that are used by smart contract developers on top of the NEAR Protocol. For example, such standards include SDK for Rust, API for fungible tokens or how to manage user's social graph. Processes include release process for spec, clients or how standards are updated. ### Contributing #### Expectations Ideas presented ultimately as NEPs will need to be driven by the author through the process. It's an exciting opportunity with a fair amount of responsibility from the contributor(s). Please put care into the details. NEPs that do not present convincing motivation, demonstrate understanding of the impact of the design, or are disingenuous about the drawbacks or alternatives tend to be poorly received. Again, by the time the NEP makes it to the pull request, it has a clear plan and path forward based on the discussions in the governance forum. #### Process Spec changes are ultimately done via pull requests to this repository (formalized process [here](neps/nep-0001.md)). In an effort to keep the pull request clean and readable, please follow these instructions to flesh out an idea. 1. Sign up for the [governance site](https://gov.near.org/) and make a post to the appropriate section. For instance, during the ideation phase of a standard, one might start a new conversation in the [Development » Standards section](https://gov.near.org/c/dev/standards/29) or the [NEP Discussions Forum](https://github.com/near/NEPs/discussions). 2. The forum has comment threading which allows the community and NEAR Collective to ideate, ask questions, wrestle with approaches, etc. If more immediate responses are desired, consider bringing the conversation to [Zulip](https://near.zulipchat.com/#narrow/stream/320497-nep-standards). 3. When the governance conversations have reached a point where a clear plan is evident, create a pull request, using the instructions below. - Clone this repository and create a branch with "my-feature". - Update relevant content in the current specification that are affected by the proposal. - Create a Pull request, using [nep-0000-template.md](nep-0000-template.md) to describe motivation and details of the new Contract or Protocol specification. In the document header, ensure the `Status` is marked as `Draft`, and any relevant discussion links are added to the `DiscussionsTo` section. Use the pull request number padded with zeroes. For instance, the pull request `219` should be created as `neps/nep-0219.md`. - Add your Draft standard to the `NEPs` section of this README.md. This helps advertise your standard via github. - Once complete, submit the pull request for editor review. - The formalization dance begins: - NEP Editors, who are unopinionated shepherds of the process, check document formatting, completeness and adherence to [NEP-0001](neps/nep-0001.md) and approve the pull request. - Once ready, the author updates the NEP status to `Review` allowing further community participation, to address any gaps or clarifications, normally part of the Review PR. - NEP Editors mark the NEP as `Last Call`, allowing a 14 day grace period for any final community feedback. Any unresolved show stoppers roll the state back to `Review`. - NEP Editors mark the NEP as `Final`, marking the standard as complete. The standard should only be updated to correct errata and add non-normative clarifications. Tip: build consensus and integrate feedback. NEPs that have broad support are much more likely to make progress than those that don't receive any comments. Feel free to reach out to the NEP assignee in particular to get help identify stakeholders and obstacles. ================================================ FILE: nep-0000-template.md ================================================ --- NEP: 0 Title: NEP Template Authors: Todd Codrington III Status: Approved DiscussionsTo: https://github.com/nearprotocol/neps/pull/0000 Type: Developer Tools Version: 1.1.0 Created: 2022-03-03 LastUpdated: 2023-03-07 --- [This is a NEP (NEAR Enhancement Proposal) template, as described in [NEP-0001](https://github.com/near/NEPs/blob/master/neps/nep-0001.md). Use this when creating a new NEP. The author should delete or replace all the comments or commented brackets when merging their NEP.] ## Summary [Provide a short human-readable (~200 words) description of the proposal. A reader should get from this section a high-level understanding about the issue this NEP is addressing.] ## Motivation [Explain why this proposal is necessary, how it will benefit the NEAR protocol or community, and what problems it solves. Also describe why the existing protocol specification is inadequate to address the problem that this NEP solves, and what potential use cases or outcomes.] ## Specification [Explain the proposal as if you were teaching it to another developer. This generally means describing the syntax and semantics, naming new concepts, and providing clear examples. The specification needs to include sufficient detail to allow interoperable implementations getting built by following only the provided specification. In cases where it is infeasible to specify all implementation details upfront, broadly describe what they are.] ## Reference Implementation [This technical section is required for Protocol proposals but optional for other categories. A draft implementation should demonstrate a minimal implementation that assists in understanding or implementing this proposal. Explain the design in sufficient detail that: * Its interaction with other features is clear. * Where possible, include a Minimum Viable Interface subsection expressing the required behavior and types in a target programming language. (ie. traits and structs for rust, interfaces and classes for javascript, function signatures and structs for c, etc.) * It is reasonably clear how the feature would be implemented. * Corner cases are dissected by example. * For protocol changes: A link to a draft PR on nearcore that shows how it can be integrated in the current code. It should at least solve the key technical challenges. The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work.] ## Security Implications [Explicitly outline any security concerns in relation to the NEP, and potential ways to resolve or mitigate them. At the very least, well-known relevant threats must be covered, e.g. person-in-the-middle, double-spend, XSS, CSRF, etc.] ## Alternatives [Explain any alternative designs that were considered and the rationale for not choosing them. Why your design is superior?] ## Future possibilities [Describe any natural extensions and evolutions to the NEP proposal, and how they would impact the project. Use this section as a tool to help fully consider all possible interactions with the project in your proposal. This is also a good place to "dump ideas"; if they are out of scope for the NEP but otherwise related. Note that having something written down in the future-possibilities section is not a reason to accept the current or a future NEP. Such notes should be in the section on motivation or rationale in this or subsequent NEPs. The section merely provides additional information.] ## Consequences [This section describes the consequences, after applying the decision. All consequences should be summarized here, not just the "positive" ones. Record any concerns raised throughout the NEP discussion.] ### Positive * p1 ### Neutral * n1 ### Negative * n1 ### Backwards Compatibility [All NEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. Author must explain a proposes to deal with these incompatibilities. Submissions without a sufficient backwards compatibility treatise may be rejected outright.] ## Unresolved Issues (Optional) [Explain any issues that warrant further discussion. Considerations * What parts of the design do you expect to resolve through the NEP process before this gets merged? * What parts of the design do you expect to resolve through the implementation of this feature before stabilization? * What related issues do you consider out of scope for this NEP that could be addressed in the future independently of the solution that comes out of this NEP?] ## Changelog [The changelog section provides historical context for how the NEP developed over time. Initial NEP submission should start with version 1.0.0, and all subsequent NEP extensions must follow [Semantic Versioning](https://semver.org/). Every version should have the benefits and concerns raised during the review. The author does not need to fill out this section for the initial draft. Instead, the assigned reviewers (Subject Matter Experts) should create the first version during the first technical review. After the final public call, the author should then finalize the last version of the decision context.] ### 1.0.0 - Initial Version > Placeholder for the context about when and who approved this NEP version. #### Benefits > List of benefits filled by the Subject Matter Experts while reviewing this version: * Benefit 1 * Benefit 2 #### Concerns > Template for Subject Matter Experts review for this version: > Status: New | Ongoing | Resolved | # | Concern | Resolution | Status | | --: | :------ | :--------- | -----: | | 1 | | | | | 2 | | | | ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/archive/0005-access-keys.md ================================================ - Proposal Code Name: access_keys - Start Date: 2019-07-08 - NEP PR: [nearprotocol/neps#0000](https://github.com/near/NEPs/blob/master/nep-0000-template.md) - Issue(s): [nearprotocol/nearcore#687](https://github.com/nearprotocol/nearcore/issues/687) # Summary Access keys provide limited access to an account. Each access key belongs to some account and identified by a unique (within the account) public key. One account may have large number of access keys. Access keys will replace original account-level public keys. Access keys allow to act on behalf of the account by restricting allowed transactions with the access key permissions. # Motivation Access keys give an ability to use dApps in a secure way without asking the user to sign every transaction in the wallet. By issuing the access key once for the application, the application can now act on behalf of the user in a restricted environment. This enables seamless experience for the user. Access keys also enable a few other use-cases that are discussed in details below. # Guide-level explanation Here are proposed changes for the AccessKey and Account structs. ```rust /// `account_id,public_key` is a key in the state struct AccessKey { /// The nonce for this access key. /// It makes sense for nonce to not start from 0, in case the access key is recreated /// with the same public key, to avoid replaying of old transactions. pub nonce: Nonce, // u64 /// Defines permissions for the AccessKey pub permission: AccessKeyPermission, } /// Defines permissions for AccessKey pub enum AccessKeyPermission { /// Restricts AccessKey to only be used for function calls. FunctionCall(FunctionCallPermission), /// Gives full access to the account. /// NOTE: It's used to replace account-level public keys. FullAccess, } pub struct FunctionCallPermission { /// `Some` amount that can be spent for transaction fees by this access key from the account balance. /// When used, both account balance and the allowance is decreased. /// To change or increase the allowance, the access key can be replaced using SwapKey. /// NOTE: If you reuse the public key, make sure to keep the nonce from the old AccessKey. /// `None` means unlimited allowance. pub allowance: Option, // u128 /// The AccountID of the receiver of the transaction. The access key will restrict transactions to /// only this receiver. pub receiver_id: AccountId, // String /// If `Some`, the access key would be restricted to calling only the given method name. /// `None` means it's restricted to calling the receiver_id contract, but any method name. pub method_name: Option, } /// NOTE: This change removes account-level nonce and public keys. /// Key is `account_id` struct Account { pub balance: Balance(u128), pub code_hash: Hash, /// Storage usage accounts for all access keys pub storage_usage: StorageUsage(u64), /// Last block index at which the storage was paid for. pub storage_paid_at: BlockIndex(u64), } ``` ### Examples #### AccessKey as account-level public key If an AccessKey has the full access to the account and the allowance set to be the max value for u128, then it essentially acts as an account-level public key. Which means we can remove account-level public keys from the account struct and rely only on access keys. An access key example from user `vasya.near` with full access: ```rust /// vasya.near,a123bca2 AccessKey { nonce: 0, permission: AccessKeyPermission::FullAccess, } ``` #### AccessKey for a dApp by a user This is a simple example where a user wants to use some dApp. The user has to authorize this dApp within their wallet, so the dApp knows who the user is, and also can issue simple function call transactions on behalf of this user. To create such AccessKey a dApp generates a new key pair and passes the new public key to the user's wallet in a URL. Then the wallet asks the user to create a new AccessKey with that points to the dApp. User has to explicitly confirm this in the wallet for AccessKey to be created. The new access key is restricted to be only used for the app’s contract_id, but is not restricted for any method name. The user also selects the allowance to some reasonable amount, enough for the application to issue regular transactions. The application might also hint the user about this desired allowance in some way. Now the app can issue function call transactions on behalf of the user’s account towards the app’s contract without requiring the user to sign each transaction. An access key example for chess app from user `vasya.near`: ```rust /// vasya.near,c5d312f3 AccessKey { nonce: 0, permission: AccessKeyPermission::FunctionCall(FunctionCallPermission { // Since the access key is stored on the Chess app front-end, the user has // limited the spending amount to some reasonable, but large enough number. // NOTE: It's needs to be multiplied to decimals, e.g. 10^-18 allowance: Some(1_000_000_000), // This access key restricts access to `chess.app` contract. receiver_id: "chess.app", // Any method name on the `chess.app` contract can be called. method_name: None, }), } ``` #### AccessKey issued by a dApp This is an example where the dApp wants to pay for the user, or it doesn't want to go through the user's sign-in flow. For whatever reason the dApp decided to issue an access key directly for their account. For this to work there should be one account with funds (that dApp controls on the backend) which creates access keys for the users. The difference from the example above is there is only one account (the same for all users) that creates multiple access keys (one per user) towards one other contract (app's contract). To differentiate users the contract has to use the public key of the access key instead of sender's account ID. If the access key wants to support user's identity from the account ID. The contract can provide a public method that links user's account ID with a given public key. Once this is done, a user can request a new access key with the linked public key (sponsored by the app), but it is linked to the user's account ID. There are some caveats with this approach: - The dApp is required to have a backend and to have some sybil resistance for users. It's needed to prevent abuse by bots. - Writing the contract is slightly more complicated, since the contract now needs to handle mapping of the public keys to the account IDs. An access key example for chess app paid by the chess app from `chess.funds` account: ```rust /// chess.funds,2bc2b3b AccessKey { nonce: 0, permission: AccessKeyPermission::FunctionCall(FunctionCallPermission { // Since the access key is given to the user, the developer wants to limit the // the spending amount to some conservative number, since a user might try to drain it. allowance: Some(5_000_000), // This access key restricts access to `chess.app` contract. receiver_id: "chess.app", // Any method name on the `chess.app` contract can be called (but some methods might just ignore this key). method_name: None, }), } ``` #### AccessKey through a proxy This examples demonstrates how to have more granular control on top of built-in access key restrictions. Let's say a user wants to: - limit the number of calls the access key can make per minute - support multiple contracts with the same access key - select which methods name can be called and which can't - transfer funds from the account up to a certain limit - stake from the account, but prevent withdrawing funds To make it work, we need to have a custom logic at every call. We can achieve this by running a portion of a smart contract code before any action. A user can deploy a code on their account and restrict access key to their account and to a method name, e.g. `proxy`. Now this access key will only be able to issue transactions on behalf of the user that goes to the user's contract code and calls method `proxy`. The `proxy` method can find out which access key is used by comparing public keys and verify the request before executing it. E.g. the access key should only be able to call `chess.app` at most 3 times per 20 block and can transfer at most 1M tokens to the `chess.app`. The `proxy` function internally can validate that this access key is used, fetch its config, validate the passed arguments and proxy the transaction. A `proxy` method might take the following arguments for a function call: ```json { "action": "call", "contractId": "chess.app", "methodName": "move", "args": "{...serialized args...}", "amount": 0 } ``` In this case the `action` is `call`, so the function checks the `amount` to be within the withdrawal limit, check that the contract name is `chess.app` and if there were the last 3 calls were not in the last 20 blocks issue an async call to the `chess.app`. The same `proxy` function in theory can handle other actions, e.g. staking or vesting. The benefit of having a proxy function on your own account is that it doesn't require additional receipt, because the account's state and the code are available at the transaction verification time. An example of an access key limited to `proxy` function: ```rust /// vasya.near,3bc2b3b AccessKey { nonce: 0, permission: AccessKeyPermission::FunctionCall(FunctionCallPermission { // Allowance can be large enough, since the user is likely trusting the app. allowance: Some(1_000_000_000), // This access key restricts access to user's account `vasya.near` contract. // Most likely, the contract code can be deployed and upgraded directly from the wallet. receiver_id: "vasya.near", // The method is restricted to proxy, which does all the security checks. method_name: Some("proxy"), }), } ``` # Reference-level explanation - Access keys are stored with the `account_id,public_key` key. Where `account_id` and `public_key` are actual Account ID and public keys, and `,` is a separator. They should be stored on the same shard as the account. - Access keys storage rent should be accounted and paid from the account directly without affecting the allowance. - Access keys allowance can exceed the account balance. - To validate a transaction signed with the AccessKey, we need to first validate the signature, then fetch the Account and the AccessKey, validate that we have enough funds and verify permissions. - Account creation should now create a full access permission access key, instead of public keys within the account. - SwapKey transaction should just replace the old access key with the given new access key. ### Technical changes #### `nonce` on the AccessKey level instead of account level Since access keys can be used by the different people or parties at the same time, we need to be able to have separate nonce for each key instead of a single nonce at the account level. With a single nonce on the account level, there is a high probability that 2 apps would use the same nonce for 2 different transactions and one of this transactions would be rejected. Previously we were ordering transactions by nonce and rejecting transactions with a duplicated or lower nonce. With the access key nonce, we still need to order transactions by nonce, but now we need to group them by `account_id,public_key` key instead of just account_id. To prevent one access key from having a priority on other access keys, we should order transactions by hash when determining which transactions should be added to the block. The suggestion from @nearmax: " We need to spec out here how transactions from different access keys are going to be ordered with respect to each other. For example: 3 access keys (A,B,C) issue 3 transactions each: A1, A2, A3; B1,B2,B3; C1, C2, C3; All these transactions operate on the same state so they need to have an order. First transaction to execute is one of \{A1,B1,C1} that has lowest hash, let's say it is B1. Second transaction to execute is one of \{A1,B2,C1} with lowest hash, etc. " We should also restrict the nonce of the next transaction to be exactly the previous nonce incremented by 1. It will help us with ordering transactions. The transaction ordering should be a separate topic which should also include security for transactions expiration and fork selection. #### `allowance` field Allowance is the amount of tokens the AccessKey can spend from the account balance. When some amount is spent, it's subtracted from both the allowance of the access key and from the account balance. If in some case the user wants to have unlimited allowance for this key, then we have a `None` allowance option. NOTE: In the previous iteration of access keys, we used balance instead of the allowance. But it required to sum up all access keys balances to get the total account balance. It also prevented sharing of the account balance between access keys. #### Permissions Almost all desired use-cases of access keys can be achieved by using the old permissions model. It restricts access keys to only issue function call with no attached tokens. The function calls are restricted to the selected `receiver_id` and potentially restricted to a single `method_name`. Anything non-trivial can be done by the contract that receives this call, e.g. through `proxy` function. To remove public keys from the account, we added a new permission that full access to the account and not limited by the allowance. #### How is `storage_usage` computed? If we use protobuf size to compute the `storage_usage` value, then protobuf might compress `u128` value and it would affect storage usage every time the `allowance` is modified. The best option would be is to change `storage_usage` only when the access key is created or removed. So that changes to the `allowance` value shouldn't change the `storage_usage` value. For this to work, we might need to update the storage computation formula for the access key, e.g. the one that ignores the compressed size of the `allowance` and instead just relies on the 16 bytes of `u128` size. Especially, because we currently don't use the proto size for the storage_usage for the account itself. # Drawbacks Currently the permission model is quite limited to either a function call with one or any method names, or a full access key. But we may add more permissions in the future in order to handle this issue. # Rationale and alternatives ## Alternatives #### More permissions directly on the access key For example we can have multiple method names, multiple contract_id/method_name pairs or different transactions types (e.g. only allow staking transactions). This can be achieved with the contract and a dedicated function that does this control. So to keep the runtime simple and secure we should avoid doing more checks, since they are not accounted for fees. It's also can be achieved if we refactor SignedTransaction to only use method_names instead of oneof body types. #### Balance instead of allowance Allowance enables sharing of a single account balance with multiple access keys. E.g. if you use 5 apps, you can give full allowance to each app instead of splitting balance into 5 parts. It's also easier to work with, than access keys balances. Previously we have AccessKey's balance owner, so the dApp could sponsor users. But it can be achieved by dApps creating access keys from their account, effectively paying for all transactions. #### Not exposing `nonce` on each AccessKey If you use 2 applications at the same time, e.g. a mobile app and a desktop wallet, you might run into a `nonce` collision at the account level, which would cancel one of the transaction. It would happen more frequently with more apps being used. As for the runtime multi nonce handling per account, we need to think and verify security a little more. #### `receiver_id` being an `Option` In the previous design, the `receiver_id` was called `contract_id` and was an option field. But it didn't remove the requirement for the receiver when it was `None`. Instead it was assuming the access key is pointed to the owner's account. We can potentially use `None` to mean unlimited key, and require user to explicitly specify their own account_id if they want to use proxy function. # Unresolved questions #### Transactions ordering and nonce restrictions That question is still unresolved. Whether we should restrict TX nonce to be +1 or not restricting. It's not a blocking change, but it would make sense to do this change with other SignedTransaction security features such as minimum hash of a block header and block expiration. #### Permissions Not clear whether a single pair of `receiver_id`/`method_name` is enough to cover all use-cases at the moment. E.g. if I want to use my account that already has some code on it, e.g. vesting locked account. I can't deploy a new code on it, so I can't use a `proxy` method. # Future possibilities For all use-cases to work we need to add all missing runtime methods that are currently only possible with `SignedTransaction` at the moment, e.g. staking, account creation, public/access key management and code deployment. Next we might consider refactoring stake out of `Account` and also refactor `SignedTransaction` to support text based method names instead of enums. We should also think about storing the same code (by hash) only once instead of storing for each account. Especially, if we adopt `proxy` model. ================================================ FILE: neps/archive/0006-bindings.md ================================================ - Proposal Name: `wasm_bindings` - Start Date: 2019-07-22 - NEP PR: [nearprotocol/neps#0000](https://github.com/near/NEPs/blob/master/nep-0000-template.md) # Summary Wasm bindings, a.k.a imports, are functions that the runtime (a.k.a host) exposes to the Wasm code (a.k.a guest) running on the virtual machine. These functions are arguably the most difficult thing to change in our entire ecosystem, after we have contracts running on our blockchain, since once the bindings change the old smart contracts will not be able to run on the new nodes. Additionally, we need a highly detailed specification of the bindings to be able to write unit tests for our contracts, since currently we only allow integration tests. Currently, writing unit tests is not possible since we cannot have a precise mock of the host in the smart contract unit tests, e.g. we don't know how to mock the range iterator (what does it do when given an empty or inverted range?). In this proposal we give a detailed specification of the functions that we will be relying on for many months to come. ## Motivation The current imports have the following issues: - **Trie API.** The behavior of trie API is currently unspecified. Many things are unclear: what happens when we try iterating over an empty range, what happens if we try accessing a non-existent key, etc. Having a trie API specification is important for being able to create a testing framework for Rust and AssemblyScript smart contracts, since in unit tests the contracts will be running on a mocked implementation of the host; - **Promise API.** Recently we have discussed the changes to our promise mechanics. The schema does not need to change, but the specification now needs to be clarified; - `data_read` currently has mixed functionality -- it can be used both for reading data from trie and to read data from the context. In former it expects pointers to be passed as arguments, in later it expects enum to be passed. It achieves juxtaposition by casting pointer type in enum when needed; - **Economics API.** The functions that provide access to balance and such might need to be added or removed since we now consider splitting attached balance into two. # Specification ## Registers Registers allow the host function to return the data into a buffer located inside the host oppose to the buffer located on the client. A special operation can be used to copy the content of the buffer into the host. Memory pointers can then be used to point either to the memory on the guest or the memory on the host, see below. Benefits: - We can have functions that return values that are not necessarily used, e.g. inserting key-value into a trie can also return the preempted old value, which might not be necessarily used. Previously, if we returned something we would have to pass the blob from host into the guest, even if it is not used; - We can pass blobs of data between host functions without going through the guest, e.g. we can remove the value from the storage and insert it into under a different key; - It makes API cleaner, because we don't need to pass `buffer_len` and `buffer_ptr` as arguments to other functions; - It allows merging certain functions together, see `storage_iter_next`; - This is consistent with other APIs that were created for high performance, e.g. allegedly Ewasm have implemented SNARK-like computations in Wasm by exposing a bignum library through stack-like interface to the guest. The guest can manipulate then with the stack of 256-bit numbers that is located on the host. #### Host → host blob passing The registers can be used to pass the blobs between host functions. For any function that takes a pair of arguments `*_len: u64, *_ptr: u64` this pair is pointing to a region of memory either on the guest or the host: - If `*_len != u64::MAX` it points to the memory on the guest; - If `*_len == u64::MAX` it points to the memory under the register `*_ptr` on the host. For example: `storage_write(u64::MAX, 0, u64::MAX, 1, 2)` -- insert key-value into storage, where key is read from register 0, value is read from register 1, and result is saved to register 2. Note, if some function takes `register_id` then it means this function can copy some data into this register. If `register_id == u64::MAX` then the copying does not happen. This allows some micro-optimizations in the future. Note, we allow multiple registers on the host, identified with `u64` number. The guest does not have to use them in order and can for instance save some blob in register `5000` and another value in register `1`. #### Specification ```rust read_register(register_id: u64, ptr: u64) ``` Writes the entire content from the register `register_id` into the memory of the guest starting with `ptr`. ###### Panics - If the content extends outside the memory allocated to the guest. In Wasmer, it returns `MemoryAccessViolation` error message; - If `register_id` is pointing to unused register returns `InvalidRegisterId` error message. ###### Undefined Behavior - If the content of register extends outside the preallocated memory on the host side, or the pointer points to a wrong location this function will overwrite memory that it is not supposed to overwrite causing an undefined behavior. --- ##### register_len ```rust register_len(register_id: u64) -> u64 ``` Returns the size of the blob stored in the given register. ###### Normal operation - If register is used, then returns the size, which can potentially be zero; - If register is not used, returns `u64::MAX` ## Trie API Here we provide a specification of trie API. After this NEP is merged, the cases where our current implementation does not follow the specification are considered to be bugs that need to be fixed. --- ##### storage_write ```rust storage_write(key_len: u64, key_ptr: u64, value_len: u64, value_ptr: u64, register_id: u64) -> u64 ``` Writes key-value into storage. ###### Normal operation - If key is not in use it inserts the key-value pair and does not modify the register; - If key is in use it inserts the key-value and copies the old value into the `register_id`. ###### Returns - If key was not used returns `0`; - If key was used returns `1`. ###### Panics - If `key_len + key_ptr` or `value_len + value_ptr` exceeds the memory container or points to an unused register it panics with `MemoryAccessViolation`. (When we say that something panics with the given error we mean that we use Wasmer API to create this error and terminate the execution of VM. For mocks of the host that would only cause a non-name panic.) - If returning the preempted value into the registers exceed the memory container it panics with `MemoryAccessViolation`; ###### Current bugs - `External::storage_set` trait can return an error which is then converted to a generic non-descriptive `StorageUpdateError`, [here](https://github.com/nearprotocol/nearcore/blob/942bd7bdbba5fb3403e5c2f1ee3c08963947d0c6/runtime/wasm/src/runtime.rs#L210) however the actual implementation does not return error at all, [see](https://github.com/nearprotocol/nearcore/blob/4773873b3cd680936bf206cebd56bdc3701ddca9/runtime/runtime/src/ext.rs#L95); - Does not return into the registers. --- ##### storage_read ```rust storage_read(key_len: u64, key_ptr: u64, register_id: u64) -> u64 ``` Reads the value stored under the given key. ###### Normal operation - If key is used copies the content of the value into the `register_id`, even if the content is zero bytes; - If key is not present then does not modify the register. ###### Returns - If key was not present returns `0`; - If key was present returns `1`. ###### Panics - If `key_len + key_ptr` exceeds the memory container or points to an unused register it panics with `MemoryAccessViolation`; - If returning the preempted value into the registers exceed the memory container it panics with `MemoryAccessViolation`; ###### Current bugs - This function currently does not exist. --- ##### storage_remove ```rust storage_remove(key_len: u64, key_ptr: u64, register_id: u64) -> u64 ``` Removes the value stored under the given key. ###### Normal operation Very similar to `storage_read`: - If key is used, removes the key-value from the trie and copies the content of the value into the `register_id`, even if the content is zero bytes. - If key is not present then does not modify the register. ###### Returns - If key was not present returns `0`; - If key was present returns `1`. ###### Panics - If `key_len + key_ptr` exceeds the memory container or points to an unused register it panics with `MemoryAccessViolation`; - If the registers exceed the memory limit panics with `MemoryAccessViolation`; - If returning the preempted value into the registers exceed the memory container it panics with `MemoryAccessViolation`; ###### Current bugs - Does not return into the registers. --- ##### storage_has_key ```rust storage_has_key(key_len: u64, key_ptr: u64) -> u64 ``` Checks if there is a key-value pair. ###### Normal operation - If key is used returns `1`, even if the value is zero bytes; - Otherwise returns `0`. ###### Panics - If `key_len + key_ptr` exceeds the memory container it panics with `MemoryAccessViolation`; --- #### storage_iter_prefix ```rust storage_iter_prefix(prefix_len: u64, prefix_ptr: u64) -> u64 ``` Creates an iterator object inside the host. Returns the identifier that uniquely differentiates the given iterator from other iterators that can be simultaneously created. ###### Normal operation - It iterates over the keys that have the provided prefix. The order of iteration is defined by the lexicographic order of the bytes in the keys. If there are no keys, it creates an empty iterator, see below on empty iterators; ###### Panics - If `prefix_len + prefix_ptr` exceeds the memory container it panics with `MemoryAccessViolation`; --- #### storage_iter_range ```rust storage_iter_range(start_len: u64, start_ptr: u64, end_len: u64, end_ptr: u64) -> u64 ``` Similarly to `storage_iter_prefix` creates an iterator object inside the host. ###### Normal operation Unless lexicographically `start < end`, it creates an empty iterator. Iterates over all key-values such that keys are between `start` and `end`, where `start` is inclusive and `end` is exclusive. Note, this definition allows for `start` or `end` keys to not actually exist on the given trie. ###### Panics - If `start_len + start_ptr` or `end_len + end_ptr` exceeds the memory container or points to an unused register it panics with `MemoryAccessViolation`; --- ##### storage_iter_next ```rust storage_iter_next(iterator_id: u64, key_register_id: u64, value_register_id: u64) -> u64 ``` Advances iterator and saves the next key and value in the register. ###### Normal operation - If iterator is not empty (after calling next it points to a key-value), copies the key into `key_register_id` and value into `value_register_id` and returns `1`; - If iterator is empty returns `0`. This allows us to iterate over the keys that have zero bytes stored in values. ###### Panics - If `key_register_id == value_register_id` panics with `MemoryAccessViolation`; - If the registers exceed the memory limit panics with `MemoryAccessViolation`; - If `iterator_id` does not correspond to an existing iterator panics with `InvalidIteratorId` - If between the creation of the iterator and calling `storage_iter_next` any modification to storage was done through `storage_write` or `storage_remove` the iterator is invalidated and the error message is `IteratorWasInvalidated`. ###### Current bugs - Not implemented, currently we have `storage_iter_next` and `data_read` + `DATA_TYPE_STORAGE_ITER` that together fulfill the purpose, but have unspecified behavior. ## Context API Context API mostly provides read-only functions that access current information about the blockchain, the accounts (that originally initiated the chain of cross-contract calls, the immediate contract that called the current one, the account of the current contract), other important information like storage usage. Many of the below functions are currently implemented through `data_read` which allows to read generic context data. However, there is no reason to have `data_read` instead of the specific functions: - `data_read` does not solve forward compatibility. If later we want to add another context function, e.g. `executed_operations` we can just declare it as a new function, instead of encoding it as `DATA_TYPE_EXECUTED_OPERATIONS = 42` which is passed as the first argument to `data_read`; - `data_read` does not help with renaming. If later we decide to rename `signer_account_id` to `originator_id` then one could argue that contracts that rely on `data_read` would not break, while contracts relying on `signer_account_id()` would. However the name change often means the change of the semantics, which means the contracts using this function are no longer safe to execute anyway. However there is one reason to not have `data_read` -- it makes `API` more human-like which is a general direction Wasm APIs, like WASI are moving towards to. --- ##### current_account_id ```rust current_account_id(register_id: u64) ``` Saves the account id of the current contract that we execute into the register. ###### Panics - If the registers exceed the memory limit panics with `MemoryAccessViolation`; --- ##### signer_account_id ```rust signer_account_id(register_id: u64) ``` All contract calls are a result of some transaction that was signed by some account using some access key and submitted into a memory pool (either through the wallet using RPC or by a node itself). This function returns the id of that account. ###### Normal operation - Saves the bytes of the signer account id into the register. ###### Panics - If the registers exceed the memory limit panics with `MemoryAccessViolation`; ###### Current bugs - Currently we conflate `originator_id` and `sender_id` in our code base. --- ##### signer_account_pk ```rust signer_account_pk(register_id: u64) ``` Saves the public key fo the access key that was used by the signer into the register. In rare situations smart contract might want to know the exact access key that was used to send the original transaction, e.g. to increase the allowance or manipulate with the public key. ###### Panics - If the registers exceed the memory limit panics with `MemoryAccessViolation`; ###### Current bugs - Not implemented. --- #### predecessor_account_id ```rust predecessor_account_id(register_id: u64) ``` All contract calls are a result of a receipt, this receipt might be created by a transaction that does function invocation on the contract or another contract as a result of cross-contract call. ###### Normal operation - Saves the bytes of the predecessor account id into the register. ###### Panics - If the registers exceed the memory limit panics with `MemoryAccessViolation`; ###### Current bugs - Not implemented. --- #### input ```rust input(register_id: u64) ``` Reads input to the contract call into the register. Input is expected to be in JSON-format. ###### Normal operation - If input is provided saves the bytes (potentially zero) of input into register. - If input is not provided does not modify the register. ###### Returns - If input was not provided returns `0`; - If input was provided returns `1`; If input is zero bytes returns `1`, too. ###### Panics - If the registers exceed the memory limit panics with `MemoryAccessViolation`; ###### Current bugs - Implemented as part of `data_read`. However there is no reason to have one unified function, like `data_read` that can be used to read all --- #### block_index ```rust block_index() -> u64 ``` Returns the current block index. --- #### storage_usage ```rust storage_usage() -> u64 ``` Returns the number of bytes used by the contract if it was saved to the trie as of the invocation. This includes: - The data written with `storage_*` functions during current and previous execution; - The bytes needed to store the account protobuf and the access keys of the given account. ## Economics API Accounts own certain balance; and each transaction and each receipt have certain amount of balance and prepaid gas attached to them. During the contract execution, the contract has access to the following `u128` values: - `account_balance` -- the balance attached to the given account. This includes the `attached_deposit` that was attached to the transaction; - `attached_deposit` -- the balance that was attached to the call that will be immediately deposited before the contract execution starts; - `prepaid_gas` -- the tokens attached to the call that can be used to pay for the gas; - `used_gas` -- the gas that was already burnt during the contract execution and attached to promises (cannot exceed `prepaid_gas`); If contract execution fails `prepaid_gas - used_gas` is refunded back to `signer_account_id` and `attached_balance` is refunded back to `predecessor_account_id`. The following spec is the same for all functions: ```rust account_balance(balance_ptr: u64) attached_deposit(balance_ptr: u64) ``` -- writes the value into the `u128` variable pointed by `balance_ptr`. ###### Panics - If `balance_ptr + 16` points outside the memory of the guest with `MemoryAccessViolation`; ###### Current bugs - Use a different name; --- ```rust prepaid_gas() -> u64 used_gas() -> u64 ``` ## Math #### random_seed ```rust random_seed(register_id: u64) ``` Returns random seed that can be used for pseudo-random number generation in deterministic way. ###### Panics - If the size of the registers exceed the set limit `MemoryAccessViolation`; --- #### sha256 ```rust sha256(value_len: u64, value_ptr: u64, register_id: u64) ``` Hashes the random sequence of bytes using sha256 and returns it into `register_id`. ###### Panics - If `value_len + value_ptr` points outside the memory or the registers use more memory than the limit with `MemoryAccessViolation`. ###### Current bugs - Current name `hash` is not specific to what hash is being used. - We have `hash32` that largely duplicates the mechanics of `hash` because it returns the first 4 bytes only. --- #### check_ethash ```rust check_ethash(block_number_ptr: u64, header_hash_ptr: u64, nonce: u64, mix_hash_ptr: u64, difficulty_ptr: u64) -> u64 ``` -- verifies hash of the header that we created using [Ethash](https://en.wikipedia.org/wiki/Ethash). Parameters are: - `block_number` -- `u256`/`[u64; 4]`, number of the block on Ethereum blockchain. We use the pointer to the slice of 32 bytes on guest memory; - `header_hash` -- `h256`/`[u8; 32]`, hash of the header on Ethereum blockchain. We use the pointer to the slice of 32 bytes on guest memory; - `nonce` -- `u64`/`h64`/`[u8; 8]`, nonce that was used to find the correct hash, passed as `u64` without pointers; - `mix_hash` -- `h256`/`[u8; 32]`, special hash that avoid griefing attack. We use the pointer to the slice of 32 bytes on guest memory; - `difficulty` -- `u256`/`[u64; 4]`, the difficulty of mining the block. We use the pointer to the slice of 32 bytes on guest memory; ###### Returns - `1` if the Ethash is valid; - `0` otherwise. ###### Panics - If `block_number_ptr + 32` or `header_hash_ptr + 32` or `mix_hash_ptr + 32` or `difficulty_ptr + 32` point outside the memory or registers use more memory than the limit with `MemoryAccessViolation`. ###### Current bugs - `block_number` and `difficulty` are currently exposed as `u64` which are casted to `u256` which breaks Ethereum compatibility; - Currently, we also pass the length together with `header_hash_ptr` and `mix_hash_ptr` which is not necessary since we know their length. ## Promises API ```rust promise_create(account_id_len: u64, account_id_ptr: u64, method_name_len: u64, method_name_ptr: u64, arguments_len: u64, arguments_ptr: u64, amount_ptr: u64, gas: u64) -> u64 ``` Creates a promise that will execute a method on account with given arguments and attaches the given amount. `amount_ptr` point to slices of bytes representing `u128`. ###### Panics - If `account_id_len + account_id_ptr` or `method_name_len + method_name_ptr` or `arguments_len + arguments_ptr` or `amount_ptr + 16` points outside the memory of the guest or host, with `MemoryAccessViolation`. ###### Returns - Index of the new promise that uniquely identifies it within the current execution of the method. --- #### promise_then ```rust promise_then(promise_idx: u64, account_id_len: u64, account_id_ptr: u64, method_name_len: u64, method_name_ptr: u64, arguments_len: u64, arguments_ptr: u64, amount_ptr: u64, gas: u64) -> u64 ``` Attaches the callback that is executed after promise pointed by `promise_idx` is complete. ###### Panics - If `promise_idx` does not correspond to an existing promise panics with `InvalidPromiseIndex`. - If `account_id_len + account_id_ptr` or `method_name_len + method_name_ptr` or `arguments_len + arguments_ptr` or `amount_ptr + 16` points outside the memory of the guest or host, with `MemoryAccessViolation`. ###### Returns - Index of the new promise that uniquely identifies it within the current execution of the method. --- #### promise_and ```rust promise_and(promise_idx_ptr: u64, promise_idx_count: u64) -> u64 ``` Creates a new promise which completes when time all promises passed as arguments complete. Cannot be used with registers. `promise_idx_ptr` points to an array of `u64` elements, with `promise_idx_count` denoting the number of elements. The array contains indices of promises that need to be waited on jointly. ###### Panics - If `promise_ids_ptr + 8 * promise_idx_count` extend outside the guest memory with `MemoryAccessViolation`; - If any of the promises in the array do not correspond to existing promises panics with `InvalidPromiseIndex`. ###### Returns - Index of the new promise that uniquely identifies it within the current execution of the method. --- #### promise_results_count ```rust promise_results_count() -> u64 ``` If the current function is invoked by a callback we can access the execution results of the promises that caused the callback. This function returns the number of complete and incomplete callbacks. Note, we are only going to have incomplete callbacks once we have `promise_or` combinator. ###### Normal execution - If there is only one callback `promise_results_count()` returns `1`; - If there are multiple callbacks (e.g. created through `promise_and`) `promise_results_count()` returns their number. - If the function was called not through the callback `promise_results_count()` returns `0`. --- #### promise_result ```rust promise_result(result_idx: u64, register_id: u64) -> u64 ``` If the current function is invoked by a callback we can access the execution results of the promises that caused the callback. This function returns the result in blob format and places it into the register. ###### Normal execution - If promise result is complete and successful copies its blob into the register; - If promise result is complete and failed or incomplete keeps register unused; ###### Returns - If promise result is not complete returns `0`; - If promise result is complete and successful returns `1`; - If promise result is complete and failed returns `2`. ###### Panics - If `result_idx` does not correspond to an existing result panics with `InvalidResultIndex`. - If copying the blob exhausts the memory limit it panics with `MemoryAccessViolation`. ###### Current bugs - We currently have two separate functions to check for result completion and copy it. --- #### promise_return ```rust promise_return(promise_idx: u64) ``` When promise `promise_idx` finishes executing its result is considered to be the result of the current function. ###### Panics - If `promise_idx` does not correspond to an existing promise panics with `InvalidPromiseIndex`. ###### Current bugs - The current name `return_promise` is inconsistent with the naming convention of Promise API. ## Miscellaneous API #### value_return ```rust value_return(value_len: u64, value_ptr: u64) ``` Sets the blob of data as the return value of the contract. ##### Panics - If `value_len + value_ptr` exceeds the memory container or points to an unused register it panics with `MemoryAccessViolation`; --- ```rust panic() ``` Terminates the execution of the program with panic `GuestPanic`. --- #### log_utf8 ```rust log_utf8(len: u64, ptr: u64) ``` Logs the UTF-8 encoded string. See https://stackoverflow.com/a/5923961 that explains that null termination is not defined through encoding. ###### Normal behavior If `len == u64::MAX` then treats the string as null-terminated with character `'\0'`; ###### Panics - If string extends outside the memory of the guest with `MemoryAccessViolation`; --- #### log_utf16 ```rust log_utf16(len: u64, ptr: u64) ``` Logs the UTF-16 encoded string. `len` is the number of bytes in the string. ###### Normal behavior If `len == u64::MAX` then treats the string as null-terminated with two-byte sequence of `0x00 0x00`. ###### Panics - If string extends outside the memory of the guest with `MemoryAccessViolation`; --- #### abort ```rust abort(msg_ptr: u32, filename_ptr: u32, line: u32, col: u32) ``` Special import kept for compatibility with AssemblyScript contracts. Not called by smart contracts directly, but instead called by the code generated by AssemblyScript. # Future Improvements In the future we can have some of the registers to be on the guest. For instance a guest can tell the host that it has some pre-allocated memory that it wants to be used for the register, e.g. ```rust set_guest_register(register_id: u64, register_ptr: u64, max_register_size: u64) ``` will assign `register_id` to a span of memory on the guest. Host then would also know the size of that buffer on guest and can throw a panic if there is an attempted copying that exceeds the guest register size. ================================================ FILE: neps/archive/0008-transaction-refactoring.md ================================================ - Proposal Name: Batched Transactions - Start Date: 2019-07-22 - NEP PR: [nearprotocol/neps#0008](https://github.com/nearprotocol/neps/pull/8) # Summary Refactor signed transactions and receipts to support batched atomic transactions and data dependency. # Motivation It simplifies account creation, by supporting batching of multiple transactions together instead of creating more complicated transaction types. For example, we want to create a new account with some account balance and one or many access keys, deploy a contract code on it and run an initialization method to restrict access keys permissions for a `proxy` function. To be able to do this now, we need to have a `CreateAccount` transaction with all the parameters of a new account. Then we need to handle it in one operation in a runtime code, which might have duplicated code for executing some WASM code with the rollback conditions. Alternative to this is to execute multiple simple transactions in a batch within the same block. It has to be done in a row without any commits to the state until the entire batch is completed. We propose to support this type of transaction batching to simplify the runtime. Currently callbacks are handled differently from async calls, this NEP simplifies data dependencies and callbacks by unifying them. # Guide-level explanation ### New transaction and receipts Previously, in the runtime to produce a block we first executed new signed transactions and then executed received receipts. It resulted in duplicated code that might be shared across similar actions, e.g. function calls for async calls, callbacks and self-calls. It also increased the complexity of the runtime implementation. This NEP proposes changing it by first converting all signed transactions into receipts and then either execute them immediately before received receipts, or put them into the list of the new receipts to be routed. To achieve this, NEP introduces a new message `Action` that represents one of atomic actions, e.g. a function call. `TransactionBody` is now called just `Transaction`. It contains the list of actions that needs to be performed in a single batch and the information shared across these actions. `Transaction` contains the following fields - `signer_id` is an account ID of the transaction signer. - `public_key` is a public key used to identify the access key and to sign the transaction. - `nonce` is used to deduplicate and order transactions (per access key). - `receiver_id` is the account ID of the destination of this transaction. It's where the generated receipt will be routed for execution. - `action` is the list of actions to perform. An `Action` can be of the following: - `CreateAccount` creates a new account with the `receiver_id` account ID. The action fails if the account already exists. `CreateAccount` also grants permission for all subsequent batched action for the newly created account. For example, permission to deploy code on the new account. Permission details are described in the reference section below. - `DeployContract` deploys given binary wasm code on the account. Either the `receiver_id` equals to the `signer_id`, or the batch of actions has started with `CreateAccount`, which granted that permission. - `FunctionCall` executes a function call on the last deployed contract. The action fails if the account or the code doesn't exist. E.g. if the previous action was `DeployContract`, then the code to execute will be the new deployed contract. `FunctionCall` has `method_name` and `args` to identify method with arguments to call. It also has `gas` and the `deposit`. `gas` is a prepaid amount of gas for this call (the price of gas is determined when a signed transaction is converted to a receipt. `deposit` is the attached deposit balance of NEAR tokens that the contract can spend, e.g. 10 tokens to pay for a crypto-corgi. - `Transfer` transfers the given `deposit` balance of tokens from the predecessor to the receiver. - `Stake` stakes the new total `stake` balance with the given `public_key`. The difference in stake is taken from the account's balance (if the new stake is greater than the current one) at the moment when this action is executed, so it's not prepaid. There is no particular reason to stake on behalf of a newly created account, so we may disallow it. - `DeleteKey` deletes an old `AccessKey` identified by the given `public_key` from the account. Fails if the access key with the given public key doesn't exist. All next batched actions will continue to execute, even if the public key that authorized that transaction was removed. - `AddKey` adds a new given `AccessKey` identified by a new given `public_key` to the account. Fails if an access key with the given public key already exists. We removed `SwapKeyTransaction`, because it can be replaced with 2 batched actions - delete an old key and add a new key. - `DeleteAccount` deletes `receiver_id` account if the account doesn't have enough balance to pay the rent, or the `receiver_id` is the `predecessor_id`. Sends the remaining balance to the `beneficiary_id` account. The new `Receipt` contains the shared information and either one of the receipt actions or a list of actions: - `predecessor_id` the account ID of the immediate previous sender (predecessor) of this receipt. It can be different from the `signer_id` in some cases, e.g. for promises. - `receiver_id` the account ID of the current account, on which we need to perform action(s). - `receipt_id` is a unique ID of this receipt (previously was called `nonce`). It's generated from either the signed transaction or the parent receipt. - `receipt` can be one of 2 types: - `ActionReceipt` is used to perform some actions on the receiver. - `DataReceipt` is used when some data needs to be passed from the predecessor to the receiver, e.g. an execution result. To support promises and callbacks we introduce a concept of cross-shard data sharing with dependencies. Each `ActionReceipt` may have a list of input `data_id`. The execution will not start until all required inputs are received. Once the execution completes and if there is `output_data_id`, it produces a `DataReceipt` that will be routed to the `output_receiver_id`. `ActionReceipt` contains the following fields: - `signer_id` the account ID of the signer, who signed the transaction. - `signer_public_key` the public key that the signer used to sign the original signed transaction. - `output_data_id` is the data ID to create DataReceipt. If it's absent, then the `DataReceipt` is not created. - `output_receiver_id` is the account ID of the data receiver. It's needed to route `DataReceipt`. It's absent if the DataReceipt is not needed. - `input_data_id` is the list of data IDs that are required for the execution of the `ActionReceipt`. If some of data IDs is not available when the receipt is received, then the `ActionReceipt` is postponed until all data is available. Once the last `DataReceipt` for the required input data arrives, the action receipt execution is triggered. - `action` is the list of actions to execute. The execution doesn't need to validate permissions of the actions, but need to fail in some cases. E.g. when the receiver's account doesn't exist and the action acts on the account, or when the action is a function call and the code is not present. `DataReceipt` contains the following fields: - `data_id` is the data ID to be used as an input. - `success` is true if the `ActionReceipt` that generated this `DataReceipt` finished the execution without any failures. - `data` is the binary data that is returned from the last action of the `ActionReceipt`. Right now, it's empty for all actions except for function calls. For function calls the data is the result of the code execution. But in the future we might introduce non-contract state reads. Data should be stored at the same shard as the receiver's account, even if the receiver's account doesn't exist. ### Refunds In case an `ActionReceipt` execution fails the runtime can generate a refund. We've removed `refund_account_id` from receipts, because the account IDs for refunds can be determined from the `signer_id` and `predecessor_id` in the `ActionReceipt`. All unused gas and action fees (also measured in gas) are always refunded back to the `signer_id`, because fees are always prepaid by the signer. The gas is converted into tokens using the `gas_price`. The deposit balances from `FunctionCall` and `Transfer` are refunded back to the `predecessor_id`, because they were deducted from predecessor's account balance. It's also important to note that the account ID of predecessor for refund receipts is `system`. It's done to prevent refund loops, e.g. when the account to receive the refund was deleted before the refund arrives. In this case the refund is burned. If the function call action with the attached `deposit` fails in the middle of the execution, then 2 refund receipts can be generated, one for the unused gas and one for the deposits. The runtime should combine them into one receipt if `signer_id` and `predecessor_id` is the same. Example of a receipt for a refund of `42000` atto-tokens to `vasya.near`: ```json { "predecessor_id": "system", "receiver_id": "vasya.near", "receipt_id": ..., "action": { "signer_id": "vasya.near", "signer_public_key": ..., "gas_price": "3", "output_data_id": null, "output_receiver_id": null, "input_data_id": [], "action": [ { "transfer": { "deposit": "42000" } } ] } } ``` ### Examples #### Account Creation To create a new account we can create a new `Transaction`: ```json { "signer_id": "vasya.near", "public_key": ..., "nonce": 42, "receiver_id": "vitalik.vasya.near", "action": [ { "create_account": { } }, { "transfer": { "deposit": "19231293123" } }, { "deploy_contract": { "code": ... } }, { "add_key": { "public_key": ..., "access_key": ... } }, { "function_call": { "method_name": "init", "args": ..., "gas": 20000, "deposit": "0" } } ] } ``` This transaction is sent from `vasya.near` signed with a `public_key`. The receiver is `vitalik.vasya.near`, which is a new account id. The transaction contains a batch of actions. First we create the account, then we transfer a few tokens to the newly created account, then we deploy code on the new account, add a new access key with some given public key, and as a final action initializing the deployed code by calling a method `init` with some arguments. For this transaction to work `vasya.near` needs to have enough balance on the account cover gas and deposits for all actions at once. Every action has some associated action gas fee with it. While `transfer` and `function_call` actions need additional balance for deposits and gas (for executions and promises). Once we validated and subtracted the total amount from `vasya.near` account, this transaction is transformed into a `Receipt`: ```json { "predecessor_id": "vasya.near", "receiver_id": "vitalik.vasya.near", "receipt_id": ..., "action": { "signer_id": "vasya.near", "signer_public_key": ..., "gas_price": "3", "output_data_id": null, "output_receiver_id": null, "input_data_id": [], "action": [...] } } ``` In this example the gas price at the moment when the transaction was processed was 3 per gas. This receipt will be sent to `vitalik.vasya.near`'s shard to be executed. In case the `vitalik.vasya.near` account already exists, the execution will fail and some amount of prepaid_fees will be refunded back to `vasya.near`. If the account creation receipt succeeds, it wouldn't create a `DataReceipt`, because `output_data_id` is `null`. But it will generate a refund receipt for the unused portion of prepaid function call `gas`. #### Deploy code example Deploying code with initialization is pretty similar to creating account, except you can't deploy code on someone else account. So the transaction's `receiver_id` has to be the same as the `signer_id`. #### Simple promise with callback Let's say the transaction contained a single action which is a function call to `a.contract.near`. It created a new promise `b.contract.near` and added a callback to itself. Once the execution completes it will result in the following new receipts: The receipt for the new promise towards `b.contract.near` ```json { "predecessor_id": "a.contract.near", "receiver_id": "b.contract.near", "receipt_id": ..., "action": { "signer_id": "vasya.near", "signer_public_key": ..., "gas_price": "3", "output_data_id": "data_123_1", "output_receiver_id": "a.contract.near", "input_data_id": [], "action": [ { "function_call": { "method_name": "sum", "args": ..., "gas": 10000, "deposit": "0" } } ] } } ``` Interesting details: - `signer_id` is still `vasya.near`, because it's the account that initialized the transaction, but not the creator of the promise. - `output_data_id` contains some unique data ID. In this example we used `data_123_1`. - `output_receiver_id` indicates where to route the result of the execution. The other receipt is for the callback which will stay in the same shard. ```json { "predecessor_id": "a.contract.near", "receiver_id": "a.contract.near", "receipt_id": ..., "action": { "signer_id": "vasya.near", "signer_public_key": ..., "gas_price": "3", "output_data_id": null, "output_receiver_id": null, "input_data_id": ["data_123_1"], "action": [ { "function_call": { "method_name": "process_sum", "args": ..., "gas": 10000, "deposit": "0" } } ] } } ``` It looks very similar to the new promise, but instead of `output_data_id` it has an `input_data_id`. This action receipt will be postponed until the other receipt is routed, executed and generated a data receipt. Once the new promise receipt is successfully executed, it will generate the following receipt: ```json { "predecessor_id": "b.contract.near", "receiver_id": "a.contract.near", "receipt_id": ..., "data": { "data_id": "data_123_1", "success": true, "data": ... } } ``` It contains the data ID `data_123_1` and routed to the `a.contract.near`. Let's say the callback receipt was processed and postponed, then this data receipt will trigger execution of the callback receipt, because the all input data is now available. #### Remote callback with 2 joined promises, with a callback on itself Let's say `a.contract.near` wants to call `b.contract.near` and `c.contract.near`, and send the result to `d.contract.near` for joining before processing the result on itself. It will generate 2 receipts for new promises, 1 receipt for the remote callback and 1 receipt for the callback on itself. Part of the receipt (#1) for the promise towards `b.contract.near`: ``` ... "output_data_id": "data_123_b", "output_receiver_id": "d.contract.near", "input_data_id": [], ... ``` Part of the receipt (#2) for the promise towards `c.contract.near`: ``` ... "output_data_id": "data_321_c", "output_receiver_id": "d.contract.near", "input_data_id": [], ... ``` The receipt (#3) for the remote callback that has to be executed on `d.contract.near` with data from `b.contract.near` and `c.contract.near`: ```json { "predecessor_id": "a.contract.near", "receiver_id": "d.contract.near", "receipt_id": ..., "action": { "signer_id": "vasya.near", "signer_public_key": ..., "gas_price": "3", "output_data_id": "bla_543", "output_receiver_id": "a.contract.near", "input_data_id": ["data_123_b", "data_321_c"], "action": [ { "function_call": { "method_name": "join_data", "args": ..., "gas": 10000, "deposit": "0" } } ] } } ``` It also has the `output_data_id` and `output_receiver_id` that is specified back towards `a.contract.near`. And finally the part of the receipt (#4) for the local callback on `a.contract.near`: ``` ... "output_data_id": null, "output_receiver_id": null, "input_data_id": ["bla_543"], ... ``` For all of this to execute the first 3 receipts needs to go to the corresponding shards and be processed. If for some reason the data arrived before the corresponding action receipt, then this data will be hold there until the action receipt arrives. An example for this is if the receipt #3 is delayed for some reason, while the receipt #2 was processed and generated a data receipt towards `d.contract.near` which arrived before #3. Also if any of the function calls fail, the receipt still going to generate a new `DataReceipt` because it has `output_data_id` and `output_receiver_id`. Here is an example of a DataReceipt for a failed execution: ```json { "predecessor_id": "b.contract.near", "receiver_id": "d.contract.near", "receipt_id": ..., "data": { "data_id": "data_123_b", "success": false, "data": null } } ``` #### Swap Key example Since there are no swap key action, we can just batch 2 actions together. One for adding a new key and one for deleting the old key. The actual order is not important if the public keys are different, but if the public key is the same then you need to first delete the old key and only after this add a new key. # Reference-level explanation ### Updated protobufs ##### public_key.proto ```proto syntax = "proto3"; message PublicKey { enum KeyType { ED25519 = 0; } KeyType key_type = 1; bytes data = 2; } ``` ##### signed_transaction.proto ```proto syntax = "proto3"; import "access_key.proto"; import "public_key.proto"; import "uint128.proto"; message Action { message CreateAccount { // empty } message DeployContract { // Binary wasm code bytes code = 1; } message FunctionCall { string method_name = 1; bytes args = 2; uint64 gas = 3; Uint128 deposit = 4; } message Transfer { Uint128 deposit = 1; } message Stake { // New total stake Uint128 stake = 1; PublicKey public_key = 2; } message AddKey { PublicKey public_key = 1; AccessKey access_key = 2; } message DeleteKey { PublicKey public_key = 1; } message DeleteAccount { // The account ID which would receive the remaining funds. string beneficiary_id = 1; } oneof action { CreateAccount create_account = 1; DeployContract deploy_contract = 2; FunctionCall function_call = 3; Transfer transfer = 4; Stake stake = 5; AddKey add_key = 6; DeleteKey delete_key = 7; DeleteAccount delete_account = 8; } } message Transaction { string signer_id = 1; PublicKey public_key = 2; uint64 nonce = 3; string receiver_id = 4; repeated Action actions = 5; } message SignedTransaction { bytes signature = 1; Transaction transaction = 2; } ``` ##### receipt.proto ```proto syntax = "proto3"; import "public_key.proto"; import "signed_transaction.proto"; import "uint128.proto"; import "wrappers.proto"; message DataReceipt { bytes data_id = 1; google.protobuf.BytesValue data = 2; } message ActionReceipt { message DataReceiver { bytes data_id = 1; string receiver_id = 2; } string signer_id = 1; PublicKey signer_public_key = 2; // The price of gas is determined when the original SignedTransaction is // converted into the Receipt. It's used for refunds. Uint128 gas_price = 3; // List of data receivers where to route the output data // (e.g. result of execution) repeated DataReceiver output_data_receivers = 4; // Ordered list of data ID to provide as input results. repeated bytes input_data_ids = 5; repeated Action actions = 6; } message Receipt { string predecessor_id = 1; string receiver_id = 2; bytes receipt_id = 3; oneof receipt { ActionReceipt action = 4; DataReceipt data = 5; } } ``` ### Validation and Permissions To validate `SignedTransaction` we need to do the following: - verify transaction hash against signature and the given public key - verify `signed_id` is a valid account ID - verify `receiver_id` is a valid account ID - fetch account for the given `signed_id` - fetch access key for the given `signed_id` and `public_key` - verify access key `nonce` - get the current price of gas - compute total required balance for the transaction, including action fees (in gas), deposits and prepaid gas. - verify account balance is larger than required balance. - verify actions are allowed by the access key permissions, e.g. if the access key only allows function call, then need to verify receiver, method name and allowance. Before we convert a `Transaction` to a new `ActionReceipt`, we don't need to validate permissions of the actions or their order. It's checked during `ActionReceipt` execution. `ActionReceipt` doesn't need to be validated before we start executing it. The actions in the `ActionReceipt` are executed in given order. Each action has to check for the validity before execution. Since `CreateAccount` gives permissions to perform actions on the new account, like it's your account, we introduce temporary variable `actor_id`. At the beginning of the execution `actor_id` is set to the value of `predecessor_id`. Validation rules for actions: - `CreateAccount` - check the account `receiver_id` doesn't exist - `DeployContract`, `Stake`, `AddKey`, `DeleteKey` - check the account `receiver_id` exists - check `actor_id` equals to `receiver_id` - `FunctionCall`, `Transfer` - check the account `receiver_id` exists When `CreateAccount` completes, the `actor_id` changes to `receiver_id`. NOTE: When we implement `DeleteAccount` action, its completion will change `actor_id` back to `predecessor_id`. Once validated, each action might still do some additional checks, e.g. `FunctionCall` might check that the code exists and `method_name` is valid. ### `DataReceipt` generation rules If `ActionReceipt` doesn't have `output_data_id` and `output_receiver_id`, then `DataReceipt` is not generated. Otherwise, `DataReceipt` depends on the last action of `ActionReceipt`. There are 4 different outcomes: 1. Last action is invalid, failed or the execution stopped on some previous action. - `DataReceipt` is generated - `data_id` is set to the value of `output_data_id` from the `ActionReceipt` - `success` is set to `false` - `data` is set to `null` 2. Last action is valid and finished successfully, but it's not a `FunctionCall`. Or a `FunctionCall`, that returned no value. - `DataReceipt` is generated - `data_id` is set to the value of `output_data_id` from the `ActionReceipt` - `success` is set to `true` - `data` is set to `null` 3. Last action is `FunctionCall`, and the result of the execution is some value. - `DataReceipt` is generated - `data_id` is set to the value of `output_data_id` from the `ActionReceipt` - `success` is set to `true` - `data` is set to the bytes of the returned value 4. Last action is `FunctionCall`, and the result of the execution is a promise ID - `DataReceipt` is NOT generated, because we don't have the value for the execution. - Instead we should modify the `ActionReceipt` generated for the returned promise ID. - In this receipt the `output_data_id` should be set to the `output_data_id` of the action receipt that we just finished executed. - `output_receiver_id` is set the same way as `output_data_id` described above. #### Example for the case #4 A user called contract `a.app`, which called `b.app` and expect a callback to `a.app`. So `a.app` generated 2 receipts: Towards `b.app`: ``` ... "receiver_id": "b.app", ... "output_data_id": "data_a", "output_receiver_id": "a.app", "input_data_id": [], ... ``` Towards itself: ``` ... "receiver_id": "a.app", ... "output_data_id": "null", "output_receiver_id": "null", "input_data_id": ["data_a"], ... ``` Now let's say `b.app` doesn't actually do the work, but it's just a middleman that charges some fees before redirecting the work to the actual contract `c.app`. In this case `b.app` creates a new promise by calling `c.app` and returns it instead of data. This triggers the case #4, so it doesn't generate the data receipt yet, instead it creates an action receipt which would look like that: ``` ... "receiver_id": "c.app", ... "output_data_id": "data_a", "output_receiver_id": "a.app", "input_data_id": [], ... ``` Once it completes, it would send a data receipt to `a.app` (unless `c.app` is a middleman as well). But let's say `b.app` doesn't want to reveal it's a middleman. In this case it would call `c.app`, but instead of returning data directly to `a.app`, `b.app` wants to wrap the result into some nice wrapper. Then instead of returning the promise to `c.app`, `b.app` would attach a callback to itself and return the promise ID of that callback. Here is how it would look: Towards `c.app`: ``` ... "receiver_id": "c.app", ... "output_data_id": "data_b", "output_receiver_id": "b.app", "input_data_id": [], ... ``` So when the callback receipt first generated, it looks like this: ``` ... "receiver_id": "b.app", ... "output_data_id": "null", "output_receiver_id": "null", "input_data_id": ["data_b"], ... ``` But once, its promise ID is returned with `promise_return`, it is updated to return data towards `a.app`: ``` ... "receiver_id": "b.app", ... "output_data_id": "data_a", "output_receiver_id": "a.app", "input_data_id": ["data_b"], ... ``` ### Data storage We should maintain the following persistent maps per account (`receiver_id`) - Received data: `data_id -> (success, data)` - Postponed receipts: `receipt_id -> Receipt` - Pending input data: `data_id -> receipt_id` When `ActionReceipt` is received, the runtime iterates through the list of `input_data_id`. If `input_data_id` is not present in the received data map, then a pair `(input_data_id, receipt_id)` is added to pending input data map and the receipt marked as postponed. At the end of the iteration if the receipt is marked as postponed, then it's added to map of postponed receipts keyed by `receipt_id`. If all `input_data_id`s are available in the received data, then `ActionReceipt` is executed. When `DataReceipt` is received, a pair `(data_id, (success, data))` is added to the received data map. Then the runtime checks if `data_id` is present in the pending input data. If it's present, then `data_id` is removed from the pending input data and the corresponding `ActionReceipt` is checked again (see above). NOTE: we can optimize by not storing `data_id` in the received data map when the pending input data is present and it was the final input data item in the receipt. When `ActionReceipt` is executed, the runtime deletes all `input_data_id` from the received data map. The `receipt_id` is deleted from the postponed receipts map (if present). ### TODO Receipt execution - input data is available to all function calls in the batched actions - TODODO # Future possibilities - We can add `or` based data selector, so data storage can be affected. ================================================ FILE: neps/archive/0013-system-methods.md ================================================ - Proposal Name: System methods in runtime API - Start Date: 2019-09-03 - NEP PR: [nearprotocol/neps#0013](https://github.com/nearprotocol/neps/pull/0013) # Summary Adds new ability for contracts to perform some system functions: - create new accounts (with possible code deploy and initialization) - deploy new code (or redeploying code for upgrades) - batched function calls - transfer money - stake - add key - delete key - delete account # Motivation Contracts should have the ability to create new accounts, transfer money without calling code and stake. It will enable full functionality of contract-based accounts. # Reference We introduce additional promise APIs to support batched actions. Firstly, we enable ability to create empty promises without any action. They act similarly to traditional promises, but don't contain function call action. Secondly, we add API to append individual actions to promises. For example we can create a promise with a function_call first using `promise_create` and then attach a transfer action on top of this promise. So the transfer will only deposit tokens if the function call succeeds. Another example is how we create accounts now using batched actions. To create a new account, we create a transaction with the following actions: `create_account`, `transfer`, `add_key`. It creates a new account, deposit some funds on it and the adds a new key. For more examples see NEP#8: https://github.com/nearprotocol/NEPs/pull/8/files?short_path=15b6752#diff-15b6752ec7d78e7b85b8c7de4a19cbd4 **NOTE: The existing promise API is a special case of the batched promise API.** - Calling `promise_batch_create` and then `promise_batch_action_function_call` will produce the same promise as calling `promise_create` directly. - Calling `promise_batch_then` and then `promise_batch_action_function_call` will produce the same promise as calling `promise_then` directly. ## Promises API #### promise_batch_create ```rust promise_batch_create(account_id_len: u64, account_id_ptr: u64) -> u64 ``` Creates a new promise towards given `account_id` without any actions attached to it. ###### Panics - If `account_id_len + account_id_ptr` points outside the memory of the guest or host, with `MemoryAccessViolation`. ###### Returns - Index of the new promise that uniquely identifies it within the current execution of the method. --- #### promise_batch_then ```rust promise_batch_then(promise_idx: u64, account_id_len: u64, account_id_ptr: u64) -> u64 ``` Attaches a new empty promise that is executed after promise pointed by `promise_idx` is complete. ###### Panics - If `promise_idx` does not correspond to an existing promise panics with `InvalidPromiseIndex`. - If `account_id_len + account_id_ptr` points outside the memory of the guest or host, with `MemoryAccessViolation`. ###### Returns - Index of the new promise that uniquely identifies it within the current execution of the method. --- ##### promise_batch_action_create_account ```rust promise_batch_action_create_account(promise_idx: u64) ``` Appends `CreateAccount` action to the batch of actions for the given promise pointed by `promise_idx`. Details for the action: https://github.com/nearprotocol/NEPs/pull/8/files#diff-15b6752ec7d78e7b85b8c7de4a19cbd4R48 ###### Panics - If `promise_idx` does not correspond to an existing promise panics with `InvalidPromiseIndex`. - If the promise pointed by the `promise_idx` is an ephemeral promise created by `promise_and`. --- #### promise_batch_action_deploy_contract ```rust promise_batch_action_deploy_contract(promise_idx: u64, code_len: u64, code_ptr: u64) ``` Appends `DeployContract` action to the batch of actions for the given promise pointed by `promise_idx`. Details for the action: https://github.com/nearprotocol/NEPs/pull/8/files#diff-15b6752ec7d78e7b85b8c7de4a19cbd4R49 ###### Panics - If `promise_idx` does not correspond to an existing promise panics with `InvalidPromiseIndex`. - If the promise pointed by the `promise_idx` is an ephemeral promise created by `promise_and`. - If `code_len + code_ptr` points outside the memory of the guest or host, with `MemoryAccessViolation`. --- #### promise_batch_action_function_call ```rust promise_batch_action_function_call(promise_idx: u64, method_name_len: u64, method_name_ptr: u64, arguments_len: u64, arguments_ptr: u64, amount_ptr: u64, gas: u64) ``` Appends `FunctionCall` action to the batch of actions for the given promise pointed by `promise_idx`. Details for the action: https://github.com/nearprotocol/NEPs/pull/8/files#diff-15b6752ec7d78e7b85b8c7de4a19cbd4R50 *NOTE: Calling `promise_batch_create` and then `promise_batch_action_function_call` will produce the same promise as calling `promise_create` directly.* ###### Panics - If `promise_idx` does not correspond to an existing promise panics with `InvalidPromiseIndex`. - If the promise pointed by the `promise_idx` is an ephemeral promise created by `promise_and`. - If `account_id_len + account_id_ptr` or `method_name_len + method_name_ptr` or `arguments_len + arguments_ptr` or `amount_ptr + 16` points outside the memory of the guest or host, with `MemoryAccessViolation`. --- #### promise_batch_action_transfer ```rust promise_batch_action_transfer(promise_idx: u64, amount_ptr: u64) ``` Appends `Transfer` action to the batch of actions for the given promise pointed by `promise_idx`. Details for the action: https://github.com/nearprotocol/NEPs/pull/8/files#diff-15b6752ec7d78e7b85b8c7de4a19cbd4R51 ###### Panics - If `promise_idx` does not correspond to an existing promise panics with `InvalidPromiseIndex`. - If the promise pointed by the `promise_idx` is an ephemeral promise created by `promise_and`. - If `amount_ptr + 16` points outside the memory of the guest or host, with `MemoryAccessViolation`. --- #### promise_batch_action_stake ```rust promise_batch_action_stake(promise_idx: u64, amount_ptr: u64, public_key_len: u64, public_key_ptr: u64) ``` Appends `Stake` action to the batch of actions for the given promise pointed by `promise_idx`. Details for the action: https://github.com/nearprotocol/NEPs/pull/8/files#diff-15b6752ec7d78e7b85b8c7de4a19cbd4R52 ###### Panics - If `promise_idx` does not correspond to an existing promise panics with `InvalidPromiseIndex`. - If the promise pointed by the `promise_idx` is an ephemeral promise created by `promise_and`. - If the given public key is not a valid public key (e.g. wrong length) `InvalidPublicKey`. - If `amount_ptr + 16` or `public_key_len + public_key_ptr` points outside the memory of the guest or host, with `MemoryAccessViolation`. --- #### promise_batch_action_add_key_with_full_access ```rust promise_batch_action_add_key_with_full_access(promise_idx: u64, public_key_len: u64, public_key_ptr: u64, nonce: u64) ``` Appends `AddKey` action to the batch of actions for the given promise pointed by `promise_idx`. Details for the action: https://github.com/nearprotocol/NEPs/pull/8/files#diff-15b6752ec7d78e7b85b8c7de4a19cbd4R54 The access key will have `FullAccess` permission, details: [0005-access-keys.md#guide-level-explanation](click here) ###### Panics - If `promise_idx` does not correspond to an existing promise panics with `InvalidPromiseIndex`. - If the promise pointed by the `promise_idx` is an ephemeral promise created by `promise_and`. - If the given public key is not a valid public key (e.g. wrong length) `InvalidPublicKey`. - If `public_key_len + public_key_ptr` points outside the memory of the guest or host, with `MemoryAccessViolation`. --- #### promise_batch_action_add_key_with_function_call ```rust promise_batch_action_add_key_with_function_call(promise_idx: u64, public_key_len: u64, public_key_ptr: u64, nonce: u64, allowance_ptr: u64, receiver_id_len: u64, receiver_id_ptr: u64, method_names_len: u64, method_names_ptr: u64) ``` Appends `AddKey` action to the batch of actions for the given promise pointed by `promise_idx`. Details for the action: https://github.com/nearprotocol/NEPs/pull/8/files#diff-156752ec7d78e7b85b8c7de4a19cbd4R54 The access key will have `FunctionCall` permission, details: [0005-access-keys.md#guide-level-explanation](click here) - If the `allowance` value (not the pointer) is `0`, the allowance is set to `None` (which means unlimited allowance). And positive value represents a `Some(...)` allowance. - Given `method_names` is a `utf-8` string with `,` used as a separator. The vm will split the given string into a vector of strings. ###### Panics - If `promise_idx` does not correspond to an existing promise panics with `InvalidPromiseIndex`. - If the promise pointed by the `promise_idx` is an ephemeral promise created by `promise_and`. - If the given public key is not a valid public key (e.g. wrong length) `InvalidPublicKey`. - if `method_names` is not a valid `utf-8` string, fails with `BadUTF8`. - If `public_key_len + public_key_ptr`, `allowance_ptr + 16`, `receiver_id_len + receiver_id_ptr` or `method_names_len + method_names_ptr` points outside the memory of the guest or host, with `MemoryAccessViolation`. --- #### promise_batch_action_delete_key ```rust promise_batch_action_delete_key(promise_idx: u64, public_key_len: u64, public_key_ptr: u64) ``` Appends `DeleteKey` action to the batch of actions for the given promise pointed by `promise_idx`. Details for the action: https://github.com/nearprotocol/NEPs/pull/8/files#diff-15b6752ec7d78e7b85b8c7de4a19cbd4R55 ###### Panics - If `promise_idx` does not correspond to an existing promise panics with `InvalidPromiseIndex`. - If the promise pointed by the `promise_idx` is an ephemeral promise created by `promise_and`. - If the given public key is not a valid public key (e.g. wrong length) `InvalidPublicKey`. - If `public_key_len + public_key_ptr` points outside the memory of the guest or host, with `MemoryAccessViolation`. --- #### promise_batch_action_delete_account ```rust promise_batch_action_delete_account(promise_idx: u64, beneficiary_id_len: u64, beneficiary_id_ptr: u64) ``` Appends `DeleteAccount` action to the batch of actions for the given promise pointed by `promise_idx`. Action is used to delete an account. It can be performed on a newly created account, on your own account or an account with insufficient funds to pay rent. Takes `beneficiary_id` to indicate where to send the remaining funds. ###### Panics - If `promise_idx` does not correspond to an existing promise panics with `InvalidPromiseIndex`. - If the promise pointed by the `promise_idx` is an ephemeral promise created by `promise_and`. - If `beneficiary_id_len + beneficiary_id_ptr` points outside the memory of the guest or host, with `MemoryAccessViolation`. --- ================================================ FILE: neps/archive/0017-execution-outcome.md ================================================ - Proposal Name: Execution Outcome - Start Date: 2019-09-23 - NEP PR: [nearprotocol/neps#0017](https://github.com/nearprotocol/neps/pull/17) - Issue(s): https://github.com/nearprotocol/nearcore/issues/1307 # Summary Refactor current TransactionResult/TransactionLog/FinalTransactionResult to improve naming, deduplicate results and provide results resolution by the front-end for async-calls. # Motivation Right now the contract calls 2 promises and doesn't return a value, the front-end will return one of the promises results as an execution result. It's because we return the last result from final transaction result. With the current API, it's impossible to know what is the actual result of the contract execution. # Guide-level explanation Here is the proposed Rust structures. Highlights: - Rename `TransactionResult` to `ExecutionOutcome` since it's used for transactions and receipts - Rename `TransactionStatus` and merge it with result into `ExecutionResult`. - In case of success `ExecutionStatus` can either be a value of a receipt_id. This helps to resolve the actual returned value by the transaction from async calls, e.g. `A->B->A->C` should return result from `C`. Also in distinguish result in case of forks, e.g. `A` calls `B` and calls `C`, but returns a result from `B`. Currently there is no way to know. - Rename `TransactionLog` to `ExecutionOutcomeWithId` which is `ExecutionOutcome` with receipt_id or transaction hash. Probably needs a better name. - Rename `FinalTransactionResult` to `FinalExecutionOutcome`. - Update `FinalTransactionStatus` to `FinalExecutionStatus`. - Provide final resolved returned result directly, so the front-end doesn't need to traverse the receipt tree. We may also expose the error directly in the execution result. - Split into final outcome into transaction and receipts. ### NEW - The `FinalExecutionStatus` contains the early result even if some dependent receipts are not yet executed. Most function call transactions contain 2 receipts. The 1st receipt is execution, the 2nd is the refund. Before this change, the transaction was not resolved until the 2nd receipt was executed. After this change, the `FinalExecutionOutcome` will have `FinalTransactionStatus::SuccessValue("")` after the execution of the 1st receipt, while the 2nd receipt execution outcome status is still `Pending`. This helps to get the transaction result on the front-end faster without waiting for all refunds. ```rust pub struct ExecutionOutcome { /// Execution status. Contains the result in case of successful execution. pub status: ExecutionStatus, /// Logs from this transaction or receipt. pub logs: Vec, /// Receipt IDs generated by this transaction or receipt. pub receipt_ids: Vec, /// The amount of the gas burnt by the given transaction or receipt. pub gas_burnt: Gas, } /// The status of execution for a transaction or a receipt. pub enum ExecutionStatus { /// The execution is pending. Pending, /// The execution has failed. Failure, /// The final action succeeded and returned some value or an empty vec. SuccessValue(Vec), /// The final action of the receipt returned a promise or the signed transaction was converted /// to a receipt. Contains the receipt_id of the generated receipt. SuccessReceiptId(CryptoHash), } // TODO: Need a better name pub struct ExecutionOutcomeWithId { /// The transaction hash or the receipt ID. pub id: CryptoHash, pub outcome: ExecutionOutcome, } #[derive(Serialize, Deserialize, PartialEq, Eq, Debug, Clone)] pub enum FinalExecutionStatus { /// The execution has not yet started. NotStarted, /// The execution has started and still going. Started, /// The execution has failed. Failure, /// The execution has succeeded and returned some value or an empty vec in base64. SuccessValue(String), } pub struct FinalExecutionOutcome { /// Execution status. Contains the result in case of successful execution. pub status: FinalExecutionStatus, /// The execution outcome of the signed transaction. pub transaction: ExecutionOutcomeWithId, /// The execution outcome of receipts. pub receipts: Vec, } ``` ================================================ FILE: neps/archive/0018-view-change-method.md ================================================ - Proposal Name: Improve view/change methods in contracts - Start Date: 2019-09-26 - NEP PR: [nearprotocol/neps#0000](https://github.com/nearprotocol/neps/pull/18) # Summary Currently the separation between view methods and change methods on the contract level is not very well defined and causes quite a bit of confusion among developers. We propose in the NEP to elucidate the difference between view methods and change methods and how they should be used. In short, we would like to restrict view methods from accessing certain context variables and do not distinguish between view and change methods on the contract level. Developers have the option to differentiate between the two in frontend or through near-shell. # Motivation From the feedback we received it seems that developers are confused by the results they get from view calls, which are mainly caused by the fact that some binding methods such as `signer_account_id`, `current_account_id`, `attached_deposit` do not make sense in a view call. To avoid such confusion and create better developer experience, it is better if those context variables are prohibited in view calls. # Guide-level explanation Among binding methods that we expose from nearcore, some do make sense in a view call, such as `block_index`, while the majority does not. Here we explicitly list the methods are not allowed in a view call and, in case they are invoked, the contract will panic with ` is not allowed in view calls`. The following methods are prohibited: - `signer_account_id` - `signer_account_pk` - `predecessor_account_id` - `attached_deposit` - `prepaid_gas` - `used_gas` - `promise_create` - `promise_then` - `promise_and` - `promise_batch_create` - `promise_batch_then` - `promise_batch_action_create_account` - `promise_batch_action_deploy_account` - `promise_batch_action_function_call` - `promise_batch_action_transfer` - `promise_batch_action_stake` - `promise_batch_action_add_key_with_full_access` - `promise_batch_action_add_key_with_function_call` - `promise_batch_action_delete_key` - `promise_batch_action_delete_account` - `promise_results_count` - `promise_result` - `promise_return` From the developer perspective, if they want to call view functions from command line on some contract, they would just call `near view [args]`. If they are building an app and want to call a view function from the frontend, they should follow the same pattern as we have right now, specifying `viewMethods` and `changeMethods` in `loadContract`. # Reference-level explanation To implement this NEP, we need to change how binding methods are handled in runtime. More specifically, we can rename `free_of_charge` to `is_view` and use that to indicate whether we are processing a view call. In addition we can add a variant `ProhibitedInView(String)` to `HostError` so that if `is_view` is true, then all the access to the prohibited methods will error with `HostError::ProhibitedInView()`. # Drawbacks In terms of not allowing context variables, I don't see any drawback as those variables do not have a proper meaning in view functions. For alternatives, see the section below. # Rationale and alternatives This design is very simple and requires very little change to the existing infrastructure. An alternative solution is to distinguish between view methods and change methods on the contract level. One way to do it is through decorators, as described [here](https://github.com/nearprotocol/NEPs/pull/3). However, enforcing such distinction on the contract level requires much more work and is not currently feasible for Rust contracts. # Unresolved questions # Future possibilities ================================================ FILE: neps/archive/0033-economics.md ================================================ - Proposal Name: NEAR economics specs - Start Date: 2020-02-23 - NEP PR: [nearprotocol/neps#0000](https://github.com/nearprotocol/NEPs/pull/33) - Issue(s): link to relevant issues in relevant repos (not required). # Summary Adding economics specification for NEAR Protocol based on the NEAR whitepaper - https://pages.near.org/papers/the-official-near-white-paper/#economics # Motivation Currently, the specification is defined by the implementation in https://github.com/near/nearcore. This codifies all the parameters and formulas and defines main concepts. # Guide-level explanation The goal is to build a set of specs about NEAR token economics, for analysts and adopters, to simplify their understanding of the protocol and its game-theoretical dynamics. This initial release will be oriented to validators and staking in general. # Reference-level explanation This part of the documentation is self-contained. It may provide material for third-party research papers, and spreadsheet analysis. # Drawbacks We might just put this in the NEAR docs. # Rationale and alternatives # Unresolved questions # Future possibilities This is an open document which may be used by NEAR's community to pull request a new economic policy. Having a formal document also for non-technical aspects opens new opportunities for the governance. ================================================ FILE: neps/archive/0040-split-states.md ================================================ - Proposal Name: Splitting States for Simple Nightshade - Start Date: 2021-07-19 - NEP PR: [near/NEPs#241](https://github.com/near/NEPs/pull/241) - Issue(s): [near/NEPs#225](https://github.com/near/NEPs/issues/225) [near/nearcore#4419](https://github.com/near/nearcore/issues/4419) # Summary This proposal proposes a way to split each shard in the blockchain into multiple shards. Currently, the near blockchain only has one shard and it needs to be split into eight shards for Simple Nightshade. # Motivation To enable sharding, specifically, phase 0 of Simple Nightshade, we need to find a way to split the current one shard state into eight shards. # Guide-level explanation The proposal assumes that all validators track all shards and that challenges are not enabled. Suppose the new sharding assignment comes into effect at epoch T. State migration is done at epoch T-1, when the validators for epoch T are catching up states for the next epoch. At the beginning of epoch T-1, they run state sync for the current shards if needed. From the existing states, they build states for the new shards, then apply changes to the new states when they process the blocks in epoch T-1. This whole process runs off-chain as the new states will be not included in blocks at epoch T-1. At the beginning of epoch T, the new validators start to build blocks based on the new state roots. The change involves three parts. ## Dynamic Shards The first issue to address in splitting shards is the assumption that the current implementation of chain and runtime makes that the number of shards never changes. This in turn involves two parts, how the validators know when and how sharding changes happen and how they store states of shards from different epochs during the transition. The former is a protocol change and the latter only affects validators' internal states. ### Protocol Change Sharding config for an epoch will be encapsulated in a struct `ShardLayout`, which not only contains the number of shards, but also layout information to decide which account ids should be mapped to which shards. The `ShardLayout` information will be stored as part of `EpochConfig`. Right now, `EpochConfig` is stored in `EpochManager` and remains static accross epochs. That will be changed in the new implementation so that `EpochConfig` can be changed according to protocol versions, similar to how `RuntimeConfig` is implemented right now. The switch to Simple Nightshade will be implemented as a protocol upgrade. `EpochManager` creates a new `EpochConfig` for each epoch from the protocol version of the epoch. When the protocol version is large enough and the `SimpleNightShade` feature is enabled, the `EpochConfig` will be use the `ShardLayout` of Simple Nightshade, otherwise it uses the genesis `ShardLayout`. Since the protocol version and the shard information of epoch T will be determined at the end of epoch T-2, the validators will have time to prepare for states of the new shards during epoch T-1. Although not ideal, the `ShardLayout` for Simple Nightshade will be added as part of the genesis config in the code. The genesis config file itself will not be changed, but the field will be set to a default value we specify in the code. This process is as hacky as it sounds, but currently we have no better way to account for changing protocol config. To completely solve this issue will be a hard problem by itself, thus we do not try to solve it in this NEP. We will discuss how the sharding transition will be managed in the next section. ### State Change In epoch T-1, the validators need to maintain two versions of states for all shards, one for the current epoch, one that is split for the next epoch. Currently, shards are identified by their `shard_id`, which is a number ranging from `0` to `NUM_SHARDS-1`.`shard_id` is also used as part of the indexing keys by which trie nodes are stored in the database. However, when shards may change accross epochs, `shard_id` can no longer be used to uniquely identify states because new shards and old shards will share the same `shard_id`s under this representation. To solve this issue, the new proposal creates a new struct `ShardUId` as an unique identifier to reference shards accross epochs. `ShardUId` will only be used for storing and managing states, for example, in `Trie` related structures, In most other places in the code, it is clear which epoch the referenced shard belongs, and `ShardId` is enough to identify the shard. There will be no change in the protocol level since `ShardId` will continue to be used in protocol level specs. `ShardUId` contains a version number and the corresponding `shard_id`. ```rust pub struct ShardUId { version: u32, shard_id: u32, } ``` The version number is different between different shard layouts, to ensure `ShardUId`s for shards from different epochs are different. `EpochManager` will be responsible for managing shard versions and `ShardUId` accross epochs. ## Build New States Currently, when receiving the first block of every epoch, validators start downloading states to prepare for the next epoch. We can modify this existing process to make the validators build states for the new shards after they finish downloading states for the existing shards. To build the new states, the validator iterates through all accounts in the current states and adds them to the new states one by one. ## Update States Similar to how validators usually catch up for the next epoch, the new states are updated as new blocks are processed. The difference is that in epoch T-1, chunks are still sharded by the current sharding assignment, but the validators need to perform updates on the new states. We cannot simply split transactions and receipts to the new shards and process updates on each new shard separately. If we do so, since each shard processes transactions and receipts with their own gas limits, some receipts may be delayed in the new states but not in the current states, or the other way around. That will lead to inconsistencies between the orderings by which transactions and receipts are applied to the current and new states. For example, for simplicity, assume there is only one shard A in epoch T-1 and there will be two shards B and C in epoch T. To process a block in epoch T-1, shard A needs to process receipts 0, 1, .., 99 while in the new sharding assignments receipts 0, 2, …, 98 belong to shard B and receipts 1, 3, …, 99 belong to shard C. Assume in shard A, the gas limit is hit after receipt 89 is processed, so receipts 90 to 99 are delayed. To achieve the same processing result, shard B must process receipt 0, 2, …, 88 and delay 90, 92, ..., 98 and shard C must process receipt 1, 3, ..., 89 and delay receipts 91, 93, …, 99. However, shard B and C have their own gas limits and which receipts will be processed and delayed cannot be guaranteed. Whether a receipt is processed in a block or delayed can affect the execution result of this receipt because transactions are charged and local receipts are processed before delayed receipts are processed. For example, let’s assume Alice’s account has 0N now and Bob sends a transaction T1 to transfer 5N to Alice. The transaction has been converted to a receipt R at block i-1 and sent to Alice's shard at block i. Let's say Alice signs another transaction T2 to send 1N to Charlie and that transaction is included in block i+1. Whether transaction T2 succeeds depends on whether receipt R is processed or delayed in block i. If R is processed in block i, Alice’s account will have 5N before block i+1 and T2 will succeed while if R is delayed in block i, Alice’s account will have 0N and T2 will be declined. Therefore, the validators must still process transactions and receipts based on the current sharding assignment. After the processing is finished, they can take the generated state changes to apply to the new states. # Reference-level explanation ## Protocol-Level Shard Representation ### `ShardLayout` ```rust pub enum ShardLayout { V0(ShardLayoutV0), V1(ShardLayoutV1), } ``` ShardLayout is a versioned struct that contains all information needed to decide which accounts belong to which shards. Note that `ShardLayout` only contains information at the protocol level, so it uses `ShardOrd` instead of `ShardId`. The API contains the following two functions. #### `get_split_shards` ``` pub fn get_split_shards(&self, parent_shard_id: ShardId) -> Option<&Vec> ``` returns the children shards of shard `parent_shard_id` (we will explain parent-children shards shortly). Note that `parent_shard_id` is a shard from the last ShardLayout, not from `self`. The returned `ShardId` represents shard in the current shard layout. This information is needed for constructing states for the new shards. We only allow adding new shards that are split from the existing shards. If shard B and C are split from shard A, we call shard A the parent shard of shard B and C. For example, if epoch T-1 has a shard layout `shardlayout0` with two shards with `shard_ord` 0 and 1 and each of them will be split to two shards in `shardlayout1` in epoch T, then `shard_layout1.get_split_shards(0)` returns `[0,1]` and `shard_layout.get_split_shards(1)` returns `[2,3]`. #### `version` ```rust pub fn version(&self) -> ShardVersion ``` returns the version number of this shard layout. This version number is used to create `ShardUId` for shards in this `ShardLayout`. The version numbers must be different for all shard layouts used in the blockchain. #### `account_id_to_shard_id` ```rust pub fn account_id_to_shard_id(account_id: &AccountId, shard_layout: ShardLayout) -> ShardId ``` maps account id to shard id given a shard layout #### `ShardLayoutV0` ```rust pub struct ShardLayoutV0 { /// map accounts evenly accross all shards num_shards: NumShards, } ``` A shard layout that maps accounts evenly accross all shards -- by calculate the hash of account id and mod number of shards. This is added to capture the current `account_id_to_shard_id` algorithm, to keep backward compatibility for some existing tests. `parent_shards` for `ShardLayoutV1` is always `None` and `version`is always `0`. #### `ShardLayoutV1` ```rust pub struct ShardLayoutV1 { /// num_shards = fixed_shards.len() + boundary_accounts.len() + 1 /// Each account and all subaccounts map to the shard of position in this array. fixed_shards: Vec, /// The rest are divided by boundary_accounts to ranges, each range is mapped to a shard boundary_accounts: Vec, /// Parent shards for the shards, useful for constructing states for the shards. /// None for the genesis shard layout parent_shards: Option>, /// Version of the shard layout, useful to uniquely identify the shard layout version: ShardVersion, } ``` A shard layout that consists some fixed shards each of which is mapped to a fixed account and other shards which are mapped to ranges of accounts. This will be the ShardLayout used by Simple Nightshade. ### `EpochConfig` `EpochConfig` will contain the shard layout for the given epoch. ```rust pub struct EpochConfig { // existing fields ... /// Shard layout of this epoch, may change from epoch to epoch pub shard_layout: ShardLayout, ``` ### `AllEpochConfig` `AllEpochConfig` stores a mapping from protocol versions to `EpochConfig`s. `EpochConfig` for a particular epoch can be retrieved from `AllEpochConfig`, given the protocol version of the epoch. For SimpleNightshade migration, it only needs to contain two configs. `AllEpochConfig` will be stored inside `EpochManager` to be used to construct `EpochConfig` for different epochs. ```rust pub struct AllEpochConfig { genesis_epoch_config: Arc, simple_nightshade_epoch_config: Arc, } ``` #### `for_protocol_version` ```rust pub fn for_protocol_version(&self, protocol_version: ProtocolVersion) -> &Arc ``` returns `EpochConfig` according to the given protocol version. `EpochManager` will call this function for every new epoch. ### `EpochManager` `EpochManager` will be responsible for managing `ShardLayout` accross epochs. As we mentioned, `EpochManager` stores an instance of `AllEpochConfig`, so it can returns the `ShardLayout` for each epoch. #### `get_shard_layout` ```rust pub fn get_shard_layout(&mut self, epoch_id: &EpochId) -> Result<&ShardLayout, EpochError> ``` ## Internal Shard Representation in Validators' State ### `ShardUId` `ShardUId` is a unique identifier that a validator uses internally to identify shards from all epochs. It only exists inside a validator's internal state and can be different among validators, thus it should never be exposed to outside APIs. ```rust pub struct ShardUId { pub version: ShardVersion, pub shard_id: u32, } ``` `version` in `ShardUId` comes from the version of `ShardLayout` that this shard belongs. This way, different shards from different shard layout will have different `ShardUId`s. ### Database storage The following database columns are stored with `ShardId` as part of the database key, it will be replaced by `ShardUId` - ColState - ColChunkExtra - ColTrieChanges #### `TrieCachingStorage` Trie storage will contruct database key from `ShardUId` and hash of the trie node. ##### `get_shard_uid_and_hash_from_key` ```rust fn get_shard_uid_and_hash_from_key(key: &[u8]) -> Result<(ShardUId, CryptoHash), std::io::Error> ``` ##### `get_key_from_shard_uid_and_hash` ```rust fn get_key_from_shard_uid_and_hash(shard_uid: ShardUId, hash: &CryptoHash) -> [u8; 40] ``` ## Build New States The following method in `Chain` will be added or modified to split a shard's current state into multiple states. ### `build_state_for_split_shards` ```rust pub fn build_state_for_split_shards(&mut self, sync_hash: &CryptoHash, shard_id: ShardId) -> Result<(), Error> ``` builds states for the new shards that the shard `shard_id` will be split to. After this function is finished, the states for the new shards should be ready in `ShardTries` to be accessed. ### `run_catchup` ```rust pub fn run_catchup(...) { ... match state_sync.run( ... )? { StateSyncResult::Unchanged => {} StateSyncResult::Changed(fetch_block) => {...} StateSyncResult::Completed => { // build states for new shards if shards will change and we will track some of the new shards if self.runtime_adapter.will_shards_change_next_epoch(epoch_id) { let mut parent_shards = HashSet::new(); let (new_shards, mapping_to_parent_shards) = self.runtime_adapter.get_shards_next_epoch(epoch_id); for shard_id in new_shards { if self.runtime_adapter.will_care_about_shard(None, &sync_hash, shard_id, true) { parent_shards.insert(mapping_to_parent_shards.get(shard_id)?); } } for shard_id in parent_shards { self.split_shards(me, &sync_hash, shard_id); } } ... } } ... } ``` ## Update States ### `split_state_changes` ```rust split_state_changes(shard_id: ShardId, state_changes: &Vec) -> HashMap> ``` splits state changes to be made to a current shard to changes that should be applid to the new shards. Note that this function call can take a long time. To avoid blocking the client actor from processing and producing blocks for the current epoch, it should be called from a separate thread. Unfortunately, as of now, catching up states and catching up blocks are both run in client actor. They should be moved to a separate actor. However, that can be a separate project, although this NEP will depend on that project. In fact, the issue has already been discussed in [#3201](https://github.com/near/nearcore/issues/3201). ### `apply_chunks` `apply_chunks` will be modified so that states of the new shards will be updated when processing chunks. In `apply_chunks`, after processing each chunk, the state changes in `apply_results` are sorted into changes to new shards. At the end, we apply these changes to the new shards. ```rust fn apply_chunks(...) -> Result<(), Error> { ... for (shard_id, (chunk_header, prev_chunk_header)) in (block.chunks().iter().zip(prev_block.chunks().iter())).enumerate() { ... let apply_result = ...; // split states to new shards let changes_to_new_shards = self.split_state_changes(trie_changes); // apply changes_to_new_changes to the new shards for (new_shard_id, new_state_changes) in changes_to_new_states { // locate the state for the new shard let trie = self.get_trie_for_shard(new_shard_id); let chunk_extra = self.chain_store_update.get_chunk_extra(&prev_block.hash(), new_shard_id)?.clone(); let mut state_update = TrieUpdate::new(trie.clone(), *chunk_extra.state_root()); // update the state for state_change in new_state_changes { state_update.set(state_change.trie_key, state_change.value); } state_update.commit(StateChangeCause::Resharding); let (trie_changes, state_changes) = state_update.finalize()?; // save the TrieChanges and ChunkExtra self.chain_store_update.save_trie_changes(WrappedTrieChanges::new( self.tries, new_shard_id, trie_changes, state_changes, *block.hash(), )); self.chain_store_update.save_chunk_extra( &block.hash(), new_shard_id, ChunkExtra::new(&trie_changes.new_root, CryptoHash::default(), Vec::new(), 0, 0, 0), ); } } ... } ``` ## Garbage Collection The old states need to be garbage collected after the resharding finishes. The garbage collection algorithm today won't automatically handle that. (#TODO: why?) Although we need to handle garbage collection eventually, it is not a pressing issue. Thus, we leave the discussion from this NEP for now and will add a detailed plan later. # Drawbacks The drawback of this approach is that it will not work when challenges are enabled since challenges to the transition to the new states will be too large to construct or verify. Thus, most of the change will likely be a one time use that only works for the Simple Nightshade transition, although part of the change involving `ShardId` may be reused in the future. # Rationale and alternatives - Why is this design the best in the space of possible designs? - It is the best because its implementation is the simplest. Considering we want to launch Simple Nightshade as soon as possible by Q4 2021 and we will not enable challenges any time soon, this is the best option we have. - What other designs have been considered and what is the rationale for not choosing them? - We have considered other designs that change states incrementally and keep state roots on chain to make it compatible with challenges. However, the implementation of those approaches are overly complicated and does not fit into our timeline for launching Simple Nightshade. - What is the impact of not doing this? - The impact will be the delay of launching Simple Nightshade, or no launch at all. # Unresolved questions - What parts of the design do you expect to resolve through the NEP process before this gets merged? - Garbage collection - State Sync? - What parts of the design do you expect to resolve through the implementation of this feature before stabilization? - There might be small changes in the detailed implementations or specifications of some of the functions described above, but the overall structure will not be changed. - What related issues do you consider out of scope for this NEP that could be addressed in the future independently of the solution that comes out of this NEP? - One issue that is related to this NEP but will be resolved indepdently is how trie nodes are stored in the database. Right now, it is a combination of `shard_id` and the node hash. Part of the change proposed in this NEP regarding `ShardId` is because of this. Plans on how to only store the node hash as keys are being discussed [here](https://github.com/near/nearcore/issues/4527), but it will happen after the Simple Nightshade migration since completely solving the issue will take some careful design and we want to prioritize launching Simple Nightshade for now. - Another issue that is not part of this NEP but must be solved for this NEP to work is to move expensive computation related to state sync / catch up into a separate actor [#3201](https://github.com/near/nearcore/issues/3201). - Lastly, we should also build a better mechanism to deal with changing protocol config. The current way of putting changing protocol config in the genesis config and changing how the genesis config file is parsed is not a long term solution. # Future possibilities ## Extension In the future, when challenges are enabled, resharding and state upgrade should be implemented on-chain. ## Affected Projects - ## Pre-mortem - Building and catching up new states takes longer than one epoch to finish. - Protocol version switched back to pre simple nightshade - Validators cannot track shards properly after resharding - Genesis State - Must load the correct `shard_version` - ShardTracker? ================================================ FILE: neps/archive/README.md ================================================ # Proposals This section contains the NEAR Enhancement Proposals (NEPs) that cover a fleshed out concept for NEAR. Before an idea is turned into a proposal, it will be fleshed out and discussed on the [NEAR Governance Forum](https://gov.near.org). These subcategories are great places to start such a discussion: - [Standards](https://gov.near.org/c/dev/standards/29) — examples might include new protocol standards, token standards, etc. - [Proposals](https://gov.near.org/c/dev/proposals/68) — ecosystem proposals that may touch tooling, node experience, wallet usage, and so on. Once and idea has been thoroughly discussed and vetted, a pull request should be made according to the instructions at the [NEP repository](https://github.com/near/NEPs). The proposals shown in this section have been merged and exist to offer as much information as possible including historical motivations, drawbacks, approaches, future concerns, etc. Once a proposal has been fully implemented it can be added as a specification, but will remain a proposal until that time. ================================================ FILE: neps/nep-0001.md ================================================ --- NEP: 1 Title: NEP Purpose and Guidelines Authors: Bowen W. , Austin Baggio , Ori A. , Vlad F. , Guillermo G. ; Status: Approved DiscussionsTo: https://github.com/near/NEPs/pull/333, https://github.com/near/NEPs/pull/619 Type: Developer Tools Version: 2.0.0 Created: 2022-03-03 Last Updated: 2025-08-04 --- ## Summary NEAR Enhancement Proposals (NEPs) are design documents that describe standards for the NEAR platform, including core protocol specifications, contract standards, and wallet APIs. Each NEP provides concise technical specifications and the rationale behind the proposed enhancement. Each NEP is championed by a community member, which builds consensus within the community and sheperds the NEP from ideation to completion. The NEP process is designed to be open and transparent, allowing anyone in the NEAR community to propose, discuss, and review ideas for improving the NEAR ecosystem. All NEPs are stored as text files in a [versioned repository](https://github.com/near/NEPs), allowing for easy historical tracking. ## Motivation The purpose of the NEP process is to give the community a way to propose, discuss, and document changes that impact the whole NEAR ecosystem in a structured manner. Given the complexity and number of participants involved across the ecosystem, a well-defined process helps ensure transparency, security, and stability. ## NEP Types There are three kinds of NEPs: 1. A **Protocol** NEP describes a new feature of the NEAR protocol (e.g. [NEP-264](https://github.com/near/NEPs/blob/master/neps/nep-0264.md), [NEP-366](https://github.com/near/NEPs/blob/master/neps/nep-0366.md)) 2. A **Contract Standards** NEP specifies NEAR smart contract interfaces for a reusable concept in the NEAR ecosystem (e.g. [NEP-141](https://github.com/near/NEPs/blob/master/neps/nep-0141.md), [NEP-171](https://github.com/near/NEPs/blob/master/neps/nep-0171.md)) 3. A **Wallet Standards** NEP specifies ecosystem-wide APIs for Wallet implementations (e.g. [NEP-413](https://github.com/near/NEPs/blob/master/neps/nep-0413.md)) ## Submit a NEP Each NEP must have a champion that proposes a new idea, shepherds the discussions in the appropriate forums to build community consensus, proposes the NEP and help it progress toward completion. ### Start with ideation Everyone in the community is welcome to propose, discuss, and review ideas to improve the NEAR protocol and standards. The NEP process begins with a new idea for the NEAR ecosystem. Before submitting a new NEP, publicly check if your idea is original and relevant to the NEAR community. This saves time and avoids proposing something already discussed or unsuitable for most users. - **Check prior proposals:** Many ideas for changing NEAR come up frequently. Please search the [issues](https://github.com/near/NEPs/issues) and NEPs in this repo before proposing something new. - **Share the idea:** Submit a new [issue](https://github.com/near/NEPs/issues) explaining the problem you want to tackle, and your proposed solution. - **Get feedback:** Share the issue to the appropriate community group: - Wallet Group: https://nearbuilders.com/tg-wallet - Protocol: https://near.zulipchat.com/ - Contract Standards: https://t.me/NEAR_Tools_Community_Group ### Submit a NEP Draft Following the above initial discussions, the author willing to champion the NEP should submit the NEP Draft in the form of a `Draft Pull Request`: 1. Fork the [NEPs repository](https://github.com/near/NEPs). 2. Copy `nep-0000-template.md` to `neps/nep-xxxx.md` (do **not** assign a NEP number yet). 3. Fill in the NEP following the NEP template guidelines. For the Header Preamble, make sure to set the status as “Draft.” 4. Push this to your GitHub fork and submit a pull request. 5. Now that your NEP has an open pull request, use the pull request number to update your `0000` prefix. For example, if the PR is 305, the NEP should be `neps/nep-0305.md`. 6. Push this to your GitHub fork and submit a pull request. Mention the @near/nep-moderators in the comment and turn the PR into a "Ready for Review" state once you believe the NEP is ready for review. ## NEP Lifecycle The NEP process begins when an author submits a [NEP draft](#submit-a-nep-draft). The NEP lifecycle consists of three stages: draft, review, and voting, with two possible outcomes: approval or rejection. Throughout the process, various roles play a critical part in moving the proposal forward. Most of the activity happens asynchronously on the NEP within GitHub, where all the roles can communicate and collaborate on revisions and improvements to the proposal. ![NEP Process](https://user-images.githubusercontent.com/110252255/201413632-f72743d6-593e-4747-9409-f56bc38de17b.png) ### NEP Stages - **Draft:** The first formally tracked stage of a new NEP. This process begins once an author submits a draft proposal and the NEP moderator merges it into the NEP repo when properly formatted. - **Review:** A NEP moderator marks a NEP as ready for Subject Matter Experts Review. If the NEP is not approved within two months, it is automatically rejected. - **Voting:** This is the final voting period for a NEP. The working group will vote on whether to accept or reject the NEP. This period is limited to two weeks. If during this period necessary normative changes are required, the NEP will revert to Review. Moderator, when moving a NEP to review stage, should update the Pull Request description to include the review summary, example: ```markdown --- ## NEP Status _(Updated by NEP moderators)_ SME reviews: - [ ] Role1: @github-handle - [ ] Role2: @github-handle Contract Standards WG voting indications (❔ | :+1: | :-1: ): - ❔ @github-handle - ❔ ... voting indications: - ❔ - ❔ ``` ### NEP Outcomes - **Approved:** If the working group votes to approve, they will move the NEP to Approved. Once approved, Standards NEPs exist in a state of finality and should only be updated to correct errata and add non-normative clarifications. - **Rejected:** If the working group votes to reject, they will move the NEP to Rejected. ### NEP Roles and Responsibilities ![author](https://user-images.githubusercontent.com/110252255/181816534-2f92b073-79e2-4e8d-b5b9-b10824958acd.png) **Author**
_Anyone can participate_ The NEP author (or champion) is responsible for creating a NEP draft that follows the guidelines. They drive the NEP forward by actively participating in discussions and incorporating feedback. During the voting stage, they may present the NEP to the working group and community, and provide a final implementation with thorough testing and documentation once approved. ![Moderator](https://user-images.githubusercontent.com/110252255/181816650-b1610c0e-6d32-4d2a-a34e-877c702139bd.png) **Moderator**
_Assigned by the working group_ The moderator is responsible for facilitating the process and validating that the NEP follows the guidelines. They do not assess the technical feasibility or write any part of the proposal. They provide comments if revisions are necessary and ensure that all roles are working together to progress the NEP forward. They also schedule and facilitate public voting calls. ![Reviewer](https://user-images.githubusercontent.com/110252255/181816664-a9485ea6-e774-4999-b11d-dc8be6b08f87.png) **NEP Reviewer** (Subject Matter Experts)
_Assigned by the working group_ The reviewer is responsible for reviewing the technical feasibility of a NEP and giving feedback to the author. While they do not have voting power, they play a critical role in providing their voting recommendations along with a summary of the benefits and concerns that were raised in the discussion. Their inputs help everyone involved make a transparent and informed decision. ![Approver](https://user-images.githubusercontent.com/110252255/181816752-521dd147-f56f-4c5c-84de-567b109f21d6.png) **Approver** (Working Groups)
_Selected by the Dev Gov DAO in the bootstrapping phase_ The working group is a selected committee of 3-7 recognized experts who are responsible for coordinating the public review and making decisions on a NEP in a fair and timely manner. There are multiple working groups, each one focusing on a specific ecosystem area, such as the Protocol or Wallet Standards. They assign reviewers to proposals, provide feedback to the author, and attend public calls to vote to approve or reject the NEP. ### NEP Communication NEP discussions should happen asynchronously within the NEP’s public thread. This allows for broad participation and ensures transparency. However, if a discussion becomes circular and could benefit from a synchronous conversation, any participants on a given NEP can suggest that the moderator schedules an ad hoc meeting. For example, if a reviewer and author have multiple rounds of comments, they may request a call. The moderator can help coordinate the call and post the registration link on the NEP. The person who requested the call should designate a note-taker to post a summary on the NEP after the call. When a NEP gets to the final voting stage, the moderator will schedule a public working group meeting to discuss the NEP with the author and formalize the decision. The moderator will first coordinate a time with the author and working group members, and then post the meeting time and registration link on the NEP at least one week in advance. All participants in the NEP process should maintain a professional and respectful code of conduct in all interactions. This includes communicating clearly and promptly and refraining from disrespectful or offensive language. ### NEP Playbook 1. Once an author [submits a NEP draft](#submit-a-nep-draft), the NEP moderators will review their pull request (PR) for structure, formatting, and other errors. Approval criteria are: - The content is complete and technically sound. The moderators do not consider whether the NEP is likely or not to get accepted. - The title accurately reflects the content. - The language, spelling, grammar, sentence structure, and code style are correct and conformant. 2. If the NEP is not ready for approval, the moderators will send it back to the author with specific instructions in the PR. The moderators must complete the review within one week. 3. Once the moderators agree that the PR is ready for review, they will ask the approvers (working group members) to nominate a team of at least two reviewers (subject matter experts) to review the NEP. At least one working group member must explicitly tag the reviewers and comment: `"As a working group member, I'd like to nominate @SME-username and @SME-username as the Subject Matter Experts to review this NEP."` If the assigned reviewers feel that they lack the relevant expertise to fully review the NEP, they can ask the working group to re-assign the reviewers for the NEP. 4. The reviewers must finish the technical review within one week. Technical Review Guidelines: - First, review the technical details of the proposals and assess their merit. If you have feedback, explicitly tag the author and comment: `"As the assigned Reviewer, I request from @author-username to [ask clarifying questions, request changes, or provide suggestions that are actionable.]."` It may take a couple of iterations to resolve any open comments. - Second, once the reviewer believes that the NEP is close to the voting stage, explicitly tag the @near/nep-moderators and comment with your technical summary. The Technical Summary must include: - A recommendation for the working group: `"As the assigned reviewer, I do not have any feedback for the author. I recommend moving this NEP forward and for the working group to [accept or reject] it based on [provide reasoning, including a sense of importance or urgency of this NEP]."` Please note that this is the reviewer's personal recommendation. - A summary of benefits that surfaced in previous discussions. This should include a concise list of all the benefits that others raised, not just the ones that the reviewer personally agrees with. - A summary of concerns or blockers, along with their current status and resolution. Again, this should reflect the collective view of all commenters, not just the reviewer's perspective. 5. The NEP author can make revisions and request further reviews from the reviewers. However, if a proposal is in the review stage for more than two months, the moderator will automatically reject it. To reopen the proposal, the author must restart the NEP process again. 6. Once both reviewers complete their technical summary, the moderators will notify the approvers (working group members) that the NEP is in the final comment period. The approvers must fully review the NEP within one week. Approver guidelines: - First, read the NEP thoroughly. If you have feedback, explicitly tag the author and comment: `"As a working group member, I request from @author-username to [ask clarifying questions, request changes, or provide actionable suggestions.]."` - Second, once the approver believes the NEP is close to the voting stage, explicitly comment with your voting indication: `"As a working group member, I lean towards [approving OR rejecting] this NEP based on [provide reasoning]."` 7. Once all the approvers indicate their voting indication, the moderator will review the voting indication for a 2/3 majority: - If the votes lean toward rejection: The moderator will summarize the feedback and close the NEP. - If the votes lean toward approval: The moderator will schedule a public call (see [NEP Communication](#nep-communication)) for the author to present the NEP and for the working group members to formalize the voting decision. If the working group members agree that the NEP is overall beneficial for the NEAR ecosystem and vote to approve it, then the proposal is considered accepted. After the call, the moderator will summarize the decision on the NEP. 8. The NEP author or other assignees will complete action items from the call. For example, the author will finalize the "Changelog" section on the NEP, which summarizes the benefits and concerns for future reference. ### Transferring NEP Ownership While a NEP is worked on, it occasionally becomes necessary to transfer ownership of NEPs to a new author. In general, it is preferable to retain the original author as a co-author of the transferred NEP, but that is up to the original author. A good reason to transfer ownership is that the original author no longer has the time or interest in updating it or following through with the NEP process. A bad reason to transfer ownership is that the author does not agree with the direction of the NEP. One aim of the NEP process is to try to build consensus around a NEP, but if that is not possible, an author can submit a competing NEP. If you are interested in assuming ownership of a NEP, you can also do this via pull request. Fork the NEP repository, modify the owner, and submit a pull request. In the PR description, tag the original author and provide a summary of the work that was previously done. Also clearly state the intent of the fork and the relationship of the new PR to the old one. For example: "Forked to address the remaining review comments in NEP \# since the original author does not have time to address them. ## What does a successful NEP look like? Each NEP should be written in markdown format and follow the [NEP-0000 template](https://github.com/near/NEPs/blob/master/nep-0000-template.md) and include all the appropriate sections, which will make it easier for the NEP reviewers and community members to understand and provide feedback. The most successful NEPs are those that go through collective iteration, with authors who actively seek feedback and support from the community. Ultimately, a successful NEP is one that addresses a specific problem or needs within the NEAR ecosystem, is well-researched, and has the support of the community and ecosystem experts. ### Auxiliary Files Images, diagrams, and auxiliary files should be included in a subdirectory of the assets folder for that NEP as follows: assets/nep-N (where N is to be replaced with the NEP number). When linking to an image in the NEP, use relative links such as `../assets/nep-1/image.png` ### Style Guide #### NEP numbers When referring to a NEP by number, it should be written in the hyphenated form NEP-X where X is the NEP's assigned number. #### RFC 2119 NEPs are encouraged to follow [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt) for terminology and to insert the following at the beginning of the Specification section: The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). ## NEP Maintenance Generally, NEPs are not modifiable after reaching their final state. However, there are occasions when updating a NEP is necessary, such as when discovering a security vulnerability or identifying misalignment with a widely-used implementation. In such cases, an author may submit a NEP extension in a pull request with the proposed changes to an existing NEP document. A NEP extension has a higher chance of approval if it introduces clear benefits to existing implementors and does not introduce breaking changes. If an author believes that a new extension meets the criteria for its own separate NEP, it is better to submit a new NEP than to modify an existing one. Just make sure to specify any dependencies on certain NEPs. ## References The content of this document was derived heavily from the PEP, BIP, Rust RFC, and EIP standards bootstrap documents: - Klock, F et al. Rust: RFC-0002: RFC Process. https://github.com/rust-lang/rfcs/blob/master/text/0002-rfc-process.md - Taaki, A. et al. Bitcoin Improvement Proposal: BIP:1, BIP Purpose and Guidelines. https://github.com/bitcoin/bips/blob/master/bip-0001.mediawiki - Warsaw, B. et al. Python Enhancement Proposal: PEP Purpose and Guidelines. https://github.com/python/peps/blob/main/peps/pep-0001.rst - Becze, M. et al. Ethereum Improvement Proposal EIP1: EIP Purpose and Guidelines. https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1.md ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0021.md ================================================ --- NEP: 21 Title: Fungible Token Standard Author: Evgeny Kuzyakov Status: Final DiscussionsTo: https://github.com/near/NEPs/pull/21 Type: Standards Track Category: Contract Created: 29-Oct-2019 SupersededBy: 141 --- ## Summary A standard interface for fungible tokens allowing for ownership, escrow and transfer, specifically targeting third-party marketplace integration. ## Motivation NEAR Protocol uses an asynchronous sharded Runtime. This means the following: Storage for different contracts and accounts can be located on the different shards. Two contracts can be executed at the same time in different shards. While this increases the transaction throughput linearly with the number of shards, it also creates some challenges for cross-contract development. For example, if one contract wants to query some information from the state of another contract (e.g. current balance), by the time the first contract receive the balance the real balance can change. It means in the async system, a contract can't rely on the state of other contract and assume it's not going to change. Instead the contract can rely on temporary partial lock of the state with a callback to act or unlock, but it requires careful engineering to avoid dead locks. ## Rationale and alternatives In this standard we're trying to avoid enforcing locks, since most actions can still be completed without locks by transferring ownership to an escrow account. Prior art: ERC-20 standard NEP#4 NEAR NFT standard: nearprotocol/neps#4 For latest lock proposals see Safes (#26) ## Specification We should be able to do the following: - Initialize contract once. The given total supply will be owned by the given account ID. - Get the total supply. - Transfer tokens to a new user. - Set a given allowance for an escrow account ID. - Escrow will be able to transfer up this allowance from your account. - Get current balance for a given account ID. - Transfer tokens from one user to another. - Get the current allowance for an escrow account on behalf of the balance owner. This should only be used in the UI, since a contract shouldn't rely on this temporary information. There are a few concepts in the scenarios above: - **Total supply**. It's the total number of tokens in circulation. - **Balance owner**. An account ID that owns some amount of tokens. - **Balance**. Some amount of tokens. - **Transfer**. Action that moves some amount from one account to another account. - **Escrow**. A different account from the balance owner who has permission to use some amount of tokens. - **Allowance**. The amount of tokens an escrow account can use on behalf of the account owner. Note, that the precision is not part of the default standard, since it's not required to perform actions. The minimum value is always 1 token. ### Simple transfer Alice wants to send 5 wBTC tokens to Bob. Assumptions: - The wBTC token contract is `wbtc`. - Alice's account is `alice`. - Bob's account is `bob`. - The precision on wBTC contract is `10^8`. - The 5 tokens is `5 * 10^8` or as a number is `500000000`. #### High-level explanation Alice needs to issue one transaction to wBTC contract to transfer 5 tokens (multiplied by precision) to Bob. #### Technical calls 1. `alice` calls `wbtc::transfer({"new_owner_id": "bob", "amount": "500000000"})`. ### Token deposit to a contract Alice wants to deposit 1000 DAI tokens to a compound interest contract to earn extra tokens. Assumptions: - The DAI token contract is `dai`. - Alice's account is `alice`. - The compound interest contract is `compound`. - The precision on DAI contract is `10^18`. - The 1000 tokens is `1000 * 10^18` or as a number is `1000000000000000000000`. - The compound contract can work with multiple token types. #### High-level explanation Alice needs to issue 2 transactions. The first one to `dai` to set an allowance for `compound` to be able to withdraw tokens from `alice`. The second transaction is to the `compound` to start the deposit process. Compound will check that the DAI tokens are supported and will try to withdraw the desired amount of DAI from `alice`. - If transfer succeeded, `compound` can increase local ownership for `alice` to 1000 DAI - If transfer fails, `compound` doesn't need to do anything in current example, but maybe can notify `alice` of unsuccessful transfer. #### Technical calls 1. `alice` calls `dai::set_allowance({"escrow_account_id": "compound", "allowance": "1000000000000000000000"})`. 1. `alice` calls `compound::deposit({"token_contract": "dai", "amount": "1000000000000000000000"})`. During the `deposit` call, `compound` does the following: 1. makes async call `dai::transfer_from({"owner_id": "alice", "new_owner_id": "compound", "amount": "1000000000000000000000"})`. 1. attaches a callback `compound::on_transfer({"owner_id": "alice", "token_contract": "dai", "amount": "1000000000000000000000"})`. ### Multi-token swap on DEX Charlie wants to exchange his wLTC to wBTC on decentralized exchange contract. Alex wants to buy wLTC and has 80 wBTC. Assumptions - The wLTC token contract is `wltc`. - The wBTC token contract is `wbtc`. - The DEX contract is `dex`. - Charlie's account is `charlie`. - Alex's account is `alex`. - The precision on both tokens contract is `10^8`. - The amount of 9001 wLTC tokens is Alex wants is `9001 * 10^8` or as a number is `900100000000`. - The 80 wBTC tokens is `80 * 10^8` or as a number is `8000000000`. - Charlie has 1000000 wLTC tokens which is `1000000 * 10^8` or as a number is `100000000000000` - Dex contract already has an open order to sell 80 wBTC tokens by `alex` towards 9001 wLTC. - Without Safes implementation, DEX has to act as an escrow and hold funds of both users before it can do an exchange. #### High-level explanation Let's first setup open order by Alex on DEX. It's similar to `Token deposit to a contract` example above. - Alex sets an allowance on wBTC to DEX - Alex calls deposit on Dex for wBTC. - Alex calls DEX to make an new sell order. Then Charlie comes and decides to fulfill the order by selling his wLTC to Alex on DEX. Charlie calls the DEX - Charlie sets the allowance on wLTC to DEX - Alex calls deposit on Dex for wLTC. - Then calls DEX to take the order from Alex. When called, DEX makes 2 async transfers calls to exchange corresponding tokens. - DEX calls wLTC to transfer tokens DEX to Alex. - DEX calls wBTC to transfer tokens DEX to Charlie. #### Technical calls 1. `alex` calls `wbtc::set_allowance({"escrow_account_id": "dex", "allowance": "8000000000"})`. 1. `alex` calls `dex::deposit({"token": "wbtc", "amount": "8000000000"})`. 1. `dex` calls `wbtc::transfer_from({"owner_id": "alex", "new_owner_id": "dex", "amount": "8000000000"})` 1. `alex` calls `dex::trade({"have": "wbtc", "have_amount": "8000000000", "want": "wltc", "want_amount": "900100000000"})`. 1. `charlie` calls `wltc::set_allowance({"escrow_account_id": "dex", "allowance": "100000000000000"})`. 1. `charlie` calls `dex::deposit({"token": "wltc", "amount": "100000000000000"})`. 1. `dex` calls `wltc::transfer_from({"owner_id": "charlie", "new_owner_id": "dex", "amount": "100000000000000"})` 1. `charlie` calls `dex::trade({"have": "wltc", "have_amount": "900100000000", "want": "wbtc", "want_amount": "8000000000"})`. - `dex` calls `wbtc::transfer({"new_owner_id": "charlie", "amount": "8000000000"})` - `dex` calls `wltc::transfer({"new_owner_id": "alex", "amount": "900100000000"})` ## Reference Implementation The full implementation in Rust can be found there: https://github.com/near/near-sdk-rs/blob/master/examples/fungible-token/ft/src/lib.rs NOTES: - All amounts, balances and allowance are limited by U128 (max value `2**128 - 1`). - Token standard uses JSON for serialization of arguments and results. - Amounts in arguments and results have are serialized as Base-10 strings, e.g. `"100"`. This is done to avoid JSON limitation of max integer value of `2**53`. Interface: ```rust /******************/ /* CHANGE METHODS */ /******************/ /// Sets the `allowance` for `escrow_account_id` on the account of the caller of this contract /// (`predecessor_id`) who is the balance owner. pub fn set_allowance(&mut self, escrow_account_id: AccountId, allowance: U128); /// Transfers the `amount` of tokens from `owner_id` to the `new_owner_id`. /// Requirements: /// * `amount` should be a positive integer. /// * `owner_id` should have balance on the account greater or equal than the transfer `amount`. /// * If this function is called by an escrow account (`owner_id != predecessor_account_id`), /// then the allowance of the caller of the function (`predecessor_account_id`) on /// the account of `owner_id` should be greater or equal than the transfer `amount`. pub fn transfer_from(&mut self, owner_id: AccountId, new_owner_id: AccountId, amount: U128); /// Transfer `amount` of tokens from the caller of the contract (`predecessor_id`) to /// `new_owner_id`. /// Act the same was as `transfer_from` with `owner_id` equal to the caller of the contract /// (`predecessor_id`). pub fn transfer(&mut self, new_owner_id: AccountId, amount: U128); /****************/ /* VIEW METHODS */ /****************/ /// Returns total supply of tokens. pub fn get_total_supply(&self) -> U128; /// Returns balance of the `owner_id` account. pub fn get_balance(&self, owner_id: AccountId) -> U128; /// Returns current allowance of `escrow_account_id` for the account of `owner_id`. /// /// NOTE: Other contracts should not rely on this information, because by the moment a contract /// receives this information, the allowance may already be changed by the owner. /// So this method should only be used on the front-end to see the current allowance. pub fn get_allowance(&self, owner_id: AccountId, escrow_account_id: AccountId) -> U128; ``` ## Drawbacks - Current interface doesn't have minting, precision (decimals), naming. But it should be done as extensions, e.g. a Precision extension. - It's not possible to exchange tokens without transferring them to escrow first. - It's not possible to transfer tokens to a contract with a single transaction without setting the allowance first. It should be possible if we introduce `transfer_with` function that transfers tokens and calls escrow contract. It needs to handle result of the execution and contracts have to be aware of this API. ## Future possibilities - Support for multiple token types - Minting and burning - Precision, naming and short token name. ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0141.md ================================================ --- NEP: 141 Title: Fungible Token Standard Author: Evgeny Kuzyakov , Robert Zaremba <@robert-zaremba>, @oysterpack Status: Final DiscussionsTo: https://github.com/near/NEPs/issues/141 Type: Standards Track Category: Contract Created: 03-Mar-2022 Replaces: 21 Requires: 297 --- ## Summary A standard interface for fungible tokens that allows for a normal transfer as well as a transfer and method call in a single transaction. The [storage standard][Storage Management] addresses the needs (and security) of storage staking. The [fungible token metadata standard][FT Metadata] provides the fields needed for ergonomics across dApps and marketplaces. ## Motivation NEAR Protocol uses an asynchronous, sharded runtime. This means the following: - Storage for different contracts and accounts can be located on the different shards. - Two contracts can be executed at the same time in different shards. While this increases the transaction throughput linearly with the number of shards, it also creates some challenges for cross-contract development. For example, if one contract wants to query some information from the state of another contract (e.g. current balance), by the time the first contract receives the balance the real balance can change. In such an async system, a contract can't rely on the state of another contract and assume it's not going to change. Instead the contract can rely on temporary partial lock of the state with a callback to act or unlock, but it requires careful engineering to avoid deadlocks. In this standard we're trying to avoid enforcing locks. A typical approach to this problem is to include an escrow system with allowances. This approach was initially developed for [NEP-21](https://github.com/near/NEPs/pull/21) which is similar to the Ethereum ERC-20 standard. There are a few issues with using an escrow as the only avenue to pay for a service with a fungible token. This frequently requires more than one transaction for common scenarios where fungible tokens are given as payment with the expectation that a method will subsequently be called. For example, an oracle contract might be paid in fungible tokens. A client contract that wishes to use the oracle must either increase the escrow allowance before each request to the oracle contract, or allocate a large allowance that covers multiple calls. Both have drawbacks and ultimately it would be ideal to be able to send fungible tokens and call a method in a single transaction. This concern is addressed in the `ft_transfer_call` method. The power of this comes from the receiver contract working in concert with the fungible token contract in a secure way. That is, if the receiver contract abides by the standard, a single transaction may transfer and call a method. Note: there is no reason why an escrow system cannot be included in a fungible token's implementation, but it is simply not necessary in the core standard. Escrow logic should be moved to a separate contract to handle that functionality. One reason for this is because the [Rainbow Bridge](https://near.org/blog/eth-near-rainbow-bridge/) will be transferring fungible tokens from Ethereum to NEAR, where the token locker (a factory) will be using the fungible token core standard. Prior art: - [ERC-20 standard](https://eips.ethereum.org/EIPS/eip-20) - NEP#4 NEAR NFT standard: [near/neps#4](https://github.com/near/neps/pull/4) Learn about NEP-141: - [Figment Learning Pathway](https://web.archive.org/web/20220621055335/https://learn.figment.io/tutorials/stake-fungible-token) ## Specification ### Guide-level explanation We should be able to do the following: - Initialize contract once. The given total supply will be owned by the given account ID. - Get the total supply. - Transfer tokens to a new user. - Transfer tokens from one user to another. - Transfer tokens to a contract, have the receiver contract call a method and "return" any fungible tokens not used. - Remove state for the key/value pair corresponding with a user's account, withdrawing a nominal balance of Ⓝ that was used for storage. There are a few concepts in the scenarios above: - **Total supply**: the total number of tokens in circulation. - **Balance owner**: an account ID that owns some amount of tokens. - **Balance**: an amount of tokens. - **Transfer**: an action that moves some amount from one account to another account, either an externally owned account or a contract account. - **Transfer and call**: an action that moves some amount from one account to a contract account where the receiver calls a method. - **Storage amount**: the amount of storage used for an account to be "registered" in the fungible token. This amount is denominated in Ⓝ, not bytes, and represents the [storage staked](https://docs.near.org/docs/concepts/storage-staking). Note that precision (the number of decimal places supported by a given token) is not part of this core standard, since it's not required to perform actions. The minimum value is always 1 token. See the [Fungible Token Metadata Standard][FT Metadata] to learn how to support precision/decimals in a standardized way. Given that multiple users will use a Fungible Token contract, and their activity will result in an increased [storage staking](https://docs.near.org/docs/concepts/storage-staking) burden for the contract's account, this standard is designed to interoperate nicely with [the Account Storage standard][Storage Management] for storage deposits and refunds. ### Example scenarios #### Simple transfer Alice wants to send 5 wBTC tokens to Bob. Assumptions - The wBTC token contract is `wbtc`. - Alice's account is `alice`. - Bob's account is `bob`. - The precision ("decimals" in the metadata standard) on wBTC contract is `10^8`. - The 5 tokens is `5 * 10^8` or as a number is `500000000`. ##### High-level explanation Alice needs to issue one transaction to wBTC contract to transfer 5 tokens (multiplied by precision) to Bob. ##### Technical calls 1. `alice` calls `wbtc::ft_transfer({"receiver_id": "bob", "amount": "500000000"})`. #### Token deposit to a contract Alice wants to deposit 1000 DAI tokens to a compound interest contract to earn extra tokens. ##### Assumptions - The DAI token contract is `dai`. - Alice's account is `alice`. - The compound interest contract is `compound`. - The precision ("decimals" in the metadata standard) on DAI contract is `10^18`. - The 1000 tokens is `1000 * 10^18` or as a number is `1000000000000000000000`. - The compound contract can work with multiple token types.
For this example, you may expand this section to see how a previous fungible token standard using escrows would deal with the scenario. ##### High-level explanation (NEP-21 standard) Alice needs to issue 2 transactions. The first one to `dai` to set an allowance for `compound` to be able to withdraw tokens from `alice`. The second transaction is to the `compound` to start the deposit process. Compound will check that the DAI tokens are supported and will try to withdraw the desired amount of DAI from `alice`. - If transfer succeeded, `compound` can increase local ownership for `alice` to 1000 DAI - If transfer fails, `compound` doesn't need to do anything in current example, but maybe can notify `alice` of unsuccessful transfer. ##### Technical calls (NEP-21 standard) 1. `alice` calls `dai::set_allowance({"escrow_account_id": "compound", "allowance": "1000000000000000000000"})`. 2. `alice` calls `compound::deposit({"token_contract": "dai", "amount": "1000000000000000000000"})`. During the `deposit` call, `compound` does the following: 1. makes async call `dai::transfer_from({"owner_id": "alice", "new_owner_id": "compound", "amount": "1000000000000000000000"})`. 2. attaches a callback `compound::on_transfer({"owner_id": "alice", "token_contract": "dai", "amount": "1000000000000000000000"})`.
##### High-level explanation Alice needs to issue 1 transaction, as opposed to 2 with a typical escrow workflow. ##### Technical calls 1. `alice` calls `dai::ft_transfer_call({"receiver_id": "compound", "amount": "1000000000000000000000", "msg": "invest"})`. During the `ft_transfer_call` call, `dai` does the following: 1. makes async call `compound::ft_on_transfer({"sender_id": "alice", "amount": "1000000000000000000000", "msg": "invest"})`. 2. attaches a callback `dai::ft_resolve_transfer({"sender_id": "alice", "receiver_id": "compound", "amount": "1000000000000000000000"})`. 3. compound finishes investing, using all attached fungible tokens `compound::invest({…})` then returns the value of the tokens that weren't used or needed. In this case, Alice asked for the tokens to be invested, so it will return 0. (In some cases a method may not need to use all the fungible tokens, and would return the remainder.) 4. the `dai::ft_resolve_transfer` function receives success/failure of the promise. If success, it will contain the unused tokens. Then the `dai` contract uses simple arithmetic (not needed in this case) and updates the balance for Alice. #### Swapping one token for another via an Automated Market Maker (AMM) like Uniswap Alice wants to swap 5 wrapped NEAR (wNEAR) for BNNA tokens at current market rate, with less than 2% slippage. ##### Assumptions - The wNEAR token contract is `wnear`. - Alice's account is `alice`. - The AMM's contract is `amm`. - BNNA's contract is `bnna`. - The precision ("decimals" in the metadata standard) on wNEAR contract is `10^24`. - The 5 tokens is `5 * 10^24` or as a number is `5000000000000000000000000`. ##### High-level explanation Alice needs to issue one transaction to wNEAR contract to transfer 5 tokens (multiplied by precision) to `amm`, specifying her desired action (swap), her destination token (BNNA) & minimum slippage (<2%) in `msg`. Alice will probably make this call via a UI that knows how to construct `msg` in a way the `amm` contract will understand. However, it's possible that the `amm` contract itself may provide view functions which take desired action, destination token, & slippage as input and return data ready to pass to `msg` for `ft_transfer_call`. For the sake of this example, let's say `amm` implements a view function called `ft_data_to_msg`. Alice needs to attach one yoctoNEAR. This will result in her seeing a confirmation page in her preferred NEAR wallet. NEAR wallet implementations will (eventually) attempt to provide useful information in this confirmation page, so receiver contracts should follow a strong convention in how they format `msg`. We will update this documentation with a recommendation, as community consensus emerges. Altogether then, Alice may take two steps, though the first may be a background detail of the app she uses. ##### Technical calls 1. View `amm::ft_data_to_msg({ action: "swap", destination_token: "bnna", min_slip: 2 })`. Using [NEAR CLI](https://docs.near.org/docs/tools/near-cli): ```shell near view amm ft_data_to_msg \ '{"action": "swap", "destination_token": "bnna", "min_slip": 2}' ``` Then Alice (or the app she uses) will hold onto the result and use it in the next step. Let's say this result is `"swap:bnna,2"`. 2. Call `wnear::ft_on_transfer`. Using NEAR CLI: ```shell near call wnear ft_transfer_call \ '{"receiver_id": "amm", "amount": "5000000000000000000000000", "msg": "swap:bnna,2"}' \ --accountId alice --depositYocto 1 ``` During the `ft_transfer_call` call, `wnear` does the following: 1. Decrease the balance of `alice` and increase the balance of `amm` by 5000000000000000000000000. 2. Makes async call `amm::ft_on_transfer({"sender_id": "alice", "amount": "5000000000000000000000000", "msg": "swap:bnna,2"})`. 3. Attaches a callback `wnear::ft_resolve_transfer({"sender_id": "alice", "receiver_id": "compound", "amount": "5000000000000000000000000"})`. 4. `amm` finishes the swap, either successfully swapping all 5 wNEAR within the desired slippage, or failing. 5. The `wnear::ft_resolve_transfer` function receives success/failure of the promise. Assuming `amm` implements all-or-nothing transfers (as in, it will not transfer less-than-the-specified amount in order to fulfill the slippage requirements), `wnear` will do nothing at this point if the swap succeeded, or it will decrease the balance of `amm` and increase the balance of `alice` by 5000000000000000000000000. ### Reference-level explanation NOTES: - All amounts, balances and allowance are limited by `U128` (max value `2**128 - 1`). - Token standard uses JSON for serialization of arguments and results. - Amounts in arguments and results have are serialized as Base-10 strings, e.g. `"100"`. This is done to avoid JSON limitation of max integer value of `2**53`. - The contract must track the change in storage when adding to and removing from collections. This is not included in this core fungible token standard but instead in the [Storage Standard][Storage Management]. - To prevent the deployed contract from being modified or deleted, it should not have any access keys on its account. #### Interface ##### ft_transfer Simple transfer to a receiver. Requirements: - Caller of the method must attach a deposit of 1 yoctoⓃ for security purposes - Caller must have greater than or equal to the `amount` being requested Arguments: - `receiver_id`: the valid NEAR account receiving the fungible tokens. - `amount`: the number of tokens to transfer, wrapped in quotes and treated like a string, although the number will be stored as an unsigned integer with 128 bits. - `memo` (optional): for use cases that may benefit from indexing or providing information for a transfer. ```ts function ft_transfer( receiver_id: string, amount: string, memo: string | null ): void; ``` ##### ft_transfer_call Transfer tokens and call a method on a receiver contract. A successful workflow will end in a success execution outcome to the callback on the same contract at the method `ft_resolve_transfer`. You can think of this as being similar to attaching native NEAR tokens to a function call. It allows you to attach any Fungible Token in a call to a receiver contract. Requirements: - Caller of the method must attach a deposit of 1 yoctoⓃ for security purposes - Caller must have greater than or equal to the `amount` being requested - The receiving contract must implement `ft_on_transfer` according to the standard. If it does not, FT contract's `ft_resolve_transfer` MUST deal with the resulting failed cross-contract call and roll back the transfer. - Contract MUST implement the behavior described in `ft_resolve_transfer` Arguments: - `receiver_id`: the valid NEAR account receiving the fungible tokens. - `amount`: the number of tokens to transfer, wrapped in quotes and treated like a string, although the number will be stored as an unsigned integer with 128 bits. - `memo` (optional): for use cases that may benefit from indexing or providing information for a transfer. - `msg`: specifies information needed by the receiving contract in order to properly handle the transfer. Can indicate both a function to call and the parameters to pass to that function. ```ts function ft_transfer_call( receiver_id: string, amount: string, memo: string | null, msg: string ): Promise; ``` ##### ft_on_transfer This function is implemented on the receiving contract. As mentioned, the `msg` argument contains information necessary for the receiving contract to know how to process the request. This may include method names and/or arguments. Returns a value, or a promise which resolves with a value. The value is the number of unused tokens in string form. For instance, if `amount` is 10 but only 9 are needed, it will return "1". ```ts function ft_on_transfer(sender_id: string, amount: string, msg: string): string; ``` ### View Methods ##### ft_total_supply Returns the total supply of fungible tokens as a string representing the value as an unsigned 128-bit integer. ```js function ft_total_supply(): string ``` ##### ft_balance_of Returns the balance of an account in string form representing a value as an unsigned 128-bit integer. If the account doesn't exist must returns `"0"`. ```ts function ft_balance_of(account_id: string): string; ``` ##### ft_resolve_transfer The following behavior is required, but contract authors may name this function something other than the conventional `ft_resolve_transfer` used here. Finalize an `ft_transfer_call` chain of cross-contract calls. The `ft_transfer_call` process: 1. Sender calls `ft_transfer_call` on FT contract 2. FT contract transfers `amount` tokens from sender to receiver 3. FT contract calls `ft_on_transfer` on receiver contract 4. [receiver contract may make other cross-contract calls] 5. FT contract resolves promise chain with `ft_resolve_transfer`, and may refund sender some or all of original `amount` Requirements: - Contract MUST forbid calls to this function by any account except self - If promise chain failed, contract MUST revert token transfer - If promise chain resolves with a non-zero amount given as a string, contract MUST return this amount of tokens to `sender_id` Arguments: - `sender_id`: the sender of `ft_transfer_call` - `receiver_id`: the `receiver_id` argument given to `ft_transfer_call` - `amount`: the `amount` argument given to `ft_transfer_call` Returns a string representing a string version of an unsigned 128-bit integer of how many total tokens were spent by sender_id. Example: if sender calls `ft_transfer_call({ "amount": "100" })`, but `receiver_id` only uses 80, `ft_on_transfer` will resolve with `"20"`, and `ft_resolve_transfer` will return `"80"`. ```ts function ft_resolve_transfer( sender_id: string, receiver_id: string, amount: string ): string; ``` ### Events Standard interfaces for FT contract actions that extend [NEP-297](nep-0297.md) NEAR and third-party applications need to track `mint`, `transfer`, `burn` events for all FT-driven apps consistently. This extension addresses that. Keep in mind that applications, including NEAR Wallet, could require implementing additional methods, such as [`ft_metadata`][FT Metadata], to display the FTs correctly. ### Event Interface Fungible Token Events MUST have `standard` set to `"nep141"`, standard version set to `"1.0.0"`, `event` value is one of `ft_mint`, `ft_burn`, `ft_transfer`, and `data` must be of one of the following relevant types: `FtMintLog[] | FtTransferLog[] | FtBurnLog[]`: ```ts interface FtEventLogData { standard: "nep141"; version: "1.0.0"; event: "ft_mint" | "ft_burn" | "ft_transfer"; data: FtMintLog[] | FtTransferLog[] | FtBurnLog[]; } ``` ```ts // An event log to capture tokens minting // Arguments // * `owner_id`: "account.near" // * `amount`: the number of tokens to mint, wrapped in quotes and treated // like a string, although the number will be stored as an unsigned integer // with 128 bits. // * `memo`: optional message interface FtMintLog { owner_id: string; amount: string; memo?: string; } // An event log to capture tokens burning // Arguments // * `owner_id`: owner of tokens to burn // * `amount`: the number of tokens to burn, wrapped in quotes and treated // like a string, although the number will be stored as an unsigned integer // with 128 bits. // * `memo`: optional message interface FtBurnLog { owner_id: string; amount: string; memo?: string; } // An event log to capture tokens transfer // Arguments // * `old_owner_id`: "owner.near" // * `new_owner_id`: "receiver.near" // * `amount`: the number of tokens to transfer, wrapped in quotes and treated // like a string, although the number will be stored as an unsigned integer // with 128 bits. // * `memo`: optional message interface FtTransferLog { old_owner_id: string; new_owner_id: string; amount: string; memo?: string; } ``` ### Event Examples Batch mint: ```js EVENT_JSON:{ "standard": "nep141", "version": "1.0.0", "event": "ft_mint", "data": [ {"owner_id": "foundation.near", "amount": "500"} ] } ``` Batch transfer: ```js EVENT_JSON:{ "standard": "nep141", "version": "1.0.0", "event": "ft_transfer", "data": [ {"old_owner_id": "from.near", "new_owner_id": "to.near", "amount": "42", "memo": "hi hello bonjour"}, {"old_owner_id": "user1.near", "new_owner_id": "user2.near", "amount": "7500"} ] } ``` Batch burn: ```js EVENT_JSON:{ "standard": "nep141", "version": "1.0.0", "event": "ft_burn", "data": [ {"owner_id": "foundation.near", "amount": "100"}, ] } ``` ### Further Event Methods Note that the example events covered above cover two different kinds of events: 1. Events that are not specified in the FT Standard (`ft_mint`, `ft_burn`) 2. An event that is covered in the [FT Core Standard][FT Core]. (`ft_transfer`) Please feel free to open pull requests for extending the events standard detailed here as needs arise. ## Reference Implementation The `near-contract-standards` cargo package of the [Near Rust SDK](https://github.com/near/near-sdk-rs) contain the following implementations of NEP-141: - [Minimum Viable Interface](https://github.com/near/near-sdk-rs/blob/master/near-contract-standards/src/fungible_token/core.rs) - The [Core Fungible Token Implementation](https://github.com/near/near-sdk-rs/blob/master/near-contract-standards/src/fungible_token/core_impl.rs) - [Optional Fungible Token Events](https://github.com/near/near-sdk-rs/blob/master/near-contract-standards/src/fungible_token/events.rs) - [Core Fungible Token tests](https://github.com/near/near-sdk-rs/blob/master/examples/fungible-token/tests/workspaces.rs) ## Drawbacks - The `msg` argument to `ft_transfer` and `ft_transfer_call` is freeform, which may necessitate conventions. - The paradigm of an escrow system may be familiar to developers and end users, and education on properly handling this in another contract may be needed. ## Future possibilities - Support for multiple token types - Minting and burning ## History See also the discussions: - [Fungible token core](https://github.com/near/NEPs/discussions/146#discussioncomment-298943) - [Fungible token metadata](https://github.com/near/NEPs/discussions/148) - [Storage standard](https://github.com/near/NEPs/discussions/145) ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). [Storage Management]: https://github.com/near/NEPs/blob/master/neps/nep-0145.md [FT Metadata]: https://github.com/near/NEPs/blob/master/neps/nep-0148.md [FT Core]: https://github.com/near/NEPs/blob/master/neps/nep-0141.md ================================================ FILE: neps/nep-0145.md ================================================ --- NEP: 145 Title: Storage Management Author: Evgeny Kuzyakov , @oysterpack Status: Final DiscussionsTo: https://github.com/near/NEPs/discussions/145 Type: Standards Track Category: Contract Created: 03-Mar-2022 --- ## Summary NEAR uses [storage staking] which means that a contract account must have sufficient balance to cover all storage added over time. This standard provides a uniform way to pass storage costs onto users. ## Motivation It allows accounts and contracts to: 1. Check an account's storage balance. 2. Determine the minimum storage needed to add account information such that the account can interact as expected with a contract. 3. Add storage balance for an account; either one's own or another. 4. Withdraw some storage deposit by removing associated account data from the contract and then making a call to remove unused deposit. 5. Unregister an account to recover full storage balance. [storage staking]: https://docs.near.org/concepts/storage/storage-staking ## Rationale and alternatives Prior art: - A previous fungible token standard ([NEP-21](https://github.com/near/NEPs/pull/21)) highlighting how [storage was paid](https://github.com/near/near-sdk-rs/blob/1d3535bd131b68f97a216e643ad1cba19e16dddf/examples/fungible-token/src/lib.rs#L92-L113) for when increasing the allowance of an escrow system. ### Example scenarios To show the flexibility and power of this standard, let's walk through two example contracts. 1. A simple Fungible Token contract which uses Storage Management in "registration only" mode, where the contract only adds storage on a user's first interaction. 1. Account registers self 2. Account registers another 3. Unnecessary attempt to re-register 4. Force-closure of account 5. Graceful closure of account 2. A social media contract, where users can add more data to the contract over time. 1. Account registers self with more than minimum required 2. Unnecessary attempt to re-register using `registration_only` param 3. Attempting to take action which exceeds paid-for storage; increasing storage deposit 4. Removing storage and reclaiming excess deposit ### Example 1: Fungible Token Contract Imagine a [fungible token][FT Core] contract deployed at `ft`. Let's say this contract saves all user balances to a Map data structure internally, and adding a key for a new user requires 0.00235Ⓝ. This contract therefore uses the Storage Management standard to pass this cost onto users, so that a new user must effectively pay a registration fee to interact with this contract of 0.00235Ⓝ, or 2350000000000000000000 yoctoⓃ ([yocto](https://www.metricconversion.us/prefixes.htm) = 10-24). For this contract, `storage_balance_bounds` will be: ```json { "min": "2350000000000000000000", "max": "2350000000000000000000" } ``` This means a user must deposit 0.00235Ⓝ to interact with this contract, and that attempts to deposit more than this will have no effect (attached deposits will be immediately refunded). Let's follow two users, Alice with account `alice` and Bob with account `bob`, as they interact with `ft` through the following scenarios: 1. Alice registers herself 2. Alice registers Bob 3. Alice tries to register Bob again 4. Alice force-closes her account 5. Bob gracefully closes his account #### 1. Account pays own registration fee ##### High-level explanation 1. Alice checks if she is registered with the `ft` contract. 2. Alice determines the needed registration fee to register with the `ft` contract. 3. Alice issues a transaction to deposit Ⓝ for her account. ##### Technical calls 1. Alice queries a view-only method to determine if she already has storage on this contract with `ft::storage_balance_of({"account_id": "alice"})`. Using [NEAR CLI](https://docs.near.org/tools/near-cli) to make this view call, the command would be: ```shell near view ft storage_balance_of '{"account_id": "alice"}' ``` The response: ```shell null ``` 2. Alice uses [NEAR CLI](https://docs.near.org/docs/tools/near-cli) to make a view call. ```shell near view ft storage_balance_bounds ``` As mentioned above, this will show that both `min` and `max` are both 2350000000000000000000 yoctoⓃ. 3. Alice converts this yoctoⓃ amount to 0.00235 Ⓝ, then calls `ft::storage_deposit` with this attached deposit. Using NEAR CLI: ```shell near call ft storage_deposit '' --accountId alice --amount 0.00235 ``` The result: ```json { "total": "2350000000000000000000", "available": "0" } ``` #### 2. Account pays for another account's storage Alice wishes to eventually send `ft` tokens to Bob who is not registered. She decides to pay for Bob's storage. ##### High-level explanation Alice issues a transaction to deposit Ⓝ for Bob's account. ##### Technical calls Alice calls `ft::storage_deposit({"account_id": "bob"})` with the attached deposit of '0.00235'. Using NEAR CLI the command would be: ```shell near call ft storage_deposit '{"account_id": "bob"}' --accountId alice --amount 0.00235 ``` The result: ```json { "total": "2350000000000000000000", "available": "0" } ``` #### 3. Unnecessary attempt to register already-registered account Alice accidentally makes the same call again, and even misses a leading zero in her deposit amount. ```shell near call ft storage_deposit '{"account_id": "bob"}' --accountId alice --amount 0.0235 ``` The result: ```json { "total": "2350000000000000000000", "available": "0" } ``` Additionally, Alice will be refunded the 0.0235Ⓝ she attached, because the `storage_deposit_bounds.max` specifies that Bob's account cannot have a total balance larger than 0.00235Ⓝ. #### 4. Account force-closes registration Alice decides she doesn't care about her `ft` tokens and wants to forcibly recover her registration fee. If the contract permits this operation, her remaining `ft` tokens will either be burned or transferred to another account, which she may or may not have the ability to specify prior to force-closing. ##### High-level explanation Alice issues a transaction to unregister her account and recover the Ⓝ from her registration fee. She must attach 1 yoctoⓃ, expressed in Ⓝ as `.000000000000000000000001`. ##### Technical calls Alice calls `ft::storage_unregister({"force": true})` with a 1 yoctoⓃ deposit. Using NEAR CLI the command would be: ```shell near call ft storage_unregister '{ "force": true }' --accountId alice --depositYocto 1 ``` The result: ```shell true ``` #### 5. Account gracefully closes registration Bob wants to close his account, but has a non-zero balance of `ft` tokens. ##### High-level explanation 1. Bob tries to gracefully close his account, calling `storage_unregister()` without specifying `force=true`. This results in an intelligible error that tells him why his account can't yet be unregistered gracefully. 2. Bob sends all of his `ft` tokens to a friend. 3. Bob retries to gracefully close his account. It works. ##### Technical calls 1. Bob calls `ft::storage_unregister()` with a 1 yoctoⓃ deposit. Using NEAR CLI the command would be: ```shell near call ft storage_unregister '' --accountId bob --depositYocto 1 ``` It fails with a message like "Cannot gracefully close account with positive remaining balance; bob has balance N" 2. Bob transfers his tokens to a friend using `ft_transfer` from the [Fungible Token Core][FT Core] standard. 3. Bob tries the call from Step 1 again. It works. ### Example 2: Social Media Contract Imagine a social media smart contract which passes storage costs onto users for posts and follower data. Let's say this contract is deployed at account `social`. Like the Fungible Token contract example above, the `storage_balance_bounds.min` is 0.00235, because this contract will likewise add a newly-registered user to an internal Map. However, this contract sets no `storage_balance_bounds.max`, since users can add more data to the contract over time and must cover the cost for this storage. So for this contract, `storage_balance_bounds` will return: ```json { "min": "2350000000000000000000", "max": null } ``` Let's follow a user, Alice with account `alice`, as she interacts with `social` through the following scenarios: 1. Registration 2. Unnecessary attempt to re-register using `registration_only` param 3. Attempting to take action which exceeds paid-for storage; increasing storage deposit 4. Removing storage and reclaiming excess deposit #### 1. Account registers with `social` ##### High-level explanation Alice issues a transaction to deposit Ⓝ for her account. While the `storage_balance_bounds.min` for this contract is 0.00235Ⓝ, the frontend she uses suggests adding 0.1Ⓝ, so that she can immediately start adding data to the app, rather than _only_ registering. ##### Technical calls Using NEAR CLI: ```shell near call social storage_deposit '' --accountId alice --amount 0.1 ``` The result: ```json { "total": "100000000000000000000000", "available": "97650000000000000000000" } ``` Here we see that she has deposited 0.1Ⓝ and that 0.00235 of it has been used to register her account, and is therefore locked by the contract. The rest is available to facilitate interaction with the contract, but could also be withdrawn by Alice by using `storage_withdraw`. #### 2. Unnecessary attempt to re-register using `registration_only` param ##### High-level explanation Alice can't remember if she already registered and re-sends the call, using the `registration_only` param to ensure she doesn't attach another 0.1Ⓝ. ##### Technical calls Using NEAR CLI: ```shell near call social storage_deposit '{"registration_only": true}' --accountId alice --amount 0.1 ``` The result: ```json { "total": "100000000000000000000000", "available": "97650000000000000000000" } ``` Additionally, Alice will be refunded the extra 0.1Ⓝ that she just attached. This makes it easy for other contracts to always attempt to register users while performing batch transactions without worrying about errors or lost deposits. Note that if Alice had not included `registration_only`, she would have ended up with a `total` of 0.2Ⓝ. #### 3. Account increases storage deposit Assumption: `social` has a `post` function which allows creating a new post with free-form text. Alice has used almost all of her available storage balance. She attempts to call `post` with a large amount of text, and the transaction aborts because she needs to pay for more storage first. Note that applications will probably want to avoid this situation in the first place by prompting users to top up storage deposits sufficiently before available balance runs out. ##### High-level explanation 1. Alice issues a transaction, let's say `social.post`, and it fails with an intelligible error message to tell her that she has an insufficient storage balance to cover the cost of the operation 2. Alice issues a transaction to increase her storage balance 3. Alice retries the initial transaction and it succeeds ##### Technical calls 1. This is outside the scope of this spec, but let's say Alice calls `near call social post '{ "text": "very long message" }'`, and that this fails with a message saying something like "Insufficient storage deposit for transaction. Please call `storage_deposit` and attach at least 0.1 NEAR, then try again." 2. Alice deposits the proper amount in a transaction by calling `social::storage_deposit` with the attached deposit of '0.1'. Using NEAR CLI: ```shell near call social storage_deposit '' --accountId alice --amount 0.1 ``` The result: ```json { "total": "200000000000000000000000", "available": "100100000000000000000000" } ``` 3. Alice tries the initial `near call social post` call again. It works. #### 4. Removing storage and reclaiming excess deposit Assumption: Alice has more deposited than she is using. ##### High-level explanation 1. Alice views her storage balance and sees that she has extra. 2. Alice withdraws her excess deposit. ##### Technical calls 1. Alice queries `social::storage_balance_of({ "account_id": "alice" })`. With NEAR CLI: ```shell near view social storage_balance_of '{"account_id": "alice"}' ``` Response: ```json { "total": "200000000000000000000000", "available": "100100000000000000000000" } ``` 2. Alice calls `storage_withdraw` with a 1 yoctoⓃ deposit. NEAR CLI command: ```shell near call social storage_withdraw '{"amount": "100100000000000000000000"}' \ --accountId alice --depositYocto 1 ``` Result: ```json { "total": "200000000000000000000000", "available": "0" } ``` ## Specification NOTES: - All amounts, balances and allowance are limited by `U128` (max value 2128 - 1). - This storage standard uses JSON for serialization of arguments and results. - Amounts in arguments and results are serialized as Base-10 strings, e.g. `"100"`. This is done to avoid JSON limitation of max integer value of 253. - To prevent the deployed contract from being modified or deleted, it should not have any access keys on its account. ### Interface ```ts // The structure that will be returned for the methods: // * `storage_deposit` // * `storage_withdraw` // * `storage_balance_of` // The `total` and `available` values are string representations of unsigned // 128-bit integers showing the balance of a specific account in yoctoⓃ. type StorageBalance = { total: string; available: string; }; // The below structure will be returned for the method `storage_balance_bounds`. // Both `min` and `max` are string representations of unsigned 128-bit integers. // // `min` is the amount of tokens required to start using this contract at all // (eg to register with the contract). If a new contract user attaches `min` // NEAR to a `storage_deposit` call, subsequent calls to `storage_balance_of` // for this user must show their `total` equal to `min` and `available=0` . // // A contract may implement `max` equal to `min` if it only charges for initial // registration, and does not adjust per-user storage over time. A contract // which implements `max` must refund deposits that would increase a user's // storage balance beyond this amount. type StorageBalanceBounds = { min: string; max: string | null; }; /************************************/ /* CHANGE METHODS on fungible token */ /************************************/ // Payable method that receives an attached deposit of Ⓝ for a given account. // // If `account_id` is omitted, the deposit MUST go toward predecessor account. // If provided, deposit MUST go toward this account. If invalid, contract MUST // panic. // // If `registration_only=true`, contract MUST refund above the minimum balance // if the account wasn't registered and refund full deposit if already // registered. // // The `storage_balance_of.total` + `attached_deposit` in excess of // `storage_balance_bounds.max` must be refunded to predecessor account. // // Returns the StorageBalance structure showing updated balances. function storage_deposit( account_id: string | null, registration_only: boolean | null ): StorageBalance {} // Withdraw specified amount of available Ⓝ for predecessor account. // // This method is safe to call. It MUST NOT remove data. // // `amount` is sent as a string representing an unsigned 128-bit integer. If // omitted, contract MUST refund full `available` balance. If `amount` exceeds // predecessor account's available balance, contract MUST panic. // // If predecessor account not registered, contract MUST panic. // // MUST require exactly 1 yoctoNEAR attached balance to prevent restricted // function-call access-key call (UX wallet security) // // Returns the StorageBalance structure showing updated balances. function storage_withdraw(amount: string | null): StorageBalance {} // Unregisters the predecessor account and returns the storage NEAR deposit. // // If the predecessor account is not registered, the function MUST return // `false` without panic. // // If `force=true` the function SHOULD ignore existing account data, such as // non-zero balances on an FT contract (that is, it should burn such balances), // and close the account. Contract MAY panic if it doesn't support forced // unregistration, or if it can't force unregister for the particular situation // (example: too much data to delete at once). // // If `force=false` or `force` is omitted, the contract MUST panic if caller // has existing account data, such as a positive registered balance (eg token // holdings). // // MUST require exactly 1 yoctoNEAR attached balance to prevent restricted // function-call access-key call (UX wallet security) // // Returns `true` iff the account was successfully unregistered. // Returns `false` iff account was not registered before. function storage_unregister(force: boolean | null): boolean {} /****************/ /* VIEW METHODS */ /****************/ // Returns minimum and maximum allowed balance amounts to interact with this // contract. See StorageBalanceBounds. function storage_balance_bounds(): StorageBalanceBounds {} // Returns the StorageBalance structure of the valid `account_id` // provided. Must panic if `account_id` is invalid. // // If `account_id` is not registered, must return `null`. function storage_balance_of(account_id: string): StorageBalance | null {} ``` ## Reference Implementation The `near-contract-standards` cargo package of the [Near Rust SDK](https://github.com/near/near-sdk-rs) contain the following implementations of NEP-145: - [Minimum Viable Interface](https://github.com/near/near-sdk-rs/blob/master/near-contract-standards/src/storage_management/mod.rs#L20) - [Implementation](https://github.com/near/near-sdk-rs/blob/master/near-contract-standards/src/storage_management/mod.rs) ## Drawbacks - The idea may confuse contract developers at first until they understand how a system with storage staking works. - Some folks in the community would rather see the storage deposit only done for the sender. That is, that no one else should be able to add storage for another user. This stance wasn't adopted in this standard, but others may have similar concerns in the future. ## Future possibilities - Ideally, contracts will update available balance for all accounts every time the NEAR blockchain's configured storage-cost-per-byte is reduced. That they _must_ do so is not enforced by this current standard. ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). [FT Core]: https://github.com/near/NEPs/blob/master/neps/nep-0141.md ================================================ FILE: neps/nep-0148.md ================================================ --- NEP: 148 Title: Fungible Token Metadata Author: Robert Zaremba , Evgeny Kuzyakov , @oysterpack Status: Final DiscussionsTo: https://github.com/near/NEPs/discussions/148 Type: Standards Track Category: Contract Created: 03-Mar-2022 Requires: 141 --- ## Summary An interface for a fungible token's metadata. The goal is to keep the metadata future-proof as well as lightweight. This will be important to dApps needing additional information about an FT's properties, and broadly compatible with other tokens standards such that the [NEAR Rainbow Bridge](https://near.org/blog/eth-near-rainbow-bridge/) can move tokens between chains. ## Motivation Custom fungible tokens play a major role in decentralized applications today. FTs can contain custom properties to differentiate themselves from other tokens or contracts in the ecosystem. In NEAR, many common properties can be stored right on-chain. Other properties are best stored off-chain or in a decentralized storage platform, in order to save on storage costs and allow rapid community experimentation. ## Rationale and alternatives As blockchain technology advances, it becomes increasingly important to provide backwards compatibility and a concept of a spec. This standard encompasses all of these concerns. Prior art: - [EIP-1046](https://eips.ethereum.org/EIPS/eip-1046) - [OpenZeppelin's ERC-721 Metadata standard](https://docs.openzeppelin.com/contracts/5.x/api/token/ERC721#IERC721Metadata) also helped, although it's for non-fungible tokens. ## Specification A fungible token smart contract allows for discoverable properties. Some properties can be determined by other contracts on-chain, or return in view method calls. Others can only be determined by an oracle system to be used on-chain, or by a frontend with the ability to access a linked reference file. ### Examples scenario #### Token provides metadata upon deploy and initialization Alice deploys a wBTC fungible token contract. ##### Assumptions - The wBTC token contract is `wbtc`. - Alice's account is `alice`. - The precision ("decimals" in this metadata standard) on wBTC contract is `10^8`. ##### High-level explanation Alice issues a transaction to deploy and initialize the fungible token contract, providing arguments to the initialization function that set metadata fields. ##### Technical calls 1. `alice` deploys a contract and calls `wbtc::new` with all metadata. If this deploy and initialization were done using [NEAR CLI](https://docs.near.org/tools/near-cli) the command would be: ```shell near deploy wbtc --wasmFile res/ft.wasm --initFunction new --initArgs '{ "owner_id": "wbtc", "total_supply": "100000000000000", "metadata": { "spec": "ft-1.0.0", "name": "Wrapped Bitcoin", "symbol": "WBTC", "icon": "data:image/svg+xml,%3C…", "reference": "https://example.com/wbtc.json", "reference_hash": "AK3YRHqKhCJNmKfV6SrutnlWW/icN5J8NUPtKsNXR1M=", "decimals": 8 }}' --accountId alice ``` ## Reference-level explanation A fungible token contract implementing the metadata standard shall contain a function named `ft_metadata`. ```ts function ft_metadata(): FungibleTokenMetadata {} ``` ##### Interface ```ts type FungibleTokenMetadata = { spec: string; name: string; symbol: string; icon: string | null; reference: string | null; reference_hash: string | null; decimals: number; }; ``` An implementing contract MUST include the following fields on-chain - `spec`: a string. Should be `ft-1.0.0` to indicate that a Fungible Token contract adheres to the current versions of this Metadata and the [Fungible Token Core][FT Core] specs. This will allow consumers of the Fungible Token to know if they support the features of a given contract. - `name`: the human-readable name of the token. - `symbol`: the abbreviation, like wETH or AMPL. - `decimals`: used in frontends to show the proper significant digits of a token. This concept is explained well in this [OpenZeppelin post](https://docs.openzeppelin.com/contracts/3.x/erc20#a-note-on-decimals). An implementing contract MAY include the following fields on-chain - `icon`: a small image associated with this token. Must be a [data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URIs), to help consumers display it quickly while protecting user data. Recommendation: use [optimized SVG](https://codepen.io/tigt/post/optimizing-svgs-in-data-uris), which can result in high-resolution images with only 100s of bytes of [storage cost](https://docs.near.org/concepts/storage/storage-staking). (Note that these storage costs are incurred to the token owner/deployer, but that querying these icons is a very cheap & cacheable read operation for all consumers of the contract and the RPC nodes that serve the data.) Recommendation: create icons that will work well with both light-mode and dark-mode websites by either using middle-tone color schemes, or by [embedding `media` queries in the SVG](https://timkadlec.com/2013/04/media-queries-within-svg/). - `reference`: a link to a valid JSON file containing various keys offering supplementary details on the token. Example: `/ipfs/QmdmQXB2mzChmMeKY47C43LxUdg1NDJ5MWcKMKxDu7RgQm`, `https://example.com/token.json`, etc. If the information given in this document conflicts with the on-chain attributes, the values in `reference` shall be considered the source of truth. - `reference_hash`: the base64-encoded sha256 hash of the JSON file contained in the `reference` field. This is to guard against off-chain tampering. ## Reference Implementation The `near-contract-standards` cargo package of the [Near Rust SDK](https://github.com/near/near-sdk-rs) contain the following implementations of NEP-148: - [Implementation](https://github.com/near/near-sdk-rs/blob/master/near-contract-standards/src/fungible_token/metadata.rs) ## Drawbacks - It could be argued that `symbol` and even `name` could belong as key/values in the `reference` JSON object. - Enforcement of `icon` to be a data URL rather than a link to an HTTP endpoint that could contain privacy-violating code cannot be done on deploy or update of contract metadata, and must be done on the consumer/app side when displaying token data. - If on-chain icon uses a data URL or is not set but the document given by `reference` contains a privacy-violating `icon` URL, consumers & apps of this data should not naïvely display the `reference` version, but should prefer the safe version. This is technically a violation of the "`reference` setting wins" policy described above. ## Future possibilities - Detailed conventions that may be enforced for versions. - A fleshed out schema for what the `reference` object should contain. ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). [FT Core]: https://github.com/near/NEPs/blob/master/neps/nep-0141.md ================================================ FILE: neps/nep-0171.md ================================================ --- NEP: 171 Title: Non Fungible Token Standard Author: Mike Purvis , Evgeny Kuzyakov , @oysterpack Status: Final DiscussionsTo: https://github.com/near/NEPs/discussions/171 Type: Standards Track Category: Contract Version: 1.2.0 Created: 03-Mar-2022 Updated: 13-Mar-2023 Requires: 297 --- ## Summary A standard interface for non-fungible tokens (NFTs). That is, tokens which each have a unique ID. ## Motivation In the three years since [ERC-721] was ratified by the Ethereum community, Non-Fungible Tokens have proven themselves as an incredible new opportunity across a wide array of disciplines: collectibles, art, gaming, finance, virtual reality, real estate, and more. This standard builds off the lessons learned in this early experimentation, and pushes the possibilities further by harnessing unique attributes of the NEAR blockchain: - an asynchronous, sharded runtime, meaning that two contracts can be executed at the same time in different shards - a [storage staking] model that separates [gas] fees from the storage demands of the network, enabling greater on-chain storage (see [Metadata] extension) and ultra-low transaction fees Given these attributes, this NFT standard can accomplish with one user interaction things for which other blockchains need two or three. Most noteworthy is `nft_transfer_call`, by which a user can essentially attach a token to a call to a separate contract. An example scenario: - An [Exquisite Corpse](https://en.wikipedia.org/wiki/Exquisite_corpse) contract allows three drawings to be submitted, one for each section of a final composition, to be minted as its own NFT and sold on a marketplace, splitting royalties amongst the original artists. - Alice draws the top third and submits it, Bob the middle third, and Carol follows up with the bottom third. Since they each use `nft_transfer_call` to both transfer their NFT to the Exquisite Corpse contract as well as call a `submit` method on it, the call from Carol can automatically kick off minting a composite NFT from the three submissions, as well as listing this composite NFT in a marketplace. - When Dan attempts to also call `nft_transfer_call` to submit an unneeded top third of the drawing, the Exquisite Corpse contract can throw an error, and the transfer will be rolled back so that Bob maintains ownership of his NFT. While this is already flexible and powerful enough to handle all sorts of existing and new use-cases, apps such as marketplaces may still benefit from the [Approval Management] extension. Prior art: - [ERC-721] - [EIP-1155 for multi-tokens](https://eips.ethereum.org/EIPS/eip-1155#non-fungible-tokens) - [NEAR's Fungible Token Standard][FT Core], which first pioneered the "transfer and call" technique ## Rationale and alternatives - Why is this design the best in the space of possible designs? - What other designs have been considered and what is the rationale for not choosing them? - What is the impact of not doing this? ## Specification **NOTES**: - All amounts, balances and allowance are limited by `U128` (max value `2**128 - 1`). - Token standard uses JSON for serialization of arguments and results. - Amounts in arguments and results are serialized as Base-10 strings, e.g. `"100"`. This is done to avoid JSON limitation of max integer value of `2**53`. - The contract must track the change in storage when adding to and removing from collections. This is not included in this core fungible token standard but instead in the [Storage Standard][Storage Management]. - To prevent the deployed contract from being modified or deleted, it should not have any access keys on its account. ### NFT Interface ```ts // The base structure that will be returned for a token. If contract is using // extensions such as Approval Management, Metadata, or other // attributes may be included in this structure. type Token = { token_id: string, owner_id: string, } /******************/ /* CHANGE METHODS */ /******************/ // Simple transfer. Transfer a given `token_id` from current owner to // `receiver_id`. // // Requirements // * Caller of the method must attach a deposit of 1 yoctoⓃ for security purposes // * Contract MUST panic if called by someone other than token owner or, // if using Approval Management, one of the approved accounts // * `approval_id` is for use with Approval Management extension, see // that document for full explanation. // * If using Approval Management, contract MUST nullify approved accounts on // successful transfer. // // Arguments: // * `receiver_id`: the valid NEAR account receiving the token // * `token_id`: the token to transfer // * `approval_id`: expected approval ID. A number smaller than // 2^53, and therefore representable as JSON. See Approval Management // standard for full explanation. // * `memo` (optional): for use cases that may benefit from indexing or // providing information for a transfer function nft_transfer( receiver_id: string, token_id: string, approval_id: number|null, memo: string|null, ) {} // Returns `true` if the token was transferred from the sender's account. // Transfer token and call a method on a receiver contract. A successful // workflow will end in a success execution outcome to the callback on the NFT // contract at the method `nft_resolve_transfer`. // // You can think of this as being similar to attaching native NEAR tokens to a // function call. It allows you to attach any Non-Fungible Token in a call to a // receiver contract. // // Requirements: // * Caller of the method must attach a deposit of 1 yoctoⓃ for security // purposes // * Contract MUST panic if called by someone other than token owner or, // if using Approval Management, one of the approved accounts // * The receiving contract must implement `nft_on_transfer` according to the // standard. If it does not, FT contract's `nft_resolve_transfer` MUST deal // with the resulting failed cross-contract call and roll back the transfer. // * Contract MUST implement the behavior described in `nft_resolve_transfer` // * `approval_id` is for use with Approval Management extension, see // that document for full explanation. // * If using Approval Management, contract MUST nullify approved accounts on // successful transfer. // // Arguments: // * `receiver_id`: the valid NEAR account receiving the token. // * `token_id`: the token to send. // * `approval_id`: expected approval ID. A number smaller than // 2^53, and therefore representable as JSON. See Approval Management // standard for full explanation. // * `memo` (optional): for use cases that may benefit from indexing or // providing information for a transfer. // * `msg`: specifies information needed by the receiving contract in // order to properly handle the transfer. Can indicate both a function to // call and the parameters to pass to that function. function nft_transfer_call( receiver_id: string, token_id: string, approval_id: number|null, memo: string|null, msg: string, ): Promise {} /****************/ /* VIEW METHODS */ /****************/ // Returns the token with the given `token_id` or `null` if no such token. function nft_token(token_id: string): Token|null {} ``` The following behavior is required, but contract authors may name this function something other than the conventional `nft_resolve_transfer` used here. ```ts // Finalize an `nft_transfer_call` chain of cross-contract calls. // // The `nft_transfer_call` process: // // 1. Sender calls `nft_transfer_call` on NFT contract // 2. NFT contract transfers token from sender to receiver // 3. NFT contract calls `nft_on_transfer` on receiver contract // 4+. [receiver contract may make other cross-contract calls] // N. NFT contract resolves promise chain with `nft_resolve_transfer`, and may // transfer token back to sender // // Requirements: // * Contract MUST forbid calls to this function by any account except self // * If promise chain failed, contract MUST revert token transfer // * If promise chain resolves with `true`, contract MUST return token to // `owner_id` // // Arguments: // * `owner_id`: the original owner of the NFT. // * `receiver_id`: the `receiver_id` argument given to `nft_transfer_call` // * `token_id`: the `token_id` argument given to `nft_transfer_call` // * `approved_account_ids `: if using Approval Management, contract MUST provide // record of original approved accounts in this argument, and restore these // approved accounts and their approval IDs in case of revert. // // Returns true if token was successfully transferred to `receiver_id`. function nft_resolve_transfer( owner_id: string, receiver_id: string, token_id: string, approved_account_ids: null|Record, ): boolean {} ``` ### Receiver Interface Contracts which want to make use of `nft_transfer_call` must implement the following: ```ts // Take some action after receiving a non-fungible token // // Requirements: // * Contract MUST restrict calls to this function to a set of whitelisted NFT // contracts // // Arguments: // * `sender_id`: the sender of `nft_transfer_call` // * `previous_owner_id`: the account that owned the NFT prior to it being // transferred to this contract, which can differ from `sender_id` if using // Approval Management extension // * `token_id`: the `token_id` argument given to `nft_transfer_call` // * `msg`: information necessary for this contract to know how to process the // request. This may include method names and/or arguments. // // Returns true if token should be returned to `sender_id` function nft_on_transfer( sender_id: string, previous_owner_id: string, token_id: string, msg: string, ): Promise; ``` ### Events NEAR and third-party applications need to track mint, transfer, burn, metadata update, and contract metadata update events for all NFT-driven apps consistently. This extension addresses that. Keep in mind that applications, including NEAR Wallet, could require implementing additional methods to display the NFTs correctly, such as [`nft_metadata`][Metadata] and [`nft_tokens_for_owner`][NFT Enumeration]). #### Events Interface Non-Fungible Token Events MUST have `standard` set to `"nep171"`, standard version set to `"1.2.0"`, `event` value is one of `nft_mint`, `nft_burn`, `nft_transfer`, `nft_metadata_update`, or `contract_metadata_update`, and `data` must be of one of the following relevant types: `NftMintLog[] | NftTransferLog[] | NftBurnLog[] | NftMetadataUpdateLog[] | NftContractMetadataUpdateLog[]`: ```ts interface NftEventLogData { standard: "nep171", version: "1.2.0", event: "nft_mint" | "nft_burn" | "nft_transfer" | "nft_metadata_update" | "contract_metadata_update", data: NftMintLog[] | NftTransferLog[] | NftBurnLog[] | NftMetadataUpdateLog[] | NftContractMetadataUpdateLog[], } ``` ```ts // An event log to capture token minting // Arguments // * `owner_id`: "account.near" // * `token_ids`: ["1", "abc"] // * `memo`: optional message interface NftMintLog { owner_id: string, token_ids: string[], memo?: string } // An event log to capture token burning // Arguments // * `owner_id`: owner of tokens to burn // * `authorized_id`: approved account_id to burn, if applicable // * `token_ids`: ["1","2"] // * `memo`: optional message interface NftBurnLog { owner_id: string, authorized_id?: string, token_ids: string[], memo?: string } // An event log to capture token transfer // Arguments // * `authorized_id`: approved account_id to transfer, if applicable // * `old_owner_id`: "owner.near" // * `new_owner_id`: "receiver.near" // * `token_ids`: ["1", "12345abc"] // * `memo`: optional message interface NftTransferLog { authorized_id?: string, old_owner_id: string, new_owner_id: string, token_ids: string[], memo?: string } // An event log to capture token metadata updating // Arguments // * `token_ids`: ["1", "abc"] // * `memo`: optional message interface NftMetadataUpdateLog { token_ids: string[], memo?: string } // An event log to capture contract metadata updates. Note that the updated contract metadata is not included in the log, as it could easily exceed the 16KB log size limit. Listeners can query `nft_metadata` to get the updated contract metadata. // Arguments // * `memo`: optional message interface NftContractMetadataUpdateLog { memo?: string } ``` #### Examples Single owner batch minting (pretty-formatted for readability purposes): ```js EVENT_JSON:{ "standard": "nep171", "version": "1.2.0", "event": "nft_mint", "data": [ {"owner_id": "foundation.near", "token_ids": ["aurora", "proximitylabs"]} ] } ``` Different owners batch minting: ```js EVENT_JSON:{ "standard": "nep171", "version": "1.2.0", "event": "nft_mint", "data": [ {"owner_id": "foundation.near", "token_ids": ["aurora", "proximitylabs"]}, {"owner_id": "user1.near", "token_ids": ["meme"]} ] } ``` Different events (separate log entries): ```js EVENT_JSON:{ "standard": "nep171", "version": "1.2.0", "event": "nft_burn", "data": [ {"owner_id": "foundation.near", "token_ids": ["aurora", "proximitylabs"]}, ] } ``` ```js EVENT_JSON:{ "standard": "nep171", "version": "1.2.0", "event": "nft_transfer", "data": [ {"old_owner_id": "user1.near", "new_owner_id": "user2.near", "token_ids": ["meme"], "memo": "have fun!"} ] } ``` Authorized id: ```js EVENT_JSON:{ "standard": "nep171", "version": "1.2.0", "event": "nft_burn", "data": [ {"owner_id": "owner.near", "token_ids": ["goodbye", "aurevoir"], "authorized_id": "thirdparty.near"} ] } ``` NFT Metadata Update: ```js EVENT_JSON:{ "standard": "nep171", "version": "1.2.0", "event": "nft_metadata_update", "data": [ {"token_ids": ["1", "2"]} ] } ``` Contract metadata update: ```js EVENT_JSON:{ "standard": "nep171", "version": "1.2.0", "event": "contract_metadata_update", "data": [] } ``` #### Events for Other NFT Methods Note that the example events above cover two different kinds of events: 1. Events that do not have a dedicated trigger function in the NFT Standard (`nft_mint`, `nft_metadata_update`, `nft_burn`, `contract_metadata_update`) 2. An event that has a relevant trigger function [NFT Core Standard](https://nomicon.io/Standards/NonFungibleToken/Core.html#nft-interface) (`nft_transfer`) This event standard also applies beyond the events highlighted here, where future events follow the same convention of as the second type. For instance, if an NFT contract uses the [approval management standard](https://nomicon.io/Standards/NonFungibleToken/ApprovalManagement.html), it may emit an event for `nft_approve` if that's deemed as important by the developer community. Please feel free to open pull requests for extending the events standard detailed here as needs arise. ## Reference Implementation [Minimum Viable Interface](https://github.com/near/near-sdk-rs/blob/master/near-contract-standards/src/non_fungible_token/core/mod.rs) [NFT Implementation](https://github.com/near/near-sdk-rs/blob/master/near-contract-standards/src/non_fungible_token/core/core_impl.rs) ## Changelog ### 1.0.0 - Initial version This NEP had several pre-1.0.0 iterations that led to the following errata updates to this NEP: - **2022-02-03**: updated `Token` struct field names. `id` was changed to `token_id`. This is to be consistent with current implementations of the standard and the rust SDK docs. - **2021-12-20**: updated `nft_resolve_transfer` argument `approved_account_ids` to be type `null|Record` instead of `null|string[]`. This gives contracts a way to restore the original approved accounts and their approval IDs. More information can be found in [this](https://github.com/near/NEPs/issues/301) discussion. - **2021-07-16**: updated `nft_transfer_call` argument `approval_id` to be type `number|null` instead of `string|null`. As stated, approval IDs are not expected to exceed the JSON limit of 2^53. ### 1.1.0 - Add `contract_metadata_update` Event The extension NEP-0423 that added Contract Metadata Update Event Kind to this NEP-0171 was approved by Contract Standards Working Group members (@frol, @abacabadabacaba, @mfornet) on January 13, 2023 ([meeting recording](https://youtu.be/pBLN9UyE6AA)). #### Benefits - This new event type will help indexers to invalidate their cached values reliably and efficiently - This NEP extension only introduces an additional event type, so there is no breaking change to the original NEP #### Concerns | # | Concern | Resolution | Status | | - | - | - | - | | 1 | Old NFT contracts do not emit JSON Events at all; more recent NFT contracts will only emit mint/burn/transfer events, so when it comes to legacy contracts support, we won’t benefit from this new event type and only further fragment the implementations | Legacy contracts usage will die out eventually and new contracts will support new features in a non-breaking way | Resolved | | 2 | There is a need to have a similar event type for individual NFT updates | It is outside of the scope of this NEP extension. Feel free to create a follow-up proposal | Resolved | ### 1.2.0 - Add `nft_metadata_update` Event The extension NEP-0469 that added Token Metadata Update Event Kind to this NEP-0171 was approved by Contract Standards Working Group members (@frol, @abacabadabacaba, @mfornet, @fadeevab, @robert-zaremba) on April 21, 2023 ([meeting recording](https://youtu.be/KOIT8XDQNjM)). #### Benefits - Apps that cache indexed NFTs will benefit from having this new event type as they won't need to have a custom logic to track potentially changing NFTs or refetch all NFTs on every transaction to NFT contract - It plays well with `contract_metadata_update` event that was introduced in version 1.1.0 of this NEP - This NEP extension only introduces an additional event type, so there is no breaking change to the original NEP #### Concerns | # | Concern | Resolution | Status | | - | - | - | - | | 1 | Ecosystem will be split where legacy contracts won't emit these new events, so legacy support will still be needed | In the future, there will be fewer legacy contracts and eventually apps will have support for this type of event | Resolved | | 2 | `nft_update` event name is ambiguous | It was decided to use `nft_metadata_update` name, instead | Resolved | ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). [ERC-721]: https://eips.ethereum.org/EIPS/eip-721 [storage staking]: https://docs.near.org/concepts/storage/storage-staking [gas]: https://docs.near.org/concepts/basics/transactions/gas [Metadata]: https://github.com/near/NEPs/blob/master/neps/nep-0177.md [Approval Management]: https://github.com/near/NEPs/blob/master/neps/nep-0178.md [FT core]: https://github.com/near/NEPs/blob/master/neps/nep-0141.md [Storage Management]: https://github.com/near/NEPs/blob/master/neps/nep-0145.md [NFT Enumeration]: https://github.com/near/NEPs/blob/master/neps/nep-0181.md ================================================ FILE: neps/nep-0177.md ================================================ --- NEP: 177 Title: Non Fungible Token Metadata Author: Chad Ostrowski <@chadoh>, Mike Purvis Status: Final DiscussionsTo: https://github.com/near/NEPs/discussions/177 Type: Standards Track Category: Contract Created: 03-Mar-2022 Requires: 171 --- ## Summary An interface for a non-fungible token's metadata. The goal is to keep the metadata future-proof as well as lightweight. This will be important to dApps needing additional information about an NFT's properties, and broadly compatible with other token standards such that the [NEAR Rainbow Bridge](https://near.org/blog/eth-near-rainbow-bridge/) can move tokens between chains. ## Motivation The primary value of non-fungible tokens comes from their metadata. While the [core standard][NFT Core] provides the minimum interface that can be considered a non-fungible token, most artists, developers, and dApps will want to associate more data with each NFT, and will want a predictable way to interact with any NFT's metadata. ## Rationale and alternatives NEAR's unique [storage staking](https://docs.near.org/concepts/storage/storage-staking) approach makes it feasible to store more data on-chain than other blockchains. This standard leverages this strength for common metadata attributes, and provides a standard way to link to additional offchain data to support rapid community experimentation. This standard also provides a `spec` version. This makes it easy for consumers of NFTs, such as marketplaces, to know if they support all the features of a given token. Prior art: - NEAR's [Fungible Token Metadata Standard][FT Metadata] - Discussion about NEAR's complete NFT standard: #171 ## Specification ## Interface Metadata applies at both the contract level (`NFTContractMetadata`) and the token level (`TokenMetadata`). The relevant metadata for each: ```ts type NFTContractMetadata = { spec: string, // required, essentially a version like "nft-1.0.0" name: string, // required, ex. "Mochi Rising — Digital Edition" or "Metaverse 3" symbol: string, // required, ex. "MOCHI" icon: string|null, // Data URL base_uri: string|null, // Centralized gateway known to have reliable access to decentralized storage assets referenced by `reference` or `media` URLs reference: string|null, // URL to a JSON file with more info reference_hash: string|null, // Base64-encoded sha256 hash of JSON from reference field. Required if `reference` is included. } type TokenMetadata = { title: string|null, // ex. "Arch Nemesis: Mail Carrier" or "Parcel #5055" description: string|null, // free-form description media: string|null, // URL to associated media, preferably to decentralized, content-addressed storage media_hash: string|null, // Base64-encoded sha256 hash of content referenced by the `media` field. Required if `media` is included. copies: number|null, // number of copies of this set of metadata in existence when token was minted. issued_at: number|null, // When token was issued or minted, Unix epoch in milliseconds expires_at: number|null, // When token expires, Unix epoch in milliseconds starts_at: number|null, // When token starts being valid, Unix epoch in milliseconds updated_at: number|null, // When token was last updated, Unix epoch in milliseconds extra: string|null, // anything extra the NFT wants to store on-chain. Can be stringified JSON. reference: string|null, // URL to an off-chain JSON file with more info. reference_hash: string|null // Base64-encoded sha256 hash of JSON from reference field. Required if `reference` is included. } ``` A new function MUST be supported on the NFT contract: ```ts function nft_metadata(): NFTContractMetadata {} ``` A new attribute MUST be added to each `Token` struct: ```diff type Token = { token_id: string, owner_id: string, + metadata: TokenMetadata, } ``` ### An implementing contract MUST include the following fields on-chain - `spec`: a string that MUST be formatted `nft-1.0.0` to indicate that a Non-Fungible Token contract adheres to the current versions of this Metadata spec. This will allow consumers of the Non-Fungible Token to know if they support the features of a given contract. - `name`: the human-readable name of the contract. - `symbol`: the abbreviated symbol of the contract, like MOCHI or MV3 - `base_uri`: Centralized gateway known to have reliable access to decentralized storage assets referenced by `reference` or `media` URLs. Can be used by other frontends for initial retrieval of assets, even if these frontends then replicate the data to their own decentralized nodes, which they are encouraged to do. ### An implementing contract MAY include the following fields on-chain For `NFTContractMetadata`: - `icon`: a small image associated with this contract. Encouraged to be a [data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URIs), to help consumers display it quickly while protecting user data. Recommendation: use [optimized SVG](https://codepen.io/tigt/post/optimizing-svgs-in-data-uris), which can result in high-resolution images with only 100s of bytes of [storage cost](https://docs.near.org/concepts/storage/storage-staking). (Note that these storage costs are incurred to the contract deployer, but that querying these icons is a very cheap & cacheable read operation for all consumers of the contract and the RPC nodes that serve the data.) Recommendation: create icons that will work well with both light-mode and dark-mode websites by either using middle-tone color schemes, or by [embedding `media` queries in the SVG](https://timkadlec.com/2013/04/media-queries-within-svg/). - `reference`: a link to a valid JSON file containing various keys offering supplementary details on the token. Example: `/ipfs/QmdmQXB2mzChmMeKY47C43LxUdg1NDJ5MWcKMKxDu7RgQm`, `https://example.com/token.json`, etc. If the information given in this document conflicts with the on-chain attributes, the values in `reference` shall be considered the source of truth. - `reference_hash`: the base64-encoded sha256 hash of the JSON file contained in the `reference` field. This is to guard against off-chain tampering. For `TokenMetadata`: - `name`: The name of this specific token. - `description`: A longer description of the token. - `media`: URL to associated media. Preferably to decentralized, content-addressed storage. - `media_hash`: the base64-encoded sha256 hash of content referenced by the `media` field. This is to guard against off-chain tampering. - `copies`: The number of tokens with this set of metadata or `media` known to exist at time of minting. - `issued_at`: Unix epoch in milliseconds when token was issued or minted (an unsigned 32-bit integer would suffice until the year 2106) - `expires_at`: Unix epoch in milliseconds when token expires - `starts_at`: Unix epoch in milliseconds when token starts being valid - `updated_at`: Unix epoch in milliseconds when token was last updated - `extra`: anything extra the NFT wants to store on-chain. Can be stringified JSON. - `reference`: URL to an off-chain JSON file with more info. - `reference_hash`: Base64-encoded sha256 hash of JSON from reference field. Required if `reference` is included. ### No incurred cost for core NFT behavior Contracts should be implemented in a way to avoid extra gas fees for serialization & deserialization of metadata for calls to `nft_*` methods other than `nft_metadata` or `nft_token`. See `near-contract-standards` [implementation using `LazyOption`](https://github.com/near/near-sdk-rs/blob/c2771af7fdfe01a4e8414046752ee16fb0d29d39/examples/fungible-token/ft/src/lib.rs#L71) as a reference example. ## Reference Implementation This is the technical portion of the NEP. Explain the design in sufficient detail that: - Its interaction with other features is clear. - Where possible, include a `Minimum Viable Interface` subsection expressing the required behavior and types in a target Near Contract language. (ie. traits and structs for rust, interfaces and classes for javascript, function signatures and structs for c, etc.) - It is reasonably clear how the feature would be implemented. - Corner cases are dissected by example. The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. ## Drawbacks - When this NFT contract is created and initialized, the storage use per-token will be higher than an NFT Core version. Frontends can account for this by adding extra deposit when minting. This could be done by padding with a reasonable amount, or by the frontend using the [RPC call detailed here](https://docs.near.org/docs/develop/front-end/rpc#genesis-config) that gets genesis configuration and actually determine precisely how much deposit is needed. - Convention of `icon` being a data URL rather than a link to an HTTP endpoint that could contain privacy-violating code cannot be done on deploy or update of contract metadata, and must be done on the consumer/app side when displaying token data. - If on-chain icon uses a data URL or is not set but the document given by `reference` contains a privacy-violating `icon` URL, consumers & apps of this data should not naïvely display the `reference` version, but should prefer the safe version. This is technically a violation of the "`reference` setting wins" policy described above. ## Future possibilities - Detailed conventions that may be enforced for versions. - A fleshed out schema for what the `reference` object should contain. ## Errata - **2022-02-03**: updated `Token` struct field names. `id` was changed to `token_id`. This is to be consistent with current implementations of the standard and the rust SDK docs. The first version (`1.0.0`) had confusing language regarding the fields: - `issued_at` - `expires_at` - `starts_at` - `updated_at` It gave those fields the type `string|null` but it was unclear whether it should be a Unix epoch in milliseconds or [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html). Upon having to revisit this, it was determined to be the most efficient to use epoch milliseconds as it would reduce the computation on the smart contract and can be derived trivially from the block timestamp. ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). [NFT Core]: https://github.com/near/NEPs/blob/master/neps/nep-0171.md [FT Metadata]: https://github.com/near/NEPs/blob/master/neps/nep-0148.md ================================================ FILE: neps/nep-0178.md ================================================ --- NEP: 178 Title: Non Fungible Token Approval Management Author: Chad Ostrowski <@chadoh>, Thor <@thor314> Status: Final DiscussionsTo: https://github.com/near/NEPs/discussions/178 Type: Standards Track Category: Contract Created: 03-Mar-2022 Requires: 171 --- ## Summary A system for allowing a set of users or contracts to transfer specific Non-Fungible Tokens on behalf of an owner. Similar to approval management systems in standards like [ERC-721]. ## Motivation People familiar with [ERC-721] may expect to need an approval management system for basic transfers, where a simple transfer from Alice to Bob requires that Alice first _approve_ Bob to spend one of her tokens, after which Bob can call `transfer_from` to actually transfer the token to himself. ## Rationale and alternatives NEAR's [core Non-Fungible Token standard][NFT Core] includes good support for safe atomic transfers without such complexity. It even provides "transfer and call" functionality (`nft_transfer_call`) which allows a specific token to be "attached" to a call to a separate contract. For many Non-Fungible Token workflows, these options may circumvent the need for a full-blown Approval Management system. However, some Non-Fungible Token developers, marketplaces, dApps, or artists may require greater control. This standard provides a uniform interface allowing token owners to approve other NEAR accounts, whether individuals or contracts, to transfer specific tokens on the owner's behalf. Prior art: - Ethereum's [ERC-721] - [NEP-4](https://github.com/near/NEPs/pull/4), NEAR's old NFT standard that does not include approved_account_ids per token ID ## Specification ## Example Scenarios Let's consider some examples. Our cast of characters & apps: - Alice: has account `alice` with no contract deployed to it - Bob: has account `bob` with no contract deployed to it - NFT: a contract with account `nft`, implementing only the [Core NFT standard][NFT Core] with this Approval Management extension - Market: a contract with account `market` which sells tokens from `nft` as well as other NFT contracts - Bazaar: similar to Market, but implemented differently (spoiler alert: has no `nft_on_approve` function!), has account `bazaar` Alice and Bob are already [registered][Storage Management] with NFT, Market, and Bazaar, and Alice owns a token on the NFT contract with ID=`"1"`. Let's examine the technical calls through the following scenarios: 1. [Simple approval](#1-simple-approval): Alice approves Bob to transfer her token. 2. [Approval with cross-contract call (XCC)](#2-approval-with-cross-contract-call): Alice approves Market to transfer one of her tokens and passes `msg` so that NFT will call `nft_on_approve` on Market's contract. 3. [Approval with XCC, edge case](#3-approval-with-cross-contract-call-edge-case): Alice approves Bazaar and passes `msg` again, but what's this? Bazaar doesn't implement `nft_on_approve`, so Alice sees an error in the transaction result. Not to worry, though, she checks `nft_is_approved` and sees that she did successfully approve Bazaar, despite the error. 4. [Approval IDs](#4-approval-ids): Bob buys Alice's token via Market. 5. [Approval IDs, edge case](#5-approval-ids-edge-case): Bob transfers same token back to Alice, Alice re-approves Market & Bazaar. Bazaar has an outdated cache. Bob tries to buy from Bazaar at the old price. 6. [Revoke one](#6-revoke-one): Alice revokes Market's approval for this token. 7. [Revoke all](#7-revoke-all): Alice revokes all approval for this token. ### 1. Simple Approval Alice approves Bob to transfer her token. ##### High-level explanation 1. Alice approves Bob 2. Alice queries the token to verify 3. Alice verifies a different way ##### Technical calls 1. Alice calls `nft::nft_approve({ "token_id": "1", "account_id": "bob" })`. She attaches 1 yoctoⓃ, (.000000000000000000000001Ⓝ). Using [NEAR CLI](https://docs.near.org/tools/near-cli) to make this call, the command would be: ```shell near call nft nft_approve \ '{ "token_id": "1", "account_id": "bob" }' \ --accountId alice --depositYocto 1 ``` The response: ```shell '' ``` 2. Alice calls view method `nft_token`: ```shell near view nft nft_token '{ "token_id": "1" }' ``` The response: ```json { "token_id": "1", "owner_id": "alice.near", "approved_account_ids": { "bob": 1 } } ``` 3. Alice calls view method `nft_is_approved`: ```shell near view nft nft_is_approved '{ "token_id": "1", "approved_account_id": "bob" }' ``` The response: ```shell true ``` ### 2. Approval with cross-contract call Alice approves Market to transfer one of her tokens and passes `msg` so that NFT will call `nft_on_approve` on Market's contract. She probably does this via Market's frontend app which would know how to construct `msg` in a useful way. ##### High-level explanation 1. Alice calls `nft_approve` to approve `market` to transfer her token, and passes a `msg` 2. Since `msg` is included, `nft` will schedule a cross-contract call to `market` 3. Market can do whatever it wants with this info, such as listing the token for sale at a given price. The result of this operation is returned as the promise outcome to the original `nft_approve` call. ##### Technical calls 1. Using near-cli: ```shell near call nft nft_approve '{ "token_id": "1", "account_id": "market", "msg": "{\"action\": \"list\", \"price\": \"100\", \"token\": \"nDAI\" }" }' --accountId alice --depositYocto 1 ``` At this point, near-cli will hang until the cross-contract call chain fully resolves, which would also be true if Alice used a Market frontend using [near-api](https://docs.near.org/tools/near-api). Alice's part is done, though. The rest happens behind the scenes. 2. `nft` schedules a call to `nft_on_approve` on `market`. Using near-cli notation for easy cross-reference with the above, this would look like: ```shell near call market nft_on_approve '{ "token_id": "1", "owner_id": "alice", "approval_id": 2, "msg": "{\"action\": \"list\", \"price\": \"100\", \"token\": \"nDAI\" }" }' --accountId nft ``` 3. `market` now knows that it can sell Alice's token for 100 [nDAI](https://explorer.mainnet.near.org/accounts/6b175474e89094c44da98b954eedeac495271d0f.factory.bridge.near), and that when it transfers it to a buyer using `nft_transfer`, it can pass along the given `approval_id` to ensure that Alice hasn't changed her mind. It can schedule any further cross-contract calls it wants, and if it returns these promises correctly, Alice's initial near-cli call will resolve with the outcome from the final step in the chain. If Alice actually made this call from a Market frontend, the frontend can use this return value for something useful. ### 3. Approval with cross-contract call, edge case Alice approves Bazaar and passes `msg` again. Maybe she actually does this via near-cli, rather than using Bazaar's frontend, because what's this? Bazaar doesn't implement `nft_on_approve`, so Alice sees an error in the transaction result. Not to worry, though, she checks `nft_is_approved` and sees that she did successfully approve Bazaar, despite the error. She will have to find a new way to list her token for sale in Bazaar, rather than using the same `msg` shortcut that worked for Market. ##### High-level explanation 1. Alice calls `nft_approve` to approve `bazaar` to transfer her token, and passes a `msg`. 2. Since `msg` is included, `nft` will schedule a cross-contract call to `bazaar`. 3. Bazaar doesn't implement `nft_on_approve`, so this call results in an error. The approval still worked, but Alice sees an error in her near-cli output. 4. Alice checks if `bazaar` is approved, and sees that it is, despite the error. ##### Technical calls 1. Using near-cli: ```shell near call nft nft_approve '{ "token_id": "1", "account_id": "bazaar", "msg": "{\"action\": \"list\", \"price\": \"100\", \"token\": \"nDAI\" }" }' --accountId alice --depositYocto 1 ``` 2. `nft` schedules a call to `nft_on_approve` on `market`. Using near-cli notation for easy cross-reference with the above, this would look like: ```shell near call bazaar nft_on_approve '{ "token_id": "1", "owner_id": "alice", "approval_id": 3, "msg": "{\"action\": \"list\", \"price\": \"100\", \"token\": \"nDAI\" }" }' --accountId nft ``` 3. 💥 `bazaar` doesn't implement this method, so the call results in an error. Alice sees this error in the output from near-cli. 4. Alice checks if the approval itself worked, despite the error on the cross-contract call: ```shell near view nft nft_is_approved \ '{ "token_id": "1", "approved_account_id": "bazaar" }' ``` The response: ```shell true ``` ### 4. Approval IDs Bob buys Alice's token via Market. Bob probably does this via Market's frontend, which will probably initiate the transfer via a call to `ft_transfer_call` on the nDAI contract to transfer 100 nDAI to `market`. Like the NFT standard's "transfer and call" function, [Fungible Token][FT Core]'s `ft_transfer_call` takes a `msg` which `market` can use to pass along information it will need to pay Alice and actually transfer the NFT. The actual transfer of the NFT is the only part we care about here. ##### High-level explanation 1. Bob signs some transaction which results in the `market` contract calling `nft_transfer` on the `nft` contract, as described above. To be trustworthy and pass security audits, `market` needs to pass along `approval_id` so that it knows it has up-to-date information. ##### Technical calls Using near-cli notation for consistency: ```shell near call nft nft_transfer '{ "receiver_id": "bob", "token_id": "1", "approval_id": 2, }' --accountId market --depositYocto 1 ``` ### 5. Approval IDs, edge case Bob transfers same token back to Alice, Alice re-approves Market & Bazaar, listing her token at a higher price than before. Bazaar is somehow unaware of these changes, and still stores `approval_id: 3` internally along with Alice's old price. Bob tries to buy from Bazaar at the old price. Like the previous example, this probably starts with a call to a different contract, which eventually results in a call to `nft_transfer` on `bazaar`. Let's consider a possible scenario from that point. ##### High-level explanation Bob signs some transaction which results in the `bazaar` contract calling `nft_transfer` on the `nft` contract, as described above. To be trustworthy and pass security audits, `bazaar` needs to pass along `approval_id` so that it knows it has up-to-date information. It does not have up-to-date information, so the call fails. If the initial `nft_transfer` call is part of a call chain originating from a call to `ft_transfer_call` on a fungible token, Bob's payment will be refunded and no assets will change hands. ##### Technical calls Using near-cli notation for consistency: ```shell near call nft nft_transfer '{ "receiver_id": "bob", "token_id": "1", "approval_id": 3, }' --accountId bazaar --depositYocto 1 ``` ### 6. Revoke one Alice revokes Market's approval for this token. ##### Technical calls Using near-cli: ```shell near call nft nft_revoke '{ "account_id": "market", "token_id": "1", }' --accountId alice --depositYocto 1 ``` Note that `market` will not get a cross-contract call in this case. The implementers of the Market app should implement [cron](https://en.wikipedia.org/wiki/Cron)-type functionality to intermittently check that Market still has the access they expect. ### 7. Revoke all Alice revokes all approval for this token. ##### Technical calls Using near-cli: ```shell near call nft nft_revoke_all '{ "token_id": "1", }' --accountId alice --depositYocto 1 ``` Again, note that no previous approvers will get cross-contract calls in this case. ## Reference-level explanation The `Token` structure returned by `nft_token` must include an `approved_account_ids` field, which is a map of account IDs to approval IDs. Using TypeScript's [Record type](https://www.typescriptlang.org/docs/handbook/utility-types.html#recordkeystype) notation: ```diff type Token = { token_id: string, owner_id: string, + approved_account_ids: Record, } ``` Example token data: ```json { "token_id": "1", "owner_id": "alice.near", "approved_account_ids": { "bob.near": 1, "carol.near": 2 } } ``` ### What is an "approval ID"? This is a unique number given to each approval that allows well-intentioned marketplaces or other 3rd-party NFT resellers to avoid a race condition. The race condition occurs when: 1. A token is listed in two marketplaces, which are both saved to the token as approved accounts. 2. One marketplace sells the token, which clears the approved accounts. 3. The new owner sells back to the original owner. 4. The original owner approves the token for the second marketplace again to list at a new price. But for some reason the second marketplace still lists the token at the previous price and is unaware of the transfers happening. 5. The second marketplace, operating from old information, attempts to again sell the token at the old price. Note that while this describes an honest mistake, the possibility of such a bug can also be taken advantage of by malicious parties via [front-running](https://users.encs.concordia.ca/~clark/papers/2019_wtsc_front.pdf). To avoid this possibility, the NFT contract generates a unique approval ID each time it approves an account. Then when calling `nft_transfer` or `nft_transfer_call`, the approved account passes `approval_id` with this value to make sure the underlying state of the token hasn't changed from what the approved account expects. Keeping with the example above, say the initial approval of the second marketplace generated the following `approved_account_ids` data: ```json { "token_id": "1", "owner_id": "alice.near", "approved_account_ids": { "marketplace_1.near": 1, "marketplace_2.near": 2 } } ``` But after the transfers and re-approval described above, the token might have `approved_account_ids` as: ```json { "token_id": "1", "owner_id": "alice.near", "approved_account_ids": { "marketplace_2.near": 3 } } ``` The marketplace then tries to call `nft_transfer`, passing outdated information: ```bash # oops! near call nft-contract.near nft_transfer '{ "approval_id": 2 }' ``` ### Interface The NFT contract must implement the following methods: ```ts /******************/ /* CHANGE METHODS */ /******************/ // Add an approved account for a specific token. // // Requirements // * Caller of the method must attach a deposit of at least 1 yoctoⓃ for // security purposes // * Contract MAY require caller to attach larger deposit, to cover cost of // storing approver data // * Contract MUST panic if called by someone other than token owner // * Contract MUST panic if addition would cause `nft_revoke_all` to exceed // single-block gas limit. See below for more info. // * Contract MUST increment approval ID even if re-approving an account // * If successfully approved or if had already been approved, and if `msg` is // present, contract MUST call `nft_on_approve` on `account_id`. See // `nft_on_approve` description below for details. // // Arguments: // * `token_id`: the token for which to add an approval // * `account_id`: the account to add to `approved_account_ids` // * `msg`: optional string to be passed to `nft_on_approve` // // Returns void, if no `msg` given. Otherwise, returns promise call to // `nft_on_approve`, which can resolve with whatever it wants. function nft_approve( token_id: TokenId, account_id: string, msg: string | null ): void | Promise {} // Revoke an approved account for a specific token. // // Requirements // * Caller of the method must attach a deposit of 1 yoctoⓃ for security // purposes // * If contract requires >1yN deposit on `nft_approve`, contract // MUST refund associated storage deposit when owner revokes approval // * Contract MUST panic if called by someone other than token owner // // Arguments: // * `token_id`: the token for which to revoke an approval // * `account_id`: the account to remove from `approved_account_ids` function nft_revoke(token_id: string, account_id: string) {} // Revoke all approved accounts for a specific token. // // Requirements // * Caller of the method must attach a deposit of 1 yoctoⓃ for security // purposes // * If contract requires >1yN deposit on `nft_approve`, contract // MUST refund all associated storage deposit when owner revokes approved_account_ids // * Contract MUST panic if called by someone other than token owner // // Arguments: // * `token_id`: the token with approved_account_ids to revoke function nft_revoke_all(token_id: string) {} /****************/ /* VIEW METHODS */ /****************/ // Check if a token is approved for transfer by a given account, optionally // checking an approval_id // // Arguments: // * `token_id`: the token for which to revoke an approval // * `approved_account_id`: the account to check the existence of in `approved_account_ids` // * `approval_id`: an optional approval ID to check against current approval ID for given account // // Returns: // if `approval_id` given, `true` if `approved_account_id` is approved with given `approval_id` // otherwise, `true` if `approved_account_id` is in list of approved accounts function nft_is_approved( token_id: string, approved_account_id: string, approval_id: number | null ): boolean {} ``` ### Why must `nft_approve` panic if `nft_revoke_all` would fail later? In the description of `nft_approve` above, it states: > Contract MUST panic if addition would cause `nft_revoke_all` to exceed > single-block gas limit. What does this mean? First, it's useful to understand what we mean by "single-block gas limit". This refers to the [hard cap on gas per block at the protocol layer](https://docs.near.org/docs/concepts/gas#thinking-in-gas). This number will increase over time. Removing data from a contract uses gas, so if an NFT had a large enough number of approved_account_ids, `nft_revoke_all` would fail, because calling it would exceed the maximum gas. Contracts must prevent this by capping the number of approved_account_ids for a given token. However, it is up to contract authors to determine a sensible cap for their contract (and the single block gas limit at the time they deploy). Since contract implementations can vary, some implementations will be able to support a larger number of approved_account_ids than others, even with the same maximum gas per block. Contract authors may choose to set a cap of something small and safe like 10 approved_account_ids, or they could dynamically calculate whether a new approval would break future calls to `nft_revoke_all`. But every contract MUST ensure that they never break the functionality of `nft_revoke_all`. ### Approved Account Contract Interface If a contract that gets approved to transfer NFTs wants to, it can implement `nft_on_approve` to update its own state when granted approval for a token: ```ts // Respond to notification that contract has been granted approval for a token. // // Notes // * Contract knows the token contract ID from `predecessor_account_id` // // Arguments: // * `token_id`: the token to which this contract has been granted approval // * `owner_id`: the owner of the token // * `approval_id`: the approval ID stored by NFT contract for this approval. // Expected to be a number within the 2^53 limit representable by JSON. // * `msg`: specifies information needed by the approved contract in order to // handle the approval. Can indicate both a function to call and the // parameters to pass to that function. function nft_on_approve( token_id: TokenId, owner_id: string, approval_id: number, msg: string ) {} ``` Note that the NFT contract will fire-and-forget this call, ignoring any return values or errors generated. This means that even if the approved account does not have a contract or does not implement `nft_on_approve`, the approval will still work correctly from the point of view of the NFT contract. Further note that there is no parallel `nft_on_revoke` when revoking either a single approval or when revoking all. This is partially because scheduling many `nft_on_revoke` calls when revoking all approved_account_ids could incur prohibitive [gas fees](https://docs.near.org/docs/concepts/gas). Apps and contracts which cache NFT approved_account_ids can therefore not rely on having up-to-date information, and should periodically refresh their caches. Since this will be the necessary reality for dealing with `nft_revoke_all`, there is no reason to complicate `nft_revoke` with an `nft_on_revoke` call. ### No incurred cost for core NFT behavior NFT contracts should be implemented in a way to avoid extra gas fees for serialization & deserialization of `approved_account_ids` for calls to `nft_*` methods other than `nft_token`. See `near-contract-standards` [implementation of `ft_metadata` using `LazyOption`](https://github.com/near/near-sdk-rs/blob/c2771af7fdfe01a4e8414046752ee16fb0d29d39/examples/fungible-token/ft/src/lib.rs#L71) as a reference example. ## Reference Implementation [NFT Approval Receiver Interface](https://github.com/near/near-sdk-rs/blob/master/near-contract-standards/src/non_fungible_token/approval/approval_receiver.rs) [NFT Approval Implementation](https://github.com/near/near-sdk-rs/blob/master/near-contract-standards/src/non_fungible_token/approval/approval_impl.rs) ## Errata - **2022-02-03**: updated `Token` struct field names. `id` was changed to `token_id` and `approvals` was changed to `approved_account_ids`. This is to be consistent with current implementations of the standard and the rust SDK docs. ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). [ERC-721]: https://eips.ethereum.org/EIPS/eip-721 [NFT Core]: https://github.com/near/NEPs/blob/master/neps/nep-0171.md [Storage Management]: https://github.com/near/NEPs/blob/master/neps/nep-0145.md [FT Core]: https://github.com/near/NEPs/blob/master/neps/nep-0141.md ================================================ FILE: neps/nep-0181.md ================================================ --- NEP: 181 Title: Non Fungible Token Enumeration Author: Chad Ostrowski <@chadoh>, Thor <@thor314> Status: Final DiscussionsTo: https://github.com/near/NEPs/discussions/181 Type: Standards Track Category: Contract Created: 03-Mar-2022 Requires: 171 --- ## Summary Standard interfaces for counting & fetching tokens, for an entire NFT contract or for a given owner. ## Motivation Apps such as marketplaces and wallets need a way to show all tokens owned by a given account and to show statistics about all tokens for a given contract. This extension provides a standard way to do so. ## Rationale and alternatives While some NFT contracts may forego this extension to save [storage] costs, this requires apps to have custom off-chain indexing layers. This makes it harder for apps to integrate with such NFTs. Apps which integrate only with NFTs that use the Enumeration extension do not even need a server-side component at all, since they can retrieve all information they need directly from the blockchain. Prior art: - [ERC-721]'s enumeration extension ## Specification The contract must implement the following view methods: ```ts // Returns the total supply of non-fungible tokens as a string representing an // unsigned 128-bit integer to avoid JSON number limit of 2^53; and "0" if there are no tokens. function nft_total_supply(): string {} // Get a list of all tokens // // Arguments: // * `from_index`: a string representing an unsigned 128-bit integer, // representing the starting index of tokens to return // * `limit`: the maximum number of tokens to return // // Returns an array of Token objects, as described in Core standard, and an empty array if there are no tokens function nft_tokens( from_index: string|null, // default: "0" limit: number|null, // default: unlimited (could fail due to gas limit) ): Token[] {} // Get number of tokens owned by a given account // // Arguments: // * `account_id`: a valid NEAR account // // Returns the number of non-fungible tokens owned by given `account_id` as // a string representing the value as an unsigned 128-bit integer to avoid JSON // number limit of 2^53; and "0" if there are no tokens. function nft_supply_for_owner( account_id: string, ): string {} // Get list of all tokens owned by a given account // // Arguments: // * `account_id`: a valid NEAR account // * `from_index`: a string representing an unsigned 128-bit integer, // representing the starting index of tokens to return // * `limit`: the maximum number of tokens to return // // Returns a paginated list of all tokens owned by this account, and an empty array if there are no tokens function nft_tokens_for_owner( account_id: string, from_index: string|null, // default: 0 limit: number|null, // default: unlimited (could fail due to gas limit) ): Token[] {} ``` ## Notes At the time of this writing, the specialized collections in the `near-sdk` Rust crate are iterable, but not all of them have implemented an `iter_from` solution. There may be efficiency gains for large collections and contract developers are encouraged to test their data structures with a large amount of entries. ## Reference Implementation [Minimum Viable Interface](https://github.com/near/near-sdk-rs/blob/master/near-contract-standards/src/non_fungible_token/enumeration/mod.rs) [Implementation](https://github.com/near/near-sdk-rs/blob/master/near-contract-standards/src/non_fungible_token/enumeration/enumeration_impl.rs) ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). [ERC-721]: https://eips.ethereum.org/EIPS/eip-721 [storage]: https://docs.near.org/concepts/storage/storage-staking ================================================ FILE: neps/nep-0199.md ================================================ --- NEP: 199 Title: Non Fungible Token Royalties and Payouts Author: Thor <@thor314>, Matt Lockyer <@mattlockyer> Status: Final DiscussionsTo: https://github.com/near/NEPs/discussions/199 Type: Standards Track Category: Contract Created: 03-Mar-2022 Requires: 171, 178 --- ## Summary An interface allowing non-fungible token contracts to request that financial contracts pay-out multiple receivers, enabling flexible royalty implementations. ## Motivation Currently, NFTs on NEAR support the field `owner_id`, but lack flexibility for ownership and payout mechanics with more complexity, including but not limited to royalties. Financial contracts, such as marketplaces, auction houses, and NFT loan contracts would benefit from a standard interface on NFT producer contracts for querying whom to pay out, and how much to pay. Therefore, the core goal of this standard is to define a set of methods for financial contracts to call, without specifying how NFT contracts define the divide of payout mechanics, and a standard `Payout` response structure. ## Specification This Payout extension standard adds two methods to NFT contracts: - a view method: `nft_payout`, accepting a `token_id` and some `balance`, returning the `Payout` mapping for the given token. - a call method: `nft_transfer_payout`, accepting all the arguments of`nft_transfer`, plus a field for some `Balance` that calculates the `Payout`, calls `nft_transfer`, and returns the `Payout` mapping. Financial contracts MUST validate several invariants on the returned `Payout`: 1. The returned `Payout` MUST be no longer than the given maximum length (`max_len_payout` parameter) if provided. Payouts of excessive length can become prohibitively gas-expensive. Financial contracts can specify the maximum length of payout the contract is willing to respect with the `max_len_payout` field on `nft_transfer_payout`. 2. The balances MUST add up to less than or equal to the `balance` argument in `nft_transfer_payout`. If the balance adds up to less than the `balance` argument, the financial contract MAY claim the remainder for itself. 3. The sum of the balances MUST NOT overflow. This is technically identical to 2, but financial contracts should be expected to handle this possibility. Financial contracts MAY specify their own maximum length payout to respect. At minimum, financial contracts MUST NOT set their maximum length below 10. If the Payout contains any addresses that do not exist, the financial contract MAY keep those wasted payout funds. Financial contracts MAY take a cut of the NFT sale price as commission, subtracting their cut from the total token sale price, and calling `nft_transfer_payout` with the remainder. ## Example Flow ```text ┌─────────────────────────────────────────────────┐ │Token Owner approves marketplace for token_id "0"│ ├─────────────────────────────────────────────────┘ │ nft_approve("0",market.near,) ▼ ┌───────────────────────────────────────────────┐ │Marketplace sells token to user.near for 10N │ ├───────────────────────────────────────────────┘ │ nft_transfer_payout(user.near,"0",0,"10000000",5) ▼ ┌───────────────────────────────────────────────┐ │NFT contract returns Payout data │ ├───────────────────────────────────────────────┘ │ Payout(, } pub trait Payouts { /// Given a `token_id` and NEAR-denominated balance, return the `Payout`. /// struct for the given token. Panic if the length of the payout exceeds /// `max_len_payout.` fn nft_payout(&self, token_id: String, balance: U128, max_len_payout: Option) -> Payout; /// Given a `token_id` and NEAR-denominated balance, transfer the token /// and return the `Payout` struct for the given token. Panic if the /// length of the payout exceeds `max_len_payout.` #[payable] fn nft_transfer_payout( &mut self, receiver_id: AccountId, token_id: String, approval_id: Option, memo: Option, balance: U128, max_len_payout: Option, ) -> Payout { assert_one_yocto(); let payout = self.nft_payout(token_id, balance); self.nft_transfer(receiver_id, token_id, approval_id, memo); payout } } ``` ## Fallback on error In the case where either the `max_len_payout` causes a panic, or a malformed `Payout` is returned, the caller contract should transfer all funds to the original token owner selling the token. ## Potential pitfalls The payout must include all accounts that should receive funds. Thus it is a mistake to assume that the original token owner will receive funds if they are not included in the payout. NFT and financial contracts vary in implementation. This means that some extra CPU cycles may occur in one NFT contract and not another. Furthermore, a financial contract may accept fungible tokens, native NEAR, or another entity as payment. Transferring native NEAR tokens is less expensive in gas than sending fungible tokens. For these reasons, the maximum length of payouts may vary according to the customization of the smart contracts. ## Drawbacks There is an introduction of trust that the contract calling `nft_transfer_payout` will indeed pay out to all intended parties. However, since the calling contract will typically be something like a marketplace used by end users, malicious actors might be found out more easily and might have less incentive. There is an assumption that NFT contracts will understand the limits of gas and not allow for a number of payouts that cannot be achieved. ## Future possibilities In the future, the NFT contract itself may be able to place an NFT transfer is a state that is "pending transfer" until all payouts have been awarded. This would keep all the information inside the NFT and remove trust. ## Errata - Version `2.1.0` adds a memo parameter to `nft_transfer_payout`, which previously forced implementers of `2.0.0` to pass `None` to the inner `nft_transfer`. Also refactors `max_len_payout` to be an option type. - Version `2.0.0` contains the intended `approval_id` of `u64` instead of the stringified `U64` version. This was an oversight, but since the standard was live for a few months before noticing, the team thought it best to bump the major version. ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0245/ApprovalManagement.md ================================================ # Multi Token Standard Approval Management :::caution This is part of the proposed spec [NEP-245](https://github.com/near/NEPs/blob/master/neps/nep-0245.md) and is subject to change. ::: Version `1.0.0` ## Summary A system for allowing a set of users or contracts to transfer specific tokens on behalf of an owner. Similar to approval management systems in standards like [ERC-721] and [ERC-1155]. [ERC-721]: https://eips.ethereum.org/EIPS/eip-721 [ERC-1155]: https://eips.ethereum.org/EIPS/eip-1155 ## Motivation People familiar with [ERC-721] may expect to need an approval management system for basic transfers, where a simple transfer from Alice to Bob requires that Alice first _approve_ Bob to spend one of her tokens, after which Bob can call `transfer_from` to actually transfer the token to himself. NEAR's [core Multi Token standard](https://github.com/near/NEPs/blob/master/neps/nep-0245.md) includes good support for safe atomic transfers without such complexity. It even provides "transfer and call" functionality (`mt_transfer_call`) which allows specific tokens to be "attached" to a call to a separate contract. For many token workflows, these options may circumvent the need for a full-blown Approval Management system. However, some Multi Token developers, marketplaces, dApps, or artists may require greater control. This standard provides a uniform interface allowing token owners to approve other NEAR accounts, whether individuals or contracts, to transfer specific tokens on the owner's behalf. Prior art: - Ethereum's [ERC-721] - Ethereum's [ERC-1155] ## Example Scenarios Let's consider some examples. Our cast of characters & apps: - Alice: has account `alice` with no contract deployed to it - Bob: has account `bob` with no contract deployed to it - MT: a contract with account `mt`, implementing only the [Multi Token Standard](https://github.com/near/NEPs/blob/master/neps/nep-0245.md) with this Approval Management extension - Market: a contract with account `market` which sells tokens from `mt` as well as other token contracts - Bazaar: similar to Market, but implemented differently (spoiler alert: has no `mt_on_approve` function!), has account `bazaar` Alice and Bob are already [registered](https://github.com/near/NEPs/blob/master/neps/nep-0145.md) with MT, Market, and Bazaar, and Alice owns a token on the MT contract with ID=`"1"` and a fungible style token with ID =`"2"` and AMOUNT =`"100"`. Let's examine the technical calls through the following scenarios: 1. [Simple approval](#1-simple-approval): Alice approves Bob to transfer her token. 2. [Approval with cross-contract call (XCC)](#2-approval-with-cross-contract-call): Alice approves Market to transfer one of her tokens and passes `msg` so that MT will call `mt_on_approve` on Market's contract. 3. [Approval with XCC, edge case](#3-approval-with-cross-contract-call-edge-case): Alice approves Bazaar and passes `msg` again, but what's this? Bazaar doesn't implement `mt_on_approve`, so Alice sees an error in the transaction result. Not to worry, though, she checks `mt_is_approved` and sees that she did successfully approve Bazaar, despite the error. 4. [Approval IDs](#4-approval-ids): Bob buys Alice's token via Market. 5. [Approval IDs, edge case](#5-approval-ids-edge-case): Bob transfers same token back to Alice, Alice re-approves Market & Bazaar. Bazaar has an outdated cache. Bob tries to buy from Bazaar at the old price. 6. [Revoke one](#6-revoke-one): Alice revokes Market's approval for this token. 7. [Revoke all](#7-revoke-all): Alice revokes all approval for this token. ### 1. Simple Approval Alice approves Bob to transfer her tokens. #### High-level explanation 1. Alice approves Bob 2. Alice queries the token to verify #### Technical calls 1. Alice calls `mt::mt_approve({ "token_ids": ["1","2"], amounts:["1","100"], "account_id": "bob" })`. She attaches 1 yoctoⓃ, (.000000000000000000000001Ⓝ). Using [NEAR CLI](https://docs.near.org/tools/near-cli) to make this call, the command would be: ```bash near call mt mt_approve \ '\{ "token_ids": ["1","2"], amounts: ["1","100"], "account_id": "bob" }' \ --accountId alice --amount .000000000000000000000001 ``` The response: ```bash '' ``` 2. Alice calls view method `mt_is_approved`: ```bash near view mt mt_is_approved \ '\{ "token_ids": ["1", "2"], amounts:["1","100"], "approved_account_id": "bob" }' ``` The response: ```bash true ``` ### 2. Approval with cross-contract call Alice approves Market to transfer some of her tokens and passes `msg` so that MT will call `mt_on_approve` on Market's contract. She probably does this via Market's frontend app which would know how to construct `msg` in a useful way. #### High-level explanation 1. Alice calls `mt_approve` to approve `market` to transfer her token, and passes a `msg` 2. Since `msg` is included, `mt` will schedule a cross-contract call to `market` 3. Market can do whatever it wants with this info, such as listing the token for sale at a given price. The result of this operation is returned as the promise outcome to the original `mt_approve` call. #### Technical calls 1. Using near-cli: ```bash near call mt mt_approve '\{ "token_ids": ["1","2"], "amounts": ["1", "100"], "account_id": "market", "msg": "\{\"action\": \"list\", \"price\": [\"100\",\"50\"],\"token\": \"nDAI\" }" }' --accountId alice --amount .000000000000000000000001 ``` At this point, near-cli will hang until the cross-contract call chain fully resolves, which would also be true if Alice used a Market frontend using [near-api](https://docs.near.org/tools/near-api). Alice's part is done, though. The rest happens behind the scenes. 2. `mt` schedules a call to `mt_on_approve` on `market`. Using near-cli notation for easy cross-reference with the above, this would look like: ```bash near call market mt_on_approve '\{ "token_ids": ["1","2"], "amounts": ["1","100"], "owner_id": "alice", "approval_ids": ["4","5"], "msg": "\{\"action\": \"list\", \"price\": [\"100\",\"50\"], \"token\": \"nDAI\" }" }' --accountId mt ``` 3. `market` now knows that it can sell Alice's tokens for 100 [nDAI](https://explorer.mainnet.near.org/accounts/6b175474e89094c44da98b954eedeac495271d0f.factory.bridge.near) and 50 [nDAI](https://explorer.mainnet.near.org/accounts/6b175474e89094c44da98b954eedeac495271d0f.factory.bridge.near), and that when it transfers it to a buyer using `mt_batch_transfer`, it can pass along the given `approval_ids` to ensure that Alice hasn't changed her mind. It can schedule any further cross-contract calls it wants, and if it returns these promises correctly, Alice's initial near-cli call will resolve with the outcome from the final step in the chain. If Alice actually made this call from a Market frontend, the frontend can use this return value for something useful. ### 3. Approval with cross-contract call, edge case Alice approves Bazaar and passes `msg` again. Maybe she actually does this via near-cli, rather than using Bazaar's frontend, because what's this? Bazaar doesn't implement `mt_on_approve`, so Alice sees an error in the transaction result. Not to worry, though, she checks `mt_is_approved` and sees that she did successfully approve Bazaar, despite the error. She will have to find a new way to list her token for sale in Bazaar, rather than using the same `msg` shortcut that worked for Market. #### High-level explanation 1. Alice calls `mt_approve` to approve `bazaar` to transfer her token, and passes a `msg`. 2. Since `msg` is included, `mt` will schedule a cross-contract call to `bazaar`. 3. Bazaar doesn't implement `mt_on_approve`, so this call results in an error. The approval still worked, but Alice sees an error in her near-cli output. 4. Alice checks if `bazaar` is approved, and sees that it is, despite the error. #### Technical calls 1. Using near-cli: ```bash near call mt mt_approve '\{ "token_ids": ["1"], "amounts: ["1000"], "account_id": "bazaar", "msg": "\{\"action\": \"list\", \"price\": \"100\", \"token\": \"nDAI\" }" }' --accountId alice --amount .000000000000000000000001 ``` 2. `mt` schedules a call to `mt_on_approve` on `market`. Using near-cli notation for easy cross-reference with the above, this would look like: ```bash near call bazaar mt_on_approve '\{ "token_ids": ["1"], "amounts": ["1000"], "owner_id": "alice", "approval_ids": [3], "msg": "\{\"action\": \"list\", \"price\": \"100\", \"token\": \"nDAI\" }" }' --accountId mt ``` 3. 💥 `bazaar` doesn't implement this method, so the call results in an error. Alice sees this error in the output from near-cli. 4. Alice checks if the approval itself worked, despite the error on the cross-contract call: ```bash near view mt mt_is_approved \ '{ "token_ids": ["1","2"], "amounts":["1","100"], "approved_account_id": "bazaar" }' ``` The response: ```bash true ``` ### 4. Approval IDs Bob buys Alice's token via Market. Bob probably does this via Market's frontend, which will probably initiate the transfer via a call to `ft_transfer_call` on the nDAI contract to transfer 100 nDAI to `market`. Like the MT standard's "transfer and call" function, [Fungible Token](https://github.com/near/NEPs/blob/master/neps/nep-0141.md)'s `ft_transfer_call` takes a `msg` which `market` can use to pass along information it will need to pay Alice and actually transfer the MT. The actual transfer of the MT is the only part we care about here. #### High-level explanation 1. Bob signs some transaction which results in the `market` contract calling `mt_transfer` on the `mt` contract, as described above. To be trustworthy and pass security audits, `market` needs to pass along `approval_id` so that it knows it has up-to-date information. #### Technical calls Using near-cli notation for consistency: ```bash near call mt mt_transfer '{ "receiver_id": "bob", "token_id": "1", "amount": "1", "approval_id": 2, }' --accountId market --amount .000000000000000000000001 ``` ### 5. Approval IDs, edge case Bob transfers same token back to Alice, Alice re-approves Market & Bazaar, listing her token at a higher price than before. Bazaar is somehow unaware of these changes, and still stores `approval_id: 3` internally along with Alice's old price. Bob tries to buy from Bazaar at the old price. Like the previous example, this probably starts with a call to a different contract, which eventually results in a call to `mt_transfer` on `bazaar`. Let's consider a possible scenario from that point. #### High-level explanation Bob signs some transaction which results in the `bazaar` contract calling `mt_transfer` on the `mt` contract, as described above. To be trustworthy and pass security audits, `bazaar` needs to pass along `approval_id` so that it knows it has up-to-date information. It does not have up-to-date information, so the call fails. If the initial `mt_transfer` call is part of a call chain originating from a call to `ft_transfer_call` on a fungible token, Bob's payment will be refunded and no assets will change hands. #### Technical calls Using near-cli notation for consistency: ```bash near call mt mt_transfer '{ "receiver_id": "bob", "token_id": "1", "amount": "1", "approval_id": 3, }' --accountId bazaar --amount .000000000000000000000001 ``` ### 6. Revoke one Alice revokes Market's approval for this token. #### Technical calls Using near-cli: ```bash near call mt mt_revoke '\{ "account_id": "market", "token_ids": ["1"], }' --accountId alice --amount .000000000000000000000001 ``` Note that `market` will not get a cross-contract call in this case. The implementors of the Market app should implement [cron](https://en.wikipedia.org/wiki/Cron)-type functionality to intermittently check that Market still has the access they expect. ### 7. Revoke all Alice revokes all approval for these tokens #### Technical calls Using near-cli: ```bash near call mt mt_revoke_all '\{ "token_ids": ["1", "2"], }' --accountId alice --amount .000000000000000000000001 ``` Again, note that no previous approvers will get cross-contract calls in this case. ## Reference-level explanation The `TokenApproval` structure returned by `mt_token_approvals` returns `approved_account_ids` field, which is a map of account IDs to `Approval` and `approval_owner_id` which is the associated account approved for removal from. The `amount` field though wrapped in quotes and treated like strings, the number will be stored as an unsigned integer with 128 bits. in approval is Using TypeScript's [Record type](https://www.typescriptlang.org/docs/handbook/utility-types.html#recordkeystype) notation: ```diff + type Approval = { + amount: string + approval_id: string + } + + type TokenApproval = { + approval_owner_id: string, + approved_account_ids: Record, + }; ``` Example token approval data: ```json [{ "approval_owner_id": "alice.near", "approved_account_ids": { "bob.near": { "amount": "100", "approval_id":1, }, "carol.near": { "amount":"2", "approval_id": 2, } } }] ``` ### What is an "approval ID"? This is a unique number given to each approval that allows well-intentioned marketplaces or other 3rd-party MT resellers to avoid a race condition. The race condition occurs when: 1. A token is listed in two marketplaces, which are both saved to the token as approved accounts. 2. One marketplace sells the token, which clears the approved accounts. 3. The new owner sells back to the original owner. 4. The original owner approves the token for the second marketplace again to list at a new price. But for some reason the second marketplace still lists the token at the previous price and is unaware of the transfers happening. 5. The second marketplace, operating from old information, attempts to again sell the token at the old price. Note that while this describes an honest mistake, the possibility of such a bug can also be taken advantage of by malicious parties via [front-running](https://users.encs.concordia.ca/~clark/papers/2019_wtsc_front.pdf). To avoid this possibility, the MT contract generates a unique approval ID each time it approves an account. Then when calling `mt_transfer`, `mt_transfer_call`, `mt_batch_transfer`, or `mt_batch_transfer_call` the approved account passes `approval_id` or `approval_ids` with this value to make sure the underlying state of the token(s) hasn't changed from what the approved account expects. Keeping with the example above, say the initial approval of the second marketplace generated the following `approved_account_ids` data: ```json { "approval_owner_id": "alice.near", "approved_account_ids": { "marketplace_1.near": { "approval_id": 1, "amount": "100", }, "marketplace_2.near": 2, "approval_id": 2, "amount": "50", } } ``` But after the transfers and re-approval described above, the token might have `approved_account_ids` as: ```json { "approval_owner_id": "alice.near", "approved_account_ids": { "marketplace_2.near": { "approval_id": 3, "amount": "50", } } } ``` The marketplace then tries to call `mt_transfer`, passing outdated information: ```bash # oops! near call mt-contract.near mt_transfer '{"account_id": "someacct", "amount":"50", "approval_id": 2 }' ``` ### Interface The MT contract must implement the following methods: ```ts /******************/ /* CHANGE METHODS */ /******************/ // Add an approved account for a specific set of tokens. // // Requirements // * Caller of the method must attach a deposit of at least 1 yoctoⓃ for // security purposes // * Contract MAY require caller to attach larger deposit, to cover cost of // storing approver data // * Contract MUST panic if called by someone other than token owner // * Contract MUST panic if addition would cause `mt_revoke_all` to exceed // single-block gas limit. See below for more info. // * Contract MUST increment approval ID even if re-approving an account // * If successfully approved or if had already been approved, and if `msg` is // present, contract MUST call `mt_on_approve` on `account_id`. See // `mt_on_approve` description below for details. // // Arguments: // * `token_ids`: the token ids for which to add an approval // * `account_id`: the account to add to `approved_account_ids` // * `amounts`: the number of tokens to approve for transfer, wrapped in quotes and treated // like an array of string, although the numbers will be stored as an array of // unsigned integer with 128 bits. // * `msg`: optional string to be passed to `mt_on_approve` // // Returns void, if no `msg` given. Otherwise, returns promise call to // `mt_on_approve`, which can resolve with whatever it wants. function mt_approve( token_ids: [string], amounts: [string], account_id: string, msg: string|null, ): void|Promise {} // Revoke an approved account for a specific token. // // Requirements // * Caller of the method must attach a deposit of 1 yoctoⓃ for security // purposes // * If contract requires >1yN deposit on `mt_approve`, contract // MUST refund associated storage deposit when owner revokes approval // * Contract MUST panic if called by someone other than token owner // // Arguments: // * `token_ids`: the token for which to revoke approved_account_ids // * `account_id`: the account to remove from `approvals` function mt_revoke( token_ids: [string], account_id: string ) {} // Revoke all approved accounts for a specific token. // // Requirements // * Caller of the method must attach a deposit of 1 yoctoⓃ for security // purposes // * If contract requires >1yN deposit on `mt_approve`, contract // MUST refund all associated storage deposit when owner revokes approved_account_ids // * Contract MUST panic if called by someone other than token owner // // Arguments: // * `token_ids`: the token ids with approved_account_ids to revoke function mt_revoke_all(token_ids: [string]) {} /****************/ /* VIEW METHODS */ /****************/ // Check if tokens are approved for transfer by a given account, optionally // checking an approval_id // // Requirements: // * Contract MUST panic if `approval_ids` is not null and the length of // `approval_ids` is not equal to `token_ids` // // Arguments: // * `token_ids`: the tokens for which to check an approval // * `approved_account_id`: the account to check the existence of in `approved_account_ids` // * `amounts`: specify the positionally corresponding amount for the `token_id` // that at least must be approved. The number of tokens to approve for transfer, // wrapped in quotes and treated like an array of string, although the numbers will be // stored as an array of unsigned integer with 128 bits. // * `approval_ids`: an optional array of approval IDs to check against // current approval IDs for given account and `token_ids`. // // Returns: // if `approval_ids` is given, `true` if `approved_account_id` is approved with given `approval_id` // and has at least the amount specified approved otherwise, `true` if `approved_account_id` // is in list of approved accounts and has at least the amount specified approved // finally it returns false for all other states function mt_is_approved( token_ids: [string], approved_account_id: string, amounts: [string], approval_ids: number[]|null ): boolean {} // Get a the list of approvals for a given token_id and account_id // // Arguments: // * `token_id`: the token for which to check an approval // * `account_id`: the account to retrieve approvals for // // Returns a TokenApproval object, as described in Approval Management standard function mt_token_approval( token_id: string, account_id: string, ): TokenApproval {} // Get a list of all approvals for a given token_id // // Arguments: // * `from_index`: a string representing an unsigned 128-bit integer, // representing the starting index of tokens to return // * `limit`: the maximum number of tokens to return // // Returns an array of TokenApproval objects, as described in Approval Management standard, and an empty array if there are no approvals function mt_token_approvals( token_id: string, from_index: string|null, // default: "0" limit: number|null, ): TokenApproval[] {} ``` ### Why must `mt_approve` panic if `mt_revoke_all` would fail later? In the description of `mt_approve` above, it states: Contract MUST panic if addition would cause `mt_revoke_all` to exceed single-block gas limit. What does this mean? First, it's useful to understand what we mean by "single-block gas limit". This refers to the [hard cap on gas per block at the protocol layer](https://docs.near.org/docs/concepts/gas#thinking-in-gas). This number will increase over time. Removing data from a contract uses gas, so if an MT had a large enough number of approvals, `mt_revoke_all` would fail, because calling it would exceed the maximum gas. Contracts must prevent this by capping the number of approvals for a given token. However, it is up to contract authors to determine a sensible cap for their contract (and the single block gas limit at the time they deploy). Since contract implementations can vary, some implementations will be able to support a larger number of approvals than others, even with the same maximum gas per block. Contract authors may choose to set a cap of something small and safe like 10 approvals, or they could dynamically calculate whether a new approval would break future calls to `mt_revoke_all`. But every contract MUST ensure that they never break the functionality of `mt_revoke_all`. ### Approved Account Contract Interface If a contract that gets approved to transfer MTs wants to, it can implement `mt_on_approve` to update its own state when granted approval for a token: ```ts // Respond to notification that contract has been granted approval for a token. // // Notes // * Contract knows the token contract ID from `predecessor_account_id` // // Arguments: // * `token_ids`: the token_ids to which this contract has been granted approval // * `amounts`: the ositionally corresponding amount for the token_id // that at must be approved. The number of tokens to approve for transfer, // wrapped in quotes and treated like an array of string, although the numbers will be // stored as an array of unsigned integer with 128 bits. // * `owner_id`: the owner of the token // * `approval_ids`: the approval ID stored by NFT contract for this approval. // Expected to be a number within the 2^53 limit representable by JSON. // * `msg`: specifies information needed by the approved contract in order to // handle the approval. Can indicate both a function to call and the // parameters to pass to that function. function mt_on_approve( token_ids: [TokenId], amounts: [string], owner_id: string, approval_ids: [number], msg: string, ) {} ``` Note that the MT contract will fire-and-forget this call, ignoring any return values or errors generated. This means that even if the approved account does not have a contract or does not implement `mt_on_approve`, the approval will still work correctly from the point of view of the MT contract. Further note that there is no parallel `mt_on_revoke` when revoking either a single approval or when revoking all. This is partially because scheduling many `mt_on_revoke` calls when revoking all approvals could incur prohibitive [gas fees](https://docs.near.org/docs/concepts/gas). Apps and contracts which cache MT approvals can therefore not rely on having up-to-date information, and should periodically refresh their caches. Since this will be the necessary reality for dealing with `mt_revoke_all`, there is no reason to complicate `mt_revoke` with an `mt_on_revoke` call. ### No incurred cost for core MT behavior MT contracts should be implemented in a way to avoid extra gas fees for serialization & deserialization of `approved_account_ids` for calls to `mt_*` methods other than `mt_tokens`. See `near-contract-standards` [implementation of `ft_metadata` using `LazyOption`](https://github.com/near/near-sdk-rs/blob/c2771af7fdfe01a4e8414046752ee16fb0d29d39/examples/fungible-token/ft/src/lib.rs#L71) as a reference example. ================================================ FILE: neps/nep-0245/Enumeration.md ================================================ # Multi Token Enumeration :::caution This is part of the proposed spec [NEP-245](https://github.com/near/NEPs/blob/master/neps/nep-0245.md) and is subject to change. ::: Version `1.0.0` ## Summary Standard interfaces for counting & fetching tokens, for an entire Multi Token contract or for a given owner. ## Motivation Apps such as marketplaces and wallets need a way to show all tokens owned by a given account and to show statistics about all tokens for a given contract. This extension provides a standard way to do so. While some Multi Token contracts may forego this extension to save [storage] costs, this requires apps to have custom off-chain indexing layers. This makes it harder for apps to integrate with such Multi Token contracts. Apps which integrate only with Multi Token Standards that use the Enumeration extension do not even need a server-side component at all, since they can retrieve all information they need directly from the blockchain. Prior art: - [ERC-721]'s enumeration extension - [Non Fungible Token Standard's](https://github.com/near/NEPs/blob/master/neps/nep-0181.md) enumeration extension ## Interface The contract must implement the following view methods: // Metadata field is optional if metadata extension is implemented. Includes the base token metadata id and the token_metadata object, that represents the token specific metadata. ```ts // Get a list of all tokens // // Arguments: // * `from_index`: a string representing an unsigned 128-bit integer, // representing the starting index of tokens to return // * `limit`: the maximum number of tokens to return // // Returns an array of `Token` objects, as described in the Core standard, // and an empty array if there are no tokens function mt_tokens( from_index: string|null, // default: "0" limit: number|null, // default: unlimited (could fail due to gas limit) ): Token[] {} // Get list of all tokens owned by a given account // // Arguments: // * `account_id`: a valid NEAR account // * `from_index`: a string representing an unsigned 128-bit integer, // representing the starting index of tokens to return // * `limit`: the maximum number of tokens to return // // Returns a paginated list of all tokens owned by this account, and an empty array if there are no tokens function mt_tokens_for_owner( account_id: string, from_index: string|null, // default: 0 limit: number|null, // default: unlimited (could fail due to gas limit) ): Token[] {} ``` The contract must implement the following view methods if using metadata extension: ```ts // Get list of all base metadata for the contract // // Arguments: // * `from_index`: a string representing an unsigned 128-bit integer, // representing the starting index of tokens to return // * `limit`: the maximum number of tokens to return // // Returns an array of `MTBaseTokenMetadata` objects, as described in the Metadata standard, and an empty array if there are no tokens function mt_tokens_base_metadata_all( from_index: string | null, limit: number | null ): MTBaseTokenMetadata[] ``` ## Notes At the time of this writing, the specialized collections in the `near-sdk` Rust crate are iterable, but not all of them have implemented an `iter_from` solution. There may be efficiency gains for large collections and contract developers are encouraged to test their data structures with a large amount of entries. [ERC-721]: https://eips.ethereum.org/EIPS/eip-721 [storage]: https://docs.near.org/concepts/storage/storage-staking ================================================ FILE: neps/nep-0245/Events.md ================================================ # Multi Token Event :::caution This is part of the proposed spec [NEP-245](https://github.com/near/NEPs/blob/master/neps/nep-0245.md) and is subject to change. ::: Version `1.0.0` ## Summary Standard interfaces for Multi Token Contract actions. Extension of [NEP-297](https://github.com/near/NEPs/blob/master/neps/nep-0297.md) ## Motivation NEAR and third-party applications need to track `mint`, `burn`, `transfer` events for all MT-driven apps consistently. This exension addresses that. Note that applications, including NEAR Wallet, could require implementing additional methods to display tokens correctly such as [`mt_metadata`](Metadata.md) and [`mt_tokens_for_owner`](Enumeration.md). ## Interface Multi Token Events MUST have `standard` set to `"nep245"`, standard version set to `"1.0.0"`, `event` value is one of `mt_mint`, `mt_burn`, `mt_transfer`, and `data` must be of one of the following relavant types: `MtMintLog[] | MtBurnLog[] | MtTransferLog[]`: ```ts interface MtEventLogData { EVENT_JSON: { standard: "nep245", version: "1.0.0", event: MtEvent, data: MtMintLog[] | MtBurnLog[] | MtTransferLog[] } } ``` ```ts // Minting event log. Emitted when a token is minted/created. // Requirements // * Contract MUST emit event when minting a token // Fields // * Contract token_ids and amounts MUST be the same length // * `owner_id`: the account receiving the minted token // * `token_ids`: the tokens minted // * `amounts`: the number of tokens minted, wrapped in quotes and treated // like a string, although the numbers will be stored as an unsigned integer // array with 128 bits. // * `memo`: optional message interface MtMintLog { owner_id: string, token_ids: string[], amounts: string[], memo?: string } // Burning event log. Emitted when a token is burned. // Requirements // * Contract MUST emit event when minting a token // Fields // * Contract token_ids and amounts MUST be the same length // * `owner_id`: the account whose token(s) are being burned // * `authorized_id`: approved account_id to burn, if applicable // * `token_ids`: the tokens being burned // * `amounts`: the number of tokens burned, wrapped in quotes and treated // like a string, although the numbers will be stored as an unsigned integer // array with 128 bits. // * `memo`: optional message interface MtBurnLog { owner_id: string, authorized_id?: string, token_ids: string[], amounts: string[], memo?: string } // Transfer event log. Emitted when a token is transferred. // Requirements // * Contract MUST emit event when transferring a token // Fields // * `authorized_id`: approved account_id to transfer // * `old_owner_id`: the account sending the tokens "sender.near" // * `new_owner_id`: the account receiving the tokens "receiver.near" // * `token_ids`: the tokens to transfer // * `amounts`: the number of tokens to transfer, wrapped in quotes and treated // like a string, although the numbers will be stored as an unsigned integer // array with 128 bits. interface MtTransferLog { authorized_id?: string, old_owner_id: string, new_owner_id: string, token_ids: string[], amounts: string[], memo?: string } ``` ## Examples Single owner minting (pretty-formatted for readability purposes): ```js EVENT_JSON:{ "standard": "nep245", "version": "1.0.0", "event": "mt_mint", "data": [ {"owner_id": "foundation.near", "token_ids": ["aurora", "proximitylabs_ft"], "amounts":["1", "100"]} ] } ``` Different owners minting: ```js EVENT_JSON:{ "standard": "nep245", "version": "1.0.0", "event": "mt_mint", "data": [ {"owner_id": "foundation.near", "token_ids": ["aurora", "proximitylabs_ft"], "amounts":["1","100"]}, {"owner_id": "user1.near", "token_ids": ["meme"], "amounts": ["1"]} ] } ``` Different events (separate log entries): ```js EVENT_JSON:{ "standard": "nep245", "version": "1.0.0", "event": "mt_burn", "data": [ {"owner_id": "foundation.near", "token_ids": ["aurora", "proximitylabs_ft"], "amounts": ["1","100"]}, ] } ``` Authorized id: ```js EVENT_JSON:{ "standard": "nep245", "version": "1.0.0", "event": "mt_burn", "data": [ {"owner_id": "foundation.near", "token_ids": ["aurora_alpha", "proximitylabs_ft"], "amounts": ["1","100"], "authorized_id": "thirdparty.near" }, ] } ``` ```js EVENT_JSON:{ "standard": "nep245", "version": "1.0.0", "event": "mt_transfer", "data": [ {"old_owner_id": "user1.near", "new_owner_id": "user2.near", "token_ids": ["meme"], "amounts":["1"], "memo": "have fun!"} ] } EVENT_JSON:{ "standard": "nep245", "version": "1.0.0", "event": "mt_transfer", "data": [ {"old_owner_id": "user2.near", "new_owner_id": "user3.near", "token_ids": ["meme"], "amounts":["1"], "authorized_id": "thirdparty.near", "memo": "have fun!"} ] } ``` ## Further methods Note that the example events covered above cover two different kinds of events: 1. Events that are not specified in the MT Standard (`mt_mint`, `mt_burn`) 2. An event that is covered in the [Multi Token Core Standard](https://github.com/near/NEPs/blob/master/neps/nep-0245.md). (`mt_transfer`) This event standard also applies beyond the three events highlighted here, where future events follow the same convention of as the second type. For instance, if an MT contract uses the [approval management standard](ApprovalManagement.md), it may emit an event for `mt_approve` if that's deemed as important by the developer community. Please feel free to open pull requests for extending the events standard detailed here as needs arise. ## Drawbacks There is a known limitation of 16kb strings when capturing logs. This can be observed from `token_ids` that may vary in length for different apps so the amount of logs that can be executed may vary. ================================================ FILE: neps/nep-0245/Metadata.md ================================================ # Multi Token Metadata :::caution This is part of the proposed spec [NEP-245](https://github.com/near/NEPs/blob/master/neps/nep-0245.md) and is subject to change. ::: Version `1.0.0` ## Summary An interface for a multi token's metadata. The goal is to keep the metadata future-proof as well as lightweight. This will be important to dApps needing additional information about multi token properties, and broadly compatible with other token standards such that the [NEAR Rainbow Bridge](https://near.org/blog/eth-near-rainbow-bridge/) can move tokens between chains. ## Motivation The primary value of tokens comes from their metadata. While the [core standard](https://github.com/near/NEPs/blob/master/neps/nep-0245.md) provides the minimum interface that can be considered a multi token, most artists, developers, and dApps will want to associate more data with each token, and will want a predictable way to interact with any MT's metadata. NEAR's unique [storage staking](https://docs.near.org/concepts/storage/storage-staking) approach makes it feasible to store more data on-chain than other blockchains. This standard leverages this strength for common metadata attributes, and provides a standard way to link to additional offchain data to support rapid community experimentation. This standard also provides a `spec` version. This makes it easy for consumers of Multi Tokens, such as marketplaces, to know if they support all the features of a given token. Prior art: - NEAR's [Fungible Token Metadata Standard](https://github.com/near/NEPs/blob/master/neps/nep-0148.md) - NEAR's [Non-Fungible Token Metadata Standard](https://github.com/near/NEPs/blob/master/neps/nep-0177.md) - Discussion about NEAR's complete NFT standard: #171 - Discussion about NEAR's complete Multi Token standard: #245 ## Interface Metadata applies at both the class level (`MTBaseTokenMetadata`) and the specific instance level (`MTTokenMetadata`). The relevant metadata for each: ```ts type MTContractMetadata = { spec: string, // required, essentially a version like "mt-1.0.0" name: string, // required Zoink's Digitial Sword Collection } type MTBaseTokenMetadata = { name: string, // required, ex. "Silver Swords" or "Metaverse 3" id: string, // required a unique identifier for the metadata symbol: string|null, // required, ex. "MOCHI" icon: string|null, // Data URL decimals: string|null // number of decimals for the token useful for FT related tokens base_uri: string|null, // Centralized gateway known to have reliable access to decentralized storage assets referenced by `reference` or `media` URLs reference: string|null, // URL to a JSON file with more info copies: number|null, // number of copies of this set of metadata in existence when token was minted. reference_hash: string|null, // Base64-encoded sha256 hash of JSON from reference field. Required if `reference` is included. } type MTTokenMetadata = { title: string|null, // ex. "Arch Nemesis: Mail Carrier" or "Parcel #5055" description: string|null, // free-form description media: string|null, // URL to associated media, preferably to decentralized, content-addressed storage media_hash: string|null, // Base64-encoded sha256 hash of content referenced by the `media` field. Required if `media` is included. issued_at: string|null, // When token was issued or minted, Unix epoch in milliseconds expires_at: string|null, // When token expires, Unix epoch in milliseconds starts_at: string|null, // When token starts being valid, Unix epoch in milliseconds updated_at: string|null, // When token was last updated, Unix epoch in milliseconds extra: string|null, // Anything extra the MT wants to store on-chain. Can be stringified JSON. reference: string|null, // URL to an off-chain JSON file with more info. reference_hash: string|null // Base64-encoded sha256 hash of JSON from reference field. Required if `reference` is included. } type MTTokenMetadataAll = { base: MTBaseTokenMetadata token: MTTokenMetadata } ``` A new set of functions MUST be supported on the MT contract: ```ts // Returns the top-level contract level metadtata function mt_metadata_contract(): MTContractMetadata {} function mt_metadata_token_all(token_ids: string[]): MTTokenMetadataAll[] function mt_metadata_token_by_token_id(token_ids: string[]): MTTokenMetadata[] function mt_metadata_base_by_token_id(token_ids: string[]): MTBaseTokenMetadata[] function mt_metadata_base_by_metadata_id(base_metadata_ids: string[]): MTBaseTokenMetadata[] ``` A new attribute MUST be added to each `Token` struct: ```diff type Token = { token_id: string, + token_metadata?: MTTokenMetadata, + base_metadata_id: string, } ``` ### An implementing contract MUST include the following fields on-chain For `MTContractMetadata`: - `spec`: a string that MUST be formatted `mt-1.0.0` to indicate that a Multi Token contract adheres to the current versions of this Metadata spec. This will allow consumers of the Multi Token to know if they support the features of a given contract. - `name`: the human-readable name of the contract. ### An implementing contract must include the following fields on-chain For `MTBaseTokenMetadata`: - `name`: the human-readable name of the Token. - `base_uri`: Centralized gateway known to have reliable access to decentralized storage assets referenced by `reference` or `media` URLs. Can be used by other frontends for initial retrieval of assets, even if these frontends then replicate the data to their own decentralized nodes, which they are encouraged to do. ### An implementing contract MAY include the following fields on-chain For `MTBaseTokenMetadata`: - `symbol`: the abbreviated symbol of the contract, like MOCHI or MV3 - `icon`: a small image associated with this contract. Encouraged to be a [data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URIs), to help consumers display it quickly while protecting user data. Recommendation: use [optimized SVG](https://codepen.io/tigt/post/optimizing-svgs-in-data-uris), which can result in high-resolution images with only 100s of bytes of [storage cost](https://docs.near.org/concepts/storage/storage-staking). (Note that these storage costs are incurred to the contract deployer, but that querying these icons is a very cheap & cacheable read operation for all consumers of the contract and the RPC nodes that serve the data.) Recommendation: create icons that will work well with both light-mode and dark-mode websites by either using middle-tone color schemes, or by [embedding `media` queries in the SVG](https://timkadlec.com/2013/04/media-queries-within-svg/). - `reference`: a link to a valid JSON file containing various keys offering supplementary details on the token. Example: `/ipfs/QmdmQXB2mzChmMeKY47C43LxUdg1NDJ5MWcKMKxDu7RgQm`, etc. If the information given in this document conflicts with the on-chain attributes, the values in `reference` shall be considered the source of truth. - `reference_hash`: the base64-encoded sha256 hash of the JSON file contained in the `reference` field. This is to guard against off-chain tampering. - `copies`: The number of tokens with this set of metadata or `media` known to exist at time of minting. Supply is a more accurate current reflection. For `MTTokenMetadata`: - `title`: The title of this specific token. - `description`: A longer description of the token. - `media`: URL to associated media. Preferably to decentralized, content-addressed storage. - `media_hash`: the base64-encoded sha256 hash of content referenced by the `media` field. This is to guard against off-chain tampering. - `copies`: The number of tokens with this set of metadata or `media` known to exist at time of minting. - `issued_at`: Unix epoch in milliseconds when token was issued or minted (an unsigned 32-bit integer would suffice until the year 2106) - `expires_at`: Unix epoch in milliseconds when token expires - `starts_at`: Unix epoch in milliseconds when token starts being valid - `updated_at`: Unix epoch in milliseconds when token was last updated - `extra`: anything extra the MT wants to store on-chain. Can be stringified JSON. - `reference`: URL to an off-chain JSON file with more info. - `reference_hash`: Base64-encoded sha256 hash of JSON from reference field. Required if `reference` is included. For `MTTokenMetadataAll `: - `base`: The base metadata that corresponds to `MTBaseTokenMetadata` for the token. - `token`: The token specific metadata that corresponds to `MTTokenMetadata`. ### No incurred cost for core MT behavior Contracts should be implemented in a way to avoid extra gas fees for serialization & deserialization of metadata for calls to `mt_*` methods other than `mt_metadata*` or `mt_tokens`. See `near-contract-standards` [implementation using `LazyOption`](https://github.com/near/near-sdk-rs/blob/c2771af7fdfe01a4e8414046752ee16fb0d29d39/examples/fungible-token/ft/src/lib.rs#L71) as a reference example. ## Drawbacks - When this MT contract is created and initialized, the storage use per-token will be higher than an MT Core version. Frontends can account for this by adding extra deposit when minting. This could be done by padding with a reasonable amount, or by the frontend using the [RPC call detailed here](https://docs.near.org/docs/develop/front-end/rpc#genesis-config) that gets genesis configuration and actually determine precisely how much deposit is needed. - Convention of `icon` being a data URL rather than a link to an HTTP endpoint that could contain privacy-violating code cannot be done on deploy or update of contract metadata, and must be done on the consumer/app side when displaying token data. - If on-chain icon uses a data URL or is not set but the document given by `reference` contains a privacy-violating `icon` URL, consumers & apps of this data should not naïvely display the `reference` version, but should prefer the safe version. This is technically a violation of the "`reference` setting wins" policy described above. ## Future possibilities - Detailed conventions that may be enforced for versions. - A fleshed out schema for what the `reference` object should contain. ================================================ FILE: neps/nep-0245.md ================================================ --- NEP: 245 Title: Multi Token Standard Author: Zane Starr , @riqi, @jriemann, @marcos.sun Status: Final DiscussionsTo: https://github.com/near/NEPs/discussions/246 Type: Standards Track Category: Contract Created: 03-Mar-2022 Requires: 297 --- ## Summary A standard interface for a multi token standard that supports fungible, semi-fungible,non-fungible, and tokens of any type, allowing for ownership, transfer, and batch transfer of tokens regardless of specific type. ### Extensions - [Approval Management](nep-0245/ApprovalManagement.md) - [Enumeration](nep-0245/Enumeration.md) - [Events](nep-0245/Events.md) - [Metadata](nep-0245/Metadata.md) ## Motivation In the three years since [ERC-1155] was ratified by the Ethereum Community, Multi Token based contracts have proven themselves valuable assets. Many blockchain projects emulate this standard for representing multiple token assets classes in a single contract. The ability to reduce transaction overhead for marketplaces, video games, DAOs, and exchanges is appealing to the blockchain ecosystem and simplifies transactions for developers. Having a single contract represent NFTs, FTs, and tokens that sit inbetween greatly improves efficiency. The standard also introduced the ability to make batch requests with multiple asset classes reducing complexity. This standard allows operations that currently require _many_ transactions to be completed in a single transaction that can transfer not only NFTs and FTs, but any tokens that are a part of same token contract. With this standard, we have sought to take advantage of the ability of the NEAR blockchain to scale. Its sharded runtime, and [storage staking] model that decouples [gas] fees from storage demand, enables ultra low transaction fees and greater on chain storage (see [Metadata][MT Metadata] extension). With the aforementioned, it is noteworthy to mention that like the [NFT] standard the Multi Token standard, implements `mt_transfer_call`, which allows, a user to attach many tokens to a call to a separate contract. Additionally, this standard includes an optional [Approval Management] extension. The extension allows marketplaces to trade on behalf of a user, providing additional flexibility for dApps. Prior art: - [ERC-721] - [ERC-1155] - [NEAR Fungible Token Standard][FT Core], which first pioneered the "transfer and call" technique - [NEAR Non-Fungible Token Standard][NFT Core] ## Rationale and alternatives Why have another standard, aren't fungible and non-fungible tokens enough? The current fungible token and non-fungible token standards, do not provide support for representing many FT tokens in a single contract, as well as the flexibility to define different token types with different behavior in a single contract. This is something that makes it difficult to be interoperable with other major blockchain networks, that implement standards that allow for representation of many different FT tokens in a single contract such as Ethereum. The standard here introduces a few concepts that evolve the original [ERC-1155] standard to have more utility, while maintaining the original flexibility of the standard. So keeping that in mind, we are defining this as a new token type. It combines two main features of FT and NFT. It allows us to represent many token types in a single contract, and it's possible to store the amount for each token. The decision to not use FT and NFT as explicit token types was taken to allow the community to define their own standards and meanings through metadata. As standards evolve on other networks, this specification allows the standard to be able to represent tokens across networks accurately, without necessarily restricting the behavior to any preset definition. The issues with this in general is a problem with defining what metadata means and how is that interpreted. We have chosen to follow the pattern that is currently in use on Ethereum in the [ERC-1155] standard. That pattern relies on people to make extensions or to make signals as to how they want the metadata to be represented for their use case. One of the areas that has broad sweeping implications from the [ERC-1155] standard is the lack of direct access to metadata. With Near's sharding we are able to have a [Metadata Extension][MT Metadata] for the standard that exists on chain. So developers and users are not required to use an indexer to understand, how to interact or interpret tokens, via token identifiers that they receive. Another extension that we made was to provide an explicit ability for developers and users to group or link together series of NFTs/FTs or any combination of tokens. This provides additional flexibility that the [ERC-1155] standard only has loose guidelines on. This was chosen to make it easy for consumers to understand the relationship between tokens within the contract. To recap, we choose to create this standard, to improve interoperability, developer ease of use, and to extend token representability beyond what was available directly in the FT or NFT standards. We believe this to be another tool in the developer's toolkit. It makes it possible to represent many types of tokens and to enable exchanges of many tokens within a single `transaction`. ## Specification **NOTES**: - All amounts, balances and allowance are limited by `U128` (max value `2**128 - 1`). - Token standard uses JSON for serialization of arguments and results. - Amounts in arguments and results are serialized as Base-10 strings, e.g. `"100"`. This is done to avoid JSON limitation of max integer value of `2**53`. - The contract must track the change in storage when adding to and removing from collections. This is not included in this core multi token standard but instead in the [Storage Standard][Storage Management]. - To prevent the deployed contract from being modified or deleted, it should not have any access keys on its account. ### MT Interface ```ts // The base structure that will be returned for a token. If contract is using // extensions such as Approval Management, Enumeration, Metadata, or other // attributes may be included in this structure. type Token = { token_id: string, owner_id: string | null } /******************/ /* CHANGE METHODS */ /******************/ // Simple transfer. Transfer a given `token_id` from current owner to // `receiver_id`. // // Requirements // * Caller of the method must attach a deposit of 1 yoctoⓃ for security purposes // * Caller must have greater than or equal to the `amount` being requested // * Contract MUST panic if called by someone other than token owner or, // if using Approval Management, one of the approved accounts // * `approval_id` is for use with Approval Management extension, see // that document for full explanation. // * If using Approval Management, contract MUST nullify approved accounts on // successful transfer. // // Arguments: // * `receiver_id`: the valid NEAR account receiving the token // * `token_id`: the token to transfer // * `amount`: the number of tokens to transfer, wrapped in quotes and treated // like a string, although the number will be stored as an unsigned integer // with 128 bits. // * `approval` (optional): is a tuple of [`owner_id`,`approval_id`]. // `owner_id` is the valid Near account that owns the tokens. // `approval_id` is the expected approval ID. A number smaller than // 2^53, and therefore representable as JSON. See Approval Management // standard for full explanation. // * `memo` (optional): for use cases that may benefit from indexing or // providing information for a transfer function mt_transfer( receiver_id: string, token_id: string, amount: string, approval: [owner_id: string, approval_id: number]|null, memo: string|null, ) {} // Simple batch transfer. Transfer a given `token_ids` from current owner to // `receiver_id`. // // Requirements // * Caller of the method must attach a deposit of 1 yoctoⓃ for security purposes // * Caller must have greater than or equal to the `amounts` being requested for the given `token_ids` // * Contract MUST panic if called by someone other than token owner or, // if using Approval Management, one of the approved accounts // * `approval_id` is for use with Approval Management extension, see // that document for full explanation. // * If using Approval Management, contract MUST nullify approved accounts on // successful transfer. // * Contract MUST panic if called with the length of `token_ids` not equal to `amounts` is not equal // * Contract MUST panic if `approval_ids` is not `null` and does not equal the length of `token_ids` // // Arguments: // * `receiver_id`: the valid NEAR account receiving the token // * `token_ids`: the tokens to transfer // * `amounts`: the number of tokens to transfer, wrapped in quotes and treated // like an array of strings, although the numbers will be stored as an array of unsigned integer // with 128 bits. // * `approvals` (optional): is an array of expected `approval` per `token_ids`. // If a `token_id` does not have a corresponding `approval` then the entry in the array // must be marked null. // `approval` is a tuple of [`owner_id`,`approval_id`]. // `owner_id` is the valid Near account that owns the tokens. // `approval_id` is the expected approval ID. A number smaller than // 2^53, and therefore representable as JSON. See Approval Management // standard for full explanation. // * `memo` (optional): for use cases that may benefit from indexing or // providing information for a transfer function mt_batch_transfer( receiver_id: string, token_ids: string[], amounts: string[], approvals: ([owner_id: string, approval_id: number]| null)[]| null, memo: string|null, ) {} // Transfer token and call a method on a receiver contract. A successful // workflow will end in a success execution outcome to the callback on the MT // contract at the method `mt_resolve_transfer`. // // You can think of this as being similar to attaching native NEAR tokens to a // function call. It allows you to attach any Multi Token, token in a call to a // receiver contract. // // Requirements: // * Caller of the method must attach a deposit of 1 yoctoⓃ for security // purposes // * Caller must have greater than or equal to the `amount` being requested // * Contract MUST panic if called by someone other than token owner or, // if using Approval Management, one of the approved accounts // * The receiving contract must implement `mt_on_transfer` according to the // standard. If it does not, MT contract's `mt_resolve_transfer` MUST deal // with the resulting failed cross-contract call and roll back the transfer. // * Contract MUST implement the behavior described in `mt_resolve_transfer` // * `approval_id` is for use with Approval Management extension, see // that document for full explanation. // * If using Approval Management, contract MUST nullify approved accounts on // successful transfer. // // Arguments: // * `receiver_id`: the valid NEAR account receiving the token. // * `token_id`: the token to send. // * `amount`: the number of tokens to transfer, wrapped in quotes and treated // like a string, although the number will be stored as an unsigned integer // with 128 bits. // * `owner_id`: the valid NEAR account that owns the token // * `approval` (optional): is a tuple of [`owner_id`,`approval_id`]. // `owner_id` is the valid Near account that owns the tokens. // `approval_id` is the expected approval ID. A number smaller than // 2^53, and therefore representable as JSON. See Approval Management // * `memo` (optional): for use cases that may benefit from indexing or // providing information for a transfer. // * `msg`: specifies information needed by the receiving contract in // order to properly handle the transfer. Can indicate both a function to // call and the parameters to pass to that function. function mt_transfer_call( receiver_id: string, token_id: string, amount: string, approval: [owner_id: string, approval_id: number]|null, memo: string|null, msg: string, ): Promise {} // Transfer tokens and call a method on a receiver contract. A successful // workflow will end in a success execution outcome to the callback on the MT // contract at the method `mt_resolve_transfer`. // // You can think of this as being similar to attaching native NEAR tokens to a // function call. It allows you to attach any Multi Token, token in a call to a // receiver contract. // // Requirements: // * Caller of the method must attach a deposit of 1 yoctoⓃ for security // purposes // * Caller must have greater than or equal to the `amount` being requested // * Contract MUST panic if called by someone other than token owner or, // if using Approval Management, one of the approved accounts // * The receiving contract must implement `mt_on_transfer` according to the // standard. If it does not, MT contract's `mt_resolve_transfer` MUST deal // with the resulting failed cross-contract call and roll back the transfer. // * Contract MUST implement the behavior described in `mt_resolve_transfer` // * `approval_id` is for use with Approval Management extension, see // that document for full explanation. // * If using Approval Management, contract MUST nullify approved accounts on // successful transfer. // * Contract MUST panic if called with the length of `token_ids` not equal to `amounts` is not equal // * Contract MUST panic if `approval_ids` is not `null` and does not equal the length of `token_ids` // // Arguments: // * `receiver_id`: the valid NEAR account receiving the token. // * `token_ids`: the tokens to transfer // * `amounts`: the number of tokens to transfer, wrapped in quotes and treated // like an array of string, although the numbers will be stored as an array of // unsigned integer with 128 bits. // * `approvals` (optional): is an array of expected `approval` per `token_ids`. // If a `token_id` does not have a corresponding `approval` then the entry in the array // must be marked null. // `approval` is a tuple of [`owner_id`,`approval_id`]. // `owner_id` is the valid Near account that owns the tokens. // `approval_id` is the expected approval ID. A number smaller than // 2^53, and therefore representable as JSON. See Approval Management // standard for full explanation. // * `memo` (optional): for use cases that may benefit from indexing or // providing information for a transfer. // * `msg`: specifies information needed by the receiving contract in // order to properly handle the transfer. Can indicate both a function to // call and the parameters to pass to that function. function mt_batch_transfer_call( receiver_id: string, token_ids: string[], amounts: string[], approvals: ([owner_id: string, approval_id: number]|null)[] | null, memo: string|null, msg: string, ): Promise {} /****************/ /* VIEW METHODS */ /****************/ // Returns the tokens with the given `token_ids` or `null` if no such token. function mt_token(token_ids: string[]) (Token | null)[] // Returns the balance of an account for the given `token_id`. // The balance though wrapped in quotes and treated like a string, // the number will be stored as an unsigned integer with 128 bits. // Arguments: // * `account_id`: the NEAR account that owns the token. // * `token_id`: the token to retrieve the balance from function mt_balance_of(account_id: string, token_id: string): string // Returns the balances of an account for the given `token_ids`. // The balances though wrapped in quotes and treated like strings, // the numbers will be stored as an unsigned integer with 128 bits. // Arguments: // * `account_id`: the NEAR account that owns the tokens. // * `token_ids`: the tokens to retrieve the balance from function mt_batch_balance_of(account_id: string, token_ids: string[]): string[] // Returns the token supply with the given `token_id` or `null` if no such token exists. // The supply though wrapped in quotes and treated like a string, the number will be stored // as an unsigned integer with 128 bits. function mt_supply(token_id: string): string | null // Returns the token supplies with the given `token_ids`, a string value is returned or `null` // if no such token exists. The supplies though wrapped in quotes and treated like strings, // the numbers will be stored as an unsigned integer with 128 bits. function mt_batch_supply(token_ids: string[]): (string | null)[] ``` The following behavior is required, but contract authors may name this function something other than the conventional `mt_resolve_transfer` used here. ```ts // Finalize an `mt_transfer_call` or `mt_batch_transfer_call` chain of cross-contract calls. Generically // referred to as `mt_transfer_call` as it applies to `mt_batch_transfer_call` as well. // // The `mt_transfer_call` process: // // 1. Sender calls `mt_transfer_call` on MT contract // 2. MT contract transfers token from sender to receiver // 3. MT contract calls `mt_on_transfer` on receiver contract // 4+. [receiver contract may make other cross-contract calls] // N. MT contract resolves promise chain with `mt_resolve_transfer`, and may // transfer token back to sender // // Requirements: // * Contract MUST forbid calls to this function by any account except self // * If promise chain failed, contract MUST revert token transfer // * If promise chain resolves with `true`, contract MUST return token to // `sender_id` // // Arguments: // * `sender_id`: the sender of `mt_transfer_call` // * `receiver_id`: the `receiver_id` argument given to `mt_transfer_call` // * `token_ids`: the `token_ids` argument given to `mt_transfer_call` // * `amounts`: the `token_ids` argument given to `mt_transfer_call` // * `approvals (optional)`: if using Approval Management, contract MUST provide // set of original approvals in this argument, and restore the // approved accounts in case of revert. // `approvals` is an array of expected `approval_list` per `token_ids`. // If a `token_id` does not have a corresponding `approvals_list` then the entry in the // array must be marked null. // `approvals_list` is an array of triplets of [`owner_id`,`approval_id`,`amount`]. // `owner_id` is the valid Near account that owns the tokens. // `approval_id` is the expected approval ID. A number smaller than // 2^53, and therefore representable as JSON. See Approval Management // standard for full explanation. // `amount`: the number of tokens to transfer, wrapped in quotes and treated // like a string, although the number will be stored as an unsigned integer // with 128 bits. // // // // Returns total amount spent by the `receiver_id`, corresponding to the `token_id`. // The amounts returned, though wrapped in quotes and treated like strings, // the numbers will be stored as an unsigned integer with 128 bits. // Example: if sender_id calls `mt_transfer_call({ "amounts": ["100"], token_ids: ["55"], receiver_id: "games" })`, // but `receiver_id` only uses 80, `mt_on_transfer` will resolve with `["20"]`, and `mt_resolve_transfer` // will return `["80"]`. function mt_resolve_transfer( sender_id: string, receiver_id: string, token_ids: string[], approvals: (null | [owner_id: string, approval_id: number, amount: string][]) []| null ):string[] {} ``` ### Receiver Interface Contracts which want to make use of `mt_transfer_call` and `mt_batch_transfer_call` must implement the following: ```ts // Take some action after receiving a multi token // // Requirements: // * Contract MUST restrict calls to this function to a set of whitelisted // contracts // * Contract MUST panic if `token_ids` length does not equals `amounts` // length // * Contract MUST panic if `previous_owner_ids` length does not equals `token_ids` // length // // Arguments: // * `sender_id`: the sender of `mt_transfer_call` // * `previous_owner_ids`: the account that owned the tokens prior to it being // transferred to this contract, which can differ from `sender_id` if using // Approval Management extension // * `token_ids`: the `token_ids` argument given to `mt_transfer_call` // * `amounts`: the `token_ids` argument given to `mt_transfer_call` // * `msg`: information necessary for this contract to know how to process the // request. This may include method names and/or arguments. // // Returns the number of unused tokens in string form. For instance, if `amounts` // is `["10"]` but only 9 are needed, it will return `["1"]`. The amounts returned, // though wrapped in quotes and treated like strings, the numbers will be stored as // an unsigned integer with 128 bits. function mt_on_transfer( sender_id: string, previous_owner_ids: string[], token_ids: string[], amounts: string[], msg: string, ): Promise; ``` ## Events NEAR and third-party applications need to track `mint`, `burn`, `transfer` events for all MT-driven apps consistently. [This extension][MT Events] addresses that. Note that applications, including NEAR Wallet, could require implementing additional methods to display tokens correctly such as [`mt_metadata`][MT Metadata] and [`mt_tokens_for_owner`][MT Enumeration]. ### Events Interface [Multi Token Events][MT Events] MUST have `standard` set to `"nep245"`, standard version set to `"1.0.0"`, `event` value is one of `mt_mint`, `mt_burn`, `mt_transfer`, and `data` must be of one of the following relevant types: `MtMintLog[] | MtBurnLog[] | MtTransferLog[]`: ```ts interface MtEventLogData { EVENT_JSON: { standard: "nep245", version: "1.0.0", event: MtEvent, data: MtMintLog[] | MtBurnLog[] | MtTransferLog[] } } ``` ```ts // Minting event log. Emitted when a token is minted/created. // Requirements // * Contract MUST emit event when minting a token // Fields // * Contract token_ids and amounts MUST be the same length // * `owner_id`: the account receiving the minted token // * `token_ids`: the tokens minted // * `amounts`: the number of tokens minted, wrapped in quotes and treated // like a string, although the numbers will be stored as an unsigned integer // array with 128 bits. // * `memo`: optional message interface MtMintLog { owner_id: string, token_ids: string[], amounts: string[], memo?: string } // Burning event log. Emitted when a token is burned. // Requirements // * Contract MUST emit event when minting a token // Fields // * Contract token_ids and amounts MUST be the same length // * `owner_id`: the account whose token(s) are being burned // * `authorized_id`: approved account_id to burn, if applicable // * `token_ids`: the tokens being burned // * `amounts`: the number of tokens burned, wrapped in quotes and treated // like a string, although the numbers will be stored as an unsigned integer // array with 128 bits. // * `memo`: optional message interface MtBurnLog { owner_id: string, authorized_id?: string, token_ids: string[], amounts: string[], memo?: string } // Transfer event log. Emitted when a token is transferred. // Requirements // * Contract MUST emit event when transferring a token // Fields // * `authorized_id`: approved account_id to transfer // * `old_owner_id`: the account sending the tokens "sender.near" // * `new_owner_id`: the account receiving the tokens "receiver.near" // * `token_ids`: the tokens to transfer // * `amounts`: the number of tokens to transfer, wrapped in quotes and treated // like a string, although the numbers will be stored as an unsigned integer // array with 128 bits. interface MtTransferLog { authorized_id?: string, old_owner_id: string, new_owner_id: string, token_ids: string[], amounts: string[], memo?: string } ``` ## Examples Single owner minting (pretty-formatted for readability purposes): ```js EVENT_JSON:{ "standard": "nep245", "version": "1.0.0", "event": "mt_mint", "data": [ {"owner_id": "foundation.near", "token_ids": ["aurora", "proximitylabs_ft"], "amounts":["1", "100"]} ] } ``` Different owners minting: ```js EVENT_JSON:{ "standard": "nep245", "version": "1.0.0", "event": "mt_mint", "data": [ {"owner_id": "foundation.near", "token_ids": ["aurora", "proximitylabs_ft"], "amounts":["1","100"]}, {"owner_id": "user1.near", "token_ids": ["meme"], "amounts": ["1"]} ] } ``` Different events (separate log entries): ```js EVENT_JSON:{ "standard": "nep245", "version": "1.0.0", "event": "mt_burn", "data": [ {"owner_id": "foundation.near", "token_ids": ["aurora", "proximitylabs_ft"], "amounts": ["1","100"]}, ] } ``` Authorized id: ```js EVENT_JSON:{ "standard": "nep245", "version": "1.0.0", "event": "mt_burn", "data": [ {"owner_id": "foundation.near", "token_ids": ["aurora_alpha", "proximitylabs_ft"], "amounts": ["1","100"], "authorized_id": "thirdparty.near" }, ] } ``` ```js EVENT_JSON:{ "standard": "nep245", "version": "1.0.0", "event": "mt_transfer", "data": [ {"old_owner_id": "user1.near", "new_owner_id": "user2.near", "token_ids": ["meme"], "amounts":["1"], "memo": "have fun!"} ] } EVENT_JSON:{ "standard": "nep245", "version": "1.0.0", "event": "mt_transfer", "data": [ {"old_owner_id": "user2.near", "new_owner_id": "user3.near", "token_ids": ["meme"], "amounts":["1"], "authorized_id": "thirdparty.near", "memo": "have fun!"} ] } ``` ## Further Event Methods Note that the example events covered above cover two different kinds of events: 1. Events that are not specified in the MT Standard (`mt_mint`, `mt_burn`) 2. An event that is covered in the [Multi Token Core Standard](https://nomicon.io/Standards/Tokens/MultiToken/Core#mt-interface). (`mt_transfer`) This event standard also applies beyond the three events highlighted here, where future events follow the same convention of as the second type. For instance, if an MT contract uses the [approval management standard][MT Approval Management], it may emit an event for `mt_approve` if that's deemed as important by the developer community. Please feel free to open pull requests for extending the events standard detailed here as needs arise. ## Reference Implementation [Minimum Viable Interface](https://github.com/jriemann/near-sdk-rs/blob/multi-token-reference-impl/near-contract-standards/src/multi_token/core/mod.rs) [MT Implementation](https://github.com/jriemann/near-sdk-rs/blob/multi-token-reference-impl/near-contract-standards/src/multi_token/core/core_impl.rs) ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). [ERC-721]: https://eips.ethereum.org/EIPS/eip-721 [ERC-1155]: https://eips.ethereum.org/EIPS/eip-1155 [storage staking]: https://docs.near.org/concepts/storage/storage-staking [gas]: https://docs.near.org/concepts/basics/transactions/gas [NFT Core]: https://github.com/near/NEPs/blob/master/neps/nep-0171.md [FT Core]: https://github.com/near/NEPs/blob/master/neps/nep-0141.md [Storage Management]: https://github.com/near/NEPs/blob/master/neps/nep-0145.md [MT Approval Management]: nep-0245/ApprovalManagement.md [MT Enumeration]: nep-0245/Enumeration.md [MT Events]: nep-0245/Events.md [MT Metadata]: nep-0245/Metadata.md ================================================ FILE: neps/nep-0256.md ================================================ --- NEP: 256 Title: Non-Fungible Token Events Author: Olga Telezhnaya , @evergreen-trading-systems Status: Final DiscussionsTo: https://github.com/near/NEPs/pull/256, https://github.com/near/NEPs/issues/254 Type: Standards Track Category: Contract Created: 8-Sep-2021 --- # Events Version `1.1.0` ## Summary Standard interface for NFT contract actions based on [NEP-297](https://github.com/near/NEPs/blob/master/neps/nep-0297.md). ## Motivation NEAR and third-party applications need to track `mint`, `transfer`, `burn` and `contract_metadata_update` events for all NFT-driven apps consistently. This extension addresses that. Keep in mind that applications, including NEAR Wallet, could require implementing additional methods to display the NFTs correctly, such as [`nft_metadata`](https://github.com/near/NEPs/blob/master/neps/nep-0177.md) and [`nft_tokens_for_owner`](https://github.com/near/NEPs/blob/master/neps/nep-0181.md). ## Interface Non-Fungible Token Events MUST have `standard` set to `"nep171"`, standard version set to `"1.1.0"`, `event` value is one of `nft_mint`, `nft_burn`, `nft_transfer`, `contract_metadata_update`, and `data` must be of one of the following relavant types: `NftMintLog[] | NftTransferLog[] | NftBurnLog[] | NftContractMetadataUpdateLog[]`: ```ts interface NftEventLogData { standard: "nep171", version: "1.1.0", event: "nft_mint" | "nft_burn" | "nft_transfer" | "contract_metadata_update", data: NftMintLog[] | NftTransferLog[] | NftBurnLog[] | NftContractMetadataUpdateLog[], } ``` ```ts // An event log to capture token minting // Arguments // * `owner_id`: "account.near" // * `token_ids`: ["1", "abc"] // * `memo`: optional message interface NftMintLog { owner_id: string, token_ids: string[], memo?: string } // An event log to capture token burning // Arguments // * `owner_id`: owner of tokens to burn // * `authorized_id`: approved account_id to burn, if applicable // * `token_ids`: ["1","2"] // * `memo`: optional message interface NftBurnLog { owner_id: string, authorized_id?: string, token_ids: string[], memo?: string } // An event log to capture token transfer // Arguments // * `authorized_id`: approved account_id to transfer, if applicable // * `old_owner_id`: "owner.near" // * `new_owner_id`: "receiver.near" // * `token_ids`: ["1", "12345abc"] // * `memo`: optional message interface NftTransferLog { authorized_id?: string, old_owner_id: string, new_owner_id: string, token_ids: string[], memo?: string } // An event log to capture contract metadata updates. Note that the updated contract metadata is not included in the log, as it could easily exceed the 16KB log size limit. Listeners can query `nft_metadata` to get the updated contract metadata. // Arguments // * `memo`: optional message interface NftContractMetadataUpdateLog { memo?: string } ``` ## Examples Single owner batch minting (pretty-formatted for readability purposes): ```js EVENT_JSON:{ "standard": "nep171", "version": "1.1.0", "event": "nft_mint", "data": [ {"owner_id": "foundation.near", "token_ids": ["aurora", "proximitylabs"]} ] } ``` Different owners batch minting: ```js EVENT_JSON:{ "standard": "nep171", "version": "1.1.0", "event": "nft_mint", "data": [ {"owner_id": "foundation.near", "token_ids": ["aurora", "proximitylabs"]}, {"owner_id": "user1.near", "token_ids": ["meme"]} ] } ``` Different events (separate log entries): ```js EVENT_JSON:{ "standard": "nep171", "version": "1.1.0", "event": "nft_burn", "data": [ {"owner_id": "foundation.near", "token_ids": ["aurora", "proximitylabs"]}, ] } ``` ```js EVENT_JSON:{ "standard": "nep171", "version": "1.1.0", "event": "nft_transfer", "data": [ {"old_owner_id": "user1.near", "new_owner_id": "user2.near", "token_ids": ["meme"], "memo": "have fun!"} ] } ``` Authorized id: ```js EVENT_JSON:{ "standard": "nep171", "version": "1.1.0", "event": "nft_burn", "data": [ {"owner_id": "owner.near", "token_ids": ["goodbye", "aurevoir"], "authorized_id": "thirdparty.near"} ] } ``` Contract metadata update: ```js EVENT_JSON:{ "standard": "nep171", "version": "1.1.0", "event": "contract_metadata_update", "data": [] } ``` ## Events for Other NFT Methods Note that the example events above cover two different kinds of events: 1. Events that do not have a dedicated trigger function in the NFT Standard (`nft_mint`, `nft_burn`, `contract_metadata_update`) 2. An event that has a relevant trigger function [NFT Core Standard](https://github.com/near/NEPs/blob/master/neps/nep-0171.md#nft-interface) (`nft_transfer`) This event standard also applies beyond the events highlighted here, where future events follow the same convention of as the second type. For instance, if an NFT contract uses the [approval management standard](https://github.com/near/NEPs/blob/master/neps/nep-0178.md), it may emit an event for `nft_approve` if that's deemed as important by the developer community. Please feel free to open pull requests for extending the events standard detailed here as needs arise. ================================================ FILE: neps/nep-0264.md ================================================ --- NEP: 264 Title: Utilization of unspent gas for promise function calls Authors: Austin Abell Status: Final DiscussionsTo: https://github.com/near/NEPs/pull/264 Type: Protocol Version: 1.0.0 Created: 2021-09-30 LastUpdated: 2022-05-26 --- # Summary This proposal is to introduce a new host function on the NEAR runtime that allows for scheduling cross-contract function calls using a percentage/weight of the remaining gas in addition to the statically defined amount. This will enable async promise execution to use the remaining gas more efficiently by utilizing unspent gas from the current transaction. # Motivation We are proposing this to be able to utilize gas more efficiently but also to improve the devX of cross-contract calls. Currently, developers must guess how much gas will remain after the current transaction finishes and if this value is too little, the transaction will fail, and if it is too large, gas will be wasted. Therefore, these cross-contract calls need a reasonable default of splitting unused gas efficiently for basic cases without sacrificing the ability to configure the gas amount attached at a granular level. Currently, gas is allocated very inefficiently, requiring more prepaid gas or failed transactions when the allocations are imprecise. # Guide-level explanation This host function is similar to [`promise_batch_action_function_call`](https://github.com/near/nearcore/blob/7d15bbc996282c8ae8f15b8f49d110fc901b84d8/runtime/near-vm-logic/src/logic.rs#L1526), except with an additional parameter that lets you specify how much of the excess gas should be attached to the function call. This parameter is a weight value that determines how much of the excess gas is attached to each function. So, for example, if there is 40 gas leftover and three function calls that select weights of 1, 5, and 2, the runtime will add 5, 25, and 10 gas to each function call. A developer can specify whether they want to attach a fixed amount of gas, a weight of remaining gas, or both. If at least one function call uses a weight of remaining gas, then all excess gas will be attached to future calls. This proposal allows developers the ability to utilize prepaid gas more efficiently than currently possible. # Reference-level explanation This host function would need to be implemented in `nearcore` and parallel [`promise_batch_action_function_call`](https://github.com/near/nearcore/blob/7d15bbc996282c8ae8f15b8f49d110fc901b84d8/runtime/near-vm-logic/src/logic.rs#L1526). Most details of these functions will be consistent, except that there will be additional bookkeeping for keeping track of which functions specified a weight for unused gas. This will not affect or replace any existing host functions, but this will likely require a slightly higher gas cost than the original `promise_batch_action_function_call` host function due to this additional overhead. This host function definition would look like this (as a Rust consumer): ```rust /// Appends `FunctionCall` action to the batch of actions for the given promise pointed by /// `promise_idx`. This function allows not specifying a specific gas value and allowing the /// runtime to assign remaining gas based on a weight. /// /// # Gas /// /// Gas can be specified using a static amount, a weight of remaining prepaid gas, or a mixture /// of both. To omit a static gas amount, `0` can be passed for the `gas` parameter. /// To omit assigning remaining gas, `0` can be passed as the `gas_weight` parameter. /// /// The gas weight parameter works as the following: /// /// All unused prepaid gas from the current function call is split among all function calls /// which supply this gas weight. The amount attached to each respective call depends on the /// value of the weight. /// /// For example, if 40 gas is leftover from the current method call and three functions specify /// the weights 1, 5, 2 then 5, 25, 10 gas will be added to each function call respectively, /// using up all remaining available gas. Any remaining gas will be allocated to the last /// function call. /// /// # Errors /// /// <...Ommitted previous errors as they do not change> /// - If `0` is passed for both `gas` and `gas_weight` parameters pub fn promise_batch_action_function_call_weight( promise_index: u64, method_name_len: u64, method_name_ptr: u64, arguments_len: u64, arguments_ptr: u64, amount_ptr: u64, gas: u64, gas_weight: u64, ); ``` The only difference from the existing API is `gas_weight` added as another parameter, as an unsigned 64-bit integer. As for calculations, the remaining gas at the end of the transaction can be floor divided by the sum of all the weights tracked. Then, after getting this value, just attach that value multiplied by the weight gas to each function call action. For example, if there are three weights, `a`, `b`, `c`: ```rust weight_sum = a + b + c a_gas += remaining_gas * a / weight_sum b_gas += remaining_gas * b / weight_sum c_gas += remaining_gas * c / weight_sum ``` Any remaining gas that is not allocated to any of these function calls will be attached to the last function call scheduled. ### SDK changes This protocol change will allow cross-contract calls to provide a fixed amount of gas and/or adjust the weight of unused gas to use. If neither is provided, it will default to using a weight of 1 for each and no static amount of gas. If no function modifies this weight, the runtime will split the unused gas evenly among all function calls. Currently, the API for a cross-contract call looks like: ```rust let contract_account_id: AccountId = todo!(); ext::some_method(/* parameters */, contract_account_id, 0 /* deposit amount */, 5_000_000_000_000 /* static amount of gas to attach */) ``` When the intended API should not require thinking about how much gas to attach by default, the API will look something like what's shown in [this PR](https://github.com/near/near-sdk-rs/pull/742), which can look like the following: ```rust cross_contract::ext(contract_account_id) // Optional config .with_attached_deposit(1 /* default deposit of 0 */) .with_static_gas(Gas(5_000_000_000_000) /* default of 0 */) .with_unused_gas_weight(2 /* default 1 */) // Then call any method to schedule the function call .some_method(/* parameters */) ``` At a basic level, a developer has only to include the parameters for the function call and specify the account id of the contract being called. Currently, only the amount can be optional because there is no way to set a reasonable default for the amount of gas to use for each function call. # Drawbacks - Complexity in refactoring to handle assigning remaining gas at the end of a transaction - Complexity in extra calculations for assigning gas will make the host function slightly more expensive than the base one. It is not easy to create an API on the SDK level that can decide which host function to call if dynamic gas assigning is needed or not. If both are used, the size of the wasm binary is trivially larger by including both host functions - Adds another host function to the runtime, which can probably never be removed - Can be confusing to have both static gas and dynamic unused gas and convey what is happening internally to a developer - If we start utilizing all prepaid gas, this will likely lead to a higher percentage of prepaid gas usage. This could be an unexpected pattern for users and require them to think about how much gas they are attaching to make sure they only attach what they are willing to spend - Since currently, we are refunding a lot of unused gas, this could be a hidden negative side effect - Keep in mind that it will also be positive because transactions will generally succeed more often due to gas more efficiently # Rationale and alternatives Alternative 1 (fraction parameters): The primary alternative is using a numerator and denominator to represent a fraction instead of a weight. This alternative would be equivalent to the one listed above except for two u64 additional parameters instead of just the one for weight. I'll list the tradeoff as pros and cons: Pros: - Can under-utilize the gas for the current transaction to limit gas allowed for certain functions - This could take responsibility away from DApp users because they would not have to worry less about attaching too much prepaid gas - Thinking in terms of fractions may be more intuitive for some developers - Might future proof better if we ever need this ability in the future, want to minimize the number of host functions created at all costs Cons: - More complicated logic/edge cases to handle to make sure the percentages don't sum to greater than 100% (or adjusting if they do) - Precision loss from dividing integers may lead to unexpected results - To get closer to expected, we could use floats for the division, but this gets messy - API for specifying a fraction would be messy (need to specify two values rather than just optionally one) - There isn't a good default for this. Unless there is a special value that indicates a pool of function calls that will split the remaining equally, but this defeats the purpose of this alternative completely - Slightly larger API (only one u64, can probably safely ignore this point) Alternative 2 (handle within contract/SDK): The other alternative is to handle all of this logic on the contract side, as seen by [this PR](https://github.com/near/near-sdk-rs/pull/523). This is much less feasible/accurate because there is only so much information available within the runtime, and gas costs and internal functionality may not always be the same. As discussed on [the respective issue](https://github.com/near/near-sdk-rs/issues/526), this alternative seems to be very infeasible. Pros: - No protocol change is needed - Can still have improved API as with protocol change Cons: - Additional bloat to every contract, even ones that don't use the pattern (~5kb in PoC, even with simple estimation logic) - Still inaccurate gas estimations, because at the point of calculation, we cannot know how much gas will be used for assigning gas values as well as gas consumed after the transaction ends - This leads to either underutilizing or having transactions fail when using too much gas if trying to estimate how much gas will be left - Prone to breaking existing contracts on protocol changes that affect gas usage or logic of runtime # Unresolved questions What needs to be addressed before this gets merged: ~~- How much refactoring exactly is needed to handle this pattern?~~ ~~- Can we keep a queue of receipt and action indices with their respective weights and update their gas values after the current method is executed? Is there a cleaner way to handle this while keeping order?~~ ~~- Do we want to attach the gas lost due to precision on division to any function?~~ - The remaining gas is now attached to the last function call What would be addressed in future independently of the solution: - How many users would expect the ability to refund part of the gas after the initial transaction? (is this worth considering the API difference of using fractions rather than weights) - Will weights be an intuitive experience for developers? # Future possibilities The future change that would extend from this being implemented is a much cleaner API for the SDKs. As mentioned previously in the alternatives section, the API changes from [the changes tested on the SDK](https://github.com/near/near-sdk-rs/pull/523) will remain, but without the overhead from implementing this on the contract level. Thus, not only can this be implemented in Rust, but it will also allow a consistent API for existing and future SDK languages to build on. The primary benefit for SDKs is that it removes the need to specify gas when making cross-contract calls explicitly. Currently, there is no easy way of knowing how many function calls will be made to split prepaid gas without a decent amount of overhead. Even if the developer does this, it's impossible to know how much gas will remain after the transaction from inside the contract. Having this host function available will simplify the DevX for contract developers and make the contracts use gas more efficiently. ================================================ FILE: neps/nep-0297.md ================================================ --- NEP: 297 Title: Events Author: Olga Telezhnaya Status: Final DiscussionsTo: https://github.com/near/NEPs/issues/297 Type: Standards Track Category: Contract Created: 03-Mar-2022 --- ## Summary Events format is a standard interface for tracking contract activity. This document is a meta-part of other standards, such as [NEP-141](https://github.com/near/NEPs/issues/141) or [NEP-171](https://github.com/near/NEPs/discussions/171). ## Motivation Apps usually perform many similar actions. Each app may have its own way of performing these actions, introducing inconsistency in capturing these events. NEAR and third-party applications need to track these and similar events consistently. If not, tracking state across many apps becomes infeasible. Events address this issue, providing other applications with the needed standardized data. Initial discussion is [here](https://github.com/near/NEPs/issues/254). ## Rationale and alternatives - Why is this design the best in the space of possible designs? - What other designs have been considered and what is the rationale for not choosing them? - What is the impact of not doing this? ## Specification Many apps use different interfaces that represent the same action. This interface standardizes that process by introducing event logs. Events use the standard logs capability of NEAR. Events are log entries that start with the `EVENT_JSON:` prefix followed by a single valid JSON string. JSON string may have any number of space characters in the beginning, the middle, or the end of the string. It's guaranteed that space characters do not break its parsing. All the examples below are pretty-formatted for better readability. JSON string should have the following interface: ```ts // Interface to capture data about an event // Arguments // * `standard`: name of standard, e.g. nep171 // * `version`: e.g. 1.0.0 // * `event`: type of the event, e.g. nft_mint // * `data`: associate event data. Strictly typed for each set {standard, version, event} inside corresponding NEP interface EventLogData { standard: string; version: string; event: string; data?: unknown; } ``` Thus, to emit an event, you only need to log a string following the rules above. Here is a bare-bones example using Rust SDK `near_sdk::log!` macro (security note: prefer using `serde_json` or alternatives to serialize the JSON string to avoid potential injections and corrupted events): ```rust use near_sdk::log; // ... log!( r#"EVENT_JSON:{"standard": "nepXXX", "version": "1.0.0", "event": "YYY", "data": {"token_id": "{}"}}"#, token_id ); // ... ``` #### Valid event logs ```js EVENT_JSON:{ "standard": "nepXXX", "version": "1.0.0", "event": "xyz_is_triggered" } ``` ```js EVENT_JSON:{ "standard": "nepXXX", "version": "1.0.0", "event": "xyz_is_triggered", "data": { "triggered_by": "foundation.near" } } ``` #### Invalid event logs - Two events in a single log entry (instead, call `log` for each individual event) ```js EVENT_JSON:{ "standard": "nepXXX", "version": "1.0.0", "event": "abc_is_triggered" } EVENT_JSON:{ "standard": "nepXXX", "version": "1.0.0", "event": "xyz_is_triggered" } ``` - Invalid JSON data ```js EVENT_JSON:invalid json ``` - Missing required fields `standard`, `version` or `event` ```js EVENT_JSON:{ "standard": "nepXXX", "event": "xyz_is_triggered", "data": { "triggered_by": "foundation.near" } } ``` ## Reference Implementation [Fungible Token Events Implementation](https://github.com/near/near-sdk-rs/blob/master/near-contract-standards/src/fungible_token/events.rs) [Non-Fungible Token Events Implementation](https://github.com/near/near-sdk-rs/blob/master/near-contract-standards/src/non_fungible_token/events.rs) ## Drawbacks There is a known limitation of 16kb strings when capturing logs. This impacts the amount of events that can be processed. ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0300.md ================================================ --- NEP: 300 Title: Fungible Token Events Author: Olga Telezhnaya Status: Final DiscussionsTo: https://github.com/near/NEPs/pull/300, https://github.com/near/NEPs/issues/271 Type: Standards Track Category: Contract Created: 15-Dec-2021 --- # Fungible Token Event Version `1.0.0` ## Summary Standard interfaces for FT contract actions. Extension of [NEP-297](https://github.com/near/NEPs/blob/master/neps/nep-0297.md) ## Motivation NEAR and third-party applications need to track `mint`, `transfer`, `burn` events for all FT-driven apps consistently. This extension addresses that. Keep in mind that applications, including NEAR Wallet, could require implementing additional methods, such as [`ft_metadata`](https://github.com/near/NEPs/blob/master/neps/nep-0148.md), to display the FTs correctly. ## Interface Fungible Token Events MUST have `standard` set to `"nep141"`, standard version set to `"1.0.0"`, `event` value is one of `ft_mint`, `ft_burn`, `ft_transfer`, and `data` must be of one of the following relevant types: `FtMintLog[] | FtTransferLog[] | FtBurnLog[]`: ```ts interface FtEventLogData { standard: "nep141", version: "1.0.0", event: "ft_mint" | "ft_burn" | "ft_transfer", data: FtMintLog[] | FtTransferLog[] | FtBurnLog[], } ``` ```ts // An event log to capture tokens minting // Arguments // * `owner_id`: "account.near" // * `amount`: the number of tokens to mint, wrapped in quotes and treated // like a string, although the number will be stored as an unsigned integer // with 128 bits. // * `memo`: optional message interface FtMintLog { owner_id: string, amount: string, memo?: string } // An event log to capture tokens burning // Arguments // * `owner_id`: owner of tokens to burn // * `amount`: the number of tokens to burn, wrapped in quotes and treated // like a string, although the number will be stored as an unsigned integer // with 128 bits. // * `memo`: optional message interface FtBurnLog { owner_id: string, amount: string, memo?: string } // An event log to capture tokens transfer // Arguments // * `old_owner_id`: "owner.near" // * `new_owner_id`: "receiver.near" // * `amount`: the number of tokens to transfer, wrapped in quotes and treated // like a string, although the number will be stored as an unsigned integer // with 128 bits. // * `memo`: optional message interface FtTransferLog { old_owner_id: string, new_owner_id: string, amount: string, memo?: string } ``` ## Examples Batch mint: ```js EVENT_JSON:{ "standard": "nep141", "version": "1.0.0", "event": "ft_mint", "data": [ {"owner_id": "foundation.near", "amount": "500"} ] } ``` Batch transfer: ```js EVENT_JSON:{ "standard": "nep141", "version": "1.0.0", "event": "ft_transfer", "data": [ {"old_owner_id": "from.near", "new_owner_id": "to.near", "amount": "42", "memo": "hi hello bonjour"}, {"old_owner_id": "user1.near", "new_owner_id": "user2.near", "amount": "7500"} ] } ``` Batch burn: ```js EVENT_JSON:{ "standard": "nep141", "version": "1.0.0", "event": "ft_burn", "data": [ {"owner_id": "foundation.near", "amount": "100"}, ] } ``` ## Further methods Note that the example events covered above cover two different kinds of events: 1. Events that are not specified in the FT Standard (`ft_mint`, `ft_burn`) 2. An event that is covered in the [FT Core Standard](https://github.com/near/NEPs/blob/master/neps/nep-0141.md). (`ft_transfer`) Please feel free to open pull requests for extending the events standard detailed here as needs arise. ================================================ FILE: neps/nep-0330.md ================================================ --- NEP: 330 Title: Source Metadata Author: Ben Kurrek , Osman Abdelnasir , Andrey Gruzdev <@canvi>, Alexey Zenin <@alexthebuildr> Status: Final DiscussionsTo: https://github.com/near/NEPs/discussions/329 Type: Standards Track Category: Contract Version: 1.2.0 Created: 27-Feb-2022 Updated: 19-Feb-2023 --- ## Summary The contract source metadata represents a standardized interface designed to facilitate the auditing and inspection of source code associated with a deployed smart contract. Adoption of this standard remains discretionary; however, it is strongly advocated for developers who maintain an open-source approach to their contracts. This initiative promotes greater accountability and transparency within the ecosystem, encouraging best practices in contract development and deployment. ## Motivation The incorporation of metadata facilitates the discovery and validation of deployed source code, thereby significantly reducing the requisite level of trust during code integration or interaction processes. The absence of an accepted protocol for identifying the source code or author contact details of a deployed smart contract presents a challenge. Establishing a standardized framework for accessing the source code of any given smart contract would foster a culture of transparency and collaborative engagement. Moreover, the current landscape does not offer a straightforward mechanism to verify the authenticity of a smart contract's deployed source code against its deployed version. To address this issue, it is imperative that metadata includes specific details that enable contract verification through reproducible builds. Furthermore, it is desirable for users and dApps to possess the capability to interpret this metadata, thereby identifying executable methods and generating UIs that facilitate such functionalities. This also extends to acquiring comprehensive insights into potential future modifications by the contract or its developers, enhancing overall system transparency and user trust. The initial discussion can be found [here](https://github.com/near/NEPs/discussions/329). ## Rationale and alternatives There is a lot of information that can be held about a contract. Ultimately, we wanted to limit it to the least amount fields while still maintaining our goal. This decision was made to not bloat the contracts with unnecessary storage and also to keep the standard simple and understandable. ## Specification Successful implementations of this standard will introduce a new (`ContractSourceMetadata`) struct that will hold all the necessary information to be queried for. This struct will be kept on the contract level. The metadata will include optional fields: - `version`: a string that references the specific commit ID or a tag of the code currently deployed on-chain. Examples: `"v0.8.1"`, `"a80bc29"`. - `link`: an URL to the currently deployed code. It must include version or a tag if using a GitHub or a GitLab link. Examples: "https://github.com/near/near-cli-rs/releases/tag/v0.8.1", "https://github.com/near/cargo-near-new-project-template/tree/9c16aaff3c0fe5bda4d8ffb418c4bb2b535eb420" or an IPFS CID. - `standards`: a list of objects (see type definition below) that enumerates the NEPs supported by the contract. If this extension is supported, it is advised to also include NEP-330 version 1.1.0 in the list (`{standard: "nep330", version: "1.1.0"}`). - `build_info`: a build details object (see type definition below) that contains all the necessary information about how the contract was built, making it possible for others to reproduce the same WASM of this contract. ```ts type ContractSourceMetadata = { version: string|null, // optional, commit hash being used for the currently deployed WASM. If the contract is not open-sourced, this could also be a numbering system for internal organization / tracking such as "1.0.0" and "2.1.0". link: string|null, // optional, link to open source code such as a Github repository or a CID to somewhere on IPFS, e.g., "https://github.com/near/cargo-near-new-project-template/tree/9c16aaff3c0fe5bda4d8ffb418c4bb2b535eb420" standards: Standard[]|null, // optional, standards and extensions implemented in the currently deployed WASM, e.g., [{standard: "nep330", version: "1.1.0"},{standard: "nep141", version: "1.0.0"}]. build_info: BuildInfo|null, // optional, details that are required for contract WASM reproducibility. } type Standard { standard: string, // standard name, e.g., "nep141" version: string, // semantic version number of the Standard, e.g., "1.0.0" } type BuildInfo { build_environment: string, // reference to a reproducible build environment docker image, e.g., "docker.io/sourcescan/cargo-near@sha256:bf488476d9c4e49e36862bbdef2c595f88d34a295fd551cc65dc291553849471" or something else pointing to the build environment. source_code_snapshot: string, // reference to the source code snapshot that was used to build the contract, e.g., "git+https://github.com/near/cargo-near-new-project-template.git#9c16aaff3c0fe5bda4d8ffb418c4bb2b535eb420" or "ipfs://". contract_path: string, // relative path to contract crate within the source code, e.g., "contracts/contract-one". Often, it is the root of the repository, so can be set to empty string. build_command: string[], // the exact command that was used to build the contract, with all the flags, e.g., ["cargo", "near", "build", "--no-abi"]. output_wasm_path: string|null, // absolute path inside build environment, where resulting `*.wasm` file was put during build } ``` In order to view this information, contracts must include a getter which will return the struct. ```ts function contract_source_metadata(): ContractSourceMetadata {} ``` ### Ensuring WASM Reproducibility #### Build Environment Docker Image When using a Docker image as a reference, it's important to specify the digest of the image to ensure reproducibility, since a tag could be reassigned to a different image. ### Paths Inside Docker Image During the build, paths from the source of the build as well as the location of the cargo registry could be saved into WASM, which affects reproducibility. Therefore, we need to ensure that everyone uses the same paths inside the Docker image. We propose using the following paths: - `/home/near/code` - Mounting volume from the host system containing the source code. - `/home/near/.cargo` - Cargo registry. #### Cargo.lock It is important to have `Cargo.lock` inside the source code snapshot to ensure reproducibility. Example: https://github.com/near/core-contracts. ## Reference Implementation As an example, consider a contract located at the root path of the repository, which was deployed using the `cargo near deploy --no-abi` and environment docker image `sourcescan/cargo-near@sha256:bf488476d9c4e49e36862bbdef2c595f88d34a295fd551cc65dc291553849471`. Its latest commit hash is `9c16aaff3c0fe5bda4d8ffb418c4bb2b535eb420`, and its open-source code can be found at `https://github.com/near/cargo-near-new-project-template`. This contract would then include a struct with the following fields: ```ts type ContractSourceMetadata = { version: "1.0.0", link: "https://github.com/near/cargo-near-new-project-template/tree/9c16aaff3c0fe5bda4d8ffb418c4bb2b535eb420", standards: [ { standard: "nep330", version: "1.1.0" } ], build_info: { build_environment: "docker.io/sourcescan/cargo-near@sha256:bf488476d9c4e49e36862bbdef2c595f88d34a295fd551cc65dc291553849471", source_code_snapshot: "git+https://github.com/near/cargo-near-new-project-template?rev=9c16aaff3c0fe5bda4d8ffb418c4bb2b535eb420", contract_path: "", build_command: ["cargo", "near", "deploy", "--no-abi"], output_wasm_path: "/home/near/code/target/near/cargo_near_new_project_name.wasm" } } ``` Calling the view function `contract_source_metadata`, the contract would return: ```json { "version": "1.0.0" "link": "https://github.com/near/cargo-near-new-project-template/tree/9c16aaff3c0fe5bda4d8ffb418c4bb2b535eb420", "standards": [ { "standard": "nep330", "version": "1.3.0" } ], "build_info": { "build_environment": "docker.io/sourcescan/cargo-near@sha256:bf488476d9c4e49e36862bbdef2c595f88d34a295fd551cc65dc291553849471", "source_code_snapshot": "git+https://github.com/near/cargo-near-new-project-template?rev=9c16aaff3c0fe5bda4d8ffb418c4bb2b535eb420", "contract_path": "", "build_command": ["cargo", "near", "deploy", "--no-abi"], "output_wasm_path": "/home/near/code/target/near/cargo_near_new_project_name.wasm" } } ``` This could be used by SourceScan to reproduce the same WASM using the build details and to verify the on-chain WASM code with the reproduced one. An example implementation can be seen below. ```rust /// Simple Implementation #[near_bindgen] pub struct Contract { pub contract_metadata: ContractSourceMetadata } /// NEP supported by the contract. pub struct Standard { pub standard: String, pub version: String } /// BuildInfo structure pub struct BuildInfo { pub build_environment: String, pub source_code_snapshot: String, pub contract_path: String, pub build_command: Vec, pub output_wasm_path: Option, } /// Contract metadata structure pub struct ContractSourceMetadata { pub version: Option, pub link: Option, pub standards: Option>, pub build_info: Option, } /// Minimum Viable Interface pub trait ContractSourceMetadataTrait { fn contract_source_metadata(&self) -> ContractSourceMetadata; } /// Implementation of the view function #[near_bindgen] impl ContractSourceMetadataTrait for Contract { fn contract_source_metadata(&self) -> ContractSourceMetadata { self.contract_source_metadata.get().unwrap() } } ``` ## Future possibilities - By having a standard outlining metadata for an arbitrary contract, any information that pertains on a contract level can be added based on the requests of the developer community. ## Decision Context ### 1.0.0 - Initial Version The initial version of NEP-330 was approved by @jlogelin on Mar 29, 2022. ### 1.1.0 - Contract Metadata Extension The extension NEP-351 that added Contract Metadata to this NEP-330 was approved by Contract Standards Working Group members on January 17, 2023 ([meeting recording](https://youtu.be/pBLN9UyE6AA)). #### Benefits - Unlocks NEP extensions that otherwise would be hard to integrate into the tooling as it would be guess-based (e.g. see "interface detection" concerns in the Non-transferrable NFT NEP) - Standardization enables composability as it makes it easier to interact with contracts when you can programmatically check compatibility - This NEP extension introduces an optional field, so there is no breaking change to the original NEP #### Concerns | # | Concern | Resolution | Status | | - | - | - | - | | 1 | Integer field as a standard reference is limiting as third-party projects may want to introduce their own standards without pushing it through the NEP process | Author accepted the proposed string-value standard reference (e.g. “nep123” instead of just 123, and allow “xyz001” as previously it was not possible to express it) | Resolved | | 2 | NEP-330 and NEP-351 should be included in the list of the supported NEPs | There seems to be a general agreement that it is a good default, so NEP was updated | Resolved | | 3 | JSON Event could be beneficial, so tooling can react to the changes in the supported standards | It is outside the scope of this NEP. Also, list of supported standards only changes with contract re-deployment, so tooling can track DEPLOY_CODE events and check the list of supported standards when new code is deployed | Won’t fix | ### 1.2.0 - Build Details Extension The NEP extension adds build details to the contract metadata, containing necessary information about how the contract was built. This makes it possible for others to reproduce the same WASM of this contract. The idea first appeared in the [cargo-near SourceScan integration thread](https://github.com/near/cargo-near/issues/131). ### 1.3.0 - Add `output_wasm_path` field to Build Details Extension 1. This field is required in order to be able to verify build in a way, that is agnostic of specific language/toolchain the contract is implemented with. 2. Field's type is `Option` (rust semantics) which allows backward compatibility with 1.2.0 contract metadata when parsing, when the field is absent. 3. If the field is present, contract metadata is considered to be at least 1.3.0 version. 4. Valid value for the field should be a valid unix path to a `wasm` file, being a subpath of `/home/near/code` (mentioned in [Paths Inside Docker Image](#paths-inside-docker-image)). #### Benefits - This NEP extension gives developers the capability to save all the required build details, making it possible to reproduce the same WASM code in the future. This ensures greater consistency in contracts and the ability to verify source code. With the assistance of tools like SourceScan and cargo-near, the development process on NEAR becomes significantly easier ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0364.md ================================================ --- NEP: 364 Title: Efficient signature verification and hashing precompile functions Author: Blas Rodriguez Irizar Status: Final DiscussionsTo: https://github.com/nearprotocol/neps/pull/364 Type: Runtime Spec Category: Contract Created: 15-Jun-2022 --- ## Summary This NEP introduces the request of adding into the NEAR runtime a pre-compiled function used to verify signatures that can help IBC compatible light clients run on-chain. ## Motivation Signature verification and hashing are ubiquitous operations in light clients, especially in PoS consensus mechanisms. Based on Polkadot's consensus mechanism there will be a need for verification of ~200 signatures every minute (Polkadot’s authority set is ~300 signers and it may be increased in the future: https://polkadot.polkassembly.io/referendum/16). Therefore, a mechanism to perform these operations cost-effectively in terms of gas and speed would be highly beneficial to have. Currently, NEAR does not have any native signature verification toolbox. This implies that a light client operating inside NEAR will have to import a library compiled to WASM as mentioned in [Zulip](https://near.zulipchat.com/#narrow/stream/295302-general/topic/light_client). Polkadot uses [three different cryptographic schemes](https://wiki.polkadot.network/docs/learn-keys) for its keys/accounts, which also translates into different signature types. However, for this NEP the focus is on: - The vanilla ed25519 implementation uses Schnorr signatures. ## Rationale and alternatives Add a signature verification signatures function into the runtime as host functions. - ED25519 signature verification function using `ed25519_dalek` crates into NEAR runtime as pre-compiled functions. Benchmarks were run using a signature verifier smart contract on-chain importing the aforementioned functions from widespread used crypto Rust crates. The biggest pitfall of these functions running wasm code instead of native is performance and gas cost. Our [benchmarks](https://github.com/blasrodri/near-test) show the following results: ```log near call sitoula-test.testnet verify_ed25519 '{"signature_p1": [145,193,203,18,114,227,14,117,33,213,121,66,130,14,25,4,36,120,46,142,226,215,7,66,122,112,97,30,249,135,61,165], "signature_p2": [221,249,252,23,105,40,56,70,31,152,236,141,154,122,207,20,75,118,79,90,168,6,221,122,213,29,126,196,216,104,191,6], "msg": [107,97,106,100,108,102,107,106,97,108,107,102,106,97,107,108,102,106,100,107,108,97,100,106,102,107,108,106,97,100,115,107], "iterations": 10}' --accountId sitoula-test.testnet --gas 300000000000000 # transaction id DZMuFHisupKW42w3giWxTRw5nhBviPu4YZLgKZ6cK4Uq ``` With `iterations = 130` **all these calls return ExecutionError**: `'Exceeded the maximum amount of gas allowed to burn per contract.'` With iterations = 50 these are the results: ```text ed25519: tx id 6DcJYfkp9fGxDGtQLZ2m6PEDBwKHXpk7Lf5VgDYLi9vB (299 Tgas) ``` - Performance in wall clock time when you compile the signature validation library directly from rust to native. Here are the results on an AMD Ryzen 9 5900X 12-Core Processor machine: ```text # 10k signature verifications ed25519: took 387ms ``` - Performance in wall clock time when you compile the library into wasm first and then use the single-pass compiler in Wasmer 1 to then compile to native. ```text ed25519: took 9926ms ``` As an extra data point, when passing `--enable-simd` instead of `--singlepass` ```text ed25519: took 3085ms ``` Steps to reproduce: commit: `31cf97fb2e155d238308f062c4b92bae716ac19f` in `https://github.com/blasrodri/near-test` ```sh # wasi singlepass cargo wasi build --bin benches --release wasmer compile --singlepass ./target/wasm32-wasi/release/benches.wasi.wasm -o benches_singlepass wasmer run ./benches_singlepass ``` ```sh # rust native cargo run --bin benches --release ``` Overall: the difference between the two versions (native vs wasi + singlepass is) ```text ed25519: 25.64x slower ``` ### What is the impact of not doing this? Costs of running IBC-compatible trustless bridges would be very high. Plus, the fact that signature verification is such an expensive operation will force the contract to split the process of batch verification of signatures into multiple transactions. ### Why is this design the best in the space of possible designs? Adding existing proved and vetted crypto crates into the runtime is a safe workaround. It will boost performance between 20-25x according to our benchmarks. This will both reduce operating costs significantly and will also enable the contract to verify all the signatures in one transaction, which will simplify the contract design. ### What other designs have been considered and what is the rationale for not choosing them? One possible alternative would be to improve the runtime implementation so that it can compile WASM code to a sufficiently fast machine code. Even when it may not be as fast as LLVM native produced code it could still be acceptable for these types of use cases (CPU intensive functions) and will certainly avoid the need of adding host functions. The effort of adding such a feature will be significantly higher than adding these host functions one by one. But on the other side, it will decrease the need of including more host functions in the future. Another alternative is to deal with the high cost of computing/verifying these signatures in some other manner. Decreasing the overall cost of gas and increasing the limits of gas available to attach to the contract could be a possibility. Introducing such modification for some contracts, and not for some others can be rather arbitrary and not straightforward in the implementation, but an alternative nevertheless. ## Specification This NEP aims to introduce the following host function: ```rust extern "C"{ /// Verify an ED25519 signature given a message and a public key. /// Ed25519 is a public-key signature system with several attractive features /// /// Proof of Stake Validator sets can contain different signature schemes. /// Ed25519 is one of the most used ones across blockchains, and hence it's importance to be added. /// For further reference, visit: https://ed25519.cr.yp.to /// # Returns /// - 1 if the signature was properly verified /// - 0 if the signature failed to be verified /// /// # Cost /// /// Each input can either be in memory or in a register. Set the length of the input to `u64::MAX` /// to declare that the input is a register number and not a pointer. /// Each input has a gas cost input_cost(num_bytes) that depends on whether it is from memory /// or from a register. It is either read_memory_base + num_bytes * read_memory_byte in the /// former case or read_register_base + num_bytes * read_register_byte in the latter. This function /// is labeled as `input_cost` below. /// /// `input_cost(num_bytes_signature) + input_cost(num_bytes_message) + input_cost(num_bytes_public_key) /// ed25519_verify_base + ed25519_verify_byte * num_bytes_message` /// /// # Errors /// /// If the signature size is not equal to 64 bytes, or public key length is not equal to 32 bytes, contract execution is terminated with an error. fn ed25519_verify( sig_len: u64, sig_ptr: u64, msg_len: u64, msg_ptr: u64, pub_key_len: u64, pub_key_ptr: u64, ) -> u64; ``` And a `rust-sdk` possible implementation could look like this: ```rs pub fn ed25519_verify(sig: &ed25519::Signature, msg: &[u8], pub_key: &ed25519::Public) -> bool; ``` Once this NEP is approved and integrated, these functions will be available in the `near_sdk` crate in the `env` module. [This blog post](https://hdevalence.ca/blog/2020-10-04-its-25519am) describes the various ways in which the existing Ed25519 implementations differ in practice. The behavior that this proposal uses, which is shared by Go `crypto/ed25519`, Rust `ed25519-dalek` (using `verify` function with `legacy_compatibility` feature turned off) and several others, makes the following decisions: - The encoding of the values `R` and `s` must be canonical, while the encoding of `A` doesn't need to. - The verification equation is `R = [s]B − [k]A`. - No additional checks are performed. In particular, the points outside of the order-l subgroup are accepted, as are the points in the torsion subgroup. Note that this implementation only refers to the `verify` function in the crate `ed25519-dalek` and **not** `verify_strict` or `verify_batch`. ## Security Implications (Optional) We have chosen this crate because it is already integrated into `nearcore`. ## Unresolved Issues (Optional) - What parts of the design do you expect to resolve through the NEP process before this gets merged? Both the function signatures and crates are up for discussion. ## Future possibilities I currently do not envision any extension in this regard. ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0366.md ================================================ --- NEP: 366 Title: Meta Transactions Author: Illia Polosukhin , Egor Uleyskiy (egor.ulieiskii@gmail.com), Alexander Fadeev (fadeevab.com@gmail.com) Status: Final DiscussionsTo: https://github.com/nearprotocol/neps/pull/366 Type: Protocol Track Category: Runtime Version: 1.1.0 Created: 19-Oct-2022 LastUpdated: 03-Aug-2023 --- ## Summary In-protocol meta transactions allow third-party accounts to initiate and pay transaction fees on behalf of the account. ## Motivation NEAR has been designed with simplicity of onboarding in mind. One of the large hurdles right now is that after creating an implicit or even named account the user does not have NEAR to pay gas fees to interact with apps. For example, apps that pay user for doing work (like NEARCrowd or Sweatcoin) or free-to-play games. [Aurora Plus](https://aurora.plus) has shown viability of the relayers that can offer some number of free transactions and a subscription model. Shifting the complexity of dealing with fees to the infrastructure from the user space. ## Rationale and alternatives The proposed design here provides the easiest way for users and developers to onboard and to pay for user transactions. An alternative is to have a proxy contract deployed on the user account. This design has severe limitations as it requires the user to deploy such contract and incur additional costs for storage. ## Specification - **User** (Sender) is the one who is going to send the `DelegateAction` to Receiver via Relayer. - **Relayer** is the one who publishes the `DelegateAction` to the protocol. - **User** and Relayer doesn't trust each other. The main flow of the meta transaction will be as follows: - User specifies `sender_id` (the user's account id), `receiver_id` (the receiver's account id) and other information (see `DelegateAction` format). - User signs `DelegateAction` specifying the set of actions that they need to be executed. - User forms `SignedDelegateAction` with the `DelegateAction` and the signature. - User forms `DelegateActionMessage` with the `SignedDelegateAction`. - User sends `DelegateActionMessage` data to the relayer. - Relayer verifies actions specified in `DelegateAction`: the total cost and whether the user included the reward for the relayer. - Relayer forms a `Transaction` with `receiver_id` equals to `delegate_action.sender_id` and `actions: [SignedDelegateAction { ... }]`. Signs it with its key. Note that such transactions can contain other actions toward user's account (for example calling a function). - This transaction is processed normally. A `Receipt` is created with a copy of the actions in the transaction. - When processing a `SignedDelegateAction`, a number of checks are done (see below), mainly a check to ensure that the `signature` matches the user account's key. - When a `Receipt` with a valid `SignedDelegateAction` in actions arrives at the user's account, it gets executed. Execution means creation of a new Receipt with `receiver_id: AccountId` and `actions: Action` matching `receiver_id` and `actions` in the `DelegateAction`. - The new `Receipt` looks like a normal receipt that could have originated from the user's account, with `predeccessor_id` equal to tbe user's account, `signer_id` equal to the relayer's account, `signer_public_key` equal to the relayer's public key. ## Diagram ![Delegate Action Diagram](assets/nep-0366/NEP-DelegateAction.png) ## Limitations - If User account exist, then deposit and gas are refunded as usual: gas is refuned to Relayer, deposit is refunded to User. - If User account doesn't exist then gas is refunded to Relayer, deposit is burnt. - `DelegateAction` actions mustn't contain another `DelegateAction` (`DelegateAction` can't contain the nested ones). ### DelegateAction Delegate actions allow an account to initiate a batch of actions on behalf of a receiving account, allowing proxy actions. This can be used to implement meta transactions. ```rust pub struct DelegateAction { /// Signer of the delegated actions sender_id: AccountId, /// Receiver of the delegated actions. receiver_id: AccountId, /// List of actions to be executed. actions: Vec, /// Nonce to ensure that the same delegate action is not sent twice by a relayer and should match for given account's `public_key`. /// After this action is processed it will increment. nonce: Nonce, /// The maximal height of the block in the blockchain below which the given DelegateAction is valid. max_block_height: BlockHeight, /// Public key that is used to sign this delegated action. public_key: PublicKey, } ``` ```rust pub struct SignedDelegateAction { delegate_action: DelegateAction, /// Signature of the `DelegateAction`. signature: Signature, } ``` Supporting batches of `actions` means `DelegateAction` can be used to initiate complex steps like creating new accounts, transferring funds, deploying contracts, and executing an initialization function all within the same transaction. ##### Validation 1. Validate `DelegateAction` doesn't contain a nested `DelegateAction` in actions. 2. To ensure that a `DelegateAction` is correct, on receipt the following signature verification is performed: `verify_signature(hash(delegate_action), delegate_action.public_key, signature)`. 3. Verify `transaction.receiver_id` matches `delegate_action.sender_id`. 4. Verify `delegate_action.max_block_height`. The `max_block_height` must be greater than the current block height (at the `DelegateAction` processing time). 5. Verify `delegate_action.sender_id` owns `delegate_action.public_key`. 6. Verify `delegate_action.nonce > sender.access_key.nonce`. A `message` is formed in the following format: ```rust struct DelegateActionMessage { signed_delegate_action: SignedDelegateAction } ``` The next set of security concerns are addressed by this format: - `sender_id` is included to ensure that the relayer sets the correct `transaction.receiver_id`. - `max_block_height` is included to ensure that the `DelegateAction` isn't expired. - `nonce` is included to ensure that the `DelegateAction` can't be replayed again. - `public_key` and `sender_id` are needed to ensure that on the right account, work across rotating keys and fetch the correct `nonce`. The permissions are verified based on the variant of `public_key`: - `AccessKeyPermission::FullAccess`, all actions are allowed. - `AccessKeyPermission::FunctionCall`, only a single `FunctionCall` action is allowed in `actions`. - `DelegateAction.receiver_id` must match to the `account[public_key].receiver_id` - `DelegateAction.actions[0].method_name` must be in the `account[public_key].method_names` ##### Outcomes - If the `signature` matches the receiver's account's `public_key`, a new receipt is created from this account with a set of `ActionReceipt { receiver_id, action }` for each action in `actions`. ##### Recommendations - Because the User doesn't trust the Relayer, the User should verify whether the Relayer has submitted the `DelegateAction` and the execution result. ### Errors - If the Sender's account doesn't exist ```rust /// Happens when TX receiver_id doesn't exist AccountDoesNotExist ``` - If the `signature` does not match the data and the `public_key` of the given key, then the following error will be returned ```rust /// Signature does not match the provided actions and given signer public key. DelegateActionInvalidSignature ``` - If the `sender_id` doesn't match the `tx.receiver_id` ```rust /// Receiver of the transaction doesn't match Sender of the delegate action DelegateActionSenderDoesNotMatchTxReceiver ``` - If the current block is equal or greater than `max_block_height` ```rust /// Delegate action has expired DelegateActionExpired ``` - If the `public_key` does not exist for Sender account ```rust /// The given public key doesn't exist for Sender account DelegateActionAccessKeyError ``` - If the `nonce` does match the `public_key` for the `sender_id` ```rust /// Nonce must be greater sender[public_key].nonce DelegateActionInvalidNonce ``` - If `nonce` is too large ```rust /// DelegateAction nonce is larger than the upper bound given by the block height (block_height * 1e6) DelegateActionNonceTooLarge ``` - If the list of Transaction actions contains several `DelegateAction` ```rust /// There should be the only one DelegateAction DelegateActionMustBeOnlyOne ``` See the [DelegateAction specification](https://nomicon.io/RuntimeSpec/Actions#DelegateAction) for details. ## Security Implications Delegate actions do not override `signer_public_key`, leaving that to the original signer that initiated the transaction (e.g. the relayer in the meta transaction case). Although it is possible to override the `signer_public_key` in the context with one from the `DelegateAction`, there is no clear value in that. See the **_Validation_** section in [DelegateAction specification](https://nomicon.io/RuntimeSpec/Actions#DelegateAction) for security considerations around what the user signs and the validation of actions with different permissions. ## Drawbacks - Increases complexity of NEAR's transactional model. - Meta transactions take an extra block to execute, as they first need to be included by the originating account, then routed to the delegate account, and only after that to the real destination. - User can't call functions from different contracts in same `DelegateAction`. This is because `DelegateAction` has only one receiver for all inner actions. - The Relayer must verify most of the parameters before submitting `DelegateAction`, making sure that one of the function calls is the reward action. Either way, this is a risk for Relayer in general. - User must not trust Relayer’s response and should check execution errors in Blockchain. ## Future possibilities Supporting ZK proofs instead of just signatures can allow for anonymous transactions, which pay fees to relayers anonymously. ## Changelog ### 1.1.0 - Adjust errors to reflect deployed reality (03-Aug`2023) - Remove the error variant `DelegateActionCantContainNestedOne` because this would already fail in the parsing stage. - Rename the error variant `DelegateActionSenderDoesNotMatchReceiver` to `DelegateActionSenderDoesNotMatchTxReceiver` to reflect published types in [near_primitives](https://docs.rs/near-primitives/0.17.0/near_primitives/errors/enum.ActionErrorKind.html#variant.DelegateActionSenderDoesNotMatchTxReceiver). ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0368.md ================================================ --- NEP: 368 Title: Bridge Wallets Author: lewis-sqa <@lewis-sqa> Status: Final DiscussionsTo: https://github.com/near/NEPs/pull/368 Type: Standards Track Category: Wallet Created: 1-Jul-2022 --- # Bridge Wallets ## Summary Standard interface for bridge wallets. ## Motivation Bridge wallets such as [WalletConnect](https://docs.walletconnect.com/2.0/) and [Nightly Connect](https://connect.nightly.app/) are powerful messaging layers for communicating with various blockchains. Since they lack an opinion on how payloads are structured, without a standard, it can be impossible for dApps and wallets to universally communicate without compatibility problems. ## Rationale and alternatives At its most basic, a wallet manages key pairs which are used to sign messages. The signed messages are typically then submitted by the wallet to the blockchain. This standard aims to define an API (based on our learning from [Wallet Selector](https://github.com/near/wallet-selector)) that achieves this requirement through a number of methods compatible with a relay architecture. There have been many iterations of this standard to help inform what we consider the best approach right now for NEAR. You can find more relevant content in the Injected Wallet Standard. ## Specification Bridge wallets use a relay architecture to forward signing requests between dApps and wallets. Requests are typically relayed using a messaging protocol such as WebSockets or by polling a REST API. The concept of a `session` wraps this connection between a dApp and a wallet. When the session is established, the wallet user typically selects which accounts that the dApp should have any awareness of. As the user interacts with the dApp and performs actions that require signing, messages are relayed to the wallet from the dApp, those messages are signed by the wallet and submitted to the blockchain on behalf of the requesting dApp. This relay architecture decouples the 'signing' context (wallet) and the 'requesting' context (dApp, which enables signing to be performed on an entirely different device than the dApp browser is running on. To establish a session, the dApp must first pair with the wallet. Pairing often includes a QR code to improve UX. Once both clients are paired, a request to initialize a session is made. During this phase, the wallet user is prompted to select one or more accounts (previously imported) to be visible to the session before approving the request. Once a session has been created, the dApp can make requests to sign transactions using either [`signTransaction`](#signtransaction) or [`signTransactions`](#signtransactions). These methods accept encoded [Transactions](https://nomicon.io/RuntimeSpec/Transactions) created with `near-api-js`. Since transactions must know the public key that will be used as the `signerId`, a call to [`getAccounts`](#getaccounts) is required to retrieve a list of the accounts visible to the session along with their associated public key. Requests to both [`signTransaction`](#signtransaction) and [`signTransactions`](#signtransactions) require explicit approval from the user since [`FullAccess`](https://nomicon.io/DataStructures/AccessKey) keys are used. For dApps that regularly sign gas-only transactions, Limited [`FunctionCall`](https://nomicon.io/DataStructures/AccessKey#accesskeypermissionfunctioncall) access keys can be added/deleted to one or more accounts by using the [`signIn`](#signin) and [`signOut`](#signout) methods. While the same functionality could be achieved with [`signTransactions`](#signtransactions), that contain actions that add specific access keys with particular permissions to a specific account, by using `signIn`, the wallet receives a direct intention that a user wishes to sign in/out of a dApp's smart contract, which can provide a cleaner UI to the wallet user and allow convenient behavior to be implemented by wallet providers such as 'sign out' automatically deleting the associated limited access key that was created when the user first signed in. Although intentionally similar to the Injected Wallet Standard, this standard focuses on the transport layer instead of the high-level abstractions found in injected wallets. Below are the key differences between the standards: - [Transactions](https://nomicon.io/RuntimeSpec/Transactions) passed to `signTransaction` and `signTransactions` must be encoded. - The result of `signTransaction` and `signTransactions` are encoded [SignedTransaction](https://nomicon.io/RuntimeSpec/Transactions#signed-transaction) models. - Accounts contain only a string representation of public keys. ### Methods #### `signTransaction` Sign a transaction. This request should require explicit approval from the user. ```ts import { transactions } from "near-api-js"; interface SignTransactionParams { // Encoded Transaction via transactions.Transaction.encode(). transaction: Uint8Array; } // Encoded SignedTransaction via transactions.SignedTransaction.encode(). type SignTransactionResponse = Uint8Array; ``` #### `signTransactions` Sign a list of transactions. This request should require explicit approval from the user. ```ts import { providers, transactions } from "near-api-js"; interface SignTransactionsParams { // Encoded Transaction via transactions.Transaction.encode(). transactions: Array; } // Encoded SignedTransaction via transactions.SignedTransaction.encode(). type SignTransactionsResponse = Array; ``` #### `signIn` For dApps that often sign gas-only transactions, `FunctionCall` access keys can be created for one or more accounts to greatly improve the UX. While this could be achieved with `signTransactions`, it suggests a direct intention that a user wishes to sign in to a dApp's smart contract. ```ts import { transactions } from "near-api-js"; interface Account { accountId: string; publicKey: string; } interface SignInParams { permission: transactions.FunctionCallPermission; accounts: Array; } type SignInResponse = null; ``` #### `signOut` Delete one or more `FunctionCall` access keys created with `signIn`. While this could be achieved with `signTransactions`, it suggests a direct intention that a user wishes to sign out from a dApp's smart contract. ```ts interface Account { accountId: string; publicKey: string; } interface SignOutParams { accounts: Array; } type SignOutResponse = null; ``` #### `getAccounts` Retrieve all accounts visible to the session. `publicKey` references the underlying `FullAccess` key linked to each account. ```ts interface Account { accountId: string; publicKey: string; } interface GetAccountsParams {} type GetAccountsResponse = Array; ``` ## Flows ### Connect 1. dApp initiates pairing via QR modal. 2. wallet establishes pairing and prompts selection of accounts for new session. 3. wallet responds with session (id and accounts). 4. dApp stores reference to session. ### Sign in (optional) 1. dApp generates a key pair for one or more accounts in the session. 2. dApp makes `signIn` request with `permission` and `accounts`. 3. wallet receives request and executes a transaction containing an `AddKey` Action for each account. 4. wallet responds with `null`. 5. dApp stores the newly generated key pairs securely. ### Sign out (optional) 1. dApp makes `signOut` request with `accounts`. 2. wallet receives request and executes a transaction containing a `DeleteKey` Action for each account. 3. wallet responds with `null`. 4. dApp clears stored key pairs. ### Sign transaction 1. dApp makes `signTransaction` request. 2. wallet prompts approval of transaction. 3. wallet signs the transaction. 4. wallet responds with `Uint8Array`. 5. dApp decodes signed transaction. 6. dApp sends signed transaction. ### Sign transactions 1. dApp makes `signTransactions` request. 2. wallet prompts approval of transactions. 3. wallet signs the transactions. 4. wallet responds with `Array`. 5. dApp decodes signed transactions. 6. dApp sends signed transactions. ================================================ FILE: neps/nep-0393.md ================================================ --- NEP: 393 Title: Soulbound Token Authors: Robert Zaremba <@robert-zaremba> Status: Final DiscussionsTo: Type: Standards Track Category: Contract Created: 12-Sep-2022 Requires: --- ## Summary Soulbound Token (SBT) is a form of a non-fungible token which represents an aspect of an account: _soul_. [Transferability](#transferability) is limited only to a case of recoverability or a _soul transfer_. The latter must coordinate with a registry to transfer all SBTs from one account to another, and _banning_ the source account. SBTs are well suited for carrying proof-of-attendance, proof-of-unique-human "stamps" and other similar credibility-carriers. ## Motivation Recent [Decentralized Society](https://www.bankless.com/decentralized-society-desoc-explained) trends open a new area of Web3 research to model various aspects of what characterizes humans. Economic and governance value is generated by humans and their relationship. SBTs can represent the commitments, credentials, and affiliations of “Souls” that encode the trust networks of the real economy to establish provenance and reputation. > More importantly, SBTs enable other applications of increasing ambition, such as community wallet recovery, Sybil-resistant governance, mechanisms for decentralization, and novel markets with decomposable, shared rights. We call this richer, pluralistic ecosystem “Decentralized Society” (DeSoc)—a co-determined sociality, where Souls and communities come together bottom-up, as emergent properties of each other to co-create plural network goods and intelligences, at a range of scales. Creating strong primitives is necessary to model new innovative systems and decentralized societies. Examples include reputation protocols, non-transferrable certificates, non-transferrable rights, undercollateralized lending, proof-of-personhood, proof-of-attendance, proof-of-skill, one-person-one-vote, fair airdrops & ICOs, universal basic income, non KYC identity systems, Human DAOs and methods for Sybil attack resistance. We propose an SBT standard to model protocols described above. _Verifiable Credentials_ (VC) could be seen as subset of SBT. However there is an important distinction: VC require set of claims and privacy protocols. It would make more sense to model VC with relation to W3 DID standard. SBT is different, it doesn't require a [resolver](https://www.w3.org/TR/did-core/#dfn-did-resolvers) nor [method](https://www.w3.org/TR/did-core/#dfn-did-methods) registry. For SBT, we need something more elastic than VC. ## Specification Main requirement for Soulbound tokens is to bound an account to a human. A **Soul** is an account with SBTs, which are used to define account identity. Often non transferrable NFT (an NFT token with a no-op transfer function) is used to implement SBTs. However, such model is rather shortsighted. Transferability is required to allow users to either recover their SBTs or merge between the accounts they own. At the same time, we need to limit transferability to assure that an SBTs is kept bound to the same _soul_. We also need an efficient on way to make composed ownership queries (for example: check if an account owns SBT of class C1, C2 and C3 issued by issuer I1, I2 and I3 respectively) - this is needed to model emergent properties discussed above. We introduce a **soul transfer**: an ability for user to move ALL SBT tokens from one account to another in a [semi atomic](#soul-transfer) way, while keeping the SBT bounded to the same _soul_. This happens when a user needs to merge his accounts (e.g. they started with a few different accounts but later decides to merge them to increase an account reputation). Soul transfer is different than a token transfer. The standard forbids a traditional token transfer functionality, where a user can transfer individual tokens. That being said, a registry can have extension functions for more advanced scenarios, which could require a governance approval. SBT standard separates the token issuer concept from the token registry in order to meet the requirements listed above. In the following sections we discuss the functionality of an issuer and registry. ### SBT Registry Traditional Token model in NEAR blockchain assumes that each token has it's own balance book and implements the authorization and issuance mechanism in the same smart contract. Such model prevents atomic _soul transfer_ in the current NEAR runtime. When token balance is kept separately in each SBT smart contract, synchronizing transfer calls to all such contracts to assure atomicity is not possible at scale. We need an additional contract, the `SBT Registry`, to provide atomic transfer of all user SBTs and efficient way to ban accounts in relation to the Ban event discussed below. This, and efficient cross-issuer queries are the main reasons SBT standards separates that token registry and token issuance concerns. Issuer is an entity which issues new tokens and potentially can update the tokens (for example execute renewal). All standard modification options are discussed in the sections below. Registry is a smart contract, where issuers register the tokens. Registry provides a balance book of all associated SBTs. Registry must ensure that each issuer has it's own "sandbox" and issuers won't overwrite each other. A registry provides an efficient way to query multiple tokens for a single user. This will allow implementation of use cases such us: - SBT based identities (main use case of the `i-am-human` protocol); - SBT classes; - decentralized societies. ```mermaid graph TB Issuer1--uses--> Registry Issuer2--uses--> Registry Issuer3--uses--> Registry ``` We can have multiple competing registries (with different purpose or different management scheme). An SBT issuer SHOULD opt-in to a registry before being able to use registry. Registries may develop different opt-in mechanisms (they could differ by the approval mechanism, be fully permissioned etc..). One SBT smart contract can opt-in to: - many registries: it MUST relay all state change functions to all registries. - or to no registry. We should think about it as a single token registry, and it MUST strictly implement all SBT Registry query functions by itself. The contract address must be part of the arguments, and it must check that it equals to the deployed account address (`require!(ctr == env::current_account_id())`). It also MUST emit related events by itself. We recommend that each issuer will use only one registry to avoid complex reconciliation and assure single source of truth. The registry fills a central and important role. But it is **not centralized**, as anyone can create their own registry and SBT issuers can choose which registry to use. It's also not too powerful, as almost all of the power (mint, revoke, burn, recover, etc) still remains with the SBT issuer and not with the registry. #### Issuer authorization A registry can limit which issuers can use registry to mint SBTs by implementing a custom issuer whitelist methods (for example simple access control list managed by a DAO) or keep it fully open (allowing any issuer minting withing the registry). Example: an `SBT_1 Issuer` wants to mint tokens using the `SBT_Registry`. The `SBT_Registry` has a DAO which votes on adding a new issuer: ```mermaid sequenceDiagram actor Issuer1 as SBT_1 Issuer actor DAO participant SBT_Registry Note over Issuer1,DAO: Issuer1 connects with the DAO
to be whitelisted. Issuer1-->>DAO: request whitelist DAO->>SBT_Registry: whitelist(SBT_1 Issuer) ``` #### Personal Identifiable Information Issuers must not include any PII into any SBT. ### Account Ban `Ban` is an event emitted by a registry signaling that the account is banned, and can't own any SBT. Registry must return zero for every SBT supply query of a banned account. Operations which trigger soul transfer must emit Ban. A registry can emit a `Ban` for use cases not discussed in this standard. Handling it depends on the registry governance. One example is to use social governance to identify fake accounts (like bots) - in that case the registry should allow to emit `Ban` and block a scam soul and block future transfers. NOTE: an SBT Issuer can have it's own list of blocked accounts or allowed only accounts. ### Minting Minting is done by issuer calling `registry.sbt_mint(tokens_to_mint)` method. Standard doesn't specify how a registry authorizes an issuer. A classical approach is a whitelist of issuers: any whitelisted issuer can mint any amount of new tokens. Registry must keep the balances and assign token IDs to newly minted tokens. Example: Alice has two accounts: `alice1` and `alice2` which she used to mint tokens. She is getting tokens from 2 issuers that use the same registry. Alice uses her `alice1` account to interact with `SBT_1 Issuer` and receives an SBT with token ID = 238: ```mermaid sequenceDiagram actor Alice actor Issuer1 as SBT_1 Issuer participant SBT1 as SBT_1 Contract participant SBT_Registry Issuer1->>SBT1: sbt_mint(alice1, metadata) activate SBT1 SBT1-)SBT_Registry: sbt_mint([[alice1, [metadata]]]) SBT_Registry->>SBT_Registry: emit Mint(SBT_1_Contract, alice1, [238]) SBT_Registry-)SBT1: [238] deactivate SBT1 Note over Alice,SBT_Registry: now Alice can query registry to check her SBT Alice-->>SBT_Registry: sbt(SBT_1_Contract, 238) SBT_Registry-->>Alice: {token: 238, owner: alice1, metadata} ``` With `SBT_2 Issuer`, Alice uses her `alice2` account. Note that `SBT_2 Contract` has different mint function (can mint many tokens at once), and validates a proof prior to requesting the registry to mint the tokens. ```mermaid sequenceDiagram actor Alice actor Issuer2 as SBT_2 Issuer participant SBT2 as SBT_2 Contract participant SBT_Registry Issuer2->>SBT2: sbt_mint_multi([[alice2, metadata2], [alice2, metadata3]], proof) activate SBT2 SBT2-)SBT_Registry: sbt_mint([[alice2, [metadata2, metadata3]]]) SBT_Registry->>SBT_Registry: emit Mint(SBT_2_Contract, alice2, [7991, 7992]) SBT_Registry-)SBT2: [7991, 1992] deactivate SBT2 Note over Alice,SBT_Registry: Alice queries one of her new tokens Alice-->>SBT_Registry: sbt(SBT_2_Contract, 7991) SBT_Registry-->>Alice: {token: 7991, owner: alice2, metadata: metadata2} ``` ### Transferability Safeguards are set against misuse of SBT transfer and keep the _soul bound_ property. SBT transfer from one account to another should be strictly limited to: - **revocation** allows issuer to invalidate or burn an SBT in case a token issuance should be reverted (for example the recipient is a Sybil account, that is an account controlled by an entity trying to create the false appearance); - **recoverability** in case a user's private key is compromised due to extortion, loss, etc. Users cannot recover an SBT only by themselves. Users must connect with issuer to request recoverability or use more advanced mechanism (like social recoverability). The recovery function provides additional economical cost preventing account trading: user should always be able to recover his SBT, and move to another, not banned account. - **soul transfer** - moving all SBT tokens from a source account (issued by all issuers) to a destination account. During such transfer, SBT registry emits `SoulTransfer` and `Ban` events. The latter signals that the account can't host nor receive any SBT in the future, effectively burning the identity of the source account. This creates an inherit cost for the source account: it's identity can't be used any more. Registry can have extension functions for more advanced scenarios, which could require a governance mechanism. `SoulTransfer` event can also trigger similar actions in other registries (specification for this is out of the scope of this NEP). This becomes especially important for proof-of-human stamps that can only be issued once per user. #### Revocation An issuer can revoke SBTs by calling `registry.sbt_revoke(tokens_to_revoke, burn)`. Example: when a related certificate or membership should be revoked, when an issuer finds out that there was an abuse or a scam, etc...). Registry, when receiving `sbt_revoke` request from an issuer must always emit the `Revoke` event. Registry must only accept revoke requests from a valid issuer, and only revoke tokens from that issuer. If `burn=true` is set in the request, then the token should be burned and `Burn` event must be emitted. Otherwise (when `burn=false`) the registry must update token metadata and set expire date to a time in the past. Registry must not ban nor emit `Ban` event when revoking a contract. That would create an attack vector, when a malicious registry would thread the registry by banning accounts. #### Recoverability Standard defines issuer recoverability. At minimum, the standard registry exposes `sbt_recover` method, which allows issuer to reassign a token issued by him from one account to another. SBT recovery MUST not trigger `SoulTransfer` nor `Ban` event: malicious issuer could compromise the system by faking the token recovery and take over all other SBTs from a user. Only the owner of the account can make a Soul Transfer transaction and merge 2 accounts they owns. #### Recoverability within an SBT Registry SBT registry can define it's own mechanism to atomically recover all tokens related to one account and execute soul transfer to another account, without going one by one through each SBT issuer (sometimes that might be even not possible). Below we list few ideas a Registry can use to implement recovery: - KYC based recovery - Social recovery ![Social Recovery, Image via “Decentralized Society”](https://bankless.ghost.io/content/images/public/images/cdb1fc23-6179-44f0-9bfe-e5e5831492f7_1399x680.png) SBT Registry based recovery is not part of this specification. #### Soul Transfer The basic use case is described above. Registry MUST provide a permissionless method to allow any user to execute soul transfer. It is essential part of the standard, but the exact interface of the method is not part of the standard, because registries may adopt different mechanism and require different arguments. Soul transfers must be _semi atomic_. That is, the holder account must be non operational (in terms of SBT supply) until the soul transfer is completed. Given the nature of NEAR blockchain, where transactions are limited by gas, big registries may require to implement the Soul Transfer operation in stages. Source and destination accounts should act as a non-soul accounts while the soul transfer operation is ongoing. For example, in the first call, contract can lock the account, and do maximum X amount of transfers. If the list of to be transferred SBTs has not been exhausted, the contract should keep locking the account and remember the last transferred SBT. Subsequent calls by the same user will resume the operation until the list is exhausted. Soul Transfer must emit the `SoulTransfer` event. Example: Alice has two accounts: `alice1` and `alice2` which she used to mint tokens (see [mint diagram](#minting)). She decides to merge the accounts by doing Soul Transfer. ```mermaid sequenceDiagram actor Alice participant SBT_Registry Alice->>SBT_Registry: sbt_soul_transfer(alice1) --accountId alice2 Alice-->>+SBT_Registry: sbt_tokens_by_owner(alice2) SBT_Registry-->>-Alice: [] Alice-->>+SBT_Registry: sbt_tokens_by_owner(alice1) SBT_Registry-->>-Alice: [[SBT_1_Contract, [238]], [SBT_2_Contract, [7991, 7992]]]} ``` Implementation Notes: - There is a risk of conflict. The standard requires that one account can't have more than one SBT of the same (issuer, class) pair. - When both `alice1` and `alice2` have SBT of the same (issuer, class) pair, then the transfer should fail. One of the accounts should burn conflicting tokens to be able to continue the soul transfer. - Soul transfer may require extra confirmation before executing a transfer. For example, if `alice1` wants to do a soul transfer to `alice2`, the contract my require `alice2` approval before continuing the transfer. - Other techniques may be used to enforce that the source account will be deleted. ### Renewal Soulbound tokens can have an _expire date_. It is useful for tokens which are related to real world certificates with expire time, or social mechanisms (e.g. community membership). Such tokens SHOULD have an option to be renewable. Examples include mandatory renewal with a frequency to check that the owner is still alive, or renew membership to a DAO that uses SBTs as membership gating. Registry defines `sbt_renew` method allowing issuers to update the token expire date. The issuer can set the _expire date_ in the past. This is useful if an issuer wants to invalidate the token without removing it. ### Burning tokens Registry MAY expose a mechanism to allow an account to burn an unwanted token. The exact mechanism is not part of the standard and it will depend on the registry implementation. We only define a standard `Burn` event, which must be emitted each time a token is removed from existence. Some registries may forbid accounts to burn their tokens in order to preserve specific claims. Ultimately, NEAR is a public blockchain, and even if a token is burned, it's trace will be preserved. ### Token Class (multitoken approach) SBT tokens can't be fractionized. Also, by definition, there should be only one SBT per token class per user. Examples: user should not be able to receive few badges of the same class, or few proof of attendance to the same event. However, we identify a need for having to support token classes (aka multitoken interface) in a single contract: - badges: one contract. Each badge will have a class (community lead, OG...), and each token will belong to a specific class; - certificates: one issuer can create certificates of a different class (eg school department can create diplomas for each major and each graduation year). We also see a trend in the NFT community and demand for market places to support multi token contracts. - In Ethereum community many projects are using [ERC-1155 Multi Token Standard](https://eips.ethereum.org/EIPS/eip-1155). NFT projects are using it for fraction ownership: each token id can have many fungible fractions. - NEAR [NEP-245](https://github.com/near/NEPs/blob/master/neps/nep-0245.md) has elaborated similar interface for both bridge compatibility with EVM chains as well as flexibility to define different token types with different behavior in a single contract. [DevGovGigs Board](https://near.social/#/mob.near/widget/MainPage.Post.Page?accountId=devgovgigs.near&blockHeight=87938945) recently also shows growing interest to move NEP-245 adoption forward. - [NEP-454](https://github.com/near/NEPs/pull/454) proposes royalties support for multi token contracts. We propose that the SBT Standard will support the multi-token idea from the get go. This won't increase the complexity of the contract (in a traditional case, where one contract will only issue tokens of the single class, the `class` argument is simply ignored in the state, and in the functions it's required to be of a constant value, eg `1`) but will unify the interface. It's up to the smart contract design how the token classes is managed. A smart contract can expose an admin function (example: `sbt_new_class() -> ClassId`) or hard code the pre-registered classes. Finally, we require that each token ID is unique within the smart contract. This will allow us to query token only by token ID, without knowing it's class. ## Smart contract interface For the Token ID type we propose `u64` rather than `U128`. `u64` capacity is more than 1e19. If we will mint 10'000 SBTs per second, then it will take us 58'494'241 years to fill the capacity. Today, the JS integer limit is `2^53-1 ~ 9e15`. Similarly, when minting 10'000 SBTs per second, it will take us 28'561 years to reach the limit. So, we don't need u128 nor the String type. However, if for some reason, we will need to get u64 support for JS, then we can always add another set of methods which will return String, so making it compatible with NFT standard (which is using `U128`, which is a string). Also, it's worth to note, that in 28'000 years JS (if it will still exists) will be completely different. The number of combinations for a single issuer is much higher in fact: the token standard uses classes. So technically that makes the number of all possible combinations for a single issuer equal `(2^64)^2 ~ 1e38`. For "today" JS it is `(2^53-1)^2 ~ 1e31`. Token IDs MUST be created in a sequence to make sure the ID space is not exhausted locally (eg if a registry would decide to introduce segments, it would potentially get into a trap where one of the segments is filled up very quickly). ```rust // TokenId and ClassId must be positive (0 is not a valid ID) pub type TokenId = u64; pub type ClassId = u64; pub struct Token { pub token: TokenId, pub owner: AccountId, pub metadata: TokenMetadata, } ``` The Soulbound Token follows the NFT [NEP-171](https://github.com/near/NEPs/blob/master/neps/nep-0171.md) interface, with few differences: - token ID is `u64` (as discussed above). - token class is `u64`, it's required when minting and it's part of the token metadata. - `TokenMetadata` doesn't have `title`, `description`, `media`, `media_hash`, `copies`, `extra`, `starts_at` nor `updated_at`. All that attributes except the `updated_at` can be part of the document stored at `reference`. `updated_at` can be tracked easily by indexers. - We don't have traditional transferability. - We propose to use more targeted events, to better reflect the event nature. Moreover events are emitted by the registry, so we need to include issuer contract address in the event. All time related attributes are defined in milliseconds (as per NEP-171). ```rust /// IssuerMetadata defines contract wide attributes, which describes the whole contract. /// Must be provided by the Issuer contract. See the `SBTIssuer` trait. pub struct IssuerMetadata { /// Version with namespace, example: "sbt-1.0.0". Required. pub spec: String, /// Issuer Name, required, ex. "Mosaics" pub name: String, /// Issuer symbol which can be used as a token symbol, eg Ⓝ, ₿, BTC, MOSAIC ... pub symbol: String, /// Icon content (SVG) or a link to an Icon. If it doesn't start with a scheme (eg: https://) /// then `base_uri` should be prepended. pub icon: Option, /// URI prefix which will be prepended to other links which don't start with a scheme /// (eg: ipfs:// or https:// ...). pub base_uri: Option, /// JSON or an URL to a JSON file with more info. If it doesn't start with a scheme /// (eg: https://) then base_uri should be prepended. pub reference: Option, /// Base64-encoded sha256 hash of JSON from reference field. Required if `reference` is included. pub reference_hash: Option, } /// ClassMetadata defines SBT class wide attributes, which are shared and default to all SBTs of /// the given class. Must be provided by the Issuer contract. See the `SBTIssuer` trait. pub struct ClassMetadata { /// Issuer class name. Required to be not empty. pub name: String, /// If defined, should be used instead of `IssuerMetadata::symbol`. pub symbol: Option, /// An URL to an Icon. To protect fellow developers from unintentionally triggering any /// SSRF vulnerabilities with URL parsers, we don't allow to set an image bytes here. /// If it doesn't start with a scheme (eg: https://) then `IssuerMetadata::base_uri` /// should be prepended. pub icon: Option, /// JSON or an URL to a JSON file with more info. If it doesn't start with a scheme /// (eg: https://) then base_uri should be prepended. pub reference: Option, /// Base64-encoded sha256 hash of JSON from reference field. Required if `reference` is included. pub reference_hash: Option, } /// TokenMetadata defines attributes for each SBT token. pub struct TokenMetadata { pub class: ClassId, // token class. Required. Must be non zero. pub issued_at: Option, // When token was issued or minted, Unix time in milliseconds pub expires_at: Option, // When token expires, Unix time in milliseconds /// JSON or an URL to a JSON file with more info. If it doesn't start with a scheme /// (eg: https://) then base_uri should be prepended. pub reference: Option, /// Base64-encoded sha256 hash of JSON from reference field. Required if `reference` is included. pub reference_hash: Option, } trait SBTRegistry { /********** * QUERIES **********/ /// Get the information about specific token ID issued by `issuer` SBT contract. fn sbt(&self, issuer: AccountId, token: TokenId) -> Option; /// Get the information about list of token IDs issued by the `issuer` SBT contract. /// If token ID is not found `None` is set in the specific return index. fn sbts(&self, issuer: AccountId, token: Vec) -> Vec>; /// Query class ID for each token ID issued by the SBT `issuer`. /// If token ID is not found, `None` is set in the specific return index. fn sbt_classes(&self, issuer: AccountId, tokens: Vec) -> Vec>; /// Returns total amount of tokens issued by `issuer` SBT contract, including expired /// tokens. If a revoke removes a token, it must not be included in the supply. fn sbt_supply(&self, issuer: AccountId) -> u64; /// Returns total amount of tokens of given class minted by `issuer`. See `sbt_supply` for /// information about revoked tokens. fn sbt_supply_by_class(&self, issuer: AccountId, class: ClassId) -> u64; /// Returns total supply of SBTs for a given owner. See `sbt_supply` for information about /// revoked tokens. /// If `class` is specified, returns only owner supply of the given class (either 0 or 1). fn sbt_supply_by_owner( &self, account: AccountId, issuer: AccountId, class: Option, ) -> u64; /// Query sbt tokens issued by a given contract. /// `limit` specifies the upper limit of how many tokens we want to return. /// If `from_token` is not specified, then `from_token` should be assumed /// to be the first valid token id. If `with_expired` is set to `true` then all the tokens are returned /// including expired ones otherwise only non-expired tokens are returned. fn sbt_tokens( &self, issuer: AccountId, from_token: Option, limit: Option, with_expired: bool, ) -> Vec; /// Query SBT tokens by owner. /// `limit` specifies the upper limit of how many tokens we want to return. /// If `from_class` is not specified, then `from_class` should be assumed to be the first /// valid class id. If `with_expired` is set to `true` then all the tokens are returned /// including expired ones otherwise only non-expired tokens are returned. /// Returns list of pairs: `(Contract address, list of token IDs)`. fn sbt_tokens_by_owner( &self, account: AccountId, issuer: Option, from_class: Option, limit: Option, with_expired: bool, ) -> Vec<(AccountId, Vec)>; /// checks if an `account` was banned by the registry. fn is_banned(&self, account: AccountId) -> bool; /************* * Transactions *************/ /// Creates a new, unique token and assigns it to the `receiver`. /// `token_spec` is a vector of pairs: owner AccountId and TokenMetadata. /// Each TokenMetadata must have non zero `class`. /// Must be called by an SBT contract. /// Must emit `Mint` event. /// Must provide enough NEAR to cover registry storage cost. // #[payable] fn sbt_mint(&mut self, token_spec: Vec<(AccountId, Vec)>) -> Vec; /// sbt_recover reassigns all tokens issued by the caller, from the old owner to a new owner. /// Must be called by a valid SBT issuer. /// Must emit `Recover` event once all the tokens have been recovered. /// Requires attaching enough tokens to cover the storage growth. /// Returns the amount of tokens recovered and a boolean: `true` if the whole /// process has finished, `false` when the process has not finished and should be /// continued by a subsequent call. User must keep calling the `sbt_recover` until `true` /// is returned. // #[payable] fn sbt_recover(&mut self, from: AccountId, to: AccountId) -> (u32, bool); /// sbt_renew will update the expire time of provided tokens. /// `expires_at` is a unix timestamp (in miliseconds). /// Must be called by an SBT contract. /// Must emit `Renew` event. fn sbt_renew(&mut self, tokens: Vec, expires_at: u64); /// Revokes SBT by burning the token or updating its expire time. /// Must be called by an SBT contract. /// Must emit `Revoke` event. /// Must also emit `Burn` event if the SBT tokens are burned (removed). fn sbt_revoke(&mut self, tokens: Vec, burn: bool); /// Similar to `sbt_revoke`. Allows SBT issuer to revoke all tokens by holder either by /// burning or updating their expire time. When an owner has many tokens from the issuer, /// the issuer may need to call this function multiple times, until all tokens are revoked. /// Retuns true if all the tokens were revoked, false otherwise. /// If false is returned issuer must call the method until true is returned /// Must be called by an SBT contract. /// Must emit `Revoke` event. /// Must also emit `Burn` event if the SBT tokens are burned (removed). fn sbt_revoke_by_owner(&mut self, owner: AccountId, burn: bool) -> bool; /// Allows issuer to update token metadata reference and reference_hash. /// * `updates` is a list of triples: (token ID, reference, reference base64-encoded sha256 hash). /// Must emit `token_reference` event. /// Panics if any of the token IDs don't exist. fn sbt_update_token_references( &mut self, updates: Vec<(TokenId, Option, Option)>, ); } ``` Example **Soul Transfer** interface: ```rust /// Transfers atomically all SBT tokens from one account to another account. /// The caller must be an SBT holder and the `recipient` must not be a banned account. /// Returns the amount of tokens transferred and a boolean: `true` if the whole /// process has finished, `false` when the process has not finished and should be /// continued by a subsequent call. /// Emits `Ban` event for the caller at the beginning of the process. /// Emits `SoulTransfer` event only once all the tokens from the caller were transferred /// and at least one token was trasnfered (caller had at least 1 sbt). /// + User must keep calling the `sbt_soul_transfer` until `true` is returned. /// + If caller does not have any tokens, nothing will be transfered, the caller /// will be banned and `Ban` event will be emitted. #[payable] fn sbt_soul_transfer( &mut self, recipient: AccountId, ) -> (u32, bool); ``` ### SBT Issuer interface SBTIssuer is the minimum required interface to be implemented by issuer. Other methods, such as a mint function, which requests the registry to proceed with token minting, is specific to an Issuer implementation (similarly, mint is not part of the FT standard). The issuer must provide metadata object of the Issuer. Optionally, Issuer can also provide metadata object for each token class. Issuer level (contract) metadata, must provide information common to all tokens and all classes defined by the issuer. Class level metadata, must provide information common to all tokens of a given class. Information should be deduplicated and denormalized whenever possible. Example: The issuer can set a default icon for all tokens (SBT) using `IssuerMetadata::icon` and additionally it can customize an icon of a particular token via `TokenMetadata::icon`. ```rust pub trait SBTIssuer { /// Returns contract metadata. fn sbt_metadata(&self) -> IssuerMetadata; /// Returns SBT class metadata, or `None` if the class is not found. fn sbt_class_metadata(&self, class: ClassId) -> Option; } ``` SBT issuer smart contracts may implement NFT query interface to make it compatible with NFT tools. In that case, the contract should proxy the calls to the related registry. Note, we use U64 type rather than U128. However, SBT issuer must not emit NFT related events. ```rust trait SBTNFT { fn nft_total_supply(&self) -> U64; // here we index by token id instead of by class id (as done in `sbt_tokens_by_owner`) fn nft_tokens_for_owner(&self, account_id: AccountId, from_index: Option, limit: Option) -> Vec; fn nft_supply_for_owner(&self, account_id: AccountId) -> U64; } ``` ### Events Event design principles: - Events don't need to repeat all function arguments - these are easy to retrieve by indexer (events are consumed by indexers anyway). - Events must include fields necessary to identify subject matters related to use case. - When possible, events should contain aggregated data, with respect to the standard function related to the event. ```typescript // only valid integer numbers (without rounding errors). type u64 = number; type Nep393Event { standard: "nep393"; version: "1.0.0"; event: "mint" | "recover" | "renew" | "revoke" | "burn" | "ban" | "soul_transfer" | "token_reference" ; data: Mint | Recover | Renew | Revoke | Burn | Ban[] | SoulTransfer | TokenReference; } /// An event emitted by the Registry when new SBT is created. type Mint { issuer: AccountId; // SBT Contract minting the tokens tokens: (AccountId, u64[])[]; // list of pairs (token owner, TokenId[]) } /// An event emitted when a recovery process succeeded to reassign SBTs, usually due to account /// access loss. This action is usually requested by the owner, but executed by an issuer, /// and doesn't trigger Soul Transfer. Registry reassigns all tokens assigned to `old_owner` /// that were ONLY issued by the `ctr` SBT Contract (hence we don't need to enumerate the /// token IDs). /// Must be emitted by an SBT registry. type Recover { issuer: AccountId // SBT Contract recovering the tokens old_owner: AccountId; // current holder of the SBT new_owner: AccountId; // destination account. } /// An event emitted when existing tokens are renewed. /// Must be emitted by an SBT registry. type Renew { issuer: AccountId; // SBT Contract renewing the tokens tokens: u64[]; // list of token ids. } /// An event emitted when existing tokens are revoked. /// Revoked tokens will continue to be listed by the registry but they should not be listed in /// a wallet. See also `Burn` event. /// Must be emitted by an SBT registry. type Revoke { issuer: AccountId; // SBT Contract revoking the tokens tokens: u64[]; // list of token ids. } /// An event emitted when existing tokens are burned and removed from the registry. /// Must be emitted by an SBT registry. type Burn { issuer: AccountId; // SBT Contract burning the tokens tokens: u64[]; // list of token ids. } /// An event emitted when an account is banned within the emitting registry. /// Registry must add the `account` to a list of accounts that are not allowed to get any SBT /// in the future. /// Must be emitted by an SBT registry. type Ban = AccountId; /// An event emitted when soul transfer is happening: all SBTs owned by `from` are transferred /// to `to`, and the `from` account is banned (can't receive any new SBT). /// Must be emitted by an SBT registry. /// Registry MUST also emit `Ban` whenever the soul transfer happens. type SoulTransfer { from: AccountId; to: AccountId; } /// An event emitted when an issuer updates token metadata reference of existing SBTs. /// Must be emitted by an SBT registry. type TokenReference { issuer: AccountId; // Issuer account tokens: u64[]; // list of token ids. } /// An event emitted when existing token metadata references are updated. type TokenReference = u64[]; // list of token ids. ``` Whenever a recovery is made in a way that an existing SBT is burned, the `Burn` event MUST be emitted. If `Revoke` burns token then `Burn` event MUST be emitted instead of `Revoke`. ### Example SBT Contract functions Although the transaction functions below are not part of the SBT smart contract standard (depending on a use case, they may have different parameters), we present here an example interface for SBT issuance and we also provide a reference implementation. These functions must relay calls to an SBT registry, which will emit appropriate events. We recommend that all functions related to an event will take an optional `memo: Option` argument for accounting purposes. ```rust trait SBT { /// Must provide enough NEAR to cover registry storage cost. // #[payable] fn sbt_mint( &mut self, account: AccountId, metadata: TokenMetadata, memo: Option, ) -> TokenId; /// Creates a new, unique token and assigns it to the `receiver`. /// `token_spec` is a vector of pairs: owner AccountId and TokenMetadata. /// Must provide enough NEAR to cover registry storage cost. // #[payable] fn sbt_mint_multi( &mut self, token_spec: Vec<(AccountId, TokenMetadata)>, memo: Option, ) -> Vec; // #[payable] fn sbt_recover(&mut self, from: AccountId, to: AccountId, memo: Option); fn sbt_renew(&mut self, tokens: Vec, expires_at: u64, memo: Option); fn sbt_revoke(token: Vec, memo: Option) -> bool; } ``` ## Reference Implementation - Common [type definitions](https://github.com/near-ndc/i-am-human/tree/master/contracts/sbt) (events, traits). - [I Am Human](https://github.com/near-ndc/i-am-human) registry and issuers. ## Consequences Being fully compatible with NFT standard is a desirable. However, given the requirements related to _soul transfer_ we didn't find an applaudable solution. Also we decided to use u64 as a Token ID, diverging further from the NFT NEP-171 standard. Given that our requirements are much striker, we had to reconsider the level of compatibility with NEP-171 NFT. There are many examples where NFT standards are improperly implemented. Adding another standard with different functionality but equal naming will cause lots of problems and misclassifications between NFT and SBT. ### Positive - Template and set of guidelines for creating SBT tokens. - Ability to create SBT aggregators. - An SBT standard with recoverability mechanism provides a unified model for multiple primitives such as non KYC identities, badges, certificates etc... - SBT can be further used for "lego" protocols, like: Proof of Humanity (discussed for NDC Governance), undercollateralized lending, role based authentication systems, innovative economic and social applications... - Standard recoverability mechanism. - SBT are considered as a basic primitive for Decentralized Societies. - new way to implement Sybil attack resistance. ### Neutral - The API partially follows the NEP-171 (NFT) standard. The proposed design is to have native SBT API and make it possible for issuer contracts to support NFT based queries if needed (such contract will have a limitation of only issuing SBTs with one `ClassId` only). ### Negative - New set of events to be handled by the indexer and wallets. - Complexity of integration with a registry: all SBT related transactions must go through Registry. ### Privacy Notes > Blockchain-based systems are public by default. Any relationship that is recorded on-chain is immediately visible not just to the participants, but also to anyone in the entire world. Some privacy can be retained by having multiple pseudonyms: a family Soul, a medical Soul, a professional Soul, a political Soul each carrying different SBTs. But done naively, it could be very easy to correlate these Souls to each other. > The consequences of this lack of privacy are serious. Indeed, without explicit measures taken to protect privacy, the “naive” vision of simply putting all SBTs on-chain may well make too much information public for many applications. -- Decentralized Society There are multiple ways how an identity can be doxxed using chain data. SBT, indeed provides more data about account. The standard allows for few anonymization methods: - not providing any data in the token metadata (reference...) or encrypt the reference. - anonymize issuers (standard allows to have many issues for the same entity) and mix it with different class ids. These are just a numbers. Perfect privacy can only be done with solid ZKP, not off-chain walls. Implementations must not store any personal information on chain. ## Changelog ### v1.0.0 The Contract Standards Working Group members approved this NEP on June 30, 2023 ([meeting recording](https://youtu.be/S1An5CDG154)). #### Benefits - SBTs as any other kind of a token are essential primitive to represent real world use cases. This standards provides a model and a guideline for developers to build SBT based solutions. - Token standards are key for composability. - Wallet and tools needs a common interface to operate tokens. #### Concerns | # | Concern | Resolution | Status | | --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | | 1 | [Robert] Should we Emit NEP-171 Mint and NEP-171 Burn by the SBT contract (in addition to SBT native events emitted by the registry)? If the events will be emitted by registry, then we need new events to include the contract address. | Don't emit NFT events. SBT is not NFT. Support: @alexastrum | resolved | | 2 | [Robert] remove `memo` in events. The `memo` is already part of the transaction, and should not be needed to identify transactions. Processes looking for events, can easily track transaction through event and recover `memo` if needed. | Removed, consequently also removed from registry transactions . Support: @alexastrum | resolved | | 3 | [Token Spam](https://github.com/near/NEPs/pull/393/#discussion_r1163938750) | We have a `Burn` event. Added example `sbt_burn` function, but keeping it not as a part of required interface. Event should be enough. | resolved | | 4 | [Multiple registries](https://github.com/near/NEPs/pull/393/#discussion_r1163951624). Registry source of truth [comment](https://github.com/near/NEPs/pull/393/#issuecomment-1531766643) | This is a part of the design: permissionless approach. [Justification for registry](https://github.com/near/NEPs/pull/393/#issuecomment-1540621077) | resolved | | 5 | [Robert] Approve the proposed multi-token | Support: @alexastrum | resolved | | 6 | [Robert] Use of milliseconds as a time unit. | Use milliseconds. | resolved | | 7 | Should a `burn` function be part of a standard or a recommendation? | We already have the Burn event. A call method should not be part of the standard interface (similarly to FT and NFT). | resolved | | 8 | [Robert] Don't include `sbt_soul_transfer` in the standard interface, [comment](https://github.com/near/NEPs/pull/393#issuecomment-1506969996). | Moved outside of the required interface. | resolved | | 9 | [Privacy](https://github.com/near/NEPs/pull/393/#issuecomment-1504309947) | Concerns have been addressed: [comment-1](https://github.com/near/NEPs/pull/393/#issuecomment-1504485420) and [comment2](https://github.com/near/NEPs/pull/393/#issuecomment-1505958549) | resolved | | 10 | @frol [suggested](https://github.com/near/NEPs/pull/393/#discussion_r1247879778) to use a struct in `sbt_recover` and `sbt_soul_transfer`. | Motivation to use pair `(number, bool)` rather than follow a common Iterator Pattern. Rust uses `Option` type for that, that works perfectly for languages with native Option type, but creates a "null" problem for anything else. Other common way to implement Iterator is the presented pair, which doesn't require extra type definition and reduces code size. | new | ### v1.1.0 In v1.0.0 we defined Issuer (an entity authorized to mint SBTs in the registry) and SBT Class. We also defined Issuer Metadata and Token Metadata, but we didn't provide interface for class metadata. This was implemented in the reference implementation (in one of the subsequent revisions), but was not backported to the NEP. This update: - Fixes the name of the issuer interface from `SBTContract` to `SBTIssuer`. The original name is wrong and we oversight it in reviews. We talk everywhere about the issuer entity and issuer contract (even the header is SBT Issuer interface). - Renames `ContractMetadata` to `IssuerMetadata`. - Adds `ClassMetadata` struct and `sbt_class_metadata` function to the `SBTIssuer` interface. Reference implementation: [ContractMetadata, ClassMetadata, TokenMetadata](https://github.com/near-ndc/i-am-human/blob/registry/v1.8.0/contracts/sbt/src/metadata.rs#L18) and [SBTIssuer interface](https://github.com/near-ndc/i-am-human/blob/registry/v1.8.0/contracts/sbt/src/lib.rs#L49). #### Benefits - Improves the documentation and meaning of the issuer entity. - Adds missing `ClassMetadata`. - Improves issuer, class and token metadata documentation. ## Copyright [Creative Commons Attribution 4.0 International Public License (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/) ================================================ FILE: neps/nep-0399.md ================================================ --- NEP: 399 Title: Flat Storage Author: Aleksandr Logunov Min Zhang Status: Final DiscussionsTo: https://github.com/nearprotocol/neps/pull/0399 Type: Protocol Track Category: Storage Created: 30-Sep-2022 --- ## Summary This NEP proposes the idea of Flat Storage, which stores a flattened map of key/value pairs of the current blockchain state on disk. Note that original Trie (persistent merkelized trie) is not removed, but Flat Storage allows to make storage reads faster, make storage fees more predictable and potentially decrease them. ## Motivation Currently, the blockchain state is stored in our storage only in the format of persistent merkelized tries. Although it is needed to compute state roots and prove the validity of states, reading from it requires a traversal from the trie root to the leaf node that contains the key value pair, which could mean up to 2 \* key_length disk accesses in the worst case. In addition, we charge receipts by the number of trie nodes they touched (TTN cost). Note that the number of touched trie node does not always equal to the key length, it depends on the internal trie structure. Based on some feedback from contract developers collected in the past, they are interested in predictable fees, but TTN costs are annoying to predict and can lead to unexpected excess of the gas limit. They are also a burden for NEAR Protocol client implementations, i.e. nearcore, as exact TTN number must be computed deterministically by all clients. This prevents storage optimizations that use other strategies than nearcore uses today. With Flat Storage, number of disk reads is reduced from worst-case 2 \* key_length to exactly 2, storage read gas fees are simplified by getting rid of TTN cost, and potentially can be further reduced because fewer disk reads are needed. ## Rationale and alternatives Q: Why is this design the best in the space of possible designs? A: Space of possible designs is quite big here, let's show some meaningful examples. The most straightforward one is just to increase TTN, or align it with biggest possible value, or alternatively increase the base fees for storage reads and writes. However, technically the biggest TTN could be 4096 as of today. And we tend to strongly avoid increasing fees, because it may break existing contract calls, not even mentioning that it would greatly reduce capacity of NEAR blocks, because for current mainnet usecases depth is usually below 20. We also consider changing tree type from Trie to AVL, B-tree, etc. to make number of traversed nodes more stable and predictable. But we approached AVL idea, and implementation turned out to be tricky, so we didn't go much further than POC here: https://github.com/near/nearcore/discussions/4815. Also for most key-value pairs tree depth will actually increase - for example, if you have 1M keys, depth is always 20, and it would cause increasing fees as well. Size of intermediate node also increases, because we have to need to store a key there to decide whether we should go to the left or right child. Separate idea is to get rid of global state root completely: https://github.com/near/NEPs/discussions/425. Instead, we could track the latest account state in the block where it was changed. But after closer look, it brings us to similar questions - if some key was untouched for a long time, it becomes harder to find exact block to find latest value for it, and we need some tree-shaped structure again. Because ideas like that could be also extremely invasive, we stopped considering them at this point. Other ideas around storage include exploring databases other than RocksDB, or moving State to a separate database. We can tweak State representation, i.e. start trie key from account id to achieve data locality within the same account. However, Flat Storage main goal is speed up reads and make their costs predictable, and these ideas are orthogonal to that, although they still can improve storage in other ways. Q: What other designs have been considered and what is the rationale for not choosing them? A: There were several ideas on Flat Storage implementation. One of questions is whether we need global flat storage for all shards or separate flat storages. Due to how sharding works, we have make flat storages separate, because in the future node may have to catchup new shard while already tracking old shards, and flat storage heads (see Specification) must be different for these shards. Flat storage deltas are another tricky part of design, but we cannot avoid them, because at certain points of time different nodes can disagree what is the current chain head, and they have to support reads for some subset of latest blocks with decent speed. We can't really fallback to Trie in such cases because storage reads for it are much slower. Another implementation detail is where to put flat storage head. The planned implementation doesn't rely on that significantly and can be changed, but for MVP we assume flat storage head = chain final head as a simplest solution. Q: What is the impact of not doing this? A: Storage reads will remain inefficiently implemented and cost more than they should, and the gas fees will remain difficult for the contract developers to predict. ## Specification The key idea of Flat Storage is to store a direct mapping from trie keys to values in the DB. Here the values of this mapping can be either the value corresponding to the trie key itself, or the value ref, a hash that points to the address of the value. If the value itself is stored, only one disk read is needed to look up a value from flat storage, otherwise two disk reads are needed if the value ref is stored. We will discuss more in the following section for whether we use values or value refs. For the purpose of high level discussion, it suffices to say that with Flat Storage, at most two disk reads are needed to perform a storage read. The simple design above won't work because there could be forks in the chain. In the following case, FlatStorage must support key value lookups for states of the blocks on both forks. ```text / Block B1 - Block B2 - ... block A \ Block C1 - Block C2 - ... ``` The handling of forks will be the main consideration of the following design. More specifically, the design should satisfy the following requirements, 1. It should support concurrent block processing. Blocks on different forks are processed concurrently in the nearcore Client code, the struct which responsibility includes receiving blocks from network, scheduling applying chunks and writing results of that to disk. Flat storage API must be aligned with that. 2. In case of long forks, block processing time should not be too much longer than the average case. We don’t want this case to be exploitable. It is acceptable that block processing time is 200ms longer, which may slow down block production, but probably won’t cause missing blocks and chunks. 10s delays are not acceptable and may lead to more forks and instability in the network. 3. The design must be able to decrease storage access cost in all cases, since we are going to change the storage read fees based on flat storage. We can't conditionally enable Flat Storage for some blocks and disable it for other, because the fees we charge must be consistent. The mapping of key value pairs FlatStorage stored on disk matches the state at some block. We call this block the head of flat storage, or the flat head. During block processing, the flat head is set to the last final block. The Doomslug consensus algorithm guarantees that if a block is final, all future final blocks must be descendants of this block. In other words, any block that is not built on top of the last final block can be discarded because they will never be finalized. As a result, if we use the last final block as the flat head, any block FlatStorage needs to process is a descendant of the flat head. To support key value lookups for other blocks that are not the flat head, FlatStorage will store key value changes(deltas) per block for these blocks. We call these deltas FlatStorageDelta (FSD). Let’s say the flat storage head is at block h, and we are applying transactions based on block h’. Since h is the last final block, h is an ancestor of h'. To access the state at block h', we need FSDs of all blocks between h and h'. Note that all these FSDs must be stored in memory, otherwise, the access of FSDs will trigger more disk reads and we will have to set storage key read fee higher. ### FSD size estimation We prefer to store deltas in memory, because memory read is much faster than disk read, and even a single extra RocksDB access requires increasing storage fees, which is not desirable. To reduce delta size, we will store hashes of trie keys instead of keys, because deltas are read-only. Now let's carefully estimate FSD size. We can do so using protocol fees as of today. Assume that flat state stores a mapping from keys to value refs. Maximal key length is ~2 KiB which is the limit of contract data key size. During wasm execution, we pay `wasm_storage_write_base` = 64 Ggas per call. Entry size is 68 B for key hash and value ref. Then the total size of keys changed in a block is at most `chunk_gas_limit / gas_per_entry * entry_size * num_shards = (1300 Tgas / 64 Ggas) * 68 B * 4 ~= 5.5 MiB`. Assuming that we can increase RAM requirements by 1 GiB, we can afford to store deltas for 100-200 blocks simultaneously. Note that if we store a value instead of value ref, size of FSDs can potentially be much larger. Because value limit is 4 MiB, we can’t apply previous argument about base cost. Since `wasm_storage_write_value_byte` = 31 Mgas, values contribution to FSD size can be estimated as `(1300 Tgas / storage_write_value_byte * num_shards)`, or ~167 MiB. Same estimation for trie keys gives 54 MiB. The advantage of storing values instead of value refs is that it saves one disk read if the key has been modified in the recent blocks. It may be beneficial if we get many transactions or receipts touching the same trie keys in consecutive blocks, but it is hard to estimate the value of such benefits without more data. We may store only short values ("inlining"), but this idea is orthogonal and can be applied separately. ### Protocol changes Flat Storage itself doesn't change protocol. We only change impacted storage costs to reflect changes in performance. Below we describe reads and writes separately. #### Storage Reads Latest proposal for shipping storage reads is [here](https://github.com/near/nearcore/issues/8006#issuecomment-1473718509). It solves several issues with costs, but the major impact of flat storage is that essentially for reads `wasm_touching_trie_node` and `wasm_read_cached_trie_node` are reduced to 0. Reason is that before we had to cover costs of reading nodes from memory or disk, and with flat storage we make only 2 DB reads. Latest up-to-date gas and compute costs can be found in nearcore repo. #### Storage Writes Storage writes are charged similarly to reads and include TTN as well, because updating the leaf trie node which stores the value to the trie key requires updating all trie nodes on the path leading to the leaf node. All writes are committed at once in one db transaction at the end of block processing, outside of runtime after all receipts in a block are executed. However, at the time of execution, runtime needs to calculate the cost, which means it needs to know how many trie nodes the write affects, so runtime will issue a read for every write to calculate the TTN cost for the write. Such reads cannot be replaced by a read in FlatStorage because FlatStorage does not provide the path to the trie node. There are multiple proposals on how storage writes can work with FlatStorage. - Keep it the same. The cost of writes remain the same. Note that this can increase the cost for writes in some cases, for example, if a contract first read from a key and then writes to the same key in the same chunk. Without FlatStorage, the key will be cached in the chunk cache after the read, so the write will cost less. With FlatStorage, the read will go through FlatStorage, the write will not find the key in the chunk cache and it will cost more. - Remove the TTN cost from storage write fees. Currently, there are two ideas in this direction. - Charge based on maximum depth of a contract’s state, instead of per-touch-trie node. - Charge based on key length only. Both of the above ideas would allow us to get rid of trie traversal ("reads-for-writes") from the critical path of block execution. However, it is unclear at this point what the new cost would look like and whether further optimizations are needed to bring down the cost for writes in the new cost model. See https://gov.near.org/t/storage-write-optimizations/30083 for more details. While storage writes are not fully implemented yet, we may increase parameter compute cost for storage writes implemented in https://github.com/near/NEPs/pull/455 as an intermediate solution. ### Migration Plan There are two main questions regarding to how to enable FlatStorage. 1. Whether there should be database migration. The main challenge of enabling FlatStorage will be to build the flat state column, which requires iterating the entire state. Estimations showed that it takes 10 hours to build flat state for archival nodes and 5 hours for rpc and validator nodes in 8 threads. The main concern is that if it takes too long for archival node to migrate, they may have a hard time catching up later since the block processing speed of archival nodes is not very fast. Alternatively, we can build the flat state in a background process while the node is running. This provides a better experience for both archival and validator nodes since the migration process is transient to them. It would require more implementation effort from our side. We currently proceed with background migration using 8 threads. 2. Whether there should be a protocol upgrade. The enabling of FlatStorage itself does not require a protocol upgrade, since it is an internal storage implementation that doesn't change protocol level. However, a protocol upgrade is needed if we want to adjust fees based on the storage performance with FlatStorage. These two changes can happen in one release, or we can be release them separately. We propose that the enabling of FlatStorage and the protocol upgrade to adjust fees should happen in separate release to reduce the risk. The period between the two releases can be used to test the stability and performance of FlatStorage. Because it is not a protocol change, it is easy to roll back the change in case any issue arises. ## Reference Implementation FlatStorage will implement the following structs. `FlatStorageChunkView`: interface for getting value or value reference from flat storage for specific shard, block hash and trie key. In current logic we plan to make it part of `Trie`, and all trie reads will be directed to this object. Though we could work with chunk hashes, we don't, because block hashes are easier to navigate. `FlatStorage`: API for interacting with flat storage for fixed shard, including updating head, adding new delta and creating `FlatStorageChunkView`s. for example, all block deltas that are stored in flat storage and the flat storage head. `FlatStorageChunkView` can access `FlatStorage` to get the list of deltas it needs to apply on top of state of current flat head in order to compute state of a target block. `FlatStorageManager`: owns flat storages for all shards, being stored in `NightshadeRuntime`, accepts updates from `Chain` side, caused by successful processing of chunk or block. `FlatStorageCreator`: handles flat storage structs creation or initiates background creation (aka migration process) if flat storage data is not presend on DB yet. `FlatStateDelta`: a HashMap that contains state changes introduced in a chunk. They can be applied on top the state at flat storage head to compute state at another block. The reason for having separate flat storages that there are two modes of block processing, normal block processing and block catchups. Since they are performed on different ranges of blocks, flat storage need to be able to support different range of blocks on different shards. Therefore, we separate the flat storage objects used for different shards. ### DB columns `DBCol::FlatState` stores a mapping from trie keys to the value corresponding to the trie keys, based on the state of the block at flat storage head. - _Rows_: trie key (`Vec`) - _Column type_: `ValueRef` `DBCol::FlatStateDeltas` stores all existing FSDs as mapping from `(shard_id, block_hash, trie_key)` to the `ValueRef`. To read the whole delta, we read all values for given key prefix. This delta stores all state changes introduced in the given shard of the given block. - _Rows_: `{ shard_id, block_hash, trie_key }` - _Column type_: `ValueRef` Note that `FlatStateDelta`s needed are stored in memory, so during block processing this column won't be used at all. This column is only used to load deltas into memory at `FlatStorage` initialization time when node starts. `DBCol::FlatStateMetadata` stores miscellaneous data about flat storage layout, including current flat storage head, current creation status and info about deltas existence. We don't specify exact format here because it is under discussion and can be tweaked until release. Similarly, flat head is also stored in `FlatStorage` in memory, so this column is only used to initialize `FlatStorage` when node starts. ### `FlatStateDelta` `FlatStateDelta` stores a mapping from trie keys to value refs. If the value is `None`, it means the key is deleted in the block. ```rust pub struct FlatStateDelta(HashMap, Option>); ``` ```rust pub fn from_state_changes(changes: &[RawStateChangesWithTrieKey]) -> FlatStateDelta ``` Converts raw state changes to flat state delta. The raw state changes will be returned as part of the result of `Runtime::apply_transactions`. They will be converted to `FlatStateDelta` to be added to `FlatStorage` during `Chain::postprocess_block` or `Chain::catch_up_postprocess`. ### `FlatStorageChunkView` `FlatStorageChunkView` will be created for a shard `shard_id` and a block `block_hash`, and it can perform key value lookup for the state of shard `shard_id` after block `block_hash` is applied. ```rust pub struct FlatStorageChunkView { /// Used to access flat state stored at the head of flat storage. store: Store, /// The block for which key-value pairs of its state will be retrieved. The flat state /// will reflect the state AFTER the block is applied. block_hash: CryptoHash, /// Stores the state of the flat storage flat_storage: FlatStorage, } ``` `FlatStorageChunkView` will provide the following interface. ```rust pub fn get_ref( &self, key: &[u8], ) -> Result, StorageError> ``` Returns the value or value reference corresponding to the given `key` for the state that this `FlatStorageChunkView` object represents, i.e., the state that after block `self.block_hash` is applied. ### `FlatStorageManager` `FlatStorageManager` will be stored as part of `ShardTries` and `NightshadeRuntime`. Similar to how `ShardTries` is used to construct new `Trie` objects given a state root and a shard id, `FlatStorageManager` is used to construct a new `FlatStorageChunkView` object given a block hash and a shard id. ```rust pub fn new_flat_storage_chunk_view( &self, shard_id: ShardId, block_hash: Option, ) -> FlatStorageChunkView ``` Creates a new `FlatStorageChunkView` to be used for performing key value lookups on the state of shard `shard_id` after block `block_hash` is applied. ```rust pub fn get_flat_storage( &self, shard_id: ShardId, ) -> Result ``` Returns the `FlatStorage` for the shard `shard_id`. This function is needed because even though `FlatStorage` is part of `NightshadeRuntime`, `Chain` also needs access to `FlatStorage` to update flat head. We will also create a function with the same in `NightshadeRuntime` that calls this function to provide `Chain` to access to `FlatStorage`. ```rust pub fn remove_flat_storage( &self, shard_id: ShardId, ) -> Result ``` Removes flat storage for shard if we stopped tracking it. ### `FlatStorage` `FlatStorage` is created per shard. It provides information to which blocks the flat storage on the given shard currently supports and what block deltas need to be applied on top the stored flat state on disk to get the state of the target block. ```rust fn get_blocks_to_head( &self, target_block_hash: &CryptoHash, ) -> Result, FlatStorageError> ``` Returns the list of deltas between blocks `target_block_hash` (inclusive) and flat head (exclusive), Returns an error if `target_block_hash` is not a direct descendent of the current flat head. This function will be used in `FlatStorageChunkView::get_ref`. Note that we can't call it once and store during applying chunk, because in parallel to that some block can be processed and flat head can be updated. ```rust fn update_flat_head(&self, new_head: &CryptoHash) -> Result<(), FlatStorageError> ``` Updates the head of the flat storage, including updating the flat head in memory and on disk, update the flat state on disk to reflect the state at the new head, and gc the `FlatStateDelta`s that are no longer needed from memory and from disk. ```rust fn add_block( &self, block_hash: &CryptoHash, delta: FlatStateDelta, ) -> Result ``` Adds `delta` to `FlatStorage`, returns a `StoreUpdate` object that includes DB transaction to be committed to persist that change. ```rust fn get_ref( &self, block_hash: &CryptoHash, key: &[u8], ) -> Result, FlatStorageError> ``` Returns `ValueRef` from flat storage state on top of `block_hash`. Returns `None` if key is not present, or an error if block is not supported. ### Thread Safety We should note that the implementation of `FlatStorage` must be thread safe because it can be concurrently accessed by multiple threads. A node can process multiple blocks at the same time if they are on different forks, and chunks from these blocks can trigger storage reads in parallel. Therefore, `FlatStorage` will be guarded by a `RwLock` so its access can be shared safely: ```rust pub struct FlatStorage(Arc>); ``` ## Drawbacks The main drawback is that we need to control total size of state updates in blocks after current final head. current testnet/mainnet load amount of blocks under final head doesn't exceed 5 in 99.99% cases, we still have to consider extreme cases, because Doomslug consensus doesn't give guarantees / upper limit on that. If we don't consider this at all and there is no finality for a long time, validator nodes can crash because of too many FSDs of memory, and chain slows down and stalls, which can have a negative impact on user/validator experience and reputation. For now, we claim that we support enough deltas in memory for chain to be finalized, and the proper discussions are likely to happen in NEPs like https://github.com/near/NEPs/pull/460. Risk of DB corruption slightly increases, and it becomes harder to replay blocks on chain. While `Trie` entries are essentially immutable (in fact, value for each key is unique, because key is a value hash), `FlatStorage` is read-modify-write, because values for the same `TrieKey` can be completely different. We believe that such flat mapping is reasonable to maintain anyway, as for newly discovered state sync idea. But if some change was applied incorrectly, we may have to recompute the whole flat storage, and for block hashes before flat head we can't access flat storage at all. Though Flat Storage significantly reduces amount of storage reads, we have to keep it up-to-date, which results in 1 extra disk write for changed key, and 1 auxiliary disk write + removal for each FSD. Disk requirements also slightly increase. We think it is acceptable, because actual disk writes are executed in background and are not a bottleneck for block processing. For storage write in general Flat Storage is even a net improvement, because it removes necessity to traverse changed nodes during write execution ("reads-for-writes"), and we can apply optimizations there (see "Storage Writes" section). Implementing FlatStorage will require a lot of engineering effort and introduce code that will make the codebase more complicated. In particular, we had to extend `RuntimeAdapter` API with flat storage-related method after thorough considerations. We are confident that FlatStorage will bring a lot of performance benefit, but we can only measure the exact improvement after the implementation. We may find that the benefit FlatStorage brings is not worth the effort, but it is very unlikely. It will make the state rollback harder in the future when we enable challenges in phase 2 of sharding. When a challenge is accepted and the state needs to be rolled back to a previous block, the entire flat state needs to be rebuilt, which could take a long time. Alternatively, we could postpone garbage collection of deltas and add support of applying them backwards. Speaking of new sharding phases, once nodes are no longer tracking all shards, Flat Storage must have support for adding or removing state for some specific shard. Adding new shard is a tricky but natural extension of catchup process. Our current approach for removal is to iterate over all entries in `DBCol::FlatState` and find out for each trie key to which shard it belongs to. We would be happy to assume that each shard is represented by set of contiguous ranges in `DBCol::FlatState` and make removals simpler, but this is still under discussion. Last but not least, resharding is not supported by current implementation yet. ## Future possibilities Flat Storage maintains all state keys in sorted order, which seems beneficial. We currently investigate opportunity to speed up state sync: instead of traversing state part in Trie, we can extract range of keys and values from Flat Storage and build range of Trie nodes based on it. It is well known that reading Trie nodes is a bottleneck for state sync as well. ## Changelog ### 1.0.0 - Initial Version The NEP was approved by Protocol Working Group members on March 16, 2023 ([meeting recording](https://www.youtube.com/watch?v=4VxRoKwLXIs)): - [Bowen's vote](https://github.com/near/NEPs/pull/399#issuecomment-1467010125) - [Marcelo's vote](https://github.com/near/NEPs/pull/399#pullrequestreview-1341069564) - [Marcin's vote](https://github.com/near/NEPs/pull/399#issuecomment-1465977749) #### Benefits - The proposal makes serving reads more efficient; making the NEAR protocol cheaper to use and increasing the capacity of the network; - The proposal makes estimating gas costs for a transaction easier as the fees for reading are no longer a function of the trie structure whose shape the smart contract developer does not know ahead of time and can continuously change. - The proposal should open doors to enabling future efficiency gains in the protocol and further simplifying gas fee estimations. - 'Secondary' index over the state data - which would allow further optimisations in the future. #### Concerns | # | Concern | Resolution | Status | | --- | ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | | 1 | The cache requires additional database storage | There is an upper bound on how much additional storage is needed. The costs for the additional disk storage should be negligible | Not an issue | | 2 | Additional implementation complexity | Given the benefits of the proposal, I believe the complexity is justified | not an issue | | 3 | Additional memory requirement | Most node operators are already operating over-provisioned machines which can handle the additional memory requirement. The minimum requirements should be raised but it appears that minimum requirements are already not enough to operate a node | This is a concern but it is not specific to this project | | 4 | Slowing down the read-update-write workload | This is common pattern in smart contracts so indeed a concern. However, there are future plans on how to address this by serving writes from the flat storage as well which will also reduce the fees of serving writes and make further improvements to the NEAR protocol | This is a concern but hopefully will be addressed in future iterations of the project | ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0408.md ================================================ --- NEP: 408 Title: Injected Wallet API Author: Daryl Collins <@MaximusHaximus>, @lewis-sqa Status: Final DiscussionsTo: https://github.com/near/NEPs/pull/408, https://github.com/near/NEPs/pull/370 Type: Standards Track Category: Wallet Created: 10-Oct-2022 --- # Injected Wallets ## Summary Standard interface for injected wallets. ## Motivation dApps are finding it increasingly difficult to support the ever expanding choice of wallets due to their wildly different implementations. While projects such as [Wallet Selector](https://github.com/near/wallet-selector) attempt to mask this problem, it's clear the ecosystem requires a standard that will not only benefit dApps but make it easier for established wallets to support NEAR. ## Rationale and alternatives At its most basic, a wallet contains key pairs required to sign messages. This standard aims to define an API (based on our learning from [Wallet Selector](https://github.com/near/wallet-selector)) that achieves this requirement through a number of methods exposed on the `window` object. The introduction of this standard makes it possible for `near-api-js` to become wallet-agnostic and eventually move away from the high amount of coupling with NEAR Wallet. It simplifies projects such as [Wallet Selector](https://github.com/near/wallet-selector) that must implement various abstractions to normalise the different APIs before it can display a modal for selecting a wallet. This standard takes a different approach to a wallet API than other blockchains such as [Ethereum's JSON-RPC Methods](https://docs.metamask.io/guide/rpc-api.html#ethereum-json-rpc-methods). Mainly, it rejects the `request` abstraction that feels unnecessary and only adds to the complexity both in terms of implementation and types. Instead, it exposes various methods directly on the top-level object that also improves discoverability. There have been many iterations of this standard to help inform what we consider the "best" approach right now for NEAR. Below is a summary of the key design choices: ### Single account vs. multiple account Almost every wallet implementation in NEAR used a single account model until we began integrating with [WalletConnect](https://walletconnect.com/). In WalletConnect, sessions can contain any number of accounts that can be modified by the dApp or wallet. The decision to use a multiple account model was influenced by the following reasons: - It future-proofs the API even if wallets (such as MetaMask) only support a single "active" account. - Other blockchains such as [Ethereum](https://docs.metamask.io/guide/rpc-api.html#eth-requestaccounts) implement this model. - Access to multiple accounts allow dApps more freedom to improve UX as users can seamlessly switch between accounts. - Aligns with WalletConnect via the [Bridge Wallet Standard](https://github.com/near/NEPs/blob/master/neps/nep-0368.md). ### Storage of key pairs for FunctionCall access keys in dApp context vs. wallet context - NEAR's unique concept of `FunctionCall` access keys allow for the concept of 'signing in' to a dApp using your wallet. 'Signing In' to a dApp is accomplished by adding `FunctionCall` type access key that the dApp owns to the account that the user is logging in as. - Once a user has 'signed in' to a dApp, the dApp can then use the keypair that it owns to execute transactions without having to prompt the user to route and approve those transactions through their wallet. - `FunctionCall` access keys have a limited quota that can only be used to pay for gas fees (typically 0.25 NEAR) and can further be restricted to only be allowed to call _specific methods_ on one **specific** smart contract. - This allows for an ideal user experience for dApps that require small gas-only transactions regularly while in use. Those transactions can be done without interrupting the user experience by requiring them to be approved through their wallet. A great example of this is evident in gaming use-cases -- take a gaming dApp where some interactions the user makes must write to the blockchain as they do common actions in the game world. Without the 'sign in' concept that provides the dApp with its own limited usage key, the user might be constantly interrupted by needing to approve transactions on their wallet as they perform common actions. If a player has their account secured with a ledger, the gameplay experience would be constantly interrupted by prompts to approve transactions on their ledger device! With the 'sign in' concept, the user will only intermittently need to approve transactions to re-sign-in, when the quota that they approved for gas usage during their last login has been used up. - Generally, it is recommended to only keep `FullAccess` keys in wallet scope and hidden from the dApp consumer. `FunctionCall` type keys should be generated and owned by the dApp, and requested to be added using the `signIn` method. They should **not** be 'hidden' inside the wallet in the way that `FullAccess` type keys are. ## Specification Injected wallets are typically browser extensions that implement the `Wallet` API (see below). References to the currently available wallets are tracked on the `window` object. To avoid namespace collisions and easily detect when they're available, wallets must mount under their own key of the object `window.near` (e.g. `window.near.sender`). **NOTE: Do not replace the entire `window.near` object with your wallet implementation, or add any objects as properties of the `window.near` object that do not conform to the Injected Wallet Standard** At the core of a wallet are [`signTransaction`](#signtransaction) and [`signTransactions`](#signtransactions). These methods, when given a [`TransactionOptions`](#wallet-api) instance, will prompt the user to sign with a key pair previously imported (with the assumption it has [`FullAccess`](https://nomicon.io/DataStructures/AccessKey) permission). In most cases, a dApp will need a reference to an account and associated public key to construct a [`Transaction`](https://nomicon.io/RuntimeSpec/Transactions). The [`connect`](#connect) method helps solve this issue by prompting the user to select one or more accounts they would like to make visible to the dApp. When at least one account is visible, the wallet considers the dApp [`connected`](#connected) and they can access a list of [`accounts`](#accounts) containing an `accountId` and `publicKey`. For dApps that often sign gas-only transactions, [`FunctionCall`](https://nomicon.io/DataStructures/AccessKey#accesskeypermissionfunctioncall) access keys can be added/deleted for one or more accounts using the [`signIn`](#signin) and [`signOut`](#signout) methods. While this functionality could be achieved with [`signTransactions`](#signtransactions), it suggests a direct intention that a user wishes to sign in/out of a dApp's smart contract. ### Wallet API Below is the entire API for injected wallets. It makes use of `near-api-js` to enable interoperability with dApps that will already use it for constructing transactions and communicating with RPC endpoints. ```ts import { transactions, utils } from "near-api-js"; interface Account { accountId: string; publicKey: utils.PublicKey; } interface Network { networkId: string; nodeUrl: string; } interface SignInParams { permission: transactions.FunctionCallPermission; account: Account; } interface SignInMultiParams { permissions: Array; account: Account; } interface SignOutParams { accounts: Array; } interface TransactionOptions { receiverId: string; actions: Array; signerId?: string; } interface SignTransactionParams { transaction: TransactionOptions; } interface SignTransactionsParams { transactions: Array; } interface Events { accountsChanged: { accounts: Array }; } interface ConnectParams { networkId: string; } type Unsubscribe = () => void; interface Wallet { id: string; connected: boolean; network: Network; accounts: Array; supportsNetwork(networkId: string): Promise; connect(params: ConnectParams): Promise>; signIn(params: SignInParams): Promise; signInMulti(params: SignInMultiParams): Promise; signOut(params: SignOutParams): Promise; signTransaction( params: SignTransactionParams ): Promise; signTransactions( params: SignTransactionsParams ): Promise>; disconnect(): Promise; on( event: EventName, callback: (params: Events[EventName]) => void ): Unsubscribe; off( event: EventName, callback?: () => void ): void; } ``` #### Properties ##### `id` Retrieve the wallet's unique identifier. ```ts const { id } = window.near.wallet; console.log(id); // "wallet" ``` ##### `connected` Determine whether we're already connected to the wallet and have visibility of at least one account. ```ts const { connected } = window.near.wallet; console.log(connected); // true ``` ##### `network` Retrieve the currently selected network. ```ts const { network } = window.near.wallet; console.log(network); // { networkId: "testnet", nodeUrl: "https://rpc.testnet.near.org" } ``` ##### `accounts` Retrieve all accounts visible to the dApp. ```ts const { accounts } = window.near.wallet; console.log(accounts); // [{ accountId: "test.testnet", publicKey: PublicKey }] ``` #### Methods ##### `connect` Request visibility for one or more accounts from the wallet. This should explicitly prompt the user to select from their list of imported accounts. dApps can use the `accounts` property once connected to retrieve the list of visible accounts. > Note: Calling this method when already connected will allow users to modify their selection, triggering the 'accountsChanged' event. ```ts const accounts = await window.near.wallet.connect(); ``` ##### `signTransaction` Sign a transaction. This request should require explicit approval from the user. ```ts import { transactions, providers, utils } from "near-api-js"; // Retrieve accounts (assuming already connected) and current network. const { network, accounts } = window.near.wallet; // Setup RPC to retrieve transaction-related prerequisites. const provider = new providers.JsonRpcProvider({ url: network.nodeUrl }); const signedTx = await window.near.wallet.signTransaction({ transaction: { signerId: accounts[0].accountId, receiverId: "guest-book.testnet", actions: [ transactions.functionCall( "addMessage", { text: "Hello World!" }, utils.format.parseNearAmount("0.00000000003"), utils.format.parseNearAmount("0.01") ), ], }, }); // Send the transaction to the blockchain. await provider.sendTransaction(signedTx); ``` ##### `signTransactions` Sign a list of transactions. This request should require explicit approval from the user. ```ts import { transactions, providers, utils } from "near-api-js"; // Retrieve accounts (assuming already connected) and current network. const { network, accounts } = window.near.wallet; // Setup RPC to retrieve transaction-related prerequisites. const provider = new providers.JsonRpcProvider({ url: network.nodeUrl }); const signedTxs = await window.near.wallet.signTransactions({ transactions: [ { signerId: accounts[0].accountId, receiverId: "guest-book.testnet", actions: [ transactions.functionCall( "addMessage", { text: "Hello World! (1/2)" }, utils.format.parseNearAmount("0.00000000003"), utils.format.parseNearAmount("0.01") ), ], }, { signerId: accounts[0].accountId, receiverId: "guest-book.testnet", actions: [ transactions.functionCall( "addMessage", { text: "Hello World! (2/2)" }, utils.format.parseNearAmount("0.00000000003"), utils.format.parseNearAmount("0.01") ), ], }, ], }); for (let i = 0; i < signedTxs.length; i += 1) { const signedTx = signedTxs[i]; // Send the transaction to the blockchain. await provider.sendTransaction(signedTx); } ``` ##### `disconnect` Remove visibility of all accounts from the wallet. ```ts await window.near.wallet.disconnect(); ``` ##### `signIn` Add one `FunctionCall` access key for one or more accounts. This request should require explicit approval from the user. ```ts import { utils } from "near-api-js"; // Retrieve the list of accounts we have visibility of. const { accounts } = window.near.wallet; // Request FunctionCall access to the 'guest-book.testnet' smart contract for each account. await window.near.wallet.signIn({ permission: { receiverId: "guest-book.testnet", methodNames: [], }, account: { accountId: accounts[0].accountId, publicKey: utils.KeyPair.fromRandom("ed25519").getPublicKey(), }, }); ``` ##### `signInMulti` Add multiple `FunctionCall` access keys for one or more accounts. This request should require explicit approval from the user. ```ts import { utils } from "near-api-js"; // Retrieve the list of accounts we have visibility of. const { accounts } = window.near.wallet; // Request FunctionCall access to the 'guest-book.testnet' and 'guest-book2.testnet' smart contract for each account. await window.near.wallet.signInMulti({ permissions: [ { receiverId: "guest-book.testnet", methodNames: [], }, { receiverId: "guest-book2.testnet", methodNames: [], }, ], account: { accountId: accounts[0].accountId, publicKey: utils.KeyPair.fromRandom("ed25519").getPublicKey(), }, }); ``` ##### Benefits This NEP will optimize UX for multi contract DApps and avoid multiple redirects. These are more and more common in the ecosystem and this NEP will benefit the UX for those DApps. ##### Concerns - The currently available keystores will have to catch up in order to support multiple keys per account - We should add the new method to the Wallet interface for clarity in the NEP doc ##### `signOut` Delete `FunctionCall` access key(s) for one or more accounts. This request should require explicit approval from the user. ```ts import { utils, keyStores } from "near-api-js"; // Setup keystore to retrieve locally stored FunctionCall access keys. const keystore = new keyStores.BrowserLocalStorageKeyStore(); // Retrieve accounts (assuming already connected) and current network. const { network, accounts } = window.near.wallet; // Remove FunctionCall access (previously granted via signIn) for each account. await window.near.wallet.signOut({ accounts: await Promise.all( accounts.map(async ({ accountId }) => { const keyPair = await keystore.getKey(network.networkId, accountId); return { accountId, publicKey: keyPair.getPublicKey(), }; }) ), }); ``` #### Events ##### `accountsChanged` Triggered whenever accounts are updated (e.g. calling `connect` or `disconnect`). ```ts window.near.wallet.on("accountsChanged", ({ accounts }) => { console.log("Accounts Changed", accounts); }); ``` ================================================ FILE: neps/nep-0413.md ================================================ --- NEP: 413 Title: Near Wallet API - support for signMessage method Author: Philip Obosi , Guillermo Gallardo Status: Final # DiscussionsTo: Type: Standards Track Category: Wallet Created: 25-Oct-2022 --- ## Summary A standardized Wallet API method, namely `signMessage`, that allows users to sign a message for a specific recipient using their NEAR account. ## Motivation NEAR users want to create messages destined to a specific recipient using their accounts. This has multiple applications, one of them being authentication in third-party services. Currently, there is no standardized way for wallets to sign a message destined to a specific recipient. ## Rationale and Alternatives Users want to sign messages for a specific recipient without incurring in GAS fees, nor compromising their account's security. This means that the message being signed: 1. Must be signed off-chain, with no transactions being involved. 2. Must include the recipient's name and a nonce. 3. Cannot represent a valid transaction. 4. Must be signed using a Full Access Key. 5. Should be simple to produce/verify, and transmitted securely. ### Why Off-Chain? So the user would not incur in GAS fees, nor the signed message gets broadcasted into a public network. ### Why The Message MUST NOT be a Transaction? How To Ensure This? An attacker could make the user inadvertently sign a valid transaction which, once signed, could be submitted into the network to execute it. #### How to Ensure the Message is not a Transaction In NEAR, transactions are encoded in Borsh before being signed. The first attribute of a transaction is a `signerId: string`, which is encoded as: (1) 4 bytes representing the string's length, (2) N bytes representing the string itself. By prepending the prefix tag $2^{31} + 413$ we can both ensure that (1) the whole message is an invalid transaction (since the string would be too long to be a valid signer account id), (2) this NEP is ready for a potential future protocol update, in which non-consensus messages are tagged using $2^{31}$ + NEP-number. ### Why The Message Needs to Include a Receiver and Nonce? To stop a malicious app from requesting the user to sign a message for them, only to relay it to a third-party. Including the recipient and making sure the user knows about it should mitigate these kind of attacks. Meanwhile, including a nonce helps to mitigate replay attacks, in which an attacker can delay or re-send a signed message. ### Why using a FullAccess Key? Why Not Simply Creating an [FunctionCall Key](https://docs.near.org/protocol/access-keys) for Signing? The most common flow for NEAR user authentication into a Web3 frontend involves the creation of a [FunctionCall Key](https://docs.near.org/protocol/access-keys). One might feel tempted to reproduce such process here, for example, by creating a key that can only be used to call a non-existing method in the user's account. This is a bad idea because: 1. The user would need to expend gas in creating a new key. 2. Any third-party can ask the user to create a `FunctionCall Key`, thus opening an attack vector. Using a FullAccess key allows us to be sure that the challenge was signed by the user (since nobody should have access to their `FullAccess Key`), while keeping the constraints of not expending gas in the process (because no new key needs to be created). ### Why The Input Needs to Include a State? Including a state helps to mitigate [CSRF attacks](https://auth0.com/docs/secure/attack-protection/state-parameters). This way, if a message needs to be signed for authentication purposes, the auth service can keep a state to make sure the auth request comes from the right author. ### How to Return the Signed Message in a Safe Way Sending the signed message in a query string to an arbitrary URL (even within the correct domain) is not secure as the data can be leaked (e.g. through headers, etc). Using URL fragments instead will improve security, since [URL fragments are not included in the `Referer`](https://greenbytes.de/tech/webdav/rfc2616.html#header.referer). ### NEAR Signatures NEAR transaction signatures are not plain Ed25519 signatures but Ed25519 signatures of a SHA-256 hash (see [near/nearcore#2835](https://github.com/near/nearcore/issues/2835)). Any protocol that signs anything with NEAR account keys should use the same signature format. ## Specification Wallets must implement a `signMessage` method, which takes a `message` destined to a specific `recipient` and transform it into a verifiable signature. ### Input Interface `signMessage` must implement the following input interface: ```jsx interface SignMessageParams { message: string ; // The message that wants to be transmitted. recipient: string; // The recipient to whom the message is destined (e.g. "alice.near" or "myapp.com"). nonce: [u8; 32] ; // A nonce that uniquely identifies this instance of the message, denoted as a 32 bytes array (a fixed `Buffer` in JS/TS). callbackUrl?: string; // Optional, applicable to browser wallets (e.g. MyNearWallet). The URL to call after the signing process. Defaults to `window.location.href`. state?: string; // Optional, applicable to browser wallets (e.g. MyNearWallet). A state for authentication purposes. } ``` ### Structure `signMessage` must embed the input `message`, `recipient` and `nonce` into the following predefined structure: ```rust struct Payload { message: string; // The same message passed in `SignMessageParams.message` nonce: [u8; 32]; // The same nonce passed in `SignMessageParams.nonce` recipient: string; // The same recipient passed in `SignMessageParams.recipient` callbackUrl?: string // The same callbackUrl passed in `SignMessageParams.callbackUrl` } ``` ### Signature In order to create a signature, `signMessage` must: 1. Create a `Payload` object. 2. Convert the `payload` into its [Borsh Representation](https://borsh.io). 3. Prepend the 4-bytes borsh representation of $2^{31}+413$, as the [prefix tag](https://github.com/near/NEPs/pull/461). 4. Compute the `SHA256` hash of the serialized-prefix + serialized-tag. 5. Sign the resulting `SHA256` hash from step 3 using a **full-access** key. > If the wallet does not hold any `full-access` keys, then it must return an error. ### Example Assuming that the `signMessage` method was invoked, and that: - The input `message` is `"hi"` - The input `nonce` is `[0,...,31]` - The input `recipient` is `"myapp.com"` - The callbackUrl is `"myapp.com/callback"` - The wallet stores a full-access private key The wallet must construct and sign the following `SHA256` hash: ```jsx // 2**31 + 413 == 2147484061 sha256.hash(Borsh.serialize(2147484061) + Borsh.serialize(Payload{message:"hi", nonce:[0,...,31], recipient:"myapp.com", callbackUrl: "myapp.com/callback"})) ``` ### Output Interface `signMessage` must return an object containing the **base64** representation of the `signature`, and all the data necessary to verify such signature. ```jsx interface SignedMessage { accountId: string; // The account name to which the publicKey corresponds as plain text (e.g. "alice.near") publicKey: string; // The public counterpart of the key used to sign, expressed as a string with format ":" (e.g. "ed25519:6TupyNrcHGTt5XRLmHTc2KGaiSbjhQi1KHtCXTgbcr4Y") signature: string; // The base64 representation of the signature. state?: string; // Optional, applicable to browser wallets (e.g. MyNearWallet). The same state passed in SignMessageParams. } ``` ### Returning the signature #### Web Wallets Web Wallets, such as [MyNearWallet](https://mynearwallet.com), should directly return the `SignedMessage` to the `SignMessageParams.callbackUrl`, passing the `accountId`,`publicKey`, `signature` and the state as URL fragments. This is: `#accountId=&publicKey=&signature=&state=`. If the signing process fails, then the wallet must return an error message and the state as string fragments: `#error=&state=`. #### Other Wallets Non-web Wallets, such as [Ledger](https://www.ledger.com) can directly return the `SignedMessage` (in preference as a JSON object) and raise an error on failure. ## References A full example on how to implement the `signMessage` method can be [found here](https://github.com/gagdiez/near-login/blob/1650e25080ab2e8a8c508638a9ba9e9732e76036/server/tests/wallet.ts#L60-L77). ## Drawbacks Accounts that do not hold a FullAccess Key will not be able to sign this kind of messages. However, this is a necessary tradeoff for security since any third-party can ask the user to create a FunctionAccess key. At the time of writing this NEP, the NEAR ledger app is unable to sign this kind of messages, since currently it can only sign pure transactions. This however can be overcomed by modifying the NEAR ledger app implementation in the near future. Non-expert subjects could use this standard to authenticate users in an unsecure way. To anyone implementing an authentication service, we urge them to read about [CSRF attacks](https://auth0.com/docs/secure/attack-protection/state-parameters), and make use of the `state` field. ## Decision Context ### 1.0.0 - Initial Version The Wallet Standards Working Group members approved this NEP on January 17, 2023 ([meeting recording](https://youtu.be/Y6z7lUJSUuA)). ### 1.1.0 - First Revison Important Security concerns were raised by a community member, driving us to change the proposed implementation. ### Benefits - Makes it possible to authenticate users without having to add new access keys. This will improve UX, save money and will not increase the on-chain storage of the users' accounts. - Makes it possible to authorize through jwt in web2 services using the NEAR account. - Removes delays in adding transactions to the blockchain and makes the experience of using projects like NEAR Social better. ### Concerns | # | Concern | Resolution | Status | | --- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | | 1 | Implementing the signMessage standard will divide wallets into those that will quickly add support for it and those that will take significantly longer. In this case, some services may not work correctly for some users | (1) Be careful when adding functionality with signMessage with legacy and ensure that alternative authorization methods are possible. For example by adding publicKey. (2) Oblige wallets to implement a standard in specific deadlines to save their support in wallet selector | Resolved | | 2 | Large number of off-chain transactions will reduce activity in the blockchain and may negatively affect NEAR rate and attractiveness to third-party developers | There seems to be a general agreement that it is a good default | Resolved | | 3 | `receiver` terminology can be misleading and confusing when existing functionality is taken into consideration (`signTransaction`) | It was recommended for the community to vote for a new name, and the NEP was updated changing `receiver` to `recipient` | Resolved | | 4 | The NEP should emphasize that `nonce` and `receiver` should be clearly displayed to the user in the signing requests by wallets to achieve the desired security from these params being included | We strongly recommend the wallet to clearly display all the elements that compose the message being signed. However, this pertains to the wallet's UI and UX, and not to the method's specification, thus the NEP was not changed. | Resolved | | 5 | NEP-408 (Injected Wallet API) should be extended with this new `signMessage` method | It is not a blocker for this NEP, but a follow-up NEP-extension proposal is welcome. | Resolved | ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0418.md ================================================ --- NEP: 418 Title: Remove attached_deposit view panic Author: Austin Abell Status: Final DiscussionsTo: https://github.com/nearprotocol/neps/pull/418 Type: Standards Track Category: Tools Version: 1.0.0 Created: 18-Oct-2022 Updated: 27-Jan-2023 --- ## Summary This proposal is to switch the behavior of the `attached_deposit` host function on the runtime from panicking in view contexts to returning 0. This results in a better devX because instead of having to configure an assertion that there was no attached deposit to a function call only for transactions and not view calls, which is impossible because you can send a transaction to any method, you could just apply this assertion without the runtime aborting in view contexts. ## Motivation This will allow contract SDK frameworks to add the `attached_deposit == 0` assertion for every function on a contract by default. This behavior matches the Solidity/Eth payable modifier and will ensure that funds aren't sent accidentally to a contract in more cases than currently possible. This can't be done at a contract level because there is no way of checking if a function call is within view context to call `attached_deposit` conditionally. This means that there is no way of restricting the sending of funds to functions intended to be view only because the abort from within `attached_deposit` can't be caught and ignored from inside the contract. Initial discussion: https://near.zulipchat.com/#narrow/stream/295306-pagoda.2Fcontract-runtime/topic/attached_deposit.20view.20error ## Rationale and alternatives The rationale for assigning `0u128` to the pointer (`u64`) passed into `attached_deposit` is that it's the least breaking change. The alternative of returning some special value, say `u128::MAX`, is that it would cause some unintended side effects for view calls using the `attached_deposit`. For example, if `attached_deposit` is called within a function, older versions of a contract that do not check the special value will return a result assuming that the attached deposit is `u128::MAX`. This is not a large concern since it would just be a view call, but it might be a bad UX in some edge cases, where returning 0 wouldn't be an issue. ## Specification The error inside `attached_deposit` for view calls will be removed, and for all view calls, `0u128` will be set at the pointer passed in. ## Reference Implementation Currently, the implementation for `attached_deposit` is as follows: ```rust pub fn attached_deposit(&mut self, balance_ptr: u64) -> Result<()> { self.gas_counter.pay_base(base)?; if self.context.is_view() { return Err(HostError::ProhibitedInView { method_name: "attached_deposit".to_string(), } .into()); } self.memory_set_u128(balance_ptr, self.context.attached_deposit) } ``` Which would just remove the check for `is_view` to no longer throw an error: ```rust pub fn attached_deposit(&mut self, balance_ptr: u64) -> Result<()> { self.gas_counter.pay_base(base)?; self.memory_set_u128(balance_ptr, self.context.attached_deposit) } ``` This assumes that in all cases, `self.context.attached_deposit` is set to 0 in all cases. This can be asserted, or just to be safe, can check if `self.context.is_view()` and set `0u128` explicitly. ## Security Implications This won't have any implications outside of view calls, so this will not affect anything that is persisted on-chain. This only affects view calls. This can only have a negative side effect if a contract is under the assumption that `attached_deposit` will panic in view contexts. The possibility that this is done _and_ has some value connected with a view call result off-chain seems extremely unlikely. ## Drawbacks This has a breaking change of the functionality of `attached_deposit` and affects the behavior of some function calls in view contexts if they use `attached_deposit` and no other prohibited host functions. ## Future possibilities - The Rust SDK, as well as other SDKs, can add the `attached_deposit() == 0` check by default to all methods for safety of use. - Potentially, other host functions can be allowed where reasonable values can be inferred. For example, `prepaid_gas`, `used_gas` could return 0. ## Decision Context ### 1.0.0 - Initial Version The initial version of NEP-418 was approved by Tools Working Group members on January 19, 2023 ([meeting recording](https://youtu.be/poVmblmc3L4)). #### Benefits - This will allow contract SDK frameworks to add the `attached_deposit == 0` assertion for every function on a contract by default. - This behavior matches the Solidity/Eth payable modifier and will ensure that funds aren't sent accidentally to a contract in more cases than currently possible. - Given that there is no way of checking if a function call is within view context to call `attached_deposit` conditionally, this NEP only changes a small surface of the API instead of introducing a new host function. #### Concerns | # | Concern | Resolution | Status | | - | - | - | - | | 1 | Proposal potentially triggers the protocol version change | It does not trigger the protocol version change. Current update could be considered a client-breaking change update. | Resolved | | 2 | The contract can assume that `attached_deposit` will panic in view contexts. | The possibility that this is done _and_ has some value connected with a view call result off-chain seems extremely unlikely. | Won't Fix | | 3 | Can we assume that in all view calls, the `attached_deposit` in the VMContext always zero? | Yes, there is no way to set `attached_deposit` in view calls context | Resolved | ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0448.md ================================================ --- NEP: 448 Title: Zero-balance Accounts Author: Bowen Wang Status: Final DiscussionsTo: https://github.com/nearprotocol/neps/pull/448 Type: Protocol Track Created: 10-Jan-2023 --- ## Summary A major blocker to a good new user onboarding experience is that users have to acquire NEAR tokens to pay for their account. With the implementation of [NEP-366](https://github.com/near/NEPs/pull/366), users don't necessarily have to first acquire NEAR tokens in order to pay transaction fees, but they still have to pay for the storage of their account. To address this problem, we propose allowing each account to have free storage for the account itself and up to four keys and account for the cost of storage in the gas cost of create account transaction. ## Motivation Ideally a new user should be able to onboard onto NEAR through any applications built on top of NEAR and do not have to understand that the application is running on top of blockchain. The ideal flow is as follows: a user hear about an interesting application from their friends or some forum and they decide to give it a try. The user opens the application in their browser and directly starts using it without worrying about registration. Under the hood, a keypair is generated for the user and the application creates an account for the user and pays for transaction fees through meta transactions. Later on, the user may find other applications that they are also interested in and give them a try as well. At some point, the user graduates from the onboarding experience by acquiring NEAR tokens either through earning or because they like some experience so much that they would like to pay for it explicitly. Overall we want to have two full access keys for recovery purposes and two function call access keys so that users can use two apps before graduating from the onboarding experience. ## Rationale and alternatives There are a few alternative ideas: - Completely disregard storage staking and do not change the account creation cost. This makes the implementation even simpler. However, there may be a risk of spamming attack given that the cost of creating an account is around 0.2Tgas. In addition, with the current design, it is easy to further reduce the cost. Going the other way is more difficult. - Do not change how storage staking is calculated when converting to gas cost. This means that account creation cost would be around 60Tgas, which is both high in gas (meaning that the throughput is limited and more likely for some contract to break) and more costly for users (around 0.006N per account creation). ## Specification There are two main changes to the protocol: - Account creation cost needs to be increased. For every account, at creation time, 770 bytes of storage are reserved for the account itself + four full access keys + two function call access keys. For function call access keys, the "free" ones cannot use `method_names` in order to minimize the storage requirement for an account. The number of bytes is calculated as follows: _ An account takes 100 bytes due to `storage_usage_config.num_bytes_account` _ A full access key takes 42 bytes and there is an additional 40 bytes required due to `storage_usage_config.num_extra_bytes_record` _ A function call access key takes 131 bytes and there is an additional 40 bytes required due to `storage_usage_config.num_extra_bytes_record` _ Therefore the total the number of bytes is `100 + (131 + 40) * 2 + (42 + 40) * 4 = 770`. The cost of these bytes is paid through transaction fee. Note that there is already [discussion](https://github.com/near/NEPs/issues/415) around the storage cost of NEAR and whether it is reasonable. While this proposal does not attempt to change the entire storage staking mechanism, the cost of storage is reduced in 10x when converting to gas. A [discussion](https://gov.near.org/t/storage-staking-price/399) from a while ago mentioned this idea, and the concerns there were proven to be not real concerns. No one is deleting data from storage in practice and the storage staking mechanism does not really serve its purpose. That conversion means we increase the account creation cost to 7.7Tgas from 0.2Tgas - Storage staking check will not be applied if an account has <= 4 full access keys and <= 2 function call access keys and does not have a contract deployed. If an account accrues more than full access keys or function call access keys, however, it must pay for the storage of everything including those 6 keys. This makes the implementation simpler and less error-prone. ## Reference Implementation (Required for Protocol Working Group proposals, optional for other categories) Details of the changes described in the section above: - Change `create_account_cost` to ```json "create_account_cost": { "send_sir": 3850000000000, "send_not_sir": 3850000000000, "execution": 3850000000000 }, ``` - Change the implementation of `get_insufficient_storage_stake` to check whether an account is zero balance account. Note that even though the intent, as described in the section above, is to limit the number of full access keys to 4 and the number of function call access keys to 2, for the ease of implementation, it makes sense to limit the size of `storage_usage` on an account to 770 bytes because `storage_usage` is already stored under `Account` and it does not require any additional storage reads. More specifically, the check looks roughly as follows: ```rust /// Returns true if an account is a zero balance account fn check_for_zero_balance_account(account: &Account) -> bool { account.storage_usage <= 770 // 4 full access keys and 2 function call access keys } ``` ## Drawbacks (Optional) - Reduction of storage cost when converting the storage cost of zero balance accounts to gas cost may be a concern. But I argue that the current storage cost is too high. A calculation shows that the current storage cost is around 36,000 times higher than S3 storage cost. In addition, when a user accrues any contract data or has more than three keys on their account, they have to pay for the storage cost of everything combined. In that sense, a user would pay slightly more than what they pay today when their account is no longer a zero-balance account. ## Unresolved Issues (Optional) ## Future possibilities - We may change the number of keys allowed for zero-balance accounts in the future. - A more radical thought: we can separate out zero-balance accounts into its own trie and manage them separately. This may allow more customization on how we want zero-balance accounts to be treated. ## Decision Context ### 1.0.0 - Initial Version The initial version of NEP-448 was approved by Protocol Working Group members on February 9, 2023 ([meeting recording](https://youtu.be/ktgWXjNTU_A)). #### Benefits - Users can now onboard with having to acquire NEAR tokens - Together with [meta transactions](https://github.com/near/NEPs/pull/366), this allows a user to start interacting with an app on NEAR directly from their device without any additional steps - Solves the problem of meta transaction for implicit accounts #### Concerns | # | Concern | Resolution | Status | | --- | -------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | | 1 | The number of full access keys allowed is too small | Could be done in a future iteration. | Resolved | | 2 | No incentive for people to remove zero balance account. | Very few people actually delete their account anyways. | Resolved | | 3 | UX of requiring balance after a user graduate from zero balance account to a regular account | The experience of graduating from zero balance account should be handled on the product side | Resolved | | 4 | Increase of account creation cost may break some existing contracts | A thorough investigation has been done and it turns out that we only need to change the contract that is deployed on `near` slightly | Resolved | | 5 | Account creation speed is slower due to increased cost | Unlikely to be a concern, especially given that the number of shards is expected to grow in the future | Resolved | | 6 | Cost of transfers to implicit account increases | Unlikely to break anything at the moment, and could be addressed in the future in a different NEP (see https://github.com/near/NEPs/issues/462 for more details) | Resolved | ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0452.md ================================================ --- NEP: 452 Title: Linkdrop Standard Author: Ben Kurrek , Ken Miyachi Status: Final DiscussionsTo: https://gov.near.org/t/official-linkdrop-standard/32463/1 Type: Standards Track Category: Contract Version: 1.0.0 Created: 24-Jan-2023 Updated: 19-Apr-2023 --- ## Summary A standard interface for linkdrops that support $NEAR, fungible tokens, non-fungible tokens, and is extensible to support new types in the future. Linkdrops are a simple way to send assets to someone by providing them with a link. This link can be embedded into a QR code, sent via email, text or any other means. Within the link, there is a private key that allows the holder to call a method that can create an account and send it assets. Alternatively, if the holder has an account, the assets can be sent there as well. By definition, anyone with an access key can interact with the blockchain and since there is a private key embedded in the link, this removes the need for the end-user to have a wallet. ## Motivation Linkdrops are an extremely powerful tool that enable seamless onboarding and instant crypto experiences with the click of a link. The original [near-linkdrop](https://github.com/near/near-linkdrop) contract provides a minimal interface allowing users to embed $NEAR within an access key and create a simple Web2 style link that can then be used as a means of onboarding. This simple $NEAR linkdrop is not enough as many artists, developers, event coordinators, and applications want to drop more digital assets such as NFTs, FTs, tickets etc. As linkdrop implementations start to push the boundaries of what’s possible, new data structures, methods, and interfaces are being developed. There needs to be a standard data model and interface put into place to ensure assets can be claimed independent of the contract they came from. If not, integrating any application with linkdrops will require customized solutions, which would become cumbersome for the developer and deteriorate the user onboarding experience. The linkdrop standard addresses these issues by providing a simple and extensible standard data model and interface. The initial discussion can be found [here](https://gov.near.org/t/official-linkdrop-standard/32463/1). ## Specification ### Example Scenarios _Pre-requisite Steps_: Linkdrop creation: The linkdrop creator that has an account with some $NEAR: - creates a keypair locally (`pubKey1`, `privKey1`). (The keypair is not written to chain at this time) - calls a method on a contract that implements the linkdrop standard in order to create the drop. The `pubKey1` and desired $NEAR amount are both passed in as arguments. - The contract maps the `pubKey1` to the desired balance for the linkdrop (`KeyInfo` record). - The contract then adds the `pubKey1` as a function call access key with the ability to call `claim` and `create_account_and_claim`. This means that anyone with the `privKey1` (see above), can sign a transaction on behalf of the contract (signer id set to contract id) with a function call to call one of the mentioned functions to claim the assets. #### Claiming a linkdrop without a NEAR Account A user with _no_ account can claim the assets associated with an existing public key, already registered in the linkdrop contract: - generates a new keypair (`pubKey2`, `privKey2`) locally. (This new keypair is not written to chain) - chooses a new account ID such as benji.near. - calls `create_account_and_claim`. The transaction is signed on behalf of the linkdrop contract (`signer_id` is set to the contract address) using `privKey1`. - the args of this function call will contain both `pubKey2` (which will be used to create a full access key for the new account) and the account ID itself. - the linkdrop contract will delete the access key associated with `pubKey1` so that it cannot be used again. - the linkdrop contract will create the new account and transfer the funds to it alongside any other assets. - the user will be able to sign transactions on behalf of the new account using `privKey2`. #### Claiming a linkdrop with a NEAR Account A user with an _existing_ account can claim the assets with an existing public key, already registered in the linkdrop contract: - calls `claim`. The transaction is signed on behalf of the linkdrop contract (`signer_id` is set to the contract address) using `privKey1`. - the args of this function call will simply contain the user's existing account ID. - the linkdrop contract will delete the access key associated with `pubKey1` so that it cannot be used again. - the linkdrop contract will transfer the funds to that account alongside any other assets. ```ts /// Information about a specific public key. type KeyInfo = { /// How much Gas should be attached when the key is used to call `claim` or `create_account_and_claim`. /// It is up to the smart contract developer to calculate the required gas (which can be done either automatically on the contract or on the client-side). required_gas: string, /// yoctoNEAR$ amount that will be sent to the account that claims the linkdrop (either new or existing) /// when the key is successfully used. yoctonear: string, /// If using the NFT standard extension, a set of NFTData can be linked to the public key /// indicating that all those assets will be sent to the account that claims the linkdrop (either new or /// existing) when the key is successfully used. nft_list: NFTData[] | null, /// If using the FT standard extension, a set of FTData can be linked to the public key /// indicating that all those assets will be sent to the account that claims the linkdrop (either new or /// existing) when the key is successfully used. ft_list: FTData[] | null /// ... other types can be introduced and the standard is easily extendable. } /// Data outlining a specific Non-Fungible Token that should be sent to the claiming account /// (either new or existing) when a key is successfully used. type NFTData = { /// the id of the token to transfer token_id: string, /// The valid NEAR account indicating the Non-Fungible Token contract. contract_id: string } /// Data outlining Fungible Tokens that should be sent to the claiming account /// (either new or existing) when a key is successfully used. type FTData = { /// The number of tokens to transfer, wrapped in quotes and treated /// like a string, although the number will be stored as an unsigned integer /// with 128 bits. amount: string, /// The valid NEAR account indicating the Fungible Token contract. contract_id: string } /****************/ /* VIEW METHODS */ /****************/ /// Allows you to query for the amount of $NEAR tokens contained in a linkdrop corresponding to a given public key. /// /// Requirements: /// * Panics if the key does not exist. /// /// Arguments: /// * `key` the public counterpart of the key used to sign, expressed as a string with format ":" (e.g. "ed25519:6TupyNrcHGTt5XRLmHTc2KGaiSbjhQi1KHtCXTgbcr4Y") /// /// Returns a string representing the $yoctoNEAR amount associated with a given public key function get_key_balance(key: string) -> string; /// Allows you to query for the `KeyInfo` corresponding to a given public key. This method is preferred over `get_key_balance` as it provides more information about the key. /// /// Requirements: /// * Panics if the key does not exist. /// /// Arguments: /// * `key` the public counterpart of the key used to sign, expressed as a string with format ":" (e.g. "ed25519:6TupyNrcHGTt5XRLmHTc2KGaiSbjhQi1KHtCXTgbcr4Y") /// /// Returns `KeyInfo` associated with a given public key function get_key_information(key: string) -> KeyInfo; /******************/ /* CHANGE METHODS */ /******************/ /// Transfer all assets linked to the signer’s public key to an `account_id`. /// If the transfer fails for whatever reason, it is up to the smart contract developer to /// choose what should happen. For example, the contract can choose to keep the assets /// or send them back to the original linkdrop creator. /// /// Requirements: /// * The predecessor account *MUST* be the current contract ID. /// * The `account_id` MUST be an *initialized* NEAR account. /// * The assets being sent *MUST* be associated with the signer’s public key. /// * The assets *MUST* be sent to the `account_id` passed in. /// /// Arguments: /// * `account_id` the account that should receive the linkdrop assets. /// /// Returns `true` if the claim was successful meaning all assets were sent to the `account_id`. function claim(account_id: string) -> Promise; /// Creates a new NEAR account and transfers all assets linked to the signer’s public key to /// the *newly created account*. If the transfer fails for whatever reason, it is up to the /// smart contract developer to choose what should happen. For example, the contract can /// choose to keep the assets or return them to the original linkdrop creator. /// /// Requirements: /// * The predecessor account *MUST* be the current contract ID. /// * The assets being sent *MUST* be associated with the signer’s public key. /// * The assets *MUST* be sent to the `new_account_id` passed in. /// * The newly created account *MUST* have a new access key added to its account (either /// full or limited access) in the same receipt that the account was created in. /// * The Public key must be in a binary format with base58 string serialization with human-readable curve. /// The key types currently supported are secp256k1 and ed25519. Ed25519 public keys accepted are 32 bytes /// and secp256k1 keys are the uncompressed 64 format. /// /// Arguments: /// * `new_account_id`: the valid NEAR account which is being created and should /// receive the linkdrop assets /// * `new_public_key`: the valid public key that should be used for the access key added to the newly created account (serialized with borsh). /// /// Returns `true` if the claim was successful meaning the `new_account_id` was created and all assets were sent to it. function create_account_and_claim(new_account_id: string, new_public_key: string) -> Promise; ``` ## Reference Implementation Below are some references for linkdrop contracts. - [Link Drop Contract](https://github.com/near/near-linkdrop) - [Keypom Contract](https://github.com/keypom/keypom) ## Security Implications 1. Linkdrop Creation Linkdrop creation involves creating keypairs that, when used, have access to assets such as $NEAR, FTs, NFTs, etc. These keys should be limited access and restricted to specific functionality. For example, they should only have permission to call `claim` and `create_account_and_claim`. Since the keys allow the holder to sign transactions on behalf of the linkdrop contract, without the proper security measures, they could be used in a malicious manner (for example executing private methods or owner-only functions). Another important security implication of linkdrop creation is to ensure that only one key is mapped to a set of assets at any given time. Externally, assets such as FTs, and NFTs belong to the overall linkdrop contract account rather than a specific access key. It is important to ensure that specific keys can only claim assets that they are mapped to. 2. Linkdrop Key Management Key management is a critical safety component of linkdrops. The linkdrop contract should implement a key management strategy for keys such that a reentrancy attack does not occur. For example, one strategy may be to "lock" or mark a key as "in transfer" such that it cannot be used again until the transfer is complete. 3. Asset Refunds & Failed Claims Given that linkdrops could contain multiple different assets such as NFTs, or fungible tokens, sending assets might happen across multiple blocks. If the claim was unsuccessful (such as passing in an invalid account ID), it is important to ensure that all state is properly managed and assets are optionally refunded depending on the linkdrop contract's implementation. 4. Fungible Tokens & Future Data Fungible token contracts require that anyone receiving tokens must be registered. For this reason, it is important to ensure that storage for accounts claiming linkdrops is paid for. This concept can be extended to any future data types that may be added. You must ensure that all the pre-requisite conditions have been met for the asset that is being transferred. 5. Tokens Properly Sent to Linkdrop Contract Since the linkdrop contract facilitates the transfer of assets including NFTs, and FTs, it is important to ensure that those tokens have been properly sent to the linkdrop contract prior to claiming. In addition, since all the tokens are in a shared pool, you must ensure that the linkdrop contract cannot claim assets that do not belong to the key that is being used to claim. It is also important to note that not every linkdrop is valid. Drops can expire, funds can be lazily sent to the contract (as seen in the case of fungible and non-fungible tokens) and the supply can be limited. ## Alternatives #### Why is this design the best in the space of possible designs? This design allows for flexibility and extensibility of the standard while providing a set of criteria that cover the majority of current linkdrop use cases. The design was heavily inspired by current, functional NEPs such as the Fungible Token and Non-Fungible Token standards. #### What other designs have been considered and what is the rationale for not choosing them? A generic data struct that all drop types needed to inherit from. This struct contained a name and some metadata in the form of stringified JSON. This made it easily extensible for any new types down the road. The rationale for not choosing this design was both simplicity and flexibility. Having one data struct requires keys to be of one type only. In reality, there can be many at once. In addition, having a generic, open-ended metadata field could lead to many interpretations and different designs. We chose to use a KeyInfo struct that can be easily extensible and can cover all use-cases by having optional vectors of different data types. The proposed standard is simple, supports drops with multiple assets, and is backwards compatible with all previous linkdrops, and can be extended very easily. A standard linkdrop creation interface. A standardized linkdrop creation interface would provide data models and functions to ensure linkdrops were created and stored in a specific format. The rationale for not choosing this design was that is was too restrictive. Standardizing linkdrop creation adds complexity and reduces flexibility by restricting linkdrop creators in the process in which linkdrops are created, and potentially limiting linkdrop functionality. The functionality of the linkdrop creation, such as refunding of assets, access keys, and batch creation, should be chosen by the linkdrop creator and live within the linkdrop creator platform. Further, linkdrop creation is often not displayed to end users and there is not an inherent value proposition for a standardized linkdrop creation interface from a client perspective. #### What is the impact of not doing this? The impact of not doing this is creating a fragmented ecosystem of linkdrops, increasing the friction for user onboarding. Linkdrop claim pages (e.g. wallet providers) would have to implement custom integrations for every linkdrop provider platform. Inherently this would lead to a bad user experience when new users are onboarding and interacting with linkdrops in general. ## Future possibilities - Linkdrop creation interface - Bulk linkdrop management (create, update, claim) - Function call data types (allowing for funder defined functions to be executed when a linkdrop is claimed) - Optional configurations added to KeyInfo which can include multi-usekeys, time-based claiming etc… - Standard process for how links connect to claim pages (i.e a standardized URL such as an app’s baseUrl/contractId= [LINKDROP_CONTRACT]&secretKey=[SECRET_KEY] - Standard for deleting keys and refunding assets. ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0455.md ================================================ --- NEP: 455 Title: Parameter Compute Costs Author: Andrei Kashin , Jakob Meier Status: Final DiscussionsTo: https://github.com/nearprotocol/neps/pull/455 Type: Protocol Track Category: Runtime Created: 26-Jan-2023 --- ## Summary Introduce compute costs decoupled from gas costs for individual parameters to safely limit the compute time it takes to process the chunk while avoiding adding breaking changes for contracts. ## Motivation For NEAR blockchain stability, we need to ensure that blocks are produced regularly and in a timely manner. The chunk gas limit is used to ensure that the time it takes to validate a chunk is strictly bounded by limiting the total gas cost of operations included in the chunk. This process relies on accurate estimates of gas costs for individual operations. Underestimating these costs leads to *undercharging* which can increase the chunk validation time and slow down the chunk production. As a concrete example, in the past we undercharged contract deployment. The responsible team has implemented a number of optimizations but a gas increase was still necessary. [Meta-pool](https://github.com/Narwallets/meta-pool/issues/21) and [Sputnik-DAO](https://github.com/near-daos/sputnik-dao-contract/issues/135) were affected by this change, among others. Finding all affected parties and reaching out to them before implementing the change took a lot of effort, prolonging the period where the network was exposed to the attack. Another motivating example is the upcoming incremental deployment of Flat Storage, where during one of the intermediate stages we expect the storage operations to be undercharged. See the explanation in the next section for more details. ## Rationale Separating compute costs from gas costs will allow us to safely limit the compute usage for processing the chunk while still keeping the gas prices the same and thus not breaking existing contracts. An important challenge with undercharging is that it is not possible to disclose them widely because it could be used to increase the chunk production time thereby impacting the stability of the network. Adjusting the compute cost for undercharged parameter eliminates the security concern and allows us to publicly discuss the ways to solve the undercharging (optimize implementation, smart contract or increasing the gas cost). This design is easy to implement and simple to reason about and provides a clear way to address existing undercharging issues. If we don't address the undercharging problems, we increase the risks that they will be exploited. Specifically for Flat Storage deployment, we [plan](https://github.com/near/nearcore/issues/8006) to stop charging TTN (touching trie node) gas costs, however the intermediate implementation (read-only Flat Storage) will still incur these costs during writes introducing undercharging. Setting temporary high compute costs for writes will ensure that this undercharging does not lead to long chunk processing times. ## Alternatives ### Increase the gas costs for undercharged operations We could increase the gas costs for the operations that are undercharged to match the computational time it takes to process them according to the rule 1ms = 1TGas. Pros: - Does not require any new code or design work (but still requires a protocol version bump) - Security implications are well-understood Cons: - Can break contracts that rely on current gas costs, in particular steeply increasing operating costs for the most active users of the blockchain (aurora and sweat) - Doing this safely and responsibly requires prior consent by the affected parties which is hard to do without disclosing security-sensitive information about undercharging in public In case of flat storage specifically, using this approach will result in a large increase in storage write costs (breaking many contracts) to enable safe deployment of read-only flat storage and later a correction of storage write costs when flat storage for writes is rolled out. With compute costs, we will be able to roll out the read-only flat storage with minimal impact on deployed contracts. ### Adjust the gas chunk limit We could continuously measure the chunk production time in nearcore clients and compare it to the gas burnt. If the chunk producer observes undercharging, it decreases the limit. If there is overcharging, the limit can be increased up to a limit of at most 1000 Tgas. To make such adjustment more predictable under spiky load, we also [limit](https://nomicon.io/Economics/Economic#transaction-fees) the magnitude of change of gas limit by 0.1% per block. Pros: - Prevents moderate undercharging from stalling the network - No protocol change necessary (as this feature is already [a part of the protocol](https://nomicon.io/Economics/Economic#transaction-fees)), we could easily experiment and revert if it does not work well Cons: - Very broad granularity --- undercharging in one parameter affects all users, even those that never use the undercharged parts - Dependence on validator hardware --- someone running overspecced hardware will continuously want to increase the limit, others might run with underspecced hardware and continuously want to decrease the limit - Malicious undercharging attacks are unlikely to be prevented by this --- a single 10x undercharged receipt still needs to be processed using the old limit. Adjusting 0.1% per block means 100 chunks can only change by a maximum of 1.1x and 1000 chunks could change up to x2.7 - Conflicts with transaction and receipt limit --- A transaction or receipt can (today) use up to 300Tgas. The effective limit per chunk is `gas_limit` + 300Tgas since receipts are added to a chunk until one exceeds the limit and the last receipt is not removed. Thus a gas limit of 0gas only reduces the effective limit from 1300Tgas to 300Tgas, which means a single 10x undercharged receipt can still result in a chunk with compute usage of 3 seconds (equivalent to 3000TGas) ### Allow skipping chunks in the chain Slow chunk production in one shard can introduce additional user-visible latency in all shards as the nodes expect a regular and timely chunk production during normal operation. If processing the chunk takes much longer than 1.3s, it can cause the corresponding block and possibly more consecutive blocks to be skipped. We could extend the protocol to produce empty chunks for some of the shards within the block (effectively skipping them) when processing the chunk takes longer than expected. This way will still ensure a regular block production, at a cost of lower throughput of the network in that shard. The chunk should still be included in a later block to avoid stalling the affected shard. Pros: - Fast and automatic adaptation to the blockchain workload Cons: - For the purpose of slashing, it is hard to distinguish situations when the honest block producer skips chunk due to slowness from the situations when the block producer is offline or is maliciously stalling the block production. We need some mechanism (e.g. on-chain voting) for nodes to agree that the chunk was skipped legitimately due to slowness as otherwise we introduce new attack vectors to stall the network ## Specification - **Chunk Compute Usage** -- total compute time spent on processing the chunk - **Chunk Compute Limit** -- upper-bound for compute time spent on processing the chunk - **Parameter Compute Cost** -- the numeric value in seconds corresponding to compute time that it takes to include an operation into the chunk Today, gas has two somewhat orthogonal roles: 1. Gas is money. It is used to avoid spam by charging users 2. Gas is CPU time. It defines how many transactions fit in a chunk so that validators can apply it within a second The idea is to decouple these two by introducing parameter compute costs. Each gas parameter still has a gas cost that determines what users have to pay. But when filling a chunk with transactions, parameter compute cost is used to estimate CPU time. Ideally, all compute costs should match corresponding gas costs. But when we discover undercharging issues, we can set a higher compute cost (this would require a protocol upgrade). The stability concern is then resolved when the compute cost becomes active. The ratio between compute cost and gas cost can be thought of as an undercharging factor. If a gas cost is 2 times too low to guarantee stability, compute cost will be twice the gas cost. A chunk will be full 2 times faster when gas for this parameter is burned. This deterministically throttles the throughput to match what validators can actually handle. Compute costs influence the gas price adjustment logic described in https://nomicon.io/Economics/Economic#transaction-fees. Specifically, we're now using compute usage instead of gas usage in the formula to make sure that the gas price increases if chunk processing time is close to the limit. Compute costs **do not** count towards the transaction/receipt gas limit of 300TGas, as that might break existing contracts by pushing their method calls over this limit. Compute costs are static for each protocol version. ### Using Compute Costs Compute costs different from gas costs are only a temporary solution. Whenever we introduce a compute cost, we as the community can discuss this publicly and find a solution to the specific problem together. For any active compute cost, a tracking GitHub issue in [`nearcore`](https://github.com/near/nearcore) should be created, tracking work towards resolving the undercharging. The reference to this issue should be added to this NEP. In the best case, we find technical optimizations that allow us to decrease the compute cost to match the existing gas cost. In other cases, the only solution is to increase the gas cost. But the dApp developers who are affected by this change should have a chance to voice their opinion, suggest alternatives, and implement necessary changes before the gas cost is increased. ## Reference Implementation The compute cost is a numeric value represented as `u64` in time units. Value 1 corresponds to `10^-15` seconds or 1fs (femtosecond) to match the gas costs scale. By default, the parameter compute cost matches the corresponding gas cost. Compute costs should be applicable to all gas parameters, specifically including: - [`ExtCosts`](https://github.com/near/nearcore/blob/6e08a41084c632010b1d4c42132ad58ecf1398a2/core/primitives-core/src/config.rs#L377) - [`ActionCosts`](https://github.com/near/nearcore/blob/6e08a41084c632010b1d4c42132ad58ecf1398a2/core/primitives-core/src/config.rs#L456) Changes necessary to support `ExtCosts`: 1. Track compute usage in [`GasCounter`](https://github.com/near/nearcore/blob/51670e593a3741342a1abc40bb65e29ba0e1b026/runtime/near-vm-logic/src/gas_counter.rs#L47) struct 2. Track compute usage in [`VMOutcome`](https://github.com/near/nearcore/blob/056c62183e31e64cd6cacfc923a357775bc2b5c9/runtime/near-vm-logic/src/logic.rs#L2868) struct (alongside `burnt_gas` and `used_gas`) 3. Store compute usage in [`ActionResult`](https://github.com/near/nearcore/blob/6d2f3fcdd8512e0071847b9d2ca10fb0268f469e/runtime/runtime/src/lib.rs#L129) and aggregate it across multiple actions by modifying [`ActionResult::merge`](https://github.com/near/nearcore/blob/6d2f3fcdd8512e0071847b9d2ca10fb0268f469e/runtime/runtime/src/lib.rs#L141) 4. Store compute costs in [`ExecutionOutcome`](https://github.com/near/nearcore/blob/578983c8df9cc36508da2fb4a205c852e92b211a/runtime/runtime/src/lib.rs#L266) and [aggregate them across all transactions](https://github.com/near/nearcore/blob/578983c8df9cc36508da2fb4a205c852e92b211a/runtime/runtime/src/lib.rs#L1279) 5. Enforce the chunk compute limit when the chunk is [applied](https://github.com/near/nearcore/blob/6d2f3fcdd8512e0071847b9d2ca10fb0268f469e/runtime/runtime/src/lib.rs#L1325) Additional changes necessary to support `ActionCosts`: 1. Return compute costs from [`total_send_fees`](https://github.com/near/nearcore/blob/578983c8df9cc36508da2fb4a205c852e92b211a/runtime/runtime/src/config.rs#L71) 2. Store aggregate compute cost in [`TransactionCost`](https://github.com/near/nearcore/blob/578983c8df9cc36508da2fb4a205c852e92b211a/runtime/runtime/src/config.rs#L22) struct 3. Propagate compute costs to [`VerificationResult`](https://github.com/near/nearcore/blob/578983c8df9cc36508da2fb4a205c852e92b211a/runtime/runtime/src/verifier.rs#L330) Additionaly, the gas price computation will need to be adjusted in [`compute_new_gas_price`](https://github.com/near/nearcore/blob/578983c8df9cc36508da2fb4a205c852e92b211a/core/primitives/src/block.rs#L328) to use compute cost instead of gas cost. ## Security Implications Changes in compute costs will be publicly known and might reveal an undercharging that can be used as a target for the attack. In practice, it is not trivial to exploit the undercharging unless you know the exact shape of the workload that realizes it. Also, after the compute cost is deployed, the undercharging should no longer be a threat for the network stability. ## Drawbacks - Changing compute costs requires a protocol version bump (and a new binary release), limiting their use to undercharging problems that we're aware of - Updating compute costs is a manual process and requires deliberately looking for potential underchargings - The compute cost would not have a full effect on the last receipt in the chunk, decreasing its effectiveness to deal with undercharging. This is because 1) a transaction or receipt today can use up to 300TGas and 2) receipts are added to a chunk until one exceeds the limit and the last receipt is not removed. Therefore, a single receipt with 300TGas filled with undercharged operations with a factor of K can lead to overshooting the chunk compute limit by (K - 1) * 300TGas - Underchargings can still be exploited to lower the throughput of the network at unfair price and increase the waiting times for other users. This is inevitable for any proposal that doesn't change the gas costs and must be resolved by improving the performance or increasing the gas costs - Even without malicious intent, the effective peak throughput of the network will decrease when the chunks include undercharged operations (as the stopping condition based on compute costs for filling the chunk becomes stricter). Most of the time, this is not the problem as the network is operating below the capacity. The effects will also be softened by the fact that undercharged operations comprise only a fraction of the workload. For example, the planned increase for TTN compute cost alongside the Flat Storage MVP is less critical because you cannot fill a receipt with only TTN costs, you will always have other storage costs and ~5Tgas overhead to even start a function call. So even with 10x difference between gas and compute costs, the DoS only becomes 5x cheaper instead of 10x ## Unresolved Issues ## Future possibilities We can also think about compute costs smaller than gas costs. For example, we charge gas instead of token balance for extra storage bytes in [NEP-448](https://github.com/near/NEPs/pull/448), it would make sense to set the compute cost to 0 for the part that covers on-chain storage if the throttling due to increased gas cost becomes problematic. Otherwise, the throughput would be throttled unnecessarily. A further option would be to change compute costs dynamically without a protocol upgrade when block production has become too slow. This would be a catch-all, self-healing solution that requires zero intervention from anyone. The network would simply throttle throughput when block time remains too high for long enough. Pursuing this approach would require additional design work: - On-chain voting to agree on new values of costs, given that inputs to the adjustment process are not deterministic (measurements of wall clock time it takes to process receipt on particular validator) - Ensuring that dynamic adjustment is done in a safe way that does not lead to erratic behavior of costs (and as a result unpredictable network throughput). Having some experience manually operating this mechanism would be valuable before introducing automation and addressing challenges described in https://github.com/near/nearcore/issues/8032#issuecomment-1362564330. The idea of introducing a chunk limit for compute resource usage naturally extends to other resource types, for example RAM usage, Disk IOPS, [Background CPU Usage](https://github.com/near/nearcore/issues/7625). This would allow us to align the pricing model with cloud offerings familiar to many users, while still using gas as a common denominator to simplify UX. ## Changelog ### 1.0.0 - Initial Version This NEP was approved by Protocol Working Group members on March 16, 2023 ([meeting recording](https://www.youtube.com/watch?v=4VxRoKwLXIs)): - [Bowen's vote](https://github.com/near/NEPs/pull/455#issuecomment-1467023424) - [Marcelo's vote](https://github.com/near/NEPs/pull/455#pullrequestreview-1340887413) - [Marcin's vote](https://github.com/near/NEPs/pull/455#issuecomment-1471882639) ### 1.0.1 - Storage Related Compute Costs Add five compute cost values for protocol version 61 and above. - wasm_touching_trie_node - wasm_storage_write_base - wasm_storage_remove_base - wasm_storage_read_base - wasm_storage_has_key_base For the exact values, please refer to the table at the bottom. The intention behind these increased compute costs is to address the issue of storage accesses taking longer than the allocated gas costs, particularly in cases where RocksDB, the underlying storage system, is too slow. These values have been chosen to ensure that validators with recommended hardware can meet the required timing constraints. ([Analysis Report](https://github.com/near/nearcore/issues/8006)) The protocol team at Pagoda is actively working on optimizing the nearcore client storage implementation. This should eventually allow to lower the compute costs parameters again. Progress on this work is tracked here: https://github.com/near/nearcore/issues/8938. #### Benefits - Among the alternatives, this is the easiest to implement. - It allows us to able to publicly discuss undercharging issues before they are fixed. #### Concerns No concerns that need to be addressed. The drawbacks listed in this NEP are minor compared to the benefits that it will bring. And implementing this NEP is strictly better than what we have today. ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ## References - https://gov.near.org/t/proposal-gas-weights-to-fight-instability-to-due-to-undercharging/30919 - https://github.com/near/nearcore/issues/8032 ## Live Compute Costs Tracking Parameter Name | Compute / Gas factor | First version | Last version | Tracking issue | -------------- | -------------------- | ------------- | ------------ | -------------- | wasm_touching_trie_node | 6.83 | 61 | *TBD* | [nearcore#8938](https://github.com/near/nearcore/issues/8938) wasm_storage_write_base | 3.12 | 61 | *TBD* | [nearcore#8938](https://github.com/near/nearcore/issues/8938) wasm_storage_remove_base | 3.74 | 61 | *TBD* | [nearcore#8938](https://github.com/near/nearcore/issues/8938) wasm_storage_read_base | 3.55 | 61 | *TBD* | [nearcore#8938](https://github.com/near/nearcore/issues/8938) wasm_storage_has_key_base | 3.70 | 61 | *TBD* | [nearcore#8938](https://github.com/near/nearcore/issues/8938) ================================================ FILE: neps/nep-0488.md ================================================ --- NEP: 488 Title: Host Functions for BLS12-381 Curve Operations Authors: Olga Kuniavskaia Status: Final DiscussionsTo: https://github.com/nearprotocol/neps/pull/488 Type: Runtime Spec Version: 0.0.1 Created: 2023-07-17 LastUpdated: 2023-11-21 --- ## Summary This NEP introduces host functions to perform operations on the BLS12-381 elliptic curve. It is a minimal set of functions needed to efficiently verify BLS signatures and zkSNARKs. ## Motivation The primary aim of this NEP is to enable fast and efficient verification of BLS signatures and zkSNARKs based on the BLS12-381[^1],[^11],[^52] elliptic curve on NEAR. To efficiently verify zkSNARKs[^19], host functions for operations on the BN254 elliptic curve (also known as Alt-BN128)[^9], [^12] have already been implemented on NEAR[^10]. For instance, the Zeropool[^20] project utilizes these host functions for verifying zkSNARKs on NEAR. However, recent research shows that the BN254 security level is lower than 100-bit[^13] and it is not recommended for use. BLS12-381, on the other hand, offers over 120 bits of security[^8] and is widely used[^2],[^3],[^4],[^5],[^6],[^7] as a robust alternative. Supporting operations for BLS12-381 elliptic curve will significantly enhance the security of projects similar to Zeropool. Another crucial objective is the verification of BLS signatures. Initially, host functions for BN254 on NEAR were designed for zkSNARK verification and are insufficient for BLS signature verification. However, even if these host functions were sufficient for BLS signature verification on the BN254 elliptic curve, this would not be enough for compatibility with other projects. In particular, projects such as ZCash[^2], Ethereum[^3], Tezos[^5], and Filecoin[^6] incorporate BLS12-381 specifically within their protocols. If we aim for compatibility with these projects, we must also utilize this elliptic curve. For instance, to create a trustless bridge[^17] between Ethereum and NEAR, we must efficiently verify BLS signatures based on BLS12-381, as these are the signatures employed within Ethereum's protocol. In this NEP, we propose to add the following host functions: - ***bls12381_p1_sum —*** computes the sum of signed points from $E(F_p)$ elliptic curve. This function is useful for aggregating public keys or signatures in the BLS signature scheme. It can be employed for simple addition in $E(F_p)$. It is kept separate from the `multiexp` function due to gas cost considerations. - ***bls12381_p2_sum —*** computes the sum of signed points from $E'(F_{p^2})$ elliptic curve. This function is useful for aggregating signatures or public keys in the BLS signature scheme. - ***bls12381_g1_multiexp —*** calculates $\sum p_i s_i$ for points $p_i \in G_1 \subset E(F_p)$ and scalars $s_i$. This operation can be used to multiply a group element by a scalar. - ***bls12381_g2_multiexp —*** calculates $\sum p_i s_i$ for points $p_i \in G_2 \subset E'(F_{p^2})$ and scalars $s_i$. - ***bls12381_map_fp_to_g1 —*** maps base field elements into $G_1$ points. It does not perform the mapping of byte strings into field elements. - ***bls12381_map_fp2_to_g2 —*** maps extension field elements into $G_2$ points. This function does not perform the mapping of byte strings into extension field elements, which would be needed to efficiently map a message into a group element. We are not implementing the `hash_to_field`[^60] function because the latter can be executed within a contract and various hashing algorithms can be used within this function. - ***bls12381_p1_decompress —*** decompresses points from $E(F_p)$ provided in a compressed form. Certain protocols offer points on the curve in a compressed form (e.g., the light client updates in Ethereum 2.0), and decompression is a time-consuming operation. All the other functions in this NEP only accept decompressed points for simplicity and optimized gas consumption. - ***bls12381_p2_decompress —*** decompresses points from $E'(F_{p^2})$ provided in a compressed form. - ***bls12381_pairing_check —*** verifies that $\prod e(p_i, q_i) = 1$, where $e$ is a pairing operation and $p_i \in G_1 \land q_i \in G_2$. This function is used to verify BLS signatures or zkSNARKs. Functions required for verifying BLS signatures[^59]: - bls12381_p1_sum - bls12381_p2_sum - bls12381_map_fp2_to_g2 - bls12381_p1_decompress - bls12381_p2_decompress - bls12381_pairing_check Functions required for verifying zkSNARKs: - bls12381_p1_sum - bls12381_g1_multiexp - bls12381_pairing_check Both zkSNARKs and BLS signatures can be implemented alternatively by swapping $G_1$ and $G_2$. Therefore, all functions have been implemented for both $G_1$ and $G_2$. An analogous proposal, EIP-2537[^15], exists in Ethereum. The functions here have been designed with compatibility with that Ethereum's proposal in mind. This design approach aims to ensure future ease in supporting corresponding precompiles for Aurora[^24]. ## Specification ### BLS12-381 Curve Specification #### Elliptic Curve **The field $F_p$** for some *prime* $p$ is a set of integer elements $\textbraceleft 0, 1, \ldots, p - 1 \textbraceright$ with two operations: multiplication $\cdot$ and addition $+$. These operations involve standard integer multiplication and addition, followed by computing the remainder modulo $p$. **The elliptic curve $E(F_p)$** is the set of all pairs $(x, y)$ with coordinates in $F_p$ satisfying: $$ y^2 \equiv x^3 + Ax + B \mod p $$ together with an imaginary point at infinity $\mathcal{O}$, where: $A, B \in F_p$, $p$ is a prime $> 3$, and $4A^3 + 27B^2 \not \equiv 0 \mod p$ In the case of BLS12-381 the equation is $y^2 \equiv x^3 + 4 \mod p$[^15],[^51],[^14],[^11] **Parameters for our case:** - $A = 0$ - $B = 4$ - $p = \mathtt{0x1a0111ea397fe69a4b1ba7b6434bacd764774b84f38512bf6730d2a0f6b0f6241eabfffeb153ffffb9feffffffffaaab}$ Let $P \in E(F_q)$ have coordinates $(x, y)$, define **$-P$** as a point on a curve with coordinates $(x, -y)$. **The addition operation for Elliptic Curve** is a function $+\colon E(F_p) \times E(F_p) \rightarrow E(F_p)$ defined with following rules: let $P$ and $Q \in E(F_p)$ - if $P \ne Q$ and $P \ne -Q$ - draw a line passing through $P$ and $Q$. This line intersects the curve at a third point $R$. - reflect the point $R$ across the $x$-axis by changing the sign of the $y$-coordinate. The resulting point is $P+Q$. - if $P=Q$ - draw a tangent line through $P$ for an elliptic curve. The line will intersect the curve at the second point $R$. - reflect the point $R$ across the $x$-axis the same way to get point $2P$ - $P = -Q$ - $P + Q = P + (-P) = \mathcal{O}$ — the point on infinity - $Q = \mathcal{O}$ - $P + Q = P + \mathcal{O} = P$ With the addition operation, Elliptic Curve forms a **group**. #### Subgroups **Subgroup** H is a subset of the group G with the following properties: - $\forall h_1, h_2 \in H\colon h_1 + h_2 \in H$ - $0 \in H$ - $\forall h \in H \colon -h \in H$ Notation: $H \subseteq G$ Group/subgroup **order** is the number of elements in group/subgroup. Notation: |G| or #G, where G represents the group. For some technical reason (related to the `pairing` operation which we will define later), we will not operate over the entire $E(F_p)$, but only over the two subgroups $G_1$ and $G_2$ having the same **order** $r$. $G_1$ is a subset of $E(F_p)$, while $G_2$ is a subgroup of another group that we will define later. The value of $r$ should be a prime number and $G_1 \ne G_2$ For the BLS12-381 Elliptic Curve, **the order r** of $G_1$ and $G_2$[^15],[^51] is given by: - $r = \mathtt{0x73eda753299d7d483339d80809a1d80553bda402fffe5bfeffffffff00000001}$ #### Field extension **The field extension $F_{p^k}$ of $F_{p}$** is a set comprising all polynomials of degree < k and coefficients from $F_p$, along with defined operations of multiplication ($\cdot$) and addition ($+$). $$ a_{k - 1}x^{k - 1} + \ldots + a_1x + a_0 = A(x) \in F_{p^k} \vert a_i \in F_p $$ The addition operation ($+$) is defined as regular polynomial addition: $$ A(x) + B(x) = C(x) $$ $$ \sum a_i x^i + \sum b_i x^i = \sum c_i x^i $$ $$ c_i = (a_i + b_i) \mod p $$ The multiplication $\cdot$ is defined as regular polynomial multiplication modulo $M(x)$, where $M(x)$ is an irreducible polynomial of degree $k$ with coefficients from $F_p$. $$ C(x) = A(x) \cdot B(x)\mod M(x) $$ Notation: $F_{p^k} = F_{p}[x] / M(x)$ In BLS12-381, we will require $F_{p^{12}}$. We'll construct this field not directly as an extension from $F_p$, but rather through a stepwise process. First, we'll build $F_{p^2}$ as a quadratic extension of the field $F_p$. Second, we'll establish $F_{p^6}$ as a cubic extension of $F_{p^2}$. Finally, we'll create $F_{p^{12}}$ as a quadratic extension of the field $F_{p^6}$. To define these fields, we'll need to set up three irreducible polynomials[^51]: - $F_{p^2} = F_p[u] / (u^2 + 1)$ - $F_{p^6} = F_{p^2}[v] / (v^3 - u - 1)$ - $F_{p^{12}} = F_{p^6}[w] / (w^2 - v)$ The second subgroup we'll utilize has order r and resides within the same elliptic curve but with elements from $F_{p^{12}}$. Specifically, $G_2 \subset E(F_{p^{12}})$, where $E: y^2 = x^3 + 4$ #### Twist Storing elements from $E(F_{p^{12}})$ consumes a significant amount of memory. The twist operation transforms the original curve $E(F_{p^{12}})$ into another curve within a different space, denoted as $E'(F_{p^2})$. It is crucial that this new curve also includes a $G'_2$ subgroup with order 'r' so that we can easily transform it back to the original $G_2$. We want to have $\psi \colon E'(F_{p^2}) \rightarrow E(F_{p^{12}})$, such as - $\forall a, b \in E'(F_{p^2}) \colon \psi(a + b) = \psi(a) + \psi(b)$ - $\forall a, b \in E'(F_{p^2}) \colon \psi(a) = \psi(b) \Rightarrow a = b$ This is referred to as an injective group homomorphism. For BLS12-381, E’ is defined as[^51]: $$ E'\colon y^2 = x^3 + 4(u + 1) $$ In most cases, we will be working with points from $G_2' \subset E'(F_{p^2})$ and will simply use the notation $G_2$ for this subgroup. #### Generators If there exists an element $g$ in the group $G$ such that $\textbraceleft g, 2 \cdot g, 3 \cdot g, \ldots, |G|g \textbraceright = G$, the group $G$ is called a ***cyclic group*** and $g$ is termed a ***generator*** $G_1$ and $G_2$ are cyclic subgroups with the following generators[^15],[^51]: $G_1$: - $x = \mathtt{0x17f1d3a73197d7942695638c4fa9ac0fc3688c4f9774b905a14e3a3f171bac586c55e83ff97a1aeffb3af00adb22c6bb}$ - $y = \mathtt{0x08b3f481e3aaa0f1a09e30ed741d8ae4fcf5e095d5d00af600db18cb2c04b3edd03cc744a2888ae40caa232946c5e7e1}$ For $(x', y') \in G_2 \subset E'(F_{p^2}):$ $$x' = x_0 + x_1u$$ $$y' = y_0 + y_1u$$ $G_2$: - $x_0 = \mathtt{0x024aa2b2f08f0a91260805272dc51051c6e47ad4fa403b02b4510b647ae3d1770bac0326a805bbefd48056c8c121bdb8}$ - $x_1 = \mathtt{0x13e02b6052719f607dacd3a088274f65596bd0d09920b61ab5da61bbdc7f5049334cf11213945d57e5ac7d055d042b7e}$ - $y_0 = \mathtt{0x0ce5d527727d6e118cc9cdc6da2e351aadfd9baa8cbdd3a76d429a695160d12c923ac9cc3baca289e193548608b82801}$ - $y_1 = \mathtt{0x0606c4a02ea734cc32acd2b02bc28b99cb3e287e85a763af267492ab572e99ab3f370d275cec1da1aaa9075ff05f79be}$ **Cofactor** is the ratio of the size of the entire group $G$ to the size of the subgroup $H$: $$ |G|/|H| $$ Cofactor $G_1\colon h = |E(F_p)|/r$[^51] $$h = \mathtt{0x396c8c005555e1568c00aaab0000aaab}$$ Cofactor $G_2\colon h' = |E'(F_{p^2})|/r$[^51] $$h' = \mathtt{0x5d543a95414e7f1091d50792876a202cd91de4547085abaa68a205b2e5a7ddfa628f1cb4d9e82ef21537e293a6691ae1616ec6e786f0c70cf1c38e31c7238e5}$$ #### Pairing Pairing is a necessary operation for the verification of BLS signatures and certain zkSNARKs. It performs the operation $e\colon G_1 \times G_2 \rightarrow G_T$, where $G_T \subset F_{p^{12}}$. The main properties of the pairing operation are: - $e(P, Q + R) = e(P, Q) \cdot e(P, R)$ - $e(P + S, R) = e(P, R)\cdot e(S, R)$ To compute this function, we utilize an algorithm called Miller Loop. For an affective implementation of this algorithm, we require a key parameter for the BLS curve, denoted as $x$: $$ x = -\mathtt{0xd201000000010000}$$ This parameter can be found in the following sources: - [^15] section specification, pairing parameters, Miller loop scalar - [^51] section 4.2.1 Parameter t - [^14] section BLS12-381, parameter u - [^11] section Curve equation and parameters, parameter x #### Summary The parameters for the BLS12-381 curve are as follows: Base field modulus: $p = \mathtt{0x1a0111ea397fe69a4b1ba7b6434bacd764774b84f38512bf6730d2a0f6b0f6241eabfffeb153ffffb9feffffffffaaab}$ $$ E\colon y^2 \equiv x^3 + 4 $$ $$ E'\colon y^2 \equiv x^3 + 4(u + 1) $$ Main subgroup order: $r = \mathtt{0x73eda753299d7d483339d80809a1d80553bda402fffe5bfeffffffff00000001}$ $$ F_{p^2} = F_p[u] / (u^2 + 1) $$ $$ F_{p^6} = F_{p^2}[v] / (v^3 - u - 1) $$ $$ F_{p^{12}} = F_{p^6}[w] / (w^2 - v) $$ Generator for $G_1$: - $x = \mathtt{0x17f1d3a73197d7942695638c4fa9ac0fc3688c4f9774b905a14e3a3f171bac586c55e83ff97a1aeffb3af00adb22c6bb}$ - $y = \mathtt{0x08b3f481e3aaa0f1a09e30ed741d8ae4fcf5e095d5d00af600db18cb2c04b3edd03cc744a2888ae40caa232946c5e7e1}$ Generator for $G_2$: - $x_0 = \mathtt{0x024aa2b2f08f0a91260805272dc51051c6e47ad4fa403b02b4510b647ae3d1770bac0326a805bbefd48056c8c121bdb8}$ - $x_1 = \mathtt{0x13e02b6052719f607dacd3a088274f65596bd0d09920b61ab5da61bbdc7f5049334cf11213945d57e5ac7d055d042b7e}$ - $y_0 = \mathtt{0x0ce5d527727d6e118cc9cdc6da2e351aadfd9baa8cbdd3a76d429a695160d12c923ac9cc3baca289e193548608b82801}$ - $y_1 = \mathtt{0x0606c4a02ea734cc32acd2b02bc28b99cb3e287e85a763af267492ab572e99ab3f370d275cec1da1aaa9075ff05f79be}$ Cofactor for $G_1$: $$h = \mathtt{0x396c8c005555e1568c00aaab0000aaab}$$ Cofactor for $G_2$: $$h' = \mathtt{0x5d543a95414e7f1091d50792876a202cd91de4547085abaa68a205b2e5a7ddfa628f1cb4d9e82ef21537e293a6691ae1616ec6e786f0c70cf1c38e31c7238e5}$$ Key BLS12-381 parameter used in Miller Loop: $$x = -\mathtt{0xd201000000010000}$$ All parameters were sourced from [^15], [^51], and [^14], and they remain consistent across these sources. ### Map to curve specification This section delineates the functionality of the `bls12381_map_fp_to_g1` and `bls12381_map_fp2_to_g2` functions, operating in accordance with the RFC9380 specification "Hashing to Elliptic Curves"[^62]. These functions map field elements in $F_p$ or $F_{p^2}$ to their corresponding subgroups: $G_1 \subset E(F_p)$ or $G_2 \subset E'(F_{p^2})$. `bls12381_map_fp_to_g1`/`bls12381_map_fp2_to_g2` combine the functionalities of `map_to_curve` and `clear_cofactor` from RFC9380[^63]. ```text fn bls12381_map_fp_to_g1(u): let Q = map_to_curve(u); return clear_cofactor(Q); ``` We choose not to implement the `hash_to_field` function as a host function due to potential changes in hashing methods. Additionally, executing this function within the contract consumes approximately 2 TGas, which is acceptable for our goals. Specific implementation parameters for `bls12381_map_fp_to_g1` and `bls12381_map_fp2_to_g2` can be found in RFC9380 under sections 8.8.1[^64] and 8.8.2[^65], respectively. ### Curve points encoding #### General comments The encoding rules for curve points and field elements align with the standards established in zkcrypto[^53] and the implementation in the milagro lib[^29]. For elements from $F_p$ the first three bits will always be $0$, because the first byte of $p$ equals $1$. As a result, we can use these bits to encode extra information: the encoding format, the point at infinity, and the points' sign. Read more in sections: Uncompressed/compressed points on curve $E(F_p)$ / $E'(F_{p^2})$. #### Sign The sign of a point on the elliptic curve is represented as a u8 type in Rust, with two possible values: 0 for a positive sign and 1 for a negative sign. Any other u8 value is considered invalid and should be treated as incorrect. #### Scalar A scalar value is encoded as little-endian [u8; 32]. All possible byte combinations are allowed. #### Fields elements $F_p$ Values from $F_p$ are encoded as big-endian [u8; 48]. Only values less than p are permitted. If the value is equal to or greater than p, an error should be returned. #### Extension fields elements $F_{p^2}$ An element $q \in F_{p^{2}}$ can be expressed as $q = c_0 + c_1 v$, where $c_0, c_1 \in F_p$. An element from $F_{p^2}$ is encoded in [u8; 96] as the byte concatenation of $c_1$ and $c_0$. The encoding for $c_1$ and $c_0$ follows the rule described in the previous section. #### Uncompressed points on curve $E(F_p)$ Points on the curve are represented by affine coordinates: $(x: F_p, y: F_p)$. Elements from $E(F_p)$ are encoded in `[u8; 96]` as the byte concatenation of the x and y point coordinates, where $x, y \in F_p$. The encoding follows the rules outlined in the section “Fields elements $F_p$”. *The second-highest bit* within the encoding serves to signify a point at infinity. When this bit is set to 1, it designates an infinity point. In this case, all other bits should be set to 0. Encoding the point at infinity: ```bash let x: [u8; 96] = [0; 96]; x[0] = x[0] | 0x40; ``` #### Compressed points on curve $E(F_p)$ The points on the curve are represented by affine coordinates: $(x: F_p, y: F_p)$. Elements from $E(F_p)$ in compressed form are encoded as `[u8; 48]`, with big-endian encoded $x \in F_p$. The $y$ coordinate is determined by the formula: $y = \pm \sqrt{x^3 + 4}$. - The highest bit indicates that the point is encoded in compressed form and thus must always be set to 1. - The second-highest bit marks the point at infinity (if set to 1). - For the point at infinity, all bits except the first two should be set to 0; other encodings should be considered as incorrect. - To represent the sign of $y$, the third-highest bit in the x encoding is utilized. - If the bit is 0, $y$ is positive; if 1, $y$ is negative. We'll consider the number positive by taking the smallest value between $y$ and $-y$, after reducing them to $[0, p)$. The encoding for $x \in F_p$ as `[u8; 48]` bytes follows the rules described in the section "Extension fields elements $F_{p}$". Encoding a point on $E(F_p)$ with a negative $y$ coordinate: ```rust let x: [u8; 48] = encodeFp(x) x[0] = x[0] | 0x80; x[0] = x[0] | 0x20; ``` Encoding the point at infinity: ```rust let x: [u8; 48] = [0; 48]; x[0] = x[0] | 0x80; x[0] = x[0] | 0x40; ``` #### Uncompressed points on the twisted curve $E'(F_{p^2})$ The points on the curve are represented by affine coordinates: $(x: F_{p^2}, y: F_{p^2})$. Elements from $E'(F_{p^2})$ are encoded in [u8; 192] as a concatenation of bytes representing x and y coordinates, where $x, y \in F_{p^2}$. The encoding for $x$ and $y$ follows the rules detailed in the "Extension Fields Elements $F_{p^2}$" section. *The second-highest bit* within the encoding serves to signify a point at infinity. When this bit is set to 1, it designates an infinity point. In this case, all other bits should be set to 0. Encoding the point at infinity: ```bash let x: [u8; 192] = [0; 192]; x[0] = x[0] | 0x40; ``` #### Compressed points on twisted curve $E'(F_{p^2})$ The points on the curve are represented by affine coordinates: $(x: F_{p^2}, y: F_{p^2})$. Elements from $E'(F_{p^2})$ in compressed form are encoded as [u8; 96], with big-endian encoded $x \in F_{p^2}$. The $y$ coordinate is determined using the formula: $y = \pm \sqrt{x^3 + 4(u + 1)}$. - The highest bit indicates if the point is encoded in compressed form and should be set to 1. - The second-highest bit marks the point at infinity (if set to 1). - For the point at infinity, all bits except the first two should be set to 0; other encodings should be considered as incorrect. - To represent the sign of $y$, the third-highest bit in the x encoding is utilized. - If the bit is 0, $y$ is positive; if 1, $y$ is negative. We'll consider the number positive by taking the smallest value between $y$ and $-y$: first compare $c_1$, then $c_0$, after reduction to $[0, p)$. The encoding of $x \in F_{p^2}$ as [u8; 96] bytes follows the rules from the section “Extension Fields Elements $F_{p^2}$”. Encoding a point on $E'(F_{p^2})$ with a negative $y$ coordinate: ```rust let x: [u8; 96] = encodeFp2(x); x[0] = x[0] | 0x80; x[0] = x[0] | 0x20; ``` Encoding the point at infinity: ```rust let x: [u8; 96] = [0; 96]; x[0] = x[0] | 0x80; x[0] = x[0] | 0x40; ``` #### ERROR_CODE Validating the input for the host functions within the contract can consume significant gas. For instance, verifying if a point belongs to the subgroup is gas-consuming. If an error is returned by the near host function, the entire execution is reverted. To mitigate this, when the input verification is complex, the host function will successfully complete its work but return an ERROR_CODE. This enables users to handle error cases independently. It's important to note that host functions might terminate with an error if it's straightforward to avoid it (e.g., incorrect input size). The ERROR_CODE is an u64 and can hold the following values: - 0: No error, execution was successful. For `bls12381_pairing_check` function, the pairing result equals the multiplicative identity. - 1: Execution finished with error due to: - Incorrect encoding (e.g., incorrectly set compression/decompression bit, coordinate >= p, etc.). - A point not on the curve (where applicable). - A point not in the expected subgroup (where applicable). - 2: Can be returned only in `bls12381_pairing_check`. No error, execution was successful, but the pairing result doesn't equal the multiplicative identity. ### Host functions #### General comments for all functions In all functions, the input is fetched from memory, beginning at `value_ptr` and extending to `value_ptr + value_len`. If `value_len` is `u64::MAX`, input will come from the register with id `value_ptr`. Execution ends only if there's an incorrect input length, input extends beyond memory bounds, or gas limits are reached. Otherwise, execution completes successfully, providing the `ERROR_CODE`. If the `ERROR_CODE` equals 0, the output data will be written to the register with the `register_id` identifier. Otherwise, nothing will be written to the register. ***Gas Estimation:*** The algorithms described above exhibit linear complexity concerning the number of elements. Gas estimation can be calculated using the following formula: ```rust let k = input_bytes/item_size let gas_consumed = A + B * k ``` Here, A and B denote empirically calculated constants unique to each algorithm. For gas estimation, the benchmark vectors outlined in EIP-2537[^46] can be used where applicable. ***Error cases (execution is terminated):*** For all functions, execution will terminate in the following cases: - The input length is not divisible by `item_size`. - The input is beyond memory bounds. #### bls12381_p1_sum ***Description:*** The function calculates the sum of signed elements on the BLS12-381 curve. It accepts an arbitrary number of pairs $(s_i, p_i)$, where $p_i \in E(F_p)$ represents a point on the elliptic curve, and $s_i \in {0, 1}$ signifies the point's sign. The output is a single point from $E(F_p)$ equivalent to $\sum (-1)^{s_i}p_i$. The operations, including the $E(F_p)$ curve, points on the curve, multiplication by -1, and the addition operation, are detailed in the BLS12-381 Curve Specification section. Note: This function accepts points from the entire curve and is not restricted to points in $G_1$. ***Input:*** The sequence of pairs $(s_i, p_i)$, where $p_i \in E(F_p)$ represents a point and $s_i \in {0, 1}$ denotes the sign. Each point is encoded in decompressed form as $(x\colon F_p, y\colon F_p)$, and the sign is encoded in one byte, taking only two allowed values: 0 or 1. Expect 97*k bytes as input, which are interpreted as byte concatenation of k slices, with each slice representing the point sign and the uncompressed point from $E(F_p)$. Further details are available in the Curve Points Encoding section. ***Output:*** The ERROR_CODE is returned. - ERROR_CODE = 0: the input is correct - Output: 96 bytes represent one point $\in E(F_p)$ in its decompressed form. In case of an empty input, it outputs a point on infinity (refer to the Curve Points Encoding section for more details). - ERROR_CODE = 1: - Points or signs are incorrectly encoded (refer to Curve points encoded section). - Point is not on the curve. ***Test cases:*** Tests for the sum of two points This section aims to verify the correctness of summing up two valid elements on the curve: - Utilize points on the curve with known addition results for comparison, such as tests from EIP-2537[^47],[^48]. - Generate random points on the curve and verify the commutative property: P + Q = Q + P. - Validate that the sum of random points from $G_1$ remains in $G_1$. - Generate random points on the curve and use another library to cross-check the results. Edge cases: - Points not from $G_1$. - $\mathcal{O} + \mathcal{O} = \mathcal{O}$. - $P + \mathcal{O} = \mathcal{O} + P = P$. - $P + (-P) = (-P) + P = \mathcal{O}$. - P + P (tangent to the curve). - The sum of two points P and (-(P + P)) (tangent to the curve at point P). Tests for inversion This section aims to validate the correctness of point inversion: - Generate random points on the curve and verify $P - P = -P + P = \mathcal{O}$. - Generate random points on the curve and verify -(-P) = P. - Generate random points from $G_1$ and ensure that -P also belong to $G_1$. - Utilize an external implementation, generate random points on the curve, and compare results. Edge cases: - Points not from $G_1$. - $-\mathcal{O}$ Tests for incorrect data This section aims to validate the handling of incorrect input data: - Incorrect input length. - Incorrect sign value (not 0 or 1). - Erroneous coding of field elements: one of the first three bits set up incorrectly. - Erroneous coding of field elements resulting in a correct element on the curve modulo p. - Erroneous coding of field elements with an incorrect extra bit in the decompressed encoding. - Point not on the curve. - Incorrect encoding of the point at infinity. - Input is beyond memory bounds. Tests for the sum of an arbitrary amount of points This section focuses on validating the summation functionality with an arbitrary number of points: - Generate random points on the curve and verify that the sum of a random permutation matches. - Generate random points on the curve and utilize another library to validate results. - Create points and cross-check the outcome with the `multiexp` function. - Generate random points from $G_1$ and confirm that the sum is also from $G_1$. Edge cases: - Empty input - Sum with the maximum number of elements - A single point ***Annotation:*** ```rust pub fn bls12381_p1_sum(&mut self, value_len: u64, value_ptr: u64, register_id: u64) -> Result; ``` #### bls12381_p2_sum ***Description:*** The function computes the sum of the signed elements on the BLS12-381 curve. It accepts an arbitrary number of pairs $(s_i, p_i)$, where $p_i \in E'(F_{p^2})$ represents a point on the elliptic curve and $s_i \in {0, 1}$ is the point's sign. The output is a single point from $E'(F_{p^2})$ equal to $\sum (-1)^{s_i}p_i$. The $E'(F_{p^2})$ curve, the points on the curve, the multiplication by -1, and the addition operation are all defined in the BLS12-381 Curve Specification section. Note: The function accepts any points on the curve and is not limited to points in $G_2$. ***Input:*** The sequence of pairs $(s_i, p_i)$, where $p_i \in E'(F_{p^2})$ is point and $s_i \in \textbraceleft 0, 1 \textbraceright$ represents a sign. Each point is encoded in decompressed form as $(x: F_{p^2}, y: F_{p^2})$, and the sign is encoded in one byte. The expected input size is 193*k bytes, interpreted as a byte concatenation of k slices, each slice representing the point sign alongside the uncompressed point from $E'(F_{p^2})$. More details are available in the Curve Points Encoding section. ***Output:*** The ERROR_CODE is returned. - ERROR_CODE = 0: the input is correct - Output: 192 bytes represent one point $\in E'(F_{p^2})$ in its decompressed form. In case of an empty input, it outputs the point at infinity (refer to the Curve Points Encoding section for more details). - ERROR_CODE = 1: - Points or signs are incorrectly encoded (refer to Curve points encoded section). - Point is not on the curve. ***Test cases:*** The test cases are identical to those of `bls12381_p1_sum`, with the only alteration being the substitution of points from $G_1$ and $E(F_p)$ with points from $G_2$ and $E'(F_{p^2})$. ***Annotation:*** ```rust pub fn bls12381_p2_sum(&mut self, value_len: u64, value_ptr: u64, register_id: u64) -> Result; ``` #### ***bls12381_g1_multiexp*** ***Description:*** The function accepts a list of pairs $(p_i, s_i)$, where $p_i \in G_1 \subset E(F_p)$ represents a point on the curve, and $s_i \in \mathbb{N}_0$ denotes a scalar. It calculates $\sum s_i \cdot p_i$. The scalar multiplication operation signifies the addition of that point a scalar number of times: $$ s \cdot p = \underbrace{p + p + \ldots + p}_{s} $$ The $E(F_p)$ curve, $G_1$ subgroup, points on the curve, and the addition operation are defined in the BLS12-381 Curve Specification section. Please note: - The function accepts only points from $G_1$. - The scalar is an arbitrary unsigned integer and can exceed the group order. - To enhance gas efficiency, the Pippenger’s algorithm[^25] can be utilized. ***Input:*** The sequence of pairs $(p_i, s_i)$, where $p_i \in G_1 \subset E(F_p)$ represents a point on the curve, and $s_i \in \mathbb{N}_0$ is a scalar. The expected input size is 128*k bytes, interpreted as byte concatenation of k slices. Each slice comprises the concatenation of an uncompressed point from $G_1 \subset E(F_p)$— 96 bytes, along with a scalar— 32 bytes. Further details are available in the Curve Points Encoding section. ***Output:*** The ERROR_CODE is returned. - ERROR_CODE = 0: the input is correct - Output: 96 bytes represent one point $\in G_1 \subset E(F_p)$ in its decompressed form. In case of an empty input, it outputs the point at infinity (refer to the Curve Points Encoding section for more details). - ERROR_CODE = 1: - Points are incorrectly encoded (refer to Curve points encoded section). - Point is not on the curve. - Point is not from $G_1$ ***Test cases:*** Tests for multiplication - Tests with known answers for multiplication from EIP-2537[^47],[^48]. - Random small scalar n and point P: - Check results with the sum function: `P + P + P + .. + P = n*P`. - Compare with results from another library. - Random scalar n and point P: - Verify against results from another library. - Implement multiplication by using the sum function and the double-and-add algorithm[^61]. Edge cases: - `group_order * P = 0` - `(scalar + groupt_order) * P = scalar * P` - `P + P + P .. + P = N*P` - `0 * P = 0` - `1 * P = P` - Scalar is a MAX_INT Tests for sum of two points These are identical test cases to those in the `bls12381_p1_sum` section, but only with points from $G_1$ subgroup. - Generate random points P and Q, then compare the results with the sum function. Tests for the sum of an arbitrary amount of points - Random number of points, random point values; compare results with the sum function. - Empty input. - Input of maximum size. Tests for the multiexp of an arbitrary amount of points - Tests with known answers from EIP-2537[^47],[^48] - Random number of points, scalars, and points: - Check with results from another library. - Check with raw implementation based on the sum function and the double-and-add algorithm. - Empty input - Maximum number of scalars and points. Tests for error cases - The same test cases as those in the `bls12381_p1_sum` section. - Points not from $G_1$. ***Annotation:*** ```rust pub fn bls12381_g1_multiexp( &mut self, value_len: u64, value_ptr: u64, register_id: u64, ) -> Result; ``` #### ***bls12381_g2_multiexp*** ***Description:*** The function takes a list of pairs $(p_i, s_i)$ as input, where $p_i \in G_2 \subset E'(F_{p^2})$ represents a point on the curve, and $s_i \in \mathbb{N}_0$ denotes a scalar. The function computes $\sum s_i \cdot p_i$. This scalar multiplication operation involves adding the point $p$ to itself a specified number of times: $$ s \cdot p = \underbrace{p + p + \ldots + p}_{s} $$ The $E'(F_{p^2})$ curve, $G_2$ subgroup, points on the curve, and the addition operation are defined in the BLS12-381 Curve Specification section. Please note: - The function accepts only points from $G_2$. - The scalar is an arbitrary unsigned integer and can exceed the group order. - To enhance gas efficiency, the Pippenger’s algorithm[^25] can be utilized. ***Input:*** the sequence of pairs $(p_i, s_i)$, where $p_i \in G_2 \subset E'(F_{p^2})$ is a point on the curve and $s_i \in \mathbb{N}_0$ is a scalar. The expected input size is `224*k` bytes, interpreted as the byte concatenation of `k` slices. Each slice is the concatenation of an uncompressed point from $G_2 \subset E'(F_{p^2})$ — `192` bytes and a scalar — `32` bytes. More details are in the Curve Points Encoding section. ***Output:*** The ERROR_CODE is returned. - ERROR_CODE = 0: the input is correct - Output: 192 bytes represent one point $\in G_2 \subset E'(F_{p^2})$ in its decompressed form. In case of an empty input, it outputs the point at infinity (refer to the Curve Points Encoding section for more details). - ERROR_CODE = 1: - Points are incorrectly encoded (refer to Curve points encoded section). - Point is not on the curve. - Point is not in $G_2$ subgroup. ***Test cases:*** The test cases are identical to those for `bls12381_g1_multiexp`, except that the points from $G_1$ and $E(F_p)$ are replaced with points from $G_2$ and $E'(F_{p^2})$ ***Annotation:*** ```rust pub fn bls12381_g2_multiexp( &mut self, value_len: u64, value_ptr: u64, register_id: u64, ) -> Result; ``` #### bls12381_map_fp_to_g1 ***Description:*** This function takes as input a list of field elements $a_i \in F_p$ and maps them to $G_1 \subset E(F_p)$. You can find the specification of this mapping function in the section titled 'Map to curve specification.' Importantly, this function does NOT perform the mapping of the byte string into $F_p$. The implementation of the mapping to $F_p$ may vary and can be effectively executed within the contract. ***Input:*** The function expects `48*k` bytes as input, representing a list of element from $F_p$ (unsigned integer $< p$). Additional information is available in the Curve Points Encoding section. ***Output:*** The ERROR_CODE is returned. - ERROR_CODE = 0: the input is correct - Output: `96*k` bytes - represents a list of points $\in G_1 \subset E(F_p)$ in decompressed format. Further information is available in the Curve Points Encoding section. - ERROR_CODE = 1: $a_i \ge p$. ***Test cases:*** Tests for general cases - Validate the results for known answers from EIP-2537[^47],[^48]. - Generate a random point $a$ from $F_p$: - Verify the result using another library. - Check that the resulting point lies on the curve in $G_1$. - Compare the results for $a$ and $-a$; they should share the same x-coordinates and have opposite y-coordinates. Edge cases: - $a = 0$ - $a = p - 1$ Tests for an arbitrary number of elements - Empty input - Maximum number of points. - Generate a random number of field elements and compare the result with another library. Tests for error cases - Input length is not divisible by 48: - Input is beyond memory bounds. - $a = p$ - Random number $\ge p$ ***Annotation:*** ```rust pub fn bls12381_map_fp_to_g1( &mut self, value_len: u64, value_ptr: u64, register_id: u64, ) -> Result; ``` #### bls12381_map_fp2_to_g2 ***Description:*** This function takes as input a list of elements $a_i \in F_{p^2}$ and maps them to $G_2 \subset E'(F_{p^2})$. You can find the mapping function specification in the "Map to Curve Specification" section. It's important to note that this function does NOT map byte strings into $F_{p^2}$. The implementation of the mapping to $F_{p^2}$ may vary and can be effectively executed within the contract. ***Input:*** the function takes as input `96*k` bytes — the elements from $F_{p^2}$ (two unsigned integers $< p$). Additional details can be found in the Curve Points Encoding section. ***Output:*** The ERROR_CODE is returned. - ERROR_CODE = 0: the input is correct - Output: `192*k` bytes - represents a list of points $\in G_2 \subset E'(F_{p^2})$ in decompressed format. More details are in the Curve Points Encoding section. - ERROR_CODE = 1: one of the values is not a valid extension field $F_{p^2}$ element ***Test cases:*** Tests for general cases - Validate the results for known answers from EIP-2537[^47],[^48] - Generate a random point $a$ from $F_{p^2}$: - Verify the result with another library. - Check that the resulting point lies in $G_2$. - Compare results for $a$ and $-a$; they should have the same x-coordinates and opposite y-coordinates. Edge cases: - $a = (0, 0)$ - $a = (p - 1, p - 1)$ Tests for an arbitrary number of elements - Empty input - Maximum number of points. - Generate a random number of field elements and compare the result with another library. Tests for error cases - Input length is not divisible by 96. - Input is beyond memory bounds. - $a = (0, p)$ - $a = (p, 0)$ - (random number $\ge p$, 0) - (0, random number $\ge p$) ***Annotation:*** ```rust pub fn bls12381_map_fp2_to_g2( &mut self, value_len: u64, value_ptr: u64, register_id: u64, ) -> Result; ``` #### bls12381_pairing_check ***Description:*** The pairing function is a bilinear function $e\colon G_1 \times G_2 \rightarrow G_T$, where $G_T \subset F_{q^{12}}$, which is used to verify BLS signatures/zkSNARKs. This function takes as input the sequence of pairs $(p_i, q_i)$, where $p_i \in G_1 \subset E(F_{p})$ and $q_i \in G_2 \subset E'(F_{p^2})$ and validates: $$ \prod e(p_i, q_i) = 1 $$ We don’t need to calculate the pairing function itself as the result would lie on a huge field, and in all known applications only this validation check is necessary. ***Input:*** A sequence of pairs $(p_i, q_i)$, where $p_i \in G_1 \subset E(F_{p})$ and $q_i \in G_2 \subset E'(F_{p^2})$. Each point is encoded in decompressed form. An expected input size of 288*k bytes is anticipated, interpreted as byte concatenation of k slices. Each slice comprises the concatenation of an uncompressed point from $G_1 \subset E(F_p)$ (occupying 96 bytes) and a point from $G_2 \subset E'(F_{p^2})$ (occupying 192 bytes). Additional details can be found in the Curve Points Encoding section. ***Output:*** The ERROR_CODE is returned. - ERROR_CODE = 0: the input is correct, the pairing result equals the multiplicative identity. - ERROR_CODE = 1: - Points encoded incorrectly (refer to the Curve Points Encoded section). - Point not on the curve. - Point not in $G_1/G_2$. - ERROR_CODE = 2: the input is correct, the pairing result doesn't equal the multiplicative identity. ***Test cases:*** Tests for one pair - Generate a random point $P \in G_1$: verify $e(P, \mathcal{O}) = 1$ - Generate a random point $Q \in G_2$: verify $e(\mathcal{O}, Q) = 1$ - Generate random points $P \ne \mathcal{O} \in G_1$ and $Q \ne \mathcal{O} \in G_2$: verify $e(P, Q) \ne 1$ Tests for two pairs - Generate random points $P \in G_1$, $Q \in G_2$ and random scalars $s_1, s_2$: - $e(P, Q) \cdot e(P, -Q) = 1$ - $e(P, Q) \cdot e(-P, Q) = 1$ - $e(s_1P, s_2Q) \cdot e(-s_2P, s_1Q) = 1$ - $e(s_1P, s_2Q) \cdot e(s_2P, -s_1Q) = 1$ - $g_1 \in G_1$, $g_2 \in G_2$ are generators defined in section 'BLS12-381 Curve Specification', r is the order of $G_1$ and $G_2$, and $p_1, p_2, q_1, q_2$ are randomly generated scalars: - if $p_1 \cdot q_1 + p_2 \cdot q_2 \not \equiv 0 (\mod r)$, verify $e(p_1 g_1, q_1 g_2) \cdot e(p_2 g_1, q_2 g_2) \ne 1$ - if $p_1 \cdot q_1 + p_2 \cdot q_2 \equiv 0 (\mod r)$, verify $e(p_1 g_1, q_1 g_2) \cdot e(p_2 g_1, q_2 g_2) = 1$ Tests for an arbitrary number of pairs - Empty input - Test with the maximum number of pairs - Tests using known answers from EIP-2537[^47],[^48] - For all possible values of 'n', generate random scalars $p_1 \cdots p_n$ and $q_1 \cdots q_n$ such that $\sum p_i \cdot q_i \not \equiv 0 (\mod r)$: - Verify $\prod e(p_i g_1, q_i g_2) \ne 1$ - For all possible values of 'n', generate random scalars $p_1 \cdots p_{n - 1}$ and $q_1 \cdots q_{n - 1}$: - Verify $(\prod e(p_i g_1, q_i g_2)) \cdot e(-(\sum p_i q_i) g_1, g_2) = 1$ - Verify $(\prod e(p_i g_1, q_i g_2)) \cdot e(g_1, -(\sum p_i q_i) g_2) = 1$ Tests for error cases - The first point is on the curve but not in $G_1$. - The second point is on the curve but not in $G_2$. - The input length is not divisible by 288. - The first point is not on the curve. - The second point is not on the curve. - Input length exceeds the memory limit. - Incorrect encoding of the point at infinity. - Incorrect encoding of a curve point: - Incorrect decompression bit. - Coordinates greater than or equal to 'p'. ***Annotation:*** ```rust pub fn bls12381_pairing_check(&mut self, value_len: u64, value_ptr: u64) -> Result; ``` #### bls12381_p1_decompress ***Description:*** The function decompresses compressed points from $E(F_p)$. It takes an arbitrary number of points $p_i \in E(F_p)$ in compressed format as input and outputs the same number of points from $E(F_p)$ in decompressed format. Further details about the decompressed and compressed formats are available in the Curve Points Encoding section. ***Input:*** A sequence of points $p_i \in E(F_p)$, with each point encoded in compressed form. An expected input size of 48*k bytes is anticipated, interpreted as the byte concatenation of k slices. Each slice represents the compressed point from $E(F_p)$. Additional details can be found in the Curve Points Encoding section. ***Output:*** The ERROR_CODE is returned. - ERROR_CODE = 0: the input is correct - Output: The sequence of points $p_i \in E(F_p)$, with each point encoded in decompressed form. An expected output of 96*k bytes, interpreted as the byte concatenation of k slices. Each slice represents the decompressed point from $E(F_p)$. k is the same as in the input. More details are available in the Curve Points Encoding section. - ERROR_CODE = 1: - Points are incorrectly encoded (refer to the Curve points encoded section). - Point is not on the curve. ***Test cases:*** Tests for decompressing a single point - Generate random points on the curve from $G_1$ and not from $G_1$: - Check that the uncompressed point lies on the curve. - Compare the result with another library. - Generate random points with a negative y: - Take the inverse and compare the y-coordinate. - Compare the result with another library. - Decompress a point on infinity. Tests for decompression of an arbitrary number of points - Empty input. - Maximum number of points. - Generate a random number of points on the curve and compare the result with another library. Tests for error cases - The input length is not divisible by 48. - The input is beyond memory bounds. - Point is not on the curve. - Incorrect decompression bit. - Incorrectly encoded point at infinity. - Point with a coordinate larger than 'p'. ***Annotation:*** ```rust pub fn bls12381_p1_decompress(&mut self, value_len: u64, value_ptr: u64, register_id: u64) -> Result; ``` #### bls12381_p2_decompress ***Description:*** The function decompresses compressed points from $E'(F_{p^2})$. It takes an arbitrary number of points $p_i \in E'(F_{p^2})$ in compressed format as input and outputs the same number of points from $E'(F_{p^2})$ in decompressed format. For more information about the decompressed and compressed formats, refer to the Curve Points Encoding section. ***Input:*** A sequence of points $p_i \in E'(F_{p^2})$, with each point encoded in compressed form. The expected input size is `96*k` bytes, interpreted as the byte concatenation of k slices. Each slice represents the compressed point from $E'(F_{p^2})$. Additional details are available in the Curve Points Encoding section. ***Output:*** The ERROR_CODE is returned. - ERROR_CODE = 0: the input is correct - Output: the sequence of point $p_i \in E'(F_{p^2})$, with each point encoded in decompressed form. The expected output is 192*k bytes, interpreted as the byte concatenation of k slices. `k` corresponds to the value specified in the input section. Each slice represents the decompressed point from $E'(F_{p^2})$. For more details, refer to the Curve Points Encoding section. - ERROR_CODE = 1: - Points are incorrectly encoded (refer to Curve points encoded section). - Point is not on the curve. ***Test cases:*** The same test cases as `bls12381_p1_decompress`, but with points from $G_2$, and the input length should be divisible by 96. ***Annotation:*** ```rust pub fn bls12381_p2_decompress(&mut self, value_len: u64, value_ptr: u64, register_id: u64) -> Result; ``` ## Reference Implementation Primarily, concerning integration with nearcore, our interest lies in Rust language libraries. The current implementations of BLS12-381 in Rust are: 1. ***Milagro Library*** [^29]. 2. ***BLST*** [^30][^31]. 3. ***Matter labs EIP-1962 implementation*** [^32] 4. ***zCash origin implementation*** [^33] 5. ***MCL Library*** [^34] 6. ***FileCoin implementation*** [^35] 7. ***zkCrypto*** [^36] To compile the list, we used links from EIP-2537[^43], the pairing-curves specification[^44], and an article containing benchmarks[^45]. This list might be incomplete, but it should encompass the primary BLS12-381 implementations. In addition, there are implementations in other languages that are less relevant to us in this context but can serve as references. 1. C++, ETH2.0 Client, ***Chia library***[^37] 2. Haskell, ***Adjoint Lib***[^38] 3. Go, ***Go-Ethereum***[^39] 4. JavaScript, ***Noble JS***[^40] 5. Go, ***Matter Labs Go EIP-1962 implementation***[^41] 6. C++, ***Matter Labs C++ EIP-1962 implementation***[^42] One of the possible libraries to use is the blst library[^30]. This library exhibits good performance[^45] and has undergone several audits[^55]. You can find a draft implementation in nearcore, which is based on this library, through this link[^54]. ## Security Implications The implementation's security depends on the chosen library's security, supporting operations with BLS curves. Within this NEP, a constant execution time for all operations isn't mandated. All the computations executed by smart contract are entirely public anyway, so there would be no advantage to a constant-time algorithm. BLS12-381 offers more security bits compared to the already existing pairing-friendly curve BN254. Consequently, the security of projects requiring a pairing-friendly curve will be enhanced. ## Alternatives In nearcore, host functions for another pairing-friendly curve, BN254, have already been implemented[^10]. Some projects[^20] might consider utilizing the supported curve as an alternative. However, recent research indicates that this curve provides less than 100 bits of security and is not recommended for use[^13]. Furthermore, projects involved in cross-chain interactions, like Rainbow Bridge, are mandated to employ the same curve as the target protocol, which, in the case of Ethereum, is currently BLS12-381[^3]. Consequently, there is no viable alternative to employing a different pairing-friendly curve. An alternative approach involves creating a single straightforward host function in nearcore for BLS signature verification. This was the initially proposed solution[^26]. However, this solution lacks flexibility[^28] for several reasons: (1) projects may utilize different hash functions; (2) some projects might employ the $G_1$ subgroup for public keys, while others use $G_2$; (3) the specifications for Ethereum 2.0 remain in draft, subject to potential changes; (4) instead of a more varied and adaptable set of functions (inspired by EIP-2537's precompiles), we are left with a single large function; (5) there will be no support for zkSNARKs verification. Another alternative is to perform BLS12-381 operations off-chain. In this scenario, applications utilizing the BLS curve will no longer maintain trustlessness. ## Future possibilities In the future, there might be support for working with various curves beyond just BLS12-381. In Ethereum, prior to EIP-2537[^15], there was a proposal, EIP-1962[^27], to introduce pairing-friendly elliptic curves in a versatile format, accommodating not only BLS curves but numerous others as well. However, this proposal wasn't adopted due to its extensive scope and complexity. Implementing every conceivable curve might not be practical, but it remains a potential extension worth considering. Another potential extension could involve supporting `hash_to_field` or `hash_to_curve` operations[^58]. Enabling their support would optimize gas usage for encoding messages into elements on the curve, which could be beneficial to BLS signatures. However, implementing the hash_to_field operation requires supporting multiple hashing algorithms simultaneously and doesn't demand a significant amount of gas for implementation within the contract. Therefore, these functions exceed the scope of this proposal. Additionally, a potential expansion might encompass supporting not only affine coordinates but also other coordinate systems, such as homogeneous or Jacobian projective coordinates. ## Consequences ### Positive - Projects currently utilizing BN254 will have the capability to transition to the BLS12-381 curve, thereby enhancing their security. - Trustless cross-chain interactions with blockchains employing BLS12-381 in protocols (like Ethereum 2.0) will become feasible. ### Neutral ### Negative - There emerges a dependency on a library that supports operations with BLS12-381 curves. - We'll have to continually maintain operations with BLS12-381 curves, even if vulnerabilities are discovered, and it becomes unsafe to use these curves. ### Backward Compatibility There are no backward compatibility questions. ## Changelog The previous NEP for supporting BLS signature based on BLS12-381[^26] [^1]: BLS 2002 [https://www.researchgate.net/publication/2894224_Constructing_Elliptic_Curves_with_Prescribed_Embedding_Degrees](https://www.researchgate.net/publication/2894224_Constructing_Elliptic_Curves_with_Prescribed_Embedding_Degrees) [^2]: ZCash protocol: [https://zips.z.cash/protocol/protocol.pdf](https://zips.z.cash/protocol/protocol.pdf) [^3]: Ethereum 2 specification: [https://github.com/ethereum/consensus-specs/blob/master/specs/phase0/beacon-chain.md](https://github.com/ethereum/consensus-specs/blob/master/specs/phase0/beacon-chain.md) [^4]: Dfinity: [https://internetcomputer.org/docs/current/references/ic-interface-spec#certificate](https://internetcomputer.org/docs/current/references/ic-interface-spec#certificate) [^5]: Tezos: [https://wiki.tezosagora.org/learn/futuredevelopments/layer2#zkchannels](https://web.archive.org/web/20210227221934/https://wiki.tezosagora.org/learn/futuredevelopments/layer2) [^6]: Filecoin: [https://spec.filecoin.io/](https://spec.filecoin.io/) [^7]: Specification of pairing friendly curves with a list of applications in the table: [https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-pairing-friendly-curves-09#name-adoption-status-of-pairing-](https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-pairing-friendly-curves-09#name-adoption-status-of-pairing-) [^8]: Specification of pairing friendly curves, the security level for BLS12-381: [https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-pairing-friendly-curves-09#section-4.2.1](https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-pairing-friendly-curves-09#section-4.2.1) [^9]: BN2005: [https://eprint.iacr.org/2005/133](https://eprint.iacr.org/2005/133) [^10]: NEP-98 for BN254 host functions on NEAR: [https://github.com/near/NEPs/issues/98](https://github.com/near/NEPs/issues/98) [^11]: BLS12-381 for the Rest of Us: [https://hackmd.io/@benjaminion/bls12-381](https://hackmd.io/@benjaminion/bls12-381) [^12]: BN254 for the Rest of Us: [https://hackmd.io/@jpw/bn254](https://hackmd.io/@jpw/bn254) [^13]: Some analytics of different curve security: [https://www.ietf.org/archive/id/draft-irtf-cfrg-pairing-friendly-curves-02.html#name-for-100-bits-of-security](https://www.ietf.org/archive/id/draft-irtf-cfrg-pairing-friendly-curves-02.html#name-for-100-bits-of-security) [^14]: ZCash Transfer from bn254 to bls12-381: [https://electriccoin.co/blog/new-snark-curve/](https://electriccoin.co/blog/new-snark-curve/) [^15]: EIP-2537 Precompiles for Ethereum for BLS12-381: [https://eips.ethereum.org/EIPS/eip-2537](https://eips.ethereum.org/EIPS/eip-2537) [^17]: Article about Rainbow Bridge [https://near.org/blog/eth-near-rainbow-bridge](https://near.org/blog/eth-near-rainbow-bridge) [^19]: Intro into zkSNARKs: [https://media.consensys.net/introduction-to-zksnarks-with-examples-3283b554fc3b](https://media.consensys.net/introduction-to-zksnarks-with-examples-3283b554fc3b) [^20]: Zeropool project: [https://zeropool.network/](https://zeropool.network/) [^24]: Precompiles on Aurora: [https://doc.aurora.dev/dev-reference/precompiles/](https://doc.aurora.dev/dev-reference/precompiles/) [^25]: Pippenger Algorithm: [https://github.com/wborgeaud/python-pippenger/blob/master/pippenger.pdf](https://github.com/wborgeaud/python-pippenger/blob/master/pippenger.pdf) [^26]: NEP-446 proposal for BLS-signature verification: [https://github.com/nearprotocol/neps/pull/446](https://github.com/nearprotocol/neps/pull/446) [^27]: EIP-1962 EC arithmetic and pairings with runtime definitions: [https://eips.ethereum.org/EIPS/eip-1962](https://eips.ethereum.org/EIPS/eip-1962) [^28]: Drawbacks of NEP-446: [https://github.com/near/NEPs/pull/446#pullrequestreview-1314601508](https://github.com/near/NEPs/pull/446#pullrequestreview-1314601508) [^29]: BLS12-381 Milagro: [https://github.com/sigp/incubator-milagro-crypto-rust/tree/057d238936c0cbbe3a59dfae6f2405db1090f474](https://github.com/sigp/incubator-milagro-crypto-rust/tree/057d238936c0cbbe3a59dfae6f2405db1090f474) [^30]: BLST: [https://github.com/supranational/blst](https://github.com/supranational/blst), [^31]: BLST EIP-2537 adaptation: [https://github.com/sean-sn/blst_eip2537](https://github.com/sean-sn/blst_eip2537) [^32]: EIP-1962 implementation matter labs Rust: https://github.com/matter-labs/eip1962 [^33]: zCash origin rust implementation: [https://github.com/zcash/zcash/tree/master/src/rust/src](https://github.com/zcash/zcash/tree/master/src/rust/src) [^34]: MCL library: [https://github.com/herumi/bls](https://github.com/herumi/bls) [^35]: filecoin/bls-signature: [https://github.com/filecoin-project/bls-signatures](https://github.com/filecoin-project/bls-signatures) [^36]: zkCrypto: [https://github.com/zkcrypto/bls12_381](https://github.com/zkcrypto/bls12_381), [https://github.com/zkcrypto/pairing](https://github.com/zkcrypto/pairing) [^37]: BLS12-381 code bases for ETH2.0 client Chia library C++: [https://github.com/Chia-Network/bls-signatures](https://github.com/Chia-Network/bls-signatures) [^38]: Adjoint Lib: [https://github.com/sdiehl/pairing](https://github.com/sdiehl/pairing) [^39]: Ethereum Go implementation for EIP-2537: [https://github.com/ethereum/go-ethereum/tree/master/core/vm/testdata/precompiles](https://github.com/ethereum/go-ethereum/tree/master/core/vm/testdata/precompiles) [^40]: Noble JS implementation: [https://github.com/paulmillr/noble-bls12-381](https://github.com/paulmillr/noble-bls12-381) [^41]: EIP-1962 implementation matter labs Go: https://github.com/kilic/eip2537, [^42]: EIP-1962 implementation matter labs C++: https://github.com/matter-labs-archive/eip1962_cpp [^43]: EIP-2537 with links: [https://github.com/matter-labs-forks/EIPs/blob/bls12_381/EIPS/eip-2537.md](https://github.com/matter-labs-forks/EIPs/blob/bls12_381/EIPS/eip-2537.md) [^44]: Pairing-friendly curves specification, crypto libs: [https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-pairing-friendly-curves-09#name-cryptographic-libraries](https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-pairing-friendly-curves-09#name-cryptographic-libraries) [^45]: Comparing different libs for pairing-friendly curves: [https://hackmd.io/@gnark/eccbench](https://hackmd.io/@gnark/eccbench) [^46]: Bench vectors from EIP2537: [https://eips.ethereum.org/assets/eip-2537/bench_vectors](https://eips.ethereum.org/assets/eip-2537/bench_vectors) [^47]: Metter Labs tests for EIP2537: [https://github.com/matter-labs/eip1962/tree/master/src/test/test_vectors/eip2537](https://github.com/matter-labs/eip1962/tree/master/src/test/test_vectors/eip2537) [^48]: Tests from Go Ethereum implementation: [https://github.com/ethereum/go-ethereum/tree/master/core/vm/testdata/precompiles](https://github.com/ethereum/go-ethereum/tree/master/core/vm/testdata/precompiles) [^51]: draft-irtf-cfrg-pairing-friendly-curves-11 [https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-pairing-friendly-curves-11#name-bls-curves-for-the-128-bit-](https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-pairing-friendly-curves-11#name-bls-curves-for-the-128-bit-) [^52]: Paper with BLS12-381: [https://eprint.iacr.org/2019/403.pdf](https://eprint.iacr.org/2019/403.pdf) [^53]: Zkcrypto points encoding: [https://github.com/zkcrypto/pairing/blob/0.14.0/src/bls12_381/README.md](https://github.com/zkcrypto/pairing/blob/0.14.0/src/bls12_381/README.md) [^54]: Draft PR for BLS12-381 operations in nearcore: [https://github.com/near/nearcore/pull/9317](https://github.com/near/nearcore/pull/9317) [^55]: Audit for BLST library: [https://research.nccgroup.com/wp-content/uploads/2021/01/NCC_Group_EthereumFoundation_ETHF002_Report_2021-01-20_v1.0.pdf](https://research.nccgroup.com/wp-content/uploads/2021/01/NCC_Group_EthereumFoundation_ETHF002_Report_2021-01-20_v1.0.pdf) [^58]: hash_to_curve and hash_to_field function: [https://datatracker.ietf.org/doc/html/rfc9380#name-hash_to_field-implementatio](https://datatracker.ietf.org/doc/html/rfc9380#name-hash_to_field-implementatio) [^59]: Implementation of BLS-signature based on these host functions: [https://github.com/olga24912/bls-signature-verificaion-poc/blob/main/src/lib.rs](https://github.com/olga24912/bls-signature-verificaion-poc/blob/main/src/lib.rs) [^60]: hash_to_field specification: [https://datatracker.ietf.org/doc/html/rfc9380#name-hash_to_field-implementatio](https://datatracker.ietf.org/doc/html/rfc9380#name-hash_to_field-implementatio) [^61]: double-and-add algorithm: [https://en.wikipedia.org/wiki/Exponentiation_by_squaring](https://en.wikipedia.org/wiki/Exponentiation_by_squaring) [^62]: RFC 9380 Hashing to Elliptic Curves specification: [https://www.rfc-editor.org/rfc/rfc9380](https://www.rfc-editor.org/rfc/rfc9380) [^63]: map_to_curve and clear_cofactor functions: [https://datatracker.ietf.org/doc/html/rfc9380#name-encoding-byte-strings-to-el](https://datatracker.ietf.org/doc/html/rfc9380#name-encoding-byte-strings-to-el) [^64]: Specification of parameters for BLS12-381 G1: [https://datatracker.ietf.org/doc/html/rfc9380#name-bls12-381-g1](https://datatracker.ietf.org/doc/html/rfc9380#name-bls12-381-g1) [^65]: Specification of parameters for BLS12-381 G2: [https://datatracker.ietf.org/doc/html/rfc9380#name-bls12-381-g2](https://datatracker.ietf.org/doc/html/rfc9380#name-bls12-381-g2) ================================================ FILE: neps/nep-0491.md ================================================ --- NEP: 491 Title: Non-Refundable Storage Staking Authors: Jakob Meier Status: Final DiscussionsTo: https://gov.near.org/t/proposal-locking-account-storage-refunds-to-avoid-faucet-draining-attacks/34155 Type: Protocol Track Version: 1.0.0 Created: 2023-07-24 LastUpdated: 2023-07-26 --- ## Summary Non-refundable storage allows to create accounts with arbitrary state for users, without being susceptible to refund abuse. This is done by tracking non-refundable balance in a separate field of the account. This balance is only useful for storage staking and otherwise can be considered burned. ## Motivation Creating new accounts on chain costs a gas fee and a storage staking fee. The more state is added to the account, the higher the storage staking fees. When deploying a contract on the account, it can quickly go above 1 NEAR per account. Some business models are okay with paying that fee for users upfront, just to get them onboarded. However, if a business does that today, their users can delete their new accounts and spend the tokens intended for storage staking in other ways. Since this is free for the user, they are financially incentivized to repeat this action for as long as the business has funds left in the faucet. The protocol should allow to create accounts in a way that is not susceptible to such refund abuse. This would at least change the incentives such that creating fake users is no longer profitable. Non-refundable storage staking is a further improvement over [NEP-448](https://github.com/near/NEPs/pull/448) (Zero Balance Accounts) which addressed the same issue but is limited to 770 bytes per account. By lifting the limit, sponsored accounts can be used in combination with smart contracts. ## Specification Users can opt-in to nonrefundable storage when creating new accounts. For that, we use the new action `ReserveStorage`. ```rust pub enum Action { ... ReserveStorage(ReserveStorageAction), ... } ``` To create a named account today, the typical pattern is a transaction with `CreateAccount`, `Transfer`, and `AddKey`. To make the funds nonrefundable, we can use action `ReserveStorage` like this: ```json "Actions": { "CreateAccount": {}, "ReserveStorage": { "deposit": "1000000000000000000000000" }, "AddKey": { "public_key": "...", "access_key": "..." } } ``` Adding a `Transfer` action allows the combination of nonrefundable balance and refundable balance. This allows the user to make calls where they need to attach balance, for example an FT transfer which requires 1 yocto NEAR. ```json "Actions": { "CreateAccount": {}, "ReserveStorage": { "deposit": "1000000000000000000000000" }, "Transfer": { "deposit": "100" }, "AddKey": { "public_key": "...", "access_key": "..." } } ``` To create implicit accounts, the current protocol requires a single `Transfer` action without further actions in the same transaction and this has not changed with this proposal: ```json "Actions": { "CreateAccount": {}, "Transfer": { "deposit": "0" }, } ``` If a non-refundable transfer arrives at an account that already exists, it will fail and the funds are returned to the predecessor. Finally, when querying an account for its balance, there will be an additional field `nonrefundable` in the output. Wallets will need to decide how they want to show it. They could, for example, add a new field called "non-refundable storage credits". ```js // Account near { "amount": "68844924385676812880674962949", "block_hash": "3d6SisRc5SuwrkJnLwQb3W5pWitZKCjGhiKZuc6tPpao", "block_height": 97314513, "code_hash": "Dmi6UTRYTT3eNirp8ndgDNh8kYk2T9SZ6PJZDUXB1VR3", "locked": "0", "storage_paid_at": 0, "storage_usage": 2511772, "formattedAmount": "68,844.924385676812880674962949", // this is new "nonrefundable": "0" } ``` ## Reference Implementation On the protocol side, we need to add new action: ```rust enum Action { CreateAccount(CreateAccountAction), DeployContract(DeployContractAction), FunctionCall(FunctionCallAction), Transfer(TransferAction), Stake(StakeAction), AddKey(AddKeyAction), DeleteKey(DeleteKeyAction), DeleteAccount(DeleteAccountAction), Delegate(super::delegate_action::SignedDelegateAction), // this gets added in the end ReserveStorage(ReserveStorageAction), } ``` and handle the new action in the `apply_action` call. Further, we have to update the account meta data representation in the state trie to track the non-refundable storage. ```rust pub struct Account { amount: Balance, locked: Balance, // this field is new nonrefundable: Balance, code_hash: CryptoHash, storage_usage: StorageUsage, // the account version will be increased from 1 to 2 version: AccountVersion, } ``` The field `nonrefundable` must be added to the normal `amount` and the `locked` balance calculate how much state the account is allowed to use. The new formula to check storage balance therefore becomes ```rust amount + locked + nonrefundable >= storage_usage * storage_amount_per_byte ``` For old accounts that don't have the new field, the non-refundable balance is always zero. Adding non-refundable balance later is not allowed. If a transfer is made to an account that already existed before the receipt's actions are applied, execution must fail with `ActionErrorKind::OnlyReserveStorageOnAccountCreation{ account_id: AccountId }`. Conceptually, these are all changes on the protocol level. However, unfortunately, the account version field is not currently serialized, hence not included in the on-chain state. Therefore, as the last change necessary for this NEP, we also introduce a new serialization format for new accounts. ```rust // new serialization format for `struct Account` // new: prefix with a sentinel value to detect V1 accounts, they will have // a real balance here which is smaller than u128::MAX writer.serialize(u128::MAX)?; // new: include version number (u8) for accounts with version 2 or more writer.serialize(version)?; writer.serialize(amount)?; writer.serialize(locked)?; writer.serialize(code_hash)?; writer.serialize(storage_usage)?; // new: this is the field we added, the type is u128 like other balances writer.serialize(nonrefundable)?; ``` Note that we are not migrating old accounts. Accounts created as version 1 will remain at version 1. A proof of concept implementation for nearcore is available in this PR: https://github.com/near/nearcore/pull/9346 ## Security Implications We were not able to come up with security relevant implications. ## Alternatives There are small variations in the implementation, and then there are completely different ways to look at the problem. Let's start with the variations. ### Variation: Allow adding nonrefundable balance to existing accounts Instead of failing when a non-refundable transfer arrives at an existing account, we could add the balance to the existing non-refundable balance. This would be more flexible to use. A business could easily add more funds for storage even after account creation. The problems are in the implementation details. It would allow to add non-refundable storage to existing accounts, which would require some form of migration of the all accounts in the state trie. This is impractical, as we have to iterate over all existing accounts and re-merklize. That's infeasible within a single block time and stopping the chain would be disruptive. We could maybe migrate lazily, i.e. read account version 1 and automatically convert it to version 2. However, that would break the assumption that every logical value in the merkle trie has a unique borsh representation, as there would be a account version 1 and a version 2 borsh serialization that both map to the same logical version 2 value. This could lead to different representations of the same chunk in memory, which might be used in attacks to force a double-sign by innocent validators. It is not 100% clear to me, the author, if this is a problem we could work around. However, the complications it would involve do not seem to be worth it, given that in the feature discussions nobody saw it as critical to add non-refundable balance to existing accounts. ### Variation: Allow refunds to original sponsor Instead of complete non-refundability, the tokens reserved for storage staking could be returned to the original account that created the account when an account is deleted. The community discussions ended with the conclusion that this feature would probably not be used and we should not implement it until there is real demand for it. ### Alternative: Don't use smart contracts on user accounts Instead of deploying contracts on the user account, one could build a similar solution that uses zero balance accounts and a single master contract that performs all smart contract functionality required. This master contract can implement the [Storage Management] (https://nomicon.io/Standards/StorageManagement) standard to limit storage usage per user. This solution is not as flexible. The master account cannot make cross-contract function calls with the user id as the predecessor. ### Alternative: Move away from storage staking We could also abandon the concept of storage staking entirely. However, coming up with a scalable, sustainable solution that does not suffer from the same refund problems is hard. One proposed design is a combination of zero balance accounts and code sharing between contracts. Basically, if somehow the deployed code is stored in a way that does not require storage staking by the user themself, maybe the per-user state is small enough to fit in the 770 bytes limit of zero balance accounts. (Questionable for non-trivial use cases.) This alternative is much harder to design and implement. The proposal that has gotten the furthest so far is [Ephemeral Storage](https://github.com/near/NEPs/pull/485), which is pretty complicated and does not have community consensus yet. Nobody is currently working on moving it forward. While we could wait for that to eventually make progress, in the meantime, the community is held back in their innovation because of the refund problem. ### Alternative: Using a proxy account As suggested by [@mfornet](https://github.com/near/NEPs/pull/491#discussion_r1349496234) another alternative is using a proxy account approach where the business creates an account with a deployed contract that has Regular (user has full access key) and Restricted mode (user doesn't have full access key and cannot delete account). In restricted mode, the user has a `FunctionCallKey` which allows the user to call methods of the contract that controls the `FullAccessKey` and allows the user some functionality but not all, e.g. not allowing account deletion. The user in restricted mode could also upgrade an account by sending the initial amount of NEAR deposited by the account creator and will attach a new `FullAccessKey`. The downside of this idea is additional complexity on the tooling side because actions like adding access keys to the account need to be converted to function calls instead of being direct actions. And the complexity on the business side is that it needs to include the proxy logic with their business logic in the same contract, increasing the complexity of development. ### Alternative: Granular access key Another suggestion is introducing another key type `GranularAccessKey`. This alternative includes a protocol change that introduces a new kind of access key which can have granular permissions set on, it e.g. not being able to delete an account. The business side gives this key to the user, and with this key comes a set of permissions that the user can do. The user can also call `Upgrade` and get `FullAccessKey` by paying for the initial amount which funded the account creation. The drawback of this approach is that it requires that the business side would have to handle the logic around `GranularAccessKey` and the `Upgrade` method making the usage more complex. ## Future possibilities - We might want to add the possibility to make non-refundable balance transfers from within a smart contract. This would require changes to the WASM smart contract to host interface. Since removing anything from there is virtually impossible, we shouldn't be too eager in adding it there but if there is demand for it, we certainly can do it without much trouble. - We could later add the possibility to refund the non-refundable tokens to the account who sent the tokens initially. - We could allow sending non-refundable balance to existing accounts. - If (cheap) code sharing between contracts is implemented in the future, this proposal will most likely work well in combination with that. Per-user data will still need to be paid for by the user, which could be sponsored as non-refundable balance without running into refund abuse. ## Consequences ### Positive - Businesses can sponsor new user accounts without the user being able to steal any tokens. ### Neutral - Non-refundable tokens are removed from the circulating supply, i.e. burnt. ### Negative - Understanding a user's balance become even more complicated than it already is. Instead of only `amount` and `locked`, there will be a third component. - There is no incentive anymore to delete an account and its state when the backing tokens are not refundable. ### Backwards Compatibility We believe this can be implemented with full backwards compatibility. ## Unresolved Issues (Optional) All of these issues already have a proposed solution above. But nevertheless, these points are likely to be challenged / discussed: - Should we allow adding non-refundable balance to existing accounts? (proposal: no) - Should we allow adding more non-refundable balance after account creation? (proposal: no) - Should this NEP include a host function to send non-refundable balance from smart contracts? (proposal: no) - How should a wallet display non-refundable balances? (proposal: up to wallet providers, probably a new separate field) ## Changelog ### 1.0.0 - Initial Version > Placeholder for the context about when and who approved this NEP version. #### Benefits > List of benefits filled by the Subject Matter Experts while reviewing this > version: - Benefit 1 - Benefit 2 #### Concerns > Template for Subject Matter Experts review for this version: Status: New | > Ongoing | Resolved | # | Concern | Resolution | Status | | --: | :------ | :--------- | -----: | | 1 | | | | | 2 | | | | ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0492.md ================================================ --- NEP: 492 Title: Restrict creation of Ethereum Addresses Authors: Bowen Wang Status: Final DiscussionsTo: https://github.com/near/NEPs/pull/492 Type: Protocol Version: 0.0.0 Created: 2023-07-27 LastUpdated: 2023-07-27 --- ## Summary This proposal aims to restrict the creation of top level accounts (other than implicit accounts) on NEAR to both prevent loss of funds due to careless user behaviors and scams and create possibilities for future interopability solutions. ## Motivation Today an [Ethereum address](https://ethereum.org/en/developers/docs/accounts/) such as "0x32400084c286cf3e17e7b677ea9583e60a000324" is a valid account on NEAR and because it is longer than 32 characters, anyone can create such an account. This has unfortunately caused a few incidents where users lose their funds due to either a scam or careless behaviors. For example, when a user withdraw USDT from an exchange to their NEAR account, it is possible that they think they withdraw to Ethereum and therefore enter their Eth address. If this address exists on NEAR, then the user would lose their fund. A malicious actor could exploit this can create known Eth smart contract addresses on NEAR to trick users to send tokens to those addresses. With the proliferation of BOS gateways, including Ethereum ones, such exploits may become more common as users switch between NEAR wallets and Ethereum wallets (mainly metamask). In addition to prevent loss of funds for users, this change allows the possibility of Ethereum wallets supporting NEAR transactions, which could enable much more adoption of NEAR. The exact details of how that would be done is outside the scope of this proposal. There are currently ~5000 Ethereum addresses already created on NEAR. It is also outside the scope of this proposal to discuss what to do with them. ## Specification The proposed change is quite simple. Only the protocol registrar account can create top-level accounts that are not implicit accounts ## Reference Implementation The implementation roughly looks as follows: ```Rust fn action_create_account(...) { ... if account_id.is_top_level() && !account_id.is_implicit() && predecessor_id != &account_creation_config.registrar_account_id { // Top level accounts that are not implicit can only be created by registrar result.result = Err(ActionErrorKind::CreateAccountOnlyByRegistrar { account_id: account_id.clone(), registrar_account_id: account_creation_config.registrar_account_id.clone(), predecessor_id: predecessor_id.clone(), } .into()); return; } ... } ``` ## Alternatives There does not appear to be a good alternative for this problem. ## Future possibilities Ethereum wallets such as Metamask could potentially support NEAR transactions through meta transactions. ## Consequences In the short term, no new top-level accounts would be allowed to be created, but this change would not create any problem for users. ### Backwards Compatibility For Ethereum addresses specifically, there are ~5000 existing ones, but this proposal per se do not deal with existing accounts. ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0508.md ================================================ --- NEP: 508 Title: Resharding v2 Authors: Waclaw Banasik, Shreyan Gupta, Yoon Hong Status: Final DiscussionsTo: https://github.com/near/nearcore/issues/8992 Type: Protocol Version: 1.0.0 Created: 2023-09-19 LastUpdated: 2023-11-14 --- ## Summary This proposal introduces a new implementation for resharding and a new shard layout for the production networks. In essence, this NEP is an extension of [NEP-40](https://github.com/near/NEPs/blob/master/specs/Proposals/0040-split-states.md), which was focused on splitting one shard into multiple shards. We are introducing resharding v2, which supports one shard splitting into two within one epoch at a pre-determined split boundary. The NEP includes performance improvement to make resharding feasible under the current state as well as actual resharding in mainnet and testnet (To be specific, splitting the largest shard into two). While the new approach addresses critical limitations left unsolved in NEP-40 and is expected to remain valid for foreseeable future, it does not serve all use cases, such as dynamic resharding. ## Motivation Currently, NEAR protocol has four shards. With more partners onboarding, we started seeing that some shards occasionally become over-crowded with respect to total state size and number of transactions. In addition, with state sync and stateless validation, validators will not need to track all shards and validator hardware requirements can be greatly reduced with smaller shard size. With future in-memory tries, it's also important to limit the size of individual shards. ## Specification ### High level assumptions * Flat storage is enabled. * Shard split boundary is predetermined and hardcoded. In other words, necessity of shard splitting is manually decided. * For the time being resharding as an event is only going to happen once but we would still like to have the infrastructure in place to handle future resharding events with ease. * Merkle Patricia Trie is the underlying data structure for the protocol state. * Epoch is at least 6 hrs long for resharding to complete. ### High level requirements * Resharding must be fast enough so that both state sync and resharding can happen within one epoch. * Resharding should work efficiently within the limits of the current hardware requirements for nodes. * Potential failures in resharding may require intervention from node operator to recover. * No transaction or receipt must be lost during resharding. * Resharding must work regardless of number of existing shards. * No apps, tools or code should hardcode the number of shards to 4. ### Out of scope * Dynamic resharding * automatically scheduling resharding based on shard usage/capacity * automatically determining the shard layout * Merging shards or boundary adjustments * Shard reshuffling ### Required protocol changes A new protocol version will be introduced specifying the new shard layout which would be picked up by the resharding logic to split the shard. ### Required state changes * For the duration of the resharding the node will need to maintain a snapshot of the flat state and related columns. As the main database and the snapshot diverge this will cause some extent of storage overhead. * For the duration of the epoch before the new shard layout takes effect, the node will need to maintain the state and flat state of shards in the old and new layout at the same time. The State and FlatState columns will grow up to approx 2x the size. The processing overhead should be minimal as the chunks will still be executed only on the parent shards. There will be increased load on the database while applying changes to both the parent and the children shards. * The total storage overhead is estimated to be on the order of 100GB for mainnet RPC nodes and 2TB for mainnet archival nodes. For testnet the overhead is expected to be much smaller. ### Resharding flow * The new shard layout will be agreed on offline by the protocol team and hardcoded in the reference implementation. * The first resharding will be scheduled soon after this NEP is merged. The new shard layout boundary accounts will be: ```["aurora", "aurora-0", "kkuuue2akv_1630967379.near", "tge-lockup.sweat"]```. * Subsequent reshardings will be scheduled as needed, without further NEPs, unless significant changes are introduced. * In epoch T, past the protocol version upgrade date, nodes will vote to switch to the new protocol version. The new protocol version will contain the new shard layout. * In epoch T, in the last block of the epoch, the EpochConfig for epoch T+2 will be set. The EpochConfig for epoch T+2 will have the new shard layout. * In epoch T + 1, all nodes will perform the state split. The child shards will be kept up to date with the blockchain up until the epoch end first via catchup, and later as part of block postprocessing state application. * In epoch T + 2, the chain will switch to the new shard layout. ## Reference Implementation The implementation heavily re-uses the implementation from [NEP-40](https://github.com/near/NEPs/blob/master/specs/Proposals/0040-split-states.md). Below are listed the major differences and additions. ### Code pointers to the proposed implementation * [new shard layout](https://github.com/near/nearcore/blob/c9836ab5b05c229da933d451fe8198d781f40509/core/primitives/src/shard_layout.rs#L161) * [the main logic for splitting states](https://github.com/near/nearcore/blob/c9836ab5b05c229da933d451fe8198d781f40509/chain/chain/src/resharding.rs#L280) * [the main logic for applying chunks to split states](https://github.com/near/nearcore/blob/c9836ab5b05c229da933d451fe8198d781f40509/chain/chain/src/update_shard.rs#L315) * [the main logic for garbage collecting state from parent shard](https://github.com/near/nearcore/blob/c9836ab5b05c229da933d451fe8198d781f40509/chain/chain/src/store.rs#L2335) ### Flat Storage The old implementation of resharding relied on iterating over the full trie state of the parent shard in order to build the state for the children shards. This implementation was suitable at the time but since then the state has grown considerably and this implementation is now too slow to fit within a single epoch. The new implementation relies on iterating through the flat storage in order to build the children shards quicker. Based on benchmarks, splitting the largest shard by using flat storage can take around 15 min without throttling and around 3 hours with throttling to maintain the block production rate. The new implementation will also propagate the flat storage for the children shards and keep it up to date with the chain until the switch to the new shard layout in the next epoch. The old implementation didn't handle this case because the flat storage didn't exist back then. In order to ensure consistent view of the flat storage while splitting the state the node will maintain a snapshot of the flat state and related columns as of the last block of the epoch prior to resharding. The existing implementation of flat state snapshots used in State Sync will be used for this purpose. ### Handling receipts, gas burnt and balance burnt When resharding, extra care should be taken when handling receipts in order to ensure that no receipts are lost or duplicated. The gas burnt and balance burnt also need to be correctly handled. The old resharding implementation for handling receipts, gas burnt and balance burnt relied on the fact in the first resharding there was only a single parent shard to begin with. The new implementation will provide a more generic and robust way of reassigning the receipts to the child shards, gas burnt, and balance burnt, that works for arbitrary splitting of shards, regardless of the previous shard layout. ### New shard layout The first release of the resharding v2 will contain a new shard layout where one of the existing shards will be split into two smaller shards. Furthermore additional reshardings can be scheduled in subsequent releases without additional NEPs unless the need for it arises. A new shard layout can be determined and will be scheduled and executed with the next protocol upgrade. Resharding will typically happen by splitting one of the existing shards into two smaller shards. The new shard layout will be created by adding a new boundary account that will be determined by analysing the storage and gas usage metrics within the shard and selecting a point that will divide the shard roughly in half in accordance to the mentioned metrics. Other metrics can also be used based on requirements. ### Removal of Fixed shards Fixed shards was a feature of the protocol that allowed for assigning specific accounts and all of their recursive sub accounts to a predetermined shard. This feature was only used for testing and was never used in production. Fixed shards feature unfortunately breaks the contiguity of shards and is not compatible with the new resharding flow. A sub account of a fixed shard account can fall in the middle of account range that belongs to a different shard. This property of fixed shards made it particularly hard to reason about and implement efficient resharding. For example in a shard layout with boundary accounts [`b`, `d`] the account space is cleanly divided into three shards, each spanning a contiguous range and account ids: * 0 - `:b` * 1 - `b:d` * 2 - `d:` Now if we add a fixed shard `f` to the same shard layout, then any we'll have 4 shards but neither is contiguous. Accounts such as `aaa.f`, `ccc.f`, `eee.f` that would otherwise belong to shards 0, 1 and 2 respectively are now all assigned to the fixed shard and create holes in the shard account ranges. It's also worth noting that there is no benefit to having accounts colocated in the same shard. Any transaction or receipt is treated the same way regardless of crossing shard boundary. This was implemented ahead of this NEP and the fixed shards feature was **removed**. ### Garbage collection In epoch T+2 once resharding is completed, we can delete the trie state and the flat state related to the parent shard. In practice, this is handled as part of the garbage collection code. While garbage collecting the last block of epoch T+1, we go ahead and clear all the data associated with the parent shard from the trie cache, flat storage, and RocksDB state associated with trie state and flat storage. ### Transaction pool The transaction pool is sharded i.e. it groups transactions by the shard where each transaction should be converted to a receipt. The transaction pool was previously sharded by the ShardId. Unfortunately ShardId is insufficient to correctly identify a shard across a resharding event as ShardIds change domain. The transaction pool was migrated to group transactions by ShardUId instead, and a transaction pool resharding was implemented to reassign transaction from parent shard to children shards right before the new shard layout takes effect. The ShardUId contains the version of the shard layout which allows differentiating between shards in different shard layouts. This was implemented ahead of this NEP and the transaction pool is now fully **migrated** to ShardUId. ## Alternatives ### Why is this design the best in the space of possible designs? This design is simple, robust, safe, and meets all requirements. ### What other designs have been considered and what is the rationale for not choosing them? #### Alternative implementations * Splitting the trie by iterating over the boundaries between children shards for each trie record type. This implementation has the potential to be faster but it is more complex and it would take longer to implement. We opted in for the much simpler one using flat storage given it is already quite performant. * Changing the trie structure to have the account id first and type of record later. This change would allow for much faster resharding by only iterating over the nodes on the boundary. This approach has two major drawbacks without providing too many benefits over the previous approach of splitting by each trie record type. 1) It would require a massive migration of trie. 2) We would need to maintain the old and the new trie structure forever. * Changing the storage structure by having the storage key to have the format of `account_id.node_hash`. This structure would make it much easier to split the trie on storage level because the children shards are simple sub-ranges of the parent shard. Unfortunately we found that the migration would not be feasible. * Changing the storage structure by having the key format as only node_hash and dropping the ShardUId prefix. This is a feasible approach but it adds complexity to the garbage collection and data deletion, specially when nodes would start tracking only one shard. We opted in for the much simpler one by using the existing scheme of prefixing storage entries by shard uid. #### Other considerations * Dynamic Resharding - we have decided to not implement the full dynamic resharding at this time. Instead we hardcode the shard layout and schedule it manually. The reasons are as follows: * We prefer incremental process of introducing resharding to make sure that it is robust and reliable, as well as give the community the time to adjust. * Each resharding increases the potential total load on the system. We don't want to allow it to grow until full sharding is in place and we can handle that increase. * Extended shard layout adjustments - we have decided to only implement shard splitting and not implement any other operations. The reasons are as follows: * In this iteration we only want to perform splitting. * The extended adjustments are currently not justified. Both merging and boundary moving may be useful in the future when the traffic patterns change and some shard become underutilized. In the nearest future we only predict needing to reduce the size of the heaviest shards. ### What is the impact of not doing this? We need resharding in order to scale up the system. Without resharding eventually shards would grow so big (in either storage or cpu usage) that a single node would not be able to handle it. Additionally, this clears up the path to implement in-memory tries as we need to store the whole trie structure in limited RAM. In the future smaller shard size would lead to faster syncing of shard data when nodes start tracking just one shard. ## Integration with State Sync There are two known issues in the integration of resharding and state sync: * When syncing the state for the first epoch where the new shard layout is used. In this case the node would need to apply the last block of the previous epoch. It cannot be done on the children shard as on chain the block was applied on the parent shards and the trie related gas costs would be different. * When generating proofs for incoming receipts. The proof for each of the children shards contains only the receipts of the shard but it's generated on the parent shard layout and so may not be verified. In this NEP we propose that resharding should be rolled out first, before any real dependency on state sync is added. We can then safely roll out the resharding logic and solve the above mentioned issues separately. We believe at least some of the issues can be mitigated by the implementation of new pre-state root and chunk execution design. ## Integration with Stateless Validation The Stateless Validation requires that chunk producers provide proof of correctness of the transition function from one state root to another. That proof for the first block after the new shard layout takes place will need to prove that the entire state split was correct as well as the state transition. In this NEP we propose that resharding should be rolled out first, before stateless validation. We can then safely roll out the resharding logic and solve the above mentioned issues separately. This issue was discussed with the stateless validation experts and we are cautiously optimistic that the integration will be possible. The most concerning part is the proof size and we believe that it should be small enough thanks to the resharding touching relatively small number of trie nodes - on the order of the depth of the trie. ## Future fast-followups ### Resharding should work even when validators stop tracking all shards As mentioned above under 'Integration with State Sync' section, initial release of resharding v2 will happen before the full implementation of state sync and we plan to tackle the integration between resharding and state sync after the next shard split (Won't need a separate NEP as the integration does not require protocol change.) ### Resharding should work after stateless validation is enabled As mentioned above under 'Integration with Stateless Validation' section, the initial release of resharding v2 will happen before the full implementation of stateless validation and we plan to tackle the integration between resharding and stateless validation after the next shard split (May need a separate NEP depending on implementation detail.) ## Future possibilities ### Further reshardings This NEP introduces both an implementation of resharding and an actual resharding to be done in the production networks. Further reshardings can also be performed in the future by adding a new shard layout and setting the shard layout for the desired protocol version in the `AllEpochConfig`. ### Dynamic resharding As noted above, dynamic resharding is out of scope for this NEP and should be implemented in the future. Dynamic resharding includes the following but not limited to: * Automatic determination of split boundary based on parameters like traffic, gas usage, state size, etc. * Automatic scheduling of resharding events ### Extended shard layout adjustments In this NEP we only propose supporting splitting shards. This operation should be more than sufficient for the near future but eventually we may want to add support for more sophisticated adjustments such as: * Merging shards together * Moving the boundary account between two shards ### Localization of resharding event to specific shard As of today, at the RocksDB storage layer, we have the ShardUId, i.e. the ShardId along with the ShardVersion, as a prefix in the key of trie state and flat state. During a resharding event, we increment the ShardVersion by one, and effectively remap all the current parent shards to new child shards. This implies we can't use the same underlying key value pairs for store and instead would need to duplicate the values with the new ShardUId prefix, even if a shard is unaffected and not split. In the future, we would like to potentially change the schema in a way such that only the shard that is splitting is impacted by a resharding event, so as to avoid additonal work done by nodes tracking other shards. ### Other useful features * Removal of shard uids and introducing globally unique shard ids * Account colocation for low latency across account call - In case we start considering synchronous execution environment, colocating associated accounts (e.g. cross contract call between them) in the same shard can increase the efficiency * Shard purchase/reservation - When someone wants to secure entirety of limitation on a single shard (e.g. state size limit), they can 'purchase/reserve' a shard so it can be dedicated for them (similar to how Aurora is set up) ## Consequences ### Positive * Workload across shards will be more evenly distributed. * Required space to maintain state (either in memory or in persistent disk) will be smaller. This is useful for in-memory tries. * State sync overhead will be smaller with smaller state size. ### Neutral * Number of shards would increase. * Underlying trie structure and data structure are not going to change. * Resharding will create dependency on flat state snapshots. * The resharding process, as of now, is not fully automated. Analyzing shard data, determining the split boundary, and triggering an actual shard split all need to be manually curated and tracked. ### Negative * During resharding, a node is expected to require more resources as it will first need to copy state data from the parent shard to the child shard, and then will have to apply trie and flat state changes twice, once for the parent shard and once for the child shards. * Increased potential for apps and tools to break without proper shard layout change handling. ### Backwards Compatibility Any light clients, tooling or frameworks external to nearcore that have the current shard layout or the current number of shards hardcoded may break and will need to be adjusted in advance. The recommended way for fixing it is querying an RPC node for the shard layout of the relevant epoch and using that information in place of the previously hardcoded shard layout or number of shards. The shard layout can be queried by using the `EXPERIMENTAL_protocol_config` rpc endpoint and reading the `shard_layout` field from the result. A dedicated endpoint may be added in the future as well. Within nearcore we do not expect anything to break with this change. Yet, shard splitting can introduce additional complexity on replayability. For instance, as target shard of a receipt and belonging shard of an account can change with shard splitting, shard splitting must be replayed along with transactions at the exact epoch boundary. ## Changelog [The changelog section provides historical context for how the NEP developed over time. Initial NEP submission should start with version 1.0.0, and all subsequent NEP extensions must follow [Semantic Versioning](https://semver.org/). Every version should have the benefits and concerns raised during the review. The author does not need to fill out this section for the initial draft. Instead, the assigned reviewers (Subject Matter Experts) should create the first version during the first technical review. After the final public call, the author should then finalize the last version of the decision context.] ### 1.0.0 - Initial Version > Placeholder for the context about when and who approved this NEP version. #### Benefits > List of benefits filled by the Subject Matter Experts while reviewing this version: * Benefit 1 * Benefit 2 #### Concerns > Template for Subject Matter Experts review for this version: > Status: New | Ongoing | Resolved | # | Concern | Resolution | Status | | --: | :------ | :--------- | -----: | | 1 | | | | | 2 | | | | ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0509.md ================================================ --- NEP: 509 Title: Stateless validation Stage 0 Authors: Robin Cheng, Anton Puhach, Alex Logunov, Yoon Hong Status: Final DiscussionsTo: https://docs.google.com/document/d/1C-w4FNeXl8ZMd_Z_YxOf30XA1JM6eMDp5Nf3N-zzNWU/edit?usp=sharing, https://docs.google.com/document/d/1TzMENFGYjwc2g5A3Yf4zilvBwuYJufsUQJwRjXGb9Xc/edit?usp=sharing Type: Protocol Version: 1.0.1 Created: 2023-09-19 LastUpdated: 2023-09-19 --- ## Summary The NEP proposes an solution to achieve phase 2 of sharding (where none of the validators needs to track all shards), with stateless validation, instead of the traditionally proposed approach of fraud proof and state rollback. The fundamental idea is that validators do not need to have state locally to validate chunks. * Under stateless validation, the responsibility of a chunk producer extends to packaging transactions and receipts and annotating them with state witnesses. This extended role will be called "chunk proposers". * The state witness of a chunk is defined to be a subset of the trie state, alongside its proof of inclusion in the trie, that is needed to execute a chunk. A state witness allows anyone to execute the chunk without having the state of its shard locally. * Then, at each block height, validators will be randomly assigned to a shard, to validate the state witness for that shard. Once a validator receives both a chunk and its state witness, it verifies the state transition of the chunk, signs a chunk endorsement and sends it to the block producer. This is similar to, but separate from, block approvals and consensus. * The block producer waits for sufficient chunk endorsements before including a chunk into the block it produces, or omits the chunk if not enough endorsements arrive in time. ## Motivation As phase 1 of sharding requires block producers to track all shards due to underlying security concerns, the team explored potential ways to achieve phase 2 of sharding, where none of the validators has to track all shards. The early design of phase 2 relied on the security assumption that as long as there is one honest validator or fisherman tracking a shard, the shard is secure; by doing so, it naturally relied on protocol's ability to handle challenges (when an honest validator or fisherman detects a malicious behavior and submits a proof of such), state rollbacks (when validators agree that the submitted challenge is valid), and slashing (to punish the malicious validator). While it sounds straightforward and simple on paper, the complex interactions between these abilities and the rest of the protocol led to concrete designs that were extremely complicated, involving several specific problems we still don't know how to solve. As a result, the team sought alternative approaches and concluded that stateless validation is the most realistic and promising one; the stateless validation approach does not assume the existence of a fishermen, does not rely on challenges, and never rolls back state. Instead, it relies on the assumption that a shard is secure if every single chunk in that shard is validated by a randomly sampled subset of all validators, to always produce valid chunks in the first place. ## Specification ### Assumptions * Not more than 1/3 of validators (by stake) is corrupted. * In memory trie is enabled - [REF](https://docs.google.com/document/d/1_X2z6CZbIsL68PiFvyrasjRdvKA_uucyIaDURziiH2U/edit?usp=sharing) * State sync is enabled (so that nodes can track different shards across epochs) * Merkle Patricia Trie continues to be the state trie implementation * Congestion Control is enabled - [NEP-539](https://github.com/near/NEPs/pull/539) ### Design requirements * No validator needs to track all shards. * Security of protocol must not degrade. * Validator assignment for both chunk validation and block validation should not create any security vulnerabilities. * Block processing time should not take significantly more than what it takes today. * Any additional load on network and compute should not negatively affect existing functionalities of any node in the blockchain. * The cost of additional network and compute should be acceptable. * Validator rewards should not be reduced. ### Design before NEP-509 The current high-level chunk production flow, excluding details and edge cases, is as follows: * Block producer at height `H`, `BP(H)`, produces block `B(H)` with chunks accessible to it and distributes it. * Chunk producer for shard `S` at height `H+1`, `CP(S, H+1)`, produces chunk `C(S, H+1)` based on `B(H)` and distributes it. * `BP(H+1)` collects all chunks at height `H+1` until certain timeout is reached. * `BP(H+1)` produces block `B(H+1)` with chunks `C(*, H+1)` accessible to it and distributes it. And the flow goes on for heights H+1, H+2, etc. The "induction base" is at genesis height, where genesis block with default chunks is accessible to everyone, so chunk producers can start right away from genesis height + 1. One can observe that there is no "chunk validation" step here. In fact, validity of chunks is implicitly guaranteed by **requirement for all block producers to track all shards**. To achieve phase 2 of sharding, we want to drop this requirement. For that, we propose the following changes to the flow: ### Design after NEP-509 * Chunk producer, in addition to producing a chunk, produces new `ChunkStateWitness` message. The `ChunkStateWitness` contains data which is enough to prove validity of the chunk's header that is being produced. * `ChunkStateWitness` proves to anyone, including those who track only block data and no shards, that this chunk header is correct. * `ChunkStateWitness` is not part of the chunk itself; it is distributed separately and is considered transient data. * The chunk producer distributes the `ChunkStateWitness` to a subset of **chunk validators** which are assigned for this shard. This is in addition to, and independent of, the existing chunk distribution logic (implemented by `ShardsManager`) today. * Chunk Validator selection and assignment are described below. * A chunk validator, upon receiving a `ChunkStateWitness`, validates the state witness and determines if the chunk header is indeed correctly produced. If so, it sends a `ChunkEndorsement` to the current block producer. * As the existing logic is today, the block producer for this block waits until either all chunks are ready, or a timeout occurs, and then proposes a block containing whatever chunks are ready. Now, the notion of readiness here is expanded to also having more than 2/3 of chunk endorsements by stake. * This means that if a chunk does not receive enough chunk endorsements by the timeout, it will not be included in the block. In other words, the block only contains chunks for which there is already a consensus of validity. **This is the key reason why we will no longer need fraud proofs / tracking all shards**. * The 2/3 fraction has the denominator being the total stake assigned to validate this shard, *not* the total stake of all validators. * The block producer, when producing the block, additionally includes the chunk endorsements (at least 2/3 needed for each chunk) in the block's body. The validity of the block is expanded to also having valid 2/3 chunk endorsements by stake for each chunk included in the block. * If a block fails validation because of not having the required chunk endorsements, it is considered a block validation failure for the purpose of Doomslug consensus, just like any other block validation failure. In other words, nodes will not apply the block on top of their blockchain, and (block) validators will not endorse the block. So the high-level specification can be described as the list of changes in the validator roles and responsibilities: * Block producers: * (Same as today) Produce blocks, (new) including waiting for chunk endorsements * (Same as today) Maintain chunk parts (i.e. participates in data availability based on Reed-Solomon erasure encoding) * (New) No longer require tracking any shard * (Same as today) Should have high barrier of entry (required stake) for security reasons, to make block double signing harder. * Chunk producers: * (Same as today) Produce chunks * (New) Produces and distributes state witnesses to chunk validators * (Same as today) Must track the shard it produces the chunk for * Block validators: * (Same as today) Validate blocks, (new) including verifying chunk endorsements * (Same as today) Vote for blocks with endorsement or skip messages * (New) No longer require tracking any shard * (Same as today) Must collectively have a majority of all the validator stake, for security reasons. * (Same as today) Should have high barrier of entry to keep `BlockHeader` size low, because it is proportional to the total byte size of block validator signatures; * (New) Chunk validators: * Validate state witnesses and sends chunk endorsements to block producers * Do not require tracking any shard * Must collectively have a majority of all the validator stake, to ensure the security of chunk validation. See the Validator Structure Change section below for more details. ### Out of scope * Resharding support. * Data size optimizations such as compression, for both chunk data and state witnesses, except basic optimizations that are practically necessary. * Separation of consensus and execution, where consensus runs independently from execution, and validators asynchronously perform state transitions after the transactions are proposed on the consensus layer, for the purpose of amortizing the computation and network transfer time. * ZK integration. * Underlying data structure change (e.g. verkle tree). ## Reference Implementation Here we carefully describe new structures and logic introduced, without going into too much technical details. ### Validator Structure Change #### Roles Currently, there are two different types of validators. Their responsibilities are defined as in the following pseudocode: ```python if index(validator) < 100: roles(validator).append("block producer") roles(validator).append("chunk producer") ``` The validators are ordered by non-increasing stake in the considered epoch. Here and below by "block production" we mean both production and validation. With stateless validation, this structure must change for several reasons: * Chunk production is the most resource consuming activity. * *Only* chunk production needs state in memory while other responsibilities can be completed via acquiring state witness * Chunk production does not have to be performed by all validators. Hence, to make transition seamless, we change the role of nodes out of top 100 to only validate chunks: ```python if index(validator) < 100: roles(validator).append("chunk producer") roles(validator).append("block producer") roles(validator).append("chunk validator") ``` The more stake validator has, the more **heavy** work it will get assigned. We expect that validators with higher stakes have more powerful hardware. With stateless validation, relative heaviness of the work changes. Comparing to the current order "block production" > "chunk production", the new order is "chunk production" > "block production" > "chunk validation". Shards are equally split among chunk producers: as in Mainnet on 12 Jun 2024 we have 6 shards, each shard would have ~16 chunk producers assigned. In the future, with increase in number of shards, we can generalise the assignment by saying that each shard should have `X` chunk producers assigned, if we have at least `X * S` validators. In such case, pseudocode for the role assignment would look as follows: ```python if index(validator) < X * S: roles(validator).append("chunk producer") if index(validator) < 100: roles(validator).append("block producer") roles(validator).append("chunk validator") ``` #### Rewards Reward for each validator is defined as `total_epoch_reward * validator_relative_stake * work_quality_ratio`, where: * `total_epoch_reward` is selected so that total inflation of the token is 2.5% per annum; * `validator_relative_stake = validator_stake / total_epoch_stake`; * `work_quality_ratio` is the measure of the work quality from 0 to 1. So, the actual reward never exceeds total reward, and when everyone does perfect work, they are equal. For the context of the NEP, it is enough to assume that `work_quality_ratio = avg_{role}({role}_quality_ratio)`. So, if node is both a block and chunk producer, we compute quality for each role separately and then take average of them. When epoch is finalized, all block headers in it uniquely determine who was expected to produce each block and chunk. Thus, if we define quality ratio for block producer as `produced_blocks/expected_blocks`, everyone is able to compute it. Similarly, `produced_chunks/expected_chunks` is a quality for chunk producer. It is more accurate to say `included_chunks/expected_chunks`, because inclusion of chunk in block is a final decision of a block producer which defines success here. Ideally, we could compute quality for chunk validator as `produced_endorsements/expected_endorsements`. Unfortunately, we won't do it in Stage 0 because: * Mask of endorsements is not part of the block header, and it would be a significant change; * Block producer doesn't have to wait for all endorsements to be collected, so it could be unfair to say that endorsement was not produced if block producer just went ahead. So for now we decided to compute quality for chunk validator as ratio of `included_chunks/expected_chunks`, where we iterate over chunks which node was expected to validate. It has clear drawbacks though: * chunk validators are not incentivized to validate the chunks, given they will be rewarded the same in either case; * if chunks are not produced at all, chunk validators will also be impacted. We plan to address them in the future releases. #### Kickouts In addition to that, if node performance is too poor, we want a mechanism to kick it out of the validator list, to ensure healthy protocol performance and validator rotation. Currently, we have a threshold for each role, and if for some role the same `{role}_quality_ratio` is lower than threshold, the node is kicked out. If we write this in pseudocode, ```python if validator is block producer and block_producer_quality_ratio < 0.8: kick out validator if validator is chunk producer and chunk_producer_quality_ratio < 0.8: kick out validator ``` For chunk validator, we apply absolutely the same formula. However, because: * the formula doesn't count endorsements explicitly * for chunk producers it kind of just makes chunk production condition stronger without adding value we apply it to nodes which **only validate chunks**. So, we add this line: ```python if validator is only chunk validator and chunk_validator_quality_ratio < 0.8: kick out validator ``` As we pointed out above, current formula `chunk_validator_quality_ratio` is problematic. Here it brings even a bigger issue: if chunk producers don't produce chunks, chunk validators will be kicked out as well, which impacts network stability. This is another reason to come up with the better formula. #### Shard assignment As chunk producer becomes the most important role, we need to ensure that every epoch has significant amount of healthy chunk producers. This is a **significant difference** with current logic, where chunk-only producers generally have low stake and their performance doesn't impact overall performance. The most challenging part of becoming a chunk producer for a shard is to download most recent shard state within previous epoch. This is called "state sync". Unfortunately, as of now, state sync is centralised on published snapshots, which is a major point of failure, until we don't have decentralised state sync. Because of that, we make additional change: if node was a chunk producer for some shard in the previous epoch, and it is a chunk producer for current epoch, it will be assigned to the same shard. This way, we minimise number of required state syncs at each epoch. The exact algorithm needs a thorough description to satisfy different edge cases, so we will just leave a link to full explanation: https://github.com/near/nearcore/issues/11213#issuecomment-2111234940. ### ChunkStateWitness The full structure is described [here](https://github.com/near/nearcore/blob/b8f08d9ded5b7cbae9d73883785902b76e4626fc/core/primitives/src/stateless_validation.rs#L247). Let's construct it sequentially together with explaining why every field is needed. Start from simple data: ```rust pub struct ChunkStateWitness { pub chunk_producer: AccountId, pub epoch_id: EpochId, /// The chunk header which this witness is proving. pub chunk_header: ShardChunkHeader, } ``` What is needed to prove `ShardChunkHeader`? The key function we have in codebase is [validate_chunk_with_chunk_extra_and_receipts_root](https://github.com/near/nearcore/blob/c2d80742187d9b8fc1bb672f16e3d5c144722742/chain/chain/src/validate.rs#L141). The main arguments there are `prev_chunk_extra: &ChunkExtra` which stands for execution result of previous chunk, and `chunk_header`. The most important field for `ShardChunkHeader` is `prev_state_root` - consider latest implementation `ShardChunkHeaderInnerV3`. It stands for state root resulted from updating shard for the previous block, which means applying previous chunk if there is no missing chunks. So, chunk validator needs some way to run transactions and receipts from the previous chunk. Let's call it a "main state transition" and add two more fields to state witness: ```rust /// The base state and post-state-root of the main transition where we /// apply transactions and receipts. Corresponds to the state transition /// that takes us from the pre-state-root of the last new chunk of this /// shard to the post-state-root of that same chunk. pub main_state_transition: ChunkStateTransition, /// The transactions to apply. These must be in the correct order in which /// they are to be applied. pub transactions: Vec, ``` where ```rust /// Represents the base state and the expected post-state-root of a chunk's state /// transition. The actual state transition itself is not included here. pub struct ChunkStateTransition { /// The block that contains the chunk; this identifies which part of the /// state transition we're talking about. pub block_hash: CryptoHash, /// The partial state before the state transition. This includes whatever /// initial state that is necessary to compute the state transition for this /// chunk. It is a list of Merkle tree nodes. pub base_state: PartialState, /// The expected final state root after applying the state transition. pub post_state_root: CryptoHash, } ``` Fine, but where do we take the receipts? Receipts are internal messages, resulting from transaction execution, sent between shards, and **by default** they are not signed by anyone. However, each receipt is an execution outcome of some transaction or other parent receipt, executed in some previous chunk. For every chunk, we conveniently store `prev_outgoing_receipts_root` which is a Merkle hash of all receipts sent to other shards resulting by execution of this chunk. So, for every receipt, there is a proof of its generation in some parent chunk. If there are no missing chunk, then it's enough to consider chunks from previous block. So we add another field: ```rust /// Non-strict superset of the receipts that must be applied, along with /// information that allows these receipts to be verifiable against the /// blockchain history. pub source_receipt_proofs: HashMap, ``` What about missing chunks though? Unfortunately, production and inclusion of any chunk **cannot be guaranteed**: * chunk producer may go offline; * chunk validators may not generate 2/3 endorsements; * block producer may not receive enough information to include chunk. Let's handle this case as well. First, each chunk producer needs not just to prove main state transition, but also all state transitions for latest missing chunks: ```rust /// For each missing chunk after the last new chunk of the shard, we need /// to carry out an implicit state transition. This is technically needed /// to handle validator rewards distribution. This list contains one for each /// such chunk, in forward chronological order. /// /// After these are applied as well, we should arrive at the pre-state-root /// of the chunk that this witness is for. pub implicit_transitions: Vec, ``` Then, while our shard was missing chunks, other shards could still produce chunks, which could generate receipts targeting our shards. So, we need to extend `source_receipt_proofs`. Field structure doesn't change, but we need to carefully pick range of set of source chunks, so different subsets will cover all source receipts without intersection. Let's say B2 is the block that contains the last new chunk of shard S before chunk which state transition we execute, and B1 is the block that contains the last new chunk of shard S before B2. Then, we will define set of blocks B as the contiguous subsequence of blocks B1 (EXCLUSIVE) to B2 (inclusive) in this chunk's chain (i.e. the linear chain that this chunk's parent block is on). Lastly, source chunks are all chunks included in blocks from B. The last caveat is **new** transactions introduced by chunk with `chunk_header`. As chunk header introduces `tx_root` for them, we need to check validity of this field as well. If we don't do it, malicious chunk producer can include invalid transaction, and if it gets its chunk endorsed, nodes which track the shard must either accept invalid transaction or refuse to process chunk, but the latter means that shard will get stuck. To validate new `tx_root`, we also need Merkle partial state to validate sender' balances, access keys, nonces, etc., which leads to two last fields to be added: ```rust pub new_transactions: Vec, pub new_transactions_validation_state: PartialState, ``` The logic to produce `ChunkStateWitness` is [here](https://github.com/near/nearcore/blob/b8f08d9ded5b7cbae9d73883785902b76e4626fc/chain/client/src/stateless_validation/state_witness_producer.rs#L79). Itself, it requires some minor changes to the logic of applying chunks, related to generating `ChunkStateTransition::base_state`. It is controlled by [this line](https://github.com/near/nearcore/blob/dc03a34101f77a17210873c4b5be28ef23443864/chain/chain/src/runtime/mod.rs#L977), which causes all nodes read during applying chunk to be put inside `TrieRecorder`. After applying chunk, its contents are saved to `StateTransitionData`. The validation logic is [here](https://github.com/near/nearcore/blob/b8f08d9ded5b7cbae9d73883785902b76e4626fc/chain/client/src/stateless_validation/chunk_validator/mod.rs#L85). First, it performs all validation steps for which access to `ChainStore` is required, `pre_validate_chunk_state_witness` is responsible for this. It is done separately because `ChainStore` is owned by a single thread. Then, it spawns a thread which runs computation-heavy `validate_chunk_state_witness` which main purpose is to apply chunk based on received state transitions and verify that execution results in chunk header are correct. If validation is successful, `ChunkEndorsement` is sent. ### ChunkEndorsement It is basically a triple of `(ChunkHash, AccountId, Signature)`. Receiving this message means that specific chunk validator account endorsed chunk with specific chunk hash. Ideally chunk validator would send chunk endorsement to just the next block producer at the same height for which chunk was produced. However, block at that height can be skipped and block producers at heights h+1, h+2, ... will have to pick up the chunk. To address that, we send `ChunkEndorsement` to all block producers at heights from `h` to `h+d-1`. We pick `d=5` as more than 5 skipped blocks in a row are very unlikely to occur. On block producer side, chunk endorsements are collected and stored in `ChunkEndorsementTracker`. Small **caveat** is that *sometimes* chunk endorsement may be received before chunk header which is required to understand that sender is indeed a validator of the chunk. Such endorsements are stored as *pending*. When chunk header is received, all pending endorsements are checked for validity and marked as *validated*. All endorsements received after that are validated right away. Finally, when block producer attempts to produce a block, in addition to checking chunk existence, it also checks that it has 2/3 endorsement stake for that chunk hash. To make chunk inclusion verifiable, we introduce [another version](https://github.com/near/nearcore/blob/cf2caa3513f58da8be758d1c93b0900ffd5d51d2/core/primitives/src/block_body.rs#L30) of block body `BlockBodyV2` which has new field `chunk_endorsements`. It is basically a `Vec>>` where element with indices `(s, i)` contains signature of i-th chunk validator for shard s if it was included and None otherwise. Lastly, we add condition to block validation, such that if chunk `s` was included in the block, then block body must contain 2/3 endorsements for that shard. This logic is triggered in `ChunkInclusionTracker` by methods [get_chunk_headers_ready_for_inclusion](https://github.com/near/nearcore/blob/6184e5dac45afb10a920cfa5532ce6b3c088deee/chain/client/src/chunk_inclusion_tracker.rs#L146) and couple similar ones. Number of ready chunks is returned by [num_chunk_headers_ready_for_inclusion](https://github.com/near/nearcore/blob/6184e5dac45afb10a920cfa5532ce6b3c088deee/chain/client/src/chunk_inclusion_tracker.rs#L178). ### Chunk validators selection Chunk validators will be randomly assigned to validate shards, for each block (or as we may decide later, for multiple blocks in a row, if required for performance reasons). A chunk validator may be assigned multiple shards at once, if it has sufficient stake. Each chunk validator's stake is divided into "mandates". There are full and partial mandates. The number of mandates per shard is a fixed parameter and the amount of stake per mandate is dynamically computed based on this parameter and the actual stake distribution; any remaining amount smaller than a full mandate is a partial mandate. A chunk validator therefore has zero or more full mandates plus up to one partial mandate. The list of full mandates and the list of partial mandates are then separately shuffled and partitioned equally (as in, no more than one mandate in difference between any two shards) across the shards. Any mandate assigned to a shard means that the chunk validator who owns the mandate is assigned to validate that shard. Because a chunk validator may have multiple mandates, it may be assigned multiple shards to validate. For Stage 0, we select **target amount of mandates per shard** to 68, which was a [result of the latest research](https://near.zulipchat.com/#narrow/stream/407237-core.2Fstateless-validation/topic/validator.20seat.20assignment/near/435252304). With this number of mandates per shard and 6 shards, we predict the protocol to be secure for 40 years at 90% confidence. Based on target number of mandates and total chunk validators stake, [here](https://github.com/near/nearcore/blob/696190b150dd2347f9f042fa99b844b67c8001d8/core/primitives/src/validator_mandates/mod.rs#L76) we compute price of a single full mandate for each new epoch using binary search. All the mandates are stored in new version of `EpochInfo` `EpochInfoV4` in [validator_mandates](https://github.com/near/nearcore/blob/164b7a367623eb651914eeaf1cbf3579c107c22d/core/primitives/src/epoch_manager.rs#L775) field. After that, for each height in the epoch, [EpochInfo::sample_chunk_validators](https://github.com/near/nearcore/blob/164b7a367623eb651914eeaf1cbf3579c107c22d/core/primitives/src/epoch_manager.rs#L1224) is called to return `ChunkValidatorStakeAssignment`. It is `Vec>` where s-th element corresponds to s-th shard in the epoch, contains ids of all chunk validator for that height and shard, alongside with its total mandate stake assigned to that shard. `sample_chunk_validators` basically just shuffles `validator_mandates` among shards using height-specific seed. If there are no more than 1/3 malicious validators, then by Chernoff bound the probability that at least one shard is corrupted is small enough. **This is a reason why we can split validators among shards and still rely on basic consensus assumption**. This way, everyone tracking block headers can compute chunk validator assignment for each height and shard. ### Size limits `ChunkStateWitness` is relatively large message. Given large number of receivers as well, its size must be strictly limited. If `ChunkStateWitness` for some state transition gets so uncontrollably large that it never can be handled by majority of validators, then its shard gets stuck. We try to limit the size of the `ChunkStateWitness` to 16 MiB. All the limits are described [in this section](https://github.com/near/nearcore/blob/b34db1e2281fbfe1d99a36b4a90df3fc7f5d00cb/docs/misc/state_witness_size_limits.md). Additionally, we have limit on currently stored partial state witnesses and chunk endorsements, because malicious chunk validators can spam these as well. ## State witness size limits A number of new limits will be introduced in order to keep the size of `ChunkStateWitness` reasonable. `ChunkStateWitness` contains all the incoming transactions and receipts that will be processed during chunk application and in theory a single receipt could be tens of megabatytes in size. Distributing a `ChunkStateWitness` this large to all chunk validators would be troublesome, so we limit the size and number of transactions, receipts, etc. The limits aim to keep the total uncompressed size of `ChunkStateWitness` under 16 MiB. There are two types of size limits: * Hard limit - The size must be below this limit, anything else is considered invalid. This is usually used in the context of having limits for a single item. * Soft limit - Things are added until the limit is exceeded, after that things stop being added. The last added thing is allowed to slightly exceed the limit. This is used in the context of having limits for a list of items. The limits are: * `max_transaction_size - 1.5 MiB` * All transactions must be below 1.5 MiB, otherwise they'll be considered invalid and rejected. * Previously was 4 MiB, now reduced to 1.5 MiB * `max_receipt_size - 4 MiB`: * All receipts must be below 4 MiB, otherwise they'll be considered invalid and rejected. * Previously there was no limit on receipt size. Set to 4 MiB, might be reduced to 1.5 MiB in the future to match the transaction limit. * `combined_transactions_size_limit - 4 MiB` * Hard limit on total size of transactions from this and previous chunk. `ChunkStateWitness` contains transactions from two chunks, this limit applies to the sum of their sizes. * `new_transactions_validation_state_size_soft_limit - 500 KiB` * Validating new transactions generates storage proof (recorded trie nodes), which has to be limited. Once transaction validation generates more storage proof than this limit, the chunk producer stops adding new transactions to the chunk. * `per_receipt_storage_proof_size_limit - 4 MB` * Executing a receipt generates storage proof. A single receipt is allowed to generate at most 4 MB of storage proof. This is a hard limit, receipts which generate more than that will fail. * `main_storage_proof_size_soft_limit - 3 MB` * This is a limit on the total size of storage proof generated by receipts in one chunk. Once receipts generate more storage proof than this limit, the chunk producer stops processing receipts and moves the rest to the delayed queue. * It's a soft limit, which means that the total size of storage proof could reach 7 MB (2.99MB + one receipt which generates 4MB of storage proof) * Due to implementation details it's hard to find the exact amount of storage proof generated by a receipt, so an upper bound estimation is used instead. This upper bound assumes that every removal generates additional 2000 bytes of storage proof, so receipts which perform a lot of trie removals might be limited more than theoretically applicable. * `outgoing_receipts_usual_size_limit - 100 KiB` * Limit on the size of outgoing receipts to another shard. Needed to keep the size of `source_receipt_proofs` small. * On most block heights a shard isn't allowed to send receipts larger than 100 KiB to another shard. * `outgoing_receipts_big_size_limit - 4.5 MiB` * On every block height there's one special "allowed shard" which is allowed to send larger receipts, up to 4.5 MiB in total. * A receiving shard will receive receipts from `num_shards - 1` shards using the usual limit and one shard using the big limit. * The "allowed shard" is the same shard as in cross-shard congestion control. It's chosen in a round-robin fashion, at height 1 the special shard is 0, at height 2 it's 1 and so on. In total that gives 4 MiB + 500 KiB + 7MB + 5*100 KiB + 4.5 MiB ~= 16 MiB of maximum witness size. Possibly a little more on missing chunks. ### New limits breaking contracts The new limits will break some existing contracts (for example, all transactions larger than 1.5 MiB). This is sad, but it's necessary. Stateless validation uses much more network bandwidth than the previous approach, as it has to send over all states on each chunk application. Because network bandwidth is limited, stateless validation can't support some operations that were allowed in the previous design. In the past year (31,536,000 blocks) there were only 679 transactions bigger than 1.5MiB, sent between 164 unique (sender -> receiver) pairs. Only 0.002% of blocks contain such transactions, so the hope is that the breakage will be minimal. Contracts generally shouldn't require more than 1.5MiB of WASM. The full list of transactions from the past year which would fail with the new limit is available here: https://gist.github.com/jancionear/4cf373aff5301a5905a5f685ff24ed6f Contract developers can take a look at this list and see if their contract will be affected. ### Validating the limits Chunk validators have to verify that chunk producer respected all of the limits while producing the chunk. This means that validators also have to keep track of recorded storage proof by recording all trie accesses and they have to enforce the limits. If it turns out that some limits weren't respected, the validators will generate a different result of chunk application and they won't endorse the chunk. ### Missing chunks When a shard is missing some chunks, the following chunk on that shard will receive receipts from multiple blocks. This could lead to large `source_receipt_proofs` so a mechanism is added to reduce the impact. If there are two or more missing chunks in a row, the shard is considered fully congested and no new receipts will be sent to it (unless it's the `allowed_shard` to avoid deadlocks). ## ChunkStateWitness distribution For chunk production, the chunk producer is required to distribute the chunk state witness to all the chunk validators. The chunk validators then validate the chunk and send the chunk endorsement to the block producer. Chunk state witness distribution is on a latency critical path. As we saw in the section above, the maximum size of the state witness can be ~16 MiB. If the chunk producer were to send the chunk state witness to all the chunk validators it would add a massive bandwidth requirement for the chunk producer. To ease and distribute the network requirements across all the chunk producers, we have a distribution mechanism similar to what we have for chunks in the shards manager. We divide the chunk state witness into a number of parts, and let the chunk validators distribute the parts among themselves, and later reconstruct the chunk state witness. ### Distribution mechanism A chunk producer divides the state witness into a set of `N` parts where `N` is the number of chunk validators. The parts or partial witnesses are represented as [PartialEncodedStateWitness](https://github.com/near/nearcore/blob/66d3b134343d9f35f6e0b437ebbdbef3e4aa1de3/core/primitives/src/stateless_validation.rs#L40). Each chunk validator is the owner of one part. The chunk producer uses the [PartialEncodedStateWitnessMessage](https://github.com/near/nearcore/blob/66d3b134343d9f35f6e0b437ebbdbef3e4aa1de3/chain/network/src/state_witness.rs#L11) to send each part to their respective owners. The chunk validator part owners, on receiving the `PartialEncodedStateWitnessMessage`, forward this part to all other chunk validators via the [PartialEncodedStateWitnessForwardMessage](https://github.com/near/nearcore/blob/66d3b134343d9f35f6e0b437ebbdbef3e4aa1de3/chain/network/src/state_witness.rs#L15). Each validator then uses the partial witnesses received to reconstruct the full chunk state witness. We have a separate [PartialWitnessActor](https://github.com/near/nearcore/blob/66d3b134343d9f35f6e0b437ebbdbef3e4aa1de3/chain/client/src/stateless_validation/partial_witness/partial_witness_actor.rs#L32) actor/module that is responsible for dividing the state witness into parts, distributing the parts, handling both partial encoded state witness message and the forward message, validating and storing the parts, and reconstructing the state witness from the parts and sending is to the chunk validation module. ### Building redundancy using Reed Solomon Erasure encoding During the distribution mechanism, it's possible that some of the chunk validators are malicious, offline, or have a high network latency. Since chunk witness distribution is on the critical path for block production, we safeguard the distribution mechanism by building in redundancy using the Reed Solomon Erasure encoding. With Reed Solomon Erasure encoding, we can divde the chunk state witness into `N` total parts with `D` number of data parts. We can reconstruct the whole state witness as long as we have `D` of the `N` parts. The ratio of data parts `r = D/N` is something we can play around with. While reducing `r`, i.e. reducing the number of data parts required to reconstruct the state witness does allow for a more robust distribution mechanism, it comes with the cost of bloating the overall size of parts we need to distribute. If `S` is the size of the state witness, after reed solomon encoding, the total size `S'` of all parts becomes `S' = S/r` or `S' = S * N / D`. For the first release of stateless validation, we've kept the ratio as `0.6` representing that ~2/3rd of all chunk validators need to be online for chunk state witness distribution mechanism to work smoothly. One thing to note here is that the redundancy and upkeep requirement of 2/3rd is the *number* of chunk validators and not the *stake* of chunk validators. ### PartialEncodedStateWitness structure The partial encoded state witness has the following fields: * `(epoch_id, shard_id, height_created)` : These are the three fields that together uniquely determine the chunk associated with the partial witness. Since the chunk and chunk header distribution mechanism is independent of the partial witness, we rely on this triplet to uniquely identify which chunk is a part associated with. * `part_ord` : The index or id of the part in the array of partial witnesses. * `part` : The data associated with the part * `encoded_length` : The total length of the state witness. This is required in the reed solomon decoding process to reconstruct the state witness. * `signature` : Each part is signed by the chunk producer. This way the validity of the partial witness can be verified by the chunk validators receiving the parts. The `PartialEncodedStateWitnessTracker` module that is responsible for the storage and decoding of partial witnesses. This module has a LRU cache to store all the partial witnesses with `(epoch_id, shard_id, height_created)` triplet as the key. We reconstruct the state witness as soon as we have `D` of the `N` parts as forward the state witness to the validation module. ### Network tradeoffs To get a sense of network requirements for validators with an without partial state witness distribution mechanism, we can do some quick back of the envelop calculations. Let `N` but the number of chunk validators, `S` be the size of the chunk state witness, `r` be the ratio of data parts to total parts for Reed Solomon Erasure encoding. Without the partial state witness distribution, each chunk producer would have to send the state witness to all chunk validators, which would require a bandwidth `B` of `B = N * S`. For the worst case of ~16 validators and ~16 MiB of state witness size, this can be a burst requirement of 2 Gbps. Partial state witness distribution takes this load off the chunk producer and distributes it evenly among all the chunk validators. However, we get an additional factor of `1/r` of extra data being transferred for redundancy. Each partial witness has a size of `P = S' / N` or `P = S / r / N`. The chunk producer and validators needs a bandwidth `B` of `B = P * N` or `B = S / r` to forward its owned part to all `N` chunk validators. For worst case of ~16 MiB of state witness size and encoding ratio of `0.6`, this works out to be ~214 Mbps, which is much more reasonable. ### Future work In the Reed Solomon Erasure encoding section we discussed that the chunk state distribution mechanism relies on 2/3rd of the *number* of chunk validators being available/non-malicious and not 2/3rd of the *total stake* of the chunk validators. This can cause a potential issue where it's possible for more than 1/3rd of the chunk validators with small enough stake to be unavailable and cause the chunk production to stall. In the future we would like to address this problem. ## Validator Role Change Currently, there are two different types of validators and their responsibilities are as follows: | | Top ~50% validators | Remaining validatiors (Chunk only producers) | |-----|:-----:|:----:| | block production | Y | N | | chunk production | Y | Y | | block validation | Y | N | ### Protocol upgrade The good property of the approach taken is that protocol upgrade happens almost seamlessly. If (main transition, implicit transitions) fully belong to the protocol version before upgrade to stateless validation, chunk validator endorsements are not distributed, chunk validators are not sampled, but the protocol is safe because of all-shards tracking, as we described in "High-level flow". If at least some transition belongs to the protocol version after upgrade, chunk header height also belongs to epoch after upgrade, so it has chunk validators assigned and requirement of 2/3 endorsements is enabled. The minor accuracy needed is that generating and saving of state transition proofs have to be saved one epoch in advance, so we won't have to re-apply chunks to generate proofs once stateless validation is enabled. But new epoch protocol version is defined by finalization of **previous previous epoch**, so this is fine. It also assumes that each epoch has at least two chunks, but if this is not the case, the chain is having a major disruption which never happened before. ## Security Implications Block validators no longer required to track any shard which means they don't have to validate state transitions proposed by the chunks in the block. Instead they trust chunk endorsements included in the block to certify the validity of the state transitions. This makes chunk validator selection algorithm correctness critical for the security of the whole protocol, which is probabilistic by nature unlike the current more strict 2/3 of non-malicious validators requirement. It is also worth mentioning that large state witness size makes witness distribution slow which could result in a missing chunk because the block producer won't get chunk endorsements in time. This design tries to address that by meticulously limiting max witness size (see [this doc](https://github.com/near/nearcore/blob/master/docs/misc/state_witness_size_limits.md)). ## Alternatives The only real alternative that was considered is the original nightshade proposal. The full overview of the differences can be found in the revised nightshade whitepaper at https://near.org/papers/nightshade. ## Future possibilities * Integration with ZK allowing to get rid of large state witness distribution. If we treat state witness as a proof and ZK-ify it, anyone can validate that state witness indeed proves the new chunk header with much lower effort. Complexity of actual proof generation and computation indeed increases, but it can be distributed among chunk producers, and we can have separate concept of finality while allowing generic users to query optimistic chunks. * Integration with resharding to further increase the number of shards and the total throughput. * The sharding of non-validating nodes and services. There are a number of services that may benefit from tracking only a subset of shards. Some examples include the RPC, archival and read-RPC nodes. ## Consequences ### Positive * The validator nodes will need to track at most one shard. * The state will be held in memory making the chunk application much faster. * The disk space hardware requirement will decrease. The top 100 nodes will need to store at most 2 shards at a time and the remaining nodes will not need to store any shards. * Thanks to the above, in the future, it will be possible to reduce the gas costs and by doing so increase the throughput of the system. ### Neutral * The current approach to resharding will need to be revised to support generating state witness. * The security assumptions will change. The responsibility will be moved from block producers to chunk validators and the security will become probabilistic. ### Negative * The network bandwidth and memory hardware requirements will increase. * The top 100 validators will need to store up to 2 shards in memory and participate in state witness distribution. * The remaining validators will need to participate in state witness distribution. * Additional limits will be put on the size of transactions, receipts and, more generally, cross shard communication. * The dependency on cloud state sync will increase the centralization of the blockchain. This will be resolved separately by the decentralized state sync. ### Backwards Compatibility [All NEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. Author must explain a proposes to deal with these incompatibilities. Submissions without a sufficient backwards compatibility treatise may be rejected outright.] ## Unresolved Issues (Optional) [Explain any issues that warrant further discussion. Considerations * What parts of the design do you expect to resolve through the NEP process before this gets merged? * What parts of the design do you expect to resolve through the implementation of this feature before stabilization? * What related issues do you consider out of scope for this NEP that could be addressed in the future independently of the solution that comes out of this NEP?] ## Changelog [The changelog section provides historical context for how the NEP developed over time. Initial NEP submission should start with version 1.0.0, and all subsequent NEP extensions must follow [Semantic Versioning](https://semver.org/). Every version should have the benefits and concerns raised during the review. The author does not need to fill out this section for the initial draft. Instead, the assigned reviewers (Subject Matter Experts) should create the first version during the first technical review. After the final public call, the author should then finalize the last version of the decision context.] ### 1.0.0 - Initial Version > Placeholder for the context about when and who approved this NEP version. ### 1.0.1 - Fix: Protocol Rewards Halving As of October 29 2025, the Protocol Rewards have been halved from 5% to 2.5%. #### Benefits > List of benefits filled by the Subject Matter Experts while reviewing this version: * Benefit 1 * Benefit 2 #### Concerns > Template for Subject Matter Experts review for this version: > Status: New | Ongoing | Resolved | # | Concern | Resolution | Status | | --: | :------ | :--------- | -----: | | 1 | | | | | 2 | | | | ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0514.md ================================================ --- NEP: 514 Title: Reducing the number of Block Producer Seats in `testnet` Authors: Nikolay Kurtov Status: Final DiscussionsTo: https://github.com/nearprotocol/neps/pull/514 Type: Protocol Version: 1.0.0 Created: 2023-10-25 LastUpdated: 2023-10-25 --- ## Summary This proposal aims to adjust the number of block producer seats on `testnet` in order to ensure a positive number of chunk-only producers present in `testnet` at all times. ## Motivation The problem is that important code paths are not exercised in `testnet`. This makes `mainnet` releases more risky than they have to be, and greatly slows down development of features related to chunk-only producers, such as State Sync. That is because `testnet` has fewer validating nodes than the number of block producer seats configured. The number of validating nodes on `testnet` is somewhere in the range of [26, 46], which means that all validating nodes are block producers and none of them are chunk-only producers. [Grafana](https://nearinc.grafana.net/goto/7Kh81P7IR?orgId=1). `testnet` configuration is currently the following: * `"num_block_producer_seats": 100,` * `"num_block_producer_seats_per_shard": [ 100, 100, 100, 100 ],` * `"num_chunk_only_producer_seats": 200,` It's evident that the 100 block producer seats significantly outnumber the validating nodes in `testnet`. An alternative solution to the problem stated above can be the following: 1. Encourage the community to run more `testnet` validating nodes 1. Release owners or developers of features start a lot of validating nodes to 1. ensure `testnet` gets some chunk-only producing nodes. 1. Exercise the unique code paths in a separate chain, a-la `localnet`. Let's consider each of these options. ### More community nodes This would be the ideal perfect situation. More nodes joining will make `testnet` more similar to `mainnet`, which will have various positive effects for protocol developers and dApp developers. However, this option is expensive, because running a validating node costs money, and most community members can't afford spending that amount of money for the good of the network. ### More protocol developer nodes While this option may seem viable, it poses significant financial challenges for protocol development. The associated computational expenses are exorbitantly high, making it an impractical choice for sustainable development. ### Test in separate chains That is the current solution, and it has significant drawbacks: * Separate chains are short-lived and may miss events critical to the unique code paths of chunk-only producers * Separate chains need special attention to be configured in a way that accommodates for chunk-only producers. Most test cases are not concerned about them, and don't exercise the unique code paths. * Separate chains can't process real transaction traffic. The traffic must either be synthetic or "inspired" by real traffic. * Each such test has a significant cost of running multiple nodes, in some cases, tens of nodes. ## Specification The proposal suggests altering the number of block producer seats to ensure that a portion of the `testnet` validating nodes become chunk-only producers. The desired `testnet` configuration is the following: * `"num_block_producer_seats": 20,` * `"num_block_producer_seats_per_shard": [ 20, 20, 20, 20 ],` * `"num_chunk_only_producer_seats": 100,` I suggest to implement the change for all networks that are not `mainnet` and have `use_production_config` in the genesis file. `use_production_config` is a sneaky parameter in `GenesisConfig` that lets protocol upgrades to change network's `GenesisConfig`. I don't have a solid argument for lowering the number of chunk producer seats, but that reflects the reality that we don't expect a lot of nodes joining `testnet`. It also makes it easier to test the case of too many validating nodes willing to join a network. ## Reference Implementation [#9563](https://github.com/near/nearcore/pull/9563) If `use_production_config`, check whether `chain_id` is eligible, then change the configuration as specified above. ## Security Implications The block production in `testnet` becomes more centralized. It's not a new concern as 50% of stake is already owned by nodes operated by the protocol developers. ## Alternatives See above. ## Future possibilities Adjust the number of block and chunk producer seats according to the development of the number of `testnet` validating nodes. ## Consequences ### Positive * Chunk-only production gets tested in `testnet` * Development of State Sync and other features related to chunk-only producers accelerates ### Neutral * `testnet` block production becomes more centralized ### Negative * Any? ### Backwards Compatibility During the protocol upgrade, some nodes will become chunk-only producers. The piece of code that updates `testnet` configuration value will need to be kept in the database in case somebody wants to generate `EpochInfo` compatible with the protocol versions containing the implementation of this NEP. ## Changelog ### 1.0.0 - Initial Version The Protocol Working Group members approved this NEP on Oct 26, 2023. [Zulip link](https://near.zulipchat.com/#narrow/stream/297873-pagoda.2Fnode/topic/How.20to.20test.20a.20chunk-only.20producer.20node.20in.20testnet.3F/near/396090090) #### Benefits See [Consequences](#consequences). #### Concerns See [Consequences](#consequences). ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0518.md ================================================ --- NEP: 518 Title: Web3-Compatible Wallets Support Authors: Aleksandr Shevchenko , Michael Birch Status: New DiscussionsTo: https://github.com/near/NEPs/issues/518 Type: Protocol Version: 1.0.0 Created: 2023-11-15 LastUpdated: 2024-07-22 --- ## Summary This NEP describes the protocol changes needed to support the usage of Ethereum-compatible wallets (Web3 wallets), for example Metamask, on Near native applications. That is to say, with this protocol change all Metamask users can become Near users without installing any additional software; from their perspective Near will appear as just another network they can choose from (similar to Aurora today). This is accomplished through two key protocol changes: 1. Ethereum-like addresses (i.e. account IDs of the form `^0x[a-f0-9]{40}$`) are implicit accounts on Near (i.e. can be created via a `Transfer` action). We call these "eth-implicit accounts". 2. Unlike the current implicit accounts (64-character hex-encoded), eth-implicit accounts do not have any access keys added to them on creation. Instead, these accounts will have a special contract deployed to them automatically called the "wallet contract". This wallet contract enables the owner of the Ethereum address corresponding to the eth-implicit account ID to sign transactions with their Ethereum private key, thus providing similar functionality to the default access key of 64-character implicit accounts. The nature of this NEP requires the reader to know some concepts from the Ethereum ecosystem. However, since this is a document for readers only familiar with the Near network, we include appendices with definitions and descriptions of the Ethereum concepts needed to understand this proposal. Terms in bold, for example **EOA**, are defined in the glossary (Appendix A). The protocol changes described here are a part of the overall eb3-Compatible Wallets Support solution. The full solution (including the protocol changes described here) are detailed in the original [NEP-518 issue description](https://github.com/near/NEPs/issues/518). ## Motivation Currently, the Ethereum ecosystem is a leading force in the smart contract blockchain space, boasting a large user base and extensive installations of Ethereum-compatible tooling and wallets. However, a significant challenge arises due to the incompatibility of these tools and wallets with NEAR Protocol. This incompatibility necessitates a complete onboarding process for users to interact with NEAR contracts and accounts, leading to confusion, decreased adoption, and the marginalization of NEAR Protocol. Implementing Web3 wallet support in NEAR Protocol, with an emphasis on user experience continuity, would significantly benefit the entire NEAR Ecosystem. ## Specification ### Eth-implicit accounts **Definition**: An eth-implicit account is a top-level Near account with ID of the form `^0x[a-f0-9]{40}$` (i.e. 42-characters with `0x` as a prefix followed by 40 characters of hex-encoded data which represents a 20-byte address). Eth-implicit accounts, as the name suggests, are implicit accounts on Near. This means if the target account ID does not exist during a `Transfer` action then it MUST be automatically created. This includes being created even if the amount being transferred is zero (per the prior [NEP on zero balance accounts](https://github.com/near/NEPs/blob/master/neps/nep-0448.md)). Eth-implicit accounts represent an Ethereum **EOA** and therefore are controlled via the Ethereum private key corresponding to the address contained in the account ID (see Appendix B for a description of how 20-byte addresses are derived from a private key in the Ethereum ecosystem). To enable this control, eth-implicit accounts all have a smart contract deploy to them called the wallet contract (specification in the next section). When an eth-implicit account is created the runtime MUST set the contract code equal to specific "magic bytes". These bytes come from a UTF-8 encoded string which is equal to the constant `near` appended with the base-58 encoding of the sha2 hash of the wallet contract code. This constant allows the contract runtime to lookup the full contract code without needing it to be stored multiple times in the state. As well as being more efficient for the protocol, setting the code equal to a hash of the contract instead of the contract itself keeps the storage requirements of a new eth-implicit account small enough to be a zero balance account. The magic bytes depend on the Near network chain id because the wallet contract (and therefore its hash) depends on the Near chain id. The magic bytes (UTF-8 encoded) for each Near chain id are listed below: - `mainnet`: `near83PPBGX9KNgC2TRJgX7mvZfFPx92bFkdYvZNARQjRt8G` - `testnet`: `near3Za8tfLX6nKa2k4u2Aq5CRrM7EmTVSL9EERxymfnSFKd` - any other id (e.g. `localnet`): `near2dQzuvePVCmkXwe1oF3AgY9pZvqtDtq43nFHph928CU4` When the runtime is executing a `FunctionCall` action on an account with these magic bytes as code then it MUST act as if the wallet contract code were stored there instead (i.e. the wallet contract Wasm module ends up being executed). ### Wallet contract This smart contract is automatically deployed to all eth-implicit accounts (see prior section). The purpose of this contract is to accept transactions encoded in an Ethereum style and create Near actions which are executed in subsequent receipts. In this way, the owner of the Ethereum private key associated with the eth-implicit account (the address contained in its account ID) controls what actions the account takes. Thus that Ethereum key effectively becomes the only access key for the account, emulating the behavior of an Ethereum **EOA**. #### API The wallet contract has two public functions: - `get_nonce` is a view function which takes no arguments and returns a 64-bit number (encoded as a base-10 string). - `rlp_execute` is the main entry point for executing user transactions. It takes two inputs (encoded as a JSON object): `target` is an account ID (i.e. string) which indicates the account that is supposed to be the target of the Near action; and `tx_bytes_b64` is a string which is the base-64 encoding of the raw bytes of an Ethereum-like transaction. The process by which a Near action is derived from the Ethereum transaction is described below. The wallet contract has two state variables: the nonce, a 64-bit number; and a boolean flag indicating if a transaction is currently in progress. As with nonce values on Near access keys, the purpose of the wallet contract nonce is to prevent replaying the same Ethereum transaction more than once. The boolean flag prevents multiple transactions from being in-flight at the same time. The reason this is needed is because of the asynchronous nature of Near as compared with the synchronous nature of the **EVM**. On Ethereum if two transactions are sent (they must have sequential nonces per the Ethereum standard) all actions of the first will happen before all actions of the second. However, on Near there is no guarantee of the order of execution for receipts in different shards. Therefore, the only way to ensure that all actions from the first transaction are executed before all the actions of the second transaction is to prevent the second transaction from starting its execution until after the first one entirely finishes. #### Details of `rlp_execute` This function is named after the **RLP** standard in Ethereum. In particular, the `tx_bytes_b64` argument is parsed into bytes from base-64; then the bytes are parsed into structured data assuming it is RLP encoded; then the structured data is parsed into an Ethereum transaction. Ethereum transactions can have multiple different forms since the Ethereum protocol has evolved over time (there are "legacy" transactions, [EIP-1559](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1559.md) type transactions, [EIP-2930](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-2930.md) transactions). All these different forms are supported by the wallet contract (they are distinguished based on the "type byte" which starts the encoding as per [EIP-2718](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-2718.md)) and are ultimately all transformed into a common data structure with the following fields: - `from`: the address associated with the private key that signed the transaction. - `chain_id`: a numerical ID that is unique per **EVM**-chain. The Near chain ID values are discussed below. - `nonce`: the nonce associated with this transaction. It must be equal to the wallet contracts's currently stored nonce for the transaction to be executed. - `gas_limit`: the maximum amount of **EVM** gas the user is willing to spend on this transaction. - `max_fee_per_gas`: the gas price the user is willing to pay. `gas_limit * max_fee_per_gas` gives the maximum amount of **Wei** the user is willing to pay for the transaction. - `to`: the address of the account the transaction is targeting. This could be another **EOA** in the case of a base token transfer or the address of a smart contract in the case of what Near would refer to as a function call. In the Ethereum standard this field is allowed to be empty to indicate a new contract is being created, however that is forbidden by the wallet contract because Near currently does not support **EVM** bytecode, so there is not a reasonable way to emulate an Ethereum contract deployment. - `value`: the amount of **Wei** attached to the transaction. - `data`: the raw bytes which will be sent as a payload to the target address. If the target address is a contract it will use these bytes as input. Note: some Ethereum transaction fields are intentionally omitted because they are unused by the wallet contract. These fields are used to validate the transaction and derive Near actions that the wallet contract will create as receipts. The details of this process are described below. ##### Ethereum transaction validation The following validation conditions MUST pass for the wallet contract to accept a transaction. 1. `from` address when formatted as hex-encoded with `0x` prefix MUST match the current account ID (i.e. the wallet contract's account ID). 2. `chain_id` MUST match one of the following values depending on the Near chain the wallet contract is deployed to: mainnet -> 397; testnet -> 398; any other chain -> 399. The mainnet and testnet values are registered with the [official Ethereum ecosystem registry of chain IDs](https://github.com/ethereum-lists/chains). 3. `nonce` MUST match the nonce value currently stored in the contract state. 4. `to` address MUST either (a) be equal to `keccak256(target)[12,32]` (where `target` is other argument passed to the `rlp_execute` function) or (b) when `to` is formatted as hex-encoded with `0x` prefix it MUST be equal to `target`. In case (b) there is an additional validation check that the `to` address is not registered in the "Ethereum Translation Contract" (ETC). The details of this check and why it is needed are discussed in Appendix C. 5. `value` MUST be less than or equal to `(2**128 - 1) // 1_000_000`. This condition arises from the mismatch is decimal places between Ether and NEAR which is discussed in the definition of **Wei** in Appendix A. Essentially, we must ensure the `value` can be mapped into a valid amount of yoctoNEAR, which means `value * 1_000_000 <= u128::MAX`. ##### Converting Ethereum transaction into Near actions Each Ethereum transaction is converted to a single Near action (batch transactions are not supported) based on the `data` field. Following the Solidity convention of the first four bytes of the data being a **method selector**, the wallet contract checks the first four bytes of the `data` to see if it is a known Near action. The **method selectors** for Near actions supported by the wallet contract are determined by mapping the actions to an equivalent Solidity function signature as follows: - `functionCall(string,string,bytes,uint64,uint32)` - `transfer(string,uint32)` - `addKey(uint8,bytes,uint64,bool,bool,uint128,string,string[])` - `deleteKey(uint8,bytes)` Note that the `uint32` fields in `functionCall` and `transfer` contain the amount of yoctoNEAR that cannot be included in the Ethereum transaction's `value` field due to the difference in decimal places (see **Wei** definition in Appendix A), therefore the value there is always less than `1_000_000` so it will easily fit in a 32-bit number. These type signatures then hash to the **method selectors**: - FunctionCall: `0x6179b707` - Transfer: `0x3ed64124` - AddKey: `0x753ce5ab` - DeleteKey: `0x3fc6d404` If the first four bytes of the `data` field matches one of these **method selectors** then the wallet contract will try to parse the remainder of the `data` into the corresponding type signature (assuming the data is Solidity ABI encoded). If this parsing succeeds then the resulting tuple of values can be converted to the corresponding Near action. Some additional validation is done in this case, depending on the action: - FunctionCall/Transfer: `target` MUST equal the first `string` parameter (interpreted as the receiver ID), the `uint32` parameter value MUST be less than `1_000_000`. - AddKey/DeleteKey: the `uint8` parameter value MUST be 0 (corresponding to an ED25519 access key) or 1 (corresponding to a Secp256k1 access key), the `bytes` MUST be the appropriate length depending on the key type, `target` MUST equal the current account ID (since these actions can only act on the current account). Additionally, the first `bool` value of `addKey` must be `false` because adding a full access key is currently not supported by the wallet contract. The reason for this is to prevent users from changing the contract code deployed to the eth-implicit contract, as it could break the account's intended functionality. However, this restriction may be lifted in the future. If the first four bytes of `data` does not match one of these known selectors then the contract tries another set of known **method selectors** which come from the Ethereum ERC-20 standard: - `balanceOf(address)` -> `0x70a08231` - `transfer(address,uint256)` -> `0xa9059cbb` - `totalSupply()` -> `0x18160ddd` These **method selectors** are included because some Web3 wallets (for example MetaMask) allow a user to transfer tokens directly within the wallet interface. This interface produces an Ethereum transaction with Solidity ABI encoded data following the ERC-20 standard rather than the encoding of the Near actions outlined above. Therefore the wallet contract also knows how to parse these ERC-20 standard methods into Near actions so that the wallet interfaces still work according to the user's expectations. This feature of the wallet contract is called Ethereum Standards Emulation because it emulates the execution of an Ethereum standard. Currently ERC-20 is the only supported standard for emulation, but perhaps more will be added in the future. ERC-20 is Ethereum's fungible token standard, thus these calls are mapped to the corresponding NEP-141 `FunctionCall` actions: - `balanceOf` -> `ft_balance_of` - `transfer` -> `ft_transfer` - `totalSupply` -> `ft_total_supply` Note: it is intentional that not all the ERC-20 functions are emulated (in particular related to `approve`/`allowance`) because there is not the corresponding functionality in NEP-141. There is additional validation in the case of `transfer` that the amount is less than `u128::MAX` because the ERC-20 standard allows 256-bit amounts while the NEP-141 standard only allows 128-bit. The NEP-141 standard also has additional complexity that ERC-20 does not have because of the storage deposit requirement (a consequence of Near's storage staking). On Ethereum a user can transfer tokens to another account that has never held that kind of token before. On Near that is only possible if the user pays for the recipient's storage deposit first. Therefore, as part of the `transfer` emulation the wallet contract includes a call to `storage_balance_of` to check if a call to `storage_storage_deposit` is also needed before calling `ft_transfer`. If none of the known selectors match the first four bytes of `data` or the remainder of `data` fails to parse into the appropriate type signature then there is one more possible emulation that the wallet contract checks for. On Ethereum base token transfers are allowed to have arbitrary data included and some wallets use this feature as a sort of messaging protocol between addresses. Therefore, if the `data` is not processed and the `target` is another eth-implicit account, then we assume this is meant to emulate a base token transfer and thus a Near `Transfer` action is created. Otherwise, the wallet contract returns an error that the transaction could not be parsed. #### Interaction with Web3 relayers Typically users will not be constructing the `rlp_execute` action themselves because the target user group are those who only have a Web3 wallet like MetaMask, not a Near wallet to sign Near transactions. Therefore, the Near transactions will be constructed and sent to the Near network on a user's behalf by relayers. These relayers expose the Ethereum standard JSON RPC so that Web3 wallets know how to as the relayer to send an Ethereum-like transaction and to query the status of that transaction. More details about relayers and the RPC they expose is found in the [NEP-518 issue description](https://github.com/near/NEPs/issues/518), but it out of scope for this document because they operated separately from the Near protocol itself. The relevant fact for the wallet contact specification is that relayers can ask their users to add a function call access key to their eth-implicit account which the relayer uses to call `rlp_execute`. By using an access key on the eth-implicit account itself, the relayer does not need to cover any gas costs for the user because the transaction originates from the wallet contract account itself. However, for this mechanism to be safe for users, relayers must be prevented from sending transactions to the wallet contract that the user did not intend. Otherwise relayers could maliciously burn the $NEAR of their users on excess calls to `rlp_execute` (even if those transactions return an error, gas is still spent in the process). For this reason, the wallet contract separates possible errors in the `rlp_execute` input into two categories: user errors and relayer errors. User errors are errors that arise from data signed by the user's private key and therefore cannot be spoofed by the relayer. Relayer errors arise from input that should not have been sent by an honest relayer in the first place. These relayer errors include: - Invalid Ethereum transaction nonce: if the nonce check fails then the relayer is at fault because it should have checked the nonce before sending the transaction. This prevents a malicious relayer from sending the same user-signed transaction over and over to burn the user's $NEAR unnecessarily. - Invalid base-64 encoding in `tx_bytes_b64`: an honest relayer should only send valid arguments. If a relayer sends garbage input then it is faulty. - Invalid Ethereum transaction encoding: similar to the error above, but with the issue occurring in the RLP-encoding instead of in the base-64 encoding. - Invalid sender: if the address extracted from the signature on the Ethereum transaction does not match the wallet contract account ID then the relayer is faulty because it sent an incorrectly signed transaction. - Invalid target: if the `target` validation relative to the `to` field in the user's signed Ethereum transaction fails then the relayer is faulty because it tried to misdirect the transaction to a different account than the user intended. - Invalid chain id: similar to the invalid sender error, the relayer should only send transaction with a valid signature, including with the correct chain id. - Insufficient gas: if the relayer does not attach as much Near gas to the transaction as the user asked for in the `gas_limit` field of their signed Ethereum transaction then it is faulty. This prevents a malicious relayer from intentionally making user transactions fail by not attaching enough gas to complete the action. If a relayer error happens then the wallet contract creates a callback to remove the relayers access key. This prevents them from repeatedly sending incorrect input. ## Reference implementation Summarizing the above, the protocol changes necessary for the Web3 wallets project include: - Creating Ethereum-like (0x) implicit accounts using `Transfer` action, - Automatically deploying the wallet contract to those 0x implicit accounts. These protocol changes are implemented in nearcore ([eth-implicit accounts PR 10224](https://github.com/near/nearcore/pull/10224), [wallet contract implementation](https://github.com/near/nearcore/tree/1ab9b42c3d723604a214e685d8ed39f7d6434ae2/runtime/near-wallet-contract/implementation)) and have been stabilized in protocol version 70 ([PR 11765](https://github.com/near/nearcore/pull/11765)). ## Security Implications The wallet contract must uphold the invariant that only the owner of the private key can make the wallet contract create Near actions. The wallet contract has been audited and is believed to be secure. ## Alternatives See the "Prior work" section of the [original NEP-518 issue](https://github.com/near/NEPs/issues/518). ## Future possibilities See the "Future Opportunities" section of the [original NEP-518 issue](https://github.com/near/NEPs/issues/518). ## Consequences ### Positive - All Ethereum users can easily onboard to Near ### Neutral - New implicit account type with a protocol-level smart contract deployed by default. ### Backwards Compatibility As pointed out in [PR 11606](https://github.com/near/nearcore/pull/11606) there are 5552 accounts on mainnet today with account IDs that would classify them as eth-implicit accounts. For backwards compatibility, these accounts will not be changed in any way (their access keys and contract code will be left in place) and therefore will in fact still be normal Near accounts as opposed to eth-implicit accounts because they have full access keys and possibly a contract different from the protocol-sanctioned wallet contract. ## Appendix A - Glossary Below is a list of Ethereum-related terms and their definitions. - **Ethereum Virtual Machine (EVM)**: the virtual machine used to execute smart contracts on the Ethereum blockchain. "EVM-compatible" is often used interchangeably with "Ethereum compatible". - **Externally owned account (EOA)**: An Ethereum account for which a user has the private key. Unlike Near, on Ethereum there is a distinction between contracts and user accounts. User accounts cannot have contract code and contract accounts cannot initiate a transaction. - **Method selector**: By convention in Solidity contracts, the first four bytes of the input to a smart contract determine which method is executed (unlike Near where a method is explicitly specified as part of the `FunctionCall` action). These bytes are obtained by taking the first four bytes of the keccak256 hash of the type signature of the function. - **Recursive Length Prefix (RLP) serialization**: An Ethereum ecosystem standard for encoding structured data as bytes. It plays a similar role to `borsh` in the Near ecosystem. - **Wei**: the smallest unit of the base token for Ethereum. It plays a similar role to yoctoNEAR in the Near ecosystem. An important difference between Wei and yoctoNEAR is that 1 Ether (the typical unit for the base token on Ethereum) is equal to 10^18 Wei, while 1 NEAR is equal to 10^24 yoctoNEAR. Phrased another way, Ether has 18 decimal places while NEAR has 24. This difference in precision creates minor complexities in the wallet contract. ## Appendix B - How addresses are derived in Ethereum On Ethereum all accounts are identified by a 20-byte address. The address of a user account is derived from a user's private key in the following way: 1. Compute the user's public key from the private key (this step can be omitted if you already, indeed only, know the public key). 2. Compute the keccak256 hash of the public key. 3. Return the rightmost 20 bytes of this hash. This is summarized by the following formula: `address = keccak256(public_key)[12,32]`. ## Appendix C - Ethereum Translation Contract (ETC) There is an additional contract which is tangentially related to the wallet contract. The [original NEP-518 issue](https://github.com/near/NEPs/issues/518) refers to it as the Ethereum Translation Contract (ETC), though perhaps a more descriptive name is the Ethereum address registrar. The implementation of this contract is not part of the protocol, however its account ID is because the account ID is hardcoded into the wallet contract. The reason is because the wallet contract occasionally needs the ETC to verify if the `target` argument to `rlp_execute` is properly set relative to the `to` field of the user's signed Ethereum transaction. The details of why ETC is needed and how it is used is described below. Recall that the user is signing an Ethereum transaction because the whole point of this project is to allow Web3 wallets like MetaMask to be used on Near. An Ethereum transaction specifies the target of a transaction using a 20-byte address because there are no named accounts on Ethereum. Therefore the user only signs over a 20-byte address to indicate their intent of what account is meant to receive this transaction. However, this is obviously insufficient information on Near because most accounts are named ones, not addresses. The purpose of the `target` argument to `rlp_execute` is to communicate the account ID of the receiver of the transaction and it must be consistent with the user's signed Ethereum transaction according to the validation conditions described in the "Ethereum transaction validation" section. Most of the time that means checking `to == keccak256(target)[12,32]` because the `target` will be some named Near account. However, it is possible that `target` be another eth-implicit account; this is the case for "emulated" base token transfers (emulated Ethereum standards are discussed in the section "Converting Ethereum transaction into Near actions"). Thus, we must also allow the possibility that `to == target`. Yet, this poses a problem because it means `target` could be set incorrectly if it was meant to be a named account satisfying the hash condition instead. The ETC closes the loophole by providing a reverse lookup from 20-byte address to named Near accounts where the association comes from the hash condition. To fully validate `target` in the case that `to == target` the wallet contract makes the following additional checks: - If the `data` field of the user's signed Ethereum transaction can be parsed into a Near action then confirm `target` matches the `receiver_id` of the corresponding action (this is statically known to be the current account ID in the case of `AddKey` and `DeleteKey`, and it is encoded with the action in the case of `FunctionCall` and `Transfer`). - If the `data` field can be parsed as ERC-20 action then call the `lookup` method of the ETC to see if the `target` is registered. If it is registered then the `target` field is set incorrectly because the relayer should have set `target` equal to the named account returned from `ETC::lookup(to)`. This validity check ensures that emulated ERC-20 transactions are sent to the correct NEP-141 token account. If `target` is not registered then the transaction is interpreted as an emulated base token transfer with a message that happens to parse like an ERC-20 function call. Notably, for this security measure to be effective all widely used NEP-141 token accounts will need to be registered with ETC. ETC has a public method `register` which permissionlessly allows anyone to add an account ID they think is important. This openness id not a feasible attack vector for the system because of the one-way nature of the keccak256 hash function preventing an attacker from coming up with a Near account ID that corresponds to an address of their choosing. ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0519.md ================================================ --- NEP: 519 Title: Yield Execution Authors: Akhi Singhania ; Saketh Are Status: Final DiscussionsTo: https://github.com/near/NEPs/pull/519 Type: Protocol Version: 0.0.0 Created: 2023-11-17 LastUpdated: 2023-11-20 --- ## Summary Today, when a smart contract is called by a user or another contract, it has no sensible way to delay responding to the caller till it has observed another future transaction. This proposal introduces this possibility into the NEAR protocol. ## Motivation There exist some situations where when a smart contract on NEAR is called, it will only be able to provide an answer at some time in the future. The callee needs a way to defer replying to the caller while the response is being prepared. Examples include a smart contract (`S`) that provides the MPC signing capability. It relies on indexers external to the NEAR protocol for computing the signatures. The rough steps are: 1. Signer contract provides a function `fn sign_payload(Payload, ...)`. 2. When called, the contract defers replying to the caller. 3. External indexers are monitoring the transactions on the contract; they observe the new signing request, compute a signature, and call another function `fn signature_available(Signature, ...)` on the signer contract. 4. The signer contract validates the signature and replies to the original caller. Today, the NEAR protocol has no sensible way to defer replying to the caller in step 2 above. This proposal proposes adding two following new host functions to the NEAR protocol: - `promise_yield_create`: allows setting up a continuation function that should only be executed after `promise_yield_resume` is invoked. Together with `promise_return` this allows delaying the reply to the caller; - `promise_yield_resume`: indicates to the protocol that the continuation to the yield may now be executed. If these two host functions were available, then `promise_yield_create` would be used to implement step 2 above and `promise_yield_resume` would be used for step 3 of the motivating example above. ## Specification The proposal is to add the following host functions to the NEAR protocol: ```rust /// Smart contracts can use this host function along with /// `promise_yield_resume()` to delay replying to their caller for up to 200 /// blocks. This host function allows the contract to provide a callback to the /// protocol that will be executed after either contract calls /// `promise_yield_resume()` or after 200 blocks have been executed. The /// callback then has the opportunity to either reply to the caller or to delay /// replying again. /// /// `method_name_len` and `method_name_ptr`: Identify the callback method that /// should be executed either after the contract calls `promise_yield_resume()` /// or after 200 blocks have been executed. /// /// `arguments_len` and `arguments_ptr` provide an initial blob of arguments /// that will be passed to the callback. These will be available via the /// `input` host function. /// /// `gas`: Similar to the `gas` parameter in /// [promise_create](https://github.com/near/nearcore/blob/a908de36ab6f75eb130447a5788007e26d05f93e/runtime/near-vm-runner/src/logic/logic.rs#L1281), /// the `gas` parameter is a prepayment for the gas that would be used to /// execute the callback. /// /// `gas_weight`: Similar to the `gas_weight` parameter in /// [promise_batch_action_function_call_weight](https://github.com/near/nearcore/blob/a908de36ab6f75eb130447a5788007e26d05f93e/runtime/near-vm-runner/src/logic/logic.rs#L1699), /// this improves the devX for the smart contract. It allows a contract to /// specify a portion of the remaining gas for executing the callback instead of /// specifying a precise amount. /// /// `register_id`: is used to identify the register that will be filled with a /// unique resumption token. This token is used with `promise_yield_resume` to /// resolve the continuation receipt set up by this function. /// /// Return value: u64: Similar to the /// [promise_create](https://github.com/near/nearcore/blob/a908de36ab6f75eb130447a5788007e26d05f93e/runtime/near-vm-runner/src/logic/logic.rs#L1281) /// host function, this function also create a promise and returns an index to /// the promise. This index can be used to create a chain of promises. pub fn promise_yield_create( method_name_len: u64, method_name_ptr: u64, arguments_len: u64, arguments_ptr: u64, gas: u64, gas_weight: u64, register_id: u64, ) -> u64; /// See `promise_yield_create()` for more details. This host function can be /// used to resolve the continuation that was set up by /// `promise_yield_create()`. The contract calling this function must be the /// same contract that called `promise_yield_create()` earlier. This host /// function cannot be called for the same resumption token twice or if the /// callback specified in `promise_yield_create()` has already executed. /// /// `data_id_len` and `data_it_ptr`: Used to pass the unique resumption token /// that was returned to the smart contract in the `promise_yield_create()` /// function (via the register). /// /// `payload_len` and `payload_ptr`: the smart contract can provide an /// additional optional blob of arguments that should be passed to the callback /// that will be resumed. These are available via the `promise_result` host /// function. /// /// This function can be called multiple times with the same data id. If it is /// called successfully multiple times, then the implementation guarantees that /// the yielded callback will execute with one of the successfully submitted /// payloads. If submission was successful, then `1` is returned. Otherwise /// (e.g. if the yield receipt has already timed out or the yielded callback has /// already been executed) `0` will be returned, indicating that this payload /// could not be submitted successfully. pub fn promise_yield_resume( data_id_len: u64, data_id_ptr: u64, payload_len: u64, payload_ptr: u64, ) -> u32; ``` ## Reference Implementation The reference implementation against the nearcore repository can be found in this [PR](https://github.com/near/nearcore/pull/10415). ## Security Implications Some potential security issues have been identified and are covered below: - Smart contracts using this functionality have to be careful not to let just any party trigger a call to `promise_yield_resume`. In the example above, it is possible that a malicious actor may pretend to be an external signer and call the `signature_available()` function with an incorrect signature. Hence contracts should be taking precautions by only letting select callers call the function (by using [this](https://github.com/aurora-is-near/near-plugins/blob/master/near-plugins/src/access_controllable.rs) service for example) and validating the payload before acting upon it. - This mechanism introduces a new way to create delayed receipts in the protocol. When the protocol is under conditions of congestion, this mechanism could be used to further aggravate the situation. This is deemed as not a terrible issue as the existing mechanisms of using promises and etc. can also be used to further exacerbate the situation. ## Alternatives Two alternatives have been identified. ### Self calls to delay replying In the `fn sign_payload(Payload, ...)` function, instead of calling `yield`, the contract can keep calling itself in a loop till external indexer replies with the signature. This would work but would be very fragile and expensive. The contract would have to pay for all the calls and function executions while it is waiting for the response. Also depending on the congestion on the network; if the shard is not busy at all, some self calls could happen within the same block meaning that the contract might not actually wait for as long as it hoped for and if the network is very busy then the call from the external indexer might be arbitrarily delayed. ### Change the flow of calls The general flow of cross contract calls in NEAR is that a contract `A` sends a request to another contract `B` to perform a service and `B` replies to `A` with the response. This flow could be altered. When a contract `A` calls `B` to perform a service, `B` could respond with a "promise to call it later with the answer". Then when the signature is eventually available, `B` can then send `A` a request with the signature. There are some problems with this approach though. After the change of flow of calls; `B` is now going to be paying for gas for various executions that `A` should have been paying for. Due to bugs or malicious intent, `B` could forget to call `A` with the signature. If `A` is calling `B` deep in a call tree and `B` replies to it without actually providing an answer, then `A` would need a mechanism to keep the call tree alive while it waits for `B` to call it with the signature in effect running into the same problem that this NEP is attempting to solve. ## Future possibilities One potential future possibility is to allow contracts to specify how long the protocol should wait (up to a certain limit) for the contract to call `promise_yield_resume`. If contracts specify a smaller value, they would potentially be charged a smaller gas fee. This would make contracts more efficient. This enhancement does lead to a more complex implementation and could even allow malicious contracts to more easily concentrate a lot of callbacks to occur at the same time increasing the congestion on the network. Hence, we decided not to include this feature for the time being. ## Consequences [This section describes the consequences, after applying the decision. All consequences should be summarized here, not just the "positive" ones. Record any concerns raised throughout the NEP discussion.] ### Positive - p1 ### Neutral - n1 ### Negative - n1 ### Backwards Compatibility We believe this can be implemented with full backwards compatibility. ## Changelog ### 1.0.0 - Initial Version > Placeholder for the context about when and who approved this NEP version. #### Benefits > List of benefits filled by the Subject Matter Experts while reviewing this version: - Benefit 1 - Benefit 2 #### Concerns > Template for Subject Matter Experts review for this version: > Status: New | Ongoing | Resolved | # | Concern | Resolution | Status | | --: | :------ | :--------- | -----: | | 1 | | | | | 2 | | | | ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0536.md ================================================ --- NEP: 536 Title: Reduce the number of gas refunds Authors: Evgeny Kuzyakov , Bowen Wang Status: Final DiscussionsTo: https://github.com/near/NEPs/pull/536 Type: Protocol Version: 1.0.0 Created: 2024-03-12 LastUpdated: 2023-03-12 --- ## Summary [Gas refund](https://docs.near.org/concepts/basics/transactions/gas#attach-extra-gas-get-refunded) is a mechanism that allows users to get refunded for gas that is not used during the execution of a smart contract. Due to [pessimistic gas pricing](https://docs.near.org/concepts/basics/transactions/gas-advanced#pessimistic-gas-price-inflation), however, even transactions that do not involve function calls generate refunds because users need to pay at a high price and get refunded the difference. Gas refunds lead to nontrivial overhead in runtime and other places, which hurts the performance of the protocol. This proposal aims to reduce the number of gas refunds and prepare for future changes that completely remove gas refunds. ## Motivation Refund receipts create nontrivial overhead: they need to be merklized and sent across shards. In addition, the processing of refund receipts requires additional storage reads and writes, which is not optimal for the performance of the protocol. In addition, when there is congestion, refund receipts may be delayed during execution. Whenever this happens, it requires two additional storage writes to store a gas refund receipt and two additional reads and writes when they are later processed, which incurs a significant overhead. To optimize the performance of the protocol under congestion, it is imperative that we reduce the number of refund receipts. ## Specification Pessimistic gas pricing is removed as a part of this change. This means that transactions that do not involve function calls will not generate gas refund receipts as a result. For function calls, this proposal introduces cost of refund to be ```rust REFUND_FIXED_COST = action_receipt_creation + action_transfer + action_add_function_call_key refund_cost = max(REFUND_FIXED_COST, 0.05 * gas_refund); ``` per receipt. The refund fixed cost includes consideration for implicit accounts (created on transfer) and refund for access key allowance, which requires an access key update. The design of refund cost is supposed to penalize developers from attaching too much gas and creating unnecessary refunds. Some examples: * If the contract wants to refund 280Tgas, burning 5% of it would be about 14Tgas, which is a significant cost and developers would be encouraged to optimize it on the frontend. * If refund is 100Tgas, then 5% is 5Tgas, which is still significant and discourages developers from doing so. * If the refund is <10Tgas (very common case for cross-contract call self-callbacks), the penalty should be just 500Ggas, which is less than the gas refund cost. So only the fixed refund cost will be charged from gas to spawn the gas refund receipt. No UX will be broken for legacy cross-contract call contracts, so long as frontend correctly estimates the required gas in worst case scenario. ## Reference Implementation The protocol changes are as follows: * When a transaction is converted to a receipt, there is no longer a `pessmistic_gas_price` multiplier when the signer balance is deducted. Instead, the signer is charged `transaction_gas_cost * gas_price`. If the transaction succeeds, then unless the transaction contains a function call action, it will not generate any refund. On the other hand, when a transaction with multiple action fails, there is gas refund for the rest of unexecuted actions, same as how the protocol works today. * For function calls, if X gas is attached during the execution of a receipt and Y gas is used+burnt, then `max(0, X-Y-refund_cost)` is refunded at the original gas price where `refund_cost = max(REFUND_FIXED_COST, 0.05 * X-Y)`. In the case the refund is 0 then no refund receipt is generated. * Tokens burnt on refund cost is counted towards tx_balance_burnt and the part over `REFUND_FIXED_COST` is not counted towards gas limit to avoid artificially limiting throughput. * Because refund cost is now separate, action costs do not need to account for refund and therefore should be recalculated and reduced. ## Security Implications This change may lead to less correct charging for transactions when there is congestion. However, the entire gas price mechanism needs to be rethought any ways and when only one or two shards are congested, the gas price wouldn't change so there is no difference. ## Alternatives One altnerative is to completely remove gas refunds by burning all prepaid gas. This idea was [discussed](https://github.com/near/NEPs/issues/107) before. However, it would be a very drastic change and may very negatively damage the developer experience. The approach outlined in this proposal has less impact on developer and user experience and may be extended to burning all prepaid gas in the future. ## Future possibilities * Burning all prepaid gas is a natural extension to this proposal, which would completely get rid of gas refunds. This, however, would be a major change to the developer experience of NEAR and should be treated cautiously. At the very least, developers should be able to easily estimate how much gas a function within a smart contract is going to consume during execution. ## Consequences [This section describes the consequences, after applying the decision. All consequences should be summarized here, not just the "positive" ones. Record any concerns raised throughout the NEP discussion.] ### Positive * p1 ### Neutral * n1 ### Negative * n1 ### Backwards Compatibility Developers may need to change the amount of gas they attach when they write client side code that interacts with smart contracts to avoid getting penalized. However, this is not too difficult to do. ## Changelog [The changelog section provides historical context for how the NEP developed over time. Initial NEP submission should start with version 1.0.0, and all subsequent NEP extensions must follow [Semantic Versioning](https://semver.org/). Every version should have the benefits and concerns raised during the review. The author does not need to fill out this section for the initial draft. Instead, the assigned reviewers (Subject Matter Experts) should create the first version during the first technical review. After the final public call, the author should then finalize the last version of the decision context.] ### 1.0.0 - Initial Version > Placeholder for the context about when and who approved this NEP version. #### Benefits > List of benefits filled by the Subject Matter Experts while reviewing this version: * Benefit 1 * Benefit 2 #### Concerns > Template for Subject Matter Experts review for this version: > Status: New | Ongoing | Resolved | # | Concern | Resolution | Status | | --: | :------ | :--------- | -----: | | 1 | | | | | 2 | | | | ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0539.md ================================================ --- NEP: 539 Title: Cross-Shard Congestion Control Authors: Waclaw Banasik , Jakob Meier Status: Final DiscussionsTo: https://github.com/nearprotocol/neps/pull/539 Type: Protocol Version: 1.0.0 Created: 2024-03-21 LastUpdated: 2024-05-15 --- ## Summary We propose to limit the transactions and outgoing receipts included in each chunk. Limits depend on the congestion level of that receiving shard and are to be enforced by the protocol. Congestion is primarily tracked in terms of the total gas of all delayed receipts. Chunk producers must ensure they stop accepting new transactions if the receiver account is on a shard with a full delayed receipts queue. Forwarding of receipts that are not directly produced by a transaction, namely cross-contract calls and delegated receipts, is limited by the receiver's overall congestion level. This includes the gas of delayed receipts and the gas of receipts that have not been forwarded, yet, due to congestion control restrictions. Additionally, the memory consumption of both receipt types can also cause congestion. This proposal extends the local congestion control rules already in place. It keeps the transaction pool size limit as is but replaces the old delayed receipt count limit with limits on gas and size of the receipts. ## Motivation We want to guarantee the Near Protocol blockchain operates stably even during congestion. Today, when users send transactions at a higher rate than what the network can process, receipts accumulate without limits. This leads to unlimited memory consumption on chunk producers' and validators' machines. Furthermore, the delay for a transaction from when it is accepted to when it finishes execution becomes larger and larger, as the receipts need to queue behind those already in the system. First attempts to limit the memory consumption have been added without protocol changes. This is known as "local congestion control" and can be summarized in two rules: - Limit the transaction pool to 100 MB. https://github.com/near/nearcore/pull/9172 - Once we have accumulated more than 20k delayed receipts in a shard, chunk-producers for that shard stop accepting more transactions. https://github.com/near/nearcore/pull/9222 But these rules do not put any limits on what other shards would accept. For example, when a particular contract is popular, the contract's shard will eventually stop accepting new transactions, but all other shards will keep accepting more and more transactions that produce receipts for the popular contract. Therefore the number of delayed receipts keeps growing indefinitely. Cross-shard congestion control addresses this issue by stopping new transactions at the source and delaying receipt forwarding when the receiving shard has reached its congestion limit. ## Specification The proposal adds fields to chunks headers, adds a new trie column, changes the transaction selection rules, and changes the chunk execution flow. After the concepts section below, the next four sections specify each of these changes in more detail. ### Concepts Below are high-level description of important concepts to make the following sections a bit easier to digest. **Delayed receipts** are all receipts that were ready for execution in the previous chunk but were delayed due to gas or compute limits. They are stored in the delayed receipt queue, which itself is stored in the state of the trie. There is exactly one delayed receipts queue per shard. **Outgoing receipts** are all newly produced receipts as a result of executing receipts or when converting a transaction to non-local receipts. In the absence of congestion, they are all stored in the output of applying receipts, as a simple list of receipts. The **outgoing receipts buffer** is a data structure added by cross-shard congestion control. Each shard has one instance of it in the state trie for every other shard. We use this buffer to store outgoing receipts temporarily when reaching receipt forwarding limits. **Receipt limits** are measured in gas and size. Gas in this context refers to the maximum amount of gas that could be burn when applying the receipt. The receipt size is how much space it takes in memory, measured in bytes. The **congestion level** is an indicator between 0 and 1 that determines how strong congestion prevention measures should be. The maximum congestion measured is reached at congestion level 1. This value is defined separately for each shard and computed as the maximum value of the following congestion types. - **Incoming congestion** increases linearly with the amount of gas in delayed receipts. - **Outgoing congestion** increases linearly with the amount of gas in any of the outgoing buffers of the shard. - **Memory congestion** increases linearly with the total size of all delayed and buffered outgoing receipts. - **Missed chunk congestion** rises linearly with the number of missed chunks since the last successful chunk, measure in block height difference. ### Chunk header changes We change the chunk header to include congestion information, adding four indicators. ```rust /// sum of gas of all receipts in the delayed receipts queue, at the end of chunk execution delayed_receipts_gas: u128, /// sum of gas of all receipts in all outgoing receipt buffers, at the end of chunk execution buffered_receipts_gas: u128, /// sum of all receipt sizes in the delayed receipts queue and all outgoing buffers receipt_bytes: u64, /// if the congestion level is 1.0, the only shard that can forward receipts to this shard in the next chunk /// not relevant if the congestion level is < 1.0 allowed_shard: u16, ``` The exact header structure and reasons for the particular integer sizes are described in more details in the [reference implementation](#efficiently-computing-the-congestion-information) section. Usage of these fields is described in [Changes to chunk execution](#changes-to-chunk-execution). This adds 42 bytes to the chunk header, increasing it from 374 bytes up to 416 bytes in the borsh representation (assuming no validator proposals are included.) This in turn increases the block size by 42 bytes *per shard*, as all chunk headers are fully included in blocks. Including all this information in the chunk header enables efficient validation. Using the previous chunk header (or alternatively, the state), combined with the list of receipts applied and forwarded, a validator can check that the congestion rules described in this NEP are fulfilled. ### Changes to transaction selection We change transaction selection to reject new transactions when the system is congested, to reduce to total workload in the system. Today, transactions are taken from the chunk producer's pool until `tx_limit` is reached, where `tx_limit` is computed as follows. ```python # old tx_limit = 500 Tgas if len(delayed_receipts) < 20_000 else 0 ``` We replace the formula for the transaction limit to depend on the `incoming_congestion` variable (between 0 and 1) computed in the previous chunk of the same shard: ```python # new MIN_TX_GAS = 20 Tgas MAX_TX_GAS = 500 Tgas tx_limit = mix(MAX_TX_GAS, MIN_TX_GAS, incoming_congestion) ``` This smoothly limits the acceptance of new work, to prioritize reducing the backlog of delayed receipts. In the pseudo code above, we borrow the [`mix`](https://docs.gl/sl4/mix) function from GLSL for linear interpolation. > `mix(x, y, a)` > > `mix` performs a linear interpolation between x and y using a to weight between > them. The return value is computed as $x \times (1 - a) + y \times a$. More importantly, we add a more targeted rule to reject all transactions *targeting* a shard with a congestion level above a certain threshold. ```python def filter_tx(tx): REJECT_TX_CONGESTION_THRESHOLD = 0.25 if congestion_level(tx.receiver_shard_id) > REJECT_TX_CONGESTION_THRESHOLD tx.reject() else tx.accept() ``` Here, `congestion_level()` is the maximum of incoming, outgoing, memory, and missed chunk congestion. This stops (some) new incoming work at the source, when a shard is using too much memory to store unprocessed receipts, or if there is already too much work piled up for that shard. Chunk validators must validate that the two rules above are respected in a produced chunk. ### Changes to chunk execution We change chunk execution to hold back receipt forwarding to congested shards. This has two effects. 1. It prevents the memory consumption of the congested shard from growing at the expense of buffering these pending receipts on the outgoing shards. 2. When user demand is consistently higher than what the system can handle, this mechanism lets backpressure propagate shard-by-shard until it reaches the shards responsible for accepting too many receipts and causes transaction filtering to kick in. To accomplish this, we add 3 new steps to chunk execution (enumerated as 1, 2, 6 below) and modify how outgoing receipts are treated in the transaction conversion step (3) and in the receipts execution step (4). The new chunk execution then follows this order. 1. (new) Read congestion information for *all* shards from the previous chunk headers. ```rust // congestion info for each shard, as computed in the last included chunk of the shard { delayed_receipts_gas: u128, buffered_receipts_gas: u128, receipt_bytes: u64, allowed_shard: u16, // extended congestion info, as computed from the latest block header missed_chunks_count: u64 } ``` 2. (new) Compute bandwidth limits to other shards based on the congestion information. The formula is: ```python for receiver in other_shards: MAX_CONGESTION_INCOMING_GAS = 20 Pgas incoming_congestion = delayed_receipts_gas[receiver] / MAX_CONGESTION_INCOMING_GAS MAX_CONGESTION_OUTGOING_GAS = 2 Pgas outgoing_congestion = buffered_receipts_gas[receiver] / MAX_CONGESTION_OUTGOING_GAS MAX_CONGESTION_MEMORY_CONSUMPTION = 1000 MB memory_congestion = receipt_bytes[receiver] / MAX_CONGESTION_MEMORY_CONSUMPTION MAX_CONGESTION_MISSED_CHUNKS = 10 missed_chunk_congestion = missed_chunks_count[receiver] / MAX_CONGESTION_MISSED_CHUNKS congestion = max(incoming_congestion, outgoing_congestion, memory_congestion, missed_chunk_congestion) if congestion >= 1.0: # Maximum congestion: reduce to minimum speed if current_shard == allowed_shard[receiver]: outgoing_gas_limit[receiver] = 1 Pgas else: outgoing_gas_limit[receiver] = 0 else: # Green or Amber # linear interpolation based on congestion level MIN_GAS_FORWARDING = 1 Pgas MAX_GAS_FORWARDING = 300 Pgas outgoing_gas_limit[receiver] = mix(MAX_GAS_FORWARDING, MIN_GAS_FORWARDING, congestion) ``` 3. (new) Drain receipts in the outgoing buffer from the previous round - Subtract `receipt.gas()` from `outgoing_gas_limit[receipt.receiver]` for each receipt drained. - Keep receipts in the buffer if the gas limit would be negative. - Subtract `receipt.gas()` from `outgoing_congestion` and `receipt.size()` from `receipt_bytes` for the local shard for every forwarded receipt. - Add the removed receipts to the outgoing receipts of the new chunk. 4. Convert all transactions to receipts included in the chunk. - Local receipts, which are receipts where the sender account id is equal to the receiver id, are set aside as local receipts for step 5. - Non-local receipts up to `outgoing_gas_limit[receipt.receiver]` for the respective shard go to the outgoing receipts list of the chunk. - (new) Non-local receipts above `outgoing_gas_limit[receipt.receiver]` for the respective shard go to the outgoing receipts buffer. - (new) For each receipt added to the outgoing buffer, add `receipt.gas()` to `outgoing_congestion` and `receipt.size()` to `receipt_bytes` for the local shard. 5. Execute receipts in the order of `local`, `delayed`, `incoming`, `yield-resume time-out`. - Don't stop before all receipts are executed or more than 1000 Tgas have been burnt. Burnt gas includes the burnt gas from step 4. - Outgoing receipts up to what is left in `outgoing_gas_limit[receipt.receiver]` per shard (after step 3) go to the outgoing receipts list of the chunk. - (new) Outgoing receipts above `outgoing_gas_limit[receipt.receiver]` go to the outgoing receipts buffer. - (new) For each delayed executed receipt, remove `receipt.gas()` from `incoming_congestion` and `receipt.size()` from `receipt_bytes`. 6. Remaining local or incoming receipts are added to the end of the `delayed` receipts queue. - (new) For each receipt added to the delayed receipts queue, add `receipt.gas()` to `incoming_congestion` and `receipt.size()` to `receipt_bytes`. 7. (new) Write own congestion information into the result, to be included in the next chunk header. - If the congestion level is >= 1.0, the `allowed_shard` can be chosen freely by the chunk producer. Selecting the own shard means nobody can send. The reference implementations uses round-robin assignment of all other shards. Further optimization can be done without requiring protocol changes. - If the congestion level is <= 1.0, the `allowed_shard` value does not affect congestion control. But chunk producer must set it to the own shard in this case. In the formula above, the receipt gas and the receipt size are defined as: ```python def gas(receipt): return receipt.attached_gas + receipt.exec_gas def size(receipt): return len(borsh(receipt)) ``` ### Changes to Trie We store the outgoing buffered receipts in the trie, similar to delayed receipts but in their own separate column. But instead of a single queue per shard, we add one queue for each other shard at the current sharding layout. We add two trie columns: - `BUFFERED_RECEIPT_INDICES: u8 = 13;` - `BUFFERED_RECEIPT: u8 = 14` The `BUFFERED_RECEIPT_INDICES` column only has one value, which stores a borsh-serialized instance of `BufferedReceiptIndices` defines as follows: ```rust pub struct BufferedReceiptIndices { pub shard_buffer_indices: BTreeMap, } pub struct ShardBufferedReceiptIndices { // First inclusive index in the queue. pub first_index: u64, // Exclusive end index of the queue pub next_available_index: u64, } ``` The `BUFFERED_RECEIPT` column stores receipts keyed by `TrieKey::BufferedReceipt{ receiving_shard: ShardId, index: u64 }`. The `BufferedReceiptIndices` map defines which queues exist, which changes during resharding. For each existing queue, all receipts in the range `[first_index, next_available_index)` (inclusive start, exclusive end) must exist under the key with the corresponding shard. ### Notes on parameter fine-tuning Below are the reasons why each parameter is set to the specific value given above. For more details, a spreadsheet with the full analysis can be found here: https://docs.google.com/spreadsheets/d/1Vt_-sgMdX1ncYleikYY8uFID_aG9RaqJOqaVMLQ37tQ/edit#gid=0 #### Queue sizes The parameters are chosen to strike a balance between guarantees for short queues and utilization. 20 PGas delayed receipts means that incoming receipts have to wait at most 20 chunks to be applied. And it can guarantee 100% utilization as long as the ratio between burnt and attached gas in receipts is above 1 to 20. A shorter delayed queue would result in lower delays but in our model simulations, we saw reduced utilization even in simple and balanced workloads. The 1 GB of memory is a target value for the control algorithm to try and stay below. With receipts in the normal range of sizes seen in today's traffic, we should never even get close to 1 GB. But the protocol allows a single receipt to be multiple MBs. In those cases, a limit of 1 GB still gives us almost 100% utilization but prevents queues from growing larger than what validators can keep in memory. #### Receipt forwarding limits `MIN_GAS_FORWARDING = 1 Pgas` and `MAX_GAS_FORWARDING = 300 Pgas` give us a large range to smooth out how much should be forwarded to other shards. Depending on the burnt to attached gas ratio of the workload, it will settle at different values for each directed shard pair. This gives the algorithm adaptability to many workload patterns. For the forwarding to work smoothly, we need a bit of an outgoing buffer queue. We found in simulations that `MAX_CONGESTION_OUTGOING_GAS = 2 Pgas` is enough for the forwarding limit to settle in the perfect spot before we are restricted by transaction rejection. Higher values did not yield better results but it does increase delays in some cases, hence we propose 2 Pgas. #### Limiting new transactions The remaining parameters work together to adapt how much new workload we accept in the system, based on how congested the chain is already. `REJECT_TX_CONGESTION_THRESHOLD = 0.25` defines how quickly we start rejecting new workload to a shard. Combined with the 20 PGas limit on the delayed receipts queue, we only reject new work if we have at least 5 PGas excess workload sent to that shard already. This is more aggressive than other mechanisms simply because rejecting more workload to known-to-be congested shards is the most effective tool to prevent the system from accumulating more transactions. The sooner we do it, the shorter the delays experienced by users who got their transactions accepted. `MIN_TX_GAS = 20 Tgas` and `MAX_TX_GAS = 500 Tgas` gives a large range to smooth out the split between gas spend on new transactions vs delayed receipts. This only looks at how many delayed receipts the local shard has, not at the receiving shard. Depending on the workload, it will settle at different values. Note that hitting `REJECT_TX_CONGESTION_THRESHOLD`, which looks at the congestion of the receiving shard, overrules this range and stops all transactions to the congested shard when it is hit. `MIN_TX_GAS = 20 Tgas` guarantees that we can still accept a decent amount of new transactions to shards that are not congested, even if the local shard itself is fully congested. This gives fairness properties under certain workloads which we could not achieve in any other of the tried congestion control strategies. This is also useful to add transaction priority in [NEP-541](https://github.com/near/NEPs/pull/541), as we can always auction off the available space for new transactions without altering the congestion control algorithm. ## Reference Implementation A reference implementation is available in this PR against nearcore: https://github.com/near/nearcore/pull/10918 Here are the most important details which are not already described in the specification above but are defined in the reference implementation. ### Efficiently computing the congestion information The congestion information is computed based on the gas and size of the incoming queue and the outgoing buffers. A naive implementation would just iterate over all of the receipts in the queue and buffers and sum up the relevant metrics. This approach is slow and, in the context of stateless validation, would add too much to the state witness size. In order to prevent those issues we consider two alternative optimizations. Both use the same principle of caching the previously calculated metrics and updating them based on the changes to the incoming queue and outgoing buffers. After applying a chunk, we store detailed information of the shard in the chunk extra. Unlike the shard header, this is only stored on the shard and not shared globally. The new fields in the chunk extra are included in `ChunkExtraV3`. ```rust pub struct ChunkExtraV3 { // ...all fields from ChunkExtraV2 pub congestion_info: CongestionInfo, } pub struct CongestionInfo { delayed_receipts_gas: u128, buffered_receipts_gas: u128, receipt_bytes: u64, allowed_shard: u16, } ``` This implementation allows to efficiently update the `StoredReceiptsInfo` during chunk application by starting with the information of the previous chunk and applying only the changes. Regarding integer sizes, `delayed_receipts_gas` and `buffered_receipts_gas` use 128-bit unsigned integers because 64-bit would not always be enough. `u64::MAX` would only be enough to store `18_446 Pgas`. This translates to roughly 5 hours of work, assuming 1 Pgas per second. While the proposed congestion control strategy should prevent congestion ever reaching such high levels, it is not possible to rule out completely. For `receipt_bytes`, a `u64` is more than enough, we have other problems if we need to store millions of Terabytes of receipts. For the id of the allowed shard, we chose a `u16` which is large enough for 65_535 shards. ### Bootstrapping The previous section explain how the gas and bytes information of unprocessed receipts is computed based on what it was for the previous chunk. But for the first chunk with this feature enabled, the information for the previous chunk is not available. In this specific case, we detect that the previous information is not available and therefore we trigger an iteration of the existing queues to compute the correct values. This computed `StoredReceiptsInfo` only applies locally. But the next value of it will be shared in the chunk header and other shards will start using it to limit the transactions they accept and receipts they forward. The congestion info of other shards is assumed to be 0 for all values for the first block with the cross-shard congestion control feature enabled. ### Missing Chunks When a chunk is missing, we use the congestion information of the last available chunk header for the shard. In practical terms this simply means we take the chunk header available in the block, even if the included height is not the latest. Additionally, we include the number of missed chunks as part of the congestion formula, treating a shard with 10 or missed chunks the same way as an otherwise fully congested shard. This is to prevent sending even more receipts to a shard that already struggles to produce chunks. ### Validation Changes The following fields in the chunk header must be validated: - `receipt_bytes`: must be equal to `receipt_bytes` of the previous chunk, plus all bytes of new receipts added to delayed or buffered receipts, minus all the receipts removed of the same types. - `delayed_receipts_gas` must be equal to `delayed_receipts_gas` of the previous chunk, plus the gas of receipts added to the delayed receipts queue, minus the gas of receipts removed from the delayed receipts queue. - `buffered_receipts_gas` must be equal to `buffered_receipts_gas` of the previous chunk, plus the gas of receipts added to any of the outgoing receipts buffers, minus the gas of all forwarded buffered receipts. - `allowed_shard` must be a valid shard id. - `allowed_shard` must be equal to the chunk's shard id if congestion is below 1. The balance checker also needs to take into account balances stored in buffered receipts. ## Security Implications With cross-shard congestion control enabled, malicious users could try to find patterns that clog up the system. This could potentially lead to cheaper denial of service attacks compared to today. If such patterns exists, most likely today's implementation would suffer from different problems, such as validators requiring unbounded amounts of memory. Therefore, we believe this feature is massive step forward in terms of security, all things considered. ## Integration with state sync What we described in [Efficiently computing the congestion information](#efficiently-computing-the-congestion-information) creates a dependence on the previous block when processing a block. For a fully synced node this requirement is always fulfilled because we keep at least 3 epochs of blocks. However in state sync we start processing from an arbitrary place in chain without access to full history. In order to integrate the congestion control and state sync features we will add extra steps in state sync to download the blocks that may be needed in order to finalize state sync. The blocks that are needed are the `sync hash` block, the `previous block` where state sync creates chunk extra in order to kick off block sync and the `previous previous block` that is now needed in order to process the `previous block`. On top of that we may need to download further blocks to ensure that every shard has at least one new chunk in the blocks leading up to the sync hash block. ## Integration with resharding Resharding is a process wherin we change the shard layout - the assignment of accound ids to shards. The centerpiece of resharding is moving the trie / state records from parent shards to children shards. It's important to preserve the ability to perform resharding while adding other protocol features such as congestion control. Below is a short description how resharding and congestion control can be integrated, in particular how to reshard the new trie columns - the outgoing buffers. For simplicity we'll only consider splitting a single parent shard into multiple children shards which is currently the only supported operation. The actual implementation of this integration will be done independently and outside of this effort. Importantly the resharding affects both the shard that is being split and all the other shards. #### Changes to the shard under resharding The outgoing buffers of the parent shard can be split among children by iterating all of the receipts in each buffer and inserting it to appropriate child shard. The assignment can in theory be arbitrary e.g. all receipts can be reassigned to a single shard. In practice it would make sense to either split the receipts equally between children or based on the sender account id of the receipt. Special consideration should be given to refund receipts where the sender account is "system" that may not belong to neither parent nor children shards. Any assignment of such receipts is fine. #### Changes to the other shards The other shards, that is all shards that are not under resharding, have an outgoing buffer to the shard under resharding. This buffer should be split into one outgoing buffer per child shard. The buffer can be split by iterating receipts and reassigning each to either of the child shards. Each receipt can be reassigned based on it's receiver account id and the new shard layout. ## Alternatives A wide range of alternatives has been discussed. It would be too much to list all suggested variations of all strategies. Instead, here is a list of different directions that were explored, with a representative strategy for each of them. 1. Use transaction fees and an open market to reduce workload added to the system. - Problem 1: This does not prevent unbounded memory requirements of validators, it just makes it more expensive. - Problem 2: In a sharded environment like Near Protocol, it is hard to implement this fairly. Because it's impossible to know where a transaction burns most of its gas, the only simple solution would require all shards to pay the price for the most congested shard. 2. Set fixed limits for delayed receipts and drop receipts beyond that. - Problem 1: Today, smart contract rely on receipts never being lost. This network-level failure mode would be completely new. - Problem 2: We either need to allow resuming with external inputs, or roll-back changes on all shards to still have consistent states in smart contracts. Both solutions means we are doing extra work when being congested, inevitably reducing the available throughput for useful work in times when the demand is the largest. 3. Stop accepting transactions globally when any shard has too long of a delayed receipts queue. ([See this issue](https://github.com/near/nearcore/issues/9228).) - Problem 1: This gives poor utilization in many workloads, as our model simulations confirmed. - Problem 2: A global stop conflicts with plans to add fee based transaction priorities which should allow sending transactions even under heavy congestion. 4. Reduce newly accepted transactions solely based on gas in delayed queues, without adding new buffers or queues to the system. Gas is tracked per shard of the transaction signer. ([Zulip Discussion](https://near.zulipchat.com/#narrow/stream/295558-core/topic/congestion.20control/near/429973223) and [idea in code](https://github.com/near/nearcore/pull/10894).) - Problem 1: This requires `N` gas numbers in each chunk header, or `N*N` numbers per block, where `N` is the number of shards. - Problem 2: We did not have time to simulate it properly. But on paper, it seems each individual delayed queue can still grow indefinitely as the number of shards in the system grows. 5. Smartly track and limit buffer space across different shards. Only accept new transactions if enough buffer space can be reserved ahead of time. - Problem 1: Without knowing which shards a transaction touches and how large receipts will be, we have to pessimistically reserve more space than most receipts will actually need. - Problem 2: If buffer space is shared globally, individual queues can still grow really large, even indefinitely if we assume the number of shards grows over time. - Problem 3: If buffer space is on a per-shard basis, we run into deadlocks when two shards have no more space left but both need to send to the other shard to make progress. 6. Require users to define which shards their transactions will touch and how much gas is burnt in each. Then use this information for global scheduling such that congestion is impossible. - Problem 1: This requires lots of changes across the infrastructure stack. It would take too long to implement as we are already facing congestion problems today. - Problem 2: This would have a strong impact on usability and it is unclear if gas usage estimating services could close the gap to make it acceptable. 7. An alternative way to what is described in [Efficiently computing the congestion information](#efficiently-computing-the-congestion-information) would be to store the total gas and total size of the incoming queue and the outgoing receipts in the state along the respective queue or buffers. Those values will be updated as receipts are added or removed from the queue. - Pro: In this case the CongestionInfo struct can remain small and only reflect the information needed by other shards. (3 bytes instead of 42 bytes) ```rust pub struct CongestionInfo { allowed_shard: u16, congestion_level: u8, } ``` - Con: Overall, it would result in more state changes per chunk, since the congestion value needs to be read before applying receipts anyway. In light of stateless validation, this would be worse for the state witness size ## Future possibilities While this proposal treats all requests the same, it sets the base for a proper transaction priority implementation. We co-designed this proposal with [NEP-541](https://github.com/near/NEPs/pull/541), which adds a transaction priority fee. On a very high level, the fee is used to auction off a part of the available gas per chunk to the highest bidders. We also expect that this proposal alone will not be the final solution for congestion control. Rather, we just want to build a solid foundation in this NEP and allow future optimization to take place on top of it. For example, estimations on how much gas is burnt on each shard could help with better load distribution in some cases. We also forsee that the round-robin scheduling of shard allowed to forward even under full congestion is not perfect. It is a key feature to make deadlocks provably impossible, since every shard is guaranteed to make a minimum progress after N rounds. But it could be beneficial to allocate more bandwidth to shards that actually have something to forward, or perhaps it would be better to stop forwarding anything for a while. The current proposal allows chunk producers to experiment with this without a protocol version change. Lastly, a future optimization could do better transaction rejection for meta transactions. Instead of looking only at the outer transaction receiver, we could also look at the receiver of the delegate action, which is most likely where most gas is going to be burnt, and use this for transaction rejection. ## Consequences ### Positive - Accepted transaction have lower latencies compared to today under congestion. - Storage and memory requirements on validator for storing receipts are bounded. ### Neutral - More transactions are rejected at the chunk producer level. ### Negative - Users need to resend transaction more often. ### Backwards Compatibility There are no observable changes on the smart contract, wallet, or API level. Thus, there are no backwards-compatibility concerns. ## Unresolved Issues (Optional) These congestion problems are out of scope for this proposal: - Malicious patterns can still cause queues to grow beyond the parameter limits. - There is no way to pay for higher priority. ([NEP-541](https://github.com/near/NEPs/pull/541) adds it.) Postponed receipts are also considered to be added to `receipt_bytes`. But at this point it seems better to not include them to avoid further complications with potential deadlocks, since postponed receipts can only be executed when incoming receipts are allowed to come in. Following the same logic, yielded receipts are also excluded from the size limits, as they require incoming receipts to resume. A solution that also address the memory space of postponed and yielded receipts could be added with future proposals but is not considered necessary for this first iteration of cross-shard congestion control. ## Changelog [The changelog section provides historical context for how the NEP developed over time. Initial NEP submission should start with version 1.0.0, and all subsequent NEP extensions must follow [Semantic Versioning](https://semver.org/). Every version should have the benefits and concerns raised during the review. The author does not need to fill out this section for the initial draft. Instead, the assigned reviewers (Subject Matter Experts) should create the first version during the first technical review. After the final public call, the author should then finalize the last version of the decision context.] ### 1.0.0 - Initial Version > Placeholder for the context about when and who approved this NEP version. #### Benefits > List of benefits filled by the Subject Matter Experts while reviewing this version: - Benefit 1 - Benefit 2 #### Concerns > Template for Subject Matter Experts review for this version: > Status: New | Ongoing | Resolved | # | Concern | Resolution | Status | | --: | :------ | :--------- | -----: | | 1 | | | | | 2 | | | | ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0568.md ================================================ --- NEP: 568 Title: Resharding V3 Authors: Adam Chudas, Aleksandr Logunov, Andrea Spurio, Marcelo Diop-Gonzalez, Shreyan Gupta, Waclaw Banasik Status: Final DiscussionsTo: https://github.com/near/nearcore/issues/11881 Type: Protocol Version: 1.0.0 Created: 2024-10-24 LastUpdated: 2024-10-24 --- ## Summary This proposal introduces a new resharding implementation and shard layout for production networks. The primary objective of Resharding V3 is to increase chain capacity by splitting overutilized shards. A secondary aim is to lay the groundwork for supporting Dynamic Resharding, Instant Resharding, and Shard Merging in future updates. ## Motivation The sharded architecture of the NEAR Protocol is a cornerstone of its design, enabling parallel and distributed execution that significantly boosts overall throughput. Resharding plays a pivotal role in this system, allowing the network to adjust the number of shards to accommodate growth. By increasing the number of shards, resharding ensures the network can scale seamlessly, alleviating existing congestion, managing rising traffic demands, and welcoming new participants. This adaptability is essential for maintaining the protocol's performance, reliability, and capacity to support a thriving, ever-expanding ecosystem. Resharding V3 is a significantly redesigned approach, addressing limitations of the previous versions, [Resharding V1][NEP-040] and [Resharding V2][NEP-508]. The earlier solutions became obsolete due to major protocol changes since Resharding V2, including the introduction of Stateless Validation, Single Shard Tracking, and Mem-Trie. ## Specification Resharding will be scheduled in advance by the NEAR developer team. The new shard layout will be hardcoded into the `neard` binary and linked to the protocol version. As the protocol upgrade progresses, resharding will be triggered during the post-processing phase of the last block of the epoch. At this point, the state of the parent shard will be split between two child shards. From the first block of the new protocol version onward, the chain will operate with the new shard layout. There are two key dimensions to consider: state storage and protocol features, along with additional details. 1. **State Storage**: Currently, the state of a shard is stored in three distinct formats: the state, the flat state, and the mem-trie. Each of these representations must be resharded. Logically, resharding is an almost instantaneous event that occurs before the first block under the new shard layout. However, in practice, some of this work may be deferred to post-processing, as long as the chain's view reflects a fully resharded state. 2. **Protocol Features**: Several protocol features must integrate smoothly with the resharding process, including: * **Stateless Validation**: Resharding must be validated and proven through stateless validation mechanisms. * **State Sync**: Nodes must be able to synchronize the states of the child shards post-resharding. * **Cross-Shard Traffic**: Receipts sent to the parent shard may need to be reassigned to one of the child shards. * **Receipt Handling**: Delayed, postponed, buffered, and promise-yield receipts must be correctly distributed between the child shards. * **ShardId Semantics**: The shard identifiers will become abstract identifiers where today they are numbers in the `0..num_shards` range. * **Congestion Info**: `CongestionInfo` in the chunk header will be recalculated for the child shards at the resharding boundary. Proof must be compatible with Stateless Validation. ### State Storage - MemTrie MemTrie is the in-memory representation of the trie that the runtime uses for all trie accesses. It is kept in sync with the Trie representation in the state. Currently, it isn't mandatory for nodes to have the MemTrie feature enabled, but going forward with Resharding V3, all nodes will be required to have MemTrie enabled for resharding to happen successfully. For resharding, we need an efficient way to split the MemTrie into two child tries based on the boundary account. This splitting happens at the epoch boundary when the new epoch is expected to have the two child shards. The requirements for MemTrie splitting are: * **Instantaneous Splitting**: MemTrie splitting needs to happen efficiently within the span of one block. The child tries need to be available for processing the next block in the new epoch. * **Compatibility with Stateless Validation**: We need to generate a proof that the MemTrie split proposed by the chunk producer is correct. * **State Witness Size Limits**: The proof generated for splitting the MemTrie needs to comply with the size limits of the state witness sent to all chunk validators. This prevents us from iterating through all trie keys for delayed receipts, etc. With the Resharding V3 design, there's no protocol change to the structure of MemTries; however, implementation constraints required us to introduce the concept of a Frozen MemTrie. More details are in the [implementation](#state-storage---memtrie-1) section below. Based on these requirements, we developed an algorithm to efficiently split the parent trie into two child tries. Trie entries can be divided into three categories based on whether the trie keys have an `account_id` prefix and the total number of such trie keys. Splitting of these keys is handled differently. #### TrieKey with AccountID Prefix This category includes most trie keys like `TrieKey::Account`, `TrieKey::ContractCode`, `TrieKey::PostponedReceipt`, etc. For these keys, we can efficiently split the trie based on the boundary account trie key. We only need to read all the intermediate nodes that form part of the split key. In the example below, if "pass" is the split key, we access all the nodes along the path of `root` ➔ `p` ➔ `a` ➔ `s` ➔ `s`, while not needing to touch other intermediate nodes like `o` ➔ `s` ➔ `t` in key "post". The accessed nodes form part of the state witness, as those are the only nodes needed by validators to verify that the resharding split is correct. This limits the size of the witness to effectively O(depth) of the trie for each trie key in this category. ![Splitting Trie diagram](assets/nep-0568/NEP-SplitState.png) #### Singleton TrieKey This category includes the trie keys `TrieKey::DelayedReceiptIndices`, `TrieKey::PromiseYieldIndices`, and `TrieKey::BufferedReceiptIndices`. These are just a single entry (or O(num_shard) entries) in the trie and are small enough to read and modify efficiently for the child tries. #### Indexed TrieKey This category includes the trie keys `TrieKey::DelayedReceipt`, `TrieKey::PromiseYieldTimeout`, and `TrieKey::BufferedReceipt`. The number of entries for these keys can potentially be arbitrarily large, making it infeasible to iterate through all entries. In the pre-stateless validation world, where we didn't care about state witness size limits, for Resharding V2 we could iterate over all delayed receipts and split them into the respective child shards. For Resharding V3, these are handled by one of two strategies: * **Duplication Across Child Shards**: `TrieKey::DelayedReceipt` and `TrieKey::PromiseYieldTimeout` are handled by duplicating entries across both child shards, as each entry could belong to either child shard. More details are in the [Delayed Receipts](#delayed-receipt-handling) and [Promise Yield](#promiseyield-receipt-handling) sections below. * **Assignment to Lower Index Child**: `TrieKey::BufferedReceipt` is independent of the `account_id` and can be sent to either of the child shards, but not both. We copy the buffered receipts and the associated metadata to the child shard with the lower index. More details are in the [Buffered Receipts](#buffered-receipt-handling) section below. ### State Storage - Flat State Flat State is a collection of key-value pairs stored on disk, with each entry containing a reference to its `ShardId`. When splitting a shard, every item inside its Flat State must be correctly reassigned to one of the new child shards; however, due to technical limitations, such an operation cannot be completed instantaneously. Flat State's main purposes are allowing the creation of State Sync snapshots and the construction of Mem Tries. Fortunately, these two operations can be delayed until resharding is completed. Note also that with Mem Tries enabled, the chain can move forward even if the current status of Flat State is not in sync with the latest block. For these reasons, the chosen strategy is to reshard Flat State in a long-running background task. The new shards' states must converge with their Mem Tries representation in a reasonable amount of time. Splitting a shard's Flat State is performed in multiple steps: 1. A post-processing "split" task is created instantaneously during the last block of the old shard layout. 2. The "split" task runs in parallel with the chain for a certain amount of time. Inside this routine, every key-value pair belonging to the shard being split (also called the parent shard) is copied into either the left or the right child Flat State. Entries linked to receipts are handled in a special way. 3. Once the task is completed, the parent shard's Flat State is cleaned up. The child shards' Flat States have their state in sync with the last block of the old shard layout. 4. Child shards must apply the delta changes from the first block of the new shard layout until the final block of the canonical chain. This operation is done in another background task to avoid slowdowns while processing blocks. 5. Child shards' Flat States are now ready and can be used to take State Sync snapshots and to reload Mem Tries. ### State Storage - State Each shard’s Trie is stored in the `State` column of the database, with keys prefixed by `ShardUId`, followed by a node's hash. This structure uniquely identifies each shard’s data. To avoid copying all entries under a new `ShardUId` during resharding, a mapping strategy allows child shards to access ancestor shard data without directly creating new entries. A naive approach to resharding would involve copying all `State` entries with a new `ShardUId` for a child shard, effectively duplicating the state. This method, while straightforward, is not feasible because copying a large state would take too much time. Resharding needs to appear complete between two blocks, so a direct copy would not allow the process to occur quickly enough. To address this, Resharding V3 employs an efficient mapping strategy, using the `DBCol::ShardUIdMapping` column to link each child shard’s `ShardUId` to the closest ancestor’s `ShardUId` holding the relevant data. This allows child shards to access and update state data under the ancestor shard’s prefix without duplicating entries. Initially, `ShardUIdMapping` is empty, as existing shards map to themselves. During resharding, a mapping entry is added to `ShardUIdMapping`, pointing each child shard’s `ShardUId` to the appropriate ancestor. Mappings persist as long as any descendant shard references the ancestor’s data. Once a node stops tracking all children and descendants of a shard, the entry for that shard can be removed, allowing its data to be garbage collected. This mapping strategy enables efficient shard management during resharding events, supporting smooth transitions without altering storage structures directly. #### Integration with Cold Storage (Archival Nodes) Cold storage uses the same mapping strategy to manage shard state during resharding: * When state data is migrated from hot to cold storage, it retains the parent shard’s `ShardUId` prefix, ensuring consistency with the mapping strategy. * While copying data for the last block of the epoch where resharding occurred, the `DBCol::StateShardUIdMapping` column is copied into cold storage. This ensures that mappings are updated alongside the shard state data. * These mappings are permanent in cold storage, aligning with its role in preserving historical state. This approach minimizes complexity while maintaining consistency across hot and cold storage. #### State cleanup Since [Stateless Validation][NEP-509], all shards tracking is no longer required. Currently, shard cleanup (e.g. when we stopped tracking one shard and started tracking another shard) has not been implemented. With resharding, we would also want to cleanup parent shard once we stop tracking all its descendants. We propose a shard cleanup mechanism that will also handle post-resharding cleanup. When garbage collection removes the last block of an epoch from the canonical chain, we determine which shards were tracked during that epoch by examining the shards present in `TrieChanges` at that block. Similarly, we collect information on shards tracked in subsequent epochs, up to the present one. A shard State is removed only if: * It was tracked in the old epoch (for which the last block has just been garbage collected). * It was not tracked in later epochs, is not currently tracked, and will not be tracked in the next epoch. To ensure compatibility with resharding, instead of checking tracked shards directly, we analyze the `ShardUId` prefixes they use. A parent shard's state is retained as long as it remains referenced in `DBCol::StateShardUIdMapping` by any descendant shard. Once all descendant shards are no longer tracked, we clean up the parent shard's state (along with its descendants) and remove all mappings to the parent from `DBCol::StateShardUIdMapping`. #### Negative refcounts Some trie keys, such as `TrieKey::DelayedReceipt`, are shared among child shards, but their corresponding State is not duplicated. The `DBCol::State` column uses reference counting, meaning that some data is counted only once, even if referenced by multiple child shards. As a result, removing the data can sometimes lead to negative refcounts. To address this, we have modified the RocksDB `refcount_merge` behavior so that negative refcounts are clamped to zero. However, this is suboptimal, as it can lead to some State being leaked. Specifically, if two operations decrement the refcount for the same key, the RocksDB compaction process may merge them before they are applied, effectively canceling each other out. As a result, the key would never be removed from disk until state sync occurs. This is a temporary solution, and we should follow up on it later. ### Stateless Validation Since only a fraction of nodes track the split shard, it is necessary to prove the transition from the state root of the parent shard to the new state roots for the child shards to other validators. Without this proof, chunk producers for the split shard could collude and provide invalid state roots, potentially compromising the protocol, such as by minting tokens out of thin air. The design ensures that generating and verifying this state transition is negligible in time compared to applying a chunk. As detailed in the [State Storage - MemTrie](#state-storage---memtrie) section, the generation and verification logic involves a constant number of trie lookups. Specifically, we implement the `retain_split_shard(boundary_account, RetainMode::{Left, Right})` method for the trie, which retains only the keys in the trie that belong to the left or right child shard. Internally, this method uses `retain_multi_range(intervals)`, where `intervals` is a vector of trie key intervals to retain. Each interval corresponds to a unique trie key type prefix byte (`Account`, `AccessKey`, etc.) and defines an interval from the empty key to the `boundary_account` key for the left shard, or from the `boundary_account` to infinity for the right shard. The `retain_multi_range` method is recursive. Based on the current trie key prefix covered by the current node, it either: * Returns the node if the subtree is fully contained within an interval. * Returns an "empty" node if the subtree is outside all intervals. * Descends into all children and constructs a new node with children returned by recursive calls. This implementation is agnostic to the trie storage used for retrieving nodes and applies to both MemTries and partial storage (state proof). * Calling it for MemTrie generates a proof and a new state root. * Calling it for partial storage generates a new state root. If the method does not fail with an error indicating that a node was not found in the proof, it means the proof was sufficient, and it remains to compare the generated state root with the one proposed by the chunk producer. ### State Witness The resharding state transition becomes one of the `implicit_transitions` in `ChunkStateWitness`. It must be validated between processing the last chunk (potentially missing) in the old epoch and the first chunk (potentially missing) in the new epoch. The `ChunkStateTransition` fields correspond to the resharding state transition: the `block_hash` stores the hash of the last block of the parent shard, the `base_state` stores the resharding proof, and the `post_state_root` stores the proposed state root. This results in **two** state transitions corresponding to the same block hash. On the chunk producer side, the first transition is stored for the `(block_hash, parent_shard_uid)` pair, and the second one is stored for the `(block_hash, child_shard_uid)` pair. The chunk validator, having all the blocks, identifies whether the implicit transition corresponds to applying a missing chunk or resharding independently. This is implemented in `get_state_witness_block_range`, which iterates from `state_witness.chunk_header.prev_block_hash()` to the block that includes the last chunk for the (parent) shard, if it exists. Then, in `validate_chunk_state_witness`, if the implicit transition corresponds to resharding, the chunk validator calls `retain_split_shard` and proves the state transition from the parent to the child shard. ### State Sync Changes to the state sync protocol are not typically considered protocol changes requiring a version bump, as they concern downloading state that is not present locally rather than the rules for executing blocks and chunks. However, it is helpful to outline some planned changes to state sync related to resharding. When nodes sync state (either because they have fallen far behind the chain or because they will become a chunk producer for a new shard in a future epoch), they first identify a point in the chain to sync to. They then download the tries corresponding to that point in the chain and apply all chunks from that point until they are caught up. Currently, the tries downloaded initially correspond to the `prev_state_root` field of the last new chunk before the first block of the current epoch. This means the state downloaded is from some point in the previous epoch. The proposed change is to move the initial state download point to one in the current epoch rather than the previous one. This keeps shard IDs consistent throughout the state sync logic, simplifies the resharding implementation, and reduces the size of the state to be downloaded. Suppose the previous epoch's shard `S` was split into shards `S'` and `S''` in the current epoch, and a chunk producer that was not tracking shard `S` or any of its children in the current epoch will become a chunk producer for `S'` in the next epoch. With the old state sync algorithm, that chunk producer would download the pre-split state for shard `S`. Then, when it is done, it would need to perform the resharding that all other nodes had already done. While this is not a correctness issue, it simplifies the implementation if we instead download only the state for shard `S'`, allowing the node to download only the state belonging to `S'`, which is much smaller. ### Cross-Shard Traffic When the shard layout changes, it is crucial to handle cross-shard traffic correctly, especially in the presence of missing chunks. Care must be taken to ensure that no receipt is lost or duplicated. There are two important receipt types that need to be considered: outgoing receipts and incoming receipts. *Note: This proposal reuses the approach taken by Resharding V2.* #### Outgoing Receipts Each new chunk in a shard contains a list of outgoing receipts generated during the processing of the previous chunk in that shard. In cases where chunks are missing at the resharding boundary, both child shards could theoretically include the outgoing receipts from their shared ancestor chunk. However, this naive approach would lead to the duplication of receipts, which must be avoided. The proposed solution is to reassign the outgoing receipts from the parent chunk to only one of the child shards. Specifically, the child shard with the lower shard ID will claim all outgoing receipts from the parent, while the other child will receive none. This ensures that all receipts are processed exactly once. #### Incoming Receipts To process a chunk in a shard, it is necessary to gather all outgoing receipts from other shards that are targeted at this shard. These receipts must then be included as incoming receipts. In the presence of missing chunks, the new chunk must collect receipts from all previous blocks, spanning the period since the last new chunk in this shard. This range may cross the resharding boundary. When this occurs, the chunk must also consider receipts that were previously targeted at its parent shard. However, it must filter these receipts to include only those where the recipient lies within the current shard, discarding those where the recipient belongs to the sibling shard in the new shard layout. This filtering process ensures that every receipt is processed exactly once and in the correct shard. ### Delayed Receipt Handling The delayed receipts queue contains all incoming receipts that could not be executed as part of a block due to resource constraints like compute cost, gas limits, etc. The entries in the delayed receipt queue can belong to any of the accounts within the shard. During a resharding event, we ideally need to split the delayed receipts across both child shards according to the associated `account_id` with the receipt. The singleton trie key `DelayedReceiptIndices` holds the `start_index` and `end_index` associated with the delayed receipt entries for the shard. The trie key `DelayedReceipt { index }` contains the actual delayed receipt associated with some `account_id`. These are processed in a FIFO queue order during chunk execution. Note that the delayed receipt trie keys do not have the `account_id` prefix. In Resharding V2, we followed the trivial solution of iterating through all the delayed receipt queue entries and assigning them to the appropriate child shard. However, due to constraints on the state witness size limits and instant resharding, this approach is no longer feasible for Resharding V3. For Resharding V3, we decided to handle the resharding by duplicating the entries of the delayed receipt queue across both child shards. This is beneficial from the perspective of state witness size and instant resharding, as we only need to access the delayed receipt queue root entry in the trie. However, it breaks the assumption that all delayed receipts in a shard belong to the accounts within that shard. To resolve this, with the new protocol version, we changed the implementation of the runtime to discard executing delayed receipts that don't belong to the `account_id` on that shard. Note that no delayed receipts are lost during resharding, as all receipts get executed exactly once based on which of the child shards the associated `account_id` belongs to. ### PromiseYield Receipt Handling Promise Yield was introduced as part of NEP-519 to enable deferring replies to the caller function while the response is being prepared. As part of the Promise Yield implementation, it introduced three new trie keys: `PromiseYieldIndices`, `PromiseYieldTimeout`, and `PromiseYieldReceipt`. * `PromiseYieldIndices`: This is a singleton key that holds the `start_index` and `end_index` of the keys in `PromiseYieldTimeout`. * `PromiseYieldTimeout { index }`: Along with the `receiver_id` and `data_id`, this stores the `expires_at` block height until which we need to wait to receive a response. * `PromiseYieldReceipt { receiver_id, data_id }`: This is the receipt created by the account. An account can call the `promise_yield_create` host function that increments the `PromiseYieldIndices` along with adding a new entry into the `PromiseYieldTimeout` and `PromiseYieldReceipt`. The `PromiseYieldTimeout` is sorted by time of creation and has an increasing value of `expires_at` block height. In the runtime, we iterate over all the expired receipts and create a blank receipt to resolve the entry in `PromiseYieldReceipt`. The account can call the `promise_yield_resume` host function multiple times, and if this is called before the expiry period, we use this to resolve the promise yield receipt. Note that the implementation allows for multiple resolution receipts to be created, including the expiry receipt, but only the first one is used for the actual resolution of the promise yield receipt. We use this implementation quirk to facilitate the resharding implementation. The resharding strategy for the three trie keys is: * **Duplicate Across Both Child Shards**: * `PromiseYieldIndices` * `PromiseYieldTimeout { index }` * **Split Based on Prefix**: * `PromiseYieldReceipt { receiver_id, data_id }`: Since this key has the `account_id` prefix, we can split the entries across both child shards based on the prefix. After duplication of the `PromiseYieldIndices` and `PromiseYieldTimeout`, when the entries of `PromiseYieldTimeout` eventually get dequeued at the expiry height, the following happens: * If the promise yield receipt associated with the dequeued entry **is not** part of the child trie, we create a timeout resolution receipt, and it gets ignored. * If the promise yield receipt associated with the dequeued entry **is** part of the child trie, the promise yield implementation continues to work as expected. This means we don't have to make any special changes in the runtime for handling the resharding of promise yield receipts. ### Buffered Receipt Handling Buffered Receipts were introduced as part of NEP-539 for cross-shard congestion control. As part of the implementation, it introduced two new trie keys: `BufferedReceiptIndices` and `BufferedReceipt`. * `BufferedReceiptIndices`: This is a singleton key that holds the `start_index` and `end_index` of the keys in `BufferedReceipt` for each `shard_id`. * `BufferedReceipt { receiving_shard, index }`: This holds the actual buffered receipt that needs to be sent to the `receiving_shard`. Note that the targets of the buffered receipts belong to external shards, and during a resharding event, we would need to handle both the set of buffered receipts in the parent shard and the set of buffered receipts in other shards that target the parent shard. #### Handling Buffered Receipts in Parent Shard Since buffered receipts target external shards, it is acceptable to assign buffered receipts to either of the child shards. For simplicity, we assign all the buffered receipts to the child shard with the lower index, i.e., copy `BufferedReceiptIndices` and `BufferedReceipt` to the child shard with the lower index while keeping `BufferedReceiptIndices` empty for the child shard with the higher index. #### Handling Buffered Receipts that Target Parent Shard This scenario is slightly more complex. At the boundary of resharding, we may have buffered receipts created before the resharding event targeting the parent shard. At the same time, we may also have buffered receipts generated after the resharding event that directly target the child shard. The receipts from both the parent and child buffered receipts queue need to be appropriately sent to the child shard depending on the `account_id`, while respecting the outgoing limits calculated by the bandwidth scheduler and congestion control. The flow of handling buffered receipts before Resharding V3 is as follows: 1. Calculate `outgoing_limit` for each shard. 2. For each shard, try to forward as many in-order receipts as possible from the buffer while respecting `outgoing_limit`. 3. Apply chunk and `try_forward` newly generated receipts. The newly generated receipts are forwarded if we have enough limit; otherwise, they are put in the buffered queue. The solution for Resharding V3 is to first try draining the parent queue before moving on to draining the child queue. The modified flow would look like this: 1. Calculate `outgoing_limit` for both child shards using congestion info from the parent. 2. Forwarding receipts: * First, try to forward as many in-order receipts as possible from the parent shard buffer. Stop either when we drain the parent buffer or when we exhaust the `outgoing_limit` of either of the child shards. * Next, try to forward as many in-order receipts as possible from the child shard buffer. 3. Applying chunk and `try_forward` newly generated receipts remains the same. The minor downside to this approach is that we don't have guarantees between the order of receipt generation and the order of receipt forwarding, but that's already the case today with buffered receipts. ### Congestion Control Along with having buffered receipts, each chunk also publishes a `CongestionInfo` to the chunk header that contains information about the congestion of the shard during block processing. ```rust pub struct CongestionInfoV1 { /// Sum of gas in currently delayed receipts. pub delayed_receipts_gas: u128, /// Sum of gas in currently buffered receipts. pub buffered_receipts_gas: u128, /// Size of borsh serialized receipts stored in state because they /// were delayed, buffered, postponed, or yielded. pub receipt_bytes: u64, /// If fully congested, only this shard can forward receipts. pub allowed_shard: u16, } ``` After a resharding event, it is essential to properly initialize the congestion info for the child shards. Here is how each field is handled: #### `delayed_receipts_gas` Since the resharding strategy for delayed receipts is to duplicate them across both child shards, we simply copy the value of `delayed_receipts_gas` to both shards. #### `buffered_receipts_gas` Given that the strategy for buffered receipts is to assign all buffered receipts to the lower index child, we copy the `buffered_receipts_gas` from the parent to the lower index child and set `buffered_receipts_gas` to zero for the higher index child. #### `receipt_bytes` This field is more complex as it includes information from both delayed receipts and buffered receipts. To calculate this field accurately, we need to know the distribution of `receipt_bytes` across both delayed receipts and buffered receipts. The current solution is to store metadata about the total `receipt_bytes` for buffered receipts in the trie. This way, we have the following: * For the child with the lower index, `receipt_bytes` is the sum of both delayed receipts bytes and buffered receipts bytes, hence `receipt_bytes = parent.receipt_bytes`. * For the child with the higher index, `receipt_bytes` is just the bytes from delayed receipts, hence `receipt_bytes = parent.receipt_bytes - parent.buffered_receipt_bytes`. #### `allowed_shard` This field is calculated using a round-robin mechanism, which can be independently determined for both child shards. Since we are changing the [ShardId semantics](#shardid-semantics), we need to update the implementation to use `ShardIndex` instead of `ShardID`, which is simply an assignment for each `shard_id` to the contiguous index `[0, num_shards)`. ### ShardId Semantics Currently, shard IDs are represented as numbers within the range `[0, n)`, where `n` is the total number of shards. These shard IDs are sorted in the same order as the account ID ranges assigned to them. While this approach is straightforward, it complicates resharding operations, particularly when splitting a shard in the middle of the range. Such a split requires reindexing all subsequent shards with higher IDs, adding complexity to the process. In this NEP, we propose updating the ShardId semantics to allow for arbitrary identifiers. Although ShardIds will remain integers, they will no longer be restricted to the `[0, n)` range, and they may appear in any order. The only requirement is that each ShardId must be unique. In practice, during resharding, the ID of a parent shard will be removed from the ShardLayout, and the new child shards will be assigned unique IDs - `max(shard_ids) + 1` and `max(shard_ids) + 2`. ## Reference Implementation ### Overview 1. Any node tracking a shard must determine if it should split the shard in the last block before the epoch where resharding should happen. ```pseudocode should_split_shard(block, shard_id): shard_layout = epoch_manager.shard_layout(block.epoch_id()) next_shard_layout = epoch_manager.shard_layout(block.next_epoch_id()) if epoch_manager.is_next_block_epoch_start(block) && shard_layout != next_shard_layout && next_shard_layout.shard_split_map().contains(shard_id): return Some(next_shard_layout.split_shard_event(shard_id)) return None ``` 2. This logic is triggered during block post-processing, which means that the block is valid and is being persisted to disk. ```pseudocode on chain.postprocess_block(block): next_shard_layout = epoch_manager.shard_layout(block.next_epoch_id()) if let Some(split_shard_event) = should_split_shard(block, shard_id): resharding_manager.split_shard(split_shard_event) ``` 3. The event triggers changes in all state storage components. ```pseudocode on resharding_manager.split_shard(split_shard_event, next_shard_layout): set State mapping start FlatState resharding process MemTrie resharding: freeze MemTrie, create HybridMemTries for each child shard: mem_tries[parent_shard].retain_split_shard(boundary_account) ``` 4. `retain_split_shard` leaves only keys in the trie that belong to the left or right child shard. It retains trie key intervals for the left or right child as described above. Simultaneously, the proof is generated. In the end, we get a new state root, hybrid MemTrie corresponding to the child shard, and the proof. Proof is saved as state transition for pair `(block, new_shard_uid)`. 5. The proof is sent as one of the implicit transitions in `ChunkStateWitness`. 6. On the chunk validation path, the chunk validator determines if resharding is part of the state transition using the same `should_split_shard` condition. 7. It calls `Trie(state_transition_proof).retain_split_shard(boundary_account)`, which should succeed if the proof is sufficient and generates a new state root. 8. Finally, it checks that the new state root matches the state root proposed in `ChunkStateWitness`. If the whole `ChunkStateWitness` is valid, then the chunk validator sends an endorsement, which also endorses the resharding. ### State Storage - MemTrie The current implementation of MemTrie uses a memory pool (`STArena`) to allocate and deallocate nodes, with internal pointers in this pool referencing child nodes. Unlike the State representation of the Trie, MemTries do not work with node hashes but with internal memory pointers directly. Additionally, MemTries are not thread-safe, and one MemTrie exists per shard. As described in the [MemTrie](#state-storage---memtrie) section above, we need an efficient way to split the MemTrie into two child MemTries within the span of one block. The challenge lies in the current implementation of MemTrie, which is not thread-safe and cannot be shared across two shards. A naive approach to creating two MemTries for the child shards would involve iterating through all entries of the parent MemTrie and populating these values into the child MemTries. However, this method is prohibitively time-consuming. The solution to this problem is to introduce the concept of a Frozen MemTrie (with a `FrozenArena`), which is a cloneable, read-only, thread-safe snapshot of a MemTrie. By calling the `freeze` method on an existing MemTrie, we convert it into a Frozen MemTrie. This process consumes the original MemTrie, making it no longer available for node allocation and deallocation. Along with `FrozenArena`, we also introduce a `HybridArena`, which effectively combines a base `FrozenArena` with a top layer of `STArena` that supports allocating and deallocating new nodes into the MemTrie. Newly allocated nodes can reference nodes in the `FrozenArena`. This Hybrid MemTrie serves as a temporary MemTrie while the flat storage is being constructed in the background. While Frozen MemTries facilitate instant resharding, they come at the cost of memory consumption. Once a MemTrie is frozen, it continues to consume the same amount of memory as it did at the time of freezing, as it does not support memory deallocation. If a node tracks only one of the child shards, a Frozen MemTrie would continue to use the same amount of memory as the parent trie. Therefore, Hybrid MemTries are only a temporary solution, and we rebuild the MemTrie for the children after resharding is completed. Additionally, a node would need to support twice the memory footprint of a single trie. After resharding, there would be two copies of the trie in memory: one from the temporary Hybrid MemTrie used for block production and another from the background MemTrie under construction. Once the background MemTrie is fully constructed and caught up with the latest block, we perform an in-place swap of the Hybrid MemTrie with the new child MemTrie and deallocate the memory from the Hybrid MemTrie. During a resharding event at the epoch boundary, when we need to split the parent shard into two child shards, we follow these steps: 1. **Freeze the Parent MemTrie**: Create a read-only frozen arena representing a snapshot of the state at the time of freezing (after post-processing the last block of the epoch). The parent MemTrie is no longer required in runtime going forward. 2. **Clone the Frozen MemTrie**: Clone the Frozen MemTrie cheaply for both child MemTries to use. This does not clone the parent arena's memory but merely increases the reference count. 3. **Create Hybrid MemTries for Each Child**: Create a new MemTrie with `HybridArena` for each child. The base of the MemTrie is the read-only `FrozenArena`, while all new node allocations occur in a dedicated `STArena` memory pool for each child MemTrie. This temporary MemTrie is used while Flat Storage is being built in the background. 4. **Rebuild MemTrie**: Once resharding is completed, we use it to load a new MemTrie and catch up to the latest block. 5. **Swap and Clean Up**: After the new child MemTrie has caught up to the latest block, we perform an in-place swap in the client and discard the Hybrid MemTrie. ![Hybrid MemTrie diagram](assets/nep-0568/NEP-HybridMemTrie.png) ### State Storage - Flat State Resharding the Flat State is a time-consuming operation that runs in parallel with block processing for several block heights. Therefore, several important aspects must be considered during implementation: * **Flat State's Status Persistence**: Flat State's status should be resilient to application crashes. * **Correct Block Height**: The parent shard's Flat State should be split at the correct block height. * **Convergence with Mem Trie**: New shards' Flat States should eventually converge to the same representation the chain uses to process blocks (MemTries). * **Chain Forks Handling**: Resharding should work correctly in the presence of chain forks. * **Retired Shards Cleanup**: Retired shards should be cleaned up. Note that the Flat States of the newly created shards will not be available until resharding is completed. This is acceptable because the temporary MemTries are built instantly and can satisfy all block processing needs. The main component responsible for carrying out resharding on Flat State is the [FlatStorageResharder](https://github.com/near/nearcore/blob/f4e9dd5d6e07089dfc789221ded8ec83bfe5f6e8/chain/chain/src/flat_storage_resharder.rs#L68). #### Flat State's Status Persistence Every shard's Flat State has a status associated with it and stored in the database, called `FlatStorageStatus`. We propose extending the existing object by adding a new enum variant named `FlatStorageStatus::Resharding`. This approach has two benefits. First, the progress of any Flat State resharding is persisted to disk, making the operation resilient to a node crash or restart. Second, resuming resharding on node restart shares the same code path as Flat State creation (see `FlatStorageShardCreator`), reducing code duplication. `FlatStorageStatus` is updated at every committable step of resharding. The commit points are as follows: * Beginning of resharding, at the last block of the old shard layout. * Scheduling of the "split parent shard" task. * Execution, cancellation, or failure of the "split parent shard" task. * Execution or failure of any "child catchup" task. #### Splitting a Shard's Flat State When the shard layout changes at the end of an epoch, we identify a **resharding block** corresponding to the last block of the current epoch. A task to split the parent shard's Flat State is scheduled to occur after the resharding block becomes final. The finality condition is necessary to avoid splitting on a block that might be excluded from the canonical chain, which would lock the node into an erroneous state. Inside the split task, we iterate over the Flat State and copy each element into either child. This routine is performed in batches to lessen the performance impact on the node. Finally, if the split completes successfully, the parent shard's Flat State is removed from the database, and the child shards' Flat States enter a catch-up phase. One current technical limitation is that, upon a node crash or restart, the "split parent shard" task will start copying all elements again from the beginning. A reference implementation of splitting a Flat State can be found in [FlatStorageResharder::split_shard_task](https://github.com/near/nearcore/blob/fecce019f0355cf89b63b066ca206a3cdbbdffff/chain/chain/src/flat_storage_resharder.rs#L295). #### Assigning Values from Parent to Child Shards Key-value pairs in the parent shard's Flat State are inherited by children according to the following rules: **Elements inherited by the child shard tracking the `account_id` contained in the key:** * `ACCOUNT` * `CONTRACT_DATA` * `CONTRACT_CODE` * `ACCESS_KEY` * `RECEIVED_DATA` * `POSTPONED_RECEIPT_ID` * `PENDING_DATA_COUNT` * `POSTPONED_RECEIPT` * `PROMISE_YIELD_RECEIPT` **Elements inherited by both children:** * `DELAYED_RECEIPT_OR_INDICES` * `PROMISE_YIELD_INDICES` * `PROMISE_YIELD_TIMEOUT` * `BANDWIDTH_SCHEDULER_STATE` **Elements inherited only by the lowest index child:** * `BUFFERED_RECEIPT_INDICES` * `BUFFERED_RECEIPT` #### Bringing Child Shards Up to Date with the Chain's Head Child shards' Flat States build a complete view of their content at the height of the resharding block sometime during the new epoch after resharding. At that point, many new blocks have already been processed, and these will most likely contain updates for the new shards. A catch-up step is necessary to apply all Flat State deltas accumulated until now. This phase of resharding does not require extra steps to handle chain forks. The catch-up task does not start until the parent shard splitting is done, ensuring the resharding block is final. Additionally, Flat State deltas can handle forks automatically. The catch-up task commits batches of Flat State deltas to the database. If the application crashes or restarts, the task will resume from where it left off. Once all Flat State deltas are applied, the child shard's status is changed to `Ready`, and cleanup of Flat State delta leftovers is performed. A reference implementation of the catch-up task can be found in [FlatStorageResharder::shard_catchup_task](https://github.com/near/nearcore/blob/fecce019f0355cf89b63b066ca206a3cdbbdffff/chain/chain/src/flat_storage_resharder.rs#L564). #### Failure of Flat State Resharding In the current proposal, any failure during Flat State resharding is considered non-recoverable. `neard` will attempt resharding again on restart, but no automatic recovery is implemented. ### State Storage - State Mapping To enable efficient shard state management during resharding, Resharding V3 uses the `DBCol::ShardUIdMapping` column. This mapping allows child shards to reference ancestor shard data, avoiding the need for immediate duplication of state entries. #### Mapping Application in Adapters The core of the mapping logic is applied in `TrieStoreAdapter` and `TrieStoreUpdateAdapter`, which act as layers over the general `Store` interface. Here’s a breakdown of the key functions involved: * **Key Resolution**: The `get_key_from_shard_uid_and_hash` function is central to determining the correct `ShardUId` for state access. At a high level, operations use the child shard's `ShardUId`, but within this function, the `DBCol::ShardUIdMapping` column is checked to determine if an ancestor `ShardUId` should be used instead. ```rust fn get_key_from_shard_uid_and_hash( store: &Store, shard_uid: ShardUId, hash: &CryptoHash, ) -> [u8; 40] { let mapped_shard_uid = store .get_ser::(DBCol::StateShardUIdMapping, &shard_uid.to_bytes()) .expect("get_key_from_shard_uid_and_hash() failed") .unwrap_or(shard_uid); let mut key = [0; 40]; key[0..8].copy_from_slice(&mapped_shard_uid.to_bytes()); key[8..].copy_from_slice(hash.as_ref()); key } ``` This function first attempts to retrieve a mapped ancestor `ShardUId` from `DBCol::ShardUIdMapping`. If no mapping exists, it defaults to the provided child `ShardUId`. This resolved `ShardUId` is then combined with the `node_hash` to form the final key used in `State` column operations. * **State Access Operations**: The `TrieStoreAdapter` and `TrieStoreUpdateAdapter` use `get_key_from_shard_uid_and_hash` to correctly resolve the key for both reads and writes. Example methods include: ```rust // In TrieStoreAdapter pub fn get(&self, shard_uid: ShardUId, hash: &CryptoHash) -> Result, StorageError> { let key = get_key_from_shard_uid_and_hash(self.store, shard_uid, hash); self.store.get(DBCol::State, &key) } // In TrieStoreUpdateAdapter pub fn increment_refcount_by( &mut self, shard_uid: ShardUId, hash: &CryptoHash, data: &[u8], increment: NonZero, ) { let key = get_key_from_shard_uid_and_hash(self.store, shard_uid, hash); self.store_update.increment_refcount_by(DBCol::State, key.as_ref(), data, increment); } ``` The `get` function retrieves data using the resolved `ShardUId` and key, while `increment_refcount_by` manages reference counts, ensuring correct tracking even when accessing data under an ancestor shard. #### Mapping Retention and Cleanup Mappings in `DBCol::ShardUIdMapping` persist as long as any descendant relies on an ancestor’s data. To manage this, the `set_shard_uid_mapping` function in `TrieStoreUpdateAdapter` adds a new mapping during resharding: ```rust fn set_shard_uid_mapping(&mut self, child_shard_uid: ShardUId, parent_shard_uid: ShardUId) { let mapped_parent_shard_uid = store .get_ser::(DBCol::StateShardUIdMapping, &parent_shard_uid.to_bytes()) .expect("set_shard_uid_mapping() failed") .unwrap_or(parent_shard_uid); self.store_update.set( DBCol::StateShardUIdMapping, child_shard_uid.to_bytes().as_ref(), &borsh::to_vec(&mapped_parent_shard_uid).expect("Borsh serialize cannot fail"), ) } ``` When a node stops tracking all descendants of a shard, garbage collection will eventually clear the last block of the last epoch in which the last descendant was tracked. The descendant will then appear in the result of: ```rust fn get_potential_shards_for_cleanup(..., last_block_of_gced_epoch) -> Result> { let mut tracked_shards = vec![]; for shard_uid in shard_layout.shard_uids() { if chain_store_update .store() .exists(DBCol::TrieChanges, &get_block_shard_uid(&last_block_of_gced_epoch, &shard_uid))? { tracked_shards.push(shard_uid); } } tracked_shards } ``` Then, `gc_state()` is called, mapping the descendant `ShardUId` to the parent `ShardUId`, making the parent shard a candidate for cleanup. We then detect that since `gced_epoch`, the parent `ShardUId` has not been used as a database key prefix. As a result, we can safely remove the state under this prefix (including the parent and all descendants) along with the associated entries from `DBCol::StateShardUIdMapping`. ```rust fn gc_state(potential_shards_for_cleanup, gced_epoch, shard_tracker, store_update) { let mut potential_shards_to_cleanup: HashSet = potential_shards_for_cleanup .iter() .map(|shard_uid| get_shard_uid_mapping(&store, *shard_uid)) .collect(); for epoch in gced_epoch+1..current_epoch { let shard_layout = get_shard_layout(epoch); let last_block_of_epoch = get_last_block_of_epoch(epoch); for shard_uid in shard_layout.shard_uids() { if !store .exists(DBCol::TrieChanges, &get_block_shard_uid(last_block_of_epoch, &shard_uid))? { continue; } let mapped_shard_uid = get_shard_uid_mapping(&store, shard_uid); potential_shards_to_cleanup.remove(&mapped_shard_uid); } } for shard_uid in shard_tracker.get_shards_tracks_this_or_next_epoch() { let mapped_shard_uid = get_shard_uid_mapping(&store, shard_uid); potential_shards_to_cleanup.remove(&mapped_shard_uid); } let shards_to_cleanup = potential_shards_to_cleanup; for kv in store.iter_ser::(DBCol::StateShardUIdMapping) { let (child_shard_uid, parent_shard_uid) = kv?; if shards_to_cleanup.contains(&parent_shard_uid) { store_update.delete(DBCol::StateShardUIdMapping, &child_shard_uid); } } for shard_uid_prefix in shards_to_cleanup { store_update.delete_shard_uid_prefixed_state(shard_uid_prefix); } } ``` For archival nodes, mappings are retained permanently to ensure access to the historical state of all shards. ### State Sync The state sync algorithm defines a `sync_hash` used in many parts of the implementation. This is always the first block of the current epoch, which the node should be aware of once it has synced headers to the current point in the chain. A node performing state sync first makes a request for a `ShardStateSyncResponseHeader` corresponding to that `sync_hash` and the Shard ID of the shard it's interested in. Among other things, this header includes the last new chunk before `sync_hash` in the shard and a `StateRootNode` with a hash equal to that chunk's `prev_state_root` field. Then the node downloads the nodes of the trie with that `StateRootNode` as its root. Afterwards, it applies new chunks in the shard until it's caught up. As described above, the state we download is the state in the shard after applying the second-to-last new chunk before `sync_hash`, which belongs to the previous epoch (since `sync_hash` is the first block of the new epoch). To move the point in the chain of the initial state download to the current epoch, we could either move the `sync_hash` forward or change the state sync protocol (perhaps changing the meaning of the `sync_hash` and the fields of the `ShardStateSyncResponseHeader`, or somehow changing these structures more significantly). The former is an easier first implementation, as it would not require any changes to the state sync protocol other than to the expected `sync_hash`. We would just need to move the `sync_hash` to a point far enough along in the chain so that the `StateRootNode` in the `ShardStateSyncResponseHeader` refers to the state in the current epoch. Currently, we plan on implementing it that way, but we may revisit making more extensive changes to the state sync protocol later. ## Security Implications ### Fork Handling In theory, it's possible that more than one candidate block finishes the last epoch with the old shard layout. For previous implementations, it didn't matter because the resharding decision was made at the beginning of the previous epoch. Now, the decision is made at the epoch boundary, so the new implementation handles this case as well. ### Proof Validation With single shard tracking, nodes can't independently validate new state roots after resharding because they don't have the state of the shard being split. That's why we generate resharding proofs, whose generation and validation may be a new weak point. However, `retain_split_shard` is equivalent to a constant number of lookups in the trie, so its overhead is negligible. Even if the proof is invalid, it will only imply that `retain_split_shard` fails early, similar to other state transitions. ## Alternatives In the solution space that would keep the blockchain stateful, we also considered an alternative to handle resharding through the mechanism of `Receipts`. The workflow would be to: * Create an empty `target_shard`. * Require `source_shard` chunk producers to create special `ReshardingReceipt(source_shard, target_shard, data)` where `data` would be an interval of key-value pairs in `source_shard` along with the proof. * Then, `target_shard` trackers and validators would process that receipt, validate the proof, and insert the key-value pairs into the new shard. However, `data` would occupy most of the state witness capacity and introduce the overhead of proving every single interval in `source_shard`. Moreover, the approach to sync the target shard "dynamically" also requires some form of catch-up, which makes it much less feasible than the chosen approach. Another question is whether we should tie resharding to epoch boundaries. This would allow us to move from the resharding decision to completion much faster. But for that, we would need to: * Agree if we should reshard in the middle of the epoch or allow "fast epoch completion," which has to be implemented. * Keep chunk producers tracking "spare shards" ready to receive items from split shards. * On resharding event, implement a specific form of state sync, on which source and target chunk producers would agree on new state roots offline. * Then, new state roots would be validated by chunk validators in the same fashion. While it is much closer to Dynamic Resharding (below), it requires many more changes to the protocol. The considered idea works well as an intermediate step toward that, if needed. ## Future Possibilities * **Dynamic Resharding**: In this proposal, resharding is scheduled in advance and hardcoded within the `neard` binary. In the future, we aim to enable the chain to dynamically trigger and execute resharding autonomously, allowing it to adjust capacity automatically based on demand. * **Fast Dynamic Resharding**: In the Dynamic Resharding extension, the new shard layout is configured for the second upcoming epoch. This means that a full epoch must pass before the chain transitions to the updated shard layout. In the future, our goal is to accelerate this process by finalizing the previous epoch more quickly, allowing the chain to adopt the new layout as soon as possible. * **Shard Merging**: In this proposal, the only allowed resharding operation is shard splitting. In the future, we aim to enable shard merging, allowing underutilized shards to be combined with neighboring shards. This would allow the chain to free up resources and reallocate them where they are most needed. ## Consequences ### Positive * The protocol can execute resharding even while only a fraction of nodes track the split shard. * State for new shard layouts is computed in a matter of minutes instead of hours, greatly increasing ecosystem stability during resharding. As before, from the point of view of NEAR users, it is instantaneous. ### Neutral N/A ### Negative * The storage components need to handle the additional complexity of controlling the shard layout change. ### Backwards Compatibility Resharding is backwards compatible with existing protocol logic. ## Unresolved Issues (Optional) ```text [Explain any issues that warrant further discussion. Considerations * What parts of the design do you expect to resolve through the NEP process before this gets merged? * What parts of the design do you expect to resolve through the implementation of this feature before stabilization? * What related issues do you consider out of scope for this NEP that could be addressed in the future independently of the solution that comes out of this NEP?] ``` ## Changelog ```text [The changelog section provides historical context for how the NEP developed over time. Initial NEP submission should start with version 1.0.0, and all subsequent NEP extensions must follow [Semantic Versioning](https://semver.org/). Every version should have the benefits and concerns raised during the review. The author does not need to fill out this section for the initial draft. Instead, the assigned reviewers (Subject Matter Experts) should create the first version during the first technical review. After the final public call, the author should then finalize the last version of the decision context.] ``` ### 1.0.0 - Initial Version > Placeholder for the context about when and who approved this NEP version. #### Benefits > List of benefits filled by the Subject Matter Experts while reviewing this version: * Benefit 1 * Benefit 2 #### Concerns > Template for Subject Matter Experts review for this version: > Status: New | Ongoing | Resolved | # | Concern | Resolution | Status | | ---: | :------ | :--------- | -----: | | 1 | | | | | 2 | | | | ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). [NEP-040]: https://github.com/near/NEPs/blob/master/specs/Proposals/0040-split-states.md [NEP-508]: https://github.com/near/NEPs/blob/master/neps/nep-0508.md [NEP-509]: https://github.com/near/NEPs/blob/master/neps/nep-0509.md ================================================ FILE: neps/nep-0584.md ================================================ --- NEP: 584 Title: Cross-shard bandwidth scheduler Authors: Jan Malinowski Status: Final DiscussionsTo: https://github.com/near/NEPs/pull/584 Type: Protocol Version: 1.0.0 Created: 2025-01-13 LastUpdated: 2025-01-13 --- ## Summary Bandwidth scheduler decides how many bytes of receipts a shard is allowed to send to different shards at every block height. Chunk application produces outgoing receipts that will be sent to other shards. Bandwidth scheduler looks at how many receipts every shard wants to send to other shards and decides how much can be sent between each pair of shards. It makes sure that every shard receives and sends a reasonable amount of data at every height. Sending or receiving too much data could cause performance problems and witness size issues. We have an existing mechanism to limit how much is sent between shards, but it's very rudimentary and inefficient. Bandwidth scheduler is a better solution to the problem. ## Motivation ### Why do we need cross-shard bandwidth limits? NEAR is a sharded blockchain - every shard is expected to do a limited amount of work at every height. Scaling is mostly achieved by adding more shards. This also means that we cannot expect a shard to send or receive more than X MB of data at every height. Without cross-shard bandwidth limits there could be a situation where this isn't respected - a shard could be forced to send or receive a ton of data at a single height. There could be a situation where all of the shards decide to send a ton of data to a single receiver shard, or a situation where one sender shard generates a lot of outgoing receipts to other shards. This problem gets worse as the number of shards increases, with 6 shards it isn't that bad, but 50 shards sending receipts to a single shard would definitely overwhelm the receiver. There are two kinds of problems that can happen when too much data is being sent: - Nodes might not be able to transfer all of the receipts in time and chunk producers might not have the data needed to produce a chunk, causing chunk misses. - With stateless validation all of the incoming receipts are kept inside `ChunkStateWitness`. The protocol is very sensitive to the size of `ChunkStateWitness` - when `ChunkStateWitness` becomes too large, the nodes are not able to distribute it in time and there are chunk misses, in extreme cases a shard can even stall. We have to make sure that the size of incoming receipts is limited to avoid witness size issues and attacks. There are plans to make the protocol more resilient to large witness size, but that is still work in progress. We need some kind of cross-shard bandwidth limits to avoid these problems. ### Existing solution There is already a rudimentary solution in place, added together with stateless validation in [NEP-509](https://github.com/near/NEPs/blob/master/neps/nep-0509.md) to limit witness size. In this solution each shard is usually allowed to send 100KiB (`outgoing_receipts_usual_size_limit`) of receipts to another shard, but there's one special shard that is allowed to send 4.5MiB (`outgoing_receipts_big_size_limit`). The special allowed shard is switched on every height in a round robin fashion. If a shards wants to send less than 100KiB it can just do it, but for larger transfers the sender needs to wait until it's the allowed shard to send the receipts. A node can only send more than 100KiB on its turn. See the PR for a more detailed description of the solution: https://github.com/near/nearcore/pull/11492 This solution was simple enough to be implemented before stateless validation launch, but there are issues with this approach: - Small throughput. If we take two shards - `1` and `2`, then `1` is able to send at most 5MiB of data to `2` every 6 blocks (assuming 6 shards). That's only 800KiB / height, even though in theory NEAR could support 5MiB / height (assuming that other shards aren't sending much). That's a lot unused throughput that we can't make use of because of the overly restrictive limits. There are some use cases that could make use of higher throughput, e.g NEAR DA, although to be fair last I heard NEAR DA was moving to a design that doesn't require a lot of cross-shard bandwidth. - High latency and bad scalability. A big receipt has to wait for up to `num_shards` heights before it can be sent. This is much higher than it could be, with bandwidth scheduler a receipt never has to wait more than one height (assuming that other shards aren't sending much). Even worse is that the more shards there are, the higher the latency. With 60 shards a receipt might need to wait for 60 blocks before it is processed. This solution doesn't scale at all. Bandwidth scheduler addresses pain points of the current solution, enabling higher throughput and scalability. ## Specification The main source of wasted bandwidth in the current algorithm is that assigning bandwidth doesn't take into account the needs of individual shards. When shard `1` needs to send 500KiB and shard `2` needs to send 20KiB, the algorithm can assign all of the bandwidth to shard `2` even though it doesn't really need it, it just happened to be the allowed shard at this height. This is wasteful, it would be much better if the algorithm could see how much bandwidth each shard needs and give to each according to their needs. This is the general idea behind the new solution: each shard requests bandwidth according to its needs and bandwidth scheduler divides the bandwidth between everyone that requested it. The bandwidth scheduler would be able to see that shard `2` needs 500KiB of bandwidth and it'd give all the bandwidth to `2`. The flow will look like this: - A chunk is applied and produces outgoing receipts to other shards. - The shard calculates the current limits and sends as many receipts as it's allowed to. - The receipts that can't be sent due to limits are buffered (saved to state), they will be sent at a later height. - The shard calculates how much bandwidth it needs to send the buffered receipts and creates a `BandwidthRequest` with this information (there's one `BandwidthRequest` for each pair of shards). - The list of `BandwidthRequest` from this shard is included in the chunk header and distributed to other nodes. - When the next chunk is applied it gathers all the `BandwidthRequests` from chunk headers at the previous height(s) and uses `BandwidthScheduler` to calculate the current bandwidth limits in a deterministic way. The same calculation is performed on all shards and all shards arrive at the same bandwidth limits. - The chunk is applied and produces outgoing receipts, receipts are sent until they hit the limits set by `BandwidthScheduler`. ![Diagram where bandwidth scheduler requests bandwidth and then sends receipts](assets/nep-0584/basic-flow.png) Details will be explained in the following sections. ### `BandwidthRequest` A `BandwidthRequest` describes the receipts that a shard would like to send to another shard. A shard looks at its queue of buffered receipts to another shard and generates a `BandwidthRequest` which describes how much bandwidth the shard would like to have. In the simplest version a `BandwidthRequest` could be a single integer containing the total size of buffered receipts, but there is a problem with this simple representation - it doesn't say anything about the size of individual receipts. Let's say that two shards want to send 4MB of data each to another shard, but the incoming limit is 5MB. Should we assign 2.5MB of bandwidth to each of the sender shards? That would work if the shards want to send a lot of small receipts, but it wouldn't work when each shard wants to send a single 4MB receipt. A shard can't send a part of the 4MB receipt, it's either the whole receipt or nothing. The scheduler should assign 2.5MB/2.5MB of bandwidth when the receipts are small and 4MB/0MB when they're large. The simple version doesn't have enough information for the scheduler to make the right decision, so we'll use a richer representation. The richer representation is a list of possible bandwidth grants that make sense for this pair of shards. For example when the outgoing buffer contains a single `4MB` receipt, the only bandwidth grant that makes sense is `4MB`. Granting more or less bandwidth wouldn't change how many receipts can be sent. In that case the bandwidth request would contain a single possible grant: `4MB`. It tells the bandwidth scheduler that for this pair of shards it can either grant `4MB` or nothing at all, other options don't really make sense. On the other hand if the outgoing buffer contains 4000 small receipts, 1kB each, there are many possible bandwidth grants that make sense. With so many small receipts the scheduler could grant 1kB, 2kB, 3kB, ..., 4000kB and each of those options would result in a different number of receipts being sent. However having 4000 options in the request would make the request pretty large. To deal with this we'll specify a list of 40 predefined options that can be requested in a bandwidth request. An option is requested when granting this much bandwidth would result in more receipts being sent. Let's take a look at an example. Let's say that the predefined list of values that can be requested is: ```rust [100kB, 200kB, 300kB, ..., 3900kB, 4MB] ``` And the outgoing receipts buffer has receipts with these sizes (receipts will be sent from left to right): ```rust [20kB, 150kB, 60kB, 400kB, 1MB, 50kB, 300kB] ``` The cumulative sum (sum from 0 to i) of sizes is: ```rust [20kB, 170kB, 230kB, 630kB, 1630kB, 1680kB, 1980kB] ``` The bandwidth grant options in the generated `BandwidthRequest` will be: ```rust [100kB, 200kB, 300kB, 700kB, 1700kB, 2000kB] ``` Explanation: - Granting 100kB of bandwidth will allow to send the first receipt. - Granting 200kB of bandwidth will allow to send the first two receipts. - Granting 300kB will allow to send the first three receipts. - Granting 400kB would give the same result as 300kB, so it's not included in the options - Granting 700kB would allow to send the first four receipts - etc etc Conceptually a `BandwidthRequest` looks like this: ```rust struct BandwidthRequest { /// Requesting bandwidth to this shard to_shard: ShardId, /// Please grant me one of the options listed here. possible_bandwidth_grants: Vec } ``` A list of such requests will be included in the chunk header: ```rust struct ChunkHeader { bandwidth_requests: Vec } ``` With this representation the `BandwidthRequest` struct could be quite big, up to about ~320 bytes. We will use a more efficient representation to bring its size down to only 7 bytes. First we could use `u16` instead `u64` for the `ShardId`, NEAR currently has only 6 shards and it'll take a while to reach 65536. There's no need to handle 10**18 shards. Second we can use a bitmap for the `possible_bandwidth_grants` field. The list of predefined options that can be requested will be computed deterministically on all nodes. The bitmap will have 40 bits, the `n-th` bit is `1` when the `n-th` value from the predefined list is requested. So the actual representation of a `BandwidthRequest` looks something like this: ```rust struct BandwidthRequest { to_shard: u16, requested_values_bitmap: [u8; 5] } ``` It's important to keep the size of `BandwidthRequest` small because bandwidth requests are included in the chunk header, and the chunk header shouldn't be too large. ### Base bandwidth In current mainnet traffic most of the time the size of outgoing receipts is small, under 50kB. It'd be nice if a shard was able to send them out without having to make a bandwidth request. It'd lower the latency (no need to wait for a grant) and make chunk headers smaller. That's why there's a concept of `base_bandwidth`. Bandwidth scheduler grants `base_bandwidth` of bandwidth for each pair of shards by default. This means that a shard doesn't need to make a request when it has less than `base_bandwidth` of receipts, it can just send them out immediately. Actual bandwidth grants based on bandwidth request happen after granting the base bandwidth. On current mainnet (with 6 shards) the base bandwidth is 61_139 (61kB) `base_bandwidth` is automatically calculated based on `max_shard_bandwidth`, `max_single_grant` and the number of shards. It gets smaller as the number of shards increases. See the next section for details. ### `BandwidthSchedulerParams` The `BandwidthSchedulerParams` struct keeps parameters used throughout the bandwidth scheduler algorithm: ```rust pub type Bandwidth = u64; /// Parameters used in the bandwidth scheduler algorithm. pub struct BandwidthSchedulerParams { /// This much bandwidth is granted by default. /// base_bandwidth = (max_shard_bandwidth - max_single_grant) / (num_shards - 1) pub base_bandwidth: Bandwidth, /// The maximum amount of data that a shard can send or receive at a single height. pub max_shard_bandwidth: Bandwidth, /// The maximum amount of bandwidth that can be granted on a single link. /// Should be at least as big as `max_receipt_size`. pub max_single_grant: Bandwidth, /// Maximum size of a single receipt. pub max_receipt_size: Bandwidth, /// Maximum bandwidth allowance that a link can accumulate. pub max_allowance: Bandwidth, } ``` The values are: ```rust max_shard_bandwidth = 4_500_000; max_single_grant = 4_194_304 max_allowance = 4_500_000; max_receipt_size = 4_194_304; base_bandwidth = min(100_000, (max_shard_bandwidth - max_single_grant) / (num_shards - 1)) = 61_139 ``` A shard must be able to send out `max_single_grant` on one link and `base_bandwidth` on all other links without exceeding `max_shard_bandwidth`. So it must hold that: ```rust base_bandwidth * (num_shards - 1) + max_single_grant <= max_shard_bandwidth ``` That's why base bandwidth is calculated by taking the bandwidth that would remain available after granting `max_single_grant` on one link and dividing it equally between the other links. There's also a limit which makes sure that `base_bandwidth` stays under 100kB, even when the number of shards is low. There are some tests which have a low number of shards, and having a lower base bandwidth allows us to fully test the bandwidth scheduler in those tests. ### `BandwidthRequestValues` The `BandwidthRequestValues` struct represents the predefined list of values that can be requested in a `BandwidthRequest`: ```rust pub struct BandwidthRequestValues { pub values: [Bandwidth; 40], } ``` The values are calculated using a linear interpolation between `base_bandwidth` and `max_single_grant`, like this: ```rust values[-1] = base_bandwidth // (here -1 is the imaginary element before 0, not the last element) values[values.len() - 1] = max_single_grant values[i] = linear interpolation between values[-1] and values[values.len() - 1] ``` The exact code is: ```rust /// Performs linear interpolation between min and max. /// interpolate(100, 200, 0, 10) = 100 /// interpolate(100, 200, 5, 10) = 150 /// interpolate(100, 200, 10, 10) = 200 fn interpolate(min: u64, max: u64, i: u64, n: u64) -> u64 { min + (max - min) * i / n } let values_len: u64 = values.len().try_into().expect("Converting usize to u64 shouldn't fail"); for i in 0..values.len() { let i_u64: u64 = i.try_into().expect("Converting usize to u64 shouldn't fail"); values[i] = interpolate( params.base_bandwidth, params.max_single_grant, i_u64 + 1, values_len, ); } ``` The final `BandwidthRequestValues` on current mainnet (6 shards) look like this: ```rust [ 164468, 267797, 371126, 474455, 577784, 681113, 784442, 887772, 991101, 1094430, 1197759, 1301088, 1404417, 1507746, 1611075, 1714405, 1817734, 1921063, 2024392, 2127721, 2231050, 2334379, 2437708, 2541038, 2644367, 2747696, 2851025, 2954354, 3057683, 3161012, 3264341, 3367671, 3471000, 3574329, 3677658, 3780987, 3884316, 3987645, 4090974, 4194304 ] ``` ### Generating bandwidth requests To generate a bandwidth request the sender shard has to look at the receipts stored in the outgoing buffer to another shard and pick bandwidth grant options that make sense. In this context "makes sense" means that having this much bandwidth would cause the sender to send more receipts than the previous requested option, as described in the `BandwidthRequest` section. The simplest implementation would be to actually walk through the list of outgoing receipts (starting from the ones that will be sent the soonest) and request values that allow to send more receipts. ```rust /// Generate a bitmap of bandwidth requests based on the size of receipts stored in the outgoing buffer. /// Returns a bitmap with requests. fn make_request_bitmap_slow( buffered_receipt_sizes: Vec, bandwidth_request_values: &BandwidthRequestValues, ) -> BandwidthRequestBitmap { let mut requested_values_bitmap = BandwidthRequestBitmap::new(); // [u8; 5] let mut total_size = 0; let values = &bandwidth_request_values.values; for receipt_size in buffered_receipt_sizes { total_size += receipt_size; for i in 0..values.len() { if values[i] >= total_size { requested_values_bitmap.set_bit(i, true); break; } } } requested_values_bitmap } ``` But this is very inefficient. Walking over all buffered receipts could take a lot of time and it'd require reading a lot of state from the Trie, which would make the `ChunkStateWitness` very large. We need a more efficient algorithm. To achieve this we will add some additional metadata about the outgoing buffer, which keeps coarse information about the receipt sizes. We will group consecutive receipts into `ReceiptGroups`. A single `ReceiptGroup` aims to have total size and gas under some threshold. If adding a new receipt to the group would cause it to exceed the threshold, a new group is started. The threshold can only be exceeded when a single receipt has size or gas above the group threshold. The size threshold is set to 100kB, the gas threshold is currently infinite. ```rust pub struct ReceiptGroupsConfig { /// All receipt groups aim to have a size below this threshold. /// A group can be larger that this if a single receipt has size larger than the limit. /// Set to 100kB pub size_upper_bound: ByteSize, /// All receipt groups aim to have gas below this threshold. /// Set to Gas::MAX pub gas_upper_bound: Gas, } ``` A `ReceiptGroup` keeps only the total size and gas of receipts in this group: ```rust pub struct ReceiptGroupV0 { /// Total size of receipts in this group. /// Should be no larger than `max_receipt_size`, otherwise the bandwidth /// scheduler will not be able to grant the bandwidth needed to send /// the receipts in this group. pub size: u64, /// Total gas of receipts in this group. pub gas: u128, } ``` All the groups are kept inside a `ReceiptGroupsQueue`, which is a Trie queue similar to the delayed receipts queue. `ReceiptGroupsQueue` additionally keeps information about the total size, gas and number of receipts in the queue. There's one `ReceiptGroupsQueue` per outgoing buffer. ```rust pub struct ReceiptGroupsQueue { /// Corresponds to receipts stored in the outgoing buffer to this shard. receiver_shard: ShardId, /// Persistent data, stored in the trie. data: ReceiptGroupsQueueDataV0, } pub struct ReceiptGroupsQueueDataV0 { /// Indices of the receipt groups TrieQueue. pub indices: TrieQueueIndices, /// Total size of all receipts in the queue. pub total_size: u64, /// Total gas of all receipts in the queue. pub total_gas: u128, /// Total number of receipts in the queue. pub total_receipts_num: u64, } ``` When a new receipt is added to the outgoing buffer, we try to add it to the last group in the `ReceiptGroupsQueue`. If there are no groups or adding the receipt would cause the last group to go over the threshold, a new group is created. When a receipt is removed from the outgoing buffer, we remove the receipt from the first group in the `ReceiptGroupsQueue` and remove the group if there are no more receipts in it. To generate a bandwidth request, we will walk over the receipt groups and request the values that will allow to send more receipts. Just like in `make_request_bitmap_slow`, only using `receipt_group_sizes` instead of `buffered_receipt_sizes`. It's important to note that `size_upper_bound` is less than difference between two consecutive values in `BandwidthRequestValues` . Thanks to this the requests are just as good as they would be if they were generated directly using individual receipt sizes. #### Example Let's say that there are five buffered receipts with sizes: ```rust 5kB, 30kB, 40kB, 120kB, 20kB ``` They would be grouped into groups of at most 100kB, like this: ```rust (5kB, 30kB, 40kB), (120kB), (20kB) ``` So the resulting groups would be: ```rust 75kB, 120kB, 20kB ``` And the bandwidth request will produced by walking over groups with sizes `35kB`, `120kB`, `70kB`, not the individual receipts. Now let's say that the first receipt with size `5kB` is forwarded. In that case it would be removed from the first group, and the groups would look like this: ```rust (30kB, 40kB), (120kB), (20kB) ``` When a new receipt is buffered, it's added to the last group, let's add a `50kB` receipt, after that the groups would look like this: ```rust (30kB, 40kB), (120kB), (20kB, 50kB) ``` When adding a new receipt would cause a group to go over the threshold, a new groups is started. So if we added another 50kB receipt, the groups would become: ```rust (30kB, 40kB), (120kB), (20kB, 50kB), (50kB) ``` #### Trie columns Two new trie columns are added to keep the receipt groups. - `BUFFERED_RECEIPT_GROUPS_QUEUE_DATA` - keeps `ReceiptGroupsQueueDataV0` for every outgoing buffer - `BUFFERED_RECEIPT_GROUPS_QUEUE_ITEM` - keeps the individual `ReceiptGroup` items from receipt group queues #### Protocol Upgrade There's a bit of additional complexity around the protocol upgrade boundary. The receipt groups are built for receipts that were buffered after protocol upgrade, but existing receipts that were buffered before the upgrade won't have corresponding receipt groups. Eventually the old buffered receipts will get sent out and we'll have full metadata for all receipts, but in the meantime we won't be able to make a proper bandwidth request without having groups for all of the buffered receipts. To deal with this we will pretend that there's one receipt with size `max_receipt_size` in the buffer until the metadata is fully initialized. Requesting `max_receipt_size` is a safe bet - it's enough to send out any buffered receipt. The effect will be similar to the previous approach - one shard will be granted most of the bandwidth (exactly `max_receipt_size`), while other will be waiting for their turn to be the "allowed shard". Once all of the old buffered receipts are sent out we can start making proper requests using the receipt groups. ### `BandwidthScheduler` `BandwidthScheduler` is an algorithm which looks at all of the `BandwidthRequests` submitted by shards and grants some bandwidth on every link (pair of shards). A shard can send only as much data as the grant allows, the remaining receipts stay in the buffer. Bandwidth scheduler tries to ensure that: - Every shard sends out at most `max_shard_bandwidth` bytes of receipts at every height. - Every shard receives at most `max_shard_bandwidth` bytes of receipts at every height. - The bandwidth is assigned in a fair way. At full load every link (pair of shards) sends and receives the same amount of bandwidth on average, there are no favorites. - Bandwidth utilization is high. The algorithm works in 4 stages: 1) Give out a fair share of allowance to every link. 2) Grant base bandwidth on every link. Decrease allowance by granted bandwidth. 3) Process bandwidth requests. Order all bandwidth requests by the link's allowance. Take the request with the highest allowance and try to grant the first proposed value. Check if it's possible to grant the value without violating any restrictions. If yes, grant the bandwidth and decrease the allowance accordingly. Then remove the granted value from the request and put it back into the queue with new allowance. If no, remove the request from the queue, it will not be fulfilled. Requests with the same allowance are processed in a random order. 4) Distribute remaining bandwidth. If there's some bandwidth left after granting base bandwidth and processing all requests, distribute it over all links in a fair manner to improve bandwidth utilization. #### Allowance There is a concept of "allowance" - every link (pair of sender and receiver shards) has an allowance. Allowance is a way to ensure fairness. Every link receives a fair amount of allowance on every height. When bandwidth is granted on a link, the link's allowance is decreased by the granted amount. Requests on links with higher allowance have priority over requests on links with lower allowance. Links that send more than their fair share are deprioritized, which keeps things fair. It's a similar idea to the [Token Bucket](https://en.wikipedia.org/wiki/Token_bucket). Link allowances are persisted in the state trie, as they're used to track fairness across multiple heights. An intuitive way to think about allowance is that it keeps track of how much each link sent recently and lowers priority of links that recently sent a lot of receipts, which gives other a fair chance. Imagine a situation where one link wants to send a 2MB receipt at every height, and other links want to send a ton of small receipts to the same shard. Without allowance, the link with 2MB receipts would always get 2MB of bandwidth assigned, and other links would get less than that, which would be unfair. Thanks to allowance, the scheduler will grant some bandwidth to the 2MB link, but then it will decrease the allowance on that link, which will deprioritize it and other links will get their fair share. When multiple requests have the same allowance, they are processed in random order. The randomness is deterministic, the scheduler uses `ChaCha20Rng` seeded using the previous block hash and requests with equal allowance are shuffled used this random generator. ```rust ChaCha20Rng::from_seed(prev_block_hash.0) ... requests.shuffle(&mut self.rng); ``` The fair share of allowance that is given out on every height is: ```rust fair_link_bandwidth = max_shard_bandwidth / num_shards ``` The reasoning is that in an ideal, fair world, every link would send the same amount of bandwidth. There would be `max_bandwidth / num_shards` sent on every link, fully saturating all senders and receivers. Allowance measures the deviation from this perfect world. Link allowance never gets larger than `max_allowance` (currently 4.5MB). When a link's allowance reaches `max_allowance` we stop adding allowance there until the link uses up some of the accumulated one. Without `max_allowance` a link that sends very little for a long time could accumulate an enormous amount of allowance and it could have priority over other links for a very long time. Capping the allowance at some value keeps the allowance fresh, information from the latest blocks should be what matters most. #### Example
Here's an example, click to expand. Let's say that there are 3 shards. The `BandwidthSchedulerParams` look like this: ```rust BandwidthSchedulerParams { base_bandwidth: 100_000, max_shard_bandwidth: 4_500_000, max_single_grant: 4_194_304, max_receipt_size: 4_194_304, max_allowance: 4_500_000, } ``` And `BandwidthRequestValues` are: ```rust [ 210000, 320000, 430000, 540000, 650000, 760000, 870000, 980000, 1090000, 1200000, 1310000, 1420000, 1530000, 1640000, 1750000, 1860000, 1970000, 2080000, 2190000, 2300000, 2410000, 2520000, 2630000, 2740000, 2850000, 2960000, 3070000, 3180000, 3290000, 3400000, 3510000, 3620000, 3730000, 3840000, 3950000, 4060000, 4194304 ] ``` (Note that these values are slightly different from the ones that would be generated by linear interpolation, but for the sake of the example let's say that they look like this, the slight difference doesn't really matter and it's easier to work with round numbers) Shard 2 is fully congested and only the allowed shard (shard 1) is allowed to send receipts to it. The outgoing buffers look like this, shards want to send receipts with these sizes: - 0->1 [3.9MB] - 1->1 [200kB, 200kB, 200kB] - 1->2 [2MB] - 2->2 [500kB] Bandwidth requests request values from the predefined list (`BandwidthRequestValues`), in this example the requests would be: - 0->1 [3950kB] - 1->1 [210kB, 430kB, 650kB] - 1->2 [2.08MB] - 2->2 [540kB] Every link has some allowance, in this example let's say that all links start with the same allowance (4MB) | Link | Allowance | | ---- | --------- | | 0->0 | 4MB | | 0->1 | 4MB | | ... | 4MB | All shards start with sender and receiver budgets set to `max_shard_bandwidth` (4.5MB). Budgets describe how much more a shard can send or receive: ![State of links before scheduler runs](assets/nep-0584/scheduler_example_1.png) First step of the algorithm is to give out a fair share of allowance on every link. In an ideally fair world every link would send the same amount of data at every height, so the fair share of allowance is: ```rust fair_link_bandwidth = max_shard_bandwidth / num_shards = 4.5MB/3 = 1.5MB ``` So every link receives `1.5MB` of allowance. But allowance can't get larger than `max_allowance`, which is set to `4.5MB`, so the allowance is set to `4.5MB` on all links: | Link | Allowance | | ---- | --------- | | 0->0 | 4.5MB | | 0->1 | 4.5MB | | ... | 4.5MB | The next step is to grant base bandwidth. Every (allowed) link is granted `base_bandwidth = 100kB`: ![State of links after granting base bandwidth](assets/nep-0584/scheduler_example_2.png) This grant is subtracted from the link's allowance, we assume that all of the granted base bandwidth will be used for sending receipts. So the allowances change to: | Link | Allowance | | ---- | --------- | | 0->2 | 4.5MB | | 2->2 | 4.5MB | | 0->0 | 4.4MB | | 0->1 | 4.4MB | | ... | 4.4MB | The next step is to process the bandwidth requests. Requests are processed in the order of decreasing link allowance, so the first one to be processed is `(2->2 [540kB])` This request can't be granted because the link `(2->2)` is not allowed. The request is rejected. The remaining requests have the same link allowance, so they'll be processed in random order. Let's first process the request `(0->1 [3950kB])`. Sender and receiver have enough budget to grant this much bandwidth and the link is allowed, so the bandwidth is granted. The grant on `(0->1)` is increased from `100kB` to `3950kB`. Allowance on `(0->1)` is reduced by `3850kB`: | Link | Allowance | | ---- | --------- | | 0->1 | 550kB | | ... | ... | ![State of links after granting 3950kB on link from 0 to 1](assets/nep-0584/scheduler_example_3.png) Then' let's process `(1->1 [210kB, 430kB, 650kB])`. Can we increase the grant on `(1->1)` to 210kB? Yes, let's do that. The bandwidth is granted and the allowance for `(1->1)` decreased. The `210kB` option is removed from the request and the request is reinserted into the priority queue with the lower allowance. ![State of links after granting 210kB on link from 1 to 1](assets/nep-0584/scheduler_example_4.png) | Link | Allowance | | ---- | --------- | | 0->1 | 550kB | | 1->1 | 4090kB | | ... | ... | Now let's process `(1->2 [2.08MB])`. The bandwidth can be granted without any issues. ![State of links after granting 2.08MB on link from 1 to 2](assets/nep-0584/scheduler_example_5.png) | Link | Allowance | | ---- | --------- | | 0->1 | 550kB | | 1->1 | 4090kB | | 1->2 | 2420kB | | ... | ... | Then `(1->1 [430kB, 650kB])` is taken back out of the priority queue. Is it ok to increase the grant on `(1->1)` to 430kB? Yes, do it. Then the `430kB` option is removed from the request, and the request is requeued. ![State of links after granting 430kB on link from 1 to 1](assets/nep-0584/scheduler_example_6.png) | Link | Allowance | | ---- | --------- | | 0->1 | 550kB | | 1->1 | 3870kB | | 1->2 | 2420kB | | ... | ... | Finally `(1->1 [650kB])` is taken out of the queue, but the request can't be granted because it would exceed the incoming limit for shard 1. The final grants are: | Link | Granted Bandwidth | | ---- | ----------------- | | 0->0 | 100kB | | 0->1 | 3950kB | | 0->2 | 0B | | 1->0 | 100kB | | 1->1 | 430kB | | 1->2 | 2080kB | | 2->0 | 100kB | | 2->1 | 100kB | | 2->2 | 0B | Notice how the big receipt sent on `(0->1)` and smaller receipts sent on `(1->1)` compete for the incoming budget of shard 1. Let's imagine a scenario where `(0->1)` always sends `3.9MB` receipts and `(1->1)` always sends many `200kB` receipts. Without allowance we would grant the first value from both bandwidth requests, which would mean that `(0->1)` always gets to send the `3.9MB` receipt and `(1->1)` gets to send a few `200kB` receipts. This isn't fair, much more data would be sent on the `(0->1)` link. With allowance the priority for `(0->1)` sharply drops after granting `3.9MB` and `(1->1)` has the space to send a fair amount of receipts. ---
All shards run the `BandwidthScheduler` algorithm with the same inputs and calculate the same bandwidth grants. The scheduler has to be run at every height, even on missing chunks, to ensure that scheduler state stays identical on all shards. `nearcore` has existing infrastructure (`apply_old_chunk`) to run things on missing chunks, there are implicit state transitions that are used for distributing validator rewards. Scheduler reuses this infrastructure to run the algorithm and modify the state on every height. ### `BandwidthSchedulerState` `BandwidthScheduler` keeps some persistent state that is modified with each run. The state is stored in the shard state trie. Each shard has identical `BandwidthSchedulerState` stored in the trie, all shards run the same algorithm with the same inputs and state and arrive at identical new state that is saved to the trie. `BandwidthSchedulerState` contains current allowance for every pair of shards. Allowance is used to ensure fairness across many heights, so it has to be persisted across heights. ```rust pub enum BandwidthSchedulerState { V1(BandwidthSchedulerStateV1), } pub struct BandwidthSchedulerStateV1 { /// Allowance for every pair of (sender, receiver). Used in the scheduler algorithm. /// Bandwidth scheduler updates the allowances on every run. pub link_allowances: Vec, /// Sanity check hash to assert that all shards run bandwidth scheduler in the exact same way. /// Hash of previous scheduler state and (some) scheduler inputs. pub sanity_check_hash: CryptoHash, } pub struct LinkAllowance { /// Sender shard pub sender: ShardId, /// Receiver shard pub receiver: ShardId, /// Link allowance, determines priority for granting bandwidth. pub allowance: Bandwidth, } ``` There's also `sanity_check_hash`. It's not used in the algorithm, it's only used for a sanity check to assert that scheduler state stays the same on all shards. It's calculated using the previous `sanity_check_hash` and the current list of shards: ```rust let mut sanity_check_bytes = Vec::new(); sanity_check_bytes.extend_from_slice(scheduler_state.sanity_check_hash.as_ref()); sanity_check_bytes.extend_from_slice(CryptoHash::hash_borsh(&all_shards).as_ref()); scheduler_state.sanity_check_hash = CryptoHash::hash_bytes(&sanity_check_bytes); ``` It would be nicer to hash all of the inputs to bandwidth scheduler, but that could require hashing tens of kilobytes of data, which could take a bit of cpu time, so it's not done. The sanity check still checks that all shards ran the algorithm the same number of times and with the same shards. A new trie column is introduced to keep the scheduler state: ```rust pub const BANDWIDTH_SCHEDULER_STATE: u8 = 15; ``` ### Congestion control Bandwidth scheduler limits only the size of outgoing receipts, the gas is limited by congestion control. It's important to make sure that these two are integrated properly. Situations where one limit allows sending receipts, but the other doesn't could lead to liveness issues. To avoid liveness problems, the scheduler checks which shards are fully congested, and doesn't grant any bandwidth on links to these shards (except for the allowed sender shard). This prevents situations where the scheduler would grant bandwidth on some link, but no receipts would be sent because of congestion. There is a guarantee that for every bandwidth grant, the shard will be able to send at least one receipt, which is enough to ensure liveness. There can still be unlucky coincidences where the scheduler grants a lot of bandwidth on a link, but the shard can send only a few receipts because of the gas limit enforced by congestion control. This is not ideal, in the future we might consider merging these two algorithm into one better algorithm, but it is good enough for now. ### Missing chunks When a chunk is missing, the incoming receipts that were aimed at this chunk are redirected to the first non-missing chunk on this shard. The non-missing chunk will be forced to consume incoming receipts meant for two chunks, or even more if there were multiple missing chunks in a row. This is dangerous because the size of receipts sent to multiple chunks could be bigger than one chunk can handle. We need to make sure that `BandwidthScheduler` is aware of this problem and stops sending data when there are missing chunks on the target shard. `BandwidthScheduler` can see when another chunk is missing and it can refrain from sending new receipts until the old ones have been processed. When a chunk is applied, it has access to the block that contains this chunk. It can take a look at other shards and see if their chunks are missing in the current block or not. If a chunk is missing, then the previously sent receipts haven't been processed and the scheduler won't send new ones. In the code this condition looks like this: ```rust fn calculate_is_link_allowed( sender_index: ShardIndex, receiver_index: ShardIndex, shards_status: &ShardIndexMap, ) -> bool { let Some(receiver_status) = shards_status.get(&receiver_index) else { // Receiver shard status unknown - don't send anything on the link, just to be safe. return false; }; if receiver_status.last_chunk_missing { // The chunk was missing, receipts sent previously were not processed. // Don't send anything to avoid accumulation of incoming receipts on the receiver shard. return false; } // ... } ``` It's forbidden to send receipts on a link if the last chunk on the receiver shard is missing. ![Diagram showing how scheduler behaves with one missing chunk](assets/nep-0584/one-missing-chunk.png) ![Diagram showing how scheduler behaves with two missing chunks](assets/nep-0584/two-missing-chunks.png) Sadly this condition isn't enough to ensure that a chunk never receives more than `max_shard_bandwidth` of receipts. This is because receipts sent from a chunk aren't included as incoming receipts until the next non-missing chunk on the sender shard appears. A chunk producer can't include incoming receipts until it has the `prev_outgoing_receipts_root` to prove the incoming receipts against. Because of this there can be a situation where the bandwidth scheduler allows to send some receipts, but they don't arrive immediately because chunks are missing on the sender shard. In the meantime other shards might send other receipts and in the end the receiver can receive receipts sent at multiple shards, which could add up to more than `max_shard_bandwidth`. ![Diagram showing that a chunk might receive more than max_shard_bandwidth](assets/nep-0584/missing-chunk-problem.png) This is still an improvement over the previous solution which allowed to send receipts to shards with missing chunks for up to `max_congestion_missed_chunks = 5` chunks. In the worst-worst case scenario a single chunk might receive `num_shards * max_shard_bandwidth` of receipts at once, but it's highly unlikely to happen. A lot of missing chunks and receipts would have to align for that too happen. To trigger it an attacker would need to have precise control over missing chunks on all shards, which they shouldn't have. A future version of bandwidth scheduler might solve this problem fully, for example by looking at how much was granted but not received and refusing to send more, but it's out of scope for the initial version of the bandwidth scheduler. The whole environment might change soon, SPICE might remove the concept of missing chunks altogether, for now we can live with this problem. ### Resharding During resharding the list of existing shards changes. Bandwidth scheduler assumes that sender and receiver shards are from the same layout, but this is not true for one height at the resharding boundary where senders are from the old layout, but receivers are from the new one. Ideally the bandwidth scheduler would make sure that bandwidth is properly granted when the sets of senders and receivers are different, but this not implemented for now. The grants will be slightly wrong (but still within limits) on the resharding boundary. They will be wrong for only one block height, after resharding the senders and receivers will be the same again and the scheduler will work properly again. The amount of work needed to properly support resharding exceeds the benefits, we can live with a slight hiccup for one height at the resharding boundary. To properly handle resharding we would have to: - Use different ShardLayouts for sender shards and receiver shards - Interpret bandwidth requests using the `BandwidthSchedulerParams` that they were created with - Make sure that `BandwidthSchedulerParams` are correct on the resharding boundary It's doable, but it's out of scope for the initial version of the bandwidth scheduler. There's one additional complication with generating bandwidth requests. When a parent shard is split into two children, the parent disappears from the current `ShardLayout`, but other shards might still have buffered receipts aimed at the parent shard. Bandwidth scheduler will not grant any bandwidth to send receipts to a shard that doesn't exist, which would prevent these buffered receipts from being sent, they'd be stuck in the buffer forever. To deal with that we have to do two things. First thing is to redirect receipts aimed at the parent to the proper child shard. When we try to forward a receipt from the buffer aimed at the parent, we will determine which child the receipt should go to and forward it to this child, using bandwidth limits meant for sending receipts to the child shard. Second thing is to generate bandwidth requests using both the parent buffer and the child buffer. A shard can't send any receipts from the parent buffer without a bandwidth grant, so we have to somehow include the parent buffer in the bandwidth requests, even though the parent doesn't exist in the current `ShardLayout`. This is done by merging (conceptually) parent and child buffers when generating a bandwidth request. First we walk over receipt groups in the parent buffer, then the receipt groups in the child buffer. This way the bandwidth grants to the child will include receipts aimed at the parent. ### Distributing remaining bandwidth After bandwidth scheduler processes all of the bandwidth requests, there's usually some leftover budget for sending and receiving data between shards. It'd be wasteful to not use the remaining bandwidth, so the scheduler will distribute it between all the links. Granting extra bandwidth helps to lower latency, it might allow a shard to send out a new receipt without having to make a bandwidth request. The algorithm for distributing remaining bandwidth works as follows: 1) Calculate how much more each shard could send and receive, call it `bandwidth_left` 2) Calculate how many active links there are to each sender and receiver. A link is active if receipts can be sent on it, i.e. it's not forbidden because of congestion or missing chunks. Call this number `links_num`. 3) Order all senders and receivers by `average_link_bandwidth = bandwidth_left/links_num`, in increasing order. Ignore shards that don't have any bandwidth or links. 4) Walk over all senders (as ordered in (3)), for each sender walk over all receivers (as ordered in (3)) and try to grant some bandwidth on this link 5) Grant `min(sender.bandwidth_left / sender.links_num, receiver.bandwidth_left / receiver.links_num)` on the link 6) Decrease `sender.links_num` and `receiver.links_num` by one. The algorithm is a bit tricky, but the intuition behind it is that if a shard can send 3MB more, and there are 3 active links connected to this shard, then it should send about 1MB on every one of these links. But it has to respect how much each of the receiver shards can receive. If one of them can receive only 500kB we can't grant 1MB on this link. That's why the algorithm takes the minimum of how much the sender should send and how much the receiver should receive. Sender and receiver negotiate the highest amount of data that can be sent between them. Shards are ordered by `average_link_bandwidth` to ensure high utilization - it gives the guarantee that all shards processed later will be able to send/receive at least as much as the shard being processed now.
Here's an example, click to expand Let's say there are three shards, each shard could send and receive a bit more data (called the remaining budget): | Shard | Sending budget | Receiving budget | | ----- | -------------- | ---------------- | | 0 | 300kB | 700kB | | 1 | 4.5MB | 100kB | | 2 | 1.5MB | 4.5MB | Shard 2 is fully congested, which means that only the allowed shard (shard 1) can send receipts to it. ![State of links before distributing remaining bandwidth](assets/nep-0584/distribute_remaining_example_1.png) First let's calculate how many active links there are to each shard. Active links are links that are not forbidden: | Shard | Active sending links | Active receiving links | | ----- | -------------------- | ---------------------- | | 0 | 2 | 3 | | 1 | 3 | 3 | | 2 | 2 | 1 | Average link bandwidth is calculated as the budget divided by the number of active links. | Shard | Average sending bandwidth | Average receiving bandwidth | | ----- | ------------------------- | --------------------------- | | 0 | 300kB/2 = 150kB | 700kB/3 = 233kB | | 1 | 4.5MB/3 = 1.5MB | 100kB/3 = 33kB | | 2 | 1.5MB/2 = 750kB | 4.5MB/1 = 4.5MB | Now let's order senders and receivers by their average link bandwidth | Sender shard | Average link bandwidth | | ------------ | ---------------------- | | 0 | 150KB | | 2 | 750kB | | 1 | 1.5MB | | Receiver shard | Average link bandwidth | | -------------- | ---------------------- | | 1 | 33kB | | 0 | 233kB | | 2 | 4.5MB | And now let's distribute the bandwidth, process senders in the order of increasing average link bandwidth and for every sender process the receiver in the same order: Link (0 -> 1): Sender proposes 300kB/2 = 150kB. Receiver proposes 100kB/3 = 33kB. Grant 33kB ![State of links after granting 33kB on link from 0 to 1](assets/nep-0584/distribute_remaining_example_2.png) Link (0 -> 0): Sender proposes 267kB/1 = 267kB. Receiver proposes 700kB/3 = 233kB. Grant 233kB ![State of links after granting 233kB on link from 0 to 0](assets/nep-0584/distribute_remaining_example_3.png) Link (0 -> 2): This link is not allowed. Nothing is granted. Link (2 -> 1): Sender proposes 1.5MB/2 = 750kB. Receiver proposes 66kB/2 = 33kB. Grant 33kB ![State of links after granting 33kB on link from 2 to 1](assets/nep-0584/distribute_remaining_example_4.png) Link (2 -> 0): Sender proposes 1467kB/1 = 1467kB. Receiver proposes 467kB/2 = 233kB. Grant 233kB ![State of links after granting 233kB on link from 2 to 0](assets/nep-0584/distribute_remaining_example_5.png) Link (2 -> 2): This link is not allowed. Nothing is granted. Link (1 -> 1): Sender proposes 4.5MB/3 = 1.5MB. Receiver proposes 33kB/1. Grant 33kB ![State of links after granting 33kB on link from 1 to 1](assets/nep-0584/distribute_remaining_example_6.png) Link (1 -> 0): Sender proposes 4467kB/2 = 2233kB. Receiver proposes 233kB/1 = 233kB. Grant 233kB ![State of links after granting 233kB on link from 1 to 0](assets/nep-0584/distribute_remaining_example_7.png) Link (1 -> 2): Sender proposes 4234kB/1 = 4234kB. Receiver proposes 4.5MB. Grant 4234kB ![State of links after granting 4234kB on link from 1 to 2](assets/nep-0584/distribute_remaining_example_8.png) And all of the bandwidth has been distributed fairly and efficiently. ---
When all links are allowed the algorithm achieves very high bandwidth utilization (99% of the theoretical maximum). When some links are not allowed the problem becomes much harder, it starts being similar to the maximum flow problem. The algorithm still achieves okay utilization (75% of the theoretical maximum), and I think this is good enough. In this case we want a fast heuristic, not a slow algorithm that will solve the max flow problem perfectly. The algorithm is safe because it never grants more than `min(sender.bandwidth_left, receiver.bandwidth_right)`, so it'll never go over the limits. The utilization and fairness is good, but I don't have a good proof for that, just an intuitive understanding. The algorithm is fast, behaves well in practice and is provably safe, and I think that is good enough. For the exact implementation see: https://github.com/near/nearcore/pull/12682 ### One block delay There's a one block delay between requesting bandwidth and receiving a grant. This is not ideal, most large receipts will have to be buffered and sent out at the next height, it'd be nicer if we could quickly negotiate bandwidth and send them immediately. It is a hard problem to solve - a shard doesn't know what other shards want to send, so it needs to contact them and negotiate. Maybe it'd be possible to negotiate it off-chain in between blocks, but that would be much more complex - we would have to make sure that the negotiation happens quickly even when latency between nodes is high and ensure that everything is fair and secure. The idea is explored further in `Option D` section, but for now I think we can go with a solution that is simpler and should be good enough, even though it has a one block delay. At first glance it might seem that the delay prevents us from using 100% of the bandwidth - a big receipt takes 2 blocks to reach the other shard, doesn't that mean that we get only 50% of the theoretical throughput? Not really, the delay increases latency, but it doesn't affect throughput. An application that wants to utilize 100% of bandwidth can submit the receipts and they'll be queued and sent over utilizing 100% of the bandwidth, just with a one block delay. There's no 50% problem. As an example one can imagine a contract that wants to send 4MB of data to another shard at every height. The contract will produce a 4MB receipt at every height, the shard will generate a 4MB `BandwidthRequest` at every height, and the bandwidth scheduler will grant the shard 4MB of bandwidth at every height (assuming no requests from other shards). At the first height the 4MB will be buffered, but for all the following heights the shard will have the 4MB grant and it'll be able to send 4MB of data to the other shard. We can utilize 100% of the bandwidth despite the delay, we just have to make sure that we can buffer ~10MB of receipts in the outgoing queue. ### Performance Complexity of the bandwidth scheduler algorithm is `O(num_shards^2 + num_requests * log(num_requests))`, which in the worst case is equal to `O(num_shards^2 * log(num_shards))`. It's hard (impossible?) to avoid the `num_shards^2` because the scheduler has to consider every pair of shards. The `log(num_requests)` comes from sorting by allowance. Scheduler works quickly for lower number of shards, but the time needed to run it grows quickly as the number of shards increases. Here's a benchmark of the worst-case scenario performance, measured on a typical `n2d-standard-8` GCP VM with an AMD EPYC 7B13 CPU: | Number of shards | time | | ---------------- | ----------| | 6 | 0.13 ms | | 10 | 0.19 ms | | 32 | 1.85 ms | | 64 | 5.80 ms | | 128 | 23.98 ms | | 256 | 97.44 ms | | 512 | 385.97 ms | It's important to note that this is worst-case performance, with all shards wanting to send a ton of small receipts to other shards. Usually the number of bandwidth requests will be lower and the scheduler will work quicker than that. The current version of the scheduler should work fine up to 50-100 shards, after that we'll probably need some modifications. A quick solution would be to randomly choose half of the shards at every height and only grant bandwidth between them, this would cut `num_shards` in half. There's also some potential for parallelization, the bandwidth grants could be calculated in parallel with application of the action receipts. I think we can worry about it when we reach 100 shards, with this many shards the environment and typical patterns will probably change a lot, we can analyze them and modify the scheduler accordingly. ### Byzantine fault tolerance Bandwidth scheduler needs to be resistant to malicious actors. All validators have to check that bandwidth requests are produced and processed correctly, and reject chunks where this isn't the case. Bandwidth scheduler is run during chunk application, all chunk validators run the same bandwidth scheduler with the same inputs and generate the same results. Any discrepancy in the scheduler would result in different state after chunk application, which would cause the validation to fail. The produced bandwidth requests are stored in the chunk header, along with other things produced when applying a chunk. Chunk validators apply the chunk and verify that produced data matches the data in chunk header, they will not endorse a chunk if it doesn't match. The data from previous chunk header (like previous bandwidth requests) can be trusted because the previous chunk headers were endorsed by chunk validators at the previous height. The logic is pretty much identical to the one used to validate CongestionInfo for congestion control. ### Testing Bandwidth scheduler is pretty complex, and it's a bit hard to reason about how things really flow, so it's important to test it well. There is a bunch of tests which run some kind of workload and check if the parameters look alright. The two main parameters are: - Utilization - are receipts sent as fast as theoretically possible? Utilization should be close to 100% for small receipts. With big receipts it should be at least 50%. (If all receipts have size `max_shard_bandwidth / 2 + 1` we can only send one such receipt per height, and we get ~50% utilization) - Fairness - is every link sending the same amount of bytes on average? As long as all outgoing buffers are full, all links should send about the same amount of data on average, there should be no favorites. The scheduler algorithm was tested in two ways: - On a blockchain simulator, which simulates a few shards sending receipts between each other, along with missing chunks and blocks. It doesn't take into account other mechanisms like congestion control. It was used to test utilization and fairness in various scenarios, without interference from other congestion mechanisms. A simulator allows to quickly run a test over thousands of blocks, which would take minutes in actual `nearcore`. - In testloop, which runs the actual blockchain code in a deterministic way. The tests are slower, but test the actual code that will run in the real world. They also allow to test interaction with other mechanisms like congestion control. The simulator tests went well. Utilization and fairness were good, the only issue that these tests found is that a chunk might sometimes receive more than `max_shard_bandwidth` because of missing chunks, which is a known issue with the design. The testloop tests were a bit below expectations. It seems like there are other mechanisms that prevent us from reaching full cross-shard bandwidth utilization. It was hard to reach a state where all of the outgoing buffers were full and the scheduler could go at full speed. I plan to add more observability which should shed more light on what exactly is going on there. Still the test results were okay, the scheduler works reasonably well. Testing in testloop also allowed to find some bugs with the congestion control integration. ## Reference Implementation Here are the PRs which implement this NEP in `nearcore`: - https://github.com/near/nearcore/pull/12234: wiring for bandwidth scheduler - https://github.com/near/nearcore/pull/12307: Do bandwidth scheduler header upgrade the same way as for congestion control - https://github.com/near/nearcore/pull/12333: implement the BandwidthRequest struct - https://github.com/near/nearcore/pull/12464: generate bandwidth requests based on receipts in outgoing buffers - https://github.com/near/nearcore/pull/12511: use u16 for shard id in bandwidth requests - https://github.com/near/nearcore/pull/12533: bandwidth scheduler - https://github.com/near/nearcore/pull/12682: distribute remaining bandwidth - https://github.com/near/nearcore/pull/12694: make BandwidthSchedulerState versioned - https://github.com/near/nearcore/pull/12719: slightly increase base_bandwidth - https://github.com/near/nearcore/pull/12728: include parent's receipts in bandwidth requests - https://github.com/near/nearcore/pull/12747: Remove BandwidthRequestValues which can never be granted ## Security Implications - Risk of too many receipts incoming receipts to a chunk. There are certain corner cases in which a chunk could end up with more than `max_shard_bandwidth` of incoming receipts, up to `num_shards * max_shard_bandwidth` in the worst-worst case. This has already been a problem before bandwidth scheduler, the scheduler slightly increases protection against such situations, but they could still happen. The consequence of too many receipts would be increased witness size, which could cause missing chunks, and in extreme cases even chain stalls. The corner case is pretty hard to trigger for an attacker, they would have to find a way to precisely cause multiple missing chunks, which would be a vulnerability by itself. - The code is a bit complex, there's a risk of a bug somewhere in the code. We trade complexity for higher performance. The most likely bug to happen would be some kind of liveness issue, e.g. receipts getting stuck in the buffer without ever being sent. Worst-case scenario would probably be a panic somewhere in the code which could cause a shard to stall. - Better protection against DoS attacks. Cross shard throughput in the previous solution was pretty low, which made it easier to run DoS attacks by generating a ton of cross-shard receipts. Bandwidth scheduler significantly increases cross-shard throughput, which makes such attacks less viable. ## Alternatives There were a few alternative designs that we considered: ### Partial receipts A lot of problems come from the fact that receipts can be big (up to 4MB in the current protocol version). It means that we have to choose which receipts can be sent and which ones must wait for their turn. It would be much easier if all of the receipts were small (say under 1kB), we would be able to divide bandwidth in a more continuous way, e.g. just grant `max_shard_bandwidth / num_shards` on every link. What if we split big receipts into parts, a 4MB receipt would be split into 4000 partial receipts, 1kB each. Partial receipts would be sent to the receiver shard over multiple block heights and saved to the receiver's state. Once all of the partial receipts are available, the whole receipt would be rebuilt from the parts stored in the state. This would work fine with stateful validation, but it isn't really viable for stateless. In stateless validation everything that is read from the state must be included in the `ChunkStateWitness`. Reconstructing a receipt from its parts would require adding all of the parts to the witness. This means that every receipt will be included in the witness twice - first as the incoming partial receipts, and then as the parts used to rebuild the whole receipt. This effectively halves the witness bandwidth available for incoming receipts. And eventually we would include the whole receipt in the witness anyway, so we might as well send the whole receipt immediately. There could also be problems with fairness when choosing which receipts to rebuild - we might not be able to rebuild all of them, as that could cause witness size to explode, so we'd need to have some sort of fair scheduler to choose which ones to include in the witness. At this point the problem becomes eerily similar to bandwidth scheduler, just with more complications. ### Chunk producers choosing incoming receipts (also called `Option D` in some docs) Bandwidth scheduler limits things from the sender side, but really it would be better to do that on the receiver side, the sender usually has limited information about the receiver's exact state. Let's allow the chunk producer to choose the incoming receipts that it includes in a chunk. All shards would produce outgoing receipts and publish some small metadata about them (e.g this many receipts, with this size, and this much gas). Chunk producers would read the metadata and fetch the receipts as fast as they can. When it's time to produce a chunk, the chunk producer would include only the incoming receipts that it was able to fetch so far, they wouldn't be forced to include all of the receipts that were sent. We would need some additional mechanism to limit how many receipts were sent, but not included, but that's doable. It would make the bandwidth limits correspond exactly to the actual networking situation in the chain. It's a similar approach as the TCP flow control uses - sender sends something, receiver receives as fast as it can and sends acks for the things it received. If sender notices that it's sending data too fast, it slows down. This solution also has potential to eliminate the one-block delay that is present in bandwidth scheduler. I feel like this is the "proper" way that things should be done, but this approach was not chosen for two reasons: - It's much harder to implement than bandwidth scheduler. Bandwidth scheduler can be added relatively painlessly in the runtime, this approach would require a lot of delicate changes to various subsystems. Bandwidth scheduler solves 80% of the problem for 50% of the effort. - There are various security considerations around giving chunk producer more choice. A malicious chunk producer could choose the receipts that it prefers, giving it extra power over what happens on the blockchain, that could lead to potential security vulnerabilities. ## Future possibilities ### Better handling of missing chunks The current way of handling missing chunks isn't as good as it could be. In some cases it's possible for a shard to receive more than `max_shard_bandwidth` of receipts. It'd be good to improve the scheduler to guard against such situations. However it's also possible that the concept of missing chunks will disappear if we move to SPICE or some other consensus mechanism. ### Merging bandwidth scheduler with congestion control Size of outgoing receipts is limited by the bandwidth scheduler, but their gas is limited by congestion control. This is awkward, there could potentially be situations where the scheduler grants a lot fo bandwidth, but receipts can't be sent because of gas limits imposed by congestion control. I think ideally they should be merged into one `CongestionScheduler` which would look how much size and gas there is in every outgoing buffer and grant bandwidth/gas based on that. It could even detect cycles and allow the receipts to progress in a smart way, which the current congestion control can't do. But that would be a big effort, for now we have two separate mechanisms for gas and size, it solves most of the problems, even if it isn't ideal. ### Don't put bandwidth requests in the chunk header The chunk header should be as small as possible, and putting bandwidth requests there could add tens of bytes to each header. We could distribute the requests separately and include only their merkle root in the chunk header. ### Optimize witness size when a chunk receives too many receipts To deal with situations where there are too many incoming receipts to a chunk, we could add a new rule for chunk application - only the first 4MB of incoming receipts are processed, the rest of incoming receipts will always be delayed. Thanks to this rule we would be able to do a trick - for the first 4MB of incoming receipts include the actual receipts in the witness, for the rest include only lightweight metadata that will be put in the delayed queue. Later when the metadata is read from the delayed queue we will include the actual receipts in the witness and they'll be executed. So if there's 8MB of incoming receipts the chunk producer would include only the first 4MB in the witness, for the rest there would metadata that will be put in the delayed queue. At the next height the metadata would be read from the delayed queue, the next chunk producer would include these receipts in the witness and they'd be processed. This would allow to keep witness size small, even when there are too many incoming receipts. ## Consequences ### Positive - Higher cross-shard throughput - Lower latency for big cross-shard receipts - Better scalability - Slightly better protection against large incoming receipts than the previous solution ### Neutral - ? ### Negative - More complexity in the runtime, higher potential for bugs - Additional compute when applying a chunk ### Backwards Compatibility The change is backwards-compatible. Everything that worked before the change will still work after it. ## Unresolved Issues (Optional) [Explain any issues that warrant further discussion. Considerations - What parts of the design do you expect to resolve through the NEP process before this gets merged? - What parts of the design do you expect to resolve through the implementation of this feature before stabilization? - What related issues do you consider out of scope for this NEP that could be addressed in the future independently of the solution that comes out of this NEP?] Most of the issues are resolved, the change should be ready for stabilization. ## Changelog [The changelog section provides historical context for how the NEP developed over time. Initial NEP submission should start with version 1.0.0, and all subsequent NEP extensions must follow [Semantic Versioning](https://semver.org/). Every version should have the benefits and concerns raised during the review. The author does not need to fill out this section for the initial draft. Instead, the assigned reviewers (Subject Matter Experts) should create the first version during the first technical review. After the final public call, the author should then finalize the last version of the decision context.] ### 1.0.0 - Initial Version > Placeholder for the context about when and who approved this NEP version. #### Benefits > List of benefits filled by the Subject Matter Experts while reviewing this version: - Benefit 1 - Benefit 2 #### Concerns > Template for Subject Matter Experts review for this version: Status: New | Ongoing | Resolved | # | Concern | Resolution | Status | | --: | :------ | :--------- | -----: | | 1 | | | | | 2 | | | | ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0591.md ================================================ --- NEP: 591 Title: Global Contracts Authors: Bowen Wang , Anton Puhach , Stefan Neamtu Status: Final DiscussionsTo: https://github.com/nearprotocol/neps/pull/591 Type: Protocol Replaces: 491 Version: 1.0.0 Created: 2025-02-11 LastUpdated: 2025-03-17 --- ## Summary This proposal introduces global contracts, a new mechanism that allows smart contracts to be deployed once and reused by any account without incurring high storage costs. Currently, deploying the same contract multiple times on different accounts leads to significant storage fees. Global contracts solve this by making contract code available globally, allowing multiple accounts to reference it instead of storing their own copies. Rather than requiring full storage costs for each deployment, accounts can simply link to an existing global contract, reducing redundancy and improving scalability. This approach optimizes storage, lowers costs, and ensures efficient contract distribution across the network. ## Motivation A common use case on NEAR is to deploy the same smart contract many times on many different accounts. For example, a multisig contract is a frequently deployed contract. However, today each time such a contract is deployed, a user has to pay for its storage and the cost is quite high. For a 300kb contract the cost is 3N. With the advent of chain signatures, the smart contract wallet use case will become more ubiquitous. As a result, it is very desirable to be able to reuse already deployed contract without having to pay for the storage cost again. Additionally, global contracts cover the underlying use case for [NEP-491](https://github.com/near/NEPs/pull/491): [#12818](https://github.com/near/nearcore/pull/12818). Some businesses onboard their users by creating an account on their behalf and deploying a contract to it. Such a use case is susceptible to refund abuse, where there is a financial incentive to repeatedly create and destroy an account, cashing in on the storage fee that was initially paid by the business to deploy the contract. ## Specification Global contract can be deployed in 2 ways: either by its hash or by owner account id. Contracts deployed by hash are effectively immutable and cannot be updated. When deployed by account id the owner can redeploy the contract updating it for all its users. Users can use contracts deployed by hash if they prefer having control over contract updates. In order to update the contract user would have to explicitly switch to a different version of the contract deployed under a different hash. Contracts deployed by account id should be used when user trusts contract developers to update the contract for them. For example if user accounts are created specifically for some application to be onchain. We introduce new receipt action for deploying global contracts: ```rust struct DeployGlobalContractAction { code: Vec, deploy_mode: GlobalContractDeployMode, } enum GlobalContractDeployMode { /// Contract is deployed under its code hash. /// Users will be able reference it by that hash. /// This effectively makes the contract immutable. CodeHash, /// Contract is deployed under the owner account id. /// Users will be able reference it by that account id. /// This allows the owner to update the contract for all its users. AccountId, } ``` User pays for storage by burning NEAR tokens from its balance depending on the deployed contract size. Global contract is not checked to be compilable wasm (just like in case of a regular contract), so it is possible to deploy invalid wasm and that still burns tokens. Also new action is added for using previously deployed global contract: ```rust struct UseGlobalContractAction { contract_identifier: GlobalContractIdentifier, } enum GlobalContractIdentifier { CodeHash(CryptoHash), AccountId(AccountId), } ``` This action is similar to deploying a regular contract to an account, except user does not cover storage deposit. Using non-existing global contract (both by its hash and account id) results in `GlobalContractDoesNotExist` action error, so users have to wait for global contract distribution to be completed in order to starting using the contract. ## Reference Implementation ### Storage In order to have global contracts available to users on all shards we store a copy in each shard's trie. A new trie key is introduced for that: ```rust pub enum TrieKey { ... GlobalContractCode { identifier: GlobalContractCodeIdentifier, }, } pub enum GlobalContractCodeIdentifier { CodeHash(CryptoHash), AccountId(AccountId), } ``` The value is contract code bytes, similar to `TrieKey::ContractCode`. ### Distribution Global contract has to be distributed to all shards after being deployed. This is implemented with a dedicated receipt type: ```rust enum ReceiptEnum { ... GlobalContractDistribution(GlobalContractDistributionReceipt), } enum GlobalContractDistributionReceipt { V1(GlobalContractDistributionReceiptV1), } struct GlobalContractDistributionReceiptV1 { id: GlobalContractIdentifier, target_shard: ShardId, already_delivered_shards: Vec, code: Arc<[u8]>, } ``` `GlobalContractDistribution` receipt is generated as a result of processing `DeployGlobalContractAction`. Such receipt contains destination shard `target_shard` as well as list of already visited shard ids `already_delivered_shards`. Applying `GlobalContractDistribution` receipt updates the corresponding `TrieKey::GlobalContractCode` in the trie for the current shard. It also generates distribution receipt for the next shard in the current shard layout. This process continues until `already_delivered_shards` contains all shards. This way we ensure that at any point in time there is at most one `GlobalContractDistribution` receipt in flight for a given deployment and eventually it will reach all shards. Such distribution also works well at the resharding boundary. If the receipt is applied with the old shard layout then storage resharding will make sure presence of the contract in both child shards. In case of application in the new layout, the distribution logic will take care of forwarding the receipt to the newly introduced child shards. Note that `GlobalContractDistribution` does not target any specific account (`system` is used as a placeholder) and `target_shard` is used for receipt routing. ### Usage We change `Account` struct to make it possible to reference global contracts. `AccountV2` is introduced changing `code_hash: CryptoHash` field to more generic `contract: AccountContract`: ```rust enum AccountContract { None, Local(CryptoHash), Global(CryptoHash), GlobalByAccount(AccountId), } ``` Applying `UseGlobalContractAction` updates user account `contract` field accordingly. `FunctionCall` action processing is updated to respect global contracts. This includes updating [contract preparation pipeline](https://github.com/near/nearcore/blob/fb95d7b7740d1fda9245afa498ce4e9ac145c8af/runtime/runtime/src/pipelining.rs#L24) as well as [recording of the executed contract to be included in the state witness](https://github.com/near/nearcore/blob/fb95d7b7740d1fda9245afa498ce4e9ac145c8af/core/store/src/trie/update.rs#L338). ### Costs For global contracts deployment we burn tokens for storage instead of locking like what we do regular contracts today. The cost per byte of global contract code `global_contract_storage_amount_per_byte` is set as 10x the storage staking cost `storage_amount_per_byte`. Additionally we add action costs for the global contract related actions: * `action_deploy_global_contract` is exactly the same as `action_deploy_contract` * `action_deploy_global_contract_per_byte`: * send costs are the same as `action_deploy_contract_per_byte` * execution costs should cover distribution of the contract to all shards: * this is pretty expensive for the network, so want want to charge significant amount of gas for that * we still want to be able to fit max allowed contracts size into single chunk space: `max_gas_burnt = 300_000_000_000_000`, `max_contract_size = 4_194_304`, so it should be at most `max_gas_burnt / max_contract_size = 71_525_573` * we need to allow for some margin for other costs, so we can round it down to `70_000_000` Using a global contract incurs two costs, as follows: * `action_use_global_contract` * mirrors `action_deploy_contract` * base cost for processing a usage action * `action_use_global_contract_per_identifier_byte` * mirrors `action_deploy_contract_per_byte`, but based on the global contract identifier length * introduced because the `AccountId` in `GlobalByAccount(AccountId)` variant can vary in length, unlike `Global(CryptoHash)` with a fixed size of 32 bytes Referencing a global contract locks tokens for storage in accordance with `storage_amount_per_byte` based on global contract identifier length, calculated similarly to `action_use_global_contract_per_identifier_byte`. Although the cost structure is similar to that of single shard contract deployments, the overall fees are significantly lower because only a few bytes are stored for the reference. This is desired, because referencing a global contract is not an expensive operation for the network. ## Security Implications One potential issue is increasing infrastructure cost for global contracts with growing number of shards. A global contract is effectively replicated on every shard, so with increase in number of shards each global contract uses more storage. This can be potentially addressed in the future by making deployment costs parametrized with the number of shards in the current epoch, but it still wouldn't address the issue for the already deployed contracts. ## Alternatives In [the original proposal](https://github.com/near/NEPs/issues/556) we considered storing global contracts in a separate global trie (managed at the block level) and introducing a dedicated distribution mechanism. We decided not to proceed with this approach as it requires significantly higher effort to implement and also introduces new critical dependencies for the protocol. ## Future possibilities Global contracts can potentially be used as part of the sharded contracts effort. Sharded contract code should be available on all shards, so using global contracts for that might be a good choice. ## Changelog [The changelog section provides historical context for how the NEP developed over time. Initial NEP submission should start with version 1.0.0, and all subsequent NEP extensions must follow [Semantic Versioning](https://semver.org/). Every version should have the benefits and concerns raised during the review. The author does not need to fill out this section for the initial draft. Instead, the assigned reviewers (Subject Matter Experts) should create the first version during the first technical review. After the final public call, the author should then finalize the last version of the decision context.] ### 1.0.0 - Initial Version > Placeholder for the context about when and who approved this NEP version. #### Benefits > List of benefits filled by the Subject Matter Experts while reviewing this version: * Benefit 1 * Benefit 2 #### Concerns > Template for Subject Matter Experts review for this version: > Status: New | Ongoing | Resolved | # | Concern | Resolution | Status | | --: | :------ | :--------- | -----: | | 1 | | | | | 2 | | | | ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0611.md ================================================ --- NEP: 611 Title: Pending Transaction Queue and Gas Keys Authors: Robin Cheng , Darioush Jalali Status: Draft DiscussionsTo: https://github.com/near/NEPs/pull/611 Type: Protocol Version: 1.2.2 Created: 2025-05-28 LastUpdated: 2026-03-05 --- ## Summary In the near future of the Near blockchain, we foresee that via the SPICE project, transaction and receipt execution will become decoupled from the blockchain itself; they will no longer run in lockstep. Instead, transactions will be included in the blocks first, and then execution will follow later. This inherently introduces a problem that we must accept transactions before we know whether they are valid. Today, when a chunk producer produces a chunk containing a transaction, it can verify using the current shard state that the transaction has a valid signature and a valid nonce, and that the corresponding account has enough balance. But as execution becomes asynchronous, we no longer have the current shard state to verify the transactions against. This NEP proposes a mechanism called the Pending Transaction Queue to solve this problem. ## Motivation ### Why is this worth solving? A potential for DoS attacks exists whenever the blockchain allows anyone to submit work without paying. Invalid transactions present such a vulnerability: if a transaction is included in a block (or more precisely, a chunk of the block) but ends up being invalid because the sender does not have enough balance, this transaction takes block space but cannot be charged against anyone. A very easy-to-perform attack exists if we do nothing to mitigate the problem: * The attacker creates two accounts, $A$ and $B$, with sufficiently many access keys each, and deploying a specific contract for both accounts that provides a `send_near` function. * The attacker deposits 10 NEAR in $A$. * The attacker then performs the following, repeatedly: * Submit a transaction to call A's `send_near` function, instructing it to send the account's remaining balance to $B$. * Right after, it floods the blockchain by signing and submitting many (arbitrary) transactions as $A$. Because execution is asynchronous, the chunk producers think that there is still enough balance in $A$'s account, so these transactions are accepted into chunks. * Some blocks later, the execution catches up, and the `send_near` function drains $A$'s account. * Subsequent executions of the following transactions all fail because $A$ has insufficient balance. * After this is done, the attacker repeats but with $A$ and $B$ swapped. This attack can be carried out with a very simple script and requires no cost other than a single contract call every few blocks, but this ends up filling up the blockchain, preventing legitimate transactions from being included. This is also very hard to defend against, because the attacker can simply create more accounts. Note that this attack pattern is not the only problematic one; instead of sending away the balance, the attacker can also delete access keys or delete the whole account. At a high level, this solution has two key ideas: 1. Limiting the number of in-flight transactions for accounts with contracts: This aims for backwards compatibility for most use cases and also limits the amount of spam they can include on chain. 2. Allowing users with sophisticated issuance needs to prepay for gas and have many in-flight transactions. This NEP additionally includes a way to associate a single signing key and balance with an array of nonces, to provide a convenient way for parallel transaction issuance for such users. ## Specification To solve this problem, we present two critical components which work together to ensure that all accepted transactions are valid. At a high level, they are: * **Access Key vs Gas Key**: We introduce a new type of access key, which we will call "Gas Key". A gas key can be funded with NEAR, and when issuing a transaction using a gas key, the transaction only consumes gas from the gas key, not from the account. * **Pending Transaction Queue**: Chunk producers keep track of pending (accepted into a block, but not executed) transactions, and ensure that transactions sent with access keys are limited in parallelism, whereas transactions sent with gas keys are limited to the available gas in the gas key. We will now specify how exactly they work. ### Definition of "Pending Transactions" In a model where execution follows but lags behind consensus, there are transactions which are accepted into consensus and thus committed to be executed in the near future, but are not yet executed. This set of transactions is called the *pending transactions*. We always discuss this in the context of one shard. Note that there are two slightly different ways to treat this definition, depending on how exactly the "execution head" (how far the execution has caught up) is defined: * It can be defined locally as the progress of execution at a node, but this will be different between nodes. * It can be defined deterministically as the last block whose execution result is certified by consensus. For the purpose of this NEP, we use the latter definition, so that the notion of pending transactions is consistent across all nodes and the determination of what transactions are eligible to be included by a chunk producer can be verified -- even though we do not plan to implement this verification right now. Another note is that the notion of pending transactions is anchored at a specific chunk that is being produced. In case of forks, we use the block that the chunk is being produced on top of to compute the set of pending transactions. Finally, this NEP does *not* depend on the implementation of SPICE. In the context where SPICE is not yet in effect, we consider the pending transactions queue to always be empty (despite technically being one chunk worth of transactions due to execution lagging one block behind in the current implementation), because we can always verify the validity of all transactions at the moment of inclusion. ### Access Key vs Gas Key We introduce a new transaction version, `TransactionV1`, which is able to either specify an access key or a gas key. Any older transaction version is equivalent to a `TransactionV1` that specifies an access key. Note: The nearcore reference implementation already includes a `TransactionV1` struct, which is unused. This struct was intended for adding support for a `priority_fee`, though this was never activated and there is no plan to activate it, so it will be removed with this NEP instead of being added to the new transaction format. This prioritizes protocol simplicity over backwards compatibility for crates that may have taken a dependency on an inactive feature. #### Access Key Parallelism Restriction We now restrict the ability to send multiple parallel pending transactions with Access Keys, from accounts that also have a contract deployed. Specifically, for any given account $A$ that has a contract deployed, the total number of access key transactions (across all access keys in the account) in the pending transaction queue whose sender is $A$ cannot exceed $P_{\mathrm {max}}$, a constant determined by the epoch; we propose $P_{\mathrm {max}} = 4$. In other words, with traditional access keys, one cannot send more than 4 transactions from an account with a contract deployed, before they are executed. If one wishes to send more transactions in parallel, they would need to create a gas key. From a UX perspective, we pick the $P_{\mathrm {max}}$ constant so that it is very likely that anyone exceeding this parallelism is a developer who needs to send parallel transactions to the contract. In such cases, we require them to use gas keys. This is not a pure limitation; as we will see later, a gas key also simplifies the developer's implementation for sending parallel transactions. Accounts that do not have a contract deployed are not restricted with any limited parallelism. #### Gas Keys This NEP adds Gas Keys, **conceptually** defined as the following: ```rust struct ConceptualGasKey { public_key: PublicKey, nonces: Vec, balance: Balance, permission: AccessKeyPermission, } ``` Gas keys are managed using the standard key management actions (`AddKey`, `DeleteKey`) along with new transfer actions to move balance to/from gas keys: * `AddKey(PublicKey, AccessKeyPermission)` with a gas key permission: Creates a gas key with the given public key, permission, and number of nonces (specified in the permission). * `DeleteKey(PublicKey)`: Deletes a gas key or access key. * `TransferToGasKey(PublicKey, Balance)`: deducts balance from the account and gives it to the gas key. * `WithdrawFromGasKey(PublicKey, Balance)`: deducts the balance from the gas key and gives it to the account. Since gas keys are a kind of access key, they share a namespace. This means that the user is not allowed to create a gas key with the same public key as an existing access key. This is important in refund handling. ### Gas Key Actions `AddKey` when adding a gas key is verified and executed as follows: * Account must already exist. * Check that the same key does not already exist as a gas key or access key. * Requested number of nonces must be less than or equal to `GasKey::MAX_NONCES` (currently suggested as 1024). This is to limit the number of trie operations for a single action. * Increases storage usage of the associated account, * Check that if the permission is a `FunctionCallPermission`, the allowance is `None` (unlimited allowance). * A new `GasKey` entry is added to the trie. * For each nonce ID (from 0 to the number of nonces minus 1), store the default nonce in the trie. * The default nonce is block height * 1e6, the same as the default nonce calculation for access keys. `TransferToGasKey` is verified and executed as follows: * The gas key must exist under the account. * The cost of this action is the amount to fund; it is then verified the same way as a `Transfer` action. * To apply this action, the balance on the gas key is increased by the same amount. `WithdrawFromGasKey` is verified and executed as follows: * The cost of this action is similar to the cost of `Transfer`, however the deposit amount is not included. * The specified gas key must have sufficient balance to perform the transfer from. * To apply this action, the balance on the gas key is decreased by the specified amount, which is credited to the account balance. `DeleteKey` when deleting a gas key is verified as executed as follows: * The gas key must exist under the account. * To apply this action, all relevant trie nodes are deleted, * Decreases storage usage of associated account, * The remaining balance left in the key is **burned**. This prevents an attack where an adversary floods the chain with transactions, then deletes the key to reclaim the balance before the transactions execute. * To protect the user from accidental loss of funds, this action fails if the balance of the key is more than 1 NEAR. * Note that this does not mean that all balance in a gas key is eventually burned; one can withdraw from a gas key using the `WithdrawFromGasKey` action. * To deter creating many keys and then deleting them all at once (which could consume excessive computation in a single chunk), a per-nonce compute cost is charged during execution. Combined with the 1024 max limit on number of nonces, this provides sufficient protection against computational DoS. #### Modifications to existing actions `AddKey` is extended to support a new gas key permission type. Since gas keys and access keys share a namespace, `AddKey` will fail if the public key is already registered as either kind of key. See [Gas Key Actions](#gas-key-actions) for full validation and execution details when adding a gas key. `DeleteKey` can delete both access keys and gas keys. When deleting a gas key, any remaining balance is burned. `DeleteAccount` can succeed even if the account has associated gas keys. * If the sum of gas key balances is less than or equal to 1 NEAR, balances are burnt. * If the sum of gas key balances is greater than 1 NEAR, the action fails, to protect the user against accidental loss. #### Cost of Gas Key actions * Cost of `AddKey` when adding a gas key will be based on the cost of adding a new access key. In addition, appropriate fees will be charged per nonce to account for trie operations. These per-nonce fees are collected during the `AddKey` action. * Cost of `DeleteKey` when deleting a gas key will be based on the cost of deleting an access key. In addition, per-nonce compute costs are charged during execution to deter creating many keys and deleting them in the same chunk. This per-nonce compute cost, combined with the `GasKey::MAX_NONCES` limit on nonces, should provide sufficient protection against computational DoS attacks. * Cost of `TransferToGasKey` and `WithdrawFromGasKey` will be based on `Transfer`, as it is the most similar existing action. `WithdrawFromGasKey` and `TransferToGasKey` do additional trie modifications on sending and on execution. These will be accounted for based on trie operations. As an alternative, we could use the [estimator](https://github.com/near/nearcore/blob/master/docs/architecture/gas/estimator.md) to calculate these fees. However, this is known to not be accurate and ignored in favor of consistency with other fees in recent additions ([example](https://github.com/near/nearcore/issues/14160)). #### Gas Key Transactions `TransactionV1` reuses the fields of `TransactionV0` except replacing the `nonce` field with an enum: ```rust enum TransactionNonce { Nonce { nonce: Nonce }, GasKeyNonce { nonce: Nonce, nonce_index: u16 } } ``` Transactions that specify a `GasKeyNonce` will use gas keys, and transactions that specify `Nonce` will use access keys. The cost of a transaction (as set in [`calculate_tx_cost`](https://github.com/near/nearcore/blob/4fefbdf90c645506beb562ecf87e84f6387aef2f/runtime/runtime/src/config.rs#L330)) will be split into: * **Gas key cost**: `burnt_amount + remaining_gas_amount`. (This includes the gas burnt for converting this tx to a receipt and pre-paid gas for function calls). In case of `WithdrawFromGasKey`, this includes the amount to withdraw from the gas key as well. * **Deposit**: Cost of transaction's actions. The semantics for gas key transactions, at the moment of execution (conversion to receipts), are: * A gas key transaction is valid iff all of the following are true: * The public key corresponds to a valid gas key; * The gas key has enough balance to cover **gas key cost**. * The account balance can cover **deposit**; this includes amounts included in `Transfer` actions. * The nonce ID < total number of nonces for the gas key, and the nonce is a valid nonce for that nonce ID (per the same nonce checks as access key); * When converting the gas key transaction to a receipt, the same logic applies as for access key transactions, except * The gas key cost is deducted from the gas key instead of the account. (Deposit cost is still deducted from account). * The new nonce is written for the specific nonce ID under the gas key. ##### What happens if account cannot cover deposit? Transaction processing is extended with a new failure case: `InvalidTxError::NotEnoughBalanceForDeposit`. A gas key transaction may be valid at submission time (the account has enough balance for the deposit), but by execution time the account may no longer have sufficient balance - for example, if a contract call drains it in between. In this case, the transaction fails, but gas is still charged from the gas key balance. Otherwise, an attacker could intentionally drain their account balance between submission and execution to get free gas key transactions, reintroducing the spam issue this NEP intends to prevent. Note that with honest chunk producers, this new failure case can only occur for gas key transactions. *Note*: As of the current date, `nearcore` ignores failed transactions in processing. After `ProtocolFeature::InvalidTxGenerateOutcomes`, invalid transactions impact the outcome but not the state. Therefore, this NEP also introduces the change that failed transactions may update the state (specifically, to deduct gas fees from the gas key balance). #### Refunds from transactions originating from a gas key This NEP suggests returning refunds to the same balance which pays for the transaction. Therefore [*balance refunds*](https://github.com/near/nearcore/blob/4fefbdf90c645506beb562ecf87e84f6387aef2f/core/primitives/src/receipt.rs#L555-L559) will be issued to the account balance, and [*gas refunds*](https://github.com/near/nearcore/blob/4fefbdf90c645506beb562ecf87e84f6387aef2f/core/primitives/src/receipt.rs#L596-L604) will be issued to the gas key. This is compatible with the existing receipt refunds, which use signer_id to [refund access key allowances](https://github.com/near/nearcore/blob/4fefbdf90c645506beb562ecf87e84f6387aef2f/runtime/runtime/src/actions.rs#L418-L428). As there is no overlap between gas key and access keys, a gas refund can be issued to the gas key's balance (without creating ambiguity if it should refund an access key's allowance instead). *Note*: We only allow `None` allowance for a `FunctionCallPermission` in gas keys. Additionally, this is compatible with `refund_to`, which only [impacts](https://github.com/near/nearcore/blob/4fefbdf90c645506beb562ecf87e84f6387aef2f/core/primitives/src/receipt.rs#L764-L766) balance refunds. ##### Refunds alternatives 1. We could route balance and gas refunds to the account balance. This trades-off user experience for simplicity of implementation and protocol: no specific changes to receipts would be needed, however the user would have to "top-off" the gas key balance more frequently. 2. We could add `ActionV3` and `PromiseYieldV3`, however this requires careful consideration of possible interactions with `Delegate` action and `refund_to`. 3. We could track the gas key for refunds by adding a `Receipt` variant, however this seems a bit out of place (as `Receipt` currently only tracks `predecessor_id`, where `ActionReceipt` tracks `signer_id` and `signer_public_key` i.e, access key). ##### Interactions with VMContext VM execution currently [provides](https://github.com/near/nearcore/blob/136acc3a524575aae1300e26901e664adb521a6f/runtime/runtime/src/actions.rs#L74) the public key of the access key used to originate the transaction in `signer_account_pk`. With gas keys, this may become the public key of a gas key or an access key. As alternative, we could provide more context to the VM, allowing contracts to distinguish gas keys and access keys. Doing so provides a richer context to applications, and trades-off simplicity and future flexibility. Without having concrete use cases or limitations, the additional VM changes seem unnecessary or best left for a future NEP. ### Gas Key Pending Transaction Constraints Unlike access key transactions, gas key transactions are not limited in parallelism; they are only limited by the amount of gas these transactions consume. Specifically, for a gas key $G$, the **gas key cost** of the transactions signed with $G$ in the pending transactions must not exceed the balance of $G$ (according to the state that the pending transactions are based on). This constraint should be good enough to cover cases of adding, funding, removing, or withdrawing from gas keys as well. For adding and funding, we do need the execution to catch up before the newly available balance can be used for pending transactions, which is suboptimal but correct. Contract execution cannot create receipts that withdraw from gas keys, as we are intentionally not adding new promise-creating host functions for them; only gas key transactions can. This intention should be documented, for example where the host functions are defined, so they are not added accidentally in the future. For deletion, balance in gas keys are not refunded, so although subsequent pending transactions may end up failing, the gas committed to those transactions are already burnt, eliminating the opportunity for the aforementioned attack. ### Pending Transaction Queue We would now maintain a new data structure that stores the pending transactions, called the Pending Transaction Queue. Although it's conceptually a queue, it is stored as a collection indexed by (account ID, transaction type), and further indexed by the block hash. Furthermore, the pending transaction queue is stored per shard, not as a single data structure. The contents of the queue is exactly the pending transactions according to the definition above. The constraints are enforced as described above; we reiterate them precisely here. For each account, Let $T_A$ be the set of transactions in the queue signed with any access key of this account, and let $T_G$ be the set of transactions in the queue signed with any gas key of this account. The following constraints must hold true: * If the account has a contract deployed (as of the latest available state), then $|T_A| \le P_{\mathrm {max}}$. * If any transaction $t\in T_A\cup T_G$ contains a `DeployContract` action, then $|T_A|=\{t\}$. In other words, deploying a contract cannot be done in parallel with any access key transactions. The same applies to `UseGlobalContract`, `DeterministicStateInit`, and `Delegate` with inner deploy-like actions. * The sum of **total costs** of all transactions in $T_A$ plus the sum of **deposit costs** of all transactions in $T_G$ does not exceed the account balance. * For each gas key $g$, the sum of the **gas key costs** of all transactions signed with $g$ does not exceed the balance of $g$. * Importantly, these costs can be calculated from the gas price and inspecting the actions contained in the transaction (does not depend on state or execution). ![Transaction Flow Overview](assets/nep-0611/pending-tx-flow.svg) The constraints are maintained at the time of chunk production: when producing a chunk, we only accept transactions that would maintain these constraints. For limiting gas key transactions, we always query the balance of the gas key from the executed state (as opposed to storing it in the queue). To compute the pending transaction queues (one for each tracked shard) for a new block, * Start from the queue from the previous block; * Subtract the transactions that are included in each newly certified block; * Add new transactions included in this block. Note: The pending transaction queue is relevant for the SPICE project, where transactions may be pending for multiple blocks before execution. Without SPICE, as mentioned above, the pending transaction queue is considered empty because transactions are validated at inclusion time. ### Trie storage To facilitate the above design, gas keys must be stored in the trie. They are stored under the same `TrieKey` as a normal access key: ```rust AccessKey { account_id: AccountId, public_key: PublicKey, } // uses col::ACCESS_KEY ``` Then, gas keys can be stored as a specific kind of a newly added `AccessKeyPermission` variant: ```rust pub struct AccessKey { pub nonce: Nonce, pub permission: AccessKeyPermission, } pub enum AccessKeyPermission { // Already exists FunctionCall(FunctionCallPermission), FullAccess, // Newly added GasKeyFunctionCall(GasKeyInfo, FunctionCallPermission), GasKeyFullAccess(GasKeyInfo), } ``` Individual nonces are stored under the following `TrieKey` as `u64` values: ```rust GasKeyNonce { account_id: AccountId, public_key: PublicKey, nonce_index: u16 } // also uses col::ACCESS_KEY ``` ### RPC and `StateChanges` modifications Gas keys are returned as other access keys using `view_access_key_list`, and `view_access_key` queries. They will show with the newly introduced `AccessKeyPermission` enum variants that contain gas key info (i.e., balance and number of nonces). New queries will be introduced for gas key nonces, such as `view_gas_key_nonces`. As an alternative, we could gather and return all nonces as part of existing queries. However, this changes the return types of existing API and potentially makes them more expensive. ### Impact to Existing Protocol without SPICE As mentioned above, this NEP applies with or without SPICE. The impact to the existing protocol is purely positive: * Normal access keys are never restricted, as the set of pending transactions is always empty. * Gas keys allow programmatic transaction senders to more easily manage multiple nonces. Rather than requiring multiple access keys to be created, they just need to create a single gas key. The implementation of gas keys before SPICE also allows programmatic users to migrate to using gas keys, preparing them for when SPICE is launched. ## Security Implications ### DoS Attack Prevention The primary security benefit of this NEP is preventing the DoS attack vector described in the Motivation section. By restricting access key parallelism for contract accounts and requiring gas keys to be pre-funded, we ensure that attackers cannot flood the blockchain with transactions that appear valid at inclusion time but become invalid during execution. All accepted transactions have committed resources that can be charged for gas consumption. ### Balance Burning on Gas Key Deletion When a gas key is deleted, any remaining balance is burned rather than refunded. This prevents an attack where an adversary attempts to include non-fee paying transactions on chain by submitting many transactions yet reclaiming the balance on deletion. ## Reference Implementation * [Initial implementation of gas key trie modifications](https://github.com/near/nearcore/pull/13687) * [Initial implementation of gas key actions](https://github.com/near/nearcore/pull/14532) * [Work in progress implementation of actions and refund receipts](https://github.com/near/nearcore/pull/14521) ## Alternatives One alternative to the gas key design is to introduce a `SenderReservedBalance`: under this model, the sender's account reserves a portion of its NEAR balance specifically for gas on future transactions. The runtime would check that the sender has enough reserved funds to cover both the gas cost and the required minimum balance, thereby enforcing validity without requiring additional trie keys. While initially it seems simpler, this approach comes with significant and subtle drawbacks: * Contract execution can lead to deletion of access keys. This creates a problem for sharing a single reserved gas balance between multiple access keys. The specific case to consider is when an `Account` does not have sufficient balance to pay for a transaction (i.e., `InvalidTxError::NotEnoughBalanceForDeposit`). This can occur if the contract transferred away its balance in a prior block. If the contract also programmatically deletes the signing key, the executor cannot distinguish between the `NotEnoughBalanceForDeposit` scenario (which should be charged gas) and an unauthorized transaction using an incorrect key (which should not be charged), based on the current state alone. * Modifies the semantics of the account's main balance, requiring checks that contract execution doesn't deplete the balance lower than `SenderReservedBalance`, potentially breaking existing assumptions about how balance operates. In contrast, the gas keys solution solves the deleted key problem, does not modify semantics of existing account balance, and also provides improved user experience for concurrent transaction issuance via `nonce_index`. ### Storing entire vector nonce of nonces in a single trie key As an alternative to having a separate trie key for each nonce, we could store all the nonces for a given gas key under a single trie key in a vector. With the vector implementation, the number of trie reads will decrease from 2 to 1, however the amount of data read from the value will be larger and depend on the number of nonces. This means we would have to charge users using more nonces a higher fee not only during addition and deletion, but also when using the gas key to sign a transaction. The larger amount of data read must also be included in state witnesses. Additionally, in the future it may be easier to reason about processing transactions in parallel when the trie keys for different nonces do not overlap. ### Alternate design for transaction parallelism (NEP-522) [NEP-522](https://github.com/near/NEPs/pull/522/files) describes an approach to transaction deduplication based on random nonces and tracking state of recently seen transaction hashes in the trie. This NEP takes an approach which does not require writing to the trie. Additionally, it does not require maintaining a data structure with historical blocks. ### Alternate possibilities for actions * An earlier version of this NEP used separate `AddGasKey` and `DeleteGasKey` actions. In this alternative, it would have been possible to issue a refund of the remaining gas balance on delete to the account. However, using the existing key operations provides better UX, by allowing contracts to manage keys. ### Alternative trie storage scheme As an alternative, gas keys may be stored under a separate trie column using the following `TrieKey`: ```rust GasKeyNonce { account_id: AccountId, public_key: PublicKey, nonce_index: Option } // uses new column col::GAS_KEY ``` In this scheme, the gas key data is stored with `nonce_index: None` and for the individual nonces are stored by with `nonce_index: Some(index)`. The advantage of this alternative is simplicity and a more additive implementation. The downside is processing refunds, and adding new access or gas keys would incur an additional trie read (which also consumes space in the state witness). While we can consider adding keys relatively uncommon, gas refunds are fairly common. This change would require 3 instead of 2 trie accesses which increases the impact of these receipts on state witness size by 50%. ## Consequences ### Positive * Enables implementing SPICE without potential for users to create unbounded spam. * Enhanced nonce management enabling parallel transaction submission for sophisticated users. ### Neutral * N/A ### Negative * Adds complexity to protocol and implementation. ### Backwards Compatibility * Adding `TransactionV1` does not change the semantics for `TransactionV0`. Users can continue to submit their transactions as such. * Adding new keys to the trie may require modifications for downstream parsers, indexers. * Existing use cases may assume they can submit an unbounded number of in-flight transactions. As changes only impact accounts with contracts, the impact of this is assumed to be low. * `VMContext` exposes the access key's public key to the contracts. Going forward, this could be a public key corresponding to either a gas key or a public key. ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0616.md ================================================ --- NEP: 616 Title: Deterministic AccountIds Authors: Arseny Mitin Status: Approved DiscussionsTo: https://github.com/near/NEPs/issues/607#issuecomment-3067136865 Type: Protocol Replaces: 605 Version: 1.0.0 Created: 2025-07-24 LastUpdated: 2025-10-06 --- ## Summary With [global contracts](https://github.com/near/NEPs/blob/master/neps/nep-0591.md) making code reuse cheaper, it's now more feasible to deploy multiple instances of the same contract code across different accounts. However, this creates challenges around permissions, storage costs, refunds, and code verification. This NEP addresses these concerns by introducing a new [backwards-compatible](#backwards-compatibility) `AccountIds` which are [deterministically derived](#deterministic-accountids) from contract initialization code and state, enabling for truly sharded contract designs. ## Motivation With global contracts introduced in [NEP-591](https://github.com/near/NEPs/blob/master/neps/nep-0591.md), it's now much cheaper to reuse the same code across different contracts in terms of storage costs. This opens up a door for sharded contract designs where each instance needs to be deployed on a separate account. However, when designing sharded contracts, following concerns naturally come up: * Who is allowed to perform different actions (deploy, call, delete, etc.) on different sharded contract instances? * Who pays for storage costs of deploying new sharded contract instances? * How does it get refunded if the contract turned out to be already deployed? * How can a contract verify that the caller was executing the same code? This NEP proposes enabling sharded contracs design by introducing fully backwards-compatible [deterministic AccountIds](#deterministic-accountids) and new protocol-level primitives required to deploy such contracts, addressing concerns above. ## Specification ### Deterministic AccountIds The core idea is to have contract account ids **deterministically** derived from its initialization code and state. Let's first define [`StateInit`](https://github.com/mitinarseny/near-sdk-rs/blob/50eb6e68544b75288145f7f0d07068a482b9e3fc/near-sdk/src/state_init.rs#L15-L24) struct: ```rust /// Initialization state for non-existing contract with /// deterministic account id, according to NEP-616. #[derive(BorshSerialize, BorshDeserialize)] pub enum StateInit { V1(StateInitV1), } #[derive(BorshSerialize, BorshDeserialize)] pub struct StateInitV1 { /// Code to deploy pub code: ContractCode, /// Optional key/value pairs to populate to storage on first initialization pub data: BTreeMap, Vec>, } /// Code to deploy for non-existing contract. #[derive(BorshSerialize, BorshDeserialize)] pub enum ContractCode { /// Reference global contract's code by its hash GlobalCodeHash(CryptoHash), /// Reference global contract's code by its [`AccountId`] GlobalAccountId(AccountId), } ``` Now, let's [derive](https://github.com/mitinarseny/near-sdk-rs/blob/50eb6e68544b75288145f7f0d07068a482b9e3fc/near-sdk/src/state_init.rs#L189-L203) `AccountId` deterministically as `"0s" .. hex(keccak256(state_init)[12..32])`: ```rust impl StateInit { /// Derives [`AccountId`] deterministically, according to NEP-616. pub fn derive_account_id(&self) -> AccountId { let serialized = borsh::to_vec(self).unwrap_or_else(|| unreachable!()); format!("0s{}", hex::encode(&env::keccak256_array(&serialized))[12..32])) .parse() .unwrap_or_else(|_| unreachable!()) } } ``` Such schema is fully backwards-compatible with existing `AccountId` types. It looks similar to existing implicit Eth addresses but we intentionally use a different prefix, so it's possible to distinguish between different kinds of accounts in runtime and apply [different rules](https://github.com/near/nearcore/blob/c5225bacaad88de3574656333bd312464a90fb6a/core/parameters/src/cost.rs#L651-L681) for estimating gas and storage costs. #### Versioning In order to provide for future protocol upgrades, [`StateInit`](#deterministic-accountids) should be a versioned enum, while the version can be thought as part of account's initialization state. So, different versions would result in different derived account ids. > Contract implementations would use specific `StateInit` versions for > [initialization](#stateinit-action) / [verification](#address-verification) of other contracts. > The information about which version to use can be hardcoded into contract's code, stored > internally or forwarded in function call arguments, depending on its business logic. ### Address Verification If a contract needs to authenticate predecessor and verify that it was executing the [same code](#other-host-functions), then it can simply [verify](https://github.com/mitinarseny/near-sdk-rs/blob/50eb6e68544b75288145f7f0d07068a482b9e3fc/examples/sharded-fungible-token/wallet/src/lib.rs#L162-L167) that it matches `AccountId` [deterministically derived](#deterministic-accountids) from its expected initialization state: ```rust // construct expected initialization state let state_init = StateInit::V1(StateInitV1 { code: env::current_contract_code(), data: Self::init_state_for(params...), }); // verify require!(env::predecessor_account_id() == state_init.derive_account_id(), "not allowed"); ``` Moreover, contracts might store references to other global contracts, so that such verification logic [can](https://github.com/mitinarseny/near-sdk-rs/blob/d9950de34084a352bd58bbb0872e686fc7d40d04/examples/sharded-fungible-token/ft2sft/src/lib.rs#L131-L135) be applied to all deterministic accounts, not just the predecessor, enabling for infinite composability between different deterministic and non-deterministic contracts. ### `StateInit` action The main issue with sharded design is that there is no synchronous way for a contract to check for existence of another contract before calling its methods. So, the only way to ensure its existence is to optionally deploy and initialize it before proceeding to actual function calls. However, since we don't know in advance whether the contract exists or needs to be deployed first, we need to reserve some amount to cover storage costs of potential deployment. Let's define a new [`StateInit` action](https://github.com/mitinarseny/near-sdk-rs/blob/50eb6e68544b75288145f7f0d07068a482b9e3fc/near-sdk/src/environment/env.rs#L1237-L1276) and a host function for it: ```rust /// Optionally, deploy the contract with deterministic account id and /// initialize its storage with [`StateInit`](crate::StateInit) according to /// [NEP-616](https://github.com/near/NEPs/pull/616). /// /// Note that the `receiver_id` of the [`Promise`](crate::Promise) MUST be /// equal to [`state_init.derive_account_id()`](StateInit::derive_account_id). /// Otherwise, this promise will fail. /// /// If non-zero, `amount` will be immediately subtracted from current /// account's balance as a "reserve" for storage costs. /// /// If the receiving contract is in `noexist` or `uninit` state by the time /// this action gets executed: /// * The contract is deployed and initialized with [`state_init`](crate::StateInit) /// * [`state_init.storage_cost`](crate::StateInit::storage_cost) is /// subtracted from attached `amount`. If the contract was in `uninit` /// state and had non-zero balance, then its balance is used first and only /// the missing part required for covering storage costs will subtracted /// from the attached `amount`. /// * The contract state is marked as `active`. /// * The remaining amount is transferred back to predecessor or /// [refund_to](promise_set_refund_to) if set. /// /// If the contract was already in `active` state, then full `amount` is /// refunded. See [`promise_set_refund_to`]. pub fn promise_batch_action_state_init( promise_index: PromiseIndex, state_init: &LazyStateInit, // may be already serialized amount: NearToken, ); ``` **Note** that [zero-balance accounts](https://github.com/near/NEPs/blob/master/neps/nep-0448.md) limits do apply here: if the contract's [`StateInit`](#deterministic-accountids) requires no more than 770 bytes of storage then no NEAR tokens are required for storage staking. Instead, storage costs will be compensated by higher gas costs. ### Permissions Since the contract address is derived deterministically from its initialization code and state, then **anyone** can deploy such contract via [`StateInit` action](#stateinit-action) and pay for it. All existing rules for current account model apply to deterministic accounts, too. In particular, only the contract itself is allowed to perform these actions on himself: * `CreateAccount`: for sub-accounts like `sub.0s123...` * `DeleteAccount` * `DeployGlobalContract` * `UseGlobalContract` * `DeployContract`, `AddKey`, `DeleteKey`: doesn't make sense and should be discouraged by implementations, but still allowed * `Stake` ### Account State There is nothing preventing accounts with deterministic ids from accepting incoming native transfers before they were deployed and initialized via [`StateInit` action](#stateinit-action). This can be used to build some mechanics where one wants to sponsor the deployment but doesn't bother to actually initialize it. To support this case, we say that each deterministic account can be in one of following states: * `nonexist`: there were no accepted receipts on this account, so it doesn't have any data (or the contract was deleted). Initially all deterministic accounts are in this state. * `uninit`: account has some data, which contains balance and meta info. An account enters this state, for example, when it was in a `nonexist` state, and another account sent native transfer to it. * `active`: account has contract code deployed, persistent data and balance. An account enters this state when it was in `nonexist` or `uninit` state and there was an incoming [`StateInit` action](#stateinit-action). Note, that to be able to deploy this account, [`state_init.derive_account_id()`](#deterministic-accountids) must be equal to its `AccountId` ### Refunds Currently, refund of total attached deposit to a failed receipt is sent via plain transfer to its predecessor. In a sharded design, the contract creating a promise is not necessarily the one who would like to receive a refund in case of failure or unused amount (e.g. [`StateInit` action](#stateinit-action)). So, we need a way to route these refunds to intended beneficiaries, which can be implementation-specific and depend on inner logic of a smart-contract. Unfortunately, Near protocol doesn't provide a way for contracts to execute some code upon receiving a native transfer if it wasn't attached deposit to a function call. Moreover, when scheduling callbacks, the runtime doesn't guarantee that the refund for a failed receipt will come before callback execution, since the refund is processed as a new independent receipt: ```mermaid --- title: "Regular Refunds: can't forward in callback" --- sequenceDiagram participant A as alice.near participant B as bob.near participant C as carol.near A ->> B: Promise::new("bob.near")
.function_call(..., attached_deposit) activate B B -x+ C: Promise::new("carol.near")
.function_call(..., attached_deposit) B -->> B: .callback() deactivate B Note over C: Failed C -->>+ B: failed receipt B -->>- A: refund??? C -)- B: refund: attached_deposit ``` To remove this limitation let's add an optional `refund_to` field to [`ActionReceipt`](https://github.com/near/nearcore/blob/685f92e3b9efafc966c9dafcb7815f172d4bb557/core/primitives/src/receipt.rs#L640-L659) and a host function for setting it: ```rust /// Set a different [`AccountId`] instead of current one to which refunds /// should go for all failed (e.g. [function_call](promise_batch_action_function_call_weight)) /// or unused (e.g. [state_init](promise_batch_action_state_init)) deposits in /// the created receipt. pub fn promise_set_refund_to( promise_idx: PromiseIndex, account_id: &AccountId, ); ``` So, when `refund_to` is used together with [`StateInit` action](#stateinit-action) the flow can look like: ```mermaid --- title: Custom Refunds via .refund_to("alice.near") --- sequenceDiagram participant A as alice.near participant B as bob.near participant C as 0s123... A ->> B: Promise::new("bob.near")
.function_call(..., attached_deposit) activate B B ->>+ C: Promise::new("0s123...")
.refund_to("alice.near")
.state_init(..., amount)
.function_call(..., attached_deposit) B -->> B: .callback() deactivate B alt uninit, success activate B C -->> B: success data receipt deactivate B else already init, success activate B C -->> B: success data receipt deactivate B C -) A: refund: state_init.amount else failed activate B C -->> B: failed receipt deactivate B C -) A: refund: state_init.amount + attached_deposit end deactivate C ``` It also makes sense to let smart-contracts know in the runtime which refund account was set for the current receipt by its predecessor via `promise_set_refund_to()`, so that they can use it according to its business logic instead of duplicating it in a FunctionCall arguments. So, we might need another host function for that: ```rust /// Returns the [`AccountId`] that was set for the current receipt by its /// predecessor via [`promise_set_refund_to()`], or [`predecessor_account_id()`] otherwise. pub fn refund_to_account_id() -> AccountId; ``` Note that setting `refund_to` to a non-existing named account id will result into burning refunded NEAR tokens. > In fact, regular refunds to predecessor also suffer from the same problem: > predecessor is not always guaranteed to exist as it could have been deleted. > You can [burn](https://github.com/near/nearcore/blob/685f92e3b9efafc966c9dafcb7815f172d4bb557/runtime/runtime/src/lib.rs#L901-L908) > NEAR today by using `DeleteAccount` action with non-existing `beneficiary_id`. ### Other host functions We would also need to implement a couple of other trivial host functions, which are required for contracts to function properly: ```rust /// Returns code deployed on current contract being executed. /// For now, only references to globally deployed contracts are supported. /// /// Note: gas cost of this for globally deployed contracts should be /// relatively small, since it would only return `ContractCode::GlobalCodeHash(code_hash)` /// or `ContractCode::GlobalAccountId(account_id)`. pub fn current_contract_code() -> ContractCode; /// If the current function is invoked by a callback, we can access the /// length of execution results of the promises that caused the callback. /// It can be used to prevent out-of-gas failures when reading too long /// execution result via [`promise_result()`]. pub fn promise_result_length(result_idx: u64) -> Result; // Currently constants, but better to be exposed as host functions, // so that changing the protocol config doesn't brick existing contracts. pub fn storage_byte_cost() -> NearToken; pub fn storage_num_bytes_account() -> StorageUsage; pub fn storage_num_extra_bytes_record() -> StorageUsage; ``` ## Reference Implementation Here is a [reference implementation](https://github.com/near/near-sdk-rs/pull/1376) of Sharded Fungible Token contracts with deterministic account ids, including FT<->sFT adaptors and optional governance functionality (useful for stablecoins), along with definition of [`StateInit`](#deterministic-accountids) structs and required [host functions](#stateinit-action). All changes are [backwards-compatible](#backwards-compatibility) with existing contracts and account ids. ## Security Implications ### 3-stage Upgrades > This concern is valid for all global contracts introduced in [NEP-591](https://github.com/near/NEPs/blob/master/neps/nep-0591.md). > However, it's worth to highlight it here for better clarity. If the upgrade of a globally deployed contract, which is used for some sharded interactions, introduces a breaking change in its internal ABI (i.e. used for XCCs only between referencing contracts), then it can be done safely only via 3-stage upgrade process: 1. Deploy a "pre-upgrade" version that still uses the old ABI but **can** understand the new one. Wait for it to be fully distributee across shards. 2. Deploy a "do-upgrade" version that starts to use new ABI, but still **can** understand the old one. Wait for it to be fully distributee across shards. 3. Deploy a "post-upgrade" version that cleans up the legacy code and now **only** uses and understands the new ABI. Even if the implementation always passes the contract version `v` as an argument for all interactions between different instances of this global contract, then it would still not help the contract to understand how to handle the new version, because the new code just hasn't reach his shard yet. The most what the contract can do upon receiving "unknown version" is to panic, which can break others' assumptions about its standard behavior. ## Alternatives ### Sharded Contexts There is an existing proposal described in [NEP-605](https://github.com/near/NEPs/pull/605) that follows a more OS-like approach by introducing a concept of non-root accounts and "sharded contexts". The main advantage of current proposal over "sharded contexts" is that there is no isolation between sharded and non-sharded contracts: any deterministic account is allowed to freely interact with any non-deterministic one and vice versa. As a result, deterministic accounts enable for truly sharded contract designs with infinite composability, while "sharded contexts" approach suffers from high complexity, less composability and inevitable bottlenecks for routing calls between root/non-root accounts. However, non-root accounts are designed to live on the same shard as their root accounts, which is a step towards [synchronous execution](https://github.com/near/NEPs/issues/497) but only within the boundaries of a single root account. At the same time, this can be seen as limitation for future scaling and dynamic resharding. With deterministic accounts approach, this might be achievable by having [`closeTo`](#co-location) analog. ### Prior Art This proposal is highly inspired by [deterministic addresses](https://docs.ton.org/v3/documentation/smart-contracts/addresses#account-id) in TON blockchain. However, we still needed to adapt their definitions to Near specifications: * Unlike TVM, Near doesn't support [message bouncing](https://docs.ton.org/v3/documentation/smart-contracts/transaction-fees/forward-fees#message-bouncing), but instead allows to schedule callbacks, which gives more control over handling of chained cross-contract calls. * Near doesn't provide a way for contracts to execute some code upon receiving a native transfer if it wasn't attached deposit to a function call. See [refunds](#refunds). * TVM doesn't differentiate between gas and attached deposit, while in Near they are not coupled, which removes some complexities. ## Future possibilities ### Sharded DeFi protocols Here is how deterministic accounts can be used to build [DEX](https://docs.dedust.io/docs/swaps) or [Lending Protocol](https://github.com/evaafi/contracts/blob/40a0e8bb32f88df8e09def536371192a824d1c3d/diagrams/liquidate-for-jetton.svg) with sharded design in mind. ### Wallet Extentions If we allow deterministic accounts to handle external messages by verifying signatures and tracking nonces by themselves instead of relying on [access keys](https://nomicon.io/DataStructures/AccessKey), then it would open up doors for upgradable [wallet implementations](https://docs.ton.org/v3/documentation/smart-contracts/contracts-specs/wallet-contracts) with [arbitrary plugins](https://docs.ton.org/v3/documentation/smart-contracts/contracts-specs/wallet-contracts/#wallet-v5) such as 2FA, social recovery, gasless transactions and [more](https://github.com/ton-blockchain/wallet-contract-v5/blob/main/README.md#suggested-extensions). This can be seen as an alternative to [contract namespaces](https://gov.near.org/t/proposal-account-extensions-contract-namespaces/34227). ### Co-location It might be possible to implement some analog of [`closeTo`](https://docs.ton.org/v3/documentation/smart-contracts/tolk/tolk-vs-func/create-message#sharding-deploying-close-to-another-contract) for deploying "close to" another contract based on shard depth (i.e. fixed prefix length). ### Storage Rent vs Fixed Storage Staking We can move from fixed storage staking fees to a more sustainable storage rent approach by adding new [account state](#account-state): > * `frozen`: account cannot perform any operations, this state contains only > two hashes of the previous state (code and state respectively). When an > account's storage charge exceeds its balance, it goes into this state. To > unfreeze it, you can send an internal message with state_init and code which > store the hashes described earlier and some NEAR. It can be difficult to > recover it, so you should not allow this situation. There is a project to > unfreeze the account, which you can find [here](https://unfreezer.ton.org). ## Consequences ### Positive * Deterministic AccountIds enable for truly **sharded** contract designs with minimalistic implementations without sacrificing composability. * Proposed [deterministic accounts](#deterministic-accountids) are fully [backwards-compatible](#backwards-compatibility) with existing ones. * Knowledge of `AccountId` (e.g. predecessor) combined with its expected initialization state is enough to [verify](#address-verification) it was executing the exact code. ### Neutral * Deterministic accounts are allowed to freely interact directly with non-deterministic ones and vice versa. * Existing contracts can be upgraded to start using [`StateInit` action](#stateinit-action) and, thus, gain the power to automatically deploy non-existing deterministic accounts. * No step towards [synchronous execution](https://github.com/near/NEPs/issues/497). Though, latency might be improved if we add support for [`closeTo`](#co-location) analog. * Indexers for sharded contracts might need to be [stateful](#stateful-indexers) in order to keep track of newly created contracts. ### Negative \- ### Backwards Compatibility Proposed [deterministic AccountIds](#deterministic-accountids) are fully **backwards-compatible** with existing ones. Deterministic accounts are allowed to freely interact directly with non-deterministic ones and vice versa. Existing contracts can be upgraded to start using [`StateInit` action](#stateinit-action) and, thus, gain the power to automatically deploy non-existing deterministic accounts. ### Stateful Indexers > This concern is valid for any sharded contracts design. > However, it's worth to highlight it here for better clarity. For some sharded contracts with deterministic account ids it doesn't make sense to emit events (e.g. `sft_transfer`) as it simply wouldn't bring any value for indexers. Even if they do emit these events, indexers still **need to track** outgoing [`StateInit` actions](#stateinit-action) to not-yet-existing sharded contracts, which will emit these events in the future. However, to properly track these cross-contract calls they would need parse function names (e.g. [`sft_transfer()`](https://github.com/mitinarseny/near-sdk-rs/blob/50eb6e68544b75288145f7f0d07068a482b9e3fc/near-contract-standards/src/sharded_fungible_token/wallet.rs#L26-L55)) and their args, while this information combined with receipt status already contains all necessary info for indexing. ## Changelog ### 1.0.0 - Initial Version #### Benefits * Deterministic AccountIds enable for truly **sharded** contract designs with minimalistic implementations without sacrificing composability. * Proposed [deterministic accounts](#deterministic-accountids) are fully [backwards-compatible](#backwards-compatibility) with existing ones. * Knowledge of `AccountId` (e.g. predecessor) combined with its expected initialization state is enough to [verify](#address-verification) it was executing the exact code. #### Concerns | # | Concern | Resolution | Status | | --: | :------ | :--------- | -----: | | 1 | Deterministic AccountIds `0s123...` are not human-readable and can be seen as a downside when compared to developer-friendly Named AccountIds. | Human-redable account names are mostly appealing for UX and can be implemented via NFTs (similar to [ENS](https://ens.domains)). | Resolved | ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0621.md ================================================ --- NEP: 621 Title: Vault NEP Authors: JY Chew , Lee Hoe Mun , Wade , Steve Kok Status: Approved DiscussionsTo: https://github.com/nearprotocol/neps/pull/0000 Type: Contract Standard Requires: 141 Version: 1.0.0 Created: 2025-08-08 LastUpdated: 2025-08-08 --- ## Summary This NEP proposes a standardized interface for implementing vault contracts on the NEAR Protocol, drawing inspiration from the ERC-4626 standard widely used on Ethereum. A vault contract allows users to deposit an underlying fungible token (FT) into the vault, in exchange for which the vault issues shares that represent proportional ownership of the vault’s assets. The underlying asset could be any NEP-141 compliant fungible token, such as a stablecoin or yield-bearing token. When deposited, the vault mints new shares to the depositor based on the current exchange rate between the vault’s total assets and total shares in circulation. Conversely, when a user redeems shares, the vault burns those shares and returns the equivalent amount of the underlying asset to the user. The issued shares themselves are also NEP-141 compliant fungible tokens, enabling them to be freely transferred between accounts or traded on decentralized exchanges (DEXs). This compatibility allows vault shares to be integrated into broader DeFi ecosystems, enabling use cases such as collateral in lending protocols, liquidity provision, or composable yield strategies. By standardizing the vault interface, this NEP aims to improve interoperability, reduce integration costs, and encourage consistent, secure practices for vault implementation across the NEAR ecosystem. ## Motivation Vault contracts are a fundamental building block in modern DeFi, enabling users to pool assets for yield generation, liquidity provision, or other strategies while receiving tokenized shares that represent their proportional ownership. However, without a standardized interface, each vault implementation on NEAR may expose different method names, return formats, and accounting mechanisms, creating unnecessary friction for developers, integrators, and auditors. A consistent vault standard, inspired by ERC-4626, would provide multiple benefits: - Interoperability – Wallets, DEXs, lending protocols, and other DeFi applications can integrate with any compliant vault without custom logic for each implementation. - Reduced Integration Costs – Developers and projects save time and resources by building once against the standard interface rather than creating one-off integrations. - Ecosystem Growth – Standardized vaults make it easier for new projects to leverage existing liquidity and composability, accelerating adoption across the NEAR DeFi ecosystem. By introducing this NEP, we aim to align vault design on NEAR with proven best practices from other blockchain ecosystems while optimizing for the unique features and requirements of NEP-141 fungible tokens. ## Specification ### Contract Interface The contract should implement the VaultCore trait. ```rust /// Specification for a fungible token vault that issues NEP-141 compliant shares. /// /// A FungibleTokenVault accepts deposits of an underlying NEP-141 compliant asset /// and issues NEP-141 compliant "shares" in return. These shares can be transferred /// and traded like any other NEP-141 token. /// /// This trait extends: /// - [`FungibleTokenCore`] to provide NEP-141 functionality for shares. /// - [`FungibleTokenReceiver`] to receive the underlying NEP-141 assets pub trait VaultCore: FungibleTokenCore + FungibleTokenReceiver { // ---------------------------- // Asset Information // ---------------------------- /// Returns the [`AccountId`] of the underlying asset token contract. /// /// ERC4626 original function name: asset() /// /// The underlying asset **must** be NEP-141 compliant. /// Implementations should store this as an immutable configuration value. fn asset_contract_id(&self) -> AccountId; /// Returns the total amount of underlying assets represented by all shares in existence. /// /// ERC4626 original function name: totalAssets() /// /// **Important:** /// - Represents the vault's *total managed value*, not just assets held in the contract. /// - If assets are staked, lent, swapped, or deployed elsewhere, this should return /// an **estimated total equivalent value**. /// - Must be denominated in the same units as [`Self::asset_contract_id`]. /// - If the vault applies any deposit or withdrawal fees, `total_asset_amount` must reflect the net value of /// all shares, i.e. the aggregate worth of the vault’s holdings after deducting all applicable fees. fn total_asset_amount(&self) -> U128; // ---------------------------- // Conversion Helpers // ---------------------------- /// Converts an amount of underlying assets to the equivalent number of shares. /// /// ERC4626 original function name: convertToShares(uint256 assets) /// /// This is a **purely view-only estimation** that: /// - Does not update state. /// - Ignores user-specific constraints such as deposit limits or fees. /// /// # Panics / Fails /// - The `convert_to_shares` method must never panic. /// It should always return the corresponding amount of shares for the given assets, /// or `0` if conversion is not possible. /// /// See also: [`Self::preview_deposit_shares`] for a version that accounts for limits and fees. fn convert_to_shares(&self, asset_amount: U128) -> U128; /// Converts an amount of shares to the equivalent amount of underlying assets. /// /// ERC4626 original function name: convertToAssets(uint256 shares) /// /// This is a **purely view-only estimation** that: /// - Does not update state. /// - Ignores withdrawal restrictions, fees, or penalties. /// /// # Panics / Fails /// - The `convert_to_asset_amount` method must never panic. /// It should always return the corresponding amount of assets for the given shares, /// or `0` if conversion is not possible. /// /// See also: [`Self::preview_redeem_amount`] for a version that accounts for real-world constraints. fn convert_to_asset_amount(&self, shares: U128) -> U128; // ---------------------------- // Deposit / Redemption Limits // ---------------------------- /// Returns the maximum amount of underlying assets that `receiver_id` can deposit. /// /// ERC4626 original function name: maxDeposit(address receiver) /// /// This may depend on: /// - Vault capacity. /// - User-specific limits. /// - Current on-chain conditions. /// /// # Panics / Fails /// - The `max_deposit_amount` method must never panic. /// It should return the maximum amount of assets that can be deposited for the given account, /// or `0` if deposits are not currently allowed. /// /// Implementations should return `U128::MAX` to signal "unlimited" deposits. fn max_deposit_amount(&self, receiver_id: AccountId) -> U128; /// Simulates depositing exactly `assets` into the vault and returns the number of shares /// that would be minted to the receiver. /// /// ERC4626 original function name: previewDeposit(uint256 assets) /// /// Differs from [`Self::convert_to_shares`] by accounting for: /// - Per-user deposit limits. /// - Protocol-specific deposit fees. /// /// # Panics / Fails /// - The `preview_deposit_shares` method must never panic. /// It should return the number of shares that would be minted for the given deposit amount, /// or `0` if deposits are not currently allowed. /// fn preview_deposit_shares(&self, asset_amount: U128) -> U128; /// Returns the maximum number of shares that `receiver_id` can mint. /// /// ERC4626 original function name: maxMint(address receiver) /// /// This may depend on: /// - Vault capacity. /// - User-specific limits. /// - Current on-chain conditions. /// /// # Panics / Fails /// - The `max_mint_shares` method must never panic. /// It should return the maximum number of shares that can be minted for the given account, /// or `0` if minting is not currently allowed. /// /// Implementations should return `U128::MAX` to signal "unlimited" minting. fn max_mint_shares(&self, receiver_id: AccountId) -> U128; /// Simulates minting exactly `shares` and returns the amount of underlying assets /// that would be required. /// /// ERC4626 original function name: previewMint(uint256 shares) /// /// Differs from [`Self::convert_to_asset_amount`] by accounting for: /// - Per-user minting limits. /// - Protocol-specific minting fees. /// /// Useful for frontends to estimate the cost of minting shares. /// /// # Panics / Fails /// - The `preview_asset_amount_required_to_mint_shares` method must never panic. /// Instead, it should return the maximum possible mint amount, or `0` if minting is currently not permitted. fn preview_asset_amount_required_to_mint_shares(&self, shares: U128) -> U128; /// Returns the maximum number of shares that `owner_id` can redeem. /// /// ERC4626 original function name: maxRedeem(address owner) /// /// This may depend on: /// - The owner's current share balance. /// - Vault withdrawal restrictions. /// - Lock-up periods or cooldowns. /// /// # Panics / Fails /// - The `max_redeem_shares` method must never panic. /// It should return the maximum number of shares that can be redeemed by the given account, /// or `0` if redemption is not currently allowed. /// /// Implementations should return `0` if redemptions are currently disabled for the owner. fn max_redeem_shares(&self, owner_id: AccountId) -> U128; /// Returns the maximum amount of assets that `owner_id` can withdraw. /// /// ERC4626 original function name: maxWithdraw(address owner) /// /// This may depend on: /// - The owner's share balance. /// - Current vault liquidity. /// - Withdrawal limits or cooldowns. /// /// # Panics / Fails /// - The `max_withdraw_amount` method must never panic. /// It should return the maximum amount of assets that can be withdrawn by the given account, /// or `0` if withdrawals are not currently allowed. /// /// Implementations should return `0` if redemptions are currently disabled for the owner. fn max_withdraw_amount(&self, owner_id: AccountId) -> U128; // ---------------------------- // Redemption Operations // ---------------------------- /// Redeems `shares` from the caller in exchange for the equivalent amount of underlying assets. /// /// ERC4626 original function name: redeem(uint256 shares, address receiver, address owner) /// /// - If `receiver_id` is `None`, defaults to sending assets to the caller. /// - Burns the caller's shares. /// - Returns the exact amount of assets redeemed. /// /// # Slippage /// - To forcefully redeem shares without accounting for slippage, the user should set `min_amount_out` to `0`. /// - To protect against slippage, the user should specify a reasonable `min_amount_out`. /// /// # Panics / Fails /// - If the caller's share balance is insufficient. /// - If withdrawal limits prevent the redemption. /// /// See also: [`Self::preview_redeem_amount`]. fn redeem(&mut self, shares: U128, min_amount_out: U128, receiver_id: Option) -> PromiseOrValue; /// Simulates redeeming `shares` into assets without executing the redemption. /// /// ERC4626 original function name: previewRedeem(uint256 shares) /// /// Differs from [`Self::convert_to_asset_amount`] by factoring in: /// - The caller's current share balance. /// - Vault withdrawal limits. /// - Applicable fees or penalties. /// /// # Panics / Fails /// - The `preview_redeem_amount` method must never panic. /// It should return the amount of assets that would be received for redeeming the given shares, /// or `0` if redemption is not currently allowed. /// /// Useful for frontends to estimate redemption outcomes. fn preview_redeem_amount(&self, shares: U128) -> U128; /// Withdraws exactly `assets` worth of underlying tokens from the vault. /// /// ERC4626 original function name: withdraw(uint256 assets, address receiver, address owner) /// /// - If `receiver_id` is `None`, defaults to sending assets to the caller. /// - Burns the required number of shares to fulfill the withdrawal. /// /// # Slippage /// - To forcefully withdraw assets without accounting for slippage, the user can omit the `max_shares_deducted` parameter. /// - To protect against slippage, the user may specify a `max_shares_deducted` parameter. /// /// # Panics / Fails /// - If the caller's share balance cannot cover the withdrawal. /// - If withdrawal limits or fees prevent the withdrawal. /// /// # Events /// - The vault **must** emit a `VaultWithdraw` event upon a successful withdrawal. /// - Since transactions on NEAR are non-atomic, the vault contract should follow one of these approaches: /// 1. Emit a `VaultWithdraw` event when the fee is deducted, and if the withdrawal later fails, emit a `VaultDeposit` event /// to revert the funds; **or** /// 2. Emit a `VaultWithdraw` event **only** if the withdrawal succeeds. /// - In the reference implementation, the contract emits an `FtBurn` event when the user calls the `withdraw` method. /// If the withdrawal is successful, a `VaultWithdraw` event is emitted in the `resolve_withdraw` callback. /// If the withdrawal fails, an `FtMint` event is emitted instead to restore the user’s balance. /// /// See also: [`Self::preview_shares_deducted_for_withdraw`]. fn withdraw(&mut self, asset_amount: U128, max_shares_deducted: Option ,receiver_id: Option) -> PromiseOrValue; /// Simulates withdrawing exactly `assets` worth of tokens without executing. /// /// ERC4626 original function name: previewWithdraw(uint256 assets) /// /// Differs from [`Self::convert_to_shares`] by factoring in: /// - The caller's current share balance. /// - Vault withdrawal limits. /// - Applicable fees or penalties. /// /// # Panics / Fails /// - The `preview_shares_deducted_for_withdraw` method must never panic. /// It should return the number of shares required to withdraw the given amount of assets, /// or `0` if withdrawals are not currently allowed. /// /// Useful for frontends to preview required shares for a given withdrawal. fn preview_shares_deducted_for_withdraw(&self, asset_amount: U128) -> U128; } ``` ### Deposit and Mint ```rust pub trait FungibleTokenReceiver { /// # Events /// - The vault **must** emit a `VaultDeposit` event upon a successful deposit. /// - The vault should also emit a `FtMint` event for the share they issued. fn ft_on_transfer(&mut self, sender_id: AccountId, amount: U128, msg: String) -> PromiseOrValue; } ``` - According to the NEP-141 standard, all fungible token transfers to a contract must invoke the `ft_on_transfer` callback. - In the context of ERC-4626, both the **`deposit`** and **`mint`** functions are implemented through this entrypoint on the vault contract. - While NEP-141 does not prescribe how the `msg` parameter should be structured, our suggested convention is to pass a JSON-encoded message with the following schema: ```rust pub struct DepositMessage { /// The minimum number of shares that must be received for this deposit to succeed. (Slippage control) min_shares: Option, /// The maximum number of shares that can be minted, used for `mint` operations. max_shares: Option, /// The account that should receive the minted shares. receiver_id: Option, /// Optional memo for logging or off-chain indexing. memo: Option, /// If `true`, the transfer is treated as a donation and no shares are minted. donate: Option, } ``` - Implementers are free to define their own schema for `msg`, but the vault must correctly handle `ft_on_transfer` according to NEP-141: - Return `PromiseOrValue::Value(0)` if all tokens were accepted. - Return a non-zero `U128` to indicate the number of tokens to refund. ### Events ```rust /// Event emitted when a deposit is received by the vault. /// /// This follows the proposed NEP vault standard, referencing the ERC-4626 pattern. /// Upon receiving assets, the vault mints and issues shares to the `owner_id`. pub struct VaultDeposit { /// The account that sends the deposit (payer of the assets). pub sender_id: AccountId, /// The account that receives the minted shares. pub owner_id: AccountId, /// Amount of underlying assets deposited into the vault. pub asset_amount: U128, /// Amount of shares minted and issued to `owner_id`. pub shares: U128, /// Optional memo provided by the sender for off-chain use. pub memo: Option, } /// Event emitted when shares are redeemed from the vault. /// /// Upon redemption, the vault burns the shares from `owner_id` /// and transfers the equivalent assets to `receiver_id`. pub struct VaultWithdraw { /// The account that owns the shares being redeemed (burned). pub owner_id: AccountId, /// The account receiving the underlying assets. pub receiver_id: AccountId, /// Amount of shares redeemed (burned from the vault). pub shares: U128, /// Amount of underlying assets withdrawn from the vault. pub asset_amount: U128, /// Optional memo provided by the redeemer for off-chain use. pub memo: Option, } ``` ## Reference Implementation [Example implementation](https://github.com/Meteor-Wallet/tokenized-vault-nep-implementation). ## Security Implications ### Exchange Rate Manipulation Vaults allow dynamic exchange rates between shares and assets, calculated by dividing total vault assets by total issued shares. If the vault has a permissionless donation mechanism, it creates vulnerability to inflation attacks where attackers manipulate the rate by donating assets to inflate share values, potentially stealing funds from subsequent depositors. Vault deployers can protect against this attack by making an initial deposit of a non-trivial amount of the asset, such that price manipulation becomes infeasible. Vaults can also implement a virtual decimal offset for issued shares, making inflation attacks significantly less feasible, this is demonstrated in the reference implementation. ### Cross-contract Calls Redeem and withdraw functions perform cross-contract calls to transfer fungible tokens, creating opportunities for reentrancy attacks and state manipulation during asynchronous execution. Vaults should implement reentrancy protection through proper state management, proper callback security, and rollback mechanisms for failed operations. ### Rounding Direction Security Vault calculations must consistently round in favor of the vault to prevent exploitation. When issuing shares for deposits or transferring assets for redemptions, round down; when calculating required shares or assets for specific amounts, round up. This asymmetric rounding prevents users from extracting value through repeated micro-transactions that exploit rounding errors and protects existing shareholders from value dilution. ### Oracle and External Price Dependencies Vaults that rely on external price oracles or cross-contract calls for exchange rate updates face additional security risks in Near's asynchronous environment. Oracle updates create temporal windows where vaults operate with stale pricing data, potentially allowing exploitation. Implementations should include staleness checks, prevent operations during oracle updates, implement proper callback security, and consider fallback pricing mechanisms for oracle failures. ## Alternatives 1. **Custom Per-Protocol Vault Implementations** - While possible, this leads to fragmentation, increases integration costs, and reduces composability between protocols. 2. **Direct ERC-4626 clone Without NEAR Adjustments** - Rejected because NEAR’s asynchronous execution model makes a one-to-one ERC-4626 clone inefficient. This proposal integrates share and vault logic in a single contract to avoid unnecessary cross-contract calls. Key Differences from ERC-4626 includes: - **Deposit & Mint**: Unlike ERC-4626, deposits and mints on NEAR must be handled through the `ft_on_transfer` callback (as defined by NEP-141), rather than direct method calls. - **Async Execution**: Since NEAR contracts execute asynchronously, results from `preview*` and `max*` view methods cannot be guaranteed to match the actual values during transaction execution. Cross-contract calls may be processed in later blocks, leading to differences between simulated values and actual outcomes. ## Future possibilities ### NEP-245 Multi Token Support Future vault implementations could extend this standard to support NEP-245 Multi Token contracts as underlying asset. We have also created a [minimal implementation](https://github.com/Meteor-Wallet/tokenized-vault-mt-nep-implementation). ### Multi-Asset Vault Extensions Future extensions could allow vaults to accept multiple assets for deposit and withdrawal. This would enable the standardization of LP vaults. ### Asynchronous Vault Operations Future vault standards could introduce asynchronous deposit and withdrawal patterns through `request_deposit` and `request_withdraw` functions. This would enable integration with cross-chain protocols and real-world asset protocols. ## Consequences ### Positive - Enables a unified, predictable vault interface. - Simplifies integration for wallets, DEXs, and aggregators. - Improves security through consistent design and accounting. - Encourages reuse of tooling, libraries, and audits. ### Neutral - Standard defines interface, not yield strategy — implementation remains flexible. - Protocols may implement only relevant parts of the interface initially. ### Negative - Migration overhead for existing vault implementations to become compliant. ### Backwards Compatibility - No breaking changes to NEP-141 itself, but existing vault-like contracts that don’t conform will need to add or rename methods to comply. - Share tokens must be NEP-141 compliant, meaning non-NEP-141 share implementations require migration. ## Changelog ### 1.0.0 - Initial Version > Placeholder for the context about when and who approved this NEP version. #### Benefits > List of benefits filled by the Subject Matter Experts while reviewing this version: - Would benefit the ecosystem by introducing a standardized way to create token vaults. Indeed, multiple projects currently exists (e.g. metapool, more markets, etc) but none of them follow a single standard making it hard to interconnect with them - Benefit 2 #### Concerns No major concerns were raised, besides the need to need to be careful with the #security implications sections described above ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ``` ``` ================================================ FILE: neps/nep-0635.md ================================================ --- NEP: 622 Title: P-256 ECDSA Signature Verification Host Function Authors: Bowen Wang Status: Draft DiscussionsTo: https://github.com/near/NEPs/pull/0635 Type: Runtime Spec Category: Protocol Version: 1.0.0 Created: 2026-01-24 Last Updated: 2026-01-24 --- ## Summary This NEP proposes adding a host function, `p256_verify`, to verify ECDSA signatures over the P-256 curve (secp256r1, prime256v1). The host function exposes a native implementation in the runtime, enabling smart contracts to verify P-256 signatures at substantially lower gas cost than a WASM-based implementation. ## Motivation P-256 verification is a hard requirement for multiple workloads on NEAR today, and the current WASM path is prohibitively expensive: - Near Intents relies on passkeys (WebAuthn), which use P-256 ECDSA. A native host function substantially reduces verification gas and enables higher throughput for intents. - TEE attestation verification depends on the `dcap-qvl` library, which uses P-256. The latest `dcap-qvl` does not fit within NEAR's gas limits when compiled to WASM, and a host function would reduce the cost of `dcap-qvl::verify::verify()` (see https://github.com/Phala-Network/dcap-qvl/issues/99). - Benchmarks show that compiling the crypto library to WASM directly consumes ~46 Tgas for verifying a 32-byte message, while the host function consumes ~0.45 Tgas, a reduction of over 100x. ## Rationale and alternatives Adding a dedicated host function provides an immediate and predictable gas reduction for a widely used cryptographic primitive. The runtime already includes mature RustCrypto implementations, and the host function design is consistent with existing cryptography host functions such as `ed25519_verify`. Alternatives considered: - Improve WASM compilation performance or use new compiler backends. This is a large, cross-cutting effort and does not guarantee acceptable costs for P-256 verification in the near term. - Increase gas limits or allow special-case gas budgets. This adds policy complexity and does not address the root performance gap. - Require contracts to verify P-256 signatures off-chain. This shifts trust and reduces on-chain verifiability, which is undesirable for intents and TEE attestations. ## Specification The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). ### Protocol feature The host function MUST be guarded by the protocol feature flag `p256_verify`. When disabled, the import MUST be unavailable to contracts. ### Host function ```rust extern "C" { /// Verify a P-256 ECDSA signature for a message and public key. /// /// - `signature` MUST be 64 bytes encoded as `r || s` (32 bytes each, big-endian). /// - `public_key` MUST be a 33-byte compressed SEC1 encoding. /// - `message` is the pre-hashed digest to verify (no hashing is performed by the host function). /// Callers MUST hash the signed data with an appropriate cryptographic hash function before /// calling `p256_verify` (e.g., WebAuthn uses SHA-256). If `message` is not exactly 32 bytes, /// it will be truncated or zero-padded to 32 bytes to match the P-256 field size. /// /// Returns: /// - 1 if the signature verifies /// - 0 if the signature does not verify or cannot be parsed /// /// # Errors /// /// - If `signature` length is not 64, the runtime MUST raise `P256VerifyInvalidInput`. /// - If `public_key` length is not 33, the runtime MUST raise `P256VerifyInvalidInput`. /// - If any pointer is out of bounds, the runtime MUST raise `MemoryAccessViolation`. fn p256_verify( sig_len: u64, sig_ptr: u64, msg_len: u64, msg_ptr: u64, public_key_len: u64, public_key_ptr: u64, ) -> u64; } ``` Each input can be in memory or in a register. If the length argument is set to `u64::MAX`, the corresponding pointer is interpreted as a register ID. The runtime MUST apply the standard input cost for reading memory or registers. ### Gas cost The gas cost MUST be computed as: `input_cost(num_bytes_signature) + input_cost(num_bytes_message) + input_cost(num_bytes_public_key) + p256_verify_base + p256_verify_byte * num_bytes_message` ### SDK exposure Once enabled, the runtime SHOULD expose a higher-level helper in `near-sdk` as: ```rust pub fn p256_verify(signature: [u8; 64], message: [u8; 32], public_key: [u8; 33]) -> bool; ``` ## Reference-level specification The reference implementation in `nearcore`: - Uses the RustCrypto `p256` crate - [audited here](https://reports.zksecurity.xyz/reports/near-p256/) - to parse `Signature::from_slice` and `VerifyingKey::from_sec1_bytes`. - Returns `0` for parsing failures or verification failures, and raises `P256VerifyInvalidInput` for invalid lengths. - Charges `p256_verify_base` once and `p256_verify_byte` per message byte, in addition to the standard input costs. ## Security Implications (Optional) This host function does not perform hashing or domain separation. Callers MUST ensure that the message passed to `p256_verify` follows the correct protocol (for example, WebAuthn uses SHA-256 over the signed data). The runtime only checks signature and key encoding constraints as specified above. ## Unresolved Issues (Optional) - Whether to support uncompressed SEC1 public keys (65 bytes) in addition to compressed keys. - Whether to support DER-encoded ECDSA signatures in addition to raw `r || s` encoding. ## Future possibilities - Batch verification for P-256 signatures if proven beneficial for intents or attestations. - Additional host functions for other curves used by standard protocols or TEEs. ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). ================================================ FILE: neps/nep-0638.md ================================================ --- NEP: 638 Title: `chain_id()` host function Authors: Arseny Mitin Status: Draft Type: Protocol Version: 1.0.0 Created: 2026-02-02 LastUpdated: 2026-02-02 --- ## Summary This NEP proposes adding a new `chain_id` host function, so that smart-contracts can retrieve the identifier of Near chain they are being executed on. ## Motivation With [NEP-616](./nep-0616.md) it's now possible to have account-abstracted [wallet-contracts](./nep-0616.md#wallet-extentions) deployed at [deterministic AccountIds](./nep-0616.md#deterministic-accountids). These wallet-contracts are responsible themselves for signature verification, nonce tracking and etc. While it enables for true account abstraction, smart-contracts on Near currently still lack knowledge of `chain_id` that they are being executed on. As a result, wallet-contracts do not have a native protection against replaying messages between mainnet, testnet and other networks. The existing [workarounds](#alternatives) come at a cost of poor UX. This NEP removes this limitation by introduing a new `chain_id` host function, enabling for account abstraction without UX tradeoffs. ## Specification The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). ### Host Function ```rust extern "C" { /// Get the current chain_id. /// /// Writes the current chain_id as UTF-8 encoded string into register /// `register_id`. fn chain_id(register_id: u64); } ``` ### Gas cost The gas cost MUST be computed as: `base + write_register_base + write_register_byte * num_bytes` ## Reference Implementation Having `chain_id` [field](https://github.com/near/nearcore/blob/b8ab0f5d578617eb287030e77ab17be512a5f8ac/core/chain-configs/src/genesis_config.rs#L120-L122) already present in validator config, it's possible to make this value accessible by smart-contracts in the runtime. ## Security Implications As already [stated](https://github.com/near/nearcore/blob/b8ab0f5d578617eb287030e77ab17be512a5f8ac/core/chain-configs/src/genesis_config.rs#L120-L122) in nearcore implementation, `chain_id` MUST be unique for every blockchain. Otherwise, messages can be replayed between networks with same chain_ids. ## Alternatives An alternative would be to encode "virtual" `chain_id` as a part of initializaton state for wallet-contract. For instance, TON's [wallet-v5](https://github.com/ton-blockchain/wallet-contract-v5/blob/main/README.md#known-security-issues) follows such approach, where clients [derive](https://github.com/ton-org/ton/blob/5deac43432fa5dfcd441f2f0100dc3f89f55bead/src/wallets/v5r1/WalletV5R1WalletId.ts#L11-L15) corresponding `wallet_id` using "virtual" `chain_id`. However, this would make users to have their wallet-contracts deployed at different [deterministic AccountIds](./nep-0616.md#deterministic-accountids) on different Near chains even if they use the same public key. As a result, this worsens the UX for end users. Having `chain_id` encoded in the signed payload enables EVM-like UX, where all wallet-contracts are deployed on the same AccountIds, regardless of a chain_id being used. ## Future possibilities Currently, [transactions](https://github.com/near/nearcore/blob/66a3dc3c2e79adb3bbe5bfd39e41ef7dcd723a95/core/primitives/src/transaction.rs#L82-L98) on Near do not include `chain_id`. So, replay protection is achieved solely by relying on the unlikelihood of recent [block_hash](https://github.com/near/nearcore/blob/66a3dc3c2e79adb3bbe5bfd39e41ef7dcd723a95/core/primitives/src/transaction.rs#L94-L95) collisions. It might make sense to add `chain_id` field in the next version of [Transaction](https://github.com/near/nearcore/blob/66a3dc3c2e79adb3bbe5bfd39e41ef7dcd723a95/core/primitives/src/transaction.rs#L109-L112) object. ## Consequences ### Positive * Wallet-contracts (and other smart-contracts) can have native protection against replaying messages between different Near chains without UX tradeoffs. ### Neutral \- ### Negative \- ### Backwards Compatibility This proposal is **backwards-compatible**: it only adds a new host function, so that new contracts can opt-in using it, while existing ones can be upgraded to start using it if necessary. ## Changelog ### 1.0.0 - Initial Version #### Benefits * Native protection against replaying messages between different Near chains. * Better UX for end users: wallet-contracts are deployed at the same AccountIds when using same public key. #### Concerns | # | Concern | Resolution | Status | | -: | :------------------------------------------------------------- | :--- | ---: | | 1 | Does consensus ensure all validators have the same `chain_id`? | Yes ([reasoning](https://github.com/near/NEPs/pull/638#issuecomment-3842705589)) | New | ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).