Showing preview only (498K chars total). Download the full file or copy to clipboard to get everything.
Repository: WhatsApp/waraft
Branch: main
Commit: befbd1818d62
Files: 40
Total size: 479.4 KB
Directory structure:
gitextract_f4__cq5a/
├── .github/
│ └── workflows/
│ └── build.yml
├── .gitignore
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── LICENSE
├── README.md
├── examples/
│ └── kvstore/
│ └── src/
│ ├── kvstore.app.src
│ ├── kvstore_app.erl
│ ├── kvstore_client.erl
│ └── kvstore_sup.erl
├── include/
│ ├── wa_raft.hrl
│ ├── wa_raft_logger.hrl
│ └── wa_raft_rpc.hrl
├── rebar.config
└── src/
├── wa_raft.app.src
├── wa_raft.erl
├── wa_raft_acceptor.erl
├── wa_raft_app.erl
├── wa_raft_app_sup.erl
├── wa_raft_dist_transport.erl
├── wa_raft_distribution.erl
├── wa_raft_durable_state.erl
├── wa_raft_env.erl
├── wa_raft_info.erl
├── wa_raft_label.erl
├── wa_raft_log.erl
├── wa_raft_log_ets.erl
├── wa_raft_metrics.erl
├── wa_raft_part_sup.erl
├── wa_raft_queue.erl
├── wa_raft_server.erl
├── wa_raft_snapshot_catchup.erl
├── wa_raft_storage.erl
├── wa_raft_storage_ets.erl
├── wa_raft_sup.erl
├── wa_raft_transport.erl
├── wa_raft_transport_cleanup.erl
├── wa_raft_transport_sup.erl
├── wa_raft_transport_target_sup.erl
└── wa_raft_transport_worker.erl
================================================
FILE CONTENTS
================================================
================================================
FILE: .github/workflows/build.yml
================================================
name: Build
on:
push:
branches:
- '*'
pull_request:
types:
- opened
- synchronize
jobs:
linux:
strategy:
matrix:
platform: [ubuntu-latest]
otp-version: [24]
runs-on: ${{ matrix.platform }}
container:
image: erlang:${{ matrix.otp-version }}
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Cache Hex packages
uses: actions/cache@v1
with:
path: ~/.cache/rebar3/hex/hexpm/packages
key: ${{ runner.os }}-hex-${{ hashFiles(format('{0}{1}', github.workspace, '/rebar.lock')) }}
restore-keys: |
${{ runner.os }}-hex-
- name: Cache Dialyzer PLTs
uses: actions/cache@v1
with:
path: ~/.cache/rebar3/rebar3_*_plt
key: ${{ runner.os }}-dialyzer-${{ hashFiles(format('{0}{1}', github.workspace, '/rebar.config')) }}
restore-keys: |
${{ runner.os }}-dialyzer-
- name: Compile
run: rebar3 compile
- name: Generate Dialyzer PLT
run: dialyzer --build_plt --apps erts kernel stdlib
- name: Run CT Tests
run: rebar3 ct
- name: Run Checks
run: rebar3 do dialyzer, xref
- name: Produce Documentation
run: rebar3 edoc
if: ${{ matrix.otp-version == '24' }}
================================================
FILE: .gitignore
================================================
.DS_Store
.rebar3
.eunit
*.o
*.beam
*.plt
*.swp
*.swo
ebin
log
erl_crash.dump
.rebar
_build
.idea
rebar3.crashdump
.edts
*.coverdata
*.log
*.log.*
doc
# Emacs Backup files
*~
# Emacs temporary files
.#*
*#
================================================
FILE: CODE_OF_CONDUCT.md
================================================
# Code of Conduct
Meta has adopted a Code of Conduct that we expect project participants to adhere to.
Please read the [full text](https://code.fb.com/codeofconduct/)
so that you can understand what actions will and will not be tolerated.
================================================
FILE: CONTRIBUTING.md
================================================
# Contributing to WhatsApp Raft
We want to make contributing to this project as easy and transparent as
possible.
## Our Development Process
We expect to ship changes to existing setup scripts and add new setup scripts on an ongoing basis.
## Pull Requests
We actively welcome your pull requests.
1. Fork the repo and create your branch from `main`.
2. Make sure your changes lint and work with all past and present versions of WhatsApp RAFT.
3. If you haven't already, complete the Contributor License Agreement ("CLA").
## Contributor License Agreement ("CLA")
In order to accept your pull request, we need you to submit a CLA. You only need
to do this once to work on any of Meta's open source projects.
Complete your CLA here: <https://code.facebook.com/cla>
## Issues
We use GitHub issues to track public bugs. Please ensure your description is
clear and has sufficient instructions to be able to reproduce the issue.
For issues on your integration with WhatsApp Raft, please use our
support channel at <https://business.facebook.com/direct-support>.
Meta has a [bounty program](https://www.facebook.com/whitehat/) for the safe
disclosure of security bugs. In those cases, please go through the process
outlined on that page and do not file a public issue.
## License
By contributing to WhatsApp Raft, you agree that your contributions will be licensed
under the LICENSE file in the root directory of this source tree.
================================================
FILE: LICENSE
================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
================================================
FILE: README.md
================================================
# WhatsApp Raft - WARaft
WARaft is a Raft library in Erlang by WhatsApp. It provides an Erlang implementation to obtain consensus among replicated state machines. Consensus is a fundamental problem in fault-tolerant distributed systems. WARaft has been used as consensus provider in WhatsApp message storage, which is a large scale strongly consistent storage system across 5+ datacenters.
## Features
* Full implementation of Raft consensus algorithm defined in https://raft.github.io/
* Extensible framework. It offers pluggable component interface for log, state machines and transport layer. Users are also allowed provide their own implementation to customize .
* Performant. It is highly optimized for large volume transactions user cases. It could support up to 200K/s transactions with in a 5 node cluster.
* Distributed key value store. WARaft provides components needed to build a distributed key-value storage.
## Get Started
The following code snippet gives a quick glance about how WARaft works. It creates a single-node WARaft cluster and writes and reads a record.
```erlang
% Setup the WARaft application and the host application
rr(wa_raft_server).
application:ensure_all_started(wa_raft).
application:set_env(test_app, raft_database, ".").
% Create a spec for partition 1 of the RAFT table "test" and start it.
Spec = wa_raft_sup:child_spec(test_app, [#{table => test, partition => 1}]).
% Here we add WARaft to the kernel's supervisor, but you should place WARaft's
% child spec underneath your application's supervisor in a real deployment.
supervisor:start_child(kernel_sup, Spec).
% Check that the RAFT server started successfully
wa_raft_server:status(raft_server_test_1).
% Make a cluster configuration with the current node as the only member
Config = wa_raft_server:make_config([#raft_identity{name = raft_server_test_1, node = node()}]).
% Bootstrap the RAFT server to get it started
wa_raft_server:bootstrap(raft_server_test_1, #raft_log_pos{index = 1, term = 1}, Config, #{}).
% Wait for the RAFT server to become the leader
wa_raft_server:status(raft_server_test_1).
% Read and write against a key
wa_raft_acceptor:commit(raft_acceptor_test_1, {make_ref(), {write, test, key, 1000}}).
wa_raft_acceptor:read(raft_acceptor_test_1, {read, test, key}).
```
A typical output would look like the following:
```erlang
1> % Setup the WARaft application and the host application
rr(wa_raft_server).
[raft_application,raft_identifier,raft_identity,raft_log,
raft_log_pos,raft_options,raft_state]
2> application:ensure_all_started(wa_raft).
{ok,[wa_raft]}
3> application:set_env(test_app, raft_database, ".").
ok
4> % Create a spec for partition 1 of the RAFT table "test" and start it.
Spec = wa_raft_sup:child_spec(test_app, [#{table => test, partition => 1}]).
#{id => wa_raft_sup,restart => permanent,shutdown => infinity,
start =>
{wa_raft_sup,start_link,
[test_app,[#{table => test,partition => 1}],#{}]},
type => supervisor,
modules => [wa_raft_sup]}
5> % Here we add WARaft to the kernel's supervisor, but you should place WARaft's
% child spec underneath your application's supervisor in a real deployment.
supervisor:start_child(kernel_sup, Spec).
{ok,<0.101.0>}
6> % Check that the RAFT server started successfully
wa_raft_server:status(raft_server_test_1).
[{state,stalled},
{id,nonode@nohost},
{table,test},
{partition,1},
{data_dir,"./test.1"},
{current_term,0},
{voted_for,undefined},
{commit_index,0},
{last_applied,0},
{leader_id,undefined},
{next_index,#{}},
{match_index,#{}},
{log_module,wa_raft_log_ets},
{log_first,0},
{log_last,0},
{votes,#{}},
{inflight_applies,0},
{disable_reason,undefined},
{config,#{version => 1,membership => [],witness => []}},
{config_index,0},
{witness,false}]
7> % Make a cluster configuration with the current node as the only member
Config = wa_raft_server:make_config([#raft_identity{name = raft_server_test_1, node = node()}]).
#{version => 1,
membership => [{raft_server_test_1,nonode@nohost}],
witness => []}
8> % Bootstrap the RAFT server to get it started
wa_raft_server:bootstrap(raft_server_test_1, #raft_log_pos{index = 1, term = 1}, Config, #{}).
ok
9> % Wait for the RAFT server to become the leader
wa_raft_server:status(raft_server_test_1).
[{state,leader},
{id,nonode@nohost},
{table,test},
{partition,1},
{data_dir,"./test.1"},
{current_term,1},
{voted_for,nonode@nohost},
{commit_index,2},
{last_applied,2},
{leader_id,nonode@nohost},
{next_index,#{}},
{match_index,#{}},
{log_module,wa_raft_log_ets},
{log_first,1},
{log_last,2},
{votes,#{}},
{inflight_applies,0},
{disable_reason,undefined},
{config,#{version => 1,
membership => [{raft_server_test_1,nonode@nohost}],
witness => []}},
{config_index,1},
{witness,false}]
10> % Read and write against a key
wa_raft_acceptor:commit(raft_acceptor_test_1, {make_ref(), {write, test, key, 1000}}).
ok
11> wa_raft_acceptor:read(raft_acceptor_test_1, {read, test, key}).
{ok,1000}
```
The [example directory](https://github.com/WhatsApp/waraft/tree/main/examples/kvstore/src) contains an example generic key-value store built on top of WARaft.
## License
WARaft is [Apache licensed](./LICENSE).
================================================
FILE: examples/kvstore/src/kvstore.app.src
================================================
%% % @format
%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%
%% This source code is licensed under the Apache 2.0 license found in
%% the LICENSE file in the root directory of this source tree.
{application, kvstore, [
{description, "Distributed Key-Value Storage"},
{vsn, "1.0.0"},
{modules, []},
{registered, [kvstore_sup]},
{applications, [
kernel,
stdlib,
wa_raft
]},
{env, [
% Specify where you want your data to be stored here
{raft_database, "/mnt/kvstore"},
% Specify your own implementations here
{raft_log_module, wa_raft_log_ets},
{raft_storage_module, wa_raft_storage_ets},
{raft_distribution_module, wa_raft_distribution},
{raft_transport_module, wa_raft_transport}
]},
{mod, {kvstore_app, []}}
]}.
================================================
FILE: examples/kvstore/src/kvstore_app.erl
================================================
%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%%
%%% This source code is licensed under the Apache 2.0 license found in
%%% the LICENSE file in the root directory of this source tree.
-module(kvstore_app).
-compile(warn_missing_spec_all).
-behaviour(application).
%% API
-export([
start/2,
stop/1
]).
-spec start(application:start_type(), term()) -> {ok, pid()}.
start(normal, _Args) ->
{ok, _Pid} = kvstore_sup:start_link().
-spec stop(term()) -> ok.
stop(_State) ->
ok.
================================================
FILE: examples/kvstore/src/kvstore_client.erl
================================================
%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%%
%%% This source code is licensed under the Apache 2.0 license found in
%%% the LICENSE file in the root directory of this source tree.
%%%
%%% This module offers APIs to access the storage.
%%%
-module(kvstore_client).
-compile(warn_missing_spec_all).
-export([
read/1,
write/2,
delete/1
]).
-include_lib("wa_raft/include/wa_raft.hrl").
-define(CALL_TIMEOUT, 5000).
-define(TABLE, kvstore).
-define(NUM_PARTITIONS, 4).
%% Read value for a given key. It's a blocking call.
-spec read(term()) -> {ok, term()} | wa_raft_acceptor:read_error().
read(Key) ->
Acceptor = ?RAFT_ACCEPTOR_NAME(?TABLE, partition(Key)),
wa_raft_acceptor:read(Acceptor, {read, ?TABLE, Key}, ?CALL_TIMEOUT).
%% Write a key/value pair to storage. It's a blocking call.
-spec write(term(), term()) -> ok | wa_raft_acceptor:commit_error().
write(Key, Value) ->
commit(Key, {write, ?TABLE, Key, Value}).
%% Delete a key/value pair. It's a blocking call.
-spec delete(term()) -> ok | wa_raft_acceptor:commit_error().
delete(Key) ->
commit(Key, {delete, ?TABLE, Key}).
-spec commit(term(), term()) -> term() | wa_raft_acceptor:commit_error().
commit(Key, Command) ->
Acceptor = ?RAFT_ACCEPTOR_NAME(?TABLE, partition(Key)),
wa_raft_acceptor:commit(Acceptor, {make_ref(), Command}, ?CALL_TIMEOUT).
-spec partition(term()) -> number().
partition(Key) ->
erlang:phash2(Key, ?NUM_PARTITIONS) + 1.
================================================
FILE: examples/kvstore/src/kvstore_sup.erl
================================================
%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%%
%%% This source code is licensed under the Apache 2.0 license found in
%%% the LICENSE file in the root directory of this source tree.
%%%
%%% This supervisor starts 4 RAFT partitions under itself.
%%%
-module(kvstore_sup).
-compile(warn_missing_spec_all).
-behaviour(supervisor).
-export([
start_link/0,
init/1
]).
-spec start_link() -> supervisor:startlink_ret().
start_link() ->
supervisor:start_link({local, ?MODULE}, ?MODULE, []).
-spec init(term()) -> {ok, {supervisor:sup_flags(), [supervisor:child_spec()]}}.
init([]) ->
Partitions = [1, 2, 3, 4],
Args = [raft_args(P) || P <- Partitions],
ChildSpecs = [
wa_raft_sup:child_spec(Args)
],
{ok, {#{}, ChildSpecs}}.
% Construct a RAFT "args" for a partition.
-spec raft_args(Partition :: wa_raft:partition()) -> wa_raft:args().
raft_args(Partition) ->
% RAFT clusters are primarily identified by their table and partition number
#{table => kvstore, partition => Partition}.
================================================
FILE: include/wa_raft.hrl
================================================
%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%%
%%% This source code is licensed under the Apache 2.0 license found in
%%% the LICENSE file in the root directory of this source tree.
%%%
%%% This file defines general macros and data structures shared across modules.
%% The name of the RAFT application.
-define(RAFT_APPLICATION, wa_raft).
%%-------------------------------------------------------------------
%% Registered information about applications and partitions
%%-------------------------------------------------------------------
%% Name of the application environment key that is expected to contain a path
%% to the directory in which RAFT should place the partition paths for the
%% RAFT partitions under a RAFT client application. This environment value uses
%% the application search order to determine the value to use.
-define(RAFT_DATABASE, raft_database).
%% Default location containing databases for RAFT partitions part of a RAFT client application
-define(RAFT_DATABASE_PATH(Application), (wa_raft_env:database_path(Application))).
%% Registered database location for the specified RAFT partition
-define(RAFT_PARTITION_PATH(Table, Partition), (wa_raft_part_sup:registered_partition_path(Table, Partition))).
%% Registered name of the RAFT partition supervisor for a RAFT partition
-define(RAFT_SUPERVISOR_NAME(Table, Partition), (wa_raft_part_sup:registered_name(Table, Partition))).
%% Registered name of the RAFT acceptor server for a RAFT partition
-define(RAFT_ACCEPTOR_NAME(Table, Partition), (wa_raft_acceptor:registered_name(Table, Partition))).
%% Registered name of the RAFT log server for a RAFT partition
-define(RAFT_LOG_NAME(Table, Partition), (wa_raft_log:registered_name(Table, Partition))).
%% Registered name of the RAFT server for a RAFT partition
-define(RAFT_SERVER_NAME(Table, Partition), (wa_raft_server:registered_name(Table, Partition))).
%% Registered name of the RAFT storage server for a RAFT partition
-define(RAFT_STORAGE_NAME(Table, Partition), (wa_raft_storage:registered_name(Table, Partition))).
%% Default distribution provider module
-define(RAFT_DEFAULT_DISTRIBUTION_MODULE, wa_raft_distribution).
%% Default log provider module
-define(RAFT_DEFAULT_LOG_MODULE, wa_raft_log_ets).
%% Default storage provider module
-define(RAFT_DEFAULT_STORAGE_MODULE, wa_raft_storage_ets).
%% Default module for handling outgoing transports
-define(RAFT_DEFAULT_TRANSPORT_MODULE, wa_raft_dist_transport).
%% Default module for log labeling
-define(RAFT_DEFAULT_LABEL_MODULE, undefined).
%% RAFT election max weight
-define(RAFT_ELECTION_MAX_WEIGHT, 10).
%% Raft election default weight
-define(RAFT_ELECTION_DEFAULT_WEIGHT, ?RAFT_ELECTION_MAX_WEIGHT).
%% Name of server state persist file
-define(STATE_FILE_NAME, "state").
%% Name prefix for snapshots
-define(SNAPSHOT_PREFIX, "snapshot").
%% Snapshot name
-define(SNAPSHOT_NAME(Index, Term), (?SNAPSHOT_PREFIX "." ++ integer_to_list(Index) ++ "." ++ integer_to_list(Term))).
%% Witness Snapshot name
-define(WITNESS_SNAPSHOT_NAME(Index, Term), (?SNAPSHOT_PREFIX "." ++ integer_to_list(Index) ++ "." ++ integer_to_list(Term) ++ ".witness")).
%% Location of a snapshot
-define(RAFT_SNAPSHOT_PATH(Path, Name), (filename:join(Path, Name))).
-define(RAFT_SNAPSHOT_PATH(Table, Partition, Name), ?RAFT_SNAPSHOT_PATH(?RAFT_PARTITION_PATH(Table, Partition), Name)).
-define(RAFT_SNAPSHOT_PATH(Table, Partition, Index, Term), ?RAFT_SNAPSHOT_PATH(Table, Partition, ?SNAPSHOT_NAME(Index, Term))).
%% [Transport] Atomics - field index for update timestamp
-define(RAFT_TRANSPORT_ATOMICS_UPDATED_TS, 1).
%% [Transport] Transport atomics - field count
-define(RAFT_TRANSPORT_TRANSPORT_ATOMICS_COUNT, 1).
%% [Transport] File atomics - field count
-define(RAFT_TRANSPORT_FILE_ATOMICS_COUNT, 1).
-define(READ_OP, '$read').
%%-------------------------------------------------------------------
%% Metrics
%%-------------------------------------------------------------------
-define(RAFT_METRICS_MODULE_KEY, {?RAFT_APPLICATION, raft_metrics_module}).
-define(RAFT_METRICS_MODULE, (persistent_term:get(?RAFT_METRICS_MODULE_KEY, wa_raft_metrics))).
-define(RAFT_COUNT(Table, Metric), ?RAFT_METRICS_MODULE:count({raft, Table, Metric})).
-define(RAFT_COUNTV(Table, Metric, Value), ?RAFT_METRICS_MODULE:countv({raft, Table, Metric}, Value)).
-define(RAFT_GATHER(Table, Metric, Value), ?RAFT_METRICS_MODULE:gather({raft, Table, Metric}, Value)).
-define(RAFT_GATHER_LATENCY(Table, Metric, Value), ?RAFT_METRICS_MODULE:gather_latency({raft, Table, Metric}, Value)).
%%-------------------------------------------------------------------
%% Global Configuration
%%-------------------------------------------------------------------
%% Get global config
-define(RAFT_CONFIG(Name), (application:get_env(?RAFT_APPLICATION, Name))).
-define(RAFT_CONFIG(Name, Default), (application:get_env(?RAFT_APPLICATION, Name, Default))).
%% Default metrics module
-define(RAFT_METRICS_MODULE(), ?RAFT_CONFIG(raft_metrics_module)).
%% Default Call timeout for all cross node gen_server:call
-define(RAFT_RPC_CALL_TIMEOUT(), ?RAFT_CONFIG(raft_rpc_call_timeout, 10000)).
%% Default call timeout for storage related operation (we need bigger default since storage can be slower)
-define(RAFT_STORAGE_CALL_TIMEOUT(), ?RAFT_CONFIG(raft_storage_call_timeout, 60000)).
%% Maximum number of concurrent outgoing snapshot transfers initiated by leaders.
-define(RAFT_MAX_CONCURRENT_SNAPSHOT_CATCHUP(), ?RAFT_CONFIG(raft_max_snapshot_catchup, 5)).
%% Maximum number of concurrent incoming snapshot transfers.
-define(RAFT_MAX_CONCURRENT_INCOMING_SNAPSHOT_TRANSFERS(), ?RAFT_CONFIG(raft_max_incoming_snapshot_transfers, 5)).
%% Maximum number of concurrent incoming snapshot transfers of witness snapshots.
-define(RAFT_MAX_CONCURRENT_INCOMING_WITNESS_SNAPSHOT_TRANSFERS(), ?RAFT_CONFIG(raft_max_incoming_witness_snapshot_transfers, 10)).
%% Default cross-node call timeout for heartbeats made for bulk logs catchup
-define(RAFT_CATCHUP_HEARTBEAT_TIMEOUT(), ?RAFT_CONFIG(raft_catchup_rpc_timeout_ms, 5000)).
%% Number of workers to use for transports
-define(RAFT_TRANSPORT_THREADS(), ?RAFT_CONFIG(raft_transport_threads, 1)).
%% Time in seconds after which a transport that has not made progress should be considered failed
-define(RAFT_TRANSPORT_IDLE_TIMEOUT(), ?RAFT_CONFIG(transport_idle_timeout_secs, 30)).
%% Maximum number of previous inactive transports to retain info for.
-define(RAFT_TRANSPORT_INACTIVE_INFO_LIMIT(), ?RAFT_CONFIG(raft_transport_inactive_info_limit, 30)).
%% Size in bytes of individual chunks (messages containing file data) to be sent during transports
%% using the dist transport provider
-define(RAFT_DIST_TRANSPORT_CHUNK_SIZE(), ?RAFT_CONFIG(dist_transport_chunk_size, 1 * 1024 * 1024)).
%% Maximum number of chunks that can be sent by the dist transport provider without being
%% acknowledged by the recipient
-define(RAFT_DIST_TRANSPORT_MAX_INFLIGHT(), ?RAFT_CONFIG(dist_transport_max_inflight, 4)).
%%-------------------------------------------------------------------
%% Configuration
%%-------------------------------------------------------------------
%% Get application-scoped config
-define(RAFT_APP_CONFIG(App, Name, Default), (wa_raft_env:get_env(App, Name, Default))).
%% Get table-scoped config (supports {table_overrides, #{Table => Value}, AppDefault})
-define(RAFT_TABLE_CONFIG(App, Table, Name, Default), (wa_raft_env:get_table_env(App, Table, Name, Default))).
%%-------------------------------------------------------------------
%% Application-scoped Configuration
%%-------------------------------------------------------------------
%% Whether or not this node is eligible to be leader.
-define(RAFT_LEADER_ELIGIBLE, raft_leader_eligible).
-define(RAFT_LEADER_ELIGIBLE(App), (?RAFT_APP_CONFIG(App, ?RAFT_LEADER_ELIGIBLE, true) =/= false)).
%% Relative "weight" at which this node will trigger elections and thereby be elected.
-define(RAFT_ELECTION_WEIGHT, raft_election_weight).
-define(RAFT_ELECTION_WEIGHT(App), ?RAFT_APP_CONFIG(App, ?RAFT_ELECTION_WEIGHT, ?RAFT_ELECTION_DEFAULT_WEIGHT)).
%% Time in seconds to retain transport destination directories after use
-define(RAFT_TRANSPORT_RETAIN_INTERVAL, transport_retain_min_secs).
-define(RAFT_TRANSPORT_RETAIN_INTERVAL(App), ?RAFT_APP_CONFIG(App, ?RAFT_TRANSPORT_RETAIN_INTERVAL, 300)).
%%-------------------------------------------------------------------
%% Table-scoped Configuration
%%-------------------------------------------------------------------
%% Maximum number of pending applies for any single RAFT partition
-define(RAFT_MAX_PENDING_APPLIES, raft_max_pending_applies).
-define(RAFT_MAX_PENDING_APPLIES(App, Table), ?RAFT_TABLE_CONFIG(App, Table, {?RAFT_MAX_PENDING_APPLIES, raft_apply_queue_max_size}, 1000)).
%% Maximum bytes of pending applies for any single RAFT partition
-define(RAFT_MAX_PENDING_APPLY_BYTES, raft_max_pending_apply_bytes).
-define(RAFT_MAX_PENDING_APPLY_BYTES(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_MAX_PENDING_APPLY_BYTES, 32_000_000)).
%% Maximum number of pending high priority commits for any single RAFT partition
-define(RAFT_MAX_PENDING_HIGH_PRIORITY_COMMITS, raft_max_pending_high_priority_commits).
-define(RAFT_MAX_PENDING_HIGH_PRIORITY_COMMITS(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_MAX_PENDING_HIGH_PRIORITY_COMMITS, 1500)).
%% Maximum number of pending low priority commits for any single RAFT partition
-define(RAFT_MAX_PENDING_LOW_PRIORITY_COMMITS, raft_max_pending_low_priority_commits).
-define(RAFT_MAX_PENDING_LOW_PRIORITY_COMMITS(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_MAX_PENDING_LOW_PRIORITY_COMMITS, 250)).
%% Maximum number of pending reads for any single RAFT partition
-define(RAFT_MAX_PENDING_READS, raft_max_pending_reads).
-define(RAFT_MAX_PENDING_READS(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_MAX_PENDING_READS, 5000)).
%% Time in milliseconds during which a leader was unable to replicate heartbeats to a
%% quorum of followers before considering the leader to be stale.
-define(RAFT_LEADER_STALE_INTERVAL, raft_max_heartbeat_age_msecs).
-define(RAFT_LEADER_STALE_INTERVAL(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_LEADER_STALE_INTERVAL, 180 * 1000)).
%% Interval in milliseconds between heartbeats sent by RAFT leaders with no pending log entries
-define(RAFT_HEARTBEAT_INTERVAL, raft_heartbeat_interval_ms).
-define(RAFT_HEARTBEAT_INTERVAL(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_HEARTBEAT_INTERVAL, 120)).
%% Maximum number of log entries to include in a single heartbeat
-define(RAFT_HEARTBEAT_MAX_ENTRIES, raft_max_log_entries_per_heartbeat).
-define(RAFT_HEARTBEAT_MAX_ENTRIES(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_HEARTBEAT_MAX_ENTRIES, 15)).
%% Maximum bytes of log entries to include in a single heartbeat
-define(RAFT_HEARTBEAT_MAX_BYTES, raft_max_heartbeat_size).
-define(RAFT_HEARTBEAT_MAX_BYTES(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_HEARTBEAT_MAX_BYTES, 1 * 1024 * 1024)).
%% Time in milliseconds to wait to collect pending log entries into a single heartbeat before
%% triggering a heartbeat due to having pending log entries
-define(RAFT_COMMIT_BATCH_INTERVAL, raft_commit_batch_interval_ms).
-define(RAFT_COMMIT_BATCH_INTERVAL(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_COMMIT_BATCH_INTERVAL, 2)).
%% Maximum number of pending log entries to collect before a heartbeat is forced. This should
%% be at most equal to the maximum number of log entries permitted per heartbeat.
-define(RAFT_COMMIT_BATCH_MAX_ENTRIES, raft_commit_batch_max).
-define(RAFT_COMMIT_BATCH_MAX_ENTRIES(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_COMMIT_BATCH_MAX_ENTRIES, 15)).
%% Maximum number of log entries to speculatively retain in the log due to followers
%% not yet reporting having replicated the log entry locally
-define(RAFT_MAX_RETAINED_ENTRIES, raft_max_retained_entries).
-define(RAFT_MAX_RETAINED_ENTRIES(App, Table), ?RAFT_TABLE_CONFIG(App, Table, {?RAFT_MAX_RETAINED_ENTRIES, max_log_rotate_delay}, 1500000)).
%% Maximum number of log entries to queue for application by storage at once before
%% continuing to process the incoming message queue on the RAFT server.
-define(RAFT_MAX_CONSECUTIVE_APPLY_ENTRIES, raft_apply_log_batch_size).
-define(RAFT_MAX_CONSECUTIVE_APPLY_ENTRIES(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_MAX_CONSECUTIVE_APPLY_ENTRIES, 200)).
%% Maximum bytes of log entries to queue for application by storage at once before
%% continuing to process the incoming message queue on the RAFT server.
-define(RAFT_MAX_CONSECUTIVE_APPLY_BYTES, raft_apply_batch_max_bytes).
-define(RAFT_MAX_CONSECUTIVE_APPLY_BYTES(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_MAX_CONSECUTIVE_APPLY_BYTES, 200 * 4 * 1024)).
%% Minimum time in milliseconds since the receiving the last valid leader heartbeat
%% before triggering a new election due to term timeout. This time should be much
%% greater than the maximum expected network delay.
-define(RAFT_ELECTION_TIMEOUT_MIN, raft_election_timeout_ms).
-define(RAFT_ELECTION_TIMEOUT_MIN(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_ELECTION_TIMEOUT_MIN, 5000)).
%% Maximum time in milliseconds since the receiving the last valid leader heartbeat
%% before triggering a new election due to term timeout. The difference between this
%% time and the minimum election timeout should be much greater than the expected
%% variance in network delay.
-define(RAFT_ELECTION_TIMEOUT_MAX, raft_election_timeout_ms_max).
-define(RAFT_ELECTION_TIMEOUT_MAX(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_ELECTION_TIMEOUT_MAX, 7500)).
%% The maximum time in milliseconds during which a leader can continue to be considered live without
%% receiving an updated heartbeat response quorum from replicas or during which a follower or witness
%% can be considered live without receiving a heartbeat from a valid leader of the current term.
-define(RAFT_LIVENESS_GRACE_PERIOD_MS, raft_liveness_grace_period_ms).
-define(RAFT_LIVENESS_GRACE_PERIOD_MS(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_LIVENESS_GRACE_PERIOD_MS, 30_000)).
%% The maximum number of log entries that can be not yet applied to a follower or witnesse's log
%% compared to the leader's commit index before the replica is considered stale.
-define(RAFT_STALE_GRACE_PERIOD_ENTRIES, raft_stale_grace_period_entries).
-define(RAFT_STALE_GRACE_PERIOD_ENTRIES(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_STALE_GRACE_PERIOD_ENTRIES, 5_000)).
%% Minium amount of time in seconds since the last successfully received
%% heartbeat from a leader of a term for non-forced promotion to be allowed.
-define(RAFT_PROMOTION_GRACE_PERIOD, raft_promotion_grace_period_secs).
-define(RAFT_PROMOTION_GRACE_PERIOD(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_PROMOTION_GRACE_PERIOD, 60)).
%% Maximum number of log entries to include in a Handover RPC to pass
%% leadership to another peer. A limit is enforced to prevent a handover
%% trying to send huge numbers of logs to catchup a peer during handover.
-define(RAFT_HANDOVER_MAX_ENTRIES, raft_max_handover_log_entries).
-define(RAFT_HANDOVER_MAX_ENTRIES(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_HANDOVER_MAX_ENTRIES, 200)).
%% Maximum number of total log entries from the leader's current log that a
%% peer has not yet confirmed to be applied. This limit helps prevent nodes who
%% may have already received all the current log entries but are behind in
%% actually applying them to the underlying storage from becoming leader due to
%% handover before they are ready. This defaults to equal to the maximum number
%% of missing log entries. (See `?RAFT_HANDOVER_MAX_ENTRIES`.)
-define(RAFT_HANDOVER_MAX_UNAPPLIED_ENTRIES, raft_handover_max_unapplied_entries).
-define(RAFT_HANDOVER_MAX_UNAPPLIED_ENTRIES(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_HANDOVER_MAX_UNAPPLIED_ENTRIES, undefined)).
%% Maximum total byte size of log entries to include in a Handover RPC.
-define(RAFT_HANDOVER_MAX_BYTES, raft_max_handover_log_size).
-define(RAFT_HANDOVER_MAX_BYTES(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_HANDOVER_MAX_BYTES, 50 * 1024 * 1024)).
%% Time in milliseconds to wait before considering a previously triggered handover failed.
-define(RAFT_HANDOVER_TIMEOUT, raft_handover_timeout_ms).
-define(RAFT_HANDOVER_TIMEOUT(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_HANDOVER_TIMEOUT, 600)).
%% Minimum nubmer of log entries past the minimum kept by the RAFT server before triggering
%% log rotation
-define(RAFT_LOG_ROTATION_INTERVAL, raft_max_log_records_per_file).
-define(RAFT_LOG_ROTATION_INTERVAL(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_LOG_ROTATION_INTERVAL, 200000)).
%% Maximum number of log entries past the minimum kept by the RAFT server to retain in
%% the log after rotation
-define(RAFT_LOG_ROTATION_KEEP, raft_max_log_records).
-define(RAFT_LOG_ROTATION_KEEP(App, Table, Interval), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_LOG_ROTATION_KEEP, Interval * 10)).
%% Whether log rotation should be controlled by local log length or by
%% leader-announced cluster trimming index
-define(RAFT_LOG_ROTATION_BY_TRIM_INDEX, raft_rotate_by_trim_index).
-define(RAFT_LOG_ROTATION_BY_TRIM_INDEX(App, Table), (?RAFT_TABLE_CONFIG(App, Table, {?RAFT_LOG_ROTATION_BY_TRIM_INDEX, use_trim_index}, false) =:= true)).
%% Whether or not the log should return entries in external term format
%% when log entries are fetched for heartbeats
-define(RAFT_LOG_HEARTBEAT_BINARY_ENTRIES, raft_log_heartbeat_binary_entries).
-define(RAFT_LOG_HEARTBEAT_BINARY_ENTRIES(App, Table),
(?RAFT_TABLE_CONFIG(App, Table, ?RAFT_LOG_HEARTBEAT_BINARY_ENTRIES, false) =:= true)
).
%% The number of log entries that have yet to be applied on a follower after
%% which leaders should send a storage snapshot in lieu of continuing regular
%% replication using log entries in heartbeats.
-define(RAFT_SNAPSHOT_CATCHUP_THRESHOLD, raft_snapshot_catchup_threshold).
-define(RAFT_SNAPSHOT_CATCHUP_THRESHOLD(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_SNAPSHOT_CATCHUP_THRESHOLD, 100000)).
%% Number of milliseconds to wait before attempting to send a new storage snapshot
%% to a follower that previously rejected a snapshot due to being overloaded.
-define(RAFT_SNAPSHOT_CATCHUP_OVERLOADED_BACKOFF_MS, raft_snapshot_catchup_overloaded_backoff_ms).
-define(RAFT_SNAPSHOT_CATCHUP_OVERLOADED_BACKOFF_MS(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_SNAPSHOT_CATCHUP_OVERLOADED_BACKOFF_MS, 1000)).
%% Number of milliseconds to wait before attempting to send a new storage snapshot
%% to a follower that previously successfully received a storage snapshot.
-define(RAFT_SNAPSHOT_CATCHUP_COMPLETED_BACKOFF_MS, raft_snapshot_catchup_completed_backoff_ms).
-define(RAFT_SNAPSHOT_CATCHUP_COMPLETED_BACKOFF_MS(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_SNAPSHOT_CATCHUP_COMPLETED_BACKOFF_MS, 20 * 1000)).
%% Number of milliseconds to wait before attempting to send a new storage snapshot
%% to a follower that previously failed to receive a storage snapshot.
-define(RAFT_SNAPSHOT_CATCHUP_FAILED_BACKOFF_MS, raft_snapshot_catchup_failed_backoff_ms).
-define(RAFT_SNAPSHOT_CATCHUP_FAILED_BACKOFF_MS(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_SNAPSHOT_CATCHUP_FAILED_BACKOFF_MS, 10 * 1000)).
%% Number of omitted log entries to skip actually applying to storage when
%% operating as a witness.
-define(RAFT_STORAGE_WITNESS_APPLY_INTERVAL, raft_storage_witness_apply_interval).
-define(RAFT_STORAGE_WITNESS_APPLY_INTERVAL(App, Table), ?RAFT_TABLE_CONFIG(App, Table, ?RAFT_STORAGE_WITNESS_APPLY_INTERVAL, 5000)).
%% Whether or not the storage server should request more log entries
%% when the apply queue is empty.
-define(RAFT_STORAGE_NOTIFY_COMPLETE, raft_storage_notify_complete).
-define(RAFT_STORAGE_NOTIFY_COMPLETE(App, Table), (?RAFT_TABLE_CONFIG(App, Table, ?RAFT_STORAGE_NOTIFY_COMPLETE, true) =:= true)).
%%-------------------------------------------------------------------
%% Records
%%-------------------------------------------------------------------
%% Log position
-record(raft_log_pos, {
%% log sequence number
index = 0 :: wa_raft_log:log_index(),
%% leader's term when log entry is created
term = 0 :: wa_raft_log:log_term()
}).
%% Log handle.
-record(raft_log, {
name :: wa_raft_log:log_name(),
application :: atom(),
table :: wa_raft:table(),
partition :: wa_raft:partition(),
provider :: module()
}).
%% This record represents the identity of a RAFT replica, usable to
%% distinguish different RAFT replicas from one another. This record
%% is not guaranteed to remain structurally compatible between versions
%% of RAFT and so should not be persisted between runtimes nor sent
%% between RAFT servers. It is generally allowed to inspect the fields
%% of this record, however, similarly, this record is subject to change
%% at any time.
-record(raft_identity, {
% The service name (registered name) of the RAFT server that this
% identity record refers to.
name :: atom(),
% The node that the RAFT server that this identity record refers
% to is located on.
node :: node()
}).
%% This record represents a RAFT instance identifier.
-record(raft_identifier, {
application :: atom(),
table :: wa_raft:table(),
partition :: wa_raft:partition()
}).
%%-------------------------------------------------------------------
%% Records for registered application and partition information
%%-------------------------------------------------------------------
%% Information about an application that has started a RAFT supervisor.
-record(raft_application, {
% Application name
name :: atom(),
% Config search path
config_search_apps :: [atom()]
}).
%% Normalized options produced by `wa_raft_part_sup` for passing into RAFT processes.
%% Not to be created externally.
-record(raft_options, {
% General options
application :: atom(),
table :: wa_raft:table(),
partition :: wa_raft:partition(),
self :: #raft_identity{},
identifier :: #raft_identifier{},
database :: file:filename(),
% Acceptor options
acceptor_name :: atom(),
% Distribution options
distribution_module :: module(),
% Label options
label_module :: undefined | module(),
% Log options
log_name :: atom(),
log_module :: module(),
% Queue options
queue_name :: atom(),
queue_counters :: atomics:atomics_ref(),
queue_reads :: atom(),
% Server options
server_name :: atom(),
% Storage options
storage_name :: atom(),
storage_module :: module(),
% Partition supervisor options
supervisor_name :: atom(),
% Transport options
transport_cleanup_name :: atom(),
transport_directory :: file:filename(),
transport_module :: module()
}).
%%-------------------------------------------------------------------
%% Internal server states
%%-------------------------------------------------------------------
%% Raft runtime state
-record(raft_state, {
%% Owning application
application :: atom(),
%% RAFT server name
name :: atom(),
%% RAFT server's cluster identity
self :: #raft_identity{},
%% RAFT replica's local identifier
identifier :: #raft_identifier{},
%% Table name
table :: wa_raft:table(),
%% Partition number
partition :: wa_raft:partition(),
%% Local path to partition data
partition_path :: string(),
%% Current view into this RAFT replica's log state
log_view :: wa_raft_log:view(),
%% Current queue handle
queues :: wa_raft_queue:queues(),
%% Active module for distribution of RPCs
distribution_module :: module(),
%% Active module for labeling of log entries
label_module :: module() | undefined,
%% Name of this RAFT replica's storage server
storage :: atom(),
%% The index of the latest log entry in the local log that is known to
%% match the log entries committed by the cluster
commit_index = 0 :: non_neg_integer(),
%% The index of the latest log entry that has been sent to storage to be
%% applied
last_applied = 0 :: non_neg_integer(),
%% The most recently written RAFT configuration and the index at which it
%% was written if a configuration exists in storage
cached_config :: undefined | {wa_raft_log:log_index(), wa_raft_server:config()},
%% [Leader] The label of the last log entry in the current log
last_label :: undefined | term(),
%% The timestamp (milliseconds monotonic clock) of the most recently
%% received (follower) or sent (leader) heartbeat.
leader_heartbeat_ts :: undefined | integer(),
%% The largest RAFT term that has been observed in the cluster or reached
%% by this RAFT replica
current_term = 0 :: non_neg_integer(),
%% The peer that this RAFT replica voted for in the current term
voted_for :: undefined | node(),
%% The affirmative votes for leadership this RAFT replica has received from
%% the cluster in the current term
votes = #{} :: #{node() => true},
%% The leader of the current RAFT term if known
leader_id :: undefined | node(),
%% The timestamp (milliseconds monotonic clock) that the current state of
%% this RAFT replica was reached
state_start_ts :: non_neg_integer(),
%% [Leader] The list of pending operations in the current commit batch
%% that are in queue to be appended and replicated after a short
%% wait to see if multiple commits can be handled at once to
%% reduce overhead
pending_high = [] :: [{gen_server:from(), wa_raft_acceptor:op()}],
pending_low = [] :: [{gen_server:from(), wa_raft_acceptor:op()}],
%% [Leader] Whether or not a read has been accepted and is waiting for the
%% leader to establish a new quorum to be handled.
pending_read = false :: boolean(),
%% [Leader] The queue of accepted commit requests that are waiting to be
%% committed and applied for response to the client.
queued = #{} :: #{wa_raft_log:log_index() => {gen_server:from(), wa_raft_acceptor:priority()}},
%% [Leader] The index of the next log entry to send in the next heartbeat
%% to each peer
next_indices = #{} :: #{node() => wa_raft_log:log_index()},
%% [Leader] The index of the latest log entry in each peer's log that is
%% confirmed by a heartbeat response to match the local log
match_indices = #{} :: #{node() => wa_raft_log:log_index()},
%% [Leader] The index of the latest log entry that has been applied to
%% each peer's underlying storage state
last_applied_indices = #{} :: #{node() => wa_raft_log:log_index()},
%% [Leader] The timestamp (milliseconds monotonic clock) of the last time
%% each peer was sent a heartbeat
last_heartbeat_ts = #{} :: #{node() => integer()},
%% [Leader] The timestamp (milliseconds monotonic clock) of the last time
%% each peer responded to this RAFT replica with a heartbeat
%% response
heartbeat_response_ts = #{} :: #{node() => integer()},
%% [Leader] The log index of the first log entry appended to the log that
%% has a log term matching the current term
first_current_term_log_index = 0 :: wa_raft_log:log_index(),
%% [Leader] Information about a currently pending handover of leadership to
%% a peer
handover :: undefined | {node(), reference(), integer()},
%% [Disabled] The reason for which this RAFT replica was disabled
disable_reason :: term()
}).
================================================
FILE: include/wa_raft_logger.hrl
================================================
%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%%
%%% This source code is licensed under the Apache 2.0 license found in
%%% the LICENSE file in the root directory of this source tree.
%%%
-include_lib("kernel/include/logger.hrl").
-define(RAFT_LOG_OPTS, #{domain => [whatsapp, wa_raft]}).
-define(RAFT_LOG_ERROR(Message), ?LOG_ERROR(Message, ?RAFT_LOG_OPTS)).
-define(RAFT_LOG_ERROR(Format, Args), ?LOG_ERROR(Format, Args, ?RAFT_LOG_OPTS)).
-define(RAFT_LOG_WARNING(Message), ?LOG_WARNING(Message, ?RAFT_LOG_OPTS)).
-define(RAFT_LOG_WARNING(Format, Args), ?LOG_WARNING(Format, Args, ?RAFT_LOG_OPTS)).
-define(RAFT_LOG_NOTICE(Message), ?LOG_NOTICE(Message, ?RAFT_LOG_OPTS)).
-define(RAFT_LOG_NOTICE(Format, Args), ?LOG_NOTICE(Format, Args, ?RAFT_LOG_OPTS)).
-define(RAFT_LOG_INFO(Message), ?LOG_INFO(Message, ?RAFT_LOG_OPTS)).
-define(RAFT_LOG_INFO(Format, Args), ?LOG_INFO(Format, Args, ?RAFT_LOG_OPTS)).
-define(RAFT_LOG_DEBUG(Message), ?LOG_DEBUG(Message, ?RAFT_LOG_OPTS)).
-define(RAFT_LOG_DEBUG(Format, Args), ?LOG_DEBUG(Format, Args, ?RAFT_LOG_OPTS)).
-define(RAFT_LOG(Level, Message), ?LOG(Level, Message, ?RAFT_LOG_OPTS)).
-define(RAFT_LOG(Level, Format, Args), ?LOG(Level, Format, Args, ?RAFT_LOG_OPTS)).
================================================
FILE: include/wa_raft_rpc.hrl
================================================
%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%%
%%% This source code is licensed under the Apache 2.0 license found in
%%% the LICENSE file in the root directory of this source tree.
%%%
%%% This file contains macros defining the form of all RPCs and API
%%% calls used as part of the RAFT protocol and RAFT server and storage API.
%%-------------------------------------------------------------------
%% RAFT Server RPC Formats
%%-------------------------------------------------------------------
%% As the RAFT process that is intended to performs the cross-node
%% communication required to provide durability against failure,
%% RAFT servers across nodes must agree on the RPC formats in use.
%% This means that RPC formats should not be changed once created.
%%-------------------------------------------------------------------
-define(RAFT_NAMED_RPC(Type, Term, SenderName, SenderNode, Payload), {rpc, Type, Term, SenderName, SenderNode, Payload}).
%% These two RPCs are used by RAFT catchup to receive the status of
%% the RAFT server being sent to and should not change.
-define(LEGACY_RAFT_RPC(Type, Term, SenderId, Payload), {rpc, Type, Term, SenderId, Payload}).
-define(LEGACY_APPEND_ENTRIES_RESPONSE_RPC(Term, SenderId, PrevLogIndex, Success, LastIndex),
?LEGACY_RAFT_RPC(append_entries_response, Term, SenderId, {PrevLogIndex, Success, LastIndex})).
%%-------------------------------------------------------------------
%% RAFT Server Procedures
%%-------------------------------------------------------------------
%% An RPC received from a peer is intended to trigger one of the
%% procedures listed below.
%%-------------------------------------------------------------------
-define(APPEND_ENTRIES, append_entries).
-define(APPEND_ENTRIES_RESPONSE, append_entries_response).
-define(REQUEST_VOTE, request_vote).
-define(VOTE, vote).
-define(HANDOVER, handover).
-define(HANDOVER_FAILED, handover_failed).
-define(NOTIFY_TERM, notify_term).
%% Definitions of each of the standard procedures.
-define(PROCEDURE(Type, Payload), {procedure, Type, Payload}).
-define(APPEND_ENTRIES(PrevLogIndex, PrevLogTerm, Entries, CommitIndex, TrimIndex), ?PROCEDURE(?APPEND_ENTRIES, {PrevLogIndex, PrevLogTerm, Entries, CommitIndex, TrimIndex})).
-define(APPEND_ENTRIES_RESPONSE(PrevLogIndex, Success, MatchIndex, LastAppliedIndex), ?PROCEDURE(?APPEND_ENTRIES_RESPONSE, {PrevLogIndex, Success, MatchIndex, LastAppliedIndex})).
-define(REQUEST_VOTE(ElectionType, LastLogIndex, LastLogTerm), ?PROCEDURE(?REQUEST_VOTE, {ElectionType, LastLogIndex, LastLogTerm})).
-define(VOTE(Vote), ?PROCEDURE(?VOTE, {Vote})).
-define(HANDOVER(Ref, PrevLogIndex, PrevLogTerm, Entries), ?PROCEDURE(?HANDOVER, {Ref, PrevLogIndex, PrevLogTerm, Entries})).
-define(HANDOVER_FAILED(Ref), ?PROCEDURE(?HANDOVER_FAILED, {Ref})).
-define(NOTIFY_TERM(), ?PROCEDURE(?NOTIFY_TERM, {})).
%% A request to execute a particular procedure. This request could
%% have been issued locally or as a result of a remote procedure
%% call. The peer (if exists and could be oneself) that issued the
%% procedure call will be provided as the sender.
-define(REMOTE(Sender, Call), {remote, Sender, Call}).
%%-------------------------------------------------------------------
%% RAFT Server Internal Events
%%-------------------------------------------------------------------
%% An event produced internally within the RAFT server.
%%-------------------------------------------------------------------
-define(ADVANCE_TERM(Term), {advance_term, Term}).
-define(FORCE_ELECTION(Term), {force_election, Term}).
%%-------------------------------------------------------------------
%% RAFT Server API
%%-------------------------------------------------------------------
%% The RAFT server also accepts commands issued from other processes
%% on the local node. These commands are not guaranteed to have the
%% same format between versions and so should only be used locally.
%% Prefer to use `wa_raft_server` module exports when possible.
%%-------------------------------------------------------------------
-define(RAFT_COMMAND(Type, Payload), {command, Type, Payload}).
-define(COMMIT_COMMAND(From, Op, Priority), ?RAFT_COMMAND(commit, {From, Op, Priority})).
-define(READ_COMMAND(Op), ?RAFT_COMMAND(read, Op)).
-define(CURRENT_CONFIG_COMMAND, ?RAFT_COMMAND(current_config, undefined)).
-define(STATUS_COMMAND, ?RAFT_COMMAND(status, undefined)).
-define(TRIGGER_ELECTION_COMMAND(TermOrOffset), ?RAFT_COMMAND(trigger_election, {TermOrOffset})).
-define(PROMOTE_COMMAND(TermOrOffset, Force), ?RAFT_COMMAND(promote, {TermOrOffset, Force})).
-define(RESIGN_COMMAND, ?RAFT_COMMAND(resign, undefined)).
-define(ADJUST_CONFIG_COMMAND(Action, Index), ?ADJUST_CONFIG_COMMAND(undefined, Action, Index)).
-define(ADJUST_CONFIG_COMMAND(From, Action, Index), ?RAFT_COMMAND(adjust_config, {From, Action, Index})).
-define(REFRESH_CONFIG_COMMAND(), ?RAFT_COMMAND(refresh_config, undefined)).
-define(SNAPSHOT_AVAILABLE_COMMAND(Root, Position), ?RAFT_COMMAND(snapshot_available, {Root, Position})).
-define(HANDOVER_CANDIDATES_COMMAND, ?RAFT_COMMAND(handover_candidates, undefined)).
-define(HANDOVER_COMMAND(Peer), ?RAFT_COMMAND(handover, Peer)).
-define(IS_PEER_READY_COMMAND(Peer), ?RAFT_COMMAND(is_peer_ready, Peer)).
-define(ENABLE_COMMAND, ?RAFT_COMMAND(enable, undefined)).
-define(DISABLE_COMMAND(Reason), ?RAFT_COMMAND(disable, Reason)).
-define(BOOTSTRAP_COMMAND(Position, Config, Data), ?RAFT_COMMAND(bootstrap, {Position, Config, Data})).
-define(NOTIFY_COMPLETE_COMMAND(), ?RAFT_COMMAND(notify_complete, undefined)).
================================================
FILE: rebar.config
================================================
{erl_opts, [ debug_info
, warnings_as_errors
, warn_export_vars
, warn_unused_import
]
}.
{deps, []}.
{dialyzer, [ {warnings, [unknown]}
, {plt_apps, all_deps}
]}.
{xref_checks, [ undefined_function_calls
, undefined_functions
, locals_not_used
, deprecated_function_calls
, deprecated_functions
]}.
================================================
FILE: src/wa_raft.app.src
================================================
%% % @format
%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%%
%%% This source code is licensed under the Apache 2.0 license found in
%%% the LICENSE file in the root directory of this source tree.
{application, wa_raft, [
{description, "Erlang implementation of RAFT Consensus Protocol"},
{vsn, "1.0.0"},
{modules, []},
%% NOTE: No more dependency is expected for this app
{applications, [
kernel,
stdlib
]},
{env, []},
{mod, {wa_raft_app, []}}
]}.
================================================
FILE: src/wa_raft.erl
================================================
%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%%
%%% This source code is licensed under the Apache 2.0 license found in
%%% the LICENSE file in the root directory of this source tree.
%%%
%%% This file defines dialyzer types.
-module(wa_raft).
-compile(warn_missing_spec_all).
-include_lib("wa_raft/include/wa_raft.hrl").
%% Public Types
-export_type([
table/0,
partition/0,
args/0,
identity/0
]).
-type table() :: atom().
-type partition() :: pos_integer().
%% Specification for starting a RAFT partition.
-type args() ::
#{
% Table name
table := table(),
% Partition number
partition := partition(),
% Distribution module
distribution_module => module(),
% Log module
log_module => module(),
% Log label module
label_module => module(),
% Storage module
storage_module => module(),
% Transport module
transport_module => module()
}.
-type identity() :: #raft_identity{}.
================================================
FILE: src/wa_raft_acceptor.erl
================================================
%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%%
%%% This source code is licensed under the Apache 2.0 license found in
%%% the LICENSE file in the root directory of this source tree.
%%%
%%% This module implements the front-end process for accepting commits / reads
-module(wa_raft_acceptor).
-compile(warn_missing_spec_all).
-behaviour(gen_server).
%% OTP supervisor
-export([
child_spec/1,
start_link/1
]).
%% Client API - data access
-export([
commit/2,
commit/3,
commit/4,
commit_async/3,
commit_async/4,
read/2,
read/3
]).
%% Client API - RAFT apis
-export([
adjust_config/3,
adjust_config_async/3,
adjust_config_async/4
]).
%% Internal API
-export([
default_name/2,
registered_name/2
]).
%% gen_server callbacks
-export([
init/1,
handle_call/3,
handle_cast/2,
terminate/2
]).
-export_type([
command/0,
key/0,
op/0,
read_op/0,
priority/0
]).
-export_type([
call_error_type/0,
call_error/0,
call_result/0,
read_error/0,
read_error_type/0,
read_result/0,
commit_error_type/0,
commit_error/0,
commit_result/0
]).
-include_lib("wa_raft/include/wa_raft.hrl").
-include_lib("wa_raft/include/wa_raft_logger.hrl").
%% Request type macros
-define(READ_REQUEST(Command), {read, Command}).
-define(COMMIT_REQUEST(Op, Priority), {commit, Op, Priority}).
-define(COMMIT_ASYNC_REQUEST(From, Op, Priority), {commit, From, Op, Priority}).
%% Commit op type macros
-define(OP_DEFAULT(Op), {default, Op}).
-define(OP_ADJUST_CONFIG(Action, Index), {adjust_config, Action, Index}).
-type command() :: noop_command() | noop_omitted_command() | config_command() | dynamic().
-type noop_command() :: noop.
-type noop_omitted_command() :: noop_omitted.
-type config_command() :: {config, Config :: wa_raft_server:config()}.
-type key() :: term().
-type op() :: {Key :: key(), Command :: command()}.
-type read_op() :: {From :: gen_server:from(), Command :: command()}.
-type priority() :: high | low.
-type call_error_type() :: timeout | unreachable | {call_error, Reason :: term()}.
-type call_error() :: {error, call_error_type()}.
-type call_result() :: Result :: dynamic() | Error :: call_error().
-type read_request() :: ?READ_REQUEST(Command :: command()).
-type read_error_type() :: not_leader | read_queue_full | apply_queue_full | {notify_redirect, Peer :: node()}.
-type read_error() :: {error, read_error_type()}.
-type read_result() :: Result :: dynamic() | Error :: read_error() | call_error().
-type commit_op() :: default_op() | adjust_config_op().
-type default_op() :: ?OP_DEFAULT(Op :: op()).
-type adjust_config_op() :: ?OP_ADJUST_CONFIG(Action :: wa_raft_server:config_action(), Index :: wa_raft_log:log_index() | undefined).
-type commit_request() :: ?COMMIT_REQUEST(Op :: commit_op(), Priority :: priority()).
-type commit_async_request() :: ?COMMIT_ASYNC_REQUEST(From :: gen_server:from(), Op :: commit_op(), Priority :: priority()).
-type commit_error_type() ::
not_supported |
not_leader |
commit_queue_full |
apply_queue_full |
{notify_redirect, Peer :: node()} |
commit_stalled |
cancelled.
-type commit_error() :: {error, commit_error_type()}.
-type commit_result() :: Result :: dynamic() | Error :: commit_error() | call_error().
%% Acceptor state
-record(state, {
% Acceptor service name
name :: atom(),
% RAFT table
table :: wa_raft:table(),
% Server service name
server :: atom(),
% Queues handle
queues :: wa_raft_queue:queues()
}).
%%-------------------------------------------------------------------
%% OTP Supervision
%%-------------------------------------------------------------------
%%-------------------------------------------------------------------
%% OTP Supervision
%%-------------------------------------------------------------------
-spec child_spec(Options :: #raft_options{}) -> supervisor:child_spec().
child_spec(Options) ->
#{
id => ?MODULE,
start => {?MODULE, start_link, [Options]},
restart => transient,
shutdown => 30000,
modules => [?MODULE]
}.
-spec start_link(Options :: #raft_options{}) -> gen_server:start_ret().
start_link(#raft_options{acceptor_name = Name} = Options) ->
gen_server:start_link({local, Name}, ?MODULE, Options, []).
%%-------------------------------------------------------------------
%% Public API
%%-------------------------------------------------------------------
%% Request that the specified RAFT server commit the provided command. The commit can only be
%% successful if the requested RAFT server is the active leader of the RAFT partition it is a
%% part of. Returns either the result returned by the storage module when applying the command
%% or an error indicating some reason for which the command was not able to be committed or
%% should be retried.
-spec commit(ServerRef :: gen_server:server_ref(), Op :: op()) -> commit_result().
commit(ServerRef, Op) ->
commit(ServerRef, Op, ?RAFT_RPC_CALL_TIMEOUT()).
-spec commit(ServerRef :: gen_server:server_ref(), Op :: op(), Timeout :: timeout()) -> commit_result().
commit(ServerRef, Op, Timeout) ->
commit(ServerRef, Op, Timeout, high).
-spec commit(ServerRef :: gen_server:server_ref(), Op :: op(), Timeout :: timeout(), Priority :: priority()) -> commit_result().
commit(ServerRef, Op, Timeout, Priority) ->
call(ServerRef, ?COMMIT_REQUEST(?OP_DEFAULT(Op), Priority), Timeout).
-spec commit_async(ServerRef :: gen_server:server_ref(), From :: {pid(), term()}, Op :: op()) -> ok.
commit_async(ServerRef, From, Op) ->
commit_async(ServerRef, From, Op, high).
-spec commit_async(ServerRef :: gen_server:server_ref(), From :: {pid(), term()}, Op :: op(), Priority :: priority()) -> ok.
commit_async(ServerRef, From, Op, Priority) ->
gen_server:cast(ServerRef, ?COMMIT_ASYNC_REQUEST(From, ?OP_DEFAULT(Op), Priority)).
% Strong-read
-spec read(ServerRef :: gen_server:server_ref(), Command :: command()) -> read_result().
read(ServerRef, Command) ->
read(ServerRef, Command, ?RAFT_RPC_CALL_TIMEOUT()).
-spec read(ServerRef :: gen_server:server_ref(), Command :: command(), Timeout :: timeout()) -> read_result().
read(ServerRef, Command, Timeout) ->
call(ServerRef, ?READ_REQUEST(Command), Timeout).
-spec adjust_config(
ServerRef :: gen_server:server_ref(),
Action :: wa_raft_server:config_action(),
Index :: wa_raft_log:log_index() | undefined
) -> commit_result().
adjust_config(ServerRef, Action, Index) ->
adjust_config(ServerRef, Action, Index, ?RAFT_RPC_CALL_TIMEOUT()).
-spec adjust_config(
ServerRef :: gen_server:server_ref(),
Action :: wa_raft_server:config_action(),
Index :: wa_raft_log:log_index() | undefined,
Timeout :: timeout()
) -> commit_result().
adjust_config(ServerRef, Action, Index, Timeout) ->
call(ServerRef, ?COMMIT_REQUEST(?OP_ADJUST_CONFIG(Action, Index), high), Timeout).
-spec adjust_config_async(
ServerRef :: gen_server:server_ref(),
From :: gen_server:from(),
Action :: wa_raft_server:config_action()
) -> ok.
adjust_config_async(ServerRef, From, Action) ->
adjust_config_async(ServerRef, From, Action, undefined).
-spec adjust_config_async(
ServerRef :: gen_server:server_ref(),
From :: gen_server:from(),
Action :: wa_raft_server:config_action(),
Index :: wa_raft_log:log_index() | undefined
) -> ok.
adjust_config_async(ServerRef, From, Action, Index) ->
gen_server:cast(ServerRef, ?COMMIT_ASYNC_REQUEST(From, ?OP_ADJUST_CONFIG(Action, Index), high)).
-spec call(ServerRef :: gen_server:server_ref(), Request :: term(), Timeout :: timeout()) -> call_result().
call(ServerRef, Request, Timeout) ->
try
gen_server:call(ServerRef, Request, Timeout)
catch
exit:{timeout, _} -> {error, timeout};
exit:{noproc, _} -> {error, unreachable};
exit:{{nodedown, _}, _} -> {error, unreachable};
exit:{shutdown, _} -> {error, unreachable};
exit:{Other, _} -> {error, {call_error, Other}}
end.
%%-------------------------------------------------------------------
%% Internal API
%%-------------------------------------------------------------------
%% Get the default name for the RAFT acceptor server associated with the
%% provided RAFT partition.
-spec default_name(Table :: wa_raft:table(), Partition :: wa_raft:partition()) -> Name :: atom().
default_name(Table, Partition) ->
% elp:ignore W0023 bounded atom, one per table/partition at startup
binary_to_atom(<<"raft_acceptor_", (atom_to_binary(Table))/binary, "_", (integer_to_binary(Partition))/binary>>).
%% Get the registered name for the RAFT acceptor server associated with the
%% provided RAFT partition or the default name if no registration exists.
-spec registered_name(Table :: wa_raft:table(), Partition :: wa_raft:partition()) -> Name :: atom().
registered_name(Table, Partition) ->
case wa_raft_part_sup:options(Table, Partition) of
undefined -> default_name(Table, Partition);
Options -> Options#raft_options.acceptor_name
end.
%%-------------------------------------------------------------------
%% RAFT Acceptor - Server Callbacks
%%-------------------------------------------------------------------
-spec init(Options :: #raft_options{}) -> {ok, #state{}}.
init(#raft_options{table = Table, partition = Partition, acceptor_name = Name, server_name = Server} = Options) ->
process_flag(trap_exit, true),
?RAFT_LOG_NOTICE("Acceptor[~0p] starting for partition ~0p/~0p", [Name, Table, Partition]),
{ok, #state{
name = Name,
table = Table,
server = Server,
queues = wa_raft_queue:queues(Options)
}}.
-spec handle_call(read_request(), gen_server:from(), #state{}) -> {reply, read_result(), #state{}} | {noreply, #state{}};
(commit_request(), gen_server:from(), #state{}) -> {reply, commit_result(), #state{}} | {noreply, #state{}}.
handle_call(?READ_REQUEST(Command), From, State) ->
case read_impl(From, Command, State) of
continue -> {noreply, State};
{error, _} = Error -> {reply, Error, State}
end;
handle_call(?COMMIT_REQUEST(Op, Priority), From, State) ->
case commit_impl(From, Op, Priority, State) of
continue -> {noreply, State};
{error, _} = Error -> {reply, Error, State}
end;
handle_call(Request, From, #state{name = Name} = State) ->
?RAFT_LOG_ERROR("Acceptor[~0p] received unexpected call ~0P from ~0p.", [Name, Request, 30, From]),
{noreply, State}.
-spec handle_cast(commit_async_request(), #state{}) -> {noreply, #state{}}.
handle_cast(?COMMIT_ASYNC_REQUEST(From, Op, Priority), State) ->
Result = commit_impl(From, Op, Priority, State),
Result =/= continue andalso gen_server:reply(From, Result),
{noreply, State};
handle_cast(Request, #state{name = Name} = State) ->
?RAFT_LOG_ERROR("Acceptor[~0p] received unexpected cast ~0P.", [Name, Request, 30]),
{noreply, State}.
-spec terminate(Reason :: term(), State :: #state{}) -> ok.
terminate(Reason, #state{name = Name}) ->
?RAFT_LOG_NOTICE("Acceptor[~0p] terminating with reason ~0P", [Name, Reason, 30]),
ok.
%%-------------------------------------------------------------------
%% RAFT Acceptor - Implementations
%%-------------------------------------------------------------------
%% Enqueue a commit.
-spec commit_impl(
From :: gen_server:from(),
CommitOp :: commit_op(),
Priority :: priority(),
State :: #state{}
) -> continue | commit_error().
commit_impl(From, CommitOp, Priority, #state{table = Table, name = Name, server = Server, queues = Queues}) ->
StartTUsec = erlang:monotonic_time(microsecond),
?RAFT_LOG_DEBUG("Acceptor[~0p] starts to handle commit of ~0P from ~0p.", [Name, CommitOp, 30, From]),
try
case wa_raft_queue:commit_started(Queues, Priority) of
ok ->
case CommitOp of
?OP_DEFAULT(Op) ->
wa_raft_server:commit(Server, From, Op, Priority),
continue;
?OP_ADJUST_CONFIG(Action, Index) ->
wa_raft_server:adjust_config(Server, From, Action, Index),
continue;
_ ->
?RAFT_LOG_WARNING(
"Acceptor[~0p] does not know how to handle commit op ~0P.",
[Name, CommitOp, 20]
),
{error, not_supported}
end;
Reason ->
?RAFT_COUNT(Table, {'acceptor.error', Reason, Priority}),
?RAFT_LOG_WARNING(
"Acceptor[~0p] is rejecting commit request from ~0p due to ~0p.",
[Name, From, Reason]
),
{error, Reason}
end
after
?RAFT_GATHER(Table, 'acceptor.commit.func', erlang:monotonic_time(microsecond) - StartTUsec)
end.
%% Enqueue a strongly-consistent read.
-spec read_impl(gen_server:from(), command(), #state{}) -> continue | read_error().
read_impl(From, Command, #state{table = Table, name = Name, server = Server, queues = Queues}) ->
StartTUsec = erlang:monotonic_time(microsecond),
?RAFT_LOG_DEBUG("Acceptor[~p] starts to handle read of ~0P from ~0p.", [Name, Command, 100, From]),
try
case wa_raft_queue:reserve_read(Queues) of
ok ->
wa_raft_server:read(Server, {From, Command}),
continue;
Reason ->
?RAFT_COUNT(Table, {'acceptor.strong_read.error', Reason}),
?RAFT_LOG_WARNING(
"Acceptor[~0p] is rejecting read request from ~0p due to ~0p.",
[Name, From, Reason]
),
{error, Reason}
end
after
?RAFT_GATHER(Table, 'acceptor.strong_read.func', erlang:monotonic_time(microsecond) - StartTUsec)
end.
================================================
FILE: src/wa_raft_app.erl
================================================
%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%%
%%% This source code is licensed under the Apache 2.0 license found in
%%% the LICENSE file in the root directory of this source tree.
%%%
%%% Application implementation for wa_raft.
-module(wa_raft_app).
-compile(warn_missing_spec_all).
-behaviour(application).
%% Application callbacks
-export([
start/2,
stop/1
]).
-spec start(StartType :: application:start_type(), StartArgs :: term()) -> {ok, pid()}.
start(normal, _Args) ->
{ok, _Pid} = wa_raft_app_sup:start_link().
-spec stop(State :: term()) -> ok.
stop(_State) ->
ok.
================================================
FILE: src/wa_raft_app_sup.erl
================================================
%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%%
%%% This source code is licensed under the Apache 2.0 license found in
%%% the LICENSE file in the root directory of this source tree.
%%%
%%% Application supervisor to be started by the wa_raft application for
%%% supervising services and resources shared between application-started
%%% RAFT processes.
-module(wa_raft_app_sup).
-compile(warn_missing_spec_all).
-behaviour(supervisor).
%% API
-export([
start_link/0
]).
%% Supervisor callbacks
-export([
init/1
]).
-include_lib("wa_raft/include/wa_raft.hrl").
-spec start_link() -> supervisor:startlink_ret().
start_link() ->
supervisor:start_link({local, ?MODULE}, ?MODULE, []).
-spec init(Arg :: term()) -> {ok, {supervisor:sup_flags(), [supervisor:child_spec()]}}.
init(_) ->
% Cache certain commonly used configuration values.
case ?RAFT_METRICS_MODULE() of
{ok, Module} -> wa_raft_metrics:install(Module);
_Other -> ok
end,
% Setup tables used by shared services.
wa_raft_info:init_tables(),
wa_raft_transport:setup_tables(),
% Configure startup of shared services.
ChildSpecs = [
wa_raft_transport:child_spec(),
wa_raft_transport_sup:child_spec(),
wa_raft_dist_transport:child_spec(),
wa_raft_snapshot_catchup:child_spec()
],
{ok, {#{strategy => one_for_one, intensity => 5, period => 1}, lists:flatten(ChildSpecs)}}.
================================================
FILE: src/wa_raft_dist_transport.erl
================================================
%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%%
%%% This source code is licensed under the Apache 2.0 license found in
%%% the LICENSE file in the root directory of this source tree.
%%%
%%% This module implements transport interface by using erlang OTP dist.
-module(wa_raft_dist_transport).
-compile(warn_missing_spec_all).
-behaviour(gen_server).
-behaviour(wa_raft_transport).
-export([
child_spec/0,
start_link/0
]).
-export([
transport_init/1,
transport_send/3
]).
-export([
init/1,
handle_call/3,
handle_cast/2,
terminate/2
]).
-include_lib("wa_raft/include/wa_raft.hrl").
-include_lib("wa_raft/include/wa_raft_logger.hrl").
-record(sender_state, {
}).
-record(receiver_state, {
fds = #{} :: #{{ID :: wa_raft_transport:transport_id(), FileID :: wa_raft_transport:file_id()} => Fd :: file:fd()}
}).
-spec child_spec() -> supervisor:child_spec().
child_spec() ->
#{
id => ?MODULE,
start => {?MODULE, start_link, []},
restart => transient,
shutdown => 5000,
modules => [?MODULE]
}.
-spec start_link() -> gen_server:start_ret().
start_link() ->
gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
-spec transport_init(Node :: node()) -> {ok, State :: #sender_state{}}.
transport_init(_Node) ->
{ok, #sender_state{}}.
-spec transport_send(ID :: wa_raft_transport:transport_id(), FileID :: wa_raft_transport:file_id(), State :: #sender_state{}) ->
{ok, NewState :: #sender_state{}} |
{stop, Reason :: term(), NewState :: #sender_state{}}.
transport_send(ID, FileID, State) ->
?RAFT_LOG_DEBUG("wa_raft_dist_transport starting to send file ~p/~p", [ID, FileID]),
case wa_raft_transport:transport_info(ID) of
{ok, #{peer := Peer}} ->
case wa_raft_transport:file_info(ID, FileID) of
{ok, #{name := File, path := Path}} ->
case prim_file:open(Path, [binary, read]) of
{ok, Fd} ->
try
try prim_file:advise(Fd, 0, 0, sequential)
catch _:_ -> ok
end,
case transport_send_loop(ID, FileID, Fd, Peer, State) of
{ok, NewState} ->
{ok, NewState};
{error, Reason, NewState} ->
{stop, Reason, NewState}
end
after
prim_file:close(Fd)
end;
{error, Reason} ->
?RAFT_LOG_ERROR(
"wa_raft_dist_transport failed to open file ~p/~p (~s) due to ~p",
[ID, FileID, File, Reason]
),
{stop, {failed_to_open_file, ID, FileID, Reason}, State}
end;
_ ->
{stop, {invalid_file, ID, FileID}, State}
end;
_ ->
{stop, {invalid_transport, ID}, State}
end.
-spec transport_send_loop(
wa_raft_transport:transport_id(),
wa_raft_transport:file_id(),
file:fd(),
node(),
#sender_state{}
) -> {ok, #sender_state{}} | {error, term(), #sender_state{}}.
transport_send_loop(ID, FileID, Fd, Peer, State) ->
ChunkSize = ?RAFT_DIST_TRANSPORT_CHUNK_SIZE(),
MaxInflight = ?RAFT_DIST_TRANSPORT_MAX_INFLIGHT(),
transport_send_loop(ID, FileID, Fd, 0, Peer, [], ChunkSize, MaxInflight, State).
-spec transport_send_loop(
wa_raft_transport:transport_id(),
wa_raft_transport:file_id(),
file:fd(),
non_neg_integer() | eof,
node(),
[gen_server:request_id()],
pos_integer(),
pos_integer(),
#sender_state{}
) -> {ok, #sender_state{}} | {error, term(), #sender_state{}}.
transport_send_loop(ID, FileID, _Fd, eof, Peer, [], _ChunkSize, _MaxInflight, State) ->
gen_server:cast({?MODULE, Peer}, {complete, ID, FileID}),
{ok, State};
transport_send_loop(ID, FileID, Fd, Offset, Peer, [RequestId | Chunks], ChunkSize, MaxInflight, State)
when Offset =:= eof ; length(Chunks) >= MaxInflight ->
case gen_server:wait_response(RequestId, 5000) of
{reply, ok} ->
transport_send_loop(ID, FileID, Fd, Offset, Peer, Chunks, ChunkSize, MaxInflight, State);
{reply, {error, Reason}} ->
?RAFT_LOG_ERROR("wa_raft_dist_transport failed to send file ~p/~p due to receiver error ~p", [ID, FileID, Reason]),
{error, {receiver_error, ID, FileID, Reason}, State};
timeout ->
?RAFT_LOG_ERROR("wa_raft_dist_transport timed out while sending file ~p/~p", [ID, FileID]),
{error, {send_timed_out, ID, FileID}, State};
{error, {Reason, _}} ->
?RAFT_LOG_ERROR("wa_raft_dist_transport failed to send file ~p/~p due to ~p", [ID, FileID, Reason]),
{error, {send_failed, ID, FileID, Reason}, State}
end;
transport_send_loop(ID, FileID, Fd, Offset, Peer, Chunks, ChunkSize, MaxInflight, State) when is_integer(Offset) ->
case prim_file:read(Fd, ChunkSize) of
{ok, Data} ->
RequestId = gen_server:send_request({?MODULE, Peer}, {chunk, ID, FileID, Offset, Data}),
wa_raft_transport:update_file_info(ID, FileID,
fun (#{completed_bytes := Completed} = Info) ->
Info#{completed_bytes := Completed + byte_size(Data)}
end),
transport_send_loop(ID, FileID, Fd, Offset + byte_size(Data), Peer, Chunks ++ [RequestId], ChunkSize, MaxInflight, State);
eof ->
transport_send_loop(ID, FileID, Fd, eof, Peer, Chunks, ChunkSize, MaxInflight, State);
{error, Reason} ->
?RAFT_LOG_ERROR("wa_raft_dist_transport failed to read file ~p/~p due to ~p", [ID, FileID, Reason]),
{error, {read_failed, ID, FileID, Reason}, State}
end.
-spec init(Args :: []) -> {ok, State :: #receiver_state{}}.
init([]) ->
process_flag(trap_exit, true),
{ok, #receiver_state{}}.
-spec handle_call(Request, From :: term(), State :: #receiver_state{}) ->
{reply, Reply :: term(), NewState :: #receiver_state{}} | {noreply, NewState :: #receiver_state{}}
when Request :: {chunk, wa_raft_transport:transport_id(), wa_raft_transport:file_id(), integer(), binary()}.
handle_call({chunk, ID, FileID, Offset, Data}, _From, #receiver_state{} = State0) ->
{Reply, NewState} = case open_file(ID, FileID, State0) of
{ok, Fd, State1} ->
case prim_file:pwrite(Fd, Offset, Data) of
ok ->
wa_raft_transport:update_file_info(ID, FileID,
fun (#{completed_bytes := Completed} = Info) ->
Info#{completed_bytes := Completed + byte_size(Data)}
end),
{ok, State1};
{error, Reason} ->
?RAFT_LOG_WARNING(
"wa_raft_dist_transport receiver failed to write file chunk ~p/~p @ ~p due to ~p",
[ID, FileID, Offset, Reason]
),
{{write_failed, Reason}, State1}
end;
{error, Reason, State1} ->
?RAFT_LOG_WARNING(
"wa_raft_dist_transport receiver failed to handle file chunk ~p/~p @ ~p due to open failing due to ~p",
[ID, FileID, Offset, Reason]
),
{{open_failed, Reason}, State1}
end,
{reply, Reply, NewState};
handle_call(Request, From, #receiver_state{} = State) ->
?RAFT_LOG_NOTICE("wa_raft_dist_transport got unrecognized call ~p from ~p", [Request, From]),
{noreply, State}.
-spec handle_cast(Request, State :: #receiver_state{}) -> {noreply, NewState :: #receiver_state{}}
when Request :: {complete, wa_raft_transport:transport_id(), wa_raft_transport:file_id()}.
handle_cast({complete, ID, FileID}, #receiver_state{} = State0) ->
case open_file(ID, FileID, State0) of
{ok, _Fd, State1} ->
{ok, State2} = close_file(ID, FileID, State1),
wa_raft_transport:complete(ID, FileID, ok),
{noreply, State2};
{error, Reason, State1} ->
?RAFT_LOG_WARNING(
"wa_raft_dist_transport receiver failed to handle file complete ~p/~p due to open failing due to ~p",
[ID, FileID, Reason]
),
{noreply, State1}
end;
handle_cast(Request, #receiver_state{} = State) ->
?RAFT_LOG_NOTICE("wa_raft_dist_transport got unrecognized cast ~p", [Request]),
{noreply, State}.
-spec terminate(Reason :: dynamic(), State :: #receiver_state{}) -> ok.
terminate(Reason, #receiver_state{}) ->
?RAFT_LOG_NOTICE("wa_raft_dist_transport terminating due to ~p", [Reason]),
ok.
-spec open_file(ID :: wa_raft_transport:transport_id(), FileID :: wa_raft_transport:file_id(), State :: #receiver_state{}) ->
{ok, Fd :: file:fd(), NewState :: #receiver_state{}} | {error, Reason :: term(), NewState :: #receiver_state{}}.
open_file(ID, FileID, #receiver_state{fds = Fds} = State0) ->
case Fds of
#{{ID, FileID} := Fd} ->
{ok, Fd, State0};
#{} ->
case wa_raft_transport:file_info(ID, FileID) of
{ok, #{name := File, path := Path}} ->
try filelib:ensure_dir(Path)
catch _:_ -> ok
end,
case prim_file:open(Path, [binary, write]) of
{ok, Fd} ->
State1 = State0#receiver_state{fds = Fds#{{ID, FileID} => Fd}},
{ok, Fd, State1};
{error, Reason} ->
?RAFT_LOG_WARNING(
"wa_raft_dist_transport receiver failed to open file ~p/~p (~p) due to ~p",
[ID, FileID, File, Reason]
),
{error, {open_failed, Reason}, State0}
end;
_ ->
{error, invalid_file, State0}
end
end.
-spec close_file(ID :: wa_raft_transport:transport_id(), FileID :: wa_raft_transport:file_id(), State :: #receiver_state{}) ->
{ok, NewState :: #receiver_state{}}.
close_file(ID, FileID, #receiver_state{fds = Fds} = State0) ->
case Fds of
#{{ID, FileID} := Fd} ->
try prim_file:close(Fd)
catch _:_ -> ok
end,
State1 = State0#receiver_state{fds = maps:remove({ID, FileID}, Fds)},
{ok, State1};
_ ->
{ok, State0}
end.
================================================
FILE: src/wa_raft_distribution.erl
================================================
%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%%
%%% This source code is licensed under the Apache 2.0 license found in
%%% the LICENSE file in the root directory of this source tree.
%%%
%%% Pluggable distribution interface. The default implementation uses Erlang
%%% distribution.
-module(wa_raft_distribution).
-compile(warn_missing_spec_all).
-export([
cast/3
]).
-include_lib("wa_raft/include/wa_raft.hrl").
-type dest_addr() :: {Name :: atom(), Node :: node()}.
-export_type([
dest_addr/0
]).
%%% ------------------------------------------------------------------------
%%% Behaviour callbacks
%%%
-callback cast(dest_addr(), #raft_identifier{}, term()) -> ok | term().
%%% ------------------------------------------------------------------------
%%% Erlang distribution default implementation
%%%
-spec cast(DestAddr :: dest_addr(), Identifier :: #raft_identifier{}, Message :: term()) -> ok | {error, Reason} when
Reason :: noconnect | nosuspend.
cast(DestAddr, _Identifier, Message) ->
case erlang:send(DestAddr, {'$gen_cast', Message}, [noconnect, nosuspend]) of
ok -> ok;
Reason -> {error, Reason}
end.
================================================
FILE: src/wa_raft_durable_state.erl
================================================
%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%%
%%% This source code is licensed under the Apache 2.0 license found in
%%% the LICENSE file in the root directory of this source tree.
%%%
%%% This module implements functions for storing / loading persistent state.
-module(wa_raft_durable_state).
-compile(warn_missing_spec_all).
-include_lib("wa_raft/include/wa_raft.hrl").
-include_lib("wa_raft/include/wa_raft_logger.hrl").
-export([
load/1,
store/1,
sync/1
]).
-spec load(StateIn :: #raft_state{}) -> {ok, StateOut :: #raft_state{}} | no_state | {error, Reason :: term()}.
load(#raft_state{name = Name, partition_path = PartitionPath} = State) ->
StateItems = [
{current_term, fun is_integer/1, fun (V, S) -> S#raft_state{current_term = V} end, required},
{voted_for, fun is_atom/1, fun (V, S) -> S#raft_state{voted_for = V} end, required},
{disable_reason, undefined, fun (V, S) -> S#raft_state{disable_reason = V} end, undefined}
],
StateFile = filename:join(PartitionPath, ?STATE_FILE_NAME),
case file:consult(StateFile) of
{ok, [{crc, SavedCRC} | StateTerms]} ->
case erlang:crc32(term_to_binary(StateTerms, [{minor_version, 1}, deterministic])) of
SavedCRC ->
try
{ok, lists:foldl(
fun ({Item, Validator, Updater, Default}, StateN) ->
case proplists:lookup(Item, StateTerms) of
none when Default =:= required ->
?RAFT_LOG_ERROR("~p read state file but cannot find ~p.", [Name, Item]),
throw({error, {missing, Item}});
none ->
Updater(Default, StateN);
{Item, Value} ->
case Validator =:= undefined orelse Validator(Value) of
true ->
Updater(Value, StateN);
false ->
?RAFT_LOG_ERROR("~p read state file but ~p has an invalid value `~p`.", [Name, Item, Value]),
throw({error, {invalid, Item}})
end
end
end, State, StateItems)}
catch
throw:{error, Reason} -> {error, Reason}
end;
InvalidCRC ->
?RAFT_LOG_ERROR("~p read state file but CRCs did not match. (saved crc: ~p, computed crc: ~p)", [Name, SavedCRC, InvalidCRC]),
{error, invalid_crc}
end;
{ok, _} ->
?RAFT_LOG_ERROR("~p read state file but no CRC was found", [Name]),
{error, no_crc};
{error, enoent} ->
?RAFT_LOG_NOTICE("~p is not loading non-existant state file.", [Name]),
no_state;
{error, Reason} ->
?RAFT_LOG_ERROR("~p could not read state file due to ~p.", [Name, Reason]),
{error, Reason}
end.
-spec store(#raft_state{}) -> ok | {error, Reason :: term()}.
store(#raft_state{name = Name, table = Table, partition_path = PartitionPath, current_term = CurrentTerm, voted_for = VotedFor, disable_reason = DisableReason}) ->
StateList = [
{current_term, CurrentTerm},
{voted_for, VotedFor},
{disable_reason, DisableReason}
],
StateListWithCRC = [{crc, erlang:crc32(term_to_binary(StateList, [{minor_version, 1}, deterministic]))} | StateList],
StateIO = [io_lib:format("~p.~n", [Term]) || Term <- StateListWithCRC],
StateFile = filename:join(PartitionPath, ?STATE_FILE_NAME),
StateFileTemp = [StateFile, ".temp"],
case filelib:ensure_dir(StateFile) of
ok ->
case prim_file:write_file(StateFileTemp, StateIO) of
ok ->
case file:rename(StateFileTemp, StateFile) of
ok ->
ok;
{error, Reason} ->
?RAFT_COUNT(Table, {'server.persist_state.error.rename', Reason}),
?RAFT_LOG_ERROR("~p failed to rename temporary state file due to ~p.", [Name, Reason]),
{error, {rename, Reason}}
end;
{error, Reason} ->
?RAFT_COUNT(Table, {'server.persist_state.error.write', Reason}),
?RAFT_LOG_ERROR("~p failed to write current state to temporary file due to ~p.", [Name, Reason]),
{error, {write, Reason}}
end;
{error, Reason} ->
?RAFT_COUNT(Table, {'server.persist_state.error.ensure_dir', Reason}),
?RAFT_LOG_ERROR("~p failed to ensure directory exists due to ~p.", [Name, Reason]),
{error, {ensure_dir, Reason}}
end.
-spec sync(StateIn :: #raft_state{}) -> ok.
sync(#raft_state{partition_path = PartitionPath}) ->
StateFile = filename:join(PartitionPath, ?STATE_FILE_NAME),
case prim_file:open(StateFile, [read, binary]) of
{ok, Fd} ->
prim_file:sync(Fd),
prim_file:close(Fd),
ok;
_ ->
ok
end.
================================================
FILE: src/wa_raft_env.erl
================================================
%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%%
%%% This source code is licensed under the Apache 2.0 license found in
%%% the LICENSE file in the root directory of this source tree.
%%%
%%% This module contains utility functions for consulting application
%%% configuration in the OTP application environment according to the search
%%% order configured for each RAFT partition.
-module(wa_raft_env).
-compile(warn_missing_spec_all).
%% Config API
-export([
database_path/1
]).
%% Internal API
-export([
get_env/2,
get_env/3,
get_table_env/3,
get_table_env/4
]).
-type scope() :: Application :: atom() | {Table :: wa_raft:table(), Partition :: wa_raft:partition()} | SearchApps :: [atom()].
-type key() :: Key :: atom() | {Primary :: atom(), Fallback :: atom()}.
-include_lib("wa_raft/include/wa_raft.hrl").
%%-------------------------------------------------------------------
%% Config API
%%-------------------------------------------------------------------
-spec database_path(Scope :: scope()) -> Root :: file:filename().
database_path(Scope) ->
case get_env(Scope, ?RAFT_DATABASE) of
{ok, Root} -> Root;
undefined -> error({no_configured_database_path, Scope})
end.
%%-------------------------------------------------------------------
%% Internal API
%%-------------------------------------------------------------------
-spec get_env(Scope :: scope(), Key :: key()) -> {ok, Value :: dynamic()} | undefined.
get_env(Scope, Key) ->
get_env_impl(search_apps(Scope), key(Key), fallback(Key)).
-spec get_env(Scope :: scope(), Key :: key(), Default :: Value) -> Value.
get_env(Scope, Key, Default) ->
case get_env(Scope, Key) of
{ok, Value} -> Value;
undefined -> Default
end.
-spec get_env_impl(SearchApps :: [atom()], Key :: atom(), FallbackKey :: atom()) -> {ok, Value :: dynamic()} | undefined.
get_env_impl([], _Key, FallbackKey) ->
?RAFT_CONFIG(FallbackKey);
get_env_impl([App | SearchApps], Key, FallbackKey) ->
case application:get_env(App, Key) of
{ok, Value} -> {ok, Value};
undefined -> get_env_impl(SearchApps, Key, FallbackKey)
end.
-doc """
Get a configuration value with table-level overrides.
Table-level configuration is supported by storing the value as a tagged tuple:
`{table_overrides, #{Table => Value}, AppDefault}`
When this format is found, the table is looked up in the map first.
If the table is not in the map, AppDefault is used. If the value is a plain
(untagged) term, it is used directly as the app-level value for all tables.
""".
-spec get_table_env(Scope :: scope(), Table :: wa_raft:table(), Key :: key()) -> {ok, Value :: dynamic()} | undefined.
get_table_env(Scope, Table, Key) ->
get_table_env_impl(search_apps(Scope), Table, key(Key), fallback(Key)).
-spec get_table_env(Scope :: scope(), Table :: wa_raft:table(), Key :: key(), Default :: Value) -> Value.
get_table_env(Scope, Table, Key, Default) ->
case get_table_env(Scope, Table, Key) of
{ok, Value} -> Value;
undefined -> Default
end.
-spec get_table_env_impl(SearchApps :: [atom()], Table :: wa_raft:table(), Key :: atom(), FallbackKey :: atom()) ->
{ok, Value :: dynamic()} | undefined.
get_table_env_impl([], _Table, _Key, FallbackKey) ->
?RAFT_CONFIG(FallbackKey);
get_table_env_impl([App | SearchApps], Table, Key, FallbackKey) ->
case application:get_env(App, Key) of
{ok, {table_overrides, TableOverrides, AppDefault}} ->
{ok, maps:get(Table, TableOverrides, AppDefault)};
{ok, Value} ->
{ok, Value};
undefined ->
get_table_env_impl(SearchApps, Table, Key, FallbackKey)
end.
-spec search_apps(Scope :: scope()) -> SearchApps :: [atom()].
search_apps(Application) when is_atom(Application) ->
case wa_raft_sup:options(Application) of
undefined -> [];
RaftApplication -> RaftApplication#raft_application.config_search_apps
end;
search_apps({Table, Partition}) ->
case wa_raft_part_sup:options(Table, Partition) of
undefined -> [];
Options -> search_apps(Options#raft_options.application)
end;
search_apps(SearchApps) ->
SearchApps.
-spec key(key()) -> atom().
key({Key, _}) -> Key;
key(Key) -> Key.
-spec fallback(key()) -> atom().
fallback({_, Fallback}) -> Fallback;
fallback(Key) -> Key.
================================================
FILE: src/wa_raft_info.erl
================================================
%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%%
%%% This source code is licensed under the Apache 2.0 license found in
%%% the LICENSE file in the root directory of this source tree.
%%%
%%% API for accessing certain useful information about the state of local
%%% RAFT servers without requiring a status request against the RAFT server
%%% itself.
-module(wa_raft_info).
-compile(warn_missing_spec_all).
%% Public API
-export([
get_current_term/2,
get_leader/2,
get_current_term_and_leader/2,
get_membership/2,
get_live/2,
get_stale/2,
get_state/2,
get_message_queue_length/1
]).
%% Internal API
-export([
init_tables/0,
delete_state/2,
set_current_term_and_leader/4,
set_membership/3,
set_live/3,
set_stale/3,
set_state/3,
set_message_queue_length/1,
set_message_queue_length/2
]).
%% Local RAFT server's current FSM state
-define(RAFT_SERVER_STATE_KEY(Table, Partition), {state, Table, Partition}).
%% Local RAFT server's most recently known term and leader
-define(RAFT_CURRENT_TERM_AND_LEADER_KEY(Table, Partition), {term, Table, Partition}).
%% Local RAFT server's current live flag - indicates if the server thinks it is part of a live cluster
-define(RAFT_LIVE_KEY(Table, Partition), {live, Table, Partition}).
%% Local RAFT server's current stale flag - indicates if the server thinks its data is stale
-define(RAFT_STALE_KEY(Table, Partition), {stale, Table, Partition}).
%% Local RAFT server's message queue length
-define(RAFT_MSG_QUEUE_LENGTH_KEY(Name), {msg_queue_length, Name}).
%% Local RAFT server's most recently known membership
-define(RAFT_MEMBERSHIP_KEY(Table, Partition), {membership, Table, Partition}).
%%-------------------------------------------------------------------
%% RAFT Info - Public API
%%-------------------------------------------------------------------
-spec get(term(), Default) -> Default.
get(Key, Default) ->
try
ets:lookup_element(?MODULE, Key, 2, Default)
catch
error:badarg ->
Default
end.
-spec get_leader(wa_raft:table(), wa_raft:partition()) -> node() | undefined.
get_leader(Table, Partition) ->
{_, Leader} = get(?RAFT_CURRENT_TERM_AND_LEADER_KEY(Table, Partition), {undefined, undefined}),
Leader.
-spec get_current_term(wa_raft:table(), wa_raft:partition()) -> wa_raft_log:log_term() | undefined.
get_current_term(Table, Partition) ->
{Term, _} = get(?RAFT_CURRENT_TERM_AND_LEADER_KEY(Table, Partition), {undefined, undefined}),
Term.
%% The RAFT server always sets both the known term and leader together, so that
%% the atomic read performed by this method will not return a known leader for
%% a different term.
-spec get_current_term_and_leader(wa_raft:table(), wa_raft:partition()) ->
{wa_raft_log:log_term() | undefined, node() | undefined}.
get_current_term_and_leader(Table, Partition) ->
get(?RAFT_CURRENT_TERM_AND_LEADER_KEY(Table, Partition), {undefined, undefined}).
-spec get_state(wa_raft:table(), wa_raft:partition()) -> wa_raft_server:state() | undefined.
get_state(Table, Partition) ->
get(?RAFT_SERVER_STATE_KEY(Table, Partition), undefined).
-spec get_live(wa_raft:table(), wa_raft:partition()) -> boolean().
get_live(Table, Partition) ->
get(?RAFT_LIVE_KEY(Table, Partition), false).
-spec get_stale(wa_raft:table(), wa_raft:partition()) -> boolean().
get_stale(Table, Partition) ->
get(?RAFT_STALE_KEY(Table, Partition), true).
-spec get_message_queue_length(atom()) -> undefined | non_neg_integer().
get_message_queue_length(Name) ->
get(?RAFT_MSG_QUEUE_LENGTH_KEY(Name), undefined).
-spec get_membership(wa_raft:table(), wa_raft:partition()) -> wa_raft_server:membership() | undefined.
get_membership(Table, Partition) ->
get(?RAFT_MEMBERSHIP_KEY(Table, Partition), undefined).
%%-------------------------------------------------------------------
%% RAFT Info - Internal API
%%-------------------------------------------------------------------
-spec init_tables() -> ok.
init_tables() ->
ets:new(?MODULE, [set, public, named_table, {write_concurrency, true}, {read_concurrency, true}]),
ok.
-spec set(term(), term()) -> true.
set(Key, Value) ->
ets:update_element(?MODULE, Key, {2, Value}) orelse ets:insert(?MODULE, {Key, Value}).
-spec delete(term()) -> true.
delete(Key) ->
ets:delete(?MODULE, Key).
-spec set_current_term_and_leader(wa_raft:table(), wa_raft:partition(), wa_raft_log:log_term(), node()) -> true.
set_current_term_and_leader(Table, Partition, Term, Leader) ->
set(?RAFT_CURRENT_TERM_AND_LEADER_KEY(Table, Partition), {Term, Leader}).
-spec set_state(wa_raft:table(), wa_raft:partition(), wa_raft_server:state()) -> true.
set_state(Table, Partition, State) ->
set(?RAFT_SERVER_STATE_KEY(Table, Partition), State).
-spec delete_state(wa_raft:table(), wa_raft:partition()) -> true.
delete_state(Table, Partition) ->
delete(?RAFT_SERVER_STATE_KEY(Table, Partition)).
-spec set_live(wa_raft:table(), wa_raft:partition(), boolean()) -> true.
set_live(Table, Partition, Live) ->
set(?RAFT_LIVE_KEY(Table, Partition), Live).
-spec set_stale(wa_raft:table(), wa_raft:partition(), boolean()) -> true.
set_stale(Table, Partition, Stale) ->
set(?RAFT_STALE_KEY(Table, Partition), Stale).
-spec set_membership(wa_raft:table(), wa_raft:partition(), wa_raft_server:membership()) -> true.
set_membership(Table, Partition, Membership) ->
set(?RAFT_MEMBERSHIP_KEY(Table, Partition), Membership).
-spec set_message_queue_length(Name :: atom()) -> true.
set_message_queue_length(Name) ->
{message_queue_len, Length} = process_info(self(), message_queue_len),
set_message_queue_length(Name, Length).
-spec set_message_queue_length(Name :: atom(), Length :: non_neg_integer()) -> true.
set_message_queue_length(Name, Length) ->
set(?RAFT_MSG_QUEUE_LENGTH_KEY(Name), Length).
================================================
FILE: src/wa_raft_label.erl
================================================
%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%%
%%% This source code is licensed under the Apache 2.0 license found in
%%% the LICENSE file in the root directory of this source tree.
%%%
%%% Pluggable module for labeling log entries before adding them to the RAFT log.
-module(wa_raft_label).
-compile(warn_missing_spec_all).
-type label() :: dynamic().
-export_type([label/0]).
%%% ------------------------------------------------------------------------
%%% Behaviour callbacks
%%%
% Produce a label for a new log record based on the log payload and the label of the preceding log entry.
-callback new_label(LastLabel :: label(), Command :: wa_raft_acceptor:command()) -> NewLabel :: label().
================================================
FILE: src/wa_raft_log.erl
================================================
%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%%
%%% This source code is licensed under the Apache 2.0 license found in
%%% the LICENSE file in the root directory of this source tree.
%%%
%%% This module is the interface for raft log. It defines the callbacks
%%% required by the specific log implementations.
-module(wa_raft_log).
-compile(warn_missing_spec_all).
-behaviour(gen_server).
%% OTP supervision
-export([
child_spec/1,
start_link/1
]).
%% APIs for writing new log data
-export([
append/2,
try_append/2,
try_append/3,
check_heartbeat/3
]).
%% APIs for accessing log data
-export([
first_index/1,
last_index/1,
fold/5,
fold/6,
fold_binary/5,
fold_binary/6,
fold_terms/5,
term/2,
get/2,
get/3,
get/4,
get_terms/3,
entries/3,
entries/4,
config/1
]).
%% APIs for managing logs and log data
-export([
open/2,
reset/2,
truncate/2,
trim/2,
rotate/2, rotate/4,
flush/1
]).
%% Internal API
-export([
default_name/2,
registered_name/2
]).
%% Internal API
-export([
log/1,
log_name/1,
provider/1
]).
%% gen_server callbacks
-export([
init/1,
handle_call/3,
handle_cast/2,
terminate/2
]).
-export_type([
log/0,
log_name/0,
log_pos/0,
log_op/0,
log_index/0,
log_term/0,
log_entry/0,
log_record/0,
view/0
]).
-include_lib("wa_raft/include/wa_raft.hrl").
-include_lib("wa_raft/include/wa_raft_logger.hrl").
%% Atom indicating that the provider has not been opened yet.
-define(PROVIDER_NOT_OPENED, '$not_opened').
%% A view of a RAFT log that is backed by a particular
%% log provider. This view keeps track of its own logical
%% start and end indices as well as a batch of pending
%% log entries so that the RAFT server is always able to
%% access a consistent view of the RAFT log given simple
%% RAFT log provider implementations.
-record(log_view, {
log :: log(),
first = 0 :: log_index(),
last = 0 :: log_index(),
config :: undefined | {log_index(), wa_raft_server:config()}
}).
%% The state stored by the RAFT log server which is
%% responsible for synchronizing destructive operations
%% on the RAFT log with operations that are performed
%% asynchronously to the RAFT server.
-record(log_state, {
log :: log(),
state = ?PROVIDER_NOT_OPENED :: term()
}).
%% Name of a raft log.
-type log() :: #raft_log{}.
-type log_name() :: atom().
-type log_index() :: non_neg_integer().
-type log_term() :: non_neg_integer().
-type log_pos() :: #raft_log_pos{}.
-type log_op() ::
undefined
| {wa_raft_acceptor:key(), wa_raft_acceptor:command()}
| {wa_raft_acceptor:key(), wa_raft_label:label(), wa_raft_acceptor:command()}.
-type log_entry() :: {log_term(), log_op()}.
-type log_record() :: {log_index(), log_entry()}.
%% A view of a RAFT log.
-opaque view() :: #log_view{}.
%%-------------------------------------------------------------------
%% RAFT log provider interface for accessing log data
%%-------------------------------------------------------------------
%% Gets the first index of the RAFT log. If there are no log entries,
%% then return 'undefined'.
-callback first_index(Log :: log()) -> undefined | FirstIndex :: log_index() | {error, Reason :: term()}.
%% Gets the last index of the RAFT log. If there are no log entries,
%% then return 'undefined'.
-callback last_index(Log :: log()) -> undefined | LastIndex :: log_index() | {error, Reason :: term()}.
%% Call the provided combining function on successive log entries in the
%% specified log starting from the specified start index (inclusive) and ending
%% at the specified end index (also inclusive) or until the total cumulative
%% size of log entries for which the combining function has been called upon
%% is at least the specified size limit. The combining function must always be
%% called with log entries in log order starting with the first log entry that
%% exists within the provided range. The combining function should not be
%% called for those log indices within the provided range that do not have a
%% stored log entry. It is suggested that the external term size of the entire
%% log entry be used as the size provided to the combining function and for
%% tracking the size limit; however, implementations are free to use any value.
%%
%% The combining function may raise an error. Implementations should take care
%% to release any resources held in the case that the fold needs to terminate
%% early.
-callback fold(Log :: log(),
Start :: log_index(),
End :: log_index(),
SizeLimit :: non_neg_integer() | infinity,
Func :: fun((Index :: log_index(), Size :: non_neg_integer(), Entry :: log_entry(), Acc) -> Acc),
Acc) -> {ok, Acc} | {error, Reason :: term()}.
%% Call the provided combining function on the external term format of
%% successive log entries in the specified log starting from the specified
%% start index (inclusive) and ending at the specified end index (also
%% inclusive) or until the total cumulative size of log entries for which the
%% combining function has been called upon is at least the specified size
%% limit. The combining function must always be called with log entries in log
%% order starting with the first log entry that exists within the provided
%% range. The combining function should not be called for those log indices
%% within the provided range that do not have a stored log entry. The byte
%% size of the binary provided to the combining function must be used as the
%% size of the entry for tracking of the size limit.
%%
%% The combining function may raise an error. Implementations should take care
%% to release any resources held in the case that the fold needs to terminate
%% early.
-callback fold_binary(
Log :: log(),
Start :: log_index(),
End :: log_index(),
SizeLimit :: non_neg_integer() | infinity,
Func :: fun((Index :: log_index(), Entry :: binary(), Acc) -> Acc),
Acc
) -> {ok, Acc} | {error, Reason :: term()}.
-optional_callbacks([fold_binary/6]).
%% Call the provided combining function on the log term of successive log
%% entries in the specified log starting from the specified start index
%% (inclusive) and ending at the specified end index (also inclusive). The
%% combining function must always be called with log entries in log order
%% starting with the first log entry that exists within the provided range. The
%% combining function should not be called for those log indices within the
%% provided range that do not have a stored log entry.
%%
%% The combining function may raise an error. Implementations should take care
%% to release any resources held in the case that the fold needs to terminate
%% early.
-callback fold_terms(Log :: log(),
Start :: log_index(),
End :: log_index(),
Func :: fun((Index :: log_index(), Term :: log_term(), Acc) -> Acc),
Acc) -> {ok, Acc} | {error, Reason :: term()}.
%% Get a single log entry at the specified index. This API is specified
%% separately because some implementations may have more efficient ways to
%% get log entries when only one log entry is required. If the log entry
%% does not exist, then return 'not_found'.
-callback get(Log :: log(), Index :: log_index()) -> {ok, Entry :: log_entry()} | not_found | {error, Reason :: term()}.
%% Get only the term of a specific log entry. This API is specified
%% separately because some implementations may have more efficient ways to
%% get just the term of a particular log entry. If the log entry does not
%% exist, then return 'not_found'.
-callback term(Log :: log(), Index :: log_index()) -> {ok, Term :: log_term()} | not_found | {error, Reason :: term()}.
%% Get the most recent configuration stored in the log. Log providers
%% should ensure that configuration operations are indexed so that this
%% call does not require a scan of the log.
-callback config(Log :: log()) -> {ok, Index :: log_index(), Config :: wa_raft_server:config()} | not_found | {error, Reason :: term()}.
%%-------------------------------------------------------------------
%% RAFT log provider interface for writing new log data
%%-------------------------------------------------------------------
%% Append new log entries to the end of the RAFT log.
%% - This function should never overwrite existing log entries.
%% - If the new log entries were written successfully, return 'ok'.
%% - Log entries may be provided either as terms directly or as a
%% binary encoding a log entry in external term format.
%% - In 'strict' mode, the append should always succeed or return an
%% error on failure.
%% - In 'relaxed' mode, if there are transient conditions that would
%% make it difficult to append to the log without blocking, then
%% the append should be skipped and 'skipped' returned. Otherwise,
%% the same conditions as the 'strict' mode apply.
-callback append(View :: view(), Entries :: [log_entry() | binary()], Mode :: strict | relaxed, Priority :: wa_raft_acceptor:priority()) ->
ok | skipped | {error, Reason :: term()}.
%%-------------------------------------------------------------------
%% RAFT log provider interface for managing underlying RAFT log
%%-------------------------------------------------------------------
%% Perform any first time setup operations before opening the RAFT log.
%% This function is called from the RAFT log server and is only called
%% once per incarnation of a RAFT partition.
%% If this setup fails such that the log is not usable, implementations
%% should raise an error or exit to interrupt the startup process.
-callback init(Log :: wa_raft_log:log()) -> ok.
%% Open the RAFT log and return a state term that will be provided to
%% subsequent calls made from the RAFT log server. During the opening
%% process, the log will be inspected to see if it contains a record
%% corresponding to the last applied index of the storage backing this
%% RAFT partition and whether or not the term of this entry matches
%% that reported by the storage. If so, then opening proceeeds normally.
%% If there is a mismatch, then the log will be reinitialized using
%% `reset/3`.
%% If this setup fails such that the log is not usable, implementations
%% should raise an error or exit to interrupt the opening process.
-callback open(Log :: wa_raft_log:log()) -> {ok, State :: term()} | {error, Reason :: term()}.
%% Close the RAFT log and release any resources used by it. This
%% is called when the RAFT log server is terminating.
-callback close(Log :: log(), State :: term()) -> term().
%% Completely clear the RAFT log and setup a new log with an initial entry
%% at the provided index with the provided term and an undefined op.
-callback reset(Log :: log(), Position :: log_pos(), State :: term()) -> {ok, NewState :: term()} | {error, Reason :: term()}.
%% Truncate the RAFT log to the given position so that all log entries
%% including and after the provided index are completely deleted from
%% the RAFT log. If the truncation failed but the log state was not
%% changed, then an error can be returned. Otherwise, a error should
%% be raised.
-callback truncate(Log :: log(), Index :: log_index(), State :: term()) -> {ok, NewState :: term()} | {error, Reason :: term()}.
%% Optionally, trim the RAFT log up to the given index.
%% - This means that all log entries before the given index can be
%% deleted (both term and op information can be removed) and the
%% log entry at the given index can have its op removed (keeping
%% only the term information).
%% - Implementations are not required to always trim the log to exactly
%% the provided index but must not trim past the provided index and
%% must always ensure that if the log were to be reloaded from disk
%% at any time that the log always remains contiguous, meaning that
%% only the first entry in the log can be missing op information and
%% that the indices of all log entries in the log are contiguous.
-callback trim(Log :: log(), Index :: log_index(), State :: term()) -> {ok, NewState :: term()} | {error, Reason :: term()}.
%% Flush log to disk on a best-effort basis. The return value is
%% ignored.
-callback flush(Log :: log()) -> term().
%%-------------------------------------------------------------------
%% RAFT log provider interface for writing new log data
%%-------------------------------------------------------------------
-spec child_spec(Options :: #raft_options{}) -> supervisor:child_spec().
child_spec(Options) ->
#{
id => ?MODULE,
start => {?MODULE, start_link, [Options]},
restart => permanent,
shutdown => 30000,
modules => [?MODULE]
}.
-spec start_link(Options :: #raft_options{}) -> {ok, Pid :: pid()} | ignore | {error, Reason :: term()}.
start_link(#raft_options{log_name = Name} = Options) ->
gen_server:start_link({local, Name}, ?MODULE, Options, []).
%%-------------------------------------------------------------------
%% APIs for writing new log data
%%-------------------------------------------------------------------
%% Append new log entries to the end of the log.
-spec append(View :: view(), Entries :: [log_entry() | binary()]) ->
{ok, NewView :: view()} | {error, Reason :: term()}.
append(View, Entries) ->
% eqwalizer:ignore - strict append cannot return skipped
append(View, Entries, strict, high).
%% Attempt to append new log entries to the end of the log if an append can be
%% serviced immediately.
-spec try_append(View :: view(), Entries :: [log_entry() | binary()]) ->
{ok, NewView :: view()} | skipped | {error, Reason :: term()}.
try_append(View, Entries) ->
try_append(View, Entries, high).
-spec try_append(View :: view(), Entries :: [log_entry() | binary()], Priority :: wa_raft_acceptor:priority()) ->
{ok, NewView :: view()} | skipped | {error, Reason :: term()}.
try_append(View, Entries, Priority) ->
append(View, Entries, relaxed, Priority).
%% Append new log entries to the end of the log.
-spec append(View :: view(), Entries :: [log_entry() | binary()], Mode :: strict | relaxed, Priority :: wa_raft_acceptor:priority()) ->
{ok, NewView :: view()} | skipped | {error, Reason :: term()}.
append(#log_view{log = #raft_log{table = Table}, last = Last} = View, Entries, Mode, Priority) ->
?RAFT_COUNT(Table, 'log.append'),
Provider = provider(View),
case Provider:append(View, Entries, Mode, Priority) of
ok ->
?RAFT_COUNT(Table, 'log.append.ok'),
{ok, refresh_config(View#log_view{last = Last + length(Entries)})};
skipped when Mode =:= relaxed ->
?RAFT_COUNT(Table, 'log.append.skipped'),
skipped;
{error, Reason} ->
?RAFT_COUNT(Table, 'log.append.error'),
{error, Reason}
end.
%% Compare the provided heartbeat log entries to the local log at the provided
%% starting position in preparation for an append operation:
%% * If the provided starting position is before the start of the log or past
%% the end of the log, the comparison will fail with an `out_of_range`
%% error.
%% * If there is a conflict between the provided heartbeat log entries and any
%% local log entries due to a term mismatch, then the comparison will fail
%% with a `conflict` tuple that contains the log index of the first log
%% entry with a conflicting term and the list containing the corresponding
%% heartbeat log entry and all subsequent heartbeat log entries.
%% * Otherwise, the comparison will succeed. Any new log entries not already
%% in the local log will be returned.
-spec check_heartbeat(View :: view(), Start :: log_index(), Entries :: [log_entry() | binary()]) ->
{ok, NewEntries :: [log_entry() | binary()]} |
{conflict, ConflictIndex :: log_index(), NewEntries :: [log_entry() | binary()]} |
{invalid, out_of_range} |
{error, term()}.
check_heartbeat(#log_view{first = First, last = Last}, Start, _Entries) when Start =< 0; Start < First; Start > Last ->
{invalid, out_of_range};
check_heartbeat(#log_view{log = #raft_log{table = Table} = Log, last = Last}, Start, Entries) ->
Provider = provider(Log),
End = Start + length(Entries) - 1,
try Provider:fold_terms(Log, Start, End, fun check_heartbeat_terms/3, {Start, Entries}) of
% The fold should not terminate early if the provider is well-behaved.
{ok, {Next, []}} when Next =:= End + 1 ->
{ok, []};
{ok, {Next, NewEntries}} when Next =:= Last + 1 ->
{ok, NewEntries};
{error, Reason} ->
?RAFT_COUNT(Table, 'log.heartbeat.error'),
{error, Reason}
catch
throw:{conflict, ConflictIndex, ConflictEntries} ->
{conflict, ConflictIndex, ConflictEntries};
throw:{missing, Index} ->
?RAFT_COUNT(Table, 'log.heartbeat.corruption'),
{error, {missing, Index}}
end.
-spec check_heartbeat_terms(Index :: wa_raft_log:log_index(), Term :: wa_raft_log:log_term(), Acc) -> Acc
when Acc :: {Next :: wa_raft_log:log_index(), Entries :: [log_entry() | binary()]}.
check_heartbeat_terms(Index, Term, {Next, [Binary | Entries]}) when is_binary(Binary) ->
check_heartbeat_terms(Index, Term, {Next, [binary_to_term(Binary) | Entries]});
check_heartbeat_terms(Index, Term, {Index, [{Term, _} | Entries]}) ->
{Index + 1, Entries};
check_heartbeat_terms(Index, _, {Index, [_ | _] = Entries}) ->
throw({conflict, Index, Entries});
check_heartbeat_terms(_, _, {Index, [_ | _]}) ->
throw({missing, Index}).
%%-------------------------------------------------------------------
%% APIs for accessing log data
%%-------------------------------------------------------------------
%% Gets the first index of the log view or as reported by the log provider.
-spec first_index(LogOrView :: log() | view()) -> FirstIndex :: log_index().
first_index(#log_view{first = First}) ->
First;
first_index(Log) ->
Provider = provider(Log),
Provider:first_index(Log).
%% Gets the last index of the log view or as reported by the log provider.
-spec last_index(LogOrView :: log() | view()) -> LastIndex :: log_index().
last_index(#log_view{last = Last}) ->
Last;
last_index(Log) ->
Provider = provider(Log),
Provider:last_index(Log).
-spec fold(LogOrView :: log() | view(),
First :: log_index(),
Last :: log_index() | infinity,
Func :: fun((Index :: log_index(), Entry :: log_entry(), Acc) -> Acc),
Acc) -> {ok, Acc} | {error, term()}.
fold(LogOrView, First, Last, Func, Acc) ->
fold(LogOrView, First, Last, infinity, Func, Acc).
%% Call the provided combining function on successive log entries in the
%% specified log starting from the specified start index (inclusive) and ending
%% at the specified end index (also inclusive). The combining function will
%% always be called with log entries in log order starting with the first log
%% entry that exists within the provided range. The combining function will
%% not be called for those log indices within the provided range that do not
%% have a stored log entry. The size provided to the combining function when
%% requested is determined by the underlying log provider.
-spec fold(
LogOrView :: log() | view(),
First :: log_index(),
Last :: log_index() | infinity,
SizeLimit :: non_neg_integer() | infinity,
Func ::
fun((Index :: log_index(), Entry :: log_entry(), Acc) -> Acc)
| fun((Index :: log_index(), Size :: non_neg_integer(), Entry :: log_entry(), Acc) -> Acc),
Acc
) -> {ok, Acc} | {error, Reason :: term()}.
fold(LogOrView, RawFirst, RawLast, SizeLimit, Func, Acc) ->
#raft_log{table = Table} = Log = log(LogOrView),
First = max(RawFirst, first_index(LogOrView)),
Last = min(RawLast, last_index(LogOrView)),
?RAFT_COUNT(Table, 'log.fold'),
?RAFT_COUNTV(Table, 'log.fold.total', Last - First + 1),
AdjFunc = if
is_function(Func, 3) -> fun (Index, _Size, Entry, InnerAcc) -> Func(Index, Entry, InnerAcc) end;
is_function(Func, 4) -> Func
end,
Provider = provider(Log),
case Provider:fold(Log, First, Last, SizeLimit, AdjFunc, Acc) of
{ok, AccOut} ->
{ok, AccOut};
{error, Reason} ->
?RAFT_COUNT(Table, 'log.fold.error'),
{error, Reason}
end.
-spec fold_binary(
LogOrView :: log() | view(),
First :: log_index(),
Last :: log_index() | infinity,
Func :: fun((Index :: log_index(), Entry :: binary(), Acc) -> Acc),
Acc
) -> {ok, Acc} | {error, term()}.
fold_binary(LogOrView, First, Last, Func, Acc) ->
fold_binary(LogOrView, First, Last, infinity, Func, Acc).
%% Call the provided combining function on the external term format of
%% successive log entries in the specified log starting from the specified
%% start index (inclusive) and ending at the specified end index (also
%% inclusive). The combining function will always be called with log entries
%% in log order starting with the first log entry that exists within the
%% provided range. The combining function will not be called for those log
%% indices within the provided range that do not have a stored log entry. The
%% size provided to the combining function when requested is determined by the
%% underlying log provider.
-spec fold_binary(
LogOrView :: log() | view(),
First :: log_index(),
Last :: log_index() | infinity,
SizeLimit :: non_neg_integer() | infinity,
Func :: fun((Index :: log_index(), Entry :: binary(), Acc) -> Acc),
Acc
) -> {ok, Acc} | {error, term()}.
fold_binary(LogOrView, RawFirst, RawLast, SizeLimit, Func, Acc) ->
#raft_log{table = Table} = Log = log(LogOrView),
First = max(RawFirst, first_index(LogOrView)),
Last = min(RawLast, last_index(LogOrView)),
?RAFT_COUNT(Table, 'log.fold_binary'),
?RAFT_COUNTV(Table, 'log.fold_binary.total', Last - First + 1),
Provider = provider(Log),
case Provider:fold_binary(Log, First, Last, SizeLimit, Func, Acc) of
{ok, AccOut} ->
{ok, AccOut};
{error, Reason} ->
?RAFT_COUNT(Table, 'log.fold_binary.error'),
{error, Reason}
end.
%% Folds over the terms in the log view of raw entries from the log provider
%% between the provided first and last log indices (inclusive).
%% If there exists a log term between the provided first and last indices then
%% the accumulator function will be called on at least that term.
%% This API provides no validation of the log indices and term passed by the
%% provider to the callback function.
-spec fold_terms(LogOrView :: log() | view(),
First :: log_index(),
Last :: log_index(),
Func :: fun((Index :: log_index(), Term :: log_term(), Acc) -> Acc),
Acc) ->
{ok, Acc} | {error, term()}.
fold_terms(#log_view{log = Log, first = LogFirst, last = LogLast}, First, Last, Func, Acc) ->
fold_terms_impl(Log, max(First, LogFirst), min(Last, LogLast), Func, Acc);
fold_terms(Log, First, Last, Func, Acc) ->
Provider = provider(Log),
LogFirst = Provider:first_index(Log),
LogLast = Provider:last_index(Log),
fold_terms_impl(Log, max(First, LogFirst), min(Last, LogLast), Func, Acc).
-spec fold_terms_impl(
Log :: log(),
First :: log_index(),
Last :: log_index(),
Func :: fun((Index :: log_index(), Term :: log_term(), Acc) -> Acc),
Acc :: term()
) -> {ok, Acc} | {error, term()}.
fold_terms_impl(#raft_log{table = Table} = Log, First, Last, Func, AccIn) ->
?RAFT_COUNT(Table, 'log.fold_terms'),
?RAFT_COUNTV(Table, 'log.fold_terms.total', Last - First + 1),
Provider = provider(Log),
case Provider:fold_terms(Log, First, Last, Func, AccIn) of
{ok, AccOut} ->
{ok, AccOut};
{error, Reason} ->
?RAFT_COUNT(Table, 'log.fold_terms.error'),
{error, Reason}
end.
%% Gets the term of entry at the provided log index. When using a log view
%% this function may return 'not_found' even if the underlying log entry still
%% exists if the entry is outside of the log view.
-spec term(LogOrView :: log() | view(), Index :: log_index()) -> {ok, Term :: log_term()} | not_found | {error, term()}.
term(#log_view{first = First, last = Last}, Index) when Index < First; Last < Index ->
not_found;
term(#log_view{log = Log}, Index) ->
Provider = provider(Log),
Provider:term(Log, Index);
term(Log, Index) ->
Provider = provider(Log),
Provider:term(Log, Index).
%% Gets the log entry at the provided log index. When using a log view
%% this function may return 'not_found' even if the underlying log entry still
%% exists if the entry is outside of the log view.
-spec get(LogOrView :: log() | view(), Index :: log_index()) -> {ok, Entry :: log_entry()} | not_found | {error, term()}.
get(#log_view{first = First, last = Last}, Index) when Index < First; Last < Index ->
not_found;
get(#log_view{log = #raft_log{table = Table} = Log}, Index) ->
?RAFT_COUNT(Table, 'log.get'),
Provider = provider(Log),
Provider:get(Log, Index);
get(#raft_log{table = Table} = Log, Index) ->
?RAFT_COUNT(Table, 'log.get'),
Provider = provider(Log),
Provider:get(Log, Index).
%% Fetch a contiguous range of log entries containing up to the specified
%% number of log entries starting at the provided index. When using a log view,
%% only those log entries that fall within the provided view will be returned.
-spec get(LogOrView :: log() | view(), Start :: log_index(), CountLimit :: non_neg_integer()) ->
{ok, Entries :: [log_entry()]} | {error, term()}.
get(LogOrView, Start, CountLimit) ->
get(LogOrView, Start, CountLimit, infinity).
%% Fetch a contiguous range of log entries containing up to the specified
%% number of log entries or the specified maximum total number of bytes (based
%% on the byte sizes reported by the underlying log provider) starting at the
%% provided index. If log entries exist at the provided starting index, then
%% at least one log entry will be returned. When using a log view, only those
%% log entries that fall within the provided view will be returned.
-spec get(
LogOrView :: log() | view(),
Start :: log_index(),
CountLimit :: non_neg_integer(),
SizeLimit :: non_neg_integer() | infinity
) -> {ok, Entries :: [log_entry()]} | {error, term()}.
get(LogOrView, Start, CountLimit, SizeLimit) ->
End = Start + CountLimit - 1,
try fold(LogOrView, Start, End, SizeLimit, fun get_method/3, {Start, []}) of
{ok, {_, EntriesRev}} ->
{ok, lists:reverse(EntriesRev)};
{error, Reason} ->
{error, Reason}
catch
throw:{missing, Index} ->
?RAFT_LOG_WARNING(
"[~0p] detected missing log entry ~0p while folding range ~0p ~~ ~0p",
[log_name(LogOrView), Index, Start, End]
),
{error, corruption}
end.
-spec get_method(Index :: log_index(), Entry :: log_entry(), Acc) -> Acc when
Acc :: {AccIndex :: log_index(), AccEntries :: [log_entry()]}.
get_method(Index, Entry, {Index, AccEntries}) ->
{Index + 1, [Entry | AccEntries]};
get_method(_, _, {AccIndex, _}) ->
throw({missing, AccIndex}).
-spec get_terms(LogOrView :: log() | view(), Start :: log_index(), CountLimit :: non_neg_integer()) ->
{ok, Terms :: [wa_raft_log:log_term()]} | {error, term()}.
get_terms(LogOrView, Start, CountLimit) ->
End = Start + CountLimit - 1,
try fold_terms(LogOrView, Start, End, fun get_terms_method/3, {Start, []}) of
{ok, {_, TermsRev}} ->
{ok, lists:reverse(TermsRev)};
{error, Reason} ->
{error, Reason}
catch
throw:{missing, Index} ->
?RAFT_LOG_WARNING(
"[~0p] detected missing log entry ~0p while folding range ~0p ~~ ~0p for terms",
[log_name(LogOrView), Index, Start, End]
),
{error, corruption}
end.
-spec get_terms_method(Index :: log_index(), Terms :: log_term(), Acc) -> Acc when
Acc :: {AccIndex :: log_index(), AccTerms :: [log_term()]}.
get_terms_method(Index, Entry, {Index, AccTerms}) ->
{Index + 1, [Entry | AccTerms]};
get_terms_method(_, _, {AccIndex, _}) ->
throw({missing, AccIndex}).
%% Produce a list of log entries in a format appropriate for inclusion within
%% a heartbeat to a peer containing up to the specified number of log entries
%% starting at the provided index. When using a log view, only those log
%% entries that fall within the provided view will be returned.
-spec entries(LogOrView :: log() | view(), Start :: log_index(), CountLimit :: non_neg_integer()) ->
{ok, Entries :: [log_entry() | binary()]} | {error, term()}.
entries(LogOrView, First, Count) ->
entries(LogOrView, First, Count, infinity).
%% Produce a list of log entries in a format appropriate for inclusion within
%% a heartbeat to a peer containing up to the specified number of log entries
%% or the specified maximum total number of bytes (based on the byte sizes
%% reported by the underlying log provider when returning log entries or the
%% byte size of each log entry binary when returning binaries) starting at the
%% provided index. If log entries exist at the provided starting index, then
%% at least one log entry will be returned. When using a log view, only those
%% log entries that fall within the provided view will be returned.
-spec entries(
LogOrView :: log() | view(),
Start :: log_index(),
CountLimit :: non_neg_integer(),
SizeLimit :: non_neg_integer() | infinity
) -> {ok, Entries :: [log_entry() | binary()]} | {error, term()}.
entries(LogOrView, Start, CountLimit, SizeLimit) ->
App = app(LogOrView),
Table = table(LogOrView),
Provider = provider(LogOrView),
End = Start + CountLimit - 1,
try
case erlang:function_exported(Provider, fold_binary, 6) andalso ?RAFT_LOG_HEARTBEAT_BINARY_ENTRIES(App, Table) of
true -> fold_binary(LogOrView, Start, End, SizeLimit, fun entries_method/3, {Start, []});
false -> fold(LogOrView, Start, End, SizeLimit, fun entries_method/3, {Start, []})
end
of
{ok, {_, EntriesRev}} ->
{ok, lists:reverse(EntriesRev)};
{error, Reason} ->
{error, Reason}
catch
throw:{missing, Index} ->
?RAFT_LOG_WARNING(
"[~0p] detected missing log entry ~0p while folding range ~0p ~~ ~0p for heartbeat",
[log_name(LogOrView), Index, Start, End]
),
{error, corruption}
end.
-spec entries_method(Index :: log_index(), Entry :: log_entry() | binary(), Acc) -> Acc when
Acc :: {AccIndex :: log_index(), AccEntries :: [log_entry() | binary()]}.
entries_method(Index, Entry, {Index, AccEntries}) ->
{Index + 1, [Entry | AccEntries]};
entries_method(_, _, {AccIndex, _}) ->
throw({missing, AccIndex}).
-spec config(LogOrView :: log() | view()) -> {ok, Index :: log_index(), Config :: wa_raft_server:config()} | not_found.
config(#log_view{config = undefined}) ->
not_found;
config(#log_view{first = First, config = {Index, _}}) when First > Index ->
% After trims, it is possible that we have a cached config from before the start
% of the log view. Don't return the cached config in this case.
not_found;
config(#log_view{config = {Index, Config}}) ->
{ok, Index, Config};
config(Log) ->
Provider = provider(Log),
case Provider:config(Log) of
{ok, Index, Config} -> {ok, Index, wa_raft_server:normalize_config(Config)};
Other -> Other
end.
%%-------------------------------------------------------------------
%% APIs for managing logs and log data
%%-------------------------------------------------------------------
%% Open the specified log (registered name or pid) at the provided position.
%% If the log does not contain the provided position, then the log is reset
%% to include it. Otherwise, the log is opened as is and may contain entries
%% before and after the provided position.
-spec open(PidOrName :: pid() | log_name(), Position :: log_pos()) -> {ok, View :: view()} | {error, term()}.
open(PidOrName, Position) ->
gen_server:call(PidOrName, {open, Position}, infinity).
%% Reset the log backing the provided log view to contain only the provided
%% position. The log entry data at the provided position will be 'undefined'.
-spec reset(View :: view(), Position :: log_pos()) -> {ok, NewView :: view()} | {error, term()}.
reset(#log_view{log = Log} = View, Position) ->
gen_server:call(log_name(Log), {reset, Position, View}, infinity).
%% Truncate the log by deleting all log entries in the log at and after the
%% provided log index. This operation is required to delete all data for the
%% affected log indices.
-spec truncate(View :: view(), Index :: log_index()) -> {ok, NewView :: view()} | {error, term()}.
truncate(#log_view{log = Log} = View, Index) ->
gen_server:call(log_name(Log), {truncate, Index, View}, infinity).
%% Trim the log by removing log entries before the provided log index.
%% This operation is not required to remove all data before the
%% provided log index immediately and can defer this work to future
%% trimming operations. This operation is asynchronous.
-spec trim(View :: view(), Index :: log_index()) -> {ok, NewView :: view()}.
trim(#log_view{log = Log, first = First} = View, Index) ->
gen_server:cast(log_name(Log), {trim, Index}),
{ok, View#log_view{first = max(Index, First)}}.
%% Perform a batched trimming (rotate) of the underlying log according
%% to application environment configuration values.
-spec rotate(View :: view(), Index :: log_index()) -> {ok, NewView :: view()}.
rotate(#log_view{log = #raft_log{application = App, table = Table}} = View, Index) ->
% Current rotation configuration is based on two configuration values,
% 'raft_max_log_records_per_file' which indicates after how many outstanding extra
% log entries are in the log should we trim and 'raft_max_log_records' which
% indicates how many additional log entries after the fully replicated index should
% be considered not extraneous and be kept by rotation.
Interval = ?RAFT_LOG_ROTATION_INTERVAL(App, Table),
Keep = ?RAFT_LOG_ROTATION_KEEP(App, Table, Interval),
rotate(View, Index, Interval, Keep).
%% Perform a batched trimming (rotate) of the underlying log where
%% we keep some number of log entries and only trigger trimming operations
%% every so often.
-spec rotate(View :: view(), Index :: log_index(), Interval :: pos_integer(), Keep :: non_neg_integer()) -> {ok, NewView :: view()}.
rotate(#log_view{log = #raft_log{table = Table}, first = First} = View, Index, Interval, Keep) when Index - Keep - First >= Interval ->
?RAFT_COUNT(Table, 'log.rotate'),
trim(View, Index - Keep);
rotate(#log_view{log = #raft_log{table = Table}} = View, _Index, _Interval, _Keep) ->
?RAFT_COUNT(Table, 'log.rotate'),
{ok, View}.
%% Try to flush any underlying log data that is not yet on disk to disk.
-spec flush(LogOrView :: log() | view()) -> ok.
flush(#log_view{log = Log}) ->
gen_server:cast(log_name(Log), flush);
flush(Log) ->
gen_server:cast(log_name(Log), flush).
%%-------------------------------------------------------------------
%% Internal API
%%-------------------------------------------------------------------
%% Get the default name for the RAFT log server associated with the
%% provided RAFT partition.
-spec default_name(Table :: wa_raft:table(), Partition :: wa_raft:partition()) -> Name :: atom().
default_name(Table, Partition) ->
% elp:ignore W0023 bounded atom, one per table/partition at startup
binary_to_atom(<<"raft_log_", (atom_to_binary(Table))/binary, "_", (integer_to_binary(Partition))/binary>>).
%% Get the registered name for the RAFT log server associated with the
%% provided RAFT partition or the default name if no registration exists.
-spec registered_name(Table :: wa_raft:table(), Partition :: wa_raft:partition()) -> Name :: atom().
registered_name(Table, Partition) ->
case wa_raft_part_sup:options(Table, Partition) of
undefined -> default_name(Table, Partition);
Options -> Options#raft_options.log_name
end.
-spec app(LogOrView :: log() | view()) -> App :: atom().
app(LogOrView) ->
(log(LogOrView))#raft_log.application.
-spec table(LogOrView :: log() | view()) -> Table :: wa_raft:table().
table(LogOrView) ->
(log(LogOrView))#raft_log.table.
-spec log(LogOrView :: log() | view()) -> Log :: log().
log(#log_view{log = Log}) ->
Log;
log(#raft_log{} = Log) ->
Log.
-spec log_name(LogOrView :: log() | view()) -> Name :: log_name().
log_name(#log_view{log = #raft_log{name = Name}}) ->
Name;
log_name(#raft_log{name = Name}) ->
Name.
-spec provider(LogOrView :: log() | view()) -> Provider :: module().
provider(#log_view{log = #raft_log{provider = Provider}}) ->
Provider;
provider(#raft_log{provider = Provider}) ->
Provider.
-spec refresh_config(View :: view()) -> NewView :: view().
refresh_config(#log_view{log = Log} = View) ->
Provider = provider(Log),
case Provider:config(Log) of
{ok, Index, Config} ->
View#log_view{config = {Index, wa_raft_server:normalize_config(Config)}};
not_found ->
View#log_view{config = undefined}
end.
%%-------------------------------------------------------------------
%% gen_server Callbacks
%%-------------------------------------------------------------------
-spec init(Options :: #raft_options{}) -> {ok, State :: #log_state{}}.
init(#raft_options{application = Application, table = Table, partition = Partition, log_name = Name, log_module = Provider}) ->
process_flag(trap_exit, true),
Log = #raft_log{
name = Name,
application = Application,
table = Table,
partition = Partition,
provider = Provider
},
ok = Provider:init(Log),
{ok, #log_state{log = Log}}.
-spec handle_call(Request, From :: term(), State :: #log_state{}) ->
{reply, Reply :: term(), NewState :: #log_state{}} |
{noreply, NewState :: #log_state{}}
when Request ::
{open, Position :: log_pos()} |
{reset, Position :: log_pos(), View :: view()} |
{truncate, Index :: log_index(), View :: view()}.
handle_call({open, Position}, _From, State) ->
{Reply, NewState} = handle_open(Position, State),
{reply, Reply, NewState};
handle_call({reset, Position, View}, _From, State) ->
case handle_reset(Position, View, State) of
{ok, NewView, NewState} ->
{reply, {ok, NewView}, NewState};
{error, Reason} ->
{reply, {error, Reason}, State}
end;
handle_call({truncate, Index, View}, _From, State) ->
case handle_truncate(Index, View, State) of
{ok, NewView, NewState} ->
{reply, {ok, NewView}, NewState};
{error, Reason} ->
{reply, {error, Reason}, State}
end;
handle_call(Request, From, #log_state{log = Log} = State) ->
?RAFT_LOG_NOTICE("[~p] got unrecognized call ~p from ~p", [log_name(Log), Request, From]),
{noreply, State}.
-spec handle_cast(Request, State :: #log_state{}) -> {noreply, NewState :: #log_state{}}
when Request :: flush | {trim, Index :: log_index()}.
handle_cast(flush, #log_state{log = #raft_log{table = Table} = Log} = State) ->
?RAFT_COUNT(Table, 'log.flush'),
Provider = provider(Log),
Provider:flush(Log),
{noreply, State};
handle_cast({trim, Index}, #log_state{log = Log} = State) ->
case handle_trim(Index, State) of
{ok, NewState} ->
{noreply, NewState};
{error, Reason} ->
?RAFT_LOG_WARNING("[~p] failed to trim log due to ~p", [log_name(Log), Reason]),
{noreply, State}
end;
handle_cast(Request, #log_state{log = Log} = State) ->
?RAFT_LOG_NOTICE("[~p] got unrecognized cast ~p", [log_name(Log), Request]),
{noreply, State}.
-spec terminate(Reason :: term(), State :: #log_state{}) -> term().
terminate(Reason, #log_state{log = Log, state = State}) ->
Provider = provider(Log),
?RAFT_LOG_NOTICE("[~p] terminating due to ~p", [Log, Reason]),
State =/= ?PROVIDER_NOT_OPENED andalso Provider:close(Log, State).
%%-------------------------------------------------------------------
%% RAFT Log Server Logic
%%-------------------------------------------------------------------
-spec handle_open(Position :: log_pos(), State :: #log_state{}) ->
{{ok, NewView :: view()} | {error, Reason :: term()}, NewState :: #log_state{}}.
handle_open(#raft_log_pos{index = Index, term = Term} = Position,
#log_state{log = #raft_log{name = Name, table = Table, provider = Provider} = Log} = State0) ->
?RAFT_COUNT(Table, 'log.open'),
?RAFT_LOG_NOTICE("[~p] opening log at position ~p:~p", [Name, Index, Term]),
case Provider:open(Log) of
{ok, ProviderState} ->
Action = case Provider:get(Log, Index) of
{ok, {Term, _Op}} ->
none;
{ok, {MismatchTerm, _Op}} ->
?RAFT_LOG_WARNING(
"[~p] resetting log due to expecting term ~p at ~p but log contains term ~p",
[Name, Term, Index, MismatchTerm]
),
reset;
not_found ->
reset;
Other ->
{failed, Other}
end,
State1 = State0#log_state{state = ProviderState},
View0 = #log_view{log = Log},
case Action of
none ->
?RAFT_COUNT(Table, 'log.open.normal'),
View1 = case Provider:first_index(Log) of
undefined ->
?RAFT_LOG_WARNING(
"[~p] opened log normally but the first index was not set",
[Name]
),
View0;
FirstIndex ->
View0#log_view{first = FirstIndex}
end,
View2 = case Provider:last_index(Log) of
undefined ->
?RAFT_LOG_WARNING(
"[~p] opened log normally but the last index was not set",
[Name]
),
View1;
LastIndex ->
View1#log_view{last = LastIndex}
end,
View3 = refresh_config(View2),
{{ok, View3}, State1};
reset ->
?RAFT_COUNT(Table, 'log.open.reset'),
case handle_reset(Position, View0, State1) of
{ok, View1, State2} ->
{{ok, View1}, State2};
{error, Reason} ->
?RAFT_COUNT(Table, 'log.open.reset.error'),
{{error, Reason}, State1}
end;
{failed, Return} ->
?RAFT_COUNT(Table, 'log.open.error'),
{Return, State1}
end;
{error, Reason} ->
?RAFT_COUNT(Table, 'log.open.error'),
{{error, Reason}, State0#log_state{state = ?PROVIDER_NOT_OPENED}}
end.
-spec handle_reset(Position :: log_pos(), View :: view(), State :: #log_state{}) ->
{ok, NewView :: view(), NewState :: #log_state{}} | {error, Reason :: term()}.
handle_reset(_Position, _View, #log_state{state = ?PROVIDER_NOT_OPENED}) ->
{error, not_open};
handle_reset(#raft_log_pos{index = 0, term = Term}, _View, #log_state{log = Log}) when Term =/= 0 ->
?RAFT_LOG_ERROR("[~p] rejects reset to index 0 with non-zero term ~p", [log_name(Log), Term]),
{error, invalid_position};
handle_reset(#raft_log_pos{index = Index, term = Term} = Position, View0,
#log_state{log = #raft_log{table = Table} = Log, state = ProviderState} = State0) ->
?RAFT_COUNT(Table, 'log.reset'),
?RAFT_LOG_NOTICE("[~p] resetting log to position ~p:~p", [log_name(Log), Index, Term]),
Provider = provider(Log),
case Provider:reset(Log, Position, ProviderState) of
{ok, NewProviderState} ->
View1 = View0#log_view{first = Index, last = Index, config = undefined},
State1 = State0#log_state{state = NewProviderState},
{ok, View1, State1};
{error, Reason} ->
?RAFT_COUNT(Table, 'log.reset.error'),
{error, Reason}
end.
-spec handle_truncate(Index :: log_index(), View :: view(), State :: #log_state{}) ->
{ok, NewView :: view(), NewState :: #log_state{}} | {error, Reason :: term()}.
handle_truncate(_Index, _View, #log_state{state = ?PROVIDER_NOT_OPENED}) ->
{error, not_open};
handle_truncate(Index, #log_view{first = First}, #log_state{log = Log}) when Index =< First ->
?RAFT_LOG_ERROR("[~p] rejects log deletion by truncation to ~p for log starting at ~p", [log_name(Log), Index, First]),
{error, invalid_position};
handle_truncate(Index, #log_view{last = Last} = View0, #log_state{log = #raft_log{table = Table} = Log, state = ProviderState} = State0) ->
?RAFT_COUNT(Table, 'log.truncate'),
?RAFT_LOG_NOTICE("[~p] truncating log from ~p to past ~p", [log_name(Log), Last, Index]),
Provider = provider(Log),
case Provider:truncate(Log, Index, ProviderState) of
{ok, NewProviderState} ->
View1 = View0#log_view{last = min(Last, Index - 1)},
View2 = refresh_config(View1),
State1 = State0#log_state{state = NewProviderState},
{ok, View2, State1};
{error, Reason} ->
?RAFT_COUNT(Table, 'log.truncate.error'),
{error, Reason}
end.
%% Trim is an asynchronous operation so we do not use the view here.
%% Rather, the wa_raft_log:trim/2 API will assume that the trim succeeded and
%% optimistically update the view to advance the start of the log to the provided index.
-spec handle_trim(Index :: log_index(), State :: #log_state{}) ->
{ok, NewState :: #log_state{}} | {error, Reason :: term()}.
handle_trim(_Index, #log_state{state = ?PROVIDER_NOT_OPENED}) ->
{error, not_open};
handle_trim(Index, #log_state{log = #raft_log{table = Table} = Log, state = ProviderState} = State) ->
?RAFT_COUNT(Table, 'log.trim'),
?RAFT_LOG_DEBUG("[~p] trimming log to ~p", [log_name(Log), Index]),
Provider = provider(Log),
case Provider:trim(Log, Index, ProviderState) of
{ok, NewProviderState} ->
{ok, State#log_state{state = NewProviderState}};
{error, Reason} ->
?RAFT_COUNT(Table, 'log.trim.error'),
{error, Reason}
end.
================================================
FILE: src/wa_raft_log_ets.erl
================================================
%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%%
%%% This source code is licensed under the Apache 2.0 license found in
%%% the LICENSE file in the root directory of this source tree.
%%%
%%% This module is an implementation of a completely in-memory RAFT
%%% log provider that uses ETS as a backing store for the log data.
%%% This module is only suitable as a log provider for an fully
%%% in-memory RAFT cluster and should not be used when any durability
%%% guarantees are required against node shutdown.
-module(wa_raft_log_ets).
-compile(warn_missing_spec_all).
-behaviour(wa_raft_log).
%% RAFT log provider interface for accessing log data
-export([
first_index/1,
last_index/1,
fold/6,
fold_terms/5,
get/2,
term/2,
config/1
]).
%% RAFT log provider interface for writing new log data
-export([
append/4
]).
%% RAFT log provider interface for managing underlying RAFT log
-export([
init/1,
open/1,
close/2,
reset/3,
truncate/3,
trim/3,
flush/1
]).
-include_lib("wa_raft/include/wa_raft.hrl").
-type state() :: undefined.
%%-------------------------------------------------------------------
%% RAFT log provider interface for accessing log data
%%-------------------------------------------------------------------
-spec first_index(Log :: wa_raft_log:log()) -> undefined | wa_raft_log:log_index().
first_index(#raft_log{name = Name}) ->
case ets:first(Name) of
'$end_of_table' -> undefined;
Key -> Key
end.
-spec last_index(Log :: wa_raft_log:log()) -> undefined | wa_raft_log:log_index().
last_index(#raft_log{name = Name}) ->
case ets:last(Name) of
'$end_of_table' -> undefined;
Key -> Key
end.
-spec fold(Log :: wa_raft_log:log(),
Start :: wa_raft_log:log_index() | '$end_of_table',
End :: wa_raft_log:log_index(),
SizeLimit :: non_neg_integer() | infinity,
Func :: fun((Index :: wa_raft_log:log_index(), Size :: non_neg_integer(), Entry :: wa_raft_log:log_entry(), Acc) -> Acc),
Acc) -> {ok, Acc}.
fold(Log, Start, End, SizeLimit, Func, Acc) ->
fold_impl(Log, Start, End, 0, SizeLimit, Func, Acc).
-spec fold_impl(
Log :: wa_raft_log:log(),
Start :: wa_raft_log:log_index() | '$end_of_table',
End :: wa_raft_log:log_index(),
Size :: non_neg_integer(),
SizeLimit :: non_neg_integer() | infinity,
Func :: fun((Index :: wa_raft_log:log_index(), Size :: non_neg_integer(), Entry :: wa_raft_log:log_entry(), Acc) -> Acc),
Acc
) -> {ok, Acc}.
fold_impl(_Log, Start, End, Size, SizeLimit, _Func, Acc) when End < Start; Size >= SizeLimit ->
{ok, Acc};
fold_impl(#raft_log{name = Name} = Log, Start, End, Size, SizeLimit, Func, Acc) ->
case ets:lookup(Name, Start) of
[{Start, Entry}] ->
EntrySize = erlang:external_size(Entry),
fold_impl(Log, ets:next(Name, Start), End, Size + EntrySize, SizeLimit, Func, Func(Start, EntrySize, Entry, Acc));
[] ->
fold_impl(Log, ets:next(Name, Start), End, Size, SizeLimit, Func, Acc)
end.
-spec fold_terms(Log :: wa_raft_log:log(),
Start :: wa_raft_log:log_index() | '$end_of_table',
End :: wa_raft_log:log_index(),
Func :: fun((Index :: wa_raft_log:log_index(), Entry :: wa_raft_log:log_term(), Acc) -> Acc),
Acc) -> {ok, Acc}.
fold_terms(Log, Start, End, Func, Acc) ->
fold_terms_impl(Log, Start, End, Func, Acc).
-spec fold_terms_impl(
Log :: wa_raft_log:log(),
Start :: wa_raft_log:log_index() | '$end_of_table',
End :: wa_raft_log:log_index(),
Func :: fun((Index :: wa_raft_log:log_index(), Term :: wa_raft_log:log_term(), Acc) -> Acc),
Acc
) -> {ok, Acc}.
fold_terms_impl(_Log, Start, End, _Func, Acc) when End < Start ->
{ok, Acc};
fold_terms_impl(#raft_log{name = Name} = Log, Start, End, Func, Acc) ->
case ets:lookup(Name, Start) of
[{Start, {Term, _Op}}] ->
fold_terms_impl(Log, ets:next(Name, Start), End, Func, Func(Start, Term, Acc));
[] ->
fold_terms_impl(Log, ets:next(Name, Start), End, Func, Acc)
end.
-spec get(Log :: wa_raft_log:log(), Index :: wa_raft_log:log_index()) -> {ok, Entry :: wa_raft_log:log_entry()} | not_found.
get(#raft_log{name = Name}, Index) ->
case ets:lookup(Name, Index) of
[{Index, Entry}] -> {ok, Entry};
[] -> not_found
end.
-spec term(Log :: wa_raft_log:log(), Index :: wa_raft_log:log_index()) -> {ok, Term :: wa_raft_log:log_term()} | not_found.
term(Log, Index) ->
case get(Log, Index) of
{ok, {Term, _Op}} -> {ok, Term};
not_found -> not_found
end.
-spec config(Log :: wa_raft_log:log()) -> {ok, Index :: wa_raft_log:log_index(), Entry :: wa_raft_server:config()} | not_found.
config(#raft_log{name = Name}) ->
case ets:select_reverse(Name, [{{'$1', {'_', {'_', {config, '$2'}}}}, [], [{{'$1', '$2'}}]}], 1) of
{[{Index, Config}], _Cont} -> {ok, Index, Config};
_ -> not_found
end.
%%-------------------------------------------------------------------
%% RAFT log provider interface for writing new log data
%%-------------------------------------------------------------------
-spec append(View :: wa_raft_log:view(), Entries :: [wa_raft_log:log_entry() | binary()], Mode :: strict | relaxed, Priority :: wa_raft_acceptor:priority()) -> ok.
append(View, Entries, _Mode, _Priority) ->
Name = wa_raft_log:log_name(View),
Last = wa_raft_log:last_index(View),
true = ets:insert(Name, append_decode(Last + 1, Entries)),
ok.
-spec append_decode(Index :: wa_raft_log:log_index(), Entries :: [wa_raft_log:log_entry() | binary()]) ->
[{wa_raft_log:log_index(), wa_raft_log:log_entry()}].
append_decode(_, []) ->
[];
append_decode(Index, [Entry | Entries]) ->
NewEntry =
case is_binary(Entry) of
true -> binary_to_term(Entry);
false -> Entry
end,
[{Index, NewEntry} | append_decode(Index + 1, Entries)].
%%-------------------------------------------------------------------
%% RAFT log provider interface for managing underlying RAFT log
%%-------------------------------------------------------------------
-spec init(Log :: wa_raft_log:log()) -> ok.
init(#raft_log{name = LogName}) ->
ets:new(LogName, [ordered_set, public, named_table]),
ok.
-spec open(Log :: wa_raft_log:log()) -> {ok, State :: state()}.
open(_Log) ->
{ok, undefined}.
-spec close(Log :: wa_raft_log:log(), State :: state()) -> ok.
close(_Log, _State) ->
ok.
-spec reset(Log :: wa_raft_log:log(), Position :: wa_raft_log:log_pos(), State :: state()) ->
{ok, NewState :: state()}.
reset(#raft_log{name = Name}, #raft_log_pos{index = Index, term = Term}, State) ->
true = ets:delete_all_objects(Name),
true = ets:insert(Name, {Index, {Term, undefined}}),
{ok, State}.
-spec truncate(Log :: wa_raft_log:log(), Index :: wa_raft_log:log_index() | '$end_of_table', State :: state()) ->
{ok, NewState :: state()}.
truncate(_Log, '$end_of_table', State) ->
{ok, State};
truncate(#raft_log{name = Name} = Log, Index, State) ->
true = ets:delete(Name, Index),
truncate(Log, ets:next(Name, Index), State).
-spec trim(Log :: wa_raft_log:log(), Index :: wa_raft_log:log_index(), State :: state()) ->
{ok, NewState :: state()}.
trim(Log, Index, State) ->
trim_impl(Log, Index - 1),
{ok, State}.
-spec trim_impl(Log :: wa_raft_log:log(), Index :: wa_raft_log:log_index() | '$end_of_table') -> ok.
trim_impl(_Log, '$end_of_table') ->
ok;
trim_impl(#raft_log{name = Name} = Log, Index) ->
true = ets:delete(Name, Index),
trim_impl(Log, ets:prev(Name, Index)).
-spec flush(Log :: wa_raft_log:log()) -> ok.
flush(_Log) ->
ok.
================================================
FILE: src/wa_raft_metrics.erl
================================================
%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%%
%%% This source code is licensed under the Apache 2.0 license found in
%%% the LICENSE file in the root directory of this source tree.
%%%
%%% Pluggable metrics interface to allow integration with different metrics system.
%%% The default implementation skips metrics logging and does nothing.
-module(wa_raft_metrics).
-compile(warn_missing_spec_all).
%% Public API
-export([
install/1
]).
%% Default Implementation
-export([
count/1,
countv/2,
gather/2,
gather_latency/2
]).
%% Public Types
-export_type([
metric/0,
value/0
]).
-include_lib("wa_raft/include/wa_raft.hrl").
%%-------------------------------------------------------------------
%% RAFT Metrics Behaviour
%%-------------------------------------------------------------------
%% Report a single occurrence of some metric.
-callback count(metric()) -> ok.
%% Report a number of occurrences of some metric.
-callback countv(metric(), value()) -> ok.
%% Report the measured value of an occurrence of some metric.
-callback gather(metric(), value()) -> ok.
%% Report the measured latency of an occurrence of some metric.
-callback gather_latency(metric(), value()) -> ok.
%%-------------------------------------------------------------------
%% Public Types
%%-------------------------------------------------------------------
-type metric() :: atom() | tuple().
-type value() :: integer().
%%-------------------------------------------------------------------
%% Public API
%%-------------------------------------------------------------------
%% Replace the previously installed or default module used to report
%% RAFT metrics with the provided module.
-spec install(Module :: module()) -> ok.
install(Module) ->
persistent_term:put(?RAFT_METRICS_MODULE_KEY, Module).
%%-------------------------------------------------------------------
%% Default Implementation
%%-------------------------------------------------------------------
-spec count(metric()) -> ok.
count(_Metric) ->
ok.
-spec countv(metric(), value()) -> ok.
countv(_Metric, _Value) ->
ok.
-spec gather(metric(), value()) -> ok.
gather(_Metric, _Value) ->
ok.
-spec gather_latency(metric(), value()) -> ok.
gather_latency(_Metric, _Value) ->
ok.
================================================
FILE: src/wa_raft_part_sup.erl
================================================
%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%%
%%% This source code is licensed under the Apache 2.0 license found in
%%% the LICENSE file in the root directory of this source tree.
%%%
%%% OTP Supervisor for monitoring RAFT processes. Correctness of RAFT
%%% relies on the consistency of the signaling between the processes,
%%% this supervisor is configured to restart all RAFT processes
%%% when any of them exits abnormally.
-module(wa_raft_part_sup).
-compile(warn_missing_spec_all).
-behaviour(supervisor).
%% OTP Supervision
-export([
child_spec/1,
child_spec/2,
start_link/2
]).
%% Options API
-export([
default_name/2,
registered_name/2
]).
%% Options API
-export([
partition_path/2,
default_partition_path/3,
registered_partition_path/2
]).
%% Options API
-export([
options/2
]).
%% Supervisor callbacks
-export([
init/1
]).
%% Test API
-export([
prepare_spec/2
]).
-include_lib("wa_raft/include/wa_raft.hrl").
-include_lib("wa_raft/include/wa_raft_logger.hrl").
%% Key in persistent_term for the options associated with a RAFT partition.
-define(OPTIONS_KEY(Table, Partition), {?MODULE, Table, Partition}).
%%-------------------------------------------------------------------
%% OTP supervision
%%-------------------------------------------------------------------
%% Returns a spec suitable for use with a `simple_one_for_one` supervisor.
-spec child_spec(Application :: atom()) -> supervisor:child_spec().
child_spec(Application) ->
#{
id => ?MODULE,
start => {?MODULE, start_link, [Application]},
restart => permanent,
shutdown => infinity,
type => supervisor,
modules => [?MODULE]
}.
-spec child_spec(Application :: atom(), Spec :: wa_raft:args()) -> supervisor:child_spec().
child_spec(Application, Spec) ->
#{
id => ?MODULE,
start => {?MODULE, start_link, [Application, Spec]},
restart => permanent,
shutdown => infinity,
type => supervisor,
modules => [?MODULE]
}.
-spec start_link(Application :: atom(), Spec :: wa_raft:args()) -> supervisor:startlink_ret().
start_link(Application, Spec) ->
%% First normalize the provided specification into a full options record.
Options = #raft_options{table = Table, partition = Partition, supervisor_name = Name} = normalize_spec(Application, Spec),
%% Then put the declared options for the current RAFT partition into
%% persistent term for access by shared resources and other services.
%% The RAFT options for a table are not expected to change during the
%% runtime of the RAFT application and so repeated updates should not
%% result in any GC load. Warn if this is case.
PrevOptions = persistent_term:get(?OPTIONS_KEY(Table, Partition), Options),
PrevOptions =/= Options andalso
?RAFT_LOG_WARNING(
?MODULE_STRING " storing changed options for RAFT partitition ~0p/~0p",
[Table, Partition]
),
ok = persistent_term:put(?OPTIONS_KEY(Table, Partition), Options),
supervisor:start_link({local, Name}, ?MODULE, Options).
%%-------------------------------------------------------------------
%% Internal API
%%-------------------------------------------------------------------
%% Get the default name for the RAFT partition supervisor associated with the
%% provided RAFT partition.
-spec default_name(Table :: wa_raft:table(), Partition :: wa_raft:partition()) -> Name :: atom().
default_name(Table, Partition) ->
% elp:ignore W0023 bounded atom for supervisor name
list_to_atom("raft_sup_" ++ atom_to_list(Table) ++ "_" ++ integer_to_list(Partition)).
%% Get the registered name for the RAFT partition supervisor associated with the
%% provided RAFT partition or the default name if no registration exists.
-spec registered_name(Table :: wa_raft:table(), Partition :: wa_raft:partition()) -> Name :: atom().
registered_name(Table, Partition) ->
case wa_raft_part_sup:options(Table, Partition) of
undefined -> default_name(Table, Partition);
Options -> Options#raft_options.supervisor_name
end.
%% Get the database directory that should be used by the RAFT partition that
%% is started under the provided application with the provided arguments.
-spec partition_path(Application :: atom(), Spec :: wa_raft:args()) -> Database :: file:filename().
partition_path(Application, #{table := Table, partition := Partition}) ->
Root = wa_raft_env:database_path(Application),
default_partition_path(Root, Table, Partition).
%% Get the default location for the database directory associated with the
%% provided RAFT partition given the database of the RAFT root.
-spec default_partition_path(Root :: file:filename(), Table :: wa_raft:table(), Partition :: wa_raft:partition()) -> Database :: file:filename().
default_partition_path(Root, Table, Partition) ->
filename:join(Root, atom_to_list(Table) ++ "." ++ integer_to_list(Partition)).
%% Get the registered database directory for the provided RAFT partition. An
%% error is raised if no registration exists.
-spec registered_partition_path(Table :: wa_raft:table(), Partition :: wa_raft:partition()) -> Database :: file:filename().
registered_partition_path(Table, Partition) ->
case wa_raft_part_sup:options(Table, Partition) of
undefined -> error({not_registered, Table, Partition});
Options -> Options#raft_options.database
end.
-spec options(Table :: wa_raft:table(), Partition :: wa_raft:partition()) -> #raft_options{} | undefined.
options(Table, Partition) ->
persistent_term:get(?OPTIONS_KEY(Table, Partition), undefined).
-spec normalize_spec(Application :: atom(), Spec :: wa_raft:args()) -> #raft_options{}.
normalize_spec(Application, #{table := Table, partition := Partition} = Spec) ->
Database = partition_path(Application, Spec),
ServerName = wa_raft_server:default_name(Table, Partition),
LogName = wa_raft_log:default_name(Table, Partition),
StorageName = wa_raft_storage:default_name(Table, Partition),
AcceptorName = wa_raft_acceptor:default_name(Table, Partition),
#raft_options{
application = Application,
table = Table,
partition = Partition,
% RAFT identity always uses the default RAFT server name for the partition
self = #raft_identity{name = wa_raft_server:default_name(Table, Partition), node = node()},
identifier = #raft_identifier{application = Application, table = Table, partition = Partition},
database = Database,
acceptor_name = AcceptorName,
distribution_module = maps:get(distribution_module, Spec, wa_raft_env:get_env(Application, raft_distribution_module, ?RAFT_DEFAULT_DISTRIBUTION_MODULE)),
log_name = LogName,
log_module = maps:get(log_module, Spec, wa_raft_env:get_env(Application, raft_log_module, ?RAFT_DEFAULT_LOG_MODULE)),
label_module = maps:get(label_module, Spec, wa_raft_env:get_env(Application, raft_label_module, ?RAFT_DEFAULT_LABEL_MODULE)),
queue_name = wa_raft_queue:default_name(Table, Partition),
queue_counters = wa_raft_queue:default_counters(),
queue_reads = wa_raft_queue:default_read_queue_name(Table, Partition),
server_name = ServerName,
storage_name = StorageName,
storage_module = maps:get(storage_module, Spec, wa_raft_env:get_env(Application, raft_storage_module, ?RAFT_DEFAULT_STORAGE_MODULE)),
supervisor_name = default_name(Table, Partition),
transport_cleanup_name = wa_raft_transport_cleanup:default_name(Table, Partition),
transport_directory = wa_raft_transport:default_directory(Database),
transport_module = maps:get(transport_module, Spec, wa_raft_env:get_env(Application, {raft_transport_module, transport_module}, ?RAFT_DEFAULT_TRANSPORT_MODULE))
}.
%%-------------------------------------------------------------------
%% Test API
%%-------------------------------------------------------------------
-spec prepare_spec(Application :: atom(), Spec :: wa_raft:args()) -> #raft_options{}.
prepare_spec(Application, Spec) ->
Options = #raft_options{table = Table, partition = Partition} = normalize_spec(Application, Spec),
ok = persistent_term:put(?OPTIONS_KEY(Table, Partition), Options),
Options.
%%-------------------------------------------------------------------
%% Supervisor callbacks
%%-------------------------------------------------------------------
-spec init(Options :: #raft_options{}) -> {ok, {supervisor:sup_flags(), [supervisor:child_spec()]}}.
init(Options) ->
ChildSpecs = [
wa_raft_queue:child_spec(Options),
wa_raft_storage:child_spec(Options),
wa_raft_log:child_spec(Options),
wa_raft_server:child_spec(Options),
wa_raft_acceptor:child_spec(Options),
wa_raft_transport_cleanup:child_spec(Options)
],
{ok, {#{strategy => one_for_all, intensity => 10, period => 1}, ChildSpecs}}.
================================================
FILE: src/wa_raft_queue.erl
================================================
%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%%
%%% This source code is licensed under the Apache 2.0 license found in
%%% the LICENSE file in the root directory of this source tree.
%%%
%%% This module implements tracking of pending requests and queue limits.
-module(wa_raft_queue).
-compile(warn_missing_spec_all).
-behaviour(gen_server).
%% PUBLIC API
-export([
queues/1,
queues/2,
commit_queue_size/2,
commit_queue_size/3,
commit_queue_full/2,
commit_queue_full/3,
apply_queue_size/1,
apply_queue_size/2,
apply_queue_byte_size/1,
apply_queue_byte_size/2,
apply_queue_full/1,
apply_queue_full/2
]).
%% INTERNAL API
-export([
default_name/2,
default_counters/0,
default_read_queue_name/2,
registered_name/2
]).
%% PENDING COMMIT QUEUE API
-export([
commit_started/2,
commit_cancelled/4,
commit_completed/4
]).
%% PENDING READ API
-export([
reserve_read/1,
submit_read/4,
query_reads/2,
fulfill_read/3,
fulfill_incomplete_read/3,
fulfill_all_reads/2
]).
%% APPLY QUEUE API
-export([
reserve_apply/2,
fulfill_apply/2
]).
%% OTP SUPERVISION
-export([
child_spec/1,
start_link/1
]).
%% QUEUE SERVER CALLBACKS
-export([
init/1,
handle_call/3,
handle_cast/2,
terminate/2
]).
%% TYPES
-export_type([
queues/0
]).
-include_lib("stdlib/include/ms_transform.hrl"). % used by ets:fun2ms
-include_lib("wa_raft/include/wa_raft.hrl").
-include_lib("wa_raft/include/wa_raft_logger.hrl").
%%-------------------------------------------------------------------
%% ETS table creation options shared by all queue tables
-define(RAFT_QUEUE_TABLE_OPTIONS, [named_table, public, {read_concurrency, true}, {write_concurrency, true}]).
%% Total number of counters for RAFT partition specfic counters
-define(RAFT_NUMBER_OF_QUEUE_SIZE_COUNTERS, 5).
%% Index into counter reference for counter tracking apply queue size
-define(RAFT_APPLY_QUEUE_SIZE_COUNTER, 1).
%% Index into counter reference for counter tracking apply total byte size
-define(RAFT_APPLY_QUEUE_BYTE_SIZE_COUNTER, 2).
%% Index into counter reference for counter tracking high priority commit queue size
-define(RAFT_HIGH_PRIORITY_COMMIT_QUEUE_SIZE_COUNTER, 3).
%% Index into counter reference for counter tracking low priority commit queue size
-define(RAFT_LOW_PRIORITY_COMMIT_QUEUE_SIZE_COUNTER, 4).
%% Index into counter reference for counter tracking read queue size
-define(RAFT_READ_QUEUE_SIZE_COUNTER, 5).
%%-------------------------------------------------------------------
%% INTERNAL TYPES
%%-------------------------------------------------------------------
-record(state, {
name :: atom()
}).
-record(queues, {
application :: atom(),
table :: wa_raft:table(),
counters :: atomics:atomics_ref(),
reads :: atom()
}).
-opaque queues() :: #queues{}.
%%-------------------------------------------------------------------
%% PUBLIC API
%%-------------------------------------------------------------------
-spec queues(Options :: #raft_options{}) -> Queues :: queues().
queues(Options) ->
#queues{
application = Options#raft_options.application,
table = Options#raft_options.table,
counters = Options#raft_options.queue_counters,
reads = Options#raft_options.queue_reads
}.
-spec queues(Table :: wa_raft:table(), Partition :: wa_raft:partition()) -> Queues :: queues() | undefined.
queues(Table, Partition) ->
case wa_raft_part_sup:options(Table, Partition) of
undefined -> undefined;
Options -> queues(Options)
end.
-spec commit_queue_size(Queues :: queues(), Priority :: wa_raft_acceptor:priority()) -> non_neg_integer().
commit_queue_size(#queues{counters = Counters}, high) ->
atomics:get(Counters, ?RAFT_HIGH_PRIORITY_COMMIT_QUEUE_SIZE_COUNTER);
commit_queue_size(#queues{counters = Counters}, low) ->
atomics:get(Counters, ?RAFT_LOW_PRIORITY_COMMIT_QUEUE_SIZE_COUNTER).
-spec commit_queue_size(Table :: wa_raft:table(), Partition :: wa_raft:partition(), Priority :: wa_raft_acceptor:priority()) -> non_neg_integer().
commit_queue_size(Table, Partition, Priority) ->
case queues(Table, Partition) of
undefined -> 0;
Queue -> commit_queue_size(Queue, Priority)
end.
-spec commit_queue_full(Queues :: queues(), Priority :: wa_raft_acceptor:priority()) -> boolean().
commit_queue_full(#queues{application = App, table = Table, counters = Counters}, high) ->
atomics:get(Counters, ?RAFT_HIGH_PRIORITY_COMMIT_QUEUE_SIZE_COUNTER) >= ?RAFT_MAX_PENDING_HIGH_PRIORITY_COMMITS(App, Table);
commit_queue_full(#queues{application = App, table = Table, counters = Counters}, low) ->
atomics:get(Counters, ?RAFT_LOW_PRIORITY_COMMIT_QUEUE_SIZE_COUNTER) >= ?RAFT_MAX_PENDING_LOW_PRIORITY_COMMITS(App, Table).
-spec commit_queue_full(Table :: wa_raft:table(), Partition :: wa_raft:partition(), Priority :: wa_raft_acceptor:priority()) -> boolean().
commit_queue_full(Table, Partition, Priority) ->
case queues(Table, Partition) of
undefined -> false;
Queues -> commit_queue_full(Queues, Priority)
end.
-spec apply_queue_size(Queues :: queues()) -> non_neg_integer().
apply_queue_size(#queues{counters = Counters}) ->
atomics:get(Counters, ?RAFT_APPLY_QUEUE_SIZE_COUNTER).
-spec apply_queue_size(Table :: wa_raft:table(), Partition :: wa_raft:partition()) -> non_neg_integer().
apply_queue_size(Table, Partition) ->
case queues(Table, Partition) of
undefined -> 0;
Queues -> apply_queue_size(Queues)
end.
-spec apply_queue_byte_size(Queues :: queues()) -> non_neg_integer().
apply_queue_byte_size(#queues{counters = Counters}) ->
atomics:get(Counters, ?RAFT_APPLY_QUEUE_BYTE_SIZE_COUNTER).
-spec apply_queue_byte_size(wa_raft:table(), wa_raft:partition()) -> non_neg_integer().
apply_queue_byte_size(Table, Partition) ->
case queues(Table, Partition) of
undefined -> 0;
Queues -> apply_queue_byte_size(Queues)
end.
-spec apply_queue_full(Queues :: queues()) -> boolean().
apply_queue_full(#queues{application = App, table = Table, counters = Counters}) ->
atomics:get(Counters, ?RAFT_APPLY_QUEUE_SIZE_COUNTER) >= ?RAFT_MAX_PENDING_APPLIES(App, Table) orelse
atomics:get(Counters, ?RAFT_APPLY_QUEUE_BYTE_SIZE_COUNTER) >= ?RAFT_MAX_PENDING_APPLY_BYTES(App, Table).
-spec apply_queue_full(wa_raft:table(), wa_raft:partition()) -> boolean().
apply_queue_full(Table, Partition) ->
case queues(Table, Partition) of
undefined -> false;
Queues -> apply_queue_full(Queues)
end.
%%-------------------------------------------------------------------
%% INTERNAL API
%%-------------------------------------------------------------------
%% Get the default name for the RAFT queue server associated with the
%% provided RAFT partition.
-spec default_name(Table :: wa_raft:table(), Partition :: wa_raft:partition()) -> Name :: atom().
default_name(Table, Partition) ->
% elp:ignore W0023 bounded atom, one per table/partition at startup
binary_to_atom(<<"raft_queue_", (atom_to_binary(Table))/bytes, "_", (integer_to_binary(Partition))/bytes>>).
%% Create a properly-sized atomics array for use by a RAFT queue
-spec default_counters() -> Counters :: atomics:atomics_ref().
default_counters() ->
atomics:new(?RAFT_NUMBER_OF_QUEUE_SIZE_COUNTERS, []).
%% Get the default name for the RAFT read queue ETS table associated with the
%% provided RAFT partition.
-spec default_read_queue_name(Table :: wa_raft:table(), Partition :: wa_raft:partition()) -> Name :: atom().
default_read_queue_name(Table, Partition) ->
% elp:ignore W0023 bounded atom, one per table/partition at startup
binary_to_atom(<<"raft_read_queue_", (atom_to_binary(Table))/bytes, "_", (integer_to_binary(Partition))/bytes>>).
%% Get the registered name for the RAFT queue server associated with the
%% provided RAFT partition or the default name if no registration exists.
-spec registered_name(Table :: wa_raft:table(), Partition :: wa_raft:partition()) -> Name :: atom().
registered_name(Table, Partition) ->
case wa_raft_part_sup:options(Table, Partition) of
undefined -> default_name(Table, Partition);
Options -> Options#raft_options.queue_name
end.
%%-------------------------------------------------------------------
%% PENDING COMMIT QUEUE API
%%-------------------------------------------------------------------
-spec commit_started(Queues :: queues(), Priority :: wa_raft_acceptor:priority()) -> ok | apply_queue_full | commit_queue_full.
commit_started(#queues{table = Table, counters = Counters} = Queues, Priority) ->
case commit_queue_full(Queues, Priority) of
true ->
?RAFT_COUNT(Table, {'acceptor.error.commit_queue_full', Priority}),
commit_queue_full;
false ->
case apply_queue_full(Queues) of
true ->
apply_queue_full;
false ->
PendingCommits =
case Priority of
high ->
atomics:add_get(Counters, ?RAFT_HIGH_PRIORITY_COMMIT_QUEUE_SIZE_COUNTER, 1);
low ->
atomics:add_get(Counters, ?RAFT_LOW_PRIORITY_COMMIT_QUEUE_SIZE_COUNTER, 1)
end,
?RAFT_GATHER(Table, {'acceptor.commit.request.pending', Priority}, PendingCommits),
ok
end
end.
-spec commit_cancelled(Queues :: queues(), From :: gen_server:from(), Reason :: wa_raft_acceptor:commit_error() | undefined, Priority :: wa_raft_acceptor:priority()) -> ok.
commit_cancelled(#queues{counters = Counters}, From, Reason, Priority) ->
case Priority of
high ->
atomics:sub(Counters, ?RAFT_HIGH_PRIORITY_COMMIT_QUEUE_SIZE_COUNTER, 1);
low ->
atomics:sub(Counters, ?RAFT_LOW_PRIORITY_COMMIT_QUEUE_SIZE_COUNTER, 1)
end,
Reason =/= undefined andalso gen_server:reply(From, Reason),
ok.
-spec commit_completed(Queues :: queues(), From :: gen_server:from(), Reply :: term(), Priority :: wa_raft_acceptor:priority()) -> ok.
commit_completed(#queues{counters = Counters}, From, Reply, Priority) ->
case Priority of
high ->
atomics:sub(Counters, ?RAFT_HIGH_PRIORITY_COMMIT_QUEUE_SIZE_COUNTER, 1);
low ->
atomics:sub(Counters, ?RAFT_LOW_PRIORITY_COMMIT_QUEUE_SIZE_COUNTER, 1)
end,
gen_server:reply(From, Reply),
ok.
%%-------------------------------------------------------------------
%% PENDING READ QUEUE API
%%-------------------------------------------------------------------
% Inspects the read and apply queues to check if a strong read is allowed
% to be submitted to the RAFT server currently. If so, then returns 'ok'
% and increments the read counter. Inspecting the queues and actually
% adding the read request to the table are done in two stages for reads
% because the acceptor does not have enough information to add the read
% to the ETS table directly.
-spec reserve_read(Queues :: queues()) -> ok | read_queue_full | apply_queue_full.
reserve_read(#queues{application = App, table = Table, counters = Counters}) ->
PendingReads = atomics:get(Counters, ?RAFT_READ_QUEUE_SIZE_COUNTER),
case PendingReads >= ?RAFT_MAX_PENDING_READS(App, Table) of
true -> read_queue_full;
false ->
case atomics:get(Counters, ?RAFT_APPLY_QUEUE_SIZE_COUNTER) >= ?RAFT_MAX_PENDING_APPLIES(App, Table) of
true -> apply_queue_full;
false ->
?RAFT_GATHER(Table, 'acceptor.strong_read.request.pending', PendingReads + 1),
atomics:add(Counters, ?RAFT_READ_QUEUE_SIZE_COUNTER, 1),
ok
end
end.
% Called from the RAFT server once it knows the proper ReadIndex for the
% read request to add the read request to the reads table for storage
% to handle upon applying.
-spec submit_read(Queues :: queues(), wa_raft_log:log_index(), From :: gen_server:from(), Command :: wa_raft_acceptor:command()) -> ok.
submit_read(#queues{reads = Reads}, ReadIndex, From, Command) ->
ets:insert(Reads, {{ReadIndex, make_ref()}, From, Command}),
ok.
-spec query_reads(Queues :: queues(), wa_raft_log:log_index() | infinity) -> [{{wa_raft_log:log_index(), reference()}, wa_raft_acceptor:command()}].
query_reads(#queues{reads = Reads}, MaxLogIndex) ->
MatchSpec = ets:fun2ms(
fun({{LogIndex, Reference}, _, Command}) when LogIndex =< MaxLogIndex ->
{{LogIndex, Reference}, Command}
end
),
ets:select(Reads, MatchSpec).
-spec fulfill_read(Queues :: queues(), {wa_raft_log:log_index(), reference()}, dynamic()) -> ok | not_found.
fulfill_read(#queues{counters = Counters, reads = Reads}, Reference, Reply) ->
case ets:take(Reads, Reference) of
[{Reference, From, _}] ->
atomics:sub(Counters, ?RAFT_READ_QUEUE_SIZE_COUNTER, 1),
gen_server:reply(From, Reply);
[] ->
not_found
end.
% Complete a read that was reserved by the RAFT acceptor but was rejected
% before it could be added to the read queue and so has no reference.
-spec fulfill_incomplete_read(Queues :: queues(), gen_server:from(), wa_raft_acceptor:read_error()) -> ok.
fulfill_incomplete_read(#queues{counters = Counters}, From, Reply) ->
atomics:sub(Counters, ?RAFT_READ_QUEUE_SIZE_COUNTER, 1),
gen_server:reply(From, Reply).
% Fulfill a pending reads with an error that indicates that the read was not completed.
-spec fulfill_all_reads(Queues :: queues(), wa_raft_acceptor:read_error()) -> ok.
fulfill_all_reads(#queues{counters = Counters, reads = Reads}, Reply) ->
lists:foreach(
fun ({Reference, _, _}) ->
case ets:take(Reads, Reference) of
[{Reference, From, _}] ->
atomics:sub(Counters, ?RAFT_READ_QUEUE_SIZE_COUNTER, 1),
gen_server:reply(From, Reply);
[] ->
ok
end
end, ets:tab2list(Reads)).
%%-------------------------------------------------------------------
%% APPLY QUEUE API
%%-------------------------------------------------------------------
-spec reserve_apply(Queues :: queues(), non_neg_integer()) -> ok.
reserve_apply(#queues{counters = Counters}, Size) ->
atomics:add(Counters, ?RAFT_APPLY_QUEUE_SIZE_COUNTER, 1),
atomics:add(Counters, ?RAFT_APPLY_QUEUE_BYTE_SIZE_COUNTER, Size).
-spec fulfill_apply(Queues :: queues(), non_neg_integer()) -> ok.
fulfill_apply(#queues{counters = Counters}, Size) ->
atomics:sub(Counters, ?RAFT_APPLY_QUEUE_SIZE_COUNTER, 1),
atomics:sub(Counters, ?RAFT_APPLY_QUEUE_BYTE_SIZE_COUNTER, Size).
%%-------------------------------------------------------------------
%% OTP SUPERVISION
%%-------------------------------------------------------------------
-spec child_spec(Options :: #raft_options{}) -> supervisor:child_spec().
child_spec(Options) ->
#{
id => ?MODULE,
start => {?MODULE, start_link, [Options]},
restart => transient,
shutdown => 1000,
modules => [?MODULE]
}.
-spec start_link(Options :: #raft_options{}) -> gen_server:start_ret().
start_link(#raft_options{queue_name = Name} = Options) ->
gen_server:start_link({local, Name}, ?MODULE, Options, []).
%%-------------------------------------------------------------------
%% QUEUE SERVER CALLBACKS
%%-------------------------------------------------------------------
-spec init(Options :: #raft_options{}) -> {ok, #state{}}.
init(
#raft_options{
table = Table,
partition = Partition,
queue_name = Name,
queue_counters = Counters,
queue_reads = ReadsName
}
) ->
process_flag(trap_exit, true),
?RAFT_LOG_NOTICE(
"Queue[~p] starting for partition ~0p/~0p with read queue ~0p",
[Name, Table, Partition, ReadsName]
),
% The queue process is the first process in the supervision for a single
% RAFT partition. The supervisor is configured to restart all processes if
% even a single process fails. Since the queue process is starting up, all
% queues tracked should be empty so reset all counters.
[atomics:put(Counters, Index, 0) || Index <- lists:seq(1, ?RAFT_NUMBER_OF_QUEUE_SIZE_COUNTERS)],
% Create ETS table for pending reads.
ets:new(ReadsName, [ordered_set | ?RAFT_QUEUE_TABLE_OPTIONS]),
{ok, #state{name = Name}}.
-spec handle_call(Request :: term(), From :: gen_server:from(), State :: #state{}) -> {noreply, #state{}}.
handle_call(Request, From, #state{name = Name} = State) ->
?RAFT_LOG_NOTICE("Queue[~p] got unexpected request ~0P from ~0p", [Name, Request, 100, From]),
{noreply, State}.
-spec handle_cast(Request :: term(), State :: #state{}) -> {noreply, #state{}}.
handle_cast(Request, #state{name = Name} = State) ->
?RAFT_LOG_NOTICE("Queue[~p] got unexpected call ~0P", [Name, Request, 100]),
{noreply, State}.
-spec terminate(Reason :: dynamic(), State :: #state{}) -> ok.
terminate(Reason, #state{name = Name}) ->
?RAFT_LOG_NOTICE("Queue[~p] terminating due to ~0P", [Name, Reason, 100]).
================================================
FILE: src/wa_raft_server.erl
================================================
%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
%%%
%%% This source code is licensed under the Apache 2.0 license found in
%%% the LICENSE file in the root directory of this source tree.
%%%
%%% This module implements RPC of raft consensus protocol. See raft spec
%%% on https://raft.github.io/. A wa_raft instance is a participant in
%%% a consensus group. Each participant plays a certain role (follower,
%%% leader or candidate). The mission of a consensus group is to
%%% implement a replicated state machine in a distributed cluster.
-module(wa_raft_server).
-compile(warn_missing_spec_all).
-behaviour(gen_statem).
-compile({inline, [require_valid_state/1]}).
%%------------------------------------------------------------------------------
%% RAFT Server - OTP Supervision
%%------------------------------------------------------------------------------
-export([
child_spec/1,
start_link/1
]).
%%------------------------------------------------------------------------------
%% RAFT Server - Public APIs - RAFT Cluster Configuration
%%------------------------------------------------------------------------------
-export([
latest_config_version/0
]).
%% Inspection of cluster configuration
-export([
get_config_version/1,
get_config_participants/1,
get_config_members/1,
get_config_full_members/1,
get_config_witness_members/1,
get_config_witnesses/1,
is_data_replica/2,
is_witness/2
]).
%% Creation and modification of cluster configuration
-export([
make_config/0,
make_config/1,
make_config/2,
make_config/3,
normalize_config/1
]).
% Stubbing log entries for witnesses
-export([
stub_entries_for_witness/1
]).
%% Modification of cluster configuration
-export([
set_config_members/2,
set_config_members/3,
set_config_members/4
]).
%%------------------------------------------------------------------------------
%% RAFT Server - Public APIs
%%------------------------------------------------------------------------------
-export([
get_current_config/1
]).
-export([
status/1,
status/2,
membership/1
]).
%%------------------------------------------------------------------------------
%% RAFT Server - Internal APIs - Local Options
%%------------------------------------------------------------------------------
-export([
default_name/2,
registered_name/2
]).
%%------------------------------------------------------------------------------
%% RAFT Server - Internal APIs - RPC Handling
%%------------------------------------------------------------------------------
-export([
make_rpc/3,
parse_rpc/2
]).
%%------------------------------------------------------------------------------
%% RAFT Server - Internal APIs - Commands
%%------------------------------------------------------------------------------
-export([
commit/4,
read/2,
snapshot_available/3,
adjust_config/2,
adjust_config/3,
adjust_config/4,
adjust_membership/3,
adjust_membership/4,
refresh_config/1,
trigger_election/1,
trigger_election/2,
promote/2,
promote/3,
resign/1,
handover/1,
handover/2,
handover_candidates/1,
is_peer_ready/2,
disable/2,
enable/1,
bootstrap/4,
notify_complete/1
]).
%%------------------------------------------------------------------------------
%% RAFT Server - State Machine Implementation
%%------------------------------------------------------------------------------
%% General callbacks
-export([
init/1,
callback_mode/0,
terminate/3
]).
%% State-specific callbacks
-export([
stalled/3,
leader/3,
follower/3,
candidate/3,
disabled/3,
witness/3
]).
%%------------------------------------------------------------------------------
%% RAFT Server - Test Exports
%%------------------------------------------------------------------------------
-ifdef(TEST).
-export([
compute_quorum/3,
config/1,
max_index_to_apply/3,
leader_adjust_config/2
]).
-endif.
%%------------------------------------------------------------------------------
%% RAFT Server - Public Types
%%------------------------------------------------------------------------------
-export_type([
state/0,
config/0,
config_all/0,
membership/0,
status/0,
config_action/0
]).
%%------------------------------------------------------------------------------
-include_lib("wa_raft/include/wa_raft.hrl").
-include_lib("wa_raft/include/wa_raft_logger.hrl").
-include_lib("wa_raft/include/wa_raft_rpc.hrl").
%%------------------------------------------------------------------------------
%% Section 5.2. Randomized election timeout for fast election and to avoid split votes
-define(ELECTION_TIMEOUT(State), {state_timeout, random_election_timeout(State), election}).
%% Timeout in milliseconds before the next heartbeat is to be sent by a RAFT leader with no pending log entries
-define(HEARTBEAT_TIMEOUT(State), {state_timeout, ?RAFT_HEARTBEAT_INTERVAL(State#raft_state.application, State#raft_state.table), heartbeat}).
%% Timeout in milliseconds before the next heartbeat is to be sent by a RAFT leader with pending log entries
-define(COMMIT_BATCH_TIMEOUT(State), {state_timeout, ?RAFT_COMMIT_BATCH_INTERVAL(State#raft_state.application, State#raft_state.table), batch_commit}).
%%------------------------------------------------------------------------------
-define(SERVER_LOG_PREFIX, "Server[~0p, term ~0p, ~0p] ").
-define(SERVER_LOG_FORMAT(Format), ?SERVER_LOG_PREFIX Format).
-define(SERVER_LOG_ARGS(State, Data, Args), [(Data)#raft_state.name, (Data)#raft_state.current_term, require_valid_state(State) | Args]).
% elp:ignore W0002 (unused_macro) - Keeping for consistency
-define(SERVER_LOG_ERROR(Data, Format, Args), ?SERVER_LOG_ERROR(?FUNCTION_NAME, Data, Format, Args)).
-define(SERVER_LOG_ERROR(State, Data, Format, Args), ?RAFT_LOG_ERROR(?SERVER_LOG_FORMAT(Format), ?SERVER_LOG_ARGS(State, Data, Args))).
-define(SERVER_LOG_WARNING(Data, Format, Args), ?SERVER_LOG_WARNING(?FUNCTION_NAME, Data, Format, Args)).
-define(SERVER_LOG_WARNING(State, Data, Format, Args), ?RAFT_LOG_WARNING(?SERVER_LOG_FORMAT(Format), ?SERVER_LOG_ARGS(State, Data, Args))).
-define(SERVER_LOG_NOTICE(Data, Format, Args), ?SERVER_LOG_NOT
gitextract_f4__cq5a/
├── .github/
│ └── workflows/
│ └── build.yml
├── .gitignore
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── LICENSE
├── README.md
├── examples/
│ └── kvstore/
│ └── src/
│ ├── kvstore.app.src
│ ├── kvstore_app.erl
│ ├── kvstore_client.erl
│ └── kvstore_sup.erl
├── include/
│ ├── wa_raft.hrl
│ ├── wa_raft_logger.hrl
│ └── wa_raft_rpc.hrl
├── rebar.config
└── src/
├── wa_raft.app.src
├── wa_raft.erl
├── wa_raft_acceptor.erl
├── wa_raft_app.erl
├── wa_raft_app_sup.erl
├── wa_raft_dist_transport.erl
├── wa_raft_distribution.erl
├── wa_raft_durable_state.erl
├── wa_raft_env.erl
├── wa_raft_info.erl
├── wa_raft_label.erl
├── wa_raft_log.erl
├── wa_raft_log_ets.erl
├── wa_raft_metrics.erl
├── wa_raft_part_sup.erl
├── wa_raft_queue.erl
├── wa_raft_server.erl
├── wa_raft_snapshot_catchup.erl
├── wa_raft_storage.erl
├── wa_raft_storage_ets.erl
├── wa_raft_sup.erl
├── wa_raft_transport.erl
├── wa_raft_transport_cleanup.erl
├── wa_raft_transport_sup.erl
├── wa_raft_transport_target_sup.erl
└── wa_raft_transport_worker.erl
Condensed preview — 40 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (505K chars).
[
{
"path": ".github/workflows/build.yml",
"chars": 1285,
"preview": "name: Build\non:\n push:\n branches:\n - '*'\n pull_request:\n types:\n - opened\n - synchronize\njobs:\n "
},
{
"path": ".gitignore",
"chars": 206,
"preview": ".DS_Store\n.rebar3\n.eunit\n*.o\n*.beam\n*.plt\n*.swp\n*.swo\nebin\nlog\nerl_crash.dump\n.rebar\n_build\n.idea\nrebar3.crashdump\n.edts"
},
{
"path": "CODE_OF_CONDUCT.md",
"chars": 240,
"preview": "# Code of Conduct\n\nMeta has adopted a Code of Conduct that we expect project participants to adhere to.\nPlease read the "
},
{
"path": "CONTRIBUTING.md",
"chars": 1433,
"preview": "# Contributing to WhatsApp Raft\nWe want to make contributing to this project as easy and transparent as\npossible.\n\n## Ou"
},
{
"path": "LICENSE",
"chars": 11325,
"preview": "Apache License\n Version 2.0, January 2004\n http://www.apache.org/licens"
},
{
"path": "README.md",
"chars": 5263,
"preview": "# WhatsApp Raft - WARaft\n\nWARaft is a Raft library in Erlang by WhatsApp. It provides an Erlang implementation to obtain"
},
{
"path": "examples/kvstore/src/kvstore.app.src",
"chars": 860,
"preview": "%% % @format\n\n%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%\n%% This source code is licen"
},
{
"path": "examples/kvstore/src/kvstore_app.erl",
"chars": 524,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "examples/kvstore/src/kvstore_client.erl",
"chars": 1488,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "examples/kvstore/src/kvstore_sup.erl",
"chars": 1063,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "include/wa_raft.hrl",
"chars": 27716,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "include/wa_raft_logger.hrl",
"chars": 1252,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "include/wa_raft_rpc.hrl",
"chars": 6198,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "rebar.config",
"chars": 439,
"preview": "{erl_opts, [ debug_info\n , warnings_as_errors\n , warn_export_vars\n , warn_unused_import\n "
},
{
"path": "src/wa_raft.app.src",
"chars": 530,
"preview": "%% % @format\n\n%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is li"
},
{
"path": "src/wa_raft.erl",
"chars": 1046,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_acceptor.erl",
"chars": 14106,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_app.erl",
"chars": 628,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_app_sup.erl",
"chars": 1471,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_dist_transport.erl",
"chars": 10876,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_distribution.erl",
"chars": 1192,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_durable_state.erl",
"chars": 5589,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_env.erl",
"chars": 4419,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_info.erl",
"chars": 5914,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_label.erl",
"chars": 737,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_log.erl",
"chars": 47245,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_log_ets.erl",
"chars": 7867,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_metrics.erl",
"chars": 2320,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_part_sup.erl",
"chars": 9040,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_queue.erl",
"chars": 17377,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_server.erl",
"chars": 174632,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_snapshot_catchup.erl",
"chars": 11715,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_storage.erl",
"chars": 43866,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_storage_ets.erl",
"chars": 8672,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_sup.erl",
"chars": 7055,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_transport.erl",
"chars": 38568,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_transport_cleanup.erl",
"chars": 6033,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_transport_sup.erl",
"chars": 1744,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_transport_target_sup.erl",
"chars": 1873,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
},
{
"path": "src/wa_raft_transport_worker.erl",
"chars": 7067,
"preview": "%%% Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n%%%\n%%% This source code is licensed under t"
}
]
About this extraction
This page contains the full source code of the WhatsApp/waraft GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 40 files (479.4 KB), approximately 114.9k tokens. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.