Showing preview only (508K chars total). Download the full file or copy to clipboard to get everything.
Repository: apple/swift-cluster-membership
Branch: main
Commit: 1e5f7b7a9039
Files: 67
Total size: 483.6 KB
Directory structure:
gitextract_bytsy_b1/
├── .editorconfig
├── .github/
│ ├── ISSUE_TEMPLATE/
│ │ └── issue-template.md
│ ├── PULL_REQUEST_TEMPLATE.md
│ ├── dependabot.yml
│ ├── release.yml
│ └── workflows/
│ ├── main.yml
│ ├── pull_request.yml
│ └── pull_request_label.yml
├── .gitignore
├── .licenseignore
├── .spi.yml
├── .swift-format
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── CONTRIBUTORS.txt
├── HANDBOOK.md
├── LICENSE.txt
├── NOTICE.txt
├── Package.swift
├── README.md
├── Samples/
│ ├── .gitignore
│ ├── Package.swift
│ ├── README.md
│ ├── Sources/
│ │ └── SWIMNIOSampleCluster/
│ │ ├── SWIMNIOSampleNode.swift
│ │ └── main.swift
│ └── Tests/
│ └── NoopTests/
│ └── SampleTest.swift
├── Sources/
│ ├── ClusterMembership/
│ │ └── Node.swift
│ ├── SWIM/
│ │ ├── Docs.docc/
│ │ │ ├── images/
│ │ │ │ ├── ping_pingreq_cycle.graffle
│ │ │ │ └── swim_lifecycle.graffle
│ │ │ └── index.md
│ │ ├── Events.swift
│ │ ├── Member.swift
│ │ ├── Metrics.swift
│ │ ├── Peer.swift
│ │ ├── SWIM.swift
│ │ ├── SWIMInstance.swift
│ │ ├── SWIMProtocol.swift
│ │ ├── Settings.swift
│ │ ├── Status.swift
│ │ └── Utils/
│ │ ├── Heap.swift
│ │ ├── String+Extensions.swift
│ │ ├── _PrettyLog.swift
│ │ └── time.swift
│ └── SWIMNIOExample/
│ ├── Coding.swift
│ ├── Logging.swift
│ ├── Message.swift
│ ├── NIOPeer.swift
│ ├── SWIMNIOHandler.swift
│ ├── SWIMNIOShell.swift
│ ├── Settings.swift
│ └── Utils/
│ ├── String+Extensions.swift
│ └── time.swift
├── Tests/
│ ├── ClusterMembershipDocumentationTests/
│ │ └── SWIMDocExamples.swift
│ ├── ClusterMembershipTests/
│ │ └── NodeTests.swift
│ ├── SWIMNIOExampleTests/
│ │ ├── CodingTests.swift
│ │ ├── SWIMNIOClusteredTests.swift
│ │ ├── SWIMNIOEventClusteredTests.swift
│ │ ├── SWIMNIOMetricsTests.swift
│ │ └── Utils/
│ │ └── BaseXCTestCases.swift
│ ├── SWIMTestKit/
│ │ ├── LogCapture.swift
│ │ └── TestMetrics.swift
│ └── SWIMTests/
│ ├── HeapTests.swift
│ ├── SWIMInstanceTests.swift
│ ├── SWIMMetricsTests.swift
│ ├── SWIMSettingsTests.swift
│ └── TestPeer.swift
└── dev/
└── git.commit.template
================================================
FILE CONTENTS
================================================
================================================
FILE: .editorconfig
================================================
root = true
[*]
indent_style = space
indent_size = 4
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true
================================================
FILE: .github/ISSUE_TEMPLATE/issue-template.md
================================================
---
name: Issue Template
about: Template for reporting general issues with the library
title: ''
labels: 0 - new
assignees: ''
---
### Expected behavior
<!-- _what you expected to happen_ -->
### Actual behavior
<!-- _what actually happened_ -->
### Steps to reproduce
<!-- _steps to reproduce the issue, or link to example app / reproducer_ -->
### If possible, minimal yet complete reproducer code (or URL to code)
<!-- _anything to help us reproducing the issue_ -->
### Version/commit hash
<!-- _the tag/commit hash_ -->
### Swift & OS version (output of `swift --version && uname -a`)
================================================
FILE: .github/PULL_REQUEST_TEMPLATE.md
================================================
_[One line description of your change]_
### Motivation:
_[Explain here the context, and why you're making that change. What is the problem you're trying to solve.]_
### Modifications:
_[Describe the modifications you've done.]_
### Result:
- Resolves #
- _[After your change, what will change.]_
================================================
FILE: .github/dependabot.yml
================================================
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
================================================
FILE: .github/release.yml
================================================
changelog:
categories:
- title: SemVer Major
labels:
- ⚠️ semver/major
- title: SemVer Minor
labels:
- 🆕 semver/minor
- title: SemVer Patch
labels:
- 🔨 semver/patch
- title: Other Changes
labels:
- semver/none
================================================
FILE: .github/workflows/main.yml
================================================
name: Main
permissions:
contents: read
on:
push:
branches: [main]
schedule:
- cron: "0 8,20 * * *"
jobs:
unit-tests:
name: Unit tests
uses: apple/swift-nio/.github/workflows/unit_tests.yml@main
with:
linux_5_10_arguments_override: "--explicit-target-dependency-import-check error"
linux_6_0_arguments_override: "--explicit-target-dependency-import-check error -Xswiftc -require-explicit-sendable"
linux_6_1_arguments_override: "--explicit-target-dependency-import-check error -Xswiftc -require-explicit-sendable"
linux_6_2_arguments_override: "--explicit-target-dependency-import-check error -Xswiftc -require-explicit-sendable"
linux_6_3_arguments_override: "--explicit-target-dependency-import-check error -Xswiftc -require-explicit-sendable"
linux_nightly_next_arguments_override: "--explicit-target-dependency-import-check error -Xswiftc -require-explicit-sendable"
linux_nightly_main_arguments_override: "--explicit-target-dependency-import-check error -Xswiftc -require-explicit-sendable"
construct-samples-matrix:
name: Construct samples matrix
runs-on: ubuntu-latest
outputs:
samples-matrix: '${{ steps.generate-matrix.outputs.samples-matrix }}'
steps:
- name: Checkout repository
uses: actions/checkout@v6
with:
persist-credentials: false
- id: generate-matrix
run: echo "samples-matrix=$(curl -s https://raw.githubusercontent.com/apple/swift-nio/main/scripts/generate_matrix.sh | bash)" >> "$GITHUB_OUTPUT"
env:
MATRIX_LINUX_COMMAND: "swift build --package-path Samples --explicit-target-dependency-import-check error"
samples:
name: Samples
needs: construct-samples-matrix
uses: apple/swift-nio/.github/workflows/swift_test_matrix.yml@main
with:
name: "Samples"
matrix_string: '${{ needs.construct-samples-matrix.outputs.samples-matrix }}'
macos-tests:
name: macOS tests
uses: apple/swift-nio/.github/workflows/macos_tests.yml@main
with:
build_scheme: "none" # no defined build schemes
macos_xcode_build_enabled: false
ios_xcode_build_enabled: false
watchos_xcode_build_enabled: false
tvos_xcode_build_enabled: false
visionos_xcode_build_enabled: false
static-sdk:
name: Static SDK
# Workaround https://github.com/nektos/act/issues/1875
uses: apple/swift-nio/.github/workflows/static_sdk.yml@main
release-builds:
name: Release builds
uses: apple/swift-nio/.github/workflows/release_builds.yml@main
================================================
FILE: .github/workflows/pull_request.yml
================================================
name: PR
permissions:
contents: read
on:
pull_request:
types: [opened, reopened, synchronize]
jobs:
soundness:
name: Soundness
uses: swiftlang/github-workflows/.github/workflows/soundness.yml@0.0.10
with:
license_header_check_project_name: "Swift Cluster Membership"
unit-tests:
name: Unit tests
uses: apple/swift-nio/.github/workflows/unit_tests.yml@main
with:
linux_5_10_arguments_override: "--explicit-target-dependency-import-check error"
linux_6_0_arguments_override: "--explicit-target-dependency-import-check error -Xswiftc -require-explicit-sendable"
linux_6_1_arguments_override: "--explicit-target-dependency-import-check error -Xswiftc -require-explicit-sendable"
linux_6_2_arguments_override: "--explicit-target-dependency-import-check error -Xswiftc -require-explicit-sendable"
linux_6_3_arguments_override: "--explicit-target-dependency-import-check error -Xswiftc -require-explicit-sendable"
linux_nightly_next_arguments_override: "--explicit-target-dependency-import-check error -Xswiftc -require-explicit-sendable"
linux_nightly_main_arguments_override: "--explicit-target-dependency-import-check error -Xswiftc -require-explicit-sendable"
construct-samples-matrix:
name: Construct samples matrix
runs-on: ubuntu-latest
outputs:
samples-matrix: '${{ steps.generate-matrix.outputs.samples-matrix }}'
steps:
- name: Checkout repository
uses: actions/checkout@v6
with:
persist-credentials: false
- id: generate-matrix
run: echo "samples-matrix=$(curl -s https://raw.githubusercontent.com/apple/swift-nio/main/scripts/generate_matrix.sh | bash)" >> "$GITHUB_OUTPUT"
env:
MATRIX_LINUX_COMMAND: "swift build --package-path Samples --explicit-target-dependency-import-check error"
samples:
name: Samples
needs: construct-samples-matrix
uses: apple/swift-nio/.github/workflows/swift_test_matrix.yml@main
with:
name: "Samples"
matrix_string: '${{ needs.construct-samples-matrix.outputs.samples-matrix }}'
cxx-interop:
name: Cxx interop
uses: apple/swift-nio/.github/workflows/cxx_interop.yml@main
static-sdk:
name: Static SDK
# Workaround https://github.com/nektos/act/issues/1875
uses: apple/swift-nio/.github/workflows/static_sdk.yml@main
release-builds:
name: Release builds
uses: apple/swift-nio/.github/workflows/release_builds.yml@main
================================================
FILE: .github/workflows/pull_request_label.yml
================================================
name: PR label
permissions:
contents: read
on:
pull_request:
types: [labeled, unlabeled, opened, reopened, synchronize]
jobs:
semver-label-check:
name: Semantic version label check
runs-on: ubuntu-latest
timeout-minutes: 1
steps:
- name: Checkout repository
uses: actions/checkout@v6
with:
persist-credentials: false
- name: Check for Semantic Version label
uses: apple/swift-nio/.github/actions/pull_request_semver_label_checker@main
================================================
FILE: .gitignore
================================================
.DS_Store
.swift-version
*.orig
*.app
/.build
/Samples/.build
/.SourceKitten
/Packages
.xcode
*.app
/*.xcodeproj
Samples/swift-cluster-membership-samples.xcodeproj
.xcode
.idea
Samples/swift-cluster-membership-samples.xcodeproj
# rendered docs output dirs
/reference/
/api/
.swiftpm/
Package.resolved
================================================
FILE: .licenseignore
================================================
.gitignore
**/.gitignore
.licenseignore
.gitattributes
.git-blame-ignore-revs
.mailfilter
.mailmap
.spi.yml
.swift-format
.editorconfig
.github/*
*.md
*.txt
*.yml
*.yaml
*.json
Package.swift
**/Package.swift
Package@-*.swift
**/Package@-*.swift
Package.resolved
**/Package.resolved
Makefile
*.modulemap
**/*.modulemap
**/*.docc/*
*.xcprivacy
**/*.xcprivacy
*.symlink
**/*.symlink
Dockerfile
**/Dockerfile
Snippets/*
dev/git.commit.template
.unacceptablelanguageignore
IntegrationTests/*.sh
Sources/SWIM/Utils/Heap.swift
Tests/SWIMTests/HeapTests.swift
================================================
FILE: .spi.yml
================================================
version: 1
builder:
configs:
- documentation_targets: [ClusterMembership, SWIM]
================================================
FILE: .swift-format
================================================
{
"version" : 1,
"indentation" : {
"spaces" : 4
},
"tabWidth" : 4,
"fileScopedDeclarationPrivacy" : {
"accessLevel" : "private"
},
"spacesAroundRangeFormationOperators" : false,
"indentConditionalCompilationBlocks" : false,
"indentSwitchCaseLabels" : false,
"lineBreakAroundMultilineExpressionChainComponents" : false,
"lineBreakBeforeControlFlowKeywords" : false,
"lineBreakBeforeEachArgument" : true,
"lineBreakBeforeEachGenericRequirement" : true,
"lineLength" : 240,
"maximumBlankLines" : 1,
"respectsExistingLineBreaks" : true,
"prioritizeKeepingFunctionOutputTogether" : true,
"noAssignmentInExpressions" : {
"allowedFunctions" : [
"XCTAssertNoThrow",
"XCTAssertThrowsError"
]
},
"rules" : {
"NoBlockComments" : false,
"ReplaceForEachWithForLoop" : false,
"AllPublicDeclarationsHaveDocumentation" : false,
"AlwaysUseLiteralForEmptyCollectionInit" : false,
"AlwaysUseLowerCamelCase" : false,
"AmbiguousTrailingClosureOverload" : true,
"BeginDocumentationCommentWithOneLineSummary" : false,
"DoNotUseSemicolons" : true,
"DontRepeatTypeInStaticProperties" : true,
"FileScopedDeclarationPrivacy" : true,
"FullyIndirectEnum" : true,
"GroupNumericLiterals" : true,
"IdentifiersMustBeASCII" : true,
"NeverForceUnwrap" : false,
"NeverUseForceTry" : false,
"NeverUseImplicitlyUnwrappedOptionals" : false,
"NoAccessLevelOnExtensionDeclaration" : true,
"NoAssignmentInExpressions" : true,
"NoBlockComments" : true,
"NoCasesWithOnlyFallthrough" : true,
"NoEmptyTrailingClosureParentheses" : true,
"NoLabelsInCasePatterns" : true,
"NoLeadingUnderscores" : false,
"NoParensAroundConditions" : true,
"NoVoidReturnOnFunctionSignature" : true,
"OmitExplicitReturns" : true,
"OneCasePerLine" : true,
"OneVariableDeclarationPerLine" : true,
"OnlyOneTrailingClosureArgument" : true,
"OrderedImports" : true,
"ReplaceForEachWithForLoop" : true,
"ReturnVoidInsteadOfEmptyTuple" : true,
"UseEarlyExits" : false,
"UseExplicitNilCheckInConditions" : false,
"UseLetInEveryBoundCaseVariable" : false,
"UseShorthandTypeNames" : true,
"UseSingleLinePropertyGetter" : false,
"UseSynthesizedInitializer" : false,
"UseTripleSlashForDocumentationComments" : true,
"UseWhereClausesInForLoops" : false,
"ValidateDocumentationComments" : false
}
}
================================================
FILE: CODE_OF_CONDUCT.md
================================================
# Code of Conduct
The code of conduct for this project can be found at https://swift.org/code-of-conduct.
================================================
FILE: CONTRIBUTING.md
================================================
## Legal
By submitting a pull request, you represent that you have the right to license
your contribution to Apple and the community, and agree by submitting the patch
that your contributions are licensed under the Apache 2.0 license (see
`LICENSE.txt`).
## How to submit a bug report
Please ensure to specify the following:
* Swift Cluster Membership commit hash
* Contextual information (e.g. what you were trying to achieve with Swift Cluster Membership)
* Simplest possible steps to reproduce
* More complex the steps are, lower the priority will be.
* A pull request with failing test case is preferred, but it's just fine to paste the test case into the issue description.
* Anything that might be relevant in your opinion, such as:
* Swift version or the output of `swift --version`
* OS version and the output of `uname -a`
* Network configuration
### Example
```
Swift Cluster Membership commit hash: 22ec043dc9d24bb011b47ece4f9ee97ee5be2757
Context:
While load testing my HTTP web server written with Swift Cluster Membership, I noticed
that one file descriptor is leaked per request.
Steps to reproduce:
1. ...
2. ...
3. ...
4. ...
$ swift --version
Swift version 4.0.2 (swift-4.0.2-RELEASE)
Target: x86_64-unknown-linux-gnu
Operating system: Ubuntu Linux 16.04 64-bit
$ uname -a
Linux beefy.machine 4.4.0-101-generic #124-Ubuntu SMP Fri Nov 10 18:29:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
My system has IPv6 disabled.
```
## Writing a Patch
A good Swift Cluster Membership patch is:
1. Concise, and contains as few changes as needed to achieve the end result.
2. Tested, ensuring that any tests provided failed before the patch and pass after it.
3. Documented, adding API documentation as needed to cover new functions and properties.
4. Adheres to our code formatting conventions and [style guide](STYLE_GUIDE.md).
5. Accompanied by a great commit message, using our commit message template.
### Code Format and Style
Swift Cluster Membership uses [SwiftFormat](https://github.com/nicklockwood/SwiftFormat) to enforce the preferred [swift code format](.swiftformat). Always run SwiftFormat before committing your code.
### Commit Message Template
We require that your commit messages match our template. The easiest way to do that is to get git to help you by explicitly using the template. To do that, `cd` to the root of our repository and run:
git config commit.template dev/git.commit.template
### Run CI checks locally
You can run the GitHub Actions workflows locally using [act](https://github.com/nektos/act). For detailed steps on how to do this please see [https://github.com/swiftlang/github-workflows?tab=readme-ov-file#running-workflows-locally](https://github.com/swiftlang/github-workflows?tab=readme-ov-file#running-workflows-locally).
## How to contribute your work
Please open a pull request at https://github.com/apple/swift-cluster-membership. Make sure the CI passes, and then wait for code review.
================================================
FILE: CONTRIBUTORS.txt
================================================
For the purpose of tracking copyright, this is the list of individuals and
organizations who have contributed source code to Swift Cluster Membership.
For employees of an organization/company where the copyright of work done
by employees of that company is held by the company itself, only the company
needs to be listed here.
## COPYRIGHT HOLDERS
- Apple Inc. (all contributors with '@apple.com')
### Contributors
- Konrad `ktoso` Malawski <ktoso@apple.com> <konrad_malawski@apple.com>
- Dario Rexin <drexin@apple.com>
- Anton Volokhov <avolokhov@apple.com>
- Tom Doron <tomer@apple.com>
================================================
FILE: HANDBOOK.md
================================================
# Contributors Handbook
## Glossary
In an attempt to form a shared vocabulary in the project, words used in the APIs have been carefully selected and we'd like to keep using them to refer to the same things in various implementations.
- CoolAlgorithm **Instance** - we refer to a specific implementation of a membership algorithm as an "Instance". An instance is:
- SHOULD be a value type; as it makes debugging the state of an instance simpler; i.e. we can record the last 10 states of the algorithm and see how things went wrong, even in running clustered systems if necessary.
- is NOT performing any IO (except for logging which we after long debates, decided that it's better to allow implementations to log if configured to do so)
- it most likely IS a finite state machine, although may not be one in the strict meaning of the pattern. If possible to express as an FSM, we recommend doing so, or carving out pieces of the protocol which can be represented as a state machine.
- an instance SHOULD be easy to test in isolation, such that crazy edge cases in algorithms can be codified in easy to run and understand test cases when necessary
- CoolAlgorithm **Shell** - inspired by shells as we know them from terminals, a shell is what handles the interaction with the environment and the instance; it is the link between the I/O and the pure instance. One can also think of it as an "interpreter."
- **Directive** - directives are how an algorithm instance may choose to interact with a Shell. Upon performing some action on an algorithm's instance it SHOULD return a directive, which will instruct the Shell to "now do this thing, then that thing". It "directs the shell" to do the right thing, following the instances protocol.
- **Peer** - a peer is a known host that has the potential to be a cluster member; we can communicate with a peer by sending messages to it (and it may send messages to us), however a peer does not have an inherent cluster membership status, in order to have a status it must be (wrapped in a) *Member*
- Peers SHOULD be Unique; meaning that if node dies and spawns again using the same host/port pair, we should consider it to be a _new peer_ rather than the same peer. This is usually solved by issuing some random UUID on node startup, and including this ID in any messaging the peer performs.
- **Cluster Member** - a member of a cluster, meaning it is _known to be (or have been) part of the cluster_ and likely has some associated cluster state (e.g. alive or dead etc.)
- It most likely is wrapping a Peer with additional information
## Tips
- When working with directives, never `return []`, always preallocate a directives array and `return directives`.
- there are many situations where it is good to bail out early, but many operations have some form of "needs to always be done"
in their directives. Using this pattern ensures you won't accidentally miss those directives.
- When "should never happen", use `precondition`s with a lot of contextual information (including the entire instance state),
so users can provide you with a good crash report.
## Testing tips
Tests have `LogCapture` installed are able to capture all logs "per node" and later present them in a readable output if a test fails automatically.
If you need to investigate test logs without the test failing, you can enable them like so:
```swift
final class SWIMNIOClusteredTests: RealClusteredXCTestCase {
override var alwaysPrintCaptureLogs: Bool {
true
}
// ...
}
```
================================================
FILE: LICENSE.txt
================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
================================================
FILE: NOTICE.txt
================================================
The Swift Cluster Membership Project
====================================
Please visit the Swift Cluster Membership web site for more information:
* https://github.com/apple/swift-cluster-membership
Copyright 2020 The Swift Cluster Membership Project
The Swift Cluster Membership Project licenses this file to you under the Apache License,
version 2.0 (the "License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at:
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
Also, please refer to each LICENSE.<component>.txt file, which is located in
the 'license' directory of the distribution file, for the license terms of the
components that this product depends on.
---
This product contains some modified data-structures from Apple/Swift-NIO.
* LICENSE (Apache 2.0):
* https://github.com/apple/swift-nio/blob/main/LICENSE.txt
* HOMEPAGE:
* https://github.com/apple/swift-nio
================================================
FILE: Package.swift
================================================
// swift-tools-version:5.10
// The swift-tools-version declares the minimum version of Swift required to build this package.
import PackageDescription
import class Foundation.ProcessInfo
// Workaround: Since we cannot include the flat just as command line options since then it applies to all targets,
// and ONE of our dependencies currently produces one warning, we have to use this workaround to enable it in _our_
// targets when the flag is set. We should remove the dependencies and then enable the flag globally though just by passing it.
let globalSwiftSettings: [SwiftSetting]
if ProcessInfo.processInfo.environment["WARNINGS_AS_ERRORS"] != nil {
print("WARNINGS_AS_ERRORS enabled, passing `-warnings-as-errors`")
globalSwiftSettings = [
SwiftSetting.unsafeFlags(["-warnings-as-errors"])
]
} else {
globalSwiftSettings = []
}
var targets: [PackageDescription.Target] = [
// ==== ------------------------------------------------------------------------------------------------------------
// MARK: SWIM
.target(
name: "ClusterMembership",
dependencies: []
),
.target(
name: "SWIM",
dependencies: [
"ClusterMembership",
.product(name: "Logging", package: "swift-log"),
.product(name: "Metrics", package: "swift-metrics"),
]
),
.target(
name: "SWIMNIOExample",
dependencies: [
"SWIM",
.product(name: "NIO", package: "swift-nio"),
.product(name: "NIOFoundationCompat", package: "swift-nio"),
.product(name: "NIOConcurrencyHelpers", package: "swift-nio"),
.product(name: "NIOExtras", package: "swift-nio-extras"),
.product(name: "Logging", package: "swift-log"),
.product(name: "Metrics", package: "swift-metrics"),
]
),
// ==== ------------------------------------------------------------------------------------------------------------
// MARK: Other Membership Protocols ...
// ==== ------------------------------------------------------------------------------------------------------------
// MARK: Documentation
.testTarget(
name: "ClusterMembershipDocumentationTests",
dependencies: [
"SWIM"
]
),
// ==== ------------------------------------------------------------------------------------------------------------
// MARK: Tests
.testTarget(
name: "ClusterMembershipTests",
dependencies: [
"ClusterMembership"
]
),
.testTarget(
name: "SWIMTests",
dependencies: [
"SWIM",
"SWIMTestKit",
]
),
.testTarget(
name: "SWIMNIOExampleTests",
dependencies: [
"SWIMNIOExample",
"SWIMTestKit",
]
),
// NOT FOR PUBLIC CONSUMPTION.
.testTarget(
name: "SWIMTestKit",
dependencies: [
"SWIM",
.product(name: "NIO", package: "swift-nio"),
.product(name: "Logging", package: "swift-log"),
.product(name: "Metrics", package: "swift-metrics"),
]
),
// ==== ------------------------------------------------------------------------------------------------------------
// MARK: Samples are defined in Samples/Package.swift
// ==== ------------------------------------------------------------------------------------------------------------
]
var dependencies: [Package.Dependency] = [
.package(url: "https://github.com/apple/swift-nio.git", from: "2.19.0"),
.package(url: "https://github.com/apple/swift-nio-ssl.git", from: "2.8.0"),
.package(url: "https://github.com/apple/swift-nio-extras.git", from: "1.5.1"),
// ~~~ SSWG APIs ~~~
.package(url: "https://github.com/apple/swift-log.git", from: "1.4.0"),
.package(url: "https://github.com/apple/swift-metrics.git", "2.3.2"..<"3.0.0"), // since latest
]
let products: [PackageDescription.Product] = [
.library(
name: "ClusterMembership",
targets: ["ClusterMembership"]
),
.library(
name: "SWIM",
targets: ["SWIM"]
),
.library(
name: "SWIMNIOExample",
targets: ["SWIMNIOExample"]
),
]
var package = Package(
name: "swift-cluster-membership",
platforms: [
.macOS(.v13),
.iOS(.v16),
.tvOS(.v16),
.watchOS(.v9),
],
products: products,
dependencies: dependencies,
targets: targets.map { target in
var swiftSettings = target.swiftSettings ?? []
swiftSettings.append(contentsOf: globalSwiftSettings)
if !swiftSettings.isEmpty {
target.swiftSettings = swiftSettings
}
return target
},
cxxLanguageStandard: .cxx11
)
// --- STANDARD CROSS-REPO SETTINGS DO NOT EDIT --- //
for target in package.targets {
switch target.type {
case .regular, .test, .executable:
var settings = target.swiftSettings ?? []
// https://github.com/swiftlang/swift-evolution/blob/main/proposals/0444-member-import-visibility.md
settings.append(.enableUpcomingFeature("MemberImportVisibility"))
target.swiftSettings = settings
case .macro, .plugin, .system, .binary:
() // not applicable
@unknown default:
() // we don't know what to do here, do nothing
}
}
// --- END: STANDARD CROSS-REPO SETTINGS DO NOT EDIT --- //
================================================
FILE: README.md
================================================
# Swift Cluster Membership
This library aims to help Swift make ground in a new space: clustered multi-node distributed systems.
With this library we provide reusable runtime agnostic membership protocol implementations which can be adopted in various clustering use-cases.
## Background
Cluster membership protocols are a crucial building block for distributed systems, such as computation intensive clusters, schedulers, databases, key-value stores and more. With the announcement of this package, we aim to make building such systems simpler, as they no longer need to rely on external services to handle service membership for them. We would also like to invite the community to collaborate on and develop additional membership protocols.
At their core, membership protocols need to provide an answer for the question "Who are my (live) peers?". This seemingly simple task turns out to be not so simple at all in a distributed system where delayed or lost messages, network partitions, and unresponsive but still "alive" nodes are the daily bread and butter. Providing a predictable, reliable answer to this question is what cluster membership protocols do.
There are various trade-offs one can take while implementing a membership protocol, and it continues to be an interesting area of research and continued refinement. As such, the cluster-membership package intends to focus not on a single implementation, but serve as a collaboration space for various distributed algorithms in this space.
## 🏊🏾♀️🏊🏻♀️🏊🏾♂️🏊🏼♂️ SWIMming with Swift
### High-level Protocol Description
> For a more in-depth discussion of the protocol and modifications in this implementation we suggest reading the [SWIM API Documentation](https://apple.github.io/swift-cluster-membership/docs/current/SWIM/Enums/SWIM.html), as well as the associated papers linked below.
The [*Scalable Weakly-consistent Infection-style process group Membership*](https://research.cs.cornell.edu/projects/Quicksilver/public_pdfs/SWIM.pdf) algorithm (also known as "SWIM"), along with a few notable protocol extensions as documented in the 2018 [*Lifeguard: Local Health Awareness for More Accurate Failure Detection*](https://arxiv.org/abs/1707.00788) paper.
SWIM is a [gossip protocol](https://en.wikipedia.org/wiki/Gossip_protocol) in which peers periodically exchange bits of information about their observations of other nodes’ statuses, eventually spreading the information to all other members in a cluster. This category of distributed algorithms are very resilient against arbitrary message loss, network partitions and similar issues.
At a high level, SWIM works like this:
* A member periodically pings a "randomly" selected peer it is aware of. It does so by sending a .ping message to that peer, expecting an [`.ack`](https://apple.github.io/swift-cluster-membership/docs/current/SWIM/Protocols/SWIMPingOriginPeer.html#/s:4SWIM18SWIMPingOriginPeerP3ack13acknowledging6target11incarnation7payloadys6UInt32V_AA8SWIMPeer_ps6UInt64VA2AO13GossipPayloadOtF) to be sent back. See how `A` probes `B` initially in the diagram below.
* The exchanged messages also carry a gossip `payload`, which is (partial) information about what other peers the sender of the message is aware of, along with their membership status (`.alive`, `.suspect`, etc.)
* If it receives an `.ack`, the peer is considered still `.alive`. Otherwise, the target peer might have terminated/crashed or is unresponsive for other reasons.
* In order to double check if the peer really is dead, the origin asks a few other peers about the state of the unresponsive peer by sending `.pingRequest` messages to a configured number of other peers, which then issue direct pings to that peer (probing peer E in the diagram below).
* If those pings fail, due to lack of .acks resulting in the peer being marked as `.suspect`,
* Our protocol implementation will also use additional `.nack` ("negative acknowledgement") messages in the situation to inform the ping request origin that the intermediary did receive those `.pingRequest` messages, however the target seems to not have responded. We use this information to adjust a Local Health Multiplier, which affects how timeouts are calculated. To learn more about this refer to the API docs and the Lifeguard paper.
The above mechanism, serves not only as a failure detection mechanism, but also as a gossip mechanism, which carries information about known members of the cluster. This way members eventually learn about the status of their peers, even without having them all listed upfront. It is worth pointing out however that this membership view is [weakly-consistent](https://en.wikipedia.org/wiki/Weak_consistency), which means there is no guarantee (or way to know, without additional information) if all members have the same exact view on the membership at any given point in time. However, it is an excellent building block for higher-level tools and systems to build their stronger guarantees on top.
Once the failure detection mechanism detects an unresponsive node, it eventually is marked as .dead resulting in its irrevocable removal from the cluster. Our implementation offers an optional extension, adding an .unreachable state to the possible states, however most users will not find it necessary and it is disabled by default. For details and rules rules about legal status transitions refer to [SWIM.Status](https://github.com/apple/swift-cluster-membership/blob/main/Sources/SWIM/Status.swift#L18-L39) or the following diagram:
The way Swift Cluster Membership implements protocols, is by offering "`Instances`" of them. For example, the SWIM implementation is encapsulated in the runtime agnostic [`SWIM.Instance`](https://github.com/apple/swift-cluster-membership/blob/main/Sources/SWIM/SWIMInstance.swift) which needs to be “driven” or “interpreted” by some glue code between a networking runtime and the instance itself. We call those glue pieces of an implementation "`Shell`s", and the library ships with a `SWIMNIOShell` implemented using [SwiftNIO](https://www.github.com/apple/swift-nio)’s `DatagramChannel` that performs all messaging asynchronously over [UDP](https://searchnetworking.techtarget.com/definition/UDP-User-Datagram-Protocol). Alternative implementations can use completely different transports, or piggy back SWIM messages on some other existing gossip system etc.
The SWIM instance also has built-in support for emitting metrics (using [swift-metrics](https://github.com/apple/swift-metrics)) and can be configured to log details about internal details by passing a [swift-log](https://github.com/apple/swift-log) `Logger`.
### Example: Reusing the SWIM protocol logic implementation
The primary purpose of this library is to share the `SWIM.Instance` implementation across various implementations which need some form of in-process membership service. Implementing a custom runtime is documented in depth in the project’s README (https://github.com/apple/swift-cluster-membership/), so please have a look there if you are interested in implementing SWIM over some different transport.
Implementing a new transport boils down a “fill in the blanks” exercise:
First, one has to implement the Peer protocols (https://github.com/apple/swift-cluster-membership/blob/main/Sources/SWIM/Peer.swift) using one’s target transport:
```swift
/// SWIM peer which can be initiated contact with, by sending ping or ping request messages.
public protocol SWIMPeer: SWIMAddressablePeer {
/// Perform a probe of this peer by sending a `ping` message.
///
/// <... more docs here - please refer to the API docs for the latest version ...>
func ping(
payload: SWIM.GossipPayload,
from origin: SWIMPingOriginPeer,
timeout: DispatchTimeInterval,
sequenceNumber: SWIM.SequenceNumber
) async throws -> SWIM.PingResponse
// ...
}
```
Which usually means wrapping some connection, channel, or other identity with the ability to send messages and invoke the appropriate callbacks when applicable.
Then, on the receiving end of a peer, one has to implement receiving those messages and invoke all the corresponding `on<SomeMessage>(...)` callbacks defined on the `SWIM.Instance` (grouped under [SWIMProtocol](https://github.com/apple/swift-cluster-membership/blob/main/Sources/SWIM/SWIMInstance.swift#L24-L85)).
A piece of the SWIMProtocol is listed below to give you an idea about it:
```swift
public protocol SWIMProtocol {
/// MUST be invoked periodically, in intervals of `self.swim.dynamicLHMProtocolInterval`.
///
/// MUST NOT be scheduled using a "repeated" task/timer", as the interval is dynamic and may change as the algorithm proceeds.
/// Implementations should schedule each next tick by handling the returned directive's `scheduleNextTick` case,
/// which includes the appropriate delay to use for the next protocol tick.
///
/// This is the heart of the protocol, as each tick corresponds to a "protocol period" in which:
/// - suspect members are checked if they're overdue and should become `.unreachable` or `.dead`,
/// - decisions are made to `.ping` a random peer for fault detection,
/// - and some internal house keeping is performed.
///
/// Note: This means that effectively all decisions are made in interval sof protocol periods.
/// It would be possible to have a secondary periodic or more ad-hoc interval to speed up
/// some operations, however this is currently not implemented and the protocol follows the fairly
/// standard mode of simply carrying payloads in periodic ping messages.
///
/// - Returns: `SWIM.Instance.PeriodicPingTickDirective` which must be interpreted by a shell implementation
mutating func onPeriodicPingTick() -> [SWIM.Instance.PeriodicPingTickDirective]
mutating func onPing( ... ) -> [SWIM.Instance.PingDirective]
mutating func onPingRequest( ... ) -> [SWIM.Instance.PingRequestDirective]
mutating func onPingResponse( ... ) -> [SWIM.Instance.PingResponseDirective]
// ...
}
```
These calls perform all SWIM protocol specific tasks internally, and return directives which are simple to interpret “commands” to an implementation about how it should react to the message. For example, upon receiving a `.pingRequest` message, the returned directive may instruct a shell to send a ping to some nodes. The directive prepares all apropriate target, timeout and additional information that makes it simpler to simply follow its instruction and implement the call correctly, e.g. like this:
```swift
self.swim.onPingRequest(
target: target,
pingRequestOrigin: pingRequestOrigin,
payload: payload,
sequenceNumber: sequenceNumber
).forEach { directive in
switch directive {
case .gossipProcessed(let gossipDirective):
self.handleGossipPayloadProcessedDirective(gossipDirective)
case .sendPing(let target, let payload, let pingRequestOriginPeer, let pingRequestSequenceNumber, let timeout, let sequenceNumber):
self.sendPing(
to: target,
payload: payload,
pingRequestOrigin: pingRequestOriginPeer,
pingRequestSequenceNumber: pingRequestSequenceNumber,
timeout: timeout,
sequenceNumber: sequenceNumber
)
}
}
```
In general this allows for all the tricky "what to do when" to be encapsulated within the protocol instance, and a Shell only has to follow instructions implementing them. The actual implementations will often need to perform some more involved concurrency and networking tasks, like awaiting for a sequence of responses, and handling them in a specific way etc, however the general outline of the protocol is orchestrated by the instance's directives.
For detailed documentation about each of the callbacks, when to invoke them, and how all this fits together, please refer to the [**API Documentation**](https://apple.github.io/swift-cluster-membership/docs/current/SWIM/index.html).
### Example: SWIMming with Swift NIO
The repository contains an [end-to-end example](Samples/Sources/SWIMNIOSampleCluster) and an example implementation called [SWIMNIOExample](Sources/SWIMNIOExample) which makes use of the `SWIM.Instance` to enable a simple UDP based peer monitoring system. This allows peers to gossip and notify each other about node failures using the SWIM protocol by sending datagrams driven by SwiftNIO.
> 📘 The `SWIMNIOExample` implementation is offered only as an example, and has not been implemented with production use in mind, however with some amount of effort it could definitely do well for some use-cases. If you are interested in learning more about cluster membership algorithms, scalability benchmarking and using SwiftNIO itself, this is a great module to get your feet wet, and perhaps once the module is mature enough we could consider making it not only an example, but a reusable component for Swift NIO based clustered applications.
In it’s simplest form, combining the provided SWIM instance and NIO shell to build a simple server, one can embedd the provided handlers like shown below, in a typical NIO channel pipeline:
```swift
let bootstrap = DatagramBootstrap(group: group)
.channelOption(ChannelOptions.socketOption(.so_reuseaddr), value: 1)
.channelInitializer { channel in
channel.pipeline
// first install the SWIM handler, which contains the SWIMNIOShell:
.addHandler(SWIMNIOHandler(settings: settings)).flatMap {
// then install some user handler, it will receive SWIM events:
channel.pipeline.addHandler(SWIMNIOExampleHandler())
}
}
bootstrap.bind(host: host, port: port)
```
The example handler can then receive and handle SWIM cluster membership change events:
```swift
final class SWIMNIOExampleHandler: ChannelInboundHandler {
public typealias InboundIn = SWIM.MemberStatusChangedEvent
let log = Logger(label: "SWIMNIOExampleHandler")
public func channelRead(context: ChannelHandlerContext, data: NIOAny) {
let change: SWIM.MemberStatusChangedEvent = self.unwrapInboundIn(data)
self.log.info("Membership status changed: [\(change.member.node)] is now [\(change.status)]", metadata: [
"swim/member": "\(change.member.node)",
"swim/member/status": "\(change.status)",
])
}
}
```
If you are interested in contributing and polishing up the SWIMNIO implementation please head over to the issues and pick up a task or propose an improvement yourself!
## Additional Membership Protocol Implementations
We are generally interested in fostering discussions and implementations of additional membership implementations using a similar "Instance" style.
If you are interested in such algorithms, and have a favourite protocol that you'd like to see implemented, please do not hesitate to reach out here via issues or the [Swift forums](https://forums.swift.org/c/server).
## Contributing
Experience reports, feedback, improvement ideas and contributions are greatly encouraged!
We look forward to hear from you.
Please refer to [CONTRIBUTING](CONTRIBUTING.md) guide to learn about the process of submitting pull requests,
and refer to the [HANDBOOK](HANDBOOK.md) for terminology and other useful tips for working with this library.
================================================
FILE: Samples/.gitignore
================================================
# The XPC sample generates an .app, so we want to ignore it
*.app
================================================
FILE: Samples/Package.swift
================================================
// swift-tools-version:5.10
// The swift-tools-version declares the minimum version of Swift required to build this package.
import PackageDescription
var targets: [PackageDescription.Target] = [
.target(
name: "SWIMNIOSampleCluster",
dependencies: [
.product(name: "SWIM", package: "swift-cluster-membership"),
.product(name: "SWIMNIOExample", package: "swift-cluster-membership"),
.product(name: "SwiftPrometheus", package: "SwiftPrometheus"),
.product(name: "Lifecycle", package: "swift-service-lifecycle"),
.product(name: "ArgumentParser", package: "swift-argument-parser"),
],
path: "Sources/SWIMNIOSampleCluster"
),
/* --- tests --- */
// no-tests placeholder project to not have `swift test` fail on Samples/
.testTarget(
name: "NoopTests",
dependencies: [
.product(name: "SWIM", package: "swift-cluster-membership")
],
path: "Tests/NoopTests"
),
]
var dependencies: [Package.Dependency] = [
// ~~~~~~~ parent ~~~~~~~
.package(path: "../"),
// ~~~~~~~ only for samples ~~~~~~~
.package(url: "https://github.com/swift-server/swift-service-lifecycle.git", from: "1.0.0-alpha"),
.package(url: "https://github.com/MrLotU/SwiftPrometheus.git", from: "1.0.0-alpha"),
.package(url: "https://github.com/apple/swift-argument-parser", from: "0.2.0"),
]
let package = Package(
name: "swift-cluster-membership-samples",
platforms: [
.macOS(.v13)
],
products: [
.executable(
name: "SWIMNIOSampleCluster",
targets: ["SWIMNIOSampleCluster"]
)
],
dependencies: dependencies,
targets: targets,
cxxLanguageStandard: .cxx11
)
================================================
FILE: Samples/README.md
================================================
## Sample applications
Use `swift run` to run the samples.
### SWIMNIOSampleCluster
This sample app runs a _single node_ per process, however it is prepared to be easily clustered up.
This mode of operation is useful to manually suspend or stop processes and see those issues be picked up by the SWIM implementation.
Recommended way to run:
```bash
# cd swift-cluster-membership
> swift run --package-path Samples SWIMNIOSampleCluster --help
```
which uses [swift argument parser](https://github.com/apple/swift-argument-parser) list all the options the sample app has available.
An example invocation would be:
```bash
swift run --package-path Samples SWIMNIOSampleCluster --port 7001
```
which spawns a node on `127.0.0.1:7001`, to spawn another node to join it and form a two node cluster you can:
```bash
swift run --package-path Samples SWIMNIOSampleCluster --port 7002 --initial-contact-points 127.0.0.1:7001,127.0.0.1:7003
# you can list multiple peers as contact points like this:
swift run --package-path Samples SWIMNIOSampleCluster --port 7003 --initial-contact-points 127.0.0.1:7001,127.0.0.1:7002
```
Once the cluster is formed, you'll see messages logged by the `SWIMNIOSampleHandler` showing when nodes become alive or dead.
You can enable debug or trace level logging to inspect more of the details of what is going on internally in the nodes.
To see the failure detection in action, you can stop processes, or "suspend" them for a little while by doing
```bash
# swift run --package-path Samples SWIMNIOSampleCluster --port 7001
<ctrl +z>
^Z
[1] + 35002 suspended swift run --package-path Samples SWIMNIOSampleCluster --port 7001
$ fg %1
```
to resume the node; This can be useful to poke around and manually get a feel about how the failure detection works.
You can also hook the systems up to a metrics dashboard to see the information propagate in real time (once it is instrumented using swift-metrics).
================================================
FILE: Samples/Sources/SWIMNIOSampleCluster/SWIMNIOSampleNode.swift
================================================
//===----------------------------------------------------------------------===//
//
// This source file is part of the Swift Cluster Membership open source project
//
// Copyright (c) 2020 Apple Inc. and the Swift Cluster Membership project authors
// Licensed under Apache License v2.0
//
// See LICENSE.txt for license information
// See CONTRIBUTORS.txt for the list of Swift Cluster Membership project authors
//
// SPDX-License-Identifier: Apache-2.0
//
//===----------------------------------------------------------------------===//
import ClusterMembership
import Logging
import NIO
import SWIM
import SWIMNIOExample
struct SampleSWIMNIONode {
let port: Int
var settings: SWIMNIO.Settings
let group: EventLoopGroup
init(port: Int, settings: SWIMNIO.Settings, group: EventLoopGroup) {
self.port = port
self.settings = settings
self.group = group
}
func start() {
let bootstrap = DatagramBootstrap(group: group)
.channelOption(ChannelOptions.socketOption(.so_reuseaddr), value: 1)
.channelInitializer { channel in
channel.pipeline
.addHandler(SWIMNIOHandler(settings: self.settings)).flatMap {
channel.pipeline.addHandler(SWIMNIOSampleHandler())
}
}
bootstrap.bind(host: "127.0.0.1", port: port).whenComplete { result in
switch result {
case .success(let res):
self.settings.logger.info("Bound to: \(res)")
()
case .failure(let error):
self.settings.logger.error("Error: \(error)")
()
}
}
}
}
final class SWIMNIOSampleHandler: ChannelInboundHandler {
typealias InboundIn = SWIM.MemberStatusChangedEvent<SWIM.NIOPeer>
let log = Logger(label: "SWIMNIOSample")
public func channelRead(context: ChannelHandlerContext, data: NIOAny) {
let change: SWIM.MemberStatusChangedEvent = self.unwrapInboundIn(data)
// we log each event (in a pretty way)
self.log.info(
"Membership status changed: [\(change.member.node)] is now [\(change.status)]",
metadata: [
"swim/member": "\(change.member.node)",
"swim/member/previousStatus": "\(change.previousStatus.map({"\($0)"}) ?? "unknown")",
"swim/member/status": "\(change.status)",
]
)
}
}
================================================
FILE: Samples/Sources/SWIMNIOSampleCluster/main.swift
================================================
//===----------------------------------------------------------------------===//
//
// This source file is part of the Swift Cluster Membership open source project
//
// Copyright (c) 2020 Apple Inc. and the Swift Cluster Membership project authors
// Licensed under Apache License v2.0
//
// See LICENSE.txt for license information
// See CONTRIBUTORS.txt for the list of Swift Cluster Membership project authors
//
// SPDX-License-Identifier: Apache-2.0
//
//===----------------------------------------------------------------------===//
import ArgumentParser
import ClusterMembership
import Lifecycle
import Logging
import Metrics
import NIO
import Prometheus
import SWIM
import SWIMNIOExample
struct SWIMNIOSampleCluster: ParsableCommand {
@Option(name: .shortAndLong, help: "The number of nodes to start, defaults to: 1")
var count: Int?
@Argument(help: "Hostname that node(s) should bind to")
var host: String?
@Option(help: "Determines which this node should bind to; Only effective when running a single node")
var port: Int?
@Option(help: "Configures which nodes should be passed in as initial contact points, format: host:port,")
var initialContactPoints: String = ""
@Option(help: "Configures log level")
var logLevel: String = "info"
mutating func run() throws {
LoggingSystem.bootstrap(_SWIMPrettyMetadataLogHandler.init)
let group = MultiThreadedEventLoopGroup(numberOfThreads: System.coreCount)
// Uncomment this if you'd like to see metrics displayed in the command line periodically;
// This bootstraps and uses the Prometheus metrics backend to report metrics periodically by printing them to the stdout (console).
//
// Note though that this will be a bit noisy, since logs are also emitted to the stdout by default, however it's a nice way
// to learn and explore what the metrics are and how they behave when toying around with a local cluster.
// let prom = PrometheusClient()
// MetricsSystem.bootstrap(prom)
//
// group.next().scheduleRepeatedTask(initialDelay: .seconds(1), delay: .seconds(10)) { _ in
// prom.collect { (string: String) in
// print("")
// print("")
// print(string)
// }
// }
let lifecycle = ServiceLifecycle()
lifecycle.registerShutdown(
label: "eventLoopGroup",
.sync(group.syncShutdownGracefully)
)
var settings = SWIMNIO.Settings()
if count == nil || count == 1 {
let nodePort = self.port ?? 7001
settings.logger = Logger(label: "swim-\(nodePort)")
settings.logger.logLevel = self.parseLogLevel()
settings.swim.logger.logLevel = self.parseLogLevel()
settings.swim.initialContactPoints = self.parseContactPoints()
let node = SampleSWIMNIONode(port: nodePort, settings: settings, group: group)
lifecycle.register(
label: "swim-\(nodePort)",
start: .sync { node.start() },
shutdown: .sync {}
)
} else {
let basePort = port ?? 7001
for i in 1...(count ?? 1) {
let nodePort = basePort + i
settings.logger = Logger(label: "swim-\(nodePort)")
settings.swim.initialContactPoints = self.parseContactPoints()
let node = SampleSWIMNIONode(
port: nodePort,
settings: settings,
group: group
)
lifecycle.register(
label: "swim\(nodePort)",
start: .sync { node.start() },
shutdown: .sync {}
)
}
}
try lifecycle.startAndWait()
}
private func parseLogLevel() -> Logger.Level {
guard let level = Logger.Level.init(rawValue: self.logLevel) else {
fatalError("Unknown log level: \(self.logLevel)")
}
return level
}
private func parseContactPoints() -> Set<ClusterMembership.Node> {
guard self.initialContactPoints.trimmingCharacters(in: .whitespacesAndNewlines) != "" else {
return []
}
let contactPoints: [Node] = self.initialContactPoints.split(separator: ",").map { hostPort in
let host = String(hostPort.split(separator: ":")[0])
let port = Int(String(hostPort.split(separator: ":")[1]))!
return Node(protocol: "udp", host: host, port: port, uid: nil)
}
return Set(contactPoints)
}
}
SWIMNIOSampleCluster.main()
================================================
FILE: Samples/Tests/NoopTests/SampleTest.swift
================================================
//===----------------------------------------------------------------------===//
//
// This source file is part of the Swift Cluster Membership open source project
//
// Copyright (c) 2018-2019 Apple Inc. and the Swift Cluster Membership project authors
// Licensed under Apache License v2.0
//
// See LICENSE.txt for license information
// See CONTRIBUTORS.txt for the list of Swift Cluster Membership project authors
//
// SPDX-License-Identifier: Apache-2.0
//
//===----------------------------------------------------------------------===//
import SWIM
import XCTest
final class SampleTest: XCTestCase {
func test_empty() {
// nothing here (so far...)
}
}
================================================
FILE: Sources/ClusterMembership/Node.swift
================================================
//===----------------------------------------------------------------------===//
//
// This source file is part of the Swift Cluster Membership open source project
//
// Copyright (c) 2020 Apple Inc. and the Swift Cluster Membership project authors
// Licensed under Apache License v2.0
//
// See LICENSE.txt for license information
// See CONTRIBUTORS.txt for the list of Swift Cluster Membership project authors
//
// SPDX-License-Identifier: Apache-2.0
//
//===----------------------------------------------------------------------===//
/// Generic representation of a (potentially unique, if `uid` is present) node in a cluster.
///
/// Generally the node represents "some node we want to contact" if the `uid` is not set,
/// and if the `uid` is available "the specific instance of a node".
public struct Node: Hashable, Sendable, Comparable, CustomStringConvertible {
/// Protocol that can be used to contact this node;
/// Does not have to be a formal protocol name and may be "swim" or a name which is understood by a membership implementation.
public var `protocol`: String
public var name: String?
public var host: String
public var port: Int
public internal(set) var uid: UInt64?
public init(protocol: String, host: String, port: Int, uid: UInt64?) {
self.protocol = `protocol`
self.name = nil
self.host = host
self.port = port
self.uid = uid
}
public init(protocol: String, name: String?, host: String, port: Int, uid: UInt64?) {
self.protocol = `protocol`
if let name = name, name.isEmpty {
self.name = nil
} else {
self.name = name
}
self.host = host
self.port = port
self.uid = uid
}
public var withoutUID: Self {
var without = self
without.uid = nil
return without
}
public var description: String {
// /// uid is not printed by default since we only care about it when we do, not in every place where we log a node
// "\(self.protocol)://\(self.host):\(self.port)"
self.detailedDescription
}
/// Prints a node's String representation including its `uid`.
public var detailedDescription: String {
"\(self.protocol)://\(self.name.map { "\($0)@" } ?? "")\(self.host):\(self.port)\(self.uid.map { "#\($0.description)" } ?? "")"
}
}
extension Node {
// Silly but good enough comparison for deciding "who is lower node"
// as we only use those for "tie-breakers" any ordering is fine to be honest here.
public static func < (lhs: Node, rhs: Node) -> Bool {
if lhs.protocol == rhs.protocol, lhs.host == rhs.host {
if lhs.port == rhs.port {
return (lhs.uid ?? 0) < (rhs.uid ?? 0)
} else {
return lhs.port < rhs.port
}
} else {
// "silly" but good enough comparison, we just need a predictable order, does not really matter what it is
return "\(lhs.protocol)\(lhs.host)" < "\(rhs.protocol)\(rhs.host)"
}
}
}
================================================
FILE: Sources/SWIM/Docs.docc/index.md
================================================
# ``SWIM``
This library aims to help Swift make ground in a new space: clustered multi-node distributed systems.
## Overview
With this library we provide reusable runtime agnostic membership protocol implementations which can be adopted in various clustering use-cases.
### Background
Cluster membership protocols are a crucial building block for distributed systems, such as computation intensive clusters, schedulers, databases, key-value stores and more. With the announcement of this package, we aim to make building such systems simpler, as they no longer need to rely on external services to handle service membership for them. We would also like to invite the community to collaborate on and develop additional membership protocols.
At their core, membership protocols need to provide an answer for the question "Who are my (live) peers?". This seemingly simple task turns out to be not so simple at all in a distributed system where delayed or lost messages, network partitions, and unresponsive but still "alive" nodes are the daily bread and butter. Providing a predictable, reliable answer to this question is what cluster membership protocols do.
There are various trade-offs one can take while implementing a membership protocol, and it continues to be an interesting area of research and continued refinement. As such, the cluster-membership package intends to focus not on a single implementation, but serve as a collaboration space for various distributed algorithms in this space.
### 🏊🏾♀️🏊🏻♀️🏊🏾♂️🏊🏼♂️ SWIMming with Swift
#### High-level Protocol Description
> For a more in-depth discussion of the protocol and modifications in this implementation we suggest reading the [SWIM API Documentation](https://apple.github.io/swift-cluster-membership/docs/current/SWIM/Enums/SWIM.html), as well as the associated papers linked below.
The [*Scalable Weakly-consistent Infection-style process group Membership*](https://research.cs.cornell.edu/projects/Quicksilver/public_pdfs/SWIM.pdf) algorithm (also known as "SWIM"), along with a few notable protocol extensions as documented in the 2018 [*Lifeguard: Local Health Awareness for More Accurate Failure Detection*](https://arxiv.org/abs/1707.00788) paper.
SWIM is a [gossip protocol](https://en.wikipedia.org/wiki/Gossip_protocol) in which peers periodically exchange bits of information about their observations of other nodes’ statuses, eventually spreading the information to all other members in a cluster. This category of distributed algorithms are very resilient against arbitrary message loss, network partitions and similar issues.
At a high level, SWIM works like this:
* A member periodically pings a "randomly" selected peer it is aware of. It does so by sending a `.ping` message to that peer, expecting an [`.ack`](https://apple.github.io/swift-cluster-membership/docs/current/SWIM/Protocols/SWIMPingOriginPeer.html#/s:4SWIM18SWIMPingOriginPeerP3ack13acknowledging6target11incarnation7payloadys6UInt32V_AA8SWIMPeer_ps6UInt64VA2AO13GossipPayloadOtF) to be sent back. See how `A` probes `B` initially in the diagram below.
* The exchanged messages also carry a gossip `payload`, which is (partial) information about what other peers the sender of the message is aware of, along with their membership status (`.alive`, `.suspect`, etc.)
* If it receives an `.ack`, the peer is considered still `.alive`. Otherwise, the target peer might have terminated/crashed or is unresponsive for other reasons.
* In order to double-check if the peer really is dead, the origin asks a few other peers about the state of the unresponsive peer by sending `.pingRequest` messages to a configured number of other peers, which then issue direct pings to that peer (probing peer E in the diagram below).
* If those pings fail, due to lack of `.ack`s resulting in the peer being marked as `.suspect`,
* Our protocol implementation will also use additional `.nack` ("negative acknowledgement") messages in the situation to inform the ping request origin that the intermediary did receive those `.pingRequest` messages, however the target seems to not have responded. We use this information to adjust a Local Health Multiplier, which affects how timeouts are calculated. To learn more about this refer to the API docs and the Lifeguard paper.
The above mechanism serves not only as a failure detection mechanism, but also as a gossip mechanism, which carries information about known members of the cluster. This way members eventually learn about the status of their peers, even without having them all listed upfront. It is worth pointing out however that this membership view is [weakly-consistent](https://en.wikipedia.org/wiki/Weak_consistency), which means there is no guarantee (or way to know, without additional information) if all members have the same exact view on the membership at any given point in time. However, it is an excellent building block for higher-level tools and systems to build their stronger guarantees on top.
Once the failure detection mechanism detects an unresponsive node, it eventually is marked as `.dead` resulting in its irrevocable removal from the cluster. Our implementation offers an optional extension, adding an `.unreachable` state to the possible states, however most users will not find it necessary and it is disabled by default. For details and rules about legal status transitions refer to [SWIM.Status](https://github.com/apple/swift-cluster-membership/blob/main/Sources/SWIM/Status.swift#L18-L39) or the following diagram:
The way Swift Cluster Membership implements protocols is by offering "`Instances`" of them. For example, the SWIM implementation is encapsulated in the runtime agnostic [`SWIM.Instance`](https://github.com/apple/swift-cluster-membership/blob/main/Sources/SWIM/SWIMInstance.swift) which needs to be “driven” or “interpreted” by some glue code between a networking runtime and the instance itself. We call those glue pieces of an implementation "`Shell`s", and the library ships with a `SWIMNIOShell` implemented using [SwiftNIO](https://www.github.com/apple/swift-nio)’s `DatagramChannel` that performs all messaging asynchronously over [UDP](https://searchnetworking.techtarget.com/definition/UDP-User-Datagram-Protocol). Alternative implementations can use completely different transports, or piggy back SWIM messages on some other existing gossip system etc.
The SWIM instance also has built-in support for emitting metrics (using [swift-metrics](https://github.com/apple/swift-metrics)) and can be configured to log details about internal details by passing a [swift-log](https://github.com/apple/swift-log) `Logger`.
#### Example: Reusing the SWIM protocol logic implementation
The primary purpose of this library is to share the `SWIM.Instance` implementation across various implementations which need some form of in-process membership service. Implementing a custom runtime is documented in depth in the project’s README (https://github.com/apple/swift-cluster-membership/), so please have a look there if you are interested in implementing SWIM over some different transport.
Implementing a new transport boils down a “fill in the blanks” exercise:
First, one has to implement the Peer protocols (https://github.com/apple/swift-cluster-membership/blob/main/Sources/SWIM/Peer.swift) using one’s target transport:
```swift
/// SWIM peer which can be initiated contact with, by sending ping or ping request messages.
public protocol SWIMPeer: SWIMAddressablePeer {
/// Perform a probe of this peer by sending a `ping` message.
///
/// <... more docs here - please refer to the API docs for the latest version ...>
func ping(
payload: SWIM.GossipPayload,
from origin: SWIMPingOriginPeer,
timeout: DispatchTimeInterval,
sequenceNumber: SWIM.SequenceNumber
) async throws -> SWIM.PingResponse
// ...
}
```
Which usually means wrapping some connection, channel, or other identity with the ability to send messages and invoke the appropriate callbacks when applicable.
Then, on the receiving end of a peer, one has to implement receiving those messages and invoke all the corresponding
`on<SomeMessage>(...)` callbacks defined on the ``SWIM/Instance`` (grouped under ``SWIMProtocol``).
A piece of the ``SWIMProtocol`` is listed below to give you an idea about it:
```swift
public protocol SWIMProtocol {
/// MUST be invoked periodically, in intervals of `self.swim.dynamicLHMProtocolInterval`.
///
/// MUST NOT be scheduled using a "repeated" task/timer", as the interval is dynamic and may change as the algorithm proceeds.
/// Implementations should schedule each next tick by handling the returned directive's `scheduleNextTick` case,
/// which includes the appropriate delay to use for the next protocol tick.
///
/// This is the heart of the protocol, as each tick corresponds to a "protocol period" in which:
/// - suspect members are checked if they're overdue and should become `.unreachable` or `.dead`,
/// - decisions are made to `.ping` a random peer for fault detection,
/// - and some internal house keeping is performed.
///
/// Note: This means that effectively all decisions are made in interval sof protocol periods.
/// It would be possible to have a secondary periodic or more ad-hoc interval to speed up
/// some operations, however this is currently not implemented and the protocol follows the fairly
/// standard mode of simply carrying payloads in periodic ping messages.
///
/// - Returns: `SWIM.Instance.PeriodicPingTickDirective` which must be interpreted by a shell implementation
mutating func onPeriodicPingTick() -> [SWIM.Instance.PeriodicPingTickDirective]
mutating func onPing( ... ) -> [SWIM.Instance.PingDirective]
mutating func onPingRequest( ... ) -> [SWIM.Instance.PingRequestDirective]
mutating func onPingResponse( ... ) -> [SWIM.Instance.PingResponseDirective]
// ...
}
```
These calls perform all SWIM protocol specific tasks internally, and return directives which are simple to interpret “commands” to an implementation about how it should react to the message. For example, upon receiving a `.pingRequest` message, the returned directive may instruct a shell to send a ping to some nodes. The directive prepares all apropriate target, timeout and additional information that makes it simpler to simply follow its instruction and implement the call correctly, e.g. like this:
```swift
self.swim.onPingRequest(
target: target,
pingRequestOrigin: pingRequestOrigin,
payload: payload,
sequenceNumber: sequenceNumber
).forEach { directive in
switch directive {
case .gossipProcessed(let gossipDirective):
self.handleGossipPayloadProcessedDirective(gossipDirective)
case .sendPing(let target, let payload, let pingRequestOriginPeer, let pingRequestSequenceNumber, let timeout, let sequenceNumber):
self.sendPing(
to: target,
payload: payload,
pingRequestOrigin: pingRequestOriginPeer,
pingRequestSequenceNumber: pingRequestSequenceNumber,
timeout: timeout,
sequenceNumber: sequenceNumber
)
}
}
```
In general this allows for all the tricky "what to do when" to be encapsulated within the protocol instance, and a Shell only has to follow instructions implementing them. The actual implementations will often need to perform some more involved concurrency and networking tasks, like awaiting a sequence of responses, and handling them in a specific way etc, however the general outline of the protocol is orchestrated by the instance's directives.
## Topics
### SWIM logic implementation
- ``SWIM/Instance``
- ``SWIM/Member``
### SWIM settings
- ``SWIMGossipSettings``
- ``SWIMLifeguardSettings``
- ``SWIMMetricsSettings``
### Protocols peer implementations must conform to
- ``SWIMPeer``
- ``SWIMAddressablePeer``
- ``SWIMPingOriginPeer``
- ``SWIMPingRequestOriginPeer``
### Namespace
- ``SWIM/SWIM``
================================================
FILE: Sources/SWIM/Events.swift
================================================
//===----------------------------------------------------------------------===//
//
// This source file is part of the Swift Cluster Membership open source project
//
// Copyright (c) 2020 Apple Inc. and the Swift Cluster Membership project authors
// Licensed under Apache License v2.0
//
// See LICENSE.txt for license information
// See CONTRIBUTORS.txt for the list of Swift Cluster Membership project authors
//
// SPDX-License-Identifier: Apache-2.0
//
//===----------------------------------------------------------------------===//
import ClusterMembership
extension SWIM {
/// Emitted whenever a membership change happens.
///
/// Use `isReachabilityChange` to detect whether the is a change from an alive to unreachable/dead state or not,
/// and is worth emitting to user-code or not.
public struct MemberStatusChangedEvent<Peer: SWIMPeer>: Equatable {
/// The member that this change event is about.
public let member: SWIM.Member<Peer>
/// The resulting ("current") status of the `member`.
public var status: SWIM.Status {
// Note if the member is marked .dead, SWIM shall continue to gossip about it for a while
// such that other nodes gain this information directly, and do not have to wait until they detect
// it as such independently.
self.member.status
}
/// Previous status of the member, needed in order to decide if the change is "effective" or if applying the
/// member did not move it in such way that we need to inform the cluster about unreachability.
public let previousStatus: SWIM.Status?
/// Create new event, representing a change of the member's status from a previous state to its current state.
public init(previousStatus: SWIM.Status?, member: SWIM.Member<Peer>) {
if let from = previousStatus, from == .dead {
precondition(
member.status == .dead,
"Change MUST NOT move status 'backwards' from [.dead] state to anything else, but did so, was: \(member)"
)
}
self.previousStatus = previousStatus
self.member = member
switch (self.previousStatus, member.status) {
case (.dead, .alive),
(.dead, .suspect),
(.dead, .unreachable):
fatalError(
"SWIM.Membership MUST NOT move status 'backwards' from .dead state to anything else, but did so, was: \(self)"
)
default:
() // ok, all other transitions are valid.
}
}
}
}
extension SWIM.MemberStatusChangedEvent {
/// Reachability changes are important events, in which a reachable node became unreachable, or vice-versa,
/// as opposed to events which only move a member between `.alive` and `.suspect` status,
/// during which the member should still be considered and no actions assuming it's death shall be performed (yet).
///
/// If true, a system may want to issue a reachability change event and handle this situation by confirming the node `.dead`,
/// and proceeding with its removal from the cluster.
public var isReachabilityChange: Bool {
guard let fromStatus = self.previousStatus else {
// i.e. nil -> anything, is always an effective reachability affecting change
return true
}
// explicitly list all changes which are affecting reachability, all others do not (i.e. flipping between
// alive and suspect does NOT affect high-level reachability).
switch (fromStatus, self.status) {
case (.alive, .unreachable),
(.alive, .dead):
return true
case (.suspect, .unreachable),
(.suspect, .dead):
return true
case (.unreachable, .alive),
(.unreachable, .suspect):
return true
default:
return false
}
}
}
extension SWIM.MemberStatusChangedEvent: CustomStringConvertible {
public var description: String {
var res = "MemberStatusChangedEvent(\(self.member), previousStatus: "
if let previousStatus = self.previousStatus {
res += "\(previousStatus)"
} else {
res += "<unknown>"
}
res += ")"
return res
}
}
================================================
FILE: Sources/SWIM/Member.swift
================================================
//===----------------------------------------------------------------------===//
//
// This source file is part of the Swift Cluster Membership open source project
//
// Copyright (c) 2020 Apple Inc. and the Swift Cluster Membership project authors
// Licensed under Apache License v2.0
//
// See LICENSE.txt for license information
// See CONTRIBUTORS.txt for the list of Swift Cluster Membership project authors
//
// SPDX-License-Identifier: Apache-2.0
//
//===----------------------------------------------------------------------===//
import ClusterMembership
@preconcurrency import struct Dispatch.DispatchTime
// ==== ----------------------------------------------------------------------------------------------------------------
// MARK: SWIM Member
extension SWIM {
/// A `SWIM.Member` represents an active participant of the cluster.
///
/// It associates a specific `SWIMAddressablePeer` with its `SWIM.Status` and a number of other SWIM specific state information.
public struct Member<Peer: SWIMPeer>: Sendable {
/// Peer reference, used to send messages to this cluster member.
///
/// Can represent the "local" member as well, use `swim.isMyself` to verify if a peer is `myself`.
public var peer: Peer
/// `Node` of the member's `peer`.
public var node: ClusterMembership.Node {
self.peer.node
}
/// Membership status of this cluster member
public var status: SWIM.Status
// Period in which protocol period was this state set
public var protocolPeriod: UInt64
/// Indicates a _local_ point in time when suspicion was started.
///
/// - Note: Only suspect members may have this value set, but having the actual field in SWIM.Member feels more natural.
/// - Note: This value is never carried across processes, as it serves only locally triggering suspicion timeouts.
public let localSuspicionStartedAt: DispatchTime? // could be "status updated at"?
/// Create a new member.
public init(peer: Peer, status: SWIM.Status, protocolPeriod: UInt64, suspicionStartedAt: DispatchTime? = nil) {
self.peer = peer
self.status = status
self.protocolPeriod = protocolPeriod
self.localSuspicionStartedAt = suspicionStartedAt
}
/// Convenience function for checking if a member is `SWIM.Status.alive`.
///
/// - Returns: `true` if the member is alive
public var isAlive: Bool {
self.status.isAlive
}
/// Convenience function for checking if a member is `SWIM.Status.suspect`.
///
/// - Returns: `true` if the member is suspect
public var isSuspect: Bool {
self.status.isSuspect
}
/// Convenience function for checking if a member is `SWIM.Status.unreachable`
///
/// - Returns: `true` if the member is unreachable
public var isUnreachable: Bool {
self.status.isUnreachable
}
/// Convenience function for checking if a member is `SWIM.Status.dead`
///
/// - Returns: `true` if the member is dead
public var isDead: Bool {
self.status.isDead
}
}
}
/// Manual Hashable conformance since we omit `suspicionStartedAt` from identity
extension SWIM.Member: Hashable, Equatable {
public static func == (lhs: SWIM.Member<Peer>, rhs: SWIM.Member<Peer>) -> Bool {
lhs.peer.node == rhs.peer.node && lhs.protocolPeriod == rhs.protocolPeriod && lhs.status == rhs.status
}
public func hash(into hasher: inout Hasher) {
hasher.combine(self.peer.node)
hasher.combine(self.protocolPeriod)
hasher.combine(self.status)
}
}
extension SWIM.Member: CustomStringConvertible, CustomDebugStringConvertible {
public var description: String {
var res = "SWIM.Member(\(self.peer), \(self.status), protocolPeriod: \(self.protocolPeriod)"
if let suspicionStartedAt = self.localSuspicionStartedAt {
res.append(", suspicionStartedAt: \(suspicionStartedAt)")
}
res.append(")")
return res
}
public var debugDescription: String {
var res = "SWIM.Member(\(String(reflecting: self.peer)), \(self.status), protocolPeriod: \(self.protocolPeriod)"
if let suspicionStartedAt = self.localSuspicionStartedAt {
res.append(", suspicionStartedAt: \(suspicionStartedAt)")
}
res.append(")")
return res
}
}
================================================
FILE: Sources/SWIM/Metrics.swift
================================================
//===----------------------------------------------------------------------===//
//
// This source file is part of the Swift Cluster Membership open source project
//
// Copyright (c) 2020 Apple Inc. and the Swift Cluster Membership project authors
// Licensed under Apache License v2.0
//
// See LICENSE.txt for license information
// See CONTRIBUTORS.txt for the list of Swift Cluster Membership project authors
//
// SPDX-License-Identifier: Apache-2.0
//
//===----------------------------------------------------------------------===//
import Metrics
extension SWIM {
/// Object containing all metrics a SWIM instance and shell should be reporting.
///
/// - SeeAlso: `SWIM.Metrics.Shell` for metrics that a specific implementation should emit
public struct Metrics {
// ==== --------------------------------------------------------------------------------------------------------
// MARK: Membership
/// Number of members (alive)
public let membersAlive: Gauge
/// Number of members (suspect)
public let membersSuspect: Gauge
/// Number of members (unreachable)
public let membersUnreachable: Gauge
// Number of members (dead) is not reported, because "dead" is considered "removed" from the cluster
// -- no metric --
/// Total number of nodes *ever* declared noticed as dead by this member
public let membersTotalDead: Counter
/// The current number of tombstones for previously known (and now dead and removed) members.
public let removedDeadMemberTombstones: Gauge
// ==== --------------------------------------------------------------------------------------------------------
// MARK: Internal metrics
/// Current value of the local health multiplier.
public let localHealthMultiplier: Gauge
// ==== --------------------------------------------------------------------------------------------------------
// MARK: Probe metrics
/// Records the incarnation of the SWIM instance.
///
/// Incarnation numbers are bumped whenever the node needs to refute some gossip about itself,
/// as such the incarnation number *growth* is an interesting indicator of cluster observation churn.
public let incarnation: Gauge
/// Total number of successful probes (pings with successful replies)
public let successfulPingProbes: Counter
/// Total number of failed probes (pings with successful replies)
public let failedPingProbes: Counter
/// Total number of successful ping request probes (pingRequest with successful replies)
/// Either an .ack or .nack from the intermediary node count as an success here
public let successfulPingRequestProbes: Counter
/// Total number of failed ping request probes (pings requests with successful replies)
/// Only a .timeout counts as a failed ping request.
public let failedPingRequestProbes: Counter
// ==== ----------------------------------------------------------------------------------------------------------------
// MARK: Shell / Transport Metrics
/// Metrics to be filled in by respective SWIM shell implementations.
public let shell: ShellMetrics
public struct ShellMetrics {
// ==== ----------------------------------------------------------------------------------------------------
// MARK: Probe metrics
/// Records time it takes for ping successful round-trips.
public let pingResponseTime: Timer
/// Records time it takes for (every) successful pingRequest round-trip
public let pingRequestResponseTimeAll: Timer
/// Records the time it takes for the (first) successful pingRequest to round trip
/// (A ping request hits multiple intermediary peers, the first reply is what counts)
public let pingRequestResponseTimeFirst: Timer
/// Number of incoming messages received
public let messageInboundCount: Counter
/// Sizes of messages received, in bytes
public let messageInboundBytes: Recorder
/// Number of messages sent
public let messageOutboundCount: Counter
/// Sizes of messages sent, in bytes
public let messageOutboundBytes: Recorder
public init(settings: SWIM.Settings) {
self.pingResponseTime = Timer(
label: settings.metrics.makeLabel("roundTripTime", "ping")
)
self.pingRequestResponseTimeAll = Timer(
label: settings.metrics.makeLabel("roundTripTime", "pingRequest"),
dimensions: [("type", "all")]
)
self.pingRequestResponseTimeFirst = Timer(
label: settings.metrics.makeLabel("roundTripTime", "pingRequest"),
dimensions: [("type", "firstAck")]
)
self.messageInboundCount = Counter(
label: settings.metrics.makeLabel("message", "count"),
dimensions: [
("direction", "in")
]
)
self.messageInboundBytes = Recorder(
label: settings.metrics.makeLabel("message", "bytes"),
dimensions: [
("direction", "in")
]
)
self.messageOutboundCount = Counter(
label: settings.metrics.makeLabel("message", "count"),
dimensions: [
("direction", "out")
]
)
self.messageOutboundBytes = Recorder(
label: settings.metrics.makeLabel("message", "bytes"),
dimensions: [
("direction", "out")
]
)
}
}
public init(settings: SWIM.Settings) {
self.membersAlive = Gauge(
label: settings.metrics.makeLabel("members"),
dimensions: [("status", "alive")]
)
self.membersSuspect = Gauge(
label: settings.metrics.makeLabel("members"),
dimensions: [("status", "suspect")]
)
self.membersUnreachable = Gauge(
label: settings.metrics.makeLabel("members"),
dimensions: [("status", "unreachable")]
)
self.membersTotalDead = Counter(
label: settings.metrics.makeLabel("members", "total"),
dimensions: [("status", "dead")]
)
self.removedDeadMemberTombstones = Gauge(
label: settings.metrics.makeLabel("removedMemberTombstones")
)
self.localHealthMultiplier = Gauge(
label: settings.metrics.makeLabel("lha")
)
self.incarnation = Gauge(label: settings.metrics.makeLabel("incarnation"))
self.successfulPingProbes = Counter(
label: settings.metrics.makeLabel("probe", "ping"),
dimensions: [("type", "successful")]
)
self.failedPingProbes = Counter(
label: settings.metrics.makeLabel("probe", "ping"),
dimensions: [("type", "failed")]
)
self.successfulPingRequestProbes = Counter(
label: settings.metrics.makeLabel("probe", "pingRequest"),
dimensions: [("type", "successful")]
)
self.failedPingRequestProbes = Counter(
label: settings.metrics.makeLabel("probe", "pingRequest"),
dimensions: [("type", "failed")]
)
self.shell = .init(settings: settings)
}
}
}
extension SWIM.Metrics {
/// Update member metrics metrics based on SWIM's membership.
public func updateMembership(_ members: SWIM.Membership<some SWIMPeer>) {
var alives = 0
var suspects = 0
var unreachables = 0
for member in members {
switch member.status {
case .alive:
alives += 1
case .suspect:
suspects += 1
case .unreachable:
unreachables += 1
case .dead:
() // dead is reported as a removal when they're removed and tombstoned, not as a gauge
}
}
self.membersAlive.record(alives)
self.membersSuspect.record(suspects)
self.membersUnreachable.record(unreachables)
}
}
================================================
FILE: Sources/SWIM/Peer.swift
================================================
//===----------------------------------------------------------------------===//
//
// This source file is part of the Swift Cluster Membership open source project
//
// Copyright (c) 2020-2022 Apple Inc. and the Swift Cluster Membership project authors
// Licensed under Apache License v2.0
//
// See LICENSE.txt for license information
// See CONTRIBUTORS.txt for the list of Swift Cluster Membership project authors
//
// SPDX-License-Identifier: Apache-2.0
//
//===----------------------------------------------------------------------===//
import ClusterMembership
/// Any peer in the cluster, can be used used to identify a peer using its unique node that it represents.
public protocol SWIMAddressablePeer: Sendable {
/// Node that this peer is representing.
nonisolated var swimNode: ClusterMembership.Node { get }
}
extension SWIMAddressablePeer {
internal var node: ClusterMembership.Node {
self.swimNode
}
}
/// SWIM A peer which originated a `ping`, should be replied to with an `ack`.
public protocol SWIMPingOriginPeer: SWIMAddressablePeer {
associatedtype Peer: SWIMPeer
/// Acknowledge a `ping`.
///
/// - parameters:
/// - sequenceNumber: the sequence number of the incoming ping that this ack should acknowledge
/// - target: target peer which received the ping (i.e. "myself" on the recipient of the `ping`).
/// - incarnation: incarnation number of the target (myself),
/// which is used to clarify which status is the most recent on the recipient of this acknowledgement.
/// - payload: additional gossip data to be carried with the message.
/// It is already trimmed to be no larger than configured in `SWIM.Settings`.
func ack(
acknowledging sequenceNumber: SWIM.SequenceNumber,
target: Peer,
incarnation: SWIM.Incarnation,
payload: SWIM.GossipPayload<Peer>
) async throws
}
/// A SWIM peer which originated a `pingRequest` and thus can receive either an `ack` or `nack` from the intermediary.
public protocol SWIMPingRequestOriginPeer: SWIMPingOriginPeer {
associatedtype NackTarget: SWIMPeer
/// "Negative acknowledge" a ping.
///
/// This message may ONLY be send in an indirect-ping scenario from the "middle" peer.
/// Meaning, only a peer which received a `pingRequest` and wants to send the `pingRequestOrigin`
/// a nack in order for it to be aware that its message did reach this member, even if it never gets an `ack`
/// through this member, e.g. since the pings `target` node is actually not reachable anymore.
///
/// - parameters:
/// - sequenceNumber: the sequence number of the incoming `pingRequest` that this nack is a response to
/// - target: the target peer which was attempted to be pinged but we didn't get an ack from it yet and are sending a nack back eagerly
func nack(
acknowledging sequenceNumber: SWIM.SequenceNumber,
target: NackTarget
) async throws
}
/// SWIM peer which can be initiated contact with, by sending ping or ping request messages.
public protocol SWIMPeer: SWIMAddressablePeer {
associatedtype Peer: SWIMPeer
associatedtype PingOrigin: SWIMPingOriginPeer
associatedtype PingRequestOrigin: SWIMPingRequestOriginPeer
/// Perform a probe of this peer by sending a `ping` message.
///
/// We expect the reply to be an `ack`.
///
/// - parameters:
/// - payload: additional gossip information to be processed by the recipient
/// - origin: the origin peer that has initiated this ping message (i.e. "myself" of the sender)
/// replies (`ack`s) from to this ping should be send to this peer
/// - timeout: timeout during which we expect the other peer to have replied to us with a `PingResponse` about the pinged node.
/// If we get no response about that peer in that time, this `ping` is considered failed, and the onResponse MUST be invoked with a `.timeout`.
/// - sequenceNumber: sequence number of the ping message
///
/// - Returns the corresponding reply (`ack`) or `timeout` event for this ping request occurs.
///
/// - Throws if the ping fails or if the reply is `nack`.
func ping(
payload: SWIM.GossipPayload<Peer>,
from origin: PingOrigin,
timeout: Duration,
sequenceNumber: SWIM.SequenceNumber
) async throws -> SWIM.PingResponse<Peer, PingRequestOrigin>
/// Send a ping request to this peer, asking it to perform an "indirect ping" of the target on our behalf.
///
/// Any resulting acknowledgements back to us. If not acknowledgements come back from the target, the intermediary
/// may send back nack messages, indicating that our connection to the intermediary is intact, however we didn't see
/// acknowledgements from the target itself.
///
/// - parameters:
/// - target: target peer that should be probed by this the recipient on our behalf
/// - payload: additional gossip information to be processed by the recipient
/// - origin: the origin peer that has initiated this `pingRequest` (i.e. "myself" on the sender);
/// replies (`ack`s) from this indirect ping should be forwarded to it.
/// - timeout: timeout during which we expect the other peer to have replied to us with a `PingResponse` about the pinged node.
/// If we get no response about that peer in that time, this `pingRequest` is considered failed, and the onResponse MUST be invoked with a `.timeout`.
/// - sequenceNumber: sequence number of the pingRequest message
///
/// - Returns the corresponding reply (`ack`, `nack`) or `timeout` event for this ping request occurs.
/// - Throws if the ping request fails
func pingRequest(
target: Peer,
payload: SWIM.GossipPayload<Peer>,
from origin: PingOrigin,
timeout: Duration,
sequenceNumber: SWIM.SequenceNumber
) async throws -> SWIM.PingResponse<Peer, PingRequestOrigin>
}
================================================
FILE: Sources/SWIM/SWIM.swift
================================================
//===----------------------------------------------------------------------===//
//
// This source file is part of the Swift Cluster Membership open source project
//
// Copyright (c) 2020 Apple Inc. and the Swift Cluster Membership project authors
// Licensed under Apache License v2.0
//
// See LICENSE.txt for license information
// See CONTRIBUTORS.txt for the list of Swift Cluster Membership project authors
//
// SPDX-License-Identifier: Apache-2.0
//
//===----------------------------------------------------------------------===//
import ClusterMembership
import struct Dispatch.DispatchTime
extension SWIM {
/// Incarnation numbers serve as sequence number and used to determine which observation
/// is "more recent" when comparing gossiped information.
public typealias Incarnation = UInt64
/// A sequence number which can be used to associate with messages in order to establish an request/response
/// relationship between ping/pingRequest and their corresponding ack/nack messages.
public typealias SequenceNumber = UInt32
/// Typealias for the underlying membership representation.
public typealias Membership<Peer: SWIMPeer> = Dictionary<Node, SWIM.Member<Peer>>.Values
}
extension SWIM {
/// Message sent in reply to a `.ping`.
///
/// The ack may be delivered directly in a request-response fashion between the probing and pinged members,
/// or indirectly, as a result of a `pingRequest` message.
public enum PingResponse<Peer: SWIMPeer, PingRequestOrigin: SWIMPingRequestOriginPeer>: Sendable {
/// - parameters:
/// - target: the target of the ping;
/// On the remote "pinged" node which is about to send an ack back to the ping origin this should be filled with the `myself` peer.
/// - incarnation: the incarnation of the peer sent in the `target` field
/// - payload: additional gossip data to be carried with the message.
/// - sequenceNumber: the `sequenceNumber` of the `ping` message this ack is a "reply" for;
/// It is used on the ping origin to co-relate the reply with its handling code.
case ack(
target: Peer,
incarnation: Incarnation,
payload: GossipPayload<Peer>,
sequenceNumber: SWIM.SequenceNumber
)
/// A `.nack` MAY ONLY be sent by an *intermediary* member which was received a `pingRequest` to perform a `ping` of some `target` member.
/// It SHOULD NOT be sent by a peer that received a `.ping` directly.
///
/// The nack allows the origin of the ping request to know if the `k` peers it asked to perform the indirect probes,
/// are still responsive to it, or if perhaps that communication by itself is also breaking down. This information is
/// used to adjust the `localHealthMultiplier`, which impacts probe and timeout intervals.
///
/// Note that nack information DOES NOT directly cause unreachability or suspicions, it only adjusts the timeouts
/// and intervals used by the swim instance in order to take into account the potential that our local node is
/// potentially not healthy.
///
/// - parameters:
/// - target: the target of the ping;
/// On the remote "pinged" node which is about to send an ack back to the ping origin this should be filled with the `myself` peer.
/// - target: the target of the ping;
/// On the remote "pinged" node which is about to send an ack back to the ping origin this should be filled with the `myself` peer.
/// - payload: The gossip payload to be carried in this message.
///
/// - SeeAlso: Lifeguard IV.A. Local Health Aware Probe
case nack(target: Peer, sequenceNumber: SWIM.SequenceNumber)
/// This is a "pseudo-message", in the sense that it is not transported over the wire, but should be triggered
/// and fired into an implementation Shell when a ping has timed out.
///
/// If a response for some reason produces a different error immediately rather than through a timeout,
/// the shell should also emit a `.timeout` response and feed it into the `SWIM.Instance` as it is important for
/// timeout adjustments that the instance makes. The instance does not need to know specifics about the reason of
/// a response not arriving, thus they are all handled via the same timeout response rather than extra "error" responses.
///
/// - parameters:
/// - target: the target of the ping;
/// On the remote "pinged" node which is about to send an ack back to the ping origin this should be filled with the `myself` peer.
/// - pingRequestOrigin: if this response/timeout is in response to a ping that was caused by a pingRequest,
/// `pingRequestOrigin` must contain the original peer which originated the ping request.
/// - timeout: the timeout interval value that caused this message to be triggered;
/// In case of "cancelled" operations or similar semantics it is allowed to use a placeholder value here.
/// - sequenceNumber: the `sequenceNumber` of the `ping` message this ack is a "reply" for;
/// It is used on the ping origin to co-relate the reply with its handling code.
case timeout(
target: Peer,
pingRequestOrigin: PingRequestOrigin?,
timeout: Duration,
sequenceNumber: SWIM.SequenceNumber
)
/// Sequence number of the initial request this is a response to.
/// Used to pair up responses to the requests which initially caused them.
///
/// All ping responses are guaranteed to have a sequence number attached to them.
public var sequenceNumber: SWIM.SequenceNumber {
switch self {
case .ack(_, _, _, let sequenceNumber):
return sequenceNumber
case .nack(_, let sequenceNumber):
return sequenceNumber
case .timeout(_, _, _, let sequenceNumber):
return sequenceNumber
}
}
}
}
// ==== ----------------------------------------------------------------------------------------------------------------
// MARK: Gossip
extension SWIM {
/// A piece of "gossip" about a specific member of the cluster.
///
/// A gossip will only be spread a limited number of times, as configured by `settings.gossip.gossipedEnoughTimes(_:members:)`.
public struct Gossip<Peer: SWIMPeer>: Equatable {
/// The specific member (including status) that this gossip is about.
///
/// A change in member status implies a new gossip must be created and the count for the rumor mongering must be reset.
public let member: SWIM.Member<Peer>
/// The number of times this specific gossip message was gossiped to another peer.
public internal(set) var numberOfTimesGossiped: Int
}
/// A `GossipPayload` is used to spread gossips about members.
public enum GossipPayload<Peer: SWIMPeer>: Sendable {
/// Explicit case to signal "no gossip payload"
///
/// Effectively equivalent to an empty `.membership([])` case.
case none
/// Gossip information about a few select members.
case membership([SWIM.Member<Peer>])
}
}
extension SWIM.GossipPayload {
/// True if the underlying gossip is empty.
public var isNone: Bool {
switch self {
case .none:
return true
case .membership:
return false
}
}
/// True if the underlying gossip contains membership information.
public var isMembership: Bool {
switch self {
case .none:
return false
case .membership:
return true
}
}
}
================================================
FILE: Sources/SWIM/SWIMInstance.swift
================================================
//===----------------------------------------------------------------------===//
//
// This source file is part of the Swift Cluster Membership open source project
//
// Copyright (c) 2020 Apple Inc. and the Swift Cluster Membership project authors
// Licensed under Apache License v2.0
//
// See LICENSE.txt for license information
// See CONTRIBUTORS.txt for the list of Swift Cluster Membership project authors
//
// SPDX-License-Identifier: Apache-2.0
//
//===----------------------------------------------------------------------===//
import ClusterMembership
import CoreMetrics
import Logging
import struct Dispatch.DispatchTime
#if canImport(Darwin)
import Darwin
#elseif canImport(Glibc)
import Glibc
#elseif canImport(Musl)
import Musl
#else
#error("Unsupported platform")
#endif
extension SWIM {
/// The `SWIM.Instance` encapsulates the complete algorithm implementation of the `SWIM` protocol.
///
/// **Please refer to `SWIM` for an in-depth discussion of the algorithm and extensions implemented in this package.**
///
/// - SeeAlso: `SWIM` for a complete and in depth discussion of the protocol.
public struct Instance<
Peer: SWIMPeer,
PingOrigin: SWIMPingOriginPeer,
PingRequestOrigin: SWIMPingRequestOriginPeer
>: SWIMProtocol {
/// The settings currently in use by this instance.
public let settings: SWIM.Settings
/// Struct containing all metrics a SWIM Instance (and implementation Shell) should emit.
public let metrics: SWIM.Metrics
/// Node which this SWIM.Instance is representing in the cluster.
public var swimNode: ClusterMembership.Node {
self.peer.node
}
// Convenience overload for internal use so we don't have to repeat "swim" all the time.
internal var node: ClusterMembership.Node {
self.swimNode
}
private var log: Logger {
self.settings.logger
}
/// The `SWIM.Member` representing this instance, also referred to as "myself".
public var member: SWIM.Member<Peer> {
if let storedMyself = self.member(forNode: self.swimNode),
!storedMyself.status.isAlive
{
return storedMyself // it is something special, like .dead
} else {
// return the always up to date "our view" on ourselves
return SWIM.Member(
peer: self.peer,
status: .alive(incarnation: self.incarnation),
protocolPeriod: self.protocolPeriod
)
}
}
// We store the owning SWIMShell peer in order avoid adding it to the `membersToPing` list
private let peer: Peer
/// Main members storage, map to values to obtain current members.
internal var _members: [ClusterMembership.Node: SWIM.Member<Peer>] {
didSet {
self.metrics.updateMembership(self.members)
}
}
/// List of members maintained in random yet stable order, see `addMember` for details.
internal var membersToPing: [SWIM.Member<Peer>]
/// Constantly mutated by `nextMemberToPing` in an effort to keep the order in which we ping nodes evenly distributed.
private var _membersToPingIndex: Int = 0
private var membersToPingIndex: Int {
self._membersToPingIndex
}
/// Tombstones are needed to avoid accidentally re-adding a member that we confirmed as dead already.
internal var removedDeadMemberTombstones: Set<MemberTombstone> = [] {
didSet {
self.metrics.removedDeadMemberTombstones.record(self.removedDeadMemberTombstones.count)
}
}
private var _sequenceNumber: SWIM.SequenceNumber = 0
/// Sequence numbers are used to identify messages and pair them up into request/replies.
/// - SeeAlso: `SWIM.SequenceNumber`
public mutating func nextSequenceNumber() -> SWIM.SequenceNumber {
// TODO: can we make it internal? it does not really hurt having public
// TODO: sequence numbers per-target node? https://github.com/apple/swift-cluster-membership/issues/39
self._sequenceNumber += 1
return self._sequenceNumber
}
/// Lifeguard IV.A. Local Health Multiplier (LHM)
/// > These different sources of feedback are combined in a Local Health Multiplier (LHM).
/// > LHM is a saturating counter, with a max value S and min value zero, meaning it will not
/// > increase above S or decrease below zero.
///
/// The local health multiplier (LHM for short) is designed to relax the `probeInterval` and `pingTimeout`.
///
/// The value MUST be >= 0.
///
/// - SeeAlso: `SWIM.Instance.LHModifierEvent` for details how and when the LHM is adjusted.
public var localHealthMultiplier = 0 {
didSet {
assert(
self.localHealthMultiplier >= 0,
"localHealthMultiplier MUST NOT be < 0, but was: \(self.localHealthMultiplier)"
)
self.metrics.localHealthMultiplier.record(self.localHealthMultiplier)
}
}
/// Dynamically adjusted probing interval.
///
/// Usually this interval will be yielded with a directive at appropriate spots, so it should not be
/// necessary to invoke it manually.
///
/// - SeeAlso: `localHealthMultiplier` for more detailed documentation.
/// - SeeAlso: Lifeguard IV.A. Local Health Multiplier (LHM)
var dynamicLHMProtocolInterval: Duration {
.nanoseconds(Int(self.settings.probeInterval.nanoseconds * Int64(1 + self.localHealthMultiplier)))
}
/// Dynamically adjusted (based on Local Health) timeout to be used when sending `ping` messages.
///
/// Usually this interval will be yielded with a directive at appropriate spots, so it should not be
/// necessary to invoke it manually.
///
/// - SeeAlso: `localHealthMultiplier` for more detailed documentation.
/// - SeeAlso: Lifeguard IV.A. Local Health Multiplier (LHM)
var dynamicLHMPingTimeout: Duration {
.nanoseconds(Int(self.settings.pingTimeout.nanoseconds * Int64(1 + self.localHealthMultiplier)))
}
/// The incarnation number is used to get a sense of ordering of events, so if an `.alive` or `.suspect`
/// state with a lower incarnation than the one currently known by a node is received, it can be dropped
/// as outdated and we don't accidentally override state with older events. The incarnation can only
/// be incremented by the respective node itself and will happen if that node receives a `.suspect` for
/// itself, to which it will respond with an `.alive` with the incremented incarnation.
var incarnation: SWIM.Incarnation {
self._incarnation
}
private var _incarnation: SWIM.Incarnation = 0 {
didSet {
self.metrics.incarnation.record(self._incarnation)
}
}
private mutating func nextIncarnation() {
self._incarnation += 1
}
/// Creates a new SWIM algorithm instance.
public init(settings: SWIM.Settings, myself: Peer) {
self.settings = settings
self.peer = myself
self._members = [:]
self.membersToPing = []
self.metrics = SWIM.Metrics(settings: settings)
_ = self.addMember(myself, status: .alive(incarnation: 0))
self.metrics.incarnation.record(self.incarnation)
self.metrics.localHealthMultiplier.record(self.localHealthMultiplier)
self.metrics.updateMembership(self.members)
}
func makeSuspicion(incarnation: SWIM.Incarnation) -> SWIM.Status {
.suspect(incarnation: incarnation, suspectedBy: [self.node])
}
func mergeSuspicions(
suspectedBy: Set<ClusterMembership.Node>,
previouslySuspectedBy: Set<ClusterMembership.Node>
) -> Set<ClusterMembership.Node> {
var newSuspectedBy = previouslySuspectedBy
for suspectedBy in suspectedBy.sorted()
where newSuspectedBy.count < self.settings.lifeguard.maxIndependentSuspicions {
newSuspectedBy.update(with: suspectedBy)
}
return newSuspectedBy
}
/// Adjust the Local Health-aware Multiplier based on the event causing it.
///
/// - Parameter event: event which causes the LHM adjustment.
public mutating func adjustLHMultiplier(_ event: LHModifierEvent) {
defer {
self.settings.logger.trace(
"Adjusted LHM multiplier",
metadata: [
"swim/lhm/event": "\(event)",
"swim/lhm": "\(self.localHealthMultiplier)",
]
)
}
self.localHealthMultiplier =
min(
max(0, self.localHealthMultiplier + event.lhmAdjustment),
self.settings.lifeguard.maxLocalHealthMultiplier
)
}
// The protocol period represents the number of times we have pinged a random member
// of the cluster. At the end of every ping cycle, the number will be incremented.
// Suspicion timeouts are based on the protocol period, i.e. if a probe did not
// reply within any of the `suspicionTimeoutPeriodsMax` rounds, it would be marked as `.suspect`.
private var _protocolPeriod: UInt64 = 0
/// In order to speed up the spreading of "fresh" rumors, we order gossips in their "number of times gossiped",
/// and thus are able to easily pick the least spread rumor and pick it for the next gossip round.
///
/// This is tremendously important in order to spread information about e.g. newly added members to others,
/// before members which are aware of them could have a chance to all terminate, leaving the rest of the cluster
/// unaware about those new members. For disseminating suspicions this is less urgent, however also serves as an
/// useful optimization.
///
/// - SeeAlso: SWIM 4.1. Infection-Style Dissemination Component
private var _messagesToGossip: Heap<SWIM.Gossip<Peer>> = Heap(
comparator: {
$0.numberOfTimesGossiped < $1.numberOfTimesGossiped
}
)
/// Note that peers without UID (in their `Node`) will NOT be added to the membership.
///
/// This is because a cluster member must be a _specific_ peer instance, and not some arbitrary "some peer on that host/port",
/// which a Node without UID represents. The only reason we allow for peers and nodes without UID, is to simplify making
/// initial contact with a node - i.e. one can construct a peer to "there should be a peer on this host/port" to send an initial ping,
/// however in reply a peer in gossip must ALWAYS include it's unique identifier in the node - such that we know it from
/// any new instance of a process on the same host/port pair.
internal mutating func addMember(_ peer: Peer, status: SWIM.Status) -> [AddMemberDirective] {
var directives: [AddMemberDirective] = []
// Guard 1) protect against adding already known dead members
if self.hasTombstone(peer.node) {
// We saw this member already and even confirmed it dead, it shall never be added again
self.log.debug("Attempt to re-add already confirmed dead peer \(peer), ignoring it.")
directives.append(.memberAlreadyKnownDead(Member(peer: peer, status: .dead, protocolPeriod: 0)))
return directives
}
// Guard 2) protect against adding non UID members
guard peer.node.uid != nil else {
self.log.warning("Ignoring attempt to add peer representing node without UID: \(peer)")
return directives
}
let maybeExistingMember = self.member(for: peer)
if let existingMember = maybeExistingMember, existingMember.status.supersedes(status) {
// we already have a newer state for this member
directives.append(.newerMemberAlreadyPresent(existingMember))
return directives
}
/// if we're adding a node, it may be a reason to declare the previous "incarnation" as dead
// TODO: could solve by another dictionary without the UIDs?
if let withoutUIDMatchMember = self._members.first(where: {
$0.value.node.withoutUID == peer.node.withoutUID
})?.value,
peer.node.uid != nil, // the incoming node has UID, so it definitely is a real peer
peer.node.uid != withoutUIDMatchMember.node.uid
{ // the peers don't agree on UID, it must be a new node on same host/port
switch self.confirmDead(peer: withoutUIDMatchMember.peer) {
case .ignored:
() // should not happen?
case .applied(let change):
directives.append(.previousHostPortMemberConfirmedDead(change))
}
}
// just in case we had a peer added manually, and thus we did not know its uuid, let us remove it
// maybe we replaced a mismatching UID node already, but let's check and remove also if we stored any "without UID" node
if let removed = self._members.removeValue(forKey: self.node.withoutUID) {
switch self.confirmDead(peer: removed.peer) {
case .ignored:
() // should not happen?
case .applied(let change):
directives.append(.previousHostPortMemberConfirmedDead(change))
}
}
let member = SWIM.Member(peer: peer, status: status, protocolPeriod: self.protocolPeriod)
self._members[member.node] = member
if self.notMyself(member), !member.isDead {
// We know this is a new member.
//
// Newly added members are inserted at a random spot in the list of members
// to ping, to have a better distribution of messages to this node from all
// other nodes. If for example all nodes would add it to the end of the list,
// it would take a longer time until it would be pinged for the first time
// and also likely receive multiple pings within a very short time frame.
let insertIndex = Int.random(in: self.membersToPing.startIndex...self.membersToPing.endIndex)
self.membersToPing.insert(member, at: insertIndex)
if insertIndex <= self.membersToPingIndex {
// If we inserted the new member before the current `membersToPingIndex`,
// we need to advance the index to avoid pinging the same member multiple
// times in a row. This is especially critical when inserting a larger
// number of members, e.g. when the cluster is just being formed, or
// on a rolling restart.
self.advanceMembersToPingIndex()
}
}
// upon each membership change we reset the gossip counters
// such that nodes have a chance to be notified about others,
// even if a node joined an otherwise quiescent cluster.
self.resetGossipPayloads(member: member)
directives.append(.added(member))
return directives
}
enum AddMemberDirective {
/// Informs an implementation that a new member was added and now has the following state.
/// An implementation should react to this by emitting a cluster membership change event.
case added(SWIM.Member<Peer>)
/// By adding a node with a new UID on the same host/port, we may actually invalidate any previous member that
/// existed on this host/port part. If this is the case, we confirm the "previous" member on the same host/port
/// pair as dead immediately.
case previousHostPortMemberConfirmedDead(SWIM.MemberStatusChangedEvent<Peer>)
/// We already have information about this exact `Member`, and our information is more recent (higher incarnation number).
/// The incoming information was discarded and the returned here member is the most up to date information we have.
case newerMemberAlreadyPresent(SWIM.Member<Peer>)
/// Member already was part of the cluster, became dead and we removed it.
/// It shall never be part of the cluster again.
///
/// This is only enforced by tombstones which are kept in the system for a period of time,
/// in the hope that all other nodes stop gossiping about this known dead member until then as well.
case memberAlreadyKnownDead(SWIM.Member<Peer>)
}
/// Implements the round-robin yet shuffled member to probe selection as proposed in the SWIM paper.
///
/// This mechanism should reduce the time until state is spread across the whole cluster,
/// by guaranteeing that each node will be gossiped to within N cycles (where N is the cluster size).
///
/// - Note:
/// SWIM 4.3: [...] The failure detection protocol at member works by maintaining a list (intuitively, an array) of the known
/// elements of the current membership list, and select-ing ping targets not randomly from this list,
/// but in a round-robin fashion. Instead, a newly joining member is inserted in the membership list at
/// a position that is chosen uniformly at random. On completing a traversal of the entire list,
/// rearranges the membership list to a random reordering.
mutating func nextPeerToPing() -> Peer? {
if self.membersToPing.isEmpty {
return nil
}
defer {
self.advanceMembersToPingIndex()
}
return self.membersToPing[self.membersToPingIndex].peer
}
/// Selects `settings.indirectProbeCount` members to send a `ping-req` to.
func membersToPingRequest(target: SWIMAddressablePeer) -> ArraySlice<SWIM.Member<Peer>> {
func notTarget(_ peer: SWIMAddressablePeer) -> Bool {
peer.node != target.node
}
func isReachable(_ status: SWIM.Status) -> Bool {
status.isAlive || status.isSuspect
}
let candidates = self._members
.values
.filter {
notTarget($0.peer) && notMyself($0.peer) && isReachable($0.status)
}
.shuffled()
return candidates.prefix(self.settings.indirectProbeCount)
}
/// Mark a specific peer/member with the new status.
mutating func mark(_ peer: Peer, as status: SWIM.Status) -> MarkedDirective {
let previousStatusOption = self.status(of: peer)
var status = status
var protocolPeriod = self.protocolPeriod
var suspicionStartedAt: DispatchTime?
if case .suspect(let incomingIncarnation, let incomingSuspectedBy) = status,
case .suspect(let previousIncarnation, let previousSuspectedBy)? = previousStatusOption,
let member = self.member(for: peer),
incomingIncarnation == previousIncarnation
{
let suspicions = self.mergeSuspicions(
suspectedBy: incomingSuspectedBy,
previouslySuspectedBy: previousSuspectedBy
)
status = .suspect(incarnation: incomingIncarnation, suspectedBy: suspicions)
// we should keep old protocol period when member is already a suspect
protocolPeriod = member.protocolPeriod
suspicionStartedAt = member.localSuspicionStartedAt
} else if case .suspect = status {
suspicionStartedAt = self.now()
} else if case .unreachable = status,
case SWIM.Settings.UnreachabilitySettings.disabled = self.settings.unreachability
{
self.log.warning(
"Attempted to mark \(peer.node) as `.unreachable`, but unreachability is disabled! Promoting to `.dead`!"
)
status = .dead
}
if let previousStatus = previousStatusOption, previousStatus.supersedes(status) {
// we already have a newer status for this member
return .ignoredDueToOlderStatus(currentStatus: previousStatus)
}
let member = SWIM.Member(
peer: peer,
status: status,
protocolPeriod: protocolPeriod,
suspicionStartedAt: suspicionStartedAt
)
self._members[peer.node] = member
if status.isDead {
if let _ = self._members.removeValue(forKey: peer.node) {
self.metrics.membersTotalDead.increment()
}
self.removeFromMembersToPing(member)
if let uid = member.node.uid {
let deadline = self.protocolPeriod + self.settings.tombstoneTimeToLiveInTicks
let tombstone = MemberTombstone(uid: uid, deadlineProtocolPeriod: deadline)
self.removedDeadMemberTombstones.insert(tombstone)
}
}
self.resetGossipPayloads(member: member)
return .applied(previousStatus: previousStatusOption, member: member)
}
enum MarkedDirective: Equatable {
/// The status that was meant to be set is "old" and was ignored.
/// We already have newer information about this peer (`currentStatus`).
case ignoredDueToOlderStatus(currentStatus: SWIM.Status)
case applied(previousStatus: SWIM.Status?, member: SWIM.Member<Peer>)
}
private mutating func resetGossipPayloads(member: SWIM.Member<Peer>) {
// seems we gained a new member, and we need to reset gossip counts in order to ensure it also receive information about all nodes
// TODO: this would be a good place to trigger a full state sync, to speed up convergence; see https://github.com/apple/swift-cluster-membership/issues/37
self.members.forEach { self.addToGossip(member: $0) }
}
mutating func incrementProtocolPeriod() {
self._protocolPeriod += 1
}
mutating func advanceMembersToPingIndex() {
self._membersToPingIndex = (self._membersToPingIndex + 1) % self.membersToPing.count
}
mutating func removeFromMembersToPing(_ member: SWIM.Member<Peer>) {
if let index = self.membersToPing.firstIndex(where: { $0.peer.node == member.peer.node }) {
self.membersToPing.remove(at: index)
if index < self.membersToPingIndex {
self._membersToPingIndex -= 1
}
if self.membersToPingIndex >= self.membersToPing.count {
self._membersToPingIndex = self.membersToPing.startIndex
}
}
}
/// Current SWIM protocol period (i.e. which round of gossip the instance is in).
public var protocolPeriod: UInt64 {
self._protocolPeriod
}
/// Debug only. Actual suspicion timeout depends on number of suspicions and calculated in `suspicionTimeout`
/// This will only show current estimate of how many intervals should pass before suspicion is reached. May change when more data is coming
var timeoutSuspectsBeforePeriodMax: Int64 {
self.settings.lifeguard.suspicionTimeoutMax.nanoseconds / self.dynamicLHMProtocolInterval.nanoseconds + 1
}
/// Debug only. Actual suspicion timeout depends on number of suspicions and calculated in `suspicionTimeout`
/// This will only show current estimate of how many intervals should pass before suspicion is reached. May change when more data is coming
var timeoutSuspectsBeforePeriodMin: Int64 {
self.settings.lifeguard.suspicionTimeoutMin.nanoseconds / self.dynamicLHMProtocolInterval.nanoseconds + 1
}
/// Local Health Aware Suspicion timeout calculation, as defined Lifeguard IV.B.
///
/// Suspicion timeout is logarithmically decaying from `suspicionTimeoutPeriodsMax` to `suspicionTimeoutPeriodsMin`
/// depending on a number of suspicion confirmations.
///
/// Suspicion timeout adjusted according to number of known independent suspicions of given member.
///
/// See: Lifeguard IV-B: Local Health Aware Suspicion
///
/// The timeout for a given suspicion is calculated as follows:
///
/// ```
/// log(C + 1)
/// SuspicionTimeout = max(Min, Max − (Max−Min) ----------)
/// log(K + 1)
/// ```
///
/// where:
/// - `Min` and `Max` are the minimum and maximum Suspicion timeout.
/// See Section `V-C` for discussion of their configuration.
/// - `K` is the number of independent suspicions required to be received before setting the suspicion timeout to `Min`.
/// We default `K` to `3`.
/// - `C` is the number of independent suspicions about that member received since the local suspicion was raised.
public func suspicionTimeout(suspectedByCount: Int) -> Duration {
let minTimeout = self.settings.lifeguard.suspicionTimeoutMin.nanoseconds
let maxTimeout = self.settings.lifeguard.suspicionTimeoutMax.nanoseconds
return .nanoseconds(
Int(
max(
minTimeout,
maxTimeout
- Int64(
round(
Double(maxTimeout - minTimeout)
* (log2(Double(suspectedByCount + 1))
/ log2(Double(self.settings.lifeguard.maxIndependentSuspicions + 1)))
)
)
)
)
)
}
/// Checks if a deadline is expired (relating to current time).
///
/// - Parameter deadline: deadline we want to check if it's expired
/// - Returns: true if the `now()` time is "past" the deadline
public func isExpired(deadline: DispatchTime) -> Bool {
deadline < self.now()
}
/// Returns the current point in time on this machine.
/// - Note: `DispatchTime` is simply a number of nanoseconds since boot on this machine, and thus is not comparable across machines.
/// We use it on purpose, as we do not intend to share our local time observations with any other peers.
private func now() -> DispatchTime {
self.settings.timeSourceNow()
}
/// Create a gossip payload (i.e. a set of `SWIM.Gossip` messages) that should be gossiped with failure detector
/// messages, or using some other medium.
///
/// - Parameter target: Allows passing the target peer this gossip will be sent to.
/// If gossiping to a specific peer, and given peer is suspect, we will always prioritize
/// letting it know that it is being suspected, such that it can refute the suspicion as soon as possible,
/// if if still is alive.
/// - Returns: The gossip payload to be gossiped.
public mutating func makeGossipPayload(to target: SWIMAddressablePeer?) -> SWIM.GossipPayload<Peer> {
var membersToGossipAbout: [SWIM.Member<Peer>] = []
// Lifeguard IV. Buddy System
// Always send to a suspect its suspicion.
// The reason for that to ensure the suspect will be notified it is being suspected,
// even if the suspicion has already been disseminated "enough times".
let targetIsSuspect: Bool
if let target = target,
let member = self.member(forNode: target.node),
member.isSuspect
{
// the member is suspect, and we must inform it about this, thus including in gossip payload:
membersToGossipAbout.append(member)
targetIsSuspect = true
} else {
targetIsSuspect = false
}
guard self._messagesToGossip.count > 0 else {
if membersToGossipAbout.isEmpty {
// if we have no pending gossips to share, at least inform the member about our state.
return .membership([self.member])
} else {
return .membership(membersToGossipAbout)
}
}
// In order to avoid duplicates within a single gossip payload, we first collect all messages we need to
// gossip out and only then re-insert them into `messagesToGossip`. Otherwise, we may end up selecting the
// same message multiple times, if e.g. the total number of messages is smaller than the maximum gossip
// size, or for newer messages that have a lower `numberOfTimesGossiped` counter than the other messages.
var gossipRoundMessages: [SWIM.Gossip<Peer>] = []
gossipRoundMessages.reserveCapacity(
min(self.settings.gossip.maxNumberOfMessagesPerGossip, self._messagesToGossip.count)
)
while gossipRoundMessages.count < self.settings.gossip.maxNumberOfMessagesPerGossip,
let gossip = self._messagesToGossip.removeRoot()
{
gossipRoundMessages.append(gossip)
}
membersToGossipAbout.reserveCapacity(gossipRoundMessages.count)
for var gossip in gossipRoundMessages {
if targetIsSuspect, target?.node == gossip.member.node {
// We do NOT add gossip to payload if it's a gossip about target and target is suspect,
// this case was handled earlier and doing it here will lead to duplicate messages
()
} else {
membersToGossipAbout.append(gossip.member)
}
gossip.numberOfTimesGossiped += 1
if self.settings.gossip.needsToBeGossipedMoreTimes(gossip, members: self.members.count) {
self._messagesToGossip.append(gossip)
}
}
return .membership(membersToGossipAbout)
}
/// Adds `Member` to gossip messages.
internal mutating func addToGossip(member: SWIM.Member<Peer>) {
// we need to remove old state before we add the new gossip, so we don't gossip out stale state
self._messagesToGossip.remove(where: { $0.member.peer.node == member.peer.node })
self._messagesToGossip.append(.init(member: member, numberOfTimesGossiped: 0))
}
}
}
// ==== ----------------------------------------------------------------------------------------------------------------
// MARK: SWIM Member helper functions
extension SWIM.Instance {
func notMyself(_ member: SWIM.Member<Peer>) -> Bool {
self.whenMyself(member) == nil
}
func notMyself(_ peer: SWIMAddressablePeer) -> Bool {
!self.isMyself(peer.node)
}
func isMyself(_ member: SWIM.Member<Peer>) -> Bool {
self.isMyself(member.node)
}
func whenMyself(_ member: SWIM.Member<Peer>) -> SWIM.Member<Peer>? {
if self.isMyself(member.peer) {
return member
} else {
return nil
}
}
func isMyself(_ peer: SWIMAddressablePeer) -> Bool {
self.isMyself(peer.node)
}
func isMyself(_ node: Node) -> Bool {
// we are exactly that node:
self.node == node
// ...or, the incoming node has no UID; there was no handshake made,
// and thus the other side does not know which specific node it is going to talk to; as such, "we" are that node
// as such, "we" are that node; we should never add such peer to our members, but we will reply to that node with "us" and thus
// inform it about our specific UID, and from then onwards it will know about specifically this node (by replacing its UID-less version with our UID-ful version).
|| self.node.withoutUID == node
}
/// Returns status of the passed in peer's member of the cluster, if known.
///
/// - Parameter peer: the peer to look up the status for.
/// - Returns: Status of the peer, if known.
public func status(of peer: SWIMAddressablePeer) -> SWIM.Status? {
if self.notMyself(peer) {
return self._members[peer.node]?.status
} else {
// we consider ourselves always as alive (enables refuting others suspecting us)
return .alive(incarnation: self.incarnation)
}
}
/// Checks if the passed in peer is already a known member of the swim cluster.
///
/// Note: `.dead` members are eventually removed from the swim instance and as such peers are not remembered forever!
///
/// - parameters:
/// - peer: Peer to check if it currently is a member
/// - ignoreUID: Whether or not to ignore the peers UID, e.g. this is useful when issuing a "join 127.0.0.1:7337"
/// command, while being unaware of the nodes specific UID. When it joins, it joins with the specific UID after all.
/// - Returns: true if the peer is currently a member of the swim cluster (regardless of status it is in)
public func isMember(_ peer: SWIMAddressablePeer, ignoreUID: Bool = false) -> Bool {
// the peer could be either:
self.isMyself(peer) // 1) "us" (i.e. the peer which hosts this SWIM instance, or
|| self._members[peer.node] != nil // 2) a "known member"
|| (ignoreUID && peer.node.uid == nil
&& self._members.contains {
// 3) a known member, however the querying peer did not know the real UID of the peer yet
$0.key.withoutUID == peer.node
})
}
/// Returns specific `SWIM.Member` instance for the passed in peer.
///
/// - Parameter peer: peer whose member should be looked up (by its node identity, including the UID)
/// - Returns: the peer's member instance, if it currently is a member of this cluster
public func member(for peer: Peer) -> SWIM.Member<Peer>? {
self.member(forNode: peer.node)
}
/// Returns specific `SWIM.Member` instance for the passed in node.
///
/// - Parameter node: node whose member should be looked up (matching also by node UID)
/// - Returns: the peer's member instance, if it currently is a member of this cluster
public func member(forNode node: ClusterMembership.Node) -> SWIM.Member<Peer>? {
self._members[node]
}
/// Count of only non-dead members.
///
/// - SeeAlso: `SWIM.Status`
public var notDeadMemberCount: Int {
self._members.lazy.filter {
!$0.value.isDead
}.count
}
/// Count of all "other" members known to this instance (meaning members other than `myself`).
///
/// This is equal to `n-1` where `n` is the number of nodes in the cluster.
public var otherMemberCount: Int {
self.allMemberCount - 1
}
/// Count of all members, including the myself node as well as any unreachable and dead nodes which are still kept in the membership.
public var allMemberCount: Int {
self._members.count
}
/// Lists all members known to this SWIM instance currently, potentially including even `.dead` nodes.
///
/// - Complexity: O(1)
/// - Returns: Returns all current members of the cluster, including suspect, unreachable and potentially dead members.
public var members: SWIM.Membership<Peer> {
self._members.values
}
/// Lists all `SWIM.Status.suspect` members.
///
/// The `myself` member will never be suspect, as we always assume ourselves to be alive,
/// even if all other cluster members think otherwise - this is what allows us to refute
/// suspicions about our unreachability after all.
///
/// - SeeAlso: `SWIM.Status.suspect`
internal var suspects: [SWIM.Member<Peer>] {
self.members.filter { $0.isSuspect }
}
}
// ==== ----------------------------------------------------------------------------------------------------------------
// MARK: Handling SWIM protocol interactions
extension SWIM.Instance {
// ==== ------------------------------------------------------------------------------------------------------------
// MARK: On Periodic Ping Tick Handler
public mutating func onPeriodicPingTick() -> [PeriodicPingTickDirective] {
defer {
self.incrementProtocolPeriod()
}
var directives: [PeriodicPingTickDirective] = []
// 1) always check suspicion timeouts, even if we no longer have anyone else to ping
directives.append(contentsOf: self.checkSuspicionTimeouts())
// 2) if we have someone to ping, let's do so
if let toPing = self.nextPeerToPing() {
directives.append(
.sendPing(
target: toPing,
payload: self.makeGossipPayload(to: toPing),
timeout: self.dynamicLHMPingTimeout,
sequenceNumber: self.nextSequenceNumber()
)
)
}
// 3) periodic cleanup of tombstones
// TODO: could be optimized a bit to keep the "oldest one" and know if we have to scan already or not yet" etc
if self.protocolPeriod % UInt64(self.settings.tombstoneCleanupIntervalInTicks) == 0 {
cleanupTombstones()
}
// 3) ALWAYS schedule the next tick
directives.append(.scheduleNextTick(delay: self.dynamicLHMProtocolInterval))
return directives
}
/// Describes how a periodic tick should be handled.
public enum PeriodicPingTickDirective {
/// The membership has changed, e.g. a member was declared unreachable or dead and an event may need to be emitted.
case membershipChanged(SWIM.MemberStatusChangedEvent<Peer>)
/// Send a ping to the requested `target` peer using the provided timeout and sequenceNumber.
case sendPing(
target: Peer,
payload: SWIM.GossipPayload<Peer>,
timeout: Duration,
sequenceNumber: SWIM.SequenceNumber
)
/// Schedule the next timer `onPeriodicPingTick` invocation in `delay` time.
case scheduleNextTick(delay: Duration)
}
/// Check all suspects if any of them have been suspect for long enough that we should promote them to unreachable or dead.
///
/// Suspicion timeouts are calculated taking into account the number of peers suspecting a given member (LHA-Suspicion).
private mutating func checkSuspicionTimeouts() -> [PeriodicPingTickDirective] {
var directives: [PeriodicPingTickDirective] = []
for suspect in self.suspects {
if case .suspect(_, let suspectedBy) = suspect.status {
let suspicionTimeout = self.suspicionTimeout(suspectedByCount: suspectedBy.count)
// proceed with suspicion escalation to .unreachable if the timeout period has been exceeded
// We don't use Deadline because tests can override TimeSource
guard let suspectSince = suspect.localSuspicionStartedAt,
self.isExpired(
deadline: DispatchTime(
uptimeNanoseconds: suspectSince.uptimeNanoseconds + UInt64(suspicionTimeout.nanoseconds)
)
)
else {
continue // skip, this suspect is not timed-out yet
}
guard let incarnation = suspect.status.incarnation else {
// suspect had no incarnation number? that means it is .dead already and should be recycled soon
continue
}
let newStatus: SWIM.Status
if self.settings.unreachability == .enabled {
newStatus = .unreachable(incarnation: incarnation)
} else {
newStatus = .dead
}
switch self.mark(suspect.peer, as: newStatus) {
case .applied(let previousStatus, let member):
directives.append(
.membershipChanged(
SWIM.MemberStatusChangedEvent(previousStatus: previousStatus, member: member)
)
)
case .ignoredDueToOlderStatus:
continue
}
}
}
self.metrics.updateMembership(self.members)
return directives
}
// ==== ------------------------------------------------------------------------------------------------------------
// MARK: On Ping Handler
public mutating func onPing(
pingOrigin: PingOrigin,
payload: SWIM.GossipPayload<Peer>,
sequenceNumber: SWIM.SequenceNumber
) -> [PingDirective] {
var directives: [PingDirective]
// 1) Process gossip
directives = self.onGossipPayload(payload).map { g in
.gossipProcessed(g)
}
// 2) Prepare reply
directives.append(
.sendAck(
to: pingOrigin,
pingedTarget: self.peer,
incarnation: self.incarnation,
payload: self.makeGossipPayload(to: pingOrigin),
acknowledging: sequenceNumber
)
)
return directives
}
/// Directs a shell implementation about how to handle an incoming `.ping`.
public enum PingDirective {
/// Indicates that incoming gossip was processed and the membership may have changed because of it,
/// inspect the `GossipProcessedDirective` to learn more about what change was applied.
case gossipProcessed(GossipProcessedDirective)
/// Send an `ack` message.
///
/// - parameters:
/// - to: the peer to which an `ack` should be sent
/// - pingedTarget: the `myself` peer, should be passed as `target` when sending the ack message
/// - incarnation: the incarnation number of this peer; used to determine which status is "the latest"
/// when comparing acknowledgement with suspicions
/// - payload: additional gossip payload to include in the ack message
/// - acknowledging: sequence number of the ack message
case sendAck(
to: PingOrigin,
pingedTarget: Peer,
incarnation: SWIM.Incarnation,
payload: SWIM.GossipPayload<Peer>,
acknowledging: SWIM.SequenceNumber
)
}
// ==== ------------------------------------------------------------------------------------------------------------
// MARK: On Ping Response Handlers
public mutating func onPingResponse(
response: SWIM.PingResponse<Peer, PingRequestOrigin>,
pingRequestOrigin: PingRequestOrigin?,
pingRequestSequenceNumber: SWIM.SequenceNumber?
) -> [PingResponseDirective] {
switch response {
case .ack(let target, let incarnation, let payload, let sequenceNumber):
return self.onPingAckResponse(
target: target,
incarnation: incarnation,
payload: payload,
pingRequestOrigin: pingRequestOrigin,
pingRequestSequenceNumber: pingRequestSequenceNumber,
sequenceNumber: sequenceNumber
)
case .nack(let target, let sequenceNumber):
return self.onPingNackResponse(
target: target,
pingRequestOrigin: pingRequestOrigin,
sequenceNumber: sequenceNumber
)
case .timeout(let target, let pingRequestOrigin, let timeout, _):
return self.onPingResponseTimeout(
target: target,
timeout: timeout,
pingRequestOrigin: pingRequestOrigin,
pingRequestSequenceNumber: pingRequestSequenceNumber
)
}
}
mutating func onPingAckResponse(
target pingedNode: Peer,
incarnation: SWIM.Incarnation,
payload: SWIM.GossipPayload<Peer>,
pingRequestOrigin: PingRequestOrigin?,
pingRequestSequenceNumber: SWIM.SequenceNumber?,
sequenceNumber: SWIM.SequenceNumber
) -> [PingResponseDirective] {
self.metrics.successfulPingProbes.increment()
var directives: [PingResponseDirective] = []
// We're proxying an ack payload from ping target back to ping source.
// If ping target was a suspect, there'll be a refutation in a payload
// and we probably want to process it asap. And since the data is already here,
// processing this payload will just make gossip convergence faster.
let gossipDirectives = self.onGossipPayload(payload)
directives.append(
contentsOf: gossipDirectives.map {
PingResponseDirective.gossipProcessed($0)
}
)
self.log.debug(
"Received ack from [\(pingedNode)] with incarnation [\(incarnation)] and payload [\(payload)]",
metadata: self.metadata
)
// The shell is already informed tha the member moved -> alive by the gossipProcessed directive
_ = self.mark(pingedNode, as: .alive(incarnation: incarnation))
if let pingRequestOrigin = pingRequestOrigin,
let pingRequestSequenceNumber = pingRequestSequenceNumber
{
directives.append(
.sendAck(
peer: pingRequestOrigin,
acknowledging: pingRequestSequenceNumber,
target: pingedNode,
incarnation: incarnation,
payload: payload
)
)
} else {
self.adjustLHMultiplier(.successfulProbe)
}
return directives
}
mutating func onPingNackResponse(
target pingedNode: Peer,
pingRequestOrigin: PingRequestOrigin?,
sequenceNumber: SWIM.SequenceNumber
) -> [PingResponseDirective] {
// yes, a nack is "successful" -- we did get a reply from the peer we contacted after all
self.metrics.successfulPingProbes.increment()
// Important:
// We do _nothing_ here, however we actually handle nacks implicitly in today's SWIMNIO implementation...
// This works because the arrival of the nack means we removed the callback from the handler,
// so the timeout also is cancelled and thus no +1 will happen since the timeout will not trigger as well
//
// we should solve this more nicely, so any implementation benefits from this;
// FIXME: .nack handling discussion https://github.com/apple/swift-cluster-membership/issues/52
return []
}
mutating func onPingResponseTimeout(
target: Peer,
timeout: Duration,
pingRequestOrigin: PingRequestOrigin?,
pingRequestSequenceNumber: SWIM.SequenceNumber?
) -> [PingResponseDirective] {
self.metrics.failedPingProbes.increment()
var directives: [PingResponseDirective] = []
if let pingRequestOrigin = pingRequestOrigin,
let pingRequestSequenceNumber = pingRequestSequenceNumber
{
// Meaning we were doing a ping on behalf of the pingReq origin, we got a timeout, and thus need to report a nack back.
directives.append(
.sendNack(
peer: pingRequestOrigin,
acknowledging: pingRequestSequenceNumber,
target: target
)
)
// Note that we do NOT adjust the LHM multiplier, this is on purpose.
// We do not adjust it if we are only an intermediary.
} else {
// We sent a direct `.ping` and it timed out; we now suspect the target node and must issue additional ping requests.
guard let pingedMember = self.member(for: target) else {
return directives // seems we are not aware of this node, ignore it
}
guard let pingedMemberLastKnownIncarnation = pingedMember.status.incarnation else {
return directives // so it is already dead, not need to suspect it
}
// The member should become suspect, it missed out ping/ack cycle:
// we do not inform the shell about -> suspect moves; only unreachable or dead moves are of interest to it.
_ = self.mark(pingedMember.peer, as: self.makeSuspicion(incarnation: pingedMemberLastKnownIncarnation))
// adjust the LHM accordingly, we failed a probe (ping/ack) cycle
self.adjustLHMultiplier(.failedProbe)
// if we have other peers, we should ping request through them,
// if not then there's no-one to ping request through and we just continue.
if let pingRequestDirective = self.preparePingRequests(target: pingedMember.peer) {
directives.append(.sendPingRequests(pingRequestDirective))
}
}
return directives
}
/// Prepare ping request directives such that the shell can easily fire those messages
mutating func preparePingRequests(target: Peer) -> SendPingRequestDirective? {
guard let lastKnownStatus = self.status(of: target) else {
// FIXME allow logging
// context.log.info("Skipping ping requests after failed ping to [\(toPing)] because node has been removed from member list")
return nil
}
// select random members to send ping requests to
let membersToPingRequest = self.membersToPingRequest(target: target)
guard !membersToPingRequest.isEmpty else {
// no nodes available to ping, so we have to assume the node suspect right away
guard let lastKnownIncarnation = lastKnownStatus.incarnation else {
// TODO logging
// log.debug("Not marking .suspect, as [\(target)] is already dead.") // "You are already dead!"
return nil
}
switch self.mark(target, as: self.makeSuspicion(incarnation: lastKnownIncarnation)) {
case .applied:
// TODO: logging
// log.debug("No members to ping-req through, marked [\(target)] immediately as [\(currentStatus)].")
return nil
case .ignoredDueToOlderStatus:
// TODO: logging
// log.debug("No members to ping-req through to [\(target)], was already [\(currentStatus)].")
return nil
}
}
let details = membersToPingRequest.map { member in
SendPingRequestDirective.PingRequestDetail(
peerToPingRequestThrough: member.peer,
payload: self.makeGossipPayload(to: target),
sequenceNumber: self.nextSequenceNumber()
)
}
return SendPingRequestDirective(target: target, timeout: self.dynamicLHMPingTimeout, requestDetails: details)
}
/// Directs a shell implementation about how to handle an incoming `.pingRequest`.
public enum PingResponseDirective {
/// Indicates that incoming gossip was processed and the membership may have changed because of it,
/// inspect the `GossipProcessedDirective` to learn more about what change was applied.
case gossipProcessed(GossipProcessedDirective)
/// Upon receiving an `ack` from `target`, if we were making this ping because of a `pingRequest` from `peer`,
/// we need to forward that acknowledgement to that peer now.
///
/// - parameters:
/// - to: the peer to which an `ack` should be sent
/// - pingedTarget: the `myself` peer, should be passed as `target` when sending the ack message
/// - incarnation: the incarnation number of this peer; used to determine which status is "the latest"
/// when comparing acknowledgement with suspicions
/// - payload: additional gossip payload to include in the ack message
/// - acknowledging: sequence number of the ack message
case sendAck(
peer: PingRequestOrigin,
acknowledging: SWIM.SequenceNumber,
target: Peer,
incarnation: UInt64,
payload: SWIM.GossipPayload<Peer>
)
/// Send a `nack` to the `peer` which originally send this peer request.
///
/// - parameters:
/// - peer: the peer to which the `nack` should be sent
/// - acknowledging: sequence number of the ack message
/// - target: the peer which we attempted to ping but it didn't reply on time
case sendNack(peer: PingRequestOrigin, acknowledging: SWIM.SequenceNumber, target: Peer)
/// Send a `pingRequest` as described by the `SendPingRequestDirective`.
///
/// The target node did not reply with an successful `.ack` and as such was now marked as `.suspect`.
/// By sending ping requests to other members of the cluster we attempt to revert this suspicion,
/// perhaps some other node is able to receive an `.ack` from it after all?
case sendPingRequests(SendPingRequestDirective)
}
/// Describes how a pingRequest should be performed.
///
/// Only a single `target` peer is used, however it may be pinged "through" a few other members.
/// The amount of fan-out in pingRequests is configurable by `swim.indirectProbeCount`.
public struct SendPingRequestDirective {
/// Target that the should be probed by the `requestDetails.memberToPingRequestThrough` peers.
public let target: Peer
/// Timeout to be used for all the ping requests about to be sent.
public let timeout: Duration
/// Describes the details how each ping request should be performed.
public let requestDetails: [PingRequestDetail]
/// Describes a specific ping request to be made.
public struct PingRequestDetail {
/// Marks the peer the `pingRequest` should be sent to.
public let peerToPingRequestThrough: Peer
/// Additional gossip to carry with the `pingRequest`
public let payload: SWIM.GossipPayload<Peer>
/// Sequence number to assign to this `pingRequest`.
public let sequenceNumber: SWIM.SequenceNumber
}
}
// ==== ------------------------------------------------------------------------------------------------------------
// MARK: On Ping Request
public mutating func onPingRequest(
target: Peer,
pingRequestOrigin: PingRequestOrigin,
payload: SWIM.GossipPayload<Peer>,
sequenceNumber: SWIM.SequenceNumber
) -> [PingRequestDirective] {
var directives: [PingRequestDirective] = []
// 1) Process gossip
let gossipDirectives: [PingRequestDirective] = self.onGossipPayload(payload).map { directive in
.gossipProcessed(directive)
}
directives.append(contentsOf: gossipDirectives)
// 2) Process the ping request itself
guard self.notMyself(target) else {
self.log.debug(
"Received pingRequest to ping myself myself, ignoring.",
metadata: self.metadata([
"swim/pingRequestOrigin": "\(pingRequestOrigin)",
"swim/pingSequenceNumber": "\(sequenceNumber)",
])
)
return directives
}
if !self.isMember(target) {
// The case when member is a suspect is already handled in `processGossipPayload`,
// since payload will always contain suspicion about target member; no need to inform the shell again about this
_ = self.addMember(target, status: .alive(incarnation: 0))
}
let pingSequenceNumber = self.nextSequenceNumber()
// Indirect ping timeout should always be shorter than pingRequest timeout.
// Setting it to a fraction of initial ping timeout as suggested in the original paper.
// - SeeAlso: Local Health Multiplier (LHM)
let indirectPingTimeout = Duration.nanoseconds(
Int(Double(self.settings.pingTimeout.nanoseconds) * self.settings.lifeguard.indirectPingTimeoutMultiplier)
)
directives.append(
.sendPing(
target: target,
payload: self.makeGossipPayload(to: target),
pingRequestOrigin: pingRequestOrigin,
pingRequestSequenceNumber: sequenceNumber,
timeout: indirectPingTimeout,
pingSequenceNumber: pingSequenceNumber
)
)
return directives
}
/// Directs a shell implementation about how to handle an incoming `.pingRequest`.
public enum PingRequestDirective {
/// Indicates that incoming gossip was processed and the membership may have changed because of it,
/// inspect the `GossipProcessedDirective` to learn more about what change was applied.
case gossipProcessed(GossipProcessedDirective)
/// Send a ping to the requested `target` peer using the provided timeout and sequenceNumber.
///
/// - parameters:
/// - target: the target peer which should be probed
/// - payload: gossip information to be processed by this peer,
/// resulting in potentially discovering new information about other members of the cluster
/// - pingRequestOrigin: peer on whose behalf we are performing this indirect ping;
/// it will be useful to pipe back replies from the target to the origin member.
/// - pingRequestSequenceNumber: sequence number that must be used when replying to the `pingRequestOrigin`
/// - timeout: timeout to be used when performing the ping probe (it MAY be smaller than a normal direct ping probe's timeout)
/// - pingSequenceNumber: sequence number to use for the `ping` message
case sendPing(
target: Peer,
payload: SWIM.GossipPayload<Peer>,
pingRequestOrigin: PingRequestOrigin,
pingRequestSequenceNumber: SWIM.SequenceNumber,
timeout: Duration,
pingSequenceNumber: SWIM.SequenceNumber
)
}
// ==== ------------------------------------------------------------------------------------------------------------
// MARK: On Ping Request Response
/// This should be called on first successful (non-nack) pingRequestResponse
public mutating func onPingRequestResponse(
_ response: SWIM.PingResponse<Peer, PingRequestOrigin>,
pinged pingedPeer: Peer
) -> [PingRequestResponseDirective] {
guard let previousStatus = self.status(of: pingedPeer) else {
// we do not process replies from an unknown member; it likely means we have removed it already for some reason.
return [.unknownMember]
}
var directives: [PingRequestResponseDirective] = []
switch response {
case .ack(let target, let incarnation, let payload, _):
assert(
target.node == pingedPeer.node,
"The ack.from member [\(target)] MUST be equal to the pinged member \(pingedPeer.node)]; The Ack message is being forwarded back to us from the pinged member."
)
let gossipDirectives = self.onGossipPayload(payload)
directives += gossipDirectives.map {
PingRequestResponseDirective.gossipProcessed($0)
}
switch self.mark(pingedPeer, as: .alive(incarnation: incarnation)) {
case .applied:
directives.append(.alive(previousStatus: previousStatus))
return directives
case .ignoredDueToOlderStatus(let currentStatus):
directives.append(.ignoredDueToOlderStatus(currentStatus: currentStatus))
return directives
}
case .nack:
// TODO: this should never happen. How do we express it?
directives.append(.nackReceived)
return directives
case .timeout:
switch previousStatus {
case .alive(let incarnation),
.suspect(let incarnation, _):
switch self.mark(pingedPeer, as: self.makeSuspicion(incarnation: incarnation)) {
case .applied:
directives.append(
.newlySuspect(previousStatus: previousStatus, suspect: self.member(forNode: pingedPeer.node)!)
)
return directives
case .ignoredDueToOlderStatus(let status):
directives.append(.ignoredDueToOlderStatus(currentStatus: status))
return directives
}
case .unreachable:
directives.append(.alreadyUnreachable)
return directives
case .dead:
directives.append(.alreadyDead)
return directives
}
}
}
public mutating func onEveryPingRequestResponse(
_ result: SWIM.PingResponse<Peer, PingRequestOrigin>,
pinged peer: Peer
) -> [PingRequestResponseDirective] {
switch result {
case .timeout:
// Failed pingRequestResponse indicates a missed nack, we should adjust LHMultiplier
self.metrics.failedPingRequestProbes.increment()
self.adjustLHMultiplier(.probeWithMissedNack)
case .ack, .nack:
// Successful pingRequestResponse should be handled only once (and thus in `onPingRequestResponse` only),
// however we can nicely handle all responses here for purposes of metrics (and NOT adjust them in the onPingRequestResponse
// since that would lead to double-counting successes)
self.metrics.successfulPingRequestProbes.increment()
}
return [] // just so happens that we never actually perform any actions here (so far, keeping the return type for future compatibility)
}
/// Directs a shell implementation about how to handle an incoming ping request response.
public enum PingRequestResponseDirective {
/// Indicates that incoming gossip was processed and the membership may have changed because of it,
/// inspect the `GossipProcessedDirective` to learn more about what change was applied.
case gossipProcessed(GossipProcessedDirective)
case alive(previousStatus: SWIM.Status) // TODO: offer a membership change option rather?
case nackReceived
/// Indicates that the `target` of the ping response is not known to this peer anymore,
/// it could be that we already marked it as dead and removed it.
///
/// No additional action, except optionally some debug logging should be performed.
case unknownMember
case newlySuspect(previousStatus: SWIM.Status, suspect: SWIM.Member<Peer>)
case alreadySuspect
case alreadyUnreachable
case alreadyDead
/// The incoming gossip is older than already known information about the target peer (by incarnation), and was (safely) ignored.
/// The current status of the peer is as returned in `currentStatus`.
case ignoredDueToOlderStatus(currentStatus: SWIM.Status)
}
internal mutating func onGossipPayload(_ payload: SWIM.GossipPayload<Peer>) -> [GossipProcessedDirective] {
switch payload {
case .none:
return []
case .membership(let members):
return members.flatMap { member in
self.onGossipPayload(about: member)
}
}
}
internal mutating func onGossipPayload(about member: SWIM.Member<Peer>) -> [GossipProcessedDirective] {
if self.isMyself(member) {
return [self.onMyselfGossipPayload(myself: member)]
} else {
return self.onOtherMemberGossipPayload(member: member)
}
}
/// ### Unreachability status handling
/// Performs all special handling of `.unreachable` such that if it is disabled members are automatically promoted to `.dead`.
/// See `settings.unreachability` for more details.
private mutating func onMyselfGossipPayload(myself incoming: SWIM.Member<Peer>) -> GossipProcessedDirective {
assert(
self.peer.node == incoming.peer.node,
"""
Attempted to process gossip as-if about myself, but was not the same peer, was: \(incoming.peer.node.detailedDescription). \
Myself: \(self.peer)
SWIM.Instance: \(self)
"""
)
// Note, we don't yield changes for myself node observations, thus the self node will never be reported as unreachable,
// after all, we can always reach ourselves. We may reconsider this if we wanted to allow SWIM to inform us about
// the fact that many other nodes think we're unreachable, and thus we could perform self-downing based upon this information
switch incoming.status {
case .alive:
// as long as other nodes see us as alive, we're happy
return .applied(change: nil)
case .suspect(let suspectedInIncarnation, _):
// someone suspected us, so we need to increment our incarnation number to spread our alive status with
// the incremented incarnation
if suspectedInIncarnation == self.incarnation {
self.adjustLHMultiplier(.refutingSuspectMessageAboutSelf)
self.nextIncarnation()
// refute the suspicion, we clearly are still alive
self.addToGossip(member: self.member)
return .applied(change: nil)
} else if suspectedInIncarnation > self.incarnation {
self.log.warning(
"""
Received gossip about self with incarnation number [\(suspectedInIncarnation)] > current incarnation [\(self._incarnation)], \
which should never happen and while harmless is highly suspicious, please raise an issue with logs. This MAY be an issue in the library.
"""
)
return .applied(change: nil)
} else {
// incoming incarnation was < than current one, i.e. the incoming information is "old" thus we discard it
return .applied(change: nil)
}
case .unreachable(let unreachableInIncarnation):
switch self.settings.unreachability {
case .enabled:
// someone suspected us,
// so we need to increment our incarnation number to spread our alive status with the incremented incarnation
if unreachableInIncarnation == self.incarnation {
self.nextIncarnation()
return .ignored
} else if unreachableInIncarnation > self.incarnation {
self.log.warning(
"""
Received gossip about self with incarnation number [\(unreachableInIncarnation)] > current incarnation [\(self._incarnation)], \
which should never happen and while harmless is highly suspicious, please raise an issue with logs. This MAY be an issue in the library.
"""
)
return .applied(change: nil)
} else {
self.log.debug(
"Incoming .unreachable about myself, however current incarnation [\(self.incarnation)] is greater than incoming \(incoming.status)"
)
return .ignored
}
case .disabled:
// we don't use unreachable states, and in any case, would not apply it to myself
// as we always consider "us" to be reachable after all
return .ignored
}
case .dead:
guard var myselfMember = self.member(for: self.peer) else {
return .applied(change: nil)
}
myselfMember.status = .dead
switch self.mark(self.peer, as: .dead) {
case .applied(.some(let previousStatus), _):
return .applied(change: .init(previousStatus: previousStatus, member: myselfMember))
default:
self.log.warning("\(self.peer) already marked .dead", metadata: self.metadata)
return .ignored
}
}
}
/// ### Unreachability status handling
/// Performs all special handling of `.unreachable` such that if it is disabled members are automatically promoted to `.dead`.
/// See `settings.unreachability` for more details.
private mutating func onOtherMemberGossipPayload(member: SWIM.Member<Peer>) -> [GossipProcessedDirective] {
assert(
self.node != member.node,
"Attempted to process gossip as-if not-myself, but WAS same peer, was: \(member). Myself: \(self.peer, orElse: "nil")"
)
guard self.isMember(member.peer) else {
// it's a new node it seems
guard member.node.uid != nil else {
self.log.debug(
"Incoming member has no `uid`, ignoring; cannot add members to membership without uid",
metadata: self.metadata([
"member": "\(member)",
"member/node": "\(member.node.detailedDescription)",
])
)
return []
}
// the Shell may need to set up a connection if we just made a move from previousStatus: nil,
// so we definitely need to emit this change
return self.addMember(member.peer, status: member.status).compactMap { directive in
switch directive {
case .added(let member):
return .applied(change: SWIM.MemberStatusChangedEvent(previousStatus: nil, member: member))
case .previousHostPortMemberConfirmedDead(let change):
return .applied(change: change)
case .memberAlreadyKnownDead:
return nil
case .newerMemberAlreadyPresent(let member):
return .applied(change: SWIM.MemberStatusChangedEvent(previousStatus: nil, member: member))
}
}
}
var directives: [GossipProcessedDirective] = []
switch self.mark(member.peer, as: member.status) {
case .applied(let previousStatus, let member):
if member.status.isSuspect, previousStatus?.isAlive ?? false {
self.log.debug(
"Member [\(member.peer.node, orElse: "<unknown-node>")] marked as suspect, via incoming gossip",
metadata: self.metadata
)
}
directives.append(.applied(change: .init(previousStatus: previousStatus, member: member)))
case .ignoredDueToOlderStatus(let currentStatus):
self.log.trace(
"Gossip about member \(member.node), incoming: [\(member.status)] does not supersede current: [\(currentStatus)]",
metadata: self.metadata
)
}
return directives
}
/// Indicates the gossip payload was processed and changes to the membership were made.
public enum GossipProcessedDirective: Equatable {
/// The gossip was applied to the local membership view and an event may want to be emitted for it.
///
/// It is up to the shell implementation which events are published, but generally it is recommended to
/// only publish changes which are `SWIM.MemberStatusChangedEvent.isReachabilityChange` as those can and should
/// usually be acted on by high level implementations.
///
/// Changes between alive and suspect are an internal implementation detail of SWIM,
/// and usually do not need to be emitted as events to users.
///
/// ### Note for connection based implementations
/// You may need to establish a new connection if the changes' `previousStatus` is `nil`, as it means we have
/// not seen this member before and in order to send messages to it, one may want to eagerly establish a connection to it.
case applied(change: SWIM.MemberStatusChangedEvent<Peer>?)
static var ignored: Self {
.applied(change: nil)
}
}
// ==== ------------------------------------------------------------------------------------------------------------
// MARK: Confirm Dead
public mutating func confirmDead(peer: Peer) -> ConfirmDeadDirective {
if self.member(for: peer) == nil,
self._members.first(where: { $0.key == peer.node }) == nil
{
return .ignored // this peer is absolutely unknown to us, we should not even emit events about it
}
switch self.mark(peer, as: .dead) {
case .applied(let previousStatus, let member):
return .applied(change: SWIM.MemberStatusChangedEvent(previousStatus: previousStatus, member: member))
case .ignoredDueToOlderStatus:
return .ignored // it was already dead for example
}
}
/// Directs how to handle the result of a `confirmDead` call.
public enum ConfirmDeadDirective {
/// The change was applied and caused a membership change.
///
/// The change should be emitted as an event by an interpreting shell.
case applied(change: SWIM.MemberStatusChangedEvent<Peer>)
/// The confirmation had not effect, either the peer was not known, or is already dead.
case ignored
}
/// Returns if this node is known to have already been marked dead at some point.
func hasTombstone(_ node: Node) -> Bool {
guard let uid = node.uid else {
return false
}
let anythingAsNotTakenIntoAccountInEquality: UInt64 = 0
return self.removedDeadMemberTombstones.contains(
.init(uid: uid, deadlineProtocolPeriod: anythingAsNotTakenIntoAccountInEquality)
)
}
private mutating func cleanupTombstones() { // time to cleanup the tombstones
self.removedDeadMemberTombstones = self.removedDeadMemberTombstones.filter {
// keep the ones where their deadline is still in the future
self.protocolPeriod < $0.deadlineProtocolPeriod
}
}
/// Used to store known "confirmed dead" member unique identifiers.
struct MemberTombstone: Hashable {
/// UID of the dead member
let uid: UInt64
/// After how many protocol periods ("ticks") should this tombstone be cleaned up
let deadlineProtocolPeriod: UInt64
func hash(into hasher: inout Hasher) {
hasher.combine(self.uid)
}
static func == (lhs: MemberTombstone, rhs: MemberTombstone) -> Bool {
lhs.uid == rhs.uid
}
}
}
extension SWIM.Instance: CustomDebugStringConvertible {
public var debugDescription: String {
// multi-line on purpose
"""
SWIM.Instance(
settings: \(settings),
myself: \(String(reflecting: peer)),
_incarnation: \(_incarnation),
_protocolPeriod: \(_protocolPeriod),
members: [
\(_members.map { "\($0.key)" }.joined(separator: "\n "))
]
membersToPing: [
\(membersToPing.map { "\($0)" }.joined(separator: "\n "))
]
_messagesToGossip: \(_messagesToGossip)
)
"""
}
}
// ==== ----------------------------------------------------------------------------------------------------------------
// MARK: SWIM Lifeguard Local Health Modifier event
extension SWIM.Instance {
/// Events which cause the modification of the Local health aware Multiplier to be adjusted.
///
/// The LHM is increased (in increments of `1`) whenever an event occurs that indicates that the instance
/// is not processing incoming messages in timely order.
///
/// It is decreased and decreased (by `1`), whenever it processes a successful ping/ack cycle,
/// meaning that is is healthy and properly processing incoming messages on time.
///
/// - SeeAlso: Lifeguard IV.A. Local Health Aware Probe, which describes the rationale behind the events.
public enum LHModifierEvent: Equatable {
/// A successful ping/ack probe cycle was completed.
case successfulProbe
/// A direct ping/ack cycle has failed (timed-out).
case failedProbe
/// Some other member has suspected this member, and we had to refute the suspicion.
case refutingSuspectMessageAboutSelf
/// During a `pingRequest` the ping request origin (us) received a timeout without seeing `.nack`
/// from the intermediary member; This could mean we are having network trouble and are a faulty node.
case probeWithMissedNack
/// - Returns: by how much the LHM should be adjusted in response to this event.
/// The adjusted value MUST be clamped between `0 <= value <= maxLocalHealthMultiplier`
var lhmAdjustment: Int {
switch self {
case .successfulProbe:
return -1 // decrease the LHM
case .failedProbe,
.refutingSuspectMessageAboutSelf,
.probeWithMissedNack:
return 1 // increase the LHM
}
}
}
}
// ==== ----------------------------------------------------------------------------------------------------------------
// MARK: SWIM Logging Metadata
extension SWIM.Instance {
/// Allows for convenient adding of additional metadata to the `SWIM.Instance.metadata`.
public func metadata(_ additional: Logger.Metadata) -> Logger.Metadata {
var metadata = self.metadata
metadata.merge(additional, uniquingKeysWith: { _, r in r })
return metadata
}
/// While the SWIM.Instance is not meant to be logging by itself, it does offer metadata for loggers to use.
public var metadata: Logger.Metadata {
[
"swim/protocolPeriod": "\(self.protocolPeriod)",
"swim/timeoutSuspectsBeforePeriodMax": "\(self.timeoutSuspectsBeforePeriodMax)",
"swim/timeoutSuspectsBeforePeriodMin": "\(self.timeoutSuspectsBeforePeriodMin)",
"swim/incarnation": "\(self.incarnation)",
"swim/members/all": Logger.Metadata.Value.array(self.members.map { "\(reflecting: $0)" }),
"swim/members/count": "\(self.notDeadMemberCount)",
"swim/suspects/count": "\(self.suspects.count)",
]
}
}
================================================
FILE: Sources/SWIM/SWIMProtocol.swift
================================================
//===----------------------------------------------------------------------===//
//
// This source file is part of the Swift Cluster Membership open source project
//
// Copyright (c) 2020 Apple Inc. and the Swift Cluster Membership project authors
// Licensed under Apache License v2.0
//
// See LICENSE.txt for license information
// See CONTRIBUTORS.txt for the list of Swift Cluster Membership project authors
//
// SPDX-License-Identifier: Apache-2.0
//
//===----------------------------------------------------------------------===//
import ClusterMembership
import Logging
import struct Dispatch.DispatchTime
#if canImport(Darwin)
import Darwin
#elseif canImport(Glibc)
import Glibc
#elseif canImport(Musl)
import Musl
#else
#error("Unsupported platform")
#endif
/// ## Scalable Weakly-consistent Infection-style Process Group Membership Protocol
///
/// > As you swim lazily through the milieu, <br/>
/// > The secrets of the world will infect you.
///
/// Implementation of the SWIM protocol in abstract terms, not dependent on any specific runtime.
/// The actual implementation resides in `SWIM.Instance`.
///
/// ### Terminology
/// This implementation follows the original terminology mostly directly, with the notable exception of the original
/// wording of "confirm" being rather represented as `SWIM.Status.dead`, as we found the "confirm" wording to be
/// confusing in practice.
///
/// ### Extensions & Modifications
///
/// This implementation has a few notable extensions and modifications implemented, some documented already in the initial
/// SWIM paper, some in the Lifeguard extensions paper and some being simple adjustments we found practical in our environments.
///
/// - The "random peer selection" is not completely ad-hoc random, but follows a _stable order_, randomized on peer insertion.
/// - Unlike the completely random selection in the original paper. This has the benefit of consistently going "around"
/// all peers participating in the cluster, enabling a more efficient spread of membership information among peers,
/// by allowing us to avoid continuously (yet randomly) selecting the same few peers.
/// - This optimization is described in the original SWIM paper, and followed by some implementations.
///
/// - Introduction of an `.unreachable` status, that is ordered after `.suspect` and before `.dead`.
/// - This is because the decision to move an unreachable peer to .dead status is a large and important decision,
/// in which user code may want to participate, e.g. by attempting "shoot the other peer in the head" or other patterns,
/// before triggering the `.dead` status (which usually implies a complete removal of information of that peer existence from the cluster),
/// after which no further communication with given peer will ever be possible anymore.
/// - The `.unreachable` status is optional and _disabled_ by default.
/// - Other SWIM implementations handle this problem by _storing_ dead members for a period of time after declaring them dead,
/// also deviating from the original paper; so we conclude that this use case is quite common and allow addressing it in various ways.
///
/// - Preservation of `.unreachable` information
/// - The original paper does not keep in memory information about dead peers,
/// it only gossips the information that a member is now dead, but does not keep tombstones for later reference.
///
/// Implementations of extensions documented in the Lifeguard paper (linked below):
///
/// - Local Health Aware Probe - which replaces the static timeouts in probing with a dynamic one, taking into account
/// recent communication failures of our member with others.
/// - Local Health Aware Suspicion - which improves the way `.suspect` states and their timeouts are handled,
/// effectively relying on more information about unreachability. See: `suspicionTimeout`.
/// - Buddy System - enables members to directly and immediately notify suspect peers about them being suspected,
/// such that they have more time and a chance to refute these
gitextract_bytsy_b1/
├── .editorconfig
├── .github/
│ ├── ISSUE_TEMPLATE/
│ │ └── issue-template.md
│ ├── PULL_REQUEST_TEMPLATE.md
│ ├── dependabot.yml
│ ├── release.yml
│ └── workflows/
│ ├── main.yml
│ ├── pull_request.yml
│ └── pull_request_label.yml
├── .gitignore
├── .licenseignore
├── .spi.yml
├── .swift-format
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── CONTRIBUTORS.txt
├── HANDBOOK.md
├── LICENSE.txt
├── NOTICE.txt
├── Package.swift
├── README.md
├── Samples/
│ ├── .gitignore
│ ├── Package.swift
│ ├── README.md
│ ├── Sources/
│ │ └── SWIMNIOSampleCluster/
│ │ ├── SWIMNIOSampleNode.swift
│ │ └── main.swift
│ └── Tests/
│ └── NoopTests/
│ └── SampleTest.swift
├── Sources/
│ ├── ClusterMembership/
│ │ └── Node.swift
│ ├── SWIM/
│ │ ├── Docs.docc/
│ │ │ ├── images/
│ │ │ │ ├── ping_pingreq_cycle.graffle
│ │ │ │ └── swim_lifecycle.graffle
│ │ │ └── index.md
│ │ ├── Events.swift
│ │ ├── Member.swift
│ │ ├── Metrics.swift
│ │ ├── Peer.swift
│ │ ├── SWIM.swift
│ │ ├── SWIMInstance.swift
│ │ ├── SWIMProtocol.swift
│ │ ├── Settings.swift
│ │ ├── Status.swift
│ │ └── Utils/
│ │ ├── Heap.swift
│ │ ├── String+Extensions.swift
│ │ ├── _PrettyLog.swift
│ │ └── time.swift
│ └── SWIMNIOExample/
│ ├── Coding.swift
│ ├── Logging.swift
│ ├── Message.swift
│ ├── NIOPeer.swift
│ ├── SWIMNIOHandler.swift
│ ├── SWIMNIOShell.swift
│ ├── Settings.swift
│ └── Utils/
│ ├── String+Extensions.swift
│ └── time.swift
├── Tests/
│ ├── ClusterMembershipDocumentationTests/
│ │ └── SWIMDocExamples.swift
│ ├── ClusterMembershipTests/
│ │ └── NodeTests.swift
│ ├── SWIMNIOExampleTests/
│ │ ├── CodingTests.swift
│ │ ├── SWIMNIOClusteredTests.swift
│ │ ├── SWIMNIOEventClusteredTests.swift
│ │ ├── SWIMNIOMetricsTests.swift
│ │ └── Utils/
│ │ └── BaseXCTestCases.swift
│ ├── SWIMTestKit/
│ │ ├── LogCapture.swift
│ │ └── TestMetrics.swift
│ └── SWIMTests/
│ ├── HeapTests.swift
│ ├── SWIMInstanceTests.swift
│ ├── SWIMMetricsTests.swift
│ ├── SWIMSettingsTests.swift
│ └── TestPeer.swift
└── dev/
└── git.commit.template
Condensed preview — 67 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (514K chars).
[
{
"path": ".editorconfig",
"chars": 130,
"preview": "root = true\n\n[*]\nindent_style = space\nindent_size = 4\nend_of_line = lf\ninsert_final_newline = true\ntrim_trailing_whitesp"
},
{
"path": ".github/ISSUE_TEMPLATE/issue-template.md",
"chars": 600,
"preview": "---\nname: Issue Template\nabout: Template for reporting general issues with the library\ntitle: ''\nlabels: 0 - new\nassigne"
},
{
"path": ".github/PULL_REQUEST_TEMPLATE.md",
"chars": 302,
"preview": "_[One line description of your change]_\n\n### Motivation:\n\n_[Explain here the context, and why you're making that change."
},
{
"path": ".github/dependabot.yml",
"chars": 118,
"preview": "version: 2\nupdates:\n - package-ecosystem: \"github-actions\"\n directory: \"/\"\n schedule:\n interval: \"weekly\"\n"
},
{
"path": ".github/release.yml",
"chars": 284,
"preview": "changelog:\n categories:\n - title: SemVer Major\n labels:\n - ⚠️ semver/major\n - title: SemVer Minor\n "
},
{
"path": ".github/workflows/main.yml",
"chars": 2580,
"preview": "name: Main\n\npermissions:\n contents: read\n\non:\n push:\n branches: [main]\n schedule:\n - cron: \"0 8,20 * * *\"\n\njobs"
},
{
"path": ".github/workflows/pull_request.yml",
"chars": 2495,
"preview": "name: PR\n\npermissions:\n contents: read\n\non:\n pull_request:\n types: [opened, reopened, synchronize]\n\njobs:\n soundne"
},
{
"path": ".github/workflows/pull_request_label.yml",
"chars": 510,
"preview": "name: PR label\n\npermissions:\n contents: read\n\non:\n pull_request:\n types: [labeled, unlabeled, opened, reopened, syn"
},
{
"path": ".gitignore",
"chars": 305,
"preview": ".DS_Store\n.swift-version\n\n*.orig\n*.app\n\n/.build\n/Samples/.build\n/.SourceKitten\n/Packages\n.xcode\n*.app\n/*.xcodeproj\nSampl"
},
{
"path": ".licenseignore",
"chars": 552,
"preview": ".gitignore\n**/.gitignore\n.licenseignore\n.gitattributes\n.git-blame-ignore-revs\n.mailfilter\n.mailmap\n.spi.yml\n.swift-forma"
},
{
"path": ".spi.yml",
"chars": 86,
"preview": "version: 1\nbuilder:\n configs:\n - documentation_targets: [ClusterMembership, SWIM]\n"
},
{
"path": ".swift-format",
"chars": 2524,
"preview": "{\n\n\n \"version\" : 1,\n \"indentation\" : {\n \"spaces\" : 4\n },\n \"tabWidth\" : 4,\n \"fileScopedDeclarationPrivacy\" "
},
{
"path": "CODE_OF_CONDUCT.md",
"chars": 107,
"preview": "# Code of Conduct\n\nThe code of conduct for this project can be found at https://swift.org/code-of-conduct.\n"
},
{
"path": "CONTRIBUTING.md",
"chars": 2978,
"preview": "## Legal\n\nBy submitting a pull request, you represent that you have the right to license\nyour contribution to Apple and "
},
{
"path": "CONTRIBUTORS.txt",
"chars": 594,
"preview": "For the purpose of tracking copyright, this is the list of individuals and\norganizations who have contributed source cod"
},
{
"path": "HANDBOOK.md",
"chars": 3562,
"preview": "# Contributors Handbook\n\n## Glossary\n\nIn an attempt to form a shared vocabulary in the project, words used in the APIs h"
},
{
"path": "LICENSE.txt",
"chars": 11358,
"preview": "\n Apache License\n Version 2.0, January 2004\n "
},
{
"path": "NOTICE.txt",
"chars": 1316,
"preview": "\n The Swift Cluster Membership Project\n =========================="
},
{
"path": "Package.swift",
"chars": 5493,
"preview": "// swift-tools-version:5.10\n// The swift-tools-version declares the minimum version of Swift required to build this pack"
},
{
"path": "README.md",
"chars": 15616,
"preview": "# Swift Cluster Membership\n\nThis library aims to help Swift make ground in a new space: clustered multi-node distributed"
},
{
"path": "Samples/.gitignore",
"chars": 66,
"preview": "# The XPC sample generates an .app, so we want to ignore it\n*.app\n"
},
{
"path": "Samples/Package.swift",
"chars": 1800,
"preview": "// swift-tools-version:5.10\n// The swift-tools-version declares the minimum version of Swift required to build this pack"
},
{
"path": "Samples/README.md",
"chars": 1948,
"preview": "## Sample applications\n\nUse `swift run` to run the samples.\n\n### SWIMNIOSampleCluster\n\nThis sample app runs a _single no"
},
{
"path": "Samples/Sources/SWIMNIOSampleCluster/SWIMNIOSampleNode.swift",
"chars": 2476,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Samples/Sources/SWIMNIOSampleCluster/main.swift",
"chars": 4789,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Samples/Tests/NoopTests/SampleTest.swift",
"chars": 678,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Sources/ClusterMembership/Node.swift",
"chars": 3104,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Sources/SWIM/Docs.docc/index.md",
"chars": 12226,
"preview": "# ``SWIM``\n\nThis library aims to help Swift make ground in a new space: clustered multi-node distributed systems.\n\n## Ov"
},
{
"path": "Sources/SWIM/Events.swift",
"chars": 4428,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Sources/SWIM/Member.swift",
"chars": 4584,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Sources/SWIM/Metrics.swift",
"chars": 8790,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Sources/SWIM/Peer.swift",
"chars": 6038,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Sources/SWIM/SWIM.swift",
"chars": 7957,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Sources/SWIM/SWIMInstance.swift",
"chars": 80941,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Sources/SWIM/SWIMProtocol.swift",
"chars": 14770,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Sources/SWIM/Settings.swift",
"chars": 20309,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Sources/SWIM/Status.swift",
"chars": 7143,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Sources/SWIM/Utils/Heap.swift",
"chars": 8695,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Sources/SWIM/Utils/String+Extensions.swift",
"chars": 1875,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Sources/SWIM/Utils/_PrettyLog.swift",
"chars": 4668,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Sources/SWIM/Utils/time.swift",
"chars": 2739,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Sources/SWIMNIOExample/Coding.swift",
"chars": 12974,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Sources/SWIMNIOExample/Logging.swift",
"chars": 2176,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Sources/SWIMNIOExample/Message.swift",
"chars": 5861,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Sources/SWIMNIOExample/NIOPeer.swift",
"chars": 7751,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Sources/SWIMNIOExample/SWIMNIOHandler.swift",
"chars": 14521,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Sources/SWIMNIOExample/SWIMNIOShell.swift",
"chars": 31561,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Sources/SWIMNIOExample/Settings.swift",
"chars": 2199,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Sources/SWIMNIOExample/Utils/String+Extensions.swift",
"chars": 1875,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Sources/SWIMNIOExample/Utils/time.swift",
"chars": 6137,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Tests/ClusterMembershipDocumentationTests/SWIMDocExamples.swift",
"chars": 649,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Tests/ClusterMembershipTests/NodeTests.swift",
"chars": 1784,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Tests/SWIMNIOExampleTests/CodingTests.swift",
"chars": 4858,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Tests/SWIMNIOExampleTests/SWIMNIOClusteredTests.swift",
"chars": 11174,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Tests/SWIMNIOExampleTests/SWIMNIOEventClusteredTests.swift",
"chars": 6749,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Tests/SWIMNIOExampleTests/SWIMNIOMetricsTests.swift",
"chars": 5490,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Tests/SWIMNIOExampleTests/Utils/BaseXCTestCases.swift",
"chars": 6988,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Tests/SWIMTestKit/LogCapture.swift",
"chars": 13826,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Tests/SWIMTestKit/TestMetrics.swift",
"chars": 12306,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Tests/SWIMTests/HeapTests.swift",
"chars": 6162,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Tests/SWIMTests/SWIMInstanceTests.swift",
"chars": 70378,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Tests/SWIMTests/SWIMMetricsTests.swift",
"chars": 11288,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Tests/SWIMTests/SWIMSettingsTests.swift",
"chars": 2303,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "Tests/SWIMTests/TestPeer.swift",
"chars": 4451,
"preview": "//===----------------------------------------------------------------------===//\n//\n// This source file is part of the S"
},
{
"path": "dev/git.commit.template",
"chars": 252,
"preview": "# One line description of your change\n\n**Motivation:**\n\n# Explain here the context and why you're making that change. Wh"
}
]
// ... and 2 more files (download for full content)
About this extraction
This page contains the full source code of the apple/swift-cluster-membership GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 67 files (483.6 KB), approximately 106.2k tokens. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.