Showing preview only (794K chars total). Download the full file or copy to clipboard to get everything.
Repository: clojure/core.rrb-vector
Branch: master
Commit: d1cc27f1698b
Files: 74
Total size: 759.9 KB
Directory structure:
gitextract_vrqmmogg/
├── .github/
│ └── workflows/
│ ├── doc-build.yml
│ ├── release.yml
│ ├── snapshot.yml
│ └── test.yml
├── .gitignore
├── CHANGES.md
├── CONTRIBUTING.md
├── LICENSE
├── README.md
├── deps.edn
├── doc/
│ ├── benchmarks/
│ │ ├── benchmarks.md
│ │ └── data/
│ │ ├── benchmarks.edn
│ │ ├── concat.csv
│ │ ├── list_construct.csv
│ │ ├── list_iterate.csv
│ │ └── list_lookup.csv
│ ├── crrbv-27/
│ │ ├── description.md
│ │ ├── proposed-fix-needs-thought-and-testing-plus-debug-prints.patch
│ │ ├── proposed-fix-needs-thought-and-testing.patch
│ │ └── use-shift-increment-2.patch
│ ├── hash-details.md
│ ├── rrb-tree-notes.md
│ └── use-transducers/
│ ├── README.md
│ └── use-transducers.patch
├── epl-v10.html
├── pom.xml
├── project.clj
├── script/
│ ├── jdo
│ ├── mvn-run-tests
│ ├── replace-params
│ ├── sdo
│ └── test
└── src/
├── main/
│ ├── cljs/
│ │ └── clojure/
│ │ └── core/
│ │ ├── rrb_vector/
│ │ │ ├── debug.cljs
│ │ │ ├── debug_platform_dependent.cljs
│ │ │ ├── interop.cljs
│ │ │ ├── macros.clj
│ │ │ ├── nodes.cljs
│ │ │ ├── protocols.cljs
│ │ │ ├── rrbt.cljs
│ │ │ ├── transients.cljs
│ │ │ └── trees.cljs
│ │ └── rrb_vector.cljs
│ └── clojure/
│ └── clojure/
│ └── core/
│ ├── rrb_vector/
│ │ ├── debug.clj
│ │ ├── debug_platform_dependent.clj
│ │ ├── fork_join.clj
│ │ ├── interop.clj
│ │ ├── nodes.clj
│ │ ├── parameters.clj
│ │ ├── protocols.clj
│ │ ├── rrbt.clj
│ │ └── transients.clj
│ └── rrb_vector.clj
├── parameterized/
│ └── clojure/
│ └── clojure/
│ └── core/
│ ├── rrb_vector/
│ │ ├── debug.clj
│ │ ├── debug_platform_dependent.clj
│ │ ├── fork_join.clj
│ │ ├── interop.clj
│ │ ├── nodes.clj
│ │ ├── parameters.clj
│ │ ├── protocols.clj
│ │ ├── rrbt.clj
│ │ └── transients.clj
│ └── rrb_vector.clj
├── test/
│ ├── cljs/
│ │ └── clojure/
│ │ └── core/
│ │ └── rrb_vector/
│ │ ├── long_test.cljs
│ │ ├── test_cljs.cljs
│ │ ├── test_cljs_only.cljs
│ │ ├── test_common.cljs
│ │ └── test_utils.cljs
│ ├── clojure/
│ │ └── clojure/
│ │ └── core/
│ │ └── rrb_vector/
│ │ ├── long_test.clj
│ │ ├── test_clj_only.clj
│ │ ├── test_cljs.clj
│ │ ├── test_common.clj
│ │ └── test_utils.clj
│ └── resources/
│ └── clojure/
│ └── core/
│ └── rrb_vector/
│ └── cljs_testsuite.clj
└── test_local/
└── clojure/
└── clojure/
└── core/
└── rrb_vector_check.clj
================================================
FILE CONTENTS
================================================
================================================
FILE: .github/workflows/doc-build.yml
================================================
name: Build API Docs
permissions:
contents: write
on:
workflow_dispatch:
jobs:
call-doc-build-workflow:
uses: clojure/build.ci/.github/workflows/doc-build.yml@master
with:
project: clojure/core.rrb-vector
================================================
FILE: .github/workflows/release.yml
================================================
name: Release on demand
permissions:
contents: write
on:
workflow_dispatch:
inputs:
releaseVersion:
description: "Version to release"
required: true
snapshotVersion:
description: "Snapshot version after release"
required: true
jobs:
call-release:
uses: clojure/build.ci/.github/workflows/release.yml@master
with:
releaseVersion: ${{ github.event.inputs.releaseVersion }}
snapshotVersion: ${{ github.event.inputs.snapshotVersion }}
secrets: inherit
================================================
FILE: .github/workflows/snapshot.yml
================================================
name: Snapshot on demand
permissions:
contents: read
on: [workflow_dispatch]
jobs:
call-snapshot:
uses: clojure/build.ci/.github/workflows/snapshot.yml@master
secrets: inherit
================================================
FILE: .github/workflows/test.yml
================================================
name: Test
permissions:
contents: read
on: [push]
jobs:
test:
strategy:
matrix:
os: [ubuntu-latest] # macOS-latest, windows-latest]
java-version: ["8", "11"] # NOTE: tests fail on Java 17 as they depend on Nashorn
clojure-version: ["1.9.0", "1.10.3", "1.11.1"]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v3
- name: Set up Java
uses: actions/setup-java@v3
with:
java-version: ${{ matrix.java-version }}
distribution: 'temurin'
cache: 'maven'
- name: Build with Maven
run: mvn -ntp -B -Dclojure.version=${{ matrix.clojure-version }} clean test
================================================
FILE: .gitignore
================================================
/target
/lib
/classes
/checkouts
*.jar
*.class
.lein-deps-sum
.lein-failures
.lein-plugins
.lein-repl-history
/.repl
/out
/repl
.\#*
/.nrepl-port
.idea
*.iml
/.cpcache
================================================
FILE: CHANGES.md
================================================
# Changes in 0.2.1
* Update parent pom and default Clojure version to 1.11.4
# Changes in 0.2.0
* Update dependencies and versions
# Changes in 0.1.2
Bug fixes:
* Correct handling of reduce-kv for empty vectors [CRRBV-29](https://clojure.atlassian.net/browse/CRRBV-29)
# Changes in 0.1.1
## Changes visible to users of the library
Bug fixes:
* Eliminate warning issued by ClojureScript compiler caused by use of a not-yet-defined type name [CRRBV-24](https://clojure.atlassian.net/browse/CRRBV-24)
Minor code cleanup:
* Eliminate redundant require of a namespace
[commit](https://github.com/clojure/core.rrb-vector/commit/8c3bdc03f4d4c73326ac0146310bf472cda4d035)
Documentation:
* Created this change log.
* Added new introductory text at beginning of README explaining
briefly why someone might want to use this library.
* Added doc/benchmarks/benchmarks.md document, with link from README,
showing some benchmark results of this library versus several other
implementations of vector data structures, some of which are based
on RRB trees, some of which have only a linear time vector
concatenation operation.
* Added doc/rrb-tree-notes.md with links to other implementations of
RRB trees, and papers and theses that have been written about them.
## Changes relevant to those who test and develop the library itself
* Added doc/use-transducers/ directory with README and proposed patch
for speeding up some of the code by using transducers, which in the
cases used provide a speedup by avoiding allocating multiple
intermediate sequences.
# Changes in 0.1.0
## Changes visible to users of the library
Bug fixes:
* Test case added that was failing before the fixes for other bugs listed below, but now works with those fixes, so likely its root cause has also been corrected [CRRBV-12](https://clojure.atlassian.net/browse/CRRBV-12)
* Fixed bug that caused assoc and assoc! to fail when used on vectors of primitives [CRRBV-13](https://clojure.atlassian.net/browse/CRRBV-13)
* Fixed bug where internal tree data structure gets too "tall and skinny", eventually exceeding the limits supported by the library's implementation [CRRBV-14](https://clojure.atlassian.net/browse/CRRBV-14)
* Bug with similar root cause, and the same fix as, CRRBV-14 [CRRBV-17](https://clojure.atlassian.net/browse/CRRBV-17)
* Fix incorrect condition check for when a subtree was full or not [CRRBV-20](https://clojure.atlassian.net/browse/CRRBV-20)
* Fix of several bugs found during implementation and testing of other issues [CRRBV-21](https://clojure.atlassian.net/browse/CRRBV-21)
* Fix off by one bug that caused incorrect results for pop and pop! operations on vectors of certain sizes [CRRBV-22](https://clojure.atlassian.net/browse/CRRBV-22)
* Fix incorrect hash calculation for empty RRB vectors in ClojureScript implementation [CRRBV-25](https://clojure.atlassian.net/browse/CRRBV-25)
* Fix potential data race for multi-threaded Clojure programs using this library where the hash value could be returned incorrectly as -1 instead of the correct value [CRRBV-26](https://clojure.atlassian.net/browse/CRRBV-26)
Enhancements:
* Support bootstrapped ClojureScript [CRRBV-16](https://clojure.atlassian.net/browse/CRRBV-16)
## Changes relevant to those who test and develop the library itself
In order to continue to support Clojure 1.6.0, core.rrb-vector uses
neither `.cljc` files nor transducers, for which Clojure added support
in Clojure 1.7.0. However, in preparation for a future
core.rrb-vector release that requires Clojure 1.7.0 or later, several
test namespaces have been made nearly identical between their Clojure
and ClojureScript versions, so that they can be replaced with a single
`.cljc` file in the future, with only a few small uses of reader
conditionals.
The implementation of core.rrb-vector still contains similar, but
independent, implementations in Clojure and ClojureScript, and no
attempt has been made to make their implementations so similar that
merging them into a combined .cljc file would be reasonable.
Replacing some or all of the Clojure implementation with Java source
code may help improve the constant factors of the execution time
enough to warrant such a change in a future version of this library.
* Updated Maven pom.xml file, Leiningen project.clj file, and Clojure
deps.edn files, so that any of them may be used for the purposes of
developing and testing this library further. Added examples of
commands for all of these tools for running this library's tests,
and commands for Ubuntu Linux and macOS for installing two different
JavaScript run time engines that can be used to test the
ClojureScript implementation.
* The ClojureScript tests are now run, using JDK's Nashorn JavaScript
run time environment, on build.clojure.org, in addition to the
previous behavior of running the Clojure tests.
* Rearranged tests in the test namespaces extensively, including some
of their namespace names. There is now a `test-common` namespace
that contains most of the tests, which can thus be run on both the
Clojure and ClojureScript implementations. There are
`test-clj-only` and `test-cljs-only` namespaces for tests unique to
one of the implementations.
* The `clojure.core.rrb-vector.debug/dbg-vec` function has been
enhanced to support showing internal details of both the built in
Clojure vectors, persistent and transient, as well as
core.rrb-vector's data structures. The `debug` namespace also has
several `checking-*` functions, e.g. `checking-catvec`,
`checking-subvec`, etc. that behave the same as their non-checking
counterparts, but perform significant sanity checking of their
return values before they return, and while they have some
configuration options, by default they throw exceptions if they find
errors in the returned values. Any such exceptions are likely to be
due to bugs in core.rrb-vector. These checking functions are
significantly slower than the non-checking variants, and only
intended for testing the core.rrb-vector library. The details of
configuring their options are not intended to be stable, and thus
likely to change in future releases of this library.
================================================
FILE: CONTRIBUTING.md
================================================
This is a [Clojure contrib] project.
Under the Clojure contrib [guidelines], this project cannot accept
pull requests. All patches must be submitted via [JIRA].
See [Contributing] on the Clojure website for
more information on how to contribute.
[Clojure contrib]: https://clojure.org/community/contrib_libs
[Contributing]: https://clojure.org/community/contributing
[JIRA]: https://clojure.atlassian.net/browse/CRRBV
[guidelines]: https://clojure.org/community/contrib_howto
================================================
FILE: LICENSE
================================================
Eclipse Public License - v 1.0
THE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THIS ECLIPSE PUBLIC
LICENSE ("AGREEMENT"). ANY USE, REPRODUCTION OR DISTRIBUTION OF THE PROGRAM
CONSTITUTES RECIPIENT'S ACCEPTANCE OF THIS AGREEMENT.
1. DEFINITIONS
"Contribution" means:
a) in the case of the initial Contributor, the initial code and documentation
distributed under this Agreement, and
b) in the case of each subsequent Contributor:
i) changes to the Program, and
ii) additions to the Program;
where such changes and/or additions to the Program originate from and are
distributed by that particular Contributor. A Contribution 'originates'
from a Contributor if it was added to the Program by such Contributor
itself or anyone acting on such Contributor's behalf. Contributions do not
include additions to the Program which: (i) are separate modules of
software distributed in conjunction with the Program under their own
license agreement, and (ii) are not derivative works of the Program.
"Contributor" means any person or entity that distributes the Program.
"Licensed Patents" mean patent claims licensable by a Contributor which are
necessarily infringed by the use or sale of its Contribution alone or when
combined with the Program.
"Program" means the Contributions distributed in accordance with this
Agreement.
"Recipient" means anyone who receives the Program under this Agreement,
including all Contributors.
2. GRANT OF RIGHTS
a) Subject to the terms of this Agreement, each Contributor hereby grants
Recipient a non-exclusive, worldwide, royalty-free copyright license to
reproduce, prepare derivative works of, publicly display, publicly
perform, distribute and sublicense the Contribution of such Contributor,
if any, and such derivative works, in source code and object code form.
b) Subject to the terms of this Agreement, each Contributor hereby grants
Recipient a non-exclusive, worldwide, royalty-free patent license under
Licensed Patents to make, use, sell, offer to sell, import and otherwise
transfer the Contribution of such Contributor, if any, in source code and
object code form. This patent license shall apply to the combination of
the Contribution and the Program if, at the time the Contribution is
added by the Contributor, such addition of the Contribution causes such
combination to be covered by the Licensed Patents. The patent license
shall not apply to any other combinations which include the Contribution.
No hardware per se is licensed hereunder.
c) Recipient understands that although each Contributor grants the licenses
to its Contributions set forth herein, no assurances are provided by any
Contributor that the Program does not infringe the patent or other
intellectual property rights of any other entity. Each Contributor
disclaims any liability to Recipient for claims brought by any other
entity based on infringement of intellectual property rights or
otherwise. As a condition to exercising the rights and licenses granted
hereunder, each Recipient hereby assumes sole responsibility to secure
any other intellectual property rights needed, if any. For example, if a
third party patent license is required to allow Recipient to distribute
the Program, it is Recipient's responsibility to acquire that license
before distributing the Program.
d) Each Contributor represents that to its knowledge it has sufficient
copyright rights in its Contribution, if any, to grant the copyright
license set forth in this Agreement.
3. REQUIREMENTS
A Contributor may choose to distribute the Program in object code form under
its own license agreement, provided that:
a) it complies with the terms and conditions of this Agreement; and
b) its license agreement:
i) effectively disclaims on behalf of all Contributors all warranties
and conditions, express and implied, including warranties or
conditions of title and non-infringement, and implied warranties or
conditions of merchantability and fitness for a particular purpose;
ii) effectively excludes on behalf of all Contributors all liability for
damages, including direct, indirect, special, incidental and
consequential damages, such as lost profits;
iii) states that any provisions which differ from this Agreement are
offered by that Contributor alone and not by any other party; and
iv) states that source code for the Program is available from such
Contributor, and informs licensees how to obtain it in a reasonable
manner on or through a medium customarily used for software exchange.
When the Program is made available in source code form:
a) it must be made available under this Agreement; and
b) a copy of this Agreement must be included with each copy of the Program.
Contributors may not remove or alter any copyright notices contained
within the Program.
Each Contributor must identify itself as the originator of its Contribution,
if
any, in a manner that reasonably allows subsequent Recipients to identify the
originator of the Contribution.
4. COMMERCIAL DISTRIBUTION
Commercial distributors of software may accept certain responsibilities with
respect to end users, business partners and the like. While this license is
intended to facilitate the commercial use of the Program, the Contributor who
includes the Program in a commercial product offering should do so in a manner
which does not create potential liability for other Contributors. Therefore,
if a Contributor includes the Program in a commercial product offering, such
Contributor ("Commercial Contributor") hereby agrees to defend and indemnify
every other Contributor ("Indemnified Contributor") against any losses,
damages and costs (collectively "Losses") arising from claims, lawsuits and
other legal actions brought by a third party against the Indemnified
Contributor to the extent caused by the acts or omissions of such Commercial
Contributor in connection with its distribution of the Program in a commercial
product offering. The obligations in this section do not apply to any claims
or Losses relating to any actual or alleged intellectual property
infringement. In order to qualify, an Indemnified Contributor must:
a) promptly notify the Commercial Contributor in writing of such claim, and
b) allow the Commercial Contributor to control, and cooperate with the
Commercial Contributor in, the defense and any related settlement
negotiations. The Indemnified Contributor may participate in any such claim at
its own expense.
For example, a Contributor might include the Program in a commercial product
offering, Product X. That Contributor is then a Commercial Contributor. If
that Commercial Contributor then makes performance claims, or offers
warranties related to Product X, those performance claims and warranties are
such Commercial Contributor's responsibility alone. Under this section, the
Commercial Contributor would have to defend claims against the other
Contributors related to those performance claims and warranties, and if a
court requires any other Contributor to pay any damages as a result, the
Commercial Contributor must pay those damages.
5. NO WARRANTY
EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, THE PROGRAM IS PROVIDED ON AN
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR
IMPLIED INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OR CONDITIONS OF TITLE,
NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Each
Recipient is solely responsible for determining the appropriateness of using
and distributing the Program and assumes all risks associated with its
exercise of rights under this Agreement , including but not limited to the
risks and costs of program errors, compliance with applicable laws, damage to
or loss of data, programs or equipment, and unavailability or interruption of
operations.
6. DISCLAIMER OF LIABILITY
EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, NEITHER RECIPIENT NOR ANY
CONTRIBUTORS SHALL HAVE ANY LIABILITY FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING WITHOUT LIMITATION
LOST PROFITS), HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OR DISTRIBUTION OF THE PROGRAM OR THE
EXERCISE OF ANY RIGHTS GRANTED HEREUNDER, EVEN IF ADVISED OF THE POSSIBILITY
OF SUCH DAMAGES.
7. GENERAL
If any provision of this Agreement is invalid or unenforceable under
applicable law, it shall not affect the validity or enforceability of the
remainder of the terms of this Agreement, and without further action by the
parties hereto, such provision shall be reformed to the minimum extent
necessary to make such provision valid and enforceable.
If Recipient institutes patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Program itself
(excluding combinations of the Program with other software or hardware)
infringes such Recipient's patent(s), then such Recipient's rights granted
under Section 2(b) shall terminate as of the date such litigation is filed.
All Recipient's rights under this Agreement shall terminate if it fails to
comply with any of the material terms or conditions of this Agreement and does
not cure such failure in a reasonable period of time after becoming aware of
such noncompliance. If all Recipient's rights under this Agreement terminate,
Recipient agrees to cease use and distribution of the Program as soon as
reasonably practicable. However, Recipient's obligations under this Agreement
and any licenses granted by Recipient relating to the Program shall continue
and survive.
Everyone is permitted to copy and distribute copies of this Agreement, but in
order to avoid inconsistency the Agreement is copyrighted and may only be
modified in the following manner. The Agreement Steward reserves the right to
publish new versions (including revisions) of this Agreement from time to
time. No one other than the Agreement Steward has the right to modify this
Agreement. The Eclipse Foundation is the initial Agreement Steward. The
Eclipse Foundation may assign the responsibility to serve as the Agreement
Steward to a suitable separate entity. Each new version of the Agreement will
be given a distinguishing version number. The Program (including
Contributions) may always be distributed subject to the version of the
Agreement under which it was received. In addition, after a new version of the
Agreement is published, Contributor may elect to distribute the Program
(including its Contributions) under the new version. Except as expressly
stated in Sections 2(a) and 2(b) above, Recipient receives no rights or
licenses to the intellectual property of any Contributor under this Agreement,
whether expressly, by implication, estoppel or otherwise. All rights in the
Program not expressly granted under this Agreement are reserved.
This Agreement is governed by the laws of the State of New York and the
intellectual property laws of the United States of America. No party to this
Agreement will bring a legal action under this Agreement more than one year
after the cause of action arose. Each party waives its rights to a jury trial in
any resulting litigation.
================================================
FILE: README.md
================================================
# core.rrb-vector
Why would anyone want to use this library? The two primary answers
are:
+ You want faster concatenation of vectors, which core.rrb-vector's
`catvec` function provides for both Clojure and ClojureScript.
+ You use vectors of Java primitive types like long, double, etc.,
returned by Clojure's `vector-of` function, e.g. to reduce memory
usage to about 1/3 of the memory required by vectors of arbitrary
objects, and
+ You want the speed enabled by using the transient versions of such
vectors. Clojure does not implement transients for primitive
vectors created via `vector-of` -- core.rrb-vector does.
Vectors are one of the most commonly used data structures within
Clojure. Likely you already know that creating a vector equal to `v`
plus a new element `e` appended to the end using the expression `(conj
v e)` has a run time that is "effectively constant", i.e. it takes
O(log N) time in the size N of `v`, where the base of the logarithm is
32, so it is a constant at most 4 for all vector sizes up to a
million, and at most 7 for all vector sizes that Clojure supports.
The fastest way to concatenate two vectors `v1` and `v2` into a single
new vector is using an expression like `(into v1 v2)`. This is
implemented by repeatedly appending a single element from the second
vector to the first, so it takes linear time in the size of `v2`
(multiplied by the effectively constant time mentioned above).
Aside: There might be another expression that has a better _constant
factor_ for its run time than `(into v1 v2)` does, and is thus faster.
However, any other such expression will still take at least linear
time in the size of the second vector.
The core.rrb-vector library uses a tree structure similar to the one
that Clojure uses internally for vectors, but generalizes it in such a
way that producing a new tree that represents the concatenation of two
input vectors using the `catvec` function can be done in O(log N)
time, where N is the size of the result.
You can give `catvec` vectors created in all of the ways you already
normally do, and while it will return a new type of object, this new
type behaves in all of the ways you expect a Clojure vector to behave.
This new type of vector is indistinguishable from a normal Clojure
vector unless you examine the value of `(type v)` or `(class v)`. In
particular, `(vector? v)` is true for this new type, you can call all
of the usual sequence-based functions on it to examine or process its
elements, you can call `conj` on it, `nth`, etc.
Thus if you have a program where frequently concatenating large
vectors to produce new vectors is useful, core.rrb-vector may help you
write a much faster program in a more natural way.
This library is an implementation of the confluently persistent vector
data structure introduced in the paper "RRB-Trees: Efficient Immutable
Vectors", EPFL-REPORT-169879, September, 2011, by Phil Bagwell and
Tiark Rompf.
RRB-Trees build upon Clojure's internal `PersistentVector` class used
to implement its built in vectors, adding logarithmic time
concatenation and slicing (i.e. create sub-vectors from input
vectors). ClojureScript is supported with the same API, except for
the absence of the `vector-of` function.
The main functions provided are `clojure.core.rrb-vector/catvec`,
performing vector concatenation, and `clojure.core.rrb-vector/subvec`,
which produces a new vector containing the appropriate subrange of the
input vector (in contrast to `clojure.core/subvec`, which returns a
view on the input vector).
Like Clojure vectors, core.rrb-vector vectors can store arbitrary
values, or using `vector-of` you can create vectors restricted to one
primitive type, e.g. long, double, etc. The core.rrb-vector
implementation provides seamless interoperability with the built in
Clojure vectors of class `clojure.lang.PersistentVector`,
`clojure.core.Vec` (vectors of primitive values) and
`clojure.lang.APersistentVector$SubVector` instances:
`clojure.core.rrb-vector/catvec` and `clojure.core.rrb-vector/subvec`
convert their inputs to `clojure.core.rrb_vector.rrbt.Vector`
instances whenever necessary (this is a very fast constant time
operation for PersistentVector and primitive vectors; for SubVector it
is O(log N), where N is the size of the underlying vector).
`clojure.core.rrb-vector` also provides its own versions of `vector`,
`vector-of`, and `vec` that always produce
`clojure.core.rrb_vector.rrbt.Vector` instances. Note that
`vector-of` accepts `:object` as one of the possible type arguments,
in addition to keywords naming primitive types.
## Usage
core.rrb-vector exports one public namespace:
(require '[clojure.core.rrb-vector :as fv])
Note that the ClojureScript version uses the same namespace name (it
*does not* use the alternative `cljs.*` prefix!). This is because the
API is precisely the same (except `clojure.core.rrb-vector/vector-of`
only makes sense on the JVM and is therefore not available in
ClojureScript).
The docstring attached to the namespace provides an overview of the
available functionality (as found at the top of this README):
(doc clojure.core.rrb-vector)
The new functionality is accessible through two functions:
`clojure.core.rrb-vector/subvec`, which provides logarithmic-time
non-view slicing (in contrast to `clojure.core/subvec`, which is a
constant-time operation producing view vectors that prevent the
underlying vector from becoming eligible for garbage collection), and
`clojure.core.rrb-vector/catvec`, which provides logarithmic-time
concatenation. Crucially, these can be applied to regular
Clojure(Script) vectors.
(doc fv/subvec)
(doc fv/catvec)
;; apply catvec and subvec to regular Clojure(Script) vectors
(fv/catvec (vec (range 1234)) (vec (range 8765)))
(fv/subvec (vec (range 1024)) 123 456)
Additionally, several functions for constructing RRB vectors are
provided. There is rarely any reason to use them, since, as mentioned
above, the interesting functions exported by core.rrb-vector work with
regular vectors. Note that `clojure.core.rrb-vector/vec`, in contrast
to `clojure.core/vec`, reuses the internal tree of its input if it
already is a vector (of any type) and does not alias short arrays.
When passed a non-vector argument, it returns an RRB vector.
(doc fv/vector)
(doc fv/vector-of)
(doc fv/vec)
The debug namespace bundled with core.rrb-vector provides several
utilities used by the test suite, as well as a function for
visualizing the internal structure of vectors that works with regular
Clojure(Script) vectors and RRB vectors.
;; for peeking under the hood
(require '[clojure.core.rrb-vector.debug :as dv])
(dv/dbg-vec (fv/catvec (vec (range 1234)) (vec (range 8765))))
## Releases and dependency information
core.rrb-vector requires Clojure >= 1.5.0. View vectors created by
`clojure.core/subvec` are correctly handled for Clojure >= 1.6.0. The
ClojureScript version is regularly tested against the most recent
ClojureScript release.
core.rrb-vector releases are available from Maven Central. Development
snapshots are available from the Sonatype OSS repository.
* [Released versions](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.clojure%22%20AND%20a%3A%22core.rrb-vector%22)
* [Development snapshots](https://oss.sonatype.org/index.html#nexus-search;gav~org.clojure~core.rrb-vector~~~)
* [Change log](CHANGES.md) of changes made in this library.
* Some [benchmark results](doc/benchmarks/benchmarks.md) comparing the run time of core.rrb-vector's JVM implementation against other vector/list implementations on the JVM.
Follow the first link above to discover the current release number.
[CLI/`deps.edn`](https://clojure.org/reference/deps_and_cli) dependency information:
```clojure
org.clojure/core.rrb-vector {:mvn/version "${version}"}
```
[Leiningen](http://leiningen.org/) dependency information:
[org.clojure/core.rrb-vector "${version}"]
[Maven](http://maven.apache.org/) dependency information:
<dependency>
<groupId>org.clojure</groupId>
<artifactId>core.rrb-vector</artifactId>
<version>${version}</version>
</dependency>
[Gradle](http://www.gradle.org/) dependency information:
compile "org.clojure:core.rrb-vector:${version}"
## TODO
1. more tests;
2. performance: general perf tuning, more efficient `catvec`
implementation (to replace current seq-ops-based impl).
## Developer information
core.rrb-vector is being developed as a Clojure Contrib project, see
the
[What is Clojure Contrib](https://clojure.org/dev/contrib_libs)
page for details. Patches will only be accepted from developers who
have signed the Clojure Contributor Agreement.
* [GitHub project](https://github.com/clojure/core.rrb-vector)
* [Bug Tracker](https://clojure.atlassian.net/browse/CRRBV)
* [Continuous Integration](https://github.com/clojure/core.rrb-vector/actions/workflows/test.yml)
### Useful Maven commands
To run Clojure and ClojureScript tests:
```bash
$ mvn -DCLOJURE_VERSION=1.10.1 -Dclojure.version=1.10.1 clean test
```
Clojure versions as old as 1.5.1 can be tested with such a command,
but the ClojureScript tests only work when using Clojure 1.8.0 or
later.
To run tests and, if successful, create a JAR file in the targets
directory:
```bash
$ mvn -DCLOJURE_VERSION=1.10.1 -Dclojure.version=1.10.1 clean package
```
Prerequisites: Only Java and Maven need to be installed. Maven will
download whatever versions of Clojure are needed for the command you
use. Both Clojure and ClojureScript tests are run with the commands
given here. They use the Nashorn JavaScript run time environment
included with Java -- no other JavaScript run time is needed.
### Useful clj CLI commands
To run relatively short Clojure tests, but no ClojureScript tests:
```bash
$ ./script/jdo test
```
To run relatively short ClojureScript tests, but no Clojure tests:
```bash
$ ./script/sdo test
```
Warning: Currently the command above for running ClojureScript tests
does _not_ show warnings from the ClojureScript compiler. I have seen
some ClojureScript compiler warnings appear when running the Maven
command above, and the Leiningen command given below for running
ClojureScript tests, that unfortunately do not appear using
`./script/sdo test`. Suggestions welcome on how to make that command
also show similar warnings.
Replace `test` in the commands above with one of the following for
other useful things:
* `sock` (or no argument at all) - start a REPL, and listen for a
socket REPL connection on TCP port 50505
* `long` - run a longer set of tests
* `coll` - run generative tests from
[`collection-check`](https://github.com/ztellman/collection-check)
library
* `east` - run Eastwood lint tool (clj version only, not cljs)
### Useful Leiningen commands
To run Clojure tests, but no ClojureScript tests:
```bash
$ lein with-profile +1.10 test
```
You can test with Clojure versions 1.5 through 1.10 by specifying that
version number after the `+`.
Prerequisites: Only Java and Leiningen. Leiningen will download
whatever versions of Clojure and other libraries are needed.
To run ClojureScript tests with Node.js and SpiderMonkey JavaScript
runtimes, but no Clojure tests:
```bash
$ lein with-profile +cljs cljsbuild test
```
Add `node` or `spidermonkey` as a separate argument after `test` to
restrict the JavaScript runtime used to only the one you specify. You
may need to adjust the command names in the `:test-commands` section
of the `project.clj` file if the command for running those JavaScript
runtimes have a different name on your system than what is used there.
Prerequisites: Java, Leiningen, and either or both of Node.js and
SpiderMonkey JavaScript run time environments.
To run normal Clojure tests, plus the
[`collection-check`](https://github.com/ztellman/collection-check)
tests, but no ClojureScript tests:
```bash
$ lein with-profile +coll,+1.7 test
```
The `collection-check` tests require Clojure 1.7.0 or later, I believe
because collection-check and/or its dependencies require that.
There is no existing command configured to run `collection-check`
tests with ClojureScript.
To start a REPL from Leiningen with Clojure versions 1.6.0 and older,
you must use Leiningen 2.8.0 (likely some other versions work, too).
### Installing other software you will need
For all of the development commands you must have Java installed.
This includes the ClojureScript compile and test commands, since the
ClojureScript compiler is at least partially written in the Java
version of Clojure.
#### Java
Install one or more of the pre-built binaries from
[AdoptOpenJDK](https://adoptopenjdk.net), or several other providers
of Java binaries.
Additional methods:
* Ubuntu 18.04 Linux: `sudo apt-get install default-jre`
#### Maven
For any `mvn` command you must install
[Maven](https://maven.apache.org).
* Ubuntu 18.04 Linux: `sudo apt-get install maven`
* macOS
* plus Homebrew: `brew install maven`
* plus MacPorts: `sudo port install maven3`, then either use the
command `mvn3`, or to use `mvn` also run the command `sudo port
select --set maven maven3`.
#### Leiningen
An install script and instructions are available on the
[Leiningen](https://leiningen.org) site.
#### Node.js JavaScript run time environment
Installation instructions for many different versions of Node.js are
available on the [Node.js web site](https://nodejs.org). You can also
install it using the commands below.
* Ubuntu 18.04 Linux: `sudo apt-get install nodejs`
* macOS
* plus Homebrew: `brew install node`
* plus MacPorts: `sudo port install nodejs10`. You can see other
versions available via the command `port list | grep nodejs`.
#### SpiderMonkey JavaScript run time environment
Installation instructions for many different versions of SpiderMonkey
are available on the [SpiderMonkey web
site](https://developer.mozilla.org/en-US/docs/Mozilla/Projects/SpiderMonkey).
You may also install it using the commands below.
* Ubuntu 18.04 Linux: `sudo apt-get install libmozjs-52-dev`
* macOS
* plus Homebrew: As of 2019-Sep-24, `brew install spidermonkey`
installs version 1.8.5 of SpiderMonkey, which according to the
Wikipedia page on SpiderMonkey was first released in 2011, with at
least one release per year after that. The ClojureScript tests
fail to run using this version of SpiderMonkey. It seems worth
avoiding this version of SpiderMonkey for the purposes of testing
`core.rrb-vector`.
* plus MacPorts: `sudo port install mozjs52`
## Clojure(Script) code reuse
core.rrb-vector's vectors support the same basic functionality regular
Clojure's vectors do (with the omissions listed above). Where
possible, this is achieved by reusing code from Clojure's gvec and
ClojureScript's PersistentVector implementations. The Clojure(Script)
source files containing the relevant code carry the following
copyright notice:
Copyright (c) Rich Hickey. All rights reserved.
The use and distribution terms for this software are covered by the
Eclipse Public License 1.0 (https://opensource.org/license/epl-1-0/)
which can be found in the file epl-v10.html at the root of this distribution.
By using this software in any fashion, you are agreeing to be bound by
the terms of this license.
You must not remove this notice, or any other, from this software.
## Licence
Copyright © Michał Marczyk, Andy Fingerhut, Rich Hickey and contributors
Distributed under the
[Eclipse Public License 1.0](https://opensource.org/license/epl-1-0/),
the same as Clojure. The licence text can be found in the
`epl-v10.html` file at the root of this distribution.
================================================
FILE: deps.edn
================================================
;; See shell scripts './script/sdo' and './script/jdo' for sample
;; useful combinations of aliases to use to acommplish common tasks.
;; One way to specify a local version of ClojureScript, in case you
;; want to test with modifications to it:
;; org.clojure/clojurescript {:local/root "/Users/jafinger/clj/clojurescript"}
{:paths ["src/main/clojure" "src/main/cljs" "src/main/cljc"]
;;:paths ["src/parameterized/clojure" "src/main/cljs" "src/main/cljc"]
:aliases
{;; Common alias to use for all Clojure/Java commands
:clj {:jvm-opts ["-XX:-OmitStackTraceInFastThrow"]}
;; Common alias to use for all ClojureScript commands
:cljs {:extra-deps {org.clojure/clojurescript {:mvn/version "1.10.520"}}
:jvm-opts ["-XX:-OmitStackTraceInFastThrow"]}
;; - start a Clojure/Java Socket REPL on port 50505
:clj-socket {:jvm-opts ["-Dclojure.server.repl={:port,50505,:accept,clojure.core.server/repl}"]}
;; start a Node-based ClojureScript socket REPL on port 50505
:cljs-socket {:jvm-opts ["-Dclojure.server.repl={:port,50505,:accept,cljs.server.node/repl}"]}
;; Common alias to use for all Clojure/Java commands that run tests
:clj-test {:extra-paths ["src/test/clojure" "src/test/cljc"]
:extra-deps {org.clojure/test.check {:mvn/version "1.1.3"}}}
;; Common alias to use for all ClojureScript commands that run tests
:cljs-test {:extra-paths ["src/test/cljs" "src/test/cljc"]
:extra-deps {org.clojure/test.check {:mvn/version "1.1.3"}}}
;; Run 'short' tests
:clj-runt {:main-opts ["-e" "(require,'[clojure.test,:as,t],'clojure.core.rrb-vector.test-clj-only,'clojure.core.rrb-vector.test-common),(t/run-tests,'clojure.core.rrb-vector.test-common),(t/run-tests,'clojure.core.rrb-vector.test-clj-only)"]}
:cljs-runt {:main-opts ["-m" "cljs.main"
"-re" "node"
"-e" "(require,'[clojure.test,:as,t],'clojure.core.rrb-vector.test-cljs-only,'clojure.core.rrb-vector.test-common),(t/run-tests,'clojure.core.rrb-vector.test-common),(t/run-tests,'clojure.core.rrb-vector.test-cljs-only)"]}
;; Run 'short' tests with extra-checks? enabled
:clj-extrachecks-runt {:main-opts ["-e" "(require,'[clojure.test,:as,t],'clojure.core.rrb-vector.test-clj-only,'clojure.core.rrb-vector.test-common),(alter-var-root,#'clojure.core.rrb-vector.test-utils/extra-checks?,(constantly,true)),(t/run-tests,'clojure.core.rrb-vector.test-common),(t/run-tests,'clojure.core.rrb-vector.test-clj-only)"]}
:cljs-extrachecks-runt {:main-opts ["-m" "cljs.main"
"-re" "node"
"-e" "(require,'[clojure.test,:as,t],'clojure.core.rrb-vector.test-cljs-only,'clojure.core.rrb-vector.test-common),(set!,clojure.core.rrb-vector.test-utils/extra-checks?,true),(t/run-tests,'clojure.core.rrb-vector.test-common),(t/run-tests,'clojure.core.rrb-vector.test-cljs-only)"]}
;; Run generative and/or 'long' tests
:clj-runlongtests {:main-opts ["-e"
"(require,'[clojure.test,:as,t],'clojure.core.rrb-vector.long-test),(t/run-tests,'clojure.core.rrb-vector.long-test)"]}
:cljs-runlongtests {:main-opts ["-m" "cljs.main"
"-re" "node"
"-e" "(require,'[clojure.test,:as,t],'clojure.core.rrb-vector.long-test),(t/run-tests,'clojure.core.rrb-vector.long-test)"]}
;; Using collections-check requires this minimum version of
;; test.check, and at least Clojure 1.7.0
:clj-check {:extra-paths ["src/test_local/clojure"]
:extra-deps {collection-check/collection-check {:mvn/version "0.1.7"}
com.gfredericks/test.chuck {:mvn/version "0.2.10"}
org.clojure/test.check {:mvn/version "1.1.3"}}}
:clj-runcheck {:main-opts ["-e" "(require,'[clojure.test,:as,t],'clojure.core.rrb-vector-check),(t/run-tests,'clojure.core.rrb-vector-check)"]}
:cljs-check {:extra-paths ["src/test_local/clojure"]
:extra-deps {collection-check/collection-check {:mvn/version "0.1.7"}
com.gfredericks/test.chuck {:mvn/version "0.2.10"}
org.clojure/test.check {:mvn/version "1.1.3"}}}
:cljs-runcheck {:main-opts ["-m" "cljs.main"
"-re" "node"
"-e" "(require,'[clojure.test,:as,t],'clojure.core.rrb-vector-check),(t/run-tests,'clojure.core.rrb-vector-check)"]}
;; Run performance tests
:clj-runperf {:main-opts ["-e" "(require,'[clojure.test,:as,t],'clojure.core.rrb-vector-performance-test),(t/run-tests,'clojure.core.rrb-vector-performance-test)"]}
:cljs-runperf {:main-opts ["-m" "cljs.main"
"-re" "node"
"-e" "(require,'[clojure.test,:as,t],'clojure.core.rrb-vector-performance-test),(t/run-tests,'clojure.core.rrb-vector-performance-test)"]}
;; Run whatever the current 'focus' tests are
:clj-runfocus {:main-opts ["-e" "(require,'[clojure.test,:as,t],'[clojure.core.rrb-vector.test-common,:as,ct]),(ct/test-reduce-subvec-catvec2)"]}
:cljs-runfocus {:main-opts ["-m" "cljs.main"
"-re" "node"
"-e" "(require,'[clojure.test,:as,t],'[clojure.core.rrb-vector.test-common,:as,ct]),(ct/test-reduce-subvec-catvec2)"]}
;; I have tried using cljs-test-runner for running clojure.test
;; tests in a modified version of core.rrb-vector, but my guess is
;; that since an older version of core.rrb-vector (version 0.0.11)
;; is in the transitive dependencies of the cljs-test-runner project
;; itself, that version conflicts with the version I am attempting
;; to test. See
;; https://github.com/Olical/cljs-test-runner/issues/34
;; :cljs-runner {:extra-deps {olical/cljs-test-runner {:mvn/version "3.7.0"}}
;; :main-opts ["-m" "cljs-test-runner.main"
;; "-d" "src/test/cljs"]}
;; :cljol {:extra-deps {cljol {:local/root "/Users/andy/clj/cljol"}
;; org.clojure/clojure {:mvn/version "1.7.0"}}}
:cljol {:extra-deps {cljol/cljol {:git/url "https://github.com/jafingerhut/cljol"
:sha "bb5549e9832e73e4a9fc5dfdf695c48e797729fa"}}}
:cap {;; recommended options from README of
;; https://github.com/clojure-goes-fast/clj-async-profiler
:jvm-opts ["-Djdk.attach.allowAttachSelf"
;; I have trouble entering password for this from
;; clj REPL. Maybe clojure command instead of clj
;; is better for this?
"-Djol.tryWithSudo=true"
"-XX:+UnlockDiagnosticVMOptions"
"-XX:+DebugNonSafepoints"]
:extra-deps {com.clojure-goes-fast/clj-async-profiler
{:mvn/version "1.6.2"}}}
:nodis {:extra-deps {com.clojure-goes-fast/clj-java-decompiler
{:mvn/version "0.3.7"}}}
:eastwood {:extra-deps {jonase/eastwood {:mvn/version "0.3.5"}}
:main-opts ["-m" "eastwood.lint"
"{:source-paths,[\"src/main/clojure\"],:test-paths,[\"src/test/clojure\",\"src/test/cljs\",\"src/test_local/clojure\"],:add-linters,[:unused-fn-args,:unused-locals,:unused-namespaces,:unused-private-vars],:exclude-linters,[:implicit-dependencies],:exclude-namespaces,[]}"]}
:clj-kondo {:extra-deps {clj-kondo/clj-kondo {:mvn/version "RELEASE"}}
:main-opts ["-m" "clj-kondo.main"]}
;; pull in specific versions of Clojure:
:1.5 {:override-deps {org.clojure/clojure {:mvn/version "1.5.1"}}}
:1.6 {:override-deps {org.clojure/clojure {:mvn/version "1.6.0"}}}
:1.7 {:override-deps {org.clojure/clojure {:mvn/version "1.7.0"}}}
:1.8 {:override-deps {org.clojure/clojure {:mvn/version "1.8.0"}}}
:1.9 {:override-deps {org.clojure/clojure {:mvn/version "1.9.0"}}}
:1.10.0 {:override-deps {org.clojure/clojure {:mvn/version "1.10.0"}}}
:1.10 {:override-deps {org.clojure/clojure {:mvn/version "1.10.1"}}}
:master {:override-deps {org.clojure/clojure {:mvn/version "1.11.0-master-SNAPSHOT"}}}}}
================================================
FILE: doc/benchmarks/benchmarks.md
================================================
# core.rrb-vector benchmarks
See the section "How the benchmarks were run" below for how the
results were created.
## Benchmark results
These benchmark results are based upon benchmark code written for the
[`bifurcan`](https://github.com/lacuna/bifurcan) library. The
[benchmarks published originally
here](https://github.com/lacuna/bifurcan/blob/master/doc/comparison.md)
include comparisons of data structures other than vectors, e.g. hash
maps and sorted sets.
Note that for the data structures we care most about here, those
results call them "lists" rather than "vectors". I will use "lists"
here to be consistent with the `bifurcan` benchmarks. The portion of
the `bifurcan` results relevant to lists can be found
[here](https://github.com/lacuna/bifurcan/blob/master/doc/comparison.md#lists).
The time measurements here were made on a different machine than the
`bifurcan` results, so you should not infer anything from the absolute
time measurements here vs. there. Only the relative time measurements
between the libraries in one set of benchmark results.
---
As in the original benchmark results, construction times are quite
consistent (at the scale shown in this graph) across different
libraries, except for vavr. The graph below shows the results for all
libraries except vavr, so that any differences between them become
more apparent. Even at that finer scale the maximum differences above
100 element vectors appears to be at most 3x.
---
Without core.rrb-vector included, Zach Tellman's original comment on
these results was "The mutable collections, which are stored
contiguously, are only moderately faster than their immutable
counterparts".
Note that the benchmark here creates a Java
[`Iterator`](https://docs.oracle.com/javase/8/docs/api/java/util/Iterator.html)
object from the collection, and then in a loop uses that interface's
`hasNext` and `next` methods to traverse the entire collection until
`hasNext` returns false. The more typical way to traverse collections
in Clojure is to call `seq` on them (or use one of many functions that
call `seq` for you under the covers), and then use `rest` or `next` to
advance one step.
core.rrb-vector version 0.1.0 is significantly slower, primarily
because it has less optimized code for its Java `Iterator`
implementation than it has for its `seq`/`rest`/`next` implementation.
The reason for the increases near 1K elements and 32K elements are
most likely because those are the sizes where the 32-way tree data
structure gets one level deeper.
The graph below shows the results for all except the core.rrb-vector
library, so any differences between those libraries can be seen.
Several of them show at least some change in run time when crossing
the 1K and 32K element points.
---
Zach's comment on the original results, which are nearly identical to
those shown above: "The mutable collections are O(1) while their
immutable counterparts are unmistakably O(log N)."
---
Zach's comment on the original results: "Concatenation is O(N) for
every library except Paguro and Bifurcan, which are O(log N) due to
their use of RRB trees."
I know for a fact that clojure.PersistentVector does not use RRB
trees, and concatenation takes at least linear time (multiplied by
some small log factor) in the length of the second vector. This
appears to be the case for vavr.Vector and java.ArrayList as well.
I would have guessed that scala.Vector used RRB trees as well, but
have not checked the implementation yet to verify. If it does use RRB
trees, it is by far the slowest of the ones that do, at least for
vectors 100K in size and larger -- perhaps the scala.Vector authors
chose to implement some more extensive tree rebalancing than other RRB
implementers did, in order to preserve faster run times for other
operations?
The next graph shows the results with only the 4 libraries that are
the fastest for concatenation. Unlike the previous graphs, the
vertical axis is the elapsed time, not "elapsed time per vector
element". RRB trees should enable O(log N) run time for
concatenation.
core.rrb-vector is the slowest of these by a large factor, probably
because of a concatenation implementation that has not been
scrutinized for optimization opportunities yet.
The next graph shows only the 3 libraries that are the fastest for
concatenation, to see any detail that might be of interest there.
Like the previous one, the vertical axis is elapsed time, not elapsed
time per vector element.
TBD: How can bifurcan.LinearList have pretty much a constant run time
for all vector sizes?
---
## How the benchmarks were run
To run benchmarks from the Bifurcan project, with small modifications
that add core.rrb-vector to the list of libraries that are measured,
follow these steps. Note that the version of the `bifurcan.List` data
structure code used in these results has a few proposed bug fixes from
the version of the code in the original `bifurcan` repository, but I
do not believe they affect the performance in any noticeable way.
```bash
$ git clone https://github.com/jafingerhut/bifurcan
$ cd bifurcan
$ git checkout 457fd0346b78392f39e4c0e79f1e43b7847ea93b
$ ./benchmarks/run-vectors-only.sh
```
To copy the data and images produced as a result of the above, to
where I copied them in this repository:
```bash
$ DST=/path/to/my/clone/of/core.rrb-vector/doc/benchmarks
$ mkdir $DST/images $DST/data
$ cp -p benchmarks/images/list*.png benchmarks/images/concat*.png $DST/images
$ cp -p benchmarks/data/benchmarks.edn benchmarks/data/concat.csv benchmarks/data/list*.csv $DST/data
```
The benchmark results here were measured on a system with these
properties:
* MacBook Pro model 11,2, 2.5 GHz Intel Core i7 with peak clock speed
3.6 GHz, 16 GB RAM
* macOS 10.14.6
* AdoptOpenJDK 11.0.4, 64-bit server build 11.0.4+11
* Leiningen 2.9.1
* To see the versions of the list libraries that were measured, look
in the project.clj file of the bifurcan project at the commit
mentioned above. For core.rrb-vector, the only measured library
that is written in Clojure, that project.clj file currently
specifies Clojure version 1.8.0. The other libraries are written in
Java.
================================================
FILE: doc/benchmarks/data/benchmarks.edn
================================================
{"bifurcan.List" {100000 {:intersection nil, :remove nil, :insert nil, :lookup 1746262, :concat 297, :equals nil, :iteration 167543, :difference nil, :clone nil, :union nil, :construct 683919}, 177827 {:intersection nil, :remove nil, :insert nil, :lookup 3062911, :concat 333, :equals nil, :iteration 297203, :difference nil, :clone nil, :union nil, :construct 1292069}, 1000000 {:intersection nil, :remove nil, :insert nil, :lookup 33788032, :concat 481, :equals nil, :iteration 1745738, :difference nil, :clone nil, :union nil, :construct 7310803}, 316227 {:intersection nil, :remove nil, :insert nil, :lookup 7754499, :concat 364, :equals nil, :iteration 526319, :difference nil, :clone nil, :union nil, :construct 2348309}, 1778 {:intersection nil, :remove nil, :insert nil, :lookup 13720, :concat 1466, :equals nil, :iteration 2558, :difference nil, :clone nil, :union nil, :construct 12905}, 31 {:intersection nil, :remove nil, :insert nil, :lookup 46, :concat 139, :equals nil, :iteration 28, :difference nil, :clone nil, :union nil, :construct 188}, 10000 {:intersection nil, :remove nil, :insert nil, :lookup 84389, :concat 447, :equals nil, :iteration 14238, :difference nil, :clone nil, :union nil, :construct 72806}, 1000 {:intersection nil, :remove nil, :insert nil, :lookup 3614, :concat 639, :equals nil, :iteration 2204, :difference nil, :clone nil, :union nil, :construct 7284}, 56 {:intersection nil, :remove nil, :insert nil, :lookup 171, :concat 144, :equals nil, :iteration 85, :difference nil, :clone nil, :union nil, :construct 450}, 562341 {:intersection nil, :remove nil, :insert nil, :lookup 16662099, :concat 480, :equals nil, :iteration 941730, :difference nil, :clone nil, :union nil, :construct 4357028}, 316 {:intersection nil, :remove nil, :insert nil, :lookup 1108, :concat 395, :equals nil, :iteration 697, :difference nil, :clone nil, :union nil, :construct 2356}, 100 {:intersection nil, :remove nil, :insert nil, :lookup 363, :concat 241, :equals nil, :iteration 218, :difference nil, :clone nil, :union nil, :construct 791}, 3162 {:intersection nil, :remove nil, :insert nil, :lookup 24404, :concat 347, :equals nil, :iteration 4462, :difference nil, :clone nil, :union nil, :construct 22781}, 56234 {:intersection nil, :remove nil, :insert nil, :lookup 903863, :concat 1475, :equals nil, :iteration 93320, :difference nil, :clone nil, :union nil, :construct 416354}, 17 {:intersection nil, :remove nil, :insert nil, :lookup 27, :concat 139, :equals nil, :iteration 19, :difference nil, :clone nil, :union nil, :construct 154}, 17782 {:intersection nil, :remove nil, :insert nil, :lookup 181189, :concat 625, :equals nil, :iteration 25257, :difference nil, :clone nil, :union nil, :construct 130015}, 562 {:intersection nil, :remove nil, :insert nil, :lookup 2028, :concat 546, :equals nil, :iteration 1241, :difference nil, :clone nil, :union nil, :construct 4229}, 5623 {:intersection nil, :remove nil, :insert nil, :lookup 43274, :concat 411, :equals nil, :iteration 7946, :difference nil, :clone nil, :union nil, :construct 41536}, 10 {:intersection nil, :remove nil, :insert nil, :lookup 17, :concat 136, :equals nil, :iteration 14, :difference nil, :clone nil, :union nil, :construct 94}, 31622 {:intersection nil, :remove nil, :insert nil, :lookup 356039, :concat 626, :equals nil, :iteration 44706, :difference nil, :clone nil, :union nil, :construct 229882}, 177 {:intersection nil, :remove nil, :insert nil, :lookup 615, :concat 314, :equals nil, :iteration 393, :difference nil, :clone nil, :union nil, :construct 1403}}, "java.ArrayList" {100000 {:intersection nil, :remove nil, :insert nil, :lookup 232778, :concat 224121, :equals nil, :iteration 56148, :difference nil, :clone nil, :union nil, :construct 666432}, 177827 {:intersection nil, :remove nil, :insert nil, :lookup 269608, :concat 412139, :equals nil, :iteration 99187, :difference nil, :clone nil, :union nil, :construct 1331189}, 1000000 {:intersection nil, :remove nil, :insert nil, :lookup 1873249, :concat 5719794, :equals nil, :iteration 562946, :difference nil, :clone nil, :union nil, :construct 14379088}, 316227 {:intersection nil, :remove nil, :insert nil, :lookup 455997, :concat 1924960, :equals nil, :iteration 176352, :difference nil, :clone nil, :union nil, :construct 3804737}, 1778 {:intersection nil, :remove nil, :insert nil, :lookup 2220, :concat 2900, :equals nil, :iteration 1001, :difference nil, :clone nil, :union nil, :construct 11630}, 31 {:intersection nil, :remove nil, :insert nil, :lookup 39, :concat 108, :equals nil, :iteration 31, :difference nil, :clone nil, :union nil, :construct 206}, 10000 {:intersection nil, :remove nil, :insert nil, :lookup 15085, :concat 18509, :equals nil, :iteration 5572, :difference nil, :clone nil, :union nil, :construct 84652}, 1000 {:intersection nil, :remove nil, :insert nil, :lookup 1236, :concat 1767, :equals nil, :iteration 582, :difference nil, :clone nil, :union nil, :construct 7784}, 56 {:intersection nil, :remove nil, :insert nil, :lookup 74, :concat 157, :equals nil, :iteration 49, :difference nil, :clone nil, :union nil, :construct 396}, 562341 {:intersection nil, :remove nil, :insert nil, :lookup 3617188, :concat 4255442, :equals nil, :iteration 312946, :difference nil, :clone nil, :union nil, :construct 9850749}, 316 {:intersection nil, :remove nil, :insert nil, :lookup 394, :concat 685, :equals nil, :iteration 206, :difference nil, :clone nil, :union nil, :construct 2525}, 100 {:intersection nil, :remove nil, :insert nil, :lookup 127, :concat 231, :equals nil, :iteration 79, :difference nil, :clone nil, :union nil, :construct 662}, 3162 {:intersection nil, :remove nil, :insert nil, :lookup 3937, :concat 5415, :equals nil, :iteration 1756, :difference nil, :clone nil, :union nil, :construct 21032}, 56234 {:intersection nil, :remove nil, :insert nil, :lookup 129834, :concat 122657, :equals nil, :iteration 31247, :difference nil, :clone nil, :union nil, :construct 382130}, 17 {:intersection nil, :remove nil, :insert nil, :lookup 22, :concat 137, :equals nil, :iteration 21, :difference nil, :clone nil, :union nil, :construct 132}, 17782 {:intersection nil, :remove nil, :insert nil, :lookup 38404, :concat 34667, :equals nil, :iteration 9918, :difference nil, :clone nil, :union nil, :construct 115303}, 562 {:intersection nil, :remove nil, :insert nil, :lookup 698, :concat 1040, :equals nil, :iteration 340, :difference nil, :clone nil, :union nil, :construct 5249}, 5623 {:intersection nil, :remove nil, :insert nil, :lookup 7047, :concat 9791, :equals nil, :iteration 3131, :difference nil, :clone nil, :union nil, :construct 36137}, 10 {:intersection nil, :remove nil, :insert nil, :lookup 14, :concat 114, :equals nil, :iteration 16, :difference nil, :clone nil, :union nil, :construct 60}, 31622 {:intersection nil, :remove nil, :insert nil, :lookup 69391, :concat 72656, :equals nil, :iteration 17507, :difference nil, :clone nil, :union nil, :construct 281678}, 177 {:intersection nil, :remove nil, :insert nil, :lookup 222, :concat 384, :equals nil, :iteration 129, :difference nil, :clone nil, :union nil, :construct 1567}}, "clojure.PersistentVector" {100000 {:intersection nil, :remove nil, :insert nil, :lookup 2228634, :concat 1779133, :equals nil, :iteration 191083, :difference nil, :clone nil, :union nil, :construct 1561274}, 177827 {:intersection nil, :remove nil, :insert nil, :lookup 4423742, :concat 3174520, :equals nil, :iteration 339029, :difference nil, :clone nil, :union nil, :construct 2788184}, 1000000 {:intersection nil, :remove nil, :insert nil, :lookup 65538555, :concat 18577365, :equals nil, :iteration 1935245, :difference nil, :clone nil, :union nil, :construct 15673957}, 316227 {:intersection nil, :remove nil, :insert nil, :lookup 15758293, :concat 5668410, :equals nil, :iteration 602441, :difference nil, :clone nil, :union nil, :construct 4990200}, 1778 {:intersection nil, :remove nil, :insert nil, :lookup 16304, :concat 31391, :equals nil, :iteration 3260, :difference nil, :clone nil, :union nil, :construct 25188}, 31 {:intersection nil, :remove nil, :insert nil, :lookup 146, :concat 978, :equals nil, :iteration 35, :difference nil, :clone nil, :union nil, :construct 561}, 10000 {:intersection nil, :remove nil, :insert nil, :lookup 114617, :concat 175732, :equals nil, :iteration 18315, :difference nil, :clone nil, :union nil, :construct 154981}, 1000 {:intersection nil, :remove nil, :insert nil, :lookup 7500, :concat 17906, :equals nil, :iteration 1779, :difference nil, :clone nil, :union nil, :construct 14172}, 56 {:intersection nil, :remove nil, :insert nil, :lookup 357, :concat 1536, :equals nil, :iteration 99, :difference nil, :clone nil, :union nil, :construct 900}, 562341 {:intersection nil, :remove nil, :insert nil, :lookup 22863239, :concat 10219007, :equals nil, :iteration 1071436, :difference nil, :clone nil, :union nil, :construct 8827463}, 316 {:intersection nil, :remove nil, :insert nil, :lookup 2479, :concat 6085, :equals nil, :iteration 568, :difference nil, :clone nil, :union nil, :construct 4536}, 100 {:intersection nil, :remove nil, :insert nil, :lookup 768, :concat 2335, :equals nil, :iteration 181, :difference nil, :clone nil, :union nil, :construct 1530}, 3162 {:intersection nil, :remove nil, :insert nil, :lookup 29530, :concat 56268, :equals nil, :iteration 5805, :difference nil, :clone nil, :union nil, :construct 49063}, 56234 {:intersection nil, :remove nil, :insert nil, :lookup 1663501, :concat 990805, :equals nil, :iteration 106731, :difference nil, :clone nil, :union nil, :construct 872348}, 17 {:intersection nil, :remove nil, :insert nil, :lookup 81, :concat 752, :equals nil, :iteration 22, :difference nil, :clone nil, :union nil, :construct 384}, 17782 {:intersection nil, :remove nil, :insert nil, :lookup 236158, :concat 310298, :equals nil, :iteration 32369, :difference nil, :clone nil, :union nil, :construct 275176}, 562 {:intersection nil, :remove nil, :insert nil, :lookup 4294, :concat 10456, :equals nil, :iteration 1002, :difference nil, :clone nil, :union nil, :construct 7998}, 5623 {:intersection nil, :remove nil, :insert nil, :lookup 54315, :concat 98658, :equals nil, :iteration 10284, :difference nil, :clone nil, :union nil, :construct 87357}, 10 {:intersection nil, :remove nil, :insert nil, :lookup 48, :concat 625, :equals nil, :iteration 17, :difference nil, :clone nil, :union nil, :construct 249}, 31622 {:intersection nil, :remove nil, :insert nil, :lookup 483401, :concat 549372, :equals nil, :iteration 57762, :difference nil, :clone nil, :union nil, :construct 490170}, 177 {:intersection nil, :remove nil, :insert nil, :lookup 1301, :concat 3748, :equals nil, :iteration 327, :difference nil, :clone nil, :union nil, :construct 2608}}, "vavr.Vector" {100000 {:intersection nil, :remove nil, :insert nil, :lookup 1032512, :concat 442338, :equals nil, :iteration 183461, :difference nil, :clone nil, :union nil, :construct 19086024}, 177827 {:intersection nil, :remove nil, :insert nil, :lookup 2412248, :concat 788452, :equals nil, :iteration 324402, :difference nil, :clone nil, :union nil, :construct 35034903}, 1000000 {:intersection nil, :remove nil, :insert nil, :lookup 35926792, :concat 4775235, :equals nil, :iteration 1891016, :difference nil, :clone nil, :union nil, :construct 203992824}, 316227 {:intersection nil, :remove nil, :insert nil, :lookup 4897287, :concat 1423772, :equals nil, :iteration 579257, :difference nil, :clone nil, :union nil, :construct 64204535}, 1778 {:intersection nil, :remove nil, :insert nil, :lookup 10737, :concat 6417, :equals nil, :iteration 3165, :difference nil, :clone nil, :union nil, :construct 324766}, 31 {:intersection nil, :remove nil, :insert nil, :lookup 46, :concat 210, :equals nil, :iteration 36, :difference nil, :clone nil, :union nil, :construct 4205}, 10000 {:intersection nil, :remove nil, :insert nil, :lookup 69566, :concat 45135, :equals nil, :iteration 17793, :difference nil, :clone nil, :union nil, :construct 1795069}, 1000 {:intersection nil, :remove nil, :insert nil, :lookup 3024, :concat 3542, :equals nil, :iteration 1909, :difference nil, :clone nil, :union nil, :construct 180283}, 56 {:intersection nil, :remove nil, :insert nil, :lookup 175, :concat 306, :equals nil, :iteration 103, :difference nil, :clone nil, :union nil, :construct 8508}, 562341 {:intersection nil, :remove nil, :insert nil, :lookup 13748592, :concat 2536907, :equals nil, :iteration 1038763, :difference nil, :clone nil, :union nil, :construct 109838874}, 316 {:intersection nil, :remove nil, :insert nil, :lookup 930, :concat 1218, :equals nil, :iteration 613, :difference nil, :clone nil, :union nil, :construct 53085}, 100 {:intersection nil, :remove nil, :insert nil, :lookup 305, :concat 572, :equals nil, :iteration 192, :difference nil, :clone nil, :union nil, :construct 17031}, 3162 {:intersection nil, :remove nil, :insert nil, :lookup 19182, :concat 14152, :equals nil, :iteration 5600, :difference nil, :clone nil, :union nil, :construct 604280}, 56234 {:intersection nil, :remove nil, :insert nil, :lookup 491455, :concat 244803, :equals nil, :iteration 103257, :difference nil, :clone nil, :union nil, :construct 11700143}, 17 {:intersection nil, :remove nil, :insert nil, :lookup 27, :concat 170, :equals nil, :iteration 24, :difference nil, :clone nil, :union nil, :construct 2139}, 17782 {:intersection nil, :remove nil, :insert nil, :lookup 152997, :concat 73002, :equals nil, :iteration 31425, :difference nil, :clone nil, :union nil, :construct 3724367}, 562 {:intersection nil, :remove nil, :insert nil, :lookup 1613, :concat 2223, :equals nil, :iteration 1056, :difference nil, :clone nil, :union nil, :construct 97549}, 5623 {:intersection nil, :remove nil, :insert nil, :lookup 34617, :concat 23687, :equals nil, :iteration 9969, :difference nil, :clone nil, :union nil, :construct 1102982}, 10 {:intersection nil, :remove nil, :insert nil, :lookup 18, :concat 150, :equals nil, :iteration 18, :difference nil, :clone nil, :union nil, :construct 1228}, 31622 {:intersection nil, :remove nil, :insert nil, :lookup 300171, :concat 138379, :equals nil, :iteration 55955, :difference nil, :clone nil, :union nil, :construct 6935015}, 177 {:intersection nil, :remove nil, :insert nil, :lookup 515, :concat 854, :equals nil, :iteration 345, :difference nil, :clone nil, :union nil, :construct 28413}}, "scala.Vector" {100000 {:intersection nil, :remove nil, :insert nil, :lookup 1443229, :concat 128174, :equals nil, :iteration 207209, :difference nil, :clone nil, :union nil, :construct 446145}, 177827 {:intersection nil, :remove nil, :insert nil, :lookup 1859958, :concat 227317, :equals nil, :iteration 366050, :difference nil, :clone nil, :union nil, :construct 797333}, 1000000 {:intersection nil, :remove nil, :insert nil, :lookup 18855904, :concat 936365, :equals nil, :iteration 2107386, :difference nil, :clone nil, :union nil, :construct 4582003}, 316227 {:intersection nil, :remove nil, :insert nil, :lookup 4385075, :concat 406149, :equals nil, :iteration 654705, :difference nil, :clone nil, :union nil, :construct 1400461}, 1778 {:intersection nil, :remove nil, :insert nil, :lookup 8606, :concat 2516, :equals nil, :iteration 3256, :difference nil, :clone nil, :union nil, :construct 9005}, 31 {:intersection nil, :remove nil, :insert nil, :lookup 54, :concat 168, :equals nil, :iteration 71, :difference nil, :clone nil, :union nil, :construct 154}, 10000 {:intersection nil, :remove nil, :insert nil, :lookup 69497, :concat 12487, :equals nil, :iteration 17861, :difference nil, :clone nil, :union nil, :construct 49662}, 1000 {:intersection nil, :remove nil, :insert nil, :lookup 2936, :concat 1546, :equals nil, :iteration 1794, :difference nil, :clone nil, :union nil, :construct 5072}, 56 {:intersection nil, :remove nil, :insert nil, :lookup 132, :concat 281, :equals nil, :iteration 115, :difference nil, :clone nil, :union nil, :construct 325}, 562341 {:intersection nil, :remove nil, :insert nil, :lookup 11807347, :concat 713960, :equals nil, :iteration 1159375, :difference nil, :clone nil, :union nil, :construct 2554273}, 316 {:intersection nil, :remove nil, :insert nil, :lookup 861, :concat 738, :equals nil, :iteration 593, :difference nil, :clone nil, :union nil, :construct 1679}, 100 {:intersection nil, :remove nil, :insert nil, :lookup 256, :concat 518, :equals nil, :iteration 213, :difference nil, :clone nil, :union nil, :construct 565}, 3162 {:intersection nil, :remove nil, :insert nil, :lookup 18950, :concat 4940, :equals nil, :iteration 5670, :difference nil, :clone nil, :union nil, :construct 15888}, 56234 {:intersection nil, :remove nil, :insert nil, :lookup 763569, :concat 70672, :equals nil, :iteration 116749, :difference nil, :clone nil, :union nil, :construct 249868}, 17 {:intersection nil, :remove nil, :insert nil, :lookup 31, :concat 170, :equals nil, :iteration 53, :difference nil, :clone nil, :union nil, :construct 118}, 17782 {:intersection nil, :remove nil, :insert nil, :lookup 156912, :concat 25782, :equals nil, :iteration 31478, :difference nil, :clone nil, :union nil, :construct 88109}, 562 {:intersection nil, :remove nil, :insert nil, :lookup 1618, :concat 1085, :equals nil, :iteration 1028, :difference nil, :clone nil, :union nil, :construct 2918}, 5623 {:intersection nil, :remove nil, :insert nil, :lookup 35728, :concat 8230, :equals nil, :iteration 10070, :difference nil, :clone nil, :union nil, :construct 27946}, 10 {:intersection nil, :remove nil, :insert nil, :lookup 19, :concat 166, :equals nil, :iteration 36, :difference nil, :clone nil, :union nil, :construct 98}, 31622 {:intersection nil, :remove nil, :insert nil, :lookup 304412, :concat 45674, :equals nil, :iteration 55700, :difference nil, :clone nil, :union nil, :construct 150112}, 177 {:intersection nil, :remove nil, :insert nil, :lookup 465, :concat 586, :equals nil, :iteration 355, :difference nil, :clone nil, :union nil, :construct 934}}, "paguro.RrbTree" {100000 {:intersection nil, :remove nil, :insert nil, :lookup 2279200, :concat 898, :equals nil, :iteration 214485, :difference nil, :clone nil, :union nil, :construct 1222418}, 177827 {:intersection nil, :remove nil, :insert nil, :lookup 3902249, :concat 922, :equals nil, :iteration 387376, :difference nil, :clone nil, :union nil, :construct 2193137}, 1000000 {:intersection nil, :remove nil, :insert nil, :lookup 45184233, :concat 507, :equals nil, :iteration 2320294, :difference nil, :clone nil, :union nil, :construct 12813206}, 316227 {:intersection nil, :remove nil, :insert nil, :lookup 10253465, :concat 803, :equals nil, :iteration 701969, :difference nil, :clone nil, :union nil, :construct 4102486}, 1778 {:intersection nil, :remove nil, :insert nil, :lookup 10448, :concat 420, :equals nil, :iteration 3737, :difference nil, :clone nil, :union nil, :construct 18476}, 31 {:intersection nil, :remove nil, :insert nil, :lookup 47, :concat 353, :equals nil, :iteration 31, :difference nil, :clone nil, :union nil, :construct 217}, 10000 {:intersection nil, :remove nil, :insert nil, :lookup 73155, :concat 655, :equals nil, :iteration 21406, :difference nil, :clone nil, :union nil, :construct 112118}, 1000 {:intersection nil, :remove nil, :insert nil, :lookup 3876, :concat 727, :equals nil, :iteration 2241, :difference nil, :clone nil, :union nil, :construct 11107}, 56 {:intersection nil, :remove nil, :insert nil, :lookup 85, :concat 788, :equals nil, :iteration 294, :difference nil, :clone nil, :union nil, :construct 558}, 562341 {:intersection nil, :remove nil, :insert nil, :lookup 46099574, :concat 1053, :equals nil, :iteration 1259165, :difference nil, :clone nil, :union nil, :construct 7316685}, 316 {:intersection nil, :remove nil, :insert nil, :lookup 1341, :concat 271, :equals nil, :iteration 763, :difference nil, :clone nil, :union nil, :construct 3330}, 100 {:intersection nil, :remove nil, :insert nil, :lookup 414, :concat 398, :equals nil, :iteration 367, :difference nil, :clone nil, :union nil, :construct 922}, 3162 {:intersection nil, :remove nil, :insert nil, :lookup 19158, :concat 902, :equals nil, :iteration 6550, :difference nil, :clone nil, :union nil, :construct 34028}, 56234 {:intersection nil, :remove nil, :insert nil, :lookup 1173421, :concat 679, :equals nil, :iteration 119708, :difference nil, :clone nil, :union nil, :construct 668771}, 17 {:intersection nil, :remove nil, :insert nil, :lookup 28, :concat 288, :equals nil, :iteration 22, :difference nil, :clone nil, :union nil, :construct 132}, 17782 {:intersection nil, :remove nil, :insert nil, :lookup 164131, :concat 649, :equals nil, :iteration 37101, :difference nil, :clone nil, :union nil, :construct 201296}, 562 {:intersection nil, :remove nil, :insert nil, :lookup 2269, :concat 342, :equals nil, :iteration 1358, :difference nil, :clone nil, :union nil, :construct 5538}, 5623 {:intersection nil, :remove nil, :insert nil, :lookup 34842, :concat 653, :equals nil, :iteration 11716, :difference nil, :clone nil, :union nil, :construct 62541}, 10 {:intersection nil, :remove nil, :insert nil, :lookup 19, :concat 265, :equals nil, :iteration 18, :difference nil, :clone nil, :union nil, :construct 118}, 31622 {:intersection nil, :remove nil, :insert nil, :lookup 328633, :concat 816, :equals nil, :iteration 66726, :difference nil, :clone nil, :union nil, :construct 363202}, 177 {:intersection nil, :remove nil, :insert nil, :lookup 747, :concat 241, :equals nil, :iteration 548, :difference nil, :clone nil, :union nil, :construct 1677}}, "bifurcan.LinearList" {100000 {:intersection nil, :remove nil, :insert nil, :lookup 253203, :concat 132, :equals nil, :iteration 69761, :difference nil, :clone nil, :union nil, :construct 463192}, 177827 {:intersection nil, :remove nil, :insert nil, :lookup 284300, :concat 129, :equals nil, :iteration 123582, :difference nil, :clone nil, :union nil, :construct 1670219}, 1000000 {:intersection nil, :remove nil, :insert nil, :lookup 1975906, :concat 135, :equals nil, :iteration 709329, :difference nil, :clone nil, :union nil, :construct 9882679}, 316227 {:intersection nil, :remove nil, :insert nil, :lookup 1869626, :concat 130, :equals nil, :iteration 219949, :difference nil, :clone nil, :union nil, :construct 4434546}, 1778 {:intersection nil, :remove nil, :insert nil, :lookup 2415, :concat 190, :equals nil, :iteration 1254, :difference nil, :clone nil, :union nil, :construct 9076}, 31 {:intersection nil, :remove nil, :insert nil, :lookup 45, :concat 187, :equals nil, :iteration 35, :difference nil, :clone nil, :union nil, :construct 189}, 10000 {:intersection nil, :remove nil, :insert nil, :lookup 16400, :concat 189, :equals nil, :iteration 6930, :difference nil, :clone nil, :union nil, :construct 59852}, 1000 {:intersection nil, :remove nil, :insert nil, :lookup 1257, :concat 187, :equals nil, :iteration 729, :difference nil, :clone nil, :union nil, :construct 4923}, 56 {:intersection nil, :remove nil, :insert nil, :lookup 81, :concat 189, :equals nil, :iteration 58, :difference nil, :clone nil, :union nil, :construct 348}, 562341 {:intersection nil, :remove nil, :insert nil, :lookup 888722, :concat 130, :equals nil, :iteration 415113, :difference nil, :clone nil, :union nil, :construct 9194265}, 316 {:intersection nil, :remove nil, :insert nil, :lookup 402, :concat 188, :equals nil, :iteration 257, :difference nil, :clone nil, :union nil, :construct 1938}, 100 {:intersection nil, :remove nil, :insert nil, :lookup 135, :concat 189, :equals nil, :iteration 97, :difference nil, :clone nil, :union nil, :construct 603}, 3162 {:intersection nil, :remove nil, :insert nil, :lookup 4462, :concat 189, :equals nil, :iteration 2201, :difference nil, :clone nil, :union nil, :construct 15648}, 56234 {:intersection nil, :remove nil, :insert nil, :lookup 140044, :concat 148, :equals nil, :iteration 39371, :difference nil, :clone nil, :union nil, :construct 279259}, 17 {:intersection nil, :remove nil, :insert nil, :lookup 26, :concat 163, :equals nil, :iteration 23, :difference nil, :clone nil, :union nil, :construct 150}, 17782 {:intersection nil, :remove nil, :insert nil, :lookup 40452, :concat 187, :equals nil, :iteration 12368, :difference nil, :clone nil, :union nil, :construct 112391}, 562 {:intersection nil, :remove nil, :insert nil, :lookup 706, :concat 189, :equals nil, :iteration 423, :difference nil, :clone nil, :union nil, :construct 3746}, 5623 {:intersection nil, :remove nil, :insert nil, :lookup 8145, :concat 189, :equals nil, :iteration 3914, :difference nil, :clone nil, :union nil, :construct 32593}, 10 {:intersection nil, :remove nil, :insert nil, :lookup 16, :concat 161, :equals nil, :iteration 17, :difference nil, :clone nil, :union nil, :construct 86}, 31622 {:intersection nil, :remove nil, :insert nil, :lookup 77626, :concat 190, :equals nil, :iteration 21930, :difference nil, :clone nil, :union nil, :construct 146753}, 177 {:intersection nil, :remove nil, :insert nil, :lookup 231, :concat 188, :equals nil, :iteration 161, :difference nil, :clone nil, :union nil, :construct 1031}}, "clojure.core.rrb-vector" {100000 {:intersection nil, :remove nil, :insert nil, :lookup 2813811, :concat 29900, :equals nil, :iteration 1835797, :difference nil, :clone nil, :union nil, :construct 1100608}, 177827 {:intersection nil, :remove nil, :insert nil, :lookup 5832712, :concat 33432, :equals nil, :iteration 3267179, :difference nil, :clone nil, :union nil, :construct 1964552}, 1000000 {:intersection nil, :remove nil, :insert nil, :lookup 63871894, :concat 42625, :equals nil, :iteration 18548726, :difference nil, :clone nil, :union nil, :construct 11052688}, 316227 {:intersection nil, :remove nil, :insert nil, :lookup 22443835, :concat 35833, :equals nil, :iteration 5833114, :difference nil, :clone nil, :union nil, :construct 3503652}, 1778 {:intersection nil, :remove nil, :insert nil, :lookup 23919, :concat 10418, :equals nil, :iteration 26883, :difference nil, :clone nil, :union nil, :construct 19724}, 31 {:intersection nil, :remove nil, :insert nil, :lookup 190, :concat 659, :equals nil, :iteration 303, :difference nil, :clone nil, :union nil, :construct 495}, 10000 {:intersection nil, :remove nil, :insert nil, :lookup 143060, :concat 7720, :equals nil, :iteration 153119, :difference nil, :clone nil, :union nil, :construct 109576}, 1000 {:intersection nil, :remove nil, :insert nil, :lookup 10760, :concat 15333, :equals nil, :iteration 11649, :difference nil, :clone nil, :union nil, :construct 11064}, 56 {:intersection nil, :remove nil, :insert nil, :lookup 556, :concat 838, :equals nil, :iteration 623, :difference nil, :clone nil, :union nil, :construct 752}, 562341 {:intersection nil, :remove nil, :insert nil, :lookup 49298836, :concat 37877, :equals nil, :iteration 10344969, :difference nil, :clone nil, :union nil, :construct 6150388}, 316 {:intersection nil, :remove nil, :insert nil, :lookup 3437, :concat 5847, :equals nil, :iteration 3622, :difference nil, :clone nil, :union nil, :construct 3584}, 100 {:intersection nil, :remove nil, :insert nil, :lookup 1083, :concat 2958, :equals nil, :iteration 1161, :difference nil, :clone nil, :union nil, :construct 1190}, 3162 {:intersection nil, :remove nil, :insert nil, :lookup 43169, :concat 4663, :equals nil, :iteration 48516, :difference nil, :clone nil, :union nil, :construct 34629}, 56234 {:intersection nil, :remove nil, :insert nil, :lookup 1426155, :concat 21259, :equals nil, :iteration 1033616, :difference nil, :clone nil, :union nil, :construct 619399}, 17 {:intersection nil, :remove nil, :insert nil, :lookup 106, :concat 481, :equals nil, :iteration 173, :difference nil, :clone nil, :union nil, :construct 330}, 17782 {:intersection nil, :remove nil, :insert nil, :lookup 291313, :concat 10993, :equals nil, :iteration 271247, :difference nil, :clone nil, :union nil, :construct 194009}, 562 {:intersection nil, :remove nil, :insert nil, :lookup 6053, :concat 9207, :equals nil, :iteration 6590, :difference nil, :clone nil, :union nil, :construct 6288}, 5623 {:intersection nil, :remove nil, :insert nil, :lookup 75496, :concat 5903, :equals nil, :iteration 85194, :difference nil, :clone nil, :union nil, :construct 61628}, 10 {:intersection nil, :remove nil, :insert nil, :lookup 62, :concat 378, :equals nil, :iteration 108, :difference nil, :clone nil, :union nil, :construct 236}, 31622 {:intersection nil, :remove nil, :insert nil, :lookup 629648, :concat 16872, :equals nil, :iteration 488417, :difference nil, :clone nil, :union nil, :construct 347757}, 177 {:intersection nil, :remove nil, :insert nil, :lookup 1932, :concat 3828, :equals nil, :iteration 2042, :difference nil, :clone nil, :union nil, :construct 2066}}}
================================================
FILE: doc/benchmarks/data/concat.csv
================================================
size,bifurcan.List,java.ArrayList,clojure.PersistentVector,vavr.Vector,scala.Vector,paguro.RrbTree,bifurcan.LinearList,clojure.core.rrb-vector
10,13.6,11.4,62.5,15.0,16.6,26.5,16.1,37.8
17,8.176471,8.058824,44.235294,10.0,10.0,16.941177,9.588235,28.294117
31,4.483871,3.483871,31.548388,6.774194,5.419355,11.387096,6.032258,21.258064
56,2.5714285,2.8035715,27.428572,5.464286,5.017857,14.071428,3.375,14.964286
100,2.41,2.31,23.35,5.72,5.18,3.98,1.89,29.58
177,1.7740113,2.1694915,21.17514,4.8248587,3.3107345,1.3615819,1.0621469,21.62712
316,1.25,2.1677215,19.256329,3.8544304,2.335443,0.8575949,0.5949367,18.503164
562,0.97153026,1.8505338,18.604982,3.955516,1.9306049,0.60854095,0.33629894,16.382563
1000,0.639,1.767,17.906,3.542,1.546,0.727,0.187,15.333
1778,0.82452196,1.6310462,17.655231,3.6091113,1.4150732,0.23622048,0.10686164,5.8593926
3162,0.10974067,1.7125237,17.795067,4.4756484,1.5623024,0.2852625,0.059772298,1.4746996
5623,0.073092654,1.7412413,17.545439,4.21252,1.4636315,0.11613018,0.03361195,1.0497955
10000,0.0447,1.8509,17.5732,4.5135,1.2487,0.0655,0.0189,0.772
17782,0.0351479,1.9495558,17.450119,4.1053877,1.4498931,0.03649758,0.010516252,0.6182094
31622,0.019796344,2.2976408,17.373095,4.3760357,1.4443742,0.02580482,0.006008475,0.5335526
56234,0.026229683,2.1811893,17.619324,4.3532915,1.2567486,0.012074546,0.0026318596,0.37804532
100000,0.00297,2.24121,17.79133,4.42338,1.28174,0.00898,0.00132,0.299
177827,0.0018726066,2.31764,17.851732,4.433815,1.2783042,0.0051848143,7.2542415E-4,0.18800294
316227,0.0011510719,6.0872726,17.925129,4.502373,1.284359,0.002539315,4.110971E-4,0.11331417
562341,8.535746E-4,7.5673695,18.17226,4.511332,1.2696211,0.0018725293,2.3117645E-4,0.06735593
1000000,4.81E-4,5.719794,18.577366,4.775235,0.936365,5.07E-4,1.35E-4,0.042625
================================================
FILE: doc/benchmarks/data/list_construct.csv
================================================
size,bifurcan.List,java.ArrayList,clojure.PersistentVector,vavr.Vector,scala.Vector,paguro.RrbTree,bifurcan.LinearList,clojure.core.rrb-vector
10,9.4,6.0,24.9,122.8,9.8,11.8,8.6,23.6
17,9.058824,7.7647057,22.588236,125.82353,6.9411764,7.7647057,8.823529,19.411764
31,6.064516,6.645161,18.096775,135.64516,4.967742,7.0,6.096774,15.967742
56,8.035714,7.071429,16.071428,151.92857,5.803571,9.964286,6.214286,13.428572
100,7.91,6.62,15.3,170.31,5.65,9.22,6.03,11.9
177,7.9265537,8.853107,14.734464,160.52542,5.2768364,9.474576,5.8248587,11.672317
316,7.455696,7.990506,14.35443,167.99051,5.313291,10.537974,6.132911,11.341772
562,7.524911,9.339858,14.231317,173.57474,5.1921706,9.854093,6.6654806,11.188612
1000,7.284,7.784,14.172,180.283,5.072,11.107,4.923,11.064
1778,7.2581553,6.5410576,14.166479,182.65804,5.0646796,10.391451,5.104612,11.093364
3162,7.2046175,6.6514864,15.516445,191.10689,5.0246677,10.761543,4.9487667,10.951612
5623,7.386804,6.4266405,15.535657,196.15543,4.969945,11.1223545,5.796372,10.959986
10000,7.2806,8.4652,15.4981,179.5069,4.9662,11.2118,5.9852,10.9576
17782,7.3116074,6.484254,15.474975,209.4459,4.9549546,11.320211,6.3204927,10.910415
31622,7.2696857,8.90766,15.500917,219.30981,4.7470746,11.485738,4.6408515,10.997312
56234,7.403955,6.7953553,15.512821,208.06172,4.4433618,11.892645,4.966017,11.01467
100000,6.83919,6.66432,15.61274,190.86024,4.46145,12.22418,4.63192,11.00608
177827,7.2658763,7.4858656,15.6791935,197.01678,4.4837565,12.332981,9.392382,11.047546
316227,7.426023,12.031664,15.7804365,203.03307,4.428657,12.973231,14.0233,11.079547
562341,7.748018,17.517395,15.697704,195.32433,4.542214,13.011118,16.349981,10.937115
1000000,7.310803,14.379088,15.673957,203.99283,4.582003,12.813206,9.882679,11.052688
================================================
FILE: doc/benchmarks/data/list_iterate.csv
================================================
size,bifurcan.List,java.ArrayList,clojure.PersistentVector,vavr.Vector,scala.Vector,paguro.RrbTree,bifurcan.LinearList,clojure.core.rrb-vector
10,1.4,1.6,1.7,1.8,3.6,1.8,1.7,10.8
17,1.117647,1.2352941,1.2941177,1.4117647,3.1176472,1.2941177,1.3529412,10.176471
31,0.9032258,1.0,1.1290323,1.1612903,2.2903225,1.0,1.1290323,9.774194
56,1.5178572,0.875,1.7678572,1.8392857,2.0535715,5.25,1.0357143,11.125
100,2.18,0.79,1.81,1.92,2.13,3.67,0.97,11.61
177,2.220339,0.7288136,1.8474576,1.9491526,2.0056498,3.0960453,0.90960455,11.536723
316,2.205696,0.65189874,1.7974683,1.9398735,1.8765823,2.414557,0.81329113,11.462026
562,2.208185,0.6049822,1.7829181,1.8790035,1.8291816,2.4163702,0.75266904,11.725979
1000,2.204,0.582,1.779,1.909,1.794,2.241,0.729,11.649
1778,1.4386952,0.56299216,1.8335208,1.78009,1.831271,2.1017997,0.70528686,15.119798
3162,1.4111322,0.5553447,1.8358634,1.771031,1.7931689,2.0714738,0.6960784,15.343453
5623,1.4131247,0.5568202,1.8289169,1.772897,1.790859,2.0835853,0.6960697,15.150987
10000,1.4238,0.5572,1.8315,1.7793,1.7861,2.1406,0.693,15.3119
17782,1.4203689,0.55775505,1.820324,1.7672366,1.7702171,2.0864358,0.6955348,15.254021
31622,1.4137626,0.5536336,1.8266397,1.769496,1.7614319,2.1101131,0.6935045,15.445481
56234,1.6594943,0.55566025,1.8979799,1.8362023,2.0761282,2.1287477,0.700128,18.380625
100000,1.67543,0.56148,1.91083,1.83461,2.07209,2.14485,0.69761,18.35797
177827,1.6713041,0.55777246,1.9065102,1.8242562,2.0584614,2.178387,0.69495636,18.372795
316227,1.6643709,0.55767536,1.9050903,1.8317759,2.070364,2.2198262,0.6955415,18.44597
562341,1.6746601,0.55650574,1.9053137,1.847212,2.061694,2.2391484,0.7381873,18.396255
1000000,1.745738,0.562946,1.935245,1.891016,2.107386,2.320294,0.709329,18.548725
================================================
FILE: doc/benchmarks/data/list_lookup.csv
================================================
size,bifurcan.List,java.ArrayList,clojure.PersistentVector,vavr.Vector,scala.Vector,paguro.RrbTree,bifurcan.LinearList,clojure.core.rrb-vector
10,1.7,1.4,4.8,1.8,1.9,1.9,1.6,6.2
17,1.5882353,1.2941177,4.7647057,1.5882353,1.8235294,1.6470588,1.5294118,6.2352943
31,1.483871,1.2580645,4.709677,1.483871,1.7419355,1.516129,1.451613,6.129032
56,3.0535715,1.3214285,6.375,3.125,2.357143,1.5178572,1.4464285,9.928572
100,3.63,1.27,7.68,3.05,2.56,4.14,1.35,10.83
177,3.4745762,1.2542373,7.3502827,2.9096045,2.6271186,4.220339,1.3050847,10.915255
316,3.506329,1.2468355,7.844937,2.943038,2.7246835,4.243671,1.272152,10.876582
562,3.608541,1.2419928,7.640569,2.8701067,2.8790035,4.0373664,1.2562277,10.770463
1000,3.614,1.236,7.5,3.024,2.936,3.876,1.257,10.76
1778,7.7165356,1.2485939,9.169854,6.038808,4.84027,5.8762655,1.3582677,13.452756
3162,7.7179003,1.245098,9.3390255,6.066414,5.9930425,6.0588236,1.4111322,13.652435
5623,7.695892,1.2532456,9.659434,6.1563225,6.353904,6.1963363,1.448515,13.426285
10000,8.4389,1.5085,11.4617,6.9566,6.9497,7.3155,1.64,14.306
17782,10.189462,2.159712,13.280733,8.604038,8.824204,9.230177,2.2748847,16.382465
31622,11.259218,2.19439,15.286858,9.492474,9.626589,10.392543,2.45481,19.911707
56234,16.073248,2.3088167,29.581766,8.739464,13.578423,20.866753,2.4903796,25.36108
100000,17.46262,2.32778,22.28634,10.32512,14.43229,22.792,2.53203,28.13811
177827,17.224106,1.5161252,24.876661,13.565139,10.459368,21.944075,1.5987449,32.799923
316227,24.521938,1.4419926,49.83222,15.486619,13.8668585,32.42438,5.912291,70.97381
562341,29.629885,6.4323745,40.65725,24.448853,20.996775,81.977974,1.5803969,87.66715
1000000,33.788033,1.873249,65.53855,35.926792,18.855904,45.184235,1.975906,63.871895
================================================
FILE: doc/crrbv-27/description.md
================================================
The production version of the Clojure/Java core.rrb-vector library is
in the directory `src/main/clojure`. It uses a maximum tree branching
factor of 32, i.e. all tree nodes have at most 32 children, the same
as Clojure's built in persistent vectors. This is good for look-up
efficiency, but when testing the code, it requires a large number of
vector elements and/or operations on the vectors in order to reach
"interesting" tree structures that exercise all parts of the code, and
find bugs that may be there.
The `src/parameterized/clojure` directory contains source code for a
modified version of the core.rrb-vector library, which uses parameters
defined in file
`src/parameterized/clojure/clojure/core/rrb_vector/parameters.clj` to
control the maximum tree branching factor. The existing code allows
you to change the value of `shift-increment` to any value that is 2 or
larger, and the maximum tree branching factor will then be 2 to the
power of `shift-increment`.
I have found a bug in the core.rrb-vector library that I do not yet
fully understand. I have a patch to the code that causes the problem
not to occur, but I do not understand the original or modified code
well enough to be confident that it is a correct fix.
I do have an easily reproducible test case that causes the problem to
occur with the parameterized version of the code when
`shift-increment` is 2, so a maximum tree branch factor of 4. I
believe it is likely that a different test sequence could be found
that exhibits the same bug with the production code's branch factor of
32, but it would likely be a far longer sequence of operations,
e.g. perhaps as long as millions of operations or more.
Here is a way to reproduce the problem with the parameterized version
of the code.
```bash
git clone https://github.com/clojure/core.rrb-vector
cd core.rrb-vector
git checkout f69df0f95e450bb4ff8e3294f3265d3d25f4e5db
patch -p1 < use-shift-increment-2.patch
```
You can see these tests fail:
```bash
./script/jdo check
```
To reproduce the same problem in the REPL, it takes a few more steps,
shown below.
```bash
# Start a JVM with a socket REPL listening on TCP port 50505
./script/jdo
```
```clojure
;; Connect to the socket REPL from your favorite dev environment, via
;; either a socket REPL, nREPL, whatever you prefer and know how to
;; set up.
(require '[clojure.core.rrb-vector :as fv]
'[clojure.core.rrb-vector.debug :as dv]
'[clojure.core.rrb-vector.rrbt :as rrbt])
;; Enable full debug options, except trace printing is off, for all
;; checking-* functions in the debug namespace.
(dv/set-debug-opts! dv/full-debug-opts)
;; The shortest sequence of operations I currently know that reaches
;; the point where the bug occurs only after about 2000 calls to
;; insert-by-sub-vcatvec, with a particular sequence of arguments.
;; It takes a couple of minutes to get there when using
;; checking-catvec and checking-subvec, but we can capture the vectors
;; that cause the problem once we get there, and as long as we keep
;; the same JVM running we can examine their contents all we want.
(def my-catvec dv/checking-catvec)
(def my-subvec dv/checking-subvec)
;; This code is slightly modified from that in the deftest named
;; test-reduce-subvec-catvec in the test-common namespace.
(defn insert-at-index-n [v n]
(my-catvec (my-subvec v 0 n)
(dv/cvec ['x])
(my-subvec v n)))
(defn insert-by-sub-catvec [v [n sz]]
(let [ret (insert-at-index-n v n)]
(when (or (< n 20)
(zero? (mod n 10)))
(println "n=" n))
(let [ret-nums (filter number? ret)]
(if (not= (filter number? ret) (range sz))
(throw (ex-info (str "Failure for sz=" sz " n=" n)
{:v v :sz sz :n n}))))
ret))
(defn repeated-subvec-catvec [sz]
(reduce insert-by-sub-catvec
(dv/cvec (range sz))
(map (fn [x] [x sz])
(range sz 0 -1))))
(count @dv/failure-data)
;; should be 0 initially
(def sz 2061)
(def x (repeated-subvec-catvec sz))
;; That caused an exception to be thrown just after "n= 15" was printed
;; Record the exception in e1
(def e1 *e)
(count @dv/failure-data)
;; should be 1 now, since data about 1 error has been appended to the vector
;; @dv/failure-data.
;; extract the data from the error
(def ed (nth @dv/failure-data 0))
(keys ed)
(:err-desc-str ed)
;; "splice-rrbts-main"
(def vret (:ret ed))
(count (:args ed))
;; should be 4. We only care about the last two args, which are
;; the two vectors that we were trying to concatenate.
(def v1 (nth (:args ed) 2))
(def v2 (nth (:args ed) 3))
(def errinf (:error-info ed))
errinf
;; {:error true, :description "One or more errors found", :data ({:error true, :kind :internal, :description "Found internal non-regular node with 1 non-nil, 3 nil children, and # children prefix sums: (39) - expected that to match stored ranges: (33 0 0 0 1)"})}
(dv/dbg-vec v1)
(dv/dbg-vec v2)
(dv/dbg-vec vret)
```
In the long output of the dbg-vec on vret, there is only one node that
has a ranges array printed as (33 0 0 0 1), so that must be the one
that the error message above is referring to.
Here are the few lines before and after that node, which are near the
beginning of the output:
```
Vector (4109 elements):
16:00 PersistentVector$Node: (33 4108 0 0 2)
14:00 PersistentVector$Node: (33 0 0 0 1)
12:00 PersistentVector$Node: (37 39 0 0 2)
10:00 PersistentVector$Node: (21 29 35 37 4)
```
The line starting with 14:00 is the node that has the reported
problem.
Its first child is the node printed on the line starting with 12:00.
Note that its ranges array contains (37 39 0 0 2), so that node has 2
children, the first with 37 vector elements as leaves beneath it, the
second with 39-37=2 vector elements with leaves beneath it (if those
values are correct -- I believe they are, else ranges-errors would
have complained about those before its parent node).
The node 14:00 should have a ranges array (39 0 0 0 1), but it is (33
0 0 0 1), so it is under-counting the number of vector elements
beneath it by 6. Its parent node, the one printed on line 16:00, is
consistent with node 14:00, but also too small by 6.
================================================
FILE: doc/crrbv-27/proposed-fix-needs-thought-and-testing-plus-debug-prints.patch
================================================
diff --git a/src/parameterized/clojure/clojure/core/rrb_vector/debug.clj b/src/parameterized/clojure/clojure/core/rrb_vector/debug.clj
index 27156d2..8208cae 100644
--- a/src/parameterized/clojure/clojure/core/rrb_vector/debug.clj
+++ b/src/parameterized/clojure/clojure/core/rrb_vector/debug.clj
@@ -248,28 +248,31 @@
(defn slow-into [to from]
(reduce conj to from))
+(defn all-nodes-in-subtree [node node-shift get-array regular?]
+ (letfn [(go [depth shift node]
+ (if node
+ (if (not= shift 0)
+ (cons
+ {:depth depth :shift shift :kind :internal :node node}
+ (apply concat
+ (map (partial go (inc depth) (- shift p/shift-increment))
+ (let [arr (get-array node)]
+ (if (regular? node)
+ arr
+ (butlast arr))))))
+ (cons {:depth depth :shift shift :kind :internal :node node}
+ (map (fn [x]
+ {:depth (inc depth) :kind :leaf :value x})
+ (get-array node))))))]
+ (go 1 node-shift node)))
+
(defn all-vector-tree-nodes [v]
(let [{:keys [v get-root get-shift get-array regular?]}
(pd/unwrap-subvec-accessors-for v)
root (get-root v)
shift (get-shift v)]
- (letfn [(go [depth shift node]
- (if node
- (if (not= shift 0)
- (cons
- {:depth depth :shift shift :kind :internal :node node}
- (apply concat
- (map (partial go (inc depth) (- shift p/shift-increment))
- (let [arr (get-array node)]
- (if (regular? node)
- arr
- (butlast arr))))))
- (cons {:depth depth :shift shift :kind :internal :node node}
- (map (fn [x]
- {:depth (inc depth) :kind :leaf :value x})
- (get-array node))))))]
- (cons {:depth 0 :kind :base :shift shift :value v}
- (go 1 shift root)))))
+ (cons {:depth 0 :kind :base :shift shift :value v}
+ (all-nodes-in-subtree root shift get-array regular?))))
;; All nodes that should be internal nodes are one of the internal
;; node types satisfying internal-node? All nodes that are less
@@ -297,6 +300,11 @@
;; a straightforward sanity check to make, to return an error if a
;; non-regular node is found with a regular ancestor in the tree.
+(defn basic-node-errors-subtree [node node-shift]
+
+ )
+
+
(defn basic-node-errors [v]
(let [{:keys [v get-shift]} (pd/unwrap-subvec-accessors-for v)
shift (get-shift v)
@@ -500,6 +508,48 @@
(/ (* 1.0 tail-off) max-tree-cap)))
+(defn ranges-errors-subtree
+ "Whereas ranges-errors performs checks on the contents of the entire
+ tree that represents a vector, ranges-errors-subtree performs these
+ checks on any subtree of such a tree, rooted at the given node. You
+ must also supply a shift value, since this is not stored in the data
+ itself."
+ ([node node-shift get-array regular? get-ranges]
+ (ranges-errors-subtree node node-shift get-array regular? get-ranges
+ false nil))
+ ([node node-shift get-array regular? get-ranges do-root-checks? root-node-cnt]
+ (letfn [
+ (go [shift node]
+ (cond
+ (nil? node) {:error false :kind :nil}
+ (zero? shift) (let [n (count (get-array node))]
+ (merge {:error (zero? n), :kind :leaves,
+ :full? (= n p/max-branches), :count n}
+ (if (zero? n)
+ {:description
+ (str "Leaf array has 0 elements."
+ " Expected > 0.")})))
+ :else ;; non-0 shift
+ (let [children (map (partial go (- shift p/shift-increment))
+ (let [arr (get-array node)]
+ (if (regular? node)
+ arr
+ (butlast arr))))
+ errs (filter :error children)]
+ (cond
+ (seq errs) {:error true, :description "One or more errors found",
+ :data errs}
+ (not= p/max-branches (count children))
+ {:error true, :kind :internal,
+ :description (str "Found internal node that has "
+ (count children) " children - expected p/max-branches.")}
+ (regular? node) (regular-node-errors (and do-root-checks?
+ (= shift node-shift))
+ root-node-cnt children)
+ :else (non-regular-node-errors node get-ranges children)))))]
+ (go node-shift node))))
+
+
(defn ranges-errors [v]
(let [{:keys [v get-root get-shift get-tail get-cnt get-array get-ranges
regular? tail-len]}
@@ -508,70 +558,48 @@
root-node-cnt (count v)
root-shift (get-shift v)
tail-off (pd/dbg-tailoff v)
- tail (get-tail v)]
- (letfn [
- (go [shift node]
- (cond
- (nil? node) {:error false :kind :nil}
- (zero? shift) (let [n (count (get-array node))]
- (merge {:error (zero? n), :kind :leaves,
- :full? (= n p/max-branches), :count n}
- (if (zero? n)
- {:description
- (str "Leaf array has 0 elements."
- " Expected > 0.")})))
- :else ;; non-0 shift
- (let [children (map (partial go (- shift p/shift-increment))
- (let [arr (get-array node)]
- (if (regular? node)
- arr
- (butlast arr))))
- errs (filter :error children)]
- (cond
- (seq errs) {:error true, :description "One or more errors found",
- :data errs}
- (not= p/max-branches (count children))
- {:error true, :kind :internal,
- :description (str "Found internal node that has "
- (count children) " children - expected p/max-branches.")}
- (regular? node) (regular-node-errors (= shift root-shift)
- root-node-cnt children)
- :else (non-regular-node-errors node get-ranges children)))))]
- (let [x (go root-shift root)]
- (cond
- (:error x) x
- (not= tail-off (:count x))
- {:error true, :kind :root,
- :description (str "Found tail-off=" tail-off " != " (:count x)
- "=count of values beneath internal nodes")
- :internal-node-leaf-count (:count x) :tail-off tail-off
- :cnt (get-cnt v)}
- (and (pd/transient-vector? v)
- (not= (tail-len tail) p/max-branches))
- {:error true, :kind :root,
- :description (str "Found transient vector with tail length "
- (tail-len tail) " - expecting p/max-branches")}
- ;; It is always a bad thing if shift becomes more than 32,
- ;; because the bit-shift-left and bit-shift-right operations
- ;; on 32-bit ints actually behave like (bit-shift-left
- ;; x (mod shift-amount 32)) for shift-amount over 32. It is
- ;; also likely a bug in the implementation if that happens.
- (>= root-shift 32)
- {:error true, :kind :root,
- :description (str "shift of root is " root-shift " >= 32,"
- " which is not supported.")}
- ;; This is not necessarily a bug, but it seems likely to be
- ;; a bug if a tree is less than 1/max-branches-squared full compared to its
- ;; max capacity. 1/(p/max-branches) full is normal when a tree becomes 1
- ;; deeper than it was before.
- (< 0 (:count x) (max-capacity-divided-by-max-branches-squared root-shift))
- {:error false, :warning true, :kind :root-too-deep,
- :description (str "For root shift=" root-shift " the maximum "
- "capacity divided by p/max-branches-squared is "
- (max-capacity-divided-by-max-branches-squared root-shift)
- " but the tree contains only "
- (:count x) " vector elements outside of the tail")}
- :else x)))))
+ tail (get-tail v)
+ x (ranges-errors-subtree root root-shift get-array regular? get-ranges
+ true root-node-cnt)]
+ (cond
+ (:error x) x
+
+ (not= tail-off (:count x))
+ {:error true, :kind :root,
+ :description (str "Found tail-off=" tail-off " != " (:count x)
+ "=count of values beneath internal nodes")
+ :internal-node-leaf-count (:count x) :tail-off tail-off
+ :cnt (get-cnt v)}
+
+ (and (pd/transient-vector? v)
+ (not= (tail-len tail) p/max-branches))
+ {:error true, :kind :root,
+ :description (str "Found transient vector with tail length "
+ (tail-len tail) " - expecting p/max-branches")}
+
+ ;; It is always a bad thing if shift becomes more than 32,
+ ;; because the bit-shift-left and bit-shift-right operations
+ ;; on 32-bit ints actually behave like (bit-shift-left
+ ;; x (mod shift-amount 32)) for shift-amount over 32. It is
+ ;; also likely a bug in the implementation if that happens.
+ (>= root-shift 32)
+ {:error true, :kind :root,
+ :description (str "shift of root is " root-shift " >= 32,"
+ " which is not supported.")}
+
+ ;; This is not necessarily a bug, but it seems likely to be
+ ;; a bug if a tree is less than 1/max-branches-squared full compared to its
+ ;; max capacity. 1/(p/max-branches) full is normal when a tree becomes 1
+ ;; deeper than it was before.
+ (< 0 (:count x) (max-capacity-divided-by-max-branches-squared root-shift))
+ {:error false, :warning true, :kind :root-too-deep,
+ :description (str "For root shift=" root-shift " the maximum "
+ "capacity divided by p/max-branches-squared is "
+ (max-capacity-divided-by-max-branches-squared root-shift)
+ " but the tree contains only "
+ (:count x) " vector elements outside of the tail")}
+
+ :else x)))
#_(defn add-return-value-checks [f err-desc-str return-value-check-fn]
(fn [& args]
@@ -1119,7 +1147,7 @@
"(type v1)=" (type v1)
"(type v2)=" (type v2)))
(let [r1 (checking-splice-rrbts-main nm am v1 v2)
- r2 (rrbt/peephole-optimize-root r1)]
+ r2 (rrbt/peephole-optimize-root v1 v2 r1)]
;; Optimize a bit by only doing all of the sanity checks on r2
;; if it is not the same identical data structure r1 that
;; checking-splice-rrbts-main already checked.
diff --git a/src/parameterized/clojure/clojure/core/rrb_vector/rrbt.clj b/src/parameterized/clojure/clojure/core/rrb_vector/rrbt.clj
index 9548b40..ee61ef4 100644
--- a/src/parameterized/clojure/clojure/core/rrb_vector/rrbt.clj
+++ b/src/parameterized/clojure/clojure/core/rrb_vector/rrbt.clj
@@ -1387,6 +1387,8 @@
(map list (take gcs arr) (take gcs (map - rngs (cons 0 rngs))))))]
(mapcat cseq (take cs arr) (take cs (map - rngs (cons 0 rngs))))))
+(def extra-trace (atom false))
+
(defn rebalance
[^NodeManager nm ^ArrayManager am shift n1 cnt1 n2 cnt2 ^Box transferred-leaves]
(if (nil? n2)
@@ -1398,14 +1400,24 @@
sbc2 (subtree-branch-count nm am n2 shift)
p (+ sbc1 sbc2)
e (- a (inc (quot (dec p) p/max-branches)))]
+ (when @extra-trace
+ (println "dbg rebalance #1 shift=" shift
+ "cnt1=" cnt1 "cnt2=" cnt2)
+ (println "dbg rebalance #2 slc1=" slc1 " slc2=" slc2
+ "a=" a "sbc1=" sbc1 "sbc2=" sbc2 "p=" p "e=" e))
(cond
(<= e max-extra-search-steps)
- (pair n1 n2)
+ (do
+ (when @extra-trace
+ (println "dbg rebalance #3a"))
+ (pair n1 n2))
(<= (+ sbc1 sbc2) p/max-branches-squared)
(let [new-arr (object-array p/non-regular-array-len)
new-rngs (int-array p/non-regular-array-len)
new-n1 (.node nm nil new-arr)]
+ (when @extra-trace
+ (println "dbg rebalance #3b"))
(loop [i 0
bs (partition-all p/max-branches
(concat (child-seq nm n1 shift cnt1)
@@ -1427,7 +1439,21 @@
(aset new-rngs p/max-branches (inc i))
(recur (inc i) (next bs)))))
(aset new-arr p/max-branches new-rngs)
- (set! (.-val transferred-leaves) cnt2)
+
+;; (when @extra-trace
+;; (println "dbg rebalance #3 cnt2=" cnt2 "-> transferred-leaves"))
+;; (set! (.-val transferred-leaves) cnt2)
+
+ (when-not (zero? (.-val transferred-leaves))
+ (println "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")
+ (println "dbg CHANGED rebalance #3"
+ "shift=" shift
+ "cnt1=" cnt1
+ "cnt2=" cnt2
+ "cnt2 added to transferred-leaves=" (.-val transferred-leaves)))
+ (set! (.-val transferred-leaves)
+ (+ (.-val transferred-leaves) cnt2))
+
(pair new-n1 nil))
:else
@@ -1437,6 +1463,8 @@
new-rngs2 (int-array p/non-regular-array-len)
new-n1 (.node nm nil new-arr1)
new-n2 (.node nm nil new-arr2)]
+ (when @extra-trace
+ (println "dbg rebalance #3c"))
(loop [i 0
bs (partition-all p/max-branches
(concat (child-seq nm n1 shift cnt1)
@@ -1457,6 +1485,10 @@
d (if (>= tbs p/max-branches)
(aget r li)
(- (aget r li) (aget r (- li tbs))))]
+ (when @extra-trace
+ (println "dbg rebalance #4 transferred-leaves was"
+ (.-val transferred-leaves)
+ "adding d=" d "to it"))
(set! (.-val transferred-leaves)
(+ (.-val transferred-leaves) d))))
(let [new-arr (if (< i p/max-branches) new-arr1 new-arr2)
@@ -1472,7 +1504,21 @@
(aset new-arr2 p/max-branches new-rngs2)
(pair new-n1 new-n2))))))
-(defn zippath
+(declare zippath)
+
+(defn show-ranges [msg node nm desc]
+ (cond
+ (nil? node)
+ (println (format "%s %s=nil" msg desc))
+
+ (= p/non-regular-array-len (alength (.array node)))
+ (println (format "%s (ranges %s)=" msg desc)
+ (seq (ranges nm node)))
+
+ :else
+ (println (format "%s %s has no ranges" msg desc))))
+
+(defn zippath-main
[^NodeManager nm ^ArrayManager am shift n1 cnt1 n2 cnt2 ^Box transferred-leaves]
(if (== shift p/shift-increment)
(rebalance-leaves nm am n1 cnt1 n2 cnt2 transferred-leaves)
@@ -1494,6 +1540,22 @@
[new-c1 new-c2] (zippath nm am (- shift p/shift-increment) c1 ccnt1 c2 ccnt2
next-transferred-leaves)
d (.-val next-transferred-leaves)]
+ (when @extra-trace
+ (println "dbg zippath #1 shift=" shift
+ "cnt1=" cnt1 "(reg? n1)=" (.regular nm n1)
+ "ccnt1=" ccnt1
+ "cnt2=" cnt2 "(reg? n2)=" (.regular nm n2)
+ "ccnt2=" ccnt2)
+ (println "dbg zippath #2 d=" d
+ "(.-val transferred-leaves)=" (.-val transferred-leaves))
+ (show-ranges "dbg zippath #3" c1 nm "c1")
+ (show-ranges "dbg zippath #3" c2 nm "c2")
+ (show-ranges "dbg zippath #3" new-c1 nm "new-c1")
+ (show-ranges "dbg zippath #3" new-c2 nm "new-c2")
+ (println "dbg zippath #4 (identical? c1 new-c1)="
+ (identical? c1 new-c1))
+ (println "dbg zippath #4 (identical? c2 new-c2)="
+ (identical? c2 new-c2)))
(set! (.-val transferred-leaves) (+ (.-val transferred-leaves) d))
(rebalance nm am shift
(if (identical? c1 new-c1)
@@ -1508,6 +1570,18 @@
(- cnt2 d)
transferred-leaves))))
+(def zippath-extra-check-fn (atom nil))
+
+(defn zippath
+ [^NodeManager nm ^ArrayManager am shift n1 cnt1 n2 cnt2 ^Box transferred-leaves]
+ (let [ret (zippath-main nm am shift n1 cnt1 n2 cnt2 transferred-leaves)
+ [n1 n2] ret]
+ (let [f @zippath-extra-check-fn]
+ (when f
+ (f n1 shift nm)
+ (f n2 shift nm)))
+ ret))
+
(defn squash-nodes [^NodeManager nm shift n1 cnt1 n2 cnt2]
(let [arr1 (.array nm n1)
arr2 (.array nm n2)
@@ -1615,7 +1689,12 @@
(let [rngs (ranges nm node)]
(aget rngs (dec (aget rngs p/max-branches))))))
-(defn peephole-optimize-root [^Vector v]
+(def peephole-extra-check (atom nil))
+
+(defn peephole-optimize-root [arg1 arg2 ^Vector v]
+ (let [f @peephole-extra-check]
+ (if f
+ (f v arg1 arg2)))
(let [config @peephole-optimization-config]
(if (<= (.-shift v) (* 2 p/shift-increment))
;; Tree depth cannot be reduced if shift <= p/shift-increment.
@@ -1766,6 +1845,8 @@
s2 (.-shift v2)
r1 (.-root v1)
o? (overflow? nm r1 s1 (+ (count v1) (- p/max-branches (.alength am (.-tail v1)))))
+ _ (when @extra-trace
+ (println "dbg #1 s1=" s1 "s2=" s2 "o?=" o?))
r1 (if o?
(let [tail (.-tail v1)
tail-node (.node nm nil tail)
@@ -1795,9 +1876,24 @@
d (.-val transferred-leaves)
ncnt1 (+ (count v1) d)
ncnt2 (- (count v2) (.alength am (.-tail v2)) d)
+ _ (when @extra-trace
+ (show-ranges "dbg #1b" n1 nm "n1")
+ (show-ranges "dbg #1b" n2 nm "n2")
+ (println "dbg #2 s=" s "(class d)=" (class d) "d=" d)
+ (println "dbg #3 (count v1)=" (count v1)
+ "(count v2)=" (count v2)
+ "ncnt1=" ncnt1 "ncnt2=" ncnt2)
+ (println "dbg #4 (identical? n2 r2)="
+ (identical? n2 r2)))
[n1 n2] (if (identical? n2 r2)
(squash-nodes nm s n1 ncnt1 n2 ncnt2)
(object-array (list n1 n2)))
+ _ (when @extra-trace
+ (println "dbg #5 (boolean n2)=" (boolean n2))
+ (let [al1 (alength (.array n1))]
+ (if (== al1 p/non-regular-array-len)
+ (println "dbg #6 n1 ranges=" (seq (aget (.array n1)
+ p/max-branches))))))
ncnt1 (if n2
(int ncnt1)
(unchecked-add-int (int ncnt1) (int ncnt2)))
@@ -1826,7 +1922,7 @@
(defn splice-rrbts [^NodeManager nm ^ArrayManager am ^Vector v1 ^Vector v2]
(let [r1 (splice-rrbts-main nm am v1 v2)
- r2 (peephole-optimize-root r1)]
+ r2 (peephole-optimize-root v1 v2 r1)]
(fallback-to-slow-splice-if-needed v1 v2 r2)))
(defn array-copy [^ArrayManager am from i to j len]
================================================
FILE: doc/crrbv-27/proposed-fix-needs-thought-and-testing.patch
================================================
diff --git a/src/main/clojure/clojure/core/rrb_vector/rrbt.clj b/src/main/clojure/clojure/core/rrb_vector/rrbt.clj
index 1d231a6..45bfb6b 100644
--- a/src/main/clojure/clojure/core/rrb_vector/rrbt.clj
+++ b/src/main/clojure/clojure/core/rrb_vector/rrbt.clj
@@ -1426,7 +1426,16 @@
(aset new-rngs 32 (inc i))
(recur (inc i) (next bs)))))
(aset new-arr 32 new-rngs)
- (set! (.-val transferred-leaves) cnt2)
+;; (set! (.-val transferred-leaves) cnt2)
+ (when-not (zero? (.-val transferred-leaves))
+ (println "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")
+ (println "dbg CHANGED rebalance #3"
+ "shift=" shift
+ "cnt1=" cnt1
+ "cnt2=" cnt2
+ "cnt2 added to transferred-leaves=" (.-val transferred-leaves)))
+ (set! (.-val transferred-leaves)
+ (+ (.-val transferred-leaves) cnt2))
(pair new-n1 nil))
:else
diff --git a/src/test/clojure/clojure/core/rrb_vector/long_test.clj b/src/test/clojure/clojure/core/rrb_vector/long_test.clj
index b68cf5a..ea2540f 100644
--- a/src/test/clojure/clojure/core/rrb_vector/long_test.clj
+++ b/src/test/clojure/clojure/core/rrb_vector/long_test.clj
@@ -84,7 +84,7 @@
(defn vector-push-f [v my-catvec extra-checks-catvec]
(loop [v v
i 0]
- (let [check? (or (zero? (mod i 10000))
+ (let [check? (or (zero? (mod i 100))
(and (> i 99000) (zero? (mod i 100)))
(and (> i 99900) (zero? (mod i 10))))]
(when check?
================================================
FILE: doc/crrbv-27/use-shift-increment-2.patch
================================================
diff --git a/deps.edn b/deps.edn
index 775cc3e..f03df7a 100644
--- a/deps.edn
+++ b/deps.edn
@@ -5,8 +5,8 @@
;; want to test with modifications to it:
;; org.clojure/clojurescript {:local/root "/Users/jafinger/clj/clojurescript"}
-{:paths ["src/main/clojure" "src/main/cljs" "src/main/cljc"]
- ;;:paths ["src/parameterized/clojure" "src/main/cljs" "src/main/cljc"]
+{;;:paths ["src/main/clojure" "src/main/cljs" "src/main/cljc"]
+ :paths ["src/parameterized/clojure" "src/main/cljs" "src/main/cljc"]
:aliases
{;; Common alias to use for all Clojure/Java commands
:clj {:jvm-opts ["-XX:-OmitStackTraceInFastThrow"]}
diff --git a/src/parameterized/clojure/clojure/core/rrb_vector/parameters.clj b/src/parameterized/clojure/clojure/core/rrb_vector/parameters.clj
index 2cd5004..c8a58ea 100644
--- a/src/parameterized/clojure/clojure/core/rrb_vector/parameters.clj
+++ b/src/parameterized/clojure/clojure/core/rrb_vector/parameters.clj
@@ -8,7 +8,7 @@
;; * when the shift-increment is 2
;; 5 3 2
-(def shift-increment 5)
+(def shift-increment 2)
;; 10 6 4
(def shift-increment-times-2 (* 2 shift-increment))
================================================
FILE: doc/hash-details.md
================================================
# Background on Clojure collection hash calculation
The persistent collections included with Clojure are immutable when
accessed via Clojure's published methods, e.g. `conj`, `assoc`,
`peek`, `seq`, etc.
Their implementation actually uses mutable fields to store cached
versions of their Java `.hashCode` and `clojure.core/hash` values
(those values are different from each other for most collection
values starting with Clojure 1.6.0).
On the JVM, all fields of a newly constructed object are first
initialized to their default JVM initial values, e.g. 0 for a
primitive `int` field, `null` for all references, etc. Then any
values assigned in the constructor are assigned. If the field is
declared `final`, then as long as the reference to the object is not
made visible to any other object before the constructor finishes
executing, any thread that later sees the object should see only the
value assigned while the constructor executed, not the default JVM
initial value. This is promised by the Java Memory Model and the
specialness of the `final` field modifier.
However, the cached hash fields in Clojure, and some other Java
objects, is intentionally stored in non-final fields, so that if no
other code ever needs to know the hash of the value, no time is ever
spent calculating it. This is a performance optimization.
If some other code does later want the hash value, then it is
calculated on demand at that time, and the calculated value is written
into the field, so that later calls to get the hash value by the same
thread are guaranteed to avoid calculating it again. If another
thread calls the function/method to get the hash value, because the
hash field has no special Java modifiers like `final` or `volatile`,
the Java Memory Model says that it might get the updated value, or it
might get the value from an older write to that field, e.g. the
initial value of 0 from the JVM default initialization of all fields
-- _even if_ the constructor assigned a non-0 value to the field.
Thus for thread safety of getting hash values of immutable values
using this "initialize a default value, and calculate on demand
later", if the field where the cached value is stored has no special
modifiers like `final` or `volatile`, and the function to get the hash
value is not declared `synchronized` (none of which are true for the
Clojure implementation), the only safe value to assign in the
constructor is none at all (leaving the field as the default JVM
initial value of 0), or to assign a value of 0 explicitly. If any
other value is assigned during the constructor, e.g. -1, other threads
calling the hash function might read a 0 from that field, or -1,
depending upon all kinds of factors that are impossible to control or
observe from a Java program.
Before Clojure 1.9.0, every collection did assign a value of -1 to
these fields during the constructor call, which was unsafe. This was
fixed in the Clojure 1.9.0 release. See this JIRA ticket for more
details: https://clojure.atlassian.net/browse/CLJ-2091
It links to this article on this pattern of writing Java code that has
intentional data races, but is still correct according to the Java
Memory Model:
http://jeremymanson.blogspot.com/2008/12/benign-data-races-in-java.html
Note that JavaScript run time environments are single threaded (not
counting WebWorkers, but as far as I know, no ClojureScript objects
are shared between the main thread and any WebWorker threads via
shared memory), so these issues do not arise, and any initial value
can be stored in the hash field without a problem.
# Java `.hashCode` vs. `clojure.core/hash`
Some background and history on why `.hashCode` and `clojure.core/hash`
return different values from each other can be found in the Clojure
equality guide, especially the ["Equality and hash"
section](https://clojure.org/guides/equality#equality_and_hash).
# Details in core.rrb-vector Clojure implementation
`core.rrb-vector`'s Clojure implementation should use an initial value
of the mutable hash fields of 0, for the same reasons described above
that any such fields should be initialized to 0 on the JVM.
In `core.rrb-vector` release 0.0.14 and earlier, these fields were
incorrectly initialized to -1. Again, this does not cause any
problems in a single-threaded program, and would only cause problems
in some timing-dependent cases (perhaps rarely, but there is no
guarantee of this) in a multi-threaded program.
This is a list of classes in `core.rrb-vector`'s Clojure
implementation that were changed to correct this problem, after the
release of version 0.0.14:
* `VecSeq` - last field is Clojure _hasheq, second to last is Java _hash
* `Vector` - same as `VecSeq`
* `Transient` - no hash fields.
I believe that calling `clojure.core/hash` on a transient collection
falls through to some default hash implementation that is based on the
identity of the mutable object, and is not the same for all transients
with "the same" contents, the way it is for immutable collections.
## `VecSeq` methods and constructor calls
The constructor to `VecSeq` in method `withMeta` actually appears safe
to me to initialize the returned object with the same values as the
original collection. The hash of the returned collection is
guaranteed to be the same as that of the collection on which
`withMeta` was called, so the initial values will either be 0, or the
correct final hash value.
All occurrences of the string VecSeq in the implementation are in the
one source file rrbt.clj, so I feel safe in saying I have corrected
all constructor calls to VecSeq.
I also corrected an unsafe data race in the implementation of method
`hasheq` for class `VecSeq`. It was reading the field `_hasheq` twice
per call, instead of only once. My fixed version reads that field at
most once. See this article for the details of the changes I made and
why:
http://jeremymanson.blogspot.com/2008/12/benign-data-races-in-java.html
## `Vector` methods and constructor calls
The constructor to `Vector` in method `withMeta` actually appears safe
to me, for the same reason as it is for the same method in class
`VecSeq` described in the previous section.
Files containing constructor calls for class `Vector`:
* rrbt.clj - many, fixed
* rrb_vector.clj - fixed
Files checked for occurrences of `-1`:
* rrb_vector.clj - none remaining after constructor calls to `Vector`
were fixed.
* debug.clj - only one, used to return a "not found" value from
`first-diff`
* fork_join.clj - none, and no constructor calls
* interop.clj.clj - none, and no constructor calls
* nodes.clj - none, and no constructor calls to `Vector` or `VecSeq`.
Many to `VecNode`, but it has no hash fields.
* protocols.clj - none, and no constructor calls
* rrbt.clj
* Many occurrences of `Vector.` constructor calls were updated to
use initial hash values of 0 instead of -1.
* Fixed racy hash function/method bugs in these methods, for the
same reason as described in previous section.
* Vector/hashCode
* Vector/hasheq
* VecSeq/hasheq
* transients.clj - none, and no mentions of `Vector` or `Vecseq` or
hash
# Details in core.rrb-vector ClojureScript implementation
Since there are no thread-safety issues here, the only thing to double
check is that the same value (e.g. 0, -1, or whatever constant value)
is used consistently in all places where a new object is created that
contains a cached hash field, and wherever the hash function is
calculated.
Classes in the ClojureScript `core.rrb-vector` implementation with a
hash field:
* `RRBChunkedSeq` - the last field of the constructor call, named `__hash`
* `Vector` - the last field of the constructor call, named `__hash`
All constructor calls for `RRBChunkedSeq` are in file rrbt.cljs, and
all use `nil` as the initial value for the field `__hash`. Its
`-hash` method calls `caching-hash` provided by the ClojureScript core
code, which uses `nil` as the "hash not calculated yet" value.
All constructor calls for `Vector` are in a few files, and all use
`nil` as the initial value for the field `__hash`. Its `-hash` method
also uses `caching-hash`, as described in previous paragraph.
================================================
FILE: doc/rrb-tree-notes.md
================================================
# Other implementations and descriptions of RRB trees
Note that most implementations have an associated paper. If they have
an author in common, then typically the paper or talk describes the
implementation they published.
Implementations:
* TBD: Tiark Rompf and Phil Bagwell's original implementation code?
* If it is available somewhere, likely it is instrumented for
experimentation on multiple variations of the algorithms, and
counting significant events that they were trying to optimize.
Thus more of a research implementation than one intended for use
in production.
* [`scala-rrb-vector`](https://github.com/nicolasstucki/scala-rrb-vector)
Scala library by Nicolas Stucki
* As of Oct 2019, this library has a few unfixed bugs that were
reported as Github issues in 2017. I have not examined them to
see how easy they might be to fix.
* [`bifurcan`](https://github.com/lacuna/bifurcan) Java library by
Zach Tellman, Java class `io.lacuna.bifurcan.List`
* [`Paguro`](https://github.com/GlenKPeterson/Paguro) Java library by
Glen Peterson, Java class `org.organicdesign.fp.collections.RrbTree`.
* From some comments in the code, it appears that perhaps this
implementation is more precisely a data structure based upon
B-trees, because those comments imply that nodes in the tree can
have a number of children varying between some branching factor B,
and be as low as B/2.
* [`c-rrb`](https://github.com/hypirion/c-rrb) C library by Jean
Niklas L'orange
* [`Array`](https://github.com/xash/Array) RRB trees implemented in
JavaScript for use in the Elm programming language.
* [`immer`](https://sinusoid.es/immer) C++ library by Juan Pedro
Bolivar Puente
* [`Vector`](https://docs.rs/im/12.3.3/im/vector/enum.Vector.html)
Rust implementation of RRB trees by Bodil Stokke
* [Source code](https://docs.rs/crate/im/12.3.3/source/)
* [`RRBVector`](https://github.com/rmunn/FSharpx.Collections/blob/rrb-vector/src/FSharpx.Collections.Experimental/RRBVector.fs)
F# library by Robin Munn.
* According to discussion on [this Github
issue](https://github.com/fsprojects/FSharpx.Collections/issues/72)
it appears to have known bugs the author would still like to find
and fix as of 2019-Oct-10.
Implementation of immutable vectors that are not RRB trees:
* [`Clojure`](https://github.com/clojure/clojure) collections library,
Java class `clojure.lang.PersistentVector` implemented in Java
* Also class `clojure.core.Vector` implemented in Clojure, with
memory/time optimizations achieved by restricting vector elements
to all be the same type of Java primitive, e.g. all `long` vector
elements, or all `double`.
* [`Scala`](https://github.com/scala/scala) collection library, Java
class `scala.collection.immutable.Vector`
* In source file `src/library/scala/collection/immutable/Vector.scala`
* As far as I can tell, as of 2019-Oct-10, it appears that this
class does _not_ use RRB trees, and thus implements concatenation
of vectors in linear time in the length of the second vector.
[This Github
issue](https://github.com/nicolasstucki/scala-rrb-vector/issues/9)
from April 2019 implies that Scala has not yet had an RRB tree
implementation incorporated into its standard library.
Published papers and theses:
* Phil Bagwell, Tiark Rompf, "RRB-Trees: Efficient Immutable Vectors",
EPFL-REPORT-169879, September, 2011
[[PDF]](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.592.5377&rep=rep1&type=pdf)
[[SemanticScholar
page]](https://www.semanticscholar.org/paper/RRB-Trees-%3A-Efficient-Immutable-Vectors-Phil-Tiark-Bagwell-Rompf/30c8c562f6421ab6b00d0b7faebd897c407de69c)
* Phil Bagwell talk
[[video]](https://www.youtube.com/watch?v=K2NYwP90bNs), "Striving
to Make Things Simple and Fast", January 2013, given at Clojure
conj conference
* Jean Niklas L'orange, "Improving RRB-Tree Performance through
Transience", Master Thesis, 2014,
[[PDF]](https://hypirion.com/thesis.pdf)
* "RRB Vector: A Practical General Purpose Immutable Sequence",
Nicolas Stucki, Tiark, Rompf, Vlad Ureche, Phil Bagwell, Proc. of
the 20th ACM SIGPLAN International Conference on Functional
Programming, 2015 [[ACM digital library
link]](http://dx.doi.org/10.1145/2784731.2784739)
[[PDF]](https://github.com/nicolasstucki/scala-rrb-vector/blob/master/documents/RRB%20Vector%20-%20A%20Practical%20General%20Purpose%20Immutable%20Sequence.pdf)
* Nicolas Stucki, "Turning Relaxed Radix Balanced Vector from Theory
into Practice for Scala Collections", Master Thesis, 2015
[[PDF]](https://github.com/nicolasstucki/scala-rrb-vector/blob/master/documents/Master%20Thesis%20-%20Nicolas%20Stucki%20-%20Turning%20Relaxed%20Radix%20Balanced%20Vector%20from%20Theory%20into%20Practice%20for%20Scala%20Collections.pdf?raw=true)
* Juan Pedro Bolivar Puente, "Persistence for the Masses: RRB-Vectors
in a Systems Language", Proc. ACM Program. Lang. 1, ICFP, Article 16
(September 2017), https://doi.org/10.1145/3110260
[[PDF]](https://public.sinusoid.es/misc/immer/immer-icfp17.pdf)
* Juan's talk [[video]](https://www.youtube.com/watch?v=sPhpelUfu8Q)
"Postmodern immutable data structures" given at CppCon 2017
* Bodil Stokke talk
[[video]](https://www.youtube.com/watch?v=cUx2b_FO8EQ) "Meetings With
Remarkable Trees" given at ClojuTRE 2018
Related things:
* Jean Niklas L'orange's series of articles on Clojure's persistent
vector data structure and how it works inside. These are good
tutorial style articles. I have not found any similar articles like
these on RRB trees.
* ["Understanding Clojure's Persistent Vectors, Part
1"](https://hypirion.com/musings/understanding-persistent-vector-pt-1),
September 2013
* ["Understanding Clojure's Persistent Vectors, Part
2](https://hypirion.com/musings/understanding-persistent-vector-pt-2),
October 2013
* ["Understanding Clojure's Persistent Vectors, Part
3](https://hypirion.com/musings/understanding-persistent-vector-pt-3)
April 2014
* ["Understanding Clojure's
Transients"](https://hypirion.com/musings/understanding-clojure-transients),
October 2014
* ["Persistent Vector
Performance"](https://hypirion.com/musings/persistent-vector-performance),
January 2015
* ["Persistent Vector Performance
Summarised"](https://hypirion.com/musings/persistent-vector-performance-summarised),
February 2015
* StackOverflow
[question](https://stackoverflow.com/questions/14007153/what-invariant-do-rrb-trees-maintain)
"What invariant do RRB-trees maintain?"
Not RRB trees, but somewhat related ideas:
* ["Theory and practice of chunked
sequences"](http://www.andrew.cmu.edu/user/mrainey//chunkedseq/chunkedseq.html)
web page has links to papers, talks, and Github repository
containing C++ of their ideas.
================================================
FILE: doc/use-transducers/README.md
================================================
The patch use-transducers.md was developed around July or August 2019,
but not yet included in the production core.rrb-vector code, because
of a desire to continue to make core.rrb-vector compatible with
Clojure 1.5.1 and later, whereas transducers were not implemented in
Clojure until version 1.7.0.
We should re-examine this patch when we are ready to require Clojure
1.7.0 or later as a minimum supported version for the core.rrb-vector
library. I may include some performance measurements with and without
these changes, to show how much they can improve the performance of
some operations.
================================================
FILE: doc/use-transducers/use-transducers.patch
================================================
diff -cr core.rrb-vector/src/main/cljs/clojure/core/rrb_vector/rrbt.cljs core.rrb-vector-pff4i/src/main/cljs/clojure/core/rrb_vector/rrbt.cljs
*** core.rrb-vector/src/main/cljs/clojure/core/rrb_vector/rrbt.cljs 2019-09-27 07:56:52.000000000 -0700
--- core.rrb-vector-pff4i/src/main/cljs/clojure/core/rrb_vector/rrbt.cljs 2019-09-30 23:36:02.000000000 -0700
***************
*** 629,635 ****
(recur (inc i) (+ sbc (slot-count child cs))))))))))
(defn leaf-seq [arr]
! (mapcat #(.-arr %) (take (index-of-nil arr) arr)))
(defn rebalance-leaves
[n1 cnt1 n2 cnt2 transferred-leaves]
--- 629,639 ----
(recur (inc i) (+ sbc (slot-count child cs))))))))))
(defn leaf-seq [arr]
! (into [] (comp (take-while (complement nil?))
! (take 32)
! (map #(.-arr %))
! cat)
! arr))
(defn rebalance-leaves
[n1 cnt1 n2 cnt2 transferred-leaves]
***************
*** 649,657 ****
new-arr (make-array (if reg? 32 33))
new-n1 (->VectorNode nil new-arr)]
(loop [i 0
! bs (partition-all 32
! (concat (leaf-seq (.-arr n1))
! (leaf-seq (.-arr n2))))]
(when-first [block bs]
(let [a (make-array (count block))]
(loop [i 0 xs (seq block)]
--- 653,662 ----
new-arr (make-array (if reg? 32 33))
new-n1 (->VectorNode nil new-arr)]
(loop [i 0
! bs (into [] (comp (map #(leaf-seq (.-arr %)))
! cat
! (partition-all 32))
! [n1 n2])]
(when-first [block bs]
(let [a (make-array (count block))]
(loop [i 0 xs (seq block)]
***************
*** 672,680 ****
new-n1 (->VectorNode nil new-arr1)
new-n2 (->VectorNode nil new-arr2)]
(loop [i 0
! bs (partition-all 32
! (concat (leaf-seq (.-arr n1))
! (leaf-seq (.-arr n2))))]
(when-first [block bs]
(let [a (make-array (count block))]
(loop [i 0 xs (seq block)]
--- 677,686 ----
new-n1 (->VectorNode nil new-arr1)
new-n2 (->VectorNode nil new-arr2)]
(loop [i 0
! bs (into [] (comp (map #(leaf-seq (.-arr %)))
! cat
! (partition-all 32))
! [n1 n2])]
(when-first [block bs]
(let [a (make-array (count block))]
(loop [i 0 xs (seq block)]
***************
*** 695,711 ****
rngs (if (regular? node)
(regular-ranges shift cnt)
(node-ranges node))
! cs (if rngs (aget rngs 32) (index-of-nil arr))
! cseq (fn cseq [c r]
(let [arr (.-arr c)
rngs (if (regular? c)
(regular-ranges (- shift 5) r)
(node-ranges c))
! gcs (if rngs (aget rngs 32) (index-of-nil arr))]
! (map list
! (take gcs arr)
! (take gcs (map - rngs (cons 0 rngs))))))]
! (mapcat cseq (take cs arr) (take cs (map - rngs (cons 0 rngs))))))
(defn rebalance
[shift n1 cnt1 n2 cnt2 transferred-leaves]
--- 701,726 ----
rngs (if (regular? node)
(regular-ranges shift cnt)
(node-ranges node))
! cs (if rngs (aget rngs 32) 32)
! cseq (fn cseq [[c r]]
(let [arr (.-arr c)
rngs (if (regular? c)
(regular-ranges (- shift 5) r)
(node-ranges c))
! gcs (if rngs (aget rngs 32) 32)
! rng-deltas (mapv - rngs (cons 0 rngs))]
! (into [] (comp (take-while (complement nil?))
! (take gcs)
! (map-indexed (fn [idx node]
! [node (rng-deltas idx)])))
! arr)))
! rng-deltas (mapv - rngs (cons 0 rngs))]
! (into [] (comp (take-while (complement nil?))
! (take cs)
! (map-indexed (fn [idx node] [node (rng-deltas idx)]))
! (map cseq)
! cat)
! arr)))
(defn rebalance
[shift n1 cnt1 n2 cnt2 transferred-leaves]
***************
*** 727,735 ****
new-rngs (make-array 33)
new-n1 (->VectorNode nil new-arr)]
(loop [i 0
! bs (partition-all 32
! (concat (child-seq n1 shift cnt1)
! (child-seq n2 shift cnt2)))]
(when-first [block bs]
(let [a (make-array 33)
r (make-array 33)]
--- 742,752 ----
new-rngs (make-array 33)
new-n1 (->VectorNode nil new-arr)]
(loop [i 0
! bs (into [] (comp (map (fn [[node cnt]]
! (child-seq node shift cnt)))
! cat
! (partition-all 32))
! [[n1 cnt1] [n2 cnt2]])]
(when-first [block bs]
(let [a (make-array 33)
r (make-array 33)]
***************
*** 758,766 ****
new-n1 (->VectorNode nil new-arr1)
new-n2 (->VectorNode nil new-arr2)]
(loop [i 0
! bs (partition-all 32
! (concat (child-seq n1 shift cnt1)
! (child-seq n2 shift cnt2)))]
(when-first [block bs]
(let [a (make-array 33)
r (make-array 33)]
--- 775,785 ----
new-n1 (->VectorNode nil new-arr1)
new-n2 (->VectorNode nil new-arr2)]
(loop [i 0
! bs (into [] (comp (map (fn [[node cnt]]
! (child-seq node shift cnt)))
! cat
! (partition-all 32))
! [[n1 cnt1] [n2 cnt2]])]
(when-first [block bs]
(let [a (make-array 33)
r (make-array 33)]
***************
*** 862,896 ****
(def peephole-optimization-config (atom {:debug-fn nil}))
(def peephole-optimization-count (atom 0))
! ;; TBD: Transducer versions of child-nodes and bounded-grandchildren
! ;; are included here for when we are willing to rely upon Clojure
! ;; 1.7.0 as the minimum version supported by the core.rrb-vector
! ;; library. They are faster.
!
! #_(defn child-nodes [node]
(into [] (comp (take-while (complement nil?))
(take 32))
(.-arr node)))
- (defn child-nodes [node]
- (->> (.-arr node)
- (take-while (complement nil?))
- (take 32)))
-
;; (take 33) is just a technique to avoid generating more
;; grandchildren than necessary. If there are at least 33, we do not
;; care how many there are.
! #_(defn bounded-grandchildren [children]
(into [] (comp (map child-nodes)
cat
(take 33))
children))
- (defn bounded-grandchildren [children]
- (->> children
- (mapcat child-nodes)
- (take 33)))
-
;; TBD: Do functions like last-non-nil-idx and
;; count-vector-elements-beneath already exist elsewhere in this
;; library? It seems like they might.
--- 881,900 ----
(def peephole-optimization-config (atom {:debug-fn nil}))
(def peephole-optimization-count (atom 0))
! (defn child-nodes [node]
(into [] (comp (take-while (complement nil?))
(take 32))
(.-arr node)))
;; (take 33) is just a technique to avoid generating more
;; grandchildren than necessary. If there are at least 33, we do not
;; care how many there are.
! (defn bounded-grandchildren [children]
(into [] (comp (map child-nodes)
cat
(take 33))
children))
;; TBD: Do functions like last-non-nil-idx and
;; count-vector-elements-beneath already exist elsewhere in this
;; library? It seems like they might.
diff -cr core.rrb-vector/src/main/clojure/clojure/core/rrb_vector/rrbt.clj core.rrb-vector-pff4i/src/main/clojure/clojure/core/rrb_vector/rrbt.clj
*** core.rrb-vector/src/main/clojure/clojure/core/rrb_vector/rrbt.clj 2019-09-29 17:29:58.000000000 -0700
--- core.rrb-vector-pff4i/src/main/clojure/clojure/core/rrb_vector/rrbt.clj 2019-09-30 23:32:01.000000000 -0700
***************
*** 1311,1317 ****
(recur (inc i) (+ sbc (long (slot-count nm am child cs)))))))))))
(defn leaf-seq [^NodeManager nm arr]
! (mapcat #(.array nm %) (take (index-of-nil arr) arr)))
(defn rebalance-leaves
[^NodeManager nm ^ArrayManager am n1 cnt1 n2 cnt2 ^Box transferred-leaves]
--- 1346,1356 ----
(recur (inc i) (+ sbc (long (slot-count nm am child cs)))))))))))
(defn leaf-seq [^NodeManager nm arr]
! (into [] (comp (take-while (complement nil?))
! (take 32)
! (map #(.array nm %))
! cat)
! arr))
(defn rebalance-leaves
[^NodeManager nm ^ArrayManager am n1 cnt1 n2 cnt2 ^Box transferred-leaves]
***************
*** 1331,1339 ****
new-arr (object-array (if reg? 32 33))
new-n1 (.node nm nil new-arr)]
(loop [i 0
! bs (partition-all 32
! (concat (leaf-seq nm (.array nm n1))
! (leaf-seq nm (.array nm n2))))]
(when-first [block bs]
(let [a (.array am (count block))]
(loop [i 0 xs (seq block)]
--- 1370,1379 ----
new-arr (object-array (if reg? 32 33))
new-n1 (.node nm nil new-arr)]
(loop [i 0
! bs (into [] (comp (map #(leaf-seq nm (.array nm %)))
! cat
! (partition-all 32))
! [n1 n2])]
(when-first [block bs]
(let [a (.array am (count block))]
(loop [i 0 xs (seq block)]
***************
*** 1354,1362 ****
new-n1 (.node nm nil new-arr1)
new-n2 (.node nm nil new-arr2)]
(loop [i 0
! bs (partition-all 32
! (concat (leaf-seq nm (.array nm n1))
! (leaf-seq nm (.array nm n2))))]
(when-first [block bs]
(let [a (.array am (count block))]
(loop [i 0 xs (seq block)]
--- 1394,1403 ----
new-n1 (.node nm nil new-arr1)
new-n2 (.node nm nil new-arr2)]
(loop [i 0
! bs (into [] (comp (map #(leaf-seq nm (.array nm %)))
! cat
! (partition-all 32))
! [n1 n2])]
(when-first [block bs]
(let [a (.array am (count block))]
(loop [i 0 xs (seq block)]
***************
*** 1377,1391 ****
rngs (if (.regular nm node)
(ints (regular-ranges shift cnt))
(ranges nm node))
! cs (if rngs (aget rngs 32) (index-of-nil arr))
! cseq (fn cseq [c r]
(let [arr (.array nm c)
rngs (if (.regular nm c)
(ints (regular-ranges (- shift 5) r))
(ranges nm c))
! gcs (if rngs (aget rngs 32) (index-of-nil arr))]
! (map list (take gcs arr) (take gcs (map - rngs (cons 0 rngs))))))]
! (mapcat cseq (take cs arr) (take cs (map - rngs (cons 0 rngs))))))
(defn rebalance
[^NodeManager nm ^ArrayManager am shift n1 cnt1 n2 cnt2 ^Box transferred-leaves]
--- 1418,1443 ----
rngs (if (.regular nm node)
(ints (regular-ranges shift cnt))
(ranges nm node))
! cs (if rngs (aget rngs 32) 32)
! cseq (fn cseq [[c r]]
(let [arr (.array nm c)
rngs (if (.regular nm c)
(ints (regular-ranges (- shift 5) r))
(ranges nm c))
! gcs (if rngs (aget rngs 32) 32)
! rng-deltas (mapv - rngs (cons 0 rngs))]
! (into [] (comp (take-while (complement nil?))
! (take gcs)
! (map-indexed (fn [idx node]
! [node (rng-deltas idx)])))
! arr)))
! rng-deltas (mapv - rngs (cons 0 rngs))]
! (into [] (comp (take-while (complement nil?))
! (take cs)
! (map-indexed (fn [idx node] [node (rng-deltas idx)]))
! (map cseq)
! cat)
! arr)))
(defn rebalance
[^NodeManager nm ^ArrayManager am shift n1 cnt1 n2 cnt2 ^Box transferred-leaves]
***************
*** 1407,1415 ****
new-rngs (int-array 33)
new-n1 (.node nm nil new-arr)]
(loop [i 0
! bs (partition-all 32
! (concat (child-seq nm n1 shift cnt1)
! (child-seq nm n2 shift cnt2)))]
(when-first [block bs]
(let [a (object-array 33)
r (int-array 33)]
--- 1459,1469 ----
new-rngs (int-array 33)
new-n1 (.node nm nil new-arr)]
(loop [i 0
! bs (into [] (comp (map (fn [[node cnt]]
! (child-seq nm node shift cnt)))
! cat
! (partition-all 32))
! [[n1 cnt1] [n2 cnt2]])]
(when-first [block bs]
(let [a (object-array 33)
r (int-array 33)]
***************
*** 1438,1446 ****
new-n1 (.node nm nil new-arr1)
new-n2 (.node nm nil new-arr2)]
(loop [i 0
! bs (partition-all 32
! (concat (child-seq nm n1 shift cnt1)
! (child-seq nm n2 shift cnt2)))]
(when-first [block bs]
(let [a (object-array 33)
r (int-array 33)]
--- 1492,1502 ----
new-n1 (.node nm nil new-arr1)
new-n2 (.node nm nil new-arr2)]
(loop [i 0
! bs (into [] (comp (map (fn [[node cnt]]
! (child-seq nm node shift cnt)))
! cat
! (partition-all 32))
! [[n1 cnt1] [n2 cnt2]])]
(when-first [block bs]
(let [a (object-array 33)
r (int-array 33)]
***************
*** 1542,1576 ****
(def peephole-optimization-config (atom {:debug-fn nil}))
(def peephole-optimization-count (atom 0))
! ;; TBD: Transducer versions of child-nodes and bounded-grandchildren
! ;; are included here for when we are willing to rely upon Clojure
! ;; 1.7.0 as the minimum version supported by the core.rrb-vector
! ;; library. They are faster.
!
! #_(defn child-nodes [node ^NodeManager nm]
(into [] (comp (take-while (complement nil?))
(take 32))
(.array nm node)))
- (defn child-nodes [node ^NodeManager nm]
- (->> (.array nm node)
- (take-while (complement nil?))
- (take 32)))
-
;; (take 33) is just a technique to avoid generating more
;; grandchildren than necessary. If there are at least 33, we do not
;; care how many there are.
! #_(defn bounded-grandchildren [nm children]
(into [] (comp (map #(child-nodes % nm))
cat
(take 33))
children))
- (defn bounded-grandchildren [nm children]
- (->> children
- (mapcat #(child-nodes % nm))
- (take 33)))
-
;; TBD: Do functions like last-non-nil-idx and
;; count-vector-elements-beneath already exist elsewhere in this
;; library? It seems like they might.
--- 1598,1617 ----
(def peephole-optimization-config (atom {:debug-fn nil}))
(def peephole-optimization-count (atom 0))
! (defn child-nodes [node ^NodeManager nm]
(into [] (comp (take-while (complement nil?))
(take 32))
(.array nm node)))
;; (take 33) is just a technique to avoid generating more
;; grandchildren than necessary. If there are at least 33, we do not
;; care how many there are.
! (defn bounded-grandchildren [nm children]
(into [] (comp (map #(child-nodes % nm))
cat
(take 33))
children))
;; TBD: Do functions like last-non-nil-idx and
;; count-vector-elements-beneath already exist elsewhere in this
;; library? It seems like they might.
================================================
FILE: epl-v10.html
================================================
<?xml version="1.0" encoding="ISO-8859-1" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1" />
<title>Eclipse Public License - Version 1.0</title>
<style type="text/css">
body {
size: 8.5in 11.0in;
margin: 0.25in 0.5in 0.25in 0.5in;
tab-interval: 0.5in;
}
p {
margin-left: auto;
margin-top: 0.5em;
margin-bottom: 0.5em;
}
p.list {
margin-left: 0.5in;
margin-top: 0.05em;
margin-bottom: 0.05em;
}
</style>
</head>
<body lang="EN-US">
<p align=center><b>Eclipse Public License - v 1.0</b></p>
<p>THE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THIS ECLIPSE
PUBLIC LICENSE ("AGREEMENT"). ANY USE, REPRODUCTION OR
DISTRIBUTION OF THE PROGRAM CONSTITUTES RECIPIENT'S ACCEPTANCE OF THIS
AGREEMENT.</p>
<p><b>1. DEFINITIONS</b></p>
<p>"Contribution" means:</p>
<p class="list">a) in the case of the initial Contributor, the initial
code and documentation distributed under this Agreement, and</p>
<p class="list">b) in the case of each subsequent Contributor:</p>
<p class="list">i) changes to the Program, and</p>
<p class="list">ii) additions to the Program;</p>
<p class="list">where such changes and/or additions to the Program
originate from and are distributed by that particular Contributor. A
Contribution 'originates' from a Contributor if it was added to the
Program by such Contributor itself or anyone acting on such
Contributor's behalf. Contributions do not include additions to the
Program which: (i) are separate modules of software distributed in
conjunction with the Program under their own license agreement, and (ii)
are not derivative works of the Program.</p>
<p>"Contributor" means any person or entity that distributes
the Program.</p>
<p>"Licensed Patents" mean patent claims licensable by a
Contributor which are necessarily infringed by the use or sale of its
Contribution alone or when combined with the Program.</p>
<p>"Program" means the Contributions distributed in accordance
with this Agreement.</p>
<p>"Recipient" means anyone who receives the Program under
this Agreement, including all Contributors.</p>
<p><b>2. GRANT OF RIGHTS</b></p>
<p class="list">a) Subject to the terms of this Agreement, each
Contributor hereby grants Recipient a non-exclusive, worldwide,
royalty-free copyright license to reproduce, prepare derivative works
of, publicly display, publicly perform, distribute and sublicense the
Contribution of such Contributor, if any, and such derivative works, in
source code and object code form.</p>
<p class="list">b) Subject to the terms of this Agreement, each
Contributor hereby grants Recipient a non-exclusive, worldwide,
royalty-free patent license under Licensed Patents to make, use, sell,
offer to sell, import and otherwise transfer the Contribution of such
Contributor, if any, in source code and object code form. This patent
license shall apply to the combination of the Contribution and the
Program if, at the time the Contribution is added by the Contributor,
such addition of the Contribution causes such combination to be covered
by the Licensed Patents. The patent license shall not apply to any other
combinations which include the Contribution. No hardware per se is
licensed hereunder.</p>
<p class="list">c) Recipient understands that although each Contributor
grants the licenses to its Contributions set forth herein, no assurances
are provided by any Contributor that the Program does not infringe the
patent or other intellectual property rights of any other entity. Each
Contributor disclaims any liability to Recipient for claims brought by
any other entity based on infringement of intellectual property rights
or otherwise. As a condition to exercising the rights and licenses
granted hereunder, each Recipient hereby assumes sole responsibility to
secure any other intellectual property rights needed, if any. For
example, if a third party patent license is required to allow Recipient
to distribute the Program, it is Recipient's responsibility to acquire
that license before distributing the Program.</p>
<p class="list">d) Each Contributor represents that to its knowledge it
has sufficient copyright rights in its Contribution, if any, to grant
the copyright license set forth in this Agreement.</p>
<p><b>3. REQUIREMENTS</b></p>
<p>A Contributor may choose to distribute the Program in object code
form under its own license agreement, provided that:</p>
<p class="list">a) it complies with the terms and conditions of this
Agreement; and</p>
<p class="list">b) its license agreement:</p>
<p class="list">i) effectively disclaims on behalf of all Contributors
all warranties and conditions, express and implied, including warranties
or conditions of title and non-infringement, and implied warranties or
conditions of merchantability and fitness for a particular purpose;</p>
<p class="list">ii) effectively excludes on behalf of all Contributors
all liability for damages, including direct, indirect, special,
incidental and consequential damages, such as lost profits;</p>
<p class="list">iii) states that any provisions which differ from this
Agreement are offered by that Contributor alone and not by any other
party; and</p>
<p class="list">iv) states that source code for the Program is available
from such Contributor, and informs licensees how to obtain it in a
reasonable manner on or through a medium customarily used for software
exchange.</p>
<p>When the Program is made available in source code form:</p>
<p class="list">a) it must be made available under this Agreement; and</p>
<p class="list">b) a copy of this Agreement must be included with each
copy of the Program.</p>
<p>Contributors may not remove or alter any copyright notices contained
within the Program.</p>
<p>Each Contributor must identify itself as the originator of its
Contribution, if any, in a manner that reasonably allows subsequent
Recipients to identify the originator of the Contribution.</p>
<p><b>4. COMMERCIAL DISTRIBUTION</b></p>
<p>Commercial distributors of software may accept certain
responsibilities with respect to end users, business partners and the
like. While this license is intended to facilitate the commercial use of
the Program, the Contributor who includes the Program in a commercial
product offering should do so in a manner which does not create
potential liability for other Contributors. Therefore, if a Contributor
includes the Program in a commercial product offering, such Contributor
("Commercial Contributor") hereby agrees to defend and
indemnify every other Contributor ("Indemnified Contributor")
against any losses, damages and costs (collectively "Losses")
arising from claims, lawsuits and other legal actions brought by a third
party against the Indemnified Contributor to the extent caused by the
acts or omissions of such Commercial Contributor in connection with its
distribution of the Program in a commercial product offering. The
obligations in this section do not apply to any claims or Losses
relating to any actual or alleged intellectual property infringement. In
order to qualify, an Indemnified Contributor must: a) promptly notify
the Commercial Contributor in writing of such claim, and b) allow the
Commercial Contributor to control, and cooperate with the Commercial
Contributor in, the defense and any related settlement negotiations. The
Indemnified Contributor may participate in any such claim at its own
expense.</p>
<p>For example, a Contributor might include the Program in a commercial
product offering, Product X. That Contributor is then a Commercial
Contributor. If that Commercial Contributor then makes performance
claims, or offers warranties related to Product X, those performance
claims and warranties are such Commercial Contributor's responsibility
alone. Under this section, the Commercial Contributor would have to
defend claims against the other Contributors related to those
performance claims and warranties, and if a court requires any other
Contributor to pay any damages as a result, the Commercial Contributor
must pay those damages.</p>
<p><b>5. NO WARRANTY</b></p>
<p>EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, THE PROGRAM IS
PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
OF ANY KIND, EITHER EXPRESS OR IMPLIED INCLUDING, WITHOUT LIMITATION,
ANY WARRANTIES OR CONDITIONS OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY
OR FITNESS FOR A PARTICULAR PURPOSE. Each Recipient is solely
responsible for determining the appropriateness of using and
distributing the Program and assumes all risks associated with its
exercise of rights under this Agreement , including but not limited to
the risks and costs of program errors, compliance with applicable laws,
damage to or loss of data, programs or equipment, and unavailability or
interruption of operations.</p>
<p><b>6. DISCLAIMER OF LIABILITY</b></p>
<p>EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, NEITHER RECIPIENT
NOR ANY CONTRIBUTORS SHALL HAVE ANY LIABILITY FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING
WITHOUT LIMITATION LOST PROFITS), HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OR
DISTRIBUTION OF THE PROGRAM OR THE EXERCISE OF ANY RIGHTS GRANTED
HEREUNDER, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.</p>
<p><b>7. GENERAL</b></p>
<p>If any provision of this Agreement is invalid or unenforceable under
applicable law, it shall not affect the validity or enforceability of
the remainder of the terms of this Agreement, and without further action
by the parties hereto, such provision shall be reformed to the minimum
extent necessary to make such provision valid and enforceable.</p>
<p>If Recipient institutes patent litigation against any entity
(including a cross-claim or counterclaim in a lawsuit) alleging that the
Program itself (excluding combinations of the Program with other
software or hardware) infringes such Recipient's patent(s), then such
Recipient's rights granted under Section 2(b) shall terminate as of the
date such litigation is filed.</p>
<p>All Recipient's rights under this Agreement shall terminate if it
fails to comply with any of the material terms or conditions of this
Agreement and does not cure such failure in a reasonable period of time
after becoming aware of such noncompliance. If all Recipient's rights
under this Agreement terminate, Recipient agrees to cease use and
distribution of the Program as soon as reasonably practicable. However,
Recipient's obligations under this Agreement and any licenses granted by
Recipient relating to the Program shall continue and survive.</p>
<p>Everyone is permitted to copy and distribute copies of this
Agreement, but in order to avoid inconsistency the Agreement is
copyrighted and may only be modified in the following manner. The
Agreement Steward reserves the right to publish new versions (including
revisions) of this Agreement from time to time. No one other than the
Agreement Steward has the right to modify this Agreement. The Eclipse
Foundation is the initial Agreement Steward. The Eclipse Foundation may
assign the responsibility to serve as the Agreement Steward to a
suitable separate entity. Each new version of the Agreement will be
given a distinguishing version number. The Program (including
Contributions) may always be distributed subject to the version of the
Agreement under which it was received. In addition, after a new version
of the Agreement is published, Contributor may elect to distribute the
Program (including its Contributions) under the new version. Except as
expressly stated in Sections 2(a) and 2(b) above, Recipient receives no
rights or licenses to the intellectual property of any Contributor under
this Agreement, whether expressly, by implication, estoppel or
otherwise. All rights in the Program not expressly granted under this
Agreement are reserved.</p>
<p>This Agreement is governed by the laws of the State of New York and
the intellectual property laws of the United States of America. No party
to this Agreement will bring a legal action under this Agreement more
than one year after the cause of action arose. Each party waives its
rights to a jury trial in any resulting litigation.</p>
</body>
</html>
================================================
FILE: pom.xml
================================================
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>core.rrb-vector</artifactId>
<version>0.2.2-SNAPSHOT</version>
<name>core.rrb-vector</name>
<description>RRB-Trees for Clojure(Script) -- see Bagwell & Rompf</description>
<licenses>
<license>
<name>Eclipse Public License 1.0</name>
<url>https://opensource.org/license/epl-1-0/</url>
<distribution>repo</distribution>
</license>
</licenses>
<parent>
<groupId>org.clojure</groupId>
<artifactId>pom.contrib</artifactId>
<version>1.4.0</version>
</parent>
<developers>
<developer>
<name>Michał Marczyk</name>
<url>https://github.com/michalmarczyk</url>
</developer>
</developers>
<scm>
<connection>scm:git:git://github.com/clojure/core.rrb-vector.git</connection>
<developerConnection>scm:git:git://github.com/clojure/core.rrb-vector.git</developerConnection>
<url>https://github.com/clojure/core.rrb-vector</url>
<tag>HEAD</tag>
</scm>
<properties>
<clojure.version>1.11.4</clojure.version>
<clojure.warnOnReflection>true</clojure.warnOnReflection>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<executions combine.children="append">
<execution>
<id>add-clojurescript-source-dirs</id>
<phase>generate-sources</phase>
<goals>
<goal>add-resource</goal>
</goals>
<configuration>
<resources>
<resource>
<directory>src/main/cljs</directory>
</resource>
</resources>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
<testResources>
<testResource>
<directory>${project.basedir}/src/test/resources</directory>
</testResource>
<testResource>
<directory>${project.basedir}/src/test/cljs</directory>
</testResource>
</testResources>
</build>
<dependencies>
<dependency>
<groupId>org.clojure</groupId>
<artifactId>test.check</artifactId>
<version>1.1.3</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.clojure</groupId>
<artifactId>clojurescript</artifactId>
<version>1.10.439</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
================================================
FILE: project.clj
================================================
(defproject org.clojure/core.rrb-vector "0.1.1-SNAPSHOT"
:description "RRB-Trees for Clojure(Script) -- see Bagwell & Rompf"
:url "https://github.com/clojure/core.rrb-vector"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:min-lein-version "2.2.0"
:parent [org.clojure/pom.contrib "1.4.0"]
:dependencies [[org.clojure/clojure "1.11.4"]]
:source-paths ["src/main/clojure" "src/main/cljs"]
;;:source-paths ["src/parameterized/clojure" "src/main/cljs"]
:test-paths ["src/test/clojure"]
:test-selectors {:default (complement :cljs-nashorn)}
:jvm-opts ^:replace ["-XX:+UseG1GC"
"-XX:-OmitStackTraceInFastThrow"]
:profiles {:dev {:dependencies [[org.clojure/test.check "1.1.3"]]
:plugins [[lein-cljsbuild "1.1.7"]]}
:coll {:test-paths ["src/test_local/clojure"]
:dependencies [[org.clojure/test.check "1.1.3"]
[collection-check "0.1.7"]]}
:cljs {:dependencies [[org.clojure/clojure "1.11.4"]
[org.clojure/clojurescript "1.10.238"]]}
:1.9 {:dependencies [[org.clojure/clojure "1.11.4"]]}
:1.10 {:dependencies [[org.clojure/clojure "1.11.4"]]}
:master {:dependencies [[org.clojure/clojure "1.13.0-master-SNAPSHOT"]]}}
:cljsbuild {:builds {:test {:source-paths ["src/main/cljs"
"src/test/cljs"]
:compiler {;;:optimizations :none
;;:optimizations :whitespace
;;:optimizations :simple
:optimizations :advanced
:output-to "out/test.js"}}}
:test-commands
{"node" ["node" "-e"
"require(\"./out/test\"); clojure.core.rrb_vector.test_cljs.run()"]
"spidermonkey" ["js52" "-f" "out/test.js"
"--execute=clojure.core.rrb_vector.test_cljs.run()"]}})
================================================
FILE: script/jdo
================================================
#! /bin/bash
# Run some task using Clojure/Java
if [ $# -eq 0 ]
then
# Default if nothing else is specified is a REPL plus listening
# for Socket REPL
TASK="socket"
elif [ $# -eq 1 ]
then
TASK="$1"
else
1>&2 echo "usage: `basename $0` arg1"
exit 1
fi
set -x
case ${TASK} in
sock*)
# Run REPL, with option to listen for Socket REPL connection, and
# test paths in classpath.
exec clj -A:clj:clj-test:clj-socket ;;
test*)
# Run 'short' tests
exec clojure -A:clj:clj-test:clj-runt ;;
chec*)
# Run 'short' tests with extra checks enabled
exec clojure -A:clj:clj-test:clj-extrachecks-runt ;;
long*)
# Run long/generative tests
exec clojure -A:clj:clj-test:clj-runlongtests ;;
coll*)
# Run collection-check generative tests
exec clojure -A:clj:clj-test:clj-check:clj-runcheck ;;
perf*)
# Run performance tests
exec clojure -A:clj:clj-test:clj-runperf ;;
focu*)
# Run whatever the current 'focus' tests are
exec clojure -A:clj:clj-test:clj-check:clj-runfocus ;;
east*|lint*)
# Run Eastwood
exec clojure -A:clj:clj-test:clj-check:eastwood ;;
*)
1>&2 echo "unknown task name: ${TASK}"
exit 1 ;;
esac
================================================
FILE: script/mvn-run-tests
================================================
#! /bin/bash
# Example of a command run by the build.clojure.org Jenkins machine to
# run Clojure/Java and ClojureScript tests for data.xml library:
# /var/lib/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.2.5/bin/mvn "-Djdk=Oracle 11 EA" -DCLOJURE_VERSION=1.7.0 -Dclojure.version=1.7.0 clean test
prog_name=`basename $0`
usage() {
1>&2 echo "usage: $prog_name <clojure_version>"
1>&2 echo ""
1>&2 echo "Examples:"
1>&2 echo ""
1>&2 echo " $prog_name 1.7.0"
1>&2 echo " $prog_name 1.10.1"
}
if [ $# -ne 1 ]
then
usage
exit 1
fi
CLOJURE_VERSION="$1"
set -x
mvn -DCLOJURE_VERSION=${CLOJURE_VERSION} -Dclojure.version=${CLOJURE_VERSION} clean test
================================================
FILE: script/replace-params
================================================
#!/bin/bash
#_(
#_DEPS is same format as deps.edn. Multiline is okay.
DEPS='{:deps {
org.clojure/clojure {:mvn/version "1.10.1"}
}}'
#_You can put other options here
OPTS='-J-XX:-OmitStackTraceInFastThrow'
exec clojure $OPTS -Sdeps "$DEPS" "$0" "$@"
)
;; For every file in this directory and its subdirectories:
;; src/parameterized/clojure/clojure/core/rrb_vector/<path_name>
;; create a corresponding file:
;; src/hardcoded/clojure/clojure/core/rrb_vector/<path_name>
;; that has the same contents, except for the string substitutions
;; specified in the var named 'substitutions' throughout the file,
;; wherever they occur.
(require '[clojure.java.io :as io]
'[clojure.java.shell :as sh]
'[clojure.edn :as edn]
'[clojure.string :as str]
'[clojure.pprint :as pp])
(def substitutions [
;; some comments have special cases like this,
;; which I want to replace before the other more
;; normal cases below.
["1/(p/max-branches)" "1/32"]
;;["by max-branches-squared" "by 1024"]
["1/max-branches-squared" "1/1024"]
["1/(p/max-branches-squared)" "1/1024"]
;; special case expression that appears in a few
;; places in the code.
["(inc p/max-branches)" "33"]
;; Note that these must be replaced before
;; p/max-branches is, because the substitution
;; code below does not know anything about
;; symbols, just raw sequences of characters. All
;; of these are suffixes of p/max-branches.
["p/max-branches-squared" "1024"]
["p/max-branches-minus-1" "31"]
["p/max-branches-minus-2" "30"]
;; Similarly, shift-increment-times-2 must be
;; replaced before shift-increment.
["p/shift-increment-times-2" "10"]
["p/shift-increment" "5"]
["p/max-branches" "32"]
["p/branch-mask" "0x1f"]
["p/non-regular-array-len" "33"]
["max-capacity-divided-by-max-branches-squared"
"max-capacity-divided-by-1024"]
])
(def source-dir-prefix "src/parameterized/")
(def target-dir-prefix "src/hardcoded/")
(def common-intermediate-path "clojure/clojure/core")
(def source-dir (str source-dir-prefix common-intermediate-path))
(def target-dir (str target-dir-prefix common-intermediate-path))
(defn source-to-target-name [source-fname]
(let [expected-path? (str/starts-with? source-fname source-dir)]
(if expected-path?
(str target-dir (subs source-fname (count source-dir)))
(throw (ex-info (format "Unexpected source path '%s' does not begin with '%s'"
source-fname source-dir)
{:source-fname source-fname
:source-dir source-dir})))))
(defn make-all-substitutions [content substitution-pairs]
(reduce (fn [content [to-replace-str replace-with-str]]
(str/replace content to-replace-str replace-with-str))
content
substitution-pairs))
(let [source-dir (str source-dir-prefix common-intermediate-path)]
(doseq [source-f (file-seq (io/file source-dir))]
(let [source-fname (str source-f)
target-fname (source-to-target-name source-fname)
target-f (io/file target-fname)]
(println)
(println "source file :" source-fname)
(println "target file1:" target-fname)
(println "target file2:" (str target-f))
(if (. source-f (isDirectory))
(println "skipping directory")
(do
(io/make-parents target-f)
(let [contents (slurp source-f)
new-contents (make-all-substitutions contents substitutions)]
(spit target-f new-contents)))))))
================================================
FILE: script/sdo
================================================
#! /bin/bash
# Run some task using ClojureScript
if [ $# -eq 0 ]
then
# Default if nothing else is specified is a REPL plus listening
# for Socket REPL
TASK="socket"
elif [ $# -eq 1 ]
then
TASK="$1"
else
1>&2 echo "usage: `basename $0` arg1"
exit 1
fi
set -x
case ${TASK} in
sock*)
# Run REPL, with option to listen for Socket REPL connection, and
# test paths in classpath.
exec clj -A:cljs:cljs-test:cljs-socket ;;
test*)
# Run 'short' tests
exec clojure -A:cljs:cljs-test:cljs-runt ;;
chec*)
# Run 'short' tests with extra checks enabled
exec clojure -A:cljs:cljs-test:cljs-extrachecks-runt ;;
long*)
# Run long/generative tests
exec clojure -A:cljs:cljs-test:cljs-runlongtests ;;
coll*)
# Run collection-check generative tests
exec clojure -A:cljs:cljs-test:cljs-check:cljs-runcheck ;;
perf*)
# Run performance tests
exec clojure -A:cljs:cljs-test:cljs-runperf ;;
focu*)
# Run whatever the current 'focus' tests are
exec clojure -A:cljs:cljs-test:cljs-check:cljs-runfocus ;;
east*|lint*)
1>&2 echo "Eastwood not supported for ClojureScript" ;;
*)
1>&2 echo "unknown task name: ${TASK}"
exit 1 ;;
esac
================================================
FILE: script/test
================================================
#!/bin/bash
# See README.md for some sample install instructions for Ubuntu 18.04
# Linux and macOS. If you use those instructions, then the following environment variable settings should work:
# export NODEJS_CMD="node"
# export SPIDERMONKEY_CMD="js52"
if [ "${NODEJS_CMD}" = "" -a "${SPIDERMONKEY_CMD}" = "" ]; then
echo "Neither NODEJS_CMD nor SPIDERMONKEY_CMD is set, cannot run tests"
exit 1
fi
rm -rf out
mkdir -p out
lein with-profile +cljs cljsbuild once test
echo "Launching test runner..."
if [ "${NODEJS_CMD}" != "" ]; then
echo "Testing with Node.js:"
"${NODEJS_CMD}" -e 'require("./out/test"); clojure.core.rrb_vector.test_cljs.run()'
fi
if [ "${SPIDERMONKEY_CMD}" != "" ]; then
echo "Testing with SpiderMonkey:"
"${SPIDERMONKEY_CMD}" -f out/test.js "--execute=clojure.core.rrb_vector.test_cljs.run()"
fi
================================================
FILE: src/main/cljs/clojure/core/rrb_vector/debug.cljs
================================================
; Copyright (c) Rich Hickey and contributors. All rights reserved.
; The use and distribution terms for this software are covered by the
; Eclipse Public License 1.0 (http://opensource.org/licenses/eclipse-1.0.php)
; which can be found in the file epl-v10.html at the root of this distribution.
; By using this software in any fashion, you are agreeing to be bound by
; the terms of this license.
; You must not remove this notice, or any other, from this software.
(ns clojure.core.rrb-vector.debug
(:require [clojure.core.rrb-vector :as fv]
[clojure.core.rrb-vector.rrbt :as rrbt]
;; This page:
;; https://clojure.org/guides/reader_conditionals refers
;; to code that can go into common cljc files as platform
;; independent, and the code in the clj or cljs files as
;; platform dependent, so I will use that terminology
;; here, too.
[clojure.core.rrb-vector.debug-platform-dependent :as pd]))
;; The intent is to keep this file as close to
;; src/main/clojure/clojure/core/rrb_vector/debug.clj as possible, so
;; that when we start requiring Clojure 1.7.0 and later for this
;; library, this file and that one can be replaced with a common file
;; with the suffix .cljc
;; Functions expected to be defined in the appropriate
;; clojure.core.rrb-vector.debug-platform-dependent namespace:
;; pd/internal-node?
;; pd/persistent-vector?
;; pd/transient-vector?
;; pd/is-vector?
;; pd/dbg-tailoff (formerly debug-tailoff)
;; pd/dbg-tidx (formerly debug-tailoff for clj, debug-tidx for cljs)
;; pd/format
;; pd/printf
;; pd/unwrap-subvec-accessors-for
;; pd/abbrev-for-type-of [vec-or-node] (formerly abbrev-type-name, but move type/class call inside)
;; pd/same-coll? (written already for clj, TBD for cljs)
;; Functions returned from unwrap-subvec-accessors-for that have
;; platform-dependent definitions, but the same general 'kind'
;; arguments and return values, where 'kind' could be: any vector,
;; persistent or transient, or a vector tree node object:
;; get-root - All get-* fns formerly called extract-* in the Java
;; platform dependent version of the debug namespace.
;; get-shift
;; get-tail
;; get-cnt
;; get-array [node] - clj (.array nm node) cljs (.-arr node)
;; get-ranges [node] - clj (ranges nm node) cljs (node-ranges node)
;; regular? [node] - clj (.regular nm node) cljs (regular? node)
;; tail-len [tail] - clj (.alength am tail) cljs (alength tail)
;; NO: nm am - cljs doesn't need them, and clj only uses them for the
;; last few functions above.
(defn children-summary [node shift get-array get-ranges regular? opts]
(let [children (get-array node)
reg? (regular? node)
rngs (if-not reg? (get-ranges node))
array-len (count children)
children-seq (if reg? children (butlast children))
non-nils (remove nil? children-seq)
regular-children (filter regular? non-nils)
num-non-nils (count non-nils)
num-regular-children (count regular-children)
num-irregular-children (- num-non-nils num-regular-children)
num-nils (- (count children-seq) num-non-nils)
exp-array-len (if reg? 32 33)
bad-array-len? (not= array-len exp-array-len)]
;; 'r' for regular, 'i' for irregular
;; For either type of node, its first 32 array elements are broken
;; down into:
;; # regular children
;; # irregular children
;; # of nil 'children' not shown, since it will always be 32 minus
;; # the total of # regular plus irregular children, unless the
;; # array is the wrong size, and in that case a BAD-ARRAY-LEN
;; # message will be included in the string.
(pd/format "%s%d+%d%s" (if reg? "r" "i")
num-regular-children
num-irregular-children
(if bad-array-len?
(pd/format " BAD-ARRAY-LEN %d != %d" array-len exp-array-len)
""))))
(defn filter-indexes
"Return a sequence of all indexes of elements e of coll for
which (pred e) returns logical true. 0 is the index of the first
element."
[pred coll]
(filter (complement nil?)
(map-indexed (fn [idx e]
(if (pred e)
idx))
coll)))
(defn dbg-vec
([v]
(dbg-vec v {:max-depth nil ;; integer to limit depth, nil for unlimited
;; force showing tree "fringes" beyond max-depth
:always-show-fringes false
;; show vector elements. false for only count
:show-elements true
;; show summary of number of children of each node, as
;; returned by function children-summary
:show-children-summary false
;; default false means show ranges arrays with their raw
;; unprocessed contents. Use true to show only the
;; first n elements, where n=(aget (get-ranges node)
;; 32), and to show the 'deltas' between consecutive
;; pairs, e.g. if the original is (32 64 96 0 ... 0 3),
;; then instead show (32 32 32), which, if the data
;; structure is correct, is the number of vector
;; elements reachable through each of the node's 3
;; children.
:show-ranges-as-deltas false}))
([v opts]
(let [{:keys [v subvector? subvec-start subvec-end get-root get-shift
get-tail get-cnt get-array get-ranges regular? tail-len]}
(pd/unwrap-subvec-accessors-for v)
root (get-root v)
shift (get-shift v)
tail (get-tail v)
cnt (get-cnt v)]
(when subvector?
(pd/printf "SubVector from start %d to end %d of vector:\n"
subvec-start subvec-end))
(letfn [(go [indent shift i node on-left-fringe? on-right-fringe?]
(when node
(dotimes [_ indent]
(print " "))
(pd/printf "%02d:%02d %s" shift i (pd/abbrev-for-type-of node))
(if (zero? shift)
;; this node has only vector elements as its children
(if (:show-elements opts)
(print ":" (vec (get-array node)))
(print ":" (count (get-array node))
"vector elements elided"))
;; else this node has only other nodes as its children
(do
(when (:show-children-summary opts)
(print " ")
(print (children-summary node shift get-array get-ranges
regular? opts)))
(if (not (regular? node))
(if (:show-ranges-as-deltas opts)
(let [rngs (get-ranges node)
r (aget rngs 32)
tmp (map - (take r rngs) (take r (cons 0 rngs)))]
(print ":" (seq tmp)))
(print ":" (seq (get-ranges node)))))))
(println)
(let [no-children? (zero? shift)
visit-all-children? (and (not no-children?)
(or (nil? (:max-depth opts))
(< (inc indent)
(:max-depth opts))))
visit-some-children? (or visit-all-children?
(and (not no-children?)
(:always-show-fringes opts)
(or on-left-fringe?
on-right-fringe?)))]
(if visit-some-children?
(dorun
(let [arr (get-array node)
a (if (regular? node) arr (butlast arr))
non-nil-idxs (filter-indexes (complement nil?) a)
first-non-nil-idx (first non-nil-idxs)
last-non-nil-idx (last non-nil-idxs)]
(map-indexed
(fn [i node]
(let [child-on-left-fringe?
(and on-left-fringe? (= i first-non-nil-idx))
child-on-right-fringe?
(and on-right-fringe? (= i last-non-nil-idx))
visit-this-child?
(or visit-all-children?
(and (:always-show-fringes opts)
(or child-on-left-fringe?
child-on-right-fringe?)))]
(if visit-this-child?
(go (inc indent) (- shift 5) i node
child-on-left-fringe?
child-on-right-fringe?))))
a)))))))]
(pd/printf "%s (%d elements):\n" (pd/abbrev-for-type-of v) (count v))
(go 0 shift 0 root true true)
(println (if (pd/transient-vector? v)
(pd/format "tail (tidx %d):" (pd/dbg-tidx v))
"tail:")
(vec tail))))))
(defn first-diff
"Compare two sequences to see if they have = elements in the same
order, and both sequences have the same number of elements. If all
of those conditions are true, and no exceptions occur while calling
seq, first, and next on the seqs of xs and ys, then return -1.
If two elements at the same index in each sequence are found not =
to each other, or the sequences differ in their number of elements,
return the index, 0 or larger, at which the first difference occurs.
If an exception occurs while calling seq, first, or next, throw an
exception that contains the index at which this exception occurred."
[xs ys]
(loop [i 0 xs (seq xs) ys (seq ys)]
(if (try (and xs ys (= (first xs) (first ys)))
(catch js/Error e
(.printStackTrace e)
i))
(let [xs (try (next xs)
(catch js/Error e
(prn :xs i)
(throw e)))
ys (try (next ys)
(catch js/Error e
(prn :ys i)
(throw e)))]
(recur (inc i) xs ys))
(if (or xs ys)
i
-1))))
;; When using non-default parameters for the tree data structure,
;; e.g. shift-increment not 5, then in test code with calls to
;; checking-* functions, they will be expecting those same non-default
;; parameter values, and will give errors if they are ever given a
;; vector returned by clojure.core/vec, because without changes to
;; Clojure itself, they always have shift-increment 5 and max-branches
;; 32.
;;
;; If we use (fv/vec coll) consistently in the test code, that in many
;; cases returns a core.rrb-vector data structure, but if given a
;; Clojure vector, it still returns that Clojure vector unmodified,
;; which has the same issues for checking-* functions. By
;; calling (fv/vec (seq coll)) when not using default parameters, we
;; force the return value of cvec to always be a core.rrb-vector data
;; structure.
;;
;; The name 'cvec' is intended to mean "construct a vector", and only
;; intended for use in test code that constructs vectors used as
;; parameters to other functions operating on vectors.
(defn cvec [coll]
(clojure.core/vec coll))
(defn slow-into [to from]
(reduce conj to from))
(defn all-vector-tree-nodes [v]
(let [{:keys [v get-root get-shift get-array regular?]}
(pd/unwrap-subvec-accessors-for v)
root (get-root v)
shift (get-shift v)]
(letfn [(go [depth shift node]
(if node
(if (not= shift 0)
(cons
{:depth depth :shift shift :kind :internal :node node}
(apply concat
(map (partial go (inc depth) (- shift 5))
(let [arr (get-array node)]
(if (regular? node)
arr
(butlast arr))))))
(cons {:depth depth :shift shift :kind :internal :node node}
(map (fn [x]
{:depth (inc depth) :kind :leaf :value x})
(get-array node))))))]
(cons {:depth 0 :kind :base :shift shift :value v}
(go 1 shift root)))))
;; All nodes that should be internal nodes are one of the internal
;; node types satisfying internal-node? All nodes that are less
;; than "leaf depth" must be internal nodes, and none of the ones
;; at "leaf depth" should be. Probably the most general restriction
;; checking for leaf values should be simply that they are any type
;; that is _not_ an internal node type. They could be objects that
;; return true for is-vector? for example, if a vector is an element
;; of another vector.
(defn leaves-with-internal-node-type [node-infos]
(filter (fn [node-info]
(and (= :leaf (:kind node-info))
(pd/internal-node? (:node node-info))))
node-infos))
(defn non-leaves-not-internal-node-type [node-infos]
(filter (fn [node-info]
(and (= :internal (:kind node-info))
(not (pd/internal-node? (:node node-info)))))
node-infos))
;; The definition of nth in deftype Vector implies that every
;; descendant of a 'regular' node must also be regular. That would be
;; a straightforward sanity check to make, to return an error if a
;; non-regular node is found with a regular ancestor in the tree.
(defn basic-node-errors [v]
(let [{:keys [v get-shift]} (pd/unwrap-subvec-accessors-for v)
shift (get-shift v)
nodes (all-vector-tree-nodes v)
by-kind (group-by :kind nodes)
leaf-depths (set (map :depth (:leaf by-kind)))
expected-leaf-depth (+ (quot shift 5) 2)
max-internal-node-depth (->> (:internal by-kind)
(map :depth)
(apply max))
;; Be a little loose in checking here. If we want to narrow
;; it down to one expected answer, we would need to look at
;; the tail to see how many elements it has, then use the
;; different between (count v) and that to determine how many
;; nodes are in the rest of the tree, whether it is 0 or
;; non-0.
expected-internal-max-depths
(cond
(= (count v) 0) #{(- expected-leaf-depth 2)}
(> (count v) 33) #{(dec expected-leaf-depth)}
:else #{(dec expected-leaf-depth)
(- expected-leaf-depth 2)})]
(cond
(not= (mod shift 5) 0)
{:error true
:description (str "shift value in root must be a multiple of 5. Found "
shift)
:data shift}
;; It is OK for this set size to be 0 if no leaves, but if there
;; are leaves, they should all be at the same depth.
(> (count leaf-depths) 1)
{:error true
:description (str "There are leaf nodes at multiple different depths: "
leaf-depths)
:data leaf-depths}
(and (= (count leaf-depths) 1)
(not= (first leaf-depths) expected-leaf-depth))
{:error true
:description (str "Expecting all leaves to be at depth " expected-leaf-depth
" because root has shift=" shift
" but found leaves at depth " (first leaf-depths))
:data leaf-depths}
(not (contains? expected-internal-max-depths max-internal-node-depth))
{:error true
:description (str "Expecting there to be some internal nodes at one of"
" these depths: "
expected-internal-max-depths
" because count=" (count v)
" and root has shift=" shift
" but max depth among all internal nodes found was "
max-internal-node-depth)}
(seq (leaves-with-internal-node-type nodes))
{:error true
:description "A leaf (at max depth) has one of the internal node types, returning true for internal-node?"
:data (first (leaves-with-internal-node-type nodes))}
(seq (non-leaves-not-internal-node-type nodes))
{:error true
:description "A non-leaf node has a type that returns false for internal-node?"
:data (first (non-leaves-not-internal-node-type nodes))}
:else
{:error false})))
;; I believe that objects-in-slot-32-of-obj-arrays and
;; ranges-not-int-array are only called directly from one test
;; namespace right now. Consider making a combined invariant checking
;; function in this debug namespace that can be used from any test
;; namespace (or other debug-time code) that a developer wants to.
(defn objects-in-slot-32-of-obj-arrays
"Function to look for errors of the form where a node's node.array
object, which is often an array of 32 or 33 java.lang.Object's, has
an element at index 32 that is not nil, and refers to an object that
is of any type _except_ an array of ints. There appears to be some
situation in which this can occur, but it seems to almost certainly
be a bug if that happens, and we should be able to detect it
whenever it occurs."
[v]
(let [{:keys [v get-array]} (pd/unwrap-subvec-accessors-for v)
node-maps (all-vector-tree-nodes v)
internal (filter #(= :internal (:kind %)) node-maps)]
(keep (fn [node-info]
;; TBD: Is there a way to do ^objects type hint for clj,
;; but none for cljs? Is it harmful for cljs to have such
;; a type hint?
;;(let [^objects arr (get-array (:node node-info))
(let [arr (get-array (:node node-info))
n (count arr)]
(if (== n 33)
(aget arr 32))))
internal)))
;; TBD: Should this function be defined in platform-specific file?
;;(defn ranges-not-int-array [x]
;; (seq (remove int-array? (objects-in-slot-32-of-obj-arrays x))))
;; edit-nodes-errors is completely defined in platform-specific source
;; files. It is simply quite different between clj/cljs.
(defn edit-nodes-errors [v]
(pd/edit-nodes-errors v all-vector-tree-nodes))
(defn regular-node-errors [root-node? root-node-cnt children]
;; For regular nodes, there should be zero or more 'full' children,
;; followed optionally by one 'partial' child, followed by nils.
(let [[full-children others] (split-with :full? children)
[partial-children others] (split-with #(and (not (:full? %))
(not= :nil (:kind %)))
others)
[nil-children others] (split-with #(= :nil (:kind %)) others)
num-full (count full-children)
num-partial (count partial-children)
num-non-nil (+ num-full num-partial)]
(cond
(not= 0 (count others))
{:error true, :kind :internal,
:description (str "Found internal regular node with "
num-full " full, " num-partial " partial, "
(count nil-children) " nil, "
(count others) " 'other' children."
" - expected 0 children after nils.")}
(> num-partial 1)
{:error true, :kind :internal,
:description (str "Found internal regular node with "
num-full " full, " num-partial " partial, "
(count nil-children) " nil children"
" - expected 0 or 1 partial.")}
(not (or (and root-node?
(<= root-node-cnt 32) ;; all elements in tail
(= 0 num-non-nil))
(<= 1 num-non-nil 32)))
{:error true, :kind :internal
:description
(str "Found internal regular node with # full + # partial=" num-non-nil
" children outside of range [1, 32]."
" root-node?=" root-node? " root-node-cnt=" root-node-cnt)
:data children}
:else
{:error false, :kind :internal,
:full? (= 32 (count full-children))
:count (reduce + (map #(or (:count %) 0) children))})))
(defn non-regular-node-errors [node get-ranges children]
(let [rng (get-ranges node)
[non-nil-children others] (split-with #(not= :nil (:kind %)) children)
[nil-children others] (split-with #(= :nil (:kind %)) others)
num-non-nil (count non-nil-children)
num-nil (count nil-children)
expected-ranges (reductions + (map :count non-nil-children))]
(cond
(not= 0 (count others))
{:error true, :kind :internal,
:description (str "Fo
gitextract_vrqmmogg/
├── .github/
│ └── workflows/
│ ├── doc-build.yml
│ ├── release.yml
│ ├── snapshot.yml
│ └── test.yml
├── .gitignore
├── CHANGES.md
├── CONTRIBUTING.md
├── LICENSE
├── README.md
├── deps.edn
├── doc/
│ ├── benchmarks/
│ │ ├── benchmarks.md
│ │ └── data/
│ │ ├── benchmarks.edn
│ │ ├── concat.csv
│ │ ├── list_construct.csv
│ │ ├── list_iterate.csv
│ │ └── list_lookup.csv
│ ├── crrbv-27/
│ │ ├── description.md
│ │ ├── proposed-fix-needs-thought-and-testing-plus-debug-prints.patch
│ │ ├── proposed-fix-needs-thought-and-testing.patch
│ │ └── use-shift-increment-2.patch
│ ├── hash-details.md
│ ├── rrb-tree-notes.md
│ └── use-transducers/
│ ├── README.md
│ └── use-transducers.patch
├── epl-v10.html
├── pom.xml
├── project.clj
├── script/
│ ├── jdo
│ ├── mvn-run-tests
│ ├── replace-params
│ ├── sdo
│ └── test
└── src/
├── main/
│ ├── cljs/
│ │ └── clojure/
│ │ └── core/
│ │ ├── rrb_vector/
│ │ │ ├── debug.cljs
│ │ │ ├── debug_platform_dependent.cljs
│ │ │ ├── interop.cljs
│ │ │ ├── macros.clj
│ │ │ ├── nodes.cljs
│ │ │ ├── protocols.cljs
│ │ │ ├── rrbt.cljs
│ │ │ ├── transients.cljs
│ │ │ └── trees.cljs
│ │ └── rrb_vector.cljs
│ └── clojure/
│ └── clojure/
│ └── core/
│ ├── rrb_vector/
│ │ ├── debug.clj
│ │ ├── debug_platform_dependent.clj
│ │ ├── fork_join.clj
│ │ ├── interop.clj
│ │ ├── nodes.clj
│ │ ├── parameters.clj
│ │ ├── protocols.clj
│ │ ├── rrbt.clj
│ │ └── transients.clj
│ └── rrb_vector.clj
├── parameterized/
│ └── clojure/
│ └── clojure/
│ └── core/
│ ├── rrb_vector/
│ │ ├── debug.clj
│ │ ├── debug_platform_dependent.clj
│ │ ├── fork_join.clj
│ │ ├── interop.clj
│ │ ├── nodes.clj
│ │ ├── parameters.clj
│ │ ├── protocols.clj
│ │ ├── rrbt.clj
│ │ └── transients.clj
│ └── rrb_vector.clj
├── test/
│ ├── cljs/
│ │ └── clojure/
│ │ └── core/
│ │ └── rrb_vector/
│ │ ├── long_test.cljs
│ │ ├── test_cljs.cljs
│ │ ├── test_cljs_only.cljs
│ │ ├── test_common.cljs
│ │ └── test_utils.cljs
│ ├── clojure/
│ │ └── clojure/
│ │ └── core/
│ │ └── rrb_vector/
│ │ ├── long_test.clj
│ │ ├── test_clj_only.clj
│ │ ├── test_cljs.clj
│ │ ├── test_common.clj
│ │ └── test_utils.clj
│ └── resources/
│ └── clojure/
│ └── core/
│ └── rrb_vector/
│ └── cljs_testsuite.clj
└── test_local/
└── clojure/
└── clojure/
└── core/
└── rrb_vector_check.clj
Condensed preview — 74 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (804K chars).
[
{
"path": ".github/workflows/doc-build.yml",
"chars": 228,
"preview": "name: Build API Docs\n\npermissions:\n contents: write\n\non:\n workflow_dispatch:\n\njobs:\n call-doc-build-workflow:\n use"
},
{
"path": ".github/workflows/release.yml",
"chars": 528,
"preview": "name: Release on demand\n\npermissions:\n contents: write\n\non:\n workflow_dispatch:\n inputs:\n releaseVersion:\n "
},
{
"path": ".github/workflows/snapshot.yml",
"chars": 191,
"preview": "name: Snapshot on demand\n\npermissions:\n contents: read\n\non: [workflow_dispatch]\n\njobs:\n call-snapshot:\n uses: cloju"
},
{
"path": ".github/workflows/test.yml",
"chars": 662,
"preview": "name: Test\n\npermissions:\n contents: read\n\non: [push]\n\njobs:\n test:\n strategy:\n matrix:\n os: [ubuntu-lat"
},
{
"path": ".gitignore",
"chars": 168,
"preview": "/target\n/lib\n/classes\n/checkouts\n*.jar\n*.class\n.lein-deps-sum\n.lein-failures\n.lein-plugins\n.lein-repl-history\n/.repl\n/ou"
},
{
"path": "CHANGES.md",
"chars": 6227,
"preview": "# Changes in 0.2.1\n\n* Update parent pom and default Clojure version to 1.11.4\n\n# Changes in 0.2.0\n\n* Update dependencies"
},
{
"path": "CONTRIBUTING.md",
"chars": 479,
"preview": "This is a [Clojure contrib] project.\n\nUnder the Clojure contrib [guidelines], this project cannot accept\npull requests. "
},
{
"path": "LICENSE",
"chars": 11516,
"preview": "Eclipse Public License - v 1.0\n\nTHE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THIS ECLIPSE PUBLIC\nLICENSE (\"AG"
},
{
"path": "README.md",
"chars": 15754,
"preview": "# core.rrb-vector\n\nWhy would anyone want to use this library? The two primary answers\nare:\n\n+ You want faster concatena"
},
{
"path": "deps.edn",
"chars": 8145,
"preview": ";; See shell scripts './script/sdo' and './script/jdo' for sample\n;; useful combinations of aliases to use to acommplish"
},
{
"path": "doc/benchmarks/benchmarks.md",
"chars": 6501,
"preview": "# core.rrb-vector benchmarks\n\nSee the section \"How the benchmarks were run\" below for how the\nresults were created.\n\n\n##"
},
{
"path": "doc/benchmarks/data/benchmarks.edn",
"chars": 28794,
"preview": "{\"bifurcan.List\" {100000 {:intersection nil, :remove nil, :insert nil, :lookup 1746262, :concat 297, :equals nil, :itera"
},
{
"path": "doc/benchmarks/data/concat.csv",
"chars": 1786,
"preview": "size,bifurcan.List,java.ArrayList,clojure.PersistentVector,vavr.Vector,scala.Vector,paguro.RrbTree,bifurcan.LinearList,c"
},
{
"path": "doc/benchmarks/data/list_construct.csv",
"chars": 1748,
"preview": "size,bifurcan.List,java.ArrayList,clojure.PersistentVector,vavr.Vector,scala.Vector,paguro.RrbTree,bifurcan.LinearList,c"
},
{
"path": "doc/benchmarks/data/list_iterate.csv",
"chars": 1736,
"preview": "size,bifurcan.List,java.ArrayList,clojure.PersistentVector,vavr.Vector,scala.Vector,paguro.RrbTree,bifurcan.LinearList,c"
},
{
"path": "doc/benchmarks/data/list_lookup.csv",
"chars": 1719,
"preview": "size,bifurcan.List,java.ArrayList,clojure.PersistentVector,vavr.Vector,scala.Vector,paguro.RrbTree,bifurcan.LinearList,c"
},
{
"path": "doc/crrbv-27/description.md",
"chars": 6218,
"preview": "The production version of the Clojure/Java core.rrb-vector library is\nin the directory `src/main/clojure`. It uses a ma"
},
{
"path": "doc/crrbv-27/proposed-fix-needs-thought-and-testing-plus-debug-prints.patch",
"chars": 20097,
"preview": "diff --git a/src/parameterized/clojure/clojure/core/rrb_vector/debug.clj b/src/parameterized/clojure/clojure/core/rrb_ve"
},
{
"path": "doc/crrbv-27/proposed-fix-needs-thought-and-testing.patch",
"chars": 1656,
"preview": "diff --git a/src/main/clojure/clojure/core/rrb_vector/rrbt.clj b/src/main/clojure/clojure/core/rrb_vector/rrbt.clj\nindex"
},
{
"path": "doc/crrbv-27/use-shift-increment-2.patch",
"chars": 1123,
"preview": "diff --git a/deps.edn b/deps.edn\nindex 775cc3e..f03df7a 100644\n--- a/deps.edn\n+++ b/deps.edn\n@@ -5,8 +5,8 @@\n ;; want to"
},
{
"path": "doc/hash-details.md",
"chars": 8184,
"preview": "# Background on Clojure collection hash calculation\n\nThe persistent collections included with Clojure are immutable when"
},
{
"path": "doc/rrb-tree-notes.md",
"chars": 6892,
"preview": "# Other implementations and descriptions of RRB trees\n\nNote that most implementations have an associated paper. If they"
},
{
"path": "doc/use-transducers/README.md",
"chars": 598,
"preview": "The patch use-transducers.md was developed around July or August 2019,\nbut not yet included in the production core.rrb-v"
},
{
"path": "doc/use-transducers/use-transducers.patch",
"chars": 17697,
"preview": "diff -cr core.rrb-vector/src/main/cljs/clojure/core/rrb_vector/rrbt.cljs core.rrb-vector-pff4i/src/main/cljs/clojure/cor"
},
{
"path": "epl-v10.html",
"chars": 12917,
"preview": "<?xml version=\"1.0\" encoding=\"ISO-8859-1\" ?>\r\n<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http://www"
},
{
"path": "pom.xml",
"chars": 2721,
"preview": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<project xmlns=\"http://maven.apache.org/POM/4.0.0\" xmlns:xsi=\"http://www.w3.org/2"
},
{
"path": "project.clj",
"chars": 2144,
"preview": "(defproject org.clojure/core.rrb-vector \"0.1.1-SNAPSHOT\"\n :description \"RRB-Trees for Clojure(Script) -- see Bagwell & "
},
{
"path": "script/jdo",
"chars": 1195,
"preview": "#! /bin/bash\n\n# Run some task using Clojure/Java\n\nif [ $# -eq 0 ]\nthen\n # Default if nothing else is specified is a R"
},
{
"path": "script/mvn-run-tests",
"chars": 703,
"preview": "#! /bin/bash\n\n# Example of a command run by the build.clojure.org Jenkins machine to\n# run Clojure/Java and ClojureScrip"
},
{
"path": "script/replace-params",
"chars": 4108,
"preview": "#!/bin/bash\n#_(\n #_DEPS is same format as deps.edn. Multiline is okay.\n DEPS='{:deps {\n org.clojure/"
},
{
"path": "script/sdo",
"chars": 1205,
"preview": "#! /bin/bash\n\n# Run some task using ClojureScript\n\nif [ $# -eq 0 ]\nthen\n # Default if nothing else is specified is a "
},
{
"path": "script/test",
"chars": 849,
"preview": "#!/bin/bash\n\n# See README.md for some sample install instructions for Ubuntu 18.04\n# Linux and macOS. If you use those "
},
{
"path": "src/main/cljs/clojure/core/rrb_vector/debug.cljs",
"chars": 59443,
"preview": "; Copyright (c) Rich Hickey and contributors. All rights reserved.\n; The use and distribution terms for this softwar"
},
{
"path": "src/main/cljs/clojure/core/rrb_vector/debug_platform_dependent.cljs",
"chars": 5763,
"preview": "; Copyright (c) Rich Hickey and contributors. All rights reserved.\n; The use and distribution terms for this softwar"
},
{
"path": "src/main/cljs/clojure/core/rrb_vector/interop.cljs",
"chars": 1126,
"preview": "; Copyright (c) Rich Hickey and contributors. All rights reserved.\n; The use and distribution terms for this softwar"
},
{
"path": "src/main/cljs/clojure/core/rrb_vector/macros.clj",
"chars": 1214,
"preview": "; Copyright (c) Rich Hickey and contributors. All rights reserved.\n; The use and distribution terms for this softwar"
},
{
"path": "src/main/cljs/clojure/core/rrb_vector/nodes.cljs",
"chars": 8054,
"preview": "; Copyright (c) Rich Hickey and contributors. All rights reserved.\n; The use and distribution terms for this softwar"
},
{
"path": "src/main/cljs/clojure/core/rrb_vector/protocols.cljs",
"chars": 632,
"preview": "; Copyright (c) Rich Hickey and contributors. All rights reserved.\n; The use and distribution terms for this softwar"
},
{
"path": "src/main/cljs/clojure/core/rrb_vector/rrbt.cljs",
"chars": 49932,
"preview": "; Copyright (c) Rich Hickey and contributors. All rights reserved.\n; The use and distribution terms for this softwar"
},
{
"path": "src/main/cljs/clojure/core/rrb_vector/transients.cljs",
"chars": 7896,
"preview": "; Copyright (c) Rich Hickey and contributors. All rights reserved.\n; The use and distribution terms for this softwar"
},
{
"path": "src/main/cljs/clojure/core/rrb_vector/trees.cljs",
"chars": 7588,
"preview": "; Copyright (c) Rich Hickey and contributors. All rights reserved.\n; The use and distribution terms for this softwar"
},
{
"path": "src/main/cljs/clojure/core/rrb_vector.cljs",
"chars": 3581,
"preview": "; Copyright (c) Rich Hickey and contributors. All rights reserved.\n; The use and distribution terms for this softwar"
},
{
"path": "src/main/clojure/clojure/core/rrb_vector/debug.clj",
"chars": 59641,
"preview": "; Copyright (c) Rich Hickey and contributors. All rights reserved.\n; The use and distribution terms for this softwar"
},
{
"path": "src/main/clojure/clojure/core/rrb_vector/debug_platform_dependent.clj",
"chars": 12825,
"preview": "; Copyright (c) Rich Hickey and contributors. All rights reserved.\n; The use and distribution terms for this softwar"
},
{
"path": "src/main/clojure/clojure/core/rrb_vector/fork_join.clj",
"chars": 689,
"preview": "; Copyright (c) Rich Hickey and contributors. All rights reserved.\n; The use and distribution terms for this softwar"
},
{
"path": "src/main/clojure/clojure/core/rrb_vector/interop.clj",
"chars": 1388,
"preview": "; Copyright (c) Rich Hickey and contributors. All rights reserved.\n; The use and distribution terms for this softwar"
},
{
"path": "src/main/clojure/clojure/core/rrb_vector/nodes.clj",
"chars": 11116,
"preview": "; Copyright (c) Rich Hickey and contributors. All rights reserved.\n; The use and distribution terms for this softwar"
},
{
"path": "src/main/clojure/clojure/core/rrb_vector/parameters.clj",
"chars": 1276,
"preview": "; Copyright (c) Rich Hickey and contributors. All rights reserved.\n; The use and distribution terms for this softwar"
},
{
"path": "src/main/clojure/clojure/core/rrb_vector/protocols.clj",
"chars": 751,
"preview": "; Copyright (c) Rich Hickey and contributors. All rights reserved.\n; The use and distribution terms for this softwar"
},
{
"path": "src/main/clojure/clojure/core/rrb_vector/rrbt.clj",
"chars": 80786,
"preview": "; Copyright (c) Rich Hickey and contributors. All rights reserved.\n; The use and distribution terms for this softwar"
},
{
"path": "src/main/clojure/clojure/core/rrb_vector/transients.clj",
"chars": 14731,
"preview": "; Copyright (c) Rich Hickey and contributors. All rights reserved.\n; The use and distribution terms for this softwar"
},
{
"path": "src/main/clojure/clojure/core/rrb_vector.clj",
"chars": 5978,
"preview": "; Copyright (c) Rich Hickey and contributors. All rights reserved.\n; The use and distribution terms for this softwar"
},
{
"path": "src/parameterized/clojure/clojure/core/rrb_vector/debug.clj",
"chars": 59883,
"preview": "(ns clojure.core.rrb-vector.debug\n (:require [clojure.core.rrb-vector.parameters :as p]\n [clojure.core.rrb-v"
},
{
"path": "src/parameterized/clojure/clojure/core/rrb_vector/debug_platform_dependent.clj",
"chars": 12372,
"preview": "(ns clojure.core.rrb-vector.debug-platform-dependent\n (:refer-clojure :exclude [format printf])\n (:require [clojure.co"
},
{
"path": "src/parameterized/clojure/clojure/core/rrb_vector/fork_join.clj",
"chars": 208,
"preview": "(ns clojure.core.rrb-vector.fork-join\n (:require [clojure.core.reducers :as r]))\n\n(def pool @#'r/pool)\n(def task @#"
},
{
"path": "src/parameterized/clojure/clojure/core/rrb_vector/interop.clj",
"chars": 907,
"preview": "(ns clojure.core.rrb-vector.interop\n (:require [clojure.core.rrb-vector.protocols\n :refer [PSliceableVector"
},
{
"path": "src/parameterized/clojure/clojure/core/rrb_vector/nodes.clj",
"chars": 12170,
"preview": "(ns clojure.core.rrb-vector.nodes\n (:require [clojure.core.rrb-vector.parameters :as p])\n (:import (clojure.core VecNo"
},
{
"path": "src/parameterized/clojure/clojure/core/rrb_vector/parameters.clj",
"chars": 685,
"preview": "(ns clojure.core.rrb-vector.parameters)\n\n;; The values in comments before each def are the value of that\n;; parameter:\n\n"
},
{
"path": "src/parameterized/clojure/clojure/core/rrb_vector/protocols.clj",
"chars": 270,
"preview": "(ns clojure.core.rrb-vector.protocols)\n\n(defprotocol PSpliceableVector\n (splicev [v1 v2]))\n\n(defprotocol PSliceableVect"
},
{
"path": "src/parameterized/clojure/clojure/core/rrb_vector/rrbt.clj",
"chars": 83985,
"preview": "(ns clojure.core.rrb-vector.rrbt\n (:refer-clojure :exclude [assert ->VecSeq])\n (:require [clojure.core.rrb-vector.para"
},
{
"path": "src/parameterized/clojure/clojure/core/rrb_vector/transients.clj",
"chars": 14959,
"preview": "(ns clojure.core.rrb-vector.transients\n (:require [clojure.core.rrb-vector.parameters :as p]\n [clojure.core."
},
{
"path": "src/parameterized/clojure/clojure/core/rrb_vector.clj",
"chars": 5529,
"preview": "(ns clojure.core.rrb-vector\n\n \"An implementation of the confluently persistent vector data\n structure introduced in Ba"
},
{
"path": "src/test/cljs/clojure/core/rrb_vector/long_test.cljs",
"chars": 4485,
"preview": "(ns clojure.core.rrb-vector.long-test\n (:require [clojure.test :as test :refer [deftest testing is are]]\n [c"
},
{
"path": "src/test/cljs/clojure/core/rrb_vector/test_cljs.cljs",
"chars": 1825,
"preview": "(ns clojure.core.rrb-vector.test-cljs\n (:require [cljs.test :as test]\n [clojure.core.rrb-vector :as fv]\n "
},
{
"path": "src/test/cljs/clojure/core/rrb_vector/test_cljs_only.cljs",
"chars": 376,
"preview": "(ns clojure.core.rrb-vector.test-cljs-only\n (:require [clojure.test :as test :refer [deftest testing is are]]\n "
},
{
"path": "src/test/cljs/clojure/core/rrb_vector/test_common.cljs",
"chars": 25136,
"preview": "(ns clojure.core.rrb-vector.test-common\n (:require [clojure.test :as test :refer [deftest testing is are]]\n "
},
{
"path": "src/test/cljs/clojure/core/rrb_vector/test_utils.cljs",
"chars": 2070,
"preview": "(ns clojure.core.rrb-vector.test-utils\n (:require [clojure.test :as test]\n [clojure.core.rrb-vector.rrbt :as"
},
{
"path": "src/test/clojure/clojure/core/rrb_vector/long_test.clj",
"chars": 4534,
"preview": "(ns clojure.core.rrb-vector.long-test\n (:require [clojure.test :as test :refer [deftest testing is are]]\n [c"
},
{
"path": "src/test/clojure/clojure/core/rrb_vector/test_clj_only.clj",
"chars": 6443,
"preview": "(ns clojure.core.rrb-vector.test-clj-only\n (:require [clojure.test :as test :refer [deftest testing is are]]\n "
},
{
"path": "src/test/clojure/clojure/core/rrb_vector/test_cljs.clj",
"chars": 1917,
"preview": "; Copyright (c) Rich Hickey. All rights reserved.\n; The use and distribution terms for this software are covered by "
},
{
"path": "src/test/clojure/clojure/core/rrb_vector/test_common.clj",
"chars": 25254,
"preview": "(ns clojure.core.rrb-vector.test-common\n (:require [clojure.test :as test :refer [deftest testing is are]]\n "
},
{
"path": "src/test/clojure/clojure/core/rrb_vector/test_utils.clj",
"chars": 2750,
"preview": "(ns clojure.core.rrb-vector.test-utils\n (:require [clojure.test :as test]\n [clojure.string :as str]\n "
},
{
"path": "src/test/resources/clojure/core/rrb_vector/cljs_testsuite.clj",
"chars": 1881,
"preview": "(ns clojure.core.rrb-vector.cljs-testsuite\n (:require\n [clojure.test :refer [is]]\n [cljs.repl.nashorn :as repl-nh]\n"
},
{
"path": "src/test_local/clojure/clojure/core/rrb_vector_check.clj",
"chars": 1606,
"preview": "(ns clojure.core.rrb-vector-check\n (:require [clojure.test :as test :refer [deftest testing is are]]\n [cloju"
}
]
About this extraction
This page contains the full source code of the clojure/core.rrb-vector GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 74 files (759.9 KB), approximately 210.5k tokens. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.