Repository: cnuernber/ham-fisted Branch: master Commit: d2f7171293f8 Files: 198 Total size: 2.0 MB Directory structure: gitextract_zg1q5mjk/ ├── .github/ │ ├── FUNDING.yml │ └── workflows/ │ └── test.yml ├── .gitignore ├── .mise.toml ├── CHANGELOG.md ├── LICENSE ├── README.md ├── build.clj ├── codegen/ │ └── gen_prim_invoke.clj ├── deps.edn ├── dev/ │ ├── resources/ │ │ └── logback.xml │ └── src/ │ ├── ham_fisted/ │ │ ├── analysis.clj │ │ ├── benchmark.clj │ │ └── protocol_perf.clj │ └── perftest.clj ├── docs/ │ ├── Reductions.html │ ├── css/ │ │ └── default.css │ ├── ham-fisted.api.html │ ├── ham-fisted.bloom-filter.html │ ├── ham-fisted.defprotocol.html │ ├── ham-fisted.fjp.html │ ├── ham-fisted.function.html │ ├── ham-fisted.hlet.html │ ├── ham-fisted.iterator.html │ ├── ham-fisted.lazy-noncaching.html │ ├── ham-fisted.mut-map.html │ ├── ham-fisted.primitive-invoke.html │ ├── ham-fisted.process.html │ ├── ham-fisted.profile.html │ ├── ham-fisted.protocols.html │ ├── ham-fisted.reduce.html │ ├── ham-fisted.set.html │ ├── ham-fisted.spliterator.html │ ├── highlight/ │ │ └── solarized-light.css │ ├── index.html │ └── js/ │ └── page_effects.js ├── java/ │ └── ham_fisted/ │ ├── ArrayHelpers.java │ ├── ArrayImmutList.java │ ├── ArrayLists.java │ ├── ArraySection.java │ ├── BatchReducer.java │ ├── BatchedList.java │ ├── BiFunctions.java │ ├── BlockSplitBloomFilter.java │ ├── Casts.java │ ├── ChunkedList.java │ ├── CljHash.java │ ├── ConstList.java │ ├── ConsumerAccumulators.java │ ├── Consumers.java │ ├── CtxIter.java │ ├── DoubleMutList.java │ ├── FJTask.java │ ├── FMapEntry.java │ ├── ForkJoinPatterns.java │ ├── HashBase.java │ ├── HashMap.java │ ├── HashNode.java │ ├── HashProvider.java │ ├── HashProviders.java │ ├── HashSet.java │ ├── IAMapEntry.java │ ├── IAPersistentMap.java │ ├── IAPersistentSet.java │ ├── IATransientMap.java │ ├── IATransientSet.java │ ├── ICollectionDef.java │ ├── IFnDef.java │ ├── IMap.java │ ├── IMutList.java │ ├── ISeqDef.java │ ├── ISet.java │ ├── ITypedReduce.java │ ├── ImmutList.java │ ├── ImmutSort.java │ ├── IndexedConsumer.java │ ├── IndexedDoubleConsumer.java │ ├── IndexedLongConsumer.java │ ├── IntegerOps.java │ ├── Iter.java │ ├── LazyChunkedSeq.java │ ├── LinkedHashMap.java │ ├── LinkedHashNode.java │ ├── LongAccum.java │ ├── LongHashBase.java │ ├── LongHashMap.java │ ├── LongHashNode.java │ ├── LongMutList.java │ ├── MapFn.java │ ├── MapForward.java │ ├── MapSetOps.java │ ├── MergeIterator.java │ ├── MethodImplCache.java │ ├── MutList.java │ ├── MutTreeList.java │ ├── MutableMap.java │ ├── ObjArray.java │ ├── ParallelOptions.java │ ├── PartitionByInner.java │ ├── PersistentHashMap.java │ ├── PersistentHashSet.java │ ├── PersistentLongHashMap.java │ ├── PersistentVector.java │ ├── ROHashMap.java │ ├── ROHashSet.java │ ├── ROLongHashMap.java │ ├── RandomAccessSpliterator.java │ ├── RangeList.java │ ├── Ranges.java │ ├── Reducible.java │ ├── Reductions.java │ ├── ReindexList.java │ ├── ReverseList.java │ ├── SetOps.java │ ├── StringCollection.java │ ├── Sum.java │ ├── Transformables.java │ ├── TransientHashMap.java │ ├── TransientHashSet.java │ ├── TransientList.java │ ├── TransientLongHashMap.java │ ├── TreeList.java │ ├── TreeListBase.java │ ├── TypedList.java │ ├── TypedNth.java │ ├── UnsharedHashMap.java │ ├── UnsharedHashSet.java │ ├── UnsharedLongHashMap.java │ └── UpdateValues.java ├── resources/ │ └── clj-kondo.exports/ │ └── cnuernber/ │ └── ham-fisted/ │ ├── config.edn │ └── hooks/ │ └── ham_fisted.clj_kondo ├── results/ │ ├── .keepme │ ├── concatv.edn │ ├── d00905c-chrisn-lt3-jdk-1.8.0_312.edn │ ├── d00905c-chrisn-lt3-jdk-17.0.1.edn │ ├── general-hashmap.edn │ ├── persistent-vector.edn │ ├── random-update.edn │ ├── sort-by.edn │ ├── typed-parallel-reductions.edn │ ├── typed-reductions-intel.edn │ ├── typed-reductions.edn │ ├── union-disj.edn │ ├── union-overlapping.edn │ ├── union-reduce-transient.edn │ ├── union-reduce.edn │ ├── update-values.edn │ └── vec-equals.edn ├── scripts/ │ ├── benchmark │ ├── compile │ ├── deploy │ ├── enable-jdk17 │ ├── install │ ├── koacha-test │ ├── lint │ ├── reformat │ └── run-tests ├── src/ │ └── ham_fisted/ │ ├── alists.clj │ ├── api.clj │ ├── bloom_filter.clj │ ├── caffeine.clj │ ├── datatypes.clj │ ├── defprotocol.clj │ ├── fjp.clj │ ├── function.clj │ ├── hlet.clj │ ├── impl.clj │ ├── iterator.clj │ ├── language.clj │ ├── lazy_caching.clj │ ├── lazy_noncaching.clj │ ├── mut_map.clj │ ├── primitive_invoke.clj │ ├── print.clj │ ├── process.clj │ ├── profile.clj │ ├── protocols.clj │ ├── reduce.clj │ ├── set.clj │ ├── spliterator.clj │ └── thread_local.clj ├── test/ │ └── ham_fisted/ │ ├── api_test.clj │ ├── bloom_filter_test.clj │ ├── cast_test.clj │ ├── defprotocol_test/ │ │ ├── examples.clj │ │ ├── hash_collisions_test.clj │ │ ├── more_examples.clj │ │ └── other_test.clj │ ├── defprotocol_test.clj │ ├── fjp_test.clj │ ├── hash_map_test.clj │ ├── hlet_test.clj │ ├── parallel_test.clj │ ├── persistent_vector_test.clj │ ├── test_setup.clj │ └── vec_like_test.clj ├── tests.edn └── topics/ └── Reductions.md ================================================ FILE CONTENTS ================================================ ================================================ FILE: .github/FUNDING.yml ================================================ github: cnuernber ================================================ FILE: .github/workflows/test.yml ================================================ name: Automated tests on: push: jobs: test: runs-on: ubuntu-22.04 steps: - uses: actions/checkout@v3 - name: Restore cached dependencies uses: actions/cache/restore@v3 with: path: | ~/.m2/repository ~/.deps.clj key: cljdeps-${{ hashFiles('deps.edn') }} - name: Mise-en-place setup uses: jdx/mise-action@v2 with: install: true cache: true - name: Setup Clojure uses: DeLaGuardo/setup-clojure@12.1 with: cli: 1.11.1.1413 - name: Run automated tests - Java 11 run: | mise use java@corretto-11 java -version scripts/run-tests - name: Run automated tests - Java 17 run: | mise use java@corretto-17 java -version scripts/run-tests - name: Run automated tests - Java 19 run: | mise use java@corretto-19 java -version scripts/run-tests - name: Run automated tests - Java 21 run: | mise use java@corretto-21 java -version scripts/run-tests - name: Run automated tests - Java 22 run: | mise use java@corretto-22 java -version scripts/run-tests - name: Cache dependencies uses: actions/cache@v3 with: path: | ~/.m2/repository ~/.deps.clj key: cljdeps-${{ hashFiles('deps.edn') }} restore-keys: cljdeps- ================================================ FILE: .gitignore ================================================ .cpcache target .nrepl-port pom.xml *.asc issue-data jdk-* .clj-kondo .lsp .dir-locals.el ================================================ FILE: .mise.toml ================================================ [tools] java = "corretto-8" ================================================ FILE: CHANGELOG.md ================================================ # 3.029 * Much more thorough support for spliterators and forkjoinpool. See spliterator and fjp namespaces respectively. # 3.025 * bloom filters handle java.time.Instants automatically # 3.023 * fix - process/launch invalid stdout for very quick processes. # 3.022 * merge-iterable returns a seq-iterable so it plays nice with the REPL. # 3.020 * seq-iterables cannot be `counted?` # 3.019 * Fix bug in with empty sequences and pmap-io. # 3.018 * iterator/merge-iterable has to be stable w/r/t iterator order - leftmost wins. # 3.017 * Fix for concat, apply-concat to make their iterators a bit lazier than they were before. # 3.015 * Fix for issue 10 - partition-all works different. * new partition fns - partition-by-cost and partition-by-comparator. * new iterator fns - iter-take-while, wrap-iter, iter-take # 3.013 * iterators created via iterator/once-iterable and iterator/iterable are as lazy as possible avoiding certain types of lookahead and hanging behavior. # 3.010 * Reduce require time for defprotocol by avoiding primitive-invoke namespace. # 3.009 * Issue fixed in process namespace. # 3.008 * [issue 22](https://github.com/cnuernber/ham-fisted/issues/21) - better clj-condo support. * new [process](https://cnuernber.github.io/ham-fisted/ham-fisted.process.html) namespace for launching and controlling sub processes. # 3.003 * New functions and docs added to 'iterator' namespace. # 3.000 * All protocols replaced with hamf's defprotocol impl. # 2.039 * Slightly faster protocol dispatch as avoids instance check for interface impl in favor of only using map lookup. * Implementation of cond that can be used in functions that need to avoid boxing. # 2.038 * Fixing some small perf issues with primitive protocol implementations. # 2.037 * Object constants work when explicitly specified via 'extend'. * Object arrays explicitly checked for protocol dispatch when dealing with array types. * 'true' and 'false' supported as object constant types - nil unsupported. * Namespace comments. # 2.036 * Work on defprotocol to support primitive and constant return types. # 2.035 * Mutable treelists now result in less data on sublist and error on double persistent call. # 2.034 * Fix to smarter sublist impl - incorrect in some cases -- tests updated to catch in future. # 2.033 * MAJOR UPGRADE - implemented vec and vector as treelists. This matches Clojure's persistent vector implementation in performance for mutable and immutable conj, reduce pathways but has a much smarter subvec implementation and faster conversion to and from object arrays. # 2.032 * Two merge-iterator implementations used to N-way merging of sorted sequences. Linear is for small n <= 32 and the priority queue method is for larger N's. The exact cutoff where performance will matter will depend on the dataset and the relative cost of the comparator. * Attempted to move all sorts to Arrays/parallelSort as it outperforms fastutils parallel sort by a bit. * added lsum, lsummary, dsummary for long-space sum, and simple summary statistics [min max mean sum n-elems] in long and double space respectively. # 2.031 * `(reduce + 0 (lznc/apply-concat nil))` works. # 2.030 * pmap-opts could produce an empty incorrect result if a custom pool was provided and parallelism was not specified. # 2.029 * Error in hash-map compute - did not remove key if compute fn was nil. # 2.028 * First class bloom filter support - uses apache parquet block-split-bloom-filter. # 2.027 * Fix for api/difference when left hand side is a java map. # 2.026 * Somewhat faster byte and short array creation when input is an IMutList impl. # 2.025 * Bugfix so update-values is consistent with persistent maps. # 2.024 * Bugfix for add-constant! default impls. # 2.023 * Bulk add-constant interface for all growable lists. # 2.022 * growable lists support clear. # 2.021 * `lines` - replacement for line-seq that returns an auto-closeable iterable and cannot cache nor hold-onto-head the data. * `re-matches` - faster version of re-matches. # 2.020 * split caffeine support off into its own namespace. # 2.019 * impl/pmap really does support user-defined thread pool. # 2.018 * Add clj-kondo exports and config, fix linting errors * Remove support for and call to `take-last` 1-arity, which was not valid. * Fix variable arity `merge-with`, which was not correctly implemented. * `apply-concat`, `concat-opts` accept cat-parallelism option allow you to specify how the concatenation should be parallelized at the creation source as opposed to at the preduce/parallel reduction callsite. # 2.017 * Faster compose-reducers especially where there really are a lot of reducers. # 2.015 * [issue 13](https://github.com/cnuernber/ham-fisted/issues/13) - any IMutList chunkedSeq was partially incorrect. # 2.014 * frequencies respects map-fn option to allow concurrent hashmaps to be used. # 2.013 * `cartesian-map` no longer has a random access variant. The cooler version of this uses the tensor address mechanism to allow parallel redution. * fixed major issue with parallel frequencies. # 2.011 * Much faster every? implementation esp. for primitive arrays and persistent vectors. * More hlet extensions - `lng-fns` and `dbl-fns` which are faster in the general case then `lngs` and `dbls` as they avoid RT/nth. * efficient `cartesian-map` which does a cartesian join across its inputs and calls f on each value. ```clojure user> (hamf/sum-fast (lznc/cartesian-map #(h/let [[a b c d](lng-fns %)] (-> (+ a b) (+ c) (+ d))) [1 2 3] [4 5 6] [7 8 9] [10 11 12 13 14])) 3645.0 ``` # 2.010 * Extensible let - hlet and helpers make using the primitive overloads of clojure functions easier. See the ham-fisted.hlet and ham-fisted.primtive-invoke namespaces. # 2.009 * typed 'nth' methods efficient for primitive manipulations - 'dnth', 'fnth', 'inth', 'lnth'. # 2.008 * Custom reduce implemented for object array wrappers. 2.007 * reduce namespace now has helper to create a parallel reducer. * hashset has optimized addall pathway when input is another hashset. # 2.006 * partition-by accepts a predicate function in options - example in docs. # 2.005 * implemented lazy noncaching partition-all - similar perf to partition-by. * Faster default dispatch for pgroups, upgroups. * Faster sum-fast if input is random access. # 2.004 * Faster sort implemented as default in several places. # 2.004 * Ensure all object sorting is done with parallelQuickSort. * Small fix to array macros to use l2i instead of RT.intCast. # 2.003 * Major issue in compose-reducers - object composition was typed to double reduction. # 2.002 * slightly faster partition-by - inner loop written in java. # 2.001 * Added lazy-noncaching partition-by. This method has somewhat higher performance than clojure.core/partition-by as it does not make intermediate containers and is strictly lazy-noncaching. # 2.000 * Rebuilt hashmaps on faster foundation especially for micro benchmarks. * Removed bits and pieces that do not provide enough return on investment. * For more in-depth comments see [PR-7](https://github.com/cnuernber/ham-fisted/pull/7). # 1.009 * new linked hashmap implementation with equiv-semantics and fast union op. # 1.008 * Fast set intersection for longer sequencers of sets (intersect-sets). # 1.007 * Fast pathways for finding min/max index of a collection of objects in a similar way to min-key and max-key. # 1.006 * Very specific upgrade to combine-reducers pathways. # 1.005 * pmap, upmap pathways now return an object that full implements seqable and ireduceinit. # 1.002 * Added n-lookahead to the parallel options pathway as for some problems this makes a major difference in the efficiency of the pmap pathway. # 1.001 * Fixed pmap implementation to release memory much more aggressively. # 1.000-beta-98 * Fixed serious but subtle issue when a transient hash map is resized. This should be considered a must-have upgrade. # 1.000-beta-96 * Fixed from upgrading dtype-next. * Major breaking changes!! API functions have been moved to make the documentation clearer and the library more maintainable in the long run. * map-union of mutable or transient maps produces mutable or transient maps! * Final refactoring before 1.000 release. * Functions to make creating java.util.function objects are moved to ham-fisted.function. * Reduction-related systems are moved to ham-fisted.reduce. * java.util.Map helpers are moved to ham-fisted.mut-map. # 1.000-beta-93 * java implementation of a batched stream reducer. This avoids adding java to the stream api. # 1.000-beta-92 * pure java reductions for situations where you have an index fn a count. These mainly just make benchmarks a bit more stable. # 1.000-beta-91 * IFnDef supports interfaces for long supplier (L), double supplier (D), and Supplier (O). # 1.000-beta-90 * Accelerated map boolean union, intersection, difference for hashtable, long hashtable. * Major bugfix in map dissoc. # 1.000-beta-89 * Added inc-consumer - returns a generic consumer that increments a long. Useful for the various situations where you need to track an incrementing variable but don't want the overhead of using a volatile variable. # 1.000-beta-88 * Immutable maps and vectors derive from APersistentMap and APersistentVector so that downtream libraries can pick them up transparently. # 1.000-beta-86 * Error in reduction of empty ranges. # 1.000-beta-85 * Slightly faster map construction pathways. * In fact both the hamf base map `mut-map` and the integer-specialized `mut-long-hashtable-map` are faster than the default `clojure.data.int-map` pathway for construction and value lookup according to the benchmarks in `clojure.data.int-map`. Interestingly enough they are fastest if you create an intermediate object array using lznc/apply-concat: ```clojure user> (count entries) 1000000 user> (c/quick-bench (into (persistent! (hamf/mut-long-hashtable-map)) entries)) Evaluation count : 6 in 6 samples of 1 calls. Execution time mean : 366.247830 ms Execution time std-deviation : 11.024896 ms Execution time lower quantile : 348.564420 ms ( 2.5%) Execution time upper quantile : 376.625750 ms (97.5%) Overhead used : 1.492920 ns nil user> (c/quick-bench (into (i/int-map) entries)) Evaluation count : 6 in 6 samples of 1 calls. Execution time mean : 568.189038 ms Execution time std-deviation : 1.677163 ms Execution time lower quantile : 566.564884 ms ( 2.5%) Execution time upper quantile : 570.564253 ms (97.5%) Overhead used : 1.492920 ns nil user> (def ll (into (persistent! (hamf/mut-long-hashtable-map)) entries)) #'user/ll user> (def il (into (i/int-map) entries)) #'user/il user> (c/quick-bench (dotimes [idx (count entries)] (.get ^java.util.Map ll idx))) Evaluation count : 84 in 6 samples of 14 calls. Execution time mean : 7.399383 ms Execution time std-deviation : 62.314286 µs Execution time lower quantile : 7.297162 ms ( 2.5%) Execution time upper quantile : 7.451677 ms (97.5%) Overhead used : 1.492920 ns nil user> (c/quick-bench (dotimes [idx (count entries)] (.get ^java.util.Map il idx))) Evaluation count : 30 in 6 samples of 5 calls. Execution time mean : 22.936125 ms Execution time std-deviation : 583.510734 µs Execution time lower quantile : 22.334216 ms ( 2.5%) Execution time upper quantile : 23.654541 ms (97.5%) Overhead used : 1.492920 ns nil user> user> (c/quick-bench (hamf/mut-long-hashtable-map (hamf/into-array Object (lznc/apply-concat entries)))) Evaluation count : 6 in 6 samples of 1 calls. Execution time mean : 271.755568 ms Execution time std-deviation : 1.545379 ms Execution time lower quantile : 270.184413 ms ( 2.5%) Execution time upper quantile : 273.736344 ms (97.5%) Overhead used : 1.492920 ns nil ``` # 1.000-beta-84 * `wrap-array`, `wrap-array-growable`, major into-array optimizations and better `map-reducible`. # 1.000-beta-83 * Opening the door to custom IReduce implementations. # 1.000-beta-82 * convert hashsets to use hashtables instead of bitmap tries. * careful analysis of various vec-like object creation mechanisms. # 1.000-beta-81 * long primitive hashtables - these are quite a bit faster but especially when used directly. # 1.000-beta-80 * Helpers for very high performance scenarios. lazy-noncaching/map-reducible, api/->long-predicate. # 1.000-beta-78 * fill-range is now property accelerated making all downstream projects that use addAll and friends far faster. # 1.000-beta-77 * see [commit 38596d8](https://github.com/cnuernber/ham-fisted/commit/38596d85541de7d2c926ac8b16522a764fcb6af9) # 1.000-beta-76 * Added group-by-consumer - this has different performance and functionality characteristics than group-by-reducer. For instance, group-by-consumer with a linked hashmap will return a map with keys in the order of keys initially encounted. group-by-reducer with the same hashmap will return a map with keys in the order of latest encountered. Group-by-consumer uses `computeIfAbsent` which is a slightly faster primitive than `compute` as it doesn't need to check the return value of the reducer, only of the initialization of the map entry. # 1.000-beta-75 * MapForward class so we can use normal java maps in normal Clojure workflows. # 1.000-beta-74 * bugfix - Map's `compute` has to accept nil keys. # 1.000-beta-73 * memoize now supports `:eviction-fn` - for callbacks when things get evicted. * More helpers for memoized fns - cache-as-map, evict-memoized-call. # 1.000-beta-72 * Switch to caffeine for memoize cache and standard java library priority queue for take-min. This removed the dependency on google guava thus drastically cutting the chances for dependency conflicts. # 1.000-beta-71 * HUGE CHANGES!!! - moved to hashtable implementation for main non-array map instead of bitmap trie. This is because in all my tests it is *much* faster for everything *aside* from non-transient (reduce assoc ...) type loops which are a waste of time to begin with. * Because there are now three full map implementations (array, trie, hashtable) there is a more defined map structure making it less error prone to test out different map backends. * Lots of inner class renaming and such - however `frequencies`, `group-by-reduce`, and `mapmap` now a bit faster - about 2X. Here is a telling performance metric: ```clojure ({:construct-μs 2.726739663709692, :access-μs 1.784634592104282, :iterate-μs 2.7345543552812073, :ds-name :java-hashmap} {:construct-μs 3.5414584143710885, :access-μs 2.761234751112207, :iterate-μs 2.1730894775185403, :ds-name :hamf-hashmap} {:construct-μs 6.475808180747403, :access-μs 2.484804237281106, :iterate-μs 1.8564705765641765, :ds-name :hamf-transient} {:construct-μs 11.43649782981362, :access-μs 5.152473242630386, :iterate-μs 8.793332955848225, :ds-name :clj-transient}) ham-fisted.hash-map-test> ``` # 1.000-beta-69 * Faster `mode`. * Faster map iteration. * Corrected clojure persistent hash map iteration. # 1.000-beta-67 * Faster `mode`. * `mmax-key` - use `(mmax-key f data)` as opposed to `(apply max-key f data)`. It is faster and handles empty sequences. Same goes for `mmin-key`. # 1.000-beta-66 * Fixed `make-comparator`. * Added `mode`. # 1.000-beta-65 * Better obj->long and obj->double pathways that will always apply the appropriate cast and thus have 0 arg variants. * Better/faster sort-by pathway that avoids potential intermediate data creation. # 1.000-beta-64 * Fixed predicate, long-consumer, double-consumer and consumer pathways. * Faster dispatch for preduce. # 1.000-beta-62 * Faster dispatch for preduce. # 1.000-beta-61 * All lists are comparable. # 1.000-beta-60 * nil is convertible to iterable and collections without fail. # 1.000-beta-59 * Container reduction must respect reduced - I think this is a design flaw but not a serious or impactful one aside from requiring more complex per-container reduction code. # 1.000-beta-58 * Perf tweaks and small fixes from TMD. # 1.000-beta-57 * Additional round of optimizations around creation of persistent vector objects. * renamed a few of the functor-creation macros. * lznc/map explicity supports long->obj transformations as these are often used as index->obj lookup systems. * Rebuilt IMutList's toArray pathway to use reduction. # 1.000-beta-56 * Switched completely to clojure.core.protocols/CollReduce. * Removed a solid amount of cruft and simplified reduction architecture. * Now loading hamf transparently makes reductions on all arrays and many java such as hashmaps datastructures faster. # 1.000-beta-55 * Removed lots of old cruft. * Added IFnDef predicates so you can use IFn-based predicates from java. # 1.000-beta-54 * Removed lots of old cruft. * Added IFnDef predicates so you can use IFn-based predicates from java. # 1.000-beta-53 * `:unmerged-result?`, `:skip-finalize?` options for `preduce` and `preduce-reducer`. This allows you to use the parallelized reductions pathway but get a sequence of results back as opposed to a single result. It also allows you to used reducers or transducing-compatible rfn's that have no parallel merge pathway and handle the parallel merge yourself after the parallelized reduction. * Fixed issue with single-map parallel reductions to ensure that it passes the parallel reduction request to its source data. # 1.000-beta-52 * bulk union, intersection operations. * Faster `equiv` for longs and doubles but equivalent for everything else. # 1.000-beta-51 * additional set operation - parallelized `unique`. * exposed indexed accumulator macros in api for use outside library. * generic protocol fn add-fn that must return a reduction compatible function for a given collection. # 1.000-beta-50 * forgot type hints on array constructors. # 1.000-beta-49 * Final round of optimizations for double array creation. Turns out reductions really are faster. # 1.000-beta-48 * macros for double, float, long, and int array creation that will inline a fastpath if the argument is a compile-time vector or integer. Bugfix for casting floats to longs. * shorthand macros, ivec, lvec, fvec, dvec to create array-backed containers that allow nth destructuring. # 1.000-beta-47 * major double-array, float-array, long-array, int-array optimizations. # 1.000-beta-46 * Set protocol to supercede the set protocol from dtype-next. * lots and lots of fixes from dataset work. # 1.000-beta-45 * Small fixes and making helpers public for dtype-next work. # 1.000-beta-43 * long lists really are long lists - copy-paste mistake from int lists. # 1.000-beta-42 * declare-double-consumer-preducer! - given type derived from DoubleConsumer and a few others, create a parallel reducer. * declare-consumer-preducer! - similar to above, incoming data is not expected to be a stream of double values. # 1.000-beta-41 * ->collection is protocol driven allowing new non-collection things like bitmaps to be turned temporarily into collections. This means that reductions and collection conversion are protocol driven. # 1.000-beta-40 * maps are iterable... # 1.000-beta-39 * Explicit protocols for serial and parallel reduction and reducers. * Explicit support for BitSet objects. * Updated (->reducible) pathways to check for protocol reducer support. * Protocol for conversion of arbitrary types to iterable for map, filter support. # 1.000-beta-38 * Fixed comparison of seq with nonseq. # 1.000-beta-37 * Small perf enhancements from tmd perf regression # 1.000-beta-36 * Added in explicit checks for long, double, and predicate objects in filter's reduction specializations. Potentially these are too expensive but it does help a bit with longer sequences. * Changed things such that Double/NaN evaluates to false. This matches that the null object evaluates to false and null evaluates to Double/NaN. # 1.000-beta-35 * Added finalize method to reducers to match transducer spec. * Exposed `compose-reducers` that produces a new reducer from a map or sequence of other reducers. * These changes simplfied `reduce-reducers` and `preduce-reducers`, `sum` and `sum-fast`. # 1.000-beta-34 * Enable parallelization for instances of clojure.core.PersistentHashMap. * protocol-based parallelization of reductions so you can extend the parallelization to new undiscovered classes. * reducer-xform->reducer - Given a reducer and a transducer xform produce a new reducer that will apply the transform to the reduction function of the reducer: ```clojure ham-fisted.api> (reduce-reducer (reducer-xform->reducer (Sum.) (clojure.core/filter even?)) (range 1000)) # ``` # 1.000-beta-33 * Finally a better api to group-by-reduce and group-by can now be implemented via group-by-reduce. group-by-reduce uses same 3 function arguments as preduce so your reduction systems are interchangable between these two systems. * Fixed `conj` for all growable array lists. * Added a protocol for parallel reductions. This allows you to pass in one object and transform it into the three functions required to do a parallel reduction. * Added preduce-reducer, preduce-reducers for a single reducer or a sequence or map of reducers, respectively. # 1.000-beta-32 * group-by, group-by-reduced fixed for large n. # 1.000-beta-31 * min-n is now a long with parallel options. * lazy-noncaching namespace now has map-indexed. Faster reductions and random access objects stay random access. # 1.000-beta-30 * Various bugfixes from dtype work. # 1.000-beta-29 * Ranges with more than Integer/MAX_VALUE elems can be accessed via their IFn overloads and support custom lgetLong and lgetDouble methods that take long indexes for long and double ranges. # 1.000-beta-28 * Use lookahead and put timeouts for all parallelization primitives so that if a long running parallelization is cancelled the forkjoin pool itself isn't hung. * Enable long and double ranges whose size is larger than Integer/MAX_VALUE. This includes parallelized reductions which even optimized take basically forever. * Add better defaults for reductions to long and double -specific IMutList interfaces. * Ensure reduction implementations do not dereference a reduced accumulator. * Fix reducible interface to it matches preduce. * Added persistent! implementation which fails gracefully if input is already persistent. * Fixed group-by, group-by-reduce, and pfrequencies implementation to use preduce. * conj works on map, filter, and concat from the lazy-noncaching library. # 1.000-beta-27 * Remove explicit support for boolean primitives. # 1.000-beta-26 * IFnDef overloads implement their appropriate java.util.function counterparts. # 1.000-beta-25 * Removed claypoole from dependencies. * Move typed clojure function interface definitions from Reductions to IFnDef. * Added overrides of keys, vals that produce parallelizable collections if the input itself is a parallelizable collection - either maps from this library or any java hashmap. # 1.000-beta-24 * preduce has new option to help parallelize concat operations - they can be parallelized two different ways, either elemwise where each container parallelizes its reduction or by sequence where an initial reduction is done with pmap then the results are merged. * all random access contains support spliterator and typed stream construction. * Fix bug in upmap causing hanging with short sequences. # 1.000-beta-23 * double conversion to long fails for NaN. * Careful combining of typed map/filter chains to avoid causing inaccuracies when converting from double to long. * Major parallelism upgrade - spliterator-based objects such as java.util.hashmap and all the hashmaps/hashsets from this library now support parallelized reduction. # 1.000-beta-22 * Numeric values are range checked on input to addLong. # 1.000-beta-21 * Removed ensureCapacity from IMutList. # 1.000-beta-20 * Moved to double reduction as opposed to double foreach. Perftested heavily and found that reduce is just as fast and more general. # 1.000-beta-18 * expose sublistcheck. # 1.000-beta-17 * Stricter correctness checking for sublist types, everything implements Associative. # 1.000-beta-16 * Correctness fixes for pmap, upmap, pgroups, upgroups. # 1.000-beta-15 * Fixed sum for large n-elems. * upgroups - Unordered parallel groupings for random access systems. * Indexed consumers for copying, broadcasting type operations. * Reducible interface for objects that can reduce themselves. # 1.000-beta-14 * ArraySection is now first-class, will rebase dtype-next array pathways on this. # 1.000-beta-12 * pmap is guaranteed *not* to require `shutdown-agents`. ================================================ FILE: LICENSE ================================================ The MIT License (MIT) Copyright © 2021 Chris Nuernberger Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================ FILE: README.md ================================================ # HAM-Fisted [![Clojars Project](https://clojars.org/com.cnuernber/ham-fisted/latest-version.svg)](https://clojars.org/com.cnuernber/ham-fisted) * [API docs](https://cnuernber.github.io/ham-fisted/) * [Clojure Conj Talk](https://www.youtube.com/watch?v=ralZ4j_ruVg) ## Summary Clojure-style immutable collections (and mutable counterparts), together with some operations, all aimed at high performance. In particular, high-performance in large-`n` and parallel contexts. Included are a namespace of lazy but not caching operations, a `ForkJoinPool` oriented `pmap`, and a system of parallel reductions. This gives the user somewhat-drop-in replacements for familiar Clojure tools that can be faster. ## History What started as a collection of efficient mutable and immutable data structures based on Phil Bagwell's bitmap trie concept became an overall reimplementation of Clojure's core datastructures and some of its base concepts specifically with performance in mind. This means, for instance, that the library prefers iterators over sequences in many situations. There are also new functional primitives developed from my experience processing data and working with Clojure over the last 10 years. My hope is this library becomes a platform to experiment and develop new and/or better functional datastructures and algorithmic primitives. Here are a few concepts to keep in mind - ## In-place Mutable -> Persistent The mutable hashmap and vector implementations allow in-place instantaneous conversion to their persistent counterparts. This allows you to build a dataset using the sometimes much faster mutable primitives (.compute and friends for instance in the hashmap case) and then return data to the rest of the program in persistent form. Using this method, for example `frequencies` is quite a bit faster while still returning a persistent datastructure. Along those lines construction of a persistent vector from an object array is very fast so it is very efficient to construct a persistent vector from an object-array-list - the array list being much faster to build. ## New Primitive Operations There are many new primitive operations than listed below - please take a moment to scan the api docs. Some standouts are: #### Map Union, Difference, Intersection Aside from simply a reimplementation of hashmaps and persistent vectors this library also introduces a few new algorithms namely map-union, map-intersection, and map-difference. These are implemented at the trie level so they avoid rehashing any keys and use the structure of the hashmap in order to boost performance. This means `merge` and `merge-with` are much faster especially if you have larger maps. But it also means you can design novel set-boolean operations as you provide a value-resolution operator for the map values. Because the hamf hashmaps have fast unions, you can now design systems where for instance each thread builds up a separate hashmap and the results are unioned together back in the main thread in a map-reduce type design. This type of design was the original target of the union system. These systems are substantially faster if the objects have a high cost to hash and do not cache their hashcode but this is rare for Clojure systems as persistent vectors, maps and keywords cache their hash values. Strings, however, are an example of something where these primitives (and things like frequencies) will perform substantially better. ### Casting and Finite Numbers Float and double values are only allowed to cast to long if they are [finite](https://docs.oracle.com/javase/8/docs/api/java/lang/Double.html#isFinite-double-). Boolean values casted to long or double are 0 for false and 1 for true. Any nonzero finite number casted to boolean is true, 0 is false, non-finite numbers are errors. `nil` casted to a floating point number is NaN. NaN casted to an object is NaN. If objects are not number then nil is false and non-nil is true. Any undefined cast falls back to `clojure.lang.RT.xCast` where `x` denotes the target type. Reading data from contains leads to unchecked casts while writing data to contains leads to checked casts. #### update-values, group-by-reduce, mapmap * `update-vals` - is far faster than map->map pathways if you want to update every value in the map but leave the keys unchanged. * `group-by-reduce` - perform a reduction during the group-by. This avoids keeping a large map of the complete intermediate values which can be both faster and more memory efficient. * `mapmap` - A techascent favorite, equivalent to: ```clojure (->> (map map-fn src-map) (remove nil?) (into {})) ``` #### Parallelized Reductions - preduce I have an entire topic on [Reductions](https://cnuernber.github.io/ham-fisted/Reductions.html) - give it a read and then send me an email with your thoughts :-). * [preduce](https://cnuernber.github.io/ham-fisted/ham-fisted.api.html#var-preduce) * [preduce-reducer](https://cnuernber.github.io/ham-fisted/ham-fisted.api.html#var-preduce-reducer) * [preduce-reducers](https://cnuernber.github.io/ham-fisted/ham-fisted.api.html#var-preduce-reducers) These parallelization primitives allow users to pass in their own forkjoin pools so you can use it for blocking tasks although it is setup be default for cpu-bound operations. Concat operations can parallelize reductions over non-finite or non-parallelizable containers using the default `:seq-wise` `:cat-parallelization` option. #### All Arrays Are First Class * Any array including any primitive array can be converted to an indexed operator with efficient sort, reduce, etc. implementations using the `lazy-noncaching` namespace's `->random-access` operator. This allows you to pass arrays as is to the rest of your clojure program without conversion to a persistent vector - something that is both not particularly efficient and explodes the data size. #### Random Access Containers Support Negative Indexes All random access containers, be it vectors, lists, or array lists support nth, ifn interfaces taking -1 to index from the end of the vector. For performance reasons, the implementation of List.get does not - ```clojure ham-fisted.persistent-vector-test> ((api/vec (range 10)) -1) 9 ham-fisted.persistent-vector-test> (nth (api/vec (range 10)) -1) 9 ham-fisted.persistent-vector-test> (.get (api/vec (range 10)) -1) Execution error (IndexOutOfBoundsException) at ham_fisted.ChunkedList/indexCheck (ChunkedList.java:210). Index underflow: -1 ham-fisted.persistent-vector-test> ((api/->random-access (api/int-array (range 10))) -1) 9 ham-fisted.persistent-vector-test> (nth (api/->random-access (api/int-array (range 10))) -1) 9 ``` Similarly `last` is constant time for all any list implementation deriving from `java.util.RandomAccess`. ## Other ideas * `lazy-noncaching` namespace contains very efficient implementations of map, filter, concat, and repeatedly which perform as good as or better than the eduction variants without chunking or requiring you to convert your code from naive clojure to transducer form. The drawback is they are lazy noncaching so for instance `(repeatedly 10 rand)` will produce 10 random values every time it is evaluated. Furthermore `map` will produce a random-access return value if passed in all random-access inputs thus preserving the random-access property of the input. * lazy-caching namespace contains inefficient implementations that do in fact cache - it appears that Clojure's base implementation is very good or at least good enough I can't haven't come up with one better. Potentially the decision to use chunking is the best optimization available here. ## Contributing The best way to contribute is to fund me through github sponsors linked to the right or to engage [TechAscent](https://techascent.com) - we are always looking for new interesting projects and partners. Aside from that as mentioned earlier my hope is this library becomes a platform that enables experimentation with various functional primitives and overall optimized ways of doing the type of programming that the Clojure community enjoys. Don't hesitate to file issues and PR's - I am happy to accept both. If you want to work on the library you need to enable the `:dev` alias. ## Benchmarks Lies, damn lies, and benchmarks - you can run the benchmarks with `./scripts/benchmark`. Results will be printed to the console and saved to results directory prefixed by the commit, your machine name and the jdk version. Results will print normalized to either the base time for clojure.core (clj) or for java.util (java). One interesting thing here is in general how much better JDK-17 is for many of these tests than JDK-8. Here are some example timings taken using my laptop plugged in with an external cooling supply (frozen peas) applied to the bottom of the machine. An interesting side note is that I get better timings often when running from the REPL for specific benchmarks than from the benchmark - perhaps due to the machine's heat management systems. #### JDK-17 | :test | :n-elems | :java | :clj | :eduction | :hamf | :norm-factor-μs | |------------------------|---------:|------:|-----:|----------:|------:|----------------:| | :assoc-in | 5 | | 1.0 | | 0.646 | 0.245 | | :assoc-in-nil | 5 | | 1.0 | | 0.371 | 0.120 | | :concatv | 100 | | 1.0 | | 0.099 | 9.827 | | :frequencies | 10000 | | 1.0 | | 0.412 | 966.154 | | :get-in | 5 | | 1.0 | | 0.564 | 0.124 | | :group-by | 10000 | | 1.0 | | 0.333 | 1414.480 | | :group-by-reduce | 10000 | | 1.0 | | 0.313 | 1408.028 | | :hashmap-access | 10000 | 0.700 | 1.0 | | 0.989 | 549.468 | | :hashmap-access | 10 | 0.837 | 1.0 | | 0.826 | 0.400 | | :hashmap-cons-obj-ary | 4 | | 1.0 | | 0.355 | 0.392 | | :hashmap-cons-obj-ary | 10 | | 1.0 | | 0.584 | 0.864 | | :hashmap-cons-obj-ary | 1000 | | 1.0 | | 0.541 | 124.811 | | :hashmap-construction | 10000 | 0.563 | 1.0 | | 0.923 | 1331.130 | | :hashmap-construction | 10 | 0.240 | 1.0 | | 0.357 | 2.337 | | :hashmap-reduce | 10000 | 0.792 | 1.0 | | 0.860 | 316.433 | | :hashmap-reduce | 10 | 0.703 | 1.0 | | 0.735 | 0.360 | | :int-list | 20000 | 1.000 | | | 1.147 | 467.994 | | :mapmap | 1000 | | 1.0 | | 0.276 | 275.786 | | :object-array | 20000 | | 1.0 | | 0.240 | 1560.975 | | :object-list | 20000 | 1.000 | | | 0.987 | 518.878 | | :sequence-summation | 20000 | | 1.0 | 0.29 | 0.409 | 1380.496 | | :shuffle | 10000 | | 1.0 | | 0.353 | 329.709 | | :sort | 10000 | | 1.0 | | 0.337 | 2418.307 | | :sort-doubles | 10000 | | 1.0 | | 0.374 | 2272.143 | | :sort-ints | 10000 | | 1.0 | | 0.291 | 2514.779 | | :union | 10 | 0.155 | 1.0 | | 0.088 | 1.785 | | :union | 10000 | 0.275 | 1.0 | | 0.174 | 1664.823 | | :union-disj | 10 | 0.156 | 1.0 | | 0.085 | 1.798 | | :union-disj | 10000 | 0.279 | 1.0 | | 0.178 | 1641.344 | | :union-reduce | 10 | 0.139 | 1.0 | | 0.220 | 23.954 | | :union-reduce | 10000 | 0.100 | 1.0 | | 0.159 | 41261.663 | | :update-in | 5 | | 1.0 | | 1.153 | 0.276 | | :update-in-nil | 5 | | 1.0 | | 0.276 | 0.158 | | :update-values | 1000 | | 1.0 | | 0.090 | 158.994 | | :vector-access | 10 | 1.568 | 1.0 | | 1.008 | 77.945 | | :vector-access | 10000 | 0.957 | 1.0 | | 1.027 | 125.778 | | :vector-cons-obj-array | 10 | 1.184 | 1.0 | | 0.356 | 0.071 | | :vector-cons-obj-array | 10000 | 0.083 | 1.0 | | 0.048 | 112.192 | | :vector-construction | 10 | 0.460 | 1.0 | | 1.124 | 0.078 | | :vector-construction | 10000 | 0.082 | 1.0 | | 0.078 | 117.432 | | :vector-reduce | 10 | 1.996 | 1.0 | | 1.088 | 0.150 | | :vector-reduce | 10000 | 1.228 | 1.0 | | 0.863 | 194.194 | | :vector-to-array | 10 | 0.256 | 1.0 | | 0.503 | 0.041 | | :vector-to-array | 10000 | 0.063 | 1.0 | | 0.124 | 69.590 | #### JDK-1.8 | :test | :n-elems | :java | :clj | :eduction | :hamf | :norm-factor-μs | |------------------------|---------:|------:|-----:|----------:|------:|----------------:| | :assoc-in | 5 | | 1.0 | | 0.801 | 0.274 | | :assoc-in-nil | 5 | | 1.0 | | 0.275 | 0.142 | | :concatv | 100 | | 1.0 | | 0.120 | 6.810 | | :frequencies | 10000 | | 1.0 | | 0.421 | 960.710 | | :get-in | 5 | | 1.0 | | 0.598 | 0.125 | | :group-by | 10000 | | 1.0 | | 0.335 | 1410.690 | | :group-by-reduce | 10000 | | 1.0 | | 0.293 | 1433.528 | | :hashmap-access | 10000 | 0.817 | 1.0 | | 1.046 | 541.540 | | :hashmap-access | 10 | 0.791 | 1.0 | | 0.904 | 0.402 | | :hashmap-cons-obj-ary | 4 | | 1.0 | | 0.398 | 0.407 | | :hashmap-cons-obj-ary | 10 | | 1.0 | | 0.696 | 0.682 | | :hashmap-cons-obj-ary | 1000 | | 1.0 | | 0.449 | 130.423 | | :hashmap-construction | 10000 | 0.586 | 1.0 | | 0.927 | 1281.371 | | :hashmap-construction | 10 | 0.278 | 1.0 | | 0.401 | 2.238 | | :hashmap-reduce | 10000 | 0.625 | 1.0 | | 0.704 | 321.295 | | :hashmap-reduce | 10 | 0.714 | 1.0 | | 0.862 | 0.312 | | :int-list | 20000 | 1.000 | | | 1.017 | 497.492 | | :mapmap | 1000 | | 1.0 | | 0.325 | 226.518 | | :object-array | 20000 | | 1.0 | | 0.391 | 1374.725 | | :object-list | 20000 | 1.000 | | | 0.981 | 493.795 | | :sequence-summation | 20000 | | 1.0 | 0.37 | 0.227 | 1342.562 | | :shuffle | 10000 | | 1.0 | | 0.430 | 286.478 | | :sort | 10000 | | 1.0 | | 0.284 | 2839.552 | | :sort-doubles | 10000 | | 1.0 | | 0.424 | 2485.058 | | :sort-ints | 10000 | | 1.0 | | 0.279 | 2885.107 | | :union | 10 | 0.147 | 1.0 | | 0.093 | 1.938 | | :union | 10000 | 0.278 | 1.0 | | 0.198 | 1446.922 | | :union-disj | 10 | 0.144 | 1.0 | | 0.091 | 1.938 | | :union-disj | 10000 | 0.285 | 1.0 | | 0.198 | 1440.777 | | :union-reduce | 10 | 0.114 | 1.0 | | 0.230 | 25.724 | | :union-reduce | 10000 | 0.081 | 1.0 | | 0.159 | 37008.010 | | :update-in | 5 | | 1.0 | | 1.401 | 0.292 | | :update-in-nil | 5 | | 1.0 | | 0.289 | 0.138 | | :update-values | 1000 | | 1.0 | | 0.083 | 166.794 | | :vector-access | 10 | 1.563 | 1.0 | | 1.131 | 86.023 | | :vector-access | 10000 | 0.945 | 1.0 | | 1.047 | 139.996 | | :vector-cons-obj-array | 10 | 1.150 | 1.0 | | 0.365 | 0.075 | | :vector-cons-obj-array | 10000 | 0.066 | 1.0 | | 0.040 | 103.369 | | :vector-construction | 10 | 0.508 | 1.0 | | 1.119 | 0.073 | | :vector-construction | 10000 | 0.062 | 1.0 | | 0.070 | 109.040 | | :vector-reduce | 10 | 2.041 | 1.0 | | 1.031 | 0.152 | | :vector-reduce | 10000 | 1.392 | 1.0 | | 1.026 | 146.636 | | :vector-to-array | 10 | 0.284 | 1.0 | | 0.569 | 0.036 | | :vector-to-array | 10000 | 0.052 | 1.0 | | 0.068 | 65.946 | ## CAVEATS!! This code is minimally tested. The datastructures especially need serious testing, potentially generative testing of edge cases. Also, microbenchmarks do not always indicate how your system will perform overall. For instance- when testing `assoc-in`, `update-in` in this project we see better performance. In at least one real world project, however, the inlining that makes the microbenchmark perform better definitely did *not* result in the project running faster -- it ran a bit slower even though the profiler of the original code indicated the sequence operations performed during assoc-in and update-in were a source of some time. The JVM is a complicated machine and there are issues with using, for instance, too many classes at a particular callsite. Overall I would recommend profiling and being careful. My honest opinion right now is that `assoc-in` and `update-in` do not improve program performance at least in some of the use cases I have tested. ## Other Interesting Projects * [clj-fast](https://github.com/bsless/clj-fast) - Great and important library more focused on compiler upgrades. * [bifurcan](https://github.com/lacuna/bifurcan) - High speed functional datastructures for Java. Perhaps ham-fisted should be based on this or we should measure the differences and take the good parts. * [Clojure Goes Fast](http://clojure-goes-fast.com/) - Grandaddy aggregator project with a lot of important information and a set of crucial github projects such as [clj-memory-meter](https://github.com/clojure-goes-fast/clj-memory-meter). ================================================ FILE: build.clj ================================================ (ns build (:require [clojure.tools.build.api :as b] [clojure.edn :as edn]) (:refer-clojure :exclude [compile])) (def deps-data (edn/read-string (slurp "deps.edn"))) (def codox-data (get-in deps-data [:aliases :codox :exec-args])) (def lib (symbol (codox-data :group-id) (codox-data :artifact-id))) (def version (codox-data :version)) (def class-dir "target/classes") (def basis (b/create-basis {:project "deps.edn"})) (def jar-file (format "target/%s.jar" (name lib))) (def uber-file (format "target/uber-%s.jar" (name lib))) (defn clean [_] (b/delete {:path "target"})) (defn compile [_] (b/javac {:src-dirs ["java"] :class-dir class-dir :basis basis :javac-opts ["-source" "8" "-target" "8" "-Xlint:unchecked" ]})) (def pom-template [[:licenses [:license [:name "MIT License"] [:url "https://github.com/cnuernber/charred/blob/master/LICENSE"]]]]) (defn jar [_] (compile nil) (b/write-pom {:class-dir class-dir :lib lib :version version :basis basis :src-dirs ["src"] :pom-data pom-template}) (b/copy-dir {:src-dirs ["src" "resources"] :target-dir class-dir}) (b/jar {:class-dir class-dir :jar-file jar-file})) (defn perftest [_] (let [basis (b/create-basis {:aliases [:dev]})] (clean nil) (compile nil) (b/copy-dir {:src-dirs ["src" "dev/resources" "dev/src"] :target-dir class-dir}) (b/compile-clj {:basis basis :src-dirs ["dev/src"] :class-dir class-dir :compile-opts {:direct-linking true} }) (b/uber {:class-dir class-dir :uber-file uber-file :basis basis :main 'ham-fisted.protocol-perf}))) ================================================ FILE: codegen/gen_prim_invoke.clj ================================================ (ns gen-prim-invoke (:require [clojure.java.io :as io] [ham-fisted.lazy-noncaching :as lznc] [ham-fisted.api :as hamf])) (defn single-arg-sigs [rv] (for [arg1 [:o :l :d]] [arg1 rv])) (defn dual-arg-sigs [rv arg1] (for [arg2 [:o :l :d]] [arg1 arg2 rv])) (defn triple-arg-sigs [rv arg1 arg2] (for [arg3 [:o :l :d]] [arg1 arg2 arg3 rv])) (defn quad-arg-sigs [rv arg1 arg2 arg3] (for [arg4 [:o :l :d]] [arg1 arg2 arg3 arg4 rv])) (def ifn-sigs (hamf/concatv [[:l] [:d]] (->> (lznc/concat (for [rv [:o :l :d]] (single-arg-sigs rv)) (for [rv [:o :l :d] arg1 [:o :l :d]] (dual-arg-sigs rv arg1)) (for [rv [:o :l :d] arg1 [:o :l :d] arg2 [:o :l :d]] (triple-arg-sigs rv arg1 arg2)) (for [rv [:o :l :d] arg1 [:o :l :d] arg2 [:o :l :d] arg3 [:o :l :d]] (quad-arg-sigs rv arg1 arg2 arg3))) lznc/apply-concat (lznc/remove #(every? (fn [a](= :o a)) %))))) (defn writeit [] (with-open [w (io/writer (io/output-stream "src/ham_fisted/primitive_invoke.clj"))] (.write w (str "(ns ham-fisted.primitive-invoke \"For statically traced calls the Clojure compiler calls the primitive version of type-hinted functions and this makes quite a difference in tight loops. Often times, however, functions are passed by values or returned from if-statements and then you need to explicitly call the primitive overload - this makes that pathway less verbose.\")\n\n")) (doseq [sig ifn-sigs] (let [sname (apply str (map name sig)) ifn-name (str "clojure.lang.IFn$" (.toUpperCase sname))] (.write w (str "(defn ->" sname " ^" ifn-name " [f] (if (instance? " ifn-name " f) f (throw (RuntimeException. (str f \" is not an instance of" ifn-name "\")))))\n")) (.write w (str "(defmacro " sname " [f")) (dotimes [i (dec (count sig))] (.write w (str " ")) (.write w (str "arg" i))) (.write w "]\n") (.write w (str "`(.invokePrim ~f")) (dotimes [i (dec (count sig))] (.write w (str " ")) (.write w (str "~arg" i))) (.write w "))\n"))))) (comment (writeit) ) ================================================ FILE: deps.edn ================================================ {:paths ["src" "resources" "target/classes"] :deps {it.unimi.dsi/fastutil-core {:mvn/version "8.5.14"} com.github.ben-manes.caffeine/caffeine {:mvn/version "2.9.3"} net.openhft/zero-allocation-hashing {:mvn/version "0.27ea0"}} :aliases {;; Run with clj -T:build function-in-build :dev {:extra-deps {;;org.clojure/clojure {:mvn/version "1.12.0-CN-SNAPSHOT"} org.clojure/clojure {:mvn/version "1.12.0"} criterium/criterium {:mvn/version "0.4.6"} techascent/tech.ml.dataset {:mvn/version "7.032"} ch.qos.logback/logback-classic {:mvn/version "1.1.3"} kixi/stats {:mvn/version "0.5.5"} org.clojure/data.int-map {:mvn/version "1.3.0"} techascent/tech.viz {:mvn/version "6.00-beta-16-4"} com.clojure-goes-fast/clj-java-decompiler {:mvn/version "0.3.6"} com.clojure-goes-fast/clj-memory-meter {:mvn/version "0.3.0"} com.clojure-goes-fast/clj-async-profiler {:mvn/version "1.6.2"}} :extra-paths ["dev/src" "test"] :jvm-opts ["-Djdk.attach.allowAttachSelf=true" "-XX:+EnableDynamicAgentLoading" "--illegal-access=permit"]} :nospec {:jvm-opts ["-Dclojure.spec.skip-macros=true" "-Xverify:none"]} :jdk-19 {:jvm-opts ["-Djdk.attach.allowAttachSelf=true" "--illegal-access=permit"]} :build {:deps {io.github.clojure/tools.build {:mvn/version "0.10.5"}} :ns-default build} :clj-kondo {:extra-deps {clj-kondo/clj-kondo {:mvn/version "2024.08.29"}} :main-opts ["-m" "clj-kondo.main"]} :kaocha-test {:extra-deps {lambdaisland/kaocha {:mvn/version "1.91.1392"} lambdaisland/kaocha-junit-xml {:mvn/version "1.17.101"} lambdaisland/kaocha-cloverage {:mvn/version "1.1.89"}} :extra-paths ["test"] :main-opts ["-m" "kaocha.runner"]} :test {:extra-deps {com.cognitect/test-runner {:git/url "https://github.com/cognitect-labs/test-runner" :sha "209b64504cb3bd3b99ecfec7937b358a879f55c1"} ch.qos.logback/logback-classic {:mvn/version "1.1.3"}} :extra-paths ["test"] :main-opts ["-m" "cognitect.test-runner"]} :codox {:extra-deps {codox-theme-rdash/codox-theme-rdash {:mvn/version "0.1.2"} nrepl/nrepl {:mvn/version "1.3.0"} cider/cider-nrepl {:mvn/version "0.50.2"} com.cnuernber/codox {:mvn/version "1.001"}} :exec-fn codox.main/-main :exec-args {:group-id "com.cnuernber" :artifact-id "ham-fisted" :version "3.029" :name "Ham-Fisted" :description "High Performance Clojure Primitives" :metadata {:doc/format :markdown} :html {:transforms [[:head] [:append [:script {:async true :src "https://www.googletagmanager.com/gtag/js?id=G-XJYNJF48RM"}]] [:head] [:append [:script "window.dataLayer = window.dataLayer || []; function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'G-XJYNJF48RM');"]]]} :themes [:rdash] :source-paths ["src"] :output-path "docs" :doc-paths ["topics"] :source-uri "https://github.com/cnuernber/ham-fisted/blob/master/{filepath}#L{line}" :namespaces [ham-fisted.api ham-fisted.lazy-noncaching ham-fisted.protocols ham-fisted.set ham-fisted.function ham-fisted.mut-map ham-fisted.reduce ham-fisted.hlet ham-fisted.primitive-invoke ham-fisted.bloom-filter ham-fisted.defprotocol ham-fisted.iterator ham-fisted.process ham-fisted.profile ham-fisted.fjp ham-fisted.spliterator]}} :deploy {:replace-deps {slipset/deps-deploy {:mvn/version "0.2.2"}} :exec-fn deps-deploy.deps-deploy/deploy :exec-args {:installer :remote :sign-releases? true :artifact "target/ham-fisted.jar"}} :install {:replace-deps {slipset/deps-deploy {:mvn/version "0.2.2"}} :exec-fn deps-deploy.deps-deploy/deploy :exec-args {:installer :local :artifact "target/ham-fisted.jar"}}}} ================================================ FILE: dev/resources/logback.xml ================================================ %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n ================================================ FILE: dev/src/ham_fisted/analysis.clj ================================================ (ns ham-fisted.analysis (:require [clojure.edn :as edn] [charred.api :as charred] [applied-science.darkstar :as darkstar] [clojure.java.io :as io])) (defn plot-perf-test [testgroup options] (let [group-item (first testgroup) testname (:test group-item) numeric? (:numeric? group-item) bytes? (= (:test group-item) :hashmap-bytes) baseline (or (:baseline options) :clj) testdata (->> testgroup (mapcat (if bytes? (fn [group-item] (let [n-elems (:n-elems group-item) clj-bytes (:clj group-item)] (->> (dissoc group-item :test :numeric? :n-elems) (map (fn [kv] {:datastructure (key kv) :n-elems n-elems :norm-bytes (double (/ (val kv) clj-bytes))}))))) (fn [group-item] (let [n-elems (:n-elems group-item) clj-val (get-in group-item [baseline :mean-μs])] (->> group-item (map (fn [kv] (let [k (key kv) val (val kv)] (when-let [us (get val :mean-μs)] {:datastructure k :norm-mean-μs (double (/ us clj-val)) :n-elems n-elems})))) (remove nil?)))))) (sort-by :datastructure) (vec)) chart-title (str (name testname) "-" (if numeric? "numeric" "non-numeric")) chartname (str "charts/" chart-title ".svg")] (io/make-parents chartname) (if bytes? (spit chartname (-> {:$schema "https://vega.github.io/schema/vega-lite/v5.1.0.json" :mark {:type :line :point true} :title chart-title :width 800 :height 600 :data {:values testdata} :encoding {:y {:field :norm-bytes, :type :quantitative :axis {:grid false}} :x {:field :n-elems :type :quantitative :scale {:type :log}} :color {:field :datastructure :type :nominal} :shape {:field :datastructure :type :nominal}}} (charred/write-json-str) (darkstar/vega-lite-spec->svg))) (spit chartname (-> {:$schema "https://vega.github.io/schema/vega-lite/v5.1.0.json" :mark {:type :line :point true} :title chart-title :width 800 :height 600 :data {:values testdata} :encoding {:y {:field :norm-mean-μs, :type :quantitative :axis {:grid false}} :x {:field :n-elems :type :quantitative :scale {:type :log}} :color {:field :datastructure :type :nominal} :shape {:field :datastructure :type :nominal}}} (charred/write-json-str) (darkstar/vega-lite-spec->svg)))) ;;testdata )) (defn- process-files ([fnames] (process-files fnames nil)) ([fnames options] (->> fnames (mapcat #(edn/read-string (slurp %))) (group-by (juxt :numeric? :test)) (vals) (map #(plot-perf-test % options)) (dorun)) :ok)) (defn general-hashmap-analysis [] (process-files ["results/general-hashmap.edn"]) ) (defn random-update-analysis [] (process-files ["results/random-update.edn"])) (defn union-analysis [] (process-files ["results/union-overlapping.edn" "results/union-disj.edn" "results/union-reduce.edn" "results/update-values.edn"])) (defn typed-reduction-analysis [] (process-files ["results/typed-reductions.edn"])) (defn typed-parallel-reduction-analysis [] (process-files ["results/typed-parallel-reductions.edn"])) (defn union-reduce-transient [] (process-files ["results/union-reduce-transient.edn"] {:baseline :hamf-hashmap}) ) (defn persistent-vector [] (process-files ["results/persistent-vector.edn"])) (defn sort-by-analysis [] (process-files ["results/sort-by.edn"])) ================================================ FILE: dev/src/ham_fisted/benchmark.clj ================================================ (ns ham-fisted.benchmark (:require [criterium.core :as c])) (defmacro benchmark-us "Benchmark an op, returning a map of :mean and :variance in μs." [op] `(let [bdata# (c/quick-benchmark ~op nil)] {:mean-μs (* (double (first (:mean bdata#))) 1e6) :variance-μs (* (double (first (:variance bdata#))) 1e6)})) ================================================ FILE: dev/src/ham_fisted/protocol_perf.clj ================================================ (ns ham-fisted.protocol-perf (:require [ham-fisted.defprotocol :as hamf-defproto] [ham-fisted.protocols :as hamf-proto] [ham-fisted.lazy-noncaching :as lznc] [ham-fisted.api :as hamf] [ham-fisted.reduce :as hamf-rf] [clojure.pprint :as pp]) (:import [java.util LongSummaryStatistics] [java.util.function LongConsumer]) (:gen-class)) (set! *unchecked-math* :warn-on-boxed) (set! *warn-on-reflection* true) (hamf-defproto/defprotocol HamfMemsize (^long hamf-memsize [m])) (hamf-defproto/extend Double HamfMemsize {:hamf-memsize 24}) (hamf-defproto/extend Long HamfMemsize {:hamf-memsize 24}) (hamf-defproto/extend clojure.lang.Keyword HamfMemsize {:hamf-memsize 48}) (hamf-defproto/extend-protocol HamfMemsize String (hamf-memsize [s] (+ 24 (.length ^String s))) java.util.Collection (hamf-memsize [c] (hamf/lsum (lznc/map (fn ^long [d] (+ 24 (hamf-memsize d))) c))) java.util.Map (hamf-memsize [c] (hamf/lsum (lznc/map (fn ^long [kv] (+ 36 (+ (hamf-memsize (key kv)) (hamf-memsize (val kv))))) c)))) (clojure.core/defprotocol CoreMemsize (core-memsize [m])) (clojure.core/extend-protocol CoreMemsize Double (core-memsize [d] 24) Long (core-memsize [l] 24) clojure.lang.Keyword (core-memsize [k] 48) String (core-memsize [s] (+ 24 (.length ^String s))) java.util.Collection (core-memsize [c] (hamf/lsum (lznc/map (fn ^long [d] (+ 24 (long (core-memsize d)))) c)) #_(reduce (fn [s v] (+ s 24 (core-memsize v))) 0 c)) java.util.Map (core-memsize [m] (hamf/lsum (lznc/map (fn ^long [kv] (+ 36 (+ (long (core-memsize (key kv))) (long (core-memsize (val kv)))))) m)) #_(reduce (fn [s [k v]] (+ s 36 (core-memsize k) (core-memsize v))) 0 m))) (def test-datastructure {:a "hello" :b 24 :c (into [] (repeat 1000 (rand))) :d (into [] (repeat 1000 1))}) (def measure-data (into [] (repeat 10000 test-datastructure))) (defn multithread-test [measure-fn] (hamf/lsum (hamf/pmap measure-fn measure-data))) (hamf-defproto/defprotocol PPrimitiveArgs (^double pargs [m ^long b])) (hamf-defproto/extend-type String PPrimitiveArgs (pargs [m b] (+ 1.0 (+ (.length m) b)))) (hamf-defproto/extend-type Double PPrimitiveArgs (pargs [m b] (+ 1.0 (+ (double m) b)))) (defprotocol CorePPrimitiveArgs (core-pargs [m b])) (extend-type String CorePPrimitiveArgs (core-pargs [m b] (+ 1.0 (+ (.length m) (long b))))) (extend-type Double CorePPrimitiveArgs (core-pargs [m b] (+ 1.0 (+ (double b) (long b))))) (def strs (mapv str (range 100000))) (def strs-and-doubles (vec (take 100000 (interleave strs (map double (range 100000)))))) (defn reduce-count [data] (reduce (fn [^long acc v] (inc acc)) 0 data)) (defprotocol TestProto (f [this a b])) (extend-protocol TestProto String (f [this a b] 1) Long (f [this a b] 1) Double (f [this a b] 1) java.util.Collection (f [this a b] (reduce-count (lznc/map #(f % [] :x) this)))) (hamf-defproto/defprotocol HFTestProto (hf [this a b])) (hamf-defproto/extend-protocol HFTestProto String (hf [this a b] 1) Long (hf [this a b] 1) Double (hf [this a b] 1) java.util.Collection (hf [this a b] (reduce-count (lznc/map #(hf % [] :x) this)))) (defn explore! [n] (println "========= Exploring general protocol pathways ========") (let [l (vec (take 10000 (cycle ["foo" 5678 3.14 (vec (take 10000 (cycle ["foo" 5678 3.14])))])))] (dotimes [i n] (println "attempt" i) (println "map f") (time (reduce-count (lznc/map #(f % [] :x) l))) (println "map fast-f") (time (reduce-count (lznc/map #(hf % [] :x) l))) (println "pmap f") (time (reduce-count (hamf/pmap #(f % [] :x) l))) (println "pmap fast-f") (time (reduce-count (lznc/map identity (hamf/pmap #(hf % [] :x) l)))))) :done) (defn -main [& args] (println "Core protocols") (dotimes [idx 5] (time (multithread-test core-memsize))) (println "hamf protocols") (dotimes [idx 5] (time (multithread-test hamf-memsize))) (println "serial single type core pargs") (dotimes [idx 5] (time (dotimes [idx 10] (hamf/sum-fast (lznc/map (fn ^double [s] (core-pargs s 100)) strs))))) (println "serial single type hamf pargs") (dotimes [idx 5] (time (dotimes [idx 10] (hamf/sum-fast (lznc/map (fn ^double [s] (pargs s 100)) strs))))) (println "parallel single type core pargs") (dotimes [idx 5] (time (dotimes [idx 10] (hamf/sum (lznc/map (fn ^double [s] (core-pargs s 100)) strs))))) (println "parallel single type hamf pargs") (dotimes [idx 5] (time (dotimes [idx 10] (hamf/sum (lznc/map (fn ^double [s] (pargs s 100)) strs))))) (println "serial dual type core pargs") (dotimes [idx 5] (time (dotimes [idx 10] (hamf/sum-fast (lznc/map (fn ^double [s] (core-pargs s 100)) strs-and-doubles))))) (println "serial dual type hamf pargs") (dotimes [idx 5] (time (dotimes [idx 10] (hamf/sum-fast (lznc/map (fn ^double [s] (pargs s 100)) strs-and-doubles))))) (println "parallel dual type core pargs") (dotimes [idx 5] (time (dotimes [idx 10] (hamf/sum (lznc/map (fn ^double [s] (core-pargs s 100)) strs-and-doubles))))) (println "parallel dual type hamf pargs") (dotimes [idx 5] (time (dotimes [idx 10] (hamf/sum (lznc/map (fn ^double [s] (pargs s 100)) strs-and-doubles))))) (explore! 4) :ok) ================================================ FILE: dev/src/perftest.clj ================================================ (ns perftest (:require [ham-fisted.api :as hamf] [ham-fisted.function :as hamf-fn] [ham-fisted.reduce :as hamf-rf] [ham-fisted.lazy-noncaching :as lznc] [ham-fisted.benchmark :refer [benchmark-us] :as bench] [clojure.data.int-map :as i] [tech.v3.datatype :as dtype] [tech.v3.datatype.functional :as dfn] [clojure.tools.logging :as log] [clojure.pprint :as pp] [clojure.java.shell :as sh] [clojure.java.io :as io] [clojure.string :as str] [criterium.core :as crit] [clj-memory-meter.core :as mm]) (:import [java.util HashMap ArrayList Map List Map$Entry ArrayList] [java.util.function BiFunction] [ham_fisted IMutList Sum$SimpleSum Sum Consumers$IncConsumer] [clojure.lang PersistentHashMap]) (:gen-class)) (set! *unchecked-math* :warn-on-boxed) (defn long-map-data [^long n-elems] (lznc/->random-access (lznc/object-array (lznc/map #(hamf/vector % %) (lznc/repeatedly n-elems #(long (rand-int 100000))))))) (defn kwd-map-data [^long n-elems] (lznc/->random-access (lznc/object-array (lznc/map #(hamf/vector (keyword (str %)) %) (lznc/repeatedly n-elems #(rand-int 100000)))))) (def map-non-numeric-constructors {:clj #(into {} %) #_#_:hamf-trie hamf/mut-trie-map :hamf-hashmap hamf/mut-map :java hamf/java-hashmap}) (def map-numeric-constructors {:hamf-long-map hamf/mut-long-hashtable-map :clj-int-map #(into (i/int-map) %)}) (defn map-constructors [numeric?] (merge map-non-numeric-constructors (when numeric? map-numeric-constructors))) (defn initial-maps [mapsc data] (hamf/mapmap #(vector (key %) ((val %) data)) mapsc)) (defn hashmap-perftest [data] (let [numeric? (number? (ffirst data)) n-elems (count data) _ (log/info (str "Running map benchmark on " (if numeric? "numeric " "non-numeric ") "data with n=" n-elems)) map-constructors (map-constructors numeric?) map-data (initial-maps map-constructors data)] [(merge (hamf/mapmap (fn [entry] [(key entry) (benchmark-us ((val entry) data))]) map-constructors) {:n-elems n-elems :test :hashmap-construction :numeric? numeric?}) (merge (hamf/mapmap #(vector (key %) (benchmark-us (reduce (fn [acc data] (.get ^Map acc (data 0)) acc) (val %) data))) map-data) {:n-elems n-elems :test :hashmap-access :numeric? numeric?}) (merge (hamf/mapmap #(vector (key %) (benchmark-us (fn [acc map] (reduce (fn [& kv] kv) nil (val %))))) map-data) {:n-elems n-elems :test :hashmap-reduction :numeric? numeric?}) (merge (hamf/mapmap #(vector (key %) (mm/measure (val %) :bytes true)) map-data) {:n-elems n-elems :test :hashmap-bytes :numeric? numeric?})])) (defn- spit-data [testname data] (let [data (vec data) fname (str "results/" testname ".edn")] (io/make-parents fname) (spit fname (pr-str data)))) (defn general-hashmap [] (->> (for [n-elems [4 10 100 1000 10000 1000000 ] numeric? [true false ]] (hashmap-perftest (if numeric? (long-map-data n-elems) (kwd-map-data n-elems)))) (lznc/apply-concat) (vec) (spit-data "general-hashmap"))) (defn random-updates [] (->> (for [n-elems [ 4 10 100 1000 10000 1000000 ] numeric? [true false ] ] (do (log/info (str "random-update benchmark on " (if numeric? "numeric " "non-numeric ") "data with n=" n-elems)) (let [cycle-size 1000 n-elems (long n-elems) data (vec (take (max cycle-size n-elems) (cycle (repeatedly (min n-elems cycle-size) #(long (rand-int 1000)))))) data (if numeric? data (map (comp keyword str) data)) init-maps (initial-maps (map-constructors numeric?) nil)] (merge (hamf/mapmap (fn [kv] (let [m (val kv) immut-path? (instance? clojure.lang.IPersistentMap m)] [(key kv) (if immut-path? (benchmark-us (reduce (fn [acc v] (assoc! acc v (unchecked-inc (long (get acc v 0))))) (transient m) data)) (let [cfn (hamf-fn/function _v (Consumers$IncConsumer.))] (benchmark-us (reduce (fn [^Map acc v] (.inc ^Consumers$IncConsumer (.computeIfAbsent acc v cfn)) acc) m data))))])) init-maps) {:n-elems n-elems :numeric? numeric? :test :random-update})))) (vec) (spit-data "random-update"))) #_(defn union-overlapping [] (->> (for [n-elems [4 10 100 1000 10000 1000000 ] numeric? [true false ] ] (do (log/info (str "union-overlapping benchmark on " (if numeric? "numeric " "non-numeric ") "data with n=" n-elems)) (let [data (if numeric? (long-map-data n-elems) (kwd-map-data n-elems)) constructors (map-constructors numeric?) init-maps (initial-maps constructors data) merge-bfn (hamf-fn/bi-function a b (+ (long a) (long b)))] (merge (hamf/mapmap (fn [kv] (let [m (hamf/persistent! (val kv))] [(key kv) (cond (instance? clojure.lang.IPersistentMap m) (benchmark-us (merge-with merge-bfn m m)) (instance? BitmapTrieCommon$MapSet m) (benchmark-us (.union ^BitmapTrieCommon$MapSet m m merge-bfn)) :else (let [map-c (get constructors (key kv))] (benchmark-us (hamf/map-union merge-bfn (map-c m) m))))])) init-maps) {:n-elems n-elems :numeric? numeric? :test :union-overlapping})))) (vec) (spit-data "union-overlapping") )) #_(defn union-disj [] (->> (for [n-elems [4 10 100 1000 10000 1000000 ] numeric? [true false ] ] (do (log/info (str "union-disj benchmark on " (if numeric? "numeric " "non-numeric ") "data with n=" n-elems)) (let [n-elems (long n-elems) data (if numeric? (long-map-data n-elems) (kwd-map-data n-elems)) lhs-data (vec (take (quot n-elems 2) data)) rhs-data (vec (drop (- n-elems (count lhs-data)) data)) constructors (map-constructors numeric?) lhs-maps (initial-maps constructors lhs-data) rhs-maps (initial-maps constructors rhs-data) merge-bfn (hamf-fn/bi-function a b (+ (long a) (long b)))] (merge (hamf/mapmap (fn [kv] ;;Make sure we can't edit lhs (let [lhs (hamf/persistent! (val kv)) rhs (rhs-maps (key kv))] [(key kv) (cond (instance? clojure.lang.IPersistentMap lhs) (benchmark-us (merge-with merge-bfn lhs rhs)) (instance? BitmapTrieCommon$MapSet lhs) (benchmark-us (.union ^BitmapTrieCommon$MapSet (hamf/transient lhs) rhs merge-bfn)) :else (let [map-c (get constructors (key kv))] (benchmark-us (hamf/map-union merge-bfn (map-c lhs ) rhs))))])) lhs-maps) {:n-elems n-elems :numeric? numeric? :test :union-disj})))) (vec) (spit-data "union-disj") )) #_(defn union-reduce [] (->> (for [n-elems [ 4 10 100 1000 10000 1000000 ] numeric? [true false ] ] (do (log/info (str "union-reduce benchmark on " (if numeric? "numeric " "non-numeric ") "data with n=" n-elems)) (let [data (if numeric? (long-map-data n-elems) (kwd-map-data n-elems)) constructors (map-constructors numeric?) init-maps (initial-maps constructors data) merge-bfn (hamf-fn/bi-function a b (+ (long a) (long b)))] (merge (hamf/mapmap (fn [kv] (let [m (hamf/persistent! (val kv)) mseq (vec (repeat 16 m))] [(key kv) (cond (instance? clojure.lang.IPersistentMap m) (benchmark-us (reduce #(merge-with merge-bfn %1 %2) m mseq)) (instance? BitmapTrieCommon$MapSet m) (benchmark-us (reduce #(.union ^BitmapTrieCommon$MapSet %1 %2 merge-bfn) m mseq)) :else (let [map-c (get constructors (key kv))] (benchmark-us (reduce #(hamf/map-union merge-bfn %1 %2) (map-c m) mseq))))])) init-maps) {:n-elems n-elems :numeric? numeric? :test :union-reduce})))) (vec) (spit-data "union-reduce"))) (defn union-reduce-transient "Comparing reducing into a single transient container vs. doing a shallow clone every union call." [] (->> (for [n-elems [ 4 10 100 1000 10000 1000000 ] numeric? [true ] ] (do (log/info (str "union-reduce-transient benchmark on " (if numeric? "numeric " "non-numeric ") "data with n=" n-elems)) (let [data (if numeric? (long-map-data n-elems) (kwd-map-data n-elems)) constructors (map-constructors numeric?) hashmap (hamf/mut-map data) long-hashmap (hamf/mut-map data) init-maps {:hamf-hashmap (persistent! hashmap) :hamf-long-map (persistent! long-hashmap) :hamf-trans-hashmap (with-meta (persistent! hashmap) {:transient? true}) :hamf-trans-long-map (with-meta (persistent! long-hashmap) {:transient? true})} merge-bfn (hamf-fn/bi-function a b (+ (long a) (long b)))] (merge (hamf/mapmap (fn [kv] (let [m (val kv) mseq (vec (repeat 16 m))] [(key kv) (benchmark-us (reduce #(.union (hamf/as-map-set %1) %2 merge-bfn) (if (:transient? (meta m)) (transient m) m) mseq))])) init-maps) {:n-elems n-elems :numeric? numeric? :test :union-reduce-transient})))) (vec) (spit-data "union-reduce-transient"))) #_(defn update-values [] (->> (for [n-elems [ 4 10 100 1000 10000 1000000 ] numeric? [true false ] ] (do (log/info (str "update-values benchmark on " (if numeric? "numeric " "non-numeric ") "data with n=" n-elems)) (let [data (if numeric? (long-map-data n-elems) (kwd-map-data n-elems)) constructors (map-constructors numeric?) init-maps (initial-maps constructors data) update-bfn (hamf-fn/bi-function k v (unchecked-inc (long v)))] (merge (hamf/mapmap (fn [kv] (let [m (val kv)] [(key kv) (cond ;;for whatever reason update-vals isn't found during uberjar build (instance? clojure.lang.IPersistentMap m) (benchmark-us (update-vals m unchecked-inc)) (instance? ImmutValues m) (benchmark-us (.immutUpdateValues ^ImmutValues m update-bfn)) :else (benchmark-us (.replaceAll ^Map ((constructors (key kv)) m) update-bfn)))])) init-maps) {:n-elems n-elems :numeric? numeric? :test :update-values})))) (vec) (spit-data "update-values"))) (deftype LongAccum [^{:unsynchronized-mutable true :tag long} val] clojure.lang.IDeref (deref [this] val) java.util.function.LongConsumer (accept [this v] (set! val (+ v val)))) (defn typed-reductions [] (->> (for [n-elems [ 4 10 100 1000 10000 1000000 ]] (do (log/info (str "typed reduction benchmark on " (if true "numeric " "non-numeric ") "data with n=" n-elems)) (merge {:clj (benchmark-us (transduce (comp (map #(+ (long %) 10)) (filter #(== 0 (rem (long %) 2)))) + 0 (hamf/range n-elems))) :clj-typed (benchmark-us (transduce (comp (map (hamf-fn/long-unary-operator a (+ a 10))) (filter (hamf-fn/long-predicate a (== 0 (rem a 2))))) (reify ham_fisted.IFnDef$LLL (invokePrim [this a b] (+ a b)) (invoke [this a] a)) 0 (hamf/range n-elems))) :hamf-partial (benchmark-us (->> (hamf/range n-elems) (lznc/map (hamf-fn/long-unary-operator a (+ a 10))) (lznc/filter (hamf-fn/long-predicate a(== 0 (rem a 2)))) (reduce (hamf-fn/long-binary-operator a b (+ a b)) 0))) :hamf-deftype-consumer (benchmark-us (->> (hamf/range n-elems) (lznc/map (hamf-fn/long-unary-operator a (+ a 10))) (lznc/filter (hamf-fn/long-predicate a(== 0 (rem a 2)))) (reduce hamf-rf/long-consumer-accumulator (LongAccum. 0)))) :hamf-java-consumer (benchmark-us (->> (hamf/range n-elems) (lznc/map (hamf-fn/long-unary-operator a (+ a 10))) (lznc/filter (hamf-fn/long-predicate a(== 0 (rem a 2)))) (reduce hamf-rf/long-consumer-accumulator (ham_fisted.LongAccum. 0))))} {:n-elems n-elems :numeric? true :test :typed-reductions}))) (vec) (spit-data "typed-reductions-intel"))) (defn typed-parallel-reductions [] (->> (for [n-elems [1000 1000000 100000000 ]] (do (log/info (str "parallel reduction benchmark on " (if true "numeric " "non-numeric ") "data with n=" n-elems)) (let [ops (hamf-rf/options->parallel-options {:min-n 10})] (merge {:clj (benchmark-us (transduce (comp (map #(+ (long %) 10)) (filter #(== 0 (rem (long %) 2)))) + 0 (hamf/range n-elems))) :hamf-untyped (benchmark-us (->> (hamf/range n-elems) (lznc/map #(+ (long %) 10)) (lznc/filter #(== 0 (rem (long %) 2))) (hamf-rf/preduce (constantly 0) #(+ (long %1) (long %2)) #(+ (long %1) (long %2)) ops))) :hamf-typed (benchmark-us (->> (hamf/range n-elems) (lznc/map (hamf-fn/long-unary-operator a (+ a 10))) (lznc/filter (hamf-fn/long-predicate a(== 0 (rem a 2)))) (hamf-rf/preduce (constantly 0) (hamf-fn/long-binary-operator a b (+ a b)) (hamf-fn/long-binary-operator a b (+ a b)) ops))) :hamf-consumer (benchmark-us (->> (hamf/range n-elems) (lznc/map (hamf-fn/long-unary-operator a (+ a 10))) (lznc/filter (hamf-fn/long-predicate a(== 0 (rem a 2)))) (hamf-rf/preduce #(ham_fisted.LongAccum. 0) hamf-rf/long-consumer-accumulator hamf-rf/reducible-merge ops)))} {:n-elems n-elems :numeric? true :test :typed-parallel-reductions})))) (vec) (spit-data "typed-parallel-reductions") )) (def persistent-vector-constructors {:clj vec :hamf hamf/immut-list :hamf-objary #(hamf/immut-list (hamf/object-array %)) :java #(doto (ArrayList.) (.addAll (hamf/->random-access %)))}) (defn persistent-vector-perftest [] (->> (for [n-elems [4 10 100 1000 10000 100000]] (do (log/info (str "persistent vector perftest with n= " n-elems)) [(merge (hamf/mapmap (fn [kv] [(key kv) (benchmark-us ((val kv) (range n-elems)))]) persistent-vector-constructors) {:n-elems n-elems :numeric? true :test :vector-construction}) (merge (hamf/mapmap (fn [kv] (let [data ((val kv) (range n-elems))] [(key kv) (benchmark-us (dotimes [idx n-elems] (.get ^List data idx)))])) persistent-vector-constructors) {:n-elems n-elems :numeric? true :test :vector-access}) (merge (hamf/mapmap (fn [kv] (let [data ((val kv) (range n-elems))] [(key kv) (benchmark-us (.toArray ^List data))])) persistent-vector-constructors) {:n-elems n-elems :numeric? true :test :to-object-array}) (merge (hamf/mapmap (fn [kv] (let [data (double-array (range n-elems))] [(key kv) (benchmark-us ((val kv) data))])) persistent-vector-constructors) {:n-elems n-elems :numeric? true :test :from-double-array}) ])) (lznc/apply-concat) (vec) (spit-data "persistent-vector"))) (defn sort-by-perftest [] (->> (for [n-elems [4 10 100 1000 10000 100000 1000000 ]] (do (log/info (str "sort-by perftest with n= " n-elems)) (let [data (mapv (fn [idx] {:a 1 :b idx}) (shuffle (range n-elems)))] {:clj (benchmark-us (sort-by :b data)) :hamf (benchmark-us (hamf/sort-by :b data)) :hamf-typed (benchmark-us (hamf/sort-by (hamf-fn/obj->long d (long (d :b))) data)) :n-elems n-elems :test :sort-by :numeric? true}))) (vec) (spit-data "sort-by"))) (defn concatv-perftest [] (->> (for [n-elems [4 10 100 1000 10000 100000 1000000 ]] (do (log/info (str "concatv perftest with n= " n-elems)) (let [data (vec (repeat 10 (range n-elems)))] {:clj (benchmark-us (into [] cat data)) :hamf (benchmark-us (hamf/apply-concatv data)) :hamf-objarry (benchmark-us (apply hamf/concata data)) :n-elems n-elems :test :concatv :numeric? true}))) (vec) (spit-data "concatv"))) (defn vec-equals-perftest [] (->> (for [n-elems [4 10 100 1000 10000 100000 ] numeric [true false]] (do (log/info (str "concatv perftest with n= " n-elems)) (let [src-data (if numeric (range n-elems) (map (comp str keyword) (range n-elems)))] {:clj (let [lhs (vec src-data) rhs (vec src-data)] (benchmark-us (= lhs rhs))) :hamf (let [lhs (hamf/vec src-data) rhs (hamf/vec src-data)] (benchmark-us (= lhs rhs))) :n-elems n-elems :test :equals :numeric? numeric}))) (vec) (spit-data "vec-equals"))) (defn -main [& args] ;;shutdown test (persistent-vector-perftest) #_(concatv-perftest) #_(vec-equals-perftest) #_(let [perf-data (process-dataset (profile)) vs (System/getProperty "java.version") mn (machine-name) gs (git-sha) fname (str "results/" gs "-" mn "-jdk-" vs ".edn")] (print-dataset perf-data) (println "Results stored to:" fname) ;;(spit fname (with-out-str (pp/pprint {:machine-name mn :git-sha gs :jdk-version vs :dataset perf-data}))) ) (println "exiting")) ================================================ FILE: docs/Reductions.html ================================================ Reductions

Reductions

The ham-fisted project extends the concept of Clojure's reduce in a few ways, taking influence from java streams and Clojure transducers. The most important way is a formal definition of a parallel reduction (analogous to pmap for map).

Most interesting is the 3 argument form of (reduce rfn init coll). Problems exist with the 2 argument form (reduce rfn coll) as the reduction function - rfn's leftmost argument is sometimes a value from the collection and at other times an accumulated value. Some reductions have the property that the accumulator is in the set of objects in the collection (such as numeric +), these reductions are not the most general. They are a special case of a reduction where the accumulator may be a different type entirely than the values in the collection.

Parallelizable Containers

Efficient parallel reductions depend on parallelizable containers.

Java has three types of containers that operate efficiently in parallel.

  1. Finite random access containers (ex: an array)
  2. Containers that can provide spliterators (ex: hashtable)
  3. A concatenation of containers suitable for parallel computation over the parts

These three types of containers we can parallelize; random access containers, maps and sets (or more generally anything with a correct spliterator implementation), and concatenations of sub-containers each of which may not itself have a parallelizable reduction.

Parallelized Reduction

A parallelized reduction works by splitting up elements of the data source. Many reduction contexts operate simultaneous each of which will perform a serial reduction. A separate step merges the results back together. This may be thought of as the "true" map-reduce, but either way it may be useful to compare a parallelized reduction in detail to a serial reduction.

To perform a parallel reduction, four things must be provided:

  • init-val-fn - a function to produce initial accumulators for each reduction context
  • rfn - a function that takes an accumulator and a value and updates the accumulator --- This is the typical reduction function passed as the first argument to Clojure's reduce
  • merge-fn - a function that takes two accumulators and merges them to produces one result accumulator.
  • coll - a collection of items to reduce to a single output

Here are the function signatures (Keep in mind ... preduce:reduce :: pmap:map):

(defn preduce [init-val-fn rfn merge-fn coll] ...)

Notably Java streams have a 'collect' method that takes the same four arguments where the collection is the this object:

interface Stream<E> {
<R> R collect(Supplier<R> supplier,
              BiConsumer<R,? super T> accumulator,
              BiConsumer<R,R> combiner);
}

The parallelizable reduction operates as a a serial reduction if the init-val-fn is called exactly once with no arguments and the entire collection is passed along with rfn to reduce:

(reduce rfn (init-val-fn) coll)

From there preduce essentially switches on the type of coll and performs one of four distinct types of reductions:

  • serial
  • parallel over and index space
  • parallel over and spliterator space
  • parallel over sub-containers

Map, Filter, Concat Chains

It is common in functional programming to implement data transformations as chains of map, filter, and concat operations. Analyzing sequences of these operations is insight with regards to reduction in general and parallelization of reductions.

The first insight is found in the Clojure transducer pathways and involves collapsing the reduction function when possible for map and filter applications. Let's start with a reduction of the form (->> coll (map x) (filter y) (reduce ...)).

The filter operator can specialize its reduce implementation by producing a new reduction function and reducing over its source collection:

public Object reduce(IFn rfn, Object init) {
	return source.reduce(new IFn() {
	  public Object invoke(Object acc, Object v) {
	    if(pred.test(v))
		  return rfn.invoke(acc, v);
	    else
		  return acc;
	  }
	}, init);
}

This results in 'collapsing' the reduction allowing the source to perform the iteration across its elements and simply dynamically creating a slightly more complex reduction function, rfn. A similar pathway exists for map as we can always delegate up the chain making a slightly more complex reduction function as long as we are reducing over a single source of data. This optimization leads to many fewer function calls and intermediate collections when compared with naive implementations of map and filter. Clojure's transducers do this automatically.

Collapsing the reduction also allows us to parallelize reductions like the initial one stated before as if the filter object has a parallelReduction method that does an identical collapsing pathway then if the source is parallelizable then the reduction itself can still parallelize:

public Object parallelReduction(IFn initValFn, IFn rfn, IFn mergeFn) {
  return source.parallelReduction(initValFn, new IFn() {...}, mergeFn);
}

If the source collection itself allows for parallel reduction, then it's possible to achieve similar 'collapsing' in preduce. Clojure's transducers do not have this particular optimization for parallel reduction, but Java streams do.

Also worth noting, these optimizations are only available if we use the 4 argument form of reduce and if we assume that map, filter, and concat are lazy and non-caching.

With those assumptions in place it is possible to parallelize a reduction over the entries, keys or values of map using simple primitive composition:

user> (require '[ham-fisted.api :as hamf])
nil
user> (require '[ham-fisted.lazy-noncaching :as lznc])
nil
user> (def data (hamf/immut-map (lznc/map #(vector % %) (range 20000))))
#'user/data
user> (type data)
ham_fisted.PersistentHashMap
user> (hamf/preduce + + + (lznc/map key data))
199990000

Stream-based Reductions

Java streams have a notion of parallel reduction built-in. Their design suffers from two flaws, one minor and one major.

The first minor flaw is that you can ask a stream for a parallel version of itself and it will give you one if possible else return a copy of itself. Unfortunately this only works on the first stream in a pipeline so for instance:

  coll.stream().map().filter().parallel().collect();

yields a serial reduction while:

  coll.stream().parallel().map().filter().collect();

yields a parallel reduction.

This is unfortunate because it means you must go back in time to get a parallel version of the stream if you want to perform a parallel collection; something that may or may not be easily done at the point in time when you decide you do in fact want to parallel reduction (especially in library code).

The second and more major flaw is that stream-based parallelization does not allow the user to pass in their own fork-join pool at any point. This limits use to the built in pool where it's pad form to park threads or do blocking operations.

reducers.clj And Parallel Folds

Clojure has an alpha namespace that provides a parallel reduction, reducers.clj. The signature for this method is:

(defn fold
  "Reduces a collection using a (potentially parallel) reduce-combine
  strategy. The collection is partitioned into groups of approximately
  n (default 512), each of which is reduced with reducef (with a seed
  value obtained by calling (combinef) with no arguments). The results
  of these reductions are then reduced with combinef (default
  reducef). combinef must be associative, and, when called with no
  arguments, (combinef) must produce its identity element. These
  operations may be performed in parallel, but the results will
  preserve order."
  {:added "1.5"}
  ([reducef coll] (fold reducef reducef coll))
  ([combinef reducef coll] (fold 512 combinef reducef coll))
  ([n combinef reducef coll]
     (coll-fold coll n combinef reducef)))

In this case we use overloads of combinef or reducef to provider the initial accumulator (called the identity element), the rfn, finalization and the merge function. combinef called with no arguments provides each thread context's accumulator and called with two arguments performs a merge of two accumulators. reducef called with 2 arguments provides the reduction from a value into the accumulator and when called with one argument finalizes both the potentially stateful reducing function and finalizes the accumulator. It prescribes the parallelization system but users can override a protocol to do it themselves.

This the same major drawback as the java stream system, namely users cannot provide their own pool for parallelization.

An interesting decision was made here as to whether one can actually parallelize the reduction or not. Transducers, the elements providing reducef, may be stateful such as (take 15). One interesting difference is that state is done with a closure in the reduction function as opposed to providing a custom accumulator that wraps the user's accumulator but tracks state.

One aspect we haven't discussed but that is also handled here in an interesting manner is that whether a reduction can be parallelized or not is a function both of the container and of the reducer. reducers.clj does a sort of double-dispatch where the transducer may choose to implement the parallel reduction, called coll-fold or not and is queried first and if it allows parallel reduction then the collection itself is dispatched. Overall this is a great, safe choice because it disallows completely parallel dispatch if the transducer or the collection do not support it.

Parallel Reducers

If we combine all three functions: init-val-fn, rfn, and merge-fn into one object then we get a ParallelReducer, defined in protocols.clj. This protocol allows the user to pass a single object into a parallelized reduction as opposed to three functions which is useful when we want to have many reducers reduce over a single source of data. A finalize method is added in order to allow compositions of reducers, and to allow reducers to hide state and information from end users:

(defprotocol ParallelReducer
  "Parallel reducers are simple a single object that you can pass into preduce as
  opposed to 3 separate functions."
  (->init-val-fn [item]
    "Returns the initial values for a parallel reduction.  This function
takes no arguments and returns the initial reduction value.")
  (->rfn [item]
    "Returns the reduction function for a parallel reduction. This function takes
two arguments, the initial value and a value from the collection and returns a new
initial value.")
  (->merge-fn [item]
    "Returns the merge function for a parallel reduction.  This function takes
two initial values and returns a new initial value.")
  (finalize [item v]
    "A finalize function called on the result of the reduction after it is
reduced but before it is returned to the user.  Identity is a reasonable default."))

There are defaults to the reducer protocol for an IFn which assumes it can be called with no arguments for a initial value and two arguments for both reduction and merge. This works for things like + and *. Additionally there are implementations provided for the ham_fisted Sum (Kahans compensated) and SimpleSum DoubleConsumer classes.

With the three functions bundled into one logical protocol or object it is easy then to create complex (aggregate) and efficient parallelized reductions:

user> (require '[ham-fisted.reduce :as hamf-rf])
nil
user> (hamf-rf/preduce-reducers {:sum + :product *} (range 1 20))
{:product 121645100408832000, :sum 190}
user>

This goes over the data in parallel, exactly once.

Consumers, Transducers, and rfn Chains

If we look at the reduction in terms of a push model as opposed to a pull model where the stream will push data into a consumer then we can implement similar chains or map and filter. These are based on creating a new consumer that takes the older consumer and the filter predicate or mapping function. In this way one can implement a pipeline on the input stream, or perhaps diverging pipelines on each reduction function in a multiple reducer scenario. Since the init and merge functions operate in accumulator space, which remains unchanged, one can develop up increasingly sophisticated reduction functions and still perform a parallelized reduction. Naturally, everything is composed in reverse (push instead of pull), which is the reason that comp works in reverse when working with transducers.

In fact, given that the covers are pulled back on composing reduction functions, the definition of the single argument clojure.core/filter becomes more clear:

(defn filter
  "Returns a lazy sequence of the items in coll for which
  (pred item) returns logical true. pred must be free of side-effects.
  Returns a transducer when no collection is provided."
  {:added "1.0"
   :static true}
  ([pred]
    (fn [rf]
      (fn
        ([] (rf))
        ([result] (rf result))
        ([result input]
           (if (pred input)
             (rf result input)
             result)))))))

It returns a function that, when given a reduction function, returns a new reduction function that when called in the two argument form is identical to the result above (although expressed in pure Clojure as opposed to Java).

Starting with the concept that a reduction begins at the collection, flows downward through the pipeline and bottoms out at the reducer then the lazy-noncaching namespace and Java streams implement parallelization flowing from the container downward. Separately consumer chains and transducers implement the pipeline flowing up from the reducer itself. Thus building the pipeline either downward from the source or upward from the final reduction produces subtly different properties. Regardless, every system must disable parallelization where it will cause an incorrect answer (to ensure correctness)

  • such as in a stateful transducer.

Broadly speaking, however, it can be faster to enable full parallelization and filter invalid results than it is to force an early serialization our problem and thus lose lots of our parallelization potential. When concerned with performance, attempt to move transformations as much as possible into a parallelizable domain.

For the take-n use case specifically mentioned above and potentially for others we can parallelize the reduction and do the take-n both in the parallelized phase and in the merge phase assuming we are using an ordered parallelization, so that doesn't itself necessarily force a serialized reduction but there are of course transformations and reductions that do. There are intermediate points however that are perhaps somewhat wasteful in terms of cpu load but do allow for more parallelization - a tradeoff that is sometimes worth it. Generically speaking we can visualize this sort of a tradeoff as triangle of three points where one point is data locality, one point parallelism, and one point redundancy. Specifically if we are willing to trade some cpu efficiency for some redundancy, for instance, then we often get more parallelization. Likewise if we are willing to save/load data from 'far' away from the CPU, then we can cut down on redundancy but at the cost of locality. For more on this line of thinking please take a moment and read at least some of Jonathan Ragan-Kelly's excellent PhD thesis - a better explanation of the above line of reasoning begins on page 20.

Primitive Typed Serial Reductions

This comes last for a good reason :-) - it doesn't make a huge difference in performance but it should be noted allowing objects to implemented typed reductions:

default Object doubleReduction(IFn.ODO op, Object init);
default Object longReduction(IFn.OLO op, Object init)

where the next incoming value is a primitive object but the accumulator is still a generic object allows us to use things like DoubleConsumers and LongConsumers and avoid boxing the stream. Furthermore if the aforementioned map and filter primitives are careful about their rfn composition we can maintain a completely primitive typed pipeline through an entire processing chain.

One Final Note About Performance

Collapsing reductions brings the source iteration pathway closer to the final reduction pathway in terms of machine stack space which allows HotSpot to optimize the entire chain more readily. Regardless of how good HotSpot gets, however, parallelizing will nearly always result in a larger win but both work together to enable peak performance on the JVM given arbitrary partially typed compositions of sequences and reductions.

When increasing the data size yet again, one can of course use the same design to distribute the computations to different machines. As some people have figured out, however, simply implementing the transformations you need efficiently reduces or completely eliminates the need to distribute computation in the first place leading to a simpler, easier to test and more robust system.

Ideally we can make achieving great performance for various algorithms clear and easy and thus avoid myriad of issues regarding distributing computing in the first place.

  • The first rule of distributed systems is to avoid distributing your computation in the first place - 1, 2.
  • The first law of distributed objects is to avoid using distributed objects. 3.
================================================ FILE: docs/css/default.css ================================================ @import url('https://fonts.googleapis.com/css?family=PT+Sans'); body { font-family: 'PT Sans', Helvetica, sans-serif; font-size: 14px; } a { color: #337ab7; text-decoration: none; } a:hover { color: #30426a; text-decoration: underline; } pre, code { font-family: Monaco, DejaVu Sans Mono, Consolas, monospace; font-size: 9pt; margin: 15px 0; } h1 { font-weight: normal; font-size: 29px; margin: 10px 0 2px 0; padding: 0; } h2 { font-weight: normal; font-size: 25px; } h3 > a:hover { text-decoration: none; } .document h1, .namespace-index h1 { font-size: 32px; margin-top: 12px; } #header, #content, .sidebar { position: fixed; } #header { top: 0; left: 0; right: 0; height: 22px; color: #f5f5f5; padding: 5px 7px; } #content { top: 32px; right: 0; bottom: 0; overflow: auto; background: #fff; color: #333; padding: 0 18px; } .sidebar { position: fixed; top: 32px; bottom: 0; overflow: auto; } .sidebar.primary { background: #30426a; border-right: solid 1px #cccccc; left: 0; width: 250px; color: white; font-size: 110%; } .sidebar.secondary { background: #f2f2f2; border-right: solid 1px #d7d7d7; left: 251px; width: 200px; font-size: 110%; } #content.namespace-index, #content.document { left: 251px; } #content.namespace-docs { left: 452px; } #content.document { padding-bottom: 10%; } #header { background: #2d3e63; box-shadow: 0 0 8px rgba(0, 0, 0, 0.4); z-index: 100; } #header h1 { margin: 0; padding: 0; font-size: 18px; font-weight: lighter; text-shadow: -1px -1px 0px #333; } #header h1 .project-version { font-weight: normal; } .project-version { padding-left: 0.15em; } #header a, .sidebar a { display: block; text-decoration: none; } #header a { color: #f5f5f5; } .sidebar.primary, .sidebar.primary a { color: #b2bfdc; } .sidebar.primary a:hover { color: white; } .sidebar.secondary, .sidebar.secondary a { color: #738bc0; } .sidebar.secondary a:hover { color: #2d3e63; } #header h2 { float: right; font-size: 9pt; font-weight: normal; margin: 4px 3px; padding: 0; color: #bbb; } #header h2 a { display: inline; } .sidebar h3 { margin: 0; padding: 10px 13px 0 13px; font-size: 19px; font-weight: lighter; } .sidebar.primary h3.no-link { text-transform: uppercase; font-size: 12px; color: #738bc0; } .sidebar.secondary h3 a { text-transform: uppercase; font-size: 12px; color: #2d3e63; } .sidebar ul { padding: 7px 0 6px 0; margin: 0; } .sidebar ul.index-link { padding-bottom: 4px; } .sidebar li { display: block; vertical-align: middle; } .sidebar li a, .sidebar li .no-link { border-left: 3px solid transparent; padding: 0 10px; white-space: nowrap; } .sidebar li .inner { display: inline-block; padding-top: 7px; height: 24px; } .sidebar li a, .sidebar li .tree { height: 31px; } .depth-1 .inner { padding-left: 2px; } .depth-2 .inner { padding-left: 6px; } .depth-3 .inner { padding-left: 20px; } .depth-4 .inner { padding-left: 34px; } .depth-5 .inner { padding-left: 48px; } .depth-6 .inner { padding-left: 62px; } .sidebar li .tree { display: block; float: left; position: relative; top: -10px; margin: 0 4px 0 0; padding: 0; } .sidebar li.depth-1 .tree { display: none; } .sidebar li .tree .top, .sidebar li .tree .bottom { display: block; margin: 0; padding: 0; width: 7px; } .sidebar li .tree .top { border-left: 1px solid #aaa; border-bottom: 1px solid #aaa; height: 19px; } .sidebar li .tree .bottom { height: 22px; } .sidebar li.branch .tree .bottom { border-left: 1px solid #aaa; } .sidebar.primary li.current a { border-left: 3px solid #e99d1a; color: white; } .sidebar.secondary li.current a { border-left: 3px solid #2d3e63; color: #33a; } .namespace-index h2 { margin: 30px 0 0 0; } .namespace-index h3 { font-size: 16px; font-weight: bold; margin-bottom: 0; letter-spacing: 0.05em; border-bottom: solid 1px #ddd; max-width: 680px; background-color: #fafafa; padding: 0.5em; } .namespace-index .topics { padding-left: 30px; margin: 11px 0 0 0; } .namespace-index .topics li { padding: 5px 0; } .namespace-docs h3 { font-size: 18px; font-weight: bold; } .public h3 { margin: 0; float: left; } .usage { clear: both; } .public { margin: 0; border-top: 1px solid #e0e0e0; padding-top: 14px; padding-bottom: 6px; } .public:last-child { margin-bottom: 20%; } .members .public:last-child { margin-bottom: 0; } .members { margin: 15px 0; } .members h4 { color: #555; font-weight: normal; font-variant: small-caps; margin: 0 0 5px 0; } .members .inner { padding-top: 5px; padding-left: 12px; margin-top: 2px; margin-left: 7px; border-left: 1px solid #bbb; } #content .members .inner h3 { font-size: 12pt; } .members .public { border-top: none; margin-top: 0; padding-top: 6px; padding-bottom: 0; } .members .public:first-child { padding-top: 0; } h4.type, h4.dynamic, h4.added, h4.deprecated { float: left; margin: 3px 10px 15px 0; font-size: 15px; font-weight: bold; font-variant: small-caps; } .public h4.type, .public h4.dynamic, .public h4.added, .public h4.deprecated { font-size: 13px; font-weight: bold; margin: 3px 0 0 10px; } .members h4.type, .members h4.added, .members h4.deprecated { margin-top: 1px; } h4.type { color: #717171; } h4.dynamic { color: #9933aa; } h4.added { color: #508820; } h4.deprecated { color: #880000; } .namespace { margin-bottom: 30px; } .namespace:last-child { margin-bottom: 10%; } .index { padding: 0; font-size: 80%; margin: 15px 0; line-height: 1.6em; } .index * { display: inline; } .index p { padding-right: 3px; } .index li { padding-right: 5px; } .index ul { padding-left: 0; } .type-sig { clear: both; color: #088; } .type-sig pre { padding-top: 10px; margin: 0; } .usage code { display: block; color: #008; margin: 2px 0; } .usage code:first-child { padding-top: 10px; } p { margin: 15px 0; } .public p:first-child, .public pre.plaintext { margin-top: 12px; } .doc { margin: 0 0 26px 0; clear: both; } .public .doc { margin: 0; } .namespace-index { font-size: 120%; } .namespace-index .doc { margin-bottom: 20px; } .namespace-index .namespace .doc { margin-bottom: 10px; } .markdown p, .markdown li, .markdown dt, .markdown dd, .markdown td { line-height: 1.6em; } .markdown h2 { font-weight: normal; font-size: 25px; } #content .markdown h3 { font-size: 20px; } .markdown h4 { font-size: 15px; } .doc, .public, .namespace .index { max-width: 680px; overflow-x: visible; } .markdown pre > code { display: block; padding: 10px; } .markdown pre > code, .src-link a { border: 1px solid #e4e4e4; border-radius: 2px; } .src-link a { background: #f6f6f6; } .markdown code:not(.hljs) { color: #c7254e; background-color: #f9f2f4; border-radius: 4px; font-size: 90%; padding: 2px 4px; } pre.deps { display: inline-block; margin: 0 10px; border: 1px solid #e4e4e4; border-radius: 2px; padding: 10px; background-color: #f6f6f6; } .markdown hr { border-style: solid; border-top: none; color: #ccc; } .doc ul, .doc ol { padding-left: 30px; } .doc table { border-collapse: collapse; margin: 0 10px; } .doc table td, .doc table th { border: 1px solid #dddddd; padding: 4px 6px; } .doc table th { background: #f2f2f2; } .doc dl { margin: 0 10px 20px 10px; } .doc dl dt { font-weight: bold; margin: 0; padding: 3px 0; border-bottom: 1px solid #ddd; } .doc dl dd { padding: 5px 0; margin: 0 0 5px 10px; } .doc abbr { border-bottom: 1px dotted #333; font-variant: none; cursor: help; } .src-link { margin-bottom: 15px; } .src-link a { font-size: 70%; padding: 1px 4px; text-decoration: none; color: #5555bb; background-color: #f6f6f6; } blockquote { opacity: 0.6; border-left: solid 2px #ddd; margin-left: 0; padding-left: 1em; } /* Responsiveness Theme */ @media (max-device-width: 480px) { .sidebar { display:none; } #content { position: relative; left: initial !important; top: 110px; padding: 0 1em; } #header { display: flex; flex-direction: column-reverse; height: 100px; } #header > h1 { font-size: 52px; } #header h2 { float: none; font-size: 20px; } .namespace-index > h1 { display: none; } .public, .doc, .namespace > .index, .namespace > .doc, .namespace > h3 { max-width: initial; } .doc { text-align: justify; } .public { padding-top: 2em; padding-bottom: 2em; } .public > h3 { font-size: 300%; } .public > h4.type, .public > h4.added, .public > h4.deprecated { font-size: 150%; margin-top: 1em; } pre > code { font-size: 200%; } } ================================================ FILE: docs/ham-fisted.api.html ================================================ ham-fisted.api documentation

ham-fisted.api

Fast mutable and immutable associative data structures based on bitmap trie hashmaps. Mutable pathways implement the java.util.Map or Set interfaces including in-place update features such as compute or computeIfPresent.

Mutable maps or sets can be turned into their immutable counterparts via the Clojure persistent! call. This allows working in a mutable space for convenience and performance then switching to an immutable pathway when necessary. Note: after persistent! one should never backdoor mutate map or set again as this will break the contract of immutability. Immutable data structures also support conversion to transient via transient.

Map keysets (.keySet) are full PersistentHashSets of keys.

Maps and sets support metadata but setting the metadata on mutable objects returns a new mutable object that shares the backing store leading to possible issues. Metadata is transferred to the persistent versions of the mutable/transient objects upon persistent!.

Very fast versions of union, difference and intersection are provided for maps and sets with the map version of union and difference requiring an extra argument, a java.util.BiFunction or an IFn taking 2 arguments to merge the left and right sides into the final map. These implementations of union, difference, and intersection are the fastest implementation of these operations we know of on the JVM.

Additionally a fast value update pathway is provided, enabling quickly updating all the values in a given map. Additionally, a new map primitive

  • mapmap - allows transforming a given map into a new map quickly by mapping across all the entries.

Unlike the standard Java objects, mutation-via-iterator is not supported.

->collection

(->collection item)

Ensure item is an implementation of java.util.Collection.

->random-access

(->random-access item)

Ensure item is derived from java.util.List and java.util.RandomAccess and thus supports constant time random addressing.

->reducible

(->reducible item)

Ensure item either implements IReduceInit or java.util.Collection. For arrays this will return an object that has a much more efficient reduction pathway than the base Clojure reducer.

add-all!

(add-all! l1 l2)

Add all items from l2 to l1. l1 is expected to be a java.util.List implementation. Returns l1.

add-constant!

(add-constant! l idx v)(add-constant! l idx count v)

apply-concat

(apply-concat args)

Faster lazy noncaching version of (apply concat)

apply-concatv

(apply-concatv data)

argsort

(argsort comp coll)(argsort coll)

Sort a collection of data returning an array of indexes. The collection must be random access and the return value is an integer array of indexes which will read the input data in sorted order. Faster implementations are provided when the collection is an integer, long, or double array. See also reindex.

array-list

(array-list data)(array-list)

Create an implementation of java.util.ArrayList.

assoc!

(assoc! obj k v)

assoc! that works on transient collections, implementations of java.util.Map and RandomAccess java.util.List implementations. Be sure to keep track of return value as some implementations return a different return value than the first argument.

boolean-array

(boolean-array)(boolean-array data)

boolean-array-list

(boolean-array-list)(boolean-array-list data)

byte-array

(byte-array)(byte-array data)

byte-array-list

(byte-array-list)(byte-array-list data)

char-array

(char-array)(char-array data)

char-array-list

(char-array-list)(char-array-list data)

clear!

(clear! map-or-coll)

Mutably clear a map, set, list or implementation of java.util.Collection.

clear-memoized-fn!

(clear-memoized-fn! memoized-fn)

Clear a memoized function backing store.

complement

(complement f)

Like clojure core complement but avoids var lookup on 'not'

concata

(concata)(concata v1)(concata v1 v2)(concata v1 v2 & args)

non-lazily concat a set of items returning an object array. This always returns an object array an may return an empty array whereas concat may return nil.

concatv

(concatv)(concatv v1)(concatv v1 v2)(concatv v1 v2 & args)

non-lazily concat a set of items returning a persistent vector.

cond

macro

(cond & clauses)

Similar to core/cond except it supports true else clauses as opposed to always returning nil on else. The last clause of an odd number of clauses will be used as the else branch similar to case. Additional if the last predicate is the constant 'true' or ':else' it will be used as the else clause. If no else is provided then returns nil same as core/cond.

Important! - in order to get correct type hinting on both branches of the if they both must return an explicit long or double:

  • Sill Boxed
  (defn memory-size
    ^long [a]
    (cond
      (instance? Long a) 24
      :else (.length ^String a) ;;string.length returns integer not long
      ))

(defn memory-size
    ^long [a]
    (cond
      (instance? Long a) 24
      :else (alength ^longs a)
      ))
  • Correctly Unboxed
(defn memory-size
    ^long [a]
    (cond
      (instance? Long a) 24
      :else (long (.length ^String a))
      ))

(defn memory-size
    ^long [a]
    (cond
      (instance? Long a) 24
      :else (long (alength ^longs a))
      ))

conj!

(conj! obj val)

conj! that works on transient collections, implementations of java.util.Set and RandomAccess java.util.List implementations. Be sure to keep track of return value as some implementations return a different return value than the first argument.

constant-count

(constant-count data)

Constant time count. Returns nil if input doesn't have a constant time count.

constant-countable?

(constant-countable? data)

Return true if data has a constant time count.

constantly

(constantly x)

count

(count m)

hamf protocol extensible count

custom-counted-ireduce

macro

(custom-counted-ireduce n-elems rfn acc & code)

Custom implementation of IReduceInit and nothing else. This can be the most efficient way to pass data to other interfaces. Also see custom-ireduce if the object does not need to be counted and see reduced-> for implementation helper.

custom-ireduce

macro

(custom-ireduce rfn acc & code)

Custom implementation of IReduceInit and nothing else. This can be the most efficient way to pass data to other interfaces. Also see custom-counted-ireduce if the object should also implement ICounted. See reduced-> for implementation helper.

darange

(darange end)(darange start end)(darange start end step)

Return a double array holding the values of the range. Use wrap-array to get an implementation of java.util.List that supports the normal Clojure interfaces.

dbl-ary-cls

difference

(difference map1 map2)

Take the difference of two maps (or sets) returning a new map. Return value is a map1 (or set1) without the keys present in map2.

dnth

macro

(dnth obj idx)

nth operation returning a primitive double. Efficient when obj is a double array.

double-array

macro

(double-array)(double-array data)

double-array-list

(double-array-list)(double-array-list cap-or-data)

An array list that is as fast as java.util.ArrayList for add,get, etc but includes many accelerated operations such as fill and an accelerated addAll when the src data is an array list.

drop

(drop n coll)

Drop the first N items of the collection. If item is random access, the return value is random-access.

drop-last

(drop-last n)(drop-last n coll)

Drop the last N values from a collection. IF the input is random access, the result will be random access.

drop-min

(drop-min n comp values)(drop-min n values)

Drop the min n values of a collection. This is not an order-preserving operation.

dsummary

(dsummary data)

Summary statistics {:mean :max :min :n-elems :sum} in double space

dvec

macro

(dvec)(dvec data)

Create a persistent-vector-compatible list backed by a double array.

empty-map

Constant persistent empty map

empty-set

Constant persistent empty set

empty-vec

Constant persistent empty vec

empty?

(empty? coll)

evict-memoized-call

(evict-memoized-call memo-fn fn-args)

filterv

(filterv pred coll)

Filter a collection into a vector.

first

(first coll)

Get the first item of a collection.

float-array

macro

(float-array)(float-array data)

float-array-list

(float-array-list)(float-array-list data)

fnth

macro

(fnth obj idx)

nth operation returning a primitive float. Efficient when obj is a float array.

freq-reducer

(freq-reducer options)(freq-reducer)

Return a hamf parallel reducer that performs a frequencies operation.

frequencies

(frequencies coll)(frequencies options coll)

Faster implementation of clojure.core/frequencies.

fvec

macro

(fvec)(fvec data)

Create a persistent-vector-compatible list backed by a float array.

group-by

(group-by f options coll)(group-by f coll)

Group items in collection by the grouping function f. Returns persistent map of keys to persistent vectors.

Options are same as group-by-reduce but this reductions defaults to an ordered reduction.

group-by-consumer

(group-by-consumer key-fn reducer coll)(group-by-consumer key-fn reducer options coll)

Perform a group-by-reduce passing in a reducer. Same options as group-by-reduce - This uses a slightly different pathway - computeIfAbsent - in order to preserve order. In this case the return value of the reduce fn is ignored. This allows things like the linked hash map to preserve initial order of keys. It map also be slightly more efficient because the map itself does not need to check the return value of rfn - something that the .compute primitive does need to do.

Options:

  • :skip-finalize? - skip finalization step.

group-by-reduce

(group-by-reduce key-fn init-val-fn rfn merge-fn options coll)(group-by-reduce key-fn init-val-fn rfn merge-fn coll)

Group by key. Apply the reduce-fn with the new value an the return of init-val-fn. Merged maps due to multithreading will be merged with merge-fn in a similar way of preduce.

This type of reduction can be both faster and more importantly use less memory than a reduction of the forms:

  (->> group-by map into)
  ;; or
  (->> group-by mapmap)

Options (which are passed to preduce):

  • :map-fn Function which takes no arguments and must return an instance of java.util.Map that supports computeIfAbsent. Some examples:
    • (constantly (java.util.concurrent.ConcurrentHashMap. ...)) Very fast update especially in the case where the keyspace is large.
    • mut-map - Fast merge, fast update, in-place immutable conversion via persistent!.
    • java-hashmap - fast merge, fast update, just a simple java.util.HashMap-based reduction.
    • #(LinkedHashMap.) - When used with options {:ordered? true} the result keys will be in order and the result values will be reduced in order.

group-by-reducer

(group-by-reducer key-fn reducer coll)(group-by-reducer key-fn reducer options coll)

Perform a group-by-reduce passing in a reducer. Same options as group-by-reduce.

Options:

  • :skip-finalize? - skip finalization step.

hash-map

(hash-map)(hash-map a b)(hash-map a b c d)(hash-map a b c d e f)(hash-map a b c d e f g h)(hash-map a b c d e f g h i j)(hash-map a b c d e f g h i j k l)(hash-map a b c d e f g h i j k l m n)(hash-map a b c d e f g h i j k l m n o p)(hash-map a b c d e f g h i j k l m n o p & args)

Drop-in replacement to Clojure's hash-map function.

iarange

(iarange end)(iarange start end)(iarange start end step)

Return an integer array holding the values of the range. Use ->collection to get a list implementation wrapping for generic access.

immut-list

(immut-list)(immut-list data)

Create a persistent list. Object arrays will be treated as if this new object owns them.

immut-map

(immut-map)(immut-map data)(immut-map options data)

Create an immutable map. This object supports conversion to a transient map via Clojure's transient function. Duplicate keys are treated as if by assoc.

If data is an object array it is treated as a flat key-value list which is distinctly different than how conj! treats object arrays. You have been warned.

If you know you will have consistently more key/val pairs than 8 you should just use (persistent! (mut-map data)) as that avoids the transition from an arraymap to a persistent hashmap.

Examples:

ham-fisted.api> (immut-map (obj-ary :a 1 :b 2 :c 3 :d 4))
{:a 1, :b 2, :c 3, :d 4}
ham-fisted.api> (type *1)
ham_fisted.PersistentArrayMap
ham-fisted.api> (immut-map (obj-ary :a 1 :b 2 :c 3 :d 4 :e 5))
{:d 4, :b 2, :c 3, :a 1, :e 5}
ham-fisted.api> (type *1)
ham_fisted.PersistentHashMap
ham-fisted.api> (immut-map [[:a 1][:b 2][:c 3][:d 4][:e 5]])
{:d 4, :b 2, :c 3, :a 1, :e 5}
ham-fisted.api> (type *1)
ham_fisted.PersistentHashMap

immut-set

(immut-set)(immut-set data)(immut-set options data)

Create an immutable hashset based on a hash table. This object supports conversion to transients via transient.

Options:

  • :hash-provider - An implementation of BitmapTrieCommon$HashProvider. Defaults to the default-hash-provider.

in-fork-join-task?

(in-fork-join-task?)

True if you are currently running in a fork-join task

inc-consumer

(inc-consumer)(inc-consumer init-value)

Return a consumer that simply increments a long. See java/ham_fisted/Consumers.java for definition.

inc-consumer-reducer

A hamf reducer that works with inc-consumers

int-array

macro

(int-array)(int-array data)

int-array-list

(int-array-list)(int-array-list cap-or-data)

An array list that is as fast as java.util.ArrayList for add,get, etc but includes many accelerated operations such as fill and an accelerated addAll when the src data is an array list.

intersect-sets

(intersect-sets sets)

Given a sequence of sets, efficiently perform the intersection of them. This algorithm is usually faster and has a more stable runtime than (reduce clojure.set/intersection sets) which degrades depending on the order of the sets and the pairwise intersection of the initial sets.

intersection

(intersection s1 s2)

Intersect the keyspace of set1 and set2 returning a new set. Also works if s1 is a map and s2 is a set - the map is trimmed to the intersecting keyspace of s1 and s2.

inth

macro

(inth obj idx)

nth operation returning a primitive int. Efficient when obj is an int array.

into

(into container data)(into container xform data)

Like clojure.core/into, but also designed to handle editable collections, transients, and base java.util.Map, List and Set containers.

into-array

(into-array aseq)(into-array ary-type aseq)(into-array ary-type mapfn aseq)

Faster version of clojure.core/into-array.

ivec

macro

(ivec)(ivec data)

Create a persistent-vector-compatible list backed by an int array.

java-concurrent-hashmap

(java-concurrent-hashmap)(java-concurrent-hashmap data)

Create a java concurrent hashmap which is still the fastest possible way to solve a few concurrent problems.

java-hashmap

(java-hashmap)(java-hashmap data)(java-hashmap xform data)(java-hashmap xform options data)

Create a java.util.HashMap. Duplicate keys are treated as if map was created by assoc.

java-hashset

(java-hashset)(java-hashset data)

Create a java hashset which is still the fastest possible way to solve a few problems.

java-linked-hashmap

(java-linked-hashmap)(java-linked-hashmap data)

Linked hash maps perform identically or very nearly so to java.util.HashMaps but they retain the order of insertion and modification.

keys

(keys m)

Return the keys of a map. This version allows parallel reduction operations on the returned sequence.

larange

(larange end)(larange start end)(larange start end step)

Return a long array holding values of the range. Use ->collection get a list implementation for generic access.

last

(last coll)

Get the last item in the collection. Constant time for random access lists.

linear-merge-iterable

(linear-merge-iterable cmp pred iterables)(linear-merge-iterable cmp iterables)(linear-merge-iterable iterables)

Create an N-way merge iterable using cmp to order the merge of provided iterables. If a predicate pred is provided the iterable itself will filter out values for which the pred returns false. In this mode it is possible for the iterator to return null if the last value is filtered out -- the hasNext method doesn't check if the next value passes the predicate.

lines

(lines fname)

Return a closeable iterable that produces an iterator that simply produces lines of the reader. Iterator does not cache more than 1 line so there is no possibility of holding onto head normal nor chunked seq-ing.

Example:

(with-open [ld (hamf/lines fname)]
  (->> ld
       (hamf/pmap (fn [line] ...))
       (reduce ...)))

linked-hashmap

(linked-hashmap)(linked-hashmap data)

Linked hash map using clojure's equiv pathways. At this time the node link order reflects insertion order. Modification and access do not affect the node link order.

lnth

macro

(lnth obj idx)

nth operation returning a primitive long. Efficient when obj is a long array.

long-array

macro

(long-array)(long-array data)

long-array-list

(long-array-list)(long-array-list cap-or-data)

An array list that is as fast as java.util.ArrayList for add,get, etc but includes many accelerated operations such as fill and an accelerated addAll when the src data is an array list.

lsum

(lsum data)

Simple summation that returns a long integer.

lsummary

(lsummary data)

Summary statistics {:mean :max :min :n-elems :sum} in long space

lvec

macro

(lvec)(lvec data)

Create a persistent-vector-compatible list backed by a long array.

make-map-entry

macro

(make-map-entry k-code v-code)

Create a dynamic implementation of clojure's IMapEntry class.

map-intersection

(map-intersection bfn map1 map2)

Intersect the keyspace of map1 and map2 returning a new map. Each value is the result of bfn applied to the map1-value and map2-value, respectively. See documentation for map-union.

Clojure's merge functionality can be duplicate via:

(map-intersection (fn [lhs rhs] rhs) map1 map2)

map-union

(map-union bfn map1 map2)

Take the union of two maps returning a new map. bfn is a function that takes 2 arguments, map1-val and map2-val and returns a new value. Has fallback if map1 and map2 aren't backed by bitmap tries.

  • bfn - A function taking two arguments and returning one. + is a fine choice.
  • map1 - the lhs of the union.
  • map2 - the rhs of the union.

Returns a persistent map if input is a persistent map else if map1 is a mutable map map1 is returned with overlapping entries merged. In this way you can pass in a normal java hashmap, a linked java hashmap, or a persistent map and get back a result that matches the input.

If map1 and map2 are the same returns map1.

map-union-java-hashmap

(map-union-java-hashmap bfn lhs rhs)

Take the union of two maps returning a new map. See documentation for map-union. Returns a java.util.HashMap.

mapmap

(mapmap map-fn src-map)

Clojure's missing piece. Map over the data in src-map, which must be a map or sequence of pairs, using map-fn. map-fn must return either a new key-value pair or nil. Then, remove nil pairs, and return a new map. If map-fn returns more than one pair with the same key later pair will overwrite the earlier pair.

Logically the same as:

(into {} (comp (map map-fn) (remove nil?)) src-map)

mapv

(mapv map-fn coll)(mapv map-fn c1 c2)(mapv map-fn c1 c2 c3)(mapv map-fn c1 c2 c3 & args)

Produce a persistent vector from a collection.

mean

(mean coll)(mean options coll)

Return the mean of the collection. Returns double/NaN for empty collections. See options for sum.

memoize

(memoize memo-fn)(memoize memo-fn {:keys [write-ttl-ms access-ttl-ms soft-values? weak-values? max-size record-stats? eviction-fn], :as options})

Efficient thread-safe version of clojure.core/memoize.

Also see clear-memoized-fn! evict-memoized-call and memoize-cache-as-map to mutably clear the backing store, manually evict a value, and get a java.util.Map view of the cache backing store.

ham-fisted.api> (def m (memoize (fn [& args] (println "fn called - " args) args)
                                {:write-ttl-ms 1000 :eviction-fn (fn [args rv cause]
                                                                   (println "evicted - " args rv cause))}))
#'ham-fisted.api/m
ham-fisted.api> (m 3)
fn called -  (3)
(3)
ham-fisted.api> (m 4)
fn called -  (4)
(4)evicted -  [3] (3) :expired
ham-fisted.api> (dotimes [idx 4] (do (m 3) (evict-memoized-call m [3])))
fn called -  (3)
fn called -  (3)
fn called -  (3)
fn called -  (3)
nil
ham-fisted.api> (dotimes [idx 4] (do (m 3) #_(evict-memoized-call m [3])))
fn called -  (3)
nil

Options:

  • :write-ttl-ms - Time that values should remain in the cache after write in milliseconds.
  • :access-ttl-ms - Time that values should remain in the cache after access in milliseconds.
  • :soft-values? - When true, the cache will store SoftReferences to the data.
  • :weak-values? - When true, the cache will store WeakReferences to the data.
  • :max-size - When set, the cache will behave like an LRU cache.
  • :record-stats? - When true, the LoadingCache will record access statistics. You can get those via the undocumented function memo-stats.
  • :eviction-fn - Function that receives 3 arguments, [args v cause], when a value is evicted. Causes the keywords:collected :expired :explicit :replaced and :size`. See caffeine documentation for cause definitions.

memoize-cache-as-map

(memoize-cache-as-map memoized-fn)

Return the memoize backing store as an implementation of java.util.Map.

merge

(merge)(merge m1)(merge m1 m2)(merge m1 m2 & args)

Merge 2 maps with the rhs values winning any intersecting keys. Uses map-union with BitmapTrieCommon/rhsWins.

Returns a new persistent map.

merge-iterator

(merge-iterator cmp iterables)

Create an N-way priority queue iterable using cmp to order the merge of provided iterables. If a predicate pred is provided the iterable itself will filter out values for which the pred returns false. In this mode it is possible for the iterator to return null if the last value is filtered out -- the hasNext method doesn't check if the next value passes the predicate.

merge-with

(merge-with f)(merge-with f m1)(merge-with f m1 m2)(merge-with f m1 m2 & args)

Merge (union) any number of maps using f as the merge operator. f gets passed two arguments, lhs-val and rhs-val and must return a new value.

Returns a new persistent map.

mmax-idx

(mmax-idx f data)

Like mmin-key but returns the max index. F should be a function from obj->long.

mmax-key

(mmax-key f data)

Faster and nil-safe version of #(apply max-key %1 %2)

mmin-idx

(mmin-idx f data)

Like mmin-key but returns the min index. F should be a function from obj->long.

mmin-key

(mmin-key f data)

Faster and nil-safe version of #(apply min-key %1 %2)

mode

(mode data)

Return the most common occurance in the data.

mut-hashtable-map

(mut-hashtable-map)(mut-hashtable-map data)(mut-hashtable-map xform data)(mut-hashtable-map xform options data)

Create a mutable implementation of java.util.Map. This object efficiently implements ITransient map so you can use assoc! and persistent! on it but you can additionally use the various members of the java.util.Map interface such as put, compute, computeIfAbsent, replaceAll and merge.

If data is an object array it is treated as a flat key-value list which is distinctly different than how conj! treats object arrays. You have been warned.

mut-list

(mut-list)(mut-list data)

Create a mutable java list that is in-place convertible to a persistent list

mut-long-hashtable-map

(mut-long-hashtable-map)(mut-long-hashtable-map data)(mut-long-hashtable-map xform data)(mut-long-hashtable-map xform options data)

Create a mutable implementation of java.util.Map. This object efficiently implements ITransient map so you can use assoc! and persistent! on it but you can additionally use the various members of the java.util.Map interface such as put, compute, computeIfAbsent, replaceAll and merge.

If data is an object array it is treated as a flat key-value list which is distinctly different than how conj! treats object arrays. You have been warned.

mut-long-map

(mut-long-map)(mut-long-map data)(mut-long-map xform data)(mut-long-map xform options data)

Create a mutable implementation of java.util.Map specialized to long keys. This object efficiently implements ITransient map so you can use assoc! and persistent! on it but you can additionally use the various members of the java.util.Map interface such as put, compute, computeIfAbsent, replaceAll and merge. Attempting to store any non-numeric value will result in an exception.

If data is an object array it is treated as a flat key-value list which is distinctly different than how conj! treats object arrays. You have been warned.

mut-map

(mut-map)(mut-map data)(mut-map xform data)(mut-map xform options data)

Create a mutable implementation of java.util.Map. This object efficiently implements ITransient map so you can use assoc! and persistent! on it but you can additionally use the various members of the java.util.Map interface such as put, compute, computeIfAbsent, replaceAll and merge.

If data is an object array it is treated as a flat key-value list which is distinctly different than how conj! treats object arrays. You have been warned.

mut-map-rf

(mut-map-rf cons-fn)(mut-map-rf cons-fn finalize-fn)

mut-map-union!

(mut-map-union! merge-bifn l r)

Very fast union that may simply update lhs and return it. Both lhs and rhs must be mutable maps. See docs for map-union.

mut-set

(mut-set)(mut-set data)(mut-set options data)

Create a mutable hashset based on the hashtable. You can create a persistent hashset via the clojure persistent! call.

Options:

  • :hash-provider - An implementation of BitmapTrieCommon$HashProvider. Defaults to the default-hash-provider.

mutable-map?

(mutable-map? m)

not

(not a)

Returns boolean opposite of passed in value

obj-ary

(obj-ary)(obj-ary v0)(obj-ary v0 v1)(obj-ary v0 v1 v2)(obj-ary v0 v1 v2 v3)(obj-ary v0 v1 v2 v3 v4)(obj-ary v0 v1 v2 v3 v4 v5)(obj-ary v0 v1 v2 v3 v4 v5 v6)(obj-ary v0 v1 v2 v3 v4 v5 v6 v7)(obj-ary v0 v1 v2 v3 v4 v5 v6 v7 v8)(obj-ary v0 v1 v2 v3 v4 v5 v6 v7 v8 v9)(obj-ary v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10)(obj-ary v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11)(obj-ary v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12)(obj-ary v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13)(obj-ary v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14)(obj-ary v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15)

As quickly as possible, produce an object array from these inputs. Very fast for arities <= 16.

object-array

(object-array item)

Faster version of object-array for java collections and strings.

object-array-list

(object-array-list)(object-array-list cap-or-data)

An array list that is as fast as java.util.ArrayList for add,get, etc but includes many accelerated operations such as fill and an accelerated addAll when the src data is an object array based list.

ovec

macro

(ovec)(ovec data)

Return an immutable persistent vector like object backed by a single object array.

persistent!

(persistent! v)

If object is an ITransientCollection, call clojure.core/persistent!. Else return collection.

pgroups

(pgroups n-elems body-fn options)(pgroups n-elems body-fn)

Run y index groups across n-elems. Y is common pool parallelism.

body-fn gets passed two longs, startidx and endidx.

Returns a sequence of the results of body-fn applied to each group of indexes.

Before using this primitive please see if ham-fisted.reduce/preduce will work.

You must wrap this in something that realizes the results if you need the parallelization to finish by a particular point in the program - (dorun (hamf/pgroups ...)).

Options:

  • :pgroup-min - when provided n-elems must be more than this value for the computation to be parallelized.
  • :batch-size - max batch size. Defaults to 64000.

pmap

(pmap map-fn & sequences)

pmap using the commonPool. This is useful for interacting with other primitives, namely pgroups which are also based on this pool. This is a change from Clojure's base pmap in that it uses the ForkJoinPool/commonPool for parallelism as opposed to the agent pool - this makes it compose with pgroups and dtype-next's parallelism system.

Before using this primitive please see if ham-fisted.reduce/preduce will work.

Is guaranteed to not trigger the need for shutdown-agents.

pmap-io

(pmap-io n-lookahead map-fn & sequences)

pmap for io bound tasks where you want to specify how far ahead to run - uses the clojure.lang.Agent/soloExecutor for execution of presumably io-bound operations.

pmap-opts

(pmap-opts opts map-fn & sequences)

pmap but takes an extra option map as the first argument. This is useful if you, for instance, want to control exactly the parallel options arguments such as :n-lookahead. See docs for ham-fisted.reduce/options->parallel-options.

range

(range)(range end)(range start end)(range start end step)

When given arguments returns a range that implements random access java list interfaces so nth, reverse and friends are efficient.

re-matches

(re-matches re s)

Much faster version of clojure.core/re-matches.

reduced->

macro

(reduced-> rfn acc & data)

Helper macro to implement reduce chains checking for if the accumulator is reduced before calling the next expression in data.

(defrecord YMC [year-month ^long count]
  clojure.lang.IReduceInit
  (reduce [this rfn init]
    (let [init (reduced-> rfn init
                   (clojure.lang.MapEntry/create :year-month year-month)
                   (clojure.lang.MapEntry/create :count count))]
      (if (and __extmap (not (reduced? init)))
        (reduce rfn init __extmap)
        init))))

reindex

(reindex coll indexes)

Permut coll by the given indexes. Result is random-access and the same length as the index collection. Indexes are expected to be in the range of [0->count(coll)).

repeat

(repeat v)(repeat n v)

When called with no arguments, produce an infinite sequence of v. When called with 2 arguments, produce a random access list that produces v at each index.

rest

(rest coll)

Version of rest that does uses subvec if collection is random access. This preserves the ability to reduce in parallel over the collection.

reverse

(reverse coll)

Reverse a collection or sequence. Constant time reverse is provided for any random access list.

short-array

(short-array)(short-array data)

short-array-list

(short-array-list)(short-array-list data)

shuffle

(shuffle coll)(shuffle coll opts)

shuffle values returning random access container. If you are calling this repeatedly on the same collection you should call ->random-access on the collection before you start as shuffle internally only works on random access collections.

Options:

  • :seed - If instance of java.util.Random, use this. If integer, use as seed. If not provided a new instance of java.util.Random is created.

sort

(sort coll)(sort comp coll)

Exact replica of clojure.core/sort but instead of wrapping the final object array in a seq which loses the fact the result is countable and random access. Faster implementations are provided when the input is an integer, long, or double array.

The default comparison is nan-last meaning null-last if the input is an undefined container and nan-last if the input is a double or float specific container.

sort-by

(sort-by keyfn coll)(sort-by keyfn comp coll)

Sort a collection by keyfn. Typehinting the return value of keyfn will somewhat increase the speed of the sort :-).

sorta

(sorta coll)(sorta comp coll)

Sort returning an object array.

splice

(splice v1 idx v2)

Splice v2 into v1 at idx. Returns a persistent vector.

subvec

(subvec m sidx eidx)(subvec m sidx)

More general version of subvec. Works for any java list implementation including persistent vectors and any array.

sum

(sum coll)(sum options coll)

Very stable high performance summation. Uses both threading and kahans compensated summation.

Options:

  • nan-strategy - defaults to :remove. Options are :keep, :remove and :exception.

sum-fast

(sum-fast coll)

Fast simple double summation. Does not do any nan checking or summation compensation.

sum-stable-nelems

(sum-stable-nelems coll)(sum-stable-nelems options coll)

Stable sum returning map of {:sum :n-elems}. See options for sum.

take

(take n)(take n coll)

Take the first N values from a collection. If the input is random access, the result will be random access.

take-last

(take-last n coll)

Take the last N values of the collection. If the input is random-access, the result will be random-access.

take-min

(take-min n comp values)(take-min n values)

Take the min n values of a collection. This is not an order-preserving operation.

transient

(transient v)

transient-map-rf

(transient-map-rf cons-fn)(transient-map-rf cons-fn finalize-fn)

union

(union s1 s2)

Union of two sets or two maps. When two maps are provided the right hand side wins in the case of an intersection - same as merge.

Result is either a set or a map, depending on if s1 is a set or map.

union-reduce-maps

(union-reduce-maps bfn maps)

Do an efficient union reduction across many maps using bfn to update values. If the first map is mutable the union is done mutably into the first map and it is returned.

update-vals

(update-vals data f)

update-values

(update-values map bfn)

Immutably (or mutably) update all values in the map returning a new map. bfn takes 2 arguments, k,v and returns a new v. Returning nil removes the key from the map. When passed a vector the keys are indexes and no nil-removal is done.

upgroups

(upgroups n-elems body-fn options)(upgroups n-elems body-fn)

Run y index groups across n-elems. Y is common pool parallelism.

body-fn gets passed two longs, startidx and endidx.

Returns a sequence of the results of body-fn applied to each group of indexes.

Before using this primitive please see if ham-fisted.reduce/preduce will work.

You must wrap this in something that realizes the results if you need the parallelization to finish by a particular point in the program - (dorun (hamf/upgroups ...)).

Options:

  • :pgroup-min - when provided n-elems must be more than this value for the computation to be parallelized.
  • :batch-size - max batch size. Defaults to 64000.

upmap

(upmap map-fn & sequences)

Unordered pmap using the commonPool. This is useful for interacting with other primitives, namely pgroups which are also based on this pool.

Before using this primitive please see if ham-fisted.reduce/preduce will work.

Like pmap this uses the commonPool so it composes with this api's pmap, pgroups, and dtype-next's parallelism primitives but it does not impose an ordering constraint on the results and thus may be significantly faster in some cases.

vals

(vals m)

Return the values of a map. This version allows parallel reduction operations on the returned sequence. Returned sequence is in same order as (keys m).

vec

(vec data)(vec)

Produce a persistent vector. Optimized pathways exist for object arrays and java List implementations.

vector

(vector)(vector a)(vector a b)(vector a b c)(vector a b c d)(vector a b c d e)(vector a b c d e f)(vector a b c d e f g)(vector a b c d e f g h)(vector a b c d e f g h i)(vector a b c d e f g h i j)(vector a b c d e f g h i j k)(vector a b c d e f g h i j k & args)

wrap-array

(wrap-array ary)

Wrap an array with an implementation of IMutList

wrap-array-growable

(wrap-array-growable ary ptr)(wrap-array-growable ary)

Wrap an array with an implementation of IMutList that supports add and addAllReducible. 'ptr is the numeric put ptr, defaults to the array length. Pass in zero for a preallocated but empty growable wrapper.

================================================ FILE: docs/ham-fisted.bloom-filter.html ================================================ ham-fisted.bloom-filter documentation

ham-fisted.bloom-filter

Simple fast bloom filter based on apache parquet BlockSplitBloomFilter.

add-uuids!

(add-uuids! bf val-seq)

bitset-size

(bitset-size fb)

Return the length of the byte array underlying this bitset

bloom-filter

(bloom-filter n f)

Create a bloom filter.

  • 'n' - Number of distinct values.
  • 'f' - Value from 1.0-0.0, defaults to 0.01. False positive rate.

bloom-filter->byte-array

(bloom-filter->byte-array bf)

byte-array->bloom-filter

(byte-array->bloom-filter data)

byte-array-cls

contains?

(contains? bf obj)

hash-obj

(hash-obj obj)

Hash an object. If integer - return integer else serialize->bytes and hash those

insert-hash!

(insert-hash! bf hc)

insert-obj

(insert-obj bf o)

make-long-hash-predicate

(make-long-hash-predicate bf)

make-obj-predicate

(make-obj-predicate bf)

make-uuid-hasher

(make-uuid-hasher)

make-uuid-pred

(make-uuid-pred bf)

serialize->bytes

(serialize->bytes o)

Serialize an object to a byte array

================================================ FILE: docs/ham-fisted.defprotocol.html ================================================ ham-fisted.defprotocol documentation

ham-fisted.defprotocol

Alternative protocol implementation.

Major features:

  • Allows subclasses to override only a subset of the methods and if the superclass has overridden the method then the superclasses implementation will be used.

  • Supports primitive typehints on function arguments and return values.

  • Much higher and more predictable multithreaded performance for protocol method invocation due to the fewer number of global variables that are read and written to for a single protocol method invocation. Does not write to global variables on a per-call basis meaning far less cpu/cache traffic in high contention scenarios.

  • Attempting to extend a protocol method that doesn't exist is an error at extension time.

  • Overriding the protocol for the base object array class overrides it for all things convertible to object array while still allowing the concrete array type to match a specific override.

Another design decision is to avoid the interface check - this simplifes the hot path a slight bit at the cost of slightly slower calltimes in the case the interface is used. For those cases often it is possible to simply typehint the interface and call it directly avoiding any protocol dispatch overhead.

Additional call overhead above and beyond a normal fn invocation in an arm mac is -6ns - the time for .getClass call into single concurrent hash map lookup.

defprotocol

macro

added in 1.2

(defprotocol name & opts+sigs)

A protocol is a named set of named methods and their signatures:

  (defprotocol AProtocolName

    ;optional doc string
    "A doc string for AProtocol abstraction"

   ;options
   :extend-via-metadata true

  ;method signatures
    (bar [this a b] "bar docs")
    (baz [this a] [this a b] [this a b c] "baz docs"))

No implementations are provided. Docs can be specified for the protocol overall and for each method. The above yields a set of polymorphic functions and a protocol object. All are namespace-qualified by the ns enclosing the definition The resulting functions dispatch on the type of their first argument, which is required and corresponds to the implicit target object ('this' in Java parlance). defprotocol is dynamic, has no special compile-time effect, and defines no new types or classes. Implementations of the protocol methods can be provided using extend.

When :extend-via-metadata is true, values can extend protocols by adding metadata where keys are fully-qualified protocol function symbols and values are function implementations. Protocol implementations are checked first for direct definitions (defrecord, deftype, reify), then metadata definitions, then external extensions (extend, extend-type, extend-protocol)

defprotocol will automatically generate a corresponding interface, with the same name as the protocol, i.e. given a protocol: my.ns/Protocol, an interface: my.ns.Protocol. The interface will have methods corresponding to the protocol functions, and the protocol will automatically work with instances of the interface.

Note that you should not use this interface with deftype or reify, as they support the protocol directly:


  (defprotocol P
    (foo [this])
    (bar-me [this] [this y]))

  (deftype Foo [a b c]
   P
    (foo [this] a)
    (bar-me [this] b)
    (bar-me [this y] (+ c y)))

  (bar-me (Foo. 1 2 3) 42)
  => 45

  (foo
    (let [x 42]
      (reify P
        (foo [this] 17)
        (bar-me [this] x)
        (bar-me [this y] x))))
  => 17

extend

added in 1.2

(extend atype & proto+mmaps)

Implementations of protocol methods can be provided using the extend construct:

  (extend AType
    AProtocol
     {:foo an-existing-fn
      :bar (fn [a b] ...)
      :baz (fn ([a]...) ([a b] ...)...)}
    BProtocol
      {...}
    ...)

extend takes a type/class (or interface, see below), and one or more protocol + method map pairs. It will extend the polymorphism of the protocol's methods to call the supplied methods when an AType is provided as the first argument.

Method maps are maps of the keyword-ized method names to ordinary fns. This facilitates easy reuse of existing fns and fn maps, for code reuse/mixins without derivation or composition. You can extend an interface to a protocol. This is primarily to facilitate interop with the host (e.g. Java) but opens the door to incidental multiple inheritance of implementation since a class can inherit from more than one interface, both of which extend the protocol. It is TBD how to specify which impl to use. You can extend a protocol on nil.

If you are supplying the definitions explicitly (i.e. not reusing exsting functions or mixin maps), you may find it more convenient to use the extend-type or extend-protocol macros.

Note that multiple independent extend clauses can exist for the same type, not all protocols need be defined in a single extend call.

See also: extends?, satisfies?, extenders

extend-protocol

macro

added in 1.2

(extend-protocol p & specs)

Useful when you want to provide several implementations of the same protocol all at once. Takes a single protocol and the implementation of that protocol for one or more types. Expands into calls to extend-type:

  (extend-protocol Protocol
    AType
      (foo [x] ...)
      (bar [x y] ...)
    BType
      (foo [x] ...)
      (bar [x y] ...)
    AClass
      (foo [x] ...)
      (bar [x y] ...)
    nil
      (foo [x] ...)
      (bar [x y] ...))

expands into:

  (do
   (clojure.core/extend-type AType Protocol
     (foo [x] ...)
     (bar [x y] ...))
   (clojure.core/extend-type BType Protocol
     (foo [x] ...)
     (bar [x y] ...))
   (clojure.core/extend-type AClass Protocol
     (foo [x] ...)
     (bar [x y] ...))
   (clojure.core/extend-type nil Protocol
     (foo [x] ...)
     (bar [x y] ...)))

extend-type

macro

added in 1.2

(extend-type t & specs)

A macro that expands into an extend call. Useful when you are supplying the definitions explicitly inline, extend-type automatically creates the maps required by extend. Propagates the class as a type hint on the first argument of all fns.

  (extend-type MyType
    Countable
      (cnt [c] ...)
    Foo
      (bar [x y] ...)
      (baz ([x] ...) ([x y & zs] ...)))

expands into:


  (extend MyType
   Countable
     {:cnt (fn [c] ...)}
   Foo
     {:baz (fn ([x] ...) ([x y & zs] ...))
      :bar (fn [x y] ...)})

extenders

(extenders protocol)

Returns a collection of the types explicitly extending protocol

extends?

(extends? protocol atype)

Returns true if atype extends protocol

find-protocol-cache-method

(find-protocol-cache-method protocol cache x)

find-protocol-method

(find-protocol-method protocol methodk x)

It may be more efficient in a tight loop to bypass the protocol dispatch on a per-call basis.

satisfies?

(satisfies? protocol x)

Returns true if x satisfies the protocol

================================================ FILE: docs/ham-fisted.fjp.html ================================================ ham-fisted.fjp documentation

ham-fisted.fjp

Support for java.util.concurrent.ForkJoinPool-specific operations such as managed block and task fork/join. Additionally supports concept of 'exception-safe' via wrapping executing code and unwrapper result post execution in order avoid wrapping exceptions and breaking calling code that may be expecting specific exception types.

Some of the api's fall back to regular executor service code when called.

Example:

(defn split-parallel-reduce
  "Perform a parallel reduction of a spliterator using the provided ExecutorService"
  [executor-service split ideal-split init-fn rfn merge-fn]
  (let [n-elems (proto/estimate-count split)
        pool (or executor-service (ForkJoinPool/commonPool))]
    (if (or (<= n-elems (long ideal-split)) (= n-elems Long/MAX_VALUE))
      (split-reduce rfn (init-fn) split)
      (if-let [[lhs rhs] (proto/split split)]
        (let [lt (fjp/safe-fork-task pool (split-parallel-reduce pool lhs ideal-split init-fn rfn merge-fn))
              rt (fjp/safe-fork-task pool (split-parallel-reduce pool rhs ideal-split init-fn rfn merge-fn))]
          (merge-fn (fjp/managed-block-unwrap lt) (fjp/managed-block-unwrap rt)))))))

common-pool

(common-pool)

Returns (ForkJoinPool/commonPool)

common-pool-parallelism

(common-pool-parallelism)

Integer parallelism assigned to the common pool

compute

(compute t)

compute a task in current thread - returns result

exception-safe

macro

(exception-safe & code)

Wrap code in an exception-safe wrapper - returns a map with either :ham-fisted.fjp/result or :ham-fisted.fjp.error.

fork-task

(fork-task pool f)

Begin a separate execution for f. If already in a fork join pool fork the task else submit f to passed in pool.

in-fork-join-pool?

(in-fork-join-pool?)

Returns true if this task is executing in a fork join pool thread

join

(join t)

join a previously forked task returning the result

managed-block

(managed-block dly)(managed-block finished? wait-till-finished get-value)

Block on a delay or future using the fjp system's managed blocking facility. Safe to call all the time whether the current system is in a forkjoinpool task or not.

managed-block-unwrap

(managed-block-unwrap dly)

managed block then safe unwrap the exception-safe result

on-cp

macro

(on-cp & code)

Run arbitrary code on the common-pool. Make sure any blocking operations are wrapped in managed-block.

safe-common-pool

(safe-common-pool safe-code)

Run safe code - see exception-safe unwrapping the result and re-throwing the wrapped exception. This allows systems based on typed exceptions to pass error info.

safe-fork-task

macro

(safe-fork-task pool & code)

Called from within an executing task, fork a executing some code and wrapping it in exception-safe then calling fork-task

task

(task f)

Create a task from a clojure IFn or something that implements IDeref

unsafe-common-pool

(unsafe-common-pool code)

Run a callable on the common pool. If the callable throws you will get a wrapped exception thrown which may confuse calling code - specifically code that relies on exact exception types or ex-info.

unwrap-safe

(unwrap-safe m)

Unwrap result created via executing code wrapped in exception-safe. Throws original exception if found.

================================================ FILE: docs/ham-fisted.function.html ================================================ ham-fisted.function documentation

ham-fisted.function

Helpers for working with java.util.function package objects.

->bi-function

(->bi-function cljfn)

Convert an object to a java.util.BiFunction. Object can either already be a bi-function or an IFn to be invoked with 2 arguments.

->function

(->function cljfn)

Convert an object to a java Function. Object can either already be a Function or an IFn to be invoked.

->long-predicate

(->long-predicate f)

bi-consumer

macro

(bi-consumer arg1 arg2 & code)

bi-function

macro

(bi-function arg1 arg2 & code)

Create an implementation of java.util.function.BiFunction.

binary-operator

macro

(binary-operator arg1 arg2 & code)

Create an implementation of java.util.function.BinaryOperator

binary-predicate

macro

(binary-predicate xvar yvar code)

Create a java.util.function.BiPredicate

binary-predicate-or-null

(binary-predicate-or-null f)

If f is null, return null. Else return f as a java.util.function.BiPredicate.

comp-nan-first

A comparator that sorts null, NAN first, natural order

comp-nan-last

A comparator that sorts null, NAN last, natural order

consumer

macro

(consumer varname & code)

Create an instance of a java.util.function.Consumer

double->long

macro

(double->long varname & code)

Create a function that receives a double and returns a long

double->obj

macro

(double->obj varname & code)

double-binary-operator

macro

(double-binary-operator lvar rvar & code)

Create a binary operator that is specialized for double values. Useful to speed up operations such as sorting or summation.

double-consumer

macro

(double-consumer varname & code)

Create an instance of a java.util.function.DoubleConsumer

double-predicate

macro

(double-predicate varname & code)

Create an implementation of java.util.Function.DoublePredicate

double-unary-operator

macro

(double-unary-operator varname & code)

Create an implementation of java.util.function.DoubleUnaryOperator

function

macro

(function arg & code)

Create a java.util.function.Function

long->double

macro

(long->double varname & code)

Create a function that receives a long and returns a double

long->obj

macro

(long->obj varname & code)

Create a function that receives a primitive long and returns an object.

long-binary-operator

macro

(long-binary-operator lvar rvar & code)

Create a binary operator that is specialized for long values. Useful to speed up operations such as sorting or summation.

long-consumer

macro

(long-consumer varname & code)

Create an instance of a java.util.function.LongConsumer

long-predicate

macro

(long-predicate varname & code)

Create an implementation of java.util.Function.LongPredicate

long-unary-operator

macro

(long-unary-operator varname & code)

Create an implementation of java.util.function.LongUnaryOperator

make-comparator

macro

(make-comparator lhsvar rhsvar & code)

Make a java comparator.

make-double-comparator

macro

(make-double-comparator lhsvar rhsvar & code)

Make a comparator that gets passed two double arguments.

make-long-comparator

macro

(make-long-comparator lhsvar rhsvar & code)

Make a comparator that gets passed two long arguments.

obj->double

macro

(obj->double)(obj->double varname & code)

Create a function that converts objects to doubles

obj->long

macro

(obj->long)(obj->long varname & code)

Create a function that converts objects to longs

predicate

macro

(predicate varname & code)

Create an implementation of java.util.Function.Predicate

rcomp

A reverse comparator that sorts in descending order

unary-operator

macro

(unary-operator varname & code)

Create an implementation of java.util.function.UnaryOperator

================================================ FILE: docs/ham-fisted.hlet.html ================================================ ham-fisted.hlet documentation

ham-fisted.hlet

Extensible let to allow efficient typed destructuring.

Registered Extensions:

dbls and lngs will most efficiently destructure java primitive arrays and fall back to casting the result of clojure.lang.RT/nth if input is not a double or long array.

dlb-fns and lng-fns call the object's IFn interface with no interface checking. This will not work with a raw array but is the fastest way - faster than RT/nth - to get data out of a persistent-vector or map like object.

obj-fns - fast IFn-based destructuring to objects - does not work with object arrays. Often much faster than RT/nth.

This can significantly reduce boxing in tight loops without needing to result in really verbose pathways.

user> (h/let [[a b] (dbls [1 2])] (+ a b))
  3.0
user> (hamf/sum-fast (lznc/cartesian-map
                      #(h/let [[a b c d](lng-fns %)]
                         (-> (+ a b) (+ c) (+ d)))
                      [1 2 3]
                      [4 5 6]
                      [7 8 9]
                      [10 11 12 13 14]))
3645.0

See also ham-fisted.primitive-invoke, ham-fisted.api/dnth ham-fisted.api/lnth.

extend-let

(extend-let sym-name code)

Code gets a tuple of lhs rhs must return a flattened sequence of left and right hand sides. This uses a special symbol that will look like a function call on the right hand side as the dispatch mechanism.

See source code of this file for example extensions.

let

macro

(let bindings & body)

Extensible let intended to allow typed destructuring of arbitrary datatypes such as primitive arrays or point types. Falls back to normal let after extension process. Several extensions are registered by default -

  • dbls and lngs which destructure into primitive doubles and primitive longs, respectively.
  • dlb-fns and lng-fns which destructure into primitive doubls and longs but use the often faster IFn overloads to get the data - avoiding RT.nth calls.
  • obj-fns which destructure into objects using the IFn interface.
user> (h/let [[x y] (dbls (hamf/double-array [1 2]))]
        (+ x y))
3.0

let-extension-names

(let-extension-names)

Return the current extension names.

================================================ FILE: docs/ham-fisted.iterator.html ================================================ ham-fisted.iterator documentation

ham-fisted.iterator

Generialized efficient pathways involving iterators.

->iterator

(->iterator item)

Convert a stream, supplier, or an iterable into an iterator.

ary-iter

(ary-iter ary-data)

Create an iterator for any primitive or object java array.

blocking-queue->iterable

(blocking-queue->iterable queue term-symbol)(blocking-queue->iterable queue timeout-us timeout-symbol term-symbol)

Given a blocking queue return an iterable that iterates until queue returns term-symbol. Uses take to block indefinitely -- will throw any throwable that comes out of the queue.

const-iterable

(const-iterable arg)

Return an iterable that always returns a arg.

ctx-iter

(ctx-iter valid? init-fn update-fn val-fn)

current-iterator

(current-iterator item)

Return a current iterator - and iterator that retains the current object. This iterator is positioned just before the first object so it's current item is nil.

dedup-first-by

(dedup-first-by key-fn cmp data)(dedup-first-by cmp data)

Given a sorted sequence remove duplicates keeping first.

doiter

macro

(doiter varname iterable & body)

Execute body for every item in the iterable. Expecting side effects, returns nil.

has-next?

(has-next? iter)

Given an iterator or nil return true if the iterator has next.

iter-cons

(iter-cons vv iter)

Produce a new iterator that points to vv then defers to passed in iterator.

iter-take

(iter-take n coll)

take n from an iterator returning a new iterator

iter-take-while

(iter-take-while pred iter)

Returns {:data :rest*} where rest* resolves to an iterator once data has been completely consumed.

iterable

(iterable valid? init-fn update-fn val-fn)(iterable init-fn update-fn)

Create an iterable. init-fn is not called until the first has-next call.

  • valid? - ctx->boolean - defaults to non-nil?
  • init-fn - creates new ctx
  • update-fn - function from ctx->ctx
  • val-fn - function from ctx->val - defaults to deref

linear-merge-iterator

(linear-merge-iterator cmp p iterators)(linear-merge-iterator cmp iterators)(linear-merge-iterator iterators)

Create a merging iterator - fast for N < 8.

maybe-next

(maybe-next iter)

Given an iterator or nil return the next element if the iterator hasNext.

merge-iterable

(merge-iterable cmp iterables)

Create an efficient stable n-way merge between a sequence of iterables using comparator. If iterables themselves are sorted result will be sorted. If two items tie then the one from the leftmost iterable wins.

next

(next iter)

Given an iterator call next.

non-nil?

(non-nil? a)

once-iterable

(once-iterable valid? init-fn update-fn val-fn)(once-iterable valid? init-fn)(once-iterable init-fn)

Create an iterable that can only be iterated once - it always returns the same (non threadsafe) iterator every iterator call. init-fn is not called until the first has-next call - also see iterable.

The arguments have different defaults as once-iterables can close over global context on construction as they can only be iterated once.

  • valid? - ctx->boolean - defaults to non-nil?
  • init-fn - creates new ctx
  • update-fn - function from ctx->ctx - defaults to init-fn ignoring argument.
  • val-fn - function from ctx->val - defaults to identity

seq-iterable

(seq-iterable iterable)

Iterable with efficient reduce but also contains a cached seq conversion so patterns like: (when (seq v) ...) still work without needing to dereference the iterable twice.

seq-once-iterable

(seq-once-iterable valid? init update val-fn)(seq-once-iterable valid? init)(seq-once-iterable init)

unstable-merge-iterable

(unstable-merge-iterable cmp iterables)

Create an efficient n-way merge between a sequence of iterables using comparator. If iterables themselves are sorted result will be sorted.

wrap-iter

(wrap-iter iter)

Wrap an iterator returning an iterable.

================================================ FILE: docs/ham-fisted.lazy-noncaching.html ================================================ ham-fisted.lazy-noncaching documentation

ham-fisted.lazy-noncaching

Lazy, noncaching implementation of many clojure.core functions. There are several benefits of carefully constructed lazy noncaching versions:

  1. No locking - better multithreading/green thread performance.
  2. Higher performance generally.
  3. More datatype flexibility - if map is passed a single randomly addressible or generically parallelizable container the result is still randomly addressible or generically perallelizable. For instance (map key {:a 1 :b 2}) returns in the generic case something that can still be parallelizable as the entry set of a map implements spliterator.

->collection

(->collection item)

Ensure an item implements java.util.Collection. This is inherently true for seqs and any implementation of java.util.List but not true for object arrays. For maps this returns the entry set.

->iterable

(->iterable a)

->random-access

(->random-access item)

->reducible

(->reducible item)

apply-concat

(apply-concat)(apply-concat data)(apply-concat opts data)

A more efficient form of (apply concat ...) that doesn't force data to be a clojure seq. See concat-opts for opts definition.

as-random-access

(as-random-access item)

If item implements RandomAccess, return List interface.

boolean-alength

(boolean-alength m)

byte-alength

(byte-alength m)

cartesian-map

(cartesian-map f)(cartesian-map f a)(cartesian-map f a b)(cartesian-map f a b & args)

Create a new sequence that is the cartesian join of the input sequence passed through f. Unlike map, f is passed the arguments as a single persistent vector. This is to enable much higher efficiency in the higher-arity applications. For tight numeric loops, see ham-fisted.hlet/let.

The argument vector is mutably updated between function calls so you can't cache it. Use (into [] args) or some variation thereof to cache the arguments as is.

user> (hamf/sum-fast (lznc/cartesian-map
                      #(h/let [[a b c d](lng-fns %)]
                         (-> (+ a b) (+ c) (+ d)))
                      [1 2 3]
                      [4 5 6]
                      [7 8 9]
                      [10 11 12 13 14]))
3645.0

char-alength

(char-alength m)

complement

(complement f)

Like clojure core complement but avoids var lookup on 'not'

concat

(concat)(concat a)(concat a & args)

concat-opts

(concat-opts opts a)(concat-opts opts a & args)

Concat where the first argument is an options map. This variation allows you to set the :cat-parallelism as you may have an idea the best way to parallelism this concatenation at time of the concatenation creation.

Options:

:cat-parallelism - Set the type of parallelism - either :elem-wise or :seq-wise - this overrides settings later passed into calls such as reduce.preduce - see reduce/options->parallel-options for definition.

cond

macro

(cond & clauses)

See documentation for ham-fisted.api/cond

constant-count

(constant-count data)

Constant time count. Returns nil if input doesn't have a constant time count.

constant-countable?

(constant-countable? data)

count

(count m)

countable-arrays

macro

(countable-arrays)

define-drop-tducer

macro

(define-drop-tducer nm iface rf-tag)

define-take-tducer

macro

(define-take-tducer nm invoke-nm iface rf-tag)

double-alength

(double-alength m)

drop

(drop n)(drop n data)

empty-vec

every?

(every? pred coll)

Faster (in most circumstances) implementation of clojure.core/every?. This can be much faster in the case of primitive arrays of values. Type-hinted functions are best if coll is primitive array - see example.

user> (type data)
[J
user> (count data)
100
user> (def vdata (vec data))
#'user/vdata
user> (crit/quick-bench (every? (fn [^long v] (> v 80)) data))
             Execution time mean : 40.248868 ns
nil
user> (crit/quick-bench (lznc/every? (fn [^long v] (> v 80)) data))
             Execution time mean : 7.601190 ns
nil
user> (crit/quick-bench (every? (fn [^long v] (< v 80)) vdata))
             Execution time mean : 1.269582 µs
nil
user> (crit/quick-bench (lznc/every? (fn [^long v] (< v 80)) vdata))
             Execution time mean : 211.645613 ns
nil
user>

filter

(filter pred)(filter pred coll)

float-alength

(float-alength m)

int-alength

(int-alength m)

into-array

(into-array aseq)(into-array ary-type aseq)(into-array ary-type mapfn aseq)

long-alength

(long-alength m)

make-readonly-list

macro

(make-readonly-list n idxvar read-code)(make-readonly-list cls-type-kwd n idxvar read-code)

Implement a readonly list. If cls-type-kwd is provided it must be, at compile time, either :int64, :float64 or :object and the getLong, getDouble or get interface methods will be filled in, respectively. In those cases read-code must return the appropriate type.

map

(map f)(map f arg)(map f arg & args)

map-indexed

(map-indexed map-fn coll)

map-reducible

(map-reducible f r)

Map a function over r - r need only be reducible. Returned value does not implement seq but is countable when r is countable.

object-alength

(object-alength m)

object-array

(object-array item)

Faster version of object-array for eductions, java collections and strings.

partition-all

(partition-all n)(partition-all n coll)(partition-all n step coll)

Lazy noncaching version of partition-all. When input is random access returns random access result.

If input is not random access then similar to partition-by each sub-collection must be entirely iterated through before requesting the next sub-collection.

user> (crit/quick-bench (mapv hamf/sum-fast (lznc/partition-all 100 (range 100000))))
             Execution time mean : 335.821098 µs
nil
user> (crit/quick-bench (mapv hamf/sum-fast (partition-all 100 (range 100000))))
             Execution time mean : 6.831242 ms
nil
user> (crit/quick-bench (into [] (comp (partition-all 100)
                                       (map hamf/sum-fast))
                              (range 100000)))
             Execution time mean : 1.645954 ms
nil

partition-by

(partition-by f)(partition-by f coll)(partition-by f options coll)

Lazy noncaching version of partition-by. For reducing partitions into a singular value please see apply-concat. Return value most efficiently implements reduce with a slightly less efficient implementation of Iterable.

Unlike clojure.core/partition-by this does not store intermediate elements nor does it build up intermediate containers. This makes it somewhat faster in most contexts.

Each sub-collection must be iterated through entirely before the next method of the parent iterator else the result will not be correct.

Options:

  • :ignore-leftover? - When true leftover items in the previous iteration do not cause an exception. Defaults to false.
  • :binary-predicate - When provided, use this for equality semantics. Defaults to equiv semantics but in a numeric context it may be useful to have (== ##NaN ##Nan).
user> ;;incorrect - inner items not iterated and non-caching!
user> (into [] (lznc/partition-by identity [1 1 1 2 2 2 3 3 3]))
Execution error at ham_fisted.lazy_noncaching.PartitionBy/reduce (lazy_noncaching.clj:514).
Sub-collection was not entirely consumed.

user> ;;correct - transducing form of into calls vec on each sub-collection
user> ;;thus iterating through it entirely.
user> (into [] (map vec) (lznc/partition-by identity [1 1 1 2 2 2 3 3 3]))
[[1 1 1] [2 2 2] [3 3 3]]
user> ;;filter,collect NaN out of sequence
user> (lznc/map hamf/vec (lznc/partition-by identity {:binary-predicate (hamf-fn/binary-predicate
                                                                         x y (let [x (double x)
                                                                                   y (double y)]
                                                                               (cond
                                                                                 (Double/isNaN x)
                                                                                 (if (Double/isNaN y)
                                                                                   true
                                                                                   false)
                                                                                 (Double/isNaN y) false
                                                                                 :else true))) }
                                            [1 2 3 ##NaN ##NaN 3 4 5]))
([1 2 3] [NaN NaN] [3 4 5])

user> (def init-data (vec (lznc/apply-concat (lznc/map #(repeat 100 %) (range 1000)))))
#'user/init-data
user> (crit/quick-bench (mapv hamf/sum-fast (lznc/partition-by identity init-data)))
             Execution time mean : 366.915796 µs
  ...
nil
user> (crit/quick-bench (mapv hamf/sum-fast (clojure.core/partition-by identity init-data)))
             Execution time mean : 6.699424 ms
  ...
nil
user> (crit/quick-bench (into [] (comp (clojure.core/partition-by identity)
                                       (map hamf/sum-fast)) init-data))
             Execution time mean : 1.705864 ms
  ...

partition-by-comparator

(partition-by-comparator cmp coll)(partition-by-comparator cmp options coll)

Partition by a comparator. Sub partitions must be fully consumed before parent is called. An optional timeout can be used if sub partitions are being consumed in a future - the parent iteration will block until the sub partition is fully consumed.

Options:

  • :timeout-ms - Timeout in milliseconds - if the sub partition isn't consumed by this time an exception is thrown.

partition-by-cost

(partition-by-cost cost-fn max-cost coll)

Partition a sequence by integer cost. Will produce partitions of average cost '(quot max-cost 2) with some partitions being larger or smaller if the input sequence ends. This function not very lazy with each new partition greedily calculated.

reindex

(reindex coll indexes)

Permut coll by the given indexes. Result is random-access and the same length as the index collection. Indexes are expected to be in the range of [0->count(coll)).

remove

added in 1.0

(remove pred coll)(remove pred)

Returns a lazy sequence of the items in coll for which (pred item) returns logical false. pred must be free of side-effects. Returns a transducer when no collection is provided.

repeatedly

(repeatedly f)(repeatedly n f)

When called with one argument, produce infinite list of calls to v. When called with two arguments, produce a non-caching random access list of length n of calls to v.

seed->random

(seed->random seed)

shift

(shift n coll)

Shift a collection forward or backward repeating either the first or the last entries. Returns a random access list with the same elements as coll.

Example:

ham-fisted.api> (shift 2 (range 10))
[0 0 0 1 2 3 4 5 6 7]
ham-fisted.api> (shift -2 (range 10))
[2 3 4 5 6 7 8 9 9 9]

short-alength

(short-alength m)

shuffle

(shuffle coll)(shuffle coll opts)

shuffle values returning random access container.

Options:

  • :seed - If instance of java.util.Random, use this. If integer, use as seed. If not provided a new instance of java.util.Random is created.

take

(take n)(take n data)

tuple-map

(tuple-map f c1)(tuple-map f c1 c2)(tuple-map f c1 c2 & cs)

Lazy nonaching map but f simply gets a single random-access list of arguments. The argument list may be mutably updated between calls.

type-single-arg-ifn

(type-single-arg-ifn ifn)

Categorize the return type of a single argument ifn. May be :float64, :int64, or :object.

type-zero-arg-ifn

(type-zero-arg-ifn ifn)

Categorize the return type of a single argument ifn. May be :float64, :int64, or :object.

================================================ FILE: docs/ham-fisted.mut-map.html ================================================ ham-fisted.mut-map documentation

ham-fisted.mut-map

Functions for working with java's mutable map interface

compute!

(compute! m k bfn)

Compute a new value in a map derived from an existing value. bfn gets passed k, v where k may be nil. If the function returns nil the corresponding key is removed from the map.

See Map.compute

An example bfn for counting occurrences would be #(if % (inc (long %)) 1).

compute-if-absent!

(compute-if-absent! m k bfn)

Compute a value if absent from the map. Useful for memoize-type operations. Must use mutable maps. bfn gets passed k.

See map.computeIfAbsent

compute-if-present!

(compute-if-present! m k bfn)

Compute a new value if the value already exists and is non-nil in the hashmap. Must use mutable maps. bfn gets passed k, v where v is non-nil.

See Map.computeIfPresent

keyset

(keyset m)

Return the keyset of the map. This may not be in the same order as (keys m) or (vals m). For hamf maps, this has the same ordering as (keys m). For both hamf and java hashmaps, the returned implementation of java.util.Set has both more utility and better performance than (keys m).

values

(values m)

Return the values collection of the map. This may not be in the same order as (keys m) or (vals m). For hamf hashmaps, this does have the same order as (vals m).

================================================ FILE: docs/ham-fisted.primitive-invoke.html ================================================ ham-fisted.primitive-invoke documentation

ham-fisted.primitive-invoke

For statically traced calls the Clojure compiler calls the primitive version of type-hinted functions and this makes quite a difference in tight loops. Often times, however, functions are passed by values or returned from if-statements and then you need to explicitly call the primitive overload - this makes that pathway less verbose. Functions must first be check-casted to their primitive types and then calling them will use their primitive overloads avoiding all casting.

(defn doit [f x y]
   (let [f (pi/->ddd f)]
     (loop [x x y y]
      (if (< x y)
        (recur (pi/ddd f x y) y)
        x))))

->d

(->d f)

->dd

(->dd f)

->ddd

(->ddd f)

->dddd

(->dddd f)

->ddddd

(->ddddd f)

->ddddl

(->ddddl f)

->ddddo

(->ddddo f)

->dddl

(->dddl f)

->dddld

(->dddld f)

->dddll

(->dddll f)

->dddlo

(->dddlo f)

->dddo

(->dddo f)

->dddod

(->dddod f)

->dddol

(->dddol f)

->dddoo

(->dddoo f)

->ddl

(->ddl f)

->ddld

(->ddld f)

->ddldd

(->ddldd f)

->ddldl

(->ddldl f)

->ddldo

(->ddldo f)

->ddll

(->ddll f)

->ddlld

(->ddlld f)

->ddlll

(->ddlll f)

->ddllo

(->ddllo f)

->ddlo

(->ddlo f)

->ddlod

(->ddlod f)

->ddlol

(->ddlol f)

->ddloo

(->ddloo f)

->ddo

(->ddo f)

->ddod

(->ddod f)

->ddodd

(->ddodd f)

->ddodl

(->ddodl f)

->ddodo

(->ddodo f)

->ddol

(->ddol f)

->ddold

(->ddold f)

->ddoll

(->ddoll f)

->ddolo

(->ddolo f)

->ddoo

(->ddoo f)

->ddood

(->ddood f)

->ddool

(->ddool f)

->ddooo

(->ddooo f)

->dl

(->dl f)

->dld

(->dld f)

->dldd

(->dldd f)

->dlddd

(->dlddd f)

->dlddl

(->dlddl f)

->dlddo

(->dlddo f)

->dldl

(->dldl f)

->dldld

(->dldld f)

->dldll

(->dldll f)

->dldlo

(->dldlo f)

->dldo

(->dldo f)

->dldod

(->dldod f)

->dldol

(->dldol f)

->dldoo

(->dldoo f)

->dll

(->dll f)

->dlld

(->dlld f)

->dlldd

(->dlldd f)

->dlldl

(->dlldl f)

->dlldo

(->dlldo f)

->dlll

(->dlll f)

->dllld

(->dllld f)

->dllll

(->dllll f)

->dlllo

(->dlllo f)

->dllo

(->dllo f)

->dllod

(->dllod f)

->dllol

(->dllol f)

->dlloo

(->dlloo f)

->dlo

(->dlo f)

->dlod

(->dlod f)

->dlodd

(->dlodd f)

->dlodl

(->dlodl f)

->dlodo

(->dlodo f)

->dlol

(->dlol f)

->dlold

(->dlold f)

->dloll

(->dloll f)

->dlolo

(->dlolo f)

->dloo

(->dloo f)

->dlood

(->dlood f)

->dlool

(->dlool f)

->dlooo

(->dlooo f)

->do

(->do f)

->dod

(->dod f)

->dodd

(->dodd f)

->doddd

(->doddd f)

->doddl

(->doddl f)

->doddo

(->doddo f)

->dodl

(->dodl f)

->dodld

(->dodld f)

->dodll

(->dodll f)

->dodlo

(->dodlo f)

->dodo

(->dodo f)

->dodod

(->dodod f)

->dodol

(->dodol f)

->dodoo

(->dodoo f)

->dol

(->dol f)

->dold

(->dold f)

->doldd

(->doldd f)

->doldl

(->doldl f)

->doldo

(->doldo f)

->doll

(->doll f)

->dolld

(->dolld f)

->dolll

(->dolll f)

->dollo

(->dollo f)

->dolo

(->dolo f)

->dolod

(->dolod f)

->dolol

(->dolol f)

->doloo

(->doloo f)

->doo

(->doo f)

->dood

(->dood f)

->doodd

(->doodd f)

->doodl

(->doodl f)

->doodo

(->doodo f)

->dool

(->dool f)

->doold

(->doold f)

->dooll

(->dooll f)

->doolo

(->doolo f)

->dooo

(->dooo f)

->doood

(->doood f)

->doool

(->doool f)

->doooo

(->doooo f)

->l

(->l f)

->ld

(->ld f)

->ldd

(->ldd f)

->lddd

(->lddd f)

->ldddd

(->ldddd f)

->ldddl

(->ldddl f)

->ldddo

(->ldddo f)

->lddl

(->lddl f)

->lddld

(->lddld f)

->lddll

(->lddll f)

->lddlo

(->lddlo f)

->lddo

(->lddo f)

->lddod

(->lddod f)

->lddol

(->lddol f)

->lddoo

(->lddoo f)

->ldl

(->ldl f)

->ldld

(->ldld f)

->ldldd

(->ldldd f)

->ldldl

(->ldldl f)

->ldldo

(->ldldo f)

->ldll

(->ldll f)

->ldlld

(->ldlld f)

->ldlll

(->ldlll f)

->ldllo

(->ldllo f)

->ldlo

(->ldlo f)

->ldlod

(->ldlod f)

->ldlol

(->ldlol f)

->ldloo

(->ldloo f)

->ldo

(->ldo f)

->ldod

(->ldod f)

->ldodd

(->ldodd f)

->ldodl

(->ldodl f)

->ldodo

(->ldodo f)

->ldol

(->ldol f)

->ldold

(->ldold f)

->ldoll

(->ldoll f)

->ldolo

(->ldolo f)

->ldoo

(->ldoo f)

->ldood

(->ldood f)

->ldool

(->ldool f)

->ldooo

(->ldooo f)

->ll

(->ll f)

->lld

(->lld f)

->lldd

(->lldd f)

->llddd

(->llddd f)

->llddl

(->llddl f)

->llddo

(->llddo f)

->lldl

(->lldl f)

->lldld

(->lldld f)

->lldll

(->lldll f)

->lldlo

(->lldlo f)

->lldo

(->lldo f)

->lldod

(->lldod f)

->lldol

(->lldol f)

->lldoo

(->lldoo f)

->lll

(->lll f)

->llld

(->llld f)

->llldd

(->llldd f)

->llldl

(->llldl f)

->llldo

(->llldo f)

->llll

(->llll f)

->lllld

(->lllld f)

->lllll

(->lllll f)

->llllo

(->llllo f)

->lllo

(->lllo f)

->lllod

(->lllod f)

->lllol

(->lllol f)

->llloo

(->llloo f)

->llo

(->llo f)

->llod

(->llod f)

->llodd

(->llodd f)

->llodl

(->llodl f)

->llodo

(->llodo f)

->llol

(->llol f)

->llold

(->llold f)

->lloll

(->lloll f)

->llolo

(->llolo f)

->lloo

(->lloo f)

->llood

(->llood f)

->llool

(->llool f)

->llooo

(->llooo f)

->lo

(->lo f)

->lod

(->lod f)

->lodd

(->lodd f)

->loddd

(->loddd f)

->loddl

(->loddl f)

->loddo

(->loddo f)

->lodl

(->lodl f)

->lodld

(->lodld f)

->lodll

(->lodll f)

->lodlo

(->lodlo f)

->lodo

(->lodo f)

->lodod

(->lodod f)

->lodol

(->lodol f)

->lodoo

(->lodoo f)

->lol

(->lol f)

->lold

(->lold f)

->loldd

(->loldd f)

->loldl

(->loldl f)

->loldo

(->loldo f)

->loll

(->loll f)

->lolld

(->lolld f)

->lolll

(->lolll f)

->lollo

(->lollo f)

->lolo

(->lolo f)

->lolod

(->lolod f)

->lolol

(->lolol f)

->loloo

(->loloo f)

->loo

(->loo f)

->lood

(->lood f)

->loodd

(->loodd f)

->loodl

(->loodl f)

->loodo

(->loodo f)

->lool

(->lool f)

->loold

(->loold f)

->looll

(->looll f)

->loolo

(->loolo f)

->looo

(->looo f)

->loood

(->loood f)

->loool

(->loool f)

->loooo

(->loooo f)

->od

(->od f)

->odd

(->odd f)

->oddd

(->oddd f)

->odddd

(->odddd f)

->odddl

(->odddl f)

->odddo

(->odddo f)

->oddl

(->oddl f)

->oddld

(->oddld f)

->oddll

(->oddll f)

->oddlo

(->oddlo f)

->oddo

(->oddo f)

->oddod

(->oddod f)

->oddol

(->oddol f)

->oddoo

(->oddoo f)

->odl

(->odl f)

->odld

(->odld f)

->odldd

(->odldd f)

->odldl

(->odldl f)

->odldo

(->odldo f)

->odll

(->odll f)

->odlld

(->odlld f)

->odlll

(->odlll f)

->odllo

(->odllo f)

->odlo

(->odlo f)

->odlod

(->odlod f)

->odlol

(->odlol f)

->odloo

(->odloo f)

->odo

(->odo f)

->odod

(->odod f)

->ododd

(->ododd f)

->ododl

(->ododl f)

->ododo

(->ododo f)

->odol

(->odol f)

->odold

(->odold f)

->odoll

(->odoll f)

->odolo

(->odolo f)

->odoo

(->odoo f)

->odood

(->odood f)

->odool

(->odool f)

->odooo

(->odooo f)

->ol

(->ol f)

->old

(->old f)

->oldd

(->oldd f)

->olddd

(->olddd f)

->olddl

(->olddl f)

->olddo

(->olddo f)

->oldl

(->oldl f)

->oldld

(->oldld f)

->oldll

(->oldll f)

->oldlo

(->oldlo f)

->oldo

(->oldo f)

->oldod

(->oldod f)

->oldol

(->oldol f)

->oldoo

(->oldoo f)

->oll

(->oll f)

->olld

(->olld f)

->olldd

(->olldd f)

->olldl

(->olldl f)

->olldo

(->olldo f)

->olll

(->olll f)

->ollld

(->ollld f)

->ollll

(->ollll f)

->olllo

(->olllo f)

->ollo

(->ollo f)

->ollod

(->ollod f)

->ollol

(->ollol f)

->olloo

(->olloo f)

->olo

(->olo f)

->olod

(->olod f)

->olodd

(->olodd f)

->olodl

(->olodl f)

->olodo

(->olodo f)

->olol

(->olol f)

->olold

(->olold f)

->ololl

(->ololl f)

->ololo

(->ololo f)

->oloo

(->oloo f)

->olood

(->olood f)

->olool

(->olool f)

->olooo

(->olooo f)

->ood

(->ood f)

->oodd

(->oodd f)

->ooddd

(->ooddd f)

->ooddl

(->ooddl f)

->ooddo

(->ooddo f)

->oodl

(->oodl f)

->oodld

(->oodld f)

->oodll

(->oodll f)

->oodlo

(->oodlo f)

->oodo

(->oodo f)

->oodod

(->oodod f)

->oodol

(->oodol f)

->oodoo

(->oodoo f)

->ool

(->ool f)

->oold

(->oold f)

->ooldd

(->ooldd f)

->ooldl

(->ooldl f)

->ooldo

(->ooldo f)

->ooll

(->ooll f)

->oolld

(->oolld f)

->oolll

(->oolll f)

->oollo

(->oollo f)

->oolo

(->oolo f)

->oolod

(->oolod f)

->oolol

(->oolol f)

->ooloo

(->ooloo f)

->oood

(->oood f)

->ooodd

(->ooodd f)

->ooodl

(->ooodl f)

->ooodo

(->ooodo f)

->oool

(->oool f)

->ooold

(->ooold f)

->oooll

(->oooll f)

->ooolo

(->ooolo f)

->ooood

(->ooood f)

->ooool

(->ooool f)

d

macro

(d f)

dd

macro

(dd f arg0)

ddd

macro

(ddd f arg0 arg1)

dddd

macro

(dddd f arg0 arg1 arg2)

ddddd

macro

(ddddd f arg0 arg1 arg2 arg3)

ddddl

macro

(ddddl f arg0 arg1 arg2 arg3)

ddddo

macro

(ddddo f arg0 arg1 arg2 arg3)

dddl

macro

(dddl f arg0 arg1 arg2)

dddld

macro

(dddld f arg0 arg1 arg2 arg3)

dddll

macro

(dddll f arg0 arg1 arg2 arg3)

dddlo

macro

(dddlo f arg0 arg1 arg2 arg3)

dddo

macro

(dddo f arg0 arg1 arg2)

dddod

macro

(dddod f arg0 arg1 arg2 arg3)

dddol

macro

(dddol f arg0 arg1 arg2 arg3)

dddoo

macro

(dddoo f arg0 arg1 arg2 arg3)

ddl

macro

(ddl f arg0 arg1)

ddld

macro

(ddld f arg0 arg1 arg2)

ddldd

macro

(ddldd f arg0 arg1 arg2 arg3)

ddldl

macro

(ddldl f arg0 arg1 arg2 arg3)

ddldo

macro

(ddldo f arg0 arg1 arg2 arg3)

ddll

macro

(ddll f arg0 arg1 arg2)

ddlld

macro

(ddlld f arg0 arg1 arg2 arg3)

ddlll

macro

(ddlll f arg0 arg1 arg2 arg3)

ddllo

macro

(ddllo f arg0 arg1 arg2 arg3)

ddlo

macro

(ddlo f arg0 arg1 arg2)

ddlod

macro

(ddlod f arg0 arg1 arg2 arg3)

ddlol

macro

(ddlol f arg0 arg1 arg2 arg3)

ddloo

macro

(ddloo f arg0 arg1 arg2 arg3)

ddo

macro

(ddo f arg0 arg1)

ddod

macro

(ddod f arg0 arg1 arg2)

ddodd

macro

(ddodd f arg0 arg1 arg2 arg3)

ddodl

macro

(ddodl f arg0 arg1 arg2 arg3)

ddodo

macro

(ddodo f arg0 arg1 arg2 arg3)

ddol

macro

(ddol f arg0 arg1 arg2)

ddold

macro

(ddold f arg0 arg1 arg2 arg3)

ddoll

macro

(ddoll f arg0 arg1 arg2 arg3)

ddolo

macro

(ddolo f arg0 arg1 arg2 arg3)

ddoo

macro

(ddoo f arg0 arg1 arg2)

ddood

macro

(ddood f arg0 arg1 arg2 arg3)

ddool

macro

(ddool f arg0 arg1 arg2 arg3)

ddooo

macro

(ddooo f arg0 arg1 arg2 arg3)

dl

macro

(dl f arg0)

dld

macro

(dld f arg0 arg1)

dldd

macro

(dldd f arg0 arg1 arg2)

dlddd

macro

(dlddd f arg0 arg1 arg2 arg3)

dlddl

macro

(dlddl f arg0 arg1 arg2 arg3)

dlddo

macro

(dlddo f arg0 arg1 arg2 arg3)

dldl

macro

(dldl f arg0 arg1 arg2)

dldld

macro

(dldld f arg0 arg1 arg2 arg3)

dldll

macro

(dldll f arg0 arg1 arg2 arg3)

dldlo

macro

(dldlo f arg0 arg1 arg2 arg3)

dldo

macro

(dldo f arg0 arg1 arg2)

dldod

macro

(dldod f arg0 arg1 arg2 arg3)

dldol

macro

(dldol f arg0 arg1 arg2 arg3)

dldoo

macro

(dldoo f arg0 arg1 arg2 arg3)

dll

macro

(dll f arg0 arg1)

dlld

macro

(dlld f arg0 arg1 arg2)

dlldd

macro

(dlldd f arg0 arg1 arg2 arg3)

dlldl

macro

(dlldl f arg0 arg1 arg2 arg3)

dlldo

macro

(dlldo f arg0 arg1 arg2 arg3)

dlll

macro

(dlll f arg0 arg1 arg2)

dllld

macro

(dllld f arg0 arg1 arg2 arg3)

dllll

macro

(dllll f arg0 arg1 arg2 arg3)

dlllo

macro

(dlllo f arg0 arg1 arg2 arg3)

dllo

macro

(dllo f arg0 arg1 arg2)

dllod

macro

(dllod f arg0 arg1 arg2 arg3)

dllol

macro

(dllol f arg0 arg1 arg2 arg3)

dlloo

macro

(dlloo f arg0 arg1 arg2 arg3)

dlo

macro

(dlo f arg0 arg1)

dlod

macro

(dlod f arg0 arg1 arg2)

dlodd

macro

(dlodd f arg0 arg1 arg2 arg3)

dlodl

macro

(dlodl f arg0 arg1 arg2 arg3)

dlodo

macro

(dlodo f arg0 arg1 arg2 arg3)

dlol

macro

(dlol f arg0 arg1 arg2)

dlold

macro

(dlold f arg0 arg1 arg2 arg3)

dloll

macro

(dloll f arg0 arg1 arg2 arg3)

dlolo

macro

(dlolo f arg0 arg1 arg2 arg3)

dloo

macro

(dloo f arg0 arg1 arg2)

dlood

macro

(dlood f arg0 arg1 arg2 arg3)

dlool

macro

(dlool f arg0 arg1 arg2 arg3)

dlooo

macro

(dlooo f arg0 arg1 arg2 arg3)

do

macro

(do f arg0)

dod

macro

(dod f arg0 arg1)

dodd

macro

(dodd f arg0 arg1 arg2)

doddd

macro

(doddd f arg0 arg1 arg2 arg3)

doddl

macro

(doddl f arg0 arg1 arg2 arg3)

doddo

macro

(doddo f arg0 arg1 arg2 arg3)

dodl

macro

(dodl f arg0 arg1 arg2)

dodld

macro

(dodld f arg0 arg1 arg2 arg3)

dodll

macro

(dodll f arg0 arg1 arg2 arg3)

dodlo

macro

(dodlo f arg0 arg1 arg2 arg3)

dodo

macro

(dodo f arg0 arg1 arg2)

dodod

macro

(dodod f arg0 arg1 arg2 arg3)

dodol

macro

(dodol f arg0 arg1 arg2 arg3)

dodoo

macro

(dodoo f arg0 arg1 arg2 arg3)

dol

macro

(dol f arg0 arg1)

dold

macro

(dold f arg0 arg1 arg2)

doldd

macro

(doldd f arg0 arg1 arg2 arg3)

doldl

macro

(doldl f arg0 arg1 arg2 arg3)

doldo

macro

(doldo f arg0 arg1 arg2 arg3)

doll

macro

(doll f arg0 arg1 arg2)

dolld

macro

(dolld f arg0 arg1 arg2 arg3)

dolll

macro

(dolll f arg0 arg1 arg2 arg3)

dollo

macro

(dollo f arg0 arg1 arg2 arg3)

dolo

macro

(dolo f arg0 arg1 arg2)

dolod

macro

(dolod f arg0 arg1 arg2 arg3)

dolol

macro

(dolol f arg0 arg1 arg2 arg3)

doloo

macro

(doloo f arg0 arg1 arg2 arg3)

doo

macro

(doo f arg0 arg1)

dood

macro

(dood f arg0 arg1 arg2)

doodd

macro

(doodd f arg0 arg1 arg2 arg3)

doodl

macro

(doodl f arg0 arg1 arg2 arg3)

doodo

macro

(doodo f arg0 arg1 arg2 arg3)

dool

macro

(dool f arg0 arg1 arg2)

doold

macro

(doold f arg0 arg1 arg2 arg3)

dooll

macro

(dooll f arg0 arg1 arg2 arg3)

doolo

macro

(doolo f arg0 arg1 arg2 arg3)

dooo

macro

(dooo f arg0 arg1 arg2)

doood

macro

(doood f arg0 arg1 arg2 arg3)

doool

macro

(doool f arg0 arg1 arg2 arg3)

doooo

macro

(doooo f arg0 arg1 arg2 arg3)

l

macro

(l f)

ld

macro

(ld f arg0)

ldd

macro

(ldd f arg0 arg1)

lddd

macro

(lddd f arg0 arg1 arg2)

ldddd

macro

(ldddd f arg0 arg1 arg2 arg3)

ldddl

macro

(ldddl f arg0 arg1 arg2 arg3)

ldddo

macro

(ldddo f arg0 arg1 arg2 arg3)

lddl

macro

(lddl f arg0 arg1 arg2)

lddld

macro

(lddld f arg0 arg1 arg2 arg3)

lddll

macro

(lddll f arg0 arg1 arg2 arg3)

lddlo

macro

(lddlo f arg0 arg1 arg2 arg3)

lddo

macro

(lddo f arg0 arg1 arg2)

lddod

macro

(lddod f arg0 arg1 arg2 arg3)

lddol

macro

(lddol f arg0 arg1 arg2 arg3)

lddoo

macro

(lddoo f arg0 arg1 arg2 arg3)

ldl

macro

(ldl f arg0 arg1)

ldld

macro

(ldld f arg0 arg1 arg2)

ldldd

macro

(ldldd f arg0 arg1 arg2 arg3)

ldldl

macro

(ldldl f arg0 arg1 arg2 arg3)

ldldo

macro

(ldldo f arg0 arg1 arg2 arg3)

ldll

macro

(ldll f arg0 arg1 arg2)

ldlld

macro

(ldlld f arg0 arg1 arg2 arg3)

ldlll

macro

(ldlll f arg0 arg1 arg2 arg3)

ldllo

macro

(ldllo f arg0 arg1 arg2 arg3)

ldlo

macro

(ldlo f arg0 arg1 arg2)

ldlod

macro

(ldlod f arg0 arg1 arg2 arg3)

ldlol

macro

(ldlol f arg0 arg1 arg2 arg3)

ldloo

macro

(ldloo f arg0 arg1 arg2 arg3)

ldo

macro

(ldo f arg0 arg1)

ldod

macro

(ldod f arg0 arg1 arg2)

ldodd

macro

(ldodd f arg0 arg1 arg2 arg3)

ldodl

macro

(ldodl f arg0 arg1 arg2 arg3)

ldodo

macro

(ldodo f arg0 arg1 arg2 arg3)

ldol

macro

(ldol f arg0 arg1 arg2)

ldold

macro

(ldold f arg0 arg1 arg2 arg3)

ldoll

macro

(ldoll f arg0 arg1 arg2 arg3)

ldolo

macro

(ldolo f arg0 arg1 arg2 arg3)

ldoo

macro

(ldoo f arg0 arg1 arg2)

ldood

macro

(ldood f arg0 arg1 arg2 arg3)

ldool

macro

(ldool f arg0 arg1 arg2 arg3)

ldooo

macro

(ldooo f arg0 arg1 arg2 arg3)

ll

macro

(ll f arg0)

lld

macro

(lld f arg0 arg1)

lldd

macro

(lldd f arg0 arg1 arg2)

llddd

macro

(llddd f arg0 arg1 arg2 arg3)

llddl

macro

(llddl f arg0 arg1 arg2 arg3)

llddo

macro

(llddo f arg0 arg1 arg2 arg3)

lldl

macro

(lldl f arg0 arg1 arg2)

lldld

macro

(lldld f arg0 arg1 arg2 arg3)

lldll

macro

(lldll f arg0 arg1 arg2 arg3)

lldlo

macro

(lldlo f arg0 arg1 arg2 arg3)

lldo

macro

(lldo f arg0 arg1 arg2)

lldod

macro

(lldod f arg0 arg1 arg2 arg3)

lldol

macro

(lldol f arg0 arg1 arg2 arg3)

lldoo

macro

(lldoo f arg0 arg1 arg2 arg3)

lll

macro

(lll f arg0 arg1)

llld

macro

(llld f arg0 arg1 arg2)

llldd

macro

(llldd f arg0 arg1 arg2 arg3)

llldl

macro

(llldl f arg0 arg1 arg2 arg3)

llldo

macro

(llldo f arg0 arg1 arg2 arg3)

llll

macro

(llll f arg0 arg1 arg2)

lllld

macro

(lllld f arg0 arg1 arg2 arg3)

lllll

macro

(lllll f arg0 arg1 arg2 arg3)

llllo

macro

(llllo f arg0 arg1 arg2 arg3)

lllo

macro

(lllo f arg0 arg1 arg2)

lllod

macro

(lllod f arg0 arg1 arg2 arg3)

lllol

macro

(lllol f arg0 arg1 arg2 arg3)

llloo

macro

(llloo f arg0 arg1 arg2 arg3)

llo

macro

(llo f arg0 arg1)

llod

macro

(llod f arg0 arg1 arg2)

llodd

macro

(llodd f arg0 arg1 arg2 arg3)

llodl

macro

(llodl f arg0 arg1 arg2 arg3)

llodo

macro

(llodo f arg0 arg1 arg2 arg3)

llol

macro

(llol f arg0 arg1 arg2)

llold

macro

(llold f arg0 arg1 arg2 arg3)

lloll

macro

(lloll f arg0 arg1 arg2 arg3)

llolo

macro

(llolo f arg0 arg1 arg2 arg3)

lloo

macro

(lloo f arg0 arg1 arg2)

llood

macro

(llood f arg0 arg1 arg2 arg3)

llool

macro

(llool f arg0 arg1 arg2 arg3)

llooo

macro

(llooo f arg0 arg1 arg2 arg3)

lo

macro

(lo f arg0)

lod

macro

(lod f arg0 arg1)

lodd

macro

(lodd f arg0 arg1 arg2)

loddd

macro

(loddd f arg0 arg1 arg2 arg3)

loddl

macro

(loddl f arg0 arg1 arg2 arg3)

loddo

macro

(loddo f arg0 arg1 arg2 arg3)

lodl

macro

(lodl f arg0 arg1 arg2)

lodld

macro

(lodld f arg0 arg1 arg2 arg3)

lodll

macro

(lodll f arg0 arg1 arg2 arg3)

lodlo

macro

(lodlo f arg0 arg1 arg2 arg3)

lodo

macro

(lodo f arg0 arg1 arg2)

lodod

macro

(lodod f arg0 arg1 arg2 arg3)

lodol

macro

(lodol f arg0 arg1 arg2 arg3)

lodoo

macro

(lodoo f arg0 arg1 arg2 arg3)

lol

macro

(lol f arg0 arg1)

lold

macro

(lold f arg0 arg1 arg2)

loldd

macro

(loldd f arg0 arg1 arg2 arg3)

loldl

macro

(loldl f arg0 arg1 arg2 arg3)

loldo

macro

(loldo f arg0 arg1 arg2 arg3)

loll

macro

(loll f arg0 arg1 arg2)

lolld

macro

(lolld f arg0 arg1 arg2 arg3)

lolll

macro

(lolll f arg0 arg1 arg2 arg3)

lollo

macro

(lollo f arg0 arg1 arg2 arg3)

lolo

macro

(lolo f arg0 arg1 arg2)

lolod

macro

(lolod f arg0 arg1 arg2 arg3)

lolol

macro

(lolol f arg0 arg1 arg2 arg3)

loloo

macro

(loloo f arg0 arg1 arg2 arg3)

loo

macro

(loo f arg0 arg1)

lood

macro

(lood f arg0 arg1 arg2)

loodd

macro

(loodd f arg0 arg1 arg2 arg3)

loodl

macro

(loodl f arg0 arg1 arg2 arg3)

loodo

macro

(loodo f arg0 arg1 arg2 arg3)

lool

macro

(lool f arg0 arg1 arg2)

loold

macro

(loold f arg0 arg1 arg2 arg3)

looll

macro

(looll f arg0 arg1 arg2 arg3)

loolo

macro

(loolo f arg0 arg1 arg2 arg3)

looo

macro

(looo f arg0 arg1 arg2)

loood

macro

(loood f arg0 arg1 arg2 arg3)

loool

macro

(loool f arg0 arg1 arg2 arg3)

loooo

macro

(loooo f arg0 arg1 arg2 arg3)

od

macro

(od f arg0)

odd

macro

(odd f arg0 arg1)

oddd

macro

(oddd f arg0 arg1 arg2)

odddd

macro

(odddd f arg0 arg1 arg2 arg3)

odddl

macro

(odddl f arg0 arg1 arg2 arg3)

odddo

macro

(odddo f arg0 arg1 arg2 arg3)

oddl

macro

(oddl f arg0 arg1 arg2)

oddld

macro

(oddld f arg0 arg1 arg2 arg3)

oddll

macro

(oddll f arg0 arg1 arg2 arg3)

oddlo

macro

(oddlo f arg0 arg1 arg2 arg3)

oddo

macro

(oddo f arg0 arg1 arg2)

oddod

macro

(oddod f arg0 arg1 arg2 arg3)

oddol

macro

(oddol f arg0 arg1 arg2 arg3)

oddoo

macro

(oddoo f arg0 arg1 arg2 arg3)

odl

macro

(odl f arg0 arg1)

odld

macro

(odld f arg0 arg1 arg2)

odldd

macro

(odldd f arg0 arg1 arg2 arg3)

odldl

macro

(odldl f arg0 arg1 arg2 arg3)

odldo

macro

(odldo f arg0 arg1 arg2 arg3)

odll

macro

(odll f arg0 arg1 arg2)

odlld

macro

(odlld f arg0 arg1 arg2 arg3)

odlll

macro

(odlll f arg0 arg1 arg2 arg3)

odllo

macro

(odllo f arg0 arg1 arg2 arg3)

odlo

macro

(odlo f arg0 arg1 arg2)

odlod

macro

(odlod f arg0 arg1 arg2 arg3)

odlol

macro

(odlol f arg0 arg1 arg2 arg3)

odloo

macro

(odloo f arg0 arg1 arg2 arg3)

odo

macro

(odo f arg0 arg1)

odod

macro

(odod f arg0 arg1 arg2)

ododd

macro

(ododd f arg0 arg1 arg2 arg3)

ododl

macro

(ododl f arg0 arg1 arg2 arg3)

ododo

macro

(ododo f arg0 arg1 arg2 arg3)

odol

macro

(odol f arg0 arg1 arg2)

odold

macro

(odold f arg0 arg1 arg2 arg3)

odoll

macro

(odoll f arg0 arg1 arg2 arg3)

odolo

macro

(odolo f arg0 arg1 arg2 arg3)

odoo

macro

(odoo f arg0 arg1 arg2)

odood

macro

(odood f arg0 arg1 arg2 arg3)

odool

macro

(odool f arg0 arg1 arg2 arg3)

odooo

macro

(odooo f arg0 arg1 arg2 arg3)

ol

macro

(ol f arg0)

old

macro

(old f arg0 arg1)

oldd

macro

(oldd f arg0 arg1 arg2)

olddd

macro

(olddd f arg0 arg1 arg2 arg3)

olddl

macro

(olddl f arg0 arg1 arg2 arg3)

olddo

macro

(olddo f arg0 arg1 arg2 arg3)

oldl

macro

(oldl f arg0 arg1 arg2)

oldld

macro

(oldld f arg0 arg1 arg2 arg3)

oldll

macro

(oldll f arg0 arg1 arg2 arg3)

oldlo

macro

(oldlo f arg0 arg1 arg2 arg3)

oldo

macro

(oldo f arg0 arg1 arg2)

oldod

macro

(oldod f arg0 arg1 arg2 arg3)

oldol

macro

(oldol f arg0 arg1 arg2 arg3)

oldoo

macro

(oldoo f arg0 arg1 arg2 arg3)

oll

macro

(oll f arg0 arg1)

olld

macro

(olld f arg0 arg1 arg2)

olldd

macro

(olldd f arg0 arg1 arg2 arg3)

olldl

macro

(olldl f arg0 arg1 arg2 arg3)

olldo

macro

(olldo f arg0 arg1 arg2 arg3)

olll

macro

(olll f arg0 arg1 arg2)

ollld

macro

(ollld f arg0 arg1 arg2 arg3)

ollll

macro

(ollll f arg0 arg1 arg2 arg3)

olllo

macro

(olllo f arg0 arg1 arg2 arg3)

ollo

macro

(ollo f arg0 arg1 arg2)

ollod

macro

(ollod f arg0 arg1 arg2 arg3)

ollol

macro

(ollol f arg0 arg1 arg2 arg3)

olloo

macro

(olloo f arg0 arg1 arg2 arg3)

olo

macro

(olo f arg0 arg1)

olod

macro

(olod f arg0 arg1 arg2)

olodd

macro

(olodd f arg0 arg1 arg2 arg3)

olodl

macro

(olodl f arg0 arg1 arg2 arg3)

olodo

macro

(olodo f arg0 arg1 arg2 arg3)

olol

macro

(olol f arg0 arg1 arg2)

olold

macro

(olold f arg0 arg1 arg2 arg3)

ololl

macro

(ololl f arg0 arg1 arg2 arg3)

ololo

macro

(ololo f arg0 arg1 arg2 arg3)

oloo

macro

(oloo f arg0 arg1 arg2)

olood

macro

(olood f arg0 arg1 arg2 arg3)

olool

macro

(olool f arg0 arg1 arg2 arg3)

olooo

macro

(olooo f arg0 arg1 arg2 arg3)

ood

macro

(ood f arg0 arg1)

oodd

macro

(oodd f arg0 arg1 arg2)

ooddd

macro

(ooddd f arg0 arg1 arg2 arg3)

ooddl

macro

(ooddl f arg0 arg1 arg2 arg3)

ooddo

macro

(ooddo f arg0 arg1 arg2 arg3)

oodl

macro

(oodl f arg0 arg1 arg2)

oodld

macro

(oodld f arg0 arg1 arg2 arg3)

oodll

macro

(oodll f arg0 arg1 arg2 arg3)

oodlo

macro

(oodlo f arg0 arg1 arg2 arg3)

oodo

macro

(oodo f arg0 arg1 arg2)

oodod

macro

(oodod f arg0 arg1 arg2 arg3)

oodol

macro

(oodol f arg0 arg1 arg2 arg3)

oodoo

macro

(oodoo f arg0 arg1 arg2 arg3)

ool

macro

(ool f arg0 arg1)

oold

macro

(oold f arg0 arg1 arg2)

ooldd

macro

(ooldd f arg0 arg1 arg2 arg3)

ooldl

macro

(ooldl f arg0 arg1 arg2 arg3)

ooldo

macro

(ooldo f arg0 arg1 arg2 arg3)

ooll

macro

(ooll f arg0 arg1 arg2)

oolld

macro

(oolld f arg0 arg1 arg2 arg3)

oolll

macro

(oolll f arg0 arg1 arg2 arg3)

oollo

macro

(oollo f arg0 arg1 arg2 arg3)

oolo

macro

(oolo f arg0 arg1 arg2)

oolod

macro

(oolod f arg0 arg1 arg2 arg3)

oolol

macro

(oolol f arg0 arg1 arg2 arg3)

ooloo

macro

(ooloo f arg0 arg1 arg2 arg3)

oood

macro

(oood f arg0 arg1 arg2)

ooodd

macro

(ooodd f arg0 arg1 arg2 arg3)

ooodl

macro

(ooodl f arg0 arg1 arg2 arg3)

ooodo

macro

(ooodo f arg0 arg1 arg2 arg3)

oool

macro

(oool f arg0 arg1 arg2)

ooold

macro

(ooold f arg0 arg1 arg2 arg3)

oooll

macro

(oooll f arg0 arg1 arg2 arg3)

ooolo

macro

(ooolo f arg0 arg1 arg2 arg3)

ooood

macro

(ooood f arg0 arg1 arg2 arg3)

ooool

macro

(ooool f arg0 arg1 arg2 arg3)
================================================ FILE: docs/ham-fisted.process.html ================================================ ham-fisted.process documentation

ham-fisted.process

destroy-forcibly!

(destroy-forcibly! proc-hdl)

Destroy the process handle's process forcibly.

launch

(launch cmd-line)(launch cmd-line {:keys [stdout-hdlr stderr-hdlr print-cmd-line?], :or {print-cmd-line? true}})

Launch a proccess.

  • cmd-line string command line.
  • stdout-hdrl, stderr-hdlr - transduce-style rf functions that receive each string read from stdout and stderr respectively.

Returns {:keys [^java.lang.ProcessHandle proc-hdl wait-or-kill]}:

  • proc-hdl - java.lang.ProcessHandle
  • wait-or-kill - function that has two arities:
    1. (proc) - kill the process returning any output as {:out :err}.
    2. (proc time-ms timeout-symbol) - wait specified time for process to terminate returning either the timeout symbol or {:out :err}.

Example:

ham-fisted.process> (launch "ls -al" {:print-cmd-line? false})
  {:proc-hdl #object[java.lang.ProcessHandleImpl 0x7f8e8742 "31019"],
  :wait-or-kill #function[ham-fisted.process/launch/wait-or-kill--65545]}
  ...

ham-fisted.process> (def result ((:wait-or-kill *1)))
#'ham-fisted.process/result
ham-fisted.process> (keys result)
(:out :err)

launch-jvm

(launch-jvm cmd-name {:keys [xmx jvm-opts], :as args})

Assumes a jvm process launched from a shell command. Will hang looking for first process descendant.

If shell command then arguments other than :xmx :jvm-opts may need to have quoted strings if they are being passed the clojure process.

Example:

ham-fisted.process> (launch-jvm "clojure" {:jvm-opts ["-A:dev" "-X" "ham-fisted.protocol-perf/-main"]})
launch-process: clojure -A:dev -X ham-fisted.protocol-perf/-main
{:proc-hdl #object[java.lang.ProcessHandleImpl 0x4f4a8dc7 "31194"],
 :wait-or-kill #function[ham-fisted.process/launch/wait-or-kill--10016],
 :jvm-pid 31199,
  :jvm-proc #object[java.lang.ProcessHandleImpl 0x2e52e7c8 "31199"]}
...
ham-fisted.process> (def result ((:wait-or-kill *1)))
#'ham-fisted.process/result

out-rf

(out-rf print?)

println-rf

Print process output using println. Example process output handler. Returns total output as a string to when finalized.

process-descendants

(process-descendants proc-hdl)

Get the first descendants of a process handle.

quiet-rf

Returns total output as a string to when finalized.

record-rf

Record all the strings and save them to a vector

sh

(sh cmd-line)(sh cmd-line {:keys [timeout-ms], :or {timeout-ms Integer/MAX_VALUE}, :as opts})

stream->strings

(stream->strings input)(stream->strings input bufsize charset)
================================================ FILE: docs/ham-fisted.profile.html ================================================ ham-fisted.profile documentation

ham-fisted.profile

current-times

(current-times)

Get the current time map

reset-times!

(reset-times!)

Clear times out of current time map

time-ms

macro

(time-ms kw & code)

Time an operation returning the results. Puts time in double milliseconds into the time map.

with-times

macro

(with-times & code)

Returns {:result :times}. Run a block of code with time map bound to a new map.

================================================ FILE: docs/ham-fisted.protocols.html ================================================ ham-fisted.protocols documentation

ham-fisted.protocols

->collection

(->collection item)

->init-val-fn

(->init-val-fn item)

Returns the initial values for a parallel reduction. This function takes no arguments and returns the initial accumulator.

->iterable

(->iterable item)

->merge-fn

(->merge-fn item)

Returns the merge function for a parallel reduction. This function takes two accumulators and returns a or modified accumulator.

->rfn

(->rfn item)

Returns the reduction function for a parallel reduction. This function takes two arguments, the accumulator and a value from the collection and returns a new or modified accumulator.

->spliterator

(->spliterator m)

add-fn

(add-fn l)

BitSet

protocol

Protocol for efficiently dealing with bitsets

bitset?

(bitset? item)

BulkSetOps

protocol

cardinality

(cardinality item)

Some sets don't work with clojure's count function.

contained-datatype

(contained-datatype o)

Datatype of contained datatype - may be nil if not a container

ContainedDatatype

protocol

contains-fn

(contains-fn item)

Return an efficient function for deciding if this set contains a single item.

contains-range?

(contains-range? item sidx eidx)

convertible-to-collection?

(convertible-to-collection? item)

convertible-to-iterable?

(convertible-to-iterable? item)

count

(count m)

Counted

protocol

Datatype

protocol

datatype

(datatype o)

Returns the datatype :int8, :int16, etc -- if known -- else the type can be assumed to be an object type. The return value may not be a keyword but it must be comparable with identical?

difference

(difference l r)

estimate-count

(estimate-count m)

EstimateCount

protocol

Finalize

protocol

Generic protocol for things that finalize results of reductions. Defaults to deref of instance of IDeref or identity.

finalize

(finalize this val)

intersection

(intersection l r)

intersects-range?

(intersects-range? item sidx eidx)

managed-blocker

(managed-blocker m)

ManagedBlocker

protocol

max-set-value

(max-set-value item)

min-set-value

(min-set-value item)

PAdd

protocol

Define a function to mutably add items to a collection. This function must return the collection -- it must be useable in a reduce as the rf.

ParallelReducer

protocol

Parallel reducers are simple a single object that you can pass into preduce as opposed to 3 separate functions.

ParallelReduction

protocol

Protocol to define a parallel reduction in a collection-specific pathway. Specializations are in impl as that is where the parallelization routines are found.

preduce

(preduce coll init-val-fn rfn merge-fn options)

Container-specific parallelized reduction. Reductions must respect the pool passed in via the options.

reduce-intersection

(reduce-intersection l data)

reduce-union

(reduce-union l data)

Reducer

protocol

Reducer is the basic reduction abstraction as a single object.

reducible?

(reducible? coll)

Reduction

protocol

Faster check than satisfies? to see if something is reducible

returned-datatype

(returned-datatype o)

ReturnedDatatype

protocol

serialize->bytes

(serialize->bytes o)

SerializeObjBytes

protocol

set?

(set? l)

SetOps

protocol

Simple protocol for set operations to make them uniformly extensible to new objects.

simplified-contained-datatype

(simplified-contained-datatype o)

Exactly :int64 :float64 :object or nil

simplified-datatype

(simplified-datatype o)

Returns exactly :int64, :float64, or :object

simplified-returned-datatype

(simplified-returned-datatype o)

Split

protocol

split

(split m)

ToCollection

protocol

ToIterable

protocol

ToSpliterator

protocol

union

(union l r)

wrap-array

(wrap-array ary)

wrap-array-growable

(wrap-array-growable ary ptr)

WrapArray

protocol

xor

(xor l r)
================================================ FILE: docs/ham-fisted.reduce.html ================================================ ham-fisted.reduce documentation

ham-fisted.reduce

Protocol-based parallel reduction architecture and helper functions.

->consumer

(->consumer cfn)

Return an instance of a consumer, double consumer, or long consumer.

bind-double-consumer-reducer!

(bind-double-consumer-reducer! cls-type ctor)(bind-double-consumer-reducer! ctor)

Bind a classtype as a double consumer parallel reducer - the consumer must implement DoubleConsumer, ham_fisted.Reducible, and IDeref.

compose-reducers

(compose-reducers reducers)(compose-reducers options reducers)

Given a map or sequence of reducers return a new reducer that produces a map or vector of results.

If data is a sequence then context is guaranteed to be an object array.

Options:

  • :rfn-datatype - One of nil, :int64, or :float64. This indicates that the rfn's should all be uniform as accepting longs, doubles, or generically objects. Defaults to nil.

consume!

(consume! consumer coll)

Consumer a collection. This is simply a reduction where the return value is ignored.

Returns the consumer.

consumer-accumulator

Generic reduction function using a consumer

consumer-preducer

(consumer-preducer constructor)

Bind a consumer as a parallel reducer.

Consumer must implement java.util.function.Consumer, ham_fisted.Reducible and clojure.lang.IDeref.

Returns instance of type bound.

See documentation for declare-double-consumer-preducer!.

consumer-reducer

(consumer-reducer ctor)

Make a parallel double consumer reducer given a function that takes no arguments and is guaranteed to produce a double consumer which also implements Reducible and IDeref

double-accumulator

macro

(double-accumulator accvar varvar & code)

Type-hinted double reduction accumulator. consumer:

  ham-fisted.api> (reduce (double-accumulator acc v (+ (double acc) v))
                             0.0
                             (range 1000))
#<SimpleSum@2fbcf20: 499500.0>
ham-fisted.api> @*1
499500.0

double-consumer-accumulator

Converts from a double consumer to a double reduction accumulator that returns the consumer:

ham-fisted.api> (reduce double-consumer-accumulator
                             (Sum$SimpleSum.)
                             (range 1000))
#<SimpleSum@2fbcf20: 499500.0>
ham-fisted.api> @*1
499500.0

double-consumer-preducer

(double-consumer-preducer constructor)

Return a preducer for a double consumer.

Consumer must implement java.util.function.DoubleConsumer, ham_fisted.Reducible and clojure.lang.IDeref.

user> (require '[ham-fisted.api :as hamf])
nil
user> (import '[java.util.function DoubleConsumer])
java.util.function.DoubleConsumer
user> (import [ham_fisted Reducible])
ham_fisted.Reducible
user> (import '[clojure.lang IDeref])
clojure.lang.IDeref
user> (deftype MeanR [^{:unsynchronized-mutable true :tag 'double} sum
                      ^{:unsynchronized-mutable true :tag 'long} n-elems]
        DoubleConsumer
        (accept [this v] (set! sum (+ sum v)) (set! n-elems (unchecked-inc n-elems)))
        Reducible
        (reduce [this o]
          (set! sum (+ sum (.-sum ^MeanR o)))
          (set! n-elems (+ n-elems (.-n-elems ^MeanR o)))
          this)
        IDeref (deref [this] (/ sum n-elems)))
user.MeanR
user> (hamf/declare-double-consumer-preducer! MeanR (MeanR. 0 0))
nil
  user> (hamf/preduce-reducer (double-consumer-preducer #(MeanR. 0 0)) (hamf/range 200000))
99999.5

double-consumer-reducer

(double-consumer-reducer ctor)

Make a parallel double consumer reducer given a function that takes no arguments and is guaranteed to produce a double consumer which also implements Reducible and IDeref

immut-map-kv

(immut-map-kv keyfn valfn data)(immut-map-kv ks vs)

indexed-accum

macro

(indexed-accum accvar idxvar varvar & code)

Create an indexed accumulator that recieves and additional long index during a reduction:

ham-fisted.api> (reduce (indexed-accum
                         acc idx v (conj acc [idx v]))
                        []
                        (range 5))
[[0 0] [1 1] [2 2] [3 3] [4 4]]

indexed-double-accum

macro

(indexed-double-accum accvar idxvar varvar & code)

Create an indexed double accumulator that recieves and additional long index during a reduction:

ham-fisted.api> (reduce (indexed-double-accum
                         acc idx v (conj acc [idx v]))
                        []
                        (range 5))
[[0 0.0] [1 1.0] [2 2.0] [3 3.0] [4 4.0]]

indexed-long-accum

macro

(indexed-long-accum accvar idxvar varvar & code)

Create an indexed long accumulator that recieves and additional long index during a reduction:

ham-fisted.api> (reduce (indexed-long-accum
                         acc idx v (conj acc [idx v]))
                        []
                        (range 5))
[[0 0] [1 1] [2 2] [3 3] [4 4]]

long-accumulator

macro

(long-accumulator accvar varvar & code)

Type-hinted double reduction accumulator. consumer:

  ham-fisted.api> (reduce (double-accumulator acc v (+ (double acc) v))
                             0.0
                             (range 1000))
#<SimpleSum@2fbcf20: 499500.0>
ham-fisted.api> @*1
499500.0

long-consumer-accumulator

Converts from a long consumer to a long reduction accumulator that returns the consumer:

ham-fisted.api> (reduce double-consumer-accumulator
                             (Sum$SimpleSum.)
                             (range 1000))
#<SimpleSum@2fbcf20: 499500.0>
ham-fisted.api> @*1
499500.0

long-consumer-reducer

(long-consumer-reducer ctor)

Make a parallel double consumer reducer given a function that takes no arguments and is guaranteed to produce a double consumer which also implements Reducible and IDeref

options->parallel-options

(options->parallel-options options)

Convert an options map to a parallel options object.

Options:

  • :pool - supply the forkjoinpool to use.
  • :max-batch-size - Defaults to 64000, used for index-based parallel pathways to control the number size of each parallelized batch.
  • :ordered? - When true process inputs and provide results in order.
  • :parallelism - The amount of parallelism to expect. Defaults to the number of threads in the fork-join pool provided.
  • :cat-parallelism - Either :seq-wise or :elem-wise - when parallelizing over a concatenation of containers either parallelize each container meaning call preduce on each container using many threads per container or use one thread per container - seq-wise. Defaults to seq-wise as this doesn't require each container itself to support parallelization but relies on the sequence of containers to be long enough to saturate the processor. Can also be set at time of container construction - see lazy-noncaching/concat-opts.
  • :put-timeout-ms - The time to wait to put data into the queue. This is a safety mechanism so that if the processing system fails we don't just keep putting things into a queue.
  • :unmerged-result? - Use with care. For parallel reductions do not perform the merge step but instead return the sequence of partially reduced results.
  • :n-lookahead - How for to look ahead for pmap and upmap to add new jobs to the queue. Defaults to `(* 2 parallelism).

parallel-reducer

(parallel-reducer init-fn rfn merge-fn fin-fn)(parallel-reducer init-fn rfn merge-fn)

Implement a parallel reducer by explicitly passing in the various required functions.

  • 'init-fn' - Takes no argumenst and returns a new accumulation target.
  • 'rfn' - clojure rf function - takes two arguments, the accumulation target and a new value and produces a new accumulation target.
  • 'merge-fn' - Given two accumulation targets returns a new combined accumulation target.
  • 'fin-fn' - optional - Given an accumulation target returns the desired final type.
user> (hamf-rf/preduce-reducer
       (hamf-rf/parallel-reducer
        hamf/mut-set
        #(do (.add ^java.util.Set %1 %2) %1)
        hamf/union
        hamf/sort)
       (lznc/map (fn ^long [^long v] (rem v 13)) (hamf/range 1000000)))
[0 1 2 3 4 5 6 7 8 9 10 11 12]

preduce

(preduce init-val-fn rfn merge-fn coll)(preduce init-val-fn rfn merge-fn options coll)

Parallelized reduction. Currently coll must either be random access or a lznc map/filter chain based on one or more random access entities, hashmaps and sets from this library or any java.util set, hashmap or concurrent versions of these. If input cannot be parallelized this lowers to a normal serial reduction.

For potentially small-n invocations providing the parallel options explicitly will improve performance surprisingly - converting the options map to the parallel options object takes a bit of time.

  • init-val-fn - Potentially called in reduction threads to produce each initial value.
  • rfn - normal clojure reduction function. Typehinting the second argument to double or long will sometimes produce a faster reduction.
  • merge-fn - Merge two reduction results into one.

Options:

  • :pool - The fork-join pool to use. Defaults to common pool which assumes reduction is cpu-bound.
  • :parallelism - What parallelism to use - defaults to pool's getParallelism method.
  • :max-batch-size - Rough maximum batch size for indexed or grouped reductions. This can both even out batch times and ensure you don't get into safepoint trouble with jdk-8.
  • :min-n - minimum number of elements before initiating a parallelized reduction - Defaults to 1000 but you should customize this particular to your specific reduction.
  • :ordered? - True if results should be in order. Unordered results sometimes are slightly faster but again you should test for your specific situation..
  • :cat-parallelism - Either :seq-wise or :elem-wise, defaults to :seq-wise. Test for your specific situation, this really is data-dependent. This contols how a concat primitive parallelizes the reduction across its contains. Elemwise means each container's reduction is individually parallelized while seqwise indicates to do a pmap style initial reduction across containers then merge the results.
  • :put-timeout-ms - Number of milliseconds to wait for queue space before throwing an exception in unordered reductions. Defaults to 50000.
  • :unmerged-result? - Defaults to false. When true, the sequence of results be returned directly without any merge steps in a lazy-noncaching container. Beware the noncaching aspect -- repeatedly evaluating this result may kick off the parallelized reduction multiple times. To ensure caching if unsure call seq on the result.

preduce-reducer

(preduce-reducer reducer options coll)(preduce-reducer reducer coll)

Given an instance of ham-fisted.protocols/ParallelReducer, perform a parallel reduction.

In the case where the result is requested unmerged then finalize will be called on each result in a lazy noncaching way. In this case you can use a non-parallelized reducer and simply get a sequence of results as opposed to one.

  • reducer - instance of ParallelReducer
  • options - Same options as preduce.
  • coll - something potentially with a parallelizable reduction.

See options for ham-fisted.reduce/preduce.

Additional Options:

  • :skip-finalize? - when true, the reducer's finalize method is not called on the result.

preduce-reducers

(preduce-reducers reducers options coll)(preduce-reducers reducers coll)

Given a map or sequence of ham-fisted.protocols/ParallelReducer, produce a map or sequence of reduced values. Reduces over input coll once in parallel if coll is large enough. See options for ham-fisted.reduce/preduce.

ham-fisted.api> (preduce-reducers {:sum (Sum.) :mult *} (range 20))
{:mult 0, :sum #<Sum@5082c3b7: {:sum 190.0, :n-elems 20}>}

reduce-reducer

(reduce-reducer reducer coll)

Serially reduce a reducer.

ham-fisted.api> (reduce-reducer (Sum.) (range 1000))
#<Sum@afbedb: {:sum 499500.0, :n-elems 1000}>

reduce-reducers

(reduce-reducers reducers coll)

Serially reduce a map or sequence of reducers into a map or sequence of results.

ham-fisted.api> (reduce-reducers {:a (Sum.) :b *} (range 1 21))
{:b 2432902008176640000, :a #<Sum@6bcebeb1: {:sum 210.0, :n-elems 20}>}

reducer->completef

(reducer->completef reducer)

Return fold-compatible pair of reducef, completef given a parallel reducer. Note that folded reducers are not finalized as of this time:

ham-fisted.api> (def data (vec (range 200000)))
#'ham-fisted.api/data
ham-fisted.api> (r/fold (reducer->completef (Sum.)) (reducer->rfn (Sum.)) data)
#<Sum@858c206: {:sum 1.99999E10, :n-elems 200000}>

reducer->rf

(reducer->rf reducer)

Given a reducer, return a transduce-compatible rf -

ham-fisted.api> (transduce (clojure.core/map #(+ % 2)) (reducer->rf (Sum.)) (range 200))
{:sum 20300.0, :n-elems 200}

reducer-with-finalize

(reducer-with-finalize reducer fin-fn)

reducer-xform->reducer

(reducer-xform->reducer reducer xform)

Given a reducer and a transducer xform produce a new reducer which will apply the transducer pipeline before is reduction function.

ham-fisted.api> (reduce-reducer (reducer-xform->reducer (Sum.) (clojure.core/filter even?))
                                (range 1000))
#<Sum@479456: {:sum 249500.0, :n-elems 500}>

!! - If you use a stateful transducer here then you must not use the reducer in a parallelized reduction.

reducible-merge

Parallel reduction merge function that expects both sides to be an instances of Reducible

================================================ FILE: docs/ham-fisted.set.html ================================================ ham-fisted.set documentation

ham-fisted.set

->integer-random-access

(->integer-random-access s)

Given a set (or bitset), return a efficient, sorted random access structure. This assumes the set contains integers.

bitset

(bitset)(bitset data)(bitset start end)

Create a java.util.Bitset. The two argument version assumes you are passing in the start, end of a monotonically incrementing range.

bitset?

(bitset? s)

Return true if this is a bitset

cardinality

(cardinality s)

Return the cardinality (size) of a given set.

contains-fn

(contains-fn s)

Return an IFn that returns efficiently returns true if the set contains a given element.

contains-range?

(contains-range? s sidx eidx)

bitset-specific query that returns true if the set contains all the integers from sidx to eidx non-inclusive.

difference

(difference l r)

set difference

intersection

(intersection)(intersection l)(intersection l r)

set intersection

intersects-range?

(intersects-range? s sidx eidx)

bitset-specific query that returns true if the set contains any the integers from sidx to eidx non-inclusive.

java-concurrent-hashset

(java-concurrent-hashset)(java-concurrent-hashset data)

Create a concurrent hashset.

java-hashset

(java-hashset)(java-hashset data)

Return a java hashset

map-invert

(map-invert m)

invert a map such that the keys are the vals and the vals are the keys

max-set-value

(max-set-value s)

Given a bitset, return the maximum set value. Errors if the bitset is empty.

min-set-value

(min-set-value s)

Given a bitset, return the minimum set value. Errors if the bitset is empty.

mut-set

(mut-set)(mut-set data)

Return a mutable set.

reduce-intersection

(reduce-intersection)(reduce-intersection l)(reduce-intersection l & data)

reduce-union

(reduce-union)(reduce-union l)(reduce-union l & data)

Reduce a number of objects into one object via union

set

(set)(set data)

Return an immutable set

set?

(set? data)

union

(union l r)

set union

unique

(unique options data)(unique data)

Create a set of unique items. Parallelized and non-lazy.

See options for unique-reducer and ham-fisted.api/preduce-reducer.

unique-reducer

(unique-reducer options)

Create a parallel reducer that creates a set.

Options:

xor

(xor l r)

set xor - difference of intersection from union

================================================ FILE: docs/ham-fisted.spliterator.html ================================================ ham-fisted.spliterator documentation

ham-fisted.spliterator

Support for spliterator reduction and parallel reduction.

->spliterator

(->spliterator ii)

deref-consumer

macro

(deref-consumer varname accept-code deref-code)

Create a Consumer with support for IDeref

deref-double-consumer

macro

(deref-double-consumer varname accept-code deref-code)

Create a DoubleConsumer with support for IDeref

deref-long-consumer

macro

(deref-long-consumer varname accept-code deref-code)

Create a LongConsumer with support for IDeref

elements

(elements data)

Return all the elements referenced by this spliterator as a persistent list

psum

(psum vv)

spliterator based parallel summation - does not account for Double/NaN

split-parallel-reduce

(split-parallel-reduce executor-service split ideal-split init-fn rfn merge-fn)

Perform a parallel reduction of a spliterator using the provided ExecutorService

split-reduce

(split-reduce rfn split)(split-reduce rfn acc split)

Reduce over a spliterator. Special support exists for IFn$LLL and IFn$DDD

split-to-max-size

(split-to-max-size split max-size op)

sum-fast

(sum-fast vv)

spliterator based serial summation

================================================ FILE: docs/highlight/solarized-light.css ================================================ /* Orginal Style from ethanschoonover.com/solarized (c) Jeremy Hull */ .hljs { display: block; overflow-x: auto; padding: 0.5em; background: #fdf6e3; color: #657b83; } .hljs-comment, .hljs-quote { color: #93a1a1; } /* Solarized Green */ .hljs-keyword, .hljs-selector-tag, .hljs-addition { color: #859900; } /* Solarized Cyan */ .hljs-number, .hljs-string, .hljs-meta .hljs-meta-string, .hljs-literal, .hljs-doctag, .hljs-regexp { color: #2aa198; } /* Solarized Blue */ .hljs-title, .hljs-section, .hljs-name, .hljs-selector-id, .hljs-selector-class { color: #268bd2; } /* Solarized Yellow */ .hljs-attribute, .hljs-attr, .hljs-variable, .hljs-template-variable, .hljs-class .hljs-title, .hljs-type { color: #b58900; } /* Solarized Orange */ .hljs-symbol, .hljs-bullet, .hljs-subst, .hljs-meta, .hljs-meta .hljs-keyword, .hljs-selector-attr, .hljs-selector-pseudo, .hljs-link { color: #cb4b16; } /* Solarized Red */ .hljs-built_in, .hljs-deletion { color: #dc322f; } .hljs-formula { background: #eee8d5; } .hljs-emphasis { font-style: italic; } .hljs-strong { font-weight: bold; } ================================================ FILE: docs/index.html ================================================ Ham-Fisted 3.029

Ham-Fisted 3.029

High Performance Clojure Primitives.

Topics

Namespaces

ham-fisted.api

Fast mutable and immutable associative data structures based on bitmap trie hashmaps. Mutable pathways implement the java.util.Map or Set interfaces including in-place update features such as compute or computeIfPresent.

Public variables and functions:

ham-fisted.fjp

Support for java.util.concurrent.ForkJoinPool-specific operations such as managed block and task fork/join. Additionally supports concept of 'exception-safe' via wrapping executing code and unwrapper result post execution in order avoid wrapping exceptions and breaking calling code that may be expecting specific exception types.

ham-fisted.hlet

Extensible let to allow efficient typed destructuring.

Public variables and functions:

ham-fisted.mut-map

Functions for working with java's mutable map interface

ham-fisted.primitive-invoke

For statically traced calls the Clojure compiler calls the primitive version of type-hinted functions and this makes quite a difference in tight loops. Often times, however, functions are passed by values or returned from if-statements and then you need to explicitly call the primitive overload - this makes that pathway less verbose. Functions must first be check-casted to their primitive types and then calling them will use their primitive overloads avoiding all casting.

Public variables and functions:

================================================ FILE: docs/js/page_effects.js ================================================ function visibleInParent(element) { var position = $(element).position().top return position > -50 && position < ($(element).offsetParent().height() - 50) } function hasFragment(link, fragment) { return $(link).attr("href").indexOf("#" + fragment) != -1 } function findLinkByFragment(elements, fragment) { return $(elements).filter(function(i, e) { return hasFragment(e, fragment)}).first() } function scrollToCurrentVarLink(elements) { var elements = $(elements); var parent = elements.offsetParent(); if (elements.length == 0) return; var top = elements.first().position().top; var bottom = elements.last().position().top + elements.last().height(); if (top >= 0 && bottom <= parent.height()) return; if (top < 0) { parent.scrollTop(parent.scrollTop() + top); } else if (bottom > parent.height()) { parent.scrollTop(parent.scrollTop() + bottom - parent.height()); } } function setCurrentVarLink() { $('.secondary a').parent().removeClass('current') $('.anchor'). filter(function(index) { return visibleInParent(this) }). each(function(index, element) { findLinkByFragment(".secondary a", element.id). parent(). addClass('current') }); scrollToCurrentVarLink('.secondary .current'); } var hasStorage = (function() { try { return localStorage.getItem } catch(e) {} }()) function scrollPositionId(element) { var directory = window.location.href.replace(/[^\/]+\.html$/, '') return 'scroll::' + $(element).attr('id') + '::' + directory } function storeScrollPosition(element) { if (!hasStorage) return; localStorage.setItem(scrollPositionId(element) + "::x", $(element).scrollLeft()) localStorage.setItem(scrollPositionId(element) + "::y", $(element).scrollTop()) } function recallScrollPosition(element) { if (!hasStorage) return; $(element).scrollLeft(localStorage.getItem(scrollPositionId(element) + "::x")) $(element).scrollTop(localStorage.getItem(scrollPositionId(element) + "::y")) } function persistScrollPosition(element) { recallScrollPosition(element) $(element).scroll(function() { storeScrollPosition(element) }) } function sidebarContentWidth(element) { var widths = $(element).find('.inner').map(function() { return $(this).innerWidth() }) return Math.max.apply(Math, widths) } function calculateSize(width, snap, margin, minimum) { if (width == 0) { return 0 } else { return Math.max(minimum, (Math.ceil(width / snap) * snap) + (margin * 2)) } } function resizeSidebars() { var primaryWidth = sidebarContentWidth('.primary') var secondaryWidth = 0 if ($('.secondary').length != 0) { secondaryWidth = sidebarContentWidth('.secondary') } // snap to grid primaryWidth = calculateSize(primaryWidth, 32, 13, 160) secondaryWidth = calculateSize(secondaryWidth, 32, 13, 160) $('.primary').css('width', primaryWidth) $('.secondary').css('width', secondaryWidth).css('left', primaryWidth + 1) if (secondaryWidth > 0) { $('#content').css('left', primaryWidth + secondaryWidth + 2) } else { $('#content').css('left', primaryWidth + 1) } } $(window).ready(resizeSidebars) $(window).ready(setCurrentVarLink) $(window).ready(function() { persistScrollPosition('.primary')}) $(window).ready(function() { $('#content').scroll(setCurrentVarLink) $(window).resize(setCurrentVarLink) $(window).resize(resizeSidebars) }) ================================================ FILE: java/ham_fisted/ArrayHelpers.java ================================================ package ham_fisted; public class ArrayHelpers { public static Object checkedAget(Object[] data, int nelems, int idx) { ArrayLists.checkIndex(idx, nelems); return data[idx]; } public static void aset(boolean[] data, int idx, boolean val) { data[idx] = val; } public static void aset(byte[] data, int idx, byte val) { data[idx] = val; } public static void aset(short[] data, int idx, short val) { data[idx] = val; } public static void aset(char[] data, int idx, char val) { data[idx] = val; } public static void aset(int[] data, int idx, int val) { data[idx] = val; } public static void aset(long[] data, int idx, long val) { data[idx] = val; } public static void aset(float[] data, int idx, float val) { data[idx] = val; } public static void aset(double[] data, int idx, double val) { data[idx] = val; } public static void aset(Object[] data, int idx, Object val) { data[idx] = val; } public static void accumPlus(byte[] data, int idx, byte val) { data[idx] += val; } public static void accumPlus(short[] data, int idx, short val) { data[idx] += val; } public static void accumPlus(char[] data, int idx, char val) { data[idx] += val; } public static void accumPlus(int[] data, int idx, int val) { data[idx] += val; } public static void accumPlus(long[] data, int idx, long val) { data[idx] += val; } public static void accumPlus(float[] data, int idx, float val) { data[idx] += val; } public static void accumPlus(double[] data, int idx, double val) { data[idx] += val; } public static void accumMul(byte[] data, int idx, byte val) { data[idx] *= val; } public static void accumMul(short[] data, int idx, short val) { data[idx] *= val; } public static void accumMul(char[] data, int idx, char val) { data[idx] *= val; } public static void accumMul(int[] data, int idx, int val) { data[idx] *= val; } public static void accumMul(long[] data, int idx, long val) { data[idx] *= val; } public static void accumMul(float[] data, int idx, float val) { data[idx] *= val; } public static void accumMul(double[] data, int idx, double val) { data[idx] *= val; } //For some sizes 4 or less manual copy is faster than system.arraycopy. //Interestingly enough, this differs depending on of this is an object array //or a primitive array with the breakeven for an object array being around 8 public static void manualCopy(Object[] src, int soff, Object[] dst, int doff, int len) { int send = soff + len; for (; soff < send; ++soff, ++doff) dst[doff] = src[soff]; } public static void manualCopy(byte[] src, int soff, byte[] dst, int doff, int len) { int send = soff + len; for (; soff < send; ++soff, ++doff) dst[doff] = src[soff]; } public static void manualCopy(short[] src, int soff, short[] dst, int doff, int len) { int send = soff + len; for (; soff < send; ++soff, ++doff) dst[doff] = src[soff]; } public static void manualCopy(char[] src, int soff, char[] dst, int doff, int len) { int send = soff + len; for (; soff < send; ++soff, ++doff) dst[doff] = src[soff]; } public static void manualCopy(int[] src, int soff, int[] dst, int doff, int len) { int send = soff + len; for (; soff < send; ++soff, ++doff) dst[doff] = src[soff]; } public static void manualCopy(long[] src, int soff, long[] dst, int doff, int len) { int send = soff + len; for (; soff < send; ++soff, ++doff) dst[doff] = src[soff]; } public static void manualCopy(float[] src, int soff, float[] dst, int doff, int len) { int send = soff + len; for (; soff < send; ++soff, ++doff) dst[doff] = src[soff]; } public static void manualCopy(double[] src, int soff, double[] dst, int doff, int len) { int send = soff + len; for (; soff < send; ++soff, ++doff) dst[doff] = src[soff]; } } ================================================ FILE: java/ham_fisted/ArrayImmutList.java ================================================ package ham_fisted; import static ham_fisted.ChunkedList.*; import static ham_fisted.HashProviders.*; import java.util.List; import java.util.RandomAccess; import java.util.Iterator; import java.util.ListIterator; import java.util.Collection; import java.util.Arrays; import java.util.Objects; import java.util.NoSuchElementException; import java.util.function.Function; import java.util.function.BiFunction; import clojure.lang.Indexed; import clojure.lang.IReduce; import clojure.lang.IKVReduce; import clojure.lang.IHashEq; import clojure.lang.Seqable; import clojure.lang.Reversible; import clojure.lang.IPersistentMap; import clojure.lang.IPersistentVector; import clojure.lang.IObj; import clojure.lang.IEditableCollection; import clojure.lang.Murmur3; import clojure.lang.Util; import clojure.lang.RT; import clojure.lang.IDeref; import clojure.lang.IFn; import clojure.lang.IteratorSeq; import clojure.lang.ISeq; import clojure.lang.MapEntry; import clojure.lang.IMapEntry; import clojure.lang.ITransientVector; import clojure.lang.APersistentVector; public class ArrayImmutList extends APersistentVector implements IMutList, RandomAccess, Indexed, IReduce, IKVReduce, IHashEq, Seqable, Reversible, ChunkedListOwner, IPersistentVector, IObj, IEditableCollection, UpdateValues, ArrayLists.ArrayOwner { final Object[] data; public final int startidx; public final int nElems; public final IPersistentMap m; int _hash = 0; public static final ArrayImmutList EMPTY = new ArrayImmutList(new Object[0], 0,0,null); public ArrayImmutList(Object[] d, int sidx, int eidx, IPersistentMap meta ) { data = d; startidx = sidx; nElems = eidx - sidx; m = meta; } public static ArrayImmutList create(boolean owning, Object[] d, int sidx, int eidx, IPersistentMap meta) { d = owning ? d : d.clone(); return new ArrayImmutList(d, sidx, eidx, meta); } public static IPersistentVector create(boolean owning, IPersistentMap meta, Object... data) { return create(owning, data, 0, data.length, meta); } final int indexCheck(int idx) { return ChunkedList.indexCheck(startidx, nElems, idx); } final int wrapIndexCheck(int idx) { return ChunkedList.wrapIndexCheck(startidx, nElems, idx); } public ChunkedListSection getChunkedList() { return new ChunkedListSection(ChunkedList.create(true, null, data).data, startidx, startidx+nElems); } public final int hashCode() { if (_hash == 0) { final int ne = nElems; final int eidx = startidx + ne; int hash = 1; final Object[] mdata = data; for(int sidx = startidx; sidx < eidx; ++sidx) { hash = 31 * hash + Util.hasheq(mdata[sidx]); } _hash = hash = Murmur3.mixCollHash(hash, ne); } return _hash; } public final int hasheq() { return hashCode(); } public final boolean equiv(HashProvider hp, Object other) { if(other == this) return true; if(other == null) return false; if(other instanceof ArrayImmutList) { final int ne = nElems; final ArrayImmutList olist = (ArrayImmutList)other; if(olist.nElems != ne) return false; final int sidx = startidx; final int osidx = olist.startidx; final Object[] mdata = data; final Object[] odata = olist.data; for(int idx = 0; idx < ne; ++idx) if(!hp.equals(mdata[idx+sidx], odata[idx+osidx])) return false; return true; } else { return CljHash.listEquiv(this, other); } } public final boolean equiv(Object other) { return equiv(defaultHashProvider, other); } public final boolean equals(Object other) { return equiv(other); } public final String toString() { return Transformables.sequenceToString(this); } public final IPersistentMap meta() { return m; } public final ArrayImmutList withMeta(IPersistentMap m) { return new ArrayImmutList(data, startidx, startidx+nElems, m); } public void clear() { throw new RuntimeException("Unimplemented"); } public boolean remove(Object c) { throw new RuntimeException("Unimplemented"); } public Character remove(int idx) { throw new RuntimeException("Unimplemented"); } public boolean add(Object c) { throw new RuntimeException("Unimplemented"); } public void add(int idx, Object c) { throw new RuntimeException("Unimplemented"); } public boolean addAll(int idx, Collection c) { throw new RuntimeException("Unimplemented"); } public Character set(int idx, Object c) { throw new RuntimeException("Unimplemented"); } public boolean retainAll(Collection c) { throw new RuntimeException("Unimplemented"); } public boolean removeAll(Collection c) { throw new RuntimeException("Unimplemented"); } public boolean addAll(Collection c) { throw new RuntimeException("Unimplemented"); } public Object get(int idx) { return data[ChunkedList.indexCheck(startidx, nElems, idx)]; } public final int indexOf(Object obj) { final int ne = nElems; final int sidx = startidx; final Object[] mdata = data; for(int idx = 0; idx < ne; ++idx) if (Objects.equals(obj, mdata[idx+sidx])) return idx; return -1; } public final int lastIndexOf(Object obj) { final int ne = nElems; final int nne = ne - 1; final int sidx = startidx; final Object[] mdata = data; for(int idx = 0; idx < ne; ++idx) { int ridx = nne - idx; if (Objects.equals(obj, mdata[ridx+sidx])) return ridx; } return -1; } public final int size() { return nElems; } public final int length() { return nElems; } public final int count() { return nElems; } public final boolean contains(Object obj) { return indexOf(obj) != -1; } public boolean isEmpty() { return nElems == 0; } public Object[] fillArray(Object[] retval) { System.arraycopy(data, startidx, retval, 0, nElems); return retval; } public Object[] toArray() { return fillArray(new Object[nElems]); } public Object[] toArray(Object[] marker) { return fillArray(Arrays.copyOf(marker, nElems)); } public ArrayImmutList subList(int sidx, int endidx) { ChunkedList.sublistCheck(sidx, endidx, nElems); return new ArrayImmutList(data, sidx+startidx, endidx + startidx, m); } static class Iter implements Iterator { final Object[] data; final int endidx; int idx; public Iter(Object[] d, int startidx, int nElems) { data = d; idx = startidx; endidx = startidx + nElems; } public final boolean hasNext() { return idx < endidx; } public final Object next() { if (!hasNext()) throw new NoSuchElementException(); final Object retval = data[idx]; ++idx; return retval; } } public final Iterator iterator() { return new Iter(data, startidx, nElems); } static class RIter implements Iterator { final Object[] data; final int nne; final int sidx; int idx; public RIter(Object[] d, int startidx, int nElems) { data = d; nne = nElems - 1; sidx = startidx + nne; idx = 0; } public final boolean hasNext() { return idx <= nne; } public final Object next() { if (!hasNext()) throw new NoSuchElementException(); final Object retval = data[sidx - idx]; ++idx; return retval; } } public final Iterator riterator() { return new RIter(data, startidx, nElems); } public final Object nth(int idx, Object notFound) { final int ne = nElems; if (idx < 0) idx = idx + ne; if (idx < 0 || idx >= ne) return notFound; return data[idx+startidx]; } public final Object nth(int idx) { return data[ChunkedList.indexCheck(startidx, nElems, idx < 0 ? idx + nElems : idx)]; } public final Object invoke(Object idx) { if (Util.isInteger(idx)) return nth(RT.intCast(idx)); return null; } public final Object invoke(Object idx, Object notFound) { if (Util.isInteger(idx)) return nth(RT.intCast(idx), notFound); return notFound; } public final Object valAt(Object idx) { return invoke(idx); } public final Object valAt(Object idx, Object notFound) { return invoke(idx, notFound); } public final IMapEntry entryAt(Object key) { if(Util.isInteger(key)) { final int idx = RT.intCast(key); if (idx >= 0 && idx < nElems) return MapEntry.create(idx, get(idx)); } return null; } public final boolean containsKey(Object key) { if(Util.isInteger(key)) { final int idx = RT.intCast(key); return idx >= 0 && idx < nElems; } return false; } public final ArrayImmutList empty() { return EMPTY.withMeta(m); } public final Object reduce(IFn f, Object init) { final int ne = startidx + nElems; final Object[] d = data; for(int idx = startidx; idx < ne; ++idx) { init = f.invoke(init, d[idx]); if(RT.isReduced(init)) return ((IDeref)init).deref(); } return init; } public final Object kvreduce(IFn fn, Object init) { final int sidx = startidx; final Object[] d = data; final int ne = nElems; for(int idx = 0; idx < ne && !RT.isReduced(init); ++idx) { init = fn.invoke(init, idx, d[sidx+idx]); } return Reductions.unreduce(init); } public final ISeq seq() { return LazyChunkedSeq.chunkIteratorSeq(iterator()); } public final ISeq rseq() { return LazyChunkedSeq.chunkIteratorSeq(riterator()); } Object[] asArray() { if(startidx == 0 && nElems == data.length) return data; return Arrays.copyOfRange(data, startidx, (int)(startidx + nElems)); } public final IPersistentVector cons(Object obj) { final int ne = nElems; final int nne = nElems + 1; Object[] newD = Arrays.copyOfRange(data, startidx, startidx + nne); newD[ne] = obj; if(nne >= 32) return TreeList.create(true, m, newD); return new ArrayImmutList(newD, 0, nne, m); } public final IPersistentVector assocN(int idx, Object obj) { final int ne = nElems; final int sidx = startidx; if (idx == ne) return cons(obj); indexCheck(idx); Object[] newD = Arrays.copyOfRange(data, sidx, sidx+ne); newD[idx] = obj; return new ArrayImmutList(newD, 0, ne, m); } public final IPersistentVector assoc(Object idx, Object obj) { if (!Util.isInteger(idx)) throw new RuntimeException("Vector indexes must be integers: " + String.valueOf(idx)); return assocN(RT.intCast(idx), obj); } public final ArrayImmutList pop() { final int ne = nElems; if (ne == 0) throw new RuntimeException("Attempt to pop empty vector"); final Object[] newD = Arrays.copyOfRange(data, startidx, startidx + ne-1); return new ArrayImmutList(newD, 0, ne-1, m); } public final Object peek() { if (nElems == 0) return null; return get(nElems-1); } public final MutTreeList asTransient() { return MutTreeList.create(false, m, asArray()); } @SuppressWarnings("unchecked") public final ArrayImmutList updateValue(Object key, Function fn) { int idx = RT.intCast(key); if (idx < 0) idx = nElems + idx; if(idx >= nElems) throw new RuntimeException("Index out of range: " + String.valueOf(key)); final Object[] newD = Arrays.copyOfRange(data, startidx, startidx+nElems); newD[idx] = fn.apply(newD[idx]); return new ArrayImmutList(newD, 0, nElems, m); } @SuppressWarnings("unchecked") public final ArrayImmutList updateValues(BiFunction fn) { final int ne = nElems; final Object[] newD = Arrays.copyOfRange(data, startidx, startidx+nElems); for(int idx = 0; idx < ne; ++idx) newD[idx] = fn.apply(idx, newD[idx]); return new ArrayImmutList(newD, 0, nElems, m); } public ArraySection getArraySection() { return new ArraySection(data, startidx, startidx+nElems); } public void move(int sidx, int eidx, int count) { throw new RuntimeException("Unimplemented"); } public void fill(int sidx, int eidx, Object v) { throw new RuntimeException("Unimplemented"); } public Object copyOfRange(int sidx, int eidx) { return Arrays.copyOfRange(data, startidx + sidx, startidx + eidx); } public Object copyOf(int len) { return Arrays.copyOfRange(data, startidx + 0, startidx + len); } } ================================================ FILE: java/ham_fisted/ArrayLists.java ================================================ package ham_fisted; import java.util.List; import java.util.Arrays; import java.lang.reflect.Array; import java.util.Comparator; import java.util.Collections; import java.util.Collection; import java.util.Random; import java.util.RandomAccess; import java.util.Iterator; import java.util.function.DoubleBinaryOperator; import java.util.function.LongBinaryOperator; import java.util.function.Consumer; import java.util.function.DoubleConsumer; import java.util.function.LongConsumer; import java.util.function.IntFunction; import clojure.lang.IPersistentMap; import clojure.lang.IObj; import clojure.lang.RT; import clojure.lang.IDeref; import clojure.lang.IFn; import clojure.lang.IReduce; import clojure.lang.IReduceInit; import clojure.lang.IPersistentVector; import clojure.lang.Associative; import clojure.lang.IMapEntry; import clojure.lang.Util; import clojure.lang.MapEntry; import clojure.lang.IPersistentStack; import clojure.lang.ITransientCollection; import clojure.lang.IEditableCollection; import clojure.lang.IPersistentCollection; import it.unimi.dsi.fastutil.bytes.ByteArrays; import it.unimi.dsi.fastutil.bytes.ByteComparator; import it.unimi.dsi.fastutil.shorts.ShortArrays; import it.unimi.dsi.fastutil.shorts.ShortComparator; import it.unimi.dsi.fastutil.chars.CharArrays; import it.unimi.dsi.fastutil.chars.CharComparator; import it.unimi.dsi.fastutil.ints.IntArrays; import it.unimi.dsi.fastutil.ints.IntComparator; import it.unimi.dsi.fastutil.longs.LongArrays; import it.unimi.dsi.fastutil.longs.LongComparator; import it.unimi.dsi.fastutil.floats.FloatArrays; import it.unimi.dsi.fastutil.floats.FloatComparator; import it.unimi.dsi.fastutil.doubles.DoubleArrays; import it.unimi.dsi.fastutil.doubles.DoubleComparator; import it.unimi.dsi.fastutil.objects.ObjectArrays; public class ArrayLists { public static int checkIndex(final int idx, final int dlen) { if (idx >= 0 && idx < dlen) return idx; return ChunkedList.indexCheck(0, dlen, idx); } public static int wrapCheckIndex(int idx, final int dlen) { if(idx < 0) idx += dlen; return checkIndex(idx, dlen); } public static void checkIndexRange(int dlen, int ssidx, int seidx) { ChunkedList.checkIndexRange(0, dlen, ssidx, seidx); } public static void checkIndexRange(long dlen, long ssidx, long seidx) { ChunkedList.checkIndexRange(0, dlen, ssidx, seidx); } public static DoubleConsumer asDoubleConsumer(Object c) { if (c instanceof DoubleConsumer) return (DoubleConsumer) c; return null; } public static LongConsumer asLongConsumer(Object c) { if (c instanceof LongConsumer) return (LongConsumer) c; return null; } public interface ArrayOwner { ArraySection getArraySection(); void fill(int sidx, int eidx, Object v); Object copyOfRange(int sidx, int eidx); Object copyOf(int len); //overwrite void move(int sourceIdx, int dstIdx, int count); } public interface ArrayPersistentVector extends IMutList, IPersistentVector { IPersistentVector unsafeImmut(); default boolean equiv(Object other) { return IMutList.super.equiv(other); } default IPersistentVector cons(Object o) { return immut().cons(o); } default IPersistentVector assocN(int i, Object o) { return immut().assocN(i, o); } default int length() { return size(); } default Associative assoc(Object idx, Object o) { return immut().assoc(idx, o); } default IPersistentStack pop() { final int nElems = size(); if (nElems == 0) throw new RuntimeException("Can't pop empty vector"); if (nElems == 1) return ImmutList.EMPTY.withMeta(meta()); return immut().pop(); } default Object peek() { final int nElems = size(); if (nElems == 0) return null; return get(nElems-1); } default ImmutList empty() { return ImmutList.EMPTY.withMeta(meta()); } } @SuppressWarnings("unchecked") static boolean fillRangeArrayCopy(Object dest, long sidx, long eidx, long startidx, Object ll) { //True means this function took care of the transfer, false means //fallback to a more generalized transfer if(ll instanceof RandomAccess) { final List l = (List)ll; if (l.isEmpty()) return true; final long endidx = startidx + l.size(); checkIndexRange(eidx-sidx, startidx, endidx); } if(ll instanceof ArrayOwner) { final ArraySection as = ((ArrayOwner)ll).getArraySection(); final int sz = as.size(); if(dest.getClass().isAssignableFrom(as.array.getClass())) { System.arraycopy(as.array, as.sidx, dest, (int)(sidx+startidx), sz); return true; } } return false; } static List immutShuffleDefault(IMutList m, Random r) { final IMutList retval = m.cloneList(); retval.shuffle(r); return retval; } @SuppressWarnings("unchecked") static List immutSortDefault(IMutList m, Comparator c) { final IMutList retval = m.cloneList(); retval.sort(c); return retval; } public interface IArrayList extends IMutList, ArrayOwner, TypedList, ArrayPersistentVector { default void add(int idx, int count, Object obj) { int nElems = size(); idx = checkIndex(idx, nElems+1); int ne = nElems + count; final Object d = ensureCapacity(ne); setSize(ne); if(idx != nElems) { move(idx, idx+count, nElems - idx); } fillRange(idx, idx+count, obj); } default Class containedType() { return getArraySection().array.getClass().getComponentType(); } default IPersistentVector unsafeImmut() { return ImmutList.create(true, meta(), (Object[])getArraySection().array); } default List immutShuffle(Random r) { return immutShuffleDefault(this, r); } default List immutSort(Comparator c) { return immutSortDefault(this, c); } default void fillRange(long startidx, long endidx, Object v) { checkIndexRange(size(), startidx, endidx); fill((int)startidx, (int)endidx, v); } default void fillRangeReducible(long startidx, Object v) { final ArraySection as = getArraySection(); if (!fillRangeArrayCopy(as.array, as.sidx, as.eidx, startidx, v)) { ArrayPersistentVector.super.fillRangeReducible(startidx, v); } } default void setSize(int size ) { throw new RuntimeException("unimplemented"); } default Object ensureCapacity(int newlen) { throw new RuntimeException("unimplemented"); } } public interface ILongArrayList extends LongMutList, ArrayOwner, TypedList, ArrayPersistentVector { default void add(int idx, int count, Object obj) { int nElems = size(); idx = checkIndex(idx, nElems+1); int ne = nElems + count; final Object d = ensureCapacity(ne); setSize(ne); if(idx != nElems) { move(idx, idx+count, nElems - idx); } fillRange(idx, idx+count, obj); } default Class containedType() { return getArraySection().array.getClass().getComponentType(); } default IPersistentVector unsafeImmut() { return ImmutList.create(true, meta(), (Object[])getArraySection().array); } default List immutShuffle(Random r) { return immutShuffleDefault(this, r); } default List immutSort(Comparator c) { return immutSortDefault(this, c); } default void fillRange(long startidx, long endidx, Object v) { checkIndexRange(size(), startidx, endidx); fill((int)startidx, (int)endidx, v); } default void fillRangeReducible(long startidx, Object v) { final ArraySection as = getArraySection(); if (!fillRangeArrayCopy(as.array, as.sidx, as.eidx, (int)startidx, v)) LongMutList.super.fillRangeReducible(startidx, v); } default Object reduce(IFn rfn, Object init) { return LongMutList.super.reduce(rfn, init); } default Object toNativeArray() { return copyOf(size()); } default void setSize(int size ) { throw new RuntimeException("unimplemented"); } default Object ensureCapacity(int newlen) { throw new RuntimeException("unimplemented"); } } public interface IDoubleArrayList extends DoubleMutList, ArrayOwner, TypedList, ArrayPersistentVector { default void add(int idx, int count, Object obj) { int nElems = size(); idx = checkIndex(idx, nElems+1); int ne = nElems + count; setSize(ne); final Object d = ensureCapacity(ne); if(idx != nElems) { move(idx, idx+count, nElems - idx); } fillRange(idx, idx+count, obj); } default Class containedType() { return getArraySection().array.getClass().getComponentType(); } default IPersistentVector unsafeImmut() { return ImmutList.create(true, meta(), (Object[])getArraySection().array); } default List immutShuffle(Random r) { return immutShuffleDefault(this, r); } default List immutSort(Comparator c) { return immutSortDefault(this, c); } default void fillRange(long startidx, long endidx, Object v) { checkIndexRange(size(), startidx, endidx); fill((int)startidx, (int)endidx, v); } default void fillRangeReducible(long startidx, Object v) { final ArraySection as = getArraySection(); if (!fillRangeArrayCopy(as.array, as.sidx, as.eidx, startidx, v)) DoubleMutList.super.fillRangeReducible(startidx, v); } default Object toNativeArray() { return copyOf(size()); } default void setSize(int size ) { throw new RuntimeException("unimplemented"); } default Object ensureCapacity(int newlen) { throw new RuntimeException("unimplemented"); } } static int fixSubArrayBinarySearch(final int sidx, final int len, final int res) { return res < 0 ? -1 - (res + sidx) : res - sidx; } public static Object[] objectArray(int len) { return new Object[len]; } public static class ObjectArraySubList implements IArrayList { public final Object[] data; public final int sidx; public final int eidx; public final int nElems; public final IPersistentMap meta; public ObjectArraySubList(Object[] d, int _sidx, int _eidx, IPersistentMap m) { data = d; sidx = _sidx; eidx = _eidx; nElems = eidx - sidx; meta = m; } public boolean isCompatible(Object other) { return other instanceof Object[]; } public String toString() { return Transformables.sequenceToString(this); } public boolean equals(Object other) { return equiv(other); } public int hashCode() { return hasheq(); } public ArraySection getArraySection() { return new ArraySection(data, sidx, eidx); } public Class containedType() { return data.getClass().getComponentType(); } public int size() { return nElems; } public Object get(int idx) { return data[checkIndex(idx, nElems) + sidx]; } public Object nth(int idx) { return data[checkIndex(idx < 0 ? idx + nElems : idx, nElems) + sidx]; } public Object set(int idx, Object obj) { idx = checkIndex(idx, nElems) + sidx; final Object retval = data[idx]; data[idx] = obj; return retval; } public IMutList subList(int ssidx, int seidx) { ChunkedList.sublistCheck(ssidx, seidx, size()); return (IMutList)toList(data, ssidx + sidx, seidx + sidx, meta()); } public IPersistentMap meta() { return meta; } public IObj withMeta(IPersistentMap m) { return (IObj)toList(data, sidx, eidx, m); } public Object[] toArray() { return Arrays.copyOfRange(data, sidx, eidx); } public void sort(Comparator c) { if(c == null) ObjectArrays.parallelQuickSort(data, sidx, eidx); else ObjectArrays.parallelQuickSort(data, sidx, eidx, c); } public void shuffle(Random r) { ObjectArrays.shuffle(data, sidx, eidx, r); } @SuppressWarnings("unchecked") public int binarySearch(Object v, Comparator c) { return fixSubArrayBinarySearch(sidx, size(), c == null ? ObjectArrays.binarySearch(data, sidx, eidx, v) : ObjectArrays.binarySearch(data, sidx, eidx, v, c)); } @SuppressWarnings("unchecked") public void forEach(Consumer c) { final int es = eidx; final Object[] d = data; for(int ss = sidx; ss < es; ++ss) c.accept(d[ss]); } public void fillRangeReducible(long startidx, Object v) { final ArraySection as = getArraySection(); if (!fillRangeArrayCopy(as.array, as.sidx, as.eidx, startidx, v)) { final int ss = (int)startidx + sidx; final int ee = sidx + size(); Reductions.serialReduction(new Reductions.IndexedAccum( startidx+sidx, new IFn.OLOO() { public Object invokePrim(Object acc, long idx, Object v) { if(idx >= ee) throw new IndexOutOfBoundsException("Index " + String.valueOf(idx - sidx) + " is out of range: " + String.valueOf(size())); ((Object[])acc)[(int)idx] = v; return acc; }}), data, v); } } public Object reduce(IFn rfn, Object acc) { final int ee = eidx; final Object[] d = data; for(int idx = sidx; idx < ee; ++idx ) { acc = rfn.invoke(acc, d[idx]); if(RT.isReduced(acc)) return ((IDeref)acc).deref(); } return acc; } public IPersistentVector immut() { return ArrayImmutList.create(true, data, sidx, eidx, meta()); } public void move(int sidx, int eidx, int count) { checkIndexRange(size(), eidx, eidx + count); System.arraycopy(data, sidx, data, eidx, count); } public void fill(int ssidx, int seidx, Object v) { checkIndexRange(size(), ssidx, seidx); Arrays.fill(data, sidx + ssidx, sidx + seidx, v); } public Object copyOfRange(int ssidx, int seidx) { return Arrays.copyOfRange(data, sidx+ssidx, sidx+seidx); } public Object copyOf(int len) { return Arrays.copyOfRange(data, sidx, sidx+len); } } public static int newArrayLen(int len) { return len < 100000 ? len * 2 : (int)(len * 1.5); } public static long newArrayLen(long len) { return len < 100000 ? len * 2 : (long)(len * 1.5); } public static class ObjectArrayList implements IArrayList, ITransientCollection { Object[] data; int nElems; IPersistentMap meta; public ObjectArrayList(Object[] d, int ne, IPersistentMap meta) { data = d; nElems = ne; } public ObjectArrayList(int capacity) { this(new Object[capacity], 0, null); } public ObjectArrayList() { this(4); } public String toString() { return Transformables.sequenceToString(this); } public boolean equals(Object other) { return equiv(other); } public int hashCode() { return hasheq(); } public IMutList cloneList() { return new ObjectArrayList((Object[])copyOf(nElems), nElems, meta); } public ArraySection getArraySection() { return new ArraySection(data, 0, nElems); } public Class containedType() { return data.getClass().getComponentType(); } public void clear() { nElems = 0; } public int size() { return nElems; } public void setSize(int sz) { nElems = sz; } public Object get(int idx) { return data[checkIndex(idx, nElems)]; } public Object set(int idx, Object obj) { idx = checkIndex(idx, nElems); final Object retval = data[idx]; data[idx] = obj; return retval; } public int capacity() { return data.length; } public Object[] ensureCapacity(int len) { Object[] d = data; if (len >= d.length) { d = data = Arrays.copyOf(d, newArrayLen(len)); } return d; } public boolean add(Object obj) { final int ne = nElems; final Object [] d = ensureCapacity(ne+1); d[ne] = obj; nElems = ne+1; return true; } public void add(int idx, Object obj) { idx = checkIndex(idx, nElems); if (idx == nElems) { add(obj); return; } final int ne = nElems; final Object [] d = ensureCapacity(ne+1); System.arraycopy(d, idx, d, idx+1, ne - idx); d[idx] = obj; nElems = ne+1; } /// Extra method because some things that implement IReduceInit are not /// collections. public boolean addAllReducible(Object c) { final int sz = size(); if (c instanceof RandomAccess) { final List cl = (List) c; if (cl.isEmpty() ) return false; final int cs = cl.size(); ensureCapacity(cs+sz); nElems += cs; //Hit fastpath fillRangeReducible(sz, cl); } else { IArrayList.super.addAllReducible(c); } return sz != size(); } public IMutList subList(int ssidx, int seidx) { ChunkedList.sublistCheck(ssidx, seidx, size()); return (IMutList)toList(data, ssidx, seidx, meta()); } public IPersistentMap meta() { return meta; } public IObj withMeta(IPersistentMap m) { meta = m; return this; } public Object[] toArray() { return Arrays.copyOf(data, nElems); } public void removeRange(int startidx, int endidx) { checkIndexRange(nElems, startidx, endidx); System.arraycopy(data, startidx, data, endidx, nElems - endidx); Arrays.fill(data, endidx, nElems, null); nElems -= endidx - startidx; } public Object reduce(IFn fn) { return ((IReduce)subList(0, nElems)).reduce(fn); } public Object reduce(IFn fn, Object init) { return ((IReduceInit)subList(0, nElems)).reduce(fn,init); } public void sort(Comparator c) { subList(0, nElems).sort(c); } public void shuffle(Random r) { ((IMutList)subList(0, nElems)).shuffle(r); } @SuppressWarnings("unchecked") public void forEach(Consumer c) { final int es = nElems; final Object[] d = data; for(int ss = 0; ss < es; ++ss) c.accept(d[ss]); } public void fillRangeReducible(long startidx, Object v) { subList(0, size()).fillRangeReducible(startidx, v); } public IPersistentVector immut() { return ArrayImmutList.create(true, data, 0, nElems, meta()); } public IPersistentCollection persistent() { return immut(); } public final ObjectArrayList conj(Object obj) { add(obj); return this; } public void move(int sidx, int eidx, int count) { checkIndexRange(size(), eidx, eidx + count); System.arraycopy(data, sidx, data, eidx, count); } public void fill(int ssidx, int seidx, Object v) { checkIndexRange(nElems, ssidx, seidx); Arrays.fill(data, ssidx, seidx, v); } public Object copyOfRange(int ssidx, int seidx) { return Arrays.copyOfRange(data, ssidx, seidx); } public Object copyOf(int len) { return Arrays.copyOf(data, len); } public static ObjectArrayList wrap(final Object[] data, int nElems, IPersistentMap m) { if (data.length < nElems) throw new RuntimeException("Array len less than required"); return new ObjectArrayList(data, nElems, m); } public static ObjectArrayList wrap(final Object[] data, IPersistentMap m) { return new ObjectArrayList(data, data.length, m); } } @SuppressWarnings("unchecked") public static Object[] toArray(Collection c) { if( c instanceof RandomAccess ) { List l = (List)c; final int ne = l.size(); Object[] rv = new Object[l.size()]; for(int idx = 0; idx < ne; ++idx) rv[idx] = l.get(idx); return rv; } else { final ObjectArrayList res = new ObjectArrayList(); res.addAllReducible(c); return res.toArray(); } } @SuppressWarnings("unchecked") public static T[] toArray(Collection c, T[] d) { if(c instanceof RandomAccess) { List l = (List)c; final int ne = l.size(); final T[] rv = Arrays.copyOf(d, ne); for(int idx = 0; idx < ne; ++idx) { rv[idx] = (T)l.get(idx); } return rv; } final ObjectArrayList res = ObjectArrayList.wrap(d, 0, null); res.addAllReducible(c); return (T[])res.toArray(); } public static IMutList toList(final Object[] data, final int sidx, final int eidx, final IPersistentMap meta) { final int dlen = eidx - sidx; return new ObjectArraySubList(data, sidx, eidx, meta); } public static IMutList toList(final Object[] data) { return toList(data, 0, data.length, null); } public static class ByteArraySubList implements ILongArrayList { public final byte[] data; public final int dlen; public final int sidx; public final IPersistentMap meta; public ByteArraySubList(byte[] d, int s, int len, IPersistentMap _meta) { data = d; sidx = s; dlen = len; meta = _meta; } public String toString() { return Transformables.sequenceToString(this); } public boolean equals(Object other) { return equiv(other); } public int hashCode() { return hasheq(); } public ArraySection getArraySection() { return new ArraySection(data, sidx, sidx + dlen); } public Class containedType() { return data.getClass().getComponentType(); } public int size() { return dlen; } public Byte get(int idx) { return data[checkIndex(idx, dlen) + sidx]; } public long getLong(int idx) { return data[checkIndex(idx, dlen) + sidx]; } public void setLong(int idx, long oobj) { data[checkIndex(idx, dlen) + sidx] = RT.byteCast(oobj); } public IMutList cloneList() { return (IMutList)toList(Arrays.copyOfRange(data, sidx, sidx+dlen)); } public IntComparator indexComparator() { return new IntComparator() { public int compare(int lidx, int ridx) { return Byte.compare(data[lidx+sidx], data[ridx+sidx]); } }; } public LongMutList subList(int ssidx, int seidx) { ChunkedList.sublistCheck(ssidx, seidx, size()); return (LongMutList)toList(data, ssidx + sidx, seidx + sidx, meta()); } public IPersistentMap meta() { return meta; } public IObj withMeta(IPersistentMap m) { return new ByteArraySubList(data, sidx, sidx + dlen, m); } public Object[] toArray() { final int sz = size(); Object[] retval = new Object[size()]; for (int idx = 0; idx < sz; ++idx) retval[idx] = data[idx+sidx]; return retval; } @SuppressWarnings("unchecked") public void sort(Comparator c) { if (c == null) Arrays.sort(data, sidx, sidx+dlen); else { ILongArrayList.super.sort(c); } } public void shuffle(Random r) { ByteArrays.shuffle(data, sidx, sidx+dlen, r); } public static ByteComparator asByteComparator(Comparator c) { if (c instanceof ByteComparator) return (ByteComparator)c; else if (c instanceof LongComparator) { final LongComparator lc = (LongComparator)c; return new ByteComparator() { public int compare(byte l, byte r) { return lc.compare(l,r); } }; } return null; } @SuppressWarnings("unchecked") public int binarySearch(Object v, Comparator c) { final byte vv = RT.byteCast(Casts.longCast(v)); final ByteComparator bc = asByteComparator(c); if(c == null || bc != null) return fixSubArrayBinarySearch(sidx, size(), bc == null ? ByteArrays.binarySearch(data, sidx, sidx+size(), vv) : ByteArrays.binarySearch(data, sidx, sidx+size(), vv, bc)); return ILongArrayList.super.binarySearch(v, c); } public Object longReduction(IFn.OLO rfn, Object init) { final int es = sidx + dlen; final byte[] d = data; for(int ss = sidx; ss < es && !RT.isReduced(init); ++ss) init = rfn.invokePrim(init, d[ss]); return Reductions.unreduce(init); } public void fillRangeReducible(long startidx, Object v) { final ArraySection as = getArraySection(); if (!fillRangeArrayCopy(as.array, as.sidx, as.eidx, startidx, v)) { final int ss = (int)startidx + sidx; final int ee = sidx + size(); Reductions.serialReduction(new Reductions.IndexedLongAccum( startidx+sidx, new IFn.OLLO() { public Object invokePrim(Object acc, long idx, long v) { if(idx >= ee) throw new IndexOutOfBoundsException("Index " + String.valueOf(idx - sidx) + " is out of range: " + String.valueOf(size())); data[(int)idx] = RT.byteCast(v); return data; }}), data, v); } } public void move(int sidx, int eidx, int count) { checkIndexRange(size(), eidx, eidx + count); System.arraycopy(data, sidx, data, eidx, count); } public void fill(int ssidx, int seidx, Object v) { checkIndexRange(size(), ssidx, seidx); Arrays.fill(data, sidx + ssidx, sidx + seidx, RT.byteCast(Casts.longCast(v))); } public Object copyOfRange(int ssidx, int seidx) { return Arrays.copyOfRange(data, sidx+ssidx, sidx+seidx); } public Object copyOf(int len) { return Arrays.copyOfRange(data, sidx, sidx+len); } } public static IMutList toList(final byte[] data, final int sidx, final int eidx, final IPersistentMap meta) { final int dlen = eidx - sidx; return new ByteArraySubList(data, sidx, dlen, meta); } public static IMutList toList(final byte[] data) { return toList(data, 0, data.length, null); } public static class ShortArraySubList implements ILongArrayList { public final short[] data; public final int sidx; public final int dlen; public final IPersistentMap meta; public ShortArraySubList(short[] d, int s, int len, IPersistentMap _meta) { data = d; sidx = s; dlen = len; meta = _meta; } public String toString() { return Transformables.sequenceToString(this); } public boolean equals(Object other) { return equiv(other); } public int hashCode() { return hasheq(); } public IMutList cloneList() { return (IMutList)toList(Arrays.copyOfRange(data, sidx, sidx+dlen)); } public ArraySection getArraySection() { return new ArraySection(data, sidx, sidx + dlen); } public Class containedType() { return data.getClass().getComponentType(); } public int size() { return dlen; } public long getLong(int idx) { return data[checkIndex(idx, dlen) + sidx]; } public void setLong(int idx, long oobj) { data[checkIndex(idx, dlen) + sidx] = RT.shortCast(Casts.longCast(oobj)); } public IntComparator indexComparator() { return new IntComparator() { public int compare(int lidx, int ridx) { return Short.compare(data[lidx+sidx], data[ridx+sidx]); } }; } public LongMutList subList(int ssidx, int seidx) { ChunkedList.sublistCheck(ssidx, seidx, size()); return (LongMutList)toList(data, ssidx + sidx, seidx + sidx, meta()); } public IPersistentMap meta() { return meta; } public IObj withMeta(IPersistentMap m) { return (IObj)toList(data, sidx, sidx + dlen, m); } public Object[] toArray() { final int sz = size(); Object[] retval = new Object[size()]; for (int idx = 0; idx < sz; ++idx) retval[idx] = data[idx+sidx]; return retval; } @SuppressWarnings("unchecked") public void sort(Comparator c) { if (c == null) Arrays.sort(data, sidx, sidx+dlen); else { final Object[] odata = toArray(); Arrays.sort(odata, c); final short[] d = data; final int sz = size(); final int ss = sidx; for (int idx = 0; idx < sz; ++idx) { d[idx+ss] = (short)odata[idx]; } } } public void shuffle(Random r) { ShortArrays.shuffle(data, sidx, sidx+dlen, r); } public static ShortComparator asShortComparator(Comparator c) { if (c instanceof ShortComparator) return (ShortComparator)c; else if (c instanceof LongComparator) { final LongComparator lc = (LongComparator)c; return new ShortComparator() { public int compare(short l, short r) { return lc.compare(l,r); } }; } return null; } @SuppressWarnings("unchecked") public int binarySearch(Object v, Comparator c) { final short vv = RT.shortCast(Casts.longCast(v)); final ShortComparator bc = asShortComparator(c); if(c == null || bc != null) return fixSubArrayBinarySearch(sidx, size(), bc == null ? ShortArrays.binarySearch(data, sidx, sidx+size(), vv) : ShortArrays.binarySearch(data, sidx, sidx+size(), vv, bc)); return ILongArrayList.super.binarySearch(v, c); } public Object longReduction(IFn.OLO rfn, Object init) { final int es = sidx + dlen; final short[] d = data; for(int ss = sidx; ss < es && !RT.isReduced(init); ++ss) init = rfn.invokePrim(init, d[ss]); return Reductions.unreduce(init); } public void fillRangeReducible(long startidx, Object v) { final ArraySection as = getArraySection(); if (!fillRangeArrayCopy(as.array, as.sidx, as.eidx, startidx, v)) { final int ss = (int)startidx + sidx; final int ee = sidx + size(); Reductions.serialReduction(new Reductions.IndexedLongAccum( startidx+sidx, new IFn.OLLO() { public Object invokePrim(Object acc, long idx, long v) { if(idx >= ee) throw new IndexOutOfBoundsException("Index " + String.valueOf(idx - sidx) + " is out of range: " + String.valueOf(size())); data[(int)idx] = RT.shortCast(v); return data; }}), data, v); } } public void move(int sidx, int eidx, int count) { checkIndexRange(size(), eidx, eidx + count); System.arraycopy(data, sidx, data, eidx, count); } public void fill(int ssidx, int seidx, Object v) { checkIndexRange(size(), ssidx, seidx); Arrays.fill(data, sidx + ssidx, sidx + seidx, RT.shortCast(Casts.longCast(v))); } public Object copyOfRange(int ssidx, int seidx) { checkIndex(ssidx, size()); return Arrays.copyOfRange(data, sidx+ssidx, sidx+seidx); } public Object copyOf(int len) { return Arrays.copyOfRange(data, sidx, sidx+len); } } public static IMutList toList(final short[] data, final int sidx, final int eidx, final IPersistentMap meta) { final int dlen = eidx - sidx; return new ShortArraySubList(data, sidx, dlen, meta); } public static IMutList toList(final short[] data) { return toList(data, 0, data.length, null); } public static int[] intArray(int len) { return new int[len]; } public static class IntArraySubList implements ILongArrayList { public final int[] data; public final int sidx; public final int eidx; public final int nElems; public final IPersistentMap meta; public IntArraySubList(int[] d, int _sidx, int _eidx, IPersistentMap m) { data = d; sidx = _sidx; eidx = _eidx; nElems = eidx - sidx; meta = m; } public String toString() { return Transformables.sequenceToString(this); } public boolean equals(Object other) { return equiv(other); } public int hashCode() { return hasheq(); } public IMutList cloneList() { return (IMutList)toList(Arrays.copyOfRange(data, sidx, eidx)); } public ArraySection getArraySection() { return new ArraySection(data, sidx, eidx); } public Class containedType() { return data.getClass().getComponentType(); } public int size() { return nElems; } public long getLong(int idx) { return data[checkIndex(idx, nElems) + sidx]; } static void setLong(final int[] d, final int sidx, final int nElems, int idx, final long obj) { int v = RT.intCast(obj); idx = checkIndex(idx, nElems) + sidx; d[idx] = v; } public void setLong(int idx, long obj) { setLong(data, sidx, nElems, idx, obj); } public LongMutList subList(int ssidx, int seidx) { ChunkedList.sublistCheck(ssidx, seidx, size()); return (LongMutList)toList(data, ssidx + sidx, seidx + sidx, meta()); } public IPersistentMap meta() { return meta; } public IObj withMeta(IPersistentMap m) { return (IObj)toList(data, sidx, eidx, m); } public Object[] toArray() { final int[] d = data; final int ss = sidx; final int ne = nElems; Object[] retval = new Object[ne]; for(int idx = 0; idx < ne; ++idx) retval[idx] = d[idx+ss]; return retval; } public int[] toIntArray() { return Arrays.copyOfRange(data, sidx, eidx); } @SuppressWarnings("unchecked") public static IntComparator indexComparator(int[] d, int sidx, Comparator c) { if (c == null) { if (sidx != 0) { return new IntComparator() { public int compare(int lidx, int ridx) { return Integer.compare(d[lidx+sidx], d[ridx+sidx]); } }; } else { return new IntComparator() { public int compare(int lidx, int ridx) { return Integer.compare(d[lidx], d[ridx]); } }; } } else { if(c instanceof LongComparator) { final LongComparator lc = (LongComparator) c; if (sidx != 0) { return new IntComparator() { public int compare(int lidx, int ridx) { return lc.compare(d[lidx+sidx], d[ridx+sidx]); } }; } else { return new IntComparator() { public int compare(int lidx, int ridx) { return lc.compare(d[lidx], d[ridx]); } }; } } else { return new IntComparator() { public int compare(int lidx, int ridx) { return c.compare(d[lidx+sidx], d[ridx+sidx]); } }; } } } public IntComparator indexComparator() { return indexComparator(data, sidx, null); } public IntComparator indexComparator(Comparator c) { return indexComparator(data, sidx, c); } @SuppressWarnings("unchecked") public static IntComparator toIntComparator(Comparator c) { if (c instanceof IntComparator) return (IntComparator) c; else return new IntComparator() { public int compare(int lhs, int rhs) { return c.compare(lhs, rhs); } }; } public void sort(Comparator c) { if(c == null) IntArrays.parallelQuickSort(data, sidx, eidx); else { IntArrays.parallelQuickSort(data, sidx, eidx, toIntComparator(c)); } } public void shuffle(Random r) { IntArrays.shuffle(data, sidx, eidx, r); } public static IntComparator asIntComparator(Comparator c) { if (c instanceof IntComparator) return (IntComparator)c; else if (c instanceof LongComparator) { final LongComparator lc = (LongComparator)c; return new IntComparator() { public int compare(int l, int r) { return lc.compare(l,r); } }; } return null; } @SuppressWarnings("unchecked") public int binarySearch(Object v, Comparator c) { final int vv = RT.intCast(Casts.longCast(v)); final IntComparator bc = asIntComparator(c); if(c == null || bc != null) return fixSubArrayBinarySearch(sidx, size(), bc == null ? IntArrays.binarySearch(data, sidx, sidx+size(), vv) : IntArrays.binarySearch(data, sidx, sidx+size(), vv, bc)); return ILongArrayList.super.binarySearch(v, c); } public int[] sortIndirect(Comparator c) { final int sz = size(); int[] retval = iarange(0, sz, 1); if(sz < 2) return retval; if(c == null) IntArrays.parallelQuickSortIndirect(retval, data, sidx, eidx); else IntArrays.parallelQuickSort(retval, indexComparator(c)); return retval; } public Object reduce(IFn rfn, Object init) { return ILongArrayList.super.reduce(rfn, init); } public Object longReduction(IFn.OLO rfn, Object init) { final int es = eidx; final int[] d = data; for(int ss = sidx; ss < es && !RT.isReduced(init); ++ss) init = rfn.invokePrim(init, d[ss]); return Reductions.unreduce(init); } public void fillRangeReducible(long startidx, Object v) { final ArraySection as = getArraySection(); if (!fillRangeArrayCopy(as.array, as.sidx, as.eidx, startidx, v)) { final int[] d = data; final int ee = sidx + size(); Reductions.serialReduction(new Reductions.IndexedLongAccum( startidx + sidx, new IFn.OLLO() { public Object invokePrim(Object acc, long idx, long v) { if(idx >= ee) throw new IndexOutOfBoundsException("Index " + String.valueOf(idx - sidx) + "> length " + String.valueOf(size())); d[(int)idx] = RT.intCast(v); return d; }}), data, v); } } public void move(int sidx, int eidx, int count) { checkIndexRange(size(), eidx, eidx + count); System.arraycopy(data, sidx, data, eidx, count); } public void fill(int ssidx, int seidx, Object v) { checkIndexRange(size(), ssidx, seidx); Arrays.fill(data, sidx + ssidx, sidx + seidx, RT.intCast(Casts.longCast(v))); } public Object copyOfRange(int ssidx, int seidx) { checkIndex(ssidx, size()); return Arrays.copyOfRange(data, sidx+ssidx, sidx+seidx); } public Object copyOf(int len) { return Arrays.copyOfRange(data, sidx, sidx+len); } } public static class IntArrayList implements ILongArrayList { int[] data; int nElems; IPersistentMap meta; public IntArrayList(int[] d, int ne, IPersistentMap meta) { data = d; nElems = ne; } public IntArrayList(int capacity) { this(new int[capacity], 0, null); } public IntArrayList() { this(4); } public String toString() { return Transformables.sequenceToString(this); } public boolean equals(Object other) { return equiv(other); } public int hashCode() { return hasheq(); } public IMutList cloneList() { return new IntArrayList((int[])copyOf(nElems), nElems, meta); } public ArraySection getArraySection() { return new ArraySection(data, 0, nElems); } public Class containedType() { return data.getClass().getComponentType(); } public int size() { return nElems; } public void setSize(int sz) { nElems = sz; } public long getLong(int idx) { return data[checkIndex(idx, nElems)]; } public void setLong(int idx, long obj) { IntArraySubList.setLong(data, 0, nElems, idx, obj); } public int capacity() { return data.length; } public void clear() { nElems = 0; } public int[] ensureCapacity(int len) { int[] d = data; if (len >= d.length) { d = data = Arrays.copyOf(d, len < 100000 ? len * 2 : (int)(len * 1.5)); } return d; } public void addLong(long obj) { int val = RT.intCast(obj); final int ne = nElems; final int[] d = ensureCapacity(ne+1); d[ne] = val; nElems = ne+1; } public void add(int idx, Object obj) { idx = wrapCheckIndex(idx, nElems); if (idx == nElems) { add(obj); return; } final int val = RT.intCast(Casts.longCast(obj)); final int ne = nElems; final int[] d = ensureCapacity(ne+1); System.arraycopy(d, idx, d, idx+1, ne - idx); d[idx] = val; nElems = ne+1; } public boolean addAllReducible(Object c) { final int sz = size(); if (c instanceof RandomAccess) { final List cl = (List) c; if (cl.isEmpty() ) return false; final int cs = cl.size(); ensureCapacity(cs+sz); nElems += cs; //Hit fastpath fillRangeReducible(sz, cl); } else { ILongArrayList.super.addAllReducible(c); } return sz != size(); } public boolean addAll(int sidx, Collection c) { sidx = wrapCheckIndex(sidx, nElems); if (c.isEmpty()) return false; final int cs = c.size(); final int sz = size(); final int eidx = sidx + cs; ensureCapacity(cs+sz); nElems += cs; System.arraycopy(data, sidx, data, eidx, sz - sidx); fillRangeReducible(sidx, c); return true; } public IMutList subList(int ssidx, int seidx) { ChunkedList.sublistCheck(ssidx, seidx, size()); return toList(data, ssidx, seidx, meta()); } public IPersistentMap meta() { return meta; } public IObj withMeta(IPersistentMap m) { meta = m; return this; } public Object[] toArray() { return subList(0, nElems).toArray(); } public int[] toIntArray() { return Arrays.copyOf(data, nElems); } public void fillRange(long startidx, long endidx, Object v) { ((RangeList)subList(0, nElems)).fillRange(startidx, endidx, v); } public void fillRangeReducible(long startidx, Object v) { subList(0,size()).fillRangeReducible(startidx, v); } public void addRange(final int startidx, final int endidx, final Object v) { final int ne = nElems; checkIndexRange(ne, startidx, endidx); final int rangeLen = endidx - startidx; final int newLen = ne + rangeLen; ensureCapacity(newLen); System.arraycopy(data, startidx, data, endidx, nElems - startidx); fillRange(startidx, endidx, v); } public void removeRange(int startidx, int endidx) { checkIndexRange(nElems, startidx, endidx); System.arraycopy(data, startidx, data, endidx, nElems - endidx); nElems -= endidx - startidx; } public Object reduce(IFn fn) { return ((IReduce)subList(0, nElems)).reduce(fn); } public Object reduce(IFn fn, Object init) { return ((IReduceInit)subList(0, nElems)).reduce(fn,init); } public Object longReduction(IFn.OLO op, long init) { return ((LongMutList)subList(0, nElems)).longReduction(op, init); } public void sort(Comparator c) { subList(0, nElems).sort(c); } public void shuffle(Random r) { ((IMutList)subList(0, nElems)).shuffle(r); } public int[] sortIndirect(Comparator c) { return ((IMutList)subList(0, nElems)).sortIndirect(c); } @SuppressWarnings("unchecked") public int binarySearch(Object v, Comparator c) { return ((IMutList)subList(0, nElems)).binarySearch(v, c); } public IntComparator indexComparator() { return IntArraySubList.indexComparator(data, 0, null); } public IntComparator indexComparator(Comparator c) { return IntArraySubList.indexComparator(data, 0, c); } public Object longReduction(IFn.OLO rfn, Object init) { final int es = nElems; final int[] d = data; for(int ss = 0; ss < es && !RT.isReduced(init); ++ss) init = rfn.invokePrim(init, d[ss]); return Reductions.unreduce(init); } public void move(int sidx, int eidx, int count) { checkIndexRange(size(), eidx, eidx + count); System.arraycopy(data, sidx, data, eidx, count); } public void fill(int ssidx, int seidx, Object v) { checkIndexRange(size(), ssidx, seidx); Arrays.fill(data, ssidx, seidx, RT.intCast(Casts.longCast(v))); } public Object copyOfRange(int ssidx, int seidx) { checkIndex(ssidx, size()); return Arrays.copyOfRange(data, ssidx, seidx); } public Object copyOf(int len) { return Arrays.copyOf(data, len); } public static IntArrayList wrap(final int[] data, int nElems, IPersistentMap m) { if (data.length < nElems) throw new RuntimeException("Array len less than required"); return new IntArrayList(data, nElems, m); } public static IntArrayList wrap(final int[] data, IPersistentMap m) { return new IntArrayList(data, data.length, m); } } @SuppressWarnings("unchecked") public static IntComparator intIndexComparator(List srcData, Comparator comp) { if(comp != null) { if(srcData instanceof IMutList) { return ((IMutList)srcData).indexComparator(comp); } else { return new IntComparator() { public int compare(int l, int r) { return comp.compare(srcData.get(l), srcData.get(r)); } }; } } else { if (srcData instanceof IMutList) { return ((IMutList)srcData).indexComparator(); } else { return new IntComparator() { public int compare(int l, int r) { return ((Comparable)srcData.get(l)).compareTo(srcData.get(r)); } }; } } } public static IMutList toList(final int[] data, final int sidx, final int eidx, IPersistentMap meta) { final int dlen = eidx - sidx; return new IntArraySubList(data, sidx, eidx, meta); } public static IMutList toList(final int[] data) { return toList(data, 0, data.length, null); } public static long[] longArray(int len) { return new long[len]; } public static class LongArraySubList implements ILongArrayList { public final long[] data; public final int sidx; public final int eidx; public final int nElems; public final IPersistentMap meta; public LongArraySubList(long[] d, int _sidx, int _eidx, IPersistentMap m) { data = d; sidx = _sidx; eidx = _eidx; nElems = eidx - sidx; meta = m; } public String toString() { return Transformables.sequenceToString(this); } public boolean equals(Object other) { return equiv(other); } public int hashCode() { return hasheq(); } public ArraySection getArraySection() { return new ArraySection(data, sidx, eidx); } public IMutList cloneList() { return (IMutList)toList(Arrays.copyOfRange(data, sidx, eidx)); } public Class containedType() { return data.getClass().getComponentType(); } public int size() { return nElems; } public long getLong(int idx) { return data[checkIndex(idx, nElems) + sidx]; } public Object get(int idx) { return data[checkIndex(idx, nElems) + sidx]; } public Object nth(int idx) { return data[checkIndex(idx < 0 ? idx + nElems : idx, nElems) + sidx]; } static void setLong(final long[] d, final int sidx, final int nElems, int idx, final long v) { idx = checkIndex(idx, nElems) + sidx; d[idx] = v; } public void setLong(int idx, long obj) { setLong(data, sidx, nElems, idx, obj); } public IMutList subList(int ssidx, int seidx) { ChunkedList.sublistCheck(ssidx, seidx, size()); return toList(data, ssidx + sidx, seidx + sidx, meta()); } public IPersistentMap meta() { return meta; } public IObj withMeta(IPersistentMap m) { return (IObj)toList(data, sidx, eidx, m); } public Object[] toArray() { final long[] d = data; final int ss = sidx; final int ne = nElems; Object[] retval = new Object[ne]; for(int idx = 0; idx < ne; ++idx) retval[idx] = d[idx+ss]; return retval; } public long[] toLongArray() { return Arrays.copyOfRange(data, sidx, eidx); } @SuppressWarnings("unchecked") public static IntComparator indexComparator(long[] d, int sidx, Comparator c) { if (c == null) { if (sidx != 0) { return new IntComparator() { public int compare(int lidx, int ridx) { return Long.compare(d[lidx+sidx], d[ridx+sidx]); } }; } else { return new IntComparator() { public int compare(int lidx, int ridx) { return Long.compare(d[lidx], d[ridx]); } }; } } else { if(c instanceof LongComparator) { final LongComparator lc = (LongComparator) c; if (sidx != 0) { return new IntComparator() { public int compare(int lidx, int ridx) { return lc.compare(d[lidx+sidx], d[ridx+sidx]); } }; } else { return new IntComparator() { public int compare(int lidx, int ridx) { return lc.compare(d[lidx], d[ridx]); } }; } } else { return new IntComparator() { public int compare(int lidx, int ridx) { return c.compare(d[lidx+sidx], d[ridx+sidx]); } }; } } } public IntComparator indexComparator() { return indexComparator(data, sidx, null); } public IntComparator indexComparator(Comparator c) { return indexComparator(data, sidx, c); } @SuppressWarnings("unchecked") public static LongComparator toLongComparator(Comparator c) { if (c instanceof LongComparator) return (LongComparator) c; else return new LongComparator() { public int compare(long lhs, long rhs) { return c.compare(lhs, rhs); } }; } public void sort(Comparator c) { if(c == null) LongArrays.parallelQuickSort(data, sidx, eidx); else { LongArrays.parallelQuickSort(data, sidx, eidx, toLongComparator(c)); } } public int[] sortIndirect(Comparator c) { final int sz = size(); int[] retval = iarange(0, sz, 1); if(sz < 2) return retval; if(c == null) LongArrays.parallelQuickSortIndirect(retval, data, sidx, eidx); else IntArrays.parallelQuickSort(retval, indexComparator(c)); return retval; } public void shuffle(Random r) { LongArrays.shuffle(data, sidx, eidx, r); } public static LongComparator asLongComparator(Comparator c) { if (c instanceof LongComparator) return (LongComparator)c; return null; } @SuppressWarnings("unchecked") public int binarySearch(Object v, Comparator c) { final long vv = Casts.longCast(v); final LongComparator bc = asLongComparator(c); if(c == null || bc != null) return fixSubArrayBinarySearch(sidx, size(), bc == null ? LongArrays.binarySearch(data, sidx, sidx+size(), vv) : LongArrays.binarySearch(data, sidx, sidx+size(), vv, bc)); return ILongArrayList.super.binarySearch(v, c); } public Object longReduction(IFn.OLO rfn, Object init) { final int ee = size(); final long[] d = data; for(int idx = 0; idx < ee && !RT.isReduced(init); ++idx) init = rfn.invokePrim(init, data[idx+sidx]); return Reductions.unreduce(init); } public void longForEach(LongConsumer c) { final int es = eidx; final long[] d = data; for(int ss = sidx; ss < es; ++ss) c.accept(d[ss]); } public void fillRangeReducible(long startidx, Object v) { final ArraySection as = getArraySection(); if (!fillRangeArrayCopy(as.array, as.sidx, as.eidx, startidx, v)) { final int ee = sidx + size(); final long[] d = data; Reductions.serialReduction(new Reductions.IndexedLongAccum( startidx + sidx, new IFn.OLLO() { public Object invokePrim(Object acc, long idx, long v) { if(idx >= ee) throw new IndexOutOfBoundsException("Index " + String.valueOf(idx-sidx) + " > length: " + String.valueOf(size())); d[(int)idx] = v; return d; }}), data, v); } } public void move(int sidx, int eidx, int count) { checkIndexRange(size(), eidx, eidx + count); System.arraycopy(data, sidx, data, eidx, count); } public void fill(int ssidx, int seidx, Object v) { checkIndexRange(size(), ssidx, seidx); Arrays.fill(data, sidx + ssidx, sidx + seidx, Casts.longCast(v)); } public Object copyOfRange(int ssidx, int seidx) { return Arrays.copyOfRange(data, sidx+ssidx, sidx+seidx); } public Object copyOf(int len) { return Arrays.copyOfRange(data, sidx, sidx+len); } } public static class LongArrayList implements ILongArrayList { long[] data; int nElems; IPersistentMap meta; public LongArrayList(long[] d, int ne, IPersistentMap meta) { data = d; nElems = ne; } public LongArrayList(int capacity) { this(new long[capacity], 0, null); } public LongArrayList() { this(4); } public String toString() { return Transformables.sequenceToString(this); } public boolean equals(Object other) { return equiv(other); } public int hashCode() { return hasheq(); } public IMutList cloneList() { return new LongArrayList((long[])copyOf(nElems), nElems, meta); } public ArraySection getArraySection() { return new ArraySection(data, 0, nElems); } public Class containedType() { return data.getClass().getComponentType(); } public void clear() { nElems = 0; } public int size() { return nElems; } public void setSize(int sz) { nElems = sz; } public long getLong(int idx) { return data[checkIndex(idx, nElems)]; } public void setLong(int idx, long obj) { LongArraySubList.setLong(data, 0, nElems, idx, obj); } public int capacity() { return data.length; } public long[] ensureCapacity(int len) { long[] d = data; if (len >= d.length) { d = data = Arrays.copyOf(d, len < 100000 ? len * 2 : (int)(len * 1.5)); } return d; } public void addLong(long val) { final int ne = nElems; final long[] d = ensureCapacity(ne+1); d[ne] = val; nElems = ne+1; } public boolean add(Object obj) { addLong(Casts.longCast(obj)); return true; } public void add(int idx, Object obj) { idx = wrapCheckIndex(idx, nElems); if (idx == nElems) { add(obj); return; } final long val = Casts.longCast(obj); final int ne = nElems; final long[] d = ensureCapacity(ne+1); System.arraycopy(d, idx, d, idx+1, ne - idx); d[idx] = val; nElems = ne+1; } public boolean addAllReducible(Object c) { final int sz = size(); if (c instanceof RandomAccess) { final List cl = (List) c; if (cl.isEmpty() ) return false; final int cs = cl.size(); ensureCapacity(cs+sz); nElems += cs; //Hit fastpath fillRangeReducible(sz, cl); } else { ILongArrayList.super.addAllReducible(c); } return sz != size(); } public boolean addAll(int sidx, Collection c) { sidx = wrapCheckIndex(sidx, nElems); if (c.isEmpty()) return false; final int cs = c.size(); final int sz = size(); final int eidx = sidx + cs; ensureCapacity(cs+sz); nElems += cs; System.arraycopy(data, sidx, data, eidx, sz - sidx); fillRangeReducible(sidx, c); return true; } public Object remove(int idx) { idx = wrapCheckIndex(idx, nElems); final int ne = nElems; final int nne = ne - 1; final long[] d = data; final long retval = d[idx]; if (idx != nne) { final int copyLen = ne - idx - 1; System.arraycopy(d, idx+1, d, idx, copyLen); } --nElems; return retval; } public IMutList subList(int ssidx, int seidx) { ChunkedList.sublistCheck(ssidx, seidx, size()); return toList(data, ssidx, seidx, meta()); } public IPersistentMap meta() { return meta; } public IObj withMeta(IPersistentMap m) { meta = m; return this; } public Object[] toArray() { return subList(0, nElems).toArray(); } public long[] toLongArray() { return Arrays.copyOf(data, nElems); } public void fillRange(long startidx, long endidx, Object v) { ((RangeList)subList(0, nElems)).fillRange(startidx, endidx, v); } public void fillRangeReducible(long startidx, List v) { ((RangeList)subList(0, nElems)).fillRangeReducible(startidx, v); } public void addRange(final int startidx, final int endidx, final Object v) { final int ne = nElems; checkIndexRange(ne, startidx, endidx); final int rangeLen = endidx - startidx; final int newLen = ne + rangeLen; ensureCapacity(newLen); System.arraycopy(data, startidx, data, endidx, nElems - startidx); } public void removeRange(int startidx, int endidx) { checkIndexRange(nElems, startidx, endidx); System.arraycopy(data, startidx, data, endidx, nElems - endidx); nElems -= endidx - startidx; } public Object reduce(IFn fn) { return ((IReduce)subList(0, nElems)).reduce(fn); } public Object reduce(IFn fn, Object init) { return ((IReduceInit)subList(0, nElems)).reduce(fn,init); } public Object longReduction(IFn.OLO op, Object init) { return ((LongMutList)subList(0, nElems)).longReduction(op, init); } public IntComparator indexComparator() { return LongArraySubList.indexComparator(data, 0, null); } public IntComparator indexComparator(Comparator c) { return LongArraySubList.indexComparator(data, 0, c); } public void sort(Comparator c) { subList(0, nElems).sort(c); } public void shuffle(Random r) { ((IMutList)subList(0, nElems)).shuffle(r); } @SuppressWarnings("unchecked") public int binarySearch(Object v, Comparator c) { return ((IMutList)subList(0, nElems)).binarySearch(v, c); } public int[] sortIndirect(Comparator c) { return ((IMutList)subList(0, nElems)).sortIndirect(c); } public void fillRangeReducible(long startidx, Object v) { subList(0,size()).fillRangeReducible(startidx, v); } public void move(int sidx, int eidx, int count) { checkIndexRange(size(), eidx, eidx + count); System.arraycopy(data, sidx, data, eidx, count); } public void fill(int ssidx, int seidx, Object v) { checkIndexRange(size(), ssidx, seidx); Arrays.fill(data, ssidx, seidx, Casts.longCast(v)); } public Object copyOfRange(int ssidx, int seidx) { checkIndex(ssidx, nElems); return Arrays.copyOfRange(data, ssidx, seidx); } public Object copyOf(int len) { return Arrays.copyOf(data, len); } public static LongArrayList wrap(final long[] data, int nElems, IPersistentMap m) { if (data.length < nElems) throw new RuntimeException("Array len less than required"); return new LongArrayList(data, nElems, m); } public static LongArrayList wrap(final long[] data, IPersistentMap m) { return new LongArrayList(data, data.length, m); } } public static IMutList toList(final long[] data, final int sidx, final int eidx, IPersistentMap meta) { return new LongArraySubList(data, sidx, eidx, meta); } public static IMutList toList(final long[] data) { return toList(data, 0, data.length, null); } public static float[] floatArray(int len) { return new float[len]; } public static class FloatArraySubList implements IDoubleArrayList { public final float[] data; public final int sidx; public final int dlen; public final IPersistentMap meta; public FloatArraySubList(float[] d, int s, int len, IPersistentMap _meta) { data = d; sidx = s; dlen = len; meta = _meta; } public String toString() { return Transformables.sequenceToString(this); } public boolean equals(Object other) { return equiv(other); } public int hashCode() { return hasheq(); } public IMutList cloneList() { return (IMutList)toList(Arrays.copyOfRange(data, sidx, sidx+dlen)); } public ArraySection getArraySection() { return new ArraySection(data, sidx, sidx + dlen); } public int size() { return dlen; } public Float get(int idx) { return data[checkIndex(idx, dlen) + sidx]; } public double getDouble(int idx) { return data[checkIndex(idx, dlen) + sidx];} public void setDouble(int idx, double v) { float obj = (float)v; idx = checkIndex(idx, dlen) + sidx; data[idx] = obj; } public IMutList subList(int ssidx, int seidx) { ChunkedList.sublistCheck(ssidx, seidx, size()); return toList(data, ssidx + sidx, seidx + sidx, meta()); } public IPersistentMap meta() { return meta; } public IObj withMeta(IPersistentMap m) { return (IObj)toList(data, sidx, sidx + dlen, m); } public Object[] toArray() { final int sz = size(); Object[] retval = new Object[size()]; for (int idx = 0; idx < sz; ++idx) retval[idx] = data[idx+sidx]; return retval; } public float[] toFloatArray() { return Arrays.copyOfRange(data, sidx, sidx + dlen); } public IntComparator indexComparator() { if (sidx == 0) { return new IntComparator() { public int compare(int lidx, int ridx) { return Float.compare(data[lidx], data[ridx]); } }; } else { return new IntComparator() { public int compare(int lidx, int ridx) { return Float.compare(data[lidx+sidx], data[ridx+sidx]); } }; } } public IntComparator indexComparator(Comparator c) { if (c == null) return indexComparator(); if (c instanceof DoubleComparator) { final DoubleComparator dc = (DoubleComparator)c; if (sidx == 0) { return new IntComparator() { public int compare(int lidx, int ridx) { return dc.compare(data[lidx], data[ridx]); } }; } else { return new IntComparator() { public int compare(int lidx, int ridx) { return dc.compare(data[lidx+sidx], data[ridx+sidx]); } }; } } else { return IDoubleArrayList.super.indexComparator(c); } } public static FloatComparator asFloatComparator(Comparator c) { if (c instanceof FloatComparator) return (FloatComparator)c; else if (c instanceof DoubleComparator) { final DoubleComparator lc = (DoubleComparator)c; return new FloatComparator() { public int compare(float l, float r) { return lc.compare(l,r); } }; } return null; } @SuppressWarnings("unchecked") public void sort(Comparator c) { if(c == null) { FloatArrays.parallelQuickSort(data, sidx, sidx+dlen); } else { FloatComparator fc = asFloatComparator(c); if (fc != null) FloatArrays.parallelQuickSort(data, sidx, sidx+dlen, fc); else IDoubleArrayList.super.sort(c); } } public void shuffle(Random r) { FloatArrays.shuffle(data, sidx, sidx+dlen, r); } @SuppressWarnings("unchecked") public int binarySearch(Object v, Comparator c) { final float vv = RT.floatCast(Casts.doubleCast(v)); final FloatComparator bc = asFloatComparator(c); if(c == null || bc != null) return fixSubArrayBinarySearch(sidx, size(), bc == null ? FloatArrays.binarySearch(data, sidx, sidx+size(), vv) : FloatArrays.binarySearch(data, sidx, sidx+size(), vv, bc)); return IDoubleArrayList.super.binarySearch(v, c); } public Object doubleReduction(IFn.ODO rfn, Object init) { final int es = sidx + dlen; final float[] d = data; for(int ss = sidx; ss < es && !RT.isReduced(init); ++ss) init = rfn.invokePrim(init, d[ss]); return Reductions.unreduce(init); } public void fillRangeReducible(long startidx, Object v) { final ArraySection as = getArraySection(); if (!fillRangeArrayCopy(as.array, as.sidx, as.eidx, startidx, v)) { final int ee = sidx + size(); final float[] d = data; Reductions.serialReduction(new Reductions.IndexedDoubleAccum( startidx+sidx, new IFn.OLDO() { public Object invokePrim(Object acc, long idx, double v) { if(idx >= ee) throw new IndexOutOfBoundsException("Index " + String.valueOf(idx - sidx) + " is out of range: " + String.valueOf(size())); d[(int)idx] = (float)v; return d; }}), data, v); } } public void move(int sidx, int eidx, int count) { checkIndexRange(size(), eidx, eidx + count); System.arraycopy(data, sidx, data, eidx, count); } public void fill(int ssidx, int seidx, Object v) { checkIndexRange(size(), ssidx, seidx); Arrays.fill(data, ssidx, seidx, (float)Casts.doubleCast(v)); } public Object copyOfRange(int ssidx, int seidx) { checkIndex(ssidx, dlen); return Arrays.copyOfRange(data, sidx+ssidx, sidx+seidx); } public Object copyOf(int len) { return Arrays.copyOfRange(data, sidx, sidx+len); } } public static IMutList toList(final float[] data, final int sidx, final int eidx, IPersistentMap meta) { final int dlen = eidx - sidx; return new FloatArraySubList(data, sidx, dlen, meta); } public static IMutList toList(final float[] data) { return toList(data, 0, data.length, null); } public static double[] doubleArray(int len) { return new double[len]; } public static class DoubleArraySubList implements IDoubleArrayList { public final double[] data; public final int sidx; public final int eidx; public final int nElems; public final IPersistentMap meta; public DoubleArraySubList(double[] d, int _sidx, int _eidx, IPersistentMap m) { data = d; sidx = _sidx; eidx = _eidx; nElems = eidx - sidx; meta = m; } public String toString() { return Transformables.sequenceToString(this); } public boolean equals(Object other) { return equiv(other); } public int hashCode() { return hasheq(); } public IMutList cloneList() { return (IMutList)toList(Arrays.copyOfRange(data, sidx, eidx)); } public ArraySection getArraySection() { return new ArraySection(data, sidx, eidx); } public Class containedType() { return data.getClass().getComponentType(); } public int size() { return nElems; } public double getDouble(int idx) { return data[checkIndex(idx, nElems) + sidx]; } public Object get(int idx) { return data[checkIndex(idx, nElems) + sidx]; } public Object nth(int idx) { if(idx < 0) idx += nElems; return data[checkIndex(idx, nElems) + sidx]; } static void setDouble(final double[] d, final int sidx, final int nElems, int idx, final double v) { idx = checkIndex(idx, nElems) + sidx; d[idx] = v; } public void setDouble(int idx, double obj) { setDouble(data, sidx, nElems, idx, obj); } public IMutList subList(int ssidx, int seidx) { ChunkedList.sublistCheck(ssidx, seidx, size()); return toList(data, ssidx + sidx, seidx + sidx, meta()); } public IPersistentMap meta() { return meta; } public IObj withMeta(IPersistentMap m) { return (IObj)toList(data, sidx, eidx, m); } public Object[] toArray() { final double[] d = data; final int ss = sidx; final int ne = nElems; Object[] retval = new Object[ne]; for(int idx = 0; idx < ne; ++idx) retval[idx] = d[idx+ss]; return retval; } public double[] toDoubleArray() { return Arrays.copyOfRange(data, sidx, eidx); } @SuppressWarnings("unchecked") public static IntComparator indexComparator(final double[] d, final int sidx, final Comparator comp) { if (comp == null) { if (sidx != 0) { return new IntComparator() { public int compare(int lidx, int ridx) { return Double.compare(d[lidx+sidx], d[ridx+sidx]); } }; } else { return new IntComparator() { public int compare(int lidx, int ridx) { return Double.compare(d[lidx], d[ridx]); } }; } } else { if (comp instanceof DoubleComparator) { final DoubleComparator dc = (DoubleComparator)comp; if(sidx != 0) { return new IntComparator() { public int compare(int lidx, int ridx) { return dc.compare(d[lidx+sidx], d[ridx+sidx]); } }; } else { return new IntComparator() { public int compare(int lidx, int ridx) { return dc.compare(d[lidx], d[ridx]); } }; } } else { return new IntComparator() { public int compare(int lidx, int ridx) { return comp.compare(d[lidx+sidx], d[ridx+sidx]); } }; } } } public IntComparator indexComparator() { return indexComparator(data, sidx, null); } public IntComparator indexComparator(Comparator c) { return indexComparator(data, sidx, c); } @SuppressWarnings("unchecked") public static DoubleComparator toDoubleComparator(Comparator c) { if (c instanceof DoubleComparator) return (DoubleComparator) c; return null; } public void sort(Comparator c) { if(c == null) DoubleArrays.parallelQuickSort(data, sidx, eidx); else { DoubleComparator dc = toDoubleComparator(c); if (dc != null) { DoubleArrays.parallelQuickSort(data, sidx, eidx, toDoubleComparator(c)); } else { IDoubleArrayList.super.sort(c); } } } public int[] sortIndirect(Comparator c) { final int sz = size(); int[] retval = iarange(0, sz, 1); if(sz < 2) return retval; if(c == null) DoubleArrays.parallelQuickSortIndirect(retval, data, sidx, eidx); else IntArrays.parallelQuickSort(retval, indexComparator(c)); return retval; } public void shuffle(Random r) { DoubleArrays.shuffle(data, sidx, eidx, r); } public static DoubleComparator asDoubleComparator(Comparator c) { if (c instanceof DoubleComparator) return (DoubleComparator)c; return null; } @SuppressWarnings("unchecked") public int binarySearch(Object v, Comparator c) { final double vv = RT.doubleCast(Casts.doubleCast(v)); final DoubleComparator bc = asDoubleComparator(c); if(c == null || bc != null) return fixSubArrayBinarySearch(sidx, size(), bc == null ? DoubleArrays.binarySearch(data, sidx, sidx+size(), vv) : DoubleArrays.binarySearch(data, sidx, sidx+size(), vv, bc)); return IDoubleArrayList.super.binarySearch(v, c); } public Object doubleReduction(IFn.ODO rfn, Object init) { final int es = eidx; final double[] d = data; for(int ss = sidx; ss < es && !RT.isReduced(init); ++ss) init = rfn.invokePrim(init, d[ss]); return Reductions.unreduce(init); } public void fillRangeReducible(long startidx, Object v) { final ArraySection as = getArraySection(); if (!fillRangeArrayCopy(as.array, as.sidx, as.eidx, startidx, v)) { final int ee = sidx + size(); final double[] d = data; Reductions.serialReduction(new Reductions.IndexedDoubleAccum( startidx+sidx, new IFn.OLDO() { public Object invokePrim(Object acc, long idx, double v) { if(idx >= ee) throw new IndexOutOfBoundsException("Index " + String.valueOf(idx - sidx) + "> length " + String.valueOf(size())); d[(int)idx] = v; return d; }}), data, v); } } public void move(int sidx, int eidx, int count) { checkIndexRange(size(), eidx, eidx + count); System.arraycopy(data, sidx, data, eidx, count); } public void fill(int ssidx, int seidx, Object v) { checkIndexRange(size(), ssidx, seidx); Arrays.fill(data, sidx + ssidx, sidx + seidx, Casts.doubleCast(v)); } public Object copyOfRange(int ssidx, int seidx) { checkIndex(ssidx, nElems); return Arrays.copyOfRange(data, sidx + ssidx, sidx + seidx); } public Object copyOf(int len) { return Arrays.copyOfRange(data, sidx, sidx + len); } } public static class DoubleArrayList implements IDoubleArrayList { double[] data; int nElems; IPersistentMap meta; public DoubleArrayList(double[] d, int ne, IPersistentMap meta) { data = d; nElems = ne; } public DoubleArrayList(int capacity) { this(new double[capacity], 0, null); } public DoubleArrayList() { this(4); } public String toString() { return Transformables.sequenceToString(this); } public boolean equals(Object other) { return equiv(other); } public int hashCode() { return hasheq(); } public IMutList cloneList() { return new DoubleArrayList((double[])copyOf(nElems), nElems, meta); } public ArraySection getArraySection() { return new ArraySection(data, 0, nElems); } public Class containedType() { return data.getClass().getComponentType(); } public void clear() { nElems = 0; } public int size() { return nElems; } public void setSize(int sz) { nElems = sz; } public double getDouble(int idx) { return data[checkIndex(idx, nElems)]; } public void setDouble(int idx, double obj) { DoubleArraySubList.setDouble(data, 0, nElems, idx, obj); } public int capacity() { return data.length; } public double[] ensureCapacity(int len) { double[] d = data; if (len >= d.length) { d = data = Arrays.copyOf(d, len < 100000 ? len * 2 : (int)(len * 1.5)); } return d; } public void addDouble(double obj) { final int ne = nElems; final double[] d = ensureCapacity(ne+1); d[ne] = obj; nElems = ne+1; } public boolean add(Object obj) { addDouble(Casts.doubleCast(obj)); return true; } public void add(int idx, Object obj) { if (idx == nElems) { add(obj); return; } idx = wrapCheckIndex(idx, nElems); final double val = Casts.doubleCast(obj); final int ne = nElems; final double[] d = ensureCapacity(ne+1); System.arraycopy(d, idx, d, idx+1, ne - idx); d[idx] = val; nElems = ne+1; } public boolean addAllReducible(Object c) { final int sz = size(); if (c instanceof RandomAccess) { final List cl = (List) c; if (cl.isEmpty() ) return false; final int cs = cl.size(); ensureCapacity(cs+sz); nElems += cs; //Hit fastpath fillRangeReducible(sz, cl); } else { IDoubleArrayList.super.addAllReducible(c); } return sz != size(); } public boolean addAll(int sidx, Collection c) { sidx = wrapCheckIndex(sidx, nElems); if (c.isEmpty()) return false; final int cs = c.size(); final int sz = size(); final int eidx = sidx + cs; ensureCapacity(cs+sz); nElems += cs; System.arraycopy(data, sidx, data, eidx, sz - sidx); if (c instanceof List) { //Hit fastpath fillRangeReducible(sidx, (List)c); } else { int idx = sidx; for(Object o: c) { set(idx, o); ++idx; } } return true; } public Object remove(int idx) { idx = wrapCheckIndex(idx, nElems); final int ne = nElems; final int nne = ne - 1; final double[] d = data; final double retval = d[idx]; if (idx != nne) { final int copyLen = ne - idx - 1; System.arraycopy(d, idx+1, d, idx, copyLen); } --nElems; return retval; } public IMutList subList(int ssidx, int seidx) { ChunkedList.sublistCheck(ssidx, seidx, size()); return toList(data, ssidx, seidx, meta()); } public void fillRangeReducible(long startidx, List v) { ((IMutList)subList(0, nElems)).fillRangeReducible(startidx, v); } public IPersistentMap meta() { return meta; } public IObj withMeta(IPersistentMap m) { meta = m; return this; } public Object[] toArray() { return subList(0, nElems).toArray(); } public double[] toDoubleArray() { return Arrays.copyOf(data, nElems); } public void removeRange(int startidx, int endidx) { checkIndexRange(size(), startidx, endidx); System.arraycopy(data, startidx, data, endidx, nElems - endidx); nElems -= endidx - startidx; } public Object reduce(IFn fn) { return ((IReduce)subList(0, nElems)).reduce(fn); } public Object reduce(IFn fn, Object init) { return ((IReduceInit)subList(0, nElems)).reduce(fn,init); } public Object doubleReduction(IFn.ODO fn, Object init) { return ((DoubleMutList)subList(0, nElems)).doubleReduction(fn, init); } public IntComparator indexComparator() { return DoubleArraySubList.indexComparator(data, 0, null); } public IntComparator indexComparator(Comparator c) { return DoubleArraySubList.indexComparator(data, 0, c); } public void sort(Comparator c) { subList(0, nElems).sort(c); } public void shuffle(Random r) { ((IMutList)subList(0, nElems)).shuffle(r); } @SuppressWarnings("unchecked") public int binarySearch(Object v, Comparator c) { return ((IMutList)subList(0, nElems)).binarySearch(v, c); } public int[] sortIndirect(Comparator c) { return ((IMutList)subList(0, nElems)).sortIndirect(c); } public void doubleForEach(DoubleConsumer c) { final int es = nElems; final double[] d = data; for(int ss = 0; ss < es; ++ss) c.accept(d[ss]); } public void move(int sidx, int eidx, int count) { checkIndexRange(size(), eidx, eidx + count); System.arraycopy(data, sidx, data, eidx, count); } public void fill(int ssidx, int seidx, Object v) { checkIndexRange(size(), ssidx, seidx); Arrays.fill(data, ssidx, seidx, Casts.doubleCast(v)); } public Object copyOfRange(int ssidx, int seidx) { checkIndex(ssidx, nElems); return Arrays.copyOfRange(data, ssidx, seidx); } public Object copyOf(int len) { return Arrays.copyOf(data, len); } public static DoubleArrayList wrap(final double[] data, int nElems, IPersistentMap m) { if (data.length < nElems) throw new RuntimeException("Array len less than required"); return new DoubleArrayList(data, nElems, m); } public static DoubleArrayList wrap(final double[] data, IPersistentMap m) { return new DoubleArrayList(data, data.length, m); } } public static IMutList toList(final double[] data, final int sidx, final int eidx, IPersistentMap meta) { return new DoubleArraySubList(data, sidx, eidx, meta); } public static IMutList toList(final double[] data) { return toList(data, 0, data.length, null); } public static class CharArraySubList implements ILongArrayList { public final char[] data; public final int sidx; public final int dlen; public final IPersistentMap meta; public CharArraySubList(char[] d, int s, int len, IPersistentMap m) { data = d; sidx = s; dlen = len; meta = m; } public String toString() { return Transformables.sequenceToString(this); } public boolean equals(Object other) { return equiv(other); } public int hashCode() { return hasheq(); } public ArraySection getArraySection() { return new ArraySection(data, sidx, sidx + dlen); } public int size() { return dlen; } public Character set(int idx, Object obj) { idx = checkIndex(idx, dlen); char rv = data[idx]; data[idx] = Casts.charCast(obj); return rv; } public Character get(int idx) { return data[checkIndex(idx, dlen) + sidx]; } public long getLong(int idx) { return data[checkIndex(idx, dlen) + sidx]; } public void setLong(int idx, long obj) { char v = Casts.charCast(obj); idx = checkIndex(idx, dlen) + sidx; data[idx] = v; } public IMutList subList(int ssidx, int seidx) { ChunkedList.sublistCheck(ssidx, seidx, size()); return toList(data, ssidx + sidx, seidx + sidx, meta()); } public IPersistentMap meta() { return meta; } public IObj withMeta(IPersistentMap m) { return (IObj)toList(data, sidx, sidx + dlen, m); } public Object[] toArray() { final int sz = size(); Object[] retval = new Object[size()]; for (int idx = 0; idx < sz; ++idx) retval[idx] = data[idx+sidx]; return retval; } public static CharComparator asCharComparator(Comparator c) { if (c instanceof CharComparator) return (CharComparator)c; else if (c instanceof LongComparator) { final LongComparator lc = (LongComparator)c; return new CharComparator() { public int compare(char l, char r) { return lc.compare(l,r); } }; } return null; } @SuppressWarnings("unchecked") public void sort(Comparator c) { if(c==null) CharArrays.parallelQuickSort(data, sidx, sidx+dlen); else { CharComparator cc = asCharComparator(c); if( cc != null) CharArrays.parallelQuickSort(data, sidx, sidx+dlen, cc); else ILongArrayList.super.sort(c); } } public void shuffle(Random r) { CharArrays.shuffle(data, sidx, sidx+dlen, r); } @SuppressWarnings("unchecked") public int binarySearch(Object v, Comparator c) { final char vv = RT.charCast(Casts.longCast(v)); final CharComparator bc = asCharComparator(c); if(c == null || bc != null) return fixSubArrayBinarySearch(sidx, size(), bc == null ? CharArrays.binarySearch(data, sidx, sidx+size(), vv) : CharArrays.binarySearch(data, sidx, sidx+size(), vv, bc)); return ILongArrayList.super.binarySearch(v, c); } public Object reduce(IFn rfn, Object init) { final int sz = size(); for (int idx = 0; idx < sz && !RT.isReduced(init); ++idx) init = rfn.invoke(init, data[idx+sidx]); return Reductions.unreduce(init); } public void fillRangeReducible(long startidx, Object v) { final ArraySection as = getArraySection(); if (!fillRangeArrayCopy(as.array, as.sidx, as.eidx, startidx, v)) { final int ss = (int)startidx + sidx; final int ee = sidx + size(); Reductions.serialReduction(new Reductions.IndexedAccum( startidx+sidx, new IFn.OLOO() { public Object invokePrim(Object acc, long idx, Object v) { if(idx >= ee) throw new IndexOutOfBoundsException("Index " + String.valueOf(idx - sidx) + " is out of range: " + String.valueOf(size())); data[(int)idx] = Casts.charCast(v); return data; }}), data, v); } } public void move(int sidx, int eidx, int count) { checkIndexRange(size(), eidx, eidx + count); System.arraycopy(data, sidx, data, eidx, count); } public void fill(int ssidx, int seidx, Object v) { checkIndexRange(size(), ssidx, seidx); Arrays.fill(data, ssidx+sidx, seidx+sidx, RT.charCast(Casts.longCast((v==null)?0:v))); } public Object copyOfRange(int ssidx, int seidx) { checkIndex(ssidx, size()); return Arrays.copyOfRange(data, ssidx+sidx, seidx+sidx); } public Object copyOf(int len) { return Arrays.copyOfRange(data, sidx, sidx+len); } } public static IMutList toList(final char[] data, final int sidx, final int eidx, IPersistentMap meta) { final int dlen = eidx - sidx; return new CharArraySubList(data, sidx, dlen, meta); } public static IMutList toList(final char[] data) { return toList(data, 0, data.length, null); } public static class BooleanArraySubList implements IArrayList { public final boolean[] data; public final int sidx; public final int dlen; public final IPersistentMap meta; public BooleanArraySubList(boolean[] d, int s, int len, IPersistentMap m) { data = d; sidx = s; dlen = len; meta = m; } public String toString() { return Transformables.sequenceToString(this); } public boolean equals(Object other) { return equiv(other); } public int hashCode() { return hasheq(); } public ArraySection getArraySection() { return new ArraySection(data, sidx, sidx + dlen); } public int size() { return dlen; } public boolean getBoolean(int idx) { return data[checkIndex(idx, dlen)+sidx]; } public void setBoolean(int idx, boolean obj) { data[checkIndex(idx, dlen)+sidx] = obj; } public long getLong(int idx) { return Casts.longCast(data[checkIndex(idx, dlen)+sidx]); } public void setLong(int idx, long obj) { data[checkIndex(idx, dlen)+sidx] = Casts.booleanCast(obj); } public Object get(int idx) { return getBoolean(idx); } public Object set(int idx, Object v) { final boolean retval = getBoolean(idx); setBoolean(idx, Casts.booleanCast(v)); return retval; } public IMutList cloneList() { return (IMutList)toList(Arrays.copyOfRange(data, sidx, sidx+dlen)); } public IMutList subList(int ssidx, int seidx) { ChunkedList.sublistCheck(ssidx, seidx, size()); return toList(data, ssidx + sidx, seidx + sidx, meta()); } public IPersistentMap meta() { return meta; } public IObj withMeta(IPersistentMap m) { return (IObj)toList(data, sidx, sidx + dlen, m); } public Object[] toArray() { final int sz = size(); Object[] retval = new Object[size()]; for (int idx = 0; idx < sz; ++idx) retval[idx] = data[idx+sidx]; return retval; } public void fillRangeReducible(long startidx, Object v) { final ArraySection as = getArraySection(); if (!fillRangeArrayCopy(as.array, as.sidx, as.eidx, startidx, v)) { final int ss = (int)startidx + sidx; final int ee = sidx + size(); Reductions.serialReduction(new Reductions.IndexedAccum( startidx+sidx, new IFn.OLOO() { public Object invokePrim(Object acc, long idx, Object v) { if(idx >= ee) throw new IndexOutOfBoundsException("Index " + String.valueOf(idx - sidx) + " is out of range: " + String.valueOf(size())); data[(int)idx] = Casts.booleanCast(v); return data; }}), data, v); } } public void move(int sidx, int eidx, int count) { checkIndexRange(size(), eidx, eidx + count); System.arraycopy(data, sidx, data, eidx, count); } public void fill(int ssidx, int seidx, Object v) { checkIndexRange(size(), ssidx, seidx); Arrays.fill(data, sidx+ssidx, sidx+seidx, Casts.booleanCast(v)); } public Object copyOfRange(int ssidx, int seidx) { checkIndexRange(size(), ssidx, seidx); return Arrays.copyOfRange(data, sidx+ssidx, sidx+seidx); } public Object copyOf(int len) { return Arrays.copyOfRange(data, sidx, sidx+len); } } public static IMutList toList(final boolean[] data, final int sidx, final int eidx, IPersistentMap meta) { final int dlen = eidx - sidx; return new BooleanArraySubList(data, sidx, dlen, meta); } public static IMutList toList(final boolean[] data) { return toList(data, 0, data.length, null); } public static IMutList toList(Object obj, int sidx, int eidx, IPersistentMap meta) { if (obj == null) return null; Class cls = obj.getClass(); if(!cls.isArray()) throw new RuntimeException("Object is not an array: " + String.valueOf(obj)); if(obj instanceof Object[]) return toList((Object[])obj, sidx, eidx, meta); else if (cls == long[].class) return toList((long[])obj, sidx, eidx, meta); else if (cls == double[].class) return toList((double[])obj, sidx, eidx, meta); else if (cls == byte[].class) return toList((byte[])obj, sidx, eidx, meta); else if (cls == short[].class) return toList((short[])obj, sidx, eidx, meta); else if (cls == int[].class) return toList((int[])obj, sidx, eidx, meta); else if (cls == float[].class) return toList((float[])obj, sidx, eidx, meta); else if (cls == char[].class) return toList((char[])obj, sidx, eidx, meta); else if (cls == boolean[].class) return toList((boolean[])obj, sidx, eidx, meta); else throw new RuntimeException("Invalid array type."); } public static IMutList toList(Object obj) { if (obj == null) return null; Class cls = obj.getClass(); if(!cls.isArray()) throw new RuntimeException("Object is not an array: " + String.valueOf(obj)); return toList(obj, 0, Array.getLength(obj), null); } @SuppressWarnings("unchecked") public static IMutList toList(ArraySection data) { if(data == null) return null; if(data instanceof IMutList) return (IMutList)data; return toList(data.array, data.sidx, data.eidx, null); } public static int[] iarange(int start, int end, int step) { final int len = (end - start)/step; if (len < 0 ) throw new RuntimeException("Invalid range - start: " + String.valueOf(start) + " end: " + String.valueOf(end) + " step: " + String.valueOf(step)); final int[] retval = new int[len]; for(int idx = 0; idx < len; ++idx) { retval[idx] = start + idx * step; } return retval; } public static long[] larange(long start, long end, long step) { final int len = RT.intCast((end - start)/step); if (len < 0 ) throw new RuntimeException("Invalid range."); final long[] retval = new long[len]; for(int idx = 0; idx < len; ++idx) { retval[idx] = start + idx * step; } return retval; } public static double[] darange(double start, double end, double step) { final int len = RT.intCast((end - start)/step); if (len < 0 ) throw new RuntimeException("Invalid range."); final double[] retval = new double[len]; for(int idx = 0; idx < len; ++idx) { retval[idx] = start + idx * step; } return retval; } } ================================================ FILE: java/ham_fisted/ArraySection.java ================================================ package ham_fisted; public class ArraySection { public final Object array; public final int sidx; public final int eidx; public ArraySection(Object ary, int _sidx, int _eidx) { if(! (_eidx >= _sidx)) throw new RuntimeException("End index: " + String.valueOf(_eidx) + " is not >= start index: " + String.valueOf(_sidx)); array = ary; sidx = _sidx; eidx = _eidx; } public ArraySection(ArraySection other) { this(other.array, other.sidx, other.eidx); } public int size() { return eidx - sidx; } public String toString() { return "ArraySection<" + array.getClass().getComponentType().getCanonicalName() + ">[" + String.valueOf(sidx) + ":" + String.valueOf(eidx) + "]"; } } ================================================ FILE: java/ham_fisted/BatchReducer.java ================================================ package ham_fisted; import clojure.lang.Sequential; import clojure.lang.IDeref; import clojure.lang.IFn; import clojure.lang.RT; import java.util.Iterator; import java.util.List; import java.util.function.Consumer; public class BatchReducer implements IFnDef.O, ITypedReduce, Sequential, Iterable { final IFn batchSrc; List batchData; int idx; int nElems; public BatchReducer(IFn batchSrc) { this.batchSrc = batchSrc; batchData = null; idx = 0; nElems = 0; } public Object reduce(IFn rfn, Object acc) { List bd = batchData; int ix = idx; int ne = nElems; do { if(ix == ne) { bd = (List)batchSrc.invoke(); ix = 0; ne = bd.size(); } acc = rfn.invoke(acc, bd.get(ix++)); } while(!RT.isReduced(acc)); batchData = bd; idx = ix; nElems = ne; return ((IDeref)acc).deref(); } public Object invoke() { if(idx == nElems) { batchData = (List)batchSrc.invoke(); idx = 0; nElems = batchData.size(); } return batchData.get(idx++); } public Iterator iterator() { final IFn invoker = this; return new Iterator() { public boolean hasNext() { return true; } public Object next() { return invoker.invoke(); } }; } @SuppressWarnings("unchecked") public void forEach(Consumer c) { ITypedReduce.super.forEach(c); } } ================================================ FILE: java/ham_fisted/BatchedList.java ================================================ package ham_fisted; import clojure.lang.IDeref; public class BatchedList implements IMutList, IDeref { public static final int tailWidth = 64; public static final int leafWidth = 64; Object[] tail = new Object[tailWidth]; int nTail; Object[][] leafTail = new Object[leafWidth][]; int nLeafTail; public static class Link { public final Object[][] leaf; public Link next; public Link(Object[][] leaf) { this.leaf = leaf; this.next = null; } } Link first; Link last; int count; public BatchedList() { first = null; last = null; count = 0; } public boolean add(Object obj) { if(nTail == tailWidth) { if(nLeafTail == leafWidth) { Link l = new Link(leafTail); leafTail = new Object[leafWidth][]; nLeafTail = 0; if(first == null) first = l; if(last != null) last.next = l; last = l; } else { leafTail[nLeafTail++] = tail; } tail = new Object[tailWidth]; nTail = 0; } tail[nTail++] = obj; ++count; return true; } public Object get(int idx) { throw new RuntimeException(); } public int count() { return count; } public int size() { return count; } public void clear() { nTail = 0; first = null; last = null; count = 0; } public TreeList deref() { return TreeList.EMPTY; } } ================================================ FILE: java/ham_fisted/BiFunctions.java ================================================ package ham_fisted; import java.util.function.BiFunction; public class BiFunctions { public static final BiFunction incBiFn = (k, v)-> v == null ? Long.valueOf(1) : Long.valueOf(((long)v) + 1); public static final BiFunction rhsWins = (v1,v2)->v2; } ================================================ FILE: java/ham_fisted/BlockSplitBloomFilter.java ================================================ /* * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * to you under the Apache License, Version 2.0 (the * "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, * software distributed under the License is distributed on an * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * KIND, either express or implied. See the License for the * specific language governing permissions and limitations * under the License. */ //package org.apache.parquet.column.values.bloomfilter; package ham_fisted; import java.io.ByteArrayOutputStream; import java.io.IOException; import java.io.OutputStream; import java.nio.ByteBuffer; import java.nio.ByteOrder; import java.nio.IntBuffer; import java.util.Arrays; import net.openhft.hashing.LongHashFunction; /* * This Bloom filter is implemented using block-based Bloom filter algorithm from Putze et al.'s * "Cache-, Hash- and Space-Efficient Bloom filters". The basic idea is to hash the item to a tiny * Bloom filter which size fit a single cache line or smaller. This implementation sets 8 bits in * each tiny Bloom filter. Each tiny Bloom filter is 32 bytes to take advantage of 32-byte SIMD * instruction. */ public class BlockSplitBloomFilter { // Bytes in a tiny Bloom filter block. private static final int BYTES_PER_BLOCK = 32; // Bits in a tiny Bloom filter block. private static final int BITS_PER_BLOCK = 256; // The lower bound of bloom filter size, set to the size of a tiny Bloom filter block. public static final int LOWER_BOUND_BYTES = 32; // The upper bound of bloom filter size, set to default row group size. public static final int UPPER_BOUND_BYTES = 128 * 1024 * 1024; // The number of bits to set in a tiny Bloom filter private static final int BITS_SET_PER_BLOCK = 8; // The metadata in the header of a serialized Bloom filter is four four-byte values: the number of bytes, // the filter algorithm, the hash algorithm, and the compression. public static final int HEADER_SIZE = 16; // The default false positive probability value public static final double DEFAULT_FPP = 0.01; // The underlying byte array for Bloom filter bitset. private byte[] bitset; // A integer array buffer of underlying bitset to help setting bits. private IntBuffer intBuffer; private int maximumBytes = UPPER_BOUND_BYTES; private int minimumBytes = LOWER_BOUND_BYTES; private int[] mask = new int[BITS_SET_PER_BLOCK]; // The block-based algorithm needs 8 odd SALT values to calculate eight indexes // of bits to set, one per 32-bit word. private static final int[] SALT = { 0x47b6137b, 0x44974d91, 0x8824ad5b, 0xa2b7289d, 0x705495c7, 0x2df1424b, 0x9efc4947, 0x5c6bfb31 }; public BlockSplitBloomFilter(int numBytes) { this.minimumBytes = LOWER_BOUND_BYTES; this.maximumBytes = UPPER_BOUND_BYTES; initBitset(numBytes); } public byte[] bitset() { return bitset; } /** * Construct the Bloom filter with given bitset, it is used when reconstructing * Bloom filter from parquet file. * * @param bitset The given bitset to construct Bloom filter. * @param hashStrategy The hash strategy Bloom filter apply. */ public BlockSplitBloomFilter(byte[] bitset) { if (bitset == null) { throw new RuntimeException("Given bitset is null"); } this.bitset = bitset; this.intBuffer = ByteBuffer.wrap(bitset).order(ByteOrder.LITTLE_ENDIAN).asIntBuffer(); } /** * Create a new bitset for Bloom filter. * * @param numBytes The number of bytes for Bloom filter bitset. The range of num_bytes should be within * [minimumBytes, maximumBytes], it will be rounded up/down * to lower/upper bound if num_bytes is out of range and also will rounded up to a power * of 2. It uses XXH64 as its default hash function and block-based algorithm * as default algorithm. */ private void initBitset(int numBytes) { if (numBytes < minimumBytes) { numBytes = minimumBytes; } // Get next power of 2 if it is not power of 2. if ((numBytes & (numBytes - 1)) != 0) { numBytes = Integer.highestOneBit(numBytes) << 1; } if (numBytes > maximumBytes || numBytes < 0) { numBytes = maximumBytes; } this.bitset = new byte[numBytes]; this.intBuffer = ByteBuffer.wrap(bitset).order(ByteOrder.LITTLE_ENDIAN).asIntBuffer(); } private int[] setMask(int key) { // The following three loops are written separately so that they could be // optimized for vectorization. for (int i = 0; i < BITS_SET_PER_BLOCK; ++i) { mask[i] = key * SALT[i]; } for (int i = 0; i < BITS_SET_PER_BLOCK; ++i) { mask[i] = mask[i] >>> 27; } for (int i = 0; i < BITS_SET_PER_BLOCK; ++i) { mask[i] = 0x1 << mask[i]; } return mask; } public void insertHash(long hash) { long numBlocks = bitset.length / BYTES_PER_BLOCK; long lowHash = hash >>> 32; int blockIndex = (int) ((lowHash * numBlocks) >> 32); int key = (int) hash; // Calculate mask for bucket. int[] mask = setMask(key); for (int i = 0; i < BITS_SET_PER_BLOCK; i++) { int value = intBuffer.get(blockIndex * (BYTES_PER_BLOCK / 4) + i); value |= mask[i]; intBuffer.put(blockIndex * (BYTES_PER_BLOCK / 4) + i, value); } } public boolean findHash(long hash) { long numBlocks = bitset.length / BYTES_PER_BLOCK; long lowHash = hash >>> 32; int blockIndex = (int) ((lowHash * numBlocks) >> 32); int key = (int) hash; // Calculate mask for the tiny Bloom filter. int[] mask = setMask(key); for (int i = 0; i < BITS_SET_PER_BLOCK; i++) { if (0 == (intBuffer.get(blockIndex * (BYTES_PER_BLOCK / 4) + i) & mask[i])) { return false; } } return true; } private static void checkArgument(boolean arg, String message) { if(!arg) throw new RuntimeException(message); } /** * Calculate optimal size according to the number of distinct values and false positive probability. * * @param n: The number of distinct values. * @param p: The false positive probability. * @return optimal number of bits of given n and p. */ public static int optimalNumOfBits(long n, double p) { checkArgument((p > 0.0 && p < 1.0), "FPP should be less than 1.0 and great than 0.0"); final double m = -8 * n / Math.log(1 - Math.pow(p, 1.0 / 8)); int numBits = (int) m; // Handle overflow. if (numBits > UPPER_BOUND_BYTES << 3 || m < 0) { numBits = UPPER_BOUND_BYTES << 3; } // Round numBits up to (k * BITS_PER_BLOCK) numBits = (numBits + BITS_PER_BLOCK - 1) & ~BITS_PER_BLOCK; if (numBits < (LOWER_BOUND_BYTES << 3)) { numBits = LOWER_BOUND_BYTES << 3; } return numBits; } public int getBitsetSize() { return this.bitset.length; } public static long hash(byte[] input) { return LongHashFunction.xx().hashBytes(input); } public boolean equals(Object object) { if (object == this) { return true; } if (object instanceof BlockSplitBloomFilter) { BlockSplitBloomFilter that = (BlockSplitBloomFilter) object; return Arrays.equals(this.bitset, that.bitset); } return false; } public boolean canMergeFrom(BlockSplitBloomFilter otherBloomFilter) { return otherBloomFilter != null && getBitsetSize() == otherBloomFilter.getBitsetSize(); } public void merge(BlockSplitBloomFilter otherBloomFilter) throws IOException { checkArgument(otherBloomFilter != null, "The BloomFilter to merge shouldn't be null"); checkArgument( canMergeFrom(otherBloomFilter), "BloomFilters must have the same size of bitset."); byte[] otherBits = otherBloomFilter.bitset; for (int i = 0; i < otherBits.length; i++) { bitset[i] |= otherBits[i]; } } } ================================================ FILE: java/ham_fisted/Casts.java ================================================ package ham_fisted; import clojure.lang.Util; import clojure.lang.RT; public class Casts { public static boolean booleanCast(Object obj) { if(obj == null) return false; if(obj instanceof Boolean) return (Boolean)obj; if(obj instanceof Number) { final Number nobj = (Number)obj; return Util.isInteger(obj) ? nobj.longValue() != 0 : booleanCast(nobj.doubleValue()); } if(obj instanceof Character) return ((Character)obj).charValue() != 0; return true; } public static boolean booleanCast(long obj) { return obj != 0; } public static boolean booleanCast(double obj) { return Double.isNaN(obj) ? false : obj != 0.0; } public static boolean booleanCast(float obj) { return Float.isNaN(obj) ? false : obj != 0.0f; } public static boolean booleanCast(boolean obj) { return obj; } public static char charCast(double obj) { if(!Double.isFinite(obj)) throw new RuntimeException("Non-finite double cannot be casted to long: " + String.valueOf(obj)); return RT.charCast(obj); } public static char charCast(float obj) { if(!Float.isFinite(obj)) throw new RuntimeException("Non-finite float cannot be casted to long: " + String.valueOf(obj)); return RT.charCast(obj); } public static char charCast(long obj) { return RT.charCast(obj); } public static char charCast(Object obj) { if(obj == null) return (char)0; else if (obj instanceof Boolean) return ((Boolean)obj) ? (char)1 : (char)0; else if (obj instanceof Number) { final Number nobj = (Number)obj; if(obj instanceof Double) return charCast(nobj.doubleValue()); else if(obj instanceof Float) return charCast(nobj.floatValue()); else return charCast(nobj.longValue()); } return RT.charCast(obj); } public static long charLongCast(Object obj) { return (long)charCast(obj); } public static long longCast(Object obj) { if (obj instanceof Long) return (long)obj; if(obj instanceof Integer) return (int)obj; if (obj instanceof Boolean) return ((Boolean)obj) ? 1 : 0; else if (obj instanceof Double) return longCast(((Double)obj).doubleValue()); else if (obj instanceof Float) return longCast(((Float)obj).floatValue()); else if (obj instanceof Number) return RT.longCast(obj); else if (obj instanceof Character) return (long)(Character)obj; else throw new RuntimeException("Object cannot be casted to long: " + obj); } public static long longCast(long obj) { return obj; } public static long longCast(char obj) { return obj; } public static long longCast(double obj) { if(!Double.isFinite(obj)) throw new RuntimeException("Non-finite double cannot be casted to long: " + String.valueOf(obj)); return (long)obj; } public static long longCast(float obj) { if(!Float.isFinite(obj)) throw new RuntimeException("Non-finite float cannot be casted to long: " + String.valueOf(obj)); return (long)obj; } public static long longCast(boolean obj) { return obj ? 1 : 0; } public static double doubleCast(Object obj) { if (obj == null) return Double.NaN; if (obj instanceof Double) return (Double)obj; if (obj instanceof Long) return (double)(Long)obj; if (obj instanceof Boolean) return ((Boolean)obj) ? 1.0 : 0.0; return RT.doubleCast(obj); } public static double doubleCast(long obj) { return (double)obj; } public static double doubleCast(double obj) { return obj; } public static double doubleCast(float obj) { return (double)obj; } public static double doubleCast(boolean obj) { return obj ? 1.0 : 0.0; } public static float floatCast(float obj) { return obj; } public static float floatCast(Object obj) { return (obj instanceof Float) ? ((Float)obj).floatValue() : (float)doubleCast(obj); } public static int intCast(int obj) { return obj; } public static int intCast(Object obj) { return (obj instanceof Integer) ? ((Integer)obj).intValue() : (int)longCast(obj); } } ================================================ FILE: java/ham_fisted/ChunkedList.java ================================================ package ham_fisted; import static ham_fisted.IntegerOps.*; import java.util.Arrays; import java.util.Iterator; import java.util.List; import java.util.RandomAccess; import java.util.NoSuchElementException; import java.util.Collection; import java.util.ListIterator; import java.util.Objects; import clojure.lang.IPersistentMap; import clojure.lang.IFn; import clojure.lang.RT; import clojure.lang.IDeref; import clojure.lang.ISeq; import clojure.lang.IteratorSeq; import clojure.lang.Util; import clojure.lang.Murmur3; import clojure.lang.Reduced; public final class ChunkedList { Object[][] data; //Global capacity ignoring offset int capacity; //nElems for this sub-chunk - true nElems is nElems + offset int nElems; final IPersistentMap meta; static final int numChunks(int capacity) { return (capacity + 31)/32; } static final int lastChunkSize(int capacity) { if (capacity == 0) return 4; final int leftover = capacity % 32; if (leftover == 0) return 32; return Math.max(4, nextPow2(leftover)); } public ChunkedList(int initSize) { initSize = Math.max(initSize, 16); final int nChunks = numChunks(initSize); final int leftover = lastChunkSize(initSize); data = new Object[nChunks][]; final int nnc = nChunks - 1; for (int idx = 0; idx < nChunks; ++idx) data[idx] = new Object[idx == nnc ? leftover : 32]; nElems = 0; capacity = (32 * (nChunks-1)) + leftover; meta = null; } public ChunkedList() {this(0);} ChunkedList(Object[][] d, int c, int e, IPersistentMap m) { data = d; capacity = c; nElems = e; meta = m; } ChunkedList(ChunkedList other, boolean shallow) { if (shallow) { data = other.data; } else { final Object[][] odata = other.data; final Object[][] mdata = odata.clone(); final int ne = mdata.length; for (int idx = 0; idx < ne; ++idx) mdata[idx] = odata[idx].clone(); data = mdata; } capacity = other.capacity; nElems = other.nElems; meta = other.meta; } static ChunkedList create(boolean owning, IPersistentMap meta, Object... data) { final int dlen = data.length; if (owning && dlen <= 32) { return new ChunkedList(new Object[][] { data }, dlen, dlen, null); } else { final int nElems = data.length; final int nChunks = numChunks(nElems); final Object[][] mdata = new Object[nChunks][]; int idx = 0; while(idx < nElems) { final int clen = Math.min(32, nElems - idx); final Object[] chunk = new Object[clen]; System.arraycopy(data, idx, chunk, 0, clen); mdata[idx/32] = chunk; idx += clen; } return new ChunkedList(mdata, nElems, nElems, meta); } } ChunkedList clone(int startidx, int endidx, int extraAlloc, boolean deep) { final int ne = endidx - startidx; final int nne = ne + extraAlloc; final boolean shallow = deep == false && ((startidx % 32) == 0); final Object[][] mdata = data; if(shallow) { final Object[][] odata = Arrays.copyOfRange(mdata, startidx/32, (endidx + extraAlloc + 31)/32); return new ChunkedList(odata, nne, nne, meta); } final int nChunks = numChunks(nne); final int nnc = nChunks - 1; final Object[][] retval = new Object[nChunks][]; final int sidx = startidx; int dstCapacity = 0; while(startidx < endidx) { final int leftover = endidx - startidx; final Object[] srcc = mdata[startidx/32]; final int eidx = startidx % 32; final int deidx = (startidx - sidx) % 32; final int dcidx = (startidx - sidx) / 32; Object[] dstc = retval[dcidx]; if (dstc == null) { dstc = dcidx == nnc ? new Object[nne%32] : new Object[32]; retval[dcidx] = dstc; dstCapacity += dstc.length; } final int copyLen = Math.min(leftover, Math.min(srcc.length - eidx, dstc.length - deidx)); // System.out.println("srcc: " + String.valueOf(srcc.length) + " eidx: " + String.valueOf(eidx) + // " dstc: " + String.valueOf(dstc.length) + " deidx: " + String.valueOf(deidx) + // " copyLen: " + String.valueOf(copyLen)); System.arraycopy(srcc, eidx, dstc, deidx, copyLen); startidx += copyLen; } return new ChunkedList(retval, dstCapacity, nne, meta); } ChunkedList clone(int startidx, int endidx) { return clone(startidx, endidx, 0, true); } void clear(int offset, int len) { if (offset == 0 && len == nElems) { final Object[][] mdata = data; final int nChunks = data.length; final int ne = nElems; int idx = 0; while(idx < ne) { final Object[] chunk = mdata[idx / 32]; final int eidx = idx % 32; Arrays.fill(chunk, eidx, chunk.length - eidx, null); idx += 32 - eidx; } nElems = 0; } else { shorten(offset, offset+len); } } void clear() { clear(0, nElems); } void enlarge(int cap) { if (cap <= capacity) return; final int nChunks = numChunks(cap); final int nnc = nChunks -1; final int leftover = lastChunkSize(cap); // System.out.println("leftover: " + String.valueOf(leftover) + " Requested Capacity: " + // String.valueOf(cap)); Object[][] mdata = data; if (nChunks != mdata.length) mdata = Arrays.copyOf(mdata, nChunks); for(int idx = 0; idx < nChunks; ++idx) { final Object[] existing = mdata[idx]; if (existing == null || existing.length != 32) { final int targetLen = idx == nnc ? leftover : 32; final int exLen = existing == null ? 0 : existing.length; if (exLen != targetLen) { mdata[idx] = existing == null ? new Object[targetLen] : Arrays.copyOf(existing, targetLen); } } } data = mdata; capacity = (32 * (nChunks-1)) + leftover; } final Object setValueRV(int idx, Object obj) { final Object[] ary = data[idx / 32]; final int eidx = idx % 32; final Object rv = ary[eidx]; ary[eidx] = obj; return rv; } final void setValue(int idx, Object obj) { data[idx / 32][idx % 32] = obj; } final Object getValue(final int idx) { return data[idx / 32][idx % 32]; } public static final void sublistCheck(long sidx, long eidx, long nElems) { if(sidx < 0 || sidx > nElems) throw new IndexOutOfBoundsException("Start index out of range: start-index(" + String.valueOf(sidx) +"), n-elems(" + String.valueOf(nElems) + ")"); if(eidx < 0 || eidx > nElems) throw new IndexOutOfBoundsException("End index out of range: end-index(" + String.valueOf(eidx) +"), n-elems(" + String.valueOf(nElems) + ")"); if(eidx < sidx) throw new IndexOutOfBoundsException("End index underflow: end-index(" + String.valueOf(eidx) +") < start-index(" + String.valueOf(sidx) + ")"); } public static final int indexCheck(int nElems, int idx) { if (idx < 0 || idx >= nElems) throw new IndexOutOfBoundsException("Index " + String.valueOf(idx) + "out of range 0-" + String.valueOf(nElems)); return idx; } public static final int indexCheck(int startidx, int nElems, int idx) { return indexCheck(nElems,idx) + startidx; } public static final long indexCheck(long startidx, long nElems, long idx) { if (idx < 0 || idx >= nElems) throw new IndexOutOfBoundsException("Index " + String.valueOf(idx) + "out of range 0-" + String.valueOf(nElems)); return idx + startidx; } static final int wrapIndexCheck(int startidx, int nElems, int idx) { if (idx < 0) idx = nElems + idx; return indexCheck(startidx, nElems, idx); } static public final void checkIndexRange(int startidx, int nElems, int sidx, int eidx) { final int rne = eidx - sidx; if(rne == 0 ) return; indexCheck(startidx, nElems, sidx); if (rne < 0) throw new RuntimeException("Range end: " + String.valueOf(eidx) + " is less than start: " + String.valueOf(sidx)); if(eidx > nElems) throw new RuntimeException("Range end point: " + String.valueOf(eidx) + " is past end of valid range: " + String.valueOf(nElems)); } static public final void checkIndexRange(long startidx, long nElems, long sidx, long eidx) { final long rne = eidx - sidx; if(rne == 0 ) return; indexCheck(startidx, nElems, sidx); if (rne < 0) throw new RuntimeException("Range end: " + String.valueOf(eidx) + " is less than start: " + String.valueOf(sidx)); if(eidx > nElems) throw new RuntimeException("Range end polong: " + String.valueOf(eidx) + " is past end of valid range: " + String.valueOf(nElems)); } final boolean add(Object obj) { final int ne = nElems; final int cap = capacity; // System.out.println("Capacity: " + String.valueOf(cap) + " ne: " + String.valueOf(ne)); if (ne >= cap) enlarge(cap+1); data[nElems/32][nElems%32] = obj; nElems = ne + 1; return true; } final void widen(final int startidx, final int endidx) { final int wne = endidx - startidx; if (wne == 0) return; final int ne = nElems; final int cap = capacity; enlarge(ne + wne); int copyNe = ne - startidx; final Object[][] mdata = data; //Copy contiguous sections starting from the end so we //do not overwrite elements we later need to move. while(copyNe > 0) { //Get the last valid index to move data into for start/end blocks final int sidx = startidx + copyNe - 1; final int eidx = endidx + copyNe - 1; //Find chunks related to those indexes. Object[] srcc = mdata[sidx / 32]; Object[] endc = mdata[eidx / 32]; //Find the relative end indexes in the blocks final int srceidx = sidx % 32; final int endeidx = eidx % 32; final int copyLen = Math.min(copyNe, Math.min(srceidx+1, endeidx+1)); // System.out.println("Widen - srceidx: " + String.valueOf(srceidx) // + " - endeidx: " + String.valueOf(endeidx) // + " - copyNe: " + String.valueOf(copyNe) // + " - copyLen: " + String.valueOf(copyLen)); System.arraycopy(srcc, srceidx - copyLen + 1, endc, endeidx - copyLen + 1, copyLen); copyNe -= copyLen; } nElems = ne + wne; } final void shorten(int startidx, int endidx) { if(startidx == 0 && endidx == nElems) { clear(); return; } final int ne = nElems; final int wne = endidx - startidx; int copyNe = ne - endidx; final Object[][] mdata = data; while(copyNe > 0) { final Object[] startc = data[startidx/32]; final Object[] endc = data[endidx/32]; final int seidx = startidx % 32; final int eeidx = endidx % 32; int copyLen = Math.min(copyNe, Math.min(startc.length - seidx, endc.length - eeidx)); // System.out.println("Shorten - startidx: " + String.valueOf(startidx) // + " - endidx: " + String.valueOf(endidx) // + " - copyNe: " + String.valueOf(copyNe) // + " - copyLen: " + String.valueOf(copyLen)); System.arraycopy(endc, eeidx, startc, seidx, copyLen); copyNe -= copyLen; startidx += copyLen; endidx += copyLen; } //Zero out remaining blocks to ensure we don't hold onto any object references. clear(startidx, wne); nElems = ne - wne; } //Extremely inefficent operation. Make another list and insert the list all at once. final void add(Object obj, int idx) { final int ne = nElems; if (idx > ne) throw new RuntimeException("Index out of range: " + String.valueOf(idx) + " > " + String.valueOf(ne)); if (idx == ne) { add(obj); return; } widen(idx, idx+1); setValue(idx,obj); } final ChunkedList conj(int startidx, int endidx, Object obj) { ChunkedList retval = clone(startidx, endidx, 1, false); final int nc = endidx - startidx; final Object[][] mdata = retval.data; final int cidx = nc/32; Object[] chunk = mdata[cidx]; final int eidx = nc % 32; if (chunk == null) { chunk = new Object[1]; } else { chunk = Arrays.copyOf(chunk, eidx+1); } mdata[cidx] = chunk; chunk[eidx] = obj; return retval; } final ChunkedList assoc(int startidx, int endidx, int idx, Object obj) { final int nc = endidx - startidx; if (idx == nc) return conj(startidx, endidx, obj); ChunkedList retval = clone(startidx, endidx, 0, false); final int cidx = idx / 32; final int eidx = idx % 32; final Object[][] mdata = retval.data; Object[] edata = mdata[cidx].clone(); edata[eidx] = obj; mdata[cidx] = edata; return retval; } final ChunkedList pop(int startidx, int endidx) { ChunkedList retval = clone(startidx, endidx-1, 0, false); final int nc = endidx - startidx; final Object[][] rdata = retval.data; final int cidx = nc/32; if ( rdata.length > cidx ) { final Object[] c = rdata[cidx]; final int eidx = nc % 32; if (c.length > eidx) { rdata[cidx] = Arrays.copyOf(c, eidx); } } return retval; } final int size() { return nElems; } final void fillRange(int startidx, int endidx, Object v) { final Object[][] mdata = data; for(; startidx < endidx; ++startidx) mdata[startidx/32][startidx%32] = v; } final void fillRangeReduce(final int startidx, Object v) { final Object[][] mdata = data; Reductions.serialReduction(new Reductions.IndexedAccum(new IFnDef.OLOO() { public Object invokePrim(Object acc, long idx, Object v) { final int ss = (int)idx+startidx; ((Object[][])acc)[ss/32][ss%32] = v; return acc; } }), mdata, v); } final void addRange(int startidx, int endidx, Object v) { widen(startidx, endidx); final Object[][] mdata = data; for(; startidx < endidx; ++startidx) mdata[startidx/32][startidx%32] = v; } final Object[] fillArray(int startidx, int endidx, Object[] retval) { final int finalCidx = endidx / 32; final int finalEidx = endidx % 32; int cidx = startidx / 32; int eidx = startidx % 32; final Object[][] mdata = data; int dstOff = 0; while(cidx <= finalCidx) { final int copyLen = cidx == finalCidx ? finalEidx - eidx : 32 - eidx; //In the case where the end idx falls exactly on a boundary we get a copyLen of 0 here. if(copyLen > 0) { System.arraycopy(mdata[cidx], eidx, retval, dstOff, copyLen); dstOff += copyLen; } eidx = 0; ++cidx; } return retval; } final Object[] fillArray(Object[] retval) { return fillArray(0, nElems, retval); } final Object[] toArray(int startidx, int endidx) { return fillArray(startidx, endidx, new Object[endidx - startidx]); } final Object[] toArray() { return toArray(0, nElems); } static class CLIter implements Iterator { final int finalCidx; final int finalEidx; final Object[][] data; int cidx; int eidx; Object[] chunk; public CLIter(int startidx, int endidx, Object[][] _data) { finalCidx = endidx / 32; finalEidx = endidx % 32; cidx = startidx / 32; eidx = (startidx % 32) - 1; data = _data; advance(); } final void advance() { ++eidx; if (eidx == 32) { ++cidx; eidx = 0; chunk = null; } } public final boolean hasNext() { if(cidx < finalCidx) return true; return eidx < finalEidx; } public final Object next() { if ( cidx >= finalCidx && eidx >= finalEidx ) throw new NoSuchElementException(); if (chunk == null) chunk = data[cidx]; final Object retval = chunk[eidx]; advance(); return retval; } } public Iterator iterator(int startidx, int endidx) { return new CLIter(startidx, endidx, data); } public Iterator iterator() { return iterator(0, nElems); } static class RIterator implements Iterator { final int startidx; final Object[][] data; int idx; public RIterator( int sidx, int eidx, Object[][] d) { startidx = sidx; idx = eidx; data = d; advance(); } void advance() { --idx; } public final boolean hasNext() { return idx >= startidx; } public final Object next() { if (idx < startidx) throw new NoSuchElementException(); final Object retval = data[idx/32][idx%32]; advance(); return retval; } } public Iterator riterator(int startidx, int endidx) { return new RIterator(startidx, endidx, data); } static class ListIter implements ListIterator { final int startidx; final int endidx; final Object[][] data; int idx; int prevIdx; ListIter(int sidx, int eidx, Object[][] d) { startidx = sidx; endidx = eidx; data = d; idx = sidx; prevIdx = idx; } public void add(E obj) { throw new RuntimeException("Unimplemented."); } public void remove() { throw new RuntimeException("Unimplemented."); } public final boolean hasNext() { return idx < endidx; } public final boolean hasPrevious() { return idx > startidx; } @SuppressWarnings("unchecked") public final E next() { if (idx == endidx) throw new NoSuchElementException(); final Object retval = data[idx/32][idx%32]; prevIdx = idx; ++idx; return (E)retval; } public final int nextIndex() { return idx - startidx; } @SuppressWarnings("unchecked") public final E previous() { --idx; if (idx < startidx) throw new NoSuchElementException(); prevIdx = idx; return (E)data[idx/32][idx%32]; } public final int previousIndex() { return idx - startidx - 1; } @SuppressWarnings("unchecked") public final void set(E v) { data[prevIdx/32][prevIdx%32] = v; } } ListIterator listIterator(int startidx, int endidx, E marker) { return new ListIter(startidx, endidx, data); } static class ChunkedListSection { public final Object[][] data; public final int startidx; public final int endidx; public int size() { return endidx - startidx; } public ChunkedListSection(Object[][] cl, int sidx, int eidx) { data = cl; startidx = sidx; endidx = eidx; } } interface ChunkedListOwner { ChunkedListSection getChunkedList(); } Object reduce(final int startidx, final int endidx, final IFn f, final Object start) { final Object[][] mdata = data; Object ret = start; int sidx = startidx; while(sidx < endidx && !RT.isReduced(ret)) { final Object[] cdata = mdata[sidx/32]; int cstart = sidx % 32; final int clen = Math.min(endidx - sidx, 32 - cstart); final int cstop = clen + cstart; for(int idx = cstart; idx < cstop && !RT.isReduced(ret); ++idx) ret = f.invoke(ret, cdata[idx]); sidx += clen; } return Reductions.unreduce(ret); } Object reduce(final int startidx, final int endidx, IFn f) { if(startidx == endidx) return f.invoke(); return reduce(startidx+1, endidx, f, getValue(startidx)); } Object kvreduce(int startidx, int endidx, IFn f, Object init) { final Object[][] mdata = data; for (int i=startidx; i= eidx) return new Reduced(false); final Object vv = d[iidx/32][iidx%32]; if(CljHash.equiv(vv, v) == false) return new Reduced(false); return acc; } }), true, o); } else { return false; } } final IPersistentMap meta() { return meta; } final ChunkedList withMeta(IPersistentMap m) { return new ChunkedList(data, capacity, nElems, m); } final int indexOf(int startidx, int endidx, Object obj) { final int ne = endidx - startidx; final Object[][] mdata = data; for(int idx = 0; idx < ne; ++idx) if (Objects.equals(obj, mdata[idx/32][idx%32])) return idx; return -1; } final int lastIndexOf(int startidx, int endidx, Object obj) { final int ne = endidx - startidx; final int nne = ne - 1; final Object[][] mdata = data; for(int idx = 0; idx < ne; ++idx) { final int ridx = nne - idx + startidx; if (Objects.equals(obj, mdata[ridx/32][ridx%32])) return ridx; } return -1; } final boolean contains(int startidx, int endidx, Object obj) { return indexOf(startidx, endidx, obj) != -1; } final boolean containsAll(int startidx, int endidx, Collection c) { final int ne = endidx - startidx; Iterator minC; Iterator maxC; if (ne < c.size()) { minC = this.iterator(startidx, endidx); maxC = c.iterator(); } else { maxC = this.iterator(startidx, endidx); minC = c.iterator(); } //This set can contain null. // HashSet hc = new HashSet(); // while(minC.hasNext()) hc.add(minC.next()); // while(maxC.hasNext()) { // if (!hc.contains(maxC.next())) // return false; // } // return true; throw new UnsupportedOperationException(); } } ================================================ FILE: java/ham_fisted/CljHash.java ================================================ package ham_fisted; import java.util.Map; import java.util.Set; import java.util.List; import java.util.Iterator; import java.util.RandomAccess; import java.util.Collection; import clojure.lang.Murmur3; import clojure.lang.APersistentMap; import clojure.lang.Util; import clojure.lang.RT; import clojure.lang.Numbers; import clojure.lang.IPersistentCollection; import clojure.lang.Reduced; public class CljHash { public static int mapHashcode(Map data) { return Murmur3.hashUnordered(data.entrySet()); } public static int setHashcode(Set data) { return Murmur3.hashUnordered(data); } public static boolean equiv(Object k1, Object k2) { if(k1 == k2) return true; //Somewhat faster version of equiv *if* both are longs or both are doubles. //which happens to be a very common case in Clojure. return k1 != null ? nonNullEquiv(k1,k2) : false; } public static boolean nonNullEquiv(Object k1, Object k2) { //Small carveout to accelerate long,long and double,double equivalence. if( k1 instanceof Number) { if( k1 instanceof Long && k2 instanceof Long) return (long)k1 == (long)k2; if ( k1 instanceof Double && k2 instanceof Double) return (double)k1 == (double)k2; if(k2 instanceof Number) return Numbers.equal((Number)k1, (Number)k2); return false; } if(k1 instanceof IPersistentCollection || k2 instanceof IPersistentCollection) return Util.pcequiv(k1,k2); return k1.equals(k2); } public static boolean mapEquiv(Map lhs, Object rhs) { if(rhs instanceof Map) { final Map rm = (Map)rhs; if(lhs.size() == rm.size()) { return (Boolean)Reductions.serialReduction(new IFnDef() { public Object invoke(Object acc, Object v) { Map.Entry me = (Map.Entry)v; if(!equiv(lhs.get(me.getKey()), me.getValue())) return new Reduced(false); return acc; } }, true, rhs); } } return false; } public static boolean setEquiv(Set data, Object rhs) { if (rhs instanceof Set) { Set rhsMap = (Set)rhs; if (data.size() != rhsMap.size()) return false; for(Object obj: rhsMap) { if (data.contains(obj) == false) return false; } return true; } return false; } public static class ListHasheqConsumer implements java.util.function.Consumer { public int hash; public int n; public ListHasheqConsumer() { hash = 1; n = 0; } public void accept(Object obj) { hash = 31 * hash + Util.hasheq(obj); ++n; } public int hash() { return Murmur3.mixCollHash(hash, n); } } public static int listHasheq(List l) { ListHasheqConsumer c = new ListHasheqConsumer(); Reductions.serialReduction(c, l); return c.hash(); } public static boolean listEquiv(List l, Object rhs) { if (l == rhs) return true; if (rhs == null) return false; if (rhs instanceof RandomAccess) { List r = (List)rhs; final int sz = l.size(); if(sz != r.size()) return false; for(int idx = 0; idx < sz; ++idx) { if(!equiv(l.get(idx), r.get(idx))) return false; } return true; } else if ( rhs instanceof Iterable) { Collection r = rhs instanceof Collection ? (Collection)rhs : (Collection)RT.seq(rhs); Iterator iter = r.iterator(); final int sz = l.size(); int idx; for(idx = 0; idx < sz && iter.hasNext(); ++idx) { if(!equiv(l.get(idx), iter.next())) return false; } return idx != sz || iter.hasNext() ? false : true; } return false; } } ================================================ FILE: java/ham_fisted/ConstList.java ================================================ package ham_fisted; import java.util.Comparator; import java.util.Random; import java.util.List; import java.util.Arrays; import clojure.lang.IPersistentMap; import clojure.lang.RT; import clojure.lang.IFn; import it.unimi.dsi.fastutil.ints.IntComparator; public class ConstList implements IMutList, TypedList { public final long nElems; public final Object value; public final IPersistentMap meta; public ConstList(long _nElems, Object _v, IPersistentMap m) { nElems = _nElems; value = _v; meta = m; } public IMutList cloneList() { return this; } public Class containedType() { return value != null ? value.getClass() : Object.class; } public static ConstList create(long nElems, Object value, IPersistentMap m) { if (value instanceof Long || value instanceof Integer || value instanceof Short || value instanceof Byte || value instanceof Character) return new LongConstList(nElems, Casts.longCast(value), m); if (value instanceof Double || value instanceof Float) return new DoubleConstList(nElems, Casts.doubleCast(value), m); return new ConstList(nElems, value, m); } public int size() { return RT.intCast(nElems); } public Object get(int idx) { return value; } public ConstList subList(long sidx, long eidx) { ChunkedList.sublistCheck(sidx, eidx, nElems); return create(eidx-sidx, value, meta); } public ConstList subList(int sidx, int eidx) { return subList((long)sidx, (long)eidx); } public void sort(Comparable c) { } public ConstList immutSort(Comparable c) { return this; } public ConstList ImmutSort() { return this; } public ConstList reverse() { return this; } public int[] sortIndirect() { return ArrayLists.iarange(0, size(), 1); } public Object[] toArray() { Object[] retval = new Object[size()]; Arrays.fill(retval, value); return retval; } public int[] toIntArray() { int v = RT.intCast(Casts.longCast(value)); int[] retval = new int[size()]; Arrays.fill(retval, v); return retval; } public long[] toLongArray() { long v = Casts.longCast(value); long[] retval = new long[size()]; Arrays.fill(retval, v); return retval; } public double[] toDoubleArray() { double v = Casts.doubleCast(value); double[] retval = new double[size()]; Arrays.fill(retval, v); return retval; } public ConstList reindex(int[] indexes) { return ConstList.create(indexes.length, value, meta); } public List immutShuffle(Random r) { return this; } public IntComparator indexComparator(Comparator c) { return new IntComparator() { public int compare(int l, int r) {return 0;}}; } public IPersistentMap meta() { return meta; } public ConstList withMeta(IPersistentMap m) { return ConstList.create(nElems, value, m); } public static class LongConstList extends ConstList implements LongMutList { public final long lval; public LongConstList(long ne, long v, IPersistentMap m ) { super(ne, v, m); lval = v; } public Class containedType() { return Long.TYPE; } public long getLong(int idx) { return lval; } public Object reduce(IFn rfn, Object init) { return LongMutList.super.reduce(rfn, init); } } public static class DoubleConstList extends ConstList implements DoubleMutList { public final double lval; public DoubleConstList(long ne, double v, IPersistentMap m ) { super(ne, v, m); lval = v; } public Class containedType() { return Double.TYPE; } public double getDouble(int idx) { return lval; } public Object reduce(IFn rfn, Object init) { return DoubleMutList.super.reduce(rfn, init); } } } ================================================ FILE: java/ham_fisted/ConsumerAccumulators.java ================================================ package ham_fisted; public class ConsumerAccumulators { public static class DoubleConsumerAccumulator implements IFnDef.ODO { public static final DoubleConsumerAccumulator INST = new DoubleConsumerAccumulator(); public DoubleConsumerAccumulator(){} public Object invokePrim(Object acc, double val) { ((java.util.function.DoubleConsumer)acc).accept(val); return acc; } } public static class LongConsumerAccumulator implements IFnDef.OLO { public static final LongConsumerAccumulator INST = new LongConsumerAccumulator(); public LongConsumerAccumulator(){} public Object invokePrim(Object acc, long val) { ((java.util.function.LongConsumer)acc).accept(val); return acc; } } public static class ConsumerAccumulator implements IFnDef.OOO { public static final ConsumerAccumulator INST = new ConsumerAccumulator(); public ConsumerAccumulator(){} @SuppressWarnings("unchecked") public Object invoke(Object acc, Object val) { ((java.util.function.Consumer)acc).accept(val); return acc; } } } ================================================ FILE: java/ham_fisted/Consumers.java ================================================ package ham_fisted; import java.util.function.Consumer; import java.util.function.DoubleConsumer; import java.util.function.LongConsumer; import java.util.function.Function; import java.util.function.DoubleUnaryOperator; import java.util.function.LongUnaryOperator; import java.util.function.Predicate; import java.util.function.DoublePredicate; import java.util.function.LongPredicate; import clojure.lang.IDeref; public class Consumers { public interface IDerefDoubleConsumer extends IDeref, DoubleConsumer, Consumer { default void accept(Object v) { accept(Casts.doubleCast(v)); } default void accept(double v) { acceptDouble(v); } void acceptDouble(double v); } public interface IDerefLongConsumer extends IDeref, LongConsumer, Consumer { default void accept(Object v) { accept(Casts.longCast(v)); } default void accept(long v) { acceptLong(v); } void acceptLong(long v); } public interface IDerefConsumer extends IDeref, Consumer {} public interface IDerefIndexedDoubleConsumer extends IDeref, IndexedDoubleConsumer {} public interface IDerefIndexedLongConsumer extends IDeref, IndexedLongConsumer {} public interface IDerefIndexedConsumer extends IDeref, IndexedConsumer {} public static IndexedDoubleConsumer asIndexedDoubleConsumer(Object obj) { if (obj instanceof IndexedDoubleConsumer) return (IndexedDoubleConsumer)obj; return null; } public static IndexedLongConsumer asIndexedLongConsumer(Object obj) { if (obj instanceof IndexedLongConsumer) return (IndexedLongConsumer)obj; return null; } public static IndexedConsumer asIndexedConsumer(Object obj) { if (obj instanceof IndexedConsumer) return (IndexedConsumer)obj; return null; } public static final class IndexedDoubleConsumerConverter implements IDerefDoubleConsumer { long idx; public final IndexedDoubleConsumer c; public IndexedDoubleConsumerConverter(long initIdx, IndexedDoubleConsumer cc) { idx = initIdx; c = cc; } public void acceptDouble(double v) { c.accept(idx, v); ++idx; } public Object deref() { return ((IDeref)c).deref(); } } public static DoubleConsumer toDoubleConsumer(long initIdx, IndexedDoubleConsumer c) { return new IndexedDoubleConsumerConverter(initIdx, c); } public static final class IndexedLongConsumerConverter implements IDerefLongConsumer { long idx; public final IndexedLongConsumer c; public IndexedLongConsumerConverter(long initIdx, IndexedLongConsumer cc) { idx = initIdx; c = cc; } public void acceptLong(long v) { c.accept(idx, v); ++idx; } public Object deref() { return ((IDeref)c).deref(); } } public static LongConsumer toLongConsumer(long initIdx, IndexedLongConsumer c) { return new IndexedLongConsumerConverter(initIdx, c); } public static final class IndexedConsumerConverter implements IDerefConsumer { long idx; public final IndexedConsumer c; public IndexedConsumerConverter(long initIdx, IndexedConsumer cc) { idx = initIdx; c = cc; } public void accept(Object v) { c.accept(idx, v); ++idx; } public Object deref() { return ((IDeref)c).deref(); } } public static Consumer toConsumer(long initIdx, IndexedConsumer c) { return new IndexedConsumerConverter(initIdx, c); } public static DoubleConsumer map(final DoubleUnaryOperator fn, final DoubleConsumer c) { return new IDerefDoubleConsumer() { public void acceptDouble(double v) { c.accept(fn.applyAsDouble(v)); } public Object deref() { return ((IDeref)c).deref(); } }; } public static LongConsumer map(final LongUnaryOperator fn, final LongConsumer c) { return new IDerefLongConsumer() { public void acceptLong(long v) { c.accept(fn.applyAsLong(v)); } public Object deref() { return ((IDeref)c).deref(); } }; } @SuppressWarnings("unchecked") public static Consumer map(final Function fn, final Consumer c) { return new IDerefConsumer() { public void accept(Object v) { c.accept(fn.apply(v)); } public Object deref() { return ((IDeref)c).deref(); } }; } public static IndexedDoubleConsumer map(final DoubleUnaryOperator fn, final IndexedDoubleConsumer c) { return new IDerefIndexedDoubleConsumer() { public void accept(long idx, double v) { c.accept(idx, fn.applyAsDouble(v)); } public Object deref() { return ((IDeref)c).deref(); } }; } public static IndexedLongConsumer map(final LongUnaryOperator fn, final IndexedLongConsumer c) { return new IDerefIndexedLongConsumer() { public void accept(long idx, long v) { c.accept(idx, fn.applyAsLong(v)); } public Object deref() { return ((IDeref)c).deref(); } }; } @SuppressWarnings("unchecked") public static IndexedConsumer map(final Function fn, final IndexedConsumer c) { return new IDerefIndexedConsumer() { public void accept(long idx, Object v) { c.accept(idx, fn.apply(v)); } public Object deref() { return ((IDeref)c).deref(); } }; } public static DoubleConsumer filter(final DoublePredicate pred, final DoubleConsumer c) { return new IDerefDoubleConsumer() { public void acceptDouble(double v) { if(pred.test(v)) c.accept(v); } public Object deref() { return ((IDeref)c).deref(); } }; } public static LongConsumer filter(final LongPredicate pred, final LongConsumer c) { return new IDerefLongConsumer() { public void acceptLong(long v) { if(pred.test(v)) c.accept(v); } public Object deref() { return ((IDeref)c).deref(); } }; } @SuppressWarnings("unchecked") public static Consumer filter(final Predicate pred, final Consumer c) { return new IDerefConsumer() { public void accept(Object v) { if(pred.test(v)) c.accept(v); } public Object deref() { return ((IDeref)c).deref(); } }; } public static IndexedDoubleConsumer filter(final DoublePredicate pred, final IndexedDoubleConsumer c) { return new IDerefIndexedDoubleConsumer() { public void accept(long idx, double v) { if(pred.test(v)) c.accept(idx, v); } public Object deref() { return ((IDeref)c).deref(); } }; } public static IndexedLongConsumer filter(final LongPredicate pred, final IndexedLongConsumer c) { return new IDerefIndexedLongConsumer() { public void accept(long idx, long v) { if(pred.test(v)) c.accept(idx, v); } public Object deref() { return ((IDeref)c).deref(); } }; } @SuppressWarnings("unchecked") public static IndexedConsumer filter(final Predicate pred, final IndexedConsumer c) { return new IDerefIndexedConsumer() { public void accept(long idx, Object v) { if(pred.test(v)) c.accept(idx, v); } public Object deref() { return ((IDeref)c).deref(); } }; } public static class IncConsumer implements Consumer, Reducible, IDeref { public static java.util.function.Function cfn = new java.util.function.Function() { public IncConsumer apply(Object obj) { return new IncConsumer(); } }; long nElems; public IncConsumer(long v) { nElems = v;} public IncConsumer() { this(0); } public void accept(Object o) { ++nElems; } public void inc() { ++nElems; } public void setValue(int v) { nElems = v;} public IncConsumer reduce(Reducible o) { nElems += ((IncConsumer)o).nElems; return this; } public long value() { return nElems; } public Object deref() { return nElems; } } } ================================================ FILE: java/ham_fisted/CtxIter.java ================================================ package ham_fisted; import java.util.Iterator; import java.util.function.Supplier; public class CtxIter implements Iterator { public static interface Ctx { public Ctx update(); public boolean valid(); public Object val(); } public final Supplier init; Ctx ctx; public static final int stepInit = 0; public static final int stepUpdate = 1; public static final int stepVal = 2; int step; public CtxIter(Supplier init) { this.init = init; step = stepInit; ctx = null; } void advance() { if(step == stepInit) { ctx = init.get(); } else if (step == stepUpdate) { ctx = ctx.update(); } step = stepVal; } public int step() { return step; } public Ctx ctx() { return ctx; } public boolean hasNext() { advance(); return ctx.valid(); } public Object next() { advance(); step = stepUpdate; return ctx.val(); } } ================================================ FILE: java/ham_fisted/DoubleMutList.java ================================================ package ham_fisted; import java.util.Random; import java.util.List; import java.util.Comparator; import java.util.Spliterator; import it.unimi.dsi.fastutil.doubles.DoubleArrays; import it.unimi.dsi.fastutil.doubles.DoubleComparator; import it.unimi.dsi.fastutil.ints.IntComparator; import java.util.function.DoubleConsumer; import clojure.lang.IFn; import clojure.lang.RT; @SuppressWarnings("unchecked") public interface DoubleMutList extends IMutList { default boolean add(Object obj) { addDouble(Casts.doubleCast(obj)); return true; } default void addLong(long obj) { addDouble(Casts.doubleCast(obj)); } default void addDouble(double v) { throw new RuntimeException("Object " + String.valueOf(getClass()) + " failed to define addLong method"); } default void add(int idx, int count, Object v) { double d = Casts.doubleCast(v); int end = idx + count; for(; idx < end; ++idx) addDouble( d ); } @SuppressWarnings("unchecked") default Object set(int idx, Object obj) { double v = getDouble(idx); setDouble(idx, Casts.doubleCast(obj)); return v; } default void setBoolean(int idx, boolean obj) { setDouble(idx, obj ? 1.0 : 0.0); } default void setLong(int idx, long obj) { setDouble(idx, (double)obj); } default Object get(int idx) { return getDouble(idx); } default long getLong(int idx) { return (long)getDouble(idx); } static class DoubleSubList extends IMutList.MutSubList implements DoubleMutList { @SuppressWarnings("unchecked") public DoubleSubList(IMutList l, int ss, int ee) { super(l, ss, ee); } public Object reduce(IFn rfn, Object init) { return DoubleMutList.super.reduce(rfn, init); } } default IMutList subList(int sidx, int eidx) { ChunkedList.sublistCheck(sidx, eidx, size()); return new DoubleSubList(this, sidx, eidx); } default boolean addAllReducible(Object obj) { final int sz = size(); Reductions.serialReduction(new IFnDef.ODO() { public Object invokePrim(Object lhs, double rhs) { ((IMutList)lhs).addDouble(rhs); return lhs; } }, this, obj); return sz != size(); } default void fillRange(long startidx, final long endidx, Object v) { ChunkedList.checkIndexRange(0, size(), startidx, endidx); double l = Casts.doubleCast(v); for(; startidx < endidx; ++startidx) { setDouble((int)startidx, l); } } default void fillRange(final long ss, List l) { if (l.isEmpty()) return; final int startidx = (int)ss; final int sz = size(); final int endidx = startidx + l.size(); ArrayLists.checkIndexRange(size(), startidx, endidx); Reductions.serialReduction(new Reductions.IndexedDoubleAccum(new IFnDef.OLDO() { public Object invokePrim(Object acc, long idx, double v) { ((IMutList)acc).setDouble((int)idx+startidx, v); return acc; } }), this, l); } default void addRange(int startidx, int endidx, Object v) { Double l = Double.valueOf(Casts.doubleCast(v)); for(; startidx < endidx; ++startidx) { add(startidx, l); } } default IntComparator indexComparator() { return new IntComparator() { public int compare(int lidx, int ridx) { return Double.compare(getDouble(lidx), getDouble(ridx)); } }; } default void sort(Comparator c) { DoubleComparator lc = ArrayLists.DoubleArraySubList.asDoubleComparator(c); if (c == null || lc != null) { final double[] data = toDoubleArray(); if(c == null) DoubleArrays.parallelQuickSort(data); else DoubleArrays.parallelQuickSort(data, lc); fillRange(0, ArrayLists.toList(data)); } else { IMutList.super.sort(c); } } default void shuffle(Random r) { fillRange(0, immutShuffle(r)); } default List immutShuffle(Random r) { final double[] bdata = toDoubleArray(); DoubleArrays.shuffle(bdata, r); return ArrayLists.toList(bdata); } default Object reduce(final IFn rfn, Object init) { return doubleReduction(Transformables.toDoubleReductionFn(rfn), init); } default Object doubleReduction(IFn.ODO rfn, Object init) { final int sz = size(); for (int idx = 0; idx < sz && !RT.isReduced(init); ++idx) init = rfn.invokePrim(init, getDouble(idx)); return Reductions.unreduce(init); } @SuppressWarnings("unchecked") default Spliterator spliterator() { return new RandomAccessSpliterator.DoubleSpliterator(this, 0, size()); } } ================================================ FILE: java/ham_fisted/FJTask.java ================================================ package ham_fisted; import clojure.lang.IFn; import clojure.lang.IDeref; import clojure.lang.Delay; import java.util.concurrent.RecursiveTask; public class FJTask extends RecursiveTask { public final IDeref c; public FJTask(Object c) { if(c instanceof IDeref) { this.c = (IDeref)c; } else { this.c = new Delay( (IFn) c); } } public Object compute() { return c.deref(); } } ================================================ FILE: java/ham_fisted/FMapEntry.java ================================================ package ham_fisted; import java.util.Map; import clojure.lang.IPersistentMap; import clojure.lang.IMapEntry; public class FMapEntry implements IMutList, Map.Entry { public final K k; public final V v; int _hash; IPersistentMap meta; public FMapEntry(K _k, V _v) { k = _k; v = _v; _hash = 0; meta = null; } public FMapEntry(FMapEntry e, IPersistentMap m) { k = e.k; v = e.v; _hash = e._hash; meta = m; } public static FMapEntry create(K k, V v) { return new FMapEntry(k,v); } public boolean equals(Object o) { return equiv(o); } public int hashCode() { return hasheq(); } public int hasheq() { if (_hash == 0) _hash = IMutList.super.hasheq(); return _hash; } public V setValue( Object v) { throw new RuntimeException("Cannot set value."); } public K getKey() { return k; } public Object key() { return k; } public V getValue() { return v; } public Object val() { return v; } public int size() { return 2; } public Object get(int idx) { if(idx == 0) return k; if(idx == 1) return v; throw new RuntimeException("Index out of range: " + String.valueOf(idx)); } public FMapEntry withMeta(IPersistentMap m) { return new FMapEntry(this, m); } public IPersistentMap meta() { return meta; } } ================================================ FILE: java/ham_fisted/ForkJoinPatterns.java ================================================ package ham_fisted; import clojure.lang.IFn; import clojure.lang.Delay; import clojure.lang.IDeref; import clojure.java.api.Clojure; import java.util.Spliterator; public class ForkJoinPatterns { private ForkJoinPatterns(){} static final Delay pgroupsPtr = new Delay(new IFnDef() { public Object invoke() { return ((IDeref)Clojure.var("ham-fisted.impl", "pgroups")).deref(); } }); public static Iterable parallelIndexGroups(long nElems, IFn bodyFn, ParallelOptions options) { return (Iterable)((IFn)pgroupsPtr.deref()).invoke(nElems, bodyFn, options); } static final Delay pmapPtr = new Delay(new IFnDef() { public Object invoke() { return ((IDeref)Clojure.var("ham-fisted.impl", "pmap")).deref(); } }); public static Iterable pmap(ParallelOptions options, IFn bodyFn, Object sequences) { return (Iterable)((IFn)pmapPtr.deref()).invoke(options, bodyFn, sequences); } static final Delay spliteratorPtr = new Delay(new IFnDef() { public Object invoke() { return ((IDeref)Clojure.var("ham-fisted.impl", "parallel-spliterator-reduce")).deref(); } }); public static Object parallelSpliteratorReduce(IFn initValFn, IFn rfn, IFn mergeFn, Spliterator s, ParallelOptions options) { return ((IFn)spliteratorPtr.deref()).invoke(initValFn, rfn, mergeFn, s, options); } } ================================================ FILE: java/ham_fisted/HashBase.java ================================================ package ham_fisted; import java.util.Arrays; import java.util.Iterator; import java.util.Spliterator; import java.util.function.Function; import java.util.function.Consumer; import java.util.Map; import clojure.lang.IPersistentMap; import clojure.lang.IHashEq; import clojure.lang.IMeta; import clojure.lang.IFn; import clojure.lang.IDeref; import clojure.lang.RT; public class HashBase implements IMeta { int capacity; int mask; int length; int threshold; float loadFactor; HashNode[] data; IPersistentMap meta; public HashBase(float loadFactor, int initialCapacity, int length, HashNode[] data, IPersistentMap meta) { this.loadFactor = loadFactor; this.capacity = IntegerOps.nextPow2(Math.max(4, initialCapacity)); this.mask = this.capacity - 1; this.length = length; this.data = data == null ? new HashNode[this.capacity] : data; this.threshold = (int)(capacity * loadFactor); this.meta = meta; } public HashBase(HashBase other, IPersistentMap m) { this.loadFactor = other.loadFactor; this.capacity = other.capacity; this.mask = other.mask; this.length = other.length; this.data = other.data; this.threshold = other.threshold; this.meta = m; } public int capacity() { return capacity; } public int size() { return length; } public int count() { return length; } //protected so clients can override as desired. protected int hash(Object k) { return k == null ? 0 : k instanceof IHashEq ? ((IHashEq)k).hasheq() : IntegerOps.mixhash(k.hashCode()); } protected boolean equals(Object lhs, Object rhs) { return lhs == rhs ? true : lhs == null || rhs == null ? false : CljHash.nonNullEquiv(lhs,rhs); } protected void inc(HashNode lf) { ++this.length; } protected void dec(HashNode lf) { --this.length; } protected void modify(HashNode lf) {} protected HashNode newNode(Object key, int hc, Object val) { return new HashNode(this,key,hc,val,null); } Object checkResize(Object rv) { if(this.length >= this.threshold) { final int newCap = this.capacity * 2; final HashNode[] newD = new HashNode[newCap]; final HashNode[] oldD = this.data; final int oldCap = oldD.length; final int mask = newCap - 1; for(int idx = 0; idx < oldCap; ++idx) { HashNode lf; if((lf = oldD[idx]) != null) { oldD[idx] = null; if(lf.nextNode == null) { newD[lf.hashcode & mask] = lf; } else { //https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/util/HashMap.java#L722 //Because we only allow capacities that are powers of two, we have //exactly 2 locations in the new data array where these can go. We want //to avoid writing to any locations more than once and instead make the //at most two new linked lists, one for the new high position and one //for the new low position. HashNode loHead = null, loTail = null, hiHead = null, hiTail = null; while(lf != null) { HashNode e = lf.setOwner(this); lf = lf.nextNode; //Check high bit if((e.hashcode & oldCap) == 0) { if(loTail == null) loHead = e; else loTail.nextNode = e; loTail = e; } else { if(hiTail == null) hiHead = e; else hiTail.nextNode = e; hiTail = e; } } if(loHead != null) { loTail.nextNode = null; newD[idx] = loHead; } if(hiHead != null) { hiTail.nextNode = null; newD[idx+oldCap] = hiHead; } } } } this.capacity = newCap; this.threshold = (int)(newCap * this.loadFactor); this.mask = mask; this.data = newD; } return rv; } public void clear() { for(int idx = 0; idx < data.length; ++idx) { for(HashNode lf = data[idx]; lf != null; lf = lf.nextNode) { dec(lf); } } length = 0; Arrays.fill(data, null); } public IPersistentMap meta() { return meta; } static class HTIter implements Iterator { final HashNode[] d; final Function fn; HashNode l; int idx; final int dlen; HTIter(HashNode[] data, Function fn) { this.d = data; this.fn = fn; this.l = null; this.idx = 0; this.dlen = d.length; advance(); } void advance() { if(l != null) l = l.nextNode; if(l == null) { for(; idx < this.dlen && l == null; ++idx) l = this.d[idx]; } } public boolean hasNext() { return l != null; } public Object next() { HashNode rv = l; advance(); return fn.apply(rv); } } static class HTSpliterator implements Spliterator, ITypedReduce { final HashNode[] d; final Function fn; int sidx; int eidx; int estimateSize; HashNode l; public HTSpliterator(HashNode[] d, int len, Function fn) { this.d = d; this.fn = fn; this.sidx = 0; this.eidx = d.length; this.estimateSize = len; this.l = null; } public HTSpliterator(HashNode[] d, int sidx, int eidx, int es, Function fn) { this.d = d; this.fn = fn; this.sidx = sidx; this.eidx = eidx; this.estimateSize = es; this.l = null; } public HTSpliterator trySplit() { final int nIdxs = this.eidx - this.sidx; if(nIdxs > 4) { final int idxLen = nIdxs/2; final int oldIdx = this.eidx; this.eidx = this.sidx + idxLen; this.estimateSize = this.estimateSize / 2; return new HTSpliterator(d, this.eidx, oldIdx, this.estimateSize, this.fn); } return null; } public int characteristics() { return Spliterator.DISTINCT | Spliterator.IMMUTABLE | Spliterator.SIZED; } public long estimateSize() { return estimateSize; } public long getExactSizeIfKnown() { return estimateSize(); } @SuppressWarnings("unchecked") public boolean tryAdvance(Consumer c) { if(this.l != null) { c.accept(this.fn.apply(this.l)); this.l = this.l.nextNode; return true; } for(; sidx < eidx; ++sidx) { final HashNode ll = this.d[sidx]; if(ll != null) { c.accept(this.fn.apply(ll)); this.l = ll.nextNode; return true; } } return false; } public Object reduce(IFn rfn, Object acc) { final HashNode[] dd = this.d; final int ee = this.eidx; final Function ffn = this.fn; for(int idx = sidx; idx < ee; ++idx) { for(HashNode e = dd[idx]; e != null; e = e.nextNode) { acc = rfn.invoke(acc, ffn.apply(e)); if(RT.isReduced(acc)) return ((IDeref)acc).deref(); } } return acc; } } final boolean containsNodeKey(Object key) { for(HashNode e = this.data[hash(key) & this.mask]; e != null; e = e.nextNode) { Object k; if((k = e.k) == key || equals(k, key)) return true; } return false; } } ================================================ FILE: java/ham_fisted/HashMap.java ================================================ package ham_fisted; import java.util.Map; import java.util.Arrays; import java.util.Iterator; import java.util.function.Function; import java.util.function.BiFunction; import java.util.function.Consumer; import java.util.function.BiConsumer; import java.util.Spliterator; import java.util.Objects; import java.util.Set; import java.util.AbstractSet; import java.util.AbstractCollection; import java.util.Collection; import clojure.lang.IPersistentMap; import clojure.lang.IFn; import clojure.lang.RT; import clojure.lang.IHashEq; import clojure.lang.IDeref; import clojure.lang.RT; import clojure.lang.IDeref; import clojure.lang.IMeta; import clojure.lang.IMapEntry; import clojure.lang.MapEntry; import clojure.lang.IReduceInit; import clojure.lang.IKVReduce; public class HashMap extends HashBase implements IMap, MapSetOps, UpdateValues { Set keySet; public HashMap(float loadFactor, int initialCapacity, int length, HashNode[] data, IPersistentMap meta) { super(loadFactor, initialCapacity, length, data, meta); } public HashMap() { this(0.75f, 0, 0, null, null); } public HashMap(IPersistentMap m) { this(0.75f, 0, 0, null, m); } public HashMap(HashMap other, IPersistentMap m) { super(other, m); } public HashMap shallowClone() { return new HashMap(loadFactor, capacity, length, data.clone(), meta); } public HashMap clone() { final int l = data.length; HashNode[] newData = new HashNode[l]; HashMap retval = new HashMap(loadFactor, capacity, length, newData, meta); for(int idx = 0; idx < l; ++idx) { HashNode orig = data[idx]; if(orig != null) newData[idx] = orig.clone(retval); } return retval; } public int hashCode() { return hasheq(); } public int hasheq() { return CljHash.mapHashcode(this); } public boolean equals(Object o) { return equiv(o); } public boolean equiv(Object o) { return CljHash.mapEquiv(this, o); } public int size() { return this.length; } public boolean isEmpty() { return this.length == 0; } public String toString() { final StringBuilder b = (StringBuilder) reduce(new IFnDef() { public Object invoke(Object acc, Object v) { final StringBuilder b = (StringBuilder)acc; final Map.Entry lf = (Map.Entry)v; if(b.length() > 2) b.append(","); return b.append(lf.getKey()) .append(" ") .append(lf.getValue()); } }, new StringBuilder().append("{")); return b.append("}").toString(); } public Object put(Object key, Object val) { final int hc = hash(key); final int idx = hc & mask; final HashNode init = data[idx]; if(init != null) { HashNode e = init; do { if(e.k == key || equals(e.k, key)) { Object rv = e.v; e.v = val; modify(e); return rv; } e = e.nextNode; } while(e != null); } HashNode lf = newNode(key,hc,val); lf.nextNode = init; data[idx] = lf; return checkResize(null); } public void putAll(Map other) { HashNode[] d = data; int mask = this.mask; for(Object o: other.entrySet()) { Map.Entry ee = (Map.Entry)o; Object k = ee.getKey(); int hashcode = hash(k); int idx = hashcode & mask; HashNode e; for(e = d[idx]; e != null && !(k == e.k || equals(k,e.k)); e = e.nextNode); if(e != null) { e.v = ee.getValue(); } else { HashNode n = newNode(k, hashcode, ee.getValue()); n.nextNode = d[idx]; d[idx] = n; checkResize(null); d = data; mask = this.mask; } } } public Object getOrDefault(Object key, Object dv) { for(HashNode e = this.data[hash(key) & this.mask]; e != null; e = e.nextNode) { Object k; if((k = e.k) == key || equals(k, key)) return e.v; } return dv; } public Object get(Object key) { for(HashNode e = this.data[hash(key) & this.mask]; e != null; e = e.nextNode) { Object k; if((k = e.k) == key || equals(k, key)) return e.v; } return null; } public IMapEntry entryAt(Object key) { for(HashNode e = this.data[hash(key) & this.mask]; e != null; e = e.nextNode) { Object k; if((k = e.k) == key || equals(k, key)) return MapEntry.create(e.k, e.v); } return null; } public boolean containsKey(Object key) { return containsNodeKey(key); } @SuppressWarnings("unchecked") public Object compute(Object k, BiFunction bfn) { final int hash = hash(k); final HashNode[] d = this.data; final int idx = hash & this.mask; HashNode e = d[idx], ee = null; for(; e != null && !(e.k == k || equals(e.k, k)); e = e.nextNode) { ee = e; } Object newV = bfn.apply(k, e == null ? null : e.v); if(e != null) { if(newV != null) { e.v = newV; modify(e); } else removeHashNode(e, ee, idx); } else if(newV != null) { HashNode nn = newNode(k, hash, newV); if(ee != null) ee.nextNode = nn; else d[idx] = nn; checkResize(null); } return newV; } @SuppressWarnings("unchecked") public Object computeIfAbsent(Object k, Function afn) { final int hash = hash(k); final HashNode[] d = this.data; final int idx = hash & this.mask; HashNode e = d[idx], ee = null; for(; e != null && !(e.k == k || equals(e.k, k)); e = e.nextNode) { ee = e; } if(e != null) { return e.v; } else { final Object newv = afn.apply(k); if(newv != null) { HashNode nn = newNode(k, hash, newv); if(ee != null) ee.nextNode = nn; else d[idx] = nn; checkResize(null); } return newv; } } Object removeHashNode (HashNode e, HashNode lastNode, int loc) { dec(e); if(lastNode != null) lastNode.nextNode = e.nextNode; else this.data[loc] = e.nextNode; return e.getValue(); } public Object remove(Object key) { HashNode lastNode = null; int loc = hash(key) & this.mask; for(HashNode e = this.data[loc]; e != null; e = e.nextNode) { Object k; if((k = e.k) == key || equals(k, key)) { return removeHashNode(e, lastNode, loc); } lastNode = e; } return null; } public Object reduce(IFn rfn, Object acc) { final int l = data.length; final HashNode[] d = data; for(int idx = 0; idx < l; ++idx) { for(HashNode e = d[idx]; e != null; e = e.nextNode) { acc = rfn.invoke(acc, e); if(RT.isReduced(acc)) return ((IDeref)acc).deref(); } } return acc; } public Object kvreduce(IFn rfn, Object acc) { final int l = data.length; final HashNode[] d = data; for(int idx = 0; idx < l; ++idx) { for(HashNode e = d[idx]; e != null; e = e.nextNode) { acc = rfn.invoke(acc, e.k, e.v); if(RT.isReduced(acc)) return ((IDeref)acc).deref(); } } return acc; } public Object parallelReduction(IFn initValFn, IFn rfn, IFn mergeFn, ParallelOptions options ) { return Reductions.parallelCollectionReduction(initValFn, rfn, mergeFn, this.entrySet(), options); } @SuppressWarnings("unchecked") public void replaceAll(BiFunction bfn) { final int l = data.length; for(int idx = 0; idx < l; ++idx) { HashNode lastNode = null; for(HashNode e = this.data[idx]; e != null; e = e.nextNode) { Object newv = bfn.apply(e.k, e.v); if(newv != null) { e.v = newv; lastNode = e; } else { dec(e); if(lastNode != null) { lastNode.nextNode = e.nextNode; } else { data[idx] = e.nextNode; } } } } } public static class HashSetKeySet extends MapKeySet implements SetOps { public HashSetKeySet(HashMap hm) { super(hm); } public PersistentHashSet union(Collection c) { return new PersistentHashSet( new HashSet(((HashMap)data).shallowClone()).union(c) ); } public PersistentHashSet intersection(Set c) { return new PersistentHashSet( new HashSet(((HashMap)data).shallowClone()).intersection(c) ); } public PersistentHashSet difference(Set c) { return new PersistentHashSet( new HashSet(((HashMap)data).shallowClone()).difference(c) ); } } public Set keySet() { if(this.keySet == null ) this.keySet = new HashSetKeySet(this); return this.keySet; } @SuppressWarnings("unchecked") public static HashMap hashMapUnion(HashMap rv, HashMap om, BiFunction bfn) { final HashNode[] od = om.data; final int l = od.length; HashNode[] rvd = rv.data; int mask = rv.mask; for(int idx = 0; idx < l; ++idx) { for(HashNode lf = od[idx]; lf != null; lf = lf.nextNode) { final Object k = lf.k; final int hashcode = lf.hashcode; final int rvidx = hashcode & mask; HashNode init = rvd[rvidx], e = init; final Object v = lf.v; for(;e != null && !(e.k==k || rv.equals(e.k, k)); e = e.nextNode); if(e != null) { rvd[rvidx] = init.assoc(rv, e.k, hashcode, bfn.apply(e.v, v)); } else { if(init != null) rvd[rvidx] = init.assoc(rv, k, hashcode, v); else rvd[rvidx] = rv.newNode(k, hashcode, v); rv.checkResize(null); mask = rv.mask; rvd = rv.data; } } } return rv; } @SuppressWarnings("unchecked") public static HashMap reduceUnion(HashMap rv, IReduceInit o, BiFunction bfn) { return (HashMap)o.reduce(new IFnDef() { public Object invoke(Object acc, Object v) { Map.Entry lf = (Map.Entry)v; final Object k = lf.getKey(); final int hashcode = rv.hash(k); final int rvidx = hashcode & rv.mask; final HashNode[] rvd = rv.data; HashNode init = rvd[rvidx], e = init; for(;e != null && !(e.k==k || rv.equals(e.k, k)); e = e.nextNode); if(e != null) { rvd[rvidx] = init.assoc(rv, e.k, hashcode, bfn.apply(e.v, lf.getValue())); } else { if(init != null) rvd[rvidx] = init.assoc(rv, k, hashcode, lf.getValue()); else rvd[rvidx] = rv.newNode(k, hashcode, lf.getValue()); rv.checkResize(null); } return rv; } }, rv); } @SuppressWarnings("unchecked") public static HashMap kvReduceUnion(HashMap rv, IKVReduce o, BiFunction bfn) { return (HashMap)o.kvreduce(new IFnDef() { public Object invoke(Object acc, Object k, Object v) { final int hashcode = rv.hash(k); final int rvidx = hashcode & rv.mask; final HashNode[] rvd = rv.data; HashNode init = rvd[rvidx], e = init; for(;e != null && !(e.k==k || rv.equals(e.k, k)); e = e.nextNode); if(e != null) { rvd[rvidx] = init.assoc(rv, e.k, hashcode, bfn.apply(e.v, v)); } else { if(init != null) rvd[rvidx] = init.assoc(rv, k, hashcode, v); else rvd[rvidx] = rv.newNode(k, hashcode, v); rv.checkResize(null); } return rv; } }, rv); } @SuppressWarnings("unchecked") public static HashMap entrySetUnion(HashMap rv, Map o, BiFunction bfn) { int mask = rv.mask; HashNode[] rvd = rv.data; for(Object ee : o.entrySet()) { Map.Entry lf = (Map.Entry)ee; final Object k = lf.getKey(); final int hashcode = rv.hash(k); final int rvidx = hashcode & mask; HashNode init = rvd[rvidx], e = init; for(;e != null && !(e.k==k || rv.equals(e.k, k)); e = e.nextNode); if(e != null) { rvd[rvidx] = init.assoc(rv, e.k, hashcode, bfn.apply(e.v, lf.getValue())); } else { if(init != null) rvd[rvidx] = init.assoc(rv, k, hashcode, lf.getValue()); else rvd[rvidx] = rv.newNode(k, hashcode, lf.getValue()); rv.checkResize(null); mask = rv.mask; rvd = rv.data; } } return rv; } public static HashMap union(HashMap rv, Map o, BiFunction bfn) { if(o instanceof HashMap) { return hashMapUnion(rv, (HashMap)o, bfn); } else if (o instanceof IReduceInit) { return reduceUnion(rv, (IReduceInit)o, bfn); } else if (o instanceof IKVReduce) { return kvReduceUnion(rv, (IKVReduce)o, bfn); } else { return entrySetUnion(rv, o, bfn); } } @SuppressWarnings("unchecked") public HashMap union(Map o, BiFunction bfn) { return union(this, o, bfn); } @SuppressWarnings("unchecked") static HashMap intersection(HashMap rv, Map o, BiFunction bfn) { final HashNode[] rvd = rv.data; final int ne = rvd.length; for (int idx = 0; idx < ne; ++idx) { HashNode lf = rvd[idx]; while(lf != null) { final HashNode curlf = lf; lf = lf.nextNode; final Object v = o.get(curlf.k); rvd[idx] = (v != null) ? rvd[idx].assoc(rv, curlf.k, curlf.hashcode, bfn.apply(curlf.v, v)) : rvd[idx].dissoc(rv, curlf.k); } } return rv; } public HashMap intersection(Map o, BiFunction bfn) { return intersection(this, o, bfn); } public static HashMap intersection(HashMap rv, Set o) { final HashNode[] rvd = rv.data; final int ne = rvd.length; for (int idx = 0; idx < ne; ++idx) { HashNode lf = rvd[idx]; while(lf != null) { final HashNode curlf = lf; final Object k = curlf.k; lf = lf.nextNode; if(!o.contains(k)) rvd[idx] = rvd[idx].dissoc(rv,k); } } return rv; } public HashMap intersection(Set o) { return intersection(this, o); } @SuppressWarnings("unchecked") static HashMap difference(HashMap rv, Collection o) { final HashNode[] rvd = rv.data; final int mask = rv.mask; for (Object k : o) { final int hashcode = rv.hash(k); final int rvidx = hashcode & mask; HashNode e = rvd[rvidx]; for(;e != null && !(e.k==k || rv.equals(e.k, k)); e = e.nextNode); if(e != null) { rvd[rvidx] = rvd[rvidx].dissoc(rv,e.k); } } return rv; } public HashMap difference(Collection o) { return difference(this, o); } @SuppressWarnings("unchecked") static HashMap updateValues(HashMap rv, BiFunction valueMap) { final HashNode[] d = rv.data; final int nl = d.length; for(int idx = 0; idx < nl; ++idx) { HashNode lf = d[idx]; while(lf != null) { HashNode cur = lf; lf = lf.nextNode; Object newv = valueMap.apply(cur.k, cur.v); d[idx] = newv == null ? d[idx].dissoc(rv, cur.k) : d[idx].assoc(rv, cur.k, cur.hashcode, newv); } } return rv; } public HashMap updateValues(BiFunction valueMap) { return updateValues(this, valueMap); } @SuppressWarnings("unchecked") static HashMap updateValue(HashMap rv, Object k, Function fn) { final int hc = rv.hash(k); final int idx = hc & rv.mask; final HashNode[] data = rv.data; HashNode e = data[idx]; for(; e != null && !((e.k == k) || rv.equals(e.k, k)); e = e.nextNode); final Object newv = e != null ? fn.apply(e.v) : fn.apply(null); data[idx] = newv == null ? data[idx].dissoc(rv, k) : data[idx].assoc(rv, k, hc, newv); if(newv != null && e == null) rv.checkResize(null); return rv; } public HashMap updateValue(Object k, Function fn) { return updateValue(this, fn); } public Iterator iterator(Function leafFn) { return new HTIter(this.data, leafFn); } public Spliterator spliterator(Function leafFn) { return new HTSpliterator(this.data, this.length, leafFn); } } ================================================ FILE: java/ham_fisted/HashNode.java ================================================ package ham_fisted; import java.util.Map; import java.util.Iterator; import clojure.lang.IMapEntry; import ham_fisted.IMutList; public class HashNode implements Map.Entry, IMutList, IMapEntry { public final HashBase owner; public final int hashcode; public final Object k; //compute-at support means we can modify v. Object v; HashNode nextNode; public HashNode(HashBase _owner, Object _k, int hc, Object _v, HashNode nn) { owner = _owner; hashcode = hc; k = _k; v = _v; nextNode = nn; _owner.inc(this); } public HashNode(HashBase _owner, Object _k, int hc, Object _v) { this(_owner, _k, hc, _v, null); } public HashNode(HashBase _owner, Object _k, int hc) { this(_owner, _k, hc, null, null); } // Cloning constructor HashNode(HashBase _owner, HashNode prev) { owner = _owner; hashcode = prev.hashcode; k = prev.k; v = prev.v; nextNode = prev.nextNode; } public HashNode setOwner(HashBase nowner) { if (owner == nowner) return this; return new HashNode(nowner, this); } public HashNode clone(HashBase nowner) { HashNode rv = new HashNode(nowner, this); if(nextNode != null) rv.nextNode = nextNode.clone(nowner); return rv; } public final Object key() { return k; } public final Object val() { return v; } public final Object getKey() { return k; } public final Object getValue() { return v; } public Object setValue(Object vv) { Object rv = v; v = vv; return rv; } public final int size() { return 2; } public final Object get(int idx) { if(idx == 0) return k; if(idx == 1) return v; throw new RuntimeException("Index out of range."); } public HashNode assoc(HashBase nowner, Object _k, int hash, Object _v) { HashNode retval = setOwner(nowner); if (k == _k || nowner.equals(_k,k)) { retval.setValue(_v); } else { if (retval.nextNode != null) { retval.nextNode = retval.nextNode.assoc(nowner, _k, hash, _v); } else { retval.nextNode = nowner.newNode(_k, hash, _v); } } return retval; } public HashNode dissoc(HashBase nowner, Object _k) { if (k == _k || owner.equals(k, _k)) { nowner.dec(this); return nextNode; } if (nextNode != null) { HashNode nn = nextNode.dissoc(nowner,_k); if (nn != nextNode) { HashNode retval = setOwner(nowner); retval.nextNode = nn; return retval; } } return this; } } ================================================ FILE: java/ham_fisted/HashProvider.java ================================================ package ham_fisted; import java.util.Objects; public interface HashProvider { public default int hash(Object obj) { return obj != null ? IntegerOps.mixhash(obj.hashCode()) : 0; } public default boolean equals(Object lhs, Object rhs) { return Objects.equals(lhs,rhs); } } ================================================ FILE: java/ham_fisted/HashProviders.java ================================================ package ham_fisted; import clojure.lang.Util; import clojure.lang.IHashEq; public class HashProviders { public static final HashProvider equalHashProvider = new HashProvider(){}; /** * Hashcode provider using Clojure's hasheq/equiv pathway */ public static final HashProvider equivHashProvider = new HashProvider() { public int hash(Object obj) { return Util.hasheq(obj); } public boolean equals(Object lhs, Object rhs) { return CljHash.equiv(lhs,rhs); } }; //Equiv-pathway with small optimizations. public static final HashProvider hybridHashProvider = new HashProvider() { public int hash(Object k) { return k == null ? 0 : k instanceof IHashEq ? ((IHashEq)k).hasheq() : IntegerOps.mixhash(k.hashCode()); } public boolean equals(Object lhs, Object rhs) { return lhs == rhs ? true : lhs == null || rhs == null ? false : CljHash.nonNullEquiv(lhs,rhs); } }; public static final HashProvider defaultHashProvider = hybridHashProvider; } ================================================ FILE: java/ham_fisted/HashSet.java ================================================ package ham_fisted; import clojure.lang.IPersistentMap; import java.util.Iterator; import java.util.Set; import java.util.Spliterator; import java.util.Collection; import clojure.lang.RT; import clojure.lang.IDeref; import clojure.lang.IFn; import clojure.lang.IReduceInit; public class HashSet extends HashBase implements ISet, SetOps { public static final Object VALUE = new Object(); public HashSet(float loadFactor, int initialCapacity, int length, HashNode[] data, IPersistentMap meta) { super(loadFactor, initialCapacity, length, data, meta); } public HashSet() { this(0.75f, 0, 0, null, null); } public HashSet(IPersistentMap m) { this(0.75f, 0, 0, null, m); } public HashSet(HashBase other, IPersistentMap m) { super(other, m); } public HashSet(HashBase other) { super(other, null); } public HashSet shallowClone() { return new HashSet(loadFactor, capacity, length, data.clone(), meta); } public HashSet clone() { final int l = data.length; HashNode[] newData = new HashNode[l]; HashSet retval = new HashSet(loadFactor, capacity, length, newData, meta); for(int idx = 0; idx < l; ++idx) { HashNode orig = data[idx]; if(orig != null) newData[idx] = orig.clone(retval); } return retval; } public int hashCode() { return hasheq(); } public int hasheq() { return CljHash.setHashcode(this); } public boolean equals(Object o) { return equiv(o); } public boolean equiv(Object o) { return CljHash.setEquiv(this, o); } public boolean add(Object key) { final int hc = hash(key); final int idx = hc & this.mask; HashNode lastNode = null; //Avoid unneeded calls to both equals and checkResize for(HashNode e = this.data[idx]; e != null; e = e.nextNode) { lastNode = e; if(e.k == key || equals(e.k, key)) return false; } HashNode lf = newNode(key,hc,VALUE); if(lastNode != null) { lastNode.nextNode = lf; } else { data[idx] = lf; } checkResize(null); return true; } public void addAllReduceGeneric(IReduceInit r) { final HashBase rv = this; r.reduce(new IFnDef() { public Object invoke(Object acc, Object k) { final int hc = rv.hash(k); final int idx = hc & rv.mask; final HashNode[] d = rv.data; HashNode e = d[idx]; for(; e != null && !(k == e.k || rv.equals(k,e.k)); e = e.nextNode); if(e == null) { final HashNode lf = rv.newNode(k, hc, VALUE); lf.nextNode = d[idx]; d[idx] = lf; rv.checkResize(null); } return this; } }, this); } public boolean addAll(Collection c) { int sz = length; if(c instanceof HashSet) { HashSet other = (HashSet) c; HashNode[] d = data; int m = mask; final HashNode[] od = other.data; final int odl = od.length; for (int idx = 0; idx < odl; ++idx) { for (HashNode e = od[idx]; e != null; e = e.nextNode) { // Its tempting to use the hashcode here but we don't know if // the other hashset has overridden hash or not. int hc = hash(e.k); int didx = hc & m; HashNode ee = d[didx]; for(; ee != null && !(ee.k == e.k || equals(ee.k, e.k)); ee = ee.nextNode); if( ee == null) { HashNode n = newNode(e.k, hc, e.v); n.nextNode = d[didx]; d[didx] = n; checkResize(null); d = data; m = mask; } } } } else if(c instanceof IReduceInit) { addAllReduceGeneric((IReduceInit)c); } else { HashNode[] d = data; int m = mask; for(Object k: c) { int hc = hash(k); int idx = hc & m; HashNode e = d[idx]; for(; e != null && !(k == e.k || equals(k,e.k)); e = e.nextNode); if(e == null) { HashNode lf = newNode(k, hc, VALUE); lf.nextNode = d[idx]; d[idx] = lf; checkResize(null); d = data; m = mask; } } } return sz != length; } public boolean remove(Object key) { HashNode lastNode = null; int loc = hash(key) & this.mask; for(HashNode e = this.data[loc]; e != null; e = e.nextNode) { Object k; if((k = e.k) == key || equals(k, key)) { dec(e); if(lastNode != null) lastNode.nextNode = e.nextNode; else this.data[loc] = e.nextNode; return true; } lastNode = e; } return false; } public boolean contains(Object key) { return containsNodeKey(key); } public Iterator iterator() { return new HTIter(this.data, (e)->e.getKey()); } public Spliterator spliterator() { return new HTSpliterator(this.data, this.length, (e)->e.getKey()); } public Object reduce(IFn rfn, Object acc) { final int l = data.length; for(int idx = 0; idx < l; ++idx) { for(HashNode e = this.data[idx]; e != null; e = e.nextNode) { acc = rfn.invoke(acc, e.k); if(RT.isReduced(acc)) return ((IDeref)acc).deref(); } } return acc; } public static HashSet union(HashSet rv, Collection rhs) { if(rhs instanceof IReduceInit) { return (HashSet)((IReduceInit)rhs).reduce(new IFnDef() { public Object invoke(Object acc, Object k) { final int hashcode = rv.hash(k); final int rvidx = hashcode & rv.mask; HashNode init = rv.data[rvidx], e = init; for(;e != null && !(e.k==k || rv.equals(e.k, k)); e = e.nextNode); if(e == null) { if(init != null) rv.data[rvidx] = init.assoc(rv, k, hashcode, VALUE); else rv.data[rvidx] = rv.newNode(k, hashcode, VALUE); rv.checkResize(null); } return rv; } }, rv); } else { HashNode[] rvd = rv.data; int mask = rv.mask; for(Object k: rhs) { final int hashcode = rv.hash(k); final int rvidx = hashcode & mask; HashNode init = rvd[rvidx], e = init; for(;e != null && !(e.k==k || rv.equals(e.k, k)); e = e.nextNode); if(e == null) { if(init != null) rvd[rvidx] = init.assoc(rv, k, hashcode, VALUE); else rvd[rvidx] = rv.newNode(k, hashcode, VALUE); rv.checkResize(null); mask = rv.mask; rvd = rv.data; } } } return rv; } public HashSet union(Collection rhs) { return union(this, rhs); } public static HashSet intersection(HashSet rv, Set rhs) { final HashNode[] rvd = rv.data; final int ne = rvd.length; for (int idx = 0; idx < ne; ++idx) { HashNode lf = rvd[idx]; while(lf != null) { final Object k = lf.k; lf = lf.nextNode; if(!rhs.contains(k)) rvd[idx] = rvd[idx].dissoc(rv, k); } } return rv; } public HashSet intersection(Set rhs) { return intersection(this, rhs); } public static HashSet difference(HashSet rv, Set rhs) { final HashNode[] rvd = rv.data; final int mask = rv.mask; if(rhs instanceof IReduceInit) { return (HashSet)((IReduceInit)rhs).reduce(new IFnDef() { public Object invoke(Object acc, Object k) { final int hashcode = rv.hash(k); final int rvidx = hashcode & mask; HashNode e = rvd[rvidx]; for(;e != null && !(e.k==k || rv.equals(e.k, k)); e = e.nextNode); if(e != null) { rvd[rvidx] = rvd[rvidx].dissoc(rv, e.k); } return rv; } }, rv); } else { for (Object k : rhs) { final int hashcode = rv.hash(k); final int rvidx = hashcode & mask; HashNode e = rvd[rvidx]; for(;e != null && !(e.k==k || rv.equals(e.k, k)); e = e.nextNode); if(e != null) { rvd[rvidx] = rvd[rvidx].dissoc(rv, e.k); } } } return rv; } public HashSet difference(Set rhs) { HashSet rv = shallowClone(); final HashNode[] rvd = rv.data; final int mask = rv.mask; for (Object k : rhs) { final int hashcode = rv.hash(k); final int rvidx = hashcode & mask; HashNode e = rvd[rvidx]; for(;e != null && !(e.k==k || rv.equals(e.k, k)); e = e.nextNode); if(e != null) { rvd[rvidx] = rvd[rvidx].dissoc(rv, k); } } return rv; } } ================================================ FILE: java/ham_fisted/IAMapEntry.java ================================================ package ham_fisted; import clojure.lang.IMapEntry; import java.util.Map; public interface IAMapEntry extends Map.Entry, IMutList, IMapEntry { default Object setValue(Object v) { throw new UnsupportedOperationException(); } default Object key() { return getKey(); } default Object val() { return getValue(); } default int size() { return 2; } default Object get(int idx) { if(idx == 0) return getKey(); if(idx == 1) return getValue(); throw new RuntimeException("Index out of range."); } } ================================================ FILE: java/ham_fisted/IAPersistentMap.java ================================================ package ham_fisted; import java.util.Map; import clojure.lang.IEditableCollection; import clojure.lang.IPersistentMap; import clojure.lang.ITransientMap; interface IAPersistentMap extends Map, IEditableCollection, IPersistentMap { default int count() { return size(); } default IPersistentMap cons(Object v) { return (IPersistentMap)(asTransient().conj(v).persistent()); } default IPersistentMap assocEx(Object key, Object val) { if(containsKey(key)) throw new RuntimeException("Object already contains key :" + String.valueOf(key)); return assoc(key, val); } default IPersistentMap assoc(Object key, Object val) { return (IPersistentMap)(asTransient().assoc(key,val).persistent()); } default IPersistentMap without(Object key) { return (IPersistentMap)(asTransient().without(key).persistent()); } ITransientMap asTransient(); } ================================================ FILE: java/ham_fisted/IAPersistentSet.java ================================================ package ham_fisted; import clojure.lang.IPersistentSet; import clojure.lang.ITransientSet; import clojure.lang.IEditableCollection; public interface IAPersistentSet extends ISet, IEditableCollection, IPersistentSet { default int count() { return size(); } default Object get(Object o) { return contains(o) ? o : null; } default IPersistentSet cons(Object o) { return (IPersistentSet)asTransient().conj(o).persistent(); } default IPersistentSet disjoin(Object o) { return (IPersistentSet)asTransient().disjoin(o).persistent(); } ITransientSet asTransient(); } ================================================ FILE: java/ham_fisted/IATransientMap.java ================================================ package ham_fisted; import java.util.Map; import clojure.lang.Indexed; import clojure.lang.ITransientMap; import clojure.lang.ITransientAssociative2; import clojure.lang.IKVReduce; public interface IATransientMap extends ITransientMap, ITransientAssociative2 { default ITransientMap conjVal(Object val) { Object k, v; if(val instanceof Indexed) { Indexed ii = (Indexed)val; k = ii.nth(0); v = ii.nth(1); } else if (val instanceof Map.Entry) { Map.Entry ii = (Map.Entry)val; k = ii.getKey(); v = ii.getValue(); } else { throw new RuntimeException("Value must be either indexed or map entry :" + String.valueOf(val != null ? val.getClass() : "null")); } return assoc(k,v); } default ITransientMap conj(Object val) { if(val instanceof Map) { if(val instanceof IKVReduce) { return (IATransientMap) ((IKVReduce)val).kvreduce(new IFnDef() { public Object invoke(Object acc, Object k, Object v) { return ((ITransientAssociative2)acc).assoc(k,v); } }, this); } else { ITransientMap m = this; for(Object o: ((Map)val).entrySet()) { Map.Entry lf = (Map.Entry)o; m = (ITransientMap)((ITransientAssociative2)m).assoc(lf.getKey(), lf.getValue()); } return m; } } else { return conjVal(val); } } } ================================================ FILE: java/ham_fisted/IATransientSet.java ================================================ package ham_fisted; import clojure.lang.ITransientSet; public interface IATransientSet extends ITransientSet { default Object get(Object o) { return contains(o) ? o : null; } } ================================================ FILE: java/ham_fisted/ICollectionDef.java ================================================ package ham_fisted; import java.util.Collection; import java.util.Iterator; import java.util.Objects; import java.util.function.Predicate; public interface ICollectionDef extends Collection { default boolean add(E e) { throw new UnsupportedOperationException("Unimplemented"); } default boolean addAll(Collection c) { throw new UnsupportedOperationException("Unimplemented"); } default void clear() { throw new UnsupportedOperationException("Unimplemented"); } default boolean contains(Object o) { for(E e: this) { if(Objects.equals(e, o)) return true; } return false; } default boolean containsAll(Collection c) { throw new UnsupportedOperationException("Unimplemented"); } default boolean isEmpty() { return iterator().hasNext() == false; } default boolean remove(Object o) { throw new UnsupportedOperationException("Unimplemented"); } default boolean removeAll(Collection c) { throw new UnsupportedOperationException("Unimplemented"); } default boolean removeIf(Predicate filter) { throw new UnsupportedOperationException("Unimplemented"); } default boolean retainAll(Collection c) { throw new UnsupportedOperationException("Unimplemented"); } default Object[] toArray() { return ArrayLists.toArray(this); } default T[] toArray(T[] d) { return ArrayLists.toArray(this, d); } } ================================================ FILE: java/ham_fisted/IFnDef.java ================================================ package ham_fisted; import clojure.lang.IFn; import clojure.lang.ISeq; import clojure.lang.Util; import clojure.lang.RT; import clojure.lang.ArityException; import java.util.List; import java.util.RandomAccess; import java.util.ArrayList; import java.util.function.Supplier; import java.util.function.DoubleSupplier; import java.util.function.LongSupplier; import java.util.function.Function; import java.util.function.LongFunction; import java.util.function.ToLongFunction; import java.util.function.DoubleFunction; import java.util.function.ToDoubleFunction; import java.util.function.UnaryOperator; import java.util.function.DoubleUnaryOperator; import java.util.function.LongUnaryOperator; import java.util.function.BinaryOperator; import java.util.function.DoubleBinaryOperator; import java.util.function.LongBinaryOperator; //UnaryOperator and BinaryOperator have mutually invalid overloads for andThen so we can't implement //those here. public interface IFnDef extends IFn { default Object call() { return invoke(); } default void run(){ invoke(); } default Object invoke() { return throwArity(0); } default Object invoke(Object arg1) { return throwArity(1); } default Object invoke(Object arg1, Object arg2) { return throwArity(2); } default Object invoke(Object arg1, Object arg2, Object arg3) { return throwArity(3); } default Object invoke(Object arg1, Object arg2, Object arg3, Object arg4) { return throwArity(4); } default Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5) { return throwArity(5); } default Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6) { return throwArity(6); } default Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7) { return throwArity(7); } default Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8) { return throwArity(8); } default Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9) { return throwArity(9); } default Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10) { return throwArity(10); } default Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10, Object arg11) { return throwArity(11); } default Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10, Object arg11, Object arg12) { return throwArity(12); } default Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10, Object arg11, Object arg12, Object arg13) { return throwArity(13); } default Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10, Object arg11, Object arg12, Object arg13, Object arg14) { return throwArity(14); } default Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10, Object arg11, Object arg12, Object arg13, Object arg14, Object arg15) { return throwArity(15); } default Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10, Object arg11, Object arg12, Object arg13, Object arg14, Object arg15, Object arg16) { return throwArity(16); } default Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10, Object arg11, Object arg12, Object arg13, Object arg14, Object arg15, Object arg16, Object arg17) { return throwArity(17); } default Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10, Object arg11, Object arg12, Object arg13, Object arg14, Object arg15, Object arg16, Object arg17, Object arg18) { return throwArity(18); } default Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10, Object arg11, Object arg12, Object arg13, Object arg14, Object arg15, Object arg16, Object arg17, Object arg18, Object arg19) { return throwArity(19); } default Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10, Object arg11, Object arg12, Object arg13, Object arg14, Object arg15, Object arg16, Object arg17, Object arg18, Object arg19, Object arg20) { return throwArity(20); } default Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10, Object arg11, Object arg12, Object arg13, Object arg14, Object arg15, Object arg16, Object arg17, Object arg18, Object arg19, Object arg20, Object... args) { return throwArity(21); } default Object applyTo(ISeq arglist) { return ifaceApplyToHelper(this, Util.ret1(arglist,arglist = null)); } static public List asRandomAccess(Object arglist) { if(arglist instanceof RandomAccess) return (List)arglist; else if (arglist instanceof Object[]) return ArrayLists.toList((Object[])arglist); else return null; } @SuppressWarnings("unchecked") static public Object ifaceApplyToHelper(IFn ifn, Object arglist) { List args = null; if( arglist != null ) { args = asRandomAccess(arglist); if(args == null) { ISeq c = RT.seq(arglist); if(c != null) { ArrayList al = new ArrayList(); for(; c != null; c = c.next()) { al.add(c.first()); } args = al; } } arglist = null; } switch(args != null ? args.size() : 0) { case 0: return ifn.invoke(); case 1: return ifn.invoke(args.get(0)); case 2: return ifn.invoke(args.get(0), args.get(1)); case 3: return ifn.invoke(args.get(0), args.get(1), args.get(2)); case 4: return ifn.invoke(args.get(0), args.get(1), args.get(2), args.get(3)); case 5: return ifn.invoke(args.get(0), args.get(1), args.get(2), args.get(3), args.get(4)); case 6: return ifn.invoke(args.get(0), args.get(1), args.get(2), args.get(3), args.get(4), args.get(5)); case 7: return ifn.invoke(args.get(0), args.get(1), args.get(2), args.get(3), args.get(4), args.get(5), args.get(6)); case 8: return ifn.invoke(args.get(0), args.get(1), args.get(2), args.get(3), args.get(4), args.get(5), args.get(6), args.get(7)); case 9: return ifn.invoke(args.get(0), args.get(1), args.get(2), args.get(3), args.get(4), args.get(5), args.get(6), args.get(7), args.get(8)); case 10: return ifn.invoke(args.get(0), args.get(1), args.get(2), args.get(3), args.get(4), args.get(5), args.get(6), args.get(7), args.get(8), args.get(9)); case 11: return ifn.invoke(args.get(0), args.get(1), args.get(2), args.get(3), args.get(4), args.get(5), args.get(6), args.get(7), args.get(8), args.get(9), args.get(10)); case 12: return ifn.invoke(args.get(0), args.get(1), args.get(2), args.get(3), args.get(4), args.get(5), args.get(6), args.get(7), args.get(8), args.get(9), args.get(10), args.get(11)); case 13: return ifn.invoke(args.get(0), args.get(1), args.get(2), args.get(3), args.get(4), args.get(5), args.get(6), args.get(7), args.get(8), args.get(9), args.get(10), args.get(11), args.get(12)); case 14: return ifn.invoke(args.get(0), args.get(1), args.get(2), args.get(3), args.get(4), args.get(5), args.get(6), args.get(7), args.get(8), args.get(9), args.get(10), args.get(11), args.get(12), args.get(13)); case 15: return ifn.invoke(args.get(0), args.get(1), args.get(2), args.get(3), args.get(4), args.get(5), args.get(6), args.get(7), args.get(8), args.get(9), args.get(10), args.get(11), args.get(12), args.get(13), args.get(14)); case 16: return ifn.invoke(args.get(0), args.get(1), args.get(2), args.get(3), args.get(4), args.get(5), args.get(6), args.get(7), args.get(8), args.get(9), args.get(10), args.get(11), args.get(12), args.get(13), args.get(14), args.get(15)); case 17: return ifn.invoke(args.get(0), args.get(1), args.get(2), args.get(3), args.get(4), args.get(5), args.get(6), args.get(7), args.get(8), args.get(9), args.get(10), args.get(11), args.get(12), args.get(13), args.get(14), args.get(15), args.get(16)); case 18: return ifn.invoke(args.get(0), args.get(1), args.get(2), args.get(3), args.get(4), args.get(5), args.get(6), args.get(7), args.get(8), args.get(9), args.get(10), args.get(11), args.get(12), args.get(13), args.get(14), args.get(15), args.get(16), args.get(17)); case 19: return ifn.invoke(args.get(0), args.get(1), args.get(2), args.get(3), args.get(4), args.get(5), args.get(6), args.get(7), args.get(8), args.get(9), args.get(10), args.get(11), args.get(12), args.get(13), args.get(14), args.get(15), args.get(16), args.get(17), args.get(18)); case 20: return ifn.invoke(args.get(0), args.get(1), args.get(2), args.get(3), args.get(4), args.get(5), args.get(6), args.get(7), args.get(8), args.get(9), args.get(10), args.get(11), args.get(12), args.get(13), args.get(14), args.get(15), args.get(16), args.get(17), args.get(18), args.get(19)); default: return ifn.invoke(args.get(0), args.get(1), args.get(2), args.get(3), args.get(4), args.get(5), args.get(6), args.get(7), args.get(8), args.get(9), args.get(10), args.get(11), args.get(12), args.get(13), args.get(14), args.get(15), args.get(16), args.get(17), args.get(18), args.get(19), args.subList(20, args.size()).toArray()); } } default Object throwArity(int n){ String name = getClass().getName(); throw new ArityException(n, name); } public interface O extends IFnDef, Supplier { default Object get() { return invoke(); } } public interface D extends IFnDef, DoubleSupplier, Supplier, IFn.D { default double getAsDouble() { return invokePrim(); } default Object get() { return invokePrim(); } default Object invoke() { return invokePrim(); } } public interface L extends IFnDef, DoubleSupplier, Supplier, IFn.L { default long getAsLong() { return invokePrim(); } default Object get() { return invokePrim(); } default Object invoke() { return invokePrim(); } } public interface OO extends IFnDef, UnaryOperator { default Object apply(Object arg) { return invoke(arg); } } public interface OOO extends IFnDef, BinaryOperator { default Object apply(Object l, Object r) { return invoke(l,r); } } public interface LO extends IFnDef, IFn.LO, LongFunction { default Object invoke(Object arg) { return invokePrim(Casts.longCast(arg)); } default Object apply(long v) { return invokePrim(v); } } public interface LongPredicate extends IFnDef, IFn.LO, LongFunction, java.util.function.LongPredicate { default Object invokePrim(long arg) { return test(arg); } default Object invoke(Object arg) { return test(Casts.longCast(arg)); } default Object apply(long v) { return test(v); } } public interface LL extends IFnDef, IFn.LL, LongUnaryOperator { default Object invoke(Object arg) { return invokePrim(Casts.longCast(arg)); } default long applyAsLong(long v) { return invokePrim(v); } } public interface OL extends IFnDef, IFn.OL, ToLongFunction { default Object invoke(Object arg) { return invokePrim(arg); } default long applyAsLong(Object v) { return invokePrim(v); } } public interface DO extends IFnDef, IFn.DO, DoubleFunction { default Object invoke(Object arg) { return invokePrim(Casts.doubleCast(arg)); } default Object apply(double v) { return invokePrim(v); } } public interface DoublePredicate extends IFnDef, IFn.DO, DoubleFunction, java.util.function.DoublePredicate { default Object invoke(Object arg) { return test(Casts.doubleCast(arg)); } default Object invokePrim(double arg) { return test(arg); } default Object apply(double v) { return test(v); } default DoublePredicate negate() { DoublePredicate prev = this; return new DoublePredicate() { public boolean test(double v) { return !prev.test(v); } }; } } public interface DD extends IFnDef, IFn.DD, DoubleUnaryOperator { default Object invoke(Object arg) { return invokePrim(Casts.doubleCast(arg)); } default double applyAsDouble(double v) { return invokePrim(v); } } public interface OD extends IFnDef, IFn.OD, ToDoubleFunction { default Object invoke(Object arg) { return invokePrim(arg); } default double applyAsDouble(Object v) { return invokePrim(v); } } public interface LD extends IFnDef, IFn.LD { default Object invoke(Object arg) { return invokePrim(Casts.longCast(arg)); } } public interface DL extends IFnDef, IFn.DL { default Object invoke(Object arg) { return invokePrim(Casts.doubleCast(arg)); } } @SuppressWarnings("unchecked") public interface Predicate extends IFnDef, UnaryOperator, java.util.function.Predicate { default Object invoke(Object v) { return test(v); } default Object apply(Object arg) { return test(arg); } } public interface DDD extends IFnDef, IFn.DDD, DoubleBinaryOperator { default Object invoke(Object lhs, Object rhs) { return invokePrim(Casts.doubleCast(lhs), Casts.doubleCast(rhs)); } default double applyAsDouble(double l, double r) { return invokePrim(l,r); } } public interface LLL extends IFnDef, IFn.LLL, LongBinaryOperator { default Object invoke(Object lhs, Object rhs) { return invokePrim(Casts.longCast(lhs), Casts.longCast(rhs)); } default long applyAsLong(long l, long r) { return invokePrim(l,r); } } public interface ODO extends IFnDef, IFn.ODO { default Object invoke(Object lhs, Object rhs) { return invokePrim(lhs, Casts.doubleCast(rhs)); } } public interface OLO extends IFnDef, IFn.OLO { default Object invoke(Object lhs, Object rhs) { return invokePrim(lhs, Casts.longCast(rhs)); } } public interface LLO extends IFnDef, IFn.LLO { default Object invoke(Object l, Object r) { return invokePrim(Casts.longCast(l), Casts.longCast(r)); } } public interface OLOO extends IFnDef, IFn.OLOO { default Object invoke(Object acc, Object idx, Object v) { return invokePrim(acc, Casts.longCast(idx), v); } } public interface OLDO extends IFnDef, IFn.OLDO { default Object invoke(Object acc, Object idx, Object v) { return invokePrim(acc, Casts.longCast(idx), Casts.doubleCast(v)); } } public interface OLLO extends IFnDef, IFn.OLLO { default Object invoke(Object acc, Object idx, Object v) { return invokePrim(acc, Casts.longCast(idx), Casts.longCast(v)); } } public interface DDDD extends IFnDef, IFn.DDDD { default Object invoke (Object a, Object b, Object c) { return invokePrim(Casts.doubleCast(a), Casts.doubleCast(b), Casts.doubleCast(c)); } } public interface LLLL extends IFnDef, IFn.LLLL { default Object invoke (Object a, Object b, Object c) { return invokePrim(Casts.longCast(a), Casts.longCast(b), Casts.longCast(c)); } } } ================================================ FILE: java/ham_fisted/IMap.java ================================================ package ham_fisted; import java.util.Map; import java.util.Set; import java.util.Iterator; import java.util.Spliterator; import java.util.Collection; import java.util.AbstractSet; import java.util.AbstractCollection; import java.util.function.Function; import java.util.function.Consumer; import java.util.function.BiConsumer; import java.util.Objects; import clojure.lang.IFn; import clojure.lang.ILookup; import clojure.lang.IMapIterable; import clojure.lang.Counted; import clojure.lang.MapEquivalence; import clojure.lang.IKVReduce; import clojure.lang.Seqable; import clojure.lang.ISeq; import clojure.lang.RT; public interface IMap extends Map, ITypedReduce, ILookup, IFnDef, Iterable, IMapIterable, Counted, MapEquivalence, IKVReduce, Seqable { Iterator iterator(Function fn); default Spliterator spliterator(Function fn) { throw new RuntimeException("Unimplemented"); } default Iterator keyIterator() { return iterator((k)->k.getKey()); } default Iterator valIterator() { return iterator((k)->k.getValue()); } default Iterator entryIterator() { return iterator((k)->k); } default Iterator iterator() { return entryIterator(); } @SuppressWarnings("unchecked") default void putAll(Map data) { data.forEach(new BiConsumer() { public void accept(Object k, Object v) { put(k,v); } }); } default boolean containsValue(Object val) { return values().contains(val); } default Object valAt(Object k) { return get(k); } @SuppressWarnings("unchecked") default Object valAt(Object k, Object defVal) { return getOrDefault(k, defVal); } default Object invoke(Object k) { return get(k); } @SuppressWarnings("unchecked") default Object invoke(Object k, Object defVal) { return getOrDefault(k, defVal); } default boolean isEmpty() { return this.size() == 0; } @SuppressWarnings("unchecked") default void forEach(Consumer c) { ITypedReduce.super.forEach(c); } default int count() { return size(); } default Object kvreduce(IFn f, Object init) { return reduce(new IFnDef() { public Object invoke(Object acc, Object v) { final Map.Entry lf = (Map.Entry)v; return f.invoke(acc, lf.getKey(), lf.getValue()); } }, init); } default ISeq seq() { return LazyChunkedSeq.chunkIteratorSeq(iterator()); } public static class MapKeySet extends AbstractSet implements ITypedReduce, IFnDef, Counted { public final IMap data; public MapKeySet(IMap data) { this.data = data; } public int size() { return data.size(); } public int count() { return data.size(); } public boolean contains(Object k) { return data.containsKey(k); } public Iterator iterator() { return data.keyIterator(); } public Spliterator spliterator() { return data.spliterator( (k)->k.getKey() ); } public void clear() { data.clear(); } static IFn wrapRfn(IFn rfn) { return new IFnDef() { public Object invoke(Object acc, Object v) { return rfn.invoke(acc, ((Map.Entry)v).getKey()); } }; } public Object reduce(IFn rfn, Object acc) { return data.reduce(wrapRfn(rfn), acc); } public Object parallelReduction(IFn initValFn, IFn rfn, IFn mergeFn, ParallelOptions options ) { return data.parallelReduction(initValFn, wrapRfn(rfn), mergeFn, options); } @SuppressWarnings("unchecked") public void forEach(Consumer c) { reduce( new IFnDef() { public Object invoke(Object lhs, Object rhs) { c.accept(rhs); return c; } }, c); } public Object invoke(Object k) { return data.containsKey(k) ? k : null; } } default Set keySet() { return new MapKeySet(this); } public static class MapEntrySet extends AbstractSet implements ITypedReduce, IFnDef, Counted { public final IMap data; public MapEntrySet(IMap data) { this.data = data; } public int size() { return data.size(); } public int count() { return data.size(); } public boolean contains(Object k) { if(!(k instanceof Map.Entry)) return false; Map.Entry e = (Map.Entry)k; return Objects.equals(data.get(e), e.getValue()); } public Iterator iterator() { return data.entryIterator(); } public Spliterator spliterator() { return data.spliterator( (k)->k ); } public void clear() { data.clear(); } public Object reduce(IFn rfn, Object acc) { return data.reduce(rfn, acc); } public Object parallelReduction(IFn initValFn, IFn rfn, IFn mergeFn, ParallelOptions options ) { return data.parallelReduction(initValFn, rfn, mergeFn, options); } @SuppressWarnings("unchecked") public void forEach(Consumer c) { data.forEach(c); } public Object invoke(Object k) { return contains(k); } } default Set entrySet() { return new MapEntrySet(this); } public static class ValueCollection extends AbstractCollection implements ITypedReduce, Counted { public final IMap data; public ValueCollection(IMap data) { this.data = data;} public final int size() { return data.size(); } public int count() { return data.size(); } public final void clear() { data.clear(); } public final Iterator iterator() { return data.valIterator(); } @SuppressWarnings("unchecked") public final Spliterator spliterator() { return data.spliterator((k)->k.getValue()); } public static IFn wrapRfn(IFn val) { return new IFnDef() { public Object invoke(Object acc, Object e) { return val.invoke(acc, ((Map.Entry)e).getValue()); } }; } public Object reduce(IFn rfn, Object acc) { return data.reduce(wrapRfn(rfn), acc); } public Object parallelReduction(IFn initValFn, IFn rfn, IFn mergeFn, ParallelOptions options ) { return data.parallelReduction(initValFn, wrapRfn(rfn), mergeFn, options); } @SuppressWarnings("unchecked") public void forEach(Consumer c) { reduce( new IFnDef() { public Object invoke(Object lhs, Object rhs) { c.accept(rhs); return c; } }, c); } } default Collection values() { return new ValueCollection(this); } } ================================================ FILE: java/ham_fisted/IMutList.java ================================================ package ham_fisted; import java.util.List; import java.util.RandomAccess; import java.util.Iterator; import java.util.ListIterator; import java.util.Objects; import java.util.NoSuchElementException; import java.util.Collection; import java.util.Arrays; import java.util.Comparator; import java.util.Random; import java.util.Collections; import java.util.Spliterator; import java.util.function.DoubleBinaryOperator; import java.util.function.LongBinaryOperator; import java.util.function.Consumer; import java.util.function.DoubleConsumer; import java.util.function.LongConsumer; import java.util.stream.IntStream; import java.util.stream.LongStream; import java.util.stream.DoubleStream; import java.util.stream.Stream; import it.unimi.dsi.fastutil.objects.ObjectArrays; import it.unimi.dsi.fastutil.ints.IntArrays; import it.unimi.dsi.fastutil.ints.IntComparator; import it.unimi.dsi.fastutil.longs.LongComparator; import it.unimi.dsi.fastutil.floats.FloatComparator; import it.unimi.dsi.fastutil.doubles.DoubleComparator; import clojure.lang.Indexed; import clojure.lang.IReduce; import clojure.lang.IKVReduce; import clojure.lang.IHashEq; import clojure.lang.Seqable; import clojure.lang.Sequential; import clojure.lang.Reversible; import clojure.lang.RT; import clojure.lang.Util; import clojure.lang.IFn; import clojure.lang.IDeref; import clojure.lang.IMapEntry; import clojure.lang.MapEntry; import clojure.lang.IteratorSeq; import clojure.lang.ISeq; import clojure.lang.IPersistentVector; import clojure.lang.IPersistentMap; import clojure.lang.IObj; import clojure.lang.ASeq; import clojure.lang.IReduceInit; import clojure.lang.Associative; import clojure.lang.IChunk; import clojure.lang.IChunkedSeq; import clojure.lang.PersistentList; import clojure.lang.ChunkedCons; import clojure.lang.ArrayChunk; public interface IMutList extends List, RandomAccess, Indexed, IFnDef, ITypedReduce, IKVReduce, IReduce, IHashEq, Seqable, Reversible, IObj, ImmutSort, RangeList, Cloneable, Sequential, Associative, Comparable { default IMutList cloneList() { return (IMutList)ArrayLists.toList(toArray()); } default void clear() { throw new UnsupportedOperationException("Unimplemented"); } default boolean add(E v) { throw new UnsupportedOperationException("Unimplemented"); } default void add(int idx, E v) { add(idx, 1, v); } default void add(int idx, int count, E v) { int end = idx + count; if(idx == size()) { for(; idx < end; ++idx) add( v ); } else { for(; idx < end; ++idx) add( idx, v ); } } @SuppressWarnings("unchecked") default void addLong(long v) { add((E)Long.valueOf(v)); } @SuppressWarnings("unchecked") default void addDouble(double v) { add((E)Double.valueOf(v)); } default void removeRange(long startidx, long endidx) { ChunkedList.checkIndexRange(0, size(), startidx, endidx); final int sidx = (int)startidx; for(; startidx < endidx; ++startidx) { remove(sidx); } } @SuppressWarnings("unchecked") default void fillRange(long startidx, final long endidx, Object v) { final int sz = size(); ChunkedList.checkIndexRange(0, (long)sz, startidx, endidx); int ss = (int)startidx; final int ee = (int)endidx; for(; ss < ee; ++ss) { set(ss, (E)v); } } @SuppressWarnings("unchecked") default void fillRangeReducible(final long startidx, Object v) { final int sz = size(); if (v instanceof RandomAccess) { ChunkedList.checkIndexRange(0, sz, startidx, startidx+((List)v).size()); } final int ss = (int)startidx; Reductions.serialReduction(new Reductions.IndexedAccum(new IFnDef.OLOO() { public Object invokePrim(Object acc, long idx, Object v) { ((List)acc).set((int)idx+ss, v); return acc; } }), this, v); } default E remove(int idx) { final E retval = get(idx); removeRange(idx, idx+1); return retval; } default boolean remove(Object o) { final int idx = indexOf(o); if ( idx == -1) return false; remove(idx); return true; } default boolean addAll(Collection c) { return addAllReducible(c); } @SuppressWarnings("unchecked") default boolean addAllReducible(Object obj) { final int sz = size(); Reductions.serialReduction(new IFnDef() { public Object invoke(Object lhs, Object rhs) { ((IMutList)lhs).add((E)rhs); return lhs; } }, this, obj); return sz != size(); } default boolean addAll(int idx, Collection c) { if (c.isEmpty()) return false; final int sz = size(); if (idx == sz) return addAll(c); for (E e: c) add(idx++, e); return true; } default boolean removeAll(Collection c) { // HashSet hs = new HashSet(); // hs.addAll(c); // int sz = size(); // final int osz = sz; // for(int idx = 0; idx < sz; ++idx) { // if(hs.contains(get(idx))) { // remove(idx); // --idx; // --sz; // } // } // return size() == osz; throw new UnsupportedOperationException(); } default boolean retainAll(Collection c) { // HashSet hs = new HashSet(); // hs.addAll(c); // int sz = size(); // final int osz = sz; // for(int idx = 0; idx < sz; ++idx) { // if(!hs.contains(get(idx))) { // remove(idx); // --idx; // --sz; // } // } // return size() == osz; throw new UnsupportedOperationException(); } public static class MutSubList implements IMutList { final IMutList list; final int sidx; final int eidx; final int nElems; public MutSubList(IMutList list, int ss, int ee) { this.list = list; sidx = ss; eidx = ee; nElems = ee - ss; } public int size() { return nElems; } public E get(int idx) { ChunkedList.indexCheck(sidx, nElems, idx); return list.get(sidx+idx); } public long getLong(int idx) { ChunkedList.indexCheck(sidx, nElems, idx); return list.getLong(sidx+idx); } public double getDouble(int idx) { ChunkedList.indexCheck(sidx, nElems, idx); return list.getDouble(sidx+idx); } @SuppressWarnings("unchecked") public E set(int idx, E v) { ChunkedList.indexCheck(sidx, nElems, idx); return list.set(sidx+idx, v); } public void setLong(int idx, long v) { ChunkedList.indexCheck(sidx, nElems, idx); list.setLong(sidx+idx, v); } public void setDouble(int idx, double v) { ChunkedList.indexCheck(sidx, nElems, idx); list.setDouble(sidx+idx, v); } public Object reduce(IFn rfn, Object init) { final int ee = eidx; final IMutList l = list; for(int idx = sidx; idx < ee && !RT.isReduced(init); ++idx) init = rfn.invoke(init, l.get(idx)); return Reductions.unreduce(init); } @SuppressWarnings("unchecked") public IMutList subList(int ssidx, int seidx) { ChunkedList.sublistCheck(ssidx, seidx, nElems); if(ssidx == 0 && seidx == nElems) return this; return list.subList(ssidx + sidx, seidx + sidx); } public IPersistentMap meta() { return list.meta(); } @SuppressWarnings("unchecked") public IMutList withMeta(IPersistentMap meta) { return ((IMutList)list.withMeta(meta)).subList(sidx, eidx); } } @SuppressWarnings("unchecked") default IMutList subList(int startidx, int endidx) { final int sz = size(); if (startidx == 0 && endidx == sz) return this; ChunkedList.sublistCheck(startidx, endidx, size()); return new MutSubList(this, startidx, endidx); } default int indexOf(Object o) { final int sz = size(); for(int idx = 0; idx < sz; ++idx) if (Objects.equals(o, get(idx))) return idx; return -1; } default int lastIndexOf(Object o) { final int sz = size(); final int ssz = sz - 1; for(int idx = 0; idx < sz; ++idx) { int ridx = ssz - idx; if (Objects.equals(o, get(ridx))) return ridx; } return -1; } default boolean contains(Object o) { return indexOf(o) != -1; } default boolean isEmpty() { return size() == 0; } default int compareTo(Object o) { final List l = (List)o; final int sz = size(); final int lsz = l.size(); if(sz < lsz) return -1; else if(sz > lsz) return 1; for(int i = 0; i < sz; i++) { int c = Util.compare(get(i),l.get(i)); if(c != 0) return c; } return 0; } public static class ListIter implements ListIterator { List list; int idx = 0; int previdx = 0; ListIter(List ls, int _idx){list = ls; idx = _idx; previdx = _idx;} public final boolean hasNext() { return idx < list.size(); } public final boolean hasPrevious() { return idx > 0; } public final E next() { if(!hasNext()) throw new NoSuchElementException(); final E retval = list.get(idx); previdx = idx; ++idx; return retval; } public final E previous() { if(!hasPrevious()) throw new NoSuchElementException(); --idx; previdx = idx; return list.get(idx); } public final int nextIndex() { return idx; } public final int previousIndex() { return idx-1; } public final void remove() { list.remove(previdx); } public final void set(E e) { list.set(previdx, e); } public final void add(E e) { list.add(previdx, e); } } default ListIterator listIterator(int idx) { if (idx < 0 || idx > size()) throw new NoSuchElementException("Index(" + String.valueOf(idx) + ") out of range 0-" + size()); return new ListIter(this, idx); } default ListIterator listIterator() { return listIterator(0); } default Iterator iterator() { return listIterator(0); } public class RIter implements Iterator { List list; int idx; public RIter(List ls) { list = ls; idx = 0; } public final boolean hasNext() { return idx < list.size(); } public final E next() { if(!hasNext()) throw new NoSuchElementException(); int ridx = list.size() - idx - 1; ++idx; return list.get(ridx); } } default Iterator riterator() { return new RIter(this); } default Spliterator spliterator() { return new RandomAccessSpliterator(this); } default Object[] fillArray(Object[] data) { Reductions.serialReduction(new Reductions.IndexedAccum(new IFnDef.OLOO() { public Object invokePrim(Object acc, long idx, Object v) { ((Object[])acc)[(int)idx] = v; return acc; } }), data, this); return data; } default Object[] toArray() { return fillArray(new Object[size()]); } default T[] toArray(T[] marker) { final T[] retval = Arrays.copyOf(marker, size()); fillArray(retval); return retval; } default Object toNativeArray() { return toArray(); } default int[] toIntArray() { final int[] retval = new int[size()]; ArrayLists.toList(retval).fillRange(0, this); return retval; } default long[] toLongArray() { final long[] retval = new long[size()]; ArrayLists.toList(retval).fillRange(0, this); return retval; } default float[] toFloatArray() { final float[] retval = new float[size()]; ArrayLists.toList(retval).fillRange(0, this); return retval; } default double[] toDoubleArray() { final double[] retval = new double[size()]; ArrayLists.toList(retval).fillRange(0, this); return retval; } @SuppressWarnings("unchecked") default IntComparator indexComparator() { return new IntComparator() { public int compare(int lidx, int ridx) { return ((Comparable)get(lidx)).compareTo(get(ridx)); } }; } @SuppressWarnings("unchecked") default IntComparator indexComparator(Comparator c) { if(c instanceof DoubleComparator) { final DoubleComparator dc = (DoubleComparator)c; return new IntComparator() { public int compare(int lidx, int ridx) { return dc.compare(getDouble(lidx), getDouble(ridx)); } }; } else if (c instanceof IntComparator) { final IntComparator dc = (IntComparator)c; return new IntComparator() { public int compare(int lidx, int ridx) { return dc.compare((int)getLong(lidx), (int)getLong(ridx)); } }; } else if (c instanceof LongComparator) { final LongComparator dc = (LongComparator)c; return new IntComparator() { public int compare(int lidx, int ridx) { return dc.compare(getLong(lidx), getLong(ridx)); } }; } else if (c instanceof FloatComparator) { final FloatComparator dc = (FloatComparator)c; return new IntComparator() { public int compare(int lidx, int ridx) { return dc.compare((float)getDouble(lidx), (float)getDouble(ridx)); } }; } else { return new IntComparator() { public int compare(int lidx, int ridx) { return c.compare(get(lidx), get(ridx)); } }; } } @SuppressWarnings("unchecked") default int[] sortIndirect(Comparator c) { final int sz = size(); int[] retval = ArrayLists.iarange(0, sz, 1); final Object[] data = toArray(); if (c == null) ObjectArrays.parallelQuickSortIndirect(retval, data); else IntArrays.parallelQuickSort(retval, new IntComparator() { public int compare(int lhs, int rhs) { return c.compare(data[lhs], data[rhs]); } }); return retval; } default Object nth(int idx) { final int sz = size(); if (idx < 0) idx = idx + sz; return get(idx); } default Object nth(int idx, Object notFound) { final int sz = size(); if (idx < 0) idx = idx + sz; return idx < sz && idx > -1 ? get(idx) : notFound; } default E set(int idx, E v) { throw new UnsupportedOperationException("Unimplemented"); } @SuppressWarnings("unchecked") default void setLong(int idx, long v) { set(idx, (E)Long.valueOf(v)); } @SuppressWarnings("unchecked") default void setDouble(int idx, double v) { set(idx, (E)Double.valueOf(v)); } default long getLong(int idx) { return Casts.longCast(get(idx)); } default double getDouble(int idx) { final Object obj = get(idx); return obj != null ? Casts.doubleCast(obj) : Double.NaN; } default void accPlusLong(int idx, long val) { setLong( idx, getLong(idx) + val ); } default void accPlusDouble(int idx, double val) { setDouble( idx, getDouble(idx) + val ); } default Object invoke(Object idx) { return nth(RT.intCast(Casts.longCast(idx))); } default Object invoke(Object idx, Object notFound) { if(Util.isInteger(idx)) return nth(RT.intCast(Casts.longCast(idx)), notFound); return notFound; } default Object valAt(Object idx) { return invoke(idx); } default Object valAt(Object idx, Object def) { return invoke(idx, def); } default IMapEntry entryAt(Object key) { if(Util.isInteger(key)) { int idx = RT.intCast(key); if (idx >= 0 && idx < size()) return MapEntry.create(idx, get(idx)); } return null; } @SuppressWarnings("unimplemented") default boolean containsAll(Collection c) { // HashSet hc = new HashSet(); // hc.addAll(c); // for(E e: this) { // if(!hc.contains(e)) // return false; // } // return true; throw new UnsupportedOperationException(); } default boolean containsKey(Object key) { if(Util.isInteger(key)) { int idx = RT.intCast(key); if (idx >= 0 && idx < size()) return true; } return false; } default int count() { return size(); } default int length() { return size(); } default Object reduce(IFn f) { final int sz = size(); if (sz == 0 ) return f.invoke(); Object init = get(0); for(int idx = 1; idx < sz && (!RT.isReduced(init)); ++idx) { init = f.invoke(init, get(idx)); } return Reductions.unreduce(init); } default Object reduce(IFn f, Object init) { final int sz = size(); for(int idx = 0; idx < sz && (!RT.isReduced(init)); ++idx) { init = f.invoke(init, get(idx)); } return Reductions.unreduce(init); } default Object kvreduce(IFn f, Object init) { final int sz = size(); for(int idx = 0; idx < sz && (!RT.isReduced(init)); ++idx) { init = f.invoke(init, idx, get(idx)); } return Reductions.unreduce(init); } default Object parallelReduction(IFn initValFn, IFn rfn, IFn mergeFn, ParallelOptions options) { return Reductions.parallelRandAccessReduction(initValFn, rfn, mergeFn, this, options); } @SuppressWarnings("unchecked") default void forEach(Consumer c) { ITypedReduce.super.forEach(c); } default int hasheq() { return CljHash.listHasheq(this); } default boolean equiv(Object other) { return CljHash.listEquiv(this, other); } public static IChunk sublistAsChunk(List data, int sidx) { final int nElems = data.size(); final int len = nElems - sidx; if(len > 0) { return new IChunk() { public int count() { return len; } public Object nth(int idx, Object defVal) { return (idx < len) ? data.get(idx+sidx) : defVal; } public Object nth(int idx) { if(idx < len) return data.get(idx+sidx); throw new IndexOutOfBoundsException(); } public IChunk dropFirst() { return len > 1 ? sublistAsChunk(data, sidx+1) : null; } public Object reduce(IFn rfn, Object acc) { final int ne = len; for(int idx = 0; idx < ne; ++idx) { acc = rfn.invoke(acc, data.get(idx+sidx)); if(RT.isReduced(acc)) return ((IDeref)acc).deref(); } return acc; } }; } return null; } public static IChunkedSeq inplaceSublistSeq(List l, int sidx, int eidx) { final int ne = eidx - sidx; if(ne > 0) { final int len = Math.min(32, ne); return new LazyChunkedSeq(new IFnDef() { public Object invoke() { return new ChunkedCons(sublistAsChunk(l.subList(sidx, sidx + len), 0), (ne - len) <= 0 ? null : inplaceSublistSeq(l, sidx+len, eidx)); } }); } else { return null; } } default IChunkedSeq inplaceSublistSeq() { if(isEmpty()) return null; return inplaceSublistSeq(this, 0, size()); } public static IChunkedSeq copyingArraySeq(List l, int sidx, int eidx) { final int ne = eidx - sidx; if(ne > 0) { final int len = Math.min(32, ne); return new LazyChunkedSeq(new IFnDef() { public Object invoke() { return new ChunkedCons(new ArrayChunk(l.subList(sidx, sidx + len).toArray()), (ne - len) <= 0 ? null : inplaceSublistSeq(l, sidx+len, eidx)); } }); } else { return null; } } default IChunkedSeq copyingArraySeq() { if(isEmpty()) return null; return copyingArraySeq(this, 0, size()); } default ISeq seq() { return copyingArraySeq(); } default ISeq rseq() { return isEmpty() ? null : new ReverseList(this, meta()).seq(); } default IPersistentMap meta() { return null; } default IObj withMeta(IPersistentMap meta ) { throw new UnsupportedOperationException("Unimplemented"); } @SuppressWarnings("unchecked") default void sort(Comparator c) { Comparator cc = (Comparator)c; Object[] finalData; if(c == null) { final Comparable[] data = toArray(new Comparable[0]); Arrays.parallelSort(data); finalData = data; } else { finalData = toArray(); Arrays.parallelSort(finalData, cc); } fillRange(0, ArrayLists.toList(finalData)); } default void shuffle(Random r) { Collections.shuffle(this, r); } default List reindex(int[] indexes) { return ReindexList.create(indexes, this, this.meta()); } default List immutShuffle(Random r) { final Object[] retval = toArray(); ObjectArrays.shuffle(retval, r); return ArrayLists.toList(retval, 0, size(), meta()); } default List reverse() { return ReverseList.create(this, meta()); } @SuppressWarnings("unchecked") default int binarySearch(E v, Comparator c) { int rv; if(c == null) rv = Collections.binarySearch(this,v,new Comparator() { public int compare(E lhs, E rhs) { return Util.compare(lhs, rhs); } }); else rv = Collections.binarySearch(this,v,c); return rv < 0 ? -1 - rv : rv; } default int binarySearch(E v) { return binarySearch(v, null); } default IPersistentVector immut() { return ArrayImmutList.create(true, toArray(), 0, size(), meta()); } default Associative assoc(Object idx, Object o) { return immut().assoc(idx, o); } default IPersistentVector cons(Object o) { return immut().cons(o); } default IPersistentVector empty() { return ArrayImmutList.EMPTY; } //Long stream to account for IMutLists that are longer than Integer.MAX_VALUE. default LongStream indexStream(boolean parallel) { LongStream retval = LongStream.range(0, size()); return parallel ? retval.parallel() : retval; } default Stream objStream(boolean parallel) { return indexStream(parallel).mapToObj((long idx)->get((int)idx)); } default DoubleStream doubleStream(boolean parallel) { return indexStream(parallel).mapToDouble((long idx)->getDouble((int)idx)); } default LongStream longStream(boolean parallel) { return indexStream(parallel).map((long idx)->getLong((int)idx)); } } ================================================ FILE: java/ham_fisted/ISeqDef.java ================================================ package ham_fisted; import clojure.lang.ISeq; import clojure.lang.Sequential; import clojure.lang.IHashEq; import clojure.lang.IPersistentCollection; import clojure.lang.PersistentList; import clojure.lang.Util; import clojure.lang.Counted; import clojure.lang.RT; import clojure.lang.SeqIterator; import clojure.lang.Murmur3; import clojure.lang.Cons; import java.util.List; import java.util.Collection; import java.util.Collections; import java.util.ArrayList; import java.util.Iterator; import java.util.ListIterator; public interface ISeqDef extends ISeq, Sequential, List, IHashEq { default IPersistentCollection empty(){ return PersistentList.EMPTY; } default boolean equiv(Object obj){ if(!(obj instanceof Sequential || obj instanceof List)) return false; if(this instanceof Counted && obj instanceof Counted && ((Counted)this).count() != ((Counted)obj).count()) return false; ISeq ms = RT.seq(obj); for(ISeq s = seq(); s != null; s = s.next(), ms = ms.next()) { if(ms == null || !Util.equiv(s.first(), ms.first())) return false; } return ms == null; } default boolean seqEquals(Object obj){ if(this == obj) return true; if(!(obj instanceof Sequential || obj instanceof List)) return false; ISeq ms = RT.seq(obj); for(ISeq s = seq(); s != null; s = s.next(), ms = ms.next()) { if(ms == null || !Util.equals(s.first(), ms.first())) return false; } return ms == null; } default int calcHashCode(){ int hash = 1; for(ISeq s = seq(); s != null; s = s.next()) { hash = 31 * hash + (s.first() == null ? 0 : s.first().hashCode()); } return hash; } default int calcHasheq(){ return Murmur3.hashOrdered(this); } default int count(){ int i = 1; for(ISeq s = next(); s != null; s = s.next(), i++) if(s instanceof Counted) return i + s.count(); return i; } default ISeq seq(){ return this; } default ISeq cons(Object o){ return new Cons(o, this); } default ISeq more(){ ISeq s = next(); if(s == null) return PersistentList.EMPTY; return s; } @SuppressWarnings("unchecked") default List toArrayList() { ArrayList retval = new ArrayList(); for(ISeq s = this; s != null; s = s.next()) retval.add(s.first()); return retval; } default Object[] toArray(){ return toArrayList().toArray(); } default boolean add(Object o){ throw new UnsupportedOperationException(); } default boolean remove(Object o){ throw new UnsupportedOperationException(); } default boolean addAll(Collection c){ throw new UnsupportedOperationException(); } default void clear(){ throw new UnsupportedOperationException(); } default boolean retainAll(Collection c){ throw new UnsupportedOperationException(); } default boolean removeAll(Collection c){ throw new UnsupportedOperationException(); } default boolean containsAll(Collection c){ for(Object o : c) { if(!contains(o)) return false; } return true; } default Object[] toArray(Object[] a){ return RT.seqToPassedArray(seq(), a); } default int size(){ return count(); } default boolean isEmpty(){ return seq() == null; } default boolean contains(Object o){ for(ISeq s = seq(); s != null; s = s.next()) { if(Util.equiv(s.first(), o)) return true; } return false; } default Iterator iterator(){ return new SeqIterator(this); } //////////// List stuff ///////////////// @SuppressWarnings("unchecked") default List reify(){ return Collections.unmodifiableList(toArrayList()); } default List subList(int fromIndex, int toIndex){ return reify().subList(fromIndex, toIndex); } default Object set(int index, Object element){ throw new UnsupportedOperationException(); } default Object remove(int index){ throw new UnsupportedOperationException(); } default int indexOf(Object o){ ISeq s = seq(); for(int i = 0; s != null; s = s.next(), i++) { if(Util.equiv(s.first(), o)) return i; } return -1; } default int lastIndexOf(Object o){ return reify().lastIndexOf(o); } default ListIterator listIterator(){ return reify().listIterator(); } default ListIterator listIterator(int index){ return reify().listIterator(index); } default Object get(int index){ return RT.nth(this, index); } default void add(int index, Object element){ throw new UnsupportedOperationException(); } default boolean addAll(int index, Collection c){ throw new UnsupportedOperationException(); } } ================================================ FILE: java/ham_fisted/ISet.java ================================================ package ham_fisted; import java.util.Collection; import java.util.Set; import java.util.Iterator; import java.util.function.Function; import java.util.function.Consumer; import clojure.lang.Counted; import clojure.lang.Seqable; import clojure.lang.ISeq; import clojure.lang.RT; public interface ISet extends Set, ITypedReduce, IFnDef, Counted, Seqable { default int count() { return size(); } @SuppressWarnings("unchecked") default boolean addAll(Collection c) { int sz = size(); for(Object o: c) add(o); return sz == size(); } @SuppressWarnings("unchecked") default void forEach(Consumer c) { ITypedReduce.super.forEach(c); } default boolean retainAll(Collection c) { int sz = size(); for(Iterator iter = iterator(); iter.hasNext();) { if(!c.contains(iter.next())) iter.remove(); } return sz == size(); } default boolean containsAll(Collection c) { for(Object o: c) { if(!contains(c)) return false; } return true; } default boolean removeAll(Collection c) { int sz = size(); for(Object o: c) { remove(o); } return sz == size(); } default ISeq seq() { return LazyChunkedSeq.chunkIteratorSeq(iterator()); } default boolean isEmpty() { return size() == 0; } default Object[] toArray() { return ArrayLists.toArray(this); } default Object[] toArray(Object[] d) { return ArrayLists.toArray(this, d); } default Object invoke(Object arg) { return contains(arg) ? arg : null; } } ================================================ FILE: java/ham_fisted/ITypedReduce.java ================================================ package ham_fisted; import java.util.function.DoubleBinaryOperator; import java.util.function.LongBinaryOperator; import java.util.function.Consumer; import java.util.function.DoubleConsumer; import java.util.function.LongConsumer; import java.util.function.IntConsumer; import java.util.concurrent.ForkJoinPool; import clojure.lang.IReduceInit; import clojure.lang.IReduce; import clojure.lang.IFn; /** * Typed reductions - a typed extension of clojure.lang.IReduceInit and * java.util.Iterable.forEach. */ public interface ITypedReduce extends IReduceInit, IReduce { default Object parallelReduction(IFn initValFn, IFn rfn, IFn mergeFn, ParallelOptions options) { return Reductions.serialParallelReduction(initValFn, rfn, options, this); } static class Reduce1 implements IFnDef { public boolean first; public IFn rfn; public Reduce1(IFn rf) { this.first = true; this.rfn = rf; } public Object invoke(Object acc, Object v) { if(first) { first = false; return v; } else { return rfn.invoke(acc,v); } } } default Object reduce(IFn rfn) { Reduce1 reducer = new Reduce1(rfn); Object rv = this.reduce(reducer, null); if(reducer.first) return rfn.invoke(); else return rv; } //Typed this way in order to match java.util.List's forEach @SuppressWarnings("unchecked") default void forEach(Consumer c) { //Reduce is strictly more powerful. You can define consume in terms of reduce with //full generality but you cannot define reduce in terms of consume. reduce( new IFnDef() { public Object invoke(Object lhs, Object rhs) { c.accept((E)rhs); return c; } }, c); } } ================================================ FILE: java/ham_fisted/ImmutList.java ================================================ package ham_fisted; import static ham_fisted.ChunkedList.*; import java.util.List; import java.util.Objects; import java.util.RandomAccess; import java.util.Iterator; import java.util.ListIterator; import java.util.Collection; import java.util.Iterator; import java.util.Arrays; import java.util.ArrayList; import java.util.function.Function; import java.util.function.BiFunction; import clojure.lang.Indexed; import clojure.lang.RT; import clojure.lang.IReduce; import clojure.lang.IKVReduce; import clojure.lang.IFn; import clojure.lang.IHashEq; import clojure.lang.Seqable; import clojure.lang.Reversible; import clojure.lang.ISeq; import clojure.lang.IPersistentVector; import clojure.lang.IPersistentMap; import clojure.lang.IObj; import clojure.lang.MapEntry; import clojure.lang.IMapEntry; import clojure.lang.Util; import clojure.lang.IEditableCollection; import clojure.lang.ITransientVector; import clojure.lang.IFn; import clojure.lang.APersistentVector; public class ImmutList extends APersistentVector implements IMutList, IHashEq, ChunkedListOwner, IPersistentVector, IEditableCollection, UpdateValues { public final int startidx; public final int nElems; final ChunkedList data; int _hash = 0; public static final ImmutList EMPTY = new ImmutList(0, 0, new ChunkedList()); ImmutList(int sidx, int eidx, ChunkedList d) { startidx = sidx; nElems = eidx - sidx; data = d; } public static IPersistentVector create(boolean owning, IPersistentMap meta, Object... data) { if (data.length <= 32) return ArrayImmutList.create(owning, meta, data); return new ImmutList(0, data.length, ChunkedList.create(owning, meta, data)); } final int indexCheck(int idx) { return ChunkedList.indexCheck(startidx, nElems, idx); } final int wrapIndexCheck(int idx) { return ChunkedList.wrapIndexCheck(startidx, nElems, idx); } public ChunkedListSection getChunkedList() { return new ChunkedListSection(data.data, startidx, startidx+nElems); } public final int hashCode() { if (_hash == 0) { _hash = data.hasheq(startidx, startidx+nElems); } return _hash; } public final int hasheq() { return hashCode(); } public final boolean equals(Object other) { if (other == this ) return true; return equiv(other); } public final String toString() { return Transformables.sequenceToString(this); } public final boolean equiv(Object other) { //return CljHash.listEquiv(this, other); return data.equiv(startidx, startidx + nElems, other); } public final int size() { return nElems; } public final int count() { return nElems; } public final int length() { return nElems; } public final void clear() { throw new RuntimeException("Unimplemented"); } public final boolean add(Object e) { throw new RuntimeException("Unimplemented"); } public final void add(int idx, Object e) { throw new RuntimeException("Unimplemented"); } public final boolean addAll(Collection e) { throw new RuntimeException("Unimplemented"); } public final boolean addAll(int idx, Collection e) { throw new RuntimeException("Unimplemented"); } public final Object remove(int idx) { throw new RuntimeException("Unimplemented"); } public final boolean remove(Object o) { throw new RuntimeException("Unimplemented"); } public final boolean removeAll(Collection c) { throw new RuntimeException("Unimplemented"); } public final boolean retainAll(Collection c) { throw new RuntimeException("Unimplemented"); } public final boolean isEmpty() { return nElems == 0; } @SuppressWarnings("unchecked") public final Object set(int idx, Object e) { throw new RuntimeException("Unimplemented"); } @SuppressWarnings("unchecked") public final Object get(int idx) { return data.getValue(indexCheck(idx)); } public final int indexOf(Object obj) { return data.indexOf(startidx, startidx+nElems, obj); } public final int lastIndexOf(Object obj) { return data.lastIndexOf(startidx, startidx+nElems, obj); } public final boolean contains(Object obj) { return data.contains(startidx, startidx+nElems, obj); } public final boolean containsAll(Collection c) { return data.containsAll(startidx, startidx+nElems, c); } @SuppressWarnings("unchecked") public final Iterator iterator() { return data.iterator(startidx, startidx + nElems); } public final IMutList subList(int sidx, int eidx) { ChunkedList.sublistCheck(sidx, eidx, nElems); return new ImmutList(sidx+startidx, eidx+startidx, data); } public final Object[] toArray() { return data.toArray(startidx, startidx+nElems); } public final Object[] toArray(Object[] marker) { return data.fillArray(startidx, startidx+nElems, Arrays.copyOf(marker, nElems)); } public IPersistentMap meta() { return data.meta(); } public ImmutList withMeta(IPersistentMap m) { return new ImmutList(startidx, startidx+nElems, data.withMeta(m)); } public final Object nth(int idx) { return data.getValue(wrapIndexCheck(idx)); } public final Object nth(int idx, Object notFound) { if (idx < 0) idx = idx + nElems; return data.getValue(indexCheck(idx)); } public final Object invoke(Object idx) { return nth(RT.intCast(idx)); } public final Object invoke(Object idx, Object notFound) { return nth(RT.intCast(idx), notFound); } public final Object reduce(IFn f) { return data.reduce(startidx, startidx+nElems, f); } public final Object reduce(IFn f, Object init) { return data.reduce(startidx, startidx+nElems, f, init); } public final Object kvreduce(IFn f, Object init) { return data.kvreduce(startidx, startidx+nElems, f, init); } public final ISeq seq() { return data.seq(startidx, startidx+nElems); } public final ISeq rseq() { return data.rseq(startidx, startidx+nElems); } public final ImmutList cons(Object obj) { return new ImmutList(0, nElems+1, data.conj(startidx, startidx+nElems, obj)); } public final ImmutList assocN(int idx, Object obj) { if(idx == nElems) return cons(obj); indexCheck(idx); return new ImmutList(0, nElems, data.assoc(startidx, startidx+nElems, idx, obj)); } public final ImmutList assoc(Object idx, Object obj) { return assocN(RT.intCast(idx), obj); } public IMapEntry entryAt(Object key) { return IMutList.super.entryAt(key); } public final boolean containsKey(Object key) { if (Util.isInteger(key)) { final int k = RT.intCast(key); return k >= 0 && k < nElems; } return false; } public final ImmutList empty() { return EMPTY.withMeta(meta()); } public final Object valAt(Object obj, Object notFound) { if(Util.isInteger(obj)) { int k = RT.intCast(obj); if (k >= 0 && k < nElems) return data.getValue(k + startidx); } return notFound; } public final Object valAt(Object obj) { return valAt(obj, null); } public final ImmutList pop() { if (nElems == 0) throw new RuntimeException("Can't pop empty vector"); if (nElems == 1) return EMPTY.withMeta(meta()); return new ImmutList(0, nElems-1, data.pop(startidx, startidx+nElems)); } public final Object peek() { if (nElems == 0) return null; return get(nElems-1); } @SuppressWarnings("unchecked") public ImmutList updateValues(BiFunction valueMap) { ChunkedList retval = data.clone(startidx, startidx+nElems, 0, true); final Object[][] rd = retval.data; final int ne = nElems; int idx = 0; while (idx < ne) { final Object[] chunk = rd[idx/32]; final int csize = Math.min(ne-idx, chunk.length); for (int eidx = 0; eidx < csize; ++eidx, ++idx) { chunk[eidx] = valueMap.apply(idx, chunk[eidx]); } } return new ImmutList(0, ne, retval); } @SuppressWarnings("unchecked") public ImmutList updateValue(Object key, Function valueMap) { if(!Util.isInteger(key)) throw new RuntimeException("Vector indexes must be integers: " + String.valueOf(key)); int idx = RT.intCast(key); if (idx == nElems) return cons(valueMap.apply(null)); indexCheck(idx); ChunkedList retval = data.clone(startidx, startidx+nElems, 0, false); final Object[][] mdata = retval.data; final int cidx = idx / 32; final int eidx = idx % 32; Object[] chunk = mdata[cidx].clone(); mdata[cidx] = chunk; chunk[eidx] = valueMap.apply(chunk[eidx]); return new ImmutList(0, nElems, retval); } public final ITransientVector asTransient() { if(nElems == 0) return new MutList(); //We know the cloning operation deep copies if startidx isn't a multiple of 32. final boolean ownsEverything = (startidx % 32) != 0; return new TransientList(data.clone(startidx, startidx + nElems, 0, false), nElems, ownsEverything); } } ================================================ FILE: java/ham_fisted/ImmutSort.java ================================================ package ham_fisted; import java.util.List; import java.util.Arrays; import java.util.Comparator; import it.unimi.dsi.fastutil.objects.ObjectArrays; import clojure.lang.RT; import clojure.lang.Util; public interface ImmutSort extends List { @SuppressWarnings("unchecked") public static final Comparator defaultComparator = new Comparator() { public int compare(Object l, Object r) { if(l == r) return 0; if(l == null) return -1; if(r == null) return 1; Class cls = l.getClass(); if(Comparable.class.isAssignableFrom(cls) && cls == r.getClass()) return ((Comparable)l).compareTo(r); return Util.compare(l,r); } }; @SuppressWarnings("unchecked") public static List immutSortList(List a, Comparator c) { if(c == null) c = defaultComparator; final Object[] data = a.toArray(); if (c != null) Arrays.parallelSort(data, 0, data.length, (Comparator)c); else Arrays.parallelSort(data, 0, data.length, c); return ArrayLists.toList(data, 0, data.length, RT.meta(a)); } default List immutSort() { return immutSort(null); } default List immutSort(Comparator c) { return immutSortList(this, c); } } ================================================ FILE: java/ham_fisted/IndexedConsumer.java ================================================ package ham_fisted; public interface IndexedConsumer { public void accept(long index, Object val); } ================================================ FILE: java/ham_fisted/IndexedDoubleConsumer.java ================================================ package ham_fisted; public interface IndexedDoubleConsumer { public void accept(long idx, double val); } ================================================ FILE: java/ham_fisted/IndexedLongConsumer.java ================================================ package ham_fisted; public interface IndexedLongConsumer { public void accept(long idx, long val); } ================================================ FILE: java/ham_fisted/IntegerOps.java ================================================ package ham_fisted; /** * Static 32 bit and 64 bit integer operations specific to the bitmap trie. */ public final class IntegerOps { /** Return the next power of 2 >= value.*/ public static final int nextPow2(int value) { int highestOneBit = Integer.highestOneBit(value); if (value == highestOneBit) { return value; } return highestOneBit << 1; } /** * 1. unsigned shifted hash right by shift bits. * 2. return last 5 bits. * * @return A number from 0-31 inclusive. */ public static final int mask(int shift, int hash) { return (hash >>> shift) & 0x01f; } /** * @return An integer with 1 bit high that indicates which value * from 0-31 inclusive is indicate by the shifted and masked hash. */ public static final int bitpos(int shift, int hash) { return 1 << mask(shift, hash); } /** * @return The number of raised bits in bitmap behind * position bit. This indicates the index in the node's packed * array where data for this bit should be stored. */ public static final int index(int bitmap, int bit){ return Integer.bitCount(bitmap & (bit - 1)); } /** * @return unchecked shift incremented by 5 to indicate * the next 5 bits of the hashcode should be used for the bitmap. */ public static final int incShift(int shift) { return shift + 5; } /** * @return unchecked shift incremented by 5 to indicate * the next 5 bits of the hashcode should be used for the bitmap. */ public static final int checkedIncShift(int shift) { if (shift >= 30) throw new RuntimeException("Invalid shift amount - already at max shift - " + String.valueOf(shift)); return shift + 5; } /** * Ripped directly from HashMap.java in openjdk source code - * * Computes key.hashCode() and spreads (XORs) higher bits of hash * to lower. Because the table uses power-of-two masking, sets of * hashes that vary only in bits above the current mask will * always collide. (Among known examples are sets of Float keys * holding consecutive whole numbers in small tables.) So we * apply a transform that spreads the impact of higher bits * downward. There is a tradeoff between speed, utility, and * quality of bit-spreading. Because many common sets of hashes * are already reasonably distributed (so don't benefit from * spreading), and because we use trees to handle large sets of * collisions in bins, we just XOR some shifted bits in the * cheapest possible way to reduce systematic lossage, as well as * to incorporate impact of the highest bits that would otherwise * never be used in index calculations because of table bounds. */ public static final int mixhash(int h) { return h ^ (h >>> 16); } public static final int mixhash(Object key) { return (key == null) ? 0 : mixhash(key.hashCode()); } /** * Set bits starting at sidx and ending at *but including* eidx. */ public static final int highBits(int sidx, int eidx) { int retval = 0; for (int idx = sidx; idx <= eidx; ++idx) retval |= 1 << idx; return retval; } /** Untested long versions of some of the above public static final long mask(long hash, long shift) { // Return the last 10 bits of hash right shifted shift bits return (hash >>> shift) & 0x03F; } public static final long bitpos(long hash, long shift) { return 1 << mask(hash, shift); } public static final long index(long bitmap, long bit){ return Long.bitCount(bitmap & (bit - 1)); } public static final long incShift(long shift) { return shift + 6; } */ } ================================================ FILE: java/ham_fisted/Iter.java ================================================ package ham_fisted; import java.util.Iterator; public interface Iter { Object get(); Iter next(); public static class IteratorIter implements Iter { public final Iterator iter; Object get; public IteratorIter(Iterator iter) { this.iter = iter; get = iter.next(); } public Object get() { return get; } public Iter next() { if(iter.hasNext()) { get = iter.next(); return this; } return null; } } public static class IterIterator implements Iterator { Iter iter; public IterIterator(Iter iter) { this.iter = iter; } public Iter iter() { return iter; } public boolean hasNext() { return iter != null; } public Object next() { Object rv = iter.get(); iter = iter.next(); return rv; } } public static Iter fromIterator(Iterator iter) { return iter != null && iter.hasNext() ? new IteratorIter(iter) : null; } public static Iterator fromIter(Iter iter) { return new IterIterator(iter); } public static Iter fromIterable(Iterable iable) { return iable != null ? fromIterator(iable.iterator()) : null; } public static Iter prepend(Object obj, Iter item) { return new Iter() { public Object get() { return obj; } public Iter next() { return item; } }; } } ================================================ FILE: java/ham_fisted/LazyChunkedSeq.java ================================================ package ham_fisted; import clojure.lang.IFn; import clojure.lang.IChunkedSeq; import clojure.lang.IChunk; import clojure.lang.ASeq; import clojure.lang.Util; import clojure.lang.PersistentList; import clojure.lang.ISeq; import clojure.lang.IPersistentMap; import clojure.lang.ChunkedCons; import clojure.lang.ArrayChunk; import java.util.Iterator; import java.util.Arrays; import java.util.concurrent.locks.Lock; import java.util.concurrent.locks.ReentrantLock; public class LazyChunkedSeq extends ASeq implements IChunkedSeq { IFn fn; IChunkedSeq mySeq; Throwable e; volatile Lock lock; public LazyChunkedSeq(IFn fn, IChunkedSeq s, Throwable e, IPersistentMap meta) { super(meta); this.fn = fn; this.mySeq = s; this.e = e; this.lock = new ReentrantLock(); } public LazyChunkedSeq(IFn fn) { this(fn, null, null, null); } public LazyChunkedSeq withMeta(IPersistentMap m) { return new LazyChunkedSeq(fn, mySeq, e, m); } IChunkedSeq unlockedUnwrap() { if(fn != null) { try { mySeq = (IChunkedSeq)fn.invoke(); } catch (Exception e) { this.e = e; } fn = null; lock = null; } if(this.e != null) throw Util.sneakyThrow(e); return mySeq; } IChunkedSeq lockedUnwrap() { if(mySeq != null) return mySeq; Lock l = lock; if(l != null) { l.lock(); try { return unlockedUnwrap(); }finally { l.unlock(); } } return unlockedUnwrap(); } public Object first() { IChunkedSeq s = lockedUnwrap(); return s != null ? s.first() : null; } public ISeq next() { IChunkedSeq s = lockedUnwrap(); return s != null ? s.next() : null; } public ISeq more() { ISeq rv = next(); return rv != null ? rv : PersistentList.EMPTY; } public IChunk chunkedFirst() { IChunkedSeq s = lockedUnwrap(); return s != null ? s.chunkedFirst() : null; } public ISeq chunkedNext() { IChunkedSeq s = lockedUnwrap(); return s != null ? s.chunkedNext() : null; } public ISeq chunkedMore() { ISeq rv = chunkedNext(); return rv != null ? rv : PersistentList.EMPTY; } public static IChunkedSeq chunkIteratorSeq(Iterator i) { if(i.hasNext()) { return new LazyChunkedSeq(new IFnDef() { public Object invoke() { Object[] ar = new Object[32]; int idx; for(idx = 0; idx < 32 && i.hasNext(); ++idx) ar[idx] = i.next(); return new ChunkedCons(new ArrayChunk(ar, 0, idx), chunkIteratorSeq(i)); } }); } else { return null; } } } ================================================ FILE: java/ham_fisted/LinkedHashMap.java ================================================ package ham_fisted; import java.util.Map; import java.util.Set; import java.util.Iterator; import java.util.NoSuchElementException; import java.util.function.Function; import java.util.function.BiFunction; import java.util.function.BiConsumer; import clojure.lang.IFn; import clojure.lang.RT; import clojure.lang.IPersistentMap; import clojure.lang.IDeref; import clojure.lang.IReduceInit; public class LinkedHashMap extends HashMap { //Most recently modified LinkedHashNode firstLink; //Least recently modified LinkedHashNode lastLink; public LinkedHashMap(IPersistentMap meta) { super(meta); } public LinkedHashMap() { this(null); } public LinkedHashMap(float loadFactor, int initialCapacity, int length, HashNode[] data, IPersistentMap meta) { super(loadFactor, initialCapacity, length, data, meta); } public LinkedHashMap clone() { LinkedHashMap rv = new LinkedHashMap(loadFactor, capacity, 0, new HashNode[data.length], meta); final HashNode[] data = this.data; final int mask = this.mask; final HashNode[] newData = rv.data; //Table is already correct size - no need to check resize. for(LinkedHashNode lf = lastLink; lf != null; lf = lf.nextLink) { int idx = lf.hashcode & mask; HashNode loc = newData[idx]; HashNode newNode = rv.newNode(lf.k, lf.hashcode, lf.v); newData[idx] = newNode; newNode.nextNode = loc; } return rv; } protected HashNode newNode(Object key, int hc, Object val) { return new LinkedHashNode(this,key,hc,val,null); } protected void inc(HashNode lf) { super.inc(lf); LinkedHashNode hn = (LinkedHashNode)lf; if(lastLink == null) lastLink = hn; hn.prevLink = firstLink; if(firstLink != null) firstLink.nextLink = hn; firstLink = hn; } protected static void removeLink(LinkedHashNode hn) { if (hn.prevLink != null) hn.prevLink.nextLink = hn.nextLink; if(hn.nextLink != null) hn.nextLink.prevLink = hn.prevLink; } protected void dec(HashNode lf) { super.dec(lf); LinkedHashNode hn = (LinkedHashNode)lf; if(hn == firstLink) firstLink = hn.prevLink; if(hn == lastLink) lastLink = hn.nextLink; removeLink(hn); hn.nextLink = hn.prevLink = null; } protected void modify(HashNode n) { // The algorithm below is lightly tested but currently linkedhashmaps only // record insertion and deletion events. // LinkedHashNode hn = (LinkedHashNode)n; // if(firstLink != hn) { // removeLink(hn); // if(hn == lastLink) // lastLink = hn.nextLink; // hn.prevLink = firstLink; // hn.nextLink = null; // if(firstLink != null) // firstLink.nextLink = hn; // hn.prevLink = firstLink; // firstLink = hn; // } } public static class LinkedIter implements Iterator { LinkedHashNode current; Function fn; public LinkedIter(Function fn, LinkedHashNode c) { this.current = c; this.fn = fn; } public boolean hasNext() { return current != null; } public Object next() { if(current == null) throw new NoSuchElementException(); Object rv = fn.apply(current); current = current.nextLink; return rv; } } public Iterator iterator(Function fn) { return new LinkedIter(fn, lastLink); } public Object reduce(IFn rfn, Object acc) { for(LinkedHashNode hn = lastLink; hn != null; hn = hn.nextLink) { acc = rfn.invoke(acc, hn); if(RT.isReduced(acc)) return ((IDeref)acc).deref(); } return acc; } public Object kvreduce(IFn rfn, Object acc) { for(LinkedHashNode hn = lastLink; hn != null; hn = hn.nextLink) { acc = rfn.invoke(acc, hn.k, hn.v); if(RT.isReduced(acc)) return ((IDeref)acc).deref(); } return acc; } public HashMap union(Map o, BiFunction bfn) { // reduce union preserves iteration order over o. if(o instanceof LinkedHashMap) { return reduceUnion(this, (IReduceInit)o, bfn); } else { return super.union(o, bfn); } } } ================================================ FILE: java/ham_fisted/LinkedHashNode.java ================================================ package ham_fisted; public class LinkedHashNode extends HashNode { //Less recently modified LinkedHashNode prevLink; //More recently modified LinkedHashNode nextLink; public LinkedHashNode(LinkedHashMap owner, Object _k, int hc, Object _v, LinkedHashNode nn) { super(owner, _k, hc, _v, nn); } public HashNode clone(HashMap nowner) { throw new UnsupportedOperationException("LinkedHashNodes cannot clone"); } public HashNode setOwner(HashMap nowner) { if(nowner != owner) throw new RuntimeException("LinkedHashMap nodes cannot be structurally shared"); return this; } //Linked node assoc/dissoc are not functional like their counterparts - //they just keep the same signature for use in the set algorithms. public final HashNode assoc(HashMap nowner, Object _k, int hash, Object _v) { if(nowner != owner) throw new RuntimeException("LinkedHashMap assoc called in functional pathway"); HashNode retval = this; if (owner.equals(_k,k)) { retval.setValue(_v); } else { if (retval.nextNode != null) { retval.nextNode = retval.nextNode.assoc(nowner, _k, hash, _v); } else { retval.nextNode = nowner.newNode(k, hash, _v); } } return retval; } public final HashNode dissoc(HashMap nowner, Object _k) { if(nowner != owner) throw new RuntimeException("LinkedHashMap assoc called in functional pathway"); if (owner.equals(k, _k)) { owner.dec(this); return nextNode; } if (nextNode != null) { nextNode = nextNode.dissoc(nowner, _k); } return this; } } ================================================ FILE: java/ham_fisted/LongAccum.java ================================================ package ham_fisted; import java.util.function.LongConsumer; import clojure.lang.IDeref; public class LongAccum implements LongConsumer, IDeref, Reducible { long val; public LongAccum(long v) { val = v; } public LongAccum() { this(0); } public void accept(long v) { val += v; } public Object deref() { return val; } public LongAccum reduce(Reducible other) { val += ((LongAccum)other).val; return this; } } ================================================ FILE: java/ham_fisted/LongHashBase.java ================================================ package ham_fisted; import java.util.Arrays; import java.util.Iterator; import java.util.Spliterator; import java.util.function.Function; import java.util.function.Consumer; import java.util.Map; import clojure.lang.IPersistentMap; import clojure.lang.IHashEq; import clojure.lang.IMeta; import clojure.lang.IFn; import clojure.lang.IDeref; import clojure.lang.RT; public class LongHashBase implements IMeta { int capacity; int mask; int length; int threshold; float loadFactor; LongHashNode[] data; IPersistentMap meta; public LongHashBase(float loadFactor, int initialCapacity, int length, LongHashNode[] data, IPersistentMap meta) { this.loadFactor = loadFactor; this.capacity = IntegerOps.nextPow2(Math.max(4, initialCapacity)); this.mask = this.capacity - 1; this.length = length; this.data = data == null ? new LongHashNode[this.capacity] : data; this.threshold = (int)(capacity * loadFactor); this.meta = meta; } public LongHashBase(LongHashBase other, IPersistentMap m) { this.loadFactor = other.loadFactor; this.capacity = other.capacity; this.mask = other.mask; this.length = other.length; this.data = other.data; this.threshold = other.threshold; this.meta = m; } public int size() { return length; } public int count() { return length; } //protected so clients can override as desired. static int hash(long k) { return IntegerOps.mixhash(Long.hashCode(k)); } static boolean equals(long lhs, long rhs) { return lhs == rhs; } protected void inc(LongHashNode lf) { ++this.length; } protected void dec(LongHashNode lf) { --this.length; } protected void modify(LongHashNode lf) {} protected LongHashNode newNode(long key, int hc, Object val) { return new LongHashNode(this,key,hc,val,null); } Object checkResize(Object rv) { if(this.length >= this.threshold) { final int newCap = this.capacity * 2; final LongHashNode[] newD = new LongHashNode[newCap]; final LongHashNode[] oldD = this.data; final int oldCap = oldD.length; final int mask = newCap - 1; for(int idx = 0; idx < oldCap; ++idx) { LongHashNode lf; if((lf = oldD[idx]) != null) { oldD[idx] = null; if(lf.nextNode == null) { newD[lf.hashcode & mask] = lf; } else { //https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/util/HashMap.java#L722 //Because we only allow capacities that are powers of two, we have //exactly 2 locations in the new data array where these can go. We want //to avoid writing to any locations more than once and instead make the //at most two new linked lists, one for the new high position and one //for the new low position. LongHashNode loHead = null, loTail = null, hiHead = null, hiTail = null; while(lf != null) { LongHashNode e = lf.setOwner(this); lf = lf.nextNode; //Check high bit if((e.hashcode & oldCap) == 0) { if(loTail == null) loHead = e; else loTail.nextNode = e; loTail = e; } else { if(hiTail == null) hiHead = e; else hiTail.nextNode = e; hiTail = e; } } if(loHead != null) { loTail.nextNode = null; newD[idx] = loHead; } if(hiHead != null) { hiTail.nextNode = null; newD[idx+oldCap] = hiHead; } } } } this.capacity = newCap; this.threshold = (int)(newCap * this.loadFactor); this.mask = mask; this.data = newD; } return rv; } public void clear() { for(int idx = 0; idx < data.length; ++idx) { for(LongHashNode lf = data[idx]; lf != null; lf = lf.nextNode) { dec(lf); } } length = 0; Arrays.fill(data, null); } public IPersistentMap meta() { return meta; } static class HTIter implements Iterator { final LongHashNode[] d; final Function fn; LongHashNode l; int idx; final int dlen; HTIter(LongHashNode[] data, Function fn) { this.d = data; this.fn = fn; this.l = null; this.idx = 0; this.dlen = d.length; advance(); } void advance() { if(l != null) l = l.nextNode; if(l == null) { for(; idx < this.dlen && l == null; ++idx) l = this.d[idx]; } } public boolean hasNext() { return l != null; } public Object next() { LongHashNode rv = l; advance(); return fn.apply(rv); } } static class HTSpliterator implements Spliterator, ITypedReduce { final LongHashNode[] d; final Function fn; int sidx; int eidx; int estimateSize; LongHashNode l; public HTSpliterator(LongHashNode[] d, int len, Function fn) { this.d = d; this.fn = fn; this.sidx = 0; this.eidx = d.length; this.estimateSize = len; this.l = null; } public HTSpliterator(LongHashNode[] d, int sidx, int eidx, int es, Function fn) { this.d = d; this.fn = fn; this.sidx = sidx; this.eidx = eidx; this.estimateSize = es; this.l = null; } public HTSpliterator trySplit() { final int nIdxs = this.eidx - this.sidx; if(nIdxs > 4) { final int idxLen = nIdxs/2; final int oldIdx = this.eidx; this.eidx = this.sidx + idxLen; this.estimateSize = this.estimateSize / 2; return new HTSpliterator(d, this.eidx, oldIdx, this.estimateSize, this.fn); } return null; } public int characteristics() { return Spliterator.DISTINCT | Spliterator.IMMUTABLE | Spliterator.SIZED; } public long estimateSize() { return estimateSize; } public long getExactSizeIfKnown() { return estimateSize(); } @SuppressWarnings("unchecked") public boolean tryAdvance(Consumer c) { if(this.l != null) { c.accept(this.fn.apply(this.l)); this.l = this.l.nextNode; return true; } for(; sidx < eidx; ++sidx) { final LongHashNode ll = this.d[sidx]; if(ll != null) { c.accept(this.fn.apply(ll)); this.l = ll.nextNode; return true; } } return false; } public Object reduce(IFn rfn, Object acc) { final LongHashNode[] dd = this.d; final int ee = this.eidx; final Function ffn = this.fn; for(int idx = sidx; idx < ee; ++idx) { for(LongHashNode e = dd[idx]; e != null; e = e.nextNode) { acc = rfn.invoke(acc, ffn.apply(e)); if(RT.isReduced(acc)) return ((IDeref)acc).deref(); } } return acc; } } final boolean containsNodeKey(Object kk) { long key = Casts.longCast(kk); for(LongHashNode e = this.data[hash(key) & this.mask]; e != null; e = e.nextNode) { if(e.k == key) return true; } return false; } } ================================================ FILE: java/ham_fisted/LongHashMap.java ================================================ package ham_fisted; import java.util.Map; import java.util.Arrays; import java.util.Iterator; import java.util.function.Function; import java.util.function.BiFunction; import java.util.function.Consumer; import java.util.function.BiConsumer; import java.util.Spliterator; import java.util.Objects; import java.util.Set; import java.util.AbstractSet; import java.util.AbstractCollection; import java.util.Collection; import clojure.lang.IPersistentMap; import clojure.lang.IFn; import clojure.lang.RT; import clojure.lang.IHashEq; import clojure.lang.IDeref; import clojure.lang.RT; import clojure.lang.IDeref; import clojure.lang.IMeta; import clojure.lang.IMapEntry; import clojure.lang.MapEntry; public class LongHashMap extends LongHashBase implements IMap, MapSetOps, UpdateValues { Set keySet = null; public LongHashMap(float loadFactor, int initialCapacity, int length, LongHashNode[] data, IPersistentMap meta) { super(loadFactor, initialCapacity, length, data, meta); } public LongHashMap() { this(0.75f, 0, 0, null, null); } public LongHashMap(IPersistentMap m) { this(0.75f, 0, 0, null, m); } public LongHashMap(LongHashMap other, IPersistentMap m) { super(other, m); } public LongHashMap shallowClone() { return new LongHashMap(loadFactor, capacity, length, data.clone(), meta); } public LongHashMap clone() { final int l = data.length; LongHashNode[] newData = new LongHashNode[l]; LongHashMap retval = new LongHashMap(loadFactor, capacity, length, newData, meta); for(int idx = 0; idx < l; ++idx) { LongHashNode orig = data[idx]; if(orig != null) newData[idx] = orig.clone(retval); } return retval; } public int hashCode() { return hasheq(); } public int hasheq() { return CljHash.mapHashcode(this); } public boolean equals(Object o) { return equiv(o); } public boolean equiv(Object o) { return CljHash.mapEquiv(this, o); } public int size() { return this.length; } public boolean isEmpty() { return this.length == 0; } public String toString() { final StringBuilder b = (StringBuilder) reduce(new IFnDef() { public Object invoke(Object acc, Object v) { final StringBuilder b = (StringBuilder)acc; final Map.Entry lf = (Map.Entry)v; if(b.length() > 2) b.append(","); return b.append(lf.getKey()) .append(" ") .append(lf.getValue()); } }, new StringBuilder().append("{")); return b.append("}").toString(); } public Object put(Object kk, Object val) { long key = Casts.longCast(kk); final int hc = hash(key); final int idx = hc & this.mask; LongHashNode lastNode = null; //Avoid unneeded calls to both equals and checkResize for(LongHashNode e = this.data[idx]; e != null; e = e.nextNode) { lastNode = e; if(e.k == key || equals(e.k, key)) { Object rv = e.v; e.v = val; modify(e); return rv; } } LongHashNode lf = newNode(key,hc,val); if(lastNode != null) { lastNode.nextNode = lf; } else { data[idx] = lf; } return checkResize(null); } public void putAll(Map other) { LongHashNode[] d = data; int mask = this.mask; for(Object o: other.entrySet()) { Map.Entry ee = (Map.Entry)o; long k = Casts.longCast(ee.getKey()); int hashcode = hash(k); int idx = hashcode & mask; LongHashNode e; for(e = d[idx]; e != null && !(k == e.k); e = e.nextNode); if(e != null) { e.v = ee.getValue(); } else { LongHashNode n = newNode(k, hashcode, ee.getValue()); n.nextNode = d[idx]; d[idx] = n; checkResize(null); d = data; mask = this.mask; } } } public Object getOrDefault(Object kk, Object dv) { if(kk instanceof Number) { long key = Casts.longCast(kk); for(LongHashNode e = this.data[hash(key) & this.mask]; e != null; e = e.nextNode) { if(e.k == key) return e.v; } } return dv; } public Object get(Object kk) { if(kk instanceof Number) { long key = Casts.longCast(kk); for(LongHashNode e = this.data[hash(key) & this.mask]; e != null; e = e.nextNode) { if(e.k == key) return e.v; } } return null; } public IMapEntry entryAt(Object kk) { if(kk instanceof Number) { long key = Casts.longCast(kk); for(LongHashNode e = this.data[hash(key) & this.mask]; e != null; e = e.nextNode) { if(e.k == key) return MapEntry.create(e.k, e.v); } } return null; } public boolean containsKey(Object key) { return containsNodeKey(key); } @SuppressWarnings("unchecked") public Object compute(Object kk, BiFunction bfn) { long k = Casts.longCast(kk); final int hash = hash(k); final LongHashNode[] d = this.data; final int idx = hash & this.mask; LongHashNode e = d[idx], ee = null; for(; e != null && !(e.k == k || equals(e.k, k)); e = e.nextNode) { ee = e; } Object newV = bfn.apply(k, e == null ? null : e.v); if(e != null) { if(newV != null) { e.v = newV; modify(e); } else remove(k, null); } else if(newV != null) { LongHashNode nn = newNode(k, hash, newV); if(ee != null) ee.nextNode = nn; else d[idx] = nn; checkResize(null); } return newV; } @SuppressWarnings("unchecked") public Object computeIfAbsent(Object kk, Function afn) { long k = Casts.longCast(kk); final int hash = hash(k); final LongHashNode[] d = this.data; final int idx = hash & this.mask; LongHashNode e = d[idx], ee = null; for(; e != null && !(e.k == k || equals(e.k, k)); e = e.nextNode) { ee = e; } if(e != null) { return e.v; } else { final Object newv = afn.apply(k); if(newv != null) { LongHashNode nn = newNode(k, hash, newv); if(ee != null) ee.nextNode = nn; else d[idx] = nn; checkResize(null); } return newv; } } public Object remove(Object kk) { long key = Casts.longCast(kk); int loc = hash(key) & this.mask; LongHashNode lastNode = null; for(LongHashNode e = this.data[loc]; e != null; e = e.nextNode) { if(e.k == key) { dec(e); if(lastNode != null) lastNode.nextNode = e.nextNode; else this.data[loc] = e.nextNode; return e.getValue(); } lastNode = e; } return null; } public Object reduce(IFn rfn, Object acc) { final int l = data.length; LongHashNode[] d = data; for(int idx = 0; idx < l; ++idx) { for(LongHashNode e = d[idx]; e != null; e = e.nextNode) { acc = rfn.invoke(acc, e); if(RT.isReduced(acc)) return ((IDeref)acc).deref(); } } return acc; } public Object kvreduce(IFn rfn, Object acc) { final int l = data.length; LongHashNode[] d = data; for(int idx = 0; idx < l; ++idx) { for(LongHashNode e = d[idx]; e != null; e = e.nextNode) { acc = rfn.invoke(acc, e.k, e.v); if(RT.isReduced(acc)) return ((IDeref)acc).deref(); } } return acc; } public Object parallelReduction(IFn initValFn, IFn rfn, IFn mergeFn, ParallelOptions options ) { return Reductions.parallelCollectionReduction(initValFn, rfn, mergeFn, this.entrySet(), options); } @SuppressWarnings("unchecked") public void replaceAll(BiFunction bfn) { final int l = data.length; for(int idx = 0; idx < l; ++idx) { LongHashNode lastNode = null; for(LongHashNode e = this.data[idx]; e != null; e = e.nextNode) { Object newv = bfn.apply(e.k, e.v); if(newv != null) { e.v = newv; lastNode = e; } else { dec(e); if(lastNode != null) { lastNode.nextNode = e.nextNode; } else { data[idx] = e.nextNode; } } } } } public Set keySet() { if(this.keySet == null ) this.keySet = IMap.super.keySet(); return this.keySet; } @SuppressWarnings("unchecked") public static LongHashMap union(LongHashMap rv, Map o, BiFunction bfn) { LongHashNode[] rvd = rv.data; int mask = rv.mask; for(Object ee : o.entrySet()) { Map.Entry lf = (Map.Entry)ee; final long k = Casts.longCast(lf.getKey()); final int hashcode = rv.hash(k); final int rvidx = hashcode & mask; LongHashNode init = rvd[rvidx], e = init; for(;e != null && !(e.k==k || rv.equals(e.k, k)); e = e.nextNode); if(e != null) { rvd[rvidx] = init.assoc(rv, k, hashcode, bfn.apply(e.v, lf.getValue())); } else { if(init != null) rvd[rvidx] = init.assoc(rv, k, hashcode, lf.getValue()); else rvd[rvidx] = rv.newNode(k, hashcode, lf.getValue()); rv.checkResize(null); mask = rv.mask; rvd = rv.data; } } return rv; } @SuppressWarnings("unchecked") public LongHashMap union(Map o, BiFunction bfn) { return union(this, o, bfn); } @SuppressWarnings("unchecked") static LongHashMap intersection(LongHashMap rv, Map o, BiFunction bfn) { final LongHashNode[] rvd = rv.data; final int ne = rvd.length; for (int idx = 0; idx < ne; ++idx) { LongHashNode lf = rvd[idx]; while(lf != null) { final LongHashNode curlf = lf; lf = lf.nextNode; final Object v = o.get(curlf.k); rvd[idx] = (v != null) ? rvd[idx].assoc(rv, curlf.k, curlf.hashcode, bfn.apply(curlf.v, v)) : rvd[idx].dissoc(rv, curlf.k); } } return rv; } public LongHashMap intersection(Map o, BiFunction bfn) { return intersection(this, o, bfn); } public static LongHashMap intersection(LongHashMap rv, Set o) { final LongHashNode[] rvd = rv.data; final int ne = rvd.length; for (int idx = 0; idx < ne; ++idx) { LongHashNode lf = rvd[idx]; while(lf != null) { final LongHashNode curlf = lf; final long k = curlf.k; lf = lf.nextNode; if(!o.contains(k)) rvd[idx] = rvd[idx].dissoc(rv,k); } } return rv; } public LongHashMap intersection(Set o) { return intersection(this, o); } @SuppressWarnings("unchecked") static LongHashMap difference(LongHashMap rv, Collection o) { final LongHashNode[] rvd = rv.data; final int mask = rv.mask; for (Object kk : o) { long k = Casts.longCast(kk); final int hashcode = rv.hash(k); final int rvidx = hashcode & mask; LongHashNode e = rvd[rvidx]; for(;e != null && !(e.k==k); e = e.nextNode); if(e != null) { rvd[rvidx] = rvd[rvidx].dissoc(rv, e.k); } } return rv; } public LongHashMap difference(Collection o) { return difference(this, o); } @SuppressWarnings("unchecked") static LongHashMap updateValues(LongHashMap rv, BiFunction valueMap) { final LongHashNode[] d = rv.data; final int nl = d.length; for(int idx = 0; idx < nl; ++idx) { LongHashNode lf = d[idx]; while(lf != null) { LongHashNode cur = lf; lf = lf.nextNode; Object newv = valueMap.apply(cur.k, cur.v); d[idx] = newv == null ? d[idx].dissoc(rv, cur.k) : d[idx].assoc(rv, cur.k, cur.hashcode, newv); } } return rv; } public LongHashMap updateValues(BiFunction valueMap) { return updateValues(this, valueMap); } @SuppressWarnings("unchecked") static LongHashMap updateValue(LongHashMap rv, Object kk, Function fn) { long k = Casts.longCast(kk); final int hc = rv.hash(k); final int idx = hc & rv.mask; final LongHashNode[] data = rv.data; LongHashNode e = data[idx]; for(; e != null && !((e.k == k)); e = e.nextNode); final Object newv = e != null ? fn.apply(e.v) : fn.apply(null); data[idx] = newv == null ? data[idx].dissoc(rv, k) : data[idx].assoc(rv, k, hc, newv); if(newv != null && e == null) rv.checkResize(null); return rv; } public LongHashMap updateValue(Object k, Function fn) { return updateValue(this, fn); } public Iterator iterator(Function leafFn) { return new HTIter(this.data, leafFn); } public Spliterator spliterator(Function leafFn) { return new HTSpliterator(this.data, this.length, leafFn); } } ================================================ FILE: java/ham_fisted/LongHashNode.java ================================================ package ham_fisted; import java.util.Map; import java.util.Iterator; import clojure.lang.IMapEntry; import ham_fisted.IMutList; public class LongHashNode implements Map.Entry, IMutList, IMapEntry { public final LongHashBase owner; public final int hashcode; public final long k; //compute-at support means we can modify v. Object v; LongHashNode nextNode; public LongHashNode(LongHashBase _owner, long _k, int hc, Object _v, LongHashNode nn) { owner = _owner; hashcode = hc; k = _k; v = _v; nextNode = nn; _owner.inc(this); } public LongHashNode(LongHashBase _owner, long _k, int hc, Object _v) { this(_owner, _k, hc, _v, null); } public LongHashNode(LongHashBase _owner, long _k, int hc) { this(_owner, _k, hc, null, null); } LongHashNode(LongHashBase _owner, LongHashNode prev) { owner = _owner; hashcode = prev.hashcode; k = prev.k; v = prev.v; nextNode = prev.nextNode; } public LongHashNode setOwner(LongHashBase nowner) { if (owner == nowner) return this; return new LongHashNode(nowner, this); } public LongHashNode clone(LongHashBase nowner) { LongHashNode rv = new LongHashNode(nowner, this); if(nextNode != null) rv.nextNode = nextNode.clone(nowner); return rv; } public final Object key() { return k; } public final Object val() { return v; } public final Object getKey() { return k; } public final Object getValue() { return v; } public Object setValue(Object vv) { Object rv = v; v = vv; return rv; } public final int size() { return 2; } public final Object get(int idx) { if(idx == 0) return k; if(idx == 1) return v; throw new RuntimeException("Index out of range."); } public LongHashNode assoc(LongHashBase nowner, long _k, int hash, Object _v) { LongHashNode retval = setOwner(nowner); if (k == _k) { retval.setValue(_v); } else { if (retval.nextNode != null) { retval.nextNode = retval.nextNode.assoc(nowner, _k, hash, _v); } else { retval.nextNode = nowner.newNode(_k, hash, _v); } } return retval; } public LongHashNode dissoc(LongHashBase nowner, long _k) { if (k == _k) { nowner.dec(this); return nextNode; } if (nextNode != null) { LongHashNode nn = nextNode.dissoc(nowner,_k); if (nn != nextNode) { LongHashNode retval = setOwner(nowner); retval.nextNode = nn; return retval; } } return this; } } ================================================ FILE: java/ham_fisted/LongMutList.java ================================================ package ham_fisted; import java.util.Random; import java.util.List; import java.util.Comparator; import java.util.function.DoubleConsumer; import java.util.function.LongConsumer; import java.util.Spliterator; import it.unimi.dsi.fastutil.longs.LongArrays; import it.unimi.dsi.fastutil.longs.LongComparator; import it.unimi.dsi.fastutil.ints.IntComparator; import java.util.function.LongConsumer; import clojure.lang.IFn; import clojure.lang.RT; @SuppressWarnings("unchecked") public interface LongMutList extends IMutList { default boolean add(Object obj) { addLong(Casts.longCast(obj)); return true; } default void add(int idx, int count, Object v) { long d = Casts.longCast(v); int end = idx + count; for(; idx < end; ++idx) addLong( d ); } default void addLong(long v) { throw new RuntimeException("Object " + String.valueOf(getClass()) + " failed to define addLong method"); } default void addBoolean( boolean obj ) { addLong(obj ? 1 : 0); } default void addDouble(double obj) { addLong(Casts.longCast(obj));} @SuppressWarnings("unchecked") default Object set(int idx, Object obj) { final long v = getLong(idx); setLong(idx, Casts.longCast(obj)); return v; } default void setBoolean(int idx, boolean obj) { setLong(idx, obj ? 1 : 0); } default void setDouble(int idx, double obj) { setLong(idx, Casts.longCast(obj)); } default Object get(int idx) { return getLong(idx); } default double getDouble(int idx) { return getLong(idx); } default void fillRange(long startidx, final long endidx, Object v) { ChunkedList.checkIndexRange(0, size(), startidx, endidx); long l = Casts.longCast(v); final int ee = (int)endidx; int ss = (int)startidx; for(; ss < ee; ++ss) { setLong(ss, l); } } static class LongSubList extends IMutList.MutSubList implements LongMutList { @SuppressWarnings("unchecked") public LongSubList(IMutList l, int ss, int ee) { super(l, ss, ee); } public Object reduce(IFn rfn, Object init) { return LongMutList.super.reduce(rfn, init); } } default IMutList subList(int ssidx, int seidx) { ChunkedList.sublistCheck(ssidx, seidx, size()); return new LongSubList(this, ssidx, seidx); } default void fillRange(final long startidx, List l) { if (l.isEmpty()) return; final int ss = (int)startidx; final int sz = size(); final int endidx = ss + l.size(); ArrayLists.checkIndexRange(size(), ss, endidx); Reductions.serialReduction(new Reductions.IndexedLongAccum(new IFnDef.OLLO() { public Object invokePrim(Object acc, long idx, long v) { ((IMutList)acc).setLong((int)idx+ss, v); return acc; } }), this, l); } default void addRange(int startidx, int endidx, Object v) { Long l = Long.valueOf(Casts.longCast(v)); for(; startidx < endidx; ++startidx) { add(startidx, l); } } default boolean addAllReducible(Object obj) { final int sz = size(); Reductions.serialReduction(new IFnDef.OLO() { public Object invokePrim(Object lhs, long rhs) { ((IMutList)lhs).addLong(rhs); return lhs; } }, this, obj); return sz != size(); } default IntComparator indexComparator() { return new IntComparator() { public int compare(int lidx, int ridx) { return Long.compare(getLong(lidx), getLong(ridx)); } }; } default void sort(Comparator c) { LongComparator lc = ArrayLists.LongArraySubList.asLongComparator(c); if (c == null || lc != null) { final long[] data = toLongArray(); if(c == null) LongArrays.parallelQuickSort(data); else LongArrays.parallelQuickSort(data, lc); fillRange(0, ArrayLists.toList(data)); } else { IMutList.super.sort(c); } } default void shuffle(Random r) { fillRange(0, immutShuffle(r)); } default List immutShuffle(Random r) { final long[] data = toLongArray(); LongArrays.shuffle(data, r); return ArrayLists.toList(data); } default Object reduce(final IFn rfn, Object init) { return longReduction(Transformables.toLongReductionFn(rfn), init); } default Object longReduction(IFn.OLO rfn, Object init) { final int sz = size(); for (int idx = 0; idx < sz && !RT.isReduced(init); ++idx) init = rfn.invokePrim(init, getLong(idx)); return Reductions.unreduce(init); } @SuppressWarnings("unchecked") default Spliterator spliterator() { return new RandomAccessSpliterator.LongSpliterator(this, 0, size()); } } ================================================ FILE: java/ham_fisted/MapFn.java ================================================ package ham_fisted; import clojure.lang.IFn; import clojure.lang.ISeq; public class MapFn implements IFnDef { public static IFn create(IFn src, IFn dst) { if(src instanceof IFn.OD && dst instanceof IFn.DD) { final IFn.OD ss = (IFn.OD)src; final IFn.DD dd = (IFn.DD)dst; return new IFnDef.OD() { public double invokePrim(Object obj) { return dd.invokePrim(ss.invokePrim(obj)); } }; } else if (src instanceof IFn.OL && dst instanceof IFn.LL) { final IFn.OL ss = (IFn.OL)src; final IFn.LL dd = (IFn.LL)dst; return new IFnDef.OL() { public long invokePrim(Object obj) { return dd.invokePrim(ss.invokePrim(obj)); } }; } else if( src instanceof IFn.LL && dst instanceof IFn.LL) { final IFn.LL ss = (IFn.LL)src; final IFn.LL dd = (IFn.LL)dst; return new IFnDef.LL() { public long invokePrim(long v) { return dd.invokePrim(ss.invokePrim(v)); } }; } else if( src instanceof IFn.DD && dst instanceof IFn.DD) { final IFn.DD ss = (IFn.DD)src; final IFn.DD dd = (IFn.DD)dst; return new IFnDef.DD() { public double invokePrim(double v) { return dd.invokePrim(ss.invokePrim(v)); } }; } else if( src instanceof IFn.DL && dst instanceof IFn.LL) { final IFn.DL ss = (IFn.DL)src; final IFn.LL dd = (IFn.LL)dst; return new IFnDef.DL() { public long invokePrim(double v) { return dd.invokePrim(ss.invokePrim(v)); } }; } else if( src instanceof IFn.LD && dst instanceof IFn.DD) { final IFn.LD ss = (IFn.LD)src; final IFn.DD dd = (IFn.DD)dst; return new IFnDef.LD() { public double invokePrim(long v) { return dd.invokePrim(ss.invokePrim(v)); } }; } //Fallthrough, no special treatment return new MapFn(src, dst); } public final IFn srcFn; public final IFn dstFn; private MapFn(IFn sfn, IFn dfn) { srcFn = sfn; dstFn = dfn; } public Object invoke() { return dstFn.invoke(srcFn.invoke()); } public Object invoke(Object arg1) { return dstFn.invoke(srcFn.invoke(arg1)); } public Object invoke(Object arg1, Object arg2) { return dstFn.invoke(srcFn.invoke(arg1, arg2)); } public Object invoke(Object arg1, Object arg2, Object arg3) { return dstFn.invoke(srcFn.invoke(arg1, arg2, arg3)); } public Object invoke(Object arg1, Object arg2, Object arg3, Object arg4) { return dstFn.invoke(srcFn.invoke(arg1, arg2, arg3, arg4)); } public Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5) { return dstFn.invoke(srcFn.invoke(arg1, arg2, arg3, arg4, arg5)); } public Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6) { return dstFn.invoke(srcFn.invoke(arg1, arg2, arg3, arg4, arg5, arg6)); } public Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7) { return dstFn.invoke(srcFn.invoke(arg1, arg2, arg3, arg4, arg5, arg6, arg7)); } public Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8) { return dstFn.invoke(srcFn.invoke(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8)); } public Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9) { return dstFn.invoke(srcFn.invoke(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9)); } public Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10) { return dstFn.invoke(srcFn.invoke(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10)); } public Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10, Object arg11) { return dstFn.invoke(srcFn.invoke(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11)); } public Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10, Object arg11, Object arg12) { return dstFn.invoke(srcFn.invoke(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12)); } public Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10, Object arg11, Object arg12, Object arg13) { return dstFn.invoke(srcFn.invoke(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12, arg13)); } public Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10, Object arg11, Object arg12, Object arg13, Object arg14) { return dstFn.invoke(srcFn.invoke(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12, arg13, arg14)); } public Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10, Object arg11, Object arg12, Object arg13, Object arg14, Object arg15) { return dstFn.invoke(srcFn.invoke(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12, arg13, arg14, arg15)); } public Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10, Object arg11, Object arg12, Object arg13, Object arg14, Object arg15, Object arg16) { return dstFn.invoke(srcFn.invoke(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12, arg13, arg14, arg15, arg16)); } public Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10, Object arg11, Object arg12, Object arg13, Object arg14, Object arg15, Object arg16, Object arg17) { return dstFn.invoke(srcFn.invoke(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12, arg13, arg14, arg15, arg16, arg17)); } public Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10, Object arg11, Object arg12, Object arg13, Object arg14, Object arg15, Object arg16, Object arg17, Object arg18) { return dstFn.invoke(srcFn.invoke(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12, arg13, arg14, arg15, arg16, arg17, arg18)); } public Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10, Object arg11, Object arg12, Object arg13, Object arg14, Object arg15, Object arg16, Object arg17, Object arg18, Object arg19) { return dstFn.invoke(srcFn.invoke(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12, arg13, arg14, arg15, arg16, arg17, arg18, arg19)); } public Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10, Object arg11, Object arg12, Object arg13, Object arg14, Object arg15, Object arg16, Object arg17, Object arg18, Object arg19, Object arg20) { return dstFn.invoke(srcFn.invoke(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12, arg13, arg14, arg15, arg16, arg17, arg18, arg19, arg20)); } public Object invoke(Object arg1, Object arg2, Object arg3, Object arg4, Object arg5, Object arg6, Object arg7, Object arg8, Object arg9, Object arg10, Object arg11, Object arg12, Object arg13, Object arg14, Object arg15, Object arg16, Object arg17, Object arg18, Object arg19, Object arg20, Object... args) { return dstFn.invoke(srcFn.invoke(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12, arg13, arg14, arg15, arg16, arg17, arg18, arg19, arg20, args)); } public Object applyTo(ISeq args) { return dstFn.invoke(srcFn.invoke(args)); } } ================================================ FILE: java/ham_fisted/MapForward.java ================================================ package ham_fisted; import java.util.Map; import java.util.Iterator; import java.util.Spliterator; import java.util.Set; import java.util.Collection; import java.util.function.BiFunction; import java.util.function.Function; import java.util.function.Consumer; import java.util.function.BiConsumer; import clojure.lang.ITransientMap; import clojure.lang.ITransientAssociative2; import clojure.lang.IMeta; import clojure.lang.IPersistentMap; import clojure.lang.IFn; import clojure.lang.IObj; import clojure.lang.ILookup; import clojure.lang.IHashEq; import clojure.lang.Counted; import clojure.lang.IKVReduce; import clojure.lang.MapEquivalence; import clojure.lang.IMapIterable; import clojure.lang.Seqable; import clojure.lang.ISeq; import clojure.lang.RT; import clojure.lang.MapEntry; import clojure.lang.IMapEntry; public class MapForward implements Map, ITypedReduce, IFnDef, IHashEq, ILookup, Counted, IMeta, IObj, IKVReduce, MapEquivalence, Seqable { final Map ht; final IPersistentMap meta; public MapForward(Map ht, IPersistentMap meta) { this.ht = ht; this.meta = meta; } public void clear() { ht.clear(); } public V compute(K key, BiFunction remappingFunction) { return ht.compute(key, remappingFunction); } public V computeIfPresent(K key, BiFunction remappingFunction) { return ht.computeIfPresent(key, remappingFunction); } public V computeIfAbsent(K key, Function mappingFunction) { return ht.computeIfAbsent(key, mappingFunction); } public boolean containsKey(Object key) { return ht.containsKey(key); } public boolean containsValue(Object v) { return ht.containsValue(v); } public Set> entrySet() { return ht.entrySet(); } public boolean equals(Object o) { return CljHash.mapEquiv(ht, o); } public boolean equiv(Object o) { return equals(o); } public void forEach(BiConsumer action) { ht.forEach(action); } public V get(Object k) { return ht.get(k); } public V getOrDefault(Object k, V dv) { return ht.getOrDefault(k, dv); } public int hashCode() { return CljHash.mapHashcode(ht); } public int hasheq() { return hashCode(); } public boolean isEmpty() { return ht.isEmpty(); } public Set keySet() { return ht.keySet(); } public V merge(K key, V value, BiFunction remappingFunction) { return ht.merge(key, value, remappingFunction); } public V put(K key, V value) { return ht.put(key, value); } public void putAll(Map m) { ht.putAll(m); } public V putIfAbsent(K key, V value) { return ht.putIfAbsent(key, value); } public V remove(Object k) { return ht.remove(k); } public boolean remove(Object key, Object value) { return ht.remove(key,value); } public V replace(K key, V value) { return ht.replace(key, value); } public boolean replace(K key, V oldValue, V newValue) { return ht.replace(key, oldValue, newValue); } public void replaceAll(BiFunction function) { ht.replaceAll(function); } public int size() { return ht.size(); } public int count() { return ht.size(); } public Collection values() { return ht.values(); } public Iterator iterator() { return entrySet().iterator(); } public Iterator keyIterator() { return keySet().iterator(); } public Iterator valIterator() { return values().iterator(); } public Object reduce(IFn rfn, Object acc) { return Reductions.serialReduction(rfn, acc, entrySet()); } @SuppressWarnings("unchecked") public final IMapEntry entryAt(Object key) { return containsKey(key) ? new MapEntry(key, get(key)) : null; } public Object valAt(Object key) { return get(key); } @SuppressWarnings("unchecked") public Object valAt(Object key, Object notFound) { return getOrDefault(key, (V)notFound); } public final Object invoke(Object arg1) { return get(arg1); } @SuppressWarnings("unchecked") public final Object invoke(Object arg1, Object notFound) { return getOrDefault(arg1, (V)notFound); } public ISeq seq() { return LazyChunkedSeq.chunkIteratorSeq(iterator()); } public IPersistentMap meta() { return meta; } public MapForward withMeta(IPersistentMap m) { return new MapForward(ht, m); } @SuppressWarnings("unchecked") public Object kvreduce(IFn f, Object init) { return reduce(new IFnDef() { public Object invoke(Object acc, Object v) { final Map.Entry me = (Map.Entry)v; return f.invoke(acc, me.getKey(), me.getValue()); } }, init); } } ================================================ FILE: java/ham_fisted/MapSetOps.java ================================================ package ham_fisted; import java.util.Map; import java.util.Set; import java.util.Collection; import java.util.function.BiFunction; public interface MapSetOps { Map union(Map rhs, BiFunction bfn); //Trim to these key-values, apply bfn to resolve vals. Map intersection(Map rhs, BiFunction bfn); //Trim to these keys Map intersection(Set rhs); default Map difference(Map rhs) { return difference(rhs.keySet()); } Map difference(Collection rhs); } ================================================ FILE: java/ham_fisted/MergeIterator.java ================================================ package ham_fisted; import java.util.Iterator; import java.util.Comparator; import java.util.ArrayList; import java.util.function.Predicate; import java.util.PriorityQueue; import java.util.NoSuchElementException; public class MergeIterator implements Iterator { public static class CurrentIterator implements Iterator { public static final Object invalid = new Object(); Object current; Iterator iter; public CurrentIterator(Iterator iter) { this.iter = iter; this.current = invalid; } public CurrentIterator(Iterator iter, Object cur) { this.iter = iter; this.current = cur; } public boolean hasNext() { return iter.hasNext(); } public Object next() { return current = iter.next(); } public Object current() { return current; } public boolean hasCurrent() { return current != invalid; } public static CurrentIterator create(Iterator iter) { if(iter.hasNext()) return new CurrentIterator(iter, iter.next()); return null; } } java.util.Comparator cmp; CurrentIterator[] iters; int curIdx; Predicate p; @SuppressWarnings("unchecked") int leastIndex() { int nIters = iters.length; if(nIters <= 1) return 0; Object leastv = iters[0].current(); int leastIdx = 0; //Potential for binary search here I guess for(int idx = 1; idx < nIters; ++idx) { Object cur = iters[idx].current(); if (cmp.compare(leastv, cur) > -1) { leastv = cur; leastIdx = idx; } } return leastIdx; } public MergeIterator(CurrentIterator[] iters, Comparator cmp, Predicate p) { this.iters = iters; this.p = p; this.cmp = cmp; curIdx = leastIndex(); } public boolean hasNext() { return iters.length != 0; } @SuppressWarnings("unchecked") public Object next() { Object rv = null; do { if(iters.length == 0) return null; CurrentIterator iter = iters[curIdx]; rv = iter.current(); if(iter.hasNext()) { iter.next(); } else { int ne = iters.length; CurrentIterator[] newIters = new CurrentIterator[ne-1]; for(int idx = 0; idx < ne; ++ idx) { if(idx < curIdx) { newIters[idx] = iters[idx]; } else if (idx > curIdx) { newIters[idx-1] = iters[idx]; } } iters = newIters; } curIdx = leastIndex(); } while(!p.test(rv)); return rv; } public static final Iterator emptyIter = new Iterator() { public boolean hasNext() { return false; } public Object next() { throw new java.util.NoSuchElementException(); } }; public static class TwoWayMergeIterator implements Iterator { CurrentIterator lhs; CurrentIterator rhs; Comparator cmp; Predicate p; boolean left; @SuppressWarnings("unchecked") public TwoWayMergeIterator(CurrentIterator lhs, CurrentIterator rhs, Comparator cmp, Predicate p) { this.lhs = lhs; this.rhs = rhs; this.cmp = cmp; this.p = p; this.left = cmp.compare(lhs.current(), rhs.current()) < 0 ? true : false; } public boolean hasNext() { return lhs != null || rhs != null; } @SuppressWarnings("unchecked") public Object next() { if(lhs == null && rhs == null) throw new java.util.NoSuchElementException(); Object rv; do { CurrentIterator update = left ? lhs : rhs; if(update == null) return null; rv = update.current(); if(update.hasNext()) { update.next(); } else { if(left) lhs = null; else rhs = null; } if(lhs == null) left = false; else if (rhs == null) left = true; else left = cmp.compare(lhs.current(), rhs.current()) < 0 ? true : false; } while(!p.test(rv)); return rv; } public static Iterator create(Iterator lhs, Iterator rhs, Comparator cmp) { return new TwoWayMergeIterator(new CurrentIterator(lhs, lhs.next()), new CurrentIterator(rhs, rhs.next()), cmp, MergeIterator.alwaysTrue); } } public static class PriorityQueueIterator implements Iterator { PriorityQueue pq; Predicate p; public PriorityQueueIterator(PriorityQueue pq, Predicate p) { this.pq = pq; this.p = p; } public boolean hasNext() { return !pq.isEmpty(); } @SuppressWarnings("unchecked") public Object next() { if(pq.isEmpty()) throw new NoSuchElementException(); while(true) { if(pq.isEmpty()) return null; final Object[] entry = (Object[])pq.poll(); Iterator iter = (Iterator)entry[0]; Object rv = entry[1]; if(iter.hasNext()) { entry[1] = iter.next(); pq.offer(entry); } if(p.test(rv)) return rv; } } @SuppressWarnings("unchecked") public static Iterator create(Iterable srcIters, Comparator cmp, Predicate p) { Comparator pqCmp = new Comparator() { public int compare(Object lhs, Object rhs) { return cmp.compare(((Object[])lhs)[1],((Object[])rhs)[1]); } }; PriorityQueue pq = new PriorityQueue(pqCmp); for(Iterator iter : srcIters) { if(iter != null && iter.hasNext()) pq.offer(new Object[] {iter, iter.next()}); } return new PriorityQueueIterator(pq, p); } public static Iterator create(Iterable srcIters, Comparator cmp) { return create(srcIters, cmp, alwaysTrue); } } @SuppressWarnings("unchecked") public static final Iterator createMergeIterator(Iterable srcIters, Comparator cmp, Predicate p) { ArrayList validIters = new ArrayList(); for(Iterator iter : srcIters) { if(iter != null && iter.hasNext()) { validIters.add(iter); } } if(validIters.isEmpty()) return emptyIter; if(validIters.size() >= 8) { return PriorityQueueIterator.create(validIters, cmp, p); } CurrentIterator[] iters = new CurrentIterator[validIters.size()]; for(int idx = 0; idx < iters.length; ++idx) { Iterator iter = validIters.get(idx); iters[idx] = new CurrentIterator(iter, iter.next()); } if(validIters.size() == 2) return new TwoWayMergeIterator(iters[0], iters[1], cmp, p); else return new MergeIterator(iters, cmp, p); } public static final Predicate alwaysTrue = new Predicate() { public boolean test(Object o){ return true; } }; public static Iterator createMergeIterator(Iterable srcIters, Comparator cmp) { return createMergeIterator(srcIters, cmp, alwaysTrue); } } ================================================ FILE: java/ham_fisted/MethodImplCache.java ================================================ /** * Copyright (c) Rich Hickey. All rights reserved. * The use and distribution terms for this software are covered by the * Eclipse Public License 1.0 (http://opensource.org/licenses/eclipse-1.0.php) * which can be found in the file epl-v10.html at the root of this distribution. * By using this software in any fashion, you are agreeing to be bound by * the terms of this license. * You must not remove this notice, or any other, from this software. **/ /* rich Nov 8, 2009 */ /* Heavily modified by Chris Nuernberger - Dec 23, 2023 */ package ham_fisted; import java.util.HashMap; import java.util.Set; import java.util.concurrent.locks.ReentrantLock; import java.util.concurrent.ConcurrentHashMap; import clojure.lang.IFn; import clojure.lang.Keyword; public final class MethodImplCache { final ReentrantLock extLock = new ReentrantLock(); //Sparse table of class to ifn. //a lock-protected hashmap as we need to perform potentially many lookups //in rapid succession. final HashMap extensions = new HashMap(); volatile Object nullExtension = null; //Potentially dense table of resolved lookups. final ConcurrentHashMap lookupCache = new ConcurrentHashMap(); public static final Object DEFAULT = new Object(); public final Keyword methodk; public final Keyword ns_methodk; public final Class iface; public final IFn ifaceFn; public MethodImplCache(Keyword methodk, Keyword ns_methodk, Class iface, IFn ifaceFn) { this.methodk = methodk; this.ns_methodk = ns_methodk; this.iface = iface; this.ifaceFn = ifaceFn; extensions.put(iface, ifaceFn); } public void extend(Class c, Object fn) { extLock.lock(); try { if(c == null) nullExtension = fn; else { if(fn == null) extensions.remove(c); else extensions.put(c, fn); } } finally { extLock.unlock(); } lookupCache.clear(); } public Set registeredClasses() { return extensions.keySet(); } @SuppressWarnings("unchecked") public Object recurCheckInterface(Set consideredSet, Class[] ifaces) { final Object defVal = DEFAULT; Object rv = defVal; int ni = ifaces.length; for(int idx = 0; idx < ni && rv == defVal; ++idx) { Class iface = ifaces[idx]; if(consideredSet.add(iface)) { rv = extensions.getOrDefault(iface, defVal); } } //check derived interface in second pass so primary interfaces get first choice if(rv == defVal) { for(int idx = 0; idx < ni && rv == defVal; ++idx) { Class iface = ifaces[idx]; rv = recurCheckInterface(consideredSet, iface.getInterfaces()); } } return rv; } static final Class objAryCls = Object[].class; @SuppressWarnings("unchecked") public Object findFnFor(Class c) { if(c == null) return nullExtension; final Object defVal = DEFAULT; Object rv = lookupCache.getOrDefault(c, defVal); //Include caching when lookup fails. if(rv != defVal) return rv; //rv is the default value at this point if(iface.isAssignableFrom(c)) { lookupCache.put(c, ifaceFn); return ifaceFn; } extLock.lock(); try { HashSet considered = new HashSet(); for(Class cc = c; cc != null && rv == defVal; cc = cc.getSuperclass()) { rv = extensions.getOrDefault(cc, defVal); if(rv == defVal && cc == c && objAryCls.isAssignableFrom(cc)) rv = extensions.getOrDefault(objAryCls, defVal); if(rv == defVal) rv = recurCheckInterface(considered, cc.getInterfaces()); } } finally { extLock.unlock(); } if(rv != defVal) lookupCache.put(c, rv); return rv == defVal ? null : rv; } } ================================================ FILE: java/ham_fisted/MutList.java ================================================ package ham_fisted; import static ham_fisted.ChunkedList.*; import java.util.List; import java.util.Objects; import java.util.RandomAccess; import java.util.Iterator; import java.util.ListIterator; import java.util.Collection; import java.util.Iterator; import java.util.Arrays; import java.util.ArrayList; import java.util.Map; import java.util.function.Function; import java.util.function.BiFunction; import clojure.lang.Indexed; import clojure.lang.RT; import clojure.lang.IReduce; import clojure.lang.Counted; import clojure.lang.IKVReduce; import clojure.lang.IFn; import clojure.lang.IHashEq; import clojure.lang.Seqable; import clojure.lang.Reversible; import clojure.lang.ISeq; import clojure.lang.IObj; import clojure.lang.IPersistentMap; import clojure.lang.ITransientVector; import clojure.lang.Util; import clojure.lang.Counted; import clojure.lang.IReduceInit; public class MutList implements IMutList, ChunkedListOwner, Cloneable, ITransientVector, UpdateValues { final ChunkedList data; public MutList() { data = new ChunkedList(); } public MutList(int capacity) { data = new ChunkedList(capacity); } public MutList(ChunkedList other) { data = new ChunkedList(other, false); } //deep cloning constructor public MutList(MutList other) { this(other.data.clone(0, other.data.nElems, 0, true)); } @SafeVarargs public static MutList create(boolean owning, IPersistentMap meta, E... data) { return new MutList(ChunkedList.create(owning, meta, data)); } public final ChunkedListSection getChunkedList() { return new ChunkedListSection(data.data, 0, data.nElems); } public final MutList clone() { return new MutList(this); } public final MutList cloneList() { return clone(); } public final boolean add(E v) { data.add(v); return true; } public final void add(int idx, E v) { data.add(v,idx); } public final boolean addAll(Collection c) { return addAllReducible(c); } @SuppressWarnings("unchecked") public final boolean addAllReducible(Object c) { if( c instanceof Map) c = ((Map)c).entrySet(); final int ssz = size(); if (c instanceof ChunkedListOwner) { final ChunkedListSection section = ((ChunkedListOwner)c).getChunkedList(); int idx = data.nElems; int oidx = section.startidx; int one = section.endidx - oidx; final int finalLen = idx + one; data.enlarge(finalLen); data.nElems = finalLen; final Object[][] mdata = data.data; final Object[][] odata = section.data; while(idx < finalLen) { final int cidx = idx / 32; final int ocidx = oidx / 32; final int leftover = finalLen - idx; final int eidx = idx % 32; final int oeidx = oidx % 32; final int copyLen = Math.min(leftover, Math.min(32-eidx, 32-oeidx)); System.arraycopy(odata[ocidx], oeidx, mdata[cidx], eidx, copyLen); idx += copyLen; oidx += copyLen; } } else if (c instanceof IReduceInit) { final ChunkedList d = data; if( c instanceof RandomAccess || c instanceof Counted) { final int cz = c instanceof RandomAccess ? ((List)c).size() : ((Counted)c).count(); if (cz == 0) return false; d.enlarge(ssz + cz); d.nElems = ssz + cz; d.fillRangeReduce(ssz, c); } else { ((IReduceInit)c).reduce(new IFnDef() { public Object invoke(Object lhs, Object rhs) { d.add(rhs); return this; } }, this); } } else if (c instanceof RandomAccess) { final List l = (List)c; final int cs = l.size(); data.enlarge(cs + ssz); int idx = ssz; data.nElems = idx + cs; final Object[][] mdata = data.data; int cidx = 0; while(cidx < cs) { final Object[] chunk = mdata[idx/32]; int eidx = idx % 32; final int groupLen = Math.min(chunk.length-eidx, cs - cidx); for (int lidx = 0; lidx < groupLen; ++lidx, ++eidx, ++cidx) { chunk[eidx] = l.get(cidx); } idx += groupLen; } } else { if(! (c instanceof Iterable)) throw new RuntimeException("Object must either be instance of IReduceInit or java.util.Iterable"); ((Iterable)c).forEach((v)->add((E)v)); } return ssz != size(); } public final boolean addAll(int idx, Collection c) { if (c.isEmpty()) return false; if (idx == data.nElems) return addAll(c); indexCheck(idx); if (! (c instanceof RandomAccess)) { ArrayList al = new ArrayList(); al.addAll(c); c = al; } final int cs = c.size(); data.widen(idx, idx + cs); for(E item: c) data.setValue(idx++, item); return true; } public final void clear() { data.clear(); } public final boolean contains(Object v) { return data.contains(0, data.nElems, v); } public final boolean containsAll(Collection c) { return data.containsAll(0, data.nElems, c); } final int indexCheck(int idx) { return ChunkedList.indexCheck(0, data.nElems, idx); } final int indexCheck(long idx) { return ChunkedList.indexCheck(0, data.nElems, (int)idx); } final int wrapIndexCheck(int idx) { return ChunkedList.wrapIndexCheck(0, data.nElems, idx); } @SuppressWarnings("unchecked") public final E set(int idx, E e) { return (E)data.setValueRV(indexCheck(idx), e); } @SuppressWarnings("unchecked") public final E get(int idx) { return (E)data.getValue(ChunkedList.indexCheck(data.nElems, idx)); } public final int indexOf(Object o) { return data.indexOf(0, data.nElems, o); } public final boolean isEmpty() { return data.nElems == 0; } @SuppressWarnings("unchecked") public final Iterator iterator() { return (Iterator)data.iterator(); } public static class SubMutList implements IMutList, ChunkedListOwner, Cloneable { final int startidx; final int nElems; final ChunkedList data; SubMutList(int sidx, int eidx, ChunkedList d) { startidx = sidx; nElems = eidx - sidx; data = d; } public final ChunkedListSection getChunkedList() { return new ChunkedListSection(data.data, startidx, startidx+nElems); } public final MutList clone() { return new MutList(data.clone(startidx, startidx+nElems)); } public final MutList cloneList() { return clone(); } final int indexCheck(int idx) { return ChunkedList.indexCheck(startidx, nElems, idx); } final int wrapIndexCheck(int idx) { return ChunkedList.wrapIndexCheck(startidx, nElems, idx); } public final int size() { return nElems; } public final boolean isEmpty() { return nElems == 0; } @SuppressWarnings("unchecked") public final E set(int idx, E e) { return (E)data.setValueRV(indexCheck(idx), e); } @SuppressWarnings("unchecked") public final E get(int idx) { return (E)data.getValue(ChunkedList.indexCheck(startidx, nElems, idx)); } public final int indexOf(Object obj) { return data.indexOf(startidx, startidx+nElems, obj); } public final int lastIndexOf(Object obj) { return data.lastIndexOf(startidx, startidx+nElems, obj); } public final boolean contains(Object obj) { return data.contains(startidx, startidx+nElems, obj); } public final boolean containsAll(Collection c) { return data.containsAll(startidx, startidx+nElems, c); } @SuppressWarnings("unchecked") public final Iterator iterator() { return (Iterator)data.iterator(startidx, startidx + nElems); } public final ListIterator listIterator(int idx) { return data.listIterator(indexCheck(idx), startidx+nElems, (E)null); } public final ListIterator listIterator() { return listIterator(0); } public final IMutList subList(int ssidx, int seidx) { ChunkedList.sublistCheck(ssidx, seidx, nElems); return new SubMutList(ssidx + startidx, seidx + startidx, data); } public final Object[] toArray() { return data.toArray(startidx, startidx + nElems); } @SuppressWarnings("unchecked") public final T[] toArray(T[] init) { return (T[])data.fillArray(startidx, startidx + nElems, Arrays.copyOf(init, nElems)); } public final Object nth(int idx) { return data.getValue(wrapIndexCheck(idx)); } public final Object nth(int idx, Object notFound) { if (idx < 0) idx = idx + data.nElems; return idx < data.nElems && idx > -1 ? data.getValue(idx+startidx) : notFound; } public final Object invoke(Object idx) { return nth(RT.intCast(idx)); } public final Object invoke(Object idx, Object notFound) { return nth(RT.intCast(idx), notFound); } public final int count() { return nElems; } public final Object reduce(IFn f) { return data.reduce(startidx, startidx+nElems, f); } public final Object reduce(IFn f, Object init) { return data.reduce(startidx, startidx+nElems, f,init); } public final Object kvreduce(IFn f, Object init) { return data.kvreduce(startidx, startidx+nElems, f, init); } public void fillRange(long sidx, long eidx, Object v) { final int ssidx = indexCheck((int)sidx); if (eidx < sidx || eidx > nElems) throw new RuntimeException("End index out of range: " + String.valueOf(eidx)); data.fillRange((int)ssidx, (int)(eidx + startidx), v); } public void fillRange(long sidx, List v) { final int ssidx = indexCheck((int)sidx); final int eidx = (int)sidx + v.size(); if (eidx > nElems) throw new RuntimeException("End index out of range: " + String.valueOf(eidx)); data.fillRangeReduce(ssidx, v); } public final int hashCode() { return data.hasheq(startidx, startidx + nElems); } public final int hasheq() { return hashCode(); } public final String toString() { return Transformables.sequenceToString(this); } public final boolean equals(Object other) { return equiv(other); } public final boolean equiv(Object other) { return data.equiv(startidx, startidx+nElems, other); } public final ISeq seq() { return data.seq(startidx, startidx+nElems); } public final ISeq rseq() { return data.rseq(startidx, startidx+nElems); } public IPersistentMap meta() { return data.meta(); } public SubMutList withMeta(IPersistentMap m) { return new SubMutList(startidx, startidx + nElems, data.withMeta(m)); } } public final IMutList subList(int startidx, int endidx) { ChunkedList.sublistCheck(startidx, endidx, size()); return new SubMutList(startidx, endidx, data); } public final ListIterator listIterator(int idx) { return data.listIterator(idx, data.nElems, (E)null); } public final ListIterator listIterator() { return listIterator(0); } public final int lastIndexOf(Object obj) { return data.lastIndexOf(0, data.nElems, obj); } @SuppressWarnings("unchecked") public final E remove(int idx) { final E retval = (E)data.getValue(idx); data.shorten(idx, idx+1); return retval; } public void fillRange(int startidx, int endidx, Object v) { indexCheck(startidx); if(endidx < startidx || endidx > data.nElems) throw new RuntimeException("End index out of range: " + String.valueOf(endidx)); data.fillRange(startidx, endidx, v); } public void fillRange(int startidx, List v) { indexCheck(startidx); final int endidx = v.size(); if(endidx < startidx || endidx > data.nElems) throw new RuntimeException("End index out of range: " + String.valueOf(endidx)); data.fillRangeReduce(startidx, v); } public void addRange(int startidx, int endidx, Object v) { indexCheck(startidx); data.addRange(startidx, endidx, v); } public void removeRange(int startidx, int endidx) { indexCheck(startidx); if (endidx == startidx) return; if(endidx < startidx || endidx > data.nElems) throw new RuntimeException("End index out of range: " + String.valueOf(endidx)); data.shorten(startidx, endidx); } public final boolean remove(Object obj) { final int idx = indexOf(obj); if (idx != -1) remove(idx); return idx != -1; } public final boolean retainAll(Collection c) { throw new RuntimeException("Unimplemented"); } public final boolean removeAll(Collection c) { throw new RuntimeException("Unimplemented"); } @SuppressWarnings("unchecked") public final T[] toArray(T[] typeMarker) { return (T[])data.fillArray(Arrays.copyOf(typeMarker, data.nElems)); } public final Object[] toArray() { return data.toArray(); } public final int size() { return data.nElems; } public final int capacity() { return data.capacity; } public final Object nth(int idx) { return data.getValue(wrapIndexCheck(idx)); } public final Object nth(int idx, Object notFound) { if (idx < 0) idx = idx + data.nElems; return idx < data.nElems && idx > -1 ? data.getValue(idx) : notFound; } public final Object invoke(Object idx) { return nth(RT.intCast(idx)); } public final Object invoke(Object idx, Object notFound) { return nth(RT.intCast(idx), notFound); } public final int count() { return data.size(); } public final Object reduce(IFn f) { return data.reduce(0, data.nElems, f); } public final Object reduce(IFn f, Object init) { return data.reduce(0, data.nElems, f,init); } public final Object kvreduce(IFn f, Object init) { return data.kvreduce(0, data.nElems, f, init); } public final ISeq seq() { return data.seq(0, data.nElems); } public final ISeq rseq() { return data.rseq(0, data.nElems); } public final int hashCode() { return data.hasheq(0, data.nElems); } public final int hasheq() { return hashCode(); } public final boolean equals(Object other) { if (other == this) return true; return equiv(other); } public final boolean equiv(Object other) { return data.equiv(0, data.nElems, other); } public final String toString() { return Transformables.sequenceToString(this); } public IPersistentMap meta() { return data.meta(); } public MutList withMeta(IPersistentMap m) { return new MutList(data.withMeta(m)); } @SuppressWarnings("unchecked") public final MutList assocN(int i, Object obj) { if (i == data.nElems) add((E)obj); set(indexCheck(i), (E)obj); return this; } public final MutList pop() { if(data.nElems == 0) throw new RuntimeException("Cannot pop empty vector."); remove(data.nElems-1); return this; } public final MutList assoc(Object i, Object obj) { if(!Util.isInteger(i)) throw new RuntimeException("Vectors must have integer keys"); return assocN(RT.intCast(i), obj); } @SuppressWarnings("unchecked") public final MutList conj(Object obj) { add((E)obj); return this; } public final Object valAt(Object key) { return valAt(key, null); } public final Object valAt(Object key, Object notFound) { if(Util.isInteger(key)) { int k = RT.intCast(key); if(k >= 0 && k < data.nElems) return data.getValue(k); } return notFound; } public ImmutList persistent() { return new ImmutList(0, data.nElems, data); } public ImmutList updateValues(BiFunction valueMap) { return persistent().updateValues(valueMap); } public ImmutList updateValue(Object key, Function valueMap) { return persistent().updateValue(key, valueMap); } } ================================================ FILE: java/ham_fisted/MutTreeList.java ================================================ package ham_fisted; import java.util.Arrays; import clojure.lang.Box; import clojure.lang.IPersistentMap; import clojure.lang.ITransientVector; import clojure.lang.RT; import clojure.lang.IPersistentVector; public class MutTreeList extends TreeListBase implements ITransientVector { int nTail; boolean persistent = false; final IPersistentMap meta; public final Object sentinel = new Object(); public int nTail() { return nTail; } public Object[] validTail() { return nTail == tail.length ? tail : Arrays.copyOf(tail, nTail); } public MutTreeList(Object root, Object[] tail, IPersistentMap meta, int shift, int count) { super(root, tail, shift, count); this.meta = meta; } public MutTreeList() { super(new Leaf(null, new Object[0][]), new Object[tailWidth], 0, 0); this.nTail = 0; this.meta = null; } public MutTreeList(TreeListBase other, IPersistentMap meta) { super(other); this.meta = meta; nTail = tail.length; this.tail = Arrays.copyOf(this.tail, tailWidth); } void consTail(Object[] tail) { Object rv = shift == 0 ? ((Leaf)root).add(sentinel, tail) : ((Branch)root).add(sentinel, shift, tail); if(rv instanceof Object[]) { shift = shift+1; root = new Branch(sentinel, (Object[])rv); } else { root = rv; } } public boolean add(Object obj) { final int tlen = nTail(); final int newCount = count+1; if(tlen == 32) { consTail(tail); tail = new Object[tailWidth]; nTail = 0; } tail[nTail++] = obj; this.count = newCount; return true; } public Object set(int idx, Object obj) { checkIndex(idx, count); int cutoff = count - nTail(); if(idx < cutoff) { Box b = new Box(null); Object newRoot = shift == 1 ? ((Leaf)root).assocN(sentinel, idx, obj, b) : ((Branch)root).assocN(sentinel, shift, idx, obj, b); root = newRoot; return b.val; } else { idx = idx - cutoff; Object rv = tail[idx]; tail[idx] = obj; return rv; } } public void setObject(int idx, Object obj) { checkIndex(idx, count); int cutoff = count - nTail(); if(idx < cutoff) { Object newRoot = shift == 1 ? ((Leaf)root).assocN(sentinel, idx, obj, null) : ((Branch)root).assocN(sentinel, shift, idx, obj, null); root = newRoot; } else { idx = idx - cutoff; tail[idx] = obj; } } public MutTreeList assocN(int i, Object val) { if(i == count) add(val); setObject(i, val); return this; } public MutTreeList pop() { if(count == 0) throw new IllegalStateException("Can't pop empty vector"); if(nTail > 0) { nTail--; } else { Object popResult = shift == 1 ? ((Leaf)root).pop(sentinel) : ((Branch)root).pop(sentinel, shift); if(popResult instanceof SublistResult) { SublistResult r = (SublistResult)popResult; this.root = r.node; nTail = tailWidth-1; System.arraycopy(tail, 0, r.tail, 0, nTail); } else { this.root = popResult; } } count--; return this; } public MutTreeList assoc(Object key, Object val) { return assocN(RT.intCast(key), val); } public MutTreeList conj(Object val) { add(val); return this; } public TreeList persistent() { if(persistent == true) throw new RuntimeException("Persistent called twice on transient vector"); persistent = true; return new TreeList( this, meta ); } public IPersistentVector immut() { return persistent(); } public static MutTreeList create(boolean owning, IPersistentMap meta, Object[] data) { int nLeaves = (data.length + tailWidth - 1) / tailWidth; MutTreeList newList = new MutTreeList(new Leaf(null, new Object[0][]), new Object[tailWidth], meta, 0, 0); for(int idx = 0; idx < nLeaves; ++idx) { int dataOff = idx*32; int dataEnd = Math.min(dataOff + 32, data.length); if(idx == (nLeaves - 1)) { System.arraycopy(data, dataOff, newList.tail, 0, dataEnd - dataOff); newList.nTail = dataEnd - dataOff; newList.count += newList.nTail; } else { newList.consTail(Arrays.copyOfRange(data, dataOff, dataEnd)); newList.count += 32; } } return newList; } } ================================================ FILE: java/ham_fisted/MutableMap.java ================================================ package ham_fisted; //Marker interface indicating that merge, compute, put, etc. are safe to use. public interface MutableMap {} ================================================ FILE: java/ham_fisted/ObjArray.java ================================================ package ham_fisted; import java.util.Arrays; import java.util.Iterator; public class ObjArray { public static final Object[] create() { return new Object[0]; } public static final Object[] create(Object a) { return new Object[] {a}; } public static final Object[] create(Object a, Object b) { return new Object[] {a, b}; } public static final Object[] create(Object a, Object b, Object c) { return new Object[] {a, b, c}; } public static final Object[] create(Object a, Object b, Object c, Object d) { return new Object[] {a, b, c, d}; } public static final Object[] create(Object a, Object b, Object c, Object d, Object e) { return new Object[] {a, b, c, d, e}; } public static final Object[] create(Object a, Object b, Object c, Object d, Object e, Object f) { return new Object[] {a, b, c, d, e, f}; } public static final Object[] createv(Object a, Object b, Object c, Object d, Object e, Object f, Object[] extra) { final int el = extra.length; final int len = 6 + el; Object[] retval = new Object[len]; retval[0] = a; retval[1] = b; retval[2] = c; retval[3] = d; retval[4] = e; retval[5] = f; System.arraycopy(extra, 0, retval, 6, el); return retval; } public static final Object[] create(Object a, Object b, Object c, Object d, Object e, Object f, Object g) { return new Object[] {a, b, c, d, e, f, g}; } public static final Object[] create(Object a, Object b, Object c, Object d, Object e, Object f, Object g, Object h) { return new Object[] {a, b, c, d, e, f, g, h}; } public static final Object[] createv(Object a, Object b, Object c, Object d, Object e, Object f, Object g, Object h, Object[] extra) { final int el = extra.length; final int len = 8 + el; Object[] retval = new Object[len]; retval[0] = a; retval[1] = b; retval[2] = c; retval[3] = d; retval[4] = e; retval[5] = f; retval[6] = g; retval[7] = h; System.arraycopy(extra, 0, retval, 8, el); return retval; } public static final Object[] create(Object a, Object b, Object c, Object d, Object e, Object f, Object g, Object h, Object i) { return new Object[] {a, b, c, d, e, f, g, h, i}; } public static final Object[] create(Object a, Object b, Object c, Object d, Object e, Object f, Object g, Object h, Object i, Object j) { return new Object[] {a, b, c, d, e, f, g, h, i, j}; } public static final Object[] create(Object a, Object b, Object c, Object d, Object e, Object f, Object g, Object h, Object i, Object j, Object k) { return new Object[] {a, b, c, d, e, f, g, h, i, j, k}; } public static final Object[] create(Object a, Object b, Object c, Object d, Object e, Object f, Object g, Object h, Object i, Object j, Object k, Object l) { return new Object[] {a, b, c, d, e, f, g, h, i, j, k, l}; } public static final Object[] create(Object a, Object b, Object c, Object d, Object e, Object f, Object g, Object h, Object i, Object j, Object k, Object l, Object m) { return new Object[] {a, b, c, d, e, f, g, h, i, j, k, l, m}; } public static final Object[] create(Object a, Object b, Object c, Object d, Object e, Object f, Object g, Object h, Object i, Object j, Object k, Object l, Object m, Object n) { return new Object[] {a, b, c, d, e, f, g, h, i, j, k, l, m, n}; } public static final Object[] create(Object a, Object b, Object c, Object d, Object e, Object f, Object g, Object h, Object i, Object j, Object k, Object l, Object m, Object n, Object o) { return new Object[] {a, b, c, d, e, f, g, h, i, j, k, l, m, n, o}; } public static final Object[] create(Object a, Object b, Object c, Object d, Object e, Object f, Object g, Object h, Object i, Object j, Object k, Object l, Object m, Object n, Object o, Object p) { return new Object[] {a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p}; } public static final Object[] createv(Object a, Object b, Object c, Object d, Object e, Object f, Object g, Object h, Object i, Object j, Object k, Object l, Object m, Object n, Object o, Object p, Object[] extra) { final int el = extra.length; final int len = 16 + el; Object[] retval = new Object[len]; retval[0] = a; retval[1] = b; retval[2] = c; retval[3] = d; retval[4] = e; retval[5] = f; retval[6] = g; retval[7] = h; retval[8] = i; retval[9] = j; retval[10] = k; retval[11] = l; retval[12] = m; retval[13] = n; retval[14] = o; retval[15] = p; System.arraycopy(extra, 0, retval, 16, el); return retval; } public static Object[] iterFill(Object[] data, int idx, Iterator iter) { while(iter.hasNext()) data[idx++] = iter.next(); return data; } } ================================================ FILE: java/ham_fisted/ParallelOptions.java ================================================ package ham_fisted; import java.util.concurrent.ExecutorService; import java.util.concurrent.ForkJoinPool; public class ParallelOptions { public final long minN; public final int maxBatchSize; public final boolean ordered; public final ExecutorService pool; public final int parallelism; public final CatParallelism catParallelism; public final int putTimeoutMs; public final int nLookahead; //In some cases users may want a sequence of unmerged results to do some //other form of merge. public final boolean unmergedResult; public enum CatParallelism { //Parallelize assuming catenation of large-N containers ELEMWISE, //Parallelize assuming large catenation of small-N containers - //default parallelism. SEQWISE, } public ParallelOptions(long _minN, int batchSize, boolean _ordered, ExecutorService _pool, int _parallelism, CatParallelism _catParallelism, int _putTimeoutMs, boolean unmergedResult, int nLookahead) { minN = _minN; maxBatchSize = batchSize; ordered = _ordered; pool = _pool; parallelism = _parallelism; catParallelism = _catParallelism; putTimeoutMs = _putTimeoutMs; this.unmergedResult = unmergedResult; this.nLookahead = nLookahead; } public ParallelOptions(long minN, int batchSize, boolean ordered) { this(minN, batchSize, ordered, ForkJoinPool.commonPool(), ForkJoinPool.getCommonPoolParallelism(), CatParallelism.SEQWISE, 5000, false, -1); } public ParallelOptions() { this(1000, 64000, true); } public ParallelOptions minN(long newMinN) { return new ParallelOptions(newMinN, maxBatchSize, ordered, pool, parallelism, catParallelism, putTimeoutMs, false, -1); } } ================================================ FILE: java/ham_fisted/PartitionByInner.java ================================================ package ham_fisted; import java.util.Iterator; import java.util.NoSuchElementException; import java.util.function.BiPredicate; import clojure.lang.IFn; import clojure.lang.Seqable; import clojure.lang.ISeq; import clojure.lang.IDeref; import clojure.lang.Util; import clojure.lang.IMeta; import clojure.lang.RT; import clojure.lang.IPersistentMap; import clojure.lang.PersistentArrayMap; import clojure.lang.Keyword; public class PartitionByInner implements ITypedReduce, Iterator, Seqable, IDeref, IMeta { public final Iterator iter; public final IFn f; public final Object fv; public final BiPredicate pred; boolean lastVValid; Object lastV; Object lastFV; public PartitionByInner(Iterator i, IFn f, Object v, BiPredicate pred) { this.iter = i; this.f = f; this.lastV = v; this.lastVValid = true; final Object fv = f.invoke(v); this.fv = fv; this.lastFV = fv; this.pred = pred == null ? (x,y)->CljHash.equiv(x,y) : pred; } Object advance() { Object rv = lastV; if (iter.hasNext()) { lastVValid = true; Object vv= iter.next(); lastV = vv; lastFV = f.invoke(vv); } else { lastVValid = false; lastV = null; lastFV = null; } return rv; } @SuppressWarnings("unchecked") public Object reduce(IFn rfn, Object acc) { final Iterator iter = this.iter; final IFn ff = f; final Object ffv = fv; if(!hasNext()) return acc; Object vv = lastV; Object fvv = lastFV; do { acc = rfn.invoke(acc, vv); if(RT.isReduced(acc)) { advance(); return ((IDeref)acc).deref(); } if(!iter.hasNext()) { advance(); return acc; } vv = iter.next(); fvv = ff.invoke(vv); } while(ffv == fvv || pred.test(ffv, fvv)); lastVValid = true; lastV = vv; lastFV = fvv; return acc; } @SuppressWarnings("unchecked") public boolean hasNext() { return lastVValid && (fv == lastFV || pred.test(fv, lastFV)); } public Object next() { if(!lastVValid) throw new NoSuchElementException(); return advance(); } public ISeq seq() { return LazyChunkedSeq.chunkIteratorSeq(this); } public Object deref() { if(hasNext()) reduce(new IFnDef() { public Object invoke(Object acc, Object v) { return v; } }, null); return lastVValid ? ImmutList.create(true, null, lastV, lastFV) : null; } public IPersistentMap meta() { return new PersistentArrayMap(new Object[] { Keyword.intern("fv"), fv }); } } ================================================ FILE: java/ham_fisted/PersistentHashMap.java ================================================ package ham_fisted; import java.util.Map; import java.util.Set; import java.util.Collection; import java.util.function.BiFunction; import java.util.function.Function; import clojure.lang.IEditableCollection; import clojure.lang.IPersistentMap; import clojure.lang.ITransientMap; import clojure.lang.IObj; import clojure.lang.Indexed; public class PersistentHashMap extends ROHashMap implements IAPersistentMap, IObj { public static final PersistentHashMap EMPTY = new PersistentHashMap(new HashMap()); int _hasheq = 0; PersistentHashMap(HashMap data) { super(data.loadFactor, data.capacity, data.length, data.data, data.meta); } public PersistentHashMap(HashMap data, boolean marker) { super(data.loadFactor, data.capacity, data.length, data.data.clone(), data.meta); } PersistentHashMap(HashMap data, IPersistentMap m) { super(data.loadFactor, data.capacity, data.length, data.data, m); } public int count() { return length; } public int hasheq() { if (_hasheq == 0) _hasheq = super.hasheq(); return _hasheq; } public PersistentHashMap assoc(Object key, Object val) { final int hashcode = hash(key); final int idx = hashcode & mask; HashNode e, init = data[idx]; Object k; for(e = init; e != null && !((k = e.k) == key || equals(k, key)); e = e.nextNode); if(e != null) { if(e.v == val) return this; PersistentHashMap rv = new PersistentHashMap(this, true); // We use e.k here for identity short circuiting in HashNode assoc rv.data[idx] = init.assoc(rv, e.k, hashcode, val); return rv; } else { PersistentHashMap rv = new PersistentHashMap(this, true); HashNode newNode = rv.newNode(key, hashcode, val); rv.data[idx] = newNode; newNode.nextNode = init; rv.checkResize(null); return rv; } } public IPersistentMap without(Object key) { final int hashcode = hash(key); final int idx = hashcode & mask; HashNode e, init = data[idx]; Object k; for(e = init; e != null && !((k = e.k) == key || equals(k, key)); e = e.nextNode); if(e == null) return this; PersistentHashMap rv = new PersistentHashMap(this, true); rv.data[idx] = init.dissoc(rv, key); return rv; } public ITransientMap asTransient() { return isEmpty() ? new UnsharedHashMap(meta) : new TransientHashMap(this); } public PersistentHashMap withMeta(IPersistentMap m) { return new PersistentHashMap(this, m); } public PersistentHashMap empty() { return EMPTY; } public PersistentHashMap union(Map o, BiFunction bfn) { return new PersistentHashMap(union(shallowClone(), o, bfn)); } public PersistentHashMap intersection(Map o, BiFunction bfn) { return new PersistentHashMap(intersection(shallowClone(), o, bfn)); } public PersistentHashMap intersection(Set o) { return new PersistentHashMap(intersection(shallowClone(), o)); } public PersistentHashMap difference(Collection c) { return new PersistentHashMap(difference(shallowClone(), c)); } public PersistentHashMap updateValues(BiFunction valueMap) { return new PersistentHashMap(updateValues(shallowClone(), valueMap)); } public PersistentHashMap updateValue(Object k, Function fn) { return new PersistentHashMap(updateValue(this, fn)); } } ================================================ FILE: java/ham_fisted/PersistentHashSet.java ================================================ package ham_fisted; import java.util.Set; import java.util.Collection; import clojure.lang.IPersistentMap; import clojure.lang.ITransientSet; import clojure.lang.IObj; public class PersistentHashSet extends ROHashSet implements IAPersistentSet, IObj { public static final PersistentHashSet EMPTY = new PersistentHashSet(); public PersistentHashSet() { super(); } public PersistentHashSet(HashBase hb) { super(hb); } public PersistentHashSet(HashBase hb, IPersistentMap meta) { super(hb,meta); } public PersistentHashSet withMeta(IPersistentMap m) { return new PersistentHashSet(this, m); } public ITransientSet asTransient() { return isEmpty() ? new UnsharedHashSet(meta) : new TransientHashSet(this, meta); } public PersistentHashSet empty() { return EMPTY; } public PersistentHashSet union(Collection rhs) { return new PersistentHashSet(union(shallowClone(), rhs)); } public PersistentHashSet intersection(Set rhs) { return new PersistentHashSet(intersection(shallowClone(), rhs)); } public PersistentHashSet difference(Set rhs) { return new PersistentHashSet(difference(shallowClone(), rhs)); } } ================================================ FILE: java/ham_fisted/PersistentLongHashMap.java ================================================ package ham_fisted; import java.util.function.Function; import java.util.function.BiFunction; import java.util.Collection; import java.util.Set; import java.util.Map; import clojure.lang.IEditableCollection; import clojure.lang.IPersistentMap; import clojure.lang.ITransientMap; import clojure.lang.IObj; import clojure.lang.Indexed; public class PersistentLongHashMap extends ROLongHashMap implements IAPersistentMap, IObj { public static final PersistentLongHashMap EMPTY = new PersistentLongHashMap(new LongHashMap()); int _hasheq = 0; public PersistentLongHashMap(LongHashMap data) { super(data.loadFactor, data.capacity, data.length, data.data, data.meta); } public PersistentLongHashMap(LongHashMap data, IPersistentMap m) { super(data.loadFactor, data.capacity, data.length, data.data, m); } public int count() { return length; } public int hasheq() { if (_hasheq == 0) _hasheq = super.hasheq(); return _hasheq; } public ITransientMap asTransient() { return isEmpty() ? new UnsharedLongHashMap(meta) : new TransientLongHashMap(this); } public PersistentLongHashMap withMeta(IPersistentMap m) { return new PersistentLongHashMap(this, m); } public PersistentLongHashMap empty() { return EMPTY; } public PersistentLongHashMap union(Map o, BiFunction bfn) { return new PersistentLongHashMap(union(shallowClone(), o, bfn)); } public PersistentLongHashMap intersection(Map o, BiFunction bfn) { return new PersistentLongHashMap(intersection(shallowClone(), o, bfn)); } public PersistentLongHashMap intersection(Set o) { return new PersistentLongHashMap(intersection(shallowClone(), o)); } public PersistentLongHashMap difference(Collection c) { return new PersistentLongHashMap(difference(shallowClone(), c)); } public PersistentLongHashMap updateValues(BiFunction valueMap) { return new PersistentLongHashMap(updateValues(shallowClone(), valueMap)); } public PersistentLongHashMap updateValue(Object k, Function fn) { return new PersistentLongHashMap(updateValue(this, fn)); } } ================================================ FILE: java/ham_fisted/PersistentVector.java ================================================ package ham_fisted; import java.util.Arrays; public class PersistentVector { } ================================================ FILE: java/ham_fisted/ROHashMap.java ================================================ package ham_fisted; import java.util.Map; import java.util.function.Function; import java.util.function.BiFunction; import clojure.lang.IPersistentMap; import clojure.lang.IMapEntry; import clojure.lang.MapEntry; public class ROHashMap extends HashMap { ROHashMap(float loadFactor, int initialCapacity, int length, HashNode[] data, IPersistentMap meta) { super(loadFactor, initialCapacity, length, data, meta); } ROHashMap(HashMap other, IPersistentMap m) { super(other, m); } public Object remove(Object k) { throw new UnsupportedOperationException(); } public Object put(Object key, Object value) { throw new UnsupportedOperationException(); } public void putAll(Map m) { throw new UnsupportedOperationException(); } public void clear() { throw new UnsupportedOperationException(); } public Object compute(Object key, BiFunction bfn) { throw new UnsupportedOperationException(); } public Object computeIfAbsent(Object key, Function mappingFunction) { throw new UnsupportedOperationException(); } public Object computeIfPresent(Object key, BiFunction remappingFunction) { throw new UnsupportedOperationException(); } public Object merge(Object key, Object value, BiFunction remappingFunction) { throw new UnsupportedOperationException(); } public void replaceAll(BiFunction function) { throw new UnsupportedOperationException(); } } ================================================ FILE: java/ham_fisted/ROHashSet.java ================================================ package ham_fisted; import java.util.Collection; import clojure.lang.IPersistentMap; public class ROHashSet extends HashSet { public ROHashSet() { super(); } public ROHashSet(HashBase hs) { this(hs, null); } public ROHashSet(HashBase hs, IPersistentMap meta) { super(hs, meta); } public boolean add(Object o) { throw new UnsupportedOperationException(); } public boolean addAll(Collection c) { throw new UnsupportedOperationException(); } public boolean remove(Object o) { throw new UnsupportedOperationException(); } } ================================================ FILE: java/ham_fisted/ROLongHashMap.java ================================================ package ham_fisted; import java.util.Map; import java.util.function.Function; import java.util.function.BiFunction; import clojure.lang.IPersistentMap; import clojure.lang.IMapEntry; import clojure.lang.MapEntry; public class ROLongHashMap extends LongHashMap { ROLongHashMap(float loadFactor, int initialCapacity, int length, LongHashNode[] data, IPersistentMap meta) { super(loadFactor, initialCapacity, length, data, meta); } ROLongHashMap(LongHashMap other, IPersistentMap m) { super(other, m); } public Object remove(Object k) { throw new UnsupportedOperationException(); } public Object put(Object key, Object value) { throw new UnsupportedOperationException(); } public void putAll(Map m) { throw new UnsupportedOperationException(); } public void clear() { throw new UnsupportedOperationException(); } public Object compute(Object key, BiFunction bfn) { throw new UnsupportedOperationException(); } public Object computeIfAbsent(Object key, Function mappingFunction) { throw new UnsupportedOperationException(); } public Object computeIfPresent(Object key, BiFunction remappingFunction) { throw new UnsupportedOperationException(); } public Object merge(Object key, Object value, BiFunction remappingFunction) { throw new UnsupportedOperationException(); } public void replaceAll(BiFunction function) { throw new UnsupportedOperationException(); } } ================================================ FILE: java/ham_fisted/RandomAccessSpliterator.java ================================================ package ham_fisted; import java.util.Spliterator; import java.util.List; import java.util.RandomAccess; import java.util.function.Consumer; import java.util.function.DoubleConsumer; import java.util.function.LongConsumer; public class RandomAccessSpliterator implements Spliterator { protected final List list; protected final int sidx; protected int eidx; int curIdx; RandomAccessSpliterator(List list, int sidx, int eidx) { assert list instanceof RandomAccess; this.list = list; this.sidx = sidx; this.eidx = eidx; curIdx = sidx; } RandomAccessSpliterator(List list) { this(list, 0, list.size()); } protected Spliterator construct(List list, int split, int eidx) { return new RandomAccessSpliterator(list, split, eidx); } protected Spliterator doSplit() { final int ne = eidx - sidx; if(ne > 1) { int split = sidx + (ne / 2); int eeidx = eidx; eidx = split; return construct(list, split, eeidx); } return null; } public Spliterator trySplit() { return doSplit(); } public long estimateSize() { return eidx - sidx; } public boolean tryAdvance(Consumer action) { if (action == null) throw new NullPointerException(); final boolean retval = curIdx < eidx; if(retval) { action.accept(list.get(curIdx)); ++curIdx; } return retval; } public void forEachRemaining(Consumer c) { final int ee = eidx; if(list instanceof IMutList) { final IMutList ll = (IMutList)list; if(c instanceof DoubleConsumer) { final DoubleConsumer dc = (DoubleConsumer)c; for(int cc = curIdx; cc < ee; ++cc) { dc.accept(ll.getDouble(cc)); } curIdx = eidx; return; } else if (c instanceof LongConsumer) { final LongConsumer dc = (LongConsumer)c; for(int cc = curIdx; cc < ee; ++cc) { dc.accept(ll.getLong(cc)); } curIdx = eidx; return; } } final List ll = list; for(int cc = curIdx; cc < ee; ++cc) { c.accept(ll.get(cc)); } curIdx = eidx; } public int characteristics() { return Spliterator.ORDERED | Spliterator.SIZED | Spliterator.SUBSIZED | Spliterator.IMMUTABLE; } public static class LongSpliterator extends RandomAccessSpliterator implements Spliterator.OfLong { @SuppressWarnings("unchecked") public LongSpliterator(IMutList list, int sidx, int eidx) { super(list, sidx, eidx); } protected Spliterator construct(List list, int sidx, int eidx) { return new LongSpliterator((IMutList)list, sidx, eidx); } public Spliterator.OfLong trySplit() { return (Spliterator.OfLong) doSplit(); } public boolean tryAdvance(LongConsumer action) { if (action == null) throw new NullPointerException(); final boolean retval = curIdx < eidx; if(retval) { action.accept(((IMutList)list).getLong(curIdx)); ++curIdx; } return retval; } public void forEachRemaining(LongConsumer dc) { final int ee = eidx; final IMutList ll = (IMutList)list; for(int cc = curIdx; cc < ee; ++cc) { dc.accept(ll.getLong(cc)); } curIdx = eidx; } } public static class DoubleSpliterator extends RandomAccessSpliterator implements Spliterator.OfDouble { @SuppressWarnings("unchecked") public DoubleSpliterator(IMutList list, int sidx, int eidx) { super(list, sidx, eidx); } protected Spliterator construct(List list, int sidx, int eidx) { return new DoubleSpliterator((IMutList)list, sidx, eidx); } public Spliterator.OfDouble trySplit() { return (Spliterator.OfDouble) doSplit(); } public boolean tryAdvance(DoubleConsumer action) { if (action == null) throw new NullPointerException(); final boolean retval = curIdx < eidx; if(retval) { action.accept(((IMutList)list).getDouble(curIdx)); ++curIdx; } return retval; } public void forEachRemaining(DoubleConsumer dc) { final int ee = eidx; final IMutList ll = (IMutList)list; for(int cc = curIdx; cc < ee; ++cc) { dc.accept(ll.getDouble(cc)); } curIdx = eidx; } } } ================================================ FILE: java/ham_fisted/RangeList.java ================================================ package ham_fisted; import java.util.List; public interface RangeList { public void fillRange(long startidx, long endidx, Object v); default void fillRange(long startidx, List v) { fillRangeReducible(startidx, v); } public void fillRangeReducible(long startidx, Object v); public void removeRange(long startidx, long endidx); } ================================================ FILE: java/ham_fisted/Ranges.java ================================================ package ham_fisted; import java.util.Random; import java.util.List; import java.util.function.LongConsumer; import java.util.function.LongBinaryOperator; import clojure.lang.RT; import clojure.lang.IPersistentMap; import clojure.lang.IFn; import clojure.lang.IDeref; import clojure.lang.Util; import clojure.lang.Reduced; import clojure.lang.ISeq; public class Ranges { public static class LongRange implements LongMutList, TypedList { public final long start; public final long end; public final long step; public final long nElems; public final IPersistentMap meta; int _hash = 0; public LongRange(long s, long e, long _step, IPersistentMap m) { start = s; end = e; step = _step; nElems = (e - s)/_step; if (nElems < 0) throw new RuntimeException("Invalid Range - start: " + String.valueOf(s) + " end: " + String.valueOf(e) + " step: " + String.valueOf(_step)); meta = m; } public IMutList cloneList() { return this; } public boolean equals(Object other) { return equiv(other); } public int hashCode() { return hasheq(); } public int hasheq() { if (_hash == 0) { _hash = LongMutList.super.hasheq(); } return _hash; } public String toString() { return Transformables.sequenceToString(this); } public Class containedType() { return Long.TYPE; } public int size() { return RT.intCast(nElems); } public long lgetLong(long idx) { final long sz = nElems; if(idx < 0 || idx >= sz) throw new IndexOutOfBoundsException("Index out of range: " + String.valueOf(idx) + " size: " + String.valueOf(sz)); return start + step*idx; } public long getLong(int idx) { return lgetLong(idx); } public int[] toIntArray() { return ArrayLists.iarange(RT.intCast(start), RT.intCast(end), RT.intCast(step)); } public long[] toLongArray() { return ArrayLists.larange(start, end, step); } public double[] toDoubleArray() { return ArrayLists.darange(start, end, step); } public Object[] toArray() { final int sz = size(); final Object[] retval = new Object[sz]; long st = start; final long se = step; for(int idx = 0; idx < sz; ++idx, st += se) retval[idx] = st; return retval; } public LongMutList subList(long sidx, long eidx) { ChunkedList.sublistCheck(sidx, eidx, nElems); return new LongRange(start + sidx*step, start + eidx*step, step, meta); } public ISeq seq() { return inplaceSublistSeq(); } public LongMutList subList(int sidx, int eidx) { return subList((long)sidx, (long)eidx); } public Object reduce(final IFn fn, Object init) { if(nElems == 0) return init; Object acc = init; long n = nElems; long i = start; if(!(fn instanceof IFn.OLO)) do { acc = fn.invoke(acc, i); if (RT.isReduced(acc)) return ((Reduced)acc).deref(); i += step; n--; } while(n > 0); else { final IFn.OLO ff = (IFn.OLO)fn; do { acc = ff.invokePrim(acc, i); if (RT.isReduced(acc)) return ((Reduced)acc).deref(); i += step; n--; } while(n > 0); } return acc; } public Object parallelReduction(IFn initValFn, IFn rfn, IFn mergeFn, ParallelOptions options) { return Reductions.parallelIndexGroupReduce(new IFnDef.LLO() { public Object invokePrim(long sidx, long eidx) { return Reductions.serialReduction(rfn, initValFn.invoke(), subList(sidx, eidx)); } }, nElems, mergeFn, options); } public Object lnth(long idx) { final long sz = nElems; if (idx < 0) idx = idx + sz; return lgetLong(idx); } public Object lnth(long idx, Object notFound) { final long sz = nElems; if (idx < 0) idx = idx + sz; return idx < sz && idx > -1 ? lgetLong(idx) : notFound; } public Object invoke(Object arg) { return lnth(Casts.longCast(arg)); } public Object invoke(Object arg, Object defVal) { if(Util.isInteger(arg)) { return lnth(Casts.longCast(arg), defVal); } else { return defVal; } } public IPersistentMap meta() { return meta; } public LongRange withMeta(IPersistentMap m) { return new LongRange(start, end, step, m); } }; public static class DoubleRange implements DoubleMutList, TypedList { public final double start; public final double end; public final double step; public final long nElems; public final IPersistentMap meta; int _hash = 0; public DoubleRange(double s, double e, double _step, IPersistentMap _meta) { start = s; end = e; step = _step; meta = _meta; //Floor to long intentional nElems = Math.max(0, (long)((e - s)/_step)); if (nElems < 0) throw new IndexOutOfBoundsException("Invalid Range - start: " + String.valueOf(s) + " end: " + String.valueOf(e) + " step: " + String.valueOf(_step)); } public IMutList cloneList() { return this; } public boolean equals(Object other) { return equiv(other); } public int hashCode() { return hasheq(); } public int hasheq() { if (_hash == 0) { _hash = DoubleMutList.super.hasheq(); } return _hash; } public String toString() { return Transformables.sequenceToString(this); } public Class containedType() { return Double.TYPE; } public int size() { return RT.intCast(nElems); } public double lgetDouble(long idx) { final long sz = nElems; if(idx < 0) idx += sz; if(idx < 0 || idx >= sz) throw new IndexOutOfBoundsException("Index out of range: " + String.valueOf(idx) + " size: " + String.valueOf(sz)); return start + step*idx; } public ISeq seq() { return inplaceSublistSeq(); } public double getDouble(int idx) { return lgetDouble(idx); } public int[] toIntArray() { final int st = RT.intCast(step); if (st != 0) return ArrayLists.iarange(RT.intCast(start), RT.intCast(end), RT.intCast(step)); throw new IndexOutOfBoundsException("Infinite range: " + String.valueOf(step) + " : " + String.valueOf(st)); } public long[] toLongArray() { final long st = Casts.longCast(step); if (st != 0) return ArrayLists.larange(Casts.longCast(start), Casts.longCast(end), Casts.longCast(step)); throw new IndexOutOfBoundsException("Infinite range: " + String.valueOf(step) + " : " + String.valueOf(st)); } public double[] toDoubleArray() { return ArrayLists.darange(start, end, step); } public DoubleMutList subList(long sidx, long eidx) { ChunkedList.sublistCheck(sidx, eidx, size()); return new DoubleRange(start + sidx*step, start + eidx*step, step, meta); } public DoubleMutList subList(int sidx, int eidx) { return subList((long)sidx, (long)eidx); } public Object reduce(final IFn fn, Object init) { final IFn.ODO rrfn = fn instanceof IFn.ODO ? (IFn.ODO)fn : new IFn.ODO() { public Object invokePrim(Object lhs, double v) { return fn.invoke(lhs, v); } }; return doubleReduction(rrfn, init); } public Object doubleReduction(IFn.ODO op, Object init) { final long sz = nElems; final double se = step; final double st = start; for(long idx = 0; idx < sz && !RT.isReduced(init); ++idx) init = op.invokePrim(init, st + (se*idx)); return Reductions.unreduce(init); } public Object parallelReduction(IFn initValFn, IFn rfn, IFn mergeFn, ParallelOptions options) { if(nElems > options.minN) { return Reductions.parallelIndexGroupReduce(new IFnDef.LLO() { public Object invokePrim(long sidx, long eidx) { return Reductions.serialReduction(rfn, initValFn.invoke(), subList(sidx, eidx)); } }, nElems, mergeFn, options); } else { return Reductions.serialReduction(rfn, initValFn.invoke(), this); } } public Object lnth(long idx) { final long sz = nElems; if (idx < 0) idx = idx + sz; return lgetDouble(idx); } public Object lnth(long idx, Object notFound) { final long sz = nElems; if (idx < 0) idx = idx + sz; return idx < sz && idx > -1 ? lgetDouble(idx) : notFound; } public Object invoke(Object arg) { return lnth(Casts.longCast(arg)); } public Object invoke(Object arg, Object defVal) { if(Util.isInteger(arg)) { return lnth(Casts.longCast(arg), defVal); } else { return defVal; } } public IPersistentMap meta() { return meta; } public DoubleRange withMeta(IPersistentMap m) { return new DoubleRange(start, end, step, m); } }; } ================================================ FILE: java/ham_fisted/Reducible.java ================================================ package ham_fisted; import java.util.Iterator; public interface Reducible { default Reducible reduceIter(Iterator rest) { Reducible retval = this; while(rest.hasNext()) retval = retval.reduce(rest.next()); return retval; } Reducible reduce(Reducible rhs); } ================================================ FILE: java/ham_fisted/Reductions.java ================================================ package ham_fisted; import clojure.lang.IFn; import clojure.lang.RT; import clojure.lang.IReduceInit; import clojure.lang.IDeref; import clojure.lang.Seqable; import clojure.java.api.Clojure; import clojure.lang.Delay; import clojure.lang.AFn; import clojure.lang.Util; import java.util.RandomAccess; import java.util.List; import java.util.Iterator; import java.util.Map; import java.util.Set; import java.util.Spliterator; import java.util.Collection; import java.util.function.Consumer; import java.util.function.DoubleConsumer; import java.util.function.LongConsumer; import java.util.concurrent.Future; import java.util.concurrent.ForkJoinPool; import java.util.concurrent.Callable; import java.util.concurrent.ArrayBlockingQueue; public class Reductions { public static IFn ToStringPrint = new AFn() { public Object invoke(Object target, Object ww) { try { ((java.io.Writer)ww).write(target.toString()); }catch (java.io.IOException e) { throw Util.sneakyThrow(e); } return null; } }; public static Object unreduce(Object obj) { return RT.isReduced(obj) ? ((IDeref)obj).deref() : obj; } public interface DoubleAccum extends IFnDef.ODO {} public interface LongAccum extends IFnDef.OLO {} public static Iterable toIterable(Object obj) { if( obj == null) return null; if( obj instanceof Iterable) return (Iterable)obj; if( obj instanceof Map) return ((Map)obj).entrySet(); if( obj.getClass().isArray()) return ArrayLists.toList(obj); if( obj instanceof String) return new StringCollection((String)obj); return (Iterable)RT.seq(obj); } public static Object iterReduce(Object obj, IFn fn) { if(obj == null) return fn.invoke(); final Iterator it = toIterable(obj).iterator(); Object init = it.hasNext() ? it.next() : null; while(it.hasNext() && !RT.isReduced(init)) { init = fn.invoke(init, it.next()); } return unreduce(init); } public static Object iterReduce(Object obj, Object init, IFn fn) { if(obj == null) return init; final Iterator it = toIterable(obj).iterator(); while(it.hasNext() && !RT.isReduced(init)) { init = fn.invoke(init, it.next()); } return unreduce(init); } public static Reducible reduceReducibles(Iterable data) { Iterator iter = data.iterator(); Reducible initial = iter.hasNext() ? iter.next() : null; return initial.reduceIter(iter); } static final Delay collReducePtr = new Delay(new IFnDef() { public Object invoke() { return ((IDeref)Clojure.var("clojure.core.protocols", "coll-reduce")).deref(); } }); public static Object serialReduction(IFn rfn, Object init, Object coll) { if( coll == null) return init; if( coll instanceof IReduceInit) { return ((IReduceInit)coll).reduce(rfn, init); } return ((IFn)(collReducePtr.deref())).invoke(coll, rfn, init); } @SuppressWarnings("unchecked") public static Consumer serialReduction(Consumer c, Object coll) { serialReduction(new IFnDef() { public Object invoke(Object acc, Object v) { c.accept(v); return c; } }, c, coll); return c; } public static Object iterableMerge(ParallelOptions options, IFn mergeFn, final Iterable groups) { if(options.unmergedResult) return groups; final Iterator giter = groups.iterator(); Object initObj = giter.hasNext() ? giter.next() : null; while(giter.hasNext()) initObj = mergeFn.invoke(initObj, giter.next()); return initObj; } public static Object parallelIndexGroupReduce(IFn groupFn, long nElems, IFn mergeFn, ParallelOptions options) { return iterableMerge(options, mergeFn, ForkJoinPatterns.parallelIndexGroups(nElems, groupFn, options)); } public static Object parallelRandAccessReduction(IFn initValFn, IFn rfn, IFn mergeFn, List l, ParallelOptions options) { return parallelIndexGroupReduce( new IFnDef.LLO() { public Object invokePrim(long sidx, long eidx) { return serialReduction(rfn, initValFn.invoke(), l.subList(RT.intCast(sidx), RT.intCast(eidx))); } }, l.size(), mergeFn, options); } public static class ReduceConsumer implements Consumer, IDeref { Object init; public final IFn rfn; public static class DoubleReduceConsumer extends ReduceConsumer implements DoubleConsumer { public final IFn.ODO dfn; public DoubleReduceConsumer(Object in, IFn _rfn) { super(in, _rfn); dfn = (IFn.ODO)_rfn; } public void accept(Object obj) { accept(Casts.doubleCast(obj)); } public void accept(double v) { if(!isReduced()) init = dfn.invokePrim(init, v); } } public static class LongReduceConsumer extends ReduceConsumer implements LongConsumer { public final IFn.OLO dfn; public LongReduceConsumer(Object in, IFn _rfn) { super(in, _rfn); dfn = (IFn.OLO)_rfn; } public void accept(Object obj) { accept(Casts.longCast(obj)); } public void accept(long v) { if(!isReduced()) init = dfn.invokePrim(init, v); } } public ReduceConsumer(Object in, IFn _rfn) { init = in; rfn = _rfn; } public void accept(Object obj) { if(!isReduced()) init = rfn.invoke(init, obj); } public boolean isReduced() { return RT.isReduced(init); } public Object deref() { return init; } public static ReduceConsumer create(Object init, IFn rfn) { if(rfn instanceof IFn.ODO) { return new DoubleReduceConsumer(init, rfn); } else if (rfn instanceof IFn.OLO) { return new LongReduceConsumer(init, rfn); } else { return new ReduceConsumer(init,rfn); } } } public static Object parallelCollectionReduction(IFn initValFn, IFn rfn, IFn mergeFn, Collection coll, ParallelOptions options ) { if(coll.size() <= options.minN) return serialReduction(rfn, initValFn.invoke(), coll); return ForkJoinPatterns.parallelSpliteratorReduce(initValFn, rfn, mergeFn, coll.spliterator(), options); } public static Delay preducePtr = new Delay( new IFnDef() { public Object invoke() { return ((IDeref)Clojure.var("ham-fisted.protocols", "preduce")).deref(); } }); public static Object parallelReduction(IFn initValFn, IFn rfn, IFn mergeFn, Object coll, ParallelOptions options) { if(coll == null) return options.unmergedResult ? ArrayImmutList.create( true, null, initValFn.invoke() ) : initValFn.invoke(); if(options.parallelism < 2) { return serialParallelReduction(initValFn, rfn, options, coll); } //Delegate back to clojure here so we can use a protocol to dispatch the //parallel reduction. return ((IFn)preducePtr.deref()).invoke(coll, initValFn, rfn, mergeFn, options); } public static Object serialParallelReduction(IFn initValFn, IFn rfn, ParallelOptions options, Object coll) { final Object retval = serialReduction(rfn, initValFn.invoke(), coll); return options.unmergedResult ? ArrayImmutList.create( true, null, retval ) : retval; } public static class IndexedDoubleAccum implements IFnDef.ODO { long idx; final IFn.OLDO rfn; public IndexedDoubleAccum(long sidx, IFn.OLDO rfn) { this.rfn = rfn; this.idx = sidx; } public IndexedDoubleAccum(IFn.OLDO rfn) { this(0, rfn); } public Object invokePrim(Object acc, double v) { return rfn.invokePrim(acc, idx++, v); } } public static class IndexedLongAccum implements IFnDef.OLO { long idx; final IFn.OLLO rfn; public IndexedLongAccum(long sidx, IFn.OLLO rfn) { this.rfn = rfn; this.idx = sidx; } public IndexedLongAccum(IFn.OLLO rfn) { this(0, rfn); } public Object invokePrim(Object acc, long v) { return rfn.invokePrim(acc, idx++, v); } } public static class IndexedAccum implements IFnDef { long idx; final IFn.OLOO rfn; public IndexedAccum(long sidx, IFn.OLOO rfn) { this.rfn = rfn; this.idx = sidx; } public IndexedAccum(IFn.OLOO rfn) { this(0, rfn); } public Object invoke(Object acc, Object v) { return rfn.invokePrim(acc, idx++, v); } } static class LongCompose implements IFnDef.OLO { final int nVals; final IFn.OLO[] rfns; public LongCompose(int nVals, Object[] rfns) { this.nVals = nVals; this.rfns = new IFn.OLO[rfns.length]; for(int idx = 0; idx < rfns.length; ++idx) { this.rfns[idx] = Transformables.toLongReductionFn(rfns[idx]); } } public Object invokePrim(Object acc, long val) { final int nv = nVals; final Object[] objs = (Object[])acc; final IFn.OLO[] rfs = this.rfns; for(int idx = 0; idx < nv; ++idx) objs[idx] = rfs[idx].invokePrim(objs[idx], val); return objs; } } public static IFn longCompose(final int nVals, final Object[] rfns) { return new LongCompose(nVals, rfns); } static class DoubleCompose implements IFnDef.ODO { final int nVals; final IFn.ODO[] rfns; public DoubleCompose(int nVals, Object[] rfns) { this.nVals = nVals; this.rfns = new IFn.ODO[rfns.length]; for(int idx = 0; idx < rfns.length; ++idx) { this.rfns[idx] = Transformables.toDoubleReductionFn(rfns[idx]); } } public Object invokePrim(Object acc, double val) { final int nv = nVals; final Object[] objs = (Object[])acc; final IFn.ODO[] rfs = this.rfns; for(int idx = 0; idx < nv; ++idx) objs[idx] = rfs[idx].invokePrim(objs[idx], val); return objs; } } public static IFn doubleCompose(final int nVals, final Object[] rfns) { return new DoubleCompose(nVals, rfns); } static class ObjCompose implements IFnDef { final int nVals; final IFn[] rfns; public ObjCompose(int nVals, Object[] rfns) { this.nVals = nVals; this.rfns = new IFn[rfns.length]; for(int idx = 0; idx < rfns.length; ++idx) { this.rfns[idx] = (IFn)rfns[idx]; } } public Object invoke(Object acc, Object val) { final int nv = nVals; final Object[] objs = (Object[])acc; final IFn[] rfs = this.rfns; for(int idx = 0; idx < nv; ++idx) objs[idx] = rfs[idx].invoke(objs[idx], val); return objs; } } public static IFn objCompose(final int nVals, final Object[] rfns) { return new ObjCompose(nVals, rfns); } public static IFn mergeCompose(final int nVals, final Object[] mergeFns) { return new IFnDef() { public Object invoke(Object lhs, Object rhs) { final Object[] l = (Object[])lhs; final Object[] r = (Object[])rhs; for(int idx = 0; idx < nVals; ++idx) { l[idx] = ((IFn)mergeFns[idx]).invoke(l[idx], r[idx]); } return l; } }; } public static Object longSamplerReduction(IFn rfn, Object acc, IFn.LL sampler, long nElems) { final IFn.OLO rrfn = Transformables.toLongReductionFn(rfn); for(long idx = 0; idx < nElems; ++idx) { acc = rrfn.invokePrim(acc, sampler.invokePrim(idx)); if(RT.isReduced(acc)) return ((IDeref)acc).deref(); } return acc; } public static Object doubleSamplerReduction(IFn rfn, Object acc, IFn.LD sampler, long nElems) { final IFn.ODO rrfn = Transformables.toDoubleReductionFn(rfn); for(long idx = 0; idx < nElems; ++idx) { acc = rrfn.invokePrim(acc, sampler.invokePrim(idx)); if(RT.isReduced(acc)) return ((IDeref)acc).deref(); } return acc; } public static Object samplerReduction(IFn rfn, Object acc, IFn sampler, long nElems) { for(long idx = 0; idx < nElems; ++idx) { acc = rfn.invoke(acc, sampler.invoke(idx)); if(RT.isReduced(acc)) return ((IDeref)acc).deref(); } return acc; } } ================================================ FILE: java/ham_fisted/ReindexList.java ================================================ package ham_fisted; import java.util.Arrays; import java.util.List; import java.util.Comparator; import it.unimi.dsi.fastutil.ints.IntComparator; import clojure.lang.IPersistentMap; public class ReindexList implements IMutList, TypedList { final int[] indexes; final List data; final IPersistentMap meta; static int checkIndex(final int idx, final int dlen) { if(idx < 0 || idx >= dlen) throw new RuntimeException("Index out of range: " + String.valueOf(idx)); return idx; } public static ReindexList create(int[] idx, List d, IPersistentMap m) { if(d instanceof IMutList) return new MutReindexList(idx, (IMutList)d, m); return new ReindexList(idx, d, m); } public ReindexList(int[] idx, List d, IPersistentMap m) { indexes = idx; data = d; meta = m; } public Class containedType() { return data instanceof TypedList ? ((TypedList)data).containedType() : null; } public int size() { return indexes.length; } public Object get(int idx) { checkIndex(idx, indexes.length); return data.get(indexes[idx]); } @SuppressWarnings("unchecked") public Object set(int idx, Object nv) { checkIndex(idx, indexes.length); return data.set(indexes[idx], nv); } public ReindexList subList(int sidx, int eidx) { return ReindexList.create(Arrays.copyOfRange(indexes, sidx, eidx), data, meta); } @SuppressWarnings("unchecked") public IntComparator indexComparator() { if(data instanceof IMutList) { final IMutList d = (IMutList)data; IntComparator srcComparator = d.indexComparator(); return new IntComparator() { public int compare(int lidx, int ridx) { return srcComparator.compare(indexes[lidx], indexes[ridx]); } }; } return new IntComparator() { public int compare(int lidx, int ridx) { return ((Comparable)(data.get(indexes[lidx]))).compareTo(data.get(indexes[ridx])); } }; } @SuppressWarnings("unchecked") public IntComparator indexComparator(Comparator comp) { if (comp == null) return indexComparator(); if (data instanceof IMutList) { final IMutList d = (IMutList)data; IntComparator srcComparator = d.indexComparator(comp); return new IntComparator() { public int compare(int lidx, int ridx) { return srcComparator.compare(indexes[lidx], indexes[ridx]); } }; } return new IntComparator() { public int compare(int lidx, int ridx) { return comp.compare(data.get(indexes[lidx]), data.get(indexes[ridx])); } }; } public IPersistentMap meta() { return meta; } public ReindexList withMeta(IPersistentMap m) { return ReindexList.create(indexes, data, m); } public List reindex(int[] newIndexes) { final int nsz = newIndexes.length; final int[] idxs = new int[nsz]; boolean inOrder = true; for (int idx = 0; idx < nsz; ++idx) { final int nidx = indexes[newIndexes[idx]]; inOrder = inOrder && nidx == idx; idxs[idx] = nidx; } //If we are in-order, no new indexing is required. if(inOrder) { if (nsz == data.size()) return data; else return data.subList(0, nsz); } //Else create one new reindex with new indexes. This keeps the cost //of indirection to a the cost of 1 reindexing operation. return ReindexList.create(idxs, data, meta); } public static class MutReindexList extends ReindexList { IMutList mlist; public MutReindexList(int[] indexes, IMutList ml, IPersistentMap meta) { super(indexes, ml, meta); mlist = ml; } public long getLong(int idx) { checkIndex(idx, indexes.length); return mlist.getLong(indexes[idx]); } public void setLong(int idx, long nv) { checkIndex(idx, indexes.length); mlist.setLong(indexes[idx], nv); } public double getDouble(int idx) { checkIndex(idx, indexes.length); return mlist.getDouble(indexes[idx]); } public void setDouble(int idx, double nv) { checkIndex(idx, indexes.length); mlist.setDouble(indexes[idx], nv); } } } ================================================ FILE: java/ham_fisted/ReverseList.java ================================================ package ham_fisted; import java.util.List; import clojure.lang.IPersistentMap; import clojure.lang.IObj; public class ReverseList implements IMutList, TypedList { final List data; final IPersistentMap meta; final int nElems; final int nne; int _hash = 0; public ReverseList(List _data, IPersistentMap _meta) { data = _data; meta = _meta; nElems = data.size(); nne = nElems - 1; } public static ReverseList create(List data, IPersistentMap meta) { return data instanceof IMutList ? new MutReverseList((IMutList)data, meta) : new ReverseList(data, meta); } public String toString() { return Transformables.sequenceToString(this); } public int hashCode() { return hasheq(); } public int hasheq() { if (_hash == 0) { _hash = IMutList.super.hasheq(); } return _hash; } public boolean equals(Object other) { return equiv(other); } public Class containedType() { return data instanceof TypedList ? ((TypedList)data).containedType() : null; } public int size() { return nElems; } public Object get(int idx) { if(idx < 0) idx += nElems; return data.get(nne-idx); } public ReverseList subList(int sidx, int eidx) { sidx = nElems - sidx; eidx = nElems - eidx; return new ReverseList(data.subList(eidx, sidx), meta); } public List reverse() { return data instanceof IObj ? (List)((IObj)data).withMeta(meta) : data; } public IPersistentMap meta() { return meta; } public ReverseList withMeta(IPersistentMap m) { return new ReverseList(data, m); } public static class MutReverseList extends ReverseList { final IMutList mlist; public MutReverseList(IMutList ml, IPersistentMap meta) { super(ml, meta); mlist = ml; } public long getLong(int idx) { return mlist.getLong(nne - idx); } public double getDouble(int idx) { return mlist.getDouble(nne - idx); } } } ================================================ FILE: java/ham_fisted/SetOps.java ================================================ package ham_fisted; import java.util.Set; import java.util.Collection; public interface SetOps { Set union(Collection rhs); Set intersection(Set rhs); Set difference(Set rhs); } ================================================ FILE: java/ham_fisted/StringCollection.java ================================================ package ham_fisted; import java.util.Collection; import java.util.List; import java.util.RandomAccess; import java.util.Iterator; import java.util.ListIterator; import java.util.Arrays; import java.util.Objects; import clojure.lang.IReduce; import clojure.lang.IReduceInit; import clojure.lang.IFn; import clojure.lang.RT; import ham_fisted.Reductions; public class StringCollection implements IMutList { public final String cs; public StringCollection(String _cs) { cs = _cs; } public final int size() { return cs.length(); } public final Character get(int idx) { return cs.charAt(idx); } public final IMutList subList(int startidx, int endidx) { return new StringCollection((String)cs.subSequence(startidx, endidx)); } public final Object[] fillArray(Object[] data) { Reductions.serialReduction(new Reductions.IndexedAccum( new IFnDef.OLOO() { public Object invokePrim(Object acc, long idx, Object v) { ((Object[])acc)[(int)idx] = v; return acc; } }), data, this); return data; } public Object reduce(IFn rfn) { final int sz = size(); if (sz == 0) return rfn.invoke(); Object acc = null; for( int idx = 0; idx < sz && !RT.isReduced(acc); ++idx) acc = idx == 0 ? cs.charAt(idx) : rfn.invoke(acc, cs.charAt(idx)); return Reductions.unreduce(acc); } public Object reduce(IFn rfn, Object acc) { final int sz = size(); for( int idx = 0; idx < sz && !RT.isReduced(acc); ++idx) rfn.invoke(acc, cs.charAt(idx)); return Reductions.unreduce(acc); } } ================================================ FILE: java/ham_fisted/Sum.java ================================================ package ham_fisted; import clojure.lang.IDeref; import clojure.lang.Keyword; import clojure.lang.IReduceInit; import clojure.lang.PersistentArrayMap; import java.util.function.DoubleConsumer; import java.util.Collection; public final class Sum implements Consumers.IDerefDoubleConsumer, Reducible { public static final Keyword sumKwd = Keyword.intern(null, "sum"); public static final Keyword nElemsKwd = Keyword.intern(null, "n-elems"); //Low order summation bits public double d0; //High order summation bits public double d1; public double simpleSum; public long nElems; /** * Incorporate a new double value using Kahan summation / * compensation summation. * * High-order bits of the sum are in intermediateSum[0], low-order * bits of the sum are in intermediateSum[1], any additional * elements are application-specific. * * @param intermediateSum the high-order and low-order words of the intermediate sum * @param value the name value to be included in the running sum */ public void sumWithCompensation(double value) { double tmp = value - d1; double sum = d0; double velvel = sum + tmp; // Little wolf of rounding error d1 = (velvel - sum) - tmp; d0 = velvel; } public double computeFinalSum() { // Better error bounds to add both terms as the final sum double tmp = d0 + d1; if (Double.isNaN(tmp) && Double.isInfinite(simpleSum)) return simpleSum; else return tmp; } public Sum(double _d0, double _d1, double _simpleSum, long _nElems) { d0 = _d0; d1 = _d1; simpleSum = _simpleSum; nElems = _nElems; } public Sum() { this(0, 0, 0, 0); } public void acceptDouble(double data) { sumWithCompensation(data); simpleSum += data; nElems++; } public void merge(Sum other) { final long ne = nElems; accept(other.computeFinalSum()); nElems = ne + other.nElems; } public Sum reduce(Reducible other) { final Sum retval = new Sum(d0, d1, simpleSum, nElems); retval.merge((Sum)other); return retval; } public Object deref() { return new PersistentArrayMap(new Object[] { sumKwd, computeFinalSum(), nElemsKwd, nElems }); } public static class SimpleSum implements Consumers.IDerefDoubleConsumer, Reducible { double simpleSum; public SimpleSum() { simpleSum = 0.0; } public SimpleSum(SimpleSum o) { simpleSum = o.simpleSum; } public void acceptDouble(double val) { simpleSum += val; } public SimpleSum reduce(Reducible other) { final SimpleSum retval = new SimpleSum(this); retval.accept(((SimpleSum)other).simpleSum); return retval; } public Object deref() { return simpleSum; } }; } ================================================ FILE: java/ham_fisted/Transformables.java ================================================ package ham_fisted; import java.util.List; import java.util.AbstractCollection; import java.util.Collection; import java.util.Iterator; import java.util.NoSuchElementException; import java.util.HashMap; import java.util.Arrays; import java.util.Map; import java.util.Objects; import java.util.RandomAccess; import java.util.Collection; import java.util.BitSet; import java.util.concurrent.atomic.AtomicReference; import java.util.function.UnaryOperator; import java.util.function.BiFunction; import java.util.function.Consumer; import java.util.function.DoubleConsumer; import java.util.function.LongConsumer; import java.util.function.Predicate; import java.util.function.LongPredicate; import java.util.function.DoublePredicate; import java.util.function.Supplier; import clojure.lang.IFn; import clojure.lang.ArraySeq; import clojure.lang.Seqable; import clojure.lang.RT; import clojure.lang.IteratorSeq; import clojure.lang.ISeq; import clojure.lang.IObj; import clojure.lang.IPersistentMap; import clojure.lang.IReduce; import clojure.lang.IReduceInit; import clojure.lang.IDeref; import clojure.lang.Sequential; import clojure.lang.Murmur3; import clojure.lang.Util; import clojure.lang.IHashEq; import clojure.lang.IPersistentCollection; import clojure.lang.PersistentList; import clojure.lang.Reduced; import clojure.lang.Delay; import clojure.lang.Box; import clojure.java.api.Clojure; public class Transformables { public static final Delay toIterableDelay = new Delay(new IFnDef() { public Object invoke() { return ((IDeref)Clojure.var("ham-fisted.protocols", "->iterable")).deref(); } } ); public static final Iterable toIterable(Object obj) { if (obj instanceof Iterable) return (Iterable) obj; //else bail to protocol return (Iterable)((IFn)toIterableDelay.deref()).invoke(obj); } public interface IMapable extends Iterable, IObj { default IMapable map(IFn fn) { return new MapIterable(fn, meta(), this); } default IMapable filter(IFn fn) { return new FilterIterable(fn, meta(), this); } default IMapable cat(Iterable iters) { return new CatIterable(meta(), new Iterable[]{ArrayLists.toList(new Iterable[] { this }), iters}); } } public static boolean truthy(final Object obj) { return Casts.booleanCast(obj); } public static boolean not(final Object obj) { return obj == null || obj == Boolean.FALSE; } public static boolean not(final boolean v) { return !v; } public static IFn toReductionFn(Object rfn) { if(rfn instanceof IFn) return (IFn)rfn; if(rfn instanceof IFn.OLO) { final IFn.OLO rrfn = (IFn.OLO)rfn; return new IFnDef() { public Object invoke(Object lhs, Object rhs) { return rrfn.invokePrim(lhs, Casts.longCast(rhs)); } }; } if(rfn instanceof IFn.ODO) { final IFn.ODO rrfn = (IFn.ODO)rfn; return new IFnDef() { public Object invoke(Object lhs, Object rhs) { return rrfn.invokePrim(lhs, Casts.doubleCast(rhs)); } }; } else throw new RuntimeException("Unrecognised function type: " + String.valueOf(rfn.getClass())); } public static IFn.OLO toLongReductionFn(Object rfn) { if(rfn instanceof IFn.OLO) { return (IFn.OLO)rfn; } if(rfn instanceof IFnDef.ODO) { final IFn.ODO rrfn = (IFn.ODO)rfn; return new IFnDef.OLO() { public Object invokePrim(Object lhs, long rhs) { return rrfn.invokePrim(lhs, (double)rhs); } }; } IFn rrfn = (IFn)rfn; return new IFnDef.OLO() { public Object invokePrim(Object lhs, long rhs) { return rrfn.invoke(lhs, rhs); } }; } public static IFn.ODO toDoubleReductionFn(Object rfn) { if(rfn instanceof IFn.ODO) { return (IFn.ODO)rfn; } if(rfn instanceof IFnDef.OLO) { final IFn.OLO rrfn = (IFn.OLO)rfn; return new IFnDef.ODO() { public Object invokePrim(Object lhs, double rhs) { return rrfn.invokePrim(lhs, Casts.longCast(rhs)); } }; } IFn rrfn = (IFn)rfn; return new IFnDef.ODO() { public Object invokePrim(Object lhs, double rhs) { return rrfn.invoke(lhs, rhs); } }; } public static int iterCount(Iterator iter) { int c = 0; while(iter.hasNext()) { c++; iter.next(); } return c; } public static IFn mapReducer(final IFn rfn, final IFn mapFn) { return new IFnDef() { public Object invoke() { return rfn.invoke(); } public Object invoke(Object res) { return rfn.invoke(res); } public Object invoke(Object lhs, Object rhs) { return rfn.invoke(lhs, mapFn.invoke(rhs)); } public Object applyTo(Object arglist) { return rfn.invoke(RT.first(arglist), mapFn.applyTo(RT.next(arglist))); } }; } public static IFn longMapReducer(final IFn rfn, final IFn mapFn) { final IFn.OLO rr = (IFn.OLO)rfn; if (mapFn instanceof IFn.OL) { final IFn.OL mfn = (IFn.OL)mapFn; return new IFnDef() { public Object invoke() { return rfn.invoke(); } public Object invoke(Object res) { return rfn.invoke(res); } public Object invoke(Object lhs, Object v) { return rr.invokePrim(lhs, mfn.invokePrim(v)); } public Object applyTo(Object arglist) { return rfn.invoke(RT.first(arglist), mapFn.applyTo(RT.next(arglist))); } }; } else if (mapFn instanceof IFn.LL) { final IFn.LL mfn = (IFn.LL)mapFn; return new Reductions.LongAccum() { public Object invoke() { return rfn.invoke(); } public Object invoke(Object res) { return rfn.invoke(res); } public Object invokePrim(Object lhs, long v) { return rr.invokePrim(lhs, mfn.invokePrim(v)); } public Object applyTo(Object arglist) { return rfn.invoke(RT.first(arglist), mapFn.applyTo(RT.next(arglist))); } }; } else { final IFn.DL mfn = (IFn.DL)mapFn; return new Reductions.DoubleAccum() { public Object invoke() { return rfn.invoke(); } public Object invoke(Object res) { return rfn.invoke(res); } public Object invokePrim(Object lhs, double v) { return rr.invokePrim(lhs, mfn.invokePrim(v)); } public Object applyTo(Object arglist) { return rfn.invoke(RT.first(arglist), mapFn.applyTo(RT.next(arglist))); } }; } } final static HashMap> mappingReducer = new HashMap>(); static { mappingReducer.put(IFn.LD.class, (IFn rfn, IFn mapFn)-> { final IFn.ODO rr = toDoubleReductionFn(rfn); final IFn.LD mfn = (IFn.LD)mapFn; return new IFnDef.OLO() { public Object invoke() { return rfn.invoke(); } public Object invoke(Object res) { return rfn.invoke(res); } public Object invokePrim(Object lhs, long v) { return rr.invokePrim(lhs, mfn.invokePrim(v)); } public Object applyTo(Object arglist) { return rfn.invoke(RT.first(arglist), mapFn.applyTo(RT.next(arglist))); } }; }); mappingReducer.put(IFnDef.LD.class, mappingReducer.get(IFn.LD.class)); mappingReducer.put(IFn.OD.class, (IFn rfn, IFn mapFn)->{ final IFn.ODO rr = toDoubleReductionFn(rfn); final IFn.OD mfn = (IFn.OD)mapFn; return new IFnDef.OOO() { public Object invoke() { return rfn.invoke(); } public Object invoke(Object res) { return rfn.invoke(res); } public Object invoke(Object lhs, Object v) { return rr.invokePrim(lhs, mfn.invokePrim(v)); } public Object applyTo(Object arglist) { return rfn.invoke(RT.first(arglist), mapFn.applyTo(RT.next(arglist))); } }; }); mappingReducer.put(IFnDef.OD.class, mappingReducer.get(IFn.OD.class)); mappingReducer.put(IFn.DD.class, (IFn rfn, IFn mapFn)-> { final IFn.ODO rr = toDoubleReductionFn(rfn); final IFn.DD mfn = (IFn.DD)mapFn; return new IFnDef.ODO() { public Object invoke() { return rfn.invoke(); } public Object invoke(Object res) { return rfn.invoke(res); } public Object invokePrim(Object lhs, double rhs) { return rr.invokePrim(lhs, mfn.invokePrim(rhs)); } public Object applyTo(Object arglist) { return rfn.invoke(RT.first(arglist), mapFn.applyTo(RT.next(arglist))); } }; }); mappingReducer.put(IFnDef.DD.class, mappingReducer.get(IFn.DD.class)); mappingReducer.put(IFn.OL.class, (IFn rfn, IFn mapFn)-> { final IFn.OLO rr = toLongReductionFn(rfn); final IFn.OL mfn = (IFn.OL)mapFn; return new IFnDef() { public Object invoke() { return rfn.invoke(); } public Object invoke(Object res) { return rfn.invoke(res); } public Object invoke(Object lhs, Object v) { return rr.invokePrim(lhs, mfn.invokePrim(v)); } public Object applyTo(Object arglist) { return rfn.invoke(RT.first(arglist), mapFn.applyTo(RT.next(arglist))); } }; }); mappingReducer.put(IFnDef.OL.class, mappingReducer.get(IFn.OL.class)); mappingReducer.put(IFn.DL.class, (IFn rfn, IFn mapFn)-> { final IFn.OLO rr = toLongReductionFn(rfn); final IFn.DL mfn = (IFn.DL)mapFn; return new IFnDef.ODO() { public Object invoke() { return rfn.invoke(); } public Object invoke(Object res) { return rfn.invoke(res); } public Object invokePrim(Object lhs, double v) { return rr.invokePrim(lhs, mfn.invokePrim(v)); } public Object applyTo(Object arglist) { return rfn.invoke(RT.first(arglist), mapFn.applyTo(RT.next(arglist))); } }; }); mappingReducer.put(IFnDef.DL.class, mappingReducer.get(IFn.DL.class)); mappingReducer.put(IFn.LL.class, (IFn rfn, IFn mapFn)-> { final IFn.OLO rr = toLongReductionFn(rfn); final IFn.LL mfn = (IFn.LL)mapFn; return new IFnDef.OLO() { public Object invoke() { return rfn.invoke(); } public Object invoke(Object res) { return rfn.invoke(res); } public Object invokePrim(Object lhs, long v) { return rr.invokePrim(lhs, mfn.invokePrim(v)); } public Object applyTo(Object arglist) { return rfn.invoke(RT.first(arglist), mapFn.applyTo(RT.next(arglist))); } }; }); mappingReducer.put(IFnDef.LL.class, mappingReducer.get(IFn.LL.class)); } public static IFn typedMapReducer(IFn rfn, IFn mapFn) { final Class[] ifaces = mapFn.getClass().getInterfaces(); final int il = ifaces.length; for(int idx = 0; idx < il; ++idx) { final BiFunction ff = mappingReducer.get(ifaces[idx]); if(ff != null) return ff.apply(rfn, mapFn); } return new IFnDef() { public Object invoke() { return rfn.invoke(); } public Object invoke(Object res) { return rfn.invoke(res); } public Object invoke(Object lhs, Object v) { return rfn.invoke(lhs, mapFn.invoke(v)); } public Object applyTo(Object arglist) { return rfn.invoke(RT.first(arglist), mapFn.applyTo(RT.next(arglist))); } }; } public static Object singleMapReduce(final Object item, final IFn rfn, final IFn mapFn, Object init) { return Reductions.serialReduction(typedMapReducer(rfn, mapFn), init, item); } public static boolean seqEquiv(Seqable ss, Object o){ ISeq s = ss.seq(); if(s != null) return s.equiv(o); else return (o instanceof Sequential || o instanceof List) && RT.seq(o) == null; } public static int seqHashCode(Seqable ss) { ISeq s = ss.seq(); if(s == null) return 1; return Util.hash(s); } public interface IterableSeq extends Collection, Seqable, IMapable, ITypedReduce, IHashEq, Sequential, IPersistentCollection { default int hasheq() { return Murmur3.hashOrdered(this); } default boolean equiv(Object o) { return seqEquiv(this, o); } default int count() { long retval = 0; final Iterator iter = iterator(); while(iter.hasNext()) { ++retval; iter.next(); } return RT.intCast(retval); } default IPersistentCollection cons(Object o) { return RT.cons(o, seq()); } default IPersistentCollection empty() { return PersistentList.EMPTY; } @SuppressWarnings("unchecked") default void forEach(Consumer c) { ITypedReduce.super.forEach(c); } default ISeq seq() { return LazyChunkedSeq.chunkIteratorSeq(iterator()); } } public static class MapIterable extends AbstractCollection implements IterableSeq { final Object[] iterables; final IFn fn; final IPersistentMap meta; public MapIterable(IFn _fn, IPersistentMap _meta, Object... _its) { fn = _fn; meta = _meta; iterables = _its; } public static MapIterable createSingle(IFn fn, IPersistentMap meta, Object single) { return new MapIterable(fn, meta, new Object[] { single }); } public MapIterable(MapIterable o, IPersistentMap m) { fn = o.fn; iterables = o.iterables; meta = m; } public String toString() { return sequenceToString(this); } public boolean isEmpty() { return iterator().hasNext() == false; } public int size() { return iterCount(iterator()); } static class SingleIterator implements Iterator { final Iterator iter; final IFn fn; public SingleIterator(IFn _fn, Iterator it) { iter = it; fn = _fn; } public boolean hasNext() { return iter.hasNext(); } public Object next() { return fn.invoke(iter.next()); } } static class DualIterator implements Iterator { final Iterator lhs; final Iterator rhs; final IFn fn; public DualIterator(IFn f, Iterator l, Iterator r) { lhs = l; rhs = r; fn = f; } public boolean hasNext() { return lhs.hasNext() && rhs.hasNext(); } public Object next() { return fn.invoke(lhs.next(), rhs.next()); } } public Iterator iterator() { final int ss = iterables.length; switch(ss) { case 1: return new SingleIterator(fn, toIterable(iterables[0]).iterator()); case 2: return new DualIterator(fn, toIterable(iterables[0]).iterator(), toIterable(iterables[1]).iterator()); default: final Iterator[] iterators = new Iterator[ss]; for(int idx = 0; idx < ss; ++idx) { iterators[idx] = toIterable(iterables[idx]).iterator(); } return new Iterator() { public boolean hasNext() { for(int idx = 0; idx < ss; ++idx) if( iterators[idx].hasNext() == false) return false; return true; } public Object next() { switch(ss) { case 3: return fn.invoke(iterators[0].next(), iterators[1].next(), iterators[2].next()); case 4: return fn.invoke(iterators[0].next(), iterators[1].next(), iterators[2].next(), iterators[3].next()); default: Object[] args = new Object[ss]; for (int idx = 0; idx < ss; ++idx) args[idx] = iterators[idx].next(); return fn.applyTo(ArraySeq.create(args)); } } }; } } public ISeq seq() { return LazyChunkedSeq.chunkIteratorSeq(iterator()); } public boolean equals(Object o) { return equiv(o); } public int hashCode(){ return hasheq(); } public MapIterable map(IFn nfn) { return new MapIterable(MapFn.create(fn, nfn), meta(), iterables); } public IPersistentMap meta() { return meta; } public MapIterable withMeta(IPersistentMap m) { return new MapIterable(this, m); } public Object reduce(IFn rfn, Object acc) { final int nLists = iterables.length; if(nLists == 1) { return singleMapReduce(iterables[0], rfn, fn, acc); } else { final Iterator[] iterators = new Iterator[nLists]; for(int idx = 0; idx < nLists; ++idx) iterators[idx] = toIterable(iterables[idx]).iterator(); switch(iterables.length) { case 2: { final Iterator l = iterators[0]; final Iterator r = iterators[1]; while(l.hasNext() && r.hasNext()) { acc = rfn.invoke(acc, fn.invoke(l.next(), r.next())); if(RT.isReduced(acc)) return ((IDeref)acc).deref(); } break; } case 3: { final Iterator x = iterators[0]; final Iterator y = iterators[1]; final Iterator z = iterators[2]; while(x.hasNext() && y.hasNext() && z.hasNext()) { acc = rfn.invoke(acc, fn.invoke(x.next(), y.next(), z.next())); if(RT.isReduced(acc)) return ((IDeref)acc).deref(); } break; } case 4: { final Iterator x = iterators[0]; final Iterator y = iterators[1]; final Iterator z = iterators[2]; final Iterator w = iterators[3]; while(x.hasNext() && y.hasNext() && z.hasNext() && w.hasNext()) { acc = rfn.invoke(acc, fn.invoke(x.next(), y.next(), z.next(), w.next())); if(RT.isReduced(acc)) return ((IDeref)acc).deref(); } break; } default: { boolean hn = true; for(int idx = 0; idx < nLists && hn; ++idx) { hn = hn && iterators[idx].hasNext(); } if(hn) { final Object[] args = new Object[nLists]; final ISeq argSeq = ArraySeq.create(args); while(hn) { for(int idx = 0; idx < nLists && hn; ++idx) { args[idx] = iterators[idx].next(); } acc = rfn.invoke(acc, fn.applyTo(argSeq)); if(RT.isReduced(acc)) return ((IDeref)acc).deref(); for(int idx = 0; idx < nLists && hn; ++idx) { hn = hn && iterators[idx].hasNext(); } } } } } } return acc; } public Object parallelReduction(IFn initValFn, IFn rfn, IFn mergeFn, ParallelOptions options) { if(iterables.length == 1) { return Reductions.parallelReduction(initValFn, typedMapReducer(rfn, fn), mergeFn, iterables[0], options); } else { return Reductions.serialParallelReduction(initValFn, rfn, options, this); } } public Object[] toArray() { return ArrayLists.toArray(this); } } public static class PredFn implements IFnDef { static IFn create(IFn src, IFn dst) { if ((src instanceof LongPredicate) && (dst instanceof LongPredicate)) { final LongPredicate ss = (LongPredicate)src; final LongPredicate dd = (LongPredicate)dst; return new IFnDef.LongPredicate() { public boolean test(long v) { return ss.test(v) && dd.test(v); } }; } else if((src instanceof IFn.LO) && (dst instanceof IFn.LO)) { final IFn.LO ss = (IFn.LO)src; final IFn.LO dd = (IFn.LO)dst; return new IFnDef.LO() { public Object invokePrim(long v) { return truthy(ss.invokePrim(v)) && truthy(dd.invokePrim(v)); } }; } else if ((src instanceof DoublePredicate) && (dst instanceof DoublePredicate)) { final DoublePredicate ss = (DoublePredicate)src; final DoublePredicate dd = (DoublePredicate)dst; return new IFnDef.DoublePredicate() { public boolean test(double v) { return ss.test(v) && dd.test(v); } }; } else if((src instanceof IFn.DO) && (dst instanceof IFn.DO)) { final IFn.DO ss = (IFn.DO)src; final IFn.DO dd = (IFn.DO)dst; return new IFnDef.DO() { public Object invokePrim(double v) { return truthy(ss.invokePrim(v)) && truthy(dd.invokePrim(v)); } }; } else if ((src instanceof Predicate) && (dst instanceof Predicate)) { final Predicate ss = (Predicate)src; final Predicate dd = (Predicate)dst; return new IFnDef.Predicate() { @SuppressWarnings("unchecked") public boolean test(Object v) { return ss.test(v) && dd.test(v); } }; } else { return new PredFn(src,dst); } } final IFn srcPred; final IFn dstPred; public PredFn(IFn sp, IFn dp) { srcPred = sp; dstPred = dp; } public Object invoke(Object v) { return truthy(srcPred.invoke(v)) && truthy(dstPred.invoke(v)); } }; public static class FilterIterable extends AbstractCollection implements IterableSeq { final IFn pred; final Object src; final IPersistentMap meta; public FilterIterable(IFn _p, IPersistentMap _meta, Object _i) { pred = _p; src = _i; meta = _meta; } public FilterIterable(FilterIterable o, IPersistentMap m) { pred = o.pred; src = o.src; meta = m; } public String toString() { return sequenceToString(this); } public boolean isEmpty() { return seq() == null; } public int size() { return iterCount(iterator()); } static class FilterIterator implements Iterator { final Iterator iter; final IFn pred; Box nextObj = new Box(null); public FilterIterator(Iterator _i, IFn p) { iter = _i; pred = p; advance(); } void advance() { while(iter.hasNext()) { final Object nobj = iter.next(); if (truthy(pred.invoke(nobj))) { nextObj.val = nobj; return; } } nextObj = null; } public boolean hasNext() { return nextObj != null; } public Object next() { if (nextObj == null) throw new NoSuchElementException(); final Object retval = nextObj.val; advance(); return retval; } } public Iterator iterator() { return new FilterIterator(toIterable(src).iterator(), pred); } public int hashCode(){ return hasheq(); } public boolean equals(Object o) { return equiv(o); } public ISeq seq() { return LazyChunkedSeq.chunkIteratorSeq(iterator()); } public IMapable filter(IFn nfn) { return new FilterIterable(PredFn.create(pred, nfn), meta(), src); } public IPersistentMap meta() { return meta; } public FilterIterable withMeta(IPersistentMap m) { return new FilterIterable(this, m); } @SuppressWarnings("unchecked") public static IFn typedReducer(final IFn rfn, final IFn pred) { if(pred instanceof LongPredicate) { final LongPredicate pfn = (LongPredicate)pred; final IFn.OLO rrfn = toLongReductionFn(rfn); return new Reductions.LongAccum() { public Object invoke() { return rfn.invoke(); } public Object invoke(Object res) { return rfn.invoke(res); } public Object invokePrim(Object lhs, long v) { return pfn.test(v) ? rrfn.invokePrim(lhs, v) : lhs; } }; } else if(pred instanceof IFn.LO) { final IFn.LO pfn = (IFn.LO)pred; final IFn.OLO rrfn = toLongReductionFn(rfn); return new Reductions.LongAccum() { public Object invoke() { return rfn.invoke(); } public Object invoke(Object res) { return rfn.invoke(res); } public Object invokePrim(Object lhs, long v) { return truthy(pfn.invokePrim(v)) ? rrfn.invokePrim(lhs, v) : lhs; } }; } else if (pred instanceof DoublePredicate) { final DoublePredicate pfn = (DoublePredicate)pred; final IFn.ODO rrfn = toDoubleReductionFn(rfn); return new Reductions.DoubleAccum() { public Object invoke() { return rfn.invoke(); } public Object invoke(Object res) { return rfn.invoke(res); } public Object invokePrim(Object lhs, double v) { return pfn.test(v) ? rrfn.invokePrim(lhs, v) : lhs; } }; } else if (pred instanceof IFn.DO) { final IFn.DO pfn = (IFn.DO)pred; final IFn.ODO rrfn = toDoubleReductionFn(rfn); return new Reductions.DoubleAccum() { public Object invoke() { return rfn.invoke(); } public Object invoke(Object res) { return rfn.invoke(res); } public Object invokePrim(Object lhs, double v) { return truthy(pfn.invokePrim(v)) ? rrfn.invokePrim(lhs, v) : lhs; } }; } else if (pred instanceof Predicate) { final Predicate pfn = (Predicate)pred; return new IFnDef() { public Object invoke() { return rfn.invoke(); } public Object invoke(Object res) { return rfn.invoke(res); } public Object invoke(Object lhs, Object v) { return pfn.test(v) ? rfn.invoke(lhs, v) : lhs; } }; } else { return new IFnDef() { public Object invoke() { return rfn.invoke(); } public Object invoke(Object res) { return rfn.invoke(res); } public Object invoke(Object lhs, Object v) { return truthy(pred.invoke(v)) ? rfn.invoke(lhs, v) : lhs; } }; } } public Object reduce(final IFn rfn, final Object init) { return Reductions.serialReduction(typedReducer(rfn,pred), init, src); } public Object parallelReduction(IFn initValFn, IFn rfn, IFn mergeFn, ParallelOptions options) { return Reductions.parallelReduction(initValFn, typedReducer(rfn, pred), mergeFn, src, options); } public Object[] toArray() { return ArrayLists.toArray(this); } } public static class CatIterable extends AbstractCollection implements IterableSeq { //this is an array of iterables of iterables. final Iterable[] data; public final IPersistentMap meta; public final ParallelOptions.CatParallelism parallelism; public CatIterable(IPersistentMap _meta, ParallelOptions.CatParallelism parallelism, Iterable arglist) { this.data = new Iterable[] {arglist}; this.meta = _meta; this.parallelism = parallelism; } public CatIterable(IPersistentMap _meta, Iterable[] f) { this.data = f; this.meta = _meta; this.parallelism = null; } public CatIterable(Iterable arglist) { data = new Iterable[]{arglist}; meta = null; parallelism = null; } public CatIterable(CatIterable other, IPersistentMap m) { data = other.data; meta = m; parallelism = other.parallelism; } public String toString() { return sequenceToString(this); } public boolean equals(Object o) { return equiv(o); } public int hashCode() { return hasheq(); } public int size() { return iterCount(iterator()); } public boolean isEmpty() { return iterator().hasNext() == false; } static class CatIteratorCtx implements CtxIter.Ctx { public final Iterator gpIter; public final Iterator parentIter; public final Iterator curIter; public CatIteratorCtx(Iterator gpIter, Iterator parentIter, Iterator curIter) { this.gpIter = gpIter; this.parentIter = parentIter; this.curIter = curIter; } public CatIteratorCtx update() { if(curIter != null && curIter.hasNext()) return this; Iterator curIter = this.curIter; Iterator parentIter = this.parentIter; do { if(parentIter != null && parentIter.hasNext()) { Iterable pp = toIterable(parentIter.next()); curIter = pp != null ? pp.iterator() : null; } else { if(gpIter.hasNext()) { Iterable gg = toIterable(gpIter.next()); parentIter = gg != null ? gg.iterator() : null; } else { return new CatIteratorCtx(gpIter, null, null); } } if(curIter != null && curIter.hasNext()) return new CatIteratorCtx(gpIter, parentIter, curIter); } while(true); } public boolean valid() { return curIter != null && curIter.hasNext(); } public Object val() { return curIter.next(); } } public Iterator iterator() { return new CtxIter(new Supplier() { public CtxIter.Ctx get() { return new CatIteratorCtx(ArrayLists.toList(data).iterator(), null, null).update(); } }); } public IMapable cat(Iterable _iters) { final int dlen = data.length; final Iterable[] newd = Arrays.copyOf(data, dlen+1); newd[dlen] = _iters; return new CatIterable(meta(), newd); } public ISeq seq() { return LazyChunkedSeq.chunkIteratorSeq(iterator()); } public IPersistentMap meta() { return meta; } public CatIterable withMeta(IPersistentMap m) { return new CatIterable(this, m); } public Object reduce(IFn fn) { return Reductions.iterReduce(this, fn); } public static IFn catReducer(IFn rfn) { if(rfn instanceof IFn.OLO) { final IFn.OLO ff = (IFn.OLO)rfn; return new IFnDef.OLO() { public Object invokePrim(Object acc, long v) { acc = ff.invokePrim(acc, v); return RT.isReduced(acc) ? new Reduced(acc) : acc; } }; } else if (rfn instanceof IFn.ODO) { final IFn.ODO ff = (IFn.ODO)rfn; return new IFnDef.ODO() { public Object invokePrim(Object acc, double v) { acc = ff.invokePrim(acc, v); return RT.isReduced(acc) ? new Reduced(acc) : acc; } }; } else { return new IFnDef() { public Object invoke(Object acc, Object v) { acc = rfn.invoke(acc,v); return RT.isReduced(acc) ? new Reduced(acc) : acc; } }; } } public Object reduce(IFn rfn, Object init) { final int nData = data.length; final Iterable[] d = data; final IFn rf = catReducer(rfn); for(int idx = 0; idx < nData && !RT.isReduced(init); ++idx) { final Iterable item = d[idx]; final Iterator iter = item != null ? item.iterator() : null; if (iter != null) { for(; iter.hasNext() && !RT.isReduced(init);) init = Reductions.serialReduction(rf, init, iter.next()); } } return Reductions.unreduce(init); } static class CatIterIter implements Iterator { Iterator gpIter; Iterator parentIter; public CatIterIter(Iterator _gpIter) { gpIter = _gpIter; parentIter = null; advance(); } public boolean hasNext() { return parentIter != null && parentIter.hasNext(); } public Object next() { final Object rv = parentIter.next(); if(!parentIter.hasNext()) advance(); return rv; } void advance() { if(hasNext()) { return; } while(gpIter != null && gpIter.hasNext()) { parentIter = null; final Iterable parent = (Iterable)gpIter.next(); if(parent != null) { parentIter = parent.iterator(); } if(hasNext()) return; } gpIter = null; parentIter = null; } } public Iterator containerIter() { return new CatIterIter(ArrayLists.toList(data).iterator()); } Object preduceSeqwise(IFn initValFn, IFn rfn, IFn mergeFn, Object init, ParallelOptions options) { final Iterable initSequence = new Iterable() { public Iterator iterator() { return containerIter(); } }; final Iterable partiallyReduced = ForkJoinPatterns.pmap(options, new IFnDef() { public Object invoke(Object arg) { return Reductions.serialReduction(rfn, initValFn.invoke(), arg); } }, ArrayLists.toList(new Object[] { initSequence })); if(options.unmergedResult) return partiallyReduced; return Reductions.serialReduction(mergeFn, init, partiallyReduced); } public Object parallelReduction(IFn initValFn, IFn rfn, IFn mergeFn, ParallelOptions options) { final int nData = data.length; final Iterable[] d = data; Object init = initValFn.invoke(); final IFn rf = catReducer(rfn); ParallelOptions.CatParallelism catP = this.parallelism != null ? this.parallelism : options.catParallelism; switch (catP) { case ELEMWISE: { final Iterable initSequence = new Iterable() { public Iterator iterator() { return new CatIterIter(ArrayLists.toList(data).iterator()); } }; final Iterable mapped = new MapIterable(new IFnDef() { public Object invoke(Object data) { return Reductions.parallelReduction(initValFn, rf, mergeFn, data, options); } }, null, initSequence); if(options.unmergedResult) { init = new CatIterable(mapped); } else { init = Reductions.iterableMerge(options, mergeFn, mapped); } break; } case SEQWISE: { init = preduceSeqwise(initValFn, rf, mergeFn, init, options); break; } }; return Reductions.unreduce(init); } public Object[] toArray() { return ArrayLists.toArray(this); } } public static class SingleMapList implements IMutList, IMapable { final int nElems; final List list; final IFn fn; final IPersistentMap meta; public SingleMapList(IFn _fn, IPersistentMap m, List l) { nElems = l.size(); list = l; fn = _fn; meta = m; } public SingleMapList(SingleMapList o, IPersistentMap m) { nElems = o.nElems; list = o.list; fn = o.fn; meta = m; } public String toString() { return sequenceToString(this); } public boolean equals(Object other) { return equiv(other); } public int hashCode() { return hasheq(); } public int size() { return nElems; } public Object get(int idx) { return fn.invoke(list.get(idx)); } public SingleMapList subList(int sidx, int eidx) { return new SingleMapList(fn, meta, list.subList(sidx, eidx)); } public IPersistentMap meta() { return meta; } public SingleMapList withMeta(IPersistentMap m) { return new SingleMapList(this, m); } public SingleMapList map(IFn nfn) { return new SingleMapList(MapFn.create(fn, nfn), meta, list); } public Object reduce(IFn rfn, Object init) { return singleMapReduce(list, rfn, fn, init); } public Object parallelReduction(IFn initValFn, IFn rfn, IFn mergeFn, ParallelOptions options) { return Reductions.parallelReduction(initValFn, typedMapReducer(rfn, fn), mergeFn, list, options); } } public static class DualMapList implements IMutList, IMapable { final int nElems; final List lhs; final List rhs; final IFn fn; final IPersistentMap meta; public DualMapList(IFn _fn, IPersistentMap m, List l, List r) { nElems = Math.min(l.size(), r.size()); lhs = l; rhs = r; fn = _fn; meta = m; } public DualMapList(DualMapList o, IPersistentMap m) { nElems = o.nElems; lhs = o.lhs; rhs = o.rhs; fn = o.fn; meta = m; } public String toString() { return sequenceToString(this); } public boolean equals(Object other) { return equiv(other); } public int hashCode() { return hasheq(); } public int size() { return nElems; } public Object get(int idx) { return fn.invoke(lhs.get(idx), rhs.get(idx)); } public DualMapList subList(int sidx, int eidx) { return new DualMapList(fn, meta, lhs.subList(sidx, eidx), rhs.subList(sidx, eidx)); } public IPersistentMap meta() { return meta; } public DualMapList withMeta(IPersistentMap m) { return new DualMapList(this, m); } public DualMapList map(IFn nfn) { return new DualMapList(MapFn.create(fn, nfn), meta, lhs, rhs); } } public static class MapList implements IMutList, IMapable { final int nElems; final List[] lists; final IFn fn; final IPersistentMap meta; final IFn.LO getter; public MapList(IFn _fn, IPersistentMap _meta, List... _lists) { final int nLists = _lists.length; if(nLists == 0) nElems = 0; else { int ne = _lists[0].size(); for(int idx = 1; idx < nLists; ++idx) ne = Math.min(ne, _lists[idx].size()); nElems = ne; } lists = _lists; fn = _fn; meta = _meta; IFn.LO getter = null; final List[] ll = lists; switch(nLists) { case 1: getter = new IFnDef.LO() { public Object invokePrim(long idx) { return fn.invoke(ll[0].get((int)idx)); } }; break; case 2: getter = new IFnDef.LO() { public Object invokePrim(long idx) { return fn.invoke(ll[0].get((int)idx), ll[1].get((int)idx)); } }; break; case 3: getter = new IFnDef.LO() { public Object invokePrim(long lidx) { final int idx = (int)lidx; return fn.invoke(ll[0].get(idx), ll[1].get(idx), ll[2].get(idx)); } }; break; case 4: getter = new IFnDef.LO() { public Object invokePrim(long lidx) { final int idx = (int)lidx; return fn.invoke(ll[0].get(idx), ll[1].get(idx), ll[2].get(idx), ll[3].get(idx)); } }; break; default: getter = new IFnDef.LO() { public Object invokePrim(long lidx) { final int idx = (int)lidx; Object[] args = new Object[nLists]; for (int aidx = 0; aidx < nLists; ++aidx) args[aidx] = lists[aidx].get(idx); return fn.applyTo(ArraySeq.create(args)); } }; } this.getter = getter; } public MapList(MapList other, IPersistentMap m) { nElems = other.nElems; lists = other.lists; fn = other.fn; getter = other.getter; meta = m; } public static IMutList create(IFn fn, IPersistentMap meta, List... lists) { if (lists.length == 1) return new SingleMapList(fn, meta, lists[0]); else if (lists.length == 2) return new DualMapList(fn, meta, lists[0], lists[1]); else return new MapList(fn, meta, lists); } public String toString() { return sequenceToString(this); } public boolean equals(Object other) { return equiv(other); } public int hashCode() { return hasheq(); } public int size() { return nElems; } public Object get(int idx) { if(idx < 0) idx += nElems; if(idx < 0 || idx >= nElems) throw new RuntimeException("Index out of range."); return getter.invokePrim(idx); } public Object reduce(IFn rfn, Object acc) { final int ne = nElems; final List[] ll = lists; final int ls = ll.length; switch(ls) { case 1: return Reductions.serialReduction( mapReducer(rfn, this.fn), acc, ll[0] ); case 2: for (int idx = 0; idx < ne; ++idx) { acc = rfn.invoke(acc, fn.invoke(lists[0].get(idx), lists[1].get(idx))); if(RT.isReduced(acc)) return ((IDeref)acc).deref(); } return acc; case 3: for (int idx = 0; idx < ne; ++idx) { acc = rfn.invoke(acc, fn.invoke(lists[0].get(idx), lists[1].get(idx), lists[2].get(idx))); if(RT.isReduced(acc)) return ((IDeref)acc).deref(); } return acc; case 4: for (int idx = 0; idx < ne; ++idx) { acc = rfn.invoke(acc, fn.invoke(lists[0].get(idx), lists[1].get(idx), lists[2].get(idx), lists[3].get(idx))); if(RT.isReduced(acc)) return ((IDeref)acc).deref(); } return acc; } //fallthrough final Object[] args = new Object[ls]; final ISeq arglist = ArraySeq.create(args); for(int oidx = 0; oidx < ne; ++oidx) { for (int aidx = 0; aidx < ls; ++aidx) args[aidx] = lists[aidx].get(oidx); acc = rfn.invoke(acc, fn.applyTo(arglist)); if(RT.isReduced(acc)) return ((IDeref)acc).deref(); } return acc; } public IMutList subList(int sidx, int eidx) { final int sz = size(); if (sidx < 0 || sidx >= sz) throw new RuntimeException("Start index out of range."); if (eidx < sidx || eidx > sz) throw new RuntimeException("End index out of range."); final int ll = lists.length; List[] newLists = new List[ll]; for(int idx = 0; idx < ll; ++idx) newLists[idx] = lists[idx].subList(sidx, eidx); return new MapList(fn, meta(), newLists); } public MapList map(IFn nfn) { return new MapList(MapFn.create(fn, nfn), meta(), lists); } public IPersistentMap meta() { return meta; } public MapList withMeta(IPersistentMap m) { return new MapList(this, m); } } public static class IndexedMapper extends AbstractCollection implements IterableSeq { final IFn mapFn; final Object src; final IPersistentMap meta; public IndexedMapper(IFn mapFn, Object src, IPersistentMap m) { this.mapFn = mapFn; this.src = src; this.meta = m; } public String toString() { return sequenceToString(this); } public boolean equals(Object other) { return equiv(other); } public int hashCode() { return hasheq(); } public boolean isEmpty() { return iterator().hasNext() == false; } public int size() { return iterCount(toIterable(src).iterator()); } public static class CountingFn implements IFnDef { long cnt; final IFn mapFn; public CountingFn(IFn mapFn) { cnt = 0; this.mapFn = mapFn; } public Object invoke(Object arg) { return mapFn.invoke(cnt++, arg); } } MapIterable mapper() { return new MapIterable(new CountingFn(mapFn), meta, new Object[] { src }); } public Iterator iterator() { return mapper().iterator(); } public Object reduce(IFn rfn, Object init) { return mapper().reduce(rfn, init); } public IMapable map(IFn fn) { final IFn srcFn = mapFn; return new IndexedMapper(new IFnDef() { public Object invoke(Object idx, Object arg) { return fn.invoke(srcFn.invoke(idx,arg)); } }, src, meta); } public IPersistentMap meta() { return meta; } public IObj withMeta(IPersistentMap m) { return new IndexedMapper(mapFn, src, meta); } } public static class CachingIterable extends AbstractCollection implements Seqable { final Iterable src; final IPersistentMap meta; AtomicReference seq; public CachingIterable(Iterable _src, IPersistentMap _meta) { src = _src; meta = _meta; seq = new AtomicReference(); } CachingIterable(CachingIterable other, IPersistentMap m) { src = other.src; meta = m; seq = other.seq; } public ISeq seq() { return seq.updateAndGet(new UnaryOperator() { public ISeq apply(ISeq v) { if(v != null) return v; return src instanceof Seqable ? ((Seqable)src).seq() : LazyChunkedSeq.chunkIteratorSeq(src.iterator()); } }); } public Iterator iterator() { return ((Collection)seq()).iterator(); } public int size() { return ((Collection)seq()).size(); } public IPersistentMap meta() { return meta; } public CachingIterable withMeta(IPersistentMap m) { return new CachingIterable(src, m); } } public static class CachingList implements IMutList { final List src; final Object[] dataCache; final BitSet cachedIndexes; final IPersistentMap meta; int _hash; public CachingList(List srcData, IPersistentMap _meta) { src = srcData; meta = _meta; dataCache = new Object[srcData.size()]; cachedIndexes = new BitSet(); } CachingList(CachingList other, IPersistentMap _meta) { src = other.src; dataCache = other.dataCache; cachedIndexes = other.cachedIndexes; meta = _meta; } public int hashCode() { return hasheq(); } public int hasheq() { if (_hash == 0) _hash = IMutList.super.hasheq(); return _hash; } public boolean equals(Object other) { return equiv(other); } public String toString() { return sequenceToString(this); } public Object get(final int idx) { synchronized(cachedIndexes) { if(cachedIndexes.get(idx)) return dataCache[idx]; final Object retval = src.get(idx); dataCache[idx] = retval; cachedIndexes.set(idx); return retval; } } public int size() { return src.size(); } public CachingList withMeta(IPersistentMap m) { return new CachingList(this, m); } } static void appendObjects(StringBuilder sb, Collection data) { boolean first = true; for(Object o: data) { if(!first) sb.append(" "); first = false; sb.append(o == null ? "nil" : o.toString()); } } public static String sequenceToString(Iterable data) { StringBuilder sb = new StringBuilder(); if(data instanceof RandomAccess) { final List ra = (List) data; final int sz = ra.size(); sb.append("["); if(sz < 50) { appendObjects(sb, ra); } else { appendObjects(sb, ra.subList(0, 20)); sb.append(" ... "); appendObjects(sb, ra.subList(sz-20, sz)); } sb.append("]"); } else { sb.append("("); if (data != null) { int idx = 0; for(Object o: data) { if(idx >= 50) { sb.append(" ..."); break; } if (idx > 0) sb.append(" "); sb.append(o == null ? "nil" : o.toString()); ++idx; } } sb.append(")"); } return sb.toString(); } } ================================================ FILE: java/ham_fisted/TransientHashMap.java ================================================ package ham_fisted; import java.util.function.BiFunction; import java.util.Map; import clojure.lang.Indexed; import clojure.lang.ITransientMap; import clojure.lang.ITransientAssociative2; import clojure.lang.IObj; import clojure.lang.IPersistentMap; import clojure.lang.IFn; public class TransientHashMap extends ROHashMap implements IATransientMap, IObj { public TransientHashMap(HashMap data) { super(data.loadFactor, data.capacity, data.length, data.data.clone(), data.meta); } public TransientHashMap(TransientHashMap data, IPersistentMap m) { super(data.loadFactor, data.capacity, data.length, data.data, m); } public TransientHashMap conj(Object val) { if(val instanceof Map) { return (TransientHashMap)union((Map)val, BiFunctions.rhsWins); } else { return (TransientHashMap)IATransientMap.super.conjVal(val); } } public TransientHashMap assoc(Object key, Object val) { int hc = hash(key); int idx = hc & mask; HashNode e = data[idx]; data[idx] = e != null ? e.assoc(this, key, hc, val) : newNode(key, hc, val); return this; } public TransientHashMap without(Object key) { int hc = hash(key); int idx = hc & mask; HashNode e = data[idx]; if(e!=null) data[idx] = e.dissoc(this, key); return this; } public PersistentHashMap persistent() { return new PersistentHashMap(this); } public TransientHashMap withMeta(IPersistentMap m) { return new TransientHashMap(this, m); } } ================================================ FILE: java/ham_fisted/TransientHashSet.java ================================================ package ham_fisted; import clojure.lang.IPersistentMap; public class TransientHashSet extends ROHashSet implements IATransientSet { public TransientHashSet(HashBase hb, IPersistentMap meta) { super(hb, meta); } public TransientHashSet conj(Object key) { int hc = hash(key); int idx = hc & mask; HashNode e = data[idx]; data[idx] = e != null ? e.assoc(this, key, hc, VALUE) : newNode(key, hc, VALUE); return this; } public TransientHashSet disjoin(Object key) { int hc = hash(key); int idx = hc & mask; HashNode e = data[idx]; if(e != null) data[idx] = e.dissoc(this, key); return this; } public PersistentHashSet persistent() { return new PersistentHashSet(this, meta); } } ================================================ FILE: java/ham_fisted/TransientList.java ================================================ package ham_fisted; import static ham_fisted.IntegerOps.*; import clojure.lang.ITransientVector; import clojure.lang.RT; import clojure.lang.Util; import java.util.BitSet; import java.util.Arrays; public class TransientList implements ITransientVector, IFnDef { final ChunkedList data; final BitSet ownedChunks; //Note there is no startidx. We cannot share structure with sub-lists so startidx is //always 0. Thus calling transient on a subvector results int nElems; public TransientList(ChunkedList _data, int _nElems, boolean _ownsEverything) { data = _data; nElems = _nElems; ownedChunks = new BitSet(); if( _ownsEverything) { final int nChunks = ChunkedList.numChunks(nElems); for(int idx = 0; idx < nChunks; ++idx) ownedChunks.set(idx); } } final int indexCheck(int idx) { return ChunkedList.indexCheck(0, nElems, idx); } final int wrapIndexCheck(int idx) { return ChunkedList.wrapIndexCheck(0, nElems, idx); } public final int count() { return nElems; } public final int size() { return nElems; } public final int length() { return nElems; } public final Object nth(int idx) { return nth(idx, null); } public final Object nth(int idx, Object notFound) { if (idx < 0) idx += nElems; if (idx >= 0 && idx < nElems) return data.getValue(idx); return notFound; } public final Object invoke(Object idx) { return nth(RT.intCast(idx)); } public final Object invoke(Object idx, Object nf) { return nth(RT.intCast(idx), nf); } public final Object valAt(Object idx) { if (Util.isInteger(idx)) return nth(RT.intCast(idx)); return null; } public final Object valAt(Object idx, Object notFound) { if (Util.isInteger(idx)) return nth(RT.intCast(idx), notFound); return notFound; } public final TransientList assocN(int idx, Object v) { if (idx == nElems) return conj(v); indexCheck(idx); int cidx = idx/32; int eidx = idx%32; final Object[][] mdata = data.data; Object[] chunk = mdata[cidx]; if (!ownedChunks.get(cidx)) { ownedChunks.set(cidx); chunk = chunk.clone(); mdata[cidx] = chunk; } chunk[eidx] = v; return this; } public final TransientList assoc(Object obj, Object v) { if (!Util.isInteger(obj)) throw new RuntimeException("Vectors must have integer indexes: " + String.valueOf(obj)); return assocN(RT.intCast(obj), v); } public final TransientList conj(Object v) { final int idx = nElems++; final int cidx = idx / 32; final int eidx = idx % 32; Object[][] mdata = data.data; Object[] chunk; if (cidx == mdata.length) { mdata = Arrays.copyOf(mdata, cidx+1); chunk = new Object[4]; mdata[cidx] = chunk; ownedChunks.set(cidx); } else { chunk = mdata[cidx]; if ((!ownedChunks.get(cidx)) || chunk.length <= eidx) { chunk = Arrays.copyOf(chunk, nextPow2(eidx+1)); mdata[cidx] = chunk; ownedChunks.set(cidx); } } chunk[eidx] = v; return this; } public final TransientList pop() { if(nElems == 0) throw new RuntimeException("Attempt to pop empty vector"); final int idx = --nElems; final int cidx = idx / 32; final int eidx = idx % 32; Object[][] mdata = data.data; Object[] chunk = mdata[cidx]; if (!ownedChunks.get(cidx)) { chunk = chunk.clone(); ownedChunks.set(cidx); mdata[cidx] = chunk; } //Release any outstanding references. chunk[eidx] = null; return this; } public final ImmutList persistent() { return new ImmutList(0, nElems, data); } } ================================================ FILE: java/ham_fisted/TransientLongHashMap.java ================================================ package ham_fisted; import java.util.Map; import clojure.lang.Indexed; import clojure.lang.ITransientMap; import clojure.lang.ITransientAssociative2; import clojure.lang.IObj; import clojure.lang.IPersistentMap; import clojure.lang.IFn; public class TransientLongHashMap extends ROLongHashMap implements IATransientMap, IObj { public TransientLongHashMap(LongHashMap data) { super(data.loadFactor, data.capacity, data.length, data.data.clone(), data.meta); } public TransientLongHashMap(TransientLongHashMap data, IPersistentMap m) { super(data.loadFactor, data.capacity, data.length, data.data, m); } public TransientLongHashMap assoc(Object kk, Object val) { long key = Casts.longCast(kk); int hc = hash(key); int idx = hc & mask; LongHashNode e = data[idx]; data[idx] = e != null ? e.assoc(this, key, hc, val) : newNode(key, hc, val); return this; } public TransientLongHashMap without(Object kk) { long key = Casts.longCast(kk); int hc = hash(key); int idx = hc & mask; LongHashNode e = data[idx]; if(e!=null) data[idx] = e.dissoc(this, key); return this; } public PersistentLongHashMap persistent() { return new PersistentLongHashMap(this); } public TransientLongHashMap withMeta(IPersistentMap m) { return new TransientLongHashMap(this, m); } } ================================================ FILE: java/ham_fisted/TreeList.java ================================================ package ham_fisted; import java.util.Arrays; import java.util.BitSet; import java.util.List; import java.util.Comparator; import java.util.Iterator; import java.util.NoSuchElementException; import java.util.function.BiFunction; import java.util.ArrayList; import clojure.lang.IPersistentVector; import clojure.lang.IFn; import clojure.lang.RT; import clojure.lang.IDeref; import clojure.lang.Util; import clojure.lang.IPersistentMap; import clojure.lang.IEditableCollection; public class TreeList extends TreeListBase implements IPersistentVector, IEditableCollection { final IPersistentMap meta; int _hash = 0; public TreeList(Object root, Object[] tail, IPersistentMap meta, int shift, int count) { super(root, tail, shift, count); this.meta = meta; } public TreeList(IPersistentMap meta) { this(new Leaf(), new Object[0], meta, 0, 0); } public TreeList() { this(null); } public TreeList(TreeListBase other, IPersistentMap meta) { super(other); this.meta = meta; } public static TreeList EMPTY = new TreeList(); public TreeList cons(Object d) { final int tlen = tail.length; final int newCount = count+1; if(tlen == 32) { Object rv = shift == 0 ? ((Leaf)root).cons(tail) : ((Branch)root).cons(shift, tail); if(rv instanceof Object[]) { return new TreeList(new Branch(null, (Object[])rv), new Object[]{d}, meta, shift+1, newCount); } else { return new TreeList(rv, new Object[]{d}, meta, shift, newCount); } } else { Object[] newTail = new Object[tlen+1]; System.arraycopy(tail, 0, newTail, 0, tail.length); newTail[tlen] = d; return new TreeList(root, newTail, meta, shift, newCount); } } public int hasheq() { if(_hash == 0) _hash = CljHash.listHasheq(this); return _hash; } public int hashCode() { return hasheq(); } public TreeList consAll(Iter data) { int tlen = tail.length; Object[] newTail = Arrays.copyOf(tail, tailWidth); while(data != null && tlen < tailWidth) { newTail[tlen++] = data.get(); data = data.next(); } if(data == null) return new TreeList(root, Arrays.copyOf(newTail, tlen), meta, shift, count); Object newRoot = root; int newShift = shift; int newCount = count; final int numSiblings = branchWidth-1; data = Iter.prepend(newTail, data); do { ConsAllResult res = shift == 0 ? ((Leaf)newRoot).consAll(null, numSiblings, data) : ((Branch)newRoot).consAll(null, numSiblings, shift, data); data = res.nextData; newCount += res.added; newTail = res.tail; Object[] nodes = res.nodes; if(nodes.length == 1) { newRoot = nodes[0]; } else { newRoot = new Branch(null, res.nodes); newShift += 1; } //should assert here that data is null of tail is nonempty } while(data != null); return new TreeList(newRoot, newTail, meta, newShift, newCount); } public TreeList withMeta(IPersistentMap newMeta) { return new TreeList(root, tail, newMeta, shift, count); } public IPersistentMap meta() { return meta; } public TreeList empty() { return EMPTY; } public TreeList assocN(int idx, Object obj) { if(idx == count) return cons(obj); checkIndex(idx, count); int cutoff = count - tail.length; if( idx < cutoff) { return new TreeList(shift == 0 ? ((Leaf)root).assocN(null, idx, obj, null) : ((Branch)root).assocN(null, shift,idx,obj, null), tail, meta, shift, count); } else { Object[] newTail = Arrays.copyOf(tail, tail.length); newTail[idx % tailWidth] = obj; return new TreeList(root, newTail, meta, shift, count); } } public TreeList assoc(Object idx, Object o) { return assocN(RT.intCast(idx), o); } public TreeList pop() { if (count == 0) throw new IllegalStateException("Can't pop empty vector"); if (count == 1) return EMPTY; Object[] newTail; Object newRoot; int newShift = shift; if(tail.length != 0) { newRoot = root; newShift = shift; newTail = Arrays.copyOf(tail, tail.length-1); } else { SublistResult res = shift == 0 ? ((Leaf)root).pop(null) : ((Branch)root).pop(null, shift); newRoot = res.node; newTail = Arrays.copyOf(res.tail, tailWidth-1); while(shift > 0) { Branch newBranch = (Branch)newRoot; if(newBranch.data.length > 1) break; newRoot = newBranch.data[0]; newShift--; } } return new TreeList(newRoot, newTail, meta, newShift, count-1); } public Object peek() { if (count == 0) return null; return get(count-1); } public MutTreeList asTransient() { return new MutTreeList(this, meta()); } public IPersistentVector immut() { return this; } public static TreeList create(boolean owning, IPersistentMap meta, Object[] data) { int nTails = (data.length + tailWidth - 1) / tailWidth; if(nTails == 1) { return new TreeList(Leaf.EMPTY, owning ? data : data.clone(), meta, 0, data.length); } return MutTreeList.create(owning, meta, data).persistent(); } public static TreeList create(IPersistentMap meta, Object[] tail, Object obj) { if(tail.length != tailWidth) throw new RuntimeException("Tail must be exactly 32 len"); return new TreeList(new Leaf(null, new Object[][] { tail }), new Object[] { obj }, meta, 0, 33); } } ================================================ FILE: java/ham_fisted/TreeListBase.java ================================================ package ham_fisted; import java.util.Iterator; import java.util.Arrays; import java.util.ArrayList; import java.util.Spliterator; import java.util.NoSuchElementException; import java.util.function.BiFunction; import java.util.function.Consumer; import clojure.lang.RT; import clojure.lang.IDeref; import clojure.lang.IFn; import clojure.lang.IPersistentVector; import clojure.lang.Box; public class TreeListBase implements IMutList { public static final int branchWidth = 32; public static final int leafWidth = branchWidth; public static final int tailWidth = branchWidth; public static final int shiftWidth = Integer.numberOfTrailingZeros(branchWidth); public static class SublistResult { public final Object node; public final Object[] tail; public SublistResult(Object node, Object[] tail) { this.node = node; this.tail = tail; } } public static int checkIndex(int idx, int nElems) { if (idx < 0 || idx >= nElems) throw new IndexOutOfBoundsException("Index: " + String.valueOf(idx) + " is out of range 0-" + String.valueOf(nElems)); return idx; } public static final void sublistCheck(long sidx, long eidx, long nElems) { if(sidx < 0 || sidx > nElems) throw new IndexOutOfBoundsException("Start index out of range: start-index(" + String.valueOf(sidx) +"), n-elems(" + String.valueOf(nElems) + ")"); if(eidx < 0 || eidx > nElems) throw new IndexOutOfBoundsException("End index out of range: end-index(" + String.valueOf(eidx) +"), n-elems(" + String.valueOf(nElems) + ")"); if(eidx < sidx) throw new IndexOutOfBoundsException("End index underflow: end-index(" + String.valueOf(eidx) +") < start-index(" + String.valueOf(sidx) + ")"); } public static final Object[][] emptyObjAryAry = new Object[0][]; public static final Object[] emptyObjAry = new Object[0]; public static class ConsAllResult { public final Object[] nodes; public final Iter nextData; public final Object[] tail; public final int added; public ConsAllResult(Object[] nodes, Iter nextData, Object[] tail, int added) { this.nodes = nodes; this.nextData = nextData; this.tail = tail; this.added = added; } } public static interface INode { public void forEachRemaining(int shift, int sidx, int eidx, Consumer cc); public Object reduce(int shift, int sidx, int eidx, IFn rfn, Object acc); } public static int outerEidxSpan(int sidx, int eidx, int level) { if(sidx == eidx) return 0; int ssidx = sidx/level; int eeidx = (eidx-1)/level; return ((eeidx - ssidx) + 1); } public static int localEidx(int aryIdx, int eidx, int level) { return Math.min(level, eidx - (aryIdx * level)); } public static class Leaf implements INode { final Object owner; Object[][] data; public Object[][] data() { return this.data; } public Leaf() { this.owner = null; this.data = emptyObjAryAry; } public Leaf(Object owner, Object[][] data) { this.owner = owner; this.data = data; } public Leaf(Object[][] data) { this(null, data); } public Leaf(Object[] tail) { this.owner = null; this.data = new Object[][]{ tail }; } public Object cons(Object owner, Object[] tail) { boolean force = owner == null || this.owner != owner; if(data.length == leafWidth) { return new Object[]{this, new Leaf(owner, new Object[][] { tail })}; } else { Object[][] newData = Arrays.copyOf(data, data.length+1); newData[data.length]=tail; if(force) return new Leaf(owner, newData); else { this.data = newData; return this; } } } @SuppressWarnings("unchecked") public void forEachRemaining(int shift, int sidx, int eidx, Consumer cc) { int aryIdx = sidx / leafWidth; int aryEnd = aryIdx + outerEidxSpan(sidx, eidx, leafWidth); int aryOffset = sidx % leafWidth; // System.out.println("leaf sidx " + String.valueOf(sidx) + // " eidx " + String.valueOf(eidx) + // " aryIdx " + String.valueOf(aryIdx) + // " aryEnd " + String.valueOf(aryEnd) + // " aryOffset " + String.valueOf(aryOffset) // ); for(; aryIdx < aryEnd; ++aryIdx) { int localEnd = localEidx(aryIdx, eidx, leafWidth); // System.out.println("Local End: " + String.valueOf(localEnd)); Object[] localData = data()[aryIdx]; for(; aryOffset < localEnd; ++aryOffset) cc.accept(localData[aryOffset]); aryOffset = 0; } } public Object reduce(int shift, int sidx, int eidx, IFn rfn, Object acc) { int aryIdx = sidx / leafWidth; int aryEnd = aryIdx + outerEidxSpan(sidx, eidx, leafWidth); int aryOffset = sidx % leafWidth; // System.out.println("leaf sidx " + String.valueOf(sidx) + // " eidx " + String.valueOf(eidx) + // " aryIdx " + String.valueOf(aryIdx) + // " aryEnd " + String.valueOf(aryEnd) + // " aryOffset " + String.valueOf(aryOffset) // ); for(; aryIdx < aryEnd; ++aryIdx) { int localEnd = localEidx(aryIdx, eidx, leafWidth); // System.out.println("Local End: " + String.valueOf(localEnd)); Object[] localData = data()[aryIdx]; for(; aryOffset < localEnd; ++aryOffset) { acc = rfn.invoke(acc, localData[aryOffset]); if(RT.isReduced(acc)) return acc; } aryOffset = 0; } return acc; } public Object cons(Object[] tail) { return cons(null, tail); } public Object add(Object owner, Object[] tail) { return cons(owner, tail); } public ConsAllResult consAll(Object owner, int maxSiblings, Iter dataIter) { int maxTails = leafWidth - data.length + leafWidth * maxSiblings; ArrayList tails = new ArrayList(); Object[] tail = new Object[32]; int nTail = 0; int added = 0; while(tails.size() < maxTails && dataIter != null) { if(nTail == tailWidth) tails.add(tail.clone()); for(nTail = 0; nTail < tailWidth && dataIter != null; ++nTail) { tail[nTail++] = dataIter.get(); dataIter = dataIter.next(); ++added; } nTail = 0; } //Rectify tail if(nTail == 0) tail = new Object[0]; else if (nTail != tailWidth) tail = Arrays.copyOf(tail, nTail); if(tails.isEmpty()) return new ConsAllResult(new Object[]{this}, dataIter, tail, added); int totalTails = tails.size(); int nLocalTails = Math.min(leafWidth - data.length, totalTails); Object[][] newData = Arrays.copyOf(data, nLocalTails); for(int idx = data.length; idx < nLocalTails; ++idx) newData[idx] = tails.get(idx-data.length); int nOtherLeaves = (totalTails - nLocalTails + leafWidth - 1) / leafWidth; Object[] leaves = new Object[1 + nOtherLeaves]; leaves[0] = new Leaf(owner, newData); int leafIdx = 1; for(int idx = nLocalTails; idx < totalTails; idx += leafWidth) { int nextIdx = Math.min(totalTails, idx + leafWidth); int nLeafTails = nextIdx - idx; leaves[leafIdx++] = new Leaf(owner, tails.subList(idx, nextIdx).toArray(emptyObjAryAry)); } return new ConsAllResult(leaves, dataIter, tail, added); } public Object[] getArray(int idx) { return data[idx/leafWidth]; } public Leaf assocN(Object owner, int idx, Object obj, Box oldVal) { boolean force = owner == null || this.owner != owner; int localIdx = idx/leafWidth; Object[] entry = force ? Arrays.copyOf(data[localIdx], tailWidth) : data[localIdx]; int objIdx = idx % tailWidth; if(oldVal != null) oldVal.val = entry[objIdx]; entry[objIdx] = obj; if(force) { Object[][] newData = Arrays.copyOf(data, data.length); newData[localIdx] = entry; return new Leaf(owner, newData); } else { return this; } } public SublistResult pop(Object owner) { int dlen = data.length; Object[] lastTail = data[dlen-1]; Object[][] newD; if(dlen-1 == 0) { newD = new Object[0][]; } else { newD = Arrays.copyOf(data, dlen-1); } boolean force = owner == null || this.owner != owner; Leaf newLeaf; if(force) { newLeaf = new Leaf(owner, newD); } else { this.data = newD; newLeaf = this; } return new SublistResult(newLeaf, lastTail); } public boolean isEmpty() { return data.length == 0; } //Assumption here is that sidx lies exactly on a tailWidth boundary. public SublistResult subList(int sidx, int eidx) { int dataSidx = sidx / tailWidth; int numTails = ((eidx - sidx) + (tailWidth-1))/tailWidth; int dataEidx = dataSidx + numTails; int len = eidx - sidx; int leftover = len % tailWidth; /* System.out.println("Leaf SubList sidx " + String.valueOf(sidx) + " eidx " + String.valueOf(eidx) */ /* + " dataSidx " + String.valueOf(dataSidx) + " dataEidx " + String.valueOf(dataEidx) */ /* + " leftover " + leftover); */ if(sidx == 0 && eidx == (data.length * tailWidth)) return new SublistResult(this, new Object[0]); if(leftover == 0) { return new SublistResult(new Leaf(Arrays.copyOfRange(data, dataSidx, dataEidx)), new Object[0]); } else { Object[] tail = data[dataEidx-1]; return new SublistResult(new Leaf(Arrays.copyOfRange(data, dataSidx, dataEidx-1)) , Arrays.copyOf(tail, leftover)); } } public static final Leaf EMPTY = new Leaf(); } public static class Branch implements INode { final Object owner; Object[] data; //either branch or leaf public Object[] data() { return this.data; } public Branch() { this.owner = null; data = new Object[0]; } public Branch(Object owner, Object[] data) { this.owner = owner; this.data = data; } public Branch(Leaf leaf) { this.owner = null; this.data = new Object[]{leaf}; } public Branch(Branch branch) { this.owner = null; this.data = new Object[]{branch}; } public Branch(Object owner, int shift, Object[] tail) { this(owner, new Object[] { shift == 1 ? new Leaf(tail) : new Branch(shift-1, tail) } ); } @SuppressWarnings("unchecked") public void forEachRemaining(int shift, int sidx, int eidx, Consumer cc) { /* System.out.println("brach sidx " + String.valueOf(sidx) + */ /* " eidx " + String.valueOf(eidx)); */ int shiftAmt = shift * shiftWidth; int level = branchWidth << shiftAmt; int aryIdx = sidx / level; int aryEnd = aryIdx + outerEidxSpan(sidx, eidx, level); int aryOffset = sidx % level; for(; aryIdx < aryEnd; ++aryIdx) { int localEnd = localEidx(aryIdx, eidx, level); ((INode)data[aryIdx]).forEachRemaining(shift-1, aryOffset, localEnd, cc); aryOffset = 0; } } public Object reduce(int shift, int sidx, int eidx, IFn rfn, Object acc) { /* System.out.println("brach sidx " + String.valueOf(sidx) + */ /* " eidx " + String.valueOf(eidx)); */ int shiftAmt = shift * shiftWidth; int level = branchWidth << shiftAmt; int aryIdx = sidx / level; int aryEnd = aryIdx + outerEidxSpan(sidx, eidx, level); int aryOffset = sidx % level; for(; aryIdx < aryEnd; ++aryIdx) { int localEnd = localEidx(aryIdx, eidx, level); acc = ((INode)data[aryIdx]).reduce(shift-1, aryOffset, localEnd, rfn, acc); if(RT.isReduced(acc)) return acc; aryOffset = 0; } return acc; } public Object cons(Object owner, int shift, Object[] tail) { if(data.length == 0) return new Branch(owner, shift, tail); int lastIdx = data.length-1; Object last = data[lastIdx]; boolean force = owner == null || this.owner != owner; Object res = shift == 1 ? ((Leaf)last).cons(owner, tail) : ((Branch)last).cons(owner, shift-1, tail); if(res instanceof Object[]) { Object newNode = ((Object[])res)[1]; if(data.length == branchWidth) { return new Object[] { this, new Branch( owner, new Object[] { newNode } ) }; } else { Object[] newData = Arrays.copyOf(data, data.length+1); newData[data.length] = newNode; if(force) { return new Branch(owner, newData); } else { data = newData; return this; } } } Object[] newData = force ? data.clone() : data; newData[newData.length-1] = res; if(force) { return new Branch(owner, newData); } else { return this; } } public Object cons(int shift, Object[] tail) { return cons(null, shift, tail); } public Object add(Object owner, int shift, Object[] tail) { return cons(owner, shift, tail); } public ConsAllResult consAll(Object owner, int shift, int maxSiblings, Iter dataIter) { int maxChildren = branchWidth - data.length + branchWidth * maxSiblings; Object lastNode = data[data.length-1]; ConsAllResult res = shift == 1 ? ((Leaf)lastNode).consAll(owner, maxChildren, dataIter) : ((Branch)lastNode).consAll(owner, shift-1, maxChildren, dataIter); int numChildren = res.nodes.length; Object[] tail = res.tail; dataIter = res.nextData; int added = res.added; if(res.nodes.length == 1 && res.nodes[0] == lastNode) return new ConsAllResult(new Object[]{this}, dataIter, tail, added); int nLocalNodes = Math.min(numChildren, branchWidth - data.length); Object[] newData = Arrays.copyOf(data, data.length + nLocalNodes); int nNewBranches = (numChildren - nLocalNodes + branchWidth -1)/branchWidth; Object[] rv = new Object[1 + nNewBranches]; rv[0] = new Branch(owner, newData); for(int idx = 0; idx < nNewBranches; ++idx) { int copyBegin = nLocalNodes + (idx * branchWidth); int copyEnd = Math.min(copyBegin + branchWidth, numChildren); rv[idx+1] = new Branch(owner, Arrays.copyOfRange(data, copyBegin, copyEnd)); } return new ConsAllResult(rv, dataIter, tail, added); } public Object getNode(int shift, int idx) { int shiftAmt = shift * shiftWidth; int level = branchWidth << shiftAmt; int localIdx = idx / level; return data[localIdx]; } public Object[] getArray(int shift, int idx) { int shiftAmt = shift * shiftWidth; int level = branchWidth << shiftAmt; int localIdx = idx / level; int leftover = idx % level; Object item = data[localIdx]; return shift == 1 ? ((Leaf)item).getArray(leftover) : ((Branch)item).getArray(shift-1, leftover); } public Branch assocN(Object owner, int shift, int idx, Object obj, Box oldVal) { boolean force = owner == null || this.owner != owner; int shiftAmt = shift * shiftWidth; int level = branchWidth << shiftAmt; int localIdx = idx / level; int leftover = idx % level; Object item = data[localIdx]; Object newItem = shift == 1 ? ((Leaf)item).assocN(owner, leftover, obj, oldVal) : ((Branch)item).assocN(owner, shift-1, leftover, obj, oldVal); Object[] newData = force ? Arrays.copyOf(data, data.length) : data; if(newItem != item) { newData[localIdx] = newItem; } return force ? new Branch(owner, newData) : this; } public boolean isEmpty() { return data.length == 0; } public SublistResult pop(Object owner, int shift) { boolean force = owner == null || owner != this.owner; int dlen = data.length; Object lastObj = data[dlen-1]; SublistResult res = shift == 1 ? ((Leaf)lastObj).pop(owner) : ((Branch)lastObj).pop(owner, shift-1); Object node = res.node; Object[] newTail = res.tail; boolean empty = shift == 1 ? ((Leaf)node).isEmpty() : ((Branch)node).isEmpty(); Object[] newD; if(empty) { if(dlen-1 == 0) { newD = new Object[0]; } else { newD = Arrays.copyOf(data, dlen-1); } }else { newD = force ? data.clone() : data; } int newLen = newD.length; if(!empty) newD[newLen-1] = node; Branch newBranch; if(force) { newBranch = new Branch(owner, newD); } else { this.data = newD; newBranch = this; } return new SublistResult(newBranch, newTail); } public static final BiFunction leafMergeRight = new BiFunction() { public Object apply(Object lhs, Object rhs) { Leaf ll = (Leaf)lhs; Leaf rr = (Leaf)rhs; // System.out.println("ll " + String.valueOf(ll.data.length) + " rr " + String.valueOf(rr.data.length)); if(ll.data.length < leafWidth) { int totalWidth = ll.data.length + rr.data.length; int newLen = Math.min(leafWidth, ll.data.length + rr.data.length); Object[][] newLeafData = Arrays.copyOf(ll.data, newLen); System.arraycopy(rr.data, 0, newLeafData, ll.data.length, newLen - ll.data.length); ll = new Leaf(newLeafData); int newRightLen = totalWidth - newLen; if(newRightLen > 0) rr = new Leaf(Arrays.copyOfRange(rr.data, rr.data.length-newRightLen, rr.data.length)); else rr = null; } return rr == null ? ll : new Object[] {ll, rr}; } }; @SuppressWarnings("unchecked") public static final BiFunction branchMergeRight = new BiFunction() { public Object apply(Object lhs, Object rhs) { Branch ll = (Branch)lhs; Branch rr = (Branch)rhs; int totalWidth = ll.data.length + rr.data.length; int newLen = Math.min(branchWidth, ll.data.length + rr.data.length); ArrayList newBranchData = new ArrayList(newLen); newBranchData.addAll(Arrays.asList(ll.data)); Object lastObj = newBranchData.get(newBranchData.size()-1); BiFunction mergeFn = lastObj instanceof Leaf ? leafMergeRight : branchMergeRight; for(int idx = 0; idx < rr.data.length; ++idx) { Object merged = mergeFn.apply(lastObj, rr.data[idx]); int lastidx = newBranchData.size()-1; if(merged instanceof Object[]) { final Object[] mm = (Object[])merged; newBranchData.set(lastidx, mm[0]); newBranchData.add(mm[1]); lastObj = mm[1]; } else { newBranchData.set(lastidx, merged); lastObj = merged; } } int newDataSize = newBranchData.size(); int llen = Math.min(branchWidth, newDataSize); ll = new Branch(null, newBranchData.subList(0, llen).toArray()); if(llen < newDataSize) { rr = new Branch(null, newBranchData.subList(llen, newDataSize).toArray()); } else { rr = null; } return rr == null ? ll : new Object[] {ll, rr}; } }; @SuppressWarnings("unchecked") public SublistResult subList(int shift, int sidx, int eidx) { int shiftAmt = shift * shiftWidth; int level = branchWidth << shiftAmt; int localSidx = sidx / level; int numNodes = (eidx - (localSidx * level) + (level - 1)) / level; int localEidx = localSidx + numNodes; ArrayList newData = new ArrayList(numNodes); Object[] newTail = null; final BiFunction mergeFunction = shift == 1 ? leafMergeRight : branchMergeRight; for(int nodeIdx = 0; nodeIdx < numNodes; ++nodeIdx) { int dataIdx = localSidx + nodeIdx; int nodeStart = dataIdx * level; int nodeEnd = nodeStart + level; int nodeSidx = Math.max(nodeStart, sidx); int nodeEidx = Math.min(nodeEnd, eidx); Object dataNode = data[dataIdx]; if(nodeStart == nodeSidx && nodeEnd == nodeEidx) { newData.add(dataNode); } else { int subSidx = nodeSidx - nodeStart; int subEidx = nodeEidx - nodeStart; SublistResult res = shift == 1 ? ((Leaf)dataNode).subList(subSidx, subEidx) : ((Branch)dataNode).subList(shift-1, subSidx, subEidx); newData.add(res.node); newTail = res.tail; } int sz = newData.size(); if(sz > 1 ) { Object merged = mergeFunction.apply(newData.get(sz-2), newData.get(sz-1)); if(merged instanceof Object[]) { Object[] mm = (Object[])merged; newData.set(sz-2, mm[0]); newData.set(sz-1, mm[1]); } else { newData.set(sz-2, merged); //right node was completely consumed by left newData.remove(sz-1); } } } return new SublistResult(new Branch(owner, newData.toArray()), newTail); } }; Object[] tail; Object root; int count; int shift; public Object[] tail() { return tail; } public Object[] validTail() { return tail; } public Object root() { return root; } public int length() { return count; } public int count() { return count; } public int size() { return count; } public int shift() { return shift; } public int nTail() { return tail.length; } public TreeListBase(Object root, Object[] tail, int shift, int count) { this.tail = tail; this.root = root; this.count = count; this.shift = shift; } public TreeListBase() { this.tail = new Object[0]; this.root = new Leaf(); this.count = 0; this.shift = 0; } public TreeListBase(TreeListBase other) { this.tail = other.validTail(); this.root = other.root; this.count = other.count; this.shift = other.shift; } public static class ArrayIterator implements Iterator { final TreeListBase data; int arySidx; final int aryEidx; public ArrayIterator(TreeListBase data, int arySidx, int aryEidx) { this.data = data; this.arySidx = arySidx; this.aryEidx = aryEidx; } public boolean hasNext() { return arySidx < aryEidx; } public Object[] next() { if(arySidx >= aryEidx) throw new NoSuchElementException(); Object[] rv = data.getArray(arySidx * tailWidth); ++arySidx; return rv; } } public Iterator arrayIterator(int sidx, int eidx) { int arySidx = sidx / tailWidth; int nArrays = (eidx - (arySidx * tailWidth) + (tailWidth - 1))/tailWidth; int aryEidx = arySidx + nArrays; return new ArrayIterator(this, arySidx, aryEidx); } public Object[] getArray(int idx) { int cutoff = count - nTail(); return (idx < cutoff) ? (shift == 0) ? ((Leaf)root).getArray(idx) : ((Branch)root).getArray(shift, idx) : tail; } public Object get(int idx) { checkIndex(idx, count); return getArray(idx)[idx % 32]; } @SuppressWarnings("unchecked") public void forEachRemaining(int sidx, int eidx, Consumer cc) { sublistCheck(sidx, eidx, size()); int cutoff = count - nTail(); if(sidx < cutoff) ((INode)root).forEachRemaining(shift, sidx, Math.min(eidx, cutoff), cc); if(eidx > cutoff) { int cidx = eidx - cutoff; for(int idx = Math.max(sidx, cutoff) - cutoff; idx < cidx; ++idx) cc.accept(tail[idx]); } } public Object reduce(int sidx, int eidx, IFn rfn, Object acc) { sublistCheck(sidx, eidx, size()); int cutoff = count - nTail(); if(sidx < cutoff) acc = ((INode)root).reduce(shift, sidx, Math.min(eidx, cutoff), rfn, acc); if (RT.isReduced(acc)) return ((IDeref)acc).deref(); if(eidx > cutoff) { int cidx = eidx - cutoff; for(int idx = Math.max(sidx, cutoff) - cutoff; idx < cidx; ++idx) { acc = rfn.invoke(acc, tail[idx]); if (RT.isReduced(acc)) return acc; } } return acc; } public Object reduce(IFn rfn, Object acc) { return reduce(0, count, rfn, acc); } public Object[] fillArray(int sidx, int eidx, Object[] data) { Iterator iter = arrayIterator(sidx, eidx); int arySidx = sidx - (sidx % tailWidth); int writeOff = 0; while(iter.hasNext()) { int startoff = Math.max(sidx, arySidx); int endoff = Math.min(eidx, arySidx + tailWidth); int copyLen = endoff - startoff; System.arraycopy(iter.next(), startoff % 32, data, writeOff, copyLen); writeOff += copyLen; arySidx += tailWidth; } return data; } public Object[] fillArray(Object[] data) { return fillArray(0, count, data); } public static class TreeListSpliterator implements Spliterator { public final TreeListBase data; int sidx; int eidx; public TreeListSpliterator(TreeListBase data, int sidx, int eidx) { this.data = data; this.sidx = sidx; this.eidx = eidx; } public int characteristics() { return Spliterator.ORDERED | Spliterator.SIZED | Spliterator.SUBSIZED | Spliterator.IMMUTABLE; } public Spliterator trySplit() { int ne = (eidx - sidx); if ( ne > 2) { int middle = sidx + (ne / 2); int eeidx = eidx; eidx = middle; return new TreeListSpliterator(data, middle, eeidx); } return null; } public long estimateSize() { return eidx - sidx; } @SuppressWarnings("unchecked") public boolean tryAdvance(Consumer c) { if(sidx < eidx) { c.accept(data.get(sidx)); ++sidx; return true; } return false; } public void forEachRemaining(Consumer c) { data.forEachRemaining(sidx, eidx, c); sidx = eidx; } } public Spliterator spliterator(int sidx, int eidx) { sublistCheck(sidx, eidx, size()); return new TreeListSpliterator(this, sidx, eidx); } public Spliterator spliterator() { return spliterator(0, size()); } public static class SubList implements IMutList, IPersistentVector { int offset; //<= 32 TreeList data; public SubList(int offset, TreeList data) { this.offset = offset; this.data = data; } public int offset() { return offset; } public TreeList data() { return data; } public int count() { return data.count() - offset; } public int size() { return data.count() - offset; } public int length() { return data.count() - offset; } public Object get(int idx) { checkIndex(idx, count()); return data.get(idx + offset); } public SubList cons(Object a) { return new SubList(offset, data.cons(a)); } public SubList assocN(int idx, Object a) { return new SubList(offset, data.assocN(idx, a)); } public IPersistentVector pop() { int cnt = count(); if ( cnt == 0 ) throw new UnsupportedOperationException("Underflow"); if ( cnt == 1 ) return TreeList.EMPTY; return new SubList(offset, data.pop()); } public Object peek() { return data.peek(); } public IMutList subList(int sidx, int eidx) { sublistCheck(sidx, eidx, size()); return data.subList(sidx+offset, eidx+offset); } public Object[] fillArray(Object[] ary) { return data.fillArray(offset, data.count(), ary); } public Object reduce(IFn rfn, Object acc) { return data.reduce(offset, data.count(), rfn, acc); } public Spliterator spliterator() { return data.spliterator(offset, data.count()); } public static IMutList create(int offset, TreeList data) { if(offset == 0) return data; return new SubList(offset, data); } } public static Object[] nonNull(Object[] data) { return data == null ? emptyObjAry : data; } public IMutList subList(int sidx, int eidx) { sublistCheck(sidx, eidx, size()); int tlen = nTail(); int cutoff = count - nTail(); if(sidx == 0 && eidx == count) return this; if(sidx == eidx) return TreeList.EMPTY; Object[] tailPart = null; if(eidx > cutoff) { int tailSidx = Math.max(sidx, cutoff); tailPart = Arrays.copyOfRange(tail, tailSidx - cutoff, eidx - cutoff); } int newShift = shift; Object newNode = null; Object[] treeTail = null; int offset = sidx % tailWidth; int roundedSidx = sidx - offset; if(sidx < cutoff) { int treeEidx = Math.min(eidx, cutoff); SublistResult treePart = shift==0 ? ((Leaf)root).subList(roundedSidx, treeEidx) : ((Branch)root).subList(shift, roundedSidx, treeEidx); newNode = treePart.node; treeTail = treePart.tail; while(newShift > 0) { Branch b = (Branch)newNode; if(b.data.length == 1) { newNode = b.data[0]; newShift--; } else { break; } } } if(newNode == null) return new TreeList(new Leaf(), tailPart, meta(), 0, tailPart.length); TreeList newList = new TreeList(newNode, nonNull(tailPart == null ? treeTail : tailPart), meta(), newShift, eidx-roundedSidx); return SubList.create(offset, newList); } } ================================================ FILE: java/ham_fisted/TypedList.java ================================================ package ham_fisted; public interface TypedList { default Class containedType() { return Object.class; } } ================================================ FILE: java/ham_fisted/TypedNth.java ================================================ package ham_fisted; import clojure.lang.RT; public class TypedNth { public static double dnth(Object v, long idx) { return (v instanceof double[]) ? dnth((double[])v, idx) : Casts.doubleCast(RT.nth(v, (int)idx)); } public static double dnth(double[] v, long idx) { return v[(int)idx]; } public static float fnth(Object v, long idx) { return (v instanceof float[]) ? fnth((float[])v, idx) : Casts.floatCast(RT.nth(v, (int)idx)); } public static float fnth(float[] v, long idx) { return v[(int)idx]; } public static long lnth(Object v, long idx) { return (v instanceof long[]) ? lnth((long[])v, idx) : Casts.longCast(RT.nth(v, (int)idx)); } public static long lnth(long[] v, long idx) { return v[(int)idx]; } public static int inth(Object v, long idx) { return (v instanceof int[]) ? inth((int[])v, idx) : Casts.intCast(RT.nth(v, (int)idx)); } public static int inth(int[] v, long idx) { return v[(int)idx]; } } ================================================ FILE: java/ham_fisted/UnsharedHashMap.java ================================================ package ham_fisted; import java.util.List; import java.util.Map; import clojure.lang.IPersistentMap; import clojure.lang.ITransientMap; import clojure.lang.ITransientAssociative2; import clojure.lang.Indexed; public class UnsharedHashMap extends HashMap implements MutableMap, IATransientMap { public UnsharedHashMap() { super(null); } public UnsharedHashMap(IPersistentMap meta) { super(meta); } public UnsharedHashMap(IPersistentMap meta, int capacity) { super(0.75f, Math.max(4, IntegerOps.nextPow2((int)(capacity/0.75f))), 0, null, meta); } public static UnsharedHashMap create(Object[] data) { final int l = data.length; if((l%2) != 0) throw new RuntimeException("Data length not evenly divisible by 2"); final UnsharedHashMap rv = new UnsharedHashMap(null, l/2); final HashNode[] d = rv.data; final int m = rv.mask; for(int idx = 0; idx < l; idx += 2) { final Object k = data[idx]; final Object v = data[idx+1]; final int hc = rv.hash(k); final int didx = hc & m; HashNode lf, init = d[didx]; for(lf = init; lf != null && !(lf.k == k || rv.equals(lf.k, k)); lf = lf.nextNode); if(lf != null) { lf.v = v; } else { final HashNode newNode = rv.newNode(k, hc, v); newNode.nextNode = d[didx]; d[didx] = newNode; } } return rv; } public static UnsharedHashMap createInterleaved(List data) { final int l = data.size(); if((l%2) != 0) throw new RuntimeException("Data length not evenly divisible by 2"); final UnsharedHashMap rv = new UnsharedHashMap(null, l/2); final HashNode[] d = rv.data; final int m = rv.mask; for(int idx = 0; idx < l; idx += 2) { final Object k = data.get(idx); final Object v = data.get(idx+1); final int hc = rv.hash(k); final int didx = hc & m; HashNode lf, init = d[didx]; for(lf = init; lf != null && !(lf.k == k || rv.equals(lf.k, k)); lf = lf.nextNode); if(lf != null) { lf.v = v; } else { final HashNode newNode = rv.newNode(k, hc, v); newNode.nextNode = d[didx]; d[didx] = newNode; } } return rv; } public UnsharedHashMap assoc(Object key, Object val) { put(key,val); return this; } public UnsharedHashMap without(Object key) { remove(key); return this; } public PersistentHashMap persistent() { return new PersistentHashMap(this); } } ================================================ FILE: java/ham_fisted/UnsharedHashSet.java ================================================ package ham_fisted; import java.util.Collection; import clojure.lang.IPersistentMap; public class UnsharedHashSet extends HashSet implements IATransientSet { public UnsharedHashSet(IPersistentMap meta) { super(meta); } public UnsharedHashSet conj(Object key) { add(key); return this; } public UnsharedHashSet disjoin(Object key) { remove(key); return this; } public HashSet union(Collection rhs) { addAll(rhs); return this; } public PersistentHashSet persistent() { return new PersistentHashSet(this, meta); } } ================================================ FILE: java/ham_fisted/UnsharedLongHashMap.java ================================================ package ham_fisted; import java.util.List; import java.util.Map; import clojure.lang.IPersistentMap; import clojure.lang.ITransientMap; import clojure.lang.ITransientAssociative2; import clojure.lang.Indexed; public class UnsharedLongHashMap extends LongHashMap implements MutableMap, IATransientMap { public UnsharedLongHashMap() { super(null); } public UnsharedLongHashMap(IPersistentMap meta) { super(meta); } public UnsharedLongHashMap(IPersistentMap meta, int capacity) { super(0.75f, Math.max(4, IntegerOps.nextPow2((int)(capacity/0.75f))), 0, null, meta); } public static UnsharedLongHashMap create(Object[] data) { final int l = data.length; if((l%2) != 0) throw new RuntimeException("Data length not evenly divisible by 2"); final UnsharedLongHashMap rv = new UnsharedLongHashMap(null, l/2); final LongHashNode[] d = rv.data; final int m = rv.mask; for(int idx = 0; idx < l; idx += 2) { final long k = Casts.longCast(data[idx]); final Object v = data[idx+1]; final int hc = rv.hash(k); final int didx = hc & m; LongHashNode lf, init = d[didx]; for(lf = init; lf != null && !(lf.k == k); lf = lf.nextNode); if(lf != null) { lf.v = v; } else { final LongHashNode newNode = rv.newNode(k, hc, v); newNode.nextNode = d[didx]; d[didx] = newNode; } } return rv; } public static UnsharedLongHashMap createInterleaved(List data) { final int l = data.size(); if((l%2) != 0) throw new RuntimeException("Data length not evenly divisible by 2"); final UnsharedLongHashMap rv = new UnsharedLongHashMap(null, l/2); final LongHashNode[] d = rv.data; final int m = rv.mask; for(int idx = 0; idx < l; idx += 2) { final long k = Casts.longCast(data.get(idx)); final Object v = data.get(idx+1); final int hc = rv.hash(k); final int didx = hc & m; LongHashNode lf, init = d[didx]; for(lf = init; lf != null && !(lf.k == k); lf = lf.nextNode); if(lf != null) { lf.v = v; } else { final LongHashNode newNode = rv.newNode(k, hc, v); newNode.nextNode = d[didx]; d[didx] = newNode; } } return rv; } public UnsharedLongHashMap assoc(Object key, Object val) { put(key,val); return this; } public UnsharedLongHashMap without(Object key) { remove(key); return this; } public PersistentLongHashMap persistent() { return new PersistentLongHashMap(this); } } ================================================ FILE: java/ham_fisted/UpdateValues.java ================================================ package ham_fisted; import java.util.function.Function; import java.util.function.BiFunction; import clojure.lang.IFn; import clojure.lang.ITransientCollection; public interface UpdateValues { public UpdateValues updateValues(BiFunction valueMap); public UpdateValues updateValue(Object key, Function fn); } ================================================ FILE: resources/clj-kondo.exports/cnuernber/ham-fisted/config.edn ================================================ {:lint-as {ham-fisted.defprotocol/defprotocol clojure.core/defprotocol ham-fisted.defprotocol/extend-protocol clojure.core/extend-protocol ham-fisted.defprotocol/extend-type clojure.core/extend-type} :hooks {:analyze-call {ham-fisted.hlet/let hooks.ham-fisted/analyze-hlet-macro ham-fisted.function/function hooks.ham-fisted/analyze-1-arg-fn-macro ham-fisted.function/long-predicate hooks.ham-fisted/analyze-1-arg-fn-macro ham-fisted.function/long-unary-operator hooks.ham-fisted/analyze-1-arg-fn-macro ham-fisted.function/double-predicate hooks.ham-fisted/analyze-1-arg-fn-macro ham-fisted.function/double-unary-operator hooks.ham-fisted/analyze-1-arg-fn-macro ham-fisted.function/obj->long hooks.ham-fisted/analyze-1-arg-fn-macro ham-fisted.function/long->double hooks.ham-fisted/analyze-1-arg-fn-macro ham-fisted.function/bi-function hooks.ham-fisted/analyze-2-arg-fn-macro ham-fisted.function/binary-predicate hooks.ham-fisted/analyze-2-arg-fn-macro ham-fisted.function/long-binary-operator hooks.ham-fisted/analyze-2-arg-fn-macro ham-fisted.function/double-binary-operator hooks.ham-fisted/analyze-2-arg-fn-macro ham-fisted.reduce/long-accumulator hooks.ham-fisted/analyze-2-arg-fn-macro ham-fisted.reduce/double-accumulator hooks.ham-fisted/analyze-2-arg-fn-macro ham-fisted.reduce/indexed-accum hooks.ham-fisted/analyze-indexed-reduce-fn-macro ham-fisted.alists/make-prim-array-list hooks.ham-fisted/analyze-make-prim-array-list-macro ham-fisted.lazy-noncaching/make-readonly-list hooks.ham-fisted/analyze-indexed-make-list-macro}}} ================================================ FILE: resources/clj-kondo.exports/cnuernber/ham-fisted/hooks/ham_fisted.clj_kondo ================================================ (ns hooks.ham-fisted (:require [clj-kondo.hooks-api :as api])) (defn node-value [node] (when node (api/sexpr node))) (defn analyze-hlet-macro [{:keys [:node]}] (let [[bindings & body] (rest (:children node)) new-node (api/list-node (list* (api/token-node 'clojure.core/let) (api/vector-node (concat [(api/token-node 'dbls) (api/token-node 'clojure.core/vec) (api/token-node 'lngs) (api/token-node 'clojure.core/vec) (api/token-node 'lng-fns) (api/token-node 'clojure.core/vec) (api/token-node 'dbl-fns) (api/token-node 'clojure.core/vec) (api/token-node 'obj-fns) (api/token-node 'clojure.core/vec)] (:children bindings))) body))] {:node new-node})) (defn analyze-1-arg-fn-macro [{:keys [:node]}] (let [[arg1 & body] (rest (:children node)) new-node (api/list-node (list* (api/token-node 'clojure.core/fn) (api/vector-node [arg1]) body))] {:node new-node})) (defn analyze-2-arg-fn-macro [{:keys [:node]}] (let [[arg1 arg2 & body] (rest (:children node)) new-node (api/list-node (list* (api/token-node 'clojure.core/fn) (api/vector-node [arg1 arg2]) body))] {:node new-node})) (defn analyze-indexed-reduce-fn-macro [{:keys [:node]}] (let [[acc-arg idx-arg obj-arg & body] (rest (:children node)) new-node (api/list-node (list* (api/token-node 'clojure.core/fn) (api/vector-node [acc-arg (api/vector-node [idx-arg obj-arg])]) body))] {:node new-node})) (defn analyze-indexed-make-list-macro [{:keys [:node]}] (let [children (rest (:children node)) _input-args (drop-last 3 children) [nel-arg idx-arg & body] (take-last 3 children) new-node (api/list-node (list (api/token-node 'clojure.core/map) (api/list-node (list* (api/token-node 'clojure.core/fn) (api/vector-node [idx-arg]) body)) (api/list-node (list (api/token-node 'clojure.core/range) (api/token-node nel-arg)))))] {:node new-node})) (defn analyze-make-prim-array-list-macro [{:keys [:node]}] (let [[lname ary-tag iface getname setname addname set-cast-fn get-cast-fn obj-cast-fn add-all-reduce] (rest (:children node)) new-node (api/list-node (list (api/token-node 'clojure.core/deftype) (vary-meta lname assoc :tag (node-value ary-tag)) (api/vector-node [(api/token-node 'data) (api/token-node 'n-elems) (api/token-node 'm)]) iface (api/list-node (list getname (api/vector-node [(api/token-node '_) (api/token-node 'idx)]) (api/list-node (list get-cast-fn (api/list-node (list (api/token-node 'clojure.core/aget) (api/token-node 'data) (api/token-node 'idx))))))) (api/list-node (list setname (api/vector-node [(api/token-node '_) (api/token-node 'idx) (api/token-node 'v)]) (api/list-node (list (api/token-node 'clojure.core/aset) (api/token-node 'data) (api/token-node 'idx) (api/list-node (list set-cast-fn (api/token-node 'v))))))) (api/list-node (list addname (api/vector-node [(api/token-node '_) (api/token-node 'v)]) (api/list-node (list (api/token-node 'clojure.core/aset) (api/token-node 'data) (api/token-node 'n-elems) (api/list-node (list obj-cast-fn (api/token-node 'v))))))) (api/list-node (list (api/token-node 'addAllReducible) (api/vector-node [(api/token-node 'this) (api/token-node 'coll)]) (api/list-node (list add-all-reduce (api/token-node 'this) (api/token-node 'coll)))))))] {:node new-node})) ================================================ FILE: results/.keepme ================================================ ================================================ FILE: results/concatv.edn ================================================ [{:clj {:mean-μs 0.24780844090292262, :variance-μs 3.764414164768287E-12}, :hamf {:mean-μs 0.43340971203396683, :variance-μs 9.387848181317607E-11}, :hamf-objarry {:mean-μs 0.2547612418625813, :variance-μs 3.302995262400414E-13}, :n-elems 4, :test :concatv, :numeric? true} {:clj {:mean-μs 0.8037494984097233, :variance-μs 2.709700728020432E-11}, :hamf {:mean-μs 0.9090962224550908, :variance-μs 5.7322329447778246E-11}, :hamf-objarry {:mean-μs 0.4451537483364495, :variance-μs 6.103257970014966E-12}, :n-elems 10, :test :concatv, :numeric? true} {:clj {:mean-μs 7.795815362657326, :variance-μs 6.281494043817672E-10}, :hamf {:mean-μs 3.394483066467648, :variance-μs 9.411949111627164E-11}, :hamf-objarry {:mean-μs 2.9224988649698935, :variance-μs 2.8498563089823355E-10}, :n-elems 100, :test :concatv, :numeric? true} {:clj {:mean-μs 81.0860135891287, :variance-μs 1.6596268671685533E-7}, :hamf {:mean-μs 37.53904281118535, :variance-μs 8.261923640048913E-8}, :hamf-objarry {:mean-μs 33.72946273712737, :variance-μs 9.40292705667369E-8}, :n-elems 1000, :test :concatv, :numeric? true} {:clj {:mean-μs 828.644979338843, :variance-μs 2.7044209184254263E-5}, :hamf {:mean-μs 339.2093104026846, :variance-μs 7.96549948444661E-6}, :hamf-objarry {:mean-μs 299.8918443443444, :variance-μs 1.0650090238108979E-6}, :n-elems 10000, :test :concatv, :numeric? true} {:clj {:mean-μs 8368.874025641026, :variance-μs 2.1625008937376858E-4}, :hamf {:mean-μs 3364.718911111111, :variance-μs 3.514570640074054E-5}, :hamf-objarry {:mean-μs 3320.8981935483876, :variance-μs 9.131112745088487E-5}, :n-elems 100000, :test :concatv, :numeric? true} {:clj {:mean-μs 84957.22483333333, :variance-μs 0.011313175909666845}, :hamf {:mean-μs 37388.642611111114, :variance-μs 0.0026630569333667073}, :hamf-objarry {:mean-μs 77076.09787499999, :variance-μs 223.1233904054363}, :n-elems 1000000, :test :concatv, :numeric? true}] ================================================ FILE: results/d00905c-chrisn-lt3-jdk-1.8.0_312.edn ================================================ {:machine-name "chrisn-lt3", :git-sha "d00905c", :jdk-version "1.8.0_312", :dataset [{:n-elems 5, :norm-factor-μs 0.2737073007766444, :hamf 0.8009695217828876, :clj 1.0, :test :assoc-in} {:n-elems 5, :norm-factor-μs 0.141872438402895, :hamf 0.2750157164152516, :clj 1.0, :test :assoc-in-nil} {:n-elems 100, :norm-factor-μs 6.810420269594472, :hamf 0.12045202287580566, :clj 1.0, :test :concatv} {:n-elems 10000, :norm-factor-μs 960.7099828660438, :hamf 0.4206960080303668, :clj 1.0, :test :frequencies} {:n-elems 5, :norm-factor-μs 0.12507571383882654, :hamf 0.5976826789421495, :clj 1.0, :test :get-in} {:n-elems 10000, :norm-factor-μs 1410.6903333333335, :hamf 0.33488278384032505, :clj 1.0, :test :group-by} {:n-elems 10000, :norm-factor-μs 1433.528372685185, :hamf 0.2931361354483846, :clj 1.0, :test :group-by-reduce} {:n-elems 10000, :java 0.8174789938125867, :norm-factor-μs 541.5395810810811, :hamf 1.0463362481300715, :clj 1.0, :test :hashmap-access} {:n-elems 10, :java 0.7914018884239628, :norm-factor-μs 0.4022970885136924, :hamf 0.903954075168239, :clj 1.0, :test :hashmap-access} {:n-elems 4, :norm-factor-μs 0.4067259476382922, :hamf 0.3981553073808861, :clj 1.0, :test :hashmap-cons-obj-ary} {:n-elems 10, :norm-factor-μs 0.6818907494639115, :hamf 0.6957442339286233, :clj 1.0, :test :hashmap-cons-obj-ary} {:n-elems 1000, :norm-factor-μs 130.4228752688172, :hamf 0.44926071566559916, :clj 1.0, :test :hashmap-cons-obj-ary} {:n-elems 10000, :java 0.5859413067880584, :norm-factor-μs 1281.3707827004218, :hamf 0.9270037369029654, :clj 1.0, :test :hashmap-construction} {:n-elems 10, :java 0.27785938738153243, :norm-factor-μs 2.237653573625657, :hamf 0.4006274017339368, :clj 1.0, :test :hashmap-construction} {:n-elems 10000, :java 0.6249505770784523, :norm-factor-μs 321.2954284963197, :hamf 0.7035802166713964, :clj 1.0, :test :hashmap-reduce} {:n-elems 10, :java 0.7135037003851468, :norm-factor-μs 0.3117122182140018, :hamf 0.8615676280736668, :clj 1.0, :test :hashmap-reduce} {:n-elems 20000, :java 1.0, :norm-factor-μs 497.4922640264026, :hamf 1.0172281545633246, :test :int-list} {:n-elems 1000, :norm-factor-μs 226.51757317073174, :hamf 0.3251372917848312, :clj 1.0, :test :mapmap} {:n-elems 20000, :norm-factor-μs 1374.724713963964, :hamf 0.3910982420215109, :clj 1.0, :test :object-array} {:n-elems 20000, :java 1.0, :norm-factor-μs 493.7951913875599, :hamf 0.981178195595189, :test :object-list} {:n-elems 20000, :norm-factor-μs 1342.562408888889, :eduction 0.370245588501266, :hamf 0.22739722394427567, :clj 1.0, :test :sequence-summation} {:n-elems 10000, :norm-factor-μs 286.4782659980898, :hamf 0.4295476338444791, :clj 1.0, :test :shuffle} {:n-elems 10000, :norm-factor-μs 2839.5518425925925, :hamf 0.28388161547650165, :clj 1.0, :test :sort} {:n-elems 10000, :norm-factor-μs 2485.0578412698414, :hamf 0.4238762907271986, :clj 1.0, :test :sort-doubles} {:n-elems 10000, :norm-factor-μs 2885.107268518519, :hamf 0.27927387790535163, :clj 1.0, :test :sort-ints} {:n-elems 10, :java 0.1465288671905732, :norm-factor-μs 1.9384014094753692, :hamf 0.09261922329193109, :clj 1.0, :test :union} {:n-elems 10000, :java 0.2776789952954786, :norm-factor-μs 1446.9220880952382, :hamf 0.19769322238863848, :clj 1.0, :test :union} {:n-elems 10, :java 0.14352703285934437, :norm-factor-μs 1.9376540425154556, :hamf 0.09116729205903847, :clj 1.0, :test :union-disj} {:n-elems 10000, :java 0.2852826007683308, :norm-factor-μs 1440.7768595238097, :hamf 0.19795254491423378, :clj 1.0, :test :union-disj} {:n-elems 10, :java 0.11367177685883537, :norm-factor-μs 25.724485906709905, :hamf 0.22980816916533875, :clj 1.0, :test :union-reduce} {:n-elems 10000, :java 0.08078332979789488, :norm-factor-μs 37008.01033333334, :hamf 0.15897321364597183, :clj 1.0, :test :union-reduce} {:n-elems 5, :norm-factor-μs 0.29246851947534236, :hamf 1.4007771267102096, :clj 1.0, :test :update-in} {:n-elems 5, :norm-factor-μs 0.1382874667400807, :hamf 0.2889718087035753, :clj 1.0, :test :update-in-nil} {:n-elems 1000, :norm-factor-μs 166.7942376344086, :hamf 0.08320271790164209, :clj 1.0, :test :update-values} {:n-elems 10, :java 1.5628249973091581, :norm-factor-μs 86.02279933938894, :hamf 1.1306972837725195, :clj 1.0, :test :vector-access} {:n-elems 10000, :java 0.9449218717052008, :norm-factor-μs 139.99571786042242, :hamf 1.0471455082754746, :clj 1.0, :test :vector-access} {:n-elems 10, :java 1.1497535407399773, :norm-factor-μs 0.07540932086272897, :hamf 0.36451148371515757, :clj 1.0, :test :vector-cons-obj-array} {:n-elems 10000, :java 0.06613494382317181, :norm-factor-μs 103.369319416499, :hamf 0.04042520509271698, :clj 1.0, :test :vector-cons-obj-array} {:n-elems 10, :java 0.507751745430425, :norm-factor-μs 0.07268104518100771, :hamf 1.1191943841960355, :clj 1.0, :test :vector-construction} {:n-elems 10000, :java 0.06228636275785186, :norm-factor-μs 109.04036126126128, :hamf 0.0700779992253449, :clj 1.0, :test :vector-construction} {:n-elems 10, :java 2.040987879747724, :norm-factor-μs 0.1518468919073785, :hamf 1.0314119616036046, :clj 1.0, :test :vector-reduce} {:n-elems 10000, :java 1.3918010263064262, :norm-factor-μs 146.63559051094893, :hamf 1.0263336595466455, :clj 1.0, :test :vector-reduce} {:n-elems 10, :java 0.284477780527412, :norm-factor-μs 0.03607406265851366, :hamf 0.5691549470684795, :clj 1.0, :test :vector-to-array} {:n-elems 10000, :java 0.05191814884413698, :norm-factor-μs 65.94629079634464, :hamf 0.06834408666272451, :clj 1.0, :test :vector-to-array}]} ================================================ FILE: results/d00905c-chrisn-lt3-jdk-17.0.1.edn ================================================ {:machine-name "chrisn-lt3", :git-sha "d00905c", :jdk-version "17.0.1", :dataset [{:n-elems 5, :norm-factor-μs 0.2454300764363296, :hamf 0.645561999028018, :clj 1.0, :test :assoc-in} {:n-elems 5, :norm-factor-μs 0.11989653723540485, :hamf 0.3710010211170179, :clj 1.0, :test :assoc-in-nil} {:n-elems 100, :norm-factor-μs 9.826760791689058, :hamf 0.0987601947642363, :clj 1.0, :test :concatv} {:n-elems 10000, :norm-factor-μs 966.1539672897197, :hamf 0.4117709827331045, :clj 1.0, :test :frequencies} {:n-elems 5, :norm-factor-μs 0.12388659038983477, :hamf 0.5635429524984066, :clj 1.0, :test :get-in} {:n-elems 10000, :norm-factor-μs 1414.4795533333333, :hamf 0.33263526123149856, :clj 1.0, :test :group-by} {:n-elems 10000, :norm-factor-μs 1408.0276894977171, :hamf 0.3133028742380794, :clj 1.0, :test :group-by-reduce} {:n-elems 10000, :java 0.700098540194207, :norm-factor-μs 549.4677786885246, :hamf 0.9885904403596032, :clj 1.0, :test :hashmap-access} {:n-elems 10, :java 0.8371851640462842, :norm-factor-μs 0.39970161594433823, :hamf 0.8257697710235989, :clj 1.0, :test :hashmap-access} {:n-elems 4, :norm-factor-μs 0.39154649377328343, :hamf 0.35532562182772665, :clj 1.0, :test :hashmap-cons-obj-ary} {:n-elems 10, :norm-factor-μs 0.863606395306321, :hamf 0.5840580056224318, :clj 1.0, :test :hashmap-cons-obj-ary} {:n-elems 1000, :norm-factor-μs 124.81141047503047, :hamf 0.5406815136043169, :clj 1.0, :test :hashmap-cons-obj-ary} {:n-elems 10000, :java 0.5630691816953767, :norm-factor-μs 1331.1297402597404, :hamf 0.9227125798469248, :clj 1.0, :test :hashmap-construction} {:n-elems 10, :java 0.24035557617165446, :norm-factor-μs 2.337265181606312, :hamf 0.3573060003056217, :clj 1.0, :test :hashmap-construction} {:n-elems 10000, :java 0.7919820197723015, :norm-factor-μs 316.4332277777778, :hamf 0.8600438031678125, :clj 1.0, :test :hashmap-reduce} {:n-elems 10, :java 0.7025734210457583, :norm-factor-μs 0.36049310535635243, :hamf 0.7353545091164955, :clj 1.0, :test :hashmap-reduce} {:n-elems 20000, :java 1.0, :norm-factor-μs 467.99419967793887, :hamf 1.1472402345498847, :test :int-list} {:n-elems 1000, :norm-factor-μs 275.7863779026218, :hamf 0.27598576781699885, :clj 1.0, :test :mapmap} {:n-elems 20000, :norm-factor-μs 1560.9749201877935, :hamf 0.24042373091680067, :clj 1.0, :test :object-array} {:n-elems 20000, :java 1.0, :norm-factor-μs 518.8781770833334, :hamf 0.9872361136470196, :test :object-list} {:n-elems 20000, :norm-factor-μs 1380.4963036529682, :eduction 0.29022715840409075, :hamf 0.40938478721051513, :clj 1.0, :test :sequence-summation} {:n-elems 10000, :norm-factor-μs 329.70861211211206, :hamf 0.35256060031168274, :clj 1.0, :test :shuffle} {:n-elems 10000, :norm-factor-μs 2418.306638888889, :hamf 0.3365951985310685, :clj 1.0, :test :sort} {:n-elems 10000, :norm-factor-μs 2272.1426259259265, :hamf 0.3744829531086446, :clj 1.0, :test :sort-doubles} {:n-elems 10000, :norm-factor-μs 2514.7790203252034, :hamf 0.29084489524894996, :clj 1.0, :test :sort-ints} {:n-elems 10, :java 0.1547125872471658, :norm-factor-μs 1.785352472507342, :hamf 0.08782960760507862, :clj 1.0, :test :union} {:n-elems 10000, :java 0.27530353772534377, :norm-factor-μs 1664.8231730769232, :hamf 0.17444121772551566, :clj 1.0, :test :union} {:n-elems 10, :java 0.1563929390039218, :norm-factor-μs 1.7984273753995366, :hamf 0.08496556764141062, :clj 1.0, :test :union-disj} {:n-elems 10000, :java 0.27923401245084806, :norm-factor-μs 1641.3437692307693, :hamf 0.17765049653015624, :clj 1.0, :test :union-disj} {:n-elems 10, :java 0.13905790023136533, :norm-factor-μs 23.95422292622533, :hamf 0.22028258205545878, :clj 1.0, :test :union-reduce} {:n-elems 10000, :java 0.09975903561943682, :norm-factor-μs 41261.662833333336, :hamf 0.15873443329519718, :clj 1.0, :test :union-reduce} {:n-elems 5, :norm-factor-μs 0.27596133989884, :hamf 1.153192661309212, :clj 1.0, :test :update-in} {:n-elems 5, :norm-factor-μs 0.15777840704317916, :hamf 0.27633859337368183, :clj 1.0, :test :update-in-nil} {:n-elems 1000, :norm-factor-μs 158.99377192982456, :hamf 0.09024145341863514, :clj 1.0, :test :update-values} {:n-elems 10, :java 1.5677072712878053, :norm-factor-μs 77.9446561934316, :hamf 1.0084726623021274, :clj 1.0, :test :vector-access} {:n-elems 10000, :java 0.9569459740380856, :norm-factor-μs 125.77780037546934, :hamf 1.026870193252271, :clj 1.0, :test :vector-access} {:n-elems 10, :java 1.1844251995426458, :norm-factor-μs 0.0711508365312022, :hamf 0.3555518514949478, :clj 1.0, :test :vector-cons-obj-array} {:n-elems 10000, :java 0.08315298066928496, :norm-factor-μs 112.19156870229008, :hamf 0.04848307511571774, :clj 1.0, :test :vector-cons-obj-array} {:n-elems 10, :java 0.4598551977815668, :norm-factor-μs 0.07780193981548053, :hamf 1.1235255104314212, :clj 1.0, :test :vector-construction} {:n-elems 10000, :java 0.08238350984081357, :norm-factor-μs 117.43194285714287, :hamf 0.07759581065745061, :clj 1.0, :test :vector-construction} {:n-elems 10, :java 1.9959541201348796, :norm-factor-μs 0.15045886766676977, :hamf 1.0879851619448582, :clj 1.0, :test :vector-reduce} {:n-elems 10000, :java 1.227779861025791, :norm-factor-μs 194.19370060606062, :hamf 0.8632717352503987, :clj 1.0, :test :vector-reduce} {:n-elems 10, :java 0.25632023563227685, :norm-factor-μs 0.040976120084999255, :hamf 0.5032633746096943, :clj 1.0, :test :vector-to-array} {:n-elems 10000, :java 0.0626329260162031, :norm-factor-μs 69.5895701754386, :hamf 0.12397017888135678, :clj 1.0, :test :vector-to-array}]} ================================================ FILE: results/general-hashmap.edn ================================================ [{:hamf-trie {:mean-μs 0.12737612204503088, :variance-μs 6.396591659858203E-11}, :java {:mean-μs 0.15931202460367855, :variance-μs 2.1921699693937754E-11}, :hamf-hashmap {:mean-μs 0.07082310724719383, :variance-μs 1.2468831270368505E-12}, :clj-int-map {:mean-μs 0.18253536926477196, :variance-μs 7.2435025867869124E-12}, :n-elems 4, :clj {:mean-μs 0.12660375158571427, :variance-μs 1.2129316208769813E-11}, :test :hashmap-construction, :hamf-long-map {:mean-μs 0.07933024944444202, :variance-μs 2.1311067935557158E-11}, :numeric? true} {:hamf-trie {:mean-μs 0.03881749141608742, :variance-μs 4.287722865119336E-15}, :java {:mean-μs 0.032855244414518744, :variance-μs 1.9405992798749638E-14}, :hamf-hashmap {:mean-μs 0.0385735657127411, :variance-μs 4.514740863848797E-15}, :clj-int-map {:mean-μs 0.037798297858484826, :variance-μs 4.1147004931921244E-14}, :n-elems 4, :clj {:mean-μs 0.05715286326051579, :variance-μs 8.809370228294369E-14}, :test :hashmap-access, :hamf-long-map {:mean-μs 0.03226057214308612, :variance-μs 3.566550010133413E-16}, :numeric? true} {:hamf-trie {:mean-μs 0.0024384717786416405, :variance-μs 5.803862412065768E-16}, :java {:mean-μs 0.0024378755410881777, :variance-μs 4.295888301027563E-16}, :hamf-hashmap {:mean-μs 0.0024450730662496133, :variance-μs 5.927874716218362E-16}, :clj-int-map {:mean-μs 0.002448551632772595, :variance-μs 1.8980594912551673E-15}, :n-elems 4, :clj {:mean-μs 0.002443443341831773, :variance-μs 5.170177435378848E-16}, :test :hashmap-reduction, :hamf-long-map {:mean-μs 0.002446382918059421, :variance-μs 1.3007366485730063E-16}, :numeric? true} {:hamf-trie 360, :java 320, :hamf-hashmap 360, :clj-int-map 632, :n-elems 4, :clj 176, :test :hashmap-bytes, :hamf-long-map 368, :numeric? true} {:hamf-trie {:mean-μs 0.13929403993989783, :variance-μs 5.450456700957539E-11}, :java {:mean-μs 0.17365940942022098, :variance-μs 1.064268216077601E-11}, :hamf-hashmap {:mean-μs 0.14058743431243464, :variance-μs 2.676404093792181E-12}, :n-elems 4, :clj {:mean-μs 0.11750402207956963, :variance-μs 1.579265281707213E-11}, :test :hashmap-construction, :numeric? false} {:hamf-trie {:mean-μs 0.06118565145565695, :variance-μs 1.396593610771331E-12}, :java {:mean-μs 0.042190656717632995, :variance-μs 2.427847078325523E-13}, :hamf-hashmap {:mean-μs 0.05205594394946113, :variance-μs 8.952374773511257E-15}, :n-elems 4, :clj {:mean-μs 0.05765036235868071, :variance-μs 1.6251377628408275E-13}, :test :hashmap-access, :numeric? false} {:hamf-trie {:mean-μs 0.00437183284375, :variance-μs 3.1404750788561035E-13}, :java {:mean-μs 0.004161787729166666, :variance-μs 1.8177328862022605E-13}, :hamf-hashmap {:mean-μs 0.0039667416278950105, :variance-μs 5.139588061720048E-15}, :n-elems 4, :clj {:mean-μs 0.003965322427083334, :variance-μs 2.6684162672671795E-15}, :test :hashmap-reduction, :numeric? false} {:hamf-trie 744, :java 704, :hamf-hashmap 744, :n-elems 4, :clj 560, :test :hashmap-bytes, :numeric? false} {:hamf-trie {:mean-μs 0.38881422524809084, :variance-μs 3.6784587621633865E-10}, :java {:mean-μs 0.36307100852637253, :variance-μs 7.394271419902345E-12}, :hamf-hashmap {:mean-μs 0.36328345928985556, :variance-μs 3.4678105142504224E-10}, :clj-int-map {:mean-μs 0.4495605552493311, :variance-μs 5.1607456622082824E-12}, :n-elems 10, :clj {:mean-μs 0.7369809755058884, :variance-μs 8.22757863590052E-11}, :test :hashmap-construction, :hamf-long-map {:mean-μs 0.2043492727775039, :variance-μs 3.475441861833042E-12}, :numeric? true} {:hamf-trie {:mean-μs 0.15733376615602068, :variance-μs 4.5775990120545246E-14}, :java {:mean-μs 0.08919611977324254, :variance-μs 8.578982120055384E-15}, :hamf-hashmap {:mean-μs 0.1495159480779122, :variance-μs 1.006378675606183E-13}, :clj-int-map {:mean-μs 0.1841453176041887, :variance-μs 7.026963173788516E-13}, :n-elems 10, :clj {:mean-μs 0.19522919080707185, :variance-μs 2.748861772918758E-13}, :test :hashmap-access, :hamf-long-map {:mean-μs 0.11958588080443483, :variance-μs 1.5187104739401225E-13}, :numeric? true} {:hamf-trie {:mean-μs 0.003928673499708508, :variance-μs 2.616775725543735E-16}, :java {:mean-μs 0.004396966968750001, :variance-μs 3.2373918098896885E-13}, :hamf-hashmap {:mean-μs 0.003949708881970929, :variance-μs 5.623070317493453E-15}, :clj-int-map {:mean-μs 0.004160355885416666, :variance-μs 1.816701020857573E-13}, :n-elems 10, :clj {:mean-μs 0.003922125917744089, :variance-μs 3.6482420051809284E-16}, :test :hashmap-reduction, :hamf-long-map {:mean-μs 0.003958190197175222, :variance-μs 6.327717781092918E-15}, :numeric? true} {:hamf-trie 808, :java 688, :hamf-hashmap 728, :clj-int-map 1328, :n-elems 10, :clj 520, :test :hashmap-bytes, :hamf-long-map 784, :numeric? true} {:hamf-trie {:mean-μs 0.391980719710974, :variance-μs 3.177861567297025E-11}, :java {:mean-μs 0.38384906786499706, :variance-μs 3.0062228263010584E-11}, :hamf-hashmap {:mean-μs 0.3834914961636829, :variance-μs 7.277915431714698E-11}, :n-elems 10, :clj {:mean-μs 0.5766033379083687, :variance-μs 1.3677833136358415E-10}, :test :hashmap-construction, :numeric? false} {:hamf-trie {:mean-μs 0.17896307771287526, :variance-μs 3.6676289532947006E-14}, :java {:mean-μs 0.09507845899989242, :variance-μs 3.1564077823872248E-15}, :hamf-hashmap {:mean-μs 0.18408497814128674, :variance-μs 6.220949136009276E-13}, :n-elems 10, :clj {:mean-μs 0.17937706624420582, :variance-μs 1.1963169589385688E-12}, :test :hashmap-access, :numeric? false} {:hamf-trie {:mean-μs 0.004129701467313691, :variance-μs 1.6406977366772879E-13}, :java {:mean-μs 0.00432857203125, :variance-μs 2.665194658832183E-13}, :hamf-hashmap {:mean-μs 0.004333073760416668, :variance-μs 2.564274172463116E-13}, :n-elems 10, :clj {:mean-μs 0.0039509407069990825, :variance-μs 3.876332490476048E-15}, :test :hashmap-reduction, :numeric? false} {:hamf-trie 1800, :java 1648, :hamf-hashmap 1688, :n-elems 10, :clj 1520, :test :hashmap-bytes, :numeric? false} {:hamf-trie {:mean-μs 5.042265099729212, :variance-μs 1.3019750130568536E-8}, :java {:mean-μs 3.4682698136391563, :variance-μs 1.2106259218308585E-8}, :hamf-hashmap {:mean-μs 3.5239084320573406, :variance-μs 2.0592874264445192E-8}, :clj-int-map {:mean-μs 5.915345205184009, :variance-μs 4.7514044719016145E-8}, :n-elems 100, :clj {:mean-μs 6.099257853377101, :variance-μs 1.1957982711306617E-8}, :test :hashmap-construction, :hamf-long-map {:mean-μs 3.254834293394778, :variance-μs 3.344082683701172E-8}, :numeric? true} {:hamf-trie {:mean-μs 2.355782991225383, :variance-μs 4.963674628469898E-11}, :java {:mean-μs 1.0801430070155806, :variance-μs 1.239219438258746E-10}, :hamf-hashmap {:mean-μs 1.583093910520642, :variance-μs 6.989749612367083E-12}, :clj-int-map {:mean-μs 1.9407344024362587, :variance-μs 5.510222310760512E-11}, :n-elems 100, :clj {:mean-μs 2.1451822321434926, :variance-μs 8.73402096040883E-11}, :test :hashmap-access, :hamf-long-map {:mean-μs 1.3615756260343472, :variance-μs 1.4378168868528625E-12}, :numeric? true} {:hamf-trie {:mean-μs 0.004355775583333334, :variance-μs 2.5841190649951006E-13}, :java {:mean-μs 0.004387476104166667, :variance-μs 3.0479993250641736E-13}, :hamf-hashmap {:mean-μs 0.00435306115625, :variance-μs 2.478671626520929E-13}, :clj-int-map {:mean-μs 0.004169804656250001, :variance-μs 1.6719513991090373E-13}, :n-elems 100, :clj {:mean-μs 0.003979964347546555, :variance-μs 7.126219212697998E-15}, :test :hashmap-reduction, :hamf-long-map {:mean-μs 0.0044084813125, :variance-μs 2.386154550975209E-13}, :numeric? true} {:hamf-trie 7864, :java 6688, :hamf-hashmap 6728, :clj-int-map 9320, :n-elems 100, :clj 5336, :test :hashmap-bytes, :hamf-long-map 7504, :numeric? true} {:hamf-trie {:mean-μs 4.81606174028822, :variance-μs 1.5665166480985434E-8}, :java {:mean-μs 3.461359238004145, :variance-μs 9.702443215715173E-9}, :hamf-hashmap {:mean-μs 3.614230712634013, :variance-μs 1.7439344392895864E-8}, :n-elems 100, :clj {:mean-μs 5.523413217039427, :variance-μs 2.78643894449881E-9}, :test :hashmap-construction, :numeric? false} {:hamf-trie {:mean-μs 1.8320056480580593, :variance-μs 5.881840598643257E-12}, :java {:mean-μs 1.0945387262778477, :variance-μs 7.725902442750274E-11}, :hamf-hashmap {:mean-μs 1.4730817014619413, :variance-μs 1.5386189994280872E-11}, :n-elems 100, :clj {:mean-μs 1.5674547419073985, :variance-μs 7.080508559610851E-11}, :test :hashmap-access, :numeric? false} {:hamf-trie {:mean-μs 0.003951741768835076, :variance-μs 4.2801113306202835E-15}, :java {:mean-μs 0.003961767246033917, :variance-μs 4.840416100237032E-15}, :hamf-hashmap {:mean-μs 0.003956974679443984, :variance-μs 5.597973639323671E-15}, :n-elems 100, :clj {:mean-μs 0.004327536010416667, :variance-μs 2.8151414833665384E-13}, :test :hashmap-reduction, :numeric? false} {:hamf-trie 17496, :java 16288, :hamf-hashmap 16328, :n-elems 100, :clj 14720, :test :hashmap-bytes, :numeric? false} {:hamf-trie {:mean-μs 65.21449017094018, :variance-μs 1.557468688019957E-6}, :java {:mean-μs 31.095526593585063, :variance-μs 7.841178970421217E-7}, :hamf-hashmap {:mean-μs 32.726058785529716, :variance-μs 6.724550020819318E-7}, :clj-int-map {:mean-μs 84.05662559374127, :variance-μs 1.2379705027684346E-7}, :n-elems 1000, :clj {:mean-μs 81.44535906763677, :variance-μs 6.374716214979322E-7}, :test :hashmap-construction, :hamf-long-map {:mean-μs 30.260220216962523, :variance-μs 1.256291216552317E-6}, :numeric? true} {:hamf-trie {:mean-μs 33.017372240437155, :variance-μs 8.592611669121406E-10}, :java {:mean-μs 11.837676194405155, :variance-μs 2.6374858885184383E-9}, :hamf-hashmap {:mean-μs 14.619642228739002, :variance-μs 8.053497666657927E-10}, :clj-int-map {:mean-μs 21.25854047472045, :variance-μs 3.1670463383734034E-7}, :n-elems 1000, :clj {:mean-μs 23.59308321507411, :variance-μs 1.2759146241946811E-8}, :test :hashmap-access, :hamf-long-map {:mean-μs 13.930387806688003, :variance-μs 1.391420621113504E-8}, :numeric? true} {:hamf-trie {:mean-μs 0.0041704717604166665, :variance-μs 2.2728670616887456E-13}, :java {:mean-μs 0.0040447757319230625, :variance-μs 2.564157454177168E-15}, :hamf-hashmap {:mean-μs 0.004120246500000001, :variance-μs 1.6619460616093238E-13}, :clj-int-map {:mean-μs 0.0039059910822086116, :variance-μs 7.498891755475981E-17}, :n-elems 1000, :clj {:mean-μs 0.003944085134378271, :variance-μs 9.088481049851367E-15}, :test :hashmap-reduction, :hamf-long-map {:mean-μs 0.003939165355000591, :variance-μs 4.414282841383754E-15}, :numeric? true} {:hamf-trie 79592, :java 64064, :hamf-hashmap 64104, :clj-int-map 97928, :n-elems 1000, :clj 72952, :test :hashmap-bytes, :hamf-long-map 71888, :numeric? true} {:hamf-trie {:mean-μs 63.17735462510189, :variance-μs 2.4447571117536376E-6}, :java {:mean-μs 33.135304131054134, :variance-μs 7.444925057237155E-7}, :hamf-hashmap {:mean-μs 35.32318542519327, :variance-μs 1.9787499436824814E-6}, :n-elems 1000, :clj {:mean-μs 73.98949730392157, :variance-μs 1.80198847544197E-6}, :test :hashmap-construction, :numeric? false} {:hamf-trie {:mean-μs 30.22536841308298, :variance-μs 6.50870367110682E-8}, :java {:mean-μs 14.414018979748537, :variance-μs 2.985518123790603E-10}, :hamf-hashmap {:mean-μs 16.950012444971218, :variance-μs 1.200396719638997E-9}, :n-elems 1000, :clj {:mean-μs 19.26251062532101, :variance-μs 8.503871035654129E-10}, :test :hashmap-access, :numeric? false} {:hamf-trie {:mean-μs 0.004413220885416667, :variance-μs 3.423062424479693E-13}, :java {:mean-μs 0.00422049603125, :variance-μs 2.9761742065081937E-13}, :hamf-hashmap {:mean-μs 0.004361120625, :variance-μs 2.406070972460215E-13}, :n-elems 1000, :clj {:mean-μs 0.0043975624687500004, :variance-μs 2.551046772970641E-13}, :test :hashmap-reduction, :numeric? false} {:hamf-trie 174824, :java 159344, :hamf-hashmap 159384, :n-elems 1000, :clj 166520, :test :hashmap-bytes, :numeric? false} {:hamf-trie {:mean-μs 780.8485737913487, :variance-μs 3.477543096084737E-5}, :java {:mean-μs 375.0044652014653, :variance-μs 7.39986466736043E-5}, :hamf-hashmap {:mean-μs 398.6921731770833, :variance-μs 8.205242788379923E-5}, :clj-int-map {:mean-μs 1154.6796074074075, :variance-μs 0.0033827290231506968}, :n-elems 10000, :clj {:mean-μs 719.7024083333334, :variance-μs 1.9683084659932206E-6}, :test :hashmap-construction, :hamf-long-map {:mean-μs 375.4870214460784, :variance-μs 3.450134214371576E-5}, :numeric? true} {:hamf-trie {:mean-μs 350.05588395904437, :variance-μs 2.774934875777946E-7}, :java {:mean-μs 147.08369096209915, :variance-μs 9.402568656767165E-9}, :hamf-hashmap {:mean-μs 193.4745440797941, :variance-μs 1.602035297694827E-7}, :clj-int-map {:mean-μs 290.3997015281758, :variance-μs 2.4641816378847545E-6}, :n-elems 10000, :clj {:mean-μs 317.73844761904763, :variance-μs 1.875548994856781E-7}, :test :hashmap-access, :hamf-long-map {:mean-μs 159.55805785562634, :variance-μs 1.9641160542432423E-8}, :numeric? true} {:hamf-trie {:mean-μs 0.004174307697916667, :variance-μs 1.8470619277218825E-13}, :java {:mean-μs 0.003958018485899186, :variance-μs 5.7057823918703085E-15}, :hamf-hashmap {:mean-μs 0.003925685516701463, :variance-μs 6.551914226599447E-17}, :clj-int-map {:mean-μs 0.003964951132591866, :variance-μs 5.682124489389675E-15}, :n-elems 10000, :clj {:mean-μs 0.003933888226433571, :variance-μs 6.4640969317443285E-15}, :test :hashmap-reduction, :hamf-long-map {:mean-μs 0.004351605875, :variance-μs 2.897003492924412E-13}, :numeric? true} {:hamf-trie 700664, :java 609184, :hamf-hashmap 609224, :clj-int-map 906416, :n-elems 10000, :clj 450320, :test :hashmap-bytes, :hamf-long-map 673552, :numeric? true} {:hamf-trie {:mean-μs 778.6050628205129, :variance-μs 1.2085644687352968E-5}, :java {:mean-μs 429.3114950564971, :variance-μs 6.238034878090834E-5}, :hamf-hashmap {:mean-μs 424.7148458856347, :variance-μs 5.306066205877626E-5}, :n-elems 10000, :clj {:mean-μs 681.7886047297299, :variance-μs 6.529311888673321E-6}, :test :hashmap-construction, :numeric? false} {:hamf-trie {:mean-μs 348.9691128472223, :variance-μs 1.271805725039338E-7}, :java {:mean-μs 174.46373507246378, :variance-μs 2.238383134084509E-8}, :hamf-hashmap {:mean-μs 193.144, :variance-μs 7.086958663804234E-8}, :n-elems 10000, :clj {:mean-μs 248.72084791666668, :variance-μs 1.1873965896110325E-6}, :test :hashmap-access, :numeric? false} {:hamf-trie {:mean-μs 0.003924769680303962, :variance-μs 6.171628630039009E-16}, :java {:mean-μs 0.00395765214327313, :variance-μs 6.60370363918879E-15}, :hamf-hashmap {:mean-μs 0.003940098602889401, :variance-μs 5.231695887552283E-15}, :n-elems 10000, :clj {:mean-μs 0.003953429604374496, :variance-μs 6.284615802717058E-15}, :test :hashmap-reduction, :numeric? false} {:hamf-trie 1626208, :java 1514008, :hamf-hashmap 1514048, :n-elems 10000, :clj 1357408, :test :hashmap-bytes, :numeric? false} {:hamf-trie {:mean-μs 105631.582, :variance-μs 0.41091841103706755}, :java {:mean-μs 34421.373666666674, :variance-μs 0.006783515112774007}, :hamf-hashmap {:mean-μs 41940.65366666667, :variance-μs 1.6065800330680753}, :clj-int-map {:mean-μs 228662.874, :variance-μs 143.98871654502028}, :n-elems 1000000, :clj {:mean-μs 87141.36675, :variance-μs 0.152704115646775}, :test :hashmap-construction, :hamf-long-map {:mean-μs 30214.13570833333, :variance-μs 0.08636392782682523}, :numeric? true} {:hamf-trie {:mean-μs 55755.943, :variance-μs 0.1451732910962674}, :java {:mean-μs 20863.527666666665, :variance-μs 0.0191632000882306}, :hamf-hashmap {:mean-μs 25654.37004166667, :variance-μs 0.021320091008268795}, :clj-int-map {:mean-μs 29977.67216666667, :variance-μs 0.110571280534535}, :n-elems 1000000, :clj {:mean-μs 49993.6235, :variance-μs 0.11235685551394138}, :test :hashmap-access, :hamf-long-map {:mean-μs 13560.8405, :variance-μs 0.00883170116229789}, :numeric? true} {:hamf-trie {:mean-μs 0.004350175133233413, :variance-μs 7.167683862078651E-13}, :java {:mean-μs 0.004548510938421815, :variance-μs 5.243753524294015E-13}, :hamf-hashmap {:mean-μs 0.004439494431695403, :variance-μs 5.254081712185421E-13}, :clj-int-map {:mean-μs 0.0045442266764172835, :variance-μs 6.287656824822641E-13}, :n-elems 1000000, :clj {:mean-μs 0.0045736030637335045, :variance-μs 6.517934758604611E-13}, :test :hashmap-reduction, :hamf-long-map {:mean-μs 0.004243031342943516, :variance-μs 3.7924330277669446E-13}, :numeric? true} {:hamf-trie 10278800, :java 9044184, :hamf-hashmap 9044224, :clj-int-map 5706776, :n-elems 1000000, :clj 7729992, :test :hashmap-bytes, :hamf-long-map 7448336, :numeric? true} {:hamf-trie {:mean-μs 111861.846, :variance-μs 0.20176143255576712}, :java {:mean-μs 47607.29483333334, :variance-μs 0.03152497361291819}, :hamf-hashmap {:mean-μs 48172.75788888889, :variance-μs 0.8034428813259324}, :n-elems 1000000, :clj {:mean-μs 77178.77258333334, :variance-μs 0.3013717195992401}, :test :hashmap-construction, :numeric? false} {:hamf-trie {:mean-μs 57660.383916666666, :variance-μs 0.8521034262750425}, :java {:mean-μs 24973.350916666666, :variance-μs 0.027276928168641783}, :hamf-hashmap {:mean-μs 26249.08016666667, :variance-μs 0.027534862330441803}, :n-elems 1000000, :clj {:mean-μs 37368.78800000001, :variance-μs 0.5087530702252877}, :test :hashmap-access, :numeric? false} {:hamf-trie {:mean-μs 0.004623724218752654, :variance-μs 6.257277938746783E-13}, :java {:mean-μs 0.0044721586994317195, :variance-μs 6.955546061472555E-13}, :hamf-hashmap {:mean-μs 0.004641498543766695, :variance-μs 7.014826910761973E-13}, :n-elems 1000000, :clj {:mean-μs 0.004608290065270957, :variance-μs 6.565473543067006E-13}, :test :hashmap-reduction, :numeric? false} {:hamf-trie 17460312, :java 16248032, :hamf-hashmap 16248072, :n-elems 1000000, :clj 14934288, :test :hashmap-bytes, :numeric? false}] ================================================ FILE: results/persistent-vector.edn ================================================ [{:java {:mean-μs 0.08404675313017497, :variance-μs 5.070139825243536E-14}, :n-elems 4, :clj {:mean-μs 0.04981935796635717, :variance-μs 1.527546946548401E-14}, :hamf {:mean-μs 0.10138344871884107, :variance-μs 7.973542842854269E-12}, :hamf-objary {:mean-μs 0.06318031677331305, :variance-μs 2.598610965473221E-12}, :test :vector-construction, :numeric? true} {:java {:mean-μs 0.014702250662715712, :variance-μs 4.403565754201952E-15}, :n-elems 4, :clj {:mean-μs 0.004808472835989959, :variance-μs 3.737505647466913E-16}, :hamf {:mean-μs 0.008112511943890463, :variance-μs 1.8442921728131405E-15}, :hamf-objary {:mean-μs 0.009619735097717748, :variance-μs 3.1724185102195438E-15}, :test :vector-access, :numeric? true} {:java {:mean-μs 0.005695914429490561, :variance-μs 2.0902311808272606E-14}, :n-elems 4, :clj {:mean-μs 0.00776486082290484, :variance-μs 4.729924999826533E-15}, :hamf {:mean-μs 0.01769129811981235, :variance-μs 2.0548635015740677E-12}, :hamf-objary {:mean-μs 0.010831710108628032, :variance-μs 7.570628868297045E-13}, :test :to-object-array, :numeric? true} {:java {:mean-μs 0.028725450802097346, :variance-μs 4.037289482015303E-14}, :n-elems 4, :clj {:mean-μs 0.03379246923815769, :variance-μs 7.104242855584067E-14}, :hamf {:mean-μs 0.032029540930047055, :variance-μs 4.248883565775089E-14}, :hamf-objary {:mean-μs 0.020358696845172882, :variance-μs 5.860664163959884E-14}, :test :from-double-array, :numeric? true} {:java {:mean-μs 0.1487238615157873, :variance-μs 1.3721503855042793E-12}, :n-elems 10, :clj {:mean-μs 0.08704663779104119, :variance-μs 4.175072297684551E-13}, :hamf {:mean-μs 0.14848439116805284, :variance-μs 3.357253148697639E-12}, :hamf-objary {:mean-μs 0.10886955594743827, :variance-μs 4.2233835999981864E-13}, :test :vector-construction, :numeric? true} {:java {:mean-μs 0.03597425499167559, :variance-μs 2.6517529741054524E-14}, :n-elems 10, :clj {:mean-μs 0.11066612996848661, :variance-μs 7.029107461479434E-13}, :hamf {:mean-μs 0.11329646938881383, :variance-μs 6.947066673472629E-13}, :hamf-objary {:mean-μs 0.10995136747021847, :variance-μs 5.183236466763794E-13}, :test :vector-access, :numeric? true} {:java {:mean-μs 0.012149834599900317, :variance-μs 2.589810189123974E-13}, :n-elems 10, :clj {:mean-μs 0.024358550404403795, :variance-μs 2.4834812156200304E-14}, :hamf {:mean-μs 0.017184634020573464, :variance-μs 7.0867003136786735E-15}, :hamf-objary {:mean-μs 0.01577686322847378, :variance-μs 2.0433189381845226E-14}, :test :to-object-array, :numeric? true} {:java {:mean-μs 0.04061378096300939, :variance-μs 4.9328568972788055E-14}, :n-elems 10, :clj {:mean-μs 0.05603033873642637, :variance-μs 1.065685429898049E-13}, :hamf {:mean-μs 0.04428793121614028, :variance-μs 1.0837386012847952E-13}, :hamf-objary {:mean-μs 0.03017124637986729, :variance-μs 1.9094853556729642E-14}, :test :from-double-array, :numeric? true} {:java {:mean-μs 0.7238227166843572, :variance-μs 8.746766611199866E-11}, :n-elems 100, :clj {:mean-μs 0.669717836801407, :variance-μs 1.2046035303641108E-10}, :hamf {:mean-μs 0.7232141750175776, :variance-μs 1.9085735524452754E-11}, :hamf-objary {:mean-μs 0.7000264403840255, :variance-μs 3.706183440159025E-11}, :test :vector-construction, :numeric? true} {:java {:mean-μs 0.3369278672420447, :variance-μs 1.751101196182606E-12}, :n-elems 100, :clj {:mean-μs 1.1870847861316134, :variance-μs 1.1862836974336482E-10}, :hamf {:mean-μs 1.060780160333358, :variance-μs 2.0927760194106986E-11}, :hamf-objary {:mean-μs 1.0629402511446575, :variance-μs 3.9832535419879046E-11}, :test :vector-access, :numeric? true} {:java {:mean-μs 0.015744159900351568, :variance-μs 9.977632866593663E-15}, :n-elems 100, :clj {:mean-μs 0.301215325717709, :variance-μs 2.1863556829924976E-13}, :hamf {:mean-μs 0.028260030657313435, :variance-μs 5.3671690610719224E-14}, :hamf-objary {:mean-μs 0.029411523935848838, :variance-μs 1.0968228427509516E-13}, :test :to-object-array, :numeric? true} {:java {:mean-μs 0.19336313799634125, :variance-μs 4.616392765668418E-13}, :n-elems 100, :clj {:mean-μs 0.88769464015771, :variance-μs 5.515768626056941E-11}, :hamf {:mean-μs 0.2263363410352146, :variance-μs 4.286090259626298E-12}, :hamf-objary {:mean-μs 0.20145931600120642, :variance-μs 1.3368500155474601E-12}, :test :from-double-array, :numeric? true} {:java {:mean-μs 7.419557924794185, :variance-μs 1.2063035500434223E-8}, :n-elems 1000, :clj {:mean-μs 6.551940577285984, :variance-μs 1.670104537251088E-9}, :hamf {:mean-μs 8.659981221466104, :variance-μs 1.2136700217014222E-8}, :hamf-objary {:mean-μs 7.445694670246351, :variance-μs 4.794667855844152E-9}, :test :vector-construction, :numeric? true} {:java {:mean-μs 3.2820573543971894, :variance-μs 4.8871499264651275E-11}, :n-elems 1000, :clj {:mean-μs 12.008160149419806, :variance-μs 9.799471756836892E-9}, :hamf {:mean-μs 10.798925809988683, :variance-μs 2.93748693952479E-9}, :hamf-objary {:mean-μs 10.704891635720601, :variance-μs 8.803949393371953E-9}, :test :vector-access, :numeric? true} {:java {:mean-μs 0.09855071867068062, :variance-μs 1.5382695957145455E-12}, :n-elems 1000, :clj {:mean-μs 2.829761194657988, :variance-μs 4.1680271617923245E-10}, :hamf {:mean-μs 0.23507436086672542, :variance-μs 3.156679760555342E-13}, :hamf-objary {:mean-μs 0.23330618642337395, :variance-μs 6.041367107732296E-12}, :test :to-object-array, :numeric? true} {:java {:mean-μs 1.9964637136988383, :variance-μs 8.286981727511509E-11}, :n-elems 1000, :clj {:mean-μs 8.903504789442467, :variance-μs 3.6010833975361855E-9}, :hamf {:mean-μs 2.2681713998193977, :variance-μs 2.345768658599516E-10}, :hamf-objary {:mean-μs 2.007464888095474, :variance-μs 1.3151750052769603E-10}, :test :from-double-array, :numeric? true} {:java {:mean-μs 41.81280116959065, :variance-μs 4.669333383154487E-8}, :n-elems 10000, :clj {:mean-μs 58.775339348079164, :variance-μs 8.901106506383374E-8}, :hamf {:mean-μs 34.812974452133794, :variance-μs 1.2146728637284742E-8}, :hamf-objary {:mean-μs 41.273131874071076, :variance-μs 7.998453376619444E-8}, :test :vector-construction, :numeric? true} {:java {:mean-μs 32.731771024464834, :variance-μs 2.170938246869297E-9}, :n-elems 10000, :clj {:mean-μs 124.72057712657714, :variance-μs 6.02334215414936E-6}, :hamf {:mean-μs 108.5272948164147, :variance-μs 7.06933875418284E-7}, :hamf-objary {:mean-μs 109.04531914893619, :variance-μs 1.2206286177146486E-6}, :test :vector-access, :numeric? true} {:java {:mean-μs 1.0006702248436885, :variance-μs 2.25595187105329E-11}, :n-elems 10000, :clj {:mean-μs 32.716528118939884, :variance-μs 1.7027873521302094E-8}, :hamf {:mean-μs 2.0961830192765865, :variance-μs 2.683216214040755E-10}, :hamf-objary {:mean-μs 2.148982849912814, :variance-μs 2.572704097985616E-10}, :test :to-object-array, :numeric? true} {:java {:mean-μs 17.91259518328316, :variance-μs 6.391510938463997E-9}, :n-elems 10000, :clj {:mean-μs 88.85198008849558, :variance-μs 6.306972398300558E-8}, :hamf {:mean-μs 20.598280486139558, :variance-μs 1.1740308329333441E-8}, :hamf-objary {:mean-μs 18.39746522515863, :variance-μs 4.176210746495778E-9}, :test :from-double-array, :numeric? true} {:java {:mean-μs 422.5867705718271, :variance-μs 1.072367608569531E-6}, :n-elems 100000, :clj {:mean-μs 637.8606199575372, :variance-μs 2.130866992003682E-6}, :hamf {:mean-μs 491.11614297385626, :variance-μs 3.786868092401166E-6}, :hamf-objary {:mean-μs 416.1541419753087, :variance-μs 9.662979704333655E-6}, :test :vector-construction, :numeric? true} {:java {:mean-μs 328.0229678649238, :variance-μs 3.3761939922963616E-7}, :n-elems 100000, :clj {:mean-μs 1326.2433640350878, :variance-μs 1.369539381780244E-4}, :hamf {:mean-μs 1104.4642857142858, :variance-μs 2.857838456217471E-5}, :hamf-objary {:mean-μs 1102.6335494505495, :variance-μs 1.366624956250049E-4}, :test :vector-access, :numeric? true} {:java {:mean-μs 7.509156500308072, :variance-μs 8.881072727459943E-8}, :n-elems 100000, :clj {:mean-μs 383.44464385297846, :variance-μs 2.1643609267547677E-6}, :hamf {:mean-μs 20.868715710637126, :variance-μs 1.049428804842173E-8}, :hamf-objary {:mean-μs 21.167191438141096, :variance-μs 3.245508258023865E-9}, :test :to-object-array, :numeric? true} {:java {:mean-μs 168.67022895622898, :variance-μs 3.924779705292658E-8}, :n-elems 100000, :clj {:mean-μs 895.1254880952381, :variance-μs 1.294896278180783E-6}, :hamf {:mean-μs 193.2052230769231, :variance-μs 4.994959728464088E-8}, :hamf-objary {:mean-μs 173.36318457538997, :variance-μs 9.811368580988553E-9}, :test :from-double-array, :numeric? true}] ================================================ FILE: results/random-update.edn ================================================ [{:hamf-trie {:mean-μs 7.473810371306137, :variance-μs 6.636768315602025E-9}, :java {:mean-μs 5.722913191296465, :variance-μs 2.7712356095487855E-9}, :hamf-hashmap {:mean-μs 6.7346819509476035, :variance-μs 2.6547603007155143E-9}, :clj-int-map {:mean-μs 28.61158458542015, :variance-μs 3.21139535674694E-7}, :n-elems 4, :clj {:mean-μs 29.315022571484526, :variance-μs 5.278260531630705E-8}, :test :random-update, :hamf-long-map {:mean-μs 4.727335746415519, :variance-μs 7.605068686913684E-10}, :numeric? true} {:hamf-trie {:mean-μs 8.249470905172416, :variance-μs 6.806187947226365E-9}, :java {:mean-μs 8.727563394056169, :variance-μs 1.4505477240971553E-9}, :hamf-hashmap {:mean-μs 11.482808723169011, :variance-μs 3.19167505795684E-9}, :n-elems 4, :clj {:mean-μs 19.57993421306107, :variance-μs 9.927578023551844E-8}, :test :random-update, :numeric? false} {:hamf-trie {:mean-μs 14.490477033168942, :variance-μs 1.6814016212846286E-8}, :java {:mean-μs 6.474124965105541, :variance-μs 2.0802148324619174E-9}, :hamf-hashmap {:mean-μs 10.617591625338537, :variance-μs 1.1670278774147662E-10}, :clj-int-map {:mean-μs 40.61000130157491, :variance-μs 3.718157660320322E-6}, :n-elems 10, :clj {:mean-μs 35.440350803142216, :variance-μs 7.336263662612312E-8}, :test :random-update, :hamf-long-map {:mean-μs 7.769649753048311, :variance-μs 1.4072701621324273E-9}, :numeric? true} {:hamf-trie {:mean-μs 16.489329767977623, :variance-μs 2.2739911162825837E-8}, :java {:mean-μs 11.329024095702461, :variance-μs 7.207390845879511E-9}, :hamf-hashmap {:mean-μs 17.204908985062758, :variance-μs 2.5615310976489953E-8}, :n-elems 10, :clj {:mean-μs 43.703390000000006, :variance-μs 4.846730085373682E-8}, :test :random-update, :numeric? false} {:hamf-trie {:mean-μs 17.366044789498034, :variance-μs 7.307864496441526E-9}, :java {:mean-μs 6.700638444548558, :variance-μs 2.891177537915405E-9}, :hamf-hashmap {:mean-μs 11.688231641721234, :variance-μs 8.932985936148656E-9}, :clj-int-map {:mean-μs 60.34284659433394, :variance-μs 4.6594863030114697E-7}, :n-elems 100, :clj {:mean-μs 47.221551500789886, :variance-μs 3.719528358275888E-8}, :test :random-update, :hamf-long-map {:mean-μs 10.178484621529858, :variance-μs 1.042940939679223E-8}, :numeric? true} {:hamf-trie {:mean-μs 23.103045288197624, :variance-μs 2.5081767176541172E-9}, :java {:mean-μs 11.483072079253018, :variance-μs 6.479275579587933E-9}, :hamf-hashmap {:mean-μs 14.500872780203787, :variance-μs 2.423372330838471E-8}, :n-elems 100, :clj {:mean-μs 41.491602569444446, :variance-μs 6.222481117254631E-8}, :test :random-update, :numeric? false} {:hamf-trie {:mean-μs 26.771196961524634, :variance-μs 9.903000450981743E-9}, :java {:mean-μs 9.280695548028806, :variance-μs 4.165138181087952E-10}, :hamf-hashmap {:mean-μs 14.475852591844586, :variance-μs 3.4677079839386018E-9}, :clj-int-map {:mean-μs 59.20099134982901, :variance-μs 8.297331301308577E-7}, :n-elems 1000, :clj {:mean-μs 77.1033328290469, :variance-μs 9.214075297517732E-9}, :test :random-update, :hamf-long-map {:mean-μs 8.895876164699619, :variance-μs 9.478380693292846E-11}, :numeric? true} {:hamf-trie {:mean-μs 37.93254213413115, :variance-μs 1.5721684133185825E-8}, :java {:mean-μs 14.728724891362727, :variance-μs 4.2628079865109866E-8}, :hamf-hashmap {:mean-μs 16.268459005810197, :variance-μs 2.1882211372855838E-8}, :n-elems 1000, :clj {:mean-μs 68.5770062742414, :variance-μs 1.2239030273183371E-6}, :test :random-update, :numeric? false} {:hamf-trie {:mean-μs 270.5718541666667, :variance-μs 5.09299341655838E-6}, :java {:mean-μs 99.79422464486292, :variance-μs 2.671491481155907E-7}, :hamf-hashmap {:mean-μs 146.36228659990462, :variance-μs 4.3048047936489657E-7}, :clj-int-map {:mean-μs 784.6103664021165, :variance-μs 5.9642458623448635E-5}, :n-elems 10000, :clj {:mean-μs 663.2450320088301, :variance-μs 2.758626783028814E-5}, :test :random-update, :hamf-long-map {:mean-μs 88.61613315618541, :variance-μs 2.5562978815447094E-8}, :numeric? true} {:hamf-trie {:mean-μs 386.008633974359, :variance-μs 1.6235898638785035E-5}, :java {:mean-μs 150.50318778280544, :variance-μs 8.595231344923378E-8}, :hamf-hashmap {:mean-μs 157.8658671421025, :variance-μs 7.728062207237752E-7}, :n-elems 10000, :clj {:mean-μs 521.2739136752136, :variance-μs 1.0895932142197636E-5}, :test :random-update, :numeric? false} {:hamf-trie {:mean-μs 27840.1305, :variance-μs 0.004096132211343695}, :java {:mean-μs 10149.956166666669, :variance-μs 0.0010089984839960011}, :hamf-hashmap {:mean-μs 14959.376523809524, :variance-μs 0.01870731796338507}, :clj-int-map {:mean-μs 70253.72091666667, :variance-μs 2.3498522960982005}, :n-elems 1000000, :clj {:mean-μs 73434.18241666668, :variance-μs 0.23153408453337476}, :test :random-update, :hamf-long-map {:mean-μs 8934.935416666667, :variance-μs 4.084387226111203E-5}, :numeric? true} {:hamf-trie {:mean-μs 38235.67222222222, :variance-μs 0.19408606879502188}, :java {:mean-μs 15777.87157142857, :variance-μs 0.001808629869948285}, :hamf-hashmap {:mean-μs 16273.244619047618, :variance-μs 2.682036882564654E-4}, :n-elems 1000000, :clj {:mean-μs 60807.11641666667, :variance-μs 0.5310154552200417}, :test :random-update, :numeric? false}] ================================================ FILE: results/sort-by.edn ================================================ [{:clj {:mean-μs 0.09368354713973853, :variance-μs 3.4749205456213755E-13}, :hamf {:mean-μs 0.14596828192467412, :variance-μs 5.984064042814943E-12}, :hamf-typed {:mean-μs 0.18029647307999264, :variance-μs 1.431082943304768E-11}, :n-elems 4, :test :sort-by, :numeric? true} {:clj {:mean-μs 0.3346017224491346, :variance-μs 2.7625131964102243E-12}, :hamf {:mean-μs 0.2159108130775009, :variance-μs 4.255235551841957E-12}, :hamf-typed {:mean-μs 0.2323090567932432, :variance-μs 4.03787766814031E-12}, :n-elems 10, :test :sort-by, :numeric? true} {:clj {:mean-μs 7.842569624343959, :variance-μs 1.971383294271551E-9}, :hamf {:mean-μs 2.118178162835768, :variance-μs 2.0992895696266137E-11}, :hamf-typed {:mean-μs 1.8497657610202387, :variance-μs 3.242863273800923E-10}, :n-elems 100, :test :sort-by, :numeric? true} {:clj {:mean-μs 148.49643230694036, :variance-μs 1.0612311214254018E-6}, :hamf {:mean-μs 38.419489832169546, :variance-μs 4.743966092745893E-6}, :hamf-typed {:mean-μs 32.32202215649647, :variance-μs 2.375869025074426E-6}, :n-elems 1000, :test :sort-by, :numeric? true} {:clj {:mean-μs 2282.277214814815, :variance-μs 1.0071167063558871E-4}, :hamf {:mean-μs 570.5418945386065, :variance-μs 6.974032847926867E-6}, :hamf-typed {:mean-μs 466.0654403100776, :variance-μs 1.58677409917289E-5}, :n-elems 10000, :test :sort-by, :numeric? true} {:clj {:mean-μs 31241.091, :variance-μs 0.017035503393293683}, :hamf {:mean-μs 3290.2144731182802, :variance-μs 1.0871166837450611E-4}, :hamf-typed {:mean-μs 2683.9532894736844, :variance-μs 6.081506563538577E-4}, :n-elems 100000, :test :sort-by, :numeric? true} {:clj {:mean-μs 556215.5891666667, :variance-μs 8.178382644753714}, :hamf {:mean-μs 37757.84622222223, :variance-μs 0.2343899319752739}, :hamf-typed {:mean-μs 28849.764583333334, :variance-μs 0.43991080556205003}, :n-elems 1000000, :test :sort-by, :numeric? true}] ================================================ FILE: results/typed-parallel-reductions.edn ================================================ [{:clj {:mean-μs 6.109750675343813, :variance-μs 6.1388779737180735E-9}, :hamf-untyped {:mean-μs 19.804417360560404, :variance-μs 1.4171681970281903E-8}, :hamf-typed {:mean-μs 15.336961038961041, :variance-μs 3.6832920022683177E-9}, :hamf-consumer {:mean-μs 14.204317738380002, :variance-μs 1.848603195751864E-9}, :n-elems 1000, :numeric? true, :test :typed-parallel-reductions} {:clj {:mean-μs 5610.183491228071, :variance-μs 0.006643800428007367}, :hamf-untyped {:mean-μs 1071.3932925531915, :variance-μs 1.0762994372732568E-5}, :hamf-typed {:mean-μs 448.69007812499996, :variance-μs 2.867989318426488E-6}, :hamf-consumer {:mean-μs 392.8345065359478, :variance-μs 1.0494138131426487E-6}, :n-elems 1000000, :numeric? true, :test :typed-parallel-reductions} {:clj {:mean-μs 572202.5613333334, :variance-μs 8.731098992905238}, :hamf-untyped {:mean-μs 86518.08233333332, :variance-μs 0.02604559385406698}, :hamf-typed {:mean-μs 34243.37627777778, :variance-μs 0.21671453178828554}, :hamf-consumer {:mean-μs 28208.393125000002, :variance-μs 0.007329199036216756}, :n-elems 100000000, :numeric? true, :test :typed-parallel-reductions}] ================================================ FILE: results/typed-reductions-intel.edn ================================================ [{:clj {:mean-μs 0.08083155079773072, :variance-μs 2.607437776221788E-13}, :clj-typed {:mean-μs 0.1103805778248939, :variance-μs 2.439651290536169E-13}, :hamf-partial {:mean-μs 0.13004735309677243, :variance-μs 5.784854897254039E-13}, :hamf-deftype-consumer {:mean-μs 0.1953989385738267, :variance-μs 4.9865286056145706E-12}, :hamf-java-consumer {:mean-μs 0.23984735686088046, :variance-μs 4.554884450446967E-12}, :n-elems 4, :numeric? true, :test :typed-reductions} {:clj {:mean-μs 0.23319472260485014, :variance-μs 6.504487014189002E-11}, :clj-typed {:mean-μs 0.2648384502886857, :variance-μs 3.0752725590709146E-12}, :hamf-partial {:mean-μs 0.46332126574765975, :variance-μs 1.3427220642780956E-11}, :hamf-deftype-consumer {:mean-μs 0.4394848976129789, :variance-μs 2.3559917416110998E-11}, :hamf-java-consumer {:mean-μs 0.44066994734026077, :variance-μs 6.5227574804474215E-12}, :n-elems 10, :numeric? true, :test :typed-reductions} {:clj {:mean-μs 1.8098834931427243, :variance-μs 3.6875297161309607E-10}, :clj-typed {:mean-μs 2.213904877494819, :variance-μs 5.903275495642295E-10}, :hamf-partial {:mean-μs 2.985199862782352, :variance-μs 6.654646560983587E-10}, :hamf-deftype-consumer {:mean-μs 2.3535886494121936, :variance-μs 4.873315280030986E-10}, :hamf-java-consumer {:mean-μs 2.3654510956687145, :variance-μs 1.6598586638706686E-10}, :n-elems 100, :numeric? true, :test :typed-reductions} {:clj {:mean-μs 19.3617035337454, :variance-μs 1.738537575972473E-8}, :clj-typed {:mean-μs 23.88976932840988, :variance-μs 4.096933742156692E-8}, :hamf-partial {:mean-μs 29.751869933712126, :variance-μs 8.461874211572359E-9}, :hamf-deftype-consumer {:mean-μs 21.85484181492208, :variance-μs 6.944969205156315E-9}, :hamf-java-consumer {:mean-μs 21.694217533635513, :variance-μs 1.8892073846351382E-8}, :n-elems 1000, :numeric? true, :test :typed-reductions} {:clj {:mean-μs 194.30773903508774, :variance-μs 4.764487843415978E-6}, :clj-typed {:mean-μs 239.42507523148151, :variance-μs 1.3964075214313341E-6}, :hamf-partial {:mean-μs 297.6285317002882, :variance-μs 1.3538777855842707E-5}, :hamf-deftype-consumer {:mean-μs 213.80862943989075, :variance-μs 3.4080759030627283E-6}, :hamf-java-consumer {:mean-μs 211.8145369220152, :variance-μs 5.458272351473067E-6}, :n-elems 10000, :numeric? true, :test :typed-reductions} {:clj {:mean-μs 19653.14427777778, :variance-μs 0.31501798511343293}, :clj-typed {:mean-μs 23928.633933333334, :variance-μs 0.07455115523551598}, :hamf-partial {:mean-μs 28838.906708333336, :variance-μs 0.03931331868764148}, :hamf-deftype-consumer {:mean-μs 20856.439233333334, :variance-μs 0.024563788894250656}, :hamf-java-consumer {:mean-μs 20685.577366666665, :variance-μs 0.01255119205303075}, :n-elems 1000000, :numeric? true, :test :typed-reductions}] ================================================ FILE: results/typed-reductions.edn ================================================ [{:clj {:mean-μs 0.020973830885365856, :variance-μs 1.5389297939653883E-13}, :clj-typed {:mean-μs 0.03145851381475773, :variance-μs 5.903242517478972E-14}, :hamf-partial {:mean-μs 0.059273714740684255, :variance-μs 1.2371784565925692E-11}, :hamf-deftype-consumer {:mean-μs 0.04533951706819582, :variance-μs 1.9485949331172755E-14}, :hamf-java-consumer {:mean-μs 0.05566604482153488, :variance-μs 2.610345084081093E-12}, :n-elems 4, :numeric? true, :test :typed-reductions} {:clj {:mean-μs 0.047609480990502294, :variance-μs 2.1159911315394183E-13}, :clj-typed {:mean-μs 0.05651426818074108, :variance-μs 1.3201867191062258E-12}, :hamf-partial {:mean-μs 0.15300628831607596, :variance-μs 3.9885298864863E-12}, :hamf-deftype-consumer {:mean-μs 0.14999821983221245, :variance-μs 9.846322079703306E-13}, :hamf-java-consumer {:mean-μs 0.14961160653301617, :variance-μs 5.251461002222532E-13}, :n-elems 10, :numeric? true, :test :typed-reductions} {:clj {:mean-μs 0.45921288628482243, :variance-μs 4.125427462132233E-11}, :clj-typed {:mean-μs 0.40030021051129266, :variance-μs 1.978201113004833E-12}, :hamf-partial {:mean-μs 1.162238944524843, :variance-μs 7.024705635842096E-11}, :hamf-deftype-consumer {:mean-μs 1.0722720725926402, :variance-μs 3.840528539811712E-11}, :hamf-java-consumer {:mean-μs 1.0578596457127247, :variance-μs 1.8860351768678773E-11}, :n-elems 100, :numeric? true, :test :typed-reductions} {:clj {:mean-μs 5.057716056512496, :variance-μs 5.441775028540305E-10}, :clj-typed {:mean-μs 5.1675323381853, :variance-μs 1.7268263670282741E-9}, :hamf-partial {:mean-μs 11.075325958213787, :variance-μs 4.197215172769934E-9}, :hamf-deftype-consumer {:mean-μs 10.362469034236076, :variance-μs 1.4804452994402536E-9}, :hamf-java-consumer {:mean-μs 10.182020976041988, :variance-μs 1.3093141731145295E-9}, :n-elems 1000, :numeric? true, :test :typed-reductions} {:clj {:mean-μs 50.798319254763115, :variance-μs 4.745629116433968E-8}, :clj-typed {:mean-μs 51.5585498403093, :variance-μs 7.521515048479323E-7}, :hamf-partial {:mean-μs 110.28216556169428, :variance-μs 1.0787136356440518E-6}, :hamf-deftype-consumer {:mean-μs 103.50167146282973, :variance-μs 1.9744936828106858E-7}, :hamf-java-consumer {:mean-μs 102.26325822312648, :variance-μs 3.310837696169239E-7}, :n-elems 10000, :numeric? true, :test :typed-reductions} {:clj {:mean-μs 5108.328216666667, :variance-μs 8.538736042629945E-4}, :clj-typed {:mean-μs 5187.716408333334, :variance-μs 0.005420315139756417}, :hamf-partial {:mean-μs 11014.015766666667, :variance-μs 0.014137288650498668}, :hamf-deftype-consumer {:mean-μs 10451.190766666667, :variance-μs 0.0034926461597226547}, :hamf-java-consumer {:mean-μs 10335.650483333333, :variance-μs 0.003020790403802994}, :n-elems 1000000, :numeric? true, :test :typed-reductions}] ================================================ FILE: results/union-disj.edn ================================================ [{:hamf-trie {:mean-μs 0.05387404263091655, :variance-μs 7.601879123779283E-12}, :java {:mean-μs 0.07206509120153398, :variance-μs 8.814124155631456E-13}, :hamf-hashmap {:mean-μs 0.06297155107772515, :variance-μs 3.568340166695999E-11}, :clj-int-map {:mean-μs 0.14015141141420245, :variance-μs 7.080369007861796E-12}, :n-elems 4, :clj {:mean-μs 0.1943523048384093, :variance-μs 1.443176048149449E-11}, :test :union-disj, :hamf-long-map {:mean-μs 0.035919899195097624, :variance-μs 4.73272322792755E-14}, :numeric? true} {:hamf-trie {:mean-μs 0.05586932456545242, :variance-μs 7.648525495599117E-12}, :java {:mean-μs 0.08033694721122178, :variance-μs 3.698528623414985E-11}, :hamf-hashmap {:mean-μs 0.05333671496349147, :variance-μs 3.939069448801596E-12}, :n-elems 4, :clj {:mean-μs 0.16365682288915165, :variance-μs 9.850253772814097E-12}, :test :union-disj, :numeric? false} {:hamf-trie {:mean-μs 0.09101537476401057, :variance-μs 2.3126389658731673E-12}, :java {:mean-μs 0.18305658882513484, :variance-μs 4.8930132832954695E-12}, :hamf-hashmap {:mean-μs 0.08930336189814504, :variance-μs 2.638454874290202E-13}, :clj-int-map {:mean-μs 0.385461959718345, :variance-μs 1.8691699696931915E-12}, :n-elems 10, :clj {:mean-μs 0.8336979928326932, :variance-μs 1.0026382184343124E-9}, :test :union-disj, :hamf-long-map {:mean-μs 0.07489276761384404, :variance-μs 4.839374073933964E-13}, :numeric? true} {:hamf-trie {:mean-μs 0.10764307121408206, :variance-μs 2.8894234351012996E-13}, :java {:mean-μs 0.18663758329506372, :variance-μs 3.439659031376218E-11}, :hamf-hashmap {:mean-μs 0.10605252259190305, :variance-μs 1.9719122525819353E-13}, :n-elems 10, :clj {:mean-μs 0.5302580652648192, :variance-μs 1.4528192279555148E-11}, :test :union-disj, :numeric? false} {:hamf-trie {:mean-μs 0.8610978648948328, :variance-μs 1.5299490300003772E-11}, :java {:mean-μs 1.6728428883700135, :variance-μs 3.6640709814869684E-9}, :hamf-hashmap {:mean-μs 0.6602829319211199, :variance-μs 1.3741791219221696E-11}, :clj-int-map {:mean-μs 4.26644650450757, :variance-μs 8.993285462112174E-11}, :n-elems 100, :clj {:mean-μs 5.057957208341729, :variance-μs 3.130108371957946E-9}, :test :union-disj, :hamf-long-map {:mean-μs 0.6505923338527289, :variance-μs 1.2013612366560051E-11}, :numeric? true} {:hamf-trie {:mean-μs 0.9934630280303927, :variance-μs 2.697913343778186E-9}, :java {:mean-μs 1.7003467974121873, :variance-μs 2.2301722095663205E-9}, :hamf-hashmap {:mean-μs 1.2105747917688385, :variance-μs 2.352238409535548E-9}, :n-elems 100, :clj {:mean-μs 4.703321511573859, :variance-μs 3.214922306780342E-9}, :test :union-disj, :numeric? false} {:hamf-trie {:mean-μs 10.033796019202594, :variance-μs 4.0936306117393106E-7}, :java {:mean-μs 18.62291703310126, :variance-μs 3.303296668707903E-7}, :hamf-hashmap {:mean-μs 6.708932768398129, :variance-μs 2.834718174727148E-7}, :clj-int-map {:mean-μs 77.57473166666666, :variance-μs 3.4202783716722863E-6}, :n-elems 1000, :clj {:mean-μs 65.11077131526768, :variance-μs 5.809813435720986E-7}, :test :union-disj, :hamf-long-map {:mean-μs 5.567049100929166, :variance-μs 2.172001389597534E-9}, :numeric? true} {:hamf-trie {:mean-μs 9.541903893978027, :variance-μs 2.025414018716051E-8}, :java {:mean-μs 21.034355885817227, :variance-μs 2.7239025039167727E-7}, :hamf-hashmap {:mean-μs 11.213639267605636, :variance-μs 3.100144207897286E-8}, :n-elems 1000, :clj {:mean-μs 61.512498178506384, :variance-μs 2.0769263334273362E-7}, :test :union-disj, :numeric? false} {:hamf-trie {:mean-μs 157.52274583333332, :variance-μs 1.1360718520883453E-5}, :java {:mean-μs 286.98687321937325, :variance-μs 1.0208708642513169E-5}, :hamf-hashmap {:mean-μs 149.84130009298002, :variance-μs 8.020519742463194E-5}, :clj-int-map {:mean-μs 962.540546474359, :variance-μs 8.667281439737801E-4}, :n-elems 10000, :clj {:mean-μs 650.8663758169935, :variance-μs 2.8210312846365386E-5}, :test :union-disj, :hamf-long-map {:mean-μs 124.9529899052328, :variance-μs 5.3337277789910735E-6}, :numeric? true} {:hamf-trie {:mean-μs 162.7947814814815, :variance-μs 8.707301176213405E-6}, :java {:mean-μs 328.07865468409585, :variance-μs 9.621875320371119E-6}, :hamf-hashmap {:mean-μs 191.68460606060606, :variance-μs 1.8009645296775687E-6}, :n-elems 10000, :clj {:mean-μs 599.7976074297188, :variance-μs 1.1665058423438299E-5}, :test :union-disj, :numeric? false} {:hamf-trie {:mean-μs 5174.599858333333, :variance-μs 0.01487523512920403}, :java {:mean-μs 4691.425795454546, :variance-μs 0.011931599349578487}, :hamf-hashmap {:mean-μs 2661.0065875, :variance-μs 0.013205702558121592}, :clj-int-map {:mean-μs 22581.566633333336, :variance-μs 1.3293161128934872}, :n-elems 1000000, :clj {:mean-μs 21840.236100000002, :variance-μs 3.8839211955426007}, :test :union-disj, :hamf-long-map {:mean-μs 1795.5331338797819, :variance-μs 0.011691181337103302}, :numeric? true} {:hamf-trie {:mean-μs 4984.485531746032, :variance-μs 0.029493650615204557}, :java {:mean-μs 8145.096730769231, :variance-μs 0.03162908116503202}, :hamf-hashmap {:mean-μs 3729.94931547619, :variance-μs 0.016524662754384047}, :n-elems 1000000, :clj {:mean-μs 20441.775111111117, :variance-μs 2.594552115168829}, :test :union-disj, :numeric? false}] ================================================ FILE: results/union-overlapping.edn ================================================ [{:hamf-trie {:mean-μs 0.10502638201374732, :variance-μs 9.426261585611141E-14}, :java {:mean-μs 0.1791063653635941, :variance-μs 7.260687089336133E-11}, :hamf-hashmap {:mean-μs 0.07032127191331726, :variance-μs 3.592262253795875E-11}, :clj-int-map {:mean-μs 0.3143601710680975, :variance-μs 3.0287656017762097E-12}, :n-elems 4, :clj {:mean-μs 0.3608711911743155, :variance-μs 1.8110786849075344E-11}, :test :union-overlapping, :hamf-long-map {:mean-μs 0.04893065959274923, :variance-μs 1.73764889882564E-13}, :numeric? true} {:hamf-trie {:mean-μs 0.1220632807960718, :variance-μs 1.0108794962904816E-13}, :java {:mean-μs 0.14789828961326076, :variance-μs 3.2273257527005675E-13}, :hamf-hashmap {:mean-μs 0.06324700870921762, :variance-μs 2.2726279112956253E-13}, :n-elems 4, :clj {:mean-μs 0.22202906084300278, :variance-μs 2.962560488187324E-11}, :test :union-overlapping, :numeric? false} {:hamf-trie {:mean-μs 0.21233526842619083, :variance-μs 9.390680559258845E-14}, :java {:mean-μs 0.33055351628735746, :variance-μs 1.826352668823769E-10}, :hamf-hashmap {:mean-μs 0.16288825709471177, :variance-μs 2.038690774759388E-12}, :clj-int-map {:mean-μs 0.7167336882052902, :variance-μs 9.729527522057529E-12}, :n-elems 10, :clj {:mean-μs 0.9405646974483907, :variance-μs 7.987818630075633E-11}, :test :union-overlapping, :hamf-long-map {:mean-μs 0.12042608814025262, :variance-μs 6.23845724318453E-13}, :numeric? true} {:hamf-trie {:mean-μs 0.2298904471968627, :variance-μs 1.1131540307538773E-12}, :java {:mean-μs 0.33493803425830315, :variance-μs 2.9542118566664694E-10}, :hamf-hashmap {:mean-μs 0.20690905870206056, :variance-μs 7.298883883877252E-12}, :n-elems 10, :clj {:mean-μs 0.8966828136432882, :variance-μs 1.887251153815189E-10}, :test :union-overlapping, :numeric? false} {:hamf-trie {:mean-μs 2.176490849349662, :variance-μs 5.502495993107607E-10}, :java {:mean-μs 3.2023792385301286, :variance-μs 2.4673814200821513E-8}, :hamf-hashmap {:mean-μs 1.5024097582103233, :variance-μs 3.392305513102097E-10}, :clj-int-map {:mean-μs 10.345330417781437, :variance-μs 4.901126268291725E-9}, :n-elems 100, :clj {:mean-μs 10.41371986549405, :variance-μs 1.7435177975930493E-9}, :test :union-overlapping, :hamf-long-map {:mean-μs 0.9984897507617002, :variance-μs 1.9496814931296483E-10}, :numeric? true} {:hamf-trie {:mean-μs 2.324429762964032, :variance-μs 7.540512787524547E-10}, :java {:mean-μs 3.3033282678013465, :variance-μs 2.7080255603092382E-8}, :hamf-hashmap {:mean-μs 1.5853071883611523, :variance-μs 3.103971504267881E-10}, :n-elems 100, :clj {:mean-μs 9.487314385333839, :variance-μs 4.117894545524918E-9}, :test :union-overlapping, :numeric? false} {:hamf-trie {:mean-μs 25.501163470434857, :variance-μs 4.711786482155854E-7}, :java {:mean-μs 37.90233383252215, :variance-μs 1.878993769774089E-8}, :hamf-hashmap {:mean-μs 16.578812079054877, :variance-μs 3.721841769265016E-8}, :clj-int-map {:mean-μs 154.41381233386497, :variance-μs 1.4142949825301602E-6}, :n-elems 1000, :clj {:mean-μs 130.17339845474615, :variance-μs 1.715224953583912E-6}, :test :union-overlapping, :hamf-long-map {:mean-μs 11.311622755505365, :variance-μs 1.0237344636433006E-8}, :numeric? true} {:hamf-trie {:mean-μs 25.732278143459922, :variance-μs 5.6240939353385624E-8}, :java {:mean-μs 44.371754378219286, :variance-μs 6.51773073340733E-7}, :hamf-hashmap {:mean-μs 19.644183598937587, :variance-μs 6.408836539021094E-8}, :n-elems 1000, :clj {:mean-μs 119.1047305882353, :variance-μs 8.702885259979214E-7}, :test :union-overlapping, :numeric? false} {:hamf-trie {:mean-μs 321.54726411075615, :variance-μs 2.7314817905134994E-5}, :java {:mean-μs 484.5736138211382, :variance-μs 7.312125421957086E-5}, :hamf-hashmap {:mean-μs 246.8647060240964, :variance-μs 1.0172197986893092E-5}, :clj-int-map {:mean-μs 1984.1942533333338, :variance-μs 0.008893258581366043}, :n-elems 10000, :clj {:mean-μs 1285.800107594937, :variance-μs 6.861328450502594E-5}, :test :union-overlapping, :hamf-long-map {:mean-μs 167.27258664459163, :variance-μs 1.208741398739636E-5}, :numeric? true} {:hamf-trie {:mean-μs 348.9733344827586, :variance-μs 5.533058293016269E-6}, :java {:mean-μs 516.8418498349835, :variance-μs 2.1618437770980092E-5}, :hamf-hashmap {:mean-μs 288.0162975517891, :variance-μs 2.2954193152688092E-5}, :n-elems 10000, :clj {:mean-μs 1189.4219392156863, :variance-μs 2.4065541122703955E-4}, :test :union-overlapping, :numeric? false} {:hamf-trie {:mean-μs 2911.2336171171173, :variance-μs 0.019254886931371996}, :java {:mean-μs 3602.4680000000003, :variance-μs 0.0035377872634102313}, :hamf-hashmap {:mean-μs 1783.2670134408604, :variance-μs 0.016966164123158312}, :clj-int-map {:mean-μs 20299.600333333336, :variance-μs 2.2236462300057527}, :n-elems 1000000, :clj {:mean-μs 18774.30525, :variance-μs 2.4784938356067845}, :test :union-overlapping, :hamf-long-map {:mean-μs 1287.4669568627453, :variance-μs 0.0011637377945428373}, :numeric? true} {:hamf-trie {:mean-μs 4279.03372, :variance-μs 0.020357493386886187}, :java {:mean-μs 6655.074855555556, :variance-μs 0.010340817931078538}, :hamf-hashmap {:mean-μs 3435.771166666667, :variance-μs 0.005673278505370591}, :n-elems 1000000, :clj {:mean-μs 20370.107333333333, :variance-μs 3.2669529107933153}, :test :union-overlapping, :numeric? false}] ================================================ FILE: results/union-reduce-transient.edn ================================================ [{:hamf-trans-long-map {:mean-μs 0.7412825333830333, :variance-μs 3.170731692172204E-10}, :hamf-hashmap {:mean-μs 0.8866651053184513, :variance-μs 3.484445195766126E-12}, :n-elems 4, :test :union-reduce-transient, :hamf-long-map {:mean-μs 0.8786651060075851, :variance-μs 1.153713343546831E-11}, :numeric? true, :hamf-trans-hashmap {:mean-μs 0.7622947671498717, :variance-μs 1.879670822353778E-10}} {:hamf-trans-long-map {:mean-μs 2.2578933011538433, :variance-μs 1.1912266153907903E-9}, :hamf-hashmap {:mean-μs 2.12513856186766, :variance-μs 3.44592015940261E-10}, :n-elems 10, :test :union-reduce-transient, :hamf-long-map {:mean-μs 2.1200784115269724, :variance-μs 1.3821808227797617E-10}, :numeric? true, :hamf-trans-hashmap {:mean-μs 2.2487650346317354, :variance-μs 1.12960205996829E-9}} {:hamf-trans-long-map {:mean-μs 20.318177081213108, :variance-μs 9.360366853700005E-8}, :hamf-hashmap {:mean-μs 18.764085931700073, :variance-μs 1.828781314253933E-8}, :n-elems 100, :test :union-reduce-transient, :hamf-long-map {:mean-μs 18.831902556409773, :variance-μs 5.1392113031581955E-9}, :numeric? true, :hamf-trans-hashmap {:mean-μs 21.68933647589471, :variance-μs 1.743532937661252E-7}} {:hamf-trans-long-map {:mean-μs 224.48590850417614, :variance-μs 1.0103915961737732E-4}, :hamf-hashmap {:mean-μs 219.95389528985507, :variance-μs 1.0649025456953546E-5}, :n-elems 1000, :test :union-reduce-transient, :hamf-long-map {:mean-μs 218.86707522123893, :variance-μs 3.0514713995301176E-5}, :numeric? true, :hamf-trans-hashmap {:mean-μs 225.72055139833714, :variance-μs 7.644137395932283E-6}} {:hamf-trans-long-map {:mean-μs 2679.216245614035, :variance-μs 3.578130407433292E-4}, :hamf-hashmap {:mean-μs 2798.3279594594596, :variance-μs 0.001752698461439497}, :n-elems 10000, :test :union-reduce-transient, :hamf-long-map {:mean-μs 2778.5944864864864, :variance-μs 0.0011648177035861703}, :numeric? true, :hamf-trans-hashmap {:mean-μs 2681.1084166666665, :variance-μs 7.942134300416677E-4}} {:hamf-trans-long-map {:mean-μs 15799.275738095239, :variance-μs 0.002852110740338103}, :hamf-hashmap {:mean-μs 17891.047555555557, :variance-μs 0.025927661871208384}, :n-elems 1000000, :test :union-reduce-transient, :hamf-long-map {:mean-μs 18630.76869444445, :variance-μs 0.0395750146766083}, :numeric? true, :hamf-trans-hashmap {:mean-μs 15486.036642857143, :variance-μs 0.046683416241262723}}] ================================================ FILE: results/union-reduce.edn ================================================ [{:hamf-trie {:mean-μs 1.7252145524853624, :variance-μs 2.011615888905509E-10}, :java {:mean-μs 38.32988724387806, :variance-μs 1.8803434110405448E-6}, :hamf-hashmap {:mean-μs 1.2381281025521231, :variance-μs 7.209768527187194E-9}, :clj-int-map {:mean-μs 5.861840907242486, :variance-μs 1.9579476680473757E-9}, :n-elems 4, :clj {:mean-μs 7.42634866750487, :variance-μs 7.627804021594806E-9}, :test :union-reduce, :hamf-long-map {:mean-μs 1.106631402508452, :variance-μs 6.2822972924059196E-9}, :numeric? true} {:hamf-trie {:mean-μs 2.17823404331, :variance-μs 1.2554786716603527E-9}, :java {:mean-μs 38.116079696273616, :variance-μs 2.067486699868819E-8}, :hamf-hashmap {:mean-μs 1.4451719973078556, :variance-μs 4.191455542888255E-10}, :n-elems 4, :clj {:mean-μs 5.422725345839924, :variance-μs 3.0843970332480513E-9}, :test :union-reduce, :numeric? false} {:hamf-trie {:mean-μs 6.059379510739388, :variance-μs 3.148597844167285E-9}, :java {:mean-μs 93.98164679911702, :variance-μs 7.586405510284673E-8}, :hamf-hashmap {:mean-μs 2.3757701729699483, :variance-μs 2.534243076758045E-10}, :clj-int-map {:mean-μs 15.311703172085648, :variance-μs 4.910261252366723E-9}, :n-elems 10, :clj {:mean-μs 18.259924595233763, :variance-μs 1.8242569414470183E-8}, :test :union-reduce, :hamf-long-map {:mean-μs 1.9576182610238315, :variance-μs 6.291706049500769E-11}, :numeric? true} {:hamf-trie {:mean-μs 3.9894582821518676, :variance-μs 1.1492966005754426E-8}, :java {:mean-μs 94.84054833647205, :variance-μs 1.5161304455261573E-7}, :hamf-hashmap {:mean-μs 2.6381999444324946, :variance-μs 3.6644632694290616E-8}, :n-elems 10, :clj {:mean-μs 13.594286938534278, :variance-μs 1.511480975985681E-8}, :test :union-reduce, :numeric? false} {:hamf-trie {:mean-μs 53.18793429744689, :variance-μs 1.4319146246254403E-7}, :java {:mean-μs 943.35361682243, :variance-μs 2.764350354297334E-5}, :hamf-hashmap {:mean-μs 24.15856529495719, :variance-μs 4.097383320823301E-6}, :clj-int-map {:mean-μs 217.43048051948054, :variance-μs 4.965510566066485E-6}, :n-elems 100, :clj {:mean-μs 179.73787843488654, :variance-μs 3.777078889569355E-6}, :test :union-reduce, :hamf-long-map {:mean-μs 17.67657798219235, :variance-μs 8.757207605184863E-9}, :numeric? true} {:hamf-trie {:mean-μs 51.102250000000005, :variance-μs 3.838505344378272E-7}, :java {:mean-μs 942.6637720125786, :variance-μs 1.2386850963142328E-5}, :hamf-hashmap {:mean-μs 27.14281319669184, :variance-μs 2.467018502603686E-7}, :n-elems 100, :clj {:mean-μs 162.32703961748635, :variance-μs 1.1839934684263025E-4}, :test :union-reduce, :numeric? false} {:hamf-trie {:mean-μs 524.5056842105263, :variance-μs 1.0863700809083028E-5}, :java {:mean-μs 9408.903106060607, :variance-μs 0.002817331073447118}, :hamf-hashmap {:mean-μs 314.069758744856, :variance-μs 5.1753992843360013E-5}, :clj-int-map {:mean-μs 2980.940348484849, :variance-μs 0.02232584565807595}, :n-elems 1000, :clj {:mean-μs 2220.5140111111114, :variance-μs 5.021514807700558E-4}, :test :union-reduce, :hamf-long-map {:mean-μs 246.04637132352943, :variance-μs 1.2300311707203191E-5}, :numeric? true} {:hamf-trie {:mean-μs 542.4614453405019, :variance-μs 5.173914501500367E-5}, :java {:mean-μs 9467.722575757574, :variance-μs 0.0029632205643559035}, :hamf-hashmap {:mean-μs 379.6645031210986, :variance-μs 2.5376491541213855E-5}, :n-elems 1000, :clj {:mean-μs 2071.348385416667, :variance-μs 0.020338011925833777}, :test :union-reduce, :numeric? false} {:hamf-trie {:mean-μs 5917.341805555556, :variance-μs 0.009515687917660094}, :java {:mean-μs 99796.31791666668, :variance-μs 11.563507553274604}, :hamf-hashmap {:mean-μs 3659.8134107142855, :variance-μs 0.001547409551530098}, :clj-int-map {:mean-μs 36213.265, :variance-μs 0.3727468710440302}, :n-elems 10000, :clj {:mean-μs 22785.483366666667, :variance-μs 0.05900913628111442}, :test :union-reduce, :hamf-long-map {:mean-μs 3027.809068627451, :variance-μs 4.251032741093422E-4}, :numeric? true} {:hamf-trie {:mean-μs 6372.496270833333, :variance-μs 0.0057097176436725116}, :java {:mean-μs 103640.92316666667, :variance-μs 58.24561118563865}, :hamf-hashmap {:mean-μs 4942.851936507937, :variance-μs 0.034681454680306936}, :n-elems 10000, :clj {:mean-μs 22197.684250000002, :variance-μs 3.0316012223332187}, :test :union-reduce, :numeric? false} {:hamf-trie {:mean-μs 58398.40116666667, :variance-μs 1.754406274143675}, :java {:mean-μs 1096804.0131666667, :variance-μs 251.4701104954116}, :hamf-hashmap {:mean-μs 25968.265233333335, :variance-μs 0.13249291644170835}, :clj-int-map {:mean-μs 410986.65883333335, :variance-μs 428.0071516463167}, :n-elems 1000000, :clj {:mean-μs 365097.666, :variance-μs 562.6211510776219}, :test :union-reduce, :hamf-long-map {:mean-μs 24091.087466666668, :variance-μs 4.293764864312841}, :numeric? true} {:hamf-trie {:mean-μs 75765.96383333334, :variance-μs 3.399043885345373}, :java {:mean-μs 1219823.3045, :variance-μs 552.5941588752235}, :hamf-hashmap {:mean-μs 59915.99491666666, :variance-μs 16.822256157834293}, :n-elems 1000000, :clj {:mean-μs 407285.2278333333, :variance-μs 1188.415009698171}, :test :union-reduce, :numeric? false}] ================================================ FILE: results/update-values.edn ================================================ [{:hamf-trie {:mean-μs 0.02327796051118421, :variance-μs 1.529798206229456E-12}, :java {:mean-μs 0.07847960494396915, :variance-μs 6.967239475562219E-11}, :hamf-hashmap {:mean-μs 0.04177343050693086, :variance-μs 2.793246669872514E-11}, :clj-int-map {:mean-μs 0.2140203792702577, :variance-μs 1.7091480124142276E-9}, :n-elems 4, :clj {:mean-μs 0.16000777451163323, :variance-μs 1.6225888151827234E-11}, :test :update-values, :hamf-long-map {:mean-μs 0.031042283155488796, :variance-μs 3.667389465617035E-14}, :numeric? true} {:hamf-trie {:mean-μs 0.14682536268574706, :variance-μs 1.1720200400734721E-8}, :java {:mean-μs 0.07505282953979409, :variance-μs 1.958352763068874E-12}, :hamf-hashmap {:mean-μs 0.07620133611255923, :variance-μs 1.9538818411170063E-10}, :n-elems 4, :clj {:mean-μs 0.12028106134618147, :variance-μs 8.664968021466646E-12}, :test :update-values, :numeric? false} {:hamf-trie {:mean-μs 0.11711380170394013, :variance-μs 1.1120362518620985E-12}, :java {:mean-μs 0.1426209117711911, :variance-μs 1.9768546530060133E-13}, :hamf-hashmap {:mean-μs 0.08490383296119978, :variance-μs 1.4406649195760521E-11}, :clj-int-map {:mean-μs 0.37913624174515115, :variance-μs 6.846181047401567E-10}, :n-elems 10, :clj {:mean-μs 0.26714631887493423, :variance-μs 1.6715492379382318E-10}, :test :update-values, :hamf-long-map {:mean-μs 0.08867522080370929, :variance-μs 9.029038656283934E-14}, :numeric? true} {:hamf-trie {:mean-μs 0.11227565937962775, :variance-μs 1.008375961266171E-11}, :java {:mean-μs 0.15195437993096086, :variance-μs 1.241621304105891E-10}, :hamf-hashmap {:mean-μs 0.08544332713548886, :variance-μs 2.136955811180962E-11}, :n-elems 10, :clj {:mean-μs 0.2590125967719688, :variance-μs 1.6083054235535771E-10}, :test :update-values, :numeric? false} {:hamf-trie {:mean-μs 1.0189347159156787, :variance-μs 8.936932055536869E-9}, :java {:mean-μs 1.424921812796662, :variance-μs 2.892741965050164E-10}, :hamf-hashmap {:mean-μs 0.5425087975874657, :variance-μs 1.201778885326809E-10}, :clj-int-map {:mean-μs 5.351526782685392, :variance-μs 1.6986481748803361E-7}, :n-elems 100, :clj {:mean-μs 2.9119777834452663, :variance-μs 1.385181291203484E-8}, :test :update-values, :hamf-long-map {:mean-μs 0.5636471016724729, :variance-μs 7.935041214888953E-10}, :numeric? true} {:hamf-trie {:mean-μs 0.963894393494081, :variance-μs 8.554152614780146E-11}, :java {:mean-μs 1.5376735663532985, :variance-μs 1.9142271267603622E-10}, :hamf-hashmap {:mean-μs 0.5565060917111692, :variance-μs 1.1399854927089318E-11}, :n-elems 100, :clj {:mean-μs 2.693525257887008, :variance-μs 4.527435201244143E-9}, :test :update-values, :numeric? false} {:hamf-trie {:mean-μs 8.083828863188979, :variance-μs 5.203264679292682E-9}, :java {:mean-μs 16.096294095665172, :variance-μs 6.799347545588575E-8}, :hamf-hashmap {:mean-μs 6.190307065890513, :variance-μs 1.8338285552011452E-9}, :clj-int-map {:mean-μs 79.01754469345698, :variance-μs 3.2383455777678053E-6}, :n-elems 1000, :clj {:mean-μs 40.62912088923967, :variance-μs 3.0511749788563086E-6}, :test :update-values, :hamf-long-map {:mean-μs 5.960749848570888, :variance-μs 9.004788710038562E-8}, :numeric? true} {:hamf-trie {:mean-μs 8.645071313757573, :variance-μs 3.9951935857527376E-8}, :java {:mean-μs 19.83810250313718, :variance-μs 5.6948459817578805E-8}, :hamf-hashmap {:mean-μs 6.33829362877976, :variance-μs 4.9963727298931594E-9}, :n-elems 1000, :clj {:mean-μs 36.64484267523503, :variance-μs 3.1799669320644587E-6}, :test :update-values, :numeric? false} {:hamf-trie {:mean-μs 89.64856090314913, :variance-μs 9.184094334200789E-7}, :java {:mean-μs 288.5613842857143, :variance-μs 4.1490216851221805E-6}, :hamf-hashmap {:mean-μs 122.14282939322301, :variance-μs 1.89917898117354E-5}, :clj-int-map {:mean-μs 1042.6436542553192, :variance-μs 4.244495402500743E-5}, :n-elems 10000, :clj {:mean-μs 326.7029265175719, :variance-μs 6.44053470881768E-5}, :test :update-values, :hamf-long-map {:mean-μs 103.01725053078556, :variance-μs 2.033621312486714E-6}, :numeric? true} {:hamf-trie {:mean-μs 94.21875124378109, :variance-μs 2.475502125683672E-7}, :java {:mean-μs 315.3385751840168, :variance-μs 9.47068701674807E-6}, :hamf-hashmap {:mean-μs 123.19073616089209, :variance-μs 1.1321857119797999E-5}, :n-elems 10000, :clj {:mean-μs 279.60671802054156, :variance-μs 1.987768575559929E-5}, :test :update-values, :numeric? false} {:hamf-trie {:mean-μs 875.124011494253, :variance-μs 2.3593347263788428E-4}, :java {:mean-μs 1883.3649781420768, :variance-μs 0.047683306558661744}, :hamf-hashmap {:mean-μs 663.4260704761905, :variance-μs 0.009436127874248483}, :clj-int-map {:mean-μs 10616.249916666668, :variance-μs 0.07341608083279194}, :n-elems 1000000, :clj {:mean-μs 3817.3415246913587, :variance-μs 0.002413966098376224}, :test :update-values, :hamf-long-map {:mean-μs 625.1630857843138, :variance-μs 6.884754374434351E-4}, :numeric? true} {:hamf-trie {:mean-μs 1664.7518924731185, :variance-μs 6.001581389875809E-4}, :java {:mean-μs 4386.828270833334, :variance-μs 0.005613395971719895}, :hamf-hashmap {:mean-μs 1365.4537822222223, :variance-μs 2.3271949921746955E-4}, :n-elems 1000000, :clj {:mean-μs 3986.0304679487176, :variance-μs 0.00559609015678013}, :test :update-values, :numeric? false}] ================================================ FILE: results/vec-equals.edn ================================================ [{:clj {:mean-μs 0.013114838133872559, :variance-μs 8.516910019923821E-16}, :hamf {:mean-μs 0.013453478507802178, :variance-μs 5.500406624628508E-16}, :n-elems 4, :test :equals, :numeric? true} {:clj {:mean-μs 0.014078313993473786, :variance-μs 4.1520987302137574E-16}, :hamf {:mean-μs 0.014942907668277137, :variance-μs 1.1622913418498456E-15}, :n-elems 4, :test :equals, :numeric? false} {:clj {:mean-μs 0.02200595405762387, :variance-μs 1.6248563316139666E-16}, :hamf {:mean-μs 0.020377683927085583, :variance-μs 4.896998172086728E-15}, :n-elems 10, :test :equals, :numeric? true} {:clj {:mean-μs 0.021947801236497873, :variance-μs 4.8453164092927975E-15}, :hamf {:mean-μs 0.020389919756287723, :variance-μs 5.517379878526508E-15}, :n-elems 10, :test :equals, :numeric? false} {:clj {:mean-μs 0.4129872683330938, :variance-μs 3.4952275396514294E-12}, :hamf {:mean-μs 0.12356661450685098, :variance-μs 6.762682580051647E-13}, :n-elems 100, :test :equals, :numeric? true} {:clj {:mean-μs 0.41086788497176263, :variance-μs 7.9542700263153E-12}, :hamf {:mean-μs 0.12288005640342069, :variance-μs 3.705936325434058E-13}, :n-elems 100, :test :equals, :numeric? false} {:clj {:mean-μs 7.628584670906521, :variance-μs 2.031587087069574E-9}, :hamf {:mean-μs 1.5920195382633124, :variance-μs 7.833875350922081E-11}, :n-elems 1000, :test :equals, :numeric? true} {:clj {:mean-μs 3.9882135278514594, :variance-μs 2.26510125976812E-10}, :hamf {:mean-μs 1.2153155253505414, :variance-μs 1.946881781725304E-11}, :n-elems 1000, :test :equals, :numeric? false} {:clj {:mean-μs 96.19576477199743, :variance-μs 6.078162864496903E-7}, :hamf {:mean-μs 17.308249798642272, :variance-μs 1.119495552027586E-8}, :n-elems 10000, :test :equals, :numeric? true} {:clj {:mean-μs 53.58084915102771, :variance-μs 1.5045705978512028E-7}, :hamf {:mean-μs 12.022921093470218, :variance-μs 1.6352125401067207E-9}, :n-elems 10000, :test :equals, :numeric? false} {:clj {:mean-μs 1105.8278095238097, :variance-μs 5.674031963247992E-5}, :hamf {:mean-μs 175.21224855324076, :variance-μs 9.207391310371014E-7}, :n-elems 100000, :test :equals, :numeric? true} {:clj {:mean-μs 678.9248702460852, :variance-μs 1.4946960727416105E-5}, :hamf {:mean-μs 121.41390271035598, :variance-μs 4.593320347574881E-7}, :n-elems 100000, :test :equals, :numeric? false}] ================================================ FILE: scripts/benchmark ================================================ #!/bin/bash echo "Building uberjar" rm -rf target clj -T:build perftest java -Djdk.attach.allowAttachSelf -jar target/uber-ham-fisted.jar ================================================ FILE: scripts/compile ================================================ #!/bin/bash rm -rf target/classes clojure -T:build compile ================================================ FILE: scripts/deploy ================================================ #!/bin/bash set -e scripts/run-tests rm -f pom.xml clj -T:build jar cp target/classes/META-INF/maven/com.cnuernber/ham-fisted/pom.xml . clj -X:codox scripts/reformat clj -X:deploy ================================================ FILE: scripts/enable-jdk17 ================================================ #!/bin/bash if [ ! -e jdk-17.0.1 ]; then wget https://download.java.net/java/GA/jdk17.0.1/2a2082e5a09d4267845be086888add4f/12/GPL/openjdk-17.0.1_linux-x64_bin.tar.gz tar -xvzf openjdk-17.0.1_linux-x64_bin.tar.gz rm openjdk-17.0.1_linux-x64_bin.tar.gz fi export PATH=$(pwd)/jdk-17.0.1/bin:$PATH export JAVA_HOME=$(pwd)/jdk-17.0.1/ ================================================ FILE: scripts/install ================================================ #!/bin/bash set -e scripts/run-tests clj -T:build jar cp target/classes/META-INF/maven/com.cnuernber/ham-fisted/pom.xml . clj -X:install ================================================ FILE: scripts/koacha-test ================================================ #!/bin/bash scripts/compile clojure -M:dev:kaocha-test ================================================ FILE: scripts/lint ================================================ #!/bin/bash scripts/compile clojure -M:dev:test:clj-kondo --copy-configs --dependencies --parallel --lint "$(clojure -A:dev:test -Spath)" clojure -M:dev:test:clj-kondo --lint "src:test" --fail-level "error" ================================================ FILE: scripts/reformat ================================================ #!/bin/bash pushd src reformat popd pushd test reformat popd ================================================ FILE: scripts/run-tests ================================================ #!/bin/bash scripts/compile clojure -M:dev:test ================================================ FILE: src/ham_fisted/alists.clj ================================================ (ns ham-fisted.alists "Generic primitive array backed array-lists. The pure clojure implementations are a bit slower than the java ones but *far* less code so these are used for the less-frequently-used primive datatypes - byte, short, char, and float." (:require [ham-fisted.iterator :as iterator] [ham-fisted.protocols :as protocols] [ham-fisted.defprotocol :refer [extend extend-type extend-protocol]]) (:import [ham_fisted ArrayLists ArrayLists$ILongArrayList ArrayLists$IDoubleArrayList Transformables ArrayHelpers Casts IMutList IFnDef$OLO IFnDef$ODO ArrayLists$ObjectArrayList ArrayLists$ObjectArraySubList ArrayLists$ByteArraySubList ArrayLists$ShortArraySubList ArrayLists$CharArraySubList ArrayLists$IntArraySubList ArrayLists$IntArrayList ArrayLists$LongArraySubList ArrayLists$LongArrayList ArrayLists$FloatArraySubList ArrayLists$BooleanArraySubList ArrayLists$DoubleArraySubList ArrayLists$DoubleArrayList ArrayLists$IArrayList ChunkedList Reductions] [clojure.lang IPersistentMap IReduceInit RT] [java.util Arrays RandomAccess List]) (:refer-clojure :exclude [extend extend-type extend-protocol])) (set! *warn-on-reflection* true) (set! *unchecked-math* true) (defmacro implement-tostring-print [tname] (require '[ham-fisted.print]) `(ham-fisted.print/implement-tostring-print ~tname)) (defn- add-long-reduce [list c] (Reductions/serialReduction (reify IFnDef$OLO (invokePrim [this acc v] (.addLong ^IMutList acc v) acc)) list c)) (defn- add-double-reduce [list c] (Reductions/serialReduction (reify IFnDef$ODO (invokePrim [this acc v] (.addDouble ^IMutList acc v) acc)) list c)) (defmacro make-prim-array-list [lname ary-tag iface getname setname addname set-cast-fn get-cast-fn obj-cast-fn add-all-reduce] `(deftype ~lname [~(with-meta 'data {:unsynchronized-mutable true :tag ary-tag}) ~(with-meta 'n-elems {:unsynchronized-mutable true :tag 'long}) ~(with-meta 'm {:tag 'IPersistentMap})] ~'Object (hashCode [this#] (.hasheq this#)) (equals [this# other#] (.equiv this# other#)) (toString [this#] (Transformables/sequenceToString this#)) ~iface (meta [this#] ~'m) (withMeta [this# newm#] (with-meta (.subList this# 0 ~'n-elems) newm#)) (cloneList [this#] (~(symbol (str (name lname) ".")) (.copyOf this# ~'n-elems) ~'n-elems ~'m)) (clone [this#] (.cloneList this#)) (clear [this#] (set! ~'n-elems 0)) (setSize [this# sz#] (set! ~'n-elems (unchecked-long sz#))) (size [this#] (unchecked-int ~'n-elems)) (~getname [this# idx#] (~get-cast-fn (aget ~'data (ArrayLists/checkIndex idx# ~'n-elems)))) (get [this# idx#] (aget ~'data (ArrayLists/checkIndex idx# ~'n-elems))) (~setname [this# idx# v#] (ArrayHelpers/aset ~'data (ArrayLists/checkIndex idx# ~'n-elems) (~set-cast-fn v#))) (set [this# idx# val#] (let [idx# (ArrayLists/checkIndex idx# ~'n-elems) rv# (aget ~'data idx#)] (ArrayHelpers/aset ~'data idx# (~set-cast-fn val#)) rv#)) (subList [this# sidx# eidx#] (ChunkedList/sublistCheck sidx# eidx# ~'n-elems) (ArrayLists/toList ~'data sidx# eidx# ~'m)) (ensureCapacity [this# newlen#] (when (> newlen# (alength ~'data)) (set! ~'data (.copyOf this# (ArrayLists/newArrayLen newlen#)))) ~'data) (~addname [this# v#] (let [curlen# ~'n-elems newlen# (unchecked-inc ~'n-elems) ~(with-meta 'b {:tag ary-tag}) (.ensureCapacity this# newlen#)] (ArrayHelpers/aset ~'b curlen# (~set-cast-fn v#)) (set! ~'n-elems newlen#))) (add [this# idx# obj#] (ArrayLists/checkIndex idx# ~'n-elems) (if (== idx# ~'n-elems) (.add this# obj#) (let [bval# (~set-cast-fn (~obj-cast-fn obj#)) curlen# ~'n-elems newlen# (unchecked-inc curlen#) ~(with-meta 'd {:tag ary-tag}) (.ensureCapacity this# newlen#)] (System/arraycopy ~'d idx# ~'d (unchecked-inc idx#) (- curlen# idx#)) (ArrayHelpers/aset ~'d idx# bval#) (set! ~'n-elems newlen#)))) (addAllReducible [this# c#] (let [sz# (.size this#)] (if (instance? RandomAccess c#) (do (when-not (== 0 (.size ^List c#)) (let [~(with-meta 'c {:tag 'List}) c# curlen# ~'n-elems newlen# (+ curlen# (.size ~'c))] (.ensureCapacity this# newlen#) (set! ~'n-elems newlen#) (.fillRangeReducible this# curlen# ~'c)))) (~add-all-reduce this# c#)) (not (== sz# ~'n-elems)))) (fillRangeReducible [this# sidx# c#] (.fillRangeReducible (.subList this# 0 (.size this#)) sidx# c#)) (removeRange [this# sidx# eidx#] (ArrayLists/checkIndexRange ~'n-elems (long sidx#) (long eidx#)) (System/arraycopy ~'data sidx# ~'data eidx# (- ~'n-elems eidx#)) (set! ~'n-elems (- ~'n-elems (- eidx# sidx#)))) (sort [~'this c#] (.sort ~(with-meta '(.subList this 0 n-elems) {:tag 'IMutList}) c#)) (sortIndirect [~'this c#] (.sortIndirect ~(with-meta '(.subList this 0 n-elems) {:tag 'IMutList}) c#)) (shuffle [~'this r#] (.shuffle ~(with-meta '(.subList this 0 n-elems) {:tag 'IMutList}) r#)) (reduce [this# rfn# init#] (reduce rfn# init# (.subList this# 0 ~'n-elems))) (binarySearch [~'this v# c#] (.binarySearch ~(with-meta '(.subList this 0 n-elems) {:tag 'IMutList}) v# c#)) (move [this# sidx# eidx# count#] (ArrayLists/checkIndexRange ~'n-elems (long eidx#) (+ (long eidx#) count#)) (System/arraycopy ~'data sidx# ~'data eidx# count#)) (fill [this# sidx# eidx# v#] (ArrayLists/checkIndexRange ~'n-elems (long sidx#) (long eidx#)) (Arrays/fill ~'data sidx# eidx# (~set-cast-fn (~obj-cast-fn v#)))) (copyOfRange [this# sidx# eidx#] (Arrays/copyOfRange ~'data sidx# eidx#)) (copyOf [this# len#] (Arrays/copyOf ~'data len#)) (getArraySection [this#] (ham_fisted.ArraySection. ~'data 0 ~'n-elems)))) (make-prim-array-list ByteArrayList bytes ArrayLists$ILongArrayList getLong setLong addLong RT/byteCast unchecked-long Casts/longCast add-long-reduce) (make-prim-array-list ShortArrayList shorts ArrayLists$ILongArrayList getLong setLong addLong RT/shortCast unchecked-long Casts/longCast add-long-reduce) (make-prim-array-list CharArrayList chars ArrayLists$ILongArrayList getLong setLong addLong Casts/charCast Casts/charLongCast Casts/charLongCast add-long-reduce) (make-prim-array-list FloatArrayList floats ArrayLists$IDoubleArrayList getDouble setDouble addDouble float unchecked-double Casts/doubleCast add-double-reduce) (deftype BooleanArrayList [^{:unsynchronized-mutable true :tag booleans} data ^{:unsynchronized-mutable true :tag long} n-elems ^IPersistentMap m] Object (hashCode [this] (.hasheq this)) (equals [this other] (.equiv this other)) (toString [this] (Transformables/sequenceToString this)) ArrayLists$IArrayList (cloneList [this] (BooleanArrayList. (.copyOf this n-elems) n-elems m)) (meta [this] m) (withMeta [this newm] (with-meta (.subList this 0 n-elems) newm)) (setSize [this sz] (set! n-elems (unchecked-long sz))) (size [this] (unchecked-int n-elems)) (clear [this] (set! n-elems 0)) (get [this idx] (aget data (ArrayLists/checkIndex idx n-elems))) (set [this idx v] (let [retval (.get this idx)] (ArrayHelpers/aset data (ArrayLists/checkIndex idx n-elems) (Casts/booleanCast v)) retval)) (subList [this sidx eidx] (ChunkedList/sublistCheck sidx eidx n-elems) (ArrayLists/toList data sidx eidx m)) (ensureCapacity [this newlen] (when (> newlen (alength data)) (set! data ^booleans (.copyOf this (ArrayLists/newArrayLen newlen)))) data) (add [this v] (let [curlen n-elems newlen (unchecked-inc n-elems) ^booleans b (.ensureCapacity this newlen)] (ArrayHelpers/aset b curlen (Casts/booleanCast v)) (set! n-elems newlen) true)) (add [this idx obj] (if (== idx n-elems) (.add this obj) (do (ArrayLists/checkIndex idx n-elems) (let [bval (Casts/booleanCast obj) curlen n-elems newlen (unchecked-inc curlen) ^booleans d (.ensureCapacity this newlen)] (System/arraycopy d idx d (unchecked-inc idx) (- curlen idx)) (ArrayHelpers/aset d idx bval) (set! n-elems newlen))))) (addAllReducible [this c] (let [sz (.size this)] (if (instance? RandomAccess c) (let [^List c c curlen n-elems newlen (+ curlen (.size c))] (.ensureCapacity this newlen) (set! n-elems newlen) (.fillRangeReducible this curlen c)) (Reductions/serialReduction (fn [acc v] (.add ^List acc v) acc) this c)) (not (== sz n-elems)))) (fillRangeReducible [this sidx c] (.fillRangeReducible (.subList this 0 (.size this)) sidx c)) (removeRange [this sidx eidx] (ArrayLists/checkIndexRange n-elems sidx eidx) (System/arraycopy data sidx data eidx (- n-elems eidx)) (set! n-elems (- n-elems (- eidx sidx)))) (sort [this c] (.sort (.subList this 0 n-elems) c)) (sortIndirect [this c] (.sortIndirect ^IMutList (.subList this 0 n-elems) c)) (shuffle [this r] (.shuffle ^IMutList (.subList this 0 n-elems) r)) (binarySearch [this v c] (.binarySearch ^IMutList (.subList this 0 n-elems) v c)) (move [this sidx eidx ct] (ArrayLists/checkIndexRange n-elems (long eidx) (+ (long eidx) ct)) (System/arraycopy data sidx data eidx ct)) (fill [this sidx eidx v] (ArrayLists/checkIndexRange n-elems (long sidx) (long eidx)) (Arrays/fill data sidx eidx (Casts/booleanCast v))) (copyOfRange [this sidx eidx] (Arrays/copyOfRange data sidx eidx)) (copyOf [this len] (Arrays/copyOf data len)) (getArraySection [this] (ham_fisted.ArraySection. data 0 n-elems))) (implement-tostring-print ByteArrayList) (implement-tostring-print ShortArrayList) (implement-tostring-print CharArrayList) (implement-tostring-print FloatArrayList) (implement-tostring-print BooleanArrayList) (defn- ladd [^IMutList m v] (.add m v) m) (defn- lmerge [^IMutList l v] (.addAllReducible l v) l) #_(extend-protocol protocols/Reducer ArrayLists$ObjectArrayList (->init-val-fn [item] #(ArrayLists$ObjectArrayList.)) (->rfn [item] ladd) ArrayLists$IntArrayList (->init-val-fn [item] #(ArrayLists$IntArrayList.)) (->rfn [item] ladd) ArrayLists$LongArrayList (->init-val-fn [item] #(ArrayLists$LongArrayList.)) (->rfn [item] ladd) ArrayLists$DoubleArrayList (->init-val-fn [item] #(ArrayLists$DoubleArrayList.)) (->rfn [item] ladd)) #_(extend-protocol protocols/ParallelReducer ArrayLists$ObjectArrayList (->merge-fn [l] lmerge) ArrayLists$IntArrayList (->merge-fn [l] lmerge) ArrayLists$LongArrayList (->merge-fn [l] lmerge) ArrayLists$DoubleArrayList (->merge-fn [l] lmerge)) (defmacro extend-array-types [] `(do ~@(->> {'(Class/forName "[Z") ['bytes 'BooleanArrayList.] '(Class/forName "[B") ['bytes 'ByteArrayList.] '(Class/forName "[S") ['shorts 'ShortArrayList.] '(Class/forName "[C") ['chars 'CharArrayList.] '(Class/forName "[I") ['ints 'ArrayLists$IntArrayList.] '(Class/forName "[J") ['longs 'ArrayLists$LongArrayList.] '(Class/forName "[F") ['floats 'FloatArrayList.] '(Class/forName "[D") ['doubles 'ArrayLists$DoubleArrayList.] '(Class/forName "[Ljava.lang.Object;") ['objects 'ArrayLists$ObjectArrayList.]} (map (fn [[cls-type [hint growable-cons]]] `(extend ~cls-type protocols/WrapArray {:wrap-array (fn [~(with-meta (symbol "ary") {:tag hint})] (ArrayLists/toList ~'ary)) :wrap-array-growable (fn [~(with-meta (symbol "ary") {:tag hint}) ~(with-meta (symbol "ptr") {:tag 'long})] (~growable-cons ~'ary ~'ptr nil))})))))) (extend-array-types) (def ^:private obj-ary-cls (Class/forName "[Ljava.lang.Object;")) (defn wrap-array "Wrap an array with an implementation of IMutList" ^IMutList [ary] (if (instance? obj-ary-cls ary) (ArrayLists/toList ^objects ary) (protocols/wrap-array ary))) (defn wrap-array-growable "Wrap an array with an implementation of IMutList that supports add and addAllReducible. 'ptr is the numeric put ptr, defaults to the array length. Pass in zero for a preallocated but empty growable wrapper." (^IMutList [ary ptr] (if (instance? obj-ary-cls ary) (ArrayLists$ObjectArrayList. ary ptr nil) (protocols/wrap-array-growable ary ptr))) (^IMutList [ary] (wrap-array-growable ary (java.lang.reflect.Array/getLength ary)))) (def array-list-types [:int8 :int16 :int32 :int64 :float32 :float64 :char :boolean :object]) (defn growable-array-list (^IMutList [dtype] (case dtype :int8 (ByteArrayList. (byte-array 8) 0 {}) :int16 (ShortArrayList. (short-array 8) 0 {}) :int32 (ArrayLists$IntArrayList. (int-array 8) 0 {}) :int64 (ArrayLists$LongArrayList. (long-array 4) 0 {}) :float32 (FloatArrayList. (float-array 8) 0 {}) :float64 (ArrayLists$DoubleArrayList. (double-array 4) 0 {}) :char (CharArrayList. (char-array 8) 0 {}) :boolean (BooleanArrayList. (boolean-array 8) 0 {}) :object (ArrayLists$ObjectArrayList. (object-array 8) 0 {}))) (^IMutList [dtype data] (let [rv (growable-array-list dtype)] (.addAllReducible rv data) rv))) ================================================ FILE: src/ham_fisted/api.clj ================================================ (ns ham-fisted.api "Fast mutable and immutable associative data structures based on bitmap trie hashmaps. Mutable pathways implement the `java.util.Map` or `Set` interfaces including in-place update features such as compute or computeIfPresent. Mutable maps or sets can be turned into their immutable counterparts via the Clojure `persistent!` call. This allows working in a mutable space for convenience and performance then switching to an immutable pathway when necessary. Note: after `persistent!` one should never backdoor mutate map or set again as this will break the contract of immutability. Immutable data structures also support conversion to transient via `transient`. Map keysets (`.keySet`) are full `PersistentHashSet`s of keys. Maps and sets support metadata but setting the metadata on mutable objects returns a new mutable object that shares the backing store leading to possible issues. Metadata is transferred to the persistent versions of the mutable/transient objects upon `persistent!`. Very fast versions of union, difference and intersection are provided for maps and sets with the map version of union and difference requiring an extra argument, a `java.util.BiFunction` or an `IFn` taking 2 arguments to merge the left and right sides into the final map. These implementations of union, difference, and intersection are the fastest implementation of these operations we know of on the JVM. Additionally a fast value update pathway is provided, enabling quickly updating all the values in a given map. Additionally, a new map primitive - [[mapmap]] - allows transforming a given map into a new map quickly by mapping across all the entries. Unlike the standard Java objects, mutation-via-iterator is not supported." (:require [ham-fisted.iterator :as iterator] [ham-fisted.function :refer [bi-function ->bi-function function bi-consumer obj->long] :as hamf-fn] [ham-fisted.lazy-noncaching :refer [map concat filter repeatedly] :as lznc] [ham-fisted.caffeine :as hamf-caffeine] [ham-fisted.lazy-caching :as lzc] [ham-fisted.alists :as alists] [ham-fisted.impl :as impl] [ham-fisted.reduce :refer [reduce-reducer preduce options->parallel-options preduce-reducer long-accumulator double-accumulator double-consumer-accumulator long-consumer-accumulator] :as hamf-rf] [ham-fisted.language :as hamf-language] [ham-fisted.protocols :as protocols] [ham-fisted.defprotocol :refer [extend extend-type extend-protocol]]) (:import [ham_fisted UnsharedHashMap UnsharedLongHashMap UnsharedHashSet PersistentHashSet PersistentHashMap PersistentLongHashMap ArrayLists$ArrayOwner MergeIterator HashProvider MapSetOps SetOps ObjArray UpdateValues MutList ImmutList StringCollection ArrayImmutList ArrayLists ImmutSort IMutList Ranges$LongRange ArrayHelpers Ranges$DoubleRange IFnDef Transformables$MapIterable Transformables$FilterIterable Transformables$CatIterable Transformables$MapList Transformables$IMapable Transformables ReindexList ConstList ArrayLists$ObjectArrayList Transformables$SingleMapList ArrayLists$IntArrayList ArrayLists$LongArrayList ArrayLists$DoubleArrayList ReverseList TypedList DoubleMutList LongMutList Consumers Sum Sum$SimpleSum Casts Reducible IndexedDoubleConsumer IndexedLongConsumer IndexedConsumer ITypedReduce ParallelOptions Reductions IFnDef$LO IFnDef$LL IFnDef$DO IFnDef$DD IFnDef$DDD IFnDef$LLL ParallelOptions$CatParallelism IFnDef$OO IFnDef$OOO IFnDef$ODO IFnDef$OLO IFnDef$OD IFnDef$OL IFnDef$LD IFnDef$DL IFnDef$OLOO IFnDef$OLDO IFnDef$OLLO IFnDef$LongPredicate IFnDef$DoublePredicate IFnDef$Predicate Consumers$IncConsumer Reductions$IndexedDoubleAccum Reductions$IndexedLongAccum Reductions$IndexedAccum MutableMap IAMapEntry MapForward TypedNth TreeList MutTreeList] [ham_fisted.alists ByteArrayList ShortArrayList CharArrayList FloatArrayList BooleanArrayList] [clojure.lang ITransientAssociative2 ITransientCollection Indexed IEditableCollection RT IPersistentMap Associative Util IFn ArraySeq Reversible IReduce IReduceInit IFn$DD IFn$DL IFn$DO IFn$LD IFn$LL IFn$LO IFn$OD IFn$OL IFn$OLO IFn$ODO IObj Util IReduceInit Seqable IteratorSeq ITransientMap Counted Box] [java.util Map Map$Entry List RandomAccess Set Collection ArrayList Arrays Comparator Random Collections Iterator PriorityQueue LinkedHashMap LongSummaryStatistics DoubleSummaryStatistics] [java.lang.reflect Array] [java.util.function Function BiFunction BiConsumer Consumer DoubleBinaryOperator LongBinaryOperator LongFunction IntFunction DoubleConsumer DoublePredicate DoubleUnaryOperator LongPredicate LongUnaryOperator LongConsumer Predicate UnaryOperator] [java.util.concurrent ForkJoinPool ExecutorService Callable Future ConcurrentHashMap ForkJoinTask ArrayBlockingQueue] [it.unimi.dsi.fastutil.ints IntComparator IntArrays] [it.unimi.dsi.fastutil.longs LongComparator] [it.unimi.dsi.fastutil.floats FloatComparator] [it.unimi.dsi.fastutil.doubles DoubleComparator DoubleArrays] [it.unimi.dsi.fastutil.objects ObjectArrays] [com.github.benmanes.caffeine.cache Caffeine LoadingCache CacheLoader Cache RemovalCause] [com.github.benmanes.caffeine.cache.stats CacheStats] [java.time Duration] [java.util.logging Logger] [java.util.stream IntStream DoubleStream]) (:refer-clojure :exclude [assoc! conj! frequencies merge merge-with memoize cond into hash-map not group-by subvec group-by mapv vec vector object-array sort int-array long-array double-array float-array range map concat filter filterv first last pmap take take-last drop drop-last sort-by repeat repeatedly shuffle into-array empty? reverse byte-array short-array char-array boolean-array keys vals persistent! rest transient update-vals re-matches complement count extend extend-type extend-protocol constantly])) (comment (require '[clj-java-decompiler.core :refer [disassemble]]) ) (set! *warn-on-reflection* true) (set! *unchecked-math* :warn-on-boxed) (declare assoc! conj! vec mapv vector object-array range first take drop into-array shuffle object-array-list int-array-list long-array-list double-array-list int-array argsort byte-array short-array char-array boolean-array repeat persistent! rest immut-map keys vals group-by-reduce reindex group-by-consumer merge constant-count mutable-map? transient update-vals apply-concat) (defn constantly [x] (hamf-language/constantly x)) (defn not "Returns boolean opposite of passed in value" {:inline (fn [x] `(Transformables/not ~x)) :inline-arities #{1}} [a] (Transformables/not a)) (defn complement "Like clojure core complement but avoids var lookup on 'not'" [f] (lznc/complement f)) (defn count "hamf protocol extensible count" ^long [m] (protocols/count m)) (defn ->collection "Ensure item is an implementation of java.util.Collection." ^Collection [item] (lznc/->collection item)) (defn ->random-access "Ensure item is derived from java.util.List and java.util.RandomAccess and thus supports constant time random addressing." ^List [item] (lznc/->random-access item)) (defn ->reducible "Ensure item either implements IReduceInit or java.util.Collection. For arrays this will return an object that has a much more efficient reduction pathway than the base Clojure reducer." [item] (lznc/->reducible item)) (defn- check-deprecated-provider [options] (when (:hash-provider options) (.warning (Logger/getGlobal) "Hash providers have been deprecated"))) (def ^{:tag PersistentHashMap :doc "Constant persistent empty map"} empty-map PersistentHashMap/EMPTY) (def ^{:tag PersistentHashSet :doc "Constant persistent empty set"} empty-set PersistentHashSet/EMPTY) (def ^{:tag ArrayImmutList :doc "Constant persistent empty vec"} empty-vec ArrayImmutList/EMPTY) (defn- empty-map? [m] (or (nil? m) (and (instance? Map m) (== 0 (.size ^Map m))))) (def ^:private empty-objs (clojure.core/object-array 0)) (def ^{:doc "As quickly as possible, produce an object array from these inputs. Very fast for arities <= 16." :arglists '([] [v0] [v0 v1] [v0 v1 v2] [v0 v1 v2 v3] [v0 v1 v2 v3 v4] [v0 v1 v2 v3 v4 v5] [v0 v1 v2 v3 v4 v5 v6] [v0 v1 v2 v3 v4 v5 v6 v7] [v0 v1 v2 v3 v4 v5 v6 v7 v8] [v0 v1 v2 v3 v4 v5 v6 v7 v8 v9] [v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10] [v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11] [v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12] [v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13] [v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14] [v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15])} obj-ary hamf-language/obj-ary) (defmacro cond "Similar to `core/cond` except it supports true else clauses as opposed to always returning nil on else. The last clause of an odd number of clauses will be used as the else branch similar to case. Additional if the last predicate is the constant 'true' or ':else' it will be used as the else clause. If no else is provided then returns nil same as `core/cond`. Important! - in order to get correct type hinting on both branches of the if they both must return an explicit long or double: * Sill Boxed ```clojure (defn memory-size ^long [a] (cond (instance? Long a) 24 :else (.length ^String a) ;;string.length returns integer not long )) (defn memory-size ^long [a] (cond (instance? Long a) 24 :else (alength ^longs a) )) ``` * Correctly Unboxed ```clojure (defn memory-size ^long [a] (cond (instance? Long a) 24 :else (long (.length ^String a)) )) (defn memory-size ^long [a] (cond (instance? Long a) 24 :else (long (alength ^longs a)) )) ```" [& clauses] `(lznc/cond ~@clauses)) (defn into "Like clojure.core/into, but also designed to handle editable collections, transients, and base java.util.Map, List and Set containers." ([container data] (cond (instance? IEditableCollection container) (-> (reduce conj! (transient container) data) (persistent!)) (instance? ITransientCollection container) (reduce conj! container data) (instance? Map container) (if (instance? Map data) (do (.putAll ^Map container ^Map data) container) (reduce conj! container data)) (instance? Collection container) (cond (instance? Collection data) (do (.addAll ^Collection container ^Collection data) container) (instance? CharSequence data) (do (.addAll ^Collection container (StringCollection. data)) container) :else (reduce conj! container data)) :else (throw (Exception. (str "Unable to ascertain container type: " (type container)))))) ([container xform data] (into container (eduction xform data)))) (defonce ^:private obj-ary-cls (type (clojure.core/object-array 0))) (defn mut-map-rf ([cons-fn] (mut-map-rf cons-fn nil)) ([cons-fn finalize-fn] (fn ([] (cons-fn)) ([acc] (if finalize-fn (finalize-fn acc) acc)) ([^Map m d] (cond (instance? Map$Entry d) (.put m (.getKey ^Map$Entry d) (.getValue ^Map$Entry d)) (instance? Indexed d) (.put m (.nth ^Indexed d 0) (.nth ^Indexed d 1)) :else (throw (Exception. (str "Unrecognized map input: " (type d))))) m)))) (defn transient-map-rf ([cons-fn] (transient-map-rf cons-fn nil)) ([cons-fn finalize-fn] (fn ([] (cons-fn)) ([acc] (if finalize-fn (finalize-fn acc) acc)) ([^ITransientCollection m d] (.conj m d))))) (defn- tduce [xform rf data] (if xform (transduce xform rf data) (rf (reduce rf (rf) data)))) (defn mut-hashtable-map "Create a mutable implementation of java.util.Map. This object efficiently implements ITransient map so you can use assoc! and persistent! on it but you can additionally use the various members of the java.util.Map interface such as put, compute, computeIfAbsent, replaceAll and merge. If data is an object array it is treated as a flat key-value list which is distinctly different than how conj! treats object arrays. You have been warned." (^UnsharedHashMap [] (UnsharedHashMap. nil)) (^UnsharedHashMap [data] (mut-hashtable-map nil nil data)) (^UnsharedHashMap [xform data] (mut-hashtable-map xform nil data)) (^UnsharedHashMap [xform options data] (cond (number? data) (UnsharedHashMap. nil (int data)) (nil? xform) (cond (instance? obj-ary-cls data) (UnsharedHashMap/create data) (instance? Map data) (doto (UnsharedHashMap. nil (.size ^Map data)) (.putAll data)) :else (into (UnsharedHashMap. nil) data)) :else (into (UnsharedHashMap. nil) xform data)))) (defn mut-long-hashtable-map "Create a mutable implementation of java.util.Map. This object efficiently implements ITransient map so you can use assoc! and persistent! on it but you can additionally use the various members of the java.util.Map interface such as put, compute, computeIfAbsent, replaceAll and merge. If data is an object array it is treated as a flat key-value list which is distinctly different than how conj! treats object arrays. You have been warned." (^UnsharedLongHashMap [] (UnsharedLongHashMap. nil)) (^UnsharedLongHashMap [data] (mut-long-hashtable-map nil nil data)) (^UnsharedLongHashMap [xform data] (mut-long-hashtable-map xform nil data)) (^UnsharedLongHashMap [xform options data] (cond (number? data) (UnsharedLongHashMap. nil (int data)) (nil? xform) (cond (instance? obj-ary-cls data) (UnsharedLongHashMap/create data) (instance? Map data) (doto (UnsharedLongHashMap. nil (.size ^Map data)) (.putAll data)) :else (into (UnsharedLongHashMap. nil) data)) :else (into (UnsharedLongHashMap. nil) xform data)))) (defn mut-map "Create a mutable implementation of java.util.Map. This object efficiently implements ITransient map so you can use assoc! and persistent! on it but you can additionally use the various members of the java.util.Map interface such as put, compute, computeIfAbsent, replaceAll and merge. If data is an object array it is treated as a flat key-value list which is distinctly different than how conj! treats object arrays. You have been warned." (^UnsharedHashMap [] (mut-hashtable-map)) (^UnsharedHashMap [data] (mut-hashtable-map data)) (^UnsharedHashMap [xform data] (mut-hashtable-map xform data)) (^UnsharedHashMap [xform options data] (mut-hashtable-map xform options data))) (defn mut-long-map "Create a mutable implementation of java.util.Map specialized to long keys. This object efficiently implements ITransient map so you can use assoc! and persistent! on it but you can additionally use the various members of the java.util.Map interface such as put, compute, computeIfAbsent, replaceAll and merge. Attempting to store any non-numeric value will result in an exception. If data is an object array it is treated as a flat key-value list which is distinctly different than how conj! treats object arrays. You have been warned." (^UnsharedLongHashMap [] (mut-long-hashtable-map)) (^UnsharedLongHashMap [data] (mut-long-hashtable-map data)) (^UnsharedLongHashMap [xform data] (mut-long-hashtable-map xform data)) (^UnsharedLongHashMap [xform options data] (mut-long-hashtable-map xform options data))) (defn constant-countable? "Return true if data has a constant time count." [data] (lznc/constant-countable? data)) (defn constant-count "Constant time count. Returns nil if input doesn't have a constant time count." [data] (lznc/constant-count data)) (defn immut-map "Create an immutable map. This object supports conversion to a transient map via Clojure's `transient` function. Duplicate keys are treated as if by assoc. If data is an object array it is treated as a flat key-value list which is distinctly different than how conj! treats object arrays. You have been warned. If you know you will have consistently more key/val pairs than 8 you should just use `(persistent! (mut-map data))` as that avoids the transition from an arraymap to a persistent hashmap. Examples: ```clojure ham-fisted.api> (immut-map (obj-ary :a 1 :b 2 :c 3 :d 4)) {:a 1, :b 2, :c 3, :d 4} ham-fisted.api> (type *1) ham_fisted.PersistentArrayMap ham-fisted.api> (immut-map (obj-ary :a 1 :b 2 :c 3 :d 4 :e 5)) {:d 4, :b 2, :c 3, :a 1, :e 5} ham-fisted.api> (type *1) ham_fisted.PersistentHashMap ham-fisted.api> (immut-map [[:a 1][:b 2][:c 3][:d 4][:e 5]]) {:d 4, :b 2, :c 3, :a 1, :e 5} ham-fisted.api> (type *1) ham_fisted.PersistentHashMap ```" (^PersistentHashMap [] empty-map) (^PersistentHashMap [data] (immut-map nil data)) (^PersistentHashMap [options data] (-> (mut-map options data) (persistent!)))) (defn hash-map "Drop-in replacement to Clojure's hash-map function." ([] empty-map) ([a b] (persistent! (doto (mut-hashtable-map) (.put a b)))) ([a b c d] (persistent! (doto (mut-hashtable-map) (.put a b) (.put c d)))) ([a b c d e f] (persistent! (doto (mut-hashtable-map) (.put a b) (.put c d) (.put e f)))) ([a b c d e f g h] (persistent! (doto (mut-hashtable-map) (.put a b) (.put c d) (.put e f) (.put g h)))) ([a b c d e f g h i j] (persistent! (doto (mut-hashtable-map) (.put a b) (.put c d) (.put e f) (.put g h) (.put i j)))) ([a b c d e f g h i j k l] (persistent! (doto (mut-hashtable-map) (.put a b) (.put c d) (.put e f) (.put g h) (.put i j) (.put k l)))) ([a b c d e f g h i j k l m n] (persistent! (doto (mut-hashtable-map) (.put a b) (.put c d) (.put e f) (.put g h) (.put i j) (.put k l) (.put m n)))) ([a b c d e f g h i j k l m n o p] (persistent! (doto (mut-hashtable-map) (.put a b) (.put c d) (.put e f) (.put g h) (.put i j) (.put k l) (.put m n) (.put o p)))) ([a b c d e f g h i j k l m n o p & args] (persistent! (UnsharedHashMap/create (ObjArray/createv a b c d e f g h i j k l m n o p (object-array args)))))) (defn java-hashmap "Create a java.util.HashMap. Duplicate keys are treated as if map was created by assoc." (^java.util.HashMap [] (java.util.HashMap.)) (^java.util.HashMap [data] (java-hashmap nil nil data)) (^java.util.HashMap [xform data] (java-hashmap xform nil data)) (^java.util.HashMap [xform options data] (cond (number? data) (java.util.HashMap. (int data)) (and (nil? xform) (instance? Map data)) (if (instance? java.util.HashMap data) (.clone ^java.util.HashMap data) (java.util.HashMap. ^Map data)) :else (tduce xform (mut-map-rf #(java.util.HashMap. (long (get options :init-size 0)))) data)))) (defn java-linked-hashmap "Linked hash maps perform identically or very nearly so to java.util.HashMaps but they retain the order of insertion and modification." (^java.util.LinkedHashMap [] (java.util.LinkedHashMap.)) (^java.util.LinkedHashMap [data] (cond (instance? java.util.LinkedHashMap data) data (number? data) (java.util.LinkedHashMap. (int data)) (instance? java.util.LinkedHashMap data) (.clone ^java.util.LinkedHashMap data) (instance? Map data) (java.util.LinkedHashMap. ^Map data) :else (tduce nil (mut-map-rf #(java.util.LinkedHashMap.)) data)))) (defn linked-hashmap "Linked hash map using clojure's equiv pathways. At this time the node link order reflects insertion order. Modification and access do not affect the node link order." (^ham_fisted.LinkedHashMap [] (ham_fisted.LinkedHashMap.)) (^ham_fisted.LinkedHashMap [data] (let [rv (ham_fisted.LinkedHashMap.)] (if (instance? Map data) (do (.putAll rv data) rv) (tduce nil (mut-map-rf (constantly rv)) data))))) (defn java-concurrent-hashmap "Create a java concurrent hashmap which is still the fastest possible way to solve a few concurrent problems." (^ConcurrentHashMap [] (ConcurrentHashMap.)) (^ConcurrentHashMap [data] (cond (instance? ConcurrentHashMap data) data (number? data) (ConcurrentHashMap. (int data)) (instance? Map data) (ConcurrentHashMap. ^Map data) :else (tduce nil (mut-map-rf #(ConcurrentHashMap.)) data)))) (defn mut-set "Create a mutable hashset based on the hashtable. You can create a persistent hashset via the clojure `persistent!` call. Options: * `:hash-provider` - An implementation of `BitmapTrieCommon$HashProvider`. Defaults to the [[default-hash-provider]]." (^UnsharedHashSet [] (UnsharedHashSet. nil)) (^UnsharedHashSet [data] (into (UnsharedHashSet. nil) data)) (^UnsharedHashSet [options data] (into (UnsharedHashSet. nil) data))) (defn immut-set "Create an immutable hashset based on a hash table. This object supports conversion to transients via `transient`. Options: * `:hash-provider` - An implementation of `BitmapTrieCommon$HashProvider`. Defaults to the [[default-hash-provider]]." (^PersistentHashSet [] empty-set) (^PersistentHashSet [data] (into empty-set data)) (^PersistentHashSet [options data] (into empty-set data))) (defn java-hashset "Create a java hashset which is still the fastest possible way to solve a few problems." (^java.util.HashSet [] (java.util.HashSet.)) (^java.util.HashSet [data] (into (java.util.HashSet.) data))) (defn mut-list "Create a mutable java list that is in-place convertible to a persistent list" (^MutTreeList [] (MutTreeList.)) (^MutTreeList [data] (cond (nil? data) (MutTreeList.) (instance? obj-ary-cls data) (MutTreeList/create false nil ^objects data) (or (instance? IReduceInit data) (instance? Collection data)) (doto (MutTreeList.) (.addAllReducible data)) (string? data) (doto (MutTreeList.) (.addAll (StringCollection. data))) (.isArray (.getClass ^Object data)) (MutTreeList/create false nil ^objects data) :else (into (MutTreeList.) data)))) (defn immut-list "Create a persistent list. Object arrays will be treated as if this new object owns them." (^TreeList [] empty-vec) (^TreeList [data] (if (instance? obj-ary-cls data) (ArrayImmutList/create true nil data) (persistent! (mut-list data))))) (defn array-list "Create an implementation of java.util.ArrayList." (^ArrayList [data] (if (instance? Collection data) (doto (ArrayList.) (.addAll (->collection data))) (into (ArrayList.) data))) (^ArrayList [] (ArrayList.))) (defn assoc! "assoc! that works on transient collections, implementations of java.util.Map and RandomAccess java.util.List implementations. Be sure to keep track of return value as some implementations return a different return value than the first argument." [obj k v] (cond (instance? ITransientAssociative2 obj) (.assoc ^ITransientAssociative2 obj k v) (instance? Map obj) (do (.put ^Map obj k v) obj) (instance? RandomAccess obj) (do (.set ^List obj (int k) v) obj) :else (throw (Exception. "Item cannot be assoc!'d")))) (defn conj! "conj! that works on transient collections, implementations of java.util.Set and RandomAccess java.util.List implementations. Be sure to keep track of return value as some implementations return a different return value than the first argument." [obj val] (cond (instance? ITransientCollection obj) (.conj ^ITransientCollection obj val) (instance? Collection obj) (do (.add ^Collection obj val) obj) (instance? Map obj) (do (cond (instance? Indexed val) (let [^Indexed val val] (.put ^Map obj (.nth val 0) (.nth val 1)) obj) (instance? Map$Entry val) (let [^Map$Entry val val] (.put ^Map obj (.getKey val) (.getValue val)) obj) :else (throw (RuntimeException. (str "Cannot conj " val " to a map."))))) :else (throw (Exception. "Item cannot be conj!'d")))) (defn add-all! "Add all items from l2 to l1. l1 is expected to be a java.util.List implementation. Returns l1." [l1 l2] (if (instance? IMutList l1) (.addAllReducible ^IMutList l1 l2) (.addAll (->collection l1) l2)) l1) (defn clear! "Mutably clear a map, set, list or implementation of java.util.Collection." [map-or-coll] (cond (instance? Map map-or-coll) (.clear ^Map map-or-coll) (instance? Collection map-or-coll) (.clear ^Collection map-or-coll) (instance? LoadingCache map-or-coll) (.invalidateAll ^LoadingCache map-or-coll) :else (throw (Exception. (str "Unrecognized type for clear!: " map-or-coll)))) map-or-coll) (defn ^:no-doc map-set? [item] (instance? MapSetOps item)) (defn ^:no-doc as-map-set ^MapSetOps [item] item) (defn- ->set ^Set [item] (cond (instance? Set item) item (instance? Map item) (.keySet ^Map item) :else (immut-set item))) (defn keys "Return the keys of a map. This version allows parallel reduction operations on the returned sequence." [m] (map key m)) (defn vals "Return the values of a map. This version allows parallel reduction operations on the returned sequence. Returned sequence is in same order as `(keys m)`." [m] (map val m)) (defmacro make-map-entry "Create a dynamic implementation of clojure's IMapEntry class." [k-code v-code] `(reify IAMapEntry (getKey [this#] ~k-code) (getValue [this#] ~v-code))) (defn map-union "Take the union of two maps returning a new map. bfn is a function that takes 2 arguments, map1-val and map2-val and returns a new value. Has fallback if map1 and map2 aren't backed by bitmap tries. * `bfn` - A function taking two arguments and returning one. `+` is a fine choice. * `map1` - the lhs of the union. * `map2` - the rhs of the union. Returns a persistent map if input is a persistent map else if map1 is a mutable map map1 is returned with overlapping entries merged. In this way you can pass in a normal java hashmap, a linked java hashmap, or a persistent map and get back a result that matches the input. If map1 and map2 are the same returns map1." [bfn map1 map2] (cond (nil? map1) map2 (nil? map2) map1 (identical? map1 map2) map1 :else (let [bfn (->bi-function bfn)] (if (map-set? map1) (.union (as-map-set map1) map2 bfn) (let [map1 (if (mutable-map? map1) map1 (mut-map map1))] (reduce (fn [acc kv] (.merge ^Map map1 (key kv) (val kv) bfn)) nil map2) map1))))) (defn map-union-java-hashmap "Take the union of two maps returning a new map. See documentation for [map-union]. Returns a java.util.HashMap." ^java.util.HashMap [bfn ^Map lhs ^Map rhs] (map-union bfn (java-hashmap lhs) rhs)) (def ^:no-doc rhs-wins (hamf-fn/bi-function l r r)) (defn ^:no-doc map-set-union-fallback [m m2] (-> (reduce (fn [^ITransientMap l e] (.conj l e)) (mut-hashtable-map m) m2) (persistent!))) (defn ^:no-doc set-union-fallback [m m2] (-> (reduce (fn [^UnsharedHashSet l e] (.conj l e)) (mut-set m) m2) (persistent!))) (defn mutable-map? [m] (or (instance? MutableMap m) (and (instance? Map m) (not (instance? IPersistentMap m)) (not (instance? ITransientMap m))))) (defn union "Union of two sets or two maps. When two maps are provided the right hand side wins in the case of an intersection - same as merge. Result is either a set or a map, depending on if s1 is a set or map." [s1 s2] (cond (nil? s1) s2 (nil? s2) s1 (identical? s1 s2) s1 (and (instance? Map s1) (instance? Map s2)) (cond (mutable-map? s1) (do (.putAll ^Map s1 s2) s1) (map-set? s1) (.union (as-map-set s1) ^Map s2 rhs-wins) :else (persistent! (reduce-kv (fn [acc k v] (assoc! acc k v)) (transient s1) s2))) (instance? SetOps s1) (.union ^SetOps s1 s2) :else (set-union-fallback s1 s2))) (defn union-reduce-maps "Do an efficient union reduction across many maps using bfn to update values. If the first map is mutable the union is done mutably into the first map and it is returned." ([bfn maps] (let [bfn (->bi-function bfn)] (-> (reduce #(map-union bfn %1 %2) maps) (persistent!))))) (defn ^:no-doc union-reduce-java-hashmap "Do an efficient union of many maps into a single java.util.HashMap." (^java.util.HashMap [bfn maps options] (let [maps (->reducible maps)] (if (nil? maps) nil (let [bfn (->bi-function bfn)] (reduce (fn [acc v] (map-union bfn acc v)) (java-hashmap (first maps)) (rest maps)))))) (^java.util.HashMap [bfn maps] (union-reduce-java-hashmap bfn maps nil))) (defn difference "Take the difference of two maps (or sets) returning a new map. Return value is a map1 (or set1) without the keys present in map2." [map1 map2] (cond (or (nil? map1) (nil? map2)) map1 (map-set? map1) (if (instance? Map map2) (.difference (as-map-set map1) ^Map map2) (.difference (as-map-set map1) (->set map2))) (instance? SetOps map1) (.difference ^SetOps map1 (->set map2)) (instance? Set map1) (let [map2 (->set map2)] (-> (reduce (fn [^Set acc v] (when-not (.contains map2 v) (.add acc v)) acc) (mut-set) map1) (persistent!))) (instance? Map map1) (let [map2 (->set map2)] (-> (reduce (fn [^Map acc kv] (when-not (.contains map2 (key kv)) (.put acc (key kv) (val kv))) acc) (mut-map) map1) (persistent!))))) (defn map-intersection "Intersect the keyspace of map1 and map2 returning a new map. Each value is the result of bfn applied to the map1-value and map2-value, respectively. See documentation for [[map-union]]. Clojure's `merge` functionality can be duplicate via: ```clojure (map-intersection (fn [lhs rhs] rhs) map1 map2) ```" [bfn map1 map2] (if (or (nil? map2) (nil? map2)) empty-map (let [bfn (->bi-function bfn)] (if (map-set? map1) (if (instance? Map map2) (.intersection (as-map-set map1) ^Map map2 bfn) (.intersection (as-map-set map1) (->set map2))) (let [retval (mut-map)] (.forEach ^Map map1 (reify BiConsumer (accept [this k v] (let [vv (.getOrDefault ^Map map2 k ::failure)] (when-not (identical? vv ::failure) (.put ^Map retval k (.apply bfn v vv))))))) (persistent! retval)))))) (defn intersection "Intersect the keyspace of set1 and set2 returning a new set. Also works if s1 is a map and s2 is a set - the map is trimmed to the intersecting keyspace of s1 and s2." [s1 s2] (cond (or (nil? s1) (nil? s2)) empty-set (instance? SetOps s1) (.intersection ^SetOps s1 (if (instance? Map s2) (.keySet ^Map s2) (->set s2))) (instance? Map s1) (map-intersection rhs-wins s1 s2) :else (let [retval (mut-set) [s1 s2] (if (< (count s1) (count s2)) [s1 s2] [s2 s1]) s2 (->set s2)] (-> (reduce (fn [rv v] (when (.contains s2 v) (.add ^Set rv v)) rv) (mut-set) s1) (persistent!))))) (defn update-values "Immutably (or mutably) update all values in the map returning a new map. bfn takes 2 arguments, k,v and returns a new v. Returning nil removes the key from the map. When passed a vector the keys are indexes and no nil-removal is done." [map bfn] (let [bfn (->bi-function bfn)] (cond (instance? UpdateValues map) (.updateValues ^UpdateValues map bfn) (or (instance? ITransientMap map) (instance? IPersistentMap map)) (-> (reduce (fn [^Map acc kv] (let [v (.apply bfn (key kv) (val kv))] (when-not (nil? v) (.put acc (key kv) v))) acc) (mut-map) map) (persistent!)) (instance? Map map) (let [iter (.iterator (.entrySet ^Map map))] (loop [continue? (.hasNext iter)] (if continue? (let [kv (.next iter)] (let [v (.apply bfn (key kv) (val kv))] (if-not (nil? v) (.setValue ^Map$Entry kv v) (.remove iter))) (recur (.hasNext iter))) map))) (instance? RandomAccess map) (mut-list (lznc/map-indexed #(.apply bfn %1 %2) map)) :else (map-indexed #(.apply bfn %1 %2) map)))) (defn update-vals [data f] (update-values data (bi-function k v (f v)))) (defn mapmap "Clojure's missing piece. Map over the data in src-map, which must be a map or sequence of pairs, using map-fn. map-fn must return either a new key-value pair or nil. Then, remove nil pairs, and return a new map. If map-fn returns more than one pair with the same key later pair will overwrite the earlier pair. Logically the same as: ```clojure (into {} (comp (map map-fn) (remove nil?)) src-map) ```" [map-fn src-map] (-> (reduce (fn [^Map m entry] (let [^Indexed result (map-fn entry)] (when result (.put m (.nth result 0) (.nth result 1))) m)) (mut-map nil {:init-size (or (constant-count src-map) 16)} nil) src-map) (persistent!))) (defn in-fork-join-task? "True if you are currently running in a fork-join task" [] (ForkJoinTask/inForkJoinPool)) (def ^:private default-pgroup-opts (options->parallel-options {:ordered? true})) (defn pgroups "Run y index groups across n-elems. Y is common pool parallelism. body-fn gets passed two longs, startidx and endidx. Returns a sequence of the results of body-fn applied to each group of indexes. Before using this primitive please see if [[ham-fisted.reduce/preduce]] will work. You *must* wrap this in something that realizes the results if you need the parallelization to finish by a particular point in the program - `(dorun (hamf/pgroups ...))`. Options: * `:pgroup-min` - when provided n-elems must be more than this value for the computation to be parallelized. * `:batch-size` - max batch size. Defaults to 64000." ([n-elems body-fn options] (impl/pgroups n-elems body-fn (options->parallel-options (assoc options :ordered? true)))) ([n-elems body-fn] (impl/pgroups n-elems body-fn default-pgroup-opts))) (def ^:private default-upgroup-opts (options->parallel-options {:ordered? false})) (defn upgroups "Run y index groups across n-elems. Y is common pool parallelism. body-fn gets passed two longs, startidx and endidx. Returns a sequence of the results of body-fn applied to each group of indexes. Before using this primitive please see if [[ham-fisted.reduce/preduce]] will work. You *must* wrap this in something that realizes the results if you need the parallelization to finish by a particular point in the program - `(dorun (hamf/upgroups ...))`. Options: * `:pgroup-min` - when provided n-elems must be more than this value for the computation to be parallelized. * `:batch-size` - max batch size. Defaults to 64000." ([n-elems body-fn options] (impl/pgroups n-elems body-fn (options->parallel-options (assoc options :ordered? false)))) ([n-elems body-fn] (impl/pgroups n-elems body-fn default-upgroup-opts))) (defn pmap "pmap using the commonPool. This is useful for interacting with other primitives, namely [[pgroups]] which are also based on this pool. This is a change from Clojure's base pmap in that it uses the ForkJoinPool/commonPool for parallelism as opposed to the agent pool - this makes it compose with pgroups and dtype-next's parallelism system. Before using this primitive please see if [[ham-fisted.reduce/preduce]] will work. Is guaranteed to *not* trigger the need for `shutdown-agents`." [map-fn & sequences] (impl/pmap (ParallelOptions. 0 64000 true) map-fn sequences)) (defn upmap "Unordered pmap using the commonPool. This is useful for interacting with other primitives, namely [[pgroups]] which are also based on this pool. Before using this primitive please see if [[ham-fisted.reduce/preduce]] will work. Like pmap this uses the commonPool so it composes with this api's pmap, pgroups, and dtype-next's parallelism primitives *but* it does not impose an ordering constraint on the results and thus may be significantly faster in some cases." [map-fn & sequences] (impl/pmap (ParallelOptions. 0 64000 false) map-fn sequences)) (defn pmap-opts "[[pmap]] but takes an extra option map as the *first* argument. This is useful if you, for instance, want to control exactly the parallel options arguments such as `:n-lookahead`. See docs for [[ham-fisted.reduce/options->parallel-options]]." [opts map-fn & sequences] (impl/pmap (hamf-rf/options->parallel-options opts) map-fn sequences)) (defn pmap-io "pmap for io bound tasks where you want to specify how far ahead to run - uses the clojure.lang.Agent/soloExecutor for execution of presumably io-bound operations." [n-lookahead map-fn & sequences] (impl/pmap (hamf-rf/options->parallel-options {:pool clojure.lang.Agent/soloExecutor :n-lookahead n-lookahead}) map-fn sequences)) (defn persistent! "If object is an ITransientCollection, call clojure.core/persistent!. Else return collection." [v] (if (instance? ITransientCollection v) (clojure.core/persistent! v) v)) (defn transient [v] (if (instance? IEditableCollection v) (clojure.core/transient v) v)) (defn mut-map-union! "Very fast union that may simply update lhs and return it. Both lhs and rhs *must* be mutable maps. See docs for [[map-union]]." [merge-bifn ^Map l ^Map r] (cond (identical? l r) l (map-set? l) (.union ^MapSetOps l r (->bi-function merge-bifn)) :else (let [merge-bifn (->bi-function merge-bifn)] (reduce (fn [acc kv] (.merge ^Map acc (key kv) (val kv) merge-bifn) acc) l r)))) (defn freq-reducer "Return a hamf parallel reducer that performs a frequencies operation." ([options] (let [map-fn (get options :map-fn mut-map) cfn (function _v (Consumers$IncConsumer.)) sk? (get options :skip-finalize?) fin-bfn (when-not sk? (bi-function k v (deref v)))] (reify protocols/Reducer (->init-val-fn [this] map-fn) (->rfn [this] (fn [acc v] (.inc ^Consumers$IncConsumer (.computeIfAbsent ^Map acc v cfn)) acc)) protocols/ParallelReducer (->merge-fn [this] #(map-union hamf-rf/reducible-merge %1 %2)) protocols/Finalize (finalize [this v] (if sk? v (update-values v fin-bfn)))))) ([] (freq-reducer nil))) (def ^:private nil-freq-reducer (freq-reducer nil)) (def ^:private freq-parallel-opts (options->parallel-options {:min-n 1000 :ordered? true})) (defn frequencies "Faster implementation of clojure.core/frequencies." ([coll] (frequencies nil coll)) ([options coll] (preduce-reducer (if options (freq-reducer options) nil-freq-reducer) (if options options freq-parallel-opts) coll))) (defn ^:no-doc inc-consumer "Return a consumer which increments a long counter. Consumer ignores its input. Deref the consumer to get the value of the counter." ^Consumer [] #(Consumers$IncConsumer.)) (def ^{:doc "A hamf reducer that works with inc-consumers"} inc-consumer-reducer (reify protocols/Finalize (finalize [this v] (deref v)) protocols/Reducer (->init-val-fn [this] #(Consumers$IncConsumer.)) (->rfn [this] hamf-rf/consumer-accumulator) protocols/ParallelReducer (->merge-fn [this] hamf-rf/reducible-merge))) (defn merge "Merge 2 maps with the rhs values winning any intersecting keys. Uses map-union with `BitmapTrieCommon/rhsWins`. Returns a new persistent map." ([] nil) ([m1] m1) ([m1 m2] (union m1 m2)) ([m1 m2 & args] ;;union on mutable maps can just be putAll. (reduce #(union %1 %2) (union m1 m2) args))) (defn merge-with "Merge (union) any number of maps using `f` as the merge operator. `f` gets passed two arguments, lhs-val and rhs-val and must return a new value. Returns a new persistent map." ([f] nil) ([f m1] m1) ([f m1 m2] (map-union f m1 m2)) ([f m1 m2 & args] (union-reduce-maps f (apply-concat [[(map-union f m1 m2)] args])))) (defn memoize "Efficient thread-safe version of clojure.core/memoize. Also see [[clear-memoized-fn!]] [[evict-memoized-call]] and [[memoize-cache-as-map]] to mutably clear the backing store, manually evict a value, and get a java.util.Map view of the cache backing store. ```clojure ham-fisted.api> (def m (memoize (fn [& args] (println \"fn called - \" args) args) {:write-ttl-ms 1000 :eviction-fn (fn [args rv cause] (println \"evicted - \" args rv cause))})) #'ham-fisted.api/m ham-fisted.api> (m 3) fn called - (3) (3) ham-fisted.api> (m 4) fn called - (4) (4)evicted - [3] (3) :expired ham-fisted.api> (dotimes [idx 4] (do (m 3) (evict-memoized-call m [3]))) fn called - (3) fn called - (3) fn called - (3) fn called - (3) nil ham-fisted.api> (dotimes [idx 4] (do (m 3) #_(evict-memoized-call m [3]))) fn called - (3) nil ``` Options: * `:write-ttl-ms` - Time that values should remain in the cache after write in milliseconds. * `:access-ttl-ms` - Time that values should remain in the cache after access in milliseconds. * `:soft-values?` - When true, the cache will store [SoftReferences](https://docs.oracle.com/javase/7/docs/api/java/lang/ref/SoftReference.html) to the data. * `:weak-values?` - When true, the cache will store [WeakReferences](https://docs.oracle.com/javase/7/docs/api/java/lang/ref/WeakReference.html) to the data. * `:max-size` - When set, the cache will behave like an LRU cache. * `:record-stats?` - When true, the LoadingCache will record access statistics. You can get those via the undocumented function memo-stats. * `:eviction-fn - Function that receives 3 arguments, [args v cause], when a value is evicted. Causes the keywords `:collected :expired :explicit :replaced and :size`. See [caffeine documentation](https://www.javadoc.io/static/com.github.ben-manes.caffeine/caffeine/2.9.3/com/github/benmanes/caffeine/cache/RemovalCause.html) for cause definitions." ([memo-fn] (memoize memo-fn nil)) ([memo-fn {:keys [write-ttl-ms access-ttl-ms soft-values? weak-values? max-size record-stats? eviction-fn] :as options}] (let [^LoadingCache cache (hamf-caffeine/cache (assoc options :load-fn (fn [args] (Box. (case (count args) 0 (memo-fn) 1 (memo-fn (args 0)) 2 (memo-fn (args 0) (args 1)) 3 (memo-fn (args 0) (args 1) (args 2)) (.applyTo ^IFn memo-fn (seq args)))))))] (-> (fn ([] (.val ^Box (.get cache []))) ([a] (.val ^Box (.get cache [a]))) ([a b] (.val ^Box (.get cache [a b]))) ([a b c] (.val ^Box (.get cache [a b c]))) ([a b c & args] (let [^IMutList obj-ary (mut-list)] (.add obj-ary a) (.add obj-ary b) (.add obj-ary c) (.addAllReducible obj-ary args) (.val ^Box (.get cache (persistent! obj-ary)))))) (with-meta {:cache cache}))))) (defn clear-memoized-fn! "Clear a memoized function backing store." [memoized-fn] (if-let [map (get (meta memoized-fn) :cache)] (hamf-caffeine/invalidate-all! map) (throw (Exception. (str "Arg is not a memoized fn - " memoized-fn)))) memoized-fn) (defn memoize-cache-as-map "Return the memoize backing store as an implementation of java.util.Map." ^Map [memoized-fn] (when-let [^Cache cache (get (meta memoized-fn) :cache)] (.asMap cache))) (defn evict-memoized-call [memo-fn fn-args] (when-let [cache (memoize-cache-as-map memo-fn)] (hamf-caffeine/invalidate! cache (vec fn-args)))) (defn ^:no-doc memo-stats "Return the statistics from a google guava cache. In order for a memoized function to produce these the :record-stats? option must be true." [memoize-fn] (when-let [cache (:cache (meta memoize-fn))] (when (instance? Cache cache) (hamf-caffeine/keyword-stats cache)))) (defn subvec "More general version of subvec. Works for any java list implementation including persistent vectors and any array." ([m sidx eidx] (let [^List m (if (instance? List m) m (->random-access m))] (.subList m sidx eidx))) ([m sidx] (subvec m sidx (count m)))) (defn group-by-reduce "Group by key. Apply the reduce-fn with the new value an the return of init-val-fn. Merged maps due to multithreading will be merged with merge-fn in a similar way of [[preduce]]. This type of reduction can be both faster and more importantly use less memory than a reduction of the forms: ```clojure (->> group-by map into) ;; or (->> group-by mapmap) ``` Options (which are passed to [[preduce]]): * `:map-fn` Function which takes no arguments and must return an instance of java.util.Map that supports `computeIfAbsent`. Some examples: - `(constantly (java.util.concurrent.ConcurrentHashMap. ...))` Very fast update especially in the case where the keyspace is large. - `mut-map` - Fast merge, fast update, in-place immutable conversion via `persistent!`. - `java-hashmap` - fast merge, fast update, just a simple java.util.HashMap-based reduction. - `#(LinkedHashMap.)` - When used with options {:ordered? true} the result keys will be in order *and* the result values will be reduced in order." ([key-fn init-val-fn rfn merge-fn options coll] (let [has-map-fn? (get :map-fn options) map-fn (get options :map-fn mut-map) merge-bifn (->bi-function merge-fn) rfn (cond (or (= identity key-fn) (nil? key-fn)) (let [bifn (bi-function k acc (rfn (or acc (init-val-fn)) k))] (fn [^Map l v] (.compute l v bifn) l)) ;;These formulations can trigger more efficient primitive reductions when, ;;for instance, you are reducing over a stream of integer indexes. (and (instance? IFn$LO key-fn) (instance? IFn$OLO rfn)) (long-accumulator l v (.compute ^Map l (.invokePrim ^IFn$LO key-fn v) (bi-function k acc (.invokePrim ^IFn$OLO rfn (or acc (init-val-fn)) v))) l) (and (instance? IFn$DO key-fn) (instance? IFn$ODO rfn)) (double-accumulator l v (.compute ^Map l (.invokePrim ^IFn$DO key-fn v) (bi-function k acc (.invokePrim ^IFn$ODO rfn (or acc (init-val-fn)) v))) l) :else (fn [^Map l v] ;;It annoys the hell out of me that I have to create a new ;;bifunction here but there is no threadsafe way to pass in the ;;new value to the reducer otherwise. (.compute l (key-fn v) (bi-function k acc (rfn (or acc (init-val-fn)) v))) l))] (cond-> (preduce map-fn rfn #(mut-map-union! merge-bifn %1 %2) (merge {:min-n 1000} options) coll) ;;In the case where no map-fn was passed in we return a persistent hash map. (not has-map-fn?) (persistent!)))) ([key-fn init-val-fn rfn merge-fn coll] (group-by-reduce key-fn init-val-fn rfn merge-fn nil coll))) (defn group-by-reducer "Perform a group-by-reduce passing in a reducer. Same options as group-by-reduce. Options: * `:skip-finalize?` - skip finalization step." ([key-fn reducer coll] (group-by-reducer key-fn reducer nil coll)) ([key-fn reducer options coll] (let [finalizer (if (:skip-finalize? options) identity #(update-values % (bi-function k v (protocols/finalize reducer v))))] (-> (group-by-reduce key-fn (protocols/->init-val-fn reducer) (protocols/->rfn reducer) (protocols/->merge-fn reducer) options coll) (finalizer))))) (defn group-by-consumer "Perform a group-by-reduce passing in a reducer. Same options as group-by-reduce - This uses a slightly different pathway - computeIfAbsent - in order to preserve order. In this case the return value of the reduce fn is ignored. This allows things like the linked hash map to preserve initial order of keys. It map also be slightly more efficient because the map itself does not need to check the return value of rfn - something that the `.compute` primitive *does* need to do. Options: * `:skip-finalize?` - skip finalization step." ([key-fn reducer coll] (group-by-reducer key-fn reducer nil coll)) ([key-fn reducer options coll] (let [finalizer (when-not (:skip-finalize? options) #(update-values % (bi-function k v (protocols/finalize reducer v)))) has-map-fn? (get :map-fn options) map-fn (get options :map-fn mut-map) merge-fn (protocols/->merge-fn reducer) merge-bifn (->bi-function merge-fn) init-fn (protocols/->init-val-fn reducer) afn (function k (init-fn)) rfn (protocols/->rfn reducer) rfn (cond (or (= identity key-fn) (nil? key-fn)) (fn [^Map l v] (-> (.computeIfAbsent l v afn) (rfn v)) l) ;;These formulations can trigger more efficient primitive reductions when, ;;for instance, you are reducing over a stream of integer indexes. (and (instance? IFn$LO key-fn) (instance? IFn$OLO rfn)) (long-accumulator l v (let [acc (.computeIfAbsent ^Map l (.invokePrim ^IFn$LO key-fn v) afn)] (.invokePrim ^IFn$OLO rfn acc v)) l) (and (instance? IFn$DO key-fn) (instance? IFn$ODO rfn)) (double-accumulator l v (let [acc (.computeIfAbsent ^Map l (.invokePrim ^IFn$DO key-fn v) afn)] (.invokePrim ^IFn$ODO rfn acc v)) l) :else (fn [^Map l v] (-> (.computeIfAbsent l (key-fn v) afn) (rfn v)) l))] (let [fin-map (preduce map-fn rfn #(mut-map-union! merge-bifn %1 %2) (merge {:min-n 1000} options) coll)] (cond finalizer (finalizer fin-map) (not has-map-fn?) (persistent! fin-map) :else fin-map))))) (defn group-by "Group items in collection by the grouping function f. Returns persistent map of keys to persistent vectors. Options are same as [[group-by-reduce]] but this reductions defaults to an ordered reduction." ([f options coll] (group-by-reduce f object-array-list conj! add-all! (merge {:ordered? true :min-n 1000} options) coll)) ([f coll] (group-by f nil coll))) (defn vec "Produce a persistent vector. Optimized pathways exist for object arrays and java List implementations." ([data] (if (vector? data) (if (instance? IObj data) (with-meta data nil) data) (immut-list data))) ([] (immut-list))) (defn vector ([] empty-vec) ([a] (ArrayImmutList/create true nil (ObjArray/create a))) ([a b] (ArrayImmutList/create true nil (ObjArray/create a b))) ([a b c] (ArrayImmutList/create true nil (ObjArray/create a b c))) ([a b c d] (ArrayImmutList/create true nil (ObjArray/create a b c d))) ([a b c d e] (ArrayImmutList/create true nil (ObjArray/create a b c d e))) ([a b c d e f] (ArrayImmutList/create true nil (ObjArray/create a b c d e f))) ([a b c d e f g] (ArrayImmutList/create true nil (ObjArray/create a b c d e f g))) ([a b c d e f g h] (ArrayImmutList/create true nil (ObjArray/create a b c d e f g h))) ([a b c d e f g h i] (ArrayImmutList/create true nil (ObjArray/create a b c d e f g h i))) ([a b c d e f g h i j] (ArrayImmutList/create true nil (ObjArray/create a b c d e f g h i j))) ([a b c d e f g h i j k] (ArrayImmutList/create true nil (ObjArray/create a b c d e f g h i j k))) ([a b c d e f g h i j k & args] (ArrayImmutList/create true nil (apply obj-ary a b c d e f g h i j k args)))) (defn splice "Splice v2 into v1 at idx. Returns a persistent vector." [v1 idx v2] (let [retval (mut-list) v1 (->collection v1)] (.addAll retval (subvec v1 0 idx)) (.addAll retval (->collection v2)) (.addAll retval (subvec v1 idx)) (persistent! retval))) (defn empty? [coll] (if coll (.isEmpty (->collection coll)) true)) (defn- concat-reducible (^IMutList [^IMutList retval v1 v2] (let [retval (mut-list)] (.addAllReducible retval (->reducible v1)) (.addAllReducible retval (->reducible v2)) retval)) (^IMutList [^IMutList retval v1 v2 args] (when-not (nil? v1) (.addAllReducible retval (->reducible v1))) (when-not (nil? v2) (.addAllReducible retval (->reducible v2))) (reduce (fn [data c] (when-not (nil? c) (.addAllReducible retval (->reducible c))) retval) retval args))) (defn apply-concatv [data] (reduce (fn [^IMutList v data] (when data (.addAllReducible v data)) v) (mut-list) data)) (defn concatv "non-lazily concat a set of items returning a persistent vector. " ([] empty-vec) ([v1] (vec v1)) ([v1 v2] (cond (nil? v1) (vec v2) (nil? v2) (vec v1) :else (-> (concat-reducible (mut-list) v1 v2) (persistent!)))) ([v1 v2 & args] (-> (concat-reducible (mut-list) v1 v2 args) (persistent!)))) (defn concata "non-lazily concat a set of items returning an object array. This always returns an object array an may return an empty array whereas concat may return nil." (^objects [] (object-array nil)) (^objects [v1] (object-array v1)) (^objects [v1 v2] (cond (nil? v1) (object-array v2) (nil? v2) (object-array v1) :else (-> (concat-reducible (ArrayLists$ObjectArrayList.) v1 v2) (.toArray)))) (^objects [v1 v2 & args] (-> (concat-reducible (ArrayLists$ObjectArrayList.) v1 v2 args) (.toArray)))) (defn apply-concat "Faster lazy noncaching version of (apply concat)" [args] (if args (lznc/apply-concat args) '())) (defn object-array "Faster version of object-array for java collections and strings." ^objects [item] (lznc/object-array item)) (defn into-array "Faster version of clojure.core/into-array." ([aseq] (lznc/into-array aseq)) ([ary-type aseq] (lznc/into-array ary-type aseq)) ([ary-type mapfn aseq] (lznc/into-array ary-type mapfn aseq))) (defn- ->comparator ^java.util.Comparator [comp] (or comp compare)) (defn sorta "Sort returning an object array." (^objects [coll] (sorta hamf-fn/comp-nan-last coll)) (^objects [comp coll] (let [coll (->reducible coll)] (if (instance? IMutList coll) (-> (.immutSort ^IMutList coll comp) (.toArray)) (let [a (object-array coll)] (if (< (alength a) 1000) (if comp (Arrays/sort a (->comparator comp)) (Arrays/sort a)) (Arrays/parallelSort a 0 (alength a) (if comp (->comparator comp) compare))) a))))) (defn sort "Exact replica of clojure.core/sort but instead of wrapping the final object array in a seq which loses the fact the result is countable and random access. Faster implementations are provided when the input is an integer, long, or double array. The default comparison is nan-last meaning null-last if the input is an undefined container and nan-last if the input is a double or float specific container." ([coll] (sort hamf-fn/comp-nan-last coll)) ([comp coll] (let [coll (->collection coll)] (if (instance? ImmutSort coll) (if (nil? comp) (.immutSort ^ImmutSort coll) (.immutSort ^ImmutSort coll (->comparator comp))) (let [a (sorta comp coll)] (ArrayLists/toList a 0 (alength a) ^IPersistentMap (meta coll))))))) (defn sort-by "Sort a collection by keyfn. Typehinting the return value of keyfn will somewhat increase the speed of the sort :-)." ([keyfn coll] (sort-by keyfn nil coll)) ([keyfn comp coll] (let [coll (->random-access coll) data (map keyfn coll) ;;Arraylists are faster to create because they do not have to be sized exactly ;;to the collection. They have very fast addAllReducible pathways that specialize ;;for constant sized containers. data (case (lznc/type-single-arg-ifn keyfn) :float64 (double-array-list data) :int64 (long-array-list data) (object-array-list data)) indexes (argsort comp data)] (reindex coll indexes)))) (defn shuffle "shuffle values returning random access container. If you are calling this repeatedly on the same collection you should call [[->random-access]] on the collection *before* you start as shuffle internally only works on random access collections. Options: * `:seed` - If instance of java.util.Random, use this. If integer, use as seed. If not provided a new instance of java.util.Random is created." (^List [coll] (lznc/shuffle coll nil)) (^List [coll opts] (lznc/shuffle coll opts))) (defn binary-search "Binary search. Coll must be a sorted random access container. comp must be an implementation of java.lang.Comparator. If you know your container's type, such as a double array, then comp should be a fastutil DoubleComparator. The most efficient method will be to convert coll to random access using ->random-access, so for a pure double array it is slightly better to call ->random-access outside this function before the function call. This search defaults to the slower java.util.Collections search using clojure's built in `compare` - reason being that that allows you to search for a double number in a vector of only longs. If you want an accelerated search you can explicitly pass in a nil comparator *but* you need to make sure that you are searching for the rough datatype in the data - e.g. long in a byte array or a double in a double for float array. Searching for doubles in integer arrays with an accelerated search will probably result in confusing results. ```clojure ham-fisted.api> (def data (->random-access (double-array (range 10)))) #'ham-fisted.api/data ham-fisted.api> (binary-search data 0) 0 ham-fisted.api> (binary-search data -1) 0 ham-fisted.api> (binary-search data 1) 1 ham-fisted.api> (binary-search data 1.1) 2 ham-fisted.api> (binary-search data 10) 10 ham-fisted.api> (binary-search data 11) 10 ham-fisted.api> ;;be wary of datatype conversions in typed containers ham-fisted.api> (def data (->random-access (int-array (range 10)))) #'ham-fisted.api/data ham-fisted.api> (binary-search data 1) 1 ham-fisted.api> (binary-search data 1.1) 2 ham-fisted.api> ;;accelerated search - flattens input to container datatype ham-fisted.api> (binary-search data 1.1 nil) 1 ```" (^long [coll v] (binary-search coll v compare)) (^long [coll v comp] (let [comp (when comp (->comparator comp)) coll (->random-access coll)] (if (instance? IMutList coll) (if comp (.binarySearch ^IMutList coll v comp) (.binarySearch ^IMutList coll v)) (let [rv (if comp (Collections/binarySearch coll v comp) ;;This corrects for things like searching 50.1 in a list that has longs (Collections/binarySearch coll v compare))] (if (< rv 0) (- -1 rv) rv)))))) (defn iarange "Return an integer array holding the values of the range. Use `->collection` to get a list implementation wrapping for generic access." (^ints [end] (iarange 0 end 1)) (^ints [start end] (iarange start end 1)) (^ints [start end step] (ArrayLists/iarange start end step))) (defn larange "Return a long array holding values of the range. Use `->collection` get a list implementation for generic access." (^longs [end] (larange 0 end 1)) (^longs [start end] (larange start end 1)) (^longs [start end step] (ArrayLists/larange start end step))) (defn darange "Return a double array holding the values of the range. Use `wrap-array` to get an implementation of java.util.List that supports the normal Clojure interfaces." (^doubles [end] (darange 0 end 1)) (^doubles [start end] (darange start end 1)) (^doubles [start end step] (ArrayLists/darange start end step))) (defn- floating? [item] (or (double? item) (float? item))) (defn range "When given arguments returns a range that implements random access java list interfaces so nth, reverse and friends are efficient." ([] (clojure.core/range)) ([end] (range 0 end 1)) ([start end] (range start end 1)) ([start end step] (if (and (integer? start) (integer? end) (integer? step)) (Ranges$LongRange. start end step nil) (Ranges$DoubleRange. start end step nil)))) (defn argsort "Sort a collection of data returning an array of indexes. The collection must be random access and the return value is an integer array of indexes which will read the input data in sorted order. Faster implementations are provided when the collection is an integer, long, or double array. See also [[reindex]]." ([comp coll] (let [^List coll (if (instance? RandomAccess coll) coll (let [coll (->collection coll)] (if (instance? RandomAccess coll) coll (object-array-list coll))))] (-> (if (instance? IMutList coll) (.sortIndirect ^IMutList coll (when comp (->comparator comp))) (let [idata (iarange (.size coll)) idx-comp (ArrayLists/intIndexComparator coll comp)] (IntArrays/parallelQuickSort idata ^IntComparator idx-comp) idata)) (->collection)))) ([coll] (argsort hamf-fn/comp-nan-last coll))) (defn ^:no-doc do-make-array [clj-ary-fn ary-ra-fn ary-list-fn data] (cond (number? data) (clj-ary-fn data) :else (let [data (->reducible data)] (if-let [c (constant-count data)] (let [retval (clj-ary-fn c)] (.fillRangeReducible ^IMutList (ary-ra-fn retval) 0 data) retval) (.toNativeArray ^IMutList (ary-list-fn data)))))) (defn byte-array-list (^IMutList [] (ByteArrayList. (clojure.core/byte-array 4) 0 nil)) (^IMutList [data] (if (number? data) (ByteArrayList. (clojure.core/byte-array data) 0 nil) (let [^IMutList retval (if (instance? RandomAccess data) (byte-array-list (.size ^List data)) (byte-array-list))] (.addAllReducible retval data) retval)))) (defn byte-array (^bytes [] (byte-array 0)) (^bytes [data] (if (instance? IMutList data) (let [^IMutList data data sz (.size data) rv (clojure.core/byte-array sz)] (dotimes [idx sz] (ArrayHelpers/aset rv idx (unchecked-byte (.getLong data idx)))) rv) (do-make-array clojure.core/byte-array #(ArrayLists/toList ^bytes %) byte-array-list data)))) (defn short-array-list (^IMutList [] (ShortArrayList. (clojure.core/short-array 4) 0 nil)) (^IMutList [data] (if (number? data) (ShortArrayList. (clojure.core/short-array data) 0 nil) (let [^IMutList retval (if (instance? RandomAccess data) (short-array-list (.size ^List data)) (short-array-list))] (.addAllReducible retval data) retval)))) (defn short-array (^shorts [] (short-array 0)) (^shorts [data] (if (instance? IMutList data) (let [^IMutList data data sz (.size data) rv (clojure.core/short-array sz)] (dotimes [idx sz] (ArrayHelpers/aset rv idx (unchecked-short (.getLong data idx)))) rv) (do-make-array clojure.core/short-array #(ArrayLists/toList ^shorts %) short-array-list data)))) (defn char-array-list (^IMutList [] (CharArrayList. (clojure.core/char-array 4) 0 nil)) (^IMutList [data] (if (number? data) (CharArrayList. (clojure.core/char-array data) 0 nil) (let [^IMutList retval (if (instance? RandomAccess data) (char-array-list (.size ^List data)) (char-array-list))] (.addAllReducible retval data) retval)))) (defn char-array (^chars [] (char-array 0)) (^chars [data] (do-make-array clojure.core/char-array #(ArrayLists/toList ^chars %) char-array-list data))) (defn boolean-array-list (^IMutList [] (BooleanArrayList. (clojure.core/boolean-array 4) 0 nil)) (^IMutList [data] (if (number? data) (BooleanArrayList. (clojure.core/boolean-array data) 0 nil) (let [^IMutList retval (if (instance? RandomAccess data) (boolean-array-list (.size ^List data)) (boolean-array-list))] (.addAllReducible retval data) retval)))) (defn boolean-array (^booleans [] (boolean-array 0)) (^booleans [data] (do-make-array clojure.core/boolean-array #(ArrayLists/toList ^booleans %) boolean-array-list data))) (defmacro ^:no-doc impl-array-macro [data ctor elem-cast vecfn] (cond (number? data) `(~ctor ~data) ;;16 chosen arbitrarily (and (vector? data) (< (count data) 16)) `(let [~'ary (~ctor (unchecked-int ~(count data)))] (do ~@(->> (range (count data)) (map (fn [^long idx] `(ArrayHelpers/aset ~'ary (unchecked-int ~idx) (~elem-cast ~(data idx)))))) ~'ary)) :else `(~vecfn ~data))) (defn int-array-list "An array list that is as fast as java.util.ArrayList for add,get, etc but includes many accelerated operations such as fill and an accelerated addAll when the src data is an array list." (^IMutList [] (ArrayLists$IntArrayList.)) (^IMutList [cap-or-data] (if (number? cap-or-data) (ArrayLists$IntArrayList. (int cap-or-data)) (doto (ArrayLists$IntArrayList.) (.addAllReducible (->reducible cap-or-data)))))) (def ^:private int-ary-cls (Class/forName "[I")) (defn ^:no-doc int-array-v ^ints [data] (cond (instance? int-ary-cls data) data (instance? IMutList data) (.toIntArray ^IMutList data) :else (do-make-array #(ArrayLists/intArray %) #(ArrayLists/toList ^ints %) int-array-list data))) (defmacro int-array ([] `(ArrayLists/intArray 0)) ([data] `(impl-array-macro ~data ArrayLists/intArray Casts/longCast int-array-v))) (defn reindex "Permut coll by the given indexes. Result is random-access and the same length as the index collection. Indexes are expected to be in the range of [0->count(coll))." [coll indexes] (lznc/reindex (->random-access coll) (int-array indexes))) (defmacro ivec "Create a persistent-vector-compatible list backed by an int array." ([] `(ArrayLists/toList (int-array))) ([data] `(ArrayLists/toList (int-array ~data)))) (defn long-array-list "An array list that is as fast as java.util.ArrayList for add,get, etc but includes many accelerated operations such as fill and an accelerated addAll when the src data is an array list." (^IMutList [] (ArrayLists$LongArrayList.)) (^IMutList [cap-or-data] (if (number? cap-or-data) (ArrayLists$LongArrayList. (int cap-or-data)) (doto (ArrayLists$LongArrayList.) (.addAllReducible (->reducible cap-or-data)))))) (def ^:no-doc long-array-cls (Class/forName "[J")) (defn ^:no-doc long-array-v ^longs [data] (cond (instance? long-array-cls data) data (instance? IMutList data) (.toLongArray ^IMutList data) :else (do-make-array #(ArrayLists/longArray %) #(ArrayLists/toList ^longs %) long-array-list data))) (defmacro long-array ([] `(ArrayLists/longArray 0)) ([data] `(impl-array-macro ~data ArrayLists/longArray Casts/longCast long-array-v))) (defmacro lvec "Create a persistent-vector-compatible list backed by a long array." ([] `(ArrayLists/toList (long-array))) ([data] `(ArrayLists/toList (long-array ~data)))) (defn float-array-list (^IMutList [] (FloatArrayList. (clojure.core/float-array 4) 0 nil)) (^IMutList [data] (if (number? data) (FloatArrayList. (clojure.core/float-array data) 0 nil) (let [^IMutList retval (if (instance? RandomAccess data) (float-array-list (.size ^List data)) (float-array-list))] (.addAllReducible retval data) retval)))) (defn ^:no-doc float-array-v ^floats [data] (if (instance? IMutList data) (.toFloatArray ^IMutList data) (do-make-array #(ArrayLists/floatArray %) #(ArrayLists/toList ^floats %) float-array-list data))) (defmacro float-array ([] `(ArrayLists/floatArray 0)) ([data] `(impl-array-macro ~data ArrayLists/floatArray Casts/doubleCast float-array-v))) (defmacro fvec "Create a persistent-vector-compatible list backed by a float array." ([] `(ArrayLists/toList (float-array))) ([data] `(ArrayLists/toList (float-array ~data)))) (defn double-array-list "An array list that is as fast as java.util.ArrayList for add,get, etc but includes many accelerated operations such as fill and an accelerated addAll when the src data is an array list." (^IMutList [] (ArrayLists$DoubleArrayList.)) (^IMutList [cap-or-data] (if (number? cap-or-data) (ArrayLists$DoubleArrayList. (int cap-or-data)) (doto (ArrayLists$DoubleArrayList.) (.addAllReducible (->reducible cap-or-data)))))) (def dbl-ary-cls (Class/forName "[D")) (defn ^:no-doc double-array-v ^doubles [data] (cond (instance? dbl-ary-cls data) data (instance? IMutList data) (.toDoubleArray ^IMutList data) :else (do-make-array #(ArrayLists/doubleArray %) #(ArrayLists/toList ^doubles %) double-array-list data))) (defmacro double-array ([] `(ArrayLists/doubleArray 0)) ([data] `(impl-array-macro ~data ArrayLists/doubleArray Casts/doubleCast double-array-v))) (defmacro dvec "Create a persistent-vector-compatible list backed by a double array." ([] `(ArrayLists/toList (double-array))) ([data] `(ArrayLists/toList (double-array ~data)))) (defn object-array-list "An array list that is as fast as java.util.ArrayList for add,get, etc but includes many accelerated operations such as fill and an accelerated addAll when the src data is an object array based list." (^IMutList [] (ArrayLists$ObjectArrayList.)) (^IMutList [cap-or-data] (if (number? cap-or-data) (ArrayLists$ObjectArrayList. (int cap-or-data)) (doto (ArrayLists$ObjectArrayList.) (.addAllReducible (->reducible cap-or-data)))))) (defn ^:no-doc ovec-v ^ArrayImmutList [data] (if (instance? obj-ary-cls data) (ArrayImmutList. ^objects data 0 (alength ^objects data) nil) (into (vec) data))) (defmacro ovec "Return an immutable persistent vector like object backed by a single object array." ([] `ArrayImmutList/EMPTY) ([data] (cond (number? data) `(ArrayImmutList. (ArrayLists/objectArray ~data) 0 ~data nil) (vector? data) `(ArrayImmutList. (obj-ary ~@data) 0 ~(count data) ~(meta data)) :else `(ovec-v ~data)))) (defn add-constant! (^List [^List l ^long idx v] (.add l (unchecked-int idx) v) l) (^IMutList [^IMutList l ^long idx ^long count v] (.add l (unchecked-int idx) (unchecked-int count) v) l)) (defmacro dnth "nth operation returning a primitive double. Efficient when obj is a double array." [obj idx] `(TypedNth/dnth ~obj ~idx)) (defmacro lnth "nth operation returning a primitive long. Efficient when obj is a long array." [obj idx] `(TypedNth/lnth ~obj ~idx)) (defmacro fnth "nth operation returning a primitive float. Efficient when obj is a float array." [obj idx] `(TypedNth/fnth ~obj ~idx)) (defmacro inth "nth operation returning a primitive int. Efficient when obj is an int array." [obj idx] `(TypedNth/inth ~obj ~idx)) (defn mapv "Produce a persistent vector from a collection." ([map-fn coll] (if-let [c (constant-count coll)] (let [c (int c) rv (ArrayLists/objectArray c)] (reduce (hamf-rf/indexed-accum acc idx v (ArrayHelpers/aset rv idx (map-fn v))) nil coll) (ArrayImmutList. rv 0 c nil)) (let [rv (ArrayLists$ObjectArrayList. (ArrayLists/objectArray 8) 0 nil) _ (reduce (fn [acc v] (.conj rv (map-fn v))) rv coll)] (persistent! rv)))) ([map-fn c1 c2] (ovec (map map-fn c1 c2))) ([map-fn c1 c2 c3] (ovec (map map-fn c1 c2 c3))) ([map-fn c1 c2 c3 & args] (ovec (apply map map-fn c1 c2 c3 args)))) (defn filterv "Filter a collection into a vector." [pred coll] (ovec (filter pred coll))) (defn sum-fast "Fast simple double summation. Does not do any nan checking or summation compensation." ^double [coll] ;;Using raw reduce call as opposed to reduce-reducer to avoid protocol dispatch for small N @(reduce double-consumer-accumulator (Sum$SimpleSum.) coll)) (defn ^:no-doc apply-nan-strategy [options coll] (case (get options :nan-strategy :remove) :remove (filter (hamf-fn/double-predicate v (not (Double/isNaN v))) coll) :keep coll :exception (map (hamf-fn/double-unary-operator v (when (Double/isNaN v) (throw (Exception. "Nan detected"))) v) coll))) (defn sum-stable-nelems "Stable sum returning map of {:sum :n-elems}. See options for [[sum]]." ([coll] (sum-stable-nelems nil coll)) ([options coll] (->> (->reducible coll) (apply-nan-strategy options) (preduce-reducer (Sum.) options)))) (defn sum "Very stable high performance summation. Uses both threading and kahans compensated summation. Options: * `nan-strategy` - defaults to `:remove`. Options are `:keep`, `:remove` and `:exception`." (^double [coll] (sum nil coll)) (^double [options coll] (get (sum-stable-nelems options coll) :sum))) (defn- long-summary-sum ^long [^LongSummaryStatistics lstats] (.getSum lstats)) (defn lsum "Simple summation that returns a long integer." ^long [data] (let [lv (long-array 1)] (reduce (fn [_ ^long v] (let [_ (aset lv 0 (+ (aget lv 0) v))]) nil) nil data) (aget lv 0))) (defn lsummary "Summary statistics {:mean :max :min :n-elems :sum} in long space" [data] (let [^LongSummaryStatistics lstats (reduce long-consumer-accumulator (LongSummaryStatistics.) data)] {:max (.getMax lstats) :min (.getMin lstats) :n-elems (.getCount lstats) :mean (.getAverage lstats) :sum (.getSum lstats)})) (defn mean "Return the mean of the collection. Returns double/NaN for empty collections. See options for [[sum]]." (^double [coll] (mean nil coll)) (^double [options coll] (let [vals (sum-stable-nelems options coll)] (/ (double (vals :sum)) (long (vals :n-elems)))))) (defn dsummary "Summary statistics {:mean :max :min :n-elems :sum} in double space" [data] (let [^DoubleSummaryStatistics lstats (reduce double-consumer-accumulator (DoubleSummaryStatistics.) data)] {:max (.getMax lstats) :min (.getMin lstats) :n-elems (.getCount lstats) :mean (.getAverage lstats) :sum (.getSum lstats)})) (defn first "Get the first item of a collection." [coll] (if (nil? coll) nil (let [coll (->reducible coll)] (if (instance? RandomAccess coll) (when-not (.isEmpty ^List coll) (.get ^List coll 0)) (clojure.core/first coll))))) (defn last "Get the last item in the collection. Constant time for random access lists." [coll] (if (nil? coll) nil (let [coll (->reducible coll)] (cond (instance? RandomAccess coll) (let [^List coll coll sz (.size coll)] (when-not (== 0 sz) (.get coll (unchecked-dec sz)))) (instance? Reversible coll) (RT/first (.rseq ^Reversible coll)) :else (clojure.core/last coll))))) (defn- key? [e] (when e (key e))) (deftype ^:private MaxKeyReducer [^{:unsynchronized-mutable true :tag long} data ^:unsynchronized-mutable value ^IFn$OL mapper] Consumer (accept [this v] (let [mval (.invokePrim mapper v)] (when (>= mval data) (set! data mval) (set! value v)))) clojure.lang.IDeref (deref [this] value) Reducible (reduce [this o] (cond (== data Long/MIN_VALUE) o (== (.-data ^MaxKeyReducer o) Long/MIN_VALUE) this :else (do (.accept this (deref o)) this)))) (defn- ensure-obj->long ^IFn$OL [f] (if (instance? IFn$OL f) f (obj->long v (f v)))) (defn mmax-key "Faster and nil-safe version of #(apply max-key %1 %2)" [f data] (let [f (ensure-obj->long f)] @(reduce hamf-rf/consumer-accumulator (MaxKeyReducer. Long/MIN_VALUE nil f) data))) (deftype ^:private MinKeyReducer [^{:unsynchronized-mutable true :tag long} data ^:unsynchronized-mutable value ^IFn$OL mapper] Consumer (accept [this v] (let [mval (.invokePrim mapper v)] (when (<= mval data) (set! data mval) (set! value v)))) clojure.lang.IDeref (deref [this] value) Reducible (reduce [this o] (cond (== data Long/MAX_VALUE) o (== (.-data ^MinKeyReducer o) Long/MAX_VALUE) this :else (do (.accept this (deref o)) this)))) (defn mmin-key "Faster and nil-safe version of #(apply min-key %1 %2)" [f data] (let [f (ensure-obj->long f)] @(reduce hamf-rf/consumer-accumulator (MinKeyReducer. Long/MAX_VALUE nil f) data))) (deftype ^:private MaxIdx [^{:unsynchronized-mutable true :tag long} data ^{:unsynchronized-mutable true :tag long} idx ^{:unsynchronized-mutable true :tag long} value ^IFn$OL mapper] Consumer (accept [this v] (let [mval (.invokePrim mapper v)] (when (>= mval data) (set! data mval) (set! value idx)) (set! idx (unchecked-inc idx)))) clojure.lang.IDeref (deref [this] value)) (defn mmax-idx "Like [[mmin-key]] but returns the max index. F should be a function from obj->long." ^long [f data] @(reduce hamf-rf/consumer-accumulator (MaxIdx. Long/MIN_VALUE 0 -1 (ensure-obj->long f)) data)) (deftype ^:private MinIdx [^{:unsynchronized-mutable true :tag long} data ^{:unsynchronized-mutable true :tag long} idx ^{:unsynchronized-mutable true :tag long} value ^IFn$OL mapper] Consumer (accept [this v] (let [mval (.invokePrim mapper v)] (when (<= mval data) (set! data mval) (set! value idx)) (set! idx (unchecked-inc idx)))) clojure.lang.IDeref (deref [this] value)) (defn mmin-idx "Like [[mmin-key]] but returns the min index. F should be a function from obj->long." ^long [f data] @(reduce hamf-rf/consumer-accumulator (MinIdx. Long/MAX_VALUE 0 -1 (ensure-obj->long f)) data)) (defn intersect-sets "Given a sequence of sets, efficiently perform the intersection of them. This algorithm is usually faster and has a more stable runtime than (reduce clojure.set/intersection sets) which degrades depending on the order of the sets and the pairwise intersection of the initial sets." [sets] (let [sets (vec sets) ns (count sets)] (case ns 0 #{} 1 (sets 0) (let [min-idx (mmin-idx (fn ^long [arg] (.size ^Set arg)) sets)] (-> (reduce (fn [^Set rv ^long idx] (cond (.isEmpty rv) (reduced rv) (== idx min-idx) rv :else (intersection rv (sets idx)))) (transient (sets min-idx)) (range ns)) (persistent!)))))) (defn mode "Return the most common occurance in the data." [data] (->> (frequencies {:map-fn java-hashmap} data) (mmax-key val) (key?))) (defn rest "Version of rest that does uses subvec if collection is random access. This preserves the ability to reduce in parallel over the collection." [coll] (cond (nil? coll) nil (instance? RandomAccess coll) (if (pos? (count coll)) (subvec coll 1) []) :else (clojure.core/rest coll))) (defn reverse "Reverse a collection or sequence. Constant time reverse is provided for any random access list." [coll] (if (instance? Comparator coll) (.reversed ^Comparator coll) (let [coll (->reducible coll)] (cond (instance? IMutList coll) (.reverse ^IMutList coll) (instance? RandomAccess coll) (ReverseList/create coll (meta coll)) :else (clojure.core/reverse coll))))) (defn take "Take the first N values from a collection. If the input is random access, the result will be random access." ([n] (clojure.core/take n)) ([n coll] (when coll (let [coll (->reducible coll)] (if (instance? RandomAccess coll) (.subList ^List coll 0 (min (long n) (.size ^List coll))) (clojure.core/take n coll)))))) (defn take-last "Take the last N values of the collection. If the input is random-access, the result will be random-access." ([n coll] (when coll (let [coll (->reducible coll)] (if (instance? RandomAccess coll) (let [ne (.size ^List coll) n (long n)] (.subList ^List coll (- ne n 1) ne)) (clojure.core/take-last n coll)))))) (defn- priority-queue-rf [^long n comp] (let [comp (when comp (.reversed (->comparator comp)))] (fn ([] (if comp (PriorityQueue. n comp) (PriorityQueue. n))) ([acc] (->random-access (.toArray ^Collection acc))) ([acc v] (let [^PriorityQueue acc acc] (.offer acc v) (when (> (.size acc) n) (.poll acc))) acc)))) (defn take-min "Take the min n values of a collection. This is not an order-preserving operation." ([n comp values] (reduce-reducer (priority-queue-rf n comp) values)) ([n values] (take-min n < values))) (defn drop "Drop the first N items of the collection. If item is random access, the return value is random-access." [n coll] (when coll (let [coll (->reducible coll)] (if (instance? RandomAccess coll) (subvec coll (min (long n) (.size ^List coll))) (clojure.core/drop n coll))))) (defn drop-min "Drop the min n values of a collection. This is not an order-preserving operation." ([n comp values] (let [values (->random-access values) tn (- (count values) (long n))] (if (<= tn 0) empty-vec (take-min tn (.reversed (->comparator comp)) values)))) ([n values] (drop-min n nil values))) (defn drop-last "Drop the last N values from a collection. IF the input is random access, the result will be random access." ([n] (clojure.core/drop-last n)) ([n coll] (when coll (let [coll (->reducible coll)] (if (instance? RandomAccess coll) (let [ne (.size ^List coll) n (min (long n) ne)] (.subList ^List coll 0 (- ne n))) (clojure.core/take-last n coll)))))) (defn repeat "When called with no arguments, produce an infinite sequence of v. When called with 2 arguments, produce a random access list that produces v at each index." ([v] (clojure.core/repeat v)) (^List [n v] (ConstList/create n v nil))) (defn ^:no-doc double-eq [lhs rhs] (if (number? lhs) (let [lhs (double lhs) rhs (double rhs)] (cond (and (Double/isNaN lhs) (Double/isNaN rhs)) true (or (Double/isNaN lhs) (Double/isNaN rhs)) false :else (== lhs rhs))) (and (== (count lhs) (count rhs)) (every? identity (map double-eq lhs rhs))))) (defmacro reduced-> "Helper macro to implement reduce chains checking for if the accumulator is reduced before calling the next expression in data. ```clojure (defrecord YMC [year-month ^long count] clojure.lang.IReduceInit (reduce [this rfn init] (let [init (reduced-> rfn init (clojure.lang.MapEntry/create :year-month year-month) (clojure.lang.MapEntry/create :count count))] (if (and __extmap (not (reduced? init))) (reduce rfn init __extmap) init)))) ```" [rfn acc & data] (reduce (fn [expr next-val] `(let [val# ~expr] (if (reduced? val#) val# (~rfn val# ~next-val)))) acc data)) (defmacro custom-ireduce "Custom implementation of IReduceInit and nothing else. This can be the most efficient way to pass data to other interfaces. Also see [[custom-counted-ireduce]] if the object should also implement ICounted. See [[reduced->]] for implementation helper." [rfn acc & code] `(reify ITypedReduce (reduce [this# ~rfn ~acc] ~@code))) (defmacro custom-counted-ireduce "Custom implementation of IReduceInit and nothing else. This can be the most efficient way to pass data to other interfaces. Also see custom-ireduce if the object does not need to be counted and see [[reduced->]] for implementation helper." [n-elems rfn acc & code] `(reify Counted (count [this] (unchecked-int ~n-elems)) ITypedReduce (reduce [this# ~rfn ~acc] ~@code))) (defn wrap-array "Wrap an array with an implementation of IMutList" ^IMutList [ary] (alists/wrap-array ary)) (defn wrap-array-growable "Wrap an array with an implementation of IMutList that supports add and addAllReducible. 'ptr is the numeric put ptr, defaults to the array length. Pass in zero for a preallocated but empty growable wrapper." (^IMutList [ary ptr] (alists/wrap-array-growable ary ptr)) (^IMutList [ary] (alists/wrap-array-growable ary))) (defn inc-consumer "Return a consumer that simply increments a long. See java/ham_fisted/Consumers.java for definition." (^Consumers$IncConsumer [] (Consumers$IncConsumer.)) (^Consumers$IncConsumer [^long init-value] (Consumers$IncConsumer. init-value))) (defn re-matches "Much faster version of clojure.core/re-matches." [^java.util.regex.Pattern re s] (let [m (re-matcher re s)] (when (. m (matches)) (let [gc (. m (groupCount))] (if (zero? gc) (.group m) (do (let [gc (inc gc) rv (object-array gc)] (dotimes [idx gc] (aset rv idx (.group m idx))) (ArrayLists/toList rv)))))))) (defn lines "Return a closeable iterable that produces an iterator that simply produces lines of the reader. Iterator does not cache more than 1 line so there is no possibility of holding onto head normal nor chunked seq-ing. Example: ```clojure (with-open [ld (hamf/lines fname)] (->> ld (hamf/pmap (fn [line] ...)) (reduce ...))) ```" ^java.lang.AutoCloseable [fname] (let [^java.io.BufferedReader rdr (cond (instance? java.io.BufferedReader fname) fname (instance? String fname) (-> (java.io.FileReader. (str fname)) (java.io.BufferedReader.)) :else (clojure.java.io/reader fname))] (reify Iterable java.lang.AutoCloseable (close [this] (.close rdr)) (iterator [this] (let [line* (volatile! (.readLine rdr))] (reify java.util.Iterator (hasNext [this] (boolean @line*)) (next [this] (let [rv @line*] (vreset! line* (.readLine rdr)) rv)))))))) (defn linear-merge-iterable "Create an N-way merge iterable using cmp to order the merge of provided iterables. If a predicate pred is provided the iterable itself will filter out values for which the pred returns false. In this mode it is possible for the iterator to return null if the last value is filtered out -- the hasNext method doesn't check if the next value passes the predicate." ([cmp pred iterables] (reify Iterable (iterator [this] (iterator/linear-merge-iterator cmp pred (lznc/map iterator/->iterator iterables))))) ([cmp iterables] (linear-merge-iterable cmp MergeIterator/alwaysTrue iterables)) ([iterables] (linear-merge-iterable compare MergeIterator/alwaysTrue iterables))) (defn merge-iterator "Create an N-way priority queue iterable using cmp to order the merge of provided iterables. If a predicate pred is provided the iterable itself will filter out values for which the pred returns false. In this mode it is possible for the iterator to return null if the last value is filtered out -- the hasNext method doesn't check if the next value passes the predicate." [cmp iterables] (iterator/merge-iterable cmp iterables)) ================================================ FILE: src/ham_fisted/bloom_filter.clj ================================================ (ns ham-fisted.bloom-filter "Simple fast bloom filter based on apache parquet BlockSplitBloomFilter." (:require [ham-fisted.protocols :as hamf-proto] [ham-fisted.function :as hamf-fn] [ham-fisted.defprotocol :refer [extend extend-type extend-protocol]]) (:import [ham_fisted BlockSplitBloomFilter] [java.util UUID] [java.time Instant] [clojure.lang IFn$OL]) (:refer-clojure :exclude [contains? extend extend-type extend-protocol])) (set! *warn-on-reflection* true) (def byte-array-cls (type (byte-array 0))) (extend-protocol hamf-proto/SerializeObjBytes nil (serialize->bytes [v] (byte-array 0)) Double (serialize->bytes [v] (-> (java.nio.ByteBuffer/allocate 8) (.putDouble v) (.array))) UUID (serialize->bytes [v] (let [^UUID v v bdata (byte-array 16) ^java.nio.ByteBuffer bbuf (-> (java.nio.ByteBuffer/wrap bdata) (.order java.nio.ByteOrder/LITTLE_ENDIAN))] (.putLong bbuf (.getMostSignificantBits v)) (.putLong bbuf (.getLeastSignificantBits v)) bdata)) String (serialize->bytes [v] (.getBytes ^String v)) Instant (serialize->bytes [inst] (let [buf (java.nio.ByteBuffer/wrap (byte-array 12))] (.putLong buf (.getEpochSecond inst)) (.putInt buf (.getNano inst)) (.array buf)))) (defn serialize->bytes "Serialize an object to a byte array" ^bytes [o] (if (instance? byte-array-cls o) o (hamf-proto/serialize->bytes o))) (defn hash-obj "Hash an object. If integer - return integer else serialize->bytes and hash those" ^long [obj] (if (or (instance? Long obj) (instance? Integer obj) (instance? Short obj) (instance? Byte obj)) (long obj) (BlockSplitBloomFilter/hash (serialize->bytes obj)))) (defn bloom-filter "Create a bloom filter. * 'n' - Number of distinct values. * 'f' - Value from 1.0-0.0, defaults to 0.01. False positive rate." [n f] (BlockSplitBloomFilter. (quot (BlockSplitBloomFilter/optimalNumOfBits n f) 8))) (defn insert-hash! ^BlockSplitBloomFilter [^BlockSplitBloomFilter bf ^long hc] (.insertHash bf hc) bf) (defn make-long-hash-predicate [^BlockSplitBloomFilter bf] (hamf-fn/long-predicate hs (.findHash bf hs))) (defn make-obj-predicate [^BlockSplitBloomFilter bf] (hamf-fn/predicate hs (.findHash bf (hash-obj hs)))) (defn contains? [^BlockSplitBloomFilter bf obj] (.findHash bf (hash-obj obj))) (defn insert-obj ^BlockSplitBloomFilter [bf o] (if (instance? Long o) (insert-hash! bf (long o)) (insert-hash! bf (BlockSplitBloomFilter/hash (serialize->bytes o))))) (defn bitset-size "Return the length of the byte array underlying this bitset" ^long [^BlockSplitBloomFilter fb] (.getBitsetSize fb)) (defn bloom-filter->byte-array ^bytes [^BlockSplitBloomFilter bf] (.bitset bf)) (defn byte-array->bloom-filter ^BlockSplitBloomFilter [^bytes data] (BlockSplitBloomFilter. data)) (defn make-uuid-hasher ^IFn$OL [] (let [bbuf (-> (java.nio.ByteBuffer/allocate 16) (.order java.nio.ByteOrder/LITTLE_ENDIAN)) bdata (.array bbuf)] (hamf-fn/obj->long v (do (.putLong bbuf (.getMostSignificantBits ^UUID v)) (.putLong bbuf (.getLeastSignificantBits ^UUID v)) (.position bbuf 0) (BlockSplitBloomFilter/hash bdata))))) (defn add-uuids! ^BlockSplitBloomFilter [bf val-seq] (let [^IFn$OL hasher (make-uuid-hasher)] (reduce (fn [bf v] (if (instance? UUID v) (insert-hash! bf (.invokePrim hasher v)) (throw (Exception. (str "Unsupported datatype: " (type v)))))) bf val-seq))) (defn make-uuid-pred [^BlockSplitBloomFilter bf] (let [hasher (make-uuid-hasher)] (fn [uuid] (.findHash bf (.invokePrim ^IFn$OL hasher uuid))))) (comment (def M (long 1e6)) (def uuids (vec (repeatedly M #(UUID/randomUUID)))) (def bf (bloom-filter M 0.01)) (reduce insert-hash! bf (map hash-obj uuids)) (def pred (make-pred bf)) (pred (uuids 0)) (pred (UUID/randomUUID)) (reduce (fn [eax _] (if (pred (UUID/randomUUID)) (inc eax) eax)) 0 (range M)) (reduce (fn [eax u] (if (pred u) (inc eax) eax)) 0 uuids) ) ================================================ FILE: src/ham_fisted/caffeine.clj ================================================ (ns ham-fisted.caffeine (:require [ham-fisted.function :as hamf-fn]) (:import [com.github.benmanes.caffeine.cache Caffeine LoadingCache CacheLoader Cache RemovalCause Weigher] [com.github.benmanes.caffeine.cache.stats CacheStats] [java.time Duration] [java.util Map])) (set! *warn-on-reflection* true) (set! *unchecked-math* :warn-on-boxed) (defn cache "Create a caffeine cache. Options: * `:write-ttl-ms` - Time that values should remain in the cache after write in milliseconds. * `:access-ttl-ms` - Time that values should remain in the cache after access in milliseconds. * `:soft-values?` - When true, the cache will store [SoftReferences](https://docs.oracle.com/javase/7/docs/api/java/lang/ref/SoftReference.html) to the data. * `:weak-values?` - When true, the cache will store [WeakReferences](https://docs.oracle.com/javase/7/docs/api/java/lang/ref/WeakReference.html) to the data. * `:max-size` - When set, the cache will behave like an LRU cache. * `:record-stats?` - When true, the LoadingCache will record access statistics. You can get those via the undocumented function memo-stats. * `:eviction-fn` - Function that receives 3 arguments, [args v cause], when a value is evicted. Causes the keywords `:collected :expired :explicit :replaced and :size`. See [caffeine documentation](https://www.javadoc.io/static/com.github.ben-manes.caffeine/caffeine/2.9.3/com/github/benmanes/caffeine/cache/RemovalCause.html) for cause definitions. * `:weight-limited` - tuple of `[weight-fn max-weight]`. weight-fn takes 2 args - k, v. * `:load-fn` - Pass in a function that will automatically compute the answer." ^Cache [{:keys [write-ttl-ms access-ttl-ms soft-values? weak-values? max-size record-stats? eviction-fn weight-limited load-fn]}] (let [^Caffeine builder (cond-> (Caffeine/newBuilder) access-ttl-ms (.expireAfterAccess (Duration/ofMillis access-ttl-ms)) write-ttl-ms (.expireAfterWrite (Duration/ofMillis write-ttl-ms)) soft-values? (.softValues) weak-values? (.weakValues) max-size (.maximumSize (long max-size)) record-stats? (.recordStats) eviction-fn (.evictionListener (reify com.github.benmanes.caffeine.cache.RemovalListener (onRemoval [this args v cause] (eviction-fn args v (condp identical? cause RemovalCause/COLLECTED :collected RemovalCause/EXPIRED :expired RemovalCause/EXPLICIT :explicit RemovalCause/REPLACED :replaced RemovalCause/SIZE :size (keyword (.toLowerCase (str cause))))))))) ^Caffeine builder (if weight-limited (let [[wfn weight] weight-limited] (-> (.maximumWeight builder (long weight)) (.weigher (reify Weigher (weigh [_ k v] (wfn k v)))))) builder)] (if load-fn (.build builder (proxy [CacheLoader] [] (load [k] (load-fn k)))) (.build builder)))) (defn get-if-present "" [^Cache c k] (.getIfPresent c k)) (defn get-or-load! [^Cache c k load-fn] (.get c k (hamf-fn/->function load-fn))) (defn invalidate! "Invalidate an entry. Returns the cache." [^Cache c k] (.invalidate c k) c) (defn invalidate-all! "Invalidate all entries or all entries with given keys. Returns cache." ([^Cache c] (.invalidateAll c) c) ([^Cache c ks] (.invalidateAll c (or ks '())) c)) (defn as-map "Return a caffeine cache as a map. Some Useful methods are .putIfAbsent and computeIfAbsent." ^Map [^Cache cache] (.asMap cache)) (defn stats "Return the caffeine cache stats." ^CacheStats [^Cache cache] (.stats cache)) (defn keyword-stats "Return a persistent map with keyword keys and caffeine stat values. Returns: `{:hit-count (.hitCount stats) :hit-rate (.hitRate stats) :miss-count (.missCount stats) :miss-rate (.missRate stats) :load-success-count (.loadSuccessCount stats) :average-load-penalty-nanos (.averageLoadPenalty stats) :total-load-time-nanos (.totalLoadTime stats) :eviction-count (.evictionCount stats)}`" [^Cache cache] (let [stats (stats cache)] {:hit-count (.hitCount stats) :hit-rate (.hitRate stats) :miss-count (.missCount stats) :miss-rate (.missRate stats) :load-success-count (.loadSuccessCount stats) :average-load-penalty-nanos (.averageLoadPenalty stats) :total-load-time-nanos (.totalLoadTime stats) :eviction-count (.evictionCount stats)})) ================================================ FILE: src/ham_fisted/datatypes.clj ================================================ (ns ham-fisted.datatypes (:require [ham-fisted.protocols :as protocols] [ham-fisted.defprotocol :as hamf-defproto] [ham-fisted.language :as hamf-language]) (:import [java.util Set HashSet])) ;; apply datatypes to all primitive types, array types (defn unsigned-type? [dt] (or (identical? :uint8 dt) (identical? :uint16 dt) (identical? :uint32 dt) (identical? :uint64 dt))) (defn integer-type? [dt] (or (identical? :int8 dt) (identical? :byte dt) (identical? :int16 dt) (identical? :short dt) (identical? :int32 dt) (identical? :int dt) (identical? :int64 dt) (identical? :long dt) (unsigned-type? dt))) (defn float-type? [dt] (or (identical? :float32 dt) (identical? :float64 dt))) (defn datatype->simplified-datatype [dt] (if (integer-type? dt) :int64 (if (float-type? dt) :float64 :object))) (defn extend-datatypes "dtype map is a map of class to datatype" [dtype-map] (->> dtype-map (run! (fn [kv] (hamf-defproto/extend (key kv) protocols/Datatype {:datatype (val kv) :simplified-datatype (datatype->simplified-datatype (val kv))}))))) (extend-datatypes {Byte/TYPE :int8 Byte :int8 Short/TYPE :int16 Short :int16 Integer/TYPE :int32 Integer :int32 Long/TYPE :int64 Long :int64 Float/TYPE :float32 Float :float32 Double/TYPE :float64 Double :float64 String :string clojure.lang.Keyword :keyword}) (extend-datatypes (into {} (map (fn [kv] [(val kv) :array])) hamf-language/array-classes)) (def java-keyword-type->datatype {:byte :int8 :short :int16 :int :int32 :integer :int32 :long :int64 :float :float32 :double :float64 :object :object}) (defn datatype->simplified-contained-datatype [dt] (when dt (datatype->simplified-datatype dt))) (defn extend-contained-datatypes [dtype-map] (->> dtype-map (run! (fn [kv] (hamf-defproto/extend (key kv) protocols/ContainedDatatype {:contained-datatype (val kv) :simplified-contained-datatype (datatype->simplified-contained-datatype (val kv))}))))) (extend-contained-datatypes {CharSequence :char Iterable :object java.util.Map :object java.util.stream.Stream :object java.util.stream.LongStream :int64 java.util.stream.IntStream :int32 java.util.stream.DoubleStream :float64}) (extend-contained-datatypes (into {} (map (fn [kv] [(val kv) (java-keyword-type->datatype (key kv))]) hamf-language/array-classes))) (defn generated-classes [argcount] (let [argcount (inc argcount) n-fns (long (Math/pow 3 argcount))] (->> (for [^long fn-idx (range n-fns)] (let [fn-vars (loop [arg-idx 0 fn-idx fn-idx data '()] (if (< arg-idx argcount) (let [local-idx (rem fn-idx 3) next-idx (quot fn-idx 3) data (conj data (case local-idx 0 "O" 1 "L" 2 "D"))] (recur (inc arg-idx) next-idx data)) data)) rval (last fn-vars)] (if (every? #(= % "O") fn-vars) [clojure.lang.IFn :object] [(Class/forName (apply str "clojure.lang.IFn$" fn-vars)) (cond (= "O" rval) :object (= "L" rval) :int64 (= "D" rval) :float64)])))))) (def fn-classes (into {clojure.lang.IFn :object} (mapcat generated-classes) (range 5))) (run! (fn [kv] (hamf-defproto/extend (key kv) protocols/ReturnedDatatype {:returned-datatype (val kv) :simplified-returned-datatype (datatype->simplified-datatype (val kv))})) fn-classes) ================================================ FILE: src/ham_fisted/defprotocol.clj ================================================ (ns ham-fisted.defprotocol "Alternative protocol implementation. Major features: * Allows subclasses to override only a subset of the methods and if the superclass has overridden the method then the superclasses implementation will be used. * Supports primitive typehints on function arguments and return values. * Much higher and more predictable multithreaded performance for protocol method invocation due to the fewer number of global variables that are read and written to for a single protocol method invocation. Does not write to global variables on a per-call basis meaning far less cpu/cache traffic in high contention scenarios. * Attempting to extend a protocol method that doesn't exist is an error at extension time. * Overriding the protocol for the base object array class overrides it for all things convertible to object array while still allowing the concrete array type to match a specific override. Another design decision is to avoid the interface check - this simplifes the hot path a slight bit at the cost of slightly slower calltimes in the case the interface is used. For those cases often it is possible to simply typehint the interface and call it directly avoiding any protocol dispatch overhead. Additional call overhead above and beyond a normal fn invocation in an arm mac is `-6ns` - the time for `.getClass` call into single concurrent hash map lookup." (:refer-clojure :exclude [defprotocol extend extend-type extend-protocol extends? satisfies? find-protocol-method find-protocol-impl extenders]) (:import [ham_fisted MethodImplCache Casts] [java.util Map])) (set! *warn-on-reflection* true) (defn find-protocol-cache-method [protocol ^MethodImplCache cache x] (when cache (let [cc (if (class? x) x (class x))] (if (.isAssignableFrom (.-iface cache) cc) (.-ifaceFn cache) (if-let [mfn (when (get protocol :extend-via-metadata) (get (meta x) (.-ns_methodk cache)))] mfn (.findFnFor cache cc)))))) (defn find-protocol-method "It may be more efficient in a tight loop to bypass the protocol dispatch on a per-call basis." ([protocol methodk x] (find-protocol-cache-method protocol @(get (get protocol :method-caches) methodk) x))) (defn- protocol? [maybe-p] (boolean (and (get maybe-p :on-interface) (get maybe-p :method-caches)))) (defn- implements? [protocol atype] (and atype (.isAssignableFrom ^Class (:on-interface protocol) atype))) (defn extends? "Returns true if atype extends protocol" [protocol atype] (boolean (or (implements? protocol atype) (get (:impls protocol) atype)))) (defn extenders "Returns a collection of the types explicitly extending protocol" [protocol] (keys (:impls protocol))) (defn satisfies? "Returns true if x satisfies the protocol" [protocol x] (or (instance? (get protocol :on-interface) x) (every? #(boolean (find-protocol-cache-method protocol % @x)) (vals (get protocol :method-caches))))) (defn- assert-same-protocol [protocol-var method-syms] (doseq [m method-syms] (let [v (resolve m) p (:protocol (meta v))] (when (and v (bound? v) (not= protocol-var p)) (binding [*out* *err*] (println "Warning: protocol" protocol-var "is overwriting" (if p (str "method " (.sym ^clojure.lang.Var v) " of protocol " (.sym ^clojure.lang.Var p)) (str "function " (.sym ^clojure.lang.Var v))))))))) (defn ^:no-doc find-fn [target ^MethodImplCache cache ns protocol] (let [rv (.findFnFor cache (class target))] (if-not (nil? rv) rv (throw (IllegalArgumentException. (format "No implementation of method: %s of protocol: #'%s/%s found for class: %s" (.-methodk cache) ns protocol (if-let [c (class target)] (.getName ^Class c) "nil"))))))) ;;Instance check is already taken care of (defn ^:no-doc find-fn-via-metadata [target ns-method cache ns protocol] (if-let [f (get (meta target) ns-method)] f (find-fn target cache ns protocol))) (defn ^:no-doc fn-tag-for-tags [arg-tags] (-> (apply str "clojure.lang.IFn$" (map (fn [arg-tag] (cond (= 'long arg-tag) "L" (= 'double arg-tag) "D" :else "O")) arg-tags)) symbol)) (defn- emit-protocol [name opts+sigs] (let [iname (symbol (str (munge (namespace-munge *ns*)) "." (munge name))) [opts sigs] (loop [opts {:on (list 'quote iname) :on-interface iname} sigs opts+sigs] (condp #(%1 %2) (first sigs) string? (recur (assoc opts :doc (first sigs)) (next sigs)) keyword? (recur (assoc opts (first sigs) (second sigs)) (nnext sigs)) [opts sigs])) sigs (when sigs (reduce (fn [m s] (let [tag-to-class (fn [tag] (if-let [c (and (instance? clojure.lang.Symbol tag) (= (.indexOf (.getName ^clojure.lang.Symbol tag) ".") -1) (not (contains? '#{int long float double char short byte boolean void ints longs floats doubles chars shorts bytes booleans objects} tag)) (resolve tag))] (symbol (.getName ^Class c)) tag)) name-meta (update-in (meta (first s)) [:tag] tag-to-class) mname (with-meta (first s) nil) [arglists doc] (loop [as [] rs (rest s)] (if (vector? (first rs)) (recur (conj as (first rs)) (next rs)) [(seq as) (first rs)])) name-kwd (keyword mname) mname (vary-meta mname assoc :doc doc :arglists arglists :tag (:tag name-meta))] (when (some #{0} (map count arglists)) (throw (IllegalArgumentException. (str "Definition of function " mname " in protocol " name " must take at least one arg.")))) (when (m (keyword mname)) (throw (IllegalArgumentException. (str "Function " mname " in protocol " name " was redefined. Specify all arities in single definition.")))) (assoc m (keyword mname) (merge name-meta {:name mname :methodk name-kwd :ns-methodk (keyword (clojure.core/name (.-name *ns*)) (clojure.core/name mname)) :arglists arglists :doc doc :cache-sym (symbol (str "-" mname "-cache")) :iface-sym (symbol (str "-" mname "-iface"))})))) {} sigs)) meths (mapcat (fn [sig] (let [m (munge (:name sig)) m-tag (or (:tag (meta (:name sig))) 'Object)] (map #(vector m (vec (repeat (dec (count %)) 'Object)) m-tag) (:arglists sig)))) (vals sigs)) opts (assoc opts :sigs sigs) name (if-let [proto-doc (:doc opts)] (with-meta name {:doc proto-doc}) name)] `(do (gen-interface :name ~iname :methods ~meths) ~@(mapcat (fn [{:keys [methodk ns-methodk cache-sym iface-sym arglists tag] mname :name}] [`(defn ~(with-meta iface-sym {:private true :tag (list 'quote tag)}) ~@(map (fn [args] (let [args (vec args) #_(mapv #(gensym (str %)) args) args (vary-meta (vec args) assoc :tag (if (class? tag) (list 'quote ) tag)) target (first args)] `(~args (. ~(with-meta target {:tag iname}) (~mname ~@(rest args)))))) arglists)) `(let [~'cache (ham_fisted.MethodImplCache. ~methodk ~ns-methodk ~iname ~iface-sym)] (def ~(with-meta cache-sym {:private true :tag 'ham_fisted.MethodImplCache}) ~'cache) (defn ~(vary-meta mname assoc :tag (list 'quote tag)) {:hamf-protocol ~(list 'quote name)} ~@(map (fn [args] (let [args (vary-meta (vec args) assoc :tag (if (class? tag) (list 'quote ) tag)) arg-tags (when (< (count args) 5) (conj (mapv (comp :tag meta) args) tag)) rval-tag (last arg-tags) invoker (when (first (filter #{'long 'double} arg-tags)) '.invokePrim) target (first args) find-data (if (:extend-via-metadata opts) `(find-fn-via-metadata ~target ~ns-methodk ~'cache ~(list 'quote (.-name *ns*)) ~(list 'quote name)) `(find-fn ~target ~'cache ~(list 'quote (.-name *ns*)) ~(list 'quote name)))] `(~args ~(if invoker `(let [~(with-meta 'ff {:tag (fn-tag-for-tags arg-tags)}) ~find-data] (~invoker ~'ff ~@args)) `(let [~'ff ~find-data] (~'ff ~@args)))))) arglists)))]) (vals sigs)) (def ~name ~(assoc (update opts :sigs (fn [sigmap] (->> sigmap (map (fn [e] [(key e) (-> (select-keys (val e) [:name :doc :arglists :tag]) (update :arglists (fn [arglists] (mapv (fn [arglist] (mapv (fn [sym] (list 'quote sym)) arglist)) arglists))) (update :tag (fn [t] (when t (list 'quote t)))) (update :name #(list 'quote %)))])) (into {})))) :method-caches (->> (map (fn [{:keys [methodk cache-sym]}] ;;Subtle issue here if you dereference the var -- ;;if the file is reloaded you can get protocol referenes that ;;point to the wrong cache var. [methodk (list 'var cache-sym)]) (vals sigs)) (into {})) ;;No more alter-var-root -- unnecessary :impls `(atom {})))))) (defmacro defprotocol "A protocol is a named set of named methods and their signatures: ```clojure (defprotocol AProtocolName ;optional doc string \"A doc string for AProtocol abstraction\" ;options :extend-via-metadata true ;method signatures (bar [this a b] \"bar docs\") (baz [this a] [this a b] [this a b c] \"baz docs\")) ``` No implementations are provided. Docs can be specified for the protocol overall and for each method. The above yields a set of polymorphic functions and a protocol object. All are namespace-qualified by the ns enclosing the definition The resulting functions dispatch on the type of their first argument, which is required and corresponds to the implicit target object ('this' in Java parlance). defprotocol is dynamic, has no special compile-time effect, and defines no new types or classes. Implementations of the protocol methods can be provided using extend. When :extend-via-metadata is true, values can extend protocols by adding metadata where keys are fully-qualified protocol function symbols and values are function implementations. Protocol implementations are checked first for direct definitions (defrecord, deftype, reify), then metadata definitions, then external extensions (extend, extend-type, extend-protocol) defprotocol will automatically generate a corresponding interface, with the same name as the protocol, i.e. given a protocol: my.ns/Protocol, an interface: my.ns.Protocol. The interface will have methods corresponding to the protocol functions, and the protocol will automatically work with instances of the interface. Note that you should not use this interface with deftype or reify, as they support the protocol directly: ```clojure (defprotocol P (foo [this]) (bar-me [this] [this y])) (deftype Foo [a b c] P (foo [this] a) (bar-me [this] b) (bar-me [this y] (+ c y))) (bar-me (Foo. 1 2 3) 42) => 45 (foo (let [x 42] (reify P (foo [this] 17) (bar-me [this] x) (bar-me [this y] x)))) => 17 ```" {:added "1.2"} [name & opts+sigs] (emit-protocol name opts+sigs)) (defn- correct-primitive-fn-type [arg-tags-v method] (doseq [arg-tags arg-tags-v] (when (first (filter #{'long 'double} arg-tags)) (let [prim-cls (Class/forName (apply str "clojure.lang.IFn$" (map (fn [tag] (cond (= tag 'long) "L" (= tag 'double) "D" :else 'O)) arg-tags)))] (when-not (.isAssignableFrom prim-cls (.getClass ^Object method)) (throw (RuntimeException. "Primitive hinted protocol methods must have primitive hinted implementations!")))))) method) (defn- clear-object-tag [tag] (cond (= tag 'long) 'long (= tag 'double) 'double)) (defn- check-constant-return [tag arglists] (when-not (and (== 1 (count arglists)) (= [nil (clear-object-tag tag)] (mapv clear-object-tag (first arglists)))) (throw (IllegalArgumentException. (str "Primitive constants only work with object->constant translations: " arglists))))) (defn extend "Implementations of protocol methods can be provided using the extend construct: ```clojure (extend AType AProtocol {:foo an-existing-fn :bar (fn [a b] ...) :baz (fn ([a]...) ([a b] ...)...)} BProtocol {...} ...) ``` extend takes a type/class (or interface, see below), and one or more protocol + method map pairs. It will extend the polymorphism of the protocol's methods to call the supplied methods when an AType is provided as the first argument. Method maps are maps of the keyword-ized method names to ordinary fns. This facilitates easy reuse of existing fns and fn maps, for code reuse/mixins without derivation or composition. You can extend an interface to a protocol. This is primarily to facilitate interop with the host (e.g. Java) but opens the door to incidental multiple inheritance of implementation since a class can inherit from more than one interface, both of which extend the protocol. It is TBD how to specify which impl to use. You can extend a protocol on nil. If you are supplying the definitions explicitly (i.e. not reusing exsting functions or mixin maps), you may find it more convenient to use the extend-type or extend-protocol macros. Note that multiple independent extend clauses can exist for the same type, not all protocols need be defined in a single extend call. See also: extends?, satisfies?, extenders" {:added "1.2"} [atype & proto+mmaps] (doseq [[proto mmap] (partition 2 proto+mmaps)] (when-not (protocol? proto) (throw (IllegalArgumentException. (str proto " is not a protocol")))) (when (implements? proto atype) (throw (IllegalArgumentException. (str atype " already directly implements " (:on-interface proto))))) (let [impls (:impls proto) method-caches (:method-caches proto)] (swap! impls assoc atype mmap) (->> mmap (run! (fn [kv] (let [methodk (key kv) method (val kv) {:keys [tag arglists]} (get-in proto [:sigs methodk]) _ (when-not arglists (throw (IllegalArgumentException. (str "method not found: " methodk " in protocol " (:on proto) " - protocol methods: " (pr-str (.keySet ^Map (get proto :sigs))))))) arg-tags (mapv #(conj (mapv (comp :tag meta) %) tag) arglists) method (cond (and (= 'long tag) (number? method)) (let [ll (Casts/longCast method)] (check-constant-return tag arg-tags) (fn ^long [o] ll)) (and (= 'double tag) (number? method)) (let [ll (Casts/doubleCast method)] (check-constant-return tag arg-tags) (fn ^double [o] ll)) (fn? method) method :else (do (check-constant-return tag arg-tags) (fn [o] method))) method (correct-primitive-fn-type arg-tags method)] (.extend ^MethodImplCache @(get method-caches methodk) atype method)))))))) (defn- normalize-specs [specs] (if (vector? (first specs)) (list specs) specs)) (defn- emit-fn-map [p hint fs] (let [proto (resolve p) pcol (when proto (deref proto)) _ (when-not (protocol? pcol) (throw (IllegalArgumentException. (str p " is not a hamf protocol")))) sigs (get pcol :sigs) define-fn (fn [fn-entry] (let [fn-name (-> fn-entry first name keyword) specs (hint (normalize-specs (drop 1 fn-entry))) {:keys [tag arglists] :as sig} (get sigs fn-name) arglist-map (into {} (map (fn [arglist] [(count arglist) (mapv (comp :tag meta) arglist)])) arglists) apply-primitive-typehints (fn [[[target & args :as arglist] & body]] (let [args (->> (map (fn [tag arg-sym] (vary-meta arg-sym update :tag #(or % tag))) (drop 1 (arglist-map (inc (count args)))) args) (into [target]))] (cons (vary-meta args update :tag #(or % tag)) body)))] [fn-name (cons 'fn (map apply-primitive-typehints specs))]))] (into {} (map define-fn) fs))) (defn- emit-impl [[p fs]] [p (emit-fn-map p identity fs)]) (defn- emit-hinted-impl [c [p fs]] (let [typehint-first-arg #(->> % (map (fn [[[target & args :as arglist] & body]] (cons (into [(vary-meta target assoc :tag c)] args) body))))] [p (emit-fn-map p typehint-first-arg fs)])) (defn- parse-impls [specs] (loop [ret {} s specs] (if (seq s) (recur (assoc ret (first s) (take-while seq? (next s))) (drop-while seq? (next s))) ret))) (defn- emit-extend-type [c specs] (let [impls (parse-impls specs)] `(extend ~c ~@(mapcat (partial emit-hinted-impl c) impls)))) (defmacro extend-type "A macro that expands into an extend call. Useful when you are supplying the definitions explicitly inline, extend-type automatically creates the maps required by extend. Propagates the class as a type hint on the first argument of all fns. ```clojure (extend-type MyType Countable (cnt [c] ...) Foo (bar [x y] ...) (baz ([x] ...) ([x y & zs] ...))) ``` expands into: ```clojure (extend MyType Countable {:cnt (fn [c] ...)} Foo {:baz (fn ([x] ...) ([x y & zs] ...)) :bar (fn [x y] ...)}) ```" {:added "1.2"} [t & specs] (emit-extend-type t specs)) (defn- emit-extend-protocol [p specs] (let [impls (parse-impls specs)] `(do ~@(map (fn [[t fs]] `(extend-type ~t ~p ~@fs)) impls)))) (defmacro extend-protocol "Useful when you want to provide several implementations of the same protocol all at once. Takes a single protocol and the implementation of that protocol for one or more types. Expands into calls to extend-type: ```clojure (extend-protocol Protocol AType (foo [x] ...) (bar [x y] ...) BType (foo [x] ...) (bar [x y] ...) AClass (foo [x] ...) (bar [x y] ...) nil (foo [x] ...) (bar [x y] ...)) ``` expands into: ```clojure (do (clojure.core/extend-type AType Protocol (foo [x] ...) (bar [x y] ...)) (clojure.core/extend-type BType Protocol (foo [x] ...) (bar [x y] ...)) (clojure.core/extend-type AClass Protocol (foo [x] ...) (bar [x y] ...)) (clojure.core/extend-type nil Protocol (foo [x] ...) (bar [x y] ...))) ```" {:added "1.2"} [p & specs] (emit-extend-protocol p specs)) ================================================ FILE: src/ham_fisted/fjp.clj ================================================ (ns ham-fisted.fjp "Support for java.util.concurrent.ForkJoinPool-specific operations such as managed block and task fork/join. Additionally supports concept of 'exception-safe' via wrapping executing code and unwrapper result post execution in order avoid wrapping exceptions and breaking calling code that may be expecting specific exception types. Some of the api's fall back to regular executor service code when called. Example: ```clojure (defn split-parallel-reduce \"Perform a parallel reduction of a spliterator using the provided ExecutorService\" [executor-service split ideal-split init-fn rfn merge-fn] (let [n-elems (proto/estimate-count split) pool (or executor-service (ForkJoinPool/commonPool))] (if (or (<= n-elems (long ideal-split)) (= n-elems Long/MAX_VALUE)) (split-reduce rfn (init-fn) split) (if-let [[lhs rhs] (proto/split split)] (let [lt (fjp/safe-fork-task pool (split-parallel-reduce pool lhs ideal-split init-fn rfn merge-fn)) rt (fjp/safe-fork-task pool (split-parallel-reduce pool rhs ideal-split init-fn rfn merge-fn))] (merge-fn (fjp/managed-block-unwrap lt) (fjp/managed-block-unwrap rt))))))) ```" (:require [ham-fisted.language :refer [cond not]] [ham-fisted.protocols :as proto] [ham-fisted.defprotocol :refer [extend-type extend-protocol]]) (:import [java.util.concurrent ForkJoinPool ForkJoinTask ForkJoinPool$ManagedBlocker RecursiveTask Future ExecutorService] [clojure.lang IDeref] [ham_fisted FJTask]) (:refer-clojure :exclude [cond not extend-type extend-protocol])) (set! *warn-on-reflection* true) (set! *unchecked-math* :warn-on-boxed) (defn common-pool "Returns (ForkJoinPool/commonPool)" ^ForkJoinPool [] (ForkJoinPool/commonPool)) (defn common-pool-parallelism "Integer parallelism assigned to the common pool " ^long [] (ForkJoinPool/getCommonPoolParallelism)) (defn in-fork-join-pool? "Returns true if this task is executing in a fork join pool thread" [] (ForkJoinTask/inForkJoinPool)) (extend-protocol proto/ManagedBlocker ForkJoinTask (managed-blocker [m] (reify ForkJoinPool$ManagedBlocker (block [_] (.quietlyComplete m) true) (isReleasable [_] (.isDone m)) clojure.lang.IDeref (deref [_] (.get m)))) Future (managed-blocker [m] (reify ForkJoinPool$ManagedBlocker (block [_] (.get m) true) (isReleasable [_] (.isDone m)) clojure.lang.IDeref (deref [_] (.get m)))) clojure.lang.IPending (managed-blocker [m] (reify ForkJoinPool$ManagedBlocker (block [_] (deref m) true) (isReleasable [_] (realized? m)) clojure.lang.IDeref (deref [_] (deref m)))) ForkJoinPool$ManagedBlocker (managed-blocker [m] m)) (defn managed-block "Block on a delay or future using the fjp system's managed blocking facility. Safe to call all the time whether the current system is in a forkjoinpool task or not." ([dly] (if (instance? ForkJoinTask dly) (.join ^ForkJoinTask dly) (let [dly (proto/managed-blocker dly)] (ForkJoinPool/managedBlock dly) @dly))) ([finished? wait-till-finished get-value] (managed-block (reify ForkJoinPool$ManagedBlocker (block [this] (wait-till-finished)) (isReleasable [this] (finished?)) clojure.lang.IDeref (deref [this] (get-value)))))) (defn task "Create a task from a clojure IFn or something that implements IDeref" ^FJTask [f] (FJTask. f)) (defn fork-task "Begin a separate execution for f. If already in a fork join pool fork the task else submit f to passed in pool." [pool f] (if (in-fork-join-pool?) (let [t (task f)] (.fork t) t) (.submit ^ExecutorService pool ^Callable f))) (defmacro exception-safe "Wrap code in an exception-safe wrapper - returns a map with either `:ham-fisted.fjp/result` or `:ham-fisted.fjp.error`." [& code] `(let [~(with-meta 'ffn {:tag 'Callable}) (^:once fn* [] (try {:ham-fisted.fjp/result (do ~@code)} (catch Throwable e# {:ham-fisted.fjp/error e#})))] ~'ffn)) (defmacro safe-fork-task "Called from within an executing task, fork a executing some code and wrapping it in [[exception-safe]] then calling [[fork-task]]" [pool & code] `(->> (do ~@code) (exception-safe) (fork-task ~pool))) (defn join "join a previously forked task returning the result" [^ForkJoinTask t] (.join t)) (defn compute "compute a task in current thread - returns result" [t] (if (instance? FJTask t) (.deref (.-c ^FJTask t)) (throw (RuntimeException. "Only recursive tasks (or tasks create via \"task\" can be computed")))) (defn unsafe-common-pool "Run a callable on the common pool. If the callable throws you will get a wrapped exception thrown which may confuse calling code - specifically code that relies on exact exception types or ex-info." [code] (-> (.submit (ForkJoinPool/commonPool) ^Callable code) (deref))) (defn unwrap-safe "Unwrap result created via executing code wrapped in [[exception-safe]]. Throws original exception if found." [m] (let [rv (get m ::result ::failure)] (if (identical? rv ::failure) (throw (get m ::error)) rv))) (defn safe-common-pool "Run safe code - see [[exception-safe]] unwrapping the result and re-throwing the wrapped exception. This allows systems based on typed exceptions to pass error info." [safe-code] (unwrap-safe (unsafe-common-pool safe-code))) (defmacro on-cp "Run arbitrary code on the common-pool. Make sure any blocking operations are wrapped in [[managed-block]]." [& code] `(safe-common-pool (exception-safe ~@code))) (defn managed-block-unwrap "managed block then safe unwrap the exception-safe result" [dly] (managed-block dly) (unwrap-safe @dly)) ================================================ FILE: src/ham_fisted/function.clj ================================================ (ns ham-fisted.function "Helpers for working with [java.util.function](https://docs.oracle.com/javase/8/docs/api/java/util/function/package-summary.html) package objects." (:import [ham_fisted IFnDef$ODO IFnDef$OOO IFnDef$OO IFnDef$OL IFnDef$LongPredicate IFnDef$DoublePredicate IFnDef$DD IFnDef$LL IFnDef IFnDef$LD IFnDef$DL IFnDef$OD IFnDef$LO Casts IFnDef$Predicate IFnDef$LLL IFnDef$DDD] [java.util.function BiFunction BiConsumer Function DoublePredicate LongPredicate Predicate Consumer LongConsumer DoubleConsumer LongBinaryOperator DoubleBinaryOperator BiPredicate] [java.util Comparator] [clojure.lang Util] [it.unimi.dsi.fastutil.longs LongComparator] [it.unimi.dsi.fastutil.doubles DoubleComparator])) (defmacro bi-function "Create an implementation of java.util.function.BiFunction." [arg1 arg2 & code] `(reify IFnDef$OOO (invoke [this# ~arg1 ~arg2] ~@code))) (defmacro bi-consumer [arg1 arg2 & code] `(reify BiConsumer (accept [this# ~arg1 ~arg2] ~@code))) (defn ->bi-function "Convert an object to a java.util.BiFunction. Object can either already be a bi-function or an IFn to be invoked with 2 arguments." ^BiFunction [cljfn] (if (instance? BiFunction cljfn) cljfn (bi-function a b (cljfn a b)))) (defmacro function "Create a java.util.function.Function" [arg & code] `(reify IFnDef$OO (invoke [this# ~arg] ~@code))) (defmacro obj->long "Create a function that converts objects to longs" ([] `(reify IFnDef$OL (invokePrim [this# v#] (Casts/longCast v#)))) ([varname & code] `(reify IFnDef$OL (invokePrim [this ~varname] (Casts/longCast ~@code))))) (defmacro obj->double "Create a function that converts objects to doubles" ([] `(reify IFnDef$OD (invokePrim [this# v#] (Casts/doubleCast v#)))) ([varname & code] `(reify IFnDef$OD (invokePrim [this# ~varname] (Casts/doubleCast ~@code))))) (defmacro long->double "Create a function that receives a long and returns a double" [varname & code] `(reify IFnDef$LD (invokePrim [this# ~varname] ~@code))) (defmacro double->long "Create a function that receives a double and returns a long" [varname & code] `(reify IFnDef$DL (invokePrim [this# ~varname] ~@code))) (defmacro long->obj "Create a function that receives a primitive long and returns an object." [varname & code] `(reify IFnDef$LO (invokePrim [this# ~varname] ~@code))) (defmacro double->obj [varname & code] `(reify IFnDef$DO (invokePrim [this# ~varname] ~@code))) (defn ->function "Convert an object to a java Function. Object can either already be a Function or an IFn to be invoked." ^Function [cljfn] (if (instance? Function cljfn) cljfn (reify IFnDef$OO (invoke [this a] (cljfn a))))) (defmacro double-predicate "Create an implementation of java.util.Function.DoublePredicate" [varname & code] `(reify IFnDef$DoublePredicate (test [this# ~varname] ~@code))) (defmacro double-unary-operator "Create an implementation of java.util.function.DoubleUnaryOperator" [varname & code] `(reify IFnDef$DD (invokePrim [this# ~varname] ~@code))) (defmacro long-predicate "Create an implementation of java.util.Function.LongPredicate" [varname & code] `(reify IFnDef$LongPredicate (test [this# ~varname] ~@code))) (defn ->long-predicate ^LongPredicate [f] (if (instance? LongPredicate f) f (long-predicate ll (boolean (f ll))))) (defmacro long-unary-operator "Create an implementation of java.util.function.LongUnaryOperator" [varname & code] `(reify IFnDef$LL (invokePrim [this# ~varname] ~@code))) (defmacro predicate "Create an implementation of java.util.Function.Predicate" [varname & code] `(reify IFnDef$Predicate (test [this# ~varname] ~@code))) (defmacro unary-operator "Create an implementation of java.util.function.UnaryOperator" [varname & code] `(function ~varname ~@code)) (defmacro binary-operator "Create an implementation of java.util.function.BinaryOperator" [arg1 arg2 & code] `(bi-function ~arg1 ~arg2 ~@code)) (defmacro double-consumer "Create an instance of a java.util.function.DoubleConsumer" [varname & code] `(reify DoubleConsumer (accept [this# ~varname] ~@code) IFnDef$DO (invokePrim [this# v#] (.accept this# v#)))) (defmacro long-consumer "Create an instance of a java.util.function.LongConsumer" [varname & code] `(reify LongConsumer (accept [this# ~varname] ~@code) IFnDef$LO (invokePrim [this# v#] (.accept this# v#)))) (defmacro consumer "Create an instance of a java.util.function.Consumer" [varname & code] `(reify Consumer (accept [this# ~varname] ~@code) IFnDef$OO (invoke [this# arg#] (.accept this# arg#)))) (defmacro make-long-comparator "Make a comparator that gets passed two long arguments." [lhsvar rhsvar & code] (let [lhsvar (with-meta lhsvar {:tag 'long}) rhsvar (with-meta rhsvar {:tag 'long}) compsym (with-meta 'compare {:tag 'int})] `(reify LongComparator (~compsym [this# ~lhsvar ~rhsvar] ~@code) IFnDef (invoke [this# l# r#] (.compare this# l# r#))))) (defmacro make-double-comparator "Make a comparator that gets passed two double arguments." [lhsvar rhsvar & code] (let [lhsvar (with-meta lhsvar {:tag 'double}) rhsvar (with-meta rhsvar {:tag 'double}) compsym (with-meta 'compare {:tag 'int})] `(reify DoubleComparator (~compsym [this# ~lhsvar ~rhsvar] ~@code) IFnDef (invoke [this# l# r#] (.compare this# l# r#))))) (defmacro make-comparator "Make a java comparator." [lhsvar rhsvar & code] `(reify Comparator (compare [this# ~lhsvar ~rhsvar] ~@code) IFnDef (invoke [this# l# r#] (.compare this# l# r#)))) (def ^{:doc "A reverse comparator that sorts in descending order" } rcomp (reify Comparator (^int compare [this ^Object l ^Object r] (Util/compare r l)) DoubleComparator (^int compare [this ^double l ^double r] (Double/compare r l)) LongComparator (^int compare [this ^long l ^long r] (Long/compare r l)) IFnDef (invoke [this l r] (.compare this l r)))) (def ^{:doc "A comparator that sorts null, NAN first, natural order"} comp-nan-first (reify Comparator (^int compare [this ^Object l ^Object r] (cond (nil? l) -1 (nil? r) 1 :else (Util/compare l r))) DoubleComparator (^int compare [this ^double l ^double r] (cond (Double/isNaN l) -1 (Double/isNaN r) 1 :else (Double/compare l r))) LongComparator (^int compare [this ^long l ^long r] (Long/compare l r)) IFnDef (invoke [this l r] (.compare this l r)))) (def ^{:doc "A comparator that sorts null, NAN last, natural order"} comp-nan-last (reify Comparator (^int compare [this ^Object l ^Object r] (cond (nil? l) 1 (nil? r) -1 :else (clojure.lang.Util/compare l r))) DoubleComparator (^int compare [this ^double l ^double r] (cond (Double/isNaN l) 1 (Double/isNaN r) -1 :else (Double/compare l r))) LongComparator (^int compare [this ^long l ^long r] (Long/compare l r)) IFnDef (invoke [this l r] (.compare this l r)))) (defmacro double-binary-operator "Create a binary operator that is specialized for double values. Useful to speed up operations such as sorting or summation." [lvar rvar & code] `(reify DoubleBinaryOperator (applyAsDouble [this# ~lvar ~rvar] ~@code) IFnDef$DDD (invokePrim [this# l# r#] (.applyAsDouble this# l# r#)))) (defmacro long-binary-operator "Create a binary operator that is specialized for long values. Useful to speed up operations such as sorting or summation." [lvar rvar & code] `(reify LongBinaryOperator (applyAsLong [this# ~lvar ~rvar] ~@code) IFnDef$LLL (invokePrim [this# l# r#] (.applyAsLong this# l# r#)))) (defmacro binary-predicate "Create a java.util.function.BiPredicate" [xvar yvar code] `(reify BiPredicate (test [this# ~xvar ~yvar] (boolean ~code)) IFnDef (invoke [this# ~xvar ~yvar] (boolean ~code)))) (defn binary-predicate-or-null "If f is null, return null. Else return f as a java.util.function.BiPredicate." ^BiPredicate [f] (when f (if (instance? BiPredicate f) f (binary-predicate x y (f x y))))) ================================================ FILE: src/ham_fisted/hlet.clj ================================================ (ns ham-fisted.hlet "Extensible let to allow efficient typed destructuring. Registered Extensions: `dbls` and `lngs` will most efficiently destructure java primitive arrays and fall back to casting the result of clojure.lang.RT/nth if input is not a double or long array. `dlb-fns` and `lng-fns` call the object's IFn interface with no interface checking. This will *not* work with a raw array but is the fastest way - faster than RT/nth - to get data out of a persistent-vector or map like object. `obj-fns` - fast IFn-based destructuring to objects - does not work with object arrays. Often much faster than RT/nth. This can significantly reduce boxing in tight loops without needing to result in really verbose pathways. ```clojure user> (h/let [[a b] (dbls [1 2])] (+ a b)) 3.0 user> (hamf/sum-fast (lznc/cartesian-map #(h/let [[a b c d](lng-fns %)] (-> (+ a b) (+ c) (+ d))) [1 2 3] [4 5 6] [7 8 9] [10 11 12 13 14])) 3645.0 ``` See also [[ham-fisted.primitive-invoke]], [[ham-fisted.api/dnth]] [[ham-fisted.api/lnth]]." (:require [ham-fisted.api :as hamf] [ham-fisted.lazy-noncaching :as lznc] [ham-fisted.reduce :as hamf-rf]) (:import [java.util List] [ham_fisted Casts]) (:refer-clojure :exclude [let])) (def ^{:private true :tag java.util.Map} extension-table (hamf/java-concurrent-hashmap)) (defn extend-let "Code gets a tuple of [lhs rhs] must return a flattened sequence of left and right hand sides. This uses a special symbol that will look like a function call on the right hand side as the dispatch mechanism. See source code of this file for example extensions." [sym-name code] (.put extension-table sym-name code)) (defn- rhs-code [r] (when (list? r) (clojure.core/let [s (first r)] (when (symbol? s) (.get extension-table s))))) (defn- add-all! [a bs] (.addAll ^List a bs) a) (defn- add! ([a b] (.add ^List a b) a) ([a b c] (.add ^List a b) (.add ^List a c) a)) (defn- process-bindings [bindings] (-> (reduce (fn [acc pair] (if-let [code (rhs-code (pair 1))] (clojure.core/let [new-pairs (code pair)] (when-not (== 0 (rem (count new-pairs) 2)) (throw (RuntimeException. (str "Code for symbol " (first (pair 1)) " returned uneven number of results")))) (add-all! acc new-pairs)) (add-all! acc pair))) (hamf/mut-list) (lznc/partition-all 2 (vec bindings))) (persistent!))) (defmacro let "Extensible let intended to allow typed destructuring of arbitrary datatypes such as primitive arrays or point types. Falls back to normal let after extension process. Several extensions are registered by default - * `dbls` and `lngs` which destructure into primitive doubles and primitive longs, respectively. * `dlb-fns` and `lng-fns` which destructure into primitive doubls and longs but use the often faster IFn overloads to get the data - avoiding RT.nth calls. * `obj-fns` which destructure into objects using the IFn interface. ```clojure user> (h/let [[x y] (dbls (hamf/double-array [1 2]))] (+ x y)) 3.0 ``` " [bindings & body] (when-not (== 0 (rem (count bindings) 2)) (throw (RuntimeException. "Bindings must be divisible by 2"))) `(clojure.core/let ~(process-bindings bindings) ~@body)) (defn ^:no-doc typed-destructure [code scalar-fn nth-fn] (clojure.core/let [lvec (code 0) rdata (second (code 1))] (if (vector? lvec) (let [rtemp (if (symbol? rdata) rdata (gensym "__tmp"))] (-> (reduce (hamf-rf/indexed-accum acc idx lv-entry (nth-fn rtemp acc idx lv-entry)) (hamf/mut-list (if (identical? rtemp rdata) nil [rtemp rdata])) lvec) (persistent!))) (scalar-fn lvec rdata)))) (defn ^:no-doc typed-nth-destructure [nth-symbol scalar-cast code] (typed-destructure code (fn [lvec rdata] [lvec `(~scalar-cast ~rdata)]) (fn [rtemp acc ^long idx lv-entry] (add! acc lv-entry `(~nth-symbol ~rtemp ~idx))))) (extend-let 'dbls #(typed-nth-destructure 'ham-fisted.api/dnth 'ham_fisted.Casts/doubleCast %)) (extend-let 'lngs #(typed-nth-destructure 'ham-fisted.api/lnth 'ham_fisted.Casts/longCast %)) (defn ^:no-doc typed-fn-destructure [scalar-cast code] (typed-destructure code (fn [lvec rdata] [lvec `(~scalar-cast ~rdata)]) (fn [rtemp acc ^long idx lv-entry] (add! acc lv-entry `(~scalar-cast (~rtemp ~idx)))))) (extend-let 'lng-fns #(typed-fn-destructure 'ham_fisted.Casts/longCast %)) (extend-let 'dbl-fns #(typed-fn-destructure 'ham_fisted.Casts/doubleCast %)) (extend-let 'obj-fns #(typed-destructure % (fn [lvec rdata] [lvec `~rdata]) (fn [rtemp acc ^long idx lv-entry] (add! acc lv-entry `(~rtemp ~idx))))) (defn let-extension-names "Return the current extension names." [] (keys extension-table)) ================================================ FILE: src/ham_fisted/impl.clj ================================================ (ns ham-fisted.impl (:require [ham-fisted.lazy-noncaching :refer [map concat] :as lznc] [ham-fisted.protocols :as protocols] [clojure.core.protocols :as cl-proto] [ham-fisted.language :refer [constantly]] [ham-fisted.iterator :as hamf-iter] [ham-fisted.defprotocol :refer [extend extend-type extend-protocol]] [ham-fisted.print :refer [implement-tostring-print]]) (:import [java.util.concurrent ForkJoinPool ForkJoinTask ArrayBlockingQueue Future TimeUnit ConcurrentHashMap] [clojure.lang MapEntry Box] [java.util Iterator Set Map RandomAccess Spliterator BitSet Collection Iterator ArrayDeque] [java.util.function Supplier] [ham_fisted ParallelOptions ITypedReduce IFnDef ICollectionDef ArrayLists$ObjectArrayList Reductions$ReduceConsumer Reductions Transformables IFnDef$OLO ArrayLists StringCollection] [clojure.lang IteratorSeq IReduceInit PersistentHashMap IFn$OLO IFn$ODO Seqable IReduce PersistentList] [java.util.logging Logger Level]) (:refer-clojure :exclude [map pmap concat extend extend-type extend-protocol constantly])) (set! *warn-on-reflection* true) (set! *unchecked-math* :warn-on-boxed) (defn in-fork-join-task? "True if you are currently running in a fork-join task" [] (ForkJoinTask/inForkJoinPool)) (defrecord ^:private GroupData [^long gsize ^long ngroups ^long leftover]) (defn- n-elems->groups "This algorithm goes somewhat over batch size but attempts to all the indexes evenly between tasks." ^GroupData [^long parallelism ^long n-elems ^long max-batch-size] (let [gsize (max 1 (min max-batch-size (quot n-elems parallelism))) ngroups (quot n-elems gsize) leftover (rem n-elems gsize) left-blocks (quot leftover ngroups) gsize (+ gsize left-blocks) leftover (rem leftover ngroups)] (GroupData. gsize ngroups leftover))) (defn- seq->lookahead [^long n-ahead data] ;;Ensure lazy-caching (let [data (seq data) n-ahead (count (take n-ahead data))] [data (concat (drop n-ahead data) (repeat n-ahead nil))])) (defn n-lookahead ^long [^ParallelOptions options] (max 1 (long (let [n-ahead (.nLookahead options)] (if (== n-ahead -1) (* (.-parallelism options) 2) n-ahead))))) (defn- pgroup-submission [^long n-elems body-fn ^ParallelOptions options] (let [pool (.-pool options) n-elems (long n-elems) parallelism (.-parallelism options) ^GroupData gdata (n-elems->groups parallelism n-elems (.-maxBatchSize options)) ;;_ (println parallelism gdata) gsize (.gsize gdata) ngroups (.ngroups gdata) leftover (.leftover gdata)] (->> (range ngroups) (map (fn [^long gidx] (let [start-idx (long (+ (* gidx gsize) (min gidx leftover))) end-idx (min n-elems (long (+ (+ start-idx gsize) (long (if (< gidx leftover) 1 0)))))] (.submit pool ^Callable #(body-fn start-idx end-idx))))) (seq->lookahead (n-lookahead options))))) (defrecord ^:private ErrorRecord [e]) (defn- queue-take [^ArrayBlockingQueue queue] (let [^Box b (.take queue) v (.-val b)] (if (instance? ErrorRecord v) (throw (.-e ^ErrorRecord v)) v))) (defmacro queue-put! [queue v put-timeout-ms] ;;You cannot put nil into a queue `(let [data# (-> (try ~v (catch Exception e# (ErrorRecord. e#))) (Box.))] (when-not (.offer ~queue data# ~put-timeout-ms TimeUnit/MILLISECONDS) (let [msg# (str ":put-timeout-ms " ~put-timeout-ms "ms exceeded")] (.warning (Logger/getLogger "ham_fisted") msg#) (throw (RuntimeException. msg#)))) data#)) (defn- iter-queue->seq [^Iterator iter ^ArrayBlockingQueue queue] ;;We are taking advantage of the fact that iteratorSeq locks the iterator so all of the ;;code below is threadsafe once it leaves this function. (IteratorSeq/create (reify Iterator (hasNext [this] (.hasNext iter)) (next [this] (.next iter) (queue-take queue))))) (defn- options->queue ^ArrayBlockingQueue [^ParallelOptions options] (when-not (.-ordered options) (ArrayBlockingQueue. (n-lookahead options)))) (defn pgroups "Run y groups across n-elems. Y is common pool parallelism. body-fn gets passed two longs, startidx and endidx. Returns a sequence of the results of body-fn applied to each group of indexes. Options: * - See the ParallelOptions java object." ([n-elems body-fn ^ParallelOptions options] (let [parallelism (.-parallelism options) n-elems (long n-elems)] (if (or (in-fork-join-task?) (< n-elems (.-minN options)) (< parallelism 2)) [(body-fn 0 n-elems)] (let [queue (options->queue options) wrapped-body-fn (if (.-ordered options) body-fn (fn [^long sidx ^long eidx] (queue-put! queue (body-fn sidx eidx) (.-putTimeoutMs options)))) [submissions lookahead] (pgroup-submission n-elems wrapped-body-fn options)] (if (.-ordered options) ;;This will correctly propagate errors (map (fn [l r] (.get ^Future l)) submissions lookahead) (iter-queue->seq (.iterator ^Iterable lookahead) queue)))))) ([n-elems body-fn] (pgroups n-elems body-fn nil))) (defn- lookahead-iterable ^Iterable [^Iterator submissions ^long n-ahead deref?] (let [queue (ArrayDeque. n-ahead) ;;in the pmap context this immediately launches all lookahead threads ;;and does no further work. _ (loop [idx 0] (when (and (.hasNext submissions) (< idx n-ahead)) (.push queue (.next submissions)) (recur (inc idx)))) update! (fn [] (let [nn (if-not (.isEmpty queue) (.removeLast queue) ::finish)] (if (.hasNext submissions) (.push queue (.next submissions)) (.push queue ::finish)) (if (identical? nn ::finish) nn (if deref? (.get ^Future nn) nn))))] (-> (hamf-iter/once-iterable #(not (identical? % ::finish)) update!) (hamf-iter/seq-iterable)))) (defn pmap [^ParallelOptions options map-fn sequences] (if (in-fork-join-task?) (apply map map-fn sequences) (let [pool (.-pool options) ;;In this case we want a caching sequence - so we call 'seq' on a lazy noncaching ;;map queue (options->queue options) submit-fn (if (.-ordered options) (case (count sequences) 1 (fn [arg] (.submit pool ^Callable (fn [] (map-fn arg)))) 2 (fn [a1 a2] (.submit pool ^Callable (fn [] (map-fn a1 a2)))) 3 (fn [a1 a2 a3] (.submit pool ^Callable (fn [] (map-fn a1 a2 a3)))) (fn [& args] (.submit pool ^Callable (fn [] (apply map-fn args))))) (case (count sequences) 1 (fn [arg] (.submit pool ^Callable (fn [] (queue-put! queue (map-fn arg) (.-putTimeoutMs options))))) 2 (fn [a1 a2] (.submit pool ^Callable (fn [] (queue-put! queue (map-fn a1 a2) (.-putTimeoutMs options))))) 3 (fn [a1 a2 a3] (.submit pool ^Callable (fn [] (queue-put! queue (map-fn a1 a2 a3) (.-putTimeoutMs options))))) (fn [& args] (.submit pool ^Callable (fn [] (queue-put! queue (apply map-fn args) (.-putTimeoutMs options))))))) submissions (.iterator ^Iterable (apply map submit-fn sequences)) n-ahead (n-lookahead options)] (if (.-ordered options) ;;lazy noncaching map - the future itself does the caching for us. (lookahead-iterable submissions n-ahead true) (iter-queue->seq (.iterator ^Iterable (lookahead-iterable submissions n-ahead false)) queue))))) (defn- split-spliterator [^long n-splits ^Spliterator spliterator] (let [lhs spliterator ;;Mutates lhs... rhs (.trySplit spliterator)] (cond ;;Sometimes the splits fail as we have too few elements (nil? rhs) [lhs] ;;Here we have split enough (<= n-splits 1) [lhs rhs] :else (let [n-splits (dec n-splits)] (concat (split-spliterator n-splits lhs) (split-spliterator n-splits rhs)))))) (defn- consume-spliterator! [rfn acc ^Spliterator s] (if (instance? IReduceInit s) (.reduce ^IReduceInit s rfn acc) (let [c (Reductions$ReduceConsumer/create acc rfn)] (.forEachRemaining s c) @c))) (defn parallel-spliterator-reduce [initValFn rfn mergeFn ^Spliterator s ^ParallelOptions options] (let [pool (.-pool options) ;;I just want enough splits so that there is decent granularity as compared ;;to the parallelism of the pool. Another more common technique is to split ;;until each spliterator has less than or equal to some threshold - this is used ;;in various places but it requires the spliterator to estimate size correctly. ;;This technique does not have that requirement. n-splits (Math/round (/ (Math/log (* 4 (.-parallelism options))) (Math/log 2))) spliterators (split-spliterator n-splits s) ^ArrayBlockingQueue queue (options->queue options) [submissions lookahead] (->> spliterators (map (fn [spliterator] (.submit pool ^Callable (fn [] (let [acc (initValFn) ^Spliterator s spliterator] (if (.-ordered options) (consume-spliterator! rfn acc s) (queue-put! queue (consume-spliterator! rfn acc s) (.-putTimeoutMs options)))))))) (seq->lookahead (n-lookahead options)))] (->> (if (.-ordered options) (map (fn [l r] (.get ^Future l)) submissions lookahead) (map (fn [l] (queue-take queue)) lookahead)) (Reductions/iterableMerge options mergeFn)))) (defn- bitset-reduce ([^BitSet coll rfn acc] (let [^IFn$OLO rfn (Transformables/toLongReductionFn rfn)] (loop [bit (.nextSetBit coll 0) acc acc] (if (and (>= bit 0) (not (reduced? acc))) (recur (.nextSetBit coll (unchecked-inc bit)) (.invokePrim rfn acc (Integer/toUnsignedLong bit))) acc)))) ([^BitSet coll rfn] (if (.isEmpty coll) (rfn) (loop [bit (.nextSetBit coll 0) first true acc nil] (if (and (>= bit 0) (not (reduced? acc))) (recur (.nextSetBit coll (unchecked-inc bit)) false (if first bit (rfn acc (Integer/toUnsignedLong bit)))) acc))))) (deftype ^:private BitSetIterator [^{:unsynchronized-mutable true :tag long} setbit ^BitSet data] Iterator (hasNext [this] (not (== setbit -1))) (next [this] (let [retval setbit nextbit (.nextSetBit data (unchecked-inc setbit))] (set! setbit (long (if (== nextbit -1) -1 nextbit))) retval))) (deftype SupplierIterator [^Supplier sup ^:unsynchronized-mutable val] Iterator (hasNext [this] (boolean val)) (next [this] (let [rv val] (set! val (.get sup)) rv))) (extend-protocol protocols/ToIterable nil (convertible-to-iterable? [item] true) (->iterable [item] PersistentList/EMPTY) java.util.function.Supplier (convertible-to-iterable? [item] true) (->iterable [item] (reify Iterable (iterator [this] (SupplierIterator. item (.get ^java.util.function.Supplier item))))) Object (convertible-to-iterable? [item] (or (instance? Iterable item) (protocols/convertible-to-collection? item))) (->iterable [item] (cond (instance? Iterable item) item (protocols/convertible-to-collection? item) (protocols/->collection item) :else (throw (RuntimeException. (str "Item is not iterable: " (type item))))))) (extend-protocol protocols/ToCollection nil (convertible-to-collection? [item] true) (->collection [item] PersistentList/EMPTY) Object (convertible-to-collection? [item] (or (instance? Collection item) (instance? Map item) (instance? Seqable item) (.isArray (.getClass item)))) (->collection [item] (cond (instance? Collection item) item (.isArray (.getClass item)) (ArrayLists/toList item) (instance? Map item) (.entrySet ^Map item) (instance? Seqable item) (seq item) :else (throw (RuntimeException. (str "Item is not convertible to collection: " (type item)))))) String (->collection [item] (StringCollection. item)) BitSet (convertible-to-collection? [item] true) (->collection [item] (reify ICollectionDef (add [c obj] (let [obj (int obj) retval (.get item obj)] (.set item (int obj)) retval)) (remove [c obj] (let [obj (int obj) retval (.get item obj)] (.clear item obj) retval)) (clear [c] (.clear item)) (size [c] (.cardinality item)) (iterator [c] (BitSetIterator. (.nextSetBit item 0) item)) (toArray [c] (let [alist (ArrayLists$ObjectArrayList. (.cardinality item))] (.addAllReducible alist c) (.toArray alist))) IReduce (reduce [c rfn] (bitset-reduce item rfn)) IReduceInit (reduce [c rfn init] (bitset-reduce item rfn init))))) (extend-protocol protocols/PAdd Collection (add-fn [c] #(do (.add ^Collection %1 %2) %1))) (extend BitSet protocols/Reduction {:reducible? (constantly true)}) (clojure.core/extend BitSet cl-proto/CollReduce {:coll-reduce bitset-reduce}) (defn- fjfork [task] (.fork ^ForkJoinTask task)) (defn- fjjoin [task] (.join ^ForkJoinTask task)) (defn- fjtask [^Callable f] (ForkJoinTask/adapt f)) (defn- ensure-fj-pool ^ForkJoinPool [p] (when-not (instance? ForkJoinPool p) (throw (RuntimeException. "Pool passed in must be a forkjoin pool"))) p) (extend-protocol protocols/ParallelReduction Object (preduce [coll init-val-fn rfn merge-fn options] (cond (instance? ITypedReduce coll) (.parallelReduction ^ITypedReduce coll init-val-fn rfn merge-fn options) (instance? RandomAccess coll) (Reductions/parallelRandAccessReduction init-val-fn rfn merge-fn coll options) (instance? IReduceInit coll) (Reductions/serialParallelReduction init-val-fn rfn options coll) (instance? Set coll) (Reductions/parallelCollectionReduction init-val-fn rfn merge-fn coll options) (instance? Map coll) (Reductions/parallelCollectionReduction init-val-fn rfn merge-fn (.entrySet ^Map coll) options) :else (Reductions/serialParallelReduction init-val-fn rfn options coll))) PersistentHashMap (preduce [coll init-val-fn rfn merge-fn options] (let [options ^ParallelOptions options pool (ensure-fj-pool (.-pool options)) n (.-minN options) _ (when (.-unmergedResult options) (throw (RuntimeException. "Persistent hash maps do not support unmerged results"))) combinef (fn ([] (init-val-fn)) ([lhs rhs] (merge-fn lhs rhs))) fjinvoke (fn [f] (if (in-fork-join-task?) (f) (.invoke pool ^ForkJoinTask (fjtask f)))) rf (fn [acc k v] (rfn acc (MapEntry/create k v)))] (.fold coll n combinef rf fjinvoke fjtask fjfork fjjoin)))) (defmacro array-fast-reduce [ary-cls] `(do (clojure.core/extend ~ary-cls cl-proto/CollReduce {:coll-reduce (fn ([~(with-meta 'coll {:tag ary-cls}) f#] (.reduce (ArrayLists/toList ~'coll) f#)) ([~(with-meta 'coll {:tag ary-cls} ) f# acc#] (.reduce (ArrayLists/toList ~'coll) f# acc#)))}) (extend ~ary-cls protocols/Reduction {:reducible? (constantly true)}))) (array-fast-reduce (Class/forName "[B")) (array-fast-reduce (Class/forName "[S")) (array-fast-reduce (Class/forName "[C")) (array-fast-reduce (Class/forName "[I")) (array-fast-reduce (Class/forName "[J")) (array-fast-reduce (Class/forName "[F")) (array-fast-reduce (Class/forName "[D")) (array-fast-reduce (Class/forName "[Z")) (array-fast-reduce (Class/forName "[Ljava.lang.Object;")) (defn map-fast-reduce [map-cls] (clojure.core/extend cl-proto/CollReduce {:coll-reduce (fn map-reducer ([coll f] (Reductions/iterReduce (.entrySet ^Map coll) f)) ([coll f init] (Reductions/iterReduce (.entrySet ^Map coll) init f)))}) (extend map-cls protocols/ToCollection {:convertible-to-collection? (constantly true) :->collection (fn [^Map m] (reify ICollectionDef (size [this] (.size m)) (iterator [this] (.iterator (.entrySet m))) IReduce (reduce [this rfn] (if (instance? IReduce m) (.reduce ^IReduce m rfn) (Reductions/iterReduce (.entrySet m) rfn))) IReduceInit (reduce [this rfn init] (if (instance? IReduceInit m) (.reduce ^IReduceInit m rfn init) (Reductions/iterReduce (.entrySet m) init rfn)))))} protocols/ToIterable {:convertible-to-iterable? (constantly true) :->iterable (fn [m] (protocols/->collection m))})) (map-fast-reduce java.util.HashMap) (map-fast-reduce java.util.LinkedHashMap) (map-fast-reduce java.util.SortedMap) (map-fast-reduce java.util.concurrent.ConcurrentHashMap) (defn iterable-fast-reduce [coll-cls] (clojure.core/extend coll-cls cl-proto/CollReduce {:coll-reduce (fn map-reducer ([coll f] (Reductions/iterReduce coll f)) ([coll f init] (Reductions/iterReduce coll init f)))})) (iterable-fast-reduce java.util.HashSet) (iterable-fast-reduce java.util.LinkedHashSet) (iterable-fast-reduce java.util.SortedSet) ================================================ FILE: src/ham_fisted/iterator.clj ================================================ (ns ham-fisted.iterator "Generialized efficient pathways involving iterators." (:require [ham-fisted.language :refer [cond not]] [ham-fisted.function :as hamf-fn] [ham-fisted.print :refer [implement-tostring-print]]) (:import [java.util Iterator PriorityQueue Comparator] [java.util.stream Stream] [java.util.function Supplier Function BiFunction Consumer Predicate BiConsumer] [java.util.concurrent BlockingQueue] [clojure.lang ArraySeq Seqable IteratorSeq] [ham_fisted StringCollection ArrayLists MergeIterator MergeIterator$CurrentIterator Transformables$MapIterable ITypedReduce Reductions Transformables]) (:refer-clojure :exclude [cond not next])) (set! *warn-on-reflection* true) (set! *unchecked-math* :warn-on-boxed) (defn has-next? "Given an iterator or nil return true if the iterator has next." [^Iterator iter] (boolean (and iter (.hasNext iter)))) (defn next "Given an iterator call next." [^Iterator iter] (.next iter)) (defn maybe-next "Given an iterator or nil return the next element if the iterator hasNext." [^Iterator iter] (when (has-next? iter) (.next iter))) (def ^{:arglists '[[a]]} non-nil? (hamf-fn/predicate a (if (nil? a) false true))) (defn- failed-coercion-message ^String [item target-type] (format "Item type %s has no coercion to %s" (type item) target-type)) (defn ary-iter "Create an iterator for any primitive or object java array." ^Iterator [ary-data] (.iterator (ArrayLists/toList ary-data))) (definterface CtxAdvance (advance [])) (deftype ^:private CtxIter [valid? init-fn update-fn val-fn ^:unsynchronized-mutable step ^:unsynchronized-mutable ctx] CtxAdvance (advance [m] (let [s step] (cond (identical? s ::empty-init) (set! ctx (init-fn)) (identical? s ::empty-update) (set! ctx (update-fn ctx))) (set! step ::value))) Iterator (hasNext [this] (.advance this) (boolean (valid? ctx))) (next [this] (.advance this) (set! step ::empty-update) (val-fn ctx))) (defn ctx-iter ^CtxIter [valid? init-fn update-fn val-fn] (CtxIter. valid? init-fn update-fn val-fn ::empty-init nil)) (defn ->iterator "Convert a stream, supplier, or an iterable into an iterator." ^Iterator [item] (cond (nil? item) nil (instance? Iterator item) item (instance? ArraySeq item) (->iterator (.array ^ArraySeq item)) (instance? Iterable item) (.iterator ^Iterable item) (.isArray (.getClass ^Object item)) (ary-iter item) (instance? CharSequence item) (.iterator (StringCollection. ^CharSequence item)) (instance? Stream item) (.iterator ^Stream item) (instance? Supplier item) (let [^Supplier item item init-update (fn ([] (.get item)) ([_] (.get item)))] (ctx-iter non-nil? init-update init-update identity)) :else (throw (Exception. (failed-coercion-message item "iterator"))))) (defmacro doiter "Execute body for every item in the iterable. Expecting side effects, returns nil." [varname iterable & body] `(let [iter# (->iterator ~iterable)] (loop [continue?# (.hasNext iter#)] (when continue?# (let [~varname (.next iter#)] ~@body (recur (.hasNext iter#))))))) (defn linear-merge-iterator "Create a merging iterator - fast for N < 8." (^Iterator [cmp p iterators] (MergeIterator/createMergeIterator iterators cmp p)) (^Iterator [cmp iterators] (MergeIterator/createMergeIterator iterators cmp)) (^Iterator [iterators] (MergeIterator/createMergeIterator iterators compare))) (deftype ^:private CurrentIterator [^Iterator iter ^:unsynchronized-mutable current] Iterator (hasNext [this] (.hasNext iter)) (next [this] (let [c (.next iter)] (set! current c) c)) clojure.lang.IDeref (deref [this] current)) (defn current-iterator "Return a current iterator - and iterator that retains the current object. This iterator is positioned just before the first object so it's current item is nil." ^CurrentIterator [item] (let [iter (->iterator item)] (cond (nil? iter) nil (.hasNext iter) (CurrentIterator. iter nil) :else nil))) (defn iterable "Create an iterable. init-fn is not called until the first has-next call. * valid? - ctx->boolean - defaults to non-nil? * init-fn - creates new ctx * update-fn - function from ctx->ctx * val-fn - function from ctx->val - defaults to deref" (^Iterable [valid? init-fn update-fn val-fn] (reify Iterable (iterator [this] (ctx-iter valid? init-fn update-fn val-fn)))) (^Iterable [init-fn update-fn] (iterable non-nil? init-fn update-fn deref))) (defn once-iterable "Create an iterable that can only be iterated once - it always returns the same (non threadsafe) iterator every `iterator` call. init-fn is not called until the first has-next call - also see [[iterable]]. The arguments have different defaults as once-iterables can close over global context on construction as they can only be iterated once. * valid? - ctx->boolean - defaults to non-nil? * init-fn - creates new ctx * update-fn - function from ctx->ctx - defaults to init-fn ignoring argument. * val-fn - function from ctx->val - defaults to identity" (^Iterable [valid? init-fn update-fn val-fn] ;;iterator defined outside of iterable so we can correctly survive patterns like (when (seq v) ...) (let [iter (.iterator (iterable valid? init-fn update-fn val-fn))] (reify Iterable (iterator [this] iter)))) (^Iterable [valid? init-fn] (once-iterable valid? init-fn (fn [_] (init-fn)) identity)) (^Iterable [init-fn] (once-iterable non-nil? init-fn))) (deftype ^:private SeqIterable [^Iterable iable seq-data* m] clojure.lang.IPersistentCollection (cons [_ o] (if-let [sq @seq-data*] (.cons ^clojure.lang.IPersistentCollection sq o) (list o))) (empty [_] '()) (equiv [this o] (clojure.lang.Util/equiv (seq this) o)) clojure.lang.Sequential clojure.lang.IHashEq (hasheq [this] (hash (or (seq this) '()))) Seqable (seq [this] (vswap! seq-data* (fn [val] (if val val (IteratorSeq/create (.iterator this)))))) ITypedReduce (reduce [this rfn acc] (if-let [seq-impl @seq-data*] (reduce rfn acc seq-impl) (Reductions/iterReduce this acc rfn))) Iterable (iterator [this] (if-let [ss @seq-data*] (.iterator ^Iterable @seq-data*) (.iterator iable))) clojure.lang.IObj (withMeta [this new-m] (SeqIterable. iable seq-data* new-m)) (meta [this] m) Object (toString [this] (Transformables/sequenceToString (or (seq this) '())))) (implement-tostring-print SeqIterable) (defn seq-iterable "Iterable with efficient reduce but also contains a cached seq conversion so patterns like: (when (seq v) ...) still work without needing to dereference the iterable twice." ^Iterable [iterable] (SeqIterable. iterable (volatile! nil) nil)) (defn seq-once-iterable (^Iterable [valid? init update val-fn] (-> (once-iterable valid? init update val-fn) (seq-iterable))) (^Iterable [valid? init] (-> (once-iterable valid? init) (seq-iterable))) (^Iterable [init] (-> (once-iterable init) (seq-iterable)))) (defn- is-not-empty? [^java.util.Collection c] (not (.isEmpty c))) (defn- obj-aget [^objects a ^long idx] (aget a idx)) (deftype ^:private ConsIter [^:unsynchronized-mutable v ^Iterator iter] Iterator (hasNext [this] (boolean (or (not (identical? v ::empty)) (.hasNext iter)))) (next [this] (if (identical? v ::empty) (.next iter) (do (let [rv v] (set! v ::empty) rv)))) clojure.lang.IDeref (deref [this] v)) (defn iter-cons "Produce a new iterator that points to vv then defers to passed in iterator." ^Iterator [vv ^Iterator iter] ;;attempt to keep stack of cons-iters as small as possible (if (and (instance? ConsIter iter) (identical? @iter ::empty)) (iter-cons vv (.-iter ^ConsIter iter)) (ConsIter. vv iter))) (defn dedup-first-by "Given a sorted sequence remove duplicates keeping first." ([key-fn ^Comparator cmp data] (let [update (fn [{:keys [iter]}] (when (has-next? iter) (let [vv (next iter) k (key-fn vv)] (loop [] (if (has-next? iter) (let [nv (next iter)] (if (== 0 (.compare cmp k (key-fn nv))) (recur) {:iter (iter-cons nv iter) :val vv})) {:val vv})))))] (seq-once-iterable :val #(update {:iter (->iterator data)}) update :val))) ([^Comparator cmp data] (dedup-first-by identity cmp data))) (defn merge-iterable "Create an efficient stable n-way merge between a sequence of iterables using comparator. If iterables themselves are sorted result will be sorted. If two items tie then the one from the leftmost iterable wins." [^Comparator cmp iterables] (seq-iterable (iterable is-not-empty? #(let [pq (PriorityQueue. (hamf-fn/make-comparator a b (let [cc (.compare cmp (obj-aget a 1) (obj-aget b 1))] (if (== cc 0) (.compareTo ^Comparable (obj-aget a 2) (obj-aget b 2)) cc))))] (loop [outer-iter (->iterator iterables) idx 0] (when (has-next? outer-iter) (when-let [iter (->iterator (.next outer-iter))] (when (has-next? iter) (.offer pq (object-array [iter (.next iter) idx])))) (recur outer-iter (inc idx)))) pq) (fn [^PriorityQueue pq] (let [^objects entry (.poll pq) ^Iterator iter (aget entry 0)] (when (.hasNext iter) (aset entry 1 (.next iter)) (.offer pq entry))) pq) #(obj-aget (.peek ^PriorityQueue %) 1)))) (defn unstable-merge-iterable "Create an efficient n-way merge between a sequence of iterables using comparator. If iterables themselves are sorted result will be sorted." [^Comparator cmp iterables] (seq-iterable (iterable is-not-empty? #(let [pq (PriorityQueue. (hamf-fn/make-comparator a b (.compare cmp (obj-aget a 1) (obj-aget b 1))))] (run! (fn [iable] (when-let [iter (->iterator iable)] (when (.hasNext iter) (.offer pq (object-array [iter (.next iter)]))))) iterables) pq) (fn [^PriorityQueue pq] (let [^objects entry (.poll pq) ^Iterator iter (aget entry 0)] (when (.hasNext iter) (aset entry 1 (.next iter)) (.offer pq entry))) pq) #(obj-aget (.peek ^PriorityQueue %) 1)))) (defn blocking-queue->iterable "Given a blocking queue return an iterable that iterates until queue returns term-symbol. Uses take to block indefinitely -- will throw any throwable that comes out of the queue." ([^BlockingQueue queue term-symbol] (seq-iterable (once-iterable #(not (identical? % term-symbol)) #(let [v (.take queue)] (when (instance? Throwable v) (throw (RuntimeException. "Error retrieving queue value" v))) v)))) ([^BlockingQueue queue timeout-us timeout-symbol term-symbol] (seq-iterable (once-iterable #(not (identical? % term-symbol)) #(let [v (.poll queue timeout-us java.util.concurrent.TimeUnit/MICROSECONDS)] (cond (instance? Throwable v) (throw (RuntimeException. "Error retrieving queue value" v)) (nil? v) timeout-symbol v)))))) (defn const-iterable "Return an iterable that always returns a arg." ^Iterable [arg] (reify Iterable (iterator [this] (reify Iterator (hasNext [this] true) (next [this] arg))))) (deftype IterTake [^Iterator iter ^long n ^{:unsynchronized-mutable true :tag long} idx] Iterator (hasNext [this] (and (.hasNext iter) (< idx n))) (next [this] (set! idx (inc idx)) (.next iter))) (defn iter-take "take n from an iterator returning a new iterator" ^Iterator [^long n coll] (IterTake. (->iterator coll) n 0)) (defn wrap-iter "Wrap an iterator returning an iterable." ^Iterable [iter] (-> (reify Iterable (iterator [this] (->iterator iter))) seq-iterable)) (defn iter-take-while "Returns {:data :rest*} where rest* resolves to an iterator once data has been completely consumed." [pred iter] (let [iter (->iterator iter)] (when (has-next? iter) (let [res (promise) updater (fn [] (try (if (has-next? iter) (let [v (next iter)] (if (pred v) v (do (deliver res (seq-iterable (reify Iterable (iterator [this] (if v (iter-cons v iter) iter))))) nil))) (do (deliver res nil) nil)) (catch Throwable e (println "During take while!!" e) (deliver res e) e)))] {:data (seq-once-iterable non-nil? updater) :rest* res})))) ================================================ FILE: src/ham_fisted/language.clj ================================================ (ns ham-fisted.language (:import [ham_fisted Transformables] [ham_fisted ObjArray]) (:refer-clojure :exclude [cond constantly not])) (defmacro cond "See documentation for [[ham-fisted.api/cond]]" [& clauses] (let [clauses (vec clauses) nc (count clauses) constant-clause? #(or (identical? % true) (keyword? %)) odd-clauses? (odd? nc) else? (or odd-clauses? (and (> nc 2) (constant-clause? (nth clauses (- nc 2)))))] (if-not else? `(clojure.core/cond ~@clauses) (let [[clauses else-branch] (if odd-clauses? [(subvec clauses 0 (dec nc)) (last clauses)] [(subvec clauses 0 (- nc 2)) (last clauses)]) pred-true-branch (reverse (partition 2 clauses))] (reduce (fn [stmts [pred true-branch]] `(if ~pred ~true-branch ~stmts)) else-branch pred-true-branch))))) (def array-classes {:byte (.getClass ^Object (clojure.core/byte-array 0)) :short (.getClass ^Object (clojure.core/short-array 0)) :char (.getClass ^Object (clojure.core/char-array 0)) :int (.getClass ^Object (clojure.core/int-array 0)) :long (.getClass ^Object (clojure.core/long-array 0)) :float (.getClass ^Object (clojure.core/float-array 0)) :double (.getClass ^Object (clojure.core/double-array 0)) :boolean (.getClass ^Object (clojure.core/boolean-array 0)) :object (.getClass ^Object (clojure.core/object-array 0))}) (defn constantly [x] (fn constantly-fn ([] x) ([a] x) ([a b] x) ([a b c] x) ([a b c d] x) ([a b c d e] x) ([a b c d e & args] x))) (defn not "Returns boolean opposite of passed in value" {:inline (fn [x] `(Transformables/not ~x)) :inline-arities #{1}} [a] (Transformables/not a)) (def ^:private empty-objs (clojure.core/object-array 0)) (defn obj-ary "As quickly as possible, produce an object array from these inputs. Very fast for arities <= 16." (^objects [] empty-objs) (^objects [v0] (ObjArray/create v0)) (^objects [v0 v1] (ObjArray/create v0 v1)) (^objects [v0 v1 v2] (ObjArray/create v0 v1 v2)) (^objects [v0 v1 v2 v3] (ObjArray/create v0 v1 v2 v3)) (^objects [v0 v1 v2 v3 v4] (ObjArray/create v0 v1 v2 v3 v4)) (^objects [v0 v1 v2 v3 v4 v5] (ObjArray/create v0 v1 v2 v3 v4 v5)) (^objects [v0 v1 v2 v3 v4 v5 v6] (ObjArray/create v0 v1 v2 v3 v4 v5 v6)) (^objects [v0 v1 v2 v3 v4 v5 v6 v7] (ObjArray/create v0 v1 v2 v3 v4 v5 v6 v7)) (^objects [v0 v1 v2 v3 v4 v5 v6 v7 v8] (ObjArray/create v0 v1 v2 v3 v4 v5 v6 v7 v8)) (^objects [v0 v1 v2 v3 v4 v5 v6 v7 v8 v9] (ObjArray/create v0 v1 v2 v3 v4 v5 v6 v7 v8 v9)) (^objects [v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10] (ObjArray/create v0 v1 v2 v3 v4 v5 v6 v7 v8 v9) v10) (^objects [v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11] (ObjArray/create v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v0 v11)) (^objects [v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12] (ObjArray/create v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v0 v11 v12)) (^objects [v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13] (ObjArray/create v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v0 v11 v12 v13)) (^objects [v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14] (ObjArray/create v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v0 v11 v12 v13 v14)) (^objects [v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15] (ObjArray/create v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v0 v11 v12 v13 v14 v15))) ================================================ FILE: src/ham_fisted/lazy_caching.clj ================================================ (ns ham-fisted.lazy-caching (:require [ham-fisted.lazy-noncaching :as lznc]) (:import [java.util RandomAccess List Iterator] [ham_fisted Transformables$CachingIterable Transformables$CachingList ArrayLists ArrayHelpers ArrayLists$ObjectArrayList LazyChunkedSeq] [clojure.lang ISeq ArraySeq IFn ChunkedCons ArrayChunk]) (:refer-clojure :exclude [map filter concat repeatedly])) (defmacro lazy-chunked-seq [chunked-cons-code] `(LazyChunkedSeq. (fn [] ~chunked-cons-code))) (defn- seq-map-recur ([f ^ISeq c1 ^ISeq c2] (let [chunk-data (ArrayLists/objectArray 32)] (loop [c1 c1 c2 c2 idx 0] (if (and c1 c2 (< idx 32)) (do (ArrayHelpers/aset chunk-data idx (f (.first c1) (.first c2))) (recur (.next c1) (.next c2) (unchecked-inc idx))) (ChunkedCons. (ArrayChunk. chunk-data 0 idx) (when (and c1 c2) (lazy-chunked-seq (seq-map-recur f c1 c2)))))))) ([f ^ISeq c1 ^ISeq c2 ^ISeq c3] (let [chunk-data (ArrayLists/objectArray 32)] (loop [c1 c1 c2 c2 c3 c3 idx 0] (if (and c1 c2 c3 (< idx 32)) (do (ArrayHelpers/aset chunk-data idx (f (.first c1) (.first c2) (.first c3))) (recur (.next c1) (.next c2) (.next c3 )(unchecked-inc idx))) (ChunkedCons. (ArrayChunk. chunk-data 0 idx) (when (and c1 c2 c3) (lazy-chunked-seq (seq-map-recur f c1 c2 c3)))))))) ([f ^ISeq c1 ^ISeq c2 ^ISeq c3 ^ISeq c4] (let [chunk-data (ArrayLists/objectArray 32)] (loop [c1 c1 c2 c2 c3 c3 c4 c4 idx 0] (if (and c1 c2 c3 c4 (< idx 32)) (do (ArrayHelpers/aset chunk-data idx (f (.first c1) (.first c2) (.first c3) (.first c4))) (recur (.next c1) (.next c2) (.next c3) (.next c4) (unchecked-inc idx))) (ChunkedCons. (ArrayChunk. chunk-data 0 idx) (when (and c1 c2 c3 c4) (lazy-chunked-seq (seq-map-recur f c1 c2 c3 c4)))))))) ([f ^List cs] (let [chunk-data (ArrayLists/objectArray 32) ncs (.size cs) args (ArrayLists/objectArray ncs) [cidx next?] (loop [cidx 0 next? true] (if (and next? (< cidx 32)) (let [next? (loop [idx 0 next? true] (if (< idx ncs) (let [^ISeq c (.get cs (unchecked-int idx)) _ (ArrayHelpers/aset args (unchecked-int idx) (.first c)) c (.next c)] (.set cs (unchecked-int idx) c) (recur (unchecked-inc idx) (and next? c))) next?))] (ArrayHelpers/aset chunk-data cidx (.applyTo ^IFn f (ArraySeq/create args))) (recur (unchecked-inc cidx) next?)) [cidx next?]))] (ChunkedCons. (ArrayChunk. chunk-data 0 cidx) (when next? (lazy-chunked-seq (seq-map-recur f cs))))))) (defn ^:no-doc seq-map ([f arg] (clojure.core/map f arg)) ([f c1 c2] (let [^ISeq c1 (seq c1) ^ISeq c2 (seq c2)] (if (and c1 c2) (seq-map-recur f c1 c2) '()))) ([f c1 c2 c3] (let [^ISeq c1 (seq c1) ^ISeq c2 (seq c2) ^ISeq c3 (seq c3)] (if (and c1 c2 c3) (seq-map-recur f c1 c2 c3) '()))) ([f c1 c2 c3 c4] (let [^ISeq c1 (seq c1) ^ISeq c2 (seq c2) ^ISeq c3 (seq c3) ^ISeq c4 (seq c4)] (if (and c1 c2 c3 c4) (seq-map-recur f c1 c2 c3 c4) '()))) ([f c1 c2 c3 c4 args] (let [cs (ArrayLists$ObjectArrayList.) c1 (seq c1) c2 (seq c2) c3 (seq c3) c4 (seq c4)] (if (and c1 c2 c3 c4) (do (.add cs c1) (.add cs c2) (.add cs c3) (.add cs c4) (let [all-valid? (reduce (fn [acc v] (if-let [s (seq v)] (do (.add cs s) acc) (reduced false))) true args)] (if all-valid? (seq-map-recur f cs) '()))) '())))) (defn map ([f arg] (clojure.core/map f arg)) ([f c1 c2] (seq-map f c1 c2)) ([f c1 c2 c3] (seq-map f c1 c2 c3)) ([f c1 c2 c3 c4] (seq-map f c1 c2 c3 c4)) ([f c1 c2 c3 c4 & args] (seq-map f c1 c2 c3 c4 args))) (defn- seq-tuple-map ([f ^ISeq c1 ^ISeq c2] (let [args (ArrayLists/objectArray 2) _ (ArrayHelpers/aset args (unchecked-int 0) (.first c1)) _ (ArrayHelpers/aset args (unchecked-int 1) (.first c2)) c1 (.next c1) c2 (.next c2)] (cons (f (ArrayLists/toList args)) (when (and c1 c2) (lazy-seq (seq-tuple-map f c1 c2)))))) ([f ^List cs] (let [n-args (.size cs) args (ArrayLists/objectArray n-args) next? (loop [idx 0 next? true] (if (< idx n-args) (let [^ISeq c (.get cs (unchecked-int idx)) _ (ArrayHelpers/aset args (unchecked-int idx) (.first c)) c (.next c)] (.set cs (unchecked-int idx) c) (recur (unchecked-inc idx) (and next? c))) next?))] (cons (f (ArrayLists/toList args)) (when next? (lazy-seq (seq-tuple-map f cs))))))) (defn tuple-map "f always receives a single tuple argument. This is *far* faster for larger argument lists." ([f arg] (clojure.core/map #(f [%]) arg)) ([f c1 c2] (let [c1 (seq c1) c2 (seq c2)] (if (and c1 c2) (seq-tuple-map f c1 c2) '()))) ([f c1 c2 & args] (let [args (doto (ArrayLists$ObjectArrayList.) (.add (seq c1)) (.add (seq c2)) (.addAll (lznc/map seq args)))] (if (lznc/every? identity args) (seq-tuple-map f args) '())))) (defn filter [pred coll] (-> (lznc/filter pred coll) (seq))) (defn concat ([] nil) ([a] a) ([a b] (->> (lznc/concat a b) (seq))) ([a b & args] (-> (apply lznc/concat a b args) (seq)))) (defn repeatedly ([f] (clojure.core/repeatedly f)) ([n f] (-> (lznc/repeatedly n f) (seq)))) ================================================ FILE: src/ham_fisted/lazy_noncaching.clj ================================================ (ns ham-fisted.lazy-noncaching "Lazy, noncaching implementation of many clojure.core functions. There are several benefits of carefully constructed lazy noncaching versions: 1. No locking - better multithreading/green thread performance. 2. Higher performance generally. 3. More datatype flexibility - if map is passed a single randomly addressible or generically parallelizable container the result is still randomly addressible or generically perallelizable. For instance (map key {:a 1 :b 2}) returns in the generic case something that can still be parallelizable as the entry set of a map implements spliterator." (:require [ham-fisted.iterator :as iterator] [ham-fisted.alists :as alists] [ham-fisted.protocols :as protocols] [ham-fisted.defprotocol :as hamf-defproto] [ham-fisted.function :as hamf-fn] [ham-fisted.print :as pp] [ham-fisted.language :as hamf-language] [ham-fisted.iterator :as hamf-iter] [ham-fisted.datatypes]) (:import [ham_fisted Transformables$MapIterable Transformables$FilterIterable Transformables$CatIterable Transformables$MapList Transformables$IMapable Transformables$SingleMapList Transformables StringCollection ArrayLists ArrayImmutList ArrayLists$ObjectArrayList IMutList TypedList LongMutList DoubleMutList ReindexList Transformables$IndexedMapper IFnDef$OLO IFnDef$ODO Reductions Reductions$IndexedAccum IFnDef$OLOO ArrayHelpers ITypedReduce PartitionByInner Casts IMutList LazyChunkedSeq ParallelOptions$CatParallelism MutTreeList IFnDef LongAccum] [java.lang.reflect Array] [it.unimi.dsi.fastutil.ints IntArrays] [java.util.concurrent.atomic AtomicLong] [java.util RandomAccess Collection Map List Random Set Iterator Map$Entry ArrayList Comparator] [clojure.lang RT IPersistentMap IReduceInit IReduce PersistentList IFn$OLO IFn$ODO IFn$DD IFn$LD IFn$OD IFn ArraySeq IFn$DL IFn$LL IFn$OL IFn$D IFn$L IFn$LO IFn$DO Counted IDeref Seqable IObj ]) (:refer-clojure :exclude [map concat filter repeatedly into-array shuffle object-array remove map-indexed partition-by partition-all every? complement cond drop take count])) (set! *warn-on-reflection* true) (set! *unchecked-math* :warn-on-boxed) (def ^{:tag ArrayImmutList} empty-vec ArrayImmutList/EMPTY) (declare concat map-reducible every?) (defmacro cond "See documentation for [[ham-fisted.api/cond]]" [& clauses] `(hamf-language/cond ~@clauses)) (defn count ^long [m] (protocols/count m)) (hamf-defproto/extend Map$Entry protocols/Counted {:count 2}) (hamf-defproto/extend nil protocols/Counted {:count 0}) (defmacro countable-arrays [] `(do ~@(->> hamf-language/array-classes (mapcat (fn [kv] (let [alen-name (symbol (str (name (key kv)) "-alength"))] `[(defn ~alen-name ~(with-meta [(with-meta 'm {:tag (.getName ^Class (val kv))})] {:tag 'long}) (alength ~'m)) (hamf-defproto/extend ~(val kv) protocols/Counted {:count ~alen-name})])))))) (countable-arrays) (hamf-defproto/extend-protocol protocols/Counted clojure.lang.Counted (count [s] (.count s)) java.util.RandomAccess (count [s] (.size ^java.util.Collection s)) CharSequence (count [s] (.length s)) Map (count [s] (.size s)) Transformables$CatIterable (count [s] (let [lc (LongAccum. 0)] (reduce (fn [^LongAccum lc _v] (.accept lc (count _v)) lc) lc (reify Iterable (iterator [this] (.containerIter ^Transformables$CatIterable s)))) (long (.deref lc)))) Object (count [s] (let [lc (LongAccum. 0)] (reduce (fn [^LongAccum lc _v] (.accept lc 1) lc) lc s) (long (.deref lc))))) (defn ->collection "Ensure an item implements java.util.Collection. This is inherently true for seqs and any implementation of java.util.List but not true for object arrays. For maps this returns the entry set." ^Collection [item] (cond (nil? item) empty-vec (instance? Collection item) item :else (protocols/->collection item))) (defn ->reducible [item] (if (or (instance? IReduceInit item) (instance? IReduce item) (instance? Iterable item) (protocols/reducible? item)) item (->collection item))) (defn ->iterable ^Iterable [a] (if (instance? Iterable a) a (protocols/->iterable a))) (def ^:private obj-ary-cls (Class/forName "[Ljava.lang.Object;")) (defn object-array "Faster version of object-array for eductions, java collections and strings." ^objects [item] (let [item (if (instance? Map item) (.entrySet ^Map item) item)] (cond (or (nil? item) (number? item)) (clojure.core/object-array item) (instance? obj-ary-cls item) item ;;Results of eduction aren't collections but do implement IReduceInit (instance? IReduceInit item) (if (or (instance? RandomAccess item) (instance? Counted item)) (let [item-size (if (instance? RandomAccess item) (.size ^List item) (count item)) retval (clojure.core/object-array item-size)] (reduce (Reductions$IndexedAccum. (reify IFnDef$OLOO (invokePrim [this acc idx v] (ArrayHelpers/aset ^objects acc (unchecked-int idx) v) acc))) retval item)) (let [retval (ArrayLists$ObjectArrayList.)] (.addAllReducible retval item) (.toArray retval))) (instance? Collection item) (.toArray ^Collection item) (instance? String item) (.toArray (StringCollection. item)) (.isArray (.getClass ^Object item)) (.toArray (ArrayLists/toList item)) (instance? Iterable item) (let [alist (ArrayLists$ObjectArrayList.)] (.addAllReducible alist item) (.toArray alist)) :else (throw (Exception. (str "Unable to coerce item of type: " (type item) " to an object array")))))) (def ^:no-doc long-array-cls (Class/forName "[J")) (def ^:no-doc double-array-cls (Class/forName "[D")) (def ^:no-doc obj-array-cls (Class/forName "[Ljava.lang.Object;")) (defn as-random-access "If item implements RandomAccess, return List interface." ^List [item] (cond (instance? RandomAccess item) item (instance? long-array-cls item) (ArrayLists/toList ^longs item) (instance? double-array-cls item) (ArrayLists/toList ^doubles item) (instance? obj-array-cls item) (ArrayLists/toList ^objects item))) (defn ->random-access ^List [item] (if (instance? RandomAccess item) item (let [c (->collection item)] (if (instance? RandomAccess c) c (-> (doto (MutTreeList.) (.addAllReducible c)) (persistent!)))))) (defn constant-countable? [data] (or (nil? data) (instance? RandomAccess data) (instance? Counted data) (instance? Set data) (instance? Map data) (.isArray (.getClass ^Object data)))) (defn constant-count "Constant time count. Returns nil if input doesn't have a constant time count." [data] (cond (nil? data) 0 (instance? RandomAccess data) (.size ^List data) (instance? Counted data) (.count ^Counted data) (instance? Map data) (.size ^Map data) (instance? Set data) (.size ^Set data) (.isArray (.getClass ^Object data)) (Array/getLength data))) (defn into-array ([aseq] (into-array (if-let [item (first aseq)] (.getClass ^Object item) Object) aseq)) ([ary-type aseq] (let [^Class ary-type (or ary-type Object) aseq (->reducible aseq)] (if-let [c (constant-count aseq)] (let [rv (Array/newInstance ary-type (int c))] (.fillRangeReducible ^IMutList (alists/wrap-array rv) 0 aseq) rv) (let [^IMutList al (alists/wrap-array-growable (Array/newInstance ary-type 4) 0)] (.addAllReducible al aseq) (.toNativeArray al))))) ([ary-type mapfn aseq] (if mapfn (into-array ary-type (map-reducible mapfn aseq)) (into-array ary-type aseq)))) (defn map ([f] (fn [rf] (let [rf (Transformables/typedMapReducer rf f)] (cond (instance? IFn$OLO rf) (reify IFnDef$OLO (invoke [this] (rf)) (invoke [this result] (rf result)) (invokePrim [this acc v] (.invokePrim ^IFn$OLO rf acc v)) (applyTo [this args] (rf (first args) (apply f (rest args))))) (instance? IFn$ODO rf) (reify IFnDef$ODO (invoke [this] (rf)) (invoke [this result] (rf result)) (invokePrim [this acc v] (.invokePrim ^IFn$ODO rf acc v)) (applyTo [this args] (rf (first args) (apply f (rest args))))) :else (fn ([] (rf)) ([result] (rf result)) ([result input] (rf result input)) ([result input & inputs] (apply rf result input inputs))))))) ([f arg] (cond (nil? arg) PersistentList/EMPTY (instance? Transformables$IMapable arg) (.map ^Transformables$IMapable arg f) (instance? RandomAccess arg) (Transformables$SingleMapList. f nil arg) :else (Transformables$MapIterable/createSingle f nil arg))) ([f arg & args] (let [args (concat [arg] args)] (if (every? #(instance? RandomAccess %) args) (Transformables$MapList/create f nil (into-array List args)) (Transformables$MapIterable. f nil (.toArray ^Collection args)))))) (pp/implement-tostring-print Transformables$SingleMapList) (pp/implement-tostring-print Transformables$MapIterable) (pp/implement-tostring-print Transformables$MapList) (defn map-indexed [map-fn coll] (cond (nil? coll) coll (instance? RandomAccess coll) (let [^List coll coll] (reify IMutList (size [this] (.size coll)) (get [this idx] (map-fn idx (.get coll idx))) (subList [this sidx eidx] (map-indexed map-fn (.subList coll sidx eidx))) (reduce [this rfn acc] (reduce (Reductions$IndexedAccum. (reify IFnDef$OLOO (invokePrim [this acc idx v] (rfn acc (map-fn idx v))))) acc coll)) Transformables$IMapable (map [this mfn] (map-indexed (fn [idx v] (-> (map-fn idx v) (mfn))) coll)))) :else (Transformables$IndexedMapper. map-fn (->iterable coll) nil))) (pp/implement-tostring-print Transformables$IndexedMapper) (defn map-reducible "Map a function over r - r need only be reducible. Returned value does not implement seq but is countable when r is countable." [f r] (if-let [c (constant-count r)] (reify Counted (count [this] c) IReduceInit (reduce [this rfn acc] (Reductions/serialReduction (Transformables/typedMapReducer rfn f) acc r))) (reify IReduceInit (reduce [this rfn acc] (Reductions/serialReduction (Transformables/typedMapReducer rfn f) acc r))))) (defn tuple-map "Lazy nonaching map but f simply gets a single random-access list of arguments. The argument list may be mutably updated between calls." ([f c1] (let [rdc (fn [rfn acc] (reduce (fn [acc v] (rfn acc (f [v]))) acc c1))] (if-let [c1 (as-random-access c1)] (reify IMutList (size [this] (.size c1)) (get [this idx] (f [(.get c1 idx)])) (subList [this sidx eidx] (tuple-map f (.subList c1 sidx eidx))) (reduce [this rfn acc] (rdc rfn acc))) (reify Iterable (iterator [this] (let [citer (.iterator (->iterable c1))] (reify Iterator (hasNext [this] (.hasNext citer)) (next [this] (f [(.next citer)]))))) Seqable (seq [this] (LazyChunkedSeq/chunkIteratorSeq (.iterator this))) ITypedReduce (reduce [this rfn acc] (rdc rfn acc)))))) ([f c1 c2] (let [c1 (->iterable c1) c2 (->iterable c2)] (reify Iterable (iterator [this] (let [c1-iter (.iterator c1) c2-iter (.iterator c2)] (reify Iterator (hasNext [this] (and (.hasNext c1-iter) (.hasNext c2-iter))) (next [this] (f [(.next c1-iter) (.next c2-iter)]))))) Seqable (seq [this] (LazyChunkedSeq/chunkIteratorSeq (.iterator this))) ITypedReduce (reduce [this rfn acc] (Reductions/iterReduce this acc rfn))))) ([f c1 c2 & cs] (let [cs (doto (ArrayLists$ObjectArrayList.) (.add c1) (.add c2) (.addAll cs)) nargs (.size cs) next-fn (fn next-fn [iters ^objects args] (loop [idx 0] (if (< idx nargs) (let [^Iterator iter (iters idx)] (if (.hasNext iter) (do (ArrayHelpers/aset args (unchecked-int idx) (.next iter)) (recur (unchecked-inc idx))) false)) true))) rdc (fn [rfn acc] (let [iters (mapv #(.iterator (->iterable %)) cs)] (loop [acc acc args (ArrayLists/objectArray nargs) next? (next-fn iters args)] (if next? (let [acc (rfn acc (f (ArrayLists/toList ^objects args)))] (if (reduced? acc) (deref acc) (let [args (ArrayLists/objectArray nargs)] (recur acc args (next-fn iters args))))) acc))))] (reify Iterable (iterator [this] (let [args (ArrayLists/objectArray nargs) argvec (ArrayLists/toList args) iters (mapv #(.iterator (->iterable %)) cs)] (reify Iterator (hasNext [this] (next-fn iters args)) (next [this] (f argvec))))) Seqable (seq [this] (LazyChunkedSeq/chunkIteratorSeq (.iterator this))) ITypedReduce (reduce [this rfn acc] (rdc rfn acc)))))) (pp/implement-tostring-print Transformables$CatIterable) (defn apply-concat "A more efficient form of (apply concat ...) that doesn't force data to be a clojure seq. See [[concat-opts]] for opts definition." ([] PersistentList/EMPTY) ([data] (Transformables$CatIterable. data)) ([opts data] (Transformables$CatIterable. nil (condp identical? (get opts :cat-parallelism) :seq-wise ParallelOptions$CatParallelism/SEQWISE :elem-wise ParallelOptions$CatParallelism/ELEMWISE nil nil) data))) (defn concat ([] PersistentList/EMPTY) ([a] (if a a PersistentList/EMPTY)) ([a & args] (if (instance? Transformables$IMapable a) (.cat ^Transformables$IMapable a args) (apply-concat (cons a args))))) (defn concat-opts "Concat where the first argument is an options map. This variation allows you to set the `:cat-parallelism` as you may have an idea the best way to parallelism this concatenation at time of the concatenation creation. Options: `:cat-parallelism` - Set the type of parallelism - either `:elem-wise` or `:seq-wise` - this overrides settings later passed into calls such as [[reduce.preduce]] - see [[reduce/options->parallel-options]] for definition." ([opts a] (if a a PersistentList/EMPTY)) ([opts a & args] (if (instance? Transformables$IMapable a) (.cat ^Transformables$IMapable a args) (apply-concat opts (cons a args))))) (defn filter ([pred] (fn [rf] (Transformables$FilterIterable/typedReducer rf pred))) ([pred coll] (cond (nil? coll) PersistentList/EMPTY (instance? Transformables$IMapable coll) (.filter ^Transformables$IMapable coll pred) :else (Transformables$FilterIterable. pred nil coll)))) (pp/implement-tostring-print Transformables$FilterIterable) (defn complement "Like clojure core complement but avoids var lookup on 'not'" [f] (fn ([] (Transformables/not (f))) ([x] (Transformables/not (f x))) ([x y] (Transformables/not (f x y))) ([x y & zs] (Transformables/not (apply f x y zs))))) (defn remove "Returns a lazy sequence of the items in coll for which (pred item) returns logical false. pred must be free of side-effects. Returns a transducer when no collection is provided." {:added "1.0" :static true} ([pred coll] (filter (complement pred) coll)) ([pred] (filter (complement pred)))) (declare drop take) (deftype ^:private DropIterable [^long n data] clojure.lang.Sequential Iterable (iterator [this] (let [src-iter (.iterator (->iterable data))] (dotimes [idx n] (when (.hasNext src-iter) (.next src-iter))) src-iter)) ITypedReduce (reduce [this rfn acc] (let [rfn ((drop n) rfn)] (reduce rfn acc data))) Object (toString [this] (Transformables/sequenceToString this))) (pp/implement-tostring-print DropIterable) (defmacro define-drop-tducer [nm iface rf-tag] (let [invoke-nm (if (= iface 'IFnDef) 'invoke 'invokePrim)] `(deftype ~(with-meta nm {:private true}) [~(with-meta 'n {:unsynchronized-mutable true :tag 'long}) ~'rf] ~iface (~invoke-nm [this# ~'result ~'input] (let [~'nn (max -1 (dec ~'n))] (set! ~'n ~'nn) (if (neg? ~'nn) (~(symbol (str "." (name invoke-nm))) ~(with-meta 'rf {:tag rf-tag}) ~'result ~'input) ~'result))) (invoke [this#] (~'rf)) (invoke [this# result#] (~'rf result#)) clojure.lang.Fn))) (define-drop-tducer DropLongTducer IFnDef$OLO clojure.lang.IFn$OLO) (define-drop-tducer DropDoubleTducer IFnDef$ODO clojure.lang.IFn$ODO) (define-drop-tducer DropTducer IFnDef clojure.lang.IFn) (defn drop ([n] (fn [rf] (cond (instance? clojure.lang.IFn$OLO rf) (DropLongTducer. n rf) (instance? clojure.lang.IFn$ODO rf) (DropDoubleTducer. n rf) :else (DropTducer. n rf)))) ([^long n data] (if(nil? data) '() (if-let [l (as-random-access data)] (let [sl (.size l)] (if (< sl n) '[] (.subList l n sl))) (DropIterable. n data))))) (defmacro define-take-tducer [nm invoke-nm iface rf-tag] `(deftype ~(with-meta nm {:private true}) [~(with-meta 'n {:unsynchronized-mutable true :tag 'long}) ~'rf] ~iface (~invoke-nm [this# ~'acc ~'v] (set! ~'n (max -1 (dec ~'n))) (if (neg? ~'n) ~'acc (let [~'acc (~(symbol (str "." (name invoke-nm))) ~(with-meta 'rf {:tag rf-tag}) ~'acc ~'v)] (if (zero? ~'n) (ensure-reduced ~'acc) ~'acc)))) (invoke [this#] (~'rf)) (invoke [this# acc#] (~'rf acc#)) clojure.lang.Fn)) (define-take-tducer TakeTducer invoke IFnDef clojure.lang.IFn) (define-take-tducer TakeLongTducer invokePrim IFnDef$OLO clojure.lang.IFn$OLO) (define-take-tducer TakeDoubleTducer invokePrim IFnDef$ODO clojure.lang.IFn$ODO) (deftype TakeIterator [^{:unsynchronized-mutable true :tag long} n ^Iterator data] Iterator (hasNext [this] (and (pos? n) (.hasNext data))) (next [this] (set! n (dec n)) (if (.hasNext data) (.next data) (throw (java.util.NoSuchElementException. "Iter out of range"))))) (deftype TakeIterable [n data] clojure.lang.Sequential Iterable (iterator [this] (TakeIterator. n (.iterator (->iterable data)))) ITypedReduce (reduce [this rfn acc] (reduce ((take n) rfn) acc data)) Object (toString [this] (Transformables/sequenceToString this))) (pp/implement-tostring-print TakeIterable) (defn take ([n] (fn [rf] (cond (instance? clojure.lang.IFn$OLO rf) (TakeLongTducer. n rf) (instance? clojure.lang.IFn$ODO rf) (TakeDoubleTducer. n rf) :else (TakeTducer. n rf)))) ([^long n data] (if (nil? data) '() (if-let [l (as-random-access data)] (.subList l 0 (min n (.size l))) (TakeIterable. n data))))) (defmacro make-readonly-list "Implement a readonly list. If cls-type-kwd is provided it must be, at compile time, either :int64, :float64 or :object and the getLong, getDouble or get interface methods will be filled in, respectively. In those cases read-code must return the appropriate type." ([n idxvar read-code] `(make-readonly-list :object ~n ~idxvar ~read-code)) ([cls-type-kwd n idxvar read-code] `(let [~'nElems (int ~n)] ~(case cls-type-kwd :int64 `(reify TypedList (containedType [this#] Long/TYPE) LongMutList (size [this#] ~'nElems) (getLong [this# ~idxvar] ~read-code)) :float64 `(reify TypedList (containedType [this#] Double/TYPE) DoubleMutList (size [this#] ~'nElems) (getDouble [this# ~idxvar] ~read-code)) :object `(reify IMutList (size [this#] ~'nElems) (get [this# ~idxvar] ~read-code)))))) (defn type-single-arg-ifn "Categorize the return type of a single argument ifn. May be :float64, :int64, or :object." [ifn] (protocols/simplified-returned-datatype ifn)) (defn type-zero-arg-ifn "Categorize the return type of a single argument ifn. May be :float64, :int64, or :object." [ifn] (protocols/simplified-returned-datatype ifn)) (defn repeatedly "When called with one argument, produce infinite list of calls to v. When called with two arguments, produce a non-caching random access list of length n of calls to v." ([f] (reify Iterable (iterator [this] (reify java.util.Iterator (hasNext [this] true) (next [this] (f)))))) (^IMutList [n f] (let [n (int n)] (case (protocols/simplified-returned-datatype f) :int64 (reify TypedList (containedType [this] Long/TYPE) LongMutList (size [this] (unchecked-int n)) (getLong [this idx] (.invokePrim ^IFn$L f))) :float64 (reify TypedList (containedType [this] Double/TYPE) DoubleMutList (size [this] (unchecked-int n)) (getDouble [this idx] (.invokePrim ^IFn$D f))) (reify IMutList (size [this] (int n)) (get [this idx] (f))))))) (defn ^:no-doc contained-type [coll] (when (instance? TypedList coll) (.containedType ^TypedList coll))) (defn- int-primitive? [cls] (or (identical? Byte/TYPE cls) (identical? Short/TYPE cls) (identical? Integer/TYPE cls) (identical? Long/TYPE cls))) (defn- double-primitive? [cls] (or (identical? Float/TYPE cls) (identical? Double/TYPE cls))) (defn shift "Shift a collection forward or backward repeating either the first or the last entries. Returns a random access list with the same elements as coll. Example: ```clojure ham-fisted.api> (shift 2 (range 10)) [0 0 0 1 2 3 4 5 6 7] ham-fisted.api> (shift -2 (range 10)) [2 3 4 5 6 7 8 9 9 9] ```" [n coll] (let [n (long n) coll (->random-access coll) n-elems (.size coll) ne (dec n-elems) ctype (contained-type coll) ^IMutList ml coll] (cond (int-primitive? ctype) (make-readonly-list :int64 n-elems idx (.getLong ml (min ne (max 0 (- idx n))))) (double-primitive? ctype) (make-readonly-list :float64 n-elems idx (.getDouble ml (min ne (max 0 (- idx n))))) :else (make-readonly-list n-elems idx (.get coll (min ne (max 0 (- idx n)))))))) (defn seed->random ^Random [seed] (cond (instance? Random seed) seed (number? seed) (Random. (int seed)) (nil? seed) (Random.) :else (throw (Exception. (str "Invalid seed type: " seed))))) (def ^:private int-ary-cls (Class/forName "[I")) (defn reindex "Permut coll by the given indexes. Result is random-access and the same length as the index collection. Indexes are expected to be in the range of [0->count(coll))." [coll indexes] (let [^ints indexes (if (instance? int-ary-cls indexes) indexes (int-array indexes)) ^List coll (if (instance? RandomAccess coll) coll (->random-access coll))] (if (instance? IMutList coll) (.reindex ^IMutList coll indexes) (ReindexList/create indexes coll (meta coll))))) (defn shuffle "shuffle values returning random access container. Options: * `:seed` - If instance of java.util.Random, use this. If integer, use as seed. If not provided a new instance of java.util.Random is created." (^List [coll] (shuffle coll nil)) (^List [coll opts] (let [coll (->random-access coll) random (seed->random (get opts :seed))] (if (instance? IMutList coll) (.immutShuffle ^IMutList coll random) (reindex coll (IntArrays/shuffle (ArrayLists/iarange 0 (.size coll) 1) random)))))) (deftype ^:private PartitionOuterIter [^Iterator iter ignore-leftover? f binary-predicate ^:unsynchronized-mutable last-iter] Iterator (hasNext [this] (if last-iter (do (when (and (not ignore-leftover?) (.hasNext ^Iterator last-iter)) (throw (RuntimeException. "Sub-collection was not completely iterated through"))) (boolean @last-iter)) (.hasNext iter))) (next [this] (if last-iter (let [piter-data @last-iter v (piter-data 0) fv (piter-data 1) rv (PartitionByInner. iter f v binary-predicate)] (set! last-iter rv) rv) (let [v (.next iter) fv (f v) rv (PartitionByInner. iter f v binary-predicate)] (set! last-iter rv) rv)))) (deftype ^:private PartitionBy [f coll ignore-leftover? m binary-predicate ^{:unsynchronized-mutable true :tag long} _hasheq] ITypedReduce (reduce [this rfn acc] (let [citer (.iterator ^Iterable (protocols/->iterable coll))] (if (.hasNext citer) (loop [acc acc v (.next citer) fv (f v)] (let [ piter (PartitionByInner. citer f v binary-predicate) ;;piter (PartitionInnerIter. citer f fv true v fv) acc (rfn acc piter) _ (when (and (not ignore-leftover?) (.hasNext piter)) (throw (RuntimeException. "Sub-collection was not entirely consumed."))) piter-data @piter] (if (reduced? acc) @acc (if piter-data (recur acc (piter-data 0) (piter-data 1)) acc)))) acc))) Iterable (iterator [this] (PartitionOuterIter. (.iterator ^Iterable (protocols/->iterable coll)) ignore-leftover? f binary-predicate nil)) Seqable (seq [this] (let [ii (clojure.lang.IteratorSeq/create (.iterator this))] (when ii (clojure.core/map vec (clojure.lang.IteratorSeq/create (.iterator this)))))) clojure.lang.Sequential clojure.lang.IHashEq (hasheq [this] (when (== _hasheq 0) (set! _hasheq (long (hash (seq this))))) _hasheq) clojure.lang.IPersistentCollection (count [this] (count (seq this))) (cons [this o] (cons (seq this) o)) (empty [this] PersistentList/EMPTY) (equiv [this o] (if (identical? this o) true (if (instance? clojure.lang.IPersistentCollection o) (clojure.lang.Util/pcequiv (seq this) o) false))) IObj (meta [this] m) (withMeta [this mm] (PartitionBy. f coll ignore-leftover? mm binary-predicate 0)) Object (toString [this] (.toString ^Object (map vec this))) (hashCode [this] (.hasheq this)) (equals [this o] (.equiv this o))) (pp/implement-tostring-print PartitionBy) (defn partition-by "Lazy noncaching version of partition-by. For reducing partitions into a singular value please see [[apply-concat]]. Return value most efficiently implements reduce with a slightly less efficient implementation of Iterable. Unlike clojure.core/partition-by this does not store intermediate elements nor does it build up intermediate containers. This makes it somewhat faster in most contexts. Each sub-collection must be iterated through entirely before the next method of the parent iterator else the result will not be correct. Options: * `:ignore-leftover?` - When true leftover items in the previous iteration do not cause an exception. Defaults to false. * `:binary-predicate` - When provided, use this for equality semantics. Defaults to equiv semantics but in a numeric context it may be useful to have `(== ##NaN ##Nan)`. ```clojure user> ;;incorrect - inner items not iterated and non-caching! user> (into [] (lznc/partition-by identity [1 1 1 2 2 2 3 3 3])) Execution error at ham_fisted.lazy_noncaching.PartitionBy/reduce (lazy_noncaching.clj:514). Sub-collection was not entirely consumed. user> ;;correct - transducing form of into calls vec on each sub-collection user> ;;thus iterating through it entirely. user> (into [] (map vec) (lznc/partition-by identity [1 1 1 2 2 2 3 3 3])) [[1 1 1] [2 2 2] [3 3 3]] user> ;;filter,collect NaN out of sequence user> (lznc/map hamf/vec (lznc/partition-by identity {:binary-predicate (hamf-fn/binary-predicate x y (let [x (double x) y (double y)] (cond (Double/isNaN x) (if (Double/isNaN y) true false) (Double/isNaN y) false :else true))) } [1 2 3 ##NaN ##NaN 3 4 5])) ([1 2 3] [NaN NaN] [3 4 5]) user> (def init-data (vec (lznc/apply-concat (lznc/map #(repeat 100 %) (range 1000))))) #'user/init-data user> (crit/quick-bench (mapv hamf/sum-fast (lznc/partition-by identity init-data))) Execution time mean : 366.915796 µs ... nil user> (crit/quick-bench (mapv hamf/sum-fast (clojure.core/partition-by identity init-data))) Execution time mean : 6.699424 ms ... nil user> (crit/quick-bench (into [] (comp (clojure.core/partition-by identity) (map hamf/sum-fast)) init-data)) Execution time mean : 1.705864 ms ... ```" ([f] (clojure.core/partition-by f)) ([f coll] (partition-by f nil coll)) ([f options coll] (PartitionBy. f coll (boolean (get options :ignore-leftover?)) nil (hamf-fn/binary-predicate-or-null (get options :binary-predicate)) 0))) (defn partition-by-comparator "Partition by a comparator. Sub partitions must be fully consumed before parent is called. An optional timeout can be used if sub partitions are being consumed in a future - the parent iteration will block until the sub partition is fully consumed. Options: * `:timeout-ms` - Timeout in milliseconds - if the sub partition isn't consumed by this time an exception is thrown." ([cmp coll] (partition-by-comparator cmp nil coll)) ([^Comparator cmp options coll] (let [tms (long (get options :timeout-ms 1)) update (fn [{:keys [rest*] :as ctx}] (let [iter (hamf-iter/->iterator (let [ii (deref rest* tms ::timeout)] (when (identical? ii ::timeout) (throw (ex-info "Sub partition not fully consumed" {:timeout-ms tms}))) ii))] (when (hamf-iter/has-next? iter) (let [next-v (hamf-iter/next iter) pred #(== 0 (.compare cmp next-v %))] (hamf-iter/iter-take-while pred (hamf-iter/iter-cons next-v iter))))))] (hamf-iter/seq-once-iterable :data #(update {:rest* (deliver (promise) (hamf-iter/->iterator coll))}) update :data)))) (defn partition-by-cost "Partition a sequence by integer cost. Will produce partitions of average cost '(quot max-cost 2) with some partitions being larger or smaller if the input sequence ends. This function not very lazy with each new partition greedily calculated." [cost-fn ^long max-cost coll] (let [^clojure.lang.IFn$OL cost-fn (if (instance? clojure.lang.IFn$OL cost-fn) cost-fn (fn ^long [o] (long (cost-fn o)))) next-partition (ArrayList.) half-max (quot max-cost 2) update (fn [{:keys [iter half-info cur-cost] :as ctx}] (loop [cur-cost (long (or cur-cost 0)) half-info half-info] (if (hamf-iter/has-next? iter) (let [next-e (.next ^Iterator iter) cur-cost (+ cur-cost (.invokePrim cost-fn next-e)) over-half? (>= cur-cost half-max)] (.add next-partition next-e) (if (< cur-cost max-cost) (recur cur-cost (if (and (nil? half-info) over-half?) [cur-cost (.size next-partition)] half-info)) (let [[half-cost half-idx] half-info next-cost (- cur-cost (long (or half-cost 0))) split-idx (long (or half-idx (.size next-partition))) subl (.subList next-partition 0 split-idx) ctx (assoc ctx :rows (vec subl) :half-cost next-cost :cur-cost next-cost :half-idx (- (.size next-partition) split-idx))] (.clear subl) ctx))) (when-not (.isEmpty next-partition) (let [rv {:rows (vec next-partition)}] (.clear next-partition) rv)))))] (hamf-iter/seq-once-iterable :rows #(update {:iter (hamf-iter/->iterator coll)}) update :rows))) (defn partition-all "Lazy noncaching version of partition-all. When input is random access returns random access result. If input is not random access then similar to [[partition-by]] each sub-collection must be entirely iterated through before requesting the next sub-collection. ```clojure user> (crit/quick-bench (mapv hamf/sum-fast (lznc/partition-all 100 (range 100000)))) Execution time mean : 335.821098 µs nil user> (crit/quick-bench (mapv hamf/sum-fast (partition-all 100 (range 100000)))) Execution time mean : 6.831242 ms nil user> (crit/quick-bench (into [] (comp (partition-all 100) (map hamf/sum-fast)) (range 100000))) Execution time mean : 1.645954 ms nil ```" ([n] (clojure.core/partition-all n)) ([n coll] (partition-all n n coll)) ([^long n ^long step coll] (if (empty? coll) '() (let [ns step] (if-let [coll (as-random-access coll)] (let [n-elems (.size coll) n-batches (quot (+ n-elems (dec ns)) ns)] (if (== n step) (reify IMutList (size [this] (unchecked-int n-batches)) (get [this outer] (when-not (and (>= outer 0) (< outer n-batches)) (throw (IndexOutOfBoundsException.))) (let [sidx (* outer ns) eidx (min n-elems (+ sidx n))] (.subList coll sidx eidx)))) (reify IMutList (size [this] (unchecked-int n-batches)) (get [this outer] (when-not (and (>= outer 0) (< outer n-batches)) (throw (IndexOutOfBoundsException.))) (let [batch-start (* outer ns) batch-n (long (min n (quot (- n-elems batch-start) step)))] (reify IMutList (size [this] (unchecked-int batch-n)) (get [this inner] (.get coll (+ batch-start (* inner step)))))))))) (if (== n step) (let [iter (hamf-iter/->iterator coll) update (fn [sub-iter] (when (hamf-iter/has-next? sub-iter) (throw (RuntimeException. "Sub iterator has more elements left"))) (when (hamf-iter/has-next? iter) (hamf-iter/iter-take n iter)))] (-> (hamf-iter/once-iterable identity #(update nil) update hamf-iter/wrap-iter) (hamf-iter/seq-iterable))) ;;No fastpath here because if step isn't n then that implies caching in the sequence (clojure.core/partition-all n step coll))))))) (defn every? "Faster (in most circumstances) implementation of clojure.core/every?. This can be much faster in the case of primitive arrays of values. Type-hinted functions are best if coll is primitive array - see example. ```clojure user> (type data) [J user> (count data) 100 user> (def vdata (vec data)) #'user/vdata user> (crit/quick-bench (every? (fn [^long v] (> v 80)) data)) Execution time mean : 40.248868 ns nil user> (crit/quick-bench (lznc/every? (fn [^long v] (> v 80)) data)) Execution time mean : 7.601190 ns nil user> (crit/quick-bench (every? (fn [^long v] (< v 80)) vdata)) Execution time mean : 1.269582 µs nil user> (crit/quick-bench (lznc/every? (fn [^long v] (< v 80)) vdata)) Execution time mean : 211.645613 ns nil user> ```" [pred coll] (if-let [coll (as-random-access coll)] (let [ne (.size coll) pred-type (if (instance? IMutList coll) (cond (instance? IFn$LO pred) :int64 (instance? IFn$DO pred) :float64 :else :object) :object)] (case pred-type :int64 (loop [idx 0] (if (< idx ne) (if (.invokePrim ^IFn$LO pred (.getLong ^IMutList coll (unchecked-int idx))) (recur (unchecked-inc idx)) false) true)) :float64 (loop [idx 0] (if (< idx ne) (if (.invokePrim ^IFn$DO pred (.getDouble ^IMutList coll (unchecked-int idx))) (recur (unchecked-inc idx)) false) true)) (loop [idx 0] (if (< idx ne) (if (pred (.get coll (unchecked-int idx))) (recur (unchecked-inc idx)) false) true)))) (cond (instance? IFn$LO pred) (reduce (fn [acc ^long v] (if-not (.invokePrim ^IFn$LO pred v) (reduced false) true)) true coll) (instance? IFn$DO pred) (reduce (fn [acc ^double v] (if-not (.invokePrim ^IFn$DO pred v) (reduced false) true)) true coll) :else (reduce (fn [acc v] (if-not (pred v) (reduced false) true)) true coll)))) (defn cartesian-map "Create a new sequence that is the cartesian join of the input sequence passed through f. Unlike map, f is passed the arguments as a single persistent vector. This is to enable much higher efficiency in the higher-arity applications. For tight numeric loops, see [[ham-fisted.hlet/let]]. The argument vector is mutably updated between function calls so you can't cache it. Use `(into [] args)` or some variation thereof to cache the arguments as is. ```clojure user> (hamf/sum-fast (lznc/cartesian-map #(h/let [[a b c d](lng-fns %)] (-> (+ a b) (+ c) (+ d))) [1 2 3] [4 5 6] [7 8 9] [10 11 12 13 14])) 3645.0 ```" ([f] '()) ([f a] (map #(f [%]) a)) ([f a b] (let [reducer (fn [rfn acc] (let [values (ArrayLists/objectArray (unchecked-int 2)) val-seq (ArrayLists/toList values) inner-reducer (fn [acc bb] (ArrayHelpers/aset values 1 bb) (rfn acc (f val-seq))) outer-reducer (if (instance? IReduceInit b) (fn [acc aa] (ArrayHelpers/aset values 0 aa) ;;In very tight loops the reduce dispatch can take some time. (.reduce ^IReduceInit b inner-reducer acc)) (fn [acc aa] (ArrayHelpers/aset values 0 aa) (reduce inner-reducer acc b)))] (reduce outer-reducer acc a)))] (reify Iterable (iterator [this] (let [a (->iterable a) b (->iterable b) ia (.iterator a) ib (clojure.lang.Box. (.iterator b)) a-v? (clojure.lang.Box. (.hasNext ia)) values (ArrayLists/objectArray (unchecked-int 2)) val-seq (ArrayLists/toList values)] (when (.-val a-v?) (aset values (unchecked-int 0) (.next ia))) (reify Iterator (hasNext [this] (and (.-val a-v?) (.hasNext ^Iterator (.-val ib)))) (next [this] (let [^Iterator iib (.-val ib) _ (aset values (unchecked-int 1) (.next iib)) rv (f val-seq)] (when-not (.hasNext iib) (set! (.-val a-v?) (.hasNext ia)) (when (.-val a-v?) (aset values (unchecked-int 0) (.next ia)) (set! (.-val ib) (.iterator b)))) rv))))) Seqable (seq [this] (LazyChunkedSeq/chunkIteratorSeq (.iterator this))) ITypedReduce (reduce [this rfn acc] (reducer rfn acc))))) ([f a b & args] (let [args (vec (concat [a b] args)) nargs (count args) dnargs (dec nargs)] (reify Iterable (iterator [this] (let [iterables (mapv ->iterable args) iterators (ArrayLists/toList (.toArray ^Collection (map #(.iterator ^Iterable %) iterables))) values-valid? (clojure.lang.Box. false) values (ArrayLists/objectArray (unchecked-int nargs)) val-seq (ArrayLists/toList values) lidx (long (loop [idx 0] (if (and (< idx dnargs) (.hasNext ^Iterator (iterators idx))) (do (aset values idx (.next ^Iterator (iterators idx))) (recur (unchecked-inc idx))) idx)))] (set! (.-val values-valid?) (== lidx dnargs)) (reify Iterator (hasNext [this] (and (.-val values-valid?) (.hasNext ^Iterator (iterators dnargs)))) (next [this] (let [last-iter ^Iterator (iterators dnargs) _ (aset values dnargs (.next last-iter)) rv (f val-seq)] (when-not (.hasNext last-iter) ;;Find the first iterator seaching backward that does have a valid next item (let [nidx (long (loop [idx 1] (if (< idx nargs) (let [ridx (- dnargs idx) ^Iterator iter (iterators ridx)] (if (.hasNext iter) ridx (recur (unchecked-inc idx)))) -1)))] (if (not (== nidx -1)) (do (aset values nidx (.next ^Iterator (iterators nidx))) (loop [idx (unchecked-inc nidx)] (when (< idx nargs) (let [iter (.iterator ^Iterable (iterables idx))] (.set iterators idx iter) (when (< idx dnargs) (aset values idx (.next iter))) (recur (unchecked-inc idx)))))) ;;If there are no more valid iterators (set! (.-val values-valid?) false)))) rv))))) Seqable (seq [this] (LazyChunkedSeq/chunkIteratorSeq (.iterator this))) ITypedReduce (reduce [this rfn acc] (let [values (ArrayLists/objectArray (unchecked-int nargs)) val-seq (ArrayLists/toList values) ;;We could cache the reducer but it wouldn't help in most cases as people aren't going to cache the ;;cartesian map object. reducer (reduce (fn [rrfn ^long idx] (let [ridx (- dnargs idx) reduce-target (args ridx) inner-reducer (if (== idx 0) (fn final-reducer [acc v] (ArrayHelpers/aset values (unchecked-int ridx) v) (rfn acc (f val-seq))) (fn intermediate-reducer [acc v] (ArrayHelpers/aset values (unchecked-int ridx) v) (rrfn acc)))] (if (instance? IReduceInit reduce-target) #(.reduce ^IReduceInit reduce-target inner-reducer %) #(reduce inner-reducer % reduce-target)))) nil (range nargs))] (reducer acc))))))) ================================================ FILE: src/ham_fisted/mut_map.clj ================================================ (ns ham-fisted.mut-map "Functions for working with java's mutable map interface" (:require [ham-fisted.function :as hamf-fn]) (:import [java.util Map Set Collection])) (defn compute! "Compute a new value in a map derived from an existing value. bfn gets passed k, v where k may be nil. If the function returns nil the corresponding key is removed from the map. See [Map.compute](https://docs.oracle.com/javase/8/docs/api/java/util/Map.html#compute-K-java.util.function.BiFunction-) An example `bfn` for counting occurrences would be `#(if % (inc (long %)) 1)`." [m k bfn] (.compute ^Map m k (hamf-fn/->bi-function bfn))) (defn compute-if-present! "Compute a new value if the value already exists and is non-nil in the hashmap. Must use mutable maps. bfn gets passed k, v where v is non-nil. See [Map.computeIfPresent](https://docs.oracle.com/javase/8/docs/api/java/util/Map.html#computeIfPresent-K-java.util.function.BiFunction-)" [m k bfn] (.computeIfPresent ^Map m k (hamf-fn/->bi-function bfn))) (defn compute-if-absent! "Compute a value if absent from the map. Useful for memoize-type operations. Must use mutable maps. bfn gets passed k. See [map.computeIfAbsent](https://docs.oracle.com/javase/8/docs/api/java/util/Map.html#computeIfAbsent-K-java.util.function.Function-)" [m k bfn] (.computeIfAbsent ^Map m k (hamf-fn/->function bfn))) (defn keyset "Return the keyset of the map. This may not be in the same order as (keys m) or (vals m). For hamf maps, this has the same ordering as (keys m). For both hamf and java hashmaps, the returned implementation of java.util.Set has both more utility and better performance than (keys m)." ^Set [^Map m] (.keySet m)) (defn values "Return the values collection of the map. This may not be in the same order as (keys m) or (vals m). For hamf hashmaps, this does have the same order as (vals m)." ^Collection [^Map m] (.values m)) ================================================ FILE: src/ham_fisted/primitive_invoke.clj ================================================ (ns ham-fisted.primitive-invoke "For statically traced calls the Clojure compiler calls the primitive version of type-hinted functions and this makes quite a difference in tight loops. Often times, however, functions are passed by values or returned from if-statements and then you need to explicitly call the primitive overload - this makes that pathway less verbose. Functions must first be check-casted to their primitive types and then calling them will use their primitive overloads avoiding all casting. ```clojure (defn doit [f x y] (let [f (pi/->ddd f)] (loop [x x y y] (if (< x y) (recur (pi/ddd f x y) y) x)))) ```") (defn ->l ^clojure.lang.IFn$L [f] (if (instance? clojure.lang.IFn$L f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$L"))))) (defmacro l [f] `(.invokePrim ~f)) (defn ->d ^clojure.lang.IFn$D [f] (if (instance? clojure.lang.IFn$D f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$D"))))) (defmacro d [f] `(.invokePrim ~f)) (defn ->lo ^clojure.lang.IFn$LO [f] (if (instance? clojure.lang.IFn$LO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LO"))))) (defmacro lo [f arg0] `(.invokePrim ~f ~arg0)) (defn ->do ^clojure.lang.IFn$DO [f] (if (instance? clojure.lang.IFn$DO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DO"))))) (defmacro do [f arg0] `(.invokePrim ~f ~arg0)) (defn ->ol ^clojure.lang.IFn$OL [f] (if (instance? clojure.lang.IFn$OL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OL"))))) (defmacro ol [f arg0] `(.invokePrim ~f ~arg0)) (defn ->ll ^clojure.lang.IFn$LL [f] (if (instance? clojure.lang.IFn$LL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LL"))))) (defmacro ll [f arg0] `(.invokePrim ~f ~arg0)) (defn ->dl ^clojure.lang.IFn$DL [f] (if (instance? clojure.lang.IFn$DL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DL"))))) (defmacro dl [f arg0] `(.invokePrim ~f ~arg0)) (defn ->od ^clojure.lang.IFn$OD [f] (if (instance? clojure.lang.IFn$OD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OD"))))) (defmacro od [f arg0] `(.invokePrim ~f ~arg0)) (defn ->ld ^clojure.lang.IFn$LD [f] (if (instance? clojure.lang.IFn$LD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LD"))))) (defmacro ld [f arg0] `(.invokePrim ~f ~arg0)) (defn ->dd ^clojure.lang.IFn$DD [f] (if (instance? clojure.lang.IFn$DD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DD"))))) (defmacro dd [f arg0] `(.invokePrim ~f ~arg0)) (defn ->olo ^clojure.lang.IFn$OLO [f] (if (instance? clojure.lang.IFn$OLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLO"))))) (defmacro olo [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->odo ^clojure.lang.IFn$ODO [f] (if (instance? clojure.lang.IFn$ODO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODO"))))) (defmacro odo [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->loo ^clojure.lang.IFn$LOO [f] (if (instance? clojure.lang.IFn$LOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOO"))))) (defmacro loo [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->llo ^clojure.lang.IFn$LLO [f] (if (instance? clojure.lang.IFn$LLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLO"))))) (defmacro llo [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->ldo ^clojure.lang.IFn$LDO [f] (if (instance? clojure.lang.IFn$LDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDO"))))) (defmacro ldo [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->doo ^clojure.lang.IFn$DOO [f] (if (instance? clojure.lang.IFn$DOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOO"))))) (defmacro doo [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->dlo ^clojure.lang.IFn$DLO [f] (if (instance? clojure.lang.IFn$DLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLO"))))) (defmacro dlo [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->ddo ^clojure.lang.IFn$DDO [f] (if (instance? clojure.lang.IFn$DDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDO"))))) (defmacro ddo [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->ool ^clojure.lang.IFn$OOL [f] (if (instance? clojure.lang.IFn$OOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOL"))))) (defmacro ool [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->oll ^clojure.lang.IFn$OLL [f] (if (instance? clojure.lang.IFn$OLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLL"))))) (defmacro oll [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->odl ^clojure.lang.IFn$ODL [f] (if (instance? clojure.lang.IFn$ODL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODL"))))) (defmacro odl [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->lol ^clojure.lang.IFn$LOL [f] (if (instance? clojure.lang.IFn$LOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOL"))))) (defmacro lol [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->lll ^clojure.lang.IFn$LLL [f] (if (instance? clojure.lang.IFn$LLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLL"))))) (defmacro lll [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->ldl ^clojure.lang.IFn$LDL [f] (if (instance? clojure.lang.IFn$LDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDL"))))) (defmacro ldl [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->dol ^clojure.lang.IFn$DOL [f] (if (instance? clojure.lang.IFn$DOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOL"))))) (defmacro dol [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->dll ^clojure.lang.IFn$DLL [f] (if (instance? clojure.lang.IFn$DLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLL"))))) (defmacro dll [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->ddl ^clojure.lang.IFn$DDL [f] (if (instance? clojure.lang.IFn$DDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDL"))))) (defmacro ddl [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->ood ^clojure.lang.IFn$OOD [f] (if (instance? clojure.lang.IFn$OOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOD"))))) (defmacro ood [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->old ^clojure.lang.IFn$OLD [f] (if (instance? clojure.lang.IFn$OLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLD"))))) (defmacro old [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->odd ^clojure.lang.IFn$ODD [f] (if (instance? clojure.lang.IFn$ODD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODD"))))) (defmacro odd [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->lod ^clojure.lang.IFn$LOD [f] (if (instance? clojure.lang.IFn$LOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOD"))))) (defmacro lod [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->lld ^clojure.lang.IFn$LLD [f] (if (instance? clojure.lang.IFn$LLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLD"))))) (defmacro lld [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->ldd ^clojure.lang.IFn$LDD [f] (if (instance? clojure.lang.IFn$LDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDD"))))) (defmacro ldd [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->dod ^clojure.lang.IFn$DOD [f] (if (instance? clojure.lang.IFn$DOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOD"))))) (defmacro dod [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->dld ^clojure.lang.IFn$DLD [f] (if (instance? clojure.lang.IFn$DLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLD"))))) (defmacro dld [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->ddd ^clojure.lang.IFn$DDD [f] (if (instance? clojure.lang.IFn$DDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDD"))))) (defmacro ddd [f arg0 arg1] `(.invokePrim ~f ~arg0 ~arg1)) (defn ->oolo ^clojure.lang.IFn$OOLO [f] (if (instance? clojure.lang.IFn$OOLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOLO"))))) (defmacro oolo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->oodo ^clojure.lang.IFn$OODO [f] (if (instance? clojure.lang.IFn$OODO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OODO"))))) (defmacro oodo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->oloo ^clojure.lang.IFn$OLOO [f] (if (instance? clojure.lang.IFn$OLOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLOO"))))) (defmacro oloo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->ollo ^clojure.lang.IFn$OLLO [f] (if (instance? clojure.lang.IFn$OLLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLLO"))))) (defmacro ollo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->oldo ^clojure.lang.IFn$OLDO [f] (if (instance? clojure.lang.IFn$OLDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLDO"))))) (defmacro oldo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->odoo ^clojure.lang.IFn$ODOO [f] (if (instance? clojure.lang.IFn$ODOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODOO"))))) (defmacro odoo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->odlo ^clojure.lang.IFn$ODLO [f] (if (instance? clojure.lang.IFn$ODLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODLO"))))) (defmacro odlo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->oddo ^clojure.lang.IFn$ODDO [f] (if (instance? clojure.lang.IFn$ODDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODDO"))))) (defmacro oddo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->looo ^clojure.lang.IFn$LOOO [f] (if (instance? clojure.lang.IFn$LOOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOOO"))))) (defmacro looo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->lolo ^clojure.lang.IFn$LOLO [f] (if (instance? clojure.lang.IFn$LOLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOLO"))))) (defmacro lolo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->lodo ^clojure.lang.IFn$LODO [f] (if (instance? clojure.lang.IFn$LODO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LODO"))))) (defmacro lodo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->lloo ^clojure.lang.IFn$LLOO [f] (if (instance? clojure.lang.IFn$LLOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLOO"))))) (defmacro lloo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->lllo ^clojure.lang.IFn$LLLO [f] (if (instance? clojure.lang.IFn$LLLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLLO"))))) (defmacro lllo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->lldo ^clojure.lang.IFn$LLDO [f] (if (instance? clojure.lang.IFn$LLDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLDO"))))) (defmacro lldo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->ldoo ^clojure.lang.IFn$LDOO [f] (if (instance? clojure.lang.IFn$LDOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDOO"))))) (defmacro ldoo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->ldlo ^clojure.lang.IFn$LDLO [f] (if (instance? clojure.lang.IFn$LDLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDLO"))))) (defmacro ldlo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->lddo ^clojure.lang.IFn$LDDO [f] (if (instance? clojure.lang.IFn$LDDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDDO"))))) (defmacro lddo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->dooo ^clojure.lang.IFn$DOOO [f] (if (instance? clojure.lang.IFn$DOOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOOO"))))) (defmacro dooo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->dolo ^clojure.lang.IFn$DOLO [f] (if (instance? clojure.lang.IFn$DOLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOLO"))))) (defmacro dolo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->dodo ^clojure.lang.IFn$DODO [f] (if (instance? clojure.lang.IFn$DODO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DODO"))))) (defmacro dodo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->dloo ^clojure.lang.IFn$DLOO [f] (if (instance? clojure.lang.IFn$DLOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLOO"))))) (defmacro dloo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->dllo ^clojure.lang.IFn$DLLO [f] (if (instance? clojure.lang.IFn$DLLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLLO"))))) (defmacro dllo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->dldo ^clojure.lang.IFn$DLDO [f] (if (instance? clojure.lang.IFn$DLDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLDO"))))) (defmacro dldo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->ddoo ^clojure.lang.IFn$DDOO [f] (if (instance? clojure.lang.IFn$DDOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDOO"))))) (defmacro ddoo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->ddlo ^clojure.lang.IFn$DDLO [f] (if (instance? clojure.lang.IFn$DDLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDLO"))))) (defmacro ddlo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->dddo ^clojure.lang.IFn$DDDO [f] (if (instance? clojure.lang.IFn$DDDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDDO"))))) (defmacro dddo [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->oool ^clojure.lang.IFn$OOOL [f] (if (instance? clojure.lang.IFn$OOOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOOL"))))) (defmacro oool [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->ooll ^clojure.lang.IFn$OOLL [f] (if (instance? clojure.lang.IFn$OOLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOLL"))))) (defmacro ooll [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->oodl ^clojure.lang.IFn$OODL [f] (if (instance? clojure.lang.IFn$OODL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OODL"))))) (defmacro oodl [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->olol ^clojure.lang.IFn$OLOL [f] (if (instance? clojure.lang.IFn$OLOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLOL"))))) (defmacro olol [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->olll ^clojure.lang.IFn$OLLL [f] (if (instance? clojure.lang.IFn$OLLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLLL"))))) (defmacro olll [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->oldl ^clojure.lang.IFn$OLDL [f] (if (instance? clojure.lang.IFn$OLDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLDL"))))) (defmacro oldl [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->odol ^clojure.lang.IFn$ODOL [f] (if (instance? clojure.lang.IFn$ODOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODOL"))))) (defmacro odol [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->odll ^clojure.lang.IFn$ODLL [f] (if (instance? clojure.lang.IFn$ODLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODLL"))))) (defmacro odll [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->oddl ^clojure.lang.IFn$ODDL [f] (if (instance? clojure.lang.IFn$ODDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODDL"))))) (defmacro oddl [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->lool ^clojure.lang.IFn$LOOL [f] (if (instance? clojure.lang.IFn$LOOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOOL"))))) (defmacro lool [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->loll ^clojure.lang.IFn$LOLL [f] (if (instance? clojure.lang.IFn$LOLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOLL"))))) (defmacro loll [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->lodl ^clojure.lang.IFn$LODL [f] (if (instance? clojure.lang.IFn$LODL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LODL"))))) (defmacro lodl [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->llol ^clojure.lang.IFn$LLOL [f] (if (instance? clojure.lang.IFn$LLOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLOL"))))) (defmacro llol [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->llll ^clojure.lang.IFn$LLLL [f] (if (instance? clojure.lang.IFn$LLLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLLL"))))) (defmacro llll [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->lldl ^clojure.lang.IFn$LLDL [f] (if (instance? clojure.lang.IFn$LLDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLDL"))))) (defmacro lldl [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->ldol ^clojure.lang.IFn$LDOL [f] (if (instance? clojure.lang.IFn$LDOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDOL"))))) (defmacro ldol [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->ldll ^clojure.lang.IFn$LDLL [f] (if (instance? clojure.lang.IFn$LDLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDLL"))))) (defmacro ldll [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->lddl ^clojure.lang.IFn$LDDL [f] (if (instance? clojure.lang.IFn$LDDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDDL"))))) (defmacro lddl [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->dool ^clojure.lang.IFn$DOOL [f] (if (instance? clojure.lang.IFn$DOOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOOL"))))) (defmacro dool [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->doll ^clojure.lang.IFn$DOLL [f] (if (instance? clojure.lang.IFn$DOLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOLL"))))) (defmacro doll [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->dodl ^clojure.lang.IFn$DODL [f] (if (instance? clojure.lang.IFn$DODL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DODL"))))) (defmacro dodl [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->dlol ^clojure.lang.IFn$DLOL [f] (if (instance? clojure.lang.IFn$DLOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLOL"))))) (defmacro dlol [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->dlll ^clojure.lang.IFn$DLLL [f] (if (instance? clojure.lang.IFn$DLLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLLL"))))) (defmacro dlll [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->dldl ^clojure.lang.IFn$DLDL [f] (if (instance? clojure.lang.IFn$DLDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLDL"))))) (defmacro dldl [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->ddol ^clojure.lang.IFn$DDOL [f] (if (instance? clojure.lang.IFn$DDOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDOL"))))) (defmacro ddol [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->ddll ^clojure.lang.IFn$DDLL [f] (if (instance? clojure.lang.IFn$DDLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDLL"))))) (defmacro ddll [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->dddl ^clojure.lang.IFn$DDDL [f] (if (instance? clojure.lang.IFn$DDDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDDL"))))) (defmacro dddl [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->oood ^clojure.lang.IFn$OOOD [f] (if (instance? clojure.lang.IFn$OOOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOOD"))))) (defmacro oood [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->oold ^clojure.lang.IFn$OOLD [f] (if (instance? clojure.lang.IFn$OOLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOLD"))))) (defmacro oold [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->oodd ^clojure.lang.IFn$OODD [f] (if (instance? clojure.lang.IFn$OODD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OODD"))))) (defmacro oodd [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->olod ^clojure.lang.IFn$OLOD [f] (if (instance? clojure.lang.IFn$OLOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLOD"))))) (defmacro olod [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->olld ^clojure.lang.IFn$OLLD [f] (if (instance? clojure.lang.IFn$OLLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLLD"))))) (defmacro olld [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->oldd ^clojure.lang.IFn$OLDD [f] (if (instance? clojure.lang.IFn$OLDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLDD"))))) (defmacro oldd [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->odod ^clojure.lang.IFn$ODOD [f] (if (instance? clojure.lang.IFn$ODOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODOD"))))) (defmacro odod [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->odld ^clojure.lang.IFn$ODLD [f] (if (instance? clojure.lang.IFn$ODLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODLD"))))) (defmacro odld [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->oddd ^clojure.lang.IFn$ODDD [f] (if (instance? clojure.lang.IFn$ODDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODDD"))))) (defmacro oddd [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->lood ^clojure.lang.IFn$LOOD [f] (if (instance? clojure.lang.IFn$LOOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOOD"))))) (defmacro lood [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->lold ^clojure.lang.IFn$LOLD [f] (if (instance? clojure.lang.IFn$LOLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOLD"))))) (defmacro lold [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->lodd ^clojure.lang.IFn$LODD [f] (if (instance? clojure.lang.IFn$LODD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LODD"))))) (defmacro lodd [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->llod ^clojure.lang.IFn$LLOD [f] (if (instance? clojure.lang.IFn$LLOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLOD"))))) (defmacro llod [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->llld ^clojure.lang.IFn$LLLD [f] (if (instance? clojure.lang.IFn$LLLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLLD"))))) (defmacro llld [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->lldd ^clojure.lang.IFn$LLDD [f] (if (instance? clojure.lang.IFn$LLDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLDD"))))) (defmacro lldd [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->ldod ^clojure.lang.IFn$LDOD [f] (if (instance? clojure.lang.IFn$LDOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDOD"))))) (defmacro ldod [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->ldld ^clojure.lang.IFn$LDLD [f] (if (instance? clojure.lang.IFn$LDLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDLD"))))) (defmacro ldld [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->lddd ^clojure.lang.IFn$LDDD [f] (if (instance? clojure.lang.IFn$LDDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDDD"))))) (defmacro lddd [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->dood ^clojure.lang.IFn$DOOD [f] (if (instance? clojure.lang.IFn$DOOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOOD"))))) (defmacro dood [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->dold ^clojure.lang.IFn$DOLD [f] (if (instance? clojure.lang.IFn$DOLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOLD"))))) (defmacro dold [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->dodd ^clojure.lang.IFn$DODD [f] (if (instance? clojure.lang.IFn$DODD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DODD"))))) (defmacro dodd [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->dlod ^clojure.lang.IFn$DLOD [f] (if (instance? clojure.lang.IFn$DLOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLOD"))))) (defmacro dlod [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->dlld ^clojure.lang.IFn$DLLD [f] (if (instance? clojure.lang.IFn$DLLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLLD"))))) (defmacro dlld [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->dldd ^clojure.lang.IFn$DLDD [f] (if (instance? clojure.lang.IFn$DLDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLDD"))))) (defmacro dldd [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->ddod ^clojure.lang.IFn$DDOD [f] (if (instance? clojure.lang.IFn$DDOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDOD"))))) (defmacro ddod [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->ddld ^clojure.lang.IFn$DDLD [f] (if (instance? clojure.lang.IFn$DDLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDLD"))))) (defmacro ddld [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->dddd ^clojure.lang.IFn$DDDD [f] (if (instance? clojure.lang.IFn$DDDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDDD"))))) (defmacro dddd [f arg0 arg1 arg2] `(.invokePrim ~f ~arg0 ~arg1 ~arg2)) (defn ->ooolo ^clojure.lang.IFn$OOOLO [f] (if (instance? clojure.lang.IFn$OOOLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOOLO"))))) (defmacro ooolo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ooodo ^clojure.lang.IFn$OOODO [f] (if (instance? clojure.lang.IFn$OOODO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOODO"))))) (defmacro ooodo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ooloo ^clojure.lang.IFn$OOLOO [f] (if (instance? clojure.lang.IFn$OOLOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOLOO"))))) (defmacro ooloo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oollo ^clojure.lang.IFn$OOLLO [f] (if (instance? clojure.lang.IFn$OOLLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOLLO"))))) (defmacro oollo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ooldo ^clojure.lang.IFn$OOLDO [f] (if (instance? clojure.lang.IFn$OOLDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOLDO"))))) (defmacro ooldo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oodoo ^clojure.lang.IFn$OODOO [f] (if (instance? clojure.lang.IFn$OODOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OODOO"))))) (defmacro oodoo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oodlo ^clojure.lang.IFn$OODLO [f] (if (instance? clojure.lang.IFn$OODLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OODLO"))))) (defmacro oodlo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ooddo ^clojure.lang.IFn$OODDO [f] (if (instance? clojure.lang.IFn$OODDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OODDO"))))) (defmacro ooddo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->olooo ^clojure.lang.IFn$OLOOO [f] (if (instance? clojure.lang.IFn$OLOOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLOOO"))))) (defmacro olooo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ololo ^clojure.lang.IFn$OLOLO [f] (if (instance? clojure.lang.IFn$OLOLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLOLO"))))) (defmacro ololo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->olodo ^clojure.lang.IFn$OLODO [f] (if (instance? clojure.lang.IFn$OLODO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLODO"))))) (defmacro olodo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->olloo ^clojure.lang.IFn$OLLOO [f] (if (instance? clojure.lang.IFn$OLLOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLLOO"))))) (defmacro olloo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->olllo ^clojure.lang.IFn$OLLLO [f] (if (instance? clojure.lang.IFn$OLLLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLLLO"))))) (defmacro olllo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->olldo ^clojure.lang.IFn$OLLDO [f] (if (instance? clojure.lang.IFn$OLLDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLLDO"))))) (defmacro olldo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oldoo ^clojure.lang.IFn$OLDOO [f] (if (instance? clojure.lang.IFn$OLDOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLDOO"))))) (defmacro oldoo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oldlo ^clojure.lang.IFn$OLDLO [f] (if (instance? clojure.lang.IFn$OLDLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLDLO"))))) (defmacro oldlo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->olddo ^clojure.lang.IFn$OLDDO [f] (if (instance? clojure.lang.IFn$OLDDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLDDO"))))) (defmacro olddo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->odooo ^clojure.lang.IFn$ODOOO [f] (if (instance? clojure.lang.IFn$ODOOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODOOO"))))) (defmacro odooo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->odolo ^clojure.lang.IFn$ODOLO [f] (if (instance? clojure.lang.IFn$ODOLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODOLO"))))) (defmacro odolo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ododo ^clojure.lang.IFn$ODODO [f] (if (instance? clojure.lang.IFn$ODODO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODODO"))))) (defmacro ododo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->odloo ^clojure.lang.IFn$ODLOO [f] (if (instance? clojure.lang.IFn$ODLOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODLOO"))))) (defmacro odloo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->odllo ^clojure.lang.IFn$ODLLO [f] (if (instance? clojure.lang.IFn$ODLLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODLLO"))))) (defmacro odllo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->odldo ^clojure.lang.IFn$ODLDO [f] (if (instance? clojure.lang.IFn$ODLDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODLDO"))))) (defmacro odldo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oddoo ^clojure.lang.IFn$ODDOO [f] (if (instance? clojure.lang.IFn$ODDOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODDOO"))))) (defmacro oddoo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oddlo ^clojure.lang.IFn$ODDLO [f] (if (instance? clojure.lang.IFn$ODDLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODDLO"))))) (defmacro oddlo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->odddo ^clojure.lang.IFn$ODDDO [f] (if (instance? clojure.lang.IFn$ODDDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODDDO"))))) (defmacro odddo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->loooo ^clojure.lang.IFn$LOOOO [f] (if (instance? clojure.lang.IFn$LOOOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOOOO"))))) (defmacro loooo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->loolo ^clojure.lang.IFn$LOOLO [f] (if (instance? clojure.lang.IFn$LOOLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOOLO"))))) (defmacro loolo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->loodo ^clojure.lang.IFn$LOODO [f] (if (instance? clojure.lang.IFn$LOODO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOODO"))))) (defmacro loodo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->loloo ^clojure.lang.IFn$LOLOO [f] (if (instance? clojure.lang.IFn$LOLOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOLOO"))))) (defmacro loloo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lollo ^clojure.lang.IFn$LOLLO [f] (if (instance? clojure.lang.IFn$LOLLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOLLO"))))) (defmacro lollo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->loldo ^clojure.lang.IFn$LOLDO [f] (if (instance? clojure.lang.IFn$LOLDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOLDO"))))) (defmacro loldo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lodoo ^clojure.lang.IFn$LODOO [f] (if (instance? clojure.lang.IFn$LODOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LODOO"))))) (defmacro lodoo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lodlo ^clojure.lang.IFn$LODLO [f] (if (instance? clojure.lang.IFn$LODLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LODLO"))))) (defmacro lodlo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->loddo ^clojure.lang.IFn$LODDO [f] (if (instance? clojure.lang.IFn$LODDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LODDO"))))) (defmacro loddo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->llooo ^clojure.lang.IFn$LLOOO [f] (if (instance? clojure.lang.IFn$LLOOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLOOO"))))) (defmacro llooo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->llolo ^clojure.lang.IFn$LLOLO [f] (if (instance? clojure.lang.IFn$LLOLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLOLO"))))) (defmacro llolo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->llodo ^clojure.lang.IFn$LLODO [f] (if (instance? clojure.lang.IFn$LLODO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLODO"))))) (defmacro llodo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->llloo ^clojure.lang.IFn$LLLOO [f] (if (instance? clojure.lang.IFn$LLLOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLLOO"))))) (defmacro llloo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->llllo ^clojure.lang.IFn$LLLLO [f] (if (instance? clojure.lang.IFn$LLLLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLLLO"))))) (defmacro llllo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->llldo ^clojure.lang.IFn$LLLDO [f] (if (instance? clojure.lang.IFn$LLLDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLLDO"))))) (defmacro llldo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lldoo ^clojure.lang.IFn$LLDOO [f] (if (instance? clojure.lang.IFn$LLDOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLDOO"))))) (defmacro lldoo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lldlo ^clojure.lang.IFn$LLDLO [f] (if (instance? clojure.lang.IFn$LLDLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLDLO"))))) (defmacro lldlo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->llddo ^clojure.lang.IFn$LLDDO [f] (if (instance? clojure.lang.IFn$LLDDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLDDO"))))) (defmacro llddo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ldooo ^clojure.lang.IFn$LDOOO [f] (if (instance? clojure.lang.IFn$LDOOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDOOO"))))) (defmacro ldooo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ldolo ^clojure.lang.IFn$LDOLO [f] (if (instance? clojure.lang.IFn$LDOLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDOLO"))))) (defmacro ldolo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ldodo ^clojure.lang.IFn$LDODO [f] (if (instance? clojure.lang.IFn$LDODO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDODO"))))) (defmacro ldodo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ldloo ^clojure.lang.IFn$LDLOO [f] (if (instance? clojure.lang.IFn$LDLOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDLOO"))))) (defmacro ldloo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ldllo ^clojure.lang.IFn$LDLLO [f] (if (instance? clojure.lang.IFn$LDLLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDLLO"))))) (defmacro ldllo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ldldo ^clojure.lang.IFn$LDLDO [f] (if (instance? clojure.lang.IFn$LDLDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDLDO"))))) (defmacro ldldo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lddoo ^clojure.lang.IFn$LDDOO [f] (if (instance? clojure.lang.IFn$LDDOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDDOO"))))) (defmacro lddoo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lddlo ^clojure.lang.IFn$LDDLO [f] (if (instance? clojure.lang.IFn$LDDLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDDLO"))))) (defmacro lddlo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ldddo ^clojure.lang.IFn$LDDDO [f] (if (instance? clojure.lang.IFn$LDDDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDDDO"))))) (defmacro ldddo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->doooo ^clojure.lang.IFn$DOOOO [f] (if (instance? clojure.lang.IFn$DOOOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOOOO"))))) (defmacro doooo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->doolo ^clojure.lang.IFn$DOOLO [f] (if (instance? clojure.lang.IFn$DOOLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOOLO"))))) (defmacro doolo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->doodo ^clojure.lang.IFn$DOODO [f] (if (instance? clojure.lang.IFn$DOODO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOODO"))))) (defmacro doodo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->doloo ^clojure.lang.IFn$DOLOO [f] (if (instance? clojure.lang.IFn$DOLOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOLOO"))))) (defmacro doloo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dollo ^clojure.lang.IFn$DOLLO [f] (if (instance? clojure.lang.IFn$DOLLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOLLO"))))) (defmacro dollo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->doldo ^clojure.lang.IFn$DOLDO [f] (if (instance? clojure.lang.IFn$DOLDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOLDO"))))) (defmacro doldo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dodoo ^clojure.lang.IFn$DODOO [f] (if (instance? clojure.lang.IFn$DODOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DODOO"))))) (defmacro dodoo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dodlo ^clojure.lang.IFn$DODLO [f] (if (instance? clojure.lang.IFn$DODLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DODLO"))))) (defmacro dodlo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->doddo ^clojure.lang.IFn$DODDO [f] (if (instance? clojure.lang.IFn$DODDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DODDO"))))) (defmacro doddo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dlooo ^clojure.lang.IFn$DLOOO [f] (if (instance? clojure.lang.IFn$DLOOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLOOO"))))) (defmacro dlooo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dlolo ^clojure.lang.IFn$DLOLO [f] (if (instance? clojure.lang.IFn$DLOLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLOLO"))))) (defmacro dlolo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dlodo ^clojure.lang.IFn$DLODO [f] (if (instance? clojure.lang.IFn$DLODO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLODO"))))) (defmacro dlodo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dlloo ^clojure.lang.IFn$DLLOO [f] (if (instance? clojure.lang.IFn$DLLOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLLOO"))))) (defmacro dlloo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dlllo ^clojure.lang.IFn$DLLLO [f] (if (instance? clojure.lang.IFn$DLLLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLLLO"))))) (defmacro dlllo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dlldo ^clojure.lang.IFn$DLLDO [f] (if (instance? clojure.lang.IFn$DLLDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLLDO"))))) (defmacro dlldo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dldoo ^clojure.lang.IFn$DLDOO [f] (if (instance? clojure.lang.IFn$DLDOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLDOO"))))) (defmacro dldoo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dldlo ^clojure.lang.IFn$DLDLO [f] (if (instance? clojure.lang.IFn$DLDLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLDLO"))))) (defmacro dldlo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dlddo ^clojure.lang.IFn$DLDDO [f] (if (instance? clojure.lang.IFn$DLDDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLDDO"))))) (defmacro dlddo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ddooo ^clojure.lang.IFn$DDOOO [f] (if (instance? clojure.lang.IFn$DDOOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDOOO"))))) (defmacro ddooo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ddolo ^clojure.lang.IFn$DDOLO [f] (if (instance? clojure.lang.IFn$DDOLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDOLO"))))) (defmacro ddolo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ddodo ^clojure.lang.IFn$DDODO [f] (if (instance? clojure.lang.IFn$DDODO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDODO"))))) (defmacro ddodo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ddloo ^clojure.lang.IFn$DDLOO [f] (if (instance? clojure.lang.IFn$DDLOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDLOO"))))) (defmacro ddloo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ddllo ^clojure.lang.IFn$DDLLO [f] (if (instance? clojure.lang.IFn$DDLLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDLLO"))))) (defmacro ddllo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ddldo ^clojure.lang.IFn$DDLDO [f] (if (instance? clojure.lang.IFn$DDLDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDLDO"))))) (defmacro ddldo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dddoo ^clojure.lang.IFn$DDDOO [f] (if (instance? clojure.lang.IFn$DDDOO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDDOO"))))) (defmacro dddoo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dddlo ^clojure.lang.IFn$DDDLO [f] (if (instance? clojure.lang.IFn$DDDLO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDDLO"))))) (defmacro dddlo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ddddo ^clojure.lang.IFn$DDDDO [f] (if (instance? clojure.lang.IFn$DDDDO f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDDDO"))))) (defmacro ddddo [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ooool ^clojure.lang.IFn$OOOOL [f] (if (instance? clojure.lang.IFn$OOOOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOOOL"))))) (defmacro ooool [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oooll ^clojure.lang.IFn$OOOLL [f] (if (instance? clojure.lang.IFn$OOOLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOOLL"))))) (defmacro oooll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ooodl ^clojure.lang.IFn$OOODL [f] (if (instance? clojure.lang.IFn$OOODL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOODL"))))) (defmacro ooodl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oolol ^clojure.lang.IFn$OOLOL [f] (if (instance? clojure.lang.IFn$OOLOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOLOL"))))) (defmacro oolol [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oolll ^clojure.lang.IFn$OOLLL [f] (if (instance? clojure.lang.IFn$OOLLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOLLL"))))) (defmacro oolll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ooldl ^clojure.lang.IFn$OOLDL [f] (if (instance? clojure.lang.IFn$OOLDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOLDL"))))) (defmacro ooldl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oodol ^clojure.lang.IFn$OODOL [f] (if (instance? clojure.lang.IFn$OODOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OODOL"))))) (defmacro oodol [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oodll ^clojure.lang.IFn$OODLL [f] (if (instance? clojure.lang.IFn$OODLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OODLL"))))) (defmacro oodll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ooddl ^clojure.lang.IFn$OODDL [f] (if (instance? clojure.lang.IFn$OODDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OODDL"))))) (defmacro ooddl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->olool ^clojure.lang.IFn$OLOOL [f] (if (instance? clojure.lang.IFn$OLOOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLOOL"))))) (defmacro olool [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ololl ^clojure.lang.IFn$OLOLL [f] (if (instance? clojure.lang.IFn$OLOLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLOLL"))))) (defmacro ololl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->olodl ^clojure.lang.IFn$OLODL [f] (if (instance? clojure.lang.IFn$OLODL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLODL"))))) (defmacro olodl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ollol ^clojure.lang.IFn$OLLOL [f] (if (instance? clojure.lang.IFn$OLLOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLLOL"))))) (defmacro ollol [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ollll ^clojure.lang.IFn$OLLLL [f] (if (instance? clojure.lang.IFn$OLLLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLLLL"))))) (defmacro ollll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->olldl ^clojure.lang.IFn$OLLDL [f] (if (instance? clojure.lang.IFn$OLLDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLLDL"))))) (defmacro olldl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oldol ^clojure.lang.IFn$OLDOL [f] (if (instance? clojure.lang.IFn$OLDOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLDOL"))))) (defmacro oldol [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oldll ^clojure.lang.IFn$OLDLL [f] (if (instance? clojure.lang.IFn$OLDLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLDLL"))))) (defmacro oldll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->olddl ^clojure.lang.IFn$OLDDL [f] (if (instance? clojure.lang.IFn$OLDDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLDDL"))))) (defmacro olddl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->odool ^clojure.lang.IFn$ODOOL [f] (if (instance? clojure.lang.IFn$ODOOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODOOL"))))) (defmacro odool [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->odoll ^clojure.lang.IFn$ODOLL [f] (if (instance? clojure.lang.IFn$ODOLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODOLL"))))) (defmacro odoll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ododl ^clojure.lang.IFn$ODODL [f] (if (instance? clojure.lang.IFn$ODODL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODODL"))))) (defmacro ododl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->odlol ^clojure.lang.IFn$ODLOL [f] (if (instance? clojure.lang.IFn$ODLOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODLOL"))))) (defmacro odlol [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->odlll ^clojure.lang.IFn$ODLLL [f] (if (instance? clojure.lang.IFn$ODLLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODLLL"))))) (defmacro odlll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->odldl ^clojure.lang.IFn$ODLDL [f] (if (instance? clojure.lang.IFn$ODLDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODLDL"))))) (defmacro odldl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oddol ^clojure.lang.IFn$ODDOL [f] (if (instance? clojure.lang.IFn$ODDOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODDOL"))))) (defmacro oddol [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oddll ^clojure.lang.IFn$ODDLL [f] (if (instance? clojure.lang.IFn$ODDLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODDLL"))))) (defmacro oddll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->odddl ^clojure.lang.IFn$ODDDL [f] (if (instance? clojure.lang.IFn$ODDDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODDDL"))))) (defmacro odddl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->loool ^clojure.lang.IFn$LOOOL [f] (if (instance? clojure.lang.IFn$LOOOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOOOL"))))) (defmacro loool [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->looll ^clojure.lang.IFn$LOOLL [f] (if (instance? clojure.lang.IFn$LOOLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOOLL"))))) (defmacro looll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->loodl ^clojure.lang.IFn$LOODL [f] (if (instance? clojure.lang.IFn$LOODL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOODL"))))) (defmacro loodl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lolol ^clojure.lang.IFn$LOLOL [f] (if (instance? clojure.lang.IFn$LOLOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOLOL"))))) (defmacro lolol [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lolll ^clojure.lang.IFn$LOLLL [f] (if (instance? clojure.lang.IFn$LOLLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOLLL"))))) (defmacro lolll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->loldl ^clojure.lang.IFn$LOLDL [f] (if (instance? clojure.lang.IFn$LOLDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOLDL"))))) (defmacro loldl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lodol ^clojure.lang.IFn$LODOL [f] (if (instance? clojure.lang.IFn$LODOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LODOL"))))) (defmacro lodol [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lodll ^clojure.lang.IFn$LODLL [f] (if (instance? clojure.lang.IFn$LODLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LODLL"))))) (defmacro lodll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->loddl ^clojure.lang.IFn$LODDL [f] (if (instance? clojure.lang.IFn$LODDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LODDL"))))) (defmacro loddl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->llool ^clojure.lang.IFn$LLOOL [f] (if (instance? clojure.lang.IFn$LLOOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLOOL"))))) (defmacro llool [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lloll ^clojure.lang.IFn$LLOLL [f] (if (instance? clojure.lang.IFn$LLOLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLOLL"))))) (defmacro lloll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->llodl ^clojure.lang.IFn$LLODL [f] (if (instance? clojure.lang.IFn$LLODL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLODL"))))) (defmacro llodl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lllol ^clojure.lang.IFn$LLLOL [f] (if (instance? clojure.lang.IFn$LLLOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLLOL"))))) (defmacro lllol [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lllll ^clojure.lang.IFn$LLLLL [f] (if (instance? clojure.lang.IFn$LLLLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLLLL"))))) (defmacro lllll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->llldl ^clojure.lang.IFn$LLLDL [f] (if (instance? clojure.lang.IFn$LLLDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLLDL"))))) (defmacro llldl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lldol ^clojure.lang.IFn$LLDOL [f] (if (instance? clojure.lang.IFn$LLDOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLDOL"))))) (defmacro lldol [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lldll ^clojure.lang.IFn$LLDLL [f] (if (instance? clojure.lang.IFn$LLDLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLDLL"))))) (defmacro lldll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->llddl ^clojure.lang.IFn$LLDDL [f] (if (instance? clojure.lang.IFn$LLDDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLDDL"))))) (defmacro llddl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ldool ^clojure.lang.IFn$LDOOL [f] (if (instance? clojure.lang.IFn$LDOOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDOOL"))))) (defmacro ldool [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ldoll ^clojure.lang.IFn$LDOLL [f] (if (instance? clojure.lang.IFn$LDOLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDOLL"))))) (defmacro ldoll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ldodl ^clojure.lang.IFn$LDODL [f] (if (instance? clojure.lang.IFn$LDODL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDODL"))))) (defmacro ldodl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ldlol ^clojure.lang.IFn$LDLOL [f] (if (instance? clojure.lang.IFn$LDLOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDLOL"))))) (defmacro ldlol [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ldlll ^clojure.lang.IFn$LDLLL [f] (if (instance? clojure.lang.IFn$LDLLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDLLL"))))) (defmacro ldlll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ldldl ^clojure.lang.IFn$LDLDL [f] (if (instance? clojure.lang.IFn$LDLDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDLDL"))))) (defmacro ldldl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lddol ^clojure.lang.IFn$LDDOL [f] (if (instance? clojure.lang.IFn$LDDOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDDOL"))))) (defmacro lddol [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lddll ^clojure.lang.IFn$LDDLL [f] (if (instance? clojure.lang.IFn$LDDLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDDLL"))))) (defmacro lddll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ldddl ^clojure.lang.IFn$LDDDL [f] (if (instance? clojure.lang.IFn$LDDDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDDDL"))))) (defmacro ldddl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->doool ^clojure.lang.IFn$DOOOL [f] (if (instance? clojure.lang.IFn$DOOOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOOOL"))))) (defmacro doool [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dooll ^clojure.lang.IFn$DOOLL [f] (if (instance? clojure.lang.IFn$DOOLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOOLL"))))) (defmacro dooll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->doodl ^clojure.lang.IFn$DOODL [f] (if (instance? clojure.lang.IFn$DOODL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOODL"))))) (defmacro doodl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dolol ^clojure.lang.IFn$DOLOL [f] (if (instance? clojure.lang.IFn$DOLOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOLOL"))))) (defmacro dolol [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dolll ^clojure.lang.IFn$DOLLL [f] (if (instance? clojure.lang.IFn$DOLLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOLLL"))))) (defmacro dolll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->doldl ^clojure.lang.IFn$DOLDL [f] (if (instance? clojure.lang.IFn$DOLDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOLDL"))))) (defmacro doldl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dodol ^clojure.lang.IFn$DODOL [f] (if (instance? clojure.lang.IFn$DODOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DODOL"))))) (defmacro dodol [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dodll ^clojure.lang.IFn$DODLL [f] (if (instance? clojure.lang.IFn$DODLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DODLL"))))) (defmacro dodll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->doddl ^clojure.lang.IFn$DODDL [f] (if (instance? clojure.lang.IFn$DODDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DODDL"))))) (defmacro doddl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dlool ^clojure.lang.IFn$DLOOL [f] (if (instance? clojure.lang.IFn$DLOOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLOOL"))))) (defmacro dlool [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dloll ^clojure.lang.IFn$DLOLL [f] (if (instance? clojure.lang.IFn$DLOLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLOLL"))))) (defmacro dloll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dlodl ^clojure.lang.IFn$DLODL [f] (if (instance? clojure.lang.IFn$DLODL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLODL"))))) (defmacro dlodl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dllol ^clojure.lang.IFn$DLLOL [f] (if (instance? clojure.lang.IFn$DLLOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLLOL"))))) (defmacro dllol [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dllll ^clojure.lang.IFn$DLLLL [f] (if (instance? clojure.lang.IFn$DLLLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLLLL"))))) (defmacro dllll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dlldl ^clojure.lang.IFn$DLLDL [f] (if (instance? clojure.lang.IFn$DLLDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLLDL"))))) (defmacro dlldl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dldol ^clojure.lang.IFn$DLDOL [f] (if (instance? clojure.lang.IFn$DLDOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLDOL"))))) (defmacro dldol [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dldll ^clojure.lang.IFn$DLDLL [f] (if (instance? clojure.lang.IFn$DLDLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLDLL"))))) (defmacro dldll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dlddl ^clojure.lang.IFn$DLDDL [f] (if (instance? clojure.lang.IFn$DLDDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLDDL"))))) (defmacro dlddl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ddool ^clojure.lang.IFn$DDOOL [f] (if (instance? clojure.lang.IFn$DDOOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDOOL"))))) (defmacro ddool [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ddoll ^clojure.lang.IFn$DDOLL [f] (if (instance? clojure.lang.IFn$DDOLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDOLL"))))) (defmacro ddoll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ddodl ^clojure.lang.IFn$DDODL [f] (if (instance? clojure.lang.IFn$DDODL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDODL"))))) (defmacro ddodl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ddlol ^clojure.lang.IFn$DDLOL [f] (if (instance? clojure.lang.IFn$DDLOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDLOL"))))) (defmacro ddlol [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ddlll ^clojure.lang.IFn$DDLLL [f] (if (instance? clojure.lang.IFn$DDLLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDLLL"))))) (defmacro ddlll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ddldl ^clojure.lang.IFn$DDLDL [f] (if (instance? clojure.lang.IFn$DDLDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDLDL"))))) (defmacro ddldl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dddol ^clojure.lang.IFn$DDDOL [f] (if (instance? clojure.lang.IFn$DDDOL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDDOL"))))) (defmacro dddol [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dddll ^clojure.lang.IFn$DDDLL [f] (if (instance? clojure.lang.IFn$DDDLL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDDLL"))))) (defmacro dddll [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ddddl ^clojure.lang.IFn$DDDDL [f] (if (instance? clojure.lang.IFn$DDDDL f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDDDL"))))) (defmacro ddddl [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ooood ^clojure.lang.IFn$OOOOD [f] (if (instance? clojure.lang.IFn$OOOOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOOOD"))))) (defmacro ooood [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ooold ^clojure.lang.IFn$OOOLD [f] (if (instance? clojure.lang.IFn$OOOLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOOLD"))))) (defmacro ooold [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ooodd ^clojure.lang.IFn$OOODD [f] (if (instance? clojure.lang.IFn$OOODD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOODD"))))) (defmacro ooodd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oolod ^clojure.lang.IFn$OOLOD [f] (if (instance? clojure.lang.IFn$OOLOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOLOD"))))) (defmacro oolod [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oolld ^clojure.lang.IFn$OOLLD [f] (if (instance? clojure.lang.IFn$OOLLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOLLD"))))) (defmacro oolld [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ooldd ^clojure.lang.IFn$OOLDD [f] (if (instance? clojure.lang.IFn$OOLDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OOLDD"))))) (defmacro ooldd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oodod ^clojure.lang.IFn$OODOD [f] (if (instance? clojure.lang.IFn$OODOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OODOD"))))) (defmacro oodod [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oodld ^clojure.lang.IFn$OODLD [f] (if (instance? clojure.lang.IFn$OODLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OODLD"))))) (defmacro oodld [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ooddd ^clojure.lang.IFn$OODDD [f] (if (instance? clojure.lang.IFn$OODDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OODDD"))))) (defmacro ooddd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->olood ^clojure.lang.IFn$OLOOD [f] (if (instance? clojure.lang.IFn$OLOOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLOOD"))))) (defmacro olood [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->olold ^clojure.lang.IFn$OLOLD [f] (if (instance? clojure.lang.IFn$OLOLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLOLD"))))) (defmacro olold [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->olodd ^clojure.lang.IFn$OLODD [f] (if (instance? clojure.lang.IFn$OLODD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLODD"))))) (defmacro olodd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ollod ^clojure.lang.IFn$OLLOD [f] (if (instance? clojure.lang.IFn$OLLOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLLOD"))))) (defmacro ollod [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ollld ^clojure.lang.IFn$OLLLD [f] (if (instance? clojure.lang.IFn$OLLLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLLLD"))))) (defmacro ollld [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->olldd ^clojure.lang.IFn$OLLDD [f] (if (instance? clojure.lang.IFn$OLLDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLLDD"))))) (defmacro olldd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oldod ^clojure.lang.IFn$OLDOD [f] (if (instance? clojure.lang.IFn$OLDOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLDOD"))))) (defmacro oldod [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oldld ^clojure.lang.IFn$OLDLD [f] (if (instance? clojure.lang.IFn$OLDLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLDLD"))))) (defmacro oldld [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->olddd ^clojure.lang.IFn$OLDDD [f] (if (instance? clojure.lang.IFn$OLDDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$OLDDD"))))) (defmacro olddd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->odood ^clojure.lang.IFn$ODOOD [f] (if (instance? clojure.lang.IFn$ODOOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODOOD"))))) (defmacro odood [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->odold ^clojure.lang.IFn$ODOLD [f] (if (instance? clojure.lang.IFn$ODOLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODOLD"))))) (defmacro odold [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ododd ^clojure.lang.IFn$ODODD [f] (if (instance? clojure.lang.IFn$ODODD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODODD"))))) (defmacro ododd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->odlod ^clojure.lang.IFn$ODLOD [f] (if (instance? clojure.lang.IFn$ODLOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODLOD"))))) (defmacro odlod [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->odlld ^clojure.lang.IFn$ODLLD [f] (if (instance? clojure.lang.IFn$ODLLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODLLD"))))) (defmacro odlld [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->odldd ^clojure.lang.IFn$ODLDD [f] (if (instance? clojure.lang.IFn$ODLDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODLDD"))))) (defmacro odldd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oddod ^clojure.lang.IFn$ODDOD [f] (if (instance? clojure.lang.IFn$ODDOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODDOD"))))) (defmacro oddod [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->oddld ^clojure.lang.IFn$ODDLD [f] (if (instance? clojure.lang.IFn$ODDLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODDLD"))))) (defmacro oddld [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->odddd ^clojure.lang.IFn$ODDDD [f] (if (instance? clojure.lang.IFn$ODDDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$ODDDD"))))) (defmacro odddd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->loood ^clojure.lang.IFn$LOOOD [f] (if (instance? clojure.lang.IFn$LOOOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOOOD"))))) (defmacro loood [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->loold ^clojure.lang.IFn$LOOLD [f] (if (instance? clojure.lang.IFn$LOOLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOOLD"))))) (defmacro loold [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->loodd ^clojure.lang.IFn$LOODD [f] (if (instance? clojure.lang.IFn$LOODD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOODD"))))) (defmacro loodd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lolod ^clojure.lang.IFn$LOLOD [f] (if (instance? clojure.lang.IFn$LOLOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOLOD"))))) (defmacro lolod [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lolld ^clojure.lang.IFn$LOLLD [f] (if (instance? clojure.lang.IFn$LOLLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOLLD"))))) (defmacro lolld [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->loldd ^clojure.lang.IFn$LOLDD [f] (if (instance? clojure.lang.IFn$LOLDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LOLDD"))))) (defmacro loldd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lodod ^clojure.lang.IFn$LODOD [f] (if (instance? clojure.lang.IFn$LODOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LODOD"))))) (defmacro lodod [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lodld ^clojure.lang.IFn$LODLD [f] (if (instance? clojure.lang.IFn$LODLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LODLD"))))) (defmacro lodld [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->loddd ^clojure.lang.IFn$LODDD [f] (if (instance? clojure.lang.IFn$LODDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LODDD"))))) (defmacro loddd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->llood ^clojure.lang.IFn$LLOOD [f] (if (instance? clojure.lang.IFn$LLOOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLOOD"))))) (defmacro llood [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->llold ^clojure.lang.IFn$LLOLD [f] (if (instance? clojure.lang.IFn$LLOLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLOLD"))))) (defmacro llold [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->llodd ^clojure.lang.IFn$LLODD [f] (if (instance? clojure.lang.IFn$LLODD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLODD"))))) (defmacro llodd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lllod ^clojure.lang.IFn$LLLOD [f] (if (instance? clojure.lang.IFn$LLLOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLLOD"))))) (defmacro lllod [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lllld ^clojure.lang.IFn$LLLLD [f] (if (instance? clojure.lang.IFn$LLLLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLLLD"))))) (defmacro lllld [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->llldd ^clojure.lang.IFn$LLLDD [f] (if (instance? clojure.lang.IFn$LLLDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLLDD"))))) (defmacro llldd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lldod ^clojure.lang.IFn$LLDOD [f] (if (instance? clojure.lang.IFn$LLDOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLDOD"))))) (defmacro lldod [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lldld ^clojure.lang.IFn$LLDLD [f] (if (instance? clojure.lang.IFn$LLDLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLDLD"))))) (defmacro lldld [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->llddd ^clojure.lang.IFn$LLDDD [f] (if (instance? clojure.lang.IFn$LLDDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LLDDD"))))) (defmacro llddd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ldood ^clojure.lang.IFn$LDOOD [f] (if (instance? clojure.lang.IFn$LDOOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDOOD"))))) (defmacro ldood [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ldold ^clojure.lang.IFn$LDOLD [f] (if (instance? clojure.lang.IFn$LDOLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDOLD"))))) (defmacro ldold [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ldodd ^clojure.lang.IFn$LDODD [f] (if (instance? clojure.lang.IFn$LDODD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDODD"))))) (defmacro ldodd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ldlod ^clojure.lang.IFn$LDLOD [f] (if (instance? clojure.lang.IFn$LDLOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDLOD"))))) (defmacro ldlod [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ldlld ^clojure.lang.IFn$LDLLD [f] (if (instance? clojure.lang.IFn$LDLLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDLLD"))))) (defmacro ldlld [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ldldd ^clojure.lang.IFn$LDLDD [f] (if (instance? clojure.lang.IFn$LDLDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDLDD"))))) (defmacro ldldd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lddod ^clojure.lang.IFn$LDDOD [f] (if (instance? clojure.lang.IFn$LDDOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDDOD"))))) (defmacro lddod [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->lddld ^clojure.lang.IFn$LDDLD [f] (if (instance? clojure.lang.IFn$LDDLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDDLD"))))) (defmacro lddld [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ldddd ^clojure.lang.IFn$LDDDD [f] (if (instance? clojure.lang.IFn$LDDDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$LDDDD"))))) (defmacro ldddd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->doood ^clojure.lang.IFn$DOOOD [f] (if (instance? clojure.lang.IFn$DOOOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOOOD"))))) (defmacro doood [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->doold ^clojure.lang.IFn$DOOLD [f] (if (instance? clojure.lang.IFn$DOOLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOOLD"))))) (defmacro doold [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->doodd ^clojure.lang.IFn$DOODD [f] (if (instance? clojure.lang.IFn$DOODD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOODD"))))) (defmacro doodd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dolod ^clojure.lang.IFn$DOLOD [f] (if (instance? clojure.lang.IFn$DOLOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOLOD"))))) (defmacro dolod [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dolld ^clojure.lang.IFn$DOLLD [f] (if (instance? clojure.lang.IFn$DOLLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOLLD"))))) (defmacro dolld [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->doldd ^clojure.lang.IFn$DOLDD [f] (if (instance? clojure.lang.IFn$DOLDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DOLDD"))))) (defmacro doldd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dodod ^clojure.lang.IFn$DODOD [f] (if (instance? clojure.lang.IFn$DODOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DODOD"))))) (defmacro dodod [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dodld ^clojure.lang.IFn$DODLD [f] (if (instance? clojure.lang.IFn$DODLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DODLD"))))) (defmacro dodld [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->doddd ^clojure.lang.IFn$DODDD [f] (if (instance? clojure.lang.IFn$DODDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DODDD"))))) (defmacro doddd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dlood ^clojure.lang.IFn$DLOOD [f] (if (instance? clojure.lang.IFn$DLOOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLOOD"))))) (defmacro dlood [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dlold ^clojure.lang.IFn$DLOLD [f] (if (instance? clojure.lang.IFn$DLOLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLOLD"))))) (defmacro dlold [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dlodd ^clojure.lang.IFn$DLODD [f] (if (instance? clojure.lang.IFn$DLODD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLODD"))))) (defmacro dlodd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dllod ^clojure.lang.IFn$DLLOD [f] (if (instance? clojure.lang.IFn$DLLOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLLOD"))))) (defmacro dllod [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dllld ^clojure.lang.IFn$DLLLD [f] (if (instance? clojure.lang.IFn$DLLLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLLLD"))))) (defmacro dllld [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dlldd ^clojure.lang.IFn$DLLDD [f] (if (instance? clojure.lang.IFn$DLLDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLLDD"))))) (defmacro dlldd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dldod ^clojure.lang.IFn$DLDOD [f] (if (instance? clojure.lang.IFn$DLDOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLDOD"))))) (defmacro dldod [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dldld ^clojure.lang.IFn$DLDLD [f] (if (instance? clojure.lang.IFn$DLDLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLDLD"))))) (defmacro dldld [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dlddd ^clojure.lang.IFn$DLDDD [f] (if (instance? clojure.lang.IFn$DLDDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DLDDD"))))) (defmacro dlddd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ddood ^clojure.lang.IFn$DDOOD [f] (if (instance? clojure.lang.IFn$DDOOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDOOD"))))) (defmacro ddood [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ddold ^clojure.lang.IFn$DDOLD [f] (if (instance? clojure.lang.IFn$DDOLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDOLD"))))) (defmacro ddold [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ddodd ^clojure.lang.IFn$DDODD [f] (if (instance? clojure.lang.IFn$DDODD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDODD"))))) (defmacro ddodd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ddlod ^clojure.lang.IFn$DDLOD [f] (if (instance? clojure.lang.IFn$DDLOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDLOD"))))) (defmacro ddlod [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ddlld ^clojure.lang.IFn$DDLLD [f] (if (instance? clojure.lang.IFn$DDLLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDLLD"))))) (defmacro ddlld [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ddldd ^clojure.lang.IFn$DDLDD [f] (if (instance? clojure.lang.IFn$DDLDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDLDD"))))) (defmacro ddldd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dddod ^clojure.lang.IFn$DDDOD [f] (if (instance? clojure.lang.IFn$DDDOD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDDOD"))))) (defmacro dddod [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->dddld ^clojure.lang.IFn$DDDLD [f] (if (instance? clojure.lang.IFn$DDDLD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDDLD"))))) (defmacro dddld [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) (defn ->ddddd ^clojure.lang.IFn$DDDDD [f] (if (instance? clojure.lang.IFn$DDDDD f) f (throw (RuntimeException. (str f " is not an instance ofclojure.lang.IFn$DDDDD"))))) (defmacro ddddd [f arg0 arg1 arg2 arg3] `(.invokePrim ~f ~arg0 ~arg1 ~arg2 ~arg3)) ================================================ FILE: src/ham_fisted/print.clj ================================================ (ns ham-fisted.print) (defmacro implement-tostring-print "Implement tostring printing for a particular type name." [typename] `(.addMethod ~(with-meta 'print-method {:tag 'clojure.lang.MultiFn}) ~typename ham_fisted.Reductions/ToStringPrint)) ================================================ FILE: src/ham_fisted/process.clj ================================================ (ns ham-fisted.process (:require [ham-fisted.iterator :as hamf-iter] [ham-fisted.reduce :as hamf-rf])) (defn stream->strings ([input] (stream->strings input 256 (java.nio.charset.Charset/defaultCharset))) ([^java.io.InputStream input bufsize ^java.nio.charset.Charset charset] (let [buffer (byte-array bufsize)] (hamf-iter/once-iterable #(let [size (long (try (.read input buffer) (catch Exception e 0)))] (when (pos? size) (String. buffer 0 size charset))))))) (defn- strip-trailing ^String [data] (.stripTrailing (str data))) (defn- naive-split-lines [^String data] (let [nl (int \newline)] (loop [data data nn (.indexOf data nl) rv []] (if (neg? nn) [rv data] (let [rv (conj rv (.substring data 0 nn)) data (.substring data (inc nn))] (recur data (.indexOf data nl) rv)))))) (defn out-rf [print?] (fn do-out-rf ([] {:temp-buf (StringBuilder.) :total-buf (StringBuilder.)}) ([{:keys [^StringBuilder temp-buf ^StringBuilder total-buf] :as acc} ^String data] (.append temp-buf data) (.append total-buf data) (let [temp-str (.toString temp-buf) [strs leftover] (naive-split-lines (.toString temp-buf))] (when print? (run! println (map strip-trailing strs))) (.delete temp-buf (int 0) (int (.length temp-buf))) (.append temp-buf leftover)) acc) ([{:keys [^StringBuilder temp-buf ^StringBuilder total-buf] :as acc}] (when print? (println (.toString temp-buf))) (.toString total-buf)))) (def ^{:doc "Print process output using println. Example process output handler. Returns total output as a string to when finalized."} println-rf (out-rf true)) (def ^{:doc "Returns total output as a string to when finalized."} quiet-rf (out-rf false)) (def ^{:doc "Record all the strings and save them to a vector"} record-rf (fn record-rf ([] (transient [])) ([acc v] (conj! acc v)) ([acc] (persistent! acc)))) (defn destroy-forcibly! "Destroy the process handle's process forcibly." [^java.lang.ProcessHandle proc-hdl] (.destroyForcibly proc-hdl)) (defn process-descendants "Get the first descendants of a process handle." [^java.lang.ProcessHandle proc-hdl] (-> (reify Iterable (iterator [this] (.iterator (.descendants proc-hdl)))) (vec))) (defn launch "Launch a proccess. * cmd-line string command line. * stdout-hdrl, stderr-hdlr - transduce-style rf functions that receive each string read from stdout and stderr respectively. Returns `{:keys [^java.lang.ProcessHandle proc-hdl wait-or-kill]}`: * `proc-hdl` - java.lang.ProcessHandle * `wait-or-kill` - function that has two arities: 1. (proc) - kill the process returning any output as {:out :err}. 2. (proc time-ms timeout-symbol) - wait specified time for process to terminate returning either the timeout symbol or {:out :err}. Example: ```clojure ham-fisted.process> (launch \"ls -al\" {:print-cmd-line? false}) {:proc-hdl #object[java.lang.ProcessHandleImpl 0x7f8e8742 \"31019\"], :wait-or-kill #function[ham-fisted.process/launch/wait-or-kill--65545]} ... ham-fisted.process> (def result ((:wait-or-kill *1))) #'ham-fisted.process/result ham-fisted.process> (keys result) (:out :err) ```" ([cmd-line] (launch cmd-line {})) ([^String cmd-line {:keys [stdout-hdlr stderr-hdlr print-cmd-line?] :or {print-cmd-line? true}}] (when print-cmd-line? (println "launch-process:" cmd-line)) (let [proc (.exec (Runtime/getRuntime) cmd-line) phandle (.toHandle proc) exit-future (.onExit phandle) stdout-hdlr (or stdout-hdlr println-rf) stderr-hdlr (or stderr-hdlr println-rf) _ (.close (.getOutputStream proc)) stdout (.getInputStream proc) stderr (.getErrorStream proc) out (future (hamf-rf/reduce-reducer stdout-hdlr (stream->strings stdout))) err (future (hamf-rf/reduce-reducer stderr-hdlr (stream->strings stderr))) cleanup (fn [close?] (when close? ;;close the streams to force termination as the process hasn't exited (.close stdout) (.close stderr)) {:out @out :err @err})] {:proc-hdl phandle :wait-or-kill (fn wait-or-kill ([] (let [desc (process-descendants phandle)] (run! destroy-forcibly! desc) (destroy-forcibly! phandle) (cleanup true))) ([^long time-ms timeout-symbol] (if-let [rv (try (.get exit-future time-ms java.util.concurrent.TimeUnit/MILLISECONDS) (catch java.util.concurrent.TimeoutException e nil))] (cleanup false) timeout-symbol)))}))) (defn sh ([cmd-line] (sh cmd-line {:print-cmd-line? false :stdout-hdlr quiet-rf :stderr-hdlr quiet-rf})) ([^String cmd-line {:keys [timeout-ms] :or {timeout-ms Integer/MAX_VALUE} :as opts}] (let [rv ((:wait-or-kill (launch cmd-line opts)) timeout-ms ::timeout)] (if (identical? rv ::timeout) (throw (RuntimeException. "Process timed out")) rv)))) (defn ^:private map->cmd-line [args] (->> (flatten (seq args)) (map str))) (defn launch-jvm "Assumes a jvm process launched from a shell command. Will hang looking for first process descendant. If shell command then arguments other than :xmx :jvm-opts may need to have quoted strings if they are being passed the clojure process. Example: ```clojure ham-fisted.process> (launch-jvm \"clojure\" {:jvm-opts [\"-A:dev\" \"-X\" \"ham-fisted.protocol-perf/-main\"]}) launch-process: clojure -A:dev -X ham-fisted.protocol-perf/-main {:proc-hdl #object[java.lang.ProcessHandleImpl 0x4f4a8dc7 \"31194\"], :wait-or-kill #function[ham-fisted.process/launch/wait-or-kill--10016], :jvm-pid 31199, :jvm-proc #object[java.lang.ProcessHandleImpl 0x2e52e7c8 \"31199\"]} ... ham-fisted.process> (def result ((:wait-or-kill *1))) #'ham-fisted.process/result ```" [cmd-name {:keys [xmx jvm-opts] :as args}] (let [jvm-opts (if xmx (conj (or jvm-opts []) (str "-Xmx" xmx)) jvm-opts) cmd-line (clojure.string/join " " (concat [cmd-name] (map->cmd-line jvm-opts) (map->cmd-line (dissoc args :xmx :jvm-opts :stdout-hdlr :stderr-hdlr)))) {:keys [^java.lang.ProcessHandle proc-hdl] :as rv} (launch cmd-line args) desc (loop [desc (process-descendants proc-hdl)] (if (== 0 (count desc) ) (do (Thread/sleep 5) (if (.isAlive proc-hdl) (recur (process-descendants proc-hdl)) desc)) desc)) ^java.lang.ProcessHandle jvm-proc (first desc)] (if jvm-proc (assoc rv :jvm-pid (.pid jvm-proc) :jvm-proc jvm-proc) rv))) ================================================ FILE: src/ham_fisted/profile.clj ================================================ (ns ham-fisted.profile (:import [java.util Map] [java.util.concurrent ConcurrentHashMap])) (set! *warn-on-reflection* true) (def ^{:dynamic true :no-doc true :tag Map} time-map (ConcurrentHashMap.)) (defmacro time-ms "Time an operation returning the results. Puts time in double milliseconds into the time map." [kw & code] `(let [start# (System/nanoTime) rv# (do ~@code) end# (System/nanoTime)] (.put time-map ~kw (* (- end# start#) 1e-6)) rv#)) (defn current-times "Get the current time map" [] (into {} time-map)) (defn reset-times! "Clear times out of current time map" [] (.clear time-map)) (defmacro with-times "Returns {:result :times}. Run a block of code with time map bound to a new map." [& code] `(with-bindings {#'time-map (ConcurrentHashMap.)} (let [rv# (do ~@code) times# (current-times)] {:result rv# :times times#}))) ================================================ FILE: src/ham_fisted/protocols.clj ================================================ (ns ham-fisted.protocols (:require [ham-fisted.defprotocol :refer [defprotocol extend-protocol extend-type extend]:as hamf-defproto]) (:import [clojure.lang IFn IReduceInit IDeref] [java.util.function DoubleConsumer] [java.util Map] [ham_fisted Sum Sum$SimpleSum Reducible IFnDef$ODO ParallelOptions Reductions IMutList]) (:refer-clojure :exclude [reduce set? count defprotocol extend-protocol extend-type extend])) (defprotocol Counted (^long count [m])) (defprotocol ToIterable (convertible-to-iterable? [item]) (->iterable [item])) (defprotocol ToCollection (convertible-to-collection? [item]) (->collection [item])) (defprotocol Reduction "Faster check than satisfies? to see if something is reducible" (reducible? [coll])) (extend-protocol Reduction nil (reducible? [this] true) Object (reducible? [this] (or (instance? IReduceInit this) (instance? Iterable this) (instance? Map this) ;;This check is dog slow (clojure.core/satisfies? clojure.core.protocols/CollReduce this)))) (defprotocol ParallelReduction "Protocol to define a parallel reduction in a collection-specific pathway. Specializations are in impl as that is where the parallelization routines are found." (preduce [coll init-val-fn rfn merge-fn ^ParallelOptions options] "Container-specific parallelized reduction. Reductions must respect the pool passed in via the options.")) (defprotocol Finalize "Generic protocol for things that finalize results of reductions. Defaults to deref of instance of IDeref or identity." (finalize [this val])) (extend-protocol Finalize Object (finalize [this val] (if (instance? IDeref val) (.deref ^IDeref val) val)) ;;clojure rfn equivalence IFn (finalize [this val] (this val))) (defprotocol Reducer "Reducer is the basic reduction abstraction as a single object." (->init-val-fn [item] "Returns the initial values for a parallel reduction. This function takes no arguments and returns the initial accumulator.") (->rfn [item] "Returns the reduction function for a parallel reduction. This function takes two arguments, the accumulator and a value from the collection and returns a new or modified accumulator.")) (extend-protocol Reducer IFn (->init-val-fn [this] this) (->rfn [this] this)) (defprotocol ParallelReducer "Parallel reducers are simple a single object that you can pass into preduce as opposed to 3 separate functions." (->merge-fn [item] "Returns the merge function for a parallel reduction. This function takes two accumulators and returns a or modified accumulator.")) (extend-protocol ParallelReducer IFn (->merge-fn [this] this)) (extend-protocol ParallelReducer Object (->merge-fn [this] (fn [_l _r] (throw (RuntimeException. (str "Object does not implement merge: " (type this))))))) (def ^:no-doc double-consumer-accumulator (reify IFnDef$ODO (invokePrim [f acc v] (.accept ^DoubleConsumer acc v) acc))) (defn- reducible-merge [^Reducible lhs rhs] (.reduce lhs rhs)) (defprotocol PAdd "Define a function to mutably add items to a collection. This function must return the collection -- it must be useable in a reduce as the rf." (add-fn [l])) (defprotocol SetOps "Simple protocol for set operations to make them uniformly extensible to new objects." (set? [l]) (union [l r]) (difference [l r]) (intersection [l r]) (xor [l r]) (contains-fn [item] "Return an efficient function for deciding if this set contains a single item.") (^long cardinality [item] "Some sets don't work with clojure's count function.")) (defprotocol BulkSetOps (reduce-union [l data]) (reduce-intersection [l data])) (defprotocol BitSet "Protocol for efficiently dealing with bitsets" (bitset? [item]) (contains-range? [item sidx eidx]) (intersects-range? [item sidx eidx]) (min-set-value [item]) (max-set-value [item])) (defprotocol WrapArray (^IMutList wrap-array [ary]) (^IMutList wrap-array-growable [ary ptr])) (defprotocol SerializeObjBytes (serialize->bytes [o])) (defprotocol Datatype (datatype [o] "Returns the datatype [:int8, :int16, etc] -- if known -- else the type can be assumed to be an object type. The return value may not be a keyword but it must be comparable with identical?") (simplified-datatype [o] "Returns exactly :int64, :float64, or :object")) (hamf-defproto/extend nil Datatype {:datatype :object :simplified-datatype :object}) (hamf-defproto/extend Object Datatype {:datatype :object :simplified-datatype :object}) (defprotocol ContainedDatatype (contained-datatype [o] "Datatype of contained datatype - may be nil if not a container") (simplified-contained-datatype [o] "Exactly :int64 :float64 :object or nil")) (extend nil ContainedDatatype {:contained-datatype nil :simplified-contained-datatype nil}) (extend Object ContainedDatatype {:contained-datatype nil :simplified-contained-datatype nil}) (defprotocol ReturnedDatatype (returned-datatype [o]) (simplified-returned-datatype [o])) (extend nil ReturnedDatatype {:returned-datatype nil :simplified-returned-datatype nil}) (extend Object ReturnedDatatype {:returned-datatype nil :simplified-returned-datatype nil}) (defprotocol EstimateCount (^long estimate-count [m])) (defprotocol Split (split [m])) (defprotocol ManagedBlocker (^java.util.concurrent.ForkJoinPool$ManagedBlocker managed-blocker [m])) (defprotocol ToSpliterator (^java.util.Spliterator ->spliterator [m])) ================================================ FILE: src/ham_fisted/reduce.clj ================================================ (ns ham-fisted.reduce "Protocol-based parallel reduction architecture and helper functions." (:require [ham-fisted.protocols :as protocols] [ham-fisted.defprotocol :refer [extend extend-type extend-protocol]] [ham-fisted.lazy-noncaching :refer [map] :as lznc] [ham-fisted.function :refer [bi-function]]) (:import [ham_fisted ParallelOptions ParallelOptions$CatParallelism Reductions Transformables Reducible IFnDef$OOO IFnDef$OLOO IFnDef$ODO IFnDef$OLO IFnDef$DDD IFnDef$LLL Sum Sum$SimpleSum Reductions$IndexedAccum Reductions$IndexedLongAccum Reductions$IndexedDoubleAccum IFnDef$OLLO IFnDef$OLDO] [clojure.lang IFn$DO IFn$LO IFn$OLO IFn$DDD IFn$LLL] [java.util Map] [java.util.function DoubleConsumer LongConsumer Consumer] [java.util.concurrent Executor ForkJoinPool]) (:refer-clojure :exclude [map extend extend-type extend-protocol])) (set! *warn-on-reflection* true) (defn- unpack-reduced [item] (if (reduced? item) (deref item) item)) (defn options->parallel-options "Convert an options map to a parallel options object. Options: * `:pool` - supply the forkjoinpool to use. * `:max-batch-size` - Defaults to 64000, used for index-based parallel pathways to control the number size of each parallelized batch. * `:ordered?` - When true process inputs and provide results in order. * `:parallelism` - The amount of parallelism to expect. Defaults to the number of threads in the fork-join pool provided. * `:cat-parallelism` - Either `:seq-wise` or `:elem-wise` - when parallelizing over a concatenation of containers either parallelize each container meaning call preduce on each container using many threads per container or use one thread per container - `seq-wise`. Defaults to `seq-wise` as this doesn't require each container itself to support parallelization but relies on the sequence of containers to be long enough to saturate the processor. Can also be set at time of container construction - see [[lazy-noncaching/concat-opts]]. * `:put-timeout-ms` - The time to wait to put data into the queue. This is a safety mechanism so that if the processing system fails we don't just keep putting things into a queue. * `:unmerged-result?` - Use with care. For parallel reductions do not perform the merge step but instead return the sequence of partially reduced results. * `:n-lookahead` - How for to look ahead for pmap and upmap to add new jobs to the queue. Defaults to `(* 2 parallelism)." ^ParallelOptions [options] (cond (instance? ParallelOptions options) options (nil? options) (ParallelOptions.) :else (let [^Map options (or options {}) ^Executor pool (.getOrDefault options :pool (ForkJoinPool/commonPool))] (ParallelOptions. (.getOrDefault options :min-n 1000) (.getOrDefault options :max-batch-size 64000) (boolean (.getOrDefault options :ordered? true)) pool (.getOrDefault options :parallelism (if (instance? ForkJoinPool pool) (.getParallelism ^ForkJoinPool pool) 1)) (case (.getOrDefault options :cat-parallelism :seq-wise) :seq-wise ParallelOptions$CatParallelism/SEQWISE :elem-wise ParallelOptions$CatParallelism/ELEMWISE) (.getOrDefault options :put-timeout-ms 5000) (.getOrDefault options :unmerged-result? false) (.getOrDefault options :n-lookahead -1))))) (defn preduce "Parallelized reduction. Currently coll must either be random access or a lznc map/filter chain based on one or more random access entities, hashmaps and sets from this library or any java.util set, hashmap or concurrent versions of these. If input cannot be parallelized this lowers to a normal serial reduction. For potentially small-n invocations providing the parallel options explicitly will improve performance surprisingly - converting the options map to the parallel options object takes a bit of time. * `init-val-fn` - Potentially called in reduction threads to produce each initial value. * `rfn` - normal clojure reduction function. Typehinting the second argument to double or long will sometimes produce a faster reduction. * `merge-fn` - Merge two reduction results into one. Options: * `:pool` - The fork-join pool to use. Defaults to common pool which assumes reduction is cpu-bound. * `:parallelism` - What parallelism to use - defaults to pool's `getParallelism` method. * `:max-batch-size` - Rough maximum batch size for indexed or grouped reductions. This can both even out batch times and ensure you don't get into safepoint trouble with jdk-8. * `:min-n` - minimum number of elements before initiating a parallelized reduction - Defaults to 1000 but you should customize this particular to your specific reduction. * `:ordered?` - True if results should be in order. Unordered results sometimes are slightly faster but again you should test for your specific situation.. * `:cat-parallelism` - Either `:seq-wise` or `:elem-wise`, defaults to `:seq-wise`. Test for your specific situation, this really is data-dependent. This contols how a concat primitive parallelizes the reduction across its contains. Elemwise means each container's reduction is individually parallelized while seqwise indicates to do a pmap style initial reduction across containers then merge the results. * `:put-timeout-ms` - Number of milliseconds to wait for queue space before throwing an exception in unordered reductions. Defaults to 50000. * `:unmerged-result?` - Defaults to false. When true, the sequence of results be returned directly without any merge steps in a lazy-noncaching container. Beware the noncaching aspect -- repeatedly evaluating this result may kick off the parallelized reduction multiple times. To ensure caching if unsure call `seq` on the result." ([init-val-fn rfn merge-fn coll] (preduce init-val-fn rfn merge-fn nil coll)) ([init-val-fn rfn merge-fn options coll] (unpack-reduced (Reductions/parallelReduction init-val-fn rfn merge-fn (lznc/->reducible coll) (options->parallel-options options))))) (defn preduce-reducer "Given an instance of [[ham-fisted.protocols/ParallelReducer]], perform a parallel reduction. In the case where the result is requested unmerged then finalize will be called on each result in a lazy noncaching way. In this case you can use a non-parallelized reducer and simply get a sequence of results as opposed to one. * reducer - instance of ParallelReducer * options - Same options as preduce. * coll - something potentially with a parallelizable reduction. See options for [[ham-fisted.reduce/preduce]]. Additional Options: * `:skip-finalize?` - when true, the reducer's finalize method is not called on the result." ([reducer options coll] (let [retval (preduce (protocols/->init-val-fn reducer) (protocols/->rfn reducer) (protocols/->merge-fn reducer) options coll)] (if (get options :skip-finalize?) retval (if (get options :unmerged-result?) (lznc/map #(protocols/finalize reducer %) retval) (protocols/finalize reducer retval))))) ([reducer coll] (preduce-reducer reducer nil coll))) (defmacro double-accumulator "Type-hinted double reduction accumulator. consumer: ```clojure ham-fisted.api> (reduce (double-accumulator acc v (+ (double acc) v)) 0.0 (range 1000)) # ham-fisted.api> @*1 499500.0 ```" [accvar varvar & code] `(reify IFnDef$ODO (invokePrim [this# ~accvar ~varvar] ~@code))) (defmacro long-accumulator "Type-hinted double reduction accumulator. consumer: ```clojure ham-fisted.api> (reduce (double-accumulator acc v (+ (double acc) v)) 0.0 (range 1000)) # ham-fisted.api> @*1 499500.0 ```" [accvar varvar & code] `(reify IFnDef$OLO (invokePrim [this# ~accvar ~varvar] ~@code))) (defn immut-map-kv ([keyfn valfn data] (-> (reduce (fn [^Map m v] (.put m (keyfn v) (valfn v)) m) (ham_fisted.UnsharedHashMap. nil) data) (persistent!))) ([ks vs] (let [rv (ham_fisted.UnsharedHashMap. nil) ki (.iterator (Transformables/toIterable ks)) vi (.iterator (Transformables/toIterable vs))] (while (and (.hasNext ki) (.hasNext vi)) (.put rv (.next ki) (.next vi))) (persistent! rv)))) (defn compose-reducers "Given a map or sequence of reducers return a new reducer that produces a map or vector of results. If data is a sequence then context is guaranteed to be an object array. Options: * `:rfn-datatype` - One of nil, :int64, or :float64. This indicates that the rfn's should all be uniform as accepting longs, doubles, or generically objects. Defaults to nil." ([reducers] (compose-reducers nil reducers)) ([options reducers] (if (instance? Map reducers) (let [reducer (compose-reducers (vals reducers))] (reify protocols/Reducer (->init-val-fn [_] (protocols/->init-val-fn reducer)) (->rfn [_] (protocols/->rfn reducer)) protocols/Finalize (finalize [_ v] (immut-map-kv (keys reducers) (protocols/finalize reducer v))) protocols/ParallelReducer (->merge-fn [_] (protocols/->merge-fn reducer)))) (let [init-fns (.toArray ^java.util.Collection (lznc/map protocols/->init-val-fn reducers)) rfn-dt (get options :rfn-datatype) ^objects rfns (object-array (case rfn-dt :int64 (->> (map protocols/->rfn reducers) (map #(Transformables/toLongReductionFn %))) :float64 (->> (map protocols/->rfn reducers) (map #(Transformables/toDoubleReductionFn %))) ;;else branch (map protocols/->rfn reducers))) ^objects mergefns (object-array (map protocols/->merge-fn reducers)) n-vals (count rfns) n-init (count init-fns)] (reify protocols/Reducer (->init-val-fn [_] (fn compose-init [] (let [rv (ham_fisted.ArrayLists/objectArray n-init)] (dotimes [idx n-init] (aset rv idx ((aget init-fns idx)))) rv))) (->rfn [_] (case rfn-dt :int64 (Reductions/longCompose n-vals rfns) :float64 (Reductions/doubleCompose n-vals rfns) (Reductions/objCompose n-vals rfns))) protocols/Finalize (finalize [_ v] (mapv #(protocols/finalize %1 %2) reducers v)) protocols/ParallelReducer (->merge-fn [_] (Reductions/mergeCompose n-vals, mergefns))))))) (defn preduce-reducers "Given a map or sequence of [[ham-fisted.protocols/ParallelReducer]], produce a map or sequence of reduced values. Reduces over input coll once in parallel if coll is large enough. See options for [[ham-fisted.reduce/preduce]]. ```clojure ham-fisted.api> (preduce-reducers {:sum (Sum.) :mult *} (range 20)) {:mult 0, :sum #} ```" ([reducers options coll] (preduce-reducer (compose-reducers reducers) options coll)) ([reducers coll] (preduce-reducers reducers nil coll))) (defn reducer-xform->reducer "Given a reducer and a transducer xform produce a new reducer which will apply the transducer pipeline before is reduction function. ```clojure ham-fisted.api> (reduce-reducer (reducer-xform->reducer (Sum.) (clojure.core/filter even?)) (range 1000)) # ``` !! - If you use a stateful transducer here then you must *not* use the reducer in a parallelized reduction." [reducer xform] (let [rfn (protocols/->rfn reducer) init-val-fn (protocols/->init-val-fn reducer) xfn (xform (fn ([] (init-val-fn)) ([v] (protocols/finalize reducer v)) ([acc v] (rfn acc v))))] (reify protocols/Reducer (->init-val-fn [this] init-val-fn) (->rfn [this] xfn) protocols/Finalize (finalize [this v] (xfn v)) protocols/ParallelReducer (->merge-fn [this] (protocols/->merge-fn reducer))))) (defn reducer->rf "Given a reducer, return a transduce-compatible rf - ```clojure ham-fisted.api> (transduce (clojure.core/map #(+ % 2)) (reducer->rf (Sum.)) (range 200)) {:sum 20300.0, :n-elems 200} ```" [reducer] (let [rfn (protocols/->rfn reducer) init-val-fn (protocols/->init-val-fn reducer)] (fn ([] (init-val-fn)) ([acc v] (rfn acc v)) ([v] (protocols/finalize reducer v))))) (defn reducer->completef "Return fold-compatible pair of [reducef, completef] given a parallel reducer. Note that folded reducers are not finalized as of this time: ```clojure ham-fisted.api> (def data (vec (range 200000))) #'ham-fisted.api/data ham-fisted.api> (r/fold (reducer->completef (Sum.)) (reducer->rfn (Sum.)) data) # ```" [reducer] (let [rfn (protocols/->rfn reducer) init-val-fn (protocols/->init-val-fn reducer) merge-fn (protocols/->merge-fn reducer)] (fn ([] (init-val-fn)) ([l r] (merge-fn l r)) ([v] (protocols/finalize reducer v))))) (defn reducer-with-finalize [reducer fin-fn] (reify protocols/Reducer (->init-val-fn [r] (protocols/->init-val-fn reducer)) (->rfn [r] (protocols/->rfn reducer)) protocols/Finalize (finalize [r v] (fin-fn v)) protocols/ParallelReducer (->merge-fn [r] (protocols/->merge-fn reducer)))) (defn reduce-reducer "Serially reduce a reducer. ```clojure ham-fisted.api> (reduce-reducer (Sum.) (range 1000)) # ```" [reducer coll] (let [rfn (protocols/->rfn reducer) init-val-fn (protocols/->init-val-fn reducer)] (->> (reduce rfn (init-val-fn) coll) (protocols/finalize reducer)))) (defn reduce-reducers "Serially reduce a map or sequence of reducers into a map or sequence of results. ```clojure ham-fisted.api> (reduce-reducers {:a (Sum.) :b *} (range 1 21)) {:b 2432902008176640000, :a #} ```" [reducers coll] (reduce-reducer (compose-reducers reducers) coll)) (defn ^:no-doc reduce-reducibles [reducibles] (let [^Reducible r (first reducibles)] (when-not (instance? Reducible r) (throw (Exception. (str "Sequence does not contain reducibles: " (type (first r)))))) (.reduce r (rest reducibles)))) (def double-consumer-accumulator "Converts from a double consumer to a double reduction accumulator that returns the consumer: ```clojure ham-fisted.api> (reduce double-consumer-accumulator (Sum$SimpleSum.) (range 1000)) # ham-fisted.api> @*1 499500.0 ```" ham_fisted.ConsumerAccumulators$DoubleConsumerAccumulator/INST) (def long-consumer-accumulator "Converts from a long consumer to a long reduction accumulator that returns the consumer: ```clojure ham-fisted.api> (reduce double-consumer-accumulator (Sum$SimpleSum.) (range 1000)) # ham-fisted.api> @*1 499500.0 ```" ham_fisted.ConsumerAccumulators$LongConsumerAccumulator/INST) (def consumer-accumulator "Generic reduction function using a consumer" ham_fisted.ConsumerAccumulators$ConsumerAccumulator/INST) (def ^{:doc "Parallel reduction merge function that expects both sides to be an instances of Reducible"} reducible-merge (bi-function lhs rhs (.reduce ^Reducible lhs rhs))) (defmacro indexed-accum "Create an indexed accumulator that recieves and additional long index during a reduction: ```clojure ham-fisted.api> (reduce (indexed-accum acc idx v (conj acc [idx v])) [] (range 5)) [[0 0] [1 1] [2 2] [3 3] [4 4]] ```" [accvar idxvar varvar & code] `(Reductions$IndexedAccum. (reify IFnDef$OLOO (invokePrim [this# ~accvar ~idxvar ~varvar] ~@code)))) (defmacro indexed-double-accum "Create an indexed double accumulator that recieves and additional long index during a reduction: ```clojure ham-fisted.api> (reduce (indexed-double-accum acc idx v (conj acc [idx v])) [] (range 5)) [[0 0.0] [1 1.0] [2 2.0] [3 3.0] [4 4.0]] ```" [accvar idxvar varvar & code] `(Reductions$IndexedDoubleAccum. (reify IFnDef$OLDO (invokePrim [this# ~accvar ~idxvar ~varvar] ~@code)))) (defmacro indexed-long-accum "Create an indexed long accumulator that recieves and additional long index during a reduction: ```clojure ham-fisted.api> (reduce (indexed-long-accum acc idx v (conj acc [idx v])) [] (range 5)) [[0 0] [1 1] [2 2] [3 3] [4 4]] ```" [accvar idxvar varvar & code] `(Reductions$IndexedLongAccum. (reify IFnDef$OLLO (invokePrim [this# ~accvar ~idxvar ~varvar] ~@code)))) (defn ->consumer "Return an instance of a consumer, double consumer, or long consumer." [cfn] (cond (or (instance? Consumer cfn) (instance? DoubleConsumer cfn) (instance? LongConsumer cfn)) cfn (instance? IFn$DO cfn) (reify DoubleConsumer (accept [this v] (.invokePrim ^IFn$DO cfn v))) (instance? IFn$LO cfn) (reify LongConsumer (accept [this v] (.invokePrim ^IFn$LO cfn v))) :else (reify Consumer (accept [this v] (cfn v))))) (defn consume! "Consumer a collection. This is simply a reduction where the return value is ignored. Returns the consumer." [consumer coll] (let [c (->consumer consumer)] (cond (instance? DoubleConsumer c) (reduce double-consumer-accumulator c coll) (instance? LongConsumer c) (reduce long-consumer-accumulator c coll) :else (reduce consumer-accumulator c coll)) consumer)) (defn double-consumer-preducer "Return a preducer for a double consumer. Consumer must implement java.util.function.DoubleConsumer, ham_fisted.Reducible and clojure.lang.IDeref. ```clojure user> (require '[ham-fisted.api :as hamf]) nil user> (import '[java.util.function DoubleConsumer]) java.util.function.DoubleConsumer user> (import [ham_fisted Reducible]) ham_fisted.Reducible user> (import '[clojure.lang IDeref]) clojure.lang.IDeref user> (deftype MeanR [^{:unsynchronized-mutable true :tag 'double} sum ^{:unsynchronized-mutable true :tag 'long} n-elems] DoubleConsumer (accept [this v] (set! sum (+ sum v)) (set! n-elems (unchecked-inc n-elems))) Reducible (reduce [this o] (set! sum (+ sum (.-sum ^MeanR o))) (set! n-elems (+ n-elems (.-n-elems ^MeanR o))) this) IDeref (deref [this] (/ sum n-elems))) user.MeanR user> (hamf/declare-double-consumer-preducer! MeanR (MeanR. 0 0)) nil user> (hamf/preduce-reducer (double-consumer-preducer #(MeanR. 0 0)) (hamf/range 200000)) 99999.5 ```" [constructor] (reify protocols/Reducer (->init-val-fn [r] constructor) (->rfn [r] double-consumer-accumulator) protocols/Finalize (finalize [r v] @v) protocols/ParallelReducer (->merge-fn [r] reducible-merge))) (deftype DDDReducer [^{:unsynchronized-mutable true :tag double} acc ^{:tag IFn$DDD} rfn merge-fn ^{:unsynchronized-mutable true :tag clojure.lang.Box} merged] DoubleConsumer (accept [this v] (set! acc (.invokePrim rfn acc v))) Reducible (reduce [this other] (if merged (set! (.-val merged) (merge-fn (.-val merged) @other)) (set! merged (clojure.lang.Box. (merge-fn acc @other)))) this) clojure.lang.IDeref (deref [this] (if merged (.-val merged) acc))) (deftype LLLReducer [^{:unsynchronized-mutable true :tag long} acc ^{:tag IFn$LLL} rfn merge-fn ^{:unsynchronized-mutable true :tag clojure.lang.Box} merged] LongConsumer (accept [this v] (set! acc (.invokePrim rfn acc v))) Reducible (reduce [this other] (if merged (set! (.-val merged) (merge-fn (.-val merged) @other)) (set! merged (clojure.lang.Box. (merge-fn acc @other)))) this) clojure.lang.IDeref (deref [this] (if merged (.-val merged) acc))) (defn parallel-reducer "Implement a parallel reducer by explicitly passing in the various required functions. * 'init-fn' - Takes no argumenst and returns a new accumulation target. * 'rfn' - clojure rf function - takes two arguments, the accumulation target and a new value and produces a new accumulation target. * 'merge-fn' - Given two accumulation targets returns a new combined accumulation target. * 'fin-fn' - optional - Given an accumulation target returns the desired final type. ```clojure user> (hamf-rf/preduce-reducer (hamf-rf/parallel-reducer hamf/mut-set #(do (.add ^java.util.Set %1 %2) %1) hamf/union hamf/sort) (lznc/map (fn ^long [^long v] (rem v 13)) (hamf/range 1000000))) [0 1 2 3 4 5 6 7 8 9 10 11 12] ``` " ([init-fn rfn merge-fn fin-fn] (cond (instance? IFn$DDD rfn) (reify protocols/Reducer (->init-val-fn [this] #(DDDReducer. (double (init-fn)) rfn merge-fn nil)) (->rfn [r] double-consumer-accumulator) protocols/ParallelReducer (->merge-fn [r] reducible-merge) protocols/Finalize (finalize [r v] (fin-fn @v))) (instance? IFn$LLL rfn) (reify protocols/Reducer (->init-val-fn [this] #(LLLReducer. (long (init-fn)) rfn merge-fn nil)) (->rfn [r] long-consumer-accumulator) protocols/ParallelReducer (->merge-fn [r] reducible-merge) protocols/Finalize (finalize [r v] (fin-fn @v))) :else (reify protocols/Reducer (->init-val-fn [this] init-fn) (->rfn [r] rfn) protocols/ParallelReducer (->merge-fn [r] merge-fn) protocols/Finalize (finalize [r v] (fin-fn v))))) ([init-fn rfn merge-fn] (parallel-reducer init-fn rfn merge-fn identity))) (defn consumer-preducer "Bind a consumer as a parallel reducer. Consumer must implement java.util.function.Consumer, ham_fisted.Reducible and clojure.lang.IDeref. Returns instance of type bound. See documentation for [[declare-double-consumer-preducer!]]. ```" [constructor] (reify protocols/Reducer (->init-val-fn [r] constructor) (->rfn [r] consumer-accumulator) protocols/Finalize (finalize [r v] @v) protocols/ParallelReducer (->merge-fn [r] reducible-merge))) (defn bind-double-consumer-reducer! "Bind a classtype as a double consumer parallel reducer - the consumer must implement DoubleConsumer, ham_fisted.Reducible, and IDeref." ([cls-type ctor] (extend cls-type protocols/Reducer {:->init-val-fn (fn [r] ctor) :->rfn (fn [r] double-consumer-accumulator)} protocols/ParallelReducer {:->merge-fn (fn [r] reducible-merge)})) ([ctor] (bind-double-consumer-reducer! (type (ctor)) ctor))) (bind-double-consumer-reducer! #(Sum.)) (bind-double-consumer-reducer! #(Sum$SimpleSum.)) (defn double-consumer-reducer "Make a parallel double consumer reducer given a function that takes no arguments and is guaranteed to produce a double consumer which also implements Reducible and IDeref" [ctor] (reify protocols/Reducer (->init-val-fn [this] ctor) (->rfn [this] double-consumer-accumulator) protocols/Finalize (finalize [this v] (deref v)) protocols/ParallelReducer (->merge-fn [this] reducible-merge))) (defn long-consumer-reducer "Make a parallel double consumer reducer given a function that takes no arguments and is guaranteed to produce a double consumer which also implements Reducible and IDeref" [ctor] (reify protocols/Reducer (->init-val-fn [this] ctor) (->rfn [this] long-consumer-accumulator) protocols/Finalize (finalize [this v] (deref v)) protocols/ParallelReducer (->merge-fn [this] reducible-merge))) (defn consumer-reducer "Make a parallel double consumer reducer given a function that takes no arguments and is guaranteed to produce a double consumer which also implements Reducible and IDeref" [ctor] (reify protocols/Reducer (->init-val-fn [this] ctor) (->rfn [this] consumer-accumulator) protocols/Finalize (finalize [this v] (deref v)) protocols/ParallelReducer (->merge-fn [this] reducible-merge))) ================================================ FILE: src/ham_fisted/set.clj ================================================ (ns ham-fisted.set (:require [ham-fisted.protocols :as hamf-proto] [ham-fisted.api :as api] [ham-fisted.function :as hamf-fn] [ham-fisted.reduce :as hamf-rf] [ham-fisted.impl] [ham-fisted.language :as hamf-language] [ham-fisted.defprotocol :refer [extend extend-type extend-protocol]]) (:import [ham_fisted UnsharedHashSet PersistentHashSet Ranges$LongRange IMutList] [java.util BitSet Set Map Collection] [java.util.concurrent ConcurrentHashMap] [clojure.lang APersistentSet]) (:refer-clojure :exclude [set set? extend extend-type extend-protocol])) (set! *warn-on-reflection* true) (declare unique bitset) (defn- set-add-all [^Set s data] (reduce (fn [^Set acc v] (.add acc v) acc) s data)) (defn mut-set "Return a mutable set." (^Set [] (UnsharedHashSet. nil)) (^Set [data] (doto (UnsharedHashSet. nil) (.addAll data)))) (defn set "Return an immutable set" (^Set [] PersistentHashSet/EMPTY) (^Set [data] (if (instance? clojure.lang.IPersistentSet data) data (-> (mut-set data) (persistent!))))) (defn set? [data] (when data (hamf-proto/set? data))) (defn java-hashset "Return a java hashset" (^Set [] (java.util.HashSet.)) (^Set [data] (if (instance? java.util.HashSet data) data (set-add-all (java.util.HashSet.) data)))) (defn java-concurrent-hashset "Create a concurrent hashset." (^Set [] (ConcurrentHashMap/newKeySet)) (^Set [data] (if (instance? Set data) data (unique {:set-constructor (hamf-language/constantly (java-concurrent-hashset))} data)))) (def ^{:private true :tag 'long} unsigned-int-max (Integer/toUnsignedLong (int -1))) (defn- unsigned-int->host ^long [^long v] (when (or (< v 0) (> v unsigned-int-max)) (throw (RuntimeException. (str "Value out of range for unsigned integer: " v)))) (unchecked-int v)) (defn- reduce->range [data] (unique {:set-constructor bitset} data)) (defn bitset "Create a java.util.Bitset. The two argument version assumes you are passing in the start, end of a monotonically incrementing range." (^BitSet [] (BitSet.)) (^BitSet [data] (cond (instance? BitSet data) data (instance? Ranges$LongRange data) (let [^Ranges$LongRange data data rstart (.-start data) rend (.-end data) step (.-step data)] (if (== 1 step) (doto (BitSet.) (.set (unchecked-int rstart) (unchecked-int rend))) (reduce->range data))) :else (reduce->range data))) (^BitSet [^long start ^long end] (doto (BitSet.) (.set (unchecked-int start) (unchecked-int end))))) (extend-protocol hamf-proto/SetOps Object (set? [l] (instance? Set l)) (union [l r] (api/union l r)) (difference [l r] (api/difference l r)) (intersection [l r] (api/intersection l r)) (xor [l r] (api/difference (api/union l r) (api/intersection l r))) (contains-fn [l] #(.contains ^Set l %)) (cardinality [l] (.size ^Set l)) BitSet (set? [l] true) (union [l r] (let [^BitSet l (.clone l)] (.or ^BitSet (.clone l) (bitset r)) l)) (difference [l r] (let [^BitSet l (.clone l)] (.andNot l (bitset r)) l)) (intersection [l r] (let [^BitSet l (.clone l)] (.and l (bitset r)) l)) (xor [l r] (let [^BitSet l (.clone l)] (.xor l (bitset r)) l)) (contains-fn [l] (hamf-fn/long-predicate v (.get l (unchecked-int v)))) (cardinality [l] (.cardinality l))) (extend-protocol hamf-proto/BulkSetOps Object (reduce-union [l data] (reduce hamf-proto/union l data)) (reduce-intersection [l data] (reduce hamf-proto/intersection l data))) (extend-protocol hamf-proto/PAdd BitSet (add-fn [c] (hamf-rf/long-accumulator b v (.set ^BitSet b (unchecked-int v)) b))) (extend-protocol hamf-proto/BitSet Object (bitset? [item] false) BitSet (bitset? [item] true) (contains-range? [item sidx eidx] (reduce (hamf-rf/long-accumulator acc v (if (and (>= v 0) (.get item (unchecked-int v))) true (reduced false))) true (api/range sidx eidx))) (intersects-range? [item ^long sidx ^long eidx] (reduce (hamf-rf/long-accumulator acc v (if (and (>= v 0) (.get item (unchecked-int v))) (reduced true) false)) false (api/range sidx eidx))) (min-set-value [item] (.nextSetBit item 0)) (max-set-value [item] (.previousSetBit item Integer/MAX_VALUE))) (defn union "set union" [l r] (if l (hamf-proto/union l r) r)) (defn reduce-union "Reduce a number of objects into one object via union" ([] (api/immut-set)) ([l] l) ([l & data] (hamf-proto/reduce-union l data))) (defn intersection "set intersection" ([] (api/immut-set)) ([l] l) ([l r] (if l (hamf-proto/intersection l r) r))) (defn reduce-intersection ([] (api/immut-set)) ([l] l) ([l & data] (hamf-proto/reduce-intersection l data))) (defn difference "set difference" [l r] (when l (hamf-proto/difference l r))) (defn xor "set xor - difference of intersection from union" [l r] (if l (hamf-proto/xor l r) r)) (defn map-invert "invert a map such that the keys are the vals and the vals are the keys" [m] (when m (-> (reduce (fn [^Map acc e] (.put acc (val e) (key e)) acc) (api/mut-map) m) (persistent!)))) (defn contains-fn "Return an IFn that returns efficiently returns true if the set contains a given element." [s] (hamf-proto/contains-fn s)) (defn cardinality "Return the cardinality (size) of a given set." ^long [s] (hamf-proto/cardinality s)) (defn bitset? "Return true if this is a bitset" [s] (hamf-proto/bitset? s)) (defn contains-range? "bitset-specific query that returns true if the set contains all the integers from sidx to eidx non-inclusive." [s ^long sidx ^long eidx] (hamf-proto/contains-range? s sidx eidx)) (defn intersects-range? "bitset-specific query that returns true if the set contains any the integers from sidx to eidx non-inclusive." [s ^long sidx ^long eidx] (hamf-proto/intersects-range? s sidx eidx)) (defn min-set-value "Given a bitset, return the minimum set value. Errors if the bitset is empty." [s] (hamf-proto/min-set-value s)) (defn max-set-value "Given a bitset, return the maximum set value. Errors if the bitset is empty." [s] (hamf-proto/max-set-value s)) (defn ->integer-random-access "Given a set (or bitset), return a efficient, sorted random access structure. This assumes the set contains integers." [s] (let [ne (cardinality s) rv (if (== 0 ne) (api/range 0) (if (bitset? s) (let [mins (long (min-set-value s)) maxs (long (max-set-value s))] (if (== ne (- (unchecked-inc maxs) mins)) (api/range mins (unchecked-inc maxs)) (if (< maxs Integer/MAX_VALUE) (api/ivec s) (api/lvec s)))) (let [^IMutList l (api/lvec s)] ;;Sorting means downstream access is in memory order. (.sort l nil) l)))] (if (== 0 ne) rv ;;Keeping track of the min, max values allows downstream functions to choose ;;perhaps different pathways. (vary-meta rv assoc :min (rv 0) :max (rv -1))))) (defn unique-reducer "Create a parallel reducer that creates a set. Options: * `:set-constructor` construct something that implements [[ham-fisted.protocols/add-fn]] and [[ham-fisted.protocols/union]] * `:add-fn` Pass in user-defined add function as opposed to using protocol lookup to find it." [options] (let [ctor (get options :set-constructor api/mut-set) add-fn (or (get options :add-fn) (hamf-proto/add-fn (ctor)))] (reify hamf-proto/Reducer (->init-val-fn [r] ctor) (->rfn [r] add-fn) hamf-proto/Finalize (finalize [r v] (api/persistent! v)) hamf-proto/ParallelReducer (->merge-fn [r] union)))) (defn unique "Create a set of unique items. Parallelized and non-lazy. See options for [[unique-reducer]] and [[ham-fisted.api/preduce-reducer]]." ([options data] (hamf-rf/preduce-reducer (unique-reducer options) (merge {:min-n 1000} options) data)) ([data] (unique nil data))) ================================================ FILE: src/ham_fisted/spliterator.clj ================================================ (ns ham-fisted.spliterator "Support for spliterator reduction and parallel reduction." (:require [ham-fisted.protocols :as proto] [ham-fisted.fjp :as fjp] [ham-fisted.defprotocol :refer [extend-type extend]] [ham-fisted.language :as hamf-language]) (:import [java.util Spliterator Spliterator$OfDouble Spliterator$OfLong List RandomAccess ArrayList] [java.util.function Consumer] [java.util.concurrent ForkJoinPool] [clojure.lang IFn$DDD IFn$LLL IFn$OLO IFn$ODO] [ham_fisted Consumers$IDerefLongConsumer Consumers$IDerefDoubleConsumer Consumers$IDerefConsumer ParallelOptions Casts ArrayLists]) (:refer-clojure :exclude [extend-type extend])) (set! *warn-on-reflection* true) (set! *unchecked-math* :warn-on-boxed) (extend-type nil proto/EstimateCount (estimate-count [m] 0) proto/Split (split [m] nil)) (extend-type Object proto/EstimateCount (estimate-count [m] (proto/count m)) proto/Split (split [m] nil)) (run! (fn [[dt-name ary-cls]] (extend ary-cls proto/ToSpliterator {:->spliterator (case dt-name :boolean (fn [a] (.spliterator (ArrayLists/toList ^booleans a))) :byte (fn [a] (.spliterator (ArrayLists/toList ^bytes a))) :short (fn [a] (.spliterator (ArrayLists/toList ^shorts a))) :char (fn [a] (.spliterator (ArrayLists/toList ^chars a))) :int (fn [a] (.spliterator (ArrayLists/toList ^ints a))) :long (fn [a] (.spliterator (ArrayLists/toList ^longs a))) :float (fn [a] (.spliterator (ArrayLists/toList ^floats a))) :double (fn [a] (.spliterator (ArrayLists/toList ^doubles a))) (fn [a] (.spliterator (ArrayLists/toList ^objects a))))})) hamf-language/array-classes) (deftype ^:private DerefLongConsumer [^{:unsynchronized-mutable true :tag long} v ^IFn$LLL rfn] Consumers$IDerefLongConsumer (acceptLong [this d] (set! v (.invokePrim rfn v d))) (deref [this] v)) (defmacro deref-long-consumer "Create a LongConsumer with support for IDeref" [varname accept-code deref-code] `(reify Consumers$IDerefLongConsumer (acceptLong [_ ~varname] (let [l# ~accept-code])) (deref [_] ~deref-code))) (deftype DerefDoubleConsumer [^{:unsynchronized-mutable true :tag double} v ^IFn$DDD rfn] Consumers$IDerefDoubleConsumer (acceptDouble [this d] (set! v (.invokePrim rfn v d))) (deref [this] v)) (defmacro deref-double-consumer "Create a DoubleConsumer with support for IDeref" [varname accept-code deref-code] `(reify Consumers$IDerefDoubleConsumer (acceptDouble [_ ~varname] (let [l# ~accept-code])) (deref [_] ~deref-code))) (defmacro deref-consumer "Create a Consumer with support for IDeref" [varname accept-code deref-code] `(reify Consumers$IDerefConsumer (accept [_ ~varname] ~accept-code) (deref [_] ~deref-code))) (defn ->spliterator ^Spliterator [ii] (if (instance? Spliterator ii) ii (proto/->spliterator ii))) (defn split-reduce "Reduce over a spliterator. Special support exists for IFn$LLL and IFn$DDD" ([rfn split] (let [acc-ary (object-array 1) cc (deref-consumer ll (aset acc-ary 0 ll) (aget acc-ary 0)) split (->spliterator split)] (if (.tryAdvance split cc) (split-reduce rfn (aget acc-ary 0) split) (rfn)))) ([rfn acc split] (cond (instance? IFn$LLL rfn) (let [cc (DerefLongConsumer. (Casts/longCast acc) rfn)] (.forEachRemaining (->spliterator split) cc) @cc) (instance? IFn$OLO rfn) (let [dd (hamf-language/obj-ary acc) cc (deref-long-consumer ll (aset dd 0 (.invokePrim ^IFn$OLO rfn (aget dd 0) ll)) (aget dd 0))] (.forEachRemaining (->spliterator split) cc) @cc) (instance? IFn$DDD rfn) (let [cc (DerefDoubleConsumer. (Casts/doubleCast acc) rfn)] (.forEachRemaining (->spliterator split) cc) @cc) (instance? IFn$ODO rfn) (let [dd (hamf-language/obj-ary acc) cc (deref-double-consumer ll (aset dd 0 (.invokePrim ^IFn$ODO rfn (aget dd 0) ll)) (aget dd 0))] (.forEachRemaining (->spliterator split) cc) @cc) :else (let [dd (hamf-language/obj-ary acc) cc (deref-consumer ll (aset dd 0 (rfn (aget dd 0) ll)) (aget dd 0)) split (->spliterator split)] (loop [] (let [c? (.tryAdvance split cc)] (if c? (let [vv @cc] (if (reduced? vv) @vv (recur))) (unreduced @cc)))))))) (clojure.core/extend Spliterator clojure.core.protocols/CollReduce {:coll-reduce (fn reduce-spliterator ([coll rfn acc] (split-reduce rfn acc coll)) ([coll rfn] (split-reduce rfn coll)))}) (defn- println-ret [v] (println v) v) (extend-type Spliterator proto/ToSpliterator (->spliterator [s] s) proto/EstimateCount (estimate-count [s] (.estimateSize s)) proto/Split (split [s] (when-let [ss (.trySplit s)] [s ss]))) (extend-type java.util.RandomAccess proto/Split (split [s] (let [^List s s ne (.size s)] (when (> ne 2) (let [n (quot ne 2)] [(.subList s 0 n) (.subList s n ne)]))))) (defn split-parallel-reduce "Perform a parallel reduction of a spliterator using the provided ExecutorService" [executor-service split ideal-split init-fn rfn merge-fn] (let [split (->spliterator split) n-elems (.estimateSize split) pool (or executor-service (ForkJoinPool/commonPool))] (if (or (<= n-elems (long ideal-split)) (= n-elems Long/MAX_VALUE)) (split-reduce rfn (init-fn) split) (if-let [rhs (.trySplit split)] (let [lt (fjp/safe-fork-task pool (split-parallel-reduce pool split ideal-split init-fn rfn merge-fn)) rt (fjp/safe-fork-task pool (split-parallel-reduce pool rhs ideal-split init-fn rfn merge-fn))] (merge-fn (fjp/managed-block-unwrap lt) (fjp/managed-block-unwrap rt))) (split-reduce rfn (init-fn) split))))) (extend-type Spliterator proto/ParallelReduction (preduce [coll init-fn rfn merge-fn options] (split-parallel-reduce (.-pool options) coll (.-minN options) init-fn rfn merge-fn))) (extend-type java.util.Collection proto/ToSpliterator (->spliterator [c] (.spliterator c))) (defn sum-fast "spliterator based serial summation" ^double [vv] (let [cc (ham_fisted.Sum$SimpleSum.)] (.forEachRemaining (->spliterator vv) cc) @cc)) (defn- dp ^double [^double a ^double b] (+ a b)) (defn psum "spliterator based parallel summation - does not account for Double/NaN" ^double [vv] (split-parallel-reduce nil (->spliterator vv) 10000 (constantly 0) dp dp)) (defn elements "Return all the elements referenced by this spliterator as a persistent list" [data] (let [split (->spliterator data) rv (ham_fisted.ArrayLists$ObjectArrayList.) cc (reify Consumer (accept [this v] (.add rv v)))] (.forEachRemaining split cc) (persistent! rv))) (defn split-to-max-size [split ^long max-size op] (let [split (->spliterator split) n-elems (.estimateSize ^Spliterator split)] (if (or (<= n-elems max-size) (== n-elems Long/MAX_VALUE)) [(op split)] (let [rv (doto (ArrayList. ) (.add split))] (loop [idx 0] (if (== idx (.size rv)) rv (let [ll (.get rv idx)] (if (<= (.estimateSize ^Spliterator ll) max-size) (do (.set rv idx (op ll)) (recur (inc idx))) (if-let [rhs (.trySplit ^Spliterator ll)] (do (.add rv (inc idx) rhs) (recur idx)) (do (.set rv idx (op ll)) (recur (inc idx)))))))))))) (comment (defn ^:no-doc ms-parallel-reduce "Perform a parallel reduction of a spliterator using the provided ExecutorService" [executor-service split ideal-split init-fn rfn merge-fn] (let [split (->spliterator split) n-elems (.estimateSize split) pool (or executor-service (ForkJoinPool/commonPool)) ideal-split (long ideal-split)] (if (or (<= n-elems ideal-split) (= n-elems Long/MAX_VALUE)) (split-reduce rfn (init-fn) split) (let [data (split-to-max-size split ideal-split #(fjp/safe-fork-task pool (split-parallel-reduce pool % ideal-split init-fn rfn merge-fn))) merge-fn #(reduce merge-fn (lznc/map fjp/managed-block-unwrap %))] (loop [merge-data data] (if (>= (count merge-data) ideal-split) (recur (split-to-max-size merge-data ideal-split #(fjp/safe-fork-task pool (merge-fn %)))) (merge-fn merge-data))))))) ) ================================================ FILE: src/ham_fisted/thread_local.clj ================================================ (ns ham-fisted.thread-local) (defn thread-local "Create a new thread local variable" (^ThreadLocal [] (thread-local nil)) (^ThreadLocal [v] (ThreadLocal/withInitial (reify java.util.function.Supplier (get [this] v))))) ================================================ FILE: test/ham_fisted/api_test.clj ================================================ (ns ham-fisted.api-test (:require [clojure.test :refer [deftest is]] [ham-fisted.api :as hamf] [ham-fisted.alists :as alists] [ham-fisted.reduce :as hamf-rf] [ham-fisted.function :as hamf-fn] [ham-fisted.lazy-noncaching :as lznc] [ham-fisted.iterator :as hamf-iter] [ham-fisted.set :as hamf-set]) (:import [java.util BitSet] [java.util.function Consumer DoubleConsumer LongConsumer] [clojure.lang IDeref] [ham_fisted Reducible])) (deftest parallism-primitives-pass-errors (is (thrown-with-msg? Exception #"Error!!" (doall (hamf/upmap (fn [^long idx] (when (== idx 77) (throw (Exception. "Error!!"))) idx) (range 100))))) (is (thrown-with-msg? Exception #"Error!!" (doall (hamf/pmap (fn [^long idx] (when (== idx 77) (throw (Exception. "Error!!"))) idx) (range 100))))) (is (thrown-with-msg? Exception #"Error!!" (doall (hamf/upgroups 1000 (fn [^long sidx ^long eidx] (when (>= sidx 10) (throw (Exception. "Error!!"))) sidx) {:batch-size 100})))) (is (thrown-with-msg? Exception #"Error!!" (doall (hamf/pgroups 1000 (fn [^long sidx ^long eidx] (when (>= sidx 10) (throw (Exception. "Error!!"))) sidx) {:batch-size 100}))))) (deftest group-by-nil (is (= {} (hamf/group-by :a nil))) (is (= {} (hamf/group-by-reduce :a + + + nil))) (is (= {} (hamf/group-by :a {}))) (is (= {} (hamf/group-by-reduce :a + + + {}))) ) (deftest map-filter-concat-nil (is (= (map + nil) (lznc/map + nil))) (is (= (filter + nil) (lznc/filter + nil))) (is (= (concat) (lznc/concat))) (is (= (concat nil) (lznc/concat nil)))) (deftest conj-with-friends (is (= (conj (map inc (list 1 2 3 4)) 4) (conj (lznc/map inc (list 1 2 3 4)) 4))) (is (= (conj (filter even? (list 1 2 3 4)) 4) (conj (lznc/filter even? (list 1 2 3 4)) 4))) (is (= (conj (concat [1 2 3] [4 5 6]) 4) (conj (lznc/concat [1 2 3] [4 5 6]) 4))) (is (= (conj (map-indexed + [1 2 3]) 4) (conj (lznc/map-indexed + (apply list [1 2 3])) 4))) (is (= (conj (map-indexed + nil) 4) (conj (lznc/map-indexed + nil) 4)))) (deftest empty-seq-preduce (is (== 0.0 (hamf/sum (list)))) (is (== 19900.0 (hamf/sum (range 200)))) (is (= 1 (hamf-rf/preduce (constantly 1) + + nil))) (is (= 1 (hamf-rf/preduce (constantly 1) + + (list))))) (deftest group-by-reduce-large-n (is (= 113 (count (hamf/group-by #(rem (unchecked-long %1) 113) (range 10000))))) (is (= 337 (count (hamf/group-by-reduce #(rem (unchecked-long %1) 337) + + + (range 10000)))))) (deftest compare-seq-with-nonseq (is (not (= (hamf/vec (range 10)) :a)))) (deftest BitsetSet (let [bs (doto (BitSet.) (.set 1) (.set 10))] (is (= [1 10] (hamf/->random-access (hamf/int-array bs)))) (is (= [1.0 10.0] (->> bs (lznc/map (hamf-fn/long->double v (double v))) (hamf/vec)))) (is (not (nil? (hamf/->collection bs)))))) (deftest char-array-reduction (let [cv (hamf/char-array [20 30])] (is (= [(char 20) (char 30)] (reduce conj [] cv)))) (let [cv (hamf/char-array "hey")] (is (= [\h \e \y] (reduce conj [] cv))))) (deftest java-maps-are-iterable (is (not (nil? (hamf/->collection (hamf/java-hashmap {:a 1 :b 2})))))) (deftest set-add-long-val (let [alist (hamf/long-array-list)] (.add alist Long/MAX_VALUE) (is (= [Long/MAX_VALUE] alist)) (.set alist 0 Long/MAX_VALUE) (is (= [Long/MAX_VALUE] alist)) (is (= [Long/MAX_VALUE] (vec alist))) (let [sl (.subList alist 0 1)] (is (= [Long/MAX_VALUE] sl)) (is (= [Long/MAX_VALUE] (vec sl)))))) (deftest tostring-empty-range (is (= "[]" (.toString (hamf/range 0)))) (is (= "[]" (.toString (hamf/range 0.0))))) (deftest map-filter-concat-reduced (let [rfn (hamf-rf/long-accumulator acc v (if (< v 4) acc (reduced v)))] (is (= 4 (reduce rfn 0 (lznc/map (hamf-fn/long-unary-operator v (inc v)) (hamf/range 20))))) (is (= 4 (reduce rfn 0 (lznc/filter even? (hamf/range 20))))) (is (= 4 (reduce rfn 0 (lznc/concat (hamf/range 20) (hamf/range 20 50))))) )) (deftest into-array-nil (is (== 0 (count (hamf/into-array Double #(double %) nil))))) (deftest mmax-key-clojure-map (is (= 3 (key (hamf/mmax-key val (frequencies (map #(rem % 7) (hamf/range 10000)))))))) (deftest reduce-empty-range (is (= 0 (reduce (fn [acc v] (inc acc)) 0 (hamf/range 0))))) (deftest assoc-basic-fail (let [m1 (hamf/immut-map {:age 2, :sex 1, "salary (binned)" 4, :job 0, :salary 3}) m2 (assoc m1 :tech.v3.dataset.reductions/_tmp_col 5)] (is (= (set (keys m1)) #{:age :sex "salary (binned)" :job :salary})))) (deftest set-api-compat (is (= #{:a} (disj (hamf-set/difference (hamf/immut-set [:a :b :c]) (hamf/immut-set [:c :d :e])) :b))) (is (= #{} (hamf/intersection (hamf/immut-set #{:a :b}) #{})))) (deftest test-partition-by (is (= [[1 1 1] [2 2 2] [3 3 3]] (vec (map vec (lznc/partition-by identity [1 1 1 2 2 2 3 3 3]))))) (is (= [[1 1 1] [2 2 2] [3 3 3]] (hamf/mapv vec (lznc/partition-by identity [1 1 1 2 2 2 3 3 3])))) (is (= (clojure.core/partition-by identity []) (lznc/partition-by identity []))) (is (= (clojure.core/partition-by identity nil) (lznc/partition-by identity nil))) ;;Ensure we catch incorrect usage when possible (is (thrown? RuntimeException (into [] (lznc/partition-by identity [1 1 1 2 2 2 3 3 3]))))) (deftype ^:private LongAccum [^:unsynchronized-mutable v] LongConsumer (accept [this val] (set! v (+ val v))) Reducible (reduce [this o] (+ v @o)) IDeref (deref [this] v)) (deftype ^:private DoubleAccum [^:unsynchronized-mutable v] DoubleConsumer (accept [this val] (set! v (+ val v))) Reducible (reduce [this o] (+ v @o)) IDeref (deref [this] v)) (deftype ^:private Accum [^:unsynchronized-mutable v] Consumer (accept [this val] (set! v (+ val v))) Reducible (reduce [this o] (+ v @o)) IDeref (deref [this] v)) (defn filter-sum-reducer [type] (case type :int64 (hamf-rf/long-consumer-reducer #(LongAccum. 0)) :float64 (hamf-rf/double-consumer-reducer #(DoubleAccum. 0)) (hamf-rf/consumer-reducer #(Accum. 0.0)))) (deftest compose-reducers (is (= {:a 49995000} (hamf-rf/preduce-reducers {:a (filter-sum-reducer :int64)} {:rfn-datatype :int64} (range 10000)))) (is (= {:a 49995000.0} (hamf-rf/preduce-reducers {:a (filter-sum-reducer :float64)} {:rfn-datatype :float64} (range 10000)))) (is (= {:a 49995000.0} (hamf-rf/preduce-reducers {:a (filter-sum-reducer nil)} (range 10000))))) (deftest test-partition-all (is (= [[0 1 2] [3 4 5] [6 7 8] [9]] (vec (map vec (lznc/partition-all 3 (range 10)))))) ;;Ensure we catch incorrect usage when possible (is (thrown? RuntimeException (into [] (lznc/partition-all 3 (range 10)))))) (deftest parallel-frequenies (is (= {0 715, 1 715, 2 715, 3 715, 4 714, 5 714, 6 714, 7 714, 8 714, 9 714, 10 714, 11 714, 12 714, 13 714} (hamf/frequencies (lznc/map #(rem (long %) 14) (hamf/range 10000)))))) (deftest range-seq (is (= (vec (range 40)) (vec (apply list (hamf/range 40)))))) (deftest drop-elems (is (empty? (hamf/drop 10 [1 2 3 4 5])))) (deftest bulk-insert-constant (let [src [0 1 2 3 4 100 100 100 5 6 7 8 9]] (reduce (fn [acc v] (if (not= v :boolean) (is (= (alists/growable-array-list v src) (-> (alists/growable-array-list v (range 10)) (hamf/add-constant! 5 3 100))) (str "Test failed for datatype: " v)) (let [src [true true false false true true]] (is (= (alists/growable-array-list v src) (-> (alists/growable-array-list v [true true true true]) (hamf/add-constant! 2 2 false))) (str "Test failed for datatype: " v))))) [] alists/array-list-types)) (let [src [0 1 2 3 4 5 6 7 8 9 100 100 100]] (reduce (fn [acc v] (if (not= v :boolean) (is (= (alists/growable-array-list v src) (-> (alists/growable-array-list v (range 10)) (hamf/add-constant! 10 3 100))) (str "Test failed for datatype: " v)) (let [src [true true true true false false]] (is (= (alists/growable-array-list v src) (-> (alists/growable-array-list v [true true true true]) (hamf/add-constant! 4 2 false))) (str "Test failed for datatype: " v))))) [] alists/array-list-types))) (deftest update-values (is (= {:a false} (hamf/update-values (array-map :a 1) (hamf-fn/bi-function k v false)))) (is (= {:a false} (into {} (hamf/update-values (hamf/java-hashmap {:a 1}) (hamf-fn/bi-function k v false))))) (is (= {:a false} (hamf/update-values (hamf/mut-map {:a 1}) (hamf-fn/bi-function k v false))))) (deftest incorrect-map-difference (is (= (hamf/java-hashmap {:a 2}) (hamf/difference (hamf/java-hashmap {:a 2 :b 2}) [:b :c])))) (deftest hash-map-compute (is (= {} (-> (let [ht (hamf/mut-map {1 2})] (.compute ht 1 (hamf-fn/bi-function k v nil)) (persistent! ht))))) (is (= {} (-> (let [ht (hamf/mut-map {1 2})] (is (= 2 (.remove ht 1))) (persistent! ht)))))) (deftest pmap-opts-empty-result (is (= [1 2 3 4] (vec (hamf/pmap-opts {:pool clojure.lang.Agent/soloExecutor :lookahead 2} identity [1 2 3 4]))))) (deftest reduce-empty-apply-concat (is (= 0 (reduce + 0 (lznc/apply-concat nil))))) (deftest drop-test (is (= (drop 3 '(1 2 3 4)) (lznc/drop 3 '(1 2 3 4))))) (deftest take-test (is (= (take 2 '(1 2 3 4)) (lznc/take 2 '(1 2 3 4))))) (deftest seq-iterable-not-counted (is (not (counted? (hamf-iter/wrap-iter (.iterator (range))))))) (comment (do (def left-chunks (mapv hamf/long-array-list (partition-all 1000 (range 10000000)))) (def right-partitions (partition-all 100 (shuffle (range 100000)))) (require '[clj-async-profiler.core :as prof]) (prof/serve-ui 8080)) (def data (prof/profile (dotimes [idx 10] (time (->> (lznc/concat left-chunks right-partitions) (lznc/apply-concat) (hamf/sort-by (fn ^long [a] a)) (hamf/lsum)))))) (def data (vec '(1 2 3 4 5 6 7))) (into [] (lznc/take 2) data) (lznc/drop 2 data) (drop 2 nil) (lznc/drop 1 [1 2]) (lznc/take 2 data) (time (->> (apply concat left-chunks) (drop 17) (hamf/lsum))) (prof/profile (dotimes [idx 10] (time (->> (lznc/apply-concat left-chunks) (lznc/take 10000000) (hamf/sum))))) ) ================================================ FILE: test/ham_fisted/bloom_filter_test.clj ================================================ (ns ham-fisted.bloom-filter-test (:require [ham-fisted.bloom-filter :as hamf-bf] [clojure.test :refer [deftest is]]) (:import [java.util UUID])) (deftest uuid-test (let [M (long 1e6) uuids (vec (repeatedly M #(UUID/randomUUID))) bf (hamf-bf/bloom-filter M 0.01) pred (hamf-bf/make-obj-predicate bf)] (reduce hamf-bf/insert-hash! bf (map hamf-bf/hash-obj uuids)) (is (pred (uuids 0))) (let [false-pos (long (reduce (fn [eax _] (if (pred (UUID/randomUUID)) (inc eax) eax)) 0 (range M))) true-pos (long (reduce (fn [eax u] (if (pred u) (inc eax) eax)) 0 uuids))] (is (< false-pos 2000)) (is (== M true-pos))))) ================================================ FILE: test/ham_fisted/cast_test.clj ================================================ (ns ham-fisted.cast-test (:require [ham-fisted.api :as api] [ham-fisted.function :as hamf-fn] [ham-fisted.lazy-noncaching :as lznc] [clojure.test :refer [deftest is]])) (deftest basic-casts (is (thrown? Exception (api/long-array [0.0 ##NaN 0.0]))) (is (api/double-eq [0.0 ##NaN 0.0] (api/double-array [0 nil 0]))) (is (thrown? Exception (api/long-array (lznc/map (fn ^double [^double v] (+ v 2.0)) [0.0 ##NaN 0.0])))) (is (thrown? Exception (api/double-array (->> [0.0 ##NaN 0.0] (lznc/map (fn ^double [^double v] (+ v 2.0))) (lznc/filter (hamf-fn/long-predicate v (not (== 0 (rem v 3))))))))) (is (thrown? Exception (vec (lznc/map (hamf-fn/long-unary-operator v (+ v 2)) [0.0 ##NaN 0.0])))) (is (thrown? Exception (vec (->> [0.0 ##NaN 0.0] (hamf-fn/double-unary-operator v (+ v 2.0)) (lznc/filter (hamf-fn/long-predicate v (not (== 0 (rem v 3))))))))) (is (= [false false false] (api/->random-access (api/boolean-array [nil ##NaN nil])))) (is (= [false false] (api/->random-access (api/boolean-array [nil nil]))))) ================================================ FILE: test/ham_fisted/defprotocol_test/examples.clj ================================================ (ns ham-fisted.defprotocol-test.examples (:refer-clojure :exclude [defprotocol]) (:require [ham-fisted.defprotocol :refer [defprotocol]])) (defprotocol ExampleProtocol "example protocol used by clojure tests" (foo [a] "method with one arg") (bar [a b] "method with two args") (^String baz [a] [a b] "method with multiple arities") (with-quux [a] "method name with a hyphen")) (defprotocol MarkerProtocol "a protocol with no methods") (defprotocol MarkerProtocol2) (definterface ExampleInterface (hinted [^int i]) (hinted [^String s])) (defprotocol LongsHintedProto (^longs longs-hinted [_])) ================================================ FILE: test/ham_fisted/defprotocol_test/hash_collisions_test.clj ================================================ (ns ham-fisted.defprotocol-test.hash-collisions-test (:refer-clojure :exclude [defprotocol extend-type extend extend-protocol satisfies? extends?]) (:require [clojure.test :refer [deftest is]] [ham-fisted.defprotocol :refer [defprotocol extend-type extend extend-protocol satisfies? extends?]])) (defprotocol TestProtocolA (method-a [this] "Test method A")) (defprotocol TestProtocolB (method-b [this] "Test method B")) (deftype TestType1 []) (deftype TestType2 []) (deftype TestType3 []) (deftype TestType4 []) (deftype TestType5 []) (deftype TestType6 []) (deftype TestType7 []) (deftype TestType8 []) (deftype TestType9 []) (deftype TestType10 []) (deftype TestType11 []) (deftype TestType12 []) (deftype TestType13 []) (deftype TestType14 []) (deftype TestType15 []) (def original-hash hash) (defn colliding-hash "Mock hash function which returns identical hash codes for the classes TestType1 and TestType2, normal hashes for everything else." [x] (if (or (= x TestType1) (= x TestType2)) 999 ;; artificial hash code, to cause a collision (original-hash x))) (deftest protocols-with-hash-collisions (with-redefs [hash colliding-hash] (extend TestType1 TestProtocolA {:method-a (constantly 1)}) (extend TestType2 TestProtocolA {:method-a (constantly 2)}) (is (= 1 (method-a (TestType1.)))) (is (= 2 (method-a (TestType2.)))))) (defn no-min-hash-in-13-bits "Mock hash function which returns hash codes for the classes TestType1 through TestType15 such that they cannot be compressed into a 13-bit min-hash table. Returns normal hashes for everything else." [x] (cond (= x TestType1) 1 (= x TestType2) 2 (= x TestType3) 4 (= x TestType4) 8 (= x TestType5) 16 (= x TestType6) 32 (= x TestType7) 64 (= x TestType8) 128 (= x TestType9) 256 (= x TestType10) 512 (= x TestType11) 1024 (= x TestType12) 2048 (= x TestType13) 4096 (= x TestType14) 8192 (= x TestType15) 16384 :else (original-hash x))) (deftest protocols-with-no-min-hash-in-13-bits (with-redefs [hash no-min-hash-in-13-bits] (extend TestType1 TestProtocolB {:method-b (constantly 1)}) (extend TestType2 TestProtocolB {:method-b (constantly 2)}) (extend TestType3 TestProtocolB {:method-b (constantly 3)}) (extend TestType4 TestProtocolB {:method-b (constantly 4)}) (extend TestType5 TestProtocolB {:method-b (constantly 5)}) (extend TestType6 TestProtocolB {:method-b (constantly 6)}) (extend TestType7 TestProtocolB {:method-b (constantly 7)}) (extend TestType8 TestProtocolB {:method-b (constantly 8)}) (extend TestType9 TestProtocolB {:method-b (constantly 9)}) (extend TestType10 TestProtocolB {:method-b (constantly 10)}) (extend TestType11 TestProtocolB {:method-b (constantly 11)}) (extend TestType12 TestProtocolB {:method-b (constantly 12)}) (extend TestType13 TestProtocolB {:method-b (constantly 13)}) (extend TestType14 TestProtocolB {:method-b (constantly 14)}) (extend TestType15 TestProtocolB {:method-b (constantly 15)}) (is (= 1 (method-b (TestType1.)))) (is (= 2 (method-b (TestType2.)))) (is (= 3 (method-b (TestType3.)))) (is (= 4 (method-b (TestType4.)))) (is (= 5 (method-b (TestType5.)))) (is (= 6 (method-b (TestType6.)))) (is (= 7 (method-b (TestType7.)))) (is (= 8 (method-b (TestType8.)))) (is (= 9 (method-b (TestType9.)))) (is (= 10 (method-b (TestType10.)))) (is (= 11 (method-b (TestType11.)))) (is (= 12 (method-b (TestType12.)))) (is (= 13 (method-b (TestType13.)))) (is (= 14 (method-b (TestType14.)))) (is (= 15 (method-b (TestType15.)))))) ================================================ FILE: test/ham_fisted/defprotocol_test/more_examples.clj ================================================ (ns ham-fisted.defprotocol-test.more-examples (:require [ham-fisted.defprotocol :as hamf-defp])) (hamf-defp/defprotocol SimpleProtocol "example protocol used by clojure tests. Note that foo collides with examples/ExampleProtocol." (foo [a] "")) ================================================ FILE: test/ham_fisted/defprotocol_test/other_test.clj ================================================ (ns ham-fisted.defprotocol-test.other-test (:refer-clojure :exclude [defprotocol extend-type extend extend-protocol satisfies? extends?]) (:require [clojure.test :refer [deftest is]] [ham-fisted.defprotocol :refer [defprotocol extend-type extend extend-protocol satisfies? extends?]] [ham-fisted.defprotocol-test]) (:refer-clojure :exclude [defprotocol extend-type extend extend-protocol satisfies? extends?])) (defn cf [val] (let [aseq (ham-fisted.defprotocol-test/f val)] (count aseq))) (extend-protocol ham-fisted.defprotocol-test/P String (f [s] (seq s))) (deftest test-resolve-type-hints-in-protocol-methods (is (= 4 (cf "test")))) ================================================ FILE: test/ham_fisted/defprotocol_test.clj ================================================ ;; Copyright (c) Rich Hickey. All rights reserved. The use and distribution terms for ;; this software are covered by the Eclipse Public License 1.0 ;; (http://opensource.org/licenses/eclipse-1.0.php) which can be found in the file ;; epl-v10.html at the root of this distribution. By using this software in any ;; fashion, you are agreeing to be bound by the terms of this license. You must not ;; remove this notice, or any other, from this software. ;; Author: Stuart Halloway ;; Heavy modification - Chris Nuernberger (ns ham-fisted.defprotocol-test (:require [ham-fisted.defprotocol-test.examples :refer :all] [ham-fisted.defprotocol-test.more-examples :as other] [ham-fisted.defprotocol :refer [defprotocol extend-type extend extend-protocol satisfies? extends?] :as defprotocol] [ham-fisted.api :as hamf] [ham-fisted.reduce :as hamf-rf] [ham-fisted.lazy-noncaching :as lznc] [clojure.set :as set] [clojure.test :refer [deftest testing are is do-report assert-expr report]]) (:import [ham_fisted.defprotocol_test.examples ExampleInterface] [java.util LongSummaryStatistics] [java.util.function LongConsumer]) (:refer-clojure :exclude [defprotocol extend-type extend extend-protocol satisfies? extends?]) ) (set! *warn-on-reflection* true) (defn causes [^Throwable throwable] (loop [causes [] t throwable] (if t (recur (conj causes t) (.getCause t)) causes))) ;; this is how I wish clojure.test/thrown? worked... ;; Does body throw expected exception, anywhere in the .getCause chain? (defmethod assert-expr 'fails-with-cause? [msg [_ exception-class msg-re & body :as form]] `(try ~@body (report {:type :fail, :message ~msg, :expected '~form, :actual nil}) (catch Throwable t# (if (some (fn [cause#] (and (= ~exception-class (class cause#)) (re-find ~msg-re (.getMessage ^Throwable cause#)))) (causes t#)) (report {:type :pass, :message ~msg, :expected '~form, :actual t#}) (report {:type :fail, :message ~msg, :expected '~form, :actual t#}))))) (defmethod clojure.test/assert-expr 'thrown-with-cause-msg? [msg form] ;; (is (thrown-with-cause-msg? c re expr)) ;; Asserts that evaluating expr throws an exception of class c. ;; Also asserts that the message string of the *cause* exception matches ;; (with re-find) the regular expression re. (let [klass (nth form 1) re (nth form 2) body (nthnext form 3)] `(try ~@body (do-report {:type :fail, :message ~msg, :expected '~form, :actual nil}) (catch ~klass e# (let [m# (if (.getCause e#) (.. e# getCause getMessage) (.getMessage e#))] (if (re-find ~re m#) (do-report {:type :pass, :message ~msg, :expected '~form, :actual e#}) (do-report {:type :fail, :message ~msg, :expected '~form, :actual e#}))) e#)))) (defn method-names "return sorted list of method names on a class" [c] (->> (.getMethods ^Class c) (map #(.getName ^java.lang.reflect.Method %)) (sort))) (defrecord EmptyRecord []) (defrecord TestRecord [a b]) (defn r ([a b] (TestRecord. a b)) ([a b meta ext] (TestRecord. a b meta ext))) (defrecord MapEntry [k v] java.util.Map$Entry (getKey [_] k) (getValue [_] v)) (deftest protocols-test (testing "protocol fns have useful metadata" (let [common-meta {:ns (find-ns 'ham-fisted.defprotocol-test.examples) :tag nil}] (are [m f] (= (merge common-meta m) (dissoc (meta (var f)) :line :column :file :hamf-protocol)) {:name 'foo :arglists '([a]) :doc "method with one arg"} foo {:name 'bar :arglists '([a b]) :doc "method with two args"} bar {:name 'baz :arglists '([a] [a b]) :doc "method with multiple arities" :tag 'java.lang.String} baz {:name 'with-quux :arglists '([a]) :doc "method name with a hyphen"} with-quux))) (testing "protocol fns throw IllegalArgumentException if no impl matches" (is (thrown-with-msg? IllegalArgumentException #"No implementation of method: :foo of protocol: #'ham-fisted.defprotocol-test.examples/ExampleProtocol found for class: java.lang.Long" (foo 10)))) (testing "protocols generate a corresponding interface using _ instead of - for method names" (is (= ["bar" "baz" "baz" "foo" "with_quux"] (method-names ham_fisted.defprotocol_test.examples.ExampleProtocol)))) (testing "protocol will work with instances of its interface (use for interop, not in Clojure!)" (let [obj (proxy [ham_fisted.defprotocol_test.examples.ExampleProtocol] [] (foo [] "foo!"))] (is (= "foo!" (.foo obj)) "call through interface") (is (= "foo!" (foo obj)) "call through protocol"))) (testing "you can implement just part of a protocol if you want" (let [obj (reify ExampleProtocol (baz [a b] "two-arg baz!"))] (is (= "two-arg baz!" (baz obj nil))) (is (thrown? AbstractMethodError (baz obj))))) (testing "error conditions checked when defining protocols" (is #_{:clj-kondo/ignore [:unresolved-symbol]} (thrown-with-cause-msg? Exception #"Definition of function m in protocol badprotdef must take at least one arg." (eval '(defprotocol badprotdef (m []))))) (is #_{:clj-kondo/ignore [:unresolved-symbol]} (thrown-with-cause-msg? Exception #"Function m in protocol badprotdef was redefined. Specify all arities in single definition." (eval '(defprotocol badprotdef (m [this arg]) (m [this arg1 arg2])))))) (testing "you can redefine a protocol with different methods" (eval '(defprotocol Elusive (old-method [x]))) (eval '(defprotocol Elusive (new-method [x]))) (is (= :new-method (eval '(new-method (reify Elusive (new-method [x] :new-method)))))) (is #_{:clj-kondo/ignore [:unresolved-symbol]} (fails-with-cause? IllegalArgumentException #"No method of interface: .*\.Elusive found for function: old-method of protocol: Elusive \(The protocol method may have been defined before and removed\.\)" (eval '(old-method (reify Elusive (new-method [x] :new-method)))))))) (deftype HasMarkers [] ExampleProtocol (foo [this] "foo") MarkerProtocol MarkerProtocol2) (deftype WillGetMarker [] ExampleProtocol (foo [this] "foo")) (extend-type WillGetMarker MarkerProtocol) (deftest marker-tests (testing "That a marker protocol has no methods" (is (= '() (method-names ham_fisted.defprotocol_test.examples.MarkerProtocol)))) (testing "That types with markers are reportedly satifying them." (let [hm (HasMarkers.) wgm (WillGetMarker.)] (is (satisfies? MarkerProtocol hm)) (is (satisfies? MarkerProtocol2 hm)) (is (satisfies? MarkerProtocol wgm))))) (deftype ExtendTestWidget [name]) (deftype HasProtocolInline [] ExampleProtocol (foo [this] :inline)) (deftest extend-test (testing "you can extend a protocol to a class" (extend String ExampleProtocol {:foo identity}) (is (= "pow" (foo "pow")))) (testing "you can have two methods with the same name. Just use namespaces!" (extend String other/SimpleProtocol {:foo (fn [s] (.toUpperCase ^String s))}) (is (= "POW" (other/foo "pow")))) (testing "you can extend deftype types" (extend ExtendTestWidget ExampleProtocol {:foo (fn [this] (str "widget " (.name ^ExtendTestWidget this)))}) (is (= "widget z" (foo (ExtendTestWidget. "z")))))) (deftest illegal-extending (testing "you cannot extend a protocol to a type that implements the protocol inline" (is #_{:clj-kondo/ignore [:unresolved-symbol]} (fails-with-cause? IllegalArgumentException #".*HasProtocolInline already directly implements interface" (eval '(extend ham_fisted.defprotocol_test.HasProtocolInline ham-fisted.defprotocol-test.examples/ExampleProtocol {:foo (fn [_] :extended)}))))) (testing "you cannot extend to an interface" (is #_{:clj-kondo/ignore [:unresolved-symbol]} (fails-with-cause? IllegalArgumentException #"interface ham_fisted.defprotocol_test.examples.ExampleProtocol is not a protocol" (eval '(extend ham_fisted.defprotocol_test.HasProtocolInline ham_fisted.defprotocol_test.examples.ExampleProtocol {:foo (fn [_] :extended)})))))) ; see CLJ-845 (defprotocol SyntaxQuoteTestProtocol (sqtp [p])) (defmacro try-extend-type [c] `(extend-type ~c SyntaxQuoteTestProtocol (sqtp [p#] p#))) (defmacro try-extend-protocol [c] `(extend-protocol SyntaxQuoteTestProtocol ~c (sqtp [p#] p#))) (try-extend-type String) (try-extend-protocol clojure.lang.Keyword) (deftest test-no-ns-capture (is (= "foo" (sqtp "foo"))) (is (= :foo (sqtp :foo)))) (defprotocol Dasherizer (-do-dashed [this])) (deftype Dashed [] Dasherizer (-do-dashed [this] 10)) (deftest test-leading-dashes (is (= 10 (-do-dashed (Dashed.)))) (is (= [10] (map -do-dashed [(Dashed.)])))) ;; see CLJ-1879 (deftest test-base-reduce-kv (is (= {1 :a 2 :b} (reduce-kv #(assoc %1 %3 %2) {} (seq {:a 1 :b 2}))))) (defn aget-long-hinted ^long [x] (aget (longs-hinted x) 0)) (deftest test-longs-hinted-proto (is (= 1 (aget-long-hinted (reify LongsHintedProto (longs-hinted [_] (long-array [1]))))))) ;; CLJ-1180 - resolve type hints in protocol methods (import 'clojure.lang.ISeq) (defprotocol P (^ISeq f [_])) ;;; continues in defprotocol_test/other.clj (defprotocol Ecount (^long ecount [m])) (extend Object Ecount {:ecount 5}) (extend Number Ecount {:ecount (fn ^long [m] 0)}) (extend-type Long Ecount (ecount [m] 10)) ;;Hamf protocols have to work with extend-type and reify (deftype TestType [] Ecount (ecount [this] 200)) (deftest primitive-hinted-test (is (== 5 (ecount :a))) (is (== 0 (ecount 1.0))) (is (== 10 (ecount 1))) (is (== 200 (ecount (TestType.)))) (is (thrown? Exception (eval '(clojure.core/extend-type String Ecount (ecount [m] (.length m))))))) (defprotocol SaneInheritance (a [r]) (b [r])) (extend-type Object SaneInheritance (a [r] :object) (b [r] :object)) (extend-type String SaneInheritance (b [r] :string)) (deftest sane-inheritance-test (is (= :object (a "hey"))) (is (= :string (b "hey")))) (defprotocol HamfMemsize (^long hamf-memsize [m])) (extend Double HamfMemsize {:hamf-memsize 24}) (extend Long HamfMemsize {:hamf-memsize 24}) (extend clojure.lang.Keyword HamfMemsize {:hamf-memsize 48}) (extend-protocol HamfMemsize String (hamf-memsize [s] (+ 24 (.length ^String s))) java.util.Collection (hamf-memsize [c] (hamf/lsum (lznc/map (fn ^long [d] (+ 24 (hamf-memsize d))) c))) java.util.Map (hamf-memsize [c] (hamf/lsum (lznc/map (fn ^long [kv] (+ 36 (+ (hamf-memsize (key kv)) (hamf-memsize (val kv))))) c)))) (clojure.core/defprotocol CoreMemsize (core-memsize [m])) (clojure.core/extend-protocol CoreMemsize Double (core-memsize [d] 24) Long (core-memsize [l] 24) clojure.lang.Keyword (core-memsize [k] 48) String (core-memsize [s] (+ 24 (.length ^String s))) java.util.Collection (core-memsize [c] (hamf/lsum (lznc/map (fn ^long [d] (+ 24 (long (core-memsize d)))) c)) #_(reduce (fn [s v] (+ s 24 (core-memsize v))) 0 c)) java.util.Map (core-memsize [m] (hamf/lsum (lznc/map (fn ^long [kv] (+ 36 (+ (long (core-memsize (key kv))) (long (core-memsize (val kv)))))) m)) #_(reduce (fn [s [k v]] (+ s 36 (core-memsize k) (core-memsize v))) 0 m))) (def test-datastructure {:a "hello" :b 24 :c (into [] (repeat 1000 (rand))) :d (into [] (repeat 1000 1))}) (def measure-data (into [] (repeat 10000 test-datastructure))) (defn multithread-test [measure-fn] (hamf/lsum (hamf/pmap measure-fn measure-data))) (defprotocol PDatatype (datatype [v])) (extend String PDatatype {:datatype :string}) (extend clojure.lang.Keyword PDatatype {:datatype :keyword}) (deftest object-constants (is (= :string (datatype "key"))) (is (= :keyword (datatype :key)))) (defn obj-cls->datatype [obj-cls] (if (identical? String obj-cls) :string :object)) (extend (type (object-array 0)) PDatatype {:datatype (fn [obj] (-> (.getClass ^Object obj) (.getComponentType) (obj-cls->datatype)))}) (extend (type (make-array clojure.lang.Keyword 0)) PDatatype {:datatype (constantly :keyword-array-overload)}) (deftest test-object-array-protocols (is (= :object (datatype (object-array 0)))) (is (= :string (datatype (make-array String 0)))) (is (= :keyword-array-overload (datatype (make-array clojure.lang.Keyword 0))))) (defprotocol PPrimitiveArgs (^double pargs [m ^long b])) (extend-type String PPrimitiveArgs (pargs [m b] (+ 1.0 (+ (.length m) b)))) (deftest test-hinted (is (instance? clojure.lang.IFn$OL hamf-memsize)) (is (instance? clojure.lang.IFn$OL -hamf-memsize-iface)) (is (instance? clojure.lang.IFn$OLD pargs)) (is (instance? clojure.lang.IFn$OLD -pargs-iface)) (is (= 14.0 (pargs "hey" 10)))) (defprotocol SubBuffer (sub-buffer [m ^long a ^long b])) (deftest test-sub-buffer (is (instance? clojure.lang.IFn$OLLO sub-buffer)) (is (instance? clojure.lang.IFn$OLLO -sub-buffer-iface))) (comment (require '[criterium.core :as crit]) ;;Single threaded calls show very little difference if any: (crit/quick-bench (core-memsize test-datastructure)) ;; Evaluation count : 23868 in 6 samples of 3978 calls. ;; Execution time mean : 25.018494 µs ;; Execution time std-deviation : 26.939284 ns ;; Execution time lower quantile : 24.969740 µs ( 2.5%) ;; Execution time upper quantile : 25.046386 µs (97.5%) ;; Overhead used : 1.539613 ns ;; Multithreaded calls however: (crit/quick-bench (hamf-memsize test-datastructure)) ;; Evaluation count : 22734 in 6 samples of 3789 calls. ;; Execution time mean : 26.336031 µs ;; Execution time std-deviation : 202.486707 ns ;; Execution time lower quantile : 25.953198 µs ( 2.5%) ;; Execution time upper quantile : 26.466131 µs (97.5%) ;; Overhead used : 1.539613 ns (crit/quick-bench (multithread-test core-memsize)) ;; Evaluation count : 6 in 6 samples of 1 calls. ;; Execution time mean : 634.046041 ms ;; Execution time std-deviation : 9.874821 ms ;; Execution time lower quantile : 622.889124 ms ( 2.5%) ;; Execution time upper quantile : 646.951202 ms (97.5%) ;; Overhead used : 1.539613 ns (crit/quick-bench (multithread-test hamf-memsize)) ;; Evaluation count : 18 in 6 samples of 3 calls. ;; Execution time mean : 45.689563 ms ;; Execution time std-deviation : 1.272542 ms ;; Execution time lower quantile : 44.302374 ms ( 2.5%) ;; Execution time upper quantile : 47.396205 ms (97.5%) ;; Overhead used : 1.539613 ns ) ================================================ FILE: test/ham_fisted/fjp_test.clj ================================================ (ns ham-fisted.fjp-test (:require [ham-fisted.api :as hamf] [ham-fisted.fjp :as fjp] [ham-fisted.protocols :as proto] [ham-fisted.function :as hamf-fn] [ham-fisted.reduce :as hamf-rf] [ham-fisted.language :refer [cond not]] [ham-fisted.spliterator :as spliterator] [clojure.test :refer [deftest is]]) (:import [java.util List Spliterator Spliterator$OfDouble Spliterator$OfLong] [java.util.function Consumer DoubleConsumer LongConsumer] [clojure.lang IDeref IFn$LLL IFn$DDD] [ham_fisted Consumers$IDerefLongConsumer Consumers$IDerefDoubleConsumer]) (:refer-clojure :exclude [cond not])) (set! *warn-on-reflection* true) (set! *unchecked-math* :warn-on-boxed) (defn lp ^long [^long a ^long b] (+ a b)) (defn dp ^double [^double a ^double b] (+ a b)) (defn sum-reducer [] (reify proto/Reducer (->init-val-fn [_] #(long-array 1)) (->rfn [_] (fn [^longs lv ^long v] (let [_ (aset lv 0 (+ (aget lv 0) v))] lv))) proto/ParallelReducer (->merge-fn [_] (fn [^longs lv ^longs rv] (let [_ (aset lv 0 (+ (aget lv 0) (aget rv 0)))] lv))) proto/Finalize (finalize [this lv] (aget ^longs lv 0)))) (deftest parallel-summation (let [data (hamf/range 1000000) total (hamf/lsum data)] (is (= total (spliterator/split-parallel-reduce (fjp/common-pool) (proto/->spliterator data) 1000 (constantly 0) lp lp))) (is (= total (spliterator/split-parallel-reduce clojure.lang.Agent/soloExecutor (proto/->spliterator data) 1000 (constantly 0) lp lp))) (is (= total (hamf-rf/preduce (constantly 0) lp lp (proto/->spliterator data)))) (is (= total (hamf-rf/preduce-reducer (sum-reducer) (proto/->spliterator data)))))) (comment (def data (hamf/range 100000000)) (def vdata (into (hamf/immut-list) data)) (def vvdata (into [] data)) (= (spliterator/sum-fast vdata) (spliterator/psum vdata)) (spliterator/split-to-max-size vdata 10 spliterator/elements) (spliterator/elements (.spliterator vdata 25 35)) (require '[clj-async-profiler.core :as prof]) (crit/quick-bench (spliterator/sum-fast vdata)) (crit/quick-bench (spliterator/sum-fast vvdata)) (prof/serve-ui 8080) (require '[criterium.core :as crit]) (crit/quick-bench (spliterator/psum vdata)) (crit/quick-bench (spliterator/psum vvdata)) ) ================================================ FILE: test/ham_fisted/hash_map_test.clj ================================================ (ns ham-fisted.hash-map-test (:require [clojure.test :refer [deftest is testing are]] [clojure.set :as set] [ham-fisted.api :as api] [ham-fisted.reduce :as hamf-rf] [ham-fisted.function :as hamf-fn] [criterium.core :as crit]) (:import [java.util ArrayList Collections Map Collection] [java.util.function BiFunction BiConsumer] [java.util.concurrent ForkJoinPool Future Callable] [ham_fisted PersistentHashMap PersistentLongHashMap])) (defonce orig api/empty-map) (deftest simple-assoc (let [orig api/empty-map] (is (= 0 (count orig))) (is (= {:a :b} (assoc orig :a :b))) (is (= 1 (count (assoc orig :a :b)))) (is (= {} (-> (assoc orig :a :b) (dissoc :a)))) (is (= 0 (-> (assoc orig :a :b) (dissoc :a) (count))))) (let [nilmap (assoc orig nil :b)] (is (= {nil :b} nilmap)) (is (= nilmap {nil :b})) (is (= {nil :b :a :b} (assoc nilmap :a :b))) (is (= (assoc nilmap :a :b) {nil :b :a :b})) (is (= 1 (count nilmap))) (is (= 1 (count (dissoc nilmap :a)))) (is (= 2 (count (assoc nilmap :a :b)))) (is (= 0 (count (dissoc nilmap nil)))) (is (= #{nil :a} (set (keys (assoc nilmap :a :b))))))) (defonce test-data* (atom {})) (deftest random-assoc-dissoc (let [n-elems 100 n-take (quot n-elems 10) n-left (- n-elems n-take) data (shuffle (range n-elems)) dissoc-vals (take n-take data) data (set data) dissoc-data (set/difference data (set dissoc-vals))] (reset! test-data* {:data data :dissoc-vals dissoc-vals :dissoc-data dissoc-data}) (testing "immutable" (let [alldata (reduce #(assoc %1 %2 %2) orig data) disdata (reduce #(dissoc %1 %2) alldata dissoc-vals)] (is (= n-left (count disdata))) (is (= n-elems (count alldata))) (is (= dissoc-data (set (keys disdata)))) (is (= data (set (keys alldata)))))) (testing "hash table mutable" (let [alldata (api/mut-map (map #(vector % %)) data) disdata (.clone alldata) _ (is (= n-elems (count disdata))) _ (reduce #(do (.remove ^Map %1 %2) %1) disdata dissoc-vals)] (is (= n-left (count disdata))) (is (= n-elems (count alldata))) (is (= dissoc-data (set (keys disdata)))) (is (= data (set (keys alldata)))))) (testing "long hash table mutable" (let [alldata (api/mut-long-map (map #(vector % %)) data) disdata (.clone alldata) _ (is (= n-elems (count disdata))) _ (reduce #(do (.remove ^Map %1 %2) %1) disdata dissoc-vals)] (is (= n-left (count disdata))) (is (= n-elems (count alldata))) (is (= dissoc-data (set (keys disdata)))) (is (= data (set (keys alldata)))))))) (def map-constructors {:mut-map api/mut-map :immut-map api/immut-map :java-hashmap api/java-hashmap}) (deftest hash-map-reduce (doseq [[k constructor] map-constructors] (is (= 2 (reduce (fn [^long acc v] (if (> acc 0) (reduced (inc acc)) (inc acc))) 0 (constructor {:a 1 :b 2 :c 3})))))) (defmacro benchmark-us [op] `(let [bdata# (crit/quick-benchmark ~op nil)] {:mean-μs (* (double (first (:mean bdata#))) 1e6) :variance-μs (* (double (first (:variance bdata#))) 1e6)})) (defn ->collection ^Collection [data] (if (instance? Collection data) data (seq data))) (defn ->array-list ^ArrayList [data] (if (instance? ArrayList data) data (let [retval (ArrayList.)] (.addAll retval (->collection data)) retval))) (defn time-dataset [data constructor] (let [dlist (api/shuffle (api/object-array-list data)) ctime (benchmark-us (constructor dlist)) ^Map ds (constructor dlist) nelems (count dlist) atime (benchmark-us (reduce (fn [acc v] (.get ds v)) nil data)) itime (benchmark-us (reduce #(+ (long %1) (val %2)) 0 ds))] {:construct-μs (:mean-μs ctime) :access-μs (:mean-μs atime) :iterate-μs (:mean-μs itime)})) (defn java-hashmap [data] (reduce (hamf-rf/indexed-accum hm idx v (.put ^Map hm v idx) hm) (api/java-hashmap (count data)) data)) (defn hamf-hashmap [data] (persistent! (reduce (hamf-rf/indexed-accum hm idx v (.put ^Map hm v idx) hm) (api/mut-hashtable-map (count data)) data))) (defn clj-transient [data] (persistent! (reduce (hamf-rf/indexed-accum hm idx v (assoc! hm v idx)) (transient {}) data))) (defn hamf-transient [data] (persistent! (reduce (hamf-rf/indexed-accum hm idx v (assoc! hm v idx)) (transient api/empty-map) data))) (def datastructures [[:java-hashmap java-hashmap] [:hamf-hashmap hamf-hashmap] [:clj-transient clj-transient] [:hamf-transient hamf-transient]]) (defn profile-datastructures [data] (->> (shuffle datastructures) (map (fn [[ds-name ctor]] (-> (time-dataset data ctor) (assoc :ds-name ds-name)))) (sort-by :construct-μs))) (defn- indexed-map ^PersistentHashMap [data] (persistent! (api/mut-map (map-indexed #(vector %2 %1) data)))) (deftest union-test (let [n-elems 100 hn-elems (quot n-elems 2) src-data (take n-elems (distinct (repeatedly #(rand-int 1000000)))) lhs (->array-list (take hn-elems src-data)) rhs (->array-list (drop hn-elems src-data)) hm1 (indexed-map lhs) bfn (reify BiFunction (apply [this a b] (+ a b))) hm2 (indexed-map rhs) un1 (.union hm1 hm2 bfn) un2 (.union un1 hm2 bfn) single-sum (reduce + (range hn-elems))] (is (= (count un1) n-elems)) (is (= (set src-data) (set (keys un1))), (str (set/difference (set src-data) (set (keys un1))) (set/difference (set (keys un1)) (set src-data)))) (is (= (count un2) n-elems)) (is (= (* 2 single-sum) (reduce + (vals un1)))) (is (= (* 3 single-sum) (reduce + (vals un2)))))) (deftest difference-test (let [n-elems 100 hn-elems (quot n-elems 2) src-data (repeatedly n-elems #(rand-int 100000000)) lhs (->array-list (take hn-elems src-data)) rhs (->array-list (drop hn-elems src-data)) llhs (->array-list (take (quot hn-elems 2) src-data)) hm1 (indexed-map lhs) hm2 (indexed-map rhs) df1 (.difference hm1 hm2) df2 (.difference hm1 (indexed-map llhs))] (is (= hn-elems (count df1))) (is (= (quot hn-elems 2) (count df2))) (is (= (set/difference (set lhs) (set llhs)) (set (keys df2)))))) (deftest intersection-test (let [n-elems 100 hn-elems (quot n-elems 2) hhn-elems (quot hn-elems 2) src-data (repeatedly n-elems #(rand-int 100000000)) lhs (->array-list (take hn-elems src-data)) rhs (->array-list (drop hn-elems src-data)) llhs (->array-list (take hhn-elems src-data)) bfn (reify BiFunction (apply [this a b] (+ a b))) hm1 (indexed-map lhs) hhm1 (indexed-map llhs) hm2 (indexed-map rhs) df1 (.intersection hm1 hm2 bfn) df2 (.intersection hm1 hhm1 bfn) hhn-sum (reduce + (range hhn-elems))] (is (= 0 (count df1))) (is (= (quot hn-elems 2) (count df2))) (is (= (set/intersection (set lhs) (set llhs)) (set (keys df2)))) (is (= (* 2 hhn-sum) (reduce + (vals df2)))))) (defn- long-indexed-map ^PersistentLongHashMap [data] (persistent! (api/mut-long-hashtable-map (map-indexed #(vector %2 %1) data)))) (deftest long-union-test (let [n-elems 100 hn-elems (quot n-elems 2) src-data (repeatedly n-elems #(rand-int 100000000)) lhs (->array-list (take hn-elems src-data)) rhs (->array-list (drop hn-elems src-data)) hm1 (long-indexed-map lhs) bfn (reify BiFunction (apply [this a b] (+ a b))) hm2 (long-indexed-map rhs) un1 (.union hm1 hm2 bfn) un2 (.union un1 hm2 bfn) single-sum (reduce + (range hn-elems))] (is (= (count un1) n-elems)) (is (= (set src-data) (set (keys un1)))) (is (= (count un2) n-elems)) (is (= (* 2 single-sum) (reduce + (vals un1)))) (is (= (* 3 single-sum) (reduce + (vals un2)))))) (deftest long-difference-test (let [n-elems 100 hn-elems (quot n-elems 2) src-data (repeatedly n-elems #(rand-int 100000000)) lhs (->array-list (take hn-elems src-data)) rhs (->array-list (drop hn-elems src-data)) llhs (->array-list (take (quot hn-elems 2) src-data)) hm1 (long-indexed-map lhs) hm2 (long-indexed-map rhs) df1 (.difference hm1 hm2) df2 (.difference hm1 (long-indexed-map llhs))] (is (= hn-elems (count df1))) (is (= (quot hn-elems 2) (count df2))) (is (= (set/difference (set lhs) (set llhs)) (set (keys df2)))))) (deftest long-intersection-test (let [n-elems 100 hn-elems (quot n-elems 2) hhn-elems (quot hn-elems 2) src-data (repeatedly n-elems #(rand-int 100000000)) lhs (->array-list (take hn-elems src-data)) rhs (->array-list (drop hn-elems src-data)) llhs (->array-list (take hhn-elems src-data)) bfn (reify BiFunction (apply [this a b] (+ a b))) hm1 (long-indexed-map lhs) hhm1 (long-indexed-map llhs) hm2 (long-indexed-map rhs) df1 (.intersection hm1 hm2 bfn) df2 (.intersection hm1 hhm1 bfn) hhn-sum (reduce + (range hhn-elems))] (is (= 0 (count df1))) (is (= (quot hn-elems 2) (count df2))) (is (= (set/intersection (set lhs) (set llhs)) (set (keys df2)))) (is (= (* 2 hhn-sum) (reduce + (vals df2)))))) (def union-data {:java-hashmap {:construct-fn java-hashmap :merge-fn api/map-union-java-hashmap :reduce-fn api/union-reduce-java-hashmap} ;;jdk-8 - 1000 ;; {:union-disj-μs 20.413211065573773, ;; :union-μs 41.34100116886689, ;; :name :java-hashmap} ;; 100000 ;; {:union-disj-μs 5050.580285714286, ;; :union-μs 8292.417935897436, ;; :name :java-hashmap} :hamf-hashmap {:construct-fn hamf-hashmap :merge-fn api/map-union :reduce-fn #(api/union-reduce-maps %1 %2)} :clj-transient (let [make-merge-fn (fn [bifn] (fn [lhs rhs] (.apply ^BiFunction bifn lhs rhs)))] {:construct-fn clj-transient :merge-fn #(merge-with (make-merge-fn %1) %2 %3) :reduce-fn #(apply merge-with (make-merge-fn %1) %2)})}) ;;jdk-8 - 1000 ;; {:union-disj-μs 11.491328581829329, ;; :union-μs 30.876507346896837, ;; :name :hamf-hashmap} ;; 100000 ;; {:union-disj-μs 2461.248479674797, ;; :union-μs 5106.405916666667, ;; :name :hamf-hashmap} (defn benchmark-union [^long n-elems mapname] (let [{:keys [construct-fn merge-fn]} (union-data mapname) hn-elems (quot n-elems 2) src-data (repeatedly n-elems #(rand-int 100000000)) lhs (->array-list (take hn-elems src-data)) rhs (->array-list (drop hn-elems src-data)) bfn (hamf-fn/->bi-function +) lhs-m (construct-fn lhs) rhs-m (construct-fn rhs) merged-m (merge-fn bfn lhs-m rhs-m)] {:union-disj-μs (:mean-μs (benchmark-us (merge-fn bfn lhs-m rhs-m))) :union-μs (:mean-μs (benchmark-us (merge-fn bfn lhs-m merged-m))) :name mapname})) (defn benchmark-reduce-union [^long n-elems mapname] (let [{:keys [construct-fn reduce-fn]} (union-data mapname) hn-elems (quot n-elems 2) src-data (repeatedly n-elems #(rand-int 100000000)) lhs (->array-list (take hn-elems src-data)) rhs (->array-list (drop hn-elems src-data)) bfn (reify BiFunction (apply [this a b] (unchecked-add (unchecked-long a) (unchecked-long b)))) lhs-m (construct-fn lhs) rhs-m (construct-fn rhs) map-seq (vec (interleave (repeat 10 lhs-m) (repeat 10 rhs-m)))] {:union-μs (:mean-μs (benchmark-us (reduce-fn bfn map-seq))) :name mapname})) (deftest hashcode-equal-hashmap (is (= (.hasheq (api/mut-map [[:a 1] [:b 2]])) (.hasheq {:a 1 :b 2}))) (is (= (.hasheq (api/mut-map [[:a 1] [:b 2] [nil 3]])) (.hasheq {:a 1 :b 2 nil 3}))) (is (= (.hasheq (api/mut-map [[:a 1] [:b 2]])) (.hasheq (api/immut-map {:a 1 :b 2})))) (is (not= (.hasheq (api/mut-map [[:a 1] [:b 3]])) (.hasheq {:a 1 :b 2}))) (is (.equals (api/immut-map [[:a 1] [:b 2]]) {:a 1 :b 2})) (is (.equals (api/immut-map [[:a 1] [:b 2]]) (api/mut-map [[:a 1] [:b 2]]))) (is (not (.equals (api/immut-map [[:a 1] [:b 2]]) nil))) (is (= (.hasheq (api/mut-set [:a :b])) (.hasheq #{:a :b}))) (is (not= (.hasheq (api/mut-set [:a :b :c])) (.hasheq #{:a :b}))) (is (.equals (api/immut-set [:a :b :c]) #{:a :b :c})) (is (.equals (api/immut-set [:a :b :c]) (api/mut-set #{:a :b :c}))) (is (not (.equals (api/immut-set [:a :b :c]) nil)))) ;;Standard tests (defn- pers-map [map-data] (api/immut-map map-data)) (defn- ary-map [map-data] (api/immut-map (api/object-array (flatten (seq map-data))))) (defn test-find-fn [map-fn] (are [x y] (= x y) (find (map-fn {}) :a) nil (find (map-fn {:a 1}) :a) [:a 1] (find (map-fn {:a 1}) :b) nil (find (map-fn {nil 1}) nil) [nil 1] (find (map-fn {:a 1 :b 2}) :a) [:a 1] (find (map-fn {:a 1 :b 2}) :b) [:b 2] (find (map-fn {:a 1 :b 2}) :c) nil (find (map-fn {}) nil) nil (find (map-fn {:a 1}) nil) nil (find (map-fn {:a 1 :b 2}) nil) nil)) (deftest test-find-ary-map (test-find-fn ary-map)) (deftest test-find-pers-map (test-find-fn pers-map)) (defn test-contains? [map-fn] (are [x y] (= x y) (contains? (map-fn {}) :a) false (contains? (map-fn {}) nil) false (contains? (map-fn {:a 1}) :a) true (contains? (map-fn {:a 1}) :b) false (contains? (map-fn {:a 1}) nil) false (contains? (map-fn {nil 1}) nil) true (contains? (map-fn {:a 1 :b 2}) :a) true (contains? (map-fn {:a 1 :b 2}) :b) true (contains? (map-fn {:a 1 :b 2}) :c) false (contains? (map-fn {:a 1 :b 2}) nil) false)) (deftest test-ary-map-contains? (test-contains? ary-map)) (deftest test-pers-map-contains? (test-contains? pers-map)) (defn diff [s1 s2] (seq (reduce disj (set s1) (set s2)))) (defn test-keys [map-fn] (are [x y] (= x y) (keys (map-fn {})) nil (keys (map-fn {:a 1})) '(:a) (keys (map-fn {nil 1})) '(nil) (diff (keys (map-fn {:a 1 :b 2})) '(:a :b)) nil (keys (api/hash-map)) nil (keys (api/hash-map :a 1)) '(:a) (diff (keys (api/hash-map :a 1 :b 2)) '(:a :b)) nil) (let [m (map-fn {:a 1 :b 2}) k (keys m)] (is (= {:hi :there} (meta (with-meta k {:hi :there})))))) (deftest test-ary-map-keys (test-keys ary-map)) (deftest test-pers-map-keys (test-keys pers-map)) (defn test-vals [map-fn] (are [x y] (= x y) (vals (map-fn {})) nil (vals (map-fn {:a 1})) '(1) (vals (map-fn {nil 1})) '(1) (diff (vals (map-fn {:a 1 :b 2})) '(1 2)) nil ; (vals {:a 1 :b 2}) '(1 2) (vals (api/hash-map)) nil (vals (api/hash-map :a 1)) '(1) (diff (vals (api/hash-map :a 1 :b 2)) '(1 2)) nil) ; (vals (hash-map :a 1 :b 2)) '(1 2) (let [m (map-fn {:a 1 :b 2}) v (vals m)] (is (= {:hi :there} (meta (with-meta v {:hi :there})))))) (deftest test-ary-map-vals (test-vals ary-map)) (deftest test-pers-map-vals (test-vals pers-map)) (deftest basic-linked-hashmap (is (= [:a :b :c] (vec (keys (api/linked-hashmap [[:a 1][:b 2][:c 3]]))))) (is (not= [:a :b :c] (vec (keys (api/mut-map [[:a 1][:b 2][:c 3]]))))) (is (= [:a :b :c :d :e] (vec (keys (.union (api/linked-hashmap [[:a 1][:b 2][:c 3]]) (api/linked-hashmap [[:d 4] [:e 5]]) (hamf-fn/bi-function v1 v2 inc)))))) (is (= [:a :b :c :d] (vec (keys (.union (api/linked-hashmap [[:a (api/long-array-list [1 2])][:b (api/long-array-list [3 4])] [:c (api/long-array-list [4 5])]]) (api/linked-hashmap [[:a (api/long-array-list [4 5])] [:d (api/long-array-list [6 7])]]) (hamf-fn/bi-function v1 v2 (.addAll ^java.util.List v1 v2) v1)))))) (is (= (.get (api/linked-hashmap [[(int 1) :a]]) 1) :a))) (comment (do (def hm (HashMap.)) (def orig PersistentHashMap/EMPTY)) (dotimes [idx 100] (.put hm idx idx)) (map iterator-seq (.splitKeys hm 8)) (dotimes [idx 10000] (.put hm idx idx)) (crit/quick-bench (let [hm (HashMap.)] (dotimes [idx 2] (.put hm idx idx)))) ;;34ns ;;jdk-17 - 24ns (crit/quick-bench (let [hm (HashMap.)] (dotimes [idx 2] (.put hm idx idx)) (dotimes [idx 2] (.get hm idx)))) (crit/quick-bench (let [hm (HashMap. PersistentHashMap/equivHashProvider)] (dotimes [idx 2] (.put hm idx idx)))) ;;70ns - twice as fast as clojure's transient for small case. hasheq is the long ;;poll in the tent. ;;jdk-17 - 61ns (crit/quick-bench (let [hm (HashMap. PersistentHashMap/equivHashProvider)] (dotimes [idx 2] (.put hm idx idx)) (dotimes [idx 2] (.get hm idx)))) (crit/quick-bench (let [hm (java.util.HashMap.)] (dotimes [idx 2] (.put hm idx idx)))) ;;36ns ;;jdk-17 31ns (crit/quick-bench (loop [idx 0 hm (transient {})] (if (< idx 2) (recur (unchecked-inc idx) (assoc! hm idx idx)) (persistent! hm)))) ;;113ns ;;jdk-17 - 136ns (crit/quick-bench (let [phm (loop [idx 0 hm (transient {})] (if (< idx 2) (recur (unchecked-inc idx) (assoc! hm idx idx)) (persistent! hm)))] (loop [idx 0] (when (< idx 2) (get phm idx) (recur (unchecked-inc idx)))))) (crit/quick-bench (loop [idx 0 hm (transient PersistentHashMap/EMPTY)] (if (< idx 2) (recur (unchecked-inc idx) (assoc! hm idx idx)) (persistent! hm)))) ;;147ns - arrayhashmap is faster for this case but still a lot slower than ;;using hashmap representation. ;;jdk-17 - 137ns (crit/quick-bench (persistent! (transient PersistentHashMap/EMPTY))) (crit/quick-bench (let [hm (HashMap.)] (dotimes [idx 1000] (.put hm idx idx)))) ;;21us ;;jdk-17 - 21us (crit/quick-bench (let [hm (HashMap.)] (dotimes [idx 1000] (.put hm idx idx)) (dotimes [idx 1000] (.get hm idx)))) ;;35us (crit/quick-bench (let [hm (HashMap. PersistentHashMap/equivHashProvider)] (dotimes [idx 1000] (.put hm idx idx)))) ;;41us ;;jdk-17 - 39us (crit/quick-bench (let [hm (HashMap. PersistentHashMap/equivHashProvider)] (dotimes [idx 1000] (.put hm idx idx)) (dotimes [idx 1000] (.get hm idx)))) ;; 75us (crit/quick-bench (let [hm (java.util.HashMap.)] (dotimes [idx 1000] (.put hm idx idx)))) ;;17us ;;jdk-17 - 21us (crit/quick-bench (let [hm (java.util.HashMap.)] (dotimes [idx 1000] (.put hm idx idx)) (dotimes [idx 1000] (.get hm idx)))) ;;20us (crit/quick-bench (loop [idx 0 hm (transient {})] (if (< idx 1000) (recur (unchecked-inc idx) (assoc! hm idx idx)) (persistent! hm)))) ;;112us ;;123us (crit/quick-bench (let [phm (loop [idx 0 hm (transient {})] (if (< idx 1000) (recur (unchecked-inc idx) (assoc! hm idx idx)) (persistent! hm)))] (loop [idx 0] (when (< idx 1000) (get phm idx) (recur (unchecked-inc idx)))))) ;;168us (crit/quick-bench (loop [idx 0 hm (transient PersistentHashMap/EMPTY)] (if (< idx 1000) (recur (unchecked-inc idx) (assoc! hm idx idx)) (persistent! hm)))) ;;47us ;;jdk-17 50us (crit/quick-bench (let [phm (loop [idx 0 hm (transient PersistentHashMap/EMPTY)] (if (< idx 1000) (recur (unchecked-inc idx) (assoc! hm idx idx)) (persistent! hm)))] (loop [idx 0] (when (< idx 1000) (get phm idx) (recur (unchecked-inc idx)))))) ;; 83us ;;Useful to profile small things sometimes. (dotimes [idx 100000000] (loop [idx 0 hm (transient {})] (if (< idx 20) (recur (unchecked-inc idx) (assoc! hm idx idx)) (persistent! hm)))) (def hm (let [hm (HashMap.) _ (dotimes [idx 60] (.put hm idx idx))] hm)) (let [nhm (HashBase.)] (.keyspaceSplit hm 146 291 false nhm) (.printNodes nhm)) (let [hm (HashMap.) _ (dotimes [idx 60] (.put hm idx idx))] (map iterator-seq (.splitKeys hm 7))) (def small-int-profile (profile-datastructures (repeatedly 2 #(rand-int 100000)))) ;;jdk-8 ;;({:construct-μs 0.04298386356008815, ;; :access-μs 0.040345482495456206, ;; :iterate-μs 0.05087595836156305, ;; :ds-name :hamf-hashmap} ;; {:construct-μs 0.044846695459721565, ;; :access-μs 0.02446975706544951, ;; :iterate-μs 0.04827733417909167, ;; :ds-name :java-hashmap} ;; {:construct-μs 0.10289826952319454, ;; :access-μs 0.08631483123589048, ;; :iterate-μs 0.05130890786182448, ;; :ds-name :hamf-equiv-hashmap} ;; {:construct-μs 0.14209144548280342, ;; :access-μs 0.08244324142027323, ;; :iterate-μs 0.04715165522789527, ;; :ds-name :clj-transient} ;; {:construct-μs 0.16351660268608148, ;; :access-μs 0.09678529907937389, ;; :iterate-μs 0.05798288225070914, ;; :ds-name :hamf-transient}) (def int-profile (profile-datastructures (repeatedly 1000 #(rand-int 100000)))) ;; jdk-8 ;; ({:construct-μs 23.099907694058732, ;; :access-μs 11.601162415160529, ;; :iterate-μs 8.334938366460666, ;; :ds-name :java-hashmap} ;; {:construct-μs 34.1529201495073, ;; :access-μs 12.681086848216172, ;; :iterate-μs 14.304917063893303, ;; :ds-name :hamf-hashmap} ;; {:construct-μs 63.69087151015229, ;; :access-μs 48.95698405726371, ;; :iterate-μs 17.277700236284005, ;; :ds-name :hamf-equiv-hashmap} ;; {:construct-μs 66.86508244206775, ;; :access-μs 50.38417851142474, ;; :iterate-μs 21.130427287409592, ;; :ds-name :hamf-transient} ;; {:construct-μs 110.31809377289377, ;; :access-μs 44.23049499782514, ;; :iterate-μs 35.53727620563613, ;; :ds-name :clj-transient}) ;; jdk-17 ;; ({:construct-μs 31.915435858163935, ;; :access-μs 12.193707814495532, ;; :iterate-μs 7.316710778209167, ;; :ds-name :java-hashmap} ;; {:construct-μs 45.07551537356322, ;; :access-μs 21.033255470685386, ;; :iterate-μs 22.979641088509506, ;; :ds-name :hamf-hashmap} ;; {:construct-μs 73.53580794223828, ;; :access-μs 49.47928398962892, ;; :iterate-μs 22.8927910698984, ;; :ds-name :hamf-equiv-hashmap} ;; {:construct-μs 77.98202366716494, ;; :access-μs 52.42915830218901, ;; :iterate-μs 20.852072754333744, ;; :ds-name :hamf-transient} ;; {:construct-μs 134.15703174603175, ;; :access-μs 47.73247937254903, ;; :iterate-μs 37.22873836295491, ;; :ds-name :clj-transient}) (def big-int-profile (profile-datastructures (repeatedly 100000 #(rand-int 100000)))) ;;jdk-8 ;; ({:construct-μs 4818.585603174603, ;; :access-μs 3115.862890625, ;; :iterate-μs 903.9770223214286, ;; :ds-name :java-hashmap} ;; {:construct-μs 8781.270125000001, ;; :access-μs 8000.021961538461, ;; :iterate-μs 1914.1505256410255, ;; :ds-name :hamf-hashmap} ;; {:construct-μs 12504.3571875, ;; :access-μs 11433.55188888889, ;; :iterate-μs 2047.054111111111, ;; :ds-name :hamf-equiv-hashmap} ;; {:construct-μs 12913.919562500001, ;; :access-μs 12543.427979166667, ;; :iterate-μs 2056.0890714285715, ;; :ds-name :hamf-transient} ;; {:construct-μs 13137.229833333335, ;; :access-μs 10124.450916666668, ;; :iterate-μs 3152.3616718750004, ;; :ds-name :clj-transient}) (def double-profile (profile-datastructures (repeatedly 1000 #(rand 100000)))) (def str-profile (profile-datastructures (map str (repeatedly 1000 #(rand-int 100000))))) ;;jdk-8 ;;({:construct-μs 23.930453143122246, ;; :access-μs 13.627138225718937, ;; :ds-name :java-hashmap} ;; {:construct-μs 43.07852368684701, ;; :access-μs 32.3816567357513, ;; :ds-name :hamf-hashmap} ;; {:construct-μs 68.1719815066939, ;; :access-μs 58.411238927738935, ;; :ds-name :hamf-equiv-hashmap} ;; {:construct-μs 73.45427547307133, ;; :access-μs 66.72418882978724, ;; :ds-name :hamf-transient} ;; {:construct-μs 128.5458824027073, ;; :access-μs 53.36163036127426, ;; :ds-name :clj-transient}) (def kw-profile (profile-datastructures (map (comp keyword str) (repeatedly 1000 #(rand-int 100000))))) ;;jdk-8 ;; ({:construct-μs 37.88887029341394, ;; :access-μs 18.842061234058516, ;; :iterate-μs 12.900347590258127, ;; :ds-name :java-hashmap} ;; {:construct-μs 40.78816281271129, ;; :access-μs 31.311148279404215, ;; :iterate-μs 26.028699949122363, ;; :ds-name :hamf-equiv-hashmap} ;; {:construct-μs 44.76396859578287, ;; :access-μs 37.260138704523115, ;; :iterate-μs 25.54532113194623, ;; :ds-name :hamf-transient} ;; {:construct-μs 48.99040834149526, ;; :access-μs 36.94391981651376, ;; :iterate-μs 26.269819758868937, ;; :ds-name :hamf-hashmap} ;; {:construct-μs 98.01407147466925, ;; :access-μs 33.85242917783735, ;; :iterate-μs 37.85340298225746, ;; :ds-name :clj-transient}) ;; joinr test (def db {:name {"bilbo" "baggins"} :age {"bilbo" 111} :location {"bilbo" "Shire"}}) (defn mutable2d [m] (let [hm (api/mut-hashtable-map nil m)] (reduce-kv (fn [acc k v] (assoc! acc k (api/mut-hashtable-map))) hm hm))) (defn eq-mutable2d [m] (let [hm (api/mut-hashtable-map nil m)] (reduce-kv (fn [acc k v] (assoc! acc k (api/mut-hashtable-map nil {:hash-provider api/equal-hash-provider} v))) hm hm))) (def mdb (mutable2d db)) (def eq-mdb (eq-mutable2d db)) (def hdb (reduce-kv (fn [^java.util.Map acc k v] (doto acc (.put k (reduce-kv (fn [^java.util.Map inner k v] (doto inner (.put k v))) (java.util.HashMap.) v)) (java.util.HashMap.))) (java.util.HashMap.) db)) (let [inner (.get mdb :name)] (crit/quick-bench (.get ^Map inner "bilbo"))) (let [inner (.get hdb :name)] (crit/quick-bench (.get ^Map inner "bilbo"))) (let [inner (.get db :name)] (crit/quick-bench (.get ^Map inner "bilbo"))) (let [inner (.get eq-mdb :name)] (println (type inner)) (crit/quick-bench (.get ^Map inner "bilbo"))) (do (def ht (ham_fisted.HashTable. api/equal-hash-provider)) (.put ht "bilbo" "baggins")) (crit/quick-bench (.get ^ham_fisted.HashTable ht "bilbo"))) ================================================ FILE: test/ham_fisted/hlet_test.clj ================================================ (ns ham-fisted.hlet-test (:require [ham-fisted.api :as hamf] [ham-fisted.primitive-invoke :as pi] [ham-fisted.hlet :as h] [clojure.test :refer [deftest is]])) (set! *unchecked-math* :warn-on-boxed) (deftest basic-hlet (h/let [[x y] (dbls (if true (hamf/double-array [1 2]) (hamf/double-array [3 4]))) look-fn (pi/->ddd (fn ^double [^double a ^double b] (+ a b))) ;;test singular case sx (dbls (rand-int 100))] (is (= 3.0 (pi/ddd look-fn x y))))) (comment (require '[clj-java-decompiler.core :refer [disassemble]]) (disassemble (h/let [[x y] (dbls (if true (hamf/double-array [1 2]) (hamf/double-array [3 4]))) look-fn (pi/->ddd (fn ^double [^double a ^double b] (+ a b)))] (look-fn x y) (pi/ddd look-fn x y))) ) ================================================ FILE: test/ham_fisted/parallel_test.clj ================================================ (ns ham-fisted.parallel-test (:require [ham-fisted.api :as api] [ham-fisted.reduce :as hamf-rf] [ham-fisted.mut-map :as hamf-map] [ham-fisted.lazy-noncaching :as lznc] [clojure.test :refer [deftest is]]) (:import [ham_fisted Sum])) (deftest spliterator-preduce (let [data (api/immut-map (lznc/map #(api/vector % %) (range 10000))) sum-data (fn [opts m] (-> (hamf-rf/preduce #(Sum.) hamf-rf/double-consumer-accumulator (fn [^Sum l ^Sum r] (.merge l r) l) opts m) (deref) :sum (long))) opts {:min-n 5 :ordered? true}] (is (= 49995000 (sum-data opts (api/keys data)))) (is (= 49995000 (sum-data opts (api/vals data)))) (is (= 49995000 (sum-data (assoc opts :ordered? false) (api/keys data)))) (is (= 49995000 (sum-data (assoc opts :ordered? false) (api/vals data)))) (is (= 49995000 (sum-data (assoc opts :ordered? false) (hamf-map/keyset data)))) (is (= 49995000 (sum-data (assoc opts :ordered? false) (hamf-map/values data)))) (let [small (api/immut-map (lznc/map #(api/vector % %) (range 8)))] (is (= 28 (sum-data opts (api/keys small)))) (is (= 28 (sum-data opts (api/vals small))))))) ;;There was an issue with small upmaps hanging (deftest small-upmap (is (= (api/range 5) (api/sort (api/upmap #(+ % 1) (range -1 4)))))) (deftest small-pmap (is (= (api/range 10) (api/pmap #(+ % 1) (range -1 9))))) (deftest small-pmap-io (is (= [] (api/pmap-io 16 identity [])))) ;;This caused a hang before wrapping queue io with boxes. (deftest upmap-nil-return (is (every? nil? (api/upgroups 10000 (fn [^long sidx ^long eidx] nil))))) (deftest custom-pmap-pool (let [p (java.util.concurrent.Executors/newCachedThreadPool)] (is (= 5050.0 (api/sum (api/pmap-opts {:n-lookahead 2 :pool p :min-n 0} #(do #_(println (.getName (Thread/currentThread))) (+ % 1)) (range 100))))))) ================================================ FILE: test/ham_fisted/persistent_vector_test.clj ================================================ (ns ham-fisted.persistent-vector-test (:require [clojure.test :refer [deftest is testing are] :as test] [clj-memory-meter.core :as mm] [ham-fisted.api :as hamf] [ham-fisted.lazy-noncaching :as lznc]) (:import [ham_fisted MutList ImmutList MutTreeList TreeList] [java.util List ArrayList Collections])) (deftest vec-nil (is (= (clojure.core/vec nil) (hamf/vec nil)))) (def vec-fns {:api-vec {:convert-fn identity :vec-fn hamf/vec} :api-vec-sublist {:convert-fn identity :vec-fn (fn ([] (hamf/subvec (hamf/vec [1 2 4]) 3)) ([data] (hamf/subvec (hamf/vec (lznc/concat [1 2 4] data)) 3)))} :immut-vec {:convert-fn identity :vec-fn (comp persistent! hamf/mut-list)} :immut-vec-sublist {:convert-fn identity :vec-fn (fn ([] (hamf/subvec (persistent! (hamf/mut-list (range 100))) 100)) ([data] (hamf/subvec (persistent! (hamf/mut-list (lznc/concat (range 100) data))) 100)))} :api-mut-list {:convert-fn identity :vec-fn hamf/mut-list} :api-mut-sublist {:convert-fn identity :vec-fn (fn ([] (hamf/subvec (hamf/mut-list (range 100)) 100)) ([data] (hamf/subvec (hamf/mut-list (lznc/concat (range 100) data)) 100)))} :byte-vec {:convert-fn unchecked-byte :vec-fn (comp hamf/->random-access hamf/byte-array)} :byte-vec-list {:convert-fn unchecked-byte :vec-fn (comp hamf/->random-access hamf/byte-array-list)} :short-vec {:convert-fn identity :vec-fn (comp hamf/->random-access hamf/short-array)} :short-vec-list {:convert-fn identity :vec-fn (comp hamf/->random-access hamf/short-array-list)} :char-vec {:convert-fn char :vec-fn (fn ([] (hamf/->random-access (hamf/char-array))) ([data] (hamf/->random-access (hamf/char-array (hamf/mapv char data)))))} :char-vec-list {:convert-fn char :vec-fn (fn ([] (hamf/->random-access (hamf/char-array-list))) ([data] (hamf/->random-access (hamf/char-array-list (hamf/mapv char data)))))} :int-vec {:convert-fn identity :vec-fn (fn ([] (hamf/ivec)) ([data] (hamf/ivec data)))} :int-list-vec {:convert-fn identity :vec-fn hamf/int-array-list} :long-vec {:convert-fn identity :vec-fn (fn ([] (hamf/lvec)) ([data] (hamf/lvec data)))} :long-list-vec {:convert-fn identity :vec-fn hamf/long-array-list} :float-vec {:convert-fn float :vec-fn (fn ([] (hamf/fvec)) ([data] (hamf/fvec data)))} :float-list-vec {:convert-fn float :vec-fn (comp hamf/->random-access hamf/float-array-list)} :double-vec {:convert-fn double :vec-fn (fn ([] (hamf/dvec)) ([data] (hamf/dvec data)))} :double-list-vec {:convert-fn double :vec-fn hamf/double-array-list} }) (defn test-reversed-vec-fn [k {:keys [convert-fn vec-fn]}] (let [r (range 6) v (vec-fn r) reversed (.rseq v)] (testing "RSeq methods" (is (= (hamf/mapv convert-fn [5 4 3 2 1 0]) reversed) (str k " " v " " reversed)) (is (= (convert-fn 5) (.first reversed)) k) (is (= (hamf/mapv convert-fn [4 3 2 1 0]) (.next reversed)) k) (is (= (hamf/mapv convert-fn [3 2 1 0]) (.. reversed next next)) k) (is (= 6 (.count reversed)) k)) (testing "clojure calling through" (is (= (convert-fn 5) (first reversed)) k) (is (= (convert-fn 5) (nth reversed 0))) k) (testing "empty reverses to nil" (is (nil? (.. v empty rseq)))))) (deftest test-reversed-vec (doseq [[k v] vec-fns] (test-reversed-vec-fn k v))) (defn all-add ([convert-fn a b] (convert-fn (+ (long a) (long b)))) ([convert-fn a b c] (convert-fn (+ (long a) (long b) (long c))))) (deftest test-subvector-reduce (doseq [[k v] vec-fns] (let [{:keys [convert-fn vec-fn]} v] (is (= (convert-fn 60) (let [prim-vec (vec-fn (lznc/map convert-fn (hamf/range 1000)))] (convert-fn (reduce (partial all-add convert-fn) (hamf/subvec prim-vec 10 15))))) k) (is (= (convert-fn 60) (let [prim-vec (hamf/vec (range 1000))] (reduce (partial all-add convert-fn) (hamf/subvec prim-vec 10 15)))) k)))) (deftest test-vec-associative (doseq [[k v] vec-fns] (let [{:keys [convert-fn vec-fn]} v] (let [empty-v (vec-fn) v (vec-fn (range 1 6))] (testing "Associative.containsKey" (are [x] (.containsKey v x) 0 1 2 3 4) (are [x] (not (.containsKey v x)) -1 -100 nil [] "" #"" #{} 5 100) (are [x] (not (.containsKey empty-v x)) 0 1)) (testing "contains?" (are [x] (contains? v x) 0 2 4) (are [x] (not (contains? v x)) -1 -100 nil "" 5 100) (are [x] (not (contains? empty-v x)) 0 1)) (testing "Associative.entryAt" (are [idx val] (= (clojure.lang.MapEntry. idx (convert-fn val)) (.entryAt v idx)) 0 1 2 3 4 5) (are [idx] (nil? (.entryAt v idx)) -5 -1 5 10 nil "") (are [idx] (nil? (.entryAt empty-v idx)) 0 1)))))) (defn =vec [expected v] (and (vector? v) (= expected v))) (deftest test-mapv (are [r c1] (=vec r (hamf/mapv + c1)) [1 2 3] [1 2 3]) (are [r c1 c2] (=vec r (hamf/mapv + c1 c2)) [2 3 4] [1 2 3] (repeat 1)) (are [r c1 c2 c3] (=vec r (hamf/mapv + c1 c2 c3)) [3 4 5] [1 2 3] (repeat 1) (repeat 1)) (are [r c1 c2 c3 c4] (=vec r (hamf/mapv + c1 c2 c3 c4)) [4 5 6] [1 2 3] [1 1 1] [1 1 1] [1 1 1])) (deftest test-filterv (are [r c1] (=vec r (hamf/filterv even? c1)) (hamf/vector) (hamf/vector 1 3 5) (hamf/vector 2 4) (hamf/vector 1 2 3 4 5))) (deftest test-subvec (doseq [[k v] vec-fns] (let [{:keys [convert-fn vec-fn]} v] (let [v1 (vec-fn (hamf/range 100)) v2 (hamf/subvec v1 50 57)] ;;nth, IFn interfaces allow negative (from the end) indexing (is (thrown? IndexOutOfBoundsException (.get v2 -1))) (is (thrown? IndexOutOfBoundsException (v2 7)) k) (is (= (v1 50) (v2 0))) (is (= (v1 56) (v2 6)))) (let [v1 (vec-fn (hamf/range 10)) v2 (hamf/subvec v1 2 7)] ;;nth, IFn interfaces allow negative (from the end) indexing (is (thrown? IndexOutOfBoundsException (.get v2 -1))) (is (thrown? IndexOutOfBoundsException (v2 7)) k) (is (= (v1 2) (v2 0))) (is (= (v1 6) (v2 4))))))) (deftest test-vec (is (= [1 2] (hamf/vec (first {1 2})))) (is (= [0 1 2 3] (hamf/vec [0 1 2 3]))) (is (= [0 1 2 3] (hamf/vec (list 0 1 2 3)))) (is (= [[1 2] [3 4]] (vec (sorted-map 1 2 3 4)))) (is (= [0 1 2 3] (hamf/vec (range 4)))) (is (= [\a \b \c \d] (hamf/vec "abcd"))) (is (= [0 1 2 3] (hamf/vec (object-array (range 4))))) (is (= [1 2 3 4] (hamf/vec (eduction (map inc) (range 4))))) (is (= [0 1 2 3] (hamf/vec (reify clojure.lang.IReduceInit (reduce [_ f start] (reduce f start (range 4)))))))) (deftest test-vector-eqv-to-non-counted-types (is (not= (range) (hamf/vector 0 1 2))) (is (not= (hamf/vector 0 1 2) (range))) (is (not= (hamf/vec (range 100)) (range))) (is (= (hamf/vec (range 100)) (hamf/range 100))) (is (= (hamf/vector 0 1 2) (take 3 (range)))) (is (= (hamf/vector 0 1 2) (new java.util.ArrayList [0 1 2]))) (is (not= (hamf/vector 1 2) (take 1 (cycle [1 2])))) (is (= (hamf/vector 1 2 3 nil 4 5 6 nil) (eduction cat [[1 2 3 nil] [4 5 6 nil]])))) (deftest test-reduce-kv-vectors (is (= 25 (reduce-kv + 10 (hamf/vector 2 4 6)))) (is (= 25 (reduce-kv + 10 (hamf/subvec (hamf/vector 0 2 4 6) 1)))) (doseq [[k v] vec-fns] (when-not (#{:byte-vec :byte-vec-list} k) (let [{:keys [convert-fn vec-fn]} v] (is (= 9811 (long (reduce-kv (partial all-add convert-fn) 10 (vec-fn (hamf/range 1 100))))) k) (is (= 9811 (long (reduce-kv (partial all-add convert-fn) 10 (hamf/subvec (vec-fn (hamf/range 100)) 1)))) k))))) (deftest test-reduced?-in-container (doseq [[k v] vec-fns] (when-not (#{:byte-vec :byte-vec-list} k) (let [{:keys [convert-fn vec-fn]} v] (is (= 4 (reduce (fn [acc v] (let [v (long v)] (if (< v 4) v (reduced v)))) 0 (vec-fn (hamf/range 1 100)))) k))))) (deftest test-reduce-kv-array (is (= 25 (reduce-kv + 10 (hamf/->random-access (hamf/int-array [2 4 6]))))) (is (= 25 (reduce-kv + 10 (hamf/subvec (hamf/->random-access (hamf/int-array [0 2 4 6])) 1))))) (deftest test-reduce-vectors (is (= 22 (reduce + 10 (hamf/vector 2 4 6)))) (is (= 22 (reduce + 10 (hamf/subvec (hamf/vector 0 2 4 6) 1)))) (is (= 4960 (reduce + 10 (hamf/vec (hamf/range 1 100))))) (is (= 4960 (reduce + 10 (hamf/subvec (hamf/vec (hamf/range 100)) 1))))) (deftest test-reduce-arrays (is (= 22 (reduce + 10 (hamf/int-array [2 4 6])))) (is (= 22 (reduce + 10 (hamf/subvec (hamf/int-array [0 2 4 6]) 1))))) (deftest concatv-special-cases (is (= (reduce + 0 (hamf/concatv [] (list 1 2 3) nil nil (clojure.core/vector 1 2 3 4 5) (hamf/array-list [1 2 3 4]) (hamf/vec (hamf/range 50)))) (reduce + 0 (concat [] (list 1 2 3) nil nil (clojure.core/vector 1 2 3 4 5) (hamf/array-list [1 2 3 4]) (hamf/vec (hamf/range 50)))))) (is (= (reduce + 0 (lznc/concat [] (list 1 2 3) nil nil (clojure.core/vector 1 2 3 4 5) (hamf/array-list [1 2 3 4]) (hamf/vec (hamf/range 50)))) (reduce + 0 (concat [] (list 1 2 3) nil nil (clojure.core/vector 1 2 3 4 5) (hamf/array-list [1 2 3 4]) (hamf/vec (hamf/range 50)))))) (is (= (hamf/concatv [] (list 1 2 3) nil nil (clojure.core/vector 1 2 3 4 5) (hamf/array-list [1 2 3 4]) (hamf/vec (hamf/range 50))) (concat [] (list 1 2 3) nil nil (clojure.core/vector 1 2 3 4 5) (hamf/array-list [1 2 3 4]) (hamf/vec (hamf/range 50))))) (is (= (lznc/concat [] (list 1 2 3) nil nil (clojure.core/vector 1 2 3 4 5) (hamf/array-list [1 2 3 4]) (hamf/vec (hamf/range 50))) (concat [] (list 1 2 3) nil nil (clojure.core/vector 1 2 3 4 5) (hamf/array-list [1 2 3 4]) (hamf/vec (hamf/range 50))))) (is (= (hamf/vec (hamf/concata [] (list 1 2 3) nil nil (clojure.core/vector 1 2 3 4 5) (hamf/object-array-list [1 2 3 4]) (hamf/vec (hamf/range 50)))) (concat [] (list 1 2 3) nil nil (clojure.core/vector 1 2 3 4 5) (hamf/object-array-list [1 2 3 4]) (hamf/vec (hamf/range 50)))))) (deftest tree-list-creation (let [data (hamf/object-array (hamf/range 100)) vdata (clojure.core/vec data)] (is (= vdata (ham_fisted.TreeList/create true nil data))) (is (= vdata (ham_fisted.MutTreeList/create true nil data))))) (deftest binary-search (let [data (hamf/shuffle (hamf/range 100))] (doseq [[k {:keys [convert-fn vec-fn]}] vec-fns] (let [init-data (vec-fn data) ;;make sure sort always works newvdata (hamf/sort init-data) ;;transform back vdata (vec-fn newvdata) subv (hamf/subvec vdata 50)] ;;Ensure that non-accelerated sort results are identical to ;;accelerated sort results (is (= vdata (hamf/sort compare init-data)) k) (is (= 50 (hamf/binary-search vdata (convert-fn 50))) k) (when-not (#{:char-vec :char-vec-list} k) (is (= 51 (hamf/binary-search vdata 50.1 compare)) k)) (when-not (#{:char-vec :char-vec-list} k) (is (= 0 (hamf/binary-search vdata -1)) k)) (is (= 100 (hamf/binary-search vdata (convert-fn 120))) k) (is (= 0 (hamf/binary-search subv (convert-fn 50))) k) (when-not (#{:char-vec :char-vec-list} k) (is (= 1 (hamf/binary-search subv 50.1 compare)) k)) (when-not (#{:char-vec :char-vec-list} k) (is (= 0 (hamf/binary-search subv -1)) k)) (is (= 50 (hamf/binary-search subv (convert-fn 120))) k))))) (deftest boolean-arrays (is (== 2.0 (hamf/sum (hamf/boolean-array [true false true false])))) (is (= [true false true false] (hamf/->random-access (hamf/boolean-array [1 0 1 0]))))) (deftest float-regression (is (= 2 (count (hamf/float-array (lznc/filter (fn [^double v] (not (Double/isNaN v))) [1 ##NaN 2])))))) (deftest sublists-are-smaller-test (let [t (into (TreeList.) (range (* 1000 1000)))] #_(println (* 2 (mm/measure (hamf/subvec t 100000 200000) :bytes true)) (* 1 (mm/measure t :bytes true))) (is (< (* 2 (mm/measure (hamf/subvec t 100000 200000) :bytes true)) (* 1 (mm/measure t :bytes true)))))) (comment (def vec-fn (get-in vec-fns [:api-mut-sublist :vec-fn])) (def data (hamf/shuffle (hamf/range 100))) (def vv (vec-fn data)) (def m (doto (MutList.) (.addAll (range 36)))) (def m ImmutList/EMPTY) (def data (doto (ArrayList.) (.addAll (range 1000)))) (crit/quick-bench (doto (ArrayList.) (.addAll data))) ;; Evaluation count : 907674 in 6 samples of 151279 calls. ;; Execution time mean : 662.299153 ns ;; Execution time std-deviation : 4.369292 ns ;; Execution time lower quantile : 653.753118 ns ( 2.5%) ;; Execution time upper quantile : 665.603474 ns (97.5%) ;; Overhead used : 1.966980 ns (crit/quick-bench (doto (MutList.) (.addAll data))) ;; Evaluation count : 395256 in 6 samples of 65876 calls. ;; Execution time mean : 1.517687 µs ;; Execution time std-deviation : 0.809702 ns ;; Execution time lower quantile : 1.516407 µs ( 2.5%) ;; Execution time upper quantile : 1.518463 µs (97.5%) ;; Overhead used : 1.969574 ns (crit/quick-bench (hamf/mut-list data)) ;;same (def vdata (vec data)) (crit/quick-bench (doto (MutList.) (.addAll vdata))) (def m (hamf/mut-list data)) (defn index-test [^List m] (let [nelems (.size m)] (crit/quick-bench (dotimes [idx nelems] (.get m idx))))) (index-test data) ;; 770ns (index-test m) ;; 1.7us (index-test vdata) ;;5.3us (crit/quick-bench (.toArray data)) ;;1.08us (crit/quick-bench (.toArray m)) ;;2.4us (crit/quick-bench (.toArray vdata)) ;;6.2us (def adata (.toArray data)) (crit/quick-bench (hamf/mut-list adata)) ;; Execution time mean : 775.067887 ns ;; Execution time std-deviation : 2.413493 ns ;; Execution time lower quantile : 770.746577 ns ( 2.5%) ;; Execution time upper quantile : 777.223862 ns (97.5%) ;; Overhead used : 1.965715 ns (crit/quick-bench (hamf/into [] data)) ;; Evaluation count : 31608 in 6 samples of 5268 calls. ;; Execution time mean : 19.045789 µs ;; Execution time std-deviation : 28.268048 ns ;; Execution time lower quantile : 19.001188 µs ( 2.5%) ;; Execution time upper quantile : 19.073174 µs (97.5%) ;; Overhead used : 1.965715 ns (crit/quick-bench (hamf/into hamf/empty-vec adata)) ;; Evaluation count : 31608 in 6 samples of 5268 calls. ;; Execution time mean : 19.045789 µs ;; Execution time std-deviation : 28.268048 ns ;; Execution time lower quantile : 19.001188 µs ( 2.5%) ;; Execution time upper quantile : 19.073174 µs (97.5%) ;; Overhead used : 1.965715 ns ) ================================================ FILE: test/ham_fisted/test_setup.clj ================================================ (ns ham-fisted.test-setup "Things we need to do in order to run tests with a smile" (:require [kaocha.hierarchy :as hierarchy])) (hierarchy/derive! ::ignore :kaocha/known-key) (defn defuse-zero-assertions "Don't fail the test suite if we hide an `is` within a `doseq`. See also https://cljdoc.org/d/lambdaisland/kaocha/1.80.1274/doc/-clojure-test-assertion-extensions#detecting-missing-assertions" [event] (if (= (:type event) :kaocha.type.var/zero-assertions) (assoc event :type ::ignore) event)) ================================================ FILE: test/ham_fisted/vec_like_test.clj ================================================ (ns ham-fisted.vec-like-test (:require [ham-fisted.api :as hamf] [clojure.test :refer [deftest is] :as test]) (:import [ham_fisted TreeList IMutList Iter MutTreeList] [java.util List])) (defn sublist-tumbler [^List data] (let [ne (count data) idx0 (rand-int ne) idx1 (rand-int ne) eidx (max idx0 idx1) sidx (min idx0 idx1) ss (.subList data sidx eidx) answer (into [] (->> (drop sidx data) (take (- eidx sidx)))) result (try (into [] ss) (catch Exception e (println e)))] (when-not (= answer result) (throw (ex-info "sublist failed:" {:data data :answer answer :result result :sidx sidx :eidx eidx}))))) (deftest sublist-test (let [tr (reduce conj (TreeList.) (range 1000000))] (dotimes [idx 50] (sublist-tumbler tr)) (is (= (count tr) 1000000)))) (defn add-all-reducible ^IMutList [^IMutList l data] (.addAllReducible l data) l) (defn ->iter [data] (when data (if (instance? Iter data) data (Iter/fromIterator (.iterator ^Iterable data))))) (defn cons-all ^TreeList [^TreeList l data] (.consAll l (->iter data))) (deftype RangeIter [^long n ^{:unsynchronized-mutable true :tag long} idx] Iter (get [this] (Long/valueOf idx)) (next [this] (set! idx (inc idx)) (when (< idx n) this))) (comment (def tr (reduce conj (TreeList.) (range 35))) (require '[criterium.core :as crit]) (def rr (into [] (range 1000000))) (crit/quick-bench (reduce conj (ham_fisted.TreeList.) rr)) (crit/quick-bench (reduce conj [] rr)) (crit/quick-bench (hamf/object-array (into [] rr))) (crit/quick-bench (add-all-reducible (hamf/object-array-list) rr)) (crit/quick-bench (hamf/object-array (add-all-reducible (ham_fisted.MutTreeList.) rr))) (crit/quick-bench (cons-all (ham_fisted.TreeList.) rr)) (crit/quick-bench (cons-all (ham_fisted.TreeList.) (RangeIter. (count rr) 0))) (crit/quick-bench (add-all-reducible (ham_fisted.BatchedList.) rr)) (def tr (reduce conj (ham_fisted.TreeList.) rr)) (def pv (reduce conj [] rr)) (def tr (reduce conj (ham_fisted.TreeList.) (range 32768))) (defn verify-structure [^TreeList tt] ) (do (def tr (reduce conj (TreeList.) (range 1000000))) (when-let [ee (try (dotimes [idx 50] (sublist-tumbler tr)) (catch Throwable e e))] (let [] (def exd (ex-data ee)) (def sidx 763476) (def eidx 877568) (def tt (.subList tr sidx eidx)) (def arrays (vec (iterator-seq (.arrayIterator (.data tt) (.offset tt) (+ (.offset tt) (.size tt)))))))) ) (do (require '[clj-async-profiler.core :as prof]) (prof/profile {:interval 10000} (dotimes [idx 50] (add-all-reducible (ham_fisted.BatchedList.) rr))) (prof/serve-ui 8080)) ) ================================================ FILE: tests.edn ================================================ #kaocha/v1 {:capture-output? false :kaocha/fail-fast? false :plugins [:kaocha.plugin/profiling :kaocha.plugin/print-invocations :kaocha.plugin/junit-xml :kaocha.plugin/cloverage :kaocha.plugin/hooks :preloads] :kaocha.plugin.junit-xml/target-file "target/junit.xml" :kaocha.plugin.junit-xml/add-location-metadata? true :cloverage/opts {:ns-exclude-regex [] :text? true :lcov? true :high-watermark 80 :fail-threshold 0 :output "target/coverage" :low-watermark 50 :summary? true :coveralls? false :emma-xml? false :html? true :nop? false :codecov? true} :kaocha.hooks/pre-report [ham-fisted.test-setup/defuse-zero-assertions] :kaocha.plugin.preloads/ns-names [ham-fisted.test-setup] :tests [{:id :unit :plugins [:kaocha.plugin/profiling :kaocha.plugin/print-invocations :kaocha.plugin/junit-xml :kaocha.plugin/hooks :preloads] :kaocha/source-paths ["src"] :kaocha/test-paths ["test"] :ns-patterns [".*-test"]} {:id :coverage :plugins [:kaocha.plugin/profiling :kaocha.plugin/print-invocations :kaocha.plugin/junit-xml :kaocha.plugin/cloverage :kaocha.plugin/hooks :preloads] :kaocha/source-paths ["src"] :kaocha/test-paths ["test"] :ns-patterns [".*-test"]}]} ================================================ FILE: topics/Reductions.md ================================================ # Reductions The ham-fisted project extends the concept of Clojure's `reduce` in a few ways, taking influence from java streams and Clojure transducers. The most important way is a formal definition of a parallel reduction (analogous to `pmap` for `map`). Most interesting is the 3 argument form of `(reduce rfn init coll)`. Problems exist with the 2 argument form `(reduce rfn coll)` as the reduction function - `rfn`'s leftmost argument is sometimes a value from the collection and at other times an accumulated value. Some reductions have the property that the accumulator is in the set of objects in the collection (such as numeric `+`), these reductions are not the most general. They are a special case of a reduction where the accumulator may be a different type entirely than the values in the collection. ## Parallelizable Containers Efficient parallel reductions depend on parallelizable containers. Java has three types of containers that operate efficiently in parallel. 1) Finite random access containers (ex: an array) 2) Containers that can provide spliterators (ex: hashtable) 3) A concatenation of containers suitable for parallel computation over the parts These three types of containers we can parallelize; random access containers, maps and sets (or more generally anything with a correct spliterator implementation), and concatenations of sub-containers each of which may not itself have a parallelizable reduction. ## Parallelized Reduction A parallelized reduction works by splitting up elements of the data source. Many reduction contexts operate simultaneous each of which will perform a serial reduction. A separate step merges the results back together. This may be thought of as the "true" map-reduce, but either way it may be useful to compare a parallelized reduction in detail to a serial reduction. To perform a parallel reduction, four things must be provided: * `init-val-fn` - a function to produce initial accumulators for each reduction context * `rfn` - a function that takes an accumulator and a value and updates the accumulator --- This is the typical reduction function passed as the first argument to Clojure's `reduce` * `merge-fn` - a function that takes two accumulators and merges them to produces one result accumulator. * `coll` - a collection of items to reduce to a single output Here are the function signatures (Keep in mind ... `preduce`:`reduce` :: `pmap`:`map`): ```clojure (defn preduce [init-val-fn rfn merge-fn coll] ...) ``` Notably Java streams have a 'collect' method that takes the same four arguments where the collection is the `this` object: ```java interface Stream { R collect(Supplier supplier, BiConsumer accumulator, BiConsumer combiner); } ``` The parallelizable reduction operates as a a serial reduction if the init-val-fn is called exactly once with no arguments and the entire collection is passed along with rfn to reduce: ```clojure (reduce rfn (init-val-fn) coll) ``` From there `preduce` essentially switches on the type of coll and performs one of four distinct types of reductions: * serial * parallel over and index space * parallel over and spliterator space * parallel over sub-containers ## Map, Filter, Concat Chains It is common in functional programming to implement data transformations as chains of `map`, `filter`, and `concat` operations. Analyzing sequences of these operations is insight with regards to reduction in general and parallelization of reductions. The first insight is found in the Clojure transducer pathways and involves collapsing the reduction function when possible for map and filter applications. Let's start with a reduction of the form `(->> coll (map x) (filter y) (reduce ...))`. The filter operator can specialize its reduce implementation by producing a new reduction function and reducing over its source collection: ```java public Object reduce(IFn rfn, Object init) { return source.reduce(new IFn() { public Object invoke(Object acc, Object v) { if(pred.test(v)) return rfn.invoke(acc, v); else return acc; } }, init); } ``` This results in 'collapsing' the reduction allowing the source to perform the iteration across its elements and simply dynamically creating a slightly more complex reduction function, `rfn`. A similar pathway exists for `map` as we can always delegate up the chain making a slightly more complex reduction function as long as we are reducing over a single source of data. This optimization leads to many fewer function calls and intermediate collections when compared with naive implementations of `map` and `filter`. Clojure's transducers do this automatically. Collapsing the reduction also allows us to parallelize reductions like the initial one stated before as if the filter object has a parallelReduction method that does an identical collapsing pathway then if the source is parallelizable then the reduction itself can still parallelize: ```java public Object parallelReduction(IFn initValFn, IFn rfn, IFn mergeFn) { return source.parallelReduction(initValFn, new IFn() {...}, mergeFn); } ``` If the source collection itself allows for parallel reduction, then it's possible to achieve similar 'collapsing' in `preduce`. Clojure's transducers do not have this particular optimization for parallel reduction, but Java streams do. Also worth noting, these optimizations are only available if we use the 4 argument form of reduce *and* if we assume that `map`, `filter`, and `concat` are lazy and non-caching. With those assumptions in place it is possible to parallelize a reduction over the entries, keys or values of map using simple primitive composition: ```clojure user> (require '[ham-fisted.api :as hamf]) nil user> (require '[ham-fisted.lazy-noncaching :as lznc]) nil user> (def data (hamf/immut-map (lznc/map #(vector % %) (range 20000)))) #'user/data user> (type data) ham_fisted.PersistentHashMap user> (hamf/preduce + + + (lznc/map key data)) 199990000 ``` ## Stream-based Reductions Java streams have a notion of parallel reduction built-in. Their design suffers from two flaws, one minor and one major. The first minor flaw is that you can ask a stream for a parallel version of itself and it will give you one if possible else return a copy of itself. Unfortunately this only works on the first stream in a pipeline so for instance: ```java coll.stream().map().filter().parallel().collect(); ``` yields a serial reduction while: ```java coll.stream().parallel().map().filter().collect(); ``` yields a parallel reduction. This is unfortunate because it means you must go back in time to get a parallel version of the stream if you want to perform a parallel collection; something that may or may not be easily done at the point in time when you decide you do in fact want to parallel reduction (especially in library code). The second and more major flaw is that stream-based parallelization does not allow the user to pass in their own fork-join pool at any point. This limits use to the built in pool where it's pad form to park threads or do blocking operations. ## reducers.clj And Parallel Folds Clojure has an alpha namespace that provides a parallel reduction, [reducers.clj](https://github.com/clojure/clojure/blob/master/src/clj/clojure/core/reducers.clj). The signature for this method is: ```clojure (defn fold "Reduces a collection using a (potentially parallel) reduce-combine strategy. The collection is partitioned into groups of approximately n (default 512), each of which is reduced with reducef (with a seed value obtained by calling (combinef) with no arguments). The results of these reductions are then reduced with combinef (default reducef). combinef must be associative, and, when called with no arguments, (combinef) must produce its identity element. These operations may be performed in parallel, but the results will preserve order." {:added "1.5"} ([reducef coll] (fold reducef reducef coll)) ([combinef reducef coll] (fold 512 combinef reducef coll)) ([n combinef reducef coll] (coll-fold coll n combinef reducef))) ``` In this case we use overloads of `combinef` or `reducef` to provider the initial accumulator (called the identity element), the rfn, finalization and the merge function. `combinef` called with no arguments provides each thread context's accumulator and called with two arguments performs a merge of two accumulators. `reducef` called with 2 arguments provides the reduction from a value into the accumulator and when called with one argument finalizes both the potentially stateful reducing function and finalizes the accumulator. It prescribes the parallelization system but users can override a protocol to do it themselves. This the same major drawback as the java stream system, namely users cannot provide their own pool for parallelization. An interesting decision was made here as to whether one can actually parallelize the reduction or not. Transducers, the elements providing `reducef`, may be stateful such as `(take 15)`. One interesting difference is that state is done with a closure in the reduction function as opposed to providing a custom accumulator that wraps the user's accumulator but tracks state. One aspect we haven't discussed but that is also handled here in an interesting manner is that whether a reduction can be parallelized or not is a function both of the container *and* of the reducer. `reducers.clj` does a sort of double-dispatch where the transducer may choose to implement the parallel reduction, called `coll-fold` or not and is queried first and if it allows parallel reduction then the collection itself is dispatched. Overall this is a great, safe choice because it disallows completely parallel dispatch if the transducer or the collection do not support it. ## Parallel Reducers If we combine all three functions: `init-val-fn`, `rfn`, and `merge-fn` into one object then we get a ParallelReducer, defined in protocols.clj. This protocol allows the user to pass a single object into a parallelized reduction as opposed to three functions which is useful when we want to have many reducers reduce over a single source of data. A `finalize` method is added in order to allow compositions of reducers, and to allow reducers to hide state and information from end users: ```clojure (defprotocol ParallelReducer "Parallel reducers are simple a single object that you can pass into preduce as opposed to 3 separate functions." (->init-val-fn [item] "Returns the initial values for a parallel reduction. This function takes no arguments and returns the initial reduction value.") (->rfn [item] "Returns the reduction function for a parallel reduction. This function takes two arguments, the initial value and a value from the collection and returns a new initial value.") (->merge-fn [item] "Returns the merge function for a parallel reduction. This function takes two initial values and returns a new initial value.") (finalize [item v] "A finalize function called on the result of the reduction after it is reduced but before it is returned to the user. Identity is a reasonable default.")) ``` There are defaults to the reducer protocol for an IFn which assumes it can be called with no arguments for a initial value and two arguments for both reduction and merge. This works for things like `+` and `*`. Additionally there are implementations provided for the ham_fisted Sum (Kahans compensated) and SimpleSum [DoubleConsumer](https://docs.oracle.com/javase/8/docs/api/java/util/function/DoubleConsumer.html) classes. With the three functions bundled into one logical protocol or object it is easy then to create complex (aggregate) and efficient parallelized reductions: ```clojure user> (require '[ham-fisted.reduce :as hamf-rf]) nil user> (hamf-rf/preduce-reducers {:sum + :product *} (range 1 20)) {:product 121645100408832000, :sum 190} user> ``` This goes over the data in parallel, exactly once. ## Consumers, Transducers, and `rfn` Chains If we look at the reduction in terms of a push model as opposed to a pull model where the stream will push data into a consumer then we can implement similar chains or map and filter. These are based on creating a new consumer that takes the older consumer and the filter predicate or mapping function. In this way one can implement a pipeline on the input stream, or perhaps diverging pipelines on each reduction function in a multiple reducer scenario. Since the init and merge functions operate in accumulator space, which remains unchanged, one can develop up increasingly sophisticated reduction functions and still perform a parallelized reduction. Naturally, everything is composed in reverse (push instead of pull), which is the reason that `comp` works in reverse when working with transducers. In fact, given that the covers are pulled back on composing reduction functions, the definition of the single argument `clojure.core/filter` becomes more clear: ```clojure (defn filter "Returns a lazy sequence of the items in coll for which (pred item) returns logical true. pred must be free of side-effects. Returns a transducer when no collection is provided." {:added "1.0" :static true} ([pred] (fn [rf] (fn ([] (rf)) ([result] (rf result)) ([result input] (if (pred input) (rf result input) result))))))) ``` It returns a function that, when given a reduction function, returns a new reduction function that when called in the two argument form is identical to the result above (although expressed in pure Clojure as opposed to Java). Starting with the concept that a reduction begins at the collection, flows downward through the pipeline and bottoms out at the reducer then the lazy-noncaching namespace and Java streams implement parallelization flowing from the container downward. Separately consumer chains and transducers implement the pipeline flowing up from the reducer itself. Thus building the pipeline either downward from the source or upward from the final reduction produces subtly different properties. Regardless, every system must disable parallelization where it will cause an incorrect answer (to ensure correctness) - such as in a stateful transducer. Broadly speaking, however, it can be faster to enable full parallelization and filter invalid results than it is to force an early serialization our problem and thus lose lots of our parallelization potential. When concerned with performance, attempt to move transformations as much as possible into a parallelizable domain. For the `take-n` use case specifically mentioned above and potentially for others we can parallelize the reduction and do the take-n both in the parallelized phase and in the merge phase assuming we are using an ordered parallelization, so that doesn't itself necessarily force a serialized reduction but there are of course transformations and reductions that do. There are intermediate points however that are perhaps somewhat wasteful in terms of cpu load but do allow for more parallelization - a tradeoff that is sometimes worth it. Generically speaking we can visualize this sort of a tradeoff as triangle of three points where one point is data locality, one point parallelism, and one point redundancy. Specifically if we are willing to trade some cpu efficiency for some redundancy, for instance, then we often get more parallelization. Likewise if we are willing to save/load data from 'far' away from the CPU, then we can cut down on redundancy but at the cost of locality. For more on this line of thinking please take a moment and read at least some of Jonathan Ragan-Kelly's [excellent PhD thesis](http://people.csail.mit.edu/jrk/jrkthesis.pdf) - a better explanation of the above line of reasoning begins on page 20. ## Primitive Typed Serial Reductions This comes last for a good reason :-) - it doesn't make a huge difference in performance but it should be noted allowing objects to implemented typed reductions: ```java default Object doubleReduction(IFn.ODO op, Object init); default Object longReduction(IFn.OLO op, Object init) ``` where the next incoming value is a primitive object but the accumulator is still a generic object allows us to use things like `DoubleConsumers` and `LongConsumers` and avoid boxing the stream. Furthermore if the aforementioned `map` and `filter` primitives are careful about their rfn composition we can maintain a completely primitive typed pipeline through an entire processing chain. ## One Final Note About Performance Collapsing reductions brings the source iteration pathway closer to the final reduction pathway in terms of machine stack space which allows HotSpot to optimize the entire chain more readily. Regardless of how good HotSpot gets, however, parallelizing will nearly always result in a larger win but both work together to enable peak performance on the JVM given arbitrary partially typed compositions of sequences and reductions. When increasing the data size yet again, one can of course use the same design to distribute the computations to different machines. As some people have figured out, however, simply implementing the transformations you need efficiently reduces or completely eliminates the need to distribute computation in the first place leading to a simpler, easier to test and more robust system. Ideally we can make achieving great performance for various algorithms clear and easy and thus avoid myriad of issues regarding distributing computing in the first place. * The first rule of distributed systems is to avoid distributing your computation in the first place - [1](https://bravenewgeek.com/service-disoriented-architecture/), [2](https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing). * The first law of distributed objects is to avoid using distributed objects. [3](https://martinfowler.com/bliki/FirstLaw.html).